2024-02-12T15:36:54.838 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-12T15:36:54.839 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633 --verbose --description rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} --name yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/orig.config.yaml 2024-02-12T15:36:54.878 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-12T15:36:54.949 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} email: null first_in_suite: false job_id: '7556633' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd backoff on peering: true osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd max backfills: 3 osd max markdown count: 1000 osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - \(MON_DOWN\) - osds down - mons down mon_bind_msgr2: false sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 workunit: branch: wip-tracker-64343-pacific sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 owner: scheduled_yuriw@teuthology priority: 60 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE= tasks: - install: branch: luminous exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-immutable-object-cache - ceph-base - python3-rados - python3-rgw - python3-rbd - python3-cephfs - librados-devel extra_packages: - librados2 - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-12_15:26:58 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-12T15:36:54.949 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa; will attempt to use it 2024-02-12T15:36:54.949 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks 2024-02-12T15:36:54.949 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-12T15:36:54.951 INFO:teuthology.task.internal:Saving configuration 2024-02-12T15:36:54.965 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-12T15:36:54.981 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T15:36:55.005 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 15:29:11.215556', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4U6ItCfOdoN1WdROC3KO40lcw/kzXeIM8M8q950kfL'} 2024-02-12T15:36:55.028 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 15:29:11.214572', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4U6ItCfOdoN1WdROC3KO40lcw/kzXeIM8M8q950kfL'} 2024-02-12T15:36:55.048 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 15:29:11.213430', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4U6ItCfOdoN1WdROC3KO40lcw/kzXeIM8M8q950kfL'} 2024-02-12T15:36:55.048 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-12T15:36:55.060 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-02-12T15:36:55.061 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-02-12T15:36:55.061 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-02-12T15:36:55.061 INFO:teuthology.run_tasks:Running task console_log... 2024-02-12T15:36:55.174 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f92f3b85e50>, signals=[15]) 2024-02-12T15:36:55.175 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-12T15:36:55.185 INFO:teuthology.task.internal:Opening connections... 2024-02-12T15:36:55.185 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-02-12T15:36:55.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:36:55.269 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-02-12T15:36:55.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:36:55.346 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-02-12T15:36:55.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:36:55.416 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-12T15:36:55.426 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-02-12T15:36:55.433 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-02-12T15:36:55.433 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-02-12T15:36:55.479 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Ubuntu" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:ID=ubuntu 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE=debian 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="18.04" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_CODENAME=bionic 2024-02-12T15:36:55.480 INFO:teuthology.orchestra.run.smithi121.stdout:UBUNTU_CODENAME=bionic 2024-02-12T15:36:55.481 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-02-12T15:36:55.500 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-02-12T15:36:55.506 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-02-12T15:36:55.506 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="18.04" 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-12T15:36:55.555 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-12T15:36:55.556 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-12T15:36:55.556 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=bionic 2024-02-12T15:36:55.556 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=bionic 2024-02-12T15:36:55.556 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-02-12T15:36:55.574 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-02-12T15:36:55.581 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-02-12T15:36:55.581 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="Ubuntu" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:ID=ubuntu 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE=debian 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="18.04" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_CODENAME=bionic 2024-02-12T15:36:55.630 INFO:teuthology.orchestra.run.smithi169.stdout:UBUNTU_CODENAME=bionic 2024-02-12T15:36:55.631 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-02-12T15:36:55.649 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-12T15:36:55.658 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-12T15:36:55.685 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-12T15:36:55.685 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-02-12T15:36:55.687 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-02-12T15:36:55.689 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-02-12T15:36:55.695 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-12T15:36:55.732 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-12T15:36:55.732 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-02-12T15:36:55.736 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-02-12T15:36:55.738 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-02-12T15:36:55.836 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-12T15:36:55.912 INFO:teuthology.run_tasks:Running task kernel... 2024-02-12T15:36:55.925 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-12T15:36:55.925 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-12T15:36:55.925 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-12T15:36:55.926 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-12T15:36:55.926 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-12T15:36:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T15:36:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T15:36:55.932 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-02-12T15:36:55.933 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-02-12T15:36:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T15:36:55.934 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-02-12T15:36:55.979 INFO:teuthology.orchestra.run.smithi123.stdout:4.15.0-200-generic 2024-02-12T15:36:55.979 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.15.0-200-generic 2024-02-12T15:36:55.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2024-02-12T15:36:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:4.15.0-200-generic 2024-02-12T15:36:55.980 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.15.0-200-generic 2024-02-12T15:36:55.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2024-02-12T15:36:55.982 INFO:teuthology.orchestra.run.smithi169.stdout:4.15.0-200-generic 2024-02-12T15:36:55.982 INFO:teuthology.task.kernel:Running kernel on smithi169: 4.15.0-200-generic 2024-02-12T15:36:55.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get clean 2024-02-12T15:36:56.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2024-02-12T15:36:56.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2024-02-12T15:36:56.086 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-02-12T15:36:56.473 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2024-02-12T15:36:56.473 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2024-02-12T15:36:56.527 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:36:56.528 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:36:56.528 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2024-02-12T15:36:56.564 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:36:56.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2024-02-12T15:36:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2024-02-12T15:36:56.592 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2024-02-12T15:36:56.865 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [1,379 kB] 2024-02-12T15:36:56.894 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [1,379 kB] 2024-02-12T15:36:56.914 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB] 2024-02-12T15:36:56.918 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB] 2024-02-12T15:36:56.923 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB] 2024-02-12T15:36:56.988 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,717 kB] 2024-02-12T15:36:57.047 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,717 kB] 2024-02-12T15:36:57.056 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,045 kB] 2024-02-12T15:36:57.083 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,045 kB] 2024-02-12T15:36:57.088 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,717 kB] 2024-02-12T15:36:57.119 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,045 kB] 2024-02-12T15:36:57.122 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [467 kB] 2024-02-12T15:36:57.130 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1,317 kB] 2024-02-12T15:36:57.156 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [33.0 kB] 2024-02-12T15:36:57.157 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [182 kB] 2024-02-12T15:36:57.162 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [1,078 kB] 2024-02-12T15:36:57.170 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [467 kB] 2024-02-12T15:36:57.189 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,303 kB] 2024-02-12T15:36:57.256 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [33.0 kB] 2024-02-12T15:36:57.257 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1,317 kB] 2024-02-12T15:36:57.273 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [308 kB] 2024-02-12T15:36:57.277 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [182 kB] 2024-02-12T15:36:57.279 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [1,078 kB] 2024-02-12T15:36:57.287 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,008 B] 2024-02-12T15:36:57.287 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.8 kB] 2024-02-12T15:36:57.289 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,303 kB] 2024-02-12T15:36:57.293 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [3,928 B] 2024-02-12T15:36:57.318 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [308 kB] 2024-02-12T15:36:57.327 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.8 kB] 2024-02-12T15:36:57.328 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,008 B] 2024-02-12T15:36:57.328 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [3,928 B] 2024-02-12T15:36:57.480 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,665 kB] 2024-02-12T15:36:57.525 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [1,379 kB] 2024-02-12T15:36:57.536 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [554 kB] 2024-02-12T15:36:57.549 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [39.7 kB] 2024-02-12T15:36:57.555 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [467 kB] 2024-02-12T15:36:57.562 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1,347 kB] 2024-02-12T15:36:57.564 INFO:teuthology.orchestra.run.smithi169.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [33.0 kB] 2024-02-12T15:36:57.565 INFO:teuthology.orchestra.run.smithi169.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1,317 kB] 2024-02-12T15:36:57.591 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [187 kB] 2024-02-12T15:36:57.595 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,663 kB] 2024-02-12T15:36:57.597 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,665 kB] 2024-02-12T15:36:57.616 INFO:teuthology.orchestra.run.smithi169.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [182 kB] 2024-02-12T15:36:57.620 INFO:teuthology.orchestra.run.smithi169.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [1,078 kB] 2024-02-12T15:36:57.629 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [554 kB] 2024-02-12T15:36:57.631 INFO:teuthology.orchestra.run.smithi169.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,665 kB] 2024-02-12T15:36:57.637 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1,347 kB] 2024-02-12T15:36:57.646 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,915 kB] 2024-02-12T15:36:57.646 INFO:teuthology.orchestra.run.smithi169.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,303 kB] 2024-02-12T15:36:57.666 INFO:teuthology.orchestra.run.smithi169.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [554 kB] 2024-02-12T15:36:57.672 INFO:teuthology.orchestra.run.smithi169.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [308 kB] 2024-02-12T15:36:57.680 INFO:teuthology.orchestra.run.smithi169.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1,347 kB] 2024-02-12T15:36:57.687 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [39.7 kB] 2024-02-12T15:36:57.688 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [187 kB] 2024-02-12T15:36:57.693 INFO:teuthology.orchestra.run.smithi169.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.8 kB] 2024-02-12T15:36:57.698 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,915 kB] 2024-02-12T15:36:57.725 INFO:teuthology.orchestra.run.smithi169.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [39.7 kB] 2024-02-12T15:36:57.726 INFO:teuthology.orchestra.run.smithi169.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [187 kB] 2024-02-12T15:36:57.734 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,663 kB] 2024-02-12T15:36:57.736 INFO:teuthology.orchestra.run.smithi169.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,915 kB] 2024-02-12T15:36:57.772 INFO:teuthology.orchestra.run.smithi169.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,663 kB] 2024-02-12T15:36:57.849 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [421 kB] 2024-02-12T15:36:57.856 INFO:teuthology.orchestra.run.smithi169.stdout:Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,008 B] 2024-02-12T15:36:57.938 INFO:teuthology.orchestra.run.smithi169.stdout:Get:24 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [3,928 B] 2024-02-12T15:36:57.958 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [421 kB] 2024-02-12T15:36:57.997 INFO:teuthology.orchestra.run.smithi169.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [421 kB] 2024-02-12T15:36:58.365 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [25.6 kB] 2024-02-12T15:36:58.369 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.2 kB] 2024-02-12T15:36:58.370 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,088 B] 2024-02-12T15:36:58.371 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/main i386 Packages [53.2 kB] 2024-02-12T15:36:58.378 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [53.3 kB] 2024-02-12T15:36:58.386 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [18.1 kB] 2024-02-12T15:36:58.388 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [18.2 kB] 2024-02-12T15:36:58.470 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.2 kB] 2024-02-12T15:36:58.472 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [25.6 kB] 2024-02-12T15:36:58.475 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,088 B] 2024-02-12T15:36:58.476 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/main i386 Packages [53.2 kB] 2024-02-12T15:36:58.483 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [53.3 kB] 2024-02-12T15:36:58.491 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [18.1 kB] 2024-02-12T15:36:58.493 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [18.2 kB] 2024-02-12T15:36:58.521 INFO:teuthology.orchestra.run.smithi169.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.2 kB] 2024-02-12T15:36:58.522 INFO:teuthology.orchestra.run.smithi169.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [25.6 kB] 2024-02-12T15:36:58.525 INFO:teuthology.orchestra.run.smithi169.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,088 B] 2024-02-12T15:36:58.526 INFO:teuthology.orchestra.run.smithi169.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/main i386 Packages [53.2 kB] 2024-02-12T15:36:58.534 INFO:teuthology.orchestra.run.smithi169.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [53.3 kB] 2024-02-12T15:36:58.541 INFO:teuthology.orchestra.run.smithi169.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [18.2 kB] 2024-02-12T15:36:58.544 INFO:teuthology.orchestra.run.smithi169.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [18.1 kB] 2024-02-12T15:36:59.828 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 20.1 MB in 3s (5,747 kB/s) 2024-02-12T15:36:59.863 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 20.1 MB in 4s (5,687 kB/s) 2024-02-12T15:37:00.309 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 20.1 MB in 4s (5,193 kB/s) 2024-02-12T15:37:01.518 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-02-12T15:37:01.532 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-12T15:37:01.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-12T15:37:01.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-12T15:37:01.657 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-02-12T15:37:01.665 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-12T15:37:01.936 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2024-02-12T15:37:01.937 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2024-02-12T15:37:01.939 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-02-12T15:37:01.939 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-02-12T15:37:02.058 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2024-02-12T15:37:02.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-12T15:37:02.155 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2024-02-12T15:37:02.155 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2024-02-12T15:37:02.155 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-generic linux-image-4.15.0-213-generic 2024-02-12T15:37:02.156 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-4.15.0-213-generic linux-modules-extra-4.15.0-213-generic 2024-02-12T15:37:02.157 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2024-02-12T15:37:02.157 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2024-02-12T15:37:02.157 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-generic linux-image-4.15.0-213-generic 2024-02-12T15:37:02.158 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2024-02-12T15:37:02.158 INFO:teuthology.orchestra.run.smithi121.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2024-02-12T15:37:02.158 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2024-02-12T15:37:02.158 INFO:teuthology.orchestra.run.smithi121.stdout: thermald 2024-02-12T15:37:02.158 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-4.15.0-213-generic linux-modules-extra-4.15.0-213-generic 2024-02-12T15:37:02.160 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2024-02-12T15:37:02.160 INFO:teuthology.orchestra.run.smithi123.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2024-02-12T15:37:02.160 INFO:teuthology.orchestra.run.smithi123.stdout:Recommended packages: 2024-02-12T15:37:02.160 INFO:teuthology.orchestra.run.smithi123.stdout: thermald 2024-02-12T15:37:02.193 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2024-02-12T15:37:02.227 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2024-02-12T15:37:02.227 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2024-02-12T15:37:02.227 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-4.15.0-213-generic linux-modules-4.15.0-213-generic 2024-02-12T15:37:02.228 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-extra-4.15.0-213-generic 2024-02-12T15:37:02.230 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages will be upgraded: 2024-02-12T15:37:02.231 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-12T15:37:02.238 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2024-02-12T15:37:02.238 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2024-02-12T15:37:02.238 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-4.15.0-213-generic linux-modules-4.15.0-213-generic 2024-02-12T15:37:02.239 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-4.15.0-213-generic 2024-02-12T15:37:02.240 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2024-02-12T15:37:02.241 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-12T15:37:02.305 INFO:teuthology.orchestra.run.smithi121.stdout:3 upgraded, 5 newly installed, 0 to remove and 165 not upgraded. 2024-02-12T15:37:02.305 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 67.4 MB of archives. 2024-02-12T15:37:02.305 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 351 MB of additional disk space will be used. 2024-02-12T15:37:02.305 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-213-generic amd64 4.15.0-213.224 [13.4 MB] 2024-02-12T15:37:02.414 INFO:teuthology.orchestra.run.smithi123.stdout:3 upgraded, 5 newly installed, 0 to remove and 165 not upgraded. 2024-02-12T15:37:02.415 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 67.4 MB of archives. 2024-02-12T15:37:02.415 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 351 MB of additional disk space will be used. 2024-02-12T15:37:02.415 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-213-generic amd64 4.15.0-213.224 [13.4 MB] 2024-02-12T15:37:02.487 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2024-02-12T15:37:02.488 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2024-02-12T15:37:02.693 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-213-generic amd64 4.15.0-213.224 [8,111 kB] 2024-02-12T15:37:02.709 INFO:teuthology.orchestra.run.smithi169.stdout:The following additional packages will be installed: 2024-02-12T15:37:02.709 INFO:teuthology.orchestra.run.smithi169.stdout: linux-generic linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2024-02-12T15:37:02.709 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-generic linux-image-4.15.0-213-generic 2024-02-12T15:37:02.710 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-4.15.0-213-generic linux-modules-extra-4.15.0-213-generic 2024-02-12T15:37:02.712 INFO:teuthology.orchestra.run.smithi169.stdout:Suggested packages: 2024-02-12T15:37:02.712 INFO:teuthology.orchestra.run.smithi169.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2024-02-12T15:37:02.712 INFO:teuthology.orchestra.run.smithi169.stdout:Recommended packages: 2024-02-12T15:37:02.712 INFO:teuthology.orchestra.run.smithi169.stdout: thermald 2024-02-12T15:37:02.785 INFO:teuthology.orchestra.run.smithi169.stdout:The following NEW packages will be installed: 2024-02-12T15:37:02.785 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-4.15.0-213 linux-headers-4.15.0-213-generic 2024-02-12T15:37:02.785 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-4.15.0-213-generic linux-modules-4.15.0-213-generic 2024-02-12T15:37:02.785 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-extra-4.15.0-213-generic 2024-02-12T15:37:02.787 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages will be upgraded: 2024-02-12T15:37:02.788 INFO:teuthology.orchestra.run.smithi169.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-12T15:37:02.822 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-213-generic amd64 4.15.0-213.224 [33.7 MB] 2024-02-12T15:37:02.973 INFO:teuthology.orchestra.run.smithi169.stdout:3 upgraded, 5 newly installed, 0 to remove and 165 not upgraded. 2024-02-12T15:37:02.973 INFO:teuthology.orchestra.run.smithi169.stdout:Need to get 67.4 MB of archives. 2024-02-12T15:37:02.973 INFO:teuthology.orchestra.run.smithi169.stdout:After this operation, 351 MB of additional disk space will be used. 2024-02-12T15:37:02.973 INFO:teuthology.orchestra.run.smithi169.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-213-generic amd64 4.15.0-213.224 [13.4 MB] 2024-02-12T15:37:03.385 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.213.196 [1,864 B] 2024-02-12T15:37:03.385 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.213.196 [2,460 B] 2024-02-12T15:37:03.385 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213 all 4.15.0-213.224 [11.0 MB] 2024-02-12T15:37:03.468 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-213-generic amd64 4.15.0-213.224 [8,111 kB] 2024-02-12T15:37:03.549 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213-generic amd64 4.15.0-213.224 [1,250 kB] 2024-02-12T15:37:03.565 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.213.196 [2,360 B] 2024-02-12T15:37:03.716 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-213-generic amd64 4.15.0-213.224 [33.7 MB] 2024-02-12T15:37:04.019 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-213-generic amd64 4.15.0-213.224 [8,111 kB] 2024-02-12T15:37:04.220 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-213-generic amd64 4.15.0-213.224 [33.7 MB] 2024-02-12T15:37:04.338 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 67.4 MB in 1s (51.7 MB/s) 2024-02-12T15:37:04.503 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-4.15.0-213-generic. 2024-02-12T15:37:04.806 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.213.196 [1,864 B] 2024-02-12T15:37:04.806 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.213.196 [2,460 B] 2024-02-12T15:37:04.806 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213 all 4.15.0-213.224 [11.0 MB] 2024-02-12T15:37:05.105 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213-generic amd64 4.15.0-213.224 [1,250 kB] 2024-02-12T15:37:05.130 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.213.196 [2,360 B] 2024-02-12T15:37:05.276 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.213.196 [1,864 B] 2024-02-12T15:37:05.277 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.213.196 [2,460 B] 2024-02-12T15:37:05.277 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213 all 4.15.0-213.224 [11.0 MB] 2024-02-12T15:37:05.474 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 135963 files and directories currently installed.) 2024-02-12T15:37:05.478 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../0-linux-modules-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:05.566 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:05.599 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-213-generic amd64 4.15.0-213.224 [1,250 kB] 2024-02-12T15:37:05.626 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.213.196 [2,360 B] 2024-02-12T15:37:05.806 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 67.4 MB in 3s (23.4 MB/s) 2024-02-12T15:37:05.985 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-4.15.0-213-generic. 2024-02-12T15:37:06.233 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 67.4 MB in 3s (23.9 MB/s) 2024-02-12T15:37:06.435 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-modules-4.15.0-213-generic. 2024-02-12T15:37:06.917 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 135963 files and directories currently installed.) 2024-02-12T15:37:06.921 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../0-linux-modules-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:06.964 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:07.409 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-4.15.0-213-generic. 2024-02-12T15:37:07.413 INFO:teuthology.orchestra.run.smithi169.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 135963 files and directories currently installed.) 2024-02-12T15:37:07.418 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../0-linux-modules-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:07.423 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../1-linux-image-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:07.482 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:07.526 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:08.095 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-213-generic. 2024-02-12T15:37:08.108 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:08.145 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:08.592 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-image-4.15.0-213-generic. 2024-02-12T15:37:08.606 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../1-linux-image-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:08.689 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:09.166 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-image-4.15.0-213-generic. 2024-02-12T15:37:09.183 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../1-linux-image-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:09.194 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-213-generic. 2024-02-12T15:37:09.208 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:09.244 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:09.281 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:09.836 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-213-generic. 2024-02-12T15:37:09.852 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:09.894 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:11.992 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../3-linux-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:12.117 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:12.402 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:12.512 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:12.873 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-4.15.0-213. 2024-02-12T15:37:12.887 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../5-linux-headers-4.15.0-213_4.15.0-213.224_all.deb ... 2024-02-12T15:37:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-4.15.0-213 (4.15.0-213.224) ... 2024-02-12T15:37:13.035 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../3-linux-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:13.145 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:13.456 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:13.582 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:13.714 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../3-linux-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:13.824 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:13.926 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-4.15.0-213. 2024-02-12T15:37:13.940 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../5-linux-headers-4.15.0-213_4.15.0-213.224_all.deb ... 2024-02-12T15:37:13.985 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-4.15.0-213 (4.15.0-213.224) ... 2024-02-12T15:37:14.094 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:14.219 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-image-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:14.531 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-headers-4.15.0-213. 2024-02-12T15:37:14.548 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../5-linux-headers-4.15.0-213_4.15.0-213.224_all.deb ... 2024-02-12T15:37:14.589 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-4.15.0-213 (4.15.0-213.224) ... 2024-02-12T15:37:17.521 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-4.15.0-213-generic. 2024-02-12T15:37:17.544 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../6-linux-headers-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:17.581 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:18.646 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-4.15.0-213-generic. 2024-02-12T15:37:18.670 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../6-linux-headers-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:18.713 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:19.261 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-headers-4.15.0-213-generic. 2024-02-12T15:37:19.287 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../6-linux-headers-4.15.0-213-generic_4.15.0-213.224_amd64.deb ... 2024-02-12T15:37:19.328 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:20.028 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:20.154 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:20.479 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:21.113 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:21.117 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:21.233 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:21.550 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:21.586 INFO:teuthology.orchestra.run.smithi121.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:21.586 INFO:teuthology.orchestra.run.smithi121.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:21.586 INFO:teuthology.orchestra.run.smithi121.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:21.586 INFO:teuthology.orchestra.run.smithi121.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:21.703 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:21.718 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.213.196_amd64.deb ... 2024-02-12T15:37:21.844 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-generic (4.15.0.213.196) over (4.15.0.200.183) ... 2024-02-12T15:37:22.161 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-modules-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:22.230 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:22.262 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-4.15.0-213 (4.15.0-213.224) ... 2024-02-12T15:37:22.388 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-generic (4.15.0.213.196) ... 2024-02-12T15:37:22.514 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:22.641 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-generic (4.15.0.213.196) ... 2024-02-12T15:37:22.719 INFO:teuthology.orchestra.run.smithi123.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:22.719 INFO:teuthology.orchestra.run.smithi123.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:22.719 INFO:teuthology.orchestra.run.smithi123.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:22.719 INFO:teuthology.orchestra.run.smithi123.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:22.766 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-generic (4.15.0.213.196) ... 2024-02-12T15:37:22.816 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:22.848 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:22.893 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:23.112 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-12T15:37:23.112 INFO:teuthology.orchestra.run.smithi121.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:23.313 INFO:teuthology.orchestra.run.smithi169.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:23.313 INFO:teuthology.orchestra.run.smithi169.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:23.313 INFO:teuthology.orchestra.run.smithi169.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:23.313 INFO:teuthology.orchestra.run.smithi169.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:23.408 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-4.15.0-213 (4.15.0-213.224) ... 2024-02-12T15:37:23.484 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-modules-extra-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:23.543 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-generic (4.15.0.213.196) ... 2024-02-12T15:37:23.669 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:23.795 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-generic (4.15.0.213.196) ... 2024-02-12T15:37:23.921 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-generic (4.15.0.213.196) ... 2024-02-12T15:37:24.039 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:24.104 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-4.15.0-213 (4.15.0-213.224) ... 2024-02-12T15:37:24.230 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-image-generic (4.15.0.213.196) ... 2024-02-12T15:37:24.254 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-12T15:37:24.255 INFO:teuthology.orchestra.run.smithi123.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:24.356 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:24.482 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-generic (4.15.0.213.196) ... 2024-02-12T15:37:24.608 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-generic (4.15.0.213.196) ... 2024-02-12T15:37:24.734 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for linux-image-4.15.0-213-generic (4.15.0-213.224) ... 2024-02-12T15:37:24.933 INFO:teuthology.orchestra.run.smithi169.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-12T15:37:24.933 INFO:teuthology.orchestra.run.smithi169.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:39.844 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-12T15:37:39.844 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub' 2024-02-12T15:37:39.865 INFO:teuthology.orchestra.run.smithi121.stdout:Generating grub configuration file ... 2024-02-12T15:37:40.255 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:40.296 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:40.696 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:40.734 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:40.807 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2024-02-12T15:37:40.831 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2024-02-12T15:37:41.719 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-12T15:37:41.720 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub' 2024-02-12T15:37:41.747 INFO:teuthology.orchestra.run.smithi123.stdout:Generating grub configuration file ... 2024-02-12T15:37:41.758 INFO:teuthology.orchestra.run.smithi121.stdout:done 2024-02-12T15:37:41.789 INFO:teuthology.orchestra.run.smithi169.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-12T15:37:41.789 INFO:teuthology.orchestra.run.smithi169.stdout:Sourcing file `/etc/default/grub' 2024-02-12T15:37:41.816 INFO:teuthology.orchestra.run.smithi169.stdout:Generating grub configuration file ... 2024-02-12T15:37:42.109 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:42.154 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:42.207 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:42.227 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2024-02-12T15:37:42.237 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:42.248 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2024-02-12T15:37:42.248 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2024-02-12T15:37:42.248 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2024-02-12T15:37:42.248 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2024-02-12T15:37:42.248 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 18 2024-02-12T15:37:42.248 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2024-02-12T15:37:42.248 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 4.15.0.213.196 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2024-02-12T15:37:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2024-02-12T15:37:42.249 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-200-generic Expected: 4.15.0-213-generic 2024-02-12T15:37:42.249 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-12T15:37:42.249 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-12T15:37:42.250 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-12T15:37:42.250 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-02-12T15:37:42.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2024-02-12T15:37:42.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2024-02-12T15:37:42.488 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:42.501 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:37:42.513 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2024-02-12T15:37:42.521 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:42.533 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2024-02-12T15:37:42.593 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2024-02-12T15:37:42.614 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2024-02-12T15:37:42.615 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2024-02-12T15:37:42.629 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:42.659 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:42.731 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2024-02-12T15:37:42.759 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2024-02-12T15:37:43.686 INFO:teuthology.orchestra.run.smithi169.stdout:done 2024-02-12T15:37:44.092 INFO:teuthology.orchestra.run.smithi123.stdout:done 2024-02-12T15:37:44.160 DEBUG:teuthology.orchestra.run.smithi169:> dpkg -s linux-image-generic 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Package: linux-image-generic 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Status: install ok installed 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Priority: optional 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Section: kernel 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Installed-Size: 18 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Maintainer: Ubuntu Kernel Team 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: amd64 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Source: linux-meta 2024-02-12T15:37:44.182 INFO:teuthology.orchestra.run.smithi169.stdout:Version: 4.15.0.213.196 2024-02-12T15:37:44.183 INFO:teuthology.orchestra.run.smithi169.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2024-02-12T15:37:44.183 INFO:teuthology.orchestra.run.smithi169.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-12T15:37:44.183 INFO:teuthology.orchestra.run.smithi169.stdout:Recommends: thermald 2024-02-12T15:37:44.183 INFO:teuthology.orchestra.run.smithi169.stdout:Description: Generic Linux kernel image 2024-02-12T15:37:44.183 INFO:teuthology.orchestra.run.smithi169.stdout: This package will always depend on the latest generic kernel image 2024-02-12T15:37:44.183 INFO:teuthology.orchestra.run.smithi169.stdout: available. 2024-02-12T15:37:44.183 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-200-generic Expected: 4.15.0-213-generic 2024-02-12T15:37:44.183 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-12T15:37:44.183 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-12T15:37:44.183 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-02-12T15:37:44.183 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-02-12T15:37:44.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get clean 2024-02-12T15:37:44.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2024-02-12T15:37:44.413 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2024-02-12T15:37:44.413 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:37:44.425 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2024-02-12T15:37:44.453 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2024-02-12T15:37:44.568 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2024-02-12T15:37:44.575 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 18 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 4.15.0.213.196 2024-02-12T15:37:44.588 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2024-02-12T15:37:44.589 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-12T15:37:44.589 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2024-02-12T15:37:44.589 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2024-02-12T15:37:44.589 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2024-02-12T15:37:44.589 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2024-02-12T15:37:44.589 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-200-generic Expected: 4.15.0-213-generic 2024-02-12T15:37:44.589 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-12T15:37:44.589 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-12T15:37:44.589 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-12T15:37:44.589 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2024-02-12T15:37:44.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2024-02-12T15:37:44.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-12T15:37:44.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-02-12T15:37:44.707 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-12T15:37:44.840 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2024-02-12T15:37:44.840 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:37:44.852 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2024-02-12T15:37:44.876 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2024-02-12T15:37:44.993 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2024-02-12T15:37:44.993 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2024-02-12T15:37:45.213 INFO:teuthology.orchestra.run.smithi121.stdout:linux-image-generic is already the newest version (4.15.0.213.196). 2024-02-12T15:37:45.214 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2024-02-12T15:37:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2024-02-12T15:37:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2024-02-12T15:37:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-4.15.0-194-generic 2024-02-12T15:37:45.214 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-12T15:37:45.267 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 0 newly installed, 0 to remove and 165 not upgraded. 2024-02-12T15:37:45.269 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2024-02-12T15:37:45.291 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2024-02-12T15:37:45.291 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2024-02-12T15:37:45.291 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2024-02-12T15:37:45.291 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2024-02-12T15:37:45.291 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 18 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 4.15.0.213.196 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2024-02-12T15:37:45.292 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2024-02-12T15:37:45.293 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-02-12T15:37:45.340 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.ZFFvWInuNN 2024-02-12T15:37:45.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ZFFvWInuNN 2024-02-12T15:37:45.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.ZFFvWInuNN 2024-02-12T15:37:45.540 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.ZFFvWInuNN is 10KB 2024-02-12T15:37:45.551 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.ZFFvWInuNN 2024-02-12T15:37:45.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-12T15:37:45.613 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:37:45.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-12T15:37:45.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-12T15:37:45.697 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-213-generic 2024-02-12T15:37:45.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo update-grub 2024-02-12T15:37:46.501 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2024-02-12T15:37:46.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-12T15:37:46.625 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2024-02-12T15:37:46.889 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2024-02-12T15:37:46.890 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2024-02-12T15:37:46.970 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-02-12T15:37:47.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-12T15:37:47.097 INFO:teuthology.orchestra.run.smithi169.stdout:linux-image-generic is already the newest version (4.15.0.213.196). 2024-02-12T15:37:47.097 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages were automatically installed and are no longer required: 2024-02-12T15:37:47.097 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2024-02-12T15:37:47.097 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2024-02-12T15:37:47.098 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-extra-4.15.0-194-generic 2024-02-12T15:37:47.098 INFO:teuthology.orchestra.run.smithi169.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-12T15:37:47.101 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-02-12T15:37:47.154 INFO:teuthology.orchestra.run.smithi169.stdout:0 upgraded, 0 newly installed, 0 to remove and 165 not upgraded. 2024-02-12T15:37:47.156 DEBUG:teuthology.orchestra.run.smithi169:> dpkg -s linux-image-generic 2024-02-12T15:37:47.205 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub' 2024-02-12T15:37:47.216 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-02-12T15:37:47.224 INFO:teuthology.orchestra.run.smithi169.stdout:Package: linux-image-generic 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Status: install ok installed 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Priority: optional 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Section: kernel 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Installed-Size: 18 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Maintainer: Ubuntu Kernel Team 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: amd64 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Source: linux-meta 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Version: 4.15.0.213.196 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Recommends: thermald 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout:Description: Generic Linux kernel image 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout: This package will always depend on the latest generic kernel image 2024-02-12T15:37:47.225 INFO:teuthology.orchestra.run.smithi169.stdout: available. 2024-02-12T15:37:47.226 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-02-12T15:37:47.274 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.1uGa9cRfT9 2024-02-12T15:37:47.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub/grub.cfg /tmp/tmp.1uGa9cRfT9 2024-02-12T15:37:47.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.1uGa9cRfT9 2024-02-12T15:37:47.377 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-02-12T15:37:47.377 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-02-12T15:37:47.458 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:47.471 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:47.479 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.1uGa9cRfT9 is 10KB 2024-02-12T15:37:47.530 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.1uGa9cRfT9 2024-02-12T15:37:47.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-12T15:37:47.586 INFO:teuthology.orchestra.run.smithi123.stdout:linux-image-generic is already the newest version (4.15.0.213.196). 2024-02-12T15:37:47.586 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-02-12T15:37:47.586 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2024-02-12T15:37:47.586 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2024-02-12T15:37:47.586 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-extra-4.15.0-194-generic 2024-02-12T15:37:47.586 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-12T15:37:47.595 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:37:47.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-12T15:37:47.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-12T15:37:47.643 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 0 newly installed, 0 to remove and 165 not upgraded. 2024-02-12T15:37:47.645 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 18 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 4.15.0.213.196 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.4), zfs-modules (= 0.7.5-1ubuntu16.12) 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-4.15.0-213-generic, linux-modules-extra-4.15.0-213-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2024-02-12T15:37:47.667 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2024-02-12T15:37:47.668 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2024-02-12T15:37:47.668 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-02-12T15:37:47.674 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-213-generic 2024-02-12T15:37:47.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo update-grub 2024-02-12T15:37:47.715 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.g39QwOYWgE 2024-02-12T15:37:47.715 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub/grub.cfg /tmp/tmp.g39QwOYWgE 2024-02-12T15:37:47.768 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:47.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.g39QwOYWgE 2024-02-12T15:37:47.779 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:47.854 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2024-02-12T15:37:47.868 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2024-02-12T15:37:47.902 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.g39QwOYWgE is 10KB 2024-02-12T15:37:47.913 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.g39QwOYWgE 2024-02-12T15:37:47.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-12T15:37:47.983 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:37:47.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-12T15:37:47.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-12T15:37:48.062 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-213-generic 2024-02-12T15:37:48.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo update-grub 2024-02-12T15:37:48.490 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-02-12T15:37:48.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-02-12T15:37:49.173 INFO:teuthology.orchestra.run.smithi169.stderr:Sourcing file `/etc/default/grub' 2024-02-12T15:37:49.184 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-02-12T15:37:49.438 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub' 2024-02-12T15:37:49.450 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-02-12T15:37:49.460 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:49.471 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:49.717 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-4.15.0-213-generic 2024-02-12T15:37:49.729 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-4.15.0-213-generic 2024-02-12T15:37:49.783 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:49.796 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:49.873 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2024-02-12T15:37:49.886 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2024-02-12T15:37:50.043 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-4.15.0-200-generic 2024-02-12T15:37:50.056 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-4.15.0-200-generic 2024-02-12T15:37:50.134 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-4.15.0-194-generic 2024-02-12T15:37:50.147 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-4.15.0-194-generic 2024-02-12T15:37:50.505 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-02-12T15:37:50.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-02-12T15:37:51.304 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-02-12T15:37:51.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-02-12T15:38:18.520 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-12T15:38:18.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-12T15:38:18.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:38:20.552 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-02-12T15:38:20.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-02-12T15:38:20.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:38:21.307 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-12T15:38:21.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-02-12T15:38:21.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:38:36.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-02-12T15:38:39.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-02-12T15:38:39.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-02-12T15:38:45.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-12T15:38:45.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:38:48.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-02-12T15:38:48.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:38:48.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-02-12T15:38:48.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:38:49.056 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-02-12T15:38:51.100 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-02-12T15:38:51.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-02-12T15:39:01.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-12T15:39:01.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:39:03.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-02-12T15:39:03.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:39:03.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-02-12T15:39:03.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:39:19.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-02-12T15:39:21.537 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-02-12T15:39:34.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-12T15:39:34.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:39:34.583 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-02-12T15:39:34.847 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-12T15:39:34.847 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-213-generic"... 2024-02-12T15:39:34.847 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-02-12T15:39:34.895 INFO:teuthology.orchestra.run.smithi121.stdout:4.15.0-213-generic 2024-02-12T15:39:34.895 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-213-generic vs 4.15.0-213-generic 2024-02-12T15:39:34.895 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-12T15:39:34.895 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-12T15:39:35.532 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-02-12T15:39:35.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-02-12T15:39:35.800 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-213-generic"... 2024-02-12T15:39:35.800 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-02-12T15:39:35.847 INFO:teuthology.orchestra.run.smithi123.stdout:4.15.0-213-generic 2024-02-12T15:39:35.847 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-213-generic vs 4.15.0-213-generic 2024-02-12T15:39:35.847 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-12T15:39:35.847 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-12T15:39:35.897 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-12T15:39:35.897 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-12T15:39:35.940 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-02-12T15:39:35.959 DEBUG:teuthology.parallel:result is None 2024-02-12T15:39:36.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-02-12T15:39:36.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:39:36.702 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-02-12T15:39:36.848 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-12T15:39:36.849 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-12T15:39:36.874 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-02-12T15:39:36.891 DEBUG:teuthology.parallel:result is None 2024-02-12T15:39:37.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-02-12T15:39:37.441 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.15.0-213-generic"... 2024-02-12T15:39:37.441 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-02-12T15:39:37.493 INFO:teuthology.orchestra.run.smithi169.stdout:4.15.0-213-generic 2024-02-12T15:39:37.494 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-213-generic vs 4.15.0-213-generic 2024-02-12T15:39:37.494 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-12T15:39:37.494 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-12T15:39:38.495 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-02-12T15:39:38.495 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-12T15:39:38.526 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-02-12T15:39:38.556 DEBUG:teuthology.parallel:result is None 2024-02-12T15:39:38.557 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-12T15:39:38.564 INFO:teuthology.task.internal:Creating test directory... 2024-02-12T15:39:38.564 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-12T15:39:38.567 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-12T15:39:38.569 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-12T15:39:38.607 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-12T15:39:38.613 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-12T15:39:38.630 INFO:teuthology.task.internal:Creating archive directory... 2024-02-12T15:39:38.631 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-12T15:39:38.633 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-12T15:39:38.635 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-12T15:39:38.741 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-12T15:39:38.747 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-12T15:39:38.747 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-12T15:39:38.749 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-12T15:39:38.751 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-12T15:39:38.772 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T15:39:38.775 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T15:39:38.775 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T15:39:38.781 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T15:39:38.781 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T15:39:38.784 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-12T15:39:38.786 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-12T15:39:38.792 INFO:teuthology.task.internal:Configuring sudo... 2024-02-12T15:39:38.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-12T15:39:38.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-12T15:39:38.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-12T15:39:38.847 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-12T15:39:38.855 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-12T15:39:38.856 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-12T15:39:38.889 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-12T15:39:38.892 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-12T15:39:38.898 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T15:39:38.942 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T15:39:38.990 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:39:38.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-12T15:39:39.063 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T15:39:39.070 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T15:39:39.118 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:39:39.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-12T15:39:39.183 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-12T15:39:39.190 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-12T15:39:39.239 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:39:39.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-12T15:39:39.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-02-12T15:39:39.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-02-12T15:39:39.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-02-12T15:39:39.436 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-12T15:39:39.443 INFO:teuthology.task.internal:Starting timer... 2024-02-12T15:39:39.443 INFO:teuthology.run_tasks:Running task pcp... 2024-02-12T15:39:39.475 INFO:teuthology.run_tasks:Running task selinux... 2024-02-12T15:39:39.499 DEBUG:teuthology.task.selinux:Excluding smithi121: OS 'ubuntu' does not support SELinux 2024-02-12T15:39:39.499 DEBUG:teuthology.task.selinux:Excluding smithi123: OS 'ubuntu' does not support SELinux 2024-02-12T15:39:39.499 DEBUG:teuthology.task.selinux:Excluding smithi169: OS 'ubuntu' does not support SELinux 2024-02-12T15:39:39.499 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-12T15:39:39.500 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-02-12T15:39:39.500 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-12T15:39:39.500 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-12T15:39:39.527 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-12T15:39:39.543 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-12T15:39:39.545 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi123.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-12T15:45:23.296 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-02-12T15:45:23.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-12T15:45:23.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:45:23.440 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-02-12T15:45:23.561 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-12T15:45:23.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-02-12T15:45:23.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:45:23.684 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-02-12T15:45:23.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-02-12T15:45:23.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-02-12T15:45:23.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-12T15:45:23.897 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-02-12T15:45:24.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-02-12T15:45:24.009 INFO:teuthology.run_tasks:Running task clock... 2024-02-12T15:45:24.019 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-12T15:45:24.019 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-12T15:45:24.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T15:45:24.024 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-12T15:45:24.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T15:45:24.027 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-12T15:45:24.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T15:45:24.089 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: ntpd 4.2.8p10@1.3728-o (1): Starting 2024-02-12T15:45:24.089 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Command line: ntpd -gq 2024-02-12T15:45:24.089 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: proto: precision = 0.124 usec (-23) 2024-02-12T15:45:24.090 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: ntpd 4.2.8p10@1.3728-o (1): Starting 2024-02-12T15:45:24.090 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Command line: ntpd -gq 2024-02-12T15:45:24.090 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: proto: precision = 0.186 usec (-22) 2024-02-12T15:45:24.090 INFO:teuthology.orchestra.run.smithi121.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-12T15:45:24.090 INFO:teuthology.orchestra.run.smithi123.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-12T15:45:24.091 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-12T15:45:24.091 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-12T15:45:24.091 INFO:teuthology.orchestra.run.smithi121.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-12T15:45:24.091 INFO:teuthology.orchestra.run.smithi123.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: restrict ::: KOD does nothing without LIMITED. 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Listen and drop on 0 v6wildcard [::]:123 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Listen normally on 2 lo 127.0.0.1:123 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Listen normally on 3 enp3s0f1 172.21.15.121:123 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Listen normally on 4 lo [::1]:123 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba3%5]:123 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:24 ntpd[14628]: Listening on routing socket on fd #22 for interface updates 2024-02-12T15:45:24.094 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: restrict ::: KOD does nothing without LIMITED. 2024-02-12T15:45:24.095 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Listen and drop on 0 v6wildcard [::]:123 2024-02-12T15:45:24.095 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-12T15:45:24.096 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Listen normally on 2 lo 127.0.0.1:123 2024-02-12T15:45:24.096 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Listen normally on 3 enp3s0f1 172.21.15.123:123 2024-02-12T15:45:24.096 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Listen normally on 4 lo [::1]:123 2024-02-12T15:45:24.096 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3f1%5]:123 2024-02-12T15:45:24.096 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:24 ntpd[14657]: Listening on routing socket on fd #22 for interface updates 2024-02-12T15:45:24.115 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: ntpd 4.2.8p10@1.3728-o (1): Starting 2024-02-12T15:45:24.115 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Command line: ntpd -gq 2024-02-12T15:45:24.115 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: proto: precision = 0.123 usec (-23) 2024-02-12T15:45:24.116 INFO:teuthology.orchestra.run.smithi169.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-12T15:45:24.116 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-12T15:45:24.117 INFO:teuthology.orchestra.run.smithi169.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: restrict ::: KOD does nothing without LIMITED. 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Listen and drop on 0 v6wildcard [::]:123 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Listen normally on 2 lo 127.0.0.1:123 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Listen normally on 3 enp3s0f1 172.21.15.169:123 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Listen normally on 4 lo [::1]:123 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f4d%5]:123 2024-02-12T15:45:24.118 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:24 ntpd[14525]: Listening on routing socket on fd #22 for interface updates 2024-02-12T15:45:25.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:25 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:25.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:25 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:25.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:25 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:25.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:25 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:25.090 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:25.090 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:25.091 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:25.091 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:25.091 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:25.091 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:25.091 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:25.091 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:25 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:25.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:25 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:25.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:25 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:25.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:25 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:25.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:25 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:26.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:26 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:26.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:26.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:26.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:26.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:26.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:26.117 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:26.117 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:26.117 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:26 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:27.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:27 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:27.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:27 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:27.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:27 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:27.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:27 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:27.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:27.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:27.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:27.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:27.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:27.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:27.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:27.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:27 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:27.115 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:27 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:27.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:27 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:27.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:27 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:27.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:27 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:28.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:28.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:28.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:28.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:28.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:28.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:28.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:28.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:28 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:28 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:28 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:28 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:28.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:28 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:28.119 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:28.120 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:28.120 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:28 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:29.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:29 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:29.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:29 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:29.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:29 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:29.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:29 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:29.088 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:29 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:29.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:29 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:29.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:29 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:29.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:29 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:29.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:29 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:30.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:30.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:30.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:30.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:30.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:30.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:30.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:30.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:30 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:30 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:30 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:30 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:30.087 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:30 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:30.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:30 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:31.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:31 ntpd[14657]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:31.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:31 ntpd[14657]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:31.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:31 ntpd[14657]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:31.084 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:31 ntpd[14657]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:31.085 INFO:teuthology.orchestra.run.smithi123.stdout:12 Feb 15:45:31 ntpd[14657]: ntpd: time slew -0.003145 s 2024-02-12T15:45:31.085 INFO:teuthology.orchestra.run.smithi123.stdout:ntpd: time slew -0.003145s 2024-02-12T15:45:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:31 ntpd[14657]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-12T15:45:31.085 INFO:teuthology.orchestra.run.smithi123.stderr:12 Feb 15:45:31 ntpd[14657]: can't open /var/log/ntpstats/loopstats.20240212: Permission denied 2024-02-12T15:45:31.093 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:31 ntpd[14628]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:31.093 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:31 ntpd[14628]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:31.093 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:31 ntpd[14628]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:31.093 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:31 ntpd[14628]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:31.094 INFO:teuthology.orchestra.run.smithi121.stdout:12 Feb 15:45:31 ntpd[14628]: ntpd: time slew +0.001872 s 2024-02-12T15:45:31.094 INFO:teuthology.orchestra.run.smithi121.stdout:ntpd: time slew +0.001872s 2024-02-12T15:45:31.094 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:31 ntpd[14628]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-12T15:45:31.094 INFO:teuthology.orchestra.run.smithi121.stderr:12 Feb 15:45:31 ntpd[14628]: can't open /var/log/ntpstats/loopstats.20240212: Permission denied 2024-02-12T15:45:31.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:31 ntpd[14525]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-12T15:45:31.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:31 ntpd[14525]: can't open /var/log/ntpstats/rawstats.20240212: Permission denied 2024-02-12T15:45:31.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:31 ntpd[14525]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-12T15:45:31.116 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:31 ntpd[14525]: can't open /var/log/ntpstats/peerstats.20240212: Permission denied 2024-02-12T15:45:31.116 INFO:teuthology.orchestra.run.smithi169.stdout:12 Feb 15:45:31 ntpd[14525]: ntpd: time slew +0.000425 s 2024-02-12T15:45:31.116 INFO:teuthology.orchestra.run.smithi169.stdout:ntpd: time slew +0.000425s 2024-02-12T15:45:31.117 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:31 ntpd[14525]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-12T15:45:31.117 INFO:teuthology.orchestra.run.smithi169.stderr:12 Feb 15:45:31 ntpd[14525]: can't open /var/log/ntpstats/loopstats.20240212: Permission denied 2024-02-12T15:45:31.145 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2024-02-12T15:45:31.145 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2024-02-12T15:45:31.145 INFO:teuthology.orchestra.run.smithi123.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.145 INFO:teuthology.orchestra.run.smithi123.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.146 INFO:teuthology.orchestra.run.smithi123.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.146 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2024-02-12T15:45:31.157 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2024-02-12T15:45:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.157 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.169 INFO:teuthology.orchestra.run.smithi169.stdout: remote refid st t when poll reach delay offset jitter 2024-02-12T15:45:31.169 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================== 2024-02-12T15:45:31.169 INFO:teuthology.orchestra.run.smithi169.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.169 INFO:teuthology.orchestra.run.smithi169.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.169 INFO:teuthology.orchestra.run.smithi169.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.170 INFO:teuthology.orchestra.run.smithi169.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-02-12T15:45:31.171 INFO:teuthology.run_tasks:Running task install... 2024-02-12T15:45:31.180 DEBUG:teuthology.task.install:project ceph 2024-02-12T15:45:31.180 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-12T15:45:31.181 DEBUG:teuthology.task.install:config {'branch': 'luminous', 'exclude_packages': ['librados3', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'cephadm', 'ceph-immutable-object-cache', 'ceph-base', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'librados-devel'], 'extra_packages': ['librados2', 'python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-12T15:45:31.181 INFO:teuthology.task.install:Using flavor: default 2024-02-12T15:45:31.187 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-mgr', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-12T15:45:31.188 INFO:teuthology.task.install:extra packages: ['librados2', 'python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'] 2024-02-12T15:45:31.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-key list | grep Ceph 2024-02-12T15:45:31.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-key list | grep Ceph 2024-02-12T15:45:31.191 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-key list | grep Ceph 2024-02-12T15:45:31.287 INFO:teuthology.orchestra.run.smithi169.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-02-12T15:45:31.290 INFO:teuthology.orchestra.run.smithi121.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-02-12T15:45:31.291 INFO:teuthology.orchestra.run.smithi123.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-02-12T15:45:31.343 INFO:teuthology.orchestra.run.smithi169.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-12T15:45:31.344 INFO:teuthology.orchestra.run.smithi169.stdout:uid [ unknown] Ceph.com (release key) 2024-02-12T15:45:31.345 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, librados2, python-rados, python-rgw, python-rbd, python-cephfs on remote deb x86_64 2024-02-12T15:45:31.345 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T15:45:31.345 INFO:teuthology.packaging:ref: None 2024-02-12T15:45:31.345 INFO:teuthology.packaging:tag: None 2024-02-12T15:45:31.345 INFO:teuthology.packaging:branch: luminous 2024-02-12T15:45:31.345 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:45:31.345 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=luminous 2024-02-12T15:45:31.350 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-12T15:45:31.351 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph.com (release key) 2024-02-12T15:45:31.351 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-12T15:45:31.351 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph.com (release key) 2024-02-12T15:45:31.351 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, librados2, python-rados, python-rgw, python-rbd, python-cephfs on remote deb x86_64 2024-02-12T15:45:31.351 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T15:45:31.351 INFO:teuthology.packaging:ref: None 2024-02-12T15:45:31.352 INFO:teuthology.packaging:tag: None 2024-02-12T15:45:31.352 INFO:teuthology.packaging:branch: luminous 2024-02-12T15:45:31.352 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:45:31.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=luminous 2024-02-12T15:45:31.353 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, librados2, python-rados, python-rgw, python-rbd, python-cephfs on remote deb x86_64 2024-02-12T15:45:31.354 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T15:45:31.354 INFO:teuthology.packaging:ref: None 2024-02-12T15:45:31.354 INFO:teuthology.packaging:tag: None 2024-02-12T15:45:31.354 INFO:teuthology.packaging:branch: luminous 2024-02-12T15:45:31.354 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:45:31.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=luminous 2024-02-12T15:45:31.526 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default/ 2024-02-12T15:45:31.526 INFO:teuthology.task.install.deb:Package version is 12.2.13-71-g707ba539-1bionic 2024-02-12T15:45:31.529 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default/ 2024-02-12T15:45:31.530 INFO:teuthology.task.install.deb:Package version is 12.2.13-71-g707ba539-1bionic 2024-02-12T15:45:31.547 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default/ 2024-02-12T15:45:31.547 INFO:teuthology.task.install.deb:Package version is 12.2.13-71-g707ba539-1bionic 2024-02-12T15:45:31.643 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:45:31.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-12T15:45:31.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-02-12T15:45:31.714 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:45:31.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-12T15:45:31.729 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:45:31.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-12T15:45:31.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2024-02-12T15:45:31.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2024-02-12T15:45:31.869 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2024-02-12T15:45:31.956 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:45:31.998 INFO:teuthology.orchestra.run.smithi169.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic InRelease 2024-02-12T15:45:31.999 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic InRelease 2024-02-12T15:45:32.022 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic Release [7,618 B] 2024-02-12T15:45:32.022 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic Release [7,618 B] 2024-02-12T15:45:32.039 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2024-02-12T15:45:32.041 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic InRelease 2024-02-12T15:45:32.042 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:45:32.042 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2024-02-12T15:45:32.044 INFO:teuthology.orchestra.run.smithi169.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic Release.gpg 2024-02-12T15:45:32.045 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic Release.gpg 2024-02-12T15:45:32.061 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2024-02-12T15:45:32.061 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2024-02-12T15:45:32.069 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main i386 Packages [432 B] 2024-02-12T15:45:32.072 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 Packages [13.8 kB] 2024-02-12T15:45:32.081 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic Release [7,618 B] 2024-02-12T15:45:32.102 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 Packages [13.8 kB] 2024-02-12T15:45:32.105 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main i386 Packages [432 B] 2024-02-12T15:45:32.113 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2024-02-12T15:45:32.119 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2024-02-12T15:45:32.125 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic Release.gpg 2024-02-12T15:45:32.130 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2024-02-12T15:45:32.166 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main i386 Packages [432 B] 2024-02-12T15:45:32.194 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2024-02-12T15:45:32.211 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 Packages [13.8 kB] 2024-02-12T15:45:32.212 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2024-02-12T15:45:32.463 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 21.8 kB in 1s (36.4 kB/s) 2024-02-12T15:45:32.642 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 21.8 kB in 1s (30.9 kB/s) 2024-02-12T15:45:32.676 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 21.8 kB in 1s (30.2 kB/s) 2024-02-12T15:45:33.992 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-02-12T15:45:34.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-71-g707ba539-1bionic ceph-mds=12.2.13-71-g707ba539-1bionic ceph-mgr=12.2.13-71-g707ba539-1bionic ceph-common=12.2.13-71-g707ba539-1bionic ceph-fuse=12.2.13-71-g707ba539-1bionic ceph-test=12.2.13-71-g707ba539-1bionic radosgw=12.2.13-71-g707ba539-1bionic libcephfs2=12.2.13-71-g707ba539-1bionic libcephfs-dev=12.2.13-71-g707ba539-1bionic librados2=12.2.13-71-g707ba539-1bionic librbd1=12.2.13-71-g707ba539-1bionic rbd-fuse=12.2.13-71-g707ba539-1bionic librados2=12.2.13-71-g707ba539-1bionic python-rados=12.2.13-71-g707ba539-1bionic python-rgw=12.2.13-71-g707ba539-1bionic python-rbd=12.2.13-71-g707ba539-1bionic python-cephfs=12.2.13-71-g707ba539-1bionic 2024-02-12T15:45:34.114 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2024-02-12T15:45:34.122 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-02-12T15:45:34.150 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-12T15:45:34.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-71-g707ba539-1bionic ceph-mds=12.2.13-71-g707ba539-1bionic ceph-mgr=12.2.13-71-g707ba539-1bionic ceph-common=12.2.13-71-g707ba539-1bionic ceph-fuse=12.2.13-71-g707ba539-1bionic ceph-test=12.2.13-71-g707ba539-1bionic radosgw=12.2.13-71-g707ba539-1bionic libcephfs2=12.2.13-71-g707ba539-1bionic libcephfs-dev=12.2.13-71-g707ba539-1bionic librados2=12.2.13-71-g707ba539-1bionic librbd1=12.2.13-71-g707ba539-1bionic rbd-fuse=12.2.13-71-g707ba539-1bionic librados2=12.2.13-71-g707ba539-1bionic python-rados=12.2.13-71-g707ba539-1bionic python-rgw=12.2.13-71-g707ba539-1bionic python-rbd=12.2.13-71-g707ba539-1bionic python-cephfs=12.2.13-71-g707ba539-1bionic 2024-02-12T15:45:34.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-71-g707ba539-1bionic ceph-mds=12.2.13-71-g707ba539-1bionic ceph-mgr=12.2.13-71-g707ba539-1bionic ceph-common=12.2.13-71-g707ba539-1bionic ceph-fuse=12.2.13-71-g707ba539-1bionic ceph-test=12.2.13-71-g707ba539-1bionic radosgw=12.2.13-71-g707ba539-1bionic libcephfs2=12.2.13-71-g707ba539-1bionic libcephfs-dev=12.2.13-71-g707ba539-1bionic librados2=12.2.13-71-g707ba539-1bionic librbd1=12.2.13-71-g707ba539-1bionic rbd-fuse=12.2.13-71-g707ba539-1bionic librados2=12.2.13-71-g707ba539-1bionic python-rados=12.2.13-71-g707ba539-1bionic python-rgw=12.2.13-71-g707ba539-1bionic python-rbd=12.2.13-71-g707ba539-1bionic python-cephfs=12.2.13-71-g707ba539-1bionic 2024-02-12T15:45:34.250 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2024-02-12T15:45:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2024-02-12T15:45:34.416 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-02-12T15:45:34.417 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-02-12T15:45:34.537 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2024-02-12T15:45:34.537 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2024-02-12T15:45:34.548 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2024-02-12T15:45:34.549 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2024-02-12T15:45:34.621 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-02-12T15:45:34.621 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2024-02-12T15:45:34.621 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2024-02-12T15:45:34.622 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-extra-4.15.0-194-generic 2024-02-12T15:45:34.622 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-12T15:45:34.622 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2024-02-12T15:45:34.622 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base ceph-mon ceph-osd formencode-i18n jq libjq1 libjs-sphinxdoc 2024-02-12T15:45:34.623 INFO:teuthology.orchestra.run.smithi123.stdout: libjs-underscore libleveldb1v5 liblttng-ust0 libonig4 libradosstriper1 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 libsnappy1v5 libxslt1.1 python-blinker python-bs4 python-certifi 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-chardet python-cherrypy3 python-click python-colorama 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-dnspython python-flask python-formencode python-html5lib 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-itsdangerous python-jinja2 python-logutils python-lxml python-mako 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-markupsafe python-openid python-openssl python-paste 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-prettytable python-pyinotify python-repoze.lru python-requests 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-routes python-scgi python-simplegeneric python-simplejson 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-singledispatch python-tempita python-urllib3 python-waitress 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-webencodings python-webob python-webtest python-werkzeug socat 2024-02-12T15:45:34.624 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout: python-blinker-doc python-flask-doc python-egenix-mxdatetime python-genshi 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout: python-jinja2-doc python-lxml-dbg python-lxml-doc python-beaker 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout: python-mako-doc python-openssl-doc python-openssl-dbg httpd-wsgi 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pastewebkit 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout: python-cheetah python-cherrypy python-flup python-pecan-doc 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout: python-pyinotify-doc python-socks python-ntlm python-waitress-doc 2024-02-12T15:45:34.626 INFO:teuthology.orchestra.run.smithi123.stdout: python-webob-doc python-webtest-doc python-pyquery ipython python-redis 2024-02-12T15:45:34.627 INFO:teuthology.orchestra.run.smithi123.stdout: python-pylibmc | python-memcache python-termcolor python-watchdog 2024-02-12T15:45:34.627 INFO:teuthology.orchestra.run.smithi123.stdout: python-werkzeug-doc 2024-02-12T15:45:34.720 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mon ceph-osd 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test formencode-i18n jq libcephfs-dev libcephfs2 libjq1 libjs-sphinxdoc 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: libjs-underscore libleveldb1v5 liblttng-ust0 libonig4 libradosstriper1 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 libsnappy1v5 libxslt1.1 python-blinker python-bs4 python-cephfs 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: python-certifi python-chardet python-cherrypy3 python-click python-colorama 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: python-dnspython python-flask python-formencode python-html5lib 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: python-itsdangerous python-jinja2 python-logutils python-lxml python-mako 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: python-markupsafe python-openid python-openssl python-paste 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2024-02-12T15:45:34.721 INFO:teuthology.orchestra.run.smithi123.stdout: python-prettytable python-pyinotify python-rados python-rbd 2024-02-12T15:45:34.722 INFO:teuthology.orchestra.run.smithi123.stdout: python-repoze.lru python-requests python-rgw python-routes python-scgi 2024-02-12T15:45:34.722 INFO:teuthology.orchestra.run.smithi123.stdout: python-simplegeneric python-simplejson python-singledispatch python-tempita 2024-02-12T15:45:34.722 INFO:teuthology.orchestra.run.smithi123.stdout: python-urllib3 python-waitress python-webencodings python-webob 2024-02-12T15:45:34.722 INFO:teuthology.orchestra.run.smithi123.stdout: python-webtest python-werkzeug radosgw rbd-fuse socat xmlstarlet 2024-02-12T15:45:34.722 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages will be upgraded: 2024-02-12T15:45:34.726 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 librbd1 2024-02-12T15:45:34.743 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages were automatically installed and are no longer required: 2024-02-12T15:45:34.743 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2024-02-12T15:45:34.743 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2024-02-12T15:45:34.744 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-extra-4.15.0-194-generic 2024-02-12T15:45:34.744 INFO:teuthology.orchestra.run.smithi169.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-12T15:45:34.744 INFO:teuthology.orchestra.run.smithi169.stdout:The following additional packages will be installed: 2024-02-12T15:45:34.744 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base ceph-mon ceph-osd formencode-i18n jq libjq1 libjs-sphinxdoc 2024-02-12T15:45:34.744 INFO:teuthology.orchestra.run.smithi169.stdout: libjs-underscore libleveldb1v5 liblttng-ust0 libonig4 libradosstriper1 2024-02-12T15:45:34.745 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 libsnappy1v5 libxslt1.1 python-blinker python-bs4 python-certifi 2024-02-12T15:45:34.745 INFO:teuthology.orchestra.run.smithi169.stdout: python-chardet python-cherrypy3 python-click python-colorama 2024-02-12T15:45:34.745 INFO:teuthology.orchestra.run.smithi169.stdout: python-dnspython python-flask python-formencode python-html5lib 2024-02-12T15:45:34.745 INFO:teuthology.orchestra.run.smithi169.stdout: python-itsdangerous python-jinja2 python-logutils python-lxml python-mako 2024-02-12T15:45:34.745 INFO:teuthology.orchestra.run.smithi169.stdout: python-markupsafe python-openid python-openssl python-paste 2024-02-12T15:45:34.745 INFO:teuthology.orchestra.run.smithi169.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2024-02-12T15:45:34.746 INFO:teuthology.orchestra.run.smithi169.stdout: python-prettytable python-pyinotify python-repoze.lru python-requests 2024-02-12T15:45:34.746 INFO:teuthology.orchestra.run.smithi169.stdout: python-routes python-scgi python-simplegeneric python-simplejson 2024-02-12T15:45:34.746 INFO:teuthology.orchestra.run.smithi169.stdout: python-singledispatch python-tempita python-urllib3 python-waitress 2024-02-12T15:45:34.746 INFO:teuthology.orchestra.run.smithi169.stdout: python-webencodings python-webob python-webtest python-werkzeug socat 2024-02-12T15:45:34.746 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet 2024-02-12T15:45:34.748 INFO:teuthology.orchestra.run.smithi169.stdout:Suggested packages: 2024-02-12T15:45:34.748 INFO:teuthology.orchestra.run.smithi169.stdout: python-blinker-doc python-flask-doc python-egenix-mxdatetime python-genshi 2024-02-12T15:45:34.748 INFO:teuthology.orchestra.run.smithi169.stdout: python-jinja2-doc python-lxml-dbg python-lxml-doc python-beaker 2024-02-12T15:45:34.749 INFO:teuthology.orchestra.run.smithi169.stdout: python-mako-doc python-openssl-doc python-openssl-dbg httpd-wsgi 2024-02-12T15:45:34.749 INFO:teuthology.orchestra.run.smithi169.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pastewebkit 2024-02-12T15:45:34.749 INFO:teuthology.orchestra.run.smithi169.stdout: python-cheetah python-cherrypy python-flup python-pecan-doc 2024-02-12T15:45:34.749 INFO:teuthology.orchestra.run.smithi169.stdout: python-pyinotify-doc python-socks python-ntlm python-waitress-doc 2024-02-12T15:45:34.749 INFO:teuthology.orchestra.run.smithi169.stdout: python-webob-doc python-webtest-doc python-pyquery ipython python-redis 2024-02-12T15:45:34.749 INFO:teuthology.orchestra.run.smithi169.stdout: python-pylibmc | python-memcache python-termcolor python-watchdog 2024-02-12T15:45:34.749 INFO:teuthology.orchestra.run.smithi169.stdout: python-werkzeug-doc 2024-02-12T15:45:34.757 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2024-02-12T15:45:34.757 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-4.15.0-194 linux-headers-4.15.0-194-generic 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-4.15.0-194-generic linux-modules-4.15.0-194-generic 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-4.15.0-194-generic 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base ceph-mon ceph-osd formencode-i18n jq libjq1 libjs-sphinxdoc 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: libjs-underscore libleveldb1v5 liblttng-ust0 libonig4 libradosstriper1 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 libsnappy1v5 libxslt1.1 python-blinker python-bs4 python-certifi 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-chardet python-cherrypy3 python-click python-colorama 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-dnspython python-flask python-formencode python-html5lib 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-itsdangerous python-jinja2 python-logutils python-lxml python-mako 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-markupsafe python-openid python-openssl python-paste 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-prettytable python-pyinotify python-repoze.lru python-requests 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-routes python-scgi python-simplegeneric python-simplejson 2024-02-12T15:45:34.758 INFO:teuthology.orchestra.run.smithi121.stdout: python-singledispatch python-tempita python-urllib3 python-waitress 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: python-webencodings python-webob python-webtest python-werkzeug socat 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: python-blinker-doc python-flask-doc python-egenix-mxdatetime python-genshi 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: python-jinja2-doc python-lxml-dbg python-lxml-doc python-beaker 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: python-mako-doc python-openssl-doc python-openssl-dbg httpd-wsgi 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pastewebkit 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: python-cheetah python-cherrypy python-flup python-pecan-doc 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: python-pyinotify-doc python-socks python-ntlm python-waitress-doc 2024-02-12T15:45:34.759 INFO:teuthology.orchestra.run.smithi121.stdout: python-webob-doc python-webtest-doc python-pyquery ipython python-redis 2024-02-12T15:45:34.760 INFO:teuthology.orchestra.run.smithi121.stdout: python-pylibmc | python-memcache python-termcolor python-watchdog 2024-02-12T15:45:34.760 INFO:teuthology.orchestra.run.smithi121.stdout: python-werkzeug-doc 2024-02-12T15:45:34.839 INFO:teuthology.orchestra.run.smithi169.stdout:The following NEW packages will be installed: 2024-02-12T15:45:34.839 INFO:teuthology.orchestra.run.smithi169.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mon ceph-osd 2024-02-12T15:45:34.840 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test formencode-i18n jq libcephfs-dev libcephfs2 libjq1 libjs-sphinxdoc 2024-02-12T15:45:34.840 INFO:teuthology.orchestra.run.smithi169.stdout: libjs-underscore libleveldb1v5 liblttng-ust0 libonig4 libradosstriper1 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 libsnappy1v5 libxslt1.1 python-blinker python-bs4 python-cephfs 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-certifi python-chardet python-cherrypy3 python-click python-colorama 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-dnspython python-flask python-formencode python-html5lib 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-itsdangerous python-jinja2 python-logutils python-lxml python-mako 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-markupsafe python-openid python-openssl python-paste 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-prettytable python-pyinotify python-rados python-rbd 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-repoze.lru python-requests python-rgw python-routes python-scgi 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-simplegeneric python-simplejson python-singledispatch python-tempita 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-urllib3 python-waitress python-webencodings python-webob 2024-02-12T15:45:34.841 INFO:teuthology.orchestra.run.smithi169.stdout: python-webtest python-werkzeug radosgw rbd-fuse socat xmlstarlet 2024-02-12T15:45:34.843 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages will be upgraded: 2024-02-12T15:45:34.843 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 librbd1 2024-02-12T15:45:34.850 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2024-02-12T15:45:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mon ceph-osd 2024-02-12T15:45:34.850 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test formencode-i18n jq libcephfs-dev libcephfs2 libjq1 libjs-sphinxdoc 2024-02-12T15:45:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: libjs-underscore libleveldb1v5 liblttng-ust0 libonig4 libradosstriper1 2024-02-12T15:45:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 libsnappy1v5 libxslt1.1 python-blinker python-bs4 python-cephfs 2024-02-12T15:45:34.851 INFO:teuthology.orchestra.run.smithi121.stdout: python-certifi python-chardet python-cherrypy3 python-click python-colorama 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-dnspython python-flask python-formencode python-html5lib 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-itsdangerous python-jinja2 python-logutils python-lxml python-mako 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-markupsafe python-openid python-openssl python-paste 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-prettytable python-pyinotify python-rados python-rbd 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-repoze.lru python-requests python-rgw python-routes python-scgi 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-simplegeneric python-simplejson python-singledispatch python-tempita 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-urllib3 python-waitress python-webencodings python-webob 2024-02-12T15:45:34.852 INFO:teuthology.orchestra.run.smithi121.stdout: python-webtest python-werkzeug radosgw rbd-fuse socat xmlstarlet 2024-02-12T15:45:34.853 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2024-02-12T15:45:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 librbd1 2024-02-12T15:45:34.918 INFO:teuthology.orchestra.run.smithi123.stdout:2 upgraded, 71 newly installed, 0 to remove and 146 not upgraded. 2024-02-12T15:45:34.918 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 86.6 MB of archives. 2024-02-12T15:45:34.918 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 356 MB of additional disk space will be used. 2024-02-12T15:45:34.918 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2024-02-12T15:45:35.001 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 12.2.13-71-g707ba539-1bionic [975 kB] 2024-02-12T15:45:35.038 INFO:teuthology.orchestra.run.smithi169.stdout:2 upgraded, 71 newly installed, 0 to remove and 146 not upgraded. 2024-02-12T15:45:35.038 INFO:teuthology.orchestra.run.smithi169.stdout:Need to get 86.6 MB of archives. 2024-02-12T15:45:35.038 INFO:teuthology.orchestra.run.smithi169.stdout:After this operation, 356 MB of additional disk space will be used. 2024-02-12T15:45:35.038 INFO:teuthology.orchestra.run.smithi169.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2024-02-12T15:45:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:2 upgraded, 71 newly installed, 0 to remove and 146 not upgraded. 2024-02-12T15:45:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 86.6 MB of archives. 2024-02-12T15:45:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 356 MB of additional disk space will be used. 2024-02-12T15:45:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2024-02-12T15:45:35.075 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 12.2.13-71-g707ba539-1bionic [975 kB] 2024-02-12T15:45:35.080 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 12.2.13-71-g707ba539-1bionic [975 kB] 2024-02-12T15:45:35.312 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 12.2.13-71-g707ba539-1bionic [2,709 kB] 2024-02-12T15:45:35.313 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 12.2.13-71-g707ba539-1bionic [2,709 kB] 2024-02-12T15:45:35.316 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2024-02-12T15:45:35.323 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2024-02-12T15:45:35.379 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2024-02-12T15:45:35.404 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 12.2.13-71-g707ba539-1bionic [2,709 kB] 2024-02-12T15:45:35.427 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.2 [86.0 kB] 2024-02-12T15:45:35.427 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2024-02-12T15:45:35.435 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2024-02-12T15:45:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2024-02-12T15:45:35.446 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2024-02-12T15:45:35.459 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2024-02-12T15:45:35.476 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2024-02-12T15:45:35.480 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2024-02-12T15:45:35.492 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2024-02-12T15:45:35.502 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2024-02-12T15:45:35.503 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2024-02-12T15:45:35.531 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2024-02-12T15:45:35.532 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-jinja2 all 2.10-1ubuntu0.18.04.1 [94.8 kB] 2024-02-12T15:45:35.539 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.2 [86.0 kB] 2024-02-12T15:45:35.553 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.2 [86.0 kB] 2024-02-12T15:45:35.571 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2024-02-12T15:45:35.581 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2024-02-12T15:45:35.589 INFO:teuthology.orchestra.run.smithi169.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2024-02-12T15:45:35.589 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 12.2.13-71-g707ba539-1bionic [405 kB] 2024-02-12T15:45:35.592 INFO:teuthology.orchestra.run.smithi169.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2024-02-12T15:45:35.593 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 12.2.13-71-g707ba539-1bionic [284 kB] 2024-02-12T15:45:35.599 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2024-02-12T15:45:35.603 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2024-02-12T15:45:35.615 INFO:teuthology.orchestra.run.smithi169.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2024-02-12T15:45:35.630 INFO:teuthology.orchestra.run.smithi169.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 12.2.13-71-g707ba539-1bionic [405 kB] 2024-02-12T15:45:35.630 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2024-02-12T15:45:35.631 INFO:teuthology.orchestra.run.smithi169.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 12.2.13-71-g707ba539-1bionic [284 kB] 2024-02-12T15:45:35.637 INFO:teuthology.orchestra.run.smithi169.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 12.2.13-71-g707ba539-1bionic [93.1 kB] 2024-02-12T15:45:35.637 INFO:teuthology.orchestra.run.smithi169.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 12.2.13-71-g707ba539-1bionic [151 kB] 2024-02-12T15:45:35.637 INFO:teuthology.orchestra.run.smithi169.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 12.2.13-71-g707ba539-1bionic [1,823 kB] 2024-02-12T15:45:35.638 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 12.2.13-71-g707ba539-1bionic [93.1 kB] 2024-02-12T15:45:35.642 INFO:teuthology.orchestra.run.smithi169.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2024-02-12T15:45:35.642 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 12.2.13-71-g707ba539-1bionic [151 kB] 2024-02-12T15:45:35.646 INFO:teuthology.orchestra.run.smithi169.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-jinja2 all 2.10-1ubuntu0.18.04.1 [94.8 kB] 2024-02-12T15:45:35.648 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 12.2.13-71-g707ba539-1bionic [1,823 kB] 2024-02-12T15:45:35.652 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2024-02-12T15:45:35.654 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-jinja2 all 2.10-1ubuntu0.18.04.1 [94.8 kB] 2024-02-12T15:45:35.712 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2024-02-12T15:45:35.769 INFO:teuthology.orchestra.run.smithi169.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 12.2.13-71-g707ba539-1bionic [101 kB] 2024-02-12T15:45:35.792 INFO:teuthology.orchestra.run.smithi169.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 12.2.13-71-g707ba539-1bionic [315 kB] 2024-02-12T15:45:35.802 INFO:teuthology.orchestra.run.smithi169.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 12.2.13-71-g707ba539-1bionic [13.0 MB] 2024-02-12T15:45:35.827 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 12.2.13-71-g707ba539-1bionic [101 kB] 2024-02-12T15:45:35.827 INFO:teuthology.orchestra.run.smithi169.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2024-02-12T15:45:35.828 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 12.2.13-71-g707ba539-1bionic [315 kB] 2024-02-12T15:45:35.834 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2024-02-12T15:45:35.835 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 12.2.13-71-g707ba539-1bionic [13.0 MB] 2024-02-12T15:45:35.870 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 12.2.13-71-g707ba539-1bionic [405 kB] 2024-02-12T15:45:35.925 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2024-02-12T15:45:35.950 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-mako all 1.0.7+ds1-1ubuntu0.2 [60.2 kB] 2024-02-12T15:45:35.955 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 12.2.13-71-g707ba539-1bionic [284 kB] 2024-02-12T15:45:35.965 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 12.2.13-71-g707ba539-1bionic [93.1 kB] 2024-02-12T15:45:35.980 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 12.2.13-71-g707ba539-1bionic [151 kB] 2024-02-12T15:45:36.034 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 12.2.13-71-g707ba539-1bionic [1,823 kB] 2024-02-12T15:45:36.045 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2024-02-12T15:45:36.048 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2024-02-12T15:45:36.069 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2024-02-12T15:45:36.070 INFO:teuthology.orchestra.run.smithi169.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2024-02-12T15:45:36.074 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-mako all 1.0.7+ds1-1ubuntu0.2 [60.2 kB] 2024-02-12T15:45:36.077 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2024-02-12T15:45:36.086 INFO:teuthology.orchestra.run.smithi169.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-mako all 1.0.7+ds1-1ubuntu0.2 [60.2 kB] 2024-02-12T15:45:36.124 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2024-02-12T15:45:36.160 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2024-02-12T15:45:36.186 INFO:teuthology.orchestra.run.smithi169.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2024-02-12T15:45:36.186 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2024-02-12T15:45:36.187 INFO:teuthology.orchestra.run.smithi169.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2024-02-12T15:45:36.197 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2024-02-12T15:45:36.197 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2024-02-12T15:45:36.198 INFO:teuthology.orchestra.run.smithi169.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2024-02-12T15:45:36.207 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2024-02-12T15:45:36.213 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2024-02-12T15:45:36.237 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2024-02-12T15:45:36.245 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 12.2.13-71-g707ba539-1bionic [101 kB] 2024-02-12T15:45:36.247 INFO:teuthology.orchestra.run.smithi169.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2024-02-12T15:45:36.251 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2024-02-12T15:45:36.280 INFO:teuthology.orchestra.run.smithi169.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2024-02-12T15:45:36.288 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2024-02-12T15:45:36.314 INFO:teuthology.orchestra.run.smithi169.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2024-02-12T15:45:36.314 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 12.2.13-71-g707ba539-1bionic [315 kB] 2024-02-12T15:45:36.322 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2024-02-12T15:45:36.333 INFO:teuthology.orchestra.run.smithi169.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2024-02-12T15:45:36.341 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2024-02-12T15:45:36.356 INFO:teuthology.orchestra.run.smithi169.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2024-02-12T15:45:36.365 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2024-02-12T15:45:36.401 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2024-02-12T15:45:36.420 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 12.2.13-71-g707ba539-1bionic [13.0 MB] 2024-02-12T15:45:36.520 INFO:teuthology.orchestra.run.smithi169.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2024-02-12T15:45:36.529 INFO:teuthology.orchestra.run.smithi121.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2024-02-12T15:45:36.695 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2024-02-12T15:45:36.703 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2024-02-12T15:45:36.819 INFO:teuthology.orchestra.run.smithi169.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2024-02-12T15:45:36.832 INFO:teuthology.orchestra.run.smithi169.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2024-02-12T15:45:36.832 INFO:teuthology.orchestra.run.smithi121.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2024-02-12T15:45:36.838 INFO:teuthology.orchestra.run.smithi121.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2024-02-12T15:45:36.929 INFO:teuthology.orchestra.run.smithi169.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 12.2.13-71-g707ba539-1bionic [3,316 kB] 2024-02-12T15:45:36.948 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2024-02-12T15:45:36.949 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2024-02-12T15:45:36.956 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2024-02-12T15:45:36.961 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2024-02-12T15:45:36.977 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-werkzeug all 0.14.1+dfsg1-1ubuntu0.2 [175 kB] 2024-02-12T15:45:37.004 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-colorama all 0.3.7-1 [22.6 kB] 2024-02-12T15:45:37.007 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-click all 6.7-3 [56.4 kB] 2024-02-12T15:45:37.013 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 12.2.13-71-g707ba539-1bionic [3,316 kB] 2024-02-12T15:45:37.015 INFO:teuthology.orchestra.run.smithi123.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-itsdangerous all 0.24+dfsg1-2 [11.9 kB] 2024-02-12T15:45:37.079 INFO:teuthology.orchestra.run.smithi169.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2024-02-12T15:45:37.080 INFO:teuthology.orchestra.run.smithi169.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2024-02-12T15:45:37.085 INFO:teuthology.orchestra.run.smithi121.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2024-02-12T15:45:37.086 INFO:teuthology.orchestra.run.smithi169.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2024-02-12T15:45:37.087 INFO:teuthology.orchestra.run.smithi121.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2024-02-12T15:45:37.099 INFO:teuthology.orchestra.run.smithi169.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2024-02-12T15:45:37.099 INFO:teuthology.orchestra.run.smithi121.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2024-02-12T15:45:37.099 INFO:teuthology.orchestra.run.smithi121.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2024-02-12T15:45:37.108 INFO:teuthology.orchestra.run.smithi169.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-werkzeug all 0.14.1+dfsg1-1ubuntu0.2 [175 kB] 2024-02-12T15:45:37.115 INFO:teuthology.orchestra.run.smithi121.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-werkzeug all 0.14.1+dfsg1-1ubuntu0.2 [175 kB] 2024-02-12T15:45:37.136 INFO:teuthology.orchestra.run.smithi169.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-colorama all 0.3.7-1 [22.6 kB] 2024-02-12T15:45:37.138 INFO:teuthology.orchestra.run.smithi169.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-click all 6.7-3 [56.4 kB] 2024-02-12T15:45:37.141 INFO:teuthology.orchestra.run.smithi121.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-colorama all 0.3.7-1 [22.6 kB] 2024-02-12T15:45:37.144 INFO:teuthology.orchestra.run.smithi169.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-itsdangerous all 0.24+dfsg1-2 [11.9 kB] 2024-02-12T15:45:37.144 INFO:teuthology.orchestra.run.smithi121.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-click all 6.7-3 [56.4 kB] 2024-02-12T15:45:37.151 INFO:teuthology.orchestra.run.smithi121.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-itsdangerous all 0.24+dfsg1-2 [11.9 kB] 2024-02-12T15:45:37.180 INFO:teuthology.orchestra.run.smithi123.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-flask all 0.12.2-3ubuntu0.1 [62.4 kB] 2024-02-12T15:45:37.226 INFO:teuthology.orchestra.run.smithi169.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 12.2.13-71-g707ba539-1bionic [3,526 kB] 2024-02-12T15:45:37.310 INFO:teuthology.orchestra.run.smithi169.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-flask all 0.12.2-3ubuntu0.1 [62.4 kB] 2024-02-12T15:45:37.316 INFO:teuthology.orchestra.run.smithi121.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-flask all 0.12.2-3ubuntu0.1 [62.4 kB] 2024-02-12T15:45:37.317 INFO:teuthology.orchestra.run.smithi123.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 12.2.13-71-g707ba539-1bionic [3,526 kB] 2024-02-12T15:45:37.457 INFO:teuthology.orchestra.run.smithi123.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2024-02-12T15:45:37.510 INFO:teuthology.orchestra.run.smithi169.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 12.2.13-71-g707ba539-1bionic [4,512 kB] 2024-02-12T15:45:37.580 INFO:teuthology.orchestra.run.smithi123.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2024-02-12T15:45:37.594 INFO:teuthology.orchestra.run.smithi121.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2024-02-12T15:45:37.598 INFO:teuthology.orchestra.run.smithi169.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2024-02-12T15:45:37.610 INFO:teuthology.orchestra.run.smithi123.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 12.2.13-71-g707ba539-1bionic [4,512 kB] 2024-02-12T15:45:37.644 INFO:teuthology.orchestra.run.smithi123.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2024-02-12T15:45:37.661 INFO:teuthology.orchestra.run.smithi123.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2024-02-12T15:45:37.711 INFO:teuthology.orchestra.run.smithi121.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2024-02-12T15:45:37.719 INFO:teuthology.orchestra.run.smithi169.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2024-02-12T15:45:37.755 INFO:teuthology.orchestra.run.smithi123.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.3 [150 kB] 2024-02-12T15:45:37.781 INFO:teuthology.orchestra.run.smithi123.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2024-02-12T15:45:37.781 INFO:teuthology.orchestra.run.smithi121.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2024-02-12T15:45:37.785 INFO:teuthology.orchestra.run.smithi169.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2024-02-12T15:45:37.792 INFO:teuthology.orchestra.run.smithi121.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2024-02-12T15:45:37.802 INFO:teuthology.orchestra.run.smithi169.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2024-02-12T15:45:37.823 INFO:teuthology.orchestra.run.smithi123.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libjs-underscore all 1.8.3~dfsg-1ubuntu0.1 [60.4 kB] 2024-02-12T15:45:37.830 INFO:teuthology.orchestra.run.smithi123.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2024-02-12T15:45:37.838 INFO:teuthology.orchestra.run.smithi123.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-blinker all 1.4+dfsg1-0.1 [13.0 kB] 2024-02-12T15:45:37.845 INFO:teuthology.orchestra.run.smithi123.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2024-02-12T15:45:37.885 INFO:teuthology.orchestra.run.smithi121.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.3 [150 kB] 2024-02-12T15:45:37.894 INFO:teuthology.orchestra.run.smithi169.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.3 [150 kB] 2024-02-12T15:45:37.908 INFO:teuthology.orchestra.run.smithi121.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2024-02-12T15:45:37.908 INFO:teuthology.orchestra.run.smithi121.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 12.2.13-71-g707ba539-1bionic [3,316 kB] 2024-02-12T15:45:37.922 INFO:teuthology.orchestra.run.smithi169.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2024-02-12T15:45:37.922 INFO:teuthology.orchestra.run.smithi169.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 12.2.13-71-g707ba539-1bionic [14.2 MB] 2024-02-12T15:45:37.964 INFO:teuthology.orchestra.run.smithi121.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libjs-underscore all 1.8.3~dfsg-1ubuntu0.1 [60.4 kB] 2024-02-12T15:45:37.971 INFO:teuthology.orchestra.run.smithi121.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2024-02-12T15:45:37.974 INFO:teuthology.orchestra.run.smithi169.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libjs-underscore all 1.8.3~dfsg-1ubuntu0.1 [60.4 kB] 2024-02-12T15:45:37.977 INFO:teuthology.orchestra.run.smithi121.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-blinker all 1.4+dfsg1-0.1 [13.0 kB] 2024-02-12T15:45:37.978 INFO:teuthology.orchestra.run.smithi121.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2024-02-12T15:45:37.979 INFO:teuthology.orchestra.run.smithi169.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2024-02-12T15:45:37.987 INFO:teuthology.orchestra.run.smithi169.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-blinker all 1.4+dfsg1-0.1 [13.0 kB] 2024-02-12T15:45:37.987 INFO:teuthology.orchestra.run.smithi169.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2024-02-12T15:45:38.004 INFO:teuthology.orchestra.run.smithi123.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2024-02-12T15:45:38.025 INFO:teuthology.orchestra.run.smithi123.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 12.2.13-71-g707ba539-1bionic [14.2 MB] 2024-02-12T15:45:38.142 INFO:teuthology.orchestra.run.smithi121.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2024-02-12T15:45:38.152 INFO:teuthology.orchestra.run.smithi169.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2024-02-12T15:45:38.321 INFO:teuthology.orchestra.run.smithi123.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.6 [898 kB] 2024-02-12T15:45:38.322 INFO:teuthology.orchestra.run.smithi121.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 12.2.13-71-g707ba539-1bionic [3,526 kB] 2024-02-12T15:45:38.467 INFO:teuthology.orchestra.run.smithi169.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.6 [898 kB] 2024-02-12T15:45:38.467 INFO:teuthology.orchestra.run.smithi121.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.6 [898 kB] 2024-02-12T15:45:38.639 INFO:teuthology.orchestra.run.smithi123.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2024-02-12T15:45:38.651 INFO:teuthology.orchestra.run.smithi123.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2024-02-12T15:45:38.661 INFO:teuthology.orchestra.run.smithi123.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2024-02-12T15:45:38.666 INFO:teuthology.orchestra.run.smithi123.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2024-02-12T15:45:38.668 INFO:teuthology.orchestra.run.smithi123.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2024-02-12T15:45:38.684 INFO:teuthology.orchestra.run.smithi123.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2024-02-12T15:45:38.760 INFO:teuthology.orchestra.run.smithi121.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 12.2.13-71-g707ba539-1bionic [4,512 kB] 2024-02-12T15:45:38.780 INFO:teuthology.orchestra.run.smithi169.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2024-02-12T15:45:38.783 INFO:teuthology.orchestra.run.smithi121.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2024-02-12T15:45:38.796 INFO:teuthology.orchestra.run.smithi121.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2024-02-12T15:45:38.796 INFO:teuthology.orchestra.run.smithi169.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2024-02-12T15:45:38.800 INFO:teuthology.orchestra.run.smithi169.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2024-02-12T15:45:38.808 INFO:teuthology.orchestra.run.smithi169.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2024-02-12T15:45:38.809 INFO:teuthology.orchestra.run.smithi169.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2024-02-12T15:45:38.809 INFO:teuthology.orchestra.run.smithi121.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2024-02-12T15:45:38.809 INFO:teuthology.orchestra.run.smithi121.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2024-02-12T15:45:38.811 INFO:teuthology.orchestra.run.smithi121.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2024-02-12T15:45:38.820 INFO:teuthology.orchestra.run.smithi169.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2024-02-12T15:45:38.822 INFO:teuthology.orchestra.run.smithi121.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2024-02-12T15:45:39.344 INFO:teuthology.orchestra.run.smithi169.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 12.2.13-71-g707ba539-1bionic [6,136 B] 2024-02-12T15:45:39.345 INFO:teuthology.orchestra.run.smithi169.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 12.2.13-71-g707ba539-1bionic [2,459 kB] 2024-02-12T15:45:39.419 INFO:teuthology.orchestra.run.smithi121.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 12.2.13-71-g707ba539-1bionic [14.2 MB] 2024-02-12T15:45:39.531 INFO:teuthology.orchestra.run.smithi123.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 12.2.13-71-g707ba539-1bionic [6,136 B] 2024-02-12T15:45:39.531 INFO:teuthology.orchestra.run.smithi123.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 12.2.13-71-g707ba539-1bionic [2,459 kB] 2024-02-12T15:45:39.611 INFO:teuthology.orchestra.run.smithi169.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 12.2.13-71-g707ba539-1bionic [3,604 kB] 2024-02-12T15:45:39.757 INFO:teuthology.orchestra.run.smithi123.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 12.2.13-71-g707ba539-1bionic [3,604 kB] 2024-02-12T15:45:39.928 INFO:teuthology.orchestra.run.smithi169.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 12.2.13-71-g707ba539-1bionic [27.0 MB] 2024-02-12T15:45:40.140 INFO:teuthology.orchestra.run.smithi123.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 12.2.13-71-g707ba539-1bionic [27.0 MB] 2024-02-12T15:45:41.263 INFO:teuthology.orchestra.run.smithi121.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 12.2.13-71-g707ba539-1bionic [6,136 B] 2024-02-12T15:45:41.263 INFO:teuthology.orchestra.run.smithi121.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 12.2.13-71-g707ba539-1bionic [2,459 kB] 2024-02-12T15:45:41.537 INFO:teuthology.orchestra.run.smithi121.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 12.2.13-71-g707ba539-1bionic [3,604 kB] 2024-02-12T15:45:41.955 INFO:teuthology.orchestra.run.smithi121.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 12.2.13-71-g707ba539-1bionic [27.0 MB] 2024-02-12T15:45:42.488 INFO:teuthology.orchestra.run.smithi169.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 12.2.13-71-g707ba539-1bionic [18.4 kB] 2024-02-12T15:45:42.488 INFO:teuthology.orchestra.run.smithi169.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 12.2.13-71-g707ba539-1bionic [3,109 kB] 2024-02-12T15:45:42.740 INFO:teuthology.orchestra.run.smithi123.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 12.2.13-71-g707ba539-1bionic [18.4 kB] 2024-02-12T15:45:42.740 INFO:teuthology.orchestra.run.smithi123.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 12.2.13-71-g707ba539-1bionic [3,109 kB] 2024-02-12T15:45:42.784 INFO:teuthology.orchestra.run.smithi169.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 12.2.13-71-g707ba539-1bionic [15.8 kB] 2024-02-12T15:45:42.990 INFO:teuthology.orchestra.run.smithi123.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 12.2.13-71-g707ba539-1bionic [15.8 kB] 2024-02-12T15:45:43.408 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 86.6 MB in 8s (10.9 MB/s) 2024-02-12T15:45:43.545 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-02-12T15:45:43.604 INFO:teuthology.orchestra.run.smithi169.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172445 files and directories currently installed.) 2024-02-12T15:45:43.609 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../00-liblttng-ust0_2.10.1-1_amd64.deb ... 2024-02-12T15:45:43.620 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 86.6 MB in 8s (10.5 MB/s) 2024-02-12T15:45:43.686 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2024-02-12T15:45:43.789 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-02-12T15:45:43.849 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172445 files and directories currently installed.) 2024-02-12T15:45:43.855 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../00-liblttng-ust0_2.10.1-1_amd64.deb ... 2024-02-12T15:45:43.939 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2024-02-12T15:45:43.981 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../01-librbd1_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:44.098 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking librbd1 (12.2.13-71-g707ba539-1bionic) over (12.2.13-0ubuntu0.18.04.11) ... 2024-02-12T15:45:44.258 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../01-librbd1_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:44.360 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librbd1 (12.2.13-71-g707ba539-1bionic) over (12.2.13-0ubuntu0.18.04.11) ... 2024-02-12T15:45:44.588 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../02-librados2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:44.625 INFO:teuthology.orchestra.run.smithi121.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 12.2.13-71-g707ba539-1bionic [18.4 kB] 2024-02-12T15:45:44.625 INFO:teuthology.orchestra.run.smithi121.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 12.2.13-71-g707ba539-1bionic [3,109 kB] 2024-02-12T15:45:44.710 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking librados2 (12.2.13-71-g707ba539-1bionic) over (12.2.13-0ubuntu0.18.04.11) ... 2024-02-12T15:45:44.780 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../02-librados2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:44.882 INFO:teuthology.orchestra.run.smithi121.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/luminous/707ba539cff23595f0220a2367c9159498148b70/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 12.2.13-71-g707ba539-1bionic [15.8 kB] 2024-02-12T15:45:44.920 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librados2 (12.2.13-71-g707ba539-1bionic) over (12.2.13-0ubuntu0.18.04.11) ... 2024-02-12T15:45:45.256 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libcephfs2. 2024-02-12T15:45:45.272 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../03-libcephfs2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:45.307 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libcephfs2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:45.507 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 86.6 MB in 10s (8,663 kB/s) 2024-02-12T15:45:45.540 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs2. 2024-02-12T15:45:45.558 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../03-libcephfs2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:45.592 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-rados. 2024-02-12T15:45:45.598 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:45.610 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../04-python-rados_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:45.647 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-02-12T15:45:45.650 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-rados (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:45.711 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172445 files and directories currently installed.) 2024-02-12T15:45:45.717 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../00-liblttng-ust0_2.10.1-1_amd64.deb ... 2024-02-12T15:45:45.780 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2024-02-12T15:45:45.867 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-rados. 2024-02-12T15:45:45.885 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../04-python-rados_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:45.921 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-rados (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:45.986 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-cephfs. 2024-02-12T15:45:46.003 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../05-python-cephfs_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:46.046 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-cephfs (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:46.082 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../01-librbd1_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:46.208 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librbd1 (12.2.13-71-g707ba539-1bionic) over (12.2.13-0ubuntu0.18.04.11) ... 2024-02-12T15:45:46.237 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-cephfs. 2024-02-12T15:45:46.254 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../05-python-cephfs_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:46.295 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-cephfs (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:46.303 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-prettytable. 2024-02-12T15:45:46.313 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../06-python-prettytable_0.7.2-3_all.deb ... 2024-02-12T15:45:46.347 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-prettytable (0.7.2-3) ... 2024-02-12T15:45:46.599 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-rbd. 2024-02-12T15:45:46.616 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../07-python-rbd_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:46.623 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-prettytable. 2024-02-12T15:45:46.639 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../06-python-prettytable_0.7.2-3_all.deb ... 2024-02-12T15:45:46.659 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-rbd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:46.670 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../02-librados2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:46.681 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-prettytable (0.7.2-3) ... 2024-02-12T15:45:46.796 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librados2 (12.2.13-71-g707ba539-1bionic) over (12.2.13-0ubuntu0.18.04.11) ... 2024-02-12T15:45:46.925 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-rbd. 2024-02-12T15:45:46.942 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../07-python-rbd_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:46.977 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-certifi. 2024-02-12T15:45:46.983 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-rbd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:46.993 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../08-python-certifi_2018.1.18-2_all.deb ... 2024-02-12T15:45:47.036 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-certifi (2018.1.18-2) ... 2024-02-12T15:45:47.261 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-certifi. 2024-02-12T15:45:47.278 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../08-python-certifi_2018.1.18-2_all.deb ... 2024-02-12T15:45:47.312 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-certifi (2018.1.18-2) ... 2024-02-12T15:45:47.349 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs2. 2024-02-12T15:45:47.366 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../03-libcephfs2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:47.382 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-chardet. 2024-02-12T15:45:47.396 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../09-python-chardet_3.0.4-1_all.deb ... 2024-02-12T15:45:47.399 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:47.438 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-chardet (3.0.4-1) ... 2024-02-12T15:45:47.632 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-chardet. 2024-02-12T15:45:47.647 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../09-python-chardet_3.0.4-1_all.deb ... 2024-02-12T15:45:47.693 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-rados. 2024-02-12T15:45:47.697 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-chardet (3.0.4-1) ... 2024-02-12T15:45:47.710 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../04-python-rados_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:47.760 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-rados (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:47.792 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-urllib3. 2024-02-12T15:45:47.810 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../10-python-urllib3_1.22-1ubuntu0.18.04.2_all.deb ... 2024-02-12T15:45:47.842 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.2) ... 2024-02-12T15:45:48.075 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-urllib3. 2024-02-12T15:45:48.088 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-cephfs. 2024-02-12T15:45:48.091 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../10-python-urllib3_1.22-1ubuntu0.18.04.2_all.deb ... 2024-02-12T15:45:48.103 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../05-python-cephfs_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:48.128 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-requests. 2024-02-12T15:45:48.140 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.2) ... 2024-02-12T15:45:48.145 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../11-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2024-02-12T15:45:48.146 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-cephfs (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:48.178 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2024-02-12T15:45:48.398 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-prettytable. 2024-02-12T15:45:48.416 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../06-python-prettytable_0.7.2-3_all.deb ... 2024-02-12T15:45:48.431 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package librgw2. 2024-02-12T15:45:48.448 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-prettytable (0.7.2-3) ... 2024-02-12T15:45:48.449 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../12-librgw2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:48.481 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking librgw2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:48.487 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-requests. 2024-02-12T15:45:48.503 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../11-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2024-02-12T15:45:48.545 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2024-02-12T15:45:48.672 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-rbd. 2024-02-12T15:45:48.687 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../07-python-rbd_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:48.726 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-rbd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:48.867 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librgw2. 2024-02-12T15:45:48.883 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../12-librgw2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:48.923 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librgw2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:48.925 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-rgw. 2024-02-12T15:45:48.943 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../13-python-rgw_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:48.983 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-rgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:49.028 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-certifi. 2024-02-12T15:45:49.050 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../08-python-certifi_2018.1.18-2_all.deb ... 2024-02-12T15:45:49.087 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-certifi (2018.1.18-2) ... 2024-02-12T15:45:49.303 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libradosstriper1. 2024-02-12T15:45:49.323 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../14-libradosstriper1_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:49.333 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-rgw. 2024-02-12T15:45:49.352 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../13-python-rgw_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:49.361 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libradosstriper1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:49.392 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-rgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:49.440 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-chardet. 2024-02-12T15:45:49.459 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../09-python-chardet_3.0.4-1_all.deb ... 2024-02-12T15:45:49.506 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-chardet (3.0.4-1) ... 2024-02-12T15:45:49.711 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libradosstriper1. 2024-02-12T15:45:49.714 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-common. 2024-02-12T15:45:49.727 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../14-libradosstriper1_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:49.731 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../15-ceph-common_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:49.764 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-common (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:49.769 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libradosstriper1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:49.859 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-urllib3. 2024-02-12T15:45:49.878 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../10-python-urllib3_1.22-1ubuntu0.18.04.2_all.deb ... 2024-02-12T15:45:49.918 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.2) ... 2024-02-12T15:45:50.197 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-common. 2024-02-12T15:45:50.217 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../15-ceph-common_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:50.322 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-common (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:50.379 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-requests. 2024-02-12T15:45:50.398 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../11-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2024-02-12T15:45:50.439 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2024-02-12T15:45:50.740 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librgw2. 2024-02-12T15:45:50.759 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../12-librgw2_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:50.799 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librgw2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:51.170 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2024-02-12T15:45:51.184 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../16-libsnappy1v5_1.1.7-1_amd64.deb ... 2024-02-12T15:45:51.193 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-rgw. 2024-02-12T15:45:51.209 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../13-python-rgw_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:51.228 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2024-02-12T15:45:51.243 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-rgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:51.529 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2024-02-12T15:45:51.544 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../17-libleveldb1v5_1.20-2_amd64.deb ... 2024-02-12T15:45:51.554 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libradosstriper1. 2024-02-12T15:45:51.570 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../14-libradosstriper1_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:51.586 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2024-02-12T15:45:51.612 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libradosstriper1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:51.694 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2024-02-12T15:45:51.710 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../16-libsnappy1v5_1.1.7-1_amd64.deb ... 2024-02-12T15:45:51.752 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2024-02-12T15:45:52.010 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-base. 2024-02-12T15:45:52.025 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../18-ceph-base_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:52.057 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-common. 2024-02-12T15:45:52.071 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2024-02-12T15:45:52.075 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../15-ceph-common_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:52.079 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-base (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:52.087 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../17-libleveldb1v5_1.20-2_amd64.deb ... 2024-02-12T15:45:52.115 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-common (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:52.130 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2024-02-12T15:45:52.583 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-base. 2024-02-12T15:45:52.600 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../18-ceph-base_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:52.651 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-base (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:52.830 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-cherrypy3. 2024-02-12T15:45:52.845 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../19-python-cherrypy3_8.9.1-2_all.deb ... 2024-02-12T15:45:52.882 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2024-02-12T15:45:53.205 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-markupsafe. 2024-02-12T15:45:53.223 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../20-python-markupsafe_1.0-1build1_amd64.deb ... 2024-02-12T15:45:53.256 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2024-02-12T15:45:53.336 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2024-02-12T15:45:53.355 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../16-libsnappy1v5_1.1.7-1_amd64.deb ... 2024-02-12T15:45:53.379 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-cherrypy3. 2024-02-12T15:45:53.395 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2024-02-12T15:45:53.395 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../19-python-cherrypy3_8.9.1-2_all.deb ... 2024-02-12T15:45:53.439 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2024-02-12T15:45:53.524 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-jinja2. 2024-02-12T15:45:53.543 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../21-python-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2024-02-12T15:45:53.577 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-jinja2 (2.10-1ubuntu0.18.04.1) ... 2024-02-12T15:45:53.698 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2024-02-12T15:45:53.716 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../17-libleveldb1v5_1.20-2_amd64.deb ... 2024-02-12T15:45:53.756 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2024-02-12T15:45:53.787 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-markupsafe. 2024-02-12T15:45:53.803 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../20-python-markupsafe_1.0-1build1_amd64.deb ... 2024-02-12T15:45:53.842 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2024-02-12T15:45:53.925 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-openssl. 2024-02-12T15:45:53.942 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../22-python-openssl_17.5.0-1ubuntu1_all.deb ... 2024-02-12T15:45:53.975 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2024-02-12T15:45:54.127 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-jinja2. 2024-02-12T15:45:54.145 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../21-python-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2024-02-12T15:45:54.188 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-jinja2 (2.10-1ubuntu0.18.04.1) ... 2024-02-12T15:45:54.209 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-base. 2024-02-12T15:45:54.225 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../18-ceph-base_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:45:54.277 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-base (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:45:54.305 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-logutils. 2024-02-12T15:45:54.323 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../23-python-logutils_0.3.3-5_all.deb ... 2024-02-12T15:45:54.363 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-logutils (0.3.3-5) ... 2024-02-12T15:45:54.532 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-openssl. 2024-02-12T15:45:54.550 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../22-python-openssl_17.5.0-1ubuntu1_all.deb ... 2024-02-12T15:45:54.587 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2024-02-12T15:45:54.625 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-mako. 2024-02-12T15:45:54.643 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../24-python-mako_1.0.7+ds1-1ubuntu0.2_all.deb ... 2024-02-12T15:45:54.675 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-mako (1.0.7+ds1-1ubuntu0.2) ... 2024-02-12T15:45:54.940 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-logutils. 2024-02-12T15:45:54.956 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../23-python-logutils_0.3.3-5_all.deb ... 2024-02-12T15:45:54.986 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-simplegeneric. 2024-02-12T15:45:54.998 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-logutils (0.3.3-5) ... 2024-02-12T15:45:55.003 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../25-python-simplegeneric_0.8.1-1_all.deb ... 2024-02-12T15:45:55.006 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-cherrypy3. 2024-02-12T15:45:55.022 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../19-python-cherrypy3_8.9.1-2_all.deb ... 2024-02-12T15:45:55.044 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2024-02-12T15:45:55.064 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2024-02-12T15:45:55.296 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-mako. 2024-02-12T15:45:55.316 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../24-python-mako_1.0.7+ds1-1ubuntu0.2_all.deb ... 2024-02-12T15:45:55.338 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-singledispatch. 2024-02-12T15:45:55.358 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../26-python-singledispatch_3.4.0.3-2_all.deb ... 2024-02-12T15:45:55.385 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-markupsafe. 2024-02-12T15:45:55.399 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2024-02-12T15:45:55.401 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../20-python-markupsafe_1.0-1build1_amd64.deb ... 2024-02-12T15:45:55.411 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-mako (1.0.7+ds1-1ubuntu0.2) ... 2024-02-12T15:45:55.436 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2024-02-12T15:45:55.708 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-simplejson. 2024-02-12T15:45:55.724 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../27-python-simplejson_3.13.2-1_amd64.deb ... 2024-02-12T15:45:55.750 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-jinja2. 2024-02-12T15:45:55.764 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../21-python-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2024-02-12T15:45:55.808 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-jinja2 (2.10-1ubuntu0.18.04.1) ... 2024-02-12T15:45:55.816 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-simplejson (3.13.2-1) ... 2024-02-12T15:45:55.903 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-simplegeneric. 2024-02-12T15:45:55.919 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../25-python-simplegeneric_0.8.1-1_all.deb ... 2024-02-12T15:45:55.953 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2024-02-12T15:45:56.178 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-openssl. 2024-02-12T15:45:56.194 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../22-python-openssl_17.5.0-1ubuntu1_all.deb ... 2024-02-12T15:45:56.230 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2024-02-12T15:45:56.241 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-singledispatch. 2024-02-12T15:45:56.261 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../26-python-singledispatch_3.4.0.3-2_all.deb ... 2024-02-12T15:45:56.294 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-webob. 2024-02-12T15:45:56.300 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2024-02-12T15:45:56.311 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../28-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2024-02-12T15:45:56.344 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2024-02-12T15:45:56.536 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-logutils. 2024-02-12T15:45:56.551 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../23-python-logutils_0.3.3-5_all.deb ... 2024-02-12T15:45:56.569 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-simplejson. 2024-02-12T15:45:56.588 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../27-python-simplejson_3.13.2-1_amd64.deb ... 2024-02-12T15:45:56.596 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-logutils (0.3.3-5) ... 2024-02-12T15:45:56.613 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-bs4. 2024-02-12T15:45:56.627 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-simplejson (3.13.2-1) ... 2024-02-12T15:45:56.628 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../29-python-bs4_4.6.0-1_all.deb ... 2024-02-12T15:45:56.663 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-bs4 (4.6.0-1) ... 2024-02-12T15:45:56.897 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-mako. 2024-02-12T15:45:56.914 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../24-python-mako_1.0.7+ds1-1ubuntu0.2_all.deb ... 2024-02-12T15:45:56.958 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-mako (1.0.7+ds1-1ubuntu0.2) ... 2024-02-12T15:45:56.971 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-webob. 2024-02-12T15:45:56.974 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-waitress. 2024-02-12T15:45:56.988 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../28-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2024-02-12T15:45:56.993 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../30-python-waitress_1.0.1-1_all.deb ... 2024-02-12T15:45:57.024 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2024-02-12T15:45:57.032 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-waitress (1.0.1-1) ... 2024-02-12T15:45:57.276 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-simplegeneric. 2024-02-12T15:45:57.291 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../25-python-simplegeneric_0.8.1-1_all.deb ... 2024-02-12T15:45:57.316 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-bs4. 2024-02-12T15:45:57.334 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2024-02-12T15:45:57.334 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../29-python-bs4_4.6.0-1_all.deb ... 2024-02-12T15:45:57.377 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-dnspython. 2024-02-12T15:45:57.377 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-bs4 (4.6.0-1) ... 2024-02-12T15:45:57.393 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../31-python-dnspython_1.15.0-1_all.deb ... 2024-02-12T15:45:57.435 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-dnspython (1.15.0-1) ... 2024-02-12T15:45:57.640 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-singledispatch. 2024-02-12T15:45:57.643 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-waitress. 2024-02-12T15:45:57.657 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../26-python-singledispatch_3.4.0.3-2_all.deb ... 2024-02-12T15:45:57.659 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../30-python-waitress_1.0.1-1_all.deb ... 2024-02-12T15:45:57.691 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2024-02-12T15:45:57.701 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-waitress (1.0.1-1) ... 2024-02-12T15:45:57.727 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package formencode-i18n. 2024-02-12T15:45:57.743 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../32-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2024-02-12T15:45:57.786 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2024-02-12T15:45:57.994 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-simplejson. 2024-02-12T15:45:58.004 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-dnspython. 2024-02-12T15:45:58.013 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../27-python-simplejson_3.13.2-1_amd64.deb ... 2024-02-12T15:45:58.022 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../31-python-dnspython_1.15.0-1_all.deb ... 2024-02-12T15:45:58.054 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-simplejson (3.13.2-1) ... 2024-02-12T15:45:58.063 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-dnspython (1.15.0-1) ... 2024-02-12T15:45:58.097 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-formencode. 2024-02-12T15:45:58.114 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../33-python-formencode_1.3.0-0ubuntu5_all.deb ... 2024-02-12T15:45:58.155 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2024-02-12T15:45:58.357 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package formencode-i18n. 2024-02-12T15:45:58.376 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../32-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2024-02-12T15:45:58.397 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-webob. 2024-02-12T15:45:58.411 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2024-02-12T15:45:58.414 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../28-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2024-02-12T15:45:58.456 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2024-02-12T15:45:58.466 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-tempita. 2024-02-12T15:45:58.484 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../34-python-tempita_0.5.2-2_all.deb ... 2024-02-12T15:45:58.525 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-tempita (0.5.2-2) ... 2024-02-12T15:45:58.704 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-formencode. 2024-02-12T15:45:58.719 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../33-python-formencode_1.3.0-0ubuntu5_all.deb ... 2024-02-12T15:45:58.760 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2024-02-12T15:45:58.766 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-bs4. 2024-02-12T15:45:58.782 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../29-python-bs4_4.6.0-1_all.deb ... 2024-02-12T15:45:58.794 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-paste. 2024-02-12T15:45:58.813 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../35-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2024-02-12T15:45:58.825 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-bs4 (4.6.0-1) ... 2024-02-12T15:45:58.852 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2024-02-12T15:45:59.087 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-tempita. 2024-02-12T15:45:59.103 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../34-python-tempita_0.5.2-2_all.deb ... 2024-02-12T15:45:59.136 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-waitress. 2024-02-12T15:45:59.145 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-tempita (0.5.2-2) ... 2024-02-12T15:45:59.155 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../30-python-waitress_1.0.1-1_all.deb ... 2024-02-12T15:45:59.163 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-12T15:45:59.180 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2024-02-12T15:45:59.194 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-waitress (1.0.1-1) ... 2024-02-12T15:45:59.218 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2024-02-12T15:45:59.419 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-paste. 2024-02-12T15:45:59.435 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../35-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2024-02-12T15:45:59.478 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2024-02-12T15:45:59.516 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-pastedeploy. 2024-02-12T15:45:59.523 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-dnspython. 2024-02-12T15:45:59.539 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../31-python-dnspython_1.15.0-1_all.deb ... 2024-02-12T15:45:59.539 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../37-python-pastedeploy_1.5.2-4_all.deb ... 2024-02-12T15:45:59.575 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2024-02-12T15:45:59.580 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-dnspython (1.15.0-1) ... 2024-02-12T15:45:59.765 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-12T15:45:59.781 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2024-02-12T15:45:59.816 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2024-02-12T15:45:59.858 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package formencode-i18n. 2024-02-12T15:45:59.877 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../32-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2024-02-12T15:45:59.917 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-webtest. 2024-02-12T15:45:59.925 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2024-02-12T15:45:59.934 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../38-python-webtest_2.0.28-1ubuntu1_all.deb ... 2024-02-12T15:45:59.976 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2024-02-12T15:46:00.101 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pastedeploy. 2024-02-12T15:46:00.120 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../37-python-pastedeploy_1.5.2-4_all.deb ... 2024-02-12T15:46:00.159 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2024-02-12T15:46:00.223 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-formencode. 2024-02-12T15:46:00.235 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../33-python-formencode_1.3.0-0ubuntu5_all.deb ... 2024-02-12T15:46:00.269 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2024-02-12T15:46:00.328 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-pecan. 2024-02-12T15:46:00.346 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../39-python-pecan_1.2.1-2_all.deb ... 2024-02-12T15:46:00.414 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-pecan (1.2.1-2) ... 2024-02-12T15:46:00.495 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-webtest. 2024-02-12T15:46:00.514 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../38-python-webtest_2.0.28-1ubuntu1_all.deb ... 2024-02-12T15:46:00.547 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-tempita. 2024-02-12T15:46:00.553 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2024-02-12T15:46:00.563 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../34-python-tempita_0.5.2-2_all.deb ... 2024-02-12T15:46:00.597 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-tempita (0.5.2-2) ... 2024-02-12T15:46:00.769 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-werkzeug. 2024-02-12T15:46:00.784 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../40-python-werkzeug_0.14.1+dfsg1-1ubuntu0.2_all.deb ... 2024-02-12T15:46:00.828 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1ubuntu0.2) ... 2024-02-12T15:46:00.917 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-paste. 2024-02-12T15:46:00.917 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pecan. 2024-02-12T15:46:00.932 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../39-python-pecan_1.2.1-2_all.deb ... 2024-02-12T15:46:00.933 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../35-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2024-02-12T15:46:00.975 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2024-02-12T15:46:00.976 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pecan (1.2.1-2) ... 2024-02-12T15:46:01.223 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mgr. 2024-02-12T15:46:01.240 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../41-ceph-mgr_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:01.280 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mgr (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:01.301 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-12T15:46:01.314 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-werkzeug. 2024-02-12T15:46:01.318 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2024-02-12T15:46:01.333 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../40-python-werkzeug_0.14.1+dfsg1-1ubuntu0.2_all.deb ... 2024-02-12T15:46:01.360 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2024-02-12T15:46:01.373 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1ubuntu0.2) ... 2024-02-12T15:46:01.654 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pastedeploy. 2024-02-12T15:46:01.673 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../37-python-pastedeploy_1.5.2-4_all.deb ... 2024-02-12T15:46:01.712 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2024-02-12T15:46:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr. 2024-02-12T15:46:01.818 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../41-ceph-mgr_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:01.860 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:01.953 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-colorama. 2024-02-12T15:46:01.968 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../42-python-colorama_0.3.7-1_all.deb ... 2024-02-12T15:46:01.982 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-webtest. 2024-02-12T15:46:01.999 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../38-python-webtest_2.0.28-1ubuntu1_all.deb ... 2024-02-12T15:46:02.010 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-colorama (0.3.7-1) ... 2024-02-12T15:46:02.040 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2024-02-12T15:46:02.321 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-click. 2024-02-12T15:46:02.340 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../43-python-click_6.7-3_all.deb ... 2024-02-12T15:46:02.379 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-click (6.7-3) ... 2024-02-12T15:46:02.410 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pecan. 2024-02-12T15:46:02.426 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../39-python-pecan_1.2.1-2_all.deb ... 2024-02-12T15:46:02.469 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-colorama. 2024-02-12T15:46:02.472 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pecan (1.2.1-2) ... 2024-02-12T15:46:02.486 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../42-python-colorama_0.3.7-1_all.deb ... 2024-02-12T15:46:02.527 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-colorama (0.3.7-1) ... 2024-02-12T15:46:02.682 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-itsdangerous. 2024-02-12T15:46:02.701 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../44-python-itsdangerous_0.24+dfsg1-2_all.deb ... 2024-02-12T15:46:02.740 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-itsdangerous (0.24+dfsg1-2) ... 2024-02-12T15:46:02.821 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-click. 2024-02-12T15:46:02.840 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../43-python-click_6.7-3_all.deb ... 2024-02-12T15:46:02.849 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-werkzeug. 2024-02-12T15:46:02.868 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../40-python-werkzeug_0.14.1+dfsg1-1ubuntu0.2_all.deb ... 2024-02-12T15:46:02.879 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-click (6.7-3) ... 2024-02-12T15:46:02.908 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1ubuntu0.2) ... 2024-02-12T15:46:03.026 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-flask. 2024-02-12T15:46:03.042 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../45-python-flask_0.12.2-3ubuntu0.1_all.deb ... 2024-02-12T15:46:03.076 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-flask (0.12.2-3ubuntu0.1) ... 2024-02-12T15:46:03.173 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-itsdangerous. 2024-02-12T15:46:03.193 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../44-python-itsdangerous_0.24+dfsg1-2_all.deb ... 2024-02-12T15:46:03.232 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-itsdangerous (0.24+dfsg1-2) ... 2024-02-12T15:46:03.309 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr. 2024-02-12T15:46:03.326 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../41-ceph-mgr_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:03.368 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:03.475 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mon. 2024-02-12T15:46:03.492 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../46-ceph-mon_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:03.535 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-flask. 2024-02-12T15:46:03.541 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mon (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:03.554 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../45-python-flask_0.12.2-3ubuntu0.1_all.deb ... 2024-02-12T15:46:03.594 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-flask (0.12.2-3ubuntu0.1) ... 2024-02-12T15:46:03.920 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mon. 2024-02-12T15:46:03.940 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../46-ceph-mon_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:03.970 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mon (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:03.991 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-colorama. 2024-02-12T15:46:04.010 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../42-python-colorama_0.3.7-1_all.deb ... 2024-02-12T15:46:04.050 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-colorama (0.3.7-1) ... 2024-02-12T15:46:04.340 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-osd. 2024-02-12T15:46:04.356 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../47-ceph-osd_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:04.361 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-click. 2024-02-12T15:46:04.378 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../43-python-click_6.7-3_all.deb ... 2024-02-12T15:46:04.398 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-osd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:04.419 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-click (6.7-3) ... 2024-02-12T15:46:04.739 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-itsdangerous. 2024-02-12T15:46:04.754 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../44-python-itsdangerous_0.24+dfsg1-2_all.deb ... 2024-02-12T15:46:04.791 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-osd. 2024-02-12T15:46:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-itsdangerous (0.24+dfsg1-2) ... 2024-02-12T15:46:04.807 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../47-ceph-osd_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:04.874 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-osd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:05.091 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-flask. 2024-02-12T15:46:05.107 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../45-python-flask_0.12.2-3ubuntu0.1_all.deb ... 2024-02-12T15:46:05.149 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-flask (0.12.2-3ubuntu0.1) ... 2024-02-12T15:46:05.686 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph. 2024-02-12T15:46:05.701 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../48-ceph_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:05.736 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:06.320 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mon. 2024-02-12T15:46:06.337 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../46-ceph-mon_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:06.378 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mon (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:06.848 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-fuse. 2024-02-12T15:46:06.866 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../49-ceph-fuse_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:06.874 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph. 2024-02-12T15:46:06.893 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../48-ceph_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:06.905 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:06.932 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:07.098 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-osd. 2024-02-12T15:46:07.116 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../47-ceph-osd_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:07.156 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-osd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:07.237 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-fuse. 2024-02-12T15:46:07.253 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../49-ceph-fuse_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:07.288 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:07.500 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-mds. 2024-02-12T15:46:07.516 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../50-ceph-mds_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:07.558 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-mds (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:07.874 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mds. 2024-02-12T15:46:07.891 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../50-ceph-mds_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:07.932 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mds (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:08.211 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libonig4:amd64. 2024-02-12T15:46:08.230 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../51-libonig4_6.7.0-1_amd64.deb ... 2024-02-12T15:46:08.269 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2024-02-12T15:46:08.444 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph. 2024-02-12T15:46:08.460 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../48-ceph_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:08.502 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:08.572 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-12T15:46:08.585 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libonig4:amd64. 2024-02-12T15:46:08.590 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../52-libjq1_1.5+dfsg-2_amd64.deb ... 2024-02-12T15:46:08.604 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../51-libonig4_6.7.0-1_amd64.deb ... 2024-02-12T15:46:08.632 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2024-02-12T15:46:08.643 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2024-02-12T15:46:08.860 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-fuse. 2024-02-12T15:46:08.878 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../49-ceph-fuse_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:08.907 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package jq. 2024-02-12T15:46:08.923 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../53-jq_1.5+dfsg-2_amd64.deb ... 2024-02-12T15:46:08.927 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:08.966 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking jq (1.5+dfsg-2) ... 2024-02-12T15:46:08.996 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-12T15:46:09.011 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../52-libjq1_1.5+dfsg-2_amd64.deb ... 2024-02-12T15:46:09.046 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2024-02-12T15:46:09.227 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package socat. 2024-02-12T15:46:09.246 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../54-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2024-02-12T15:46:09.285 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2024-02-12T15:46:09.306 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package jq. 2024-02-12T15:46:09.323 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../53-jq_1.5+dfsg-2_amd64.deb ... 2024-02-12T15:46:09.365 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking jq (1.5+dfsg-2) ... 2024-02-12T15:46:09.507 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mds. 2024-02-12T15:46:09.525 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../50-ceph-mds_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:09.565 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mds (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:09.609 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package socat. 2024-02-12T15:46:09.626 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../54-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2024-02-12T15:46:09.667 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2024-02-12T15:46:09.671 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libxslt1.1:amd64. 2024-02-12T15:46:09.688 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../55-libxslt1.1_1.1.29-5ubuntu0.3_amd64.deb ... 2024-02-12T15:46:09.729 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.3) ... 2024-02-12T15:46:10.015 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package xmlstarlet. 2024-02-12T15:46:10.032 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../56-xmlstarlet_1.6.1-2_amd64.deb ... 2024-02-12T15:46:10.046 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libxslt1.1:amd64. 2024-02-12T15:46:10.062 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../55-libxslt1.1_1.1.29-5ubuntu0.3_amd64.deb ... 2024-02-12T15:46:10.065 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2024-02-12T15:46:10.103 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.3) ... 2024-02-12T15:46:10.227 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libonig4:amd64. 2024-02-12T15:46:10.243 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../51-libonig4_6.7.0-1_amd64.deb ... 2024-02-12T15:46:10.286 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2024-02-12T15:46:10.381 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package xmlstarlet. 2024-02-12T15:46:10.398 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../56-xmlstarlet_1.6.1-2_amd64.deb ... 2024-02-12T15:46:10.431 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package ceph-test. 2024-02-12T15:46:10.439 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2024-02-12T15:46:10.444 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../57-ceph-test_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:10.486 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking ceph-test (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:10.613 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-12T15:46:10.632 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../52-libjq1_1.5+dfsg-2_amd64.deb ... 2024-02-12T15:46:10.690 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2024-02-12T15:46:10.793 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-test. 2024-02-12T15:46:10.811 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../57-ceph-test_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:10.851 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-test (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:10.890 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package jq. 2024-02-12T15:46:10.907 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../53-jq_1.5+dfsg-2_amd64.deb ... 2024-02-12T15:46:10.949 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking jq (1.5+dfsg-2) ... 2024-02-12T15:46:11.218 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package socat. 2024-02-12T15:46:11.244 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../54-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2024-02-12T15:46:11.285 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2024-02-12T15:46:11.671 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libxslt1.1:amd64. 2024-02-12T15:46:11.687 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../55-libxslt1.1_1.1.29-5ubuntu0.3_amd64.deb ... 2024-02-12T15:46:11.730 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.3) ... 2024-02-12T15:46:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package xmlstarlet. 2024-02-12T15:46:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../56-xmlstarlet_1.6.1-2_amd64.deb ... 2024-02-12T15:46:12.032 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2024-02-12T15:46:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-test. 2024-02-12T15:46:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../57-ceph-test_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-test (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:12.680 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-12T15:46:12.697 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../58-libcephfs-dev_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:12.738 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libcephfs-dev (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:12.999 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libjs-underscore. 2024-02-12T15:46:13.020 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../59-libjs-underscore_1.8.3~dfsg-1ubuntu0.1_all.deb ... 2024-02-12T15:46:13.057 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1ubuntu0.1) ... 2024-02-12T15:46:13.079 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-12T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../58-libcephfs-dev_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:13.137 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs-dev (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:13.344 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-02-12T15:46:13.359 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../60-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2024-02-12T15:46:13.381 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libjs-underscore. 2024-02-12T15:46:13.400 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../59-libjs-underscore_1.8.3~dfsg-1ubuntu0.1_all.deb ... 2024-02-12T15:46:13.401 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2024-02-12T15:46:13.439 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1ubuntu0.1) ... 2024-02-12T15:46:13.716 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-02-12T15:46:13.732 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../60-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2024-02-12T15:46:13.745 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-blinker. 2024-02-12T15:46:13.763 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../61-python-blinker_1.4+dfsg1-0.1_all.deb ... 2024-02-12T15:46:13.775 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2024-02-12T15:46:13.804 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-blinker (1.4+dfsg1-0.1) ... 2024-02-12T15:46:14.065 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-webencodings. 2024-02-12T15:46:14.077 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-blinker. 2024-02-12T15:46:14.083 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../62-python-webencodings_0.5-2_all.deb ... 2024-02-12T15:46:14.096 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../61-python-blinker_1.4+dfsg1-0.1_all.deb ... 2024-02-12T15:46:14.115 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-webencodings (0.5-2) ... 2024-02-12T15:46:14.135 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-blinker (1.4+dfsg1-0.1) ... 2024-02-12T15:46:14.417 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-html5lib. 2024-02-12T15:46:14.421 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-webencodings. 2024-02-12T15:46:14.436 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../63-python-html5lib_0.999999999-1_all.deb ... 2024-02-12T15:46:14.440 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../62-python-webencodings_0.5-2_all.deb ... 2024-02-12T15:46:14.476 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-html5lib (0.999999999-1) ... 2024-02-12T15:46:14.479 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-webencodings (0.5-2) ... 2024-02-12T15:46:14.540 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-12T15:46:14.555 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../58-libcephfs-dev_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:14.597 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs-dev (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:14.732 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-html5lib. 2024-02-12T15:46:14.751 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../63-python-html5lib_0.999999999-1_all.deb ... 2024-02-12T15:46:14.782 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-html5lib (0.999999999-1) ... 2024-02-12T15:46:14.812 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-lxml:amd64. 2024-02-12T15:46:14.828 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../64-python-lxml_4.2.1-1ubuntu0.6_amd64.deb ... 2024-02-12T15:46:14.858 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-underscore. 2024-02-12T15:46:14.870 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.6) ... 2024-02-12T15:46:14.874 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../59-libjs-underscore_1.8.3~dfsg-1ubuntu0.1_all.deb ... 2024-02-12T15:46:14.916 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1ubuntu0.1) ... 2024-02-12T15:46:15.135 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-lxml:amd64. 2024-02-12T15:46:15.151 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../64-python-lxml_4.2.1-1ubuntu0.6_amd64.deb ... 2024-02-12T15:46:15.193 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.6) ... 2024-02-12T15:46:15.210 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-02-12T15:46:15.227 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../60-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2024-02-12T15:46:15.268 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2024-02-12T15:46:15.315 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-openid. 2024-02-12T15:46:15.332 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../65-python-openid_2.2.5-6_all.deb ... 2024-02-12T15:46:15.373 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-openid (2.2.5-6) ... 2024-02-12T15:46:15.621 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-openid. 2024-02-12T15:46:15.621 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-blinker. 2024-02-12T15:46:15.637 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../65-python-openid_2.2.5-6_all.deb ... 2024-02-12T15:46:15.638 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../61-python-blinker_1.4+dfsg1-0.1_all.deb ... 2024-02-12T15:46:15.679 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-openid (2.2.5-6) ... 2024-02-12T15:46:15.679 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-blinker (1.4+dfsg1-0.1) ... 2024-02-12T15:46:15.707 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-pastescript. 2024-02-12T15:46:15.726 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../66-python-pastescript_2.0.2-2_all.deb ... 2024-02-12T15:46:15.765 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-pastescript (2.0.2-2) ... 2024-02-12T15:46:15.949 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-webencodings. 2024-02-12T15:46:15.968 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../62-python-webencodings_0.5-2_all.deb ... 2024-02-12T15:46:16.007 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-webencodings (0.5-2) ... 2024-02-12T15:46:16.068 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pastescript. 2024-02-12T15:46:16.069 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-pyinotify. 2024-02-12T15:46:16.086 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../66-python-pastescript_2.0.2-2_all.deb ... 2024-02-12T15:46:16.089 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../67-python-pyinotify_0.9.6-1_all.deb ... 2024-02-12T15:46:16.126 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pastescript (2.0.2-2) ... 2024-02-12T15:46:16.127 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2024-02-12T15:46:16.310 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-html5lib. 2024-02-12T15:46:16.326 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../63-python-html5lib_0.999999999-1_all.deb ... 2024-02-12T15:46:16.368 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-html5lib (0.999999999-1) ... 2024-02-12T15:46:16.437 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pyinotify. 2024-02-12T15:46:16.453 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../67-python-pyinotify_0.9.6-1_all.deb ... 2024-02-12T15:46:16.454 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-repoze.lru. 2024-02-12T15:46:16.473 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../68-python-repoze.lru_0.7-1_all.deb ... 2024-02-12T15:46:16.495 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2024-02-12T15:46:16.507 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-repoze.lru (0.7-1) ... 2024-02-12T15:46:16.688 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-lxml:amd64. 2024-02-12T15:46:16.707 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../64-python-lxml_4.2.1-1ubuntu0.6_amd64.deb ... 2024-02-12T15:46:16.746 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.6) ... 2024-02-12T15:46:16.768 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-repoze.lru. 2024-02-12T15:46:16.783 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../68-python-repoze.lru_0.7-1_all.deb ... 2024-02-12T15:46:16.820 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-routes. 2024-02-12T15:46:16.822 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-repoze.lru (0.7-1) ... 2024-02-12T15:46:16.838 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../69-python-routes_2.4.1-1_all.deb ... 2024-02-12T15:46:16.879 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-routes (2.4.1-1) ... 2024-02-12T15:46:17.106 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-routes. 2024-02-12T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../69-python-routes_2.4.1-1_all.deb ... 2024-02-12T15:46:17.165 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-routes (2.4.1-1) ... 2024-02-12T15:46:17.176 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-openid. 2024-02-12T15:46:17.191 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../65-python-openid_2.2.5-6_all.deb ... 2024-02-12T15:46:17.222 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package python-scgi. 2024-02-12T15:46:17.232 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-openid (2.2.5-6) ... 2024-02-12T15:46:17.238 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../70-python-scgi_1.13-1.1build1_amd64.deb ... 2024-02-12T15:46:17.280 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2024-02-12T15:46:17.485 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-scgi. 2024-02-12T15:46:17.503 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../70-python-scgi_1.13-1.1build1_amd64.deb ... 2024-02-12T15:46:17.527 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pastescript. 2024-02-12T15:46:17.543 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2024-02-12T15:46:17.545 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../66-python-pastescript_2.0.2-2_all.deb ... 2024-02-12T15:46:17.577 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pastescript (2.0.2-2) ... 2024-02-12T15:46:17.645 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package radosgw. 2024-02-12T15:46:17.661 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../71-radosgw_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:17.703 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking radosgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:17.877 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package radosgw. 2024-02-12T15:46:17.882 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pyinotify. 2024-02-12T15:46:17.897 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../71-radosgw_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:17.897 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../67-python-pyinotify_0.9.6-1_all.deb ... 2024-02-12T15:46:17.935 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking radosgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:17.939 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2024-02-12T15:46:18.216 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-repoze.lru. 2024-02-12T15:46:18.236 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../68-python-repoze.lru_0.7-1_all.deb ... 2024-02-12T15:46:18.252 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package rbd-fuse. 2024-02-12T15:46:18.268 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../72-rbd-fuse_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:18.275 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-repoze.lru (0.7-1) ... 2024-02-12T15:46:18.311 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking rbd-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:18.480 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package rbd-fuse. 2024-02-12T15:46:18.496 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../72-rbd-fuse_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:18.533 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking rbd-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:18.590 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-routes. 2024-02-12T15:46:18.590 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-dnspython (1.15.0-1) ... 2024-02-12T15:46:18.606 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../69-python-routes_2.4.1-1_all.deb ... 2024-02-12T15:46:18.648 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-routes (2.4.1-1) ... 2024-02-12T15:46:18.842 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-simplejson (3.13.2-1) ... 2024-02-12T15:46:18.844 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-dnspython (1.15.0-1) ... 2024-02-12T15:46:18.979 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-scgi. 2024-02-12T15:46:18.995 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../70-python-scgi_1.13-1.1build1_amd64.deb ... 2024-02-12T15:46:19.037 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2024-02-12T15:46:19.053 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-repoze.lru (0.7-1) ... 2024-02-12T15:46:19.110 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-simplejson (3.13.2-1) ... 2024-02-12T15:46:19.254 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-openid (2.2.5-6) ... 2024-02-12T15:46:19.338 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-repoze.lru (0.7-1) ... 2024-02-12T15:46:19.391 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package radosgw. 2024-02-12T15:46:19.407 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../71-radosgw_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:19.445 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking radosgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:19.480 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-prettytable (0.7.2-3) ... 2024-02-12T15:46:19.542 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-openid (2.2.5-6) ... 2024-02-12T15:46:19.656 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.2) ... 2024-02-12T15:46:19.769 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-prettytable (0.7.2-3) ... 2024-02-12T15:46:19.883 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2024-02-12T15:46:19.962 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.2) ... 2024-02-12T15:46:20.000 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package rbd-fuse. 2024-02-12T15:46:20.020 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../72-rbd-fuse_12.2.13-71-g707ba539-1bionic_amd64.deb ... 2024-02-12T15:46:20.059 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking rbd-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:20.076 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-chardet (3.0.4-1) ... 2024-02-12T15:46:20.180 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2024-02-12T15:46:20.349 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-dnspython (1.15.0-1) ... 2024-02-12T15:46:20.373 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-chardet (3.0.4-1) ... 2024-02-12T15:46:20.377 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-blinker (1.4+dfsg1-0.1) ... 2024-02-12T15:46:20.569 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libjs-underscore (1.8.3~dfsg-1ubuntu0.1) ... 2024-02-12T15:46:20.601 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-simplejson (3.13.2-1) ... 2024-02-12T15:46:20.640 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-blinker (1.4+dfsg1-0.1) ... 2024-02-12T15:46:20.695 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2024-02-12T15:46:20.833 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libjs-underscore (1.8.3~dfsg-1ubuntu0.1) ... 2024-02-12T15:46:20.897 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2024-02-12T15:46:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2024-02-12T15:46:20.998 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2024-02-12T15:46:21.079 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-repoze.lru (0.7-1) ... 2024-02-12T15:46:21.123 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-certifi (2018.1.18-2) ... 2024-02-12T15:46:21.169 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2024-02-12T15:46:21.282 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-openid (2.2.5-6) ... 2024-02-12T15:46:21.286 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2024-02-12T15:46:21.316 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:21.396 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-certifi (2018.1.18-2) ... 2024-02-12T15:46:21.519 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-prettytable (0.7.2-3) ... 2024-02-12T15:46:21.534 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-12T15:46:21.534 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-12T15:46:21.564 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:21.745 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.2) ... 2024-02-12T15:46:21.784 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-12T15:46:21.785 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-12T15:46:21.836 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2024-02-12T15:46:21.963 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2024-02-12T15:46:21.972 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2024-02-12T15:46:22.057 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2024-02-12T15:46:22.080 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2024-02-12T15:46:22.156 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-chardet (3.0.4-1) ... 2024-02-12T15:46:22.182 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2024-02-12T15:46:22.290 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-colorama (0.3.7-1) ... 2024-02-12T15:46:22.308 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2024-02-12T15:46:22.424 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-blinker (1.4+dfsg1-0.1) ... 2024-02-12T15:46:22.474 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-waitress (1.0.1-1) ... 2024-02-12T15:46:22.543 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-colorama (0.3.7-1) ... 2024-02-12T15:46:22.617 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-underscore (1.8.3~dfsg-1ubuntu0.1) ... 2024-02-12T15:46:22.736 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2024-02-12T15:46:22.745 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-waitress (1.0.1-1) ... 2024-02-12T15:46:22.944 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2024-02-12T15:46:22.946 INFO:teuthology.orchestra.run.smithi169.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2024-02-12T15:46:23.070 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2024-02-12T15:46:23.105 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2024-02-12T15:46:23.127 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-markupsafe (1.0-1build1) ... 2024-02-12T15:46:23.197 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-certifi (2018.1.18-2) ... 2024-02-12T15:46:23.288 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-markupsafe (1.0-1build1) ... 2024-02-12T15:46:23.329 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1ubuntu0.2) ... 2024-02-12T15:46:23.390 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:23.481 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1ubuntu0.2) ... 2024-02-12T15:46:23.572 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-pyinotify (0.9.6-1) ... 2024-02-12T15:46:23.606 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-12T15:46:23.606 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-12T15:46:23.741 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pyinotify (0.9.6-1) ... 2024-02-12T15:46:23.748 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-scgi (1.13-1.1build1) ... 2024-02-12T15:46:23.933 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.3) ... 2024-02-12T15:46:23.942 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-scgi (1.13-1.1build1) ... 2024-02-12T15:46:23.966 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2024-02-12T15:46:24.059 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2024-02-12T15:46:24.092 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2024-02-12T15:46:24.143 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.3) ... 2024-02-12T15:46:24.201 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2024-02-12T15:46:24.202 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-bs4 (4.6.0-1) ... 2024-02-12T15:46:24.269 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2024-02-12T15:46:24.387 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-bs4 (4.6.0-1) ... 2024-02-12T15:46:24.419 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-colorama (0.3.7-1) ... 2024-02-12T15:46:24.436 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-mako (1.0.7+ds1-1ubuntu0.2) ... 2024-02-12T15:46:24.604 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-mako (1.0.7+ds1-1ubuntu0.2) ... 2024-02-12T15:46:24.612 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-waitress (1.0.1-1) ... 2024-02-12T15:46:24.654 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2024-02-12T15:46:24.772 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.6) ... 2024-02-12T15:46:24.847 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2024-02-12T15:46:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.6) ... 2024-02-12T15:46:24.981 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2024-02-12T15:46:24.996 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2024-02-12T15:46:25.174 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2024-02-12T15:46:25.181 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-markupsafe (1.0-1build1) ... 2024-02-12T15:46:25.191 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2024-02-12T15:46:25.366 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1ubuntu0.2) ... 2024-02-12T15:46:25.366 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2024-02-12T15:46:25.400 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2024-02-12T15:46:25.492 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-itsdangerous (0.24+dfsg1-2) ... 2024-02-12T15:46:25.601 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2024-02-12T15:46:25.634 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pyinotify (0.9.6-1) ... 2024-02-12T15:46:25.685 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-tempita (0.5.2-2) ... 2024-02-12T15:46:25.728 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-itsdangerous (0.24+dfsg1-2) ... 2024-02-12T15:46:25.810 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-scgi (1.13-1.1build1) ... 2024-02-12T15:46:25.895 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2024-02-12T15:46:25.920 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-tempita (0.5.2-2) ... 2024-02-12T15:46:25.986 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.3) ... 2024-02-12T15:46:26.021 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-webencodings (0.5-2) ... 2024-02-12T15:46:26.112 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2024-02-12T15:46:26.208 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2024-02-12T15:46:26.238 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-bs4 (4.6.0-1) ... 2024-02-12T15:46:26.346 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2024-02-12T15:46:26.447 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-webencodings (0.5-2) ... 2024-02-12T15:46:26.448 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-logutils (0.3.3-5) ... 2024-02-12T15:46:26.465 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-mako (1.0.7+ds1-1ubuntu0.2) ... 2024-02-12T15:46:26.639 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2024-02-12T15:46:26.682 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2024-02-12T15:46:26.808 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.6) ... 2024-02-12T15:46:26.899 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-logutils (0.3.3-5) ... 2024-02-12T15:46:26.916 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-routes (2.4.1-1) ... 2024-02-12T15:46:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2024-02-12T15:46:27.100 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-routes (2.4.1-1) ... 2024-02-12T15:46:27.134 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2024-02-12T15:46:27.211 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2024-02-12T15:46:27.301 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2024-02-12T15:46:27.360 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up jq (1.5+dfsg-2) ... 2024-02-12T15:46:27.387 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2024-02-12T15:46:27.486 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-jinja2 (2.10-1ubuntu0.18.04.1) ... 2024-02-12T15:46:27.521 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-itsdangerous (0.24+dfsg1-2) ... 2024-02-12T15:46:27.535 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up jq (1.5+dfsg-2) ... 2024-02-12T15:46:27.661 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-jinja2 (2.10-1ubuntu0.18.04.1) ... 2024-02-12T15:46:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-tempita (0.5.2-2) ... 2024-02-12T15:46:27.720 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-click (6.7-3) ... 2024-02-12T15:46:27.879 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-click (6.7-3) ... 2024-02-12T15:46:27.915 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2024-02-12T15:46:27.938 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2024-02-12T15:46:28.041 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-webencodings (0.5-2) ... 2024-02-12T15:46:28.080 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2024-02-12T15:46:28.231 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2024-02-12T15:46:28.233 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2024-02-12T15:46:28.349 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up xmlstarlet (1.6.1-2) ... 2024-02-12T15:46:28.373 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2024-02-12T15:46:28.475 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up librados2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:28.485 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-logutils (0.3.3-5) ... 2024-02-12T15:46:28.499 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up xmlstarlet (1.6.1-2) ... 2024-02-12T15:46:28.601 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libcephfs2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:28.625 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librados2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:28.661 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-routes (2.4.1-1) ... 2024-02-12T15:46:28.727 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libcephfs-dev (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:28.752 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:28.853 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-html5lib (0.999999999-1) ... 2024-02-12T15:46:28.877 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs-dev (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:28.879 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2024-02-12T15:46:29.003 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-html5lib (0.999999999-1) ... 2024-02-12T15:46:29.105 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up jq (1.5+dfsg-2) ... 2024-02-12T15:46:29.121 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-flask (0.12.2-3ubuntu0.1) ... 2024-02-12T15:46:29.231 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-jinja2 (2.10-1ubuntu0.18.04.1) ... 2024-02-12T15:46:29.254 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-flask (0.12.2-3ubuntu0.1) ... 2024-02-12T15:46:29.331 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-rados (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.440 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-cephfs (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.456 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-rados (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.465 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-click (6.7-3) ... 2024-02-12T15:46:29.564 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-cephfs (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.633 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up libradosstriper1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.675 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2024-02-12T15:46:29.759 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up librgw2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.766 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libradosstriper1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.885 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-rgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.891 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librgw2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:29.959 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2024-02-12T15:46:30.012 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2024-02-12T15:46:30.017 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-rgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.077 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up xmlstarlet (1.6.1-2) ... 2024-02-12T15:46:30.144 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2024-02-12T15:46:30.186 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librados2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.212 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up librbd1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.336 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librbd1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.363 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-pastescript (2.0.2-2) ... 2024-02-12T15:46:30.447 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs-dev (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.479 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pastescript (2.0.2-2) ... 2024-02-12T15:46:30.573 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-rbd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.573 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-html5lib (0.999999999-1) ... 2024-02-12T15:46:30.682 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-common (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.688 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-rbd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.789 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-common (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:30.841 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-flask (0.12.2-3ubuntu0.1) ... 2024-02-12T15:46:31.050 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-rados (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:31.176 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-cephfs (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:31.259 INFO:teuthology.orchestra.run.smithi169.stdout:Adding group ceph....done 2024-02-12T15:46:31.361 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libradosstriper1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:31.366 INFO:teuthology.orchestra.run.smithi123.stdout:Adding group ceph....done 2024-02-12T15:46:31.462 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librgw2 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:31.588 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-rgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:31.678 INFO:teuthology.orchestra.run.smithi169.stdout:Adding system user ceph....done 2024-02-12T15:46:31.714 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2024-02-12T15:46:31.770 INFO:teuthology.orchestra.run.smithi169.stdout:Setting system user ceph properties....done 2024-02-12T15:46:31.801 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user ceph....done 2024-02-12T15:46:31.892 INFO:teuthology.orchestra.run.smithi123.stdout:Setting system user ceph properties....done 2024-02-12T15:46:31.915 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librbd1 (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:32.001 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-12T15:46:32.041 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pastescript (2.0.2-2) ... 2024-02-12T15:46:32.117 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-12T15:46:32.242 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-12T15:46:32.259 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-rbd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:32.324 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-12T15:46:32.385 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-common (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:32.542 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up radosgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:32.615 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up radosgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:32.937 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-12T15:46:32.938 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-12T15:46:32.953 INFO:teuthology.orchestra.run.smithi121.stdout:Adding group ceph....done 2024-02-12T15:46:33.002 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-12T15:46:33.002 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-12T15:46:33.337 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up rbd-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:33.373 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user ceph....done 2024-02-12T15:46:33.418 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up rbd-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:33.463 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2024-02-12T15:46:33.463 INFO:teuthology.orchestra.run.smithi121.stdout:Setting system user ceph properties....done 2024-02-12T15:46:33.544 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2024-02-12T15:46:33.683 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-12T15:46:33.697 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-test (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:33.754 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-test (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:33.823 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-base (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:33.879 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-base (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:33.900 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-12T15:46:34.142 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up python-pecan (1.2.1-2) ... 2024-02-12T15:46:34.195 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up radosgw (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:34.231 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pecan (1.2.1-2) ... 2024-02-12T15:46:34.232 INFO:teuthology.orchestra.run.smithi169.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2024-02-12T15:46:34.264 INFO:teuthology.orchestra.run.smithi169.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2024-02-12T15:46:34.305 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2024-02-12T15:46:34.328 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2024-02-12T15:46:34.462 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-osd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:34.534 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-osd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:34.603 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-12T15:46:34.603 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-12T15:46:34.915 INFO:teuthology.orchestra.run.smithi169.stdout:chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory 2024-02-12T15:46:34.989 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up rbd-fuse (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:35.007 INFO:teuthology.orchestra.run.smithi123.stdout:chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory 2024-02-12T15:46:35.115 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2024-02-12T15:46:35.180 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-12T15:46:35.180 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-12T15:46:35.275 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-12T15:46:35.275 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-12T15:46:35.325 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-test (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:35.451 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-base (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:35.465 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mds (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:35.554 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mds (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:35.803 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pecan (1.2.1-2) ... 2024-02-12T15:46:35.805 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-12T15:46:35.805 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-12T15:46:35.885 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2024-02-12T15:46:35.908 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-12T15:46:35.909 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-12T15:46:35.909 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2024-02-12T15:46:36.077 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mon (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:36.098 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-osd (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:36.173 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mon (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:36.425 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-12T15:46:36.425 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-12T15:46:36.504 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-12T15:46:36.505 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-12T15:46:36.551 INFO:teuthology.orchestra.run.smithi121.stdout:chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory 2024-02-12T15:46:36.721 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph-mgr (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:36.800 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:36.810 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-12T15:46:36.810 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-12T15:46:37.076 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mds (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:37.170 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-12T15:46:37.170 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-12T15:46:37.264 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-12T15:46:37.264 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-12T15:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-12T15:46:37.416 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-12T15:46:37.451 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up ceph (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:37.541 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:37.550 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for systemd (237-3ubuntu10.56) ... 2024-02-12T15:46:37.663 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for systemd (237-3ubuntu10.56) ... 2024-02-12T15:46:37.704 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mon (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:37.726 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2024-02-12T15:46:37.855 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2024-02-12T15:46:38.042 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-12T15:46:38.042 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-12T15:46:38.339 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:38.730 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2024-02-12T15:46:38.803 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-12T15:46:38.803 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-12T15:46:38.831 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.6) ... 2024-02-12T15:46:38.850 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2024-02-12T15:46:38.951 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.6) ... 2024-02-12T15:46:39.092 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph (12.2.13-71-g707ba539-1bionic) ... 2024-02-12T15:46:39.219 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for systemd (237-3ubuntu10.56) ... 2024-02-12T15:46:39.311 INFO:teuthology.orchestra.run.smithi169.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-12T15:46:39.317 DEBUG:teuthology.parallel:result is None 2024-02-12T15:46:39.420 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2024-02-12T15:46:39.446 INFO:teuthology.orchestra.run.smithi123.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-12T15:46:39.450 DEBUG:teuthology.parallel:result is None 2024-02-12T15:46:40.440 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2024-02-12T15:46:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.6) ... 2024-02-12T15:46:41.030 INFO:teuthology.orchestra.run.smithi121.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-12T15:46:41.034 DEBUG:teuthology.parallel:result is None 2024-02-12T15:46:41.035 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T15:46:41.035 INFO:teuthology.packaging:ref: None 2024-02-12T15:46:41.035 INFO:teuthology.packaging:tag: None 2024-02-12T15:46:41.035 INFO:teuthology.packaging:branch: luminous 2024-02-12T15:46:41.035 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:46:41.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=luminous 2024-02-12T15:46:41.286 DEBUG:teuthology.orchestra.run.smithi121:> dpkg-query -W -f '${Version}' ceph 2024-02-12T15:46:41.312 INFO:teuthology.orchestra.run.smithi121.stdout:12.2.13-71-g707ba539-1bionic 2024-02-12T15:46:41.313 INFO:teuthology.packaging:The installed version of ceph is 12.2.13-71-g707ba539-1bionic 2024-02-12T15:46:41.313 INFO:teuthology.task.install:The correct ceph version 12.2.13-71-g707ba539-1bionic is installed. 2024-02-12T15:46:41.315 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T15:46:41.315 INFO:teuthology.packaging:ref: None 2024-02-12T15:46:41.315 INFO:teuthology.packaging:tag: None 2024-02-12T15:46:41.315 INFO:teuthology.packaging:branch: luminous 2024-02-12T15:46:41.315 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:46:41.315 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=luminous 2024-02-12T15:46:41.570 DEBUG:teuthology.orchestra.run.smithi123:> dpkg-query -W -f '${Version}' ceph 2024-02-12T15:46:41.594 INFO:teuthology.orchestra.run.smithi123.stdout:12.2.13-71-g707ba539-1bionic 2024-02-12T15:46:41.594 INFO:teuthology.packaging:The installed version of ceph is 12.2.13-71-g707ba539-1bionic 2024-02-12T15:46:41.594 INFO:teuthology.task.install:The correct ceph version 12.2.13-71-g707ba539-1bionic is installed. 2024-02-12T15:46:41.596 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-12T15:46:41.597 INFO:teuthology.packaging:ref: None 2024-02-12T15:46:41.597 INFO:teuthology.packaging:tag: None 2024-02-12T15:46:41.597 INFO:teuthology.packaging:branch: luminous 2024-02-12T15:46:41.597 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:46:41.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=luminous 2024-02-12T15:46:41.780 DEBUG:teuthology.orchestra.run.smithi169:> dpkg-query -W -f '${Version}' ceph 2024-02-12T15:46:41.803 INFO:teuthology.orchestra.run.smithi169.stdout:12.2.13-71-g707ba539-1bionic 2024-02-12T15:46:41.804 INFO:teuthology.packaging:The installed version of ceph is 12.2.13-71-g707ba539-1bionic 2024-02-12T15:46:41.804 INFO:teuthology.task.install:The correct ceph version 12.2.13-71-g707ba539-1bionic is installed. 2024-02-12T15:46:41.806 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-12T15:46:41.806 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:46:41.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-12T15:46:41.822 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:46:41.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-12T15:46:41.838 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:46:41.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-12T15:46:41.858 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-12T15:46:41.858 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:46:41.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-02-12T15:46:41.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-12T15:46:41.933 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:46:41.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-02-12T15:46:41.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-12T15:46:42.006 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:46:42.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-02-12T15:46:42.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-12T15:46:42.079 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-12T15:46:42.079 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:46:42.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-12T15:46:42.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-12T15:46:42.149 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:46:42.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-12T15:46:42.164 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-12T15:46:42.221 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:46:42.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-12T15:46:42.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-12T15:46:42.295 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-12T15:46:42.295 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:46:42.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-02-12T15:46:42.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-12T15:46:42.370 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:46:42.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-02-12T15:46:42.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-12T15:46:42.446 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:46:42.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-02-12T15:46:42.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-12T15:46:42.519 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-12T15:46:42.601 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(MON_DOWN\\)', 'osds down', 'mons down'], 'mon_bind_msgr2': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-12T15:46:42.602 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:46:42.602 INFO:tasks.cephadm:Cluster fsid is eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:46:42.602 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-12T15:46:42.602 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.121:6789]', 'mon.b': '[v1:172.21.15.123:6789]', 'mon.c': '[v1:172.21.15.169:6789]'} 2024-02-12T15:46:42.602 INFO:tasks.cephadm:First mon is mon.a on smithi121 2024-02-12T15:46:42.603 INFO:tasks.cephadm:First mgr is y 2024-02-12T15:46:42.603 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-12T15:46:42.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-02-12T15:46:42.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-02-12T15:46:42.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-02-12T15:46:42.656 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 0e714d9a4bd2a821113e6318adb87bd06cf81ec1)... 2024-02-12T15:46:42.656 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:42.834 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Feb 12 15:46 /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:42.834 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:42.986 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Feb 12 15:46 /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:42.987 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/0e714d9a4bd2a821113e6318adb87bd06cf81ec1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:43.139 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Feb 12 15:46 /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:43.140 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:43.149 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:43.158 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-12T15:46:43.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-02-12T15:46:43.206 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-02-12T15:46:43.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-02-12T15:46:43.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-02-12T15:46:43.260 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-02-12T15:46:43.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-02-12T15:46:43.292 INFO:tasks.cephadm:Writing seed config... 2024-02-12T15:46:43.293 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-02-12T15:46:43.293 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-12T15:46:43.293 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-12T15:46:43.293 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-12T15:46:43.293 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [global] ms inject socket failures = 5000 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-12T15:46:43.294 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 2 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-02-12T15:46:43.295 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-02-12T15:46:43.296 INFO:tasks.cephadm: override: [osd] osd heartbeat use min delay socket = True 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-12T15:46:43.297 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-12T15:46:43.298 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:46:43.298 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-12T15:46:43.309 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eb2349b2-c9bd-11ee-95b9-87774f69a715 mon client directed command retry = 5 mon election default strategy = 1 ms bind msgr2 = False ms inject socket failures = 5000 ms type = async osd_pool_default_min_size = 2 osd_pool_default_size = 2 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd backoff on peering = True osd debug reject backfill probability = 0.3 osd heartbeat use min delay socket = True osd max backfills = 3 osd max markdown count = 1000 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-12T15:46:43.310 DEBUG:teuthology.orchestra.run.smithi121:mon.a> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.a.service 2024-02-12T15:46:43.354 DEBUG:teuthology.orchestra.run.smithi121:mgr.y> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.y.service 2024-02-12T15:46:43.400 INFO:tasks.cephadm:Bootstrapping... 2024-02-12T15:46:43.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 -v bootstrap --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.121:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-12T15:46:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:46:43.856 INFO:journalctl@ceph.mgr.y.smithi121.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:46:43.930 INFO:teuthology.orchestra.run.smithi121.stderr:-------------------------------------------------------------------------------- 2024-02-12T15:46:43.931 INFO:teuthology.orchestra.run.smithi121.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1', '-v', 'bootstrap', '--fsid', 'eb2349b2-c9bd-11ee-95b9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.121:6789]', '--skip-admin-label'] 2024-02-12T15:46:43.931 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying podman|docker is present... 2024-02-12T15:46:43.931 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying lvm2 is present... 2024-02-12T15:46:43.931 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying time synchronization is in place... 2024-02-12T15:46:43.939 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-12T15:46:43.947 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:43.955 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2024-02-12T15:46:43.963 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:43.972 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: enabled 2024-02-12T15:46:43.980 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:43.988 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2024-02-12T15:46:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:44.006 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: enabled 2024-02-12T15:46:44.015 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: active 2024-02-12T15:46:44.016 INFO:teuthology.orchestra.run.smithi121.stderr:Unit ntp.service is enabled and running 2024-02-12T15:46:44.016 INFO:teuthology.orchestra.run.smithi121.stderr:Repeating the final host check... 2024-02-12T15:46:44.016 INFO:teuthology.orchestra.run.smithi121.stderr:docker (/usr/bin/docker) is present 2024-02-12T15:46:44.017 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl is present 2024-02-12T15:46:44.017 INFO:teuthology.orchestra.run.smithi121.stderr:lvcreate is present 2024-02-12T15:46:44.025 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-12T15:46:44.033 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:44.042 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2024-02-12T15:46:44.050 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:44.059 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: enabled 2024-02-12T15:46:44.067 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:44.074 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2024-02-12T15:46:44.082 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-02-12T15:46:44.089 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: enabled 2024-02-12T15:46:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: active 2024-02-12T15:46:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:Unit ntp.service is enabled and running 2024-02-12T15:46:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:Host looks OK 2024-02-12T15:46:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:Cluster fsid: eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:46:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:Acquiring lock 139681932825880 on /run/cephadm/eb2349b2-c9bd-11ee-95b9-87774f69a715.lock 2024-02-12T15:46:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:Lock 139681932825880 acquired on /run/cephadm/eb2349b2-c9bd-11ee-95b9-87774f69a715.lock 2024-02-12T15:46:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying IP 172.21.15.121 port 6789 ... 2024-02-12T15:46:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:Base mon IP(s) is [172.21.15.121:6789], mon addrv is [v1:172.21.15.121:6789] 2024-02-12T15:46:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 2024-02-12T15:46:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.121 metric 100 2024-02-12T15:46:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-02-12T15:46:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.121 2024-02-12T15:46:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.121 metric 100 2024-02-12T15:46:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-12T15:46:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-02-12T15:46:44.112 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-12T15:46:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-12T15:46:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-12T15:46:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-12T15:46:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-02-12T15:46:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-12T15:46:44.129 INFO:teuthology.orchestra.run.smithi121.stderr:Mon IP `172.21.15.121` is in CIDR network `172.21.0.0/20` 2024-02-12T15:46:44.129 INFO:teuthology.orchestra.run.smithi121.stderr:Mon IP `172.21.15.121` is in CIDR network `172.21.15.254/32` 2024-02-12T15:46:44.129 INFO:teuthology.orchestra.run.smithi121.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.15.254/32'] 2024-02-12T15:46:44.129 INFO:teuthology.orchestra.run.smithi121.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-12T15:46:44.129 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1... 2024-02-12T15:46:44.500 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1: Pulling from ceph-ci/ceph 2024-02-12T15:46:44.504 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 58df4e03ba06: Pulling fs layer 2024-02-12T15:46:44.504 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 955bfa9dba4a: Pulling fs layer 2024-02-12T15:46:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 58df4e03ba06: Download complete 2024-02-12T15:46:47.022 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 955bfa9dba4a: Verifying Checksum 2024-02-12T15:46:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 955bfa9dba4a: Download complete 2024-02-12T15:46:47.666 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 58df4e03ba06: Pull complete 2024-02-12T15:46:57.475 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: 955bfa9dba4a: Pull complete 2024-02-12T15:46:57.592 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: Digest: sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc 2024-02-12T15:46:57.634 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: Status: Downloaded newer image for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:46:57.659 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/docker: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-12T15:47:07.425 INFO:teuthology.orchestra.run.smithi121.stderr:ceph: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-12T15:47:07.951 INFO:teuthology.orchestra.run.smithi121.stderr:Ceph version: ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable) 2024-02-12T15:47:07.951 INFO:teuthology.orchestra.run.smithi121.stderr:Extracting ceph user uid/gid from container image... 2024-02-12T15:47:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:stat: 167 167 2024-02-12T15:47:09.281 INFO:teuthology.orchestra.run.smithi121.stderr:Creating initial keys... 2024-02-12T15:47:10.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-authtool: AQB+PcplfLcjCxAAIgMecKVctwiQVZ8K21YlqA== 2024-02-12T15:47:11.565 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-authtool: AQB/PcplzKZcIRAATfnn/s1n5Y9pdWCno+rEIQ== 2024-02-12T15:47:12.940 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-authtool: AQCAPcpldPSzNxAAJAqQdZ4K161vTNn7UppGuQ== 2024-02-12T15:47:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:Creating initial monmap... 2024-02-12T15:47:14.402 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-12T15:47:14.402 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:14.402 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-12T15:47:14.918 INFO:teuthology.orchestra.run.smithi121.stderr:monmaptool for a [v1:172.21.15.121:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-12T15:47:14.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: set fsid to eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:14.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-12T15:47:14.918 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:47:14.918 INFO:teuthology.orchestra.run.smithi121.stderr:Creating mon... 2024-02-12T15:47:15.815 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 1 imported monmap: 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: last_changed 2024-02-12T15:47:14.396629+0000 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: created 2024-02-12T15:47:14.396629+0000 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 0: v1:172.21.15.121:6789/0 mon.a 2024-02-12T15:47:15.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.817 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 0 /usr/bin/ceph-mon: set fsid to eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:15.817 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T15:47:15.817 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.817 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: DB SUMMARY 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.create_if_missing: 1 2024-02-12T15:47:15.819 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.env: 0x5608a6321080 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.fs: Posix File System 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.info_log: 0x5608a7b9da80 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.statistics: (nil) 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.use_fsync: 0 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T15:47:15.820 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.db_log_dir: 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T15:47:15.821 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.write_buffer_manager: 0x5608a7ba3980 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T15:47:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.unordered_write: 0 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.row_cache: None 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.wal_filter: None 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T15:47:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T15:47:15.853 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.max_open_files: -1 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T15:47:15.854 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.804+0000 7fdf6f95d880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Compression algorithms supported: 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kZSTD supported: 0 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.855 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.merge_operator: 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_filter: None 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5608a7ab1d30) 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-12T15:47:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache: 0x5608a7ae8d10 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-12T15:47:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-12T15:47:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compression: NoCompression 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.num_levels: 7 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T15:47:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T15:47:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T15:47:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T15:47:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.ttl: 2592000 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.808+0000 7fdf6f95d880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.812+0000 7fdf6f95d880 4 rocksdb: DB pointer 0x5608a7bb1800 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.812+0000 7fdf587cf700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T15:47:15.863 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.812+0000 7fdf587cf700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:47:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:47:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-12T15:47:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.812+0000 7fdf6f95d880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.812+0000 7fdf6f95d880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-12T15:47:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-02-12T15:47:15.812+0000 7fdf6f95d880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-02-12T15:47:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:create mon.a on 2024-02-12T15:47:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-12T15:47:16.421 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-02-12T15:47:16.545 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715.target -> /etc/systemd/system/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715.target. 2024-02-12T15:47:16.545 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715.target -> /etc/systemd/system/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715.target. 2024-02-12T15:47:16.798 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to reset failed state of unit ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.a.service: Unit ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.a.service is not loaded. 2024-02-12T15:47:16.807 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715.target.wants/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.a.service -> /etc/systemd/system/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@.service. 2024-02-12T15:47:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld does not appear to be present 2024-02-12T15:47:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to enable service . firewalld.service is not available 2024-02-12T15:47:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mon to start... 2024-02-12T15:47:16.923 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mon... 2024-02-12T15:47:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:16 smithi121 systemd[1]: Started Ceph mon.a for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T15:47:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:18 smithi121 bash[17841]: cluster 2024-02-12T15:47:18.274158+0000 mon.a (mon.0) 0 : [INF] mkfs eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:18 smithi121 bash[17841]: cluster 2024-02-12T15:47:18.257223+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-12T15:47:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: cluster: 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: id: eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: services: 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.394902s) 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: data: 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: pgs: 2024-02-12T15:47:18.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:mon is available 2024-02-12T15:47:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:Assimilating anything we can from ceph.conf... 2024-02-12T15:47:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:19 smithi121 bash[17841]: cluster 2024-02-12T15:47:18.269438+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-12T15:47:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:19 smithi121 bash[17841]: cluster 2024-02-12T15:47:18.270997+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.121:6789/0} removed_ranks: {} 2024-02-12T15:47:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:19 smithi121 bash[17841]: cluster 2024-02-12T15:47:18.280549+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-12T15:47:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:19 smithi121 bash[17841]: cluster 2024-02-12T15:47:18.290207+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-12T15:47:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:19 smithi121 bash[17841]: cluster 2024-02-12T15:47:18.291714+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-12T15:47:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:19 smithi121 bash[17841]: audit 2024-02-12T15:47:18.664260+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.121:0/1277600177' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-12T15:47:20.443 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:20.444 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [global] 2024-02-12T15:47:20.444 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: fsid = eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:20.444 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-12T15:47:20.444 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_host = [v1:172.21.15.121:6789] 2024-02-12T15:47:20.445 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-12T15:47:20.445 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-12T15:47:20.445 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-12T15:47:20.445 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-12T15:47:20.445 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:20.446 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [mgr] 2024-02-12T15:47:20.446 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-12T15:47:20.446 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:20.446 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [osd] 2024-02-12T15:47:20.446 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-12T15:47:20.446 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-12T15:47:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:20 smithi121 bash[17841]: audit 2024-02-12T15:47:20.429794+0000 mon.a (mon.0) 8 : audit [INF] from='client.? v1:172.21.15.121:0/1668811881' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-12T15:47:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:20 smithi121 bash[17841]: audit 2024-02-12T15:47:20.436942+0000 mon.a (mon.0) 9 : audit [INF] from='client.? v1:172.21.15.121:0/1668811881' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-12T15:47:20.962 INFO:teuthology.orchestra.run.smithi121.stderr:Generating new minimal ceph.conf... 2024-02-12T15:47:22.644 INFO:teuthology.orchestra.run.smithi121.stderr:Restarting the monitor... 2024-02-12T15:47:22.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:22 smithi121 bash[17841]: audit 2024-02-12T15:47:22.152796+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.121:0/4075474134' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:47:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:22 smithi121 systemd[1]: Stopping Ceph mon.a for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T15:47:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:22 smithi121 bash[18368]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-mon.a 2024-02-12T15:47:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:22 smithi121 bash[17841]: debug 2024-02-12T15:47:22.815+0000 7f599839a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T15:47:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:22 smithi121 bash[17841]: debug 2024-02-12T15:47:22.815+0000 7f599839a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-12T15:47:23.630 INFO:teuthology.orchestra.run.smithi121.stderr:Setting public_network to 172.21.15.254/32,172.21.0.0/20 in mon config section 2024-02-12T15:47:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:23 smithi121 bash[18368]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-mon-a 2024-02-12T15:47:23.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:23 smithi121 systemd[1]: Stopped Ceph mon.a for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T15:47:23.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:23 smithi121 systemd[1]: Started Ceph mon.a for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.847+0000 7fbbbefe8880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.847+0000 7fbbbefe8880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 7 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.847+0000 7fbbbefe8880 0 pidfile_write: ignore empty --pid-file 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.851+0000 7fbbbefe8880 0 load: jerasure load: lrc load: isa 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: DB SUMMARY 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: CURRENT file: CURRENT 2024-02-12T15:47:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 95045 ; 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.create_if_missing: 0 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.env: 0x55bdb8340080 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.fs: Posix File System 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.info_log: 0x55bdb9bd21c0 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.statistics: (nil) 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.use_fsync: 0 2024-02-12T15:47:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.db_log_dir: 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-12T15:47:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.write_buffer_manager: 0x55bdba8c6390 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T15:47:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T15:47:25.165 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T15:47:25.165 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T15:47:25.165 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T15:47:25.165 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T15:47:25.166 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T15:47:25.166 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.unordered_write: 0 2024-02-12T15:47:25.166 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.row_cache: None 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.wal_filter: None 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T15:47:25.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_open_files: -1 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T15:47:25.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Compression algorithms supported: 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kZSTD supported: 0 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T15:47:25.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.merge_operator: 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_filter: None 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bdb9b75d28) 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: cache_index_and_filter_blocks: 1 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: pin_top_level_index_and_filter: 1 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: index_type: 0 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: data_block_index_type: 0 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: index_shortening: 1 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: data_block_hash_table_util_ratio: 0.750000 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: hash_index_allow_collision: 1 2024-02-12T15:47:25.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: checksum: 1 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: no_block_cache: 0 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_cache: 0x55bdb9bacf10 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_cache_name: BinnedLRUCache 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_cache_options: 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: capacity : 536870912 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: num_shard_bits : 4 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: strict_capacity_limit : 0 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: high_pri_pool_ratio: 0.000 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_cache_compressed: (nil) 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: persistent_cache: (nil) 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_size: 4096 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_size_deviation: 10 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_restart_interval: 16 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: index_block_restart_interval: 1 2024-02-12T15:47:25.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: metadata_block_size: 4096 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: partition_filters: 0 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: use_delta_encoding: 1 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: whole_key_filtering: 1 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: verify_compression: 0 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: read_amp_bytes_per_bit: 0 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: format_version: 2 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: enable_index_compression: 1 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: block_align: 0 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compression: NoCompression 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T15:47:25.172 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.num_levels: 7 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T15:47:25.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T15:47:25.174 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T15:47:25.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T15:47:25.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.ttl: 2592000 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752844861117, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752844862169, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 90578, "table_properties": {"data_size": 88676, "index_size": 379, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12357, "raw_average_key_size": 50, "raw_value_size": 81911, "raw_average_value_size": 332, "num_data_blocks": 12, "num_entries": 246, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707752844, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.855+0000 7fbbbefe8880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752844862899, "job": 1, "event": "recovery_finished"} 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 4 rocksdb: DB pointer 0x55bdb9c75800 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 0 starting mon.a rank 0 at public addrs v1:172.21.15.121:6789/0 at bind addrs v1:172.21.15.121:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:25.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbba5dd9700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T15:47:25.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbba5dd9700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T15:47:25.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ** DB Stats ** 2024-02-12T15:47:25.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ** Compaction Stats [default] ** 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: L0 2/0 90.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.4 0.00 0.00 1 0.001 0 0 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Sum 2/0 90.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.4 0.00 0.00 1 0.001 0 0 2024-02-12T15:47:25.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.4 0.00 0.00 1 0.001 0 0 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ** Compaction Stats [default] ** 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.4 0.00 0.00 1 0.001 0 0 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:47:25.180 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Cumulative compaction: 0.00 GB write, 17.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Interval compaction: 0.00 GB write, 17.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ** Compaction Stats [default] ** 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: L0 2/0 90.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.4 0.00 0.00 1 0.001 0 0 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Sum 2/0 90.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.4 0.00 0.00 1 0.001 0 0 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ** Compaction Stats [default] ** 2024-02-12T15:47:25.181 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.4 0.00 0.00 1 0.001 0 0 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Cumulative compaction: 0.00 GB write, 17.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:47:25.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 1 mon.a@-1(???) e1 preinit fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 0 mon.a@-1(???).mds e1 new map 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 0 mon.a@-1(???).mds e1 print_map 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: e1 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: legacy client fscid: -1 2024-02-12T15:47:25.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: No filesystems configured 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: audit 2024-02-12T15:47:22.152796+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.121:0/4075474134' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: debug 2024-02-12T15:47:24.859+0000 7fbbbefe8880 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: cluster 2024-02-12T15:47:24.870715+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: cluster 2024-02-12T15:47:24.870804+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.121:6789/0} removed_ranks: {} 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: cluster 2024-02-12T15:47:24.871411+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: cluster 2024-02-12T15:47:24.871472+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-12T15:47:25.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:24 smithi121 bash[18489]: cluster 2024-02-12T15:47:24.872003+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-12T15:47:25.846 INFO:teuthology.orchestra.run.smithi121.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-12T15:47:25.846 INFO:teuthology.orchestra.run.smithi121.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-12T15:47:25.846 INFO:teuthology.orchestra.run.smithi121.stderr:Creating mgr... 2024-02-12T15:47:25.846 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying port 9283 ... 2024-02-12T15:47:25.992 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to reset failed state of unit ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.y.service: Unit ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.y.service is not loaded. 2024-02-12T15:47:26.000 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715.target.wants/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.y.service -> /etc/systemd/system/ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@.service. 2024-02-12T15:47:26.125 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld does not appear to be present 2024-02-12T15:47:26.125 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to enable service . firewalld.service is not available 2024-02-12T15:47:26.125 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld does not appear to be present 2024-02-12T15:47:26.125 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-12T15:47:26.125 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr to start... 2024-02-12T15:47:26.125 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr... 2024-02-12T15:47:26.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:26 smithi121 bash[18489]: audit 2024-02-12T15:47:25.187889+0000 mon.a (mon.0) 6 : audit [INF] from='client.? v1:172.21.15.121:0/1423488219' entity='client.admin' 2024-02-12T15:47:26.411 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:26 smithi121 systemd[1]: Started Ceph mgr.y for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T15:47:28.022 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:28.022 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "health": { 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 0 2024-02-12T15:47:28.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "a" 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_age": 3, 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T15:47:28.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T15:47:28.027 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T15:47:28.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T15:47:28.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T15:47:28.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T15:47:28.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:28.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T15:47:28.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T15:47:28.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T15:47:28.029 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T15:47:28.029 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T15:47:28.032 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": false, 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "iostat", 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "nfs", 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "restful" 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modified": "2024-02-12T15:47:18.275308+0000", 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-02-12T15:47:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:28.034 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T15:47:28.034 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-02-12T15:47:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:28 smithi121 bash[18489]: audit 2024-02-12T15:47:28.015607+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.121:0/1601932173' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T15:47:28.411 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:27 smithi121 bash[18805]: debug 2024-02-12T15:47:27.923+0000 7f9958e7f680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T15:47:28.411 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:28 smithi121 bash[18805]: debug 2024-02-12T15:47:28.027+0000 7f9958e7f680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T15:47:28.544 INFO:teuthology.orchestra.run.smithi121.stderr:mgr not available, waiting (1/15)... 2024-02-12T15:47:28.910 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:28 smithi121 bash[18805]: debug 2024-02-12T15:47:28.467+0000 7f9958e7f680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T15:47:29.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:29 smithi121 bash[18805]: debug 2024-02-12T15:47:29.355+0000 7f9958e7f680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T15:47:29.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:29 smithi121 bash[18805]: debug 2024-02-12T15:47:29.443+0000 7f9958e7f680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T15:47:30.160 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:29 smithi121 bash[18805]: debug 2024-02-12T15:47:29.683+0000 7f9958e7f680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T15:47:31.034 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:30 smithi121 bash[18805]: debug 2024-02-12T15:47:30.771+0000 7f9958e7f680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T15:47:31.034 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:30 smithi121 bash[18805]: debug 2024-02-12T15:47:30.927+0000 7f9958e7f680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:47:31.358 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:31 smithi121 bash[18805]: debug 2024-02-12T15:47:31.023+0000 7f9958e7f680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T15:47:31.359 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:31 smithi121 bash[18805]: debug 2024-02-12T15:47:31.131+0000 7f9958e7f680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T15:47:31.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:31 smithi121 bash[18805]: debug 2024-02-12T15:47:31.347+0000 7f9958e7f680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T15:47:31.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:31.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-02-12T15:47:31.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T15:47:31.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "health": { 2024-02-12T15:47:31.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T15:47:31.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T15:47:31.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T15:47:31.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:31.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T15:47:31.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T15:47:31.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 0 2024-02-12T15:47:31.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:31.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "a" 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T15:47:31.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T15:47:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:31.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T15:47:31.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:31.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T15:47:31.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T15:47:31.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": false, 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "iostat", 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "nfs", 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "restful" 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modified": "2024-02-12T15:47:18.275308+0000", 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:31.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T15:47:31.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-02-12T15:47:31.987 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:31 smithi121 bash[18489]: audit 2024-02-12T15:47:31.897715+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? v1:172.21.15.121:0/3451711456' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T15:47:31.987 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:31 smithi121 bash[18805]: debug 2024-02-12T15:47:31.707+0000 7f9958e7f680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T15:47:31.987 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:31 smithi121 bash[18805]: debug 2024-02-12T15:47:31.827+0000 7f9958e7f680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T15:47:32.492 INFO:teuthology.orchestra.run.smithi121.stderr:mgr not available, waiting (2/15)... 2024-02-12T15:47:33.112 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:32 smithi121 bash[18805]: debug 2024-02-12T15:47:32.679+0000 7f9958e7f680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T15:47:33.112 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:32 smithi121 bash[18805]: debug 2024-02-12T15:47:32.779+0000 7f9958e7f680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T15:47:33.112 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:32 smithi121 bash[18805]: debug 2024-02-12T15:47:32.891+0000 7f9958e7f680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T15:47:33.112 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:33 smithi121 bash[18805]: debug 2024-02-12T15:47:33.099+0000 7f9958e7f680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T15:47:33.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:33 smithi121 bash[18805]: debug 2024-02-12T15:47:33.379+0000 7f9958e7f680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T15:47:33.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:33 smithi121 bash[18805]: debug 2024-02-12T15:47:33.539+0000 7f9958e7f680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T15:47:34.025 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:33 smithi121 bash[18805]: debug 2024-02-12T15:47:33.703+0000 7f9958e7f680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:47:34.026 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:33 smithi121 bash[18805]: debug 2024-02-12T15:47:33.911+0000 7f9958e7f680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T15:47:34.026 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18805]: debug 2024-02-12T15:47:34.015+0000 7f9958e7f680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T15:47:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: cluster 2024-02-12T15:47:34.040604+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-02-12T15:47:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: cluster 2024-02-12T15:47:34.045911+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00546461s) 2024-02-12T15:47:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.048768+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T15:47:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.049036+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T15:47:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.049305+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T15:47:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.049814+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T15:47:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.050400+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T15:47:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.050873+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T15:47:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.051232+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:47:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.051588+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T15:47:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: cluster 2024-02-12T15:47:34.054988+0000 mon.a (mon.0) 19 : cluster [INF] Manager daemon y is now available 2024-02-12T15:47:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.070159+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' 2024-02-12T15:47:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.080618+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T15:47:34.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.083779+0000 mon.a (mon.0) 22 : audit [INF] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T15:47:34.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.088082+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' 2024-02-12T15:47:34.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:34 smithi121 bash[18489]: audit 2024-02-12T15:47:34.091215+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "health": { 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "checks": {}, 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mutes": [] 2024-02-12T15:47:36.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:36.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-12T15:47:36.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum": [ 2024-02-12T15:47:36.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 0 2024-02-12T15:47:36.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:36.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "a" 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "monmap": { 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osdmap": { 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-12T15:47:36.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgmap": { 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-12T15:47:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-12T15:47:36.241 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsmap": { 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": true, 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modules": [ 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "iostat", 2024-02-12T15:47:36.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "nfs", 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "restful" 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "servicemap": { 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modified": "2024-02-12T15:47:18.275308+0000", 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-12T15:47:36.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-02-12T15:47:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:36 smithi121 bash[18489]: cluster 2024-02-12T15:47:35.049493+0000 mon.a (mon.0) 25 : cluster [DBG] mgrmap e3: y(active, since 1.00905s) 2024-02-12T15:47:36.765 INFO:teuthology.orchestra.run.smithi121.stderr:mgr is available 2024-02-12T15:47:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:37 smithi121 bash[18489]: cluster 2024-02-12T15:47:36.051456+0000 mon.a (mon.0) 26 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-02-12T15:47:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:37 smithi121 bash[18489]: audit 2024-02-12T15:47:36.226712+0000 mon.a (mon.0) 27 : audit [DBG] from='client.14110 v1:172.21.15.121:0/3394545057' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-12T15:47:38.143 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [global] 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: fsid = eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [mgr] 2024-02-12T15:47:38.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-12T15:47:38.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-02-12T15:47:38.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [osd] 2024-02-12T15:47:38.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-12T15:47:38.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-12T15:47:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:38 smithi121 bash[18489]: audit 2024-02-12T15:47:38.044906+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.14100 v1:172.21.15.121:0/3538725953' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:47:38.680 INFO:teuthology.orchestra.run.smithi121.stderr:Enabling cephadm module... 2024-02-12T15:47:39.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:39 smithi121 bash[18489]: audit 2024-02-12T15:47:38.134969+0000 mon.a (mon.0) 29 : audit [INF] from='client.? v1:172.21.15.121:0/164138883' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-12T15:47:41.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:40 smithi121 bash[18489]: audit 2024-02-12T15:47:40.156506+0000 mon.a (mon.0) 30 : audit [INF] from='client.? v1:172.21.15.121:0/1897080579' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-12T15:47:41.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:40 smithi121 bash[18805]: ignoring --setuser ceph since I am not root 2024-02-12T15:47:41.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:40 smithi121 bash[18805]: ignoring --setgroup ceph since I am not root 2024-02-12T15:47:41.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:41 smithi121 bash[18805]: debug 2024-02-12T15:47:41.083+0000 7fb41ee73680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T15:47:41.633 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:41 smithi121 bash[18805]: debug 2024-02-12T15:47:41.187+0000 7fb41ee73680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T15:47:41.887 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:41 smithi121 bash[18805]: debug 2024-02-12T15:47:41.623+0000 7fb41ee73680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T15:47:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:41 smithi121 bash[18489]: audit 2024-02-12T15:47:40.885002+0000 mon.a (mon.0) 31 : audit [INF] from='client.? v1:172.21.15.121:0/1897080579' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-12T15:47:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:41 smithi121 bash[18489]: cluster 2024-02-12T15:47:40.885159+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e5: y(active, since 6s) 2024-02-12T15:47:42.634 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:42 smithi121 bash[18805]: debug 2024-02-12T15:47:42.527+0000 7fb41ee73680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T15:47:42.910 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:42 smithi121 bash[18805]: debug 2024-02-12T15:47:42.623+0000 7fb41ee73680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T15:47:42.910 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:42 smithi121 bash[18805]: debug 2024-02-12T15:47:42.863+0000 7fb41ee73680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T15:47:43.133 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-02-12T15:47:43.133 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-12T15:47:43.133 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": true, 2024-02-12T15:47:43.133 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-12T15:47:43.133 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-12T15:47:43.133 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-02-12T15:47:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:43 smithi121 bash[18489]: audit 2024-02-12T15:47:43.125990+0000 mon.a (mon.0) 33 : audit [DBG] from='client.? v1:172.21.15.121:0/639031943' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-12T15:47:43.673 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for the mgr to restart... 2024-02-12T15:47:43.673 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr epoch 5... 2024-02-12T15:47:44.228 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:43 smithi121 bash[18805]: debug 2024-02-12T15:47:43.963+0000 7fb41ee73680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T15:47:44.228 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:44 smithi121 bash[18805]: debug 2024-02-12T15:47:44.119+0000 7fb41ee73680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:47:44.561 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:44 smithi121 bash[18805]: debug 2024-02-12T15:47:44.215+0000 7fb41ee73680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T15:47:44.562 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:44 smithi121 bash[18805]: debug 2024-02-12T15:47:44.327+0000 7fb41ee73680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T15:47:44.910 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:44 smithi121 bash[18805]: debug 2024-02-12T15:47:44.551+0000 7fb41ee73680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T15:47:45.410 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:44 smithi121 bash[18805]: debug 2024-02-12T15:47:44.911+0000 7fb41ee73680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T15:47:45.411 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:45 smithi121 bash[18805]: debug 2024-02-12T15:47:45.031+0000 7fb41ee73680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T15:47:46.160 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:45 smithi121 bash[18805]: debug 2024-02-12T15:47:45.867+0000 7fb41ee73680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T15:47:46.160 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:45 smithi121 bash[18805]: debug 2024-02-12T15:47:45.967+0000 7fb41ee73680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T15:47:46.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:46 smithi121 bash[18805]: debug 2024-02-12T15:47:46.079+0000 7fb41ee73680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T15:47:46.581 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:46 smithi121 bash[18805]: debug 2024-02-12T15:47:46.287+0000 7fb41ee73680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T15:47:46.904 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:46 smithi121 bash[18805]: debug 2024-02-12T15:47:46.571+0000 7fb41ee73680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T15:47:46.904 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:46 smithi121 bash[18805]: debug 2024-02-12T15:47:46.727+0000 7fb41ee73680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T15:47:47.160 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:46 smithi121 bash[18805]: debug 2024-02-12T15:47:46.891+0000 7fb41ee73680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:47:47.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18805]: debug 2024-02-12T15:47:47.099+0000 7fb41ee73680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T15:47:47.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18805]: debug 2024-02-12T15:47:47.203+0000 7fb41ee73680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: cluster 2024-02-12T15:47:47.232725+0000 mon.a (mon.0) 34 : cluster [INF] Active manager daemon y restarted 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: cluster 2024-02-12T15:47:47.233174+0000 mon.a (mon.0) 35 : cluster [INF] Activating manager daemon y 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: cluster 2024-02-12T15:47:47.237310+0000 mon.a (mon.0) 36 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: cluster 2024-02-12T15:47:47.237442+0000 mon.a (mon.0) 37 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00444016s) 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: audit 2024-02-12T15:47:47.240433+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: audit 2024-02-12T15:47:47.242083+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: audit 2024-02-12T15:47:47.243425+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: audit 2024-02-12T15:47:47.243732+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: audit 2024-02-12T15:47:47.244055+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: cluster 2024-02-12T15:47:47.249105+0000 mon.a (mon.0) 43 : cluster [INF] Manager daemon y is now available 2024-02-12T15:47:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:47 smithi121 bash[18489]: audit 2024-02-12T15:47:47.269266+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:47:48.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-02-12T15:47:48.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-12T15:47:48.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "initialized": true 2024-02-12T15:47:48.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: cephadm 2024-02-12T15:47:47.266074+0000 mgr.y (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: audit 2024-02-12T15:47:47.393220+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: audit 2024-02-12T15:47:47.395013+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: audit 2024-02-12T15:47:47.398094+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: audit 2024-02-12T15:47:47.402455+0000 mon.a (mon.0) 48 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: audit 2024-02-12T15:47:47.430023+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: audit 2024-02-12T15:47:47.436174+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T15:47:48.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:48 smithi121 bash[18489]: cluster 2024-02-12T15:47:48.244861+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e7: y(active, since 1.01185s) 2024-02-12T15:47:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:mgr epoch 5 is available 2024-02-12T15:47:48.880 INFO:teuthology.orchestra.run.smithi121.stderr:Setting orchestrator backend to cephadm... 2024-02-12T15:47:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:49 smithi121 bash[18489]: audit 2024-02-12T15:47:48.242602+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 v1:172.21.15.121:0/2580526437' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-12T15:47:49.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:49 smithi121 bash[18489]: audit 2024-02-12T15:47:48.259243+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14120 v1:172.21.15.121:0/2580526437' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-12T15:47:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:50 smithi121 bash[18489]: cluster 2024-02-12T15:47:49.399376+0000 mon.a (mon.0) 52 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-02-12T15:47:50.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:50 smithi121 bash[18489]: audit 2024-02-12T15:47:50.331547+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:47:50.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:50 smithi121 bash[18489]: audit 2024-02-12T15:47:50.387224+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:47:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:51 smithi121 bash[18489]: audit 2024-02-12T15:47:50.322574+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14126 v1:172.21.15.121:0/3750601393' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:47:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: value unchanged 2024-02-12T15:47:52.773 INFO:teuthology.orchestra.run.smithi121.stderr:Generating ssh key... 2024-02-12T15:47:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:52 smithi121 bash[18489]: audit 2024-02-12T15:47:52.225280+0000 mgr.y (mgr.14116) 5 : audit [DBG] from='client.14128 v1:172.21.15.121:0/3929292788' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: Generating public/private rsa key pair. 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: Your identification has been saved in /tmp/tmpl5164xsn/key. 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: Your public key has been saved in /tmp/tmpl5164xsn/key.pub. 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: The key fingerprint is: 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: SHA256:OwsKOU8sTx4JuTQUdYAEDGcmxBFRu0c3bwATn4J+sX8 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: The key's randomart image is: 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: +---[RSA 3072]----+ 2024-02-12T15:47:54.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: |OB@+o=o | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | B..o.+ . | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | .o + * | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | ...o = + | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | =o + So | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | . *o.. .. | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | * * ..oE | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | X o ..o | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: | = . | 2024-02-12T15:47:54.912 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:54 smithi121 bash[18805]: +----[SHA256]-----+ 2024-02-12T15:47:55.801 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:55 smithi121 bash[18489]: audit 2024-02-12T15:47:54.164438+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 v1:172.21.15.121:0/1746648629' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:47:55.801 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:55 smithi121 bash[18489]: cephadm 2024-02-12T15:47:54.164834+0000 mgr.y (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-02-12T15:47:55.801 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:55 smithi121 bash[18489]: audit 2024-02-12T15:47:54.474577+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:47:55.801 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:55 smithi121 bash[18489]: audit 2024-02-12T15:47:54.479111+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:47:56.367 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLD0s8gfdVv8R10+qI9CiZxflmuQZ8YOYBi9FA42Ohpmh/5WYIJ/U8biRhaPV69HtQHkFfOGTj8XzDG+boC6mBvfFm7U18KL5KTEq/kFs6e0uvBpU8rcI0I1BhQGd7fDszmqd9xkercRNgtQva6eTGSVlxlxk+kZV62Y8Lw3kExBdbkXT97uIxS5vqTPA+eN4/817aJiY3a5a2MmPuysFEFNiOroxqXFz6Vnuhttjj4HWtfGQYmHxK81t+kbdtd4MVXJ8WMCVJY/9sy0/Z8Z8LjtxF2VAfw5zG9pJ4IUNSzOWpb57AOTP9xn0EjmsGXyRACiSIUf3OFrAwx9wuc5l3gyk6ArmLId/Cf7m417yvX2Bmq8xvzUozfcAEJFlbqzpCgFh6KR36oZFFdistc5trv76BuIj/ACmVH4CAWSQkOhuLzSnmpbknN3Ewqgl/Oj87ZWx6df5Wt78BtFZHb5gXH5rhMd0VsrFoS0in7Y7Y9R4IWZCZS+HEPwJZnN6dk58= ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:47:56.911 INFO:teuthology.orchestra.run.smithi121.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-12T15:47:56.911 INFO:teuthology.orchestra.run.smithi121.stderr:Adding key to root@localhost authorized_keys... 2024-02-12T15:47:56.912 INFO:teuthology.orchestra.run.smithi121.stderr:Adding host smithi121... 2024-02-12T15:47:57.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:56 smithi121 bash[18489]: audit 2024-02-12T15:47:56.360445+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14132 v1:172.21.15.121:0/1545565358' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:47:58.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:47:58 smithi121 bash[18805]: Warning: Permanently added '172.21.15.121' (ECDSA) to the list of known hosts. 2024-02-12T15:47:59.177 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Added host 'smithi121' with addr '172.21.15.121' 2024-02-12T15:47:59.247 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:47:58 smithi121 bash[18489]: audit 2024-02-12T15:47:58.352754+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 v1:172.21.15.121:0/944257287' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:48:00.009 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying unmanaged mon service... 2024-02-12T15:48:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:00 smithi121 bash[18489]: audit 2024-02-12T15:47:59.169861+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:48:00.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:00 smithi121 bash[18489]: cephadm 2024-02-12T15:47:59.170583+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Added host smithi121 2024-02-12T15:48:00.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:00 smithi121 bash[18489]: audit 2024-02-12T15:47:59.178962+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:48:02.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-12T15:48:02.678 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying unmanaged mgr service... 2024-02-12T15:48:03.400 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:03 smithi121 bash[18489]: audit 2024-02-12T15:48:02.041026+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 v1:172.21.15.121:0/859582814' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:48:03.400 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:03 smithi121 bash[18489]: cephadm 2024-02-12T15:48:02.043101+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-12T15:48:03.400 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:03 smithi121 bash[18489]: audit 2024-02-12T15:48:02.046272+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:48:03.400 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:03 smithi121 bash[18489]: audit 2024-02-12T15:48:02.301152+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:48:03.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:03 smithi121 bash[18489]: audit 2024-02-12T15:48:02.593882+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:48:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-12T15:48:05.364 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:05 smithi121 bash[18489]: audit 2024-02-12T15:48:04.081589+0000 mgr.y (mgr.14116) 13 : audit [DBG] from='client.14138 v1:172.21.15.121:0/2493849607' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:48:05.365 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:05 smithi121 bash[18489]: cephadm 2024-02-12T15:48:04.084007+0000 mgr.y (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-12T15:48:05.365 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:05 smithi121 bash[18489]: audit 2024-02-12T15:48:04.088568+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:48:07.296 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:07 smithi121 bash[18489]: audit 2024-02-12T15:48:06.011594+0000 mon.a (mon.0) 63 : audit [INF] from='client.? v1:172.21.15.121:0/199622248' entity='client.admin' 2024-02-12T15:48:08.462 INFO:teuthology.orchestra.run.smithi121.stderr:Enabling the dashboard module... 2024-02-12T15:48:09.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:08 smithi121 bash[18489]: cluster 2024-02-12T15:48:07.245185+0000 mgr.y (mgr.14116) 15 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:09.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:08 smithi121 bash[18489]: audit 2024-02-12T15:48:07.749941+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14116 v1:172.21.15.121:0/3668811578' entity='mgr.y' 2024-02-12T15:48:09.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:08 smithi121 bash[18489]: audit 2024-02-12T15:48:07.910171+0000 mon.a (mon.0) 65 : audit [INF] from='client.? v1:172.21.15.121:0/3165629181' entity='client.admin' 2024-02-12T15:48:11.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:10 smithi121 bash[18489]: cluster 2024-02-12T15:48:09.245650+0000 mgr.y (mgr.14116) 16 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:11.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:10 smithi121 bash[18489]: audit 2024-02-12T15:48:10.073118+0000 mon.a (mon.0) 66 : audit [INF] from='client.? v1:172.21.15.121:0/2067449594' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-12T15:48:11.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:10 smithi121 bash[18805]: ignoring --setuser ceph since I am not root 2024-02-12T15:48:11.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:10 smithi121 bash[18805]: ignoring --setgroup ceph since I am not root 2024-02-12T15:48:11.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:11 smithi121 bash[18805]: debug 2024-02-12T15:48:11.082+0000 7fb599620680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T15:48:11.611 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:11 smithi121 bash[18805]: debug 2024-02-12T15:48:11.174+0000 7fb599620680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T15:48:11.895 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:11 smithi121 bash[18805]: debug 2024-02-12T15:48:11.602+0000 7fb599620680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T15:48:12.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:11 smithi121 bash[18489]: audit 2024-02-12T15:48:10.886899+0000 mon.a (mon.0) 67 : audit [INF] from='client.? v1:172.21.15.121:0/2067449594' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-12T15:48:12.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:11 smithi121 bash[18489]: cluster 2024-02-12T15:48:10.887065+0000 mon.a (mon.0) 68 : cluster [DBG] mgrmap e9: y(active, since 23s) 2024-02-12T15:48:12.660 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:12 smithi121 bash[18805]: debug 2024-02-12T15:48:12.510+0000 7fb599620680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T15:48:12.661 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:12 smithi121 bash[18805]: debug 2024-02-12T15:48:12.606+0000 7fb599620680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T15:48:13.151 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-02-12T15:48:13.151 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-12T15:48:13.151 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": true, 2024-02-12T15:48:13.151 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-12T15:48:13.151 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-12T15:48:13.151 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-02-12T15:48:13.160 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:12 smithi121 bash[18805]: debug 2024-02-12T15:48:12.862+0000 7fb599620680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T15:48:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:13 smithi121 bash[18489]: audit 2024-02-12T15:48:13.143472+0000 mon.a (mon.0) 69 : audit [DBG] from='client.? v1:172.21.15.121:0/2436368178' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-12T15:48:13.731 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for the mgr to restart... 2024-02-12T15:48:13.731 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr epoch 9... 2024-02-12T15:48:14.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:13 smithi121 bash[18805]: debug 2024-02-12T15:48:13.962+0000 7fb599620680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T15:48:14.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:14 smithi121 bash[18805]: debug 2024-02-12T15:48:14.122+0000 7fb599620680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:48:14.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:14 smithi121 bash[18805]: debug 2024-02-12T15:48:14.210+0000 7fb599620680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T15:48:14.661 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:14 smithi121 bash[18805]: debug 2024-02-12T15:48:14.318+0000 7fb599620680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T15:48:14.661 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:14 smithi121 bash[18805]: debug 2024-02-12T15:48:14.534+0000 7fb599620680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T15:48:15.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:14 smithi121 bash[18805]: debug 2024-02-12T15:48:14.894+0000 7fb599620680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T15:48:15.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:15 smithi121 bash[18805]: debug 2024-02-12T15:48:15.018+0000 7fb599620680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T15:48:16.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:15 smithi121 bash[18805]: debug 2024-02-12T15:48:15.886+0000 7fb599620680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T15:48:16.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:15 smithi121 bash[18805]: debug 2024-02-12T15:48:15.990+0000 7fb599620680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T15:48:16.161 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:16 smithi121 bash[18805]: debug 2024-02-12T15:48:16.102+0000 7fb599620680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T15:48:16.607 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:16 smithi121 bash[18805]: debug 2024-02-12T15:48:16.314+0000 7fb599620680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T15:48:16.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:16 smithi121 bash[18805]: debug 2024-02-12T15:48:16.598+0000 7fb599620680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T15:48:16.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:16 smithi121 bash[18805]: debug 2024-02-12T15:48:16.758+0000 7fb599620680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T15:48:17.246 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:16 smithi121 bash[18805]: debug 2024-02-12T15:48:16.926+0000 7fb599620680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:48:17.246 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18805]: debug 2024-02-12T15:48:17.134+0000 7fb599620680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T15:48:17.539 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18805]: debug 2024-02-12T15:48:17.238+0000 7fb599620680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: cluster 2024-02-12T15:48:17.261045+0000 mon.a (mon.0) 70 : cluster [INF] Active manager daemon y restarted 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: cluster 2024-02-12T15:48:17.261474+0000 mon.a (mon.0) 71 : cluster [INF] Activating manager daemon y 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: cluster 2024-02-12T15:48:17.267598+0000 mon.a (mon.0) 72 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: cluster 2024-02-12T15:48:17.267833+0000 mon.a (mon.0) 73 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00652315s) 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: audit 2024-02-12T15:48:17.270089+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: audit 2024-02-12T15:48:17.271731+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: audit 2024-02-12T15:48:17.272918+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: audit 2024-02-12T15:48:17.273138+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: audit 2024-02-12T15:48:17.273361+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-12T15:48:17.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18489]: cluster 2024-02-12T15:48:17.278180+0000 mon.a (mon.0) 79 : cluster [INF] Manager daemon y is now available 2024-02-12T15:48:17.911 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:48:17 smithi121 bash[18805]: Warning: Permanently added '172.21.15.121' (ECDSA) to the list of known hosts. 2024-02-12T15:48:18.296 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-02-12T15:48:18.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-12T15:48:18.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "initialized": true 2024-02-12T15:48:18.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-02-12T15:48:18.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:18 smithi121 bash[18489]: audit 2024-02-12T15:48:17.411514+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:48:18.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:18 smithi121 bash[18489]: audit 2024-02-12T15:48:17.415609+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:48:18.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:18 smithi121 bash[18489]: audit 2024-02-12T15:48:17.418285+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:48:18.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:18 smithi121 bash[18489]: audit 2024-02-12T15:48:17.468118+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-12T15:48:18.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:18 smithi121 bash[18489]: audit 2024-02-12T15:48:17.473601+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-12T15:48:18.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:18 smithi121 bash[18489]: cluster 2024-02-12T15:48:18.272097+0000 mon.a (mon.0) 85 : cluster [DBG] mgrmap e11: y(active, since 1.01078s) 2024-02-12T15:48:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:mgr epoch 9 is available 2024-02-12T15:48:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:Generating a dashboard self-signed certificate... 2024-02-12T15:48:19.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:19 smithi121 bash[18489]: audit 2024-02-12T15:48:18.269846+0000 mgr.y (mgr.14146) 1 : audit [DBG] from='client.14150 v1:172.21.15.121:0/3083002654' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-12T15:48:19.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:19 smithi121 bash[18489]: audit 2024-02-12T15:48:18.289441+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 v1:172.21.15.121:0/3083002654' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-12T15:48:20.440 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-12T15:48:20.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:20 smithi121 bash[18489]: cluster 2024-02-12T15:48:19.332926+0000 mon.a (mon.0) 86 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-02-12T15:48:20.997 INFO:teuthology.orchestra.run.smithi121.stderr:Creating initial admin user... 2024-02-12T15:48:21.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:21 smithi121 bash[18489]: audit 2024-02-12T15:48:20.347501+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 v1:172.21.15.121:0/276360060' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:48:21.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:21 smithi121 bash[18489]: audit 2024-02-12T15:48:20.429506+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:21.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:21 smithi121 bash[18489]: audit 2024-02-12T15:48:20.433599+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:22.701 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$cf7GZo37rt6RQKYf7q5HMeG87BDAZi7ddvN/gzSCR226rVy20lNaO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707752902, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-12T15:48:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:Fetching dashboard port number... 2024-02-12T15:48:24.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:23 smithi121 bash[18489]: audit 2024-02-12T15:48:22.453452+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 v1:172.21.15.121:0/4294819903' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:48:24.034 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:23 smithi121 bash[18489]: audit 2024-02-12T15:48:22.694049+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:24.708 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 8443 2024-02-12T15:48:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:24 smithi121 bash[18489]: cluster 2024-02-12T15:48:23.698017+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-02-12T15:48:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:24 smithi121 bash[18489]: audit 2024-02-12T15:48:24.310140+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:24 smithi121 bash[18489]: audit 2024-02-12T15:48:24.701153+0000 mon.a (mon.0) 92 : audit [DBG] from='client.? v1:172.21.15.121:0/1041076695' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-12T15:48:25.335 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld does not appear to be present 2024-02-12T15:48:25.335 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-12T15:48:25.338 INFO:teuthology.orchestra.run.smithi121.stderr:Ceph Dashboard is now available at: 2024-02-12T15:48:25.338 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:25.339 INFO:teuthology.orchestra.run.smithi121.stderr: URL: https://smithi121.front.sepia.ceph.com:8443/ 2024-02-12T15:48:25.339 INFO:teuthology.orchestra.run.smithi121.stderr: User: admin 2024-02-12T15:48:25.339 INFO:teuthology.orchestra.run.smithi121.stderr: Password: 4lnfam31dn 2024-02-12T15:48:25.339 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:25.339 INFO:teuthology.orchestra.run.smithi121.stderr:Enabling autotune for osd_memory_target 2024-02-12T15:48:28.784 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-12T15:48:29.328 INFO:teuthology.orchestra.run.smithi121.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-12T15:48:29.328 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.328 INFO:teuthology.orchestra.run.smithi121.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:Or, if you are only running a single cluster on this host: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: ceph telemetry on 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:For more information see: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T15:48:29.329 INFO:teuthology.orchestra.run.smithi121.stderr:Bootstrap complete. 2024-02-12T15:48:29.368 INFO:tasks.cephadm:Fetching config... 2024-02-12T15:48:29.369 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:48:29.369 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-12T15:48:29.374 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-12T15:48:29.374 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:48:29.374 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-12T15:48:29.422 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-12T15:48:29.422 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:48:29.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-12T15:48:29.479 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-12T15:48:29.479 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:48:29.479 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-12T15:48:29.526 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-12T15:48:29.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLD0s8gfdVv8R10+qI9CiZxflmuQZ8YOYBi9FA42Ohpmh/5WYIJ/U8biRhaPV69HtQHkFfOGTj8XzDG+boC6mBvfFm7U18KL5KTEq/kFs6e0uvBpU8rcI0I1BhQGd7fDszmqd9xkercRNgtQva6eTGSVlxlxk+kZV62Y8Lw3kExBdbkXT97uIxS5vqTPA+eN4/817aJiY3a5a2MmPuysFEFNiOroxqXFz6Vnuhttjj4HWtfGQYmHxK81t+kbdtd4MVXJ8WMCVJY/9sy0/Z8Z8LjtxF2VAfw5zG9pJ4IUNSzOWpb57AOTP9xn0EjmsGXyRACiSIUf3OFrAwx9wuc5l3gyk6ArmLId/Cf7m417yvX2Bmq8xvzUozfcAEJFlbqzpCgFh6KR36oZFFdistc5trv76BuIj/ACmVH4CAWSQkOhuLzSnmpbknN3Ewqgl/Oj87ZWx6df5Wt78BtFZHb5gXH5rhMd0VsrFoS0in7Y7Y9R4IWZCZS+HEPwJZnN6dk58= ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-12T15:48:29.593 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLD0s8gfdVv8R10+qI9CiZxflmuQZ8YOYBi9FA42Ohpmh/5WYIJ/U8biRhaPV69HtQHkFfOGTj8XzDG+boC6mBvfFm7U18KL5KTEq/kFs6e0uvBpU8rcI0I1BhQGd7fDszmqd9xkercRNgtQva6eTGSVlxlxk+kZV62Y8Lw3kExBdbkXT97uIxS5vqTPA+eN4/817aJiY3a5a2MmPuysFEFNiOroxqXFz6Vnuhttjj4HWtfGQYmHxK81t+kbdtd4MVXJ8WMCVJY/9sy0/Z8Z8LjtxF2VAfw5zG9pJ4IUNSzOWpb57AOTP9xn0EjmsGXyRACiSIUf3OFrAwx9wuc5l3gyk6ArmLId/Cf7m417yvX2Bmq8xvzUozfcAEJFlbqzpCgFh6KR36oZFFdistc5trv76BuIj/ACmVH4CAWSQkOhuLzSnmpbknN3Ewqgl/Oj87ZWx6df5Wt78BtFZHb5gXH5rhMd0VsrFoS0in7Y7Y9R4IWZCZS+HEPwJZnN6dk58= ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:48:29.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLD0s8gfdVv8R10+qI9CiZxflmuQZ8YOYBi9FA42Ohpmh/5WYIJ/U8biRhaPV69HtQHkFfOGTj8XzDG+boC6mBvfFm7U18KL5KTEq/kFs6e0uvBpU8rcI0I1BhQGd7fDszmqd9xkercRNgtQva6eTGSVlxlxk+kZV62Y8Lw3kExBdbkXT97uIxS5vqTPA+eN4/817aJiY3a5a2MmPuysFEFNiOroxqXFz6Vnuhttjj4HWtfGQYmHxK81t+kbdtd4MVXJ8WMCVJY/9sy0/Z8Z8LjtxF2VAfw5zG9pJ4IUNSzOWpb57AOTP9xn0EjmsGXyRACiSIUf3OFrAwx9wuc5l3gyk6ArmLId/Cf7m417yvX2Bmq8xvzUozfcAEJFlbqzpCgFh6KR36oZFFdistc5trv76BuIj/ACmVH4CAWSQkOhuLzSnmpbknN3Ewqgl/Oj87ZWx6df5Wt78BtFZHb5gXH5rhMd0VsrFoS0in7Y7Y9R4IWZCZS+HEPwJZnN6dk58= ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-12T15:48:29.631 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLD0s8gfdVv8R10+qI9CiZxflmuQZ8YOYBi9FA42Ohpmh/5WYIJ/U8biRhaPV69HtQHkFfOGTj8XzDG+boC6mBvfFm7U18KL5KTEq/kFs6e0uvBpU8rcI0I1BhQGd7fDszmqd9xkercRNgtQva6eTGSVlxlxk+kZV62Y8Lw3kExBdbkXT97uIxS5vqTPA+eN4/817aJiY3a5a2MmPuysFEFNiOroxqXFz6Vnuhttjj4HWtfGQYmHxK81t+kbdtd4MVXJ8WMCVJY/9sy0/Z8Z8LjtxF2VAfw5zG9pJ4IUNSzOWpb57AOTP9xn0EjmsGXyRACiSIUf3OFrAwx9wuc5l3gyk6ArmLId/Cf7m417yvX2Bmq8xvzUozfcAEJFlbqzpCgFh6KR36oZFFdistc5trv76BuIj/ACmVH4CAWSQkOhuLzSnmpbknN3Ewqgl/Oj87ZWx6df5Wt78BtFZHb5gXH5rhMd0VsrFoS0in7Y7Y9R4IWZCZS+HEPwJZnN6dk58= ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:48:29.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLD0s8gfdVv8R10+qI9CiZxflmuQZ8YOYBi9FA42Ohpmh/5WYIJ/U8biRhaPV69HtQHkFfOGTj8XzDG+boC6mBvfFm7U18KL5KTEq/kFs6e0uvBpU8rcI0I1BhQGd7fDszmqd9xkercRNgtQva6eTGSVlxlxk+kZV62Y8Lw3kExBdbkXT97uIxS5vqTPA+eN4/817aJiY3a5a2MmPuysFEFNiOroxqXFz6Vnuhttjj4HWtfGQYmHxK81t+kbdtd4MVXJ8WMCVJY/9sy0/Z8Z8LjtxF2VAfw5zG9pJ4IUNSzOWpb57AOTP9xn0EjmsGXyRACiSIUf3OFrAwx9wuc5l3gyk6ArmLId/Cf7m417yvX2Bmq8xvzUozfcAEJFlbqzpCgFh6KR36oZFFdistc5trv76BuIj/ACmVH4CAWSQkOhuLzSnmpbknN3Ewqgl/Oj87ZWx6df5Wt78BtFZHb5gXH5rhMd0VsrFoS0in7Y7Y9R4IWZCZS+HEPwJZnN6dk58= ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-12T15:48:29.667 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLD0s8gfdVv8R10+qI9CiZxflmuQZ8YOYBi9FA42Ohpmh/5WYIJ/U8biRhaPV69HtQHkFfOGTj8XzDG+boC6mBvfFm7U18KL5KTEq/kFs6e0uvBpU8rcI0I1BhQGd7fDszmqd9xkercRNgtQva6eTGSVlxlxk+kZV62Y8Lw3kExBdbkXT97uIxS5vqTPA+eN4/817aJiY3a5a2MmPuysFEFNiOroxqXFz6Vnuhttjj4HWtfGQYmHxK81t+kbdtd4MVXJ8WMCVJY/9sy0/Z8Z8LjtxF2VAfw5zG9pJ4IUNSzOWpb57AOTP9xn0EjmsGXyRACiSIUf3OFrAwx9wuc5l3gyk6ArmLId/Cf7m417yvX2Bmq8xvzUozfcAEJFlbqzpCgFh6KR36oZFFdistc5trv76BuIj/ACmVH4CAWSQkOhuLzSnmpbknN3Ewqgl/Oj87ZWx6df5Wt78BtFZHb5gXH5rhMd0VsrFoS0in7Y7Y9R4IWZCZS+HEPwJZnN6dk58= ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:48:29.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-12T15:48:30.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:29 smithi121 bash[18489]: audit 2024-02-12T15:48:28.774259+0000 mon.a (mon.0) 93 : audit [INF] from='client.? v1:172.21.15.121:0/3459277301' entity='client.admin' 2024-02-12T15:48:34.830 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-12T15:48:34.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-12T15:48:35.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:35 smithi121 bash[18489]: audit 2024-02-12T15:48:34.251488+0000 mon.a (mon.0) 94 : audit [INF] from='client.? v1:172.21.15.121:0/1603845971' entity='client.admin' 2024-02-12T15:48:37.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:36 smithi121 bash[18489]: audit 2024-02-12T15:48:35.856782+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:37.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:36 smithi121 bash[18489]: audit 2024-02-12T15:48:36.176800+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:48:37.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:36 smithi121 bash[18489]: audit 2024-02-12T15:48:36.181437+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:37.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:36 smithi121 bash[18489]: audit 2024-02-12T15:48:36.183132+0000 mon.a (mon.0) 98 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:48:37.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:36 smithi121 bash[18489]: audit 2024-02-12T15:48:36.188947+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:38.569 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi123 2024-02-12T15:48:38.570 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:48:38.570 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.conf 2024-02-12T15:48:38.576 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:48:38.577 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:48:38.626 INFO:tasks.cephadm:Adding host smithi123 to orchestrator... 2024-02-12T15:48:38.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch host add smithi123 2024-02-12T15:48:39.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:38 smithi121 bash[18489]: cluster 2024-02-12T15:48:37.274510+0000 mgr.y (mgr.14146) 5 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:39.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:38 smithi121 bash[18489]: audit 2024-02-12T15:48:37.961957+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:39.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:38 smithi121 bash[18489]: audit 2024-02-12T15:48:38.008612+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:48:39.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:38 smithi121 bash[18489]: audit 2024-02-12T15:48:38.012220+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:48:39.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:38 smithi121 bash[18489]: audit 2024-02-12T15:48:38.014252+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:48:39.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:38 smithi121 bash[18489]: audit 2024-02-12T15:48:38.095818+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:39.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:38 smithi121 bash[18489]: audit 2024-02-12T15:48:38.100190+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:48:40.165 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:39 smithi121 bash[18489]: cephadm 2024-02-12T15:48:38.016355+0000 mgr.y (mgr.14146) 7 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2024-02-12T15:48:40.165 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:39 smithi121 bash[18489]: cephadm 2024-02-12T15:48:38.039148+0000 mgr.y (mgr.14146) 8 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:48:41.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:40 smithi121 bash[18489]: cluster 2024-02-12T15:48:39.274917+0000 mgr.y (mgr.14146) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:43.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:42 smithi121 bash[18489]: cluster 2024-02-12T15:48:41.275358+0000 mgr.y (mgr.14146) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:45.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:44 smithi121 bash[18489]: cluster 2024-02-12T15:48:43.275788+0000 mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:47.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:46 smithi121 bash[18489]: cluster 2024-02-12T15:48:45.276239+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:49.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:48 smithi121 bash[18489]: cluster 2024-02-12T15:48:47.276682+0000 mgr.y (mgr.14146) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:51.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:50 smithi121 bash[18489]: cluster 2024-02-12T15:48:49.277132+0000 mgr.y (mgr.14146) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:53.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:52 smithi121 bash[18489]: cluster 2024-02-12T15:48:51.277594+0000 mgr.y (mgr.14146) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:54 smithi121 bash[18489]: cluster 2024-02-12T15:48:53.278007+0000 mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:56 smithi121 bash[18489]: cluster 2024-02-12T15:48:55.278404+0000 mgr.y (mgr.14146) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:48:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:48:58 smithi121 bash[18489]: cluster 2024-02-12T15:48:57.278811+0000 mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:00 smithi121 bash[18489]: cluster 2024-02-12T15:48:59.279210+0000 mgr.y (mgr.14146) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:03.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:02 smithi121 bash[18489]: cluster 2024-02-12T15:49:01.279676+0000 mgr.y (mgr.14146) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:04.410 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:49:03 smithi121 bash[18805]: Warning: Permanently added 'smithi123,172.21.15.123' (ECDSA) to the list of known hosts. 2024-02-12T15:49:04.708 INFO:teuthology.orchestra.run.smithi123.stdout:Added host 'smithi123' with addr '172.21.15.123' 2024-02-12T15:49:05.159 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:49:04 smithi121 bash[18805]: Warning: Permanently added '172.21.15.123' (ECDSA) to the list of known hosts. 2024-02-12T15:49:05.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:04 smithi121 bash[18489]: cluster 2024-02-12T15:49:03.280108+0000 mgr.y (mgr.14146) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:05.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:04 smithi121 bash[18489]: audit 2024-02-12T15:49:03.871425+0000 mgr.y (mgr.14146) 22 : audit [DBG] from='client.14170 v1:172.21.15.123:0/1332884091' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:49:05.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:04 smithi121 bash[18489]: audit 2024-02-12T15:49:04.702248+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:05.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:04 smithi121 bash[18489]: audit 2024-02-12T15:49:04.748460+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:49:05.434 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-12T15:49:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:05 smithi121 bash[18489]: cephadm 2024-02-12T15:49:04.703073+0000 mgr.y (mgr.14146) 23 : cephadm [INF] Added host smithi123 2024-02-12T15:49:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:05 smithi121 bash[18489]: audit 2024-02-12T15:49:05.330736+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:06 smithi121 bash[18489]: cluster 2024-02-12T15:49:05.280619+0000 mgr.y (mgr.14146) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:06 smithi121 bash[18489]: audit 2024-02-12T15:49:06.630674+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:08.226 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-02-12T15:49:08.227 INFO:teuthology.orchestra.run.smithi123.stdout:[{"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}] 2024-02-12T15:49:09.016 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi169 2024-02-12T15:49:09.016 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:49:09.016 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.conf 2024-02-12T15:49:09.024 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:49:09.024 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:49:09.071 INFO:tasks.cephadm:Adding host smithi169 to orchestrator... 2024-02-12T15:49:09.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch host add smithi169 2024-02-12T15:49:09.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:08 smithi121 bash[18489]: cluster 2024-02-12T15:49:07.281026+0000 mgr.y (mgr.14146) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:10.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:09 smithi121 bash[18489]: audit 2024-02-12T15:49:08.220775+0000 mgr.y (mgr.14146) 26 : audit [DBG] from='client.14172 v1:172.21.15.123:0/3326951233' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T15:49:11.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:10 smithi121 bash[18489]: cluster 2024-02-12T15:49:09.281445+0000 mgr.y (mgr.14146) 27 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: cluster 2024-02-12T15:49:11.281833+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: audit 2024-02-12T15:49:12.104237+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: audit 2024-02-12T15:49:12.394518+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: audit 2024-02-12T15:49:12.396150+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: audit 2024-02-12T15:49:12.398296+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: audit 2024-02-12T15:49:12.400007+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: audit 2024-02-12T15:49:12.439712+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:12 smithi121 bash[18489]: audit 2024-02-12T15:49:12.446025+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:14.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:13 smithi121 bash[18489]: cephadm 2024-02-12T15:49:12.402681+0000 mgr.y (mgr.14146) 29 : cephadm [INF] Updating smithi123:/etc/ceph/ceph.conf 2024-02-12T15:49:14.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:13 smithi121 bash[18489]: cephadm 2024-02-12T15:49:12.417756+0000 mgr.y (mgr.14146) 30 : cephadm [INF] Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:49:15.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:14 smithi121 bash[18489]: cluster 2024-02-12T15:49:13.282225+0000 mgr.y (mgr.14146) 31 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:17.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:16 smithi121 bash[18489]: cluster 2024-02-12T15:49:15.282604+0000 mgr.y (mgr.14146) 32 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:19.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:18 smithi121 bash[18489]: cluster 2024-02-12T15:49:17.283021+0000 mgr.y (mgr.14146) 33 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:21.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:20 smithi121 bash[18489]: cluster 2024-02-12T15:49:19.283452+0000 mgr.y (mgr.14146) 34 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:23.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:22 smithi121 bash[18489]: cluster 2024-02-12T15:49:21.283860+0000 mgr.y (mgr.14146) 35 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:25.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:24 smithi121 bash[18489]: cluster 2024-02-12T15:49:23.284243+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:26 smithi121 bash[18489]: cluster 2024-02-12T15:49:25.284645+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:29.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:28 smithi121 bash[18489]: cluster 2024-02-12T15:49:27.285060+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:31.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:30 smithi121 bash[18489]: cluster 2024-02-12T15:49:29.285437+0000 mgr.y (mgr.14146) 39 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:32.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:31 smithi121 bash[18489]: cluster 2024-02-12T15:49:31.285881+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:34.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:34 smithi121 bash[18489]: cluster 2024-02-12T15:49:33.286290+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:34.658 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:49:34 smithi121 bash[18805]: Warning: Permanently added 'smithi169,172.21.15.169' (ECDSA) to the list of known hosts. 2024-02-12T15:49:34.973 INFO:teuthology.orchestra.run.smithi169.stdout:Added host 'smithi169' with addr '172.21.15.169' 2024-02-12T15:49:35.408 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:49:35 smithi121 bash[18805]: Warning: Permanently added '172.21.15.169' (ECDSA) to the list of known hosts. 2024-02-12T15:49:35.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-12T15:49:36.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:35 smithi121 bash[18489]: audit 2024-02-12T15:49:34.134424+0000 mgr.y (mgr.14146) 42 : audit [DBG] from='client.14174 v1:172.21.15.169:0/1450358254' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:49:36.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:35 smithi121 bash[18489]: audit 2024-02-12T15:49:34.969299+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:36.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:35 smithi121 bash[18489]: audit 2024-02-12T15:49:34.977974+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:49:36.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:35 smithi121 bash[18489]: audit 2024-02-12T15:49:35.543862+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:37.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:36 smithi121 bash[18489]: cephadm 2024-02-12T15:49:34.970048+0000 mgr.y (mgr.14146) 43 : cephadm [INF] Added host smithi169 2024-02-12T15:49:37.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:36 smithi121 bash[18489]: cluster 2024-02-12T15:49:35.286951+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:37.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:36 smithi121 bash[18489]: audit 2024-02-12T15:49:36.834637+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:38.674 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-02-12T15:49:38.674 INFO:teuthology.orchestra.run.smithi169.stdout:[{"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}] 2024-02-12T15:49:39.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:38 smithi121 bash[18489]: cluster 2024-02-12T15:49:37.287456+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:39.467 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-12T15:49:39.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-12T15:49:40.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:39 smithi121 bash[18489]: audit 2024-02-12T15:49:38.669602+0000 mgr.y (mgr.14146) 46 : audit [DBG] from='client.14176 v1:172.21.15.169:0/3901110923' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T15:49:41.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:40 smithi121 bash[18489]: cluster 2024-02-12T15:49:39.287856+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:42.931 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2024-02-12T15:49:43.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:42 smithi121 bash[18489]: cluster 2024-02-12T15:49:41.288295+0000 mgr.y (mgr.14146) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:43.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:42 smithi121 bash[18489]: audit 2024-02-12T15:49:42.419782+0000 mon.a (mon.0) 121 : audit [INF] from='client.? v1:172.21.15.121:0/3372754846' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-12T15:49:43.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:42 smithi121 bash[18489]: audit 2024-02-12T15:49:42.733048+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:43.497 INFO:tasks.cephadm:Adding mon.a on smithi121 2024-02-12T15:49:43.498 INFO:tasks.cephadm:Adding mon.b on smithi123 2024-02-12T15:49:43.498 INFO:tasks.cephadm:Adding mon.c on smithi169 2024-02-12T15:49:43.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch apply mon '3;smithi121:[v1:172.21.15.121:6789]=a;smithi123:[v1:172.21.15.123:6789]=b;smithi169:[v1:172.21.15.169:6789]=c' 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: audit 2024-02-12T15:49:42.921858+0000 mon.a (mon.0) 123 : audit [INF] from='client.? v1:172.21.15.121:0/3372754846' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: cluster 2024-02-12T15:49:42.921978+0000 mon.a (mon.0) 124 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: audit 2024-02-12T15:49:43.027273+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: audit 2024-02-12T15:49:43.028692+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: audit 2024-02-12T15:49:43.030429+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: audit 2024-02-12T15:49:43.031862+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: audit 2024-02-12T15:49:43.088200+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:44.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:43 smithi121 bash[18489]: audit 2024-02-12T15:49:43.094290+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:44 smithi121 bash[18489]: cephadm 2024-02-12T15:49:43.034485+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2024-02-12T15:49:45.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:44 smithi121 bash[18489]: cephadm 2024-02-12T15:49:43.057759+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:49:45.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:44 smithi121 bash[18489]: cluster 2024-02-12T15:49:43.288725+0000 mgr.y (mgr.14146) 51 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:46.308 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mon update... 2024-02-12T15:49:46.827 DEBUG:teuthology.orchestra.run.smithi123:mon.b> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.b.service 2024-02-12T15:49:46.829 DEBUG:teuthology.orchestra.run.smithi169:mon.c> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.c.service 2024-02-12T15:49:46.831 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-12T15:49:46.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T15:49:46.931 INFO:journalctl@ceph.mon.c.smithi169.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:49:47.207 INFO:journalctl@ceph.mon.b.smithi123.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:49:47.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: cluster 2024-02-12T15:49:45.289174+0000 mgr.y (mgr.14146) 52 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: audit 2024-02-12T15:49:46.304741+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: audit 2024-02-12T15:49:46.317629+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:49:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: audit 2024-02-12T15:49:46.600619+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:49:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: audit 2024-02-12T15:49:46.602114+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:49:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: audit 2024-02-12T15:49:46.609396+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: audit 2024-02-12T15:49:46.612299+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:49:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:46 smithi121 bash[18489]: audit 2024-02-12T15:49:46.613328+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:49:48.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:47 smithi121 bash[18489]: audit 2024-02-12T15:49:46.294245+0000 mgr.y (mgr.14146) 53 : audit [DBG] from='client.14180 v1:172.21.15.169:0/1598936936' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:[v1:172.21.15.121:6789]=a;smithi123:[v1:172.21.15.123:6789]=b;smithi169:[v1:172.21.15.169:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:49:48.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:47 smithi121 bash[18489]: cephadm 2024-02-12T15:49:46.299959+0000 mgr.y (mgr.14146) 54 : cephadm [INF] Saving service mon spec with placement smithi121:[v1:172.21.15.121:6789]=a;smithi123:[v1:172.21.15.123:6789]=b;smithi169:[v1:172.21.15.169:6789]=c;count:3 2024-02-12T15:49:48.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:47 smithi121 bash[18489]: cephadm 2024-02-12T15:49:46.614332+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Deploying daemon mon.c on smithi169 2024-02-12T15:49:49.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:48 smithi121 bash[18489]: cluster 2024-02-12T15:49:47.289613+0000 mgr.y (mgr.14146) 56 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:50.694 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-02-12T15:49:50.694 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","modified":"2024-02-12T15:47:14.396629Z","created":"2024-02-12T15:47:14.396629Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-12T15:49:50.697 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-02-12T15:49:51.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:50 smithi121 bash[18489]: cluster 2024-02-12T15:49:49.290057+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:51.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:50 smithi121 bash[18489]: audit 2024-02-12T15:49:50.691475+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.169:0/520879038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T15:49:52.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:51 smithi169 systemd[1]: Started Ceph mon.c for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T15:49:52.389 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-12T15:49:52.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.695+0000 7ff671a13880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.695+0000 7ff671a13880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 8 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.695+0000 7ff671a13880 0 pidfile_write: ignore empty --pid-file 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 0 load: jerasure load: lrc load: isa 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: DB SUMMARY 2024-02-12T15:49:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: CURRENT file: CURRENT 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 476 ; 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.create_if_missing: 0 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.env: 0x55e0534a7080 2024-02-12T15:49:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.fs: Posix File System 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.info_log: 0x55e0554ef980 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.statistics: (nil) 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.use_fsync: 0 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T15:49:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.db_log_dir: 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T15:49:53.050 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.write_buffer_manager: 0x55e0561ee3f0 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T15:49:53.051 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T15:49:53.053 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.unordered_write: 0 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.row_cache: None 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.wal_filter: None 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T15:49:53.054 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T15:49:53.055 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_open_files: -1 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Compression algorithms supported: 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kZSTD supported: 0 2024-02-12T15:49:53.056 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.merge_operator: 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_filter: None 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T15:49:53.057 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e05549dd28) 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: cache_index_and_filter_blocks: 1 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: pin_top_level_index_and_filter: 1 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: index_type: 0 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: data_block_index_type: 0 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: index_shortening: 1 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: data_block_hash_table_util_ratio: 0.750000 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: hash_index_allow_collision: 1 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: checksum: 1 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: no_block_cache: 0 2024-02-12T15:49:53.058 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_cache: 0x55e0554d4f10 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_cache_name: BinnedLRUCache 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_cache_options: 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: capacity : 536870912 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: num_shard_bits : 4 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: strict_capacity_limit : 0 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: high_pri_pool_ratio: 0.000 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_cache_compressed: (nil) 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: persistent_cache: (nil) 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_size: 4096 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_size_deviation: 10 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_restart_interval: 16 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: index_block_restart_interval: 1 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: metadata_block_size: 4096 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: partition_filters: 0 2024-02-12T15:49:53.059 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: use_delta_encoding: 1 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: whole_key_filtering: 1 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: verify_compression: 0 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: read_amp_bytes_per_bit: 0 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: format_version: 2 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: enable_index_compression: 1 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: block_align: 0 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compression: NoCompression 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T15:49:53.060 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.num_levels: 7 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T15:49:53.061 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T15:49:53.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T15:49:53.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T15:49:53.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.ttl: 2592000 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.699+0000 7ff671a13880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752992704615, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-12T15:49:53.065 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752992705371, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1402, "table_properties": {"data_size": 488, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707752992, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752992706424, "job": 1, "event": "recovery_finished"} 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff671a13880 4 rocksdb: DB pointer 0x55e05559d800 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff658804700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.703+0000 7ff658804700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ** DB Stats ** 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T15:49:53.066 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ** Compaction Stats [default] ** 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ** Compaction Stats [default] ** 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:53.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Cumulative compaction: 0.00 GB write, 0.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Interval compaction: 0.00 GB write, 0.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ** Compaction Stats [default] ** 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T15:49:53.068 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ** Compaction Stats [default] ** 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Cumulative compaction: 0.00 GB write, 0.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:49:53.069 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.707+0000 7ff671a13880 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.707+0000 7ff671a13880 0 using public_addrv v1:172.21.15.169:6789/0 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.707+0000 7ff671a13880 0 starting mon.c rank -1 at public addrs v1:172.21.15.169:6789/0 at bind addrs v1:172.21.15.169:6789/0 mon_data /var/lib/ceph/mon/ceph-c fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.707+0000 7ff671a13880 1 mon.c@-1(???) e0 preinit fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.743+0000 7ff656800700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.743+0000 7ff660814700 4 rocksdb: (Original Log Time 2024/02/12-15:49:52.748144) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.743+0000 7ff660814700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.743+0000 7ff660814700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752992748236, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1238, "num_deletes": 21, "total_data_size": 4218756, "memory_usage": 4243544, "flush_reason": "Manual Compaction"} 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.743+0000 7ff660814700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 0 mon.c@-1(synchronizing).mds e1 new map 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: e1 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-12T15:49:53.070 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: legacy client fscid: -1 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: No filesystems configured 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.747+0000 7ff65b009700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.751+0000 7ff65b009700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.751+0000 7ff65b009700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: cluster 2024-02-12T15:49:51.290467+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: audit 2024-02-12T15:49:51.679790+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: audit 2024-02-12T15:49:51.681723+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:49:53.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: audit 2024-02-12T15:49:51.683501+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:49:53.072 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: cephadm 2024-02-12T15:49:51.685139+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Deploying daemon mon.b on smithi123 2024-02-12T15:49:53.072 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:52 smithi169 bash[19604]: debug 2024-02-12T15:49:52.751+0000 7ff65b009700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-12T15:49:53.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:52 smithi121 bash[18489]: cluster 2024-02-12T15:49:51.290467+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:53.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:52 smithi121 bash[18489]: audit 2024-02-12T15:49:51.679790+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:53.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:52 smithi121 bash[18489]: audit 2024-02-12T15:49:51.681723+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:49:53.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:52 smithi121 bash[18489]: audit 2024-02-12T15:49:51.683501+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:49:53.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:52 smithi121 bash[18489]: cephadm 2024-02-12T15:49:51.685139+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Deploying daemon mon.b on smithi123 2024-02-12T15:49:55.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:55 smithi123 systemd[1]: Started Ceph mon.b for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.314+0000 7f0ac2caf880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.314+0000 7f0ac2caf880 0 ceph version 16.2.14-521-g0e714d9a (0e714d9a4bd2a821113e6318adb87bd06cf81ec1) pacific (stable), process ceph-mon, pid 8 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.314+0000 7f0ac2caf880 0 pidfile_write: ignore empty --pid-file 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 0 load: jerasure load: lrc load: isa 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: RocksDB version: 6.8.1 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Compile date Feb 8 2024 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: DB SUMMARY 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: CURRENT file: CURRENT 2024-02-12T15:49:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 476 ; 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.error_if_exists: 0 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.create_if_missing: 0 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.paranoid_checks: 1 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.env: 0x560bf9dbd080 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.fs: Posix File System 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.info_log: 0x560bfb35f980 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.statistics: (nil) 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.use_fsync: 0 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_log_file_size: 0 2024-02-12T15:49:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.allow_fallocate: 1 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.use_direct_reads: 0 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.db_log_dir: 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_subcompactions: 1 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_background_flushes: -1 2024-02-12T15:49:56.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.write_buffer_manager: 0x560bfc05e3f0 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-12T15:49:56.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.unordered_write: 0 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.row_cache: None 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.wal_filter: None 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-12T15:49:56.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.preserve_deletes: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.two_write_queues: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.atomic_flush: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.log_readahead_size: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_background_jobs: 2 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_background_compactions: -1 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-12T15:49:56.645 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_open_files: -1 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Compression algorithms supported: 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kZSTD supported: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kXpressCompression supported: 0 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kLZ4Compression supported: 1 2024-02-12T15:49:56.646 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kBZip2Compression supported: 0 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kZlibCompression supported: 1 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: kSnappyCompression supported: 1 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.merge_operator: 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_filter: None 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560bfb30dd28) 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: cache_index_and_filter_blocks: 1 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-12T15:49:56.647 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: pin_top_level_index_and_filter: 1 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: index_type: 0 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: data_block_index_type: 0 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: index_shortening: 1 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: data_block_hash_table_util_ratio: 0.750000 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: hash_index_allow_collision: 1 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: checksum: 1 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: no_block_cache: 0 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_cache: 0x560bfb344f10 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_cache_name: BinnedLRUCache 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_cache_options: 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: capacity : 536870912 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: num_shard_bits : 4 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: strict_capacity_limit : 0 2024-02-12T15:49:56.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: high_pri_pool_ratio: 0.000 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_cache_compressed: (nil) 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: persistent_cache: (nil) 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_size: 4096 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_size_deviation: 10 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_restart_interval: 16 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: index_block_restart_interval: 1 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: metadata_block_size: 4096 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: partition_filters: 0 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: use_delta_encoding: 1 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: whole_key_filtering: 1 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: verify_compression: 0 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: read_amp_bytes_per_bit: 0 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: format_version: 2 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: enable_index_compression: 1 2024-02-12T15:49:56.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: block_align: 0 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compression: NoCompression 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.num_levels: 7 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-12T15:49:56.650 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-12T15:49:56.651 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-12T15:49:56.652 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.table_properties_collectors: 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.inplace_update_support: 0 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-12T15:49:56.653 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.bloom_locality: 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.max_successive_merges: 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.ttl: 2592000 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.318+0000 7f0ac2caf880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.322+0000 7f0ac2caf880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.322+0000 7f0ac2caf880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.322+0000 7f0ac2caf880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752996325993, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.322+0000 7f0ac2caf880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.322+0000 7f0ac2caf880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.322+0000 7f0ac2caf880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752996327095, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1402, "table_properties": {"data_size": 488, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707752996, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.322+0000 7f0ac2caf880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.326+0000 7f0ac2caf880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752996328298, "job": 1, "event": "recovery_finished"} 2024-02-12T15:49:56.654 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.330+0000 7f0ac2caf880 4 rocksdb: DB pointer 0x560bfb40d800 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.330+0000 7f0aa9aa0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.330+0000 7f0aa9aa0700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ** DB Stats ** 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ** Compaction Stats [default] ** 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:56.655 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ** Compaction Stats [default] ** 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Cumulative compaction: 0.00 GB write, 0.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Interval compaction: 0.00 GB write, 0.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:49:56.656 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ** Compaction Stats [default] ** 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.3 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ** Compaction Stats [default] ** 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0.00 0.00 1 0.001 0 0 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(Total Files): cumulative 0, interval 0 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-12T15:49:56.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: AddFile(Keys): cumulative 0, interval 0 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Cumulative compaction: 0.00 GB write, 0.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: ** File Read Latency Histogram By Level [default] ** 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.330+0000 7f0ac2caf880 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.330+0000 7f0ac2caf880 0 using public_addrv v1:172.21.15.123:6789/0 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.330+0000 7f0ac2caf880 0 starting mon.b rank -1 at public addrs v1:172.21.15.123:6789/0 at bind addrs v1:172.21.15.123:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.330+0000 7f0ac2caf880 1 mon.b@-1(???) e0 preinit fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.362+0000 7f0aa7a9c700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.362+0000 7f0ab1ab0700 4 rocksdb: (Original Log Time 2024/02/12-15:49:56.365415) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.362+0000 7f0ab1ab0700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.362+0000 7f0ab1ab0700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707752996365537, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1238, "num_deletes": 21, "total_data_size": 4218756, "memory_usage": 4243592, "flush_reason": "Manual Compaction"} 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.362+0000 7f0ab1ab0700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 0 mon.b@-1(synchronizing).mds e1 new map 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-02-12T15:49:56.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: e1 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: legacy client fscid: -1 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: No filesystems configured 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: cluster 2024-02-12T15:49:51.290467+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:56.659 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: audit 2024-02-12T15:49:51.679790+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:56.660 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: audit 2024-02-12T15:49:51.681723+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:49:56.660 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: audit 2024-02-12T15:49:51.683501+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:49:56.660 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: cephadm 2024-02-12T15:49:51.685139+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Deploying daemon mon.b on smithi123 2024-02-12T15:49:56.660 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:49:56 smithi123 bash[19119]: debug 2024-02-12T15:49:56.366+0000 7f0aac2a5700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-12T15:49:58.086 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-02-12T15:49:58.087 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":2,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","modified":"2024-02-12T15:49:52.759277Z","created":"2024-02-12T15:47:14.396629Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-12T15:49:58.090 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 2 2024-02-12T15:49:58.144 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:52.764326+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:49:58.144 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:52.764730+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-02-12T15:49:58.144 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:52.766846+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:53.290879+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:53.757855+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:54.757914+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:54.762736+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:55.291251+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:55.758041+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:56.376660+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:56.758190+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:57.376413+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:57.758369+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:57.788725+0000 mon.a (mon.0) 153 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:57.795362+0000 mon.a (mon.0) 154 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.121:6789/0,c=v1:172.21.15.169:6789/0} removed_ranks: {} 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:57.796253+0000 mon.a (mon.0) 155 : cluster [DBG] fsmap 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:57.796304+0000 mon.a (mon.0) 156 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T15:49:58.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:57.796611+0000 mon.a (mon.0) 157 : cluster [DBG] mgrmap e13: y(active, since 100s) 2024-02-12T15:49:58.146 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: cluster 2024-02-12T15:49:57.803561+0000 mon.a (mon.0) 158 : cluster [INF] overall HEALTH_OK 2024-02-12T15:49:58.146 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:57.807667+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:58.146 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:49:57 smithi169 bash[19604]: audit 2024-02-12T15:49:57.810136+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:52.764326+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:52.764730+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:52.766846+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:53.290879+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:53.757855+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:54.757914+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:54.762736+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:55.291251+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:55.758041+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:56.376660+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:56.758190+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:57.376413+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:49:58.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:57.758369+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:57.788725+0000 mon.a (mon.0) 153 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:57.795362+0000 mon.a (mon.0) 154 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.121:6789/0,c=v1:172.21.15.169:6789/0} removed_ranks: {} 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:57.796253+0000 mon.a (mon.0) 155 : cluster [DBG] fsmap 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:57.796304+0000 mon.a (mon.0) 156 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:57.796611+0000 mon.a (mon.0) 157 : cluster [DBG] mgrmap e13: y(active, since 100s) 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: cluster 2024-02-12T15:49:57.803561+0000 mon.a (mon.0) 158 : cluster [INF] overall HEALTH_OK 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:57.807667+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:49:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:49:57 smithi121 bash[18489]: audit 2024-02-12T15:49:57.810136+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:49:59.157 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:49:58 smithi121 bash[18805]: debug 2024-02-12T15:49:58.756+0000 7fb564e0b700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-12T15:49:59.687 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-12T15:49:59.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-12T15:50:03.707 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-02-12T15:50:03.707 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":3,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","modified":"2024-02-12T15:49:58.378559Z","created":"2024-02-12T15:47:14.396629Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-12T15:50:03.710 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 3 2024-02-12T15:50:03.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:49:58.384813+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:50:03.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:49:58.385254+0000 mon.a (mon.0) 164 : cluster [INF] mon.a calling monitor election 2024-02-12T15:50:03.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:49:58.387556+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-12T15:50:03.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:49:58.388047+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:49:58.388455+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:49:59.292062+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:49:59.376601+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:00.377000+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:00.381571+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:01.292477+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:01.377037+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:02.377315+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:03.377489+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:03.409437+0000 mon.a (mon.0) 172 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:03.416419+0000 mon.a (mon.0) 173 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.121:6789/0,b=v1:172.21.15.123:6789/0,c=v1:172.21.15.169:6789/0} removed_ranks: {} 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:03.417474+0000 mon.a (mon.0) 174 : cluster [DBG] fsmap 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:03.417533+0000 mon.a (mon.0) 175 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T15:50:03.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:03.417871+0000 mon.a (mon.0) 176 : cluster [DBG] mgrmap e13: y(active, since 106s) 2024-02-12T15:50:03.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: cluster 2024-02-12T15:50:03.426042+0000 mon.a (mon.0) 177 : cluster [INF] overall HEALTH_OK 2024-02-12T15:50:03.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:03.429787+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:03.436457+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:03.446027+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:03.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:03.449708+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.765 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:03 smithi169 bash[19604]: audit 2024-02-12T15:50:03.451560+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:52.764326+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:50:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:52.764730+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:52.766846+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:53.290879+0000 mgr.y (mgr.14146) 60 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:53.757855+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:54.757914+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:54.762736+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:55.291251+0000 mgr.y (mgr.14146) 61 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:55.758041+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:56.376660+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:56.758190+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:57.376413+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:57.758369+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:57.788725+0000 mon.a (mon.0) 153 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:57.795362+0000 mon.a (mon.0) 154 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.121:6789/0,c=v1:172.21.15.169:6789/0} removed_ranks: {} 2024-02-12T15:50:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:57.796253+0000 mon.a (mon.0) 155 : cluster [DBG] fsmap 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:57.796304+0000 mon.a (mon.0) 156 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:57.796611+0000 mon.a (mon.0) 157 : cluster [DBG] mgrmap e13: y(active, since 100s) 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:57.803561+0000 mon.a (mon.0) 158 : cluster [INF] overall HEALTH_OK 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:57.807667+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:57.810136+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:58.384813+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:58.385254+0000 mon.a (mon.0) 164 : cluster [INF] mon.a calling monitor election 2024-02-12T15:50:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:58.387556+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:58.388047+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:58.388455+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:49:59.292062+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:49:59.376601+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:00.377000+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:00.381571+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:01.292477+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:01.377037+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:02.377315+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:03.377489+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:03.409437+0000 mon.a (mon.0) 172 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:03.416419+0000 mon.a (mon.0) 173 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.121:6789/0,b=v1:172.21.15.123:6789/0,c=v1:172.21.15.169:6789/0} removed_ranks: {} 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:03.417474+0000 mon.a (mon.0) 174 : cluster [DBG] fsmap 2024-02-12T15:50:03.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:03.417533+0000 mon.a (mon.0) 175 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T15:50:03.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:03.417871+0000 mon.a (mon.0) 176 : cluster [DBG] mgrmap e13: y(active, since 106s) 2024-02-12T15:50:03.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: cluster 2024-02-12T15:50:03.426042+0000 mon.a (mon.0) 177 : cluster [INF] overall HEALTH_OK 2024-02-12T15:50:03.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:03.429787+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:03.436457+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:03.446027+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:03.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:03.449708+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:03 smithi123 bash[19119]: audit 2024-02-12T15:50:03.451560+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:03.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:49:58.384813+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:50:03.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:49:58.385254+0000 mon.a (mon.0) 164 : cluster [INF] mon.a calling monitor election 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:49:58.387556+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:49:58.388047+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:49:58.388455+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:49:59.292062+0000 mgr.y (mgr.14146) 63 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:49:59.376601+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:00.377000+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:00.381571+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:01.292477+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:01.377037+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:02.377315+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:03.377489+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:03.409437+0000 mon.a (mon.0) 172 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-12T15:50:03.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:03.416419+0000 mon.a (mon.0) 173 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.121:6789/0,b=v1:172.21.15.123:6789/0,c=v1:172.21.15.169:6789/0} removed_ranks: {} 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:03.417474+0000 mon.a (mon.0) 174 : cluster [DBG] fsmap 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:03.417533+0000 mon.a (mon.0) 175 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:03.417871+0000 mon.a (mon.0) 176 : cluster [DBG] mgrmap e13: y(active, since 106s) 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: cluster 2024-02-12T15:50:03.426042+0000 mon.a (mon.0) 177 : cluster [INF] overall HEALTH_OK 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:03.429787+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:03.436457+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:03.446027+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:03.449708+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:03.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:03 smithi121 bash[18489]: audit 2024-02-12T15:50:03.451560+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:04.354 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-12T15:50:04.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph config generate-minimal-conf 2024-02-12T15:50:04.657 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18805]: debug 2024-02-12T15:50:04.373+0000 7fb564e0b700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-12T15:50:04.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: cephadm 2024-02-12T15:50:03.453422+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2024-02-12T15:50:04.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: cephadm 2024-02-12T15:50:03.454034+0000 mgr.y (mgr.14146) 67 : cephadm [INF] Updating smithi123:/etc/ceph/ceph.conf 2024-02-12T15:50:04.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: cephadm 2024-02-12T15:50:03.454406+0000 mgr.y (mgr.14146) 68 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.490424+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.498125+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.552216+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.558657+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: cephadm 2024-02-12T15:50:03.561559+0000 mgr.y (mgr.14146) 69 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.562148+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.563571+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.566208+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.566521+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.566711+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.570815+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: cephadm 2024-02-12T15:50:03.574881+0000 mgr.y (mgr.14146) 70 : cephadm [INF] Reconfiguring daemon mon.a on smithi121 2024-02-12T15:50:04.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:04 smithi121 bash[18489]: audit 2024-02-12T15:50:03.704687+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? v1:172.21.15.169:0/767910442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T15:50:04.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: cephadm 2024-02-12T15:50:03.453422+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2024-02-12T15:50:04.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: cephadm 2024-02-12T15:50:03.454034+0000 mgr.y (mgr.14146) 67 : cephadm [INF] Updating smithi123:/etc/ceph/ceph.conf 2024-02-12T15:50:04.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: cephadm 2024-02-12T15:50:03.454406+0000 mgr.y (mgr.14146) 68 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2024-02-12T15:50:04.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.490424+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.498125+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.552216+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.558657+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: cephadm 2024-02-12T15:50:03.561559+0000 mgr.y (mgr.14146) 69 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.562148+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.563571+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.566208+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.566521+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.566711+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.570815+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: cephadm 2024-02-12T15:50:03.574881+0000 mgr.y (mgr.14146) 70 : cephadm [INF] Reconfiguring daemon mon.a on smithi121 2024-02-12T15:50:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:04 smithi169 bash[19604]: audit 2024-02-12T15:50:03.704687+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? v1:172.21.15.169:0/767910442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T15:50:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: cephadm 2024-02-12T15:50:03.453422+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Updating smithi121:/etc/ceph/ceph.conf 2024-02-12T15:50:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: cephadm 2024-02-12T15:50:03.454034+0000 mgr.y (mgr.14146) 67 : cephadm [INF] Updating smithi123:/etc/ceph/ceph.conf 2024-02-12T15:50:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: cephadm 2024-02-12T15:50:03.454406+0000 mgr.y (mgr.14146) 68 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2024-02-12T15:50:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.490424+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.498125+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.552216+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.558657+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: cephadm 2024-02-12T15:50:03.561559+0000 mgr.y (mgr.14146) 69 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.562148+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.563571+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.566208+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.566521+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.566711+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.570815+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: cephadm 2024-02-12T15:50:03.574881+0000 mgr.y (mgr.14146) 70 : cephadm [INF] Reconfiguring daemon mon.a on smithi121 2024-02-12T15:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:04 smithi123 bash[19119]: audit 2024-02-12T15:50:03.704687+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? v1:172.21.15.169:0/767910442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T15:50:07.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:06 smithi169 bash[19604]: cluster 2024-02-12T15:50:05.293300+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:07.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:06 smithi169 bash[19604]: audit 2024-02-12T15:50:05.762194+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:07.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:06 smithi169 bash[19604]: cephadm 2024-02-12T15:50:05.763323+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-12T15:50:07.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:06 smithi169 bash[19604]: audit 2024-02-12T15:50:05.763775+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:07.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:06 smithi169 bash[19604]: audit 2024-02-12T15:50:05.765050+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:07.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:06 smithi169 bash[19604]: audit 2024-02-12T15:50:05.766248+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:07.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:06 smithi169 bash[19604]: cephadm 2024-02-12T15:50:05.767412+0000 mgr.y (mgr.14146) 73 : cephadm [INF] Reconfiguring daemon mon.b on smithi123 2024-02-12T15:50:07.097 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:06 smithi123 bash[19119]: cluster 2024-02-12T15:50:05.293300+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:07.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:06 smithi123 bash[19119]: audit 2024-02-12T15:50:05.762194+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:07.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:06 smithi123 bash[19119]: cephadm 2024-02-12T15:50:05.763323+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-12T15:50:07.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:06 smithi123 bash[19119]: audit 2024-02-12T15:50:05.763775+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:07.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:06 smithi123 bash[19119]: audit 2024-02-12T15:50:05.765050+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:07.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:06 smithi123 bash[19119]: audit 2024-02-12T15:50:05.766248+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:07.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:06 smithi123 bash[19119]: cephadm 2024-02-12T15:50:05.767412+0000 mgr.y (mgr.14146) 73 : cephadm [INF] Reconfiguring daemon mon.b on smithi123 2024-02-12T15:50:07.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:06 smithi121 bash[18489]: cluster 2024-02-12T15:50:05.293300+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:07.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:06 smithi121 bash[18489]: audit 2024-02-12T15:50:05.762194+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:07.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:06 smithi121 bash[18489]: cephadm 2024-02-12T15:50:05.763323+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-12T15:50:07.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:06 smithi121 bash[18489]: audit 2024-02-12T15:50:05.763775+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:07.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:06 smithi121 bash[18489]: audit 2024-02-12T15:50:05.765050+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:07.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:06 smithi121 bash[18489]: audit 2024-02-12T15:50:05.766248+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:07.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:06 smithi121 bash[18489]: cephadm 2024-02-12T15:50:05.767412+0000 mgr.y (mgr.14146) 73 : cephadm [INF] Reconfiguring daemon mon.b on smithi123 2024-02-12T15:50:07.838 INFO:teuthology.orchestra.run.smithi121.stdout:# minimal ceph.conf for eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:50:07.839 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-02-12T15:50:07.839 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T15:50:07.839 INFO:teuthology.orchestra.run.smithi121.stdout: mon_host = 172.21.15.121:6789/0 172.21.15.123:6789/0 172.21.15.169:6789/0 2024-02-12T15:50:08.372 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-12T15:50:08.372 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:50:08.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2024-02-12T15:50:08.381 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:50:08.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:50:08.438 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:50:08.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2024-02-12T15:50:08.454 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:50:08.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:50:08.511 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:50:08.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2024-02-12T15:50:08.526 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:50:08.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-12T15:50:08.587 INFO:tasks.cephadm:Adding mgr.y on smithi121 2024-02-12T15:50:08.587 INFO:tasks.cephadm:Adding mgr.x on smithi123 2024-02-12T15:50:08.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch apply mgr '2;smithi121=y;smithi123=x' 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: cluster 2024-02-12T15:50:07.293711+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: audit 2024-02-12T15:50:07.788926+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: cephadm 2024-02-12T15:50:07.790048+0000 mgr.y (mgr.14146) 75 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: audit 2024-02-12T15:50:07.790585+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: audit 2024-02-12T15:50:07.792116+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: audit 2024-02-12T15:50:07.793616+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: cephadm 2024-02-12T15:50:07.794970+0000 mgr.y (mgr.14146) 76 : cephadm [INF] Reconfiguring daemon mon.c on smithi169 2024-02-12T15:50:09.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:08 smithi169 bash[19604]: audit 2024-02-12T15:50:07.836684+0000 mon.a (mon.0) 202 : audit [DBG] from='client.? v1:172.21.15.121:0/4068678761' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:09.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: cluster 2024-02-12T15:50:07.293711+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:09.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: audit 2024-02-12T15:50:07.788926+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:09.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: cephadm 2024-02-12T15:50:07.790048+0000 mgr.y (mgr.14146) 75 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-12T15:50:09.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: audit 2024-02-12T15:50:07.790585+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:09.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: audit 2024-02-12T15:50:07.792116+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:09.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: audit 2024-02-12T15:50:07.793616+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: cephadm 2024-02-12T15:50:07.794970+0000 mgr.y (mgr.14146) 76 : cephadm [INF] Reconfiguring daemon mon.c on smithi169 2024-02-12T15:50:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:08 smithi123 bash[19119]: audit 2024-02-12T15:50:07.836684+0000 mon.a (mon.0) 202 : audit [DBG] from='client.? v1:172.21.15.121:0/4068678761' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: cluster 2024-02-12T15:50:07.293711+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: audit 2024-02-12T15:50:07.788926+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: cephadm 2024-02-12T15:50:07.790048+0000 mgr.y (mgr.14146) 75 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: audit 2024-02-12T15:50:07.790585+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: audit 2024-02-12T15:50:07.792116+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: audit 2024-02-12T15:50:07.793616+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: cephadm 2024-02-12T15:50:07.794970+0000 mgr.y (mgr.14146) 76 : cephadm [INF] Reconfiguring daemon mon.c on smithi169 2024-02-12T15:50:09.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:08 smithi121 bash[18489]: audit 2024-02-12T15:50:07.836684+0000 mon.a (mon.0) 202 : audit [DBG] from='client.? v1:172.21.15.121:0/4068678761' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:11.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:10 smithi169 bash[19604]: cluster 2024-02-12T15:50:09.294132+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:11.315 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:10 smithi169 bash[19604]: audit 2024-02-12T15:50:10.073856+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:11.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:10 smithi169 bash[19604]: audit 2024-02-12T15:50:10.075884+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:11.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:10 smithi169 bash[19604]: audit 2024-02-12T15:50:10.079752+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:11.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:10 smithi169 bash[19604]: audit 2024-02-12T15:50:10.081456+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:11.316 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:10 smithi169 bash[19604]: audit 2024-02-12T15:50:10.091774+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:10 smithi123 bash[19119]: cluster 2024-02-12T15:50:09.294132+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:10 smithi123 bash[19119]: audit 2024-02-12T15:50:10.073856+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:10 smithi123 bash[19119]: audit 2024-02-12T15:50:10.075884+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:10 smithi123 bash[19119]: audit 2024-02-12T15:50:10.079752+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:10 smithi123 bash[19119]: audit 2024-02-12T15:50:10.081456+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:10 smithi123 bash[19119]: audit 2024-02-12T15:50:10.091774+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:11.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:10 smithi121 bash[18489]: cluster 2024-02-12T15:50:09.294132+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:11.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:10 smithi121 bash[18489]: audit 2024-02-12T15:50:10.073856+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:11.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:10 smithi121 bash[18489]: audit 2024-02-12T15:50:10.075884+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:11.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:10 smithi121 bash[18489]: audit 2024-02-12T15:50:10.079752+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:11.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:10 smithi121 bash[18489]: audit 2024-02-12T15:50:10.081456+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:11.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:10 smithi121 bash[18489]: audit 2024-02-12T15:50:10.091774+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:12.058 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mgr update... 2024-02-12T15:50:12.617 DEBUG:teuthology.orchestra.run.smithi123:mgr.x> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.x.service 2024-02-12T15:50:12.619 INFO:tasks.cephadm:Deploying OSDs... 2024-02-12T15:50:12.619 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:50:12.620 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-02-12T15:50:12.626 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-12T15:50:12.627 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 338 Links: 1 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-12 15:43:50.527105087 +0000 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-12 15:43:50.315110132 +0000 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-12 15:43:50.315110132 +0000 2024-02-12T15:50:12.675 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-02-12T15:50:12.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-12T15:50:12.729 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-12T15:50:12.729 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-12T15:50:12.729 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000410492 s, 1.2 MB/s 2024-02-12T15:50:12.731 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-12T15:50:12.780 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-02-12T15:50:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-12T15:50:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:12.830 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 344 Links: 1 2024-02-12T15:50:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-12 15:43:50.875096806 +0000 2024-02-12T15:50:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-12 15:43:50.639102422 +0000 2024-02-12T15:50:12.831 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-12 15:43:50.639102422 +0000 2024-02-12T15:50:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-02-12T15:50:12.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-12T15:50:12.885 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-12T15:50:12.885 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-12T15:50:12.885 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000397316 s, 1.3 MB/s 2024-02-12T15:50:12.886 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-12T15:50:12.936 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 350 Links: 1 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-12 15:43:51.003093760 +0000 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-12 15:43:50.999093856 +0000 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-12 15:43:50.999093856 +0000 2024-02-12T15:50:12.987 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-02-12T15:50:12.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-12T15:50:13.039 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: cluster 2024-02-12T15:50:11.294546+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:13.039 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.055755+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:13.039 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.065322+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:13.040 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.069268+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:13.040 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.070964+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:13.040 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.080219+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:13.040 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.094141+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T15:50:13.040 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.100073+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-12T15:50:13.040 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.101681+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T15:50:13.040 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:12 smithi121 bash[18489]: audit 2024-02-12T15:50:12.103783+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:13.041 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-12T15:50:13.041 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-12T15:50:13.041 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000339037 s, 1.5 MB/s 2024-02-12T15:50:13.042 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-12T15:50:13.092 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-02-12T15:50:13.142 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-12T15:50:13.143 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 356 Links: 1 2024-02-12T15:50:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-12 15:43:51.379084814 +0000 2024-02-12T15:50:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-12 15:43:51.375084907 +0000 2024-02-12T15:50:13.143 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-12 15:43:51.375084907 +0000 2024-02-12T15:50:13.143 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-02-12T15:50:13.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-12T15:50:13.197 INFO:journalctl@ceph.mgr.x.smithi123.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:50:13.197 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-12T15:50:13.197 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-12T15:50:13.198 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000394308 s, 1.3 MB/s 2024-02-12T15:50:13.198 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-12T15:50:13.247 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:50:13.248 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-02-12T15:50:13.254 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-12T15:50:13.254 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-02-12T15:50:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: cluster 2024-02-12T15:50:11.294546+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.055755+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.065322+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.069268+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.070964+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.080219+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.094141+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T15:50:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.100073+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-12T15:50:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.101681+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T15:50:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:12 smithi169 bash[19604]: audit 2024-02-12T15:50:12.103783+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-12T15:50:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:13.303 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 343 Links: 1 2024-02-12T15:50:13.303 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:13.303 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-02-12 15:43:51.007737320 +0000 2024-02-12T15:50:13.303 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-02-12 15:43:50.779742748 +0000 2024-02-12T15:50:13.304 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-02-12 15:43:50.779742748 +0000 2024-02-12T15:50:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-02-12T15:50:13.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-12T15:50:13.356 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: cluster 2024-02-12T15:50:11.294546+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:13.356 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.055755+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.065322+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.069268+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.070964+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.080219+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.094141+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.100073+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.101681+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T15:50:13.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:12 smithi123 bash[19119]: audit 2024-02-12T15:50:12.103783+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:13.358 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-02-12T15:50:13.358 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-02-12T15:50:13.358 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000410734 s, 1.2 MB/s 2024-02-12T15:50:13.359 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-12T15:50:13.409 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-02-12T15:50:13.460 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-12T15:50:13.461 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:13.461 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 350 Links: 1 2024-02-12T15:50:13.461 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:13.461 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-02-12 15:43:51.159733702 +0000 2024-02-12T15:50:13.461 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-02-12 15:43:51.155733797 +0000 2024-02-12T15:50:13.462 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-02-12 15:43:51.155733797 +0000 2024-02-12T15:50:13.462 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-02-12T15:50:13.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-12T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-02-12T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-02-12T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000395512 s, 1.3 MB/s 2024-02-12T15:50:13.518 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-12T15:50:13.568 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-02-12T15:50:13.623 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-12T15:50:13.624 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:13.624 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 356 Links: 1 2024-02-12T15:50:13.624 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:13.624 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-02-12 15:43:51.679721325 +0000 2024-02-12T15:50:13.624 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-02-12 15:43:51.471726275 +0000 2024-02-12T15:50:13.624 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-02-12 15:43:51.471726275 +0000 2024-02-12T15:50:13.624 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-02-12T15:50:13.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-12T15:50:13.678 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-02-12T15:50:13.678 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-02-12T15:50:13.678 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000390778 s, 1.3 MB/s 2024-02-12T15:50:13.680 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-12T15:50:13.727 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 362 Links: 1 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-02-12 15:43:52.055712377 +0000 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-02-12 15:43:51.811718183 +0000 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-02-12 15:43:51.811718183 +0000 2024-02-12T15:50:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-02-12T15:50:13.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-12T15:50:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-02-12T15:50:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-02-12T15:50:13.830 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000337285 s, 1.5 MB/s 2024-02-12T15:50:13.832 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-12T15:50:13.881 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:50:13.881 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-02-12T15:50:13.888 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-12T15:50:13.888 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-02-12T15:50:13.938 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-12T15:50:13.938 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:13.939 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 342 Links: 1 2024-02-12T15:50:13.939 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:13.939 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-02-12 15:43:56.291099379 +0000 2024-02-12T15:50:13.939 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-02-12 15:43:56.083104330 +0000 2024-02-12T15:50:13.939 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-02-12 15:43:56.083104330 +0000 2024-02-12T15:50:13.939 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-02-12T15:50:13.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-12T15:50:13.994 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-02-12T15:50:13.994 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-02-12T15:50:13.994 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000385348 s, 1.3 MB/s 2024-02-12T15:50:13.995 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-12T15:50:14.045 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-02-12T15:50:14.095 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-12T15:50:14.095 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:14.096 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 348 Links: 1 2024-02-12T15:50:14.096 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:14.096 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-02-12 15:43:56.623091476 +0000 2024-02-12T15:50:14.096 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-02-12 15:43:56.443095760 +0000 2024-02-12T15:50:14.096 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-02-12 15:43:56.443095760 +0000 2024-02-12T15:50:14.096 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-02-12T15:50:14.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-12T15:50:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:13 smithi123 bash[19119]: audit 2024-02-12T15:50:12.045925+0000 mgr.y (mgr.14146) 79 : audit [DBG] from='client.24109 v1:172.21.15.169:0/813284808' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi121=y;smithi123=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:50:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:13 smithi123 bash[19119]: cephadm 2024-02-12T15:50:12.049373+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Saving service mgr spec with placement smithi121=y;smithi123=x;count:2 2024-02-12T15:50:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:13 smithi123 bash[19119]: cephadm 2024-02-12T15:50:12.105233+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Deploying daemon mgr.x on smithi123 2024-02-12T15:50:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-02-12T15:50:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-02-12T15:50:14.150 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.0003919 s, 1.3 MB/s 2024-02-12T15:50:14.151 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-12T15:50:14.201 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-02-12T15:50:14.251 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-12T15:50:14.251 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:14.251 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 354 Links: 1 2024-02-12T15:50:14.251 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:14.252 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-02-12 15:43:56.971083192 +0000 2024-02-12T15:50:14.252 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-02-12 15:43:56.755088333 +0000 2024-02-12T15:50:14.252 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-02-12 15:43:56.755088333 +0000 2024-02-12T15:50:14.252 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-02-12T15:50:14.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-12T15:50:14.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:13 smithi169 bash[19604]: audit 2024-02-12T15:50:12.045925+0000 mgr.y (mgr.14146) 79 : audit [DBG] from='client.24109 v1:172.21.15.169:0/813284808' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi121=y;smithi123=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:50:14.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:13 smithi169 bash[19604]: cephadm 2024-02-12T15:50:12.049373+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Saving service mgr spec with placement smithi121=y;smithi123=x;count:2 2024-02-12T15:50:14.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:13 smithi169 bash[19604]: cephadm 2024-02-12T15:50:12.105233+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Deploying daemon mgr.x on smithi123 2024-02-12T15:50:14.306 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-02-12T15:50:14.306 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-02-12T15:50:14.306 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000389607 s, 1.3 MB/s 2024-02-12T15:50:14.307 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-12T15:50:14.357 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-02-12T15:50:14.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:13 smithi121 bash[18489]: audit 2024-02-12T15:50:12.045925+0000 mgr.y (mgr.14146) 79 : audit [DBG] from='client.24109 v1:172.21.15.169:0/813284808' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi121=y;smithi123=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:50:14.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:13 smithi121 bash[18489]: cephadm 2024-02-12T15:50:12.049373+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Saving service mgr spec with placement smithi121=y;smithi123=x;count:2 2024-02-12T15:50:14.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:13 smithi121 bash[18489]: cephadm 2024-02-12T15:50:12.105233+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Deploying daemon mgr.x on smithi123 2024-02-12T15:50:14.408 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-12T15:50:14.408 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-12T15:50:14.408 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 360 Links: 1 2024-02-12T15:50:14.408 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-12T15:50:14.408 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-02-12 15:43:57.299075384 +0000 2024-02-12T15:50:14.408 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-02-12 15:43:57.119079669 +0000 2024-02-12T15:50:14.409 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-02-12 15:43:57.119079669 +0000 2024-02-12T15:50:14.409 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2024-02-12T15:50:14.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-12T15:50:14.462 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-02-12T15:50:14.462 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-02-12T15:50:14.462 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000322291 s, 1.6 MB/s 2024-02-12T15:50:14.463 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-12T15:50:14.513 INFO:tasks.cephadm:Deploying osd.0 on smithi121 with /dev/vg_nvme/lv_4... 2024-02-12T15:50:14.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-12T15:50:14.888 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:14 smithi123 systemd[1]: Started Ceph mgr.x for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T15:50:15.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:14 smithi169 bash[19604]: cluster 2024-02-12T15:50:13.294983+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:15.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:14 smithi169 bash[19604]: audit 2024-02-12T15:50:14.571325+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:15.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:14 smithi169 bash[19604]: audit 2024-02-12T15:50:14.575179+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:14 smithi123 bash[19119]: cluster 2024-02-12T15:50:13.294983+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:14 smithi123 bash[19119]: audit 2024-02-12T15:50:14.571325+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:14 smithi123 bash[19119]: audit 2024-02-12T15:50:14.575179+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:15.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:14 smithi121 bash[18489]: cluster 2024-02-12T15:50:13.294983+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:15.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:14 smithi121 bash[18489]: audit 2024-02-12T15:50:14.571325+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:15.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:14 smithi121 bash[18489]: audit 2024-02-12T15:50:14.575179+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:16.138 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:15 smithi123 bash[19706]: debug 2024-02-12T15:50:15.822+0000 7ff954c70680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-12T15:50:16.138 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:15 smithi123 bash[19706]: debug 2024-02-12T15:50:15.922+0000 7ff954c70680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-12T15:50:16.638 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:16 smithi123 bash[19706]: debug 2024-02-12T15:50:16.346+0000 7ff954c70680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-12T15:50:17.241 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:16 smithi123 bash[19119]: cluster 2024-02-12T15:50:15.295459+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:17.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:16 smithi121 bash[18489]: cluster 2024-02-12T15:50:15.295459+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:17.295 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:16 smithi169 bash[19604]: cluster 2024-02-12T15:50:15.295459+0000 mgr.y (mgr.14146) 83 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:17.572 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:17 smithi123 bash[19706]: debug 2024-02-12T15:50:17.238+0000 7ff954c70680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-12T15:50:17.572 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:17 smithi123 bash[19706]: debug 2024-02-12T15:50:17.326+0000 7ff954c70680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-12T15:50:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-12T15:50:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-12T15:50:17.750 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-12T15:50:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-12T15:50:17.750 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0183603 s, 571 MB/s 2024-02-12T15:50:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-12T15:50:17.888 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:17 smithi123 bash[19706]: debug 2024-02-12T15:50:17.566+0000 7ff954c70680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-12T15:50:18.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2024-02-12T15:50:18.918 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:18 smithi123 bash[19706]: debug 2024-02-12T15:50:18.658+0000 7ff954c70680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-12T15:50:18.918 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:18 smithi123 bash[19706]: debug 2024-02-12T15:50:18.818+0000 7ff954c70680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:50:19.241 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:18 smithi123 bash[19119]: cluster 2024-02-12T15:50:17.295791+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:19.241 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:18 smithi123 bash[19119]: audit 2024-02-12T15:50:18.459308+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:19.241 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:18 smithi123 bash[19706]: debug 2024-02-12T15:50:18.914+0000 7ff954c70680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-12T15:50:19.241 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:19 smithi123 bash[19706]: debug 2024-02-12T15:50:19.022+0000 7ff954c70680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-12T15:50:19.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:18 smithi169 bash[19604]: cluster 2024-02-12T15:50:17.295791+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:19.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:18 smithi169 bash[19604]: audit 2024-02-12T15:50:18.459308+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:19.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:18 smithi121 bash[18489]: cluster 2024-02-12T15:50:17.295791+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:19.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:18 smithi121 bash[18489]: audit 2024-02-12T15:50:18.459308+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:19.630 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:19 smithi123 bash[19706]: debug 2024-02-12T15:50:19.234+0000 7ff954c70680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-12T15:50:19.888 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:19 smithi123 bash[19706]: debug 2024-02-12T15:50:19.626+0000 7ff954c70680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-12T15:50:19.888 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:19 smithi123 bash[19706]: debug 2024-02-12T15:50:19.750+0000 7ff954c70680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-12T15:50:20.888 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19706]: debug 2024-02-12T15:50:20.622+0000 7ff954c70680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-12T15:50:20.888 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19706]: debug 2024-02-12T15:50:20.722+0000 7ff954c70680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-12T15:50:20.888 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19706]: debug 2024-02-12T15:50:20.834+0000 7ff954c70680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-12T15:50:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: cluster 2024-02-12T15:50:19.296204+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: audit 2024-02-12T15:50:19.656760+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: audit 2024-02-12T15:50:19.923386+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: audit 2024-02-12T15:50:19.925051+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: audit 2024-02-12T15:50:19.933599+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: cephadm 2024-02-12T15:50:19.936870+0000 mgr.y (mgr.14146) 86 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: audit 2024-02-12T15:50:19.940038+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: audit 2024-02-12T15:50:19.941957+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: audit 2024-02-12T15:50:19.943557+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:20 smithi123 bash[19119]: cephadm 2024-02-12T15:50:19.944877+0000 mgr.y (mgr.14146) 87 : cephadm [INF] Reconfiguring daemon mgr.y on smithi121 2024-02-12T15:50:20.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: cluster 2024-02-12T15:50:19.296204+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:20.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: audit 2024-02-12T15:50:19.656760+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:20.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: audit 2024-02-12T15:50:19.923386+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:20.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: audit 2024-02-12T15:50:19.925051+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:20.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: audit 2024-02-12T15:50:19.933599+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:20.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: cephadm 2024-02-12T15:50:19.936870+0000 mgr.y (mgr.14146) 86 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-12T15:50:20.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: audit 2024-02-12T15:50:19.940038+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T15:50:20.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: audit 2024-02-12T15:50:19.941957+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T15:50:20.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: audit 2024-02-12T15:50:19.943557+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:20.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:20 smithi121 bash[18489]: cephadm 2024-02-12T15:50:19.944877+0000 mgr.y (mgr.14146) 87 : cephadm [INF] Reconfiguring daemon mgr.y on smithi121 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: cluster 2024-02-12T15:50:19.296204+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: audit 2024-02-12T15:50:19.656760+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: audit 2024-02-12T15:50:19.923386+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: audit 2024-02-12T15:50:19.925051+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: audit 2024-02-12T15:50:19.933599+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: cephadm 2024-02-12T15:50:19.936870+0000 mgr.y (mgr.14146) 86 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: audit 2024-02-12T15:50:19.940038+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-12T15:50:21.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: audit 2024-02-12T15:50:19.941957+0000 mon.a (mon.0) 225 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-12T15:50:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: audit 2024-02-12T15:50:19.943557+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:20 smithi169 bash[19604]: cephadm 2024-02-12T15:50:19.944877+0000 mgr.y (mgr.14146) 87 : cephadm [INF] Reconfiguring daemon mgr.y on smithi121 2024-02-12T15:50:21.332 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19706]: debug 2024-02-12T15:50:21.046+0000 7ff954c70680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-12T15:50:21.638 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19706]: debug 2024-02-12T15:50:21.326+0000 7ff954c70680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-12T15:50:21.638 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19706]: debug 2024-02-12T15:50:21.486+0000 7ff954c70680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-12T15:50:21.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:21 smithi121 bash[18489]: audit 2024-02-12T15:50:21.632649+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:50:21.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:21 smithi121 bash[18489]: audit 2024-02-12T15:50:21.638681+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:50:21.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:21 smithi121 bash[18489]: audit 2024-02-12T15:50:21.639884+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:21.968 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19706]: debug 2024-02-12T15:50:21.650+0000 7ff954c70680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-12T15:50:21.968 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19706]: debug 2024-02-12T15:50:21.862+0000 7ff954c70680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-12T15:50:21.968 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19706]: debug 2024-02-12T15:50:21.962+0000 7ff954c70680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-12T15:50:21.968 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19119]: audit 2024-02-12T15:50:21.632649+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:50:21.969 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19119]: audit 2024-02-12T15:50:21.638681+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:50:21.969 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:21 smithi123 bash[19119]: audit 2024-02-12T15:50:21.639884+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:22.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:21 smithi169 bash[19604]: audit 2024-02-12T15:50:21.632649+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:50:22.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:21 smithi169 bash[19604]: audit 2024-02-12T15:50:21.638681+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:50:22.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:21 smithi169 bash[19604]: audit 2024-02-12T15:50:21.639884+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:23.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: cluster 2024-02-12T15:50:21.296576+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:23.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.629546+0000 mgr.y (mgr.14146) 89 : audit [DBG] from='client.14214 v1:172.21.15.121:0/1073601887' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:50:23.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.981426+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:23.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.983265+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.985509+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.986632+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: cluster 2024-02-12T15:50:21.987081+0000 mon.a (mon.0) 234 : cluster [DBG] Standby manager daemon x started 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.988590+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.989135+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.990570+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.991240+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T15:50:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:22 smithi169 bash[19604]: audit 2024-02-12T15:50:21.993903+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: cluster 2024-02-12T15:50:21.296576+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.629546+0000 mgr.y (mgr.14146) 89 : audit [DBG] from='client.14214 v1:172.21.15.121:0/1073601887' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.981426+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.983265+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.985509+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.986632+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: cluster 2024-02-12T15:50:21.987081+0000 mon.a (mon.0) 234 : cluster [DBG] Standby manager daemon x started 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.988590+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.989135+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.990570+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.991240+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T15:50:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:22 smithi123 bash[19119]: audit 2024-02-12T15:50:21.993903+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:23.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: cluster 2024-02-12T15:50:21.296576+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:23.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.629546+0000 mgr.y (mgr.14146) 89 : audit [DBG] from='client.14214 v1:172.21.15.121:0/1073601887' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:50:23.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.981426+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:23.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.983265+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:23.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.985509+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:23.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.986632+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:23.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: cluster 2024-02-12T15:50:21.987081+0000 mon.a (mon.0) 234 : cluster [DBG] Standby manager daemon x started 2024-02-12T15:50:23.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.988590+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-12T15:50:23.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.989135+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-12T15:50:23.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.990570+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-12T15:50:23.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.991240+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? v1:172.21.15.123:0/2861118376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-12T15:50:23.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:22 smithi121 bash[18489]: audit 2024-02-12T15:50:21.993903+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:24.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:23 smithi169 bash[19604]: cluster 2024-02-12T15:50:22.991368+0000 mon.a (mon.0) 236 : cluster [DBG] mgrmap e14: y(active, since 2m), standbys: x 2024-02-12T15:50:24.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:23 smithi169 bash[19604]: audit 2024-02-12T15:50:22.991664+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T15:50:24.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:23 smithi169 bash[19604]: cluster 2024-02-12T15:50:23.296967+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:24.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:23 smithi123 bash[19119]: cluster 2024-02-12T15:50:22.991368+0000 mon.a (mon.0) 236 : cluster [DBG] mgrmap e14: y(active, since 2m), standbys: x 2024-02-12T15:50:24.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:23 smithi123 bash[19119]: audit 2024-02-12T15:50:22.991664+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T15:50:24.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:23 smithi123 bash[19119]: cluster 2024-02-12T15:50:23.296967+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:24.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:23 smithi121 bash[18489]: cluster 2024-02-12T15:50:22.991368+0000 mon.a (mon.0) 236 : cluster [DBG] mgrmap e14: y(active, since 2m), standbys: x 2024-02-12T15:50:24.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:23 smithi121 bash[18489]: audit 2024-02-12T15:50:22.991664+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-12T15:50:24.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:23 smithi121 bash[18489]: cluster 2024-02-12T15:50:23.296967+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:27.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:26 smithi121 bash[18489]: cluster 2024-02-12T15:50:25.297412+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:27.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:26 smithi169 bash[19604]: cluster 2024-02-12T15:50:25.297412+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:26 smithi123 bash[19119]: cluster 2024-02-12T15:50:25.297412+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:29.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:28 smithi121 bash[18489]: cluster 2024-02-12T15:50:27.297939+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:29.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:28 smithi169 bash[19604]: cluster 2024-02-12T15:50:27.297939+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:29.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:28 smithi123 bash[19119]: cluster 2024-02-12T15:50:27.297939+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:30.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:29 smithi169 bash[19604]: audit 2024-02-12T15:50:29.645930+0000 mon.a (mon.0) 238 : audit [INF] from='client.? v1:172.21.15.121:0/1068215495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0"}]: dispatch 2024-02-12T15:50:30.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:29 smithi169 bash[19604]: audit 2024-02-12T15:50:29.653412+0000 mon.a (mon.0) 239 : audit [INF] from='client.? v1:172.21.15.121:0/1068215495' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0"}]': finished 2024-02-12T15:50:30.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:29 smithi169 bash[19604]: cluster 2024-02-12T15:50:29.653514+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-12T15:50:30.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:29 smithi169 bash[19604]: audit 2024-02-12T15:50:29.653698+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:30.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:29 smithi123 bash[19119]: audit 2024-02-12T15:50:29.645930+0000 mon.a (mon.0) 238 : audit [INF] from='client.? v1:172.21.15.121:0/1068215495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0"}]: dispatch 2024-02-12T15:50:30.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:29 smithi123 bash[19119]: audit 2024-02-12T15:50:29.653412+0000 mon.a (mon.0) 239 : audit [INF] from='client.? v1:172.21.15.121:0/1068215495' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0"}]': finished 2024-02-12T15:50:30.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:29 smithi123 bash[19119]: cluster 2024-02-12T15:50:29.653514+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-12T15:50:30.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:29 smithi123 bash[19119]: audit 2024-02-12T15:50:29.653698+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:30.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:29 smithi121 bash[18489]: audit 2024-02-12T15:50:29.645930+0000 mon.a (mon.0) 238 : audit [INF] from='client.? v1:172.21.15.121:0/1068215495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0"}]: dispatch 2024-02-12T15:50:30.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:29 smithi121 bash[18489]: audit 2024-02-12T15:50:29.653412+0000 mon.a (mon.0) 239 : audit [INF] from='client.? v1:172.21.15.121:0/1068215495' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0"}]': finished 2024-02-12T15:50:30.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:29 smithi121 bash[18489]: cluster 2024-02-12T15:50:29.653514+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-12T15:50:30.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:29 smithi121 bash[18489]: audit 2024-02-12T15:50:29.653698+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:31.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:30 smithi169 bash[19604]: cluster 2024-02-12T15:50:29.298415+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:31.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:30 smithi169 bash[19604]: audit 2024-02-12T15:50:30.255306+0000 mon.a (mon.0) 242 : audit [DBG] from='client.? v1:172.21.15.121:0/954761252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:50:31.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:30 smithi123 bash[19119]: cluster 2024-02-12T15:50:29.298415+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:31.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:30 smithi123 bash[19119]: audit 2024-02-12T15:50:30.255306+0000 mon.a (mon.0) 242 : audit [DBG] from='client.? v1:172.21.15.121:0/954761252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:50:31.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:30 smithi121 bash[18489]: cluster 2024-02-12T15:50:29.298415+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:31.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:30 smithi121 bash[18489]: audit 2024-02-12T15:50:30.255306+0000 mon.a (mon.0) 242 : audit [DBG] from='client.? v1:172.21.15.121:0/954761252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:50:33.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:32 smithi169 bash[19604]: cluster 2024-02-12T15:50:31.298928+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:33.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:32 smithi123 bash[19119]: cluster 2024-02-12T15:50:31.298928+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:33.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:32 smithi121 bash[18489]: cluster 2024-02-12T15:50:31.298928+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:35.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:34 smithi169 bash[19604]: cluster 2024-02-12T15:50:33.299464+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:35.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:34 smithi123 bash[19119]: cluster 2024-02-12T15:50:33.299464+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:35.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:34 smithi121 bash[18489]: cluster 2024-02-12T15:50:33.299464+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:37.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:36 smithi169 bash[19604]: cluster 2024-02-12T15:50:35.300032+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:36 smithi123 bash[19119]: cluster 2024-02-12T15:50:35.300032+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:37.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:36 smithi121 bash[18489]: cluster 2024-02-12T15:50:35.300032+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:39.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:38 smithi121 bash[18489]: cluster 2024-02-12T15:50:37.300488+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:39.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:38 smithi169 bash[19604]: cluster 2024-02-12T15:50:37.300488+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:39.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:38 smithi123 bash[19119]: cluster 2024-02-12T15:50:37.300488+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:41.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:40 smithi121 bash[18489]: cluster 2024-02-12T15:50:39.301011+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:41.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:40 smithi169 bash[19604]: cluster 2024-02-12T15:50:39.301011+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:40 smithi123 bash[19119]: cluster 2024-02-12T15:50:39.301011+0000 mgr.y (mgr.14146) 98 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:42.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:41 smithi169 bash[19604]: audit 2024-02-12T15:50:41.393711+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-12T15:50:42.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:41 smithi169 bash[19604]: audit 2024-02-12T15:50:41.395226+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:42.373 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:41 smithi121 bash[18489]: audit 2024-02-12T15:50:41.393711+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-12T15:50:42.373 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:41 smithi121 bash[18489]: audit 2024-02-12T15:50:41.395226+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:41 smithi123 bash[19119]: audit 2024-02-12T15:50:41.393711+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-12T15:50:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:41 smithi123 bash[19119]: audit 2024-02-12T15:50:41.395226+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:43.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:42 smithi169 bash[19604]: cluster 2024-02-12T15:50:41.301570+0000 mgr.y (mgr.14146) 99 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:43.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:42 smithi169 bash[19604]: cephadm 2024-02-12T15:50:41.396794+0000 mgr.y (mgr.14146) 100 : cephadm [INF] Deploying daemon osd.0 on smithi121 2024-02-12T15:50:43.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:42 smithi123 bash[19119]: cluster 2024-02-12T15:50:41.301570+0000 mgr.y (mgr.14146) 99 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:43.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:42 smithi123 bash[19119]: cephadm 2024-02-12T15:50:41.396794+0000 mgr.y (mgr.14146) 100 : cephadm [INF] Deploying daemon osd.0 on smithi121 2024-02-12T15:50:43.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:42 smithi121 bash[18489]: cluster 2024-02-12T15:50:41.301570+0000 mgr.y (mgr.14146) 99 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:43.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:42 smithi121 bash[18489]: cephadm 2024-02-12T15:50:41.396794+0000 mgr.y (mgr.14146) 100 : cephadm [INF] Deploying daemon osd.0 on smithi121 2024-02-12T15:50:45.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:44 smithi121 bash[18489]: cluster 2024-02-12T15:50:43.302058+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:45.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:44 smithi169 bash[19604]: cluster 2024-02-12T15:50:43.302058+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:44 smithi123 bash[19119]: cluster 2024-02-12T15:50:43.302058+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:46.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:46 smithi169 bash[19604]: audit 2024-02-12T15:50:45.250879+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:46.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:46 smithi169 bash[19604]: audit 2024-02-12T15:50:45.268439+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:46.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:46 smithi169 bash[19604]: cluster 2024-02-12T15:50:45.302428+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:46 smithi123 bash[19119]: audit 2024-02-12T15:50:45.250879+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:46 smithi123 bash[19119]: audit 2024-02-12T15:50:45.268439+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:46 smithi123 bash[19119]: cluster 2024-02-12T15:50:45.302428+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:46.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:46 smithi121 bash[18489]: audit 2024-02-12T15:50:45.250879+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:46.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:46 smithi121 bash[18489]: audit 2024-02-12T15:50:45.268439+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:46.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:46 smithi121 bash[18489]: cluster 2024-02-12T15:50:45.302428+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:48 smithi123 bash[19119]: cluster 2024-02-12T15:50:47.302849+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:48.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:48 smithi121 bash[18489]: cluster 2024-02-12T15:50:47.302849+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:48.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:48 smithi169 bash[19604]: cluster 2024-02-12T15:50:47.302849+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:51.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:50 smithi121 bash[18489]: cluster 2024-02-12T15:50:49.303388+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:51.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:50 smithi121 bash[18489]: audit 2024-02-12T15:50:50.564470+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:51.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:50 smithi121 bash[18489]: audit 2024-02-12T15:50:50.835898+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:51.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:50 smithi121 bash[18489]: audit 2024-02-12T15:50:50.837221+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:51.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:50 smithi121 bash[18489]: audit 2024-02-12T15:50:50.843494+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:50 smithi169 bash[19604]: cluster 2024-02-12T15:50:49.303388+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:50 smithi169 bash[19604]: audit 2024-02-12T15:50:50.564470+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:50 smithi169 bash[19604]: audit 2024-02-12T15:50:50.835898+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:50 smithi169 bash[19604]: audit 2024-02-12T15:50:50.837221+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:50 smithi169 bash[19604]: audit 2024-02-12T15:50:50.843494+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:50 smithi123 bash[19119]: cluster 2024-02-12T15:50:49.303388+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:50 smithi123 bash[19119]: audit 2024-02-12T15:50:50.564470+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:50 smithi123 bash[19119]: audit 2024-02-12T15:50:50.835898+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:50:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:50 smithi123 bash[19119]: audit 2024-02-12T15:50:50.837221+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:50:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:50 smithi123 bash[19119]: audit 2024-02-12T15:50:50.843494+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:52.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:51 smithi169 bash[19604]: cluster 2024-02-12T15:50:51.303745+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:52.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:51 smithi169 bash[19604]: audit 2024-02-12T15:50:51.922042+0000 mon.a (mon.0) 251 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T15:50:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:51 smithi123 bash[19119]: cluster 2024-02-12T15:50:51.303745+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:51 smithi123 bash[19119]: audit 2024-02-12T15:50:51.922042+0000 mon.a (mon.0) 251 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T15:50:52.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:51 smithi121 bash[18489]: cluster 2024-02-12T15:50:51.303745+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:52.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:51 smithi121 bash[18489]: audit 2024-02-12T15:50:51.922042+0000 mon.a (mon.0) 251 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-12T15:50:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:52 smithi169 bash[19604]: audit 2024-02-12T15:50:51.978906+0000 mon.a (mon.0) 252 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-12T15:50:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:52 smithi169 bash[19604]: cluster 2024-02-12T15:50:51.979124+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-12T15:50:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:52 smithi169 bash[19604]: audit 2024-02-12T15:50:51.979882+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:52 smithi169 bash[19604]: audit 2024-02-12T15:50:51.980834+0000 mon.a (mon.0) 255 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:50:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:52 smithi169 bash[19604]: audit 2024-02-12T15:50:52.980794+0000 mon.a (mon.0) 256 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:50:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:52 smithi169 bash[19604]: cluster 2024-02-12T15:50:52.980918+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-12T15:50:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:52 smithi169 bash[19604]: audit 2024-02-12T15:50:52.981201+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:52 smithi123 bash[19119]: audit 2024-02-12T15:50:51.978906+0000 mon.a (mon.0) 252 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-12T15:50:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:52 smithi123 bash[19119]: cluster 2024-02-12T15:50:51.979124+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-12T15:50:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:52 smithi123 bash[19119]: audit 2024-02-12T15:50:51.979882+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:52 smithi123 bash[19119]: audit 2024-02-12T15:50:51.980834+0000 mon.a (mon.0) 255 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:50:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:52 smithi123 bash[19119]: audit 2024-02-12T15:50:52.980794+0000 mon.a (mon.0) 256 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:50:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:52 smithi123 bash[19119]: cluster 2024-02-12T15:50:52.980918+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-12T15:50:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:52 smithi123 bash[19119]: audit 2024-02-12T15:50:52.981201+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:53.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:52 smithi121 bash[18489]: audit 2024-02-12T15:50:51.978906+0000 mon.a (mon.0) 252 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-12T15:50:53.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:52 smithi121 bash[18489]: cluster 2024-02-12T15:50:51.979124+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-12T15:50:53.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:52 smithi121 bash[18489]: audit 2024-02-12T15:50:51.979882+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:53.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:52 smithi121 bash[18489]: audit 2024-02-12T15:50:51.980834+0000 mon.a (mon.0) 255 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:50:53.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:52 smithi121 bash[18489]: audit 2024-02-12T15:50:52.980794+0000 mon.a (mon.0) 256 : audit [INF] from='osd.0 v1:172.21.15.121:6801/1815232143' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:50:53.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:52 smithi121 bash[18489]: cluster 2024-02-12T15:50:52.980918+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-12T15:50:53.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:52 smithi121 bash[18489]: audit 2024-02-12T15:50:52.981201+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:53.744 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 0 on host 'smithi121' 2024-02-12T15:50:54.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:53 smithi121 bash[18489]: audit 2024-02-12T15:50:52.991449+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:54.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:53 smithi121 bash[18489]: cluster 2024-02-12T15:50:53.304126+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:54.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:53 smithi121 bash[18489]: audit 2024-02-12T15:50:53.742242+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:54.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:53 smithi121 bash[18489]: audit 2024-02-12T15:50:53.768752+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:54.270 DEBUG:teuthology.orchestra.run.smithi121:osd.0> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.0.service 2024-02-12T15:50:54.272 INFO:tasks.cephadm:Deploying osd.1 on smithi121 with /dev/vg_nvme/lv_3... 2024-02-12T15:50:54.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-12T15:50:54.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:53 smithi169 bash[19604]: audit 2024-02-12T15:50:52.991449+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:54.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:53 smithi169 bash[19604]: cluster 2024-02-12T15:50:53.304126+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:54.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:53 smithi169 bash[19604]: audit 2024-02-12T15:50:53.742242+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:54.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:53 smithi169 bash[19604]: audit 2024-02-12T15:50:53.768752+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:53 smithi123 bash[19119]: audit 2024-02-12T15:50:52.991449+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:53 smithi123 bash[19119]: cluster 2024-02-12T15:50:53.304126+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-12T15:50:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:53 smithi123 bash[19119]: audit 2024-02-12T15:50:53.742242+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:50:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:53 smithi123 bash[19119]: audit 2024-02-12T15:50:53.768752+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:50:54.413 INFO:journalctl@ceph.osd.0.smithi121.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:50:55.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:54 smithi169 bash[19604]: cluster 2024-02-12T15:50:52.941697+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:50:55.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:54 smithi169 bash[19604]: cluster 2024-02-12T15:50:52.941828+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:50:55.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:54 smithi169 bash[19604]: audit 2024-02-12T15:50:53.987248+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:55.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:54 smithi169 bash[19604]: cluster 2024-02-12T15:50:53.997332+0000 mon.a (mon.0) 263 : cluster [INF] osd.0 v1:172.21.15.121:6801/1815232143 boot 2024-02-12T15:50:55.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:54 smithi169 bash[19604]: cluster 2024-02-12T15:50:53.997374+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-12T15:50:55.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:54 smithi169 bash[19604]: audit 2024-02-12T15:50:53.997548+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:55.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:54 smithi121 bash[18489]: cluster 2024-02-12T15:50:52.941697+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:50:55.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:54 smithi121 bash[18489]: cluster 2024-02-12T15:50:52.941828+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:50:55.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:54 smithi121 bash[18489]: audit 2024-02-12T15:50:53.987248+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:55.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:54 smithi121 bash[18489]: cluster 2024-02-12T15:50:53.997332+0000 mon.a (mon.0) 263 : cluster [INF] osd.0 v1:172.21.15.121:6801/1815232143 boot 2024-02-12T15:50:55.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:54 smithi121 bash[18489]: cluster 2024-02-12T15:50:53.997374+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-12T15:50:55.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:54 smithi121 bash[18489]: audit 2024-02-12T15:50:53.997548+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:54 smithi123 bash[19119]: cluster 2024-02-12T15:50:52.941697+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:50:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:54 smithi123 bash[19119]: cluster 2024-02-12T15:50:52.941828+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:50:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:54 smithi123 bash[19119]: audit 2024-02-12T15:50:53.987248+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:54 smithi123 bash[19119]: cluster 2024-02-12T15:50:53.997332+0000 mon.a (mon.0) 263 : cluster [INF] osd.0 v1:172.21.15.121:6801/1815232143 boot 2024-02-12T15:50:55.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:54 smithi123 bash[19119]: cluster 2024-02-12T15:50:53.997374+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-12T15:50:55.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:54 smithi123 bash[19119]: audit 2024-02-12T15:50:53.997548+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-12T15:50:56.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:56 smithi169 bash[19604]: cluster 2024-02-12T15:50:55.002925+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-12T15:50:56.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:56 smithi169 bash[19604]: cluster 2024-02-12T15:50:55.304673+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:50:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:56 smithi123 bash[19119]: cluster 2024-02-12T15:50:55.002925+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-12T15:50:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:56 smithi123 bash[19119]: cluster 2024-02-12T15:50:55.304673+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:50:56.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:56 smithi121 bash[18489]: cluster 2024-02-12T15:50:55.002925+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-12T15:50:56.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:56 smithi121 bash[18489]: cluster 2024-02-12T15:50:55.304673+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:50:57.599 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-12T15:50:57.599 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-12T15:50:57.599 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-12T15:50:57.599 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-12T15:50:57.599 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138374 s, 758 MB/s 2024-02-12T15:50:57.600 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-12T15:50:58.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2024-02-12T15:50:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:50:58 smithi123 bash[19119]: cluster 2024-02-12T15:50:57.305167+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:50:58.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:50:58 smithi121 bash[18489]: cluster 2024-02-12T15:50:57.305167+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:50:58.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:50:58 smithi169 bash[19604]: cluster 2024-02-12T15:50:57.305167+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:01.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:00 smithi169 bash[19604]: cluster 2024-02-12T15:50:59.305693+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:00 smithi123 bash[19119]: cluster 2024-02-12T15:50:59.305693+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:01.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:00 smithi121 bash[18489]: cluster 2024-02-12T15:50:59.305693+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:03.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:02 smithi121 bash[18489]: cluster 2024-02-12T15:51:01.306200+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:03.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:02 smithi121 bash[18489]: audit 2024-02-12T15:51:02.387485+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:51:03.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:02 smithi121 bash[18489]: audit 2024-02-12T15:51:02.390016+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:51:03.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:02 smithi121 bash[18489]: audit 2024-02-12T15:51:02.390649+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:03.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:02 smithi169 bash[19604]: cluster 2024-02-12T15:51:01.306200+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:03.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:02 smithi169 bash[19604]: audit 2024-02-12T15:51:02.387485+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:51:03.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:02 smithi169 bash[19604]: audit 2024-02-12T15:51:02.390016+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:51:03.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:02 smithi169 bash[19604]: audit 2024-02-12T15:51:02.390649+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:02 smithi123 bash[19119]: cluster 2024-02-12T15:51:01.306200+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:02 smithi123 bash[19119]: audit 2024-02-12T15:51:02.387485+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:51:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:02 smithi123 bash[19119]: audit 2024-02-12T15:51:02.390016+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:51:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:02 smithi123 bash[19119]: audit 2024-02-12T15:51:02.390649+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:04.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:03 smithi169 bash[19604]: audit 2024-02-12T15:51:02.383081+0000 mgr.y (mgr.14146) 111 : audit [DBG] from='client.14241 v1:172.21.15.121:0/2556710264' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:51:04.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:03 smithi169 bash[19604]: audit 2024-02-12T15:51:03.474084+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:03 smithi169 bash[19604]: audit 2024-02-12T15:51:03.475690+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:04.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:03 smithi169 bash[19604]: audit 2024-02-12T15:51:03.484128+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:03 smithi169 bash[19604]: audit 2024-02-12T15:51:03.485677+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:04.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:03 smithi169 bash[19604]: audit 2024-02-12T15:51:03.487110+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:03 smithi169 bash[19604]: audit 2024-02-12T15:51:03.495330+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:03 smithi123 bash[19119]: audit 2024-02-12T15:51:02.383081+0000 mgr.y (mgr.14146) 111 : audit [DBG] from='client.14241 v1:172.21.15.121:0/2556710264' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:51:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:03 smithi123 bash[19119]: audit 2024-02-12T15:51:03.474084+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:03 smithi123 bash[19119]: audit 2024-02-12T15:51:03.475690+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:03 smithi123 bash[19119]: audit 2024-02-12T15:51:03.484128+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:03 smithi123 bash[19119]: audit 2024-02-12T15:51:03.485677+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:03 smithi123 bash[19119]: audit 2024-02-12T15:51:03.487110+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:03 smithi123 bash[19119]: audit 2024-02-12T15:51:03.495330+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:03 smithi121 bash[18489]: audit 2024-02-12T15:51:02.383081+0000 mgr.y (mgr.14146) 111 : audit [DBG] from='client.14241 v1:172.21.15.121:0/2556710264' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:51:04.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:03 smithi121 bash[18489]: audit 2024-02-12T15:51:03.474084+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:03 smithi121 bash[18489]: audit 2024-02-12T15:51:03.475690+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:04.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:03 smithi121 bash[18489]: audit 2024-02-12T15:51:03.484128+0000 mon.a (mon.0) 272 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:04.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:03 smithi121 bash[18489]: audit 2024-02-12T15:51:03.485677+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:04.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:03 smithi121 bash[18489]: audit 2024-02-12T15:51:03.487110+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:04.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:03 smithi121 bash[18489]: audit 2024-02-12T15:51:03.495330+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:05.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:04 smithi169 bash[19604]: cluster 2024-02-12T15:51:03.306701+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:05.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:04 smithi169 bash[19604]: cephadm 2024-02-12T15:51:03.467817+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:51:05.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:04 smithi169 bash[19604]: cephadm 2024-02-12T15:51:03.477037+0000 mgr.y (mgr.14146) 114 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 17317M 2024-02-12T15:51:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:04 smithi123 bash[19119]: cluster 2024-02-12T15:51:03.306701+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:04 smithi123 bash[19119]: cephadm 2024-02-12T15:51:03.467817+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:51:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:04 smithi123 bash[19119]: cephadm 2024-02-12T15:51:03.477037+0000 mgr.y (mgr.14146) 114 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 17317M 2024-02-12T15:51:05.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:04 smithi121 bash[18489]: cluster 2024-02-12T15:51:03.306701+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:05.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:04 smithi121 bash[18489]: cephadm 2024-02-12T15:51:03.467817+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:51:05.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:04 smithi121 bash[18489]: cephadm 2024-02-12T15:51:03.477037+0000 mgr.y (mgr.14146) 114 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 17317M 2024-02-12T15:51:07.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:06 smithi169 bash[19604]: cluster 2024-02-12T15:51:05.307312+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:06 smithi123 bash[19119]: cluster 2024-02-12T15:51:05.307312+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:07.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:06 smithi121 bash[18489]: cluster 2024-02-12T15:51:05.307312+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:09.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:08 smithi169 bash[19604]: cluster 2024-02-12T15:51:07.307828+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:08 smithi123 bash[19119]: cluster 2024-02-12T15:51:07.307828+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:09.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:08 smithi121 bash[18489]: cluster 2024-02-12T15:51:07.307828+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:11.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:10 smithi169 bash[19604]: cluster 2024-02-12T15:51:09.308314+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:11.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:10 smithi169 bash[19604]: audit 2024-02-12T15:51:10.438966+0000 mon.a (mon.0) 276 : audit [INF] from='client.? v1:172.21.15.121:0/3400073384' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6140309-5703-4538-9739-276fbbdaa880"}]: dispatch 2024-02-12T15:51:11.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:10 smithi169 bash[19604]: audit 2024-02-12T15:51:10.445331+0000 mon.a (mon.0) 277 : audit [INF] from='client.? v1:172.21.15.121:0/3400073384' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6140309-5703-4538-9739-276fbbdaa880"}]': finished 2024-02-12T15:51:11.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:10 smithi169 bash[19604]: cluster 2024-02-12T15:51:10.445376+0000 mon.a (mon.0) 278 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-12T15:51:11.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:10 smithi169 bash[19604]: audit 2024-02-12T15:51:10.445550+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:10 smithi123 bash[19119]: cluster 2024-02-12T15:51:09.308314+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:10 smithi123 bash[19119]: audit 2024-02-12T15:51:10.438966+0000 mon.a (mon.0) 276 : audit [INF] from='client.? v1:172.21.15.121:0/3400073384' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6140309-5703-4538-9739-276fbbdaa880"}]: dispatch 2024-02-12T15:51:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:10 smithi123 bash[19119]: audit 2024-02-12T15:51:10.445331+0000 mon.a (mon.0) 277 : audit [INF] from='client.? v1:172.21.15.121:0/3400073384' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6140309-5703-4538-9739-276fbbdaa880"}]': finished 2024-02-12T15:51:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:10 smithi123 bash[19119]: cluster 2024-02-12T15:51:10.445376+0000 mon.a (mon.0) 278 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-12T15:51:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:10 smithi123 bash[19119]: audit 2024-02-12T15:51:10.445550+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:11.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:10 smithi121 bash[18489]: cluster 2024-02-12T15:51:09.308314+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:11.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:10 smithi121 bash[18489]: audit 2024-02-12T15:51:10.438966+0000 mon.a (mon.0) 276 : audit [INF] from='client.? v1:172.21.15.121:0/3400073384' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6140309-5703-4538-9739-276fbbdaa880"}]: dispatch 2024-02-12T15:51:11.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:10 smithi121 bash[18489]: audit 2024-02-12T15:51:10.445331+0000 mon.a (mon.0) 277 : audit [INF] from='client.? v1:172.21.15.121:0/3400073384' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6140309-5703-4538-9739-276fbbdaa880"}]': finished 2024-02-12T15:51:11.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:10 smithi121 bash[18489]: cluster 2024-02-12T15:51:10.445376+0000 mon.a (mon.0) 278 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-12T15:51:11.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:10 smithi121 bash[18489]: audit 2024-02-12T15:51:10.445550+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:12.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:11 smithi169 bash[19604]: audit 2024-02-12T15:51:11.024000+0000 mon.a (mon.0) 280 : audit [DBG] from='client.? v1:172.21.15.121:0/3879513912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:51:12.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:11 smithi123 bash[19119]: audit 2024-02-12T15:51:11.024000+0000 mon.a (mon.0) 280 : audit [DBG] from='client.? v1:172.21.15.121:0/3879513912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:51:12.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:11 smithi121 bash[18489]: audit 2024-02-12T15:51:11.024000+0000 mon.a (mon.0) 280 : audit [DBG] from='client.? v1:172.21.15.121:0/3879513912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:51:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:12 smithi169 bash[19604]: cluster 2024-02-12T15:51:11.308852+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:12 smithi123 bash[19119]: cluster 2024-02-12T15:51:11.308852+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:13.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:12 smithi121 bash[18489]: cluster 2024-02-12T15:51:11.308852+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:15.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:14 smithi169 bash[19604]: cluster 2024-02-12T15:51:13.309242+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:14 smithi123 bash[19119]: cluster 2024-02-12T15:51:13.309242+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:15.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:14 smithi121 bash[18489]: cluster 2024-02-12T15:51:13.309242+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:17.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:16 smithi169 bash[19604]: cluster 2024-02-12T15:51:15.309714+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:16 smithi123 bash[19119]: cluster 2024-02-12T15:51:15.309714+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:17.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:16 smithi121 bash[18489]: cluster 2024-02-12T15:51:15.309714+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:18.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:18 smithi169 bash[19604]: cluster 2024-02-12T15:51:17.310144+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:18 smithi123 bash[19119]: cluster 2024-02-12T15:51:17.310144+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:18.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:18 smithi121 bash[18489]: cluster 2024-02-12T15:51:17.310144+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:21.192 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:20 smithi121 bash[18489]: cluster 2024-02-12T15:51:19.310630+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:21.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:20 smithi169 bash[19604]: cluster 2024-02-12T15:51:19.310630+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:21.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:20 smithi123 bash[19119]: cluster 2024-02-12T15:51:19.310630+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:23.254 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:22 smithi121 bash[18489]: cluster 2024-02-12T15:51:21.311197+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:23.254 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:22 smithi121 bash[18489]: audit 2024-02-12T15:51:22.233051+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-12T15:51:23.254 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:22 smithi121 bash[18489]: audit 2024-02-12T15:51:22.234595+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:23.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:22 smithi169 bash[19604]: cluster 2024-02-12T15:51:21.311197+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:23.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:22 smithi169 bash[19604]: audit 2024-02-12T15:51:22.233051+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-12T15:51:23.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:22 smithi169 bash[19604]: audit 2024-02-12T15:51:22.234595+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:23.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:22 smithi123 bash[19119]: cluster 2024-02-12T15:51:21.311197+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:23.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:22 smithi123 bash[19119]: audit 2024-02-12T15:51:22.233051+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-12T15:51:23.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:22 smithi123 bash[19119]: audit 2024-02-12T15:51:22.234595+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:24.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:23 smithi169 bash[19604]: cephadm 2024-02-12T15:51:22.235954+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.1 on smithi121 2024-02-12T15:51:24.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:23 smithi123 bash[19119]: cephadm 2024-02-12T15:51:22.235954+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.1 on smithi121 2024-02-12T15:51:24.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:23 smithi121 bash[18489]: cephadm 2024-02-12T15:51:22.235954+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.1 on smithi121 2024-02-12T15:51:25.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:24 smithi121 bash[18489]: cluster 2024-02-12T15:51:23.311612+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:25.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:24 smithi169 bash[19604]: cluster 2024-02-12T15:51:23.311612+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:25.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:24 smithi123 bash[19119]: cluster 2024-02-12T15:51:23.311612+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:26 smithi169 bash[19604]: cluster 2024-02-12T15:51:25.312134+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:26 smithi169 bash[19604]: audit 2024-02-12T15:51:26.110949+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:26 smithi169 bash[19604]: audit 2024-02-12T15:51:26.119574+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:51:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:26 smithi123 bash[19119]: cluster 2024-02-12T15:51:25.312134+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:26 smithi123 bash[19119]: audit 2024-02-12T15:51:26.110949+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:26 smithi123 bash[19119]: audit 2024-02-12T15:51:26.119574+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:51:27.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:26 smithi121 bash[18489]: cluster 2024-02-12T15:51:25.312134+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:27.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:26 smithi121 bash[18489]: audit 2024-02-12T15:51:26.110949+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:27.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:26 smithi121 bash[18489]: audit 2024-02-12T15:51:26.119574+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:51:29.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:28 smithi121 bash[18489]: cluster 2024-02-12T15:51:27.312681+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:29.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:28 smithi169 bash[19604]: cluster 2024-02-12T15:51:27.312681+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:29.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:28 smithi123 bash[19119]: cluster 2024-02-12T15:51:27.312681+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:31.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:30 smithi169 bash[19604]: cluster 2024-02-12T15:51:29.313211+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:31.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:30 smithi123 bash[19119]: cluster 2024-02-12T15:51:29.313211+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:31.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:30 smithi121 bash[18489]: cluster 2024-02-12T15:51:29.313211+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:32.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:32 smithi169 bash[19604]: cluster 2024-02-12T15:51:31.313617+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:32.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:32 smithi169 bash[19604]: audit 2024-02-12T15:51:31.416432+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:32.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:32 smithi169 bash[19604]: audit 2024-02-12T15:51:31.417564+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:32.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:32 smithi169 bash[19604]: audit 2024-02-12T15:51:31.418923+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:32.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:32 smithi169 bash[19604]: audit 2024-02-12T15:51:31.428474+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:32 smithi123 bash[19119]: cluster 2024-02-12T15:51:31.313617+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:32 smithi123 bash[19119]: audit 2024-02-12T15:51:31.416432+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:32 smithi123 bash[19119]: audit 2024-02-12T15:51:31.417564+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:32 smithi123 bash[19119]: audit 2024-02-12T15:51:31.418923+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:32 smithi123 bash[19119]: audit 2024-02-12T15:51:31.428474+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:32.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:32 smithi121 bash[18489]: cluster 2024-02-12T15:51:31.313617+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:32.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:32 smithi121 bash[18489]: audit 2024-02-12T15:51:31.416432+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:32.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:32 smithi121 bash[18489]: audit 2024-02-12T15:51:31.417564+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:32.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:32 smithi121 bash[18489]: audit 2024-02-12T15:51:31.418923+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:32.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:32 smithi121 bash[18489]: audit 2024-02-12T15:51:31.428474+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:33.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:33 smithi169 bash[19604]: audit 2024-02-12T15:51:32.517070+0000 mon.a (mon.0) 289 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T15:51:33.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:33 smithi169 bash[19604]: audit 2024-02-12T15:51:32.517135+0000 mon.b (mon.2) 6 : audit [INF] from='osd.1 v1:172.21.15.121:6805/2968433670' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T15:51:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:33 smithi123 bash[19119]: audit 2024-02-12T15:51:32.517070+0000 mon.a (mon.0) 289 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T15:51:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:33 smithi123 bash[19119]: audit 2024-02-12T15:51:32.517135+0000 mon.b (mon.2) 6 : audit [INF] from='osd.1 v1:172.21.15.121:6805/2968433670' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T15:51:33.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:33 smithi121 bash[18489]: audit 2024-02-12T15:51:32.517070+0000 mon.a (mon.0) 289 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T15:51:33.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:33 smithi121 bash[18489]: audit 2024-02-12T15:51:32.517135+0000 mon.b (mon.2) 6 : audit [INF] from='osd.1 v1:172.21.15.121:6805/2968433670' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-12T15:51:34.609 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 1 on host 'smithi121' 2024-02-12T15:51:34.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:34 smithi169 bash[19604]: cluster 2024-02-12T15:51:33.314204+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:34.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:34 smithi169 bash[19604]: audit 2024-02-12T15:51:33.428860+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T15:51:34.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:34 smithi169 bash[19604]: cluster 2024-02-12T15:51:33.428931+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-12T15:51:34.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:34 smithi169 bash[19604]: audit 2024-02-12T15:51:33.429053+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:34.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:34 smithi169 bash[19604]: audit 2024-02-12T15:51:33.432181+0000 mon.b (mon.2) 7 : audit [INF] from='osd.1 v1:172.21.15.121:6805/2968433670' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:51:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:34 smithi169 bash[19604]: audit 2024-02-12T15:51:33.432401+0000 mon.a (mon.0) 293 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:51:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:34 smithi123 bash[19119]: cluster 2024-02-12T15:51:33.314204+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:34 smithi123 bash[19119]: audit 2024-02-12T15:51:33.428860+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T15:51:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:34 smithi123 bash[19119]: cluster 2024-02-12T15:51:33.428931+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-12T15:51:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:34 smithi123 bash[19119]: audit 2024-02-12T15:51:33.429053+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:34 smithi123 bash[19119]: audit 2024-02-12T15:51:33.432181+0000 mon.b (mon.2) 7 : audit [INF] from='osd.1 v1:172.21.15.121:6805/2968433670' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:51:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:34 smithi123 bash[19119]: audit 2024-02-12T15:51:33.432401+0000 mon.a (mon.0) 293 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:51:34.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:34 smithi121 bash[18489]: cluster 2024-02-12T15:51:33.314204+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:34.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:34 smithi121 bash[18489]: audit 2024-02-12T15:51:33.428860+0000 mon.a (mon.0) 290 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-12T15:51:34.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:34 smithi121 bash[18489]: cluster 2024-02-12T15:51:33.428931+0000 mon.a (mon.0) 291 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-12T15:51:34.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:34 smithi121 bash[18489]: audit 2024-02-12T15:51:33.429053+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:34.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:34 smithi121 bash[18489]: audit 2024-02-12T15:51:33.432181+0000 mon.b (mon.2) 7 : audit [INF] from='osd.1 v1:172.21.15.121:6805/2968433670' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:51:34.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:34 smithi121 bash[18489]: audit 2024-02-12T15:51:33.432401+0000 mon.a (mon.0) 293 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:51:35.236 DEBUG:teuthology.orchestra.run.smithi121:osd.1> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.1.service 2024-02-12T15:51:35.239 INFO:tasks.cephadm:Deploying osd.2 on smithi121 with /dev/vg_nvme/lv_2... 2024-02-12T15:51:35.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-12T15:51:35.410 INFO:journalctl@ceph.osd.1.smithi121.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:51:36.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:35 smithi121 bash[18489]: audit 2024-02-12T15:51:34.432865+0000 mon.a (mon.0) 294 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:51:36.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:35 smithi121 bash[18489]: cluster 2024-02-12T15:51:34.433050+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-12T15:51:36.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:35 smithi121 bash[18489]: audit 2024-02-12T15:51:34.433776+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:36.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:35 smithi121 bash[18489]: audit 2024-02-12T15:51:34.440780+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:36.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:35 smithi121 bash[18489]: audit 2024-02-12T15:51:34.606142+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:36.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:35 smithi121 bash[18489]: audit 2024-02-12T15:51:34.606642+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:51:36.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:35 smithi169 bash[19604]: audit 2024-02-12T15:51:34.432865+0000 mon.a (mon.0) 294 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:51:36.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:35 smithi169 bash[19604]: cluster 2024-02-12T15:51:34.433050+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-12T15:51:36.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:35 smithi169 bash[19604]: audit 2024-02-12T15:51:34.433776+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:35 smithi169 bash[19604]: audit 2024-02-12T15:51:34.440780+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:35 smithi169 bash[19604]: audit 2024-02-12T15:51:34.606142+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:35 smithi169 bash[19604]: audit 2024-02-12T15:51:34.606642+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:51:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:35 smithi123 bash[19119]: audit 2024-02-12T15:51:34.432865+0000 mon.a (mon.0) 294 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:51:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:35 smithi123 bash[19119]: cluster 2024-02-12T15:51:34.433050+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-12T15:51:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:35 smithi123 bash[19119]: audit 2024-02-12T15:51:34.433776+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:35 smithi123 bash[19119]: audit 2024-02-12T15:51:34.440780+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:35 smithi123 bash[19119]: audit 2024-02-12T15:51:34.606142+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:35 smithi123 bash[19119]: audit 2024-02-12T15:51:34.606642+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: cluster 2024-02-12T15:51:33.547334+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: cluster 2024-02-12T15:51:33.547450+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: cluster 2024-02-12T15:51:35.314619+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: cluster 2024-02-12T15:51:35.443271+0000 mon.a (mon.0) 300 : cluster [INF] osd.1 v1:172.21.15.121:6805/2968433670 boot 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: cluster 2024-02-12T15:51:35.443309+0000 mon.a (mon.0) 301 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: audit 2024-02-12T15:51:35.443779+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: audit 2024-02-12T15:51:35.444139+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T15:51:37.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: audit 2024-02-12T15:51:36.447095+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-12T15:51:37.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: cluster 2024-02-12T15:51:36.447173+0000 mon.a (mon.0) 305 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-12T15:51:37.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:36 smithi121 bash[18489]: audit 2024-02-12T15:51:36.448550+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T15:51:37.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: cluster 2024-02-12T15:51:33.547334+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:51:37.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: cluster 2024-02-12T15:51:33.547450+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: cluster 2024-02-12T15:51:35.314619+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: cluster 2024-02-12T15:51:35.443271+0000 mon.a (mon.0) 300 : cluster [INF] osd.1 v1:172.21.15.121:6805/2968433670 boot 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: cluster 2024-02-12T15:51:35.443309+0000 mon.a (mon.0) 301 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: audit 2024-02-12T15:51:35.443779+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: audit 2024-02-12T15:51:35.444139+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: audit 2024-02-12T15:51:36.447095+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: cluster 2024-02-12T15:51:36.447173+0000 mon.a (mon.0) 305 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-12T15:51:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:36 smithi169 bash[19604]: audit 2024-02-12T15:51:36.448550+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: cluster 2024-02-12T15:51:33.547334+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: cluster 2024-02-12T15:51:33.547450+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: cluster 2024-02-12T15:51:35.314619+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: cluster 2024-02-12T15:51:35.443271+0000 mon.a (mon.0) 300 : cluster [INF] osd.1 v1:172.21.15.121:6805/2968433670 boot 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: cluster 2024-02-12T15:51:35.443309+0000 mon.a (mon.0) 301 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: audit 2024-02-12T15:51:35.443779+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: audit 2024-02-12T15:51:35.444139+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: audit 2024-02-12T15:51:36.447095+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-12T15:51:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: cluster 2024-02-12T15:51:36.447173+0000 mon.a (mon.0) 305 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-12T15:51:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:36 smithi123 bash[19119]: audit 2024-02-12T15:51:36.448550+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-12T15:51:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-12T15:51:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-12T15:51:38.622 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-12T15:51:38.622 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-12T15:51:38.622 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.015552 s, 674 MB/s 2024-02-12T15:51:38.622 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-12T15:51:38.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:38 smithi121 bash[18489]: cluster 2024-02-12T15:51:37.315187+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v104: 1 pgs: 1 unknown; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:38.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:38 smithi121 bash[18489]: audit 2024-02-12T15:51:37.448993+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T15:51:38.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:38 smithi121 bash[18489]: cluster 2024-02-12T15:51:37.449084+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-02-12T15:51:38.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:38 smithi169 bash[19604]: cluster 2024-02-12T15:51:37.315187+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v104: 1 pgs: 1 unknown; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:38.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:38 smithi169 bash[19604]: audit 2024-02-12T15:51:37.448993+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T15:51:38.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:38 smithi169 bash[19604]: cluster 2024-02-12T15:51:37.449084+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-02-12T15:51:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:38 smithi123 bash[19119]: cluster 2024-02-12T15:51:37.315187+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v104: 1 pgs: 1 unknown; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:38 smithi123 bash[19119]: audit 2024-02-12T15:51:37.448993+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-12T15:51:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:38 smithi123 bash[19119]: cluster 2024-02-12T15:51:37.449084+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-02-12T15:51:39.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2024-02-12T15:51:39.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:39 smithi121 bash[18489]: cluster 2024-02-12T15:51:38.450225+0000 mon.a (mon.0) 309 : cluster [DBG] osdmap e16: 2 total, 2 up, 2 in 2024-02-12T15:51:39.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:39 smithi169 bash[19604]: cluster 2024-02-12T15:51:38.450225+0000 mon.a (mon.0) 309 : cluster [DBG] osdmap e16: 2 total, 2 up, 2 in 2024-02-12T15:51:39.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:39 smithi123 bash[19119]: cluster 2024-02-12T15:51:38.450225+0000 mon.a (mon.0) 309 : cluster [DBG] osdmap e16: 2 total, 2 up, 2 in 2024-02-12T15:51:41.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:40 smithi121 bash[18489]: cluster 2024-02-12T15:51:39.315765+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v107: 1 pgs: 1 unknown; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:41.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:40 smithi169 bash[19604]: cluster 2024-02-12T15:51:39.315765+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v107: 1 pgs: 1 unknown; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:40 smithi123 bash[19119]: cluster 2024-02-12T15:51:39.315765+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v107: 1 pgs: 1 unknown; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:43.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:42 smithi169 bash[19604]: cluster 2024-02-12T15:51:41.316328+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v108: 1 pgs: 1 unknown; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:43.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:42 smithi123 bash[19119]: cluster 2024-02-12T15:51:41.316328+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v108: 1 pgs: 1 unknown; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:43.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:42 smithi121 bash[18489]: cluster 2024-02-12T15:51:41.316328+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v108: 1 pgs: 1 unknown; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:44.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:43 smithi121 bash[18489]: audit 2024-02-12T15:51:43.526637+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:51:44.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:43 smithi121 bash[18489]: audit 2024-02-12T15:51:43.530395+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:51:44.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:43 smithi121 bash[18489]: audit 2024-02-12T15:51:43.531022+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:44.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:43 smithi169 bash[19604]: audit 2024-02-12T15:51:43.526637+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:51:44.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:43 smithi169 bash[19604]: audit 2024-02-12T15:51:43.530395+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:51:44.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:43 smithi169 bash[19604]: audit 2024-02-12T15:51:43.531022+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:43 smithi123 bash[19119]: audit 2024-02-12T15:51:43.526637+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:51:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:43 smithi123 bash[19119]: audit 2024-02-12T15:51:43.530395+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:51:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:43 smithi123 bash[19119]: audit 2024-02-12T15:51:43.531022+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:45.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: cluster 2024-02-12T15:51:43.316738+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:45.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:43.523168+0000 mgr.y (mgr.14146) 136 : audit [DBG] from='client.14265 v1:172.21.15.121:0/873595915' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:51:45.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:44.691262+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:44.692893+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:44.694199+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:44.702154+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:44.703833+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:44.705177+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:44 smithi169 bash[19604]: audit 2024-02-12T15:51:44.713061+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: cluster 2024-02-12T15:51:43.316738+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:43.523168+0000 mgr.y (mgr.14146) 136 : audit [DBG] from='client.14265 v1:172.21.15.121:0/873595915' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:44.691262+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:44.692893+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:44.694199+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:44.702154+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:44.703833+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:44.705177+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:44 smithi123 bash[19119]: audit 2024-02-12T15:51:44.713061+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: cluster 2024-02-12T15:51:43.316738+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:43.523168+0000 mgr.y (mgr.14146) 136 : audit [DBG] from='client.14265 v1:172.21.15.121:0/873595915' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:44.691262+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:44.692893+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:44.694199+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:44.702154+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:44.703833+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:44.705177+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:51:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:44 smithi121 bash[18489]: audit 2024-02-12T15:51:44.713061+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:51:46.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:45 smithi169 bash[19604]: cephadm 2024-02-12T15:51:44.684753+0000 mgr.y (mgr.14146) 137 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:51:46.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:45 smithi169 bash[19604]: cephadm 2024-02-12T15:51:44.694830+0000 mgr.y (mgr.14146) 138 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 8658M 2024-02-12T15:51:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:45 smithi123 bash[19119]: cephadm 2024-02-12T15:51:44.684753+0000 mgr.y (mgr.14146) 137 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:51:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:45 smithi123 bash[19119]: cephadm 2024-02-12T15:51:44.694830+0000 mgr.y (mgr.14146) 138 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 8658M 2024-02-12T15:51:46.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:45 smithi121 bash[18489]: cephadm 2024-02-12T15:51:44.684753+0000 mgr.y (mgr.14146) 137 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:51:46.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:45 smithi121 bash[18489]: cephadm 2024-02-12T15:51:44.694830+0000 mgr.y (mgr.14146) 138 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 8658M 2024-02-12T15:51:47.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:46 smithi169 bash[19604]: cluster 2024-02-12T15:51:45.317226+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:46 smithi123 bash[19119]: cluster 2024-02-12T15:51:45.317226+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:47.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:46 smithi121 bash[18489]: cluster 2024-02-12T15:51:45.317226+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:49.253 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:48 smithi121 bash[18489]: cluster 2024-02-12T15:51:47.317762+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:49.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:48 smithi169 bash[19604]: cluster 2024-02-12T15:51:47.317762+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:49.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:48 smithi123 bash[19119]: cluster 2024-02-12T15:51:47.317762+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:51.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:50 smithi121 bash[18489]: cluster 2024-02-12T15:51:49.318267+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:50 smithi169 bash[19604]: cluster 2024-02-12T15:51:49.318267+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:50 smithi123 bash[19119]: cluster 2024-02-12T15:51:49.318267+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:52.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:51 smithi169 bash[19604]: audit 2024-02-12T15:51:51.578350+0000 mon.a (mon.0) 320 : audit [INF] from='client.? v1:172.21.15.121:0/1546705632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170"}]: dispatch 2024-02-12T15:51:52.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:51 smithi169 bash[19604]: audit 2024-02-12T15:51:51.586036+0000 mon.a (mon.0) 321 : audit [INF] from='client.? v1:172.21.15.121:0/1546705632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170"}]': finished 2024-02-12T15:51:52.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:51 smithi169 bash[19604]: cluster 2024-02-12T15:51:51.586140+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-12T15:51:52.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:51 smithi169 bash[19604]: audit 2024-02-12T15:51:51.586294+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:51:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:51 smithi123 bash[19119]: audit 2024-02-12T15:51:51.578350+0000 mon.a (mon.0) 320 : audit [INF] from='client.? v1:172.21.15.121:0/1546705632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170"}]: dispatch 2024-02-12T15:51:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:51 smithi123 bash[19119]: audit 2024-02-12T15:51:51.586036+0000 mon.a (mon.0) 321 : audit [INF] from='client.? v1:172.21.15.121:0/1546705632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170"}]': finished 2024-02-12T15:51:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:51 smithi123 bash[19119]: cluster 2024-02-12T15:51:51.586140+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-12T15:51:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:51 smithi123 bash[19119]: audit 2024-02-12T15:51:51.586294+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:51:52.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:51 smithi121 bash[18489]: audit 2024-02-12T15:51:51.578350+0000 mon.a (mon.0) 320 : audit [INF] from='client.? v1:172.21.15.121:0/1546705632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170"}]: dispatch 2024-02-12T15:51:52.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:51 smithi121 bash[18489]: audit 2024-02-12T15:51:51.586036+0000 mon.a (mon.0) 321 : audit [INF] from='client.? v1:172.21.15.121:0/1546705632' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170"}]': finished 2024-02-12T15:51:52.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:51 smithi121 bash[18489]: cluster 2024-02-12T15:51:51.586140+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-12T15:51:52.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:51 smithi121 bash[18489]: audit 2024-02-12T15:51:51.586294+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:51:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:52 smithi169 bash[19604]: cluster 2024-02-12T15:51:51.318826+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:52 smithi169 bash[19604]: audit 2024-02-12T15:51:52.174396+0000 mon.a (mon.0) 324 : audit [DBG] from='client.? v1:172.21.15.121:0/993277178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:51:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:52 smithi123 bash[19119]: cluster 2024-02-12T15:51:51.318826+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:52 smithi123 bash[19119]: audit 2024-02-12T15:51:52.174396+0000 mon.a (mon.0) 324 : audit [DBG] from='client.? v1:172.21.15.121:0/993277178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:51:53.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:52 smithi121 bash[18489]: cluster 2024-02-12T15:51:51.318826+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:53.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:52 smithi121 bash[18489]: audit 2024-02-12T15:51:52.174396+0000 mon.a (mon.0) 324 : audit [DBG] from='client.? v1:172.21.15.121:0/993277178' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:51:54.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:53 smithi169 bash[19604]: cluster 2024-02-12T15:51:53.319348+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:53 smithi123 bash[19119]: cluster 2024-02-12T15:51:53.319348+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:54.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:53 smithi121 bash[18489]: cluster 2024-02-12T15:51:53.319348+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:57.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:56 smithi169 bash[19604]: cluster 2024-02-12T15:51:55.319899+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:57.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:56 smithi169 bash[19604]: cluster 2024-02-12T15:51:56.793547+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:51:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:56 smithi169 bash[19604]: cluster 2024-02-12T15:51:56.793644+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:51:57.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:56 smithi123 bash[19119]: cluster 2024-02-12T15:51:55.319899+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:57.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:56 smithi123 bash[19119]: cluster 2024-02-12T15:51:56.793547+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:51:57.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:56 smithi123 bash[19119]: cluster 2024-02-12T15:51:56.793644+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:51:57.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:56 smithi121 bash[18489]: cluster 2024-02-12T15:51:55.319899+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:57.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:56 smithi121 bash[18489]: cluster 2024-02-12T15:51:56.793547+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:51:57.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:56 smithi121 bash[18489]: cluster 2024-02-12T15:51:56.793644+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:51:58.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:57 smithi169 bash[19604]: cluster 2024-02-12T15:51:56.965698+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-02-12T15:51:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:57 smithi169 bash[19604]: audit 2024-02-12T15:51:56.965960+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:51:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:57 smithi123 bash[19119]: cluster 2024-02-12T15:51:56.965698+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-02-12T15:51:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:57 smithi123 bash[19119]: audit 2024-02-12T15:51:56.965960+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:51:58.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:57 smithi121 bash[18489]: cluster 2024-02-12T15:51:56.965698+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-02-12T15:51:58.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:57 smithi121 bash[18489]: audit 2024-02-12T15:51:56.965960+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:51:59.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:51:58 smithi169 bash[19604]: cluster 2024-02-12T15:51:57.320391+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:59.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:51:58 smithi123 bash[19119]: cluster 2024-02-12T15:51:57.320391+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:51:59.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:51:58 smithi121 bash[18489]: cluster 2024-02-12T15:51:57.320391+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:01.119 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:00 smithi121 bash[18489]: cluster 2024-02-12T15:51:59.320959+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:01.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:00 smithi169 bash[19604]: cluster 2024-02-12T15:51:59.320959+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:00 smithi123 bash[19119]: cluster 2024-02-12T15:51:59.320959+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:03.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:02 smithi121 bash[18489]: cluster 2024-02-12T15:52:01.321529+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:03.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:02 smithi169 bash[19604]: cluster 2024-02-12T15:52:01.321529+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:02 smithi123 bash[19119]: cluster 2024-02-12T15:52:01.321529+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:04.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:03 smithi169 bash[19604]: cluster 2024-02-12T15:52:03.322021+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:03 smithi169 bash[19604]: audit 2024-02-12T15:52:03.401267+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-12T15:52:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:03 smithi169 bash[19604]: audit 2024-02-12T15:52:03.402914+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:03 smithi169 bash[19604]: cephadm 2024-02-12T15:52:03.404369+0000 mgr.y (mgr.14146) 149 : cephadm [INF] Deploying daemon osd.2 on smithi121 2024-02-12T15:52:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:03 smithi123 bash[19119]: cluster 2024-02-12T15:52:03.322021+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:03 smithi123 bash[19119]: audit 2024-02-12T15:52:03.401267+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-12T15:52:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:03 smithi123 bash[19119]: audit 2024-02-12T15:52:03.402914+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:03 smithi123 bash[19119]: cephadm 2024-02-12T15:52:03.404369+0000 mgr.y (mgr.14146) 149 : cephadm [INF] Deploying daemon osd.2 on smithi121 2024-02-12T15:52:04.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:03 smithi121 bash[18489]: cluster 2024-02-12T15:52:03.322021+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:04.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:03 smithi121 bash[18489]: audit 2024-02-12T15:52:03.401267+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-12T15:52:04.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:03 smithi121 bash[18489]: audit 2024-02-12T15:52:03.402914+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:04.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:03 smithi121 bash[18489]: cephadm 2024-02-12T15:52:03.404369+0000 mgr.y (mgr.14146) 149 : cephadm [INF] Deploying daemon osd.2 on smithi121 2024-02-12T15:52:07.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:06 smithi121 bash[18489]: cluster 2024-02-12T15:52:05.322542+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:07.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:06 smithi169 bash[19604]: cluster 2024-02-12T15:52:05.322542+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:06 smithi123 bash[19119]: cluster 2024-02-12T15:52:05.322542+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:08 smithi123 bash[19119]: cluster 2024-02-12T15:52:07.323061+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:08 smithi123 bash[19119]: audit 2024-02-12T15:52:07.382149+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:08 smithi123 bash[19119]: audit 2024-02-12T15:52:07.388252+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:08.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:08 smithi121 bash[18489]: cluster 2024-02-12T15:52:07.323061+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:08.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:08 smithi121 bash[18489]: audit 2024-02-12T15:52:07.382149+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:08.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:08 smithi121 bash[18489]: audit 2024-02-12T15:52:07.388252+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:08.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:08 smithi169 bash[19604]: cluster 2024-02-12T15:52:07.323061+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:08 smithi169 bash[19604]: audit 2024-02-12T15:52:07.382149+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:08 smithi169 bash[19604]: audit 2024-02-12T15:52:07.388252+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:11.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:10 smithi121 bash[18489]: cluster 2024-02-12T15:52:09.323567+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:11.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:10 smithi169 bash[19604]: cluster 2024-02-12T15:52:09.323567+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:10 smithi123 bash[19119]: cluster 2024-02-12T15:52:09.323567+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:12 smithi169 bash[19604]: cluster 2024-02-12T15:52:11.324139+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:12 smithi169 bash[19604]: audit 2024-02-12T15:52:12.777154+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:13.361 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:12 smithi121 bash[18489]: cluster 2024-02-12T15:52:11.324139+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:13.361 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:12 smithi121 bash[18489]: audit 2024-02-12T15:52:12.777154+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:12 smithi123 bash[19119]: cluster 2024-02-12T15:52:11.324139+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:12 smithi123 bash[19119]: audit 2024-02-12T15:52:12.777154+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:14.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:13 smithi169 bash[19604]: audit 2024-02-12T15:52:13.057578+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:14.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:13 smithi169 bash[19604]: audit 2024-02-12T15:52:13.058861+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:13 smithi169 bash[19604]: audit 2024-02-12T15:52:13.065966+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:13 smithi169 bash[19604]: audit 2024-02-12T15:52:13.580526+0000 mon.a (mon.0) 335 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T15:52:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:13 smithi123 bash[19119]: audit 2024-02-12T15:52:13.057578+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:13 smithi123 bash[19119]: audit 2024-02-12T15:52:13.058861+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:13 smithi123 bash[19119]: audit 2024-02-12T15:52:13.065966+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:13 smithi123 bash[19119]: audit 2024-02-12T15:52:13.580526+0000 mon.a (mon.0) 335 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T15:52:14.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:13 smithi121 bash[18489]: audit 2024-02-12T15:52:13.057578+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:13 smithi121 bash[18489]: audit 2024-02-12T15:52:13.058861+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:13 smithi121 bash[18489]: audit 2024-02-12T15:52:13.065966+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:13 smithi121 bash[18489]: audit 2024-02-12T15:52:13.580526+0000 mon.a (mon.0) 335 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-12T15:52:15.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:14 smithi121 bash[18489]: cluster 2024-02-12T15:52:13.324698+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:15.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:14 smithi121 bash[18489]: audit 2024-02-12T15:52:14.073385+0000 mon.a (mon.0) 336 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-12T15:52:15.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:14 smithi121 bash[18489]: cluster 2024-02-12T15:52:14.073455+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e19: 3 total, 2 up, 3 in 2024-02-12T15:52:15.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:14 smithi121 bash[18489]: audit 2024-02-12T15:52:14.073783+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:15.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:14 smithi121 bash[18489]: audit 2024-02-12T15:52:14.074567+0000 mon.a (mon.0) 339 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:52:15.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:14 smithi169 bash[19604]: cluster 2024-02-12T15:52:13.324698+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:15.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:14 smithi169 bash[19604]: audit 2024-02-12T15:52:14.073385+0000 mon.a (mon.0) 336 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-12T15:52:15.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:14 smithi169 bash[19604]: cluster 2024-02-12T15:52:14.073455+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e19: 3 total, 2 up, 3 in 2024-02-12T15:52:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:14 smithi169 bash[19604]: audit 2024-02-12T15:52:14.073783+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:14 smithi169 bash[19604]: audit 2024-02-12T15:52:14.074567+0000 mon.a (mon.0) 339 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:52:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:14 smithi123 bash[19119]: cluster 2024-02-12T15:52:13.324698+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:14 smithi123 bash[19119]: audit 2024-02-12T15:52:14.073385+0000 mon.a (mon.0) 336 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-12T15:52:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:14 smithi123 bash[19119]: cluster 2024-02-12T15:52:14.073455+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e19: 3 total, 2 up, 3 in 2024-02-12T15:52:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:14 smithi123 bash[19119]: audit 2024-02-12T15:52:14.073783+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:14 smithi123 bash[19119]: audit 2024-02-12T15:52:14.074567+0000 mon.a (mon.0) 339 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:52:15.957 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 2 on host 'smithi121' 2024-02-12T15:52:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:16 smithi123 bash[19119]: audit 2024-02-12T15:52:15.076202+0000 mon.a (mon.0) 340 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:52:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:16 smithi123 bash[19119]: cluster 2024-02-12T15:52:15.076250+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 2 up, 3 in 2024-02-12T15:52:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:16 smithi123 bash[19119]: audit 2024-02-12T15:52:15.076469+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:16 smithi123 bash[19119]: audit 2024-02-12T15:52:15.080437+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:16 smithi123 bash[19119]: cluster 2024-02-12T15:52:15.325092+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:16 smithi123 bash[19119]: audit 2024-02-12T15:52:15.953125+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:16 smithi123 bash[19119]: audit 2024-02-12T15:52:15.989849+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:16.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:16 smithi121 bash[18489]: audit 2024-02-12T15:52:15.076202+0000 mon.a (mon.0) 340 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:52:16.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:16 smithi121 bash[18489]: cluster 2024-02-12T15:52:15.076250+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 2 up, 3 in 2024-02-12T15:52:16.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:16 smithi121 bash[18489]: audit 2024-02-12T15:52:15.076469+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:16.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:16 smithi121 bash[18489]: audit 2024-02-12T15:52:15.080437+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:16.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:16 smithi121 bash[18489]: cluster 2024-02-12T15:52:15.325092+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:16.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:16 smithi121 bash[18489]: audit 2024-02-12T15:52:15.953125+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:16.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:16 smithi121 bash[18489]: audit 2024-02-12T15:52:15.989849+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:16.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:16 smithi169 bash[19604]: audit 2024-02-12T15:52:15.076202+0000 mon.a (mon.0) 340 : audit [INF] from='osd.2 v1:172.21.15.121:6809/2158187852' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:52:16.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:16 smithi169 bash[19604]: cluster 2024-02-12T15:52:15.076250+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e20: 3 total, 2 up, 3 in 2024-02-12T15:52:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:16 smithi169 bash[19604]: audit 2024-02-12T15:52:15.076469+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:16 smithi169 bash[19604]: audit 2024-02-12T15:52:15.080437+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:16 smithi169 bash[19604]: cluster 2024-02-12T15:52:15.325092+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-12T15:52:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:16 smithi169 bash[19604]: audit 2024-02-12T15:52:15.953125+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:16 smithi169 bash[19604]: audit 2024-02-12T15:52:15.989849+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:16.560 DEBUG:teuthology.orchestra.run.smithi121:osd.2> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.2.service 2024-02-12T15:52:16.562 INFO:tasks.cephadm:Deploying osd.3 on smithi121 with /dev/vg_nvme/lv_1... 2024-02-12T15:52:16.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-12T15:52:16.699 INFO:journalctl@ceph.osd.2.smithi121.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:52:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:17 smithi123 bash[19119]: cluster 2024-02-12T15:52:14.609024+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:17 smithi123 bash[19119]: cluster 2024-02-12T15:52:14.609129+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:17 smithi123 bash[19119]: cluster 2024-02-12T15:52:16.078396+0000 mon.a (mon.0) 346 : cluster [INF] osd.2 v1:172.21.15.121:6809/2158187852 boot 2024-02-12T15:52:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:17 smithi123 bash[19119]: cluster 2024-02-12T15:52:16.078430+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-12T15:52:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:17 smithi123 bash[19119]: audit 2024-02-12T15:52:16.078984+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:17.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:17 smithi121 bash[18489]: cluster 2024-02-12T15:52:14.609024+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:17.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:17 smithi121 bash[18489]: cluster 2024-02-12T15:52:14.609129+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:17.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:17 smithi121 bash[18489]: cluster 2024-02-12T15:52:16.078396+0000 mon.a (mon.0) 346 : cluster [INF] osd.2 v1:172.21.15.121:6809/2158187852 boot 2024-02-12T15:52:17.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:17 smithi121 bash[18489]: cluster 2024-02-12T15:52:16.078430+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-12T15:52:17.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:17 smithi121 bash[18489]: audit 2024-02-12T15:52:16.078984+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:17.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:17 smithi169 bash[19604]: cluster 2024-02-12T15:52:14.609024+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:17.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:17 smithi169 bash[19604]: cluster 2024-02-12T15:52:14.609129+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:17.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:17 smithi169 bash[19604]: cluster 2024-02-12T15:52:16.078396+0000 mon.a (mon.0) 346 : cluster [INF] osd.2 v1:172.21.15.121:6809/2158187852 boot 2024-02-12T15:52:17.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:17 smithi169 bash[19604]: cluster 2024-02-12T15:52:16.078430+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-12T15:52:17.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:17 smithi169 bash[19604]: audit 2024-02-12T15:52:16.078984+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-12T15:52:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:18 smithi123 bash[19119]: cluster 2024-02-12T15:52:17.098867+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-02-12T15:52:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:18 smithi123 bash[19119]: cluster 2024-02-12T15:52:17.325582+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:18.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:18 smithi121 bash[18489]: cluster 2024-02-12T15:52:17.098867+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-02-12T15:52:18.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:18 smithi121 bash[18489]: cluster 2024-02-12T15:52:17.325582+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:18.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:18 smithi169 bash[19604]: cluster 2024-02-12T15:52:17.098867+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-02-12T15:52:18.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:18 smithi169 bash[19604]: cluster 2024-02-12T15:52:17.325582+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-12T15:52:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-12T15:52:19.899 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-12T15:52:19.899 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-12T15:52:19.900 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134779 s, 778 MB/s 2024-02-12T15:52:19.900 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-12T15:52:20.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2024-02-12T15:52:21.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:20 smithi169 bash[19604]: cluster 2024-02-12T15:52:19.326027+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:21.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:20 smithi123 bash[19119]: cluster 2024-02-12T15:52:19.326027+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:21.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:20 smithi121 bash[18489]: cluster 2024-02-12T15:52:19.326027+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:23.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:22 smithi169 bash[19604]: cluster 2024-02-12T15:52:21.326525+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:23.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:22 smithi123 bash[19119]: cluster 2024-02-12T15:52:21.326525+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:23.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:22 smithi121 bash[18489]: cluster 2024-02-12T15:52:21.326525+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:25.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:24 smithi169 bash[19604]: cluster 2024-02-12T15:52:23.327115+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:25.357 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:24 smithi121 bash[18489]: cluster 2024-02-12T15:52:23.327115+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:25.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:24 smithi123 bash[19119]: cluster 2024-02-12T15:52:23.327115+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:26.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:25 smithi169 bash[19604]: audit 2024-02-12T15:52:25.007583+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:52:26.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:25 smithi169 bash[19604]: audit 2024-02-12T15:52:25.010467+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:52:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:25 smithi169 bash[19604]: audit 2024-02-12T15:52:25.011424+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:25 smithi123 bash[19119]: audit 2024-02-12T15:52:25.007583+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:52:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:25 smithi123 bash[19119]: audit 2024-02-12T15:52:25.010467+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:52:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:25 smithi123 bash[19119]: audit 2024-02-12T15:52:25.011424+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:26.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:25 smithi121 bash[18489]: audit 2024-02-12T15:52:25.007583+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:52:26.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:25 smithi121 bash[18489]: audit 2024-02-12T15:52:25.010467+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:52:26.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:25 smithi121 bash[18489]: audit 2024-02-12T15:52:25.011424+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:25.004743+0000 mgr.y (mgr.14146) 160 : audit [DBG] from='client.14292 v1:172.21.15.121:0/331470242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: cluster 2024-02-12T15:52:25.327505+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.212662+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.214054+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.215222+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.216351+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.224771+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.226381+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.227852+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:26 smithi169 bash[19604]: audit 2024-02-12T15:52:26.235973+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:25.004743+0000 mgr.y (mgr.14146) 160 : audit [DBG] from='client.14292 v1:172.21.15.121:0/331470242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:52:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: cluster 2024-02-12T15:52:25.327505+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.212662+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.214054+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.215222+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.216351+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.224771+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.226381+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.227852+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:26 smithi123 bash[19119]: audit 2024-02-12T15:52:26.235973+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:25.004743+0000 mgr.y (mgr.14146) 160 : audit [DBG] from='client.14292 v1:172.21.15.121:0/331470242' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: cluster 2024-02-12T15:52:25.327505+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.212662+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.214054+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.215222+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.216351+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.224771+0000 mon.a (mon.0) 357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.226381+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.227852+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:26 smithi121 bash[18489]: audit 2024-02-12T15:52:26.235973+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:28.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:27 smithi169 bash[19604]: cephadm 2024-02-12T15:52:26.205326+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:52:28.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:27 smithi169 bash[19604]: cephadm 2024-02-12T15:52:26.217574+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 5772M 2024-02-12T15:52:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:27 smithi123 bash[19119]: cephadm 2024-02-12T15:52:26.205326+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:52:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:27 smithi123 bash[19119]: cephadm 2024-02-12T15:52:26.217574+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 5772M 2024-02-12T15:52:28.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:27 smithi121 bash[18489]: cephadm 2024-02-12T15:52:26.205326+0000 mgr.y (mgr.14146) 162 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:52:28.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:27 smithi121 bash[18489]: cephadm 2024-02-12T15:52:26.217574+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 5772M 2024-02-12T15:52:29.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:28 smithi169 bash[19604]: cluster 2024-02-12T15:52:27.328072+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:29.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:28 smithi123 bash[19119]: cluster 2024-02-12T15:52:27.328072+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:29.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:28 smithi121 bash[18489]: cluster 2024-02-12T15:52:27.328072+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:31.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:30 smithi169 bash[19604]: cluster 2024-02-12T15:52:29.328620+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:31.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:30 smithi123 bash[19119]: cluster 2024-02-12T15:52:29.328620+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:31.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:30 smithi121 bash[18489]: cluster 2024-02-12T15:52:29.328620+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:32.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:32 smithi169 bash[19604]: cluster 2024-02-12T15:52:31.329174+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:32.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:32 smithi121 bash[18489]: cluster 2024-02-12T15:52:31.329174+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:32 smithi123 bash[19119]: cluster 2024-02-12T15:52:31.329174+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:33.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:33 smithi123 bash[19119]: audit 2024-02-12T15:52:33.069981+0000 mon.a (mon.0) 361 : audit [INF] from='client.? v1:172.21.15.121:0/741432404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd"}]: dispatch 2024-02-12T15:52:33.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:33 smithi123 bash[19119]: audit 2024-02-12T15:52:33.074961+0000 mon.a (mon.0) 362 : audit [INF] from='client.? v1:172.21.15.121:0/741432404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd"}]': finished 2024-02-12T15:52:33.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:33 smithi123 bash[19119]: cluster 2024-02-12T15:52:33.074999+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-12T15:52:33.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:33 smithi123 bash[19119]: audit 2024-02-12T15:52:33.075099+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:33.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:33 smithi121 bash[18489]: audit 2024-02-12T15:52:33.069981+0000 mon.a (mon.0) 361 : audit [INF] from='client.? v1:172.21.15.121:0/741432404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd"}]: dispatch 2024-02-12T15:52:33.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:33 smithi121 bash[18489]: audit 2024-02-12T15:52:33.074961+0000 mon.a (mon.0) 362 : audit [INF] from='client.? v1:172.21.15.121:0/741432404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd"}]': finished 2024-02-12T15:52:33.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:33 smithi121 bash[18489]: cluster 2024-02-12T15:52:33.074999+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-12T15:52:33.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:33 smithi121 bash[18489]: audit 2024-02-12T15:52:33.075099+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:33.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:33 smithi169 bash[19604]: audit 2024-02-12T15:52:33.069981+0000 mon.a (mon.0) 361 : audit [INF] from='client.? v1:172.21.15.121:0/741432404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd"}]: dispatch 2024-02-12T15:52:33.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:33 smithi169 bash[19604]: audit 2024-02-12T15:52:33.074961+0000 mon.a (mon.0) 362 : audit [INF] from='client.? v1:172.21.15.121:0/741432404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd"}]': finished 2024-02-12T15:52:33.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:33 smithi169 bash[19604]: cluster 2024-02-12T15:52:33.074999+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-12T15:52:33.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:33 smithi169 bash[19604]: audit 2024-02-12T15:52:33.075099+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:34.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:34 smithi121 bash[18489]: cluster 2024-02-12T15:52:33.329713+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:34.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:34 smithi121 bash[18489]: audit 2024-02-12T15:52:33.659133+0000 mon.a (mon.0) 365 : audit [DBG] from='client.? v1:172.21.15.121:0/2795970523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:52:34.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:34 smithi169 bash[19604]: cluster 2024-02-12T15:52:33.329713+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:34.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:34 smithi169 bash[19604]: audit 2024-02-12T15:52:33.659133+0000 mon.a (mon.0) 365 : audit [DBG] from='client.? v1:172.21.15.121:0/2795970523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:52:34.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:34 smithi123 bash[19119]: cluster 2024-02-12T15:52:33.329713+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:34.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:34 smithi123 bash[19119]: audit 2024-02-12T15:52:33.659133+0000 mon.a (mon.0) 365 : audit [DBG] from='client.? v1:172.21.15.121:0/2795970523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:52:37.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:36 smithi169 bash[19604]: cluster 2024-02-12T15:52:35.330382+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:36 smithi123 bash[19119]: cluster 2024-02-12T15:52:35.330382+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:37.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:36 smithi121 bash[18489]: cluster 2024-02-12T15:52:35.330382+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:39.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:38 smithi169 bash[19604]: cluster 2024-02-12T15:52:37.331059+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:39.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:38 smithi123 bash[19119]: cluster 2024-02-12T15:52:37.331059+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:39.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:38 smithi121 bash[18489]: cluster 2024-02-12T15:52:37.331059+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:41.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:40 smithi169 bash[19604]: cluster 2024-02-12T15:52:39.331610+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:40 smithi123 bash[19119]: cluster 2024-02-12T15:52:39.331610+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:41.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:40 smithi121 bash[18489]: cluster 2024-02-12T15:52:39.331610+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:43.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:42 smithi169 bash[19604]: cluster 2024-02-12T15:52:41.332176+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:43.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:42 smithi123 bash[19119]: cluster 2024-02-12T15:52:41.332176+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:43.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:42 smithi121 bash[18489]: cluster 2024-02-12T15:52:41.332176+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:45.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:44 smithi169 bash[19604]: cluster 2024-02-12T15:52:43.332722+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:44 smithi123 bash[19119]: cluster 2024-02-12T15:52:43.332722+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:45.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:44 smithi121 bash[18489]: cluster 2024-02-12T15:52:43.332722+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:46.246 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:45 smithi121 bash[18489]: audit 2024-02-12T15:52:44.999754+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-12T15:52:46.246 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:45 smithi121 bash[18489]: audit 2024-02-12T15:52:45.001392+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:46.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:45 smithi169 bash[19604]: audit 2024-02-12T15:52:44.999754+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-12T15:52:46.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:45 smithi169 bash[19604]: audit 2024-02-12T15:52:45.001392+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:45 smithi123 bash[19119]: audit 2024-02-12T15:52:44.999754+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-12T15:52:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:45 smithi123 bash[19119]: audit 2024-02-12T15:52:45.001392+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:47.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:46 smithi169 bash[19604]: cephadm 2024-02-12T15:52:45.002835+0000 mgr.y (mgr.14146) 173 : cephadm [INF] Deploying daemon osd.3 on smithi121 2024-02-12T15:52:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:46 smithi169 bash[19604]: cluster 2024-02-12T15:52:45.333274+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:46 smithi123 bash[19119]: cephadm 2024-02-12T15:52:45.002835+0000 mgr.y (mgr.14146) 173 : cephadm [INF] Deploying daemon osd.3 on smithi121 2024-02-12T15:52:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:46 smithi123 bash[19119]: cluster 2024-02-12T15:52:45.333274+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:46 smithi121 bash[18489]: cephadm 2024-02-12T15:52:45.002835+0000 mgr.y (mgr.14146) 173 : cephadm [INF] Deploying daemon osd.3 on smithi121 2024-02-12T15:52:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:46 smithi121 bash[18489]: cluster 2024-02-12T15:52:45.333274+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:48.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:48 smithi169 bash[19604]: cluster 2024-02-12T15:52:47.333929+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:48 smithi123 bash[19119]: cluster 2024-02-12T15:52:47.333929+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:48.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:48 smithi121 bash[18489]: cluster 2024-02-12T15:52:47.333929+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:50.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:49 smithi121 bash[18489]: audit 2024-02-12T15:52:48.919211+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:50.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:49 smithi121 bash[18489]: audit 2024-02-12T15:52:48.960323+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:50.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:49 smithi169 bash[19604]: audit 2024-02-12T15:52:48.919211+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:50.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:49 smithi169 bash[19604]: audit 2024-02-12T15:52:48.960323+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:50.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:49 smithi123 bash[19119]: audit 2024-02-12T15:52:48.919211+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:50.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:49 smithi123 bash[19119]: audit 2024-02-12T15:52:48.960323+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:50 smithi169 bash[19604]: cluster 2024-02-12T15:52:49.334636+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:50 smithi169 bash[19604]: cluster 2024-02-12T15:52:50.135650+0000 osd.1 (osd.1) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:52:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:50 smithi169 bash[19604]: cluster 2024-02-12T15:52:50.139381+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:52:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:50 smithi123 bash[19119]: cluster 2024-02-12T15:52:49.334636+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:50 smithi123 bash[19119]: cluster 2024-02-12T15:52:50.135650+0000 osd.1 (osd.1) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:52:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:50 smithi123 bash[19119]: cluster 2024-02-12T15:52:50.139381+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:52:51.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:50 smithi121 bash[18489]: cluster 2024-02-12T15:52:49.334636+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:51.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:50 smithi121 bash[18489]: cluster 2024-02-12T15:52:50.135650+0000 osd.1 (osd.1) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:52:51.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:50 smithi121 bash[18489]: cluster 2024-02-12T15:52:50.139381+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:52:53.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:52 smithi121 bash[18489]: cluster 2024-02-12T15:52:51.335217+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:53.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:52 smithi169 bash[19604]: cluster 2024-02-12T15:52:51.335217+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:52 smithi123 bash[19119]: cluster 2024-02-12T15:52:51.335217+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:55.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:54 smithi169 bash[19604]: cluster 2024-02-12T15:52:53.335767+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:54 smithi123 bash[19119]: cluster 2024-02-12T15:52:53.335767+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:55.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:54 smithi121 bash[18489]: cluster 2024-02-12T15:52:53.335767+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:56 smithi123 bash[19119]: audit 2024-02-12T15:52:55.322853+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:56 smithi123 bash[19119]: audit 2024-02-12T15:52:55.324285+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:56 smithi123 bash[19119]: audit 2024-02-12T15:52:55.325428+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:56 smithi123 bash[19119]: audit 2024-02-12T15:52:55.330891+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:56 smithi123 bash[19119]: cluster 2024-02-12T15:52:55.336272+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:56 smithi123 bash[19119]: audit 2024-02-12T15:52:55.775845+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T15:52:56.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:56 smithi121 bash[18489]: audit 2024-02-12T15:52:55.322853+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:56.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:56 smithi121 bash[18489]: audit 2024-02-12T15:52:55.324285+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:56.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:56 smithi121 bash[18489]: audit 2024-02-12T15:52:55.325428+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:56.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:56 smithi121 bash[18489]: audit 2024-02-12T15:52:55.330891+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:56.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:56 smithi121 bash[18489]: cluster 2024-02-12T15:52:55.336272+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:56.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:56 smithi121 bash[18489]: audit 2024-02-12T15:52:55.775845+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T15:52:56.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:56 smithi169 bash[19604]: audit 2024-02-12T15:52:55.322853+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:56 smithi169 bash[19604]: audit 2024-02-12T15:52:55.324285+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:52:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:56 smithi169 bash[19604]: audit 2024-02-12T15:52:55.325428+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:52:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:56 smithi169 bash[19604]: audit 2024-02-12T15:52:55.330891+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:56 smithi169 bash[19604]: cluster 2024-02-12T15:52:55.336272+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:56 smithi169 bash[19604]: audit 2024-02-12T15:52:55.775845+0000 mon.a (mon.0) 374 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T15:52:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:57 smithi123 bash[19119]: audit 2024-02-12T15:52:56.339840+0000 mon.a (mon.0) 375 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T15:52:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:57 smithi123 bash[19119]: cluster 2024-02-12T15:52:56.339910+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-12T15:52:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:57 smithi123 bash[19119]: audit 2024-02-12T15:52:56.340045+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:57 smithi123 bash[19119]: audit 2024-02-12T15:52:56.340704+0000 mon.a (mon.0) 378 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:52:57.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:57 smithi121 bash[18489]: audit 2024-02-12T15:52:56.339840+0000 mon.a (mon.0) 375 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T15:52:57.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:57 smithi121 bash[18489]: cluster 2024-02-12T15:52:56.339910+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-12T15:52:57.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:57 smithi121 bash[18489]: audit 2024-02-12T15:52:56.340045+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:57.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:57 smithi121 bash[18489]: audit 2024-02-12T15:52:56.340704+0000 mon.a (mon.0) 378 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:52:57.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:57 smithi169 bash[19604]: audit 2024-02-12T15:52:56.339840+0000 mon.a (mon.0) 375 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T15:52:57.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:57 smithi169 bash[19604]: cluster 2024-02-12T15:52:56.339910+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-12T15:52:57.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:57 smithi169 bash[19604]: audit 2024-02-12T15:52:56.340045+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:57.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:57 smithi169 bash[19604]: audit 2024-02-12T15:52:56.340704+0000 mon.a (mon.0) 378 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T15:52:58.527 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 3 on host 'smithi121' 2024-02-12T15:52:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:58 smithi123 bash[19119]: cluster 2024-02-12T15:52:56.872861+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:58 smithi123 bash[19119]: cluster 2024-02-12T15:52:56.872944+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:58 smithi123 bash[19119]: cluster 2024-02-12T15:52:57.336857+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:58 smithi123 bash[19119]: audit 2024-02-12T15:52:57.342567+0000 mon.a (mon.0) 379 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:52:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:58 smithi123 bash[19119]: cluster 2024-02-12T15:52:57.342637+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-12T15:52:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:58 smithi123 bash[19119]: audit 2024-02-12T15:52:57.343243+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:58 smithi123 bash[19119]: audit 2024-02-12T15:52:57.350691+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:58.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:58 smithi121 bash[18489]: cluster 2024-02-12T15:52:56.872861+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:58.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:58 smithi121 bash[18489]: cluster 2024-02-12T15:52:56.872944+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:58.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:58 smithi121 bash[18489]: cluster 2024-02-12T15:52:57.336857+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:58.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:58 smithi121 bash[18489]: audit 2024-02-12T15:52:57.342567+0000 mon.a (mon.0) 379 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:52:58.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:58 smithi121 bash[18489]: cluster 2024-02-12T15:52:57.342637+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-12T15:52:58.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:58 smithi121 bash[18489]: audit 2024-02-12T15:52:57.343243+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:58.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:58 smithi121 bash[18489]: audit 2024-02-12T15:52:57.350691+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:58.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:58 smithi169 bash[19604]: cluster 2024-02-12T15:52:56.872861+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:58.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:58 smithi169 bash[19604]: cluster 2024-02-12T15:52:56.872944+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:58.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:58 smithi169 bash[19604]: cluster 2024-02-12T15:52:57.336857+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:52:58.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:58 smithi169 bash[19604]: audit 2024-02-12T15:52:57.342567+0000 mon.a (mon.0) 379 : audit [INF] from='osd.3 v1:172.21.15.121:6813/1415189366' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-12T15:52:58.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:58 smithi169 bash[19604]: cluster 2024-02-12T15:52:57.342637+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-12T15:52:58.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:58 smithi169 bash[19604]: audit 2024-02-12T15:52:57.343243+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:58.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:58 smithi169 bash[19604]: audit 2024-02-12T15:52:57.350691+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:59.165 DEBUG:teuthology.orchestra.run.smithi121:osd.3> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.3.service 2024-02-12T15:52:59.167 INFO:tasks.cephadm:Deploying osd.4 on smithi123 with /dev/vg_nvme/lv_4... 2024-02-12T15:52:59.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-12T15:52:59.358 INFO:journalctl@ceph.osd.3.smithi121.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:52:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: cluster 2024-02-12T15:52:56.734905+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: cluster 2024-02-12T15:52:56.735020+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: audit 2024-02-12T15:52:58.348035+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: cluster 2024-02-12T15:52:58.354978+0000 mon.a (mon.0) 384 : cluster [INF] osd.3 v1:172.21.15.121:6813/1415189366 boot 2024-02-12T15:52:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: cluster 2024-02-12T15:52:58.355090+0000 mon.a (mon.0) 385 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-12T15:52:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: audit 2024-02-12T15:52:58.356081+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: audit 2024-02-12T15:52:58.523225+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:52:59 smithi123 bash[19119]: audit 2024-02-12T15:52:58.555715+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: cluster 2024-02-12T15:52:56.734905+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: cluster 2024-02-12T15:52:56.735020+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: audit 2024-02-12T15:52:58.348035+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: cluster 2024-02-12T15:52:58.354978+0000 mon.a (mon.0) 384 : cluster [INF] osd.3 v1:172.21.15.121:6813/1415189366 boot 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: cluster 2024-02-12T15:52:58.355090+0000 mon.a (mon.0) 385 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: audit 2024-02-12T15:52:58.356081+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: audit 2024-02-12T15:52:58.523225+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:59.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:52:59 smithi121 bash[18489]: audit 2024-02-12T15:52:58.555715+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:52:59.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: cluster 2024-02-12T15:52:56.734905+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:52:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: cluster 2024-02-12T15:52:56.735020+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:52:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: audit 2024-02-12T15:52:58.348035+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: cluster 2024-02-12T15:52:58.354978+0000 mon.a (mon.0) 384 : cluster [INF] osd.3 v1:172.21.15.121:6813/1415189366 boot 2024-02-12T15:52:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: cluster 2024-02-12T15:52:58.355090+0000 mon.a (mon.0) 385 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-12T15:52:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: audit 2024-02-12T15:52:58.356081+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T15:52:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: audit 2024-02-12T15:52:58.523225+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:52:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:52:59 smithi169 bash[19604]: audit 2024-02-12T15:52:58.555715+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:53:00.605 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:00 smithi123 bash[19119]: cluster 2024-02-12T15:52:59.337465+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:53:00.605 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:00 smithi123 bash[19119]: cluster 2024-02-12T15:52:59.530649+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-12T15:53:00.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:00 smithi169 bash[19604]: cluster 2024-02-12T15:52:59.337465+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:53:00.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:00 smithi169 bash[19604]: cluster 2024-02-12T15:52:59.530649+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-12T15:53:00.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:00 smithi121 bash[18489]: cluster 2024-02-12T15:52:59.337465+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-12T15:53:00.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:00 smithi121 bash[18489]: cluster 2024-02-12T15:52:59.530649+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-12T15:53:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-12T15:53:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-12T15:53:02.408 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-02-12T15:53:02.408 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-02-12T15:53:02.408 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0163099 s, 643 MB/s 2024-02-12T15:53:02.409 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-02-12T15:53:02.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:02 smithi169 bash[19604]: cluster 2024-02-12T15:53:01.338155+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:02 smithi123 bash[19119]: cluster 2024-02-12T15:53:01.338155+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:02.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:02 smithi121 bash[18489]: cluster 2024-02-12T15:53:01.338155+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:02.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_4 2024-02-12T15:53:04.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:04 smithi169 bash[19604]: cluster 2024-02-12T15:53:03.338735+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:04 smithi123 bash[19119]: cluster 2024-02-12T15:53:03.338735+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:04.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:04 smithi121 bash[18489]: cluster 2024-02-12T15:53:03.338735+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:06.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:05 smithi121 bash[18489]: audit 2024-02-12T15:53:05.758916+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:53:06.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:05 smithi121 bash[18489]: audit 2024-02-12T15:53:05.764783+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:53:06.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:05 smithi121 bash[18489]: audit 2024-02-12T15:53:05.766257+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:05 smithi169 bash[19604]: audit 2024-02-12T15:53:05.758916+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:53:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:05 smithi169 bash[19604]: audit 2024-02-12T15:53:05.764783+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:53:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:05 smithi169 bash[19604]: audit 2024-02-12T15:53:05.766257+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:05 smithi123 bash[19119]: audit 2024-02-12T15:53:05.758916+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:53:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:05 smithi123 bash[19119]: audit 2024-02-12T15:53:05.764783+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:53:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:05 smithi123 bash[19119]: audit 2024-02-12T15:53:05.766257+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:07.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:06 smithi121 bash[18489]: cluster 2024-02-12T15:53:05.339338+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:07.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:06 smithi121 bash[18489]: audit 2024-02-12T15:53:05.751580+0000 mgr.y (mgr.14146) 185 : audit [DBG] from='client.24233 v1:172.21.15.123:0/428698506' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:53:07.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:06 smithi169 bash[19604]: cluster 2024-02-12T15:53:05.339338+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:07.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:06 smithi169 bash[19604]: audit 2024-02-12T15:53:05.751580+0000 mgr.y (mgr.14146) 185 : audit [DBG] from='client.24233 v1:172.21.15.123:0/428698506' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:53:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:06 smithi123 bash[19119]: cluster 2024-02-12T15:53:05.339338+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:06 smithi123 bash[19119]: audit 2024-02-12T15:53:05.751580+0000 mgr.y (mgr.14146) 185 : audit [DBG] from='client.24233 v1:172.21.15.123:0/428698506' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:53:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: cluster 2024-02-12T15:53:07.339943+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: cephadm 2024-02-12T15:53:07.555555+0000 mgr.y (mgr.14146) 187 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:53:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.566017+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.568163+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.569678+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.571101+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.572487+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: cephadm 2024-02-12T15:53:07.573507+0000 mgr.y (mgr.14146) 188 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 4329M 2024-02-12T15:53:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.579282+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.580605+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.581877+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:08 smithi123 bash[19119]: audit 2024-02-12T15:53:07.588573+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:08.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: cluster 2024-02-12T15:53:07.339943+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:08.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: cephadm 2024-02-12T15:53:07.555555+0000 mgr.y (mgr.14146) 187 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:53:08.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.566017+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:08.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.568163+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.569678+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.571101+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.572487+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: cephadm 2024-02-12T15:53:07.573507+0000 mgr.y (mgr.14146) 188 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 4329M 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.579282+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.580605+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.581877+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:08 smithi121 bash[18489]: audit 2024-02-12T15:53:07.588573+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: cluster 2024-02-12T15:53:07.339943+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: cephadm 2024-02-12T15:53:07.555555+0000 mgr.y (mgr.14146) 187 : cephadm [INF] Detected new or changed devices on smithi121 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.566017+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.568163+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.569678+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.571101+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.572487+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: cephadm 2024-02-12T15:53:07.573507+0000 mgr.y (mgr.14146) 188 : cephadm [INF] Adjusting osd_memory_target on smithi121 to 4329M 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.579282+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.580605+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.581877+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:08 smithi169 bash[19604]: audit 2024-02-12T15:53:07.588573+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:11.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:10 smithi169 bash[19604]: cluster 2024-02-12T15:53:09.340526+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:10 smithi123 bash[19119]: cluster 2024-02-12T15:53:09.340526+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:11.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:10 smithi121 bash[18489]: cluster 2024-02-12T15:53:09.340526+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:13.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:12 smithi169 bash[19604]: cluster 2024-02-12T15:53:11.341078+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:12 smithi123 bash[19119]: cluster 2024-02-12T15:53:11.341078+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:13.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:12 smithi121 bash[18489]: cluster 2024-02-12T15:53:11.341078+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:14 smithi169 bash[19604]: audit 2024-02-12T15:53:13.794162+0000 mon.a (mon.0) 402 : audit [INF] from='client.24239 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]: dispatch 2024-02-12T15:53:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:14 smithi169 bash[19604]: audit 2024-02-12T15:53:13.795270+0000 mon.b (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.123:0/2780962681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]: dispatch 2024-02-12T15:53:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:14 smithi169 bash[19604]: audit 2024-02-12T15:53:13.802779+0000 mon.a (mon.0) 403 : audit [INF] from='client.24239 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]': finished 2024-02-12T15:53:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:14 smithi169 bash[19604]: cluster 2024-02-12T15:53:13.802958+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-12T15:53:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:14 smithi169 bash[19604]: audit 2024-02-12T15:53:13.803225+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:14 smithi123 bash[19119]: audit 2024-02-12T15:53:13.794162+0000 mon.a (mon.0) 402 : audit [INF] from='client.24239 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]: dispatch 2024-02-12T15:53:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:14 smithi123 bash[19119]: audit 2024-02-12T15:53:13.795270+0000 mon.b (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.123:0/2780962681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]: dispatch 2024-02-12T15:53:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:14 smithi123 bash[19119]: audit 2024-02-12T15:53:13.802779+0000 mon.a (mon.0) 403 : audit [INF] from='client.24239 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]': finished 2024-02-12T15:53:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:14 smithi123 bash[19119]: cluster 2024-02-12T15:53:13.802958+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-12T15:53:14.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:14 smithi123 bash[19119]: audit 2024-02-12T15:53:13.803225+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:13 smithi121 bash[18489]: audit 2024-02-12T15:53:13.794162+0000 mon.a (mon.0) 402 : audit [INF] from='client.24239 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]: dispatch 2024-02-12T15:53:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:13 smithi121 bash[18489]: audit 2024-02-12T15:53:13.795270+0000 mon.b (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.123:0/2780962681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]: dispatch 2024-02-12T15:53:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:13 smithi121 bash[18489]: audit 2024-02-12T15:53:13.802779+0000 mon.a (mon.0) 403 : audit [INF] from='client.24239 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31"}]': finished 2024-02-12T15:53:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:13 smithi121 bash[18489]: cluster 2024-02-12T15:53:13.802958+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-12T15:53:14.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:13 smithi121 bash[18489]: audit 2024-02-12T15:53:13.803225+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:15 smithi169 bash[19604]: cluster 2024-02-12T15:53:13.341633+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:15 smithi169 bash[19604]: audit 2024-02-12T15:53:14.389740+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.123:0/3031718076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:53:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:15 smithi123 bash[19119]: cluster 2024-02-12T15:53:13.341633+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:15 smithi123 bash[19119]: audit 2024-02-12T15:53:14.389740+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.123:0/3031718076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:53:15.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:15 smithi121 bash[18489]: cluster 2024-02-12T15:53:13.341633+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:15.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:15 smithi121 bash[18489]: audit 2024-02-12T15:53:14.389740+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? v1:172.21.15.123:0/3031718076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:53:16.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:16 smithi169 bash[19604]: cluster 2024-02-12T15:53:15.342164+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:16 smithi123 bash[19119]: cluster 2024-02-12T15:53:15.342164+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:16.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:16 smithi121 bash[18489]: cluster 2024-02-12T15:53:15.342164+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:18.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:18 smithi121 bash[18489]: cluster 2024-02-12T15:53:17.342748+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:18.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:18 smithi169 bash[19604]: cluster 2024-02-12T15:53:17.342748+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:18 smithi123 bash[19119]: cluster 2024-02-12T15:53:17.342748+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:19.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:19 smithi169 bash[19604]: cluster 2024-02-12T15:53:18.855723+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:19.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:19 smithi169 bash[19604]: cluster 2024-02-12T15:53:18.855812+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:19 smithi123 bash[19119]: cluster 2024-02-12T15:53:18.855723+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:19 smithi123 bash[19119]: cluster 2024-02-12T15:53:18.855812+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:19.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:19 smithi121 bash[18489]: cluster 2024-02-12T15:53:18.855723+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:19.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:19 smithi121 bash[18489]: cluster 2024-02-12T15:53:18.855812+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:20.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:20 smithi169 bash[19604]: cluster 2024-02-12T15:53:18.590255+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:20.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:20 smithi169 bash[19604]: cluster 2024-02-12T15:53:18.590346+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:20.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:20 smithi169 bash[19604]: cluster 2024-02-12T15:53:19.343304+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:20.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:20 smithi169 bash[19604]: cluster 2024-02-12T15:53:19.414501+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-12T15:53:20.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:20 smithi169 bash[19604]: audit 2024-02-12T15:53:19.414917+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:20 smithi123 bash[19119]: cluster 2024-02-12T15:53:18.590255+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:20 smithi123 bash[19119]: cluster 2024-02-12T15:53:18.590346+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:20 smithi123 bash[19119]: cluster 2024-02-12T15:53:19.343304+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:20 smithi123 bash[19119]: cluster 2024-02-12T15:53:19.414501+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-12T15:53:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:20 smithi123 bash[19119]: audit 2024-02-12T15:53:19.414917+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:20.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:20 smithi121 bash[18489]: cluster 2024-02-12T15:53:18.590255+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:20.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:20 smithi121 bash[18489]: cluster 2024-02-12T15:53:18.590346+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:20.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:20 smithi121 bash[18489]: cluster 2024-02-12T15:53:19.343304+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:20.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:20 smithi121 bash[18489]: cluster 2024-02-12T15:53:19.414501+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-12T15:53:20.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:20 smithi121 bash[18489]: audit 2024-02-12T15:53:19.414917+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:22.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:22 smithi169 bash[19604]: cluster 2024-02-12T15:53:21.343766+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:22 smithi123 bash[19119]: cluster 2024-02-12T15:53:21.343766+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:22.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:22 smithi121 bash[18489]: cluster 2024-02-12T15:53:21.343766+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:24.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:24 smithi123 bash[19119]: cluster 2024-02-12T15:53:23.344289+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:24.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:24 smithi169 bash[19604]: cluster 2024-02-12T15:53:23.344289+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:24.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:24 smithi121 bash[18489]: cluster 2024-02-12T15:53:23.344289+0000 mgr.y (mgr.14146) 196 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:25 smithi169 bash[19604]: audit 2024-02-12T15:53:25.334179+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-12T15:53:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:25 smithi169 bash[19604]: audit 2024-02-12T15:53:25.335885+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:26.305 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:25 smithi123 bash[19119]: audit 2024-02-12T15:53:25.334179+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-12T15:53:26.305 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:25 smithi123 bash[19119]: audit 2024-02-12T15:53:25.335885+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:26.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:25 smithi121 bash[18489]: audit 2024-02-12T15:53:25.334179+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-12T15:53:26.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:25 smithi121 bash[18489]: audit 2024-02-12T15:53:25.335885+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:26 smithi169 bash[19604]: cephadm 2024-02-12T15:53:25.337364+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon osd.4 on smithi123 2024-02-12T15:53:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:26 smithi169 bash[19604]: cluster 2024-02-12T15:53:25.344756+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:26 smithi123 bash[19119]: cephadm 2024-02-12T15:53:25.337364+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon osd.4 on smithi123 2024-02-12T15:53:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:26 smithi123 bash[19119]: cluster 2024-02-12T15:53:25.344756+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:26 smithi121 bash[18489]: cephadm 2024-02-12T15:53:25.337364+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon osd.4 on smithi123 2024-02-12T15:53:27.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:26 smithi121 bash[18489]: cluster 2024-02-12T15:53:25.344756+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:29.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:28 smithi169 bash[19604]: cluster 2024-02-12T15:53:27.345295+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:28 smithi169 bash[19604]: audit 2024-02-12T15:53:28.980663+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:29.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:28 smithi123 bash[19119]: cluster 2024-02-12T15:53:27.345295+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:29.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:28 smithi123 bash[19119]: audit 2024-02-12T15:53:28.980663+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:29.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:28 smithi121 bash[18489]: cluster 2024-02-12T15:53:27.345295+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:29.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:28 smithi121 bash[18489]: audit 2024-02-12T15:53:28.980663+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:30.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:29 smithi123 bash[19119]: audit 2024-02-12T15:53:29.014804+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:53:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:29 smithi169 bash[19604]: audit 2024-02-12T15:53:29.014804+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:53:30.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:29 smithi121 bash[18489]: audit 2024-02-12T15:53:29.014804+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:53:31.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:31 smithi169 bash[19604]: cluster 2024-02-12T15:53:29.345882+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:31.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:31 smithi123 bash[19119]: cluster 2024-02-12T15:53:29.345882+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:31.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:30 smithi121 bash[18489]: cluster 2024-02-12T15:53:29.345882+0000 mgr.y (mgr.14146) 200 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:32.311 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:32 smithi123 bash[19119]: cluster 2024-02-12T15:53:31.346482+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:32.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:32 smithi121 bash[18489]: cluster 2024-02-12T15:53:31.346482+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:32.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:32 smithi169 bash[19604]: cluster 2024-02-12T15:53:31.346482+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:34.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:34 smithi121 bash[18489]: cluster 2024-02-12T15:53:33.347002+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:34.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:34 smithi121 bash[18489]: audit 2024-02-12T15:53:34.283436+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:34.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:34 smithi121 bash[18489]: audit 2024-02-12T15:53:34.284912+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:34.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:34 smithi121 bash[18489]: audit 2024-02-12T15:53:34.286445+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:34.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:34 smithi121 bash[18489]: audit 2024-02-12T15:53:34.295673+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:34.681 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:34 smithi123 bash[19119]: cluster 2024-02-12T15:53:33.347002+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:34.681 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:34 smithi123 bash[19119]: audit 2024-02-12T15:53:34.283436+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:34.681 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:34 smithi123 bash[19119]: audit 2024-02-12T15:53:34.284912+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:34.681 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:34 smithi123 bash[19119]: audit 2024-02-12T15:53:34.286445+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:34.681 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:34 smithi123 bash[19119]: audit 2024-02-12T15:53:34.295673+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:34 smithi169 bash[19604]: cluster 2024-02-12T15:53:33.347002+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:34 smithi169 bash[19604]: audit 2024-02-12T15:53:34.283436+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:34 smithi169 bash[19604]: audit 2024-02-12T15:53:34.284912+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:34 smithi169 bash[19604]: audit 2024-02-12T15:53:34.286445+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:34 smithi169 bash[19604]: audit 2024-02-12T15:53:34.295673+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:36.253 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:35 smithi123 bash[19119]: audit 2024-02-12T15:53:35.298020+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T15:53:36.253 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:35 smithi123 bash[19119]: audit 2024-02-12T15:53:35.299565+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 v1:172.21.15.123:6800/3447700123' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T15:53:36.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:35 smithi169 bash[19604]: audit 2024-02-12T15:53:35.298020+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T15:53:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:35 smithi169 bash[19604]: audit 2024-02-12T15:53:35.299565+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 v1:172.21.15.123:6800/3447700123' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T15:53:36.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:35 smithi121 bash[18489]: audit 2024-02-12T15:53:35.298020+0000 mon.a (mon.0) 416 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T15:53:36.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:35 smithi121 bash[18489]: audit 2024-02-12T15:53:35.299565+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 v1:172.21.15.123:6800/3447700123' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-12T15:53:37.184 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 4 on host 'smithi123' 2024-02-12T15:53:37.261 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:37 smithi123 bash[19119]: cluster 2024-02-12T15:53:35.347548+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:37.261 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:37 smithi123 bash[19119]: audit 2024-02-12T15:53:35.994549+0000 mon.a (mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-12T15:53:37.261 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:37 smithi123 bash[19119]: cluster 2024-02-12T15:53:35.994775+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-12T15:53:37.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:37 smithi123 bash[19119]: audit 2024-02-12T15:53:35.994976+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:37.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:37 smithi123 bash[19119]: audit 2024-02-12T15:53:35.996541+0000 mon.a (mon.0) 420 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:53:37.262 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:37 smithi123 bash[19119]: audit 2024-02-12T15:53:35.998080+0000 mon.b (mon.2) 11 : audit [INF] from='osd.4 v1:172.21.15.123:6800/3447700123' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:53:37.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:37 smithi169 bash[19604]: cluster 2024-02-12T15:53:35.347548+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:37.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:37 smithi169 bash[19604]: audit 2024-02-12T15:53:35.994549+0000 mon.a (mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-12T15:53:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:37 smithi169 bash[19604]: cluster 2024-02-12T15:53:35.994775+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-12T15:53:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:37 smithi169 bash[19604]: audit 2024-02-12T15:53:35.994976+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:37 smithi169 bash[19604]: audit 2024-02-12T15:53:35.996541+0000 mon.a (mon.0) 420 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:53:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:37 smithi169 bash[19604]: audit 2024-02-12T15:53:35.998080+0000 mon.b (mon.2) 11 : audit [INF] from='osd.4 v1:172.21.15.123:6800/3447700123' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:53:37.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:36 smithi121 bash[18489]: cluster 2024-02-12T15:53:35.347548+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:37.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:36 smithi121 bash[18489]: audit 2024-02-12T15:53:35.994549+0000 mon.a (mon.0) 417 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-12T15:53:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:36 smithi121 bash[18489]: cluster 2024-02-12T15:53:35.994775+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-12T15:53:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:37 smithi121 bash[18489]: audit 2024-02-12T15:53:35.994976+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:37 smithi121 bash[18489]: audit 2024-02-12T15:53:35.996541+0000 mon.a (mon.0) 420 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:53:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:37 smithi121 bash[18489]: audit 2024-02-12T15:53:35.998080+0000 mon.b (mon.2) 11 : audit [INF] from='osd.4 v1:172.21.15.123:6800/3447700123' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:53:37.800 DEBUG:teuthology.orchestra.run.smithi123:osd.4> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.4.service 2024-02-12T15:53:37.802 INFO:tasks.cephadm:Deploying osd.5 on smithi123 with /dev/vg_nvme/lv_3... 2024-02-12T15:53:37.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-12T15:53:37.921 INFO:journalctl@ceph.osd.4.smithi123.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:53:38.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:38 smithi123 bash[19119]: audit 2024-02-12T15:53:36.997016+0000 mon.a (mon.0) 421 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:53:38.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:38 smithi123 bash[19119]: cluster 2024-02-12T15:53:36.997171+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-02-12T15:53:38.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:38 smithi123 bash[19119]: audit 2024-02-12T15:53:36.997793+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:38.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:38 smithi123 bash[19119]: audit 2024-02-12T15:53:37.005191+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:38.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:38 smithi123 bash[19119]: audit 2024-02-12T15:53:37.172850+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:53:38.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:38 smithi123 bash[19119]: audit 2024-02-12T15:53:37.178664+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:38 smithi169 bash[19604]: audit 2024-02-12T15:53:36.997016+0000 mon.a (mon.0) 421 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:53:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:38 smithi169 bash[19604]: cluster 2024-02-12T15:53:36.997171+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-02-12T15:53:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:38 smithi169 bash[19604]: audit 2024-02-12T15:53:36.997793+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:38 smithi169 bash[19604]: audit 2024-02-12T15:53:37.005191+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:38 smithi169 bash[19604]: audit 2024-02-12T15:53:37.172850+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:53:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:38 smithi169 bash[19604]: audit 2024-02-12T15:53:37.178664+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:38.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:38 smithi121 bash[18489]: audit 2024-02-12T15:53:36.997016+0000 mon.a (mon.0) 421 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:53:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:38 smithi121 bash[18489]: cluster 2024-02-12T15:53:36.997171+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-02-12T15:53:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:38 smithi121 bash[18489]: audit 2024-02-12T15:53:36.997793+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:38 smithi121 bash[18489]: audit 2024-02-12T15:53:37.005191+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:38 smithi121 bash[18489]: audit 2024-02-12T15:53:37.172850+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:53:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:38 smithi121 bash[18489]: audit 2024-02-12T15:53:37.178664+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:39.066 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:39 smithi123 bash[19119]: cluster 2024-02-12T15:53:36.257454+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:39.066 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:39 smithi123 bash[19119]: cluster 2024-02-12T15:53:36.257566+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:39.067 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:39 smithi123 bash[19119]: cluster 2024-02-12T15:53:37.348078+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:39.067 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:39 smithi123 bash[19119]: audit 2024-02-12T15:53:38.008232+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:39.067 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:39 smithi123 bash[19119]: cluster 2024-02-12T15:53:38.013006+0000 mon.a (mon.0) 428 : cluster [INF] osd.4 v1:172.21.15.123:6800/3447700123 boot 2024-02-12T15:53:39.067 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:39 smithi123 bash[19119]: cluster 2024-02-12T15:53:38.013051+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-12T15:53:39.067 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:39 smithi123 bash[19119]: audit 2024-02-12T15:53:38.013169+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:39 smithi169 bash[19604]: cluster 2024-02-12T15:53:36.257454+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:39 smithi169 bash[19604]: cluster 2024-02-12T15:53:36.257566+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:39 smithi169 bash[19604]: cluster 2024-02-12T15:53:37.348078+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:39 smithi169 bash[19604]: audit 2024-02-12T15:53:38.008232+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:39 smithi169 bash[19604]: cluster 2024-02-12T15:53:38.013006+0000 mon.a (mon.0) 428 : cluster [INF] osd.4 v1:172.21.15.123:6800/3447700123 boot 2024-02-12T15:53:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:39 smithi169 bash[19604]: cluster 2024-02-12T15:53:38.013051+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-12T15:53:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:39 smithi169 bash[19604]: audit 2024-02-12T15:53:38.013169+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:39.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:39 smithi121 bash[18489]: cluster 2024-02-12T15:53:36.257454+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:53:39.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:39 smithi121 bash[18489]: cluster 2024-02-12T15:53:36.257566+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:53:39.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:39 smithi121 bash[18489]: cluster 2024-02-12T15:53:37.348078+0000 mgr.y (mgr.14146) 204 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-12T15:53:39.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:39 smithi121 bash[18489]: audit 2024-02-12T15:53:38.008232+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:39.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:39 smithi121 bash[18489]: cluster 2024-02-12T15:53:38.013006+0000 mon.a (mon.0) 428 : cluster [INF] osd.4 v1:172.21.15.123:6800/3447700123 boot 2024-02-12T15:53:39.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:39 smithi121 bash[18489]: cluster 2024-02-12T15:53:38.013051+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-12T15:53:39.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:39 smithi121 bash[18489]: audit 2024-02-12T15:53:38.013169+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-12T15:53:40.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:40 smithi169 bash[19604]: cluster 2024-02-12T15:53:39.183925+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-12T15:53:40.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:40 smithi169 bash[19604]: cluster 2024-02-12T15:53:39.348581+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:40.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:40 smithi123 bash[19119]: cluster 2024-02-12T15:53:39.183925+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-12T15:53:40.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:40 smithi123 bash[19119]: cluster 2024-02-12T15:53:39.348581+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:40.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:40 smithi121 bash[18489]: cluster 2024-02-12T15:53:39.183925+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-12T15:53:40.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:40 smithi121 bash[18489]: cluster 2024-02-12T15:53:39.348581+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:40.747 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-12T15:53:40.747 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-12T15:53:40.747 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-02-12T15:53:40.747 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-02-12T15:53:40.747 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0188404 s, 557 MB/s 2024-02-12T15:53:40.747 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-02-12T15:53:41.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_3 2024-02-12T15:53:42.658 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:42 smithi121 bash[18489]: cluster 2024-02-12T15:53:41.349135+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:42.684 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:42 smithi123 bash[19119]: cluster 2024-02-12T15:53:41.349135+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:42.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:42 smithi169 bash[19604]: cluster 2024-02-12T15:53:41.349135+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:44.766 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:44 smithi123 bash[19119]: cluster 2024-02-12T15:53:43.349668+0000 mgr.y (mgr.14146) 207 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:44.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:44 smithi169 bash[19604]: cluster 2024-02-12T15:53:43.349668+0000 mgr.y (mgr.14146) 207 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:44.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:44 smithi121 bash[18489]: cluster 2024-02-12T15:53:43.349668+0000 mgr.y (mgr.14146) 207 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:46.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:45 smithi169 bash[19604]: audit 2024-02-12T15:53:45.194335+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:53:46.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:45 smithi169 bash[19604]: audit 2024-02-12T15:53:45.200194+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:53:46.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:45 smithi169 bash[19604]: audit 2024-02-12T15:53:45.201626+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:45 smithi123 bash[19119]: audit 2024-02-12T15:53:45.194335+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:53:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:45 smithi123 bash[19119]: audit 2024-02-12T15:53:45.200194+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:53:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:45 smithi123 bash[19119]: audit 2024-02-12T15:53:45.201626+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:46.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:45 smithi121 bash[18489]: audit 2024-02-12T15:53:45.194335+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:53:46.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:45 smithi121 bash[18489]: audit 2024-02-12T15:53:45.200194+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:53:46.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:45 smithi121 bash[18489]: audit 2024-02-12T15:53:45.201626+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: audit 2024-02-12T15:53:45.187509+0000 mgr.y (mgr.14146) 208 : audit [DBG] from='client.14346 v1:172.21.15.123:0/2657397210' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: cluster 2024-02-12T15:53:45.350205+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: audit 2024-02-12T15:53:46.430680+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: audit 2024-02-12T15:53:46.432685+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: audit 2024-02-12T15:53:46.442303+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: audit 2024-02-12T15:53:46.444190+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: audit 2024-02-12T15:53:46.445546+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:47.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:46 smithi169 bash[19604]: audit 2024-02-12T15:53:46.452895+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: audit 2024-02-12T15:53:45.187509+0000 mgr.y (mgr.14146) 208 : audit [DBG] from='client.14346 v1:172.21.15.123:0/2657397210' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: cluster 2024-02-12T15:53:45.350205+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: audit 2024-02-12T15:53:46.430680+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: audit 2024-02-12T15:53:46.432685+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: audit 2024-02-12T15:53:46.442303+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: audit 2024-02-12T15:53:46.444190+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: audit 2024-02-12T15:53:46.445546+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:47.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:46 smithi123 bash[19119]: audit 2024-02-12T15:53:46.452895+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: audit 2024-02-12T15:53:45.187509+0000 mgr.y (mgr.14146) 208 : audit [DBG] from='client.14346 v1:172.21.15.123:0/2657397210' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:53:47.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: cluster 2024-02-12T15:53:45.350205+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: audit 2024-02-12T15:53:46.430680+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: audit 2024-02-12T15:53:46.432685+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:53:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: audit 2024-02-12T15:53:46.442303+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: audit 2024-02-12T15:53:46.444190+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:53:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: audit 2024-02-12T15:53:46.445546+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:53:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:46 smithi121 bash[18489]: audit 2024-02-12T15:53:46.452895+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:53:48.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:47 smithi169 bash[19604]: cephadm 2024-02-12T15:53:46.420372+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:53:48.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:47 smithi169 bash[19604]: cephadm 2024-02-12T15:53:46.433916+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 17318M 2024-02-12T15:53:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:48 smithi123 bash[19119]: cephadm 2024-02-12T15:53:46.420372+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:53:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:48 smithi123 bash[19119]: cephadm 2024-02-12T15:53:46.433916+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 17318M 2024-02-12T15:53:48.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:47 smithi121 bash[18489]: cephadm 2024-02-12T15:53:46.420372+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:53:48.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:47 smithi121 bash[18489]: cephadm 2024-02-12T15:53:46.433916+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 17318M 2024-02-12T15:53:49.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:49 smithi169 bash[19604]: cluster 2024-02-12T15:53:47.350785+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:49.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:49 smithi123 bash[19119]: cluster 2024-02-12T15:53:47.350785+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:49.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:48 smithi121 bash[18489]: cluster 2024-02-12T15:53:47.350785+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:51 smithi169 bash[19604]: cluster 2024-02-12T15:53:49.351440+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:51 smithi123 bash[19119]: cluster 2024-02-12T15:53:49.351440+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:51.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:51 smithi121 bash[18489]: cluster 2024-02-12T15:53:49.351440+0000 mgr.y (mgr.14146) 213 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:52 smithi123 bash[19119]: cluster 2024-02-12T15:53:51.352003+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:52.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:52 smithi121 bash[18489]: cluster 2024-02-12T15:53:51.352003+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:52.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:52 smithi169 bash[19604]: cluster 2024-02-12T15:53:51.352003+0000 mgr.y (mgr.14146) 214 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:53 smithi123 bash[19119]: audit 2024-02-12T15:53:52.975711+0000 mon.a (mon.0) 441 : audit [INF] from='client.24266 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]: dispatch 2024-02-12T15:53:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:53 smithi123 bash[19119]: audit 2024-02-12T15:53:52.977237+0000 mon.b (mon.2) 12 : audit [INF] from='client.? v1:172.21.15.123:0/208300237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]: dispatch 2024-02-12T15:53:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:53 smithi123 bash[19119]: audit 2024-02-12T15:53:52.984403+0000 mon.a (mon.0) 442 : audit [INF] from='client.24266 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]': finished 2024-02-12T15:53:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:53 smithi123 bash[19119]: cluster 2024-02-12T15:53:52.984596+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-12T15:53:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:53 smithi123 bash[19119]: audit 2024-02-12T15:53:52.984865+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:53:53.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:53 smithi121 bash[18489]: audit 2024-02-12T15:53:52.975711+0000 mon.a (mon.0) 441 : audit [INF] from='client.24266 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]: dispatch 2024-02-12T15:53:53.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:53 smithi121 bash[18489]: audit 2024-02-12T15:53:52.977237+0000 mon.b (mon.2) 12 : audit [INF] from='client.? v1:172.21.15.123:0/208300237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]: dispatch 2024-02-12T15:53:53.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:53 smithi121 bash[18489]: audit 2024-02-12T15:53:52.984403+0000 mon.a (mon.0) 442 : audit [INF] from='client.24266 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]': finished 2024-02-12T15:53:53.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:53 smithi121 bash[18489]: cluster 2024-02-12T15:53:52.984596+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-12T15:53:53.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:53 smithi121 bash[18489]: audit 2024-02-12T15:53:52.984865+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:53:53.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:53 smithi169 bash[19604]: audit 2024-02-12T15:53:52.975711+0000 mon.a (mon.0) 441 : audit [INF] from='client.24266 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]: dispatch 2024-02-12T15:53:53.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:53 smithi169 bash[19604]: audit 2024-02-12T15:53:52.977237+0000 mon.b (mon.2) 12 : audit [INF] from='client.? v1:172.21.15.123:0/208300237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]: dispatch 2024-02-12T15:53:53.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:53 smithi169 bash[19604]: audit 2024-02-12T15:53:52.984403+0000 mon.a (mon.0) 442 : audit [INF] from='client.24266 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602"}]': finished 2024-02-12T15:53:53.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:53 smithi169 bash[19604]: cluster 2024-02-12T15:53:52.984596+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-12T15:53:53.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:53 smithi169 bash[19604]: audit 2024-02-12T15:53:52.984865+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:53:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:54 smithi123 bash[19119]: cluster 2024-02-12T15:53:53.352523+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:54 smithi123 bash[19119]: audit 2024-02-12T15:53:53.579556+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.123:0/4042336817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:53:54.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:54 smithi121 bash[18489]: cluster 2024-02-12T15:53:53.352523+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:54.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:54 smithi121 bash[18489]: audit 2024-02-12T15:53:53.579556+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.123:0/4042336817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:53:54.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:54 smithi169 bash[19604]: cluster 2024-02-12T15:53:53.352523+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:54.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:54 smithi169 bash[19604]: audit 2024-02-12T15:53:53.579556+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.123:0/4042336817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:53:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:56 smithi169 bash[19604]: cluster 2024-02-12T15:53:55.353059+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:57.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:56 smithi123 bash[19119]: cluster 2024-02-12T15:53:55.353059+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:57.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:56 smithi121 bash[18489]: cluster 2024-02-12T15:53:55.353059+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:53:58 smithi169 bash[19604]: cluster 2024-02-12T15:53:57.353610+0000 mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:59.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:53:58 smithi123 bash[19119]: cluster 2024-02-12T15:53:57.353610+0000 mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:53:59.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:53:58 smithi121 bash[18489]: cluster 2024-02-12T15:53:57.353610+0000 mgr.y (mgr.14146) 217 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:01 smithi169 bash[19604]: cluster 2024-02-12T15:53:59.354218+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:01 smithi123 bash[19119]: cluster 2024-02-12T15:53:59.354218+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:01.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:01 smithi121 bash[18489]: cluster 2024-02-12T15:53:59.354218+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:03.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:03 smithi169 bash[19604]: cluster 2024-02-12T15:54:01.354786+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:03 smithi123 bash[19119]: cluster 2024-02-12T15:54:01.354786+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:03.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:03 smithi121 bash[18489]: cluster 2024-02-12T15:54:01.354786+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:04 smithi123 bash[19119]: cluster 2024-02-12T15:54:03.355345+0000 mgr.y (mgr.14146) 220 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:04 smithi123 bash[19119]: cluster 2024-02-12T15:54:04.015544+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-12T15:54:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:04 smithi123 bash[19119]: audit 2024-02-12T15:54:04.015719+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:04.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:04 smithi121 bash[18489]: cluster 2024-02-12T15:54:03.355345+0000 mgr.y (mgr.14146) 220 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:04.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:04 smithi121 bash[18489]: cluster 2024-02-12T15:54:04.015544+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-12T15:54:04.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:04 smithi121 bash[18489]: audit 2024-02-12T15:54:04.015719+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:04.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:04 smithi169 bash[19604]: cluster 2024-02-12T15:54:03.355345+0000 mgr.y (mgr.14146) 220 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:04.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:04 smithi169 bash[19604]: cluster 2024-02-12T15:54:04.015544+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-12T15:54:04.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:04 smithi169 bash[19604]: audit 2024-02-12T15:54:04.015719+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:05 smithi123 bash[19119]: cluster 2024-02-12T15:54:03.785394+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:05 smithi123 bash[19119]: cluster 2024-02-12T15:54:03.785486+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:05 smithi123 bash[19119]: audit 2024-02-12T15:54:04.555238+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-12T15:54:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:05 smithi123 bash[19119]: audit 2024-02-12T15:54:04.556933+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:05 smithi123 bash[19119]: cephadm 2024-02-12T15:54:04.558431+0000 mgr.y (mgr.14146) 221 : cephadm [INF] Deploying daemon osd.5 on smithi123 2024-02-12T15:54:05.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:05 smithi121 bash[18489]: cluster 2024-02-12T15:54:03.785394+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:05 smithi121 bash[18489]: cluster 2024-02-12T15:54:03.785486+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:05 smithi121 bash[18489]: audit 2024-02-12T15:54:04.555238+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-12T15:54:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:05 smithi121 bash[18489]: audit 2024-02-12T15:54:04.556933+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:05 smithi121 bash[18489]: cephadm 2024-02-12T15:54:04.558431+0000 mgr.y (mgr.14146) 221 : cephadm [INF] Deploying daemon osd.5 on smithi123 2024-02-12T15:54:05.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:05 smithi169 bash[19604]: cluster 2024-02-12T15:54:03.785394+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:05.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:05 smithi169 bash[19604]: cluster 2024-02-12T15:54:03.785486+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:05.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:05 smithi169 bash[19604]: audit 2024-02-12T15:54:04.555238+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-12T15:54:05.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:05 smithi169 bash[19604]: audit 2024-02-12T15:54:04.556933+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:05.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:05 smithi169 bash[19604]: cephadm 2024-02-12T15:54:04.558431+0000 mgr.y (mgr.14146) 221 : cephadm [INF] Deploying daemon osd.5 on smithi123 2024-02-12T15:54:06.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:06 smithi123 bash[19119]: cluster 2024-02-12T15:54:05.355865+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:06.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:06 smithi121 bash[18489]: cluster 2024-02-12T15:54:05.355865+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:06.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:06 smithi169 bash[19604]: cluster 2024-02-12T15:54:05.355865+0000 mgr.y (mgr.14146) 222 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:08 smithi123 bash[19119]: cluster 2024-02-12T15:54:07.356448+0000 mgr.y (mgr.14146) 223 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:08 smithi123 bash[19119]: audit 2024-02-12T15:54:08.231978+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:08 smithi123 bash[19119]: audit 2024-02-12T15:54:08.239073+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:08 smithi169 bash[19604]: cluster 2024-02-12T15:54:07.356448+0000 mgr.y (mgr.14146) 223 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:08 smithi169 bash[19604]: audit 2024-02-12T15:54:08.231978+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:08 smithi169 bash[19604]: audit 2024-02-12T15:54:08.239073+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:08.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:08 smithi121 bash[18489]: cluster 2024-02-12T15:54:07.356448+0000 mgr.y (mgr.14146) 223 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:08 smithi121 bash[18489]: audit 2024-02-12T15:54:08.231978+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:08 smithi121 bash[18489]: audit 2024-02-12T15:54:08.239073+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:10 smithi169 bash[19604]: cluster 2024-02-12T15:54:08.824499+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:10 smithi169 bash[19604]: cluster 2024-02-12T15:54:08.824594+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:10 smithi169 bash[19604]: cluster 2024-02-12T15:54:09.356996+0000 mgr.y (mgr.14146) 224 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:10 smithi169 bash[19604]: cluster 2024-02-12T15:54:09.424593+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-12T15:54:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:10 smithi169 bash[19604]: audit 2024-02-12T15:54:09.424992+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:10 smithi123 bash[19119]: cluster 2024-02-12T15:54:08.824499+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:10 smithi123 bash[19119]: cluster 2024-02-12T15:54:08.824594+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:10 smithi123 bash[19119]: cluster 2024-02-12T15:54:09.356996+0000 mgr.y (mgr.14146) 224 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:10 smithi123 bash[19119]: cluster 2024-02-12T15:54:09.424593+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-12T15:54:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:10 smithi123 bash[19119]: audit 2024-02-12T15:54:09.424992+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:10.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:10 smithi121 bash[18489]: cluster 2024-02-12T15:54:08.824499+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:10.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:10 smithi121 bash[18489]: cluster 2024-02-12T15:54:08.824594+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:10.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:10 smithi121 bash[18489]: cluster 2024-02-12T15:54:09.356996+0000 mgr.y (mgr.14146) 224 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:10.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:10 smithi121 bash[18489]: cluster 2024-02-12T15:54:09.424593+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-12T15:54:10.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:10 smithi121 bash[18489]: audit 2024-02-12T15:54:09.424992+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:11.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:11 smithi169 bash[19604]: cluster 2024-02-12T15:54:10.123923+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:54:11.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:11 smithi169 bash[19604]: cluster 2024-02-12T15:54:10.127362+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:54:11.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:11 smithi123 bash[19119]: cluster 2024-02-12T15:54:10.123923+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:54:11.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:11 smithi123 bash[19119]: cluster 2024-02-12T15:54:10.127362+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:54:11.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:11 smithi121 bash[18489]: cluster 2024-02-12T15:54:10.123923+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:54:11.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:11 smithi121 bash[18489]: cluster 2024-02-12T15:54:10.127362+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:54:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:12 smithi169 bash[19604]: cluster 2024-02-12T15:54:11.357560+0000 mgr.y (mgr.14146) 225 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:12 smithi123 bash[19119]: cluster 2024-02-12T15:54:11.357560+0000 mgr.y (mgr.14146) 225 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:12.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:12 smithi121 bash[18489]: cluster 2024-02-12T15:54:11.357560+0000 mgr.y (mgr.14146) 225 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:14 smithi123 bash[19119]: cluster 2024-02-12T15:54:13.358119+0000 mgr.y (mgr.14146) 226 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:14 smithi123 bash[19119]: audit 2024-02-12T15:54:13.564629+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:14 smithi123 bash[19119]: audit 2024-02-12T15:54:13.856004+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:14 smithi123 bash[19119]: audit 2024-02-12T15:54:13.857925+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:14 smithi123 bash[19119]: audit 2024-02-12T15:54:13.868039+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:14 smithi123 bash[19119]: audit 2024-02-12T15:54:14.511668+0000 mon.a (mon.0) 457 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T15:54:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:14 smithi123 bash[19119]: audit 2024-02-12T15:54:14.513452+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 v1:172.21.15.123:6804/961317306' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T15:54:14.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:14 smithi121 bash[18489]: cluster 2024-02-12T15:54:13.358119+0000 mgr.y (mgr.14146) 226 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:14 smithi121 bash[18489]: audit 2024-02-12T15:54:13.564629+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:14 smithi121 bash[18489]: audit 2024-02-12T15:54:13.856004+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:14 smithi121 bash[18489]: audit 2024-02-12T15:54:13.857925+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:14 smithi121 bash[18489]: audit 2024-02-12T15:54:13.868039+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:14 smithi121 bash[18489]: audit 2024-02-12T15:54:14.511668+0000 mon.a (mon.0) 457 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T15:54:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:14 smithi121 bash[18489]: audit 2024-02-12T15:54:14.513452+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 v1:172.21.15.123:6804/961317306' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T15:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:14 smithi169 bash[19604]: cluster 2024-02-12T15:54:13.358119+0000 mgr.y (mgr.14146) 226 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:14 smithi169 bash[19604]: audit 2024-02-12T15:54:13.564629+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:14 smithi169 bash[19604]: audit 2024-02-12T15:54:13.856004+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:14 smithi169 bash[19604]: audit 2024-02-12T15:54:13.857925+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:14 smithi169 bash[19604]: audit 2024-02-12T15:54:13.868039+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:14 smithi169 bash[19604]: audit 2024-02-12T15:54:14.511668+0000 mon.a (mon.0) 457 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T15:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:14 smithi169 bash[19604]: audit 2024-02-12T15:54:14.513452+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 v1:172.21.15.123:6804/961317306' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-12T15:54:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:15 smithi123 bash[19119]: audit 2024-02-12T15:54:14.870880+0000 mon.a (mon.0) 458 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-12T15:54:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:15 smithi123 bash[19119]: cluster 2024-02-12T15:54:14.871084+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-12T15:54:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:15 smithi123 bash[19119]: audit 2024-02-12T15:54:14.871377+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:15 smithi123 bash[19119]: audit 2024-02-12T15:54:14.872913+0000 mon.a (mon.0) 461 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:15 smithi123 bash[19119]: audit 2024-02-12T15:54:14.874664+0000 mon.b (mon.2) 15 : audit [INF] from='osd.5 v1:172.21.15.123:6804/961317306' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:16.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:15 smithi121 bash[18489]: audit 2024-02-12T15:54:14.870880+0000 mon.a (mon.0) 458 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-12T15:54:16.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:15 smithi121 bash[18489]: cluster 2024-02-12T15:54:14.871084+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-12T15:54:16.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:15 smithi121 bash[18489]: audit 2024-02-12T15:54:14.871377+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:16.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:15 smithi121 bash[18489]: audit 2024-02-12T15:54:14.872913+0000 mon.a (mon.0) 461 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:16.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:15 smithi121 bash[18489]: audit 2024-02-12T15:54:14.874664+0000 mon.b (mon.2) 15 : audit [INF] from='osd.5 v1:172.21.15.123:6804/961317306' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:15 smithi169 bash[19604]: audit 2024-02-12T15:54:14.870880+0000 mon.a (mon.0) 458 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-12T15:54:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:15 smithi169 bash[19604]: cluster 2024-02-12T15:54:14.871084+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-12T15:54:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:15 smithi169 bash[19604]: audit 2024-02-12T15:54:14.871377+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:15 smithi169 bash[19604]: audit 2024-02-12T15:54:14.872913+0000 mon.a (mon.0) 461 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:15 smithi169 bash[19604]: audit 2024-02-12T15:54:14.874664+0000 mon.b (mon.2) 15 : audit [INF] from='osd.5 v1:172.21.15.123:6804/961317306' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:16.484 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 5 on host 'smithi123' 2024-02-12T15:54:17.067 DEBUG:teuthology.orchestra.run.smithi123:osd.5> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.5.service 2024-02-12T15:54:17.070 INFO:tasks.cephadm:Deploying osd.6 on smithi123 with /dev/vg_nvme/lv_2... 2024-02-12T15:54:17.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-12T15:54:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:16 smithi123 bash[19119]: cluster 2024-02-12T15:54:15.358682+0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:16 smithi123 bash[19119]: audit 2024-02-12T15:54:15.874519+0000 mon.a (mon.0) 462 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:54:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:16 smithi123 bash[19119]: cluster 2024-02-12T15:54:15.874790+0000 mon.a (mon.0) 463 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-02-12T15:54:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:16 smithi123 bash[19119]: audit 2024-02-12T15:54:15.875449+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:16 smithi123 bash[19119]: audit 2024-02-12T15:54:15.883126+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:16 smithi123 bash[19119]: audit 2024-02-12T15:54:16.479139+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:16 smithi123 bash[19119]: audit 2024-02-12T15:54:16.494880+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:17.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:16 smithi121 bash[18489]: cluster 2024-02-12T15:54:15.358682+0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:17.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:16 smithi121 bash[18489]: audit 2024-02-12T15:54:15.874519+0000 mon.a (mon.0) 462 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:54:17.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:16 smithi121 bash[18489]: cluster 2024-02-12T15:54:15.874790+0000 mon.a (mon.0) 463 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-02-12T15:54:17.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:16 smithi121 bash[18489]: audit 2024-02-12T15:54:15.875449+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:17.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:16 smithi121 bash[18489]: audit 2024-02-12T15:54:15.883126+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:17.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:16 smithi121 bash[18489]: audit 2024-02-12T15:54:16.479139+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:17.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:16 smithi121 bash[18489]: audit 2024-02-12T15:54:16.494880+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:17.192 INFO:journalctl@ceph.osd.5.smithi123.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:54:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:16 smithi169 bash[19604]: cluster 2024-02-12T15:54:15.358682+0000 mgr.y (mgr.14146) 227 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-12T15:54:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:16 smithi169 bash[19604]: audit 2024-02-12T15:54:15.874519+0000 mon.a (mon.0) 462 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:54:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:16 smithi169 bash[19604]: cluster 2024-02-12T15:54:15.874790+0000 mon.a (mon.0) 463 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-02-12T15:54:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:16 smithi169 bash[19604]: audit 2024-02-12T15:54:15.875449+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:16 smithi169 bash[19604]: audit 2024-02-12T15:54:15.883126+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:16 smithi169 bash[19604]: audit 2024-02-12T15:54:16.479139+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:16 smithi169 bash[19604]: audit 2024-02-12T15:54:16.494880+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:18.156 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:17 smithi123 bash[19119]: cluster 2024-02-12T15:54:15.537678+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:18.156 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:17 smithi123 bash[19119]: cluster 2024-02-12T15:54:15.537796+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:18.156 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:17 smithi123 bash[19119]: cluster 2024-02-12T15:54:16.877778+0000 mon.a (mon.0) 468 : cluster [INF] osd.5 v1:172.21.15.123:6804/961317306 boot 2024-02-12T15:54:18.156 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:17 smithi123 bash[19119]: cluster 2024-02-12T15:54:16.877833+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-12T15:54:18.156 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:17 smithi123 bash[19119]: audit 2024-02-12T15:54:16.878015+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:18.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:17 smithi121 bash[18489]: cluster 2024-02-12T15:54:15.537678+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:18.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:17 smithi121 bash[18489]: cluster 2024-02-12T15:54:15.537796+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:18.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:17 smithi121 bash[18489]: cluster 2024-02-12T15:54:16.877778+0000 mon.a (mon.0) 468 : cluster [INF] osd.5 v1:172.21.15.123:6804/961317306 boot 2024-02-12T15:54:18.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:17 smithi121 bash[18489]: cluster 2024-02-12T15:54:16.877833+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-12T15:54:18.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:17 smithi121 bash[18489]: audit 2024-02-12T15:54:16.878015+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:17 smithi169 bash[19604]: cluster 2024-02-12T15:54:15.537678+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:17 smithi169 bash[19604]: cluster 2024-02-12T15:54:15.537796+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:17 smithi169 bash[19604]: cluster 2024-02-12T15:54:16.877778+0000 mon.a (mon.0) 468 : cluster [INF] osd.5 v1:172.21.15.123:6804/961317306 boot 2024-02-12T15:54:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:17 smithi169 bash[19604]: cluster 2024-02-12T15:54:16.877833+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-12T15:54:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:17 smithi169 bash[19604]: audit 2024-02-12T15:54:16.878015+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-12T15:54:19.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:18 smithi121 bash[18489]: cluster 2024-02-12T15:54:17.359312+0000 mgr.y (mgr.14146) 228 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:19.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:18 smithi121 bash[18489]: cluster 2024-02-12T15:54:17.880205+0000 mon.a (mon.0) 471 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-12T15:54:19.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:18 smithi169 bash[19604]: cluster 2024-02-12T15:54:17.359312+0000 mgr.y (mgr.14146) 228 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:19.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:18 smithi169 bash[19604]: cluster 2024-02-12T15:54:17.880205+0000 mon.a (mon.0) 471 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-12T15:54:19.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:18 smithi123 bash[19119]: cluster 2024-02-12T15:54:17.359312+0000 mgr.y (mgr.14146) 228 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:19.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:18 smithi123 bash[19119]: cluster 2024-02-12T15:54:17.880205+0000 mon.a (mon.0) 471 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-12T15:54:20.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:19 smithi123 bash[19119]: cluster 2024-02-12T15:54:18.899889+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-02-12T15:54:20.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:19 smithi121 bash[18489]: cluster 2024-02-12T15:54:18.899889+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-02-12T15:54:20.281 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-12T15:54:20.281 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-12T15:54:20.281 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-02-12T15:54:20.281 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-02-12T15:54:20.281 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0189463 s, 553 MB/s 2024-02-12T15:54:20.281 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 2024-02-12T15:54:20.281 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-02-12T15:54:20.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:19 smithi169 bash[19604]: cluster 2024-02-12T15:54:18.899889+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-02-12T15:54:20.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_2 2024-02-12T15:54:21.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:20 smithi123 bash[19119]: cluster 2024-02-12T15:54:19.359838+0000 mgr.y (mgr.14146) 229 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:21.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:20 smithi169 bash[19604]: cluster 2024-02-12T15:54:19.359838+0000 mgr.y (mgr.14146) 229 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:20 smithi121 bash[18489]: cluster 2024-02-12T15:54:19.359838+0000 mgr.y (mgr.14146) 229 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:23.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:23 smithi169 bash[19604]: cluster 2024-02-12T15:54:21.360382+0000 mgr.y (mgr.14146) 230 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:23.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:23 smithi123 bash[19119]: cluster 2024-02-12T15:54:21.360382+0000 mgr.y (mgr.14146) 230 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:23.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:22 smithi121 bash[18489]: cluster 2024-02-12T15:54:21.360382+0000 mgr.y (mgr.14146) 230 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:25 smithi169 bash[19604]: cluster 2024-02-12T15:54:23.360929+0000 mgr.y (mgr.14146) 231 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:25 smithi169 bash[19604]: cluster 2024-02-12T15:54:24.385336+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:25 smithi169 bash[19604]: cluster 2024-02-12T15:54:24.385424+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:25.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:25 smithi123 bash[19119]: cluster 2024-02-12T15:54:23.360929+0000 mgr.y (mgr.14146) 231 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:25.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:25 smithi123 bash[19119]: cluster 2024-02-12T15:54:24.385336+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:25.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:25 smithi123 bash[19119]: cluster 2024-02-12T15:54:24.385424+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:25.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:25 smithi121 bash[18489]: cluster 2024-02-12T15:54:23.360929+0000 mgr.y (mgr.14146) 231 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:25.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:25 smithi121 bash[18489]: cluster 2024-02-12T15:54:24.385336+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:25.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:25 smithi121 bash[18489]: cluster 2024-02-12T15:54:24.385424+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:26 smithi169 bash[19604]: cluster 2024-02-12T15:54:25.004872+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-02-12T15:54:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:26 smithi169 bash[19604]: audit 2024-02-12T15:54:25.072712+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:54:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:26 smithi169 bash[19604]: audit 2024-02-12T15:54:25.078662+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:54:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:26 smithi169 bash[19604]: audit 2024-02-12T15:54:25.079920+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:26 smithi123 bash[19119]: cluster 2024-02-12T15:54:25.004872+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-02-12T15:54:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:26 smithi123 bash[19119]: audit 2024-02-12T15:54:25.072712+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:54:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:26 smithi123 bash[19119]: audit 2024-02-12T15:54:25.078662+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:54:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:26 smithi123 bash[19119]: audit 2024-02-12T15:54:25.079920+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:26.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:26 smithi121 bash[18489]: cluster 2024-02-12T15:54:25.004872+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-02-12T15:54:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:26 smithi121 bash[18489]: audit 2024-02-12T15:54:25.072712+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:54:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:26 smithi121 bash[18489]: audit 2024-02-12T15:54:25.078662+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:54:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:26 smithi121 bash[18489]: audit 2024-02-12T15:54:25.079920+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:25.065823+0000 mgr.y (mgr.14146) 232 : audit [DBG] from='client.24287 v1:172.21.15.123:0/202267911' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:54:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: cluster 2024-02-12T15:54:25.361515+0000 mgr.y (mgr.14146) 233 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: cephadm 2024-02-12T15:54:26.201539+0000 mgr.y (mgr.14146) 234 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:54:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:26.211503+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:26.213603+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:54:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:26.215239+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:54:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: cephadm 2024-02-12T15:54:26.216475+0000 mgr.y (mgr.14146) 235 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 8659M 2024-02-12T15:54:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:26.224781+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:26.226825+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:26.228632+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:27 smithi169 bash[19604]: audit 2024-02-12T15:54:26.237499+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:25.065823+0000 mgr.y (mgr.14146) 232 : audit [DBG] from='client.24287 v1:172.21.15.123:0/202267911' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:54:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: cluster 2024-02-12T15:54:25.361515+0000 mgr.y (mgr.14146) 233 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: cephadm 2024-02-12T15:54:26.201539+0000 mgr.y (mgr.14146) 234 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:54:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:26.211503+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:26.213603+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:54:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:26.215239+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:54:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: cephadm 2024-02-12T15:54:26.216475+0000 mgr.y (mgr.14146) 235 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 8659M 2024-02-12T15:54:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:26.224781+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:26.226825+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:26.228632+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:27 smithi123 bash[19119]: audit 2024-02-12T15:54:26.237499+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:25.065823+0000 mgr.y (mgr.14146) 232 : audit [DBG] from='client.24287 v1:172.21.15.123:0/202267911' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: cluster 2024-02-12T15:54:25.361515+0000 mgr.y (mgr.14146) 233 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: cephadm 2024-02-12T15:54:26.201539+0000 mgr.y (mgr.14146) 234 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:26.211503+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:26.213603+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:26.215239+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: cephadm 2024-02-12T15:54:26.216475+0000 mgr.y (mgr.14146) 235 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 8659M 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:26.224781+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:26.226825+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:26.228632+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:27 smithi121 bash[18489]: audit 2024-02-12T15:54:26.237499+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:28.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:28 smithi169 bash[19604]: cluster 2024-02-12T15:54:27.362099+0000 mgr.y (mgr.14146) 236 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:28 smithi123 bash[19119]: cluster 2024-02-12T15:54:27.362099+0000 mgr.y (mgr.14146) 236 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:28.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:28 smithi121 bash[18489]: cluster 2024-02-12T15:54:27.362099+0000 mgr.y (mgr.14146) 236 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:30 smithi169 bash[19604]: cluster 2024-02-12T15:54:29.362642+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:31.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:30 smithi123 bash[19119]: cluster 2024-02-12T15:54:29.362642+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:31.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:30 smithi121 bash[18489]: cluster 2024-02-12T15:54:29.362642+0000 mgr.y (mgr.14146) 237 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:33 smithi169 bash[19604]: cluster 2024-02-12T15:54:31.363215+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:33.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:33 smithi123 bash[19119]: cluster 2024-02-12T15:54:31.363215+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:33.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:32 smithi121 bash[18489]: cluster 2024-02-12T15:54:31.363215+0000 mgr.y (mgr.14146) 238 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:34 smithi121 bash[18489]: audit 2024-02-12T15:54:33.052265+0000 mon.a (mon.0) 484 : audit [INF] from='client.24293 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]: dispatch 2024-02-12T15:54:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:34 smithi121 bash[18489]: audit 2024-02-12T15:54:33.054296+0000 mon.b (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.123:0/375460634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]: dispatch 2024-02-12T15:54:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:34 smithi121 bash[18489]: audit 2024-02-12T15:54:33.059969+0000 mon.a (mon.0) 485 : audit [INF] from='client.24293 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]': finished 2024-02-12T15:54:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:34 smithi121 bash[18489]: cluster 2024-02-12T15:54:33.060061+0000 mon.a (mon.0) 486 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-12T15:54:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:34 smithi121 bash[18489]: audit 2024-02-12T15:54:33.060323+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:34 smithi121 bash[18489]: audit 2024-02-12T15:54:33.658233+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.123:0/3493886024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:54:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:34 smithi169 bash[19604]: audit 2024-02-12T15:54:33.052265+0000 mon.a (mon.0) 484 : audit [INF] from='client.24293 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]: dispatch 2024-02-12T15:54:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:34 smithi169 bash[19604]: audit 2024-02-12T15:54:33.054296+0000 mon.b (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.123:0/375460634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]: dispatch 2024-02-12T15:54:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:34 smithi169 bash[19604]: audit 2024-02-12T15:54:33.059969+0000 mon.a (mon.0) 485 : audit [INF] from='client.24293 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]': finished 2024-02-12T15:54:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:34 smithi169 bash[19604]: cluster 2024-02-12T15:54:33.060061+0000 mon.a (mon.0) 486 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-12T15:54:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:34 smithi169 bash[19604]: audit 2024-02-12T15:54:33.060323+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:34 smithi169 bash[19604]: audit 2024-02-12T15:54:33.658233+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.123:0/3493886024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:54:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:34 smithi123 bash[19119]: audit 2024-02-12T15:54:33.052265+0000 mon.a (mon.0) 484 : audit [INF] from='client.24293 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]: dispatch 2024-02-12T15:54:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:34 smithi123 bash[19119]: audit 2024-02-12T15:54:33.054296+0000 mon.b (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.123:0/375460634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]: dispatch 2024-02-12T15:54:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:34 smithi123 bash[19119]: audit 2024-02-12T15:54:33.059969+0000 mon.a (mon.0) 485 : audit [INF] from='client.24293 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0"}]': finished 2024-02-12T15:54:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:34 smithi123 bash[19119]: cluster 2024-02-12T15:54:33.060061+0000 mon.a (mon.0) 486 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-12T15:54:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:34 smithi123 bash[19119]: audit 2024-02-12T15:54:33.060323+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:34 smithi123 bash[19119]: audit 2024-02-12T15:54:33.658233+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.123:0/3493886024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:54:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:35 smithi169 bash[19604]: cluster 2024-02-12T15:54:33.363735+0000 mgr.y (mgr.14146) 239 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:35.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:35 smithi123 bash[19119]: cluster 2024-02-12T15:54:33.363735+0000 mgr.y (mgr.14146) 239 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:35.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:35 smithi121 bash[18489]: cluster 2024-02-12T15:54:33.363735+0000 mgr.y (mgr.14146) 239 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:36 smithi169 bash[19604]: cluster 2024-02-12T15:54:35.020499+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:36 smithi169 bash[19604]: cluster 2024-02-12T15:54:35.020600+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:36 smithi123 bash[19119]: cluster 2024-02-12T15:54:35.020499+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:36 smithi123 bash[19119]: cluster 2024-02-12T15:54:35.020600+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:36.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:36 smithi121 bash[18489]: cluster 2024-02-12T15:54:35.020499+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:36.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:36 smithi121 bash[18489]: cluster 2024-02-12T15:54:35.020600+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:37 smithi169 bash[19604]: cluster 2024-02-12T15:54:35.364362+0000 mgr.y (mgr.14146) 240 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:37 smithi169 bash[19604]: cluster 2024-02-12T15:54:36.020768+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-12T15:54:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:37 smithi169 bash[19604]: audit 2024-02-12T15:54:36.020955+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:37 smithi123 bash[19119]: cluster 2024-02-12T15:54:35.364362+0000 mgr.y (mgr.14146) 240 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:37 smithi123 bash[19119]: cluster 2024-02-12T15:54:36.020768+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-12T15:54:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:37 smithi123 bash[19119]: audit 2024-02-12T15:54:36.020955+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:37 smithi121 bash[18489]: cluster 2024-02-12T15:54:35.364362+0000 mgr.y (mgr.14146) 240 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:37 smithi121 bash[18489]: cluster 2024-02-12T15:54:36.020768+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-12T15:54:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:37 smithi121 bash[18489]: audit 2024-02-12T15:54:36.020955+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:38 smithi169 bash[19604]: cluster 2024-02-12T15:54:37.364947+0000 mgr.y (mgr.14146) 241 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:38 smithi123 bash[19119]: cluster 2024-02-12T15:54:37.364947+0000 mgr.y (mgr.14146) 241 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:38 smithi121 bash[18489]: cluster 2024-02-12T15:54:37.364947+0000 mgr.y (mgr.14146) 241 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:41 smithi169 bash[19604]: cluster 2024-02-12T15:54:39.365537+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:41 smithi123 bash[19119]: cluster 2024-02-12T15:54:39.365537+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:41.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:40 smithi121 bash[18489]: cluster 2024-02-12T15:54:39.365537+0000 mgr.y (mgr.14146) 242 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:43.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:43 smithi169 bash[19604]: cluster 2024-02-12T15:54:41.366076+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:43.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:43 smithi123 bash[19119]: cluster 2024-02-12T15:54:41.366076+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:43 smithi121 bash[18489]: cluster 2024-02-12T15:54:41.366076+0000 mgr.y (mgr.14146) 243 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:44 smithi169 bash[19604]: cluster 2024-02-12T15:54:42.655052+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:44 smithi169 bash[19604]: cluster 2024-02-12T15:54:42.655149+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:44 smithi169 bash[19604]: cluster 2024-02-12T15:54:43.006161+0000 mon.a (mon.0) 490 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2024-02-12T15:54:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:44 smithi169 bash[19604]: audit 2024-02-12T15:54:43.006784+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:44 smithi123 bash[19119]: cluster 2024-02-12T15:54:42.655052+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:44 smithi123 bash[19119]: cluster 2024-02-12T15:54:42.655149+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:44 smithi123 bash[19119]: cluster 2024-02-12T15:54:43.006161+0000 mon.a (mon.0) 490 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2024-02-12T15:54:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:44 smithi123 bash[19119]: audit 2024-02-12T15:54:43.006784+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:44.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:44 smithi121 bash[18489]: cluster 2024-02-12T15:54:42.655052+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:44.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:44 smithi121 bash[18489]: cluster 2024-02-12T15:54:42.655149+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:44.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:44 smithi121 bash[18489]: cluster 2024-02-12T15:54:43.006161+0000 mon.a (mon.0) 490 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2024-02-12T15:54:44.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:44 smithi121 bash[18489]: audit 2024-02-12T15:54:43.006784+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:45 smithi169 bash[19604]: cluster 2024-02-12T15:54:43.366614+0000 mgr.y (mgr.14146) 244 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:45 smithi169 bash[19604]: audit 2024-02-12T15:54:44.736178+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-12T15:54:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:45 smithi169 bash[19604]: audit 2024-02-12T15:54:44.738065+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:45 smithi169 bash[19604]: cephadm 2024-02-12T15:54:44.739777+0000 mgr.y (mgr.14146) 245 : cephadm [INF] Deploying daemon osd.6 on smithi123 2024-02-12T15:54:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:45 smithi123 bash[19119]: cluster 2024-02-12T15:54:43.366614+0000 mgr.y (mgr.14146) 244 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:45 smithi123 bash[19119]: audit 2024-02-12T15:54:44.736178+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-12T15:54:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:45 smithi123 bash[19119]: audit 2024-02-12T15:54:44.738065+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:45.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:45 smithi123 bash[19119]: cephadm 2024-02-12T15:54:44.739777+0000 mgr.y (mgr.14146) 245 : cephadm [INF] Deploying daemon osd.6 on smithi123 2024-02-12T15:54:45.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:45 smithi121 bash[18489]: cluster 2024-02-12T15:54:43.366614+0000 mgr.y (mgr.14146) 244 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:45.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:45 smithi121 bash[18489]: audit 2024-02-12T15:54:44.736178+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-12T15:54:45.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:45 smithi121 bash[18489]: audit 2024-02-12T15:54:44.738065+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:45.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:45 smithi121 bash[18489]: cephadm 2024-02-12T15:54:44.739777+0000 mgr.y (mgr.14146) 245 : cephadm [INF] Deploying daemon osd.6 on smithi123 2024-02-12T15:54:46.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:46 smithi169 bash[19604]: cluster 2024-02-12T15:54:45.367226+0000 mgr.y (mgr.14146) 246 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:46 smithi123 bash[19119]: cluster 2024-02-12T15:54:45.367226+0000 mgr.y (mgr.14146) 246 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:46 smithi121 bash[18489]: cluster 2024-02-12T15:54:45.367226+0000 mgr.y (mgr.14146) 246 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:48 smithi123 bash[19119]: cluster 2024-02-12T15:54:47.367802+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:48 smithi123 bash[19119]: audit 2024-02-12T15:54:48.364965+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:48 smithi123 bash[19119]: audit 2024-02-12T15:54:48.368545+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:48 smithi169 bash[19604]: cluster 2024-02-12T15:54:47.367802+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:48 smithi169 bash[19604]: audit 2024-02-12T15:54:48.364965+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:48 smithi169 bash[19604]: audit 2024-02-12T15:54:48.368545+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:48.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:48 smithi121 bash[18489]: cluster 2024-02-12T15:54:47.367802+0000 mgr.y (mgr.14146) 247 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:48.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:48 smithi121 bash[18489]: audit 2024-02-12T15:54:48.364965+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:48.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:48 smithi121 bash[18489]: audit 2024-02-12T15:54:48.368545+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:51.261 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:51 smithi123 bash[19119]: cluster 2024-02-12T15:54:49.368343+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:51 smithi169 bash[19604]: cluster 2024-02-12T15:54:49.368343+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:51.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:51 smithi121 bash[18489]: cluster 2024-02-12T15:54:49.368343+0000 mgr.y (mgr.14146) 248 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:53.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:53 smithi169 bash[19604]: cluster 2024-02-12T15:54:51.368886+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:53 smithi123 bash[19119]: cluster 2024-02-12T15:54:51.368886+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:53 smithi121 bash[18489]: cluster 2024-02-12T15:54:51.368886+0000 mgr.y (mgr.14146) 249 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:54 smithi169 bash[19604]: cluster 2024-02-12T15:54:53.369447+0000 mgr.y (mgr.14146) 250 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:54 smithi169 bash[19604]: audit 2024-02-12T15:54:53.779545+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:54 smithi169 bash[19604]: audit 2024-02-12T15:54:53.781100+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:54 smithi169 bash[19604]: audit 2024-02-12T15:54:53.782835+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:54 smithi169 bash[19604]: audit 2024-02-12T15:54:53.791283+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:54 smithi123 bash[19119]: cluster 2024-02-12T15:54:53.369447+0000 mgr.y (mgr.14146) 250 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:54 smithi123 bash[19119]: audit 2024-02-12T15:54:53.779545+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:54 smithi123 bash[19119]: audit 2024-02-12T15:54:53.781100+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:54 smithi123 bash[19119]: audit 2024-02-12T15:54:53.782835+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:54 smithi123 bash[19119]: audit 2024-02-12T15:54:53.791283+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:54 smithi121 bash[18489]: cluster 2024-02-12T15:54:53.369447+0000 mgr.y (mgr.14146) 250 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:54 smithi121 bash[18489]: audit 2024-02-12T15:54:53.779545+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:54 smithi121 bash[18489]: audit 2024-02-12T15:54:53.781100+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:54:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:54 smithi121 bash[18489]: audit 2024-02-12T15:54:53.782835+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:54:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:54 smithi121 bash[18489]: audit 2024-02-12T15:54:53.791283+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:55 smithi169 bash[19604]: audit 2024-02-12T15:54:54.855447+0000 mon.a (mon.0) 500 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T15:54:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:55 smithi169 bash[19604]: audit 2024-02-12T15:54:54.856178+0000 mon.c (mon.1) 3 : audit [INF] from='osd.6 v1:172.21.15.123:6808/1377261784' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T15:54:56.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:55 smithi123 bash[19119]: audit 2024-02-12T15:54:54.855447+0000 mon.a (mon.0) 500 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T15:54:56.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:55 smithi123 bash[19119]: audit 2024-02-12T15:54:54.856178+0000 mon.c (mon.1) 3 : audit [INF] from='osd.6 v1:172.21.15.123:6808/1377261784' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T15:54:56.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:55 smithi121 bash[18489]: audit 2024-02-12T15:54:54.855447+0000 mon.a (mon.0) 500 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T15:54:56.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:55 smithi121 bash[18489]: audit 2024-02-12T15:54:54.856178+0000 mon.c (mon.1) 3 : audit [INF] from='osd.6 v1:172.21.15.123:6808/1377261784' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-12T15:54:56.942 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 6 on host 'smithi123' 2024-02-12T15:54:57.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:56 smithi123 bash[19119]: cluster 2024-02-12T15:54:55.370025+0000 mgr.y (mgr.14146) 251 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:57.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:56 smithi123 bash[19119]: audit 2024-02-12T15:54:55.787664+0000 mon.a (mon.0) 501 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-12T15:54:57.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:56 smithi123 bash[19119]: cluster 2024-02-12T15:54:55.787851+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e46: 7 total, 6 up, 7 in 2024-02-12T15:54:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:56 smithi123 bash[19119]: audit 2024-02-12T15:54:55.788576+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:56 smithi123 bash[19119]: audit 2024-02-12T15:54:55.791215+0000 mon.c (mon.1) 4 : audit [INF] from='osd.6 v1:172.21.15.123:6808/1377261784' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:56 smithi123 bash[19119]: audit 2024-02-12T15:54:55.795010+0000 mon.a (mon.0) 504 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:56 smithi121 bash[18489]: cluster 2024-02-12T15:54:55.370025+0000 mgr.y (mgr.14146) 251 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:56 smithi121 bash[18489]: audit 2024-02-12T15:54:55.787664+0000 mon.a (mon.0) 501 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-12T15:54:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:56 smithi121 bash[18489]: cluster 2024-02-12T15:54:55.787851+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e46: 7 total, 6 up, 7 in 2024-02-12T15:54:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:56 smithi121 bash[18489]: audit 2024-02-12T15:54:55.788576+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:56 smithi121 bash[18489]: audit 2024-02-12T15:54:55.791215+0000 mon.c (mon.1) 4 : audit [INF] from='osd.6 v1:172.21.15.123:6808/1377261784' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:56 smithi121 bash[18489]: audit 2024-02-12T15:54:55.795010+0000 mon.a (mon.0) 504 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:56 smithi169 bash[19604]: cluster 2024-02-12T15:54:55.370025+0000 mgr.y (mgr.14146) 251 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:56 smithi169 bash[19604]: audit 2024-02-12T15:54:55.787664+0000 mon.a (mon.0) 501 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-12T15:54:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:56 smithi169 bash[19604]: cluster 2024-02-12T15:54:55.787851+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e46: 7 total, 6 up, 7 in 2024-02-12T15:54:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:56 smithi169 bash[19604]: audit 2024-02-12T15:54:55.788576+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:56 smithi169 bash[19604]: audit 2024-02-12T15:54:55.791215+0000 mon.c (mon.1) 4 : audit [INF] from='osd.6 v1:172.21.15.123:6808/1377261784' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:57.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:56 smithi169 bash[19604]: audit 2024-02-12T15:54:55.795010+0000 mon.a (mon.0) 504 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:54:57.571 DEBUG:teuthology.orchestra.run.smithi123:osd.6> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.6.service 2024-02-12T15:54:57.573 INFO:tasks.cephadm:Deploying osd.7 on smithi123 with /dev/vg_nvme/lv_1... 2024-02-12T15:54:57.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-12T15:54:57.721 INFO:journalctl@ceph.osd.6.smithi123.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:54:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:57 smithi123 bash[19119]: audit 2024-02-12T15:54:56.798591+0000 mon.a (mon.0) 505 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:54:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:57 smithi123 bash[19119]: cluster 2024-02-12T15:54:56.798760+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e47: 7 total, 6 up, 7 in 2024-02-12T15:54:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:57 smithi123 bash[19119]: audit 2024-02-12T15:54:56.799346+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:57 smithi123 bash[19119]: audit 2024-02-12T15:54:56.805686+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:57 smithi123 bash[19119]: audit 2024-02-12T15:54:56.936741+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:57 smithi123 bash[19119]: audit 2024-02-12T15:54:56.968825+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:57 smithi121 bash[18489]: audit 2024-02-12T15:54:56.798591+0000 mon.a (mon.0) 505 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:54:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:57 smithi121 bash[18489]: cluster 2024-02-12T15:54:56.798760+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e47: 7 total, 6 up, 7 in 2024-02-12T15:54:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:57 smithi121 bash[18489]: audit 2024-02-12T15:54:56.799346+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:57 smithi121 bash[18489]: audit 2024-02-12T15:54:56.805686+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:57 smithi121 bash[18489]: audit 2024-02-12T15:54:56.936741+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:57 smithi121 bash[18489]: audit 2024-02-12T15:54:56.968825+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:57 smithi169 bash[19604]: audit 2024-02-12T15:54:56.798591+0000 mon.a (mon.0) 505 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:54:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:57 smithi169 bash[19604]: cluster 2024-02-12T15:54:56.798760+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e47: 7 total, 6 up, 7 in 2024-02-12T15:54:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:57 smithi169 bash[19604]: audit 2024-02-12T15:54:56.799346+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:57 smithi169 bash[19604]: audit 2024-02-12T15:54:56.805686+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:57 smithi169 bash[19604]: audit 2024-02-12T15:54:56.936741+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:54:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:57 smithi169 bash[19604]: audit 2024-02-12T15:54:56.968825+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:54:59.026 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:58 smithi123 bash[19119]: cluster 2024-02-12T15:54:55.865669+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:59.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:58 smithi123 bash[19119]: cluster 2024-02-12T15:54:55.865752+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:59.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:58 smithi123 bash[19119]: cluster 2024-02-12T15:54:57.370659+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:59.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:58 smithi123 bash[19119]: cluster 2024-02-12T15:54:57.808520+0000 mon.a (mon.0) 511 : cluster [INF] osd.6 v1:172.21.15.123:6808/1377261784 boot 2024-02-12T15:54:59.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:58 smithi123 bash[19119]: cluster 2024-02-12T15:54:57.808627+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-12T15:54:59.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:58 smithi123 bash[19119]: audit 2024-02-12T15:54:57.809267+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:58 smithi121 bash[18489]: cluster 2024-02-12T15:54:55.865669+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:58 smithi121 bash[18489]: cluster 2024-02-12T15:54:55.865752+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:58 smithi121 bash[18489]: cluster 2024-02-12T15:54:57.370659+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:58 smithi121 bash[18489]: cluster 2024-02-12T15:54:57.808520+0000 mon.a (mon.0) 511 : cluster [INF] osd.6 v1:172.21.15.123:6808/1377261784 boot 2024-02-12T15:54:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:58 smithi121 bash[18489]: cluster 2024-02-12T15:54:57.808627+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-12T15:54:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:58 smithi121 bash[18489]: audit 2024-02-12T15:54:57.809267+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:54:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:58 smithi169 bash[19604]: cluster 2024-02-12T15:54:55.865669+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:54:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:58 smithi169 bash[19604]: cluster 2024-02-12T15:54:55.865752+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:54:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:58 smithi169 bash[19604]: cluster 2024-02-12T15:54:57.370659+0000 mgr.y (mgr.14146) 252 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-12T15:54:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:58 smithi169 bash[19604]: cluster 2024-02-12T15:54:57.808520+0000 mon.a (mon.0) 511 : cluster [INF] osd.6 v1:172.21.15.123:6808/1377261784 boot 2024-02-12T15:54:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:58 smithi169 bash[19604]: cluster 2024-02-12T15:54:57.808627+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-12T15:54:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:58 smithi169 bash[19604]: audit 2024-02-12T15:54:57.809267+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-12T15:55:00.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:54:59 smithi123 bash[19119]: cluster 2024-02-12T15:54:58.810858+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2024-02-12T15:55:00.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:54:59 smithi121 bash[18489]: cluster 2024-02-12T15:54:58.810858+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2024-02-12T15:55:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:54:59 smithi169 bash[19604]: cluster 2024-02-12T15:54:58.810858+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2024-02-12T15:55:00.779 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-12T15:55:00.779 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-12T15:55:00.779 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-02-12T15:55:00.780 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-02-12T15:55:00.780 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121836 s, 861 MB/s 2024-02-12T15:55:00.780 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-02-12T15:55:01.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:00 smithi123 bash[19119]: cluster 2024-02-12T15:54:59.371294+0000 mgr.y (mgr.14146) 253 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:01.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:00 smithi123 bash[19119]: cluster 2024-02-12T15:54:59.827372+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e50: 7 total, 7 up, 7 in 2024-02-12T15:55:01.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:00 smithi121 bash[18489]: cluster 2024-02-12T15:54:59.371294+0000 mgr.y (mgr.14146) 253 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:01.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:00 smithi121 bash[18489]: cluster 2024-02-12T15:54:59.827372+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e50: 7 total, 7 up, 7 in 2024-02-12T15:55:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:00 smithi169 bash[19604]: cluster 2024-02-12T15:54:59.371294+0000 mgr.y (mgr.14146) 253 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:00 smithi169 bash[19604]: cluster 2024-02-12T15:54:59.827372+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e50: 7 total, 7 up, 7 in 2024-02-12T15:55:01.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_1 2024-02-12T15:55:03.105 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:02 smithi123 bash[19119]: cluster 2024-02-12T15:55:01.371861+0000 mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:03.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:02 smithi121 bash[18489]: cluster 2024-02-12T15:55:01.371861+0000 mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:03.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:02 smithi169 bash[19604]: cluster 2024-02-12T15:55:01.371861+0000 mgr.y (mgr.14146) 254 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:04.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.412812+0000 mon.a (mon.0) 516 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=17,config=2,health=12,kv=61,logm=8} crc {auth=1722902203,config=1248726048,health=1444605944,kv=1026518164,logm=3533100778}) 2024-02-12T15:55:04.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.414435+0000 mon.a (mon.0) 517 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1236544048}) 2024-02-12T15:55:04.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.416161+0000 mon.a (mon.0) 518 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1482775686}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.418435+0000 mon.a (mon.0) 519 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3013584798}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.424661+0000 mon.a (mon.0) 520 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3703978978}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.429546+0000 mon.a (mon.0) 521 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3271606841}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.432788+0000 mon.a (mon.0) 522 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=73,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=4} crc {logm=3594722449,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2272510050}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.433647+0000 mon.a (mon.0) 523 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mgrstat=8,mon_config_key=92} crc {mgrstat=3055558525,mon_config_key=3308045456}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.434355+0000 mon.a (mon.0) 524 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3545407195}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.435252+0000 mon.a (mon.0) 525 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=33,monmap=5,osd_metadata=7,osd_pg_creating=1,osdmap=54} crc {mon_config_key=1923849884,monmap=2720272532,osd_metadata=1177587031,osd_pg_creating=1009218749,osdmap=2224936412}) 2024-02-12T15:55:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:03 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.440998+0000 mon.a (mon.0) 526 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=49} crc {osdmap=521164729}) 2024-02-12T15:55:04.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.412812+0000 mon.a (mon.0) 516 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=17,config=2,health=12,kv=61,logm=8} crc {auth=1722902203,config=1248726048,health=1444605944,kv=1026518164,logm=3533100778}) 2024-02-12T15:55:04.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.414435+0000 mon.a (mon.0) 517 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1236544048}) 2024-02-12T15:55:04.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.416161+0000 mon.a (mon.0) 518 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1482775686}) 2024-02-12T15:55:04.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.418435+0000 mon.a (mon.0) 519 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3013584798}) 2024-02-12T15:55:04.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.424661+0000 mon.a (mon.0) 520 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3703978978}) 2024-02-12T15:55:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.429546+0000 mon.a (mon.0) 521 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3271606841}) 2024-02-12T15:55:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.432788+0000 mon.a (mon.0) 522 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=73,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=4} crc {logm=3594722449,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2272510050}) 2024-02-12T15:55:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.433647+0000 mon.a (mon.0) 523 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mgrstat=8,mon_config_key=92} crc {mgrstat=3055558525,mon_config_key=3308045456}) 2024-02-12T15:55:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.434355+0000 mon.a (mon.0) 524 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3545407195}) 2024-02-12T15:55:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.435252+0000 mon.a (mon.0) 525 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=33,monmap=5,osd_metadata=7,osd_pg_creating=1,osdmap=54} crc {mon_config_key=1923849884,monmap=2720272532,osd_metadata=1177587031,osd_pg_creating=1009218749,osdmap=2224936412}) 2024-02-12T15:55:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:03 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.440998+0000 mon.a (mon.0) 526 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=49} crc {osdmap=521164729}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.412812+0000 mon.a (mon.0) 516 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=17,config=2,health=12,kv=61,logm=8} crc {auth=1722902203,config=1248726048,health=1444605944,kv=1026518164,logm=3533100778}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.414435+0000 mon.a (mon.0) 517 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1236544048}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.416161+0000 mon.a (mon.0) 518 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1482775686}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.418435+0000 mon.a (mon.0) 519 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3013584798}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.424661+0000 mon.a (mon.0) 520 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3703978978}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.429546+0000 mon.a (mon.0) 521 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3271606841}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.432788+0000 mon.a (mon.0) 522 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=73,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=4} crc {logm=3594722449,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2272510050}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.433647+0000 mon.a (mon.0) 523 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mgrstat=8,mon_config_key=92} crc {mgrstat=3055558525,mon_config_key=3308045456}) 2024-02-12T15:55:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.434355+0000 mon.a (mon.0) 524 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3545407195}) 2024-02-12T15:55:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.435252+0000 mon.a (mon.0) 525 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=33,monmap=5,osd_metadata=7,osd_pg_creating=1,osdmap=54} crc {mon_config_key=1923849884,monmap=2720272532,osd_metadata=1177587031,osd_pg_creating=1009218749,osdmap=2224936412}) 2024-02-12T15:55:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:03 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.440998+0000 mon.a (mon.0) 526 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=49} crc {osdmap=521164729}) 2024-02-12T15:55:05.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:04 smithi123 bash[19119]: cluster 2024-02-12T15:55:03.372419+0000 mgr.y (mgr.14146) 255 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:05.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:04 smithi121 bash[18489]: cluster 2024-02-12T15:55:03.372419+0000 mgr.y (mgr.14146) 255 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:04 smithi169 bash[19604]: cluster 2024-02-12T15:55:03.372419+0000 mgr.y (mgr.14146) 255 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:06 smithi169 bash[19604]: audit 2024-02-12T15:55:05.732224+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:55:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:06 smithi169 bash[19604]: audit 2024-02-12T15:55:05.738335+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:55:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:06 smithi169 bash[19604]: audit 2024-02-12T15:55:05.739935+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:06 smithi123 bash[19119]: audit 2024-02-12T15:55:05.732224+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:55:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:06 smithi123 bash[19119]: audit 2024-02-12T15:55:05.738335+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:55:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:06 smithi123 bash[19119]: audit 2024-02-12T15:55:05.739935+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:06 smithi121 bash[18489]: audit 2024-02-12T15:55:05.732224+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:55:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:06 smithi121 bash[18489]: audit 2024-02-12T15:55:05.738335+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:55:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:06 smithi121 bash[18489]: audit 2024-02-12T15:55:05.739935+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:07 smithi169 bash[19604]: cluster 2024-02-12T15:55:05.373027+0000 mgr.y (mgr.14146) 256 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:07 smithi169 bash[19604]: audit 2024-02-12T15:55:05.725325+0000 mgr.y (mgr.14146) 257 : audit [DBG] from='client.24314 v1:172.21.15.123:0/1056961005' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:55:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:07 smithi123 bash[19119]: cluster 2024-02-12T15:55:05.373027+0000 mgr.y (mgr.14146) 256 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:07 smithi123 bash[19119]: audit 2024-02-12T15:55:05.725325+0000 mgr.y (mgr.14146) 257 : audit [DBG] from='client.24314 v1:172.21.15.123:0/1056961005' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:55:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:07 smithi121 bash[18489]: cluster 2024-02-12T15:55:05.373027+0000 mgr.y (mgr.14146) 256 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:07 smithi121 bash[18489]: audit 2024-02-12T15:55:05.725325+0000 mgr.y (mgr.14146) 257 : audit [DBG] from='client.24314 v1:172.21.15.123:0/1056961005' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:55:08.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: cephadm 2024-02-12T15:55:07.003893+0000 mgr.y (mgr.14146) 258 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:55:08.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.014111+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.016600+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.022781+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.024220+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.031687+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.033227+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.034602+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:08 smithi169 bash[19604]: audit 2024-02-12T15:55:07.042983+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: cephadm 2024-02-12T15:55:07.003893+0000 mgr.y (mgr.14146) 258 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.014111+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.016600+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.022781+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.024220+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.031687+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.033227+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.034602+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:08.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:08 smithi123 bash[19119]: audit 2024-02-12T15:55:07.042983+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: cephadm 2024-02-12T15:55:07.003893+0000 mgr.y (mgr.14146) 258 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:55:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.014111+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.016600+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.022781+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.024220+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.031687+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.033227+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.034602+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:08 smithi121 bash[18489]: audit 2024-02-12T15:55:07.042983+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:09 smithi169 bash[19604]: cephadm 2024-02-12T15:55:07.025224+0000 mgr.y (mgr.14146) 259 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 5772M 2024-02-12T15:55:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:09 smithi169 bash[19604]: cluster 2024-02-12T15:55:07.373619+0000 mgr.y (mgr.14146) 260 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:09 smithi169 bash[19604]: cluster 2024-02-12T15:55:08.554790+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:09 smithi169 bash[19604]: cluster 2024-02-12T15:55:08.554881+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:09 smithi123 bash[19119]: cephadm 2024-02-12T15:55:07.025224+0000 mgr.y (mgr.14146) 259 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 5772M 2024-02-12T15:55:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:09 smithi123 bash[19119]: cluster 2024-02-12T15:55:07.373619+0000 mgr.y (mgr.14146) 260 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:09 smithi123 bash[19119]: cluster 2024-02-12T15:55:08.554790+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:09 smithi123 bash[19119]: cluster 2024-02-12T15:55:08.554881+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:09.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:09 smithi121 bash[18489]: cephadm 2024-02-12T15:55:07.025224+0000 mgr.y (mgr.14146) 259 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 5772M 2024-02-12T15:55:09.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:09 smithi121 bash[18489]: cluster 2024-02-12T15:55:07.373619+0000 mgr.y (mgr.14146) 260 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:09.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:09 smithi121 bash[18489]: cluster 2024-02-12T15:55:08.554790+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:09.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:09 smithi121 bash[18489]: cluster 2024-02-12T15:55:08.554881+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:10.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:10 smithi123 bash[19119]: cluster 2024-02-12T15:55:09.050256+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e51: 7 total, 7 up, 7 in 2024-02-12T15:55:10.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:10 smithi123 bash[19119]: cluster 2024-02-12T15:55:09.374257+0000 mgr.y (mgr.14146) 261 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:10.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:10 smithi121 bash[18489]: cluster 2024-02-12T15:55:09.050256+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e51: 7 total, 7 up, 7 in 2024-02-12T15:55:10.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:10 smithi121 bash[18489]: cluster 2024-02-12T15:55:09.374257+0000 mgr.y (mgr.14146) 261 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:10.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:10 smithi169 bash[19604]: cluster 2024-02-12T15:55:09.050256+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e51: 7 total, 7 up, 7 in 2024-02-12T15:55:10.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:10 smithi169 bash[19604]: cluster 2024-02-12T15:55:09.374257+0000 mgr.y (mgr.14146) 261 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:12 smithi169 bash[19604]: cluster 2024-02-12T15:55:11.374875+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:12 smithi123 bash[19119]: cluster 2024-02-12T15:55:11.374875+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:12.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:12 smithi121 bash[18489]: cluster 2024-02-12T15:55:11.374875+0000 mgr.y (mgr.14146) 262 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:14 smithi169 bash[19604]: cluster 2024-02-12T15:55:13.375450+0000 mgr.y (mgr.14146) 263 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:14 smithi169 bash[19604]: audit 2024-02-12T15:55:13.813464+0000 mon.a (mon.0) 539 : audit [INF] from='client.24320 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]: dispatch 2024-02-12T15:55:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:14 smithi169 bash[19604]: audit 2024-02-12T15:55:13.815557+0000 mon.b (mon.2) 18 : audit [INF] from='client.? v1:172.21.15.123:0/2440652714' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]: dispatch 2024-02-12T15:55:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:14 smithi169 bash[19604]: audit 2024-02-12T15:55:13.824102+0000 mon.a (mon.0) 540 : audit [INF] from='client.24320 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]': finished 2024-02-12T15:55:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:14 smithi169 bash[19604]: cluster 2024-02-12T15:55:13.824162+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-12T15:55:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:14 smithi169 bash[19604]: audit 2024-02-12T15:55:13.824360+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:14 smithi169 bash[19604]: audit 2024-02-12T15:55:14.394462+0000 mon.b (mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.123:0/762263217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:55:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:14 smithi123 bash[19119]: cluster 2024-02-12T15:55:13.375450+0000 mgr.y (mgr.14146) 263 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:14 smithi123 bash[19119]: audit 2024-02-12T15:55:13.813464+0000 mon.a (mon.0) 539 : audit [INF] from='client.24320 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]: dispatch 2024-02-12T15:55:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:14 smithi123 bash[19119]: audit 2024-02-12T15:55:13.815557+0000 mon.b (mon.2) 18 : audit [INF] from='client.? v1:172.21.15.123:0/2440652714' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]: dispatch 2024-02-12T15:55:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:14 smithi123 bash[19119]: audit 2024-02-12T15:55:13.824102+0000 mon.a (mon.0) 540 : audit [INF] from='client.24320 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]': finished 2024-02-12T15:55:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:14 smithi123 bash[19119]: cluster 2024-02-12T15:55:13.824162+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-12T15:55:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:14 smithi123 bash[19119]: audit 2024-02-12T15:55:13.824360+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:14 smithi123 bash[19119]: audit 2024-02-12T15:55:14.394462+0000 mon.b (mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.123:0/762263217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:55:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:14 smithi121 bash[18489]: cluster 2024-02-12T15:55:13.375450+0000 mgr.y (mgr.14146) 263 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:14 smithi121 bash[18489]: audit 2024-02-12T15:55:13.813464+0000 mon.a (mon.0) 539 : audit [INF] from='client.24320 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]: dispatch 2024-02-12T15:55:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:14 smithi121 bash[18489]: audit 2024-02-12T15:55:13.815557+0000 mon.b (mon.2) 18 : audit [INF] from='client.? v1:172.21.15.123:0/2440652714' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]: dispatch 2024-02-12T15:55:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:14 smithi121 bash[18489]: audit 2024-02-12T15:55:13.824102+0000 mon.a (mon.0) 540 : audit [INF] from='client.24320 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515"}]': finished 2024-02-12T15:55:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:14 smithi121 bash[18489]: cluster 2024-02-12T15:55:13.824162+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-12T15:55:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:14 smithi121 bash[18489]: audit 2024-02-12T15:55:13.824360+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:14 smithi121 bash[18489]: audit 2024-02-12T15:55:14.394462+0000 mon.b (mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.123:0/762263217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:55:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:17 smithi123 bash[19119]: cluster 2024-02-12T15:55:15.376048+0000 mgr.y (mgr.14146) 264 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:17 smithi169 bash[19604]: cluster 2024-02-12T15:55:15.376048+0000 mgr.y (mgr.14146) 264 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:17.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:17 smithi121 bash[18489]: cluster 2024-02-12T15:55:15.376048+0000 mgr.y (mgr.14146) 264 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:18 smithi169 bash[19604]: cluster 2024-02-12T15:55:16.690411+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:18 smithi169 bash[19604]: cluster 2024-02-12T15:55:16.690497+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:18 smithi169 bash[19604]: cluster 2024-02-12T15:55:17.023479+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2024-02-12T15:55:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:18 smithi169 bash[19604]: audit 2024-02-12T15:55:17.024312+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:18 smithi169 bash[19604]: cluster 2024-02-12T15:55:17.376620+0000 mgr.y (mgr.14146) 265 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:18 smithi123 bash[19119]: cluster 2024-02-12T15:55:16.690411+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:18 smithi123 bash[19119]: cluster 2024-02-12T15:55:16.690497+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:18 smithi123 bash[19119]: cluster 2024-02-12T15:55:17.023479+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2024-02-12T15:55:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:18 smithi123 bash[19119]: audit 2024-02-12T15:55:17.024312+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:18 smithi123 bash[19119]: cluster 2024-02-12T15:55:17.376620+0000 mgr.y (mgr.14146) 265 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:18 smithi121 bash[18489]: cluster 2024-02-12T15:55:16.690411+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:18 smithi121 bash[18489]: cluster 2024-02-12T15:55:16.690497+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:18 smithi121 bash[18489]: cluster 2024-02-12T15:55:17.023479+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2024-02-12T15:55:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:18 smithi121 bash[18489]: audit 2024-02-12T15:55:17.024312+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:18 smithi121 bash[18489]: cluster 2024-02-12T15:55:17.376620+0000 mgr.y (mgr.14146) 265 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:21.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:21 smithi169 bash[19604]: cluster 2024-02-12T15:55:19.377194+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:21.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:21 smithi123 bash[19119]: cluster 2024-02-12T15:55:19.377194+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:21 smithi121 bash[18489]: cluster 2024-02-12T15:55:19.377194+0000 mgr.y (mgr.14146) 266 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:23.279 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:23 smithi123 bash[19119]: cluster 2024-02-12T15:55:21.377751+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:23.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:23 smithi169 bash[19604]: cluster 2024-02-12T15:55:21.377751+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:23 smithi121 bash[18489]: cluster 2024-02-12T15:55:21.377751+0000 mgr.y (mgr.14146) 267 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:24.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:24 smithi169 bash[19604]: cluster 2024-02-12T15:55:23.378291+0000 mgr.y (mgr.14146) 268 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:24.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:24 smithi123 bash[19119]: cluster 2024-02-12T15:55:23.378291+0000 mgr.y (mgr.14146) 268 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:24.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:24 smithi121 bash[18489]: cluster 2024-02-12T15:55:23.378291+0000 mgr.y (mgr.14146) 268 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:26 smithi169 bash[19604]: audit 2024-02-12T15:55:25.691140+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-12T15:55:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:26 smithi169 bash[19604]: audit 2024-02-12T15:55:25.692813+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:26 smithi123 bash[19119]: audit 2024-02-12T15:55:25.691140+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-12T15:55:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:26 smithi123 bash[19119]: audit 2024-02-12T15:55:25.692813+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:26 smithi121 bash[18489]: audit 2024-02-12T15:55:25.691140+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-12T15:55:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:26 smithi121 bash[18489]: audit 2024-02-12T15:55:25.692813+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:27 smithi169 bash[19604]: cluster 2024-02-12T15:55:25.378891+0000 mgr.y (mgr.14146) 269 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:27 smithi169 bash[19604]: cephadm 2024-02-12T15:55:25.694346+0000 mgr.y (mgr.14146) 270 : cephadm [INF] Deploying daemon osd.7 on smithi123 2024-02-12T15:55:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:27 smithi123 bash[19119]: cluster 2024-02-12T15:55:25.378891+0000 mgr.y (mgr.14146) 269 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:27.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:27 smithi123 bash[19119]: cephadm 2024-02-12T15:55:25.694346+0000 mgr.y (mgr.14146) 270 : cephadm [INF] Deploying daemon osd.7 on smithi123 2024-02-12T15:55:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:27 smithi121 bash[18489]: cluster 2024-02-12T15:55:25.378891+0000 mgr.y (mgr.14146) 269 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:27 smithi121 bash[18489]: cephadm 2024-02-12T15:55:25.694346+0000 mgr.y (mgr.14146) 270 : cephadm [INF] Deploying daemon osd.7 on smithi123 2024-02-12T15:55:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:28 smithi123 bash[19119]: cluster 2024-02-12T15:55:27.379449+0000 mgr.y (mgr.14146) 271 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:28.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:28 smithi121 bash[18489]: cluster 2024-02-12T15:55:27.379449+0000 mgr.y (mgr.14146) 271 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:28.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:28 smithi169 bash[19604]: cluster 2024-02-12T15:55:27.379449+0000 mgr.y (mgr.14146) 271 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:30.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:30 smithi123 bash[19119]: cluster 2024-02-12T15:55:29.380061+0000 mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:30.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:30 smithi123 bash[19119]: audit 2024-02-12T15:55:29.464627+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:30.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:30 smithi123 bash[19119]: audit 2024-02-12T15:55:29.472532+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:55:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:30 smithi169 bash[19604]: cluster 2024-02-12T15:55:29.380061+0000 mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:30 smithi169 bash[19604]: audit 2024-02-12T15:55:29.464627+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:30 smithi169 bash[19604]: audit 2024-02-12T15:55:29.472532+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:55:30.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:30 smithi121 bash[18489]: cluster 2024-02-12T15:55:29.380061+0000 mgr.y (mgr.14146) 272 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:30.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:30 smithi121 bash[18489]: audit 2024-02-12T15:55:29.464627+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:30.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:30 smithi121 bash[18489]: audit 2024-02-12T15:55:29.472532+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:55:32.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:32 smithi169 bash[19604]: cluster 2024-02-12T15:55:31.380684+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:32.857 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:32 smithi123 bash[19119]: cluster 2024-02-12T15:55:31.380684+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:32.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:32 smithi121 bash[18489]: cluster 2024-02-12T15:55:31.380684+0000 mgr.y (mgr.14146) 273 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:34 smithi169 bash[19604]: cluster 2024-02-12T15:55:33.381245+0000 mgr.y (mgr.14146) 274 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:34 smithi123 bash[19119]: cluster 2024-02-12T15:55:33.381245+0000 mgr.y (mgr.14146) 274 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:34.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:34 smithi121 bash[18489]: cluster 2024-02-12T15:55:33.381245+0000 mgr.y (mgr.14146) 274 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:36.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:35 smithi123 bash[19119]: audit 2024-02-12T15:55:34.902594+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:36.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:35 smithi123 bash[19119]: audit 2024-02-12T15:55:35.190988+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:36.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:35 smithi123 bash[19119]: audit 2024-02-12T15:55:35.192939+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:36.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:35 smithi123 bash[19119]: audit 2024-02-12T15:55:35.203836+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:36.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:35 smithi123 bash[19119]: audit 2024-02-12T15:55:35.817377+0000 mon.a (mon.0) 553 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T15:55:36.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:35 smithi123 bash[19119]: audit 2024-02-12T15:55:35.819986+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 v1:172.21.15.123:6812/1702613957' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T15:55:36.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:35 smithi121 bash[18489]: audit 2024-02-12T15:55:34.902594+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:36.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:35 smithi121 bash[18489]: audit 2024-02-12T15:55:35.190988+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:35 smithi121 bash[18489]: audit 2024-02-12T15:55:35.192939+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:35 smithi121 bash[18489]: audit 2024-02-12T15:55:35.203836+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:35 smithi121 bash[18489]: audit 2024-02-12T15:55:35.817377+0000 mon.a (mon.0) 553 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T15:55:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:35 smithi121 bash[18489]: audit 2024-02-12T15:55:35.819986+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 v1:172.21.15.123:6812/1702613957' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T15:55:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:35 smithi169 bash[19604]: audit 2024-02-12T15:55:34.902594+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:35 smithi169 bash[19604]: audit 2024-02-12T15:55:35.190988+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:35 smithi169 bash[19604]: audit 2024-02-12T15:55:35.192939+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:35 smithi169 bash[19604]: audit 2024-02-12T15:55:35.203836+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:35 smithi169 bash[19604]: audit 2024-02-12T15:55:35.817377+0000 mon.a (mon.0) 553 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T15:55:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:35 smithi169 bash[19604]: audit 2024-02-12T15:55:35.819986+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 v1:172.21.15.123:6812/1702613957' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T15:55:37.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:36 smithi123 bash[19119]: cluster 2024-02-12T15:55:35.381805+0000 mgr.y (mgr.14146) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:37.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:36 smithi123 bash[19119]: audit 2024-02-12T15:55:36.212936+0000 mon.a (mon.0) 554 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T15:55:37.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:36 smithi123 bash[19119]: cluster 2024-02-12T15:55:36.213087+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e54: 8 total, 7 up, 8 in 2024-02-12T15:55:37.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:36 smithi123 bash[19119]: audit 2024-02-12T15:55:36.213377+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:37.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:36 smithi123 bash[19119]: audit 2024-02-12T15:55:36.215623+0000 mon.a (mon.0) 557 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:55:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:36 smithi123 bash[19119]: audit 2024-02-12T15:55:36.218058+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 v1:172.21.15.123:6812/1702613957' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:55:37.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:36 smithi121 bash[18489]: cluster 2024-02-12T15:55:35.381805+0000 mgr.y (mgr.14146) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:37.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:36 smithi121 bash[18489]: audit 2024-02-12T15:55:36.212936+0000 mon.a (mon.0) 554 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T15:55:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:36 smithi121 bash[18489]: cluster 2024-02-12T15:55:36.213087+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e54: 8 total, 7 up, 8 in 2024-02-12T15:55:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:36 smithi121 bash[18489]: audit 2024-02-12T15:55:36.213377+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:36 smithi121 bash[18489]: audit 2024-02-12T15:55:36.215623+0000 mon.a (mon.0) 557 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:55:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:36 smithi121 bash[18489]: audit 2024-02-12T15:55:36.218058+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 v1:172.21.15.123:6812/1702613957' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:55:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:36 smithi169 bash[19604]: cluster 2024-02-12T15:55:35.381805+0000 mgr.y (mgr.14146) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:36 smithi169 bash[19604]: audit 2024-02-12T15:55:36.212936+0000 mon.a (mon.0) 554 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T15:55:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:36 smithi169 bash[19604]: cluster 2024-02-12T15:55:36.213087+0000 mon.a (mon.0) 555 : cluster [DBG] osdmap e54: 8 total, 7 up, 8 in 2024-02-12T15:55:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:36 smithi169 bash[19604]: audit 2024-02-12T15:55:36.213377+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:36 smithi169 bash[19604]: audit 2024-02-12T15:55:36.215623+0000 mon.a (mon.0) 557 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:55:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:36 smithi169 bash[19604]: audit 2024-02-12T15:55:36.218058+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 v1:172.21.15.123:6812/1702613957' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T15:55:37.938 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 7 on host 'smithi123' 2024-02-12T15:55:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: cluster 2024-02-12T15:55:36.921819+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: cluster 2024-02-12T15:55:36.921935+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: audit 2024-02-12T15:55:37.219542+0000 mon.a (mon.0) 558 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:55:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: cluster 2024-02-12T15:55:37.219697+0000 mon.a (mon.0) 559 : cluster [DBG] osdmap e55: 8 total, 7 up, 8 in 2024-02-12T15:55:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: audit 2024-02-12T15:55:37.219890+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: audit 2024-02-12T15:55:37.224053+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: cluster 2024-02-12T15:55:37.382331+0000 mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: audit 2024-02-12T15:55:37.931947+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:38 smithi123 bash[19119]: audit 2024-02-12T15:55:37.932662+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:55:38.462 DEBUG:teuthology.orchestra.run.smithi123:osd.7> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.7.service 2024-02-12T15:55:38.464 INFO:tasks.cephadm:Deploying osd.8 on smithi169 with /dev/vg_nvme/lv_4... 2024-02-12T15:55:38.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-12T15:55:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: cluster 2024-02-12T15:55:36.921819+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: cluster 2024-02-12T15:55:36.921935+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: audit 2024-02-12T15:55:37.219542+0000 mon.a (mon.0) 558 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:55:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: cluster 2024-02-12T15:55:37.219697+0000 mon.a (mon.0) 559 : cluster [DBG] osdmap e55: 8 total, 7 up, 8 in 2024-02-12T15:55:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: audit 2024-02-12T15:55:37.219890+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: audit 2024-02-12T15:55:37.224053+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: cluster 2024-02-12T15:55:37.382331+0000 mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: audit 2024-02-12T15:55:37.931947+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:38 smithi169 bash[19604]: audit 2024-02-12T15:55:37.932662+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:55:38.653 INFO:journalctl@ceph.osd.7.smithi123.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:55:38.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: cluster 2024-02-12T15:55:36.921819+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:38.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: cluster 2024-02-12T15:55:36.921935+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: audit 2024-02-12T15:55:37.219542+0000 mon.a (mon.0) 558 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-02-12T15:55:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: cluster 2024-02-12T15:55:37.219697+0000 mon.a (mon.0) 559 : cluster [DBG] osdmap e55: 8 total, 7 up, 8 in 2024-02-12T15:55:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: audit 2024-02-12T15:55:37.219890+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: audit 2024-02-12T15:55:37.224053+0000 mon.a (mon.0) 561 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: cluster 2024-02-12T15:55:37.382331+0000 mgr.y (mgr.14146) 276 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-12T15:55:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: audit 2024-02-12T15:55:37.931947+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:38 smithi121 bash[18489]: audit 2024-02-12T15:55:37.932662+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: cluster 2024-02-12T15:55:36.865403+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: cluster 2024-02-12T15:55:36.865537+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: cluster 2024-02-12T15:55:38.224646+0000 mon.a (mon.0) 564 : cluster [INF] osd.7 v1:172.21.15.123:6812/1702613957 boot 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: cluster 2024-02-12T15:55:38.224726+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: audit 2024-02-12T15:55:38.225272+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: cluster 2024-02-12T15:55:38.782720+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: cluster 2024-02-12T15:55:38.782815+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:39.511 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:39 smithi169 bash[19604]: cluster 2024-02-12T15:55:39.220892+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: cluster 2024-02-12T15:55:36.865403+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: cluster 2024-02-12T15:55:36.865537+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: cluster 2024-02-12T15:55:38.224646+0000 mon.a (mon.0) 564 : cluster [INF] osd.7 v1:172.21.15.123:6812/1702613957 boot 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: cluster 2024-02-12T15:55:38.224726+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: audit 2024-02-12T15:55:38.225272+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: cluster 2024-02-12T15:55:38.782720+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: cluster 2024-02-12T15:55:38.782815+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:39 smithi123 bash[19119]: cluster 2024-02-12T15:55:39.220892+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-12T15:55:39.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: cluster 2024-02-12T15:55:36.865403+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:39.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: cluster 2024-02-12T15:55:36.865537+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:39.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: cluster 2024-02-12T15:55:38.224646+0000 mon.a (mon.0) 564 : cluster [INF] osd.7 v1:172.21.15.123:6812/1702613957 boot 2024-02-12T15:55:39.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: cluster 2024-02-12T15:55:38.224726+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-12T15:55:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: audit 2024-02-12T15:55:38.225272+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T15:55:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: cluster 2024-02-12T15:55:38.782720+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: cluster 2024-02-12T15:55:38.782815+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:39 smithi121 bash[18489]: cluster 2024-02-12T15:55:39.220892+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-12T15:55:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:41 smithi169 bash[19604]: cluster 2024-02-12T15:55:39.382968+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:41 smithi169 bash[19604]: cluster 2024-02-12T15:55:40.224502+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-12T15:55:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:41 smithi123 bash[19119]: cluster 2024-02-12T15:55:39.382968+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:41 smithi123 bash[19119]: cluster 2024-02-12T15:55:40.224502+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-12T15:55:41.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:41 smithi121 bash[18489]: cluster 2024-02-12T15:55:39.382968+0000 mgr.y (mgr.14146) 277 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:41.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:41 smithi121 bash[18489]: cluster 2024-02-12T15:55:40.224502+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-12T15:55:41.556 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-12T15:55:41.556 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-12T15:55:41.556 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-02-12T15:55:41.556 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-02-12T15:55:41.556 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0162241 s, 646 MB/s 2024-02-12T15:55:41.556 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 2024-02-12T15:55:41.556 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-02-12T15:55:42.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2024-02-12T15:55:42.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:42 smithi169 bash[19604]: cluster 2024-02-12T15:55:41.750375+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:55:42.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:42 smithi169 bash[19604]: cluster 2024-02-12T15:55:41.754102+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:55:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:42 smithi123 bash[19119]: cluster 2024-02-12T15:55:41.750375+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:55:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:42 smithi123 bash[19119]: cluster 2024-02-12T15:55:41.754102+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:55:42.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:42 smithi121 bash[18489]: cluster 2024-02-12T15:55:41.750375+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:55:42.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:42 smithi121 bash[18489]: cluster 2024-02-12T15:55:41.754102+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:55:43.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:43 smithi169 bash[19604]: cluster 2024-02-12T15:55:41.383602+0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:43.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:43 smithi123 bash[19119]: cluster 2024-02-12T15:55:41.383602+0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:43 smithi121 bash[18489]: cluster 2024-02-12T15:55:41.383602+0000 mgr.y (mgr.14146) 278 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:44 smithi169 bash[19604]: cluster 2024-02-12T15:55:43.384163+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:44 smithi169 bash[19604]: cluster 2024-02-12T15:55:43.425546+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:44 smithi169 bash[19604]: cluster 2024-02-12T15:55:43.425635+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:44 smithi123 bash[19119]: cluster 2024-02-12T15:55:43.384163+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:44 smithi123 bash[19119]: cluster 2024-02-12T15:55:43.425546+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:44.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:44 smithi123 bash[19119]: cluster 2024-02-12T15:55:43.425635+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:44.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:44 smithi121 bash[18489]: cluster 2024-02-12T15:55:43.384163+0000 mgr.y (mgr.14146) 279 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:44.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:44 smithi121 bash[18489]: cluster 2024-02-12T15:55:43.425546+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:44.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:44 smithi121 bash[18489]: cluster 2024-02-12T15:55:43.425635+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:45.337 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:45 smithi123 bash[19119]: cluster 2024-02-12T15:55:44.043995+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-12T15:55:45.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:45 smithi121 bash[18489]: cluster 2024-02-12T15:55:44.043995+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-12T15:55:45.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:45 smithi169 bash[19604]: cluster 2024-02-12T15:55:44.043995+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-12T15:55:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:46 smithi123 bash[19119]: audit 2024-02-12T15:55:45.218745+0000 mgr.y (mgr.14146) 280 : audit [DBG] from='client.24334 v1:172.21.15.169:0/2883766835' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:55:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:46 smithi123 bash[19119]: audit 2024-02-12T15:55:45.226409+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:55:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:46 smithi123 bash[19119]: audit 2024-02-12T15:55:45.230590+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:55:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:46 smithi123 bash[19119]: audit 2024-02-12T15:55:45.231735+0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:46 smithi123 bash[19119]: cluster 2024-02-12T15:55:45.384701+0000 mgr.y (mgr.14146) 281 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:46 smithi121 bash[18489]: audit 2024-02-12T15:55:45.218745+0000 mgr.y (mgr.14146) 280 : audit [DBG] from='client.24334 v1:172.21.15.169:0/2883766835' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:55:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:46 smithi121 bash[18489]: audit 2024-02-12T15:55:45.226409+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:55:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:46 smithi121 bash[18489]: audit 2024-02-12T15:55:45.230590+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:55:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:46 smithi121 bash[18489]: audit 2024-02-12T15:55:45.231735+0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:46 smithi121 bash[18489]: cluster 2024-02-12T15:55:45.384701+0000 mgr.y (mgr.14146) 281 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:46 smithi169 bash[19604]: audit 2024-02-12T15:55:45.218745+0000 mgr.y (mgr.14146) 280 : audit [DBG] from='client.24334 v1:172.21.15.169:0/2883766835' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:55:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:46 smithi169 bash[19604]: audit 2024-02-12T15:55:45.226409+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:55:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:46 smithi169 bash[19604]: audit 2024-02-12T15:55:45.230590+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:55:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:46 smithi169 bash[19604]: audit 2024-02-12T15:55:45.231735+0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:46 smithi169 bash[19604]: cluster 2024-02-12T15:55:45.384701+0000 mgr.y (mgr.14146) 281 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:48.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: cephadm 2024-02-12T15:55:46.952718+0000 mgr.y (mgr.14146) 282 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.963315+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.964966+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.966817+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.968566+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.970259+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: cephadm 2024-02-12T15:55:46.971524+0000 mgr.y (mgr.14146) 283 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 4329M 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.980212+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.981846+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.983608+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:47 smithi169 bash[19604]: audit 2024-02-12T15:55:46.992542+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: cephadm 2024-02-12T15:55:46.952718+0000 mgr.y (mgr.14146) 282 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:55:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.963315+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.964966+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.966817+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.968566+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.970259+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: cephadm 2024-02-12T15:55:46.971524+0000 mgr.y (mgr.14146) 283 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 4329M 2024-02-12T15:55:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.980212+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.981846+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.983608+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:47 smithi123 bash[19119]: audit 2024-02-12T15:55:46.992542+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: cephadm 2024-02-12T15:55:46.952718+0000 mgr.y (mgr.14146) 282 : cephadm [INF] Detected new or changed devices on smithi123 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.963315+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.964966+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.966817+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.968566+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.970259+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: cephadm 2024-02-12T15:55:46.971524+0000 mgr.y (mgr.14146) 283 : cephadm [INF] Adjusting osd_memory_target on smithi123 to 4329M 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.980212+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.981846+0000 mon.a (mon.0) 579 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.983608+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:55:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:47 smithi121 bash[18489]: audit 2024-02-12T15:55:46.992542+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:55:49.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:48 smithi169 bash[19604]: cluster 2024-02-12T15:55:47.385252+0000 mgr.y (mgr.14146) 284 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:49.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:48 smithi123 bash[19119]: cluster 2024-02-12T15:55:47.385252+0000 mgr.y (mgr.14146) 284 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:48 smithi121 bash[18489]: cluster 2024-02-12T15:55:47.385252+0000 mgr.y (mgr.14146) 284 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:51.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:51 smithi169 bash[19604]: cluster 2024-02-12T15:55:49.385845+0000 mgr.y (mgr.14146) 285 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:51.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:51 smithi123 bash[19119]: cluster 2024-02-12T15:55:49.385845+0000 mgr.y (mgr.14146) 285 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:51.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:51 smithi121 bash[18489]: cluster 2024-02-12T15:55:49.385845+0000 mgr.y (mgr.14146) 285 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:53.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:53 smithi169 bash[19604]: cluster 2024-02-12T15:55:51.386418+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:53 smithi123 bash[19119]: cluster 2024-02-12T15:55:51.386418+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:53 smithi121 bash[18489]: cluster 2024-02-12T15:55:51.386418+0000 mgr.y (mgr.14146) 286 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:54.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:54 smithi169 bash[19604]: audit 2024-02-12T15:55:53.271768+0000 mon.a (mon.0) 582 : audit [INF] from='client.24340 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]: dispatch 2024-02-12T15:55:54.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:54 smithi169 bash[19604]: audit 2024-02-12T15:55:53.272953+0000 mon.c (mon.1) 5 : audit [INF] from='client.? v1:172.21.15.169:0/4013194757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]: dispatch 2024-02-12T15:55:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:54 smithi169 bash[19604]: audit 2024-02-12T15:55:53.280613+0000 mon.a (mon.0) 583 : audit [INF] from='client.24340 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]': finished 2024-02-12T15:55:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:54 smithi169 bash[19604]: cluster 2024-02-12T15:55:53.280714+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e60: 9 total, 8 up, 9 in 2024-02-12T15:55:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:54 smithi169 bash[19604]: audit 2024-02-12T15:55:53.280872+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:55:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:54 smithi169 bash[19604]: audit 2024-02-12T15:55:53.871884+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? v1:172.21.15.169:0/2658598353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:55:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:54 smithi123 bash[19119]: audit 2024-02-12T15:55:53.271768+0000 mon.a (mon.0) 582 : audit [INF] from='client.24340 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]: dispatch 2024-02-12T15:55:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:54 smithi123 bash[19119]: audit 2024-02-12T15:55:53.272953+0000 mon.c (mon.1) 5 : audit [INF] from='client.? v1:172.21.15.169:0/4013194757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]: dispatch 2024-02-12T15:55:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:54 smithi123 bash[19119]: audit 2024-02-12T15:55:53.280613+0000 mon.a (mon.0) 583 : audit [INF] from='client.24340 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]': finished 2024-02-12T15:55:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:54 smithi123 bash[19119]: cluster 2024-02-12T15:55:53.280714+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e60: 9 total, 8 up, 9 in 2024-02-12T15:55:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:54 smithi123 bash[19119]: audit 2024-02-12T15:55:53.280872+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:55:54.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:54 smithi123 bash[19119]: audit 2024-02-12T15:55:53.871884+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? v1:172.21.15.169:0/2658598353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:55:54.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:54 smithi121 bash[18489]: audit 2024-02-12T15:55:53.271768+0000 mon.a (mon.0) 582 : audit [INF] from='client.24340 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]: dispatch 2024-02-12T15:55:54.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:54 smithi121 bash[18489]: audit 2024-02-12T15:55:53.272953+0000 mon.c (mon.1) 5 : audit [INF] from='client.? v1:172.21.15.169:0/4013194757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]: dispatch 2024-02-12T15:55:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:54 smithi121 bash[18489]: audit 2024-02-12T15:55:53.280613+0000 mon.a (mon.0) 583 : audit [INF] from='client.24340 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75"}]': finished 2024-02-12T15:55:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:54 smithi121 bash[18489]: cluster 2024-02-12T15:55:53.280714+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e60: 9 total, 8 up, 9 in 2024-02-12T15:55:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:54 smithi121 bash[18489]: audit 2024-02-12T15:55:53.280872+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:55:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:54 smithi121 bash[18489]: audit 2024-02-12T15:55:53.871884+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? v1:172.21.15.169:0/2658598353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:55:55.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:55 smithi169 bash[19604]: cluster 2024-02-12T15:55:53.386964+0000 mgr.y (mgr.14146) 287 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:55 smithi123 bash[19119]: cluster 2024-02-12T15:55:53.386964+0000 mgr.y (mgr.14146) 287 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:55.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:55 smithi121 bash[18489]: cluster 2024-02-12T15:55:53.386964+0000 mgr.y (mgr.14146) 287 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:56.331 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:56 smithi169 bash[19604]: cluster 2024-02-12T15:55:55.387576+0000 mgr.y (mgr.14146) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:56 smithi123 bash[19119]: cluster 2024-02-12T15:55:55.387576+0000 mgr.y (mgr.14146) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:56.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:56 smithi121 bash[18489]: cluster 2024-02-12T15:55:55.387576+0000 mgr.y (mgr.14146) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:58 smithi123 bash[19119]: cluster 2024-02-12T15:55:56.777005+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:58 smithi123 bash[19119]: cluster 2024-02-12T15:55:56.777100+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:58 smithi123 bash[19119]: cluster 2024-02-12T15:55:57.050611+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e61: 9 total, 8 up, 9 in 2024-02-12T15:55:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:58 smithi123 bash[19119]: audit 2024-02-12T15:55:57.050822+0000 mon.a (mon.0) 587 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:55:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:55:58 smithi123 bash[19119]: cluster 2024-02-12T15:55:57.388155+0000 mgr.y (mgr.14146) 289 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:58 smithi121 bash[18489]: cluster 2024-02-12T15:55:56.777005+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:58 smithi121 bash[18489]: cluster 2024-02-12T15:55:56.777100+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:58 smithi121 bash[18489]: cluster 2024-02-12T15:55:57.050611+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e61: 9 total, 8 up, 9 in 2024-02-12T15:55:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:58 smithi121 bash[18489]: audit 2024-02-12T15:55:57.050822+0000 mon.a (mon.0) 587 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:55:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:55:58 smithi121 bash[18489]: cluster 2024-02-12T15:55:57.388155+0000 mgr.y (mgr.14146) 289 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:55:58.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:58 smithi169 bash[19604]: cluster 2024-02-12T15:55:56.777005+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:55:58.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:58 smithi169 bash[19604]: cluster 2024-02-12T15:55:56.777100+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:55:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:58 smithi169 bash[19604]: cluster 2024-02-12T15:55:57.050611+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e61: 9 total, 8 up, 9 in 2024-02-12T15:55:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:58 smithi169 bash[19604]: audit 2024-02-12T15:55:57.050822+0000 mon.a (mon.0) 587 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:55:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:55:58 smithi169 bash[19604]: cluster 2024-02-12T15:55:57.388155+0000 mgr.y (mgr.14146) 289 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:01 smithi169 bash[19604]: cluster 2024-02-12T15:55:59.388739+0000 mgr.y (mgr.14146) 290 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:01 smithi123 bash[19119]: cluster 2024-02-12T15:55:59.388739+0000 mgr.y (mgr.14146) 290 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:01 smithi121 bash[18489]: cluster 2024-02-12T15:55:59.388739+0000 mgr.y (mgr.14146) 290 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:03.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:03 smithi169 bash[19604]: cluster 2024-02-12T15:56:01.389301+0000 mgr.y (mgr.14146) 291 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:03 smithi123 bash[19119]: cluster 2024-02-12T15:56:01.389301+0000 mgr.y (mgr.14146) 291 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:03.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:03 smithi121 bash[18489]: cluster 2024-02-12T15:56:01.389301+0000 mgr.y (mgr.14146) 291 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:04 smithi169 bash[19604]: cluster 2024-02-12T15:56:03.389880+0000 mgr.y (mgr.14146) 292 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:04.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:04 smithi123 bash[19119]: cluster 2024-02-12T15:56:03.389880+0000 mgr.y (mgr.14146) 292 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:04 smithi121 bash[18489]: cluster 2024-02-12T15:56:03.389880+0000 mgr.y (mgr.14146) 292 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:06 smithi169 bash[19604]: audit 2024-02-12T15:56:05.244921+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-12T15:56:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:06 smithi169 bash[19604]: audit 2024-02-12T15:56:05.246695+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:06 smithi123 bash[19119]: audit 2024-02-12T15:56:05.244921+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-12T15:56:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:06 smithi123 bash[19119]: audit 2024-02-12T15:56:05.246695+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:06 smithi121 bash[18489]: audit 2024-02-12T15:56:05.244921+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-12T15:56:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:06 smithi121 bash[18489]: audit 2024-02-12T15:56:05.246695+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:07 smithi169 bash[19604]: cephadm 2024-02-12T15:56:05.248316+0000 mgr.y (mgr.14146) 293 : cephadm [INF] Deploying daemon osd.8 on smithi169 2024-02-12T15:56:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:07 smithi169 bash[19604]: cluster 2024-02-12T15:56:05.390427+0000 mgr.y (mgr.14146) 294 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:07 smithi123 bash[19119]: cephadm 2024-02-12T15:56:05.248316+0000 mgr.y (mgr.14146) 293 : cephadm [INF] Deploying daemon osd.8 on smithi169 2024-02-12T15:56:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:07 smithi123 bash[19119]: cluster 2024-02-12T15:56:05.390427+0000 mgr.y (mgr.14146) 294 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:07 smithi121 bash[18489]: cephadm 2024-02-12T15:56:05.248316+0000 mgr.y (mgr.14146) 293 : cephadm [INF] Deploying daemon osd.8 on smithi169 2024-02-12T15:56:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:07 smithi121 bash[18489]: cluster 2024-02-12T15:56:05.390427+0000 mgr.y (mgr.14146) 294 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:08.232 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:08 smithi169 bash[19604]: cluster 2024-02-12T15:56:07.390977+0000 mgr.y (mgr.14146) 295 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:08.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:08 smithi123 bash[19119]: cluster 2024-02-12T15:56:07.390977+0000 mgr.y (mgr.14146) 295 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:08 smithi121 bash[18489]: cluster 2024-02-12T15:56:07.390977+0000 mgr.y (mgr.14146) 295 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:10.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:10 smithi123 bash[19119]: audit 2024-02-12T15:56:09.067859+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:10.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:10 smithi123 bash[19119]: audit 2024-02-12T15:56:09.069933+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:10.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:10 smithi123 bash[19119]: cluster 2024-02-12T15:56:09.391564+0000 mgr.y (mgr.14146) 296 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:10 smithi121 bash[18489]: audit 2024-02-12T15:56:09.067859+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:10 smithi121 bash[18489]: audit 2024-02-12T15:56:09.069933+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:10 smithi121 bash[18489]: cluster 2024-02-12T15:56:09.391564+0000 mgr.y (mgr.14146) 296 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:10.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:10 smithi169 bash[19604]: audit 2024-02-12T15:56:09.067859+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:10.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:10 smithi169 bash[19604]: audit 2024-02-12T15:56:09.069933+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:10.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:10 smithi169 bash[19604]: cluster 2024-02-12T15:56:09.391564+0000 mgr.y (mgr.14146) 296 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:11 smithi123 bash[19119]: cluster 2024-02-12T15:56:10.087462+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:11.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:11 smithi123 bash[19119]: cluster 2024-02-12T15:56:10.087575+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:11.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:11 smithi121 bash[18489]: cluster 2024-02-12T15:56:10.087462+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:11 smithi121 bash[18489]: cluster 2024-02-12T15:56:10.087575+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:11.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:11 smithi169 bash[19604]: cluster 2024-02-12T15:56:10.087462+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:11.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:11 smithi169 bash[19604]: cluster 2024-02-12T15:56:10.087575+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:12.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:12 smithi123 bash[19119]: cluster 2024-02-12T15:56:11.085957+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e62: 9 total, 8 up, 9 in 2024-02-12T15:56:12.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:12 smithi123 bash[19119]: audit 2024-02-12T15:56:11.086271+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:12.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:12 smithi123 bash[19119]: cluster 2024-02-12T15:56:11.392097+0000 mgr.y (mgr.14146) 297 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:12 smithi121 bash[18489]: cluster 2024-02-12T15:56:11.085957+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e62: 9 total, 8 up, 9 in 2024-02-12T15:56:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:12 smithi121 bash[18489]: audit 2024-02-12T15:56:11.086271+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:12 smithi121 bash[18489]: cluster 2024-02-12T15:56:11.392097+0000 mgr.y (mgr.14146) 297 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:12.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:12 smithi169 bash[19604]: cluster 2024-02-12T15:56:11.085957+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e62: 9 total, 8 up, 9 in 2024-02-12T15:56:12.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:12 smithi169 bash[19604]: audit 2024-02-12T15:56:11.086271+0000 mon.a (mon.0) 593 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:12.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:12 smithi169 bash[19604]: cluster 2024-02-12T15:56:11.392097+0000 mgr.y (mgr.14146) 297 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:14 smithi169 bash[19604]: cluster 2024-02-12T15:56:13.392663+0000 mgr.y (mgr.14146) 298 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:14 smithi123 bash[19119]: cluster 2024-02-12T15:56:13.392663+0000 mgr.y (mgr.14146) 298 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:14 smithi121 bash[18489]: cluster 2024-02-12T15:56:13.392663+0000 mgr.y (mgr.14146) 298 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:16 smithi169 bash[19604]: audit 2024-02-12T15:56:15.262762+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:16 smithi169 bash[19604]: cluster 2024-02-12T15:56:15.393250+0000 mgr.y (mgr.14146) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:16 smithi169 bash[19604]: audit 2024-02-12T15:56:15.539714+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:16 smithi169 bash[19604]: audit 2024-02-12T15:56:15.541431+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:16 smithi169 bash[19604]: audit 2024-02-12T15:56:15.552782+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:16 smithi169 bash[19604]: audit 2024-02-12T15:56:15.749598+0000 mon.a (mon.0) 598 : audit [INF] from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-12T15:56:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:16 smithi169 bash[19604]: audit 2024-02-12T15:56:15.751247+0000 mon.c (mon.1) 7 : audit [INF] from='osd.8 v1:172.21.15.169:6800/2551075225' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-12T15:56:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:16 smithi123 bash[19119]: audit 2024-02-12T15:56:15.262762+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:16 smithi123 bash[19119]: cluster 2024-02-12T15:56:15.393250+0000 mgr.y (mgr.14146) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:16 smithi123 bash[19119]: audit 2024-02-12T15:56:15.539714+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:16 smithi123 bash[19119]: audit 2024-02-12T15:56:15.541431+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:16 smithi123 bash[19119]: audit 2024-02-12T15:56:15.552782+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:16 smithi123 bash[19119]: audit 2024-02-12T15:56:15.749598+0000 mon.a (mon.0) 598 : audit [INF] from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-12T15:56:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:16 smithi123 bash[19119]: audit 2024-02-12T15:56:15.751247+0000 mon.c (mon.1) 7 : audit [INF] from='osd.8 v1:172.21.15.169:6800/2551075225' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-12T15:56:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:16 smithi121 bash[18489]: audit 2024-02-12T15:56:15.262762+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:16 smithi121 bash[18489]: cluster 2024-02-12T15:56:15.393250+0000 mgr.y (mgr.14146) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:16 smithi121 bash[18489]: audit 2024-02-12T15:56:15.539714+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:16 smithi121 bash[18489]: audit 2024-02-12T15:56:15.541431+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:16 smithi121 bash[18489]: audit 2024-02-12T15:56:15.552782+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:16 smithi121 bash[18489]: audit 2024-02-12T15:56:15.749598+0000 mon.a (mon.0) 598 : audit [INF] from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-12T15:56:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:16 smithi121 bash[18489]: audit 2024-02-12T15:56:15.751247+0000 mon.c (mon.1) 7 : audit [INF] from='osd.8 v1:172.21.15.169:6800/2551075225' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-12T15:56:17.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:17 smithi169 bash[19604]: audit 2024-02-12T15:56:16.559078+0000 mon.a (mon.0) 599 : audit [INF] from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-12T15:56:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:17 smithi169 bash[19604]: cluster 2024-02-12T15:56:16.559234+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e63: 9 total, 8 up, 9 in 2024-02-12T15:56:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:17 smithi169 bash[19604]: audit 2024-02-12T15:56:16.559480+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:17 smithi169 bash[19604]: audit 2024-02-12T15:56:16.560969+0000 mon.a (mon.0) 602 : audit [INF] from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:17 smithi169 bash[19604]: audit 2024-02-12T15:56:16.562491+0000 mon.c (mon.1) 8 : audit [INF] from='osd.8 v1:172.21.15.169:6800/2551075225' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:17.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:17 smithi123 bash[19119]: audit 2024-02-12T15:56:16.559078+0000 mon.a (mon.0) 599 : audit [INF] from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-12T15:56:17.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:17 smithi123 bash[19119]: cluster 2024-02-12T15:56:16.559234+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e63: 9 total, 8 up, 9 in 2024-02-12T15:56:17.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:17 smithi123 bash[19119]: audit 2024-02-12T15:56:16.559480+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:17.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:17 smithi123 bash[19119]: audit 2024-02-12T15:56:16.560969+0000 mon.a (mon.0) 602 : audit [INF] from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:17.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:17 smithi123 bash[19119]: audit 2024-02-12T15:56:16.562491+0000 mon.c (mon.1) 8 : audit [INF] from='osd.8 v1:172.21.15.169:6800/2551075225' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:17 smithi121 bash[18489]: audit 2024-02-12T15:56:16.559078+0000 mon.a (mon.0) 599 : audit [INF] from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-12T15:56:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:17 smithi121 bash[18489]: cluster 2024-02-12T15:56:16.559234+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e63: 9 total, 8 up, 9 in 2024-02-12T15:56:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:17 smithi121 bash[18489]: audit 2024-02-12T15:56:16.559480+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:17 smithi121 bash[18489]: audit 2024-02-12T15:56:16.560969+0000 mon.a (mon.0) 602 : audit [INF] from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:17 smithi121 bash[18489]: audit 2024-02-12T15:56:16.562491+0000 mon.c (mon.1) 8 : audit [INF] from='osd.8 v1:172.21.15.169:6800/2551075225' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:18.503 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 8 on host 'smithi169' 2024-02-12T15:56:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:18 smithi123 bash[19119]: cluster 2024-02-12T15:56:17.393788+0000 mgr.y (mgr.14146) 300 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:18 smithi123 bash[19119]: audit 2024-02-12T15:56:17.567238+0000 mon.a (mon.0) 603 : audit [INF] from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:56:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:18 smithi123 bash[19119]: cluster 2024-02-12T15:56:17.567472+0000 mon.a (mon.0) 604 : cluster [DBG] osdmap e64: 9 total, 8 up, 9 in 2024-02-12T15:56:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:18 smithi123 bash[19119]: audit 2024-02-12T15:56:17.568019+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:18 smithi123 bash[19119]: audit 2024-02-12T15:56:17.574137+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:18.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:18 smithi123 bash[19119]: audit 2024-02-12T15:56:18.496836+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:18.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:18 smithi123 bash[19119]: audit 2024-02-12T15:56:18.528897+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:18 smithi121 bash[18489]: cluster 2024-02-12T15:56:17.393788+0000 mgr.y (mgr.14146) 300 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:18 smithi121 bash[18489]: audit 2024-02-12T15:56:17.567238+0000 mon.a (mon.0) 603 : audit [INF] from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:56:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:18 smithi121 bash[18489]: cluster 2024-02-12T15:56:17.567472+0000 mon.a (mon.0) 604 : cluster [DBG] osdmap e64: 9 total, 8 up, 9 in 2024-02-12T15:56:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:18 smithi121 bash[18489]: audit 2024-02-12T15:56:17.568019+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:18 smithi121 bash[18489]: audit 2024-02-12T15:56:17.574137+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:18 smithi121 bash[18489]: audit 2024-02-12T15:56:18.496836+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:18 smithi121 bash[18489]: audit 2024-02-12T15:56:18.528897+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:18 smithi169 bash[19604]: cluster 2024-02-12T15:56:17.393788+0000 mgr.y (mgr.14146) 300 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-12T15:56:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:18 smithi169 bash[19604]: audit 2024-02-12T15:56:17.567238+0000 mon.a (mon.0) 603 : audit [INF] from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:56:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:18 smithi169 bash[19604]: cluster 2024-02-12T15:56:17.567472+0000 mon.a (mon.0) 604 : cluster [DBG] osdmap e64: 9 total, 8 up, 9 in 2024-02-12T15:56:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:18 smithi169 bash[19604]: audit 2024-02-12T15:56:17.568019+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:18 smithi169 bash[19604]: audit 2024-02-12T15:56:17.574137+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:18 smithi169 bash[19604]: audit 2024-02-12T15:56:18.496836+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:18 smithi169 bash[19604]: audit 2024-02-12T15:56:18.528897+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:19.149 DEBUG:teuthology.orchestra.run.smithi169:osd.8> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.8.service 2024-02-12T15:56:19.152 INFO:tasks.cephadm:Deploying osd.9 on smithi169 with /dev/vg_nvme/lv_3... 2024-02-12T15:56:19.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-12T15:56:19.484 INFO:journalctl@ceph.osd.8.smithi169.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:56:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:19 smithi123 bash[19119]: cluster 2024-02-12T15:56:16.766844+0000 osd.8 (osd.8) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:19 smithi123 bash[19119]: cluster 2024-02-12T15:56:16.766951+0000 osd.8 (osd.8) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:19 smithi123 bash[19119]: cluster 2024-02-12T15:56:18.568814+0000 mon.a (mon.0) 609 : cluster [INF] osd.8 v1:172.21.15.169:6800/2551075225 boot 2024-02-12T15:56:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:19 smithi123 bash[19119]: cluster 2024-02-12T15:56:18.568920+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e65: 9 total, 9 up, 9 in 2024-02-12T15:56:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:19 smithi123 bash[19119]: audit 2024-02-12T15:56:18.569638+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:19.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:19 smithi121 bash[18489]: cluster 2024-02-12T15:56:16.766844+0000 osd.8 (osd.8) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:20.022 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:19 smithi121 bash[18489]: cluster 2024-02-12T15:56:16.766951+0000 osd.8 (osd.8) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:20.022 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:19 smithi121 bash[18489]: cluster 2024-02-12T15:56:18.568814+0000 mon.a (mon.0) 609 : cluster [INF] osd.8 v1:172.21.15.169:6800/2551075225 boot 2024-02-12T15:56:20.022 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:19 smithi121 bash[18489]: cluster 2024-02-12T15:56:18.568920+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e65: 9 total, 9 up, 9 in 2024-02-12T15:56:20.022 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:19 smithi121 bash[18489]: audit 2024-02-12T15:56:18.569638+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:19 smithi169 bash[19604]: cluster 2024-02-12T15:56:16.766844+0000 osd.8 (osd.8) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:19 smithi169 bash[19604]: cluster 2024-02-12T15:56:16.766951+0000 osd.8 (osd.8) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:19 smithi169 bash[19604]: cluster 2024-02-12T15:56:18.568814+0000 mon.a (mon.0) 609 : cluster [INF] osd.8 v1:172.21.15.169:6800/2551075225 boot 2024-02-12T15:56:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:19 smithi169 bash[19604]: cluster 2024-02-12T15:56:18.568920+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e65: 9 total, 9 up, 9 in 2024-02-12T15:56:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:19 smithi169 bash[19604]: audit 2024-02-12T15:56:18.569638+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-12T15:56:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:20 smithi123 bash[19119]: cluster 2024-02-12T15:56:19.394353+0000 mgr.y (mgr.14146) 301 : cluster [DBG] pgmap v296: 1 pgs: 1 peering; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:20 smithi123 bash[19119]: cluster 2024-02-12T15:56:19.587173+0000 mon.a (mon.0) 612 : cluster [DBG] osdmap e66: 9 total, 9 up, 9 in 2024-02-12T15:56:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:20 smithi121 bash[18489]: cluster 2024-02-12T15:56:19.394353+0000 mgr.y (mgr.14146) 301 : cluster [DBG] pgmap v296: 1 pgs: 1 peering; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:20 smithi121 bash[18489]: cluster 2024-02-12T15:56:19.587173+0000 mon.a (mon.0) 612 : cluster [DBG] osdmap e66: 9 total, 9 up, 9 in 2024-02-12T15:56:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:20 smithi169 bash[19604]: cluster 2024-02-12T15:56:19.394353+0000 mgr.y (mgr.14146) 301 : cluster [DBG] pgmap v296: 1 pgs: 1 peering; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:20 smithi169 bash[19604]: cluster 2024-02-12T15:56:19.587173+0000 mon.a (mon.0) 612 : cluster [DBG] osdmap e66: 9 total, 9 up, 9 in 2024-02-12T15:56:21.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:21 smithi123 bash[19119]: cluster 2024-02-12T15:56:20.587587+0000 mon.a (mon.0) 613 : cluster [DBG] osdmap e67: 9 total, 9 up, 9 in 2024-02-12T15:56:21.894 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:21 smithi169 bash[19604]: cluster 2024-02-12T15:56:20.587587+0000 mon.a (mon.0) 613 : cluster [DBG] osdmap e67: 9 total, 9 up, 9 in 2024-02-12T15:56:21.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:21 smithi121 bash[18489]: cluster 2024-02-12T15:56:20.587587+0000 mon.a (mon.0) 613 : cluster [DBG] osdmap e67: 9 total, 9 up, 9 in 2024-02-12T15:56:22.420 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-12T15:56:22.421 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-12T15:56:22.421 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-02-12T15:56:22.421 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-02-12T15:56:22.421 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0228168 s, 460 MB/s 2024-02-12T15:56:22.421 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-02-12T15:56:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:22 smithi169 bash[19604]: cluster 2024-02-12T15:56:21.394957+0000 mgr.y (mgr.14146) 302 : cluster [DBG] pgmap v299: 1 pgs: 1 peering; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:22 smithi123 bash[19119]: cluster 2024-02-12T15:56:21.394957+0000 mgr.y (mgr.14146) 302 : cluster [DBG] pgmap v299: 1 pgs: 1 peering; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:22 smithi121 bash[18489]: cluster 2024-02-12T15:56:21.394957+0000 mgr.y (mgr.14146) 302 : cluster [DBG] pgmap v299: 1 pgs: 1 peering; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:23.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2024-02-12T15:56:24.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:24 smithi169 bash[19604]: cluster 2024-02-12T15:56:23.395607+0000 mgr.y (mgr.14146) 303 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:24 smithi123 bash[19119]: cluster 2024-02-12T15:56:23.395607+0000 mgr.y (mgr.14146) 303 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:24 smithi121 bash[18489]: cluster 2024-02-12T15:56:23.395607+0000 mgr.y (mgr.14146) 303 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:26.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:26 smithi123 bash[19119]: cluster 2024-02-12T15:56:25.396195+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:26 smithi121 bash[18489]: cluster 2024-02-12T15:56:25.396195+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:26.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:26 smithi169 bash[19604]: cluster 2024-02-12T15:56:25.396195+0000 mgr.y (mgr.14146) 304 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:27.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:27 smithi123 bash[19119]: audit 2024-02-12T15:56:27.255483+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:56:27.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:27 smithi123 bash[19119]: audit 2024-02-12T15:56:27.261508+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:56:27.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:27 smithi123 bash[19119]: audit 2024-02-12T15:56:27.263095+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:27.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:27 smithi121 bash[18489]: audit 2024-02-12T15:56:27.255483+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:56:27.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:27 smithi121 bash[18489]: audit 2024-02-12T15:56:27.261508+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:56:27.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:27 smithi121 bash[18489]: audit 2024-02-12T15:56:27.263095+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:27.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:27 smithi169 bash[19604]: audit 2024-02-12T15:56:27.255483+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:56:27.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:27 smithi169 bash[19604]: audit 2024-02-12T15:56:27.261508+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:56:27.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:27 smithi169 bash[19604]: audit 2024-02-12T15:56:27.263095+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:28.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:28 smithi123 bash[19119]: audit 2024-02-12T15:56:27.248503+0000 mgr.y (mgr.14146) 305 : audit [DBG] from='client.24361 v1:172.21.15.169:0/402518533' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:56:28.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:28 smithi123 bash[19119]: cluster 2024-02-12T15:56:27.396758+0000 mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:28.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:28 smithi121 bash[18489]: audit 2024-02-12T15:56:27.248503+0000 mgr.y (mgr.14146) 305 : audit [DBG] from='client.24361 v1:172.21.15.169:0/402518533' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:56:28.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:28 smithi121 bash[18489]: cluster 2024-02-12T15:56:27.396758+0000 mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:28.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:28 smithi169 bash[19604]: audit 2024-02-12T15:56:27.248503+0000 mgr.y (mgr.14146) 305 : audit [DBG] from='client.24361 v1:172.21.15.169:0/402518533' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:56:28.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:28 smithi169 bash[19604]: cluster 2024-02-12T15:56:27.396758+0000 mgr.y (mgr.14146) 306 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: cephadm 2024-02-12T15:56:28.616099+0000 mgr.y (mgr.14146) 307 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:56:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: audit 2024-02-12T15:56:28.624765+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: audit 2024-02-12T15:56:28.626648+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:56:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: cephadm 2024-02-12T15:56:28.628123+0000 mgr.y (mgr.14146) 308 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 21414M 2024-02-12T15:56:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: audit 2024-02-12T15:56:28.635497+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: audit 2024-02-12T15:56:28.637149+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: audit 2024-02-12T15:56:28.638434+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:29 smithi123 bash[19119]: audit 2024-02-12T15:56:28.647227+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: cephadm 2024-02-12T15:56:28.616099+0000 mgr.y (mgr.14146) 307 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: audit 2024-02-12T15:56:28.624765+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: audit 2024-02-12T15:56:28.626648+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: cephadm 2024-02-12T15:56:28.628123+0000 mgr.y (mgr.14146) 308 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 21414M 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: audit 2024-02-12T15:56:28.635497+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: audit 2024-02-12T15:56:28.637149+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: audit 2024-02-12T15:56:28.638434+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:29 smithi121 bash[18489]: audit 2024-02-12T15:56:28.647227+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:30.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: cephadm 2024-02-12T15:56:28.616099+0000 mgr.y (mgr.14146) 307 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:56:30.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: audit 2024-02-12T15:56:28.624765+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: audit 2024-02-12T15:56:28.626648+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:56:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: cephadm 2024-02-12T15:56:28.628123+0000 mgr.y (mgr.14146) 308 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 21414M 2024-02-12T15:56:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: audit 2024-02-12T15:56:28.635497+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: audit 2024-02-12T15:56:28.637149+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: audit 2024-02-12T15:56:28.638434+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:29 smithi169 bash[19604]: audit 2024-02-12T15:56:28.647227+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T15:56:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:31 smithi169 bash[19604]: cluster 2024-02-12T15:56:29.397339+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:31.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:31 smithi123 bash[19119]: cluster 2024-02-12T15:56:29.397339+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:31 smithi121 bash[18489]: cluster 2024-02-12T15:56:29.397339+0000 mgr.y (mgr.14146) 309 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:32 smithi123 bash[19119]: cluster 2024-02-12T15:56:31.397915+0000 mgr.y (mgr.14146) 310 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:32 smithi121 bash[18489]: cluster 2024-02-12T15:56:31.397915+0000 mgr.y (mgr.14146) 310 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:32.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:32 smithi169 bash[19604]: cluster 2024-02-12T15:56:31.397915+0000 mgr.y (mgr.14146) 310 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:34 smithi169 bash[19604]: cluster 2024-02-12T15:56:33.398488+0000 mgr.y (mgr.14146) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:34 smithi123 bash[19119]: cluster 2024-02-12T15:56:33.398488+0000 mgr.y (mgr.14146) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:34 smithi121 bash[18489]: cluster 2024-02-12T15:56:33.398488+0000 mgr.y (mgr.14146) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:36 smithi123 bash[19119]: audit 2024-02-12T15:56:35.282672+0000 mon.a (mon.0) 623 : audit [INF] from='client.24367 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]: dispatch 2024-02-12T15:56:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:36 smithi123 bash[19119]: audit 2024-02-12T15:56:35.284314+0000 mon.c (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.169:0/3842225105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]: dispatch 2024-02-12T15:56:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:36 smithi123 bash[19119]: audit 2024-02-12T15:56:35.291864+0000 mon.a (mon.0) 624 : audit [INF] from='client.24367 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]': finished 2024-02-12T15:56:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:36 smithi123 bash[19119]: cluster 2024-02-12T15:56:35.292089+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e68: 10 total, 9 up, 10 in 2024-02-12T15:56:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:36 smithi123 bash[19119]: audit 2024-02-12T15:56:35.292307+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:36 smithi123 bash[19119]: cluster 2024-02-12T15:56:35.399085+0000 mgr.y (mgr.14146) 312 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:36 smithi123 bash[19119]: audit 2024-02-12T15:56:35.879492+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.169:0/1239442524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:56:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:36 smithi121 bash[18489]: audit 2024-02-12T15:56:35.282672+0000 mon.a (mon.0) 623 : audit [INF] from='client.24367 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]: dispatch 2024-02-12T15:56:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:36 smithi121 bash[18489]: audit 2024-02-12T15:56:35.284314+0000 mon.c (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.169:0/3842225105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]: dispatch 2024-02-12T15:56:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:36 smithi121 bash[18489]: audit 2024-02-12T15:56:35.291864+0000 mon.a (mon.0) 624 : audit [INF] from='client.24367 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]': finished 2024-02-12T15:56:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:36 smithi121 bash[18489]: cluster 2024-02-12T15:56:35.292089+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e68: 10 total, 9 up, 10 in 2024-02-12T15:56:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:36 smithi121 bash[18489]: audit 2024-02-12T15:56:35.292307+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:36 smithi121 bash[18489]: cluster 2024-02-12T15:56:35.399085+0000 mgr.y (mgr.14146) 312 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:36 smithi121 bash[18489]: audit 2024-02-12T15:56:35.879492+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.169:0/1239442524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:56:36.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:36 smithi169 bash[19604]: audit 2024-02-12T15:56:35.282672+0000 mon.a (mon.0) 623 : audit [INF] from='client.24367 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]: dispatch 2024-02-12T15:56:36.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:36 smithi169 bash[19604]: audit 2024-02-12T15:56:35.284314+0000 mon.c (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.169:0/3842225105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]: dispatch 2024-02-12T15:56:36.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:36 smithi169 bash[19604]: audit 2024-02-12T15:56:35.291864+0000 mon.a (mon.0) 624 : audit [INF] from='client.24367 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15"}]': finished 2024-02-12T15:56:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:36 smithi169 bash[19604]: cluster 2024-02-12T15:56:35.292089+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e68: 10 total, 9 up, 10 in 2024-02-12T15:56:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:36 smithi169 bash[19604]: audit 2024-02-12T15:56:35.292307+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:36 smithi169 bash[19604]: cluster 2024-02-12T15:56:35.399085+0000 mgr.y (mgr.14146) 312 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:36 smithi169 bash[19604]: audit 2024-02-12T15:56:35.879492+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.169:0/1239442524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:56:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: audit 2024-02-12T15:56:36.057712+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:56:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: audit 2024-02-12T15:56:36.058111+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:56:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: audit 2024-02-12T15:56:36.058464+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:56:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: audit 2024-02-12T15:56:36.089990+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: cluster 2024-02-12T15:56:36.195804+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: cluster 2024-02-12T15:56:36.195896+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: audit 2024-02-12T15:56:36.406409+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: audit 2024-02-12T15:56:36.408299+0000 mon.b (mon.2) 27 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:37 smithi123 bash[19119]: audit 2024-02-12T15:56:36.416669+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: audit 2024-02-12T15:56:36.057712+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:56:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: audit 2024-02-12T15:56:36.058111+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:56:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: audit 2024-02-12T15:56:36.058464+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:56:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: audit 2024-02-12T15:56:36.089990+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: cluster 2024-02-12T15:56:36.195804+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:37.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: cluster 2024-02-12T15:56:36.195896+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:37.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: audit 2024-02-12T15:56:36.406409+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:37.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: audit 2024-02-12T15:56:36.408299+0000 mon.b (mon.2) 27 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:37.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:37 smithi121 bash[18489]: audit 2024-02-12T15:56:36.416669+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: audit 2024-02-12T15:56:36.057712+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: audit 2024-02-12T15:56:36.058111+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: audit 2024-02-12T15:56:36.058464+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: audit 2024-02-12T15:56:36.089990+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: cluster 2024-02-12T15:56:36.195804+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: cluster 2024-02-12T15:56:36.195896+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: audit 2024-02-12T15:56:36.406409+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: audit 2024-02-12T15:56:36.408299+0000 mon.b (mon.2) 27 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:37 smithi169 bash[19604]: audit 2024-02-12T15:56:36.416669+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:38.332 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:38 smithi169 bash[19604]: cluster 2024-02-12T15:56:37.054311+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e69: 10 total, 9 up, 10 in 2024-02-12T15:56:38.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:38 smithi169 bash[19604]: audit 2024-02-12T15:56:37.054749+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:38.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:38 smithi169 bash[19604]: cluster 2024-02-12T15:56:37.399673+0000 mgr.y (mgr.14146) 313 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:38.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:38 smithi169 bash[19604]: cluster 2024-02-12T15:56:37.745107+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:38.333 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:38 smithi169 bash[19604]: cluster 2024-02-12T15:56:37.745190+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:38 smithi123 bash[19119]: cluster 2024-02-12T15:56:37.054311+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e69: 10 total, 9 up, 10 in 2024-02-12T15:56:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:38 smithi123 bash[19119]: audit 2024-02-12T15:56:37.054749+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:38 smithi123 bash[19119]: cluster 2024-02-12T15:56:37.399673+0000 mgr.y (mgr.14146) 313 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:38 smithi123 bash[19119]: cluster 2024-02-12T15:56:37.745107+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:38 smithi123 bash[19119]: cluster 2024-02-12T15:56:37.745190+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:38 smithi121 bash[18489]: cluster 2024-02-12T15:56:37.054311+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e69: 10 total, 9 up, 10 in 2024-02-12T15:56:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:38 smithi121 bash[18489]: audit 2024-02-12T15:56:37.054749+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:38 smithi121 bash[18489]: cluster 2024-02-12T15:56:37.399673+0000 mgr.y (mgr.14146) 313 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:38 smithi121 bash[18489]: cluster 2024-02-12T15:56:37.745107+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:38 smithi121 bash[18489]: cluster 2024-02-12T15:56:37.745190+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:39.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:39 smithi123 bash[19119]: cluster 2024-02-12T15:56:38.074802+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e70: 10 total, 9 up, 10 in 2024-02-12T15:56:39.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:39 smithi123 bash[19119]: audit 2024-02-12T15:56:38.080155+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:39.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:39 smithi121 bash[18489]: cluster 2024-02-12T15:56:38.074802+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e70: 10 total, 9 up, 10 in 2024-02-12T15:56:39.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:39 smithi121 bash[18489]: audit 2024-02-12T15:56:38.080155+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:39.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:39 smithi169 bash[19604]: cluster 2024-02-12T15:56:38.074802+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e70: 10 total, 9 up, 10 in 2024-02-12T15:56:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:39 smithi169 bash[19604]: audit 2024-02-12T15:56:38.080155+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:41 smithi169 bash[19604]: cluster 2024-02-12T15:56:39.400292+0000 mgr.y (mgr.14146) 314 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:41 smithi123 bash[19119]: cluster 2024-02-12T15:56:39.400292+0000 mgr.y (mgr.14146) 314 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:41 smithi121 bash[18489]: cluster 2024-02-12T15:56:39.400292+0000 mgr.y (mgr.14146) 314 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:42 smithi123 bash[19119]: cluster 2024-02-12T15:56:41.400879+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:42.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:42 smithi121 bash[18489]: cluster 2024-02-12T15:56:41.400879+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:42.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:42 smithi169 bash[19604]: cluster 2024-02-12T15:56:41.400879+0000 mgr.y (mgr.14146) 315 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:44.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:44 smithi169 bash[19604]: cluster 2024-02-12T15:56:43.401433+0000 mgr.y (mgr.14146) 316 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:44 smithi123 bash[19119]: cluster 2024-02-12T15:56:43.401433+0000 mgr.y (mgr.14146) 316 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:44 smithi121 bash[18489]: cluster 2024-02-12T15:56:43.401433+0000 mgr.y (mgr.14146) 316 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:46 smithi123 bash[19119]: cluster 2024-02-12T15:56:45.402015+0000 mgr.y (mgr.14146) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:46 smithi123 bash[19119]: cluster 2024-02-12T15:56:45.735293+0000 osd.7 (osd.7) 5 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:56:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:46 smithi123 bash[19119]: cluster 2024-02-12T15:56:45.739280+0000 osd.7 (osd.7) 6 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:56:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:46 smithi121 bash[18489]: cluster 2024-02-12T15:56:45.402015+0000 mgr.y (mgr.14146) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:46 smithi121 bash[18489]: cluster 2024-02-12T15:56:45.735293+0000 osd.7 (osd.7) 5 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:56:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:46 smithi121 bash[18489]: cluster 2024-02-12T15:56:45.739280+0000 osd.7 (osd.7) 6 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:56:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:46 smithi169 bash[19604]: cluster 2024-02-12T15:56:45.402015+0000 mgr.y (mgr.14146) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:46 smithi169 bash[19604]: cluster 2024-02-12T15:56:45.735293+0000 osd.7 (osd.7) 5 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:56:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:46 smithi169 bash[19604]: cluster 2024-02-12T15:56:45.739280+0000 osd.7 (osd.7) 6 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:56:47.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:47 smithi169 bash[19604]: audit 2024-02-12T15:56:47.139806+0000 mon.b (mon.2) 30 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-12T15:56:47.553 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:47 smithi169 bash[19604]: audit 2024-02-12T15:56:47.141809+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:47 smithi123 bash[19119]: audit 2024-02-12T15:56:47.139806+0000 mon.b (mon.2) 30 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-12T15:56:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:47 smithi123 bash[19119]: audit 2024-02-12T15:56:47.141809+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:47 smithi121 bash[18489]: audit 2024-02-12T15:56:47.139806+0000 mon.b (mon.2) 30 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-12T15:56:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:47 smithi121 bash[18489]: audit 2024-02-12T15:56:47.141809+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:48.462 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:48 smithi169 bash[19604]: cephadm 2024-02-12T15:56:47.139525+0000 mgr.y (mgr.14146) 318 : cephadm [INF] Deploying daemon osd.9 on smithi169 2024-02-12T15:56:48.462 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:48 smithi169 bash[19604]: cluster 2024-02-12T15:56:47.402654+0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:48 smithi123 bash[19119]: cephadm 2024-02-12T15:56:47.139525+0000 mgr.y (mgr.14146) 318 : cephadm [INF] Deploying daemon osd.9 on smithi169 2024-02-12T15:56:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:48 smithi123 bash[19119]: cluster 2024-02-12T15:56:47.402654+0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:48 smithi121 bash[18489]: cephadm 2024-02-12T15:56:47.139525+0000 mgr.y (mgr.14146) 318 : cephadm [INF] Deploying daemon osd.9 on smithi169 2024-02-12T15:56:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:48 smithi121 bash[18489]: cluster 2024-02-12T15:56:47.402654+0000 mgr.y (mgr.14146) 319 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:50.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:50 smithi169 bash[19604]: cluster 2024-02-12T15:56:49.403300+0000 mgr.y (mgr.14146) 320 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:50 smithi123 bash[19119]: cluster 2024-02-12T15:56:49.403300+0000 mgr.y (mgr.14146) 320 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:50 smithi121 bash[18489]: cluster 2024-02-12T15:56:49.403300+0000 mgr.y (mgr.14146) 320 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:51 smithi121 bash[18489]: audit 2024-02-12T15:56:50.889689+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:51 smithi121 bash[18489]: audit 2024-02-12T15:56:50.930912+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:52.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:51 smithi169 bash[19604]: audit 2024-02-12T15:56:50.889689+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:52.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:51 smithi169 bash[19604]: audit 2024-02-12T15:56:50.930912+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:51 smithi123 bash[19119]: audit 2024-02-12T15:56:50.889689+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:52.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:51 smithi123 bash[19119]: audit 2024-02-12T15:56:50.930912+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:56:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:52 smithi121 bash[18489]: cluster 2024-02-12T15:56:51.403856+0000 mgr.y (mgr.14146) 321 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:53.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:52 smithi169 bash[19604]: cluster 2024-02-12T15:56:51.403856+0000 mgr.y (mgr.14146) 321 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:53.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:52 smithi123 bash[19119]: cluster 2024-02-12T15:56:51.403856+0000 mgr.y (mgr.14146) 321 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:54 smithi169 bash[19604]: cluster 2024-02-12T15:56:53.404452+0000 mgr.y (mgr.14146) 322 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:54 smithi121 bash[18489]: cluster 2024-02-12T15:56:53.404452+0000 mgr.y (mgr.14146) 322 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:54 smithi123 bash[19119]: cluster 2024-02-12T15:56:53.404452+0000 mgr.y (mgr.14146) 322 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:56 smithi123 bash[19119]: cluster 2024-02-12T15:56:55.405056+0000 mgr.y (mgr.14146) 323 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:56 smithi121 bash[18489]: cluster 2024-02-12T15:56:55.405056+0000 mgr.y (mgr.14146) 323 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:56.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:56 smithi169 bash[19604]: cluster 2024-02-12T15:56:55.405056+0000 mgr.y (mgr.14146) 323 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: cluster 2024-02-12T15:56:56.764293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: cluster 2024-02-12T15:56:56.764390+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: cluster 2024-02-12T15:56:57.064194+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e71: 10 total, 9 up, 10 in 2024-02-12T15:56:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: audit 2024-02-12T15:56:57.072931+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: audit 2024-02-12T15:56:57.209183+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: audit 2024-02-12T15:56:57.215739+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:58.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: audit 2024-02-12T15:56:57.217714+0000 mon.b (mon.2) 35 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: audit 2024-02-12T15:56:57.224305+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: cluster 2024-02-12T15:56:57.252634+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: cluster 2024-02-12T15:56:57.252724+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: cluster 2024-02-12T15:56:57.405628+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: audit 2024-02-12T15:56:57.734632+0000 mon.a (mon.0) 634 : audit [INF] from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-12T15:56:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:58 smithi123 bash[19119]: audit 2024-02-12T15:56:57.736317+0000 mon.c (mon.1) 11 : audit [INF] from='osd.9 v1:172.21.15.169:6804/2339555071' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-12T15:56:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: cluster 2024-02-12T15:56:56.764293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: cluster 2024-02-12T15:56:56.764390+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: cluster 2024-02-12T15:56:57.064194+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e71: 10 total, 9 up, 10 in 2024-02-12T15:56:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: audit 2024-02-12T15:56:57.072931+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: audit 2024-02-12T15:56:57.209183+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: audit 2024-02-12T15:56:57.215739+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: audit 2024-02-12T15:56:57.217714+0000 mon.b (mon.2) 35 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: audit 2024-02-12T15:56:57.224305+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: cluster 2024-02-12T15:56:57.252634+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: cluster 2024-02-12T15:56:57.252724+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: cluster 2024-02-12T15:56:57.405628+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: audit 2024-02-12T15:56:57.734632+0000 mon.a (mon.0) 634 : audit [INF] from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-12T15:56:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:58 smithi121 bash[18489]: audit 2024-02-12T15:56:57.736317+0000 mon.c (mon.1) 11 : audit [INF] from='osd.9 v1:172.21.15.169:6804/2339555071' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-12T15:56:58.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: cluster 2024-02-12T15:56:56.764293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: cluster 2024-02-12T15:56:56.764390+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: cluster 2024-02-12T15:56:57.064194+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e71: 10 total, 9 up, 10 in 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: audit 2024-02-12T15:56:57.072931+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: audit 2024-02-12T15:56:57.209183+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: audit 2024-02-12T15:56:57.215739+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: audit 2024-02-12T15:56:57.217714+0000 mon.b (mon.2) 35 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: audit 2024-02-12T15:56:57.224305+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: cluster 2024-02-12T15:56:57.252634+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: cluster 2024-02-12T15:56:57.252724+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: cluster 2024-02-12T15:56:57.405628+0000 mgr.y (mgr.14146) 324 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: audit 2024-02-12T15:56:57.734632+0000 mon.a (mon.0) 634 : audit [INF] from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-12T15:56:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:58 smithi169 bash[19604]: audit 2024-02-12T15:56:57.736317+0000 mon.c (mon.1) 11 : audit [INF] from='osd.9 v1:172.21.15.169:6804/2339555071' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-12T15:56:59.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:59 smithi169 bash[19604]: audit 2024-02-12T15:56:58.234963+0000 mon.a (mon.0) 635 : audit [INF] from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-12T15:56:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:59 smithi169 bash[19604]: cluster 2024-02-12T15:56:58.235132+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e72: 10 total, 9 up, 10 in 2024-02-12T15:56:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:59 smithi169 bash[19604]: audit 2024-02-12T15:56:58.238303+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:59 smithi169 bash[19604]: audit 2024-02-12T15:56:58.240194+0000 mon.a (mon.0) 637 : audit [INF] from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:59 smithi169 bash[19604]: audit 2024-02-12T15:56:58.242105+0000 mon.c (mon.1) 12 : audit [INF] from='osd.9 v1:172.21.15.169:6804/2339555071' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:59 smithi169 bash[19604]: cluster 2024-02-12T15:56:58.492711+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:56:59 smithi169 bash[19604]: cluster 2024-02-12T15:56:58.492796+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:59 smithi123 bash[19119]: audit 2024-02-12T15:56:58.234963+0000 mon.a (mon.0) 635 : audit [INF] from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-12T15:56:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:59 smithi123 bash[19119]: cluster 2024-02-12T15:56:58.235132+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e72: 10 total, 9 up, 10 in 2024-02-12T15:56:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:59 smithi123 bash[19119]: audit 2024-02-12T15:56:58.238303+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:59 smithi123 bash[19119]: audit 2024-02-12T15:56:58.240194+0000 mon.a (mon.0) 637 : audit [INF] from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:59 smithi123 bash[19119]: audit 2024-02-12T15:56:58.242105+0000 mon.c (mon.1) 12 : audit [INF] from='osd.9 v1:172.21.15.169:6804/2339555071' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:59 smithi123 bash[19119]: cluster 2024-02-12T15:56:58.492711+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:56:59 smithi123 bash[19119]: cluster 2024-02-12T15:56:58.492796+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:56:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:59 smithi121 bash[18489]: audit 2024-02-12T15:56:58.234963+0000 mon.a (mon.0) 635 : audit [INF] from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-12T15:56:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:59 smithi121 bash[18489]: cluster 2024-02-12T15:56:58.235132+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e72: 10 total, 9 up, 10 in 2024-02-12T15:56:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:59 smithi121 bash[18489]: audit 2024-02-12T15:56:58.238303+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:56:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:59 smithi121 bash[18489]: audit 2024-02-12T15:56:58.240194+0000 mon.a (mon.0) 637 : audit [INF] from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:59 smithi121 bash[18489]: audit 2024-02-12T15:56:58.242105+0000 mon.c (mon.1) 12 : audit [INF] from='osd.9 v1:172.21.15.169:6804/2339555071' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:56:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:59 smithi121 bash[18489]: cluster 2024-02-12T15:56:58.492711+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:56:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:56:59 smithi121 bash[18489]: cluster 2024-02-12T15:56:58.492796+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:00.348 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 9 on host 'smithi169' 2024-02-12T15:57:00.557 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:00 smithi169 bash[19604]: audit 2024-02-12T15:56:59.237524+0000 mon.a (mon.0) 638 : audit [INF] from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:57:00.557 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:00 smithi169 bash[19604]: cluster 2024-02-12T15:56:59.237688+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e73: 10 total, 9 up, 10 in 2024-02-12T15:57:00.557 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:00 smithi169 bash[19604]: audit 2024-02-12T15:56:59.241029+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:00.557 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:00 smithi169 bash[19604]: audit 2024-02-12T15:56:59.250271+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:00.557 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:00 smithi169 bash[19604]: cluster 2024-02-12T15:56:59.406163+0000 mgr.y (mgr.14146) 325 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:57:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:00 smithi123 bash[19119]: audit 2024-02-12T15:56:59.237524+0000 mon.a (mon.0) 638 : audit [INF] from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:57:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:00 smithi123 bash[19119]: cluster 2024-02-12T15:56:59.237688+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e73: 10 total, 9 up, 10 in 2024-02-12T15:57:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:00 smithi123 bash[19119]: audit 2024-02-12T15:56:59.241029+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:00 smithi123 bash[19119]: audit 2024-02-12T15:56:59.250271+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:00 smithi123 bash[19119]: cluster 2024-02-12T15:56:59.406163+0000 mgr.y (mgr.14146) 325 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:57:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:00 smithi121 bash[18489]: audit 2024-02-12T15:56:59.237524+0000 mon.a (mon.0) 638 : audit [INF] from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:57:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:00 smithi121 bash[18489]: cluster 2024-02-12T15:56:59.237688+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e73: 10 total, 9 up, 10 in 2024-02-12T15:57:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:00 smithi121 bash[18489]: audit 2024-02-12T15:56:59.241029+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:00 smithi121 bash[18489]: audit 2024-02-12T15:56:59.250271+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:00.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:00 smithi121 bash[18489]: cluster 2024-02-12T15:56:59.406163+0000 mgr.y (mgr.14146) 325 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 2.6 GiB used, 802 GiB / 805 GiB avail 2024-02-12T15:57:00.969 DEBUG:teuthology.orchestra.run.smithi169:osd.9> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.9.service 2024-02-12T15:57:00.971 INFO:tasks.cephadm:Deploying osd.10 on smithi169 with /dev/vg_nvme/lv_2... 2024-02-12T15:57:00.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-12T15:57:01.083 INFO:journalctl@ceph.osd.9.smithi169.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:57:01.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:01 smithi169 bash[19604]: cluster 2024-02-12T15:56:58.723582+0000 osd.9 (osd.9) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:01 smithi169 bash[19604]: cluster 2024-02-12T15:56:58.723691+0000 osd.9 (osd.9) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:01 smithi169 bash[19604]: cluster 2024-02-12T15:57:00.247902+0000 mon.a (mon.0) 640 : cluster [INF] osd.9 v1:172.21.15.169:6804/2339555071 boot 2024-02-12T15:57:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:01 smithi169 bash[19604]: cluster 2024-02-12T15:57:00.247979+0000 mon.a (mon.0) 641 : cluster [DBG] osdmap e74: 10 total, 10 up, 10 in 2024-02-12T15:57:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:01 smithi169 bash[19604]: audit 2024-02-12T15:57:00.251386+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:01 smithi169 bash[19604]: audit 2024-02-12T15:57:00.340124+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:01 smithi169 bash[19604]: audit 2024-02-12T15:57:00.355081+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:01 smithi123 bash[19119]: cluster 2024-02-12T15:56:58.723582+0000 osd.9 (osd.9) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:01 smithi123 bash[19119]: cluster 2024-02-12T15:56:58.723691+0000 osd.9 (osd.9) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:01 smithi123 bash[19119]: cluster 2024-02-12T15:57:00.247902+0000 mon.a (mon.0) 640 : cluster [INF] osd.9 v1:172.21.15.169:6804/2339555071 boot 2024-02-12T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:01 smithi123 bash[19119]: cluster 2024-02-12T15:57:00.247979+0000 mon.a (mon.0) 641 : cluster [DBG] osdmap e74: 10 total, 10 up, 10 in 2024-02-12T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:01 smithi123 bash[19119]: audit 2024-02-12T15:57:00.251386+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:01 smithi123 bash[19119]: audit 2024-02-12T15:57:00.340124+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:01 smithi123 bash[19119]: audit 2024-02-12T15:57:00.355081+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:01 smithi121 bash[18489]: cluster 2024-02-12T15:56:58.723582+0000 osd.9 (osd.9) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:01 smithi121 bash[18489]: cluster 2024-02-12T15:56:58.723691+0000 osd.9 (osd.9) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:01 smithi121 bash[18489]: cluster 2024-02-12T15:57:00.247902+0000 mon.a (mon.0) 640 : cluster [INF] osd.9 v1:172.21.15.169:6804/2339555071 boot 2024-02-12T15:57:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:01 smithi121 bash[18489]: cluster 2024-02-12T15:57:00.247979+0000 mon.a (mon.0) 641 : cluster [DBG] osdmap e74: 10 total, 10 up, 10 in 2024-02-12T15:57:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:01 smithi121 bash[18489]: audit 2024-02-12T15:57:00.251386+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-12T15:57:01.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:01 smithi121 bash[18489]: audit 2024-02-12T15:57:00.340124+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:01.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:01 smithi121 bash[18489]: audit 2024-02-12T15:57:00.355081+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:02.444 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:02 smithi169 bash[19604]: cluster 2024-02-12T15:57:01.342529+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e75: 10 total, 10 up, 10 in 2024-02-12T15:57:02.444 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:02 smithi169 bash[19604]: cluster 2024-02-12T15:57:01.406730+0000 mgr.y (mgr.14146) 326 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:02.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:02 smithi123 bash[19119]: cluster 2024-02-12T15:57:01.342529+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e75: 10 total, 10 up, 10 in 2024-02-12T15:57:02.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:02 smithi123 bash[19119]: cluster 2024-02-12T15:57:01.406730+0000 mgr.y (mgr.14146) 326 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:02 smithi121 bash[18489]: cluster 2024-02-12T15:57:01.342529+0000 mon.a (mon.0) 643 : cluster [DBG] osdmap e75: 10 total, 10 up, 10 in 2024-02-12T15:57:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:02 smithi121 bash[18489]: cluster 2024-02-12T15:57:01.406730+0000 mgr.y (mgr.14146) 326 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:04.333 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-12T15:57:04.333 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-12T15:57:04.333 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-02-12T15:57:04.334 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-02-12T15:57:04.334 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0175934 s, 596 MB/s 2024-02-12T15:57:04.334 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-02-12T15:57:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:04 smithi169 bash[19604]: cluster 2024-02-12T15:57:03.407364+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:04 smithi123 bash[19119]: cluster 2024-02-12T15:57:03.407364+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:04 smithi121 bash[18489]: cluster 2024-02-12T15:57:03.407364+0000 mgr.y (mgr.14146) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:04.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2024-02-12T15:57:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:06 smithi123 bash[19119]: cluster 2024-02-12T15:57:05.407989+0000 mgr.y (mgr.14146) 328 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:06.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:06 smithi121 bash[18489]: cluster 2024-02-12T15:57:05.407989+0000 mgr.y (mgr.14146) 328 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:06.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:06 smithi169 bash[19604]: cluster 2024-02-12T15:57:05.407989+0000 mgr.y (mgr.14146) 328 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:08.787 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:08 smithi169 bash[19604]: cluster 2024-02-12T15:57:07.408566+0000 mgr.y (mgr.14146) 329 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:08 smithi123 bash[19119]: cluster 2024-02-12T15:57:07.408566+0000 mgr.y (mgr.14146) 329 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:08.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:08 smithi121 bash[18489]: cluster 2024-02-12T15:57:07.408566+0000 mgr.y (mgr.14146) 329 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:09 smithi169 bash[19604]: cluster 2024-02-12T15:57:08.932014+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:09 smithi169 bash[19604]: cluster 2024-02-12T15:57:08.932108+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:09 smithi169 bash[19604]: audit 2024-02-12T15:57:09.131844+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:57:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:09 smithi169 bash[19604]: audit 2024-02-12T15:57:09.138665+0000 mon.b (mon.2) 42 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:57:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:09 smithi169 bash[19604]: audit 2024-02-12T15:57:09.140374+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:09 smithi123 bash[19119]: cluster 2024-02-12T15:57:08.932014+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:09 smithi123 bash[19119]: cluster 2024-02-12T15:57:08.932108+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:09 smithi123 bash[19119]: audit 2024-02-12T15:57:09.131844+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:57:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:09 smithi123 bash[19119]: audit 2024-02-12T15:57:09.138665+0000 mon.b (mon.2) 42 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:57:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:09 smithi123 bash[19119]: audit 2024-02-12T15:57:09.140374+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:09 smithi121 bash[18489]: cluster 2024-02-12T15:57:08.932014+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:09 smithi121 bash[18489]: cluster 2024-02-12T15:57:08.932108+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:09 smithi121 bash[18489]: audit 2024-02-12T15:57:09.131844+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:57:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:09 smithi121 bash[18489]: audit 2024-02-12T15:57:09.138665+0000 mon.b (mon.2) 42 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:57:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:09 smithi121 bash[18489]: audit 2024-02-12T15:57:09.140374+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:09.120645+0000 mgr.y (mgr.14146) 330 : audit [DBG] from='client.24388 v1:172.21.15.169:0/1340239076' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: cluster 2024-02-12T15:57:09.409162+0000 mgr.y (mgr.14146) 331 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: cluster 2024-02-12T15:57:09.485036+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e76: 10 total, 10 up, 10 in 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.359882+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.364659+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.367678+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.368070+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.371153+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.377977+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.386736+0000 mon.b (mon.2) 46 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.388916+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:10 smithi169 bash[19604]: audit 2024-02-12T15:57:10.395612+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:09.120645+0000 mgr.y (mgr.14146) 330 : audit [DBG] from='client.24388 v1:172.21.15.169:0/1340239076' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:57:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: cluster 2024-02-12T15:57:09.409162+0000 mgr.y (mgr.14146) 331 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: cluster 2024-02-12T15:57:09.485036+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e76: 10 total, 10 up, 10 in 2024-02-12T15:57:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.359882+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.364659+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.367678+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.368070+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.371153+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.377977+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.386736+0000 mon.b (mon.2) 46 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.388916+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:10 smithi123 bash[19119]: audit 2024-02-12T15:57:10.395612+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:09.120645+0000 mgr.y (mgr.14146) 330 : audit [DBG] from='client.24388 v1:172.21.15.169:0/1340239076' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:57:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: cluster 2024-02-12T15:57:09.409162+0000 mgr.y (mgr.14146) 331 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: cluster 2024-02-12T15:57:09.485036+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e76: 10 total, 10 up, 10 in 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.359882+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.364659+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.367678+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.368070+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.371153+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.377977+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.386736+0000 mon.b (mon.2) 46 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.388916+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:10 smithi121 bash[18489]: audit 2024-02-12T15:57:10.395612+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:11.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:11 smithi169 bash[19604]: cephadm 2024-02-12T15:57:10.349145+0000 mgr.y (mgr.14146) 332 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:57:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:11 smithi169 bash[19604]: cephadm 2024-02-12T15:57:10.369572+0000 mgr.y (mgr.14146) 333 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 10707M 2024-02-12T15:57:11.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:11 smithi123 bash[19119]: cephadm 2024-02-12T15:57:10.349145+0000 mgr.y (mgr.14146) 332 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:57:11.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:11 smithi123 bash[19119]: cephadm 2024-02-12T15:57:10.369572+0000 mgr.y (mgr.14146) 333 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 10707M 2024-02-12T15:57:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:11 smithi121 bash[18489]: cephadm 2024-02-12T15:57:10.349145+0000 mgr.y (mgr.14146) 332 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:57:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:11 smithi121 bash[18489]: cephadm 2024-02-12T15:57:10.369572+0000 mgr.y (mgr.14146) 333 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 10707M 2024-02-12T15:57:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:12 smithi169 bash[19604]: cluster 2024-02-12T15:57:11.409769+0000 mgr.y (mgr.14146) 334 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:12 smithi123 bash[19119]: cluster 2024-02-12T15:57:11.409769+0000 mgr.y (mgr.14146) 334 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:12 smithi121 bash[18489]: cluster 2024-02-12T15:57:11.409769+0000 mgr.y (mgr.14146) 334 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:14 smithi169 bash[19604]: cluster 2024-02-12T15:57:13.410325+0000 mgr.y (mgr.14146) 335 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:14 smithi123 bash[19119]: cluster 2024-02-12T15:57:13.410325+0000 mgr.y (mgr.14146) 335 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:14 smithi121 bash[18489]: cluster 2024-02-12T15:57:13.410325+0000 mgr.y (mgr.14146) 335 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:16.327 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:16 smithi169 bash[19604]: cluster 2024-02-12T15:57:15.410930+0000 mgr.y (mgr.14146) 336 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:16 smithi123 bash[19119]: cluster 2024-02-12T15:57:15.410930+0000 mgr.y (mgr.14146) 336 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:16.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:16 smithi121 bash[18489]: cluster 2024-02-12T15:57:15.410930+0000 mgr.y (mgr.14146) 336 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:17 smithi123 bash[19119]: audit 2024-02-12T15:57:17.057653+0000 mon.a (mon.0) 650 : audit [INF] from='client.24394 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]: dispatch 2024-02-12T15:57:17.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:17 smithi123 bash[19119]: audit 2024-02-12T15:57:17.059313+0000 mon.c (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.169:0/1921965246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]: dispatch 2024-02-12T15:57:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:17 smithi121 bash[18489]: audit 2024-02-12T15:57:17.057653+0000 mon.a (mon.0) 650 : audit [INF] from='client.24394 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]: dispatch 2024-02-12T15:57:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:17 smithi121 bash[18489]: audit 2024-02-12T15:57:17.059313+0000 mon.c (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.169:0/1921965246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]: dispatch 2024-02-12T15:57:17.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:17 smithi169 bash[19604]: audit 2024-02-12T15:57:17.057653+0000 mon.a (mon.0) 650 : audit [INF] from='client.24394 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]: dispatch 2024-02-12T15:57:17.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:17 smithi169 bash[19604]: audit 2024-02-12T15:57:17.059313+0000 mon.c (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.169:0/1921965246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]: dispatch 2024-02-12T15:57:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:18 smithi123 bash[19119]: audit 2024-02-12T15:57:17.067364+0000 mon.a (mon.0) 651 : audit [INF] from='client.24394 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]': finished 2024-02-12T15:57:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:18 smithi123 bash[19119]: cluster 2024-02-12T15:57:17.067422+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e77: 11 total, 10 up, 11 in 2024-02-12T15:57:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:18 smithi123 bash[19119]: audit 2024-02-12T15:57:17.071578+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:18 smithi123 bash[19119]: cluster 2024-02-12T15:57:17.411479+0000 mgr.y (mgr.14146) 337 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:18.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:18 smithi123 bash[19119]: audit 2024-02-12T15:57:17.674304+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.169:0/3871810423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:57:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:18 smithi121 bash[18489]: audit 2024-02-12T15:57:17.067364+0000 mon.a (mon.0) 651 : audit [INF] from='client.24394 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]': finished 2024-02-12T15:57:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:18 smithi121 bash[18489]: cluster 2024-02-12T15:57:17.067422+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e77: 11 total, 10 up, 11 in 2024-02-12T15:57:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:18 smithi121 bash[18489]: audit 2024-02-12T15:57:17.071578+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:18 smithi121 bash[18489]: cluster 2024-02-12T15:57:17.411479+0000 mgr.y (mgr.14146) 337 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:18.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:18 smithi121 bash[18489]: audit 2024-02-12T15:57:17.674304+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.169:0/3871810423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:57:18.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:18 smithi169 bash[19604]: audit 2024-02-12T15:57:17.067364+0000 mon.a (mon.0) 651 : audit [INF] from='client.24394 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f74dca-503a-45d1-9647-df4804d58515"}]': finished 2024-02-12T15:57:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:18 smithi169 bash[19604]: cluster 2024-02-12T15:57:17.067422+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e77: 11 total, 10 up, 11 in 2024-02-12T15:57:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:18 smithi169 bash[19604]: audit 2024-02-12T15:57:17.071578+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:18 smithi169 bash[19604]: cluster 2024-02-12T15:57:17.411479+0000 mgr.y (mgr.14146) 337 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:18 smithi169 bash[19604]: audit 2024-02-12T15:57:17.674304+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.169:0/3871810423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:57:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:20 smithi169 bash[19604]: cluster 2024-02-12T15:57:19.412081+0000 mgr.y (mgr.14146) 338 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:20 smithi123 bash[19119]: cluster 2024-02-12T15:57:19.412081+0000 mgr.y (mgr.14146) 338 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:20 smithi121 bash[18489]: cluster 2024-02-12T15:57:19.412081+0000 mgr.y (mgr.14146) 338 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:22 smithi169 bash[19604]: cluster 2024-02-12T15:57:21.187716+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:22 smithi169 bash[19604]: cluster 2024-02-12T15:57:21.187809+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:22 smithi169 bash[19604]: cluster 2024-02-12T15:57:21.412683+0000 mgr.y (mgr.14146) 339 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:22 smithi169 bash[19604]: cluster 2024-02-12T15:57:21.482151+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e78: 11 total, 10 up, 11 in 2024-02-12T15:57:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:22 smithi169 bash[19604]: audit 2024-02-12T15:57:21.489594+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:22 smithi123 bash[19119]: cluster 2024-02-12T15:57:21.187716+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:22 smithi123 bash[19119]: cluster 2024-02-12T15:57:21.187809+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:22 smithi123 bash[19119]: cluster 2024-02-12T15:57:21.412683+0000 mgr.y (mgr.14146) 339 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:22 smithi123 bash[19119]: cluster 2024-02-12T15:57:21.482151+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e78: 11 total, 10 up, 11 in 2024-02-12T15:57:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:22 smithi123 bash[19119]: audit 2024-02-12T15:57:21.489594+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:22 smithi121 bash[18489]: cluster 2024-02-12T15:57:21.187716+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:22 smithi121 bash[18489]: cluster 2024-02-12T15:57:21.187809+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:22 smithi121 bash[18489]: cluster 2024-02-12T15:57:21.412683+0000 mgr.y (mgr.14146) 339 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:22 smithi121 bash[18489]: cluster 2024-02-12T15:57:21.482151+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e78: 11 total, 10 up, 11 in 2024-02-12T15:57:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:22 smithi121 bash[18489]: audit 2024-02-12T15:57:21.489594+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:24.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:24 smithi169 bash[19604]: cluster 2024-02-12T15:57:23.413260+0000 mgr.y (mgr.14146) 340 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:24 smithi123 bash[19119]: cluster 2024-02-12T15:57:23.413260+0000 mgr.y (mgr.14146) 340 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:24 smithi121 bash[18489]: cluster 2024-02-12T15:57:23.413260+0000 mgr.y (mgr.14146) 340 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:25 smithi169 bash[19604]: cluster 2024-02-12T15:57:23.785890+0000 osd.8 (osd.8) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:25 smithi169 bash[19604]: cluster 2024-02-12T15:57:23.785983+0000 osd.8 (osd.8) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:25 smithi169 bash[19604]: cluster 2024-02-12T15:57:24.496099+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e79: 11 total, 10 up, 11 in 2024-02-12T15:57:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:25 smithi169 bash[19604]: audit 2024-02-12T15:57:24.501065+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:25 smithi123 bash[19119]: cluster 2024-02-12T15:57:23.785890+0000 osd.8 (osd.8) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:25 smithi123 bash[19119]: cluster 2024-02-12T15:57:23.785983+0000 osd.8 (osd.8) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:25 smithi123 bash[19119]: cluster 2024-02-12T15:57:24.496099+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e79: 11 total, 10 up, 11 in 2024-02-12T15:57:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:25 smithi123 bash[19119]: audit 2024-02-12T15:57:24.501065+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:25 smithi121 bash[18489]: cluster 2024-02-12T15:57:23.785890+0000 osd.8 (osd.8) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:25 smithi121 bash[18489]: cluster 2024-02-12T15:57:23.785983+0000 osd.8 (osd.8) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:25 smithi121 bash[18489]: cluster 2024-02-12T15:57:24.496099+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e79: 11 total, 10 up, 11 in 2024-02-12T15:57:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:25 smithi121 bash[18489]: audit 2024-02-12T15:57:24.501065+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:26.696 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:26 smithi169 bash[19604]: cluster 2024-02-12T15:57:25.413853+0000 mgr.y (mgr.14146) 341 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:26.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:26 smithi123 bash[19119]: cluster 2024-02-12T15:57:25.413853+0000 mgr.y (mgr.14146) 341 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:26 smithi121 bash[18489]: cluster 2024-02-12T15:57:25.413853+0000 mgr.y (mgr.14146) 341 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:28 smithi169 bash[19604]: cluster 2024-02-12T15:57:27.392342+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:28 smithi169 bash[19604]: cluster 2024-02-12T15:57:27.392426+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:28 smithi169 bash[19604]: cluster 2024-02-12T15:57:27.414415+0000 mgr.y (mgr.14146) 342 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:28 smithi169 bash[19604]: cluster 2024-02-12T15:57:27.508210+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e80: 11 total, 10 up, 11 in 2024-02-12T15:57:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:28 smithi169 bash[19604]: audit 2024-02-12T15:57:27.511996+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:28 smithi123 bash[19119]: cluster 2024-02-12T15:57:27.392342+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:28 smithi123 bash[19119]: cluster 2024-02-12T15:57:27.392426+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:28 smithi123 bash[19119]: cluster 2024-02-12T15:57:27.414415+0000 mgr.y (mgr.14146) 342 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:28 smithi123 bash[19119]: cluster 2024-02-12T15:57:27.508210+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e80: 11 total, 10 up, 11 in 2024-02-12T15:57:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:28 smithi123 bash[19119]: audit 2024-02-12T15:57:27.511996+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:28.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:28 smithi121 bash[18489]: cluster 2024-02-12T15:57:27.392342+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:28.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:28 smithi121 bash[18489]: cluster 2024-02-12T15:57:27.392426+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:28.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:28 smithi121 bash[18489]: cluster 2024-02-12T15:57:27.414415+0000 mgr.y (mgr.14146) 342 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:28.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:28 smithi121 bash[18489]: cluster 2024-02-12T15:57:27.508210+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e80: 11 total, 10 up, 11 in 2024-02-12T15:57:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:28 smithi121 bash[18489]: audit 2024-02-12T15:57:27.511996+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:29.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:29 smithi169 bash[19604]: cephadm 2024-02-12T15:57:28.950880+0000 mgr.y (mgr.14146) 343 : cephadm [INF] Deploying daemon osd.10 on smithi169 2024-02-12T15:57:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:29 smithi169 bash[19604]: audit 2024-02-12T15:57:28.951346+0000 mon.b (mon.2) 52 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-12T15:57:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:29 smithi169 bash[19604]: audit 2024-02-12T15:57:28.953517+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:29 smithi123 bash[19119]: cephadm 2024-02-12T15:57:28.950880+0000 mgr.y (mgr.14146) 343 : cephadm [INF] Deploying daemon osd.10 on smithi169 2024-02-12T15:57:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:29 smithi123 bash[19119]: audit 2024-02-12T15:57:28.951346+0000 mon.b (mon.2) 52 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-12T15:57:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:29 smithi123 bash[19119]: audit 2024-02-12T15:57:28.953517+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:29 smithi121 bash[18489]: cephadm 2024-02-12T15:57:28.950880+0000 mgr.y (mgr.14146) 343 : cephadm [INF] Deploying daemon osd.10 on smithi169 2024-02-12T15:57:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:29 smithi121 bash[18489]: audit 2024-02-12T15:57:28.951346+0000 mon.b (mon.2) 52 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-12T15:57:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:29 smithi121 bash[18489]: audit 2024-02-12T15:57:28.953517+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:30 smithi169 bash[19604]: cluster 2024-02-12T15:57:29.414999+0000 mgr.y (mgr.14146) 344 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:30.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:30 smithi123 bash[19119]: cluster 2024-02-12T15:57:29.414999+0000 mgr.y (mgr.14146) 344 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:30.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:30 smithi121 bash[18489]: cluster 2024-02-12T15:57:29.414999+0000 mgr.y (mgr.14146) 344 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:32.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:32 smithi169 bash[19604]: cluster 2024-02-12T15:57:31.415632+0000 mgr.y (mgr.14146) 345 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:32 smithi123 bash[19119]: cluster 2024-02-12T15:57:31.415632+0000 mgr.y (mgr.14146) 345 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:32 smithi121 bash[18489]: cluster 2024-02-12T15:57:31.415632+0000 mgr.y (mgr.14146) 345 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:34.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:33 smithi169 bash[19604]: audit 2024-02-12T15:57:32.824643+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:33 smithi169 bash[19604]: audit 2024-02-12T15:57:32.830825+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:34.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:33 smithi123 bash[19119]: audit 2024-02-12T15:57:32.824643+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:34.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:33 smithi123 bash[19119]: audit 2024-02-12T15:57:32.830825+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:33 smithi121 bash[18489]: audit 2024-02-12T15:57:32.824643+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:34.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:33 smithi121 bash[18489]: audit 2024-02-12T15:57:32.830825+0000 mon.b (mon.2) 54 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:35.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:34 smithi123 bash[19119]: cluster 2024-02-12T15:57:33.416214+0000 mgr.y (mgr.14146) 346 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:34 smithi121 bash[18489]: cluster 2024-02-12T15:57:33.416214+0000 mgr.y (mgr.14146) 346 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:34 smithi169 bash[19604]: cluster 2024-02-12T15:57:33.416214+0000 mgr.y (mgr.14146) 346 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:36 smithi169 bash[19604]: cluster 2024-02-12T15:57:35.416818+0000 mgr.y (mgr.14146) 347 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:36 smithi123 bash[19119]: cluster 2024-02-12T15:57:35.416818+0000 mgr.y (mgr.14146) 347 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:36 smithi121 bash[18489]: cluster 2024-02-12T15:57:35.416818+0000 mgr.y (mgr.14146) 347 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:38 smithi169 bash[19604]: cluster 2024-02-12T15:57:37.417405+0000 mgr.y (mgr.14146) 348 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:38 smithi169 bash[19604]: audit 2024-02-12T15:57:38.175142+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:38 smithi169 bash[19604]: audit 2024-02-12T15:57:38.461746+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:38 smithi169 bash[19604]: audit 2024-02-12T15:57:38.463929+0000 mon.b (mon.2) 56 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:38 smithi123 bash[19119]: cluster 2024-02-12T15:57:37.417405+0000 mgr.y (mgr.14146) 348 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:38 smithi123 bash[19119]: audit 2024-02-12T15:57:38.175142+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:38 smithi123 bash[19119]: audit 2024-02-12T15:57:38.461746+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:38 smithi123 bash[19119]: audit 2024-02-12T15:57:38.463929+0000 mon.b (mon.2) 56 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:38.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:38 smithi121 bash[18489]: cluster 2024-02-12T15:57:37.417405+0000 mgr.y (mgr.14146) 348 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:38.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:38 smithi121 bash[18489]: audit 2024-02-12T15:57:38.175142+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:38.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:38 smithi121 bash[18489]: audit 2024-02-12T15:57:38.461746+0000 mon.b (mon.2) 55 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:38.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:38 smithi121 bash[18489]: audit 2024-02-12T15:57:38.463929+0000 mon.b (mon.2) 56 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:39 smithi169 bash[19604]: audit 2024-02-12T15:57:38.471272+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:39.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:39 smithi123 bash[19119]: audit 2024-02-12T15:57:38.471272+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:39 smithi121 bash[18489]: audit 2024-02-12T15:57:38.471272+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:40 smithi169 bash[19604]: cluster 2024-02-12T15:57:39.417989+0000 mgr.y (mgr.14146) 349 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:40 smithi169 bash[19604]: audit 2024-02-12T15:57:39.552800+0000 mon.a (mon.0) 659 : audit [INF] from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-12T15:57:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:40 smithi169 bash[19604]: audit 2024-02-12T15:57:39.554906+0000 mon.c (mon.1) 15 : audit [INF] from='osd.10 v1:172.21.15.169:6808/391306610' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-12T15:57:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:40 smithi123 bash[19119]: cluster 2024-02-12T15:57:39.417989+0000 mgr.y (mgr.14146) 349 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:40 smithi123 bash[19119]: audit 2024-02-12T15:57:39.552800+0000 mon.a (mon.0) 659 : audit [INF] from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-12T15:57:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:40 smithi123 bash[19119]: audit 2024-02-12T15:57:39.554906+0000 mon.c (mon.1) 15 : audit [INF] from='osd.10 v1:172.21.15.169:6808/391306610' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-12T15:57:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:40 smithi121 bash[18489]: cluster 2024-02-12T15:57:39.417989+0000 mgr.y (mgr.14146) 349 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:40 smithi121 bash[18489]: audit 2024-02-12T15:57:39.552800+0000 mon.a (mon.0) 659 : audit [INF] from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-12T15:57:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:40 smithi121 bash[18489]: audit 2024-02-12T15:57:39.554906+0000 mon.c (mon.1) 15 : audit [INF] from='osd.10 v1:172.21.15.169:6808/391306610' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-12T15:57:41.720 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 10 on host 'smithi169' 2024-02-12T15:57:41.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:41 smithi169 bash[19604]: audit 2024-02-12T15:57:40.498383+0000 mon.a (mon.0) 660 : audit [INF] from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-12T15:57:41.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:41 smithi169 bash[19604]: cluster 2024-02-12T15:57:40.498560+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e81: 11 total, 10 up, 11 in 2024-02-12T15:57:41.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:41 smithi169 bash[19604]: audit 2024-02-12T15:57:40.500739+0000 mon.a (mon.0) 662 : audit [INF] from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:57:41.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:41 smithi169 bash[19604]: audit 2024-02-12T15:57:40.502368+0000 mon.c (mon.1) 16 : audit [INF] from='osd.10 v1:172.21.15.169:6808/391306610' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:57:41.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:41 smithi169 bash[19604]: audit 2024-02-12T15:57:40.503799+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:41 smithi123 bash[19119]: audit 2024-02-12T15:57:40.498383+0000 mon.a (mon.0) 660 : audit [INF] from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-12T15:57:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:41 smithi123 bash[19119]: cluster 2024-02-12T15:57:40.498560+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e81: 11 total, 10 up, 11 in 2024-02-12T15:57:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:41 smithi123 bash[19119]: audit 2024-02-12T15:57:40.500739+0000 mon.a (mon.0) 662 : audit [INF] from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:57:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:41 smithi123 bash[19119]: audit 2024-02-12T15:57:40.502368+0000 mon.c (mon.1) 16 : audit [INF] from='osd.10 v1:172.21.15.169:6808/391306610' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:57:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:41 smithi123 bash[19119]: audit 2024-02-12T15:57:40.503799+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:41 smithi121 bash[18489]: audit 2024-02-12T15:57:40.498383+0000 mon.a (mon.0) 660 : audit [INF] from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-12T15:57:41.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:41 smithi121 bash[18489]: cluster 2024-02-12T15:57:40.498560+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e81: 11 total, 10 up, 11 in 2024-02-12T15:57:41.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:41 smithi121 bash[18489]: audit 2024-02-12T15:57:40.500739+0000 mon.a (mon.0) 662 : audit [INF] from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:57:41.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:41 smithi121 bash[18489]: audit 2024-02-12T15:57:40.502368+0000 mon.c (mon.1) 16 : audit [INF] from='osd.10 v1:172.21.15.169:6808/391306610' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:57:41.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:41 smithi121 bash[18489]: audit 2024-02-12T15:57:40.503799+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:42.274 DEBUG:teuthology.orchestra.run.smithi169:osd.10> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.10.service 2024-02-12T15:57:42.276 INFO:tasks.cephadm:Deploying osd.11 on smithi169 with /dev/vg_nvme/lv_1... 2024-02-12T15:57:42.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-12T15:57:42.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:42 smithi169 bash[19604]: cluster 2024-02-12T15:57:41.418581+0000 mgr.y (mgr.14146) 350 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:42 smithi169 bash[19604]: audit 2024-02-12T15:57:41.501709+0000 mon.a (mon.0) 663 : audit [INF] from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:57:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:42 smithi169 bash[19604]: cluster 2024-02-12T15:57:41.501858+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e82: 11 total, 10 up, 11 in 2024-02-12T15:57:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:42 smithi169 bash[19604]: audit 2024-02-12T15:57:41.509645+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:42 smithi169 bash[19604]: audit 2024-02-12T15:57:41.712211+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:42 smithi169 bash[19604]: audit 2024-02-12T15:57:41.755369+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:42.554 INFO:journalctl@ceph.osd.10.smithi169.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:57:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:42 smithi123 bash[19119]: cluster 2024-02-12T15:57:41.418581+0000 mgr.y (mgr.14146) 350 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:42 smithi123 bash[19119]: audit 2024-02-12T15:57:41.501709+0000 mon.a (mon.0) 663 : audit [INF] from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:57:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:42 smithi123 bash[19119]: cluster 2024-02-12T15:57:41.501858+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e82: 11 total, 10 up, 11 in 2024-02-12T15:57:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:42 smithi123 bash[19119]: audit 2024-02-12T15:57:41.509645+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:42 smithi123 bash[19119]: audit 2024-02-12T15:57:41.712211+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:42 smithi123 bash[19119]: audit 2024-02-12T15:57:41.755369+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:42 smithi121 bash[18489]: cluster 2024-02-12T15:57:41.418581+0000 mgr.y (mgr.14146) 350 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:42.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:42 smithi121 bash[18489]: audit 2024-02-12T15:57:41.501709+0000 mon.a (mon.0) 663 : audit [INF] from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:57:42.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:42 smithi121 bash[18489]: cluster 2024-02-12T15:57:41.501858+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e82: 11 total, 10 up, 11 in 2024-02-12T15:57:42.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:42 smithi121 bash[18489]: audit 2024-02-12T15:57:41.509645+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:42.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:42 smithi121 bash[18489]: audit 2024-02-12T15:57:41.712211+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:42.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:42 smithi121 bash[18489]: audit 2024-02-12T15:57:41.755369+0000 mon.b (mon.2) 59 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:57:43.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:43 smithi169 bash[19604]: cluster 2024-02-12T15:57:40.571448+0000 osd.10 (osd.10) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:43.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:43 smithi169 bash[19604]: cluster 2024-02-12T15:57:40.571568+0000 osd.10 (osd.10) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:43.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:43 smithi169 bash[19604]: audit 2024-02-12T15:57:42.512515+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:43.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:43 smithi169 bash[19604]: cluster 2024-02-12T15:57:42.522420+0000 mon.a (mon.0) 666 : cluster [INF] osd.10 v1:172.21.15.169:6808/391306610 boot 2024-02-12T15:57:43.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:43 smithi169 bash[19604]: cluster 2024-02-12T15:57:42.522523+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e83: 11 total, 11 up, 11 in 2024-02-12T15:57:43.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:43 smithi169 bash[19604]: audit 2024-02-12T15:57:42.523987+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:43 smithi123 bash[19119]: cluster 2024-02-12T15:57:40.571448+0000 osd.10 (osd.10) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:43 smithi123 bash[19119]: cluster 2024-02-12T15:57:40.571568+0000 osd.10 (osd.10) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:43 smithi123 bash[19119]: audit 2024-02-12T15:57:42.512515+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:43 smithi123 bash[19119]: cluster 2024-02-12T15:57:42.522420+0000 mon.a (mon.0) 666 : cluster [INF] osd.10 v1:172.21.15.169:6808/391306610 boot 2024-02-12T15:57:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:43 smithi123 bash[19119]: cluster 2024-02-12T15:57:42.522523+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e83: 11 total, 11 up, 11 in 2024-02-12T15:57:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:43 smithi123 bash[19119]: audit 2024-02-12T15:57:42.523987+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:43 smithi121 bash[18489]: cluster 2024-02-12T15:57:40.571448+0000 osd.10 (osd.10) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:43 smithi121 bash[18489]: cluster 2024-02-12T15:57:40.571568+0000 osd.10 (osd.10) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:43 smithi121 bash[18489]: audit 2024-02-12T15:57:42.512515+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:43 smithi121 bash[18489]: cluster 2024-02-12T15:57:42.522420+0000 mon.a (mon.0) 666 : cluster [INF] osd.10 v1:172.21.15.169:6808/391306610 boot 2024-02-12T15:57:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:43 smithi121 bash[18489]: cluster 2024-02-12T15:57:42.522523+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e83: 11 total, 11 up, 11 in 2024-02-12T15:57:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:43 smithi121 bash[18489]: audit 2024-02-12T15:57:42.523987+0000 mon.b (mon.2) 61 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-12T15:57:44.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:44 smithi169 bash[19604]: cluster 2024-02-12T15:57:43.419220+0000 mgr.y (mgr.14146) 351 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:44.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:44 smithi169 bash[19604]: cluster 2024-02-12T15:57:43.531717+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e84: 11 total, 11 up, 11 in 2024-02-12T15:57:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:44 smithi123 bash[19119]: cluster 2024-02-12T15:57:43.419220+0000 mgr.y (mgr.14146) 351 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:44 smithi123 bash[19119]: cluster 2024-02-12T15:57:43.531717+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e84: 11 total, 11 up, 11 in 2024-02-12T15:57:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:44 smithi121 bash[18489]: cluster 2024-02-12T15:57:43.419220+0000 mgr.y (mgr.14146) 351 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 2.8 GiB used, 891 GiB / 894 GiB avail 2024-02-12T15:57:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:44 smithi121 bash[18489]: cluster 2024-02-12T15:57:43.531717+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e84: 11 total, 11 up, 11 in 2024-02-12T15:57:45.441 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-12T15:57:45.441 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-12T15:57:45.441 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2024-02-12T15:57:45.442 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2024-02-12T15:57:45.442 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143847 s, 729 MB/s 2024-02-12T15:57:45.442 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2024-02-12T15:57:46.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_1 2024-02-12T15:57:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:46 smithi123 bash[19119]: cluster 2024-02-12T15:57:45.419949+0000 mgr.y (mgr.14146) 352 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:46 smithi121 bash[18489]: cluster 2024-02-12T15:57:45.419949+0000 mgr.y (mgr.14146) 352 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:46 smithi169 bash[19604]: cluster 2024-02-12T15:57:45.419949+0000 mgr.y (mgr.14146) 352 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:48 smithi169 bash[19604]: cluster 2024-02-12T15:57:47.420561+0000 mgr.y (mgr.14146) 353 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:48 smithi123 bash[19119]: cluster 2024-02-12T15:57:47.420561+0000 mgr.y (mgr.14146) 353 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:48 smithi121 bash[18489]: cluster 2024-02-12T15:57:47.420561+0000 mgr.y (mgr.14146) 353 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:50 smithi169 bash[19604]: cluster 2024-02-12T15:57:49.421200+0000 mgr.y (mgr.14146) 354 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:50 smithi169 bash[19604]: audit 2024-02-12T15:57:50.357949+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:57:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:50 smithi169 bash[19604]: audit 2024-02-12T15:57:50.363953+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:57:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:50 smithi169 bash[19604]: audit 2024-02-12T15:57:50.365692+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:50 smithi123 bash[19119]: cluster 2024-02-12T15:57:49.421200+0000 mgr.y (mgr.14146) 354 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:50 smithi123 bash[19119]: audit 2024-02-12T15:57:50.357949+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:57:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:50 smithi123 bash[19119]: audit 2024-02-12T15:57:50.363953+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:57:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:50 smithi123 bash[19119]: audit 2024-02-12T15:57:50.365692+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:50 smithi121 bash[18489]: cluster 2024-02-12T15:57:49.421200+0000 mgr.y (mgr.14146) 354 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:50 smithi121 bash[18489]: audit 2024-02-12T15:57:50.357949+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-12T15:57:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:50 smithi121 bash[18489]: audit 2024-02-12T15:57:50.363953+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-12T15:57:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:50 smithi121 bash[18489]: audit 2024-02-12T15:57:50.365692+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:51 smithi169 bash[19604]: audit 2024-02-12T15:57:50.346809+0000 mgr.y (mgr.14146) 355 : audit [DBG] from='client.24415 v1:172.21.15.169:0/3059059232' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:57:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:51 smithi123 bash[19119]: audit 2024-02-12T15:57:50.346809+0000 mgr.y (mgr.14146) 355 : audit [DBG] from='client.24415 v1:172.21.15.169:0/3059059232' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:57:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:51 smithi121 bash[18489]: audit 2024-02-12T15:57:50.346809+0000 mgr.y (mgr.14146) 355 : audit [DBG] from='client.24415 v1:172.21.15.169:0/3059059232' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T15:57:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: cluster 2024-02-12T15:57:51.421833+0000 mgr.y (mgr.14146) 356 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: cephadm 2024-02-12T15:57:51.614498+0000 mgr.y (mgr.14146) 357 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:57:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.625858+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.629610+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.632392+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.633491+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.635140+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.636308+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: cephadm 2024-02-12T15:57:51.636810+0000 mgr.y (mgr.14146) 358 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 7138M 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.639024+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.645257+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.654490+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.656325+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: audit 2024-02-12T15:57:51.661719+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: cluster 2024-02-12T15:57:52.288514+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:52 smithi123 bash[19119]: cluster 2024-02-12T15:57:52.288604+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: cluster 2024-02-12T15:57:51.421833+0000 mgr.y (mgr.14146) 356 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: cephadm 2024-02-12T15:57:51.614498+0000 mgr.y (mgr.14146) 357 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.625858+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.629610+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.632392+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.633491+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.635140+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.636308+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: cephadm 2024-02-12T15:57:51.636810+0000 mgr.y (mgr.14146) 358 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 7138M 2024-02-12T15:57:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.639024+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.645257+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.654490+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.656325+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: audit 2024-02-12T15:57:51.661719+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: cluster 2024-02-12T15:57:52.288514+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:52 smithi121 bash[18489]: cluster 2024-02-12T15:57:52.288604+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: cluster 2024-02-12T15:57:51.421833+0000 mgr.y (mgr.14146) 356 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: cephadm 2024-02-12T15:57:51.614498+0000 mgr.y (mgr.14146) 357 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.625858+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.629610+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.632392+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.633491+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.635140+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.636308+0000 mon.b (mon.2) 66 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: cephadm 2024-02-12T15:57:51.636810+0000 mgr.y (mgr.14146) 358 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 7138M 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.639024+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.645257+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.654490+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.656325+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: audit 2024-02-12T15:57:51.661719+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: cluster 2024-02-12T15:57:52.288514+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:57:53.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:52 smithi169 bash[19604]: cluster 2024-02-12T15:57:52.288604+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:57:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:53 smithi169 bash[19604]: cluster 2024-02-12T15:57:52.663997+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e85: 11 total, 11 up, 11 in 2024-02-12T15:57:54.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:53 smithi123 bash[19119]: cluster 2024-02-12T15:57:52.663997+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e85: 11 total, 11 up, 11 in 2024-02-12T15:57:54.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:53 smithi121 bash[18489]: cluster 2024-02-12T15:57:52.663997+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e85: 11 total, 11 up, 11 in 2024-02-12T15:57:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:54 smithi169 bash[19604]: cluster 2024-02-12T15:57:53.422421+0000 mgr.y (mgr.14146) 359 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:54 smithi123 bash[19119]: cluster 2024-02-12T15:57:53.422421+0000 mgr.y (mgr.14146) 359 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:55.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:54 smithi121 bash[18489]: cluster 2024-02-12T15:57:53.422421+0000 mgr.y (mgr.14146) 359 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:56 smithi169 bash[19604]: cluster 2024-02-12T15:57:55.423110+0000 mgr.y (mgr.14146) 360 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:56 smithi123 bash[19119]: cluster 2024-02-12T15:57:55.423110+0000 mgr.y (mgr.14146) 360 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:56 smithi121 bash[18489]: cluster 2024-02-12T15:57:55.423110+0000 mgr.y (mgr.14146) 360 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:58 smithi169 bash[19604]: cluster 2024-02-12T15:57:57.423717+0000 mgr.y (mgr.14146) 361 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:58 smithi169 bash[19604]: audit 2024-02-12T15:57:58.356813+0000 mon.a (mon.0) 676 : audit [INF] from='client.24421 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]: dispatch 2024-02-12T15:57:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:58 smithi169 bash[19604]: audit 2024-02-12T15:57:58.358861+0000 mon.c (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.169:0/4044845642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]: dispatch 2024-02-12T15:57:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:58 smithi169 bash[19604]: audit 2024-02-12T15:57:58.369312+0000 mon.a (mon.0) 677 : audit [INF] from='client.24421 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]': finished 2024-02-12T15:57:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:58 smithi169 bash[19604]: cluster 2024-02-12T15:57:58.369440+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e86: 12 total, 11 up, 12 in 2024-02-12T15:57:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:58 smithi169 bash[19604]: audit 2024-02-12T15:57:58.372245+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:57:58.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:58 smithi123 bash[19119]: cluster 2024-02-12T15:57:57.423717+0000 mgr.y (mgr.14146) 361 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:58.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:58 smithi123 bash[19119]: audit 2024-02-12T15:57:58.356813+0000 mon.a (mon.0) 676 : audit [INF] from='client.24421 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]: dispatch 2024-02-12T15:57:58.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:58 smithi123 bash[19119]: audit 2024-02-12T15:57:58.358861+0000 mon.c (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.169:0/4044845642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]: dispatch 2024-02-12T15:57:58.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:58 smithi123 bash[19119]: audit 2024-02-12T15:57:58.369312+0000 mon.a (mon.0) 677 : audit [INF] from='client.24421 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]': finished 2024-02-12T15:57:58.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:58 smithi123 bash[19119]: cluster 2024-02-12T15:57:58.369440+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e86: 12 total, 11 up, 12 in 2024-02-12T15:57:58.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:58 smithi123 bash[19119]: audit 2024-02-12T15:57:58.372245+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:57:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:58 smithi121 bash[18489]: cluster 2024-02-12T15:57:57.423717+0000 mgr.y (mgr.14146) 361 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:57:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:58 smithi121 bash[18489]: audit 2024-02-12T15:57:58.356813+0000 mon.a (mon.0) 676 : audit [INF] from='client.24421 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]: dispatch 2024-02-12T15:57:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:58 smithi121 bash[18489]: audit 2024-02-12T15:57:58.358861+0000 mon.c (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.169:0/4044845642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]: dispatch 2024-02-12T15:57:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:58 smithi121 bash[18489]: audit 2024-02-12T15:57:58.369312+0000 mon.a (mon.0) 677 : audit [INF] from='client.24421 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf"}]': finished 2024-02-12T15:57:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:58 smithi121 bash[18489]: cluster 2024-02-12T15:57:58.369440+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e86: 12 total, 11 up, 12 in 2024-02-12T15:57:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:58 smithi121 bash[18489]: audit 2024-02-12T15:57:58.372245+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:57:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:57:59 smithi169 bash[19604]: audit 2024-02-12T15:57:58.968162+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.169:0/3128857996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:57:59.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:57:59 smithi123 bash[19119]: audit 2024-02-12T15:57:58.968162+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.169:0/3128857996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:57:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:57:59 smithi121 bash[18489]: audit 2024-02-12T15:57:58.968162+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.169:0/3128857996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-12T15:58:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:00 smithi169 bash[19604]: cluster 2024-02-12T15:57:59.424315+0000 mgr.y (mgr.14146) 362 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:00 smithi123 bash[19119]: cluster 2024-02-12T15:57:59.424315+0000 mgr.y (mgr.14146) 362 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:00 smithi121 bash[18489]: cluster 2024-02-12T15:57:59.424315+0000 mgr.y (mgr.14146) 362 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:02 smithi169 bash[19604]: cluster 2024-02-12T15:58:01.424922+0000 mgr.y (mgr.14146) 363 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:02 smithi169 bash[19604]: cluster 2024-02-12T15:58:01.758432+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:02 smithi169 bash[19604]: cluster 2024-02-12T15:58:01.758525+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:02 smithi123 bash[19119]: cluster 2024-02-12T15:58:01.424922+0000 mgr.y (mgr.14146) 363 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:02 smithi123 bash[19119]: cluster 2024-02-12T15:58:01.758432+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:02 smithi123 bash[19119]: cluster 2024-02-12T15:58:01.758525+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:02.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:02 smithi121 bash[18489]: cluster 2024-02-12T15:58:01.424922+0000 mgr.y (mgr.14146) 363 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:02.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:02 smithi121 bash[18489]: cluster 2024-02-12T15:58:01.758432+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:02.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:02 smithi121 bash[18489]: cluster 2024-02-12T15:58:01.758525+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:03 smithi169 bash[19604]: cluster 2024-02-12T15:58:02.509296+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e87: 12 total, 11 up, 12 in 2024-02-12T15:58:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:03 smithi169 bash[19604]: audit 2024-02-12T15:58:02.513402+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:03 smithi123 bash[19119]: cluster 2024-02-12T15:58:02.509296+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e87: 12 total, 11 up, 12 in 2024-02-12T15:58:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:03 smithi123 bash[19119]: audit 2024-02-12T15:58:02.513402+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:03 smithi121 bash[18489]: cluster 2024-02-12T15:58:02.509296+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e87: 12 total, 11 up, 12 in 2024-02-12T15:58:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:03 smithi121 bash[18489]: audit 2024-02-12T15:58:02.513402+0000 mon.b (mon.2) 71 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:04.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:04 smithi169 bash[19604]: cluster 2024-02-12T15:58:03.425584+0000 mgr.y (mgr.14146) 364 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:04 smithi123 bash[19119]: cluster 2024-02-12T15:58:03.425584+0000 mgr.y (mgr.14146) 364 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:04 smithi121 bash[18489]: cluster 2024-02-12T15:58:03.425584+0000 mgr.y (mgr.14146) 364 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:05 smithi169 bash[19604]: cluster 2024-02-12T15:58:04.608460+0000 osd.2 (osd.2) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:05 smithi169 bash[19604]: cluster 2024-02-12T15:58:04.608550+0000 osd.2 (osd.2) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:05 smithi123 bash[19119]: cluster 2024-02-12T15:58:04.608460+0000 osd.2 (osd.2) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:05 smithi123 bash[19119]: cluster 2024-02-12T15:58:04.608550+0000 osd.2 (osd.2) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:05.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:05 smithi121 bash[18489]: cluster 2024-02-12T15:58:04.608460+0000 osd.2 (osd.2) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:05.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:05 smithi121 bash[18489]: cluster 2024-02-12T15:58:04.608550+0000 osd.2 (osd.2) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:06 smithi169 bash[19604]: cluster 2024-02-12T15:58:05.426198+0000 mgr.y (mgr.14146) 365 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:06 smithi169 bash[19604]: cluster 2024-02-12T15:58:05.521478+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e88: 12 total, 11 up, 12 in 2024-02-12T15:58:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:06 smithi169 bash[19604]: audit 2024-02-12T15:58:05.526279+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:06 smithi123 bash[19119]: cluster 2024-02-12T15:58:05.426198+0000 mgr.y (mgr.14146) 365 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:06 smithi123 bash[19119]: cluster 2024-02-12T15:58:05.521478+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e88: 12 total, 11 up, 12 in 2024-02-12T15:58:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:06 smithi123 bash[19119]: audit 2024-02-12T15:58:05.526279+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:06.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:06 smithi121 bash[18489]: cluster 2024-02-12T15:58:05.426198+0000 mgr.y (mgr.14146) 365 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:06.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:06 smithi121 bash[18489]: cluster 2024-02-12T15:58:05.521478+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e88: 12 total, 11 up, 12 in 2024-02-12T15:58:06.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:06 smithi121 bash[18489]: audit 2024-02-12T15:58:05.526279+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:08 smithi169 bash[19604]: cluster 2024-02-12T15:58:07.426825+0000 mgr.y (mgr.14146) 366 : cluster [DBG] pgmap v373: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:08 smithi123 bash[19119]: cluster 2024-02-12T15:58:07.426825+0000 mgr.y (mgr.14146) 366 : cluster [DBG] pgmap v373: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:08.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:08 smithi121 bash[18489]: cluster 2024-02-12T15:58:07.426825+0000 mgr.y (mgr.14146) 366 : cluster [DBG] pgmap v373: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:10 smithi123 bash[19119]: cluster 2024-02-12T15:58:09.352630+0000 osd.7 (osd.7) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:10 smithi123 bash[19119]: cluster 2024-02-12T15:58:09.352743+0000 osd.7 (osd.7) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:10 smithi123 bash[19119]: cluster 2024-02-12T15:58:09.427466+0000 mgr.y (mgr.14146) 367 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:10 smithi123 bash[19119]: cluster 2024-02-12T15:58:09.543200+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e89: 12 total, 11 up, 12 in 2024-02-12T15:58:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:10 smithi123 bash[19119]: audit 2024-02-12T15:58:09.548715+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:10 smithi123 bash[19119]: audit 2024-02-12T15:58:10.284116+0000 mon.b (mon.2) 74 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-12T15:58:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:10 smithi123 bash[19119]: audit 2024-02-12T15:58:10.286209+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:10 smithi121 bash[18489]: cluster 2024-02-12T15:58:09.352630+0000 osd.7 (osd.7) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:10 smithi121 bash[18489]: cluster 2024-02-12T15:58:09.352743+0000 osd.7 (osd.7) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:10 smithi121 bash[18489]: cluster 2024-02-12T15:58:09.427466+0000 mgr.y (mgr.14146) 367 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:10 smithi121 bash[18489]: cluster 2024-02-12T15:58:09.543200+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e89: 12 total, 11 up, 12 in 2024-02-12T15:58:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:10 smithi121 bash[18489]: audit 2024-02-12T15:58:09.548715+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:10 smithi121 bash[18489]: audit 2024-02-12T15:58:10.284116+0000 mon.b (mon.2) 74 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-12T15:58:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:10 smithi121 bash[18489]: audit 2024-02-12T15:58:10.286209+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:11.052 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:10 smithi169 bash[19604]: cluster 2024-02-12T15:58:09.352630+0000 osd.7 (osd.7) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:11.052 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:10 smithi169 bash[19604]: cluster 2024-02-12T15:58:09.352743+0000 osd.7 (osd.7) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:11.052 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:10 smithi169 bash[19604]: cluster 2024-02-12T15:58:09.427466+0000 mgr.y (mgr.14146) 367 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:11.052 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:10 smithi169 bash[19604]: cluster 2024-02-12T15:58:09.543200+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e89: 12 total, 11 up, 12 in 2024-02-12T15:58:11.052 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:10 smithi169 bash[19604]: audit 2024-02-12T15:58:09.548715+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:11.052 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:10 smithi169 bash[19604]: audit 2024-02-12T15:58:10.284116+0000 mon.b (mon.2) 74 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-12T15:58:11.053 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:10 smithi169 bash[19604]: audit 2024-02-12T15:58:10.286209+0000 mon.b (mon.2) 75 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:11.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:11 smithi169 bash[19604]: cephadm 2024-02-12T15:58:10.283125+0000 mgr.y (mgr.14146) 368 : cephadm [INF] Deploying daemon osd.11 on smithi169 2024-02-12T15:58:11.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:11 smithi123 bash[19119]: cephadm 2024-02-12T15:58:10.283125+0000 mgr.y (mgr.14146) 368 : cephadm [INF] Deploying daemon osd.11 on smithi169 2024-02-12T15:58:11.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:11 smithi121 bash[18489]: cephadm 2024-02-12T15:58:10.283125+0000 mgr.y (mgr.14146) 368 : cephadm [INF] Deploying daemon osd.11 on smithi169 2024-02-12T15:58:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:12 smithi123 bash[19119]: cluster 2024-02-12T15:58:11.428093+0000 mgr.y (mgr.14146) 369 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:12.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:12 smithi121 bash[18489]: cluster 2024-02-12T15:58:11.428093+0000 mgr.y (mgr.14146) 369 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:13.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:12 smithi169 bash[19604]: cluster 2024-02-12T15:58:11.428093+0000 mgr.y (mgr.14146) 369 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:14 smithi169 bash[19604]: cluster 2024-02-12T15:58:13.428691+0000 mgr.y (mgr.14146) 370 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:14 smithi169 bash[19604]: audit 2024-02-12T15:58:14.360963+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:14 smithi169 bash[19604]: audit 2024-02-12T15:58:14.398428+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:58:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:14 smithi123 bash[19119]: cluster 2024-02-12T15:58:13.428691+0000 mgr.y (mgr.14146) 370 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:14 smithi123 bash[19119]: audit 2024-02-12T15:58:14.360963+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:14 smithi123 bash[19119]: audit 2024-02-12T15:58:14.398428+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:58:14.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:14 smithi121 bash[18489]: cluster 2024-02-12T15:58:13.428691+0000 mgr.y (mgr.14146) 370 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:14.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:14 smithi121 bash[18489]: audit 2024-02-12T15:58:14.360963+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:14.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:14 smithi121 bash[18489]: audit 2024-02-12T15:58:14.398428+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:58:16.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:15 smithi169 bash[19604]: cluster 2024-02-12T15:58:13.852480+0000 osd.7 (osd.7) 11 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:58:16.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:15 smithi169 bash[19604]: cluster 2024-02-12T15:58:13.856177+0000 osd.7 (osd.7) 12 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:58:16.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:15 smithi169 bash[19604]: audit 2024-02-12T15:58:14.698488+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:15 smithi123 bash[19119]: cluster 2024-02-12T15:58:13.852480+0000 osd.7 (osd.7) 11 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:58:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:15 smithi123 bash[19119]: cluster 2024-02-12T15:58:13.856177+0000 osd.7 (osd.7) 12 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:58:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:15 smithi123 bash[19119]: audit 2024-02-12T15:58:14.698488+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:15 smithi121 bash[18489]: cluster 2024-02-12T15:58:13.852480+0000 osd.7 (osd.7) 11 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:58:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:15 smithi121 bash[18489]: cluster 2024-02-12T15:58:13.856177+0000 osd.7 (osd.7) 12 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:58:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:15 smithi121 bash[18489]: audit 2024-02-12T15:58:14.698488+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:16 smithi169 bash[19604]: cluster 2024-02-12T15:58:15.429332+0000 mgr.y (mgr.14146) 371 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:16 smithi123 bash[19119]: cluster 2024-02-12T15:58:15.429332+0000 mgr.y (mgr.14146) 371 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:16 smithi121 bash[18489]: cluster 2024-02-12T15:58:15.429332+0000 mgr.y (mgr.14146) 371 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:18.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:17 smithi169 bash[19604]: cluster 2024-02-12T15:58:17.255115+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:18.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:17 smithi169 bash[19604]: cluster 2024-02-12T15:58:17.255205+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:18.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:17 smithi123 bash[19119]: cluster 2024-02-12T15:58:17.255115+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:18.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:17 smithi123 bash[19119]: cluster 2024-02-12T15:58:17.255205+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:18.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:17 smithi121 bash[18489]: cluster 2024-02-12T15:58:17.255115+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:18.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:17 smithi121 bash[18489]: cluster 2024-02-12T15:58:17.255205+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:18 smithi169 bash[19604]: cluster 2024-02-12T15:58:17.429921+0000 mgr.y (mgr.14146) 372 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:18 smithi169 bash[19604]: cluster 2024-02-12T15:58:17.715779+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e90: 12 total, 11 up, 12 in 2024-02-12T15:58:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:18 smithi169 bash[19604]: audit 2024-02-12T15:58:17.718066+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:18 smithi123 bash[19119]: cluster 2024-02-12T15:58:17.429921+0000 mgr.y (mgr.14146) 372 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:18 smithi123 bash[19119]: cluster 2024-02-12T15:58:17.715779+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e90: 12 total, 11 up, 12 in 2024-02-12T15:58:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:18 smithi123 bash[19119]: audit 2024-02-12T15:58:17.718066+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:18 smithi121 bash[18489]: cluster 2024-02-12T15:58:17.429921+0000 mgr.y (mgr.14146) 372 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:18 smithi121 bash[18489]: cluster 2024-02-12T15:58:17.715779+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e90: 12 total, 11 up, 12 in 2024-02-12T15:58:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:18 smithi121 bash[18489]: audit 2024-02-12T15:58:17.718066+0000 mon.b (mon.2) 77 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:20 smithi169 bash[19604]: cluster 2024-02-12T15:58:19.430544+0000 mgr.y (mgr.14146) 373 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:21.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:20 smithi123 bash[19119]: cluster 2024-02-12T15:58:19.430544+0000 mgr.y (mgr.14146) 373 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:21.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:20 smithi121 bash[18489]: cluster 2024-02-12T15:58:19.430544+0000 mgr.y (mgr.14146) 373 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:21 smithi169 bash[19604]: audit 2024-02-12T15:58:20.770110+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:21 smithi169 bash[19604]: audit 2024-02-12T15:58:20.778188+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:21 smithi169 bash[19604]: audit 2024-02-12T15:58:20.780222+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:58:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:21 smithi169 bash[19604]: audit 2024-02-12T15:58:20.783871+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:21 smithi169 bash[19604]: audit 2024-02-12T15:58:21.236715+0000 mon.a (mon.0) 687 : audit [INF] from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-12T15:58:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:21 smithi169 bash[19604]: audit 2024-02-12T15:58:21.239120+0000 mon.c (mon.1) 19 : audit [INF] from='osd.11 v1:172.21.15.169:6812/3696812305' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-12T15:58:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:21 smithi123 bash[19119]: audit 2024-02-12T15:58:20.770110+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:21 smithi123 bash[19119]: audit 2024-02-12T15:58:20.778188+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:21 smithi123 bash[19119]: audit 2024-02-12T15:58:20.780222+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:58:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:21 smithi123 bash[19119]: audit 2024-02-12T15:58:20.783871+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:21 smithi123 bash[19119]: audit 2024-02-12T15:58:21.236715+0000 mon.a (mon.0) 687 : audit [INF] from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-12T15:58:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:21 smithi123 bash[19119]: audit 2024-02-12T15:58:21.239120+0000 mon.c (mon.1) 19 : audit [INF] from='osd.11 v1:172.21.15.169:6812/3696812305' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-12T15:58:22.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:21 smithi121 bash[18489]: audit 2024-02-12T15:58:20.770110+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:22.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:21 smithi121 bash[18489]: audit 2024-02-12T15:58:20.778188+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:22.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:21 smithi121 bash[18489]: audit 2024-02-12T15:58:20.780222+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:58:22.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:21 smithi121 bash[18489]: audit 2024-02-12T15:58:20.783871+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:22.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:21 smithi121 bash[18489]: audit 2024-02-12T15:58:21.236715+0000 mon.a (mon.0) 687 : audit [INF] from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-12T15:58:22.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:21 smithi121 bash[18489]: audit 2024-02-12T15:58:21.239120+0000 mon.c (mon.1) 19 : audit [INF] from='osd.11 v1:172.21.15.169:6812/3696812305' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-12T15:58:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:22 smithi169 bash[19604]: cluster 2024-02-12T15:58:21.431195+0000 mgr.y (mgr.14146) 374 : cluster [DBG] pgmap v382: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:22 smithi169 bash[19604]: audit 2024-02-12T15:58:21.792011+0000 mon.a (mon.0) 688 : audit [INF] from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-12T15:58:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:22 smithi169 bash[19604]: cluster 2024-02-12T15:58:21.792081+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e91: 12 total, 11 up, 12 in 2024-02-12T15:58:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:22 smithi169 bash[19604]: audit 2024-02-12T15:58:21.794922+0000 mon.a (mon.0) 690 : audit [INF] from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:58:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:22 smithi169 bash[19604]: audit 2024-02-12T15:58:21.797204+0000 mon.c (mon.1) 20 : audit [INF] from='osd.11 v1:172.21.15.169:6812/3696812305' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:58:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:22 smithi169 bash[19604]: audit 2024-02-12T15:58:21.801864+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:22 smithi123 bash[19119]: cluster 2024-02-12T15:58:21.431195+0000 mgr.y (mgr.14146) 374 : cluster [DBG] pgmap v382: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:22 smithi123 bash[19119]: audit 2024-02-12T15:58:21.792011+0000 mon.a (mon.0) 688 : audit [INF] from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-12T15:58:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:22 smithi123 bash[19119]: cluster 2024-02-12T15:58:21.792081+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e91: 12 total, 11 up, 12 in 2024-02-12T15:58:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:22 smithi123 bash[19119]: audit 2024-02-12T15:58:21.794922+0000 mon.a (mon.0) 690 : audit [INF] from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:58:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:22 smithi123 bash[19119]: audit 2024-02-12T15:58:21.797204+0000 mon.c (mon.1) 20 : audit [INF] from='osd.11 v1:172.21.15.169:6812/3696812305' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:58:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:22 smithi123 bash[19119]: audit 2024-02-12T15:58:21.801864+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:22 smithi121 bash[18489]: cluster 2024-02-12T15:58:21.431195+0000 mgr.y (mgr.14146) 374 : cluster [DBG] pgmap v382: 1 pgs: 1 active+clean; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:22 smithi121 bash[18489]: audit 2024-02-12T15:58:21.792011+0000 mon.a (mon.0) 688 : audit [INF] from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-12T15:58:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:22 smithi121 bash[18489]: cluster 2024-02-12T15:58:21.792081+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e91: 12 total, 11 up, 12 in 2024-02-12T15:58:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:22 smithi121 bash[18489]: audit 2024-02-12T15:58:21.794922+0000 mon.a (mon.0) 690 : audit [INF] from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:58:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:22 smithi121 bash[18489]: audit 2024-02-12T15:58:21.797204+0000 mon.c (mon.1) 20 : audit [INF] from='osd.11 v1:172.21.15.169:6812/3696812305' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-02-12T15:58:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:22 smithi121 bash[18489]: audit 2024-02-12T15:58:21.801864+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:23.810 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 11 on host 'smithi169' 2024-02-12T15:58:24.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:23 smithi123 bash[19119]: audit 2024-02-12T15:58:22.796414+0000 mon.a (mon.0) 691 : audit [INF] from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:58:24.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:23 smithi123 bash[19119]: cluster 2024-02-12T15:58:22.796570+0000 mon.a (mon.0) 692 : cluster [DBG] osdmap e92: 12 total, 11 up, 12 in 2024-02-12T15:58:24.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:23 smithi123 bash[19119]: audit 2024-02-12T15:58:22.801547+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:24.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:23 smithi123 bash[19119]: audit 2024-02-12T15:58:22.810796+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:24.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:23 smithi121 bash[18489]: audit 2024-02-12T15:58:22.796414+0000 mon.a (mon.0) 691 : audit [INF] from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:58:24.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:23 smithi121 bash[18489]: cluster 2024-02-12T15:58:22.796570+0000 mon.a (mon.0) 692 : cluster [DBG] osdmap e92: 12 total, 11 up, 12 in 2024-02-12T15:58:24.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:23 smithi121 bash[18489]: audit 2024-02-12T15:58:22.801547+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:24.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:23 smithi121 bash[18489]: audit 2024-02-12T15:58:22.810796+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:23 smithi169 bash[19604]: audit 2024-02-12T15:58:22.796414+0000 mon.a (mon.0) 691 : audit [INF] from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-02-12T15:58:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:23 smithi169 bash[19604]: cluster 2024-02-12T15:58:22.796570+0000 mon.a (mon.0) 692 : cluster [DBG] osdmap e92: 12 total, 11 up, 12 in 2024-02-12T15:58:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:23 smithi169 bash[19604]: audit 2024-02-12T15:58:22.801547+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:23 smithi169 bash[19604]: audit 2024-02-12T15:58:22.810796+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:24.551 DEBUG:teuthology.orchestra.run.smithi169:osd.11> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.11.service 2024-02-12T15:58:24.553 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-02-12T15:58:24.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-12T15:58:24.840 INFO:journalctl@ceph.osd.11.smithi169.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: cluster 2024-02-12T15:58:22.275526+0000 osd.11 (osd.11) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: cluster 2024-02-12T15:58:22.275633+0000 osd.11 (osd.11) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: cluster 2024-02-12T15:58:23.431791+0000 mgr.y (mgr.14146) 375 : cluster [DBG] pgmap v385: 1 pgs: 1 peering; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: audit 2024-02-12T15:58:23.801111+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: audit 2024-02-12T15:58:23.806851+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: cluster 2024-02-12T15:58:23.810899+0000 mon.a (mon.0) 694 : cluster [INF] osd.11 v1:172.21.15.169:6812/3696812305 boot 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: cluster 2024-02-12T15:58:23.810978+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e93: 12 total, 12 up, 12 in 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: audit 2024-02-12T15:58:23.815996+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:24 smithi123 bash[19119]: audit 2024-02-12T15:58:23.822516+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:58:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: cluster 2024-02-12T15:58:22.275526+0000 osd.11 (osd.11) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: cluster 2024-02-12T15:58:22.275633+0000 osd.11 (osd.11) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: cluster 2024-02-12T15:58:23.431791+0000 mgr.y (mgr.14146) 375 : cluster [DBG] pgmap v385: 1 pgs: 1 peering; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: audit 2024-02-12T15:58:23.801111+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: audit 2024-02-12T15:58:23.806851+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: cluster 2024-02-12T15:58:23.810899+0000 mon.a (mon.0) 694 : cluster [INF] osd.11 v1:172.21.15.169:6812/3696812305 boot 2024-02-12T15:58:25.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: cluster 2024-02-12T15:58:23.810978+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e93: 12 total, 12 up, 12 in 2024-02-12T15:58:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: audit 2024-02-12T15:58:23.815996+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:25.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:24 smithi121 bash[18489]: audit 2024-02-12T15:58:23.822516+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: cluster 2024-02-12T15:58:22.275526+0000 osd.11 (osd.11) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: cluster 2024-02-12T15:58:22.275633+0000 osd.11 (osd.11) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: cluster 2024-02-12T15:58:23.431791+0000 mgr.y (mgr.14146) 375 : cluster [DBG] pgmap v385: 1 pgs: 1 peering; 0 B data, 3.1 GiB used, 980 GiB / 983 GiB avail 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: audit 2024-02-12T15:58:23.801111+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: audit 2024-02-12T15:58:23.806851+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: cluster 2024-02-12T15:58:23.810899+0000 mon.a (mon.0) 694 : cluster [INF] osd.11 v1:172.21.15.169:6812/3696812305 boot 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: cluster 2024-02-12T15:58:23.810978+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e93: 12 total, 12 up, 12 in 2024-02-12T15:58:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: audit 2024-02-12T15:58:23.815996+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-12T15:58:25.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:24 smithi169 bash[19604]: audit 2024-02-12T15:58:23.822516+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:58:26.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:25 smithi121 bash[18489]: cluster 2024-02-12T15:58:24.821147+0000 mon.a (mon.0) 696 : cluster [DBG] osdmap e94: 12 total, 12 up, 12 in 2024-02-12T15:58:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:25 smithi123 bash[19119]: cluster 2024-02-12T15:58:24.821147+0000 mon.a (mon.0) 696 : cluster [DBG] osdmap e94: 12 total, 12 up, 12 in 2024-02-12T15:58:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:25 smithi169 bash[19604]: cluster 2024-02-12T15:58:24.821147+0000 mon.a (mon.0) 696 : cluster [DBG] osdmap e94: 12 total, 12 up, 12 in 2024-02-12T15:58:27.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:26 smithi123 bash[19119]: cluster 2024-02-12T15:58:25.432427+0000 mgr.y (mgr.14146) 376 : cluster [DBG] pgmap v388: 1 pgs: 1 peering; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:27.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:26 smithi123 bash[19119]: cluster 2024-02-12T15:58:25.831124+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e95: 12 total, 12 up, 12 in 2024-02-12T15:58:27.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:26 smithi123 bash[19119]: cluster 2024-02-12T15:58:26.052075+0000 osd.8 (osd.8) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:27.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:26 smithi123 bash[19119]: cluster 2024-02-12T15:58:26.052184+0000 osd.8 (osd.8) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:27.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:26 smithi123 bash[19119]: cluster 2024-02-12T15:58:26.726810+0000 osd.9 (osd.9) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:27.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:26 smithi123 bash[19119]: cluster 2024-02-12T15:58:26.726904+0000 osd.9 (osd.9) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:27.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:26 smithi121 bash[18489]: cluster 2024-02-12T15:58:25.432427+0000 mgr.y (mgr.14146) 376 : cluster [DBG] pgmap v388: 1 pgs: 1 peering; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:27.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:26 smithi121 bash[18489]: cluster 2024-02-12T15:58:25.831124+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e95: 12 total, 12 up, 12 in 2024-02-12T15:58:27.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:26 smithi121 bash[18489]: cluster 2024-02-12T15:58:26.052075+0000 osd.8 (osd.8) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:27.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:26 smithi121 bash[18489]: cluster 2024-02-12T15:58:26.052184+0000 osd.8 (osd.8) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:27.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:26 smithi121 bash[18489]: cluster 2024-02-12T15:58:26.726810+0000 osd.9 (osd.9) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:27.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:26 smithi121 bash[18489]: cluster 2024-02-12T15:58:26.726904+0000 osd.9 (osd.9) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:26 smithi169 bash[19604]: cluster 2024-02-12T15:58:25.432427+0000 mgr.y (mgr.14146) 376 : cluster [DBG] pgmap v388: 1 pgs: 1 peering; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:26 smithi169 bash[19604]: cluster 2024-02-12T15:58:25.831124+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e95: 12 total, 12 up, 12 in 2024-02-12T15:58:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:26 smithi169 bash[19604]: cluster 2024-02-12T15:58:26.052075+0000 osd.8 (osd.8) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:26 smithi169 bash[19604]: cluster 2024-02-12T15:58:26.052184+0000 osd.8 (osd.8) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:26 smithi169 bash[19604]: cluster 2024-02-12T15:58:26.726810+0000 osd.9 (osd.9) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:26 smithi169 bash[19604]: cluster 2024-02-12T15:58:26.726904+0000 osd.9 (osd.9) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:27.959 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T15:58:28.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:27 smithi123 bash[19119]: cluster 2024-02-12T15:58:26.841868+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e96: 12 total, 12 up, 12 in 2024-02-12T15:58:28.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:27 smithi121 bash[18489]: cluster 2024-02-12T15:58:26.841868+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e96: 12 total, 12 up, 12 in 2024-02-12T15:58:28.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:27 smithi169 bash[19604]: cluster 2024-02-12T15:58:26.841868+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e96: 12 total, 12 up, 12 in 2024-02-12T15:58:28.622 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":96,"num_osds":12,"num_up_osds":12,"osd_up_since":1707753503,"num_in_osds":12,"osd_in_since":1707753478,"num_remapped_pgs":0} 2024-02-12T15:58:28.623 INFO:tasks.cephadm:Setting up client nodes... 2024-02-12T15:58:28.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-12T15:58:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:28 smithi121 bash[18489]: cluster 2024-02-12T15:58:27.433006+0000 mgr.y (mgr.14146) 377 : cluster [DBG] pgmap v391: 1 pgs: 1 peering; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:28 smithi121 bash[18489]: audit 2024-02-12T15:58:27.952497+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? v1:172.21.15.121:0/1703229800' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-12T15:58:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:28 smithi123 bash[19119]: cluster 2024-02-12T15:58:27.433006+0000 mgr.y (mgr.14146) 377 : cluster [DBG] pgmap v391: 1 pgs: 1 peering; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:28 smithi123 bash[19119]: audit 2024-02-12T15:58:27.952497+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? v1:172.21.15.121:0/1703229800' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-12T15:58:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:28 smithi169 bash[19604]: cluster 2024-02-12T15:58:27.433006+0000 mgr.y (mgr.14146) 377 : cluster [DBG] pgmap v391: 1 pgs: 1 peering; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:28 smithi169 bash[19604]: audit 2024-02-12T15:58:27.952497+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? v1:172.21.15.121:0/1703229800' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-12T15:58:31.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:30 smithi123 bash[19119]: cluster 2024-02-12T15:58:29.433597+0000 mgr.y (mgr.14146) 378 : cluster [DBG] pgmap v392: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:31.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:30 smithi121 bash[18489]: cluster 2024-02-12T15:58:29.433597+0000 mgr.y (mgr.14146) 378 : cluster [DBG] pgmap v392: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:31.203 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:30 smithi169 bash[19604]: cluster 2024-02-12T15:58:29.433597+0000 mgr.y (mgr.14146) 378 : cluster [DBG] pgmap v392: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:31.928 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-02-12T15:58:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAnQMplkQyQNhAAONXOZDwVrgK5xEpX8beLIw== 2024-02-12T15:58:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:32 smithi123 bash[19119]: cluster 2024-02-12T15:58:30.399947+0000 osd.4 (osd.4) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:32 smithi123 bash[19119]: cluster 2024-02-12T15:58:30.400040+0000 osd.4 (osd.4) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:32 smithi123 bash[19119]: cluster 2024-02-12T15:58:31.085030+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e97: 12 total, 12 up, 12 in 2024-02-12T15:58:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:32 smithi123 bash[19119]: cluster 2024-02-12T15:58:31.434224+0000 mgr.y (mgr.14146) 379 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:32 smithi123 bash[19119]: audit 2024-02-12T15:58:31.914976+0000 mon.a (mon.0) 701 : audit [INF] from='client.? v1:172.21.15.121:0/2570423102' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:32.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:32 smithi123 bash[19119]: audit 2024-02-12T15:58:31.922242+0000 mon.a (mon.0) 702 : audit [INF] from='client.? v1:172.21.15.121:0/2570423102' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:32.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:32 smithi121 bash[18489]: cluster 2024-02-12T15:58:30.399947+0000 osd.4 (osd.4) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:32.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:32 smithi121 bash[18489]: cluster 2024-02-12T15:58:30.400040+0000 osd.4 (osd.4) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:32.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:32 smithi121 bash[18489]: cluster 2024-02-12T15:58:31.085030+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e97: 12 total, 12 up, 12 in 2024-02-12T15:58:32.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:32 smithi121 bash[18489]: cluster 2024-02-12T15:58:31.434224+0000 mgr.y (mgr.14146) 379 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:32.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:32 smithi121 bash[18489]: audit 2024-02-12T15:58:31.914976+0000 mon.a (mon.0) 701 : audit [INF] from='client.? v1:172.21.15.121:0/2570423102' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:32.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:32 smithi121 bash[18489]: audit 2024-02-12T15:58:31.922242+0000 mon.a (mon.0) 702 : audit [INF] from='client.? v1:172.21.15.121:0/2570423102' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:32 smithi169 bash[19604]: cluster 2024-02-12T15:58:30.399947+0000 osd.4 (osd.4) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:32 smithi169 bash[19604]: cluster 2024-02-12T15:58:30.400040+0000 osd.4 (osd.4) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:32 smithi169 bash[19604]: cluster 2024-02-12T15:58:31.085030+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e97: 12 total, 12 up, 12 in 2024-02-12T15:58:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:32 smithi169 bash[19604]: cluster 2024-02-12T15:58:31.434224+0000 mgr.y (mgr.14146) 379 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:32 smithi169 bash[19604]: audit 2024-02-12T15:58:31.914976+0000 mon.a (mon.0) 701 : audit [INF] from='client.? v1:172.21.15.121:0/2570423102' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:32 smithi169 bash[19604]: audit 2024-02-12T15:58:31.922242+0000 mon.a (mon.0) 702 : audit [INF] from='client.? v1:172.21.15.121:0/2570423102' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:32.568 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-12T15:58:32.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-12T15:58:32.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-12T15:58:32.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-12T15:58:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: cephadm 2024-02-12T15:58:32.961227+0000 mgr.y (mgr.14146) 380 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:58:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.973568+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.977842+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.980867+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.982031+0000 mon.b (mon.2) 86 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.983585+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.985277+0000 mon.b (mon.2) 87 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.985862+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: cephadm 2024-02-12T15:58:32.987435+0000 mgr.y (mgr.14146) 381 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 5353M 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.988163+0000 mon.b (mon.2) 88 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.990532+0000 mon.b (mon.2) 89 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:32.995766+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:33.004135+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:33.005987+0000 mon.b (mon.2) 91 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:58:34.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:33 smithi169 bash[19604]: audit 2024-02-12T15:58:33.011075+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: cephadm 2024-02-12T15:58:32.961227+0000 mgr.y (mgr.14146) 380 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:58:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.973568+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.977842+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.980867+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.982031+0000 mon.b (mon.2) 86 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.983585+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.985277+0000 mon.b (mon.2) 87 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.985862+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: cephadm 2024-02-12T15:58:32.987435+0000 mgr.y (mgr.14146) 381 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 5353M 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.988163+0000 mon.b (mon.2) 88 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.990532+0000 mon.b (mon.2) 89 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:32.995766+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:33.004135+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:33.005987+0000 mon.b (mon.2) 91 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:58:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:33 smithi123 bash[19119]: audit 2024-02-12T15:58:33.011075+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: cephadm 2024-02-12T15:58:32.961227+0000 mgr.y (mgr.14146) 380 : cephadm [INF] Detected new or changed devices on smithi169 2024-02-12T15:58:34.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.973568+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.977842+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.980867+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.982031+0000 mon.b (mon.2) 86 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.983585+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.985277+0000 mon.b (mon.2) 87 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.985862+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: cephadm 2024-02-12T15:58:32.987435+0000 mgr.y (mgr.14146) 381 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 5353M 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.988163+0000 mon.b (mon.2) 88 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.990532+0000 mon.b (mon.2) 89 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:32.995766+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:33.004135+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:33.005987+0000 mon.b (mon.2) 91 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:58:34.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:33 smithi121 bash[18489]: audit 2024-02-12T15:58:33.011075+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:58:35.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:35 smithi123 bash[19119]: cluster 2024-02-12T15:58:33.434879+0000 mgr.y (mgr.14146) 382 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:35 smithi169 bash[19604]: cluster 2024-02-12T15:58:33.434879+0000 mgr.y (mgr.14146) 382 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:35.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:35 smithi121 bash[18489]: cluster 2024-02-12T15:58:33.434879+0000 mgr.y (mgr.14146) 382 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:35.694 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2024-02-12T15:58:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQArQMpl+iCDKBAAsyHxMSeTU9/voxxJULJT+w== 2024-02-12T15:58:36.258 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-02-12T15:58:36.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-12T15:58:36.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-12T15:58:36.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-12T15:58:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:36 smithi123 bash[19119]: cluster 2024-02-12T15:58:35.435499+0000 mgr.y (mgr.14146) 383 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:36 smithi123 bash[19119]: audit 2024-02-12T15:58:35.679233+0000 mon.a (mon.0) 710 : audit [INF] from='client.24461 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:36 smithi123 bash[19119]: audit 2024-02-12T15:58:35.683116+0000 mon.b (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.123:0/3277600886' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:36 smithi123 bash[19119]: audit 2024-02-12T15:58:35.687438+0000 mon.a (mon.0) 711 : audit [INF] from='client.24461 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:36 smithi121 bash[18489]: cluster 2024-02-12T15:58:35.435499+0000 mgr.y (mgr.14146) 383 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:36 smithi121 bash[18489]: audit 2024-02-12T15:58:35.679233+0000 mon.a (mon.0) 710 : audit [INF] from='client.24461 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:36 smithi121 bash[18489]: audit 2024-02-12T15:58:35.683116+0000 mon.b (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.123:0/3277600886' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:36 smithi121 bash[18489]: audit 2024-02-12T15:58:35.687438+0000 mon.a (mon.0) 711 : audit [INF] from='client.24461 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:36 smithi169 bash[19604]: cluster 2024-02-12T15:58:35.435499+0000 mgr.y (mgr.14146) 383 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:36 smithi169 bash[19604]: audit 2024-02-12T15:58:35.679233+0000 mon.a (mon.0) 710 : audit [INF] from='client.24461 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:36 smithi169 bash[19604]: audit 2024-02-12T15:58:35.683116+0000 mon.b (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.123:0/3277600886' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:36 smithi169 bash[19604]: audit 2024-02-12T15:58:35.687438+0000 mon.a (mon.0) 711 : audit [INF] from='client.24461 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:37 smithi123 bash[19119]: cluster 2024-02-12T15:58:36.101105+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:37.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:37 smithi123 bash[19119]: cluster 2024-02-12T15:58:36.101194+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:37.393 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:37 smithi169 bash[19604]: cluster 2024-02-12T15:58:36.101105+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:37.393 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:37 smithi169 bash[19604]: cluster 2024-02-12T15:58:36.101194+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:37.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:37 smithi121 bash[18489]: cluster 2024-02-12T15:58:36.101105+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:37.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:37 smithi121 bash[18489]: cluster 2024-02-12T15:58:36.101194+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:38 smithi123 bash[19119]: cluster 2024-02-12T15:58:37.106145+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e98: 12 total, 12 up, 12 in 2024-02-12T15:58:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:38 smithi123 bash[19119]: cluster 2024-02-12T15:58:37.436164+0000 mgr.y (mgr.14146) 384 : cluster [DBG] pgmap v398: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:38.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:38 smithi121 bash[18489]: cluster 2024-02-12T15:58:37.106145+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e98: 12 total, 12 up, 12 in 2024-02-12T15:58:38.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:38 smithi121 bash[18489]: cluster 2024-02-12T15:58:37.436164+0000 mgr.y (mgr.14146) 384 : cluster [DBG] pgmap v398: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:38 smithi169 bash[19604]: cluster 2024-02-12T15:58:37.106145+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e98: 12 total, 12 up, 12 in 2024-02-12T15:58:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:38 smithi169 bash[19604]: cluster 2024-02-12T15:58:37.436164+0000 mgr.y (mgr.14146) 384 : cluster [DBG] pgmap v398: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:39.731 INFO:teuthology.orchestra.run.smithi169.stdout:[client.2] 2024-02-12T15:58:39.732 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQAvQMplH9HLKhAA3yYR2bFp0ilW08uDsK1JgA== 2024-02-12T15:58:40.334 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-02-12T15:58:40.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-02-12T15:58:40.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-02-12T15:58:40.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T15:58:40.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:40 smithi121 bash[18489]: cluster 2024-02-12T15:58:39.436847+0000 mgr.y (mgr.14146) 385 : cluster [DBG] pgmap v399: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:40.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:40 smithi121 bash[18489]: audit 2024-02-12T15:58:39.717623+0000 mon.a (mon.0) 713 : audit [INF] from='client.24460 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:40.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:40 smithi121 bash[18489]: audit 2024-02-12T15:58:39.719950+0000 mon.c (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.169:0/1572876704' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:40.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:40 smithi121 bash[18489]: audit 2024-02-12T15:58:39.724658+0000 mon.a (mon.0) 714 : audit [INF] from='client.24460 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:40 smithi169 bash[19604]: cluster 2024-02-12T15:58:39.436847+0000 mgr.y (mgr.14146) 385 : cluster [DBG] pgmap v399: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:40 smithi169 bash[19604]: audit 2024-02-12T15:58:39.717623+0000 mon.a (mon.0) 713 : audit [INF] from='client.24460 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:40 smithi169 bash[19604]: audit 2024-02-12T15:58:39.719950+0000 mon.c (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.169:0/1572876704' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:40 smithi169 bash[19604]: audit 2024-02-12T15:58:39.724658+0000 mon.a (mon.0) 714 : audit [INF] from='client.24460 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:40 smithi123 bash[19119]: cluster 2024-02-12T15:58:39.436847+0000 mgr.y (mgr.14146) 385 : cluster [DBG] pgmap v399: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:40 smithi123 bash[19119]: audit 2024-02-12T15:58:39.717623+0000 mon.a (mon.0) 713 : audit [INF] from='client.24460 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:40 smithi123 bash[19119]: audit 2024-02-12T15:58:39.719950+0000 mon.c (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.169:0/1572876704' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-12T15:58:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:40 smithi123 bash[19119]: audit 2024-02-12T15:58:39.724658+0000 mon.a (mon.0) 714 : audit [INF] from='client.24460 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-12T15:58:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:42 smithi169 bash[19604]: cluster 2024-02-12T15:58:41.437444+0000 mgr.y (mgr.14146) 386 : cluster [DBG] pgmap v400: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:42 smithi123 bash[19119]: cluster 2024-02-12T15:58:41.437444+0000 mgr.y (mgr.14146) 386 : cluster [DBG] pgmap v400: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:42.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:42 smithi121 bash[18489]: cluster 2024-02-12T15:58:41.437444+0000 mgr.y (mgr.14146) 386 : cluster [DBG] pgmap v400: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:44 smithi169 bash[19604]: cluster 2024-02-12T15:58:43.438008+0000 mgr.y (mgr.14146) 387 : cluster [DBG] pgmap v401: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:44 smithi123 bash[19119]: cluster 2024-02-12T15:58:43.438008+0000 mgr.y (mgr.14146) 387 : cluster [DBG] pgmap v401: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:44 smithi121 bash[18489]: cluster 2024-02-12T15:58:43.438008+0000 mgr.y (mgr.14146) 387 : cluster [DBG] pgmap v401: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:45.157 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:58:46.376 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:46 smithi121 bash[18489]: cluster 2024-02-12T15:58:45.438645+0000 mgr.y (mgr.14146) 388 : cluster [DBG] pgmap v402: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:46.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:46 smithi123 bash[19119]: cluster 2024-02-12T15:58:45.438645+0000 mgr.y (mgr.14146) 388 : cluster [DBG] pgmap v402: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:46 smithi169 bash[19604]: cluster 2024-02-12T15:58:45.438645+0000 mgr.y (mgr.14146) 388 : cluster [DBG] pgmap v402: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:48.163 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T15:58:48.164 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":98,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","created":"2024-02-12T15:47:18.273813+0000","modified":"2024-02-12T15:58:37.092239+0000","last_up_change":"2024-02-12T15:58:23.801839+0000","last_in_change":"2024-02-12T15:57:58.358258+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T15:51:35.886878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"3a3badac-97ea-4ee1-ad82-1b67a4fe94f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1815232143}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1815232143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":1815232143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1815232143}]},"public_addr":"172.21.15.121:6801/1815232143","cluster_addr":"172.21.15.121:6802/1815232143","heartbeat_back_addr":"172.21.15.121:6804/1815232143","heartbeat_front_addr":"172.21.15.121:6803/1815232143","state":["exists","up"]},{"osd":1,"uuid":"c6140309-5703-4538-9739-276fbbdaa880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2968433670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2968433670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":2968433670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2968433670}]},"public_addr":"172.21.15.121:6805/2968433670","cluster_addr":"172.21.15.121:6806/2968433670","heartbeat_back_addr":"172.21.15.121:6808/2968433670","heartbeat_front_addr":"172.21.15.121:6807/2968433670","state":["exists","up"]},{"osd":2,"uuid":"7c81e1a4-cae0-48d4-98ab-0becb6b31170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":2158187852}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":2158187852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":2158187852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":2158187852}]},"public_addr":"172.21.15.121:6809/2158187852","cluster_addr":"172.21.15.121:6810/2158187852","heartbeat_back_addr":"172.21.15.121:6812/2158187852","heartbeat_front_addr":"172.21.15.121:6811/2158187852","state":["exists","up"]},{"osd":3,"uuid":"5f824526-596d-42e1-8b7e-cd8676ded6dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":1415189366}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":1415189366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6816","nonce":1415189366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":1415189366}]},"public_addr":"172.21.15.121:6813/1415189366","cluster_addr":"172.21.15.121:6814/1415189366","heartbeat_back_addr":"172.21.15.121:6816/1415189366","heartbeat_front_addr":"172.21.15.121:6815/1415189366","state":["exists","up"]},{"osd":4,"uuid":"14c2f878-99ce-4b51-8d9c-131c1d883b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6800","nonce":3447700123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":3447700123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":3447700123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":3447700123}]},"public_addr":"172.21.15.123:6800/3447700123","cluster_addr":"172.21.15.123:6801/3447700123","heartbeat_back_addr":"172.21.15.123:6803/3447700123","heartbeat_front_addr":"172.21.15.123:6802/3447700123","state":["exists","up"]},{"osd":5,"uuid":"75ecf26a-d7ed-46e7-a40e-6f719e2c3602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":961317306}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":961317306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":961317306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":961317306}]},"public_addr":"172.21.15.123:6804/961317306","cluster_addr":"172.21.15.123:6805/961317306","heartbeat_back_addr":"172.21.15.123:6807/961317306","heartbeat_front_addr":"172.21.15.123:6806/961317306","state":["exists","up"]},{"osd":6,"uuid":"71e715e6-eac1-4386-927e-51df0cc419c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":1377261784}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":1377261784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":1377261784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":1377261784}]},"public_addr":"172.21.15.123:6808/1377261784","cluster_addr":"172.21.15.123:6809/1377261784","heartbeat_back_addr":"172.21.15.123:6811/1377261784","heartbeat_front_addr":"172.21.15.123:6810/1377261784","state":["exists","up"]},{"osd":7,"uuid":"b419b04f-0c0c-495a-aa9b-25c334159515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":1702613957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":1702613957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":1702613957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":1702613957}]},"public_addr":"172.21.15.123:6812/1702613957","cluster_addr":"172.21.15.123:6813/1702613957","heartbeat_back_addr":"172.21.15.123:6815/1702613957","heartbeat_front_addr":"172.21.15.123:6814/1702613957","state":["exists","up"]},{"osd":8,"uuid":"1591e7bc-5b86-4401-803d-5325062ffb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6800","nonce":2551075225}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6801","nonce":2551075225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6803","nonce":2551075225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6802","nonce":2551075225}]},"public_addr":"172.21.15.169:6800/2551075225","cluster_addr":"172.21.15.169:6801/2551075225","heartbeat_back_addr":"172.21.15.169:6803/2551075225","heartbeat_front_addr":"172.21.15.169:6802/2551075225","state":["exists","up"]},{"osd":9,"uuid":"522d61df-8dc1-4915-83fd-f94d0d572c15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6804","nonce":2339555071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6805","nonce":2339555071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6807","nonce":2339555071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6806","nonce":2339555071}]},"public_addr":"172.21.15.169:6804/2339555071","cluster_addr":"172.21.15.169:6805/2339555071","heartbeat_back_addr":"172.21.15.169:6807/2339555071","heartbeat_front_addr":"172.21.15.169:6806/2339555071","state":["exists","up"]},{"osd":10,"uuid":"91f74dca-503a-45d1-9647-df4804d58515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6808","nonce":391306610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6809","nonce":391306610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6811","nonce":391306610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6810","nonce":391306610}]},"public_addr":"172.21.15.169:6808/391306610","cluster_addr":"172.21.15.169:6809/391306610","heartbeat_back_addr":"172.21.15.169:6811/391306610","heartbeat_front_addr":"172.21.15.169:6810/391306610","state":["exists","up"]},{"osd":11,"uuid":"788b4f47-eadf-4601-a5b4-51fb6f04fbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":93,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6812","nonce":3696812305}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6813","nonce":3696812305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6815","nonce":3696812305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6814","nonce":3696812305}]},"public_addr":"172.21.15.169:6812/3696812305","cluster_addr":"172.21.15.169:6813/3696812305","heartbeat_back_addr":"172.21.15.169:6815/3696812305","heartbeat_front_addr":"172.21.15.169:6814/3696812305","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:36.101199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:57:52.288612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:04.608555+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:01.758531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:30.400045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:17.255210+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:57:21.187814+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:09.352749+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:26.052191+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:26.726908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:57:40.571578+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:22.275639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/2474849789":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/253689008":"2024-02-13T15:48:17.261083+0000","172.21.15.121:6800/3531151725":"2024-02-13T15:47:47.232761+0000","172.21.15.121:0/1778611055":"2024-02-13T15:47:47.232761+0000","172.21.15.121:6800/28718860":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/445290372":"2024-02-13T15:47:47.232761+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T15:58:48.781 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:48 smithi121 bash[18489]: cluster 2024-02-12T15:58:47.439392+0000 mgr.y (mgr.14146) 389 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:48.781 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:48 smithi121 bash[18489]: audit 2024-02-12T15:58:48.155219+0000 mon.a (mon.0) 715 : audit [DBG] from='client.? v1:172.21.15.121:0/2135640395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:58:48.783 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2024-02-12T15:58:48.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-02-12T15:58:48.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:48 smithi169 bash[19604]: cluster 2024-02-12T15:58:47.439392+0000 mgr.y (mgr.14146) 389 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:48.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:48 smithi169 bash[19604]: audit 2024-02-12T15:58:48.155219+0000 mon.a (mon.0) 715 : audit [DBG] from='client.? v1:172.21.15.121:0/2135640395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:58:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:48 smithi123 bash[19119]: cluster 2024-02-12T15:58:47.439392+0000 mgr.y (mgr.14146) 389 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:48 smithi123 bash[19119]: audit 2024-02-12T15:58:48.155219+0000 mon.a (mon.0) 715 : audit [DBG] from='client.? v1:172.21.15.121:0/2135640395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:58:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:50 smithi169 bash[19604]: cluster 2024-02-12T15:58:49.206637+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:50 smithi169 bash[19604]: cluster 2024-02-12T15:58:49.206725+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:50 smithi169 bash[19604]: cluster 2024-02-12T15:58:49.440095+0000 mgr.y (mgr.14146) 390 : cluster [DBG] pgmap v404: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:50 smithi169 bash[19604]: cluster 2024-02-12T15:58:49.508962+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e99: 12 total, 12 up, 12 in 2024-02-12T15:58:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:50 smithi123 bash[19119]: cluster 2024-02-12T15:58:49.206637+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:50 smithi123 bash[19119]: cluster 2024-02-12T15:58:49.206725+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:50 smithi123 bash[19119]: cluster 2024-02-12T15:58:49.440095+0000 mgr.y (mgr.14146) 390 : cluster [DBG] pgmap v404: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:50 smithi123 bash[19119]: cluster 2024-02-12T15:58:49.508962+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e99: 12 total, 12 up, 12 in 2024-02-12T15:58:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:50 smithi121 bash[18489]: cluster 2024-02-12T15:58:49.206637+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:50 smithi121 bash[18489]: cluster 2024-02-12T15:58:49.206725+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:50 smithi121 bash[18489]: cluster 2024-02-12T15:58:49.440095+0000 mgr.y (mgr.14146) 390 : cluster [DBG] pgmap v404: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:50 smithi121 bash[18489]: cluster 2024-02-12T15:58:49.508962+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e99: 12 total, 12 up, 12 in 2024-02-12T15:58:52.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:52 smithi169 bash[19604]: cluster 2024-02-12T15:58:51.440693+0000 mgr.y (mgr.14146) 391 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:52.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:52 smithi123 bash[19119]: cluster 2024-02-12T15:58:51.440693+0000 mgr.y (mgr.14146) 391 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:52 smithi121 bash[18489]: cluster 2024-02-12T15:58:51.440693+0000 mgr.y (mgr.14146) 391 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:58:54.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:54 smithi121 bash[18489]: cluster 2024-02-12T15:58:53.441309+0000 mgr.y (mgr.14146) 392 : cluster [DBG] pgmap v407: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:54 smithi169 bash[19604]: cluster 2024-02-12T15:58:53.441309+0000 mgr.y (mgr.14146) 392 : cluster [DBG] pgmap v407: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:54.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:54 smithi123 bash[19119]: cluster 2024-02-12T15:58:53.441309+0000 mgr.y (mgr.14146) 392 : cluster [DBG] pgmap v407: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:56.544 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-02-12T15:58:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:56 smithi169 bash[19604]: cluster 2024-02-12T15:58:54.882737+0000 osd.10 (osd.10) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:56 smithi169 bash[19604]: cluster 2024-02-12T15:58:54.882827+0000 osd.10 (osd.10) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:56 smithi169 bash[19604]: cluster 2024-02-12T15:58:55.441951+0000 mgr.y (mgr.14146) 393 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:56 smithi169 bash[19604]: cluster 2024-02-12T15:58:55.526031+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e100: 12 total, 12 up, 12 in 2024-02-12T15:58:56.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:56 smithi123 bash[19119]: cluster 2024-02-12T15:58:54.882737+0000 osd.10 (osd.10) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:56.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:56 smithi123 bash[19119]: cluster 2024-02-12T15:58:54.882827+0000 osd.10 (osd.10) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:56.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:56 smithi123 bash[19119]: cluster 2024-02-12T15:58:55.441951+0000 mgr.y (mgr.14146) 393 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:56.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:56 smithi123 bash[19119]: cluster 2024-02-12T15:58:55.526031+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e100: 12 total, 12 up, 12 in 2024-02-12T15:58:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:56 smithi121 bash[18489]: cluster 2024-02-12T15:58:54.882737+0000 osd.10 (osd.10) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:58:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:56 smithi121 bash[18489]: cluster 2024-02-12T15:58:54.882827+0000 osd.10 (osd.10) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:58:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:56 smithi121 bash[18489]: cluster 2024-02-12T15:58:55.441951+0000 mgr.y (mgr.14146) 393 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:56 smithi121 bash[18489]: cluster 2024-02-12T15:58:55.526031+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e100: 12 total, 12 up, 12 in 2024-02-12T15:58:57.146 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-12T15:58:57.146 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-12T15:58:57.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph mgr dump --format=json 2024-02-12T15:58:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:57 smithi169 bash[19604]: audit 2024-02-12T15:58:56.538003+0000 mon.a (mon.0) 718 : audit [DBG] from='client.? v1:172.21.15.121:0/1013519393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-12T15:58:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:57 smithi123 bash[19119]: audit 2024-02-12T15:58:56.538003+0000 mon.a (mon.0) 718 : audit [DBG] from='client.? v1:172.21.15.121:0/1013519393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-12T15:58:57.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:57 smithi121 bash[18489]: audit 2024-02-12T15:58:56.538003+0000 mon.a (mon.0) 718 : audit [DBG] from='client.? v1:172.21.15.121:0/1013519393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-12T15:58:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:58:58 smithi169 bash[19604]: cluster 2024-02-12T15:58:57.442574+0000 mgr.y (mgr.14146) 394 : cluster [DBG] pgmap v410: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:58.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:58:58 smithi123 bash[19119]: cluster 2024-02-12T15:58:57.442574+0000 mgr.y (mgr.14146) 394 : cluster [DBG] pgmap v410: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:58:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:58:58 smithi121 bash[18489]: cluster 2024-02-12T15:58:57.442574+0000 mgr.y (mgr.14146) 394 : cluster [DBG] pgmap v410: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:00 smithi169 bash[19604]: cluster 2024-02-12T15:58:59.443278+0000 mgr.y (mgr.14146) 395 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:00 smithi123 bash[19119]: cluster 2024-02-12T15:58:59.443278+0000 mgr.y (mgr.14146) 395 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:00.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:00 smithi121 bash[18489]: cluster 2024-02-12T15:58:59.443278+0000 mgr.y (mgr.14146) 395 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:01.955 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:59:02.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:02 smithi121 bash[18489]: cluster 2024-02-12T15:59:01.443780+0000 mgr.y (mgr.14146) 396 : cluster [DBG] pgmap v412: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:02.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:02 smithi121 bash[18489]: cluster 2024-02-12T15:59:01.537161+0000 osd.1 (osd.1) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:02.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:02 smithi121 bash[18489]: cluster 2024-02-12T15:59:01.537255+0000 osd.1 (osd.1) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:02.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:02 smithi121 bash[18489]: cluster 2024-02-12T15:59:01.594315+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e101: 12 total, 12 up, 12 in 2024-02-12T15:59:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:02 smithi123 bash[19119]: cluster 2024-02-12T15:59:01.443780+0000 mgr.y (mgr.14146) 396 : cluster [DBG] pgmap v412: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:02 smithi123 bash[19119]: cluster 2024-02-12T15:59:01.537161+0000 osd.1 (osd.1) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:02 smithi123 bash[19119]: cluster 2024-02-12T15:59:01.537255+0000 osd.1 (osd.1) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:02 smithi123 bash[19119]: cluster 2024-02-12T15:59:01.594315+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e101: 12 total, 12 up, 12 in 2024-02-12T15:59:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:02 smithi169 bash[19604]: cluster 2024-02-12T15:59:01.443780+0000 mgr.y (mgr.14146) 396 : cluster [DBG] pgmap v412: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:02 smithi169 bash[19604]: cluster 2024-02-12T15:59:01.537161+0000 osd.1 (osd.1) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:02 smithi169 bash[19604]: cluster 2024-02-12T15:59:01.537255+0000 osd.1 (osd.1) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:02 smithi169 bash[19604]: cluster 2024-02-12T15:59:01.594315+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e101: 12 total, 12 up, 12 in 2024-02-12T15:59:04.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:04 smithi123 bash[19119]: cluster 2024-02-12T15:59:03.444372+0000 mgr.y (mgr.14146) 397 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:04 smithi121 bash[18489]: cluster 2024-02-12T15:59:03.444372+0000 mgr.y (mgr.14146) 397 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:04.999 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T15:59:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:04 smithi169 bash[19604]: cluster 2024-02-12T15:59:03.444372+0000 mgr.y (mgr.14146) 397 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:05.660 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":14,"active_gid":14146,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":890731717}]},"active_addr":"172.21.15.121:6800/890731717","active_change":"2024-02-12T15:48:17.261290+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":24119,"name":"x","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.121:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":3137298256}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":3383257331}]}]} 2024-02-12T15:59:05.664 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-12T15:59:05.664 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-12T15:59:05.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T15:59:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:05 smithi123 bash[19119]: audit 2024-02-12T15:59:04.987395+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? v1:172.21.15.121:0/3661760412' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-12T15:59:05.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:05 smithi121 bash[18489]: audit 2024-02-12T15:59:04.987395+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? v1:172.21.15.121:0/3661760412' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-12T15:59:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:05 smithi169 bash[19604]: audit 2024-02-12T15:59:04.987395+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? v1:172.21.15.121:0/3661760412' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-12T15:59:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:06 smithi123 bash[19119]: cluster 2024-02-12T15:59:05.445011+0000 mgr.y (mgr.14146) 398 : cluster [DBG] pgmap v415: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:06.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:06 smithi121 bash[18489]: cluster 2024-02-12T15:59:05.445011+0000 mgr.y (mgr.14146) 398 : cluster [DBG] pgmap v415: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:06 smithi169 bash[19604]: cluster 2024-02-12T15:59:05.445011+0000 mgr.y (mgr.14146) 398 : cluster [DBG] pgmap v415: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:08.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:08 smithi123 bash[19119]: cluster 2024-02-12T15:59:07.445705+0000 mgr.y (mgr.14146) 399 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:08.924 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:08 smithi121 bash[18489]: cluster 2024-02-12T15:59:07.445705+0000 mgr.y (mgr.14146) 399 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:08 smithi169 bash[19604]: cluster 2024-02-12T15:59:07.445705+0000 mgr.y (mgr.14146) 399 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:10.469 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:59:10.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:10 smithi123 bash[19119]: cluster 2024-02-12T15:59:09.446368+0000 mgr.y (mgr.14146) 400 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:10.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:10 smithi121 bash[18489]: cluster 2024-02-12T15:59:09.446368+0000 mgr.y (mgr.14146) 400 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:10 smithi169 bash[19604]: cluster 2024-02-12T15:59:09.446368+0000 mgr.y (mgr.14146) 400 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:12 smithi123 bash[19119]: cluster 2024-02-12T15:59:11.447151+0000 mgr.y (mgr.14146) 401 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:12.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:12 smithi121 bash[18489]: cluster 2024-02-12T15:59:11.447151+0000 mgr.y (mgr.14146) 401 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:12 smithi169 bash[19604]: cluster 2024-02-12T15:59:11.447151+0000 mgr.y (mgr.14146) 401 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T15:59:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":101,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","created":"2024-02-12T15:47:18.273813+0000","modified":"2024-02-12T15:59:01.588623+0000","last_up_change":"2024-02-12T15:58:23.801839+0000","last_in_change":"2024-02-12T15:57:58.358258+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T15:51:35.886878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"3a3badac-97ea-4ee1-ad82-1b67a4fe94f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1815232143}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1815232143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":1815232143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1815232143}]},"public_addr":"172.21.15.121:6801/1815232143","cluster_addr":"172.21.15.121:6802/1815232143","heartbeat_back_addr":"172.21.15.121:6804/1815232143","heartbeat_front_addr":"172.21.15.121:6803/1815232143","state":["exists","up"]},{"osd":1,"uuid":"c6140309-5703-4538-9739-276fbbdaa880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2968433670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2968433670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":2968433670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2968433670}]},"public_addr":"172.21.15.121:6805/2968433670","cluster_addr":"172.21.15.121:6806/2968433670","heartbeat_back_addr":"172.21.15.121:6808/2968433670","heartbeat_front_addr":"172.21.15.121:6807/2968433670","state":["exists","up"]},{"osd":2,"uuid":"7c81e1a4-cae0-48d4-98ab-0becb6b31170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":2158187852}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":2158187852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":2158187852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":2158187852}]},"public_addr":"172.21.15.121:6809/2158187852","cluster_addr":"172.21.15.121:6810/2158187852","heartbeat_back_addr":"172.21.15.121:6812/2158187852","heartbeat_front_addr":"172.21.15.121:6811/2158187852","state":["exists","up"]},{"osd":3,"uuid":"5f824526-596d-42e1-8b7e-cd8676ded6dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":1415189366}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":1415189366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6816","nonce":1415189366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":1415189366}]},"public_addr":"172.21.15.121:6813/1415189366","cluster_addr":"172.21.15.121:6814/1415189366","heartbeat_back_addr":"172.21.15.121:6816/1415189366","heartbeat_front_addr":"172.21.15.121:6815/1415189366","state":["exists","up"]},{"osd":4,"uuid":"14c2f878-99ce-4b51-8d9c-131c1d883b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6800","nonce":3447700123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":3447700123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":3447700123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":3447700123}]},"public_addr":"172.21.15.123:6800/3447700123","cluster_addr":"172.21.15.123:6801/3447700123","heartbeat_back_addr":"172.21.15.123:6803/3447700123","heartbeat_front_addr":"172.21.15.123:6802/3447700123","state":["exists","up"]},{"osd":5,"uuid":"75ecf26a-d7ed-46e7-a40e-6f719e2c3602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":961317306}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":961317306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":961317306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":961317306}]},"public_addr":"172.21.15.123:6804/961317306","cluster_addr":"172.21.15.123:6805/961317306","heartbeat_back_addr":"172.21.15.123:6807/961317306","heartbeat_front_addr":"172.21.15.123:6806/961317306","state":["exists","up"]},{"osd":6,"uuid":"71e715e6-eac1-4386-927e-51df0cc419c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":1377261784}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":1377261784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":1377261784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":1377261784}]},"public_addr":"172.21.15.123:6808/1377261784","cluster_addr":"172.21.15.123:6809/1377261784","heartbeat_back_addr":"172.21.15.123:6811/1377261784","heartbeat_front_addr":"172.21.15.123:6810/1377261784","state":["exists","up"]},{"osd":7,"uuid":"b419b04f-0c0c-495a-aa9b-25c334159515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":1702613957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":1702613957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":1702613957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":1702613957}]},"public_addr":"172.21.15.123:6812/1702613957","cluster_addr":"172.21.15.123:6813/1702613957","heartbeat_back_addr":"172.21.15.123:6815/1702613957","heartbeat_front_addr":"172.21.15.123:6814/1702613957","state":["exists","up"]},{"osd":8,"uuid":"1591e7bc-5b86-4401-803d-5325062ffb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6800","nonce":2551075225}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6801","nonce":2551075225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6803","nonce":2551075225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6802","nonce":2551075225}]},"public_addr":"172.21.15.169:6800/2551075225","cluster_addr":"172.21.15.169:6801/2551075225","heartbeat_back_addr":"172.21.15.169:6803/2551075225","heartbeat_front_addr":"172.21.15.169:6802/2551075225","state":["exists","up"]},{"osd":9,"uuid":"522d61df-8dc1-4915-83fd-f94d0d572c15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6804","nonce":2339555071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6805","nonce":2339555071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6807","nonce":2339555071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6806","nonce":2339555071}]},"public_addr":"172.21.15.169:6804/2339555071","cluster_addr":"172.21.15.169:6805/2339555071","heartbeat_back_addr":"172.21.15.169:6807/2339555071","heartbeat_front_addr":"172.21.15.169:6806/2339555071","state":["exists","up"]},{"osd":10,"uuid":"91f74dca-503a-45d1-9647-df4804d58515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6808","nonce":391306610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6809","nonce":391306610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6811","nonce":391306610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6810","nonce":391306610}]},"public_addr":"172.21.15.169:6808/391306610","cluster_addr":"172.21.15.169:6809/391306610","heartbeat_back_addr":"172.21.15.169:6811/391306610","heartbeat_front_addr":"172.21.15.169:6810/391306610","state":["exists","up"]},{"osd":11,"uuid":"788b4f47-eadf-4601-a5b4-51fb6f04fbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":93,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6812","nonce":3696812305}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6813","nonce":3696812305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6815","nonce":3696812305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6814","nonce":3696812305}]},"public_addr":"172.21.15.169:6812/3696812305","cluster_addr":"172.21.15.169:6813/3696812305","heartbeat_back_addr":"172.21.15.169:6815/3696812305","heartbeat_front_addr":"172.21.15.169:6814/3696812305","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:36.101199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:59:01.537259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:04.608555+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:01.758531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:30.400045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:17.255210+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:49.206729+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:09.352749+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:26.052191+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:26.726908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:54.882832+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:22.275639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/2474849789":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/253689008":"2024-02-13T15:48:17.261083+0000","172.21.15.121:6800/3531151725":"2024-02-13T15:47:47.232761+0000","172.21.15.121:0/1778611055":"2024-02-13T15:47:47.232761+0000","172.21.15.121:6800/28718860":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/445290372":"2024-02-13T15:47:47.232761+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T15:59:13.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:13 smithi123 bash[19119]: audit 2024-02-12T15:59:13.420164+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.121:0/2996628143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:59:13.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:13 smithi121 bash[18489]: audit 2024-02-12T15:59:13.420164+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.121:0/2996628143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:59:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:13 smithi169 bash[19604]: audit 2024-02-12T15:59:13.420164+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.121:0/2996628143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:59:14.095 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-12T15:59:14.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T15:59:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:14 smithi123 bash[19119]: cluster 2024-02-12T15:59:13.447762+0000 mgr.y (mgr.14146) 402 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:14.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:14 smithi121 bash[18489]: cluster 2024-02-12T15:59:13.447762+0000 mgr.y (mgr.14146) 402 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:14 smithi169 bash[19604]: cluster 2024-02-12T15:59:13.447762+0000 mgr.y (mgr.14146) 402 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:16.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:16 smithi123 bash[19119]: cluster 2024-02-12T15:59:15.448424+0000 mgr.y (mgr.14146) 403 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:16.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:16 smithi121 bash[18489]: cluster 2024-02-12T15:59:15.448424+0000 mgr.y (mgr.14146) 403 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:16 smithi169 bash[19604]: cluster 2024-02-12T15:59:15.448424+0000 mgr.y (mgr.14146) 403 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:18 smithi169 bash[19604]: cluster 2024-02-12T15:59:17.449137+0000 mgr.y (mgr.14146) 404 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:18 smithi169 bash[19604]: cluster 2024-02-12T15:59:17.843189+0000 osd.7 (osd.7) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:18 smithi169 bash[19604]: cluster 2024-02-12T15:59:17.843277+0000 osd.7 (osd.7) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:18.880 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:59:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:18 smithi123 bash[19119]: cluster 2024-02-12T15:59:17.449137+0000 mgr.y (mgr.14146) 404 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:18 smithi123 bash[19119]: cluster 2024-02-12T15:59:17.843189+0000 osd.7 (osd.7) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:18 smithi123 bash[19119]: cluster 2024-02-12T15:59:17.843277+0000 osd.7 (osd.7) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:18 smithi121 bash[18489]: cluster 2024-02-12T15:59:17.449137+0000 mgr.y (mgr.14146) 404 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:18 smithi121 bash[18489]: cluster 2024-02-12T15:59:17.843189+0000 osd.7 (osd.7) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:18 smithi121 bash[18489]: cluster 2024-02-12T15:59:17.843277+0000 osd.7 (osd.7) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:19 smithi169 bash[19604]: cluster 2024-02-12T15:59:18.517700+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e102: 12 total, 12 up, 12 in 2024-02-12T15:59:19.845 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:19 smithi121 bash[18489]: cluster 2024-02-12T15:59:18.517700+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e102: 12 total, 12 up, 12 in 2024-02-12T15:59:19.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:19 smithi123 bash[19119]: cluster 2024-02-12T15:59:18.517700+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e102: 12 total, 12 up, 12 in 2024-02-12T15:59:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:20 smithi169 bash[19604]: cluster 2024-02-12T15:59:19.449896+0000 mgr.y (mgr.14146) 405 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:20 smithi123 bash[19119]: cluster 2024-02-12T15:59:19.449896+0000 mgr.y (mgr.14146) 405 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:20 smithi121 bash[18489]: cluster 2024-02-12T15:59:19.449896+0000 mgr.y (mgr.14146) 405 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:21.872 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T15:59:21.872 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":103,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","created":"2024-02-12T15:47:18.273813+0000","modified":"2024-02-12T15:59:21.523959+0000","last_up_change":"2024-02-12T15:58:23.801839+0000","last_in_change":"2024-02-12T15:57:58.358258+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T15:51:35.886878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"3a3badac-97ea-4ee1-ad82-1b67a4fe94f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1815232143}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1815232143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":1815232143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1815232143}]},"public_addr":"172.21.15.121:6801/1815232143","cluster_addr":"172.21.15.121:6802/1815232143","heartbeat_back_addr":"172.21.15.121:6804/1815232143","heartbeat_front_addr":"172.21.15.121:6803/1815232143","state":["exists","up"]},{"osd":1,"uuid":"c6140309-5703-4538-9739-276fbbdaa880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2968433670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2968433670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":2968433670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2968433670}]},"public_addr":"172.21.15.121:6805/2968433670","cluster_addr":"172.21.15.121:6806/2968433670","heartbeat_back_addr":"172.21.15.121:6808/2968433670","heartbeat_front_addr":"172.21.15.121:6807/2968433670","state":["exists","up"]},{"osd":2,"uuid":"7c81e1a4-cae0-48d4-98ab-0becb6b31170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":2158187852}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":2158187852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":2158187852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":2158187852}]},"public_addr":"172.21.15.121:6809/2158187852","cluster_addr":"172.21.15.121:6810/2158187852","heartbeat_back_addr":"172.21.15.121:6812/2158187852","heartbeat_front_addr":"172.21.15.121:6811/2158187852","state":["exists","up"]},{"osd":3,"uuid":"5f824526-596d-42e1-8b7e-cd8676ded6dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":1415189366}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":1415189366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6816","nonce":1415189366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":1415189366}]},"public_addr":"172.21.15.121:6813/1415189366","cluster_addr":"172.21.15.121:6814/1415189366","heartbeat_back_addr":"172.21.15.121:6816/1415189366","heartbeat_front_addr":"172.21.15.121:6815/1415189366","state":["exists","up"]},{"osd":4,"uuid":"14c2f878-99ce-4b51-8d9c-131c1d883b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6800","nonce":3447700123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":3447700123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":3447700123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":3447700123}]},"public_addr":"172.21.15.123:6800/3447700123","cluster_addr":"172.21.15.123:6801/3447700123","heartbeat_back_addr":"172.21.15.123:6803/3447700123","heartbeat_front_addr":"172.21.15.123:6802/3447700123","state":["exists","up"]},{"osd":5,"uuid":"75ecf26a-d7ed-46e7-a40e-6f719e2c3602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":961317306}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":961317306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":961317306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":961317306}]},"public_addr":"172.21.15.123:6804/961317306","cluster_addr":"172.21.15.123:6805/961317306","heartbeat_back_addr":"172.21.15.123:6807/961317306","heartbeat_front_addr":"172.21.15.123:6806/961317306","state":["exists","up"]},{"osd":6,"uuid":"71e715e6-eac1-4386-927e-51df0cc419c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":1377261784}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":1377261784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":1377261784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":1377261784}]},"public_addr":"172.21.15.123:6808/1377261784","cluster_addr":"172.21.15.123:6809/1377261784","heartbeat_back_addr":"172.21.15.123:6811/1377261784","heartbeat_front_addr":"172.21.15.123:6810/1377261784","state":["exists","up"]},{"osd":7,"uuid":"b419b04f-0c0c-495a-aa9b-25c334159515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":1702613957}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":1702613957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":1702613957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":1702613957}]},"public_addr":"172.21.15.123:6812/1702613957","cluster_addr":"172.21.15.123:6813/1702613957","heartbeat_back_addr":"172.21.15.123:6815/1702613957","heartbeat_front_addr":"172.21.15.123:6814/1702613957","state":["exists","up"]},{"osd":8,"uuid":"1591e7bc-5b86-4401-803d-5325062ffb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6800","nonce":2551075225}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6801","nonce":2551075225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6803","nonce":2551075225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6802","nonce":2551075225}]},"public_addr":"172.21.15.169:6800/2551075225","cluster_addr":"172.21.15.169:6801/2551075225","heartbeat_back_addr":"172.21.15.169:6803/2551075225","heartbeat_front_addr":"172.21.15.169:6802/2551075225","state":["exists","up"]},{"osd":9,"uuid":"522d61df-8dc1-4915-83fd-f94d0d572c15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6804","nonce":2339555071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6805","nonce":2339555071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6807","nonce":2339555071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6806","nonce":2339555071}]},"public_addr":"172.21.15.169:6804/2339555071","cluster_addr":"172.21.15.169:6805/2339555071","heartbeat_back_addr":"172.21.15.169:6807/2339555071","heartbeat_front_addr":"172.21.15.169:6806/2339555071","state":["exists","up"]},{"osd":10,"uuid":"91f74dca-503a-45d1-9647-df4804d58515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6808","nonce":391306610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6809","nonce":391306610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6811","nonce":391306610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6810","nonce":391306610}]},"public_addr":"172.21.15.169:6808/391306610","cluster_addr":"172.21.15.169:6809/391306610","heartbeat_back_addr":"172.21.15.169:6811/391306610","heartbeat_front_addr":"172.21.15.169:6810/391306610","state":["exists","up"]},{"osd":11,"uuid":"788b4f47-eadf-4601-a5b4-51fb6f04fbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":93,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6812","nonce":3696812305}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6813","nonce":3696812305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6815","nonce":3696812305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6814","nonce":3696812305}]},"public_addr":"172.21.15.169:6812/3696812305","cluster_addr":"172.21.15.169:6813/3696812305","heartbeat_back_addr":"172.21.15.169:6815/3696812305","heartbeat_front_addr":"172.21.15.169:6814/3696812305","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:36.101199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:59:01.537259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:04.608555+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:59:20.862245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:30.400045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:17.255210+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:49.206729+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:59:17.843282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:26.052191+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:26.726908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:54.882832+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T15:58:22.275639+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/2474849789":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/253689008":"2024-02-13T15:48:17.261083+0000","172.21.15.121:6800/3531151725":"2024-02-13T15:47:47.232761+0000","172.21.15.121:0/1778611055":"2024-02-13T15:47:47.232761+0000","172.21.15.121:6800/28718860":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/445290372":"2024-02-13T15:47:47.232761+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T15:59:22.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-12T15:59:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:22 smithi169 bash[19604]: cluster 2024-02-12T15:59:20.862148+0000 osd.3 (osd.3) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:22 smithi169 bash[19604]: cluster 2024-02-12T15:59:20.862241+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:22 smithi169 bash[19604]: cluster 2024-02-12T15:59:21.450533+0000 mgr.y (mgr.14146) 406 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:22 smithi169 bash[19604]: cluster 2024-02-12T15:59:21.539572+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e103: 12 total, 12 up, 12 in 2024-02-12T15:59:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:22 smithi169 bash[19604]: audit 2024-02-12T15:59:21.864078+0000 mon.a (mon.0) 724 : audit [DBG] from='client.? v1:172.21.15.121:0/1257068231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:59:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:22 smithi123 bash[19119]: cluster 2024-02-12T15:59:20.862148+0000 osd.3 (osd.3) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:22 smithi123 bash[19119]: cluster 2024-02-12T15:59:20.862241+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:22 smithi123 bash[19119]: cluster 2024-02-12T15:59:21.450533+0000 mgr.y (mgr.14146) 406 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:22 smithi123 bash[19119]: cluster 2024-02-12T15:59:21.539572+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e103: 12 total, 12 up, 12 in 2024-02-12T15:59:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:22 smithi123 bash[19119]: audit 2024-02-12T15:59:21.864078+0000 mon.a (mon.0) 724 : audit [DBG] from='client.? v1:172.21.15.121:0/1257068231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:59:22.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:22 smithi121 bash[18489]: cluster 2024-02-12T15:59:20.862148+0000 osd.3 (osd.3) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:22.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:22 smithi121 bash[18489]: cluster 2024-02-12T15:59:20.862241+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:22.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:22 smithi121 bash[18489]: cluster 2024-02-12T15:59:21.450533+0000 mgr.y (mgr.14146) 406 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:22.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:22 smithi121 bash[18489]: cluster 2024-02-12T15:59:21.539572+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e103: 12 total, 12 up, 12 in 2024-02-12T15:59:22.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:22 smithi121 bash[18489]: audit 2024-02-12T15:59:21.864078+0000 mon.a (mon.0) 724 : audit [DBG] from='client.? v1:172.21.15.121:0/1257068231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T15:59:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:24 smithi123 bash[19119]: cluster 2024-02-12T15:59:23.451166+0000 mgr.y (mgr.14146) 407 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:24.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:24 smithi121 bash[18489]: cluster 2024-02-12T15:59:23.451166+0000 mgr.y (mgr.14146) 407 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:24 smithi169 bash[19604]: cluster 2024-02-12T15:59:23.451166+0000 mgr.y (mgr.14146) 407 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:25.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:25 smithi123 bash[19119]: cluster 2024-02-12T15:59:24.847568+0000 osd.7 (osd.7) 15 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:59:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:25 smithi123 bash[19119]: cluster 2024-02-12T15:59:24.851552+0000 osd.7 (osd.7) 16 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:59:25.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:25 smithi121 bash[18489]: cluster 2024-02-12T15:59:24.847568+0000 osd.7 (osd.7) 15 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:59:25.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:25 smithi121 bash[18489]: cluster 2024-02-12T15:59:24.851552+0000 osd.7 (osd.7) 16 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:59:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:25 smithi169 bash[19604]: cluster 2024-02-12T15:59:24.847568+0000 osd.7 (osd.7) 15 : cluster [DBG] 1.0 scrub starts 2024-02-12T15:59:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:25 smithi169 bash[19604]: cluster 2024-02-12T15:59:24.851552+0000 osd.7 (osd.7) 16 : cluster [DBG] 1.0 scrub ok 2024-02-12T15:59:26.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:26 smithi123 bash[19119]: cluster 2024-02-12T15:59:25.451813+0000 mgr.y (mgr.14146) 408 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:26.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:26 smithi121 bash[18489]: cluster 2024-02-12T15:59:25.451813+0000 mgr.y (mgr.14146) 408 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:26 smithi169 bash[19604]: cluster 2024-02-12T15:59:25.451813+0000 mgr.y (mgr.14146) 408 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:27.253 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:59:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:28 smithi123 bash[19119]: cluster 2024-02-12T15:59:26.538528+0000 osd.11 (osd.11) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:28 smithi123 bash[19119]: cluster 2024-02-12T15:59:26.538644+0000 osd.11 (osd.11) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:28 smithi123 bash[19119]: cluster 2024-02-12T15:59:27.103578+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e104: 12 total, 12 up, 12 in 2024-02-12T15:59:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:28 smithi123 bash[19119]: cluster 2024-02-12T15:59:27.439516+0000 osd.2 (osd.2) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:28 smithi123 bash[19119]: cluster 2024-02-12T15:59:27.439624+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:28.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:28 smithi123 bash[19119]: cluster 2024-02-12T15:59:27.452393+0000 mgr.y (mgr.14146) 409 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:28.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:28 smithi121 bash[18489]: cluster 2024-02-12T15:59:26.538528+0000 osd.11 (osd.11) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:28.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:28 smithi121 bash[18489]: cluster 2024-02-12T15:59:26.538644+0000 osd.11 (osd.11) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:28.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:28 smithi121 bash[18489]: cluster 2024-02-12T15:59:27.103578+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e104: 12 total, 12 up, 12 in 2024-02-12T15:59:28.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:28 smithi121 bash[18489]: cluster 2024-02-12T15:59:27.439516+0000 osd.2 (osd.2) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:28.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:28 smithi121 bash[18489]: cluster 2024-02-12T15:59:27.439624+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:28.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:28 smithi121 bash[18489]: cluster 2024-02-12T15:59:27.452393+0000 mgr.y (mgr.14146) 409 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:28 smithi169 bash[19604]: cluster 2024-02-12T15:59:26.538528+0000 osd.11 (osd.11) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:28 smithi169 bash[19604]: cluster 2024-02-12T15:59:26.538644+0000 osd.11 (osd.11) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:28 smithi169 bash[19604]: cluster 2024-02-12T15:59:27.103578+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e104: 12 total, 12 up, 12 in 2024-02-12T15:59:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:28 smithi169 bash[19604]: cluster 2024-02-12T15:59:27.439516+0000 osd.2 (osd.2) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:28.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:28 smithi169 bash[19604]: cluster 2024-02-12T15:59:27.439624+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:28.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:28 smithi169 bash[19604]: cluster 2024-02-12T15:59:27.452393+0000 mgr.y (mgr.14146) 409 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:29.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:29 smithi123 bash[19119]: cluster 2024-02-12T15:59:28.113560+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e105: 12 total, 12 up, 12 in 2024-02-12T15:59:29.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:29 smithi121 bash[18489]: cluster 2024-02-12T15:59:28.113560+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e105: 12 total, 12 up, 12 in 2024-02-12T15:59:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:29 smithi169 bash[19604]: cluster 2024-02-12T15:59:28.113560+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e105: 12 total, 12 up, 12 in 2024-02-12T15:59:30.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:30 smithi121 bash[18489]: cluster 2024-02-12T15:59:29.453095+0000 mgr.y (mgr.14146) 410 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:30.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:30 smithi123 bash[19119]: cluster 2024-02-12T15:59:29.453095+0000 mgr.y (mgr.14146) 410 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:30 smithi169 bash[19604]: cluster 2024-02-12T15:59:29.453095+0000 mgr.y (mgr.14146) 410 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:30.829 INFO:teuthology.orchestra.run.smithi121.stdout:34359738475 2024-02-12T15:59:30.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-12T15:59:32.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:32 smithi169 bash[19604]: cluster 2024-02-12T15:59:31.453689+0000 mgr.y (mgr.14146) 411 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:32.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:32 smithi123 bash[19119]: cluster 2024-02-12T15:59:31.453689+0000 mgr.y (mgr.14146) 411 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:32.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:32 smithi121 bash[18489]: cluster 2024-02-12T15:59:31.453689+0000 mgr.y (mgr.14146) 411 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:33 smithi169 bash[19604]: cluster 2024-02-12T15:59:32.228351+0000 osd.8 (osd.8) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:33 smithi169 bash[19604]: cluster 2024-02-12T15:59:32.228444+0000 osd.8 (osd.8) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:33 smithi169 bash[19604]: cluster 2024-02-12T15:59:32.516786+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e106: 12 total, 12 up, 12 in 2024-02-12T15:59:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:33 smithi169 bash[19604]: cluster 2024-02-12T15:59:32.666906+0000 osd.9 (osd.9) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:33 smithi169 bash[19604]: cluster 2024-02-12T15:59:32.667017+0000 osd.9 (osd.9) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:33 smithi169 bash[19604]: audit 2024-02-12T15:59:33.023912+0000 mon.b (mon.2) 93 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:59:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:33 smithi169 bash[19604]: audit 2024-02-12T15:59:33.337258+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:59:33.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:33 smithi123 bash[19119]: cluster 2024-02-12T15:59:32.228351+0000 osd.8 (osd.8) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:33 smithi123 bash[19119]: cluster 2024-02-12T15:59:32.228444+0000 osd.8 (osd.8) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:33 smithi123 bash[19119]: cluster 2024-02-12T15:59:32.516786+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e106: 12 total, 12 up, 12 in 2024-02-12T15:59:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:33 smithi123 bash[19119]: cluster 2024-02-12T15:59:32.666906+0000 osd.9 (osd.9) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:33 smithi123 bash[19119]: cluster 2024-02-12T15:59:32.667017+0000 osd.9 (osd.9) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:33 smithi123 bash[19119]: audit 2024-02-12T15:59:33.023912+0000 mon.b (mon.2) 93 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:59:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:33 smithi123 bash[19119]: audit 2024-02-12T15:59:33.337258+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:59:33.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:33 smithi121 bash[18489]: cluster 2024-02-12T15:59:32.228351+0000 osd.8 (osd.8) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:33.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:33 smithi121 bash[18489]: cluster 2024-02-12T15:59:32.228444+0000 osd.8 (osd.8) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:33.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:33 smithi121 bash[18489]: cluster 2024-02-12T15:59:32.516786+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e106: 12 total, 12 up, 12 in 2024-02-12T15:59:33.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:33 smithi121 bash[18489]: cluster 2024-02-12T15:59:32.666906+0000 osd.9 (osd.9) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:33.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:33 smithi121 bash[18489]: cluster 2024-02-12T15:59:32.667017+0000 osd.9 (osd.9) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:33.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:33 smithi121 bash[18489]: audit 2024-02-12T15:59:33.023912+0000 mon.b (mon.2) 93 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T15:59:33.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:33 smithi121 bash[18489]: audit 2024-02-12T15:59:33.337258+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:59:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:34 smithi169 bash[19604]: cluster 2024-02-12T15:59:33.454225+0000 mgr.y (mgr.14146) 412 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:34 smithi169 bash[19604]: cluster 2024-02-12T15:59:33.533080+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e107: 12 total, 12 up, 12 in 2024-02-12T15:59:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:34 smithi169 bash[19604]: audit 2024-02-12T15:59:33.653129+0000 mon.b (mon.2) 94 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:59:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:34 smithi169 bash[19604]: audit 2024-02-12T15:59:33.654936+0000 mon.b (mon.2) 95 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:59:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:34 smithi169 bash[19604]: audit 2024-02-12T15:59:33.659915+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:59:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:34 smithi123 bash[19119]: cluster 2024-02-12T15:59:33.454225+0000 mgr.y (mgr.14146) 412 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:34 smithi123 bash[19119]: cluster 2024-02-12T15:59:33.533080+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e107: 12 total, 12 up, 12 in 2024-02-12T15:59:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:34 smithi123 bash[19119]: audit 2024-02-12T15:59:33.653129+0000 mon.b (mon.2) 94 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:59:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:34 smithi123 bash[19119]: audit 2024-02-12T15:59:33.654936+0000 mon.b (mon.2) 95 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:59:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:34 smithi123 bash[19119]: audit 2024-02-12T15:59:33.659915+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:59:34.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:34 smithi121 bash[18489]: cluster 2024-02-12T15:59:33.454225+0000 mgr.y (mgr.14146) 412 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:34.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:34 smithi121 bash[18489]: cluster 2024-02-12T15:59:33.533080+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e107: 12 total, 12 up, 12 in 2024-02-12T15:59:34.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:34 smithi121 bash[18489]: audit 2024-02-12T15:59:33.653129+0000 mon.b (mon.2) 94 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T15:59:34.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:34 smithi121 bash[18489]: audit 2024-02-12T15:59:33.654936+0000 mon.b (mon.2) 95 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T15:59:34.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:34 smithi121 bash[18489]: audit 2024-02-12T15:59:33.659915+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T15:59:35.625 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:59:36.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:36 smithi123 bash[19119]: cluster 2024-02-12T15:59:35.454867+0000 mgr.y (mgr.14146) 413 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:36 smithi121 bash[18489]: cluster 2024-02-12T15:59:35.454867+0000 mgr.y (mgr.14146) 413 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:36 smithi169 bash[19604]: cluster 2024-02-12T15:59:35.454867+0000 mgr.y (mgr.14146) 413 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:38 smithi123 bash[19119]: cluster 2024-02-12T15:59:36.671910+0000 osd.4 (osd.4) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:38 smithi123 bash[19119]: cluster 2024-02-12T15:59:36.672003+0000 osd.4 (osd.4) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:38 smithi123 bash[19119]: cluster 2024-02-12T15:59:37.119673+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e108: 12 total, 12 up, 12 in 2024-02-12T15:59:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:38 smithi123 bash[19119]: cluster 2024-02-12T15:59:37.455487+0000 mgr.y (mgr.14146) 414 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:38 smithi121 bash[18489]: cluster 2024-02-12T15:59:36.671910+0000 osd.4 (osd.4) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:38.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:38 smithi121 bash[18489]: cluster 2024-02-12T15:59:36.672003+0000 osd.4 (osd.4) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:38 smithi121 bash[18489]: cluster 2024-02-12T15:59:37.119673+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e108: 12 total, 12 up, 12 in 2024-02-12T15:59:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:38 smithi121 bash[18489]: cluster 2024-02-12T15:59:37.455487+0000 mgr.y (mgr.14146) 414 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:38 smithi169 bash[19604]: cluster 2024-02-12T15:59:36.671910+0000 osd.4 (osd.4) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:38 smithi169 bash[19604]: cluster 2024-02-12T15:59:36.672003+0000 osd.4 (osd.4) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:38 smithi169 bash[19604]: cluster 2024-02-12T15:59:37.119673+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e108: 12 total, 12 up, 12 in 2024-02-12T15:59:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:38 smithi169 bash[19604]: cluster 2024-02-12T15:59:37.455487+0000 mgr.y (mgr.14146) 414 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:39.010 INFO:teuthology.orchestra.run.smithi121.stdout:55834574946 2024-02-12T15:59:39.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-12T15:59:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:40 smithi169 bash[19604]: cluster 2024-02-12T15:59:39.456121+0000 mgr.y (mgr.14146) 415 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:40.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:40 smithi123 bash[19119]: cluster 2024-02-12T15:59:39.456121+0000 mgr.y (mgr.14146) 415 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:40.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:40 smithi121 bash[18489]: cluster 2024-02-12T15:59:39.456121+0000 mgr.y (mgr.14146) 415 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:41 smithi169 bash[19604]: cluster 2024-02-12T15:59:40.113951+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:41 smithi169 bash[19604]: cluster 2024-02-12T15:59:40.114046+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:41 smithi169 bash[19604]: cluster 2024-02-12T15:59:41.104967+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e109: 12 total, 12 up, 12 in 2024-02-12T15:59:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:41 smithi123 bash[19119]: cluster 2024-02-12T15:59:40.113951+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:41 smithi123 bash[19119]: cluster 2024-02-12T15:59:40.114046+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:41 smithi123 bash[19119]: cluster 2024-02-12T15:59:41.104967+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e109: 12 total, 12 up, 12 in 2024-02-12T15:59:41.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:41 smithi121 bash[18489]: cluster 2024-02-12T15:59:40.113951+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:41.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:41 smithi121 bash[18489]: cluster 2024-02-12T15:59:40.114046+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:41.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:41 smithi121 bash[18489]: cluster 2024-02-12T15:59:41.104967+0000 mon.a (mon.0) 732 : cluster [DBG] osdmap e109: 12 total, 12 up, 12 in 2024-02-12T15:59:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:42 smithi169 bash[19604]: cluster 2024-02-12T15:59:41.456806+0000 mgr.y (mgr.14146) 416 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:42 smithi169 bash[19604]: cluster 2024-02-12T15:59:41.995403+0000 osd.0 (osd.0) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:42 smithi169 bash[19604]: cluster 2024-02-12T15:59:41.995493+0000 osd.0 (osd.0) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:42.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:42 smithi123 bash[19119]: cluster 2024-02-12T15:59:41.456806+0000 mgr.y (mgr.14146) 416 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:42 smithi123 bash[19119]: cluster 2024-02-12T15:59:41.995403+0000 osd.0 (osd.0) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:42 smithi123 bash[19119]: cluster 2024-02-12T15:59:41.995493+0000 osd.0 (osd.0) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:42.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:42 smithi121 bash[18489]: cluster 2024-02-12T15:59:41.456806+0000 mgr.y (mgr.14146) 416 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:42.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:42 smithi121 bash[18489]: cluster 2024-02-12T15:59:41.995403+0000 osd.0 (osd.0) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:42.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:42 smithi121 bash[18489]: cluster 2024-02-12T15:59:41.995493+0000 osd.0 (osd.0) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:43 smithi169 bash[19604]: cluster 2024-02-12T15:59:42.533112+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e110: 12 total, 12 up, 12 in 2024-02-12T15:59:43.807 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:59:43.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:43 smithi123 bash[19119]: cluster 2024-02-12T15:59:42.533112+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e110: 12 total, 12 up, 12 in 2024-02-12T15:59:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:43 smithi121 bash[18489]: cluster 2024-02-12T15:59:42.533112+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e110: 12 total, 12 up, 12 in 2024-02-12T15:59:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:44 smithi169 bash[19604]: cluster 2024-02-12T15:59:43.457433+0000 mgr.y (mgr.14146) 417 : cluster [DBG] pgmap v443: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:44.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:44 smithi121 bash[18489]: cluster 2024-02-12T15:59:43.457433+0000 mgr.y (mgr.14146) 417 : cluster [DBG] pgmap v443: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:44 smithi123 bash[19119]: cluster 2024-02-12T15:59:43.457433+0000 mgr.y (mgr.14146) 417 : cluster [DBG] pgmap v443: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:46 smithi123 bash[19119]: cluster 2024-02-12T15:59:45.458033+0000 mgr.y (mgr.14146) 418 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:46.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:46 smithi121 bash[18489]: cluster 2024-02-12T15:59:45.458033+0000 mgr.y (mgr.14146) 418 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:46 smithi169 bash[19604]: cluster 2024-02-12T15:59:45.458033+0000 mgr.y (mgr.14146) 418 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:47.316 INFO:teuthology.orchestra.run.smithi121.stdout:90194313309 2024-02-12T15:59:47.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-12T15:59:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:48 smithi169 bash[19604]: cluster 2024-02-12T15:59:47.458609+0000 mgr.y (mgr.14146) 419 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:48 smithi123 bash[19119]: cluster 2024-02-12T15:59:47.458609+0000 mgr.y (mgr.14146) 419 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:48.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:48 smithi121 bash[18489]: cluster 2024-02-12T15:59:47.458609+0000 mgr.y (mgr.14146) 419 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:50 smithi169 bash[19604]: cluster 2024-02-12T15:59:49.459236+0000 mgr.y (mgr.14146) 420 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:50 smithi123 bash[19119]: cluster 2024-02-12T15:59:49.459236+0000 mgr.y (mgr.14146) 420 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:50.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:50 smithi121 bash[18489]: cluster 2024-02-12T15:59:49.459236+0000 mgr.y (mgr.14146) 420 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:52.109 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T15:59:52.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:52 smithi169 bash[19604]: cluster 2024-02-12T15:59:51.459809+0000 mgr.y (mgr.14146) 421 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:52.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:52 smithi123 bash[19119]: cluster 2024-02-12T15:59:51.459809+0000 mgr.y (mgr.14146) 421 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:52 smithi121 bash[18489]: cluster 2024-02-12T15:59:51.459809+0000 mgr.y (mgr.14146) 421 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:54 smithi169 bash[19604]: cluster 2024-02-12T15:59:53.460443+0000 mgr.y (mgr.14146) 422 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:54.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:54 smithi123 bash[19119]: cluster 2024-02-12T15:59:53.460443+0000 mgr.y (mgr.14146) 422 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:54.899 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:54 smithi121 bash[18489]: cluster 2024-02-12T15:59:53.460443+0000 mgr.y (mgr.14146) 422 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:55.478 INFO:teuthology.orchestra.run.smithi121.stdout:111669149782 2024-02-12T15:59:55.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-12T15:59:56.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:56 smithi123 bash[19119]: cluster 2024-02-12T15:59:55.461015+0000 mgr.y (mgr.14146) 423 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:56 smithi123 bash[19119]: cluster 2024-02-12T15:59:55.821676+0000 osd.6 (osd.6) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:56 smithi123 bash[19119]: cluster 2024-02-12T15:59:55.821771+0000 osd.6 (osd.6) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:56.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:56 smithi121 bash[18489]: cluster 2024-02-12T15:59:55.461015+0000 mgr.y (mgr.14146) 423 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:56.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:56 smithi121 bash[18489]: cluster 2024-02-12T15:59:55.821676+0000 osd.6 (osd.6) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:56.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:56 smithi121 bash[18489]: cluster 2024-02-12T15:59:55.821771+0000 osd.6 (osd.6) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:56 smithi169 bash[19604]: cluster 2024-02-12T15:59:55.461015+0000 mgr.y (mgr.14146) 423 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:56 smithi169 bash[19604]: cluster 2024-02-12T15:59:55.821676+0000 osd.6 (osd.6) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T15:59:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:56 smithi169 bash[19604]: cluster 2024-02-12T15:59:55.821771+0000 osd.6 (osd.6) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T15:59:57.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:57 smithi123 bash[19119]: cluster 2024-02-12T15:59:56.123568+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e111: 12 total, 12 up, 12 in 2024-02-12T15:59:57.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:57 smithi121 bash[18489]: cluster 2024-02-12T15:59:56.123568+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e111: 12 total, 12 up, 12 in 2024-02-12T15:59:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:57 smithi169 bash[19604]: cluster 2024-02-12T15:59:56.123568+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e111: 12 total, 12 up, 12 in 2024-02-12T15:59:58.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 15:59:58 smithi121 bash[18489]: cluster 2024-02-12T15:59:57.461619+0000 mgr.y (mgr.14146) 424 : cluster [DBG] pgmap v451: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 15:59:58 smithi169 bash[19604]: cluster 2024-02-12T15:59:57.461619+0000 mgr.y (mgr.14146) 424 : cluster [DBG] pgmap v451: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T15:59:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 15:59:58 smithi123 bash[19119]: cluster 2024-02-12T15:59:57.461619+0000 mgr.y (mgr.14146) 424 : cluster [DBG] pgmap v451: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:00.278 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:00 smithi169 bash[19604]: cluster 2024-02-12T15:59:59.462240+0000 mgr.y (mgr.14146) 425 : cluster [DBG] pgmap v452: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:00 smithi169 bash[19604]: cluster 2024-02-12T16:00:00.000154+0000 mon.a (mon.0) 735 : cluster [INF] overall HEALTH_OK 2024-02-12T16:00:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:00 smithi123 bash[19119]: cluster 2024-02-12T15:59:59.462240+0000 mgr.y (mgr.14146) 425 : cluster [DBG] pgmap v452: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:00 smithi123 bash[19119]: cluster 2024-02-12T16:00:00.000154+0000 mon.a (mon.0) 735 : cluster [INF] overall HEALTH_OK 2024-02-12T16:00:00.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:00 smithi121 bash[18489]: cluster 2024-02-12T15:59:59.462240+0000 mgr.y (mgr.14146) 425 : cluster [DBG] pgmap v452: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:00.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:00 smithi121 bash[18489]: cluster 2024-02-12T16:00:00.000154+0000 mon.a (mon.0) 735 : cluster [INF] overall HEALTH_OK 2024-02-12T16:00:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:02 smithi169 bash[19604]: cluster 2024-02-12T16:00:01.462675+0000 mgr.y (mgr.14146) 426 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:02.802 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:02 smithi121 bash[18489]: cluster 2024-02-12T16:00:01.462675+0000 mgr.y (mgr.14146) 426 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:02.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:02 smithi123 bash[19119]: cluster 2024-02-12T16:00:01.462675+0000 mgr.y (mgr.14146) 426 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:03.682 INFO:teuthology.orchestra.run.smithi121.stdout:137438953550 2024-02-12T16:00:03.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-12T16:00:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.443363+0000 mon.a (mon.0) 736 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1681112297,config=1984608554,health=3279304564,kv=3887129236}) 2024-02-12T16:00:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.444888+0000 mon.a (mon.0) 737 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291550467}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.446436+0000 mon.a (mon.0) 738 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637083144}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.447951+0000 mon.a (mon.0) 739 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2501103854}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.449459+0000 mon.a (mon.0) 740 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=285957272}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.451020+0000 mon.a (mon.0) 741 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603942514}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.457308+0000 mon.a (mon.0) 742 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110981359}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.462626+0000 mon.a (mon.0) 743 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929067949}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.467214+0000 mon.a (mon.0) 744 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3868058736}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.471349+0000 mon.a (mon.0) 745 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193729246}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.474909+0000 mon.a (mon.0) 746 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3098284492,mdsmap=1531151339,mgr=629502609}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.476266+0000 mon.a (mon.0) 747 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=72} crc {mgr=1827276921,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2151962723,mon_config_key=1963666918}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.477310+0000 mon.a (mon.0) 748 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3632343759}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.478507+0000 mon.a (mon.0) 749 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=1390371099,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=1291903796,osdmap=666480663}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.479820+0000 mon.a (mon.0) 750 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1924881157}) 2024-02-12T16:00:03.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:03 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.487124+0000 mon.a (mon.0) 751 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=6} crc {osdmap=652125247}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.443363+0000 mon.a (mon.0) 736 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1681112297,config=1984608554,health=3279304564,kv=3887129236}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.444888+0000 mon.a (mon.0) 737 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291550467}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.446436+0000 mon.a (mon.0) 738 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637083144}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.447951+0000 mon.a (mon.0) 739 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2501103854}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.449459+0000 mon.a (mon.0) 740 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=285957272}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.451020+0000 mon.a (mon.0) 741 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603942514}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.457308+0000 mon.a (mon.0) 742 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110981359}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.462626+0000 mon.a (mon.0) 743 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929067949}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.467214+0000 mon.a (mon.0) 744 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3868058736}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.471349+0000 mon.a (mon.0) 745 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193729246}) 2024-02-12T16:00:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.474909+0000 mon.a (mon.0) 746 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3098284492,mdsmap=1531151339,mgr=629502609}) 2024-02-12T16:00:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.476266+0000 mon.a (mon.0) 747 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=72} crc {mgr=1827276921,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2151962723,mon_config_key=1963666918}) 2024-02-12T16:00:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.477310+0000 mon.a (mon.0) 748 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3632343759}) 2024-02-12T16:00:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.478507+0000 mon.a (mon.0) 749 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=1390371099,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=1291903796,osdmap=666480663}) 2024-02-12T16:00:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.479820+0000 mon.a (mon.0) 750 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1924881157}) 2024-02-12T16:00:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:03 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.487124+0000 mon.a (mon.0) 751 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=6} crc {osdmap=652125247}) 2024-02-12T16:00:03.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.443363+0000 mon.a (mon.0) 736 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=1681112297,config=1984608554,health=3279304564,kv=3887129236}) 2024-02-12T16:00:03.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.444888+0000 mon.a (mon.0) 737 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291550467}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.446436+0000 mon.a (mon.0) 738 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=637083144}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.447951+0000 mon.a (mon.0) 739 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2501103854}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.449459+0000 mon.a (mon.0) 740 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=285957272}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.451020+0000 mon.a (mon.0) 741 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1603942514}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.457308+0000 mon.a (mon.0) 742 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110981359}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.462626+0000 mon.a (mon.0) 743 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929067949}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.467214+0000 mon.a (mon.0) 744 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3868058736}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.471349+0000 mon.a (mon.0) 745 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193729246}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.474909+0000 mon.a (mon.0) 746 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3098284492,mdsmap=1531151339,mgr=629502609}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.476266+0000 mon.a (mon.0) 747 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mgr=13,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=72} crc {mgr=1827276921,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2151962723,mon_config_key=1963666918}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.477310+0000 mon.a (mon.0) 748 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3632343759}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.478507+0000 mon.a (mon.0) 749 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=1390371099,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=1291903796,osdmap=666480663}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.479820+0000 mon.a (mon.0) 750 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1924881157}) 2024-02-12T16:00:03.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:03 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.487124+0000 mon.a (mon.0) 751 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=6} crc {osdmap=652125247}) 2024-02-12T16:00:04.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:04 smithi169 bash[19604]: cluster 2024-02-12T16:00:03.463281+0000 mgr.y (mgr.14146) 427 : cluster [DBG] pgmap v454: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:04.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:04 smithi123 bash[19119]: cluster 2024-02-12T16:00:03.463281+0000 mgr.y (mgr.14146) 427 : cluster [DBG] pgmap v454: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:04 smithi121 bash[18489]: cluster 2024-02-12T16:00:03.463281+0000 mgr.y (mgr.14146) 427 : cluster [DBG] pgmap v454: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:06 smithi123 bash[19119]: cluster 2024-02-12T16:00:05.463902+0000 mgr.y (mgr.14146) 428 : cluster [DBG] pgmap v455: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:06.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:06 smithi121 bash[18489]: cluster 2024-02-12T16:00:05.463902+0000 mgr.y (mgr.14146) 428 : cluster [DBG] pgmap v455: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:06 smithi169 bash[19604]: cluster 2024-02-12T16:00:05.463902+0000 mgr.y (mgr.14146) 428 : cluster [DBG] pgmap v455: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:08.478 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:08 smithi169 bash[19604]: cluster 2024-02-12T16:00:07.464520+0000 mgr.y (mgr.14146) 429 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:08.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:08 smithi123 bash[19119]: cluster 2024-02-12T16:00:07.464520+0000 mgr.y (mgr.14146) 429 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:08.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:08 smithi121 bash[18489]: cluster 2024-02-12T16:00:07.464520+0000 mgr.y (mgr.14146) 429 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:10 smithi169 bash[19604]: cluster 2024-02-12T16:00:09.465039+0000 mgr.y (mgr.14146) 430 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:10.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:10 smithi123 bash[19119]: cluster 2024-02-12T16:00:09.465039+0000 mgr.y (mgr.14146) 430 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:10.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:10 smithi121 bash[18489]: cluster 2024-02-12T16:00:09.465039+0000 mgr.y (mgr.14146) 430 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:11 smithi169 bash[19604]: cluster 2024-02-12T16:00:10.184353+0000 osd.1 (osd.1) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:11 smithi169 bash[19604]: cluster 2024-02-12T16:00:10.184450+0000 osd.1 (osd.1) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:11 smithi169 bash[19604]: cluster 2024-02-12T16:00:11.115728+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e112: 12 total, 12 up, 12 in 2024-02-12T16:00:11.834 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:11 smithi121 bash[18489]: cluster 2024-02-12T16:00:10.184353+0000 osd.1 (osd.1) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:11.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:11 smithi121 bash[18489]: cluster 2024-02-12T16:00:10.184450+0000 osd.1 (osd.1) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:11.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:11 smithi121 bash[18489]: cluster 2024-02-12T16:00:11.115728+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e112: 12 total, 12 up, 12 in 2024-02-12T16:00:11.835 INFO:teuthology.orchestra.run.smithi121.stdout:167503724617 2024-02-12T16:00:11.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-12T16:00:11.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:11 smithi123 bash[19119]: cluster 2024-02-12T16:00:10.184353+0000 osd.1 (osd.1) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:11.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:11 smithi123 bash[19119]: cluster 2024-02-12T16:00:10.184450+0000 osd.1 (osd.1) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:11.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:11 smithi123 bash[19119]: cluster 2024-02-12T16:00:11.115728+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e112: 12 total, 12 up, 12 in 2024-02-12T16:00:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:12 smithi169 bash[19604]: cluster 2024-02-12T16:00:11.465632+0000 mgr.y (mgr.14146) 431 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:12.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:12 smithi123 bash[19119]: cluster 2024-02-12T16:00:11.465632+0000 mgr.y (mgr.14146) 431 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:12.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:12 smithi121 bash[18489]: cluster 2024-02-12T16:00:11.465632+0000 mgr.y (mgr.14146) 431 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:14.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:14 smithi169 bash[19604]: cluster 2024-02-12T16:00:13.466240+0000 mgr.y (mgr.14146) 432 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:14.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:14 smithi123 bash[19119]: cluster 2024-02-12T16:00:13.466240+0000 mgr.y (mgr.14146) 432 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:14.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:14 smithi121 bash[18489]: cluster 2024-02-12T16:00:13.466240+0000 mgr.y (mgr.14146) 432 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:16.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:16 smithi123 bash[19119]: cluster 2024-02-12T16:00:15.466862+0000 mgr.y (mgr.14146) 433 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:16.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:16 smithi121 bash[18489]: cluster 2024-02-12T16:00:15.466862+0000 mgr.y (mgr.14146) 433 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:16 smithi169 bash[19604]: cluster 2024-02-12T16:00:15.466862+0000 mgr.y (mgr.14146) 433 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:18 smithi169 bash[19604]: cluster 2024-02-12T16:00:17.467477+0000 mgr.y (mgr.14146) 434 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:18 smithi123 bash[19119]: cluster 2024-02-12T16:00:17.467477+0000 mgr.y (mgr.14146) 434 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:18.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:18 smithi121 bash[18489]: cluster 2024-02-12T16:00:17.467477+0000 mgr.y (mgr.14146) 434 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:20.156 INFO:teuthology.orchestra.run.smithi121.stdout:206158430276 2024-02-12T16:00:20.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-12T16:00:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:20 smithi169 bash[19604]: cluster 2024-02-12T16:00:19.468091+0000 mgr.y (mgr.14146) 435 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:20.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:20 smithi123 bash[19119]: cluster 2024-02-12T16:00:19.468091+0000 mgr.y (mgr.14146) 435 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:20.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:20 smithi121 bash[18489]: cluster 2024-02-12T16:00:19.468091+0000 mgr.y (mgr.14146) 435 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:22 smithi169 bash[19604]: cluster 2024-02-12T16:00:21.468731+0000 mgr.y (mgr.14146) 436 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:22.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:22 smithi123 bash[19119]: cluster 2024-02-12T16:00:21.468731+0000 mgr.y (mgr.14146) 436 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:22.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:22 smithi121 bash[18489]: cluster 2024-02-12T16:00:21.468731+0000 mgr.y (mgr.14146) 436 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:24.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:24 smithi169 bash[19604]: cluster 2024-02-12T16:00:23.469374+0000 mgr.y (mgr.14146) 437 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:24 smithi123 bash[19119]: cluster 2024-02-12T16:00:23.469374+0000 mgr.y (mgr.14146) 437 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:24.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:24 smithi121 bash[18489]: cluster 2024-02-12T16:00:23.469374+0000 mgr.y (mgr.14146) 437 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:24.950 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:25.826 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:25 smithi121 bash[18489]: cluster 2024-02-12T16:00:23.796269+0000 osd.10 (osd.10) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:25.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:25 smithi121 bash[18489]: cluster 2024-02-12T16:00:23.796401+0000 osd.10 (osd.10) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:25.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:25 smithi121 bash[18489]: cluster 2024-02-12T16:00:24.550295+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e113: 12 total, 12 up, 12 in 2024-02-12T16:00:25.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:25 smithi123 bash[19119]: cluster 2024-02-12T16:00:23.796269+0000 osd.10 (osd.10) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:25 smithi123 bash[19119]: cluster 2024-02-12T16:00:23.796401+0000 osd.10 (osd.10) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:25 smithi123 bash[19119]: cluster 2024-02-12T16:00:24.550295+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e113: 12 total, 12 up, 12 in 2024-02-12T16:00:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:25 smithi169 bash[19604]: cluster 2024-02-12T16:00:23.796269+0000 osd.10 (osd.10) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:25 smithi169 bash[19604]: cluster 2024-02-12T16:00:23.796401+0000 osd.10 (osd.10) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:25 smithi169 bash[19604]: cluster 2024-02-12T16:00:24.550295+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e113: 12 total, 12 up, 12 in 2024-02-12T16:00:26.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:26 smithi123 bash[19119]: cluster 2024-02-12T16:00:25.469975+0000 mgr.y (mgr.14146) 438 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:26.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:26 smithi121 bash[18489]: cluster 2024-02-12T16:00:25.469975+0000 mgr.y (mgr.14146) 438 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:26 smithi169 bash[19604]: cluster 2024-02-12T16:00:25.469975+0000 mgr.y (mgr.14146) 438 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:240518168636 2024-02-12T16:00:28.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-02-12T16:00:28.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:28 smithi123 bash[19119]: cluster 2024-02-12T16:00:27.470546+0000 mgr.y (mgr.14146) 439 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:28.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:28 smithi121 bash[18489]: cluster 2024-02-12T16:00:27.470546+0000 mgr.y (mgr.14146) 439 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:28 smithi169 bash[19604]: cluster 2024-02-12T16:00:27.470546+0000 mgr.y (mgr.14146) 439 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:30 smithi123 bash[19119]: cluster 2024-02-12T16:00:29.471177+0000 mgr.y (mgr.14146) 440 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:30.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:30 smithi121 bash[18489]: cluster 2024-02-12T16:00:29.471177+0000 mgr.y (mgr.14146) 440 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:30 smithi169 bash[19604]: cluster 2024-02-12T16:00:29.471177+0000 mgr.y (mgr.14146) 440 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:32.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:32 smithi123 bash[19119]: cluster 2024-02-12T16:00:31.471819+0000 mgr.y (mgr.14146) 441 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:32.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:32 smithi121 bash[18489]: cluster 2024-02-12T16:00:31.471819+0000 mgr.y (mgr.14146) 441 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:32 smithi169 bash[19604]: cluster 2024-02-12T16:00:31.471819+0000 mgr.y (mgr.14146) 441 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:33.275 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:34 smithi123 bash[19119]: cluster 2024-02-12T16:00:33.472484+0000 mgr.y (mgr.14146) 442 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:34 smithi123 bash[19119]: audit 2024-02-12T16:00:33.674065+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:00:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:34 smithi123 bash[19119]: audit 2024-02-12T16:00:33.982680+0000 mon.a (mon.0) 754 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:00:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:34 smithi123 bash[19119]: audit 2024-02-12T16:00:34.293486+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:00:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:34 smithi123 bash[19119]: audit 2024-02-12T16:00:34.295582+0000 mon.b (mon.2) 98 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:00:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:34 smithi123 bash[19119]: audit 2024-02-12T16:00:34.301040+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:00:34.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:34 smithi121 bash[18489]: cluster 2024-02-12T16:00:33.472484+0000 mgr.y (mgr.14146) 442 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:34.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:34 smithi121 bash[18489]: audit 2024-02-12T16:00:33.674065+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:00:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:34 smithi121 bash[18489]: audit 2024-02-12T16:00:33.982680+0000 mon.a (mon.0) 754 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:00:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:34 smithi121 bash[18489]: audit 2024-02-12T16:00:34.293486+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:00:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:34 smithi121 bash[18489]: audit 2024-02-12T16:00:34.295582+0000 mon.b (mon.2) 98 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:00:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:34 smithi121 bash[18489]: audit 2024-02-12T16:00:34.301040+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:00:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:34 smithi169 bash[19604]: cluster 2024-02-12T16:00:33.472484+0000 mgr.y (mgr.14146) 442 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:34 smithi169 bash[19604]: audit 2024-02-12T16:00:33.674065+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:00:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:34 smithi169 bash[19604]: audit 2024-02-12T16:00:33.982680+0000 mon.a (mon.0) 754 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:00:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:34 smithi169 bash[19604]: audit 2024-02-12T16:00:34.293486+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:00:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:34 smithi169 bash[19604]: audit 2024-02-12T16:00:34.295582+0000 mon.b (mon.2) 98 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:00:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:34 smithi169 bash[19604]: audit 2024-02-12T16:00:34.301040+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:00:35.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:35 smithi123 bash[19119]: cluster 2024-02-12T16:00:34.503511+0000 osd.7 (osd.7) 17 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:00:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:35 smithi123 bash[19119]: cluster 2024-02-12T16:00:34.507040+0000 osd.7 (osd.7) 18 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:00:35.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:35 smithi121 bash[18489]: cluster 2024-02-12T16:00:34.503511+0000 osd.7 (osd.7) 17 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:00:35.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:35 smithi121 bash[18489]: cluster 2024-02-12T16:00:34.507040+0000 osd.7 (osd.7) 18 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:00:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:35 smithi169 bash[19604]: cluster 2024-02-12T16:00:34.503511+0000 osd.7 (osd.7) 17 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:00:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:35 smithi169 bash[19604]: cluster 2024-02-12T16:00:34.507040+0000 osd.7 (osd.7) 18 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:00:36.787 INFO:teuthology.orchestra.run.smithi121.stdout:279172874293 2024-02-12T16:00:36.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-02-12T16:00:36.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:36 smithi123 bash[19119]: cluster 2024-02-12T16:00:35.473117+0000 mgr.y (mgr.14146) 443 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:36.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:36 smithi121 bash[18489]: cluster 2024-02-12T16:00:35.473117+0000 mgr.y (mgr.14146) 443 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:36 smithi169 bash[19604]: cluster 2024-02-12T16:00:35.473117+0000 mgr.y (mgr.14146) 443 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:37.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:37 smithi123 bash[19119]: cluster 2024-02-12T16:00:35.907346+0000 osd.7 (osd.7) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:37.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:37 smithi123 bash[19119]: cluster 2024-02-12T16:00:35.907444+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:37.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:37 smithi123 bash[19119]: cluster 2024-02-12T16:00:36.587487+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e114: 12 total, 12 up, 12 in 2024-02-12T16:00:37.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:37 smithi121 bash[18489]: cluster 2024-02-12T16:00:35.907346+0000 osd.7 (osd.7) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:37.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:37 smithi121 bash[18489]: cluster 2024-02-12T16:00:35.907444+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:37.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:37 smithi121 bash[18489]: cluster 2024-02-12T16:00:36.587487+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e114: 12 total, 12 up, 12 in 2024-02-12T16:00:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:37 smithi169 bash[19604]: cluster 2024-02-12T16:00:35.907346+0000 osd.7 (osd.7) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:37 smithi169 bash[19604]: cluster 2024-02-12T16:00:35.907444+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:37 smithi169 bash[19604]: cluster 2024-02-12T16:00:36.587487+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e114: 12 total, 12 up, 12 in 2024-02-12T16:00:38.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:38 smithi123 bash[19119]: cluster 2024-02-12T16:00:37.473715+0000 mgr.y (mgr.14146) 444 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:38.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:38 smithi121 bash[18489]: cluster 2024-02-12T16:00:37.473715+0000 mgr.y (mgr.14146) 444 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:38 smithi169 bash[19604]: cluster 2024-02-12T16:00:37.473715+0000 mgr.y (mgr.14146) 444 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:40.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:40 smithi123 bash[19119]: cluster 2024-02-12T16:00:39.474346+0000 mgr.y (mgr.14146) 445 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:40.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:40 smithi121 bash[18489]: cluster 2024-02-12T16:00:39.474346+0000 mgr.y (mgr.14146) 445 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:40 smithi169 bash[19604]: cluster 2024-02-12T16:00:39.474346+0000 mgr.y (mgr.14146) 445 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:41.580 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:42.744 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:42 smithi121 bash[18489]: cluster 2024-02-12T16:00:41.474956+0000 mgr.y (mgr.14146) 446 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:42.744 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:42 smithi121 bash[18489]: cluster 2024-02-12T16:00:41.815576+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:42.744 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:42 smithi121 bash[18489]: cluster 2024-02-12T16:00:41.815661+0000 osd.3 (osd.3) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:42.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:42 smithi123 bash[19119]: cluster 2024-02-12T16:00:41.474956+0000 mgr.y (mgr.14146) 446 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:42 smithi123 bash[19119]: cluster 2024-02-12T16:00:41.815576+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:42 smithi123 bash[19119]: cluster 2024-02-12T16:00:41.815661+0000 osd.3 (osd.3) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:42 smithi169 bash[19604]: cluster 2024-02-12T16:00:41.474956+0000 mgr.y (mgr.14146) 446 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:42 smithi169 bash[19604]: cluster 2024-02-12T16:00:41.815576+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:42 smithi169 bash[19604]: cluster 2024-02-12T16:00:41.815661+0000 osd.3 (osd.3) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:43.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:43 smithi123 bash[19119]: cluster 2024-02-12T16:00:42.392671+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:43 smithi123 bash[19119]: cluster 2024-02-12T16:00:42.392785+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:43 smithi123 bash[19119]: cluster 2024-02-12T16:00:42.614263+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e115: 12 total, 12 up, 12 in 2024-02-12T16:00:43.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:43 smithi121 bash[18489]: cluster 2024-02-12T16:00:42.392671+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:43.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:43 smithi121 bash[18489]: cluster 2024-02-12T16:00:42.392785+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:43.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:43 smithi121 bash[18489]: cluster 2024-02-12T16:00:42.614263+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e115: 12 total, 12 up, 12 in 2024-02-12T16:00:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:43 smithi169 bash[19604]: cluster 2024-02-12T16:00:42.392671+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:43 smithi169 bash[19604]: cluster 2024-02-12T16:00:42.392785+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:43 smithi169 bash[19604]: cluster 2024-02-12T16:00:42.614263+0000 mon.a (mon.0) 757 : cluster [DBG] osdmap e115: 12 total, 12 up, 12 in 2024-02-12T16:00:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:44 smithi123 bash[19119]: cluster 2024-02-12T16:00:43.475585+0000 mgr.y (mgr.14146) 447 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:44.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:44 smithi121 bash[18489]: cluster 2024-02-12T16:00:43.475585+0000 mgr.y (mgr.14146) 447 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:44 smithi169 bash[19604]: cluster 2024-02-12T16:00:43.475585+0000 mgr.y (mgr.14146) 447 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:45.109 INFO:teuthology.orchestra.run.smithi121.stdout:317827579952 2024-02-12T16:00:45.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-02-12T16:00:46.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:46 smithi121 bash[18489]: cluster 2024-02-12T16:00:45.476271+0000 mgr.y (mgr.14146) 448 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:46 smithi169 bash[19604]: cluster 2024-02-12T16:00:45.476271+0000 mgr.y (mgr.14146) 448 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:46.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:46 smithi123 bash[19119]: cluster 2024-02-12T16:00:45.476271+0000 mgr.y (mgr.14146) 448 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:48 smithi169 bash[19604]: cluster 2024-02-12T16:00:47.476896+0000 mgr.y (mgr.14146) 449 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:48.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:48 smithi123 bash[19119]: cluster 2024-02-12T16:00:47.476896+0000 mgr.y (mgr.14146) 449 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:48.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:48 smithi121 bash[18489]: cluster 2024-02-12T16:00:47.476896+0000 mgr.y (mgr.14146) 449 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:49.908 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:50 smithi169 bash[19604]: cluster 2024-02-12T16:00:48.957981+0000 osd.9 (osd.9) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:50 smithi169 bash[19604]: cluster 2024-02-12T16:00:48.958079+0000 osd.9 (osd.9) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:50 smithi169 bash[19604]: cluster 2024-02-12T16:00:49.477495+0000 mgr.y (mgr.14146) 450 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:50.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:50 smithi169 bash[19604]: cluster 2024-02-12T16:00:49.540074+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e116: 12 total, 12 up, 12 in 2024-02-12T16:00:50.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:50 smithi121 bash[18489]: cluster 2024-02-12T16:00:48.957981+0000 osd.9 (osd.9) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:50.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:50 smithi121 bash[18489]: cluster 2024-02-12T16:00:48.958079+0000 osd.9 (osd.9) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:50.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:50 smithi121 bash[18489]: cluster 2024-02-12T16:00:49.477495+0000 mgr.y (mgr.14146) 450 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:50 smithi121 bash[18489]: cluster 2024-02-12T16:00:49.540074+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e116: 12 total, 12 up, 12 in 2024-02-12T16:00:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:50 smithi123 bash[19119]: cluster 2024-02-12T16:00:48.957981+0000 osd.9 (osd.9) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:50 smithi123 bash[19119]: cluster 2024-02-12T16:00:48.958079+0000 osd.9 (osd.9) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:50 smithi123 bash[19119]: cluster 2024-02-12T16:00:49.477495+0000 mgr.y (mgr.14146) 450 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:50 smithi123 bash[19119]: cluster 2024-02-12T16:00:49.540074+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e116: 12 total, 12 up, 12 in 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: audit 2024-02-12T16:00:50.549261+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: audit 2024-02-12T16:00:50.549881+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: audit 2024-02-12T16:00:50.550273+0000 mon.a (mon.0) 761 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: audit 2024-02-12T16:00:50.561569+0000 mon.a (mon.0) 762 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: audit 2024-02-12T16:00:50.563681+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: audit 2024-02-12T16:00:50.564784+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: audit 2024-02-12T16:00:50.571042+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: cluster 2024-02-12T16:00:50.790073+0000 osd.11 (osd.11) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:51 smithi123 bash[19119]: cluster 2024-02-12T16:00:50.790192+0000 osd.11 (osd.11) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:51.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: audit 2024-02-12T16:00:50.549261+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: audit 2024-02-12T16:00:50.549881+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: audit 2024-02-12T16:00:50.550273+0000 mon.a (mon.0) 761 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: audit 2024-02-12T16:00:50.561569+0000 mon.a (mon.0) 762 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: audit 2024-02-12T16:00:50.563681+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: audit 2024-02-12T16:00:50.564784+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: audit 2024-02-12T16:00:50.571042+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: cluster 2024-02-12T16:00:50.790073+0000 osd.11 (osd.11) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:51 smithi121 bash[18489]: cluster 2024-02-12T16:00:50.790192+0000 osd.11 (osd.11) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: audit 2024-02-12T16:00:50.549261+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: audit 2024-02-12T16:00:50.549881+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: audit 2024-02-12T16:00:50.550273+0000 mon.a (mon.0) 761 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: audit 2024-02-12T16:00:50.561569+0000 mon.a (mon.0) 762 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: audit 2024-02-12T16:00:50.563681+0000 mon.a (mon.0) 763 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: audit 2024-02-12T16:00:50.564784+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: audit 2024-02-12T16:00:50.571042+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: cluster 2024-02-12T16:00:50.790073+0000 osd.11 (osd.11) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:51 smithi169 bash[19604]: cluster 2024-02-12T16:00:50.790192+0000 osd.11 (osd.11) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:52.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:52 smithi123 bash[19119]: cluster 2024-02-12T16:00:51.478231+0000 mgr.y (mgr.14146) 451 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:52.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:52 smithi123 bash[19119]: cluster 2024-02-12T16:00:51.582064+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e117: 12 total, 12 up, 12 in 2024-02-12T16:00:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:52 smithi121 bash[18489]: cluster 2024-02-12T16:00:51.478231+0000 mgr.y (mgr.14146) 451 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:52 smithi121 bash[18489]: cluster 2024-02-12T16:00:51.582064+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e117: 12 total, 12 up, 12 in 2024-02-12T16:00:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:52 smithi169 bash[19604]: cluster 2024-02-12T16:00:51.478231+0000 mgr.y (mgr.14146) 451 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:52 smithi169 bash[19604]: cluster 2024-02-12T16:00:51.582064+0000 mon.a (mon.0) 766 : cluster [DBG] osdmap e117: 12 total, 12 up, 12 in 2024-02-12T16:00:53.510 INFO:teuthology.orchestra.run.smithi121.stdout:356482285608 2024-02-12T16:00:53.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-02-12T16:00:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:53 smithi123 bash[19119]: cluster 2024-02-12T16:00:52.307247+0000 osd.8 (osd.8) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:53 smithi123 bash[19119]: cluster 2024-02-12T16:00:52.307329+0000 osd.8 (osd.8) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:53 smithi123 bash[19119]: cluster 2024-02-12T16:00:52.591096+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e118: 12 total, 12 up, 12 in 2024-02-12T16:00:53.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:53 smithi121 bash[18489]: cluster 2024-02-12T16:00:52.307247+0000 osd.8 (osd.8) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:53.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:53 smithi121 bash[18489]: cluster 2024-02-12T16:00:52.307329+0000 osd.8 (osd.8) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:53.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:53 smithi121 bash[18489]: cluster 2024-02-12T16:00:52.591096+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e118: 12 total, 12 up, 12 in 2024-02-12T16:00:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:53 smithi169 bash[19604]: cluster 2024-02-12T16:00:52.307247+0000 osd.8 (osd.8) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:53 smithi169 bash[19604]: cluster 2024-02-12T16:00:52.307329+0000 osd.8 (osd.8) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:53 smithi169 bash[19604]: cluster 2024-02-12T16:00:52.591096+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e118: 12 total, 12 up, 12 in 2024-02-12T16:00:54.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:54 smithi123 bash[19119]: cluster 2024-02-12T16:00:53.478984+0000 mgr.y (mgr.14146) 452 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:54.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:54 smithi121 bash[18489]: cluster 2024-02-12T16:00:53.478984+0000 mgr.y (mgr.14146) 452 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:54 smithi169 bash[19604]: cluster 2024-02-12T16:00:53.478984+0000 mgr.y (mgr.14146) 452 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:56.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:56 smithi121 bash[18489]: cluster 2024-02-12T16:00:55.479724+0000 mgr.y (mgr.14146) 453 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:56 smithi169 bash[19604]: cluster 2024-02-12T16:00:55.479724+0000 mgr.y (mgr.14146) 453 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:56 smithi123 bash[19119]: cluster 2024-02-12T16:00:55.479724+0000 mgr.y (mgr.14146) 453 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:00:58.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:58 smithi121 bash[18489]: cluster 2024-02-12T16:00:56.631845+0000 osd.4 (osd.4) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:58.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:58 smithi121 bash[18489]: cluster 2024-02-12T16:00:56.631931+0000 osd.4 (osd.4) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:58.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:58 smithi121 bash[18489]: cluster 2024-02-12T16:00:57.146939+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e119: 12 total, 12 up, 12 in 2024-02-12T16:00:58.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:58 smithi121 bash[18489]: cluster 2024-02-12T16:00:57.480275+0000 mgr.y (mgr.14146) 454 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:58 smithi169 bash[19604]: cluster 2024-02-12T16:00:56.631845+0000 osd.4 (osd.4) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:58 smithi169 bash[19604]: cluster 2024-02-12T16:00:56.631931+0000 osd.4 (osd.4) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:58 smithi169 bash[19604]: cluster 2024-02-12T16:00:57.146939+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e119: 12 total, 12 up, 12 in 2024-02-12T16:00:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:58 smithi169 bash[19604]: cluster 2024-02-12T16:00:57.480275+0000 mgr.y (mgr.14146) 454 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:58 smithi123 bash[19119]: cluster 2024-02-12T16:00:56.631845+0000 osd.4 (osd.4) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:58 smithi123 bash[19119]: cluster 2024-02-12T16:00:56.631931+0000 osd.4 (osd.4) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:58 smithi123 bash[19119]: cluster 2024-02-12T16:00:57.146939+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e119: 12 total, 12 up, 12 in 2024-02-12T16:00:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:58 smithi123 bash[19119]: cluster 2024-02-12T16:00:57.480275+0000 mgr.y (mgr.14146) 454 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:00:59.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:59 smithi121 bash[18489]: cluster 2024-02-12T16:00:58.436914+0000 osd.0 (osd.0) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:59.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:00:59 smithi121 bash[18489]: cluster 2024-02-12T16:00:58.437038+0000 osd.0 (osd.0) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:59 smithi169 bash[19604]: cluster 2024-02-12T16:00:58.436914+0000 osd.0 (osd.0) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:00:59 smithi169 bash[19604]: cluster 2024-02-12T16:00:58.437038+0000 osd.0 (osd.0) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:00:59.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:59 smithi123 bash[19119]: cluster 2024-02-12T16:00:58.436914+0000 osd.0 (osd.0) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:00:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:00:59 smithi123 bash[19119]: cluster 2024-02-12T16:00:58.437038+0000 osd.0 (osd.0) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:00 smithi169 bash[19604]: cluster 2024-02-12T16:00:59.156188+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e120: 12 total, 12 up, 12 in 2024-02-12T16:01:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:00 smithi169 bash[19604]: cluster 2024-02-12T16:00:59.480911+0000 mgr.y (mgr.14146) 455 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:00 smithi123 bash[19119]: cluster 2024-02-12T16:00:59.156188+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e120: 12 total, 12 up, 12 in 2024-02-12T16:01:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:00 smithi123 bash[19119]: cluster 2024-02-12T16:00:59.480911+0000 mgr.y (mgr.14146) 455 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:00.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:00 smithi121 bash[18489]: cluster 2024-02-12T16:00:59.156188+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e120: 12 total, 12 up, 12 in 2024-02-12T16:01:00.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:00 smithi121 bash[18489]: cluster 2024-02-12T16:00:59.480911+0000 mgr.y (mgr.14146) 455 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:01 smithi169 bash[19604]: cluster 2024-02-12T16:00:59.760597+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:01 smithi169 bash[19604]: cluster 2024-02-12T16:00:59.760684+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:01 smithi169 bash[19604]: cluster 2024-02-12T16:01:00.190043+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e121: 12 total, 12 up, 12 in 2024-02-12T16:01:01.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:01 smithi123 bash[19119]: cluster 2024-02-12T16:00:59.760597+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:01 smithi123 bash[19119]: cluster 2024-02-12T16:00:59.760684+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:01 smithi123 bash[19119]: cluster 2024-02-12T16:01:00.190043+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e121: 12 total, 12 up, 12 in 2024-02-12T16:01:01.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:01 smithi121 bash[18489]: cluster 2024-02-12T16:00:59.760597+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:01.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:01 smithi121 bash[18489]: cluster 2024-02-12T16:00:59.760684+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:01.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:01 smithi121 bash[18489]: cluster 2024-02-12T16:01:00.190043+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e121: 12 total, 12 up, 12 in 2024-02-12T16:01:01.835 INFO:teuthology.orchestra.run.smithi121.stdout:399431958562 2024-02-12T16:01:01.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-12T16:01:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:02 smithi169 bash[19604]: cluster 2024-02-12T16:01:01.481572+0000 mgr.y (mgr.14146) 456 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:02.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:02 smithi123 bash[19119]: cluster 2024-02-12T16:01:01.481572+0000 mgr.y (mgr.14146) 456 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:02.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:02 smithi121 bash[18489]: cluster 2024-02-12T16:01:01.481572+0000 mgr.y (mgr.14146) 456 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:04.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:04 smithi169 bash[19604]: cluster 2024-02-12T16:01:03.482174+0000 mgr.y (mgr.14146) 457 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:04.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:04 smithi123 bash[19119]: cluster 2024-02-12T16:01:03.482174+0000 mgr.y (mgr.14146) 457 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:04.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:04 smithi121 bash[18489]: cluster 2024-02-12T16:01:03.482174+0000 mgr.y (mgr.14146) 457 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:06.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:06 smithi121 bash[18489]: cluster 2024-02-12T16:01:05.482798+0000 mgr.y (mgr.14146) 458 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:06 smithi169 bash[19604]: cluster 2024-02-12T16:01:05.482798+0000 mgr.y (mgr.14146) 458 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:06.630 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:01:06.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:06 smithi123 bash[19119]: cluster 2024-02-12T16:01:05.482798+0000 mgr.y (mgr.14146) 458 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:08 smithi169 bash[19604]: cluster 2024-02-12T16:01:07.483442+0000 mgr.y (mgr.14146) 459 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:08 smithi169 bash[19604]: cluster 2024-02-12T16:01:07.644415+0000 osd.6 (osd.6) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:08 smithi169 bash[19604]: cluster 2024-02-12T16:01:07.644508+0000 osd.6 (osd.6) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:08.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:08 smithi123 bash[19119]: cluster 2024-02-12T16:01:07.483442+0000 mgr.y (mgr.14146) 459 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:08.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:08 smithi123 bash[19119]: cluster 2024-02-12T16:01:07.644415+0000 osd.6 (osd.6) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:08.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:08 smithi123 bash[19119]: cluster 2024-02-12T16:01:07.644508+0000 osd.6 (osd.6) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:08.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:08 smithi121 bash[18489]: cluster 2024-02-12T16:01:07.483442+0000 mgr.y (mgr.14146) 459 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:08.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:08 smithi121 bash[18489]: cluster 2024-02-12T16:01:07.644415+0000 osd.6 (osd.6) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:08.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:08 smithi121 bash[18489]: cluster 2024-02-12T16:01:07.644508+0000 osd.6 (osd.6) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:09.570 INFO:teuthology.orchestra.run.smithi121.stdout:34359738494 2024-02-12T16:01:09.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:09 smithi123 bash[19119]: cluster 2024-02-12T16:01:08.550496+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e122: 12 total, 12 up, 12 in 2024-02-12T16:01:09.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:09 smithi121 bash[18489]: cluster 2024-02-12T16:01:08.550496+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e122: 12 total, 12 up, 12 in 2024-02-12T16:01:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:09 smithi169 bash[19604]: cluster 2024-02-12T16:01:08.550496+0000 mon.a (mon.0) 771 : cluster [DBG] osdmap e122: 12 total, 12 up, 12 in 2024-02-12T16:01:10.194 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738475 got 34359738494 for osd.0 2024-02-12T16:01:10.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-12T16:01:10.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:10 smithi123 bash[19119]: cluster 2024-02-12T16:01:09.484128+0000 mgr.y (mgr.14146) 460 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:10.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:10 smithi123 bash[19119]: audit 2024-02-12T16:01:09.562108+0000 mon.a (mon.0) 772 : audit [DBG] from='client.? v1:172.21.15.121:0/3118673675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:01:10.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:10 smithi121 bash[18489]: cluster 2024-02-12T16:01:09.484128+0000 mgr.y (mgr.14146) 460 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:10.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:10 smithi121 bash[18489]: audit 2024-02-12T16:01:09.562108+0000 mon.a (mon.0) 772 : audit [DBG] from='client.? v1:172.21.15.121:0/3118673675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:01:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:10 smithi169 bash[19604]: cluster 2024-02-12T16:01:09.484128+0000 mgr.y (mgr.14146) 460 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:10 smithi169 bash[19604]: audit 2024-02-12T16:01:09.562108+0000 mon.a (mon.0) 772 : audit [DBG] from='client.? v1:172.21.15.121:0/3118673675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:01:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:12 smithi123 bash[19119]: cluster 2024-02-12T16:01:11.484890+0000 mgr.y (mgr.14146) 461 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:12.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:12 smithi121 bash[18489]: cluster 2024-02-12T16:01:11.484890+0000 mgr.y (mgr.14146) 461 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:12 smithi169 bash[19604]: cluster 2024-02-12T16:01:11.484890+0000 mgr.y (mgr.14146) 461 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:14.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:14 smithi123 bash[19119]: cluster 2024-02-12T16:01:13.485554+0000 mgr.y (mgr.14146) 462 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:14.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:14 smithi121 bash[18489]: cluster 2024-02-12T16:01:13.485554+0000 mgr.y (mgr.14146) 462 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:01:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:14 smithi169 bash[19604]: cluster 2024-02-12T16:01:13.485554+0000 mgr.y (mgr.14146) 462 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:16.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:16 smithi121 bash[18489]: cluster 2024-02-12T16:01:15.486228+0000 mgr.y (mgr.14146) 463 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:16 smithi169 bash[19604]: cluster 2024-02-12T16:01:15.486228+0000 mgr.y (mgr.14146) 463 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:16.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:16 smithi123 bash[19119]: cluster 2024-02-12T16:01:15.486228+0000 mgr.y (mgr.14146) 463 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:17.838 INFO:teuthology.orchestra.run.smithi121.stdout:55834574966 2024-02-12T16:01:18.449 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574946 got 55834574966 for osd.1 2024-02-12T16:01:18.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-12T16:01:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:18 smithi123 bash[19119]: cluster 2024-02-12T16:01:17.486859+0000 mgr.y (mgr.14146) 464 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:18 smithi123 bash[19119]: audit 2024-02-12T16:01:17.829719+0000 mon.a (mon.0) 773 : audit [DBG] from='client.? v1:172.21.15.121:0/949667036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:01:18.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:18 smithi121 bash[18489]: cluster 2024-02-12T16:01:17.486859+0000 mgr.y (mgr.14146) 464 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:18.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:18 smithi121 bash[18489]: audit 2024-02-12T16:01:17.829719+0000 mon.a (mon.0) 773 : audit [DBG] from='client.? v1:172.21.15.121:0/949667036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:01:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:18 smithi169 bash[19604]: cluster 2024-02-12T16:01:17.486859+0000 mgr.y (mgr.14146) 464 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:18 smithi169 bash[19604]: audit 2024-02-12T16:01:17.829719+0000 mon.a (mon.0) 773 : audit [DBG] from='client.? v1:172.21.15.121:0/949667036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:01:20.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:20 smithi123 bash[19119]: cluster 2024-02-12T16:01:19.078577+0000 osd.1 (osd.1) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:20 smithi123 bash[19119]: cluster 2024-02-12T16:01:19.078664+0000 osd.1 (osd.1) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:20 smithi123 bash[19119]: cluster 2024-02-12T16:01:19.487521+0000 mgr.y (mgr.14146) 465 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:20 smithi123 bash[19119]: cluster 2024-02-12T16:01:19.555324+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e123: 12 total, 12 up, 12 in 2024-02-12T16:01:20.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:20 smithi121 bash[18489]: cluster 2024-02-12T16:01:19.078577+0000 osd.1 (osd.1) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:20.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:20 smithi121 bash[18489]: cluster 2024-02-12T16:01:19.078664+0000 osd.1 (osd.1) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:20.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:20 smithi121 bash[18489]: cluster 2024-02-12T16:01:19.487521+0000 mgr.y (mgr.14146) 465 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:20.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:20 smithi121 bash[18489]: cluster 2024-02-12T16:01:19.555324+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e123: 12 total, 12 up, 12 in 2024-02-12T16:01:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:20 smithi169 bash[19604]: cluster 2024-02-12T16:01:19.078577+0000 osd.1 (osd.1) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:20 smithi169 bash[19604]: cluster 2024-02-12T16:01:19.078664+0000 osd.1 (osd.1) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:20 smithi169 bash[19604]: cluster 2024-02-12T16:01:19.487521+0000 mgr.y (mgr.14146) 465 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:20 smithi169 bash[19604]: cluster 2024-02-12T16:01:19.555324+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e123: 12 total, 12 up, 12 in 2024-02-12T16:01:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:22 smithi123 bash[19119]: cluster 2024-02-12T16:01:21.488108+0000 mgr.y (mgr.14146) 466 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:22.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:22 smithi121 bash[18489]: cluster 2024-02-12T16:01:21.488108+0000 mgr.y (mgr.14146) 466 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:22 smithi169 bash[19604]: cluster 2024-02-12T16:01:21.488108+0000 mgr.y (mgr.14146) 466 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:23.249 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:01:24.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:24 smithi123 bash[19119]: cluster 2024-02-12T16:01:23.488722+0000 mgr.y (mgr.14146) 467 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:24.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:24 smithi121 bash[18489]: cluster 2024-02-12T16:01:23.488722+0000 mgr.y (mgr.14146) 467 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:24 smithi169 bash[19604]: cluster 2024-02-12T16:01:23.488722+0000 mgr.y (mgr.14146) 467 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:26.211 INFO:teuthology.orchestra.run.smithi121.stdout:90194313328 2024-02-12T16:01:26.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:26 smithi121 bash[18489]: cluster 2024-02-12T16:01:25.489344+0000 mgr.y (mgr.14146) 468 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:26 smithi169 bash[19604]: cluster 2024-02-12T16:01:25.489344+0000 mgr.y (mgr.14146) 468 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:26.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:26 smithi123 bash[19119]: cluster 2024-02-12T16:01:25.489344+0000 mgr.y (mgr.14146) 468 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:26.840 INFO:tasks.cephadm.ceph_manager.ceph:need seq 90194313309 got 90194313328 for osd.2 2024-02-12T16:01:26.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-12T16:01:27.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:27 smithi121 bash[18489]: audit 2024-02-12T16:01:26.203425+0000 mon.a (mon.0) 775 : audit [DBG] from='client.? v1:172.21.15.121:0/2379482346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:01:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:27 smithi169 bash[19604]: audit 2024-02-12T16:01:26.203425+0000 mon.a (mon.0) 775 : audit [DBG] from='client.? v1:172.21.15.121:0/2379482346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:01:27.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:27 smithi123 bash[19119]: audit 2024-02-12T16:01:26.203425+0000 mon.a (mon.0) 775 : audit [DBG] from='client.? v1:172.21.15.121:0/2379482346' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:01:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:28 smithi169 bash[19604]: cluster 2024-02-12T16:01:27.489946+0000 mgr.y (mgr.14146) 469 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:28.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:28 smithi123 bash[19119]: cluster 2024-02-12T16:01:27.489946+0000 mgr.y (mgr.14146) 469 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:28.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:28 smithi121 bash[18489]: cluster 2024-02-12T16:01:27.489946+0000 mgr.y (mgr.14146) 469 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:30 smithi123 bash[19119]: cluster 2024-02-12T16:01:29.490533+0000 mgr.y (mgr.14146) 470 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:30.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:30 smithi121 bash[18489]: cluster 2024-02-12T16:01:29.490533+0000 mgr.y (mgr.14146) 470 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:30 smithi169 bash[19604]: cluster 2024-02-12T16:01:29.490533+0000 mgr.y (mgr.14146) 470 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:31.634 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:01:32.788 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:32 smithi121 bash[18489]: cluster 2024-02-12T16:01:31.491156+0000 mgr.y (mgr.14146) 471 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:32.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:32 smithi123 bash[19119]: cluster 2024-02-12T16:01:31.491156+0000 mgr.y (mgr.14146) 471 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:32 smithi169 bash[19604]: cluster 2024-02-12T16:01:31.491156+0000 mgr.y (mgr.14146) 471 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:34.708 INFO:teuthology.orchestra.run.smithi121.stdout:111669149801 2024-02-12T16:01:34.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:34 smithi123 bash[19119]: cluster 2024-02-12T16:01:33.491806+0000 mgr.y (mgr.14146) 472 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:34 smithi121 bash[18489]: cluster 2024-02-12T16:01:33.491806+0000 mgr.y (mgr.14146) 472 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:35.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:34 smithi169 bash[19604]: cluster 2024-02-12T16:01:33.491806+0000 mgr.y (mgr.14146) 472 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:35.360 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149782 got 111669149801 for osd.3 2024-02-12T16:01:35.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-12T16:01:35.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:35 smithi121 bash[18489]: audit 2024-02-12T16:01:34.700501+0000 mon.a (mon.0) 776 : audit [DBG] from='client.? v1:172.21.15.121:0/3952756968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:01:35.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:35 smithi123 bash[19119]: audit 2024-02-12T16:01:34.700501+0000 mon.a (mon.0) 776 : audit [DBG] from='client.? v1:172.21.15.121:0/3952756968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:01:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:35 smithi169 bash[19604]: audit 2024-02-12T16:01:34.700501+0000 mon.a (mon.0) 776 : audit [DBG] from='client.? v1:172.21.15.121:0/3952756968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:01:36.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:36 smithi123 bash[19119]: cluster 2024-02-12T16:01:35.492413+0000 mgr.y (mgr.14146) 473 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:36.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:36 smithi121 bash[18489]: cluster 2024-02-12T16:01:35.492413+0000 mgr.y (mgr.14146) 473 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:36 smithi169 bash[19604]: cluster 2024-02-12T16:01:35.492413+0000 mgr.y (mgr.14146) 473 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:38.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:38 smithi123 bash[19119]: cluster 2024-02-12T16:01:37.493005+0000 mgr.y (mgr.14146) 474 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:38.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:38 smithi121 bash[18489]: cluster 2024-02-12T16:01:37.493005+0000 mgr.y (mgr.14146) 474 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:38 smithi169 bash[19604]: cluster 2024-02-12T16:01:37.493005+0000 mgr.y (mgr.14146) 474 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:40.158 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:01:40.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:40 smithi123 bash[19119]: cluster 2024-02-12T16:01:39.493595+0000 mgr.y (mgr.14146) 475 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:40.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:40 smithi121 bash[18489]: cluster 2024-02-12T16:01:39.493595+0000 mgr.y (mgr.14146) 475 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:40 smithi169 bash[19604]: cluster 2024-02-12T16:01:39.493595+0000 mgr.y (mgr.14146) 475 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:42 smithi123 bash[19119]: cluster 2024-02-12T16:01:41.494196+0000 mgr.y (mgr.14146) 476 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:42.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:42 smithi121 bash[18489]: cluster 2024-02-12T16:01:41.494196+0000 mgr.y (mgr.14146) 476 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:42 smithi169 bash[19604]: cluster 2024-02-12T16:01:41.494196+0000 mgr.y (mgr.14146) 476 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:43.126 INFO:teuthology.orchestra.run.smithi121.stdout:137438953570 2024-02-12T16:01:43.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953550 got 137438953570 for osd.4 2024-02-12T16:01:43.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-12T16:01:43.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:43 smithi123 bash[19119]: audit 2024-02-12T16:01:43.118594+0000 mon.a (mon.0) 777 : audit [DBG] from='client.? v1:172.21.15.121:0/3042069849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:01:43.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:43 smithi121 bash[18489]: audit 2024-02-12T16:01:43.118594+0000 mon.a (mon.0) 777 : audit [DBG] from='client.? v1:172.21.15.121:0/3042069849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:01:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:43 smithi169 bash[19604]: audit 2024-02-12T16:01:43.118594+0000 mon.a (mon.0) 777 : audit [DBG] from='client.? v1:172.21.15.121:0/3042069849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:01:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:44 smithi123 bash[19119]: cluster 2024-02-12T16:01:43.494791+0000 mgr.y (mgr.14146) 477 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:44.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:44 smithi121 bash[18489]: cluster 2024-02-12T16:01:43.494791+0000 mgr.y (mgr.14146) 477 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:44 smithi169 bash[19604]: cluster 2024-02-12T16:01:43.494791+0000 mgr.y (mgr.14146) 477 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:45.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:45 smithi123 bash[19119]: cluster 2024-02-12T16:01:44.461815+0000 osd.7 (osd.7) 21 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:01:45.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:45 smithi123 bash[19119]: cluster 2024-02-12T16:01:44.465477+0000 osd.7 (osd.7) 22 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:01:45.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:45 smithi121 bash[18489]: cluster 2024-02-12T16:01:44.461815+0000 osd.7 (osd.7) 21 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:01:45.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:45 smithi121 bash[18489]: cluster 2024-02-12T16:01:44.465477+0000 osd.7 (osd.7) 22 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:01:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:45 smithi169 bash[19604]: cluster 2024-02-12T16:01:44.461815+0000 osd.7 (osd.7) 21 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:01:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:45 smithi169 bash[19604]: cluster 2024-02-12T16:01:44.465477+0000 osd.7 (osd.7) 22 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:01:46.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:46 smithi123 bash[19119]: cluster 2024-02-12T16:01:45.495402+0000 mgr.y (mgr.14146) 478 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:46.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:46 smithi121 bash[18489]: cluster 2024-02-12T16:01:45.495402+0000 mgr.y (mgr.14146) 478 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:47.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:46 smithi169 bash[19604]: cluster 2024-02-12T16:01:45.495402+0000 mgr.y (mgr.14146) 478 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:48.565 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:01:48.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:48 smithi123 bash[19119]: cluster 2024-02-12T16:01:47.495985+0000 mgr.y (mgr.14146) 479 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:48.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:48 smithi121 bash[18489]: cluster 2024-02-12T16:01:47.495985+0000 mgr.y (mgr.14146) 479 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:49.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:48 smithi169 bash[19604]: cluster 2024-02-12T16:01:47.495985+0000 mgr.y (mgr.14146) 479 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:50 smithi123 bash[19119]: cluster 2024-02-12T16:01:49.496641+0000 mgr.y (mgr.14146) 480 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:50 smithi123 bash[19119]: audit 2024-02-12T16:01:50.575760+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:01:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:50 smithi121 bash[18489]: cluster 2024-02-12T16:01:49.496641+0000 mgr.y (mgr.14146) 480 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:50 smithi121 bash[18489]: audit 2024-02-12T16:01:50.575760+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:01:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:50 smithi169 bash[19604]: cluster 2024-02-12T16:01:49.496641+0000 mgr.y (mgr.14146) 480 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:50 smithi169 bash[19604]: audit 2024-02-12T16:01:50.575760+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:01:51.436 INFO:teuthology.orchestra.run.smithi121.stdout:167503724637 2024-02-12T16:01:51.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:51 smithi123 bash[19119]: audit 2024-02-12T16:01:50.900880+0000 mon.a (mon.0) 779 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:01:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:51 smithi123 bash[19119]: audit 2024-02-12T16:01:50.902278+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:01:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:51 smithi123 bash[19119]: audit 2024-02-12T16:01:50.911222+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:01:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:51 smithi123 bash[19119]: cluster 2024-02-12T16:01:51.140226+0000 osd.10 (osd.10) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:51 smithi123 bash[19119]: cluster 2024-02-12T16:01:51.140321+0000 osd.10 (osd.10) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:51 smithi123 bash[19119]: audit 2024-02-12T16:01:51.428014+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? v1:172.21.15.121:0/945598903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:01:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:51 smithi121 bash[18489]: audit 2024-02-12T16:01:50.900880+0000 mon.a (mon.0) 779 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:01:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:51 smithi121 bash[18489]: audit 2024-02-12T16:01:50.902278+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:01:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:51 smithi121 bash[18489]: audit 2024-02-12T16:01:50.911222+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:01:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:51 smithi121 bash[18489]: cluster 2024-02-12T16:01:51.140226+0000 osd.10 (osd.10) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:51 smithi121 bash[18489]: cluster 2024-02-12T16:01:51.140321+0000 osd.10 (osd.10) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:51.914 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:51 smithi121 bash[18489]: audit 2024-02-12T16:01:51.428014+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? v1:172.21.15.121:0/945598903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:01:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:51 smithi169 bash[19604]: audit 2024-02-12T16:01:50.900880+0000 mon.a (mon.0) 779 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:01:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:51 smithi169 bash[19604]: audit 2024-02-12T16:01:50.902278+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:01:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:51 smithi169 bash[19604]: audit 2024-02-12T16:01:50.911222+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:01:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:51 smithi169 bash[19604]: cluster 2024-02-12T16:01:51.140226+0000 osd.10 (osd.10) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:51 smithi169 bash[19604]: cluster 2024-02-12T16:01:51.140321+0000 osd.10 (osd.10) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:52.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:51 smithi169 bash[19604]: audit 2024-02-12T16:01:51.428014+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? v1:172.21.15.121:0/945598903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:01:52.081 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724617 got 167503724637 for osd.5 2024-02-12T16:01:52.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-12T16:01:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:52 smithi169 bash[19604]: cluster 2024-02-12T16:01:51.497240+0000 mgr.y (mgr.14146) 481 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:52 smithi169 bash[19604]: cluster 2024-02-12T16:01:51.925274+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e124: 12 total, 12 up, 12 in 2024-02-12T16:01:53.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:52 smithi123 bash[19119]: cluster 2024-02-12T16:01:51.497240+0000 mgr.y (mgr.14146) 481 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:53.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:52 smithi123 bash[19119]: cluster 2024-02-12T16:01:51.925274+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e124: 12 total, 12 up, 12 in 2024-02-12T16:01:53.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:52 smithi121 bash[18489]: cluster 2024-02-12T16:01:51.497240+0000 mgr.y (mgr.14146) 481 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:53.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:52 smithi121 bash[18489]: cluster 2024-02-12T16:01:51.925274+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e124: 12 total, 12 up, 12 in 2024-02-12T16:01:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:53 smithi169 bash[19604]: cluster 2024-02-12T16:01:53.591076+0000 osd.3 (osd.3) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:53 smithi169 bash[19604]: cluster 2024-02-12T16:01:53.591163+0000 osd.3 (osd.3) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:54.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:53 smithi123 bash[19119]: cluster 2024-02-12T16:01:53.591076+0000 osd.3 (osd.3) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:54.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:53 smithi123 bash[19119]: cluster 2024-02-12T16:01:53.591163+0000 osd.3 (osd.3) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:54.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:53 smithi121 bash[18489]: cluster 2024-02-12T16:01:53.591076+0000 osd.3 (osd.3) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:54.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:53 smithi121 bash[18489]: cluster 2024-02-12T16:01:53.591163+0000 osd.3 (osd.3) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:55.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:54 smithi169 bash[19604]: cluster 2024-02-12T16:01:53.497839+0000 mgr.y (mgr.14146) 482 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:55.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:54 smithi169 bash[19604]: cluster 2024-02-12T16:01:53.935161+0000 mon.a (mon.0) 784 : cluster [DBG] osdmap e125: 12 total, 12 up, 12 in 2024-02-12T16:01:55.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:54 smithi169 bash[19604]: cluster 2024-02-12T16:01:54.084829+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:55.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:54 smithi169 bash[19604]: cluster 2024-02-12T16:01:54.084916+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:55.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:54 smithi123 bash[19119]: cluster 2024-02-12T16:01:53.497839+0000 mgr.y (mgr.14146) 482 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:55.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:54 smithi123 bash[19119]: cluster 2024-02-12T16:01:53.935161+0000 mon.a (mon.0) 784 : cluster [DBG] osdmap e125: 12 total, 12 up, 12 in 2024-02-12T16:01:55.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:54 smithi123 bash[19119]: cluster 2024-02-12T16:01:54.084829+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:55.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:54 smithi123 bash[19119]: cluster 2024-02-12T16:01:54.084916+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:55.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:54 smithi121 bash[18489]: cluster 2024-02-12T16:01:53.497839+0000 mgr.y (mgr.14146) 482 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:55.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:54 smithi121 bash[18489]: cluster 2024-02-12T16:01:53.935161+0000 mon.a (mon.0) 784 : cluster [DBG] osdmap e125: 12 total, 12 up, 12 in 2024-02-12T16:01:55.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:54 smithi121 bash[18489]: cluster 2024-02-12T16:01:54.084829+0000 osd.2 (osd.2) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:55.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:54 smithi121 bash[18489]: cluster 2024-02-12T16:01:54.084916+0000 osd.2 (osd.2) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:55 smithi169 bash[19604]: cluster 2024-02-12T16:01:54.942526+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e126: 12 total, 12 up, 12 in 2024-02-12T16:01:56.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:55 smithi123 bash[19119]: cluster 2024-02-12T16:01:54.942526+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e126: 12 total, 12 up, 12 in 2024-02-12T16:01:56.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:55 smithi121 bash[18489]: cluster 2024-02-12T16:01:54.942526+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e126: 12 total, 12 up, 12 in 2024-02-12T16:01:56.887 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:01:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:56 smithi169 bash[19604]: cluster 2024-02-12T16:01:55.498461+0000 mgr.y (mgr.14146) 483 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:57.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:56 smithi123 bash[19119]: cluster 2024-02-12T16:01:55.498461+0000 mgr.y (mgr.14146) 483 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:57.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:56 smithi121 bash[18489]: cluster 2024-02-12T16:01:55.498461+0000 mgr.y (mgr.14146) 483 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:58 smithi169 bash[19604]: cluster 2024-02-12T16:01:57.499013+0000 mgr.y (mgr.14146) 484 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:58 smithi169 bash[19604]: cluster 2024-02-12T16:01:57.853300+0000 osd.9 (osd.9) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:58 smithi169 bash[19604]: cluster 2024-02-12T16:01:57.853411+0000 osd.9 (osd.9) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:01:58 smithi169 bash[19604]: cluster 2024-02-12T16:01:57.957043+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e127: 12 total, 12 up, 12 in 2024-02-12T16:01:59.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:58 smithi123 bash[19119]: cluster 2024-02-12T16:01:57.499013+0000 mgr.y (mgr.14146) 484 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:59.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:58 smithi123 bash[19119]: cluster 2024-02-12T16:01:57.853300+0000 osd.9 (osd.9) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:59.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:58 smithi123 bash[19119]: cluster 2024-02-12T16:01:57.853411+0000 osd.9 (osd.9) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:59.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:01:58 smithi123 bash[19119]: cluster 2024-02-12T16:01:57.957043+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e127: 12 total, 12 up, 12 in 2024-02-12T16:01:59.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:58 smithi121 bash[18489]: cluster 2024-02-12T16:01:57.499013+0000 mgr.y (mgr.14146) 484 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:01:59.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:58 smithi121 bash[18489]: cluster 2024-02-12T16:01:57.853300+0000 osd.9 (osd.9) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:01:59.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:58 smithi121 bash[18489]: cluster 2024-02-12T16:01:57.853411+0000 osd.9 (osd.9) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:01:59.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:01:58 smithi121 bash[18489]: cluster 2024-02-12T16:01:57.957043+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e127: 12 total, 12 up, 12 in 2024-02-12T16:02:00.039 INFO:teuthology.orchestra.run.smithi121.stdout:206158430296 2024-02-12T16:02:00.661 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430276 got 206158430296 for osd.6 2024-02-12T16:02:00.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-12T16:02:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:00 smithi169 bash[19604]: cluster 2024-02-12T16:01:59.499656+0000 mgr.y (mgr.14146) 485 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:00 smithi169 bash[19604]: audit 2024-02-12T16:02:00.030428+0000 mon.a (mon.0) 787 : audit [DBG] from='client.? v1:172.21.15.121:0/2617958866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:02:01.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:00 smithi123 bash[19119]: cluster 2024-02-12T16:01:59.499656+0000 mgr.y (mgr.14146) 485 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:01.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:00 smithi123 bash[19119]: audit 2024-02-12T16:02:00.030428+0000 mon.a (mon.0) 787 : audit [DBG] from='client.? v1:172.21.15.121:0/2617958866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:02:01.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:00 smithi121 bash[18489]: cluster 2024-02-12T16:01:59.499656+0000 mgr.y (mgr.14146) 485 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:01.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:00 smithi121 bash[18489]: audit 2024-02-12T16:02:00.030428+0000 mon.a (mon.0) 787 : audit [DBG] from='client.? v1:172.21.15.121:0/2617958866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:02:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:02 smithi169 bash[19604]: cluster 2024-02-12T16:02:01.500242+0000 mgr.y (mgr.14146) 486 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:03.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:02 smithi123 bash[19119]: cluster 2024-02-12T16:02:01.500242+0000 mgr.y (mgr.14146) 486 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:03.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:02 smithi121 bash[18489]: cluster 2024-02-12T16:02:01.500242+0000 mgr.y (mgr.14146) 486 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:04 smithi169 bash[19604]: cluster 2024-02-12T16:02:03.500834+0000 mgr.y (mgr.14146) 487 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:05.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:04 smithi123 bash[19119]: cluster 2024-02-12T16:02:03.500834+0000 mgr.y (mgr.14146) 487 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:05.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:04 smithi121 bash[18489]: cluster 2024-02-12T16:02:03.500834+0000 mgr.y (mgr.14146) 487 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:02:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:05 smithi169 bash[19604]: cluster 2024-02-12T16:02:04.721539+0000 osd.11 (osd.11) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:05 smithi169 bash[19604]: cluster 2024-02-12T16:02:04.721633+0000 osd.11 (osd.11) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:05 smithi169 bash[19604]: cluster 2024-02-12T16:02:04.975293+0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e128: 12 total, 12 up, 12 in 2024-02-12T16:02:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:05 smithi169 bash[19604]: cluster 2024-02-12T16:02:05.807951+0000 osd.7 (osd.7) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:05 smithi169 bash[19604]: cluster 2024-02-12T16:02:05.808046+0000 osd.7 (osd.7) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:06.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:05 smithi121 bash[18489]: cluster 2024-02-12T16:02:04.721539+0000 osd.11 (osd.11) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:06.381 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:05 smithi121 bash[18489]: cluster 2024-02-12T16:02:04.721633+0000 osd.11 (osd.11) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:06.381 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:05 smithi121 bash[18489]: cluster 2024-02-12T16:02:04.975293+0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e128: 12 total, 12 up, 12 in 2024-02-12T16:02:06.381 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:05 smithi121 bash[18489]: cluster 2024-02-12T16:02:05.807951+0000 osd.7 (osd.7) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:06.381 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:05 smithi121 bash[18489]: cluster 2024-02-12T16:02:05.808046+0000 osd.7 (osd.7) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:06.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:05 smithi123 bash[19119]: cluster 2024-02-12T16:02:04.721539+0000 osd.11 (osd.11) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:06.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:05 smithi123 bash[19119]: cluster 2024-02-12T16:02:04.721633+0000 osd.11 (osd.11) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:06.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:05 smithi123 bash[19119]: cluster 2024-02-12T16:02:04.975293+0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e128: 12 total, 12 up, 12 in 2024-02-12T16:02:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:05 smithi123 bash[19119]: cluster 2024-02-12T16:02:05.807951+0000 osd.7 (osd.7) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:06.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:05 smithi123 bash[19119]: cluster 2024-02-12T16:02:05.808046+0000 osd.7 (osd.7) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:06 smithi169 bash[19604]: cluster 2024-02-12T16:02:05.501418+0000 mgr.y (mgr.14146) 488 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:06 smithi169 bash[19604]: cluster 2024-02-12T16:02:06.171617+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e129: 12 total, 12 up, 12 in 2024-02-12T16:02:07.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:06 smithi123 bash[19119]: cluster 2024-02-12T16:02:05.501418+0000 mgr.y (mgr.14146) 488 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:07.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:06 smithi123 bash[19119]: cluster 2024-02-12T16:02:06.171617+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e129: 12 total, 12 up, 12 in 2024-02-12T16:02:07.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:06 smithi121 bash[18489]: cluster 2024-02-12T16:02:05.501418+0000 mgr.y (mgr.14146) 488 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:07.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:06 smithi121 bash[18489]: cluster 2024-02-12T16:02:06.171617+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e129: 12 total, 12 up, 12 in 2024-02-12T16:02:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:07 smithi169 bash[19604]: cluster 2024-02-12T16:02:07.240063+0000 osd.8 (osd.8) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:07 smithi169 bash[19604]: cluster 2024-02-12T16:02:07.240159+0000 osd.8 (osd.8) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:08.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:07 smithi121 bash[18489]: cluster 2024-02-12T16:02:07.240063+0000 osd.8 (osd.8) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:08.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:07 smithi121 bash[18489]: cluster 2024-02-12T16:02:07.240159+0000 osd.8 (osd.8) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:08.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:08 smithi123 bash[19119]: cluster 2024-02-12T16:02:07.240063+0000 osd.8 (osd.8) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:08.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:08 smithi123 bash[19119]: cluster 2024-02-12T16:02:07.240159+0000 osd.8 (osd.8) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:08.645 INFO:teuthology.orchestra.run.smithi121.stdout:240518168656 2024-02-12T16:02:09.283 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:08 smithi121 bash[18489]: cluster 2024-02-12T16:02:07.501995+0000 mgr.y (mgr.14146) 489 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:09.283 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:08 smithi121 bash[18489]: cluster 2024-02-12T16:02:07.990873+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e130: 12 total, 12 up, 12 in 2024-02-12T16:02:09.283 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:08 smithi121 bash[18489]: audit 2024-02-12T16:02:08.636729+0000 mon.a (mon.0) 791 : audit [DBG] from='client.? v1:172.21.15.121:0/201866014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:02:09.284 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168636 got 240518168656 for osd.7 2024-02-12T16:02:09.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-02-12T16:02:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:09 smithi169 bash[19604]: cluster 2024-02-12T16:02:07.501995+0000 mgr.y (mgr.14146) 489 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:09 smithi169 bash[19604]: cluster 2024-02-12T16:02:07.990873+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e130: 12 total, 12 up, 12 in 2024-02-12T16:02:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:09 smithi169 bash[19604]: audit 2024-02-12T16:02:08.636729+0000 mon.a (mon.0) 791 : audit [DBG] from='client.? v1:172.21.15.121:0/201866014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:02:09.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:09 smithi123 bash[19119]: cluster 2024-02-12T16:02:07.501995+0000 mgr.y (mgr.14146) 489 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:09 smithi123 bash[19119]: cluster 2024-02-12T16:02:07.990873+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e130: 12 total, 12 up, 12 in 2024-02-12T16:02:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:09 smithi123 bash[19119]: audit 2024-02-12T16:02:08.636729+0000 mon.a (mon.0) 791 : audit [DBG] from='client.? v1:172.21.15.121:0/201866014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:02:11.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:11 smithi169 bash[19604]: cluster 2024-02-12T16:02:09.502429+0000 mgr.y (mgr.14146) 490 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:11.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:11 smithi123 bash[19119]: cluster 2024-02-12T16:02:09.502429+0000 mgr.y (mgr.14146) 490 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:11.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:11 smithi121 bash[18489]: cluster 2024-02-12T16:02:09.502429+0000 mgr.y (mgr.14146) 490 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:13 smithi169 bash[19604]: cluster 2024-02-12T16:02:11.426955+0000 osd.4 (osd.4) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:13 smithi169 bash[19604]: cluster 2024-02-12T16:02:11.427042+0000 osd.4 (osd.4) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:13 smithi169 bash[19604]: cluster 2024-02-12T16:02:11.503000+0000 mgr.y (mgr.14146) 491 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:13 smithi169 bash[19604]: cluster 2024-02-12T16:02:12.023018+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e131: 12 total, 12 up, 12 in 2024-02-12T16:02:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:13 smithi169 bash[19604]: cluster 2024-02-12T16:02:12.706793+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:13.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:13 smithi169 bash[19604]: cluster 2024-02-12T16:02:12.706872+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:13.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:13 smithi123 bash[19119]: cluster 2024-02-12T16:02:11.426955+0000 osd.4 (osd.4) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:13 smithi123 bash[19119]: cluster 2024-02-12T16:02:11.427042+0000 osd.4 (osd.4) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:13 smithi123 bash[19119]: cluster 2024-02-12T16:02:11.503000+0000 mgr.y (mgr.14146) 491 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:13 smithi123 bash[19119]: cluster 2024-02-12T16:02:12.023018+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e131: 12 total, 12 up, 12 in 2024-02-12T16:02:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:13 smithi123 bash[19119]: cluster 2024-02-12T16:02:12.706793+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:13.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:13 smithi123 bash[19119]: cluster 2024-02-12T16:02:12.706872+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:13 smithi121 bash[18489]: cluster 2024-02-12T16:02:11.426955+0000 osd.4 (osd.4) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:13 smithi121 bash[18489]: cluster 2024-02-12T16:02:11.427042+0000 osd.4 (osd.4) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:13 smithi121 bash[18489]: cluster 2024-02-12T16:02:11.503000+0000 mgr.y (mgr.14146) 491 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:13 smithi121 bash[18489]: cluster 2024-02-12T16:02:12.023018+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e131: 12 total, 12 up, 12 in 2024-02-12T16:02:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:13 smithi121 bash[18489]: cluster 2024-02-12T16:02:12.706793+0000 osd.6 (osd.6) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:13 smithi121 bash[18489]: cluster 2024-02-12T16:02:12.706872+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:14.089 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:02:14.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:14 smithi169 bash[19604]: cluster 2024-02-12T16:02:13.034168+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e132: 12 total, 12 up, 12 in 2024-02-12T16:02:14.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:14 smithi123 bash[19119]: cluster 2024-02-12T16:02:13.034168+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e132: 12 total, 12 up, 12 in 2024-02-12T16:02:14.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:14 smithi121 bash[18489]: cluster 2024-02-12T16:02:13.034168+0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e132: 12 total, 12 up, 12 in 2024-02-12T16:02:15.238 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:15 smithi121 bash[18489]: cluster 2024-02-12T16:02:13.503628+0000 mgr.y (mgr.14146) 492 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:15.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:15 smithi123 bash[19119]: cluster 2024-02-12T16:02:13.503628+0000 mgr.y (mgr.14146) 492 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:15 smithi169 bash[19604]: cluster 2024-02-12T16:02:13.503628+0000 mgr.y (mgr.14146) 492 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:16 smithi169 bash[19604]: cluster 2024-02-12T16:02:15.504265+0000 mgr.y (mgr.14146) 493 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:16.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:16 smithi123 bash[19119]: cluster 2024-02-12T16:02:15.504265+0000 mgr.y (mgr.14146) 493 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:16.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:16 smithi121 bash[18489]: cluster 2024-02-12T16:02:15.504265+0000 mgr.y (mgr.14146) 493 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:17.151 INFO:teuthology.orchestra.run.smithi121.stdout:279172874313 2024-02-12T16:02:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:17 smithi169 bash[19604]: audit 2024-02-12T16:02:17.143285+0000 mon.a (mon.0) 794 : audit [DBG] from='client.? v1:172.21.15.121:0/1590089998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-12T16:02:17.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:17 smithi123 bash[19119]: audit 2024-02-12T16:02:17.143285+0000 mon.a (mon.0) 794 : audit [DBG] from='client.? v1:172.21.15.121:0/1590089998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-12T16:02:17.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:17 smithi121 bash[18489]: audit 2024-02-12T16:02:17.143285+0000 mon.a (mon.0) 794 : audit [DBG] from='client.? v1:172.21.15.121:0/1590089998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-12T16:02:17.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874293 got 279172874313 for osd.8 2024-02-12T16:02:17.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-02-12T16:02:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:18 smithi169 bash[19604]: cluster 2024-02-12T16:02:17.504851+0000 mgr.y (mgr.14146) 494 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:18.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:18 smithi123 bash[19119]: cluster 2024-02-12T16:02:17.504851+0000 mgr.y (mgr.14146) 494 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:18.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:18 smithi121 bash[18489]: cluster 2024-02-12T16:02:17.504851+0000 mgr.y (mgr.14146) 494 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:20.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:20 smithi123 bash[19119]: cluster 2024-02-12T16:02:19.505433+0000 mgr.y (mgr.14146) 495 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:20.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:20 smithi121 bash[18489]: cluster 2024-02-12T16:02:19.505433+0000 mgr.y (mgr.14146) 495 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:20 smithi169 bash[19604]: cluster 2024-02-12T16:02:19.505433+0000 mgr.y (mgr.14146) 495 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:21.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:21 smithi123 bash[19119]: audit 2024-02-12T16:02:21.509676+0000 mon.a (mon.0) 795 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:02:21.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:21 smithi123 bash[19119]: audit 2024-02-12T16:02:21.510078+0000 mon.a (mon.0) 796 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:02:21.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:21 smithi123 bash[19119]: audit 2024-02-12T16:02:21.510468+0000 mon.a (mon.0) 797 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:02:21.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:21 smithi123 bash[19119]: audit 2024-02-12T16:02:21.556305+0000 mon.a (mon.0) 798 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:02:21.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:21 smithi123 bash[19119]: audit 2024-02-12T16:02:21.559724+0000 mon.a (mon.0) 799 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:02:21.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:21 smithi121 bash[18489]: audit 2024-02-12T16:02:21.509676+0000 mon.a (mon.0) 795 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:02:21.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:21 smithi121 bash[18489]: audit 2024-02-12T16:02:21.510078+0000 mon.a (mon.0) 796 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:02:21.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:21 smithi121 bash[18489]: audit 2024-02-12T16:02:21.510468+0000 mon.a (mon.0) 797 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:02:21.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:21 smithi121 bash[18489]: audit 2024-02-12T16:02:21.556305+0000 mon.a (mon.0) 798 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:02:21.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:21 smithi121 bash[18489]: audit 2024-02-12T16:02:21.559724+0000 mon.a (mon.0) 799 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:02:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:21 smithi169 bash[19604]: audit 2024-02-12T16:02:21.509676+0000 mon.a (mon.0) 795 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:02:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:21 smithi169 bash[19604]: audit 2024-02-12T16:02:21.510078+0000 mon.a (mon.0) 796 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:02:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:21 smithi169 bash[19604]: audit 2024-02-12T16:02:21.510468+0000 mon.a (mon.0) 797 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:02:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:21 smithi169 bash[19604]: audit 2024-02-12T16:02:21.556305+0000 mon.a (mon.0) 798 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:02:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:21 smithi169 bash[19604]: audit 2024-02-12T16:02:21.559724+0000 mon.a (mon.0) 799 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:02:22.570 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:02:22.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:22 smithi123 bash[19119]: cluster 2024-02-12T16:02:21.458719+0000 osd.0 (osd.0) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:22 smithi123 bash[19119]: cluster 2024-02-12T16:02:21.458821+0000 osd.0 (osd.0) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:22 smithi123 bash[19119]: audit 2024-02-12T16:02:21.562141+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:02:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:22 smithi123 bash[19119]: cluster 2024-02-12T16:02:21.568259+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e133: 12 total, 12 up, 12 in 2024-02-12T16:02:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:22 smithi123 bash[19119]: audit 2024-02-12T16:02:21.580755+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:02:22.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:22 smithi121 bash[18489]: cluster 2024-02-12T16:02:21.458719+0000 osd.0 (osd.0) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:22.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:22 smithi121 bash[18489]: cluster 2024-02-12T16:02:21.458821+0000 osd.0 (osd.0) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:22.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:22 smithi121 bash[18489]: audit 2024-02-12T16:02:21.562141+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:02:22.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:22 smithi121 bash[18489]: cluster 2024-02-12T16:02:21.568259+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e133: 12 total, 12 up, 12 in 2024-02-12T16:02:22.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:22 smithi121 bash[18489]: audit 2024-02-12T16:02:21.580755+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:02:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:22 smithi169 bash[19604]: cluster 2024-02-12T16:02:21.458719+0000 osd.0 (osd.0) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:22 smithi169 bash[19604]: cluster 2024-02-12T16:02:21.458821+0000 osd.0 (osd.0) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:22 smithi169 bash[19604]: audit 2024-02-12T16:02:21.562141+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:02:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:22 smithi169 bash[19604]: cluster 2024-02-12T16:02:21.568259+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e133: 12 total, 12 up, 12 in 2024-02-12T16:02:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:22 smithi169 bash[19604]: audit 2024-02-12T16:02:21.580755+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:02:24.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:24 smithi123 bash[19119]: cluster 2024-02-12T16:02:23.506571+0000 mgr.y (mgr.14146) 497 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:24.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:24 smithi121 bash[18489]: cluster 2024-02-12T16:02:23.506571+0000 mgr.y (mgr.14146) 497 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:24 smithi169 bash[19604]: cluster 2024-02-12T16:02:23.506571+0000 mgr.y (mgr.14146) 497 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:25.451 INFO:teuthology.orchestra.run.smithi121.stdout:317827579972 2024-02-12T16:02:25.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:25 smithi123 bash[19119]: cluster 2024-02-12T16:02:23.810946+0000 osd.5 (osd.5) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:25 smithi123 bash[19119]: cluster 2024-02-12T16:02:23.811058+0000 osd.5 (osd.5) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:25 smithi123 bash[19119]: cluster 2024-02-12T16:02:24.589451+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e134: 12 total, 12 up, 12 in 2024-02-12T16:02:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:25 smithi123 bash[19119]: audit 2024-02-12T16:02:25.443489+0000 mon.a (mon.0) 804 : audit [DBG] from='client.? v1:172.21.15.121:0/65985957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-12T16:02:25.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:25 smithi121 bash[18489]: cluster 2024-02-12T16:02:23.810946+0000 osd.5 (osd.5) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:25.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:25 smithi121 bash[18489]: cluster 2024-02-12T16:02:23.811058+0000 osd.5 (osd.5) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:25.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:25 smithi121 bash[18489]: cluster 2024-02-12T16:02:24.589451+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e134: 12 total, 12 up, 12 in 2024-02-12T16:02:25.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:25 smithi121 bash[18489]: audit 2024-02-12T16:02:25.443489+0000 mon.a (mon.0) 804 : audit [DBG] from='client.? v1:172.21.15.121:0/65985957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-12T16:02:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:25 smithi169 bash[19604]: cluster 2024-02-12T16:02:23.810946+0000 osd.5 (osd.5) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:25 smithi169 bash[19604]: cluster 2024-02-12T16:02:23.811058+0000 osd.5 (osd.5) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:25 smithi169 bash[19604]: cluster 2024-02-12T16:02:24.589451+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e134: 12 total, 12 up, 12 in 2024-02-12T16:02:26.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:25 smithi169 bash[19604]: audit 2024-02-12T16:02:25.443489+0000 mon.a (mon.0) 804 : audit [DBG] from='client.? v1:172.21.15.121:0/65985957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-12T16:02:26.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579952 got 317827579972 for osd.9 2024-02-12T16:02:26.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-02-12T16:02:26.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:26 smithi123 bash[19119]: cluster 2024-02-12T16:02:25.507151+0000 mgr.y (mgr.14146) 498 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:26.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:26 smithi121 bash[18489]: cluster 2024-02-12T16:02:25.507151+0000 mgr.y (mgr.14146) 498 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:26 smithi169 bash[19604]: cluster 2024-02-12T16:02:25.507151+0000 mgr.y (mgr.14146) 498 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:28.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:28 smithi123 bash[19119]: cluster 2024-02-12T16:02:27.507852+0000 mgr.y (mgr.14146) 499 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:28.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:28 smithi121 bash[18489]: cluster 2024-02-12T16:02:27.507852+0000 mgr.y (mgr.14146) 499 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:28 smithi169 bash[19604]: cluster 2024-02-12T16:02:27.507852+0000 mgr.y (mgr.14146) 499 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:30 smithi123 bash[19119]: cluster 2024-02-12T16:02:29.508502+0000 mgr.y (mgr.14146) 500 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:30.903 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:02:30.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:30 smithi121 bash[18489]: cluster 2024-02-12T16:02:29.508502+0000 mgr.y (mgr.14146) 500 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:30 smithi169 bash[19604]: cluster 2024-02-12T16:02:29.508502+0000 mgr.y (mgr.14146) 500 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:32.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:32 smithi123 bash[19119]: cluster 2024-02-12T16:02:31.509075+0000 mgr.y (mgr.14146) 501 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:32.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:32 smithi121 bash[18489]: cluster 2024-02-12T16:02:31.509075+0000 mgr.y (mgr.14146) 501 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:32 smithi169 bash[19604]: cluster 2024-02-12T16:02:31.509075+0000 mgr.y (mgr.14146) 501 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:33.980 INFO:teuthology.orchestra.run.smithi121.stdout:356482285628 2024-02-12T16:02:34.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285608 got 356482285628 for osd.10 2024-02-12T16:02:34.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-02-12T16:02:34.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:34 smithi123 bash[19119]: cluster 2024-02-12T16:02:33.509702+0000 mgr.y (mgr.14146) 502 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:34.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:34 smithi123 bash[19119]: audit 2024-02-12T16:02:33.972436+0000 mon.a (mon.0) 805 : audit [DBG] from='client.? v1:172.21.15.121:0/2765614775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-12T16:02:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:34 smithi121 bash[18489]: cluster 2024-02-12T16:02:33.509702+0000 mgr.y (mgr.14146) 502 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:34 smithi121 bash[18489]: audit 2024-02-12T16:02:33.972436+0000 mon.a (mon.0) 805 : audit [DBG] from='client.? v1:172.21.15.121:0/2765614775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-12T16:02:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:34 smithi169 bash[19604]: cluster 2024-02-12T16:02:33.509702+0000 mgr.y (mgr.14146) 502 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:34 smithi169 bash[19604]: audit 2024-02-12T16:02:33.972436+0000 mon.a (mon.0) 805 : audit [DBG] from='client.? v1:172.21.15.121:0/2765614775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-12T16:02:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:36 smithi169 bash[19604]: cluster 2024-02-12T16:02:35.510358+0000 mgr.y (mgr.14146) 503 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:36.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:36 smithi123 bash[19119]: cluster 2024-02-12T16:02:35.510358+0000 mgr.y (mgr.14146) 503 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:36.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:36 smithi121 bash[18489]: cluster 2024-02-12T16:02:35.510358+0000 mgr.y (mgr.14146) 503 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:38.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:38 smithi123 bash[19119]: cluster 2024-02-12T16:02:37.511018+0000 mgr.y (mgr.14146) 504 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:38.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:38 smithi121 bash[18489]: cluster 2024-02-12T16:02:37.511018+0000 mgr.y (mgr.14146) 504 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:38 smithi169 bash[19604]: cluster 2024-02-12T16:02:37.511018+0000 mgr.y (mgr.14146) 504 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:39.397 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:02:40.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:40 smithi121 bash[18489]: cluster 2024-02-12T16:02:39.511675+0000 mgr.y (mgr.14146) 505 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:40.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:40 smithi123 bash[19119]: cluster 2024-02-12T16:02:39.511675+0000 mgr.y (mgr.14146) 505 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:40 smithi169 bash[19604]: cluster 2024-02-12T16:02:39.511675+0000 mgr.y (mgr.14146) 505 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:42.284 INFO:teuthology.orchestra.run.smithi121.stdout:399431958582 2024-02-12T16:02:42.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:42 smithi121 bash[18489]: cluster 2024-02-12T16:02:41.512261+0000 mgr.y (mgr.14146) 506 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:42.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:42 smithi121 bash[18489]: audit 2024-02-12T16:02:42.276376+0000 mon.a (mon.0) 806 : audit [DBG] from='client.? v1:172.21.15.121:0/3876234610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-12T16:02:42.875 INFO:tasks.cephadm.ceph_manager.ceph:need seq 399431958562 got 399431958582 for osd.11 2024-02-12T16:02:42.875 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-12T16:02:42.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:02:42.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:42 smithi123 bash[19119]: cluster 2024-02-12T16:02:41.512261+0000 mgr.y (mgr.14146) 506 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:42 smithi123 bash[19119]: audit 2024-02-12T16:02:42.276376+0000 mon.a (mon.0) 806 : audit [DBG] from='client.? v1:172.21.15.121:0/3876234610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-12T16:02:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:42 smithi169 bash[19604]: cluster 2024-02-12T16:02:41.512261+0000 mgr.y (mgr.14146) 506 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:42 smithi169 bash[19604]: audit 2024-02-12T16:02:42.276376+0000 mon.a (mon.0) 806 : audit [DBG] from='client.? v1:172.21.15.121:0/3876234610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-12T16:02:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:44 smithi123 bash[19119]: cluster 2024-02-12T16:02:43.512862+0000 mgr.y (mgr.14146) 507 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:44.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:44 smithi121 bash[18489]: cluster 2024-02-12T16:02:43.512862+0000 mgr.y (mgr.14146) 507 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:44 smithi169 bash[19604]: cluster 2024-02-12T16:02:43.512862+0000 mgr.y (mgr.14146) 507 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:46 smithi169 bash[19604]: cluster 2024-02-12T16:02:45.513515+0000 mgr.y (mgr.14146) 508 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:46.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:46 smithi123 bash[19119]: cluster 2024-02-12T16:02:45.513515+0000 mgr.y (mgr.14146) 508 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:46.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:46 smithi121 bash[18489]: cluster 2024-02-12T16:02:45.513515+0000 mgr.y (mgr.14146) 508 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:47 smithi169 bash[19604]: cluster 2024-02-12T16:02:46.533313+0000 osd.1 (osd.1) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:47 smithi169 bash[19604]: cluster 2024-02-12T16:02:46.533448+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:47.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:47 smithi123 bash[19119]: cluster 2024-02-12T16:02:46.533313+0000 osd.1 (osd.1) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:47.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:47 smithi123 bash[19119]: cluster 2024-02-12T16:02:46.533448+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:47.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:47 smithi121 bash[18489]: cluster 2024-02-12T16:02:46.533313+0000 osd.1 (osd.1) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:02:47.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:47 smithi121 bash[18489]: cluster 2024-02-12T16:02:46.533448+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:02:47.674 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:02:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:48 smithi169 bash[19604]: cluster 2024-02-12T16:02:47.181019+0000 mon.a (mon.0) 807 : cluster [DBG] osdmap e135: 12 total, 12 up, 12 in 2024-02-12T16:02:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:48 smithi169 bash[19604]: cluster 2024-02-12T16:02:47.514094+0000 mgr.y (mgr.14146) 509 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:48.562 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:48 smithi121 bash[18489]: cluster 2024-02-12T16:02:47.181019+0000 mon.a (mon.0) 807 : cluster [DBG] osdmap e135: 12 total, 12 up, 12 in 2024-02-12T16:02:48.562 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:48 smithi121 bash[18489]: cluster 2024-02-12T16:02:47.514094+0000 mgr.y (mgr.14146) 509 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:48.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:48 smithi123 bash[19119]: cluster 2024-02-12T16:02:47.181019+0000 mon.a (mon.0) 807 : cluster [DBG] osdmap e135: 12 total, 12 up, 12 in 2024-02-12T16:02:48.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:48 smithi123 bash[19119]: cluster 2024-02-12T16:02:47.514094+0000 mgr.y (mgr.14146) 509 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:02:50.559 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:02:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:50 smithi123 bash[19119]: cluster 2024-02-12T16:02:49.514712+0000 mgr.y (mgr.14146) 510 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:50 smithi123 bash[19119]: cluster 2024-02-12T16:02:49.546650+0000 osd.7 (osd.7) 25 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:02:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:50 smithi123 bash[19119]: cluster 2024-02-12T16:02:49.550612+0000 osd.7 (osd.7) 26 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:02:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:50 smithi121 bash[18489]: cluster 2024-02-12T16:02:49.514712+0000 mgr.y (mgr.14146) 510 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:50 smithi121 bash[18489]: cluster 2024-02-12T16:02:49.546650+0000 osd.7 (osd.7) 25 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:02:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:50 smithi121 bash[18489]: cluster 2024-02-12T16:02:49.550612+0000 osd.7 (osd.7) 26 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:02:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:50 smithi169 bash[19604]: cluster 2024-02-12T16:02:49.514712+0000 mgr.y (mgr.14146) 510 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:50 smithi169 bash[19604]: cluster 2024-02-12T16:02:49.546650+0000 osd.7 (osd.7) 25 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:02:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:50 smithi169 bash[19604]: cluster 2024-02-12T16:02:49.550612+0000 osd.7 (osd.7) 26 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:02:51.174 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2024-02-12T16:02:49.514400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":3573792,"kb_used_data":8640,"kb_used_omap":0,"kb_used_meta":3565056,"kb_avail":1121368032,"statfs":{"total":1151940427776,"available":1148280864768,"internally_reserved":0,"allocated":8847360,"data_stored":3972628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3650617344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":134,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-02-12T16:02:25.436656+0000","last_change":"2024-02-12T16:01:44.465700+0000","last_active":"2024-02-12T16:02:25.436656+0000","last_peered":"2024-02-12T16:02:25.436656+0000","last_clean":"2024-02-12T16:02:25.436656+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:02:25.436656+0000","last_undegraded":"2024-02-12T16:02:25.436656+0000","last_fullsized":"2024-02-12T16:02:25.436656+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:01:44.465493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:00:34.507052+0000","last_clean_scrub_stamp":"2024-02-12T16:01:44.465493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958583,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0409999999999999,"15min":1.0389999999999999},"min":{"1min":0.71899999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.482,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.089,"15min":1.123},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.4379999999999999,"5min":2.871,"15min":2.871},"last":1.206}]},{"osd":1,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.048,"15min":1.1020000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1990000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0780000000000001,"15min":1.0880000000000001},"min":{"1min":0.73199999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5920000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.349}]},{"osd":2,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.119,"15min":1.157},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.466,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.1519999999999999,"15min":1.24},"min":{"1min":0.77200000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3210000000000002,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.1279999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1739999999999999,"15min":1.2190000000000001},"min":{"1min":0.92600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6839999999999999,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":1.085},{"interface":"front","average":{"1min":1.032,"5min":1.151,"15min":1.2070000000000001},"min":{"1min":0.75800000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.393,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":1.2729999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2749999999999999,"15min":1.3720000000000001},"min":{"1min":0.81200000000000006,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4770000000000001,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.355,"15min":1.4390000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8400000000000001,"5min":4.0789999999999997,"15min":4.0789999999999997},"last":1.47}]},{"osd":5,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.421,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.754,"5min":3.044,"15min":3.044},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.423,"15min":1.5069999999999999},"min":{"1min":0.93500000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.6279999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":1.0669999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4379999999999999,"15min":1.5349999999999999},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.827,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.5349999999999999,"15min":1.619},"min":{"1min":1.038,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9279999999999999,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.534,"15min":1.6259999999999999},"min":{"1min":0.90000000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.8779999999999999,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.635},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6259999999999999,"15min":1.722},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.921,"5min":3.8370000000000002,"15min":3.8370000000000002},"last":1.595}]},{"osd":8,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.569,"15min":1.6519999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.974,"5min":4,"15min":4},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.647,"15min":1.742},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.9490000000000001,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.647}]},{"osd":10,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.5700000000000001,"15min":1.694},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.772,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.411},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.742,"15min":1.8360000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0230000000000001,"5min":4.3949999999999996,"15min":4.3949999999999996},"last":1.724}]}]},{"osd":10,"up_from":83,"seq":356482285632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.88800000000000001,"15min":0.89200000000000002},"min":{"1min":0.66100000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1779999999999999,"5min":1.413,"15min":1.413},"last":1.002},{"interface":"front","average":{"1min":1.044,"5min":1.036,"15min":1.0700000000000001},"min":{"1min":0.72499999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.377,"5min":1.573,"15min":1.573},"last":1.1040000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.995,"15min":0.98099999999999998},"min":{"1min":0.70399999999999996,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.3049999999999999,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.052,"5min":0.995,"15min":0.96499999999999997},"min":{"1min":0.72699999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3069999999999999,"5min":1.492,"15min":1.492},"last":0.75600000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0229999999999999,"15min":1.0109999999999999},"min":{"1min":0.73399999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0049999999999999,"15min":1.0169999999999999},"min":{"1min":0.76500000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.256,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.81799999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.1930000000000001,"15min":1.1180000000000001},"min":{"1min":0.88200000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5860000000000001,"5min":1.647,"15min":1.647},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.242,"15min":1.234},"min":{"1min":0.84799999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.5209999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":0.77600000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1739999999999999,"15min":1.1839999999999999},"min":{"1min":0.89600000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.464,"5min":2.048,"15min":2.048},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3260000000000001,"15min":1.3080000000000001},"min":{"1min":1.038,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.5129999999999999,"5min":1.788,"15min":1.788},"last":0.85899999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.222,"15min":1.23},"min":{"1min":0.91300000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.5089999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.413,"15min":1.357},"min":{"1min":1.113,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6120000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.4059999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.351,"15min":1.2909999999999999},"min":{"1min":1.1499999999999999,"5min":0.755,"15min":0.755},"max":{"1min":1.665,"5min":1.764,"15min":1.764},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.3799999999999999,"15min":1.3819999999999999},"min":{"1min":1.0580000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.5429999999999999,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.3759999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.3999999999999999,"15min":1.351},"min":{"1min":0.96899999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.891,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4450000000000001,"15min":1.351},"min":{"1min":1.1499999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8160000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.496}]},{"osd":9,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.5189999999999999,"15min":1.4510000000000001},"min":{"1min":1.2589999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.7370000000000001,"5min":2.02,"15min":2.02},"last":1.454},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.502,"15min":1.4179999999999999},"min":{"1min":1.321,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7689999999999999,"5min":1.952,"15min":1.952},"last":1.4219999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.595,"15min":1.6100000000000001},"min":{"1min":1.2989999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.482},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.6180000000000001,"15min":1.6379999999999999},"min":{"1min":1.2989999999999999,"5min":1.25,"15min":1.25},"max":{"1min":1.919,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.5469999999999999}]}]},{"osd":9,"up_from":74,"seq":317827579977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.95199999999999996,"15min":0.93600000000000005},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.218,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.95899999999999996,"15min":0.97799999999999998},"min":{"1min":0.72799999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.347,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":0.998},"min":{"1min":0.71899999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.399,"5min":1.48,"15min":1.48},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0589999999999999,"15min":1.0369999999999999},"min":{"1min":0.79100000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.427,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.86899999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95499999999999996,"15min":0.93300000000000005},"min":{"1min":0.69099999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.343,"5min":1.738,"15min":1.738},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":1.0309999999999999},"min":{"1min":0.73099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3049999999999999,"5min":1.601,"15min":1.601},"last":0.71799999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.98799999999999999,"15min":0.97199999999999998},"min":{"1min":0.629,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.371,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.1180000000000001,"15min":1.1100000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5029999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.85399999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1919999999999999,"15min":1.177},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6739999999999999,"5min":2.113,"15min":2.113},"last":1.012},{"interface":"front","average":{"1min":1.165,"5min":1.1930000000000001,"15min":1.1919999999999999},"min":{"1min":0.78500000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5680000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2350000000000001,"15min":1.2270000000000001},"min":{"1min":0.85199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.756,"5min":2.4399999999999999,"15min":2.4399999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.296,"15min":1.264},"min":{"1min":0.92600000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.661,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.4259999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.325},"min":{"1min":1.0960000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.5820000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3640000000000001,"15min":1.3600000000000001},"min":{"1min":0.93400000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.573,"5min":2.48,"15min":2.48},"last":1.4079999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3149999999999999,"15min":1.3260000000000001},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.575,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":1.121},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4359999999999999,"15min":1.411},"min":{"1min":1.2350000000000001,"5min":0.875,"15min":0.875},"max":{"1min":1.7609999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.3460000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.407,"15min":1.3720000000000001},"min":{"1min":0.91900000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.792,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.3939999999999999,"15min":1.3600000000000001},"min":{"1min":0.93700000000000006,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.706,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":1.3160000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4490000000000001,"15min":1.4350000000000001},"min":{"1min":1.01,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7769999999999999,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.228},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.383,"15min":1.351},"min":{"1min":0.85899999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.8300000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.379}]}]},{"osd":8,"up_from":65,"seq":279172874320,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95999999999999996,"15min":0.85599999999999998},"min":{"1min":0.61699999999999999,"5min":0.57899999999999996,"15min":0.57699999999999996},"max":{"1min":1.1839999999999999,"5min":1.488,"15min":1.488},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.96499999999999997,"15min":0.89500000000000002},"min":{"1min":0.71699999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.589,"5min":1.623,"15min":1.623},"last":0.83199999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.88800000000000001,"15min":0.84999999999999998},"min":{"1min":0.64900000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.3480000000000001,"5min":1.508,"15min":1.508},"last":0.878},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0249999999999999,"15min":0.95599999999999996},"min":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.355,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.94099999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.01,"15min":0.88},"min":{"1min":0.68700000000000006,"5min":0.59499999999999997,"15min":0.59199999999999997},"max":{"1min":1.194,"5min":1.764,"15min":1.764},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.173,"5min":1.141,"15min":1.0309999999999999},"min":{"1min":0.91400000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.637,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.339}]},{"osd":3,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.083,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.64800000000000002,"15min":0.60499999999999998},"max":{"1min":1.365,"5min":1.698,"15min":1.698},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.22,"15min":1.04},"min":{"1min":0.96099999999999997,"5min":0.623,"15min":0.623},"max":{"1min":1.3280000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.0029999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2629999999999999,"15min":1.157},"min":{"1min":0.94599999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.4339999999999999,"5min":1.677,"15min":1.839},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.3080000000000001,"15min":1.1160000000000001},"min":{"1min":0.80800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.772,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.252}]},{"osd":5,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1990000000000001,"15min":1.155},"min":{"1min":0.76900000000000002,"5min":0.76700000000000002,"15min":0.75800000000000001},"max":{"1min":1.4379999999999999,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.3420000000000001,"15min":1.2030000000000001},"min":{"1min":1.157,"5min":0.71899999999999997,"15min":0.68899999999999995},"max":{"1min":1.508,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":1.4470000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.1970000000000001},"min":{"1min":0.92800000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.915,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.115},{"interface":"front","average":{"1min":1.429,"5min":1.3939999999999999,"15min":1.25},"min":{"1min":1.073,"5min":0.91500000000000004,"15min":0.876},"max":{"1min":1.5549999999999999,"5min":2.339,"15min":2.339},"last":1.5109999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4530000000000001,"15min":1.2629999999999999},"min":{"1min":1.0860000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6120000000000001,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.272},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4930000000000001,"15min":1.292},"min":{"1min":1.194,"5min":0.997,"15min":0.85499999999999998},"max":{"1min":1.7909999999999999,"5min":1.984,"15min":1.984},"last":1.593}]},{"osd":9,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4610000000000001,"15min":1.2470000000000001},"min":{"1min":1.0680000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.8620000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3380000000000001,"15min":1.218},"min":{"1min":0.96899999999999997,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7330000000000001,"5min":2.456,"15min":2.456},"last":1.1839999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:01:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4039999999999999,"15min":1.3620000000000001},"min":{"1min":0.98699999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.518,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.498,"15min":1.4610000000000001},"min":{"1min":1.298,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.905,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.538}]}]},{"osd":7,"up_from":56,"seq":240518168665,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0349999999999999,"15min":0.98299999999999998},"min":{"1min":0.67500000000000004,"5min":0.436,"15min":0.436},"max":{"1min":1.3140000000000001,"5min":1.675,"15min":1.675},"last":1.349},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.95999999999999996,"15min":0.96199999999999997},"min":{"1min":0.64100000000000001,"5min":0.55100000000000005,"15min":0.53400000000000003},"max":{"1min":1.274,"5min":1.353,"15min":1.353},"last":0.99199999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.98299999999999998,"15min":1.0269999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2849999999999999,"5min":1.857,"15min":1.8580000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.046,"5min":1.0549999999999999,"15min":1.042},"min":{"1min":0.74099999999999999,"5min":0.58299999999999996,"15min":0.56499999999999995},"max":{"1min":1.4159999999999999,"5min":1.6279999999999999,"15min":1.794},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0780000000000001,"15min":1.0820000000000001},"min":{"1min":0.71099999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.635,"5min":1.635,"15min":2.0049999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.01,"5min":1.05,"15min":1.1000000000000001},"min":{"1min":0.69899999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4390000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.2090000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.95699999999999996,"15min":1.0129999999999999},"min":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.51,"5min":1.51,"15min":2.0449999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0569999999999999,"15min":1.04},"min":{"1min":0.71399999999999997,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.49,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.0860000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.095,"15min":1.071},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.60099999999999998},"max":{"1min":1.5960000000000001,"5min":1.76,"15min":1.76},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.149,"15min":1.1850000000000001},"min":{"1min":0.77300000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.677,"5min":1.677,"15min":2.1709999999999998},"last":1.278}]},{"osd":6,"last update":"Mon Feb 12 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.167,"15min":1.204},"min":{"1min":0.77300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.54,"5min":1.667,"15min":1.8069999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.226},"min":{"1min":0.86499999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.552,"5min":2.0960000000000001,"15min":2.1139999999999999},"last":1.327}]},{"osd":8,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":1.1910000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.5},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4099999999999999,"15min":1.446},"min":{"1min":1.028,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8180000000000001,"5min":1.905,"15min":2.0760000000000001},"last":1.6339999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.4319999999999999,"15min":1.4330000000000001},"min":{"1min":1.032,"5min":0.751,"15min":0.751},"max":{"1min":1.8420000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.427,"15min":1.421},"min":{"1min":0.98599999999999999,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":1.712,"5min":1.944,"15min":1.944},"last":1.4199999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.502,"15min":1.534},"min":{"1min":1.1619999999999999,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.911,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.835},{"interface":"front","average":{"1min":1.456,"5min":1.5109999999999999,"15min":1.5620000000000001},"min":{"1min":1.101,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":1.986,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.615}]},{"osd":11,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.55,"15min":1.5820000000000001},"min":{"1min":1.169,"5min":1.071,"15min":1.071},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.7290000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.522,"15min":1.5209999999999999},"min":{"1min":1.087,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7849999999999999,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.7090000000000001}]}]},{"osd":6,"up_from":48,"seq":206158430306,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.997,"15min":0.94899999999999995},"min":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.498},"max":{"1min":1.3460000000000001,"5min":1.492,"15min":1.492},"last":1.335},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":0.92500000000000004},"min":{"1min":0.72099999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.254,"5min":1.573,"15min":1.573},"last":1.1679999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.81100000000000005,"5min":0.67200000000000004,"15min":0.64300000000000002},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.0429999999999999,"15min":0.998},"min":{"1min":0.64000000000000001,"5min":0.62,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.93100000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0589999999999999,"15min":0.96799999999999997},"min":{"1min":0.76100000000000001,"5min":0.66000000000000003,"15min":0.55500000000000005},"max":{"1min":1.3129999999999999,"5min":2.383,"15min":2.383},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.113,"5min":1.109,"15min":1.004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.98299999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.032,"15min":0.95399999999999996},"min":{"1min":0.75700000000000001,"5min":0.68400000000000005,"15min":0.68100000000000005},"max":{"1min":1.3799999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.095,"15min":1.042},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.363,"5min":1.623,"15min":1.623},"last":1.4650000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.038,"15min":0.91000000000000003},"min":{"1min":0.68500000000000005,"5min":0.57199999999999995,"15min":0.41899999999999998},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.42299999999999999},"max":{"1min":1.554,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.264}]},{"osd":5,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.202,"15min":1.0069999999999999},"min":{"1min":0.79200000000000004,"5min":0.60299999999999998,"15min":0.46999999999999997},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.214,"15min":1.034},"min":{"1min":0.82999999999999996,"5min":0.69499999999999995,"15min":0.51800000000000002},"max":{"1min":1.538,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.597}]},{"osd":7,"last update":"Mon Feb 12 16:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.321,"15min":1.1890000000000001},"min":{"1min":0.875,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8839999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.337,"15min":1.2729999999999999},"min":{"1min":1.147,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.631,"5min":1.889,"15min":1.889},"last":1.502}]},{"osd":8,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.502,"15min":1.397},"min":{"1min":1.0960000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.534,"15min":1.4259999999999999},"min":{"1min":1.083,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.0739999999999998,"5min":2.093,"15min":2.093},"last":1.7390000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:02:09 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.5629999999999999,"15min":1.4730000000000001},"min":{"1min":1.3320000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.96},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.4870000000000001,"15min":1.407},"min":{"1min":1.2929999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.125,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":2.0150000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6359999999999999,"5min":1.6299999999999999,"15min":1.6180000000000001},"min":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1339999999999999},"max":{"1min":1.9350000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":2.0409999999999999},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.5740000000000001,"15min":1.5629999999999999},"min":{"1min":1.3640000000000001,"5min":1.081,"15min":1.081},"max":{"1min":2.008,"5min":2.367,"15min":2.367},"last":1.853}]}]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84399999999999997,"15min":0.66500000000000004},"min":{"1min":0.61499999999999999,"5min":0.41099999999999998,"15min":0.377},"max":{"1min":1.1160000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.93899999999999995,"15min":0.73099999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.28899999999999998},"max":{"1min":1.345,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.94499999999999995,"15min":0.75900000000000001},"min":{"1min":0.63800000000000001,"5min":0.50700000000000001,"15min":0.379},"max":{"1min":1.252,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96799999999999997,"15min":0.79100000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.34999999999999998},"max":{"1min":1.2669999999999999,"5min":1.54,"15min":1.54},"last":1.0780000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:02:16 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98999999999999999,"15min":0.83799999999999997},"min":{"1min":0.71299999999999997,"5min":0.61799999999999999,"15min":0.61099999999999999},"max":{"1min":1.3,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0389999999999999,"15min":0.90000000000000002},"min":{"1min":0.72899999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.54,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":1.3700000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.155,"15min":1.0309999999999999},"min":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.232,"15min":1.1060000000000001},"min":{"1min":0.77100000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.534,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.143}]},{"osd":5,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3160000000000001,"15min":1.1439999999999999},"min":{"1min":1.034,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.302},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4390000000000001,"15min":1.234},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":1.5600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.327,"15min":1.2170000000000001},"min":{"1min":1.073,"5min":0.81399999999999995,"15min":0.80500000000000005},"max":{"1min":1.6439999999999999,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.246},{"interface":"front","average":{"1min":1.512,"5min":1.486,"15min":1.3280000000000001},"min":{"1min":1.0640000000000001,"5min":0.872,"15min":0.872},"max":{"1min":2.0739999999999998,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.194}]},{"osd":7,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.472,"15min":1.3819999999999999},"min":{"1min":0.99199999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.0139999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.472},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.3779999999999999,"15min":1.3400000000000001},"min":{"1min":1.034,"5min":0.871,"15min":0.871},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.2769999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.49,"15min":1.403},"min":{"1min":1.0069999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8049999999999999,"5min":2.355,"15min":2.355},"last":1.452},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6479999999999999,"15min":1.506},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1349999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.7210000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.653,"15min":1.635},"min":{"1min":1.196,"5min":1.125,"15min":1.125},"max":{"1min":2.3410000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.518,"15min":1.534},"min":{"1min":1.21,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.4390000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.548,"15min":1.571},"min":{"1min":1.175,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.712,"5min":1.72,"15min":1.756},"min":{"1min":1.2490000000000001,"5min":1.089,"15min":1.089},"max":{"1min":2.238,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.8029999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738514,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81000000000000005,"15min":0.65200000000000002},"min":{"1min":0.57599999999999996,"5min":0.44900000000000001,"15min":0.32500000000000001},"max":{"1min":1.0620000000000001,"5min":1.2350000000000001,"15min":1.3009999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.84199999999999997,"15min":0.68700000000000006},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.34599999999999997},"max":{"1min":1.0069999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.754}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.873,"15min":0.76200000000000001},"min":{"1min":0.65300000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.2390000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.93300000000000005,"15min":0.86499999999999999},"min":{"1min":0.61099999999999999,"5min":0.53500000000000003,"15min":0.48099999999999998},"max":{"1min":1.1459999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.72599999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.169,"15min":1.0680000000000001},"min":{"1min":0.878,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3400000000000001,"5min":1.577,"15min":1.577},"last":1.044},{"interface":"front","average":{"1min":0.995,"5min":1.05,"15min":0.999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.27,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.999}]},{"osd":5,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.94199999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3680000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.2170000000000001,"15min":1.202},"min":{"1min":1.014,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.4239999999999999,"5min":1.772,"15min":1.772},"last":1.1950000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.198,"15min":1.1299999999999999},"min":{"1min":0.80400000000000005,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.254,"5min":1.2809999999999999,"15min":1.2090000000000001},"min":{"1min":0.96299999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.623,"5min":1.768,"15min":1.768},"last":1.2809999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.387,"15min":1.3600000000000001},"min":{"1min":0.90300000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6719999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4390000000000001,"15min":1.3169999999999999},"min":{"1min":1.248,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.784,"5min":2.1379999999999999,"15min":2.214},"last":1.4410000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4910000000000001,"15min":1.403},"min":{"1min":1.3120000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.988,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.46,"15min":1.4870000000000001},"min":{"1min":0.95499999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.6140000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.3999999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.514,"15min":1.478},"min":{"1min":1.0449999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.506,"15min":1.5800000000000001},"min":{"1min":1.1659999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.0779999999999998,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":1.331}]},{"osd":10,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.542,"15min":1.6200000000000001},"min":{"1min":1.1990000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.756,"5min":2.335,"15min":2.335},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.3979999999999999,"15min":1.4199999999999999},"min":{"1min":1.0149999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.6259999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.4099999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.5960000000000001,"15min":1.546},"min":{"1min":1.367,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.536},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.407,"15min":1.383},"min":{"1min":0.88500000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.5169999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.872,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88800000000000001,"15min":0.73499999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.47099999999999997},"max":{"1min":1.278,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.84199999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.98499999999999999,"15min":0.80000000000000004},"min":{"1min":0.59599999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3959999999999999,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.093,"15min":0.89200000000000002},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.42799999999999999},"max":{"1min":1.5209999999999999,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.99399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0669999999999999,"15min":0.871},"min":{"1min":0.91000000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.2310000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.0229999999999999,"15min":0.878},"min":{"1min":0.79100000000000004,"5min":0.61099999999999999,"15min":0.59199999999999997},"max":{"1min":1.4319999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.72999999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.206,"15min":1.0469999999999999},"min":{"1min":0.88,"5min":0.86599999999999999,"15min":0.70599999999999996},"max":{"1min":1.3400000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96899999999999997,"15min":0.89500000000000002},"min":{"1min":0.66000000000000003,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.1000000000000001,"5min":1.323,"15min":1.323},"last":0.90600000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2310000000000001,"15min":1.155},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.528,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.147,"15min":1.0600000000000001},"min":{"1min":0.85199999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.571,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":1.1220000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.147,"15min":1.083},"min":{"1min":0.84399999999999997,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.6539999999999999,"5min":1.746,"15min":1.746},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3169999999999999,"15min":1.2010000000000001},"min":{"1min":1.127,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.748,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.2549999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3819999999999999,"15min":1.337},"min":{"1min":1.012,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.913,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.403},{"interface":"front","average":{"1min":1.302,"5min":1.306,"15min":1.264},"min":{"1min":0.97799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.756,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.149}]},{"osd":8,"last update":"Mon Feb 12 16:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5109999999999999,"15min":1.427},"min":{"1min":1.1240000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9099999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4890000000000001,"15min":1.4810000000000001},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.5249999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.4079999999999999,"15min":1.4339999999999999},"min":{"1min":1.0449999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":1.887,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.502,"15min":1.484},"min":{"1min":1.292,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.1709999999999998,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.46}]},{"osd":10,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.464,"15min":1.47},"min":{"1min":1.046,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8500000000000001,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5329999999999999,"15min":1.5489999999999999},"min":{"1min":1.1739999999999999,"5min":1.046,"15min":1.046},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.3680000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.879,"15min":0.81399999999999995},"min":{"1min":0.67000000000000004,"5min":0.434,"15min":0.42999999999999999},"max":{"1min":0.98499999999999999,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92700000000000005,"15min":0.81299999999999994},"min":{"1min":0.70199999999999996,"5min":0.56599999999999995,"15min":0.371},"max":{"1min":1.2350000000000001,"5min":1.3979999999999999,"15min":1.4419999999999999},"last":0.84899999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.94399999999999995,"15min":0.84399999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.51900000000000002},"max":{"1min":1.464,"5min":1.466,"15min":1.5529999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.054,"15min":0.90000000000000002},"min":{"1min":0.79400000000000004,"5min":0.77500000000000002,"15min":0.44800000000000001},"max":{"1min":1.3620000000000001,"5min":1.401,"15min":1.4470000000000001},"last":1.0649999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:01:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.0069999999999999,"15min":0.97299999999999998},"min":{"1min":0.76300000000000001,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.103,"5min":1.389,"15min":1.593},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1419999999999999,"15min":1.038},"min":{"1min":0.94799999999999995,"5min":0.73399999999999999,"15min":0.60699999999999998},"max":{"1min":1.4039999999999999,"5min":1.669,"15min":1.669},"last":1.119}]},{"osd":5,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2370000000000001,"15min":1.153},"min":{"1min":0.96599999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6140000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.089,"15min":1.095},"min":{"1min":0.76900000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1830000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":0.90000000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.2290000000000001,"15min":1.1459999999999999},"min":{"1min":0.95699999999999996,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.4650000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":1.24},{"interface":"front","average":{"1min":1.3,"5min":1.321,"15min":1.2490000000000001},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.5669999999999999,"5min":1.7689999999999999,"15min":1.8120000000000001},"last":1.3169999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.335,"15min":1.3160000000000001},"min":{"1min":1.042,"5min":0.82099999999999995,"15min":0.66600000000000004},"max":{"1min":1.6000000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.3899999999999999,"15min":1.3340000000000001},"min":{"1min":0.93600000000000005,"5min":0.85299999999999998,"15min":0.77800000000000002},"max":{"1min":1.669,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.3859999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.3160000000000001,"15min":1.3740000000000001},"min":{"1min":0.93600000000000005,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.669,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.49,"15min":1.4850000000000001},"min":{"1min":0.98999999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.8080000000000001,"5min":2.242,"15min":2.242},"last":1.4370000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.55,"15min":1.554},"min":{"1min":1.24,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.97,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.542},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.5549999999999999},"min":{"1min":1.095,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.877,"5min":1.9099999999999999,"15min":1.9099999999999999},"last":1.472}]},{"osd":10,"last update":"Mon Feb 12 16:01:50 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.46,"15min":1.484},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"max":{"1min":1.73,"5min":2.113,"15min":2.113},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5720000000000001,"15min":1.5840000000000001},"min":{"1min":1.335,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.8300000000000001,"5min":1.974,"15min":1.974},"last":1.6339999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.4910000000000001,"15min":1.4950000000000001},"min":{"1min":1.2330000000000001,"5min":1.073,"15min":1.073},"max":{"1min":1.867,"5min":1.925,"15min":1.925},"last":1.415},{"interface":"front","average":{"1min":1.597,"5min":1.5780000000000001,"15min":1.536},"min":{"1min":1.0469999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.895,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.732}]}]},{"osd":4,"up_from":32,"seq":137438953584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.044,"15min":0.93400000000000005},"min":{"1min":0.72699999999999998,"5min":0.64400000000000002,"15min":0.56999999999999995},"max":{"1min":1.4530000000000001,"5min":1.736,"15min":1.736},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.042,"15min":0.91600000000000004},"min":{"1min":0.72099999999999997,"5min":0.70699999999999996,"15min":0.58699999999999997},"max":{"1min":1.47,"5min":1.617,"15min":1.617},"last":0.97799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.069,"15min":0.95399999999999996},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.63900000000000001},"max":{"1min":1.3859999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.0860000000000001,"15min":0.94599999999999995},"min":{"1min":0.79900000000000004,"5min":0.68799999999999994,"15min":0.51400000000000001},"max":{"1min":1.452,"5min":1.806,"15min":1.806},"last":1.216}]},{"osd":2,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.036,"15min":0.92100000000000004},"min":{"1min":0.79300000000000004,"5min":0.68899999999999995,"15min":0.60299999999999998},"max":{"1min":1.2829999999999999,"5min":1.593,"15min":1.593},"last":1.085},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.0269999999999999,"15min":0.95199999999999996},"min":{"1min":0.745,"5min":0.68000000000000005,"15min":0.64900000000000002},"max":{"1min":1.23,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.0329999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1180000000000001,"15min":0.97499999999999998},"min":{"1min":0.746,"5min":0.72599999999999998,"15min":0.57599999999999996},"max":{"1min":1.546,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.113,"15min":0.95399999999999996},"min":{"1min":0.73299999999999998,"5min":0.72199999999999998,"15min":0.55100000000000005},"max":{"1min":1.494,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.1850000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1060000000000001,"15min":0.92400000000000004},"min":{"1min":0.69599999999999995,"5min":0.625,"15min":0.55200000000000005},"max":{"1min":1.611,"5min":1.889,"15min":1.889},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1850000000000001,"15min":0.98899999999999999},"min":{"1min":0.73599999999999999,"5min":0.67200000000000004,"15min":0.57799999999999996},"max":{"1min":1.476,"5min":1.581,"15min":1.581},"last":1.1579999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1279999999999999,"15min":1.042},"min":{"1min":0.82999999999999996,"5min":0.66900000000000004,"15min":0.65600000000000003},"max":{"1min":1.401,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.242,"15min":1.1359999999999999},"min":{"1min":0.85199999999999998,"5min":0.74199999999999999,"15min":0.58799999999999997},"max":{"1min":1.5740000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.78500000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.165,"15min":1.129},"min":{"1min":0.78700000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.4330000000000001,"5min":1.718,"15min":1.718},"last":1.274},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3919999999999999,"15min":1.3360000000000001},"min":{"1min":0.96399999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.6699999999999999,"5min":2.0110000000000001,"15min":2.302},"last":1.01}]},{"osd":8,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5249999999999999,"15min":1.5129999999999999},"min":{"1min":1.073,"5min":1.073,"15min":0.88400000000000001},"max":{"1min":1.8100000000000001,"5min":2.093,"15min":2.1949999999999998},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4590000000000001,"15min":1.423},"min":{"1min":1.133,"5min":1.038,"15min":0.93700000000000006},"max":{"1min":1.931,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.335}]},{"osd":9,"last update":"Mon Feb 12 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.601,"15min":1.5609999999999999},"min":{"1min":1.1739999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"max":{"1min":1.897,"5min":2.173,"15min":2.173},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.528,"15min":1.474},"min":{"1min":1.1539999999999999,"5min":1.109,"15min":1.109},"max":{"1min":2.0009999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.25}]},{"osd":10,"last update":"Mon Feb 12 16:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.5429999999999999,"15min":1.5149999999999999},"min":{"1min":1.236,"5min":1.097,"15min":1.097},"max":{"1min":1.9790000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.546,"15min":1.5629999999999999},"min":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"max":{"1min":2.0009999999999999,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":1.375}]}]},{"osd":5,"up_from":39,"seq":167503724649,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297812,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447340,"statfs":{"total":95995035648,"available":95690076160,"internally_reserved":0,"allocated":733184,"data_stored":329197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0609999999999999,"15min":0.90800000000000003},"min":{"1min":0.65100000000000002,"5min":0.63100000000000001,"15min":0.57699999999999996},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0349999999999999,"15min":0.93100000000000005},"min":{"1min":0.79500000000000004,"5min":0.66300000000000003,"15min":0.46999999999999997},"max":{"1min":1.4279999999999999,"5min":1.524,"15min":1.524},"last":0.90100000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1080000000000001,"15min":0.96399999999999997},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.59999999999999998},"max":{"1min":1.3600000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.026},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.028,"15min":0.92400000000000004},"min":{"1min":0.75800000000000001,"5min":0.74099999999999999,"15min":0.59099999999999997},"max":{"1min":1.3660000000000001,"5min":1.381,"15min":1.381},"last":0.69199999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1160000000000001,"15min":0.99399999999999999},"min":{"1min":0.75800000000000001,"5min":0.622,"15min":0.57999999999999996},"max":{"1min":1.5309999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.169,"5min":1.099,"15min":1.0329999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.58899999999999997},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.1020000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1100000000000001,"15min":1.0149999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70299999999999996},"max":{"1min":1.544,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.232},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.202,"15min":1.0920000000000001},"min":{"1min":0.82599999999999996,"5min":0.72699999999999998,"15min":0.66400000000000003},"max":{"1min":1.5589999999999999,"5min":1.726,"15min":1.726},"last":1.0669999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.999,"15min":0.92000000000000004},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.56499999999999995},"max":{"1min":1.4510000000000001,"5min":1.889,"15min":1.889},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.208,"5min":1.181,"15min":1.02},"min":{"1min":0.86599999999999999,"5min":0.68200000000000005,"15min":0.52700000000000002},"max":{"1min":1.712,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.002}]},{"osd":6,"last update":"Mon Feb 12 16:02:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.21,"15min":1.087},"min":{"1min":0.69999999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.871},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2849999999999999,"15min":1.1379999999999999},"min":{"1min":0.86699999999999999,"5min":0.68300000000000005,"15min":0.40400000000000003},"max":{"1min":1.5940000000000001,"5min":1.8360000000000001,"15min":1.903},"last":0.92800000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.274,"15min":1.161},"min":{"1min":1.0720000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.339},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.4159999999999999,"15min":1.2290000000000001},"min":{"1min":1.107,"5min":0.69099999999999995,"15min":0.435},"max":{"1min":1.8300000000000001,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.1359999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.532,"15min":1.427},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.9850000000000001,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.484},{"interface":"front","average":{"1min":1.603,"5min":1.466,"15min":1.331},"min":{"1min":1.347,"5min":1.0860000000000001,"15min":0.83999999999999997},"max":{"1min":1.96,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.3620000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:02:09 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5569999999999999,"15min":1.492},"min":{"1min":1.256,"5min":1.0469999999999999,"15min":1.0469999999999999},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.542},{"interface":"front","average":{"1min":1.663,"5min":1.6319999999999999,"15min":1.587},"min":{"1min":1.397,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":2,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.2849999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:01:52 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4850000000000001,"15min":1.492},"min":{"1min":1.3260000000000001,"5min":1.145,"15min":1.145},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.585},{"interface":"front","average":{"1min":1.609,"5min":1.6299999999999999,"15min":1.6359999999999999},"min":{"1min":1.2629999999999999,"5min":1.222,"15min":1.222},"max":{"1min":2.246,"5min":2.254,"15min":2.254},"last":1.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:02:51.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:02:51.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:51 smithi123 bash[19119]: audit 2024-02-12T16:02:50.534342+0000 mgr.y (mgr.14146) 511 : audit [DBG] from='client.14730 v1:172.21.15.121:0/4216751182' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:02:51.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:51 smithi121 bash[18489]: audit 2024-02-12T16:02:50.534342+0000 mgr.y (mgr.14146) 511 : audit [DBG] from='client.14730 v1:172.21.15.121:0/4216751182' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:02:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:51 smithi169 bash[19604]: audit 2024-02-12T16:02:50.534342+0000 mgr.y (mgr.14146) 511 : audit [DBG] from='client.14730 v1:172.21.15.121:0/4216751182' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:02:52.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:52 smithi123 bash[19119]: cluster 2024-02-12T16:02:51.515338+0000 mgr.y (mgr.14146) 512 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:52.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:52 smithi121 bash[18489]: cluster 2024-02-12T16:02:51.515338+0000 mgr.y (mgr.14146) 512 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:52 smithi169 bash[19604]: cluster 2024-02-12T16:02:51.515338+0000 mgr.y (mgr.14146) 512 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:54.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:54 smithi123 bash[19119]: cluster 2024-02-12T16:02:53.515948+0000 mgr.y (mgr.14146) 513 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:54.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:54 smithi121 bash[18489]: cluster 2024-02-12T16:02:53.515948+0000 mgr.y (mgr.14146) 513 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:54 smithi169 bash[19604]: cluster 2024-02-12T16:02:53.515948+0000 mgr.y (mgr.14146) 513 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:55.982 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:02:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:56 smithi169 bash[19604]: cluster 2024-02-12T16:02:55.516625+0000 mgr.y (mgr.14146) 514 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:56 smithi123 bash[19119]: cluster 2024-02-12T16:02:55.516625+0000 mgr.y (mgr.14146) 514 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:56.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:56 smithi121 bash[18489]: cluster 2024-02-12T16:02:55.516625+0000 mgr.y (mgr.14146) 514 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:58.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:58 smithi123 bash[19119]: cluster 2024-02-12T16:02:57.517216+0000 mgr.y (mgr.14146) 515 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:58.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:58 smithi121 bash[18489]: cluster 2024-02-12T16:02:57.517216+0000 mgr.y (mgr.14146) 515 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:58.956 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:02:58.958 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:02:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:58 smithi169 bash[19604]: cluster 2024-02-12T16:02:57.517216+0000 mgr.y (mgr.14146) 515 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:02:59.610 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2024-02-12T16:02:57.516904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":3573888,"kb_used_data":8736,"kb_used_omap":0,"kb_used_meta":3565056,"kb_avail":1121367936,"statfs":{"total":1151940427776,"available":1148280766464,"internally_reserved":0,"allocated":8945664,"data_stored":4017236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3650617344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":139,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-02-12T16:02:49.550827+0000","last_change":"2024-02-12T16:02:49.550827+0000","last_active":"2024-02-12T16:02:49.550827+0000","last_peered":"2024-02-12T16:02:49.550827+0000","last_clean":"2024-02-12T16:02:49.550827+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:02:49.550827+0000","last_undegraded":"2024-02-12T16:02:49.550827+0000","last_fullsized":"2024-02-12T16:02:49.550827+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:02:49.550626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:00:34.507052+0000","last_clean_scrub_stamp":"2024-02-12T16:02:49.550626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0409999999999999,"15min":1.0389999999999999},"min":{"1min":0.71899999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.482,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.089,"15min":1.123},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.4379999999999999,"5min":2.871,"15min":2.871},"last":0.81100000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.048,"15min":1.1020000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1990000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0780000000000001,"15min":1.0880000000000001},"min":{"1min":0.73199999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5920000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.97699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.119,"15min":1.157},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.466,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.1519999999999999,"15min":1.24},"min":{"1min":0.77200000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3210000000000002,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.879}]},{"osd":3,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1739999999999999,"15min":1.2190000000000001},"min":{"1min":0.92600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6839999999999999,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":1.091},{"interface":"front","average":{"1min":1.032,"5min":1.151,"15min":1.2070000000000001},"min":{"1min":0.75800000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.393,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":1.0049999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2749999999999999,"15min":1.3720000000000001},"min":{"1min":0.81200000000000006,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4770000000000001,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.355,"15min":1.4390000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8400000000000001,"5min":4.0789999999999997,"15min":4.0789999999999997},"last":1.367}]},{"osd":5,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.421,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.754,"5min":3.044,"15min":3.044},"last":1.321},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.423,"15min":1.5069999999999999},"min":{"1min":0.93500000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.6279999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":1.0720000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4379999999999999,"15min":1.5349999999999999},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.827,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.5349999999999999,"15min":1.619},"min":{"1min":1.038,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9279999999999999,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.605}]},{"osd":7,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.534,"15min":1.6259999999999999},"min":{"1min":0.90000000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.8779999999999999,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.478},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6259999999999999,"15min":1.722},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.921,"5min":3.8370000000000002,"15min":3.8370000000000002},"last":1.3919999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.569,"15min":1.6519999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.974,"5min":4,"15min":4},"last":1.651},{"interface":"front","average":{"1min":1.534,"5min":1.647,"15min":1.742},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.9490000000000001,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.448}]},{"osd":10,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.5700000000000001,"15min":1.694},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.772,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.163},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.742,"15min":1.8360000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0230000000000001,"5min":4.3949999999999996,"15min":4.3949999999999996},"last":1.54}]}]},{"osd":10,"up_from":83,"seq":356482285633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.88800000000000001,"15min":0.89200000000000002},"min":{"1min":0.66100000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1779999999999999,"5min":1.413,"15min":1.413},"last":1.123},{"interface":"front","average":{"1min":1.044,"5min":1.036,"15min":1.0700000000000001},"min":{"1min":0.72499999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.377,"5min":1.573,"15min":1.573},"last":0.83199999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.995,"15min":0.98099999999999998},"min":{"1min":0.70399999999999996,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.3049999999999999,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.052,"5min":0.995,"15min":0.96499999999999997},"min":{"1min":0.72699999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3069999999999999,"5min":1.492,"15min":1.492},"last":1.0109999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0229999999999999,"15min":1.0109999999999999},"min":{"1min":0.73399999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0049999999999999,"15min":1.0169999999999999},"min":{"1min":0.76500000000000001,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.256,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.875}]},{"osd":4,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.1930000000000001,"15min":1.1180000000000001},"min":{"1min":0.88200000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5860000000000001,"5min":1.647,"15min":1.647},"last":1.248},{"interface":"front","average":{"1min":1.196,"5min":1.242,"15min":1.234},"min":{"1min":0.84799999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.5209999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.2070000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.1739999999999999,"15min":1.1839999999999999},"min":{"1min":0.89600000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.464,"5min":2.048,"15min":2.048},"last":1.224},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3260000000000001,"15min":1.3080000000000001},"min":{"1min":1.038,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.5129999999999999,"5min":1.788,"15min":1.788},"last":1.403}]},{"osd":6,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.222,"15min":1.23},"min":{"1min":0.91300000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.5089999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.04},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.413,"15min":1.357},"min":{"1min":1.113,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6120000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.302}]},{"osd":7,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.351,"15min":1.2909999999999999},"min":{"1min":1.1499999999999999,"5min":0.755,"15min":0.755},"max":{"1min":1.665,"5min":1.764,"15min":1.764},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.3799999999999999,"15min":1.3819999999999999},"min":{"1min":1.0580000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.5429999999999999,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.3280000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.3999999999999999,"15min":1.351},"min":{"1min":0.96899999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.891,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.4450000000000001,"15min":1.351},"min":{"1min":1.1499999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8160000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.554}]},{"osd":9,"last update":"Mon Feb 12 16:01:51 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.5189999999999999,"15min":1.4510000000000001},"min":{"1min":1.2589999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.7370000000000001,"5min":2.02,"15min":2.02},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.502,"15min":1.4179999999999999},"min":{"1min":1.321,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7689999999999999,"5min":1.952,"15min":1.952},"last":1.5209999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.595,"15min":1.6100000000000001},"min":{"1min":1.2989999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.484},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.6180000000000001,"15min":1.6379999999999999},"min":{"1min":1.2989999999999999,"5min":1.25,"15min":1.25},"max":{"1min":1.919,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.4610000000000001}]}]},{"osd":9,"up_from":74,"seq":317827579979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.95199999999999996,"15min":0.93600000000000005},"min":{"1min":0.67400000000000004,"5min":0.623,"15min":0.623},"max":{"1min":1.218,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.95899999999999996,"15min":0.97799999999999998},"min":{"1min":0.72799999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.347,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.71899999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":0.998},"min":{"1min":0.71899999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.399,"5min":1.48,"15min":1.48},"last":1.097},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0589999999999999,"15min":1.0369999999999999},"min":{"1min":0.79100000000000004,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.427,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.0860000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95499999999999996,"15min":0.93300000000000005},"min":{"1min":0.69099999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.343,"5min":1.738,"15min":1.738},"last":1},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":1.0309999999999999},"min":{"1min":0.73099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3049999999999999,"5min":1.601,"15min":1.601},"last":0.83599999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.98799999999999999,"15min":0.97199999999999998},"min":{"1min":0.629,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.371,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.123,"5min":1.1180000000000001,"15min":1.1100000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5029999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.874}]},{"osd":4,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1919999999999999,"15min":1.177},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.6739999999999999,"5min":2.113,"15min":2.113},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.165,"5min":1.1930000000000001,"15min":1.1919999999999999},"min":{"1min":0.78500000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.5680000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.02}]},{"osd":5,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2350000000000001,"15min":1.2270000000000001},"min":{"1min":0.85199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.756,"5min":2.4399999999999999,"15min":2.4399999999999999},"last":1.147},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.296,"15min":1.264},"min":{"1min":0.92600000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.661,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.3049999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.325,"15min":1.325},"min":{"1min":1.0960000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.5820000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.218},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.3640000000000001,"15min":1.3600000000000001},"min":{"1min":0.93400000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.573,"5min":2.48,"15min":2.48},"last":1.24}]},{"osd":7,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3149999999999999,"15min":1.3260000000000001},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.575,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4359999999999999,"15min":1.411},"min":{"1min":1.2350000000000001,"5min":0.875,"15min":0.875},"max":{"1min":1.7609999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.375}]},{"osd":8,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.407,"15min":1.3720000000000001},"min":{"1min":0.91900000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.792,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.159},{"interface":"front","average":{"1min":1.369,"5min":1.3939999999999999,"15min":1.3600000000000001},"min":{"1min":0.93700000000000006,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.706,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":1.1739999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.4330000000000001,"15min":1.4299999999999999},"min":{"1min":1.0780000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.597,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.3979999999999999,"15min":1.3560000000000001},"min":{"1min":1.05,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.8400000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.272}]}]},{"osd":8,"up_from":65,"seq":279172874321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95999999999999996,"15min":0.85599999999999998},"min":{"1min":0.61699999999999999,"5min":0.57899999999999996,"15min":0.57699999999999996},"max":{"1min":1.1839999999999999,"5min":1.488,"15min":1.488},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.96499999999999997,"15min":0.89500000000000002},"min":{"1min":0.71699999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.589,"5min":1.623,"15min":1.623},"last":0.82399999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.88800000000000001,"15min":0.84999999999999998},"min":{"1min":0.64900000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.3480000000000001,"5min":1.508,"15min":1.508},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0249999999999999,"15min":0.95599999999999996},"min":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.355,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.02}]},{"osd":2,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.01,"15min":0.88},"min":{"1min":0.68700000000000006,"5min":0.59499999999999997,"15min":0.59199999999999997},"max":{"1min":1.194,"5min":1.764,"15min":1.764},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.173,"5min":1.141,"15min":1.0309999999999999},"min":{"1min":0.91400000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.637,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.0009999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.083,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.64800000000000002,"15min":0.60499999999999998},"max":{"1min":1.365,"5min":1.698,"15min":1.698},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.22,"15min":1.04},"min":{"1min":0.96099999999999997,"5min":0.623,"15min":0.623},"max":{"1min":1.3280000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.1579999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2629999999999999,"15min":1.157},"min":{"1min":0.94599999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.4339999999999999,"5min":1.677,"15min":1.839},"last":1.456},{"interface":"front","average":{"1min":1.3,"5min":1.3080000000000001,"15min":1.1160000000000001},"min":{"1min":0.80800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.772,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.2210000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1990000000000001,"15min":1.155},"min":{"1min":0.76900000000000002,"5min":0.76700000000000002,"15min":0.75800000000000001},"max":{"1min":1.4379999999999999,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.341,"5min":1.3420000000000001,"15min":1.2030000000000001},"min":{"1min":1.157,"5min":0.71899999999999997,"15min":0.68899999999999995},"max":{"1min":1.508,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.1970000000000001},"min":{"1min":0.92800000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.915,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.429,"5min":1.3939999999999999,"15min":1.25},"min":{"1min":1.073,"5min":0.91500000000000004,"15min":0.876},"max":{"1min":1.5549999999999999,"5min":2.339,"15min":2.339},"last":1.077}]},{"osd":7,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4530000000000001,"15min":1.2629999999999999},"min":{"1min":1.0860000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6120000000000001,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.371},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4930000000000001,"15min":1.292},"min":{"1min":1.194,"5min":0.997,"15min":0.85499999999999998},"max":{"1min":1.7909999999999999,"5min":1.984,"15min":1.984},"last":1.2949999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4610000000000001,"15min":1.2470000000000001},"min":{"1min":1.0680000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.8620000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.349},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3380000000000001,"15min":1.218},"min":{"1min":0.96899999999999997,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7330000000000001,"5min":2.456,"15min":2.456},"last":1.1419999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4079999999999999,"15min":1.3640000000000001},"min":{"1min":1.002,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.502,"15min":1.462},"min":{"1min":0.99399999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.8069999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.52}]}]},{"osd":7,"up_from":56,"seq":240518168666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.036,"15min":0.98999999999999999},"min":{"1min":0.63900000000000001,"5min":0.60899999999999999,"15min":0.436},"max":{"1min":1.671,"5min":1.675,"15min":1.675},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.74199999999999999,"5min":0.64100000000000001,"15min":0.53400000000000003},"max":{"1min":1.171,"5min":1.353,"15min":1.353},"last":1.081}]},{"osd":2,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.98399999999999999,"15min":1.022},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3680000000000001,"5min":1.5980000000000001,"15min":1.8580000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.052,"15min":1.044},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.56499999999999995},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.1240000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.089,"15min":1.083},"min":{"1min":0.628,"5min":0.628,"15min":0.53400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":2.0049999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.014,"5min":1.046,"15min":1.093},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.355,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.91800000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95399999999999996,"15min":1.0069999999999999},"min":{"1min":0.69599999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4019999999999999,"5min":1.51,"15min":2.0449999999999999},"last":0.751},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.042},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4790000000000001,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.80500000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.095,"15min":1.077},"min":{"1min":0.82799999999999996,"5min":0.626,"15min":0.60099999999999998},"max":{"1min":1.4590000000000001,"5min":1.76,"15min":1.76},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1419999999999999,"15min":1.181},"min":{"1min":0.65800000000000003,"5min":0.62,"15min":0.46100000000000002},"max":{"1min":1.4259999999999999,"5min":1.677,"15min":2.1709999999999998},"last":0.98599999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1719999999999999,"15min":1.2030000000000001},"min":{"1min":0.78700000000000003,"5min":0.73099999999999998,"15min":0.64300000000000002},"max":{"1min":1.744,"5min":1.744,"15min":1.8069999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.204,"5min":1.1870000000000001,"15min":1.2230000000000001},"min":{"1min":0.77900000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.486,"5min":2.0960000000000001,"15min":2.1139999999999999},"last":1.028}]},{"osd":8,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":1.1910000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.486},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4099999999999999,"15min":1.446},"min":{"1min":1.028,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8180000000000001,"5min":1.905,"15min":2.0760000000000001},"last":1.3009999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.4319999999999999,"15min":1.4330000000000001},"min":{"1min":1.032,"5min":0.751,"15min":0.751},"max":{"1min":1.8420000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.387},{"interface":"front","average":{"1min":1.417,"5min":1.427,"15min":1.421},"min":{"1min":0.98599999999999999,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":1.712,"5min":1.944,"15min":1.944},"last":1.109}]},{"osd":10,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.504,"15min":1.5349999999999999},"min":{"1min":0.96499999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":1.637},{"interface":"front","average":{"1min":1.542,"5min":1.502,"15min":1.5589999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9450000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.411}]},{"osd":11,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.55,"15min":1.5820000000000001},"min":{"1min":1.169,"5min":1.071,"15min":1.071},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.522,"15min":1.5209999999999999},"min":{"1min":1.087,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7849999999999999,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.47}]}]},{"osd":6,"up_from":48,"seq":206158430307,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.997,"15min":0.94899999999999995},"min":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.498},"max":{"1min":1.3460000000000001,"5min":1.492,"15min":1.492},"last":1.121},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":0.92500000000000004},"min":{"1min":0.72099999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.254,"5min":1.573,"15min":1.573},"last":1.0369999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.81100000000000005,"5min":0.67200000000000004,"15min":0.64300000000000002},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.048,"5min":1.0429999999999999,"15min":0.998},"min":{"1min":0.64000000000000001,"5min":0.62,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.95399999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0589999999999999,"15min":0.96799999999999997},"min":{"1min":0.76100000000000001,"5min":0.66000000000000003,"15min":0.55500000000000005},"max":{"1min":1.3129999999999999,"5min":2.383,"15min":2.383},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.109,"15min":1.004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.24}]},{"osd":3,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.032,"15min":0.95399999999999996},"min":{"1min":0.75700000000000001,"5min":0.68400000000000005,"15min":0.68100000000000005},"max":{"1min":1.3799999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.095,"15min":1.042},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.363,"5min":1.623,"15min":1.623},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.038,"15min":0.91000000000000003},"min":{"1min":0.68500000000000005,"5min":0.57199999999999995,"15min":0.41899999999999998},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.155},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.42299999999999999},"max":{"1min":1.554,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.0189999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.202,"15min":1.0069999999999999},"min":{"1min":0.79200000000000004,"5min":0.60299999999999998,"15min":0.46999999999999997},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.214,"15min":1.034},"min":{"1min":0.82999999999999996,"5min":0.69499999999999995,"15min":0.51800000000000002},"max":{"1min":1.538,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.0980000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.321,"15min":1.1890000000000001},"min":{"1min":0.875,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8839999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.337,"15min":1.2729999999999999},"min":{"1min":1.147,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.631,"5min":1.889,"15min":1.889},"last":1.2849999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.502,"15min":1.397},"min":{"1min":1.0960000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.804},{"interface":"front","average":{"1min":1.544,"5min":1.534,"15min":1.4259999999999999},"min":{"1min":1.083,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.0739999999999998,"5min":2.093,"15min":2.093},"last":1.6060000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:02:09 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.5629999999999999,"15min":1.4730000000000001},"min":{"1min":1.3320000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.883},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.4870000000000001,"15min":1.407},"min":{"1min":1.2929999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.125,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.673}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":1.6559999999999999,"15min":1.627},"min":{"1min":1.3759999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"max":{"1min":2.1200000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.9390000000000001},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.591,"15min":1.569},"min":{"1min":1.3320000000000001,"5min":1.081,"15min":1.081},"max":{"1min":2.0070000000000001,"5min":2.367,"15min":2.367},"last":1.724}]}]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84399999999999997,"15min":0.66500000000000004},"min":{"1min":0.61499999999999999,"5min":0.41099999999999998,"15min":0.377},"max":{"1min":1.1160000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.93899999999999995,"15min":0.73099999999999998},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.28899999999999998},"max":{"1min":1.345,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.179}]},{"osd":2,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.94499999999999995,"15min":0.75900000000000001},"min":{"1min":0.63800000000000001,"5min":0.50700000000000001,"15min":0.379},"max":{"1min":1.252,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96799999999999997,"15min":0.79100000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.34999999999999998},"max":{"1min":1.2669999999999999,"5min":1.54,"15min":1.54},"last":0.89500000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:02:16 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98999999999999999,"15min":0.83799999999999997},"min":{"1min":0.71299999999999997,"5min":0.61799999999999999,"15min":0.61099999999999999},"max":{"1min":1.3,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0389999999999999,"15min":0.90000000000000002},"min":{"1min":0.72899999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.54,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.93200000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.155,"15min":1.0309999999999999},"min":{"1min":0.73699999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.502},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.232,"15min":1.1060000000000001},"min":{"1min":0.77100000000000002,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.534,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.476}]},{"osd":5,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3160000000000001,"15min":1.1439999999999999},"min":{"1min":1.034,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4390000000000001,"15min":1.234},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":1.367}]},{"osd":6,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.327,"15min":1.2170000000000001},"min":{"1min":1.073,"5min":0.81399999999999995,"15min":0.80500000000000005},"max":{"1min":1.6439999999999999,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.29},{"interface":"front","average":{"1min":1.512,"5min":1.486,"15min":1.3280000000000001},"min":{"1min":1.0640000000000001,"5min":0.872,"15min":0.872},"max":{"1min":2.0739999999999998,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":1.383}]},{"osd":7,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.472,"15min":1.3819999999999999},"min":{"1min":0.99199999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.0139999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.3779999999999999,"15min":1.3400000000000001},"min":{"1min":1.034,"5min":0.871,"15min":0.871},"max":{"1min":1.887,"5min":1.887,"15min":1.887},"last":1.262}]},{"osd":8,"last update":"Mon Feb 12 16:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.49,"15min":1.403},"min":{"1min":1.0069999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8049999999999999,"5min":2.355,"15min":2.355},"last":1.73},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6479999999999999,"15min":1.506},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1349999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.748}]},{"osd":9,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.653,"15min":1.635},"min":{"1min":1.196,"5min":1.125,"15min":1.125},"max":{"1min":2.3410000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.827},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.518,"15min":1.534},"min":{"1min":1.21,"5min":1.1180000000000001,"15min":1.1180000000000001},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":1.6060000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.548,"15min":1.571},"min":{"1min":1.175,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.804},{"interface":"front","average":{"1min":1.712,"5min":1.72,"15min":1.756},"min":{"1min":1.2490000000000001,"5min":1.089,"15min":1.089},"max":{"1min":2.238,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.6679999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.81599999999999995,"15min":0.67400000000000004},"min":{"1min":0.621,"5min":0.44900000000000001,"15min":0.32500000000000001},"max":{"1min":1.0449999999999999,"5min":1.2170000000000001,"15min":1.3009999999999999},"last":0.88},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.84099999999999997,"15min":0.70899999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.242,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.873,"15min":0.76200000000000001},"min":{"1min":0.65300000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.2390000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.01},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.93300000000000005,"15min":0.86499999999999999},"min":{"1min":0.61099999999999999,"5min":0.53500000000000003,"15min":0.48099999999999998},"max":{"1min":1.1459999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.83999999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1599999999999999,"15min":1.081},"min":{"1min":0.78300000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.202},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0489999999999999,"15min":1.0049999999999999},"min":{"1min":0.71299999999999997,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.97599999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.94199999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3680000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.2170000000000001,"15min":1.202},"min":{"1min":1.014,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.4239999999999999,"5min":1.772,"15min":1.772},"last":1.1220000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.198,"15min":1.1299999999999999},"min":{"1min":0.80400000000000005,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.2809999999999999,"15min":1.2090000000000001},"min":{"1min":0.96299999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.623,"5min":1.768,"15min":1.768},"last":1.246}]},{"osd":7,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.353,"15min":1.3560000000000001},"min":{"1min":0.97699999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7649999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.486,"15min":1.3360000000000001},"min":{"1min":1.262,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0059999999999998,"5min":2.1379999999999999,"15min":2.214},"last":1.363}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4910000000000001,"15min":1.403},"min":{"1min":1.3120000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.988,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.46,"15min":1.4870000000000001},"min":{"1min":0.95499999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.6140000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.5609999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.514,"15min":1.478},"min":{"1min":1.0449999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.506,"15min":1.5800000000000001},"min":{"1min":1.1659999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.0779999999999998,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":1.649}]},{"osd":10,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.51,"15min":1.6100000000000001},"min":{"1min":1.0229999999999999,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.8899999999999999,"5min":2.335,"15min":2.335},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3859999999999999,"15min":1.4159999999999999},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.339}]},{"osd":11,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.5960000000000001,"15min":1.546},"min":{"1min":1.367,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.407,"15min":1.383},"min":{"1min":0.88500000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.3979999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313347,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.872,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88800000000000001,"15min":0.73499999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.47099999999999997},"max":{"1min":1.278,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.98499999999999999,"15min":0.80000000000000004},"min":{"1min":0.59599999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3959999999999999,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.093,"15min":0.89200000000000002},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.42799999999999999},"max":{"1min":1.5209999999999999,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0669999999999999,"15min":0.871},"min":{"1min":0.91000000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.2310000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0229999999999999,"15min":0.878},"min":{"1min":0.79100000000000004,"5min":0.61099999999999999,"15min":0.59199999999999997},"max":{"1min":1.4319999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.2150000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.206,"15min":1.0469999999999999},"min":{"1min":0.88,"5min":0.86599999999999999,"15min":0.70599999999999996},"max":{"1min":1.3400000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.367},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96899999999999997,"15min":0.89500000000000002},"min":{"1min":0.66000000000000003,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.1000000000000001,"5min":1.323,"15min":1.323},"last":0.98699999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2310000000000001,"15min":1.155},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.528,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.085},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.147,"15min":1.0600000000000001},"min":{"1min":0.85199999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.571,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":1.486}]},{"osd":6,"last update":"Mon Feb 12 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.147,"15min":1.083},"min":{"1min":0.84399999999999997,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.6539999999999999,"5min":1.746,"15min":1.746},"last":1.28},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3169999999999999,"15min":1.2010000000000001},"min":{"1min":1.127,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.748,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.387}]},{"osd":7,"last update":"Mon Feb 12 16:01:53 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3819999999999999,"15min":1.337},"min":{"1min":1.012,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.913,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.306,"15min":1.264},"min":{"1min":0.97799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.756,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.1890000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5109999999999999,"15min":1.427},"min":{"1min":1.1240000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9099999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.514},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4890000000000001,"15min":1.4810000000000001},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.671}]},{"osd":9,"last update":"Mon Feb 12 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.4079999999999999,"15min":1.4339999999999999},"min":{"1min":1.0449999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":1.887,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.502,"15min":1.484},"min":{"1min":1.292,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.1709999999999998,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.3049999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.464,"15min":1.47},"min":{"1min":1.046,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8500000000000001,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.548},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5329999999999999,"15min":1.5489999999999999},"min":{"1min":1.1739999999999999,"5min":1.046,"15min":1.046},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.611}]}]},{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.879,"15min":0.81399999999999995},"min":{"1min":0.67000000000000004,"5min":0.434,"15min":0.42999999999999999},"max":{"1min":0.98499999999999999,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92700000000000005,"15min":0.81299999999999994},"min":{"1min":0.70199999999999996,"5min":0.56599999999999995,"15min":0.371},"max":{"1min":1.2350000000000001,"5min":1.3979999999999999,"15min":1.4419999999999999},"last":1.008}]},{"osd":2,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.94399999999999995,"15min":0.84399999999999997},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.51900000000000002},"max":{"1min":1.464,"5min":1.466,"15min":1.5529999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.054,"15min":0.90000000000000002},"min":{"1min":0.79400000000000004,"5min":0.77500000000000002,"15min":0.44800000000000001},"max":{"1min":1.3620000000000001,"5min":1.401,"15min":1.4470000000000001},"last":1.1379999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:01:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.0069999999999999,"15min":0.97299999999999998},"min":{"1min":0.76300000000000001,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.103,"5min":1.389,"15min":1.593},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1419999999999999,"15min":1.038},"min":{"1min":0.94799999999999995,"5min":0.73399999999999999,"15min":0.60699999999999998},"max":{"1min":1.4039999999999999,"5min":1.669,"15min":1.669},"last":1.081}]},{"osd":5,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2370000000000001,"15min":1.153},"min":{"1min":0.96599999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6140000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.415},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.089,"15min":1.095},"min":{"1min":0.76900000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1830000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.204}]},{"osd":6,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.2290000000000001,"15min":1.1459999999999999},"min":{"1min":0.95699999999999996,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.4650000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":1.232},{"interface":"front","average":{"1min":1.3,"5min":1.321,"15min":1.2490000000000001},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.5669999999999999,"5min":1.7689999999999999,"15min":1.8120000000000001},"last":1.6919999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.335,"15min":1.3160000000000001},"min":{"1min":1.042,"5min":0.82099999999999995,"15min":0.66600000000000004},"max":{"1min":1.6000000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.357},{"interface":"front","average":{"1min":1.335,"5min":1.3899999999999999,"15min":1.3340000000000001},"min":{"1min":0.93600000000000005,"5min":0.85299999999999998,"15min":0.77800000000000002},"max":{"1min":1.669,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.623}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.3160000000000001,"15min":1.3740000000000001},"min":{"1min":0.93600000000000005,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.669,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.304},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.49,"15min":1.4850000000000001},"min":{"1min":0.98999999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.8080000000000001,"5min":2.242,"15min":2.242},"last":1.597}]},{"osd":9,"last update":"Mon Feb 12 16:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.55,"15min":1.554},"min":{"1min":1.24,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.97,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.5549999999999999},"min":{"1min":1.095,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.877,"5min":1.9099999999999999,"15min":1.9099999999999999},"last":1.7529999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4450000000000001,"15min":1.4790000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":1.7869999999999999,"5min":2.113,"15min":2.113},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.546,"5min":1.5629999999999999,"15min":1.581},"min":{"1min":1.1279999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.819,"5min":1.974,"15min":1.974},"last":1.446}]},{"osd":11,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.4910000000000001,"15min":1.4950000000000001},"min":{"1min":1.2330000000000001,"5min":1.073,"15min":1.073},"max":{"1min":1.867,"5min":1.925,"15min":1.925},"last":1.462},{"interface":"front","average":{"1min":1.597,"5min":1.5780000000000001,"15min":1.536},"min":{"1min":1.0469999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.895,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.6659999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.044,"15min":0.93400000000000005},"min":{"1min":0.72699999999999998,"5min":0.64400000000000002,"15min":0.56999999999999995},"max":{"1min":1.4530000000000001,"5min":1.736,"15min":1.736},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.042,"15min":0.91600000000000004},"min":{"1min":0.72099999999999997,"5min":0.70699999999999996,"15min":0.58699999999999997},"max":{"1min":1.47,"5min":1.617,"15min":1.617},"last":1.1970000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.069,"15min":0.95399999999999996},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.63900000000000001},"max":{"1min":1.3859999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.115},{"interface":"front","average":{"1min":1.103,"5min":1.0860000000000001,"15min":0.94599999999999995},"min":{"1min":0.79900000000000004,"5min":0.68799999999999994,"15min":0.51400000000000001},"max":{"1min":1.452,"5min":1.806,"15min":1.806},"last":1.3280000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.036,"15min":0.92100000000000004},"min":{"1min":0.79300000000000004,"5min":0.68899999999999995,"15min":0.60299999999999998},"max":{"1min":1.2829999999999999,"5min":1.593,"15min":1.593},"last":1.282},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.0269999999999999,"15min":0.95199999999999996},"min":{"1min":0.745,"5min":0.68000000000000005,"15min":0.64900000000000002},"max":{"1min":1.23,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.429}]},{"osd":3,"last update":"Mon Feb 12 16:02:00 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1180000000000001,"15min":0.97499999999999998},"min":{"1min":0.746,"5min":0.72599999999999998,"15min":0.57599999999999996},"max":{"1min":1.546,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.113,"15min":0.95399999999999996},"min":{"1min":0.73299999999999998,"5min":0.72199999999999998,"15min":0.55100000000000005},"max":{"1min":1.494,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.0429999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1060000000000001,"15min":0.92400000000000004},"min":{"1min":0.69599999999999995,"5min":0.625,"15min":0.55200000000000005},"max":{"1min":1.611,"5min":1.889,"15min":1.889},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1850000000000001,"15min":0.98899999999999999},"min":{"1min":0.73599999999999999,"5min":0.67200000000000004,"15min":0.57799999999999996},"max":{"1min":1.476,"5min":1.581,"15min":1.581},"last":1.008}]},{"osd":6,"last update":"Mon Feb 12 16:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1279999999999999,"15min":1.042},"min":{"1min":0.82999999999999996,"5min":0.66900000000000004,"15min":0.65600000000000003},"max":{"1min":1.401,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.224},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.242,"15min":1.1359999999999999},"min":{"1min":0.85199999999999998,"5min":0.74199999999999999,"15min":0.58799999999999997},"max":{"1min":1.5740000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.2929999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.175,"15min":1.1339999999999999},"min":{"1min":0.879,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.4970000000000001,"5min":1.718,"15min":1.718},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.403,"15min":1.343},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.7909999999999999,"5min":2.0110000000000001,"15min":2.302},"last":1.532}]},{"osd":8,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5249999999999999,"15min":1.5129999999999999},"min":{"1min":1.073,"5min":1.073,"15min":0.88400000000000001},"max":{"1min":1.8100000000000001,"5min":2.093,"15min":2.1949999999999998},"last":1.476},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4590000000000001,"15min":1.423},"min":{"1min":1.133,"5min":1.038,"15min":0.93700000000000006},"max":{"1min":1.931,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.7849999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.601,"15min":1.5609999999999999},"min":{"1min":1.1739999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"max":{"1min":1.897,"5min":2.173,"15min":2.173},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.528,"15min":1.474},"min":{"1min":1.1539999999999999,"5min":1.109,"15min":1.109},"max":{"1min":2.0009999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.7130000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.5429999999999999,"15min":1.5149999999999999},"min":{"1min":1.236,"5min":1.097,"15min":1.097},"max":{"1min":1.9790000000000001,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.851},{"interface":"front","average":{"1min":1.466,"5min":1.546,"15min":1.5629999999999999},"min":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"max":{"1min":2.0009999999999999,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":1.6619999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297824,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447328,"statfs":{"total":95995035648,"available":95690063872,"internally_reserved":0,"allocated":745472,"data_stored":334773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0609999999999999,"15min":0.90800000000000003},"min":{"1min":0.65100000000000002,"5min":0.63100000000000001,"15min":0.57699999999999996},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.448},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0349999999999999,"15min":0.93100000000000005},"min":{"1min":0.79500000000000004,"5min":0.66300000000000003,"15min":0.46999999999999997},"max":{"1min":1.4279999999999999,"5min":1.524,"15min":1.524},"last":1.089}]},{"osd":1,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1080000000000001,"15min":0.96399999999999997},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.59999999999999998},"max":{"1min":1.3600000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.028,"15min":0.92400000000000004},"min":{"1min":0.75800000000000001,"5min":0.74099999999999999,"15min":0.59099999999999997},"max":{"1min":1.3660000000000001,"5min":1.381,"15min":1.381},"last":1.2190000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1160000000000001,"15min":0.99399999999999999},"min":{"1min":0.75800000000000001,"5min":0.622,"15min":0.57999999999999996},"max":{"1min":1.5309999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.169,"5min":1.099,"15min":1.0329999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.58899999999999997},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.89900000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1100000000000001,"15min":1.0149999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70299999999999996},"max":{"1min":1.544,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.2},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.202,"15min":1.0920000000000001},"min":{"1min":0.82599999999999996,"5min":0.72699999999999998,"15min":0.66400000000000003},"max":{"1min":1.5589999999999999,"5min":1.726,"15min":1.726},"last":1.3520000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.999,"15min":0.92000000000000004},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.56499999999999995},"max":{"1min":1.4510000000000001,"5min":1.889,"15min":1.889},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.181,"15min":1.02},"min":{"1min":0.86599999999999999,"5min":0.68200000000000005,"15min":0.52700000000000002},"max":{"1min":1.712,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.117}]},{"osd":6,"last update":"Mon Feb 12 16:02:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.21,"15min":1.087},"min":{"1min":0.69999999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2849999999999999,"15min":1.1379999999999999},"min":{"1min":0.86699999999999999,"5min":0.68300000000000005,"15min":0.40400000000000003},"max":{"1min":1.5940000000000001,"5min":1.8360000000000001,"15min":1.903},"last":1.2450000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2829999999999999,"15min":1.1699999999999999},"min":{"1min":0.90300000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.6739999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.429,"15min":1.2470000000000001},"min":{"1min":1.1359999999999999,"5min":0.69099999999999995,"15min":0.435},"max":{"1min":1.772,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.7569999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.532,"15min":1.427},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.9850000000000001,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.603,"5min":1.466,"15min":1.331},"min":{"1min":1.347,"5min":1.0860000000000001,"15min":0.83999999999999997},"max":{"1min":1.96,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.8380000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:02:09 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.5569999999999999,"15min":1.492},"min":{"1min":1.256,"5min":1.0469999999999999,"15min":1.0469999999999999},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.569},{"interface":"front","average":{"1min":1.663,"5min":1.6319999999999999,"15min":1.587},"min":{"1min":1.397,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":2,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.903}]},{"osd":10,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4790000000000001,"15min":1.49},"min":{"1min":1.149,"5min":1.145,"15min":1.145},"max":{"1min":1.8340000000000001,"5min":1.835,"15min":1.835},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.617,"15min":1.631},"min":{"1min":1.258,"5min":1.222,"15min":1.222},"max":{"1min":1.9410000000000001,"5min":2.254,"15min":2.254},"last":1.722}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:02:59.612 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-12T16:02:59.613 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-12T16:02:59.613 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-12T16:02:59.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph health --format=json 2024-02-12T16:02:59.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:02:59 smithi123 bash[19119]: audit 2024-02-12T16:02:58.933811+0000 mgr.y (mgr.14146) 516 : audit [DBG] from='client.14736 v1:172.21.15.121:0/217496051' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:02:59.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:02:59 smithi121 bash[18489]: audit 2024-02-12T16:02:58.933811+0000 mgr.y (mgr.14146) 516 : audit [DBG] from='client.14736 v1:172.21.15.121:0/217496051' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:02:59 smithi169 bash[19604]: audit 2024-02-12T16:02:58.933811+0000 mgr.y (mgr.14146) 516 : audit [DBG] from='client.14736 v1:172.21.15.121:0/217496051' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:00 smithi123 bash[19119]: cluster 2024-02-12T16:02:59.044876+0000 osd.10 (osd.10) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:00 smithi123 bash[19119]: cluster 2024-02-12T16:02:59.044963+0000 osd.10 (osd.10) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:00 smithi123 bash[19119]: cluster 2024-02-12T16:02:59.517856+0000 mgr.y (mgr.14146) 517 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:00 smithi123 bash[19119]: cluster 2024-02-12T16:02:59.586257+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e136: 12 total, 12 up, 12 in 2024-02-12T16:03:00.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:00 smithi121 bash[18489]: cluster 2024-02-12T16:02:59.044876+0000 osd.10 (osd.10) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:00.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:00 smithi121 bash[18489]: cluster 2024-02-12T16:02:59.044963+0000 osd.10 (osd.10) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:00.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:00 smithi121 bash[18489]: cluster 2024-02-12T16:02:59.517856+0000 mgr.y (mgr.14146) 517 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:00.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:00 smithi121 bash[18489]: cluster 2024-02-12T16:02:59.586257+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e136: 12 total, 12 up, 12 in 2024-02-12T16:03:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:00 smithi169 bash[19604]: cluster 2024-02-12T16:02:59.044876+0000 osd.10 (osd.10) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:00 smithi169 bash[19604]: cluster 2024-02-12T16:02:59.044963+0000 osd.10 (osd.10) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:00 smithi169 bash[19604]: cluster 2024-02-12T16:02:59.517856+0000 mgr.y (mgr.14146) 517 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:00 smithi169 bash[19604]: cluster 2024-02-12T16:02:59.586257+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e136: 12 total, 12 up, 12 in 2024-02-12T16:03:02.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:02 smithi123 bash[19119]: cluster 2024-02-12T16:03:01.518448+0000 mgr.y (mgr.14146) 518 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:02.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:02 smithi121 bash[18489]: cluster 2024-02-12T16:03:01.518448+0000 mgr.y (mgr.14146) 518 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:02 smithi169 bash[19604]: cluster 2024-02-12T16:03:01.518448+0000 mgr.y (mgr.14146) 518 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:04.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:04 smithi123 bash[19119]: cluster 2024-02-12T16:03:03.519089+0000 mgr.y (mgr.14146) 519 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:04.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:04 smithi123 bash[19119]: cluster 2024-02-12T16:03:03.789425+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:04.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:04 smithi123 bash[19119]: cluster 2024-02-12T16:03:03.789533+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:04.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:04 smithi121 bash[18489]: cluster 2024-02-12T16:03:03.519089+0000 mgr.y (mgr.14146) 519 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:04.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:04 smithi121 bash[18489]: cluster 2024-02-12T16:03:03.789425+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:04.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:04 smithi121 bash[18489]: cluster 2024-02-12T16:03:03.789533+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:04 smithi169 bash[19604]: cluster 2024-02-12T16:03:03.519089+0000 mgr.y (mgr.14146) 519 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:04 smithi169 bash[19604]: cluster 2024-02-12T16:03:03.789425+0000 osd.3 (osd.3) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:04 smithi169 bash[19604]: cluster 2024-02-12T16:03:03.789533+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:05.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:05 smithi121 bash[18489]: cluster 2024-02-12T16:03:03.914567+0000 osd.2 (osd.2) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:05.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:05 smithi121 bash[18489]: cluster 2024-02-12T16:03:03.914664+0000 osd.2 (osd.2) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:05.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:05 smithi121 bash[18489]: cluster 2024-02-12T16:03:04.599006+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e137: 12 total, 12 up, 12 in 2024-02-12T16:03:05.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:05 smithi123 bash[19119]: cluster 2024-02-12T16:03:03.914567+0000 osd.2 (osd.2) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:05.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:05 smithi123 bash[19119]: cluster 2024-02-12T16:03:03.914664+0000 osd.2 (osd.2) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:05.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:05 smithi123 bash[19119]: cluster 2024-02-12T16:03:04.599006+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e137: 12 total, 12 up, 12 in 2024-02-12T16:03:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:05 smithi169 bash[19604]: cluster 2024-02-12T16:03:03.914567+0000 osd.2 (osd.2) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:05 smithi169 bash[19604]: cluster 2024-02-12T16:03:03.914664+0000 osd.2 (osd.2) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:05 smithi169 bash[19604]: cluster 2024-02-12T16:03:04.599006+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e137: 12 total, 12 up, 12 in 2024-02-12T16:03:06.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:06 smithi123 bash[19119]: cluster 2024-02-12T16:03:05.519774+0000 mgr.y (mgr.14146) 520 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:06.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:06 smithi121 bash[18489]: cluster 2024-02-12T16:03:05.519774+0000 mgr.y (mgr.14146) 520 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:06 smithi169 bash[19604]: cluster 2024-02-12T16:03:05.519774+0000 mgr.y (mgr.14146) 520 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:07.421 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:03:07.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:07 smithi123 bash[19119]: audit 2024-02-12T16:03:07.413042+0000 mon.a (mon.0) 810 : audit [DBG] from='client.? v1:172.21.15.121:0/3487535028' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-12T16:03:07.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:07 smithi121 bash[18489]: audit 2024-02-12T16:03:07.413042+0000 mon.a (mon.0) 810 : audit [DBG] from='client.? v1:172.21.15.121:0/3487535028' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-12T16:03:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:07 smithi169 bash[19604]: audit 2024-02-12T16:03:07.413042+0000 mon.a (mon.0) 810 : audit [DBG] from='client.? v1:172.21.15.121:0/3487535028' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-12T16:03:08.097 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-12T16:03:08.098 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-12T16:03:08.098 INFO:tasks.cephadm:Setup complete, yielding 2024-02-12T16:03:08.098 INFO:teuthology.run_tasks:Running task exec... 2024-02-12T16:03:08.108 INFO:teuthology.task.exec:Executing custom commands... 2024-02-12T16:03:08.108 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi121.front.sepia.ceph.com 2024-02-12T16:03:08.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "active": true, 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "last_optimize_duration": "0:00:00.000449", 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "last_optimize_started": "Mon Feb 12 16:02:17 2024", 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "upmap", 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "no_optimization_needed": true, 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-02-12T16:03:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "plans": [] 2024-02-12T16:03:08.622 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:03:08.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-02-12T16:03:08.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:08 smithi123 bash[19119]: cluster 2024-02-12T16:03:07.520389+0000 mgr.y (mgr.14146) 521 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:08.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:08 smithi121 bash[18489]: cluster 2024-02-12T16:03:07.520389+0000 mgr.y (mgr.14146) 521 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:08 smithi169 bash[19604]: cluster 2024-02-12T16:03:07.520389+0000 mgr.y (mgr.14146) 521 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:09.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-02-12T16:03:09.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:09 smithi121 bash[18489]: audit 2024-02-12T16:03:08.611196+0000 mgr.y (mgr.14146) 522 : audit [DBG] from='client.14745 v1:172.21.15.121:0/1185265298' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:09.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:09 smithi121 bash[18489]: audit 2024-02-12T16:03:08.965835+0000 mgr.y (mgr.14146) 523 : audit [DBG] from='client.14748 v1:172.21.15.121:0/479649506' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:09.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:09 smithi121 bash[18489]: audit 2024-02-12T16:03:08.967666+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-02-12T16:03:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:09 smithi169 bash[19604]: audit 2024-02-12T16:03:08.611196+0000 mgr.y (mgr.14146) 522 : audit [DBG] from='client.14745 v1:172.21.15.121:0/1185265298' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:09 smithi169 bash[19604]: audit 2024-02-12T16:03:08.965835+0000 mgr.y (mgr.14146) 523 : audit [DBG] from='client.14748 v1:172.21.15.121:0/479649506' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:09 smithi169 bash[19604]: audit 2024-02-12T16:03:08.967666+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-02-12T16:03:10.063 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-12T16:03:10.074 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-12T16:03:10.074 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri10-testing-2024-02-08-0854-pacific'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(MON_DOWN\\)', 'osds down', 'mons down'], 'mon_bind_msgr2': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}}, 'workunit': {'branch': 'wip-tracker-64343-pacific', 'sha1': '57bd6abdec7bb457ae7999d9c96682e9ac678e27'}} 2024-02-12T16:03:10.074 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-12T16:03:10.074 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-12T16:03:10.075 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-12T16:03:10.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:03:10.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:09 smithi123 bash[19119]: audit 2024-02-12T16:03:08.611196+0000 mgr.y (mgr.14146) 522 : audit [DBG] from='client.14745 v1:172.21.15.121:0/1185265298' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:10.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:09 smithi123 bash[19119]: audit 2024-02-12T16:03:08.965835+0000 mgr.y (mgr.14146) 523 : audit [DBG] from='client.14748 v1:172.21.15.121:0/479649506' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:10.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:09 smithi123 bash[19119]: audit 2024-02-12T16:03:08.967666+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-02-12T16:03:10.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:09.631382+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-02-12T16:03:10.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: cluster 2024-02-12T16:03:09.631497+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e138: 12 total, 12 up, 12 in 2024-02-12T16:03:10.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:09.632315+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-02-12T16:03:10.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:09.636931+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:03:10.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:09.637143+0000 mon.a (mon.0) 816 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:03:10.914 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:09.637378+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:03:10.914 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:09.642417+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:10.914 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:09.668201+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:03:10.914 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:10 smithi121 bash[18489]: audit 2024-02-12T16:03:10.038680+0000 mgr.y (mgr.14146) 525 : audit [DBG] from='client.24655 v1:172.21.15.121:0/281987705' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:09.631382+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-02-12T16:03:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: cluster 2024-02-12T16:03:09.631497+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e138: 12 total, 12 up, 12 in 2024-02-12T16:03:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:09.632315+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-02-12T16:03:11.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:09.636931+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:03:11.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:09.637143+0000 mon.a (mon.0) 816 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:03:11.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:09.637378+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:03:11.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:09.642417+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:11.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:09.668201+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:03:11.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:10 smithi169 bash[19604]: audit 2024-02-12T16:03:10.038680+0000 mgr.y (mgr.14146) 525 : audit [DBG] from='client.24655 v1:172.21.15.121:0/281987705' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:11.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:09.631382+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-02-12T16:03:11.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: cluster 2024-02-12T16:03:09.631497+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e138: 12 total, 12 up, 12 in 2024-02-12T16:03:11.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:09.632315+0000 mon.a (mon.0) 814 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-02-12T16:03:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:09.636931+0000 mon.a (mon.0) 815 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:03:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:09.637143+0000 mon.a (mon.0) 816 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:03:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:09.637378+0000 mon.a (mon.0) 817 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:03:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:09.642417+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:09.668201+0000 mon.a (mon.0) 819 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:03:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:10 smithi123 bash[19119]: audit 2024-02-12T16:03:10.038680+0000 mgr.y (mgr.14146) 525 : audit [DBG] from='client.24655 v1:172.21.15.121:0/281987705' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-02-12T16:03:12.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:12 smithi121 bash[18489]: cluster 2024-02-12T16:03:11.521532+0000 mgr.y (mgr.14146) 526 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:12 smithi169 bash[19604]: cluster 2024-02-12T16:03:11.521532+0000 mgr.y (mgr.14146) 526 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:13.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:12 smithi123 bash[19119]: cluster 2024-02-12T16:03:11.521532+0000 mgr.y (mgr.14146) 526 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:13.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:13 smithi121 bash[18489]: cluster 2024-02-12T16:03:12.713721+0000 osd.7 (osd.7) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:13.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:13 smithi121 bash[18489]: cluster 2024-02-12T16:03:12.713806+0000 osd.7 (osd.7) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:13 smithi169 bash[19604]: cluster 2024-02-12T16:03:12.713721+0000 osd.7 (osd.7) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:13 smithi169 bash[19604]: cluster 2024-02-12T16:03:12.713806+0000 osd.7 (osd.7) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:14.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:13 smithi123 bash[19119]: cluster 2024-02-12T16:03:12.713721+0000 osd.7 (osd.7) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:14.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:13 smithi123 bash[19119]: cluster 2024-02-12T16:03:12.713806+0000 osd.7 (osd.7) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:14.490 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:14.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:14 smithi121 bash[18489]: cluster 2024-02-12T16:03:13.522117+0000 mgr.y (mgr.14146) 527 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:14.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:14 smithi121 bash[18489]: cluster 2024-02-12T16:03:13.651742+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e139: 12 total, 12 up, 12 in 2024-02-12T16:03:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:14 smithi169 bash[19604]: cluster 2024-02-12T16:03:13.522117+0000 mgr.y (mgr.14146) 527 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:14 smithi169 bash[19604]: cluster 2024-02-12T16:03:13.651742+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e139: 12 total, 12 up, 12 in 2024-02-12T16:03:15.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:14 smithi123 bash[19119]: cluster 2024-02-12T16:03:13.522117+0000 mgr.y (mgr.14146) 527 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:15.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:14 smithi123 bash[19119]: cluster 2024-02-12T16:03:13.651742+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e139: 12 total, 12 up, 12 in 2024-02-12T16:03:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:16 smithi169 bash[19604]: audit 2024-02-12T16:03:15.179119+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:16 smithi169 bash[19604]: audit 2024-02-12T16:03:15.462102+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:03:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:16 smithi169 bash[19604]: audit 2024-02-12T16:03:15.463101+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:03:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:16 smithi169 bash[19604]: audit 2024-02-12T16:03:15.470466+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:16 smithi169 bash[19604]: cluster 2024-02-12T16:03:15.522583+0000 mgr.y (mgr.14146) 528 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:16.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:16 smithi123 bash[19119]: audit 2024-02-12T16:03:15.179119+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:16.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:16 smithi123 bash[19119]: audit 2024-02-12T16:03:15.462102+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:03:16.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:16 smithi123 bash[19119]: audit 2024-02-12T16:03:15.463101+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:03:16.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:16 smithi123 bash[19119]: audit 2024-02-12T16:03:15.470466+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:16.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:16 smithi123 bash[19119]: cluster 2024-02-12T16:03:15.522583+0000 mgr.y (mgr.14146) 528 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:16.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:16 smithi121 bash[18489]: audit 2024-02-12T16:03:15.179119+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:16.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:16 smithi121 bash[18489]: audit 2024-02-12T16:03:15.462102+0000 mon.a (mon.0) 822 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:03:16.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:16 smithi121 bash[18489]: audit 2024-02-12T16:03:15.463101+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:03:16.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:16 smithi121 bash[18489]: audit 2024-02-12T16:03:15.470466+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:03:16.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:16 smithi121 bash[18489]: cluster 2024-02-12T16:03:15.522583+0000 mgr.y (mgr.14146) 528 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:17.807 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:03:17.809 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:03:18.405 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":580,"stamp":"2024-02-12T16:03:17.522763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":3572940,"kb_used_data":7788,"kb_used_omap":0,"kb_used_meta":3565056,"kb_avail":1121368884,"statfs":{"total":1151940427776,"available":1148281737216,"internally_reserved":0,"allocated":7974912,"data_stored":3620932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3650617344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":142,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-12T16:03:04.602959+0000","last_change":"2024-02-12T16:02:49.550827+0000","last_active":"2024-02-12T16:03:04.602959+0000","last_peered":"2024-02-12T16:03:04.602959+0000","last_clean":"2024-02-12T16:03:04.602959+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:03:04.602959+0000","last_undegraded":"2024-02-12T16:03:04.602959+0000","last_fullsized":"2024-02-12T16:03:04.602959+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:02:49.550626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:00:34.507052+0000","last_clean_scrub_stamp":"2024-02-12T16:02:49.550626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":299296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0409999999999999,"15min":1.0389999999999999},"min":{"1min":0.71899999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.482,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.089,"15min":1.123},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.4379999999999999,"5min":2.871,"15min":2.871},"last":1.1299999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.048,"15min":1.1020000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1990000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0780000000000001,"15min":1.0880000000000001},"min":{"1min":0.73199999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5920000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.85199999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.119,"15min":1.157},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.466,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.117,"5min":1.1519999999999999,"15min":1.24},"min":{"1min":0.77200000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3210000000000002,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.3440000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1739999999999999,"15min":1.2190000000000001},"min":{"1min":0.92600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6839999999999999,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.745},{"interface":"front","average":{"1min":1.032,"5min":1.151,"15min":1.2070000000000001},"min":{"1min":0.75800000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.393,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":0.91400000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2749999999999999,"15min":1.3720000000000001},"min":{"1min":0.81200000000000006,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4770000000000001,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":1.081},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.355,"15min":1.4390000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8400000000000001,"5min":4.0789999999999997,"15min":4.0789999999999997},"last":1.236}]},{"osd":5,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.421,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.754,"5min":3.044,"15min":3.044},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.423,"15min":1.5069999999999999},"min":{"1min":0.93500000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.6279999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":1.272}]},{"osd":6,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4379999999999999,"15min":1.5349999999999999},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.827,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.464,"5min":1.5349999999999999,"15min":1.619},"min":{"1min":1.038,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9279999999999999,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.4159999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.534,"15min":1.6259999999999999},"min":{"1min":0.90000000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.8779999999999999,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.159},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6259999999999999,"15min":1.722},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.921,"5min":3.8370000000000002,"15min":3.8370000000000002},"last":1.214}]},{"osd":8,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.569,"15min":1.6519999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.974,"5min":4,"15min":4},"last":1.038},{"interface":"front","average":{"1min":1.534,"5min":1.647,"15min":1.742},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.9490000000000001,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.3049999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.5700000000000001,"15min":1.694},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.772,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.181},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.742,"15min":1.8360000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0230000000000001,"5min":4.3949999999999996,"15min":4.3949999999999996},"last":1.4710000000000001}]}]},{"osd":10,"up_from":83,"seq":356482285637,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":299296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.90000000000000002,"15min":0.89600000000000002},"min":{"1min":0.69699999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1919999999999999,"5min":1.413,"15min":1.413},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.036,"5min":1.026,"15min":1.0660000000000001},"min":{"1min":0.80200000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.335,"5min":1.573,"15min":1.573},"last":0.81799999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0009999999999999,"15min":0.98299999999999998},"min":{"1min":0.66400000000000003,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.4490000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1,"15min":0.96599999999999997},"min":{"1min":0.72599999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.395,"5min":1.492,"15min":1.492},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.04,"15min":1.016},"min":{"1min":0.73499999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.998,"15min":1.0149999999999999},"min":{"1min":0.80600000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.294,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.169}]},{"osd":4,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2230000000000001,"15min":1.1279999999999999},"min":{"1min":0.83299999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":1.093},{"interface":"front","average":{"1min":1.157,"5min":1.228,"15min":1.2290000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.5640000000000001,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.6020000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.171,"15min":1.1830000000000001},"min":{"1min":0.872,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.421,"5min":2.048,"15min":2.048},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.3200000000000001,"15min":1.306},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7110000000000001,"5min":1.788,"15min":1.788},"last":1.3620000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2050000000000001,"15min":1.224},"min":{"1min":0.82899999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.407,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.075},{"interface":"front","average":{"1min":1.387,"5min":1.4239999999999999,"15min":1.3600000000000001},"min":{"1min":1.0309999999999999,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.8089999999999999,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.458}]},{"osd":7,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3759999999999999,"15min":1.3},"min":{"1min":1.0720000000000001,"5min":0.755,"15min":0.755},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.3740000000000001,"15min":1.3799999999999999},"min":{"1min":1.056,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.5409999999999999,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.343}]},{"osd":8,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.423,"15min":1.359},"min":{"1min":0.84499999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":2,"5min":2,"15min":2},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.48,"15min":1.3620000000000001},"min":{"1min":1.0049999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.659}]},{"osd":9,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.5389999999999999,"15min":1.4570000000000001},"min":{"1min":1.1439999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.982,"5min":2.02,"15min":2.02},"last":1.708},{"interface":"front","average":{"1min":1.538,"5min":1.5349999999999999,"15min":1.429},"min":{"1min":1.1319999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.5009999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.595,"15min":1.6100000000000001},"min":{"1min":1.2989999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.581},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.6180000000000001,"15min":1.6379999999999999},"min":{"1min":1.2989999999999999,"5min":1.25,"15min":1.25},"max":{"1min":1.919,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.75}]}]},{"osd":9,"up_from":74,"seq":317827579983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":305152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.94299999999999995,"15min":0.93300000000000005},"min":{"1min":0.68000000000000005,"5min":0.623,"15min":0.623},"max":{"1min":1.1539999999999999,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.93300000000000005,"15min":0.96899999999999997},"min":{"1min":0.57899999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3,"5min":1.4419999999999999,"15min":2.2480000000000002},"last":0.69299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.052,"15min":1},"min":{"1min":0.65600000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.399,"5min":1.48,"15min":1.48},"last":1.006},{"interface":"front","average":{"1min":0.995,"5min":1.0529999999999999,"15min":1.0349999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3879999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96799999999999997,"15min":0.93700000000000006},"min":{"1min":0.71799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.252,"5min":1.738,"15min":1.738},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96499999999999997,"15min":1.022},"min":{"1min":0.71799999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.194,"5min":1.369,"15min":1.601},"last":1.1699999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.97499999999999998,"15min":0.96799999999999997},"min":{"1min":0.68300000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.198,"5min":1.4470000000000001,"15min":2.1549999999999998},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.121,"15min":1.111},"min":{"1min":0.72999999999999998,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.5469999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.177,"15min":1.1719999999999999},"min":{"1min":0.91400000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.2629999999999999,"5min":1.9330000000000001,"15min":2.113},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1759999999999999,"15min":1.1859999999999999},"min":{"1min":0.88900000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.355,"5min":1.619,"15min":2.3260000000000001},"last":0.96599999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2290000000000001,"15min":1.2250000000000001},"min":{"1min":1.0209999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.4970000000000001,"5min":1.756,"15min":2.4399999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.329,"15min":1.2749999999999999},"min":{"1min":1.234,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.633,"5min":1.851,"15min":2.2599999999999998},"last":1.234}]},{"osd":6,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3220000000000001,"15min":1.3240000000000001},"min":{"1min":0.995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.9139999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.351},"min":{"1min":0.996,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8660000000000001,"15min":2.48},"last":1.0960000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.284,"15min":1.3160000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.401,"5min":2.012,"15min":2.3519999999999999},"last":1.367},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4350000000000001,"15min":1.411},"min":{"1min":0.98099999999999998,"5min":0.90200000000000002,"15min":0.875},"max":{"1min":1.772,"5min":1.772,"15min":2.0249999999999999},"last":1.4490000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.393,"15min":1.367},"min":{"1min":0.86399999999999999,"5min":0.64700000000000002,"15min":0.61099999999999999},"max":{"1min":1.704,"5min":1.792,"15min":2.1299999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.377,"15min":1.355},"min":{"1min":1.069,"5min":0.62,"15min":0.58499999999999996},"max":{"1min":1.4379999999999999,"5min":1.9570000000000001,"15min":2.3889999999999998},"last":1.286}]},{"osd":10,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.4330000000000001,"15min":1.4299999999999999},"min":{"1min":1.0780000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.597,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.3979999999999999,"15min":1.3560000000000001},"min":{"1min":1.05,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.8400000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.482}]}]},{"osd":8,"up_from":65,"seq":279172874325,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":299316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95999999999999996,"15min":0.85599999999999998},"min":{"1min":0.61699999999999999,"5min":0.57899999999999996,"15min":0.57699999999999996},"max":{"1min":1.1839999999999999,"5min":1.488,"15min":1.488},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.96499999999999997,"15min":0.89500000000000002},"min":{"1min":0.71699999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.589,"5min":1.623,"15min":1.623},"last":0.70499999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.88800000000000001,"15min":0.84999999999999998},"min":{"1min":0.64900000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.3480000000000001,"5min":1.508,"15min":1.508},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0249999999999999,"15min":0.95599999999999996},"min":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.355,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.01,"15min":0.88},"min":{"1min":0.68700000000000006,"5min":0.59499999999999997,"15min":0.59199999999999997},"max":{"1min":1.194,"5min":1.764,"15min":1.764},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.173,"5min":1.141,"15min":1.0309999999999999},"min":{"1min":0.91400000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.637,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.86799999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.083,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.64800000000000002,"15min":0.60499999999999998},"max":{"1min":1.365,"5min":1.698,"15min":1.698},"last":1.25},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.22,"15min":1.04},"min":{"1min":0.96099999999999997,"5min":0.623,"15min":0.623},"max":{"1min":1.3280000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.0449999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2629999999999999,"15min":1.157},"min":{"1min":0.94599999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.4339999999999999,"5min":1.677,"15min":1.839},"last":1.171},{"interface":"front","average":{"1min":1.3,"5min":1.3080000000000001,"15min":1.1160000000000001},"min":{"1min":0.80800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.772,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.444}]},{"osd":5,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1990000000000001,"15min":1.155},"min":{"1min":0.76900000000000002,"5min":0.76700000000000002,"15min":0.75800000000000001},"max":{"1min":1.4379999999999999,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.341,"5min":1.3420000000000001,"15min":1.2030000000000001},"min":{"1min":1.157,"5min":0.71899999999999997,"15min":0.68899999999999995},"max":{"1min":1.508,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":1.3280000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.1970000000000001},"min":{"1min":0.92800000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.915,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.3939999999999999,"15min":1.25},"min":{"1min":1.073,"5min":0.91500000000000004,"15min":0.876},"max":{"1min":1.5549999999999999,"5min":2.339,"15min":2.339},"last":1.196}]},{"osd":7,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4530000000000001,"15min":1.2629999999999999},"min":{"1min":1.0860000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6120000000000001,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4930000000000001,"15min":1.292},"min":{"1min":1.194,"5min":0.997,"15min":0.85499999999999998},"max":{"1min":1.7909999999999999,"5min":1.984,"15min":1.984},"last":1.3839999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:02:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4610000000000001,"15min":1.2470000000000001},"min":{"1min":1.0680000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.8620000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.294},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3380000000000001,"15min":1.218},"min":{"1min":0.96899999999999997,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7330000000000001,"5min":2.456,"15min":2.456},"last":1.1339999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4079999999999999,"15min":1.3640000000000001},"min":{"1min":1.002,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.23},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.502,"15min":1.462},"min":{"1min":0.99399999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.8069999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.468}]}]},{"osd":7,"up_from":56,"seq":240518168670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":299316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.036,"15min":0.98999999999999999},"min":{"1min":0.63900000000000001,"5min":0.60899999999999999,"15min":0.436},"max":{"1min":1.671,"5min":1.675,"15min":1.675},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.74199999999999999,"5min":0.64100000000000001,"15min":0.53400000000000003},"max":{"1min":1.171,"5min":1.353,"15min":1.353},"last":1.097}]},{"osd":2,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.98399999999999999,"15min":1.022},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3680000000000001,"5min":1.5980000000000001,"15min":1.8580000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.052,"15min":1.044},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.56499999999999995},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.93400000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.089,"15min":1.083},"min":{"1min":0.628,"5min":0.628,"15min":0.53400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":2.0049999999999999},"last":1},{"interface":"front","average":{"1min":1.014,"5min":1.046,"15min":1.093},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.355,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.91000000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95399999999999996,"15min":1.0069999999999999},"min":{"1min":0.69599999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4019999999999999,"5min":1.51,"15min":2.0449999999999999},"last":0.879},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.042},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4790000000000001,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.1659999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.095,"15min":1.077},"min":{"1min":0.82799999999999996,"5min":0.626,"15min":0.60099999999999998},"max":{"1min":1.4590000000000001,"5min":1.76,"15min":1.76},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1419999999999999,"15min":1.181},"min":{"1min":0.65800000000000003,"5min":0.62,"15min":0.46100000000000002},"max":{"1min":1.4259999999999999,"5min":1.677,"15min":2.1709999999999998},"last":1.1950000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1719999999999999,"15min":1.2030000000000001},"min":{"1min":0.78700000000000003,"5min":0.73099999999999998,"15min":0.64300000000000002},"max":{"1min":1.744,"5min":1.744,"15min":1.8069999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.204,"5min":1.1870000000000001,"15min":1.2230000000000001},"min":{"1min":0.77900000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.486,"5min":2.0960000000000001,"15min":2.1139999999999999},"last":0.79700000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":1.1910000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.081},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4099999999999999,"15min":1.446},"min":{"1min":1.028,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8180000000000001,"5min":1.905,"15min":2.0760000000000001},"last":1.5549999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.446,"15min":1.4379999999999999},"min":{"1min":1.0629999999999999,"5min":0.81699999999999995,"15min":0.751},"max":{"1min":1.8180000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4259999999999999,"15min":1.421},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.266}]},{"osd":10,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.504,"15min":1.5349999999999999},"min":{"1min":0.96499999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.542,"5min":1.502,"15min":1.5589999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9450000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.4299999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.55,"15min":1.5820000000000001},"min":{"1min":1.169,"5min":1.071,"15min":1.071},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.228},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.522,"15min":1.5209999999999999},"min":{"1min":1.087,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7849999999999999,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.639}]}]},{"osd":6,"up_from":48,"seq":206158430311,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":299316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.997,"15min":0.94899999999999995},"min":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.498},"max":{"1min":1.3460000000000001,"5min":1.492,"15min":1.492},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.98499999999999999,"15min":0.92500000000000004},"min":{"1min":0.72099999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.254,"5min":1.573,"15min":1.573},"last":1.421}]},{"osd":1,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.81100000000000005,"5min":0.67200000000000004,"15min":0.64300000000000002},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.0429999999999999,"15min":0.998},"min":{"1min":0.64000000000000001,"5min":0.62,"15min":0.60999999999999999},"max":{"1min":1.419,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0589999999999999,"15min":0.96799999999999997},"min":{"1min":0.76100000000000001,"5min":0.66000000000000003,"15min":0.55500000000000005},"max":{"1min":1.3129999999999999,"5min":2.383,"15min":2.383},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.109,"15min":1.004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.339}]},{"osd":3,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.032,"15min":0.95399999999999996},"min":{"1min":0.75700000000000001,"5min":0.68400000000000005,"15min":0.68100000000000005},"max":{"1min":1.3799999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.02},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.095,"15min":1.042},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.363,"5min":1.623,"15min":1.623},"last":1.0349999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.038,"15min":0.91000000000000003},"min":{"1min":0.68500000000000005,"5min":0.57199999999999995,"15min":0.41899999999999998},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.242},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.42299999999999999},"max":{"1min":1.554,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.97299999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.202,"15min":1.0069999999999999},"min":{"1min":0.79200000000000004,"5min":0.60299999999999998,"15min":0.46999999999999997},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.635},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.214,"15min":1.034},"min":{"1min":0.82999999999999996,"5min":0.69499999999999995,"15min":0.51800000000000002},"max":{"1min":1.538,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.88800000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.321,"15min":1.1890000000000001},"min":{"1min":0.875,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8839999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.337,"15min":1.2729999999999999},"min":{"1min":1.147,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.631,"5min":1.889,"15min":1.889},"last":1.2190000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.502,"15min":1.397},"min":{"1min":1.0960000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.708},{"interface":"front","average":{"1min":1.544,"5min":1.534,"15min":1.4259999999999999},"min":{"1min":1.083,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.0739999999999998,"5min":2.093,"15min":2.093},"last":1.1799999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:03:09 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.6100000000000001,"15min":1.4890000000000001},"min":{"1min":1.3149999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":2.258},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5309999999999999,"15min":1.421},"min":{"1min":1.2829999999999999,"5min":1.046,"15min":0.95899999999999996},"max":{"1min":2.0150000000000001,"5min":2.125,"15min":2.2469999999999999},"last":1.3899999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":1.6559999999999999,"15min":1.627},"min":{"1min":1.3759999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"max":{"1min":2.1200000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":2.343},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.591,"15min":1.569},"min":{"1min":1.3320000000000001,"5min":1.081,"15min":1.081},"max":{"1min":2.0070000000000001,"5min":2.367,"15min":2.367},"last":1.319}]}]},{"osd":1,"up_from":13,"seq":55834574990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":305152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.85499999999999998,"15min":0.68899999999999995},"min":{"1min":0.66200000000000003,"5min":0.41099999999999998,"15min":0.377},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.749},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.94799999999999995,"15min":0.75700000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.28899999999999998},"max":{"1min":1.179,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.94499999999999995,"15min":0.75900000000000001},"min":{"1min":0.63800000000000001,"5min":0.50700000000000001,"15min":0.379},"max":{"1min":1.252,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96799999999999997,"15min":0.79100000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.34999999999999998},"max":{"1min":1.2669999999999999,"5min":1.54,"15min":1.54},"last":0.79800000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:02:16 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.98999999999999999,"15min":0.83799999999999997},"min":{"1min":0.71299999999999997,"5min":0.61799999999999999,"15min":0.61099999999999999},"max":{"1min":1.3,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0389999999999999,"15min":0.90000000000000002},"min":{"1min":0.72899999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.54,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.95099999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.181,"15min":1.048},"min":{"1min":0.92100000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.746,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.073},{"interface":"front","average":{"1min":1.325,"5min":1.264,"15min":1.1259999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.8720000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.181}]},{"osd":5,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3160000000000001,"15min":1.1439999999999999},"min":{"1min":1.034,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4390000000000001,"15min":1.234},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":1.0840000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3400000000000001,"15min":1.2330000000000001},"min":{"1min":0.91700000000000004,"5min":0.81399999999999995,"15min":0.80500000000000005},"max":{"1min":1.921,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.462,"15min":1.3380000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.573,"5min":2.573,"15min":2.573},"last":1.337}]},{"osd":7,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.4950000000000001,"15min":1.3939999999999999},"min":{"1min":1.0600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.974,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.387},{"interface":"front","average":{"1min":1.367,"5min":1.3700000000000001,"15min":1.343},"min":{"1min":0.89600000000000002,"5min":0.871,"15min":0.871},"max":{"1min":1.722,"5min":1.887,"15min":1.887},"last":1.4570000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.49,"15min":1.403},"min":{"1min":1.0069999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8049999999999999,"5min":2.355,"15min":2.355},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6479999999999999,"15min":1.506},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1349999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.599}]},{"osd":9,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":1.6759999999999999,"15min":1.6419999999999999},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":2.1669999999999998,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.5109999999999999,"15min":1.5309999999999999},"min":{"1min":1.075,"5min":1.075,"15min":1.075},"max":{"1min":1.772,"5min":2.133,"15min":2.133},"last":1.569}]},{"osd":10,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.5469999999999999,"15min":1.571},"min":{"1min":1.212,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.377},{"interface":"front","average":{"1min":1.738,"5min":1.7130000000000001,"15min":1.7529999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.4220000000000002,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.6739999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":305152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.81599999999999995,"15min":0.67400000000000004},"min":{"1min":0.621,"5min":0.44900000000000001,"15min":0.32500000000000001},"max":{"1min":1.0449999999999999,"5min":1.2170000000000001,"15min":1.3009999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.84099999999999997,"15min":0.70899999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.242,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.92500000000000004}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.873,"15min":0.76200000000000001},"min":{"1min":0.65300000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.2390000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.93300000000000005,"15min":0.86499999999999999},"min":{"1min":0.61099999999999999,"5min":0.53500000000000003,"15min":0.48099999999999998},"max":{"1min":1.1459999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":1.2050000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1599999999999999,"15min":1.081},"min":{"1min":0.78300000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0489999999999999,"15min":1.0049999999999999},"min":{"1min":0.71299999999999997,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.94199999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3680000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.2,"5min":1.2170000000000001,"15min":1.202},"min":{"1min":1.014,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.4239999999999999,"5min":1.772,"15min":1.772},"last":1.1339999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.198,"15min":1.1299999999999999},"min":{"1min":0.80400000000000005,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.254,"5min":1.2809999999999999,"15min":1.2090000000000001},"min":{"1min":0.96299999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.623,"5min":1.768,"15min":1.768},"last":1.3009999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.353,"15min":1.3560000000000001},"min":{"1min":0.97699999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7649999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.486,"15min":1.3360000000000001},"min":{"1min":1.262,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0059999999999998,"5min":2.1379999999999999,"15min":2.214},"last":1.5}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4910000000000001,"15min":1.403},"min":{"1min":1.3120000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.988,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.46,"15min":1.4870000000000001},"min":{"1min":0.95499999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.6140000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.022}]},{"osd":9,"last update":"Mon Feb 12 16:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.514,"15min":1.478},"min":{"1min":1.0449999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.506,"15min":1.5800000000000001},"min":{"1min":1.1659999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.0779999999999998,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":1.165}]},{"osd":10,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.51,"15min":1.6100000000000001},"min":{"1min":1.0229999999999999,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.8899999999999999,"5min":2.335,"15min":2.335},"last":1.109},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3859999999999999,"15min":1.4159999999999999},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.323}]},{"osd":11,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.5960000000000001,"15min":1.546},"min":{"1min":1.367,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.407,"15min":1.383},"min":{"1min":0.88500000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.444}]}]},{"osd":2,"up_from":21,"seq":90194313351,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":305152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.872,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.181},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88800000000000001,"15min":0.73499999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.47099999999999997},"max":{"1min":1.278,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.86599999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.98499999999999999,"15min":0.80000000000000004},"min":{"1min":0.59599999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3959999999999999,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.034,"5min":1.093,"15min":0.89200000000000002},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.42799999999999999},"max":{"1min":1.5209999999999999,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.96799999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.046,"15min":0.89000000000000001},"min":{"1min":0.79100000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.294,"5min":1.4710000000000001,"15min":1.5089999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.004,"5min":1.0149999999999999,"15min":0.89400000000000002},"min":{"1min":0.72999999999999998,"5min":0.61099999999999999,"15min":0.59199999999999997},"max":{"1min":1.3140000000000001,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1439999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1950000000000001,"15min":1.0640000000000001},"min":{"1min":0.92300000000000004,"5min":0.86599999999999999,"15min":0.70599999999999996},"max":{"1min":1.387,"5min":1.74,"15min":1.7669999999999999},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.96799999999999997,"15min":0.90600000000000003},"min":{"1min":0.71799999999999997,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.294,"5min":1.306,"15min":1.323},"last":1.038}]},{"osd":5,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2310000000000001,"15min":1.155},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.528,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.147,"15min":1.0600000000000001},"min":{"1min":0.85199999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.571,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":1.3180000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:03:07 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.149,"15min":1.089},"min":{"1min":0.94599999999999995,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.613,"5min":1.746,"15min":1.746},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.304,"15min":1.2090000000000001},"min":{"1min":0.84199999999999997,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7110000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.4550000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:59 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.383,"15min":1.341},"min":{"1min":0.85999999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.7490000000000001,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.492},{"interface":"front","average":{"1min":1.262,"5min":1.296,"15min":1.2649999999999999},"min":{"1min":0.91900000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.581,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.2190000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5109999999999999,"15min":1.427},"min":{"1min":1.1240000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9099999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.671},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4890000000000001,"15min":1.4810000000000001},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.6120000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:03:07 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.387,"15min":1.427},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.498,"15min":1.4830000000000001},"min":{"1min":1.024,"5min":1.024,"15min":1.024},"max":{"1min":2.117,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.728}]},{"osd":10,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.464,"15min":1.47},"min":{"1min":1.046,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8500000000000001,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5329999999999999,"15min":1.5489999999999999},"min":{"1min":1.1739999999999999,"5min":1.046,"15min":1.046},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.573}]}]},{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":299316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.84099999999999997,"15min":0.81899999999999995},"min":{"1min":0.56599999999999995,"5min":0.434,"15min":0.42999999999999999},"max":{"1min":1.1559999999999999,"5min":1.2589999999999999,"15min":1.3260000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.94599999999999995,"15min":0.82999999999999996},"min":{"1min":0.80900000000000005,"5min":0.56599999999999995,"15min":0.371},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.90500000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92800000000000005,"15min":0.85599999999999998},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.51900000000000002},"max":{"1min":1.278,"5min":1.466,"15min":1.5529999999999999},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.077,"15min":0.92000000000000004},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.44800000000000001},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.4319999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.751,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.2749999999999999,"5min":1.278,"15min":1.593},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.145,"15min":1.0489999999999999},"min":{"1min":0.81299999999999994,"5min":0.75900000000000001,"15min":0.60699999999999998},"max":{"1min":1.4710000000000001,"5min":1.669,"15min":1.669},"last":1.3220000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2370000000000001,"15min":1.153},"min":{"1min":0.96599999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6140000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.74},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.089,"15min":1.095},"min":{"1min":0.76900000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1830000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.0269999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.194,"15min":1.1519999999999999},"min":{"1min":0.84599999999999997,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.7629999999999999,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.2989999999999999,"15min":1.258},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.7030000000000001,"5min":1.7689999999999999,"15min":1.8120000000000001},"last":1.591}]},{"osd":7,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3129999999999999,"15min":1.3149999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.66600000000000004},"max":{"1min":1.7450000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3819999999999999,"15min":1.343},"min":{"1min":1.0309999999999999,"5min":0.85299999999999998,"15min":0.77800000000000002},"max":{"1min":1.768,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.282}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.3160000000000001,"15min":1.3740000000000001},"min":{"1min":0.93600000000000005,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.669,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.7769999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.49,"15min":1.4850000000000001},"min":{"1min":0.98999999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.8080000000000001,"5min":2.242,"15min":2.242},"last":1.6739999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.548,"15min":1.554},"min":{"1min":1.026,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.901,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.571},{"interface":"front","average":{"1min":1.548,"5min":1.5009999999999999,"15min":1.5529999999999999},"min":{"1min":1.1439999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":1.8720000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4450000000000001,"15min":1.4790000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":1.7869999999999999,"5min":2.113,"15min":2.113},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.546,"5min":1.5629999999999999,"15min":1.581},"min":{"1min":1.1279999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.819,"5min":1.974,"15min":1.974},"last":1.421}]},{"osd":11,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.4910000000000001,"15min":1.4950000000000001},"min":{"1min":1.2330000000000001,"5min":1.073,"15min":1.073},"max":{"1min":1.867,"5min":1.925,"15min":1.925},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.597,"5min":1.5780000000000001,"15min":1.536},"min":{"1min":1.0469999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.895,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.51}]}]},{"osd":4,"up_from":32,"seq":137438953589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":299316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0720000000000001,"15min":0.94899999999999995},"min":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.56999999999999995},"max":{"1min":1.476,"5min":1.736,"15min":1.736},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.069,"5min":1.0600000000000001,"15min":0.93200000000000005},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.58699999999999997},"max":{"1min":1.3140000000000001,"5min":1.617,"15min":1.617},"last":1.244}]},{"osd":1,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0609999999999999,"15min":0.96199999999999997},"min":{"1min":0.78600000000000003,"5min":0.72099999999999997,"15min":0.63900000000000001},"max":{"1min":1.4019999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.349},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.115,"15min":0.96599999999999997},"min":{"1min":0.81599999999999995,"5min":0.68799999999999994,"15min":0.51400000000000001},"max":{"1min":1.423,"5min":1.806,"15min":1.806},"last":0.97299999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.052,"15min":0.93300000000000005},"min":{"1min":0.68999999999999995,"5min":0.68899999999999995,"15min":0.60299999999999998},"max":{"1min":1.3540000000000001,"5min":1.593,"15min":1.593},"last":0.877},{"interface":"front","average":{"1min":0.997,"5min":1.0209999999999999,"15min":0.95799999999999996},"min":{"1min":0.71499999999999997,"5min":0.68000000000000005,"15min":0.64900000000000002},"max":{"1min":1.429,"5min":1.4450000000000001,"15min":1.5549999999999999},"last":1.05}]},{"osd":3,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.121,"15min":0.98799999999999999},"min":{"1min":0.79600000000000004,"5min":0.72599999999999998,"15min":0.57599999999999996},"max":{"1min":1.47,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.119},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.109,"15min":0.97399999999999998},"min":{"1min":0.91400000000000003,"5min":0.73299999999999998,"15min":0.55100000000000005},"max":{"1min":1.4470000000000001,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.93400000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1060000000000001,"15min":0.92400000000000004},"min":{"1min":0.69599999999999995,"5min":0.625,"15min":0.55200000000000005},"max":{"1min":1.611,"5min":1.889,"15min":1.889},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1850000000000001,"15min":0.98899999999999999},"min":{"1min":0.73599999999999999,"5min":0.67200000000000004,"15min":0.57799999999999996},"max":{"1min":1.476,"5min":1.581,"15min":1.581},"last":1.097}]},{"osd":6,"last update":"Mon Feb 12 16:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1279999999999999,"15min":1.042},"min":{"1min":0.82999999999999996,"5min":0.66900000000000004,"15min":0.65600000000000003},"max":{"1min":1.401,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.242,"15min":1.1359999999999999},"min":{"1min":0.85199999999999998,"5min":0.74199999999999999,"15min":0.58799999999999997},"max":{"1min":1.5740000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.2729999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.175,"15min":1.1339999999999999},"min":{"1min":0.879,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.4970000000000001,"5min":1.718,"15min":1.718},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.403,"15min":1.343},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.7909999999999999,"5min":2.0110000000000001,"15min":2.302},"last":1.399}]},{"osd":8,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5249999999999999,"15min":1.5129999999999999},"min":{"1min":1.073,"5min":1.073,"15min":0.88400000000000001},"max":{"1min":1.8100000000000001,"5min":2.093,"15min":2.1949999999999998},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4590000000000001,"15min":1.423},"min":{"1min":1.133,"5min":1.038,"15min":0.93700000000000006},"max":{"1min":1.931,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.508}]},{"osd":9,"last update":"Mon Feb 12 16:03:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.615,"15min":1.5649999999999999},"min":{"1min":1.401,"5min":1.1659999999999999,"15min":1.1659999999999999},"max":{"1min":1.9910000000000001,"5min":2.173,"15min":2.173},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.55,"15min":1.482},"min":{"1min":1.2190000000000001,"5min":1.117,"15min":1.109},"max":{"1min":1.9299999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.6479999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.5529999999999999,"15min":1.5189999999999999},"min":{"1min":1.2789999999999999,"5min":1.097,"15min":1.097},"max":{"1min":2.0609999999999999,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5389999999999999,"15min":1.5609999999999999},"min":{"1min":1.202,"5min":1.1819999999999999,"15min":1.1819999999999999},"max":{"1min":2.0259999999999998,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":1.609}]}]},{"osd":5,"up_from":39,"seq":167503724655,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":305152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0609999999999999,"15min":0.90800000000000003},"min":{"1min":0.65100000000000002,"5min":0.63100000000000001,"15min":0.57699999999999996},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0349999999999999,"15min":0.93100000000000005},"min":{"1min":0.79500000000000004,"5min":0.66300000000000003,"15min":0.46999999999999997},"max":{"1min":1.4279999999999999,"5min":1.524,"15min":1.524},"last":0.878}]},{"osd":1,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1080000000000001,"15min":0.96399999999999997},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.59999999999999998},"max":{"1min":1.3600000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.141},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.028,"15min":0.92400000000000004},"min":{"1min":0.75800000000000001,"5min":0.74099999999999999,"15min":0.59099999999999997},"max":{"1min":1.3660000000000001,"5min":1.381,"15min":1.381},"last":1.389}]},{"osd":2,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1160000000000001,"15min":0.99399999999999999},"min":{"1min":0.75800000000000001,"5min":0.622,"15min":0.57999999999999996},"max":{"1min":1.5309999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.099,"15min":1.0329999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.58899999999999997},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.91300000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1100000000000001,"15min":1.0149999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70299999999999996},"max":{"1min":1.544,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.111},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.202,"15min":1.0920000000000001},"min":{"1min":0.82599999999999996,"5min":0.72699999999999998,"15min":0.66400000000000003},"max":{"1min":1.5589999999999999,"5min":1.726,"15min":1.726},"last":1.0900000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.999,"15min":0.92000000000000004},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.56499999999999995},"max":{"1min":1.4510000000000001,"5min":1.889,"15min":1.889},"last":1.222},{"interface":"front","average":{"1min":1.208,"5min":1.181,"15min":1.02},"min":{"1min":0.86599999999999999,"5min":0.68200000000000005,"15min":0.52700000000000002},"max":{"1min":1.712,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.2430000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.206,"15min":1.1000000000000001},"min":{"1min":0.83099999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.655,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.246,"15min":1.143},"min":{"1min":0.76500000000000001,"5min":0.68300000000000005,"15min":0.40400000000000003},"max":{"1min":1.603,"5min":1.8360000000000001,"15min":1.903},"last":1.3440000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2829999999999999,"15min":1.1699999999999999},"min":{"1min":0.90300000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.6739999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.429,"15min":1.2470000000000001},"min":{"1min":1.1359999999999999,"5min":0.69099999999999995,"15min":0.435},"max":{"1min":1.772,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.5649999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.532,"15min":1.427},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.9850000000000001,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.429},{"interface":"front","average":{"1min":1.603,"5min":1.466,"15min":1.331},"min":{"1min":1.347,"5min":1.0860000000000001,"15min":0.83999999999999997},"max":{"1min":1.96,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.714}]},{"osd":9,"last update":"Mon Feb 12 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.5700000000000001,"15min":1.496},"min":{"1min":1.216,"5min":1.177,"15min":1.0469999999999999},"max":{"1min":1.845,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.647,"15min":1.5920000000000001},"min":{"1min":1.2849999999999999,"5min":1.1850000000000001,"15min":1.1619999999999999},"max":{"1min":1.923,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.643}]},{"osd":10,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4790000000000001,"15min":1.49},"min":{"1min":1.149,"5min":1.145,"15min":1.145},"max":{"1min":1.8340000000000001,"5min":1.835,"15min":1.835},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.617,"15min":1.631},"min":{"1min":1.258,"5min":1.222,"15min":1.222},"max":{"1min":1.9410000000000001,"5min":2.254,"15min":2.254},"last":1.5089999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:03:18.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:03:18.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:18 smithi121 bash[18489]: cluster 2024-02-12T16:03:16.893073+0000 osd.9 (osd.9) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:18.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:18 smithi121 bash[18489]: cluster 2024-02-12T16:03:16.893168+0000 osd.9 (osd.9) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:18.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:18 smithi121 bash[18489]: cluster 2024-02-12T16:03:17.478225+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e140: 12 total, 12 up, 12 in 2024-02-12T16:03:18.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:18 smithi121 bash[18489]: cluster 2024-02-12T16:03:17.522981+0000 mgr.y (mgr.14146) 529 : cluster [DBG] pgmap v580: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:18.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:18 smithi121 bash[18489]: audit 2024-02-12T16:03:17.784428+0000 mgr.y (mgr.14146) 530 : audit [DBG] from='client.14754 v1:172.21.15.121:0/4060168286' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:18 smithi169 bash[19604]: cluster 2024-02-12T16:03:16.893073+0000 osd.9 (osd.9) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:18 smithi169 bash[19604]: cluster 2024-02-12T16:03:16.893168+0000 osd.9 (osd.9) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:18 smithi169 bash[19604]: cluster 2024-02-12T16:03:17.478225+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e140: 12 total, 12 up, 12 in 2024-02-12T16:03:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:18 smithi169 bash[19604]: cluster 2024-02-12T16:03:17.522981+0000 mgr.y (mgr.14146) 529 : cluster [DBG] pgmap v580: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:18 smithi169 bash[19604]: audit 2024-02-12T16:03:17.784428+0000 mgr.y (mgr.14146) 530 : audit [DBG] from='client.14754 v1:172.21.15.121:0/4060168286' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:18 smithi123 bash[19119]: cluster 2024-02-12T16:03:16.893073+0000 osd.9 (osd.9) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:18 smithi123 bash[19119]: cluster 2024-02-12T16:03:16.893168+0000 osd.9 (osd.9) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:18 smithi123 bash[19119]: cluster 2024-02-12T16:03:17.478225+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e140: 12 total, 12 up, 12 in 2024-02-12T16:03:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:18 smithi123 bash[19119]: cluster 2024-02-12T16:03:17.522981+0000 mgr.y (mgr.14146) 529 : cluster [DBG] pgmap v580: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:18 smithi123 bash[19119]: audit 2024-02-12T16:03:17.784428+0000 mgr.y (mgr.14146) 530 : audit [DBG] from='client.14754 v1:172.21.15.121:0/4060168286' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:20.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:20 smithi123 bash[19119]: cluster 2024-02-12T16:03:19.523622+0000 mgr.y (mgr.14146) 531 : cluster [DBG] pgmap v581: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:20.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:20 smithi121 bash[18489]: cluster 2024-02-12T16:03:19.523622+0000 mgr.y (mgr.14146) 531 : cluster [DBG] pgmap v581: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:20 smithi169 bash[19604]: cluster 2024-02-12T16:03:19.523622+0000 mgr.y (mgr.14146) 531 : cluster [DBG] pgmap v581: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:22.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:22 smithi123 bash[19119]: cluster 2024-02-12T16:03:21.524265+0000 mgr.y (mgr.14146) 532 : cluster [DBG] pgmap v582: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:22.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:22 smithi121 bash[18489]: cluster 2024-02-12T16:03:21.524265+0000 mgr.y (mgr.14146) 532 : cluster [DBG] pgmap v582: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:22 smithi169 bash[19604]: cluster 2024-02-12T16:03:21.524265+0000 mgr.y (mgr.14146) 532 : cluster [DBG] pgmap v582: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:24.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:24 smithi123 bash[19119]: cluster 2024-02-12T16:03:23.524893+0000 mgr.y (mgr.14146) 533 : cluster [DBG] pgmap v583: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:24.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:24 smithi121 bash[18489]: cluster 2024-02-12T16:03:23.524893+0000 mgr.y (mgr.14146) 533 : cluster [DBG] pgmap v583: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:24 smithi169 bash[19604]: cluster 2024-02-12T16:03:23.524893+0000 mgr.y (mgr.14146) 533 : cluster [DBG] pgmap v583: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:26.226 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:03:26.229 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:03:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:26 smithi169 bash[19604]: cluster 2024-02-12T16:03:25.525307+0000 mgr.y (mgr.14146) 534 : cluster [DBG] pgmap v584: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:26.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:26 smithi123 bash[19119]: cluster 2024-02-12T16:03:25.525307+0000 mgr.y (mgr.14146) 534 : cluster [DBG] pgmap v584: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:26.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:26 smithi121 bash[18489]: cluster 2024-02-12T16:03:25.525307+0000 mgr.y (mgr.14146) 534 : cluster [DBG] pgmap v584: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:26.837 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2024-02-12T16:03:25.525115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":3573168,"kb_used_data":8016,"kb_used_omap":0,"kb_used_meta":3565056,"kb_avail":1121368656,"statfs":{"total":1151940427776,"available":1148281503744,"internally_reserved":0,"allocated":8208384,"data_stored":3731816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3650617344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":144,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-12T16:03:14.070817+0000","last_change":"2024-02-12T16:02:49.550827+0000","last_active":"2024-02-12T16:03:14.070817+0000","last_peered":"2024-02-12T16:03:14.070817+0000","last_clean":"2024-02-12T16:03:14.070817+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:03:14.070817+0000","last_undegraded":"2024-02-12T16:03:14.070817+0000","last_fullsized":"2024-02-12T16:03:14.070817+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:02:49.550626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:00:34.507052+0000","last_clean_scrub_stamp":"2024-02-12T16:02:49.550626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0409999999999999,"15min":1.0389999999999999},"min":{"1min":0.71899999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.482,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.873},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.089,"15min":1.123},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.4379999999999999,"5min":2.871,"15min":2.871},"last":1.204}]},{"osd":1,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.048,"15min":1.1020000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1990000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.0780000000000001,"15min":1.0880000000000001},"min":{"1min":0.73199999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.5920000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.3720000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.119,"15min":1.157},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.466,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":1.468},{"interface":"front","average":{"1min":1.117,"5min":1.1519999999999999,"15min":1.24},"min":{"1min":0.77200000000000002,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.3210000000000002,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.125}]},{"osd":3,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1739999999999999,"15min":1.2190000000000001},"min":{"1min":0.92600000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6839999999999999,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.032,"5min":1.151,"15min":1.2070000000000001},"min":{"1min":0.75800000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.393,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":1.1659999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2749999999999999,"15min":1.3720000000000001},"min":{"1min":0.81200000000000006,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4770000000000001,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":1.099},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.355,"15min":1.4390000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.8400000000000001,"5min":4.0789999999999997,"15min":4.0789999999999997},"last":1.284}]},{"osd":5,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.421,"15min":1.4750000000000001},"min":{"1min":1.016,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.754,"5min":3.044,"15min":3.044},"last":1.528},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.423,"15min":1.5069999999999999},"min":{"1min":0.93500000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.6279999999999999,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":1.3300000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4379999999999999,"15min":1.5349999999999999},"min":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":1.827,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.5349999999999999,"15min":1.619},"min":{"1min":1.038,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.9279999999999999,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":1.5469999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.534,"15min":1.6259999999999999},"min":{"1min":0.90000000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.8779999999999999,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":1.655},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6259999999999999,"15min":1.722},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.921,"5min":3.8370000000000002,"15min":3.8370000000000002},"last":1.5820000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.569,"15min":1.6519999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.974,"5min":4,"15min":4},"last":1.7310000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.647,"15min":1.742},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.9490000000000001,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.633}]},{"osd":10,"last update":"Mon Feb 12 16:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.5700000000000001,"15min":1.694},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.772,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.742,"15min":1.8360000000000001},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0230000000000001,"5min":4.3949999999999996,"15min":4.3949999999999996},"last":1.794}]}]},{"osd":10,"up_from":83,"seq":356482285639,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.90000000000000002,"15min":0.89600000000000002},"min":{"1min":0.69699999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1919999999999999,"5min":1.413,"15min":1.413},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.026,"15min":1.0660000000000001},"min":{"1min":0.80200000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.335,"5min":1.573,"15min":1.573},"last":1.1499999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0009999999999999,"15min":0.98299999999999998},"min":{"1min":0.66400000000000003,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.4490000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1,"15min":0.96599999999999997},"min":{"1min":0.72599999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.395,"5min":1.492,"15min":1.492},"last":0.80200000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.04,"15min":1.016},"min":{"1min":0.73499999999999999,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.998,"15min":1.0149999999999999},"min":{"1min":0.80600000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.294,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.026}]},{"osd":4,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.2230000000000001,"15min":1.1279999999999999},"min":{"1min":0.83299999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.228,"15min":1.2290000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.5640000000000001,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.056}]},{"osd":5,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.171,"15min":1.1830000000000001},"min":{"1min":0.872,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.421,"5min":2.048,"15min":2.048},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.3200000000000001,"15min":1.306},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7110000000000001,"5min":1.788,"15min":1.788},"last":1.4330000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2050000000000001,"15min":1.224},"min":{"1min":0.82899999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.407,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.212},{"interface":"front","average":{"1min":1.387,"5min":1.4239999999999999,"15min":1.3600000000000001},"min":{"1min":1.0309999999999999,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.8089999999999999,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.2490000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3759999999999999,"15min":1.3},"min":{"1min":1.0720000000000001,"5min":0.755,"15min":0.755},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.169},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.3740000000000001,"15min":1.3799999999999999},"min":{"1min":1.056,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.5409999999999999,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.29}]},{"osd":8,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.423,"15min":1.359},"min":{"1min":0.84499999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":2,"5min":2,"15min":2},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.48,"15min":1.3620000000000001},"min":{"1min":1.0049999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.4830000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.5389999999999999,"15min":1.4570000000000001},"min":{"1min":1.1439999999999999,"5min":0.752,"15min":0.752},"max":{"1min":1.982,"5min":2.02,"15min":2.02},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.5349999999999999,"15min":1.429},"min":{"1min":1.1319999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.327}]},{"osd":11,"last update":"Mon Feb 12 16:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.595,"15min":1.6100000000000001},"min":{"1min":1.2989999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.6180000000000001,"15min":1.6379999999999999},"min":{"1min":1.2989999999999999,"5min":1.25,"15min":1.25},"max":{"1min":1.919,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.522}]}]},{"osd":9,"up_from":74,"seq":317827579984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.94299999999999995,"15min":0.93300000000000005},"min":{"1min":0.68000000000000005,"5min":0.623,"15min":0.623},"max":{"1min":1.1539999999999999,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.93300000000000005,"15min":0.96899999999999997},"min":{"1min":0.57899999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3,"5min":1.4419999999999999,"15min":2.2480000000000002},"last":0.99299999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.052,"15min":1},"min":{"1min":0.65600000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.399,"5min":1.48,"15min":1.48},"last":1.212},{"interface":"front","average":{"1min":0.995,"5min":1.0529999999999999,"15min":1.0349999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3879999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.125}]},{"osd":2,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96799999999999997,"15min":0.93700000000000006},"min":{"1min":0.71799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.252,"5min":1.738,"15min":1.738},"last":1.3140000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.96499999999999997,"15min":1.022},"min":{"1min":0.71799999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.194,"5min":1.369,"15min":1.601},"last":0.77200000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.97499999999999998,"15min":0.96799999999999997},"min":{"1min":0.68300000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.198,"5min":1.4470000000000001,"15min":2.1549999999999998},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.121,"15min":1.111},"min":{"1min":0.72999999999999998,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.82499999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.177,"15min":1.1719999999999999},"min":{"1min":0.91400000000000003,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.2629999999999999,"5min":1.9330000000000001,"15min":2.113},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1759999999999999,"15min":1.1859999999999999},"min":{"1min":0.88900000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.355,"5min":1.619,"15min":2.3260000000000001},"last":1.46}]},{"osd":5,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2290000000000001,"15min":1.2250000000000001},"min":{"1min":1.0209999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.4970000000000001,"5min":1.756,"15min":2.4399999999999999},"last":1.274},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.329,"15min":1.2749999999999999},"min":{"1min":1.234,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.633,"5min":1.851,"15min":2.2599999999999998},"last":1.4870000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3220000000000001,"15min":1.3240000000000001},"min":{"1min":0.995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.9139999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.351},"min":{"1min":0.996,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8660000000000001,"15min":2.48},"last":1.3560000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.284,"15min":1.3160000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.401,"5min":2.012,"15min":2.3519999999999999},"last":1.018},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4350000000000001,"15min":1.411},"min":{"1min":0.98099999999999998,"5min":0.90200000000000002,"15min":0.875},"max":{"1min":1.772,"5min":1.772,"15min":2.0249999999999999},"last":1.5880000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.393,"15min":1.367},"min":{"1min":0.86399999999999999,"5min":0.64700000000000002,"15min":0.61099999999999999},"max":{"1min":1.704,"5min":1.792,"15min":2.1299999999999999},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.377,"15min":1.355},"min":{"1min":1.069,"5min":0.62,"15min":0.58499999999999996},"max":{"1min":1.4379999999999999,"5min":1.9570000000000001,"15min":2.3889999999999998},"last":1.538}]},{"osd":10,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.4330000000000001,"15min":1.4299999999999999},"min":{"1min":1.0780000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.597,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":1.623},{"interface":"front","average":{"1min":1.409,"5min":1.3979999999999999,"15min":1.3560000000000001},"min":{"1min":1.05,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.8400000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.3360000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874327,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95999999999999996,"15min":0.85599999999999998},"min":{"1min":0.61699999999999999,"5min":0.57899999999999996,"15min":0.57699999999999996},"max":{"1min":1.1839999999999999,"5min":1.488,"15min":1.488},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.96499999999999997,"15min":0.89500000000000002},"min":{"1min":0.71699999999999997,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.589,"5min":1.623,"15min":1.623},"last":1.169}]},{"osd":1,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.88800000000000001,"15min":0.84999999999999998},"min":{"1min":0.64900000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.3480000000000001,"5min":1.508,"15min":1.508},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0249999999999999,"15min":0.95599999999999996},"min":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.355,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.101}]},{"osd":2,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.01,"15min":0.88},"min":{"1min":0.68700000000000006,"5min":0.59499999999999997,"15min":0.59199999999999997},"max":{"1min":1.194,"5min":1.764,"15min":1.764},"last":1.278},{"interface":"front","average":{"1min":1.173,"5min":1.141,"15min":1.0309999999999999},"min":{"1min":0.91400000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.637,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.3560000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.083,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.64800000000000002,"15min":0.60499999999999998},"max":{"1min":1.365,"5min":1.698,"15min":1.698},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.22,"15min":1.04},"min":{"1min":0.96099999999999997,"5min":0.623,"15min":0.623},"max":{"1min":1.3280000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":1.0209999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2629999999999999,"15min":1.157},"min":{"1min":0.94599999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.4339999999999999,"5min":1.677,"15min":1.839},"last":0.995},{"interface":"front","average":{"1min":1.3,"5min":1.3080000000000001,"15min":1.1160000000000001},"min":{"1min":0.80800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.772,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.534}]},{"osd":5,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1990000000000001,"15min":1.155},"min":{"1min":0.76900000000000002,"5min":0.76700000000000002,"15min":0.75800000000000001},"max":{"1min":1.4379999999999999,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.341,"5min":1.3420000000000001,"15min":1.2030000000000001},"min":{"1min":1.157,"5min":0.71899999999999997,"15min":0.68899999999999995},"max":{"1min":1.508,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":1.325}]},{"osd":6,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.333,"15min":1.1970000000000001},"min":{"1min":0.92800000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.915,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.3939999999999999,"15min":1.25},"min":{"1min":1.073,"5min":0.91500000000000004,"15min":0.876},"max":{"1min":1.5549999999999999,"5min":2.339,"15min":2.339},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4530000000000001,"15min":1.2629999999999999},"min":{"1min":1.0860000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6120000000000001,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4930000000000001,"15min":1.292},"min":{"1min":1.194,"5min":0.997,"15min":0.85499999999999998},"max":{"1min":1.7909999999999999,"5min":1.984,"15min":1.984},"last":1.4490000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:03:18 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5229999999999999,"15min":1.268},"min":{"1min":0.98899999999999999,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.8149999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.589},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.3440000000000001,"15min":1.22},"min":{"1min":0.90700000000000003,"5min":0.72999999999999998,"15min":0.67200000000000004},"max":{"1min":1.579,"5min":2.456,"15min":2.456},"last":0.95499999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4079999999999999,"15min":1.3640000000000001},"min":{"1min":1.002,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.843,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.147},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.502,"15min":1.462},"min":{"1min":0.99399999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.8069999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.5029999999999999}]}]},{"osd":7,"up_from":56,"seq":240518168672,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.036,"15min":0.98999999999999999},"min":{"1min":0.63900000000000001,"5min":0.60899999999999999,"15min":0.436},"max":{"1min":1.671,"5min":1.675,"15min":1.675},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.74199999999999999,"5min":0.64100000000000001,"15min":0.53400000000000003},"max":{"1min":1.171,"5min":1.353,"15min":1.353},"last":1.1220000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.98399999999999999,"15min":1.022},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3680000000000001,"5min":1.5980000000000001,"15min":1.8580000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.052,"15min":1.044},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.56499999999999995},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.78200000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.089,"15min":1.083},"min":{"1min":0.628,"5min":0.628,"15min":0.53400000000000003},"max":{"1min":1.954,"5min":1.954,"15min":2.0049999999999999},"last":1.159},{"interface":"front","average":{"1min":1.014,"5min":1.046,"15min":1.093},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.355,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.202}]},{"osd":4,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95399999999999996,"15min":1.0069999999999999},"min":{"1min":0.69599999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4019999999999999,"5min":1.51,"15min":2.0449999999999999},"last":0.874},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.042},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4790000000000001,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.83499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.095,"15min":1.077},"min":{"1min":0.82799999999999996,"5min":0.626,"15min":0.60099999999999998},"max":{"1min":1.4590000000000001,"5min":1.76,"15min":1.76},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1419999999999999,"15min":1.181},"min":{"1min":0.65800000000000003,"5min":0.62,"15min":0.46100000000000002},"max":{"1min":1.4259999999999999,"5min":1.677,"15min":2.1709999999999998},"last":1.3029999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1719999999999999,"15min":1.2030000000000001},"min":{"1min":0.78700000000000003,"5min":0.73099999999999998,"15min":0.64300000000000002},"max":{"1min":1.744,"5min":1.744,"15min":1.8069999999999999},"last":1.236},{"interface":"front","average":{"1min":1.204,"5min":1.1870000000000001,"15min":1.2230000000000001},"min":{"1min":0.77900000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.486,"5min":2.0960000000000001,"15min":2.1139999999999999},"last":1.288}]},{"osd":8,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.3999999999999999,"15min":1.425},"min":{"1min":1.1910000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.256},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.4099999999999999,"15min":1.446},"min":{"1min":1.028,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8180000000000001,"5min":1.905,"15min":2.0760000000000001},"last":1.5049999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.446,"15min":1.4379999999999999},"min":{"1min":1.0629999999999999,"5min":0.81699999999999995,"15min":0.751},"max":{"1min":1.8180000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4259999999999999,"15min":1.421},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.4430000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.504,"15min":1.5349999999999999},"min":{"1min":0.96499999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":1.333},{"interface":"front","average":{"1min":1.542,"5min":1.502,"15min":1.5589999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9450000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.54}]},{"osd":11,"last update":"Mon Feb 12 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.55,"15min":1.5820000000000001},"min":{"1min":1.169,"5min":1.071,"15min":1.071},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.522,"15min":1.5209999999999999},"min":{"1min":1.087,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":1.7849999999999999,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.607}]}]},{"osd":6,"up_from":48,"seq":206158430313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.99199999999999999,"15min":0.95399999999999996},"min":{"1min":0.73899999999999999,"5min":0.59599999999999997,"15min":0.498},"max":{"1min":1.335,"5min":1.492,"15min":1.492},"last":1.23},{"interface":"front","average":{"1min":1.032,"5min":1.01,"15min":0.93400000000000005},"min":{"1min":0.76000000000000001,"5min":0.66500000000000004,"15min":0.64200000000000002},"max":{"1min":1.421,"5min":1.573,"15min":1.573},"last":0.98499999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:03:19 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.002,"15min":0.91100000000000003},"min":{"1min":0.72299999999999998,"5min":0.68899999999999995,"15min":0.64300000000000002},"max":{"1min":1.343,"5min":1.484,"15min":1.484},"last":1.032},{"interface":"front","average":{"1min":1.002,"5min":1.0309999999999999,"15min":1},"min":{"1min":0.70599999999999996,"5min":0.62,"15min":0.60999999999999999},"max":{"1min":1.3360000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.0589999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:03:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.054,"15min":0.97499999999999998},"min":{"1min":0.81299999999999994,"5min":0.66000000000000003,"15min":0.55500000000000005},"max":{"1min":1.1990000000000001,"5min":2.383,"15min":2.383},"last":1.113},{"interface":"front","average":{"1min":1.115,"5min":1.1279999999999999,"15min":1.0149999999999999},"min":{"1min":0.83499999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.5649999999999999,"5min":1.621,"15min":1.621},"last":1.1859999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:03:19 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.056,"15min":0.96499999999999997},"min":{"1min":0.77900000000000003,"5min":0.75700000000000001,"15min":0.68100000000000005},"max":{"1min":1.2809999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0800000000000001,"15min":1.0469999999999999},"min":{"1min":0.77000000000000002,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.623},"last":1.4410000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:03:19 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.0620000000000001,"15min":0.92800000000000005},"min":{"1min":0.66300000000000003,"5min":0.57199999999999995,"15min":0.41899999999999998},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.755},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.117,"15min":0.97899999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.42299999999999999},"max":{"1min":1.476,"5min":1.591,"15min":1.6299999999999999},"last":0.90500000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:03:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.2070000000000001,"15min":1.0269999999999999},"min":{"1min":0.71199999999999997,"5min":0.60299999999999998,"15min":0.46999999999999997},"max":{"1min":1.635,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":1.085},{"interface":"front","average":{"1min":1.208,"5min":1.218,"15min":1.052},"min":{"1min":0.749,"5min":0.69499999999999995,"15min":0.51800000000000002},"max":{"1min":1.597,"5min":1.7709999999999999,"15min":1.7949999999999999},"last":0.81999999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.321,"15min":1.1890000000000001},"min":{"1min":0.875,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.8839999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.3},{"interface":"front","average":{"1min":1.319,"5min":1.337,"15min":1.2729999999999999},"min":{"1min":1.147,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.631,"5min":1.889,"15min":1.889},"last":1.284}]},{"osd":8,"last update":"Mon Feb 12 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.502,"15min":1.397},"min":{"1min":1.0960000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.47},{"interface":"front","average":{"1min":1.544,"5min":1.534,"15min":1.4259999999999999},"min":{"1min":1.083,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.0739999999999998,"5min":2.093,"15min":2.093},"last":1.351}]},{"osd":9,"last update":"Mon Feb 12 16:03:09 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.6100000000000001,"15min":1.4890000000000001},"min":{"1min":1.3149999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5309999999999999,"15min":1.421},"min":{"1min":1.2829999999999999,"5min":1.046,"15min":0.95899999999999996},"max":{"1min":2.0150000000000001,"5min":2.125,"15min":2.2469999999999999},"last":1.1499999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":1.6559999999999999,"15min":1.627},"min":{"1min":1.3759999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"max":{"1min":2.1200000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.591,"15min":1.569},"min":{"1min":1.3320000000000001,"5min":1.081,"15min":1.081},"max":{"1min":2.0070000000000001,"5min":2.367,"15min":2.367},"last":1.258}]}]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.85499999999999998,"15min":0.68899999999999995},"min":{"1min":0.66200000000000003,"5min":0.41099999999999998,"15min":0.377},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.753},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.94799999999999995,"15min":0.75700000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.28899999999999998},"max":{"1min":1.179,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.998}]},{"osd":2,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.94499999999999995,"15min":0.75900000000000001},"min":{"1min":0.63800000000000001,"5min":0.50700000000000001,"15min":0.379},"max":{"1min":1.252,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96799999999999997,"15min":0.79100000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.34999999999999998},"max":{"1min":1.2669999999999999,"5min":1.54,"15min":1.54},"last":0.86299999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.98599999999999999,"15min":0.85499999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2649999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.0609999999999999,"15min":0.92300000000000004},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.9570000000000001,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.80100000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.181,"15min":1.048},"min":{"1min":0.92100000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.746,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.264,"15min":1.1259999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.8720000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.1619999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3160000000000001,"15min":1.1439999999999999},"min":{"1min":1.034,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.4390000000000001,"15min":1.234},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":1.45}]},{"osd":6,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3400000000000001,"15min":1.2330000000000001},"min":{"1min":0.91700000000000004,"5min":0.81399999999999995,"15min":0.80500000000000005},"max":{"1min":1.921,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.462,"15min":1.3380000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.573,"5min":2.573,"15min":2.573},"last":1.4279999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.4950000000000001,"15min":1.3939999999999999},"min":{"1min":1.0600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.974,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.3700000000000001,"15min":1.343},"min":{"1min":0.89600000000000002,"5min":0.871,"15min":0.871},"max":{"1min":1.722,"5min":1.887,"15min":1.887},"last":1.3009999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.49,"15min":1.403},"min":{"1min":1.0069999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8049999999999999,"5min":2.355,"15min":2.355},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6479999999999999,"15min":1.506},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.1349999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.5880000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.744,"5min":1.6759999999999999,"15min":1.6419999999999999},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":2.1669999999999998,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.52},{"interface":"front","average":{"1min":1.508,"5min":1.5109999999999999,"15min":1.5309999999999999},"min":{"1min":1.075,"5min":1.075,"15min":1.075},"max":{"1min":1.772,"5min":2.133,"15min":2.133},"last":1.5069999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:58 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.5469999999999999,"15min":1.571},"min":{"1min":1.212,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.738,"5min":1.7130000000000001,"15min":1.7529999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.4220000000000002,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":1.651}]}]},{"osd":0,"up_from":8,"seq":34359738522,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.81599999999999995,"15min":0.67400000000000004},"min":{"1min":0.621,"5min":0.44900000000000001,"15min":0.32500000000000001},"max":{"1min":1.0449999999999999,"5min":1.2170000000000001,"15min":1.3009999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.84099999999999997,"15min":0.70899999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.242,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.91800000000000004}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.88500000000000001,"15min":0.77900000000000003},"min":{"1min":0.70199999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.3799999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.89000000000000001,"15min":0.87},"min":{"1min":0.64200000000000002,"5min":0.53500000000000003,"15min":0.48099999999999998},"max":{"1min":1.2050000000000001,"5min":1.3440000000000001,"15min":1.3839999999999999},"last":1.1459999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1599999999999999,"15min":1.081},"min":{"1min":0.78300000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0489999999999999,"15min":1.0049999999999999},"min":{"1min":0.71299999999999997,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":1.0349999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0960000000000001,"15min":1.069},"min":{"1min":0.94199999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.3680000000000001,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.2170000000000001,"15min":1.202},"min":{"1min":1.014,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.4239999999999999,"5min":1.772,"15min":1.772},"last":1.097}]},{"osd":6,"last update":"Mon Feb 12 16:03:24 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.1850000000000001,"15min":1.137},"min":{"1min":0.871,"5min":0.755,"15min":0.73799999999999999},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.871},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2549999999999999,"15min":1.2150000000000001},"min":{"1min":0.89900000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5629999999999999,"5min":1.7310000000000001,"15min":1.768},"last":1.2829999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.353,"15min":1.3560000000000001},"min":{"1min":0.97699999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.7649999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.232},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.486,"15min":1.3360000000000001},"min":{"1min":1.262,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0059999999999998,"5min":2.1379999999999999,"15min":2.214},"last":1.3340000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.4910000000000001,"15min":1.403},"min":{"1min":1.3120000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.988,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.46,"15min":1.4870000000000001},"min":{"1min":0.95499999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.6140000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.3080000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:03:18 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.542,"15min":1.4870000000000001},"min":{"1min":1.2909999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.0800000000000001,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":1.427},{"interface":"front","average":{"1min":1.446,"5min":1.4710000000000001,"15min":1.569},"min":{"1min":1.0720000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":1.9330000000000001,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":1.5149999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.51,"15min":1.6100000000000001},"min":{"1min":1.0229999999999999,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.8899999999999999,"5min":2.335,"15min":2.335},"last":1.605},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3859999999999999,"15min":1.4159999999999999},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.173}]},{"osd":11,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.5960000000000001,"15min":1.546},"min":{"1min":1.367,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.407,"15min":1.383},"min":{"1min":0.88500000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.4470000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313352,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.872,"15min":0.71399999999999997},"min":{"1min":0.63300000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88800000000000001,"15min":0.73499999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.47099999999999997},"max":{"1min":1.278,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.0820000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.98499999999999999,"15min":0.80000000000000004},"min":{"1min":0.59599999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3959999999999999,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.093,"15min":0.89200000000000002},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.42799999999999999},"max":{"1min":1.5209999999999999,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.2070000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.046,"15min":0.89000000000000001},"min":{"1min":0.79100000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.294,"5min":1.4710000000000001,"15min":1.5089999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.004,"5min":1.0149999999999999,"15min":0.89400000000000002},"min":{"1min":0.72999999999999998,"5min":0.61099999999999999,"15min":0.59199999999999997},"max":{"1min":1.3140000000000001,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.81100000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:02:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1950000000000001,"15min":1.0640000000000001},"min":{"1min":0.92300000000000004,"5min":0.86599999999999999,"15min":0.70599999999999996},"max":{"1min":1.387,"5min":1.74,"15min":1.7669999999999999},"last":1.294},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.96799999999999997,"15min":0.90600000000000003},"min":{"1min":0.71799999999999997,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.294,"5min":1.306,"15min":1.323},"last":0.98799999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2310000000000001,"15min":1.155},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.528,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.147,"15min":1.0600000000000001},"min":{"1min":0.85199999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.571,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":1.444}]},{"osd":6,"last update":"Mon Feb 12 16:03:07 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.149,"15min":1.089},"min":{"1min":0.94599999999999995,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.613,"5min":1.746,"15min":1.746},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.304,"15min":1.2090000000000001},"min":{"1min":0.84199999999999997,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7110000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.401}]},{"osd":7,"last update":"Mon Feb 12 16:02:59 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.383,"15min":1.341},"min":{"1min":0.85999999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.7490000000000001,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.296,"15min":1.2649999999999999},"min":{"1min":0.91900000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.581,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.2350000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5109999999999999,"15min":1.427},"min":{"1min":1.1240000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9099999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4890000000000001,"15min":1.4810000000000001},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.6080000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:03:07 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.387,"15min":1.427},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.498,"15min":1.4830000000000001},"min":{"1min":1.024,"5min":1.024,"15min":1.024},"max":{"1min":2.117,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":1.496}]},{"osd":10,"last update":"Mon Feb 12 16:02:44 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.464,"15min":1.47},"min":{"1min":1.046,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.8500000000000001,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.478},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5329999999999999,"15min":1.5489999999999999},"min":{"1min":1.1739999999999999,"5min":1.046,"15min":1.046},"max":{"1min":2.206,"5min":2.206,"15min":2.206},"last":1.538}]}]},{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.84099999999999997,"15min":0.81899999999999995},"min":{"1min":0.56599999999999995,"5min":0.434,"15min":0.42999999999999999},"max":{"1min":1.1559999999999999,"5min":1.2589999999999999,"15min":1.3260000000000001},"last":0.876},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.94599999999999995,"15min":0.82999999999999996},"min":{"1min":0.80900000000000005,"5min":0.56599999999999995,"15min":0.371},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.91100000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92800000000000005,"15min":0.85599999999999998},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.51900000000000002},"max":{"1min":1.278,"5min":1.466,"15min":1.5529999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.109,"5min":1.077,"15min":0.92000000000000004},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.44800000000000001},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.117}]},{"osd":4,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.751,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.2749999999999999,"5min":1.278,"15min":1.593},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.145,"15min":1.0489999999999999},"min":{"1min":0.81299999999999994,"5min":0.75900000000000001,"15min":0.60699999999999998},"max":{"1min":1.4710000000000001,"5min":1.669,"15min":1.669},"last":1.1479999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2370000000000001,"15min":1.153},"min":{"1min":0.96599999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6140000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.262},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.089,"15min":1.095},"min":{"1min":0.76900000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.1830000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.0289999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.194,"15min":1.1519999999999999},"min":{"1min":0.84599999999999997,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.7629999999999999,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.2989999999999999,"15min":1.258},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.7030000000000001,"5min":1.7689999999999999,"15min":1.8120000000000001},"last":1.363}]},{"osd":7,"last update":"Mon Feb 12 16:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.3129999999999999,"15min":1.3149999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.66600000000000004},"max":{"1min":1.7450000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3819999999999999,"15min":1.343},"min":{"1min":1.0309999999999999,"5min":0.85299999999999998,"15min":0.77800000000000002},"max":{"1min":1.768,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.4970000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.3160000000000001,"15min":1.3740000000000001},"min":{"1min":0.93600000000000005,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.669,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.49,"15min":1.4850000000000001},"min":{"1min":0.98999999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.8080000000000001,"5min":2.242,"15min":2.242},"last":1.6539999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.548,"15min":1.554},"min":{"1min":1.026,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.901,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.548},{"interface":"front","average":{"1min":1.548,"5min":1.5009999999999999,"15min":1.5529999999999999},"min":{"1min":1.1439999999999999,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.117,"5min":2.117,"15min":2.117},"last":1.679}]},{"osd":10,"last update":"Mon Feb 12 16:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4450000000000001,"15min":1.4790000000000001},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":1.7869999999999999,"5min":2.113,"15min":2.113},"last":1.421},{"interface":"front","average":{"1min":1.546,"5min":1.5629999999999999,"15min":1.581},"min":{"1min":1.1279999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.819,"5min":1.974,"15min":1.974},"last":1.732}]},{"osd":11,"last update":"Mon Feb 12 16:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.4910000000000001,"15min":1.4950000000000001},"min":{"1min":1.2330000000000001,"5min":1.073,"15min":1.073},"max":{"1min":1.867,"5min":1.925,"15min":1.925},"last":1.526},{"interface":"front","average":{"1min":1.597,"5min":1.5780000000000001,"15min":1.536},"min":{"1min":1.0469999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.895,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.796}]}]},{"osd":4,"up_from":32,"seq":137438953591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.0720000000000001,"15min":0.94899999999999995},"min":{"1min":0.65400000000000003,"5min":0.64400000000000002,"15min":0.56999999999999995},"max":{"1min":1.476,"5min":1.736,"15min":1.736},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.0600000000000001,"15min":0.93200000000000005},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.58699999999999997},"max":{"1min":1.3140000000000001,"5min":1.617,"15min":1.617},"last":1.0980000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0609999999999999,"15min":0.96199999999999997},"min":{"1min":0.78600000000000003,"5min":0.72099999999999997,"15min":0.63900000000000001},"max":{"1min":1.4019999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.379},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.115,"15min":0.96599999999999997},"min":{"1min":0.81599999999999995,"5min":0.68799999999999994,"15min":0.51400000000000001},"max":{"1min":1.423,"5min":1.806,"15min":1.806},"last":1.4099999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.052,"15min":0.93300000000000005},"min":{"1min":0.68999999999999995,"5min":0.68899999999999995,"15min":0.60299999999999998},"max":{"1min":1.3540000000000001,"5min":1.593,"15min":1.593},"last":1.232},{"interface":"front","average":{"1min":0.997,"5min":1.0209999999999999,"15min":0.95799999999999996},"min":{"1min":0.71499999999999997,"5min":0.68000000000000005,"15min":0.64900000000000002},"max":{"1min":1.429,"5min":1.4450000000000001,"15min":1.5549999999999999},"last":1.0640000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.121,"15min":0.98799999999999999},"min":{"1min":0.79600000000000004,"5min":0.72599999999999998,"15min":0.57599999999999996},"max":{"1min":1.47,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.109,"15min":0.97399999999999998},"min":{"1min":0.91400000000000003,"5min":0.73299999999999998,"15min":0.55100000000000005},"max":{"1min":1.4470000000000001,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.2549999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1060000000000001,"15min":0.92400000000000004},"min":{"1min":0.69599999999999995,"5min":0.625,"15min":0.55200000000000005},"max":{"1min":1.611,"5min":1.889,"15min":1.889},"last":1.214},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1850000000000001,"15min":0.98899999999999999},"min":{"1min":0.73599999999999999,"5min":0.67200000000000004,"15min":0.57799999999999996},"max":{"1min":1.476,"5min":1.581,"15min":1.581},"last":1.4330000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:03:15 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.153,"15min":1.054},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":1.486},{"interface":"front","average":{"1min":1.119,"5min":1.208,"15min":1.139},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.58799999999999997},"max":{"1min":1.5,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.3440000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.175,"15min":1.1339999999999999},"min":{"1min":0.879,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.4970000000000001,"5min":1.718,"15min":1.718},"last":1.399},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.403,"15min":1.343},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.7909999999999999,"5min":2.0110000000000001,"15min":2.302},"last":1.5840000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5249999999999999,"15min":1.5129999999999999},"min":{"1min":1.073,"5min":1.073,"15min":0.88400000000000001},"max":{"1min":1.8100000000000001,"5min":2.093,"15min":2.1949999999999998},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4590000000000001,"15min":1.423},"min":{"1min":1.133,"5min":1.038,"15min":0.93700000000000006},"max":{"1min":1.931,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.5189999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:03:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.615,"15min":1.5649999999999999},"min":{"1min":1.401,"5min":1.1659999999999999,"15min":1.1659999999999999},"max":{"1min":1.9910000000000001,"5min":2.173,"15min":2.173},"last":1.804},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.55,"15min":1.482},"min":{"1min":1.2190000000000001,"5min":1.117,"15min":1.109},"max":{"1min":1.9299999999999999,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.74}]},{"osd":10,"last update":"Mon Feb 12 16:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.5529999999999999,"15min":1.5189999999999999},"min":{"1min":1.2789999999999999,"5min":1.097,"15min":1.097},"max":{"1min":2.0609999999999999,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.665},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5389999999999999,"15min":1.5609999999999999},"min":{"1min":1.202,"5min":1.1819999999999999,"15min":1.1819999999999999},"max":{"1min":2.0259999999999998,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":1.552}]}]},{"osd":5,"up_from":39,"seq":167503724656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":310988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0609999999999999,"15min":0.90800000000000003},"min":{"1min":0.65100000000000002,"5min":0.63100000000000001,"15min":0.57699999999999996},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0349999999999999,"15min":0.93100000000000005},"min":{"1min":0.79500000000000004,"5min":0.66300000000000003,"15min":0.46999999999999997},"max":{"1min":1.4279999999999999,"5min":1.524,"15min":1.524},"last":1.3160000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.1080000000000001,"15min":0.96399999999999997},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.59999999999999998},"max":{"1min":1.3600000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.028,"15min":0.92400000000000004},"min":{"1min":0.75800000000000001,"5min":0.74099999999999999,"15min":0.59099999999999997},"max":{"1min":1.3660000000000001,"5min":1.381,"15min":1.381},"last":1.242}]},{"osd":2,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1160000000000001,"15min":0.99399999999999999},"min":{"1min":0.75800000000000001,"5min":0.622,"15min":0.57999999999999996},"max":{"1min":1.5309999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.099,"15min":1.0329999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.58899999999999997},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.0940000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1100000000000001,"15min":1.0149999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.70299999999999996},"max":{"1min":1.544,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.274},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.202,"15min":1.0920000000000001},"min":{"1min":0.82599999999999996,"5min":0.72699999999999998,"15min":0.66400000000000003},"max":{"1min":1.5589999999999999,"5min":1.726,"15min":1.726},"last":1.3480000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.999,"15min":0.92000000000000004},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.56499999999999995},"max":{"1min":1.4510000000000001,"5min":1.889,"15min":1.889},"last":1.169},{"interface":"front","average":{"1min":1.208,"5min":1.181,"15min":1.02},"min":{"1min":0.86599999999999999,"5min":0.68200000000000005,"15min":0.52700000000000002},"max":{"1min":1.712,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.95799999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.206,"15min":1.1000000000000001},"min":{"1min":0.83099999999999996,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.655,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.246,"15min":1.143},"min":{"1min":0.76500000000000001,"5min":0.68300000000000005,"15min":0.40400000000000003},"max":{"1min":1.603,"5min":1.8360000000000001,"15min":1.903},"last":1.4590000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2829999999999999,"15min":1.1699999999999999},"min":{"1min":0.90300000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.6739999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.381},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.429,"15min":1.2470000000000001},"min":{"1min":1.1359999999999999,"5min":0.69099999999999995,"15min":0.435},"max":{"1min":1.772,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.4390000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:02:38 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.532,"15min":1.427},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.9850000000000001,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.603,"5min":1.466,"15min":1.331},"min":{"1min":1.347,"5min":1.0860000000000001,"15min":0.83999999999999997},"max":{"1min":1.96,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.7030000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.5700000000000001,"15min":1.496},"min":{"1min":1.216,"5min":1.177,"15min":1.0469999999999999},"max":{"1min":1.845,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.639,"5min":1.647,"15min":1.5920000000000001},"min":{"1min":1.2849999999999999,"5min":1.1850000000000001,"15min":1.1619999999999999},"max":{"1min":1.923,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.5700000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4790000000000001,"15min":1.49},"min":{"1min":1.149,"5min":1.145,"15min":1.145},"max":{"1min":1.8340000000000001,"5min":1.835,"15min":1.835},"last":1.514},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.617,"15min":1.631},"min":{"1min":1.258,"5min":1.222,"15min":1.222},"max":{"1min":1.9410000000000001,"5min":2.254,"15min":2.254},"last":1.7190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:03:26.839 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-12T16:03:26.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump 2024-02-12T16:03:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:27 smithi169 bash[19604]: audit 2024-02-12T16:03:26.208447+0000 mgr.y (mgr.14146) 535 : audit [DBG] from='client.14760 v1:172.21.15.121:0/2449769494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:27.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:27 smithi123 bash[19119]: audit 2024-02-12T16:03:26.208447+0000 mgr.y (mgr.14146) 535 : audit [DBG] from='client.14760 v1:172.21.15.121:0/2449769494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:27.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:27 smithi121 bash[18489]: audit 2024-02-12T16:03:26.208447+0000 mgr.y (mgr.14146) 535 : audit [DBG] from='client.14760 v1:172.21.15.121:0/2449769494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:03:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:28 smithi169 bash[19604]: cluster 2024-02-12T16:03:26.880414+0000 osd.11 (osd.11) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:28 smithi169 bash[19604]: cluster 2024-02-12T16:03:26.880500+0000 osd.11 (osd.11) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:28 smithi169 bash[19604]: cluster 2024-02-12T16:03:27.208124+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e141: 12 total, 12 up, 12 in 2024-02-12T16:03:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:28 smithi169 bash[19604]: cluster 2024-02-12T16:03:27.525943+0000 mgr.y (mgr.14146) 536 : cluster [DBG] pgmap v586: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:28.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:28 smithi123 bash[19119]: cluster 2024-02-12T16:03:26.880414+0000 osd.11 (osd.11) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:28.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:28 smithi123 bash[19119]: cluster 2024-02-12T16:03:26.880500+0000 osd.11 (osd.11) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:28.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:28 smithi123 bash[19119]: cluster 2024-02-12T16:03:27.208124+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e141: 12 total, 12 up, 12 in 2024-02-12T16:03:28.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:28 smithi123 bash[19119]: cluster 2024-02-12T16:03:27.525943+0000 mgr.y (mgr.14146) 536 : cluster [DBG] pgmap v586: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:28.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:28 smithi121 bash[18489]: cluster 2024-02-12T16:03:26.880414+0000 osd.11 (osd.11) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:28.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:28 smithi121 bash[18489]: cluster 2024-02-12T16:03:26.880500+0000 osd.11 (osd.11) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:28.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:28 smithi121 bash[18489]: cluster 2024-02-12T16:03:27.208124+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e141: 12 total, 12 up, 12 in 2024-02-12T16:03:28.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:28 smithi121 bash[18489]: cluster 2024-02-12T16:03:27.525943+0000 mgr.y (mgr.14146) 536 : cluster [DBG] pgmap v586: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:30 smithi169 bash[19604]: cluster 2024-02-12T16:03:29.465695+0000 osd.8 (osd.8) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:30 smithi169 bash[19604]: cluster 2024-02-12T16:03:29.465782+0000 osd.8 (osd.8) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:30 smithi169 bash[19604]: cluster 2024-02-12T16:03:29.518938+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e142: 12 total, 12 up, 12 in 2024-02-12T16:03:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:30 smithi169 bash[19604]: cluster 2024-02-12T16:03:29.526494+0000 mgr.y (mgr.14146) 537 : cluster [DBG] pgmap v588: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:30 smithi123 bash[19119]: cluster 2024-02-12T16:03:29.465695+0000 osd.8 (osd.8) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:30 smithi123 bash[19119]: cluster 2024-02-12T16:03:29.465782+0000 osd.8 (osd.8) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:30 smithi123 bash[19119]: cluster 2024-02-12T16:03:29.518938+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e142: 12 total, 12 up, 12 in 2024-02-12T16:03:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:30 smithi123 bash[19119]: cluster 2024-02-12T16:03:29.526494+0000 mgr.y (mgr.14146) 537 : cluster [DBG] pgmap v588: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:30.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:30 smithi121 bash[18489]: cluster 2024-02-12T16:03:29.465695+0000 osd.8 (osd.8) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:30.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:30 smithi121 bash[18489]: cluster 2024-02-12T16:03:29.465782+0000 osd.8 (osd.8) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:30.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:30 smithi121 bash[18489]: cluster 2024-02-12T16:03:29.518938+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e142: 12 total, 12 up, 12 in 2024-02-12T16:03:30.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:30 smithi121 bash[18489]: cluster 2024-02-12T16:03:29.526494+0000 mgr.y (mgr.14146) 537 : cluster [DBG] pgmap v588: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:31.642 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:32.756 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:32 smithi121 bash[18489]: cluster 2024-02-12T16:03:31.527246+0000 mgr.y (mgr.14146) 538 : cluster [DBG] pgmap v589: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:32.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:32 smithi123 bash[19119]: cluster 2024-02-12T16:03:31.527246+0000 mgr.y (mgr.14146) 538 : cluster [DBG] pgmap v589: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:32 smithi169 bash[19604]: cluster 2024-02-12T16:03:31.527246+0000 mgr.y (mgr.14146) 538 : cluster [DBG] pgmap v589: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:34.434 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 143 2024-02-12T16:03:34.434 INFO:teuthology.orchestra.run.smithi121.stdout:fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 2024-02-12T16:03:34.434 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-02-12T15:47:18.273813+0000 2024-02-12T16:03:34.434 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-02-12T16:03:33.583186+0000 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:crush_version 27 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:full_ratio 0.95 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:backfillfull_ratio 0.9 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:nearfull_ratio 0.85 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:require_min_compat_client luminous 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:min_compat_client jewel 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:require_osd_release pacific 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:stretch_mode_enabled false 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:pool 1 'device_health_metrics' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:max_osd 12 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 up in weight 1 up_from 8 up_thru 56 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6801/1815232143 v1:172.21.15.121:6802/1815232143 exists,up 3a3badac-97ea-4ee1-ad82-1b67a4fe94f0 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6805/2968433670 v1:172.21.15.121:6806/2968433670 exists,up c6140309-5703-4538-9739-276fbbdaa880 2024-02-12T16:03:34.435 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6809/2158187852 v1:172.21.15.121:6810/2158187852 exists,up 7c81e1a4-cae0-48d4-98ab-0becb6b31170 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6813/1415189366 v1:172.21.15.121:6814/1415189366 exists,up 5f824526-596d-42e1-8b7e-cd8676ded6dd 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6800/3447700123 v1:172.21.15.123:6801/3447700123 exists,up 14c2f878-99ce-4b51-8d9c-131c1d883b31 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6804/961317306 v1:172.21.15.123:6805/961317306 exists,up 75ecf26a-d7ed-46e7-a40e-6f719e2c3602 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6 up in weight 1 up_from 48 up_thru 49 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6808/1377261784 v1:172.21.15.123:6809/1377261784 exists,up 71e715e6-eac1-4386-927e-51df0cc419c0 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7 up in weight 1 up_from 56 up_thru 94 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6812/1702613957 v1:172.21.15.123:6813/1702613957 exists,up b419b04f-0c0c-495a-aa9b-25c334159515 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6800/2551075225 v1:172.21.15.169:6801/2551075225 exists,up 1591e7bc-5b86-4401-803d-5325062ffb75 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6804/2339555071 v1:172.21.15.169:6805/2339555071 exists,up 522d61df-8dc1-4915-83fd-f94d0d572c15 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6808/391306610 v1:172.21.15.169:6809/391306610 exists,up 91f74dca-503a-45d1-9647-df4804d58515 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11 up in weight 1 up_from 93 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6812/3696812305 v1:172.21.15.169:6813/3696812305 exists,up 788b4f47-eadf-4601-a5b4-51fb6f04fbdf 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/2474849789 expires 2024-02-13T15:48:17.261083+0000 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/253689008 expires 2024-02-13T15:48:17.261083+0000 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:6800/3531151725 expires 2024-02-13T15:47:47.232761+0000 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/1778611055 expires 2024-02-13T15:47:47.232761+0000 2024-02-12T16:03:34.436 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:6800/28718860 expires 2024-02-13T15:48:17.261083+0000 2024-02-12T16:03:34.437 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/445290372 expires 2024-02-13T15:47:47.232761+0000 2024-02-12T16:03:34.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:34 smithi123 bash[19119]: cluster 2024-02-12T16:03:33.527923+0000 mgr.y (mgr.14146) 539 : cluster [DBG] pgmap v590: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:34.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:34 smithi123 bash[19119]: cluster 2024-02-12T16:03:33.568058+0000 osd.4 (osd.4) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:34.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:34 smithi123 bash[19119]: cluster 2024-02-12T16:03:33.568148+0000 osd.4 (osd.4) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:34.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:34 smithi123 bash[19119]: cluster 2024-02-12T16:03:33.590306+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e143: 12 total, 12 up, 12 in 2024-02-12T16:03:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:34 smithi123 bash[19119]: audit 2024-02-12T16:03:34.426000+0000 mon.a (mon.0) 829 : audit [DBG] from='client.? v1:172.21.15.121:0/3182002762' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-12T16:03:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:34 smithi121 bash[18489]: cluster 2024-02-12T16:03:33.527923+0000 mgr.y (mgr.14146) 539 : cluster [DBG] pgmap v590: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:34 smithi121 bash[18489]: cluster 2024-02-12T16:03:33.568058+0000 osd.4 (osd.4) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:34 smithi121 bash[18489]: cluster 2024-02-12T16:03:33.568148+0000 osd.4 (osd.4) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:34 smithi121 bash[18489]: cluster 2024-02-12T16:03:33.590306+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e143: 12 total, 12 up, 12 in 2024-02-12T16:03:34.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:34 smithi121 bash[18489]: audit 2024-02-12T16:03:34.426000+0000 mon.a (mon.0) 829 : audit [DBG] from='client.? v1:172.21.15.121:0/3182002762' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-12T16:03:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:34 smithi169 bash[19604]: cluster 2024-02-12T16:03:33.527923+0000 mgr.y (mgr.14146) 539 : cluster [DBG] pgmap v590: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:34 smithi169 bash[19604]: cluster 2024-02-12T16:03:33.568058+0000 osd.4 (osd.4) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:34 smithi169 bash[19604]: cluster 2024-02-12T16:03:33.568148+0000 osd.4 (osd.4) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:34 smithi169 bash[19604]: cluster 2024-02-12T16:03:33.590306+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e143: 12 total, 12 up, 12 in 2024-02-12T16:03:35.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:34 smithi169 bash[19604]: audit 2024-02-12T16:03:34.426000+0000 mon.a (mon.0) 829 : audit [DBG] from='client.? v1:172.21.15.121:0/3182002762' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-12T16:03:35.081 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 56 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6801/1815232143 v1:172.21.15.121:6802/1815232143 exists,up 3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6805/2968433670 v1:172.21.15.121:6806/2968433670 exists,up c6140309-5703-4538-9739-276fbbdaa880', 'osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6809/2158187852 v1:172.21.15.121:6810/2158187852 exists,up 7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6813/1415189366 v1:172.21.15.121:6814/1415189366 exists,up 5f824526-596d-42e1-8b7e-cd8676ded6dd', 'osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6800/3447700123 v1:172.21.15.123:6801/3447700123 exists,up 14c2f878-99ce-4b51-8d9c-131c1d883b31', 'osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6804/961317306 v1:172.21.15.123:6805/961317306 exists,up 75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'osd.6 up in weight 1 up_from 48 up_thru 49 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6808/1377261784 v1:172.21.15.123:6809/1377261784 exists,up 71e715e6-eac1-4386-927e-51df0cc419c0', 'osd.7 up in weight 1 up_from 56 up_thru 94 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6812/1702613957 v1:172.21.15.123:6813/1702613957 exists,up b419b04f-0c0c-495a-aa9b-25c334159515', 'osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6800/2551075225 v1:172.21.15.169:6801/2551075225 exists,up 1591e7bc-5b86-4401-803d-5325062ffb75', 'osd.9 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6804/2339555071 v1:172.21.15.169:6805/2339555071 exists,up 522d61df-8dc1-4915-83fd-f94d0d572c15', 'osd.10 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6808/391306610 v1:172.21.15.169:6809/391306610 exists,up 91f74dca-503a-45d1-9647-df4804d58515', 'osd.11 up in weight 1 up_from 93 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.169:6812/3696812305 v1:172.21.15.169:6813/3696812305 exists,up 788b4f47-eadf-4601-a5b4-51fb6f04fbdf'] 2024-02-12T16:03:35.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon mon.a config show 2024-02-12T16:03:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:35 smithi123 bash[19119]: cluster 2024-02-12T16:03:33.655403+0000 osd.5 (osd.5) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:35 smithi123 bash[19119]: cluster 2024-02-12T16:03:33.655487+0000 osd.5 (osd.5) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:35 smithi123 bash[19119]: cluster 2024-02-12T16:03:34.599707+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e144: 12 total, 12 up, 12 in 2024-02-12T16:03:35.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:35 smithi121 bash[18489]: cluster 2024-02-12T16:03:33.655403+0000 osd.5 (osd.5) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:35.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:35 smithi121 bash[18489]: cluster 2024-02-12T16:03:33.655487+0000 osd.5 (osd.5) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:35.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:35 smithi121 bash[18489]: cluster 2024-02-12T16:03:34.599707+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e144: 12 total, 12 up, 12 in 2024-02-12T16:03:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:35 smithi169 bash[19604]: cluster 2024-02-12T16:03:33.655403+0000 osd.5 (osd.5) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:35 smithi169 bash[19604]: cluster 2024-02-12T16:03:33.655487+0000 osd.5 (osd.5) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:35 smithi169 bash[19604]: cluster 2024-02-12T16:03:34.599707+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e144: 12 total, 12 up, 12 in 2024-02-12T16:03:36.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:36 smithi123 bash[19119]: cluster 2024-02-12T16:03:35.528563+0000 mgr.y (mgr.14146) 540 : cluster [DBG] pgmap v593: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:36.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:36 smithi121 bash[18489]: cluster 2024-02-12T16:03:35.528563+0000 mgr.y (mgr.14146) 540 : cluster [DBG] pgmap v593: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:36 smithi169 bash[19604]: cluster 2024-02-12T16:03:35.528563+0000 mgr.y (mgr.14146) 540 : cluster [DBG] pgmap v593: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:37.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:37 smithi123 bash[19119]: cluster 2024-02-12T16:03:36.819830+0000 osd.0 (osd.0) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:37.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:37 smithi123 bash[19119]: cluster 2024-02-12T16:03:36.819938+0000 osd.0 (osd.0) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:37.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:37 smithi121 bash[18489]: cluster 2024-02-12T16:03:36.819830+0000 osd.0 (osd.0) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:37.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:37 smithi121 bash[18489]: cluster 2024-02-12T16:03:36.819938+0000 osd.0 (osd.0) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:37 smithi169 bash[19604]: cluster 2024-02-12T16:03:36.819830+0000 osd.0 (osd.0) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:37 smithi169 bash[19604]: cluster 2024-02-12T16:03:36.819938+0000 osd.0 (osd.0) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:38.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:38 smithi123 bash[19119]: cluster 2024-02-12T16:03:37.529209+0000 mgr.y (mgr.14146) 541 : cluster [DBG] pgmap v594: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:38.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:38 smithi123 bash[19119]: cluster 2024-02-12T16:03:37.607393+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e145: 12 total, 12 up, 12 in 2024-02-12T16:03:38.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:38 smithi121 bash[18489]: cluster 2024-02-12T16:03:37.529209+0000 mgr.y (mgr.14146) 541 : cluster [DBG] pgmap v594: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:38.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:38 smithi121 bash[18489]: cluster 2024-02-12T16:03:37.607393+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e145: 12 total, 12 up, 12 in 2024-02-12T16:03:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:38 smithi169 bash[19604]: cluster 2024-02-12T16:03:37.529209+0000 mgr.y (mgr.14146) 541 : cluster [DBG] pgmap v594: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:38 smithi169 bash[19604]: cluster 2024-02-12T16:03:37.607393+0000 mon.a (mon.0) 831 : cluster [DBG] osdmap e145: 12 total, 12 up, 12 in 2024-02-12T16:03:39.880 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:40.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:40 smithi123 bash[19119]: cluster 2024-02-12T16:03:39.529842+0000 mgr.y (mgr.14146) 542 : cluster [DBG] pgmap v596: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:40.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:40 smithi121 bash[18489]: cluster 2024-02-12T16:03:39.529842+0000 mgr.y (mgr.14146) 542 : cluster [DBG] pgmap v596: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:40 smithi169 bash[19604]: cluster 2024-02-12T16:03:39.529842+0000 mgr.y (mgr.14146) 542 : cluster [DBG] pgmap v596: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:41.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:41 smithi123 bash[19119]: cluster 2024-02-12T16:03:40.362521+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:41 smithi123 bash[19119]: cluster 2024-02-12T16:03:40.362611+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:41 smithi123 bash[19119]: cluster 2024-02-12T16:03:41.206455+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e146: 12 total, 12 up, 12 in 2024-02-12T16:03:41.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:41 smithi121 bash[18489]: cluster 2024-02-12T16:03:40.362521+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:41.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:41 smithi121 bash[18489]: cluster 2024-02-12T16:03:40.362611+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:41.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:41 smithi121 bash[18489]: cluster 2024-02-12T16:03:41.206455+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e146: 12 total, 12 up, 12 in 2024-02-12T16:03:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:41 smithi169 bash[19604]: cluster 2024-02-12T16:03:40.362521+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:03:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:41 smithi169 bash[19604]: cluster 2024-02-12T16:03:40.362611+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:03:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:41 smithi169 bash[19604]: cluster 2024-02-12T16:03:41.206455+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e146: 12 total, 12 up, 12 in 2024-02-12T16:03:42.731 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mon.a", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster": "ceph", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "admin_socket_mode": "", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_client_required": "cephx, none", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_cluster_required": "cephx", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_debug": "true", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_service_required": "cephx", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-12T16:03:42.732 INFO:teuthology.orchestra.run.smithi121.stdout: "auth_supported": "", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_aio": "true", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_aio_poll_ms": "250", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_aio_reap_max": "16", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_async_discard": "false", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_block_size": "4096", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_debug_aio": "false", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_debug_inflight_ios": "false", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_enable_discard": "false", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_flock_retry": "3", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-12T16:03:42.733 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_inject_crash": "0", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_ioring": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_ioring_hipri": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bdev_type": "", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_alloc_size": "1048576", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_allocator": "hybrid", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_buffered_io": "true", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_check_for_zeros": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_compact_log_sync": "false", 2024-02-12T16:03:42.734 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_max_log_runway": "4194304", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_max_prefetch": "1048576", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_min_flush_size": "524288", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_min_log_runway": "1048576", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_replay_recovery": "false", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_sync_write": "false", 2024-02-12T16:03:42.735 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_allocator": "hybrid", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_create": "true", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_db_create": "false", 2024-02-12T16:03:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_db_path": "", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_db_size": "0", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_path": "", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_preallocate_file": "false", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_size": "107374182400", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_wal_create": "false", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_wal_path": "", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_block_wal_size": "100663296", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_bluefs": "true", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_autotune": "true", 2024-02-12T16:03:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_size": "0", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_type": "2q", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_clone_cow": "true", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-12T16:03:42.738 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_mode": "none", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_csum_type": "crc32c", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_freelist": "false", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_inject_bug21040": "false", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-12T16:03:42.739 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_small_allocations": "0", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_default_buffered_read": "true", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_default_buffered_write": "false", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-12T16:03:42.740 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_on_mount": "false", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-12T16:03:42.741 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_on_umount": "false", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_ignore_data_csum": "false", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-12T16:03:42.742 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_log_op_age": "5.000000", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_max_alloc_size": "0", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_max_blob_size": "0", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_max_deferred_txc": "32", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_min_alloc_size": "0", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_nid_prealloc": "1024", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-12T16:03:42.743 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_retry_disk_reads": "3", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_rocksdb_cf": "true", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_spdk_mem": "512", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-12T16:03:42.744 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_tracing": "false", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-12T16:03:42.745 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephx_cluster_require_signatures": "false", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephx_cluster_require_version": "2", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephx_require_signatures": "false", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephx_require_version": "2", 2024-02-12T16:03:42.746 INFO:teuthology.orchestra.run.smithi121.stdout: "cephx_service_require_signatures": "false", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "cephx_service_require_version": "2", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "cephx_sign_messages": "true", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "chdir": "", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_acl_type": "", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_asio_thread_count": "2", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_cache_mid": "0.750000", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_cache_size": "16384", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_caps_release_delay": "5", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_check_pool_perm": "true", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_debug_force_sync_read": "false", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_debug_getattr_caps": "false", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_debug_inject_tick_delay": "0", 2024-02-12T16:03:42.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_die_on_failed_remount": "false", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_dirsize_rbytes": "true", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_force_lazyio": "false", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_fs": "", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_inject_release_failure": "false", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_max_inline_size": "4096", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_mds_namespace": "", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_metadata": "", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_mount_gid": "-1", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_mount_timeout": "300", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_mount_uid": "-1", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_mountpoint": "/", 2024-02-12T16:03:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_notify_timeout": "10", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_oc": "true", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_oc_max_dirty": "104857600", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_oc_max_objects": "1000", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_oc_size": "209715200", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_oc_target_dirty": "8388608", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_permissions": "true", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_quota": "true", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_quota_df": "true", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_readahead_max_bytes": "0", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_readahead_max_periods": "4", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_readahead_min": "131072", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_reconnect_stale": "false", 2024-02-12T16:03:42.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_shutdown_timeout": "30", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_snapdir": ".snap", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_tick_interval": "1", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_trace": "", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_try_dentry_invalidate": "false", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_use_faked_inos": "false", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_use_random_mds": "false", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "clog_to_graylog": "false", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "clog_to_graylog_port": "12201", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "clog_to_monitors": "default=true", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "clog_to_syslog": "false", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "clog_to_syslog_level": "info", 2024-02-12T16:03:42.750 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "-", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_network": "", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_network_interface": "", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "compressor_zlib_isal": "false", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "compressor_zlib_level": "5", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "compressor_zlib_winsize": "-15", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "compressor_zstd_level": "1", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a9c1a3337ae6b9328bb9087151b372dee3bd15abdc1220ab67654538650c50dc", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crimson_osd_obc_lru_size": "10", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location_hook": "", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location_hook_timeout": "10", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "daemonize": "false", 2024-02-12T16:03:42.751 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_allow_any_pool_priority": "false", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_asok": "1/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_asok_assert_abort": "false", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_asserts_on_shutdown": "true", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_auth": "1/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_bdev": "1/3", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_bluefs": "1/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_bluestore": "1/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_buffer": "0/1", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_cephfs_mirror": "0/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_cephsqlite": "0/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_civetweb": "1/10", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_client": "0/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_compressor": "1/5", 2024-02-12T16:03:42.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_context": "0/1", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_crush": "1/1", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_crypto": "1/5", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_deliberately_leak_memory": "false", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_disable_randomized_ping": "false", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_dpdk": "1/5", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_eventtrace": "1/5", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_filer": "0/1", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_filestore": "1/3", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_finisher": "1/1", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_fuse": "1/5", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_heartbeat_testing_span": "0", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_heartbeatmap": "1/5", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_javaclient": "1/5", 2024-02-12T16:03:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_journal": "1/3", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_journaler": "0/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_kstore": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_leveldb": "4/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_lockdep": "0/1", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mds": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mds_balancer": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mds_locker": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mds_log": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mds_log_expire": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mds_migrator": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_memdb": "4/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mgr": "2/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mgrc": "1/5", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_mon": "20/20", 2024-02-12T16:03:42.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_monc": "0/10", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_ms": "1/1", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_none": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_objclass": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_objectcacher": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_objecter": "0/1", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_optracker": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_osd": "1/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_paxos": "20/20", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_perfcounter": "1/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_prioritycache": "1/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rados": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rbd": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rbd_mirror": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rbd_pwl": "0/5", 2024-02-12T16:03:42.755 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rbd_replay": "0/5", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_refs": "0/0", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_reserver": "1/1", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rgw": "1/5", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rgw_sync": "1/5", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_rocksdb": "4/5", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_striper": "0/1", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_test": "0/5", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_throttle": "1/1", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_timer": "0/1", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "debug_tp": "0/5", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "device_failure_prediction_mode": "none", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-12T16:03:42.756 INFO:teuthology.orchestra.run.smithi121.stdout: "err_to_graylog": "false", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "err_to_stderr": "true", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "err_to_syslog": "false", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event_tracing": "false", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "fake_statfs_for_testing": "0", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "fatal_signal_handlers": "true", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filer_max_purge_ops": "10", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filer_max_truncate_ops": "128", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_apply_finisher_threads": "1", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_blackhole": "false", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_btrfs_clone_range": "true", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_btrfs_snap": "true", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_caller_concurrency": "10", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_collect_device_partition_information": "true", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_commit_timeout": "600.000000", 2024-02-12T16:03:42.757 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_debug_inject_read_err": "false", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_debug_omap_check": "false", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_debug_verify_split": "false", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_dump_file": "", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_fadvise": "true", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_fail_eio": "true", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_fd_cache_shards": "16", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_fd_cache_size": "128", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_fiemap": "false", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_fiemap_threshold": "4096", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-12T16:03:42.758 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_inject_stall": "0", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_journal_parallel": "false", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_journal_trailing": "false", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_journal_writeahead": "false", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_kill_at": "0", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattrs": "0", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_xattr_value_size": "0", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-12T16:03:42.759 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_merge_threshold": "-10", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_odsync_write": "false", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_omap_backend": "rocksdb", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_omap_backend_path": "", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_op_thread_timeout": "60", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_op_threads": "2", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_punch_hole": "false", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-12T16:03:42.760 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_queue_max_ops": "50", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_seek_data_hole": "false", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_sloppy_crc": "false", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_splice": "false", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_split_multiple": "2", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_split_rand_factor": "20", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_update_to": "1000", 2024-02-12T16:03:42.761 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_enable": "true", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_xfs_extsize": "false", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "filestore_zfs_snap": "false", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "fio_dir": "/tmp/fio", 2024-02-12T16:03:42.762 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_allow_other": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_atomic_o_trunc": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_big_writes": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_debug": "false", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_default_permissions": "false", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_disable_pagecache": "false", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_max_write": "0", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_multithreaded": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_require_active_mds": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_set_user_groups": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_splice_move": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_splice_read": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_splice_write": "true", 2024-02-12T16:03:42.763 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "fuse_use_invalidate_cb": "true", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "gss_target_name": "ceph", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_file": "", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_inject_failure": "0", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_interval": "5", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "host": "", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-12T16:03:42.764 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "inject_early_sigterm": "false", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_aio": "true", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_align_min_size": "65536", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_block_align": "true", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_block_size": "4096", 2024-02-12T16:03:42.765 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_dio": "true", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_discard": "false", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_force_aio": "false", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_ignore_corruption": "false", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_max_write_bytes": "10485760", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_max_write_entries": "100", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_replay_from": "0", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_write_header_frequency": "0", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journal_zero_on_create": "false", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journaler_prefetch_periods": "10", 2024-02-12T16:03:42.766 INFO:teuthology.orchestra.run.smithi121.stdout: "journaler_prezero_periods": "5", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "journaler_write_head_interval": "15", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "key": "", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "keyfile": "", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_backend": "rocksdb", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_default_stripe_size": "65536", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_fsck_on_mount": "false", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_max_bytes": "67108864", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_max_ops": "512", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_nid_prealloc": "1024", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_onode_map_size": "1024", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_sync_submit_transaction": "false", 2024-02-12T16:03:42.767 INFO:teuthology.orchestra.run.smithi121.stdout: "kstore_sync_transaction": "false", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_block_size": "65536", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_bloom_size": "0", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_cache_size": "536870912", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_compact_on_mount": "false", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_compression": "false", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_log": "", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_log_to_ceph_log": "true", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_max_open_files": "0", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_paranoid": "false", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "leveldb_write_buffer_size": "33554432", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "librados_thread_count": "2", 2024-02-12T16:03:42.768 INFO:teuthology.orchestra.run.smithi121.stdout: "lockdep": "false", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "lockdep_force_backtrace": "false", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_coarse_timestamps": "true", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_flush_on_exit": "false", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_graylog_host": "127.0.0.1", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_graylog_port": "12201", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_max_new": "1000", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_max_recent": "10000", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_stderr_prefix": "debug ", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_stop_at_utilization": "0.970000", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": "true", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_graylog": "false", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_stderr": "false", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_syslog": "false", 2024-02-12T16:03:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: "max_rotating_auth_attempts": "10", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_action_on_write_error": "1", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_alternate_name_max": "8192", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_asio_thread_count": "2", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_export_pin": "true", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_fragment_dirs": "true", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_fragment_interval": "5", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_interval": "10", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_max": "-1", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_max_until": "-1", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_merge_size": "50", 2024-02-12T16:03:42.770 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_midchunk": "0.300000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_min_start": "0.200000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_minchunk": "0.001000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_mode": "0", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_need_max": "1.200000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_need_min": "0.800000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_split_bits": "3", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_split_size": "10000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_target_decay": "10.000000", 2024-02-12T16:03:42.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_beacon_grace": "15.000000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_beacon_interval": "4.000000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache_mid": "0.700000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache_release_free_interval": "10", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache_reservation": "0.050000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache_trim_interval": "1", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache_trim_threshold": "262144", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_client_prealloc_inos": "1000", 2024-02-12T16:03:42.772 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_connect_bootstrapping": "false", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_damage_table_max_entries": "10000", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_debug_auth_pins": "false", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_debug_frag": "false", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_debug_scatterstat": "false", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_debug_subtrees": "false", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_decay_halflife": "5.000000", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_default_dir_hash": "2", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_defer_session_stale": "true", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_deny_all_reconnect": "false", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dir_keys_per_op": "16384", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dir_max_commit_size": "10", 2024-02-12T16:03:42.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dir_max_entries": "0", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dump_cache_on_map": "false", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_early_reply": "true", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_enable_op_tracker": "true", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_enforce_unique_name": "true", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_export_ephemeral_random": "true", 2024-02-12T16:03:42.774 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_health_summarize_threshold": "10", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_inject_health_dummy": "false", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_inject_migrator_session_race": "false", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_join_fs": "", 2024-02-12T16:03:42.775 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_journal_format": "1", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_create_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_export_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_import_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_journal_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_journal_expire_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_journal_replay_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_link_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_mdstable_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_openc_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_rename_at": "0", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log_events_per_segment": "1024", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log_max_events": "-1", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log_max_segments": "128", 2024-02-12T16:03:42.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log_pause": "false", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log_segment_size": "0", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log_warn_factor": "2.000000", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_caps_per_client": "1048576", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_completed_flushes": "100000", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_completed_requests": "100000", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_export_size": "20971520", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_file_recover": "32", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_purge_files": "64", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_purge_ops": "8192", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_snaps_per_dir": "100", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_max_xattr_pairs_size": "65536", 2024-02-12T16:03:42.777 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_metrics_update_interval": "2", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_min_caps_per_client": "100", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_min_caps_working_set": "10000", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_numa_node": "-1", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_op_complaint_time": "30.000000", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_op_history_duration": "600", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_op_history_size": "20", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_op_log_threshold": "5", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_ping_grace": "15", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_ping_interval": "5", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_recall_max_caps": "30000", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-12T16:03:42.778 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_recall_warning_threshold": "262144", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_replay_interval": "1.000000", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_root_ino_gid": "0", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_root_ino_uid": "0", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-12T16:03:42.779 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-12T16:03:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-12T16:03:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-12T16:03:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-12T16:03:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_shutdown_check": "0", 2024-02-12T16:03:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_skip_ino": "0", 2024-02-12T16:03:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-12T16:03:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_snap_max_uid": "4294967294", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_snap_min_uid": "0", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_snap_rstat": "false", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_standby_replay_damaged": "false", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_thrash_exports": "0", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_thrash_fragments": "0", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_tick_interval": "5.000000", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_valgrind_exit": "false", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_verify_backtrace": "1", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_verify_scatter": "false", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_wipe_sessions": "false", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool_debug": "false", 2024-02-12T16:03:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "memstore_device_bytes": "1073741824", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "memstore_page_set": "false", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "memstore_page_size": "65536", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_client_bytes": "134217728", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_client_messages": "512", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_disabled_modules": "", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_max_pg_num_change": "128", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_mds_bytes": "134217728", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_mds_messages": "128", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_mon_bytes": "134217728", 2024-02-12T16:03:42.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_mon_messages": "128", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_osd_bytes": "536870912", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_osd_messages": "8192", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_standby_modules": "true", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_stats_period": "5", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_stats_threshold": "5", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_tick_period": "2", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_allow_pool_delete": "true", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_allow_pool_size_one": "false", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_auth_validate_all_caps": "true", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-12T16:03:42.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_bytes": "104857600", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_directed_command_retry": "5", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_hunt_parallel": "3", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_log_interval": "1.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_ping_interval": "10.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-12T16:03:42.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_file_level": "debug", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_file": "true", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_stderr": "true", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_compact_on_bootstrap": "false", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_compact_on_start": "false", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_compact_on_trim": "true", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-12T16:03:42.785 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_cpu_threads": "4", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_crush_min_required_version": "hammer", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_daemon_bytes": "419430400", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_data_avail_crit": "5", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_data_avail_warn": "5", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_data_size_warn": "16106127360", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_dump_json": "false", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_dump_transactions": "false", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_extra_checks": "false", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-12T16:03:42.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_no_require_octopus": "false", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_no_require_pacific": "false", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_down_mkfs_grace": "60", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_election_default_strategy": "1", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_election_timeout": "5.000000", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_elector_ping_divisor": "2", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_enable_op_tracker": "true", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_fake_pool_delete": "false", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_force_quorum_join": "false", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_globalid_prealloc": "10000", 2024-02-12T16:03:42.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_health_detail_to_clog": "true", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_health_log_update_period": "5", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_health_max_detail": "50", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_health_to_clog": "true", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_health_to_clog_interval": "600", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_host": "172.21.15.121:6789/0", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_host_override": "", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_initial_members": "", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_lease": "5.000000", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-12T16:03:42.788 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_log_max_summary": "50", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_log_epochs": "500", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_osd": "10000", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_pg_per_osd": "10000", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_pool_pg_num": "65536", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mds_force_trim_to": "0", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mds_skip_sanity": "false", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_memory_autotune": "true", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_memory_target": "2147483648", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mgr_beacon_grace": "30", 2024-02-12T16:03:42.789 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mgr_digest_period": "5", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mgr_inactive_grace": "60", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_min_osdmap_epochs": "50", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_op_complaint_time": "30", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_op_history_duration": "600", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_op_history_size": "20", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_op_history_slow_op_size": "20", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_op_log_threshold": "5", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-12T16:03:42.790 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_auto_mark_in": "false", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_cache_size": "500", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_down_out_interval": "600", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_force_trim_to": "0", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-12T16:03:42.791 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_min_down_reporters": "2", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_report_timeout": "900", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-12T16:03:42.792 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pg_stuck_threshold": "60", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_probe_timeout": "2.000000", 2024-02-12T16:03:42.793 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_reweight_max_change": "0.050000", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_reweight_max_osds": "4", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_scrub_interval": "300", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_scrub_max_keys": "100", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_scrub_timeout": "300", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_session_timeout": "300", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_smart_report_timeout": "5", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_stat_smooth_intervals": "6", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-12T16:03:42.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_stretch_pool_min_size": "2", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_stretch_pool_size": "4", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_sync_debug": "false", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_sync_provider_kill_at": "0", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_sync_requester_kill_at": "0", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_sync_timeout": "60.000000", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_target_pg_per_osd": "100", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_tick_interval": "5", 2024-02-12T16:03:42.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_timecheck_interval": "300.000000", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_older_version_delay": "604800", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_misplaced": "false", 2024-02-12T16:03:42.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_older_version": "true", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": "", 2024-02-12T16:03:42.797 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_max_op_threads": "5", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_op_threads": "3", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_cm": "false", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_device_name": "", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_dscp": "96", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_local_gid": "", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_port_num": "1", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-12T16:03:42.798 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_sl": "3", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_support_srq": "true", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_rdma_type": "ib", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_async_reap_threshold": "5", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_before_connect": "false", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_ipv4": "true", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_ipv6": "false", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_msgr1": "true", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_msgr2": "false", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_port_max": "7300", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_port_min": "6800", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_retry_count": "3", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_bind_retry_delay": "5", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_blackhole_client": "false", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_blackhole_mds": "false", 2024-02-12T16:03:42.799 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_blackhole_mgr": "false", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_blackhole_mon": "false", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_blackhole_osd": "false", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_client_mode": "crc secure", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_cluster_mode": "crc secure", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_cluster_type": "", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_connection_idle_timeout": "900", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_connection_ready_timeout": "10", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_crc_data": "true", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_crc_header": "true", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_die_on_bad_msg": "false", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_die_on_bug": "true", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_die_on_old_message": "true", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_die_on_skipped_message": "false", 2024-02-12T16:03:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_coremask": "0xF", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_hugepages": "", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_lro": "true", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_memory_channel": "4", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_pmd": "", 2024-02-12T16:03:42.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_port_id": "0", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_dump_on_send": "false", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_initial_backoff": "0.200000", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_inject_delay_max": "1.000000", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_inject_delay_type": "", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_inject_socket_failures": "5000", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_learn_addr_from_peer": "true", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_max_accept_failures": "4", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_max_backoff": "15.000000", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_mon_client_mode": "secure crc", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_mon_service_mode": "secure crc", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-12T16:03:42.802 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_pq_min_cost": "65536", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_public_type": "", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_service_mode": "crc secure", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_tcp_listen_backlog": "512", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_tcp_nodelay": "true", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_tcp_rcvbuf": "0", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "ms_type": "async+posix", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "no_config_file": "false", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_completion_locks_per_session": "32", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_inflight_ops": "1024", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_tick_interval": "5.000000", 2024-02-12T16:03:42.803 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter_timeout": "10.000000", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "objectstore_blackhole": "false", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "openssl_engine_opts": "", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_agent_delay_time": "5.000000", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_agent_hist_halflife": "1000", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_agent_max_low_ops": "2", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_agent_max_ops": "4", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_agent_slop": "0.020000", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_asio_thread_count": "2", 2024-02-12T16:03:42.804 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_async_recovery_min_cost": "100", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_backfill_scan_max": "512", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_backfill_scan_min": "64", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_backoff_on_degraded": "false", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_backoff_on_peering": "false", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_backoff_on_unfound": "true", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_beacon_report_interval": "300", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bench_duration": "30", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bench_max_block_size": "67108864", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_blkin_trace_all": "false", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-12T16:03:42.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_check_for_log_corruption": "false", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_class_update_on_start": "true", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_client_message_cap": "256", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_client_message_size_cap": "524288000", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_client_op_priority": "63", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_client_watch_timeout": "30", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_command_max_records": "256", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_command_thread_suicide_timeout": "900", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_command_thread_timeout": "600", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_compact_on_start": "false", 2024-02-12T16:03:42.806 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_crush_update_on_start": "true", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_crush_update_weight_set": "true", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_feed_pullee": "-1", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-12T16:03:42.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_misdirected_ops": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_no_acting_change": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_no_purge_strays": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_op_order": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_shutdown": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-12T16:03:42.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_verify_snaps": "false", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_deep_scrub_keys": "1024", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_deep_scrub_stride": "524288", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_default_notify_timeout": "30", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_delete_sleep": "0.000000", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_discard_disconnected_ops": "true", 2024-02-12T16:03:42.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_enable_op_tracker": "true", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_fast_info": "true", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_fast_shutdown": "true", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_function_tracing": "false", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_heartbeat_grace": "20", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_heartbeat_interval": "6", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_heartbeat_min_peers": "10", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_heartbeat_min_size": "2000", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_heartbeat_stale": "600", 2024-02-12T16:03:42.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_hit_set_max_size": "100000", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_hit_set_min_size": "1000", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_journal_size": "5120", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_kill_backfill_at": "0", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_map_cache_size": "50", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_map_dedup": "true", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_map_message_max": "40", 2024-02-12T16:03:42.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_map_share_max_epochs": "40", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_attr_name_len": "100", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_attr_size": "0", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_backfills": "1", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_markdown_count": "5", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_markdown_period": "600", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_object_name_len": "2048", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_object_namespace_len": "256", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_object_size": "134217728", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_pg_blocked_by": "16", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_pg_log_entries": "10000", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_pgls": "1024", 2024-02-12T16:03:42.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_push_cost": "8388608", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_push_objects": "10", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_scrubs": "1", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_trimming_pgs": "2", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_write_op_reply_len": "64", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_max_write_size": "90", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2024-02-12T16:03:42.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_profile": "high_client_ops", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2024-02-12T16:03:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_client_lim": "999999", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_client_res": "1", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_memory_base": "805306368", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_memory_cache_min": "134217728", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_memory_target": "4294967296", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_memory_target_autotune": "false", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_min_pg_log_entries": "250", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_min_recovery_priority": "0", 2024-02-12T16:03:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mon_report_interval": "5", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_num_cache_shards": "32", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_num_op_tracker_shard": "32", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_numa_auto_affinity": "true", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_numa_node": "-1", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_numa_prefer_iface": "true", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_objecter_finishers": "1", 2024-02-12T16:03:42.816 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_objectstore": "bluestore", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_objectstore_fuse": "false", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_objectstore_tracing": "false", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_complaint_time": "30.000000", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_history_duration": "600", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_history_size": "20", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_history_slow_op_size": "20", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_log_threshold": "5", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_num_shards": "0", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_num_shards_hdd": "5", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_num_shards_ssd": "8", 2024-02-12T16:03:42.817 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_pq_min_cost": "65536", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_queue": "wpq", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_queue_cut_off": "high", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_op_thread_timeout": "15", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_open_classes_on_start": "true", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_os_flags": "0", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_peering_op_priority": "255", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_delete_cost": "1048576", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_delete_priority": "5", 2024-02-12T16:03:42.818 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_log_trim_max": "10000", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_log_trim_min": "100", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pg_stat_report_interval_max": "500", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-12T16:03:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_flags": "0", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_min_size": "2", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_pg_num": "32", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_pgp_num": "0", 2024-02-12T16:03:42.820 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_size": "2", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_default_type": "replicated", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_push_per_object_cost": "1000", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recover_clone_overlap": "true", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_cost": "20971520", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_max_active": "0", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-12T16:03:42.821 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_max_single_start": "1", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_op_priority": "3", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_priority": "5", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_sleep": "0.000000", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_repair_during_recovery": "false", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_requested_scrub_priority": "120", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_auto_repair": "false", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-12T16:03:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_begin_hour": "0", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_begin_week_day": "0", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_chunk_max": "25", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_chunk_min": "5", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_cost": "52428800", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_during_recovery": "false", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_end_hour": "0", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_end_week_day": "0", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_invalid_stats": "true", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_max_preemptions": "5", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-12T16:03:42.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_priority": "5", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_scrub_sleep": "0.000000", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_skip_data_digest": "false", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_smart_report_timeout": "5", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_snap_trim_cost": "1048576", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_snap_trim_priority": "5", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_target_transaction_size": "30", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-12T16:03:42.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_tracing": "false", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_use_stale_snap": "false", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "osdc_blkin_trace_all": "false", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_kill_at": "0", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_max_join_drift": "10", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_min": "500", 2024-02-12T16:03:42.825 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_min_wait": "0.050000", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_propose_interval": "1.000000", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_service_trim_max": "500", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_service_trim_min": "10", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_stash_full_interval": "25", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_trim_max": "500", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "paxos_trim_min": "250", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "perf": "true", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "pid_file": "", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "-", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrv": "", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "public_bind_addr": "-", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "public_network": "", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "public_network_interface": "", 2024-02-12T16:03:42.826 INFO:teuthology.orchestra.run.smithi121.stdout: "qat_compressor_enabled": "false", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rados_mon_op_timeout": "0", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rados_osd_op_timeout": "0", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rados_tracing": "false", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_atime_update_interval": "60", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_balance_parent_reads": "false", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_balance_snap_reads": "false", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_blkin_trace_all": "false", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache": "true", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-12T16:03:42.827 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_policy": "writearound", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_size": "33554432", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_clone_copy_on_read": "false", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_compression_hint": "none", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_concurrent_management_ops": "10", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_clone_format": "auto", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_data_pool": "", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_features": "61", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_format": "2", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_map_options": "", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_order": "22", 2024-02-12T16:03:42.828 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_pool": "rbd", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_stripe_count": "0", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_default_stripe_unit": "0", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_enable_alloc_hint": "true", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_io_scheduler": "simple", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-12T16:03:42.829 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_order": "24", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_pool": "", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_journal_splay_width": "4", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_localize_parent_reads": "false", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_localize_snap_reads": "false", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-12T16:03:42.830 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-12T16:03:42.831 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_mtime_update_interval": "60", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_non_blocking_aio": "true", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_op_thread_timeout": "60", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_op_threads": "1", 2024-02-12T16:03:42.832 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_parent_cache_enabled": "false", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_plugins": "", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_bps_burst": "0", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_bps_limit": "0", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_iops_burst": "0", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_iops_limit": "0", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-12T16:03:42.833 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_read_from_replica_policy": "default", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_skip_partial_discard": "true", 2024-02-12T16:03:42.834 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_tracing": "false", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_validate_names": "true", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_validate_pool": "true", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "restapi_base_url": "", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "restapi_log_level": "", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_acl_grants_max_num": "100", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_admin_entry": "admin", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_barbican_url": "", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_cache_enabled": "true", 2024-02-12T16:03:42.835 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_cache_expiry_interval": "900", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_cache_lru_size": "10000", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_content_length_compat": "false", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_copy_obj_progress": "true", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_cors_rules_max_num": "100", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_cross_domain_policy": "", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_addr": "", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-12T16:03:42.836 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_password": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_kmip_username": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_require_ssl": "true", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_addr": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_auth": "token", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_namespace": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_prefix": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-12T16:03:42.837 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_token_file": "", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_curl_buffersize": "524288", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_curl_low_speed_time": "300", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_data_log_changes_size": "1000", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_data_log_num_shards": "128", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_data_log_window": "30", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_data_notify_interval_msec": "200", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-12T16:03:42.838 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-12T16:03:42.839 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dns_name": "", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dns_s3website_name": "", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_dynamic_resharding": "true", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enable_gc_threads": "true", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enable_lc_threads": "true", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enable_ops_log": "false", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enable_quota_threads": "true", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enable_static_website": "false", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enable_usage_log": "false", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_enforce_swift_acls": "true", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_exit_timeout_secs": "120", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_expose_bucket": "false", 2024-02-12T16:03:42.840 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_extended_http_attrs": "", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_fcgi_socket_backlog": "1024", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_frontends": "beast port=7480", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_max_deferred": "50", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_max_objs": "32", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_gc_processor_period": "3600", 2024-02-12T16:03:42.841 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_host": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_init_timeout": "300", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_domain": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_password": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_password_path": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_project": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_tenant": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_token": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_token_path": "", 2024-02-12T16:03:42.842 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_admin_user": "", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_api_version": "2", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_barbican_domain": "", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_barbican_password": "", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_barbican_project": "", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_barbican_user": "", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_url": "", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-12T16:03:42.843 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lc_debug_interval": "-1", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lc_lock_max_time": "90", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lc_max_objs": "32", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lc_max_rules": "1000", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lc_max_worker": "3", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lc_thread_delay": "0", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ldap_dnattr": "uid", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ldap_searchfilter": "", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-12T16:03:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_log_http_headers": "", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_log_object_name_utc": "false", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_attr_name_len": "0", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_attr_size": "0", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_chunk_size": "4194304", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_listing_results": "1000", 2024-02-12T16:03:42.845 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_notify_retries": "10", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_put_param_size": "1048576", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_put_size": "5368709120", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_max_slo_entries": "1000", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_md_log_max_shards": "64", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_mp_lock_max_time": "600", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-12T16:03:42.846 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_max_gc": "300", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_num_async_rados_threads": "32", 2024-02-12T16:03:42.847 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_num_control_oids": "8", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_num_rados_handles": "1", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_numa_node": "-1", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_objexp_chunk_size": "100", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_objexp_gc_interval": "600", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_op_thread_timeout": "600", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_op_tracing": "false", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_opa_token": "", 2024-02-12T16:03:42.848 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_opa_url": "", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_opa_verify_ssl": "true", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ops_log_file_path": "", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ops_log_rados": "true", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_ops_log_socket_path": "", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_period_push_interval": "2.000000", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_port": "", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_print_continue": "true", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-12T16:03:42.849 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_rados_tracing": "false", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_realm": "", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_region": "", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_request_uri": "", 2024-02-12T16:03:42.850 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_reshard_batch_size": "64", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_reshard_max_aio": "128", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_reshard_num_logs": "16", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_reshard_thread_interval": "600", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_resolve_cname": "false", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_run_sync_thread": "true", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-12T16:03:42.851 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_scheduler_type": "throttler", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_script_uri": "", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_service_provider_name": "", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_socket_path": "", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sts_client_id": "", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sts_client_secret": "", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sts_entry": "sts", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sts_key": "sts", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sts_min_session_duration": "900", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sts_token_introspection_url": "", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_account_in_url": "false", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_auth_entry": "auth", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_auth_url": "", 2024-02-12T16:03:42.852 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_custom_header": "", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_need_stats": "true", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_tenant_name": "", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_token_expiration": "86400", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_url": "", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_url_prefix": "swift", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_lease_period": "120", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-12T16:03:42.853 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_thread_pool_size": "512", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_torrent_comment": "", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_torrent_createby": "", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_torrent_encoding": "", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_torrent_flag": "false", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_torrent_origin": "", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_torrent_tracker": "", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_trust_forwarded_https": "false", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-12T16:03:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_usage_max_shards": "32", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_usage_max_user_shards": "1", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_use_opa_authz": "false", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_max_buckets": "1000", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_user_unique_email": "true", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_verify_ssl": "true", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-12T16:03:42.855 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_zone": "", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_zonegroup": "", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_block_size": "4096", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cache_size": "536870912", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cf_compact_on_deletion": "false", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-12T16:03:42.856 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_index_type": "binary_search", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_partition_filters": "false", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_perf": "false", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "run_dir": "/var/run/ceph", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "setgroup": "ceph", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "setuser": "ceph", 2024-02-12T16:03:42.857 INFO:teuthology.orchestra.run.smithi121.stdout: "setuser_match_path": "", 2024-02-12T16:03:42.858 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-12T16:03:42.858 INFO:teuthology.orchestra.run.smithi121.stdout: "thp": "false", 2024-02-12T16:03:42.858 INFO:teuthology.orchestra.run.smithi121.stdout: "threadpool_default_timeout": "60", 2024-02-12T16:03:42.858 INFO:teuthology.orchestra.run.smithi121.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-12T16:03:42.858 INFO:teuthology.orchestra.run.smithi121.stdout: "throttler_perf_counter": "true" 2024-02-12T16:03:42.858 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:03:42.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:42 smithi121 bash[18489]: cluster 2024-02-12T16:03:41.530421+0000 mgr.y (mgr.14146) 543 : cluster [DBG] pgmap v598: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:42.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:42 smithi123 bash[19119]: cluster 2024-02-12T16:03:41.530421+0000 mgr.y (mgr.14146) 543 : cluster [DBG] pgmap v598: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:42 smithi169 bash[19604]: cluster 2024-02-12T16:03:41.530421+0000 mgr.y (mgr.14146) 543 : cluster [DBG] pgmap v598: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:43.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-12T16:03:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:44 smithi123 bash[19119]: cluster 2024-02-12T16:03:43.531099+0000 mgr.y (mgr.14146) 544 : cluster [DBG] pgmap v599: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:44.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:44 smithi121 bash[18489]: cluster 2024-02-12T16:03:43.531099+0000 mgr.y (mgr.14146) 544 : cluster [DBG] pgmap v599: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:44 smithi169 bash[19604]: cluster 2024-02-12T16:03:43.531099+0000 mgr.y (mgr.14146) 544 : cluster [DBG] pgmap v599: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:46 smithi169 bash[19604]: cluster 2024-02-12T16:03:45.531848+0000 mgr.y (mgr.14146) 545 : cluster [DBG] pgmap v600: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:46.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:46 smithi123 bash[19119]: cluster 2024-02-12T16:03:45.531848+0000 mgr.y (mgr.14146) 545 : cluster [DBG] pgmap v600: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:46.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:46 smithi121 bash[18489]: cluster 2024-02-12T16:03:45.531848+0000 mgr.y (mgr.14146) 545 : cluster [DBG] pgmap v600: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:48.176 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:48.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:48 smithi123 bash[19119]: cluster 2024-02-12T16:03:47.532487+0000 mgr.y (mgr.14146) 546 : cluster [DBG] pgmap v601: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:48.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:48 smithi121 bash[18489]: cluster 2024-02-12T16:03:47.532487+0000 mgr.y (mgr.14146) 546 : cluster [DBG] pgmap v601: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:49.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:48 smithi169 bash[19604]: cluster 2024-02-12T16:03:47.532487+0000 mgr.y (mgr.14146) 546 : cluster [DBG] pgmap v601: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:50.816 INFO:teuthology.orchestra.run.smithi121.stdout:mon.a: {} 2024-02-12T16:03:50.834 INFO:teuthology.orchestra.run.smithi121.stdout:mon.c: {} 2024-02-12T16:03:50.851 INFO:teuthology.orchestra.run.smithi121.stdout:mon.b: {} 2024-02-12T16:03:50.852 INFO:teuthology.orchestra.run.smithi121.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-12T16:03:50.853 INFO:teuthology.orchestra.run.smithi121.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-12T16:03:50.853 INFO:teuthology.orchestra.run.smithi121.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-12T16:03:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:50 smithi123 bash[19119]: cluster 2024-02-12T16:03:49.533152+0000 mgr.y (mgr.14146) 547 : cluster [DBG] pgmap v602: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:50.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:50 smithi121 bash[18489]: cluster 2024-02-12T16:03:49.533152+0000 mgr.y (mgr.14146) 547 : cluster [DBG] pgmap v602: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:50 smithi169 bash[19604]: cluster 2024-02-12T16:03:49.533152+0000 mgr.y (mgr.14146) 547 : cluster [DBG] pgmap v602: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:51.488 INFO:teuthology.run_tasks:Running task full_sequential... 2024-02-12T16:03:51.497 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:03:51.498 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:03:51.498 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-12T16:03:51.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-02-12T16:03:51.499 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-12T16:03:51.499 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:03:51.499 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:03:51.499 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-12T16:03:51.499 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:03:51.499 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:03:51.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-12T16:03:51.500 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-12T16:03:51.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:51.501 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-12T16:03:51.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:03:51.501 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-12T16:03:51.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:03:51.502 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-12T16:03:51.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:03:51.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:51.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:03:51.645 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:03:51.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:03:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:51.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:03:51.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:51.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:51 smithi121 bash[18489]: audit 2024-02-12T16:03:50.793517+0000 mon.a (mon.0) 833 : audit [DBG] from='client.? v1:172.21.15.121:0/2302714477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:03:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:51 smithi169 bash[19604]: audit 2024-02-12T16:03:50.793517+0000 mon.a (mon.0) 833 : audit [DBG] from='client.? v1:172.21.15.121:0/2302714477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:03:51.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:51.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:51.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:51 smithi123 bash[19119]: audit 2024-02-12T16:03:50.793517+0000 mon.a (mon.0) 833 : audit [DBG] from='client.? v1:172.21.15.121:0/2302714477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:03:51.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:51.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:51.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:03:52.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:52.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:52.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:52.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:52.238 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:52.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:03:52.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:52.381 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:52.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:03:52.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:52.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:52.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:03:52.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:52.662 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:52.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:03:52.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:03:52.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:03:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:52.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:03:52.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:52 smithi123 bash[19119]: cluster 2024-02-12T16:03:51.533659+0000 mgr.y (mgr.14146) 548 : cluster [DBG] pgmap v603: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:52.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:52 smithi121 bash[18489]: cluster 2024-02-12T16:03:51.533659+0000 mgr.y (mgr.14146) 548 : cluster [DBG] pgmap v603: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:52.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:52.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:52.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:03:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:52 smithi169 bash[19604]: cluster 2024-02-12T16:03:51.533659+0000 mgr.y (mgr.14146) 548 : cluster [DBG] pgmap v603: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:53.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:03:53.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:03:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:53.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:03:53.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:53.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:53.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:53.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:53.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:53.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:03:53.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:53.517 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:53.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:03:53.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:53.657 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:53.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:03:53.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:53.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:03:53.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:53.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:53.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:03:54.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:54.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:54.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:03:54.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:54.218 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:54.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:03:54.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:54.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:54.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:54.492 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:54.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:54.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:54.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:54.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:03:54.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:03:54.776 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:03:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:54.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:03:54.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:54.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:54 smithi123 bash[19119]: cluster 2024-02-12T16:03:53.534256+0000 mgr.y (mgr.14146) 549 : cluster [DBG] pgmap v604: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:54.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:54 smithi121 bash[18489]: cluster 2024-02-12T16:03:53.534256+0000 mgr.y (mgr.14146) 549 : cluster [DBG] pgmap v604: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:54.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:54.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:03:55.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:54 smithi169 bash[19604]: cluster 2024-02-12T16:03:53.534256+0000 mgr.y (mgr.14146) 549 : cluster [DBG] pgmap v604: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:55.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:55.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:03:55.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:55.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:55.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:03:55.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:55.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:55.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:03:55.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:55.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:55.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:03:55.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:55.622 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:55.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:03:55.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:55.765 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:55.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:03:55.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:55.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:55.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:55.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:03:56.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:56.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:56.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:03:56.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:56.186 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:56.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:03:56.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:56.295 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:03:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:56 smithi169 bash[19604]: cluster 2024-02-12T16:03:55.534927+0000 mgr.y (mgr.14146) 550 : cluster [DBG] pgmap v605: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:56.306 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:56.306 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:56.306 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:56.307 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:03:56.322 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:03:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:56.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:03:56.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:56.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:56.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:03:56.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:03:56.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:56 smithi121 bash[18489]: cluster 2024-02-12T16:03:55.534927+0000 mgr.y (mgr.14146) 550 : cluster [DBG] pgmap v605: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:56.609 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:03:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:56.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:03:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:56 smithi123 bash[19119]: cluster 2024-02-12T16:03:55.534927+0000 mgr.y (mgr.14146) 550 : cluster [DBG] pgmap v605: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:56.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:56.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:56.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:56.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:56.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:03:56.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:03:57.026 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:03:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:57.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:03:57.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:57.166 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:57.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:57.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:03:57.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:57.304 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:03:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:57.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:03:57.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:57.448 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:57.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:03:57.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:57.590 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:57.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:03:57.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:57.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:03:57.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:57.870 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:57.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:03:57.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:58.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:03:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:03:58.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:03:58.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:03:58.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:58.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:03:58.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:03:58.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:58.701 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:58.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:03:58 smithi121 bash[18489]: cluster 2024-02-12T16:03:57.535588+0000 mgr.y (mgr.14146) 551 : cluster [DBG] pgmap v606: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:58.841 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:03:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:03:58.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:03:58 smithi123 bash[19119]: cluster 2024-02-12T16:03:57.535588+0000 mgr.y (mgr.14146) 551 : cluster [DBG] pgmap v606: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:58.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:58.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:03:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:58.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:03:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:03:58 smithi169 bash[19604]: cluster 2024-02-12T16:03:57.535588+0000 mgr.y (mgr.14146) 551 : cluster [DBG] pgmap v606: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:03:59.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:59.121 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:03:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:59.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:03:59.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.237 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:03:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T16:03:59.237 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:03:59.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:59.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:03:59.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:59.401 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:03:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:59.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:03:59.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:59.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:03:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:59.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:03:59.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.684 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:59.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:03:59.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.817 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:59.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:03:59.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.957 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:03:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:03:59.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:00.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:00.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:00.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:00.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:00.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:00.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:00.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:00.374 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:00.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:00.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:00.509 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:00.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:00.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:00.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:00.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:00.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:00.765 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:00 smithi121 bash[18489]: cluster 2024-02-12T16:03:59.536226+0000 mgr.y (mgr.14146) 552 : cluster [DBG] pgmap v607: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:00.765 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:00 smithi121 bash[18489]: cluster 2024-02-12T16:03:59.773182+0000 osd.7 (osd.7) 29 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:04:00.766 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:00 smithi121 bash[18489]: cluster 2024-02-12T16:03:59.776668+0000 osd.7 (osd.7) 30 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:04:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:00.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:00 smithi123 bash[19119]: cluster 2024-02-12T16:03:59.536226+0000 mgr.y (mgr.14146) 552 : cluster [DBG] pgmap v607: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:00 smithi123 bash[19119]: cluster 2024-02-12T16:03:59.773182+0000 osd.7 (osd.7) 29 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:04:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:00 smithi123 bash[19119]: cluster 2024-02-12T16:03:59.776668+0000 osd.7 (osd.7) 30 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:04:00.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:00.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:00 smithi169 bash[19604]: cluster 2024-02-12T16:03:59.536226+0000 mgr.y (mgr.14146) 552 : cluster [DBG] pgmap v607: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:00.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:00 smithi169 bash[19604]: cluster 2024-02-12T16:03:59.773182+0000 osd.7 (osd.7) 29 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:04:00.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:00 smithi169 bash[19604]: cluster 2024-02-12T16:03:59.776668+0000 osd.7 (osd.7) 30 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:04:00.928 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:00.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:01.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:01.069 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:01.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:01.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:01.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:01.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:01.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:01.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:01.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:01.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:01.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:01.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:01.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:01.628 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:01.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:01.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:01.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:01.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:01.808 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:04:01.828 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:04:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:04:01.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:01.867 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:04:01.892 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:04:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:01.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:01.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:04:01.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:04:01.969 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:04:02.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:04:02.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:04:02.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:02.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:02.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:04:02.070 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_0' created 2024-02-12T16:04:02.077 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:04:02.078 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:04:02.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:02 smithi121 bash[18489]: cluster 2024-02-12T16:04:00.903897+0000 osd.10 (osd.10) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:02.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:02 smithi121 bash[18489]: cluster 2024-02-12T16:04:00.903997+0000 osd.10 (osd.10) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:02.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:02 smithi121 bash[18489]: audit 2024-02-12T16:04:01.588428+0000 mon.a (mon.0) 834 : audit [INF] from='client.? v1:172.21.15.121:0/1362756154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-12T16:04:02.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:02 smithi121 bash[18489]: audit 2024-02-12T16:04:01.783629+0000 mon.b (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.121:0/2898449090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:02.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:02 smithi121 bash[18489]: audit 2024-02-12T16:04:02.057235+0000 mon.a (mon.0) 835 : audit [INF] from='client.? v1:172.21.15.121:0/1362756154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-12T16:04:02.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:02 smithi121 bash[18489]: cluster 2024-02-12T16:04:02.057308+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e147: 12 total, 12 up, 12 in 2024-02-12T16:04:02.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:02.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:02.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:02.228 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:04:02.228 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:04:02.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:02.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:02.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:02.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:02 smithi123 bash[19119]: cluster 2024-02-12T16:04:00.903897+0000 osd.10 (osd.10) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:02.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:02 smithi123 bash[19119]: cluster 2024-02-12T16:04:00.903997+0000 osd.10 (osd.10) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:02.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:02 smithi123 bash[19119]: audit 2024-02-12T16:04:01.588428+0000 mon.a (mon.0) 834 : audit [INF] from='client.? v1:172.21.15.121:0/1362756154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-12T16:04:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:02 smithi123 bash[19119]: audit 2024-02-12T16:04:01.783629+0000 mon.b (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.121:0/2898449090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:02 smithi123 bash[19119]: audit 2024-02-12T16:04:02.057235+0000 mon.a (mon.0) 835 : audit [INF] from='client.? v1:172.21.15.121:0/1362756154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-12T16:04:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:02 smithi123 bash[19119]: cluster 2024-02-12T16:04:02.057308+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e147: 12 total, 12 up, 12 in 2024-02-12T16:04:02.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:02.459 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:02.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:02 smithi169 bash[19604]: cluster 2024-02-12T16:04:00.903897+0000 osd.10 (osd.10) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:02 smithi169 bash[19604]: cluster 2024-02-12T16:04:00.903997+0000 osd.10 (osd.10) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:02 smithi169 bash[19604]: audit 2024-02-12T16:04:01.588428+0000 mon.a (mon.0) 834 : audit [INF] from='client.? v1:172.21.15.121:0/1362756154' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-12T16:04:02.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:02 smithi169 bash[19604]: audit 2024-02-12T16:04:01.783629+0000 mon.b (mon.2) 99 : audit [DBG] from='client.? v1:172.21.15.121:0/2898449090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:02.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:02 smithi169 bash[19604]: audit 2024-02-12T16:04:02.057235+0000 mon.a (mon.0) 835 : audit [INF] from='client.? v1:172.21.15.121:0/1362756154' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-12T16:04:02.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:02 smithi169 bash[19604]: cluster 2024-02-12T16:04:02.057308+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e147: 12 total, 12 up, 12 in 2024-02-12T16:04:02.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:02.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:02.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:02.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:02.741 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:02.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:02.844 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:02.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:02.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:02.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:03.024 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:03.080 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:04:03.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:03 smithi123 bash[19119]: cluster 2024-02-12T16:04:01.536653+0000 mgr.y (mgr.14146) 553 : cluster [DBG] pgmap v608: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:03.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:03 smithi123 bash[19119]: audit 2024-02-12T16:04:02.463263+0000 mon.a (mon.0) 837 : audit [INF] from='client.? v1:172.21.15.121:0/3966937601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:04:03.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:03 smithi123 bash[19119]: cluster 2024-02-12T16:04:03.054734+0000 mon.a (mon.0) 838 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:03.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:03 smithi123 bash[19119]: audit 2024-02-12T16:04:03.065234+0000 mon.a (mon.0) 839 : audit [INF] from='client.? v1:172.21.15.121:0/3966937601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:04:03.113 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:03 smithi123 bash[19119]: cluster 2024-02-12T16:04:03.065410+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e148: 12 total, 12 up, 12 in 2024-02-12T16:04:03.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:03.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:03.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:03 smithi169 bash[19604]: cluster 2024-02-12T16:04:01.536653+0000 mgr.y (mgr.14146) 553 : cluster [DBG] pgmap v608: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:03 smithi169 bash[19604]: audit 2024-02-12T16:04:02.463263+0000 mon.a (mon.0) 837 : audit [INF] from='client.? v1:172.21.15.121:0/3966937601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:04:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:03 smithi169 bash[19604]: cluster 2024-02-12T16:04:03.054734+0000 mon.a (mon.0) 838 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:03 smithi169 bash[19604]: audit 2024-02-12T16:04:03.065234+0000 mon.a (mon.0) 839 : audit [INF] from='client.? v1:172.21.15.121:0/3966937601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:04:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:03 smithi169 bash[19604]: cluster 2024-02-12T16:04:03.065410+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e148: 12 total, 12 up, 12 in 2024-02-12T16:04:03.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:03 smithi121 bash[18489]: cluster 2024-02-12T16:04:01.536653+0000 mgr.y (mgr.14146) 553 : cluster [DBG] pgmap v608: 1 pgs: 1 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:03.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:03 smithi121 bash[18489]: audit 2024-02-12T16:04:02.463263+0000 mon.a (mon.0) 837 : audit [INF] from='client.? v1:172.21.15.121:0/3966937601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:04:03.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:03 smithi121 bash[18489]: cluster 2024-02-12T16:04:03.054734+0000 mon.a (mon.0) 838 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:03.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:03 smithi121 bash[18489]: audit 2024-02-12T16:04:03.065234+0000 mon.a (mon.0) 839 : audit [INF] from='client.? v1:172.21.15.121:0/3966937601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:04:03.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:03 smithi121 bash[18489]: cluster 2024-02-12T16:04:03.065410+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e148: 12 total, 12 up, 12 in 2024-02-12T16:04:03.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-12T16:04:03.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:03.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:03.537 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:03.574 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:03.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:03.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:03.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:03.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:03.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:03.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:04:03.991 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:03.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:04.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:04.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:04.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:04.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:04.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:04.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:04.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:04.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:04.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:04.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:04.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:04.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:04.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:04.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:04.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:04.836 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:04:04.836 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:04:04.836 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-12T16:04:04.836 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:04:04.836 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:04:04.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-12T16:04:04.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:04.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:05.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:05.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:05 smithi121 bash[18489]: cluster 2024-02-12T16:04:03.537241+0000 mgr.y (mgr.14146) 554 : cluster [DBG] pgmap v611: 17 pgs: 3 active+clean, 14 unknown; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:05.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:05 smithi121 bash[18489]: cluster 2024-02-12T16:04:04.063691+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e149: 12 total, 12 up, 12 in 2024-02-12T16:04:05.107 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:05.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:05.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:05.252 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:05.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:05.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:05.368 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:05 smithi123 bash[19119]: cluster 2024-02-12T16:04:03.537241+0000 mgr.y (mgr.14146) 554 : cluster [DBG] pgmap v611: 17 pgs: 3 active+clean, 14 unknown; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:05.368 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:05 smithi123 bash[19119]: cluster 2024-02-12T16:04:04.063691+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e149: 12 total, 12 up, 12 in 2024-02-12T16:04:05.394 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:05.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:05.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:05.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:05.533 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:05.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:05.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:04:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:05 smithi169 bash[19604]: cluster 2024-02-12T16:04:03.537241+0000 mgr.y (mgr.14146) 554 : cluster [DBG] pgmap v611: 17 pgs: 3 active+clean, 14 unknown; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:05 smithi169 bash[19604]: cluster 2024-02-12T16:04:04.063691+0000 mon.a (mon.0) 841 : cluster [DBG] osdmap e149: 12 total, 12 up, 12 in 2024-02-12T16:04:05.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:05.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:05.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:05.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:05.806 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:05.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:05.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:05.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:06.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:06.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:04:06.090 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:06.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:06.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:06.231 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:06.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:06.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:06.347 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:06 smithi123 bash[19119]: cluster 2024-02-12T16:04:05.537979+0000 mgr.y (mgr.14146) 555 : cluster [DBG] pgmap v613: 17 pgs: 9 active+clean, 8 unknown; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:06.377 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:06.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:06.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:06 smithi121 bash[18489]: cluster 2024-02-12T16:04:05.537979+0000 mgr.y (mgr.14146) 555 : cluster [DBG] pgmap v613: 17 pgs: 9 active+clean, 8 unknown; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:06.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:06.493 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:06 smithi169 bash[19604]: cluster 2024-02-12T16:04:05.537979+0000 mgr.y (mgr.14146) 555 : cluster [DBG] pgmap v613: 17 pgs: 9 active+clean, 8 unknown; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:06.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:06.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:06.623 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:06.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:06.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:06.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:06.801 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:06.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:06.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:06.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:06.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:07.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:07.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:07.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:07.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:07.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:07.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:07.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:07.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:07.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:07.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:07.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:07.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:07.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:07.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:07.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:07.796 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:07.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:07.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:07.938 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:07.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:08.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:08.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:08.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:08.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:08.202 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:08.203 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:08.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:08.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:08.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:08.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:08.471 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:08.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:08.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:08.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:08.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:08.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:08.756 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:08.791 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:08.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:08.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:08 smithi123 bash[19119]: cluster 2024-02-12T16:04:07.538737+0000 mgr.y (mgr.14146) 556 : cluster [DBG] pgmap v614: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:08.896 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:08.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:08 smithi169 bash[19604]: cluster 2024-02-12T16:04:07.538737+0000 mgr.y (mgr.14146) 556 : cluster [DBG] pgmap v614: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:08.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:08 smithi121 bash[18489]: cluster 2024-02-12T16:04:07.538737+0000 mgr.y (mgr.14146) 556 : cluster [DBG] pgmap v614: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:08.931 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:08.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:09.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:09.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:09.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:09.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:09.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:09.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:09.218 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:09.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:09.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:09.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:09.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:09.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:09.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:09.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:09.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:09.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:09.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:09.918 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:09.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:10.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:10.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:10.162 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:10.198 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:10.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:10.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:10.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:10.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:10.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:10.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:10.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:10.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:10.620 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:10.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:10.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:10.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:10 smithi169 bash[19604]: cluster 2024-02-12T16:04:09.539470+0000 mgr.y (mgr.14146) 557 : cluster [DBG] pgmap v615: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:10.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:10.875 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:10 smithi123 bash[19119]: cluster 2024-02-12T16:04:09.539470+0000 mgr.y (mgr.14146) 557 : cluster [DBG] pgmap v615: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:10.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:10.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:10.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:10 smithi121 bash[18489]: cluster 2024-02-12T16:04:09.539470+0000 mgr.y (mgr.14146) 557 : cluster [DBG] pgmap v615: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:11.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:11.039 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:11.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:11.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:11.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:11.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:11.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:11.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:11.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:11.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:11.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:11.458 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:11.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:11.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:11.597 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:11.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:11.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:11.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:11.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:11.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:11.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:11.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:11.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:12.026 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:12.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:12.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:12.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:12.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:12.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:12.306 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:12.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:12.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:12.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:12.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:12.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:12.592 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:12.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:12.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:12.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:12.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:12.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:12.853 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:12 smithi123 bash[19119]: cluster 2024-02-12T16:04:11.540169+0000 mgr.y (mgr.14146) 558 : cluster [DBG] pgmap v616: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:12.875 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:12.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:12.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:12 smithi121 bash[18489]: cluster 2024-02-12T16:04:11.540169+0000 mgr.y (mgr.14146) 558 : cluster [DBG] pgmap v616: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:12.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:13.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:13.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:12 smithi169 bash[19604]: cluster 2024-02-12T16:04:11.540169+0000 mgr.y (mgr.14146) 558 : cluster [DBG] pgmap v616: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:13.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:13.158 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:13.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:13.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:13.301 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:13.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:13.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:13.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:13.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:13.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:13.582 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:13.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:13.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:13 smithi169 bash[19604]: cluster 2024-02-12T16:04:12.477271+0000 osd.1 (osd.1) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:13 smithi169 bash[19604]: cluster 2024-02-12T16:04:12.477339+0000 osd.1 (osd.1) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:13 smithi169 bash[19604]: cluster 2024-02-12T16:04:12.610126+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e150: 12 total, 12 up, 12 in 2024-02-12T16:04:13.722 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:13.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:13.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:13.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:13.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:13.884 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:13 smithi121 bash[18489]: cluster 2024-02-12T16:04:12.477271+0000 osd.1 (osd.1) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:13.884 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:13 smithi121 bash[18489]: cluster 2024-02-12T16:04:12.477339+0000 osd.1 (osd.1) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:13.884 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:13 smithi121 bash[18489]: cluster 2024-02-12T16:04:12.610126+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e150: 12 total, 12 up, 12 in 2024-02-12T16:04:13.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:13 smithi123 bash[19119]: cluster 2024-02-12T16:04:12.477271+0000 osd.1 (osd.1) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:13.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:13 smithi123 bash[19119]: cluster 2024-02-12T16:04:12.477339+0000 osd.1 (osd.1) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:13.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:13 smithi123 bash[19119]: cluster 2024-02-12T16:04:12.610126+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e150: 12 total, 12 up, 12 in 2024-02-12T16:04:13.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:14.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:14.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:14.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:14.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:14.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:14.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:14.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:14.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:14.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:14.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:14.705 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:14.722 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:14 smithi121 bash[18489]: cluster 2024-02-12T16:04:13.540801+0000 mgr.y (mgr.14146) 559 : cluster [DBG] pgmap v618: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:14.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:14.846 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:14.946 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:14 smithi123 bash[19119]: cluster 2024-02-12T16:04:13.540801+0000 mgr.y (mgr.14146) 559 : cluster [DBG] pgmap v618: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:14.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:14.962 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:14 smithi169 bash[19604]: cluster 2024-02-12T16:04:13.540801+0000 mgr.y (mgr.14146) 559 : cluster [DBG] pgmap v618: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:14.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:14.991 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:04:14.991 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:04:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:14.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:15.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:15.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:15.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:15.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:15.261 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:15.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:15.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:15.404 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:15.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:15.453 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:04:15.453 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T16:04:15.453 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:04:15.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:15.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:15.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:15.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:15.648 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-12T16:04:15.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:15.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:15.784 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:15.821 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:15.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:15.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:15 smithi123 bash[19119]: audit 2024-02-12T16:04:15.136591+0000 mon.a (mon.0) 843 : audit [INF] from='client.? v1:172.21.15.121:0/2035550293' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:04:15.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:15 smithi123 bash[19119]: audit 2024-02-12T16:04:15.474300+0000 mon.a (mon.0) 844 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:04:15.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:15.929 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:15 smithi121 bash[18489]: audit 2024-02-12T16:04:15.136591+0000 mon.a (mon.0) 843 : audit [INF] from='client.? v1:172.21.15.121:0/2035550293' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:04:15.929 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:15 smithi121 bash[18489]: audit 2024-02-12T16:04:15.474300+0000 mon.a (mon.0) 844 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:04:15.962 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:15.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:15 smithi169 bash[19604]: audit 2024-02-12T16:04:15.136591+0000 mon.a (mon.0) 843 : audit [INF] from='client.? v1:172.21.15.121:0/2035550293' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:04:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:15 smithi169 bash[19604]: audit 2024-02-12T16:04:15.474300+0000 mon.a (mon.0) 844 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:04:16.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:16.104 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:16.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:16.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:16.246 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:16.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:16.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:16.386 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:16.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:16.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:16.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:16.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:16.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:04:16.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:16.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:16.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:16.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:16.806 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:16.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:16.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:16.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:16 smithi121 bash[18489]: cluster 2024-02-12T16:04:15.541386+0000 mgr.y (mgr.14146) 560 : cluster [DBG] pgmap v619: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:16.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:16 smithi121 bash[18489]: audit 2024-02-12T16:04:15.636566+0000 mon.a (mon.0) 845 : audit [INF] from='client.? v1:172.21.15.121:0/2035550293' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:04:16.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:16 smithi121 bash[18489]: cluster 2024-02-12T16:04:15.636616+0000 mon.a (mon.0) 846 : cluster [DBG] osdmap e151: 12 total, 12 up, 12 in 2024-02-12T16:04:16.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:16 smithi121 bash[18489]: audit 2024-02-12T16:04:15.805655+0000 mon.a (mon.0) 847 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:04:16.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:16 smithi121 bash[18489]: audit 2024-02-12T16:04:15.806731+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:04:16.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:16 smithi121 bash[18489]: audit 2024-02-12T16:04:15.812932+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:04:16.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:16.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:16 smithi169 bash[19604]: cluster 2024-02-12T16:04:15.541386+0000 mgr.y (mgr.14146) 560 : cluster [DBG] pgmap v619: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:17.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:16 smithi169 bash[19604]: audit 2024-02-12T16:04:15.636566+0000 mon.a (mon.0) 845 : audit [INF] from='client.? v1:172.21.15.121:0/2035550293' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:04:17.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:16 smithi169 bash[19604]: cluster 2024-02-12T16:04:15.636616+0000 mon.a (mon.0) 846 : cluster [DBG] osdmap e151: 12 total, 12 up, 12 in 2024-02-12T16:04:17.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:16 smithi169 bash[19604]: audit 2024-02-12T16:04:15.805655+0000 mon.a (mon.0) 847 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:04:17.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:16 smithi169 bash[19604]: audit 2024-02-12T16:04:15.806731+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:04:17.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:16 smithi169 bash[19604]: audit 2024-02-12T16:04:15.812932+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:04:17.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:17.084 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:17.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:17.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:16 smithi123 bash[19119]: cluster 2024-02-12T16:04:15.541386+0000 mgr.y (mgr.14146) 560 : cluster [DBG] pgmap v619: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:17.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:16 smithi123 bash[19119]: audit 2024-02-12T16:04:15.636566+0000 mon.a (mon.0) 845 : audit [INF] from='client.? v1:172.21.15.121:0/2035550293' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:04:17.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:16 smithi123 bash[19119]: cluster 2024-02-12T16:04:15.636616+0000 mon.a (mon.0) 846 : cluster [DBG] osdmap e151: 12 total, 12 up, 12 in 2024-02-12T16:04:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:16 smithi123 bash[19119]: audit 2024-02-12T16:04:15.805655+0000 mon.a (mon.0) 847 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:04:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:16 smithi123 bash[19119]: audit 2024-02-12T16:04:15.806731+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:04:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:16 smithi123 bash[19119]: audit 2024-02-12T16:04:15.812932+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:04:17.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:17.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:17.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:17.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:17.370 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:17.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:17.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:17.514 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:17.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:17.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:17.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:17.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:17.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:17.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:17.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:17.938 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:17.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:18.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:18.075 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:04:18.078 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:18.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:18.098 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:04:18.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:04:18.137 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:04:18.159 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:04:18.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:18.189 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:04:18.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:18.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:04:18.240 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:04:18.265 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:04:18.292 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:04:18.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:18.317 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:04:18.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:04:18.356 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:18.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:18.446 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-02-12T16:04:18.449 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:04:18.449 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:04:18.459 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:18.494 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:18.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:18.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:18.634 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:18.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:18.641 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:04:18.641 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:04:18.641 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_42458 2024-02-12T16:04:18.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:04:18.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 5 5 0 0 0 - 0 2024-02-12T16:04:18.670 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:04:18.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:18.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:18.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:18.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:18.886 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:18 smithi123 bash[19119]: cluster 2024-02-12T16:04:17.542054+0000 mgr.y (mgr.14146) 561 : cluster [DBG] pgmap v621: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:18 smithi123 bash[19119]: audit 2024-02-12T16:04:18.039006+0000 mon.a (mon.0) 850 : audit [DBG] from='client.? v1:172.21.15.121:0/565632905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:18 smithi123 bash[19119]: audit 2024-02-12T16:04:18.287150+0000 mon.a (mon.0) 851 : audit [INF] from='client.? v1:172.21.15.121:0/3554404607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:04:18.907 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:18.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:18.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:18 smithi121 bash[18489]: cluster 2024-02-12T16:04:17.542054+0000 mgr.y (mgr.14146) 561 : cluster [DBG] pgmap v621: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:18.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:18 smithi121 bash[18489]: audit 2024-02-12T16:04:18.039006+0000 mon.a (mon.0) 850 : audit [DBG] from='client.? v1:172.21.15.121:0/565632905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:18.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:18 smithi121 bash[18489]: audit 2024-02-12T16:04:18.287150+0000 mon.a (mon.0) 851 : audit [INF] from='client.? v1:172.21.15.121:0/3554404607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:04:19.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:19.019 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:18 smithi169 bash[19604]: cluster 2024-02-12T16:04:17.542054+0000 mgr.y (mgr.14146) 561 : cluster [DBG] pgmap v621: 17 pgs: 17 active+clean; 0 B data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:19.019 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:18 smithi169 bash[19604]: audit 2024-02-12T16:04:18.039006+0000 mon.a (mon.0) 850 : audit [DBG] from='client.? v1:172.21.15.121:0/565632905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:19.019 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:18 smithi169 bash[19604]: audit 2024-02-12T16:04:18.287150+0000 mon.a (mon.0) 851 : audit [INF] from='client.? v1:172.21.15.121:0/3554404607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:04:19.040 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:19.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:19.168 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:19.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:19.302 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:19.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:19.434 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:19.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:19.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:19.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 16 6181 6165 385.166 385.312 0.00236505 0.00259134 2024-02-12T16:04:19.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:19.725 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:19.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:19.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:19.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:19 smithi121 bash[18489]: cluster 2024-02-12T16:04:18.642291+0000 mon.a (mon.0) 852 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:19.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:19 smithi121 bash[18489]: audit 2024-02-12T16:04:18.656460+0000 mon.a (mon.0) 853 : audit [INF] from='client.? v1:172.21.15.121:0/3554404607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:04:19.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:19 smithi121 bash[18489]: cluster 2024-02-12T16:04:18.656581+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e152: 12 total, 12 up, 12 in 2024-02-12T16:04:19.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:19.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:19 smithi169 bash[19604]: cluster 2024-02-12T16:04:18.642291+0000 mon.a (mon.0) 852 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:19 smithi169 bash[19604]: audit 2024-02-12T16:04:18.656460+0000 mon.a (mon.0) 853 : audit [INF] from='client.? v1:172.21.15.121:0/3554404607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:04:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:19 smithi169 bash[19604]: cluster 2024-02-12T16:04:18.656581+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e152: 12 total, 12 up, 12 in 2024-02-12T16:04:20.095 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:20.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:19 smithi123 bash[19119]: cluster 2024-02-12T16:04:18.642291+0000 mon.a (mon.0) 852 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:20.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:19 smithi123 bash[19119]: audit 2024-02-12T16:04:18.656460+0000 mon.a (mon.0) 853 : audit [INF] from='client.? v1:172.21.15.121:0/3554404607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:04:20.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:19 smithi123 bash[19119]: cluster 2024-02-12T16:04:18.656581+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e152: 12 total, 12 up, 12 in 2024-02-12T16:04:20.120 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:20.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:20.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:20.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:20.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:20.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:20.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:20.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:20.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:20.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:20.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:20.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:20.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:20.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:20.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 15 11767 11752 367.159 349.188 0.00307428 0.00272093 2024-02-12T16:04:20.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:20.749 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:20 smithi121 bash[18489]: cluster 2024-02-12T16:04:19.542606+0000 mgr.y (mgr.14146) 562 : cluster [DBG] pgmap v623: 17 pgs: 17 active+clean; 4.8 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:20.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:20.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:20.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:20.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:20.901 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:20.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:20.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:21.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:21.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:20 smithi169 bash[19604]: cluster 2024-02-12T16:04:19.542606+0000 mgr.y (mgr.14146) 562 : cluster [DBG] pgmap v623: 17 pgs: 17 active+clean; 4.8 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:21.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:21.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:21.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:20 smithi123 bash[19119]: cluster 2024-02-12T16:04:19.542606+0000 mgr.y (mgr.14146) 562 : cluster [DBG] pgmap v623: 17 pgs: 17 active+clean; 4.8 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:04:21.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:21.161 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:21.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:21.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:21.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:21.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:21.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:21.420 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:21.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:21.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:04:21.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:21.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:21.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:21.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 17064 17048 355.098 331 0.00281971 0.00281368 2024-02-12T16:04:21.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:21.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:21.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:21.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:04:21.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:21.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:21.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:21.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:22.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:22.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:22.217 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:22.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:22.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:22.313 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:04:22.313 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:04:22.313 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2024-02-12T16:04:22.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:04:22.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:22.415 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:22.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:22.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:22.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:22.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 16 21711 21695 338.928 290.438 0.00191334 0.00294813 2024-02-12T16:04:22.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:22.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:22.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:22.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:22 smithi123 bash[19119]: cluster 2024-02-12T16:04:21.543104+0000 mgr.y (mgr.14146) 563 : cluster [DBG] pgmap v624: 17 pgs: 17 active+clean; 426 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 210 MiB/s wr, 3.36k op/s 2024-02-12T16:04:22.806 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:22.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:22.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:22 smithi121 bash[18489]: cluster 2024-02-12T16:04:21.543104+0000 mgr.y (mgr.14146) 563 : cluster [DBG] pgmap v624: 17 pgs: 17 active+clean; 426 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 210 MiB/s wr, 3.36k op/s 2024-02-12T16:04:22.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:22.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:23.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:22 smithi169 bash[19604]: cluster 2024-02-12T16:04:21.543104+0000 mgr.y (mgr.14146) 563 : cluster [DBG] pgmap v624: 17 pgs: 17 active+clean; 426 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 210 MiB/s wr, 3.36k op/s 2024-02-12T16:04:23.076 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:23.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:23.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:23.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:23.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:23.470 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:23.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:23.596 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:23.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 25301 25285 316.015 224.375 0.00199877 0.00316189 2024-02-12T16:04:23.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:23.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:23.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:23.860 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:23.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:23.992 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:23.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:24.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:24.132 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:24.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:24.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:24.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:24.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:24.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:24.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:24.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:24.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:24.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:24.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:24.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:24.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:24.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 29525 29509 307.343 264 0.0021416 0.00325087 2024-02-12T16:04:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:24.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:24.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:24.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:24.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:24.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:24.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:24 smithi123 bash[19119]: cluster 2024-02-12T16:04:23.543497+0000 mgr.y (mgr.14146) 564 : cluster [DBG] pgmap v625: 17 pgs: 17 active+clean; 763 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 190 MiB/s wr, 3.03k op/s 2024-02-12T16:04:24.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:24 smithi123 bash[19119]: cluster 2024-02-12T16:04:23.547368+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:24.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:24 smithi123 bash[19119]: cluster 2024-02-12T16:04:23.547423+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:24.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:24 smithi123 bash[19119]: cluster 2024-02-12T16:04:23.670980+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e153: 12 total, 12 up, 12 in 2024-02-12T16:04:24.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:24.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:24.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:24 smithi121 bash[18489]: cluster 2024-02-12T16:04:23.543497+0000 mgr.y (mgr.14146) 564 : cluster [DBG] pgmap v625: 17 pgs: 17 active+clean; 763 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 190 MiB/s wr, 3.03k op/s 2024-02-12T16:04:24.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:24 smithi121 bash[18489]: cluster 2024-02-12T16:04:23.547368+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:24.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:24 smithi121 bash[18489]: cluster 2024-02-12T16:04:23.547423+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:24.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:24 smithi121 bash[18489]: cluster 2024-02-12T16:04:23.670980+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e153: 12 total, 12 up, 12 in 2024-02-12T16:04:25.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:25.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:24 smithi169 bash[19604]: cluster 2024-02-12T16:04:23.543497+0000 mgr.y (mgr.14146) 564 : cluster [DBG] pgmap v625: 17 pgs: 17 active+clean; 763 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 190 MiB/s wr, 3.03k op/s 2024-02-12T16:04:25.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:24 smithi169 bash[19604]: cluster 2024-02-12T16:04:23.547368+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:25.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:24 smithi169 bash[19604]: cluster 2024-02-12T16:04:23.547423+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:25.010 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:24 smithi169 bash[19604]: cluster 2024-02-12T16:04:23.670980+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e153: 12 total, 12 up, 12 in 2024-02-12T16:04:25.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:25.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:25.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:25.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:25.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:25.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:25.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:25.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:25.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:25.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 16 33455 33439 298.524 245.625 0.003374 0.0033474 2024-02-12T16:04:25.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:25.686 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:25 smithi123 bash[19119]: cluster 2024-02-12T16:04:24.114105+0000 osd.7 (osd.7) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:25.686 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:25 smithi123 bash[19119]: cluster 2024-02-12T16:04:24.114152+0000 osd.7 (osd.7) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:25.686 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:25 smithi123 bash[19119]: cluster 2024-02-12T16:04:24.673994+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e154: 12 total, 12 up, 12 in 2024-02-12T16:04:25.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:25.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:25.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:25.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:25.935 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:25 smithi121 bash[18489]: cluster 2024-02-12T16:04:24.114105+0000 osd.7 (osd.7) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:25.935 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:25 smithi121 bash[18489]: cluster 2024-02-12T16:04:24.114152+0000 osd.7 (osd.7) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:25.936 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:25 smithi121 bash[18489]: cluster 2024-02-12T16:04:24.673994+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e154: 12 total, 12 up, 12 in 2024-02-12T16:04:25.954 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:25.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:25.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:25 smithi169 bash[19604]: cluster 2024-02-12T16:04:24.114105+0000 osd.7 (osd.7) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:25 smithi169 bash[19604]: cluster 2024-02-12T16:04:24.114152+0000 osd.7 (osd.7) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:25 smithi169 bash[19604]: cluster 2024-02-12T16:04:24.673994+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e154: 12 total, 12 up, 12 in 2024-02-12T16:04:26.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:26.090 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:26.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:26.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:26.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:26.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:26.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:26.352 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:26.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:26.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:26.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:26.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:26.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:26.620 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:26.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:26.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:26.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 16 37014 36998 289.011 222.438 0.00354246 0.00345802 2024-02-12T16:04:26.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:26.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:26 smithi121 bash[18489]: cluster 2024-02-12T16:04:25.543891+0000 mgr.y (mgr.14146) 565 : cluster [DBG] pgmap v628: 17 pgs: 17 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.55k op/s 2024-02-12T16:04:26.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:26.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:26.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:26.859 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:26.865 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:26.872 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:26.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:26.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:26.982 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:26 smithi123 bash[19119]: cluster 2024-02-12T16:04:25.543891+0000 mgr.y (mgr.14146) 565 : cluster [DBG] pgmap v628: 17 pgs: 17 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.55k op/s 2024-02-12T16:04:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:26 smithi169 bash[19604]: cluster 2024-02-12T16:04:25.543891+0000 mgr.y (mgr.14146) 565 : cluster [DBG] pgmap v628: 17 pgs: 17 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.55k op/s 2024-02-12T16:04:27.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:27.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:27.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:27.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:27.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:27.389 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:27.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:27.518 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:27.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:27.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 40617 40601 281.918 225.188 0.00599942 0.0035445 2024-02-12T16:04:27.647 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:27.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:27.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:27.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:27.900 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:27.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:28.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:28.033 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:28.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:28.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:28.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:28.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:28.436 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:28.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:28.571 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:28.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 16 43874 43858 274.081 203.562 0.00239277 0.00364648 2024-02-12T16:04:28.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:28.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:28.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:28.841 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:28.915 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:28 smithi121 bash[18489]: cluster 2024-02-12T16:04:27.544292+0000 mgr.y (mgr.14146) 566 : cluster [DBG] pgmap v629: 17 pgs: 17 active+clean; 1.9 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 245 MiB/s wr, 3.92k op/s 2024-02-12T16:04:28.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:28.953 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:28 smithi123 bash[19119]: cluster 2024-02-12T16:04:27.544292+0000 mgr.y (mgr.14146) 566 : cluster [DBG] pgmap v629: 17 pgs: 17 active+clean; 1.9 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 245 MiB/s wr, 3.92k op/s 2024-02-12T16:04:28.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:28.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:28 smithi169 bash[19604]: cluster 2024-02-12T16:04:27.544292+0000 mgr.y (mgr.14146) 566 : cluster [DBG] pgmap v629: 17 pgs: 17 active+clean; 1.9 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 245 MiB/s wr, 3.92k op/s 2024-02-12T16:04:29.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:29.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:29.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:29.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:29.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:29.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:29.366 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:29.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:29.436 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:04:29.436 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:04:29.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:04:29.436 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:04:29.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.436 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:9683 2.4 e152/147 2:284c42ba:::benchmark_data_smithi169_42458_object9682:head v 152'618, mlcod=152'616)", 2024-02-12T16:04:29.436 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:20.234733+0000", 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.1900171989999997, 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019887583, 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234733+0000", 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234733+0000", 2024-02-12T16:04:29.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234731+0000", 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5652000000000001e-05 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234756+0000", 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234771+0000", 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0709999999999995e-06 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234779+0000", 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4152000000000005e-05 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234843+0000", 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4338999999999999e-05 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.234867+0000", 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01972285 2024-02-12T16:04:29.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.254590+0000", 2024-02-12T16:04:29.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9756000000000001e-05 2024-02-12T16:04:29.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.254620+0000", 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019887583 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:9839 2.4 e152/147 2:2c072216:::benchmark_data_smithi169_42458_object9838:head v 152'629, mlcod=152'627)", 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:20.286230+0000", 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.1385201970000001, 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021282995999999998, 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.442 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286230+0000", 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286230+0000", 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286228+0000", 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5257999999999998e-05 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286283+0000", 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2622999999999999e-05 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286296+0000", 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9800000000000003e-06 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286302+0000", 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041442300000000003 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286716+0000", 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4527000000000001e-05 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.286741+0000", 2024-02-12T16:04:29.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020739539000000001 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.307480+0000", 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2363000000000002e-05 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:20.307512+0000", 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021282995999999998 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:16383 2.4 e152/147 2:25a009f8:::benchmark_data_smithi169_42458_object16382:head v 152'1055, mlcod=152'1052)", 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:21.491144+0000", 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.933605676, 2024-02-12T16:04:29.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084936260000000006, 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491144+0000", 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491144+0000", 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491143+0000", 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7536e-05 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491190+0000", 2024-02-12T16:04:29.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1181e-05 2024-02-12T16:04:29.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491201+0000", 2024-02-12T16:04:29.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8749999999999997e-06 2024-02-12T16:04:29.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491207+0000", 2024-02-12T16:04:29.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9312000000000001e-05 2024-02-12T16:04:29.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491297+0000", 2024-02-12T16:04:29.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1284000000000002e-05 2024-02-12T16:04:29.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491318+0000", 2024-02-12T16:04:29.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082842590000000004 2024-02-12T16:04:29.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.499602+0000", 2024-02-12T16:04:29.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5519999999999999e-05 2024-02-12T16:04:29.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.499638+0000", 2024-02-12T16:04:29.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084936260000000006 2024-02-12T16:04:29.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.455 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:16377 2.c e152/147 2:3bdbb69d:::benchmark_data_smithi169_42458_object16376:head v 152'1016, mlcod=152'1014)", 2024-02-12T16:04:29.455 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:21.491223+0000", 2024-02-12T16:04:29.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.9335271819999997, 2024-02-12T16:04:29.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00865959, 2024-02-12T16:04:29.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491223+0000", 2024-02-12T16:04:29.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491223+0000", 2024-02-12T16:04:29.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:04:29.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491221+0000", 2024-02-12T16:04:29.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6044000000000004e-05 2024-02-12T16:04:29.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491317+0000", 2024-02-12T16:04:29.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9739999999999997e-06 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491327+0000", 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.711e-06 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491333+0000", 2024-02-12T16:04:29.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2719999999999996e-05 2024-02-12T16:04:29.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491395+0000", 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7263999999999999e-05 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.491412+0000", 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084358820000000004 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.499848+0000", 2024-02-12T16:04:29.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3729999999999997e-05 2024-02-12T16:04:29.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:21.499882+0000", 2024-02-12T16:04:29.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00865959 2024-02-12T16:04:29.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.464 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.465 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:20268 2.5 e152/147 2:a2cdc3c2:::benchmark_data_smithi169_42458_object20267:head v 152'1256, mlcod=152'1254)", 2024-02-12T16:04:29.465 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:22.324980+0000", 2024-02-12T16:04:29.465 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.0997694449999997, 2024-02-12T16:04:29.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010194412999999999, 2024-02-12T16:04:29.466 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.466 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.466 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.324980+0000", 2024-02-12T16:04:29.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.324980+0000", 2024-02-12T16:04:29.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.324979+0000", 2024-02-12T16:04:29.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107188 2024-02-12T16:04:29.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325086+0000", 2024-02-12T16:04:29.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.134e-05 2024-02-12T16:04:29.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325097+0000", 2024-02-12T16:04:29.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5309999999999998e-06 2024-02-12T16:04:29.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325104+0000", 2024-02-12T16:04:29.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3508000000000002e-05 2024-02-12T16:04:29.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325167+0000", 2024-02-12T16:04:29.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9383000000000001e-05 2024-02-12T16:04:29.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325187+0000", 2024-02-12T16:04:29.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099178240000000004 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.335105+0000", 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0053000000000002e-05 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.335175+0000", 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010194412999999999 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:20273 2.5 e152/147 2:af640554:::benchmark_data_smithi169_42458_object20272:head v 152'1257, mlcod=152'1254)", 2024-02-12T16:04:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:22.325285+0000", 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 7.0994645050000003, 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099282450000000005, 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325285+0000", 2024-02-12T16:04:29.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325285+0000", 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325284+0000", 2024-02-12T16:04:29.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0949e-05 2024-02-12T16:04:29.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325365+0000", 2024-02-12T16:04:29.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4099999999999997e-06 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325374+0000", 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6280000000000004e-06 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325379+0000", 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051419200000000001 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325893+0000", 2024-02-12T16:04:29.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8164000000000001e-05 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.325911+0000", 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092682319999999995 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.335179+0000", 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3976999999999997e-05 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:22.335213+0000", 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099282450000000005 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:27883 2.c e153/147 2:39790f93:::benchmark_data_smithi169_42458_object27882:head v 153'1766, mlcod=153'1763)", 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:24.241713+0000", 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1830369019999996, 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011142312999999999, 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241713+0000", 2024-02-12T16:04:29.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241713+0000", 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241710+0000", 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6105000000000001e-05 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241747+0000", 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6545000000000002e-05 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241763+0000", 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4710000000000007e-06 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241772+0000", 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3310999999999998e-05 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241825+0000", 2024-02-12T16:04:29.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0571999999999999e-05 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.241845+0000", 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010977840000000001 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252823+0000", 2024-02-12T16:04:29.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1782999999999997e-05 2024-02-12T16:04:29.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252855+0000", 2024-02-12T16:04:29.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011142312999999999 2024-02-12T16:04:29.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.486 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.486 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:27886 2.4 e153/147 2:2b62b518:::benchmark_data_smithi169_42458_object27885:head v 153'1741, mlcod=153'1739)", 2024-02-12T16:04:29.486 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:24.242370+0000", 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1823800640000002, 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097812999999999997, 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242370+0000", 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242370+0000", 2024-02-12T16:04:29.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242368+0000", 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8654999999999999e-05 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242396+0000", 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5294999999999998e-05 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242412+0000", 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2890000000000003e-06 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242418+0000", 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0064999999999997e-05 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242458+0000", 2024-02-12T16:04:29.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5294e-05 2024-02-12T16:04:29.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242473+0000", 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096430149999999996 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252116+0000", 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4691999999999999e-05 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252151+0000", 2024-02-12T16:04:29.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097812999999999997 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:27893 2.c e153/147 2:3174bd61:::benchmark_data_smithi169_42458_object27892:head v 153'1767, mlcod=153'1764)", 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:24.242547+0000", 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1822024960000004, 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010346888, 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.492 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242547+0000", 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242547+0000", 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242546+0000", 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6878999999999994e-05 2024-02-12T16:04:29.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242613+0000", 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2123000000000001e-05 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242625+0000", 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2299999999999999e-06 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.242630+0000", 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075030590000000003 2024-02-12T16:04:29.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.250133+0000", 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0973000000000001e-05 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.250154+0000", 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027252610000000001 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252879+0000", 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4773999999999999e-05 2024-02-12T16:04:29.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252894+0000", 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010346888 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:27892 2.4 e153/147 2:21a6da21:::benchmark_data_smithi169_42458_object27891:head v 153'1742, mlcod=153'1739)", 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:24.243545+0000", 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1812048849999996, 2024-02-12T16:04:29.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093310460000000008, 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.243545+0000", 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.243545+0000", 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.243543+0000", 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9799999999999999e-05 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.243573+0000", 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4551e-05 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.243587+0000", 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.435e-06 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.243595+0000", 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036713240000000001 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.247266+0000", 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1727999999999999e-05 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.247288+0000", 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055704559999999997 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252858+0000", 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7445999999999999e-05 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252876+0000", 2024-02-12T16:04:29.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093310460000000008 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:27908 2.c e153/147 2:34202b79:::benchmark_data_smithi169_42458_object27907:head v 153'1769, mlcod=153'1764)", 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:24.248590+0000", 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1761598070000003, 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091541899999999995, 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.248590+0000", 2024-02-12T16:04:29.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.248590+0000", 2024-02-12T16:04:29.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:04:29.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.248589+0000", 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1019000000000006e-05 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.248660+0000", 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3050000000000004e-06 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.248667+0000", 2024-02-12T16:04:29.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8500000000000002e-06 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.248672+0000", 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034478579999999998 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252120+0000", 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3552000000000001e-05 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.252144+0000", 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055635149999999998 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.257707+0000", 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6835999999999998e-05 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:24.257744+0000", 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091541899999999995 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:44009 2.4 e154/147 2:2ef6bab6:::benchmark_data_smithi169_42458_object44008:head v 154'2738, mlcod=154'2735)", 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:28.682849+0000", 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.74190116900000003, 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091264889999999998, 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.682849+0000", 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.682849+0000", 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.682847+0000", 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1623000000000003e-05 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.682909+0000", 2024-02-12T16:04:29.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2429999999999994e-06 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.682918+0000", 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7089999999999996e-06 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.682925+0000", 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001268684 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.684194+0000", 2024-02-12T16:04:29.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1600000000000002e-05 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.684225+0000", 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076932670000000002 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.691918+0000", 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6539999999999997e-05 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.691975+0000", 2024-02-12T16:04:29.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091264889999999998 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45010 2.4 e155/147 2:2005025c:::benchmark_data_smithi169_42458_object45009:head v 155'2796, mlcod=155'2792)", 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:28.982232+0000", 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.44251781400000001, 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011076021, 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982232+0000", 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982232+0000", 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982230+0000", 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5454000000000001e-05 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982256+0000", 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4890000000000001e-06 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982265+0000", 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3180000000000003e-06 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982272+0000", 2024-02-12T16:04:29.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014048859999999999 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983676+0000", 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8763999999999999e-05 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983695+0000", 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.009584268 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.993279+0000", 2024-02-12T16:04:29.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8452000000000001e-05 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.993308+0000", 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011076021 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45012 2.5 e155/147 2:a4d88954:::benchmark_data_smithi169_42458_object45011:head v 155'2743, mlcod=155'2740)", 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:28.982780+0000", 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.44196961099999998, 2024-02-12T16:04:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010315446000000001, 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982780+0000", 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982780+0000", 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982779+0000", 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4221000000000001e-05 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982803+0000", 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5519000000000002e-05 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982818+0000", 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2640000000000002e-06 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982823+0000", 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5145999999999999e-05 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982858+0000", 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8834e-05 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.982877+0000", 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010182073999999999 2024-02-12T16:04:29.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.993059+0000", 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6850999999999997e-05 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.993096+0000", 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010315446000000001 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45013 2.4 e155/147 2:2121475a:::benchmark_data_smithi169_42458_object45012:head v 155'2797, mlcod=155'2792)", 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:28.983600+0000", 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.44114932000000001, 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097350609999999997, 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983600+0000", 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983600+0000", 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983599+0000", 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2099999999999998e-05 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983621+0000", 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3109999999999995e-06 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983631+0000", 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3569999999999997e-06 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.983636+0000", 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012827559999999999 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.984919+0000", 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1861e-05 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.984941+0000", 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083709390000000009 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.993312+0000", 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3895000000000001e-05 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:28.993335+0000", 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097350609999999997 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45293 2.4 e155/147 2:291fcd98:::benchmark_data_smithi169_42458_object45292:head v 155'2812, mlcod=155'2810)", 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:29.070882+0000", 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.35386810400000002, 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011885428, 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.070882+0000", 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.070882+0000", 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.070880+0000", 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3611000000000001e-05 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.070904+0000", 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0268999999999999e-05 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.070914+0000", 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7719999999999997e-06 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.070921+0000", 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042150879999999996 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.075136+0000", 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.19e-05 2024-02-12T16:04:29.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.075158+0000", 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075636870000000004 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.082722+0000", 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5383999999999998e-05 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.082767+0000", 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011885428 2024-02-12T16:04:29.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45712 2.5 e155/147 2:a3438308:::benchmark_data_smithi169_42458_object45711:head v 155'2790, mlcod=155'2786)", 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:29.202418+0000", 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.22233182600000001, 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010715554, 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.202418+0000", 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.202418+0000", 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.202417+0000", 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9981999999999999e-05 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.202447+0000", 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3991e-05 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.202461+0000", 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6390000000000003e-06 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.202467+0000", 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086339099999999998 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.211101+0000", 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3425e-05 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.211125+0000", 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019752810000000002 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.213100+0000", 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3637000000000003e-05 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.213133+0000", 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010715554 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45863 2.4 e155/147 2:20731381:::benchmark_data_smithi169_42458_object45862:head v 155'2847, mlcod=155'2845)", 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:29.257201+0000", 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.16754875, 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098853589999999998, 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.532 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257201+0000", 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257201+0000", 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257200+0000", 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5225999999999999e-05 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257225+0000", 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6800000000000005e-06 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257234+0000", 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5339999999999999e-06 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257240+0000", 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8943999999999998e-05 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257279+0000", 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2711999999999999e-05 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.257302+0000", 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097455609999999998 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.267047+0000", 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9131000000000002e-05 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.267086+0000", 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098853589999999998 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45869 2.c e155/147 2:346ce5ce:::benchmark_data_smithi169_42458_object45868:head v 155'2842, mlcod=155'2839)", 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:29.258484+0000", 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.166265568, 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086482360000000001, 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.536 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258484+0000", 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258484+0000", 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258483+0000", 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6987000000000001e-05 2024-02-12T16:04:29.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258550+0000", 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2305e-05 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258562+0000", 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8759999999999999e-06 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258569+0000", 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1919000000000002e-05 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258621+0000", 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7719000000000001e-05 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258639+0000", 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084533590000000006 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.267092+0000", 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0441000000000001e-05 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.267132+0000", 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086482360000000001 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:45866 2.4 e155/147 2:284bcbf8:::benchmark_data_smithi169_42458_object45865:head v 155'2848, mlcod=155'2845)", 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:29.258611+0000", 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.16613858000000001, 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085479879999999994, 2024-02-12T16:04:29.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258611+0000", 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258611+0000", 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:04:29.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258610+0000", 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6542000000000001e-05 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258637+0000", 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1005999999999999e-05 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258648+0000", 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3920000000000002e-06 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:04:29.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258653+0000", 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3071000000000002e-05 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258686+0000", 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6514000000000001e-05 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.258703+0000", 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084326650000000006 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:04:29.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.267135+0000", 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3785999999999999e-05 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:29.267159+0000", 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085479879999999994 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:04:29.544 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:04:29.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:29.575 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:29.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:29.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 47008 46992 266.97 195.875 0.00591316 0.00374402 2024-02-12T16:04:29.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:29.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:29.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:29.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:29.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:29.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:29 smithi121 bash[18489]: cluster 2024-02-12T16:04:28.521477+0000 osd.11 (osd.11) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:29.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:29 smithi121 bash[18489]: cluster 2024-02-12T16:04:28.521534+0000 osd.11 (osd.11) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:29.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:29 smithi121 bash[18489]: cluster 2024-02-12T16:04:28.686893+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e155: 12 total, 12 up, 12 in 2024-02-12T16:04:29.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:29.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:29.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:29 smithi169 bash[19604]: cluster 2024-02-12T16:04:28.521477+0000 osd.11 (osd.11) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:29 smithi169 bash[19604]: cluster 2024-02-12T16:04:28.521534+0000 osd.11 (osd.11) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:29 smithi169 bash[19604]: cluster 2024-02-12T16:04:28.686893+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e155: 12 total, 12 up, 12 in 2024-02-12T16:04:30.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:30.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:30.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:29 smithi123 bash[19119]: cluster 2024-02-12T16:04:28.521477+0000 osd.11 (osd.11) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:30.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:29 smithi123 bash[19119]: cluster 2024-02-12T16:04:28.521534+0000 osd.11 (osd.11) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:30.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:29 smithi123 bash[19119]: cluster 2024-02-12T16:04:28.686893+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e155: 12 total, 12 up, 12 in 2024-02-12T16:04:30.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:30.222 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:30.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:30.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:30.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:30.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:30.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:30.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:30.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:30.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:30.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:30.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 50046 50030 260.544 189.875 0.00329239 0.00383633 2024-02-12T16:04:30.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:30.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:30.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:30.790 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:30 smithi121 bash[18489]: cluster 2024-02-12T16:04:28.933529+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:30.790 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:30 smithi121 bash[18489]: cluster 2024-02-12T16:04:28.933576+0000 osd.2 (osd.2) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:30.790 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:30 smithi121 bash[18489]: cluster 2024-02-12T16:04:29.544681+0000 mgr.y (mgr.14146) 567 : cluster [DBG] pgmap v631: 17 pgs: 17 active+clean; 2.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 273 MiB/s wr, 4.38k op/s 2024-02-12T16:04:30.790 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:30 smithi121 bash[18489]: cluster 2024-02-12T16:04:29.698379+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e156: 12 total, 12 up, 12 in 2024-02-12T16:04:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:30 smithi169 bash[19604]: cluster 2024-02-12T16:04:28.933529+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:30 smithi169 bash[19604]: cluster 2024-02-12T16:04:28.933576+0000 osd.2 (osd.2) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:30 smithi169 bash[19604]: cluster 2024-02-12T16:04:29.544681+0000 mgr.y (mgr.14146) 567 : cluster [DBG] pgmap v631: 17 pgs: 17 active+clean; 2.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 273 MiB/s wr, 4.38k op/s 2024-02-12T16:04:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:30 smithi169 bash[19604]: cluster 2024-02-12T16:04:29.698379+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e156: 12 total, 12 up, 12 in 2024-02-12T16:04:30.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:30.892 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:30.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:30.996 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:31.002 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:30 smithi123 bash[19119]: cluster 2024-02-12T16:04:28.933529+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:31.003 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:30 smithi123 bash[19119]: cluster 2024-02-12T16:04:28.933576+0000 osd.2 (osd.2) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:31.003 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:30 smithi123 bash[19119]: cluster 2024-02-12T16:04:29.544681+0000 mgr.y (mgr.14146) 567 : cluster [DBG] pgmap v631: 17 pgs: 17 active+clean; 2.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 273 MiB/s wr, 4.38k op/s 2024-02-12T16:04:31.003 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:30 smithi123 bash[19119]: cluster 2024-02-12T16:04:29.698379+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e156: 12 total, 12 up, 12 in 2024-02-12T16:04:31.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:31.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:04:31.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:31.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:31.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:31.279 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:31.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:31.416 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:31.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:31.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:31.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 51951 51935 249.661 119.062 0.011914 0.00400308 2024-02-12T16:04:31.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:31.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:31.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:31.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:31.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:31.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:31.956 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:31.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:32.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:32.094 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:32.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:32.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:32.222 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:32.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:32.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:32.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:32.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:32.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:32.489 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:32.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:32.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:32.628 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:32.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:32.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 54506 54490 243.233 159.688 0.00283145 0.00410938 2024-02-12T16:04:32.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:32.755 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:32.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:32.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:32.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:32.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:32.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:32 smithi121 bash[18489]: cluster 2024-02-12T16:04:31.350489+0000 osd.8 (osd.8) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:32.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:32 smithi121 bash[18489]: cluster 2024-02-12T16:04:31.350572+0000 osd.8 (osd.8) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:32.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:32 smithi121 bash[18489]: cluster 2024-02-12T16:04:31.545103+0000 mgr.y (mgr.14146) 568 : cluster [DBG] pgmap v633: 17 pgs: 17 active+clean; 2.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 254 MiB/s wr, 4.06k op/s 2024-02-12T16:04:32.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:32 smithi121 bash[18489]: cluster 2024-02-12T16:04:31.704452+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e157: 12 total, 12 up, 12 in 2024-02-12T16:04:32.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:32.988 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:32 smithi123 bash[19119]: cluster 2024-02-12T16:04:31.350489+0000 osd.8 (osd.8) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:32.989 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:32 smithi123 bash[19119]: cluster 2024-02-12T16:04:31.350572+0000 osd.8 (osd.8) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:32.989 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:32 smithi123 bash[19119]: cluster 2024-02-12T16:04:31.545103+0000 mgr.y (mgr.14146) 568 : cluster [DBG] pgmap v633: 17 pgs: 17 active+clean; 2.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 254 MiB/s wr, 4.06k op/s 2024-02-12T16:04:32.989 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:32 smithi123 bash[19119]: cluster 2024-02-12T16:04:31.704452+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e157: 12 total, 12 up, 12 in 2024-02-12T16:04:33.003 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:32 smithi169 bash[19604]: cluster 2024-02-12T16:04:31.350489+0000 osd.8 (osd.8) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:32 smithi169 bash[19604]: cluster 2024-02-12T16:04:31.350572+0000 osd.8 (osd.8) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:32 smithi169 bash[19604]: cluster 2024-02-12T16:04:31.545103+0000 mgr.y (mgr.14146) 568 : cluster [DBG] pgmap v633: 17 pgs: 17 active+clean; 2.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 254 MiB/s wr, 4.06k op/s 2024-02-12T16:04:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:32 smithi169 bash[19604]: cluster 2024-02-12T16:04:31.704452+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e157: 12 total, 12 up, 12 in 2024-02-12T16:04:33.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:33.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:33.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:33.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:33.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:33.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:33.397 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:33.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:33.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:33.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:33.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 57360 57344 238.908 178.375 0.00408179 0.00418396 2024-02-12T16:04:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:33.765 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:33.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:33.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:33.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:33.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:34.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:34.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:34.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:34.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:34.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:34.313 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:34.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:34.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:34.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:34.575 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:34.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 16 59822 59806 233.593 153.875 0.00856246 0.00427916 2024-02-12T16:04:34.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:34.704 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:34.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:34.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:34 smithi169 bash[19604]: cluster 2024-02-12T16:04:33.545498+0000 mgr.y (mgr.14146) 569 : cluster [DBG] pgmap v635: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 192 MiB/s wr, 3.07k op/s 2024-02-12T16:04:34.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:34 smithi169 bash[19604]: cluster 2024-02-12T16:04:33.703611+0000 osd.9 (osd.9) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:34.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:34 smithi169 bash[19604]: cluster 2024-02-12T16:04:33.703665+0000 osd.9 (osd.9) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:34.817 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:34 smithi169 bash[19604]: cluster 2024-02-12T16:04:33.714648+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e158: 12 total, 12 up, 12 in 2024-02-12T16:04:34.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:34.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:34.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:34.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:35.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:35.074 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:34 smithi121 bash[18489]: cluster 2024-02-12T16:04:33.545498+0000 mgr.y (mgr.14146) 569 : cluster [DBG] pgmap v635: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 192 MiB/s wr, 3.07k op/s 2024-02-12T16:04:35.074 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:34 smithi121 bash[18489]: cluster 2024-02-12T16:04:33.703611+0000 osd.9 (osd.9) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:35.074 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:34 smithi121 bash[18489]: cluster 2024-02-12T16:04:33.703665+0000 osd.9 (osd.9) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:35.074 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:34 smithi121 bash[18489]: cluster 2024-02-12T16:04:33.714648+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e158: 12 total, 12 up, 12 in 2024-02-12T16:04:35.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:35.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:35.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:34 smithi123 bash[19119]: cluster 2024-02-12T16:04:33.545498+0000 mgr.y (mgr.14146) 569 : cluster [DBG] pgmap v635: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 192 MiB/s wr, 3.07k op/s 2024-02-12T16:04:35.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:34 smithi123 bash[19119]: cluster 2024-02-12T16:04:33.703611+0000 osd.9 (osd.9) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:35.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:34 smithi123 bash[19119]: cluster 2024-02-12T16:04:33.703665+0000 osd.9 (osd.9) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:35.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:34 smithi123 bash[19119]: cluster 2024-02-12T16:04:33.714648+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e158: 12 total, 12 up, 12 in 2024-02-12T16:04:35.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:35.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:35.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:35.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:35.365 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:04:35.397 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:04:35.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:35.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:35.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:35.424 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:04:35.455 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:04:35.485 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:04:35.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:35.529 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:04:35.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:35.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:35.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:04:35.592 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:04:35.623 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:04:35.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 62233 62217 228.716 150.688 0.00281138 0.00437069 2024-02-12T16:04:35.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:35.657 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:04:35.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:35.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:35.686 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:04:35.734 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:04:35.738 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.738 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.738 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.739 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:04:35.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:35.799 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:35.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:35.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:35.928 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:35.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:36.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:36.054 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:36.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:36.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:36.239 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:04:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:04:36.255 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:04:36.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:36.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:36.434 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:36.458 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:36.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:36 smithi121 bash[18489]: audit 2024-02-12T16:04:35.326534+0000 mon.a (mon.0) 861 : audit [DBG] from='client.? v1:172.21.15.121:0/4079840756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:36.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:36 smithi121 bash[18489]: cluster 2024-02-12T16:04:35.545911+0000 mgr.y (mgr.14146) 570 : cluster [DBG] pgmap v637: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s wr, 2.71k op/s 2024-02-12T16:04:36.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:36 smithi121 bash[18489]: audit 2024-02-12T16:04:36.033276+0000 mon.a (mon.0) 862 : audit [INF] from='client.? v1:172.21.15.121:0/931020053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:04:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:36 smithi169 bash[19604]: audit 2024-02-12T16:04:35.326534+0000 mon.a (mon.0) 861 : audit [DBG] from='client.? v1:172.21.15.121:0/4079840756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:36 smithi169 bash[19604]: cluster 2024-02-12T16:04:35.545911+0000 mgr.y (mgr.14146) 570 : cluster [DBG] pgmap v637: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s wr, 2.71k op/s 2024-02-12T16:04:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:36 smithi169 bash[19604]: audit 2024-02-12T16:04:36.033276+0000 mon.a (mon.0) 862 : audit [INF] from='client.? v1:172.21.15.121:0/931020053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:04:36.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:36.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:36.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:36 smithi123 bash[19119]: audit 2024-02-12T16:04:35.326534+0000 mon.a (mon.0) 861 : audit [DBG] from='client.? v1:172.21.15.121:0/4079840756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:36.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:36 smithi123 bash[19119]: cluster 2024-02-12T16:04:35.545911+0000 mgr.y (mgr.14146) 570 : cluster [DBG] pgmap v637: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s wr, 2.71k op/s 2024-02-12T16:04:36.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:36 smithi123 bash[19119]: audit 2024-02-12T16:04:36.033276+0000 mon.a (mon.0) 862 : audit [INF] from='client.? v1:172.21.15.121:0/931020053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:04:36.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 64603 64587 224.238 148.125 0.00370198 0.00445751 2024-02-12T16:04:36.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:36.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:36.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:36.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:36.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:36.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:36.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:36.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:36.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:37.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:37.110 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:37.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:37.215 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:37.240 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:37.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:37.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:37.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:37.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:37.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:37.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:37 smithi169 bash[19604]: cluster 2024-02-12T16:04:35.513071+0000 osd.4 (osd.4) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:37.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:37 smithi169 bash[19604]: cluster 2024-02-12T16:04:35.513144+0000 osd.4 (osd.4) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:37.492 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:37 smithi169 bash[19604]: cluster 2024-02-12T16:04:36.219181+0000 mon.a (mon.0) 863 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:37.493 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:37 smithi169 bash[19604]: audit 2024-02-12T16:04:36.227890+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.121:0/931020053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:04:37.493 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:37 smithi169 bash[19604]: cluster 2024-02-12T16:04:36.227955+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e159: 12 total, 12 up, 12 in 2024-02-12T16:04:37.493 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:37 smithi169 bash[19604]: audit 2024-02-12T16:04:36.229168+0000 mgr.y (mgr.14146) 571 : audit [DBG] from='client.14823 v1:172.21.15.121:0/3548612631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:04:37.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:37.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:37.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:37.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:37 smithi121 bash[18489]: cluster 2024-02-12T16:04:35.513071+0000 osd.4 (osd.4) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:37.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:37 smithi121 bash[18489]: cluster 2024-02-12T16:04:35.513144+0000 osd.4 (osd.4) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:37.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:37 smithi121 bash[18489]: cluster 2024-02-12T16:04:36.219181+0000 mon.a (mon.0) 863 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:37.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:37 smithi121 bash[18489]: audit 2024-02-12T16:04:36.227890+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.121:0/931020053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:04:37.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:37 smithi121 bash[18489]: cluster 2024-02-12T16:04:36.227955+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e159: 12 total, 12 up, 12 in 2024-02-12T16:04:37.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:37 smithi121 bash[18489]: audit 2024-02-12T16:04:36.229168+0000 mgr.y (mgr.14146) 571 : audit [DBG] from='client.14823 v1:172.21.15.121:0/3548612631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:04:37.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:37 smithi123 bash[19119]: cluster 2024-02-12T16:04:35.513071+0000 osd.4 (osd.4) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:37.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:37 smithi123 bash[19119]: cluster 2024-02-12T16:04:35.513144+0000 osd.4 (osd.4) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:37.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:37 smithi123 bash[19119]: cluster 2024-02-12T16:04:36.219181+0000 mon.a (mon.0) 863 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:04:37.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:37 smithi123 bash[19119]: audit 2024-02-12T16:04:36.227890+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.121:0/931020053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:04:37.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:37 smithi123 bash[19119]: cluster 2024-02-12T16:04:36.227955+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e159: 12 total, 12 up, 12 in 2024-02-12T16:04:37.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:37 smithi123 bash[19119]: audit 2024-02-12T16:04:36.229168+0000 mgr.y (mgr.14146) 571 : audit [DBG] from='client.14823 v1:172.21.15.121:0/3548612631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:04:37.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 67152 67136 220.82 159.312 0.00223022 0.0045259 2024-02-12T16:04:37.650 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:37.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:37.727 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2024-02-12T16:04:36.224458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3693084672,"num_objects":56352,"num_object_clones":0,"num_object_copies":112704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56352,"num_write_kb":3606528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56352,"ondisk_log_size":56352,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10795204,"kb_used_data":7130468,"kb_used_omap":0,"kb_used_meta":3664640,"kb_avail":1114146620,"statfs":{"total":1151940427776,"available":1140886138880,"internally_reserved":0,"allocated":7301599232,"data_stored":7296635971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3752591360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,5,9,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1171914752,"num_objects":17882,"num_object_clones":0,"num_object_copies":35764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17882,"num_write_kb":1144448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.525905"},"pg_stats":[{"pgid":"2.f","version":"157'3549","reported_seq":3566,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-02-12T16:04:33.640316+0000","last_change":"2024-02-12T16:04:03.074738+0000","last_active":"2024-02-12T16:04:33.640316+0000","last_peered":"2024-02-12T16:04:33.640316+0000","last_clean":"2024-02-12T16:04:33.640316+0000","last_became_active":"2024-02-12T16:04:03.074484+0000","last_became_peered":"2024-02-12T16:04:03.074484+0000","last_unstale":"2024-02-12T16:04:33.640316+0000","last_undegraded":"2024-02-12T16:04:33.640316+0000","last_fullsized":"2024-02-12T16:04:33.640316+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3549,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":232587264,"num_objects":3549,"num_object_clones":0,"num_object_copies":7098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3549,"num_write_kb":227136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"156'3231","reported_seq":3245,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-12T16:04:31.500708+0000","last_change":"2024-02-12T16:04:03.106274+0000","last_active":"2024-02-12T16:04:31.500708+0000","last_peered":"2024-02-12T16:04:31.500708+0000","last_clean":"2024-02-12T16:04:31.500708+0000","last_became_active":"2024-02-12T16:04:03.105925+0000","last_became_peered":"2024-02-12T16:04:03.105925+0000","last_unstale":"2024-02-12T16:04:31.500708+0000","last_undegraded":"2024-02-12T16:04:31.500708+0000","last_fullsized":"2024-02-12T16:04:31.500708+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3231,"ondisk_log_size":3231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":211746816,"num_objects":3231,"num_object_clones":0,"num_object_copies":6462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3231,"num_write_kb":206784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"157'3220","reported_seq":3237,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-02-12T16:04:31.818038+0000","last_change":"2024-02-12T16:04:03.078000+0000","last_active":"2024-02-12T16:04:31.818038+0000","last_peered":"2024-02-12T16:04:31.818038+0000","last_clean":"2024-02-12T16:04:31.818038+0000","last_became_active":"2024-02-12T16:04:03.077754+0000","last_became_peered":"2024-02-12T16:04:03.077754+0000","last_unstale":"2024-02-12T16:04:31.818038+0000","last_undegraded":"2024-02-12T16:04:31.818038+0000","last_fullsized":"2024-02-12T16:04:31.818038+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3220,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":211025920,"num_objects":3220,"num_object_clones":0,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":206080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"158'3961","reported_seq":3979,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-02-12T16:04:36.130182+0000","last_change":"2024-02-12T16:04:03.069405+0000","last_active":"2024-02-12T16:04:36.130182+0000","last_peered":"2024-02-12T16:04:36.130182+0000","last_clean":"2024-02-12T16:04:36.130182+0000","last_became_active":"2024-02-12T16:04:03.069103+0000","last_became_peered":"2024-02-12T16:04:03.069103+0000","last_unstale":"2024-02-12T16:04:36.130182+0000","last_undegraded":"2024-02-12T16:04:36.130182+0000","last_fullsized":"2024-02-12T16:04:36.130182+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3961,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":259588096,"num_objects":3961,"num_object_clones":0,"num_object_copies":7922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":253504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"157'3291","reported_seq":3308,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-02-12T16:04:31.825152+0000","last_change":"2024-02-12T16:04:03.076121+0000","last_active":"2024-02-12T16:04:31.825152+0000","last_peered":"2024-02-12T16:04:31.825152+0000","last_clean":"2024-02-12T16:04:31.825152+0000","last_became_active":"2024-02-12T16:04:03.075819+0000","last_became_peered":"2024-02-12T16:04:03.075819+0000","last_unstale":"2024-02-12T16:04:31.825152+0000","last_undegraded":"2024-02-12T16:04:31.825152+0000","last_fullsized":"2024-02-12T16:04:31.825152+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3291,"ondisk_log_size":3291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":215678976,"num_objects":3291,"num_object_clones":0,"num_object_copies":6582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3291,"num_write_kb":210624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":166,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-12T16:04:29.708418+0000","last_change":"2024-02-12T16:03:59.776773+0000","last_active":"2024-02-12T16:04:29.708418+0000","last_peered":"2024-02-12T16:04:29.708418+0000","last_clean":"2024-02-12T16:04:29.708418+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:04:29.708418+0000","last_undegraded":"2024-02-12T16:04:29.708418+0000","last_fullsized":"2024-02-12T16:04:29.708418+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:03:59.776674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"157'3543","reported_seq":3560,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-02-12T16:04:33.307253+0000","last_change":"2024-02-12T16:04:03.113429+0000","last_active":"2024-02-12T16:04:33.307253+0000","last_peered":"2024-02-12T16:04:33.307253+0000","last_clean":"2024-02-12T16:04:33.307253+0000","last_became_active":"2024-02-12T16:04:03.113129+0000","last_became_peered":"2024-02-12T16:04:03.113129+0000","last_unstale":"2024-02-12T16:04:33.307253+0000","last_undegraded":"2024-02-12T16:04:33.307253+0000","last_fullsized":"2024-02-12T16:04:33.307253+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3543,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":232194048,"num_objects":3543,"num_object_clones":0,"num_object_copies":7086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3543,"num_write_kb":226752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"157'3536","reported_seq":3553,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-02-12T16:04:33.636953+0000","last_change":"2024-02-12T16:04:03.111231+0000","last_active":"2024-02-12T16:04:33.636953+0000","last_peered":"2024-02-12T16:04:33.636953+0000","last_clean":"2024-02-12T16:04:33.636953+0000","last_became_active":"2024-02-12T16:04:03.110555+0000","last_became_peered":"2024-02-12T16:04:03.110555+0000","last_unstale":"2024-02-12T16:04:33.636953+0000","last_undegraded":"2024-02-12T16:04:33.636953+0000","last_fullsized":"2024-02-12T16:04:33.636953+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3536,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":231735296,"num_objects":3536,"num_object_clones":0,"num_object_copies":7072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":226304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"156'3134","reported_seq":3150,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-12T16:04:31.509912+0000","last_change":"2024-02-12T16:04:03.071933+0000","last_active":"2024-02-12T16:04:31.509912+0000","last_peered":"2024-02-12T16:04:31.509912+0000","last_clean":"2024-02-12T16:04:31.509912+0000","last_became_active":"2024-02-12T16:04:03.071599+0000","last_became_peered":"2024-02-12T16:04:03.071599+0000","last_unstale":"2024-02-12T16:04:31.509912+0000","last_undegraded":"2024-02-12T16:04:31.509912+0000","last_fullsized":"2024-02-12T16:04:31.509912+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3134,"ondisk_log_size":3134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":205389824,"num_objects":3134,"num_object_clones":0,"num_object_copies":6268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3134,"num_write_kb":200576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"158'3617","reported_seq":3635,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-02-12T16:04:33.853364+0000","last_change":"2024-02-12T16:04:03.114770+0000","last_active":"2024-02-12T16:04:33.853364+0000","last_peered":"2024-02-12T16:04:33.853364+0000","last_clean":"2024-02-12T16:04:33.853364+0000","last_became_active":"2024-02-12T16:04:03.114419+0000","last_became_peered":"2024-02-12T16:04:03.114419+0000","last_unstale":"2024-02-12T16:04:33.853364+0000","last_undegraded":"2024-02-12T16:04:33.853364+0000","last_fullsized":"2024-02-12T16:04:33.853364+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3617,"ondisk_log_size":3617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":237043712,"num_objects":3617,"num_object_clones":0,"num_object_copies":7234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":231488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"157'3656","reported_seq":3672,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-02-12T16:04:33.316801+0000","last_change":"2024-02-12T16:04:03.076402+0000","last_active":"2024-02-12T16:04:33.316801+0000","last_peered":"2024-02-12T16:04:33.316801+0000","last_clean":"2024-02-12T16:04:33.316801+0000","last_became_active":"2024-02-12T16:04:03.075975+0000","last_became_peered":"2024-02-12T16:04:03.075975+0000","last_unstale":"2024-02-12T16:04:33.316801+0000","last_undegraded":"2024-02-12T16:04:33.316801+0000","last_fullsized":"2024-02-12T16:04:33.316801+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3656,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":239599616,"num_objects":3656,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3656,"num_write_kb":233984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"156'3218","reported_seq":3234,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-12T16:04:31.512462+0000","last_change":"2024-02-12T16:04:03.070646+0000","last_active":"2024-02-12T16:04:31.512462+0000","last_peered":"2024-02-12T16:04:31.512462+0000","last_clean":"2024-02-12T16:04:31.512462+0000","last_became_active":"2024-02-12T16:04:03.070386+0000","last_became_peered":"2024-02-12T16:04:03.070386+0000","last_unstale":"2024-02-12T16:04:31.512462+0000","last_undegraded":"2024-02-12T16:04:31.512462+0000","last_fullsized":"2024-02-12T16:04:31.512462+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3218,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":6436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3218,"num_write_kb":205952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"158'3821","reported_seq":3839,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-02-12T16:04:34.845939+0000","last_change":"2024-02-12T16:04:03.073707+0000","last_active":"2024-02-12T16:04:34.845939+0000","last_peered":"2024-02-12T16:04:34.845939+0000","last_clean":"2024-02-12T16:04:34.845939+0000","last_became_active":"2024-02-12T16:04:03.073456+0000","last_became_peered":"2024-02-12T16:04:03.073456+0000","last_unstale":"2024-02-12T16:04:34.845939+0000","last_undegraded":"2024-02-12T16:04:34.845939+0000","last_fullsized":"2024-02-12T16:04:34.845939+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3821,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":250413056,"num_objects":3821,"num_object_clones":0,"num_object_copies":7642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"157'3592","reported_seq":3609,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-02-12T16:04:33.616110+0000","last_change":"2024-02-12T16:04:03.073798+0000","last_active":"2024-02-12T16:04:33.616110+0000","last_peered":"2024-02-12T16:04:33.616110+0000","last_clean":"2024-02-12T16:04:33.616110+0000","last_became_active":"2024-02-12T16:04:03.073550+0000","last_became_peered":"2024-02-12T16:04:03.073550+0000","last_unstale":"2024-02-12T16:04:33.616110+0000","last_undegraded":"2024-02-12T16:04:33.616110+0000","last_fullsized":"2024-02-12T16:04:33.616110+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3592,"ondisk_log_size":3592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":235405312,"num_objects":3592,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3592,"num_write_kb":229888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"158'3722","reported_seq":3740,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-02-12T16:04:34.845884+0000","last_change":"2024-02-12T16:04:03.111009+0000","last_active":"2024-02-12T16:04:34.845884+0000","last_peered":"2024-02-12T16:04:34.845884+0000","last_clean":"2024-02-12T16:04:34.845884+0000","last_became_active":"2024-02-12T16:04:03.110305+0000","last_became_peered":"2024-02-12T16:04:03.110305+0000","last_unstale":"2024-02-12T16:04:34.845884+0000","last_undegraded":"2024-02-12T16:04:34.845884+0000","last_fullsized":"2024-02-12T16:04:34.845884+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3722,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":243924992,"num_objects":3722,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":238208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"156'3210","reported_seq":3226,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-12T16:04:31.093520+0000","last_change":"2024-02-12T16:04:03.110487+0000","last_active":"2024-02-12T16:04:31.093520+0000","last_peered":"2024-02-12T16:04:31.093520+0000","last_clean":"2024-02-12T16:04:31.093520+0000","last_became_active":"2024-02-12T16:04:03.110162+0000","last_became_peered":"2024-02-12T16:04:03.110162+0000","last_unstale":"2024-02-12T16:04:31.093520+0000","last_undegraded":"2024-02-12T16:04:31.093520+0000","last_fullsized":"2024-02-12T16:04:31.093520+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":210370560,"num_objects":3210,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":205440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"158'4051","reported_seq":4069,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-02-12T16:04:36.152649+0000","last_change":"2024-02-12T16:04:03.069883+0000","last_active":"2024-02-12T16:04:36.152649+0000","last_peered":"2024-02-12T16:04:36.152649+0000","last_clean":"2024-02-12T16:04:36.152649+0000","last_became_active":"2024-02-12T16:04:03.069615+0000","last_became_peered":"2024-02-12T16:04:03.069615+0000","last_unstale":"2024-02-12T16:04:36.152649+0000","last_undegraded":"2024-02-12T16:04:36.152649+0000","last_fullsized":"2024-02-12T16:04:36.152649+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":4051,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":265486336,"num_objects":4051,"num_object_clones":0,"num_object_copies":8102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4051,"num_write_kb":259264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3693084672,"num_objects":56352,"num_object_clones":0,"num_object_copies":112704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56352,"num_write_kb":3606528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7478050816,"data_stored":7478050816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56352,"ondisk_log_size":56352,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958605,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004740,"kb_used_data":699324,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92740412,"statfs":{"total":95995035648,"available":94966181888,"internally_reserved":0,"allocated":716107776,"data_stored":715689589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.0129999999999999,"15min":1.03},"min":{"1min":0.69799999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.2190000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0660000000000001,"15min":1.115},"min":{"1min":0.72699999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.319,"5min":2.871,"15min":2.871},"last":0.44}]},{"osd":1,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.008,"15min":1.089},"min":{"1min":0.751,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1950000000000001,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.077,"15min":1.087},"min":{"1min":0.76600000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.3720000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.105,"15min":1.1519999999999999},"min":{"1min":0.71199999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.5680000000000001,"5min":3.2429999999999999,"15min":3.2429999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.006,"5min":1.0960000000000001,"15min":1.2210000000000001},"min":{"1min":0.76500000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.3440000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.81100000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.131,"15min":1.2050000000000001},"min":{"1min":0.745,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.3520000000000001,"5min":2.5299999999999998,"15min":2.5299999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.004,"5min":1.105,"15min":1.1919999999999999},"min":{"1min":0.73699999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.762,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":0.749}]},{"osd":4,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.2050000000000001,"15min":1.349},"min":{"1min":0.755,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.339,"5min":2.9500000000000002,"15min":2.9500000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.3169999999999999,"15min":1.427},"min":{"1min":1.0289999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.5429999999999999,"5min":4.0789999999999997,"15min":4.0789999999999997},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3939999999999999,"15min":1.466},"min":{"1min":1.135,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":1.528,"5min":3.044,"15min":3.044},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.3680000000000001,"15min":1.488},"min":{"1min":1.0529999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.956,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":0.68700000000000006}]},{"osd":6,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.375,"15min":1.514},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":1.538,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":0.623},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.492,"15min":1.6040000000000001},"min":{"1min":1.1639999999999999,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.6799999999999999,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4870000000000001,"15min":1.611},"min":{"1min":1.159,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.655,"5min":4.3129999999999997,"15min":4.3129999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5649999999999999,"15min":1.702},"min":{"1min":1.214,"5min":0.748,"15min":0.748},"max":{"1min":1.722,"5min":3.8370000000000002,"15min":3.8370000000000002},"last":0.69899999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.522,"15min":1.6359999999999999},"min":{"1min":1.038,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.788,"5min":4,"15min":4},"last":0.497},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5880000000000001,"15min":1.722},"min":{"1min":1.2390000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.647,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":0.73999999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.492,"15min":1.6679999999999999},"min":{"1min":1.083,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0209999999999999,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6899999999999999,"15min":1.8180000000000001},"min":{"1min":1.4470000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.855,"5min":4.3949999999999996,"15min":4.3949999999999996},"last":0.76100000000000001}]}]},{"osd":10,"up_from":83,"seq":356482285654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":748856,"kb_used_data":443440,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92996296,"statfs":{"total":95995035648,"available":95228207104,"internally_reserved":0,"allocated":454082560,"data_stored":453670460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.90400000000000003,"15min":0.89700000000000002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.1020000000000001},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.0009999999999999,"15min":1.0580000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.391,"5min":1.538,"15min":1.573},"last":3.4119999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.0029999999999999,"15min":0.98399999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2529999999999999,"5min":1.4490000000000001,"15min":1.4850000000000001},"last":3.6349999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.98999999999999999,"15min":0.96299999999999997},"min":{"1min":0.68100000000000005,"5min":0.65600000000000003,"15min":0.60999999999999999},"max":{"1min":1.3540000000000001,"5min":1.492,"15min":1.492},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.034,"15min":1.014},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.538,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.97699999999999998,"15min":1.008},"min":{"1min":0.71599999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.45}]},{"osd":4,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.246,"15min":1.1359999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.64600000000000002},"max":{"1min":1.466,"5min":1.776,"15min":1.776},"last":0.434},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.206,"15min":1.222},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6020000000000001,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":3.4630000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1619999999999999,"15min":1.1799999999999999},"min":{"1min":0.85499999999999998,"5min":0.83499999999999996,"15min":0.72299999999999998},"max":{"1min":1.7010000000000001,"5min":2.048,"15min":2.048},"last":1.139},{"interface":"front","average":{"1min":1.248,"5min":1.3100000000000001,"15min":1.3029999999999999},"min":{"1min":0.92700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.6279999999999999,"5min":1.726,"15min":1.788},"last":3.556}]},{"osd":6,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.181,"15min":1.216},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.597,"5min":1.661,"15min":1.7569999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4179999999999999,"15min":1.3580000000000001},"min":{"1min":1.002,"5min":1.002,"15min":0.68500000000000005},"max":{"1min":1.5580000000000001,"5min":1.8700000000000001,"15min":2.0270000000000001},"last":3.597}]},{"osd":7,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.379,"15min":1.3009999999999999},"min":{"1min":1.0409999999999999,"5min":1.026,"15min":0.755},"max":{"1min":1.641,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.256,"5min":1.349,"15min":1.3720000000000001},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.83499999999999996},"max":{"1min":1.877,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.2070000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.4019999999999999,"15min":1.3520000000000001},"min":{"1min":0.91300000000000003,"5min":0.84499999999999997,"15min":0.54700000000000004},"max":{"1min":1.587,"5min":2,"15min":2},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.339,"5min":1.4870000000000001,"15min":1.365},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.48399999999999999},"max":{"1min":1.6739999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":3.6819999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:03:58 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.5409999999999999,"15min":1.458},"min":{"1min":1.131,"5min":1.089,"15min":0.752},"max":{"1min":1.756,"5min":2.02,"15min":2.02},"last":3.5219999999999998},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.54,"15min":1.4299999999999999},"min":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":0.52400000000000002},"max":{"1min":1.744,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.50700000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.5680000000000001,"15min":1.601},"min":{"1min":1.167,"5min":1.167,"15min":1.167},"max":{"1min":1.833,"5min":2.077,"15min":2.077},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5920000000000001,"15min":1.629},"min":{"1min":1.175,"5min":1.175,"15min":1.175},"max":{"1min":1.7689999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":3.4929999999999999}]}]},{"osd":9,"up_from":74,"seq":317827579998,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899308,"kb_used_data":602084,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92845844,"statfs":{"total":95995035648,"available":95074144256,"internally_reserved":0,"allocated":616534016,"data_stored":616128023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.92300000000000004,"15min":0.92900000000000005},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2370000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":1.4670000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.93000000000000005,"15min":0.96599999999999997},"min":{"1min":0.65200000000000002,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0860000000000001,"5min":1.4419999999999999,"15min":2.2480000000000002},"last":1.252}]},{"osd":1,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.069,"15min":1.014},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.04,"15min":1.032},"min":{"1min":0.65300000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2789999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":1.3049999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.94599999999999995,"15min":0.93400000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.3140000000000001,"5min":1.343,"15min":1.738},"last":1.502},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.94099999999999995,"15min":1.014},"min":{"1min":0.77200000000000002,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.601},"last":0.438}]},{"osd":3,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94899999999999995,"15min":0.96299999999999997},"min":{"1min":0.71999999999999997,"5min":0.61399999999999999,"15min":0.60699999999999998},"max":{"1min":1.3120000000000001,"5min":1.4470000000000001,"15min":2.1549999999999998},"last":1.631},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1020000000000001,"15min":1.1080000000000001},"min":{"1min":0.69699999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.538,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.405}]},{"osd":4,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.139,"15min":1.165},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.401,"5min":1.6739999999999999,"15min":2.113},"last":0.498},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1459999999999999,"15min":1.1819999999999999},"min":{"1min":0.83599999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.476,"5min":1.5680000000000001,"15min":2.3260000000000001},"last":1.601}]},{"osd":5,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.224,"15min":1.2290000000000001},"min":{"1min":1.0780000000000001,"5min":0.77000000000000002,"15min":0.71499999999999997},"max":{"1min":1.4390000000000001,"5min":1.756,"15min":2.4399999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.383,"5min":1.3440000000000001,"15min":1.284},"min":{"1min":0.98899999999999999,"5min":0.92600000000000005,"15min":0.76300000000000001},"max":{"1min":1.639,"5min":1.6830000000000001,"15min":2.2599999999999998},"last":1.4510000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3009999999999999,"15min":1.319},"min":{"1min":1.1040000000000001,"5min":0.89700000000000002,"15min":0.82199999999999995},"max":{"1min":1.415,"5min":1.8340000000000001,"15min":1.9139999999999999},"last":1.351},{"interface":"front","average":{"1min":1.224,"5min":1.29,"15min":1.3420000000000001},"min":{"1min":0.93200000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.5469999999999999,"5min":1.585,"15min":2.48},"last":1.575}]},{"osd":7,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.254,"15min":1.3100000000000001},"min":{"1min":0.876,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.5549999999999999,"5min":1.7330000000000001,"15min":2.3519999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.381,"5min":1.421,"15min":1.4099999999999999},"min":{"1min":1.054,"5min":0.98099999999999998,"15min":0.875},"max":{"1min":1.5880000000000001,"5min":1.772,"15min":2.0249999999999999},"last":1.536}]},{"osd":8,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.353,"15min":1.3600000000000001},"min":{"1min":0.95299999999999996,"5min":0.86399999999999999,"15min":0.61099999999999999},"max":{"1min":1.6080000000000001,"5min":1.792,"15min":2.1299999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.355,"15min":1.355},"min":{"1min":1.083,"5min":0.82899999999999996,"15min":0.58499999999999996},"max":{"1min":1.5740000000000001,"5min":1.7649999999999999,"15min":2.3889999999999998},"last":2.1970000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:03:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.4139999999999999,"15min":1.4239999999999999},"min":{"1min":1.1100000000000001,"5min":0.90700000000000003,"15min":0.77400000000000002},"max":{"1min":1.623,"5min":2.02,"15min":2.5110000000000001},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.389,"15min":1.353},"min":{"1min":0.94199999999999995,"5min":0.85899999999999999,"15min":0.64200000000000002},"max":{"1min":1.5129999999999999,"5min":1.8400000000000001,"15min":2.0819999999999999},"last":0.54400000000000004}]}]},{"osd":8,"up_from":65,"seq":279172874341,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193528,"kb_used_data":896304,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92551624,"statfs":{"total":95995035648,"available":94772862976,"internally_reserved":0,"allocated":917815296,"data_stored":917403216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.96799999999999997,"15min":0.86499999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.57699999999999996},"max":{"1min":1.3640000000000001,"5min":1.488,"15min":1.488},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.95399999999999996,"15min":0.89900000000000002},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.53400000000000003},"max":{"1min":1.5589999999999999,"5min":1.623,"15min":1.623},"last":0.747}]},{"osd":1,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.88800000000000001,"15min":0.85099999999999998},"min":{"1min":0.65500000000000003,"5min":0.64800000000000002,"15min":0.60199999999999998},"max":{"1min":1.0369999999999999,"5min":1.508,"15min":1.508},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.0169999999999999,"15min":0.96199999999999997},"min":{"1min":0.76800000000000002,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.8480000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0309999999999999,"15min":0.89500000000000002},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.59199999999999997},"max":{"1min":1.3420000000000001,"5min":1.764,"15min":1.764},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.165,"5min":1.1779999999999999,"15min":1.0429999999999999},"min":{"1min":0.86799999999999999,"5min":0.78200000000000003,"15min":0.65600000000000003},"max":{"1min":1.409,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.95799999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.105,"15min":1.0069999999999999},"min":{"1min":0.68300000000000005,"5min":0.64800000000000002,"15min":0.60499999999999998},"max":{"1min":1.5029999999999999,"5min":1.6919999999999999,"15min":1.698},"last":0.375},{"interface":"front","average":{"1min":1.167,"5min":1.2430000000000001,"15min":1.054},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.623},"max":{"1min":1.5920000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.79200000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2689999999999999,"15min":1.163},"min":{"1min":0.92400000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.4910000000000001,"5min":1.677,"15min":1.839},"last":0.307},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.343,"15min":1.131},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.69999999999999996},"max":{"1min":1.647,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.82299999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.177,"15min":1.151},"min":{"1min":0.84099999999999997,"5min":0.76900000000000002,"15min":0.75800000000000001},"max":{"1min":1.6870000000000001,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.335,"5min":1.3700000000000001,"15min":1.2170000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.68899999999999995},"max":{"1min":1.631,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.34100000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.3380000000000001,"15min":1.2},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.65600000000000003},"max":{"1min":1.5740000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4079999999999999,"15min":1.262},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.876},"max":{"1min":1.8340000000000001,"5min":2.339,"15min":2.339},"last":0.86299999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.47,"15min":1.274},"min":{"1min":1.0529999999999999,"5min":1.0369999999999999,"15min":0.73599999999999999},"max":{"1min":1.6060000000000001,"5min":1.97,"15min":2.0179999999999998},"last":1.008},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.51,"15min":1.3100000000000001},"min":{"1min":1.1140000000000001,"5min":1.036,"15min":0.85499999999999998},"max":{"1min":1.776,"5min":1.798,"15min":1.984},"last":0.66500000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.522,"15min":1.294},"min":{"1min":0.95399999999999996,"5min":0.70499999999999996,"15min":0.50900000000000001},"max":{"1min":1.8680000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3200000000000001,"15min":1.232},"min":{"1min":0.85699999999999998,"5min":0.72999999999999998,"15min":0.67200000000000004},"max":{"1min":1.802,"5min":2.456,"15min":2.456},"last":0.69099999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:03:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.4099999999999999,"15min":1.3640000000000001},"min":{"1min":1.0589999999999999,"5min":0.91700000000000004,"15min":0.73399999999999999},"max":{"1min":1.8640000000000001,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.516,"15min":1.4670000000000001},"min":{"1min":1.319,"5min":0.88200000000000001,"15min":0.76300000000000001},"max":{"1min":1.732,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.40500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":7,"up_from":56,"seq":240518168686,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963640,"kb_used_data":658224,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92781512,"statfs":{"total":95995035648,"available":95008268288,"internally_reserved":0,"allocated":674021376,"data_stored":673609296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:03:51 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0189999999999999,"15min":0.996},"min":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.436},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96599999999999997,"15min":0.96299999999999997},"min":{"1min":0.76800000000000002,"5min":0.64100000000000001,"15min":0.53400000000000003},"max":{"1min":1.288,"5min":1.353,"15min":1.353},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.97499999999999998,"15min":1.0169999999999999},"min":{"1min":0.748,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.3049999999999999,"5min":1.413,"15min":1.8580000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0429999999999999,"15min":1.0449999999999999},"min":{"1min":0.74299999999999999,"5min":0.63800000000000001,"15min":0.56499999999999995},"max":{"1min":1.599,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.518}]},{"osd":3,"last update":"Mon Feb 12 16:03:51 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0509999999999999,"15min":1.0780000000000001},"min":{"1min":0.80200000000000005,"5min":0.628,"15min":0.53400000000000003},"max":{"1min":1.3720000000000001,"5min":1.954,"15min":2.0049999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.018,"15min":1.085},"min":{"1min":0.67400000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.8180000000000001},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.94099999999999995,"15min":0.998},"min":{"1min":0.66700000000000004,"5min":0.55600000000000005,"15min":0.55400000000000005},"max":{"1min":1.212,"5min":1.51,"15min":2.0449999999999999},"last":0.373},{"interface":"front","average":{"1min":1.091,"5min":1.052,"15min":1.046},"min":{"1min":0.748,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8779999999999999},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:03:51 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.69699999999999995,"5min":0.626,"15min":0.60099999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.131,"15min":1.1759999999999999},"min":{"1min":0.68799999999999994,"5min":0.65800000000000003,"15min":0.46100000000000002},"max":{"1min":1.367,"5min":1.677,"15min":2.1709999999999998},"last":0.57899999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:03:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.165,"15min":1.1990000000000001},"min":{"1min":0.82099999999999995,"5min":0.77300000000000002,"15min":0.64300000000000002},"max":{"1min":1.502,"5min":1.744,"15min":1.8069999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1759999999999999,"15min":1.2190000000000001},"min":{"1min":0.79700000000000004,"5min":0.72499999999999998,"15min":0.65200000000000002},"max":{"1min":1.5940000000000001,"5min":1.6499999999999999,"15min":2.1139999999999999},"last":0.503}]},{"osd":8,"last update":"Mon Feb 12 16:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.353,"15min":1.4119999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.611,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.802,"5min":1.9490000000000001,"15min":2.0760000000000001},"last":0.60599999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:04:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":1.133,"5min":0.91500000000000004,"15min":0.751},"max":{"1min":1.8400000000000001,"5min":1.9259999999999999,"15min":1.9630000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.423,"15min":1.4239999999999999},"min":{"1min":1.089,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8220000000000001,"5min":1.944,"15min":1.944},"last":0.53000000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:03:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4870000000000001,"15min":1.5289999999999999},"min":{"1min":1.1539999999999999,"5min":0.96499999999999997,"15min":0.78100000000000003},"max":{"1min":1.879,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4830000000000001,"15min":1.5529999999999999},"min":{"1min":1.1619999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9099999999999999,"5min":1.986,"15min":2.1589999999999998},"last":0.55500000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.5069999999999999,"15min":1.5669999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.8979999999999999,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.506,"15min":1.516},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.77,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.72599999999999998}]}]},{"osd":6,"up_from":48,"seq":206158430327,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534904,"kb_used_data":225520,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93210248,"statfs":{"total":95995035648,"available":95447293952,"internally_reserved":0,"allocated":230932480,"data_stored":230520400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.0269999999999999,"15min":0.97099999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.806,"5min":3.806,"15min":3.806},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.085,"5min":1.0349999999999999,"15min":0.94699999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.29399999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.028,"15min":0.92800000000000005},"min":{"1min":0.84999999999999998,"5min":0.69199999999999995,"15min":0.64300000000000002},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0489999999999999,"15min":1.004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.309}]},{"osd":2,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.077,"15min":0.98799999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.55500000000000005},"max":{"1min":1.6639999999999999,"5min":2.383,"15min":2.383},"last":0.434},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1279999999999999,"15min":1.0269999999999999},"min":{"1min":0.68200000000000005,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.611,"5min":1.621,"15min":1.621},"last":0.371}]},{"osd":3,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0569999999999999,"15min":0.97599999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.28,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.0980000000000001,"15min":1.0580000000000001},"min":{"1min":0.80300000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.623},"last":0.499}]},{"osd":4,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0840000000000001,"15min":0.94599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.5229999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.276,"5min":1.1599999999999999,"15min":1.006},"min":{"1min":0.90500000000000003,"5min":0.64800000000000002,"15min":0.42299999999999999},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.238,"15min":1.05},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.506,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.24,"15min":1.075},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.752,"5min":1.7709999999999999,"15min":1.7949999999999999},"last":0.39700000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:03:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3280000000000001,"15min":1.2010000000000001},"min":{"1min":0.746,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.7669999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3260000000000001,"15min":1.2749999999999999},"min":{"1min":0.877,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.472,"5min":1.889,"15min":1.889},"last":0.47199999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.53,"15min":1.4099999999999999},"min":{"1min":1.1910000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"max":{"1min":2.0529999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.526,"15min":1.4330000000000001},"min":{"1min":1.1799999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":1.7390000000000001,"5min":2.0739999999999998,"15min":2.093},"last":0.92100000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.6299999999999999,"15min":1.5009999999999999},"min":{"1min":1.151,"5min":1.151,"15min":1.0589999999999999},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.546,"15min":1.4319999999999999},"min":{"1min":1.1499999999999999,"5min":1.1379999999999999,"15min":0.95899999999999996},"max":{"1min":1.788,"5min":2.125,"15min":2.2469999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:03:56 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.6639999999999999,"15min":1.6299999999999999},"min":{"1min":1.2729999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.591,"5min":1.597,"15min":1.571},"min":{"1min":1.25,"5min":1.081,"15min":1.081},"max":{"1min":1.9299999999999999,"5min":2.008,"15min":2.367},"last":0.52800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124460,"kb_used_data":814948,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92620692,"statfs":{"total":95995035648,"available":94843588608,"internally_reserved":0,"allocated":834506752,"data_stored":834100759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.871,"15min":0.71499999999999997},"min":{"1min":0.69199999999999995,"5min":0.41099999999999998,"15min":0.377},"max":{"1min":1.329,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":8.3450000000000006},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.93600000000000005,"15min":0.78300000000000003},"min":{"1min":0.72999999999999998,"5min":0.40500000000000003,"15min":0.28899999999999998},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":2.1949999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:03:42 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.96599999999999997,"15min":0.78600000000000003},"min":{"1min":0.77600000000000002,"5min":0.50700000000000001,"15min":0.379},"max":{"1min":1.2190000000000001,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.93999999999999995,"15min":0.81000000000000005},"min":{"1min":0.71699999999999997,"5min":0.45800000000000002,"15min":0.34999999999999998},"max":{"1min":1.1899999999999999,"5min":1.54,"15min":1.54},"last":1.3089999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.98499999999999999,"15min":0.871},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.236,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.345},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.071,"15min":0.93799999999999994},"min":{"1min":0.59499999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.3009999999999999,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":1.3859999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.179,"15min":1.0620000000000001},"min":{"1min":0.95099999999999996,"5min":0.71999999999999997,"15min":0.70299999999999996},"max":{"1min":1.399,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.262,"15min":1.1399999999999999},"min":{"1min":0.79000000000000004,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.802,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.47699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:03:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.341,"15min":1.169},"min":{"1min":1.0740000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.7889999999999999,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.411,"5min":1.421,"15min":1.2549999999999999},"min":{"1min":0.94799999999999995,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.7749999999999999,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.379,"15min":1.252},"min":{"1min":0.83499999999999996,"5min":0.81399999999999995,"15min":0.80500000000000005},"max":{"1min":1.8720000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.363,"5min":1.4570000000000001,"15min":1.3460000000000001},"min":{"1min":0.80800000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.768,"5min":2.573,"15min":2.573},"last":1.417}]},{"osd":7,"last update":"Mon Feb 12 16:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.4890000000000001,"15min":1.4019999999999999},"min":{"1min":0.874,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.74,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.373,"15min":1.3440000000000001},"min":{"1min":0.96799999999999997,"5min":0.871,"15min":0.871},"max":{"1min":1.679,"5min":1.887,"15min":1.887},"last":1.3759999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:03:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.504,"15min":1.415},"min":{"1min":1.274,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.9119999999999999,"5min":2.355,"15min":2.355},"last":0.877},{"interface":"front","average":{"1min":1.696,"5min":1.641,"15min":1.524},"min":{"1min":1.4330000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.016,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.60899999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6279999999999999,"5min":1.6910000000000001,"15min":1.643},"min":{"1min":1.125,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":1.9670000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.534,"15min":1.532},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":2.113,"5min":2.133,"15min":2.133},"last":1.468}]},{"osd":10,"last update":"Mon Feb 12 16:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.526,"15min":1.5640000000000001},"min":{"1min":1.002,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.6779999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.6819999999999999,"15min":1.7430000000000001},"min":{"1min":1.036,"5min":0.87,"15min":0.87},"max":{"1min":1.982,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.68799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738536,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004164,"kb_used_data":698748,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92740988,"statfs":{"total":95995035648,"available":94966771712,"internally_reserved":0,"allocated":715517952,"data_stored":715099785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:03:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.81000000000000005,"15min":0.69799999999999995},"min":{"1min":0.68100000000000005,"5min":0.44900000000000001,"15min":0.32500000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.3009999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.84099999999999997,"15min":0.73299999999999998},"min":{"1min":0.71499999999999997,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.234,"5min":1.242,"15min":1.2589999999999999},"last":3.1459999999999999}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.79200000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.2689999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85799999999999998,"15min":0.871},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.012,"5min":1.337,"15min":1.3839999999999999},"last":1.8500000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:03:59 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.137,"15min":1.0900000000000001},"min":{"1min":0.84199999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.5680000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.036,"5min":1.024,"15min":1.01},"min":{"1min":0.81399999999999995,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.3129999999999999,"5min":1.736,"15min":1.736},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1060000000000001,"15min":1.081},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.609,"5min":1.609,"15min":2.0009999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.139,"15min":1.1850000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.171,"15min":1.1379999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.228,"15min":1.2170000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:03:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.353},"min":{"1min":0.995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.734,"5min":1.9059999999999999,"15min":2.0449999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.488,"15min":1.3500000000000001},"min":{"1min":1.1279999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.214},"last":0.67700000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.466,"15min":1.411},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.4399999999999999,"15min":1.4850000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":4.4539999999999997,"5min":4.4539999999999997,"15min":4.4539999999999997},"last":0.60299999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:04:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5429999999999999,"15min":1.4910000000000001},"min":{"1min":0.92600000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.3559999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4550000000000001,"15min":1.5600000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.873,"5min":2.0779999999999998,"15min":2.4380000000000002},"last":0.54600000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:03:59 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.474,"15min":1.5980000000000001},"min":{"1min":1.109,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.931,"5min":1.931,"15min":2.335},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.383,"15min":1.415},"min":{"1min":1.1619999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":1.714,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.438}]},{"osd":11,"last update":"Mon Feb 12 16:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.577,"15min":1.5389999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.492,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.286,"5min":1.4019999999999999,"15min":1.381},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.95,"5min":2.198,"15min":2.198},"last":0.46700000000000003}]}]},{"osd":2,"up_from":21,"seq":90194313367,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055556,"kb_used_data":746044,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92689596,"statfs":{"total":95995035648,"available":94914146304,"internally_reserved":0,"allocated":763949056,"data_stored":763530889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86499999999999999,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4930000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":2.6579999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.871,"15min":0.752},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.133,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.45000000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.97399999999999998,"15min":0.81999999999999995},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.6100000000000001},"last":1.794},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.075,"15min":0.91500000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42799999999999999},"max":{"1min":1.8009999999999999,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.80900000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.048,"15min":0.90900000000000003},"min":{"1min":0.52200000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.312},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.98799999999999999,"15min":0.90700000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.3460000000000001,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.758}]},{"osd":4,"last update":"Mon Feb 12 16:04:00 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.1919999999999999,"15min":1.0800000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.597,"5min":1.74,"15min":1.7669999999999999},"last":0.626},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.97699999999999998,"15min":0.91900000000000004},"min":{"1min":0.77800000000000002,"5min":0.60999999999999999,"15min":0.56000000000000005},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.95199999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:03:50 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.177,"15min":1.149},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.355,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.997},{"interface":"front","average":{"1min":1.351,"5min":1.1870000000000001,"15min":1.0820000000000001},"min":{"1min":0.73299999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.77700000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.145,"15min":1.097},"min":{"1min":0.89500000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.663,"5min":1.746,"15min":1.746},"last":1.036},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3100000000000001,"15min":1.222},"min":{"1min":0.995,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7669999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.7110000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:04:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3839999999999999,"15min":1.347},"min":{"1min":0.90900000000000003,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.1640000000000001,"5min":2.1640000000000001,"15min":2.1640000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3100000000000001,"15min":1.2729999999999999},"min":{"1min":1.038,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.0720000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.4450000000000001,"15min":1.425},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.72,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.446,"15min":1.4710000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.9430000000000001,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.70999999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3979999999999999,"15min":1.425},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.508,"15min":1.4850000000000001},"min":{"1min":1.04,"5min":1.024,"15min":1.024},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":1.7709999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:03:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4590000000000001,"15min":1.4690000000000001},"min":{"1min":0.96499999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.546,"5min":1.53,"15min":1.548},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.7410000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1003972,"kb_used_data":698556,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92741180,"statfs":{"total":95995035648,"available":94966968320,"internally_reserved":0,"allocated":715321344,"data_stored":714903177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84399999999999997,"15min":0.82599999999999996},"min":{"1min":0.627,"5min":0.434,"15min":0.42999999999999999},"max":{"1min":1.099,"5min":1.2589999999999999,"15min":1.3260000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.96999999999999997,"15min":0.84999999999999998},"min":{"1min":0.65500000000000003,"5min":0.56599999999999995,"15min":0.371},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.194}]},{"osd":2,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93799999999999994,"15min":0.86899999999999999},"min":{"1min":0.69999999999999996,"5min":0.60399999999999998,"15min":0.51900000000000002},"max":{"1min":1.3660000000000001,"5min":1.466,"15min":1.5529999999999999},"last":1.115},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1000000000000001,"15min":0.94399999999999995},"min":{"1min":0.82899999999999996,"5min":0.67100000000000004,"15min":0.44800000000000001},"max":{"1min":1.4410000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.45700000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:03:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.96099999999999997,"15min":0.97499999999999998},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.1899999999999999,"5min":1.278,"15min":1.593},"last":1.581},{"interface":"front","average":{"1min":1.127,"5min":1.1399999999999999,"15min":1.0589999999999999},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.60699999999999998},"max":{"1min":1.343,"5min":1.669,"15min":1.669},"last":0.79200000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:03:38 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.238,"15min":1.1639999999999999},"min":{"1min":0.998,"5min":0.89600000000000002,"15min":0.58299999999999996},"max":{"1min":1.74,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.038,"15min":1.0940000000000001},"min":{"1min":0.73299999999999998,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.548,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.6339999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.226,"15min":1.1659999999999999},"min":{"1min":0.98499999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.573,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.327,"15min":1.2709999999999999},"min":{"1min":1.145,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.696,"5min":1.7689999999999999,"15min":1.8120000000000001},"last":0.60299999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:03:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3140000000000001,"15min":1.3180000000000001},"min":{"1min":0.93200000000000005,"5min":0.748,"15min":0.66600000000000004},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.387,"15min":1.3500000000000001},"min":{"1min":1.089,"5min":0.85299999999999998,"15min":0.77800000000000002},"max":{"1min":1.7050000000000001,"5min":1.768,"15min":2.0059999999999998},"last":0.51600000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.3180000000000001,"15min":1.3759999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":3.2719999999999998,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":2.1160000000000001},{"interface":"front","average":{"1min":1.47,"5min":1.4810000000000001,"15min":1.484},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.0899999999999999,"5min":2.242,"15min":2.242},"last":0.81799999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.5700000000000001,"15min":1.5549999999999999},"min":{"1min":1.1519999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.032,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.601,"5min":1.528,"15min":1.5549999999999999},"min":{"1min":1.2330000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.917,"5min":2.117,"15min":2.117},"last":0.64700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:03:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.448,"15min":1.48},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.081,"5min":2.113,"15min":2.113},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.5600000000000001,"15min":1.5800000000000001},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":0.84799999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:03:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.496,"15min":1.4970000000000001},"min":{"1min":1.073,"5min":1.073,"15min":1.073},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":1.663},{"interface":"front","average":{"1min":1.591,"5min":1.593,"15min":1.5409999999999999},"min":{"1min":1.2310000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.0550000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":0.876}]}]},{"osd":4,"up_from":32,"seq":137438953605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541060,"kb_used_data":231676,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93204092,"statfs":{"total":95995035648,"available":95440990208,"internally_reserved":0,"allocated":237236224,"data_stored":236818057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0529999999999999,"15min":0.95899999999999996},"min":{"1min":0.76000000000000001,"5min":0.64400000000000002,"15min":0.56999999999999995},"max":{"1min":1.377,"5min":1.5209999999999999,"15min":1.736},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.0620000000000001,"15min":0.94899999999999995},"min":{"1min":0.71999999999999997,"5min":0.69399999999999995,"15min":0.58699999999999997},"max":{"1min":1.484,"5min":1.617,"15min":1.617},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0620000000000001,"15min":0.97499999999999998},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.63900000000000001},"max":{"1min":1.4870000000000001,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.1060000000000001,"15min":0.98199999999999998},"min":{"1min":0.75600000000000001,"5min":0.68799999999999994,"15min":0.51400000000000001},"max":{"1min":1.5049999999999999,"5min":1.7490000000000001,"15min":1.806},"last":0.497}]},{"osd":2,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0589999999999999,"15min":0.94499999999999995},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.60299999999999998},"max":{"1min":1.351,"5min":1.593,"15min":1.593},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0109999999999999,"15min":0.96299999999999997},"min":{"1min":0.78000000000000003,"5min":0.68000000000000005,"15min":0.64900000000000002},"max":{"1min":1.3220000000000001,"5min":1.4450000000000001,"15min":1.5549999999999999},"last":0.31}]},{"osd":3,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1160000000000001,"15min":1.0049999999999999},"min":{"1min":0.79800000000000004,"5min":0.72599999999999998,"15min":0.57599999999999996},"max":{"1min":1.5129999999999999,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.069,"5min":1.103,"15min":0.98799999999999999},"min":{"1min":0.86199999999999999,"5min":0.73299999999999998,"15min":0.55100000000000005},"max":{"1min":1.26,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.41099999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.1100000000000001,"15min":0.94199999999999995},"min":{"1min":0.69799999999999995,"5min":0.625,"15min":0.55200000000000005},"max":{"1min":1.5529999999999999,"5min":1.889,"15min":1.889},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.171,"15min":1.0089999999999999},"min":{"1min":0.747,"5min":0.67200000000000004,"15min":0.57799999999999996},"max":{"1min":1.474,"5min":1.552,"15min":1.581},"last":2.7149999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.1919999999999999,"15min":1.0720000000000001},"min":{"1min":0.76000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6020000000000001,"5min":1.653,"15min":1.653},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.101,"5min":1.161,"15min":1.1399999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.4019999999999999,"5min":1.831,"15min":1.9330000000000001},"last":0.48399999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:03:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1699999999999999,"15min":1.1399999999999999},"min":{"1min":0.89400000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.411,"5min":1.577,"15min":1.718},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3959999999999999,"15min":1.3480000000000001},"min":{"1min":1.123,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.75,"5min":1.835,"15min":2.302},"last":0.73999999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:03:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5349999999999999,"15min":1.514},"min":{"1min":1.216,"5min":1.073,"15min":0.88400000000000001},"max":{"1min":1.954,"5min":2.0699999999999998,"15min":2.1949999999999998},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.417,"5min":1.452,"15min":1.4239999999999999},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.8,"5min":1.931,"15min":2.1699999999999999},"last":1.175}]},{"osd":9,"last update":"Mon Feb 12 16:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.617,"15min":1.571},"min":{"1min":1.2869999999999999,"5min":1.1739999999999999,"15min":1.1659999999999999},"max":{"1min":2.0299999999999998,"5min":2.1309999999999998,"15min":2.173},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.516,"5min":1.5489999999999999,"15min":1.486},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":1.9670000000000001,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.70999999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:03:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.554,"15min":1.5189999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.8939999999999999,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.474,"5min":1.5189999999999999,"15min":1.554},"min":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"max":{"1min":1.796,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":0.65900000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721016,"kb_used_data":415600,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93024136,"statfs":{"total":95995035648,"available":95256715264,"internally_reserved":0,"allocated":425574400,"data_stored":425162320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:03:39 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.101,"15min":0.93100000000000005},"min":{"1min":0.77100000000000002,"5min":0.65100000000000002,"15min":0.57699999999999996},"max":{"1min":1.448,"5min":1.71,"15min":1.71},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0229999999999999,"15min":0.94199999999999995},"min":{"1min":0.85999999999999999,"5min":0.66300000000000003,"15min":0.46999999999999997},"max":{"1min":1.3160000000000001,"5min":1.524,"15min":1.524},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:03:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1140000000000001,"15min":0.97899999999999998},"min":{"1min":0.84299999999999997,"5min":0.68400000000000005,"15min":0.59999999999999998},"max":{"1min":1.421,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0329999999999999,"15min":0.93799999999999994},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.59099999999999997},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Feb 12 16:03:39 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.135,"15min":1.0089999999999999},"min":{"1min":0.67100000000000004,"5min":0.622,"15min":0.57999999999999996},"max":{"1min":1.498,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.1020000000000001,"15min":1.04},"min":{"1min":0.79500000000000004,"5min":0.65200000000000002,"15min":0.58899999999999997},"max":{"1min":1.3340000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:03:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.131,"15min":1.03},"min":{"1min":0.86899999999999999,"5min":0.751,"15min":0.70299999999999996},"max":{"1min":1.5569999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.234,"5min":1.1990000000000001,"15min":1.105},"min":{"1min":0.83799999999999997,"5min":0.753,"15min":0.66400000000000003},"max":{"1min":1.575,"5min":1.726,"15min":1.726},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:03:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.002,"15min":0.92900000000000005},"min":{"1min":0.68700000000000006,"5min":0.59099999999999997,"15min":0.56499999999999995},"max":{"1min":1.335,"5min":1.889,"15min":1.889},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.175,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71499999999999997,"15min":0.52700000000000002},"max":{"1min":1.458,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.212,"15min":1.1120000000000001},"min":{"1min":0.66600000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7010000000000001,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.2470000000000001,"15min":1.167},"min":{"1min":0.89400000000000002,"5min":0.68300000000000005,"15min":0.40400000000000003},"max":{"1min":1.7849999999999999,"5min":1.8360000000000001,"15min":1.903},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:03:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2929999999999999,"15min":1.1830000000000001},"min":{"1min":0.88700000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.7350000000000001,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.552,"5min":1.4550000000000001,"15min":1.2749999999999999},"min":{"1min":1.137,"5min":0.69099999999999995,"15min":0.435},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":0.60999999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:03:39 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5189999999999999,"15min":1.4350000000000001},"min":{"1min":1.052,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.1509999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.669,"5min":1.506,"15min":1.3580000000000001},"min":{"1min":1.286,"5min":1.0860000000000001,"15min":0.83999999999999997},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.66200000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.5580000000000001,"15min":1.5009999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.036,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.6719999999999999,"15min":1.6060000000000001},"min":{"1min":1.3600000000000001,"5min":1.1850000000000001,"15min":1.1619999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":0.63600000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:03:54 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.5,"15min":1.4970000000000001},"min":{"1min":1.1719999999999999,"5min":1.145,"15min":1.145},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.641,"5min":1.6180000000000001,"15min":1.6319999999999999},"min":{"1min":1.329,"5min":1.258,"15min":1.222},"max":{"1min":2.2160000000000002,"5min":2.254,"15min":2.254},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":742064128,"data_stored":742064128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":835256320,"data_stored":835256320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":786890752,"data_stored":786890752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":735051776,"data_stored":735051776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":239271936,"data_stored":239271936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":426704896,"data_stored":426704896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":237043712,"data_stored":237043712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":702939136,"data_stored":702939136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":928645120,"data_stored":928645120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":634257408,"data_stored":634257408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":468779008,"data_stored":468779008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":741146624,"data_stored":741146624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:04:37.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set device_health_metrics pgp_num 1 2024-02-12T16:04:37.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:37.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:37.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:37.931 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:37.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:38.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:38.123 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:38.173 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:38.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:38.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:38.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:38.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:38.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:38.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:38.544 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:38 smithi169 bash[19604]: cluster 2024-02-12T16:04:36.689422+0000 osd.5 (osd.5) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:38.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:38.563 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:38 smithi123 bash[19119]: cluster 2024-02-12T16:04:36.689422+0000 osd.5 (osd.5) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:38.563 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:38 smithi123 bash[19119]: cluster 2024-02-12T16:04:36.689461+0000 osd.5 (osd.5) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:38.563 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:38 smithi123 bash[19119]: cluster 2024-02-12T16:04:37.232658+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e160: 12 total, 12 up, 12 in 2024-02-12T16:04:38.563 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:38 smithi123 bash[19119]: cluster 2024-02-12T16:04:37.546263+0000 mgr.y (mgr.14146) 572 : cluster [DBG] pgmap v640: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.53k op/s 2024-02-12T16:04:38.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:38.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:04:38.641937 min lat: 0.000954878 max lat: 0.270964 avg lat: 0.00459918 2024-02-12T16:04:38.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:04:38.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 16 69577 69561 217.357 151.562 0.00465208 0.00459918 2024-02-12T16:04:38.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:38 smithi121 bash[18489]: cluster 2024-02-12T16:04:36.689422+0000 osd.5 (osd.5) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:38.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:38 smithi121 bash[18489]: cluster 2024-02-12T16:04:36.689461+0000 osd.5 (osd.5) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:38.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:38 smithi121 bash[18489]: cluster 2024-02-12T16:04:37.232658+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e160: 12 total, 12 up, 12 in 2024-02-12T16:04:38.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:38 smithi121 bash[18489]: cluster 2024-02-12T16:04:37.546263+0000 mgr.y (mgr.14146) 572 : cluster [DBG] pgmap v640: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.53k op/s 2024-02-12T16:04:38.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:38.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:38 smithi169 bash[19604]: cluster 2024-02-12T16:04:36.689461+0000 osd.5 (osd.5) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:38 smithi169 bash[19604]: cluster 2024-02-12T16:04:37.232658+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e160: 12 total, 12 up, 12 in 2024-02-12T16:04:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:38 smithi169 bash[19604]: cluster 2024-02-12T16:04:37.546263+0000 mgr.y (mgr.14146) 572 : cluster [DBG] pgmap v640: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 158 MiB/s wr, 2.53k op/s 2024-02-12T16:04:38.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:38.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:38.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:04:38.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:38.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:38.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:39.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:39.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:39.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:39.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:39.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:39.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:39.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:39.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:39.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:39.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:39.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:39.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:39.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:39.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:39.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:39.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 71887 71871 213.881 144.375 0.00619698 0.00467362 2024-02-12T16:04:39.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:04:39.719 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:39.752 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:39.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:39.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:39.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:39.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:39.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:40.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:40.138 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:40.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:40.267 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:40.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:40.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:40.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:40.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:40 smithi123 bash[19119]: cluster 2024-02-12T16:04:39.546724+0000 mgr.y (mgr.14146) 573 : cluster [DBG] pgmap v641: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.64k op/s 2024-02-12T16:04:40.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 73806 73790 209.605 119.938 0.00879517 0.00476887 2024-02-12T16:04:40.657 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:40.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:40.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:40.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:40.902 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:40 smithi169 bash[19604]: cluster 2024-02-12T16:04:39.546724+0000 mgr.y (mgr.14146) 573 : cluster [DBG] pgmap v641: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.64k op/s 2024-02-12T16:04:40.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:40 smithi121 bash[18489]: cluster 2024-02-12T16:04:39.546724+0000 mgr.y (mgr.14146) 573 : cluster [DBG] pgmap v641: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.64k op/s 2024-02-12T16:04:40.920 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:40.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:40.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:41.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:41.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:41.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:41.182 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:41.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:41.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:41.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:41.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:41.453 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:41.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:41.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:41.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:41.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:41.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 76123 76107 206.788 144.812 0.00908202 0.00483434 2024-02-12T16:04:41.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:41.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:41.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:41.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:41.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:41.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:41.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:42.123 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:04:42.123 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:04:42.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:42.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:42.249 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:42.257 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:42.279 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:42.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:42.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:42.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:42.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:42.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 78196 78180 203.57 129.562 0.00732985 0.00491057 2024-02-12T16:04:42.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:42.675 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:42.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:42 smithi169 bash[19604]: cluster 2024-02-12T16:04:41.547123+0000 mgr.y (mgr.14146) 574 : cluster [DBG] pgmap v642: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.44k op/s 2024-02-12T16:04:42.800 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:42.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:42 smithi123 bash[19119]: cluster 2024-02-12T16:04:41.547123+0000 mgr.y (mgr.14146) 574 : cluster [DBG] pgmap v642: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.44k op/s 2024-02-12T16:04:42.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:42.929 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:42 smithi121 bash[18489]: cluster 2024-02-12T16:04:41.547123+0000 mgr.y (mgr.14146) 574 : cluster [DBG] pgmap v642: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.44k op/s 2024-02-12T16:04:42.932 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:42.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:43.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:43.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:43.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:43.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:43.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:43.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.334 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:43.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:43.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.462 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:43.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:43.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:04:43.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:43.621 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:43.622 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:43.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:43.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 80408 80392 200.956 138.25 0.0143914 0.00497377 2024-02-12T16:04:43.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.773 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:43.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:43.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:43.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:43.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:44.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:44.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:44.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:44.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:44.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:44.302 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:44.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:44.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:44.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:44.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:44.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 16 82542 82526 198.357 133.375 0.00261934 0.00503958 2024-02-12T16:04:44.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:44.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:44 smithi169 bash[19604]: cluster 2024-02-12T16:04:43.547592+0000 mgr.y (mgr.14146) 575 : cluster [DBG] pgmap v643: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.45k op/s 2024-02-12T16:04:44.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:44.842 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:44 smithi123 bash[19119]: cluster 2024-02-12T16:04:43.547592+0000 mgr.y (mgr.14146) 575 : cluster [DBG] pgmap v643: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.45k op/s 2024-02-12T16:04:44.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:44 smithi121 bash[18489]: cluster 2024-02-12T16:04:43.547592+0000 mgr.y (mgr.14146) 575 : cluster [DBG] pgmap v643: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.45k op/s 2024-02-12T16:04:44.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:44.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:44.984 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:44.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:45.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:45.115 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:45.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:45.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:45.247 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:45.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:45.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:45.375 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:45.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:45.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:45.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:45.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:45.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:45.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 84388 84372 195.283 115.375 0.0092947 0.00511872 2024-02-12T16:04:45.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:45.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:45.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:45.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:45.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:45.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:45.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:45 smithi123 bash[19119]: cluster 2024-02-12T16:04:44.412670+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:45.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:45 smithi123 bash[19119]: cluster 2024-02-12T16:04:44.412725+0000 osd.6 (osd.6) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:45.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:45 smithi123 bash[19119]: cluster 2024-02-12T16:04:44.622962+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e161: 12 total, 12 up, 12 in 2024-02-12T16:04:45.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:45 smithi123 bash[19119]: cluster 2024-02-12T16:04:45.218590+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e162: 12 total, 12 up, 12 in 2024-02-12T16:04:45.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:45.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:45.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:45 smithi121 bash[18489]: cluster 2024-02-12T16:04:44.412670+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:45.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:45 smithi121 bash[18489]: cluster 2024-02-12T16:04:44.412725+0000 osd.6 (osd.6) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:45.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:45 smithi121 bash[18489]: cluster 2024-02-12T16:04:44.622962+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e161: 12 total, 12 up, 12 in 2024-02-12T16:04:45.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:45 smithi121 bash[18489]: cluster 2024-02-12T16:04:45.218590+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e162: 12 total, 12 up, 12 in 2024-02-12T16:04:46.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:46.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:45 smithi169 bash[19604]: cluster 2024-02-12T16:04:44.412670+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:45 smithi169 bash[19604]: cluster 2024-02-12T16:04:44.412725+0000 osd.6 (osd.6) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:45 smithi169 bash[19604]: cluster 2024-02-12T16:04:44.622962+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e161: 12 total, 12 up, 12 in 2024-02-12T16:04:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:45 smithi169 bash[19604]: cluster 2024-02-12T16:04:45.218590+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e162: 12 total, 12 up, 12 in 2024-02-12T16:04:46.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:46.170 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:46.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:46.307 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:46.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:46.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:46.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:46.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:46.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 86463 86447 192.94 129.688 0.00287545 0.00517436 2024-02-12T16:04:46.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:46.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: cluster 2024-02-12T16:04:45.547969+0000 mgr.y (mgr.14146) 576 : cluster [DBG] pgmap v646: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 MiB/s wr, 2.15k op/s 2024-02-12T16:04:46.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: audit 2024-02-12T16:04:46.254466+0000 mon.a (mon.0) 869 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:04:46.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: audit 2024-02-12T16:04:46.254666+0000 mon.a (mon.0) 870 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:04:46.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: audit 2024-02-12T16:04:46.255328+0000 mon.a (mon.0) 871 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:04:46.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: audit 2024-02-12T16:04:46.276662+0000 mon.a (mon.0) 872 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:04:46.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: audit 2024-02-12T16:04:46.278747+0000 mon.a (mon.0) 873 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:04:46.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: audit 2024-02-12T16:04:46.279660+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:04:46.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:46 smithi121 bash[18489]: audit 2024-02-12T16:04:46.287196+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:04:46.700 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:46.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:46.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: cluster 2024-02-12T16:04:45.547969+0000 mgr.y (mgr.14146) 576 : cluster [DBG] pgmap v646: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 MiB/s wr, 2.15k op/s 2024-02-12T16:04:46.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: audit 2024-02-12T16:04:46.254466+0000 mon.a (mon.0) 869 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:04:46.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: audit 2024-02-12T16:04:46.254666+0000 mon.a (mon.0) 870 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:04:46.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: audit 2024-02-12T16:04:46.255328+0000 mon.a (mon.0) 871 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:04:46.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: audit 2024-02-12T16:04:46.276662+0000 mon.a (mon.0) 872 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:04:46.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: audit 2024-02-12T16:04:46.278747+0000 mon.a (mon.0) 873 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:04:46.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: audit 2024-02-12T16:04:46.279660+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:04:46.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:46 smithi123 bash[19119]: audit 2024-02-12T16:04:46.287196+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:04:46.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:46.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: cluster 2024-02-12T16:04:45.547969+0000 mgr.y (mgr.14146) 576 : cluster [DBG] pgmap v646: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 MiB/s wr, 2.15k op/s 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: audit 2024-02-12T16:04:46.254466+0000 mon.a (mon.0) 869 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: audit 2024-02-12T16:04:46.254666+0000 mon.a (mon.0) 870 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: audit 2024-02-12T16:04:46.255328+0000 mon.a (mon.0) 871 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: audit 2024-02-12T16:04:46.276662+0000 mon.a (mon.0) 872 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: audit 2024-02-12T16:04:46.278747+0000 mon.a (mon.0) 873 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: audit 2024-02-12T16:04:46.279660+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:04:46.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:46 smithi169 bash[19604]: audit 2024-02-12T16:04:46.287196+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:04:46.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:46.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:47.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:47.089 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:47.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:47.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:47.222 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:47.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:47.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:47.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:47.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:47.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:47.478 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:47.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:47.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:47.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:47.613 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:47.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:47.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 88425 88409 190.515 122.625 0.00288677 0.00524516 2024-02-12T16:04:47.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:47.745 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:47.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:47.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:47.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:47.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:47.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:48.005 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:48.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:48.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:48.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:48.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:48.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:48.409 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:48.511 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:48.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:48.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 89737 89721 186.898 82 0.00357304 0.00531252 2024-02-12T16:04:48.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:48.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:48.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:48.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:48.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:48 smithi123 bash[19119]: cluster 2024-02-12T16:04:47.548395+0000 mgr.y (mgr.14146) 577 : cluster [DBG] pgmap v647: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.25k op/s 2024-02-12T16:04:48.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:48 smithi121 bash[18489]: cluster 2024-02-12T16:04:47.548395+0000 mgr.y (mgr.14146) 577 : cluster [DBG] pgmap v647: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.25k op/s 2024-02-12T16:04:48.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:48.956 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:48.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:48.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:49.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:49.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:49.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:49.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:49.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:49.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:49.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:49.371 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:49.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:49.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:49.499 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:49.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:49 smithi169 bash[19604]: cluster 2024-02-12T16:04:47.548395+0000 mgr.y (mgr.14146) 577 : cluster [DBG] pgmap v647: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.25k op/s 2024-02-12T16:04:49.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:49.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:49.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:49.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 90734 90718 182.879 62.3125 0.00838716 0.00546653 2024-02-12T16:04:49.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:49.762 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:49.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:49.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:49.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:49.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:49.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:50.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:50.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:50.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 1 pgp_num to 1 2024-02-12T16:04:50.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:50.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:50.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:50.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:50.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:50.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:50.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 92624 92608 180.855 118.125 0.0085104 0.00552736 2024-02-12T16:04:50.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:50.737 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:04:50.764 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:04:50.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:50.801 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:04:50.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:50.831 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:04:50.860 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:04:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:50 smithi123 bash[19119]: cluster 2024-02-12T16:04:49.548781+0000 mgr.y (mgr.14146) 578 : cluster [DBG] pgmap v648: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-12T16:04:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:50 smithi123 bash[19119]: audit 2024-02-12T16:04:50.093623+0000 mon.a (mon.0) 876 : audit [INF] from='client.24745 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-12T16:04:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:50 smithi123 bash[19119]: audit 2024-02-12T16:04:50.097763+0000 mon.c (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.121:0/761773756' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-12T16:04:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:50 smithi123 bash[19119]: audit 2024-02-12T16:04:50.217859+0000 mon.a (mon.0) 877 : audit [INF] from='client.24745 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-12T16:04:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:50 smithi123 bash[19119]: cluster 2024-02-12T16:04:50.217943+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e163: 12 total, 12 up, 12 in 2024-02-12T16:04:50.895 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:04:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:50 smithi121 bash[18489]: cluster 2024-02-12T16:04:49.548781+0000 mgr.y (mgr.14146) 578 : cluster [DBG] pgmap v648: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-12T16:04:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:50 smithi121 bash[18489]: audit 2024-02-12T16:04:50.093623+0000 mon.a (mon.0) 876 : audit [INF] from='client.24745 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-12T16:04:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:50 smithi121 bash[18489]: audit 2024-02-12T16:04:50.097763+0000 mon.c (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.121:0/761773756' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-12T16:04:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:50 smithi121 bash[18489]: audit 2024-02-12T16:04:50.217859+0000 mon.a (mon.0) 877 : audit [INF] from='client.24745 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-12T16:04:50.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:50 smithi121 bash[18489]: cluster 2024-02-12T16:04:50.217943+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e163: 12 total, 12 up, 12 in 2024-02-12T16:04:50.923 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:04:50.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:50.954 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:04:50.959 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:50.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:50.986 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:04:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:04:51.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:04:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:50 smithi169 bash[19604]: cluster 2024-02-12T16:04:49.548781+0000 mgr.y (mgr.14146) 578 : cluster [DBG] pgmap v648: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-12T16:04:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:50 smithi169 bash[19604]: audit 2024-02-12T16:04:50.093623+0000 mon.a (mon.0) 876 : audit [INF] from='client.24745 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-12T16:04:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:50 smithi169 bash[19604]: audit 2024-02-12T16:04:50.097763+0000 mon.c (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.121:0/761773756' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-12T16:04:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:50 smithi169 bash[19604]: audit 2024-02-12T16:04:50.217859+0000 mon.a (mon.0) 877 : audit [INF] from='client.24745 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-12T16:04:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:50 smithi169 bash[19604]: cluster 2024-02-12T16:04:50.217943+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e163: 12 total, 12 up, 12 in 2024-02-12T16:04:51.053 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.056 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.057 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.057 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:04:51.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:51.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:51.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:51.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:51.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:51.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:51.359 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:51.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:51.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:51.492 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:51.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:51.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:51.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:51.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:51.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:51.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 94495 94479 178.918 116.938 0.00602004 0.00558754 2024-02-12T16:04:51.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:51.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:51.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:51 smithi169 bash[19604]: audit 2024-02-12T16:04:50.696933+0000 mon.a (mon.0) 879 : audit [DBG] from='client.? v1:172.21.15.121:0/1147349448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:51.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:51.898 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:51.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:51 smithi121 bash[18489]: audit 2024-02-12T16:04:50.696933+0000 mon.a (mon.0) 879 : audit [DBG] from='client.? v1:172.21.15.121:0/1147349448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:52.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:52.034 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:52.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:51 smithi123 bash[19119]: audit 2024-02-12T16:04:50.696933+0000 mon.a (mon.0) 879 : audit [DBG] from='client.? v1:172.21.15.121:0/1147349448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:04:52.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:52.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:52.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:52.307 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:52.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:52.439 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:52.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:52.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:52.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 96404 96388 177.164 119.312 0.0155428 0.00564318 2024-02-12T16:04:52.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:52 smithi121 bash[18489]: cluster 2024-02-12T16:04:51.549175+0000 mgr.y (mgr.14146) 579 : cluster [DBG] pgmap v650: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-02-12T16:04:52.670 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:52.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:52.718 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:52.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:52.830 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:52 smithi123 bash[19119]: cluster 2024-02-12T16:04:51.549175+0000 mgr.y (mgr.14146) 579 : cluster [DBG] pgmap v650: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-02-12T16:04:52.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:52.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:52.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:04:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:52.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:04:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:52 smithi169 bash[19604]: cluster 2024-02-12T16:04:51.549175+0000 mgr.y (mgr.14146) 579 : cluster [DBG] pgmap v650: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-02-12T16:04:53.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:53.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:53.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:53.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:53.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:53.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:53.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:53.384 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:53.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:53.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:53.520 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:53.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:53.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:53.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 98286 98270 175.463 117.625 0.0058152 0.0056975 2024-02-12T16:04:53.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:53 smithi169 bash[19604]: audit 2024-02-12T16:04:52.877859+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.121:0/230085895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:04:53.655 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:53.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:53.663 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:04:53.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:53.788 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:53.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:53.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:53.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:53 smithi121 bash[18489]: audit 2024-02-12T16:04:52.877859+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.121:0/230085895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:04:53.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:53.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:53.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:54.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:54.029 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:53 smithi123 bash[19119]: audit 2024-02-12T16:04:52.877859+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.121:0/230085895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:04:54.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:54.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:54.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:54.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:54.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:54.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:04:54.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:54.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:54.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:04:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:54.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:04:54.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:54.598 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:54.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:54.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 16 100095 100079 173.729 113.062 0.00740952 0.00575442 2024-02-12T16:04:54.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:54.732 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:54.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:54.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:54.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:54.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:54 smithi169 bash[19604]: cluster 2024-02-12T16:04:53.549520+0000 mgr.y (mgr.14146) 580 : cluster [DBG] pgmap v651: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-02-12T16:04:54.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:54 smithi169 bash[19604]: cluster 2024-02-12T16:04:53.642033+0000 mon.a (mon.0) 881 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:04:54.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:54 smithi169 bash[19604]: cluster 2024-02-12T16:04:53.642068+0000 mon.a (mon.0) 882 : cluster [INF] Cluster is now healthy 2024-02-12T16:04:54.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:54 smithi169 bash[19604]: audit 2024-02-12T16:04:53.651569+0000 mon.a (mon.0) 883 : audit [INF] from='client.? v1:172.21.15.121:0/230085895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:04:54.842 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:54 smithi169 bash[19604]: cluster 2024-02-12T16:04:53.651661+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e164: 12 total, 12 up, 12 in 2024-02-12T16:04:54.865 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:54.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:54.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:54.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:54 smithi121 bash[18489]: cluster 2024-02-12T16:04:53.549520+0000 mgr.y (mgr.14146) 580 : cluster [DBG] pgmap v651: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-02-12T16:04:54.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:54 smithi121 bash[18489]: cluster 2024-02-12T16:04:53.642033+0000 mon.a (mon.0) 881 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:04:54.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:54 smithi121 bash[18489]: cluster 2024-02-12T16:04:53.642068+0000 mon.a (mon.0) 882 : cluster [INF] Cluster is now healthy 2024-02-12T16:04:54.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:54 smithi121 bash[18489]: audit 2024-02-12T16:04:53.651569+0000 mon.a (mon.0) 883 : audit [INF] from='client.? v1:172.21.15.121:0/230085895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:04:54.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:54 smithi121 bash[18489]: cluster 2024-02-12T16:04:53.651661+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e164: 12 total, 12 up, 12 in 2024-02-12T16:04:54.970 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:54.995 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:55.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:55.128 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:55.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:54 smithi123 bash[19119]: cluster 2024-02-12T16:04:53.549520+0000 mgr.y (mgr.14146) 580 : cluster [DBG] pgmap v651: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-02-12T16:04:55.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:54 smithi123 bash[19119]: cluster 2024-02-12T16:04:53.642033+0000 mon.a (mon.0) 881 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:04:55.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:54 smithi123 bash[19119]: cluster 2024-02-12T16:04:53.642068+0000 mon.a (mon.0) 882 : cluster [INF] Cluster is now healthy 2024-02-12T16:04:55.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:54 smithi123 bash[19119]: audit 2024-02-12T16:04:53.651569+0000 mon.a (mon.0) 883 : audit [INF] from='client.? v1:172.21.15.121:0/230085895' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:04:55.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:54 smithi123 bash[19119]: cluster 2024-02-12T16:04:53.651661+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e164: 12 total, 12 up, 12 in 2024-02-12T16:04:55.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:55.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:55.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:55.404 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:55.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:55.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:55.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:55.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 101867 101851 172.027 110.75 0.0070581 0.00581103 2024-02-12T16:04:55.671 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:04:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:04:55.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:55.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:55.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:55.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:55.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:56.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:56.073 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:56.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:56.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.205 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:56.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:56.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:56.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:56.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:56.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:56.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:56 smithi121 bash[18489]: cluster 2024-02-12T16:04:55.221814+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e165: 12 total, 12 up, 12 in 2024-02-12T16:04:56.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:56 smithi121 bash[18489]: cluster 2024-02-12T16:04:55.549879+0000 mgr.y (mgr.14146) 581 : cluster [DBG] pgmap v654: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-02-12T16:04:56.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:56.449 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:56 smithi169 bash[19604]: cluster 2024-02-12T16:04:55.221814+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e165: 12 total, 12 up, 12 in 2024-02-12T16:04:56.449 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:56 smithi169 bash[19604]: cluster 2024-02-12T16:04:55.549879+0000 mgr.y (mgr.14146) 581 : cluster [DBG] pgmap v654: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-02-12T16:04:56.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:56.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:56.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.607 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:04:56.608 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:04:56.608 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-02-12T16:04:56.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:04:56.612 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:56.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:56 smithi123 bash[19119]: cluster 2024-02-12T16:04:55.221814+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e165: 12 total, 12 up, 12 in 2024-02-12T16:04:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:56 smithi123 bash[19119]: cluster 2024-02-12T16:04:55.549879+0000 mgr.y (mgr.14146) 581 : cluster [DBG] pgmap v654: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-02-12T16:04:56.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 16 103622 103606 170.386 109.688 0.00379579 0.00586643 2024-02-12T16:04:56.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:04:56.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.742 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:56.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:56.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:04:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:56.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:04:56.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:57.008 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:57.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:57.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:57.221 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:04:57.221 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:04:57.221 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:04:57.221 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:04:57.221 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:04:57.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:57.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:57.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:57.416 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:57.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:57.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:57.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 105288 105272 168.687 104.125 0.00989542 0.00592661 2024-02-12T16:04:57.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:57.694 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:57.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:57.823 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:57.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:57.949 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:57.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:57.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:04:58.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:58.077 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:58.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:58.214 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:58.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:58.344 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:04:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:04:58.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:58.469 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:58.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:58.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:58.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:04:58.644176 min lat: 0.000954878 max lat: 0.731447 avg lat: 0.00597867 2024-02-12T16:04:58.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:04:58.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 107044 107028 167.214 109.75 0.0106991 0.00597867 2024-02-12T16:04:58.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:58.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:58 smithi121 bash[18489]: cluster 2024-02-12T16:04:57.550287+0000 mgr.y (mgr.14146) 582 : cluster [DBG] pgmap v655: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-12T16:04:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:58.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:58.841 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:58 smithi123 bash[19119]: cluster 2024-02-12T16:04:57.550287+0000 mgr.y (mgr.14146) 582 : cluster [DBG] pgmap v655: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-12T16:04:58.860 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:04:58.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:58.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:58 smithi169 bash[19604]: cluster 2024-02-12T16:04:57.550287+0000 mgr.y (mgr.14146) 582 : cluster [DBG] pgmap v655: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-12T16:04:58.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:04:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:58.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:04:59.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:59.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:04:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:59.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:04:59.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:59.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:04:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:59.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:04:59.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:59.389 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:59.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:59.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:59.495 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:04:59.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:04:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:59.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:04:59.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:59 smithi123 bash[19119]: cluster 2024-02-12T16:04:59.038636+0000 osd.0 (osd.0) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:59.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:04:59 smithi123 bash[19119]: cluster 2024-02-12T16:04:59.038688+0000 osd.0 (osd.0) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:59.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:59 smithi169 bash[19604]: cluster 2024-02-12T16:04:59.038636+0000 osd.0 (osd.0) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:59.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:59.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 16 108797 108781 165.807 109.562 0.00466062 0.0060292 2024-02-12T16:04:59.655 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:04:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:59.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:04:59.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:59.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:04:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:59.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:04:59.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:59.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:59 smithi121 bash[18489]: cluster 2024-02-12T16:04:59.038636+0000 osd.0 (osd.0) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:04:59.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:04:59 smithi121 bash[18489]: cluster 2024-02-12T16:04:59.038688+0000 osd.0 (osd.0) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:04:59.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:04:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:04:59.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:00.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:04:59 smithi169 bash[19604]: cluster 2024-02-12T16:04:59.038688+0000 osd.0 (osd.0) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:00.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:00.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:00.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:00.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:00.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:00.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:00.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:00.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:00.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:00.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:00.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:00.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:00.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:00 smithi169 bash[19604]: cluster 2024-02-12T16:04:59.550729+0000 mgr.y (mgr.14146) 583 : cluster [DBG] pgmap v656: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-12T16:05:00.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:00 smithi169 bash[19604]: cluster 2024-02-12T16:04:59.616360+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e166: 12 total, 12 up, 12 in 2024-02-12T16:05:00.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:00 smithi169 bash[19604]: cluster 2024-02-12T16:05:00.222733+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e167: 12 total, 12 up, 12 in 2024-02-12T16:05:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:00.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:00.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 110462 110446 164.337 104.062 0.00474519 0.00608301 2024-02-12T16:05:00.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:00.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:00.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:00.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:00.868 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:00 smithi121 bash[18489]: cluster 2024-02-12T16:04:59.550729+0000 mgr.y (mgr.14146) 583 : cluster [DBG] pgmap v656: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-12T16:05:00.868 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:00 smithi121 bash[18489]: cluster 2024-02-12T16:04:59.616360+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e166: 12 total, 12 up, 12 in 2024-02-12T16:05:00.868 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:00 smithi121 bash[18489]: cluster 2024-02-12T16:05:00.222733+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e167: 12 total, 12 up, 12 in 2024-02-12T16:05:00.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:00 smithi123 bash[19119]: cluster 2024-02-12T16:04:59.550729+0000 mgr.y (mgr.14146) 583 : cluster [DBG] pgmap v656: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-12T16:05:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:00 smithi123 bash[19119]: cluster 2024-02-12T16:04:59.616360+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e166: 12 total, 12 up, 12 in 2024-02-12T16:05:00.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:00 smithi123 bash[19119]: cluster 2024-02-12T16:05:00.222733+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e167: 12 total, 12 up, 12 in 2024-02-12T16:05:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:00.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:00.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:01.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:01.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:01.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:01.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:01.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:01.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:01.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:01.525 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:01.526 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:01.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:01.565 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:01.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 16 112106 112090 162.905 102.75 0.00829431 0.00613719 2024-02-12T16:05:01.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:01.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:01.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:01.836 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:01.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:01.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:02.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:02.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:02.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:02.222 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:02.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:02.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:02.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:02.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:02.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:02.476 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:02.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:02.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:02.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:02.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:02 smithi123 bash[19119]: cluster 2024-02-12T16:05:01.551125+0000 mgr.y (mgr.14146) 584 : cluster [DBG] pgmap v659: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s 2024-02-12T16:05:02.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 113777 113761 161.576 104.438 0.00429637 0.00618757 2024-02-12T16:05:02.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:02.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:02 smithi169 bash[19604]: cluster 2024-02-12T16:05:01.551125+0000 mgr.y (mgr.14146) 584 : cluster [DBG] pgmap v659: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s 2024-02-12T16:05:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:02.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:02.758 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:02.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:02.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:02 smithi121 bash[18489]: cluster 2024-02-12T16:05:01.551125+0000 mgr.y (mgr.14146) 584 : cluster [DBG] pgmap v659: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s 2024-02-12T16:05:02.878 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:02.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:02.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:03.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:03.146 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:03.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:03.282 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:03.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:03.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:03.521 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:03.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:03.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 115345 115329 160.163 98 0.00464418 0.00624171 2024-02-12T16:05:03.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.488679+0000 mon.a (mon.0) 888 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=462360833,config=2903391811,health=256908861,kv=3761071382}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.491200+0000 mon.a (mon.0) 889 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2321845214,logm=1894287832}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.492030+0000 mon.a (mon.0) 890 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559662840}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.492869+0000 mon.a (mon.0) 891 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208735753}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.493662+0000 mon.a (mon.0) 892 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066359861}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.494465+0000 mon.a (mon.0) 893 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1410093565}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.497744+0000 mon.a (mon.0) 894 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1230780804}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.501461+0000 mon.a (mon.0) 895 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142472375}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.504747+0000 mon.a (mon.0) 896 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3909342886}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.509070+0000 mon.a (mon.0) 897 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486394724}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.512407+0000 mon.a (mon.0) 898 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231922165}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.514148+0000 mon.a (mon.0) 899 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=16,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=52} crc {logm=2904024843,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=1477226476,mon_config_key=803339340}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.514976+0000 mon.a (mon.0) 900 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3875129731}) 2024-02-12T16:05:03.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.515696+0000 mon.a (mon.0) 901 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=94,monmap=5,osd_metadata=1} crc {mon_config_key=3107573434,monmap=2720272532,osd_metadata=3627385991}) 2024-02-12T16:05:03.664 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.516595+0000 mon.a (mon.0) 902 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=11,osd_pg_creating=1,osdmap=88} crc {osd_metadata=2338078737,osd_pg_creating=3841215591,osdmap=1745822062}) 2024-02-12T16:05:03.664 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:03 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.522478+0000 mon.a (mon.0) 903 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=50} crc {osdmap=2457113304}) 2024-02-12T16:05:03.679 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:03.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:03.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:03.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:03.958 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:03.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.488679+0000 mon.a (mon.0) 888 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=462360833,config=2903391811,health=256908861,kv=3761071382}) 2024-02-12T16:05:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.491200+0000 mon.a (mon.0) 889 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2321845214,logm=1894287832}) 2024-02-12T16:05:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.492030+0000 mon.a (mon.0) 890 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559662840}) 2024-02-12T16:05:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.492869+0000 mon.a (mon.0) 891 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208735753}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.493662+0000 mon.a (mon.0) 892 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066359861}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.494465+0000 mon.a (mon.0) 893 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1410093565}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.497744+0000 mon.a (mon.0) 894 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1230780804}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.501461+0000 mon.a (mon.0) 895 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142472375}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.504747+0000 mon.a (mon.0) 896 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3909342886}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.509070+0000 mon.a (mon.0) 897 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486394724}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.512407+0000 mon.a (mon.0) 898 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231922165}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.514148+0000 mon.a (mon.0) 899 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=16,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=52} crc {logm=2904024843,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=1477226476,mon_config_key=803339340}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.514976+0000 mon.a (mon.0) 900 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3875129731}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.515696+0000 mon.a (mon.0) 901 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=94,monmap=5,osd_metadata=1} crc {mon_config_key=3107573434,monmap=2720272532,osd_metadata=3627385991}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.516595+0000 mon.a (mon.0) 902 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=11,osd_pg_creating=1,osdmap=88} crc {osd_metadata=2338078737,osd_pg_creating=3841215591,osdmap=1745822062}) 2024-02-12T16:05:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:03 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.522478+0000 mon.a (mon.0) 903 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=50} crc {osdmap=2457113304}) 2024-02-12T16:05:04.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:04.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:04.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:04.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:04.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:04.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:04.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:03 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.488679+0000 mon.a (mon.0) 888 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=462360833,config=2903391811,health=256908861,kv=3761071382}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:03 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.491200+0000 mon.a (mon.0) 889 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2321845214,logm=1894287832}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.492030+0000 mon.a (mon.0) 890 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559662840}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.492869+0000 mon.a (mon.0) 891 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208735753}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.493662+0000 mon.a (mon.0) 892 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066359861}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.494465+0000 mon.a (mon.0) 893 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1410093565}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.497744+0000 mon.a (mon.0) 894 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1230780804}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.501461+0000 mon.a (mon.0) 895 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2142472375}) 2024-02-12T16:05:04.380 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.504747+0000 mon.a (mon.0) 896 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3909342886}) 2024-02-12T16:05:04.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.509070+0000 mon.a (mon.0) 897 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486394724}) 2024-02-12T16:05:04.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.512407+0000 mon.a (mon.0) 898 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231922165}) 2024-02-12T16:05:04.383 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:04.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:04.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:04.520 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:04.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:04.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:04.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.514148+0000 mon.a (mon.0) 899 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=16,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=52} crc {logm=2904024843,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=1477226476,mon_config_key=803339340}) 2024-02-12T16:05:04.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.514976+0000 mon.a (mon.0) 900 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3875129731}) 2024-02-12T16:05:04.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.515696+0000 mon.a (mon.0) 901 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=94,monmap=5,osd_metadata=1} crc {mon_config_key=3107573434,monmap=2720272532,osd_metadata=3627385991}) 2024-02-12T16:05:04.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.516595+0000 mon.a (mon.0) 902 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=11,osd_pg_creating=1,osdmap=88} crc {osd_metadata=2338078737,osd_pg_creating=3841215591,osdmap=1745822062}) 2024-02-12T16:05:04.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.522478+0000 mon.a (mon.0) 903 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=50} crc {osdmap=2457113304}) 2024-02-12T16:05:04.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 16 115866 115850 157.389 32.5625 0.00924913 0.00635224 2024-02-12T16:05:04.662 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:04.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:04.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:04 smithi169 bash[19604]: cluster 2024-02-12T16:05:03.551535+0000 mgr.y (mgr.14146) 585 : cluster [DBG] pgmap v660: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s 2024-02-12T16:05:04.802 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:04.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:04 smithi121 bash[18489]: cluster 2024-02-12T16:05:03.551535+0000 mgr.y (mgr.14146) 585 : cluster [DBG] pgmap v660: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s 2024-02-12T16:05:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:04.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:04.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:04.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:04 smithi123 bash[19119]: cluster 2024-02-12T16:05:03.551535+0000 mgr.y (mgr.14146) 585 : cluster [DBG] pgmap v660: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.78k op/s 2024-02-12T16:05:04.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:04.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:05.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:05.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:05.169 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:05.194 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:05.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:05.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:05.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:05.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:05.356 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:05:05.389 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:05:05.425 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:05:05.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:05.462 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:05:05.473 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:05.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:05.490 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:05:05.516 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:05:05.542 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:05:05.570 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:05:05.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:05.601 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:05:05.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:05.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:05.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:05:05.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 16 117375 117359 156.047 94.3125 0.0155807 0.0064064 2024-02-12T16:05:05.660 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:05:05.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:05:05.685 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.685 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.685 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.685 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.686 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:05:05.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:05.738 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:05.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:05.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:05.864 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:05.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:05.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:06.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.144 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:06.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:06.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:06.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:06.526 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:06.533 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:06 smithi123 bash[19119]: audit 2024-02-12T16:05:05.303318+0000 mon.a (mon.0) 904 : audit [DBG] from='client.? v1:172.21.15.121:0/1125070606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:06.533 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:06 smithi123 bash[19119]: cluster 2024-02-12T16:05:05.551942+0000 mgr.y (mgr.14146) 586 : cluster [DBG] pgmap v661: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:05:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:06 smithi169 bash[19604]: audit 2024-02-12T16:05:05.303318+0000 mon.a (mon.0) 904 : audit [DBG] from='client.? v1:172.21.15.121:0/1125070606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:06 smithi169 bash[19604]: cluster 2024-02-12T16:05:05.551942+0000 mgr.y (mgr.14146) 586 : cluster [DBG] pgmap v661: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:05:06.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:06.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 118872 118856 154.745 93.5625 0.00427173 0.00646075 2024-02-12T16:05:06.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:06 smithi121 bash[18489]: audit 2024-02-12T16:05:05.303318+0000 mon.a (mon.0) 904 : audit [DBG] from='client.? v1:172.21.15.121:0/1125070606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:06.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:06 smithi121 bash[18489]: cluster 2024-02-12T16:05:05.551942+0000 mgr.y (mgr.14146) 586 : cluster [DBG] pgmap v661: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:05:06.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:06.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:06.829 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:06.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:06.961 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:06.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:07.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:07.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:07.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:07.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:07.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:07.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:07.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:07.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:07.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:07.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:07.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:07.480 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:07.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:07.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:07.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:07.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:07.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 120291 120275 153.397 88.6875 0.00409551 0.00651684 2024-02-12T16:05:07.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:07.742 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:07.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:07.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:07.879 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:07.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:07.987 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:08.008 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:08.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:08.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:08.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:08.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:08.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:08.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:08.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:08.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:08.615 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:08 smithi123 bash[19119]: cluster 2024-02-12T16:05:07.552398+0000 mgr.y (mgr.14146) 587 : cluster [DBG] pgmap v662: 17 pgs: 17 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-02-12T16:05:08.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:08.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 121869 121853 152.301 98.625 0.00794912 0.0065643 2024-02-12T16:05:08.656 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:08.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:08 smithi121 bash[18489]: cluster 2024-02-12T16:05:07.552398+0000 mgr.y (mgr.14146) 587 : cluster [DBG] pgmap v662: 17 pgs: 17 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-02-12T16:05:08.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:08.769 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:08 smithi169 bash[19604]: cluster 2024-02-12T16:05:07.552398+0000 mgr.y (mgr.14146) 587 : cluster [DBG] pgmap v662: 17 pgs: 17 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-02-12T16:05:08.795 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:08.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:08.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:08.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:09.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:09.067 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:09.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:09.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:09.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:09.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:09.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:09.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:09.576 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:09.597 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:09.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 123372 123356 151.156 93.9375 0.0200498 0.00661388 2024-02-12T16:05:09.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:09.722 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:09.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:05:09.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:09.858 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:09.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:09.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:09.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:10.066 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:05:10.066 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:05:10.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:10.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:10.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:10.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:10.254 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:10.259 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:05:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:10.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:10.271 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:05:10.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:10.388 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:10.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:10.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:10.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:10.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:10.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:10.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:10.630 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:05:10.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 16 124719 124703 149.868 84.1875 0.0116536 0.00667111 2024-02-12T16:05:10.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:10.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:10.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:10.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:10 smithi169 bash[19604]: cluster 2024-02-12T16:05:09.552834+0000 mgr.y (mgr.14146) 588 : cluster [DBG] pgmap v663: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-12T16:05:10.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:10 smithi169 bash[19604]: audit 2024-02-12T16:05:09.760109+0000 mon.a (mon.0) 905 : audit [INF] from='client.? v1:172.21.15.121:0/3524314402' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:05:10.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:10.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:10.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:10 smithi123 bash[19119]: cluster 2024-02-12T16:05:09.552834+0000 mgr.y (mgr.14146) 588 : cluster [DBG] pgmap v663: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-12T16:05:10.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:10 smithi123 bash[19119]: audit 2024-02-12T16:05:09.760109+0000 mon.a (mon.0) 905 : audit [INF] from='client.? v1:172.21.15.121:0/3524314402' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:05:10.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:10.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:10 smithi121 bash[18489]: cluster 2024-02-12T16:05:09.552834+0000 mgr.y (mgr.14146) 588 : cluster [DBG] pgmap v663: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-12T16:05:10.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:10 smithi121 bash[18489]: audit 2024-02-12T16:05:09.760109+0000 mon.a (mon.0) 905 : audit [INF] from='client.? v1:172.21.15.121:0/3524314402' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:05:10.918 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:10.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:11.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:11.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:11.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:11.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:11.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:11.318 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:11.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:11.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:05:11.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:11.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:11.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:11.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:11 smithi169 bash[19604]: cluster 2024-02-12T16:05:10.233862+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:05:11.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:11 smithi169 bash[19604]: audit 2024-02-12T16:05:10.239606+0000 mgr.y (mgr.14146) 589 : audit [DBG] from='client.24782 v1:172.21.15.121:0/2336618943' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:05:11.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:11 smithi169 bash[19604]: cluster 2024-02-12T16:05:10.608232+0000 mon.a (mon.0) 906 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:11.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:11 smithi169 bash[19604]: audit 2024-02-12T16:05:10.619740+0000 mon.a (mon.0) 907 : audit [INF] from='client.? v1:172.21.15.121:0/3524314402' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:05:11.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:11 smithi169 bash[19604]: cluster 2024-02-12T16:05:10.619835+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e168: 12 total, 12 up, 12 in 2024-02-12T16:05:11.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 126262 126246 148.86 96.4375 0.00550102 0.0067159 2024-02-12T16:05:11.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:11.715 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:11.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:11.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:11.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:11 smithi123 bash[19119]: cluster 2024-02-12T16:05:10.233862+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:05:11.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:11 smithi123 bash[19119]: audit 2024-02-12T16:05:10.239606+0000 mgr.y (mgr.14146) 589 : audit [DBG] from='client.24782 v1:172.21.15.121:0/2336618943' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:05:11.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:11 smithi123 bash[19119]: cluster 2024-02-12T16:05:10.608232+0000 mon.a (mon.0) 906 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:11.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:11 smithi123 bash[19119]: audit 2024-02-12T16:05:10.619740+0000 mon.a (mon.0) 907 : audit [INF] from='client.? v1:172.21.15.121:0/3524314402' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:05:11.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:11 smithi123 bash[19119]: cluster 2024-02-12T16:05:10.619835+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e168: 12 total, 12 up, 12 in 2024-02-12T16:05:11.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:11 smithi121 bash[18489]: cluster 2024-02-12T16:05:10.233862+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.7 scrub starts 2024-02-12T16:05:11.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:11 smithi121 bash[18489]: audit 2024-02-12T16:05:10.239606+0000 mgr.y (mgr.14146) 589 : audit [DBG] from='client.24782 v1:172.21.15.121:0/2336618943' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:05:11.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:11 smithi121 bash[18489]: cluster 2024-02-12T16:05:10.608232+0000 mon.a (mon.0) 906 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:11.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:11 smithi121 bash[18489]: audit 2024-02-12T16:05:10.619740+0000 mon.a (mon.0) 907 : audit [INF] from='client.? v1:172.21.15.121:0/3524314402' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:05:11.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:11 smithi121 bash[18489]: cluster 2024-02-12T16:05:10.619835+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e168: 12 total, 12 up, 12 in 2024-02-12T16:05:11.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:11.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:11.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:11.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:12.011 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-02-12T16:05:09.552581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7833649152,"num_objects":119532,"num_object_clones":0,"num_object_copies":239064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119532,"num_write_kb":7650048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119532,"ondisk_log_size":119532,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18866880,"kb_used_data":15214176,"kb_used_omap":0,"kb_used_meta":3652608,"kb_avail":1106074944,"statfs":{"total":1151940427776,"available":1132620742656,"internally_reserved":0,"allocated":15579316224,"data_stored":15573642380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3740270592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":990445568,"num_objects":15113,"num_object_clones":0,"num_object_copies":30226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15113,"num_write_kb":967232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.938721"},"pg_stats":[{"pgid":"2.f","version":"167'7539","reported_seq":7565,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:08.657744+0000","last_change":"2024-02-12T16:04:03.074738+0000","last_active":"2024-02-12T16:05:08.657744+0000","last_peered":"2024-02-12T16:05:08.657744+0000","last_clean":"2024-02-12T16:05:08.657744+0000","last_became_active":"2024-02-12T16:04:03.074484+0000","last_became_peered":"2024-02-12T16:04:03.074484+0000","last_unstale":"2024-02-12T16:05:08.657744+0000","last_undegraded":"2024-02-12T16:05:08.657744+0000","last_fullsized":"2024-02-12T16:05:08.657744+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7539,"ondisk_log_size":7539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":494075904,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7539,"num_write_kb":482496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"167'7421","reported_seq":7448,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.525895+0000","last_change":"2024-02-12T16:04:03.106274+0000","last_active":"2024-02-12T16:05:06.525895+0000","last_peered":"2024-02-12T16:05:06.525895+0000","last_clean":"2024-02-12T16:05:06.525895+0000","last_became_active":"2024-02-12T16:04:03.105925+0000","last_became_peered":"2024-02-12T16:04:03.105925+0000","last_unstale":"2024-02-12T16:05:06.525895+0000","last_undegraded":"2024-02-12T16:05:06.525895+0000","last_fullsized":"2024-02-12T16:05:06.525895+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7421,"ondisk_log_size":7421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":486342656,"num_objects":7421,"num_object_clones":0,"num_object_copies":14842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":474944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"167'7456","reported_seq":7483,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.903209+0000","last_change":"2024-02-12T16:04:03.078000+0000","last_active":"2024-02-12T16:05:06.903209+0000","last_peered":"2024-02-12T16:05:06.903209+0000","last_clean":"2024-02-12T16:05:06.903209+0000","last_became_active":"2024-02-12T16:04:03.077754+0000","last_became_peered":"2024-02-12T16:04:03.077754+0000","last_unstale":"2024-02-12T16:05:06.903209+0000","last_undegraded":"2024-02-12T16:05:06.903209+0000","last_fullsized":"2024-02-12T16:05:06.903209+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7456,"ondisk_log_size":7456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":488636416,"num_objects":7456,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7456,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"167'7321","reported_seq":7346,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.177924+0000","last_change":"2024-02-12T16:04:03.069405+0000","last_active":"2024-02-12T16:05:06.177924+0000","last_peered":"2024-02-12T16:05:06.177924+0000","last_clean":"2024-02-12T16:05:06.177924+0000","last_became_active":"2024-02-12T16:04:03.069103+0000","last_became_peered":"2024-02-12T16:04:03.069103+0000","last_unstale":"2024-02-12T16:05:06.177924+0000","last_undegraded":"2024-02-12T16:05:06.177924+0000","last_fullsized":"2024-02-12T16:05:06.177924+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7321,"ondisk_log_size":7321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":479789056,"num_objects":7321,"num_object_clones":0,"num_object_copies":14642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7321,"num_write_kb":468544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"167'7555","reported_seq":7581,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.943109+0000","last_change":"2024-02-12T16:04:03.076121+0000","last_active":"2024-02-12T16:05:06.943109+0000","last_peered":"2024-02-12T16:05:06.943109+0000","last_clean":"2024-02-12T16:05:06.943109+0000","last_became_active":"2024-02-12T16:04:03.075819+0000","last_became_peered":"2024-02-12T16:04:03.075819+0000","last_unstale":"2024-02-12T16:05:06.943109+0000","last_undegraded":"2024-02-12T16:05:06.943109+0000","last_fullsized":"2024-02-12T16:05:06.943109+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7555,"ondisk_log_size":7555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":495124480,"num_objects":7555,"num_object_clones":0,"num_object_copies":15110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":483520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":176,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-02-12T16:04:59.629002+0000","last_change":"2024-02-12T16:03:59.776773+0000","last_active":"2024-02-12T16:04:59.629002+0000","last_peered":"2024-02-12T16:04:59.629002+0000","last_clean":"2024-02-12T16:04:59.629002+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:04:59.629002+0000","last_undegraded":"2024-02-12T16:04:59.629002+0000","last_fullsized":"2024-02-12T16:04:59.629002+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:03:59.776674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"167'7638","reported_seq":7665,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:08.306784+0000","last_change":"2024-02-12T16:04:03.113429+0000","last_active":"2024-02-12T16:05:08.306784+0000","last_peered":"2024-02-12T16:05:08.306784+0000","last_clean":"2024-02-12T16:05:08.306784+0000","last_became_active":"2024-02-12T16:04:03.113129+0000","last_became_peered":"2024-02-12T16:04:03.113129+0000","last_unstale":"2024-02-12T16:05:08.306784+0000","last_undegraded":"2024-02-12T16:05:08.306784+0000","last_fullsized":"2024-02-12T16:05:08.306784+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7638,"ondisk_log_size":7638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":500563968,"num_objects":7638,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7638,"num_write_kb":488832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"167'7506","reported_seq":7533,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:08.669971+0000","last_change":"2024-02-12T16:04:03.111231+0000","last_active":"2024-02-12T16:05:08.669971+0000","last_peered":"2024-02-12T16:05:08.669971+0000","last_clean":"2024-02-12T16:05:08.669971+0000","last_became_active":"2024-02-12T16:04:03.110555+0000","last_became_peered":"2024-02-12T16:04:03.110555+0000","last_unstale":"2024-02-12T16:05:08.669971+0000","last_undegraded":"2024-02-12T16:05:08.669971+0000","last_fullsized":"2024-02-12T16:05:08.669971+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7506,"ondisk_log_size":7506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":491913216,"num_objects":7506,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7506,"num_write_kb":480384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"167'7396","reported_seq":7423,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.509805+0000","last_change":"2024-02-12T16:04:03.071933+0000","last_active":"2024-02-12T16:05:06.509805+0000","last_peered":"2024-02-12T16:05:06.509805+0000","last_clean":"2024-02-12T16:05:06.509805+0000","last_became_active":"2024-02-12T16:04:03.071599+0000","last_became_peered":"2024-02-12T16:04:03.071599+0000","last_unstale":"2024-02-12T16:05:06.509805+0000","last_undegraded":"2024-02-12T16:05:06.509805+0000","last_fullsized":"2024-02-12T16:05:06.509805+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7396,"ondisk_log_size":7396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":484704256,"num_objects":7396,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"167'7631","reported_seq":7658,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:08.859083+0000","last_change":"2024-02-12T16:04:03.114770+0000","last_active":"2024-02-12T16:05:08.859083+0000","last_peered":"2024-02-12T16:05:08.859083+0000","last_clean":"2024-02-12T16:05:08.859083+0000","last_became_active":"2024-02-12T16:04:03.114419+0000","last_became_peered":"2024-02-12T16:04:03.114419+0000","last_unstale":"2024-02-12T16:05:08.859083+0000","last_undegraded":"2024-02-12T16:05:08.859083+0000","last_fullsized":"2024-02-12T16:05:08.859083+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7631,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":500105216,"num_objects":7631,"num_object_clones":0,"num_object_copies":15262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":488384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"167'7734","reported_seq":7760,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:08.322368+0000","last_change":"2024-02-12T16:04:03.076402+0000","last_active":"2024-02-12T16:05:08.322368+0000","last_peered":"2024-02-12T16:05:08.322368+0000","last_clean":"2024-02-12T16:05:08.322368+0000","last_became_active":"2024-02-12T16:04:03.075975+0000","last_became_peered":"2024-02-12T16:04:03.075975+0000","last_unstale":"2024-02-12T16:05:08.322368+0000","last_undegraded":"2024-02-12T16:05:08.322368+0000","last_fullsized":"2024-02-12T16:05:08.322368+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7734,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":506855424,"num_objects":7734,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7734,"num_write_kb":494976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"167'7501","reported_seq":7526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.538814+0000","last_change":"2024-02-12T16:04:03.070646+0000","last_active":"2024-02-12T16:05:06.538814+0000","last_peered":"2024-02-12T16:05:06.538814+0000","last_clean":"2024-02-12T16:05:06.538814+0000","last_became_active":"2024-02-12T16:04:03.070386+0000","last_became_peered":"2024-02-12T16:04:03.070386+0000","last_unstale":"2024-02-12T16:05:06.538814+0000","last_undegraded":"2024-02-12T16:05:06.538814+0000","last_fullsized":"2024-02-12T16:05:06.538814+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7501,"ondisk_log_size":7501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":491585536,"num_objects":7501,"num_object_clones":0,"num_object_copies":15002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7501,"num_write_kb":480064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"167'7197","reported_seq":7224,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:04.862938+0000","last_change":"2024-02-12T16:04:03.073707+0000","last_active":"2024-02-12T16:05:04.862938+0000","last_peered":"2024-02-12T16:05:04.862938+0000","last_clean":"2024-02-12T16:05:04.862938+0000","last_became_active":"2024-02-12T16:04:03.073456+0000","last_became_peered":"2024-02-12T16:04:03.073456+0000","last_unstale":"2024-02-12T16:05:04.862938+0000","last_undegraded":"2024-02-12T16:05:04.862938+0000","last_fullsized":"2024-02-12T16:05:04.862938+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7197,"ondisk_log_size":7197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":471662592,"num_objects":7197,"num_object_clones":0,"num_object_copies":14394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7197,"num_write_kb":460608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"167'7501","reported_seq":7528,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:08.656753+0000","last_change":"2024-02-12T16:04:03.073798+0000","last_active":"2024-02-12T16:05:08.656753+0000","last_peered":"2024-02-12T16:05:08.656753+0000","last_clean":"2024-02-12T16:05:08.656753+0000","last_became_active":"2024-02-12T16:04:03.073550+0000","last_became_peered":"2024-02-12T16:04:03.073550+0000","last_unstale":"2024-02-12T16:05:08.656753+0000","last_undegraded":"2024-02-12T16:05:08.656753+0000","last_fullsized":"2024-02-12T16:05:08.656753+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7501,"ondisk_log_size":7501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":491585536,"num_objects":7501,"num_object_clones":0,"num_object_copies":15002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7501,"num_write_kb":480064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"167'7277","reported_seq":7304,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:04.847949+0000","last_change":"2024-02-12T16:04:03.111009+0000","last_active":"2024-02-12T16:05:04.847949+0000","last_peered":"2024-02-12T16:05:04.847949+0000","last_clean":"2024-02-12T16:05:04.847949+0000","last_became_active":"2024-02-12T16:04:03.110305+0000","last_became_peered":"2024-02-12T16:04:03.110305+0000","last_unstale":"2024-02-12T16:05:04.847949+0000","last_undegraded":"2024-02-12T16:05:04.847949+0000","last_fullsized":"2024-02-12T16:05:04.847949+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7277,"ondisk_log_size":7277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":476905472,"num_objects":7277,"num_object_clones":0,"num_object_copies":14554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7277,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"167'7448","reported_seq":7475,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.300377+0000","last_change":"2024-02-12T16:04:03.110487+0000","last_active":"2024-02-12T16:05:06.300377+0000","last_peered":"2024-02-12T16:05:06.300377+0000","last_clean":"2024-02-12T16:05:06.300377+0000","last_became_active":"2024-02-12T16:04:03.110162+0000","last_became_peered":"2024-02-12T16:04:03.110162+0000","last_unstale":"2024-02-12T16:05:06.300377+0000","last_undegraded":"2024-02-12T16:05:06.300377+0000","last_fullsized":"2024-02-12T16:05:06.300377+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7448,"ondisk_log_size":7448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":488112128,"num_objects":7448,"num_object_clones":0,"num_object_copies":14896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7448,"num_write_kb":476672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"167'7411","reported_seq":7438,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-12T16:05:06.142119+0000","last_change":"2024-02-12T16:04:03.069883+0000","last_active":"2024-02-12T16:05:06.142119+0000","last_peered":"2024-02-12T16:05:06.142119+0000","last_clean":"2024-02-12T16:05:06.142119+0000","last_became_active":"2024-02-12T16:04:03.069615+0000","last_became_peered":"2024-02-12T16:04:03.069615+0000","last_unstale":"2024-02-12T16:05:06.142119+0000","last_undegraded":"2024-02-12T16:05:06.142119+0000","last_fullsized":"2024-02-12T16:05:06.142119+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:04:02.047306+0000","last_clean_scrub_stamp":"2024-02-12T16:04:02.047306+0000","log_size":7411,"ondisk_log_size":7411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":485687296,"num_objects":7411,"num_object_clones":0,"num_object_copies":14822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7411,"num_write_kb":474304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":7833649152,"num_objects":119532,"num_object_clones":0,"num_object_copies":239064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119532,"num_write_kb":7650048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15692070912,"data_stored":15692070912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119532,"ondisk_log_size":119532,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958611,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689200,"kb_used_data":1375592,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":92055952,"statfs":{"total":95995035648,"available":94265294848,"internally_reserved":0,"allocated":1408606208,"data_stored":1408133371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.995,"15min":1.024},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.05,"15min":1.1100000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.647,"5min":1.647,"15min":2.871},"last":0.70099999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.94299999999999995,"15min":1.0669999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.2669999999999999,"5min":1.9590000000000001,"15min":2.7109999999999999},"last":1.01},{"interface":"front","average":{"1min":1.097,"5min":1.077,"15min":1.087},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.7789999999999999,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.73399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.073,"15min":1.141},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.724,"5min":1.798,"15min":3.2429999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.02,"15min":1.196},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.4419999999999999,"5min":2.3210000000000002,"15min":2.6280000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1000000000000001,"15min":1.194},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":2.5299999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.056,"15min":1.1759999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5660000000000001,"5min":1.8839999999999999,"15min":3.1429999999999998},"last":1.0780000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.127,"15min":1.323},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.6299999999999999,"5min":2.1930000000000001,"15min":2.9500000000000002},"last":1.107},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.226,"15min":1.3959999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6040000000000001,"5min":1.8400000000000001,"15min":4.0789999999999997},"last":0.36799999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.3420000000000001,"15min":1.448},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.758,"5min":1.758,"15min":3.044},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.085,"5min":1.2749999999999999,"15min":1.4570000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.794,"5min":2.4990000000000001,"15min":4.2190000000000003},"last":1.0600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.2989999999999999,"15min":1.4890000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.919,"5min":1.919,"15min":4.4989999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.389,"15min":1.5700000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":3.3340000000000001},"last":0.441}]},{"osd":7,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.393,"15min":1.579},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":4.3129999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.371,"5min":1.4850000000000001,"15min":1.675},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.96,"5min":2.0070000000000001,"15min":3.8370000000000002},"last":0.66500000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.431,"15min":1.6060000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.9179999999999999,"5min":1.9910000000000001,"15min":4},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.5169999999999999,"15min":1.6990000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":2.0779999999999998,"5min":2.2290000000000001,"15min":4.5789999999999997},"last":0.60199999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.381,"15min":1.631},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.873,"5min":2.4049999999999998,"15min":4.2610000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.573,"15min":1.7789999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":4.3949999999999996},"last":0.68400000000000005}]}]},{"osd":10,"up_from":83,"seq":356482285661,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1273264,"kb_used_data":967848,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92471888,"statfs":{"total":95995035648,"available":94691213312,"internally_reserved":0,"allocated":991076352,"data_stored":990603515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.90100000000000002,"15min":0.89700000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.97999999999999998,"15min":1.0429999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.4119999999999999,"5min":3.4119999999999999,"15min":3.4119999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.96599999999999997,"15min":0.97899999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6349999999999998,"5min":3.6349999999999998,"15min":3.6349999999999998},"last":4.8300000000000001},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.95499999999999996,"15min":0.95399999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.97199999999999998,"15min":0.999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.542,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":4.8620000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.96399999999999997,"15min":1},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.4100000000000001,"5min":5.4100000000000001,"15min":5.4100000000000001},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.1830000000000001,"15min":1.127},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.597,"5min":1.776,"15min":1.776},"last":4.9189999999999996},{"interface":"front","average":{"1min":0.85499999999999998,"5min":1.1100000000000001,"15min":1.1970000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.4630000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.1140000000000001,"15min":1.165},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.4420000000000002,"5min":5.4420000000000002,"15min":5.4420000000000002},"last":0.379},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.2350000000000001,"15min":1.2829999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.556,"5min":3.556,"15min":3.556},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.1399999999999999,"15min":1.202},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":5.4729999999999999,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.3109999999999999,"15min":1.337},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.597,"5min":3.597,"15min":3.597},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.2889999999999999,"15min":1.282},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.615,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":4.8920000000000003},{"interface":"front","average":{"1min":1.232,"5min":1.3140000000000001,"15min":1.3620000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":5.5099999999999998,"5min":5.5099999999999998,"15min":5.5099999999999998},"last":0.45300000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.306,"15min":1.3300000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5880000000000001,"5min":2,"15min":2},"last":4.7309999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4419999999999999,"15min":1.367},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":7.3810000000000002,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":0.63300000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.452,"15min":1.4419999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.5219999999999998,"5min":3.5219999999999998,"15min":3.5219999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.194,"5min":1.46,"15min":1.4179999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":4.7789999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:04:39 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.498,"15min":1.5780000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0529999999999999,"5min":2.077,"15min":2.077},"last":4.8010000000000002},{"interface":"front","average":{"1min":1.444,"5min":1.5509999999999999,"15min":1.615},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":3.4929999999999999,"5min":3.4929999999999999,"15min":3.4929999999999999},"last":0.68300000000000005}]}]},{"osd":9,"up_from":74,"seq":317827580005,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725552,"kb_used_data":1428328,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92019600,"statfs":{"total":95995035648,"available":94228070400,"internally_reserved":0,"allocated":1462607872,"data_stored":1462135055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.92300000000000004,"15min":0.92900000000000005},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2370000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.93000000000000005,"15min":0.96599999999999997},"min":{"1min":0.65200000000000002,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0860000000000001,"5min":1.4419999999999999,"15min":2.2480000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.069,"15min":1.014},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.744,"5min":1.744,"15min":1.744},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.04,"15min":1.032},"min":{"1min":0.65300000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2789999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.94599999999999995,"15min":0.93400000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.3140000000000001,"5min":1.343,"15min":1.738},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.94099999999999995,"15min":1.014},"min":{"1min":0.77200000000000002,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.601},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94899999999999995,"15min":0.96299999999999997},"min":{"1min":0.71999999999999997,"5min":0.61399999999999999,"15min":0.60699999999999998},"max":{"1min":1.3120000000000001,"5min":1.4470000000000001,"15min":2.1549999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1020000000000001,"15min":1.1080000000000001},"min":{"1min":0.69699999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.538,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.84499999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.139,"15min":1.165},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.401,"5min":1.6739999999999999,"15min":2.113},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1459999999999999,"15min":1.1819999999999999},"min":{"1min":0.83599999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.476,"5min":1.5680000000000001,"15min":2.3260000000000001},"last":1.0860000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.224,"15min":1.2290000000000001},"min":{"1min":1.0780000000000001,"5min":0.77000000000000002,"15min":0.71499999999999997},"max":{"1min":1.4390000000000001,"5min":1.756,"15min":2.4399999999999999},"last":617.95899999999995},{"interface":"front","average":{"1min":1.383,"5min":1.3440000000000001,"15min":1.284},"min":{"1min":0.98899999999999999,"5min":0.92600000000000005,"15min":0.76300000000000001},"max":{"1min":1.639,"5min":1.6830000000000001,"15min":2.2599999999999998},"last":617.88099999999997}]},{"osd":6,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3009999999999999,"15min":1.319},"min":{"1min":1.1040000000000001,"5min":0.89700000000000002,"15min":0.82199999999999995},"max":{"1min":1.415,"5min":1.8340000000000001,"15min":1.9139999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.224,"5min":1.29,"15min":1.3420000000000001},"min":{"1min":0.93200000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.5469999999999999,"5min":1.585,"15min":2.48},"last":1.1439999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.254,"15min":1.3100000000000001},"min":{"1min":0.876,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.5549999999999999,"5min":1.7330000000000001,"15min":2.3519999999999999},"last":613.28200000000004},{"interface":"front","average":{"1min":1.381,"5min":1.421,"15min":1.4099999999999999},"min":{"1min":1.054,"5min":0.98099999999999998,"15min":0.875},"max":{"1min":1.5880000000000001,"5min":1.772,"15min":2.0249999999999999},"last":613.25199999999995}]},{"osd":8,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.353,"15min":1.3600000000000001},"min":{"1min":0.95299999999999996,"5min":0.86399999999999999,"15min":0.61099999999999999},"max":{"1min":1.6080000000000001,"5min":1.792,"15min":2.1299999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.355,"15min":1.355},"min":{"1min":1.083,"5min":0.82899999999999996,"15min":0.58499999999999996},"max":{"1min":1.5740000000000001,"5min":1.7649999999999999,"15min":2.3889999999999998},"last":0.96899999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.345,"15min":1.409},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.5110000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.3300000000000001,"15min":1.343},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.83199999999999996}]}]},{"osd":8,"up_from":65,"seq":279172874348,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2214512,"kb_used_data":1909096,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91530640,"statfs":{"total":95995035648,"available":93727375360,"internally_reserved":0,"allocated":1954914304,"data_stored":1954441487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.91600000000000004,"15min":0.86599999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.621},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.94799999999999995,"15min":0.90700000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:03:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.88800000000000001,"15min":0.85099999999999998},"min":{"1min":0.65500000000000003,"5min":0.64800000000000002,"15min":0.60199999999999998},"max":{"1min":1.0369999999999999,"5min":1.508,"15min":1.508},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.0169999999999999,"15min":0.96199999999999997},"min":{"1min":0.76800000000000002,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.8480000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.04,"15min":0.91200000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.2110000000000001,"15min":1.0600000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":0.90200000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.0609999999999999,"15min":1.0089999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.7869999999999999,"5min":1.7869999999999999,"15min":1.7869999999999999},"last":1.079},{"interface":"front","average":{"1min":1.196,"5min":1.2130000000000001,"15min":1.071},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.204,"15min":1.1579999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.258,"5min":1.3180000000000001,"15min":1.147},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8029999999999999,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.113,"15min":1.147},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.5580000000000001,"5min":1.6870000000000001,"15min":2.2200000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.095,"5min":1.3009999999999999,"15min":1.214},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.546,"5min":1.8879999999999999,"15min":1.8999999999999999},"last":1.2050000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.246,"15min":1.196},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.6180000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.409,"5min":1.3879999999999999,"15min":1.278},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.879,"5min":1.879,"15min":2.339},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.395,"15min":1.2809999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.7090000000000001,"5min":1.97,"15min":2.0179999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.482,"15min":1.3260000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.944,"5min":1.944,"15min":1.984},"last":0.89000000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:04:18 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.522,"15min":1.294},"min":{"1min":0.95399999999999996,"5min":0.70499999999999996,"15min":0.50900000000000001},"max":{"1min":1.8680000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3200000000000001,"15min":1.232},"min":{"1min":0.85699999999999998,"5min":0.72999999999999998,"15min":0.67200000000000004},"max":{"1min":1.802,"5min":2.456,"15min":2.456},"last":0.84899999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:04:54 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.323,"15min":1.3520000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.496,"5min":2.496,"15min":2.5209999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.391,"15min":1.4359999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.617,"5min":1.905,"15min":2.0009999999999999},"last":1.25}]},{"osd":11,"last update":"Mon Feb 12 16:05:05 2024","interfaces":[{"interface":"back","average":{"1min":36.581000000000003,"5min":36.581000000000003,"15min":36.581000000000003},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":635.66399999999999,"5min":635.66399999999999,"15min":635.66399999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":36.253999999999998,"5min":36.253999999999998,"15min":36.253999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":636.28099999999995,"5min":636.28099999999995,"15min":636.28099999999995},"last":1.284}]}]},{"osd":7,"up_from":56,"seq":240518168693,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1764528,"kb_used_data":1459112,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91980624,"statfs":{"total":95995035648,"available":94188158976,"internally_reserved":0,"allocated":1494130688,"data_stored":1493657871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.97499999999999998,"15min":0.98599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.6850000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.92700000000000005,"15min":0.94699999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":2.411}]},{"osd":2,"last update":"Mon Feb 12 16:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.94699999999999995,"15min":1.002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.8580000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.032,"15min":1.042},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.0129999999999999,"15min":1.0640000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.95599999999999996,"15min":1.0609999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":0.73199999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.89600000000000002,"15min":0.97599999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":2.0449999999999999},"last":0.37},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.008,"15min":1.0329999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:04:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0580000000000001,"15min":1.075},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.9329999999999998,"5min":3.9329999999999998,"15min":3.9329999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":1.069,"15min":1.155},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":2.1709999999999998},"last":0.45600000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.0920000000000001,"15min":1.173},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.143,"15min":1.2050000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.6080000000000001,"5min":3.6080000000000001,"15min":3.6080000000000001},"last":0.64400000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.353,"15min":1.4119999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.611,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.802,"5min":1.9490000000000001,"15min":2.0760000000000001},"last":0.51800000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:04:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":1.133,"5min":0.91500000000000004,"15min":0.751},"max":{"1min":1.8400000000000001,"5min":1.9259999999999999,"15min":1.9630000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.423,"15min":1.4239999999999999},"min":{"1min":1.089,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8220000000000001,"5min":1.944,"15min":1.944},"last":0.97199999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:04:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.423,"15min":1.5049999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.4470000000000001,"15min":1.536},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.93000000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:04:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.5069999999999999,"15min":1.5669999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.8979999999999999,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.506,"15min":1.516},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.77,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.67600000000000005}]}]},{"osd":6,"up_from":48,"seq":206158430334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794800,"kb_used_data":485288,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92950352,"statfs":{"total":95995035648,"available":95181160448,"internally_reserved":0,"allocated":496934912,"data_stored":496462095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.0269999999999999,"15min":0.97099999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.806,"5min":3.806,"15min":3.806},"last":0.379},{"interface":"front","average":{"1min":1.085,"5min":1.0349999999999999,"15min":0.94699999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.028,"15min":0.92800000000000005},"min":{"1min":0.84999999999999998,"5min":0.69199999999999995,"15min":0.64300000000000002},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0489999999999999,"15min":1.004},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.077,"15min":0.98799999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.55500000000000005},"max":{"1min":1.6639999999999999,"5min":2.383,"15min":2.383},"last":3.0289999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1279999999999999,"15min":1.0269999999999999},"min":{"1min":0.68200000000000005,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.611,"5min":1.621,"15min":1.621},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0569999999999999,"15min":0.97599999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.28,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.0980000000000001,"15min":1.0580000000000001},"min":{"1min":0.80300000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.623},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0840000000000001,"15min":0.94599999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.5229999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.442},{"interface":"front","average":{"1min":1.276,"5min":1.1599999999999999,"15min":1.006},"min":{"1min":0.90500000000000003,"5min":0.64800000000000002,"15min":0.42299999999999999},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.238,"15min":1.05},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.506,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.256,"5min":1.24,"15min":1.075},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.752,"5min":1.7709999999999999,"15min":1.7949999999999999},"last":0.51300000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:04:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.284,"15min":1.204},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.2709999999999999,"15min":1.2649999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.8400000000000001,"5min":1.889,"15min":1.889},"last":0.68100000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.528,"15min":1.4119999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.0489999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4930000000000001,"15min":1.4350000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.966,"5min":2.0739999999999998,"15min":2.093},"last":0.55600000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.6299999999999999,"15min":1.5009999999999999},"min":{"1min":1.151,"5min":1.151,"15min":1.0589999999999999},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.546,"15min":1.4319999999999999},"min":{"1min":1.1499999999999999,"5min":1.1379999999999999,"15min":0.95899999999999996},"max":{"1min":1.788,"5min":2.125,"15min":2.2469999999999999},"last":0.64200000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.6060000000000001,"15min":1.611},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.9260000000000002,"5min":2.9260000000000002,"15min":2.9260000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.141,"5min":1.524,"15min":1.5429999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.367},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2191536,"kb_used_data":1886120,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91553616,"statfs":{"total":95995035648,"available":93750902784,"internally_reserved":0,"allocated":1931386880,"data_stored":1930914063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:05:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.012,"15min":0.78200000000000003},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.3450000000000006,"5min":8.3450000000000006,"15min":8.3450000000000006},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.96899999999999997,"15min":0.81899999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":5.3659999999999997,"5min":5.3659999999999997,"15min":5.3659999999999997},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:04:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0149999999999999,"15min":0.82099999999999995},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.1710000000000003,"5min":6.1710000000000003,"15min":6.1710000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.95199999999999996,"15min":0.83399999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34999999999999998},"max":{"1min":3.0880000000000001,"5min":3.0880000000000001,"15min":3.0880000000000001},"last":0.24299999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.98499999999999999,"15min":0.871},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.236,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.071,"15min":0.93799999999999994},"min":{"1min":0.59499999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.3009999999999999,"5min":1.9570000000000001,"15min":1.9570000000000001},"last":0.255}]},{"osd":4,"last update":"Mon Feb 12 16:05:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.2330000000000001,"15min":1.0880000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.3159999999999998,"5min":6.3159999999999998,"15min":6.3159999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.2030000000000001,"15min":1.133},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:04:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.347,"15min":1.1879999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.2729999999999997,"5min":6.2729999999999997,"15min":6.2729999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.087,"5min":1.363,"15min":1.254},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.667,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.42299999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.379,"15min":1.252},"min":{"1min":0.83499999999999996,"5min":0.81399999999999995,"15min":0.80500000000000005},"max":{"1min":1.8720000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.363,"5min":1.4570000000000001,"15min":1.3460000000000001},"min":{"1min":0.80800000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.768,"5min":2.573,"15min":2.573},"last":0.35199999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:05:06 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.427,"15min":1.387},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.9399999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.111,"5min":1.337,"15min":1.3300000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.46200000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:04:43 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.514,"15min":1.425},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":6.2270000000000003,"5min":6.2270000000000003,"15min":6.2270000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.341,"5min":1.5900000000000001,"15min":1.5169999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":0.60699999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6279999999999999,"5min":1.6910000000000001,"15min":1.643},"min":{"1min":1.125,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":1.9670000000000001,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.534,"15min":1.532},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":2.113,"5min":2.133,"15min":2.133},"last":0.628}]},{"osd":10,"last update":"Mon Feb 12 16:05:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.536,"15min":1.5580000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":6.3529999999999998,"5min":6.3529999999999998,"15min":6.3529999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.649,"15min":1.7210000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":3.8660000000000001,"5min":3.8660000000000001,"15min":3.8660000000000001},"last":0.51800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1685424,"kb_used_data":1388200,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92059728,"statfs":{"total":95995035648,"available":94269161472,"internally_reserved":0,"allocated":1421516800,"data_stored":1421043983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.81899999999999995,"15min":0.71899999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.694,"5min":1.0229999999999999,"15min":0.81100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":8.266,"5min":8.266,"15min":8.266},"last":4.3470000000000004}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90700000000000003,"15min":0.79200000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.2689999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.215},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85799999999999998,"15min":0.871},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.012,"5min":1.337,"15min":1.3839999999999999},"last":0.35399999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.101,"15min":1.083},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.371},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0289999999999999,"15min":1.012},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.9169999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":2.8690000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1060000000000001,"15min":1.081},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.609,"5min":1.609,"15min":2.0009999999999999},"last":0.313},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.139,"15min":1.1850000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.39700000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.171,"15min":1.1379999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.433},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.228,"15min":1.2170000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.47199999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:05:01 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.2629999999999999,"15min":1.337},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.196,"5min":1.4299999999999999,"15min":1.345},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":0.59799999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.466,"15min":1.411},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.4399999999999999,"15min":1.4850000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":4.4539999999999997,"5min":4.4539999999999997,"15min":4.4539999999999997},"last":0.41699999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:04:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5429999999999999,"15min":1.4910000000000001},"min":{"1min":0.92600000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.3559999999999999,"5min":2.4809999999999999,"15min":2.4809999999999999},"last":0.502},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.4550000000000001,"15min":1.5600000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.873,"5min":2.0779999999999998,"15min":2.4380000000000002},"last":0.81499999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:05:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.4399999999999999,"15min":1.573},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":4.6449999999999996,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.204,"5min":1.3560000000000001,"15min":1.399},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.956,"5min":2.956,"15min":2.956},"last":2.9460000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:04:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.577,"15min":1.5389999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.492,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.627},{"interface":"front","average":{"1min":1.286,"5min":1.4019999999999999,"15min":1.381},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.95,"5min":2.198,"15min":2.198},"last":2.915}]}]},{"osd":2,"up_from":21,"seq":90194313373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1704368,"kb_used_data":1407144,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92040784,"statfs":{"total":95995035648,"available":94249762816,"internally_reserved":0,"allocated":1440915456,"data_stored":1440442639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.86499999999999999,"15min":0.75600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.86199999999999999,"15min":0.76800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.99099999999999999,"15min":0.85099999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":1.849},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0509999999999999,"15min":0.93400000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":1.867}]},{"osd":3,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.048,"15min":0.90900000000000003},"min":{"1min":0.52200000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":5.4889999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.98799999999999999,"15min":0.90700000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.3460000000000001,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.6910000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:05:03 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.1060000000000001,"15min":1.073},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.93100000000000005,"15min":0.91300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3440000000000001,"5min":1.526,"15min":1.526},"last":0.371}]},{"osd":5,"last update":"Mon Feb 12 16:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.097,"15min":1.133},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.208,"5min":1.764,"15min":1.9139999999999999},"last":0.434},{"interface":"front","average":{"1min":0.996,"5min":1.163,"15min":1.081},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.9159999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.145,"15min":1.097},"min":{"1min":0.89500000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.663,"5min":1.746,"15min":1.746},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3100000000000001,"15min":1.222},"min":{"1min":0.995,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.7669999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:05:04 2024","interfaces":[{"interface":"back","average":{"1min":33.115000000000002,"5min":7.726,"15min":3.4670000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":611.79600000000005,"5min":611.79600000000005,"15min":611.79600000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":33.128,"5min":7.6639999999999997,"15min":3.3980000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":611.83900000000006,"5min":611.83900000000006,"15min":611.83900000000006},"last":0.46600000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.4450000000000001,"15min":1.425},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.72,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.503},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.446,"15min":1.4710000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.9430000000000001,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":2.3490000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:04:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3979999999999999,"15min":1.425},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.508,"15min":1.4850000000000001},"min":{"1min":1.04,"5min":1.024,"15min":1.024},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.53800000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.3759999999999999,"15min":1.448},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.694,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.5089999999999999,"15min":1.54},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8839999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":0.70599999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1761328,"kb_used_data":1460008,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91983824,"statfs":{"total":95995035648,"available":94191435776,"internally_reserved":0,"allocated":1495048192,"data_stored":1494575375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.84399999999999997,"15min":0.82599999999999996},"min":{"1min":0.627,"5min":0.434,"15min":0.42999999999999999},"max":{"1min":1.099,"5min":1.2589999999999999,"15min":1.3260000000000001},"last":3.4510000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.96999999999999997,"15min":0.84999999999999998},"min":{"1min":0.65500000000000003,"5min":0.56599999999999995,"15min":0.371},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.77}]},{"osd":2,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93799999999999994,"15min":0.86899999999999999},"min":{"1min":0.69999999999999996,"5min":0.60399999999999998,"15min":0.51900000000000002},"max":{"1min":1.3660000000000001,"5min":1.466,"15min":1.5529999999999999},"last":2.782},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1000000000000001,"15min":0.94399999999999995},"min":{"1min":0.82899999999999996,"5min":0.67100000000000004,"15min":0.44800000000000001},"max":{"1min":1.4410000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.796}]},{"osd":4,"last update":"Mon Feb 12 16:05:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.96399999999999997,"15min":0.98099999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.1160000000000001,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":3.8340000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0800000000000001,"15min":1.052},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.54,"5min":1.54,"15min":1.669},"last":0.495}]},{"osd":5,"last update":"Mon Feb 12 16:04:39 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.238,"15min":1.1699999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.835,"5min":2.835,"15min":2.835},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.0780000000000001,"15min":1.1020000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.157,"5min":3.157,"15min":3.157},"last":3.8809999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.226,"15min":1.1659999999999999},"min":{"1min":0.98499999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.573,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.389,"5min":1.327,"15min":1.2709999999999999},"min":{"1min":1.145,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":1.696,"5min":1.7689999999999999,"15min":1.8120000000000001},"last":1.8540000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.248,"15min":1.2969999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.7250000000000001,"5min":1.946,"15min":1.946},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.353,"5min":1.381,"15min":1.353},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":3.9849999999999999,"5min":3.9849999999999999,"15min":3.9849999999999999},"last":1.8839999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.3180000000000001,"15min":1.3759999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":3.2719999999999998,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":3.948},{"interface":"front","average":{"1min":1.47,"5min":1.4810000000000001,"15min":1.484},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.0899999999999999,"5min":2.242,"15min":2.242},"last":0.52000000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:04:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.5700000000000001,"15min":1.5549999999999999},"min":{"1min":1.1519999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.032,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.601,"5min":1.528,"15min":1.5549999999999999},"min":{"1min":1.2330000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.917,"5min":2.117,"15min":2.117},"last":4.1040000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4330000000000001,"15min":1.476},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":3.2389999999999999,"5min":3.2389999999999999,"15min":3.2389999999999999},"last":4.1799999999999997},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.51,"15min":1.5589999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.8959999999999999,"5min":2.133,"15min":2.133},"last":4.1449999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:04:39 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.506,"15min":1.5},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":3.2999999999999998,"5min":3.2999999999999998,"15min":3.2999999999999998},"last":4},{"interface":"front","average":{"1min":1.468,"5min":1.5840000000000001,"15min":1.538},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.9850000000000001,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":4.2140000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953612,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802672,"kb_used_data":493160,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92942480,"statfs":{"total":95995035648,"available":95173099520,"internally_reserved":0,"allocated":504995840,"data_stored":504523023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0529999999999999,"15min":0.95899999999999996},"min":{"1min":0.76000000000000001,"5min":0.64400000000000002,"15min":0.56999999999999995},"max":{"1min":1.377,"5min":1.5209999999999999,"15min":1.736},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.0620000000000001,"15min":0.94899999999999995},"min":{"1min":0.71999999999999997,"5min":0.69399999999999995,"15min":0.58699999999999997},"max":{"1min":1.484,"5min":1.617,"15min":1.617},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0620000000000001,"15min":0.97499999999999998},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.63900000000000001},"max":{"1min":1.4870000000000001,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.1060000000000001,"15min":0.98199999999999998},"min":{"1min":0.75600000000000001,"5min":0.68799999999999994,"15min":0.51400000000000001},"max":{"1min":1.5049999999999999,"5min":1.7490000000000001,"15min":1.806},"last":1.4730000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.0589999999999999,"15min":0.94499999999999995},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.60299999999999998},"max":{"1min":1.351,"5min":1.593,"15min":1.593},"last":1.8859999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.0109999999999999,"15min":0.96299999999999997},"min":{"1min":0.78000000000000003,"5min":0.68000000000000005,"15min":0.64900000000000002},"max":{"1min":1.3220000000000001,"5min":1.4450000000000001,"15min":1.5549999999999999},"last":0.49299999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:04:05 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1160000000000001,"15min":1.0049999999999999},"min":{"1min":0.79800000000000004,"5min":0.72599999999999998,"15min":0.57599999999999996},"max":{"1min":1.5129999999999999,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.069,"5min":1.103,"15min":0.98799999999999999},"min":{"1min":0.86199999999999999,"5min":0.73299999999999998,"15min":0.55100000000000005},"max":{"1min":1.26,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.372}]},{"osd":5,"last update":"Mon Feb 12 16:04:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.069,"15min":0.95199999999999996},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.383,"5min":1.611,"15min":1.889},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.089,"5min":1.155,"15min":1.0249999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7149999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:04:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.1919999999999999,"15min":1.0720000000000001},"min":{"1min":0.76000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.6020000000000001,"5min":1.653,"15min":1.653},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.101,"5min":1.161,"15min":1.1399999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.4019999999999999,"5min":1.831,"15min":1.9330000000000001},"last":0.44}]},{"osd":7,"last update":"Mon Feb 12 16:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.1120000000000001,"15min":1.1240000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.494,"5min":1.4970000000000001,"15min":1.718},"last":1.46},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.3009999999999999,"15min":1.3200000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.5609999999999999,"5min":1.835,"15min":2.302},"last":0.71999999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:04:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.494,"15min":1.4990000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.948,"5min":2.0699999999999998,"15min":2.1949999999999998},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.427,"15min":1.413},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":2.1699999999999999},"last":0.75600000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:04:10 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.617,"15min":1.571},"min":{"1min":1.2869999999999999,"5min":1.1739999999999999,"15min":1.1659999999999999},"max":{"1min":2.0299999999999998,"5min":2.1309999999999998,"15min":2.173},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.516,"5min":1.5489999999999999,"15min":1.486},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":1.9670000000000001,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.69499999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:05:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.464,"15min":1.4910000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.4199999999999999,"15min":1.5169999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.873,"5min":2.5430000000000001,"15min":2.5430000000000001},"last":0.57699999999999996}]}]},{"osd":5,"up_from":39,"seq":167503724677,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1259696,"kb_used_data":954280,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92485456,"statfs":{"total":95995035648,"available":94705106944,"internally_reserved":0,"allocated":977182720,"data_stored":976709903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:04:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.0329999999999999,"15min":0.93100000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.4239999999999999,"5min":1.71,"15min":1.71},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":0.94599999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.42199999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:04:40 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0820000000000001,"15min":0.98499999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.6480000000000001,"5min":2.6480000000000001,"15min":2.6480000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.004,"15min":0.93999999999999995},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.5389999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:04:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.0880000000000001,"15min":1.0049999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.361,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.503},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.0569999999999999,"15min":1.032},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3180000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.0089999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:04:40 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.0840000000000001,"15min":1.024},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4119999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":3.4319999999999999},{"interface":"front","average":{"1min":0.998,"5min":1.163,"15min":1.1020000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.694,"5min":1.718,"15min":1.726},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:04:40 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.98199999999999998,"15min":0.92600000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.446,"5min":1.889,"15min":1.889},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.123,"15min":1.03},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.4810000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.93300000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.212,"15min":1.1120000000000001},"min":{"1min":0.66600000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.7010000000000001,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.2470000000000001,"15min":1.167},"min":{"1min":0.89400000000000002,"5min":0.68300000000000005,"15min":0.40400000000000003},"max":{"1min":1.7849999999999999,"5min":1.8360000000000001,"15min":1.903},"last":0.69199999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:04:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.2190000000000001,"15min":1.1679999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.5469999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.363,"15min":1.2629999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.435},"max":{"1min":1.776,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":0.77600000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:04:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.4450000000000001,"15min":1.4219999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.0459999999999998,"5min":2.1509999999999998,"15min":2.1949999999999998},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.478,"15min":1.3600000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":2.0950000000000002,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.042}]},{"osd":9,"last update":"Mon Feb 12 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.5580000000000001,"15min":1.5009999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.036,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.6719999999999999,"15min":1.6060000000000001},"min":{"1min":1.3600000000000001,"5min":1.1850000000000001,"15min":1.1619999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":0.82299999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:04:54 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.4350000000000001,"15min":1.47},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.0499999999999998,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.51,"15min":1.595},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.9299999999999999,"5min":2.246,"15min":2.254},"last":1.129}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1426718720,"data_stored":1426718720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1948319744,"data_stored":1948319744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1455751168,"data_stored":1455751168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1506476032,"data_stored":1506476032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":505610240,"data_stored":505610240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":983826432,"data_stored":983826432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":500105216,"data_stored":500105216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1510014976,"data_stored":1510014976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1976303616,"data_stored":1976303616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1462632448,"data_stored":1462632448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":999751680,"data_stored":999751680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1416560640,"data_stored":1416560640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:05:12.014 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T16:05:12.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 26 2024-02-12T16:05:12.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:12.114 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:12.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:12.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:12.243 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:12.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:12.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:12.374 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:12.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:12.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:12.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:12.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:12.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:12.642 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:12.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:12.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 127782 127766 147.863 95 0.0110122 0.00676139 2024-02-12T16:05:12.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:12.770 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:12.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:12 smithi169 bash[19604]: cluster 2024-02-12T16:05:11.553250+0000 mgr.y (mgr.14146) 590 : cluster [DBG] pgmap v665: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-12T16:05:12.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:12.880 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:12 smithi123 bash[19119]: cluster 2024-02-12T16:05:11.553250+0000 mgr.y (mgr.14146) 590 : cluster [DBG] pgmap v665: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-12T16:05:12.897 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:12.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:12.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:12 smithi121 bash[18489]: cluster 2024-02-12T16:05:11.553250+0000 mgr.y (mgr.14146) 590 : cluster [DBG] pgmap v665: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-02-12T16:05:13.000 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:13.024 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:13.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:13.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:13.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:13.297 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:13.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:13.426 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:13.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:13.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:13.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 129316 129300 146.917 95.875 0.00628892 0.00680505 2024-02-12T16:05:13.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:13.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:13.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:13.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:13.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:13.953 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:13.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:14.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:14.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:14.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:14.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:05:14.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:14.227 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:14.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:14.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:14.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:14.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:14.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:14.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:14.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:14.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:14.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:14.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 130809 130793 145.96 93.3125 0.00763946 0.0068495 2024-02-12T16:05:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:14.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:14.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:14.772 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:14.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:14.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:14.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:14.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:14.911 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:14 smithi123 bash[19119]: cluster 2024-02-12T16:05:13.356073+0000 osd.1 (osd.1) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:14.911 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:14 smithi123 bash[19119]: cluster 2024-02-12T16:05:13.356121+0000 osd.1 (osd.1) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:14.911 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:14 smithi123 bash[19119]: cluster 2024-02-12T16:05:13.553680+0000 mgr.y (mgr.14146) 591 : cluster [DBG] pgmap v666: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:05:14.912 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:14 smithi123 bash[19119]: cluster 2024-02-12T16:05:13.634908+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e169: 12 total, 12 up, 12 in 2024-02-12T16:05:14.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:14 smithi121 bash[18489]: cluster 2024-02-12T16:05:13.356073+0000 osd.1 (osd.1) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:14.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:14 smithi121 bash[18489]: cluster 2024-02-12T16:05:13.356121+0000 osd.1 (osd.1) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:14.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:14 smithi121 bash[18489]: cluster 2024-02-12T16:05:13.553680+0000 mgr.y (mgr.14146) 591 : cluster [DBG] pgmap v666: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:05:14.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:14 smithi121 bash[18489]: cluster 2024-02-12T16:05:13.634908+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e169: 12 total, 12 up, 12 in 2024-02-12T16:05:15.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:14 smithi169 bash[19604]: cluster 2024-02-12T16:05:13.356073+0000 osd.1 (osd.1) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:14 smithi169 bash[19604]: cluster 2024-02-12T16:05:13.356121+0000 osd.1 (osd.1) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:14 smithi169 bash[19604]: cluster 2024-02-12T16:05:13.553680+0000 mgr.y (mgr.14146) 591 : cluster [DBG] pgmap v666: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:05:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:14 smithi169 bash[19604]: cluster 2024-02-12T16:05:13.634908+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e169: 12 total, 12 up, 12 in 2024-02-12T16:05:15.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:15.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:15.304 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:15.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:15.444 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:15.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.576 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:15.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:15.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 132267 132251 144.998 91.125 0.0113777 0.00689497 2024-02-12T16:05:15.670 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:15.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:15.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:15.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:15.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:15.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:16.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:16.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:16.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:16.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:16.241 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:16 smithi169 bash[19604]: cluster 2024-02-12T16:05:15.554130+0000 mgr.y (mgr.14146) 592 : cluster [DBG] pgmap v668: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-12T16:05:16.249 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:16.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:16.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:16.377 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:16.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:16.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:16.491 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:16 smithi121 bash[18489]: cluster 2024-02-12T16:05:15.554130+0000 mgr.y (mgr.14146) 592 : cluster [DBG] pgmap v668: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-12T16:05:16.510 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:16.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:16.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:16.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:16 smithi123 bash[19119]: cluster 2024-02-12T16:05:15.554130+0000 mgr.y (mgr.14146) 592 : cluster [DBG] pgmap v668: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-12T16:05:16.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 16 133739 133723 144.084 92 0.00478048 0.00693889 2024-02-12T16:05:16.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:16.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:16.686 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:16.686 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:16.756 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:16.778 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:16.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:16.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:16.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:16.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:17.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:17.045 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:17.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:17.178 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:17.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:17.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:17.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:17.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:17.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:17.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 135192 135176 143.181 90.8125 0.010145 0.00698203 2024-02-12T16:05:17.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:17.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:17.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:17.844 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:17.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:17.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:17.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:18.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:18.112 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:18.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:18.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:18.243 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:18.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:18.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:18.373 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:18.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:18.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:18.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:18.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:18.616 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:18 smithi123 bash[19119]: cluster 2024-02-12T16:05:17.554565+0000 mgr.y (mgr.14146) 593 : cluster [DBG] pgmap v669: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-12T16:05:18.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:18.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:05:18.645826 min lat: 0.000954878 max lat: 0.731447 avg lat: 0.00702983 2024-02-12T16:05:18.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:05:18.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 136553 136537 142.212 85.0625 0.0120854 0.00702983 2024-02-12T16:05:18.650 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:18.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:18.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:18 smithi121 bash[18489]: cluster 2024-02-12T16:05:17.554565+0000 mgr.y (mgr.14146) 593 : cluster [DBG] pgmap v669: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-12T16:05:18.715 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:18.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:18.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:18.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:18.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:18.895 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:18 smithi169 bash[19604]: cluster 2024-02-12T16:05:17.554565+0000 mgr.y (mgr.14146) 593 : cluster [DBG] pgmap v669: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-12T16:05:18.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:18.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:19.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:19.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:19.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:19.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:19.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:19.313 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:19.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:19.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:19.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:19.574 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:19.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 137983 137967 141.346 89.375 0.0116334 0.0070734 2024-02-12T16:05:19.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:19.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:19.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:19.841 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:19.949 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:19.972 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:19.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:20.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:20.111 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:20.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:20.217 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:20.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:20.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:20.385 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:20.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:20.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:20.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:20.618 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:20 smithi121 bash[18489]: cluster 2024-02-12T16:05:19.554993+0000 mgr.y (mgr.14146) 594 : cluster [DBG] pgmap v670: 17 pgs: 17 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:05:20.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:20.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:20.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 139231 139215 140.324 78 0.0256577 0.00712451 2024-02-12T16:05:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:20.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:20.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:20.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:20.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:20.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:20.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:20 smithi123 bash[19119]: cluster 2024-02-12T16:05:19.554993+0000 mgr.y (mgr.14146) 594 : cluster [DBG] pgmap v670: 17 pgs: 17 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:05:20.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:20.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:21.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:21.020 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:20 smithi169 bash[19604]: cluster 2024-02-12T16:05:19.554993+0000 mgr.y (mgr.14146) 594 : cluster [DBG] pgmap v670: 17 pgs: 17 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:05:21.046 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:21.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:21.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:21.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:21.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:21.435 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:21.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:21.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:21.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 16 140538 140522 139.393 81.6875 0.00531607 0.00717156 2024-02-12T16:05:21.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:21.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:21.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:21.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:21.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:21.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:22.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:22.088 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:22.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:22.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:22.220 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:22.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:22.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:22.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:22.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:22.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:22.476 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:22.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:22.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:22.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:22.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 141749 141733 138.398 75.6875 0.0140622 0.00722348 2024-02-12T16:05:22.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:22.754 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:22.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:22 smithi169 bash[19604]: cluster 2024-02-12T16:05:21.555454+0000 mgr.y (mgr.14146) 595 : cluster [DBG] pgmap v671: 17 pgs: 17 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-12T16:05:22.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:22.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:22 smithi123 bash[19119]: cluster 2024-02-12T16:05:21.555454+0000 mgr.y (mgr.14146) 595 : cluster [DBG] pgmap v671: 17 pgs: 17 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-12T16:05:22.889 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:22.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:22.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:22 smithi121 bash[18489]: cluster 2024-02-12T16:05:21.555454+0000 mgr.y (mgr.14146) 595 : cluster [DBG] pgmap v671: 17 pgs: 17 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-12T16:05:22.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:23.025 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:23.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:23.161 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:23.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:23.286 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:23.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:23.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:23.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:23.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:23.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 143127 143111 137.593 86.125 0.0105631 0.00726561 2024-02-12T16:05:23.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:23.697 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:23.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:23.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:23.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:23.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:23.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:23.976 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:05:23.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:05:24.030 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:05:24.055 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:05:24.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:24.090 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:05:24.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:24.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:05:24.154 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:05:24.182 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:05:24.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:24.217 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:05:24.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:24.246 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:05:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:24.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:24.273 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:05:24.312 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.327 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.327 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.327 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:05:24.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:24.375 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:24.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:24.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:24.499 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:24.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:24.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:24.644 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:24.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:24.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 144475 144459 136.785 84.25 0.010281 0.00730876 2024-02-12T16:05:24.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:24 smithi121 bash[18489]: cluster 2024-02-12T16:05:23.555838+0000 mgr.y (mgr.14146) 596 : cluster [DBG] pgmap v672: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:05:24.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:24 smithi121 bash[18489]: audit 2024-02-12T16:05:23.934455+0000 mon.b (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.121:0/2143905945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:24.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:24.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:24.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:24.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:24 smithi123 bash[19119]: cluster 2024-02-12T16:05:23.555838+0000 mgr.y (mgr.14146) 596 : cluster [DBG] pgmap v672: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:05:24.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:24 smithi123 bash[19119]: audit 2024-02-12T16:05:23.934455+0000 mon.b (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.121:0/2143905945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:24.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:24.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:24.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:25.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:25.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:24 smithi169 bash[19604]: cluster 2024-02-12T16:05:23.555838+0000 mgr.y (mgr.14146) 596 : cluster [DBG] pgmap v672: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:05:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:24 smithi169 bash[19604]: audit 2024-02-12T16:05:23.934455+0000 mon.b (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.121:0/2143905945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:25.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:25.091 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:05:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:05:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:145016 2.3 e169/147 2:c71d3ae1:::benchmark_data_smithi169_42458_object145015:head v 169'9165, mlcod=169'9163)", 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:05:25.079750+0000", 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.003891865, 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039212309999999998, 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079750+0000", 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079750+0000", 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079748+0000", 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0884000000000002e-05 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079809+0000", 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.942e-05 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079828+0000", 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6690000000000006e-06 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079836+0000", 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0237e-05 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079886+0000", 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8464000000000001e-05 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:05:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:25.079915+0000", 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000164819 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 1 2024-02-12T16:05:25.095 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:05:25.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:25.180 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:25.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:25.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:25.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:25.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:25.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:25.574 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:25.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 145753 145737 135.936 79.875 0.0087425 0.00735464 2024-02-12T16:05:25.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:25.704 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:25.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:25.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:25.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:25.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:25.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:26.072 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:26.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:26.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:26.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:26.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:26.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 2 pg_num to 26 2024-02-12T16:05:26.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:05:26.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:26.348 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:26 smithi123 bash[19119]: audit 2024-02-12T16:05:25.295174+0000 mon.a (mon.0) 910 : audit [INF] from='client.? v1:172.21.15.121:0/2436247993' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:05:26.348 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:26 smithi123 bash[19119]: cluster 2024-02-12T16:05:25.346981+0000 osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:26.348 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:26 smithi123 bash[19119]: cluster 2024-02-12T16:05:25.347051+0000 osd.7 (osd.7) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:26.349 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:26 smithi123 bash[19119]: cluster 2024-02-12T16:05:25.556225+0000 mgr.y (mgr.14146) 597 : cluster [DBG] pgmap v673: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-12T16:05:26.366 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:26.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:26.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:26.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:26.500 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:26 smithi169 bash[19604]: audit 2024-02-12T16:05:25.295174+0000 mon.a (mon.0) 910 : audit [INF] from='client.? v1:172.21.15.121:0/2436247993' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:05:26.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:26 smithi169 bash[19604]: cluster 2024-02-12T16:05:25.346981+0000 osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:26.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:26 smithi169 bash[19604]: cluster 2024-02-12T16:05:25.347051+0000 osd.7 (osd.7) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:26.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:26 smithi169 bash[19604]: cluster 2024-02-12T16:05:25.556225+0000 mgr.y (mgr.14146) 597 : cluster [DBG] pgmap v673: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-12T16:05:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:26.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:26.503 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:26 smithi121 bash[18489]: audit 2024-02-12T16:05:25.295174+0000 mon.a (mon.0) 910 : audit [INF] from='client.? v1:172.21.15.121:0/2436247993' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:05:26.503 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:26 smithi121 bash[18489]: cluster 2024-02-12T16:05:25.346981+0000 osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:26.503 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:26 smithi121 bash[18489]: cluster 2024-02-12T16:05:25.347051+0000 osd.7 (osd.7) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:26.503 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:26 smithi121 bash[18489]: cluster 2024-02-12T16:05:25.556225+0000 mgr.y (mgr.14146) 597 : cluster [DBG] pgmap v673: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-02-12T16:05:26.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:26.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:26.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:26.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 16 147066 147050 135.143 82.0625 0.00565356 0.00739816 2024-02-12T16:05:26.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:26.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:26.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:26.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:26.897 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:26.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:27.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:27.037 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:27.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:27.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:27.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:27.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:27.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:27.292 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:27 smithi121 bash[18489]: audit 2024-02-12T16:05:26.236099+0000 mon.a (mon.0) 911 : audit [INF] from='client.? v1:172.21.15.121:0/2436247993' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:05:27.292 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:27 smithi121 bash[18489]: cluster 2024-02-12T16:05:26.236178+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e170: 12 total, 12 up, 12 in 2024-02-12T16:05:27.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:27.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:27.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:27.460 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:27.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:27 smithi169 bash[19604]: audit 2024-02-12T16:05:26.236099+0000 mon.a (mon.0) 911 : audit [INF] from='client.? v1:172.21.15.121:0/2436247993' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:05:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:27 smithi169 bash[19604]: cluster 2024-02-12T16:05:26.236178+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e170: 12 total, 12 up, 12 in 2024-02-12T16:05:27.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:27.594 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:27.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:27.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:27 smithi123 bash[19119]: audit 2024-02-12T16:05:26.236099+0000 mon.a (mon.0) 911 : audit [INF] from='client.? v1:172.21.15.121:0/2436247993' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:05:27.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:27 smithi123 bash[19119]: cluster 2024-02-12T16:05:26.236178+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e170: 12 total, 12 up, 12 in 2024-02-12T16:05:27.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 148357 148341 134.354 80.6875 0.0246906 0.00744146 2024-02-12T16:05:27.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:27.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:27.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:27.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:27.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:27.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:27.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:28.104 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:28.127 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:28.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:28.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:28.267 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:05:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:28.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:05:28.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:28.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:28 smithi123 bash[19119]: audit 2024-02-12T16:05:27.457542+0000 mon.a (mon.0) 913 : audit [INF] from='client.? v1:172.21.15.121:0/1847631883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:05:28.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:28 smithi123 bash[19119]: cluster 2024-02-12T16:05:27.556725+0000 mgr.y (mgr.14146) 598 : cluster [DBG] pgmap v675: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-12T16:05:28.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:28 smithi123 bash[19119]: audit 2024-02-12T16:05:27.557896+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:05:28.405 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:28.508 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:28.533 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:28 smithi169 bash[19604]: audit 2024-02-12T16:05:27.457542+0000 mon.a (mon.0) 913 : audit [INF] from='client.? v1:172.21.15.121:0/1847631883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:05:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:28 smithi169 bash[19604]: cluster 2024-02-12T16:05:27.556725+0000 mgr.y (mgr.14146) 598 : cluster [DBG] pgmap v675: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-12T16:05:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:28 smithi169 bash[19604]: audit 2024-02-12T16:05:27.557896+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:05:28.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:28.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 16 149178 149162 133.168 51.3125 0.0121814 0.00746624 2024-02-12T16:05:28.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:28 smithi121 bash[18489]: audit 2024-02-12T16:05:27.457542+0000 mon.a (mon.0) 913 : audit [INF] from='client.? v1:172.21.15.121:0/1847631883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:05:28.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:28 smithi121 bash[18489]: cluster 2024-02-12T16:05:27.556725+0000 mgr.y (mgr.14146) 598 : cluster [DBG] pgmap v675: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-12T16:05:28.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:28 smithi121 bash[18489]: audit 2024-02-12T16:05:27.557896+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:05:28.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:28.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:28.813 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:28.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:28.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:28.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:29.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:29.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:29.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:29.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:29.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:29.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:29.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:29.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:29 smithi169 bash[19604]: cluster 2024-02-12T16:05:27.764978+0000 osd.10 (osd.10) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:29.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:29 smithi169 bash[19604]: cluster 2024-02-12T16:05:27.765079+0000 osd.10 (osd.10) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:29.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:29 smithi169 bash[19604]: cluster 2024-02-12T16:05:28.233054+0000 mon.a (mon.0) 915 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:29.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:29 smithi169 bash[19604]: audit 2024-02-12T16:05:28.241028+0000 mon.a (mon.0) 916 : audit [INF] from='client.? v1:172.21.15.121:0/1847631883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:05:29.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:29 smithi169 bash[19604]: audit 2024-02-12T16:05:28.241099+0000 mon.a (mon.0) 917 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:05:29.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:29 smithi169 bash[19604]: cluster 2024-02-12T16:05:28.241150+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e171: 12 total, 12 up, 12 in 2024-02-12T16:05:29.370 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:29.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:29.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:29.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:29.498 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:29 smithi123 bash[19119]: cluster 2024-02-12T16:05:27.764978+0000 osd.10 (osd.10) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:29.498 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:29 smithi123 bash[19119]: cluster 2024-02-12T16:05:27.765079+0000 osd.10 (osd.10) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:29.498 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:29 smithi123 bash[19119]: cluster 2024-02-12T16:05:28.233054+0000 mon.a (mon.0) 915 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:29.498 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:29 smithi123 bash[19119]: audit 2024-02-12T16:05:28.241028+0000 mon.a (mon.0) 916 : audit [INF] from='client.? v1:172.21.15.121:0/1847631883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:05:29.498 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:29 smithi123 bash[19119]: audit 2024-02-12T16:05:28.241099+0000 mon.a (mon.0) 917 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:05:29.498 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:29 smithi123 bash[19119]: cluster 2024-02-12T16:05:28.241150+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e171: 12 total, 12 up, 12 in 2024-02-12T16:05:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:29.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:29.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:29.607 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:29 smithi121 bash[18489]: cluster 2024-02-12T16:05:27.764978+0000 osd.10 (osd.10) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:29.607 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:29 smithi121 bash[18489]: cluster 2024-02-12T16:05:27.765079+0000 osd.10 (osd.10) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:29.607 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:29 smithi121 bash[18489]: cluster 2024-02-12T16:05:28.233054+0000 mon.a (mon.0) 915 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:29.608 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:29 smithi121 bash[18489]: audit 2024-02-12T16:05:28.241028+0000 mon.a (mon.0) 916 : audit [INF] from='client.? v1:172.21.15.121:0/1847631883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:05:29.608 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:29 smithi121 bash[18489]: audit 2024-02-12T16:05:28.241099+0000 mon.a (mon.0) 917 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:05:29.608 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:29 smithi121 bash[18489]: cluster 2024-02-12T16:05:28.241150+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e171: 12 total, 12 up, 12 in 2024-02-12T16:05:29.625 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:29.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:29.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 149804 149788 131.843 39.125 0.00615065 0.00758279 2024-02-12T16:05:29.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:29.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:29.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:29.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:29.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:29.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:29.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:30.019 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:30.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:30.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:30.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:30.278 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:30.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:30.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:30 smithi123 bash[19119]: cluster 2024-02-12T16:05:29.247943+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e172: 12 total, 12 up, 12 in 2024-02-12T16:05:30.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:30 smithi123 bash[19119]: cluster 2024-02-12T16:05:29.557140+0000 mgr.y (mgr.14146) 599 : cluster [DBG] pgmap v678: 27 pgs: 4 peering, 13 active+clean, 10 unknown; 7.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:05:30.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:30 smithi121 bash[18489]: cluster 2024-02-12T16:05:29.247943+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e172: 12 total, 12 up, 12 in 2024-02-12T16:05:30.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:30 smithi121 bash[18489]: cluster 2024-02-12T16:05:29.557140+0000 mgr.y (mgr.14146) 599 : cluster [DBG] pgmap v678: 27 pgs: 4 peering, 13 active+clean, 10 unknown; 7.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:05:30.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:30.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:30.542 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:30.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:30 smithi169 bash[19604]: cluster 2024-02-12T16:05:29.247943+0000 mon.a (mon.0) 919 : cluster [DBG] osdmap e172: 12 total, 12 up, 12 in 2024-02-12T16:05:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:30 smithi169 bash[19604]: cluster 2024-02-12T16:05:29.557140+0000 mgr.y (mgr.14146) 599 : cluster [DBG] pgmap v678: 27 pgs: 4 peering, 13 active+clean, 10 unknown; 7.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:05:30.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:30.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 151591 151575 131.563 111.688 0.00261902 0.00759971 2024-02-12T16:05:30.670 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:30.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:30.801 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:30.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:30.941 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:30.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:31.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:31.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:31.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:31.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:31.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:31.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:31.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:05:31.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:31.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:31.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:31.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:31.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:31.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:31.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:31.629 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:31.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:31.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 153309 153293 131.231 107.375 0.0207026 0.00761882 2024-02-12T16:05:31.718 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:31.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:31.763 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:31.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:31.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:31.902 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:31.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:32.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:32.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:32.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:32.170 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:32.226 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:05:32.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T16:05:32.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:32.314 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:32.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:32 smithi121 bash[18489]: cluster 2024-02-12T16:05:30.374875+0000 osd.11 (osd.11) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:32.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:32 smithi121 bash[18489]: cluster 2024-02-12T16:05:30.374996+0000 osd.11 (osd.11) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:32.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:32 smithi121 bash[18489]: cluster 2024-02-12T16:05:31.258287+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e173: 12 total, 12 up, 12 in 2024-02-12T16:05:32.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:32 smithi121 bash[18489]: cluster 2024-02-12T16:05:31.557556+0000 mgr.y (mgr.14146) 600 : cluster [DBG] pgmap v680: 27 pgs: 4 peering, 14 active+clean, 9 unknown; 8.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:05:32.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:32.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:32 smithi169 bash[19604]: cluster 2024-02-12T16:05:30.374875+0000 osd.11 (osd.11) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:32 smithi169 bash[19604]: cluster 2024-02-12T16:05:30.374996+0000 osd.11 (osd.11) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:32 smithi169 bash[19604]: cluster 2024-02-12T16:05:31.258287+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e173: 12 total, 12 up, 12 in 2024-02-12T16:05:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:32 smithi169 bash[19604]: cluster 2024-02-12T16:05:31.557556+0000 mgr.y (mgr.14146) 600 : cluster [DBG] pgmap v680: 27 pgs: 4 peering, 14 active+clean, 9 unknown; 8.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:05:32.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:32.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:32.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:32 smithi123 bash[19119]: cluster 2024-02-12T16:05:30.374875+0000 osd.11 (osd.11) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:32.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:32 smithi123 bash[19119]: cluster 2024-02-12T16:05:30.374996+0000 osd.11 (osd.11) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:32.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:32 smithi123 bash[19119]: cluster 2024-02-12T16:05:31.258287+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e173: 12 total, 12 up, 12 in 2024-02-12T16:05:32.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:32 smithi123 bash[19119]: cluster 2024-02-12T16:05:31.557556+0000 mgr.y (mgr.14146) 600 : cluster [DBG] pgmap v680: 27 pgs: 4 peering, 14 active+clean, 9 unknown; 8.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:05:32.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 155043 155027 130.922 108.375 0.00827142 0.00763664 2024-02-12T16:05:32.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:32.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:32.747 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:32.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:32.841 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:32.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:32.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:32.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:33.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:33.124 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:33.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:33.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:33.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:33.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:33.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:33.405 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:33.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:33.511 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:33.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:33.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:33.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 15 156384 156369 130.295 83.875 0.00284332 0.00767276 2024-02-12T16:05:33.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:33.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:33.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:33.798 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:33.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:33.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:33.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:33.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:34.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:34.074 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:34.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:34.184 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:34.211 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:34.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:34.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:34.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:34 smithi169 bash[19604]: cluster 2024-02-12T16:05:33.272232+0000 osd.8 (osd.8) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:34.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:34 smithi169 bash[19604]: cluster 2024-02-12T16:05:33.272276+0000 osd.8 (osd.8) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:34.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:34 smithi169 bash[19604]: cluster 2024-02-12T16:05:33.324090+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e174: 12 total, 12 up, 12 in 2024-02-12T16:05:34.339 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:34 smithi169 bash[19604]: cluster 2024-02-12T16:05:33.558077+0000 mgr.y (mgr.14146) 601 : cluster [DBG] pgmap v682: 27 pgs: 2 peering, 23 active+clean, 2 unknown; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s, 0 objects/s recovering 2024-02-12T16:05:34.347 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:34.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:34.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:34.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:34.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:34.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:34.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:34 smithi123 bash[19119]: cluster 2024-02-12T16:05:33.272232+0000 osd.8 (osd.8) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:34.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:34 smithi123 bash[19119]: cluster 2024-02-12T16:05:33.272276+0000 osd.8 (osd.8) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:34.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:34 smithi123 bash[19119]: cluster 2024-02-12T16:05:33.324090+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e174: 12 total, 12 up, 12 in 2024-02-12T16:05:34.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:34 smithi123 bash[19119]: cluster 2024-02-12T16:05:33.558077+0000 mgr.y (mgr.14146) 601 : cluster [DBG] pgmap v682: 27 pgs: 2 peering, 23 active+clean, 2 unknown; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s, 0 objects/s recovering 2024-02-12T16:05:34.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:34.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:34.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 158095 158079 129.987 106.875 0.0034284 0.00769148 2024-02-12T16:05:34.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:34 smithi121 bash[18489]: cluster 2024-02-12T16:05:33.272232+0000 osd.8 (osd.8) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:34.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:34 smithi121 bash[18489]: cluster 2024-02-12T16:05:33.272276+0000 osd.8 (osd.8) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:34.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:34 smithi121 bash[18489]: cluster 2024-02-12T16:05:33.324090+0000 mon.a (mon.0) 921 : cluster [DBG] osdmap e174: 12 total, 12 up, 12 in 2024-02-12T16:05:34.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:34 smithi121 bash[18489]: cluster 2024-02-12T16:05:33.558077+0000 mgr.y (mgr.14146) 601 : cluster [DBG] pgmap v682: 27 pgs: 2 peering, 23 active+clean, 2 unknown; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s, 0 objects/s recovering 2024-02-12T16:05:34.719 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:34.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:34.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:34.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:34.881 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:34.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:34.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:35.015 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:35.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:35.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:35.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:35.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:35.422 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:35.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:35.564 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:35.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 16 159770 159754 129.658 104.688 0.00393064 0.00771091 2024-02-12T16:05:35.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:35.699 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:35.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:35.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:35.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:35.968 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:35.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:36.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:36.107 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:36.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:36.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:36.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:36.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:36.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:36.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:36 smithi169 bash[19604]: cluster 2024-02-12T16:05:35.558533+0000 mgr.y (mgr.14146) 602 : cluster [DBG] pgmap v683: 27 pgs: 27 active+clean; 9.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 259 MiB/s wr, 4.14k op/s; 10 KiB/s, 0 objects/s recovering 2024-02-12T16:05:36.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:36 smithi169 bash[19604]: audit 2024-02-12T16:05:35.559120+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:05:36.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:36.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:36.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:36.516 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:36.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:36.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:36.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:36 smithi123 bash[19119]: cluster 2024-02-12T16:05:35.558533+0000 mgr.y (mgr.14146) 602 : cluster [DBG] pgmap v683: 27 pgs: 27 active+clean; 9.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 259 MiB/s wr, 4.14k op/s; 10 KiB/s, 0 objects/s recovering 2024-02-12T16:05:36.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:36 smithi123 bash[19119]: audit 2024-02-12T16:05:35.559120+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:05:36.644 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:36.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:36.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 161295 161279 129.218 95.3125 0.00347911 0.0077227 2024-02-12T16:05:36.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:36 smithi121 bash[18489]: cluster 2024-02-12T16:05:35.558533+0000 mgr.y (mgr.14146) 602 : cluster [DBG] pgmap v683: 27 pgs: 27 active+clean; 9.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 259 MiB/s wr, 4.14k op/s; 10 KiB/s, 0 objects/s recovering 2024-02-12T16:05:36.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:36 smithi121 bash[18489]: audit 2024-02-12T16:05:35.559120+0000 mon.a (mon.0) 922 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:05:36.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:36.760 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:36.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:36.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:36.789 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:36.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:36.907 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:36.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:37.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:37.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:37.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:37.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:37.184 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:37.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:37.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:37.319 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:37.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:37.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:37 smithi121 bash[18489]: audit 2024-02-12T16:05:36.244107+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:05:37.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:37 smithi121 bash[18489]: cluster 2024-02-12T16:05:36.244356+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e175: 12 total, 12 up, 12 in 2024-02-12T16:05:37.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:37.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:37 smithi123 bash[19119]: audit 2024-02-12T16:05:36.244107+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:05:37.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:37 smithi123 bash[19119]: cluster 2024-02-12T16:05:36.244356+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e175: 12 total, 12 up, 12 in 2024-02-12T16:05:37.459 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:37.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:37 smithi169 bash[19604]: audit 2024-02-12T16:05:36.244107+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:05:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:37 smithi169 bash[19604]: cluster 2024-02-12T16:05:36.244356+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e175: 12 total, 12 up, 12 in 2024-02-12T16:05:37.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:37.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:37.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:37.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 161361 161345 127.634 4.125 0.00424086 0.00772136 2024-02-12T16:05:37.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:37.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:37.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:37.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:37.874 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:37.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:37.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:38.014 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:38.108 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:05:38.108 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:05:38.108 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:05:38.108 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:05:38.108 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:05:38.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:38.149 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:38.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:38.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:38.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:38.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:38 smithi123 bash[19119]: cluster 2024-02-12T16:05:37.255700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e176: 12 total, 12 up, 12 in 2024-02-12T16:05:38.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:38 smithi123 bash[19119]: cluster 2024-02-12T16:05:37.559418+0000 mgr.y (mgr.14146) 603 : cluster [DBG] pgmap v686: 27 pgs: 27 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 295 MiB/s wr, 4.72k op/s; 11 KiB/s, 0 objects/s recovering 2024-02-12T16:05:38.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:38 smithi123 bash[19119]: audit 2024-02-12T16:05:37.560889+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:05:38.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:38.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:38.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:38 smithi169 bash[19604]: cluster 2024-02-12T16:05:37.255700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e176: 12 total, 12 up, 12 in 2024-02-12T16:05:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:38 smithi169 bash[19604]: cluster 2024-02-12T16:05:37.559418+0000 mgr.y (mgr.14146) 603 : cluster [DBG] pgmap v686: 27 pgs: 27 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 295 MiB/s wr, 4.72k op/s; 11 KiB/s, 0 objects/s recovering 2024-02-12T16:05:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:38 smithi169 bash[19604]: audit 2024-02-12T16:05:37.560889+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:05:38.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:38.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:05:38.647644 min lat: 0.000954878 max lat: 1.07555 avg lat: 0.00772136 2024-02-12T16:05:38.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:05:38.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 161361 161345 126.039 0 - 0.00772136 2024-02-12T16:05:38.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:38.693 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:38 smithi121 bash[18489]: cluster 2024-02-12T16:05:37.255700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e176: 12 total, 12 up, 12 in 2024-02-12T16:05:38.693 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:38 smithi121 bash[18489]: cluster 2024-02-12T16:05:37.559418+0000 mgr.y (mgr.14146) 603 : cluster [DBG] pgmap v686: 27 pgs: 27 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 295 MiB/s wr, 4.72k op/s; 11 KiB/s, 0 objects/s recovering 2024-02-12T16:05:38.693 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:38 smithi121 bash[18489]: audit 2024-02-12T16:05:37.560889+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:05:38.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:38.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:38.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:38.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:38.991 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:38.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:39.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:39.125 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:39.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:39.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:39.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:39.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:39.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:05:39.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:39.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:39.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:39.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:39.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:39 smithi169 bash[19604]: cluster 2024-02-12T16:05:37.604325+0000 osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:39.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:39 smithi169 bash[19604]: cluster 2024-02-12T16:05:37.604434+0000 osd.4 (osd.4) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:39.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:39 smithi169 bash[19604]: audit 2024-02-12T16:05:38.258549+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:05:39.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:39 smithi169 bash[19604]: cluster 2024-02-12T16:05:38.258748+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e177: 12 total, 12 up, 12 in 2024-02-12T16:05:39.489 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:39 smithi169 bash[19604]: cluster 2024-02-12T16:05:39.260195+0000 mon.a (mon.0) 929 : cluster [DBG] osdmap e178: 12 total, 12 up, 12 in 2024-02-12T16:05:39.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:39.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:39.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:05:39.572 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:05:39.593 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:05:39.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:39.614 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:39 smithi121 bash[18489]: cluster 2024-02-12T16:05:37.604325+0000 osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:39.614 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:39 smithi121 bash[18489]: cluster 2024-02-12T16:05:37.604434+0000 osd.4 (osd.4) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:39.614 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:39 smithi121 bash[18489]: audit 2024-02-12T16:05:38.258549+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:05:39.614 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:39 smithi121 bash[18489]: cluster 2024-02-12T16:05:38.258748+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e177: 12 total, 12 up, 12 in 2024-02-12T16:05:39.614 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:39 smithi121 bash[18489]: cluster 2024-02-12T16:05:39.260195+0000 mon.a (mon.0) 929 : cluster [DBG] osdmap e178: 12 total, 12 up, 12 in 2024-02-12T16:05:39.621 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:05:39.621 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:39 smithi123 bash[19119]: cluster 2024-02-12T16:05:37.604325+0000 osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:39.621 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:39 smithi123 bash[19119]: cluster 2024-02-12T16:05:37.604434+0000 osd.4 (osd.4) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:39.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:39 smithi123 bash[19119]: audit 2024-02-12T16:05:38.258549+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:05:39.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:39 smithi123 bash[19119]: cluster 2024-02-12T16:05:38.258748+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e177: 12 total, 12 up, 12 in 2024-02-12T16:05:39.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:39 smithi123 bash[19119]: cluster 2024-02-12T16:05:39.260195+0000 mon.a (mon.0) 929 : cluster [DBG] osdmap e178: 12 total, 12 up, 12 in 2024-02-12T16:05:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:39.641 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:05:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:39.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:39.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 161931 161915 124.922 17.8125 0.00294176 0.00796755 2024-02-12T16:05:39.666 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:05:39.689 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:05:39.715 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:05:39.741 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:05:39.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:39.762 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:05:39.771 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:39.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:39.787 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:05:39.807 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:05:39.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.809 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:05:39.874 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:39.908 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:39.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:40.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:40.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:40.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:40.286 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:40.325 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:40.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:40.454 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:40 smithi169 bash[19604]: cluster 2024-02-12T16:05:39.168386+0000 osd.5 (osd.5) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:40 smithi169 bash[19604]: cluster 2024-02-12T16:05:39.168456+0000 osd.5 (osd.5) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:40 smithi169 bash[19604]: cluster 2024-02-12T16:05:39.410037+0000 osd.3 (osd.3) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:40 smithi169 bash[19604]: cluster 2024-02-12T16:05:39.410089+0000 osd.3 (osd.3) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:40 smithi169 bash[19604]: audit 2024-02-12T16:05:39.528992+0000 mon.b (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.121:0/2567134961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:40 smithi169 bash[19604]: cluster 2024-02-12T16:05:39.560173+0000 mgr.y (mgr.14146) 604 : cluster [DBG] pgmap v689: 27 pgs: 1 activating+remapped, 26 active+clean; 9.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 5055/321712 objects degraded (1.571%); 5055/321712 objects misplaced (1.571%) 2024-02-12T16:05:40.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:40.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:40 smithi123 bash[19119]: cluster 2024-02-12T16:05:39.168386+0000 osd.5 (osd.5) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:40 smithi123 bash[19119]: cluster 2024-02-12T16:05:39.168456+0000 osd.5 (osd.5) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:40 smithi123 bash[19119]: cluster 2024-02-12T16:05:39.410037+0000 osd.3 (osd.3) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:40 smithi123 bash[19119]: cluster 2024-02-12T16:05:39.410089+0000 osd.3 (osd.3) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:40 smithi123 bash[19119]: audit 2024-02-12T16:05:39.528992+0000 mon.b (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.121:0/2567134961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:40 smithi123 bash[19119]: cluster 2024-02-12T16:05:39.560173+0000 mgr.y (mgr.14146) 604 : cluster [DBG] pgmap v689: 27 pgs: 1 activating+remapped, 26 active+clean; 9.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 5055/321712 objects degraded (1.571%); 5055/321712 objects misplaced (1.571%) 2024-02-12T16:05:40.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 161969 161953 123.428 2.375 0.00742388 0.00796682 2024-02-12T16:05:40.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:40 smithi121 bash[18489]: cluster 2024-02-12T16:05:39.168386+0000 osd.5 (osd.5) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:40.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:40 smithi121 bash[18489]: cluster 2024-02-12T16:05:39.168456+0000 osd.5 (osd.5) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:40.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:40 smithi121 bash[18489]: cluster 2024-02-12T16:05:39.410037+0000 osd.3 (osd.3) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:40.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:40 smithi121 bash[18489]: cluster 2024-02-12T16:05:39.410089+0000 osd.3 (osd.3) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:40.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:40 smithi121 bash[18489]: audit 2024-02-12T16:05:39.528992+0000 mon.b (mon.2) 101 : audit [DBG] from='client.? v1:172.21.15.121:0/2567134961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:40.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:40 smithi121 bash[18489]: cluster 2024-02-12T16:05:39.560173+0000 mgr.y (mgr.14146) 604 : cluster [DBG] pgmap v689: 27 pgs: 1 activating+remapped, 26 active+clean; 9.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 5055/321712 objects degraded (1.571%); 5055/321712 objects misplaced (1.571%) 2024-02-12T16:05:40.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:40.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:40.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:40.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:40.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:40.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:40.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:41.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:41.127 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:41.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:41.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:41.267 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:41.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:41.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:41 smithi123 bash[19119]: cluster 2024-02-12T16:05:40.263662+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e179: 12 total, 12 up, 12 in 2024-02-12T16:05:41.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:41.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:41 smithi169 bash[19604]: cluster 2024-02-12T16:05:40.263662+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e179: 12 total, 12 up, 12 in 2024-02-12T16:05:41.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:41.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:41.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:41.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:41.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:41.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:41.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:41 smithi121 bash[18489]: cluster 2024-02-12T16:05:40.263662+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e179: 12 total, 12 up, 12 in 2024-02-12T16:05:41.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:41.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 163306 163290 122.947 83.5625 0.00514388 0.00813201 2024-02-12T16:05:41.691 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:41.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:41.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:41.821 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:41.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:41.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:41.948 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:41.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:42.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:42.088 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:42.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:42.191 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:42.218 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:42.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:42.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:42.328 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:42 smithi123 bash[19119]: cluster 2024-02-12T16:05:41.560676+0000 mgr.y (mgr.14146) 605 : cluster [DBG] pgmap v691: 27 pgs: 1 active+recovering+remapped, 1 activating+remapped, 25 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 605 op/s; 5055/323268 objects degraded (1.564%); 15001/323268 objects misplaced (4.640%); 89 KiB/s, 1 objects/s recovering 2024-02-12T16:05:42.346 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:42.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:42.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:42.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:42.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:42.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:42 smithi169 bash[19604]: cluster 2024-02-12T16:05:41.560676+0000 mgr.y (mgr.14146) 605 : cluster [DBG] pgmap v691: 27 pgs: 1 active+recovering+remapped, 1 activating+remapped, 25 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 605 op/s; 5055/323268 objects degraded (1.564%); 15001/323268 objects misplaced (4.640%); 89 KiB/s, 1 objects/s recovering 2024-02-12T16:05:42.576 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:42.611 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:42.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:42.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 164674 164658 122.501 85.5 0.00900832 0.00816145 2024-02-12T16:05:42.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:42 smithi121 bash[18489]: cluster 2024-02-12T16:05:41.560676+0000 mgr.y (mgr.14146) 605 : cluster [DBG] pgmap v691: 27 pgs: 1 active+recovering+remapped, 1 activating+remapped, 25 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 605 op/s; 5055/323268 objects degraded (1.564%); 15001/323268 objects misplaced (4.640%); 89 KiB/s, 1 objects/s recovering 2024-02-12T16:05:42.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:42.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:42.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:42.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:42.883 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:42.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:42.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:43.013 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:43.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:43.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:43.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:43.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:43.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:43.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:43.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:43.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:43.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:43.593 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:43.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:05:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:43.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:43.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 166062 166046 122.081 86.75 0.00577027 0.00818949 2024-02-12T16:05:43.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:43.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:43.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:43.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:43.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:44.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:44.046 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:44.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:44.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:44.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:44.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:44.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:44.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:44.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:44.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:44.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:44.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:44.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:44.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:44.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:44.640 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:05:44.646 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T16:05:44.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 167446 167430 121.667 86.5 0.0119173 0.0082176 2024-02-12T16:05:44.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:44.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:44 smithi169 bash[19604]: cluster 2024-02-12T16:05:43.561420+0000 mgr.y (mgr.14146) 606 : cluster [DBG] pgmap v692: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s; 19952/325778 objects misplaced (6.124%); 618 KiB/s, 9 objects/s recovering 2024-02-12T16:05:44.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:44 smithi169 bash[19604]: audit 2024-02-12T16:05:43.740005+0000 mon.a (mon.0) 931 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:05:44.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:44 smithi169 bash[19604]: audit 2024-02-12T16:05:43.743770+0000 mon.c (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.121:0/3951503911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:05:44.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:44 smithi169 bash[19604]: audit 2024-02-12T16:05:43.919731+0000 mon.a (mon.0) 932 : audit [INF] from='client.? v1:172.21.15.121:0/693172070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:05:44.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:44.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:44.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:44.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:44 smithi121 bash[18489]: cluster 2024-02-12T16:05:43.561420+0000 mgr.y (mgr.14146) 606 : cluster [DBG] pgmap v692: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s; 19952/325778 objects misplaced (6.124%); 618 KiB/s, 9 objects/s recovering 2024-02-12T16:05:44.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:44 smithi121 bash[18489]: audit 2024-02-12T16:05:43.740005+0000 mon.a (mon.0) 931 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:05:44.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:44 smithi121 bash[18489]: audit 2024-02-12T16:05:43.743770+0000 mon.c (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.121:0/3951503911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:05:44.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:44 smithi121 bash[18489]: audit 2024-02-12T16:05:43.919731+0000 mon.a (mon.0) 932 : audit [INF] from='client.? v1:172.21.15.121:0/693172070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:05:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:44.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:44 smithi123 bash[19119]: cluster 2024-02-12T16:05:43.561420+0000 mgr.y (mgr.14146) 606 : cluster [DBG] pgmap v692: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s; 19952/325778 objects misplaced (6.124%); 618 KiB/s, 9 objects/s recovering 2024-02-12T16:05:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:44 smithi123 bash[19119]: audit 2024-02-12T16:05:43.740005+0000 mon.a (mon.0) 931 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:05:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:44 smithi123 bash[19119]: audit 2024-02-12T16:05:43.743770+0000 mon.c (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.121:0/3951503911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:05:44.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:44 smithi123 bash[19119]: audit 2024-02-12T16:05:43.919731+0000 mon.a (mon.0) 932 : audit [INF] from='client.? v1:172.21.15.121:0/693172070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:05:44.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:45.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:45.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:45.108 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:45.143 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:45.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:45.277 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:45.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:45.408 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:45.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:45.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:45.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:45.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 168900 168884 121.313 90.875 0.0188318 0.00824166 2024-02-12T16:05:45.669 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:45.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:45.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:45 smithi123 bash[19119]: cluster 2024-02-12T16:05:44.614106+0000 mon.a (mon.0) 933 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:45.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:45 smithi123 bash[19119]: audit 2024-02-12T16:05:44.627810+0000 mon.a (mon.0) 934 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:05:45.781 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:45 smithi123 bash[19119]: audit 2024-02-12T16:05:44.627875+0000 mon.a (mon.0) 935 : audit [INF] from='client.? v1:172.21.15.121:0/693172070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:05:45.781 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:45 smithi123 bash[19119]: cluster 2024-02-12T16:05:44.627915+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e180: 12 total, 12 up, 12 in 2024-02-12T16:05:45.798 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:45.901 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:45.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:45 smithi121 bash[18489]: cluster 2024-02-12T16:05:44.614106+0000 mon.a (mon.0) 933 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:45.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:45 smithi121 bash[18489]: audit 2024-02-12T16:05:44.627810+0000 mon.a (mon.0) 934 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:05:45.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:45 smithi121 bash[18489]: audit 2024-02-12T16:05:44.627875+0000 mon.a (mon.0) 935 : audit [INF] from='client.? v1:172.21.15.121:0/693172070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:05:45.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:45 smithi121 bash[18489]: cluster 2024-02-12T16:05:44.627915+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e180: 12 total, 12 up, 12 in 2024-02-12T16:05:45.930 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:45.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:46.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:45 smithi169 bash[19604]: cluster 2024-02-12T16:05:44.614106+0000 mon.a (mon.0) 933 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:05:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:45 smithi169 bash[19604]: audit 2024-02-12T16:05:44.627810+0000 mon.a (mon.0) 934 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:05:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:45 smithi169 bash[19604]: audit 2024-02-12T16:05:44.627875+0000 mon.a (mon.0) 935 : audit [INF] from='client.? v1:172.21.15.121:0/693172070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:05:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:45 smithi169 bash[19604]: cluster 2024-02-12T16:05:44.627915+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e180: 12 total, 12 up, 12 in 2024-02-12T16:05:46.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:46.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:46.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:46.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:46.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:46.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:46.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:46.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:46.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:46.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:46.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:46.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:46 smithi121 bash[18489]: cluster 2024-02-12T16:05:45.561857+0000 mgr.y (mgr.14146) 607 : cluster [DBG] pgmap v694: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 639 op/s; 19952/329778 objects misplaced (6.050%); 589 KiB/s, 9 objects/s recovering 2024-02-12T16:05:46.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:46.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 170306 170290 120.933 87.875 0.00619014 0.00826719 2024-02-12T16:05:46.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:46.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:46.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:46.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T16:05:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:46 smithi169 bash[19604]: cluster 2024-02-12T16:05:45.561857+0000 mgr.y (mgr.14146) 607 : cluster [DBG] pgmap v694: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 639 op/s; 19952/329778 objects misplaced (6.050%); 589 KiB/s, 9 objects/s recovering 2024-02-12T16:05:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:46 smithi169 bash[19604]: cluster 2024-02-12T16:05:46.203655+0000 osd.11 (osd.11) 15 : cluster [DBG] 2.c scrub starts 2024-02-12T16:05:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:46 smithi169 bash[19604]: audit 2024-02-12T16:05:46.294042+0000 mon.a (mon.0) 937 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:05:46.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:46.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:46.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:46 smithi123 bash[19119]: cluster 2024-02-12T16:05:45.561857+0000 mgr.y (mgr.14146) 607 : cluster [DBG] pgmap v694: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 639 op/s; 19952/329778 objects misplaced (6.050%); 589 KiB/s, 9 objects/s recovering 2024-02-12T16:05:46.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:46 smithi123 bash[19119]: cluster 2024-02-12T16:05:46.203655+0000 osd.11 (osd.11) 15 : cluster [DBG] 2.c scrub starts 2024-02-12T16:05:46.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:46 smithi123 bash[19119]: audit 2024-02-12T16:05:46.294042+0000 mon.a (mon.0) 937 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:05:46.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:46 smithi121 bash[18489]: cluster 2024-02-12T16:05:46.203655+0000 osd.11 (osd.11) 15 : cluster [DBG] 2.c scrub starts 2024-02-12T16:05:46.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:46 smithi121 bash[18489]: audit 2024-02-12T16:05:46.294042+0000 mon.a (mon.0) 937 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:05:46.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:46.957 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:46.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:47.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:47.092 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:47.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:47.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:47.221 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:47.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:47.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:47.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:47.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:47.480 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:47.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:47.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:47.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:47.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:47.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 171639 171623 120.51 83.3125 0.036948 0.00829617 2024-02-12T16:05:47.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:47.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:47.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:47.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:47.879 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:47.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:47.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:48.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:48.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:48.147 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:48.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:48.284 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:48.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:48.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:48.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:48.560 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:48.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 90 2 173031 173029 120.147 87.875 0.00994329 0.00832189 2024-02-12T16:05:48.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:48 smithi121 bash[18489]: cluster 2024-02-12T16:05:47.562335+0000 mgr.y (mgr.14146) 608 : cluster [DBG] pgmap v695: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 936 op/s; 19836/336698 objects misplaced (5.891%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:05:48.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:48.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:48.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:48.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:05:48.825 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:48.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:48 smithi123 bash[19119]: cluster 2024-02-12T16:05:47.562335+0000 mgr.y (mgr.14146) 608 : cluster [DBG] pgmap v695: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 936 op/s; 19836/336698 objects misplaced (5.891%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:05:48.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:48.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:48.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:49.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:48 smithi169 bash[19604]: cluster 2024-02-12T16:05:47.562335+0000 mgr.y (mgr.14146) 608 : cluster [DBG] pgmap v695: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 936 op/s; 19836/336698 objects misplaced (5.891%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:05:49.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:49.100 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:49.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:49.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:49.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:49.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:49.288 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:49.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:49.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:49.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:49.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:49.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:49.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:49.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:49.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:49.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.019097 2024-02-12T16:05:49.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 173031 2024-02-12T16:05:49.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:05:49.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 120.135 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 67.7876 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 385.312 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 0 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 1922 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 1084 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 6165 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 0 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.00832202 2024-02-12T16:05:49.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.0316841 2024-02-12T16:05:49.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 2.70355 2024-02-12T16:05:49.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.000954878 2024-02-12T16:05:49.663 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:05:49.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:49.767 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:49.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:49.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:49.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:49.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:50.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:50.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:50.139 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:50.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:50.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:50.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:50.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:50.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:50.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:50.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:50.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:50.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:50.836 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:50 smithi169 bash[19604]: cluster 2024-02-12T16:05:49.008720+0000 osd.9 (osd.9) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:50.836 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:50 smithi169 bash[19604]: cluster 2024-02-12T16:05:49.008767+0000 osd.9 (osd.9) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:50.836 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:50 smithi169 bash[19604]: cluster 2024-02-12T16:05:49.523688+0000 osd.11 (osd.11) 16 : cluster [DBG] 2.c scrub ok 2024-02-12T16:05:50.836 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:50 smithi169 bash[19604]: cluster 2024-02-12T16:05:49.562963+0000 mgr.y (mgr.14146) 609 : cluster [DBG] pgmap v696: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s; 19718/341894 objects misplaced (5.767%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:05:50.836 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:50 smithi169 bash[19604]: cluster 2024-02-12T16:05:49.631991+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e181: 12 total, 12 up, 12 in 2024-02-12T16:05:50.858 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:50 smithi123 bash[19119]: cluster 2024-02-12T16:05:49.008720+0000 osd.9 (osd.9) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:50 smithi123 bash[19119]: cluster 2024-02-12T16:05:49.008767+0000 osd.9 (osd.9) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:50 smithi123 bash[19119]: cluster 2024-02-12T16:05:49.523688+0000 osd.11 (osd.11) 16 : cluster [DBG] 2.c scrub ok 2024-02-12T16:05:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:50 smithi123 bash[19119]: cluster 2024-02-12T16:05:49.562963+0000 mgr.y (mgr.14146) 609 : cluster [DBG] pgmap v696: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s; 19718/341894 objects misplaced (5.767%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:05:50.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:50 smithi123 bash[19119]: cluster 2024-02-12T16:05:49.631991+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e181: 12 total, 12 up, 12 in 2024-02-12T16:05:50.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:50 smithi121 bash[18489]: cluster 2024-02-12T16:05:49.008720+0000 osd.9 (osd.9) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:50.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:50 smithi121 bash[18489]: cluster 2024-02-12T16:05:49.008767+0000 osd.9 (osd.9) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:50.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:50 smithi121 bash[18489]: cluster 2024-02-12T16:05:49.523688+0000 osd.11 (osd.11) 16 : cluster [DBG] 2.c scrub ok 2024-02-12T16:05:50.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:50 smithi121 bash[18489]: cluster 2024-02-12T16:05:49.562963+0000 mgr.y (mgr.14146) 609 : cluster [DBG] pgmap v696: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s; 19718/341894 objects misplaced (5.767%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:05:50.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:50 smithi121 bash[18489]: cluster 2024-02-12T16:05:49.631991+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e181: 12 total, 12 up, 12 in 2024-02-12T16:05:50.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:50.985 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:05:50.986 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:05:50.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:50.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:51.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:51.122 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:51.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:51.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:51.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:51.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:51.316 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:51.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:51.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:51.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:51.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:51.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:51.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:51.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:51.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:51.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:51.790 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:51.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:51.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:51.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:51.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:52.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:52.046 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:52.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:52.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:52.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:52.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:52.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:52.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:52.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:52.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:52.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:52.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:52.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:52.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:05:52.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:52.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:52.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:52.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:52.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:52.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:52.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:52.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:52 smithi121 bash[18489]: cluster 2024-02-12T16:05:51.563463+0000 mgr.y (mgr.14146) 610 : cluster [DBG] pgmap v698: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 73 MiB/s wr, 1.22k op/s; 19602/343390 objects misplaced (5.708%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:05:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:52 smithi121 bash[18489]: audit 2024-02-12T16:05:51.652777+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:05:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:52 smithi121 bash[18489]: audit 2024-02-12T16:05:51.987065+0000 mon.a (mon.0) 940 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:05:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:52 smithi121 bash[18489]: audit 2024-02-12T16:05:51.987944+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:05:52.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:52 smithi121 bash[18489]: audit 2024-02-12T16:05:51.993737+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:05:52.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:52.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:52 smithi169 bash[19604]: cluster 2024-02-12T16:05:51.563463+0000 mgr.y (mgr.14146) 610 : cluster [DBG] pgmap v698: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 73 MiB/s wr, 1.22k op/s; 19602/343390 objects misplaced (5.708%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:05:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:52 smithi169 bash[19604]: audit 2024-02-12T16:05:51.652777+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:05:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:52 smithi169 bash[19604]: audit 2024-02-12T16:05:51.987065+0000 mon.a (mon.0) 940 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:05:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:52 smithi169 bash[19604]: audit 2024-02-12T16:05:51.987944+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:05:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:52 smithi169 bash[19604]: audit 2024-02-12T16:05:51.993737+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:05:53.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:53.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:53.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:52 smithi123 bash[19119]: cluster 2024-02-12T16:05:51.563463+0000 mgr.y (mgr.14146) 610 : cluster [DBG] pgmap v698: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 73 MiB/s wr, 1.22k op/s; 19602/343390 objects misplaced (5.708%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:05:53.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:52 smithi123 bash[19119]: audit 2024-02-12T16:05:51.652777+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:05:53.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:52 smithi123 bash[19119]: audit 2024-02-12T16:05:51.987065+0000 mon.a (mon.0) 940 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:05:53.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:52 smithi123 bash[19119]: audit 2024-02-12T16:05:51.987944+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:05:53.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:52 smithi123 bash[19119]: audit 2024-02-12T16:05:51.993737+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:05:53.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:53.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:53.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:53.368 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:05:53.398 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:53.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:53.530 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:53.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:53.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:53.666 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:53 smithi169 bash[19604]: cluster 2024-02-12T16:05:52.717429+0000 osd.6 (osd.6) 20 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:53.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:53.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:53 smithi123 bash[19119]: cluster 2024-02-12T16:05:52.717429+0000 osd.6 (osd.6) 20 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:53.773 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:53 smithi123 bash[19119]: cluster 2024-02-12T16:05:52.717481+0000 osd.6 (osd.6) 21 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:53.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:53.926 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:53.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:53.963 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:53 smithi121 bash[18489]: cluster 2024-02-12T16:05:52.717429+0000 osd.6 (osd.6) 20 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:53.963 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:53 smithi121 bash[18489]: cluster 2024-02-12T16:05:52.717481+0000 osd.6 (osd.6) 21 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:54.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:53 smithi169 bash[19604]: cluster 2024-02-12T16:05:52.717481+0000 osd.6 (osd.6) 21 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:54.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:54.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:54.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:54.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:54.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:54.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:54.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:54.457 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:05:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:05:54.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:54.595 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:54.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:54.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:05:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:05:54.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:54.839 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:54 smithi123 bash[19119]: cluster 2024-02-12T16:05:53.563932+0000 mgr.y (mgr.14146) 611 : cluster [DBG] pgmap v699: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 114 B/s rd, 62 MiB/s wr, 1.23k op/s; 19482/341790 objects misplaced (5.700%); 1.6 MiB/s, 26 objects/s recovering 2024-02-12T16:05:54.839 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:54 smithi123 bash[19119]: cluster 2024-02-12T16:05:53.659872+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e182: 12 total, 12 up, 12 in 2024-02-12T16:05:54.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:54.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:54 smithi121 bash[18489]: cluster 2024-02-12T16:05:53.563932+0000 mgr.y (mgr.14146) 611 : cluster [DBG] pgmap v699: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 114 B/s rd, 62 MiB/s wr, 1.23k op/s; 19482/341790 objects misplaced (5.700%); 1.6 MiB/s, 26 objects/s recovering 2024-02-12T16:05:54.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:54 smithi121 bash[18489]: cluster 2024-02-12T16:05:53.659872+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e182: 12 total, 12 up, 12 in 2024-02-12T16:05:54.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:54.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:54.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:54 smithi169 bash[19604]: cluster 2024-02-12T16:05:53.563932+0000 mgr.y (mgr.14146) 611 : cluster [DBG] pgmap v699: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 114 B/s rd, 62 MiB/s wr, 1.23k op/s; 19482/341790 objects misplaced (5.700%); 1.6 MiB/s, 26 objects/s recovering 2024-02-12T16:05:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:54 smithi169 bash[19604]: cluster 2024-02-12T16:05:53.659872+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e182: 12 total, 12 up, 12 in 2024-02-12T16:05:55.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:55.114 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:55.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:55.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:55.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:55.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:55.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:55.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:55.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:55.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:55.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:55.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:55.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:55.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:55.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:55.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:55.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:55.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:55.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:55.921 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:55.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:56.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:56.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:56.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:56.191 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:56.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:56.337 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:56.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:56.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:56 smithi169 bash[19604]: cluster 2024-02-12T16:05:55.035754+0000 osd.11 (osd.11) 17 : cluster [DBG] 2.a scrub starts 2024-02-12T16:05:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:56 smithi169 bash[19604]: cluster 2024-02-12T16:05:55.236641+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e183: 12 total, 12 up, 12 in 2024-02-12T16:05:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:56 smithi169 bash[19604]: cluster 2024-02-12T16:05:55.564318+0000 mgr.y (mgr.14146) 612 : cluster [DBG] pgmap v702: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 22 MiB/s wr, 1.10k op/s; 19482/337044 objects misplaced (5.780%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:05:56.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:56.602 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:56 smithi123 bash[19119]: cluster 2024-02-12T16:05:55.035754+0000 osd.11 (osd.11) 17 : cluster [DBG] 2.a scrub starts 2024-02-12T16:05:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:56 smithi123 bash[19119]: cluster 2024-02-12T16:05:55.236641+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e183: 12 total, 12 up, 12 in 2024-02-12T16:05:56.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:56 smithi123 bash[19119]: cluster 2024-02-12T16:05:55.564318+0000 mgr.y (mgr.14146) 612 : cluster [DBG] pgmap v702: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 22 MiB/s wr, 1.10k op/s; 19482/337044 objects misplaced (5.780%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:05:56.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:56 smithi121 bash[18489]: cluster 2024-02-12T16:05:55.035754+0000 osd.11 (osd.11) 17 : cluster [DBG] 2.a scrub starts 2024-02-12T16:05:56.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:56 smithi121 bash[18489]: cluster 2024-02-12T16:05:55.236641+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e183: 12 total, 12 up, 12 in 2024-02-12T16:05:56.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:56 smithi121 bash[18489]: cluster 2024-02-12T16:05:55.564318+0000 mgr.y (mgr.14146) 612 : cluster [DBG] pgmap v702: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 22 MiB/s wr, 1.10k op/s; 19482/337044 objects misplaced (5.780%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:05:56.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:56.731 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:56.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:56.860 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:56.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:56.979 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:05:56.987 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:56.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:57.010 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:05:57.046 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:05:57.076 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:05:57.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:57.106 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:05:57.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:57.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:57.131 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:05:57.157 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:05:57.186 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:05:57.214 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:05:57.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:57.243 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:05:57.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:05:57.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:57.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:05:57.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:57 smithi169 bash[19604]: audit 2024-02-12T16:05:56.943115+0000 mon.a (mon.0) 945 : audit [DBG] from='client.? v1:172.21.15.121:0/2693251077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:57.277 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:05:57.308 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.314 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:05:57.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:57.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:57.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:57.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:57.511 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:57 smithi121 bash[18489]: audit 2024-02-12T16:05:56.943115+0000 mon.a (mon.0) 945 : audit [DBG] from='client.? v1:172.21.15.121:0/2693251077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:57.530 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:57.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:57.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:57.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:57 smithi123 bash[19119]: audit 2024-02-12T16:05:56.943115+0000 mon.a (mon.0) 945 : audit [DBG] from='client.? v1:172.21.15.121:0/2693251077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:05:57.662 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:57.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:57.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:57.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:57.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:05:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:57.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:05:58.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:58.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:58.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.208 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:58.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:58.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:58.318 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:58 smithi169 bash[19604]: cluster 2024-02-12T16:05:57.564712+0000 mgr.y (mgr.14146) 613 : cluster [DBG] pgmap v703: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 16 MiB/s wr, 1.26k op/s; 19156/330200 objects misplaced (5.801%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:05:58.318 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:58 smithi169 bash[19604]: cluster 2024-02-12T16:05:57.654562+0000 osd.2 (osd.2) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:58.318 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:58 smithi169 bash[19604]: cluster 2024-02-12T16:05:57.654591+0000 osd.2 (osd.2) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:58.337 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:58.440 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:58.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:58.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:58.599 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:05:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.605 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:05:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:58 smithi123 bash[19119]: cluster 2024-02-12T16:05:57.564712+0000 mgr.y (mgr.14146) 613 : cluster [DBG] pgmap v703: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 16 MiB/s wr, 1.26k op/s; 19156/330200 objects misplaced (5.801%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:05:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:58 smithi123 bash[19119]: cluster 2024-02-12T16:05:57.654562+0000 osd.2 (osd.2) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:58 smithi123 bash[19119]: cluster 2024-02-12T16:05:57.654591+0000 osd.2 (osd.2) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:58.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:58 smithi121 bash[18489]: cluster 2024-02-12T16:05:57.564712+0000 mgr.y (mgr.14146) 613 : cluster [DBG] pgmap v703: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 16 MiB/s wr, 1.26k op/s; 19156/330200 objects misplaced (5.801%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:05:58.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:58 smithi121 bash[18489]: cluster 2024-02-12T16:05:57.654562+0000 osd.2 (osd.2) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:05:58.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:58 smithi121 bash[18489]: cluster 2024-02-12T16:05:57.654591+0000 osd.2 (osd.2) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:05:58.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:58.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:05:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:05:58.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:58.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:05:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:05:58.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:58.996 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:05:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:58.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:05:59.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:59.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:05:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:59.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:05:59.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:59.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:59 smithi121 bash[18489]: cluster 2024-02-12T16:05:57.714546+0000 osd.11 (osd.11) 18 : cluster [DBG] 2.a scrub ok 2024-02-12T16:05:59.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:59 smithi121 bash[18489]: cluster 2024-02-12T16:05:58.264976+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e184: 12 total, 12 up, 12 in 2024-02-12T16:05:59.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:05:59 smithi121 bash[18489]: audit 2024-02-12T16:05:59.219292+0000 mon.a (mon.0) 947 : audit [INF] from='client.? v1:172.21.15.121:0/1764193990' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:05:59.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:59 smithi169 bash[19604]: cluster 2024-02-12T16:05:57.714546+0000 osd.11 (osd.11) 18 : cluster [DBG] 2.a scrub ok 2024-02-12T16:05:59.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:59 smithi169 bash[19604]: cluster 2024-02-12T16:05:58.264976+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e184: 12 total, 12 up, 12 in 2024-02-12T16:05:59.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:05:59 smithi169 bash[19604]: audit 2024-02-12T16:05:59.219292+0000 mon.a (mon.0) 947 : audit [INF] from='client.? v1:172.21.15.121:0/1764193990' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:05:59.279 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T16:05:59.290 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:05:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:59.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:05:59.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:59.416 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:59.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:59.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:59.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:59.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:59.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:59 smithi123 bash[19119]: cluster 2024-02-12T16:05:57.714546+0000 osd.11 (osd.11) 18 : cluster [DBG] 2.a scrub ok 2024-02-12T16:05:59.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:59 smithi123 bash[19119]: cluster 2024-02-12T16:05:58.264976+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e184: 12 total, 12 up, 12 in 2024-02-12T16:05:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:05:59 smithi123 bash[19119]: audit 2024-02-12T16:05:59.219292+0000 mon.a (mon.0) 947 : audit [INF] from='client.? v1:172.21.15.121:0/1764193990' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:05:59.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:59.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:59.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:05:59.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:59.825 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:05:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:59.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:05:59.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:59.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:05:59.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:05:59.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:00.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:00.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:00.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:00.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:00.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:00.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:00.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:00.354 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:00.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:00.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:00.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T16:06:00.510 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:00.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:00.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:00 smithi169 bash[19604]: audit 2024-02-12T16:05:59.266384+0000 mon.a (mon.0) 948 : audit [INF] from='client.? v1:172.21.15.121:0/1764193990' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:06:00.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:00 smithi169 bash[19604]: cluster 2024-02-12T16:05:59.266443+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e185: 12 total, 12 up, 12 in 2024-02-12T16:06:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:00 smithi169 bash[19604]: cluster 2024-02-12T16:05:59.565124+0000 mgr.y (mgr.14146) 614 : cluster [DBG] pgmap v706: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 18764/325396 objects misplaced (5.767%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T16:06:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:00 smithi169 bash[19604]: cluster 2024-02-12T16:06:00.235600+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e186: 12 total, 12 up, 12 in 2024-02-12T16:06:00.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:00.628 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:00 smithi121 bash[18489]: audit 2024-02-12T16:05:59.266384+0000 mon.a (mon.0) 948 : audit [INF] from='client.? v1:172.21.15.121:0/1764193990' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:06:00.628 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:00 smithi121 bash[18489]: cluster 2024-02-12T16:05:59.266443+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e185: 12 total, 12 up, 12 in 2024-02-12T16:06:00.628 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:00 smithi121 bash[18489]: cluster 2024-02-12T16:05:59.565124+0000 mgr.y (mgr.14146) 614 : cluster [DBG] pgmap v706: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 18764/325396 objects misplaced (5.767%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T16:06:00.629 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:00 smithi121 bash[18489]: cluster 2024-02-12T16:06:00.235600+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e186: 12 total, 12 up, 12 in 2024-02-12T16:06:00.637 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:00.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:00 smithi123 bash[19119]: audit 2024-02-12T16:05:59.266384+0000 mon.a (mon.0) 948 : audit [INF] from='client.? v1:172.21.15.121:0/1764193990' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:06:00.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:00 smithi123 bash[19119]: cluster 2024-02-12T16:05:59.266443+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e185: 12 total, 12 up, 12 in 2024-02-12T16:06:00.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:00 smithi123 bash[19119]: cluster 2024-02-12T16:05:59.565124+0000 mgr.y (mgr.14146) 614 : cluster [DBG] pgmap v706: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 18764/325396 objects misplaced (5.767%); 3.8 MiB/s, 60 objects/s recovering 2024-02-12T16:06:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:00 smithi123 bash[19119]: cluster 2024-02-12T16:06:00.235600+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e186: 12 total, 12 up, 12 in 2024-02-12T16:06:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:00.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:00.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:00.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:00.875 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:00.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:00.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:00.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:01.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:06:01.043 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:01.066 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:01.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:01.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:01.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:06:01.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:01.351 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:01.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:01.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:01 smithi123 bash[19119]: audit 2024-02-12T16:06:01.235636+0000 mon.a (mon.0) 951 : audit [INF] from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:01.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:01 smithi123 bash[19119]: audit 2024-02-12T16:06:01.239324+0000 mon.c (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.121:0/102371346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:01.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:01.485 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:01.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:01.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:01 smithi169 bash[19604]: audit 2024-02-12T16:06:01.235636+0000 mon.a (mon.0) 951 : audit [INF] from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:01 smithi169 bash[19604]: audit 2024-02-12T16:06:01.239324+0000 mon.c (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.121:0/102371346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:01.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:01.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:01 smithi121 bash[18489]: audit 2024-02-12T16:06:01.235636+0000 mon.a (mon.0) 951 : audit [INF] from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:01.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:01 smithi121 bash[18489]: audit 2024-02-12T16:06:01.239324+0000 mon.c (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.121:0/102371346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:01.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:01.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:01.759 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:01.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:01.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:01.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:01.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:02.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:02.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:02.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:02.161 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:02.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:02.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:02.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:02.403 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:02 smithi123 bash[19119]: cluster 2024-02-12T16:06:01.276094+0000 mon.a (mon.0) 952 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:06:02.403 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:02 smithi123 bash[19119]: cluster 2024-02-12T16:06:01.276129+0000 mon.a (mon.0) 953 : cluster [INF] Cluster is now healthy 2024-02-12T16:06:02.403 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:02 smithi123 bash[19119]: audit 2024-02-12T16:06:01.290098+0000 mon.a (mon.0) 954 : audit [INF] from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:06:02.403 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:02 smithi123 bash[19119]: cluster 2024-02-12T16:06:01.290180+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e187: 12 total, 12 up, 12 in 2024-02-12T16:06:02.403 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:02 smithi123 bash[19119]: cluster 2024-02-12T16:06:01.565547+0000 mgr.y (mgr.14146) 615 : cluster [DBG] pgmap v709: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 18396/321396 objects misplaced (5.724%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:06:02.420 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:02.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:02.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:02 smithi169 bash[19604]: cluster 2024-02-12T16:06:01.276094+0000 mon.a (mon.0) 952 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:06:02.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:02 smithi169 bash[19604]: cluster 2024-02-12T16:06:01.276129+0000 mon.a (mon.0) 953 : cluster [INF] Cluster is now healthy 2024-02-12T16:06:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:02 smithi169 bash[19604]: audit 2024-02-12T16:06:01.290098+0000 mon.a (mon.0) 954 : audit [INF] from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:06:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:02 smithi169 bash[19604]: cluster 2024-02-12T16:06:01.290180+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e187: 12 total, 12 up, 12 in 2024-02-12T16:06:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:02 smithi169 bash[19604]: cluster 2024-02-12T16:06:01.565547+0000 mgr.y (mgr.14146) 615 : cluster [DBG] pgmap v709: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 18396/321396 objects misplaced (5.724%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:06:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:02.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:02 smithi121 bash[18489]: cluster 2024-02-12T16:06:01.276094+0000 mon.a (mon.0) 952 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:06:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:02 smithi121 bash[18489]: cluster 2024-02-12T16:06:01.276129+0000 mon.a (mon.0) 953 : cluster [INF] Cluster is now healthy 2024-02-12T16:06:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:02 smithi121 bash[18489]: audit 2024-02-12T16:06:01.290098+0000 mon.a (mon.0) 954 : audit [INF] from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:06:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:02 smithi121 bash[18489]: cluster 2024-02-12T16:06:01.290180+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e187: 12 total, 12 up, 12 in 2024-02-12T16:06:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:02 smithi121 bash[18489]: cluster 2024-02-12T16:06:01.565547+0000 mgr.y (mgr.14146) 615 : cluster [DBG] pgmap v709: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 18396/321396 objects misplaced (5.724%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:06:02.690 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:02.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:02.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:02.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:02.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:02.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:03.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:03.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:03.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:03.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:03.230 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:03.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:03.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:03.365 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:03.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:03.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:03.506 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:03.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:03.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:03.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:03.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:03.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:03.766 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:03.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:03.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:03.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:03.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:04.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:04.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:04.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:04.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:04.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:04.320 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:04.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:04.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:04.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:04.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:04.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:06:04.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:04.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:04.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:04 smithi169 bash[19604]: cluster 2024-02-12T16:06:03.566011+0000 mgr.y (mgr.14146) 616 : cluster [DBG] pgmap v710: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 17980/315562 objects misplaced (5.698%); 6.9 MiB/s, 110 objects/s recovering 2024-02-12T16:06:04.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:04.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:04 smithi121 bash[18489]: cluster 2024-02-12T16:06:03.566011+0000 mgr.y (mgr.14146) 616 : cluster [DBG] pgmap v710: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 17980/315562 objects misplaced (5.698%); 6.9 MiB/s, 110 objects/s recovering 2024-02-12T16:06:04.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:04.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:04.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:05.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:05.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:05.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:04 smithi123 bash[19119]: cluster 2024-02-12T16:06:03.566011+0000 mgr.y (mgr.14146) 616 : cluster [DBG] pgmap v710: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 17980/315562 objects misplaced (5.698%); 6.9 MiB/s, 110 objects/s recovering 2024-02-12T16:06:05.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:05.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:05.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:05.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:05.406 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:05.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:05.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:05.525 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:06:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:06:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:193570 2.1 e188/171 2:84bbb579:::benchmark_data_smithi169_42458_object20536:head v 188'10693, mlcod=188'10691)", 2024-02-12T16:06:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:05.511182+0000", 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0064536109999999997, 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064841999999999999, 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511182+0000", 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511182+0000", 2024-02-12T16:06:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511180+0000", 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6919999999999997e-06 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511189+0000", 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7559999999999994e-06 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511199+0000", 2024-02-12T16:06:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0329999999999997e-06 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511205+0000", 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2237000000000006e-05 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511287+0000", 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9038e-05 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.511306+0000", 2024-02-12T16:06:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012462500000000001 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:193574 2.1 e188/171 2:844e5a7c:::benchmark_data_smithi169_42458_object20540:head v 188'10694, mlcod=188'10691)", 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:05.514014+0000", 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0036209710000000002, 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037232150000000002, 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514014+0000", 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514014+0000", 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514013+0000", 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5380000000000008e-06 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514022+0000", 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3912e-05 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514036+0000", 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3149999999999997e-06 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514043+0000", 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310606 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514353+0000", 2024-02-12T16:06:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4700000000000003e-05 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.514388+0000", 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037359900000000001 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 2 2024-02-12T16:06:05.532 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:06:05.534 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:05.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:05.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:05.661 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:05.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:05.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:05.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:05.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:05.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:05.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:05.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:05.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:06.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:06.057 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:06.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:06.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:06.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:06.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:06.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:06.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:06.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:06.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:06.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:06:06.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:06.456 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:06.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:06 smithi169 bash[19604]: cluster 2024-02-12T16:06:05.237572+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e188: 12 total, 12 up, 12 in 2024-02-12T16:06:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:06 smithi169 bash[19604]: cluster 2024-02-12T16:06:05.566428+0000 mgr.y (mgr.14146) 617 : cluster [DBG] pgmap v712: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 17980/310880 objects misplaced (5.784%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:06:06.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:06.569 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:06 smithi121 bash[18489]: cluster 2024-02-12T16:06:05.237572+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e188: 12 total, 12 up, 12 in 2024-02-12T16:06:06.569 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:06 smithi121 bash[18489]: cluster 2024-02-12T16:06:05.566428+0000 mgr.y (mgr.14146) 617 : cluster [DBG] pgmap v712: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 17980/310880 objects misplaced (5.784%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:06:06.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:06.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:06.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:06 smithi123 bash[19119]: cluster 2024-02-12T16:06:05.237572+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e188: 12 total, 12 up, 12 in 2024-02-12T16:06:06.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:06 smithi123 bash[19119]: cluster 2024-02-12T16:06:05.566428+0000 mgr.y (mgr.14146) 617 : cluster [DBG] pgmap v712: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 17980/310880 objects misplaced (5.784%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:06:06.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:06.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:06.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:06.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:06.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:06.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:06.867 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:06.869 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:06.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:06.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:07.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:07.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:07.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:07.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:07.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:07.394 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:07.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:07.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:07.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:07.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:07.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:07.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:07.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:07.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:07.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:08.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:08.067 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:08.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:08.173 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:08.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:08.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:08.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:08.328 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:08.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:08.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:08.459 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:08.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:08.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:08.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:08.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:08.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:08 smithi123 bash[19119]: cluster 2024-02-12T16:06:07.566815+0000 mgr.y (mgr.14146) 618 : cluster [DBG] pgmap v713: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 17590/305076 objects misplaced (5.766%); 5.0 MiB/s, 80 objects/s recovering 2024-02-12T16:06:08.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:08 smithi123 bash[19119]: cluster 2024-02-12T16:06:07.949323+0000 osd.0 (osd.0) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:08.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:08 smithi123 bash[19119]: cluster 2024-02-12T16:06:07.949361+0000 osd.0 (osd.0) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:08.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:08.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:08.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:08 smithi169 bash[19604]: cluster 2024-02-12T16:06:07.566815+0000 mgr.y (mgr.14146) 618 : cluster [DBG] pgmap v713: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 17590/305076 objects misplaced (5.766%); 5.0 MiB/s, 80 objects/s recovering 2024-02-12T16:06:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:08 smithi169 bash[19604]: cluster 2024-02-12T16:06:07.949323+0000 osd.0 (osd.0) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:08 smithi169 bash[19604]: cluster 2024-02-12T16:06:07.949361+0000 osd.0 (osd.0) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:08.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:08.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:08.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:08.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:08 smithi121 bash[18489]: cluster 2024-02-12T16:06:07.566815+0000 mgr.y (mgr.14146) 618 : cluster [DBG] pgmap v713: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 17590/305076 objects misplaced (5.766%); 5.0 MiB/s, 80 objects/s recovering 2024-02-12T16:06:08.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:08 smithi121 bash[18489]: cluster 2024-02-12T16:06:07.949323+0000 osd.0 (osd.0) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:08.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:08 smithi121 bash[18489]: cluster 2024-02-12T16:06:07.949361+0000 osd.0 (osd.0) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:08.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:09.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:09.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:09.140 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:09.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:09.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:09.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:09.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:09.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:09.538 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:09.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:09.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:09.779 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:09 smithi121 bash[18489]: cluster 2024-02-12T16:06:08.631886+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e189: 12 total, 12 up, 12 in 2024-02-12T16:06:09.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:09 smithi169 bash[19604]: cluster 2024-02-12T16:06:08.631886+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e189: 12 total, 12 up, 12 in 2024-02-12T16:06:09.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:09.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:09.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:09 smithi123 bash[19119]: cluster 2024-02-12T16:06:08.631886+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e189: 12 total, 12 up, 12 in 2024-02-12T16:06:09.944 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:09.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:10.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:10.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:10.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:10.202 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:10.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:10.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:10.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:10.456 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:10.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:10.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:10.691 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:10.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:10.726 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:10.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:10.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:10.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:10 smithi121 bash[18489]: cluster 2024-02-12T16:06:09.567264+0000 mgr.y (mgr.14146) 619 : cluster [DBG] pgmap v715: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 17226/300558 objects misplaced (5.731%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T16:06:10.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:10 smithi121 bash[18489]: cluster 2024-02-12T16:06:10.238665+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e190: 12 total, 12 up, 12 in 2024-02-12T16:06:10.923 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:10.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:10.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:10 smithi169 bash[19604]: cluster 2024-02-12T16:06:09.567264+0000 mgr.y (mgr.14146) 619 : cluster [DBG] pgmap v715: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 17226/300558 objects misplaced (5.731%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T16:06:10.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:10 smithi169 bash[19604]: cluster 2024-02-12T16:06:10.238665+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e190: 12 total, 12 up, 12 in 2024-02-12T16:06:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:10.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:11.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:11.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:10 smithi123 bash[19119]: cluster 2024-02-12T16:06:09.567264+0000 mgr.y (mgr.14146) 619 : cluster [DBG] pgmap v715: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 17226/300558 objects misplaced (5.731%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T16:06:11.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:10 smithi123 bash[19119]: cluster 2024-02-12T16:06:10.238665+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e190: 12 total, 12 up, 12 in 2024-02-12T16:06:11.118 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:11.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:11.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:11.252 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:11.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:11.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:11.381 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:11.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:11.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:11.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:11.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:11.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:11.640 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:11.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:11.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:11.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:11.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:11.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:11.905 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:11.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:11.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:12.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:12.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:12.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:12.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:12.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:12.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:12.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:12.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:12.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:12.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:12.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:12.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:12.852 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:12.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:12 smithi121 bash[18489]: cluster 2024-02-12T16:06:11.567639+0000 mgr.y (mgr.14146) 620 : cluster [DBG] pgmap v717: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 16838/296632 objects misplaced (5.676%); 5.6 MiB/s, 90 objects/s recovering 2024-02-12T16:06:12.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:12.962 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:12 smithi169 bash[19604]: cluster 2024-02-12T16:06:11.567639+0000 mgr.y (mgr.14146) 620 : cluster [DBG] pgmap v717: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 16838/296632 objects misplaced (5.676%); 5.6 MiB/s, 90 objects/s recovering 2024-02-12T16:06:12.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:12.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:12.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:13.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:13.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:12 smithi123 bash[19119]: cluster 2024-02-12T16:06:11.567639+0000 mgr.y (mgr.14146) 620 : cluster [DBG] pgmap v717: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 16838/296632 objects misplaced (5.676%); 5.6 MiB/s, 90 objects/s recovering 2024-02-12T16:06:13.116 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:13.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:13.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:06:13.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:13.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:06:13.319 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:06:13.361 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:06:13.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:13.403 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:06:13.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:13.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:13.449 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:06:13.481 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:06:13.513 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:06:13.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:13.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:13.565 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:06:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:06:13.642 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:06:13.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:13.674 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:06:13.676 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.676 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:06:13.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:13.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:13.746 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:13 smithi121 bash[18489]: audit 2024-02-12T16:06:13.181361+0000 mon.a (mon.0) 959 : audit [DBG] from='client.? v1:172.21.15.121:0/1380905957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:13.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:13.810 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:13.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:13.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:13.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:13.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:13 smithi169 bash[19604]: audit 2024-02-12T16:06:13.181361+0000 mon.a (mon.0) 959 : audit [DBG] from='client.? v1:172.21.15.121:0/1380905957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:14.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:14.059 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:13 smithi123 bash[19119]: audit 2024-02-12T16:06:13.181361+0000 mon.a (mon.0) 959 : audit [DBG] from='client.? v1:172.21.15.121:0/1380905957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:14.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:14.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:14.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:14.206 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:14.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:14.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:14.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:14.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:14.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:14.488 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:14.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:14.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:14.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:14.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:14.673 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T16:06:14.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:14.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:14 smithi169 bash[19604]: cluster 2024-02-12T16:06:13.568091+0000 mgr.y (mgr.14146) 621 : cluster [DBG] pgmap v718: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 16426/290998 objects misplaced (5.645%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:06:14.731 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:14 smithi169 bash[19604]: audit 2024-02-12T16:06:13.698216+0000 mon.a (mon.0) 960 : audit [INF] from='client.? v1:172.21.15.121:0/3412810326' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:06:14.759 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:14.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:14.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:14.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:14 smithi123 bash[19119]: cluster 2024-02-12T16:06:13.568091+0000 mgr.y (mgr.14146) 621 : cluster [DBG] pgmap v718: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 16426/290998 objects misplaced (5.645%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:06:14.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:14 smithi123 bash[19119]: audit 2024-02-12T16:06:13.698216+0000 mon.a (mon.0) 960 : audit [INF] from='client.? v1:172.21.15.121:0/3412810326' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:06:14.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:14.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:15.011 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:15.038 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:15.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:15.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:14 smithi121 bash[18489]: cluster 2024-02-12T16:06:13.568091+0000 mgr.y (mgr.14146) 621 : cluster [DBG] pgmap v718: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 16426/290998 objects misplaced (5.645%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:06:15.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:14 smithi121 bash[18489]: audit 2024-02-12T16:06:13.698216+0000 mon.a (mon.0) 960 : audit [INF] from='client.? v1:172.21.15.121:0/3412810326' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:06:15.168 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:15.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:15.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:15.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:15.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:15.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:15.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:15.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:15.717 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:15.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:15 smithi169 bash[19604]: audit 2024-02-12T16:06:14.654947+0000 mon.a (mon.0) 961 : audit [INF] from='client.? v1:172.21.15.121:0/3412810326' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:06:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:15 smithi169 bash[19604]: cluster 2024-02-12T16:06:14.655003+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e191: 12 total, 12 up, 12 in 2024-02-12T16:06:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:15 smithi169 bash[19604]: cluster 2024-02-12T16:06:15.239057+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e192: 12 total, 12 up, 12 in 2024-02-12T16:06:15.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:15 smithi121 bash[18489]: audit 2024-02-12T16:06:14.654947+0000 mon.a (mon.0) 961 : audit [INF] from='client.? v1:172.21.15.121:0/3412810326' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:06:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:15 smithi121 bash[18489]: cluster 2024-02-12T16:06:14.655003+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e191: 12 total, 12 up, 12 in 2024-02-12T16:06:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:15 smithi121 bash[18489]: cluster 2024-02-12T16:06:15.239057+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e192: 12 total, 12 up, 12 in 2024-02-12T16:06:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:15.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:15.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:15.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:16.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:16.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:15 smithi123 bash[19119]: audit 2024-02-12T16:06:14.654947+0000 mon.a (mon.0) 961 : audit [INF] from='client.? v1:172.21.15.121:0/3412810326' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:06:16.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:15 smithi123 bash[19119]: cluster 2024-02-12T16:06:14.655003+0000 mon.a (mon.0) 962 : cluster [DBG] osdmap e191: 12 total, 12 up, 12 in 2024-02-12T16:06:16.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:15 smithi123 bash[19119]: cluster 2024-02-12T16:06:15.239057+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e192: 12 total, 12 up, 12 in 2024-02-12T16:06:16.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:16.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:16.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-12T16:06:16.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:16.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:16.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:16.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:16.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:16.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:16.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:16.593 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:16.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:16.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:16.706 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:16 smithi121 bash[18489]: cluster 2024-02-12T16:06:15.568504+0000 mgr.y (mgr.14146) 622 : cluster [DBG] pgmap v721: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 16426/286304 objects misplaced (5.737%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T16:06:16.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:16.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:16.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:16.856 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:16.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:16.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:16.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:16.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:16.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:17.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:16 smithi169 bash[19604]: cluster 2024-02-12T16:06:15.568504+0000 mgr.y (mgr.14146) 622 : cluster [DBG] pgmap v721: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 16426/286304 objects misplaced (5.737%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T16:06:17.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:17.099 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:16 smithi123 bash[19119]: cluster 2024-02-12T16:06:15.568504+0000 mgr.y (mgr.14146) 622 : cluster [DBG] pgmap v721: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 16426/286304 objects misplaced (5.737%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T16:06:17.118 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:17.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:17.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:17.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:17.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:17.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:17.393 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:17.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:17.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:17.529 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:17.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:17.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:17.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:17.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:17.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:17.796 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:17.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:17.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:17.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:17.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:18.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:18.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:18.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:18.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:18.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:06:18.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:18.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:18.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:18.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:18.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:18.344 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:06:18.344 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:06:18.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:18.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:18.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:18.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:18.629 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:18.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:18 smithi169 bash[19604]: cluster 2024-02-12T16:06:17.569010+0000 mgr.y (mgr.14146) 623 : cluster [DBG] pgmap v722: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 16012/280028 objects misplaced (5.718%); 5.2 MiB/s, 82 objects/s recovering 2024-02-12T16:06:18.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:18 smithi169 bash[19604]: audit 2024-02-12T16:06:18.632586+0000 mon.a (mon.0) 964 : audit [INF] from='client.? v1:172.21.15.121:0/2940748079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:06:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:06:18.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:18.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:18.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:18.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:18 smithi123 bash[19119]: cluster 2024-02-12T16:06:17.569010+0000 mgr.y (mgr.14146) 623 : cluster [DBG] pgmap v722: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 16012/280028 objects misplaced (5.718%); 5.2 MiB/s, 82 objects/s recovering 2024-02-12T16:06:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:18 smithi123 bash[19119]: audit 2024-02-12T16:06:18.632586+0000 mon.a (mon.0) 964 : audit [INF] from='client.? v1:172.21.15.121:0/2940748079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:06:18.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:18.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:19.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:19.032 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:19.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:19.143 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:18 smithi121 bash[18489]: cluster 2024-02-12T16:06:17.569010+0000 mgr.y (mgr.14146) 623 : cluster [DBG] pgmap v722: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 16012/280028 objects misplaced (5.718%); 5.2 MiB/s, 82 objects/s recovering 2024-02-12T16:06:19.143 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:18 smithi121 bash[18489]: audit 2024-02-12T16:06:18.632586+0000 mon.a (mon.0) 964 : audit [INF] from='client.? v1:172.21.15.121:0/2940748079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:06:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:19.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:19.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:19.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:19.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:19.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:19.564 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:19.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:19.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:19.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:19 smithi169 bash[19604]: cluster 2024-02-12T16:06:18.660735+0000 mon.a (mon.0) 965 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:19 smithi169 bash[19604]: audit 2024-02-12T16:06:18.665548+0000 mon.a (mon.0) 966 : audit [INF] from='client.? v1:172.21.15.121:0/2940748079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:06:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:19 smithi169 bash[19604]: cluster 2024-02-12T16:06:18.665610+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e193: 12 total, 12 up, 12 in 2024-02-12T16:06:19.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:19.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:19.939 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:19 smithi121 bash[18489]: cluster 2024-02-12T16:06:18.660735+0000 mon.a (mon.0) 965 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:19.939 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:19 smithi121 bash[18489]: audit 2024-02-12T16:06:18.665548+0000 mon.a (mon.0) 966 : audit [INF] from='client.? v1:172.21.15.121:0/2940748079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:06:19.939 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:19 smithi121 bash[18489]: cluster 2024-02-12T16:06:18.665610+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e193: 12 total, 12 up, 12 in 2024-02-12T16:06:19.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:19.955 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:19 smithi123 bash[19119]: cluster 2024-02-12T16:06:18.660735+0000 mon.a (mon.0) 965 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:19.955 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:19 smithi123 bash[19119]: audit 2024-02-12T16:06:18.665548+0000 mon.a (mon.0) 966 : audit [INF] from='client.? v1:172.21.15.121:0/2940748079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:06:19.955 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:19 smithi123 bash[19119]: cluster 2024-02-12T16:06:18.665610+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e193: 12 total, 12 up, 12 in 2024-02-12T16:06:19.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:19.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:19.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:06:20.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:20.108 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:20.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:20.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:20.258 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:20.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:20.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:20.396 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:20.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:20.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:20.530 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:20.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:20.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:20.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:20.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:20.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:20.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:20.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:20.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:20.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:20.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:21.020 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:21.028 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:20 smithi123 bash[19119]: cluster 2024-02-12T16:06:19.569475+0000 mgr.y (mgr.14146) 624 : cluster [DBG] pgmap v724: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s; 15606/275182 objects misplaced (5.671%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:06:21.028 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:20 smithi123 bash[19119]: cluster 2024-02-12T16:06:20.242466+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e194: 12 total, 12 up, 12 in 2024-02-12T16:06:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:20 smithi169 bash[19604]: cluster 2024-02-12T16:06:19.569475+0000 mgr.y (mgr.14146) 624 : cluster [DBG] pgmap v724: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s; 15606/275182 objects misplaced (5.671%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:06:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:20 smithi169 bash[19604]: cluster 2024-02-12T16:06:20.242466+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e194: 12 total, 12 up, 12 in 2024-02-12T16:06:21.053 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:21.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:21.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:21.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:20 smithi121 bash[18489]: cluster 2024-02-12T16:06:19.569475+0000 mgr.y (mgr.14146) 624 : cluster [DBG] pgmap v724: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s; 15606/275182 objects misplaced (5.671%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:06:21.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:20 smithi121 bash[18489]: cluster 2024-02-12T16:06:20.242466+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e194: 12 total, 12 up, 12 in 2024-02-12T16:06:21.188 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:21.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:21.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:21.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.322 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:21.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:21.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:21.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:21.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:21.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:21.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:21.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:21.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:21.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:21.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:21.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:06:21.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:21.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:22.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:22.126 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:22.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:22.259 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:22.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:22.410 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:22.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:22.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:22.650 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:22.689 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:22.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:22 smithi169 bash[19604]: cluster 2024-02-12T16:06:21.569957+0000 mgr.y (mgr.14146) 625 : cluster [DBG] pgmap v726: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 15220/271500 objects misplaced (5.606%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:06:22.835 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:22.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:22.947 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:22 smithi121 bash[18489]: cluster 2024-02-12T16:06:21.569957+0000 mgr.y (mgr.14146) 625 : cluster [DBG] pgmap v726: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 15220/271500 objects misplaced (5.606%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:06:22.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:22.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:23.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:23.088 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:22 smithi123 bash[19119]: cluster 2024-02-12T16:06:21.569957+0000 mgr.y (mgr.14146) 625 : cluster [DBG] pgmap v726: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 15220/271500 objects misplaced (5.606%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:06:23.104 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:23.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:23.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:23.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:23.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:23.339 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:23.368 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:23.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:23.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:23.426 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:23.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:23.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:23.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:23.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:23.642 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:23.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:23.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:23.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:23.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:23.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:23.909 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:23.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:24.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:24.046 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:24.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:24.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:24.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:24.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:24.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:24.450 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:24.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:24.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:24.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:24.721 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:24.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:24.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:24.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:24 smithi123 bash[19119]: cluster 2024-02-12T16:06:23.570392+0000 mgr.y (mgr.14146) 626 : cluster [DBG] pgmap v727: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 14794/266138 objects misplaced (5.559%); 6.4 MiB/s, 101 objects/s recovering 2024-02-12T16:06:24.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:24.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:24.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:25.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:24 smithi169 bash[19604]: cluster 2024-02-12T16:06:23.570392+0000 mgr.y (mgr.14146) 626 : cluster [DBG] pgmap v727: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 14794/266138 objects misplaced (5.559%); 6.4 MiB/s, 101 objects/s recovering 2024-02-12T16:06:25.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:25.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:24 smithi121 bash[18489]: cluster 2024-02-12T16:06:23.570392+0000 mgr.y (mgr.14146) 626 : cluster [DBG] pgmap v727: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 14794/266138 objects misplaced (5.559%); 6.4 MiB/s, 101 objects/s recovering 2024-02-12T16:06:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:25.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:25.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:25.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:25.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:25.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:25.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:25.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:25.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:25.550 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:25.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:25.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:25.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:25.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:25.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:25.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:25.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:25.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:25.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:25.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:26.043 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:26.074 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:26.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:26.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:26.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:26.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:26.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:26.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:26.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:26.471 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:26.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:26.496 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:26.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:26 smithi169 bash[19604]: cluster 2024-02-12T16:06:25.570793+0000 mgr.y (mgr.14146) 627 : cluster [DBG] pgmap v728: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 14794/261406 objects misplaced (5.659%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:06:26.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:26.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:26.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:26.635 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:26 smithi123 bash[19119]: cluster 2024-02-12T16:06:25.570793+0000 mgr.y (mgr.14146) 627 : cluster [DBG] pgmap v728: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 14794/261406 objects misplaced (5.659%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:06:26.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:26 smithi121 bash[18489]: cluster 2024-02-12T16:06:25.570793+0000 mgr.y (mgr.14146) 627 : cluster [DBG] pgmap v728: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 14794/261406 objects misplaced (5.659%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:06:26.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:26.736 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:26.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:26.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:26.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:26.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:26.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:27.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:27.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:27.139 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:27.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:27.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:27.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:27.402 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:27.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:27.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:27.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:27.661 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:27.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:27.793 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:27.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:27.923 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:27.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:27.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:28.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:28.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:28.187 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:28.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:28.318 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:28.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:28.458 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:28.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:28.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:28.634 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:28 smithi123 bash[19119]: cluster 2024-02-12T16:06:27.571244+0000 mgr.y (mgr.14146) 628 : cluster [DBG] pgmap v729: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 14446/254994 objects misplaced (5.665%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:06:28.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:28.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:28.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:28 smithi169 bash[19604]: cluster 2024-02-12T16:06:27.571244+0000 mgr.y (mgr.14146) 628 : cluster [DBG] pgmap v729: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 14446/254994 objects misplaced (5.665%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:06:28.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:28.857 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:28 smithi121 bash[18489]: cluster 2024-02-12T16:06:27.571244+0000 mgr.y (mgr.14146) 628 : cluster [DBG] pgmap v729: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 14446/254994 objects misplaced (5.665%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:06:28.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:28.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:28.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:29.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:29.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:29.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:29.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:29.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:29.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:29.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:29.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:29.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:29.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:29.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:29.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:29.648 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:29.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:29.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:29.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:29.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:29.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:29.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:29.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:30.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:30.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:30.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:30.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:30.184 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:30.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:30.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:30.316 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:30.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:30.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:30.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:30.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:30.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:30.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:30.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:30.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:30.725 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:30.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:30 smithi169 bash[19604]: cluster 2024-02-12T16:06:29.571660+0000 mgr.y (mgr.14146) 629 : cluster [DBG] pgmap v730: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 14092/250614 objects misplaced (5.623%); 4.7 MiB/s, 75 objects/s recovering 2024-02-12T16:06:30.827 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:30.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:30.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:30.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:30 smithi121 bash[18489]: cluster 2024-02-12T16:06:29.571660+0000 mgr.y (mgr.14146) 629 : cluster [DBG] pgmap v730: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 14092/250614 objects misplaced (5.623%); 4.7 MiB/s, 75 objects/s recovering 2024-02-12T16:06:30.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:30.980 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:30 smithi123 bash[19119]: cluster 2024-02-12T16:06:29.571660+0000 mgr.y (mgr.14146) 629 : cluster [DBG] pgmap v730: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 14092/250614 objects misplaced (5.623%); 4.7 MiB/s, 75 objects/s recovering 2024-02-12T16:06:30.999 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:31.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:31.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:31.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:31.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:31.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:31.407 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:31.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:31.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:31.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:31.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:31.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:31.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:31.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:31.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:32.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:32.106 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:32.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:32.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:32.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:32.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:32.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:32.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:32.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:32.502 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:32.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:32.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:32.633 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:32.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:32.653 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.3 to 1 (10000) 2024-02-12T16:06:32.714 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:06:32.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:32.765 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:06:32.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:32.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:32.817 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:06:32.850 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:06:32.885 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:06:32.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:32 smithi123 bash[19119]: cluster 2024-02-12T16:06:31.572101+0000 mgr.y (mgr.14146) 630 : cluster [DBG] pgmap v731: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 13720/247040 objects misplaced (5.554%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T16:06:32.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:32 smithi123 bash[19119]: audit 2024-02-12T16:06:32.064489+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.121:0/863571254' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:06:32.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:32.906 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:32 smithi169 bash[19604]: cluster 2024-02-12T16:06:31.572101+0000 mgr.y (mgr.14146) 630 : cluster [DBG] pgmap v731: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 13720/247040 objects misplaced (5.554%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T16:06:32.906 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:32 smithi169 bash[19604]: audit 2024-02-12T16:06:32.064489+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.121:0/863571254' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:06:32.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:32 smithi121 bash[18489]: cluster 2024-02-12T16:06:31.572101+0000 mgr.y (mgr.14146) 630 : cluster [DBG] pgmap v731: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 13720/247040 objects misplaced (5.554%); 5.2 MiB/s, 83 objects/s recovering 2024-02-12T16:06:32.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:32 smithi121 bash[18489]: audit 2024-02-12T16:06:32.064489+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.121:0/863571254' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:06:32.920 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:06:32.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:32.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:32.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:32.951 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:06:32.984 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:06:33.017 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:06:33.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:33.038 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:06:33.038 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:06:33.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51085 2.a e156/147 2:50205334:::benchmark_data_smithi169_42458_object51084:head v 156'3243, mlcod=156'3240)", 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.096434+0000", 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 121.92796892699999, 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26604782599999999, 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096434+0000", 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096434+0000", 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999814 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096416+0000", 2024-02-12T16:06:33.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7355000000000003e-05 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096463+0000", 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0964e-05 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096474+0000", 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.959e-06 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096480+0000", 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3524999999999999e-05 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096543+0000", 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9534e-05 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096563+0000", 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26581924499999998 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362382+0000", 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100048 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362482+0000", 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26604782599999999 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51084 2.e e156/147 2:705710c5:::benchmark_data_smithi169_42458_object51083:head v 156'3209, mlcod=156'3207)", 2024-02-12T16:06:33.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.096526+0000", 2024-02-12T16:06:33.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 121.927876966, 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25884117499999998, 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096526+0000", 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096526+0000", 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096525+0000", 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1891000000000001e-05 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096547+0000", 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-02-12T16:06:33.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096554+0000", 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6019999999999996e-06 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096561+0000", 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4433e-05 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096605+0000", 2024-02-12T16:06:33.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0227000000000001e-05 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096625+0000", 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25866377600000001 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355289+0000", 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8609000000000006e-05 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355368+0000", 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25884117499999998 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51088 2.a e156/147 2:522a68c4:::benchmark_data_smithi169_42458_object51087:head v 156'3244, mlcod=156'3240)", 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.097911+0000", 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 121.926492049, 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26463948399999998, 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097911+0000", 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097911+0000", 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097910+0000", 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0844000000000002e-05 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097971+0000", 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0606e-05 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097981+0000", 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8599999999999998e-06 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097987+0000", 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015630200000000001 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.098143+0000", 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8615e-05 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.098162+0000", 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26432980900000003 2024-02-12T16:06:33.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362492+0000", 2024-02-12T16:06:33.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8909000000000001e-05 2024-02-12T16:06:33.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362551+0000", 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26463948399999998 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51087 2.1 e156/147 2:85827259:::benchmark_data_smithi169_42458_object51086:head v 156'3177, mlcod=156'3175)", 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.099054+0000", 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 121.92534891699999, 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26445508099999998, 2024-02-12T16:06:33.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099054+0000", 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099054+0000", 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099053+0000", 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8713000000000001e-05 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099081+0000", 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25612307000000001 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355205+0000", 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2753000000000001e-05 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355227+0000", 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106338 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355334+0000", 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0373000000000001e-05 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355374+0000", 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.008059933 2024-02-12T16:06:33.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.363434+0000", 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5575999999999996e-05 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.363510+0000", 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26445508099999998 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:57692 2.1 e158/147 2:88048732:::benchmark_data_smithi169_42458_object57691:head v 158'3599, mlcod=158'3597)", 2024-02-12T16:06:33.059 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:33.756651+0000", 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.267752043, 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.043594604000000002, 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.756651+0000", 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.756651+0000", 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.756650+0000", 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000312916 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.756963+0000", 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5724999999999999e-05 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.756978+0000", 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6309999999999997e-06 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.756985+0000", 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.037169336999999997 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.794154+0000", 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5993e-05 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.794180+0000", 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006026376 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.800207+0000", 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9135999999999999e-05 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.800246+0000", 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.043594604000000002 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:57694 2.1 e158/147 2:8c24c3ae:::benchmark_data_smithi169_42458_object57693:head v 158'3600, mlcod=158'3597)", 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:33.760642+0000", 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.26376179099999, 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.039628343000000003, 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.760642+0000", 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.760642+0000", 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.760640+0000", 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7854000000000001e-05 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.760668+0000", 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033987007999999999 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.794655+0000", 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0129000000000001e-05 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.794665+0000", 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012425299999999999 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.795907+0000", 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3540000000000002e-05 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.795931+0000", 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004320656 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.800251+0000", 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8513999999999999e-05 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:33.800270+0000", 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.039628343000000003 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:147721 2.e e170/147 2:7a0a603b:::benchmark_data_smithi169_42458_object147720:head v 170'9252, mlcod=170'9247)", 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:05:27.136305+0000", 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.888097999999999, 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027312741000000001, 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.136305+0000", 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.136305+0000", 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.136304+0000", 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5007999999999998e-05 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.136329+0000", 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2459999999999992e-06 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.136338+0000", 2024-02-12T16:06:33.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8719999999999999e-06 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.136344+0000", 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062113259999999997 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.142555+0000", 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4420999999999999e-05 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.142580+0000", 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020946533999999999 2024-02-12T16:06:33.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.163526+0000", 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1927000000000003e-05 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:27.163618+0000", 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027312741000000001 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:187384 2.e e185/147 2:7d698d3c:::benchmark_data_smithi169_42458_object14350:head v 185'11664, mlcod=185'11660)", 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:00.194844+0000", 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.829558959000003, 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028207863999999999, 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194844+0000", 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194844+0000", 2024-02-12T16:06:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194843+0000", 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4619999999999996e-06 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194851+0000", 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8120000000000002e-06 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194861+0000", 2024-02-12T16:06:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5300000000000002e-06 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194868+0000", 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0244999999999998e-05 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194918+0000", 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4987999999999999e-05 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.194943+0000", 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028054289 2024-02-12T16:06:33.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.222997+0000", 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5229000000000003e-05 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:00.223052+0000", 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028207863999999999 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:189118 2.a e187/147 2:54afa137:::benchmark_data_smithi169_42458_object16084:head v 187'11789, mlcod=187'11785)", 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:01.582391+0000", 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.442012325, 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027668914999999999, 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.582391+0000", 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.582391+0000", 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.582389+0000", 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2758999999999999e-05 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.582401+0000", 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4498000000000001e-05 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.582416+0000", 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.582423+0000", 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041098920000000004 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.586533+0000", 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0157999999999999e-05 2024-02-12T16:06:33.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.586553+0000", 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023472489999999999 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.610025+0000", 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4588999999999997e-05 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:01.610060+0000", 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027668914999999999 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.079 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.080 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:193366 2.e e188/147 2:7fbfb780:::benchmark_data_smithi169_42458_object20332:head v 188'12055, mlcod=188'12051)", 2024-02-12T16:06:33.080 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:05.349134+0000", 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.675269868000001, 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033721407000000002, 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.349134+0000", 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.349134+0000", 2024-02-12T16:06:33.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.349131+0000", 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1747e-05 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.349143+0000", 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6303e-05 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.349159+0000", 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.764e-06 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.349166+0000", 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017767799999999999 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.350943+0000", 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4937e-05 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.350988+0000", 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.031821334999999999 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.382809+0000", 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5753000000000001e-05 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.382855+0000", 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033721407000000002 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:193382 2.e e188/147 2:7054f580:::benchmark_data_smithi169_42458_object20348:head v 188'12056, mlcod=188'12051)", 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:05.353713+0000", 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.670690622999999, 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029168840000000001, 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.353713+0000", 2024-02-12T16:06:33.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.353713+0000", 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.353711+0000", 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.353719+0000", 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2126999999999999e-05 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.353731+0000", 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3349999999999997e-06 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.353736+0000", 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013844827000000001 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.367581+0000", 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7637e-05 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.367609+0000", 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015250689 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.382860+0000", 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1946999999999998e-05 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:05.382882+0000", 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029168840000000001 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:196660 2.e e188/147 2:7ac7a4f0:::benchmark_data_smithi169_42458_object23626:head v 188'12244, mlcod=188'12241)", 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:07.881958+0000", 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.142445835, 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.041267195, 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.089 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.881958+0000", 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.881958+0000", 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.881954+0000", 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4653e-05 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.881969+0000", 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5503999999999999e-05 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.881984+0000", 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6889999999999993e-06 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.881993+0000", 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043686519999999998 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886362+0000", 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6548000000000001e-05 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886388+0000", 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036798619999999997 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923187+0000", 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7942999999999999e-05 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923225+0000", 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.041267195 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:196661 2.e e188/147 2:741baeab:::benchmark_data_smithi169_42458_object23627:head v 188'12245, mlcod=188'12241)", 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:07.886313+0000", 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.138090343000002, 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036932653000000003, 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886313+0000", 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886313+0000", 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886311+0000", 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0566e-05 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886321+0000", 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3044000000000001e-05 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886335+0000", 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0560000000000005e-06 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.886343+0000", 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082291629999999994 2024-02-12T16:06:33.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.894572+0000", 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2288000000000001e-05 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.894594+0000", 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028633788 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923228+0000", 2024-02-12T16:06:33.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7847e-05 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923246+0000", 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036932653000000003 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:196668 2.e e188/147 2:78adf2e5:::benchmark_data_smithi169_42458_object23634:head v 188'12246, mlcod=188'12241)", 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:07.890076+0000", 2024-02-12T16:06:33.097 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.134327307, 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033187115000000003, 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.890076+0000", 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.890076+0000", 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.890074+0000", 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1368e-05 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.890086+0000", 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4554e-05 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.890100+0000", 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6490000000000002e-06 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.890107+0000", 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086398489999999998 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.898747+0000", 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5084e-05 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.898762+0000", 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024485864999999999 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923248+0000", 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5651999999999999e-05 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923263+0000", 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033187115000000003 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:196677 2.e e188/147 2:70e145d5:::benchmark_data_smithi169_42458_object23643:head v 188'12247, mlcod=188'12241)", 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:07.895506+0000", 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.128897388999999, 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027775854999999999, 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.895506+0000", 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.895506+0000", 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.895504+0000", 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4830000000000005e-06 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.895514+0000", 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2829e-05 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.895527+0000", 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6659999999999996e-06 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.895532+0000", 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011683153 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.907216+0000", 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7523999999999999e-05 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.907243+0000", 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016021662999999998 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923265+0000", 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7149000000000001e-05 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:07.923282+0000", 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027775854999999999 2024-02-12T16:06:33.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:203288 2.a e190/147 2:5d3c59d0:::benchmark_data_smithi169_42458_object30254:head v 190'12681, mlcod=190'12678)", 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:13.048749+0000", 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.975654602999999, 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055411479, 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048749+0000", 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048749+0000", 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048747+0000", 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6565000000000001e-05 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048763+0000", 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5498e-05 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048779+0000", 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.216e-06 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048786+0000", 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8720000000000006e-05 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048855+0000", 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4019000000000002e-05 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.048879+0000", 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055245966000000001 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104125+0000", 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5701999999999998e-05 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104160+0000", 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055411479 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:203299 2.a e190/147 2:56dee156:::benchmark_data_smithi169_42458_object30265:head v 190'12682, mlcod=190'12678)", 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:13.063583+0000", 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.960820784999999, 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.040595671999999999, 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.063583+0000", 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.063583+0000", 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.063581+0000", 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3511e-05 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.063594+0000", 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.378e-05 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.063608+0000", 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999993e-06 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.063616+0000", 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004162163 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.067778+0000", 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6565000000000001e-05 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.067804+0000", 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036358743999999998 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104163+0000", 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5213e-05 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104178+0000", 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.040595671999999999 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:203303 2.a e190/147 2:5abd2e9d:::benchmark_data_smithi169_42458_object30269:head v 190'12683, mlcod=190'12678)", 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:13.068539+0000", 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.955863931, 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035654555999999997, 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.068539+0000", 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.068539+0000", 2024-02-12T16:06:33.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.068537+0000", 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0900999999999999e-05 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.068548+0000", 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3349999999999999e-05 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.068561+0000", 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.934e-06 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.068568+0000", 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038604609999999999 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.072429+0000", 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0205e-05 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.072459+0000", 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.031722093 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104181+0000", 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2967e-05 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104194+0000", 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035654555999999997 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:203308 2.a e190/147 2:595be0f5:::benchmark_data_smithi169_42458_object30274:head v 190'12684, mlcod=190'12679)", 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:13.075070+0000", 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.949333676999998, 2024-02-12T16:06:33.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029139743999999999, 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.075070+0000", 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.075070+0000", 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.075068+0000", 2024-02-12T16:06:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2031000000000001e-05 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.075080+0000", 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4079e-05 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.075094+0000", 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1910000000000003e-06 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.075101+0000", 2024-02-12T16:06:33.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022426590000000001 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.077344+0000", 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7928000000000001e-05 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.077371+0000", 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026824064000000002 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104195+0000", 2024-02-12T16:06:33.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3943e-05 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104209+0000", 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029139743999999999 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:203310 2.a e190/147 2:5ae879b4:::benchmark_data_smithi169_42458_object30276:head v 190'12685, mlcod=190'12679)", 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:06:13.076548+0000", 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.947855327999999, 2024-02-12T16:06:33.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027678512999999998, 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.076548+0000", 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.076548+0000", 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:06:33.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.076546+0000", 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3478999999999999e-05 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.076559+0000", 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.288e-05 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.076572+0000", 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.888e-06 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.076579+0000", 2024-02-12T16:06:33.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018545961 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.095125+0000", 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1576000000000001e-05 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.095146+0000", 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090677039999999993 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104214+0000", 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2404e-05 2024-02-12T16:06:33.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:06:13.104227+0000", 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027678512999999998 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:06:33.125 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:06:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.127 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:06:33.128 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:33.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:33.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:33.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:33.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:33.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:33.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:33.396 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:33.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:33.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:33.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:33.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:33.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:33.662 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:33.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:33.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:33.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:33 smithi121 bash[18489]: audit 2024-02-12T16:06:32.642156+0000 mon.a (mon.0) 970 : audit [INF] from='client.? v1:172.21.15.121:0/863571254' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:06:33.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:33 smithi121 bash[18489]: cluster 2024-02-12T16:06:32.642212+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e195: 12 total, 12 up, 12 in 2024-02-12T16:06:33.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:33 smithi121 bash[18489]: audit 2024-02-12T16:06:32.664509+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? v1:172.21.15.121:0/3379052552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:33.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:33 smithi121 bash[18489]: cluster 2024-02-12T16:06:32.811712+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:33.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:33 smithi121 bash[18489]: cluster 2024-02-12T16:06:32.811770+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:33.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:33.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:33.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:33 smithi123 bash[19119]: audit 2024-02-12T16:06:32.642156+0000 mon.a (mon.0) 970 : audit [INF] from='client.? v1:172.21.15.121:0/863571254' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:06:33.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:33 smithi123 bash[19119]: cluster 2024-02-12T16:06:32.642212+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e195: 12 total, 12 up, 12 in 2024-02-12T16:06:33.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:33 smithi123 bash[19119]: audit 2024-02-12T16:06:32.664509+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? v1:172.21.15.121:0/3379052552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:33.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:33 smithi123 bash[19119]: cluster 2024-02-12T16:06:32.811712+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:33 smithi123 bash[19119]: cluster 2024-02-12T16:06:32.811770+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:33.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:33.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:33 smithi169 bash[19604]: audit 2024-02-12T16:06:32.642156+0000 mon.a (mon.0) 970 : audit [INF] from='client.? v1:172.21.15.121:0/863571254' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:06:33.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:33 smithi169 bash[19604]: cluster 2024-02-12T16:06:32.642212+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e195: 12 total, 12 up, 12 in 2024-02-12T16:06:33.922 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:33 smithi169 bash[19604]: audit 2024-02-12T16:06:32.664509+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? v1:172.21.15.121:0/3379052552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:33.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:33 smithi169 bash[19604]: cluster 2024-02-12T16:06:32.811712+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:33.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:33 smithi169 bash[19604]: cluster 2024-02-12T16:06:32.811770+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:33.952 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:33.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:34.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:34.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:34.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:34.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:34.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:34.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:34.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T16:06:34.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:34.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:34.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:34.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:34.474 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:34.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:34.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:34.602 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:34.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:34.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:34 smithi121 bash[18489]: cluster 2024-02-12T16:06:33.572486+0000 mgr.y (mgr.14146) 631 : cluster [DBG] pgmap v733: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 13414/241942 objects misplaced (5.544%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:06:34.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:34 smithi121 bash[18489]: cluster 2024-02-12T16:06:33.657705+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e196: 12 total, 12 up, 12 in 2024-02-12T16:06:34.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:34.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:34.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:34.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:34.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:06:34.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:34.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:34.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:34.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:34 smithi169 bash[19604]: cluster 2024-02-12T16:06:33.572486+0000 mgr.y (mgr.14146) 631 : cluster [DBG] pgmap v733: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 13414/241942 objects misplaced (5.544%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:06:34.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:34 smithi169 bash[19604]: cluster 2024-02-12T16:06:33.657705+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e196: 12 total, 12 up, 12 in 2024-02-12T16:06:35.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:35.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:35.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:35.144 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:34 smithi123 bash[19119]: cluster 2024-02-12T16:06:33.572486+0000 mgr.y (mgr.14146) 631 : cluster [DBG] pgmap v733: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 13414/241942 objects misplaced (5.544%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:06:35.144 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:34 smithi123 bash[19119]: cluster 2024-02-12T16:06:33.657705+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e196: 12 total, 12 up, 12 in 2024-02-12T16:06:35.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:35.283 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:35.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:35.416 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:35.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:35.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:35.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:35.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:35.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:35.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:35.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:35.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:36.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:36.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:36.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:36.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:36.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:36.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:36.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:36 smithi121 bash[18489]: audit 2024-02-12T16:06:35.480330+0000 mon.a (mon.0) 974 : audit [INF] from='client.? v1:172.21.15.121:0/3697322904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:36.257 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:36 smithi169 bash[19604]: audit 2024-02-12T16:06:35.480330+0000 mon.a (mon.0) 974 : audit [INF] from='client.? v1:172.21.15.121:0/3697322904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:36.257 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:36 smithi169 bash[19604]: cluster 2024-02-12T16:06:35.572922+0000 mgr.y (mgr.14146) 632 : cluster [DBG] pgmap v735: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 13414/237468 objects misplaced (5.649%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:06:36.266 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:06:36.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:36.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:36.463 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:36.470 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:36 smithi123 bash[19119]: audit 2024-02-12T16:06:35.480330+0000 mon.a (mon.0) 974 : audit [INF] from='client.? v1:172.21.15.121:0/3697322904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:06:36.470 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:36 smithi123 bash[19119]: cluster 2024-02-12T16:06:35.572922+0000 mgr.y (mgr.14146) 632 : cluster [DBG] pgmap v735: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 13414/237468 objects misplaced (5.649%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:06:36.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:36.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:36.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:36.628 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:36.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:36.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:36 smithi121 bash[18489]: cluster 2024-02-12T16:06:35.572922+0000 mgr.y (mgr.14146) 632 : cluster [DBG] pgmap v735: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 13414/237468 objects misplaced (5.649%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:06:36.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:36.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:36.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:36.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:06:36.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:36.911 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:36.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:37.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:37.045 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:37.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:37.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:37.177 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:37.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:37 smithi169 bash[19604]: cluster 2024-02-12T16:06:36.242495+0000 mon.a (mon.0) 975 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:37 smithi169 bash[19604]: audit 2024-02-12T16:06:36.253287+0000 mon.a (mon.0) 976 : audit [INF] from='client.? v1:172.21.15.121:0/3697322904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:06:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:37 smithi169 bash[19604]: cluster 2024-02-12T16:06:36.253329+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e197: 12 total, 12 up, 12 in 2024-02-12T16:06:37.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:37.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:37.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:37.451 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:37.512 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:37 smithi121 bash[18489]: cluster 2024-02-12T16:06:36.242495+0000 mon.a (mon.0) 975 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:37.512 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:37 smithi121 bash[18489]: audit 2024-02-12T16:06:36.253287+0000 mon.a (mon.0) 976 : audit [INF] from='client.? v1:172.21.15.121:0/3697322904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:06:37.512 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:37 smithi121 bash[18489]: cluster 2024-02-12T16:06:36.253329+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e197: 12 total, 12 up, 12 in 2024-02-12T16:06:37.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:37.570 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:37 smithi123 bash[19119]: cluster 2024-02-12T16:06:36.242495+0000 mon.a (mon.0) 975 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:37.570 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:37 smithi123 bash[19119]: audit 2024-02-12T16:06:36.253287+0000 mon.a (mon.0) 976 : audit [INF] from='client.? v1:172.21.15.121:0/3697322904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:06:37.570 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:37 smithi123 bash[19119]: cluster 2024-02-12T16:06:36.253329+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e197: 12 total, 12 up, 12 in 2024-02-12T16:06:37.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:37.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:37.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:37.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:37.861 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:37.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:37.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:37.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:37.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:38.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:38.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:38.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:38.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:38.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:38.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:38.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:38.399 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:38.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:38.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:38 smithi121 bash[18489]: cluster 2024-02-12T16:06:37.573378+0000 mgr.y (mgr.14146) 633 : cluster [DBG] pgmap v737: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 13068/231806 objects misplaced (5.637%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:06:38.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:38.512 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:38 smithi169 bash[19604]: cluster 2024-02-12T16:06:37.573378+0000 mgr.y (mgr.14146) 633 : cluster [DBG] pgmap v737: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 13068/231806 objects misplaced (5.637%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:06:38.534 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:38.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:38.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:38.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:38 smithi123 bash[19119]: cluster 2024-02-12T16:06:37.573378+0000 mgr.y (mgr.14146) 633 : cluster [DBG] pgmap v737: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 13068/231806 objects misplaced (5.637%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:06:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:38.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:38.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:38.802 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:38.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:38.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:38.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:38.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:39.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:39.079 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:39.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:39.206 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:39.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:39.338 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:39.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:39.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:39.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:06:39.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:39.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:39.617 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:06:39.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:39.725 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:39.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:39.858 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:39.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:39.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:39.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:40.003 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:40.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:40.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:40.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:40.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:40.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:40.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:40.277 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:40 smithi123 bash[19119]: cluster 2024-02-12T16:06:38.709757+0000 osd.10 (osd.10) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:40.277 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:40 smithi123 bash[19119]: cluster 2024-02-12T16:06:38.709815+0000 osd.10 (osd.10) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:40.277 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:40 smithi123 bash[19119]: cluster 2024-02-12T16:06:39.260048+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e198: 12 total, 12 up, 12 in 2024-02-12T16:06:40.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:40.403 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:40.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:40 smithi121 bash[18489]: cluster 2024-02-12T16:06:38.709757+0000 osd.10 (osd.10) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:40 smithi121 bash[18489]: cluster 2024-02-12T16:06:38.709815+0000 osd.10 (osd.10) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:40 smithi121 bash[18489]: cluster 2024-02-12T16:06:39.260048+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e198: 12 total, 12 up, 12 in 2024-02-12T16:06:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:40 smithi121 bash[18489]: cluster 2024-02-12T16:06:39.573791+0000 mgr.y (mgr.14146) 634 : cluster [DBG] pgmap v739: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 12658/227396 objects misplaced (5.567%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T16:06:40.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:40.538 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:40.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:40.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:40 smithi169 bash[19604]: cluster 2024-02-12T16:06:38.709757+0000 osd.10 (osd.10) 15 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:40.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:40 smithi169 bash[19604]: cluster 2024-02-12T16:06:38.709815+0000 osd.10 (osd.10) 16 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:40.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:40 smithi169 bash[19604]: cluster 2024-02-12T16:06:39.260048+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e198: 12 total, 12 up, 12 in 2024-02-12T16:06:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:40 smithi169 bash[19604]: cluster 2024-02-12T16:06:39.573791+0000 mgr.y (mgr.14146) 634 : cluster [DBG] pgmap v739: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 12658/227396 objects misplaced (5.567%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T16:06:40.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:40 smithi123 bash[19119]: cluster 2024-02-12T16:06:39.573791+0000 mgr.y (mgr.14146) 634 : cluster [DBG] pgmap v739: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 12658/227396 objects misplaced (5.567%); 3.9 MiB/s, 62 objects/s recovering 2024-02-12T16:06:40.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:40.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:40.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:40.779 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:40.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:40.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:40.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:40.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:40.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:41.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:41.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:41.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:41.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:41.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:41.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:41.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:41.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:41.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:41.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:41.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:41.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:41.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:41.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:41.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:41.750 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:41.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:41.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:41.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:41.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:42.032 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:42.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:42.155 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:42.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:42.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:42.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:42.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:42.556 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:42.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:42.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:42.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:42.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:42 smithi123 bash[19119]: cluster 2024-02-12T16:06:41.574272+0000 mgr.y (mgr.14146) 635 : cluster [DBG] pgmap v740: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s; 12268/220824 objects misplaced (5.556%); 4.5 MiB/s, 72 objects/s recovering 2024-02-12T16:06:42.927 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:42 smithi121 bash[18489]: cluster 2024-02-12T16:06:41.574272+0000 mgr.y (mgr.14146) 635 : cluster [DBG] pgmap v740: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s; 12268/220824 objects misplaced (5.556%); 4.5 MiB/s, 72 objects/s recovering 2024-02-12T16:06:42.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:42.986 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:42.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:42 smithi169 bash[19604]: cluster 2024-02-12T16:06:41.574272+0000 mgr.y (mgr.14146) 635 : cluster [DBG] pgmap v740: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s; 12268/220824 objects misplaced (5.556%); 4.5 MiB/s, 72 objects/s recovering 2024-02-12T16:06:43.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:43.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:43.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:43.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:43.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:43.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:43.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:43.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:43.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:43.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:43.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:43.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:43.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:43.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:43.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:43.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:43.772 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:43.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:43.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:43.931 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:43.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:44.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:44.076 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:44.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:44.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:44.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:44.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:44.308 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:44.328 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:44.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:44.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:44.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:44.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:44.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:241753 2.d e198/147 2:b3c16430:::benchmark_data_smithi169_42458_object68719:head v 198'15179, mlcod=198'15176)", 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:44.532066+0000", 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0020411520000000001, 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020751340000000002, 2024-02-12T16:06:44.535 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532066+0000", 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532066+0000", 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:06:44.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532064+0000", 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2397e-05 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532077+0000", 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4209000000000001e-05 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532091+0000", 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.596e-06 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:06:44.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532097+0000", 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4728999999999998e-05 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532142+0000", 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9859000000000001e-05 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:44.532162+0000", 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6124000000000003e-05 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:06:44.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:06:44.539 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:06:44.539 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:06:44.539 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 1 2024-02-12T16:06:44.539 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:06:44.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:44.606 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:44.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:44.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:44.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:44 smithi169 bash[19604]: cluster 2024-02-12T16:06:43.574661+0000 mgr.y (mgr.14146) 636 : cluster [DBG] pgmap v741: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 12268/219248 objects misplaced (5.595%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:06:44.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:44.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:44.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:44.868 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:44.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:44.871 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:44 smithi123 bash[19119]: cluster 2024-02-12T16:06:43.574661+0000 mgr.y (mgr.14146) 636 : cluster [DBG] pgmap v741: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 12268/219248 objects misplaced (5.595%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:06:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:44 smithi121 bash[18489]: cluster 2024-02-12T16:06:43.574661+0000 mgr.y (mgr.14146) 636 : cluster [DBG] pgmap v741: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 12268/219248 objects misplaced (5.595%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:06:44.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:44.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:45.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:45.128 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:45.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:45.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:45.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:45.391 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:06:45.405 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:45.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:45.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:45.664 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:45.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:45.794 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:45.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:45.939 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:45.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:46.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:46.066 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:46.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:46.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:46.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:46.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:46.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:46.340 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:46.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:46.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:46.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:46.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:46 smithi169 bash[19604]: cluster 2024-02-12T16:06:45.349152+0000 osd.7 (osd.7) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:46 smithi169 bash[19604]: cluster 2024-02-12T16:06:45.349209+0000 osd.7 (osd.7) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:46 smithi169 bash[19604]: cluster 2024-02-12T16:06:45.575154+0000 mgr.y (mgr.14146) 637 : cluster [DBG] pgmap v742: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 11884/213084 objects misplaced (5.577%); 5.1 MiB/s, 82 objects/s recovering 2024-02-12T16:06:46.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:46.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:46 smithi121 bash[18489]: cluster 2024-02-12T16:06:45.349152+0000 osd.7 (osd.7) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:46.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:46 smithi121 bash[18489]: cluster 2024-02-12T16:06:45.349209+0000 osd.7 (osd.7) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:46.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:46 smithi121 bash[18489]: cluster 2024-02-12T16:06:45.575154+0000 mgr.y (mgr.14146) 637 : cluster [DBG] pgmap v742: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 11884/213084 objects misplaced (5.577%); 5.1 MiB/s, 82 objects/s recovering 2024-02-12T16:06:46.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:46.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:46.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:46 smithi123 bash[19119]: cluster 2024-02-12T16:06:45.349152+0000 osd.7 (osd.7) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:46.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:46 smithi123 bash[19119]: cluster 2024-02-12T16:06:45.349209+0000 osd.7 (osd.7) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:46.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:46 smithi123 bash[19119]: cluster 2024-02-12T16:06:45.575154+0000 mgr.y (mgr.14146) 637 : cluster [DBG] pgmap v742: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 11884/213084 objects misplaced (5.577%); 5.1 MiB/s, 82 objects/s recovering 2024-02-12T16:06:46.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:46.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:46.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:46.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:46.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:46.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:46.995 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:47.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:47.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:47.163 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:47.265 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:47 smithi169 bash[19604]: cluster 2024-02-12T16:06:46.261267+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e199: 12 total, 12 up, 12 in 2024-02-12T16:06:47.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:47.302 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:47.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:47.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:47.552 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:47 smithi121 bash[18489]: cluster 2024-02-12T16:06:46.261267+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e199: 12 total, 12 up, 12 in 2024-02-12T16:06:47.571 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:47.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:47 smithi123 bash[19119]: cluster 2024-02-12T16:06:46.261267+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e199: 12 total, 12 up, 12 in 2024-02-12T16:06:47.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:47.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:47.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:47.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:47.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:47.958 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:47.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:48.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:48.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:48.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:48.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:48.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:48.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:48.280 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T16:06:48.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:48.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:48.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:48.458 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:48 smithi121 bash[18489]: cluster 2024-02-12T16:06:47.575607+0000 mgr.y (mgr.14146) 638 : cluster [DBG] pgmap v744: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 11464/206764 objects misplaced (5.544%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:06:48.458 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:48 smithi121 bash[18489]: audit 2024-02-12T16:06:47.868770+0000 mon.a (mon.0) 980 : audit [INF] from='client.? v1:172.21.15.121:0/1239303063' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:06:48.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:48.494 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:06:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:48.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:06:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:48 smithi169 bash[19604]: cluster 2024-02-12T16:06:47.575607+0000 mgr.y (mgr.14146) 638 : cluster [DBG] pgmap v744: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 11464/206764 objects misplaced (5.544%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:06:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:48 smithi169 bash[19604]: audit 2024-02-12T16:06:47.868770+0000 mon.a (mon.0) 980 : audit [INF] from='client.? v1:172.21.15.121:0/1239303063' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:06:48.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:48.605 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:48 smithi123 bash[19119]: cluster 2024-02-12T16:06:47.575607+0000 mgr.y (mgr.14146) 638 : cluster [DBG] pgmap v744: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 11464/206764 objects misplaced (5.544%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:06:48.605 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:48 smithi123 bash[19119]: audit 2024-02-12T16:06:47.868770+0000 mon.a (mon.0) 980 : audit [INF] from='client.? v1:172.21.15.121:0/1239303063' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:06:48.623 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:48.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:48.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:48.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:48.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:48.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:48.775 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:06:48.809 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:06:48.845 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:06:48.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:48.876 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:06:48.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:48.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:48.908 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:06:48.937 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:06:48.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:06:48.996 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:06:49.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:49.029 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:06:49.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:49.060 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:06:49.097 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:06:49.126 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:06:49.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:49.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:49.284 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:49 smithi121 bash[18489]: audit 2024-02-12T16:06:48.269458+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.121:0/1239303063' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:06:49.284 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:49 smithi121 bash[18489]: cluster 2024-02-12T16:06:48.269516+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e200: 12 total, 12 up, 12 in 2024-02-12T16:06:49.284 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:49 smithi121 bash[18489]: audit 2024-02-12T16:06:48.730191+0000 mon.b (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.121:0/2025954315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:49.301 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:49.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:49.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:49 smithi169 bash[19604]: audit 2024-02-12T16:06:48.269458+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.121:0/1239303063' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:06:49.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:49 smithi169 bash[19604]: cluster 2024-02-12T16:06:48.269516+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e200: 12 total, 12 up, 12 in 2024-02-12T16:06:49.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:49 smithi169 bash[19604]: audit 2024-02-12T16:06:48.730191+0000 mon.b (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.121:0/2025954315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:49.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:49.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:49.545 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:49 smithi123 bash[19119]: audit 2024-02-12T16:06:48.269458+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.121:0/1239303063' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:06:49.545 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:49 smithi123 bash[19119]: cluster 2024-02-12T16:06:48.269516+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e200: 12 total, 12 up, 12 in 2024-02-12T16:06:49.545 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:49 smithi123 bash[19119]: audit 2024-02-12T16:06:48.730191+0000 mon.b (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.121:0/2025954315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:06:49.566 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:49.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:49.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:49.836 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:49.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T16:06:49.938 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:49.965 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:49.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:50.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:50.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:50.213 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:06:50.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:50.256 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:50.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:50.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:50.383 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:50.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:50.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:50 smithi121 bash[18489]: cluster 2024-02-12T16:06:48.948181+0000 osd.3 (osd.3) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:50.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:50 smithi121 bash[18489]: cluster 2024-02-12T16:06:48.948238+0000 osd.3 (osd.3) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:50.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:50 smithi121 bash[18489]: cluster 2024-02-12T16:06:49.281995+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e201: 12 total, 12 up, 12 in 2024-02-12T16:06:50.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:50 smithi121 bash[18489]: cluster 2024-02-12T16:06:49.576016+0000 mgr.y (mgr.14146) 639 : cluster [DBG] pgmap v747: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.43k op/s; 11061/202104 objects misplaced (5.473%); 6.3 MiB/s, 100 objects/s recovering 2024-02-12T16:06:50.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:50 smithi121 bash[18489]: cluster 2024-02-12T16:06:50.252181+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e202: 12 total, 12 up, 12 in 2024-02-12T16:06:50.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:50.513 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:50.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:50.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:50 smithi169 bash[19604]: cluster 2024-02-12T16:06:48.948181+0000 osd.3 (osd.3) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:50 smithi169 bash[19604]: cluster 2024-02-12T16:06:48.948238+0000 osd.3 (osd.3) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:50 smithi169 bash[19604]: cluster 2024-02-12T16:06:49.281995+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e201: 12 total, 12 up, 12 in 2024-02-12T16:06:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:50 smithi169 bash[19604]: cluster 2024-02-12T16:06:49.576016+0000 mgr.y (mgr.14146) 639 : cluster [DBG] pgmap v747: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.43k op/s; 11061/202104 objects misplaced (5.473%); 6.3 MiB/s, 100 objects/s recovering 2024-02-12T16:06:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:50 smithi169 bash[19604]: cluster 2024-02-12T16:06:50.252181+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e202: 12 total, 12 up, 12 in 2024-02-12T16:06:50.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:50.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:50 smithi123 bash[19119]: cluster 2024-02-12T16:06:48.948181+0000 osd.3 (osd.3) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:50.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:50 smithi123 bash[19119]: cluster 2024-02-12T16:06:48.948238+0000 osd.3 (osd.3) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:50.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:50 smithi123 bash[19119]: cluster 2024-02-12T16:06:49.281995+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e201: 12 total, 12 up, 12 in 2024-02-12T16:06:50.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:50 smithi123 bash[19119]: cluster 2024-02-12T16:06:49.576016+0000 mgr.y (mgr.14146) 639 : cluster [DBG] pgmap v747: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.43k op/s; 11061/202104 objects misplaced (5.473%); 6.3 MiB/s, 100 objects/s recovering 2024-02-12T16:06:50.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:50 smithi123 bash[19119]: cluster 2024-02-12T16:06:50.252181+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e202: 12 total, 12 up, 12 in 2024-02-12T16:06:50.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:50.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:50.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:50.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:50.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:50.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:50.908 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:50.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:51.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:51.036 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:51.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:51.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:51.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:51.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:51.307 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:51.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:51.313 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:51 smithi123 bash[19119]: audit 2024-02-12T16:06:51.026848+0000 mon.a (mon.0) 985 : audit [INF] from='client.? v1:172.21.15.121:0/1932799666' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:06:51.331 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:06:51.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:51 smithi121 bash[18489]: audit 2024-02-12T16:06:51.026848+0000 mon.a (mon.0) 985 : audit [INF] from='client.? v1:172.21.15.121:0/1932799666' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:06:51.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:51.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:51.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:51 smithi169 bash[19604]: audit 2024-02-12T16:06:51.026848+0000 mon.a (mon.0) 985 : audit [INF] from='client.? v1:172.21.15.121:0/1932799666' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:06:51.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:51.585 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:51.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:51.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:51.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:51.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:51.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:51.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:51.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:51.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:51.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:52.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:52.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:52.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:52.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:52.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:52.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:52.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:52.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:06:52.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:52.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:52.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:52.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:52.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:52 smithi169 bash[19604]: cluster 2024-02-12T16:06:51.304390+0000 mon.a (mon.0) 986 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:52.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:52 smithi169 bash[19604]: audit 2024-02-12T16:06:51.316341+0000 mon.a (mon.0) 987 : audit [INF] from='client.? v1:172.21.15.121:0/1932799666' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:06:52.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:52 smithi169 bash[19604]: cluster 2024-02-12T16:06:51.316441+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e203: 12 total, 12 up, 12 in 2024-02-12T16:06:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:52 smithi169 bash[19604]: cluster 2024-02-12T16:06:51.576464+0000 mgr.y (mgr.14146) 640 : cluster [DBG] pgmap v750: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 10683/196016 objects misplaced (5.450%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:06:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:52 smithi169 bash[19604]: cluster 2024-02-12T16:06:51.795439+0000 osd.9 (osd.9) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:52 smithi169 bash[19604]: cluster 2024-02-12T16:06:51.795498+0000 osd.9 (osd.9) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:52 smithi169 bash[19604]: audit 2024-02-12T16:06:51.997549+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:06:52.581 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:52.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:52 smithi121 bash[18489]: cluster 2024-02-12T16:06:51.304390+0000 mon.a (mon.0) 986 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:52.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:52 smithi121 bash[18489]: audit 2024-02-12T16:06:51.316341+0000 mon.a (mon.0) 987 : audit [INF] from='client.? v1:172.21.15.121:0/1932799666' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:06:52.587 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:52 smithi121 bash[18489]: cluster 2024-02-12T16:06:51.316441+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e203: 12 total, 12 up, 12 in 2024-02-12T16:06:52.587 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:52 smithi121 bash[18489]: cluster 2024-02-12T16:06:51.576464+0000 mgr.y (mgr.14146) 640 : cluster [DBG] pgmap v750: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 10683/196016 objects misplaced (5.450%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:06:52.587 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:52 smithi121 bash[18489]: cluster 2024-02-12T16:06:51.795439+0000 osd.9 (osd.9) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:52.587 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:52 smithi121 bash[18489]: cluster 2024-02-12T16:06:51.795498+0000 osd.9 (osd.9) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:52.587 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:52 smithi121 bash[18489]: audit 2024-02-12T16:06:51.997549+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:06:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:52.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:52.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:52 smithi123 bash[19119]: cluster 2024-02-12T16:06:51.304390+0000 mon.a (mon.0) 986 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:06:52.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:52 smithi123 bash[19119]: audit 2024-02-12T16:06:51.316341+0000 mon.a (mon.0) 987 : audit [INF] from='client.? v1:172.21.15.121:0/1932799666' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:06:52.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:52 smithi123 bash[19119]: cluster 2024-02-12T16:06:51.316441+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e203: 12 total, 12 up, 12 in 2024-02-12T16:06:52.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:52 smithi123 bash[19119]: cluster 2024-02-12T16:06:51.576464+0000 mgr.y (mgr.14146) 640 : cluster [DBG] pgmap v750: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 10683/196016 objects misplaced (5.450%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:06:52.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:52 smithi123 bash[19119]: cluster 2024-02-12T16:06:51.795439+0000 osd.9 (osd.9) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:52.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:52 smithi123 bash[19119]: cluster 2024-02-12T16:06:51.795498+0000 osd.9 (osd.9) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:52.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:52 smithi123 bash[19119]: audit 2024-02-12T16:06:51.997549+0000 mon.a (mon.0) 989 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:06:52.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:52.719 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:52.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:52.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:52.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:52.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:52.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:53.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:53.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:53.159 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:53.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:53.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:53 smithi121 bash[18489]: cluster 2024-02-12T16:06:52.323577+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e204: 12 total, 12 up, 12 in 2024-02-12T16:06:53.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:53.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:53.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:53.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:53.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:53 smithi169 bash[19604]: cluster 2024-02-12T16:06:52.323577+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e204: 12 total, 12 up, 12 in 2024-02-12T16:06:53.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:53 smithi169 bash[19604]: audit 2024-02-12T16:06:52.358598+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:06:53.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:53 smithi169 bash[19604]: audit 2024-02-12T16:06:52.359477+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:06:53.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:53 smithi169 bash[19604]: audit 2024-02-12T16:06:52.365512+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:06:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:53 smithi123 bash[19119]: cluster 2024-02-12T16:06:52.323577+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e204: 12 total, 12 up, 12 in 2024-02-12T16:06:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:53 smithi123 bash[19119]: audit 2024-02-12T16:06:52.358598+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:06:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:53 smithi123 bash[19119]: audit 2024-02-12T16:06:52.359477+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:06:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:53 smithi123 bash[19119]: audit 2024-02-12T16:06:52.365512+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:06:53.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:53 smithi121 bash[18489]: audit 2024-02-12T16:06:52.358598+0000 mon.a (mon.0) 991 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:06:53.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:53 smithi121 bash[18489]: audit 2024-02-12T16:06:52.359477+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:06:53.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:53 smithi121 bash[18489]: audit 2024-02-12T16:06:52.365512+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:06:53.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:53.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:53.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:53.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:53.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:53.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:06:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:53.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:06:54.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:54.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:54.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:54.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:54.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:54.226 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:54.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:54.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:54.336 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:54 smithi123 bash[19119]: cluster 2024-02-12T16:06:53.129851+0000 osd.11 (osd.11) 19 : cluster [DBG] 2.c scrub starts 2024-02-12T16:06:54.336 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:54 smithi123 bash[19119]: cluster 2024-02-12T16:06:53.583290+0000 mgr.y (mgr.14146) 641 : cluster [DBG] pgmap v752: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 10683/194516 objects misplaced (5.492%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T16:06:54.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:54.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:54.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:06:54.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:54.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:54.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:54.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:54.617 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:54 smithi121 bash[18489]: cluster 2024-02-12T16:06:53.129851+0000 osd.11 (osd.11) 19 : cluster [DBG] 2.c scrub starts 2024-02-12T16:06:54.617 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:54 smithi121 bash[18489]: cluster 2024-02-12T16:06:53.583290+0000 mgr.y (mgr.14146) 641 : cluster [DBG] pgmap v752: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 10683/194516 objects misplaced (5.492%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T16:06:54.666 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:54.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:54.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:54.779 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:54 smithi169 bash[19604]: cluster 2024-02-12T16:06:53.129851+0000 osd.11 (osd.11) 19 : cluster [DBG] 2.c scrub starts 2024-02-12T16:06:54.779 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:54 smithi169 bash[19604]: cluster 2024-02-12T16:06:53.583290+0000 mgr.y (mgr.14146) 641 : cluster [DBG] pgmap v752: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 10683/194516 objects misplaced (5.492%); 2.7 MiB/s, 43 objects/s recovering 2024-02-12T16:06:54.795 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:54.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:54.814 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:06:54.815 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:06:54.890 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:54.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:54.924 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:54.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:55.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:55.052 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:55.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:55.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:55.184 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:55.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:55.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:55.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:55.450 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:55.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:55.577 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:55.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:06:55.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:55.717 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:55.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:55.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:55.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:55.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:56.099 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:56.126 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:56.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:56.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:56.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:56.264 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:56 smithi169 bash[19604]: cluster 2024-02-12T16:06:55.583697+0000 mgr.y (mgr.14146) 642 : cluster [DBG] pgmap v753: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 10258/188324 objects misplaced (5.447%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T16:06:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:56.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:56.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:56.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:56.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:56.512 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:56.519 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:56 smithi123 bash[19119]: cluster 2024-02-12T16:06:55.583697+0000 mgr.y (mgr.14146) 642 : cluster [DBG] pgmap v753: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 10258/188324 objects misplaced (5.447%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T16:06:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:56.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:56.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:56.647 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:56 smithi121 bash[18489]: cluster 2024-02-12T16:06:55.583697+0000 mgr.y (mgr.14146) 642 : cluster [DBG] pgmap v753: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 10258/188324 objects misplaced (5.447%); 4.2 MiB/s, 66 objects/s recovering 2024-02-12T16:06:56.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:56.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:56.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:56.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:56.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:56.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:56.922 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:56.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:56.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:57.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:57.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:57.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:57.227 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:57.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:57.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:57 smithi121 bash[18489]: cluster 2024-02-12T16:06:55.597149+0000 osd.11 (osd.11) 20 : cluster [DBG] 2.c scrub ok 2024-02-12T16:06:57.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:57 smithi121 bash[18489]: cluster 2024-02-12T16:06:55.823490+0000 osd.11 (osd.11) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:57.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:57 smithi121 bash[18489]: cluster 2024-02-12T16:06:55.823544+0000 osd.11 (osd.11) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:57.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:57 smithi121 bash[18489]: cluster 2024-02-12T16:06:56.260098+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e205: 12 total, 12 up, 12 in 2024-02-12T16:06:57.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:57 smithi121 bash[18489]: cluster 2024-02-12T16:06:56.409422+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:57.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:57 smithi121 bash[18489]: cluster 2024-02-12T16:06:56.409481+0000 osd.6 (osd.6) 23 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:57.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:57.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:57.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:57.509 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:57.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:57.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:57 smithi169 bash[19604]: cluster 2024-02-12T16:06:55.597149+0000 osd.11 (osd.11) 20 : cluster [DBG] 2.c scrub ok 2024-02-12T16:06:57.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:57 smithi169 bash[19604]: cluster 2024-02-12T16:06:55.823490+0000 osd.11 (osd.11) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:57 smithi169 bash[19604]: cluster 2024-02-12T16:06:55.823544+0000 osd.11 (osd.11) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:57 smithi169 bash[19604]: cluster 2024-02-12T16:06:56.260098+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e205: 12 total, 12 up, 12 in 2024-02-12T16:06:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:57 smithi169 bash[19604]: cluster 2024-02-12T16:06:56.409422+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:57 smithi169 bash[19604]: cluster 2024-02-12T16:06:56.409481+0000 osd.6 (osd.6) 23 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:57.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:57.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:57 smithi123 bash[19119]: cluster 2024-02-12T16:06:55.597149+0000 osd.11 (osd.11) 20 : cluster [DBG] 2.c scrub ok 2024-02-12T16:06:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:57 smithi123 bash[19119]: cluster 2024-02-12T16:06:55.823490+0000 osd.11 (osd.11) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:57 smithi123 bash[19119]: cluster 2024-02-12T16:06:55.823544+0000 osd.11 (osd.11) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:57 smithi123 bash[19119]: cluster 2024-02-12T16:06:56.260098+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e205: 12 total, 12 up, 12 in 2024-02-12T16:06:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:57 smithi123 bash[19119]: cluster 2024-02-12T16:06:56.409422+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:06:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:57 smithi123 bash[19119]: cluster 2024-02-12T16:06:56.409481+0000 osd.6 (osd.6) 23 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:06:57.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:06:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:57.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:06:57.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:57.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:06:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:57.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:06:57.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:57.905 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:57.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:57.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:58.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:58.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:58.139 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:58.163 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:58.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:58.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:06:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:06:58.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:58.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:58.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:58.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:58 smithi169 bash[19604]: cluster 2024-02-12T16:06:57.264753+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e206: 12 total, 12 up, 12 in 2024-02-12T16:06:58.532 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:06:58 smithi169 bash[19604]: cluster 2024-02-12T16:06:57.584112+0000 mgr.y (mgr.14146) 643 : cluster [DBG] pgmap v756: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 9914/182256 objects misplaced (5.440%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:06:58.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:58 smithi123 bash[19119]: cluster 2024-02-12T16:06:57.264753+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e206: 12 total, 12 up, 12 in 2024-02-12T16:06:58.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:06:58 smithi123 bash[19119]: cluster 2024-02-12T16:06:57.584112+0000 mgr.y (mgr.14146) 643 : cluster [DBG] pgmap v756: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 9914/182256 objects misplaced (5.440%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:06:58.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:58.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:58 smithi121 bash[18489]: cluster 2024-02-12T16:06:57.264753+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e206: 12 total, 12 up, 12 in 2024-02-12T16:06:58.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:06:58 smithi121 bash[18489]: cluster 2024-02-12T16:06:57.584112+0000 mgr.y (mgr.14146) 643 : cluster [DBG] pgmap v756: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 9914/182256 objects misplaced (5.440%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:06:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:58.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:58.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:06:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:06:58.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:58.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:58.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:59.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:59.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:06:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:06:59.156 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:06:59.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:06:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:06:59.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:59.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:59.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:59.466 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:06:59.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:59.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:06:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:06:59.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:59.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:06:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:06:59.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:59.857 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:06:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:06:59.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:59.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:06:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:06:59.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:00.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:00.125 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:00.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:00.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:00.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:00.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:00.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:00.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:00 smithi123 bash[19119]: cluster 2024-02-12T16:06:58.902712+0000 osd.8 (osd.8) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:00.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:00 smithi123 bash[19119]: cluster 2024-02-12T16:06:58.902773+0000 osd.8 (osd.8) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:00.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:00 smithi123 bash[19119]: cluster 2024-02-12T16:06:58.926881+0000 osd.5 (osd.5) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:00.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:00 smithi123 bash[19119]: cluster 2024-02-12T16:06:58.926933+0000 osd.5 (osd.5) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:00.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:00 smithi123 bash[19119]: cluster 2024-02-12T16:06:59.273211+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e207: 12 total, 12 up, 12 in 2024-02-12T16:07:00.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:00 smithi123 bash[19119]: cluster 2024-02-12T16:06:59.584552+0000 mgr.y (mgr.14146) 644 : cluster [DBG] pgmap v758: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 9554/177806 objects misplaced (5.373%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:07:00.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:00 smithi123 bash[19119]: cluster 2024-02-12T16:07:00.252974+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e208: 12 total, 12 up, 12 in 2024-02-12T16:07:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:00.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:00.429 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:07:00.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:00.515 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:00.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:00 smithi169 bash[19604]: cluster 2024-02-12T16:06:58.902712+0000 osd.8 (osd.8) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:00 smithi169 bash[19604]: cluster 2024-02-12T16:06:58.902773+0000 osd.8 (osd.8) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:00 smithi169 bash[19604]: cluster 2024-02-12T16:06:58.926881+0000 osd.5 (osd.5) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:00 smithi169 bash[19604]: cluster 2024-02-12T16:06:58.926933+0000 osd.5 (osd.5) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:00 smithi169 bash[19604]: cluster 2024-02-12T16:06:59.273211+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e207: 12 total, 12 up, 12 in 2024-02-12T16:07:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:00 smithi169 bash[19604]: cluster 2024-02-12T16:06:59.584552+0000 mgr.y (mgr.14146) 644 : cluster [DBG] pgmap v758: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 9554/177806 objects misplaced (5.373%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:07:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:00 smithi169 bash[19604]: cluster 2024-02-12T16:07:00.252974+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e208: 12 total, 12 up, 12 in 2024-02-12T16:07:00.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:00.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:00 smithi121 bash[18489]: cluster 2024-02-12T16:06:58.902712+0000 osd.8 (osd.8) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:00.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:00 smithi121 bash[18489]: cluster 2024-02-12T16:06:58.902773+0000 osd.8 (osd.8) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:00.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:00 smithi121 bash[18489]: cluster 2024-02-12T16:06:58.926881+0000 osd.5 (osd.5) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:00.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:00 smithi121 bash[18489]: cluster 2024-02-12T16:06:58.926933+0000 osd.5 (osd.5) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:00.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:00 smithi121 bash[18489]: cluster 2024-02-12T16:06:59.273211+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e207: 12 total, 12 up, 12 in 2024-02-12T16:07:00.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:00 smithi121 bash[18489]: cluster 2024-02-12T16:06:59.584552+0000 mgr.y (mgr.14146) 644 : cluster [DBG] pgmap v758: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 9554/177806 objects misplaced (5.373%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:07:00.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:00 smithi121 bash[18489]: cluster 2024-02-12T16:07:00.252974+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e208: 12 total, 12 up, 12 in 2024-02-12T16:07:00.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:00.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:00.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:00.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:00.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:00.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:00.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:00.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:00.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:01.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:01.042 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:01.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:01.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:01.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:01.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:01.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:01.336 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:01.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:01.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:01.477 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:01.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:01.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:01.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:01.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:01.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:01.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:01.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:01.881 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:01.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:01.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:02.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:02.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:02.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:02.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:02.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:02.411 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:02.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:02.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:02.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:02.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:02.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:02.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:02.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:02 smithi121 bash[18489]: cluster 2024-02-12T16:07:01.585007+0000 mgr.y (mgr.14146) 645 : cluster [DBG] pgmap v760: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.54k op/s; 9208/171882 objects misplaced (5.357%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:07:02.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:02.926 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:02 smithi169 bash[19604]: cluster 2024-02-12T16:07:01.585007+0000 mgr.y (mgr.14146) 645 : cluster [DBG] pgmap v760: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.54k op/s; 9208/171882 objects misplaced (5.357%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:07:02.944 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:02.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:03.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:03.056 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:02 smithi123 bash[19119]: cluster 2024-02-12T16:07:01.585007+0000 mgr.y (mgr.14146) 645 : cluster [DBG] pgmap v760: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.54k op/s; 9208/171882 objects misplaced (5.357%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:07:03.073 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:03.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:03.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:03.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:03.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:03.481 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:03.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:03.608 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:03.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:03 smithi169 bash[19604]: audit 2024-02-12T16:07:02.905643+0000 mon.a (mon.0) 998 : audit [INF] from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:07:03.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:03 smithi169 bash[19604]: audit 2024-02-12T16:07:02.908399+0000 mon.c (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.121:0/2935745573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:07:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T16:07:03.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:03.720 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:03 smithi123 bash[19119]: audit 2024-02-12T16:07:02.905643+0000 mon.a (mon.0) 998 : audit [INF] from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:07:03.720 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:03 smithi123 bash[19119]: audit 2024-02-12T16:07:02.908399+0000 mon.c (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.121:0/2935745573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:07:03.727 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:03 smithi121 bash[18489]: audit 2024-02-12T16:07:02.905643+0000 mon.a (mon.0) 998 : audit [INF] from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:07:03.727 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:03 smithi121 bash[18489]: audit 2024-02-12T16:07:02.908399+0000 mon.c (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.121:0/2935745573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:07:03.736 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:03.785 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:07:03.812 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:07:03.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:03.844 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:07:03.866 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:03.878 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:07:03.913 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:07:03.946 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:07:03.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:03.979 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:07:03.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:03.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:04.005 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:07:04.026 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:07:04.046 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:07:04.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:07:04.098 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:07:04.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:07:04.125 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:04.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:04.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:04.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:04.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:04.428 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:04.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:04.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:04.561 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:04.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:04.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:04.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:04.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:04.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:04.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:04 smithi123 bash[19119]: cluster 2024-02-12T16:07:03.551813+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:04.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:04 smithi123 bash[19119]: cluster 2024-02-12T16:07:03.551863+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:04.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:04 smithi123 bash[19119]: cluster 2024-02-12T16:07:03.585392+0000 mgr.y (mgr.14146) 646 : cluster [DBG] pgmap v761: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 9208/170364 objects misplaced (5.405%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:07:04.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:04 smithi123 bash[19119]: audit 2024-02-12T16:07:03.647538+0000 mon.a (mon.0) 999 : audit [INF] from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:07:04.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:04 smithi123 bash[19119]: cluster 2024-02-12T16:07:03.647625+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e209: 12 total, 12 up, 12 in 2024-02-12T16:07:04.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:04 smithi123 bash[19119]: audit 2024-02-12T16:07:03.753882+0000 mon.b (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.121:0/3489137178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:04.836 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:04.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:04.881 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:07:04.881 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:07:04.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:9665 2.d e152/147 2:be7de64e:::benchmark_data_smithi169_42458_object9664:head v 152'636, mlcod=152'634)", 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:04:20.237780+0000", 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 164.639438928, 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018845417999999999, 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237780+0000", 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237780+0000", 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237779+0000", 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4888000000000004e-05 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237874+0000", 2024-02-12T16:07:04.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3467e-05 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237887+0000", 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6799999999999998e-06 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237893+0000", 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4140000000000001e-05 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237937+0000", 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2656e-05 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.237950+0000", 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018645261 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.256595+0000", 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0886999999999998e-05 2024-02-12T16:07:04.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.256626+0000", 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018845417999999999 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:9677 2.d e152/147 2:b8b50609:::benchmark_data_smithi169_42458_object9676:head v 152'637, mlcod=152'634)", 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:04:20.240250+0000", 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 164.636969722, 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017801615, 2024-02-12T16:07:04.886 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240250+0000", 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240250+0000", 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240248+0000", 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7106000000000003e-05 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240335+0000", 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0530000000000001e-05 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240345+0000", 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3529999999999997e-06 2024-02-12T16:07:04.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240353+0000", 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7141000000000002e-05 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240390+0000", 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7889000000000001e-05 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.240408+0000", 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017598272000000002 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.258006+0000", 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5178e-05 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.258051+0000", 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017801615 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.890 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:9810 2.d e152/147 2:be99bf19:::benchmark_data_smithi169_42458_object9809:head v 152'642, mlcod=152'640)", 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:04:20.285470+0000", 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 164.59174950799999, 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020305178, 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285470+0000", 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285470+0000", 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999938 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285463+0000", 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9976999999999999e-05 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285523+0000", 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0028e-05 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285533+0000", 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6190000000000002e-06 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285538+0000", 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9379999999999999e-05 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285577+0000", 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2211000000000001e-05 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.285600+0000", 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020137859000000001 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.305738+0000", 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.752e-05 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:20.305775+0000", 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020305178 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.895 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:101180 2.d e165/147 2:baa37fdc:::benchmark_data_smithi169_42458_object101179:head v 165'6416, mlcod=165'6414)", 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:04:55.252524+0000", 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 129.62469536099999, 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021537298999999999, 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252524+0000", 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252524+0000", 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252513+0000", 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3443999999999999e-05 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252567+0000", 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4462999999999999e-05 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252591+0000", 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0650000000000005e-06 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252600+0000", 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1167999999999994e-05 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252661+0000", 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5341999999999999e-05 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.252687+0000", 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021324124 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.274011+0000", 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0445999999999998e-05 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:04:55.274061+0000", 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021537298999999999 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:190127 2.11 e187/178 2:89093cb3:::benchmark_data_smithi169_42458_object17093:head v 187'10535, mlcod=152'1042)", 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:02.320478+0000", 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.556741160999998, 2024-02-12T16:07:04.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017404670000000001, 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320478+0000", 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320478+0000", 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320477+0000", 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6750000000000008e-06 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320485+0000", 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0975000000000001e-05 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320496+0000", 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1659999999999999e-06 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320502+0000", 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6400000000000002e-05 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320559+0000", 2024-02-12T16:07:04.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6886999999999999e-05 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.320586+0000", 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017261379 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.337847+0000", 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5741999999999998e-05 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:02.337883+0000", 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017404670000000001 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:192666 2.11 e187/178 2:8b0afc2d:::benchmark_data_smithi169_42458_object19632:head v 187'10624, mlcod=152'1203)", 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:04.286781+0000", 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 60.590438605999999, 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020628878999999999, 2024-02-12T16:07:04.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286781+0000", 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286781+0000", 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:07:04.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286780+0000", 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2840000000000001e-06 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286787+0000", 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5021000000000001e-05 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286802+0000", 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9920000000000004e-06 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286808+0000", 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7521000000000002e-05 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286845+0000", 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9125999999999999e-05 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.286874+0000", 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020497372 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.307372+0000", 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7815999999999998e-05 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.307410+0000", 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020628878999999999 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:193294 2.d e188/147 2:b60fa4bf:::benchmark_data_smithi169_42458_object20260:head v 188'12140, mlcod=188'12138)", 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:05.277987+0000", 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.599232649999998, 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019890788999999999, 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.910 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.277987+0000", 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.277987+0000", 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.277985+0000", 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3875e-05 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.277999+0000", 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6208000000000001e-05 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.278015+0000", 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1450000000000008e-06 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.278023+0000", 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0722e-05 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.278074+0000", 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5306000000000001e-05 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.278099+0000", 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019731301 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.297830+0000", 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7061000000000002e-05 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.297878+0000", 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019890788999999999 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:193661 2.11 e188/178 2:8fc93381:::benchmark_data_smithi169_42458_object20627:head v 188'10653, mlcod=152'1256)", 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:05.601663+0000", 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.275556764000001, 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021020086, 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.601663+0000", 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.601663+0000", 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.601661+0000", 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4011e-05 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.601675+0000", 2024-02-12T16:07:04.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6509000000000001e-05 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.601692+0000", 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1160000000000001e-06 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.601699+0000", 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001170372 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.602869+0000", 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3085999999999999e-05 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.602892+0000", 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019744787999999999 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.622637+0000", 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5806000000000002e-05 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.622683+0000", 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021020086 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:193668 2.11 e188/178 2:8f394fad:::benchmark_data_smithi169_42458_object20634:head v 188'10654, mlcod=152'1263)", 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:05.615298+0000", 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.261921528999999, 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023976404999999999, 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.615298+0000", 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.615298+0000", 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.615296+0000", 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1439e-05 2024-02-12T16:07:04.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.615307+0000", 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7283e-05 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.615324+0000", 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.157e-06 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.615333+0000", 2024-02-12T16:07:04.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001771463 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.617104+0000", 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7169000000000003e-05 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.617141+0000", 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022116186999999999 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.639257+0000", 2024-02-12T16:07:04.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6940999999999999e-05 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.639274+0000", 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023976404999999999 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:198456 2.11 e189/178 2:884aadcf:::benchmark_data_smithi169_42458_object25422:head v 189'10798, mlcod=152'1555)", 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:09.270932+0000", 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 55.606287191, 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.02088636, 2024-02-12T16:07:04.923 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.270932+0000", 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.270932+0000", 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.270930+0000", 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5505000000000001e-05 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.270945+0000", 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0426000000000001e-05 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.270965+0000", 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.79e-06 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.270973+0000", 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8452000000000003e-05 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.271022+0000", 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0910000000000001e-05 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.271053+0000", 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020712679000000001 2024-02-12T16:07:04.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.291765+0000", 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3224999999999999e-05 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:09.291819+0000", 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.02088636 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:200177 2.d e190/147 2:b9064bc4:::benchmark_data_smithi169_42458_object27143:head v 190'12589, mlcod=190'12585)", 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:10.705461+0000", 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.171758052000001, 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018851349999999999, 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705461+0000", 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705461+0000", 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705459+0000", 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8408999999999999e-05 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705487+0000", 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5150999999999999e-05 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705503+0000", 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4989999999999997e-06 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705510+0000", 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1564000000000003e-05 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705562+0000", 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013484099999999999 2024-02-12T16:07:04.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.705696+0000", 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012897862 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.718594+0000", 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057183260000000001 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:10.724313+0000", 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018851349999999999 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:236358 2.d e198/147 2:b8c58c01:::benchmark_data_smithi169_42458_object63324:head v 198'14802, mlcod=198'14800)", 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:40.277504+0000", 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.599715538000002, 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022200948000000002, 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.932 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277504+0000", 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277504+0000", 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277502+0000", 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.313e-05 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277515+0000", 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6988000000000001e-05 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277532+0000", 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5859999999999999e-06 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277540+0000", 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3468999999999998e-05 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277603+0000", 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3658999999999997e-05 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.277637+0000", 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022019515999999999 2024-02-12T16:07:04.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.299656+0000", 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8572000000000002e-05 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:40.299705+0000", 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022200948000000002 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:239080 2.d e198/147 2:b5cd159c:::benchmark_data_smithi169_42458_object66046:head v 198'14994, mlcod=198'14991)", 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:42.352173+0000", 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.525046852999999, 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020164826, 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352173+0000", 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352173+0000", 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352170+0000", 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8142e-05 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352188+0000", 2024-02-12T16:07:04.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8178000000000001e-05 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352206+0000", 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3050000000000004e-06 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352213+0000", 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00051435599999999999 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352728+0000", 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8744e-05 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.352767+0000", 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019527441999999999 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.372294+0000", 2024-02-12T16:07:04.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3322000000000003e-05 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:42.372337+0000", 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020164826 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:240681 2.11 e198/178 2:8f5263a7:::benchmark_data_smithi169_42458_object67647:head v 198'12111, mlcod=160'4221)", 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:43.691400+0000", 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.185819794, 2024-02-12T16:07:04.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01780296, 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691400+0000", 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691400+0000", 2024-02-12T16:07:04.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691397+0000", 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3114e-05 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691410+0000", 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8705000000000001e-05 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691429+0000", 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0129999999999999e-06 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691438+0000", 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9416000000000003e-05 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691487+0000", 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4267999999999998e-05 2024-02-12T16:07:04.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.691522+0000", 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017633738999999999 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.709155+0000", 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7239e-05 2024-02-12T16:07:04.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:43.709203+0000", 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01780296 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.946 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:242645 2.11 e198/178 2:8e0f8e37:::benchmark_data_smithi169_42458_object69611:head v 198'12182, mlcod=160'4341)", 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:45.277658+0000", 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.599561803, 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023340339000000002, 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277658+0000", 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277658+0000", 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277655+0000", 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.358e-05 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277669+0000", 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5749999999999999e-05 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277694+0000", 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7850000000000003e-06 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277701+0000", 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1805999999999998e-05 2024-02-12T16:07:04.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277753+0000", 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8963e-05 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.277782+0000", 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023131558 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.300914+0000", 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4336e-05 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.300998+0000", 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023340339000000002 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:242897 2.d e198/147 2:be78eff1:::benchmark_data_smithi169_42458_object69863:head v 198'15252, mlcod=198'15249)", 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:45.473438+0000", 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.403781015, 2024-02-12T16:07:04.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017679443, 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.473438+0000", 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.473438+0000", 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.473437+0000", 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3070000000000003e-06 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.473445+0000", 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.695e-06 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.473453+0000", 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6100000000000002e-06 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.473457+0000", 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040884930000000003 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.477545+0000", 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6547000000000001e-05 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.477582+0000", 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013495293 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.491077+0000", 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0680999999999999e-05 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:45.491118+0000", 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017679443 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:244090 2.d e199/147 2:bd717e0a:::benchmark_data_smithi169_42458_object71056:head v 199'15321, mlcod=199'15315)", 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:46.430214+0000", 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.447005424, 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017173198000000001, 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.430214+0000", 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.430214+0000", 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.430212+0000", 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2933e-05 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.430225+0000", 2024-02-12T16:07:04.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2261e-05 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.430237+0000", 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7630000000000003e-06 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.430244+0000", 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049926249999999997 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.435237+0000", 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1866999999999999e-05 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.435259+0000", 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012037861 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.447296+0000", 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0700999999999998e-05 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:46.447387+0000", 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017173198000000001 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:265850 2.d e209/147 2:b9c1e117:::benchmark_data_smithi169_42458_object92816:head v 209'16719, mlcod=209'16716)", 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:04.462455+0000", 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.41476434000000001, 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017568884, 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.960 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.462455+0000", 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.462455+0000", 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.462453+0000", 2024-02-12T16:07:04.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4450000000000002e-06 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.462463+0000", 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4207e-05 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.462477+0000", 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1340000000000001e-06 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.462483+0000", 2024-02-12T16:07:04.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037529439999999998 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.466236+0000", 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.951e-05 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.466266+0000", 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013713360000000001 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.479979+0000", 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4839999999999998e-05 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.480024+0000", 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017568884 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:266053 2.11 e209/178 2:8d8ec325:::benchmark_data_smithi169_42458_object93019:head v 209'12884, mlcod=163'5789)", 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:04.698476+0000", 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.17874358900000001, 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021410696999999999, 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.698476+0000", 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.698476+0000", 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.698474+0000", 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4408e-05 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.698488+0000", 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3572e-05 2024-02-12T16:07:04.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.698502+0000", 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3249999999999997e-06 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.698511+0000", 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00091928500000000005 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.699430+0000", 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5343000000000001e-05 2024-02-12T16:07:04.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.699456+0000", 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020379400999999998 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.719835+0000", 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1536000000000003e-05 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.719886+0000", 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021410696999999999 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.14811.0:266056 2.11 e209/178 2:8d23ccc2:::benchmark_data_smithi169_42458_object93022:head v 209'12885, mlcod=163'5789)", 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:04.704289+0000", 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.17293056700000001, 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020104909000000001, 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704289+0000", 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704289+0000", 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704287+0000", 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4069e-05 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704301+0000", 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0264e-05 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704311+0000", 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0389999999999999e-06 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:04.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704318+0000", 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6990000000000002e-05 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704365+0000", 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9598000000000003e-05 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.704404+0000", 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019898951000000002 2024-02-12T16:07:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.724303+0000", 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0311999999999999e-05 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:04.724394+0000", 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020104909000000001 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:04.974 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:04.974 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:07:04.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:04.975 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:04 smithi121 bash[18489]: cluster 2024-02-12T16:07:03.551813+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:04.975 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:04 smithi121 bash[18489]: cluster 2024-02-12T16:07:03.551863+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:04.975 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:04 smithi121 bash[18489]: cluster 2024-02-12T16:07:03.585392+0000 mgr.y (mgr.14146) 646 : cluster [DBG] pgmap v761: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 9208/170364 objects misplaced (5.405%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:07:04.975 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:04 smithi121 bash[18489]: audit 2024-02-12T16:07:03.647538+0000 mon.a (mon.0) 999 : audit [INF] from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:07:04.975 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:04 smithi121 bash[18489]: cluster 2024-02-12T16:07:03.647625+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e209: 12 total, 12 up, 12 in 2024-02-12T16:07:04.975 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:04 smithi121 bash[18489]: audit 2024-02-12T16:07:03.753882+0000 mon.b (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.121:0/3489137178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:05.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:05.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:04 smithi169 bash[19604]: cluster 2024-02-12T16:07:03.551813+0000 osd.4 (osd.4) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:04 smithi169 bash[19604]: cluster 2024-02-12T16:07:03.551863+0000 osd.4 (osd.4) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:04 smithi169 bash[19604]: cluster 2024-02-12T16:07:03.585392+0000 mgr.y (mgr.14146) 646 : cluster [DBG] pgmap v761: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 9208/170364 objects misplaced (5.405%); 3.5 MiB/s, 55 objects/s recovering 2024-02-12T16:07:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:04 smithi169 bash[19604]: audit 2024-02-12T16:07:03.647538+0000 mon.a (mon.0) 999 : audit [INF] from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:07:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:04 smithi169 bash[19604]: cluster 2024-02-12T16:07:03.647625+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e209: 12 total, 12 up, 12 in 2024-02-12T16:07:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:04 smithi169 bash[19604]: audit 2024-02-12T16:07:03.753882+0000 mon.b (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.121:0/3489137178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:05.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:05.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:05.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:05.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:05.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:05.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:05.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T16:07:05.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:05.563 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:05.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:05.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:07:05.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:05.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:05.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:05.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:05.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:06.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:06.125 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:06.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:06.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:06.258 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:06 smithi123 bash[19119]: cluster 2024-02-12T16:07:05.252067+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e210: 12 total, 12 up, 12 in 2024-02-12T16:07:06.258 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:06 smithi123 bash[19119]: cluster 2024-02-12T16:07:05.585802+0000 mgr.y (mgr.14146) 647 : cluster [DBG] pgmap v764: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 8842/164442 objects misplaced (5.377%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T16:07:06.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:06.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:06.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:06.406 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:06.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:06.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:06.519 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:06 smithi121 bash[18489]: cluster 2024-02-12T16:07:05.252067+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e210: 12 total, 12 up, 12 in 2024-02-12T16:07:06.520 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:06 smithi121 bash[18489]: cluster 2024-02-12T16:07:05.585802+0000 mgr.y (mgr.14146) 647 : cluster [DBG] pgmap v764: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 8842/164442 objects misplaced (5.377%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T16:07:06.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:06 smithi169 bash[19604]: cluster 2024-02-12T16:07:05.252067+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e210: 12 total, 12 up, 12 in 2024-02-12T16:07:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:06 smithi169 bash[19604]: cluster 2024-02-12T16:07:05.585802+0000 mgr.y (mgr.14146) 647 : cluster [DBG] pgmap v764: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 8842/164442 objects misplaced (5.377%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T16:07:06.553 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:06.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:06.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:06.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:06.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:06.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:06.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:06.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:06.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:06.963 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:06.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:07.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:07.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:07.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:07.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:07.222 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:07.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:07.282 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:07:07.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:07 smithi121 bash[18489]: audit 2024-02-12T16:07:06.323954+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? v1:172.21.15.121:0/1346947922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:07:07.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:07.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:07.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:07.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:07 smithi169 bash[19604]: audit 2024-02-12T16:07:06.323954+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? v1:172.21.15.121:0/1346947922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:07:07.487 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:07.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:07.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:07.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:07:07.602 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:07 smithi123 bash[19119]: audit 2024-02-12T16:07:06.323954+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? v1:172.21.15.121:0/1346947922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:07:07.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:07.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:07.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:07.749 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:07.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:07.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:07.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:07.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:07.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:08.004 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:08.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:08.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:08.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:08.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:08.266 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:08 smithi169 bash[19604]: cluster 2024-02-12T16:07:07.256488+0000 mon.a (mon.0) 1003 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:07:08.266 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:08 smithi169 bash[19604]: cluster 2024-02-12T16:07:07.256529+0000 mon.a (mon.0) 1004 : cluster [INF] Cluster is now healthy 2024-02-12T16:07:08.266 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:08 smithi169 bash[19604]: audit 2024-02-12T16:07:07.267581+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? v1:172.21.15.121:0/1346947922' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:07:08.266 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:08 smithi169 bash[19604]: cluster 2024-02-12T16:07:07.267658+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e211: 12 total, 12 up, 12 in 2024-02-12T16:07:08.266 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:08 smithi169 bash[19604]: cluster 2024-02-12T16:07:07.586187+0000 mgr.y (mgr.14146) 648 : cluster [DBG] pgmap v766: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 8448/158460 objects misplaced (5.331%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T16:07:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:08.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:08.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:08 smithi123 bash[19119]: cluster 2024-02-12T16:07:07.256488+0000 mon.a (mon.0) 1003 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:07:08.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:08 smithi123 bash[19119]: cluster 2024-02-12T16:07:07.256529+0000 mon.a (mon.0) 1004 : cluster [INF] Cluster is now healthy 2024-02-12T16:07:08.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:08 smithi123 bash[19119]: audit 2024-02-12T16:07:07.267581+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? v1:172.21.15.121:0/1346947922' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:07:08.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:08 smithi123 bash[19119]: cluster 2024-02-12T16:07:07.267658+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e211: 12 total, 12 up, 12 in 2024-02-12T16:07:08.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:08 smithi123 bash[19119]: cluster 2024-02-12T16:07:07.586187+0000 mgr.y (mgr.14146) 648 : cluster [DBG] pgmap v766: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 8448/158460 objects misplaced (5.331%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T16:07:08.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:08.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:08.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:08.553 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:08 smithi121 bash[18489]: cluster 2024-02-12T16:07:07.256488+0000 mon.a (mon.0) 1003 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:07:08.553 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:08 smithi121 bash[18489]: cluster 2024-02-12T16:07:07.256529+0000 mon.a (mon.0) 1004 : cluster [INF] Cluster is now healthy 2024-02-12T16:07:08.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:08 smithi121 bash[18489]: audit 2024-02-12T16:07:07.267581+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? v1:172.21.15.121:0/1346947922' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:07:08.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:08 smithi121 bash[18489]: cluster 2024-02-12T16:07:07.267658+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e211: 12 total, 12 up, 12 in 2024-02-12T16:07:08.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:08 smithi121 bash[18489]: cluster 2024-02-12T16:07:07.586187+0000 mgr.y (mgr.14146) 648 : cluster [DBG] pgmap v766: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 8448/158460 objects misplaced (5.331%); 4.0 MiB/s, 63 objects/s recovering 2024-02-12T16:07:08.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:08.657 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:08.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:08.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:08.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:08.925 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:08.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:09.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:09.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:09.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:09.188 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:09.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:09.317 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:09.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:09.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:09.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:09 smithi169 bash[19604]: cluster 2024-02-12T16:07:09.024374+0000 osd.11 (osd.11) 23 : cluster [DBG] 2.a scrub starts 2024-02-12T16:07:09.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:09.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:09 smithi123 bash[19119]: cluster 2024-02-12T16:07:09.024374+0000 osd.11 (osd.11) 23 : cluster [DBG] 2.a scrub starts 2024-02-12T16:07:09.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:09 smithi121 bash[18489]: cluster 2024-02-12T16:07:09.024374+0000 osd.11 (osd.11) 23 : cluster [DBG] 2.a scrub starts 2024-02-12T16:07:09.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:09.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:09.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:09.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:09.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:09.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:09.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:10.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:10.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:10.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:10.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:10.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:10.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:10.391 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:10 smithi121 bash[18489]: cluster 2024-02-12T16:07:09.586649+0000 mgr.y (mgr.14146) 649 : cluster [DBG] pgmap v767: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s; 8090/154142 objects misplaced (5.248%); 5.8 MiB/s, 93 objects/s recovering 2024-02-12T16:07:10.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:10.420 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:10.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:10.471 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:07:10.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:10.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:10.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:07:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:10.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:10.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:10.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:10 smithi121 bash[18489]: cluster 2024-02-12T16:07:10.253538+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e212: 12 total, 12 up, 12 in 2024-02-12T16:07:10.664 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:10 smithi169 bash[19604]: cluster 2024-02-12T16:07:09.586649+0000 mgr.y (mgr.14146) 649 : cluster [DBG] pgmap v767: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s; 8090/154142 objects misplaced (5.248%); 5.8 MiB/s, 93 objects/s recovering 2024-02-12T16:07:10.664 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:10 smithi169 bash[19604]: cluster 2024-02-12T16:07:10.253538+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e212: 12 total, 12 up, 12 in 2024-02-12T16:07:10.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:10.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:10.784 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:10.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:10.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:10.935 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:10.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:10.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:11.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:11.044 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:11.068 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:11.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:11.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:10 smithi123 bash[19119]: cluster 2024-02-12T16:07:09.586649+0000 mgr.y (mgr.14146) 649 : cluster [DBG] pgmap v767: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s; 8090/154142 objects misplaced (5.248%); 5.8 MiB/s, 93 objects/s recovering 2024-02-12T16:07:11.168 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:10 smithi123 bash[19119]: cluster 2024-02-12T16:07:10.253538+0000 mon.a (mon.0) 1007 : cluster [DBG] osdmap e212: 12 total, 12 up, 12 in 2024-02-12T16:07:11.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:11.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:11.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:11.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:11.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:11.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:11.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:11.480 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:11.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:11.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:11.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:11.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:11.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:11 smithi123 bash[19119]: cluster 2024-02-12T16:07:10.461260+0000 osd.11 (osd.11) 24 : cluster [DBG] 2.a scrub ok 2024-02-12T16:07:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:11 smithi121 bash[18489]: cluster 2024-02-12T16:07:10.461260+0000 osd.11 (osd.11) 24 : cluster [DBG] 2.a scrub ok 2024-02-12T16:07:11.723 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:11.748 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:11.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:11.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:11 smithi169 bash[19604]: cluster 2024-02-12T16:07:10.461260+0000 osd.11 (osd.11) 24 : cluster [DBG] 2.a scrub ok 2024-02-12T16:07:11.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:11.881 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:11.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:11.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:12.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:12.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:12.163 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:12.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:12.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:12.400 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:12 smithi169 bash[19604]: cluster 2024-02-12T16:07:11.587075+0000 mgr.y (mgr.14146) 650 : cluster [DBG] pgmap v769: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s; 7730/148194 objects misplaced (5.216%); 5.5 MiB/s, 87 objects/s recovering 2024-02-12T16:07:12.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:12.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:12.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:12.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:12.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:12 smithi121 bash[18489]: cluster 2024-02-12T16:07:11.587075+0000 mgr.y (mgr.14146) 650 : cluster [DBG] pgmap v769: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s; 7730/148194 objects misplaced (5.216%); 5.5 MiB/s, 87 objects/s recovering 2024-02-12T16:07:12.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:12.695 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:12 smithi123 bash[19119]: cluster 2024-02-12T16:07:11.587075+0000 mgr.y (mgr.14146) 650 : cluster [DBG] pgmap v769: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s; 7730/148194 objects misplaced (5.216%); 5.5 MiB/s, 87 objects/s recovering 2024-02-12T16:07:12.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:12.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:12.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:12.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:12.981 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:12.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:13.075 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:13.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:13.135 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:13.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:13.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:13.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:13.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:13.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:13.387 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:13.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:13.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:13.532 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:13.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:13.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:13.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:13.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:13.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:13.801 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:13.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:13.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:13.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:13.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:14.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:14.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:14.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:14.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:14.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:14.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:14.357 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:14.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:14.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:14.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:14.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:14.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:14.626 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:14.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:14.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:14.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:14.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:14.784 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:14 smithi123 bash[19119]: cluster 2024-02-12T16:07:13.587511+0000 mgr.y (mgr.14146) 651 : cluster [DBG] pgmap v770: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 7730/146632 objects misplaced (5.272%); 4.3 MiB/s, 69 objects/s recovering 2024-02-12T16:07:14.875 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:14.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:14.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:278420 2.12 2:49f226ce:::benchmark_data_smithi169_42458_object105386:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:14.883783+0000", 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.060788835999999999, 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.060832036999999999, 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:07:14.946 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 278420 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.883783+0000", 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.883783+0000", 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.883780+0000", 2024-02-12T16:07:14.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2424e-05 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.883793+0000", 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4725999999999999e-05 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.883808+0000", 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0699999999999996e-06 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:14.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.883817+0000", 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1172999999999999e-05 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.883868+0000", 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014734500000000001 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.884015+0000", 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00080738499999999998 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.884823+0000", 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034326890000000001 2024-02-12T16:07:14.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.888255+0000", 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044719779999999997 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:278436 2.13 2:c91ab338:::benchmark_data_smithi169_42458_object105402:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:14.900031+0000", 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.044540623000000001, 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044694004000000002, 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:14.950 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 278436 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900031+0000", 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900031+0000", 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900029+0000", 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0989999999999999e-06 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900037+0000", 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1522000000000001e-05 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900049+0000", 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8790000000000002e-06 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900054+0000", 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5079000000000001e-05 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900089+0000", 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3812000000000003e-05 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900173+0000", 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059184500000000002 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2", 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.900765+0000", 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029179729999999999 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.903683+0000", 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003651086 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:278444 2.13 2:cbcce20a:::benchmark_data_smithi169_42458_object105410:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:14.919128+0000", 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.025443865999999999, 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025678725999999999, 2024-02-12T16:07:14.954 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 278444 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919128+0000", 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919128+0000", 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999928 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919121+0000", 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0936000000000001e-05 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919142+0000", 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6358000000000001e-05 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919168+0000", 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0190999999999999e-05 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919179+0000", 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3117999999999999e-05 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919262+0000", 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019569100000000001 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.919457+0000", 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083660899999999999 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2", 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.920294+0000", 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039111170000000004 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:14.924205+0000", 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005076901 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 3 2024-02-12T16:07:14.958 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:07:14.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:14 smithi121 bash[18489]: cluster 2024-02-12T16:07:13.587511+0000 mgr.y (mgr.14146) 651 : cluster [DBG] pgmap v770: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 7730/146632 objects misplaced (5.272%); 4.3 MiB/s, 69 objects/s recovering 2024-02-12T16:07:15.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:15.013 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:14 smithi169 bash[19604]: cluster 2024-02-12T16:07:13.587511+0000 mgr.y (mgr.14146) 651 : cluster [DBG] pgmap v770: 27 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 24 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 7730/146632 objects misplaced (5.272%); 4.3 MiB/s, 69 objects/s recovering 2024-02-12T16:07:15.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:15.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:15.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:15.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:15.294 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:15.317 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:15.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:15.443 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:15.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:15.574 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:15.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:15.702 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:15.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:07:15.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:15.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:15.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:15.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:16.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:16.116 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:16.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:16.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:16.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:16.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:16.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:16.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:16.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:16.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:16.512 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:16.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:16 smithi169 bash[19604]: cluster 2024-02-12T16:07:15.587988+0000 mgr.y (mgr.14146) 652 : cluster [DBG] pgmap v771: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 7306/140606 objects misplaced (5.196%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:07:16.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:16.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:16 smithi123 bash[19119]: cluster 2024-02-12T16:07:15.587988+0000 mgr.y (mgr.14146) 652 : cluster [DBG] pgmap v771: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 7306/140606 objects misplaced (5.196%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:07:16.639 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:16.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:16.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:16 smithi121 bash[18489]: cluster 2024-02-12T16:07:15.587988+0000 mgr.y (mgr.14146) 652 : cluster [DBG] pgmap v771: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 7306/140606 objects misplaced (5.196%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:07:16.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:16.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:16.874 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:16.895 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:16.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:16.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:17.024 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:17.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:17.161 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:17.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:17.295 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:17.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:17.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:17.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:17.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:17.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:17.698 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:17.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:17.832 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:17.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:17.965 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:17.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:18.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:18.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:18.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:18.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:18.218 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:18.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:18.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:18.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:18.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:18.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:18.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:18.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:18.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:18.622 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:18.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:18.648 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:18 smithi121 bash[18489]: cluster 2024-02-12T16:07:17.588410+0000 mgr.y (mgr.14146) 653 : cluster [DBG] pgmap v772: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 6956/134440 objects misplaced (5.174%); 4.7 MiB/s, 74 objects/s recovering 2024-02-12T16:07:18.648 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:18 smithi121 bash[18489]: audit 2024-02-12T16:07:18.111060+0000 mon.a (mon.0) 1008 : audit [INF] from='client.24896 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:07:18.648 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:18 smithi121 bash[18489]: audit 2024-02-12T16:07:18.116013+0000 mon.b (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.121:0/2602034616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:07:18.661 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T16:07:18.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:18.759 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:18.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:18.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:18.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:18.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:19.009 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:19.017 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:18 smithi123 bash[19119]: cluster 2024-02-12T16:07:17.588410+0000 mgr.y (mgr.14146) 653 : cluster [DBG] pgmap v772: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 6956/134440 objects misplaced (5.174%); 4.7 MiB/s, 74 objects/s recovering 2024-02-12T16:07:19.017 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:18 smithi123 bash[19119]: audit 2024-02-12T16:07:18.111060+0000 mon.a (mon.0) 1008 : audit [INF] from='client.24896 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:07:19.017 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:18 smithi123 bash[19119]: audit 2024-02-12T16:07:18.116013+0000 mon.b (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.121:0/2602034616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:07:19.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:18 smithi169 bash[19604]: cluster 2024-02-12T16:07:17.588410+0000 mgr.y (mgr.14146) 653 : cluster [DBG] pgmap v772: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 6956/134440 objects misplaced (5.174%); 4.7 MiB/s, 74 objects/s recovering 2024-02-12T16:07:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:18 smithi169 bash[19604]: audit 2024-02-12T16:07:18.111060+0000 mon.a (mon.0) 1008 : audit [INF] from='client.24896 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:07:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:18 smithi169 bash[19604]: audit 2024-02-12T16:07:18.116013+0000 mon.b (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.121:0/2602034616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:07:19.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:19.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:19.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:19.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:19.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:19.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:19.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:19.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:19.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:19.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:19.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:19 smithi121 bash[18489]: audit 2024-02-12T16:07:18.649533+0000 mon.a (mon.0) 1009 : audit [INF] from='client.24896 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:07:19.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:19 smithi121 bash[18489]: cluster 2024-02-12T16:07:18.649756+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e213: 12 total, 12 up, 12 in 2024-02-12T16:07:19.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:19 smithi121 bash[18489]: cluster 2024-02-12T16:07:18.730448+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:19.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:19 smithi121 bash[18489]: cluster 2024-02-12T16:07:18.730507+0000 osd.2 (osd.2) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:19.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:19.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:19.974 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:19 smithi169 bash[19604]: audit 2024-02-12T16:07:18.649533+0000 mon.a (mon.0) 1009 : audit [INF] from='client.24896 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:07:19.974 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:19 smithi169 bash[19604]: cluster 2024-02-12T16:07:18.649756+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e213: 12 total, 12 up, 12 in 2024-02-12T16:07:19.974 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:19 smithi169 bash[19604]: cluster 2024-02-12T16:07:18.730448+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:19.974 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:19 smithi169 bash[19604]: cluster 2024-02-12T16:07:18.730507+0000 osd.2 (osd.2) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:19.992 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:19.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:20.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:20.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:19 smithi123 bash[19119]: audit 2024-02-12T16:07:18.649533+0000 mon.a (mon.0) 1009 : audit [INF] from='client.24896 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:07:20.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:19 smithi123 bash[19119]: cluster 2024-02-12T16:07:18.649756+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e213: 12 total, 12 up, 12 in 2024-02-12T16:07:20.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:19 smithi123 bash[19119]: cluster 2024-02-12T16:07:18.730448+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:20.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:19 smithi123 bash[19119]: cluster 2024-02-12T16:07:18.730507+0000 osd.2 (osd.2) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:20.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:20.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:20.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:20.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:20.285 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:07:20.322 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:07:20.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T16:07:20.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:20.362 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:07:20.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:20.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:20.393 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:07:20.430 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:07:20.459 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:07:20.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:20.490 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:07:20.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:20.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:20.514 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:07:20.515 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:07:20.553 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:07:20.584 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:07:20.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:20.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:07:20.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:20.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:20.638 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:07:20.649 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.649 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.649 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.649 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.650 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:07:20.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:20.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:20.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:20.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:20 smithi169 bash[19604]: cluster 2024-02-12T16:07:19.588876+0000 mgr.y (mgr.14146) 654 : cluster [DBG] pgmap v774: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 6612/129778 objects misplaced (5.095%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T16:07:20.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:20 smithi169 bash[19604]: cluster 2024-02-12T16:07:19.654037+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e214: 12 total, 12 up, 12 in 2024-02-12T16:07:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:20 smithi169 bash[19604]: audit 2024-02-12T16:07:20.242687+0000 mon.a (mon.0) 1012 : audit [DBG] from='client.? v1:172.21.15.121:0/1690503587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:20.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:20.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:20.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:20 smithi121 bash[18489]: cluster 2024-02-12T16:07:19.588876+0000 mgr.y (mgr.14146) 654 : cluster [DBG] pgmap v774: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 6612/129778 objects misplaced (5.095%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T16:07:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:20 smithi121 bash[18489]: cluster 2024-02-12T16:07:19.654037+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e214: 12 total, 12 up, 12 in 2024-02-12T16:07:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:20 smithi121 bash[18489]: audit 2024-02-12T16:07:20.242687+0000 mon.a (mon.0) 1012 : audit [DBG] from='client.? v1:172.21.15.121:0/1690503587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:20.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:21.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:21.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:21.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:20 smithi123 bash[19119]: cluster 2024-02-12T16:07:19.588876+0000 mgr.y (mgr.14146) 654 : cluster [DBG] pgmap v774: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 6612/129778 objects misplaced (5.095%); 4.9 MiB/s, 79 objects/s recovering 2024-02-12T16:07:21.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:20 smithi123 bash[19119]: cluster 2024-02-12T16:07:19.654037+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e214: 12 total, 12 up, 12 in 2024-02-12T16:07:21.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:20 smithi123 bash[19119]: audit 2024-02-12T16:07:20.242687+0000 mon.a (mon.0) 1012 : audit [DBG] from='client.? v1:172.21.15.121:0/1690503587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:21.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:21.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:21.302 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:21.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:21.429 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:21.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:21.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:21.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:21.685 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:21.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:21.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:21.959 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:21.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:22.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:22.102 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:22.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:22.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:22.229 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:22.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:22.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:22.360 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:22.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:22.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:22.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:22.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:22.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:22.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:22.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:22.668 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:07:22.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:22.756 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:22 smithi121 bash[18489]: cluster 2024-02-12T16:07:21.589344+0000 mgr.y (mgr.14146) 655 : cluster [DBG] pgmap v776: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s; 6252/123974 objects misplaced (5.043%); 5.8 MiB/s, 92 objects/s recovering 2024-02-12T16:07:22.757 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:22 smithi121 bash[18489]: audit 2024-02-12T16:07:22.191530+0000 mon.a (mon.0) 1013 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:07:22.757 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:22 smithi121 bash[18489]: audit 2024-02-12T16:07:22.194327+0000 mon.c (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.121:0/1169377345' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:07:22.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:22.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:22.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:22.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:22 smithi123 bash[19119]: cluster 2024-02-12T16:07:21.589344+0000 mgr.y (mgr.14146) 655 : cluster [DBG] pgmap v776: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s; 6252/123974 objects misplaced (5.043%); 5.8 MiB/s, 92 objects/s recovering 2024-02-12T16:07:22.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:22 smithi123 bash[19119]: audit 2024-02-12T16:07:22.191530+0000 mon.a (mon.0) 1013 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:07:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:22 smithi123 bash[19119]: audit 2024-02-12T16:07:22.194327+0000 mon.c (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.121:0/1169377345' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:07:22.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:22.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:23.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:22 smithi169 bash[19604]: cluster 2024-02-12T16:07:21.589344+0000 mgr.y (mgr.14146) 655 : cluster [DBG] pgmap v776: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s; 6252/123974 objects misplaced (5.043%); 5.8 MiB/s, 92 objects/s recovering 2024-02-12T16:07:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:22 smithi169 bash[19604]: audit 2024-02-12T16:07:22.191530+0000 mon.a (mon.0) 1013 : audit [INF] from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:07:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:22 smithi169 bash[19604]: audit 2024-02-12T16:07:22.194327+0000 mon.c (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.121:0/1169377345' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:07:23.049 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:23.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:23.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:23.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:23.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:23.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:07:23.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:23.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:23.722 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:23.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:23.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:23 smithi121 bash[18489]: cluster 2024-02-12T16:07:22.653456+0000 mon.a (mon.0) 1014 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:23.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:23 smithi121 bash[18489]: audit 2024-02-12T16:07:22.657667+0000 mon.a (mon.0) 1015 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:07:23.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:23 smithi121 bash[18489]: cluster 2024-02-12T16:07:22.657727+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e215: 12 total, 12 up, 12 in 2024-02-12T16:07:23.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:23.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:23.966 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:23 smithi123 bash[19119]: cluster 2024-02-12T16:07:22.653456+0000 mon.a (mon.0) 1014 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:23.966 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:23 smithi123 bash[19119]: audit 2024-02-12T16:07:22.657667+0000 mon.a (mon.0) 1015 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:07:23.966 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:23 smithi123 bash[19119]: cluster 2024-02-12T16:07:22.657727+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e215: 12 total, 12 up, 12 in 2024-02-12T16:07:23.982 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:23.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:24.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:23 smithi169 bash[19604]: cluster 2024-02-12T16:07:22.653456+0000 mon.a (mon.0) 1014 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:23 smithi169 bash[19604]: audit 2024-02-12T16:07:22.657667+0000 mon.a (mon.0) 1015 : audit [INF] from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:07:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:23 smithi169 bash[19604]: cluster 2024-02-12T16:07:22.657727+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e215: 12 total, 12 up, 12 in 2024-02-12T16:07:24.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:24.113 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:24.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:24.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:24.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:24.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:24.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:24.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:24.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:24.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:24.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:24.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:24.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:24.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:24.810 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:24.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:24.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:24.868 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:07:24.869 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:07:24.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:24 smithi121 bash[18489]: cluster 2024-02-12T16:07:23.589780+0000 mgr.y (mgr.14146) 656 : cluster [DBG] pgmap v778: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s; 6252/122440 objects misplaced (5.106%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T16:07:24.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:24.933 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:24 smithi123 bash[19119]: cluster 2024-02-12T16:07:23.589780+0000 mgr.y (mgr.14146) 656 : cluster [DBG] pgmap v778: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s; 6252/122440 objects misplaced (5.106%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T16:07:24.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:24.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:24 smithi169 bash[19604]: cluster 2024-02-12T16:07:23.589780+0000 mgr.y (mgr.14146) 656 : cluster [DBG] pgmap v778: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s; 6252/122440 objects misplaced (5.106%); 3.7 MiB/s, 58 objects/s recovering 2024-02-12T16:07:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:24.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:25.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:25.074 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:25.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:25.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:25.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:25.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:25.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:25.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:25.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:25.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:25.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:25.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:25.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:25.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:25.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:25.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:07:25.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:25.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:07:25.748 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:25.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:25.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:25.883 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:25.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:25.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:26.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:26.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:26.172 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:26.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:26.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:26.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:26.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:26.439 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:26.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:26.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:26 smithi169 bash[19604]: cluster 2024-02-12T16:07:25.590242+0000 mgr.y (mgr.14146) 657 : cluster [DBG] pgmap v779: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 5886/116692 objects misplaced (5.044%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:07:26.562 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:26.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:26 smithi123 bash[19119]: cluster 2024-02-12T16:07:25.590242+0000 mgr.y (mgr.14146) 657 : cluster [DBG] pgmap v779: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 5886/116692 objects misplaced (5.044%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:07:26.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:26 smithi121 bash[18489]: cluster 2024-02-12T16:07:25.590242+0000 mgr.y (mgr.14146) 657 : cluster [DBG] pgmap v779: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 5886/116692 objects misplaced (5.044%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:07:26.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:26.704 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:26.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:26.844 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:26.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:26.979 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:26.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:27.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:27.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:27.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:27.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:27.245 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:27.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:27.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:27.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:27.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:27.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:27.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:27.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:27.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:27.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:27.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:27.790 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:27.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:27.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:27.931 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:27.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:27.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:28.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:28.068 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:28.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:28.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:28.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:28.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:28.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:28.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:28.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:28.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:28.466 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:28.467 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:28.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:28.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:28.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:28.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:28.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:28.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:28.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:28 smithi169 bash[19604]: cluster 2024-02-12T16:07:27.590738+0000 mgr.y (mgr.14146) 658 : cluster [DBG] pgmap v780: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 5556/110524 objects misplaced (5.027%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:07:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:28 smithi169 bash[19604]: cluster 2024-02-12T16:07:27.800254+0000 osd.0 (osd.0) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:28 smithi169 bash[19604]: cluster 2024-02-12T16:07:27.800317+0000 osd.0 (osd.0) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:28.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:28.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:28.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:28 smithi121 bash[18489]: cluster 2024-02-12T16:07:27.590738+0000 mgr.y (mgr.14146) 658 : cluster [DBG] pgmap v780: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 5556/110524 objects misplaced (5.027%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:07:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:28 smithi121 bash[18489]: cluster 2024-02-12T16:07:27.800254+0000 osd.0 (osd.0) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:28 smithi121 bash[18489]: cluster 2024-02-12T16:07:27.800317+0000 osd.0 (osd.0) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:28.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:28.984 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:28 smithi123 bash[19119]: cluster 2024-02-12T16:07:27.590738+0000 mgr.y (mgr.14146) 658 : cluster [DBG] pgmap v780: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 5556/110524 objects misplaced (5.027%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:07:28.984 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:28 smithi123 bash[19119]: cluster 2024-02-12T16:07:27.800254+0000 osd.0 (osd.0) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:28.984 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:28 smithi123 bash[19119]: cluster 2024-02-12T16:07:27.800317+0000 osd.0 (osd.0) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:29.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:29.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:29.132 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:29.240 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:29.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:29.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:29.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:29.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:29.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:29.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:29.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:29.808 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:29.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:29.920 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:29 smithi121 bash[18489]: cluster 2024-02-12T16:07:28.647322+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e216: 12 total, 12 up, 12 in 2024-02-12T16:07:29.920 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:29 smithi121 bash[18489]: audit 2024-02-12T16:07:29.591851+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:07:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:29.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:30.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:29 smithi169 bash[19604]: cluster 2024-02-12T16:07:28.647322+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e216: 12 total, 12 up, 12 in 2024-02-12T16:07:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:29 smithi169 bash[19604]: audit 2024-02-12T16:07:29.591851+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:07:30.071 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:30.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:30.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:29 smithi123 bash[19119]: cluster 2024-02-12T16:07:28.647322+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e216: 12 total, 12 up, 12 in 2024-02-12T16:07:30.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:29 smithi123 bash[19119]: audit 2024-02-12T16:07:29.591851+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:07:30.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:30.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:30.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:30.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:30.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:30.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:30.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:30.468 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:30.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:30.479 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:07:30.545 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:30.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:30.604 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:30.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:30.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:30.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:30.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:30.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:30.874 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:30.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:30.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:30 smithi123 bash[19119]: cluster 2024-02-12T16:07:29.591199+0000 mgr.y (mgr.14146) 659 : cluster [DBG] pgmap v782: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 5178/105866 objects misplaced (4.891%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T16:07:30.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:30 smithi123 bash[19119]: audit 2024-02-12T16:07:29.659623+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:07:30.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:30 smithi123 bash[19119]: cluster 2024-02-12T16:07:29.659680+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e217: 12 total, 12 up, 12 in 2024-02-12T16:07:30.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:30 smithi123 bash[19119]: cluster 2024-02-12T16:07:30.267120+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e218: 12 total, 12 up, 12 in 2024-02-12T16:07:30.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:30 smithi121 bash[18489]: cluster 2024-02-12T16:07:29.591199+0000 mgr.y (mgr.14146) 659 : cluster [DBG] pgmap v782: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 5178/105866 objects misplaced (4.891%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T16:07:30.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:30 smithi121 bash[18489]: audit 2024-02-12T16:07:29.659623+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:07:30.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:30 smithi121 bash[18489]: cluster 2024-02-12T16:07:29.659680+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e217: 12 total, 12 up, 12 in 2024-02-12T16:07:30.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:30 smithi121 bash[18489]: cluster 2024-02-12T16:07:30.267120+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e218: 12 total, 12 up, 12 in 2024-02-12T16:07:30.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:31.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:30 smithi169 bash[19604]: cluster 2024-02-12T16:07:29.591199+0000 mgr.y (mgr.14146) 659 : cluster [DBG] pgmap v782: 27 pgs: 2 active+recovering+remapped, 25 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 5178/105866 objects misplaced (4.891%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T16:07:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:30 smithi169 bash[19604]: audit 2024-02-12T16:07:29.659623+0000 mon.a (mon.0) 1019 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:07:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:30 smithi169 bash[19604]: cluster 2024-02-12T16:07:29.659680+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e217: 12 total, 12 up, 12 in 2024-02-12T16:07:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:30 smithi169 bash[19604]: cluster 2024-02-12T16:07:30.267120+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e218: 12 total, 12 up, 12 in 2024-02-12T16:07:31.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:31.150 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:31.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:31.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:31.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:31.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:31.555 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:31.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:31.692 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:31.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:31.828 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:31.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:31.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:31.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:31.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:32.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:32.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:32.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:32.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:32.251 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:32.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:32.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:32.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:32.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:32.438 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:32 smithi123 bash[19119]: cluster 2024-02-12T16:07:31.275897+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e219: 12 total, 12 up, 12 in 2024-02-12T16:07:32.438 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:32 smithi123 bash[19119]: cluster 2024-02-12T16:07:31.591899+0000 mgr.y (mgr.14146) 660 : cluster [DBG] pgmap v786: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 5400/100610 objects degraded (5.367%); 10222/100610 objects misplaced (10.160%); 5.7 MiB/s, 91 objects/s recovering 2024-02-12T16:07:32.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:32.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:32.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:32.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:32 smithi169 bash[19604]: cluster 2024-02-12T16:07:31.275897+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e219: 12 total, 12 up, 12 in 2024-02-12T16:07:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:32 smithi169 bash[19604]: cluster 2024-02-12T16:07:31.591899+0000 mgr.y (mgr.14146) 660 : cluster [DBG] pgmap v786: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 5400/100610 objects degraded (5.367%); 10222/100610 objects misplaced (10.160%); 5.7 MiB/s, 91 objects/s recovering 2024-02-12T16:07:32.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:32.652 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:32.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:32.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:32 smithi121 bash[18489]: cluster 2024-02-12T16:07:31.275897+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e219: 12 total, 12 up, 12 in 2024-02-12T16:07:32.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:32 smithi121 bash[18489]: cluster 2024-02-12T16:07:31.591899+0000 mgr.y (mgr.14146) 660 : cluster [DBG] pgmap v786: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 5400/100610 objects degraded (5.367%); 10222/100610 objects misplaced (10.160%); 5.7 MiB/s, 91 objects/s recovering 2024-02-12T16:07:32.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:32.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:32.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:32.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:32.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:32.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:33.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:33.045 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:33.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:33.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:33.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:33.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:33.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:33.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:33.554 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T16:07:33.556 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:33.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:33.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:33.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:33 smithi169 bash[19604]: audit 2024-02-12T16:07:33.482774+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? v1:172.21.15.121:0/891648186' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:07:33.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:33.862 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:33.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:33 smithi123 bash[19119]: audit 2024-02-12T16:07:33.482774+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? v1:172.21.15.121:0/891648186' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:07:33.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:33 smithi121 bash[18489]: audit 2024-02-12T16:07:33.482774+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? v1:172.21.15.121:0/891648186' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:07:33.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:33.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:33.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:34.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:34.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:34.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:34.251 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:34.279 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:34.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:34.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:34.415 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:34.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:34.517 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:07:34.517 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:07:34.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:34.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:34.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:34.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:34.689 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:34.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:34.793 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:34 smithi169 bash[19604]: audit 2024-02-12T16:07:33.539082+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? v1:172.21.15.121:0/891648186' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:07:34.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:34 smithi169 bash[19604]: cluster 2024-02-12T16:07:33.539308+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e220: 12 total, 12 up, 12 in 2024-02-12T16:07:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:34 smithi169 bash[19604]: cluster 2024-02-12T16:07:33.592338+0000 mgr.y (mgr.14146) 661 : cluster [DBG] pgmap v788: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s; 5400/99878 objects degraded (5.407%); 10222/99878 objects misplaced (10.234%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T16:07:34.817 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:34.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:34 smithi123 bash[19119]: audit 2024-02-12T16:07:33.539082+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? v1:172.21.15.121:0/891648186' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:07:34.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:34 smithi123 bash[19119]: cluster 2024-02-12T16:07:33.539308+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e220: 12 total, 12 up, 12 in 2024-02-12T16:07:34.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:34 smithi123 bash[19119]: cluster 2024-02-12T16:07:33.592338+0000 mgr.y (mgr.14146) 661 : cluster [DBG] pgmap v788: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s; 5400/99878 objects degraded (5.407%); 10222/99878 objects misplaced (10.234%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T16:07:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:34.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:34.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:34 smithi121 bash[18489]: audit 2024-02-12T16:07:33.539082+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? v1:172.21.15.121:0/891648186' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:07:34.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:34 smithi121 bash[18489]: cluster 2024-02-12T16:07:33.539308+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e220: 12 total, 12 up, 12 in 2024-02-12T16:07:34.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:34 smithi121 bash[18489]: cluster 2024-02-12T16:07:33.592338+0000 mgr.y (mgr.14146) 661 : cluster [DBG] pgmap v788: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s; 5400/99878 objects degraded (5.407%); 10222/99878 objects misplaced (10.234%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T16:07:34.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:34.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:35.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:35.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:35.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:35.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:35.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:35.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:35.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:07:35.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:35.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:35.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:35.440 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:35.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 8 1 2024-02-12T16:07:35.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:35.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:35.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:35.606 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:35.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:35.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:35.743 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:35.747 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:07:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:35.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:35.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:07:35.814 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:07:35.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:35.867 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:07:35.898 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:07:35.906 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:35.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:35.935 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:07:35.964 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:07:35.996 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:07:36.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:36.027 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:07:36.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:36.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:36.062 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:07:36.096 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:07:36.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:07:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:07:36.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:36.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:36.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:36.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:36.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:36 smithi121 bash[18489]: cluster 2024-02-12T16:07:35.592839+0000 mgr.y (mgr.14146) 662 : cluster [DBG] pgmap v789: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 2.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 5400/96488 objects degraded (5.597%); 9994/96488 objects misplaced (10.358%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:07:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:36 smithi121 bash[18489]: audit 2024-02-12T16:07:35.698262+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? v1:172.21.15.121:0/2596707373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:36.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:36.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:36.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:36 smithi169 bash[19604]: cluster 2024-02-12T16:07:35.592839+0000 mgr.y (mgr.14146) 662 : cluster [DBG] pgmap v789: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 2.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 5400/96488 objects degraded (5.597%); 9994/96488 objects misplaced (10.358%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:07:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:36 smithi169 bash[19604]: audit 2024-02-12T16:07:35.698262+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? v1:172.21.15.121:0/2596707373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:36.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:36.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:36.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:36.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:36 smithi123 bash[19119]: cluster 2024-02-12T16:07:35.592839+0000 mgr.y (mgr.14146) 662 : cluster [DBG] pgmap v789: 27 pgs: 1 activating+remapped, 2 active+recovering+remapped, 24 active+clean; 2.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 5400/96488 objects degraded (5.597%); 9994/96488 objects misplaced (10.358%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:07:36.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:36 smithi123 bash[19119]: audit 2024-02-12T16:07:35.698262+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? v1:172.21.15.121:0/2596707373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:36.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:36.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:36.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:36.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:36.892 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:36.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:36.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:37.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:37.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:37.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:37.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:37.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:37.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:37.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:37.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:37.573 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:37.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:37.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:37.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:37.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:37.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:37.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:37.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:38.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:38.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:38.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:38.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:38.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:38.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:38.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:38.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:38.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:38.364 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:38.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:38.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:38.495 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:38.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:38.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:38.626 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:38.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:07:38.729 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:38.736 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:38 smithi123 bash[19119]: cluster 2024-02-12T16:07:37.593315+0000 mgr.y (mgr.14146) 663 : cluster [DBG] pgmap v790: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.8 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 7135/91992 objects misplaced (7.756%); 4.9 MiB/s, 78 objects/s recovering 2024-02-12T16:07:38.736 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:38 smithi123 bash[19119]: audit 2024-02-12T16:07:38.087199+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? v1:172.21.15.121:0/2046048973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:07:38.751 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:38.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:38.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:38.864 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:38 smithi169 bash[19604]: cluster 2024-02-12T16:07:37.593315+0000 mgr.y (mgr.14146) 663 : cluster [DBG] pgmap v790: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.8 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 7135/91992 objects misplaced (7.756%); 4.9 MiB/s, 78 objects/s recovering 2024-02-12T16:07:38.865 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:38 smithi169 bash[19604]: audit 2024-02-12T16:07:38.087199+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? v1:172.21.15.121:0/2046048973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:07:38.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:38.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:38.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:38 smithi121 bash[18489]: cluster 2024-02-12T16:07:37.593315+0000 mgr.y (mgr.14146) 663 : cluster [DBG] pgmap v790: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.8 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s; 7135/91992 objects misplaced (7.756%); 4.9 MiB/s, 78 objects/s recovering 2024-02-12T16:07:38.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:38 smithi121 bash[18489]: audit 2024-02-12T16:07:38.087199+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? v1:172.21.15.121:0/2046048973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:07:38.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:39.042 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:39.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:39.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:39.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:39.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:39.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:39.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:39.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:39.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:07:39.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:39.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:39.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:39.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:39.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:39.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:39.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:39.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:39.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:39.847 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:39 smithi121 bash[18489]: cluster 2024-02-12T16:07:38.645820+0000 mon.a (mon.0) 1027 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:39.847 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:39 smithi121 bash[18489]: audit 2024-02-12T16:07:38.655715+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? v1:172.21.15.121:0/2046048973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:07:39.847 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:39 smithi121 bash[18489]: cluster 2024-02-12T16:07:38.655897+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e221: 12 total, 12 up, 12 in 2024-02-12T16:07:39.885 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:39.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:39.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:40.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:39 smithi169 bash[19604]: cluster 2024-02-12T16:07:38.645820+0000 mon.a (mon.0) 1027 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:40.001 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:39 smithi169 bash[19604]: audit 2024-02-12T16:07:38.655715+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? v1:172.21.15.121:0/2046048973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:07:40.001 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:39 smithi169 bash[19604]: cluster 2024-02-12T16:07:38.655897+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e221: 12 total, 12 up, 12 in 2024-02-12T16:07:40.023 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:40.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:40.073 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:07:40.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:40.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:39 smithi123 bash[19119]: cluster 2024-02-12T16:07:38.645820+0000 mon.a (mon.0) 1027 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:40.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:39 smithi123 bash[19119]: audit 2024-02-12T16:07:38.655715+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? v1:172.21.15.121:0/2046048973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:07:40.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:39 smithi123 bash[19119]: cluster 2024-02-12T16:07:38.655897+0000 mon.a (mon.0) 1029 : cluster [DBG] osdmap e221: 12 total, 12 up, 12 in 2024-02-12T16:07:40.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:40.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:40.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:40.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:40.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:40.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:40.436 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:40.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:40.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:40.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:40.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:40.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:40 smithi121 bash[18489]: cluster 2024-02-12T16:07:39.602212+0000 mgr.y (mgr.14146) 664 : cluster [DBG] pgmap v792: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 6825/88200 objects misplaced (7.738%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:07:40.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:40.688 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:40 smithi123 bash[19119]: cluster 2024-02-12T16:07:39.602212+0000 mgr.y (mgr.14146) 664 : cluster [DBG] pgmap v792: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 6825/88200 objects misplaced (7.738%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:07:40.705 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:40.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:40.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:40.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:40.819 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:40 smithi169 bash[19604]: cluster 2024-02-12T16:07:39.602212+0000 mgr.y (mgr.14146) 664 : cluster [DBG] pgmap v792: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 6825/88200 objects misplaced (7.738%); 4.3 MiB/s, 68 objects/s recovering 2024-02-12T16:07:40.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:40.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:40.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:41.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:41.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:41.146 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:41.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:41.278 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:41.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:41.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:41.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:41.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:41.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:41.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:41.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:41 smithi169 bash[19604]: cluster 2024-02-12T16:07:40.768728+0000 osd.10 (osd.10) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:41.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:41 smithi169 bash[19604]: cluster 2024-02-12T16:07:40.768788+0000 osd.10 (osd.10) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:41.806 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:41.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:41.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:41.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:41.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:41 smithi123 bash[19119]: cluster 2024-02-12T16:07:40.768728+0000 osd.10 (osd.10) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:41.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:41 smithi123 bash[19119]: cluster 2024-02-12T16:07:40.768788+0000 osd.10 (osd.10) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:42.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:42.066 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:42.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:07:42.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:41 smithi121 bash[18489]: cluster 2024-02-12T16:07:40.768728+0000 osd.10 (osd.10) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:42.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:41 smithi121 bash[18489]: cluster 2024-02-12T16:07:40.768788+0000 osd.10 (osd.10) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:42.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:42.199 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:42.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:42.332 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:42.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:42.456 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:42.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:42.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:42.590 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:42.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:42.698 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:42 smithi169 bash[19604]: cluster 2024-02-12T16:07:41.602680+0000 mgr.y (mgr.14146) 665 : cluster [DBG] pgmap v793: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 6236/82780 objects misplaced (7.533%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:07:42.698 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:42 smithi169 bash[19604]: cluster 2024-02-12T16:07:41.670704+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e222: 12 total, 12 up, 12 in 2024-02-12T16:07:42.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:42.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:42.831 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:42 smithi123 bash[19119]: cluster 2024-02-12T16:07:41.602680+0000 mgr.y (mgr.14146) 665 : cluster [DBG] pgmap v793: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 6236/82780 objects misplaced (7.533%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:07:42.831 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:42 smithi123 bash[19119]: cluster 2024-02-12T16:07:41.670704+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e222: 12 total, 12 up, 12 in 2024-02-12T16:07:42.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:42.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:42.978 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:42.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:43.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:43.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:43.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:43.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:43.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:43.281 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:42 smithi121 bash[18489]: cluster 2024-02-12T16:07:41.602680+0000 mgr.y (mgr.14146) 665 : cluster [DBG] pgmap v793: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 6236/82780 objects misplaced (7.533%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:07:43.281 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:42 smithi121 bash[18489]: cluster 2024-02-12T16:07:41.670704+0000 mon.a (mon.0) 1030 : cluster [DBG] osdmap e222: 12 total, 12 up, 12 in 2024-02-12T16:07:43.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:43.405 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:43.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:43.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:43.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:43.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:43.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:43.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:43.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:43.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:43.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:43.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:43.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:43.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:43.993 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:43.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:44.099 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:44.133 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:44.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:309209 2.7 2:e7dbc83b:::benchmark_data_smithi169_42458_object136175:head [delete] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:44.205664+0000", 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0017563660000000001, 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017966620000000001, 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:07:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 309209 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.205664+0000", 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.205664+0000", 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.205662+0000", 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2461e-05 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.205674+0000", 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6602000000000001e-05 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.205691+0000", 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6080000000000001e-06 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.205697+0000", 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001046653 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206744+0000", 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011306000000000001 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206857+0000", 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001192706 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.14811.0:309204 2.7 e222/171)", 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:44.206222+0000", 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.001199194, 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001317758, 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "queued for pg", 2024-02-12T16:07:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206222+0000", 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206222+0000", 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206220+0000", 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9760000000000001e-06 2024-02-12T16:07:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206229+0000", 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2639999999999999e-05 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206242+0000", 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7499999999999997e-06 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.206248+0000", 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6783000000000001e-05 2024-02-12T16:07:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:309204 2.7 2:e54f491d:::benchmark_data_smithi169_42458_object136170:head [delete] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:07:44.200903+0000", 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0065181500000000003, 2024-02-12T16:07:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066990050000000001, 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 309204 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.200903+0000", 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:07:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.200903+0000", 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.200900+0000", 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.17e-05 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.200911+0000", 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1729000000000002e-05 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.200933+0000", 2024-02-12T16:07:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7250000000000003e-06 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.200942+0000", 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8486999999999997e-05 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.201010+0000", 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015922500000000001 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.201169+0000", 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00082144100000000001 2024-02-12T16:07:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:07:44.201991+0000", 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001088236 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 3 2024-02-12T16:07:44.220 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:07:44.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:44.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:44.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:44.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:44.410 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:44.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:44.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:44.535 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:44.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:44.624 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:44.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:44.663 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:44.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:44.768 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:44.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:44 smithi123 bash[19119]: cluster 2024-02-12T16:07:43.603121+0000 mgr.y (mgr.14146) 666 : cluster [DBG] pgmap v795: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 6236/81810 objects misplaced (7.623%); 5.7 MiB/s, 91 objects/s recovering 2024-02-12T16:07:44.794 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:44.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:44.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:44.906 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:44 smithi169 bash[19604]: cluster 2024-02-12T16:07:43.603121+0000 mgr.y (mgr.14146) 666 : cluster [DBG] pgmap v795: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 6236/81810 objects misplaced (7.623%); 5.7 MiB/s, 91 objects/s recovering 2024-02-12T16:07:44.931 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:44.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:44.992 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:07:45.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:45.038 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:44 smithi121 bash[18489]: cluster 2024-02-12T16:07:43.603121+0000 mgr.y (mgr.14146) 666 : cluster [DBG] pgmap v795: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 6236/81810 objects misplaced (7.623%); 5.7 MiB/s, 91 objects/s recovering 2024-02-12T16:07:45.058 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:45.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:45.191 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:45.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:45.319 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:45.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:45.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:45.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:45.591 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:45.696 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:45.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:45.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:45.857 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:45.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:45.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:45.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:46.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:46.138 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:46.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:46.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:46.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:46.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:46.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:46.379 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:46 smithi121 bash[18489]: cluster 2024-02-12T16:07:45.603597+0000 mgr.y (mgr.14146) 667 : cluster [DBG] pgmap v796: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 5906/76598 objects misplaced (7.710%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:07:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:46.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:46.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:46 smithi169 bash[19604]: cluster 2024-02-12T16:07:45.603597+0000 mgr.y (mgr.14146) 667 : cluster [DBG] pgmap v796: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 5906/76598 objects misplaced (7.710%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:07:46.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:46.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:46.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:46 smithi123 bash[19119]: cluster 2024-02-12T16:07:45.603597+0000 mgr.y (mgr.14146) 667 : cluster [DBG] pgmap v796: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 5906/76598 objects misplaced (7.710%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:07:46.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:46.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:46.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:46.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:46.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:46.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:46.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:46.963 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:46.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:47.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:47.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:47.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:47.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:47.231 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:47.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:47.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:47.362 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:47.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:47.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:47.476 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:47 smithi169 bash[19604]: cluster 2024-02-12T16:07:46.752802+0000 osd.1 (osd.1) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:47.476 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:47 smithi169 bash[19604]: cluster 2024-02-12T16:07:46.752853+0000 osd.1 (osd.1) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:47.502 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:47.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:47.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:47 smithi121 bash[18489]: cluster 2024-02-12T16:07:46.752802+0000 osd.1 (osd.1) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:47.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:47 smithi121 bash[18489]: cluster 2024-02-12T16:07:46.752853+0000 osd.1 (osd.1) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:47.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:47.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:47.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:47.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:47 smithi123 bash[19119]: cluster 2024-02-12T16:07:46.752802+0000 osd.1 (osd.1) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:47.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:47 smithi123 bash[19119]: cluster 2024-02-12T16:07:46.752853+0000 osd.1 (osd.1) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:47.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:47.758 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:47.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:47.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:47.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:47.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:47.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:48.024 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:48.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:48.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:48.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:48.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:48.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:48 smithi169 bash[19604]: cluster 2024-02-12T16:07:47.282266+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e223: 12 total, 12 up, 12 in 2024-02-12T16:07:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:48 smithi169 bash[19604]: cluster 2024-02-12T16:07:47.604020+0000 mgr.y (mgr.14146) 668 : cluster [DBG] pgmap v798: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5290/71350 objects misplaced (7.414%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:07:48.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:48.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:48.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:48 smithi121 bash[18489]: cluster 2024-02-12T16:07:47.282266+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e223: 12 total, 12 up, 12 in 2024-02-12T16:07:48.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:48 smithi121 bash[18489]: cluster 2024-02-12T16:07:47.604020+0000 mgr.y (mgr.14146) 668 : cluster [DBG] pgmap v798: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5290/71350 objects misplaced (7.414%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:07:48.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:48.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:48.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:48.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:48.588 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:48.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:48.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:48 smithi123 bash[19119]: cluster 2024-02-12T16:07:47.282266+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e223: 12 total, 12 up, 12 in 2024-02-12T16:07:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:48 smithi123 bash[19119]: cluster 2024-02-12T16:07:47.604020+0000 mgr.y (mgr.14146) 668 : cluster [DBG] pgmap v798: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5290/71350 objects misplaced (7.414%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:07:48.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:48.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:48.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:48.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:48.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:48.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:48.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:49.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:49.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:49.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:49.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:49.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:49.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:49.307 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:49.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:49.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:49.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:49.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:49.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:49.603 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:49.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:49.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:49.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:49.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:49.786 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:07:49.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:49.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:49.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:50.015 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:50.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:50.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:50.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:50.265 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:50.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:50.394 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:50.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:50.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:50.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:50.697 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:50.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:50.814 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:50 smithi121 bash[18489]: cluster 2024-02-12T16:07:49.604529+0000 mgr.y (mgr.14146) 669 : cluster [DBG] pgmap v799: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 4934/67528 objects misplaced (7.307%); 5.1 MiB/s, 81 objects/s recovering 2024-02-12T16:07:50.814 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:50 smithi121 bash[18489]: audit 2024-02-12T16:07:50.513338+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.121:0/3772706697' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:07:50.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:50.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:50.959 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:50 smithi169 bash[19604]: cluster 2024-02-12T16:07:49.604529+0000 mgr.y (mgr.14146) 669 : cluster [DBG] pgmap v799: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 4934/67528 objects misplaced (7.307%); 5.1 MiB/s, 81 objects/s recovering 2024-02-12T16:07:50.959 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:50 smithi169 bash[19604]: audit 2024-02-12T16:07:50.513338+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.121:0/3772706697' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:07:50.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:50.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:51.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:51.107 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:51.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:51.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:50 smithi123 bash[19119]: cluster 2024-02-12T16:07:49.604529+0000 mgr.y (mgr.14146) 669 : cluster [DBG] pgmap v799: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 2.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 4934/67528 objects misplaced (7.307%); 5.1 MiB/s, 81 objects/s recovering 2024-02-12T16:07:51.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:50 smithi123 bash[19119]: audit 2024-02-12T16:07:50.513338+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.121:0/3772706697' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:07:51.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:51.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:51.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:51.288 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.8 to 1 (10000) 2024-02-12T16:07:51.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:51.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:51.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:51.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:51.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:51.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:51.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:51.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:51.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:51.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:51.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:51.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:51 smithi123 bash[19119]: cluster 2024-02-12T16:07:50.663079+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:51.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:51 smithi123 bash[19119]: cluster 2024-02-12T16:07:50.663132+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:51.791 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:51 smithi123 bash[19119]: audit 2024-02-12T16:07:51.278299+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? v1:172.21.15.121:0/3772706697' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:07:51.791 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:51 smithi123 bash[19119]: cluster 2024-02-12T16:07:51.278509+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e224: 12 total, 12 up, 12 in 2024-02-12T16:07:51.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:51.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:51.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:51.902 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:07:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:51 smithi121 bash[18489]: cluster 2024-02-12T16:07:50.663079+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:51 smithi121 bash[18489]: cluster 2024-02-12T16:07:50.663132+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:51 smithi121 bash[18489]: audit 2024-02-12T16:07:51.278299+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? v1:172.21.15.121:0/3772706697' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:07:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:51 smithi121 bash[18489]: cluster 2024-02-12T16:07:51.278509+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e224: 12 total, 12 up, 12 in 2024-02-12T16:07:51.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:51.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:51.935 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:07:51.968 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:07:51.997 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:07:52.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:07:52.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:51 smithi169 bash[19604]: cluster 2024-02-12T16:07:50.663079+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:07:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:51 smithi169 bash[19604]: cluster 2024-02-12T16:07:50.663132+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:07:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:51 smithi169 bash[19604]: audit 2024-02-12T16:07:51.278299+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? v1:172.21.15.121:0/3772706697' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:07:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:51 smithi169 bash[19604]: cluster 2024-02-12T16:07:51.278509+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e224: 12 total, 12 up, 12 in 2024-02-12T16:07:52.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:52.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:52.058 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:07:52.089 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:07:52.124 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:07:52.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:52.167 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:07:52.186 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:52.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:52.200 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:07:52.228 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:07:52.258 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:07:52.259 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.260 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:07:52.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:52.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:52.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:52.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:52.459 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:52.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:52.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:52.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:52.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:52.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:52.706 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:52 smithi169 bash[19604]: cluster 2024-02-12T16:07:51.605009+0000 mgr.y (mgr.14146) 670 : cluster [DBG] pgmap v801: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 4314/62248 objects misplaced (6.930%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T16:07:52.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:52 smithi169 bash[19604]: audit 2024-02-12T16:07:51.846013+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.121:0/3944115644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:52.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:52 smithi169 bash[19604]: audit 2024-02-12T16:07:52.369714+0000 mon.a (mon.0) 1036 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:07:52.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:52.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:52.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:52.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:52.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:52.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:52.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:52 smithi123 bash[19119]: cluster 2024-02-12T16:07:51.605009+0000 mgr.y (mgr.14146) 670 : cluster [DBG] pgmap v801: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 4314/62248 objects misplaced (6.930%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T16:07:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:52 smithi123 bash[19119]: audit 2024-02-12T16:07:51.846013+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.121:0/3944115644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:52 smithi123 bash[19119]: audit 2024-02-12T16:07:52.369714+0000 mon.a (mon.0) 1036 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:07:52.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:52.999 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:53.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:53.129 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:53.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:52 smithi121 bash[18489]: cluster 2024-02-12T16:07:51.605009+0000 mgr.y (mgr.14146) 670 : cluster [DBG] pgmap v801: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 4314/62248 objects misplaced (6.930%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T16:07:53.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:52 smithi121 bash[18489]: audit 2024-02-12T16:07:51.846013+0000 mon.a (mon.0) 1035 : audit [DBG] from='client.? v1:172.21.15.121:0/3944115644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:07:53.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:52 smithi121 bash[18489]: audit 2024-02-12T16:07:52.369714+0000 mon.a (mon.0) 1036 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:07:53.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:53.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:53.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 9 1 2024-02-12T16:07:53.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:53.395 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:53.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:53.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:53.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:53.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:53.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:53.673 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:53.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:53.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:53 smithi123 bash[19119]: audit 2024-02-12T16:07:52.742961+0000 mon.a (mon.0) 1037 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:07:53.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:53 smithi123 bash[19119]: audit 2024-02-12T16:07:52.744400+0000 mon.a (mon.0) 1038 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:07:53.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:53 smithi123 bash[19119]: audit 2024-02-12T16:07:52.753646+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:07:53.807 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:53.863 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:07:53.863 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:07:53.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:53 smithi121 bash[18489]: audit 2024-02-12T16:07:52.742961+0000 mon.a (mon.0) 1037 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:07:53.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:53 smithi121 bash[18489]: audit 2024-02-12T16:07:52.744400+0000 mon.a (mon.0) 1038 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:07:53.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:53 smithi121 bash[18489]: audit 2024-02-12T16:07:52.753646+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:07:53.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:53.939 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:53.944 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:53 smithi169 bash[19604]: audit 2024-02-12T16:07:52.742961+0000 mon.a (mon.0) 1037 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:07:53.944 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:53 smithi169 bash[19604]: audit 2024-02-12T16:07:52.744400+0000 mon.a (mon.0) 1038 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:07:53.944 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:53 smithi169 bash[19604]: audit 2024-02-12T16:07:52.753646+0000 mon.a (mon.0) 1039 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:07:53.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:53.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:54.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:54.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:54.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:54.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:54.223 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:54.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:54.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:54.362 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:54.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:54.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:54.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:54.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:54.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:07:54.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:54.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:54.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:54.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:54.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:54 smithi121 bash[18489]: cluster 2024-02-12T16:07:53.605435+0000 mgr.y (mgr.14146) 671 : cluster [DBG] pgmap v802: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 4314/61392 objects misplaced (7.027%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:07:54.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:54.941 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:54 smithi123 bash[19119]: cluster 2024-02-12T16:07:53.605435+0000 mgr.y (mgr.14146) 671 : cluster [DBG] pgmap v802: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 4314/61392 objects misplaced (7.027%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:07:54.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:54.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:54 smithi169 bash[19604]: cluster 2024-02-12T16:07:53.605435+0000 mgr.y (mgr.14146) 671 : cluster [DBG] pgmap v802: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 4314/61392 objects misplaced (7.027%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:07:55.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:55.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:55.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:55.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:55.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:55.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:55.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:55.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:55.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:55.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:55.527 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:55.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:55.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:55.664 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:55.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:55.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:55 smithi169 bash[19604]: audit 2024-02-12T16:07:54.762770+0000 mon.a (mon.0) 1040 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:07:55.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:55 smithi169 bash[19604]: audit 2024-02-12T16:07:54.765326+0000 mon.c (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.121:0/2777203908' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:07:55.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:55.780 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:07:55.793 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:55.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:55.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:07:55.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:55.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:55.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:56.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:56.064 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:55 smithi123 bash[19119]: audit 2024-02-12T16:07:54.762770+0000 mon.a (mon.0) 1040 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:07:56.065 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:55 smithi123 bash[19119]: audit 2024-02-12T16:07:54.765326+0000 mon.c (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.121:0/2777203908' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:07:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:56.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:56.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:55 smithi121 bash[18489]: audit 2024-02-12T16:07:54.762770+0000 mon.a (mon.0) 1040 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:07:56.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:55 smithi121 bash[18489]: audit 2024-02-12T16:07:54.765326+0000 mon.c (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.121:0/2777203908' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:07:56.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:56.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:56.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:56.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:56.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:56.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:07:56.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:56.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:56.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:56.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:56.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:56.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:56.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:56.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:56 smithi169 bash[19604]: cluster 2024-02-12T16:07:55.605842+0000 mgr.y (mgr.14146) 672 : cluster [DBG] pgmap v803: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 4006/56696 objects misplaced (7.066%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:07:56.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:56 smithi169 bash[19604]: cluster 2024-02-12T16:07:55.754055+0000 mon.a (mon.0) 1041 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:56.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:56 smithi169 bash[19604]: audit 2024-02-12T16:07:55.767138+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:07:56.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:56 smithi169 bash[19604]: cluster 2024-02-12T16:07:55.767269+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e225: 12 total, 12 up, 12 in 2024-02-12T16:07:56.766 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:56.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:56.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:56.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:56.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:56 smithi121 bash[18489]: cluster 2024-02-12T16:07:55.605842+0000 mgr.y (mgr.14146) 672 : cluster [DBG] pgmap v803: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 4006/56696 objects misplaced (7.066%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:07:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:56 smithi121 bash[18489]: cluster 2024-02-12T16:07:55.754055+0000 mon.a (mon.0) 1041 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:56 smithi121 bash[18489]: audit 2024-02-12T16:07:55.767138+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:07:56.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:56 smithi121 bash[18489]: cluster 2024-02-12T16:07:55.767269+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e225: 12 total, 12 up, 12 in 2024-02-12T16:07:57.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.060 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:57.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:57.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:56 smithi123 bash[19119]: cluster 2024-02-12T16:07:55.605842+0000 mgr.y (mgr.14146) 672 : cluster [DBG] pgmap v803: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 4006/56696 objects misplaced (7.066%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:07:57.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:56 smithi123 bash[19119]: cluster 2024-02-12T16:07:55.754055+0000 mon.a (mon.0) 1041 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:07:57.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:56 smithi123 bash[19119]: audit 2024-02-12T16:07:55.767138+0000 mon.a (mon.0) 1042 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:07:57.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:56 smithi123 bash[19119]: cluster 2024-02-12T16:07:55.767269+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e225: 12 total, 12 up, 12 in 2024-02-12T16:07:57.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:57.188 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:07:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:57.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:07:57.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:07:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:57.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:07:57.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:57.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:57.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:57.586 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:07:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:57.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:07:57.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:57.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:57.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.874 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:57.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:57.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:58.008 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:58.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:58.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:07:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:07:58.218 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:07:58.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:58.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:58.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:58.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:58.566 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:07:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:07:58.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:58.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:58.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:07:58 smithi169 bash[19604]: cluster 2024-02-12T16:07:57.606305+0000 mgr.y (mgr.14146) 673 : cluster [DBG] pgmap v805: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 3347/51754 objects misplaced (6.467%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:07:58.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:58.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:07:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:07:58.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:07:58 smithi123 bash[19119]: cluster 2024-02-12T16:07:57.606305+0000 mgr.y (mgr.14146) 673 : cluster [DBG] pgmap v805: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 3347/51754 objects misplaced (6.467%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:07:58.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:58.969 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:58.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:59.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:59.098 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:59.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:59.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:07:59.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:07:58 smithi121 bash[18489]: cluster 2024-02-12T16:07:57.606305+0000 mgr.y (mgr.14146) 673 : cluster [DBG] pgmap v805: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 3347/51754 objects misplaced (6.467%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:07:59.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:59.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:07:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:59.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:07:59.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:59.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:07:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:59.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:07:59.460 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:07:59.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:59.504 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:07:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:59.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:07:59.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:59.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:07:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:59.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:07:59.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:59.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:07:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:59.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:07:59.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:59.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:07:59.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:07:59.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:00.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:00.049 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:00.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:00.155 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:00.183 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:00.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:00.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:00.248 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:00.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:00.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:00.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:00.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:00.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:00.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:00.544 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:00.567 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:00.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:00.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:00.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:00.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:00.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:00 smithi169 bash[19604]: cluster 2024-02-12T16:07:59.606762+0000 mgr.y (mgr.14146) 674 : cluster [DBG] pgmap v806: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 3015/47782 objects misplaced (6.310%); 6.1 MiB/s, 96 objects/s recovering 2024-02-12T16:08:00.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:00.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:00.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:00.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:00.998 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:01.031 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:00 smithi123 bash[19119]: cluster 2024-02-12T16:07:59.606762+0000 mgr.y (mgr.14146) 674 : cluster [DBG] pgmap v806: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 3015/47782 objects misplaced (6.310%); 6.1 MiB/s, 96 objects/s recovering 2024-02-12T16:08:01.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:01.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:00 smithi121 bash[18489]: cluster 2024-02-12T16:07:59.606762+0000 mgr.y (mgr.14146) 674 : cluster [DBG] pgmap v806: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 3015/47782 objects misplaced (6.310%); 6.1 MiB/s, 96 objects/s recovering 2024-02-12T16:08:01.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:01.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:01.279 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:01.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:01.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:01.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:01.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:01.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:01.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:01.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:01.863 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:01.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:01.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:01.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:02.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:02.126 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:02.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:02.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:02.255 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:02.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:02.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:02.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:02.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:02.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:02.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:02.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:02.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:02.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:02.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:02.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:02.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:02 smithi121 bash[18489]: cluster 2024-02-12T16:08:01.607224+0000 mgr.y (mgr.14146) 675 : cluster [DBG] pgmap v807: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s; 2360/42248 objects misplaced (5.586%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:08:02.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:02.939 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:02.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:03.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:03.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:02 smithi169 bash[19604]: cluster 2024-02-12T16:08:01.607224+0000 mgr.y (mgr.14146) 675 : cluster [DBG] pgmap v807: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s; 2360/42248 objects misplaced (5.586%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:08:03.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:03.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:03.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:02 smithi123 bash[19119]: cluster 2024-02-12T16:08:01.607224+0000 mgr.y (mgr.14146) 675 : cluster [DBG] pgmap v807: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s; 2360/42248 objects misplaced (5.586%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:08:03.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:03.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:03.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:03.298 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:03.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:03.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:03.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:03.369 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:08:03.369 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:08:03.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:03.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:03.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:03.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:03.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:03.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:03.742 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:03.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:03.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:03.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:03.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:03.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:04.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:04.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:08:04.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:04.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:04.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:04.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:04.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:04.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:04.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:04.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:04.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:04.816 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:04.823 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:04 smithi123 bash[19119]: cluster 2024-02-12T16:08:03.607639+0000 mgr.y (mgr.14146) 676 : cluster [DBG] pgmap v808: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s; 2360/41306 objects misplaced (5.713%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:08:04.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:04.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:04.988 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:04.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:05.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:04 smithi169 bash[19604]: cluster 2024-02-12T16:08:03.607639+0000 mgr.y (mgr.14146) 676 : cluster [DBG] pgmap v808: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s; 2360/41306 objects misplaced (5.713%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:08:05.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:05.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:05.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:05.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:04 smithi121 bash[18489]: cluster 2024-02-12T16:08:03.607639+0000 mgr.y (mgr.14146) 676 : cluster [DBG] pgmap v808: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s; 2360/41306 objects misplaced (5.713%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:08:05.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:05.255 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:05.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:05.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:05.399 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:05.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:05.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:05.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:05.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:05.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:05.656 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:05.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:05.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:05.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:05.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:05.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:05.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:05.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:06.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:06.060 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:06.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:06.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:06.198 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:06.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:06.293 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.9 to 1 (10000) 2024-02-12T16:08:06.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:06.336 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:06.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:06.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:06.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:06.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:06.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:06 smithi169 bash[19604]: cluster 2024-02-12T16:08:05.608126+0000 mgr.y (mgr.14146) 677 : cluster [DBG] pgmap v809: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 2044/36144 objects misplaced (5.655%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T16:08:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:06 smithi169 bash[19604]: audit 2024-02-12T16:08:06.169751+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? v1:172.21.15.121:0/1485285138' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:08:06.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:06.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:06 smithi123 bash[19119]: cluster 2024-02-12T16:08:05.608126+0000 mgr.y (mgr.14146) 677 : cluster [DBG] pgmap v809: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 2044/36144 objects misplaced (5.655%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T16:08:06.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:06 smithi123 bash[19119]: audit 2024-02-12T16:08:06.169751+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? v1:172.21.15.121:0/1485285138' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:08:06.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:06.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:06 smithi121 bash[18489]: cluster 2024-02-12T16:08:05.608126+0000 mgr.y (mgr.14146) 677 : cluster [DBG] pgmap v809: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 2044/36144 objects misplaced (5.655%); 6.1 MiB/s, 98 objects/s recovering 2024-02-12T16:08:06.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:06 smithi121 bash[18489]: audit 2024-02-12T16:08:06.169751+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? v1:172.21.15.121:0/1485285138' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:08:06.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:06.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:06.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:06.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:06.891 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:06.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:06.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:06.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:07.049 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:07.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:07.155 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:07.178 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:07.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:07.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:07.315 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:07.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:07.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:07.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:07.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:07.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:07 smithi169 bash[19604]: audit 2024-02-12T16:08:06.283833+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? v1:172.21.15.121:0/1485285138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:08:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:07 smithi169 bash[19604]: cluster 2024-02-12T16:08:06.283899+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e226: 12 total, 12 up, 12 in 2024-02-12T16:08:07.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:07.566 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:07 smithi121 bash[18489]: audit 2024-02-12T16:08:06.283833+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? v1:172.21.15.121:0/1485285138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:08:07.566 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:07 smithi121 bash[18489]: cluster 2024-02-12T16:08:06.283899+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e226: 12 total, 12 up, 12 in 2024-02-12T16:08:07.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:07.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:07.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:07 smithi123 bash[19119]: audit 2024-02-12T16:08:06.283833+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? v1:172.21.15.121:0/1485285138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:08:07.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:07 smithi123 bash[19119]: cluster 2024-02-12T16:08:06.283899+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e226: 12 total, 12 up, 12 in 2024-02-12T16:08:07.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:07.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:07.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:07.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 10 1 2024-02-12T16:08:07.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:07.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:07.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:08.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:08.048 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:08.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:08.183 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:08.286 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:08.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:08.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:08.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:08.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:08 smithi169 bash[19604]: cluster 2024-02-12T16:08:07.608503+0000 mgr.y (mgr.14146) 678 : cluster [DBG] pgmap v811: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 966 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 1418/30928 objects misplaced (4.585%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:08:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:08 smithi169 bash[19604]: audit 2024-02-12T16:08:07.609393+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:08:08.560 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:08 smithi121 bash[18489]: cluster 2024-02-12T16:08:07.608503+0000 mgr.y (mgr.14146) 678 : cluster [DBG] pgmap v811: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 966 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 1418/30928 objects misplaced (4.585%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:08:08.560 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:08 smithi121 bash[18489]: audit 2024-02-12T16:08:07.609393+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:08:08.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:08 smithi123 bash[19119]: cluster 2024-02-12T16:08:07.608503+0000 mgr.y (mgr.14146) 678 : cluster [DBG] pgmap v811: 27 pgs: 3 active+recovering+remapped, 24 active+clean; 966 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 1418/30928 objects misplaced (4.585%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:08:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:08 smithi123 bash[19119]: audit 2024-02-12T16:08:07.609393+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:08:08.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:08.704 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:08.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:08:08.745 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:08:08.767 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:08:08.792 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:08:08.807 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:08.813 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:08:08.837 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:08:08.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:08.862 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:08:08.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:08:08.892 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:08:08.906 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:08:08.925 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:08:08.945 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:08:08.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:08.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:08:08.968 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.968 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.968 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:08:08.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:08.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:09.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:09.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:09.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:09.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:09.241 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:09.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:09.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:09.380 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:09.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:09.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:09.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:09.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:09.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:09.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:09.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:09 smithi169 bash[19604]: audit 2024-02-12T16:08:08.301652+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:08:09.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:09 smithi169 bash[19604]: cluster 2024-02-12T16:08:08.301726+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e227: 12 total, 12 up, 12 in 2024-02-12T16:08:09.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:09 smithi169 bash[19604]: audit 2024-02-12T16:08:08.701713+0000 mon.a (mon.0) 1050 : audit [DBG] from='client.? v1:172.21.15.121:0/4101843559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:09.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:09 smithi123 bash[19119]: audit 2024-02-12T16:08:08.301652+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:08:09.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:09 smithi123 bash[19119]: cluster 2024-02-12T16:08:08.301726+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e227: 12 total, 12 up, 12 in 2024-02-12T16:08:09.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:09 smithi123 bash[19119]: audit 2024-02-12T16:08:08.701713+0000 mon.a (mon.0) 1050 : audit [DBG] from='client.? v1:172.21.15.121:0/4101843559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:09.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:09.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:09.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:09 smithi121 bash[18489]: audit 2024-02-12T16:08:08.301652+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:08:09.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:09 smithi121 bash[18489]: cluster 2024-02-12T16:08:08.301726+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e227: 12 total, 12 up, 12 in 2024-02-12T16:08:09.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:09 smithi121 bash[18489]: audit 2024-02-12T16:08:08.701713+0000 mon.a (mon.0) 1050 : audit [DBG] from='client.? v1:172.21.15.121:0/4101843559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:09.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:09.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:09.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:09.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:09.900 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:09.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:10.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:10.041 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:10.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:10.175 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:10.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:10.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:10.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:10 smithi121 bash[18489]: cluster 2024-02-12T16:08:08.884105+0000 osd.8 (osd.8) 21 : cluster [DBG] 2.b scrub starts 2024-02-12T16:08:10.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:10 smithi121 bash[18489]: cluster 2024-02-12T16:08:09.040100+0000 osd.8 (osd.8) 22 : cluster [DBG] 2.b scrub ok 2024-02-12T16:08:10.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:10 smithi121 bash[18489]: cluster 2024-02-12T16:08:09.313307+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e228: 12 total, 12 up, 12 in 2024-02-12T16:08:10.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:10 smithi121 bash[18489]: cluster 2024-02-12T16:08:09.609006+0000 mgr.y (mgr.14146) 679 : cluster [DBG] pgmap v814: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 2 active+recovering+remapped, 23 active+clean; 856 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 1124/27388 objects misplaced (4.104%); 6.4 MiB/s, 102 objects/s recovering 2024-02-12T16:08:10.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:10 smithi121 bash[18489]: cluster 2024-02-12T16:08:10.276880+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e229: 12 total, 12 up, 12 in 2024-02-12T16:08:10.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:10.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:10 smithi169 bash[19604]: cluster 2024-02-12T16:08:08.884105+0000 osd.8 (osd.8) 21 : cluster [DBG] 2.b scrub starts 2024-02-12T16:08:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:10 smithi169 bash[19604]: cluster 2024-02-12T16:08:09.040100+0000 osd.8 (osd.8) 22 : cluster [DBG] 2.b scrub ok 2024-02-12T16:08:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:10 smithi169 bash[19604]: cluster 2024-02-12T16:08:09.313307+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e228: 12 total, 12 up, 12 in 2024-02-12T16:08:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:10 smithi169 bash[19604]: cluster 2024-02-12T16:08:09.609006+0000 mgr.y (mgr.14146) 679 : cluster [DBG] pgmap v814: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 2 active+recovering+remapped, 23 active+clean; 856 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 1124/27388 objects misplaced (4.104%); 6.4 MiB/s, 102 objects/s recovering 2024-02-12T16:08:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:10 smithi169 bash[19604]: cluster 2024-02-12T16:08:10.276880+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e229: 12 total, 12 up, 12 in 2024-02-12T16:08:10.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:10.582 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:10.619 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:10 smithi123 bash[19119]: cluster 2024-02-12T16:08:08.884105+0000 osd.8 (osd.8) 21 : cluster [DBG] 2.b scrub starts 2024-02-12T16:08:10.619 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:10 smithi123 bash[19119]: cluster 2024-02-12T16:08:09.040100+0000 osd.8 (osd.8) 22 : cluster [DBG] 2.b scrub ok 2024-02-12T16:08:10.619 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:10 smithi123 bash[19119]: cluster 2024-02-12T16:08:09.313307+0000 mon.a (mon.0) 1051 : cluster [DBG] osdmap e228: 12 total, 12 up, 12 in 2024-02-12T16:08:10.619 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:10 smithi123 bash[19119]: cluster 2024-02-12T16:08:09.609006+0000 mgr.y (mgr.14146) 679 : cluster [DBG] pgmap v814: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 2 active+recovering+remapped, 23 active+clean; 856 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 1124/27388 objects misplaced (4.104%); 6.4 MiB/s, 102 objects/s recovering 2024-02-12T16:08:10.619 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:10 smithi123 bash[19119]: cluster 2024-02-12T16:08:10.276880+0000 mon.a (mon.0) 1052 : cluster [DBG] osdmap e229: 12 total, 12 up, 12 in 2024-02-12T16:08:10.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:10.715 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:10.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:10.851 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:10.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:10.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:10.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:11.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:11.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:11.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:11.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:11.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:11.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:11.295 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:08:11.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:11.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:09.954930+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:08:11.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:10.030882+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:08:11.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:10.038299+0000 osd.9 (osd.9) 19 : cluster [DBG] 2.d scrub starts 2024-02-12T16:08:11.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:10.200252+0000 osd.9 (osd.9) 20 : cluster [DBG] 2.d scrub ok 2024-02-12T16:08:11.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:10.274558+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.13 starting backfill to osd.4 from (0'0,0'0] MAX to 226'15204 2024-02-12T16:08:11.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:10.298524+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.13 starting backfill to osd.9 from (0'0,0'0] MAX to 226'15204 2024-02-12T16:08:11.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:10.308278+0000 mon.a (mon.0) 1053 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:08:11.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: audit 2024-02-12T16:08:10.580640+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? v1:172.21.15.121:0/3123275677' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:08:11.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:11.279257+0000 mon.a (mon.0) 1055 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:08:11.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: audit 2024-02-12T16:08:11.285479+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? v1:172.21.15.121:0/3123275677' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:08:11.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:11 smithi123 bash[19119]: cluster 2024-02-12T16:08:11.285538+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e230: 12 total, 12 up, 12 in 2024-02-12T16:08:11.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:11.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:11.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:11.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:11.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:11.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:09.954930+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:10.030882+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:10.038299+0000 osd.9 (osd.9) 19 : cluster [DBG] 2.d scrub starts 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:10.200252+0000 osd.9 (osd.9) 20 : cluster [DBG] 2.d scrub ok 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:10.274558+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.13 starting backfill to osd.4 from (0'0,0'0] MAX to 226'15204 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:10.298524+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.13 starting backfill to osd.9 from (0'0,0'0] MAX to 226'15204 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:10.308278+0000 mon.a (mon.0) 1053 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: audit 2024-02-12T16:08:10.580640+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? v1:172.21.15.121:0/3123275677' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:11.279257+0000 mon.a (mon.0) 1055 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: audit 2024-02-12T16:08:11.285479+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? v1:172.21.15.121:0/3123275677' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:08:11.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:11 smithi169 bash[19604]: cluster 2024-02-12T16:08:11.285538+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e230: 12 total, 12 up, 12 in 2024-02-12T16:08:11.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:11.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:09.954930+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.3 scrub starts 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:10.030882+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.3 scrub ok 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:10.038299+0000 osd.9 (osd.9) 19 : cluster [DBG] 2.d scrub starts 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:10.200252+0000 osd.9 (osd.9) 20 : cluster [DBG] 2.d scrub ok 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:10.274558+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.13 starting backfill to osd.4 from (0'0,0'0] MAX to 226'15204 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:10.298524+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.13 starting backfill to osd.9 from (0'0,0'0] MAX to 226'15204 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:10.308278+0000 mon.a (mon.0) 1053 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: audit 2024-02-12T16:08:10.580640+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? v1:172.21.15.121:0/3123275677' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:11.279257+0000 mon.a (mon.0) 1055 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:08:11.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: audit 2024-02-12T16:08:11.285479+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? v1:172.21.15.121:0/3123275677' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:08:11.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:11 smithi121 bash[18489]: cluster 2024-02-12T16:08:11.285538+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e230: 12 total, 12 up, 12 in 2024-02-12T16:08:11.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:11.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:11.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:11.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:11.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:12.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:12.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:12.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:12.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:08:12.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:12.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:12.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:12.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:12.377 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:12.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:12.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:12.520 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:08:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:08:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:335685 2.0 2:02180f24:::benchmark_data_smithi169_42458_object162651:head [delete] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-02-12T16:08:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:08:12.502556+0000", 2024-02-12T16:08:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.015526032, 2024-02-12T16:08:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015550118999999999, 2024-02-12T16:08:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:08:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:08:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:08:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:08:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:08:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 335685 2024-02-12T16:08:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:08:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502556+0000", 2024-02-12T16:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502556+0000", 2024-02-12T16:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502553+0000", 2024-02-12T16:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3278000000000001e-05 2024-02-12T16:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502566+0000", 2024-02-12T16:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1844999999999998e-05 2024-02-12T16:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502588+0000", 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8190000000000003e-06 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502597+0000", 2024-02-12T16:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2473999999999997e-05 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502680+0000", 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000196391 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:08:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.502876+0000", 2024-02-12T16:08:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00097104400000000001 2024-02-12T16:08:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:08:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.503847+0000", 2024-02-12T16:08:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061442049999999998 2024-02-12T16:08:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:08:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.509992+0000", 2024-02-12T16:08:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074355189999999998 2024-02-12T16:08:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:08:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:335690 2.8 2:16f8b526:::benchmark_data_smithi169_42458_object162656:head [delete] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-02-12T16:08:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:08:12.508029+0000", 2024-02-12T16:08:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.010052877999999999, 2024-02-12T16:08:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010152105999999999, 2024-02-12T16:08:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:08:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:08:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:08:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:08:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:08:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 335690 2024-02-12T16:08:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:08:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:08:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508029+0000", 2024-02-12T16:08:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:08:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508029+0000", 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508027+0000", 2024-02-12T16:08:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1260000000000001e-05 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508038+0000", 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8284e-05 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:08:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508056+0000", 2024-02-12T16:08:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9559999999999997e-06 2024-02-12T16:08:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:08:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508064+0000", 2024-02-12T16:08:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4920000000000006e-05 2024-02-12T16:08:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:08:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508159+0000", 2024-02-12T16:08:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017303899999999999 2024-02-12T16:08:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:08:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.508332+0000", 2024-02-12T16:08:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00093349999999999998 2024-02-12T16:08:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:08:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.509266+0000", 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001236663 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:335695 2.8 2:12700fb7:::benchmark_data_smithi169_42458_object162661:head [delete] snapc 0=[] ondisk+write+known_if_redirected e231)", 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:08:12.510323+0000", 2024-02-12T16:08:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0077594379999999996, 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079061229999999993, 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 335695 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.510323+0000", 2024-02-12T16:08:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.510323+0000", 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.510321+0000", 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3770000000000001e-06 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:08:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.510327+0000", 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2043e-05 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.510339+0000", 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0930000000000002e-06 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.510344+0000", 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043877619999999999 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:08:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.514732+0000", 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015650000000000001 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.514889+0000", 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083287900000000002 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.515722+0000", 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053988969999999997 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.14811.0:335690 2.8 e231/171)", 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:08:12.513902+0000", 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0041801609999999999, 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043712949999999999, 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "queued for pg", 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.513902+0000", 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.513902+0000", 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.513900+0000", 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7070000000000004e-06 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.513910+0000", 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4268e-05 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.513924+0000", 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0329999999999997e-06 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:08:12.513930+0000", 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8387000000000001e-05 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 4 2024-02-12T16:08:12.546 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:08:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:12.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:12.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:12.585 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:12 smithi123 bash[19119]: cluster 2024-02-12T16:08:11.609436+0000 mgr.y (mgr.14146) 680 : cluster [DBG] pgmap v817: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+remapped, 22 active+clean; 795 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 1908/25452 objects misplaced (7.496%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T16:08:12.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:12.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:12 smithi121 bash[18489]: cluster 2024-02-12T16:08:11.609436+0000 mgr.y (mgr.14146) 680 : cluster [DBG] pgmap v817: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+remapped, 22 active+clean; 795 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 1908/25452 objects misplaced (7.496%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T16:08:12.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:12.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:12.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:12 smithi169 bash[19604]: cluster 2024-02-12T16:08:11.609436+0000 mgr.y (mgr.14146) 680 : cluster [DBG] pgmap v817: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+remapped, 22 active+clean; 795 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 1908/25452 objects misplaced (7.496%); 2.4 MiB/s, 37 objects/s recovering 2024-02-12T16:08:12.798 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:12.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:12.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:12.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:12.931 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:12.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:13.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:13.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:13.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:13.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:13.211 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:13.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:13.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:08:13.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:13.345 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:13.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:13.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:13.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:13.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:13 smithi169 bash[19604]: cluster 2024-02-12T16:08:12.228308+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:13 smithi169 bash[19604]: cluster 2024-02-12T16:08:12.228355+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:13 smithi169 bash[19604]: cluster 2024-02-12T16:08:12.377803+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e231: 12 total, 12 up, 12 in 2024-02-12T16:08:13.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:13.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:13.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:13 smithi123 bash[19119]: cluster 2024-02-12T16:08:12.228308+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:13.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:13 smithi123 bash[19119]: cluster 2024-02-12T16:08:12.228355+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:13.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:13 smithi123 bash[19119]: cluster 2024-02-12T16:08:12.377803+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e231: 12 total, 12 up, 12 in 2024-02-12T16:08:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:13.641 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:13 smithi121 bash[18489]: cluster 2024-02-12T16:08:12.228308+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:13.641 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:13 smithi121 bash[18489]: cluster 2024-02-12T16:08:12.228355+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:13.641 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:13 smithi121 bash[18489]: cluster 2024-02-12T16:08:12.377803+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e231: 12 total, 12 up, 12 in 2024-02-12T16:08:13.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:13.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:13.781 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:13.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:13.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:13.923 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:13.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:14.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:14.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:14.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:14.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:14.177 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:14.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:14.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:14.303 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:14.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:14.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:08:14.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:14.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:14.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:14.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:14.601 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:14.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:14.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:14 smithi123 bash[19119]: cluster 2024-02-12T16:08:12.875448+0000 osd.7 (osd.7) 37 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:08:14.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:14 smithi123 bash[19119]: cluster 2024-02-12T16:08:12.878481+0000 osd.7 (osd.7) 38 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:08:14.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:14 smithi123 bash[19119]: cluster 2024-02-12T16:08:13.609873+0000 mgr.y (mgr.14146) 681 : cluster [DBG] pgmap v819: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+remapped, 22 active+clean; 748 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1654/23938 objects misplaced (6.910%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:08:14.664 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:14 smithi121 bash[18489]: cluster 2024-02-12T16:08:12.875448+0000 osd.7 (osd.7) 37 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:08:14.664 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:14 smithi121 bash[18489]: cluster 2024-02-12T16:08:12.878481+0000 osd.7 (osd.7) 38 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:08:14.664 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:14 smithi121 bash[18489]: cluster 2024-02-12T16:08:13.609873+0000 mgr.y (mgr.14146) 681 : cluster [DBG] pgmap v819: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+remapped, 22 active+clean; 748 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1654/23938 objects misplaced (6.910%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:08:14.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:14.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:14 smithi169 bash[19604]: cluster 2024-02-12T16:08:12.875448+0000 osd.7 (osd.7) 37 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:08:14.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:14 smithi169 bash[19604]: cluster 2024-02-12T16:08:12.878481+0000 osd.7 (osd.7) 38 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:08:14.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:14 smithi169 bash[19604]: cluster 2024-02-12T16:08:13.609873+0000 mgr.y (mgr.14146) 681 : cluster [DBG] pgmap v819: 27 pgs: 1 creating+peering, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+remapped, 22 active+clean; 748 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1654/23938 objects misplaced (6.910%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:08:14.743 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:14.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:14.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:14.878 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:14.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:14.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:14.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:15.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:15.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:15.155 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:15.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:15.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:15.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:15.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:15.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:15.565 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:15.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:15.695 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:15.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:15.943 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:15.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:15.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:16.072 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:16.101 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:16.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:16.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:16.233 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:16.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:16.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:16.382 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:16.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:16.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:16.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:16.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:14.673561+0000 osd.4 (osd.4) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:14.673646+0000 osd.4 (osd.4) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:15.208943+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:15.208987+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:15.395805+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e232: 12 total, 12 up, 12 in 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:15.610286+0000 mgr.y (mgr.14146) 682 : cluster [DBG] pgmap v821: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 25 active+clean; 622 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1470/19908 objects misplaced (7.384%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:15.935734+0000 osd.11 (osd.11) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:16 smithi169 bash[19604]: cluster 2024-02-12T16:08:15.935788+0000 osd.11 (osd.11) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:16.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:16.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:14.673561+0000 osd.4 (osd.4) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:14.673646+0000 osd.4 (osd.4) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:15.208943+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:15.208987+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:15.395805+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e232: 12 total, 12 up, 12 in 2024-02-12T16:08:16.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:15.610286+0000 mgr.y (mgr.14146) 682 : cluster [DBG] pgmap v821: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 25 active+clean; 622 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1470/19908 objects misplaced (7.384%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:08:16.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:15.935734+0000 osd.11 (osd.11) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:16 smithi121 bash[18489]: cluster 2024-02-12T16:08:15.935788+0000 osd.11 (osd.11) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:14.673561+0000 osd.4 (osd.4) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:14.673646+0000 osd.4 (osd.4) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:15.208943+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:15.208987+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:15.395805+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e232: 12 total, 12 up, 12 in 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:15.610286+0000 mgr.y (mgr.14146) 682 : cluster [DBG] pgmap v821: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 25 active+clean; 622 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1470/19908 objects misplaced (7.384%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:15.935734+0000 osd.11 (osd.11) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:16.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:16 smithi123 bash[19119]: cluster 2024-02-12T16:08:15.935788+0000 osd.11 (osd.11) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:16.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:16.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:16.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:16.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:16.907 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:16.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:17.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:17.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:17.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:17.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:17.187 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:17.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:17.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:17.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:17.327 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:17.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:17.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:17.460 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:17.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:17.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:17.586 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:17.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:17.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:17 smithi121 bash[18489]: cluster 2024-02-12T16:08:16.390185+0000 mon.a (mon.0) 1060 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T16:08:17.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:17 smithi121 bash[18489]: cluster 2024-02-12T16:08:16.390227+0000 mon.a (mon.0) 1061 : cluster [INF] Cluster is now healthy 2024-02-12T16:08:17.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:17 smithi121 bash[18489]: cluster 2024-02-12T16:08:16.402465+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e233: 12 total, 12 up, 12 in 2024-02-12T16:08:17.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:17.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:17.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:17 smithi169 bash[19604]: cluster 2024-02-12T16:08:16.390185+0000 mon.a (mon.0) 1060 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T16:08:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:17 smithi169 bash[19604]: cluster 2024-02-12T16:08:16.390227+0000 mon.a (mon.0) 1061 : cluster [INF] Cluster is now healthy 2024-02-12T16:08:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:17 smithi169 bash[19604]: cluster 2024-02-12T16:08:16.402465+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e233: 12 total, 12 up, 12 in 2024-02-12T16:08:17.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:17.877 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:17.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:17.887 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:17 smithi123 bash[19119]: cluster 2024-02-12T16:08:16.390185+0000 mon.a (mon.0) 1060 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T16:08:17.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:17 smithi123 bash[19119]: cluster 2024-02-12T16:08:16.390227+0000 mon.a (mon.0) 1061 : cluster [INF] Cluster is now healthy 2024-02-12T16:08:17.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:17 smithi123 bash[19119]: cluster 2024-02-12T16:08:16.402465+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e233: 12 total, 12 up, 12 in 2024-02-12T16:08:17.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:18.012 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:08:18.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:18.119 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:18.145 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:18.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:18.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:18.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:18.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:18.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:18.551 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:18.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:18 smithi123 bash[19119]: cluster 2024-02-12T16:08:17.610743+0000 mgr.y (mgr.14146) 683 : cluster [DBG] pgmap v823: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 479 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772/15330 objects misplaced (5.036%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T16:08:18.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:18.665 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:18 smithi121 bash[18489]: cluster 2024-02-12T16:08:17.610743+0000 mgr.y (mgr.14146) 683 : cluster [DBG] pgmap v823: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 479 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772/15330 objects misplaced (5.036%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T16:08:18.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:18.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:18 smithi169 bash[19604]: cluster 2024-02-12T16:08:17.610743+0000 mgr.y (mgr.14146) 683 : cluster [DBG] pgmap v823: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 479 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 772/15330 objects misplaced (5.036%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T16:08:18.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:18.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:18.990 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:18.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:19.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:19.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:19.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:19.267 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:19.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:19.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:19.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:19.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:19.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:19.539 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:19.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:19.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:19 smithi169 bash[19604]: cluster 2024-02-12T16:08:18.730136+0000 osd.8 (osd.8) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:19.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:19 smithi169 bash[19604]: cluster 2024-02-12T16:08:18.730187+0000 osd.8 (osd.8) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:19.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:19.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:19 smithi121 bash[18489]: cluster 2024-02-12T16:08:18.730136+0000 osd.8 (osd.8) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:19.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:19 smithi121 bash[18489]: cluster 2024-02-12T16:08:18.730187+0000 osd.8 (osd.8) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:19.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:19.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:19 smithi123 bash[19119]: cluster 2024-02-12T16:08:18.730136+0000 osd.8 (osd.8) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:19.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:19 smithi123 bash[19119]: cluster 2024-02-12T16:08:18.730187+0000 osd.8 (osd.8) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:19.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:19.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:19.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:19.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:19.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:19.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:19.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:20.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:20.110 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:20.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:20.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:20.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:20.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:20.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:20.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:20.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:20.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:20.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:20 smithi121 bash[18489]: cluster 2024-02-12T16:08:19.414985+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e234: 12 total, 12 up, 12 in 2024-02-12T16:08:20.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:20 smithi121 bash[18489]: cluster 2024-02-12T16:08:19.611163+0000 mgr.y (mgr.14146) 684 : cluster [DBG] pgmap v825: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 354 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 772/11338 objects misplaced (6.809%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:08:20.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:20 smithi121 bash[18489]: cluster 2024-02-12T16:08:19.829221+0000 osd.11 (osd.11) 27 : cluster [DBG] 2.c scrub starts 2024-02-12T16:08:20.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:20 smithi121 bash[18489]: cluster 2024-02-12T16:08:19.954600+0000 osd.11 (osd.11) 28 : cluster [DBG] 2.c scrub ok 2024-02-12T16:08:20.547 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:20.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:20.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:20.680 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:20.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:20.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:20.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:20 smithi169 bash[19604]: cluster 2024-02-12T16:08:19.414985+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e234: 12 total, 12 up, 12 in 2024-02-12T16:08:20.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:20 smithi169 bash[19604]: cluster 2024-02-12T16:08:19.611163+0000 mgr.y (mgr.14146) 684 : cluster [DBG] pgmap v825: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 354 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 772/11338 objects misplaced (6.809%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:08:20.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:20 smithi169 bash[19604]: cluster 2024-02-12T16:08:19.829221+0000 osd.11 (osd.11) 27 : cluster [DBG] 2.c scrub starts 2024-02-12T16:08:20.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:20 smithi169 bash[19604]: cluster 2024-02-12T16:08:19.954600+0000 osd.11 (osd.11) 28 : cluster [DBG] 2.c scrub ok 2024-02-12T16:08:20.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:20.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:20 smithi123 bash[19119]: cluster 2024-02-12T16:08:19.414985+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e234: 12 total, 12 up, 12 in 2024-02-12T16:08:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:20 smithi123 bash[19119]: cluster 2024-02-12T16:08:19.611163+0000 mgr.y (mgr.14146) 684 : cluster [DBG] pgmap v825: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 354 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 772/11338 objects misplaced (6.809%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:08:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:20 smithi123 bash[19119]: cluster 2024-02-12T16:08:19.829221+0000 osd.11 (osd.11) 27 : cluster [DBG] 2.c scrub starts 2024-02-12T16:08:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:20 smithi123 bash[19119]: cluster 2024-02-12T16:08:19.954600+0000 osd.11 (osd.11) 28 : cluster [DBG] 2.c scrub ok 2024-02-12T16:08:20.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:20.943 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:20.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:21.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:21.082 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:21.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:21.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:21.207 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:21.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:21.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:21.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:21.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:21.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:21.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:21.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:21.534 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:08:21.534 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:08:21.534 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2024-02-12T16:08:21.534 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2024-02-12T16:08:21.534 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:08:21.585 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:21.621 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:21.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:21.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:21.762 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:21.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:21.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:21.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:21.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:21.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:22.019 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:22.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:22.147 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:22.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:22.278 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:22.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:08:22.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:22.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:22.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:22.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:22.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:22.665 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:22.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.10 to 1 (10000) 2024-02-12T16:08:22.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:22.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:22 smithi121 bash[18489]: cluster 2024-02-12T16:08:21.611618+0000 mgr.y (mgr.14146) 685 : cluster [DBG] pgmap v826: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 25 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s; 488/8280 objects misplaced (5.894%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:08:22.774 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:22 smithi121 bash[18489]: cluster 2024-02-12T16:08:21.715533+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e235: 12 total, 12 up, 12 in 2024-02-12T16:08:22.774 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:22 smithi121 bash[18489]: cluster 2024-02-12T16:08:21.772496+0000 osd.9 (osd.9) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:22.774 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:22 smithi121 bash[18489]: cluster 2024-02-12T16:08:21.772554+0000 osd.9 (osd.9) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:22.774 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:22 smithi121 bash[18489]: cluster 2024-02-12T16:08:21.870541+0000 osd.8 (osd.8) 25 : cluster [DBG] 2.f scrub starts 2024-02-12T16:08:22.774 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:22 smithi121 bash[18489]: audit 2024-02-12T16:08:22.169722+0000 mon.a (mon.0) 1065 : audit [INF] from='client.? v1:172.21.15.121:0/976157641' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:08:22.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:22.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:22.934 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:22.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:23.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:22 smithi169 bash[19604]: cluster 2024-02-12T16:08:21.611618+0000 mgr.y (mgr.14146) 685 : cluster [DBG] pgmap v826: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 25 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s; 488/8280 objects misplaced (5.894%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:08:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:22 smithi169 bash[19604]: cluster 2024-02-12T16:08:21.715533+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e235: 12 total, 12 up, 12 in 2024-02-12T16:08:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:22 smithi169 bash[19604]: cluster 2024-02-12T16:08:21.772496+0000 osd.9 (osd.9) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:22 smithi169 bash[19604]: cluster 2024-02-12T16:08:21.772554+0000 osd.9 (osd.9) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:22 smithi169 bash[19604]: cluster 2024-02-12T16:08:21.870541+0000 osd.8 (osd.8) 25 : cluster [DBG] 2.f scrub starts 2024-02-12T16:08:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:22 smithi169 bash[19604]: audit 2024-02-12T16:08:22.169722+0000 mon.a (mon.0) 1065 : audit [INF] from='client.? v1:172.21.15.121:0/976157641' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:08:23.077 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:23.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:22 smithi123 bash[19119]: cluster 2024-02-12T16:08:21.611618+0000 mgr.y (mgr.14146) 685 : cluster [DBG] pgmap v826: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 25 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s; 488/8280 objects misplaced (5.894%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:08:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:22 smithi123 bash[19119]: cluster 2024-02-12T16:08:21.715533+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e235: 12 total, 12 up, 12 in 2024-02-12T16:08:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:22 smithi123 bash[19119]: cluster 2024-02-12T16:08:21.772496+0000 osd.9 (osd.9) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:22 smithi123 bash[19119]: cluster 2024-02-12T16:08:21.772554+0000 osd.9 (osd.9) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:22 smithi123 bash[19119]: cluster 2024-02-12T16:08:21.870541+0000 osd.8 (osd.8) 25 : cluster [DBG] 2.f scrub starts 2024-02-12T16:08:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:22 smithi123 bash[19119]: audit 2024-02-12T16:08:22.169722+0000 mon.a (mon.0) 1065 : audit [INF] from='client.? v1:172.21.15.121:0/976157641' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:08:23.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:23.218 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:23.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:23.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:23.349 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:23.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:23.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:23.480 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:23.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:23.518 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 173031 objects 2024-02-12T16:08:23.519 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :153.854676 2024-02-12T16:08:23.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:23.625 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:23.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:23.729 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:23.765 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:23.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:23.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:23 smithi123 bash[19119]: cluster 2024-02-12T16:08:21.889012+0000 osd.8 (osd.8) 26 : cluster [DBG] 2.f scrub ok 2024-02-12T16:08:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:23 smithi123 bash[19119]: cluster 2024-02-12T16:08:22.636449+0000 osd.6 (osd.6) 24 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:23 smithi123 bash[19119]: cluster 2024-02-12T16:08:22.636537+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:23 smithi123 bash[19119]: audit 2024-02-12T16:08:22.718765+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? v1:172.21.15.121:0/976157641' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:08:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:23 smithi123 bash[19119]: cluster 2024-02-12T16:08:22.718929+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e236: 12 total, 12 up, 12 in 2024-02-12T16:08:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:23 smithi123 bash[19119]: audit 2024-02-12T16:08:23.612997+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:08:23.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:23.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:23.943 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:08:23.964 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:08:23.992 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:08:24.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:24.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:23 smithi121 bash[18489]: cluster 2024-02-12T16:08:21.889012+0000 osd.8 (osd.8) 26 : cluster [DBG] 2.f scrub ok 2024-02-12T16:08:24.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:23 smithi121 bash[18489]: cluster 2024-02-12T16:08:22.636449+0000 osd.6 (osd.6) 24 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:24.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:23 smithi121 bash[18489]: cluster 2024-02-12T16:08:22.636537+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:24.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:23 smithi121 bash[18489]: audit 2024-02-12T16:08:22.718765+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? v1:172.21.15.121:0/976157641' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:08:24.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:23 smithi121 bash[18489]: cluster 2024-02-12T16:08:22.718929+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e236: 12 total, 12 up, 12 in 2024-02-12T16:08:24.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:23 smithi121 bash[18489]: audit 2024-02-12T16:08:23.612997+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:08:24.019 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:08:24.032 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:24.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 11 1 2024-02-12T16:08:24.042 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:08:24.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:23 smithi169 bash[19604]: cluster 2024-02-12T16:08:21.889012+0000 osd.8 (osd.8) 26 : cluster [DBG] 2.f scrub ok 2024-02-12T16:08:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:23 smithi169 bash[19604]: cluster 2024-02-12T16:08:22.636449+0000 osd.6 (osd.6) 24 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:23 smithi169 bash[19604]: cluster 2024-02-12T16:08:22.636537+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:23 smithi169 bash[19604]: audit 2024-02-12T16:08:22.718765+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? v1:172.21.15.121:0/976157641' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:08:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:23 smithi169 bash[19604]: cluster 2024-02-12T16:08:22.718929+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e236: 12 total, 12 up, 12 in 2024-02-12T16:08:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:23 smithi169 bash[19604]: audit 2024-02-12T16:08:23.612997+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:08:24.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:08:24.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:08:24.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:08:24.135 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:24.148 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:08:24.170 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:08:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:24.197 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:08:24.221 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:08:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.223 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:08:24.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:24.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:24.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:24.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:24.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:24.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:24.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:24.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:24.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:24.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:24.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:24 smithi121 bash[18489]: cluster 2024-02-12T16:08:23.612025+0000 mgr.y (mgr.14146) 686 : cluster [DBG] pgmap v829: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 170 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 180/5438 objects misplaced (3.310%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:08:24.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:24 smithi121 bash[18489]: audit 2024-02-12T16:08:23.738769+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:08:24.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:24 smithi121 bash[18489]: cluster 2024-02-12T16:08:23.738842+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e237: 12 total, 12 up, 12 in 2024-02-12T16:08:24.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:24 smithi121 bash[18489]: audit 2024-02-12T16:08:23.911122+0000 mon.a (mon.0) 1071 : audit [DBG] from='client.? v1:172.21.15.121:0/4002114293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:24.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:24.992 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:24.994 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:24 smithi169 bash[19604]: cluster 2024-02-12T16:08:23.612025+0000 mgr.y (mgr.14146) 686 : cluster [DBG] pgmap v829: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 170 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 180/5438 objects misplaced (3.310%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:08:24.994 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:24 smithi169 bash[19604]: audit 2024-02-12T16:08:23.738769+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:08:24.995 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:24 smithi169 bash[19604]: cluster 2024-02-12T16:08:23.738842+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e237: 12 total, 12 up, 12 in 2024-02-12T16:08:24.995 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:24 smithi169 bash[19604]: audit 2024-02-12T16:08:23.911122+0000 mon.a (mon.0) 1071 : audit [DBG] from='client.? v1:172.21.15.121:0/4002114293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:24.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:25.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:25.121 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:25.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:24 smithi123 bash[19119]: cluster 2024-02-12T16:08:23.612025+0000 mgr.y (mgr.14146) 686 : cluster [DBG] pgmap v829: 27 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 25 active+clean; 170 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 180/5438 objects misplaced (3.310%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:08:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:24 smithi123 bash[19119]: audit 2024-02-12T16:08:23.738769+0000 mon.a (mon.0) 1069 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:08:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:24 smithi123 bash[19119]: cluster 2024-02-12T16:08:23.738842+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e237: 12 total, 12 up, 12 in 2024-02-12T16:08:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:24 smithi123 bash[19119]: audit 2024-02-12T16:08:23.911122+0000 mon.a (mon.0) 1071 : audit [DBG] from='client.? v1:172.21.15.121:0/4002114293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:25.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:25.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:25.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:25.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:25.400 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:25.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:25.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:25.541 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:25.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:25.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:25.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:25.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:25.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:25.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:25.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:25 smithi123 bash[19119]: cluster 2024-02-12T16:08:24.135588+0000 osd.9 (osd.9) 23 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:08:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:25 smithi123 bash[19119]: cluster 2024-02-12T16:08:24.146103+0000 osd.9 (osd.9) 24 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:08:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:25 smithi123 bash[19119]: cluster 2024-02-12T16:08:24.747095+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e238: 12 total, 12 up, 12 in 2024-02-12T16:08:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:25 smithi123 bash[19119]: cluster 2024-02-12T16:08:25.283560+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e239: 12 total, 12 up, 12 in 2024-02-12T16:08:25.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:25.937 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:25.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:26.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:26.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:25 smithi169 bash[19604]: cluster 2024-02-12T16:08:24.135588+0000 osd.9 (osd.9) 23 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:08:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:25 smithi169 bash[19604]: cluster 2024-02-12T16:08:24.146103+0000 osd.9 (osd.9) 24 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:08:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:25 smithi169 bash[19604]: cluster 2024-02-12T16:08:24.747095+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e238: 12 total, 12 up, 12 in 2024-02-12T16:08:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:25 smithi169 bash[19604]: cluster 2024-02-12T16:08:25.283560+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e239: 12 total, 12 up, 12 in 2024-02-12T16:08:26.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:25 smithi121 bash[18489]: cluster 2024-02-12T16:08:24.135588+0000 osd.9 (osd.9) 23 : cluster [DBG] 2.13 scrub starts 2024-02-12T16:08:26.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:25 smithi121 bash[18489]: cluster 2024-02-12T16:08:24.146103+0000 osd.9 (osd.9) 24 : cluster [DBG] 2.13 scrub ok 2024-02-12T16:08:26.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:25 smithi121 bash[18489]: cluster 2024-02-12T16:08:24.747095+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e238: 12 total, 12 up, 12 in 2024-02-12T16:08:26.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:25 smithi121 bash[18489]: cluster 2024-02-12T16:08:25.283560+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e239: 12 total, 12 up, 12 in 2024-02-12T16:08:26.070 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:26.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:26.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:26.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:26.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:26.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:26.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:26.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:26.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:26.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:26.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:26.609 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:26.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:26.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:26.750 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:26.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:26.766 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:08:26.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:26.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:26 smithi169 bash[19604]: cluster 2024-02-12T16:08:25.281988+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.14 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15449 2024-02-12T16:08:26.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:26 smithi169 bash[19604]: cluster 2024-02-12T16:08:25.300089+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.14 starting backfill to osd.9 from (0'0,0'0] MAX to 236'15449 2024-02-12T16:08:26.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:26 smithi169 bash[19604]: cluster 2024-02-12T16:08:25.612670+0000 mgr.y (mgr.14146) 687 : cluster [DBG] pgmap v833: 27 pgs: 1 remapped+peering, 1 active+recovering+remapped, 1 active+remapped+backfilling, 24 active+clean; 46 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 180/1482 objects misplaced (12.146%) 2024-02-12T16:08:26.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:26 smithi169 bash[19604]: audit 2024-02-12T16:08:25.852423+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.121:0/3909428435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:08:26.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:26.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:26.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:27.005 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:26 smithi121 bash[18489]: cluster 2024-02-12T16:08:25.281988+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.14 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15449 2024-02-12T16:08:27.005 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:26 smithi121 bash[18489]: cluster 2024-02-12T16:08:25.300089+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.14 starting backfill to osd.9 from (0'0,0'0] MAX to 236'15449 2024-02-12T16:08:27.005 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:26 smithi121 bash[18489]: cluster 2024-02-12T16:08:25.612670+0000 mgr.y (mgr.14146) 687 : cluster [DBG] pgmap v833: 27 pgs: 1 remapped+peering, 1 active+recovering+remapped, 1 active+remapped+backfilling, 24 active+clean; 46 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 180/1482 objects misplaced (12.146%) 2024-02-12T16:08:27.005 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:26 smithi121 bash[18489]: audit 2024-02-12T16:08:25.852423+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.121:0/3909428435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:08:27.026 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:27.079 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:08:27.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:27.153 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:26 smithi123 bash[19119]: cluster 2024-02-12T16:08:25.281988+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.14 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15449 2024-02-12T16:08:27.153 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:26 smithi123 bash[19119]: cluster 2024-02-12T16:08:25.300089+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.14 starting backfill to osd.9 from (0'0,0'0] MAX to 236'15449 2024-02-12T16:08:27.153 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:26 smithi123 bash[19119]: cluster 2024-02-12T16:08:25.612670+0000 mgr.y (mgr.14146) 687 : cluster [DBG] pgmap v833: 27 pgs: 1 remapped+peering, 1 active+recovering+remapped, 1 active+remapped+backfilling, 24 active+clean; 46 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 180/1482 objects misplaced (12.146%) 2024-02-12T16:08:27.153 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:26 smithi123 bash[19119]: audit 2024-02-12T16:08:25.852423+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? v1:172.21.15.121:0/3909428435' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:08:27.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:27.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:27.298 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:27.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:08:27.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:27.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:27.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:27.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:27.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:27.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:27.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:27 smithi169 bash[19604]: cluster 2024-02-12T16:08:26.744454+0000 mon.a (mon.0) 1075 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:27.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:27 smithi169 bash[19604]: audit 2024-02-12T16:08:26.756533+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? v1:172.21.15.121:0/3909428435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:08:27.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:27 smithi169 bash[19604]: cluster 2024-02-12T16:08:26.756614+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e240: 12 total, 12 up, 12 in 2024-02-12T16:08:27.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:27.937 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:27.948 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:27 smithi123 bash[19119]: cluster 2024-02-12T16:08:26.744454+0000 mon.a (mon.0) 1075 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:27.948 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:27 smithi123 bash[19119]: audit 2024-02-12T16:08:26.756533+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? v1:172.21.15.121:0/3909428435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:08:27.949 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:27 smithi123 bash[19119]: cluster 2024-02-12T16:08:26.756614+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e240: 12 total, 12 up, 12 in 2024-02-12T16:08:27.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:27.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:28.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:28.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:28.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:27 smithi121 bash[18489]: cluster 2024-02-12T16:08:26.744454+0000 mon.a (mon.0) 1075 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:28.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:27 smithi121 bash[18489]: audit 2024-02-12T16:08:26.756533+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? v1:172.21.15.121:0/3909428435' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:08:28.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:27 smithi121 bash[18489]: cluster 2024-02-12T16:08:26.756614+0000 mon.a (mon.0) 1077 : cluster [DBG] osdmap e240: 12 total, 12 up, 12 in 2024-02-12T16:08:28.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:28.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:28.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:28.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:28.388 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:28.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:28.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:28.525 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:28.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:28.602 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-12T16:08:28.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-12T16:08:28.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:28.664 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:28.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:28.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:28.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:28.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:28.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:28.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:28 smithi121 bash[18489]: cluster 2024-02-12T16:08:27.613377+0000 mgr.y (mgr.14146) 688 : cluster [DBG] pgmap v835: 27 pgs: 1 active+recovering+degraded+remapped, 1 remapped+peering, 25 active+clean; 13 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:28.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:28 smithi121 bash[18489]: cluster 2024-02-12T16:08:27.752617+0000 mon.a (mon.0) 1078 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-02-12T16:08:28.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:28 smithi121 bash[18489]: cluster 2024-02-12T16:08:27.766596+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e241: 12 total, 12 up, 12 in 2024-02-12T16:08:28.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:28 smithi169 bash[19604]: cluster 2024-02-12T16:08:27.613377+0000 mgr.y (mgr.14146) 688 : cluster [DBG] pgmap v835: 27 pgs: 1 active+recovering+degraded+remapped, 1 remapped+peering, 25 active+clean; 13 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:28.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:28 smithi169 bash[19604]: cluster 2024-02-12T16:08:27.752617+0000 mon.a (mon.0) 1078 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-02-12T16:08:28.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:28 smithi169 bash[19604]: cluster 2024-02-12T16:08:27.766596+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e241: 12 total, 12 up, 12 in 2024-02-12T16:08:28.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:28.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:29.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:29.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:29.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:28 smithi123 bash[19119]: cluster 2024-02-12T16:08:27.613377+0000 mgr.y (mgr.14146) 688 : cluster [DBG] pgmap v835: 27 pgs: 1 active+recovering+degraded+remapped, 1 remapped+peering, 25 active+clean; 13 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:28 smithi123 bash[19119]: cluster 2024-02-12T16:08:27.752617+0000 mon.a (mon.0) 1078 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-02-12T16:08:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:28 smithi123 bash[19119]: cluster 2024-02-12T16:08:27.766596+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e241: 12 total, 12 up, 12 in 2024-02-12T16:08:29.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:29.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:29.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:29.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:29.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:29.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:29.452 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:29.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:29.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:29.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:29.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:29.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:08:29.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:29.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:29.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:29.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:29 smithi123 bash[19119]: audit 2024-02-12T16:08:29.614647+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:08:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:29 smithi121 bash[18489]: audit 2024-02-12T16:08:29.614647+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:08:29.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:29.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:30.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:30.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:29 smithi169 bash[19604]: audit 2024-02-12T16:08:29.614647+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:08:30.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:30.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:30.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:30.163 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:08:30.163 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:08:30.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:30.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:30.203 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:30.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:30.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:30.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:30.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:30.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:30.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:30.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:30.587 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:30.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:30.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:30.728 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:30.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:30.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-12T16:08:30.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:30.871 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:30.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:30.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:30.986 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:30 smithi123 bash[19119]: cluster 2024-02-12T16:08:29.613877+0000 mgr.y (mgr.14146) 689 : cluster [DBG] pgmap v837: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:30.986 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:30 smithi123 bash[19119]: audit 2024-02-12T16:08:29.761140+0000 mon.a (mon.0) 1081 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:08:30.986 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:30 smithi123 bash[19119]: cluster 2024-02-12T16:08:29.761239+0000 mon.a (mon.0) 1082 : cluster [DBG] osdmap e242: 12 total, 12 up, 12 in 2024-02-12T16:08:30.986 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:30 smithi123 bash[19119]: cluster 2024-02-12T16:08:30.283055+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e243: 12 total, 12 up, 12 in 2024-02-12T16:08:31.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:31.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:30 smithi169 bash[19604]: cluster 2024-02-12T16:08:29.613877+0000 mgr.y (mgr.14146) 689 : cluster [DBG] pgmap v837: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:30 smithi169 bash[19604]: audit 2024-02-12T16:08:29.761140+0000 mon.a (mon.0) 1081 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:08:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:30 smithi169 bash[19604]: cluster 2024-02-12T16:08:29.761239+0000 mon.a (mon.0) 1082 : cluster [DBG] osdmap e242: 12 total, 12 up, 12 in 2024-02-12T16:08:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:30 smithi169 bash[19604]: cluster 2024-02-12T16:08:30.283055+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e243: 12 total, 12 up, 12 in 2024-02-12T16:08:31.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:31.152 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:31.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:30 smithi121 bash[18489]: cluster 2024-02-12T16:08:29.613877+0000 mgr.y (mgr.14146) 689 : cluster [DBG] pgmap v837: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:31.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:30 smithi121 bash[18489]: audit 2024-02-12T16:08:29.761140+0000 mon.a (mon.0) 1081 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:08:31.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:30 smithi121 bash[18489]: cluster 2024-02-12T16:08:29.761239+0000 mon.a (mon.0) 1082 : cluster [DBG] osdmap e242: 12 total, 12 up, 12 in 2024-02-12T16:08:31.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:30 smithi121 bash[18489]: cluster 2024-02-12T16:08:30.283055+0000 mon.a (mon.0) 1083 : cluster [DBG] osdmap e243: 12 total, 12 up, 12 in 2024-02-12T16:08:31.255 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:31.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:31.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:31.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:31.436 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:31.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:31.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:31.574 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:31.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:31.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:31.718 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:31.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:31.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:31.858 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:31.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:31.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:31.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:32.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:32.135 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:32.232 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:32.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:32.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:32.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:32.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:32.409 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:32.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:32.537 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:32.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:32.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:32 smithi123 bash[19119]: cluster 2024-02-12T16:08:31.290905+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e244: 12 total, 12 up, 12 in 2024-02-12T16:08:32.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:32 smithi123 bash[19119]: cluster 2024-02-12T16:08:31.614502+0000 mgr.y (mgr.14146) 690 : cluster [DBG] pgmap v841: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:32.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:32 smithi123 bash[19119]: audit 2024-02-12T16:08:31.615425+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:08:32.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:32.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:32 smithi169 bash[19604]: cluster 2024-02-12T16:08:31.290905+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e244: 12 total, 12 up, 12 in 2024-02-12T16:08:32.646 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:32 smithi169 bash[19604]: cluster 2024-02-12T16:08:31.614502+0000 mgr.y (mgr.14146) 690 : cluster [DBG] pgmap v841: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:32.647 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:32 smithi169 bash[19604]: audit 2024-02-12T16:08:31.615425+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:08:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:32 smithi121 bash[18489]: cluster 2024-02-12T16:08:31.290905+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e244: 12 total, 12 up, 12 in 2024-02-12T16:08:32.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:32 smithi121 bash[18489]: cluster 2024-02-12T16:08:31.614502+0000 mgr.y (mgr.14146) 690 : cluster [DBG] pgmap v841: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:32.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:32 smithi121 bash[18489]: audit 2024-02-12T16:08:31.615425+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:08:32.668 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:32.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:32.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:32.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:32.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:32.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:33.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:33.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:33.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:33.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:33.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:33.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:33.292 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:33.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:33.327 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:33 smithi121 bash[18489]: audit 2024-02-12T16:08:32.291640+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:08:33.327 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:33 smithi121 bash[18489]: cluster 2024-02-12T16:08:32.291692+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e245: 12 total, 12 up, 12 in 2024-02-12T16:08:33.327 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:33 smithi121 bash[18489]: cluster 2024-02-12T16:08:32.295786+0000 osd.0 (osd.0) 31 : cluster [DBG] 2.15 starting backfill to osd.1 from (0'0,0'0] MAX to 236'15326 2024-02-12T16:08:33.327 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:33 smithi121 bash[18489]: cluster 2024-02-12T16:08:32.418026+0000 osd.0 (osd.0) 32 : cluster [DBG] 2.15 starting backfill to osd.9 from (0'0,0'0] MAX to 236'15326 2024-02-12T16:08:33.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:33.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:33.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:33.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:33.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:33.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:33.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:33.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:33 smithi169 bash[19604]: audit 2024-02-12T16:08:32.291640+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:08:33.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:33 smithi169 bash[19604]: cluster 2024-02-12T16:08:32.291692+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e245: 12 total, 12 up, 12 in 2024-02-12T16:08:33.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:33 smithi169 bash[19604]: cluster 2024-02-12T16:08:32.295786+0000 osd.0 (osd.0) 31 : cluster [DBG] 2.15 starting backfill to osd.1 from (0'0,0'0] MAX to 236'15326 2024-02-12T16:08:33.605 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:33 smithi169 bash[19604]: cluster 2024-02-12T16:08:32.418026+0000 osd.0 (osd.0) 32 : cluster [DBG] 2.15 starting backfill to osd.9 from (0'0,0'0] MAX to 236'15326 2024-02-12T16:08:33.625 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:33.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:33.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:33 smithi123 bash[19119]: audit 2024-02-12T16:08:32.291640+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:08:33.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:33 smithi123 bash[19119]: cluster 2024-02-12T16:08:32.291692+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e245: 12 total, 12 up, 12 in 2024-02-12T16:08:33.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:33 smithi123 bash[19119]: cluster 2024-02-12T16:08:32.295786+0000 osd.0 (osd.0) 31 : cluster [DBG] 2.15 starting backfill to osd.1 from (0'0,0'0] MAX to 236'15326 2024-02-12T16:08:33.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:33 smithi123 bash[19119]: cluster 2024-02-12T16:08:32.418026+0000 osd.0 (osd.0) 32 : cluster [DBG] 2.15 starting backfill to osd.9 from (0'0,0'0] MAX to 236'15326 2024-02-12T16:08:33.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:33.766 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:33.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:33.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:33.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:33.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:34.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:34.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:34.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:34.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:34.186 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:34.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:34.264 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:34.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:34.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:34.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:34.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:34.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:34 smithi169 bash[19604]: cluster 2024-02-12T16:08:33.306287+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e246: 12 total, 12 up, 12 in 2024-02-12T16:08:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:34 smithi169 bash[19604]: cluster 2024-02-12T16:08:33.615330+0000 mgr.y (mgr.14146) 691 : cluster [DBG] pgmap v844: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 15 objects/s recovering 2024-02-12T16:08:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:34 smithi169 bash[19604]: audit 2024-02-12T16:08:33.616049+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:08:34.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:34.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:34.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:34.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:34 smithi123 bash[19119]: cluster 2024-02-12T16:08:33.306287+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e246: 12 total, 12 up, 12 in 2024-02-12T16:08:34.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:34 smithi123 bash[19119]: cluster 2024-02-12T16:08:33.615330+0000 mgr.y (mgr.14146) 691 : cluster [DBG] pgmap v844: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 15 objects/s recovering 2024-02-12T16:08:34.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:34 smithi123 bash[19119]: audit 2024-02-12T16:08:33.616049+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:08:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:34 smithi121 bash[18489]: cluster 2024-02-12T16:08:33.306287+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e246: 12 total, 12 up, 12 in 2024-02-12T16:08:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:34 smithi121 bash[18489]: cluster 2024-02-12T16:08:33.615330+0000 mgr.y (mgr.14146) 691 : cluster [DBG] pgmap v844: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 15 objects/s recovering 2024-02-12T16:08:34.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:34 smithi121 bash[18489]: audit 2024-02-12T16:08:33.616049+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:08:34.704 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:34.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:34.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:34.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:34.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:34.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:34.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:35.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:35.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:35.149 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:35.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:35.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:35.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:35.418 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:35.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:35.535 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:35 smithi169 bash[19604]: audit 2024-02-12T16:08:34.314703+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:08:35.535 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:35 smithi169 bash[19604]: cluster 2024-02-12T16:08:34.314747+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e247: 12 total, 12 up, 12 in 2024-02-12T16:08:35.535 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:35 smithi169 bash[19604]: cluster 2024-02-12T16:08:34.320338+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.16 starting backfill to osd.5 from (0'0,0'0] MAX to 236'15424 2024-02-12T16:08:35.535 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:35 smithi169 bash[19604]: cluster 2024-02-12T16:08:34.340549+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.16 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15424 2024-02-12T16:08:35.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:35.607 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:08:35.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:35 smithi123 bash[19119]: audit 2024-02-12T16:08:34.314703+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:08:35.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:35 smithi123 bash[19119]: cluster 2024-02-12T16:08:34.314747+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e247: 12 total, 12 up, 12 in 2024-02-12T16:08:35.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:35 smithi123 bash[19119]: cluster 2024-02-12T16:08:34.320338+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.16 starting backfill to osd.5 from (0'0,0'0] MAX to 236'15424 2024-02-12T16:08:35.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:35 smithi123 bash[19119]: cluster 2024-02-12T16:08:34.340549+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.16 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15424 2024-02-12T16:08:35.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:35.686 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:35 smithi121 bash[18489]: audit 2024-02-12T16:08:34.314703+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:08:35.687 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:35 smithi121 bash[18489]: cluster 2024-02-12T16:08:34.314747+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e247: 12 total, 12 up, 12 in 2024-02-12T16:08:35.687 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:35 smithi121 bash[18489]: cluster 2024-02-12T16:08:34.320338+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.16 starting backfill to osd.5 from (0'0,0'0] MAX to 236'15424 2024-02-12T16:08:35.687 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:35 smithi121 bash[18489]: cluster 2024-02-12T16:08:34.340549+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.16 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15424 2024-02-12T16:08:35.699 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:35.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:35.841 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:35.945 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:35.984 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:35.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:35.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:36.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:36.124 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:36.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:36.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:36.265 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:36.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:36.339 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.11 to 1 (10000) 2024-02-12T16:08:36.368 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:36.404 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:36.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:36.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:36.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:36.609 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:36 smithi121 bash[18489]: cluster 2024-02-12T16:08:35.616075+0000 mgr.y (mgr.14146) 692 : cluster [DBG] pgmap v846: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 13 objects/s recovering 2024-02-12T16:08:36.609 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:36 smithi121 bash[18489]: audit 2024-02-12T16:08:35.617059+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:08:36.609 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:36 smithi121 bash[18489]: audit 2024-02-12T16:08:35.792168+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.121:0/1383048488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:08:36.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:36 smithi123 bash[19119]: cluster 2024-02-12T16:08:35.616075+0000 mgr.y (mgr.14146) 692 : cluster [DBG] pgmap v846: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 13 objects/s recovering 2024-02-12T16:08:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:36 smithi123 bash[19119]: audit 2024-02-12T16:08:35.617059+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:08:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:36 smithi123 bash[19119]: audit 2024-02-12T16:08:35.792168+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.121:0/1383048488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:08:36.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:36.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:36.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:36.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:36 smithi169 bash[19604]: cluster 2024-02-12T16:08:35.616075+0000 mgr.y (mgr.14146) 692 : cluster [DBG] pgmap v846: 27 pgs: 1 active+recovering+degraded+remapped, 26 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 13 objects/s recovering 2024-02-12T16:08:36.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:36 smithi169 bash[19604]: audit 2024-02-12T16:08:35.617059+0000 mon.a (mon.0) 1092 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:08:36.786 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:36 smithi169 bash[19604]: audit 2024-02-12T16:08:35.792168+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? v1:172.21.15.121:0/1383048488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:08:36.806 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:36.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:36.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:36.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:36.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:37.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:37.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:37.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:37.170 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:37.203 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:37.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:37.308 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:37.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:37.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:37.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:37.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:37.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:37.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:37 smithi121 bash[18489]: audit 2024-02-12T16:08:36.329706+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:08:37.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:37 smithi121 bash[18489]: audit 2024-02-12T16:08:36.329825+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? v1:172.21.15.121:0/1383048488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:08:37.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:37 smithi121 bash[18489]: cluster 2024-02-12T16:08:36.329920+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e248: 12 total, 12 up, 12 in 2024-02-12T16:08:37.606 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:37.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:37 smithi123 bash[19119]: audit 2024-02-12T16:08:36.329706+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:08:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:37 smithi123 bash[19119]: audit 2024-02-12T16:08:36.329825+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? v1:172.21.15.121:0/1383048488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:08:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:37 smithi123 bash[19119]: cluster 2024-02-12T16:08:36.329920+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e248: 12 total, 12 up, 12 in 2024-02-12T16:08:37.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:37.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:37.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:37.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:37 smithi169 bash[19604]: audit 2024-02-12T16:08:36.329706+0000 mon.a (mon.0) 1094 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:08:37.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:37 smithi169 bash[19604]: audit 2024-02-12T16:08:36.329825+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? v1:172.21.15.121:0/1383048488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:08:37.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:37 smithi169 bash[19604]: cluster 2024-02-12T16:08:36.329920+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e248: 12 total, 12 up, 12 in 2024-02-12T16:08:37.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:37.875 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:37.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:37.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:38.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:38.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:38.255 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:38.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:38.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:38.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:38.474 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T16:08:38.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:08:38.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:38.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:38.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:38 smithi123 bash[19119]: cluster 2024-02-12T16:08:37.616795+0000 mgr.y (mgr.14146) 693 : cluster [DBG] pgmap v848: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 11 objects/s recovering 2024-02-12T16:08:38.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:38 smithi123 bash[19119]: audit 2024-02-12T16:08:37.617438+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:08:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:38 smithi121 bash[18489]: cluster 2024-02-12T16:08:37.616795+0000 mgr.y (mgr.14146) 693 : cluster [DBG] pgmap v848: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 11 objects/s recovering 2024-02-12T16:08:38.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:38 smithi121 bash[18489]: audit 2024-02-12T16:08:37.617438+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:08:38.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:38.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:38.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:38 smithi169 bash[19604]: cluster 2024-02-12T16:08:37.616795+0000 mgr.y (mgr.14146) 693 : cluster [DBG] pgmap v848: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 11 objects/s recovering 2024-02-12T16:08:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:38 smithi169 bash[19604]: audit 2024-02-12T16:08:37.617438+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:08:38.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:38.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:38.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:38.977 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:38.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:39.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:39.110 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:39.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:39.214 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:39.251 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:39.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:39.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:39.392 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:39.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:39.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:39.503 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:39 smithi123 bash[19119]: cluster 2024-02-12T16:08:38.328518+0000 mon.a (mon.0) 1098 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:08:39.503 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:39 smithi123 bash[19119]: audit 2024-02-12T16:08:38.341198+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T16:08:39.503 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:39 smithi123 bash[19119]: cluster 2024-02-12T16:08:38.341390+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e249: 12 total, 12 up, 12 in 2024-02-12T16:08:39.524 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:39.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:39.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:39.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:39 smithi169 bash[19604]: cluster 2024-02-12T16:08:38.328518+0000 mon.a (mon.0) 1098 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:08:39.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:39 smithi169 bash[19604]: audit 2024-02-12T16:08:38.341198+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T16:08:39.637 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:39 smithi169 bash[19604]: cluster 2024-02-12T16:08:38.341390+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e249: 12 total, 12 up, 12 in 2024-02-12T16:08:39.657 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:39 smithi121 bash[18489]: cluster 2024-02-12T16:08:38.328518+0000 mon.a (mon.0) 1098 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:08:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:39 smithi121 bash[18489]: audit 2024-02-12T16:08:38.341198+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T16:08:39.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:39 smithi121 bash[18489]: cluster 2024-02-12T16:08:38.341390+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e249: 12 total, 12 up, 12 in 2024-02-12T16:08:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:39.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:39.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:39.798 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:39.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:39.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:39.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:39.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:40.029 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:08:40.036 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:40.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:08:40.073 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:08:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:40.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:40.099 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:08:40.121 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:08:40.124 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:08:40.124 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:08:40.145 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:08:40.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:40.180 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:08:40.206 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:40.206 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:08:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:40.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:40.228 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:08:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:08:40.270 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:08:40.296 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:08:40.298 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.298 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.298 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.298 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:08:40.308 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:40.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:40.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:40.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:40 smithi123 bash[19119]: cluster 2024-02-12T16:08:39.342264+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e250: 12 total, 12 up, 12 in 2024-02-12T16:08:40.359 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:40 smithi123 bash[19119]: cluster 2024-02-12T16:08:39.350530+0000 osd.9 (osd.9) 25 : cluster [DBG] 2.18 starting backfill to osd.4 from (0'0,0'0] MAX to 236'15566 2024-02-12T16:08:40.359 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:40 smithi123 bash[19119]: cluster 2024-02-12T16:08:39.617544+0000 mgr.y (mgr.14146) 694 : cluster [DBG] pgmap v851: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 11 objects/s recovering 2024-02-12T16:08:40.359 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:40 smithi123 bash[19119]: audit 2024-02-12T16:08:40.002946+0000 mon.a (mon.0) 1102 : audit [DBG] from='client.? v1:172.21.15.121:0/1234520032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:40.359 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:40 smithi123 bash[19119]: cluster 2024-02-12T16:08:40.282422+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e251: 12 total, 12 up, 12 in 2024-02-12T16:08:40.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:40.474 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:40.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:40 smithi169 bash[19604]: cluster 2024-02-12T16:08:39.342264+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e250: 12 total, 12 up, 12 in 2024-02-12T16:08:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:40 smithi169 bash[19604]: cluster 2024-02-12T16:08:39.350530+0000 osd.9 (osd.9) 25 : cluster [DBG] 2.18 starting backfill to osd.4 from (0'0,0'0] MAX to 236'15566 2024-02-12T16:08:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:40 smithi169 bash[19604]: cluster 2024-02-12T16:08:39.617544+0000 mgr.y (mgr.14146) 694 : cluster [DBG] pgmap v851: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 11 objects/s recovering 2024-02-12T16:08:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:40 smithi169 bash[19604]: audit 2024-02-12T16:08:40.002946+0000 mon.a (mon.0) 1102 : audit [DBG] from='client.? v1:172.21.15.121:0/1234520032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:40 smithi169 bash[19604]: cluster 2024-02-12T16:08:40.282422+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e251: 12 total, 12 up, 12 in 2024-02-12T16:08:40.574 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:40 smithi121 bash[18489]: cluster 2024-02-12T16:08:39.342264+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e250: 12 total, 12 up, 12 in 2024-02-12T16:08:40.575 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:40 smithi121 bash[18489]: cluster 2024-02-12T16:08:39.350530+0000 osd.9 (osd.9) 25 : cluster [DBG] 2.18 starting backfill to osd.4 from (0'0,0'0] MAX to 236'15566 2024-02-12T16:08:40.575 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:40 smithi121 bash[18489]: cluster 2024-02-12T16:08:39.617544+0000 mgr.y (mgr.14146) 694 : cluster [DBG] pgmap v851: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 11 objects/s recovering 2024-02-12T16:08:40.575 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:40 smithi121 bash[18489]: audit 2024-02-12T16:08:40.002946+0000 mon.a (mon.0) 1102 : audit [DBG] from='client.? v1:172.21.15.121:0/1234520032' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:40.575 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:40 smithi121 bash[18489]: cluster 2024-02-12T16:08:40.282422+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e251: 12 total, 12 up, 12 in 2024-02-12T16:08:40.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:40.606 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:40.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:40.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:40.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:40.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:40.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:40.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-12T16:08:40.869 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:40.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:40.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:41.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:41.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:41.144 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:41.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:41.282 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:41.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:41.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:41 smithi123 bash[19119]: cluster 2024-02-12T16:08:40.284920+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.19 starting backfill to osd.3 from (0'0,0'0] MAX to 236'15676 2024-02-12T16:08:41.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:41 smithi123 bash[19119]: cluster 2024-02-12T16:08:40.306178+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.19 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15676 2024-02-12T16:08:41.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:41 smithi123 bash[19119]: cluster 2024-02-12T16:08:41.292357+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e252: 12 total, 12 up, 12 in 2024-02-12T16:08:41.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:41 smithi123 bash[19119]: audit 2024-02-12T16:08:41.340560+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? v1:172.21.15.121:0/2424041979' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:08:41.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:41 smithi123 bash[19119]: audit 2024-02-12T16:08:41.344902+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? v1:172.21.15.121:0/3302505144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:08:41.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:41.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:41 smithi169 bash[19604]: cluster 2024-02-12T16:08:40.284920+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.19 starting backfill to osd.3 from (0'0,0'0] MAX to 236'15676 2024-02-12T16:08:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:41 smithi169 bash[19604]: cluster 2024-02-12T16:08:40.306178+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.19 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15676 2024-02-12T16:08:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:41 smithi169 bash[19604]: cluster 2024-02-12T16:08:41.292357+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e252: 12 total, 12 up, 12 in 2024-02-12T16:08:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:41 smithi169 bash[19604]: audit 2024-02-12T16:08:41.340560+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? v1:172.21.15.121:0/2424041979' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:08:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:41 smithi169 bash[19604]: audit 2024-02-12T16:08:41.344902+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? v1:172.21.15.121:0/3302505144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:08:41.562 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:41 smithi121 bash[18489]: cluster 2024-02-12T16:08:40.284920+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.19 starting backfill to osd.3 from (0'0,0'0] MAX to 236'15676 2024-02-12T16:08:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:41 smithi121 bash[18489]: cluster 2024-02-12T16:08:40.306178+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.19 starting backfill to osd.6 from (0'0,0'0] MAX to 236'15676 2024-02-12T16:08:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:41 smithi121 bash[18489]: cluster 2024-02-12T16:08:41.292357+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e252: 12 total, 12 up, 12 in 2024-02-12T16:08:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:41 smithi121 bash[18489]: audit 2024-02-12T16:08:41.340560+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? v1:172.21.15.121:0/2424041979' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:08:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:41 smithi121 bash[18489]: audit 2024-02-12T16:08:41.344902+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? v1:172.21.15.121:0/3302505144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:08:41.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:41.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:41.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:41.842 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:41.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:41.984 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:41.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:42.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:42.124 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:42.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:42.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:42.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:42.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:08:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_0' removed 2024-02-12T16:08:42.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:42.413 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:42.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:42.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:42.560 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:42.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:42 smithi123 bash[19119]: cluster 2024-02-12T16:08:41.618125+0000 mgr.y (mgr.14146) 695 : cluster [DBG] pgmap v854: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:42 smithi123 bash[19119]: cluster 2024-02-12T16:08:41.636186+0000 osd.2 (osd.2) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:42 smithi123 bash[19119]: cluster 2024-02-12T16:08:41.636260+0000 osd.2 (osd.2) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:42 smithi123 bash[19119]: cluster 2024-02-12T16:08:42.288779+0000 mon.a (mon.0) 1107 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:42 smithi123 bash[19119]: audit 2024-02-12T16:08:42.294907+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? v1:172.21.15.121:0/2424041979' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:08:42.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:42 smithi123 bash[19119]: audit 2024-02-12T16:08:42.295048+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? v1:172.21.15.121:0/3302505144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:08:42.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:42 smithi123 bash[19119]: cluster 2024-02-12T16:08:42.295144+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e253: 12 total, 12 up, 12 in 2024-02-12T16:08:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:42 smithi121 bash[18489]: cluster 2024-02-12T16:08:41.618125+0000 mgr.y (mgr.14146) 695 : cluster [DBG] pgmap v854: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:42 smithi121 bash[18489]: cluster 2024-02-12T16:08:41.636186+0000 osd.2 (osd.2) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:42 smithi121 bash[18489]: cluster 2024-02-12T16:08:41.636260+0000 osd.2 (osd.2) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:42 smithi121 bash[18489]: cluster 2024-02-12T16:08:42.288779+0000 mon.a (mon.0) 1107 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:42 smithi121 bash[18489]: audit 2024-02-12T16:08:42.294907+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? v1:172.21.15.121:0/2424041979' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:08:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:42 smithi121 bash[18489]: audit 2024-02-12T16:08:42.295048+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? v1:172.21.15.121:0/3302505144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:08:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:42 smithi121 bash[18489]: cluster 2024-02-12T16:08:42.295144+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e253: 12 total, 12 up, 12 in 2024-02-12T16:08:42.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:42.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:42.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:42.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:42 smithi169 bash[19604]: cluster 2024-02-12T16:08:41.618125+0000 mgr.y (mgr.14146) 695 : cluster [DBG] pgmap v854: 27 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 25 active+clean; 13 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:42 smithi169 bash[19604]: cluster 2024-02-12T16:08:41.636186+0000 osd.2 (osd.2) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:42 smithi169 bash[19604]: cluster 2024-02-12T16:08:41.636260+0000 osd.2 (osd.2) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:42 smithi169 bash[19604]: cluster 2024-02-12T16:08:42.288779+0000 mon.a (mon.0) 1107 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:42.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:42 smithi169 bash[19604]: audit 2024-02-12T16:08:42.294907+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? v1:172.21.15.121:0/2424041979' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:08:42.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:42 smithi169 bash[19604]: audit 2024-02-12T16:08:42.295048+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? v1:172.21.15.121:0/3302505144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:08:42.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:42 smithi169 bash[19604]: cluster 2024-02-12T16:08:42.295144+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e253: 12 total, 12 up, 12 in 2024-02-12T16:08:42.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:42.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:42.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:42.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:42.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:42.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:43.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:43.125 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:43.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:43.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:08:43.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:43.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:43.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:43.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:43.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:43.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:43.450 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:08:43.451 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:08:43.451 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-02-12T16:08:43.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-02-12T16:08:43.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:43.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:43.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:43.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:43.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:43.686 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:43.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:43.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:43.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:43.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:43 smithi121 bash[18489]: cluster 2024-02-12T16:08:43.669964+0000 mon.a (mon.0) 1111 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:08:43.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:43 smithi121 bash[18489]: cluster 2024-02-12T16:08:43.670054+0000 mon.a (mon.0) 1112 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-02-12T16:08:43.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:43.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:43.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:43 smithi169 bash[19604]: cluster 2024-02-12T16:08:43.669964+0000 mon.a (mon.0) 1111 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:08:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:43 smithi169 bash[19604]: cluster 2024-02-12T16:08:43.670054+0000 mon.a (mon.0) 1112 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-02-12T16:08:44.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:44.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:44.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:43 smithi123 bash[19119]: cluster 2024-02-12T16:08:43.669964+0000 mon.a (mon.0) 1111 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:08:44.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:43 smithi123 bash[19119]: cluster 2024-02-12T16:08:43.670054+0000 mon.a (mon.0) 1112 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-02-12T16:08:44.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:44.250 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:44.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:44.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:44.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:44.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:44.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:44.533 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:44.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:44.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:44.675 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:44.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:44.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:44.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:44 smithi169 bash[19604]: cluster 2024-02-12T16:08:43.618675+0000 mgr.y (mgr.14146) 696 : cluster [DBG] pgmap v856: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:44.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:44.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:44.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:44.958 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:44.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:45.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:45.074 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:44 smithi121 bash[18489]: cluster 2024-02-12T16:08:43.618675+0000 mgr.y (mgr.14146) 696 : cluster [DBG] pgmap v856: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:45.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:45.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:45.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:44 smithi123 bash[19119]: cluster 2024-02-12T16:08:43.618675+0000 mgr.y (mgr.14146) 696 : cluster [DBG] pgmap v856: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:45.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:45.240 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:45.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:45.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:45.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:45.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:45.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:45.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:08:45.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:45.519 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:45.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:45.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:45.624 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:08:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:45.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:45.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:45.796 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:45.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:45.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:45.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:45.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:45.976 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:46.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:46.080 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:46.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:46.184 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:46.222 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:46.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:46.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:46.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:46.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:46.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:46.501 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:46.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:46.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:46.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:46 smithi169 bash[19604]: cluster 2024-02-12T16:08:45.619177+0000 mgr.y (mgr.14146) 697 : cluster [DBG] pgmap v857: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:46 smithi123 bash[19119]: cluster 2024-02-12T16:08:45.619177+0000 mgr.y (mgr.14146) 697 : cluster [DBG] pgmap v857: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:46.646 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:46.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:46 smithi121 bash[18489]: cluster 2024-02-12T16:08:45.619177+0000 mgr.y (mgr.14146) 697 : cluster [DBG] pgmap v857: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:46.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:46.787 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:46.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:46.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:46.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:46.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:47.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:47.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:47.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:47.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:47.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:47.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:47.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:47.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:47.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:47.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:47.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:47.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:47.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:47.620 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:47.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:47.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:47.762 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:47.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:47.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:47.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:47.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:48.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:48.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:48.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:48.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:48.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:48.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:48.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:48.328 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:48.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:48.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:48.470 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:48.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:48.489 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:48.490 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:48.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:48.610 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:48.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:48.703 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:08:48.704 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:08:48.704 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:08:48.704 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:08:48.704 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:08:48.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:48.726 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:48 smithi123 bash[19119]: cluster 2024-02-12T16:08:47.619760+0000 mgr.y (mgr.14146) 698 : cluster [DBG] pgmap v858: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:48.751 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:48.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:48.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:48.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:49.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:48 smithi121 bash[18489]: cluster 2024-02-12T16:08:47.619760+0000 mgr.y (mgr.14146) 698 : cluster [DBG] pgmap v858: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:49.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:49.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:49.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:48 smithi169 bash[19604]: cluster 2024-02-12T16:08:47.619760+0000 mgr.y (mgr.14146) 698 : cluster [DBG] pgmap v858: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:49.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:49.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:49.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:49.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:49.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:49.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-12T16:08:49.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:49.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:49.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:49.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:49.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:49.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:49.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:49.892 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:49.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:49.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:50.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:50.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:08:50.064 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:08:50.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:50.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:50.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:50.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:50.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:50.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:50.454 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:50.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:50.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:50.596 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:50.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:50.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:50.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:50.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:50.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:50.871 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":859,"stamp":"2024-02-12T16:08:49.620040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":5,"kb":1124941824,"kb_used":4263376,"kb_used_data":598384,"kb_used_omap":0,"kb_used_meta":3664896,"kb_avail":1120678448,"statfs":{"total":1151940427776,"available":1147574730752,"internally_reserved":0,"allocated":612745216,"data_stored":606692520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3752853504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13664256,"num_objects":-210,"num_object_clones":0,"num_object_copies":-420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-210,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-324684,"num_write_kb":-10377985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11743,"num_bytes_recovered":-759496704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.953707"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":267,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-12T16:08:41.297996+0000","last_change":"2024-02-12T16:08:12.878558+0000","last_active":"2024-02-12T16:08:41.297996+0000","last_peered":"2024-02-12T16:08:41.297996+0000","last_clean":"2024-02-12T16:08:41.297996+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:08:41.297996+0000","last_undegraded":"2024-02-12T16:08:41.297996+0000","last_fullsized":"2024-02-12T16:08:41.297996+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:12.878487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:08:12.878487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958655,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302316,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442836,"statfs":{"total":95995035648,"available":95685464064,"internally_reserved":0,"allocated":1019904,"data_stored":515496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.74399999999999999,"15min":0.94099999999999995},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":1.4730000000000001,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.92000000000000004,"15min":1.05},"min":{"1min":0.32200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.8650000000000002,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":0.97099999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.0649999999999999,"15min":1.081},"min":{"1min":0.311,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.2629999999999999,"5min":5.6509999999999998,"15min":5.6509999999999998},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.23,"5min":1.0800000000000001,"15min":1.0840000000000001},"min":{"1min":0.28499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.873}]},{"osd":2,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.141,"15min":1.145},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.1369999999999996,"5min":5.274,"15min":5.274},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.90200000000000002,"15min":1.1120000000000001},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":2.1819999999999999,"5min":5.375,"15min":5.375},"last":1.036}]},{"osd":3,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.129,"15min":1.181},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.266,"5min":4.266,"15min":4.266},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.23,"5min":1.0840000000000001,"15min":1.157},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.315},"max":{"1min":6.5750000000000002,"5min":6.5750000000000002,"15min":6.5750000000000002},"last":1.0649999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.96499999999999997,"15min":1.22},"min":{"1min":0.40999999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.7549999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":1.1919999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.89500000000000002,"15min":1.244},"min":{"1min":0.41399999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89300000000000002,"5min":3.1299999999999999,"15min":4.0789999999999997},"last":1.1759999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.0549999999999999,"15min":1.3260000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2889999999999999,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.081,"15min":1.351},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.7189999999999999,"5min":4.8380000000000001,"15min":4.8380000000000001},"last":1.3129999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.0009999999999999,"15min":1.341},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":1.3200000000000001,"5min":3.2290000000000001,"15min":4.4989999999999997},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.751,"5min":1.079,"15min":1.425},"min":{"1min":0.47399999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.5700000000000001,"5min":3.2610000000000001,"15min":3.3340000000000001},"last":1.4059999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.083,"15min":1.4299999999999999},"min":{"1min":0.441,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.6519999999999999,"5min":3.4710000000000001,"15min":4.3129999999999997},"last":1.335},{"interface":"front","average":{"1min":0.81799999999999995,"5min":1.129,"15min":1.5089999999999999},"min":{"1min":0.49199999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.389,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.165}]},{"osd":8,"last update":"Mon Feb 12 16:07:46 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.1040000000000001,"15min":1.4550000000000001},"min":{"1min":0.44700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.431,"5min":3.4359999999999999,"15min":4},"last":1.498},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.1459999999999999,"15min":1.528},"min":{"1min":0.376,"5min":0.375,"15min":0.375},"max":{"1min":1.444,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":1.1060000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:07:51 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.125,"15min":1.484},"min":{"1min":0.499,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.6709999999999998,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":1.3819999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.1579999999999999,"15min":1.5940000000000001},"min":{"1min":0.50800000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.8839999999999999,"5min":4.6689999999999996,"15min":4.6689999999999996},"last":1.454}]}]},{"osd":10,"up_from":83,"seq":356482285705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563372,"kb_used_data":249764,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93181780,"statfs":{"total":95995035648,"available":95418142720,"internally_reserved":0,"allocated":255758336,"data_stored":255253928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.95699999999999996,"15min":0.91700000000000004},"min":{"1min":0.26100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.1890000000000001,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":1.29},{"interface":"front","average":{"1min":1.3999999999999999,"5min":0.96399999999999997,"15min":1.0249999999999999},"min":{"1min":0.32300000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.2590000000000003,"5min":7.6580000000000004,"15min":7.6580000000000004},"last":0.93700000000000006}]},{"osd":2,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.879,"15min":0.95099999999999996},"min":{"1min":0.29799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.4610000000000003,"5min":4.8300000000000001,"15min":4.8300000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.9550000000000001,"5min":1.052,"15min":1.0009999999999999},"min":{"1min":0.31900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.9829999999999997,"5min":7.9829999999999997,"15min":7.9829999999999997},"last":0.755}]},{"osd":3,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.048,"15min":1.0309999999999999},"min":{"1min":0.35099999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.343,"5min":4.8620000000000001,"15min":4.8620000000000001},"last":1.075},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.96399999999999997,"15min":0.995},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.4039999999999999,"5min":5.4100000000000001,"15min":5.4100000000000001},"last":1.2270000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.92100000000000004,"15min":1.075},"min":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3740000000000001,"5min":4.9189999999999996,"15min":4.9189999999999996},"last":1.26},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.88700000000000001,"15min":1.115},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.5919999999999996,"5min":4.5919999999999996,"15min":4.5919999999999996},"last":1.198}]},{"osd":5,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":7.2439999999999998,"15min":3.202},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.9079999999999999,"5min":602.05399999999997,"15min":602.05399999999997},"last":1.321},{"interface":"front","average":{"1min":1.3400000000000001,"5min":7.2949999999999999,"15min":3.3050000000000002},"min":{"1min":0.374,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.6769999999999996,"5min":598.875,"15min":598.875},"last":1.379}]},{"osd":6,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.98099999999999998,"15min":1.1399999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.9470000000000001,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":1.359},{"interface":"front","average":{"1min":1.181,"5min":1.046,"15min":1.266},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.6660000000000004,"5min":4.6660000000000004,"15min":4.6660000000000004},"last":1.3049999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":7.3460000000000001,"15min":3.3279999999999998},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":3.3319999999999999,"5min":597.48299999999995,"15min":597.48299999999995},"last":1.171},{"interface":"front","average":{"1min":0.92600000000000005,"5min":7.4219999999999997,"15min":3.3929999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.9930000000000003,"5min":600.23099999999999,"15min":600.23099999999999},"last":1.403}]},{"osd":8,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.1499999999999999,"15min":1.3},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":7.2720000000000002,"5min":7.2720000000000002,"15min":7.2720000000000002},"last":1.337},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.179,"15min":1.321},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":4.694,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":1.4590000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.1399999999999999,"15min":1.365},"min":{"1min":0.57799999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":4.7089999999999996,"5min":4.7089999999999996,"15min":4.7089999999999996},"last":1.532},{"interface":"front","average":{"1min":1.077,"5min":1.139,"15min":1.3500000000000001},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.4249999999999998,"5min":4.7789999999999999,"15min":4.7789999999999999},"last":1.4199999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.244,"15min":1.4850000000000001},"min":{"1min":0.52200000000000002,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":3.4830000000000001,"5min":5.2279999999999998,"15min":5.2279999999999998},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.2230000000000001,"15min":1.496},"min":{"1min":0.59999999999999998,"5min":0.5,"15min":0.5},"max":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":1.573}]}]},{"osd":9,"up_from":74,"seq":317827580049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":306412,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438740,"statfs":{"total":95995035648,"available":95681269760,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.82199999999999995,"15min":0.89700000000000002},"min":{"1min":0.24399999999999999,"5min":0.215,"15min":0.215},"max":{"1min":4.6879999999999997,"5min":10.788,"15min":10.788},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.8819999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.33500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.9630000000000001,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.91100000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":1.4590000000000001,"15min":1.1619999999999999},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":7.8479999999999999,"5min":11.991,"15min":11.991},"last":0.749},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.357,"15min":1.145},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":7.8360000000000003,"5min":11.977,"15min":11.977},"last":1.105}]},{"osd":2,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.234,"15min":1.0409999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.6230000000000002,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.802,"5min":1.2509999999999999,"15min":1.089},"min":{"1min":0.315,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.4989999999999997,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":0.94699999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1100000000000001,"15min":1.016},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":1.034},{"interface":"front","average":{"1min":2.2149999999999999,"5min":1.2470000000000001,"15min":1.1579999999999999},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.0650000000000004,"5min":5.0650000000000004,"15min":5.0650000000000004},"last":1.3169999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.304,"15min":1.2170000000000001},"min":{"1min":0.38600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.399,"5min":25.552,"15min":25.552},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":1.113,"15min":1.1599999999999999},"min":{"1min":0.35399999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.4540000000000002,"5min":25.847999999999999,"15min":25.847999999999999},"last":1.2070000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":7.9550000000000001,"15min":3.4689999999999999},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.3230000000000004,"5min":617.95899999999995,"15min":617.95899999999995},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.45,"5min":7.96,"15min":3.512},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":4.9960000000000004,"5min":617.88099999999997,"15min":617.88099999999997},"last":1.5149999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.105,"15min":1.25},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.0960000000000001,"5min":5.0960000000000001,"15min":5.0960000000000001},"last":1.3959999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.88300000000000001,"15min":1.1919999999999999},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.97099999999999997,"5min":3.4900000000000002,"15min":3.4900000000000002},"last":1.446}]},{"osd":7,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":11.598000000000001,"15min":4.7380000000000004},"min":{"1min":0.39000000000000001,"5min":0.379,"15min":0.379},"max":{"1min":4.3780000000000001,"5min":613.28200000000004,"15min":613.28200000000004},"last":1.171},{"interface":"front","average":{"1min":1.617,"5min":11.587,"15min":4.8070000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.7460000000000004,"5min":613.25199999999995,"15min":613.25199999999995},"last":1.4570000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.2310000000000001,"15min":1.3260000000000001},"min":{"1min":0.51300000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.96,"5min":4.96,"15min":4.96},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0129999999999999,"15min":1.2450000000000001},"min":{"1min":0.46500000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4399999999999999,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":1.294}]},{"osd":10,"last update":"Mon Feb 12 16:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":1.03,"15min":1.2969999999999999},"min":{"1min":0.46000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.4399999999999999,"5min":5.5,"15min":5.5},"last":1.349},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.234,"15min":1.3220000000000001},"min":{"1min":0.53000000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.2400000000000002,"5min":4.7619999999999996,"15min":4.7619999999999996},"last":1.3660000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874392,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":302316,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442836,"statfs":{"total":95995035648,"available":95685464064,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.309,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.0430000000000001,"5min":5.383,"15min":5.383},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1739999999999999,"5min":0.90800000000000003,"15min":0.91500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.5330000000000004,"5min":4.5330000000000004,"15min":4.5330000000000004},"last":0.86699999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.8939999999999999,"15min":2.0230000000000001},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":7.3789999999999996,"5min":23.661000000000001,"15min":23.661000000000001},"last":1.385},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.9970000000000001,"15min":2.161},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.4340000000000002,"5min":23.672000000000001,"15min":23.672000000000001},"last":1.4319999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":12.574,"5min":3.3340000000000001,"15min":1.696},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":193.929,"5min":193.929,"15min":193.929},"last":0.52300000000000002},{"interface":"front","average":{"1min":12.651,"5min":3.484,"15min":1.8380000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":193.82499999999999,"5min":193.82499999999999,"15min":193.82499999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0629999999999999,"15min":1.046},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.5659999999999998,"5min":5.4020000000000001,"15min":5.4020000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0620000000000001,"15min":1.0920000000000001},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.8879999999999999,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":1.129}]},{"osd":4,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.92300000000000004,"15min":1.1020000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":3.8420000000000001,"5min":6.4619999999999997,"15min":6.4619999999999997},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.091,"5min":1.087,"15min":1.1539999999999999},"min":{"1min":0.376,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.9279999999999999,"5min":5.71,"15min":5.71},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.052,"15min":1.1240000000000001},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.956,"5min":13.961,"15min":13.961},"last":1.006},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.052,"15min":1.1890000000000001},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.0760000000000001,"5min":13.103,"15min":13.103},"last":1.5369999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1,"15min":1.157},"min":{"1min":0.46400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.0289999999999999,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.2050000000000001,"15min":1.2709999999999999},"min":{"1min":0.40100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.4720000000000004,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":1.5660000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2070000000000001,"15min":1.2869999999999999},"min":{"1min":0.51600000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.8719999999999999,"5min":13.242000000000001,"15min":13.242000000000001},"last":1.649},{"interface":"front","average":{"1min":1.206,"5min":1.3260000000000001,"15min":1.355},"min":{"1min":0.52600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":3.9820000000000002,"5min":13.388,"15min":13.388},"last":1.337}]},{"osd":9,"last update":"Mon Feb 12 16:08:26 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1419999999999999,"15min":1.2689999999999999},"min":{"1min":0.40899999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.7909999999999999,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.702},{"interface":"front","average":{"1min":1.056,"5min":1.212,"15min":1.238},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.8119999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"last":1.754}]},{"osd":10,"last update":"Mon Feb 12 16:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.1879999999999999,"15min":1.3120000000000001},"min":{"1min":0.41199999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.0030000000000001,"5min":5.4189999999999996,"15min":5.4189999999999996},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.0489999999999999,"15min":1.331},"min":{"1min":0.48799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":3.1520000000000001,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":1.6699999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:08:15 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":15.471,"15min":29.544},"min":{"1min":0.47899999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":4.2539999999999996,"5min":635.66399999999999,"15min":635.66399999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":15.201000000000001,"15min":29.236000000000001},"min":{"1min":0.51600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":5.0800000000000001,"5min":636.28099999999995,"15min":636.28099999999995},"last":1.613}]}]},{"osd":7,"up_from":56,"seq":240518168737,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310508,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434644,"statfs":{"total":95995035648,"available":95677075456,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":4.101,"5min":1.508,"15min":1.173},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":60.152000000000001,"5min":60.152000000000001,"15min":60.152000000000001},"last":1.03},{"interface":"front","average":{"1min":4.4569999999999999,"5min":1.7410000000000001,"15min":1.222},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":62.020000000000003,"5min":62.020000000000003,"15min":62.020000000000003},"last":1.143}]},{"osd":2,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":1.7829999999999999,"15min":1.266},"min":{"1min":0.36199999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":60.204999999999998,"5min":60.204999999999998,"15min":60.204999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":4.4580000000000002,"5min":1.651,"15min":1.2490000000000001},"min":{"1min":0.33800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":59.365000000000002,"5min":59.365000000000002,"15min":59.365000000000002},"last":0.89800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.1140000000000001,"15min":1.093},"min":{"1min":0.28000000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.4020000000000001,"5min":4.8200000000000003,"15min":4.8200000000000003},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.097,"15min":1.083},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.1600000000000001,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.79800000000000004,"15min":0.92600000000000005},"min":{"1min":0.371,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.8220000000000001,"5min":6.2519999999999998,"15min":6.2519999999999998},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84499999999999997,"15min":0.97999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.4260000000000002,"5min":4.7430000000000003,"15min":4.7430000000000003},"last":1.3009999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.91000000000000003,"15min":1.028},"min":{"1min":0.33200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.8969999999999998,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.89700000000000002,"15min":1.0780000000000001},"min":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":3.907,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":1.0700000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0489999999999999,"15min":1.1459999999999999},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.9809999999999999,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":1.343},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.95199999999999996,"15min":1.1240000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.5119999999999996,"5min":4.7149999999999999,"15min":4.7149999999999999},"last":1.4870000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.1699999999999999,"15min":1.3360000000000001},"min":{"1min":0.5,"5min":0.432,"15min":0.432},"max":{"1min":4.7779999999999996,"5min":4.7779999999999996,"15min":4.7779999999999996},"last":1.679},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.2649999999999999,"15min":1.3819999999999999},"min":{"1min":0.54500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7919999999999998,"5min":4.851,"15min":4.851},"last":1.6279999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:08:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":4.3230000000000004,"15min":2.4009999999999998},"min":{"1min":0.53800000000000003,"5min":0.441,"15min":0.441},"max":{"1min":4.5960000000000001,"5min":278.54599999999999,"15min":278.54599999999999},"last":1.528},{"interface":"front","average":{"1min":1.3420000000000001,"5min":4.2670000000000003,"15min":2.3700000000000001},"min":{"1min":0.57099999999999995,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.4870000000000001,"5min":281.09199999999998,"15min":281.09199999999998},"last":1.728}]},{"osd":10,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.155,"15min":1.403},"min":{"1min":0.54900000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":3.8919999999999999,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.46},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.208,"15min":1.4330000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":4.4560000000000004,"5min":4.9450000000000003,"15min":4.9450000000000003},"last":1.607}]},{"osd":11,"last update":"Mon Feb 12 16:08:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":4.1340000000000003,"15min":2.4260000000000002},"min":{"1min":0.57599999999999996,"5min":0.498,"15min":0.498},"max":{"1min":3.593,"5min":280.83999999999997,"15min":280.83999999999997},"last":1.706},{"interface":"front","average":{"1min":1.528,"5min":4.048,"15min":2.367},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.8339999999999996,"5min":278.63799999999998,"15min":278.63799999999998},"last":1.7689999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":314604,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93430548,"statfs":{"total":95995035648,"available":95672881152,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.7959999999999998,"5min":1.429,"15min":1.121},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":67.093000000000004,"5min":67.093000000000004,"15min":67.093000000000004},"last":1.3939999999999999},{"interface":"front","average":{"1min":3.8149999999999999,"5min":1.462,"15min":1.1240000000000001},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":68.549000000000007,"5min":68.549000000000007,"15min":68.549000000000007},"last":1.0589999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9489999999999998,"5min":1.6679999999999999,"15min":1.1879999999999999},"min":{"1min":0.317,"5min":0.308,"15min":0.308},"max":{"1min":66.936000000000007,"5min":66.936000000000007,"15min":66.936000000000007},"last":1.157},{"interface":"front","average":{"1min":4.3890000000000002,"5min":1.6359999999999999,"15min":1.222},"min":{"1min":0.34499999999999997,"5min":0.309,"15min":0.309},"max":{"1min":68.683000000000007,"5min":68.683000000000007,"15min":68.683000000000007},"last":1.139}]},{"osd":2,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":4.1319999999999997,"5min":2.452,"15min":1.488},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":68.462000000000003,"5min":68.462000000000003,"15min":68.462000000000003},"last":1.254},{"interface":"front","average":{"1min":3.9399999999999999,"5min":2.456,"15min":1.5189999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":67.322999999999993,"5min":67.322999999999993,"15min":67.322999999999993},"last":1.28}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.512,"15min":1.1659999999999999},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":68.332999999999998,"5min":68.332999999999998,"15min":68.332999999999998},"last":1.0169999999999999},{"interface":"front","average":{"1min":4.1600000000000001,"5min":1.55,"15min":1.228},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":67.281999999999996,"5min":67.281999999999996,"15min":67.281999999999996},"last":1.3500000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.7919999999999998,"5min":1.474,"15min":1.141},"min":{"1min":0.38300000000000001,"5min":0.316,"15min":0.316},"max":{"1min":68.605999999999995,"5min":68.605999999999995,"15min":68.605999999999995},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.77600000000000002,"15min":0.94899999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.3999999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.369}]},{"osd":5,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9750000000000001,"5min":1.593,"15min":1.2589999999999999},"min":{"1min":0.36099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":69.260000000000005,"5min":69.260000000000005,"15min":69.260000000000005},"last":1.52},{"interface":"front","average":{"1min":3.839,"5min":1.4259999999999999,"15min":1.22},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":68.742000000000004,"5min":68.742000000000004,"15min":68.742000000000004},"last":1.3140000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":9.2140000000000004,"5min":2.7429999999999999,"15min":1.732},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":67.144999999999996,"5min":67.144999999999996,"15min":67.144999999999996},"last":1.4330000000000001},{"interface":"front","average":{"1min":8.9420000000000002,"5min":2.6629999999999998,"15min":1.7490000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":67.230999999999995,"5min":67.230999999999995,"15min":67.230999999999995},"last":1.508}]},{"osd":8,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":4.7350000000000003,"5min":1.8779999999999999,"15min":1.5880000000000001},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":68.653000000000006,"5min":68.653000000000006,"15min":68.653000000000006},"last":1.649},{"interface":"front","average":{"1min":4.8259999999999996,"5min":1.752,"15min":1.5589999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":68.620999999999995,"5min":68.620999999999995,"15min":68.620999999999995},"last":1.635}]},{"osd":9,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":4.2510000000000003,"5min":1.8140000000000001,"15min":1.617},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":67.188000000000002,"5min":67.188000000000002,"15min":67.188000000000002},"last":1.5649999999999999},{"interface":"front","average":{"1min":4.2519999999999998,"5min":1.79,"15min":1.5620000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":68.506,"5min":68.506,"15min":68.506},"last":1.704}]},{"osd":10,"last update":"Mon Feb 12 16:08:09 2024","interfaces":[{"interface":"back","average":{"1min":4.2149999999999999,"5min":1.8759999999999999,"15min":1.7150000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":68.722999999999999,"5min":68.722999999999999,"15min":68.722999999999999},"last":1.75},{"interface":"front","average":{"1min":4.2560000000000002,"5min":1.72,"15min":1.623},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":68.573999999999998,"5min":68.573999999999998,"15min":68.573999999999998},"last":1.8320000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575056,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":298220,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.2390000000000001,"15min":0.93200000000000005},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.0030000000000001,"5min":9.8740000000000006,"15min":9.8740000000000006},"last":1.046},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.244,"15min":0.98699999999999999},"min":{"1min":0.27700000000000002,"5min":0.255,"15min":0.255},"max":{"1min":5.9130000000000003,"5min":10.513,"15min":10.513},"last":0.98699999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:07:51 2024","interfaces":[{"interface":"back","average":{"1min":3.1960000000000002,"5min":1.881,"15min":1.1899999999999999},"min":{"1min":0.28000000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":40.542999999999999,"5min":40.542999999999999,"15min":40.542999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":3.5830000000000002,"5min":1.7849999999999999,"15min":1.1759999999999999},"min":{"1min":0.26900000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":40.811999999999998,"5min":40.811999999999998,"15min":40.811999999999998},"last":0.85399999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.169,"15min":1.004},"min":{"1min":0.28499999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.6900000000000004,"5min":6.0170000000000003,"15min":6.0170000000000003},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.962,"5min":1.405,"15min":1.1259999999999999},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":10.225,"5min":10.648999999999999,"15min":10.648999999999999},"last":0.89100000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.4199999999999999,"15min":1.202},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.1050000000000004,"5min":9.6780000000000008,"15min":9.6780000000000008},"last":2.093},{"interface":"front","average":{"1min":1.212,"5min":1.381,"15min":1.2410000000000001},"min":{"1min":0.40300000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.1870000000000003,"5min":10.244999999999999,"15min":10.244999999999999},"last":2.0089999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:07:51 2024","interfaces":[{"interface":"back","average":{"1min":1.843,"5min":1.4770000000000001,"15min":1.2929999999999999},"min":{"1min":0.46100000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":8.3610000000000007,"5min":9.8019999999999996,"15min":9.8019999999999996},"last":1.8979999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.206,"15min":1.2689999999999999},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.637,"5min":10.420999999999999,"15min":10.420999999999999},"last":1.5600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.234,"15min":1.26},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.1779999999999999,"5min":10.455,"15min":10.455},"last":1.946},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.242,"15min":1.3380000000000001},"min":{"1min":0.58599999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.21,"5min":10.488,"15min":10.488},"last":2.04}]},{"osd":7,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3580000000000001,"15min":1.3999999999999999},"min":{"1min":0.50600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":3.7869999999999999,"5min":6.2869999999999999,"15min":6.2869999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.2529999999999999,"15min":1.3200000000000001},"min":{"1min":0.503,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.7389999999999999,"5min":5.1429999999999998,"15min":5.1429999999999998},"last":1.6000000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:07:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.6559999999999999,"15min":1.502},"min":{"1min":0.42499999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.3300000000000001,"5min":9.9049999999999994,"15min":9.9049999999999994},"last":1.6659999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.667,"15min":1.583},"min":{"1min":0.48599999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":8.0180000000000007,"5min":10.313000000000001,"15min":10.313000000000001},"last":1.46}]},{"osd":9,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.522,"15min":1.6080000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":7.9409999999999998,"5min":10.352,"15min":10.352},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.2949999999999999,"15min":1.4490000000000001},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.7839999999999998,"5min":4.7839999999999998,"15min":4.7839999999999998},"last":1.3759999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.6060000000000001,"15min":1.579},"min":{"1min":0.55500000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":5.181,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":1.7949999999999999},{"interface":"front","average":{"1min":1.948,"5min":1.6040000000000001,"15min":1.6970000000000001},"min":{"1min":0.622,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":7.984,"5min":10.387,"15min":10.387},"last":1.7609999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":302316,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442836,"statfs":{"total":95995035648,"available":95685464064,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":8.5429999999999993,"5min":2.464,"15min":1.3320000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":114.01600000000001,"5min":114.01600000000001,"15min":114.01600000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":7.6890000000000001,"5min":2.6160000000000001,"15min":1.405},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":114.342,"5min":114.342,"15min":114.342},"last":0.77700000000000002}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.048,"15min":0.89900000000000002},"min":{"1min":0.27600000000000002,"5min":0.215,"15min":0.215},"max":{"1min":5.633,"5min":5.633,"15min":5.633},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.0409999999999999,"15min":0.95799999999999996},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.5140000000000002,"5min":7.5140000000000002,"15min":7.5140000000000002},"last":0.752}]},{"osd":4,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.86499999999999999,"15min":1.0349999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.804,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.052},{"interface":"front","average":{"1min":0.879,"5min":0.97699999999999998,"15min":1.008},"min":{"1min":0.29699999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.585,"5min":4.5860000000000003,"15min":4.5860000000000003},"last":1.397}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.93300000000000005,"15min":1.034},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"last":1.371},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.86699999999999999,"15min":1.0960000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.75,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.96699999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.93600000000000005,"15min":1.089},"min":{"1min":0.376,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6699999999999999,"5min":5.4699999999999998,"15min":5.4699999999999998},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.97299999999999998,"15min":1.153},"min":{"1min":0.502,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.6579999999999999,"5min":5.5389999999999997,"15min":5.5389999999999997},"last":1.0680000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.0169999999999999,"15min":1.2470000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.123,"5min":5.5060000000000002,"15min":5.5060000000000002},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.1200000000000001,"15min":1.29},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2070000000000001,"5min":5.5720000000000001,"15min":5.5720000000000001},"last":1.175}]},{"osd":8,"last update":"Mon Feb 12 16:08:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.1850000000000001,"15min":1.3560000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.7140000000000004,"5min":5.6059999999999999,"15min":5.6059999999999999},"last":1.476},{"interface":"front","average":{"1min":1.395,"5min":1.3180000000000001,"15min":1.419},"min":{"1min":0.55100000000000005,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":7.5599999999999996,"5min":7.5599999999999996,"15min":7.5599999999999996},"last":1.288}]},{"osd":9,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.3049999999999999,"15min":1.4350000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.75,"5min":5.7969999999999997,"15min":5.7969999999999997},"last":1.242},{"interface":"front","average":{"1min":1.117,"5min":1.206,"15min":1.4319999999999999},"min":{"1min":0.59699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2229999999999999,"5min":5.6890000000000001,"15min":5.6890000000000001},"last":1.1220000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.23,"15min":1.466},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.909,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.3069999999999999,"15min":1.375},"min":{"1min":0.44500000000000001,"5min":0.437,"15min":0.437},"max":{"1min":2.657,"5min":5.8010000000000002,"15min":5.8010000000000002},"last":1.296}]},{"osd":11,"last update":"Mon Feb 12 16:08:40 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.2609999999999999,"15min":1.462},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.9359999999999999,"5min":7.9660000000000002,"15min":7.9660000000000002},"last":1.222},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.5149999999999999,"15min":1.4350000000000001},"min":{"1min":0.48199999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.1920000000000002,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":1.5009999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313417,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":314604,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93430548,"statfs":{"total":95995035648,"available":95672881152,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.208,"15min":0.92300000000000004},"min":{"1min":0.247,"5min":0.222,"15min":0.222},"max":{"1min":6.2359999999999998,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.931,"5min":1.2290000000000001,"15min":0.95199999999999996},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":6.2779999999999996,"5min":6.2779999999999996,"15min":6.2779999999999996},"last":0.749}]},{"osd":1,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.3819999999999999,"15min":1.0409999999999999},"min":{"1min":0.29399999999999998,"5min":0.223,"15min":0.223},"max":{"1min":6.5629999999999997,"5min":11.423,"15min":11.423},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.8100000000000001,"5min":1.3919999999999999,"15min":1.125},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.1749999999999998,"5min":7.2290000000000001,"15min":7.2290000000000001},"last":0.86299999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:08:24 2024","interfaces":[{"interface":"back","average":{"1min":1.956,"5min":1.542,"15min":1.163},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":9.4909999999999997,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.1990000000000001,"15min":1.0389999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.9910000000000001,"5min":9.4399999999999995,"15min":9.4399999999999995},"last":0.82399999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3680000000000001,"15min":1.204},"min":{"1min":0.39900000000000002,"5min":0.311,"15min":0.311},"max":{"1min":6.2469999999999999,"5min":9.657,"15min":9.657},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.157,"5min":1.167,"15min":1.0129999999999999},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.4569999999999999,"5min":9.5530000000000008,"15min":9.5530000000000008},"last":1.083}]},{"osd":5,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":7.6539999999999999,"15min":3.3410000000000002},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.1059999999999999,"5min":616.34199999999998,"15min":616.34199999999998},"last":0.999},{"interface":"front","average":{"1min":1.71,"5min":7.875,"15min":3.3479999999999999},"min":{"1min":0.42299999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.202,"5min":615.18899999999996,"15min":615.18899999999996},"last":1.383}]},{"osd":6,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.212,"15min":1.145},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.0219999999999998,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.905,"5min":1.2989999999999999,"15min":1.2629999999999999},"min":{"1min":0.433,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":6.3979999999999997,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":1.2230000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":7.9809999999999999,"15min":3.5630000000000002},"min":{"1min":0.41899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":3.0510000000000002,"5min":611.79600000000005,"15min":611.79600000000005},"last":1.363},{"interface":"front","average":{"1min":2.0840000000000001,"5min":8.0980000000000008,"15min":3.5499999999999998},"min":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.1520000000000001,"5min":611.83900000000006,"15min":611.83900000000006},"last":1.4730000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:38 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":1.5269999999999999,"15min":1.48},"min":{"1min":0.46700000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":6.4960000000000004,"5min":6.4960000000000004,"15min":6.4960000000000004},"last":1.419},{"interface":"front","average":{"1min":2.383,"5min":1.75,"15min":1.5760000000000001},"min":{"1min":0.52000000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":8.5920000000000005,"5min":8.5920000000000005,"15min":8.5920000000000005},"last":1.2829999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.367,"15min":1.401},"min":{"1min":0.47899999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":3.0790000000000002,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":1.2629999999999999},{"interface":"front","average":{"1min":2.2549999999999999,"5min":1.5369999999999999,"15min":1.504},"min":{"1min":0.52800000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":6.4699999999999998,"5min":6.4699999999999998,"15min":6.4699999999999998},"last":1.518}]},{"osd":10,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.51900000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":4.8819999999999997,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.452},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.5209999999999999,"15min":1.5369999999999999},"min":{"1min":0.51100000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":6.4359999999999999,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":1.329}]}]},{"osd":3,"up_from":26,"seq":111669149889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":558252,"kb_used_data":248740,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93186900,"statfs":{"total":95995035648,"available":95423385600,"internally_reserved":0,"allocated":254709760,"data_stored":254205372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":2.0659999999999998,"15min":1.2549999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":8.3680000000000003,"5min":70.031999999999996,"15min":70.031999999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.621,"5min":2.25,"15min":1.339},"min":{"1min":0.317,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.3399999999999999,"5min":69.944999999999993,"15min":69.944999999999993},"last":1.042}]},{"osd":2,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":1.8720000000000001,"15min":1.2290000000000001},"min":{"1min":0.28399999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.3129999999999999,"5min":68.405000000000001,"15min":68.405000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.752,"15min":1.238},"min":{"1min":0.31900000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.101,"5min":69.009,"15min":69.009},"last":1.119}]},{"osd":4,"last update":"Mon Feb 12 16:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.96599999999999997,"15min":0.98099999999999998},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.0600000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.86899999999999999,"15min":1.0149999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.8940000000000001,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":0.98799999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.92900000000000005,"15min":1.109},"min":{"1min":0.35799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5840000000000001,"5min":4.1840000000000002,"15min":4.1840000000000002},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.0389999999999999,"15min":1.0820000000000001},"min":{"1min":0.34699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.1179999999999999,"5min":5.4370000000000003,"15min":5.4370000000000003},"last":0.92600000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.04,"15min":1.137},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.8899999999999997,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.0609999999999999,"15min":1.2110000000000001},"min":{"1min":0.39200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.171,"5min":4.1429999999999998,"15min":4.1429999999999998},"last":1.3089999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.9950000000000001,"15min":1.544},"min":{"1min":0.433,"5min":0.432,"15min":0.432},"max":{"1min":3.024,"5min":91.718999999999994,"15min":91.718999999999994},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":2.0179999999999998,"15min":1.5800000000000001},"min":{"1min":0.52100000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.9249999999999998,"5min":90.367999999999995,"15min":90.367999999999995},"last":1.244}]},{"osd":8,"last update":"Mon Feb 12 16:08:38 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.333,"15min":1.347},"min":{"1min":0.44700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.9489999999999998,"5min":6.5899999999999999,"15min":6.5899999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2010000000000001,"15min":1.391},"min":{"1min":0.505,"5min":0.442,"15min":0.442},"max":{"1min":8.516,"5min":8.516,"15min":8.516},"last":1.2589999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.2210000000000001,"15min":1.4419999999999999},"min":{"1min":0.501,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.9740000000000002,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":1.198},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.333,"15min":1.4710000000000001},"min":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2529999999999999,"5min":6.5350000000000001,"15min":6.5350000000000001},"last":1.4419999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:07:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4690000000000001,"15min":1.47},"min":{"1min":0.50700000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":5.5069999999999997,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":1.393},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.3759999999999999,"15min":1.51},"min":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.3419999999999996,"5min":5.3419999999999996,"15min":5.3419999999999996},"last":1.4690000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.8200000000000001,"15min":1.6040000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.468,"5min":49.167000000000002,"15min":49.167000000000002},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.75,"15min":1.619},"min":{"1min":0.55800000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":5.0289999999999999,"5min":48.548999999999999,"15min":48.548999999999999},"last":1.347}]}]},{"osd":4,"up_from":32,"seq":137438953658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":298220,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":1019904,"data_stored":515516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.78000000000000003,"15min":0.92000000000000004},"min":{"1min":0.29499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.4029999999999996,"5min":4.4029999999999996,"15min":4.4029999999999996},"last":0.875},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0109999999999999,"15min":0.99199999999999999},"min":{"1min":0.30399999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3869999999999996,"5min":10.754,"15min":10.754},"last":0.91500000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.643,"15min":1.214},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4690000000000003,"5min":42.226999999999997,"15min":42.226999999999997},"last":1.044},{"interface":"front","average":{"1min":1.345,"5min":1.3240000000000001,"15min":1.1220000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.4459999999999997,"5min":43.689,"15min":43.689},"last":1.2330000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.4430000000000001,"15min":1.1319999999999999},"min":{"1min":0.34200000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.5490000000000004,"5min":41.546999999999997,"15min":41.546999999999997},"last":1.254},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.55,"15min":1.173},"min":{"1min":0.46200000000000002,"5min":0.31,"15min":0.31},"max":{"1min":3.843,"5min":41.57,"15min":41.57},"last":1.161}]},{"osd":3,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.83499999999999996,"15min":0.97099999999999997},"min":{"1min":0.375,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.234,"5min":0.86899999999999999,"15min":0.97799999999999998},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.7029999999999998,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":1.3080000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.79800000000000004,"15min":0.94699999999999995},"min":{"1min":0.318,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.25,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":1.1020000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90300000000000002,"15min":1.026},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3759999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":1.0549999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.81899999999999995,"15min":0.99299999999999999},"min":{"1min":0.41899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.004,"5min":3.004,"15min":3.004},"last":1.609},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71999999999999997,"15min":1.02},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.1799999999999999,"5min":1.5249999999999999,"15min":1.9330000000000001},"last":1.341}]},{"osd":7,"last update":"Mon Feb 12 16:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":2.1859999999999999,"15min":1.492},"min":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.9169999999999998,"5min":116.273,"15min":116.273},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.999,"5min":2.3130000000000002,"15min":1.6779999999999999},"min":{"1min":0.53800000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":3.0739999999999998,"5min":116.24299999999999,"15min":116.24299999999999},"last":1.4279999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.1599999999999999,"15min":1.399},"min":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.9180000000000001,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":1.536},{"interface":"front","average":{"1min":1.129,"5min":1.0760000000000001,"15min":1.3140000000000001},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.742,"5min":4.8609999999999998,"15min":4.8609999999999998},"last":1.585}]},{"osd":9,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":4.2779999999999996,"15min":2.4780000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":4.9050000000000002,"5min":297.55900000000003,"15min":297.55900000000003},"last":1.365},{"interface":"front","average":{"1min":1.2170000000000001,"5min":4.2569999999999997,"15min":2.4180000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":3.867,"5min":297.61900000000003,"15min":297.61900000000003},"last":1.6899999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0620000000000001,"15min":1.3720000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.944,"5min":4.944,"15min":4.944},"last":1.5009999999999999},{"interface":"front","average":{"1min":0.997,"5min":1.0429999999999999,"15min":1.3839999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":4.0289999999999999,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":1.4830000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392236,"kb_used_data":90916,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93352916,"statfs":{"total":95995035648,"available":95593385984,"internally_reserved":0,"allocated":93097984,"data_stored":92593596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:07:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":0.98799999999999999,"15min":0.96599999999999997},"min":{"1min":0.30499999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":13.031000000000001,"5min":13.031000000000001,"15min":13.031000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.5740000000000001,"5min":0.96999999999999997,"15min":0.96299999999999997},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":14.018000000000001,"5min":14.018000000000001,"15min":14.018000000000001},"last":0.76900000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:07:45 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.5129999999999999,"15min":1.171},"min":{"1min":0.27700000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":13.095000000000001,"5min":33.084000000000003,"15min":33.084000000000003},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.204,"15min":1.0389999999999999},"min":{"1min":0.31900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":3.956,"5min":33.052999999999997,"15min":33.052999999999997},"last":1.546}]},{"osd":2,"last update":"Mon Feb 12 16:07:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.3600000000000001,"15min":1.139},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":13.117000000000001,"5min":26.797000000000001,"15min":26.797000000000001},"last":1.452},{"interface":"front","average":{"1min":1.359,"5min":1.3260000000000001,"15min":1.1459999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.4210000000000003,"5min":26.905999999999999,"15min":26.905999999999999},"last":0.81499999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:07:45 2024","interfaces":[{"interface":"back","average":{"1min":2.423,"5min":1.3300000000000001,"15min":1.1359999999999999},"min":{"1min":0.36499999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":12.991,"5min":12.991,"15min":12.991},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.677,"5min":1.1990000000000001,"15min":1.145},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":13.935,"5min":13.935,"15min":13.935},"last":1.6830000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:07:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.97899999999999998,"15min":0.95099999999999996},"min":{"1min":0.32500000000000001,"5min":0.308,"15min":0.308},"max":{"1min":14.259,"5min":14.259,"15min":14.259},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.625,"5min":0.95199999999999996,"15min":1.0309999999999999},"min":{"1min":0.33600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":13.268000000000001,"5min":13.268000000000001,"15min":13.268000000000001},"last":1.238}]},{"osd":6,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":2.4060000000000001,"5min":1.145,"15min":1.1439999999999999},"min":{"1min":0.37,"5min":0.318,"15min":0.318},"max":{"1min":13.148,"5min":13.148,"15min":13.148},"last":1.0620000000000001},{"interface":"front","average":{"1min":2.2480000000000002,"5min":1.141,"15min":1.175},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":1.77}]},{"osd":7,"last update":"Mon Feb 12 16:07:59 2024","interfaces":[{"interface":"back","average":{"1min":7.2270000000000003,"5min":2.2250000000000001,"15min":1.5429999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":98.971000000000004,"5min":98.971000000000004,"15min":98.971000000000004},"last":1.1080000000000001},{"interface":"front","average":{"1min":6.9850000000000003,"5min":2.2909999999999999,"15min":1.6319999999999999},"min":{"1min":0.53400000000000003,"5min":0.45200000000000001,"15min":0.435},"max":{"1min":99.218000000000004,"5min":99.218000000000004,"15min":99.218000000000004},"last":1.8400000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:07:45 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.3009999999999999,"15min":1.4019999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":14.372999999999999,"5min":14.372999999999999,"15min":14.372999999999999},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.274,"15min":1.3340000000000001},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":14.074999999999999,"5min":14.074999999999999,"15min":14.074999999999999},"last":1.921}]},{"osd":9,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1970000000000001,"5min":1.2749999999999999,"15min":1.4350000000000001},"min":{"1min":0.499,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":13.375999999999999,"5min":13.375999999999999,"15min":13.375999999999999},"last":1.629},{"interface":"front","average":{"1min":2.097,"5min":1.268,"15min":1.4930000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":14.132,"5min":14.132,"15min":14.132},"last":2.0059999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.012,"5min":1.333,"15min":1.4359999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":14.214,"5min":14.214,"15min":14.214},"last":1.6639999999999999},{"interface":"front","average":{"1min":2.109,"5min":1.3,"15min":1.5189999999999999},"min":{"1min":0.60399999999999998,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":13.464,"5min":13.464,"15min":13.464},"last":1.716}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:08:50.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:08:50.879 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:50.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:50 smithi123 bash[19119]: cluster 2024-02-12T16:08:49.620258+0000 mgr.y (mgr.14146) 699 : cluster [DBG] pgmap v859: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:50 smithi123 bash[19119]: audit 2024-02-12T16:08:50.041366+0000 mgr.y (mgr.14146) 700 : audit [DBG] from='client.15078 v1:172.21.15.121:0/1884465966' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:08:50.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:50.995 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:50 smithi169 bash[19604]: cluster 2024-02-12T16:08:49.620258+0000 mgr.y (mgr.14146) 699 : cluster [DBG] pgmap v859: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:50.995 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:50 smithi169 bash[19604]: audit 2024-02-12T16:08:50.041366+0000 mgr.y (mgr.14146) 700 : audit [DBG] from='client.15078 v1:172.21.15.121:0/1884465966' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:08:51.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:51.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:51.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:51.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:50 smithi121 bash[18489]: cluster 2024-02-12T16:08:49.620258+0000 mgr.y (mgr.14146) 699 : cluster [DBG] pgmap v859: 1 pgs: 1 active+clean; 0 B data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:51.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:50 smithi121 bash[18489]: audit 2024-02-12T16:08:50.041366+0000 mgr.y (mgr.14146) 700 : audit [DBG] from='client.15078 v1:172.21.15.121:0/1884465966' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:08:51.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:51.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:51.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:51.297 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:51.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:51.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:51.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:51.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:51.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:51.583 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:51.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:51.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:51.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:51.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:51.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:51.871 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:51.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:51.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:52.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:52.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:52.146 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:52.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:52.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:52.287 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:52.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:52.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:52.407 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:52.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:52.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:52.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:52.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:52.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:52.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:52.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:52.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:52.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: cluster 2024-02-12T16:08:51.631987+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: cluster 2024-02-12T16:08:51.632070+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: cluster 2024-02-12T16:08:51.694119+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e254: 12 total, 12 up, 12 in 2024-02-12T16:08:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: audit 2024-02-12T16:08:51.695784+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:08:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: audit 2024-02-12T16:08:51.695965+0000 mon.a (mon.0) 1115 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:08:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: audit 2024-02-12T16:08:51.696138+0000 mon.a (mon.0) 1116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:08:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: audit 2024-02-12T16:08:51.714385+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:08:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:52 smithi121 bash[18489]: audit 2024-02-12T16:08:52.003255+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:52.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:52.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:52.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:52.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: cluster 2024-02-12T16:08:51.631987+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:52.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: cluster 2024-02-12T16:08:51.632070+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:52.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: cluster 2024-02-12T16:08:51.694119+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e254: 12 total, 12 up, 12 in 2024-02-12T16:08:52.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: audit 2024-02-12T16:08:51.695784+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:08:52.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: audit 2024-02-12T16:08:51.695965+0000 mon.a (mon.0) 1115 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:08:52.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: audit 2024-02-12T16:08:51.696138+0000 mon.a (mon.0) 1116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:08:52.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: audit 2024-02-12T16:08:51.714385+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:08:52.972 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:52 smithi169 bash[19604]: audit 2024-02-12T16:08:52.003255+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:52.998 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:53.102 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: cluster 2024-02-12T16:08:51.631987+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: cluster 2024-02-12T16:08:51.632070+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: cluster 2024-02-12T16:08:51.694119+0000 mon.a (mon.0) 1113 : cluster [DBG] osdmap e254: 12 total, 12 up, 12 in 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: audit 2024-02-12T16:08:51.695784+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: audit 2024-02-12T16:08:51.695965+0000 mon.a (mon.0) 1115 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: audit 2024-02-12T16:08:51.696138+0000 mon.a (mon.0) 1116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: audit 2024-02-12T16:08:51.714385+0000 mon.a (mon.0) 1117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:08:53.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:52 smithi123 bash[19119]: audit 2024-02-12T16:08:52.003255+0000 mon.a (mon.0) 1118 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:53.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:53.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:53.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:53.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:53.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:53.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:53.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:53.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:53.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:53.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:53 smithi169 bash[19604]: cluster 2024-02-12T16:08:52.678120+0000 osd.1 (osd.1) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:53 smithi169 bash[19604]: cluster 2024-02-12T16:08:52.678222+0000 osd.1 (osd.1) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:53 smithi169 bash[19604]: cluster 2024-02-12T16:08:53.008563+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e255: 12 total, 12 up, 12 in 2024-02-12T16:08:53.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:53.835 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.838 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:53.939 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:53.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:53 smithi123 bash[19119]: cluster 2024-02-12T16:08:52.678120+0000 osd.1 (osd.1) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:53.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:53 smithi123 bash[19119]: cluster 2024-02-12T16:08:52.678222+0000 osd.1 (osd.1) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:53.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:53 smithi123 bash[19119]: cluster 2024-02-12T16:08:53.008563+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e255: 12 total, 12 up, 12 in 2024-02-12T16:08:53.978 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:53.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:54.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:53 smithi121 bash[18489]: cluster 2024-02-12T16:08:52.678120+0000 osd.1 (osd.1) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:08:54.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:53 smithi121 bash[18489]: cluster 2024-02-12T16:08:52.678222+0000 osd.1 (osd.1) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:08:54.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:53 smithi121 bash[18489]: cluster 2024-02-12T16:08:53.008563+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e255: 12 total, 12 up, 12 in 2024-02-12T16:08:54.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:08:54.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:54.094 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:08:54.113 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:54.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:54.129 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:08:54.137 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:08:54.155 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:08:54.179 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:08:54.209 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:08:54.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:54.231 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:08:54.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:54.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:54.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:08:54.321 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:08:54.343 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:08:54.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:54.370 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:08:54.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:54.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:54.394 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:08:54.396 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.396 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.396 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.396 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.396 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.396 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.397 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.397 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.397 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.397 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.397 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.397 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:08:54.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:54.526 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:54.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:54.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:54.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:54.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:54.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:54.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:54.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:54.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:54.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:54.949 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:54.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:55.026 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_1' created 2024-02-12T16:08:55.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:55.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:55 smithi121 bash[18489]: cluster 2024-02-12T16:08:53.621363+0000 mgr.y (mgr.14146) 702 : cluster [DBG] pgmap v863: 1 pgs: 1 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:55.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:55 smithi121 bash[18489]: audit 2024-02-12T16:08:54.027318+0000 mon.a (mon.0) 1120 : audit [DBG] from='client.? v1:172.21.15.121:0/3308447875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:55.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:55 smithi121 bash[18489]: audit 2024-02-12T16:08:54.194472+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? v1:172.21.15.121:0/123567438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:08:55.089 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:55.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:55.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:55.224 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:55.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:55 smithi169 bash[19604]: cluster 2024-02-12T16:08:53.621363+0000 mgr.y (mgr.14146) 702 : cluster [DBG] pgmap v863: 1 pgs: 1 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:55 smithi169 bash[19604]: audit 2024-02-12T16:08:54.027318+0000 mon.a (mon.0) 1120 : audit [DBG] from='client.? v1:172.21.15.121:0/3308447875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:55.233 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:55 smithi169 bash[19604]: audit 2024-02-12T16:08:54.194472+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? v1:172.21.15.121:0/123567438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:08:55.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:55.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:55.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:55 smithi123 bash[19119]: cluster 2024-02-12T16:08:53.621363+0000 mgr.y (mgr.14146) 702 : cluster [DBG] pgmap v863: 1 pgs: 1 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:55 smithi123 bash[19119]: audit 2024-02-12T16:08:54.027318+0000 mon.a (mon.0) 1120 : audit [DBG] from='client.? v1:172.21.15.121:0/3308447875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:08:55.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:55 smithi123 bash[19119]: audit 2024-02-12T16:08:54.194472+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? v1:172.21.15.121:0/123567438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-12T16:08:55.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:55.507 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:55.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:55.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:55.648 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:55.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:55.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:55.784 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:55.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:55.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:55.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:55.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:56.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:56.049 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:56.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:56.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:56.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:56.191 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:56.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:56.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:56 smithi169 bash[19604]: audit 2024-02-12T16:08:55.015146+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? v1:172.21.15.121:0/123567438' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-12T16:08:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:56 smithi169 bash[19604]: cluster 2024-02-12T16:08:55.015389+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e256: 12 total, 12 up, 12 in 2024-02-12T16:08:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:56 smithi169 bash[19604]: cluster 2024-02-12T16:08:56.017343+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e257: 12 total, 12 up, 12 in 2024-02-12T16:08:56.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:56 smithi121 bash[18489]: audit 2024-02-12T16:08:55.015146+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? v1:172.21.15.121:0/123567438' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-12T16:08:56.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:56 smithi121 bash[18489]: cluster 2024-02-12T16:08:55.015389+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e256: 12 total, 12 up, 12 in 2024-02-12T16:08:56.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:56 smithi121 bash[18489]: cluster 2024-02-12T16:08:56.017343+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e257: 12 total, 12 up, 12 in 2024-02-12T16:08:56.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:56.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:56 smithi123 bash[19119]: audit 2024-02-12T16:08:55.015146+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? v1:172.21.15.121:0/123567438' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-12T16:08:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:56 smithi123 bash[19119]: cluster 2024-02-12T16:08:55.015389+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e256: 12 total, 12 up, 12 in 2024-02-12T16:08:56.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:56 smithi123 bash[19119]: cluster 2024-02-12T16:08:56.017343+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e257: 12 total, 12 up, 12 in 2024-02-12T16:08:56.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:56.464 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:08:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:56.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:08:56.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-02-12T16:08:56.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:56.635 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:56.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:56.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:56.776 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:56.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:56.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:56.915 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:08:56.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:56.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:57.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:57.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:57 smithi169 bash[19604]: cluster 2024-02-12T16:08:55.621744+0000 mgr.y (mgr.14146) 703 : cluster [DBG] pgmap v865: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:57.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:57 smithi169 bash[19604]: cluster 2024-02-12T16:08:57.017624+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e258: 12 total, 12 up, 12 in 2024-02-12T16:08:57.055 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:57.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:57.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:57.195 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:57.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:57.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:57.301 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:08:57.301 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:08:57.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:57 smithi123 bash[19119]: cluster 2024-02-12T16:08:55.621744+0000 mgr.y (mgr.14146) 703 : cluster [DBG] pgmap v865: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:57.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:57 smithi123 bash[19119]: cluster 2024-02-12T16:08:57.017624+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e258: 12 total, 12 up, 12 in 2024-02-12T16:08:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:57.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:57.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:57 smithi121 bash[18489]: cluster 2024-02-12T16:08:55.621744+0000 mgr.y (mgr.14146) 703 : cluster [DBG] pgmap v865: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:57.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:57 smithi121 bash[18489]: cluster 2024-02-12T16:08:57.017624+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e258: 12 total, 12 up, 12 in 2024-02-12T16:08:57.440 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:57.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:57.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:57.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:57.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:57.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:57.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:57.749 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:57.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:57.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:57.886 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:08:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:57.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:08:57.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:58.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-12T16:08:58.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:08:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:58.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:08:58.108 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:08:58.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:58.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:58 smithi123 bash[19119]: audit 2024-02-12T16:08:57.081999+0000 mon.a (mon.0) 1126 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:58.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:58 smithi123 bash[19119]: audit 2024-02-12T16:08:57.173359+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? v1:172.21.15.121:0/1214694143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:08:58.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:58 smithi123 bash[19119]: audit 2024-02-12T16:08:57.377959+0000 mon.a (mon.0) 1128 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:08:58.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:58 smithi123 bash[19119]: audit 2024-02-12T16:08:57.379855+0000 mon.a (mon.0) 1129 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:08:58.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:58 smithi123 bash[19119]: audit 2024-02-12T16:08:57.389399+0000 mon.a (mon.0) 1130 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:58.164 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:08:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:58.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:08:58.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:58 smithi169 bash[19604]: audit 2024-02-12T16:08:57.081999+0000 mon.a (mon.0) 1126 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:58 smithi169 bash[19604]: audit 2024-02-12T16:08:57.173359+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? v1:172.21.15.121:0/1214694143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:08:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:58 smithi169 bash[19604]: audit 2024-02-12T16:08:57.377959+0000 mon.a (mon.0) 1128 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:08:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:58 smithi169 bash[19604]: audit 2024-02-12T16:08:57.379855+0000 mon.a (mon.0) 1129 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:08:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:58 smithi169 bash[19604]: audit 2024-02-12T16:08:57.389399+0000 mon.a (mon.0) 1130 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:58.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:58.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:58.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:58.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:08:58.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:58 smithi121 bash[18489]: audit 2024-02-12T16:08:57.081999+0000 mon.a (mon.0) 1126 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:58.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:58 smithi121 bash[18489]: audit 2024-02-12T16:08:57.173359+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? v1:172.21.15.121:0/1214694143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:08:58.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:58 smithi121 bash[18489]: audit 2024-02-12T16:08:57.377959+0000 mon.a (mon.0) 1128 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:08:58.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:58 smithi121 bash[18489]: audit 2024-02-12T16:08:57.379855+0000 mon.a (mon.0) 1129 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:08:58.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:58 smithi121 bash[18489]: audit 2024-02-12T16:08:57.389399+0000 mon.a (mon.0) 1130 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:08:58.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:58.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:58.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:08:58.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:58.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:58.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:58.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:58.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:58.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:58.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:08:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:58.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:08:58.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:59.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:08:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:59.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:08:59.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:59.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:59.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:59.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:59.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:59 smithi123 bash[19119]: cluster 2024-02-12T16:08:57.622417+0000 mgr.y (mgr.14146) 704 : cluster [DBG] pgmap v868: 17 pgs: 2 creating+peering, 11 unknown, 4 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:59.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:59 smithi123 bash[19119]: cluster 2024-02-12T16:08:58.079363+0000 mon.a (mon.0) 1131 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:59.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:59 smithi123 bash[19119]: audit 2024-02-12T16:08:58.098771+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.121:0/1214694143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:08:59.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:08:59 smithi123 bash[19119]: cluster 2024-02-12T16:08:58.098899+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e259: 12 total, 12 up, 12 in 2024-02-12T16:08:59.289 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:08:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:59.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:08:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:59 smithi169 bash[19604]: cluster 2024-02-12T16:08:57.622417+0000 mgr.y (mgr.14146) 704 : cluster [DBG] pgmap v868: 17 pgs: 2 creating+peering, 11 unknown, 4 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:59 smithi169 bash[19604]: cluster 2024-02-12T16:08:58.079363+0000 mon.a (mon.0) 1131 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:59 smithi169 bash[19604]: audit 2024-02-12T16:08:58.098771+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.121:0/1214694143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:08:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:08:59 smithi169 bash[19604]: cluster 2024-02-12T16:08:58.098899+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e259: 12 total, 12 up, 12 in 2024-02-12T16:08:59.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:59 smithi121 bash[18489]: cluster 2024-02-12T16:08:57.622417+0000 mgr.y (mgr.14146) 704 : cluster [DBG] pgmap v868: 17 pgs: 2 creating+peering, 11 unknown, 4 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:08:59.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:59 smithi121 bash[18489]: cluster 2024-02-12T16:08:58.079363+0000 mon.a (mon.0) 1131 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:08:59.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:59 smithi121 bash[18489]: audit 2024-02-12T16:08:58.098771+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? v1:172.21.15.121:0/1214694143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:08:59.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:08:59 smithi121 bash[18489]: cluster 2024-02-12T16:08:58.098899+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e259: 12 total, 12 up, 12 in 2024-02-12T16:08:59.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:08:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:59.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:08:59.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:59.571 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:08:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:59.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:08:59.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:59.713 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:08:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:59.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:08:59.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:08:59.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:08:59.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:08:59.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:08:59.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:00.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:00.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:00.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:00.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:00.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:00.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:00.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:00.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:00.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:00.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:00.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:00.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:00.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:00.977 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:00.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:01.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:09:01.057 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:09:01.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:01.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:01.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:01.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:01.256 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:01.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:01.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:01.371 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:01 smithi169 bash[19604]: cluster 2024-02-12T16:08:59.623183+0000 mgr.y (mgr.14146) 705 : cluster [DBG] pgmap v870: 17 pgs: 2 creating+peering, 3 unknown, 12 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:01.371 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:01 smithi169 bash[19604]: cluster 2024-02-12T16:09:00.652396+0000 osd.3 (osd.3) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:01.371 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:01 smithi169 bash[19604]: cluster 2024-02-12T16:09:00.652510+0000 osd.3 (osd.3) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:01 smithi123 bash[19119]: cluster 2024-02-12T16:08:59.623183+0000 mgr.y (mgr.14146) 705 : cluster [DBG] pgmap v870: 17 pgs: 2 creating+peering, 3 unknown, 12 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:01 smithi123 bash[19119]: cluster 2024-02-12T16:09:00.652396+0000 osd.3 (osd.3) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:01 smithi123 bash[19119]: cluster 2024-02-12T16:09:00.652510+0000 osd.3 (osd.3) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:01.396 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:01.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:01 smithi121 bash[18489]: cluster 2024-02-12T16:08:59.623183+0000 mgr.y (mgr.14146) 705 : cluster [DBG] pgmap v870: 17 pgs: 2 creating+peering, 3 unknown, 12 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:01.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:01 smithi121 bash[18489]: cluster 2024-02-12T16:09:00.652396+0000 osd.3 (osd.3) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:01.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:01 smithi121 bash[18489]: cluster 2024-02-12T16:09:00.652510+0000 osd.3 (osd.3) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:01.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:01.535 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:01.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:09:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:01.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:01.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:01.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:01.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:01.765 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":870,"stamp":"2024-02-12T16:08:59.622735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":28,"acting":28,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":4204564,"kb_used_data":538840,"kb_used_omap":0,"kb_used_meta":3664896,"kb_avail":1120737260,"statfs":{"total":1151940427776,"available":1147634954240,"internally_reserved":0,"allocated":551772160,"data_stored":547658191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3752853504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13664256,"num_objects":-210,"num_object_clones":0,"num_object_copies":-420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-210,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-324684,"num_write_kb":-10377985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11743,"num_bytes_recovered":-759496704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.953707"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-02-12T16:08:56.027598+0000","last_change":"2024-02-12T16:08:56.026943+0000","last_active":"2024-02-12T16:08:56.027598+0000","last_peered":"2024-02-12T16:08:56.027598+0000","last_clean":"2024-02-12T16:08:56.027598+0000","last_became_active":"2024-02-12T16:08:56.025977+0000","last_became_peered":"2024-02-12T16:08:56.025977+0000","last_unstale":"2024-02-12T16:08:56.027598+0000","last_undegraded":"2024-02-12T16:08:56.027598+0000","last_fullsized":"2024-02-12T16:08:56.027598+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:08:55.005799+0000","last_change":"2024-02-12T16:08:55.005799+0000","last_active":"2024-02-12T16:08:55.005799+0000","last_peered":"2024-02-12T16:08:55.005799+0000","last_clean":"2024-02-12T16:08:55.005799+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:08:55.005799+0000","last_undegraded":"2024-02-12T16:08:55.005799+0000","last_fullsized":"2024-02-12T16:08:55.005799+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-02-12T16:08:56.027650+0000","last_change":"2024-02-12T16:08:56.026138+0000","last_active":"2024-02-12T16:08:56.027650+0000","last_peered":"2024-02-12T16:08:56.027650+0000","last_clean":"2024-02-12T16:08:56.027650+0000","last_became_active":"2024-02-12T16:08:56.025860+0000","last_became_peered":"2024-02-12T16:08:56.025860+0000","last_unstale":"2024-02-12T16:08:56.027650+0000","last_undegraded":"2024-02-12T16:08:56.027650+0000","last_fullsized":"2024-02-12T16:08:56.027650+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":8,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-02-12T16:08:56.027997+0000","last_change":"2024-02-12T16:08:56.027997+0000","last_active":"2024-02-12T16:08:56.027997+0000","last_peered":"2024-02-12T16:08:56.027997+0000","last_clean":"2024-02-12T16:08:56.027997+0000","last_became_active":"2024-02-12T16:08:56.027658+0000","last_became_peered":"2024-02-12T16:08:56.027658+0000","last_unstale":"2024-02-12T16:08:56.027997+0000","last_undegraded":"2024-02-12T16:08:56.027997+0000","last_fullsized":"2024-02-12T16:08:56.027997+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-02-12T16:08:57.533201+0000","last_change":"2024-02-12T16:08:56.021048+0000","last_active":"2024-02-12T16:08:57.533201+0000","last_peered":"2024-02-12T16:08:57.533201+0000","last_clean":"2024-02-12T16:08:57.533201+0000","last_became_active":"2024-02-12T16:08:56.020761+0000","last_became_peered":"2024-02-12T16:08:56.020761+0000","last_unstale":"2024-02-12T16:08:57.533201+0000","last_undegraded":"2024-02-12T16:08:57.533201+0000","last_fullsized":"2024-02-12T16:08:57.533201+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-02-12T16:08:57.714050+0000","last_change":"2024-02-12T16:08:56.027375+0000","last_active":"2024-02-12T16:08:57.714050+0000","last_peered":"2024-02-12T16:08:57.714050+0000","last_clean":"2024-02-12T16:08:57.714050+0000","last_became_active":"2024-02-12T16:08:56.027087+0000","last_became_peered":"2024-02-12T16:08:56.027087+0000","last_unstale":"2024-02-12T16:08:57.714050+0000","last_undegraded":"2024-02-12T16:08:57.714050+0000","last_fullsized":"2024-02-12T16:08:57.714050+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-02-12T16:08:56.025062+0000","last_change":"2024-02-12T16:08:56.024538+0000","last_active":"2024-02-12T16:08:56.025062+0000","last_peered":"2024-02-12T16:08:56.025062+0000","last_clean":"2024-02-12T16:08:56.025062+0000","last_became_active":"2024-02-12T16:08:56.024064+0000","last_became_peered":"2024-02-12T16:08:56.024064+0000","last_unstale":"2024-02-12T16:08:56.025062+0000","last_undegraded":"2024-02-12T16:08:56.025062+0000","last_fullsized":"2024-02-12T16:08:56.025062+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:08:55.005799+0000","last_change":"2024-02-12T16:08:55.005799+0000","last_active":"2024-02-12T16:08:55.005799+0000","last_peered":"2024-02-12T16:08:55.005799+0000","last_clean":"2024-02-12T16:08:55.005799+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:08:55.005799+0000","last_undegraded":"2024-02-12T16:08:55.005799+0000","last_fullsized":"2024-02-12T16:08:55.005799+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-02-12T16:08:57.533221+0000","last_change":"2024-02-12T16:08:56.022243+0000","last_active":"2024-02-12T16:08:57.533221+0000","last_peered":"2024-02-12T16:08:57.533221+0000","last_clean":"2024-02-12T16:08:57.533221+0000","last_became_active":"2024-02-12T16:08:56.021958+0000","last_became_peered":"2024-02-12T16:08:56.021958+0000","last_unstale":"2024-02-12T16:08:57.533221+0000","last_undegraded":"2024-02-12T16:08:57.533221+0000","last_fullsized":"2024-02-12T16:08:57.533221+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-12T16:08:55.005799+0000","last_change":"2024-02-12T16:08:55.015555+0000","last_active":"2024-02-12T16:08:55.005799+0000","last_peered":"2024-02-12T16:08:55.005799+0000","last_clean":"2024-02-12T16:08:55.005799+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:08:55.005799+0000","last_undegraded":"2024-02-12T16:08:55.005799+0000","last_fullsized":"2024-02-12T16:08:55.005799+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":8,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-02-12T16:08:56.026672+0000","last_change":"2024-02-12T16:08:56.026672+0000","last_active":"2024-02-12T16:08:56.026672+0000","last_peered":"2024-02-12T16:08:56.026672+0000","last_clean":"2024-02-12T16:08:56.026672+0000","last_became_active":"2024-02-12T16:08:56.026413+0000","last_became_peered":"2024-02-12T16:08:56.026413+0000","last_unstale":"2024-02-12T16:08:56.026672+0000","last_undegraded":"2024-02-12T16:08:56.026672+0000","last_fullsized":"2024-02-12T16:08:56.026672+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":8,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-02-12T16:08:56.021364+0000","last_change":"2024-02-12T16:08:56.021364+0000","last_active":"2024-02-12T16:08:56.021364+0000","last_peered":"2024-02-12T16:08:56.021364+0000","last_clean":"2024-02-12T16:08:56.021364+0000","last_became_active":"2024-02-12T16:08:56.021115+0000","last_became_peered":"2024-02-12T16:08:56.021115+0000","last_unstale":"2024-02-12T16:08:56.021364+0000","last_undegraded":"2024-02-12T16:08:56.021364+0000","last_fullsized":"2024-02-12T16:08:56.021364+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-02-12T16:08:57.020782+0000","last_change":"2024-02-12T16:08:56.026095+0000","last_active":"2024-02-12T16:08:57.020782+0000","last_peered":"2024-02-12T16:08:57.020782+0000","last_clean":"2024-02-12T16:08:57.020782+0000","last_became_active":"2024-02-12T16:08:56.025422+0000","last_became_peered":"2024-02-12T16:08:56.025422+0000","last_unstale":"2024-02-12T16:08:57.020782+0000","last_undegraded":"2024-02-12T16:08:57.020782+0000","last_fullsized":"2024-02-12T16:08:57.020782+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-12T16:08:55.005799+0000","last_change":"2024-02-12T16:08:55.015847+0000","last_active":"2024-02-12T16:08:55.005799+0000","last_peered":"2024-02-12T16:08:55.005799+0000","last_clean":"2024-02-12T16:08:55.005799+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:08:55.005799+0000","last_undegraded":"2024-02-12T16:08:55.005799+0000","last_fullsized":"2024-02-12T16:08:55.005799+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:08:55.005799+0000","last_change":"2024-02-12T16:08:55.005799+0000","last_active":"2024-02-12T16:08:55.005799+0000","last_peered":"2024-02-12T16:08:55.005799+0000","last_clean":"2024-02-12T16:08:55.005799+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:08:55.005799+0000","last_undegraded":"2024-02-12T16:08:55.005799+0000","last_fullsized":"2024-02-12T16:08:55.005799+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-02-12T16:08:57.533021+0000","last_change":"2024-02-12T16:08:56.274192+0000","last_active":"2024-02-12T16:08:57.533021+0000","last_peered":"2024-02-12T16:08:57.533021+0000","last_clean":"2024-02-12T16:08:57.533021+0000","last_became_active":"2024-02-12T16:08:56.273670+0000","last_became_peered":"2024-02-12T16:08:56.273670+0000","last_unstale":"2024-02-12T16:08:57.533021+0000","last_undegraded":"2024-02-12T16:08:57.533021+0000","last_fullsized":"2024-02-12T16:08:57.533021+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":272,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-02-12T16:08:56.024362+0000","last_change":"2024-02-12T16:08:12.878558+0000","last_active":"2024-02-12T16:08:56.024362+0000","last_peered":"2024-02-12T16:08:56.024362+0000","last_clean":"2024-02-12T16:08:56.024362+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:08:56.024362+0000","last_undegraded":"2024-02-12T16:08:56.024362+0000","last_fullsized":"2024-02-12T16:08:56.024362+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:12.878487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:08:12.878487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":26,"acting":26,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958657,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302328,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442824,"statfs":{"total":95995035648,"available":95685451776,"internally_reserved":0,"allocated":663552,"data_stored":333844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.745,"15min":0.93200000000000005},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":3.3929999999999998,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.91100000000000003,"15min":1.0389999999999999},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.1909999999999998,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":1.2270000000000001,"15min":1.1220000000000001},"min":{"1min":0.47399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.228,"5min":1.109,"15min":1.093},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.786,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.97099999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0980000000000001,"15min":1.1259999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.7109999999999999,"5min":5.274,"15min":5.274},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":0.96999999999999997,"15min":1.1160000000000001},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":3.7360000000000002,"5min":5.375,"15min":5.375},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.077,"15min":1.1499999999999999},"min":{"1min":0.33200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.712,"5min":4.266,"15min":4.266},"last":1.3560000000000001},{"interface":"front","average":{"1min":2.242,"5min":1.3320000000000001,"15min":1.224},"min":{"1min":0.34799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":8.9819999999999993,"5min":8.9819999999999993,"15min":8.9819999999999993},"last":1.2849999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.0660000000000001,"15min":1.2310000000000001},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6420000000000003,"5min":6.6420000000000003,"15min":6.6420000000000003},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.83199999999999996,"15min":1.21},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.788,"5min":3.1299999999999999,"15min":4.0789999999999997},"last":1.2210000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.95799999999999996,"15min":1.284},"min":{"1min":0.376,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.474,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":1.262},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.1839999999999999,"15min":1.367},"min":{"1min":0.45900000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.6849999999999996,"5min":6.6849999999999996,"15min":6.6849999999999996},"last":1.1890000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.97599999999999998,"15min":1.3120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.105,"5min":3.2290000000000001,"15min":4.4989999999999997},"last":1.012},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.0509999999999999,"15min":1.401},"min":{"1min":0.501,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7709999999999999,"5min":3.7709999999999999,"15min":3.7709999999999999},"last":1.5580000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.006,"15min":1.389},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.909,"5min":3.4710000000000001,"15min":4.3129999999999997},"last":1.327},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.46},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.7589999999999999,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.4690000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.054,"15min":1.423},"min":{"1min":0.42399999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.6579999999999999,"5min":3.6579999999999999,"15min":4},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.048,"15min":1.4750000000000001},"min":{"1min":0.40999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.7370000000000001,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":1.575}]},{"osd":10,"last update":"Mon Feb 12 16:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.131,"15min":1.46},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.903,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.093,"15min":1.5549999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.73,"5min":4.6689999999999996,"15min":4.6689999999999996},"last":1.4339999999999999}]}]},{"osd":10,"up_from":83,"seq":356482285707,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543860,"kb_used_data":230252,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93201292,"statfs":{"total":95995035648,"available":95438123008,"internally_reserved":0,"allocated":235778048,"data_stored":235430288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.95699999999999996,"15min":0.91700000000000004},"min":{"1min":0.26100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.1890000000000001,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.3999999999999999,"5min":0.96399999999999997,"15min":1.0249999999999999},"min":{"1min":0.32300000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.2590000000000003,"5min":7.6580000000000004,"15min":7.6580000000000004},"last":1.179}]},{"osd":2,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.879,"15min":0.95099999999999996},"min":{"1min":0.29799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.4610000000000003,"5min":4.8300000000000001,"15min":4.8300000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.9550000000000001,"5min":1.052,"15min":1.0009999999999999},"min":{"1min":0.31900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.9829999999999997,"5min":7.9829999999999997,"15min":7.9829999999999997},"last":1.325}]},{"osd":3,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.048,"15min":1.0309999999999999},"min":{"1min":0.35099999999999998,"5min":0.308,"15min":0.308},"max":{"1min":4.343,"5min":4.8620000000000001,"15min":4.8620000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.96399999999999997,"15min":0.995},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.4039999999999999,"5min":5.4100000000000001,"15min":5.4100000000000001},"last":0.69699999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.92100000000000004,"15min":1.075},"min":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3740000000000001,"5min":4.9189999999999996,"15min":4.9189999999999996},"last":1.095},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.88700000000000001,"15min":1.115},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.5919999999999996,"5min":4.5919999999999996,"15min":4.5919999999999996},"last":1.631}]},{"osd":5,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":7.2439999999999998,"15min":3.202},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.9079999999999999,"5min":602.05399999999997,"15min":602.05399999999997},"last":0.748},{"interface":"front","average":{"1min":1.3400000000000001,"5min":7.2949999999999999,"15min":3.3050000000000002},"min":{"1min":0.374,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.6769999999999996,"5min":598.875,"15min":598.875},"last":1.512}]},{"osd":6,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.98099999999999998,"15min":1.1399999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.9470000000000001,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.181,"5min":1.046,"15min":1.266},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.6660000000000004,"5min":4.6660000000000004,"15min":4.6660000000000004},"last":1.407}]},{"osd":7,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":7.3460000000000001,"15min":3.3279999999999998},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":3.3319999999999999,"5min":597.48299999999995,"15min":597.48299999999995},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":7.4219999999999997,"15min":3.3929999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":4.9930000000000003,"5min":600.23099999999999,"15min":600.23099999999999},"last":0.84199999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.1499999999999999,"15min":1.3},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":7.2720000000000002,"5min":7.2720000000000002,"15min":7.2720000000000002},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.179,"15min":1.321},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":4.694,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":1.4610000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.1399999999999999,"15min":1.365},"min":{"1min":0.57799999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":4.7089999999999996,"5min":4.7089999999999996,"15min":4.7089999999999996},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.139,"15min":1.3500000000000001},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.4249999999999998,"5min":4.7789999999999999,"15min":4.7789999999999999},"last":1.3}]},{"osd":11,"last update":"Mon Feb 12 16:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.2190000000000001,"15min":1.468},"min":{"1min":0.57699999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.8730000000000002,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1659999999999999,"15min":1.4690000000000001},"min":{"1min":0.58599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":4.1479999999999997,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":1.7390000000000001}]}]},{"osd":9,"up_from":74,"seq":317827580051,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306088,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93439064,"statfs":{"total":95995035648,"available":95681601536,"internally_reserved":0,"allocated":688128,"data_stored":346211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.82199999999999995,"15min":0.89700000000000002},"min":{"1min":0.24399999999999999,"5min":0.215,"15min":0.215},"max":{"1min":4.6879999999999997,"5min":10.788,"15min":10.788},"last":0.746},{"interface":"front","average":{"1min":1.8819999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.33500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.9630000000000001,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.68600000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":1.4590000000000001,"15min":1.1619999999999999},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":7.8479999999999999,"5min":11.991,"15min":11.991},"last":0.71899999999999997},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.357,"15min":1.145},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":7.8360000000000003,"5min":11.977,"15min":11.977},"last":0.82899999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.234,"15min":1.0409999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.6230000000000002,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.802,"5min":1.2509999999999999,"15min":1.089},"min":{"1min":0.315,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.4989999999999997,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":0.83999999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1100000000000001,"15min":1.016},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":1.105},{"interface":"front","average":{"1min":2.2149999999999999,"5min":1.2470000000000001,"15min":1.1579999999999999},"min":{"1min":0.439,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.0650000000000004,"5min":5.0650000000000004,"15min":5.0650000000000004},"last":1.0669999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.304,"15min":1.2170000000000001},"min":{"1min":0.38600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.399,"5min":25.552,"15min":25.552},"last":1.169},{"interface":"front","average":{"1min":0.66200000000000003,"5min":1.113,"15min":1.1599999999999999},"min":{"1min":0.35399999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.4540000000000002,"5min":25.847999999999999,"15min":25.847999999999999},"last":1.2090000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":7.9550000000000001,"15min":3.4689999999999999},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.3230000000000004,"5min":617.95899999999995,"15min":617.95899999999995},"last":1.262},{"interface":"front","average":{"1min":1.45,"5min":7.96,"15min":3.512},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":4.9960000000000004,"5min":617.88099999999997,"15min":617.88099999999997},"last":1.3180000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.105,"15min":1.25},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.0960000000000001,"5min":5.0960000000000001,"15min":5.0960000000000001},"last":1.4730000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.88300000000000001,"15min":1.1919999999999999},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.97099999999999997,"5min":3.4900000000000002,"15min":3.4900000000000002},"last":1.1499999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":11.598000000000001,"15min":4.7380000000000004},"min":{"1min":0.39000000000000001,"5min":0.379,"15min":0.379},"max":{"1min":4.3780000000000001,"5min":613.28200000000004,"15min":613.28200000000004},"last":1.345},{"interface":"front","average":{"1min":1.617,"5min":11.587,"15min":4.8070000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":5.7460000000000004,"5min":613.25199999999995,"15min":613.25199999999995},"last":1.407}]},{"osd":8,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.2310000000000001,"15min":1.3260000000000001},"min":{"1min":0.51300000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.96,"5min":4.96,"15min":4.96},"last":1.226},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0129999999999999,"15min":1.2450000000000001},"min":{"1min":0.46500000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4399999999999999,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":1.3799999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":1.03,"15min":1.2969999999999999},"min":{"1min":0.46000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.4399999999999999,"5min":5.5,"15min":5.5},"last":1.083},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.234,"15min":1.3220000000000001},"min":{"1min":0.53000000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.2400000000000002,"5min":4.7619999999999996,"15min":4.7619999999999996},"last":1.246}]}]},{"osd":8,"up_from":65,"seq":279172874394,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302004,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93443148,"statfs":{"total":95995035648,"available":95685783552,"internally_reserved":0,"allocated":700416,"data_stored":352676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.309,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.0430000000000001,"5min":5.383,"15min":5.383},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":0.90800000000000003,"15min":0.91500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.5330000000000004,"5min":4.5330000000000004,"15min":4.5330000000000004},"last":0.96099999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.8939999999999999,"15min":2.0230000000000001},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":7.3789999999999996,"5min":23.661000000000001,"15min":23.661000000000001},"last":1.032},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.9970000000000001,"15min":2.161},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.4340000000000002,"5min":23.672000000000001,"15min":23.672000000000001},"last":1.421}]},{"osd":2,"last update":"Mon Feb 12 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":12.574,"5min":3.3340000000000001,"15min":1.696},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":193.929,"5min":193.929,"15min":193.929},"last":0.52300000000000002},{"interface":"front","average":{"1min":12.651,"5min":3.484,"15min":1.8380000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":193.82499999999999,"5min":193.82499999999999,"15min":193.82499999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0629999999999999,"15min":1.046},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.5659999999999998,"5min":5.4020000000000001,"15min":5.4020000000000001},"last":1.256},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0620000000000001,"15min":1.0920000000000001},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.8879999999999999,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":1.0800000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.92300000000000004,"15min":1.1020000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":3.8420000000000001,"5min":6.4619999999999997,"15min":6.4619999999999997},"last":1.643},{"interface":"front","average":{"1min":1.091,"5min":1.087,"15min":1.1539999999999999},"min":{"1min":0.376,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.9279999999999999,"5min":5.71,"15min":5.71},"last":1.29}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.052,"15min":1.1240000000000001},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.956,"5min":13.961,"15min":13.961},"last":1.347},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.052,"15min":1.1890000000000001},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.0760000000000001,"5min":13.103,"15min":13.103},"last":1.5569999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1,"15min":1.157},"min":{"1min":0.46400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.0289999999999999,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.2050000000000001,"15min":1.2709999999999999},"min":{"1min":0.40100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.4720000000000004,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":1.7190000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2070000000000001,"15min":1.2869999999999999},"min":{"1min":0.51600000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.8719999999999999,"5min":13.242000000000001,"15min":13.242000000000001},"last":1.829},{"interface":"front","average":{"1min":1.206,"5min":1.3260000000000001,"15min":1.355},"min":{"1min":0.52600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":3.9820000000000002,"5min":13.388,"15min":13.388},"last":1.47}]},{"osd":9,"last update":"Mon Feb 12 16:08:26 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1419999999999999,"15min":1.2689999999999999},"min":{"1min":0.40899999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.7909999999999999,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.776},{"interface":"front","average":{"1min":1.056,"5min":1.212,"15min":1.238},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.8119999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"last":1.8720000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.214,"15min":1.3220000000000001},"min":{"1min":0.57499999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.2670000000000003,"5min":5.4189999999999996,"15min":5.4189999999999996},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.0429999999999999,"15min":1.3340000000000001},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":3.4860000000000002,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":1.5029999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:08:15 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":15.471,"15min":29.544},"min":{"1min":0.47899999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":4.2539999999999996,"5min":635.66399999999999,"15min":635.66399999999999},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":15.201000000000001,"15min":29.236000000000001},"min":{"1min":0.51600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":5.0800000000000001,"5min":636.28099999999995,"15min":636.28099999999995},"last":1.587}]}]},{"osd":7,"up_from":56,"seq":240518168739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310196,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434956,"statfs":{"total":95995035648,"available":95677394944,"internally_reserved":0,"allocated":700416,"data_stored":352676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":4.101,"5min":1.508,"15min":1.173},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":60.152000000000001,"5min":60.152000000000001,"15min":60.152000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":4.4569999999999999,"5min":1.7410000000000001,"15min":1.222},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":62.020000000000003,"5min":62.020000000000003,"15min":62.020000000000003},"last":1.101}]},{"osd":2,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":4.9379999999999997,"5min":1.7829999999999999,"15min":1.266},"min":{"1min":0.36199999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":60.204999999999998,"5min":60.204999999999998,"15min":60.204999999999998},"last":0.86599999999999999},{"interface":"front","average":{"1min":4.4580000000000002,"5min":1.651,"15min":1.2490000000000001},"min":{"1min":0.33800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":59.365000000000002,"5min":59.365000000000002,"15min":59.365000000000002},"last":1.1259999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.1140000000000001,"15min":1.093},"min":{"1min":0.28000000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.4020000000000001,"5min":4.8200000000000003,"15min":4.8200000000000003},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.097,"15min":1.083},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.1600000000000001,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":1.016}]},{"osd":4,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.79800000000000004,"15min":0.92600000000000005},"min":{"1min":0.371,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.8220000000000001,"5min":6.2519999999999998,"15min":6.2519999999999998},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.84499999999999997,"15min":0.97999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.4260000000000002,"5min":4.7430000000000003,"15min":4.7430000000000003},"last":1.1619999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.91000000000000003,"15min":1.028},"min":{"1min":0.33200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.8969999999999998,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":1.3049999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.89700000000000002,"15min":1.0780000000000001},"min":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":3.907,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":1.427}]},{"osd":6,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0489999999999999,"15min":1.1459999999999999},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.9809999999999999,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":1.204},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.95199999999999996,"15min":1.1240000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.5119999999999996,"5min":4.7149999999999999,"15min":4.7149999999999999},"last":1.52}]},{"osd":8,"last update":"Mon Feb 12 16:08:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.1699999999999999,"15min":1.3360000000000001},"min":{"1min":0.5,"5min":0.432,"15min":0.432},"max":{"1min":4.7779999999999996,"5min":4.7779999999999996,"15min":4.7779999999999996},"last":1.571},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.2649999999999999,"15min":1.3819999999999999},"min":{"1min":0.54500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7919999999999998,"5min":4.851,"15min":4.851},"last":0.94599999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:08:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":4.3230000000000004,"15min":2.4009999999999998},"min":{"1min":0.53800000000000003,"5min":0.441,"15min":0.441},"max":{"1min":4.5960000000000001,"5min":278.54599999999999,"15min":278.54599999999999},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":4.2670000000000003,"15min":2.3700000000000001},"min":{"1min":0.57099999999999995,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.4870000000000001,"5min":281.09199999999998,"15min":281.09199999999998},"last":1.379}]},{"osd":10,"last update":"Mon Feb 12 16:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.155,"15min":1.403},"min":{"1min":0.54900000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":3.8919999999999999,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.208,"15min":1.4330000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":4.4560000000000004,"5min":4.9450000000000003,"15min":4.9450000000000003},"last":1.27}]},{"osd":11,"last update":"Mon Feb 12 16:08:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":4.1340000000000003,"15min":2.4260000000000002},"min":{"1min":0.57599999999999996,"5min":0.498,"15min":0.498},"max":{"1min":3.593,"5min":280.83999999999997,"15min":280.83999999999997},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.528,"5min":4.048,"15min":2.367},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.8339999999999996,"5min":278.63799999999998,"15min":278.63799999999998},"last":1.647}]}]},{"osd":6,"up_from":48,"seq":206158430380,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314304,"kb_used_data":696,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93430848,"statfs":{"total":95995035648,"available":95673188352,"internally_reserved":0,"allocated":712704,"data_stored":358814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.7959999999999998,"5min":1.429,"15min":1.121},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":67.093000000000004,"5min":67.093000000000004,"15min":67.093000000000004},"last":0.93400000000000005},{"interface":"front","average":{"1min":3.8149999999999999,"5min":1.462,"15min":1.1240000000000001},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":68.549000000000007,"5min":68.549000000000007,"15min":68.549000000000007},"last":1.113}]},{"osd":1,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9489999999999998,"5min":1.6679999999999999,"15min":1.1879999999999999},"min":{"1min":0.317,"5min":0.308,"15min":0.308},"max":{"1min":66.936000000000007,"5min":66.936000000000007,"15min":66.936000000000007},"last":1.2829999999999999},{"interface":"front","average":{"1min":4.3890000000000002,"5min":1.6359999999999999,"15min":1.222},"min":{"1min":0.34499999999999997,"5min":0.309,"15min":0.309},"max":{"1min":68.683000000000007,"5min":68.683000000000007,"15min":68.683000000000007},"last":1.1619999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":4.1319999999999997,"5min":2.452,"15min":1.488},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":68.462000000000003,"5min":68.462000000000003,"15min":68.462000000000003},"last":1.071},{"interface":"front","average":{"1min":3.9399999999999999,"5min":2.456,"15min":1.5189999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":67.322999999999993,"5min":67.322999999999993,"15min":67.322999999999993},"last":1.6419999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.512,"15min":1.1659999999999999},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":68.332999999999998,"5min":68.332999999999998,"15min":68.332999999999998},"last":0.97799999999999998},{"interface":"front","average":{"1min":4.1600000000000001,"5min":1.55,"15min":1.228},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":67.281999999999996,"5min":67.281999999999996,"15min":67.281999999999996},"last":1.51}]},{"osd":4,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.7919999999999998,"5min":1.474,"15min":1.141},"min":{"1min":0.38300000000000001,"5min":0.316,"15min":0.316},"max":{"1min":68.605999999999995,"5min":68.605999999999995,"15min":68.605999999999995},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.77600000000000002,"15min":0.94899999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.3999999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.3169999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9750000000000001,"5min":1.593,"15min":1.2589999999999999},"min":{"1min":0.36099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":69.260000000000005,"5min":69.260000000000005,"15min":69.260000000000005},"last":1.3979999999999999},{"interface":"front","average":{"1min":3.839,"5min":1.4259999999999999,"15min":1.22},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":68.742000000000004,"5min":68.742000000000004,"15min":68.742000000000004},"last":1.2290000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":9.2140000000000004,"5min":2.7429999999999999,"15min":1.732},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":67.144999999999996,"5min":67.144999999999996,"15min":67.144999999999996},"last":1.784},{"interface":"front","average":{"1min":8.9420000000000002,"5min":2.6629999999999998,"15min":1.7490000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":67.230999999999995,"5min":67.230999999999995,"15min":67.230999999999995},"last":1.9370000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":4.7350000000000003,"5min":1.8779999999999999,"15min":1.5880000000000001},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":68.653000000000006,"5min":68.653000000000006,"15min":68.653000000000006},"last":1.75},{"interface":"front","average":{"1min":4.8259999999999996,"5min":1.752,"15min":1.5589999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":68.620999999999995,"5min":68.620999999999995,"15min":68.620999999999995},"last":1.5720000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":4.2510000000000003,"5min":1.8140000000000001,"15min":1.617},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":67.188000000000002,"5min":67.188000000000002,"15min":67.188000000000002},"last":2.0499999999999998},{"interface":"front","average":{"1min":4.2519999999999998,"5min":1.79,"15min":1.5620000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":68.506,"5min":68.506,"15min":68.506},"last":1.702}]},{"osd":10,"last update":"Mon Feb 12 16:08:09 2024","interfaces":[{"interface":"back","average":{"1min":4.2149999999999999,"5min":1.8759999999999999,"15min":1.7150000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":68.722999999999999,"5min":68.722999999999999,"15min":68.722999999999999},"last":1.8720000000000001},{"interface":"front","average":{"1min":4.2560000000000002,"5min":1.72,"15min":1.623},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":68.573999999999998,"5min":68.573999999999998,"15min":68.573999999999998},"last":2.0030000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297896,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447256,"statfs":{"total":95995035648,"available":95689990144,"internally_reserved":0,"allocated":688128,"data_stored":346211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.2390000000000001,"15min":0.93200000000000005},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.0030000000000001,"5min":9.8740000000000006,"15min":9.8740000000000006},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.244,"15min":0.98699999999999999},"min":{"1min":0.27700000000000002,"5min":0.255,"15min":0.255},"max":{"1min":5.9130000000000003,"5min":10.513,"15min":10.513},"last":0.73999999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.861,"15min":1.2110000000000001},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.1529999999999996,"5min":40.542999999999999,"15min":40.542999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.865,"5min":1.9750000000000001,"15min":1.258},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":9.2119999999999997,"5min":40.811999999999998,"15min":40.811999999999998},"last":0.871}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.169,"15min":1.004},"min":{"1min":0.28499999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.6900000000000004,"5min":6.0170000000000003,"15min":6.0170000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.962,"5min":1.405,"15min":1.1259999999999999},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":10.225,"5min":10.648999999999999,"15min":10.648999999999999},"last":0.60499999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.4199999999999999,"15min":1.202},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.1050000000000004,"5min":9.6780000000000008,"15min":9.6780000000000008},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.381,"15min":1.2410000000000001},"min":{"1min":0.40300000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":5.1870000000000003,"5min":10.244999999999999,"15min":10.244999999999999},"last":1.181}]},{"osd":5,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.399,"15min":1.292},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.8979999999999999,"5min":9.8019999999999996,"15min":9.8019999999999996},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.173,"15min":1.2789999999999999},"min":{"1min":0.46999999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0390000000000001,"5min":10.420999999999999,"15min":10.420999999999999},"last":1.21}]},{"osd":6,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.234,"15min":1.26},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.1779999999999999,"5min":10.455,"15min":10.455},"last":1.296},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.242,"15min":1.3380000000000001},"min":{"1min":0.58599999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.21,"5min":10.488,"15min":10.488},"last":1.3680000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3580000000000001,"15min":1.3999999999999999},"min":{"1min":0.50600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":3.7869999999999999,"5min":6.2869999999999999,"15min":6.2869999999999999},"last":1.091},{"interface":"front","average":{"1min":1.448,"5min":1.2529999999999999,"15min":1.3200000000000001},"min":{"1min":0.503,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.7389999999999999,"5min":5.1429999999999998,"15min":5.1429999999999998},"last":1.1399999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.5449999999999999,"15min":1.4770000000000001},"min":{"1min":0.50900000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":9.9049999999999994,"15min":9.9049999999999994},"last":1.472},{"interface":"front","average":{"1min":1.347,"5min":1.597,"15min":1.577},"min":{"1min":0.625,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.2410000000000001,"5min":10.313000000000001,"15min":10.313000000000001},"last":1.4490000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.522,"15min":1.6080000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":7.9409999999999998,"5min":10.352,"15min":10.352},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.2949999999999999,"15min":1.4490000000000001},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.7839999999999998,"5min":4.7839999999999998,"15min":4.7839999999999998},"last":1.258}]},{"osd":10,"last update":"Mon Feb 12 16:08:14 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.6060000000000001,"15min":1.579},"min":{"1min":0.55500000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":5.181,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":1.556},{"interface":"front","average":{"1min":1.948,"5min":1.6040000000000001,"15min":1.6970000000000001},"min":{"1min":0.622,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":7.984,"5min":10.387,"15min":10.387},"last":1.629}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738588,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302340,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442812,"statfs":{"total":95995035648,"available":95685439488,"internally_reserved":0,"allocated":663552,"data_stored":333491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":8.5429999999999993,"5min":2.464,"15min":1.3320000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":114.01600000000001,"5min":114.01600000000001,"15min":114.01600000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":7.6890000000000001,"5min":2.6160000000000001,"15min":1.405},"min":{"1min":0.29899999999999999,"5min":0.254,"15min":0.254},"max":{"1min":114.342,"5min":114.342,"15min":114.342},"last":0.93899999999999995}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.048,"15min":0.89900000000000002},"min":{"1min":0.27600000000000002,"5min":0.215,"15min":0.215},"max":{"1min":5.633,"5min":5.633,"15min":5.633},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.0409999999999999,"15min":0.95799999999999996},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.5140000000000002,"5min":7.5140000000000002,"15min":7.5140000000000002},"last":0.96999999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.86499999999999999,"15min":1.0349999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.804,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":1.1990000000000001},{"interface":"front","average":{"1min":0.879,"5min":0.97699999999999998,"15min":1.008},"min":{"1min":0.29699999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.585,"5min":4.5860000000000003,"15min":4.5860000000000003},"last":1.0549999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.93300000000000005,"15min":1.034},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"last":1.264},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.86699999999999999,"15min":1.0960000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.75,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.5469999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.93600000000000005,"15min":1.089},"min":{"1min":0.376,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6699999999999999,"5min":5.4699999999999998,"15min":5.4699999999999998},"last":1.423},{"interface":"front","average":{"1min":0.875,"5min":0.97299999999999998,"15min":1.153},"min":{"1min":0.502,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.6579999999999999,"5min":5.5389999999999997,"15min":5.5389999999999997},"last":1.6379999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.0169999999999999,"15min":1.2470000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.123,"5min":5.5060000000000002,"15min":5.5060000000000002},"last":1.7},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.1200000000000001,"15min":1.29},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2070000000000001,"5min":5.5720000000000001,"15min":5.5720000000000001},"last":1.8049999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:08:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.1850000000000001,"15min":1.3560000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.7140000000000004,"5min":5.6059999999999999,"15min":5.6059999999999999},"last":1.855},{"interface":"front","average":{"1min":1.395,"5min":1.3180000000000001,"15min":1.419},"min":{"1min":0.55100000000000005,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":7.5599999999999996,"5min":7.5599999999999996,"15min":7.5599999999999996},"last":1.946}]},{"osd":9,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.3049999999999999,"15min":1.4350000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.75,"5min":5.7969999999999997,"15min":5.7969999999999997},"last":1.516},{"interface":"front","average":{"1min":1.117,"5min":1.206,"15min":1.4319999999999999},"min":{"1min":0.59699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2229999999999999,"5min":5.6890000000000001,"15min":5.6890000000000001},"last":1.8759999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.23,"15min":1.466},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.909,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":1.613},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.3069999999999999,"15min":1.375},"min":{"1min":0.44500000000000001,"5min":0.437,"15min":0.437},"max":{"1min":2.657,"5min":5.8010000000000002,"15min":5.8010000000000002},"last":1.575}]},{"osd":11,"last update":"Mon Feb 12 16:08:40 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.2609999999999999,"15min":1.462},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.9359999999999999,"5min":7.9660000000000002,"15min":7.9660000000000002},"last":1.748},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.5149999999999999,"15min":1.4350000000000001},"min":{"1min":0.48199999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.1920000000000002,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":1.4750000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313419,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":314268,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93430884,"statfs":{"total":95995035648,"available":95673225216,"internally_reserved":0,"allocated":675840,"data_stored":340001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.208,"15min":0.92300000000000004},"min":{"1min":0.247,"5min":0.222,"15min":0.222},"max":{"1min":6.2359999999999998,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.931,"5min":1.2290000000000001,"15min":0.95199999999999996},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":6.2779999999999996,"5min":6.2779999999999996,"15min":6.2779999999999996},"last":0.97999999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.3819999999999999,"15min":1.0409999999999999},"min":{"1min":0.29399999999999998,"5min":0.223,"15min":0.223},"max":{"1min":6.5629999999999997,"5min":11.423,"15min":11.423},"last":1.117},{"interface":"front","average":{"1min":1.8100000000000001,"5min":1.3919999999999999,"15min":1.125},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.1749999999999998,"5min":7.2290000000000001,"15min":7.2290000000000001},"last":1.3220000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:08:24 2024","interfaces":[{"interface":"back","average":{"1min":1.956,"5min":1.542,"15min":1.163},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":9.4909999999999997,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.1990000000000001,"15min":1.0389999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.9910000000000001,"5min":9.4399999999999995,"15min":9.4399999999999995},"last":0.90000000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3680000000000001,"15min":1.204},"min":{"1min":0.39900000000000002,"5min":0.311,"15min":0.311},"max":{"1min":6.2469999999999999,"5min":9.657,"15min":9.657},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.167,"15min":1.0129999999999999},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.4569999999999999,"5min":9.5530000000000008,"15min":9.5530000000000008},"last":2.5390000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":7.6539999999999999,"15min":3.3410000000000002},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.1059999999999999,"5min":616.34199999999998,"15min":616.34199999999998},"last":1.181},{"interface":"front","average":{"1min":1.71,"5min":7.875,"15min":3.3479999999999999},"min":{"1min":0.42299999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.202,"5min":615.18899999999996,"15min":615.18899999999996},"last":1.4790000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.212,"15min":1.145},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.0219999999999998,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.706},{"interface":"front","average":{"1min":1.905,"5min":1.2989999999999999,"15min":1.2629999999999999},"min":{"1min":0.433,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":6.3979999999999997,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":2.3650000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":7.9809999999999999,"15min":3.5630000000000002},"min":{"1min":0.41899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":3.0510000000000002,"5min":611.79600000000005,"15min":611.79600000000005},"last":1.575},{"interface":"front","average":{"1min":2.0840000000000001,"5min":8.0980000000000008,"15min":3.5499999999999998},"min":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.1520000000000001,"5min":611.83900000000006,"15min":611.83900000000006},"last":1.635}]},{"osd":8,"last update":"Mon Feb 12 16:08:38 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":1.5269999999999999,"15min":1.48},"min":{"1min":0.46700000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":6.4960000000000004,"5min":6.4960000000000004,"15min":6.4960000000000004},"last":2.6560000000000001},{"interface":"front","average":{"1min":2.383,"5min":1.75,"15min":1.5760000000000001},"min":{"1min":0.52000000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":8.5920000000000005,"5min":8.5920000000000005,"15min":8.5920000000000005},"last":1.806}]},{"osd":9,"last update":"Mon Feb 12 16:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.367,"15min":1.401},"min":{"1min":0.47899999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":3.0790000000000002,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":1.929},{"interface":"front","average":{"1min":2.2549999999999999,"5min":1.5369999999999999,"15min":1.504},"min":{"1min":0.52800000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":6.4699999999999998,"5min":6.4699999999999998,"15min":6.4699999999999998},"last":2.7690000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:07:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.51900000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":4.8819999999999997,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":2.1539999999999999},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.5209999999999999,"15min":1.5369999999999999},"min":{"1min":0.51100000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":6.4359999999999999,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":2.714}]}]},{"osd":3,"up_from":26,"seq":111669149891,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538740,"kb_used_data":229228,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93206412,"statfs":{"total":95995035648,"available":95443365888,"internally_reserved":0,"allocated":234729472,"data_stored":234381732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.70499999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":2.0659999999999998,"15min":1.2549999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":8.3680000000000003,"5min":70.031999999999996,"15min":70.031999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.621,"5min":2.25,"15min":1.339},"min":{"1min":0.317,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.3399999999999999,"5min":69.944999999999993,"15min":69.944999999999993},"last":0.95299999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":1.8720000000000001,"15min":1.2290000000000001},"min":{"1min":0.28399999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.3129999999999999,"5min":68.405000000000001,"15min":68.405000000000001},"last":0.872},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.752,"15min":1.238},"min":{"1min":0.31900000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.101,"5min":69.009,"15min":69.009},"last":0.621}]},{"osd":4,"last update":"Mon Feb 12 16:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.96599999999999997,"15min":0.98099999999999998},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.0600000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.86899999999999999,"15min":1.0149999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.8940000000000001,"5min":3.8029999999999999,"15min":3.8029999999999999},"last":1.452}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.92900000000000005,"15min":1.109},"min":{"1min":0.35799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5840000000000001,"5min":4.1840000000000002,"15min":4.1840000000000002},"last":1.125},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.0389999999999999,"15min":1.0820000000000001},"min":{"1min":0.34699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.1179999999999999,"5min":5.4370000000000003,"15min":5.4370000000000003},"last":1.2909999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.04,"15min":1.137},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":4.8899999999999997,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.0609999999999999,"15min":1.2110000000000001},"min":{"1min":0.39200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.171,"5min":4.1429999999999998,"15min":4.1429999999999998},"last":1.2669999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.9950000000000001,"15min":1.544},"min":{"1min":0.433,"5min":0.432,"15min":0.432},"max":{"1min":3.024,"5min":91.718999999999994,"15min":91.718999999999994},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.83699999999999997,"5min":2.0179999999999998,"15min":1.5800000000000001},"min":{"1min":0.52100000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.9249999999999998,"5min":90.367999999999995,"15min":90.367999999999995},"last":1.391}]},{"osd":8,"last update":"Mon Feb 12 16:08:38 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.333,"15min":1.347},"min":{"1min":0.44700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.9489999999999998,"5min":6.5899999999999999,"15min":6.5899999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2010000000000001,"15min":1.391},"min":{"1min":0.505,"5min":0.442,"15min":0.442},"max":{"1min":8.516,"5min":8.516,"15min":8.516},"last":1.0640000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.2210000000000001,"15min":1.4419999999999999},"min":{"1min":0.501,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.9740000000000002,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":1.635},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.333,"15min":1.4710000000000001},"min":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2529999999999999,"5min":6.5350000000000001,"15min":6.5350000000000001},"last":1.5189999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.385,"15min":1.4430000000000001},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8740000000000001,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":1.679},{"interface":"front","average":{"1min":1.304,"5min":1.3220000000000001,"15min":1.4910000000000001},"min":{"1min":0.52300000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.0019999999999998,"5min":5.3419999999999996,"15min":5.3419999999999996},"last":1.5469999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.8200000000000001,"15min":1.6040000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.468,"5min":49.167000000000002,"15min":49.167000000000002},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.75,"15min":1.619},"min":{"1min":0.55800000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":5.0289999999999999,"5min":48.548999999999999,"15min":48.548999999999999},"last":1.171}]}]},{"osd":4,"up_from":32,"seq":137438953660,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297908,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447244,"statfs":{"total":95995035648,"available":95689977856,"internally_reserved":0,"allocated":700416,"data_stored":352676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.78000000000000003,"15min":0.92000000000000004},"min":{"1min":0.29499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.4029999999999996,"5min":4.4029999999999996,"15min":4.4029999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0109999999999999,"15min":0.99199999999999999},"min":{"1min":0.30399999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3869999999999996,"5min":10.754,"15min":10.754},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.643,"15min":1.214},"min":{"1min":0.32800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4690000000000003,"5min":42.226999999999997,"15min":42.226999999999997},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.3240000000000001,"15min":1.1220000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.4459999999999997,"5min":43.689,"15min":43.689},"last":1.3620000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.4430000000000001,"15min":1.1319999999999999},"min":{"1min":0.34200000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.5490000000000004,"5min":41.546999999999997,"15min":41.546999999999997},"last":1.153},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.55,"15min":1.173},"min":{"1min":0.46200000000000002,"5min":0.31,"15min":0.31},"max":{"1min":3.843,"5min":41.57,"15min":41.57},"last":1.5960000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.83499999999999996,"15min":0.97099999999999997},"min":{"1min":0.375,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.234,"5min":0.86899999999999999,"15min":0.97799999999999998},"min":{"1min":0.38200000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.7029999999999998,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":1.468}]},{"osd":5,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.79800000000000004,"15min":0.94699999999999995},"min":{"1min":0.318,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.25,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90300000000000002,"15min":1.026},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3759999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":1.3340000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.81899999999999995,"15min":0.99299999999999999},"min":{"1min":0.41899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.004,"5min":3.004,"15min":3.004},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71999999999999997,"15min":1.02},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.1799999999999999,"5min":1.5249999999999999,"15min":1.9330000000000001},"last":1.431}]},{"osd":7,"last update":"Mon Feb 12 16:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":2.1859999999999999,"15min":1.492},"min":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.9169999999999998,"5min":116.273,"15min":116.273},"last":1.3120000000000001},{"interface":"front","average":{"1min":0.999,"5min":2.3130000000000002,"15min":1.6779999999999999},"min":{"1min":0.53800000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":3.0739999999999998,"5min":116.24299999999999,"15min":116.24299999999999},"last":1.79}]},{"osd":8,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.1599999999999999,"15min":1.399},"min":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.9180000000000001,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":1.7210000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.0760000000000001,"15min":1.3140000000000001},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.742,"5min":4.8609999999999998,"15min":4.8609999999999998},"last":1.569}]},{"osd":9,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":4.2779999999999996,"15min":2.4780000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":4.9050000000000002,"5min":297.55900000000003,"15min":297.55900000000003},"last":1.667},{"interface":"front","average":{"1min":1.2170000000000001,"5min":4.2569999999999997,"15min":2.4180000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":3.867,"5min":297.61900000000003,"15min":297.61900000000003},"last":1.8839999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0620000000000001,"15min":1.3720000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":4.944,"5min":4.944,"15min":4.944},"last":1.7490000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.0429999999999999,"15min":1.3839999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":4.0289999999999999,"5min":4.0289999999999999,"15min":4.0289999999999999},"last":1.6990000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724723,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":374632,"kb_used_data":73312,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93370520,"statfs":{"total":95995035648,"available":95611412480,"internally_reserved":0,"allocated":75071488,"data_stored":74729571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94599999999999995,"15min":0.97399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":3.4950000000000001,"5min":13.031000000000001,"15min":13.031000000000001},"last":0.755},{"interface":"front","average":{"1min":1.248,"5min":1.0129999999999999,"15min":0.98799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5140000000000002,"5min":14.018000000000001,"15min":14.018000000000001},"last":1.3009999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.627,"15min":1.224},"min":{"1min":0.29399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":7.7610000000000001,"5min":33.084000000000003,"15min":33.084000000000003},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.329,"15min":1.095},"min":{"1min":0.34499999999999997,"5min":0.253,"15min":0.253},"max":{"1min":7.8140000000000001,"5min":33.052999999999997,"15min":33.052999999999997},"last":0.85199999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5150000000000001,"5min":1.631,"15min":1.2450000000000001},"min":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":15.673,"5min":26.797000000000001,"15min":26.797000000000001},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3700000000000001,"15min":1.1679999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":8.298,"5min":26.905999999999999,"15min":26.905999999999999},"last":0.92500000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.4079999999999999,"15min":1.177},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.032,"5min":12.991,"15min":12.991},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.238,"15min":1.171},"min":{"1min":0.42899999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.6749999999999998,"5min":13.935,"15min":13.935},"last":1.214}]},{"osd":4,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.96999999999999997,"15min":0.95699999999999996},"min":{"1min":0.33800000000000002,"5min":0.308,"15min":0.308},"max":{"1min":3.5609999999999999,"5min":14.259,"15min":14.259},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":0.97599999999999998,"15min":1.0509999999999999},"min":{"1min":0.41499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.75,"5min":13.268000000000001,"15min":13.268000000000001},"last":0.79000000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":2.4060000000000001,"5min":1.145,"15min":1.1439999999999999},"min":{"1min":0.37,"5min":0.318,"15min":0.318},"max":{"1min":13.148,"5min":13.148,"15min":13.148},"last":1.133},{"interface":"front","average":{"1min":2.2480000000000002,"5min":1.141,"15min":1.175},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":1.347}]},{"osd":7,"last update":"Mon Feb 12 16:07:59 2024","interfaces":[{"interface":"back","average":{"1min":7.2270000000000003,"5min":2.2250000000000001,"15min":1.5429999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":98.971000000000004,"5min":98.971000000000004,"15min":98.971000000000004},"last":1.2450000000000001},{"interface":"front","average":{"1min":6.9850000000000003,"5min":2.2909999999999999,"15min":1.6319999999999999},"min":{"1min":0.53400000000000003,"5min":0.45200000000000001,"15min":0.435},"max":{"1min":99.218000000000004,"5min":99.218000000000004,"15min":99.218000000000004},"last":1.401}]},{"osd":8,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.238,"15min":1.389},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.7839999999999998,"5min":14.372999999999999,"15min":14.372999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.619,"5min":1.264,"15min":1.3580000000000001},"min":{"1min":0.52100000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.7249999999999996,"5min":14.074999999999999,"15min":14.074999999999999},"last":1.5129999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1970000000000001,"5min":1.2749999999999999,"15min":1.4350000000000001},"min":{"1min":0.499,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":13.375999999999999,"5min":13.375999999999999,"15min":13.375999999999999},"last":1.3140000000000001},{"interface":"front","average":{"1min":2.097,"5min":1.268,"15min":1.4930000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":14.132,"5min":14.132,"15min":14.132},"last":1.462}]},{"osd":10,"last update":"Mon Feb 12 16:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.012,"5min":1.333,"15min":1.4359999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":14.214,"5min":14.214,"15min":14.214},"last":1.1859999999999999},{"interface":"front","average":{"1min":2.109,"5min":1.3,"15min":1.5189999999999999},"min":{"1min":0.60399999999999998,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":13.464,"5min":13.464,"15min":13.464},"last":1.365}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:09:01.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T16:09:01.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:01.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:01.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:01.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:01.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:01.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:02.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:02.094 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:02.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:02.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:02.235 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:02.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:02.247 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:02.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:02 smithi169 bash[19604]: audit 2024-02-12T16:09:01.034756+0000 mgr.y (mgr.14146) 706 : audit [DBG] from='client.15099 v1:172.21.15.121:0/1906230784' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:02.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:02 smithi169 bash[19604]: cluster 2024-02-12T16:09:01.298075+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e260: 12 total, 12 up, 12 in 2024-02-12T16:09:02.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:02.374 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:02.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:02 smithi123 bash[19119]: audit 2024-02-12T16:09:01.034756+0000 mgr.y (mgr.14146) 706 : audit [DBG] from='client.15099 v1:172.21.15.121:0/1906230784' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:02 smithi123 bash[19119]: cluster 2024-02-12T16:09:01.298075+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e260: 12 total, 12 up, 12 in 2024-02-12T16:09:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:02 smithi121 bash[18489]: audit 2024-02-12T16:09:01.034756+0000 mgr.y (mgr.14146) 706 : audit [DBG] from='client.15099 v1:172.21.15.121:0/1906230784' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:02 smithi121 bash[18489]: cluster 2024-02-12T16:09:01.298075+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e260: 12 total, 12 up, 12 in 2024-02-12T16:09:02.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:02.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:02.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:02.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:02.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:02.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:02.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:02.792 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:02.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:02.807 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:09:02.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:02.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:02.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:03.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:03.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:03.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:03 smithi121 bash[18489]: cluster 2024-02-12T16:09:01.623814+0000 mgr.y (mgr.14146) 707 : cluster [DBG] pgmap v872: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:03.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:03 smithi121 bash[18489]: cluster 2024-02-12T16:09:01.962187+0000 osd.10 (osd.10) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:03.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:03 smithi121 bash[18489]: cluster 2024-02-12T16:09:01.962306+0000 osd.10 (osd.10) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:03.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:03 smithi121 bash[18489]: cluster 2024-02-12T16:09:02.306335+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e261: 12 total, 12 up, 12 in 2024-02-12T16:09:03.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:03.216 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:03.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:03.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:03.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:03 smithi169 bash[19604]: cluster 2024-02-12T16:09:01.623814+0000 mgr.y (mgr.14146) 707 : cluster [DBG] pgmap v872: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:03 smithi169 bash[19604]: cluster 2024-02-12T16:09:01.962187+0000 osd.10 (osd.10) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:03 smithi169 bash[19604]: cluster 2024-02-12T16:09:01.962306+0000 osd.10 (osd.10) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:03.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:03 smithi169 bash[19604]: cluster 2024-02-12T16:09:02.306335+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e261: 12 total, 12 up, 12 in 2024-02-12T16:09:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:03.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:03 smithi123 bash[19119]: cluster 2024-02-12T16:09:01.623814+0000 mgr.y (mgr.14146) 707 : cluster [DBG] pgmap v872: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:03 smithi123 bash[19119]: cluster 2024-02-12T16:09:01.962187+0000 osd.10 (osd.10) 19 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:03 smithi123 bash[19119]: cluster 2024-02-12T16:09:01.962306+0000 osd.10 (osd.10) 20 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:03.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:03 smithi123 bash[19119]: cluster 2024-02-12T16:09:02.306335+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e261: 12 total, 12 up, 12 in 2024-02-12T16:09:03.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:03.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:03.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:03.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:03.640 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:03.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:03.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:03.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:03.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:03.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:03.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:03.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:04.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:04.060 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:04.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:04.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:04.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:04.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:04.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:04.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:04.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:04.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:04.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:04.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:04.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:04.626 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:04.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:04.687 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:04.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:04.768 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:04.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:04.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:04.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:04.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:05.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:05.046 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:05.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:05.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:05 smithi121 bash[18489]: cluster 2024-02-12T16:09:03.624452+0000 mgr.y (mgr.14146) 708 : cluster [DBG] pgmap v874: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:05.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:05.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:05.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:05.302 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:05 smithi123 bash[19119]: cluster 2024-02-12T16:09:03.624452+0000 mgr.y (mgr.14146) 708 : cluster [DBG] pgmap v874: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:05.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:05.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:05.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:05.470 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:05.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:05.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:05 smithi169 bash[19604]: cluster 2024-02-12T16:09:03.624452+0000 mgr.y (mgr.14146) 708 : cluster [DBG] pgmap v874: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:05.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:05.612 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:05.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:05.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:05.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:05.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:05.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:05.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:05.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:05.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:06.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:06.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:06.039 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:09:06.039 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:09:06.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:06.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:06.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:06.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:06.317 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:06.318 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:06.319 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:06.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:06.325 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-02-12T16:09:06.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:06 smithi121 bash[18489]: cluster 2024-02-12T16:09:05.625210+0000 mgr.y (mgr.14146) 709 : cluster [DBG] pgmap v875: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:06.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:06 smithi121 bash[18489]: audit 2024-02-12T16:09:06.168201+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? v1:172.21.15.121:0/411028773' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:09:06.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:06.460 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:06.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:06.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:06.576 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:06 smithi169 bash[19604]: cluster 2024-02-12T16:09:05.625210+0000 mgr.y (mgr.14146) 709 : cluster [DBG] pgmap v875: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:06.576 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:06 smithi169 bash[19604]: audit 2024-02-12T16:09:06.168201+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? v1:172.21.15.121:0/411028773' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:09:06.603 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:06.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:06.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:06 smithi123 bash[19119]: cluster 2024-02-12T16:09:05.625210+0000 mgr.y (mgr.14146) 709 : cluster [DBG] pgmap v875: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:06.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:06 smithi123 bash[19119]: audit 2024-02-12T16:09:06.168201+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? v1:172.21.15.121:0/411028773' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:09:06.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:06.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-12T16:09:06.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:06.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:06.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:06.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:06.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:06.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:07.015 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:07.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:07.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:07.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:07.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:07 smithi123 bash[19119]: audit 2024-02-12T16:09:06.313117+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? v1:172.21.15.121:0/411028773' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:09:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:07 smithi123 bash[19119]: cluster 2024-02-12T16:09:06.313242+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e262: 12 total, 12 up, 12 in 2024-02-12T16:09:07.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:07.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:07.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:07.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:07 smithi169 bash[19604]: audit 2024-02-12T16:09:06.313117+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? v1:172.21.15.121:0/411028773' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:09:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:07 smithi169 bash[19604]: cluster 2024-02-12T16:09:06.313242+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e262: 12 total, 12 up, 12 in 2024-02-12T16:09:07.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:07 smithi121 bash[18489]: audit 2024-02-12T16:09:06.313117+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? v1:172.21.15.121:0/411028773' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:09:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:07 smithi121 bash[18489]: cluster 2024-02-12T16:09:06.313242+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e262: 12 total, 12 up, 12 in 2024-02-12T16:09:07.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:07.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:07.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:07.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:07.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:07.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:08.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:08.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:08.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:08.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:08.272 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:08.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:08.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:08.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:08.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:08 smithi121 bash[18489]: cluster 2024-02-12T16:09:07.625913+0000 mgr.y (mgr.14146) 710 : cluster [DBG] pgmap v877: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:08.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:08.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:08 smithi123 bash[19119]: cluster 2024-02-12T16:09:07.625913+0000 mgr.y (mgr.14146) 710 : cluster [DBG] pgmap v877: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:08.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:08.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:08.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:08.690 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:08.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:08.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:08 smithi169 bash[19604]: cluster 2024-02-12T16:09:07.625913+0000 mgr.y (mgr.14146) 710 : cluster [DBG] pgmap v877: 17 pgs: 17 active+clean; 0 B data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:08.827 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:08.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:08.909 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-02-12T16:09:08.912 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:09:08.912 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:09:08.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:08.950 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:09:08.950 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:09:08.950 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_51390 2024-02-12T16:09:08.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:09:08.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:09:08.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:08.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:09.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:09.101 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:09.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:09.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:09.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:09.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:09.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:09.339 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:09:09.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:09.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:09:09.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:09 smithi123 bash[19119]: audit 2024-02-12T16:09:09.297979+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.121:0/61090527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:09.402 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:09:09.433 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:09:09.466 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:09:09.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:09.495 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:09:09.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:09.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:09.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:09.519 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:09:09.536 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:09:09.562 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:09:09.590 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:09:09.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:09.613 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:09 smithi169 bash[19604]: audit 2024-02-12T16:09:09.297979+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.121:0/61090527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:09.618 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:09:09.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:09.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:09.645 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.648 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.648 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:09:09.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:09 smithi121 bash[18489]: audit 2024-02-12T16:09:09.297979+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.121:0/61090527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:09.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:09.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:09.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:09.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:09.908 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:09.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:09.950 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 15 3102 3087 192.928 192.938 0.0026395 0.00517523 2024-02-12T16:09:10.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:10.039 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:10.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:10.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:10.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:10.300 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:10.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:10.431 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:10.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:10.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:10.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:10 smithi123 bash[19119]: cluster 2024-02-12T16:09:09.626662+0000 mgr.y (mgr.14146) 711 : cluster [DBG] pgmap v878: 17 pgs: 17 active+clean; 19 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:10 smithi121 bash[18489]: cluster 2024-02-12T16:09:09.626662+0000 mgr.y (mgr.14146) 711 : cluster [DBG] pgmap v878: 17 pgs: 17 active+clean; 19 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:10.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:10.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:10 smithi169 bash[19604]: cluster 2024-02-12T16:09:09.626662+0000 mgr.y (mgr.14146) 711 : cluster [DBG] pgmap v878: 17 pgs: 17 active+clean; 19 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:10.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:10.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:10.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:10.950 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 16 9270 9254 289.164 385.438 0.00232451 0.00345492 2024-02-12T16:09:10.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:10.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:11.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:11.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:11.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:11.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:11.219 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:11.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:11.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:11.346 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:11.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:11.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:11.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:11.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:11.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:11.506 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:09:11.588 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:11.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:11.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:11.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:11.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:11.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:11.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:11.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:11.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:11.950 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 14607 14591 303.954 333.562 0.00368723 0.00328736 2024-02-12T16:09:11.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:12.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:12.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:12.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:12.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:12.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:12.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:12.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:12.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:12.539 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:12.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:12.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:12.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:12.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:12 smithi123 bash[19119]: cluster 2024-02-12T16:09:11.627129+0000 mgr.y (mgr.14146) 712 : cluster [DBG] pgmap v879: 17 pgs: 17 active+clean; 88 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:12.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:12.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:12.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:12.950 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 16 18478 18462 288.447 241.938 0.00470806 0.00346464 2024-02-12T16:09:13.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:13.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:12 smithi169 bash[19604]: cluster 2024-02-12T16:09:11.627129+0000 mgr.y (mgr.14146) 712 : cluster [DBG] pgmap v879: 17 pgs: 17 active+clean; 88 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:13.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:12 smithi121 bash[18489]: cluster 2024-02-12T16:09:11.627129+0000 mgr.y (mgr.14146) 712 : cluster [DBG] pgmap v879: 17 pgs: 17 active+clean; 88 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:13.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:13.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:13.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:13.310 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.340 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:13.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:13.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:13.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:13.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:13.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:13.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:13.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:13.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:13.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:13.854 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:13.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 23065 23049 288.09 286.688 0.00240899 0.00346859 2024-02-12T16:09:13.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:13.982 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:13.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:14.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:14.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:14.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:14.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:14.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:14.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:14.259 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:09:14.259 INFO:teuthology.orchestra.run.smithi121.stdout:{"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1150,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":262,"num_osds":12,"num_up_osds":12,"osd_up_since":1707753503,"num_in_osds":12,"osd_in_since":1707753478,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":4115,"data_bytes":269680640,"bytes_used":4637376512,"bytes_avail":1147303051264,"bytes_total":1151940427776},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.121:8443/"}},"servicemap":{"epoch":29,"modified":"2024-02-12T16:08:59.622886+0000","services":{}},"progress_events":{}} 2024-02-12T16:09:14.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:09:14.348 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:14.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:14.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:14.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:14.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:14.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:14.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:14.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:14.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:14.697 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:09:14.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:14.776 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:14.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:14.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:14.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:14.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:14.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:14 smithi121 bash[18489]: cluster 2024-02-12T16:09:13.627561+0000 mgr.y (mgr.14146) 713 : cluster [DBG] pgmap v880: 17 pgs: 17 active+clean; 257 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:14 smithi121 bash[18489]: audit 2024-02-12T16:09:14.220049+0000 mon.a (mon.0) 1139 : audit [INF] from='client.25040 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:14 smithi121 bash[18489]: audit 2024-02-12T16:09:14.223721+0000 mon.b (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.121:0/1644740763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:14 smithi121 bash[18489]: audit 2024-02-12T16:09:14.253913+0000 mon.a (mon.0) 1140 : audit [DBG] from='client.? v1:172.21.15.121:0/1304427963' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:09:14.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 26221 26205 272.947 197.25 0.00693028 0.00366122 2024-02-12T16:09:15.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:15.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:14 smithi169 bash[19604]: cluster 2024-02-12T16:09:13.627561+0000 mgr.y (mgr.14146) 713 : cluster [DBG] pgmap v880: 17 pgs: 17 active+clean; 257 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:15.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:14 smithi169 bash[19604]: audit 2024-02-12T16:09:14.220049+0000 mon.a (mon.0) 1139 : audit [INF] from='client.25040 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:15.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:14 smithi169 bash[19604]: audit 2024-02-12T16:09:14.223721+0000 mon.b (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.121:0/1644740763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:15.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:14 smithi169 bash[19604]: audit 2024-02-12T16:09:14.253913+0000 mon.a (mon.0) 1140 : audit [DBG] from='client.? v1:172.21.15.121:0/1304427963' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:09:15.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:15.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:14 smithi123 bash[19119]: cluster 2024-02-12T16:09:13.627561+0000 mgr.y (mgr.14146) 713 : cluster [DBG] pgmap v880: 17 pgs: 17 active+clean; 257 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:15.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:14 smithi123 bash[19119]: audit 2024-02-12T16:09:14.220049+0000 mon.a (mon.0) 1139 : audit [INF] from='client.25040 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:15.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:14 smithi123 bash[19119]: audit 2024-02-12T16:09:14.223721+0000 mon.b (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.121:0/1644740763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:15.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:14 smithi123 bash[19119]: audit 2024-02-12T16:09:14.253913+0000 mon.a (mon.0) 1140 : audit [DBG] from='client.? v1:172.21.15.121:0/1304427963' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:09:15.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:15.164 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:15.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:15.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:15.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:15.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:15.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:15.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:15.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:15.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:15.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:15.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:15 smithi169 bash[19604]: cluster 2024-02-12T16:09:14.682678+0000 mon.a (mon.0) 1141 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:09:15.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:15 smithi169 bash[19604]: cluster 2024-02-12T16:09:14.682717+0000 mon.a (mon.0) 1142 : cluster [INF] Cluster is now healthy 2024-02-12T16:09:15.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:15 smithi169 bash[19604]: audit 2024-02-12T16:09:14.687136+0000 mon.a (mon.0) 1143 : audit [INF] from='client.25040 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:09:15.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:15 smithi169 bash[19604]: cluster 2024-02-12T16:09:14.687214+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e263: 12 total, 12 up, 12 in 2024-02-12T16:09:15.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:15.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:15.929 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:15 smithi123 bash[19119]: cluster 2024-02-12T16:09:14.682678+0000 mon.a (mon.0) 1141 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:09:15.929 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:15 smithi123 bash[19119]: cluster 2024-02-12T16:09:14.682717+0000 mon.a (mon.0) 1142 : cluster [INF] Cluster is now healthy 2024-02-12T16:09:15.929 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:15 smithi123 bash[19119]: audit 2024-02-12T16:09:14.687136+0000 mon.a (mon.0) 1143 : audit [INF] from='client.25040 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:09:15.929 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:15 smithi123 bash[19119]: cluster 2024-02-12T16:09:14.687214+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e263: 12 total, 12 up, 12 in 2024-02-12T16:09:15.947 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:15.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 15 29354 29339 261.934 195.875 0.00485747 0.00381619 2024-02-12T16:09:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:15.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:16.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:16.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:15 smithi121 bash[18489]: cluster 2024-02-12T16:09:14.682678+0000 mon.a (mon.0) 1141 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:09:16.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:15 smithi121 bash[18489]: cluster 2024-02-12T16:09:14.682717+0000 mon.a (mon.0) 1142 : cluster [INF] Cluster is now healthy 2024-02-12T16:09:16.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:15 smithi121 bash[18489]: audit 2024-02-12T16:09:14.687136+0000 mon.a (mon.0) 1143 : audit [INF] from='client.25040 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:09:16.063 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:15 smithi121 bash[18489]: cluster 2024-02-12T16:09:14.687214+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e263: 12 total, 12 up, 12 in 2024-02-12T16:09:16.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:16.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:16.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:16.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:16.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:16.268 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-02-12T16:09:16.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:09:16.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:16.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:16.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:16.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:16.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:16.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:16.544 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:09:16.544 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:09:16.544 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:09:16.544 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:09:16.544 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:09:16.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:16.598 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:16.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:16.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:16.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:16.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:16.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:16.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:16.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:16 smithi169 bash[19604]: cluster 2024-02-12T16:09:15.628122+0000 mgr.y (mgr.14146) 714 : cluster [DBG] pgmap v882: 17 pgs: 17 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:16.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:16.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:16.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:16 smithi123 bash[19119]: cluster 2024-02-12T16:09:15.628122+0000 mgr.y (mgr.14146) 714 : cluster [DBG] pgmap v882: 17 pgs: 17 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:16 smithi121 bash[18489]: cluster 2024-02-12T16:09:15.628122+0000 mgr.y (mgr.14146) 714 : cluster [DBG] pgmap v882: 17 pgs: 17 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:16.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 16 33111 33095 258.534 234.75 0.00553751 0.00386638 2024-02-12T16:09:16.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:17.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:17.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:17.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:17.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:17.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:17.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:17.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:17.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-12T16:09:17.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:17.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:17.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:17.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:17.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:17.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:17.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:17.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:17.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 36833 36817 255.653 232.625 0.0034721 0.00390952 2024-02-12T16:09:18.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:18.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:18.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:18.290 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:18.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:18.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:18.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:18.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:09:18.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:18.553 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:18.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:18.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:18.683 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:18.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:18.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:18.795 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:18 smithi121 bash[18489]: cluster 2024-02-12T16:09:17.628542+0000 mgr.y (mgr.14146) 715 : cluster [DBG] pgmap v883: 17 pgs: 17 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:18.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:18 smithi169 bash[19604]: cluster 2024-02-12T16:09:17.628542+0000 mgr.y (mgr.14146) 715 : cluster [DBG] pgmap v883: 17 pgs: 17 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:18.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:18.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:18.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:18.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:18.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 16 40356 40340 252.105 220.188 0.00363365 0.00396479 2024-02-12T16:09:19.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:19.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:18 smithi123 bash[19119]: cluster 2024-02-12T16:09:17.628542+0000 mgr.y (mgr.14146) 715 : cluster [DBG] pgmap v883: 17 pgs: 17 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:09:19.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:19.210 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:19.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:19.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:19.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:19.476 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:19.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:19.604 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:19.708 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:19.737 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:19.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:19.865 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:19.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 43782 43766 248.651 214.125 0.00211567 0.00401957 2024-02-12T16:09:19.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:19.994 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:19.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:20.030 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:20.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:20.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:20.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:20.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:20.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:20.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:20.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:20.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:20.378 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:20.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:20.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:20.504 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:20.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:20.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:20.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:20.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:20.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:20.763 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:20.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:20.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:20.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:20 smithi169 bash[19604]: cluster 2024-02-12T16:09:19.629005+0000 mgr.y (mgr.14146) 716 : cluster [DBG] pgmap v884: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 215 MiB/s wr, 3.44k op/s 2024-02-12T16:09:20.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:20.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:20.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 47055 47039 244.975 204.562 0.00875948 0.0040806 2024-02-12T16:09:20.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:21.024 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:21.044 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:21.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:21.136 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:20 smithi123 bash[19119]: cluster 2024-02-12T16:09:19.629005+0000 mgr.y (mgr.14146) 716 : cluster [DBG] pgmap v884: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 215 MiB/s wr, 3.44k op/s 2024-02-12T16:09:21.158 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:20 smithi121 bash[18489]: cluster 2024-02-12T16:09:19.629005+0000 mgr.y (mgr.14146) 716 : cluster [DBG] pgmap v884: 17 pgs: 17 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 215 MiB/s wr, 3.44k op/s 2024-02-12T16:09:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:21.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:21.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:21.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:21.420 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:21.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:21.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:21.687 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:21.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:21.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:21.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:21.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:21.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 50156 50140 241.039 193.812 0.00262891 0.00414717 2024-02-12T16:09:22.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:22.076 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:22.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:22.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:22.272 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:09:22.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:22.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:22.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:22.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:22.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:22.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:22.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:22.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:22.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:22.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:22.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:22 smithi123 bash[19119]: cluster 2024-02-12T16:09:21.629366+0000 mgr.y (mgr.14146) 717 : cluster [DBG] pgmap v885: 17 pgs: 17 active+clean; 2.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.88k op/s 2024-02-12T16:09:22.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 53031 53015 236.655 179.688 0.00431484 0.00422387 2024-02-12T16:09:22.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:22.985 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:22.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:22.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:22 smithi169 bash[19604]: cluster 2024-02-12T16:09:21.629366+0000 mgr.y (mgr.14146) 717 : cluster [DBG] pgmap v885: 17 pgs: 17 active+clean; 2.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.88k op/s 2024-02-12T16:09:23.078 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:23.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:23.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:23.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:23.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:22 smithi121 bash[18489]: cluster 2024-02-12T16:09:21.629366+0000 mgr.y (mgr.14146) 717 : cluster [DBG] pgmap v885: 17 pgs: 17 active+clean; 2.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.88k op/s 2024-02-12T16:09:23.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:23.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:23.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:23.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:23.373 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:23.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:23.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:23.504 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:23.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:23.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:23.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:23.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:23.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:23.773 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:23.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:23.803 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:23 smithi121 bash[18489]: cluster 2024-02-12T16:09:23.296835+0000 osd.5 (osd.5) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:23.803 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:23 smithi121 bash[18489]: cluster 2024-02-12T16:09:23.296883+0000 osd.5 (osd.5) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:23.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:23.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:23.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:23.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 55453 55437 230.969 151.375 0.00471437 0.00432807 2024-02-12T16:09:24.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:24.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:23 smithi169 bash[19604]: cluster 2024-02-12T16:09:23.296835+0000 osd.5 (osd.5) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:24.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:23 smithi169 bash[19604]: cluster 2024-02-12T16:09:23.296883+0000 osd.5 (osd.5) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:24.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:24.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:24.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:23 smithi123 bash[19119]: cluster 2024-02-12T16:09:23.296835+0000 osd.5 (osd.5) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:24.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:23 smithi123 bash[19119]: cluster 2024-02-12T16:09:23.296883+0000 osd.5 (osd.5) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:24.166 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:24.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:24.304 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:24.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:24.433 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:24.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:24.564 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:24.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:24.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:24.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:24 smithi169 bash[19604]: cluster 2024-02-12T16:09:23.629806+0000 mgr.y (mgr.14146) 718 : cluster [DBG] pgmap v886: 17 pgs: 17 active+clean; 2.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 245 MiB/s wr, 3.92k op/s 2024-02-12T16:09:24.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:24 smithi169 bash[19604]: cluster 2024-02-12T16:09:23.716870+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e264: 12 total, 12 up, 12 in 2024-02-12T16:09:24.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:24.829 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:24.928 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:24 smithi121 bash[18489]: cluster 2024-02-12T16:09:23.629806+0000 mgr.y (mgr.14146) 718 : cluster [DBG] pgmap v886: 17 pgs: 17 active+clean; 2.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 245 MiB/s wr, 3.92k op/s 2024-02-12T16:09:24.928 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:24 smithi121 bash[18489]: cluster 2024-02-12T16:09:23.716870+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e264: 12 total, 12 up, 12 in 2024-02-12T16:09:24.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:24.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 16 58012 57996 226.529 159.938 0.00294302 0.00441316 2024-02-12T16:09:24.958 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:24.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:25.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:25.090 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:25.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:24 smithi123 bash[19119]: cluster 2024-02-12T16:09:23.629806+0000 mgr.y (mgr.14146) 718 : cluster [DBG] pgmap v886: 17 pgs: 17 active+clean; 2.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 245 MiB/s wr, 3.92k op/s 2024-02-12T16:09:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:24 smithi123 bash[19119]: cluster 2024-02-12T16:09:23.716870+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e264: 12 total, 12 up, 12 in 2024-02-12T16:09:25.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:25.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:25.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:25.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:25.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:25.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:25.471 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:25.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:25.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:25.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:25.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:25.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:25.728 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:25 smithi123 bash[19119]: cluster 2024-02-12T16:09:24.565334+0000 osd.9 (osd.9) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:25.728 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:25 smithi123 bash[19119]: cluster 2024-02-12T16:09:24.565391+0000 osd.9 (osd.9) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:25.728 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:25 smithi123 bash[19119]: cluster 2024-02-12T16:09:24.720773+0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e265: 12 total, 12 up, 12 in 2024-02-12T16:09:25.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:25.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:25.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:25.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:25.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:25.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:25.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:25 smithi121 bash[18489]: cluster 2024-02-12T16:09:24.565334+0000 osd.9 (osd.9) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:25.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:25 smithi121 bash[18489]: cluster 2024-02-12T16:09:24.565391+0000 osd.9 (osd.9) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:25.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:25 smithi121 bash[18489]: cluster 2024-02-12T16:09:24.720773+0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e265: 12 total, 12 up, 12 in 2024-02-12T16:09:25.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 60284 60268 221.556 142 0.00334584 0.00451048 2024-02-12T16:09:25.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:26.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:26.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:25 smithi169 bash[19604]: cluster 2024-02-12T16:09:24.565334+0000 osd.9 (osd.9) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:25 smithi169 bash[19604]: cluster 2024-02-12T16:09:24.565391+0000 osd.9 (osd.9) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:25 smithi169 bash[19604]: cluster 2024-02-12T16:09:24.720773+0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e265: 12 total, 12 up, 12 in 2024-02-12T16:09:26.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:26.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:26.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:26.277 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:26.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:26.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:26.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:26.512 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:26.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:26.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:26.675 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:26.779 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:26.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:26 smithi123 bash[19119]: cluster 2024-02-12T16:09:25.630196+0000 mgr.y (mgr.14146) 719 : cluster [DBG] pgmap v889: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 230 MiB/s wr, 3.67k op/s 2024-02-12T16:09:26.808 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:26.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:26.939 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:26 smithi121 bash[18489]: cluster 2024-02-12T16:09:25.630196+0000 mgr.y (mgr.14146) 719 : cluster [DBG] pgmap v889: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 230 MiB/s wr, 3.67k op/s 2024-02-12T16:09:26.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:26.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:26.951 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 62924 62908 218.413 165 0.00672114 0.0045769 2024-02-12T16:09:27.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:26 smithi169 bash[19604]: cluster 2024-02-12T16:09:25.630196+0000 mgr.y (mgr.14146) 719 : cluster [DBG] pgmap v889: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 230 MiB/s wr, 3.67k op/s 2024-02-12T16:09:27.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:27.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:27.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:27.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:27.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:27.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:27.339 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:27.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:27.434 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:09:27.434 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:09:27.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:09:27.434 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:09:27.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22194 2.d 2:b7672c5f:::benchmark_data_smithi169_42458_object22193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.747440+0000", 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.68117848200001, 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20546419899999999, 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22194 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747440+0000", 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747440+0000", 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747437+0000", 2024-02-12T16:09:27.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9731e-05 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747467+0000", 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0679999999999999e-05 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747477+0000", 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4320000000000004e-06 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747484+0000", 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3286000000000004e-05 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747547+0000", 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020886900000000001 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747756+0000", 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011899 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.747875+0000", 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00090532600000000003 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.748780+0000", 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20384081600000001 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.952621+0000", 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016218600000000001 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.952783+0000", 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000120504 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.952904+0000", 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20546419899999999 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22205 2.d 2:bcb98dda:::benchmark_data_smithi169_42458_object22204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.750103+0000", 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.67851484800002, 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20371917, 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.441 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22205 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750103+0000", 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750103+0000", 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750101+0000", 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9635999999999999e-05 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750131+0000", 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4192999999999999e-05 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750145+0000", 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3200000000000002e-06 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750152+0000", 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2228999999999997e-05 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750205+0000", 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012974500000000001 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750334+0000", 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000179008 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750513+0000", 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002421613 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.752935+0000", 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20064776600000001 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.953583+0000", 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000127575 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.953710+0000", 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112139 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.953822+0000", 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20371917 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22209 2.d 2:b7fad188:::benchmark_data_smithi169_42458_object22208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.750430+0000", 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.67818843200001, 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.204339097, 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.447 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22209 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750430+0000", 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750430+0000", 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:27.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750428+0000", 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.262e-05 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750451+0000", 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1861e-05 2024-02-12T16:09:27.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750473+0000", 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9240000000000007e-06 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.750481+0000", 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086805300000000003 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751349+0000", 2024-02-12T16:09:27.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013502800000000001 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751484+0000", 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000156904 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751641+0000", 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001303654 2024-02-12T16:09:27.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.752944+0000", 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20161942199999999 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.954564+0000", 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7429999999999996e-05 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.954641+0000", 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000127701 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.954769+0000", 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.204339097 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22217 2.d 2:bb6fa0d0:::benchmark_data_smithi169_42458_object22216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.751902+0000", 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.67671597399999, 2024-02-12T16:09:27.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20333037100000001, 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22217 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751902+0000", 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751902+0000", 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751901+0000", 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0468000000000001e-05 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751921+0000", 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0624e-05 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751932+0000", 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7339999999999999e-06 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.751936+0000", 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000246557 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.752183+0000", 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000125406 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.752308+0000", 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017124800000000001 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.752480+0000", 2024-02-12T16:09:27.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00063109799999999997 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.753111+0000", 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.201868879 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.954980+0000", 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014715400000000001 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955127+0000", 2024-02-12T16:09:27.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010582900000000001 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955233+0000", 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20333037100000001 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22246 2.d 2:b71b0264:::benchmark_data_smithi169_42458_object22245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.758517+0000", 2024-02-12T16:09:27.459 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.67010146500002, 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197204398, 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22246 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758517+0000", 2024-02-12T16:09:27.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758517+0000", 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758514+0000", 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4301999999999999e-05 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758538+0000", 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1715999999999999e-05 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758550+0000", 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5280000000000003e-06 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758556+0000", 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015616800000000001 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758712+0000", 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013525200000000001 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.758847+0000", 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017673799999999999 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.759024+0000", 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014652199999999999 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.760489+0000", 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19509021700000001 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955579+0000", 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9352999999999997e-05 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955659+0000", 2024-02-12T16:09:27.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2460999999999997e-05 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955721+0000", 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197204398 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22263 2.d 2:b61bd371:::benchmark_data_smithi169_42458_object22262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.763379+0000", 2024-02-12T16:09:27.465 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.665238633, 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19258627, 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22263 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763379+0000", 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763379+0000", 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763377+0000", 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6974e-05 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763404+0000", 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1455e-05 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763416+0000", 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2979999999999997e-06 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763422+0000", 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9791999999999999e-05 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763492+0000", 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014050299999999999 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763632+0000", 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000170328 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763802+0000", 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017576289999999999 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.765560+0000", 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19027650099999999 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955837+0000", 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010784 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955944+0000", 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1336999999999999e-05 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.955966+0000", 2024-02-12T16:09:27.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19258627 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22264 2.d 2:b6ec6274:::benchmark_data_smithi169_42458_object22263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.763520+0000", 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.665098484, 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19327575899999999, 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.471 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22264 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763520+0000", 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763520+0000", 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763518+0000", 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5629000000000001e-05 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763544+0000", 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2129e-05 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763556+0000", 2024-02-12T16:09:27.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0779999999999997e-06 2024-02-12T16:09:27.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.763560+0000", 2024-02-12T16:09:27.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001049861 2024-02-12T16:09:27.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.764610+0000", 2024-02-12T16:09:27.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013516400000000001 2024-02-12T16:09:27.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.764745+0000", 2024-02-12T16:09:27.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014993600000000001 2024-02-12T16:09:27.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.764895+0000", 2024-02-12T16:09:27.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00081220299999999995 2024-02-12T16:09:27.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.765707+0000", 2024-02-12T16:09:27.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19103247500000001 2024-02-12T16:09:27.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.956739+0000", 2024-02-12T16:09:27.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1202999999999999e-05 2024-02-12T16:09:27.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.956771+0000", 2024-02-12T16:09:27.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4689999999999999e-05 2024-02-12T16:09:27.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.956795+0000", 2024-02-12T16:09:27.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19327575899999999 2024-02-12T16:09:27.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.482 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.482 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.482 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22284 2.d 2:bcb7a6ef:::benchmark_data_smithi169_42458_object22283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.482 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.768524+0000", 2024-02-12T16:09:27.483 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.66009377799998, 2024-02-12T16:09:27.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18837168000000001, 2024-02-12T16:09:27.483 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.483 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.483 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.483 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.483 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.484 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22284 2024-02-12T16:09:27.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.484 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768524+0000", 2024-02-12T16:09:27.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768524+0000", 2024-02-12T16:09:27.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:27.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768522+0000", 2024-02-12T16:09:27.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6169999999999998e-05 2024-02-12T16:09:27.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768548+0000", 2024-02-12T16:09:27.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4599e-05 2024-02-12T16:09:27.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768563+0000", 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1870000000000002e-06 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768569+0000", 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8801999999999998e-05 2024-02-12T16:09:27.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768618+0000", 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011217000000000001 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768730+0000", 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013886899999999999 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.768869+0000", 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010823759999999999 2024-02-12T16:09:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.769951+0000", 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18687904499999999 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.956830+0000", 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5861000000000001e-05 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.956866+0000", 2024-02-12T16:09:27.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9750000000000001e-05 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.956896+0000", 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18837168000000001 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22320 2.d 2:bee78249:::benchmark_data_smithi169_42458_object22319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.777889+0000", 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.650728921, 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18112668600000001, 2024-02-12T16:09:27.491 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22320 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.777889+0000", 2024-02-12T16:09:27.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.777889+0000", 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.777886+0000", 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0127000000000001e-05 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.777917+0000", 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.624e-05 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.777933+0000", 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4819999999999997e-06 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.777940+0000", 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5903999999999998e-05 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778006+0000", 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013230699999999999 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778139+0000", 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015978699999999999 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778298+0000", 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001524678 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.779823+0000", 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17909944 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.958922+0000", 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4264000000000001e-05 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.958977+0000", 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9153000000000003e-05 2024-02-12T16:09:27.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959016+0000", 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18112668600000001 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22321 2.d 2:bbae9c32:::benchmark_data_smithi169_42458_object22320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.778190+0000", 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.65042847799998, 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.180965402, 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.498 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22321 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778190+0000", 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778190+0000", 2024-02-12T16:09:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778188+0000", 2024-02-12T16:09:27.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1882000000000002e-05 2024-02-12T16:09:27.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778210+0000", 2024-02-12T16:09:27.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3570000000000004e-06 2024-02-12T16:09:27.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778219+0000", 2024-02-12T16:09:27.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8700000000000003e-06 2024-02-12T16:09:27.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.778226+0000", 2024-02-12T16:09:27.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00084464999999999996 2024-02-12T16:09:27.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.779071+0000", 2024-02-12T16:09:27.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106094 2024-02-12T16:09:27.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.779177+0000", 2024-02-12T16:09:27.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017727699999999999 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.779354+0000", 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00062728299999999996 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.779981+0000", 2024-02-12T16:09:27.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17907836699999999 2024-02-12T16:09:27.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959060+0000", 2024-02-12T16:09:27.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6100000000000002e-05 2024-02-12T16:09:27.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959116+0000", 2024-02-12T16:09:27.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9209999999999999e-05 2024-02-12T16:09:27.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959155+0000", 2024-02-12T16:09:27.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.180965402 2024-02-12T16:09:27.508 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.508 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.508 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22328 2.d 2:b6e057ce:::benchmark_data_smithi169_42458_object22327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.780549+0000", 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.64806923100002, 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.178783093, 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22328 2024-02-12T16:09:27.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780549+0000", 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780549+0000", 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780547+0000", 2024-02-12T16:09:27.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6106999999999999e-05 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780573+0000", 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6138e-05 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780589+0000", 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8550000000000002e-06 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780597+0000", 2024-02-12T16:09:27.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5927999999999998e-05 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780643+0000", 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013381600000000001 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780776+0000", 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013329100000000001 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.780910+0000", 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015820039999999999 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.782492+0000", 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.176771818 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959264+0000", 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9692e-05 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959293+0000", 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8736000000000003e-05 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959332+0000", 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.178783093 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22332 2.d 2:ba0e1ab2:::benchmark_data_smithi169_42458_object22331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.781650+0000", 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.64696803599998, 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.177809516, 2024-02-12T16:09:27.514 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22332 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781650+0000", 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781650+0000", 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:27.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781648+0000", 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8541e-05 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781677+0000", 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4167e-05 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781691+0000", 2024-02-12T16:09:27.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3360000000000002e-06 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781696+0000", 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6949000000000001e-05 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781743+0000", 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7105e-05 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781831+0000", 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109128 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.781940+0000", 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069428700000000003 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.782634+0000", 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.176739908 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959374+0000", 2024-02-12T16:09:27.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0939999999999998e-05 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959415+0000", 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4799000000000003e-05 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959460+0000", 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.177809516 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.519 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22342 2.d 2:b006609b:::benchmark_data_smithi169_42458_object22341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.786677+0000", 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.641940858, 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17337311799999999, 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22342 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786677+0000", 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786677+0000", 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786675+0000", 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2427999999999999e-05 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786698+0000", 2024-02-12T16:09:27.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1629e-05 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786710+0000", 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3210000000000004e-06 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786715+0000", 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9931e-05 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786765+0000", 2024-02-12T16:09:27.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5084e-05 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786860+0000", 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100289 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.786960+0000", 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069220399999999997 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.787652+0000", 2024-02-12T16:09:27.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17223228900000001 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959885+0000", 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6320999999999999e-05 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.959931+0000", 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119417 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960050+0000", 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17337311799999999 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22423 2.d 2:b7c27d3c:::benchmark_data_smithi169_42458_object22422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.830312+0000", 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.59830639400002, 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12987616099999999, 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.525 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22423 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830312+0000", 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830312+0000", 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830309+0000", 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4415000000000001e-05 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830354+0000", 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1574e-05 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830365+0000", 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5909999999999998e-06 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830373+0000", 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7413000000000002e-05 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830420+0000", 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8775000000000003e-05 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830509+0000", 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113502 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.830623+0000", 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079859600000000005 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.831421+0000", 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12866395899999999 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960085+0000", 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5526999999999999e-05 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960141+0000", 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7117999999999998e-05 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960188+0000", 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12987616099999999 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22426 2.d 2:bb87a8df:::benchmark_data_smithi169_42458_object22425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.832235+0000", 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.59638265299998, 2024-02-12T16:09:27.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12814287499999999, 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22426 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832235+0000", 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832235+0000", 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832234+0000", 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5129000000000002e-05 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832269+0000", 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1865000000000001e-05 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832281+0000", 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6760000000000001e-06 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832288+0000", 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7277000000000001e-05 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832345+0000", 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011631 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832461+0000", 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016181799999999999 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.832623+0000", 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00074569499999999999 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.833369+0000", 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12690837099999999 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960277+0000", 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0911e-05 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960338+0000", 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0213999999999998e-05 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960378+0000", 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12814287499999999 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:22430 2.d 2:b973f1ea:::benchmark_data_smithi169_42458_object22429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:22.837491+0000", 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 304.591126686, 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12305055099999999, 2024-02-12T16:09:27.536 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 22430 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837491+0000", 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837491+0000", 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837489+0000", 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7174e-05 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837526+0000", 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0855e-05 2024-02-12T16:09:27.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837537+0000", 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8940000000000001e-06 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837542+0000", 2024-02-12T16:09:27.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9240000000000001e-05 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837571+0000", 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3139000000000005e-05 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837655+0000", 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9628000000000002e-05 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.837754+0000", 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00067527099999999997 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.838429+0000", 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.121989062 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960419+0000", 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0129999999999997e-05 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960459+0000", 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3313999999999995e-05 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:22.960542+0000", 2024-02-12T16:09:27.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12305055099999999 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:345195 2.13 2:ccbe5bfd:::benchmark_data_smithi169_42458_object172161:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e236)", 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:22.722607+0000", 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 64.706010770000006, 2024-02-12T16:09:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.105252624, 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 345195 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722607+0000", 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722607+0000", 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722607+0000", 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.858e-06 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722610+0000", 2024-02-12T16:09:27.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9049999999999999e-06 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722614+0000", 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7939999999999998e-06 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722616+0000", 2024-02-12T16:09:27.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.089845922999999994 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.812462+0000", 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100364 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.812563+0000", 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00057328499999999998 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.813136+0000", 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014531702000000001 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827668+0000", 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150443 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827818+0000", 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2065999999999998e-05 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827840+0000", 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9779000000000002e-05 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827860+0000", 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.105252624 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:345244 2.13 2:c9bacafa:::benchmark_data_smithi169_42458_object172210:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e236)", 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:22.722619+0000", 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 64.705998942999997, 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.105333522, 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:09:27.553 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 345244 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722619+0000", 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722619+0000", 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722619+0000", 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.142e-06 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722621+0000", 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7299999999999996e-06 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722626+0000", 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5210000000000001e-06 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.722628+0000", 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.093443746999999994 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.816072+0000", 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104281 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.816176+0000", 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047668800000000002 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.820943+0000", 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067641280000000003 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827707+0000", 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016137700000000001 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827869+0000", 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6285999999999998e-05 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827905+0000", 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7803999999999998e-05 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:22.827953+0000", 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.105333522 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15108.0:24083 3.0 e262/256 3:0f0d65c3:::benchmark_data_smithi169_51390_object24082:head v 262'1521, mlcod=262'1519)", 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:14.207659+0000", 2024-02-12T16:09:27.558 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.220959521999999, 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24796887400000001, 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207659+0000", 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207659+0000", 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207657+0000", 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5877999999999999e-05 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207683+0000", 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0091e-05 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207693+0000", 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2389999999999999e-06 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207699+0000", 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8883e-05 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207738+0000", 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1916000000000001e-05 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207750+0000", 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.247786915 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.455537+0000", 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0620999999999999e-05 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.455627+0000", 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24796887400000001 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15108.0:24091 3.2 e262/256 3:45d66431:::benchmark_data_smithi169_51390_object24090:head v 262'1466, mlcod=262'1464)", 2024-02-12T16:09:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:14.209311+0000", 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.219306644, 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24952506399999999, 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209311+0000", 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209311+0000", 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209310+0000", 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9494999999999998e-05 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209380+0000", 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24128054600000001 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450660+0000", 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.385e-05 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450684+0000", 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012295700000000001 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450807+0000", 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0756000000000001e-05 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450848+0000", 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079541049999999995 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.458802+0000", 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4740000000000003e-05 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.458837+0000", 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24952506399999999 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:27.566 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:09:27.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:27.599 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:27.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:27.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:27.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:27.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:27.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:27.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:27.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:27.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 65154 65138 214.253 139.375 0.00937988 0.00466591 2024-02-12T16:09:27.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:27.985 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:27.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:28.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:28.116 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:28.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:28.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:28.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:28.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:28.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-12T16:09:28.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:28.382 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:28.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:28.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:28.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:28.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:28.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:28.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:28.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:28.732 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:28 smithi169 bash[19604]: cluster 2024-02-12T16:09:27.630623+0000 mgr.y (mgr.14146) 720 : cluster [DBG] pgmap v890: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.89k op/s 2024-02-12T16:09:28.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:28.769 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:28.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:28.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:28.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:28.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:28.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:28 smithi121 bash[18489]: cluster 2024-02-12T16:09:27.630623+0000 mgr.y (mgr.14146) 720 : cluster [DBG] pgmap v890: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.89k op/s 2024-02-12T16:09:28.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:09:28.949632 min lat: 0.000952532 max lat: 0.293457 avg lat: 0.0047325 2024-02-12T16:09:28.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:09:28.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 16 67616 67600 211.233 153.875 0.00527813 0.0047325 2024-02-12T16:09:29.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:29.009 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:28 smithi123 bash[19119]: cluster 2024-02-12T16:09:27.630623+0000 mgr.y (mgr.14146) 720 : cluster [DBG] pgmap v890: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.89k op/s 2024-02-12T16:09:29.024 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:29.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:29.152 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:29.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:29.286 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:29.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:29.426 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:29.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:29.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:29.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:29.687 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:29.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:29.817 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:29.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:29.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:29.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:29.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 69360 69344 206.364 109 0.00628186 0.00484355 2024-02-12T16:09:30.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:30.073 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:30.103 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:09:30.104 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:09:30.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:30.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:30.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:30.346 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:30.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:30.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:30.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:30.604 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:30.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:30.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:30.760 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:09:30.785 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:09:30.825 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:09:30.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:30.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:30 smithi123 bash[19119]: cluster 2024-02-12T16:09:29.631064+0000 mgr.y (mgr.14146) 721 : cluster [DBG] pgmap v891: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-02-12T16:09:30.841 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:30 smithi123 bash[19119]: audit 2024-02-12T16:09:30.084463+0000 mgr.y (mgr.14146) 722 : audit [DBG] from='client.25043 v1:172.21.15.121:0/1376182731' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:30.841 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:30 smithi123 bash[19119]: audit 2024-02-12T16:09:30.725271+0000 mon.b (mon.2) 106 : audit [DBG] from='client.? v1:172.21.15.121:0/3619395102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:30.855 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:09:30.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:30.885 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:09:30.911 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:09:30.939 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:09:30.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 70757 70741 200.953 87.3125 0.00460217 0.00497434 2024-02-12T16:09:30.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:09:30.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:30.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:30.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:30.995 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:09:31.024 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:09:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:30 smithi169 bash[19604]: cluster 2024-02-12T16:09:29.631064+0000 mgr.y (mgr.14146) 721 : cluster [DBG] pgmap v891: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-02-12T16:09:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:30 smithi169 bash[19604]: audit 2024-02-12T16:09:30.084463+0000 mgr.y (mgr.14146) 722 : audit [DBG] from='client.25043 v1:172.21.15.121:0/1376182731' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:30 smithi169 bash[19604]: audit 2024-02-12T16:09:30.725271+0000 mon.b (mon.2) 106 : audit [DBG] from='client.? v1:172.21.15.121:0/3619395102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:31.053 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:09:31.082 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:09:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:09:31.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:31.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:31.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:31.150 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:30 smithi121 bash[18489]: cluster 2024-02-12T16:09:29.631064+0000 mgr.y (mgr.14146) 721 : cluster [DBG] pgmap v891: 17 pgs: 17 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-02-12T16:09:31.150 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:30 smithi121 bash[18489]: audit 2024-02-12T16:09:30.084463+0000 mgr.y (mgr.14146) 722 : audit [DBG] from='client.25043 v1:172.21.15.121:0/1376182731' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:31.150 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:30 smithi121 bash[18489]: audit 2024-02-12T16:09:30.725271+0000 mon.b (mon.2) 106 : audit [DBG] from='client.? v1:172.21.15.121:0/3619395102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:31.227 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:31.252 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:31.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:31.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:31.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:31.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:31.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:31.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:31.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:31.530 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":891,"stamp":"2024-02-12T16:09:29.630814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4174118912,"num_objects":63692,"num_object_clones":0,"num_object_copies":127384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63692,"num_write_kb":4076288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63692,"ondisk_log_size":63692,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":11968596,"kb_used_data":8303604,"kb_used_omap":0,"kb_used_meta":3664896,"kb_avail":1112973228,"statfs":{"total":1151940427776,"available":1139684585472,"internally_reserved":0,"allocated":8502890496,"data_stored":8498214167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3752853504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,8,7,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1537736704,"num_objects":23464,"num_object_clones":0,"num_object_copies":46928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23464,"num_write_kb":1501696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001630"},"pg_stats":[{"pgid":"3.f","version":"265'4229","reported_seq":4245,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:28.661591+0000","last_change":"2024-02-12T16:08:56.026943+0000","last_active":"2024-02-12T16:09:28.661591+0000","last_peered":"2024-02-12T16:09:28.661591+0000","last_clean":"2024-02-12T16:09:28.661591+0000","last_became_active":"2024-02-12T16:08:56.025977+0000","last_became_peered":"2024-02-12T16:08:56.025977+0000","last_unstale":"2024-02-12T16:09:28.661591+0000","last_undegraded":"2024-02-12T16:09:28.661591+0000","last_fullsized":"2024-02-12T16:09:28.661591+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4229,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":277151744,"num_objects":4229,"num_object_clones":0,"num_object_copies":8458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4229,"num_write_kb":270656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"265'3625","reported_seq":3640,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:24.948520+0000","last_change":"2024-02-12T16:08:56.029433+0000","last_active":"2024-02-12T16:09:24.948520+0000","last_peered":"2024-02-12T16:09:24.948520+0000","last_clean":"2024-02-12T16:09:24.948520+0000","last_became_active":"2024-02-12T16:08:56.029081+0000","last_became_peered":"2024-02-12T16:08:56.029081+0000","last_unstale":"2024-02-12T16:09:24.948520+0000","last_undegraded":"2024-02-12T16:09:24.948520+0000","last_fullsized":"2024-02-12T16:09:24.948520+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3625,"ondisk_log_size":3625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":237568000,"num_objects":3625,"num_object_clones":0,"num_object_copies":7250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3625,"num_write_kb":232000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"265'4101","reported_seq":4117,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:28.656042+0000","last_change":"2024-02-12T16:08:56.026138+0000","last_active":"2024-02-12T16:09:28.656042+0000","last_peered":"2024-02-12T16:09:28.656042+0000","last_clean":"2024-02-12T16:09:28.656042+0000","last_became_active":"2024-02-12T16:08:56.025860+0000","last_became_peered":"2024-02-12T16:08:56.025860+0000","last_unstale":"2024-02-12T16:09:28.656042+0000","last_undegraded":"2024-02-12T16:09:28.656042+0000","last_fullsized":"2024-02-12T16:09:28.656042+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4101,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":268763136,"num_objects":4101,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4101,"num_write_kb":262464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"265'3943","reported_seq":3959,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:26.891892+0000","last_change":"2024-02-12T16:08:56.027997+0000","last_active":"2024-02-12T16:09:26.891892+0000","last_peered":"2024-02-12T16:09:26.891892+0000","last_clean":"2024-02-12T16:09:26.891892+0000","last_became_active":"2024-02-12T16:08:56.027658+0000","last_became_peered":"2024-02-12T16:08:56.027658+0000","last_unstale":"2024-02-12T16:09:26.891892+0000","last_undegraded":"2024-02-12T16:09:26.891892+0000","last_fullsized":"2024-02-12T16:09:26.891892+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3943,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":258408448,"num_objects":3943,"num_object_clones":0,"num_object_copies":7886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3943,"num_write_kb":252352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"265'4243","reported_seq":4258,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:29.188902+0000","last_change":"2024-02-12T16:08:56.021048+0000","last_active":"2024-02-12T16:09:29.188902+0000","last_peered":"2024-02-12T16:09:29.188902+0000","last_clean":"2024-02-12T16:09:29.188902+0000","last_became_active":"2024-02-12T16:08:56.020761+0000","last_became_peered":"2024-02-12T16:08:56.020761+0000","last_unstale":"2024-02-12T16:09:29.188902+0000","last_undegraded":"2024-02-12T16:09:29.188902+0000","last_fullsized":"2024-02-12T16:09:29.188902+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4243,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":278069248,"num_objects":4243,"num_object_clones":0,"num_object_copies":8486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4243,"num_write_kb":271552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"265'4219","reported_seq":4235,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:28.909037+0000","last_change":"2024-02-12T16:08:56.027375+0000","last_active":"2024-02-12T16:09:28.909037+0000","last_peered":"2024-02-12T16:09:28.909037+0000","last_clean":"2024-02-12T16:09:28.909037+0000","last_became_active":"2024-02-12T16:08:56.027087+0000","last_became_peered":"2024-02-12T16:08:56.027087+0000","last_unstale":"2024-02-12T16:09:28.909037+0000","last_undegraded":"2024-02-12T16:09:28.909037+0000","last_fullsized":"2024-02-12T16:09:28.909037+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4219,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":276496384,"num_objects":4219,"num_object_clones":0,"num_object_copies":8438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4219,"num_write_kb":270016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"265'4211","reported_seq":4227,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:28.906598+0000","last_change":"2024-02-12T16:08:56.024538+0000","last_active":"2024-02-12T16:09:28.906598+0000","last_peered":"2024-02-12T16:09:28.906598+0000","last_clean":"2024-02-12T16:09:28.906598+0000","last_became_active":"2024-02-12T16:08:56.024064+0000","last_became_peered":"2024-02-12T16:08:56.024064+0000","last_unstale":"2024-02-12T16:09:28.906598+0000","last_undegraded":"2024-02-12T16:09:28.906598+0000","last_fullsized":"2024-02-12T16:09:28.906598+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4211,"ondisk_log_size":4211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":275972096,"num_objects":4211,"num_object_clones":0,"num_object_copies":8422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4211,"num_write_kb":269504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"265'3516","reported_seq":3531,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:24.951530+0000","last_change":"2024-02-12T16:08:56.026436+0000","last_active":"2024-02-12T16:09:24.951530+0000","last_peered":"2024-02-12T16:09:24.951530+0000","last_clean":"2024-02-12T16:09:24.951530+0000","last_became_active":"2024-02-12T16:08:56.026173+0000","last_became_peered":"2024-02-12T16:08:56.026173+0000","last_unstale":"2024-02-12T16:09:24.951530+0000","last_undegraded":"2024-02-12T16:09:24.951530+0000","last_fullsized":"2024-02-12T16:09:24.951530+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3516,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":230424576,"num_objects":3516,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":225024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"265'4257","reported_seq":4272,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:29.198160+0000","last_change":"2024-02-12T16:08:56.022243+0000","last_active":"2024-02-12T16:09:29.198160+0000","last_peered":"2024-02-12T16:09:29.198160+0000","last_clean":"2024-02-12T16:09:29.198160+0000","last_became_active":"2024-02-12T16:08:56.021958+0000","last_became_peered":"2024-02-12T16:08:56.021958+0000","last_unstale":"2024-02-12T16:09:29.198160+0000","last_undegraded":"2024-02-12T16:09:29.198160+0000","last_fullsized":"2024-02-12T16:09:29.198160+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4257,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":278986752,"num_objects":4257,"num_object_clones":0,"num_object_copies":8514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":272448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"265'3659","reported_seq":3675,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:25.074653+0000","last_change":"2024-02-12T16:08:56.020936+0000","last_active":"2024-02-12T16:09:25.074653+0000","last_peered":"2024-02-12T16:09:25.074653+0000","last_clean":"2024-02-12T16:09:25.074653+0000","last_became_active":"2024-02-12T16:08:56.020715+0000","last_became_peered":"2024-02-12T16:08:56.020715+0000","last_unstale":"2024-02-12T16:09:25.074653+0000","last_undegraded":"2024-02-12T16:09:25.074653+0000","last_fullsized":"2024-02-12T16:09:25.074653+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3659,"ondisk_log_size":3659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":239796224,"num_objects":3659,"num_object_clones":0,"num_object_copies":7318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3659,"num_write_kb":234176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"265'3833","reported_seq":3849,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:26.892358+0000","last_change":"2024-02-12T16:08:56.026672+0000","last_active":"2024-02-12T16:09:26.892358+0000","last_peered":"2024-02-12T16:09:26.892358+0000","last_clean":"2024-02-12T16:09:26.892358+0000","last_became_active":"2024-02-12T16:08:56.026413+0000","last_became_peered":"2024-02-12T16:08:56.026413+0000","last_unstale":"2024-02-12T16:09:26.892358+0000","last_undegraded":"2024-02-12T16:09:26.892358+0000","last_fullsized":"2024-02-12T16:09:26.892358+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3833,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":251199488,"num_objects":3833,"num_object_clones":0,"num_object_copies":7666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":245312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"265'3907","reported_seq":3923,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:26.655736+0000","last_change":"2024-02-12T16:08:56.021364+0000","last_active":"2024-02-12T16:09:26.655736+0000","last_peered":"2024-02-12T16:09:26.655736+0000","last_clean":"2024-02-12T16:09:26.655736+0000","last_became_active":"2024-02-12T16:08:56.021115+0000","last_became_peered":"2024-02-12T16:08:56.021115+0000","last_unstale":"2024-02-12T16:09:26.655736+0000","last_undegraded":"2024-02-12T16:09:26.655736+0000","last_fullsized":"2024-02-12T16:09:26.655736+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3907,"ondisk_log_size":3907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256049152,"num_objects":3907,"num_object_clones":0,"num_object_copies":7814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3907,"num_write_kb":250048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"265'4278","reported_seq":4291,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:29.130718+0000","last_change":"2024-02-12T16:08:56.026095+0000","last_active":"2024-02-12T16:09:29.130718+0000","last_peered":"2024-02-12T16:09:29.130718+0000","last_clean":"2024-02-12T16:09:29.130718+0000","last_became_active":"2024-02-12T16:08:56.025422+0000","last_became_peered":"2024-02-12T16:08:56.025422+0000","last_unstale":"2024-02-12T16:09:29.130718+0000","last_undegraded":"2024-02-12T16:09:29.130718+0000","last_fullsized":"2024-02-12T16:09:29.130718+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4278,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":280363008,"num_objects":4278,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":273792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"265'3698","reported_seq":3714,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:25.069829+0000","last_change":"2024-02-12T16:08:56.022404+0000","last_active":"2024-02-12T16:09:25.069829+0000","last_peered":"2024-02-12T16:09:25.069829+0000","last_clean":"2024-02-12T16:09:25.069829+0000","last_became_active":"2024-02-12T16:08:56.022130+0000","last_became_peered":"2024-02-12T16:08:56.022130+0000","last_unstale":"2024-02-12T16:09:25.069829+0000","last_undegraded":"2024-02-12T16:09:25.069829+0000","last_fullsized":"2024-02-12T16:09:25.069829+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3698,"ondisk_log_size":3698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":242352128,"num_objects":3698,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3698,"num_write_kb":236672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"265'3737","reported_seq":3752,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:24.990201+0000","last_change":"2024-02-12T16:08:56.275603+0000","last_active":"2024-02-12T16:09:24.990201+0000","last_peered":"2024-02-12T16:09:24.990201+0000","last_clean":"2024-02-12T16:09:24.990201+0000","last_became_active":"2024-02-12T16:08:56.275308+0000","last_became_peered":"2024-02-12T16:08:56.275308+0000","last_unstale":"2024-02-12T16:09:24.990201+0000","last_undegraded":"2024-02-12T16:09:24.990201+0000","last_fullsized":"2024-02-12T16:09:24.990201+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":3737,"ondisk_log_size":3737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":244908032,"num_objects":3737,"num_object_clones":0,"num_object_copies":7474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":239168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"265'4236","reported_seq":4251,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-02-12T16:09:29.197166+0000","last_change":"2024-02-12T16:08:56.274192+0000","last_active":"2024-02-12T16:09:29.197166+0000","last_peered":"2024-02-12T16:09:29.197166+0000","last_clean":"2024-02-12T16:09:29.197166+0000","last_became_active":"2024-02-12T16:08:56.273670+0000","last_became_peered":"2024-02-12T16:08:56.273670+0000","last_unstale":"2024-02-12T16:09:29.197166+0000","last_undegraded":"2024-02-12T16:09:29.197166+0000","last_fullsized":"2024-02-12T16:09:29.197166+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":4236,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":277610496,"num_objects":4236,"num_object_clones":0,"num_object_copies":8472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":271104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":279,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-02-12T16:09:23.720805+0000","last_change":"2024-02-12T16:08:12.878558+0000","last_active":"2024-02-12T16:09:23.720805+0000","last_peered":"2024-02-12T16:09:23.720805+0000","last_clean":"2024-02-12T16:09:23.720805+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:09:23.720805+0000","last_undegraded":"2024-02-12T16:09:23.720805+0000","last_fullsized":"2024-02-12T16:09:23.720805+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:12.878487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:08:12.878487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":4174118912,"num_objects":63692,"num_object_clones":0,"num_object_copies":127384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63692,"num_write_kb":4076288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8272347136,"data_stored":8272347136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63692,"ondisk_log_size":63692,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958663,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1928968,"kb_used_data":1619456,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91816184,"statfs":{"total":95995035648,"available":94019772416,"internally_reserved":0,"allocated":1658322944,"data_stored":1657932726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.745,"15min":0.93200000000000005},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":3.3929999999999998,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.91100000000000003,"15min":1.0389999999999999},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.1909999999999998,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":0.78400000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":1.2270000000000001,"15min":1.1220000000000001},"min":{"1min":0.47399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.109,"15min":1.093},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.786,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.69999999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0980000000000001,"15min":1.1259999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.7109999999999999,"5min":5.274,"15min":5.274},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":0.96999999999999997,"15min":1.1160000000000001},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":3.7360000000000002,"5min":5.375,"15min":5.375},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.077,"15min":1.1499999999999999},"min":{"1min":0.33200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.712,"5min":4.266,"15min":4.266},"last":0.61399999999999999},{"interface":"front","average":{"1min":2.242,"5min":1.3320000000000001,"15min":1.224},"min":{"1min":0.34799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":8.9819999999999993,"5min":8.9819999999999993,"15min":8.9819999999999993},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.0660000000000001,"15min":1.2310000000000001},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6420000000000003,"5min":6.6420000000000003,"15min":6.6420000000000003},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.83199999999999996,"15min":1.21},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.788,"5min":3.1299999999999999,"15min":4.0789999999999997},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.95799999999999996,"15min":1.284},"min":{"1min":0.376,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.474,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.1839999999999999,"15min":1.367},"min":{"1min":0.45900000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.6849999999999996,"5min":6.6849999999999996,"15min":6.6849999999999996},"last":0.753}]},{"osd":6,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.97599999999999998,"15min":1.3120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.105,"5min":3.2290000000000001,"15min":4.4989999999999997},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.0509999999999999,"15min":1.401},"min":{"1min":0.501,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7709999999999999,"5min":3.7709999999999999,"15min":3.7709999999999999},"last":0.74299999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.006,"15min":1.389},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.909,"5min":3.4710000000000001,"15min":4.3129999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.46},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.7589999999999999,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":0.84699999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.054,"15min":1.423},"min":{"1min":0.42399999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.6579999999999999,"5min":3.6579999999999999,"15min":4},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.048,"15min":1.4750000000000001},"min":{"1min":0.40999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.7370000000000001,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.81399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.131,"15min":1.46},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.903,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":1.099},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.093,"15min":1.5549999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.73,"5min":4.6689999999999996,"15min":4.6689999999999996},"last":2.9790000000000001}]}]},{"osd":10,"up_from":83,"seq":356482285713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":486344,"kb_used_data":172736,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93258808,"statfs":{"total":95995035648,"available":95497019392,"internally_reserved":0,"allocated":176881664,"data_stored":176491446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.97799999999999998,"15min":0.92500000000000004},"min":{"1min":0.38900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.0739999999999998,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.96599999999999997,"15min":1.018},"min":{"1min":0.32600000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.6070000000000002,"5min":7.6580000000000004,"15min":7.6580000000000004},"last":0.80600000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85299999999999998,"15min":0.94299999999999995},"min":{"1min":0.309,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.4240000000000004,"5min":4.8300000000000001,"15min":4.8300000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0369999999999999,"15min":0.99199999999999999},"min":{"1min":0.377,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.621,"5min":7.9829999999999997,"15min":7.9829999999999997},"last":0.94099999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.022,"15min":1.02},"min":{"1min":0.36699999999999999,"5min":0.308,"15min":0.308},"max":{"1min":3.1509999999999998,"5min":4.8620000000000001,"15min":4.8620000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.95499999999999996,"15min":0.98499999999999999},"min":{"1min":0.35599999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.7749999999999999,"5min":5.4100000000000001,"15min":5.4100000000000001},"last":0.89800000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.88200000000000001,"15min":1.069},"min":{"1min":0.40999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.605,"5min":4.9189999999999996,"15min":4.9189999999999996},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.103,"5min":0.88300000000000001,"15min":1.1060000000000001},"min":{"1min":0.48399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.704,"5min":4.5919999999999996,"15min":4.5919999999999996},"last":1.2290000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":7.2210000000000001,"15min":3.1909999999999998},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.8849999999999998,"5min":602.05399999999997,"15min":602.05399999999997},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":7.2750000000000004,"15min":3.2949999999999999},"min":{"1min":0.59799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.825,"5min":598.875,"15min":598.875},"last":1.1859999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.98199999999999998,"15min":1.1319999999999999},"min":{"1min":0.39600000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.8519999999999999,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.012,"15min":1.2529999999999999},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.8140000000000001,"5min":4.6660000000000004,"15min":4.6660000000000004},"last":1.0409999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":7.3570000000000002,"15min":3.3330000000000002},"min":{"1min":0.50900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":5.3220000000000001,"5min":597.48299999999995,"15min":597.48299999999995},"last":1.073},{"interface":"front","average":{"1min":1.1970000000000001,"5min":7.4100000000000001,"15min":3.3809999999999998},"min":{"1min":0.57199999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":3.8149999999999999,"5min":600.23099999999999,"15min":600.23099999999999},"last":1.113}]},{"osd":8,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.157,"15min":1.2949999999999999},"min":{"1min":0.51000000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.0640000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"last":1.012},{"interface":"front","average":{"1min":1.403,"5min":1.1919999999999999,"15min":1.327},"min":{"1min":0.63700000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.278,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":1.135}]},{"osd":9,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.1160000000000001,"15min":1.3580000000000001},"min":{"1min":0.67100000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.8859999999999999,"5min":4.7089999999999996,"15min":4.7089999999999996},"last":1.087},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.1200000000000001,"15min":1.345},"min":{"1min":0.60799999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.0309999999999997,"5min":4.7789999999999999,"15min":4.7789999999999999},"last":1.1519999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.2190000000000001,"15min":1.468},"min":{"1min":0.57699999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.8730000000000002,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":5.0499999999999998},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1659999999999999,"15min":1.4690000000000001},"min":{"1min":0.58599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":4.1479999999999997,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":4.5640000000000001}]}]},{"osd":9,"up_from":74,"seq":317827580057,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":782472,"kb_used_data":481152,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92962680,"statfs":{"total":95995035648,"available":95193784320,"internally_reserved":0,"allocated":492699648,"data_stored":492309450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.79300000000000004,"15min":0.88500000000000001},"min":{"1min":0.31900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":1.198,"5min":10.788,"15min":10.788},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.93400000000000005,"15min":0.94799999999999995},"min":{"1min":0.308,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.8680000000000001,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.73999999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":7.694,"5min":2.7599999999999998,"15min":1.6100000000000001},"min":{"1min":0.29399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":175.33500000000001,"5min":175.33500000000001,"15min":175.33500000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":7.6369999999999996,"5min":2.6869999999999998,"15min":1.5860000000000001},"min":{"1min":0.33900000000000002,"5min":0.317,"15min":0.317},"max":{"1min":175.24299999999999,"5min":175.24299999999999,"15min":175.24299999999999},"last":1.1870000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.2390000000000001,"15min":1.0389999999999999},"min":{"1min":0.52900000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.2370000000000001,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.256,"15min":1.0820000000000001},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.3479999999999999,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":1.016}]},{"osd":3,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":2.2080000000000002,"5min":1.3720000000000001,"15min":1.099},"min":{"1min":0.42699999999999999,"5min":0.308,"15min":0.308},"max":{"1min":34.591000000000001,"5min":34.591000000000001,"15min":34.591000000000001},"last":2.399},{"interface":"front","average":{"1min":2.3500000000000001,"5min":1.502,"15min":1.2410000000000001},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":34.494,"5min":34.494,"15min":34.494},"last":2.3500000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.3240000000000001,"15min":1.2170000000000001},"min":{"1min":0.41399999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.391,"5min":25.552,"15min":25.552},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1020000000000001,"15min":1.1519999999999999},"min":{"1min":0.40699999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4199999999999999,"5min":25.847999999999999,"15min":25.847999999999999},"last":1.0880000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":7.9480000000000004,"15min":3.4700000000000002},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.4630000000000001,"5min":617.95899999999995,"15min":617.95899999999995},"last":1.173},{"interface":"front","average":{"1min":1.2250000000000001,"5min":7.9279999999999999,"15min":3.5110000000000001},"min":{"1min":0.41599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.5860000000000001,"5min":617.88099999999997,"15min":617.88099999999997},"last":1.278}]},{"osd":6,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1000000000000001,"15min":1.244},"min":{"1min":0.49199999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.754,"5min":5.0960000000000001,"15min":5.0960000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.254,"5min":0.88900000000000001,"15min":1.1850000000000001},"min":{"1min":0.46500000000000002,"5min":0.377,"15min":0.377},"max":{"1min":2.2719999999999998,"5min":3.4900000000000002,"15min":3.4900000000000002},"last":1.496}]},{"osd":7,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":11.624000000000001,"15min":4.7409999999999997},"min":{"1min":0.78500000000000003,"5min":0.379,"15min":0.379},"max":{"1min":4.5030000000000001,"5min":613.28200000000004,"15min":613.28200000000004},"last":1.141},{"interface":"front","average":{"1min":1.3009999999999999,"5min":11.571,"15min":4.7999999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.825,"5min":613.25199999999995,"15min":613.25199999999995},"last":1.333}]},{"osd":8,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.2490000000000001,"15min":1.325},"min":{"1min":0.55700000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.3330000000000002,"5min":4.96,"15min":4.96},"last":1.355},{"interface":"front","average":{"1min":1.238,"5min":0.99199999999999999,"15min":1.238},"min":{"1min":0.57499999999999996,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.3559999999999999,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":1.4079999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:09:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.0249999999999999,"15min":1.2889999999999999},"min":{"1min":0.49199999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.4219999999999997,"5min":5.5,"15min":5.5},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.202,"15min":1.3089999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.8839999999999999,"5min":4.7619999999999996,"15min":4.7619999999999996},"last":1.623}]}]},{"osd":8,"up_from":65,"seq":279172874400,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571208,"kb_used_data":269888,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93173944,"statfs":{"total":95995035648,"available":95410118656,"internally_reserved":0,"allocated":276365312,"data_stored":275975114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.80700000000000005,"15min":0.86599999999999999},"min":{"1min":0.309,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.0430000000000001,"5min":5.383,"15min":5.383},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.1739999999999999,"5min":0.90800000000000003,"15min":0.91500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.5330000000000004,"5min":4.5330000000000004,"15min":4.5330000000000004},"last":0.505}]},{"osd":1,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8819999999999999,"5min":1.8939999999999999,"15min":2.0230000000000001},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":7.3789999999999996,"5min":23.661000000000001,"15min":23.661000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.9970000000000001,"15min":2.161},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.4340000000000002,"5min":23.672000000000001,"15min":23.672000000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":12.574,"5min":3.3340000000000001,"15min":1.696},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":193.929,"5min":193.929,"15min":193.929},"last":0.52300000000000002},{"interface":"front","average":{"1min":12.651,"5min":3.484,"15min":1.8380000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":193.82499999999999,"5min":193.82499999999999,"15min":193.82499999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0629999999999999,"15min":1.046},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.5659999999999998,"5min":5.4020000000000001,"15min":5.4020000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0620000000000001,"15min":1.0920000000000001},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.8879999999999999,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.92300000000000004,"15min":1.1020000000000001},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":3.8420000000000001,"5min":6.4619999999999997,"15min":6.4619999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.091,"5min":1.087,"15min":1.1539999999999999},"min":{"1min":0.376,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.9279999999999999,"5min":5.71,"15min":5.71},"last":0.36799999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.052,"15min":1.1240000000000001},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.956,"5min":13.961,"15min":13.961},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.052,"15min":1.1890000000000001},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.0760000000000001,"5min":13.103,"15min":13.103},"last":0.66300000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1,"15min":1.157},"min":{"1min":0.46400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.0289999999999999,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.333,"5min":1.2050000000000001,"15min":1.2709999999999999},"min":{"1min":0.40100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.4720000000000004,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.71499999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.2070000000000001,"15min":1.2869999999999999},"min":{"1min":0.51600000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.8719999999999999,"5min":13.242000000000001,"15min":13.242000000000001},"last":0.747},{"interface":"front","average":{"1min":1.206,"5min":1.3260000000000001,"15min":1.355},"min":{"1min":0.52600000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":3.9820000000000002,"5min":13.388,"15min":13.388},"last":0.40400000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:08:26 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1419999999999999,"15min":1.2689999999999999},"min":{"1min":0.40899999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.7909999999999999,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.056,"5min":1.212,"15min":1.238},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.8119999999999998,"5min":6.2000000000000002,"15min":6.2000000000000002},"last":0.68700000000000006}]},{"osd":10,"last update":"Mon Feb 12 16:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.214,"15min":1.3220000000000001},"min":{"1min":0.57499999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.2670000000000003,"5min":5.4189999999999996,"15min":5.4189999999999996},"last":0.873},{"interface":"front","average":{"1min":1.48,"5min":1.0429999999999999,"15min":1.3340000000000001},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":3.4860000000000002,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.94899999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:09:16 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":8.4209999999999994,"15min":27.193999999999999},"min":{"1min":0.58999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.8220000000000001,"5min":635.66399999999999,"15min":635.66399999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.4450000000000001,"5min":8.2390000000000008,"15min":26.914999999999999},"min":{"1min":0.59599999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.1000000000000001,"5min":636.28099999999995,"15min":636.28099999999995},"last":0.621}]}]},{"osd":7,"up_from":56,"seq":240518168745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840648,"kb_used_data":531136,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92904504,"statfs":{"total":95995035648,"available":95134212096,"internally_reserved":0,"allocated":543883264,"data_stored":543493066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.4670000000000001,"15min":1.1659999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.8380000000000001,"5min":60.152000000000001,"15min":60.152000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.74,"15min":1.2210000000000001},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.3879999999999999,"5min":62.020000000000003,"15min":62.020000000000003},"last":5.444}]},{"osd":2,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.8069999999999999,"15min":1.27},"min":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.4249999999999998,"5min":60.204999999999998,"15min":60.204999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.641,"15min":1.2470000000000001},"min":{"1min":0.45600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.952,"5min":59.365000000000002,"15min":59.365000000000002},"last":0.40400000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.143,"15min":1.0980000000000001},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.1079999999999997,"5min":4.8200000000000003,"15min":4.8200000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.997,"5min":1.095,"15min":1.075},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6879999999999999,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":0.37}]},{"osd":4,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79400000000000004,"15min":0.91500000000000004},"min":{"1min":0.42399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.272,"5min":6.2519999999999998,"15min":6.2519999999999998},"last":5.3129999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":0.871,"15min":0.99299999999999999},"min":{"1min":0.50900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2130000000000001,"5min":4.7430000000000003,"15min":4.7430000000000003},"last":0.32800000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.89800000000000002,"15min":1.0249999999999999},"min":{"1min":0.48599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.3480000000000001,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":5.375},{"interface":"front","average":{"1min":1.135,"5min":0.89600000000000002,"15min":1.0740000000000001},"min":{"1min":0.53100000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.899,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.57899999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0309999999999999,"15min":1.135},"min":{"1min":0.46400000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.411,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":5.4119999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":0.95399999999999996,"15min":1.1200000000000001},"min":{"1min":0.48499999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6180000000000001,"5min":4.7149999999999999,"15min":4.7149999999999999},"last":0.46300000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:08:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.1699999999999999,"15min":1.3360000000000001},"min":{"1min":0.5,"5min":0.432,"15min":0.432},"max":{"1min":4.7779999999999996,"5min":4.7779999999999996,"15min":4.7779999999999996},"last":5.5300000000000002},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.2649999999999999,"15min":1.3819999999999999},"min":{"1min":0.54500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7919999999999998,"5min":4.851,"15min":4.851},"last":0.47899999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":4.3070000000000004,"15min":2.395},"min":{"1min":0.52300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":2.2949999999999999,"5min":278.54599999999999,"15min":278.54599999999999},"last":5.5030000000000001},{"interface":"front","average":{"1min":1.284,"5min":4.2300000000000004,"15min":2.3610000000000002},"min":{"1min":0.51400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.8240000000000001,"5min":281.09199999999998,"15min":281.09199999999998},"last":0.5}]},{"osd":10,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1379999999999999,"15min":1.3919999999999999},"min":{"1min":0.61599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0430000000000001,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.52,"5min":1.214,"15min":1.4279999999999999},"min":{"1min":0.75,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.1059999999999999,"5min":4.9450000000000003,"15min":4.9450000000000003},"last":0.55900000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:08:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":4.1340000000000003,"15min":2.4260000000000002},"min":{"1min":0.57599999999999996,"5min":0.498,"15min":0.498},"max":{"1min":3.593,"5min":280.83999999999997,"15min":280.83999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.528,"5min":4.048,"15min":2.367},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.8339999999999996,"5min":278.63799999999998,"15min":278.63799999999998},"last":5.476}]}]},{"osd":6,"up_from":48,"seq":206158430386,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582856,"kb_used_data":269248,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93162296,"statfs":{"total":95995035648,"available":95398191104,"internally_reserved":0,"allocated":275709952,"data_stored":275319754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.7959999999999998,"5min":1.429,"15min":1.121},"min":{"1min":0.26500000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":67.093000000000004,"5min":67.093000000000004,"15min":67.093000000000004},"last":1.488},{"interface":"front","average":{"1min":3.8149999999999999,"5min":1.462,"15min":1.1240000000000001},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":68.549000000000007,"5min":68.549000000000007,"15min":68.549000000000007},"last":0.61499999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9489999999999998,"5min":1.6679999999999999,"15min":1.1879999999999999},"min":{"1min":0.317,"5min":0.308,"15min":0.308},"max":{"1min":66.936000000000007,"5min":66.936000000000007,"15min":66.936000000000007},"last":1.081},{"interface":"front","average":{"1min":4.3890000000000002,"5min":1.6359999999999999,"15min":1.222},"min":{"1min":0.34499999999999997,"5min":0.309,"15min":0.309},"max":{"1min":68.683000000000007,"5min":68.683000000000007,"15min":68.683000000000007},"last":0.32800000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":4.1319999999999997,"5min":2.452,"15min":1.488},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":68.462000000000003,"5min":68.462000000000003,"15min":68.462000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":3.9399999999999999,"5min":2.456,"15min":1.5189999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":67.322999999999993,"5min":67.322999999999993,"15min":67.322999999999993},"last":0.378}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.512,"15min":1.1659999999999999},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":68.332999999999998,"5min":68.332999999999998,"15min":68.332999999999998},"last":2.173},{"interface":"front","average":{"1min":4.1600000000000001,"5min":1.55,"15min":1.228},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":67.281999999999996,"5min":67.281999999999996,"15min":67.281999999999996},"last":1.7410000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.7919999999999998,"5min":1.474,"15min":1.141},"min":{"1min":0.38300000000000001,"5min":0.316,"15min":0.316},"max":{"1min":68.605999999999995,"5min":68.605999999999995,"15min":68.605999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.77600000000000002,"15min":0.94899999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.3999999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.42199999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9750000000000001,"5min":1.593,"15min":1.2589999999999999},"min":{"1min":0.36099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":69.260000000000005,"5min":69.260000000000005,"15min":69.260000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":3.839,"5min":1.4259999999999999,"15min":1.22},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":68.742000000000004,"5min":68.742000000000004,"15min":68.742000000000004},"last":0.63100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":2.7349999999999999,"15min":1.7410000000000001},"min":{"1min":0.46100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.419,"5min":67.144999999999996,"15min":67.144999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.3680000000000001,"5min":2.6819999999999999,"15min":1.758},"min":{"1min":0.54300000000000004,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.5139999999999998,"5min":67.230999999999995,"15min":67.230999999999995},"last":0.67100000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":4.7350000000000003,"5min":1.8779999999999999,"15min":1.5880000000000001},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":68.653000000000006,"5min":68.653000000000006,"15min":68.653000000000006},"last":0.60099999999999998},{"interface":"front","average":{"1min":4.8259999999999996,"5min":1.752,"15min":1.5589999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":68.620999999999995,"5min":68.620999999999995,"15min":68.620999999999995},"last":0.55900000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.7869999999999999,"15min":1.621},"min":{"1min":0.623,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.4769999999999999,"5min":67.188000000000002,"15min":67.188000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.335,"5min":1.7529999999999999,"15min":1.5600000000000001},"min":{"1min":0.61799999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.784,"5min":68.506,"15min":68.506},"last":0.59099999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.8280000000000001,"15min":1.702},"min":{"1min":0.63500000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9019999999999999,"5min":68.722999999999999,"15min":68.722999999999999},"last":2.2909999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.6970000000000001,"15min":1.617},"min":{"1min":0.60299999999999998,"5min":0.505,"15min":0.505},"max":{"1min":2.0030000000000001,"5min":68.573999999999998,"15min":68.573999999999998},"last":2.3199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022280,"kb_used_data":716864,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92722872,"statfs":{"total":95995035648,"available":94948220928,"internally_reserved":0,"allocated":734068736,"data_stored":733678538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.194,"15min":0.93799999999999994},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.101,"5min":9.8740000000000006,"15min":9.8740000000000006},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.214,"15min":0.998},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":1.3300000000000001,"5min":10.513,"15min":10.513},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.861,"15min":1.2110000000000001},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.1529999999999996,"5min":40.542999999999999,"15min":40.542999999999999},"last":0.307},{"interface":"front","average":{"1min":1.865,"5min":1.9750000000000001,"15min":1.258},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":9.2119999999999997,"5min":40.811999999999998,"15min":40.811999999999998},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.169,"15min":1.004},"min":{"1min":0.28499999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.6900000000000004,"5min":6.0170000000000003,"15min":6.0170000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.962,"5min":1.405,"15min":1.1259999999999999},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":10.225,"5min":10.648999999999999,"15min":10.648999999999999},"last":0.60499999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.3919999999999999,"15min":1.206},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.093,"5min":9.6780000000000008,"15min":9.6780000000000008},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.103,"5min":1.353,"15min":1.2450000000000001},"min":{"1min":0.41299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.0089999999999999,"5min":10.244999999999999,"15min":10.244999999999999},"last":0.63500000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.399,"15min":1.292},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.8979999999999999,"5min":9.8019999999999996,"15min":9.8019999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.173,"15min":1.2789999999999999},"min":{"1min":0.46999999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0390000000000001,"5min":10.420999999999999,"15min":10.420999999999999},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1839999999999999,"15min":1.262},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.946,"5min":10.455,"15min":10.455},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.232,"15min":1.3420000000000001},"min":{"1min":0.54800000000000004,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.04,"5min":10.488,"15min":10.488},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.286,"15min":1.385},"min":{"1min":0.58099999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.6319999999999999,"5min":6.2869999999999999,"15min":6.2869999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.1890000000000001,"15min":1.3},"min":{"1min":0.57099999999999995,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.6000000000000001,"5min":5.1429999999999998,"15min":5.1429999999999998},"last":0.503}]},{"osd":8,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.5449999999999999,"15min":1.4770000000000001},"min":{"1min":0.50900000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":9.9049999999999994,"15min":9.9049999999999994},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.597,"15min":1.577},"min":{"1min":0.625,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.2410000000000001,"5min":10.313000000000001,"15min":10.313000000000001},"last":0.58699999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.456,"15min":1.5860000000000001},"min":{"1min":0.60299999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.8400000000000001,"5min":10.352,"15min":10.352},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.2030000000000001,"15min":1.419},"min":{"1min":0.56899999999999995,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.4259999999999999,"5min":4.7839999999999998,"15min":4.7839999999999998},"last":0.53300000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.5509999999999999,"15min":1.5529999999999999},"min":{"1min":0.68799999999999994,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7949999999999999,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.5489999999999999,"15min":1.6679999999999999},"min":{"1min":0.64200000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.798,"5min":10.387,"15min":10.387},"last":0.61099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738594,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228732,"kb_used_data":923316,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92516420,"statfs":{"total":95995035648,"available":94736814080,"internally_reserved":0,"allocated":945475584,"data_stored":945091473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":2.4980000000000002,"15min":1.361},"min":{"1min":0.28499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.8369999999999997,"5min":114.01600000000001,"15min":114.01600000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.014,"5min":2.6389999999999998,"15min":1.429},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.254},"max":{"1min":3.3919999999999999,"5min":114.342,"15min":114.342},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.048,"15min":0.89900000000000002},"min":{"1min":0.27600000000000002,"5min":0.215,"15min":0.215},"max":{"1min":5.633,"5min":5.633,"15min":5.633},"last":2.0830000000000002},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.0409999999999999,"15min":0.95799999999999996},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.5140000000000002,"5min":7.5140000000000002,"15min":7.5140000000000002},"last":1.294}]},{"osd":4,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.83099999999999996,"15min":1.032},"min":{"1min":0.312,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.9770000000000001,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.34899999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.1080000000000001,"5min":4.5860000000000003,"15min":4.5860000000000003},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.93300000000000005,"15min":1.034},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.86699999999999999,"15min":1.0960000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.75,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.2410000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.93600000000000005,"15min":1.089},"min":{"1min":0.376,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6699999999999999,"5min":5.4699999999999998,"15min":5.4699999999999998},"last":1.4590000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.97299999999999998,"15min":1.153},"min":{"1min":0.502,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.6579999999999999,"5min":5.5389999999999997,"15min":5.5389999999999997},"last":1.321}]},{"osd":7,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.0109999999999999,"15min":1.2410000000000001},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.966,"5min":5.5060000000000002,"15min":5.5060000000000002},"last":1.421},{"interface":"front","average":{"1min":1.373,"5min":1.103,"15min":1.298},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.1709999999999998,"5min":5.5720000000000001,"15min":5.5720000000000001},"last":1.4410000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.1850000000000001,"15min":1.3560000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.7140000000000004,"5min":5.6059999999999999,"15min":5.6059999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.3180000000000001,"15min":1.419},"min":{"1min":0.55100000000000005,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":7.5599999999999996,"5min":7.5599999999999996,"15min":7.5599999999999996},"last":1.345}]},{"osd":9,"last update":"Mon Feb 12 16:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.3049999999999999,"15min":1.4350000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.75,"5min":5.7969999999999997,"15min":5.7969999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.117,"5min":1.206,"15min":1.4319999999999999},"min":{"1min":0.59699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.2229999999999999,"5min":5.6890000000000001,"15min":5.6890000000000001},"last":1.399}]},{"osd":10,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.2569999999999999,"15min":1.464},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":4.8079999999999998,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.288,"15min":1.367},"min":{"1min":0.44900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":3.1549999999999998,"5min":5.8010000000000002,"15min":5.8010000000000002},"last":0.54400000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:08:40 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.2609999999999999,"15min":1.462},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.9359999999999999,"5min":7.9660000000000002,"15min":7.9660000000000002},"last":1.371},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.5149999999999999,"15min":1.4350000000000001},"min":{"1min":0.48199999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.1920000000000002,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.51500000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313425,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":314376,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93430776,"statfs":{"total":95995035648,"available":95673114624,"internally_reserved":0,"allocated":786432,"data_stored":396234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.181,"15min":0.93500000000000005},"min":{"1min":0.248,"5min":0.222,"15min":0.222},"max":{"1min":2.4449999999999998,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3320000000000001,"15min":1.0029999999999999},"min":{"1min":0.48699999999999999,"5min":0.216,"15min":0.216},"max":{"1min":6.3079999999999998,"5min":6.3079999999999998,"15min":6.3079999999999998},"last":2.1269999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.4530000000000001,"15min":1.085},"min":{"1min":0.26600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":9.1500000000000004,"5min":11.423,"15min":11.423},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.4319999999999999,"15min":1.1619999999999999},"min":{"1min":0.56000000000000005,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.4649999999999999,"5min":7.2290000000000001,"15min":7.2290000000000001},"last":1.966}]},{"osd":3,"last update":"Mon Feb 12 16:09:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.512,"15min":1.1719999999999999},"min":{"1min":0.36699999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.1970000000000001,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":2.1970000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.1799999999999999,"15min":1.0449999999999999},"min":{"1min":0.34100000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.3700000000000001,"5min":9.4399999999999995,"15min":9.4399999999999995},"last":2.3700000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.3080000000000001,"15min":1.2},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.4179999999999999,"5min":9.657,"15min":9.657},"last":1.8480000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1719999999999999,"15min":1.0269999999999999},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.5390000000000001,"5min":9.5530000000000008,"15min":9.5530000000000008},"last":0.38}]},{"osd":5,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":7.6550000000000002,"15min":3.3359999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.4159999999999999,"5min":616.34199999999998,"15min":616.34199999999998},"last":2.0910000000000002},{"interface":"front","average":{"1min":1.7010000000000001,"5min":7.9450000000000003,"15min":3.3940000000000001},"min":{"1min":0.80600000000000005,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":5.6390000000000002,"5min":615.18899999999996,"15min":615.18899999999996},"last":1.9059999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.1859999999999999,"15min":1.1439999999999999},"min":{"1min":0.41499999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.706,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":2.0649999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.2689999999999999,"15min":1.266},"min":{"1min":0.50900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.3650000000000002,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.41499999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":7.9189999999999996,"15min":3.5489999999999999},"min":{"1min":0.47699999999999998,"5min":0.377,"15min":0.377},"max":{"1min":1.6140000000000001,"5min":611.79600000000005,"15min":611.79600000000005},"last":1.9450000000000001},{"interface":"front","average":{"1min":1.151,"5min":8.0579999999999998,"15min":3.544},"min":{"1min":0.54400000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.7470000000000001,"5min":611.83900000000006,"15min":611.83900000000006},"last":2.1640000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:38 2024","interfaces":[{"interface":"back","average":{"1min":2.415,"5min":1.5269999999999999,"15min":1.48},"min":{"1min":0.46700000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":6.4960000000000004,"5min":6.4960000000000004,"15min":6.4960000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":2.383,"5min":1.75,"15min":1.5760000000000001},"min":{"1min":0.52000000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":8.5920000000000005,"5min":8.5920000000000005,"15min":8.5920000000000005},"last":2.0489999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.3180000000000001,"15min":1.3819999999999999},"min":{"1min":0.498,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.929,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":1.9870000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.504,"15min":1.4950000000000001},"min":{"1min":0.59799999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.7690000000000001,"5min":6.4699999999999998,"15min":6.4699999999999998},"last":0.52900000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4259999999999999,"15min":1.4530000000000001},"min":{"1min":0.65100000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.1539999999999999,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":2.0310000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5289999999999999,"15min":1.5389999999999999},"min":{"1min":0.66900000000000004,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":5.5220000000000002,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.59699999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149897,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1791112,"kb_used_data":1493888,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":91954040,"statfs":{"total":95995035648,"available":94160936960,"internally_reserved":0,"allocated":1529741312,"data_stored":1529351114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":2.6890000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.3540000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":2.0659999999999998,"15min":1.2549999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":8.3680000000000003,"5min":70.031999999999996,"15min":70.031999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.621,"5min":2.25,"15min":1.339},"min":{"1min":0.317,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.3399999999999999,"5min":69.944999999999993,"15min":69.944999999999993},"last":0.95299999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.8620000000000001,"15min":1.238},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9329999999999998,"5min":68.405000000000001,"15min":68.405000000000001},"last":1.7549999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.694,"15min":1.2410000000000001},"min":{"1min":0.32800000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.5569999999999999,"5min":69.009,"15min":69.009},"last":1.8240000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92500000000000004,"15min":0.96899999999999997},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.407,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":2.5640000000000001},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.84799999999999998,"15min":1.018},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"last":0.32100000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.92900000000000005,"15min":1.109},"min":{"1min":0.35799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5840000000000001,"5min":4.1840000000000002,"15min":4.1840000000000002},"last":1.796},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.0389999999999999,"15min":1.0820000000000001},"min":{"1min":0.34699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.1179999999999999,"5min":5.4370000000000003,"15min":5.4370000000000003},"last":2.6040000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.94699999999999995,"15min":1.119},"min":{"1min":0.52700000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.399,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.01,"15min":1.2070000000000001},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.4990000000000001,"5min":4.1429999999999998,"15min":4.1429999999999998},"last":1.925}]},{"osd":7,"last update":"Mon Feb 12 16:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.9359999999999999,"15min":1.528},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.9210000000000003,"5min":91.718999999999994,"15min":91.718999999999994},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.968,"15min":1.571},"min":{"1min":0.47799999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.46,"5min":90.367999999999995,"15min":90.367999999999995},"last":1.8899999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:08:38 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.333,"15min":1.347},"min":{"1min":0.44700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.9489999999999998,"5min":6.5899999999999999,"15min":6.5899999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2010000000000001,"15min":1.391},"min":{"1min":0.505,"5min":0.442,"15min":0.442},"max":{"1min":8.516,"5min":8.516,"15min":8.516},"last":0.51100000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1240000000000001,"15min":1.4099999999999999},"min":{"1min":0.61099999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.667,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.274,"15min":1.4530000000000001},"min":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.5289999999999999,"5min":6.5350000000000001,"15min":6.5350000000000001},"last":1.8560000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.385,"15min":1.4430000000000001},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8740000000000001,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":2.6659999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.3220000000000001,"15min":1.4910000000000001},"min":{"1min":0.52300000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.0019999999999998,"5min":5.3419999999999996,"15min":5.3419999999999996},"last":0.54200000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.8200000000000001,"15min":1.6040000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.468,"5min":49.167000000000002,"15min":49.167000000000002},"last":2.637},{"interface":"front","average":{"1min":1.228,"5min":1.75,"15min":1.619},"min":{"1min":0.55800000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":5.0289999999999999,"5min":48.548999999999999,"15min":48.548999999999999},"last":0.56899999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953666,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367176,"kb_used_data":1069952,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92377976,"statfs":{"total":95995035648,"available":94595047424,"internally_reserved":0,"allocated":1095630848,"data_stored":1095240650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.77500000000000002,"15min":0.92800000000000005},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.125,"5min":4.4029999999999996,"15min":4.4029999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.98399999999999999,"15min":0.999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.4950000000000001,"5min":10.754,"15min":10.754},"last":0.68799999999999994}]},{"osd":1,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.659,"15min":1.232},"min":{"1min":0.46899999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.1079999999999997,"5min":42.226999999999997,"15min":42.226999999999997},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.3180000000000001,"15min":1.135},"min":{"1min":0.51500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6950000000000001,"5min":43.689,"15min":43.689},"last":0.79900000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.45,"15min":1.147},"min":{"1min":0.435,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.1429999999999998,"5min":41.546999999999997,"15min":41.546999999999997},"last":2.1400000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.599,"15min":1.1950000000000001},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":4.9340000000000002,"5min":41.57,"15min":41.57},"last":0.85699999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.80200000000000005,"15min":0.97799999999999998},"min":{"1min":0.48199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5980000000000001,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":2.1989999999999998},{"interface":"front","average":{"1min":1.167,"5min":0.88800000000000001,"15min":0.999},"min":{"1min":0.55100000000000005,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.5669999999999999,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":0.76600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.79800000000000004,"15min":0.94699999999999995},"min":{"1min":0.318,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.25,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90300000000000002,"15min":1.026},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3759999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.79700000000000004,"15min":1.004},"min":{"1min":0.39300000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.641,"5min":3.004,"15min":3.004},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.72099999999999997,"15min":1.0209999999999999},"min":{"1min":0.37,"5min":0.307,"15min":0.307},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.9330000000000001},"last":2.173}]},{"osd":7,"last update":"Mon Feb 12 16:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.1789999999999998,"15min":1.496},"min":{"1min":0.45400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.669,"5min":116.273,"15min":116.273},"last":2.5369999999999999},{"interface":"front","average":{"1min":1.383,"5min":2.3119999999999998,"15min":1.6830000000000001},"min":{"1min":0.69499999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.9370000000000001,"5min":116.24299999999999,"15min":116.24299999999999},"last":1.073}]},{"osd":8,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.1599999999999999,"15min":1.399},"min":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.9180000000000001,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.0760000000000001,"15min":1.3140000000000001},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.742,"5min":4.8609999999999998,"15min":4.8609999999999998},"last":0.621}]},{"osd":9,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":4.2199999999999998,"15min":2.464},"min":{"1min":0.745,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.881,"5min":297.55900000000003,"15min":297.55900000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.4730000000000001,"5min":4.2480000000000002,"15min":2.4199999999999999},"min":{"1min":0.52900000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.966,"5min":297.61900000000003,"15min":297.61900000000003},"last":0.92200000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.0349999999999999,"15min":1.363},"min":{"1min":0.63900000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.7490000000000001,"5min":4.944,"15min":4.944},"last":2.2949999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.0649999999999999,"15min":1.385},"min":{"1min":0.55600000000000005,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":5.2469999999999999,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":2.2490000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724729,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052424,"kb_used_data":755200,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92692728,"statfs":{"total":95995035648,"available":94917353472,"internally_reserved":0,"allocated":773324800,"data_stored":772934602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94599999999999995,"15min":0.97399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":3.4950000000000001,"5min":13.031000000000001,"15min":13.031000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.248,"5min":1.0129999999999999,"15min":0.98799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5140000000000002,"5min":14.018000000000001,"15min":14.018000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.627,"15min":1.224},"min":{"1min":0.29399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":7.7610000000000001,"5min":33.084000000000003,"15min":33.084000000000003},"last":0.496},{"interface":"front","average":{"1min":1.708,"5min":1.329,"15min":1.095},"min":{"1min":0.34499999999999997,"5min":0.253,"15min":0.253},"max":{"1min":7.8140000000000001,"5min":33.052999999999997,"15min":33.052999999999997},"last":0.89500000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5150000000000001,"5min":1.631,"15min":1.2450000000000001},"min":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":15.673,"5min":26.797000000000001,"15min":26.797000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.319,"5min":1.3700000000000001,"15min":1.1679999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":8.298,"5min":26.905999999999999,"15min":26.905999999999999},"last":0.61499999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.4079999999999999,"15min":1.177},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.032,"5min":12.991,"15min":12.991},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.429,"5min":1.238,"15min":1.171},"min":{"1min":0.42899999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.6749999999999998,"5min":13.935,"15min":13.935},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.96999999999999997,"15min":0.95699999999999996},"min":{"1min":0.33800000000000002,"5min":0.308,"15min":0.308},"max":{"1min":3.5609999999999999,"5min":14.259,"15min":14.259},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":0.97599999999999998,"15min":1.0509999999999999},"min":{"1min":0.41499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.75,"5min":13.268000000000001,"15min":13.268000000000001},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":2.4060000000000001,"5min":1.145,"15min":1.1439999999999999},"min":{"1min":0.37,"5min":0.318,"15min":0.318},"max":{"1min":13.148,"5min":13.148,"15min":13.148},"last":0.57099999999999995},{"interface":"front","average":{"1min":2.2480000000000002,"5min":1.141,"15min":1.175},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":13.214,"5min":13.214,"15min":13.214},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:09:00 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.282,"15min":1.5760000000000001},"min":{"1min":0.57899999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":5.6669999999999998,"5min":98.971000000000004,"15min":98.971000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.627,"5min":2.306,"15min":1.665},"min":{"1min":0.56699999999999995,"5min":0.45200000000000001,"15min":0.435},"max":{"1min":4.6020000000000003,"5min":99.218000000000004,"15min":99.218000000000004},"last":0.625}]},{"osd":8,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.238,"15min":1.389},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.7839999999999998,"5min":14.372999999999999,"15min":14.372999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.619,"5min":1.264,"15min":1.3580000000000001},"min":{"1min":0.52100000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.7249999999999996,"5min":14.074999999999999,"15min":14.074999999999999},"last":0.76900000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:08:25 2024","interfaces":[{"interface":"back","average":{"1min":2.1970000000000001,"5min":1.2749999999999999,"15min":1.4350000000000001},"min":{"1min":0.499,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":13.375999999999999,"5min":13.375999999999999,"15min":13.375999999999999},"last":0.63},{"interface":"front","average":{"1min":2.097,"5min":1.268,"15min":1.4930000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":14.132,"5min":14.132,"15min":14.132},"last":0.71799999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:09:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.27,"15min":1.4219999999999999},"min":{"1min":0.495,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.712,"5min":14.214,"15min":14.214},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.2769999999999999,"15min":1.5109999999999999},"min":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.8849999999999998,"5min":13.464,"15min":13.464},"last":0.74099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":965345280,"data_stored":965345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":758972416,"data_stored":758972416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1391132672,"data_stored":1391132672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1120731136,"data_stored":1120731136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":760676352,"data_stored":760676352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":276496384,"data_stored":276496384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":546045952,"data_stored":546045952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":275972096,"data_stored":275972096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":506986496,"data_stored":506986496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1669988352,"data_stored":1669988352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:09:31.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:31.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:31.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:31.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:31.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:31.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:31.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:31.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:31.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 72898 72882 198.033 133.812 0.00577897 0.0050477 2024-02-12T16:09:32.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:32.056 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:32.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:32.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:32.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:32.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:32.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:32.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:32.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:32.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:32.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:32.758 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:09:32.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:32.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:32 smithi123 bash[19119]: cluster 2024-02-12T16:09:31.439374+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:09:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:32 smithi123 bash[19119]: cluster 2024-02-12T16:09:31.470983+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:09:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:32 smithi123 bash[19119]: cluster 2024-02-12T16:09:31.631463+0000 mgr.y (mgr.14146) 723 : cluster [DBG] pgmap v892: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 186 MiB/s wr, 2.98k op/s 2024-02-12T16:09:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:32 smithi123 bash[19119]: audit 2024-02-12T16:09:32.541305+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? v1:172.21.15.121:0/1724828845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:09:32.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:32.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 75139 75123 195.617 140.062 0.00386356 0.00510996 2024-02-12T16:09:32.975 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:32.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:32 smithi169 bash[19604]: cluster 2024-02-12T16:09:31.439374+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:09:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:32 smithi169 bash[19604]: cluster 2024-02-12T16:09:31.470983+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:09:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:32 smithi169 bash[19604]: cluster 2024-02-12T16:09:31.631463+0000 mgr.y (mgr.14146) 723 : cluster [DBG] pgmap v892: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 186 MiB/s wr, 2.98k op/s 2024-02-12T16:09:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:32 smithi169 bash[19604]: audit 2024-02-12T16:09:32.541305+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? v1:172.21.15.121:0/1724828845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:09:33.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:33.102 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:33.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:33.135 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:09:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:32 smithi121 bash[18489]: cluster 2024-02-12T16:09:31.439374+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:09:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:32 smithi121 bash[18489]: cluster 2024-02-12T16:09:31.470983+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:09:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:32 smithi121 bash[18489]: cluster 2024-02-12T16:09:31.631463+0000 mgr.y (mgr.14146) 723 : cluster [DBG] pgmap v892: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 186 MiB/s wr, 2.98k op/s 2024-02-12T16:09:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:32 smithi121 bash[18489]: audit 2024-02-12T16:09:32.541305+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? v1:172.21.15.121:0/1724828845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:09:33.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:33.235 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:33.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:33.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:33.362 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:33.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:33.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:33.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:33.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:33.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:33.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:33.624 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:33.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:33.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:33.745 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:33 smithi121 bash[18489]: cluster 2024-02-12T16:09:32.731242+0000 mon.a (mon.0) 1148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:09:33.745 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:33 smithi121 bash[18489]: audit 2024-02-12T16:09:32.748212+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? v1:172.21.15.121:0/1724828845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:09:33.745 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:33 smithi121 bash[18489]: cluster 2024-02-12T16:09:32.748373+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e266: 12 total, 12 up, 12 in 2024-02-12T16:09:33.756 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:33.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:33.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:33.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:33 smithi169 bash[19604]: cluster 2024-02-12T16:09:32.731242+0000 mon.a (mon.0) 1148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:09:33.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:33 smithi169 bash[19604]: audit 2024-02-12T16:09:32.748212+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? v1:172.21.15.121:0/1724828845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:09:33.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:33 smithi169 bash[19604]: cluster 2024-02-12T16:09:32.748373+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e266: 12 total, 12 up, 12 in 2024-02-12T16:09:33.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:33.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:33.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 77402 77386 193.45 141.438 0.0127695 0.00516791 2024-02-12T16:09:33.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:34.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:34.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:34.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:33 smithi123 bash[19119]: cluster 2024-02-12T16:09:32.731242+0000 mon.a (mon.0) 1148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:09:34.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:33 smithi123 bash[19119]: audit 2024-02-12T16:09:32.748212+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? v1:172.21.15.121:0/1724828845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:09:34.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:33 smithi123 bash[19119]: cluster 2024-02-12T16:09:32.748373+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e266: 12 total, 12 up, 12 in 2024-02-12T16:09:34.150 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:34.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:34.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:09:34.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:34.416 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:34.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:34.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:09:34.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:34.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:34.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:34.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:34 smithi169 bash[19604]: cluster 2024-02-12T16:09:33.239829+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:34.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:34 smithi169 bash[19604]: cluster 2024-02-12T16:09:33.239885+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:34.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:34 smithi169 bash[19604]: cluster 2024-02-12T16:09:33.631792+0000 mgr.y (mgr.14146) 724 : cluster [DBG] pgmap v894: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-12T16:09:34.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:34 smithi169 bash[19604]: cluster 2024-02-12T16:09:33.750376+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e267: 12 total, 12 up, 12 in 2024-02-12T16:09:34.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:34.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:34 smithi123 bash[19119]: cluster 2024-02-12T16:09:33.239829+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:34 smithi123 bash[19119]: cluster 2024-02-12T16:09:33.239885+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:34 smithi123 bash[19119]: cluster 2024-02-12T16:09:33.631792+0000 mgr.y (mgr.14146) 724 : cluster [DBG] pgmap v894: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-12T16:09:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:34 smithi123 bash[19119]: cluster 2024-02-12T16:09:33.750376+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e267: 12 total, 12 up, 12 in 2024-02-12T16:09:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:34 smithi121 bash[18489]: cluster 2024-02-12T16:09:33.239829+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:34 smithi121 bash[18489]: cluster 2024-02-12T16:09:33.239885+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:34 smithi121 bash[18489]: cluster 2024-02-12T16:09:33.631792+0000 mgr.y (mgr.14146) 724 : cluster [DBG] pgmap v894: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-12T16:09:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:34 smithi121 bash[18489]: cluster 2024-02-12T16:09:33.750376+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e267: 12 total, 12 up, 12 in 2024-02-12T16:09:34.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:34.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 16 79533 79517 191.131 133.188 0.00310257 0.00523035 2024-02-12T16:09:34.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:34.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:35.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:35.092 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:35.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:35.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:35.226 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:35.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:35.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:35.369 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:35.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:35.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:35.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:35.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:35.605 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:35.632 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:35.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:35.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:35.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:35.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:35.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:09:35.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:35.896 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:35.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:35.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 81740 81724 189.161 137.938 0.00833968 0.00528494 2024-02-12T16:09:35.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:36.026 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:36.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:36.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:36.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:36.291 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:36.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:36 smithi121 bash[18489]: cluster 2024-02-12T16:09:35.632220+0000 mgr.y (mgr.14146) 725 : cluster [DBG] pgmap v896: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.43k op/s 2024-02-12T16:09:36.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:36.531 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:36.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:36 smithi169 bash[19604]: cluster 2024-02-12T16:09:35.632220+0000 mgr.y (mgr.14146) 725 : cluster [DBG] pgmap v896: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.43k op/s 2024-02-12T16:09:36.560 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:36.564 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:36 smithi123 bash[19119]: cluster 2024-02-12T16:09:35.632220+0000 mgr.y (mgr.14146) 725 : cluster [DBG] pgmap v896: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.43k op/s 2024-02-12T16:09:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:36.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:36.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:36.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:36.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:36.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:36.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 83845 83829 187.103 131.562 0.00293975 0.00534299 2024-02-12T16:09:36.961 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:36.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:37.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:37.095 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:37.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:37.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:37.224 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:37.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:37.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:37.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:37.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:37.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:37.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:37.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:37.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:37.613 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:37.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:37.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:37.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:37.743 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:37.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:37.760 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:09:37.760 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:09:37.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:37.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:37.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:37.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 85934 85918 185.153 130.562 0.0138941 0.00539906 2024-02-12T16:09:37.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:38.016 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:38.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:38.152 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:38.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:38.283 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:38.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:38.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:38.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:38.561 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:38.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:38.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:38.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-12T16:09:38.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:38.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:38 smithi123 bash[19119]: cluster 2024-02-12T16:09:37.632640+0000 mgr.y (mgr.14146) 726 : cluster [DBG] pgmap v897: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-12T16:09:38.824 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:38.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:38.952 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 87406 87390 182.048 92 0.00872473 0.00549124 2024-02-12T16:09:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:38.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:38 smithi121 bash[18489]: cluster 2024-02-12T16:09:37.632640+0000 mgr.y (mgr.14146) 726 : cluster [DBG] pgmap v897: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-12T16:09:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:38.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:39.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:38 smithi169 bash[19604]: cluster 2024-02-12T16:09:37.632640+0000 mgr.y (mgr.14146) 726 : cluster [DBG] pgmap v897: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-12T16:09:39.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:39.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:39.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:39.168 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:39.169 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:39.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:39.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:39.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:39.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:39.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:39.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:39.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:39.500 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:39.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:39.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:39.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:39.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:39.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:39 smithi169 bash[19604]: cluster 2024-02-12T16:09:38.785644+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:39.704 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:39 smithi169 bash[19604]: cluster 2024-02-12T16:09:38.785697+0000 osd.11 (osd.11) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:39.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:39.745 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:39 smithi121 bash[18489]: cluster 2024-02-12T16:09:38.785644+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:39.745 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:39 smithi121 bash[18489]: cluster 2024-02-12T16:09:38.785697+0000 osd.11 (osd.11) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:39.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:39.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:39.891 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:39.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:39.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 89435 89419 180.266 126.812 0.00339039 0.00554529 2024-02-12T16:09:39.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:40.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:40.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:40.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:39 smithi123 bash[19119]: cluster 2024-02-12T16:09:38.785644+0000 osd.11 (osd.11) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:40.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:39 smithi123 bash[19119]: cluster 2024-02-12T16:09:38.785697+0000 osd.11 (osd.11) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:40.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:40.181 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:40.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:40.286 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:40.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:40.424 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:40.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:40.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:40.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:40.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:40.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:40.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:40 smithi121 bash[18489]: cluster 2024-02-12T16:09:39.633100+0000 mgr.y (mgr.14146) 727 : cluster [DBG] pgmap v898: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-12T16:09:40.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:40 smithi121 bash[18489]: cluster 2024-02-12T16:09:39.695564+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e268: 12 total, 12 up, 12 in 2024-02-12T16:09:40.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:40 smithi121 bash[18489]: cluster 2024-02-12T16:09:40.304914+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e269: 12 total, 12 up, 12 in 2024-02-12T16:09:40.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:40.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:40.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:40 smithi123 bash[19119]: cluster 2024-02-12T16:09:39.633100+0000 mgr.y (mgr.14146) 727 : cluster [DBG] pgmap v898: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-12T16:09:40.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:40 smithi123 bash[19119]: cluster 2024-02-12T16:09:39.695564+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e268: 12 total, 12 up, 12 in 2024-02-12T16:09:40.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:40 smithi123 bash[19119]: cluster 2024-02-12T16:09:40.304914+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e269: 12 total, 12 up, 12 in 2024-02-12T16:09:40.943 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:40.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:40.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 91171 91155 178.023 108.5 0.00480947 0.00561576 2024-02-12T16:09:41.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:40 smithi169 bash[19604]: cluster 2024-02-12T16:09:39.633100+0000 mgr.y (mgr.14146) 727 : cluster [DBG] pgmap v898: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-12T16:09:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:40 smithi169 bash[19604]: cluster 2024-02-12T16:09:39.695564+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e268: 12 total, 12 up, 12 in 2024-02-12T16:09:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:40 smithi169 bash[19604]: cluster 2024-02-12T16:09:40.304914+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e269: 12 total, 12 up, 12 in 2024-02-12T16:09:41.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:41.074 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:41.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:41.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:41.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:41.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:41.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:41.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:41.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:41.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:41.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:41.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:41.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:41.849 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:41.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 93140 93124 176.357 123.062 0.00516993 0.00566864 2024-02-12T16:09:41.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:41.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:41.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:42.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:42.123 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:42.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:42.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:42.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:42.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:42.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:42.386 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:42 smithi123 bash[19119]: cluster 2024-02-12T16:09:40.693314+0000 osd.4 (osd.4) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:42 smithi123 bash[19119]: cluster 2024-02-12T16:09:40.693355+0000 osd.4 (osd.4) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:42 smithi123 bash[19119]: cluster 2024-02-12T16:09:40.973153+0000 osd.8 (osd.8) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:42 smithi123 bash[19119]: cluster 2024-02-12T16:09:40.973207+0000 osd.8 (osd.8) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:42 smithi123 bash[19119]: cluster 2024-02-12T16:09:41.311473+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e270: 12 total, 12 up, 12 in 2024-02-12T16:09:42.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:42 smithi123 bash[19119]: cluster 2024-02-12T16:09:41.633433+0000 mgr.y (mgr.14146) 728 : cluster [DBG] pgmap v902: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-02-12T16:09:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:42.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:42.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:42 smithi121 bash[18489]: cluster 2024-02-12T16:09:40.693314+0000 osd.4 (osd.4) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:42.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:42 smithi121 bash[18489]: cluster 2024-02-12T16:09:40.693355+0000 osd.4 (osd.4) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:42.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:42 smithi121 bash[18489]: cluster 2024-02-12T16:09:40.973153+0000 osd.8 (osd.8) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:42.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:42 smithi121 bash[18489]: cluster 2024-02-12T16:09:40.973207+0000 osd.8 (osd.8) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:42.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:42 smithi121 bash[18489]: cluster 2024-02-12T16:09:41.311473+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e270: 12 total, 12 up, 12 in 2024-02-12T16:09:42.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:42 smithi121 bash[18489]: cluster 2024-02-12T16:09:41.633433+0000 mgr.y (mgr.14146) 728 : cluster [DBG] pgmap v902: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-02-12T16:09:42.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:42.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:42.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:42.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:42.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:42.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:42.789 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:42.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:42.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:42 smithi169 bash[19604]: cluster 2024-02-12T16:09:40.693314+0000 osd.4 (osd.4) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:42 smithi169 bash[19604]: cluster 2024-02-12T16:09:40.693355+0000 osd.4 (osd.4) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:42 smithi169 bash[19604]: cluster 2024-02-12T16:09:40.973153+0000 osd.8 (osd.8) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:42 smithi169 bash[19604]: cluster 2024-02-12T16:09:40.973207+0000 osd.8 (osd.8) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:42 smithi169 bash[19604]: cluster 2024-02-12T16:09:41.311473+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e270: 12 total, 12 up, 12 in 2024-02-12T16:09:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:42 smithi169 bash[19604]: cluster 2024-02-12T16:09:41.633433+0000 mgr.y (mgr.14146) 728 : cluster [DBG] pgmap v902: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-02-12T16:09:42.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:42.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:42.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:42.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 95032 95016 174.648 118.25 0.00328905 0.00572403 2024-02-12T16:09:43.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:43.049 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:43.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:43.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:43.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:43.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:43.460 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:43.528 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:09:43.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:43.594 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:43.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:43.720 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:43.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:43.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:43.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 96821 96805 172.852 111.812 0.00353416 0.00578355 2024-02-12T16:09:43.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:43.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:43.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:44.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:44.111 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:44.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:44.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:44.240 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:44.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:44.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:44.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:44.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:44.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:44.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:44.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:44.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:44.647 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:44.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:44.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:44.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:44.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:44.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:44.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:44 smithi121 bash[18489]: cluster 2024-02-12T16:09:43.633835+0000 mgr.y (mgr.14146) 729 : cluster [DBG] pgmap v903: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:09:44.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:44 smithi121 bash[18489]: cluster 2024-02-12T16:09:43.750063+0000 osd.7 (osd.7) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:44.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:44 smithi121 bash[18489]: cluster 2024-02-12T16:09:43.750117+0000 osd.7 (osd.7) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:44.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:44.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 16 98650 98634 171.226 114.312 0.00854946 0.00583879 2024-02-12T16:09:45.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:45.034 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:44 smithi123 bash[19119]: cluster 2024-02-12T16:09:43.633835+0000 mgr.y (mgr.14146) 729 : cluster [DBG] pgmap v903: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:09:45.034 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:44 smithi123 bash[19119]: cluster 2024-02-12T16:09:43.750063+0000 osd.7 (osd.7) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:45.034 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:44 smithi123 bash[19119]: cluster 2024-02-12T16:09:43.750117+0000 osd.7 (osd.7) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:44 smithi169 bash[19604]: cluster 2024-02-12T16:09:43.633835+0000 mgr.y (mgr.14146) 729 : cluster [DBG] pgmap v903: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:09:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:44 smithi169 bash[19604]: cluster 2024-02-12T16:09:43.750063+0000 osd.7 (osd.7) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:44 smithi169 bash[19604]: cluster 2024-02-12T16:09:43.750117+0000 osd.7 (osd.7) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:45.054 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:45.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:45.182 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:45.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:45.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:45.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:45.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:45.578 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:45.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:45.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:45.836 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:45.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:45 smithi123 bash[19119]: cluster 2024-02-12T16:09:44.693299+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e271: 12 total, 12 up, 12 in 2024-02-12T16:09:45.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:45 smithi123 bash[19119]: cluster 2024-02-12T16:09:45.302950+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e272: 12 total, 12 up, 12 in 2024-02-12T16:09:45.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:45.946 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:45 smithi169 bash[19604]: cluster 2024-02-12T16:09:44.693299+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e271: 12 total, 12 up, 12 in 2024-02-12T16:09:45.946 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:45 smithi169 bash[19604]: cluster 2024-02-12T16:09:45.302950+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e272: 12 total, 12 up, 12 in 2024-02-12T16:09:45.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 100417 100401 169.582 110.438 0.00465301 0.00589441 2024-02-12T16:09:45.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:45.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:45.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:46.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:46.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:46.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:46.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:45 smithi121 bash[18489]: cluster 2024-02-12T16:09:44.693299+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e271: 12 total, 12 up, 12 in 2024-02-12T16:09:46.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:45 smithi121 bash[18489]: cluster 2024-02-12T16:09:45.302950+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e272: 12 total, 12 up, 12 in 2024-02-12T16:09:46.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:46.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:46.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:46.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:46.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:46.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:46.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:46.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:46.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:46.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:46.613 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:46.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:46.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:46.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:46.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:46.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:46.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:46.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:46.953 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 16 102040 102024 167.789 101.438 0.0064603 0.00595828 2024-02-12T16:09:46.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:46.976 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:46 smithi121 bash[18489]: cluster 2024-02-12T16:09:45.634164+0000 mgr.y (mgr.14146) 730 : cluster [DBG] pgmap v906: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:09:46.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:46 smithi169 bash[19604]: cluster 2024-02-12T16:09:45.634164+0000 mgr.y (mgr.14146) 730 : cluster [DBG] pgmap v906: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:09:47.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:47.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:47.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:46 smithi123 bash[19119]: cluster 2024-02-12T16:09:45.634164+0000 mgr.y (mgr.14146) 730 : cluster [DBG] pgmap v906: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:09:47.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:47.266 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:47.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:47.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:47.500 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:09:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:47.538 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:09:47.575 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:09:47.607 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:09:47.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:47.647 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:09:47.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:09:47.686 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:47.733 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:09:47.760 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:09:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:09:47.776 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:09:47.790 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:09:47.793 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:47.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:47 smithi123 bash[19119]: audit 2024-02-12T16:09:47.445966+0000 mon.a (mon.0) 1157 : audit [DBG] from='client.? v1:172.21.15.121:0/907949899' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:47.819 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:47.829 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:09:47.864 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:09:47.893 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.906 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.906 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.906 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:09:47.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:47 smithi121 bash[18489]: audit 2024-02-12T16:09:47.445966+0000 mon.a (mon.0) 1157 : audit [DBG] from='client.? v1:172.21.15.121:0/907949899' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:47.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:47.940 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:09:47.940 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:09:47.940 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:09:47.941 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:09:47.941 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:09:47.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:47.954 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 103596 103580 165.98 97.25 0.00727667 0.00602337 2024-02-12T16:09:47.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:47.993 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:47 smithi169 bash[19604]: audit 2024-02-12T16:09:47.445966+0000 mon.a (mon.0) 1157 : audit [DBG] from='client.? v1:172.21.15.121:0/907949899' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:09:48.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:48.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:48.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:48.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:48.209 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:48.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:48.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:48.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:48.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:48.469 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:48.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:48.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:48.600 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:48.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:48.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:09:48.733 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:48.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:48.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-12T16:09:48.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:48.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:48.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:48.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:09:48.953304 min lat: 0.000952532 max lat: 0.346497 avg lat: 0.00607951 2024-02-12T16:09:48.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:09:48.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 105267 105251 164.433 104.438 0.0167083 0.00607951 2024-02-12T16:09:49.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:49.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:48 smithi169 bash[19604]: cluster 2024-02-12T16:09:47.067423+0000 osd.2 (osd.2) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:49.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:48 smithi169 bash[19604]: cluster 2024-02-12T16:09:47.067488+0000 osd.2 (osd.2) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:49.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:48 smithi169 bash[19604]: cluster 2024-02-12T16:09:47.634586+0000 mgr.y (mgr.14146) 731 : cluster [DBG] pgmap v907: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:09:49.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:48 smithi169 bash[19604]: cluster 2024-02-12T16:09:47.702125+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e273: 12 total, 12 up, 12 in 2024-02-12T16:09:49.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:48 smithi169 bash[19604]: audit 2024-02-12T16:09:47.752873+0000 mgr.y (mgr.14146) 732 : audit [DBG] from='client.15156 v1:172.21.15.121:0/1287913865' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:49.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:48 smithi169 bash[19604]: audit 2024-02-12T16:09:47.768584+0000 mon.a (mon.0) 1159 : audit [INF] from='client.25061 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:49.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:48 smithi169 bash[19604]: audit 2024-02-12T16:09:47.772066+0000 mon.b (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.121:0/1161894110' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:49.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:49.135 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:48 smithi123 bash[19119]: cluster 2024-02-12T16:09:47.067423+0000 osd.2 (osd.2) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:48 smithi123 bash[19119]: cluster 2024-02-12T16:09:47.067488+0000 osd.2 (osd.2) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:48 smithi123 bash[19119]: cluster 2024-02-12T16:09:47.634586+0000 mgr.y (mgr.14146) 731 : cluster [DBG] pgmap v907: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:09:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:48 smithi123 bash[19119]: cluster 2024-02-12T16:09:47.702125+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e273: 12 total, 12 up, 12 in 2024-02-12T16:09:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:48 smithi123 bash[19119]: audit 2024-02-12T16:09:47.752873+0000 mgr.y (mgr.14146) 732 : audit [DBG] from='client.15156 v1:172.21.15.121:0/1287913865' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:48 smithi123 bash[19119]: audit 2024-02-12T16:09:47.768584+0000 mon.a (mon.0) 1159 : audit [INF] from='client.25061 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:48 smithi123 bash[19119]: audit 2024-02-12T16:09:47.772066+0000 mon.b (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.121:0/1161894110' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:49.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:48 smithi121 bash[18489]: cluster 2024-02-12T16:09:47.067423+0000 osd.2 (osd.2) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:09:49.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:48 smithi121 bash[18489]: cluster 2024-02-12T16:09:47.067488+0000 osd.2 (osd.2) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:09:49.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:48 smithi121 bash[18489]: cluster 2024-02-12T16:09:47.634586+0000 mgr.y (mgr.14146) 731 : cluster [DBG] pgmap v907: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:09:49.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:48 smithi121 bash[18489]: cluster 2024-02-12T16:09:47.702125+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e273: 12 total, 12 up, 12 in 2024-02-12T16:09:49.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:48 smithi121 bash[18489]: audit 2024-02-12T16:09:47.752873+0000 mgr.y (mgr.14146) 732 : audit [DBG] from='client.15156 v1:172.21.15.121:0/1287913865' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:09:49.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:48 smithi121 bash[18489]: audit 2024-02-12T16:09:47.768584+0000 mon.a (mon.0) 1159 : audit [INF] from='client.25061 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:49.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:48 smithi121 bash[18489]: audit 2024-02-12T16:09:47.772066+0000 mon.b (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.121:0/1161894110' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:09:49.164 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:49.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:49.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:49.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:49.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:49.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:49.556 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":908,"stamp":"2024-02-12T16:09:47.698757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6456410112,"num_objects":98517,"num_object_clones":0,"num_object_copies":197034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98517,"num_write_kb":6305088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98517,"ondisk_log_size":98517,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":16518556,"kb_used_data":12841276,"kb_used_omap":0,"kb_used_meta":3677184,"kb_avail":1108423268,"statfs":{"total":1151940427776,"available":1135025426432,"internally_reserved":0,"allocated":13149466624,"data_stored":13144363405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3765436416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":725221376,"num_objects":11066,"num_object_clones":0,"num_object_copies":22132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11066,"num_write_kb":708224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.065482"},"pg_stats":[{"pgid":"3.f","version":"270'6040","reported_seq":6061,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:43.661547+0000","last_change":"2024-02-12T16:08:56.026943+0000","last_active":"2024-02-12T16:09:43.661547+0000","last_peered":"2024-02-12T16:09:43.661547+0000","last_clean":"2024-02-12T16:09:43.661547+0000","last_became_active":"2024-02-12T16:08:56.025977+0000","last_became_peered":"2024-02-12T16:08:56.025977+0000","last_unstale":"2024-02-12T16:09:43.661547+0000","last_undegraded":"2024-02-12T16:09:43.661547+0000","last_fullsized":"2024-02-12T16:09:43.661547+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6040,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":395837440,"num_objects":6040,"num_object_clones":0,"num_object_copies":12080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":386560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"271'6086","reported_seq":6107,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-02-12T16:09:44.987895+0000","last_change":"2024-02-12T16:08:56.029433+0000","last_active":"2024-02-12T16:09:44.987895+0000","last_peered":"2024-02-12T16:09:44.987895+0000","last_clean":"2024-02-12T16:09:44.987895+0000","last_became_active":"2024-02-12T16:08:56.029081+0000","last_became_peered":"2024-02-12T16:08:56.029081+0000","last_unstale":"2024-02-12T16:09:44.987895+0000","last_undegraded":"2024-02-12T16:09:44.987895+0000","last_fullsized":"2024-02-12T16:09:44.987895+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6086,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":398852096,"num_objects":6086,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6086,"num_write_kb":389504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"270'5959","reported_seq":5980,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:43.658235+0000","last_change":"2024-02-12T16:08:56.026138+0000","last_active":"2024-02-12T16:09:43.658235+0000","last_peered":"2024-02-12T16:09:43.658235+0000","last_clean":"2024-02-12T16:09:43.658235+0000","last_became_active":"2024-02-12T16:08:56.025860+0000","last_became_peered":"2024-02-12T16:08:56.025860+0000","last_unstale":"2024-02-12T16:09:43.658235+0000","last_undegraded":"2024-02-12T16:09:43.658235+0000","last_fullsized":"2024-02-12T16:09:43.658235+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":5959,"ondisk_log_size":5959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":390529024,"num_objects":5959,"num_object_clones":0,"num_object_copies":11918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5959,"num_write_kb":381376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"272'6491","reported_seq":6514,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-02-12T16:09:46.896131+0000","last_change":"2024-02-12T16:08:56.027997+0000","last_active":"2024-02-12T16:09:46.896131+0000","last_peered":"2024-02-12T16:09:46.896131+0000","last_clean":"2024-02-12T16:09:46.896131+0000","last_became_active":"2024-02-12T16:08:56.027658+0000","last_became_peered":"2024-02-12T16:08:56.027658+0000","last_unstale":"2024-02-12T16:09:46.896131+0000","last_undegraded":"2024-02-12T16:09:46.896131+0000","last_fullsized":"2024-02-12T16:09:46.896131+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6491,"ondisk_log_size":6491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":425394176,"num_objects":6491,"num_object_clones":0,"num_object_copies":12982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"270'6103","reported_seq":6123,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:44.382663+0000","last_change":"2024-02-12T16:08:56.021048+0000","last_active":"2024-02-12T16:09:44.382663+0000","last_peered":"2024-02-12T16:09:44.382663+0000","last_clean":"2024-02-12T16:09:44.382663+0000","last_became_active":"2024-02-12T16:08:56.020761+0000","last_became_peered":"2024-02-12T16:08:56.020761+0000","last_unstale":"2024-02-12T16:09:44.382663+0000","last_undegraded":"2024-02-12T16:09:44.382663+0000","last_fullsized":"2024-02-12T16:09:44.382663+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6103,"ondisk_log_size":6103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":399966208,"num_objects":6103,"num_object_clones":0,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6103,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"270'6023","reported_seq":6041,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:43.903714+0000","last_change":"2024-02-12T16:08:56.027375+0000","last_active":"2024-02-12T16:09:43.903714+0000","last_peered":"2024-02-12T16:09:43.903714+0000","last_clean":"2024-02-12T16:09:43.903714+0000","last_became_active":"2024-02-12T16:08:56.027087+0000","last_became_peered":"2024-02-12T16:08:56.027087+0000","last_unstale":"2024-02-12T16:09:43.903714+0000","last_undegraded":"2024-02-12T16:09:43.903714+0000","last_fullsized":"2024-02-12T16:09:43.903714+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6023,"ondisk_log_size":6023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":394723328,"num_objects":6023,"num_object_clones":0,"num_object_copies":12046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6023,"num_write_kb":385472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"270'6118","reported_seq":6139,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:43.917480+0000","last_change":"2024-02-12T16:08:56.024538+0000","last_active":"2024-02-12T16:09:43.917480+0000","last_peered":"2024-02-12T16:09:43.917480+0000","last_clean":"2024-02-12T16:09:43.917480+0000","last_became_active":"2024-02-12T16:08:56.024064+0000","last_became_peered":"2024-02-12T16:08:56.024064+0000","last_unstale":"2024-02-12T16:09:43.917480+0000","last_undegraded":"2024-02-12T16:09:43.917480+0000","last_fullsized":"2024-02-12T16:09:43.917480+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6118,"ondisk_log_size":6118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":400949248,"num_objects":6118,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":391552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"271'5979","reported_seq":6000,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-02-12T16:09:44.970560+0000","last_change":"2024-02-12T16:08:56.026436+0000","last_active":"2024-02-12T16:09:44.970560+0000","last_peered":"2024-02-12T16:09:44.970560+0000","last_clean":"2024-02-12T16:09:44.970560+0000","last_became_active":"2024-02-12T16:08:56.026173+0000","last_became_peered":"2024-02-12T16:08:56.026173+0000","last_unstale":"2024-02-12T16:09:44.970560+0000","last_undegraded":"2024-02-12T16:09:44.970560+0000","last_fullsized":"2024-02-12T16:09:44.970560+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":5979,"ondisk_log_size":5979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391839744,"num_objects":5979,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":382656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"270'6083","reported_seq":6103,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:44.398475+0000","last_change":"2024-02-12T16:08:56.022243+0000","last_active":"2024-02-12T16:09:44.398475+0000","last_peered":"2024-02-12T16:09:44.398475+0000","last_clean":"2024-02-12T16:09:44.398475+0000","last_became_active":"2024-02-12T16:08:56.021958+0000","last_became_peered":"2024-02-12T16:08:56.021958+0000","last_unstale":"2024-02-12T16:09:44.398475+0000","last_undegraded":"2024-02-12T16:09:44.398475+0000","last_fullsized":"2024-02-12T16:09:44.398475+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6083,"ondisk_log_size":6083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":398655488,"num_objects":6083,"num_object_clones":0,"num_object_copies":12166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":389312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"271'6281","reported_seq":6303,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-02-12T16:09:45.125953+0000","last_change":"2024-02-12T16:08:56.020936+0000","last_active":"2024-02-12T16:09:45.125953+0000","last_peered":"2024-02-12T16:09:45.125953+0000","last_clean":"2024-02-12T16:09:45.125953+0000","last_became_active":"2024-02-12T16:08:56.020715+0000","last_became_peered":"2024-02-12T16:08:56.020715+0000","last_unstale":"2024-02-12T16:09:45.125953+0000","last_undegraded":"2024-02-12T16:09:45.125953+0000","last_fullsized":"2024-02-12T16:09:45.125953+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6281,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":411631616,"num_objects":6281,"num_object_clones":0,"num_object_copies":12562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6281,"num_write_kb":401984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"272'6363","reported_seq":6386,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-02-12T16:09:46.865314+0000","last_change":"2024-02-12T16:08:56.026672+0000","last_active":"2024-02-12T16:09:46.865314+0000","last_peered":"2024-02-12T16:09:46.865314+0000","last_clean":"2024-02-12T16:09:46.865314+0000","last_became_active":"2024-02-12T16:08:56.026413+0000","last_became_peered":"2024-02-12T16:08:56.026413+0000","last_unstale":"2024-02-12T16:09:46.865314+0000","last_undegraded":"2024-02-12T16:09:46.865314+0000","last_fullsized":"2024-02-12T16:09:46.865314+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6363,"ondisk_log_size":6363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":417005568,"num_objects":6363,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"272'6363","reported_seq":6386,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-02-12T16:09:46.645338+0000","last_change":"2024-02-12T16:08:56.021364+0000","last_active":"2024-02-12T16:09:46.645338+0000","last_peered":"2024-02-12T16:09:46.645338+0000","last_clean":"2024-02-12T16:09:46.645338+0000","last_became_active":"2024-02-12T16:08:56.021115+0000","last_became_peered":"2024-02-12T16:08:56.021115+0000","last_unstale":"2024-02-12T16:09:46.645338+0000","last_undegraded":"2024-02-12T16:09:46.645338+0000","last_fullsized":"2024-02-12T16:09:46.645338+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6363,"ondisk_log_size":6363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":417005568,"num_objects":6363,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"270'6095","reported_seq":6114,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:44.142506+0000","last_change":"2024-02-12T16:08:56.026095+0000","last_active":"2024-02-12T16:09:44.142506+0000","last_peered":"2024-02-12T16:09:44.142506+0000","last_clean":"2024-02-12T16:09:44.142506+0000","last_became_active":"2024-02-12T16:08:56.025422+0000","last_became_peered":"2024-02-12T16:08:56.025422+0000","last_unstale":"2024-02-12T16:09:44.142506+0000","last_undegraded":"2024-02-12T16:09:44.142506+0000","last_fullsized":"2024-02-12T16:09:44.142506+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6095,"ondisk_log_size":6095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":399441920,"num_objects":6095,"num_object_clones":0,"num_object_copies":12190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6095,"num_write_kb":390080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"271'6205","reported_seq":6227,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-02-12T16:09:45.140058+0000","last_change":"2024-02-12T16:08:56.022404+0000","last_active":"2024-02-12T16:09:45.140058+0000","last_peered":"2024-02-12T16:09:45.140058+0000","last_clean":"2024-02-12T16:09:45.140058+0000","last_became_active":"2024-02-12T16:08:56.022130+0000","last_became_peered":"2024-02-12T16:08:56.022130+0000","last_unstale":"2024-02-12T16:09:45.140058+0000","last_undegraded":"2024-02-12T16:09:45.140058+0000","last_fullsized":"2024-02-12T16:09:45.140058+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6205,"ondisk_log_size":6205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":406650880,"num_objects":6205,"num_object_clones":0,"num_object_copies":12410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6205,"num_write_kb":397120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"271'6278","reported_seq":6299,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-02-12T16:09:44.971029+0000","last_change":"2024-02-12T16:08:56.275603+0000","last_active":"2024-02-12T16:09:44.971029+0000","last_peered":"2024-02-12T16:09:44.971029+0000","last_clean":"2024-02-12T16:09:44.971029+0000","last_became_active":"2024-02-12T16:08:56.275308+0000","last_became_peered":"2024-02-12T16:08:56.275308+0000","last_unstale":"2024-02-12T16:09:44.971029+0000","last_undegraded":"2024-02-12T16:09:44.971029+0000","last_fullsized":"2024-02-12T16:09:44.971029+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6278,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":411435008,"num_objects":6278,"num_object_clones":0,"num_object_copies":12556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":401792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"270'6050","reported_seq":6070,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-02-12T16:09:44.391635+0000","last_change":"2024-02-12T16:08:56.274192+0000","last_active":"2024-02-12T16:09:44.391635+0000","last_peered":"2024-02-12T16:09:44.391635+0000","last_clean":"2024-02-12T16:09:44.391635+0000","last_became_active":"2024-02-12T16:08:56.273670+0000","last_became_peered":"2024-02-12T16:08:56.273670+0000","last_unstale":"2024-02-12T16:09:44.391635+0000","last_undegraded":"2024-02-12T16:09:44.391635+0000","last_fullsized":"2024-02-12T16:09:44.391635+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":256,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:08:55.005799+0000","last_clean_scrub_stamp":"2024-02-12T16:08:55.005799+0000","log_size":6050,"ondisk_log_size":6050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":396492800,"num_objects":6050,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6050,"num_write_kb":387200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":289,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-02-12T16:09:40.309389+0000","last_change":"2024-02-12T16:09:31.471108+0000","last_active":"2024-02-12T16:09:40.309389+0000","last_peered":"2024-02-12T16:09:40.309389+0000","last_clean":"2024-02-12T16:09:40.309389+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:09:40.309389+0000","last_undegraded":"2024-02-12T16:09:40.309389+0000","last_fullsized":"2024-02-12T16:09:40.309389+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:09:31.470997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:09:31.470997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6456410112,"num_objects":98517,"num_object_clones":0,"num_object_copies":197034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98517,"num_write_kb":6305088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12956467200,"data_stored":12956467200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98517,"ondisk_log_size":98517,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958667,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3065936,"kb_used_data":2756424,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":90679216,"statfs":{"total":95995035648,"available":92855517184,"internally_reserved":0,"allocated":2822578176,"data_stored":2822151386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.745,"15min":0.93200000000000005},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":3.3929999999999998,"5min":4.7850000000000001,"15min":4.7850000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.91100000000000003,"15min":1.0389999999999999},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.1909999999999998,"5min":3.8650000000000002,"15min":3.8650000000000002},"last":0.378}]},{"osd":1,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":1.2270000000000001,"15min":1.1220000000000001},"min":{"1min":0.47399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.7569999999999997,"5min":6.7569999999999997,"15min":6.7569999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.228,"5min":1.109,"15min":1.093},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.786,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0980000000000001,"15min":1.1259999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.7109999999999999,"5min":5.274,"15min":5.274},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":0.96999999999999997,"15min":1.1160000000000001},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":3.7360000000000002,"5min":5.375,"15min":5.375},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.077,"15min":1.1499999999999999},"min":{"1min":0.33200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.712,"5min":4.266,"15min":4.266},"last":0.68500000000000005},{"interface":"front","average":{"1min":2.242,"5min":1.3320000000000001,"15min":1.224},"min":{"1min":0.34799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":8.9819999999999993,"5min":8.9819999999999993,"15min":8.9819999999999993},"last":2.2309999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.0660000000000001,"15min":1.2310000000000001},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6420000000000003,"5min":6.6420000000000003,"15min":6.6420000000000003},"last":0.442},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.83199999999999996,"15min":1.21},"min":{"1min":0.39300000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.788,"5min":3.1299999999999999,"15min":4.0789999999999997},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.95799999999999996,"15min":1.284},"min":{"1min":0.376,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.474,"5min":4.0759999999999996,"15min":4.0759999999999996},"last":0.621},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.1839999999999999,"15min":1.367},"min":{"1min":0.45900000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":6.6849999999999996,"5min":6.6849999999999996,"15min":6.6849999999999996},"last":0.47499999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.97599999999999998,"15min":1.3120000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.105,"5min":3.2290000000000001,"15min":4.4989999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.0509999999999999,"15min":1.401},"min":{"1min":0.501,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":3.7709999999999999,"5min":3.7709999999999999,"15min":3.7709999999999999},"last":0.752}]},{"osd":7,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.006,"15min":1.389},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.909,"5min":3.4710000000000001,"15min":4.3129999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.46},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.7589999999999999,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":0.65600000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.054,"15min":1.423},"min":{"1min":0.42399999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.6579999999999999,"5min":3.6579999999999999,"15min":4},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.048,"15min":1.4750000000000001},"min":{"1min":0.40999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.7370000000000001,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.70999999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.131,"15min":1.46},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.903,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.093,"15min":1.5549999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.73,"5min":4.6689999999999996,"15min":4.6689999999999996},"last":0.58699999999999997}]}]},{"osd":10,"up_from":83,"seq":356482285716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":441220,"kb_used_data":143996,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93303932,"statfs":{"total":95995035648,"available":95543226368,"internally_reserved":0,"allocated":147451904,"data_stored":147031201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.97799999999999998,"15min":0.92500000000000004},"min":{"1min":0.38900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.0739999999999998,"5min":5.3650000000000002,"15min":5.3650000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.96599999999999997,"15min":1.018},"min":{"1min":0.32600000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.6070000000000002,"5min":7.6580000000000004,"15min":7.6580000000000004},"last":4.0890000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85299999999999998,"15min":0.94299999999999995},"min":{"1min":0.309,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.4240000000000004,"5min":4.8300000000000001,"15min":4.8300000000000001},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0369999999999999,"15min":0.99199999999999999},"min":{"1min":0.377,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.621,"5min":7.9829999999999997,"15min":7.9829999999999997},"last":0.34499999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.022,"15min":1.02},"min":{"1min":0.36699999999999999,"5min":0.308,"15min":0.308},"max":{"1min":3.1509999999999998,"5min":4.8620000000000001,"15min":4.8620000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.95499999999999996,"15min":0.98499999999999999},"min":{"1min":0.35599999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.7749999999999999,"5min":5.4100000000000001,"15min":5.4100000000000001},"last":3.8999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.88200000000000001,"15min":1.069},"min":{"1min":0.40999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.605,"5min":4.9189999999999996,"15min":4.9189999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.88300000000000001,"15min":1.1060000000000001},"min":{"1min":0.48399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.704,"5min":4.5919999999999996,"15min":4.5919999999999996},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":7.2210000000000001,"15min":3.1909999999999998},"min":{"1min":0.42399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.8849999999999998,"5min":602.05399999999997,"15min":602.05399999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.1499999999999999,"5min":7.2750000000000004,"15min":3.2949999999999999},"min":{"1min":0.59799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.825,"5min":598.875,"15min":598.875},"last":0.55100000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.98199999999999998,"15min":1.1319999999999999},"min":{"1min":0.39600000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.8519999999999999,"5min":5.4729999999999999,"15min":5.4729999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.012,"15min":1.2529999999999999},"min":{"1min":0.53500000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.8140000000000001,"5min":4.6660000000000004,"15min":4.6660000000000004},"last":0.44}]},{"osd":7,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":7.3570000000000002,"15min":3.3330000000000002},"min":{"1min":0.50900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":5.3220000000000001,"5min":597.48299999999995,"15min":597.48299999999995},"last":0.5},{"interface":"front","average":{"1min":1.1970000000000001,"5min":7.4100000000000001,"15min":3.3809999999999998},"min":{"1min":0.57199999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":3.8149999999999999,"5min":600.23099999999999,"15min":600.23099999999999},"last":0.47199999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.157,"15min":1.2949999999999999},"min":{"1min":0.51000000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.0640000000000001,"5min":7.2720000000000002,"15min":7.2720000000000002},"last":0.37},{"interface":"front","average":{"1min":1.403,"5min":1.1919999999999999,"15min":1.327},"min":{"1min":0.63700000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.278,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":0.38200000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:09:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.1160000000000001,"15min":1.3580000000000001},"min":{"1min":0.67100000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.8859999999999999,"5min":4.7089999999999996,"15min":4.7089999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.1200000000000001,"15min":1.345},"min":{"1min":0.60799999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.0309999999999997,"5min":4.7789999999999999,"15min":4.7789999999999999},"last":0.64800000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.2190000000000001,"15min":1.468},"min":{"1min":0.57699999999999996,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":5.8730000000000002,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1659999999999999,"15min":1.4690000000000001},"min":{"1min":0.58599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":4.1479999999999997,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":0.48499999999999999}]}]},{"osd":9,"up_from":74,"seq":317827580061,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1108828,"kb_used_data":803412,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92636324,"statfs":{"total":95995035648,"available":94859595776,"internally_reserved":0,"allocated":822693888,"data_stored":822260981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.79300000000000004,"15min":0.88500000000000001},"min":{"1min":0.31900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":1.198,"5min":10.788,"15min":10.788},"last":1.76},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.93400000000000005,"15min":0.94799999999999995},"min":{"1min":0.308,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.8680000000000001,"5min":7.9630000000000001,"15min":7.9630000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":7.694,"5min":2.7599999999999998,"15min":1.6100000000000001},"min":{"1min":0.29399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":175.33500000000001,"5min":175.33500000000001,"15min":175.33500000000001},"last":1.032},{"interface":"front","average":{"1min":7.6369999999999996,"5min":2.6869999999999998,"15min":1.5860000000000001},"min":{"1min":0.33900000000000002,"5min":0.317,"15min":0.317},"max":{"1min":175.24299999999999,"5min":175.24299999999999,"15min":175.24299999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.2390000000000001,"15min":1.0389999999999999},"min":{"1min":0.52900000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.2370000000000001,"5min":7.8140000000000001,"15min":7.8140000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.256,"15min":1.0820000000000001},"min":{"1min":0.37,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.3479999999999999,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":1.1120000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":2.2080000000000002,"5min":1.3720000000000001,"15min":1.099},"min":{"1min":0.42699999999999999,"5min":0.308,"15min":0.308},"max":{"1min":34.591000000000001,"5min":34.591000000000001,"15min":34.591000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":2.3500000000000001,"5min":1.502,"15min":1.2410000000000001},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":34.494,"5min":34.494,"15min":34.494},"last":0.84899999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.3240000000000001,"15min":1.2170000000000001},"min":{"1min":0.41399999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.391,"5min":25.552,"15min":25.552},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1020000000000001,"15min":1.1519999999999999},"min":{"1min":0.40699999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4199999999999999,"5min":25.847999999999999,"15min":25.847999999999999},"last":1.075}]},{"osd":5,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":7.9480000000000004,"15min":3.4700000000000002},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.4630000000000001,"5min":617.95899999999995,"15min":617.95899999999995},"last":1.145},{"interface":"front","average":{"1min":1.2250000000000001,"5min":7.9279999999999999,"15min":3.5110000000000001},"min":{"1min":0.41599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.5860000000000001,"5min":617.88099999999997,"15min":617.88099999999997},"last":0.92000000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.1000000000000001,"15min":1.244},"min":{"1min":0.49199999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.754,"5min":5.0960000000000001,"15min":5.0960000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.254,"5min":0.88900000000000001,"15min":1.1850000000000001},"min":{"1min":0.46500000000000002,"5min":0.377,"15min":0.377},"max":{"1min":2.2719999999999998,"5min":3.4900000000000002,"15min":3.4900000000000002},"last":1.01}]},{"osd":7,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":11.624000000000001,"15min":4.7409999999999997},"min":{"1min":0.78500000000000003,"5min":0.379,"15min":0.379},"max":{"1min":4.5030000000000001,"5min":613.28200000000004,"15min":613.28200000000004},"last":1.175},{"interface":"front","average":{"1min":1.3009999999999999,"5min":11.571,"15min":4.7999999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.825,"5min":613.25199999999995,"15min":613.25199999999995},"last":0.99399999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.2490000000000001,"15min":1.325},"min":{"1min":0.55700000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.3330000000000002,"5min":4.96,"15min":4.96},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.238,"5min":0.99199999999999999,"15min":1.238},"min":{"1min":0.57499999999999996,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.3559999999999999,"5min":3.4009999999999998,"15min":3.4009999999999998},"last":0.67800000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:09:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.0249999999999999,"15min":1.2889999999999999},"min":{"1min":0.49199999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.4219999999999997,"5min":5.5,"15min":5.5},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.202,"15min":1.3089999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.8839999999999999,"5min":4.7619999999999996,"15min":4.7619999999999996},"last":3.581}]}]},{"osd":8,"up_from":65,"seq":279172874403,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":700932,"kb_used_data":391420,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93044220,"statfs":{"total":95995035648,"available":95277281280,"internally_reserved":0,"allocated":400814080,"data_stored":400393397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.78500000000000003,"15min":0.85999999999999999},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.254,"5min":5.383,"15min":5.383},"last":0.442},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91300000000000003},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.2879999999999998,"5min":4.5330000000000004,"15min":4.5330000000000004},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":1.6319999999999999,"15min":1.9359999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.385,"5min":23.661000000000001,"15min":23.661000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.7170000000000001,"15min":2.0670000000000002},"min":{"1min":0.35399999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.5369999999999999,"5min":23.672000000000001,"15min":23.672000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":12.574,"5min":3.3340000000000001,"15min":1.696},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":193.929,"5min":193.929,"15min":193.929},"last":0.52300000000000002},{"interface":"front","average":{"1min":12.651,"5min":3.484,"15min":1.8380000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":193.82499999999999,"5min":193.82499999999999,"15min":193.82499999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.069,"15min":1.0489999999999999},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.3620000000000001,"5min":5.4020000000000001,"15min":5.4020000000000001},"last":1.216},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.0049999999999999,"15min":1.089},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.3519999999999999,"5min":5.3929999999999998,"15min":5.3929999999999998},"last":1.2270000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.89200000000000002,"15min":1.087},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":1.643,"5min":6.4619999999999997,"15min":6.4619999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.98199999999999998,"15min":1.1339999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4610000000000001,"5min":5.71,"15min":5.71},"last":0.32400000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.0129999999999999,"15min":1.107},"min":{"1min":0.37,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.371,"5min":13.961,"15min":13.961},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0389999999999999,"15min":1.1819999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.0379999999999998,"5min":13.103,"15min":13.103},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.95899999999999996,"15min":1.139},"min":{"1min":0.39900000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.5979999999999999,"5min":4.3440000000000003,"15min":4.3440000000000003},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.1240000000000001,"15min":1.26},"min":{"1min":0.39000000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.7350000000000001,"5min":4.4720000000000004,"15min":4.4720000000000004},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.1970000000000001,"15min":1.29},"min":{"1min":0.45600000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.4239999999999999,"5min":13.242000000000001,"15min":13.242000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.224,"15min":1.337},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.841,"5min":13.388,"15min":13.388},"last":0.48599999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:09:29 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.0640000000000001,"15min":1.268},"min":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.9359999999999999,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2050000000000001,"15min":1.248},"min":{"1min":0.59899999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":2.008,"5min":6.2000000000000002,"15min":6.2000000000000002},"last":0.76000000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.214,"15min":1.3220000000000001},"min":{"1min":0.57499999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.2670000000000003,"5min":5.4189999999999996,"15min":5.4189999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.48,"5min":1.0429999999999999,"15min":1.3340000000000001},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":3.4860000000000002,"5min":3.6779999999999999,"15min":3.6779999999999999},"last":0.61399999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:09:16 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":8.4209999999999994,"15min":27.193999999999999},"min":{"1min":0.58999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.8220000000000001,"5min":635.66399999999999,"15min":635.66399999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.4450000000000001,"5min":8.2390000000000008,"15min":26.914999999999999},"min":{"1min":0.59599999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":3.1000000000000001,"5min":636.28099999999995,"15min":636.28099999999995},"last":2.097}]}]},{"osd":7,"up_from":56,"seq":240518168748,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1075012,"kb_used_data":765500,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92670140,"statfs":{"total":95995035648,"available":94894223360,"internally_reserved":0,"allocated":783872000,"data_stored":783451317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.4670000000000001,"15min":1.1659999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.8380000000000001,"5min":60.152000000000001,"15min":60.152000000000001},"last":1.506},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.74,"15min":1.2210000000000001},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.3879999999999999,"5min":62.020000000000003,"15min":62.020000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.8069999999999999,"15min":1.27},"min":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.4249999999999998,"5min":60.204999999999998,"15min":60.204999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.641,"15min":1.2470000000000001},"min":{"1min":0.45600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.952,"5min":59.365000000000002,"15min":59.365000000000002},"last":0.28799999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.143,"15min":1.0980000000000001},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.1079999999999997,"5min":4.8200000000000003,"15min":4.8200000000000003},"last":2.7999999999999998},{"interface":"front","average":{"1min":0.997,"5min":1.095,"15min":1.075},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6879999999999999,"5min":5.1600000000000001,"15min":5.1600000000000001},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79400000000000004,"15min":0.91500000000000004},"min":{"1min":0.42399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.272,"5min":6.2519999999999998,"15min":6.2519999999999998},"last":3.843},{"interface":"front","average":{"1min":1.2190000000000001,"5min":0.871,"15min":0.99299999999999999},"min":{"1min":0.50900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2130000000000001,"5min":4.7430000000000003,"15min":4.7430000000000003},"last":0.73499999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.89800000000000002,"15min":1.0249999999999999},"min":{"1min":0.48599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.3480000000000001,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":1.345},{"interface":"front","average":{"1min":1.135,"5min":0.89600000000000002,"15min":1.0740000000000001},"min":{"1min":0.53100000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.899,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":1.7170000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0309999999999999,"15min":1.135},"min":{"1min":0.46400000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.411,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.1990000000000001,"5min":0.95399999999999996,"15min":1.1200000000000001},"min":{"1min":0.48499999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6180000000000001,"5min":4.7149999999999999,"15min":4.7149999999999999},"last":0.47999999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:09:39 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.2010000000000001,"15min":1.333},"min":{"1min":0.47999999999999998,"5min":0.432,"15min":0.432},"max":{"1min":5.5300000000000002,"5min":5.5300000000000002,"15min":5.5300000000000002},"last":0.432},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.194,"15min":1.3460000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.746,"5min":4.851,"15min":4.851},"last":1.601}]},{"osd":9,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":4.3070000000000004,"15min":2.395},"min":{"1min":0.52300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":2.2949999999999999,"5min":278.54599999999999,"15min":278.54599999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.284,"5min":4.2300000000000004,"15min":2.3610000000000002},"min":{"1min":0.51400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.8240000000000001,"5min":281.09199999999998,"15min":281.09199999999998},"last":0.77700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:09:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.1379999999999999,"15min":1.3919999999999999},"min":{"1min":0.61599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0430000000000001,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.214,"15min":1.4279999999999999},"min":{"1min":0.75,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.1059999999999999,"5min":4.9450000000000003,"15min":4.9450000000000003},"last":0.81299999999999994}]},{"osd":11,"last update":"Mon Feb 12 16:09:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":4.0759999999999996,"15min":2.3940000000000001},"min":{"1min":0.51500000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.9790000000000001,"5min":280.83999999999997,"15min":280.83999999999997},"last":1.669},{"interface":"front","average":{"1min":1.611,"5min":4.093,"15min":2.3730000000000002},"min":{"1min":0.64100000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":5.476,"5min":278.63799999999998,"15min":278.63799999999998},"last":1.639}]}]},{"osd":6,"up_from":48,"seq":206158430389,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":685892,"kb_used_data":384572,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93059260,"statfs":{"total":95995035648,"available":95292682240,"internally_reserved":0,"allocated":393801728,"data_stored":393381045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.4019999999999999,"15min":1.129},"min":{"1min":0.30199999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.6899999999999999,"5min":67.093000000000004,"15min":67.093000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.4370000000000001,"15min":1.1279999999999999},"min":{"1min":0.40200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.4770000000000001,"5min":68.549000000000007,"15min":68.549000000000007},"last":1.589}]},{"osd":1,"last update":"Mon Feb 12 16:09:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.6639999999999999,"15min":1.204},"min":{"1min":0.32200000000000001,"5min":0.308,"15min":0.308},"max":{"1min":2.931,"5min":66.936000000000007,"15min":66.936000000000007},"last":1.6299999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.6180000000000001,"15min":1.22},"min":{"1min":0.32800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.478,"5min":68.683000000000007,"15min":68.683000000000007},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:09:29 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":2.4169999999999998,"15min":1.4890000000000001},"min":{"1min":0.46600000000000003,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.452,"5min":68.462000000000003,"15min":68.462000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.0229999999999999,"5min":2.4350000000000001,"15min":1.524},"min":{"1min":0.378,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.6419999999999999,"5min":67.322999999999993,"15min":67.322999999999993},"last":1.657}]},{"osd":3,"last update":"Mon Feb 12 16:09:29 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.496,"15min":1.1719999999999999},"min":{"1min":0.45800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.173,"5min":68.332999999999998,"15min":68.332999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.5609999999999999,"15min":1.242},"min":{"1min":0.40000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.9650000000000001,"5min":67.281999999999996,"15min":67.281999999999996},"last":1.6799999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:09:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.458,"15min":1.153},"min":{"1min":0.36799999999999999,"5min":0.316,"15min":0.316},"max":{"1min":1.577,"5min":68.605999999999995,"15min":68.605999999999995},"last":0.499},{"interface":"front","average":{"1min":1.03,"5min":0.72699999999999998,"15min":0.95999999999999996},"min":{"1min":0.42199999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.5549999999999999,"5min":1.589,"15min":1.6459999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:09:29 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.5669999999999999,"15min":1.2729999999999999},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.597,"5min":69.260000000000005,"15min":69.260000000000005},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.391,"15min":1.23},"min":{"1min":0.373,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.6419999999999999,"5min":68.742000000000004,"15min":68.742000000000004},"last":1.0049999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":2.7349999999999999,"15min":1.7410000000000001},"min":{"1min":0.46100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.419,"5min":67.144999999999996,"15min":67.144999999999996},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":2.6819999999999999,"15min":1.758},"min":{"1min":0.54300000000000004,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.5139999999999998,"5min":67.230999999999995,"15min":67.230999999999995},"last":1.7030000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":4.7350000000000003,"5min":1.8779999999999999,"15min":1.5880000000000001},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":68.653000000000006,"5min":68.653000000000006,"15min":68.653000000000006},"last":0.67100000000000004},{"interface":"front","average":{"1min":4.8259999999999996,"5min":1.752,"15min":1.5589999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":68.620999999999995,"5min":68.620999999999995,"15min":68.620999999999995},"last":0.75600000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.7869999999999999,"15min":1.621},"min":{"1min":0.623,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.4769999999999999,"5min":67.188000000000002,"15min":67.188000000000002},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.7529999999999999,"15min":1.5600000000000001},"min":{"1min":0.61799999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.784,"5min":68.506,"15min":68.506},"last":0.65700000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.8280000000000001,"15min":1.702},"min":{"1min":0.63500000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.9019999999999999,"5min":68.722999999999999,"15min":68.722999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.476,"5min":1.6970000000000001,"15min":1.617},"min":{"1min":0.60299999999999998,"5min":0.505,"15min":0.505},"max":{"1min":2.0030000000000001,"5min":68.573999999999998,"15min":68.573999999999998},"last":0.76900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575068,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499740,"kb_used_data":1198420,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92245412,"statfs":{"total":95995035648,"available":94459301888,"internally_reserved":0,"allocated":1227182080,"data_stored":1226749173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":1.194,"15min":0.93799999999999994},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.101,"5min":9.8740000000000006,"15min":9.8740000000000006},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.214,"15min":0.998},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":1.3300000000000001,"5min":10.513,"15min":10.513},"last":1.9419999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.861,"15min":1.2110000000000001},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.1529999999999996,"5min":40.542999999999999,"15min":40.542999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.865,"5min":1.9750000000000001,"15min":1.258},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":9.2119999999999997,"5min":40.811999999999998,"15min":40.811999999999998},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.169,"15min":1.004},"min":{"1min":0.28499999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.6900000000000004,"5min":6.0170000000000003,"15min":6.0170000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.962,"5min":1.405,"15min":1.1259999999999999},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":10.225,"5min":10.648999999999999,"15min":10.648999999999999},"last":0.60499999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.3919999999999999,"15min":1.206},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.093,"5min":9.6780000000000008,"15min":9.6780000000000008},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.103,"5min":1.353,"15min":1.2450000000000001},"min":{"1min":0.41299999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.0089999999999999,"5min":10.244999999999999,"15min":10.244999999999999},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.399,"15min":1.292},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.8979999999999999,"5min":9.8019999999999996,"15min":9.8019999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.173,"15min":1.2789999999999999},"min":{"1min":0.46999999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0390000000000001,"5min":10.420999999999999,"15min":10.420999999999999},"last":2.2309999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1839999999999999,"15min":1.262},"min":{"1min":0.42699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.946,"5min":10.455,"15min":10.455},"last":2.2599999999999998},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.232,"15min":1.3420000000000001},"min":{"1min":0.54800000000000004,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.04,"5min":10.488,"15min":10.488},"last":2.2869999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.286,"15min":1.385},"min":{"1min":0.58099999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.6319999999999999,"5min":6.2869999999999999,"15min":6.2869999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.1890000000000001,"15min":1.3},"min":{"1min":0.57099999999999995,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.6000000000000001,"5min":5.1429999999999998,"15min":5.1429999999999998},"last":2.202}]},{"osd":8,"last update":"Mon Feb 12 16:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.5449999999999999,"15min":1.4770000000000001},"min":{"1min":0.50900000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":9.9049999999999994,"15min":9.9049999999999994},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.347,"5min":1.597,"15min":1.577},"min":{"1min":0.625,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.2410000000000001,"5min":10.313000000000001,"15min":10.313000000000001},"last":2.0790000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.456,"15min":1.5860000000000001},"min":{"1min":0.60299999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.8400000000000001,"5min":10.352,"15min":10.352},"last":2.0459999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.2030000000000001,"15min":1.419},"min":{"1min":0.56899999999999995,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.4259999999999999,"5min":4.7839999999999998,"15min":4.7839999999999998},"last":0.92700000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.5509999999999999,"15min":1.5529999999999999},"min":{"1min":0.68799999999999994,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7949999999999999,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.5489999999999999,"15min":1.6679999999999999},"min":{"1min":0.64200000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.798,"5min":10.387,"15min":10.387},"last":2.1099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1440000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738598,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1885700,"kb_used_data":1576188,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91859452,"statfs":{"total":95995035648,"available":94064078848,"internally_reserved":0,"allocated":1614016512,"data_stored":1613595829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":2.4980000000000002,"15min":1.361},"min":{"1min":0.28499999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.8369999999999997,"5min":114.01600000000001,"15min":114.01600000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.014,"5min":2.6389999999999998,"15min":1.429},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.254},"max":{"1min":3.3919999999999999,"5min":114.342,"15min":114.342},"last":2.6070000000000002}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:09:28 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.121,"15min":0.93700000000000006},"min":{"1min":0.26300000000000001,"5min":0.215,"15min":0.215},"max":{"1min":7.4740000000000002,"5min":7.4740000000000002,"15min":7.4740000000000002},"last":0.751},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.0289999999999999,"15min":0.95599999999999996},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.294,"5min":7.5140000000000002,"15min":7.5140000000000002},"last":2.7759999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.83099999999999996,"15min":1.032},"min":{"1min":0.312,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.9770000000000001,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.34899999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.1080000000000001,"5min":4.5860000000000003,"15min":4.5860000000000003},"last":2.6499999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.93300000000000005,"15min":1.034},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":4.6379999999999999,"5min":4.6379999999999999,"15min":4.6379999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.86699999999999999,"15min":1.0960000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.75,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":2.8090000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.91500000000000004,"15min":1.0840000000000001},"min":{"1min":0.36199999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.579,"5min":5.4699999999999998,"15min":5.4699999999999998},"last":2.8570000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.94899999999999995,"15min":1.1459999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.0619999999999998,"5min":5.5389999999999997,"15min":5.5389999999999997},"last":2.9359999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.0109999999999999,"15min":1.2410000000000001},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.966,"5min":5.5060000000000002,"15min":5.5060000000000002},"last":2.835},{"interface":"front","average":{"1min":1.373,"5min":1.103,"15min":1.298},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.1709999999999998,"5min":5.5720000000000001,"15min":5.5720000000000001},"last":2.9060000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:09:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.149,"15min":1.341},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":2.5129999999999999,"5min":5.6059999999999999,"15min":5.6059999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.246,"15min":1.401},"min":{"1min":0.44400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.8759999999999999,"5min":7.5599999999999996,"15min":7.5599999999999996},"last":2.8759999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:09:28 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2290000000000001,"15min":1.415},"min":{"1min":0.51200000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.794,"5min":5.7969999999999997,"15min":5.7969999999999997},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.224,"5min":1.1519999999999999,"15min":1.405},"min":{"1min":0.51700000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.8759999999999999,"5min":5.6890000000000001,"15min":5.6890000000000001},"last":2.9649999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:09:10 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.2569999999999999,"15min":1.464},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":4.8079999999999998,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":4.2030000000000003},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.288,"15min":1.367},"min":{"1min":0.44900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":3.1549999999999998,"5min":5.8010000000000002,"15min":5.8010000000000002},"last":2.996}]},{"osd":11,"last update":"Mon Feb 12 16:09:40 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.236,"15min":1.4450000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":3.0259999999999998,"5min":7.9660000000000002,"15min":7.9660000000000002},"last":3.0259999999999998},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.5069999999999999,"15min":1.4259999999999999},"min":{"1min":0.51500000000000001,"5min":0.48199999999999998,"15min":0.46700000000000003},"max":{"1min":2.992,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":2.6949999999999998}]}]},{"osd":2,"up_from":21,"seq":90194313429,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":314460,"kb_used_data":852,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93430692,"statfs":{"total":95995035648,"available":95673028608,"internally_reserved":0,"allocated":872448,"data_stored":439541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.181,"15min":0.93500000000000005},"min":{"1min":0.248,"5min":0.222,"15min":0.222},"max":{"1min":2.4449999999999998,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3320000000000001,"15min":1.0029999999999999},"min":{"1min":0.48699999999999999,"5min":0.216,"15min":0.216},"max":{"1min":6.3079999999999998,"5min":6.3079999999999998,"15min":6.3079999999999998},"last":0.72699999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.4530000000000001,"15min":1.085},"min":{"1min":0.26600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":9.1500000000000004,"5min":11.423,"15min":11.423},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.4319999999999999,"15min":1.1619999999999999},"min":{"1min":0.56000000000000005,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.4649999999999999,"5min":7.2290000000000001,"15min":7.2290000000000001},"last":1.177}]},{"osd":3,"last update":"Mon Feb 12 16:09:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.512,"15min":1.1719999999999999},"min":{"1min":0.36699999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.1970000000000001,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.1799999999999999,"15min":1.0449999999999999},"min":{"1min":0.34100000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.3700000000000001,"5min":9.4399999999999995,"15min":9.4399999999999995},"last":1.1259999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.3080000000000001,"15min":1.2},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.4179999999999999,"5min":9.657,"15min":9.657},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.1719999999999999,"15min":1.0269999999999999},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.5390000000000001,"5min":9.5530000000000008,"15min":9.5530000000000008},"last":0.89700000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":7.6550000000000002,"15min":3.3359999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.4159999999999999,"5min":616.34199999999998,"15min":616.34199999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.7010000000000001,"5min":7.9450000000000003,"15min":3.3940000000000001},"min":{"1min":0.80600000000000005,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":5.6390000000000002,"5min":615.18899999999996,"15min":615.18899999999996},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.1859999999999999,"15min":1.1439999999999999},"min":{"1min":0.41499999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.706,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.2689999999999999,"15min":1.266},"min":{"1min":0.50900000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.3650000000000002,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":7.9189999999999996,"15min":3.5489999999999999},"min":{"1min":0.47699999999999998,"5min":0.377,"15min":0.377},"max":{"1min":1.6140000000000001,"5min":611.79600000000005,"15min":611.79600000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.151,"5min":8.0579999999999998,"15min":3.544},"min":{"1min":0.54400000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.7470000000000001,"5min":611.83900000000006,"15min":611.83900000000006},"last":0.76600000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:09:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.4930000000000001,"15min":1.46},"min":{"1min":0.56200000000000006,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.6560000000000001,"5min":6.4960000000000004,"15min":6.4960000000000004},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.734,"15min":1.5569999999999999},"min":{"1min":0.58199999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.0489999999999999,"5min":8.5920000000000005,"15min":8.5920000000000005},"last":0.86899999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:09:21 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.3180000000000001,"15min":1.3819999999999999},"min":{"1min":0.498,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.929,"5min":9.6460000000000008,"15min":9.6460000000000008},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.343,"5min":1.504,"15min":1.4950000000000001},"min":{"1min":0.59799999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.7690000000000001,"5min":6.4699999999999998,"15min":6.4699999999999998},"last":0.92500000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4259999999999999,"15min":1.4530000000000001},"min":{"1min":0.65100000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.1539999999999999,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5289999999999999,"15min":1.5389999999999999},"min":{"1min":0.66900000000000004,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":5.5220000000000002,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":2.4700000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149900,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2365060,"kb_used_data":2063740,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91380092,"statfs":{"total":95995035648,"available":93573214208,"internally_reserved":0,"allocated":2113269760,"data_stored":2112849077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:58:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.72499999999999998,"15min":0.66900000000000004},"min":{"1min":0.63200000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.68100000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":2.0659999999999998,"15min":1.2549999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":8.3680000000000003,"5min":70.031999999999996,"15min":70.031999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.621,"5min":2.25,"15min":1.339},"min":{"1min":0.317,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.3399999999999999,"5min":69.944999999999993,"15min":69.944999999999993},"last":0.95299999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.8620000000000001,"15min":1.238},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9329999999999998,"5min":68.405000000000001,"15min":68.405000000000001},"last":0.442},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.694,"15min":1.2410000000000001},"min":{"1min":0.32800000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.5569999999999999,"5min":69.009,"15min":69.009},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.92500000000000004,"15min":0.96899999999999997},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.407,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.84799999999999998,"15min":1.018},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.92900000000000005,"15min":1.109},"min":{"1min":0.35799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5840000000000001,"5min":4.1840000000000002,"15min":4.1840000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.0389999999999999,"15min":1.0820000000000001},"min":{"1min":0.34699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.1179999999999999,"5min":5.4370000000000003,"15min":5.4370000000000003},"last":0.81200000000000006}]},{"osd":6,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.94699999999999995,"15min":1.119},"min":{"1min":0.52700000000000002,"5min":0.37,"15min":0.37},"max":{"1min":1.399,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.01,"15min":1.2070000000000001},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.4990000000000001,"5min":4.1429999999999998,"15min":4.1429999999999998},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.9359999999999999,"15min":1.528},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.9210000000000003,"5min":91.718999999999994,"15min":91.718999999999994},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.968,"15min":1.571},"min":{"1min":0.47799999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.46,"5min":90.367999999999995,"15min":90.367999999999995},"last":0.65200000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:09:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2849999999999999,"15min":1.3360000000000001},"min":{"1min":0.45200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":2.8919999999999999,"5min":6.5899999999999999,"15min":6.5899999999999999},"last":1.024},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1499999999999999,"15min":1.373},"min":{"1min":0.51100000000000001,"5min":0.442,"15min":0.442},"max":{"1min":2.911,"5min":8.516,"15min":8.516},"last":0.99299999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1240000000000001,"15min":1.4099999999999999},"min":{"1min":0.61099999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.667,"5min":4.9850000000000003,"15min":4.9850000000000003},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.274,"15min":1.4530000000000001},"min":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.5289999999999999,"5min":6.5350000000000001,"15min":6.5350000000000001},"last":0.745}]},{"osd":10,"last update":"Mon Feb 12 16:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.385,"15min":1.4430000000000001},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8740000000000001,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.304,"5min":1.3220000000000001,"15min":1.4910000000000001},"min":{"1min":0.52300000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":5.0019999999999998,"5min":5.3419999999999996,"15min":5.3419999999999996},"last":1.0569999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:08:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.8200000000000001,"15min":1.6040000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.468,"5min":49.167000000000002,"15min":49.167000000000002},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.228,"5min":1.75,"15min":1.619},"min":{"1min":0.55800000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":5.0289999999999999,"5min":48.548999999999999,"15min":48.548999999999999},"last":0.96099999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953669,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1842308,"kb_used_data":1532796,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91902844,"statfs":{"total":95995035648,"available":94108512256,"internally_reserved":0,"allocated":1569583104,"data_stored":1569162421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.77500000000000002,"15min":0.92800000000000005},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.125,"5min":4.4029999999999996,"15min":4.4029999999999996},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.98399999999999999,"15min":0.999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.4950000000000001,"5min":10.754,"15min":10.754},"last":3.621}]},{"osd":1,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.659,"15min":1.232},"min":{"1min":0.46899999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.1079999999999997,"5min":42.226999999999997,"15min":42.226999999999997},"last":3.5059999999999998},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.3180000000000001,"15min":1.135},"min":{"1min":0.51500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6950000000000001,"5min":43.689,"15min":43.689},"last":3.5459999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.45,"15min":1.147},"min":{"1min":0.435,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.1429999999999998,"5min":41.546999999999997,"15min":41.546999999999997},"last":2.7519999999999998},{"interface":"front","average":{"1min":1.234,"5min":1.599,"15min":1.1950000000000001},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":4.9340000000000002,"5min":41.57,"15min":41.57},"last":3.5750000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.80200000000000005,"15min":0.97799999999999998},"min":{"1min":0.48199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5980000000000001,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":2.7909999999999999},{"interface":"front","average":{"1min":1.167,"5min":0.88800000000000001,"15min":0.999},"min":{"1min":0.55100000000000005,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.5669999999999999,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":3.5979999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.79800000000000004,"15min":0.94699999999999995},"min":{"1min":0.318,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.25,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90300000000000002,"15min":1.026},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3759999999999999,"5min":4.8109999999999999,"15min":4.8109999999999999},"last":2.9689999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:09:25 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.79700000000000004,"15min":1.004},"min":{"1min":0.39300000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.641,"5min":3.004,"15min":3.004},"last":3.7229999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.72099999999999997,"15min":1.0209999999999999},"min":{"1min":0.37,"5min":0.307,"15min":0.307},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.9330000000000001},"last":2.6930000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.1789999999999998,"15min":1.496},"min":{"1min":0.45400000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.669,"5min":116.273,"15min":116.273},"last":3.4049999999999998},{"interface":"front","average":{"1min":1.383,"5min":2.3119999999999998,"15min":1.6830000000000001},"min":{"1min":0.69499999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.9370000000000001,"5min":116.24299999999999,"15min":116.24299999999999},"last":3.742}]},{"osd":8,"last update":"Mon Feb 12 16:08:46 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.1599999999999999,"15min":1.399},"min":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.9180000000000001,"5min":8.4190000000000005,"15min":8.4190000000000005},"last":3.7989999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.0760000000000001,"15min":1.3140000000000001},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.742,"5min":4.8609999999999998,"15min":4.8609999999999998},"last":0.42999999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":4.2199999999999998,"15min":2.464},"min":{"1min":0.745,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.881,"5min":297.55900000000003,"15min":297.55900000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.4730000000000001,"5min":4.2480000000000002,"15min":2.4199999999999999},"min":{"1min":0.52900000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.966,"5min":297.61900000000003,"15min":297.61900000000003},"last":3.8740000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.0349999999999999,"15min":1.363},"min":{"1min":0.63900000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.7490000000000001,"5min":4.944,"15min":4.944},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.0649999999999999,"15min":1.385},"min":{"1min":0.55600000000000005,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":5.2469999999999999,"5min":5.2469999999999999,"15min":5.2469999999999999},"last":2.8290000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724733,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1533468,"kb_used_data":1223956,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92211684,"statfs":{"total":95995035648,"available":94424764416,"internally_reserved":0,"allocated":1253330944,"data_stored":1252898037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94599999999999995,"15min":0.97399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":3.4950000000000001,"5min":13.031000000000001,"15min":13.031000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.0129999999999999,"15min":0.98799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5140000000000002,"5min":14.018000000000001,"15min":14.018000000000001},"last":0.65400000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.627,"15min":1.224},"min":{"1min":0.29399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":7.7610000000000001,"5min":33.084000000000003,"15min":33.084000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.708,"5min":1.329,"15min":1.095},"min":{"1min":0.34499999999999997,"5min":0.253,"15min":0.253},"max":{"1min":7.8140000000000001,"5min":33.052999999999997,"15min":33.052999999999997},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5150000000000001,"5min":1.631,"15min":1.2450000000000001},"min":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":15.673,"5min":26.797000000000001,"15min":26.797000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.319,"5min":1.3700000000000001,"15min":1.1679999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":8.298,"5min":26.905999999999999,"15min":26.905999999999999},"last":0.434}]},{"osd":3,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.4079999999999999,"15min":1.177},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.032,"5min":12.991,"15min":12.991},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.429,"5min":1.238,"15min":1.171},"min":{"1min":0.42899999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":4.6749999999999998,"5min":13.935,"15min":13.935},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.96999999999999997,"15min":0.95699999999999996},"min":{"1min":0.33800000000000002,"5min":0.308,"15min":0.308},"max":{"1min":3.5609999999999999,"5min":14.259,"15min":14.259},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.2270000000000001,"5min":0.97599999999999998,"15min":1.0509999999999999},"min":{"1min":0.41499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.75,"5min":13.268000000000001,"15min":13.268000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:09:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.107,"15min":1.143},"min":{"1min":0.39700000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.6140000000000001,"5min":13.148,"15min":13.148},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.329,"5min":1.127,"15min":1.194},"min":{"1min":0.54100000000000004,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.4630000000000001,"5min":13.214,"15min":13.214},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:09:00 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":2.282,"15min":1.5760000000000001},"min":{"1min":0.57899999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":5.6669999999999998,"5min":98.971000000000004,"15min":98.971000000000004},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.627,"5min":2.306,"15min":1.665},"min":{"1min":0.56699999999999995,"5min":0.45200000000000001,"15min":0.435},"max":{"1min":4.6020000000000003,"5min":99.218000000000004,"15min":99.218000000000004},"last":0.58599999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.238,"15min":1.389},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.7839999999999998,"5min":14.372999999999999,"15min":14.372999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.619,"5min":1.264,"15min":1.3580000000000001},"min":{"1min":0.52100000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.7249999999999996,"5min":14.074999999999999,"15min":14.074999999999999},"last":0.74299999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:09:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.22,"15min":1.421},"min":{"1min":0.59299999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.8240000000000001,"5min":13.375999999999999,"15min":13.375999999999999},"last":1},{"interface":"front","average":{"1min":1.413,"5min":1.1950000000000001,"15min":1.4830000000000001},"min":{"1min":0.64000000000000001,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0859999999999999,"5min":14.132,"15min":14.132},"last":0.77300000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:09:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.27,"15min":1.4219999999999999},"min":{"1min":0.495,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.712,"5min":14.214,"15min":14.214},"last":0.503},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.2769999999999999,"15min":1.5109999999999999},"min":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.8849999999999998,"5min":13.464,"15min":13.464},"last":1.0349999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1630732288,"data_stored":1630732288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1241710592,"data_stored":1241710592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2001076224,"data_stored":2001076224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1587544064,"data_stored":1587544064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1255604224,"data_stored":1255604224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":394788864,"data_stored":394788864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":786432000,"data_stored":786432000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401014784,"data_stored":401014784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":832831488,"data_stored":832831488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2824732672,"data_stored":2824732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:09:49.560 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T16:09:49.560 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:09:49.560 INFO:tasks.thrashosds.thrasher:Removing osd 8, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:09:49.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd out 8 2024-02-12T16:09:49.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:49.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:49.708 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:49.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:49.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 16 106906 106890 162.921 102.438 0.00495644 0.00613622 2024-02-12T16:09:49.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:49.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:49 smithi169 bash[19604]: cluster 2024-02-12T16:09:48.698567+0000 mon.a (mon.0) 1160 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:09:49.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:49 smithi169 bash[19604]: audit 2024-02-12T16:09:48.706699+0000 mon.a (mon.0) 1161 : audit [INF] from='client.25061 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:09:49.971 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:49 smithi169 bash[19604]: cluster 2024-02-12T16:09:48.706754+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e274: 12 total, 12 up, 12 in 2024-02-12T16:09:49.992 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:49.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:50.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:50.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:49 smithi121 bash[18489]: cluster 2024-02-12T16:09:48.698567+0000 mon.a (mon.0) 1160 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:09:50.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:49 smithi121 bash[18489]: audit 2024-02-12T16:09:48.706699+0000 mon.a (mon.0) 1161 : audit [INF] from='client.25061 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:09:50.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:49 smithi121 bash[18489]: cluster 2024-02-12T16:09:48.706754+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e274: 12 total, 12 up, 12 in 2024-02-12T16:09:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:50.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:50.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:49 smithi123 bash[19119]: cluster 2024-02-12T16:09:48.698567+0000 mon.a (mon.0) 1160 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:09:50.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:49 smithi123 bash[19119]: audit 2024-02-12T16:09:48.706699+0000 mon.a (mon.0) 1161 : audit [INF] from='client.25061 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:09:50.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:49 smithi123 bash[19119]: cluster 2024-02-12T16:09:48.706754+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e274: 12 total, 12 up, 12 in 2024-02-12T16:09:50.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:50.258 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:50.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:50.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:50.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:50.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:50.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:50.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:50.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:50.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:50.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:50.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:50.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:50.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:50.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:50.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:50.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 108666 108650 161.661 110 0.0125921 0.00618451 2024-02-12T16:09:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:50 smithi169 bash[19604]: cluster 2024-02-12T16:09:49.635073+0000 mgr.y (mgr.14146) 733 : cluster [DBG] pgmap v910: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-12T16:09:51.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:51.071 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:51.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:51.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:50 smithi123 bash[19119]: cluster 2024-02-12T16:09:49.635073+0000 mgr.y (mgr.14146) 733 : cluster [DBG] pgmap v910: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-12T16:09:51.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:50 smithi121 bash[18489]: cluster 2024-02-12T16:09:49.635073+0000 mgr.y (mgr.14146) 733 : cluster [DBG] pgmap v910: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.78k op/s 2024-02-12T16:09:51.176 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:51.207 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:51.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:51.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:51.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:51.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:51.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:51.479 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:51.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:51.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:51.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:51.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:51.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:51.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:51.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:09:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:51.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:51.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:09:51.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:51.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:51.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:51.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 16 110359 110343 160.362 105.812 0.00841719 0.00623439 2024-02-12T16:09:51.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:52.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:52.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:52.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:52.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:52.271 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:52.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:52.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:52.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:52.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:52.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:52.676 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:52.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:52.803 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:52 smithi123 bash[19119]: cluster 2024-02-12T16:09:51.635487+0000 mgr.y (mgr.14146) 734 : cluster [DBG] pgmap v911: 17 pgs: 17 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-12T16:09:52.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:52.942 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:52.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:52.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 112035 112019 159.098 104.75 0.0088627 0.00628378 2024-02-12T16:09:53.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:52 smithi169 bash[19604]: cluster 2024-02-12T16:09:51.635487+0000 mgr.y (mgr.14146) 734 : cluster [DBG] pgmap v911: 17 pgs: 17 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-12T16:09:53.068 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:52 smithi121 bash[18489]: cluster 2024-02-12T16:09:51.635487+0000 mgr.y (mgr.14146) 734 : cluster [DBG] pgmap v911: 17 pgs: 17 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-02-12T16:09:53.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:53.197 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:53.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:53.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:53.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:53.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:53.530 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:09:53.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:53.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:53.713 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:53.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:53.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:53.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:53.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:53.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 113705 113689 157.881 104.375 0.00479258 0.00633189 2024-02-12T16:09:53.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:53.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:54.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:54.100 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:54.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:54.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:54.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:54.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:54.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:54.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:54.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:54.405 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:54.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:54.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:54.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:54.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:54.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:54.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:54.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:54.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:54 smithi123 bash[19119]: cluster 2024-02-12T16:09:53.635916+0000 mgr.y (mgr.14146) 735 : cluster [DBG] pgmap v912: 17 pgs: 17 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:09:54.769 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:54.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:54.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:54.899 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:54 smithi121 bash[18489]: cluster 2024-02-12T16:09:53.635916+0000 mgr.y (mgr.14146) 735 : cluster [DBG] pgmap v912: 17 pgs: 17 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:09:54.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:54.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:54.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 16 115253 115237 156.552 96.75 0.00404596 0.00638625 2024-02-12T16:09:55.018 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:54 smithi169 bash[19604]: cluster 2024-02-12T16:09:53.635916+0000 mgr.y (mgr.14146) 735 : cluster [DBG] pgmap v912: 17 pgs: 17 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:09:55.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:55.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:55.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:55.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:55.304 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:55.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:55.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:55.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:55.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:55.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:55.701 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:55.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:55.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:09:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:09:55.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:55.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 16 116554 116538 154.951 81.3125 0.00444154 0.00645198 2024-02-12T16:09:55.958 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:55.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:56.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:56.096 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:56.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:56.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:56.226 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:56.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:56.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:56.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:56.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:56.432 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:56.435 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:09:56.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:56.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:56.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:56.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:56.605 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:56 smithi123 bash[19119]: cluster 2024-02-12T16:09:55.636334+0000 mgr.y (mgr.14146) 736 : cluster [DBG] pgmap v913: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:09:56.626 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:09:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:56.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:09:56.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:56 smithi121 bash[18489]: cluster 2024-02-12T16:09:55.636334+0000 mgr.y (mgr.14146) 736 : cluster [DBG] pgmap v913: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:09:56.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:56.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:56.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:56.805 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:56 smithi169 bash[19604]: cluster 2024-02-12T16:09:55.636334+0000 mgr.y (mgr.14146) 736 : cluster [DBG] pgmap v913: 17 pgs: 17 active+clean; 6.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:09:56.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:56.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:56.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:56.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 118170 118154 153.827 101 0.00413816 0.0064991 2024-02-12T16:09:56.996 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:57.026 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:09:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:57.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:09:57.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:57.162 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:57.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:57.265 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:09:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:57.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:09:57.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:57.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:57.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:57.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:57.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:57.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:57 smithi123 bash[19119]: audit 2024-02-12T16:09:57.395805+0000 mon.a (mon.0) 1163 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:09:57.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:57.668 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:57 smithi169 bash[19604]: audit 2024-02-12T16:09:57.395805+0000 mon.a (mon.0) 1163 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:09:57.694 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:57.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:57.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:57.820 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:09:57.820 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:09:57.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:09:57.820 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:09:57.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.820 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:61398 2.0 e158/147 2:0eb47553:::benchmark_data_smithi169_42458_object61397:head v 158'3893, mlcod=158'3891)", 2024-02-12T16:09:57.820 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:35.244572+0000", 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 322.57105457400002, 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025654824999999999, 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244572+0000", 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244572+0000", 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244570+0000", 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2656000000000005e-05 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244632+0000", 2024-02-12T16:09:57.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4761e-05 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244647+0000", 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.661e-06 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244655+0000", 2024-02-12T16:09:57.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6017000000000003e-05 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244711+0000", 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4700000000000001e-05 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.244735+0000", 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025455586999999998 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.270191+0000", 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5630999999999998e-05 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.270227+0000", 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025654824999999999 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:61401 2.0 e158/147 2:04cc44cf:::benchmark_data_smithi169_42458_object61400:head v 158'3894, mlcod=158'3891)", 2024-02-12T16:09:57.825 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:35.246168+0000", 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 322.56945795899998, 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029808085000000002, 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246168+0000", 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246168+0000", 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:57.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246167+0000", 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8685999999999999e-05 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246236+0000", 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0230000000000001e-05 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246256+0000", 2024-02-12T16:09:57.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.827e-06 2024-02-12T16:09:57.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246263+0000", 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00028676100000000001 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246549+0000", 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4459000000000001e-05 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.246574+0000", 2024-02-12T16:09:57.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029363798999999999 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.275938+0000", 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8849e-05 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:35.275977+0000", 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029808085000000002 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:236357 2.0 e198/171 2:079c0eab:::benchmark_data_smithi169_42458_object63323:head v 198'12171, mlcod=198'12169)", 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:06:40.275547+0000", 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 197.54007987200001, 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025213045, 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.275547+0000", 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.275547+0000", 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.275544+0000", 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2001000000000001e-05 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.275556+0000", 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9890000000000001e-05 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.275576+0000", 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4089999999999997e-06 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.275584+0000", 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079017300000000003 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276374+0000", 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0035e-05 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276394+0000", 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024322231 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.300716+0000", 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3616999999999999e-05 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.300760+0000", 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025213045 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:236360 2.7 e198/171 2:e74921d1:::benchmark_data_smithi169_42458_object63326:head v 198'11932, mlcod=198'11930)", 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:06:40.276108+0000", 2024-02-12T16:09:57.835 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 197.53951839300001, 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024473063999999999, 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276108+0000", 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276108+0000", 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:57.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276107+0000", 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276116+0000", 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3122000000000001e-05 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276129+0000", 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8640000000000004e-06 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276134+0000", 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9833000000000003e-05 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276174+0000", 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9465000000000001e-05 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.276193+0000", 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024338583 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.300532+0000", 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9131000000000001e-05 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:06:40.300581+0000", 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024473063999999999 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:301011 2.12 e220/218 2:48a4effa:::benchmark_data_smithi169_42458_object127977:head v 220'14103, mlcod=168'7991)", 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:36.344795+0000", 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 141.47083149700001, 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025659056999999999, 2024-02-12T16:09:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344795+0000", 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344795+0000", 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344793+0000", 2024-02-12T16:09:57.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1710000000000002e-06 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344802+0000", 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0682999999999999e-05 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344812+0000", 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1109999999999996e-06 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344818+0000", 2024-02-12T16:09:57.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0152000000000003e-05 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344878+0000", 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8787999999999999e-05 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.344907+0000", 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025508459000000001 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.370416+0000", 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8247000000000002e-05 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:36.370454+0000", 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025659056999999999 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:302389 2.12 e220/218 2:48c92ffb:::benchmark_data_smithi169_42458_object129355:head v 220'14132, mlcod=169'8068)", 2024-02-12T16:09:57.844 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:37.606471+0000", 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.209155213, 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027753578000000001, 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.606471+0000", 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.606471+0000", 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:09:57.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.606471+0000", 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9009999999999999e-06 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.606476+0000", 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3088000000000001e-05 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.606489+0000", 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7350000000000004e-06 2024-02-12T16:09:57.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.606494+0000", 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075204340000000003 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.614015+0000", 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1118999999999998e-05 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.614046+0000", 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020142856000000001 2024-02-12T16:09:57.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.634189+0000", 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6149000000000002e-05 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.634225+0000", 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027753578000000001 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.848 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:302396 2.12 e220/218 2:4be77dd9:::benchmark_data_smithi169_42458_object129362:head v 220'14133, mlcod=169'8068)", 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:37.610450+0000", 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.20517617100001, 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023791855000000001, 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.610450+0000", 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.610450+0000", 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.610448+0000", 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4123e-05 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.610462+0000", 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5157000000000001e-05 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.610477+0000", 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1215e-05 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.610489+0000", 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010190075999999999 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.851 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.620679+0000", 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2257999999999999e-05 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.620701+0000", 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013527323000000001 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.634228+0000", 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3913e-05 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.634242+0000", 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023791855000000001 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:302597 2.12 e220/218 2:4f45e974:::benchmark_data_smithi169_42458_object129563:head v 220'14143, mlcod=169'8080)", 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:37.876680+0000", 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 139.938946536, 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031987163999999998, 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876680+0000", 2024-02-12T16:09:57.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876680+0000", 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876678+0000", 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6469999999999993e-06 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876688+0000", 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2145e-05 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876700+0000", 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7230000000000001e-06 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876705+0000", 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1357000000000003e-05 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876757+0000", 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0149e-05 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.876777+0000", 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031844033000000001 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.908621+0000", 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6047999999999997e-05 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.908667+0000", 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031987163999999998 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.856 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:302624 2.12 e220/218 2:4f208e59:::benchmark_data_smithi169_42458_object129590:head v 220'14144, mlcod=169'8080)", 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:37.884717+0000", 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 139.93090911100001, 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.026542403999999999, 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.884717+0000", 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.884717+0000", 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.884715+0000", 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1511e-05 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.884727+0000", 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5605999999999999e-05 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.884743+0000", 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1790000000000002e-06 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.884750+0000", 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.009594455 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.894344+0000", 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9271000000000001e-05 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.894373+0000", 2024-02-12T16:09:57.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016865630999999999 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.911239+0000", 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.065e-05 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:37.911260+0000", 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.026542403999999999 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:305759 2.12 e221/218 2:49dac128:::benchmark_data_smithi169_42458_object132725:head v 221'14223, mlcod=169'8276)", 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:40.869011+0000", 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 136.94661589399999, 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024141874000000001, 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869011+0000", 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869011+0000", 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869009+0000", 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6920000000000008e-06 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869018+0000", 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1625e-05 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869030+0000", 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7509999999999997e-06 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869035+0000", 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2021999999999998e-05 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869087+0000", 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7395000000000001e-05 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.869115+0000", 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023993311999999999 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.893108+0000", 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4419000000000003e-05 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:40.893152+0000", 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024141874000000001 2024-02-12T16:09:57.864 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:312359 2.12 e222/218 2:4d750fce:::benchmark_data_smithi169_42458_object139325:head v 222'14408, mlcod=169'8679)", 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:47.208995+0000", 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 130.60663096900001, 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025794655999999999, 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.865 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.208995+0000", 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.208995+0000", 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:09:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.208992+0000", 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.05e-05 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.209013+0000", 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7419999999999999e-05 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.209030+0000", 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4070000000000004e-06 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.209038+0000", 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045601499999999998 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.213598+0000", 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5874e-05 2024-02-12T16:09:57.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.213624+0000", 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021118938 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.234743+0000", 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7505e-05 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:47.234790+0000", 2024-02-12T16:09:57.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025794655999999999 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:316242 2.12 e224/218 2:4b815b00:::benchmark_data_smithi169_42458_object143208:head v 224'14542, mlcod=169'8935)", 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:51.357799+0000", 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 126.457827671, 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024538693, 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.870 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357799+0000", 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357799+0000", 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357797+0000", 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0750000000000004e-06 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357806+0000", 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7439999999999999e-05 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357824+0000", 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3720000000000004e-06 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357829+0000", 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1823000000000002e-05 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357881+0000", 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0639000000000003e-05 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.357912+0000", 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024378499000000001 2024-02-12T16:09:57.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.382290+0000", 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7212000000000001e-05 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:51.382337+0000", 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024538693 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:334538 2.12 e230/218 2:48ad901a:::benchmark_data_smithi169_42458_object161504:head v 230'15133, mlcod=179'9755)", 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:11.437290+0000", 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 106.378336922, 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024605267, 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.437290+0000", 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.437290+0000", 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.437288+0000", 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0750000000000004e-06 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.437297+0000", 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1674e-05 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.437309+0000", 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7570000000000001e-06 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.437315+0000", 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028859369999999999 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.440201+0000", 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4828999999999999e-05 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.440226+0000", 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021617351999999999 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.461843+0000", 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1814000000000002e-05 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:11.461895+0000", 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024605267 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:336411 2.12 e231/218 2:488f7028:::benchmark_data_smithi169_42458_object163377:head v 231'15198, mlcod=179'9802)", 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:13.190359+0000", 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 104.62526700700001, 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027739764, 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.190359+0000", 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.190359+0000", 2024-02-12T16:09:57.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.190358+0000", 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.747e-06 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.190368+0000", 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1606e-05 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.190379+0000", 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8250000000000001e-06 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.190385+0000", 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048033060000000002 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.195189+0000", 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5837999999999999e-05 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.195214+0000", 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022800028 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.218014+0000", 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4724999999999999e-05 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:13.218099+0000", 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027739764 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:337988 2.12 e231/218 2:48d579dc:::benchmark_data_smithi169_42458_object164954:head v 231'15251, mlcod=179'9849)", 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:14.726474+0000", 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 103.089152872, 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025923728, 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726474+0000", 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726474+0000", 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726471+0000", 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5960999999999998e-05 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726487+0000", 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3363999999999999e-05 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726501+0000", 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7580000000000003e-06 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726508+0000", 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029872900000000002 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726806+0000", 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5230000000000001e-05 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.726832+0000", 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025517186000000001 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.752349+0000", 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8559e-05 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:14.752397+0000", 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025923728 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:338602 2.12 e232/218 2:4c621709:::benchmark_data_smithi169_42458_object165568:head v 232'15278, mlcod=179'9877)", 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:15.610872+0000", 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 102.204754299, 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029880439000000002, 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.610872+0000", 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.610872+0000", 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.610869+0000", 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8026999999999999e-05 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.610887+0000", 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5619999999999999e-05 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.610913+0000", 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1871999999999999e-05 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.610925+0000", 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062621550000000002 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.617187+0000", 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4204000000000001e-05 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.617211+0000", 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023497726 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.640709+0000", 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3723000000000001e-05 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:15.640753+0000", 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029880439000000002 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.890 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:340573 2.12 e233/218 2:4991e067:::benchmark_data_smithi169_42458_object167539:head v 233'15336, mlcod=180'9935)", 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:17.549455+0000", 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 100.266171106, 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028857679000000001, 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549455+0000", 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549455+0000", 2024-02-12T16:09:57.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549453+0000", 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0821e-05 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549464+0000", 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6492000000000001e-05 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549481+0000", 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0319999999999999e-05 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549491+0000", 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3542999999999999e-05 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549545+0000", 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3448999999999998e-05 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.549578+0000", 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028703515999999998 2024-02-12T16:09:57.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.578281+0000", 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1529999999999998e-05 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:17.578313+0000", 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028857679000000001 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:344690 2.12 e234/218 2:4f27530c:::benchmark_data_smithi169_42458_object171656:head v 234'15460, mlcod=180'10059)", 2024-02-12T16:09:57.894 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:21.556712+0000", 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 96.258914419999996, 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024212498999999998, 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556712+0000", 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556712+0000", 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556710+0000", 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0672999999999999e-05 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556721+0000", 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0871e-05 2024-02-12T16:09:57.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556732+0000", 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-06 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556738+0000", 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3391000000000003e-05 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556802+0000", 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6627000000000001e-05 2024-02-12T16:09:57.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.556828+0000", 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024030411000000002 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.580859+0000", 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5631999999999998e-05 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:21.580925+0000", 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024212498999999998 2024-02-12T16:09:57.898 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:345640 2.12 e236/218 2:4b9c230e:::benchmark_data_smithi169_42458_object172606:head v 236'15489, mlcod=180'10095)", 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:23.138484+0000", 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 94.677142095999997, 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032013906000000002, 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.138484+0000", 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.138484+0000", 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.138483+0000", 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8209999999999997e-06 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.138493+0000", 2024-02-12T16:09:57.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3674e-05 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.138506+0000", 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5810000000000002e-06 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.138513+0000", 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022470229999999999 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.140760+0000", 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0204000000000001e-05 2024-02-12T16:09:57.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.140790+0000", 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029668160999999998 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.170458+0000", 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9894000000000001e-05 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.170498+0000", 2024-02-12T16:09:57.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032013906000000002 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.14811.0:345648 2.12 e236/218 2:49548c22:::benchmark_data_smithi169_42458_object172614:head v 236'15490, mlcod=180'10095)", 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:08:23.142971+0000", 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 94.672655818999999, 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027552851, 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:09:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.142971+0000", 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.142971+0000", 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.142969+0000", 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2913e-05 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:09:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.142982+0000", 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8301e-05 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.143000+0000", 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3379999999999999e-06 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.143007+0000", 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096170049999999997 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.152624+0000", 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3315999999999997e-05 2024-02-12T16:09:57.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.152657+0000", 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.017848315 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.170506+0000", 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7657999999999999e-05 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:08:23.170523+0000", 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027552851 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.906 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.907 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:09:57.907 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:09:57.907 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:09:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:57.908 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:57.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:57.917 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:57 smithi121 bash[18489]: audit 2024-02-12T16:09:57.395805+0000 mon.a (mon.0) 1163 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:09:57.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 119630 119614 152.55 91.25 0.0119725 0.00655343 2024-02-12T16:09:58.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:58.034 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:58.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:58.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:58.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:09:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:58.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:09:58.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:58.314 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:09:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:58.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:09:58.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:58.450 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:09:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:58.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:09:58.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:58.601 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:58.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:58.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-12T16:09:58.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:58.737 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:58.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:58.844 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:58.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:58.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:58 smithi121 bash[18489]: cluster 2024-02-12T16:09:57.636742+0000 mgr.y (mgr.14146) 737 : cluster [DBG] pgmap v914: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:09:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:58 smithi121 bash[18489]: audit 2024-02-12T16:09:57.698106+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:09:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:58 smithi121 bash[18489]: audit 2024-02-12T16:09:58.025925+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:09:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:58 smithi121 bash[18489]: audit 2024-02-12T16:09:58.028062+0000 mon.a (mon.0) 1166 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:09:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:09:58 smithi121 bash[18489]: audit 2024-02-12T16:09:58.033810+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:09:58.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 121245 121229 151.518 100.938 0.0163477 0.00659836 2024-02-12T16:09:58.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:58.981 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:58 smithi123 bash[19119]: cluster 2024-02-12T16:09:57.636742+0000 mgr.y (mgr.14146) 737 : cluster [DBG] pgmap v914: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:09:58.981 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:58 smithi123 bash[19119]: audit 2024-02-12T16:09:57.698106+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:09:58.981 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:58 smithi123 bash[19119]: audit 2024-02-12T16:09:58.025925+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:09:58.981 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:58 smithi123 bash[19119]: audit 2024-02-12T16:09:58.028062+0000 mon.a (mon.0) 1166 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:09:58.981 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:09:58 smithi123 bash[19119]: audit 2024-02-12T16:09:58.033810+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:09:58.999 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:58 smithi169 bash[19604]: cluster 2024-02-12T16:09:57.636742+0000 mgr.y (mgr.14146) 737 : cluster [DBG] pgmap v914: 17 pgs: 17 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:09:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:58 smithi169 bash[19604]: audit 2024-02-12T16:09:57.698106+0000 mon.a (mon.0) 1164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:09:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:58 smithi169 bash[19604]: audit 2024-02-12T16:09:58.025925+0000 mon.a (mon.0) 1165 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:09:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:58 smithi169 bash[19604]: audit 2024-02-12T16:09:58.028062+0000 mon.a (mon.0) 1166 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:09:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:09:58 smithi169 bash[19604]: audit 2024-02-12T16:09:58.033810+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:09:59.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:59.136 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:09:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:09:59.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:59.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:09:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:09:59.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:59.397 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:59.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:59.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:09:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:09:59.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:59.659 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:09:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:09:59.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:59.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:59.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:59.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:09:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:09:59.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:09:59.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 122870 122854 150.538 101.562 0.0117326 0.00664139 2024-02-12T16:10:00.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:00.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:00.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:00.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:00.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:00.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:00.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:00.460 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:00.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:00.592 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:00.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:00.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:00.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:00.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:00 smithi123 bash[19119]: cluster 2024-02-12T16:09:59.637184+0000 mgr.y (mgr.14146) 738 : cluster [DBG] pgmap v915: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-02-12T16:10:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:00 smithi123 bash[19119]: cluster 2024-02-12T16:10:00.000101+0000 mon.a (mon.0) 1168 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:10:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:00 smithi123 bash[19119]: cluster 2024-02-12T16:10:00.000133+0000 mon.a (mon.0) 1169 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-12T16:10:00.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:00.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 16 124183 124167 149.221 82.0625 0.00447768 0.00669947 2024-02-12T16:10:00.963 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:00 smithi169 bash[19604]: cluster 2024-02-12T16:09:59.637184+0000 mgr.y (mgr.14146) 738 : cluster [DBG] pgmap v915: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-02-12T16:10:00.964 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:00 smithi169 bash[19604]: cluster 2024-02-12T16:10:00.000101+0000 mon.a (mon.0) 1168 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:10:00.964 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:00 smithi169 bash[19604]: cluster 2024-02-12T16:10:00.000133+0000 mon.a (mon.0) 1169 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-12T16:10:00.981 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:00.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:01.047 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:00 smithi121 bash[18489]: cluster 2024-02-12T16:09:59.637184+0000 mgr.y (mgr.14146) 738 : cluster [DBG] pgmap v915: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-02-12T16:10:01.047 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:00 smithi121 bash[18489]: cluster 2024-02-12T16:10:00.000101+0000 mon.a (mon.0) 1168 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:10:01.047 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:00 smithi121 bash[18489]: cluster 2024-02-12T16:10:00.000133+0000 mon.a (mon.0) 1169 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-12T16:10:01.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:01.111 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:01.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:01.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:01.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:01.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:01.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:01.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:01.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:01.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:01.502 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:01.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:01.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:01.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:01.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:01.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:01.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:01.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:01.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:01.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:01.956 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 125426 125410 147.871 77.6875 0.00595498 0.00676088 2024-02-12T16:10:02.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:02.060 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:02.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:02.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:02.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:02.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:02.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:02.458 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:02.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:02.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:02 smithi121 bash[18489]: debug 2024-02-12T16:10:02.234+0000 7fbba85de700 -1 mon.a@0(leader).osd e274 definitely_dead 0 2024-02-12T16:10:02.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:02.715 INFO:teuthology.orchestra.run.smithi121.stderr:marked out osd.8. 2024-02-12T16:10:02.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:02.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:02.845 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:02.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:02.956 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:02 smithi169 bash[19604]: cluster 2024-02-12T16:10:01.637612+0000 mgr.y (mgr.14146) 739 : cluster [DBG] pgmap v916: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:10:02.956 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:02 smithi169 bash[19604]: audit 2024-02-12T16:10:02.234952+0000 mon.a (mon.0) 1170 : audit [INF] from='client.25030 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:10:02.956 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:02 smithi169 bash[19604]: cluster 2024-02-12T16:10:02.235093+0000 mon.a (mon.0) 1171 : cluster [INF] Client client.admin marked osd.8 out, while it was still marked up 2024-02-12T16:10:02.956 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:02 smithi169 bash[19604]: audit 2024-02-12T16:10:02.236997+0000 mon.c (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.121:0/2841024463' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:10:02.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 126773 126757 146.692 84.1875 0.00522713 0.00679464 2024-02-12T16:10:02.976 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:02.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:03.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:03.092 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:02 smithi123 bash[19119]: cluster 2024-02-12T16:10:01.637612+0000 mgr.y (mgr.14146) 739 : cluster [DBG] pgmap v916: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:10:03.092 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:02 smithi123 bash[19119]: audit 2024-02-12T16:10:02.234952+0000 mon.a (mon.0) 1170 : audit [INF] from='client.25030 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:10:03.092 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:02 smithi123 bash[19119]: cluster 2024-02-12T16:10:02.235093+0000 mon.a (mon.0) 1171 : cluster [INF] Client client.admin marked osd.8 out, while it was still marked up 2024-02-12T16:10:03.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:02 smithi123 bash[19119]: audit 2024-02-12T16:10:02.236997+0000 mon.c (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.121:0/2841024463' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:10:03.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:03.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:02 smithi121 bash[18489]: cluster 2024-02-12T16:10:01.637612+0000 mgr.y (mgr.14146) 739 : cluster [DBG] pgmap v916: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:10:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:02 smithi121 bash[18489]: audit 2024-02-12T16:10:02.234952+0000 mon.a (mon.0) 1170 : audit [INF] from='client.25030 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:10:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:02 smithi121 bash[18489]: cluster 2024-02-12T16:10:02.235093+0000 mon.a (mon.0) 1171 : cluster [INF] Client client.admin marked osd.8 out, while it was still marked up 2024-02-12T16:10:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:02 smithi121 bash[18489]: audit 2024-02-12T16:10:02.236997+0000 mon.c (mon.1) 30 : audit [INF] from='client.? v1:172.21.15.121:0/2841024463' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:10:03.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:03.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:03.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:03.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:03.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:03.443 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:10:03.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:03.535 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:03.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:03.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:03.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:03.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:03.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:03.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:03.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:03.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:03.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:03.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:03.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 126773 126757 144.025 0 - 0.00679464 2024-02-12T16:10:03.968 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: audit 2024-02-12T16:10:02.706242+0000 mon.a (mon.0) 1172 : audit [INF] from='client.25030 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:02.706316+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e275: 12 total, 12 up, 11 in 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.524611+0000 mon.a (mon.0) 1174 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=1995728569,config=2903391811,health=438477494,kv=2355815837}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.526392+0000 mon.a (mon.0) 1175 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2558004167,logm=255244303}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.527863+0000 mon.a (mon.0) 1176 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=654397940}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.529253+0000 mon.a (mon.0) 1177 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4273462403}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.530802+0000 mon.a (mon.0) 1178 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2368527503}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.532393+0000 mon.a (mon.0) 1179 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3070668759}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.539219+0000 mon.a (mon.0) 1180 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1959311415}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.545883+0000 mon.a (mon.0) 1181 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2722760719}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.551937+0000 mon.a (mon.0) 1182 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592798890}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.557155+0000 mon.a (mon.0) 1183 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3539499822}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.561921+0000 mon.a (mon.0) 1184 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996003035}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.563880+0000 mon.a (mon.0) 1185 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=43} crc {logm=1761306972,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=479563086,mon_config_key=1175164893}) 2024-02-12T16:10:03.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.564723+0000 mon.a (mon.0) 1186 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2534505019}) 2024-02-12T16:10:03.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.565523+0000 mon.a (mon.0) 1187 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=690416649}) 2024-02-12T16:10:03.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.566399+0000 mon.a (mon.0) 1188 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=3709686934,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=2619014876,osdmap=270461205}) 2024-02-12T16:10:03.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.574590+0000 mon.a (mon.0) 1189 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=1351570964}) 2024-02-12T16:10:03.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:03 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.706010+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e276: 12 total, 12 up, 11 in 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: audit 2024-02-12T16:10:02.706242+0000 mon.a (mon.0) 1172 : audit [INF] from='client.25030 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:02.706316+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e275: 12 total, 12 up, 11 in 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.524611+0000 mon.a (mon.0) 1174 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=1995728569,config=2903391811,health=438477494,kv=2355815837}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.526392+0000 mon.a (mon.0) 1175 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2558004167,logm=255244303}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.527863+0000 mon.a (mon.0) 1176 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=654397940}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.529253+0000 mon.a (mon.0) 1177 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4273462403}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.530802+0000 mon.a (mon.0) 1178 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2368527503}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.532393+0000 mon.a (mon.0) 1179 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3070668759}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.539219+0000 mon.a (mon.0) 1180 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1959311415}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.545883+0000 mon.a (mon.0) 1181 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2722760719}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.551937+0000 mon.a (mon.0) 1182 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592798890}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.557155+0000 mon.a (mon.0) 1183 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3539499822}) 2024-02-12T16:10:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.561921+0000 mon.a (mon.0) 1184 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996003035}) 2024-02-12T16:10:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.563880+0000 mon.a (mon.0) 1185 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=43} crc {logm=1761306972,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=479563086,mon_config_key=1175164893}) 2024-02-12T16:10:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.564723+0000 mon.a (mon.0) 1186 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2534505019}) 2024-02-12T16:10:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.565523+0000 mon.a (mon.0) 1187 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=690416649}) 2024-02-12T16:10:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.566399+0000 mon.a (mon.0) 1188 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=3709686934,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=2619014876,osdmap=270461205}) 2024-02-12T16:10:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.574590+0000 mon.a (mon.0) 1189 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=1351570964}) 2024-02-12T16:10:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:03 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.706010+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e276: 12 total, 12 up, 11 in 2024-02-12T16:10:04.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:04.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:04.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:04.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: audit 2024-02-12T16:10:02.706242+0000 mon.a (mon.0) 1172 : audit [INF] from='client.25030 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-02-12T16:10:04.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:02.706316+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e275: 12 total, 12 up, 11 in 2024-02-12T16:10:04.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.524611+0000 mon.a (mon.0) 1174 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=1995728569,config=2903391811,health=438477494,kv=2355815837}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.526392+0000 mon.a (mon.0) 1175 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2558004167,logm=255244303}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.527863+0000 mon.a (mon.0) 1176 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=654397940}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.529253+0000 mon.a (mon.0) 1177 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4273462403}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.530802+0000 mon.a (mon.0) 1178 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2368527503}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.532393+0000 mon.a (mon.0) 1179 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3070668759}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.539219+0000 mon.a (mon.0) 1180 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1959311415}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.545883+0000 mon.a (mon.0) 1181 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2722760719}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.551937+0000 mon.a (mon.0) 1182 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592798890}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.557155+0000 mon.a (mon.0) 1183 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3539499822}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.561921+0000 mon.a (mon.0) 1184 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996003035}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.563880+0000 mon.a (mon.0) 1185 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=17,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=43} crc {logm=1761306972,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=479563086,mon_config_key=1175164893}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.564723+0000 mon.a (mon.0) 1186 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2534505019}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.565523+0000 mon.a (mon.0) 1187 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=690416649}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.566399+0000 mon.a (mon.0) 1188 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=3,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=79} crc {mon_config_key=3709686934,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=2619014876,osdmap=270461205}) 2024-02-12T16:10:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.574590+0000 mon.a (mon.0) 1189 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=1351570964}) 2024-02-12T16:10:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:03 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.706010+0000 mon.a (mon.0) 1190 : cluster [DBG] osdmap e276: 12 total, 12 up, 11 in 2024-02-12T16:10:04.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:10:04.177 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:10:04.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:10:04.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:04.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:04.235 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:10:04.276 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:10:04.300 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:10:04.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:04.332 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:10:04.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:04.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:04.362 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:10:04.393 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:10:04.426 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:10:04.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:04.467 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:10:04.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:04.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:04.503 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:10:04.505 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.506 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:10:04.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:04.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:04.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:04.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:04 smithi169 bash[19604]: cluster 2024-02-12T16:10:03.638197+0000 mgr.y (mgr.14146) 740 : cluster [DBG] pgmap v918: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:10:04.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:04 smithi169 bash[19604]: audit 2024-02-12T16:10:04.116200+0000 mon.a (mon.0) 1191 : audit [DBG] from='client.? v1:172.21.15.121:0/912673989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:04.771 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:04.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:04.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:04.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:04.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:04.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 127854 127838 142.659 33.7812 0.00883876 0.00700769 2024-02-12T16:10:05.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:05.019 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:04 smithi123 bash[19119]: cluster 2024-02-12T16:10:03.638197+0000 mgr.y (mgr.14146) 740 : cluster [DBG] pgmap v918: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:10:05.019 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:04 smithi123 bash[19119]: audit 2024-02-12T16:10:04.116200+0000 mon.a (mon.0) 1191 : audit [DBG] from='client.? v1:172.21.15.121:0/912673989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:05.041 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:05.148 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:04 smithi121 bash[18489]: cluster 2024-02-12T16:10:03.638197+0000 mgr.y (mgr.14146) 740 : cluster [DBG] pgmap v918: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:10:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:04 smithi121 bash[18489]: audit 2024-02-12T16:10:04.116200+0000 mon.a (mon.0) 1191 : audit [DBG] from='client.? v1:172.21.15.121:0/912673989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:05.176 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:05.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:05.315 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:05.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:05.451 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:05.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:05.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:05.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:05.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:05.781 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:10:05.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:05.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:05.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:05.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 129088 129072 141.51 77.125 0.00479585 0.0070641 2024-02-12T16:10:05.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:05.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:06.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:05 smithi169 bash[19604]: audit 2024-02-12T16:10:04.846050+0000 mon.a (mon.0) 1192 : audit [INF] from='client.25085 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:10:06.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:05 smithi169 bash[19604]: audit 2024-02-12T16:10:04.849426+0000 mon.b (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.121:0/349510283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:10:06.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:06.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:05 smithi123 bash[19119]: audit 2024-02-12T16:10:04.846050+0000 mon.a (mon.0) 1192 : audit [INF] from='client.25085 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:10:06.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:05 smithi123 bash[19119]: audit 2024-02-12T16:10:04.849426+0000 mon.b (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.121:0/349510283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:10:06.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:06.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:05 smithi121 bash[18489]: audit 2024-02-12T16:10:04.846050+0000 mon.a (mon.0) 1192 : audit [INF] from='client.25085 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:10:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:05 smithi121 bash[18489]: audit 2024-02-12T16:10:04.849426+0000 mon.b (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.121:0/349510283' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:10:06.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:06.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:06.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:06.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:06.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:06.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:06.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:06.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:06.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:06.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:06.626 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:06.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:06.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:06.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:06.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:06.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:06 smithi123 bash[19119]: cluster 2024-02-12T16:10:05.638790+0000 mgr.y (mgr.14146) 741 : cluster [DBG] pgmap v920: 17 pgs: 1 active+recovering, 16 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s; 7877/254060 objects misplaced (3.100%); 184 KiB/s, 2 objects/s recovering 2024-02-12T16:10:06.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:06 smithi123 bash[19119]: cluster 2024-02-12T16:10:05.755135+0000 mon.a (mon.0) 1193 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:06.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:06 smithi123 bash[19119]: audit 2024-02-12T16:10:05.763130+0000 mon.a (mon.0) 1194 : audit [INF] from='client.25085 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:10:06.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:06 smithi123 bash[19119]: cluster 2024-02-12T16:10:05.763220+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e277: 12 total, 12 up, 11 in 2024-02-12T16:10:06.770 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:06 smithi123 bash[19119]: cluster 2024-02-12T16:10:05.980747+0000 osd.10 (osd.10) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:06.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:06 smithi123 bash[19119]: cluster 2024-02-12T16:10:05.980807+0000 osd.10 (osd.10) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:06.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:06 smithi123 bash[19119]: cluster 2024-02-12T16:10:06.545964+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.5 scrub starts 2024-02-12T16:10:06.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:06.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:06.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:06.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:06.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 16 130160 130144 140.225 67 0.0111681 0.00712989 2024-02-12T16:10:06.994 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:07.030 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:07.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:06 smithi169 bash[19604]: cluster 2024-02-12T16:10:05.638790+0000 mgr.y (mgr.14146) 741 : cluster [DBG] pgmap v920: 17 pgs: 1 active+recovering, 16 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s; 7877/254060 objects misplaced (3.100%); 184 KiB/s, 2 objects/s recovering 2024-02-12T16:10:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:06 smithi169 bash[19604]: cluster 2024-02-12T16:10:05.755135+0000 mon.a (mon.0) 1193 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:06 smithi169 bash[19604]: audit 2024-02-12T16:10:05.763130+0000 mon.a (mon.0) 1194 : audit [INF] from='client.25085 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:10:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:06 smithi169 bash[19604]: cluster 2024-02-12T16:10:05.763220+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e277: 12 total, 12 up, 11 in 2024-02-12T16:10:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:06 smithi169 bash[19604]: cluster 2024-02-12T16:10:05.980747+0000 osd.10 (osd.10) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:06 smithi169 bash[19604]: cluster 2024-02-12T16:10:05.980807+0000 osd.10 (osd.10) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:06 smithi169 bash[19604]: cluster 2024-02-12T16:10:06.545964+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.5 scrub starts 2024-02-12T16:10:07.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:06 smithi121 bash[18489]: cluster 2024-02-12T16:10:05.638790+0000 mgr.y (mgr.14146) 741 : cluster [DBG] pgmap v920: 17 pgs: 1 active+recovering, 16 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s; 7877/254060 objects misplaced (3.100%); 184 KiB/s, 2 objects/s recovering 2024-02-12T16:10:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:06 smithi121 bash[18489]: cluster 2024-02-12T16:10:05.755135+0000 mon.a (mon.0) 1193 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:06 smithi121 bash[18489]: audit 2024-02-12T16:10:05.763130+0000 mon.a (mon.0) 1194 : audit [INF] from='client.25085 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:10:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:06 smithi121 bash[18489]: cluster 2024-02-12T16:10:05.763220+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e277: 12 total, 12 up, 11 in 2024-02-12T16:10:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:06 smithi121 bash[18489]: cluster 2024-02-12T16:10:05.980747+0000 osd.10 (osd.10) 21 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:06 smithi121 bash[18489]: cluster 2024-02-12T16:10:05.980807+0000 osd.10 (osd.10) 22 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:06 smithi121 bash[18489]: cluster 2024-02-12T16:10:06.545964+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.5 scrub starts 2024-02-12T16:10:07.161 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:07.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:07.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:07.289 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:07.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:07.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:10:07.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:07.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:07.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:07.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:07.597 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:07.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:07.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:07.740 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:07.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:07.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:07.853 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:07 smithi123 bash[19119]: cluster 2024-02-12T16:10:06.773713+0000 mon.a (mon.0) 1196 : cluster [DBG] osdmap e278: 12 total, 12 up, 11 in 2024-02-12T16:10:07.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:07.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:07.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:07.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 131371 131355 139.131 75.6875 0.0183921 0.00718341 2024-02-12T16:10:07.958 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:10:07.958 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:10:07.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:08.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:08.023 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:07 smithi169 bash[19604]: cluster 2024-02-12T16:10:06.773713+0000 mon.a (mon.0) 1196 : cluster [DBG] osdmap e278: 12 total, 12 up, 11 in 2024-02-12T16:10:08.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:08.114 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:07 smithi121 bash[18489]: cluster 2024-02-12T16:10:06.773713+0000 mon.a (mon.0) 1196 : cluster [DBG] osdmap e278: 12 total, 12 up, 11 in 2024-02-12T16:10:08.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:08.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:08.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:08.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:08.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:08.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:08.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:08.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:08.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:08.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:10:08.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:08.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:08.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-12T16:10:08.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:08.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:08.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:08.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:08.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:10:08.955036 min lat: 0.000952532 max lat: 1.3461 avg lat: 0.00724427 2024-02-12T16:10:08.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:10:08.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 132510 132494 137.998 71.1875 0.00516425 0.00724427 2024-02-12T16:10:09.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:09.040 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:09.041 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:08 smithi123 bash[19119]: cluster 2024-02-12T16:10:07.639234+0000 mgr.y (mgr.14146) 742 : cluster [DBG] pgmap v923: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 7877/255862 objects misplaced (3.079%); 298 KiB/s, 4 objects/s recovering 2024-02-12T16:10:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:08 smithi169 bash[19604]: cluster 2024-02-12T16:10:07.639234+0000 mgr.y (mgr.14146) 742 : cluster [DBG] pgmap v923: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 7877/255862 objects misplaced (3.079%); 298 KiB/s, 4 objects/s recovering 2024-02-12T16:10:09.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:09.150 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:08 smithi121 bash[18489]: cluster 2024-02-12T16:10:07.639234+0000 mgr.y (mgr.14146) 742 : cluster [DBG] pgmap v923: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 7.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 7877/255862 objects misplaced (3.079%); 298 KiB/s, 4 objects/s recovering 2024-02-12T16:10:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:09.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:09.294 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:10:09.294 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:10:09.294 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-12T16:10:09.294 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:10:09.294 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:10:09.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-12T16:10:09.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:09.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:09.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:09.571 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:09.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:09.702 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:09.802 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:09 smithi121 bash[18489]: cluster 2024-02-12T16:10:08.499624+0000 osd.0 (osd.0) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:09.802 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:09 smithi121 bash[18489]: cluster 2024-02-12T16:10:08.499678+0000 osd.0 (osd.0) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:09.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:09.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:09 smithi123 bash[19119]: cluster 2024-02-12T16:10:08.499624+0000 osd.0 (osd.0) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:09 smithi123 bash[19119]: cluster 2024-02-12T16:10:08.499678+0000 osd.0 (osd.0) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:09 smithi123 bash[19119]: cluster 2024-02-12T16:10:08.751866+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.5 scrub ok 2024-02-12T16:10:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:09 smithi123 bash[19119]: cluster 2024-02-12T16:10:08.779527+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e279: 12 total, 12 up, 11 in 2024-02-12T16:10:09.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:09.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 133684 133668 136.939 73.375 0.00739765 0.00730031 2024-02-12T16:10:09.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:09.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:10.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:09 smithi169 bash[19604]: cluster 2024-02-12T16:10:08.499624+0000 osd.0 (osd.0) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:10.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:09 smithi169 bash[19604]: cluster 2024-02-12T16:10:08.499678+0000 osd.0 (osd.0) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:09 smithi169 bash[19604]: cluster 2024-02-12T16:10:08.751866+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.5 scrub ok 2024-02-12T16:10:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:09 smithi169 bash[19604]: cluster 2024-02-12T16:10:08.779527+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e279: 12 total, 12 up, 11 in 2024-02-12T16:10:10.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:10.095 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:10.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:10.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:09 smithi121 bash[18489]: cluster 2024-02-12T16:10:08.751866+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.5 scrub ok 2024-02-12T16:10:10.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:09 smithi121 bash[18489]: cluster 2024-02-12T16:10:08.779527+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e279: 12 total, 12 up, 11 in 2024-02-12T16:10:10.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:10.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:10.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:10.381 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:10.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:10.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:10.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:10.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:10.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:10.652 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:10.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:10.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:10.781 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:10.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:10.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:10.909 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:10.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:10.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 134829 134813 135.884 71.5625 0.00845759 0.00735646 2024-02-12T16:10:11.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:10 smithi169 bash[19604]: cluster 2024-02-12T16:10:09.279697+0000 osd.0 (osd.0) 37 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:10:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:10 smithi169 bash[19604]: cluster 2024-02-12T16:10:09.639791+0000 mgr.y (mgr.14146) 743 : cluster [DBG] pgmap v925: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.0 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 7877/260848 objects misplaced (3.020%); 248 KiB/s, 3 objects/s recovering 2024-02-12T16:10:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:10 smithi169 bash[19604]: cluster 2024-02-12T16:10:09.651161+0000 osd.3 (osd.3) 29 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:10:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:10 smithi169 bash[19604]: cluster 2024-02-12T16:10:10.310099+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e280: 12 total, 12 up, 11 in 2024-02-12T16:10:11.056 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:11.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:10 smithi121 bash[18489]: cluster 2024-02-12T16:10:09.279697+0000 osd.0 (osd.0) 37 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:10:11.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:10 smithi121 bash[18489]: cluster 2024-02-12T16:10:09.639791+0000 mgr.y (mgr.14146) 743 : cluster [DBG] pgmap v925: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.0 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 7877/260848 objects misplaced (3.020%); 248 KiB/s, 3 objects/s recovering 2024-02-12T16:10:11.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:10 smithi121 bash[18489]: cluster 2024-02-12T16:10:09.651161+0000 osd.3 (osd.3) 29 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:10:11.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:10 smithi121 bash[18489]: cluster 2024-02-12T16:10:10.310099+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e280: 12 total, 12 up, 11 in 2024-02-12T16:10:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:11.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:10 smithi123 bash[19119]: cluster 2024-02-12T16:10:09.279697+0000 osd.0 (osd.0) 37 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:10:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:10 smithi123 bash[19119]: cluster 2024-02-12T16:10:09.639791+0000 mgr.y (mgr.14146) 743 : cluster [DBG] pgmap v925: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.0 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 7877/260848 objects misplaced (3.020%); 248 KiB/s, 3 objects/s recovering 2024-02-12T16:10:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:10 smithi123 bash[19119]: cluster 2024-02-12T16:10:09.651161+0000 osd.3 (osd.3) 29 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:10:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:10 smithi123 bash[19119]: cluster 2024-02-12T16:10:10.310099+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e280: 12 total, 12 up, 11 in 2024-02-12T16:10:11.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:11.194 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:11.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:11.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:11.330 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:11.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:11.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:11.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:11.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:11.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:11.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:11.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:11.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:11.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:11.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:11.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:11.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:11.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 16 135985 135969 134.874 72.25 0.0055343 0.00741106 2024-02-12T16:10:11.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:12.002 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:12.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:12.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:12.180 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:12.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:12.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:12.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:12.404 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:12.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:12.535 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:12.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:12.668 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:12.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:12.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:12.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:12.903 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:12.934 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:12.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:12.957 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 137116 137100 133.871 70.6875 0.00872382 0.00746794 2024-02-12T16:10:13.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:13.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:13.076 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:12 smithi169 bash[19604]: cluster 2024-02-12T16:10:11.208951+0000 osd.0 (osd.0) 38 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:10:13.077 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:12 smithi169 bash[19604]: cluster 2024-02-12T16:10:11.539372+0000 osd.3 (osd.3) 30 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:10:13.077 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:12 smithi169 bash[19604]: cluster 2024-02-12T16:10:11.640235+0000 mgr.y (mgr.14146) 744 : cluster [DBG] pgmap v927: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 8.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 59 MiB/s wr, 947 op/s; 7796/265204 objects misplaced (2.940%); 882 KiB/s, 13 objects/s recovering 2024-02-12T16:10:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:13.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:13.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:12 smithi123 bash[19119]: cluster 2024-02-12T16:10:11.208951+0000 osd.0 (osd.0) 38 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:10:13.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:12 smithi123 bash[19119]: cluster 2024-02-12T16:10:11.539372+0000 osd.3 (osd.3) 30 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:10:13.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:12 smithi123 bash[19119]: cluster 2024-02-12T16:10:11.640235+0000 mgr.y (mgr.14146) 744 : cluster [DBG] pgmap v927: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 8.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 59 MiB/s wr, 947 op/s; 7796/265204 objects misplaced (2.940%); 882 KiB/s, 13 objects/s recovering 2024-02-12T16:10:13.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:12 smithi121 bash[18489]: cluster 2024-02-12T16:10:11.208951+0000 osd.0 (osd.0) 38 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:10:13.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:12 smithi121 bash[18489]: cluster 2024-02-12T16:10:11.539372+0000 osd.3 (osd.3) 30 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:10:13.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:12 smithi121 bash[18489]: cluster 2024-02-12T16:10:11.640235+0000 mgr.y (mgr.14146) 744 : cluster [DBG] pgmap v927: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 8.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 59 MiB/s wr, 947 op/s; 7796/265204 objects misplaced (2.940%); 882 KiB/s, 13 objects/s recovering 2024-02-12T16:10:13.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:13.192 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:13.193 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:13.208 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:13.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:13.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:13.356 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:13.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:13.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:13.490 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:13.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:13.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:13.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:13.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:13.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:13.666 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:10:13.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:13.747 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:13.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:13.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:13.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:13.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:13.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 138050 138034 132.71 58.375 0.0102454 0.00753304 2024-02-12T16:10:13.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:13.992 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:13 smithi123 bash[19119]: cluster 2024-02-12T16:10:13.525714+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.c scrub starts 2024-02-12T16:10:14.018 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:14.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:14.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:13 smithi121 bash[18489]: cluster 2024-02-12T16:10:13.525714+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.c scrub starts 2024-02-12T16:10:14.174 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:14.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:14.288 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:13 smithi169 bash[19604]: cluster 2024-02-12T16:10:13.525714+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.c scrub starts 2024-02-12T16:10:14.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:14.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:14.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:14.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:14.574 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:14.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:14.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:14.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:14.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:14.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:14.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 139170 139154 131.76 70 0.00575617 0.00758761 2024-02-12T16:10:14.971 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:14.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:15.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:15.083 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:14 smithi123 bash[19119]: cluster 2024-02-12T16:10:13.371757+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:10:15.084 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:14 smithi123 bash[19119]: cluster 2024-02-12T16:10:13.640680+0000 mgr.y (mgr.14146) 745 : cluster [DBG] pgmap v928: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 58 MiB/s wr, 922 op/s; 7796/267382 objects misplaced (2.916%); 755 KiB/s, 11 objects/s recovering 2024-02-12T16:10:15.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:14 smithi121 bash[18489]: cluster 2024-02-12T16:10:13.371757+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:10:15.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:14 smithi121 bash[18489]: cluster 2024-02-12T16:10:13.640680+0000 mgr.y (mgr.14146) 745 : cluster [DBG] pgmap v928: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 58 MiB/s wr, 922 op/s; 7796/267382 objects misplaced (2.916%); 755 KiB/s, 11 objects/s recovering 2024-02-12T16:10:15.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:15.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:15.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:15.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:15.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:14 smithi169 bash[19604]: cluster 2024-02-12T16:10:13.371757+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:10:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:14 smithi169 bash[19604]: cluster 2024-02-12T16:10:13.640680+0000 mgr.y (mgr.14146) 745 : cluster [DBG] pgmap v928: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 58 MiB/s wr, 922 op/s; 7796/267382 objects misplaced (2.916%); 755 KiB/s, 11 objects/s recovering 2024-02-12T16:10:15.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:15.405 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:15.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:15.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:15.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:15.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:15.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:15.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:15.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:15.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:15.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:15.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:15.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:15.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:15.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 140263 140247 130.813 68.3125 0.00855932 0.00764134 2024-02-12T16:10:16.038 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:16.066 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:16.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:16.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:16.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:16.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:16.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:16.321 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:16 smithi123 bash[19119]: cluster 2024-02-12T16:10:15.641109+0000 mgr.y (mgr.14146) 746 : cluster [DBG] pgmap v929: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.4 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s; 7713/276442 objects misplaced (2.790%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:16.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:16.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:16.434 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:16 smithi169 bash[19604]: cluster 2024-02-12T16:10:15.641109+0000 mgr.y (mgr.14146) 746 : cluster [DBG] pgmap v929: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.4 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s; 7713/276442 objects misplaced (2.790%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:16.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:16.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:16.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:16.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:16.626 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:16.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:16 smithi121 bash[18489]: cluster 2024-02-12T16:10:15.641109+0000 mgr.y (mgr.14146) 746 : cluster [DBG] pgmap v929: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.4 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s; 7713/276442 objects misplaced (2.790%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:16.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:16.756 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:16.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:16.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:16.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:16.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:16.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 16 141297 141281 129.839 64.625 0.00631291 0.0076996 2024-02-12T16:10:16.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:17.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:17.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:17.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:17.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:17.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:17.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:17.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:17.426 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:17.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:17.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:17 smithi123 bash[19119]: cluster 2024-02-12T16:10:15.947227+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.c scrub ok 2024-02-12T16:10:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:17 smithi123 bash[19119]: cluster 2024-02-12T16:10:16.539840+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:10:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:17 smithi123 bash[19119]: cluster 2024-02-12T16:10:16.657343+0000 osd.3 (osd.3) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:17 smithi123 bash[19119]: cluster 2024-02-12T16:10:16.657402+0000 osd.3 (osd.3) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:17 smithi121 bash[18489]: cluster 2024-02-12T16:10:15.947227+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.c scrub ok 2024-02-12T16:10:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:17 smithi121 bash[18489]: cluster 2024-02-12T16:10:16.539840+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:10:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:17 smithi121 bash[18489]: cluster 2024-02-12T16:10:16.657343+0000 osd.3 (osd.3) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:17 smithi121 bash[18489]: cluster 2024-02-12T16:10:16.657402+0000 osd.3 (osd.3) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:17.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:17.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:17 smithi169 bash[19604]: cluster 2024-02-12T16:10:15.947227+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.c scrub ok 2024-02-12T16:10:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:17 smithi169 bash[19604]: cluster 2024-02-12T16:10:16.539840+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:10:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:17 smithi169 bash[19604]: cluster 2024-02-12T16:10:16.657343+0000 osd.3 (osd.3) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:17 smithi169 bash[19604]: cluster 2024-02-12T16:10:16.657402+0000 osd.3 (osd.3) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:17.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:17.832 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:17.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:17.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 142240 142224 128.812 58.9375 0.00567003 0.00776128 2024-02-12T16:10:17.978 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:17.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:18.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:18.112 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:18.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:18.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:18.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:18.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:18.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:18 smithi123 bash[19119]: cluster 2024-02-12T16:10:17.238999+0000 osd.4 (osd.4) 29 : cluster [DBG] 3.3 scrub starts 2024-02-12T16:10:18.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:18 smithi123 bash[19119]: cluster 2024-02-12T16:10:17.324493+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e281: 12 total, 12 up, 11 in 2024-02-12T16:10:18.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:18 smithi123 bash[19119]: cluster 2024-02-12T16:10:17.641486+0000 mgr.y (mgr.14146) 747 : cluster [DBG] pgmap v931: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 7713/278404 objects misplaced (2.770%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:18.375 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:18.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:18.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:18.486 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:18 smithi121 bash[18489]: cluster 2024-02-12T16:10:17.238999+0000 osd.4 (osd.4) 29 : cluster [DBG] 3.3 scrub starts 2024-02-12T16:10:18.486 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:18 smithi121 bash[18489]: cluster 2024-02-12T16:10:17.324493+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e281: 12 total, 12 up, 11 in 2024-02-12T16:10:18.486 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:18 smithi121 bash[18489]: cluster 2024-02-12T16:10:17.641486+0000 mgr.y (mgr.14146) 747 : cluster [DBG] pgmap v931: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 7713/278404 objects misplaced (2.770%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:18.508 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:18.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:18.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:18.617 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:18 smithi169 bash[19604]: cluster 2024-02-12T16:10:17.238999+0000 osd.4 (osd.4) 29 : cluster [DBG] 3.3 scrub starts 2024-02-12T16:10:18.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:18 smithi169 bash[19604]: cluster 2024-02-12T16:10:17.324493+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e281: 12 total, 12 up, 11 in 2024-02-12T16:10:18.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:18 smithi169 bash[19604]: cluster 2024-02-12T16:10:17.641486+0000 mgr.y (mgr.14146) 747 : cluster [DBG] pgmap v931: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 7713/278404 objects misplaced (2.770%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:18.642 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:18.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:18.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:18.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:18.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:18.805 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:10:18.805 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:10:18.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:18.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:18.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:18.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 15 143356 143341 127.969 69.8125 0.0149316 0.0078122 2024-02-12T16:10:19.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:19.040 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:19.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:19.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:19.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:19.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:19.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:19.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:19.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:19.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:19.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:19.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:19.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:19.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:19.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:19.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:19.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:19.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-12T16:10:19.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:19.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:19.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:19.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 144303 144287 126.999 59.125 0.00491081 0.00787117 2024-02-12T16:10:19.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:19.980 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:10:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:20.008 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:10:20.008 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T16:10:20.008 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:10:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:20.011 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:10:20.039 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:10:20.065 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:10:20.101 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:10:20.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:20.128 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:10:20.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:20.149 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:10:20.175 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:10:20.204 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:10:20.234 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:10:20.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:20.260 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:10:20.270 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:20.292 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:10:20.293 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.293 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.293 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.293 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:10:20.321 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:10:20.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:20.403 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:20.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:20.541 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:20.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:20.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:20.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:20.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: cluster 2024-02-12T16:10:19.641931+0000 mgr.y (mgr.14146) 748 : cluster [DBG] pgmap v932: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 8.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s; 7713/282914 objects misplaced (2.726%); 569 KiB/s, 8 objects/s recovering 2024-02-12T16:10:20.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: audit 2024-02-12T16:10:19.763462+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? v1:172.21.15.121:0/3376372152' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:20.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: audit 2024-02-12T16:10:19.944999+0000 mon.a (mon.0) 1201 : audit [DBG] from='client.? v1:172.21.15.121:0/1487392977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:20.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: cluster 2024-02-12T16:10:20.059342+0000 osd.4 (osd.4) 30 : cluster [DBG] 3.3 scrub ok 2024-02-12T16:10:20.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: cluster 2024-02-12T16:10:20.307674+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:10:20.786 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: cluster 2024-02-12T16:10:20.307709+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-02-12T16:10:20.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: audit 2024-02-12T16:10:20.312125+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? v1:172.21.15.121:0/3376372152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:10:20.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:20 smithi123 bash[19119]: cluster 2024-02-12T16:10:20.312207+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e282: 12 total, 12 up, 11 in 2024-02-12T16:10:20.806 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:20.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:20.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: cluster 2024-02-12T16:10:19.641931+0000 mgr.y (mgr.14146) 748 : cluster [DBG] pgmap v932: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 8.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s; 7713/282914 objects misplaced (2.726%); 569 KiB/s, 8 objects/s recovering 2024-02-12T16:10:20.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: audit 2024-02-12T16:10:19.763462+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? v1:172.21.15.121:0/3376372152' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:20.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: audit 2024-02-12T16:10:19.944999+0000 mon.a (mon.0) 1201 : audit [DBG] from='client.? v1:172.21.15.121:0/1487392977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:20.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: cluster 2024-02-12T16:10:20.059342+0000 osd.4 (osd.4) 30 : cluster [DBG] 3.3 scrub ok 2024-02-12T16:10:20.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: cluster 2024-02-12T16:10:20.307674+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:10:20.920 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: cluster 2024-02-12T16:10:20.307709+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-02-12T16:10:20.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: audit 2024-02-12T16:10:20.312125+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? v1:172.21.15.121:0/3376372152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:10:20.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:20 smithi169 bash[19604]: cluster 2024-02-12T16:10:20.312207+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e282: 12 total, 12 up, 11 in 2024-02-12T16:10:20.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:20.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:20.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 145409 145393 126.195 69.125 0.00564686 0.00792159 2024-02-12T16:10:21.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:21.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:21.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:21.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: cluster 2024-02-12T16:10:19.641931+0000 mgr.y (mgr.14146) 748 : cluster [DBG] pgmap v932: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 8.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 948 op/s; 7713/282914 objects misplaced (2.726%); 569 KiB/s, 8 objects/s recovering 2024-02-12T16:10:21.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: audit 2024-02-12T16:10:19.763462+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? v1:172.21.15.121:0/3376372152' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:21.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: audit 2024-02-12T16:10:19.944999+0000 mon.a (mon.0) 1201 : audit [DBG] from='client.? v1:172.21.15.121:0/1487392977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:21.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: cluster 2024-02-12T16:10:20.059342+0000 osd.4 (osd.4) 30 : cluster [DBG] 3.3 scrub ok 2024-02-12T16:10:21.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: cluster 2024-02-12T16:10:20.307674+0000 mon.a (mon.0) 1202 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:10:21.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: cluster 2024-02-12T16:10:20.307709+0000 mon.a (mon.0) 1203 : cluster [INF] Cluster is now healthy 2024-02-12T16:10:21.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: audit 2024-02-12T16:10:20.312125+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? v1:172.21.15.121:0/3376372152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:10:21.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:20 smithi121 bash[18489]: cluster 2024-02-12T16:10:20.312207+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e282: 12 total, 12 up, 11 in 2024-02-12T16:10:21.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:21.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:21.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:21.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:21.332 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:21.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:21.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:21.461 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:21.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:21.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:21.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:21.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:21.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:21.734 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:21.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:21.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:21.866 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:21.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:21.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 146528 146512 125.424 69.9375 0.0263089 0.0079704 2024-02-12T16:10:21.970 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:22.006 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:22.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:22.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:22.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:22.268 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:22.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:22.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:22.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:22.541 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:22.645 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:22.670 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:22.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:22 smithi169 bash[19604]: cluster 2024-02-12T16:10:21.132099+0000 osd.1 (osd.1) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:22 smithi169 bash[19604]: cluster 2024-02-12T16:10:21.132162+0000 osd.1 (osd.1) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:22 smithi169 bash[19604]: cluster 2024-02-12T16:10:21.642434+0000 mgr.y (mgr.14146) 749 : cluster [DBG] pgmap v934: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s; 7630/286808 objects misplaced (2.660%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:22 smithi169 bash[19604]: cluster 2024-02-12T16:10:21.711147+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e283: 12 total, 12 up, 11 in 2024-02-12T16:10:22.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:22.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:22.937 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:22.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:22.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:22.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 147654 147638 124.68 70.375 0.00524485 0.00801792 2024-02-12T16:10:23.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:23.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:23.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:22 smithi123 bash[19119]: cluster 2024-02-12T16:10:21.132099+0000 osd.1 (osd.1) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:22 smithi123 bash[19119]: cluster 2024-02-12T16:10:21.132162+0000 osd.1 (osd.1) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:22 smithi123 bash[19119]: cluster 2024-02-12T16:10:21.642434+0000 mgr.y (mgr.14146) 749 : cluster [DBG] pgmap v934: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s; 7630/286808 objects misplaced (2.660%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:22 smithi123 bash[19119]: cluster 2024-02-12T16:10:21.711147+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e283: 12 total, 12 up, 11 in 2024-02-12T16:10:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:22 smithi121 bash[18489]: cluster 2024-02-12T16:10:21.132099+0000 osd.1 (osd.1) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:22 smithi121 bash[18489]: cluster 2024-02-12T16:10:21.132162+0000 osd.1 (osd.1) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:22 smithi121 bash[18489]: cluster 2024-02-12T16:10:21.642434+0000 mgr.y (mgr.14146) 749 : cluster [DBG] pgmap v934: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s; 7630/286808 objects misplaced (2.660%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:22 smithi121 bash[18489]: cluster 2024-02-12T16:10:21.711147+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e283: 12 total, 12 up, 11 in 2024-02-12T16:10:23.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:23.200 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:23.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:23.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:23.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:23.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:23.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:23.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:23.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:23.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:23.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:23.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:23.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:23.696 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:23.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:10:23.733 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:23.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:23.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:10:23.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:23.881 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:23.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:23.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 16 148787 148771 123.962 70.8125 0.00615163 0.00806408 2024-02-12T16:10:23.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:24.009 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:24.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:24.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:24.257 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:24.286 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:24.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:24.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:24.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:24.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:24.591 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:10:24.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:24.689 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:24.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:24.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:24 smithi121 bash[18489]: cluster 2024-02-12T16:10:23.642881+0000 mgr.y (mgr.14146) 750 : cluster [DBG] pgmap v936: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 7630/288836 objects misplaced (2.642%); 840 KiB/s, 13 objects/s recovering 2024-02-12T16:10:24.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:24.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:24 smithi169 bash[19604]: cluster 2024-02-12T16:10:23.642881+0000 mgr.y (mgr.14146) 750 : cluster [DBG] pgmap v936: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 7630/288836 objects misplaced (2.642%); 840 KiB/s, 13 objects/s recovering 2024-02-12T16:10:24.957 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:24.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:24.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 149740 149724 123.115 59.5625 0.0144644 0.00812099 2024-02-12T16:10:25.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:25.086 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:25.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:24 smithi123 bash[19119]: cluster 2024-02-12T16:10:23.642881+0000 mgr.y (mgr.14146) 750 : cluster [DBG] pgmap v936: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 8.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 7630/288836 objects misplaced (2.642%); 840 KiB/s, 13 objects/s recovering 2024-02-12T16:10:25.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:25.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:25.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:25.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:25.356 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:25.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:25.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:25.495 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:25.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:25.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:25.624 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:25.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:25.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:25.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:25.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:25.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:25.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:25.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:25.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 16 150750 150734 122.336 63.125 0.0140027 0.00817208 2024-02-12T16:10:26.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:26.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:25 smithi169 bash[19604]: cluster 2024-02-12T16:10:24.403915+0000 osd.6 (osd.6) 28 : cluster [DBG] 3.a scrub starts 2024-02-12T16:10:26.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:25 smithi169 bash[19604]: cluster 2024-02-12T16:10:25.273041+0000 osd.5 (osd.5) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:26.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:25 smithi169 bash[19604]: cluster 2024-02-12T16:10:25.273097+0000 osd.5 (osd.5) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:26.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:25 smithi169 bash[19604]: cluster 2024-02-12T16:10:25.312714+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e284: 12 total, 12 up, 11 in 2024-02-12T16:10:26.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:26.050 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:10:26.050 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:10:26.050 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-12T16:10:26.050 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-02-12T16:10:26.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-12T16:10:26.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:25 smithi121 bash[18489]: cluster 2024-02-12T16:10:24.403915+0000 osd.6 (osd.6) 28 : cluster [DBG] 3.a scrub starts 2024-02-12T16:10:26.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:25 smithi121 bash[18489]: cluster 2024-02-12T16:10:25.273041+0000 osd.5 (osd.5) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:26.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:25 smithi121 bash[18489]: cluster 2024-02-12T16:10:25.273097+0000 osd.5 (osd.5) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:26.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:25 smithi121 bash[18489]: cluster 2024-02-12T16:10:25.312714+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e284: 12 total, 12 up, 11 in 2024-02-12T16:10:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:25 smithi123 bash[19119]: cluster 2024-02-12T16:10:24.403915+0000 osd.6 (osd.6) 28 : cluster [DBG] 3.a scrub starts 2024-02-12T16:10:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:25 smithi123 bash[19119]: cluster 2024-02-12T16:10:25.273041+0000 osd.5 (osd.5) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:25 smithi123 bash[19119]: cluster 2024-02-12T16:10:25.273097+0000 osd.5 (osd.5) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:25 smithi123 bash[19119]: cluster 2024-02-12T16:10:25.312714+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e284: 12 total, 12 up, 11 in 2024-02-12T16:10:26.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:26.166 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:26.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:26.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:26.301 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:26.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:26.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:26.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:26.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:26.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:26.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:26.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:26.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:26.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:26.958 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 151789 151773 121.6 64.9375 0.00522662 0.00822151 2024-02-12T16:10:26.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:26.976 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:26 smithi121 bash[18489]: cluster 2024-02-12T16:10:25.643231+0000 mgr.y (mgr.14146) 751 : cluster [DBG] pgmap v938: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 7549/297684 objects misplaced (2.536%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:10:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:26.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:26 smithi169 bash[19604]: cluster 2024-02-12T16:10:25.643231+0000 mgr.y (mgr.14146) 751 : cluster [DBG] pgmap v938: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 7549/297684 objects misplaced (2.536%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:10:27.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:27.103 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:27.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:27.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:26 smithi123 bash[19119]: cluster 2024-02-12T16:10:25.643231+0000 mgr.y (mgr.14146) 751 : cluster [DBG] pgmap v938: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 7549/297684 objects misplaced (2.536%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:10:27.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:27.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:27.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:27.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:27.382 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:27.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:27.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:27.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:27.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:27.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:27.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:27.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:27.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:27.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:27.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:27 smithi121 bash[18489]: cluster 2024-02-12T16:10:26.671858+0000 osd.6 (osd.6) 29 : cluster [DBG] 3.a scrub ok 2024-02-12T16:10:27.928 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:27.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:27.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 152764 152748 120.832 60.9375 0.0055959 0.0082726 2024-02-12T16:10:28.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:28.038 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:27 smithi123 bash[19119]: cluster 2024-02-12T16:10:26.671858+0000 osd.6 (osd.6) 29 : cluster [DBG] 3.a scrub ok 2024-02-12T16:10:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:27 smithi169 bash[19604]: cluster 2024-02-12T16:10:26.671858+0000 osd.6 (osd.6) 29 : cluster [DBG] 3.a scrub ok 2024-02-12T16:10:28.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:28.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:28.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:28.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:28.328 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:28.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:28.457 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:28.537 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:28.538 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:28.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:28.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:28.715 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:28.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:28.826 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:28 smithi123 bash[19119]: cluster 2024-02-12T16:10:27.433192+0000 osd.9 (osd.9) 28 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:28.826 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:28 smithi123 bash[19119]: cluster 2024-02-12T16:10:27.433248+0000 osd.9 (osd.9) 29 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:28.826 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:28 smithi123 bash[19119]: cluster 2024-02-12T16:10:27.643699+0000 mgr.y (mgr.14146) 752 : cluster [DBG] pgmap v939: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 55 MiB/s wr, 873 op/s; 7549/299626 objects misplaced (2.519%); 707 KiB/s, 11 objects/s recovering 2024-02-12T16:10:28.826 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:28 smithi123 bash[19119]: cluster 2024-02-12T16:10:27.730329+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e285: 12 total, 12 up, 11 in 2024-02-12T16:10:28.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:28.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:28.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:10:28.956731 min lat: 0.000952532 max lat: 1.3461 avg lat: 0.00832377 2024-02-12T16:10:28.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:10:28.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 153764 153748 120.103 62.5 0.00782511 0.00832377 2024-02-12T16:10:28.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:28.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:28 smithi169 bash[19604]: cluster 2024-02-12T16:10:27.433192+0000 osd.9 (osd.9) 28 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:28 smithi169 bash[19604]: cluster 2024-02-12T16:10:27.433248+0000 osd.9 (osd.9) 29 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:28 smithi169 bash[19604]: cluster 2024-02-12T16:10:27.643699+0000 mgr.y (mgr.14146) 752 : cluster [DBG] pgmap v939: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 55 MiB/s wr, 873 op/s; 7549/299626 objects misplaced (2.519%); 707 KiB/s, 11 objects/s recovering 2024-02-12T16:10:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:28 smithi169 bash[19604]: cluster 2024-02-12T16:10:27.730329+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e285: 12 total, 12 up, 11 in 2024-02-12T16:10:29.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:29.107 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:29.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:29.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:29.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:28 smithi121 bash[18489]: cluster 2024-02-12T16:10:27.433192+0000 osd.9 (osd.9) 28 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:29.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:28 smithi121 bash[18489]: cluster 2024-02-12T16:10:27.433248+0000 osd.9 (osd.9) 29 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:29.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:28 smithi121 bash[18489]: cluster 2024-02-12T16:10:27.643699+0000 mgr.y (mgr.14146) 752 : cluster [DBG] pgmap v939: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 55 MiB/s wr, 873 op/s; 7549/299626 objects misplaced (2.519%); 707 KiB/s, 11 objects/s recovering 2024-02-12T16:10:29.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:28 smithi121 bash[18489]: cluster 2024-02-12T16:10:27.730329+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e285: 12 total, 12 up, 11 in 2024-02-12T16:10:29.252 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:29.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:29.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:29.380 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:29.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:29.477 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:10:29.477 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:10:29.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:10:29.477 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:10:29.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.477 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:51085 2.a 2:50205334:::benchmark_data_smithi169_42458_object51084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-02-12T16:10:29.477 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:31.100378+0000", 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.36593707399999, 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26710181900000002, 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 51085 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100378+0000", 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100378+0000", 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100377+0000", 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8478000000000002e-05 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100395+0000", 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5480000000000004e-06 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100403+0000", 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3810000000000001e-06 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100407+0000", 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0719000000000002e-05 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100438+0000", 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113529 2024-02-12T16:10:29.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100551+0000", 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015132599999999999 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100703+0000", 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003122901 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.103826+0000", 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26346567700000001 2024-02-12T16:10:29.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.367291+0000", 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4165999999999993e-05 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.367375+0000", 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104505 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.367480+0000", 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26710181900000002 2024-02-12T16:10:29.482 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:51088 2.a 2:522a68c4:::benchmark_data_smithi169_42458_object51087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:04:31.100926+0000", 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 358.36538896799999, 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26694769499999998, 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 51088 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.483 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100926+0000", 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100926+0000", 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100925+0000", 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1386000000000002e-05 2024-02-12T16:10:29.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100946+0000", 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3570999999999999e-05 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100959+0000", 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2829999999999998e-06 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.100966+0000", 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001093934 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.102060+0000", 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102364 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.102162+0000", 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015711100000000001 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.102319+0000", 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00164646 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.103966+0000", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.263658801 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.367624+0000", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013148700000000001 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.367756+0000", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011798600000000001 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:04:31.367874+0000", 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26694769499999998 2024-02-12T16:10:29.487 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:68181 3.8 3:1cc2c950:::benchmark_data_smithi169_51390_object68180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:29.186859+0000", 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 60.279456303000003, 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28095186999999999, 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.488 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 68181 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.186859+0000", 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.186859+0000", 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.186857+0000", 2024-02-12T16:10:29.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1260999999999999e-05 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.186878+0000", 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0161e-05 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.186889+0000", 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8299999999999996e-06 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.186896+0000", 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1851000000000003e-05 2024-02-12T16:10:29.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.186938+0000", 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017637299999999999 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187115+0000", 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000134425 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187249+0000", 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003352935 2024-02-12T16:10:29.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.190602+0000", 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.277101402 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.467703+0000", 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7167999999999999e-05 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.467761+0000", 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0188000000000001e-05 2024-02-12T16:10:29.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.467811+0000", 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28095186999999999 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:68188 3.8 3:10a34daf:::benchmark_data_smithi169_51390_object68187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:29.187574+0000", 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 60.278740792999997, 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28047881299999999, 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.493 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 68188 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187574+0000", 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187574+0000", 2024-02-12T16:10:29.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187573+0000", 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2103000000000002e-05 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187595+0000", 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1635e-05 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187607+0000", 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.516e-06 2024-02-12T16:10:29.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.187613+0000", 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013518339999999999 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.188965+0000", 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000135356 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.189100+0000", 2024-02-12T16:10:29.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014305399999999999 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.189243+0000", 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001528405 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.190772+0000", 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.277112356 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.467884+0000", 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9161999999999997e-05 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.467983+0000", 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9801000000000005e-05 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.468053+0000", 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28047881299999999 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:68202 3.8 3:1440727b:::benchmark_data_smithi169_51390_object68201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:29.194389+0000", 2024-02-12T16:10:29.498 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 60.271926661000002, 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27382580099999998, 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 68202 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194389+0000", 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194389+0000", 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194387+0000", 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2815000000000001e-05 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194409+0000", 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2777e-05 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194422+0000", 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.908e-06 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194428+0000", 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2617000000000001e-05 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194481+0000", 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012386799999999999 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194605+0000", 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015823899999999999 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.194763+0000", 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021483779999999998 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.196911+0000", 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27120877300000001 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.468120+0000", 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2899999999999998e-05 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.468173+0000", 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1457e-05 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.468214+0000", 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27382580099999998 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:68222 3.8 3:18ff2cc3:::benchmark_data_smithi169_51390_object68221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:29.202865+0000", 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 60.263449868000002, 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26558862999999999, 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 68222 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.202865+0000", 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.202865+0000", 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.202863+0000", 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0656999999999999e-05 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.202884+0000", 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1072e-05 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.202895+0000", 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7639999999999996e-06 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.202900+0000", 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5577000000000001e-05 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.202945+0000", 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101877 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.203047+0000", 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000110931 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.203158+0000", 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022976239999999998 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.205456+0000", 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.262822422 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.468278+0000", 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109688 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.468388+0000", 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5969000000000005e-05 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.468454+0000", 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26558862999999999 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:68229 3.8 3:1e44d020:::benchmark_data_smithi169_51390_object68228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:29.208477+0000", 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 60.257838149000001, 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26060746000000001, 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 68229 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208477+0000", 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208477+0000", 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208476+0000", 2024-02-12T16:10:29.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7880000000000002e-05 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208493+0000", 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0057e-05 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208504+0000", 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3109999999999999e-06 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208508+0000", 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6022999999999997e-05 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208544+0000", 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6286000000000006e-05 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208640+0000", 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000125585 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:10:29.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.208766+0000", 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015414809999999999 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.210307+0000", 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.258712426 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.469020+0000", 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2199000000000001e-05 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.469052+0000", 2024-02-12T16:10:29.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2660999999999998e-05 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:29.469084+0000", 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26060746000000001 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126611 3.9 3:950c2c2c:::benchmark_data_smithi169_51390_object126610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791388+0000", 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674926671000001, 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26772749899999998, 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.514 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126611 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791388+0000", 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791388+0000", 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791388+0000", 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8183999999999998e-05 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791436+0000", 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2046e-05 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791448+0000", 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4779999999999999e-06 2024-02-12T16:10:29.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791453+0000", 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001870895 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793324+0000", 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.732e-06 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793328+0000", 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.217792496 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.011120+0000", 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012451000000000001 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.011245+0000", 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015789499999999999 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.011403+0000", 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010676267 2024-02-12T16:10:29.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.022079+0000", 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.034770759999999998 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.056850+0000", 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0664e-05 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.056920+0000", 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021955590000000001 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.059116+0000", 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26772749899999998 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126612 3.9 3:9fa155c8:::benchmark_data_smithi169_51390_object126611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791459+0000", 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674856022, 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27031909300000001, 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.520 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126612 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791459+0000", 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791459+0000", 2024-02-12T16:10:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791458+0000", 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9629e-05 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791488+0000", 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1253000000000001e-05 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791499+0000", 2024-02-12T16:10:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1429999999999997e-06 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791504+0000", 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001851253 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793356+0000", 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7689999999999998e-06 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793359+0000", 2024-02-12T16:10:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.224912001 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.018271+0000", 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012406900000000001 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.018395+0000", 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017242700000000001 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.018568+0000", 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013158391 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.031726+0000", 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027515121 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.059241+0000", 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6867000000000006e-05 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.059338+0000", 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002440078 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.061778+0000", 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27031909300000001 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126669 3.9 3:95895975:::benchmark_data_smithi169_51390_object126668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791509+0000", 2024-02-12T16:10:29.526 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674805782, 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.29773799299999998, 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126669 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791509+0000", 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791509+0000", 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791508+0000", 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7809000000000001e-05 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791536+0000", 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1215e-05 2024-02-12T16:10:29.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791547+0000", 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5040000000000002e-06 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791553+0000", 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001833108 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793386+0000", 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7829999999999998e-06 2024-02-12T16:10:29.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793391+0000", 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.228738468 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.022129+0000", 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013214300000000001 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.022261+0000", 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014894299999999999 2024-02-12T16:10:29.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.022410+0000", 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0093271550000000002 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.031737+0000", 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054540457000000001 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.086278+0000", 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6519000000000005e-05 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.086344+0000", 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029029339999999998 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.089247+0000", 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.29773799299999998 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126670 3.9 3:9c3dba13:::benchmark_data_smithi169_51390_object126669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791558+0000", 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674757315000001, 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30037945199999999, 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126670 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791558+0000", 2024-02-12T16:10:29.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791558+0000", 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791557+0000", 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9747999999999999e-05 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791607+0000", 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1654000000000001e-05 2024-02-12T16:10:29.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791618+0000", 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4729999999999999e-06 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791624+0000", 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001795645 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793419+0000", 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.16e-06 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793424+0000", 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23126442899999999 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.024688+0000", 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118513 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.024807+0000", 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014610900000000001 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.024953+0000", 2024-02-12T16:10:29.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010938662 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.035891+0000", 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.053436615999999999 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.089328+0000", 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9655999999999999e-05 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.089398+0000", 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025396759999999998 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.091937+0000", 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30037945199999999 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126680 3.9 3:93d06668:::benchmark_data_smithi169_51390_object126679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791628+0000", 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674686780999998, 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30235235100000002, 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.538 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126680 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791628+0000", 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791628+0000", 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791627+0000", 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6857999999999997e-05 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791684+0000", 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0973e-05 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791695+0000", 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1560000000000003e-06 2024-02-12T16:10:29.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791700+0000", 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001762237 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793463+0000", 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.473e-06 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793466+0000", 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23365667400000001 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.027123+0000", 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012397999999999999 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.027247+0000", 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000199168 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.027446+0000", 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0084561519999999998 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.035902+0000", 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.056097201999999999 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.091999+0000", 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3037000000000001e-05 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.092042+0000", 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001938426 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.093981+0000", 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30235235100000002 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126692 3.9 3:99eb602a:::benchmark_data_smithi169_51390_object126691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791705+0000", 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674609769, 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30405978700000003, 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.544 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126692 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791705+0000", 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791705+0000", 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791705+0000", 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5506999999999995e-05 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791770+0000", 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1719000000000001e-05 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791782+0000", 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.339e-06 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791787+0000", 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001737607 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793525+0000", 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.467e-06 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793528+0000", 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23598254499999999 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.029511+0000", 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106338 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.029617+0000", 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012442200000000001 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.029741+0000", 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012472631 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.042214+0000", 2024-02-12T16:10:29.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.051831923000000002 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.094046+0000", 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1319999999999997e-05 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.094087+0000", 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016778520000000001 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.095765+0000", 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30405978700000003 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.549 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126710 3.9 3:992fdf5f:::benchmark_data_smithi169_51390_object126709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791792+0000", 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674523245, 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30585088199999999, 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126710 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.550 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791792+0000", 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791792+0000", 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791791+0000", 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015660500000000001 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791948+0000", 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2594e-05 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791960+0000", 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.643e-06 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791966+0000", 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001589001 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793555+0000", 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1320000000000004e-06 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793559+0000", 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24489108100000001 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.038450+0000", 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010658 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.038557+0000", 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011963099999999999 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.038676+0000", 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020453209999999999 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.059129+0000", 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036694110000000002 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.095824+0000", 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8099000000000002e-05 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.095862+0000", 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017811330000000001 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.097643+0000", 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30585088199999999 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.555 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126717 3.9 3:99973c74:::benchmark_data_smithi169_51390_object126716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.791971+0000", 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674344006999998, 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30742333199999999, 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126717 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791971+0000", 2024-02-12T16:10:29.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791971+0000", 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.791970+0000", 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0322000000000003e-05 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792030+0000", 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2211e-05 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792043+0000", 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0980000000000001e-06 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792048+0000", 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015362349999999999 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793584+0000", 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0350000000000003e-06 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793588+0000", 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.248662667 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.042251+0000", 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016377500000000001 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.042414+0000", 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022080799999999999 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.042635+0000", 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016502446 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.059138+0000", 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.038549304999999999 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.097687+0000", 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1607999999999999e-05 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.097729+0000", 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016658720000000001 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.099394+0000", 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30742333199999999 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126721 3.9 3:982993fa:::benchmark_data_smithi169_51390_object126720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.792053+0000", 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674262589000001, 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.324857124, 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.561 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126721 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792053+0000", 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792053+0000", 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792052+0000", 2024-02-12T16:10:29.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8649999999999997e-05 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792100+0000", 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1049e-05 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792111+0000", 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5210000000000002e-06 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792117+0000", 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014950269999999999 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793612+0000", 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.444e-06 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793615+0000", 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.25311559700000003 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.046731+0000", 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126095 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.046857+0000", 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012723199999999999 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.046984+0000", 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012160794000000001 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.059145+0000", 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054352958 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.113498+0000", 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8776000000000001e-05 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.113557+0000", 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033529020000000001 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.116910+0000", 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.324857124 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126738 3.9 3:9aa59732:::benchmark_data_smithi169_51390_object126737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.792125+0000", 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674190006, 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32854916000000001, 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126738 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792125+0000", 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792125+0000", 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792124+0000", 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7546999999999997e-05 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792172+0000", 2024-02-12T16:10:29.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0508e-05 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792182+0000", 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3739999999999997e-06 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792188+0000", 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014510809999999999 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793639+0000", 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.252e-06 2024-02-12T16:10:29.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793642+0000", 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.25670302099999998 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.050345+0000", 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000165188 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.050510+0000", 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025177700000000001 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.050762+0000", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014608652999999999 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.065371+0000", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.051629860999999999 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.117001+0000", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1799999999999996e-05 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.117082+0000", 2024-02-12T16:10:29.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0035919910000000001 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.120674+0000", 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32854916000000001 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126756 3.9 3:95220020:::benchmark_data_smithi169_51390_object126755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.792193+0000", 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674122512, 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34631441000000002, 2024-02-12T16:10:29.572 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126756 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792193+0000", 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792193+0000", 2024-02-12T16:10:29.573 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792192+0000", 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3798000000000003e-05 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792236+0000", 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0404000000000001e-05 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792246+0000", 2024-02-12T16:10:29.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0819999999999998e-06 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792251+0000", 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014136940000000001 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793665+0000", 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0960000000000003e-06 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793669+0000", 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26814528100000001 2024-02-12T16:10:29.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.061814+0000", 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100738 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.061915+0000", 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117008 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.062032+0000", 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010410706 2024-02-12T16:10:29.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.072443+0000", 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.063212880999999999 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.135655+0000", 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3616999999999996e-05 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.135739+0000", 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027679879999999999 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.138507+0000", 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34631441000000002 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126758 3.9 3:9e14ba8d:::benchmark_data_smithi169_51390_object126757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.792256+0000", 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.674059333999999, 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34844240300000001, 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.578 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126758 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792256+0000", 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792256+0000", 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792255+0000", 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7743e-05 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792303+0000", 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0842999999999999e-05 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792314+0000", 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1699999999999996e-06 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792319+0000", 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001374765 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793693+0000", 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1210000000000003e-06 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793700+0000", 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27172925799999997 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.065429+0000", 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020852800000000001 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.065637+0000", 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030110199999999998 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.065938+0000", 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065167879999999999 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.072455+0000", 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.06610212 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.138557+0000", 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1142e-05 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.138599+0000", 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00209969 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.140698+0000", 2024-02-12T16:10:29.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34844240300000001 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:126773 3.9 3:9ed5d68c:::benchmark_data_smithi169_51390_object126772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e275)", 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:10:03.792324+0000", 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 25.673991636, 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36345487999999998, 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:10:29.584 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126773 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792324+0000", 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792324+0000", 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792323+0000", 2024-02-12T16:10:29.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8300000000000007e-05 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792391+0000", 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1633e-05 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792403+0000", 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3369999999999999e-06 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.792408+0000", 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013049069999999999 2024-02-12T16:10:29.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793713+0000", 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4410000000000002e-06 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:03.793716+0000", 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27530588099999997 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.069022+0000", 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016848199999999999 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.069191+0000", 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020155800000000001 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.069392+0000", 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016826988000000001 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.086219+0000", 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.066627025000000006 2024-02-12T16:10:29.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.152846+0000", 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2219999999999998e-05 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.152898+0000", 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028799979999999999 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:10:04.155778+0000", 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36345487999999998 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.590 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.590 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:10:29.590 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:10:29.590 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:10:29.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:29.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:29.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:29.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:29.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:29.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:29.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:29.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:29.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:29.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 154458 154442 119.155 43.375 0.0096497 0.00839047 2024-02-12T16:10:29.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:30.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:30.131 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:30.158 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:30.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:30.288 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:30.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:10:30.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:30.434 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:30.539 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:30.564 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:30.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:30.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:30.702 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:30.737 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:30 smithi123 bash[19119]: cluster 2024-02-12T16:10:29.644100+0000 mgr.y (mgr.14146) 753 : cluster [DBG] pgmap v941: 17 pgs: 1 active+recovering, 16 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s; 7549/303842 objects misplaced (2.485%); 653 KiB/s, 10 objects/s recovering 2024-02-12T16:10:30.737 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:30 smithi123 bash[19119]: cluster 2024-02-12T16:10:30.315224+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e286: 12 total, 12 up, 11 in 2024-02-12T16:10:30.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:30.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:30.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:30.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 155538 155522 118.525 67.5 0.0229034 0.00843522 2024-02-12T16:10:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:30.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:30 smithi169 bash[19604]: cluster 2024-02-12T16:10:29.644100+0000 mgr.y (mgr.14146) 753 : cluster [DBG] pgmap v941: 17 pgs: 1 active+recovering, 16 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s; 7549/303842 objects misplaced (2.485%); 653 KiB/s, 10 objects/s recovering 2024-02-12T16:10:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:30 smithi169 bash[19604]: cluster 2024-02-12T16:10:30.315224+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e286: 12 total, 12 up, 11 in 2024-02-12T16:10:31.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:31.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:31.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:30 smithi121 bash[18489]: cluster 2024-02-12T16:10:29.644100+0000 mgr.y (mgr.14146) 753 : cluster [DBG] pgmap v941: 17 pgs: 1 active+recovering, 16 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s; 7549/303842 objects misplaced (2.485%); 653 KiB/s, 10 objects/s recovering 2024-02-12T16:10:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:30 smithi121 bash[18489]: cluster 2024-02-12T16:10:30.315224+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e286: 12 total, 12 up, 11 in 2024-02-12T16:10:31.212 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:31.238 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:31.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:31.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:31.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:31.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:31.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:31.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:31.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:31.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:31.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:31.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:31.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:31.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:31.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:31.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:31.915 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:31.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:31.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 156603 156587 117.899 66.5625 0.0144429 0.0084802 2024-02-12T16:10:32.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:32.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:32.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:32.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:32.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:32.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:32.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:32.451 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:32.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:32.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:32.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:32.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:32.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:32 smithi123 bash[19119]: cluster 2024-02-12T16:10:31.644498+0000 mgr.y (mgr.14146) 754 : cluster [DBG] pgmap v943: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 48 MiB/s wr, 764 op/s; 7455/307362 objects misplaced (2.425%); 950 KiB/s, 14 objects/s recovering 2024-02-12T16:10:32.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:32.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 157639 157623 117.266 64.75 0.00678682 0.00852513 2024-02-12T16:10:32.963 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:32 smithi169 bash[19604]: cluster 2024-02-12T16:10:31.644498+0000 mgr.y (mgr.14146) 754 : cluster [DBG] pgmap v943: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 48 MiB/s wr, 764 op/s; 7455/307362 objects misplaced (2.425%); 950 KiB/s, 14 objects/s recovering 2024-02-12T16:10:32.982 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:32.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:33.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:33.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:33.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:32 smithi121 bash[18489]: cluster 2024-02-12T16:10:31.644498+0000 mgr.y (mgr.14146) 754 : cluster [DBG] pgmap v943: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 48 MiB/s wr, 764 op/s; 7455/307362 objects misplaced (2.425%); 950 KiB/s, 14 objects/s recovering 2024-02-12T16:10:33.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:33.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:33.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:33.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:33.385 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:33.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:33.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:33.519 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:33.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:33.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:33.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:33.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:33.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:33.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:33.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:33.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:33.925 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:33.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:33.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 158694 158678 116.662 65.9375 0.00633928 0.00856977 2024-02-12T16:10:34.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:34.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:34.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:34.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:34.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:34.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:34.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:34.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:34.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:34.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:34.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:34.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:34.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:34.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:34.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:34.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:34.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:34.743 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:34 smithi123 bash[19119]: cluster 2024-02-12T16:10:33.644959+0000 mgr.y (mgr.14146) 755 : cluster [DBG] pgmap v944: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 45 MiB/s wr, 717 op/s; 7455/309172 objects misplaced (2.411%); 752 KiB/s, 11 objects/s recovering 2024-02-12T16:10:34.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:34.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:34.867 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:34 smithi121 bash[18489]: cluster 2024-02-12T16:10:33.644959+0000 mgr.y (mgr.14146) 755 : cluster [DBG] pgmap v944: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 45 MiB/s wr, 717 op/s; 7455/309172 objects misplaced (2.411%); 752 KiB/s, 11 objects/s recovering 2024-02-12T16:10:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:34.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:34.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 159635 159619 115.99 58.8125 0.0647663 0.00861843 2024-02-12T16:10:34.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:35.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:34 smithi169 bash[19604]: cluster 2024-02-12T16:10:33.644959+0000 mgr.y (mgr.14146) 755 : cluster [DBG] pgmap v944: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 45 MiB/s wr, 717 op/s; 7455/309172 objects misplaced (2.411%); 752 KiB/s, 11 objects/s recovering 2024-02-12T16:10:35.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:35.133 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:35.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:35.264 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:35.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:35.397 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:35.501 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:35.533 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:35.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:35.664 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:35.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:35.798 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:35.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:35.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:35.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:35.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:35.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 160666 160650 115.397 64.4375 0.00664174 0.00866435 2024-02-12T16:10:36.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:36.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:36.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:36.189 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:36.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:36.317 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:36.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:36.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:36.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:36 smithi169 bash[19604]: cluster 2024-02-12T16:10:35.645352+0000 mgr.y (mgr.14146) 756 : cluster [DBG] pgmap v945: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 7354/317602 objects misplaced (2.315%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:36.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:36.588 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:36.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:36.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:36 smithi123 bash[19119]: cluster 2024-02-12T16:10:35.645352+0000 mgr.y (mgr.14146) 756 : cluster [DBG] pgmap v945: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 7354/317602 objects misplaced (2.315%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:36.648 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:10:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:36 smithi121 bash[18489]: cluster 2024-02-12T16:10:35.645352+0000 mgr.y (mgr.14146) 756 : cluster [DBG] pgmap v945: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 7354/317602 objects misplaced (2.315%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:36.675 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:10:36.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:36.705 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:10:36.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:36.733 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:10:36.762 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:10:36.783 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:10:36.812 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:10:36.815 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:36.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:36.846 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:10:36.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:36.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:36.876 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:10:36.918 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:10:36.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:36.958 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:10:36.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 161682 161666 114.807 63.5 0.00669128 0.0087081 2024-02-12T16:10:36.985 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:36.987 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:10:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:36.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.992 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.993 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.993 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:10:36.993 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:10:37.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:37.114 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:37.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:37.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:37.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:37.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:37.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:37 smithi121 bash[18489]: audit 2024-02-12T16:10:36.605149+0000 mon.a (mon.0) 1210 : audit [DBG] from='client.? v1:172.21.15.121:0/1648301938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:37.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:37 smithi121 bash[18489]: audit 2024-02-12T16:10:37.269008+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? v1:172.21.15.121:0/2872973711' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:10:37.344 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:10:37.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:37.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:37.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:37.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:37.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:37 smithi169 bash[19604]: audit 2024-02-12T16:10:36.605149+0000 mon.a (mon.0) 1210 : audit [DBG] from='client.? v1:172.21.15.121:0/1648301938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:37.485 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:37 smithi169 bash[19604]: audit 2024-02-12T16:10:37.269008+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? v1:172.21.15.121:0/2872973711' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:10:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:37.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:37.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:37 smithi123 bash[19119]: audit 2024-02-12T16:10:36.605149+0000 mon.a (mon.0) 1210 : audit [DBG] from='client.? v1:172.21.15.121:0/1648301938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:37 smithi123 bash[19119]: audit 2024-02-12T16:10:37.269008+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? v1:172.21.15.121:0/2872973711' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:10:37.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:37.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:37.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:37.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:37.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:37.874 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:37.910 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:37.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:37.959 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 162711 162695 114.24 64.3125 0.0220072 0.00875209 2024-02-12T16:10:38.018 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:38.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:38.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:38.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:38.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:38.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:38.428 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:38 smithi123 bash[19119]: cluster 2024-02-12T16:10:37.325029+0000 mon.a (mon.0) 1212 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:38.428 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:38 smithi123 bash[19119]: audit 2024-02-12T16:10:37.328503+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.121:0/2872973711' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:10:38.428 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:38 smithi123 bash[19119]: cluster 2024-02-12T16:10:37.328545+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e287: 12 total, 12 up, 11 in 2024-02-12T16:10:38.428 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:38 smithi123 bash[19119]: cluster 2024-02-12T16:10:37.645705+0000 mgr.y (mgr.14146) 757 : cluster [DBG] pgmap v947: 17 pgs: 1 active+recovering, 16 active+clean; 9.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 61 MiB/s wr, 979 op/s; 7354/319520 objects misplaced (2.302%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:38.446 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:38.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:38.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:38 smithi121 bash[18489]: cluster 2024-02-12T16:10:37.325029+0000 mon.a (mon.0) 1212 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:38 smithi121 bash[18489]: audit 2024-02-12T16:10:37.328503+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.121:0/2872973711' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:10:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:38 smithi121 bash[18489]: cluster 2024-02-12T16:10:37.328545+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e287: 12 total, 12 up, 11 in 2024-02-12T16:10:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:38 smithi121 bash[18489]: cluster 2024-02-12T16:10:37.645705+0000 mgr.y (mgr.14146) 757 : cluster [DBG] pgmap v947: 17 pgs: 1 active+recovering, 16 active+clean; 9.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 61 MiB/s wr, 979 op/s; 7354/319520 objects misplaced (2.302%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:38.684 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:38.716 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:38.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:38 smithi169 bash[19604]: cluster 2024-02-12T16:10:37.325029+0000 mon.a (mon.0) 1212 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:38.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:38 smithi169 bash[19604]: audit 2024-02-12T16:10:37.328503+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.121:0/2872973711' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:10:38.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:38 smithi169 bash[19604]: cluster 2024-02-12T16:10:37.328545+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e287: 12 total, 12 up, 11 in 2024-02-12T16:10:38.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:38 smithi169 bash[19604]: cluster 2024-02-12T16:10:37.645705+0000 mgr.y (mgr.14146) 757 : cluster [DBG] pgmap v947: 17 pgs: 1 active+recovering, 16 active+clean; 9.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 61 MiB/s wr, 979 op/s; 7354/319520 objects misplaced (2.302%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:38.820 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:38.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:38.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:38.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 90 12 163781 163769 113.716 67.125 0.0289222 0.00879162 2024-02-12T16:10:38.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:38.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:39.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:39.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:39.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:39.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:39.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:39.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:39.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:39.385 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:39.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:39.486 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-12T16:10:39.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:39.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:39.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:39.621 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:39.654 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:39.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:39.693 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:39 smithi123 bash[19119]: audit 2024-02-12T16:10:39.420917+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.121:0/2904380102' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:10:39.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:39.790 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:39.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:39.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:39 smithi169 bash[19604]: audit 2024-02-12T16:10:39.420917+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.121:0/2904380102' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:10:39.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:39 smithi121 bash[18489]: audit 2024-02-12T16:10:39.420917+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.121:0/2904380102' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:10:39.925 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:39.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:39.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.028000 2024-02-12T16:10:39.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 163781 2024-02-12T16:10:39.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:10:39.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:10:39.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 113.701 2024-02-12T16:10:39.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 65.0157 2024-02-12T16:10:39.960 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 385.438 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 0 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 1819 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 1040 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 6167 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 0 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.0087936 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.0178903 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 1.3461 2024-02-12T16:10:39.961 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.000952532 2024-02-12T16:10:39.971 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:10:40.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:40.053 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:40.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:40.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:40.184 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:40.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:40.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:40.317 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:40.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:40.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:40.447 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:40.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:40.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:40.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:40.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:40.731 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:40.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:40 smithi169 bash[19604]: audit 2024-02-12T16:10:39.476714+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? v1:172.21.15.121:0/2904380102' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:10:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:40 smithi169 bash[19604]: cluster 2024-02-12T16:10:39.476795+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e288: 12 total, 12 up, 11 in 2024-02-12T16:10:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:40 smithi169 bash[19604]: cluster 2024-02-12T16:10:39.646333+0000 mgr.y (mgr.14146) 758 : cluster [DBG] pgmap v949: 17 pgs: 1 active+recovering, 16 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 7354/323612 objects misplaced (2.272%); 808 KiB/s, 12 objects/s recovering 2024-02-12T16:10:40.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:40.843 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:40 smithi121 bash[18489]: audit 2024-02-12T16:10:39.476714+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? v1:172.21.15.121:0/2904380102' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:10:40.844 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:40 smithi121 bash[18489]: cluster 2024-02-12T16:10:39.476795+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e288: 12 total, 12 up, 11 in 2024-02-12T16:10:40.844 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:40 smithi121 bash[18489]: cluster 2024-02-12T16:10:39.646333+0000 mgr.y (mgr.14146) 758 : cluster [DBG] pgmap v949: 17 pgs: 1 active+recovering, 16 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 7354/323612 objects misplaced (2.272%); 808 KiB/s, 12 objects/s recovering 2024-02-12T16:10:40.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:40.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:10:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:40.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:40 smithi123 bash[19119]: audit 2024-02-12T16:10:39.476714+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? v1:172.21.15.121:0/2904380102' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:10:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:40 smithi123 bash[19119]: cluster 2024-02-12T16:10:39.476795+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e288: 12 total, 12 up, 11 in 2024-02-12T16:10:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:40 smithi123 bash[19119]: cluster 2024-02-12T16:10:39.646333+0000 mgr.y (mgr.14146) 758 : cluster [DBG] pgmap v949: 17 pgs: 1 active+recovering, 16 active+clean; 9.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 7354/323612 objects misplaced (2.272%); 808 KiB/s, 12 objects/s recovering 2024-02-12T16:10:40.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:40.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:10:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:41.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:41.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:41.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:41.306 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:41.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:41.431 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:41.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:41.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:41.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:41.685 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:41.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:41.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:41.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:41.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:41.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:41.954 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:10:41.954 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:10:42.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:42.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:42.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:42.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:42.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:42.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:42.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:42.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:42.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:42.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:42.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:42.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:42.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:42 smithi169 bash[19604]: cluster 2024-02-12T16:10:41.018463+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:42.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:42.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:42.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:42 smithi123 bash[19119]: cluster 2024-02-12T16:10:41.018463+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:42 smithi123 bash[19119]: cluster 2024-02-12T16:10:41.018526+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:42 smithi123 bash[19119]: cluster 2024-02-12T16:10:41.493668+0000 mon.a (mon.0) 1218 : cluster [DBG] osdmap e289: 12 total, 12 up, 11 in 2024-02-12T16:10:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:42 smithi123 bash[19119]: cluster 2024-02-12T16:10:41.646738+0000 mgr.y (mgr.14146) 759 : cluster [DBG] pgmap v951: 17 pgs: 1 active+recovering, 16 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 47 MiB/s wr, 758 op/s; 7235/326448 objects misplaced (2.216%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:10:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:42.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:42.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:42.767 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:42.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:42 smithi169 bash[19604]: cluster 2024-02-12T16:10:41.018526+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:42.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:42 smithi169 bash[19604]: cluster 2024-02-12T16:10:41.493668+0000 mon.a (mon.0) 1218 : cluster [DBG] osdmap e289: 12 total, 12 up, 11 in 2024-02-12T16:10:42.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:42 smithi169 bash[19604]: cluster 2024-02-12T16:10:41.646738+0000 mgr.y (mgr.14146) 759 : cluster [DBG] pgmap v951: 17 pgs: 1 active+recovering, 16 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 47 MiB/s wr, 758 op/s; 7235/326448 objects misplaced (2.216%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:10:42.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:42.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:42.820 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:42 smithi121 bash[18489]: cluster 2024-02-12T16:10:41.018463+0000 osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:42.820 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:42 smithi121 bash[18489]: cluster 2024-02-12T16:10:41.018526+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:42.820 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:42 smithi121 bash[18489]: cluster 2024-02-12T16:10:41.493668+0000 mon.a (mon.0) 1218 : cluster [DBG] osdmap e289: 12 total, 12 up, 11 in 2024-02-12T16:10:42.820 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:42 smithi121 bash[18489]: cluster 2024-02-12T16:10:41.646738+0000 mgr.y (mgr.14146) 759 : cluster [DBG] pgmap v951: 17 pgs: 1 active+recovering, 16 active+clean; 10 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 47 MiB/s wr, 758 op/s; 7235/326448 objects misplaced (2.216%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:10:42.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:10:42.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:42.898 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:42.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:43.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:43.033 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:43.139 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:43.163 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:43.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:43.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:43.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:43.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:43.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:43.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:43.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:43.685 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:43.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:43.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:43.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:43.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:43.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:44.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:44.079 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:44.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:44.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:44.232 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:44.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:44.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:44.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:44.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:44.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:44.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:44.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:44.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:44.634 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:44.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:44.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:44.772 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:44.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:44.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:44.905 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:44.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:44 smithi121 bash[18489]: cluster 2024-02-12T16:10:43.647142+0000 mgr.y (mgr.14146) 760 : cluster [DBG] pgmap v952: 17 pgs: 1 active+recovering, 16 active+clean; 9.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 162 B/s rd, 40 MiB/s wr, 774 op/s; 7235/325814 objects misplaced (2.221%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:10:45.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:45.017 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:44 smithi169 bash[19604]: cluster 2024-02-12T16:10:43.647142+0000 mgr.y (mgr.14146) 760 : cluster [DBG] pgmap v952: 17 pgs: 1 active+recovering, 16 active+clean; 9.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 162 B/s rd, 40 MiB/s wr, 774 op/s; 7235/325814 objects misplaced (2.221%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:10:45.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:45.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:45.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:45.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:44 smithi123 bash[19119]: cluster 2024-02-12T16:10:43.647142+0000 mgr.y (mgr.14146) 760 : cluster [DBG] pgmap v952: 17 pgs: 1 active+recovering, 16 active+clean; 9.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 162 B/s rd, 40 MiB/s wr, 774 op/s; 7235/325814 objects misplaced (2.221%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:10:45.176 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:45.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:45.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:45.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:45.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:45.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:45.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:45.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:45.582 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:45.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:45.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:45.719 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:45.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:45.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:45.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:45.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:45.961 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:10:45.961 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:10:45.961 INFO:tasks.thrashosds.thrasher:Adding osd 8 2024-02-12T16:10:45.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd in 8 2024-02-12T16:10:45.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:46.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:46.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:46.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:46.269 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:46.332 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:46.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:46.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:46 smithi123 bash[19119]: cluster 2024-02-12T16:10:45.647586+0000 mgr.y (mgr.14146) 761 : cluster [DBG] pgmap v953: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 31 MiB/s wr, 1.06k op/s; 7152/318704 objects misplaced (2.244%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:10:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:46.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:46.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:46 smithi169 bash[19604]: cluster 2024-02-12T16:10:45.647586+0000 mgr.y (mgr.14146) 761 : cluster [DBG] pgmap v953: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 31 MiB/s wr, 1.06k op/s; 7152/318704 objects misplaced (2.244%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:10:46.644 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:46 smithi121 bash[18489]: cluster 2024-02-12T16:10:45.647586+0000 mgr.y (mgr.14146) 761 : cluster [DBG] pgmap v953: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 31 MiB/s wr, 1.06k op/s; 7152/318704 objects misplaced (2.244%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:10:46.686 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:46.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:46.821 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:46.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:46.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:47.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:47.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:47.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:47.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:47.214 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:47.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:47.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:47.342 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:47.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:47.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:47.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:47.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:47.480 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:47.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:47.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:47.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:47.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:47.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:47.742 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:47.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:47.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:47.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:47.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:47.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:47.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:48.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:48.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:48.242 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:48.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:48.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:48.546 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:48.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:48.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:48.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:48.825 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:48 smithi121 bash[18489]: cluster 2024-02-12T16:10:47.648017+0000 mgr.y (mgr.14146) 762 : cluster [DBG] pgmap v954: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 125 B/s rd, 22 MiB/s wr, 1.02k op/s; 7152/316646 objects misplaced (2.259%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:48 smithi123 bash[19119]: cluster 2024-02-12T16:10:47.648017+0000 mgr.y (mgr.14146) 762 : cluster [DBG] pgmap v954: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 125 B/s rd, 22 MiB/s wr, 1.02k op/s; 7152/316646 objects misplaced (2.259%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:48.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:48.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:48.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:49.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:48 smithi169 bash[19604]: cluster 2024-02-12T16:10:47.648017+0000 mgr.y (mgr.14146) 762 : cluster [DBG] pgmap v954: 17 pgs: 1 active+recovering, 16 active+clean; 9.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 125 B/s rd, 22 MiB/s wr, 1.02k op/s; 7152/316646 objects misplaced (2.259%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:10:49.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:49.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:49.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:49.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:49.238 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:49.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:49.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:49.370 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:49.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:49.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:49.516 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:49.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:49.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:49.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:49.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:49.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:49.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:49.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:49.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:49.913 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:49.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:50.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:50.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:50.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:50.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:50.175 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:50.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:50.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:50.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:50.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:50.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:50.517 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:50.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:50.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:50.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:50.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:50.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:50.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:50 smithi169 bash[19604]: cluster 2024-02-12T16:10:49.648424+0000 mgr.y (mgr.14146) 763 : cluster [DBG] pgmap v955: 17 pgs: 1 active+recovering, 16 active+clean; 9.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 102 B/s rd, 12 MiB/s wr, 985 op/s; 7152/311810 objects misplaced (2.294%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:50.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:50.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:50.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:50.914 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:50.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:51.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:51.026 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:50 smithi121 bash[18489]: cluster 2024-02-12T16:10:49.648424+0000 mgr.y (mgr.14146) 763 : cluster [DBG] pgmap v955: 17 pgs: 1 active+recovering, 16 active+clean; 9.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 102 B/s rd, 12 MiB/s wr, 985 op/s; 7152/311810 objects misplaced (2.294%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:51.041 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:51.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:50 smithi123 bash[19119]: cluster 2024-02-12T16:10:49.648424+0000 mgr.y (mgr.14146) 763 : cluster [DBG] pgmap v955: 17 pgs: 1 active+recovering, 16 active+clean; 9.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 102 B/s rd, 12 MiB/s wr, 985 op/s; 7152/311810 objects misplaced (2.294%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:10:51.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:51.170 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:51.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:51.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:51.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:51.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:51.432 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:51.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:51.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:51.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:51.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:51.754 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:10:51.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:51.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:51.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:51.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:51.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:51.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:52.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:52.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:52.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:52.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:52.233 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:52.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:52.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:52.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:52.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:52.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:52.489 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:52.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:52.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:52.635 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:52.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:52.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:52.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:52.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:52.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:52.878 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:52 smithi123 bash[19119]: cluster 2024-02-12T16:10:51.479665+0000 osd.4 (osd.4) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:52.878 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:52 smithi123 bash[19119]: cluster 2024-02-12T16:10:51.479713+0000 osd.4 (osd.4) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:52.878 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:52 smithi123 bash[19119]: cluster 2024-02-12T16:10:51.652581+0000 mgr.y (mgr.14146) 764 : cluster [DBG] pgmap v956: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 100 B/s rd, 3.0 MiB/s wr, 1.02k op/s; 7054/307664 objects misplaced (2.293%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:10:52.878 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:52 smithi123 bash[19119]: cluster 2024-02-12T16:10:51.715753+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e290: 12 total, 12 up, 11 in 2024-02-12T16:10:52.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:52.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:52.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:52.934 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:52 smithi121 bash[18489]: cluster 2024-02-12T16:10:51.479665+0000 osd.4 (osd.4) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:52.935 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:52 smithi121 bash[18489]: cluster 2024-02-12T16:10:51.479713+0000 osd.4 (osd.4) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:52.935 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:52 smithi121 bash[18489]: cluster 2024-02-12T16:10:51.652581+0000 mgr.y (mgr.14146) 764 : cluster [DBG] pgmap v956: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 100 B/s rd, 3.0 MiB/s wr, 1.02k op/s; 7054/307664 objects misplaced (2.293%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:10:52.935 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:52 smithi121 bash[18489]: cluster 2024-02-12T16:10:51.715753+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e290: 12 total, 12 up, 11 in 2024-02-12T16:10:53.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:53.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:52 smithi169 bash[19604]: cluster 2024-02-12T16:10:51.479665+0000 osd.4 (osd.4) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:52 smithi169 bash[19604]: cluster 2024-02-12T16:10:51.479713+0000 osd.4 (osd.4) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:52 smithi169 bash[19604]: cluster 2024-02-12T16:10:51.652581+0000 mgr.y (mgr.14146) 764 : cluster [DBG] pgmap v956: 17 pgs: 1 active+recovering, 16 active+clean; 9.4 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 100 B/s rd, 3.0 MiB/s wr, 1.02k op/s; 7054/307664 objects misplaced (2.293%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:10:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:52 smithi169 bash[19604]: cluster 2024-02-12T16:10:51.715753+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e290: 12 total, 12 up, 11 in 2024-02-12T16:10:53.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:53.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:53.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:53.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:53.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:53.428 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:53.459 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:10:53.485 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:10:53.509 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:10:53.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:53.534 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:10:53.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:53.566 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:10:53.591 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:10:53.618 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:10:53.641 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:10:53.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:53.664 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:10:53.686 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:53.690 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:10:53.719 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:10:53.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:10:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:10:53.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:53.796 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:53 smithi121 bash[18489]: cluster 2024-02-12T16:10:52.639928+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:53.796 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:53 smithi121 bash[18489]: cluster 2024-02-12T16:10:52.639977+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:53.796 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:53 smithi121 bash[18489]: cluster 2024-02-12T16:10:52.722255+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e291: 12 total, 12 up, 11 in 2024-02-12T16:10:53.796 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:53 smithi121 bash[18489]: audit 2024-02-12T16:10:53.431149+0000 mon.a (mon.0) 1221 : audit [DBG] from='client.? v1:172.21.15.121:0/1727597546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:53.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:53.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:53.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:53.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:53 smithi169 bash[19604]: cluster 2024-02-12T16:10:52.639928+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:53 smithi169 bash[19604]: cluster 2024-02-12T16:10:52.639977+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:53 smithi169 bash[19604]: cluster 2024-02-12T16:10:52.722255+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e291: 12 total, 12 up, 11 in 2024-02-12T16:10:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:53 smithi169 bash[19604]: audit 2024-02-12T16:10:53.431149+0000 mon.a (mon.0) 1221 : audit [DBG] from='client.? v1:172.21.15.121:0/1727597546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:54.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:54.090 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:54.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:54.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:53 smithi123 bash[19119]: cluster 2024-02-12T16:10:52.639928+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:10:54.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:53 smithi123 bash[19119]: cluster 2024-02-12T16:10:52.639977+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:10:54.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:53 smithi123 bash[19119]: cluster 2024-02-12T16:10:52.722255+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e291: 12 total, 12 up, 11 in 2024-02-12T16:10:54.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:53 smithi123 bash[19119]: audit 2024-02-12T16:10:53.431149+0000 mon.a (mon.0) 1221 : audit [DBG] from='client.? v1:172.21.15.121:0/1727597546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:10:54.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:54.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:54.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:54.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:54.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:54.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:54.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:54.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:54.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:54.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:54.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:54.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:54.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:54.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:54 smithi121 bash[18489]: cluster 2024-02-12T16:10:53.652943+0000 mgr.y (mgr.14146) 765 : cluster [DBG] pgmap v959: 17 pgs: 1 active+recovering, 16 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 7054/304184 objects misplaced (2.319%); 783 KiB/s, 12 objects/s recovering 2024-02-12T16:10:54.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:54.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:54.809 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:10:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:10:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:10:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:10:54.810 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:10:54.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:54.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:54.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:55.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:55.007 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:54 smithi169 bash[19604]: cluster 2024-02-12T16:10:53.652943+0000 mgr.y (mgr.14146) 765 : cluster [DBG] pgmap v959: 17 pgs: 1 active+recovering, 16 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 7054/304184 objects misplaced (2.319%); 783 KiB/s, 12 objects/s recovering 2024-02-12T16:10:55.024 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:55.126 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:55.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:54 smithi123 bash[19119]: cluster 2024-02-12T16:10:53.652943+0000 mgr.y (mgr.14146) 765 : cluster [DBG] pgmap v959: 17 pgs: 1 active+recovering, 16 active+clean; 9.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 7054/304184 objects misplaced (2.319%); 783 KiB/s, 12 objects/s recovering 2024-02-12T16:10:55.159 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:10:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:10:55.264 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:55.288 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:55.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:55.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:55.549 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:55.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:55.684 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:55.752 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:10:55.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:55.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:55 smithi123 bash[19119]: audit 2024-02-12T16:10:55.045307+0000 mon.a (mon.0) 1222 : audit [INF] from='client.25121 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:55 smithi123 bash[19119]: audit 2024-02-12T16:10:55.048401+0000 mon.b (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.121:0/1977219395' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:55.923 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:55.948 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:55.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:55 smithi169 bash[19604]: audit 2024-02-12T16:10:55.045307+0000 mon.a (mon.0) 1222 : audit [INF] from='client.25121 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:55 smithi169 bash[19604]: audit 2024-02-12T16:10:55.048401+0000 mon.b (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.121:0/1977219395' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:56.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:56.077 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:56.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:56.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:55 smithi121 bash[18489]: audit 2024-02-12T16:10:55.045307+0000 mon.a (mon.0) 1222 : audit [INF] from='client.25121 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:56.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:55 smithi121 bash[18489]: audit 2024-02-12T16:10:55.048401+0000 mon.b (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.121:0/1977219395' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:10:56.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:10:56.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:56.204 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:56.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:56.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:56.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:56.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:56.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:56.467 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:56.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:56.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:56.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:56.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:56.709 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:56.746 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:56.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:56.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:56.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:10:56.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:56.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:56.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:56.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:56.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:56 smithi123 bash[19119]: cluster 2024-02-12T16:10:55.653314+0000 mgr.y (mgr.14146) 766 : cluster [DBG] pgmap v960: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s; 6858/296996 objects misplaced (2.309%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:10:56.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:56 smithi123 bash[19119]: cluster 2024-02-12T16:10:55.730916+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:56.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:56 smithi123 bash[19119]: audit 2024-02-12T16:10:55.741683+0000 mon.a (mon.0) 1224 : audit [INF] from='client.25121 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:10:56.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:56 smithi123 bash[19119]: cluster 2024-02-12T16:10:55.741745+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e292: 12 total, 12 up, 11 in 2024-02-12T16:10:57.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:56 smithi121 bash[18489]: cluster 2024-02-12T16:10:55.653314+0000 mgr.y (mgr.14146) 766 : cluster [DBG] pgmap v960: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s; 6858/296996 objects misplaced (2.309%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:10:57.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:56 smithi121 bash[18489]: cluster 2024-02-12T16:10:55.730916+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:57.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:56 smithi121 bash[18489]: audit 2024-02-12T16:10:55.741683+0000 mon.a (mon.0) 1224 : audit [INF] from='client.25121 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:10:57.004 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:56 smithi121 bash[18489]: cluster 2024-02-12T16:10:55.741745+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e292: 12 total, 12 up, 11 in 2024-02-12T16:10:57.014 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:56 smithi169 bash[19604]: cluster 2024-02-12T16:10:55.653314+0000 mgr.y (mgr.14146) 766 : cluster [DBG] pgmap v960: 17 pgs: 1 active+recovering, 16 active+clean; 9.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s; 6858/296996 objects misplaced (2.309%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:10:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:56 smithi169 bash[19604]: cluster 2024-02-12T16:10:55.730916+0000 mon.a (mon.0) 1223 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:10:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:56 smithi169 bash[19604]: audit 2024-02-12T16:10:55.741683+0000 mon.a (mon.0) 1224 : audit [INF] from='client.25121 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:10:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:56 smithi169 bash[19604]: cluster 2024-02-12T16:10:55.741745+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e292: 12 total, 12 up, 11 in 2024-02-12T16:10:57.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:57.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:57.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:57.269 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:57.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:57.397 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:57.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:57.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:57.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:57.660 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:57.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:57.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:57.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:57.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:10:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:57.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:10:58.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:58.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:58.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:58.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:10:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:10:58.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:58.331 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:10:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:10:58.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:58.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:58.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:58.601 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:10:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:10:58.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: debug 2024-02-12T16:10:58.241+0000 7fbba85de700 -1 mon.a@0(leader).osd e292 definitely_dead 0 2024-02-12T16:10:58.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:58.732 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:marked in osd.8. 2024-02-12T16:10:58.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:58 smithi123 bash[19119]: cluster 2024-02-12T16:10:57.653667+0000 mgr.y (mgr.14146) 767 : cluster [DBG] pgmap v962: 17 pgs: 1 active+recovering, 16 active+clean; 9.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 6858/295272 objects misplaced (2.323%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T16:10:58.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:58 smithi123 bash[19119]: audit 2024-02-12T16:10:58.037724+0000 mon.a (mon.0) 1226 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:10:58.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:58 smithi123 bash[19119]: audit 2024-02-12T16:10:58.244036+0000 mon.a (mon.0) 1227 : audit [DBG] from='client.? v1:172.21.15.121:0/2980802118' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-02-12T16:10:58.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:58 smithi123 bash[19119]: audit 2024-02-12T16:10:58.378207+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:10:58.751 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:58 smithi123 bash[19119]: audit 2024-02-12T16:10:58.720326+0000 mon.a (mon.0) 1229 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:10:58.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:58.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:10:58.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:58.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:58.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:59.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:10:59.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: cluster 2024-02-12T16:10:57.653667+0000 mgr.y (mgr.14146) 767 : cluster [DBG] pgmap v962: 17 pgs: 1 active+recovering, 16 active+clean; 9.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 6858/295272 objects misplaced (2.323%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T16:10:59.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: audit 2024-02-12T16:10:58.037724+0000 mon.a (mon.0) 1226 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:10:59.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: audit 2024-02-12T16:10:58.244036+0000 mon.a (mon.0) 1227 : audit [DBG] from='client.? v1:172.21.15.121:0/2980802118' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-02-12T16:10:59.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: audit 2024-02-12T16:10:58.378207+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:10:59.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: audit 2024-02-12T16:10:58.720326+0000 mon.a (mon.0) 1229 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:10:59.013 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: audit 2024-02-12T16:10:58.721240+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:10:59.014 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:58 smithi121 bash[18489]: audit 2024-02-12T16:10:58.728709+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:10:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:58 smithi169 bash[19604]: cluster 2024-02-12T16:10:57.653667+0000 mgr.y (mgr.14146) 767 : cluster [DBG] pgmap v962: 17 pgs: 1 active+recovering, 16 active+clean; 9.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 6858/295272 objects misplaced (2.323%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T16:10:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:58 smithi169 bash[19604]: audit 2024-02-12T16:10:58.037724+0000 mon.a (mon.0) 1226 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:10:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:58 smithi169 bash[19604]: audit 2024-02-12T16:10:58.244036+0000 mon.a (mon.0) 1227 : audit [DBG] from='client.? v1:172.21.15.121:0/2980802118' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-02-12T16:10:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:58 smithi169 bash[19604]: audit 2024-02-12T16:10:58.378207+0000 mon.a (mon.0) 1228 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:10:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:58 smithi169 bash[19604]: audit 2024-02-12T16:10:58.720326+0000 mon.a (mon.0) 1229 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:10:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:58 smithi169 bash[19604]: audit 2024-02-12T16:10:58.721240+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:10:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:58 smithi169 bash[19604]: audit 2024-02-12T16:10:58.728709+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:10:59.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:59.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:10:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:59.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:10:59.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:58 smithi123 bash[19119]: audit 2024-02-12T16:10:58.721240+0000 mon.a (mon.0) 1230 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:10:59.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:58 smithi123 bash[19119]: audit 2024-02-12T16:10:58.728709+0000 mon.a (mon.0) 1231 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:10:59.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:59.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:10:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:59.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:10:59.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:59.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:10:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:59.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:10:59.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:59.524 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:10:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:59.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:10:59.607 INFO:tasks.thrashosds.thrasher:Added osd 8 2024-02-12T16:10:59.628 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:59.666 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:10:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:59.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:10:59.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:59.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:10:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:59.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:10:59.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:59.944 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:10:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:10:59.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:59 smithi169 bash[19604]: audit 2024-02-12T16:10:58.742521+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? v1:172.21.15.121:0/2980802118' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-02-12T16:11:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:10:59 smithi169 bash[19604]: cluster 2024-02-12T16:10:58.742594+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e293: 12 total, 12 up, 12 in 2024-02-12T16:11:00.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:00.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:59 smithi121 bash[18489]: audit 2024-02-12T16:10:58.742521+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? v1:172.21.15.121:0/2980802118' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-02-12T16:11:00.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:10:59 smithi121 bash[18489]: cluster 2024-02-12T16:10:58.742594+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e293: 12 total, 12 up, 12 in 2024-02-12T16:11:00.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:00.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:00.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:59 smithi123 bash[19119]: audit 2024-02-12T16:10:58.742521+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? v1:172.21.15.121:0/2980802118' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-02-12T16:11:00.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:10:59 smithi123 bash[19119]: cluster 2024-02-12T16:10:58.742594+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e293: 12 total, 12 up, 12 in 2024-02-12T16:11:00.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:00.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:00.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:00.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:00.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:00.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:00.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:00.500 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:00.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:00.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:00.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:00.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:00.775 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:00.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:00.879 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:00 smithi121 bash[18489]: cluster 2024-02-12T16:10:59.654340+0000 mgr.y (mgr.14146) 768 : cluster [DBG] pgmap v964: 17 pgs: 1 peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 533 op/s 2024-02-12T16:11:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:00 smithi121 bash[18489]: cluster 2024-02-12T16:10:59.752393+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e294: 12 total, 12 up, 12 in 2024-02-12T16:11:00.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:00.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:01.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:01.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:00 smithi169 bash[19604]: cluster 2024-02-12T16:10:59.654340+0000 mgr.y (mgr.14146) 768 : cluster [DBG] pgmap v964: 17 pgs: 1 peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 533 op/s 2024-02-12T16:11:01.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:00 smithi169 bash[19604]: cluster 2024-02-12T16:10:59.752393+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e294: 12 total, 12 up, 12 in 2024-02-12T16:11:01.049 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:01.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:01.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:00 smithi123 bash[19119]: cluster 2024-02-12T16:10:59.654340+0000 mgr.y (mgr.14146) 768 : cluster [DBG] pgmap v964: 17 pgs: 1 peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 533 op/s 2024-02-12T16:11:01.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:00 smithi123 bash[19119]: cluster 2024-02-12T16:10:59.752393+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e294: 12 total, 12 up, 12 in 2024-02-12T16:11:01.154 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:01.195 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:01.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:01.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:01.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:01.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:01.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:01.463 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:01.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:01.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:01.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:01.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:01.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:01.743 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:01.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:01.847 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:01.854 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:01 smithi123 bash[19119]: cluster 2024-02-12T16:11:00.771226+0000 mon.a (mon.0) 1235 : cluster [DBG] osdmap e295: 12 total, 12 up, 12 in 2024-02-12T16:11:01.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:01.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:01.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:02.001 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:02.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:01 smithi169 bash[19604]: cluster 2024-02-12T16:11:00.771226+0000 mon.a (mon.0) 1235 : cluster [DBG] osdmap e295: 12 total, 12 up, 12 in 2024-02-12T16:11:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:02.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:02.146 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:02.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:01 smithi121 bash[18489]: cluster 2024-02-12T16:11:00.771226+0000 mon.a (mon.0) 1235 : cluster [DBG] osdmap e295: 12 total, 12 up, 12 in 2024-02-12T16:11:02.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:02.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:02.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:02.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:02.519 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:02.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:02.650 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:02.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:02.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:02.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:02.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:02.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:02.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:03.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:02 smithi169 bash[19604]: cluster 2024-02-12T16:11:01.654819+0000 mgr.y (mgr.14146) 769 : cluster [DBG] pgmap v967: 17 pgs: 1 remapped+peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-02-12T16:11:03.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:03.079 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:02 smithi123 bash[19119]: cluster 2024-02-12T16:11:01.654819+0000 mgr.y (mgr.14146) 769 : cluster [DBG] pgmap v967: 17 pgs: 1 remapped+peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-02-12T16:11:03.102 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:03.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:03.116 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:02 smithi121 bash[18489]: cluster 2024-02-12T16:11:01.654819+0000 mgr.y (mgr.14146) 769 : cluster [DBG] pgmap v967: 17 pgs: 1 remapped+peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-02-12T16:11:03.204 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:03.232 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:03.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:03.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:03.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:03.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:03.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:03.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:03.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:03.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:03.635 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:03.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:03.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:03.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:03.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:03.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:03 smithi123 bash[19119]: cluster 2024-02-12T16:11:02.275177+0000 osd.8 (osd.8) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:03.794 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:03 smithi123 bash[19119]: cluster 2024-02-12T16:11:02.275236+0000 osd.8 (osd.8) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:03.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:03 smithi169 bash[19604]: cluster 2024-02-12T16:11:02.275177+0000 osd.8 (osd.8) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:03.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:03 smithi169 bash[19604]: cluster 2024-02-12T16:11:02.275236+0000 osd.8 (osd.8) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:03 smithi169 bash[19604]: cluster 2024-02-12T16:11:02.777924+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e296: 12 total, 12 up, 12 in 2024-02-12T16:11:03.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:03.894 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:03.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:03.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:04.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:04.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:04.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:03 smithi121 bash[18489]: cluster 2024-02-12T16:11:02.275177+0000 osd.8 (osd.8) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:04.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:03 smithi121 bash[18489]: cluster 2024-02-12T16:11:02.275236+0000 osd.8 (osd.8) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:04.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:03 smithi121 bash[18489]: cluster 2024-02-12T16:11:02.777924+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e296: 12 total, 12 up, 12 in 2024-02-12T16:11:04.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:03 smithi123 bash[19119]: cluster 2024-02-12T16:11:02.777924+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e296: 12 total, 12 up, 12 in 2024-02-12T16:11:04.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:04.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:04.296 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:04.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:04.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:04.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:04.553 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:04.608 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:11:04.608 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:11:04.608 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-02-12T16:11:04.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-02-12T16:11:04.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:04.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:04.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:04.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:04.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:04.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:05.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:05.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:05.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:05.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:05.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:05 smithi169 bash[19604]: cluster 2024-02-12T16:11:03.655275+0000 mgr.y (mgr.14146) 770 : cluster [DBG] pgmap v969: 17 pgs: 1 remapped+peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 2.9 MiB/s, 47 objects/s recovering 2024-02-12T16:11:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:05 smithi169 bash[19604]: audit 2024-02-12T16:11:04.022693+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:11:05.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:05 smithi121 bash[18489]: cluster 2024-02-12T16:11:03.655275+0000 mgr.y (mgr.14146) 770 : cluster [DBG] pgmap v969: 17 pgs: 1 remapped+peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 2.9 MiB/s, 47 objects/s recovering 2024-02-12T16:11:05.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:05 smithi121 bash[18489]: audit 2024-02-12T16:11:04.022693+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:11:05.368 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:05.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:05 smithi123 bash[19119]: cluster 2024-02-12T16:11:03.655275+0000 mgr.y (mgr.14146) 770 : cluster [DBG] pgmap v969: 17 pgs: 1 remapped+peering, 16 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 2.9 MiB/s, 47 objects/s recovering 2024-02-12T16:11:05.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:05 smithi123 bash[19119]: audit 2024-02-12T16:11:04.022693+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:11:05.410 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:05.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:05.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:05.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:05.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:05.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:05.675 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:05.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:05.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:05.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:05.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:05.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:05.941 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:05.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:06.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:06.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:06.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:06.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:06.209 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:06.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:06.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:06.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:06.345 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:06.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:06.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:06.460 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:06 smithi169 bash[19604]: cluster 2024-02-12T16:11:05.655738+0000 mgr.y (mgr.14146) 771 : cluster [DBG] pgmap v970: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 1.40k op/s; 2243/281336 objects misplaced (0.797%); 15 MiB/s, 237 objects/s recovering 2024-02-12T16:11:06.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:06.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:06.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:06.589 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:06 smithi123 bash[19119]: cluster 2024-02-12T16:11:05.655738+0000 mgr.y (mgr.14146) 771 : cluster [DBG] pgmap v970: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 1.40k op/s; 2243/281336 objects misplaced (0.797%); 15 MiB/s, 237 objects/s recovering 2024-02-12T16:11:06.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:06.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:06 smithi121 bash[18489]: cluster 2024-02-12T16:11:05.655738+0000 mgr.y (mgr.14146) 771 : cluster [DBG] pgmap v970: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 1.40k op/s; 2243/281336 objects misplaced (0.797%); 15 MiB/s, 237 objects/s recovering 2024-02-12T16:11:06.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:06.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:06.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:06.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:06.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:06.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:06.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:06.996 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:07.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:07.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:07.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:07.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:07.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:07 smithi123 bash[19119]: cluster 2024-02-12T16:11:06.072522+0000 osd.11 (osd.11) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:07 smithi123 bash[19119]: cluster 2024-02-12T16:11:06.072602+0000 osd.11 (osd.11) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:07.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:07 smithi123 bash[19119]: cluster 2024-02-12T16:11:06.338811+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e297: 12 total, 12 up, 12 in 2024-02-12T16:11:07.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:07.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:07.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:07 smithi121 bash[18489]: cluster 2024-02-12T16:11:06.072522+0000 osd.11 (osd.11) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:07.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:07 smithi121 bash[18489]: cluster 2024-02-12T16:11:06.072602+0000 osd.11 (osd.11) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:07.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:07 smithi121 bash[18489]: cluster 2024-02-12T16:11:06.338811+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e297: 12 total, 12 up, 12 in 2024-02-12T16:11:07.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:07.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:07.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:07.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:07.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:07 smithi169 bash[19604]: cluster 2024-02-12T16:11:06.072522+0000 osd.11 (osd.11) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:07.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:07 smithi169 bash[19604]: cluster 2024-02-12T16:11:06.072602+0000 osd.11 (osd.11) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:07.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:07 smithi169 bash[19604]: cluster 2024-02-12T16:11:06.338811+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e297: 12 total, 12 up, 12 in 2024-02-12T16:11:07.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:07.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:07.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:07.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:08.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:08.063 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:08.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:08.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:08.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:08.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:08.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:08.463 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:08.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:08.597 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:08.599 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:08 smithi123 bash[19119]: cluster 2024-02-12T16:11:07.656166+0000 mgr.y (mgr.14146) 772 : cluster [DBG] pgmap v972: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 2243/279194 objects misplaced (0.803%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:11:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:08 smithi121 bash[18489]: cluster 2024-02-12T16:11:07.656166+0000 mgr.y (mgr.14146) 772 : cluster [DBG] pgmap v972: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 2243/279194 objects misplaced (0.803%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:11:08.701 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:08.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:08 smithi169 bash[19604]: cluster 2024-02-12T16:11:07.656166+0000 mgr.y (mgr.14146) 772 : cluster [DBG] pgmap v972: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 2243/279194 objects misplaced (0.803%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:11:08.836 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:08.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:08.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:08.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:08.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:09.044 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:11:09.073 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:11:09.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:09.101 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:11:09.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:09.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:09.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:11:09.163 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:11:09.194 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:11:09.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:09.228 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:11:09.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:09.260 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:11:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:09.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:09.295 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:11:09.330 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:11:09.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:09.365 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:11:09.390 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:09.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:09.394 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:11:09.394 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:11:09.400 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:11:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:09 smithi121 bash[18489]: audit 2024-02-12T16:11:09.011463+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.121:0/2336997377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:09.427 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:09.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:09.499 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:09 smithi123 bash[19119]: audit 2024-02-12T16:11:09.011463+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.121:0/2336997377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:09.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:09.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:09.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:09.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:09.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:09.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:09.766 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:09 smithi169 bash[19604]: audit 2024-02-12T16:11:09.011463+0000 mon.a (mon.0) 1239 : audit [DBG] from='client.? v1:172.21.15.121:0/2336997377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:09.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:09.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:09.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:09.931 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:09.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:10.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:10.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:10.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:10.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:10.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:10.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:10.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:10.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:10.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:10.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:10 smithi121 bash[18489]: cluster 2024-02-12T16:11:09.656554+0000 mgr.y (mgr.14146) 773 : cluster [DBG] pgmap v973: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 2243/274330 objects misplaced (0.818%); 1.1 MiB/s, 18 objects/s recovering 2024-02-12T16:11:10.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:10 smithi121 bash[18489]: audit 2024-02-12T16:11:09.681998+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? v1:172.21.15.121:0/1659056623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:11:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:11:10.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:10.452 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:10 smithi123 bash[19119]: cluster 2024-02-12T16:11:09.656554+0000 mgr.y (mgr.14146) 773 : cluster [DBG] pgmap v973: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 2243/274330 objects misplaced (0.818%); 1.1 MiB/s, 18 objects/s recovering 2024-02-12T16:11:10.452 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:10 smithi123 bash[19119]: audit 2024-02-12T16:11:09.681998+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? v1:172.21.15.121:0/1659056623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:11:10.469 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:10.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:10.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:10.598 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:10.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:10.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:10.710 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:10 smithi169 bash[19604]: cluster 2024-02-12T16:11:09.656554+0000 mgr.y (mgr.14146) 773 : cluster [DBG] pgmap v973: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 2243/274330 objects misplaced (0.818%); 1.1 MiB/s, 18 objects/s recovering 2024-02-12T16:11:10.710 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:10 smithi169 bash[19604]: audit 2024-02-12T16:11:09.681998+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? v1:172.21.15.121:0/1659056623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:11:10.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:10.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:10.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:10.868 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:10.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:10.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:10.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:11.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:11.132 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:11.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:11:11.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:11.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:11.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:11.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:11.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:11.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:11.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:11 smithi123 bash[19119]: cluster 2024-02-12T16:11:10.342322+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:11:11.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:11 smithi123 bash[19119]: audit 2024-02-12T16:11:10.353155+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? v1:172.21.15.121:0/1659056623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:11:11.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:11 smithi123 bash[19119]: cluster 2024-02-12T16:11:10.353207+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e298: 12 total, 12 up, 12 in 2024-02-12T16:11:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:11 smithi121 bash[18489]: cluster 2024-02-12T16:11:10.342322+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:11:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:11 smithi121 bash[18489]: audit 2024-02-12T16:11:10.353155+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? v1:172.21.15.121:0/1659056623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:11:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:11 smithi121 bash[18489]: cluster 2024-02-12T16:11:10.353207+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e298: 12 total, 12 up, 12 in 2024-02-12T16:11:11.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:11.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:11 smithi169 bash[19604]: cluster 2024-02-12T16:11:10.342322+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:11:11.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:11 smithi169 bash[19604]: audit 2024-02-12T16:11:10.353155+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? v1:172.21.15.121:0/1659056623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:11:11.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:11 smithi169 bash[19604]: cluster 2024-02-12T16:11:10.353207+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e298: 12 total, 12 up, 12 in 2024-02-12T16:11:11.708 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:11.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:11.852 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:11.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:11.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:11.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:11.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:12.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:12.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:12.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:12.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:12.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:12.261 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:12.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:12.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:12 smithi169 bash[19604]: cluster 2024-02-12T16:11:11.656960+0000 mgr.y (mgr.14146) 774 : cluster [DBG] pgmap v975: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 2053/270108 objects misplaced (0.760%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T16:11:12.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:12.391 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:12.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:12.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:12.528 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:12.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:12.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:12.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:12 smithi123 bash[19119]: cluster 2024-02-12T16:11:11.656960+0000 mgr.y (mgr.14146) 774 : cluster [DBG] pgmap v975: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 2053/270108 objects misplaced (0.760%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T16:11:12.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:12 smithi121 bash[18489]: cluster 2024-02-12T16:11:11.656960+0000 mgr.y (mgr.14146) 774 : cluster [DBG] pgmap v975: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 2053/270108 objects misplaced (0.760%); 2.6 MiB/s, 42 objects/s recovering 2024-02-12T16:11:12.660 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:12.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:12.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:12.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:12.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:12.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:12.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:12.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:13.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:13.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:13.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:13.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:13.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:13.316 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:13.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:13.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:13.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:13.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:13.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:13.714 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:13.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:11:13.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:11:13.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:13.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:13.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:13.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:13.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:14.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:14.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:14.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:14.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:14.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:14.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:14.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:14.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:14.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:14.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:14.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:14.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:14.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:14.645 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:14.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:14.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:14.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:14.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:14.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:14 smithi123 bash[19119]: cluster 2024-02-12T16:11:13.657321+0000 mgr.y (mgr.14146) 775 : cluster [DBG] pgmap v976: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 2053/266670 objects misplaced (0.770%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:11:14.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:14 smithi121 bash[18489]: cluster 2024-02-12T16:11:13.657321+0000 mgr.y (mgr.14146) 775 : cluster [DBG] pgmap v976: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 2053/266670 objects misplaced (0.770%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:11:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:14.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:15.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:15.041 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:14 smithi169 bash[19604]: cluster 2024-02-12T16:11:13.657321+0000 mgr.y (mgr.14146) 775 : cluster [DBG] pgmap v976: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 2053/266670 objects misplaced (0.770%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:11:15.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:15.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:15.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:15.303 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:15.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:15.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:15.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:15.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-02-12T16:11:15.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:15.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:15.717 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:15.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:15.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:15 smithi169 bash[19604]: cluster 2024-02-12T16:11:14.789519+0000 osd.2 (osd.2) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:15.827 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:15 smithi169 bash[19604]: cluster 2024-02-12T16:11:14.789584+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:15.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:15.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:15 smithi121 bash[18489]: cluster 2024-02-12T16:11:14.789519+0000 osd.2 (osd.2) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:15.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:15 smithi121 bash[18489]: cluster 2024-02-12T16:11:14.789584+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:15.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:15.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:15.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:15 smithi123 bash[19119]: cluster 2024-02-12T16:11:14.789519+0000 osd.2 (osd.2) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:15.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:15 smithi123 bash[19119]: cluster 2024-02-12T16:11:14.789584+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:15.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:16.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:16.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:16.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:16.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:16.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:16.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:16.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:16.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:16.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:16.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:16.509 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:16.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:16.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:16.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:16.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:16.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:16.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:16.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:16.907 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:16 smithi121 bash[18489]: cluster 2024-02-12T16:11:15.657772+0000 mgr.y (mgr.14146) 776 : cluster [DBG] pgmap v977: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1843/259100 objects misplaced (0.711%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:11:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:16 smithi121 bash[18489]: cluster 2024-02-12T16:11:15.719662+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e299: 12 total, 12 up, 12 in 2024-02-12T16:11:16.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:16.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:17.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:17.018 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:16 smithi169 bash[19604]: cluster 2024-02-12T16:11:15.657772+0000 mgr.y (mgr.14146) 776 : cluster [DBG] pgmap v977: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1843/259100 objects misplaced (0.711%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:11:17.018 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:16 smithi169 bash[19604]: cluster 2024-02-12T16:11:15.719662+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e299: 12 total, 12 up, 12 in 2024-02-12T16:11:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:17.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:16 smithi123 bash[19119]: cluster 2024-02-12T16:11:15.657772+0000 mgr.y (mgr.14146) 776 : cluster [DBG] pgmap v977: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1843/259100 objects misplaced (0.711%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:11:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:16 smithi123 bash[19119]: cluster 2024-02-12T16:11:15.719662+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e299: 12 total, 12 up, 12 in 2024-02-12T16:11:17.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:17.170 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:17.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:17.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:17.419 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:17.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:17.559 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:17.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:17.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:17.718 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:17.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:17.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:17.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:17.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:17.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:18.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:18.118 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:18.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:18.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:18.252 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:18.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:18.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:18.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:18.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:18.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:18.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:18.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:18.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:18.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:18.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:18.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:18.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:18 smithi121 bash[18489]: cluster 2024-02-12T16:11:17.584436+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:11:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:18 smithi121 bash[18489]: cluster 2024-02-12T16:11:17.658192+0000 mgr.y (mgr.14146) 777 : cluster [DBG] pgmap v979: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1843/257036 objects misplaced (0.717%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:11:18.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:18.944 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:18.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:18.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:18 smithi169 bash[19604]: cluster 2024-02-12T16:11:17.584436+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:11:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:18 smithi169 bash[19604]: cluster 2024-02-12T16:11:17.658192+0000 mgr.y (mgr.14146) 777 : cluster [DBG] pgmap v979: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1843/257036 objects misplaced (0.717%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:11:19.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:19.078 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:19.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:18 smithi123 bash[19119]: cluster 2024-02-12T16:11:17.584436+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:11:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:18 smithi123 bash[19119]: cluster 2024-02-12T16:11:17.658192+0000 mgr.y (mgr.14146) 777 : cluster [DBG] pgmap v979: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1843/257036 objects misplaced (0.717%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:11:19.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:19.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:19.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:19.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:19.356 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:19.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:19.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:19.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:19.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:19.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:19.625 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:19.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:19.729 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:19.761 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:19.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:19.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:19.888 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:19.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:19.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:20.077 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:20.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:20.155 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:20.260 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:20.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:20.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:20.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:20.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:20.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:20.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:20.673 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:20.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:20.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:20 smithi169 bash[19604]: cluster 2024-02-12T16:11:19.510559+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:11:20.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:20 smithi169 bash[19604]: cluster 2024-02-12T16:11:19.658646+0000 mgr.y (mgr.14146) 778 : cluster [DBG] pgmap v980: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s; 1843/252156 objects misplaced (0.731%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:11:20.803 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:20.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:20 smithi121 bash[18489]: cluster 2024-02-12T16:11:19.510559+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:11:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:20 smithi121 bash[18489]: cluster 2024-02-12T16:11:19.658646+0000 mgr.y (mgr.14146) 778 : cluster [DBG] pgmap v980: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s; 1843/252156 objects misplaced (0.731%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:11:20.941 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:20.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:21.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:21.053 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:20 smithi123 bash[19119]: cluster 2024-02-12T16:11:19.510559+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:11:21.053 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:20 smithi123 bash[19119]: cluster 2024-02-12T16:11:19.658646+0000 mgr.y (mgr.14146) 778 : cluster [DBG] pgmap v980: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s; 1843/252156 objects misplaced (0.731%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:11:21.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:21.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:21.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:21.210 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:21.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:21.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:21.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:21.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:21.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:21.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:21.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:21.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:21.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:21.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:21.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:21.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:21.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:21.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:21.848 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:21 smithi123 bash[19119]: cluster 2024-02-12T16:11:20.366874+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:21.848 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:21 smithi123 bash[19119]: cluster 2024-02-12T16:11:20.366981+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:21.848 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:21 smithi123 bash[19119]: cluster 2024-02-12T16:11:21.340772+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e300: 12 total, 12 up, 12 in 2024-02-12T16:11:21.868 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:21.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:21.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:21.980 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:21 smithi169 bash[19604]: cluster 2024-02-12T16:11:20.366874+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:21.980 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:21 smithi169 bash[19604]: cluster 2024-02-12T16:11:20.366981+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:21.980 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:21 smithi169 bash[19604]: cluster 2024-02-12T16:11:21.340772+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e300: 12 total, 12 up, 12 in 2024-02-12T16:11:22.004 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:21 smithi121 bash[18489]: cluster 2024-02-12T16:11:20.366874+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:22.004 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:21 smithi121 bash[18489]: cluster 2024-02-12T16:11:20.366981+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:22.004 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:21 smithi121 bash[18489]: cluster 2024-02-12T16:11:21.340772+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e300: 12 total, 12 up, 12 in 2024-02-12T16:11:22.011 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:22.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:22.144 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:22.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:22.282 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:22.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:22.412 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:22.515 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:22.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:22.644 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:22.670 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:22.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:22 smithi121 bash[18489]: cluster 2024-02-12T16:11:21.659036+0000 mgr.y (mgr.14146) 779 : cluster [DBG] pgmap v982: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 1619/247988 objects misplaced (0.653%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:11:22.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:22.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:22.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:22.943 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:22.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:22 smithi169 bash[19604]: cluster 2024-02-12T16:11:21.659036+0000 mgr.y (mgr.14146) 779 : cluster [DBG] pgmap v982: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 1619/247988 objects misplaced (0.653%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:11:23.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:23.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:23.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:22 smithi123 bash[19119]: cluster 2024-02-12T16:11:21.659036+0000 mgr.y (mgr.14146) 779 : cluster [DBG] pgmap v982: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 1619/247988 objects misplaced (0.653%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:11:23.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:23.226 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:23.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:23.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:23.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:23.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:23.463 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:23.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:23.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:23.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:23.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:23.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:23.743 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:23 smithi123 bash[19119]: cluster 2024-02-12T16:11:22.463881+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:23.743 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:23 smithi123 bash[19119]: cluster 2024-02-12T16:11:22.463937+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:23.743 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:23 smithi123 bash[19119]: cluster 2024-02-12T16:11:23.345055+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e301: 12 total, 12 up, 12 in 2024-02-12T16:11:23.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:23.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:23.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:23.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:23 smithi169 bash[19604]: cluster 2024-02-12T16:11:22.463881+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:23.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:23 smithi169 bash[19604]: cluster 2024-02-12T16:11:22.463937+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:23.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:23 smithi169 bash[19604]: cluster 2024-02-12T16:11:23.345055+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e301: 12 total, 12 up, 12 in 2024-02-12T16:11:23.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:23.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:24.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:24.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:23 smithi121 bash[18489]: cluster 2024-02-12T16:11:22.463881+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:24.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:23 smithi121 bash[18489]: cluster 2024-02-12T16:11:22.463937+0000 osd.1 (osd.1) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:24.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:23 smithi121 bash[18489]: cluster 2024-02-12T16:11:23.345055+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e301: 12 total, 12 up, 12 in 2024-02-12T16:11:24.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:24.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:24.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:24.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:24.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:24.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:24.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:24.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:24.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:24.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:24.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:24 smithi121 bash[18489]: cluster 2024-02-12T16:11:23.659380+0000 mgr.y (mgr.14146) 780 : cluster [DBG] pgmap v984: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 1619/244572 objects misplaced (0.662%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:11:24.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:24.950 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:24 smithi123 bash[19119]: cluster 2024-02-12T16:11:23.659380+0000 mgr.y (mgr.14146) 780 : cluster [DBG] pgmap v984: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 1619/244572 objects misplaced (0.662%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:11:24.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:24.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:24.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:25.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:24 smithi169 bash[19604]: cluster 2024-02-12T16:11:23.659380+0000 mgr.y (mgr.14146) 780 : cluster [DBG] pgmap v984: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 1619/244572 objects misplaced (0.662%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:11:25.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:25.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:25.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:25.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:25.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:25.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:25.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:25.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:25.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:25.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:25.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:25.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:25.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:25.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:25.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:25.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:25.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:25.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:25.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:25.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:26.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:26.040 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:26.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:26.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:26.171 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:26.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:26.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:26.307 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:11:26.307 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:11:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:26.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:26.435 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:26.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:26.545 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:26 smithi121 bash[18489]: cluster 2024-02-12T16:11:25.476888+0000 osd.10 (osd.10) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:26.545 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:26 smithi121 bash[18489]: cluster 2024-02-12T16:11:25.476947+0000 osd.10 (osd.10) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:26.545 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:26 smithi121 bash[18489]: cluster 2024-02-12T16:11:25.660455+0000 mgr.y (mgr.14146) 781 : cluster [DBG] pgmap v985: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 1394/236922 objects misplaced (0.588%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T16:11:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:26.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:26 smithi123 bash[19119]: cluster 2024-02-12T16:11:25.476888+0000 osd.10 (osd.10) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:26.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:26 smithi123 bash[19119]: cluster 2024-02-12T16:11:25.476947+0000 osd.10 (osd.10) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:26.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:26 smithi123 bash[19119]: cluster 2024-02-12T16:11:25.660455+0000 mgr.y (mgr.14146) 781 : cluster [DBG] pgmap v985: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 1394/236922 objects misplaced (0.588%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T16:11:26.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:26.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:26.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:26 smithi169 bash[19604]: cluster 2024-02-12T16:11:25.476888+0000 osd.10 (osd.10) 23 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:26.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:26 smithi169 bash[19604]: cluster 2024-02-12T16:11:25.476947+0000 osd.10 (osd.10) 24 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:26.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:26 smithi169 bash[19604]: cluster 2024-02-12T16:11:25.660455+0000 mgr.y (mgr.14146) 781 : cluster [DBG] pgmap v985: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 1394/236922 objects misplaced (0.588%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T16:11:26.812 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:26.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:26.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:26.976 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:26.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:27.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:27.104 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:27.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:27.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:27.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:27.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:27.372 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:27.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:27.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:27.488 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:27 smithi121 bash[18489]: cluster 2024-02-12T16:11:25.646231+0000 osd.0 (osd.0) 45 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:11:27.489 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:27 smithi121 bash[18489]: cluster 2024-02-12T16:11:26.345268+0000 mon.a (mon.0) 1247 : cluster [DBG] osdmap e302: 12 total, 12 up, 12 in 2024-02-12T16:11:27.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:27.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:27.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:27.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:27 smithi169 bash[19604]: cluster 2024-02-12T16:11:25.646231+0000 osd.0 (osd.0) 45 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:11:27.625 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:27 smithi169 bash[19604]: cluster 2024-02-12T16:11:26.345268+0000 mon.a (mon.0) 1247 : cluster [DBG] osdmap e302: 12 total, 12 up, 12 in 2024-02-12T16:11:27.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:27 smithi123 bash[19119]: cluster 2024-02-12T16:11:25.646231+0000 osd.0 (osd.0) 45 : cluster [DBG] 3.1 scrub starts 2024-02-12T16:11:27.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:27 smithi123 bash[19119]: cluster 2024-02-12T16:11:26.345268+0000 mon.a (mon.0) 1247 : cluster [DBG] osdmap e302: 12 total, 12 up, 12 in 2024-02-12T16:11:27.648 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:27.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:27.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:27.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:27.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:27.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:27.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:27.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:28.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:28.039 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:28.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:28.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:28.174 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:28.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:28.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:28.312 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:28.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:28.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:28.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:28.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:28.539 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:28 smithi121 bash[18489]: cluster 2024-02-12T16:11:27.661041+0000 mgr.y (mgr.14146) 782 : cluster [DBG] pgmap v987: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 1394/234988 objects misplaced (0.593%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:11:28.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:28.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:11:28.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:28.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:28.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:28 smithi123 bash[19119]: cluster 2024-02-12T16:11:27.661041+0000 mgr.y (mgr.14146) 782 : cluster [DBG] pgmap v987: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 1394/234988 objects misplaced (0.593%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:11:28.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:28.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:28.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:28.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:28 smithi169 bash[19604]: cluster 2024-02-12T16:11:27.661041+0000 mgr.y (mgr.14146) 782 : cluster [DBG] pgmap v987: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 1394/234988 objects misplaced (0.593%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:11:28.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:28.888 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:11:28.894 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:28.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:28.936 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:11:28.970 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:11:29.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:29.015 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:11:29.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:29.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:29.056 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:11:29.097 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:11:29.128 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:11:29.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:29.162 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:11:29.199 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:11:29.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:29.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:29.224 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:11:29.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:11:29.286 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:11:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.288 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:11:29.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:29.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:29.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:29.364 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:11:29.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:29.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:29.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:29.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:29.605 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:29.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:29.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:29 smithi123 bash[19119]: cluster 2024-02-12T16:11:27.728997+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.5 scrub starts 2024-02-12T16:11:29.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:29 smithi123 bash[19119]: cluster 2024-02-12T16:11:27.950675+0000 osd.0 (osd.0) 46 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:11:29.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:29 smithi123 bash[19119]: audit 2024-02-12T16:11:28.833420+0000 mon.b (mon.2) 110 : audit [DBG] from='client.? v1:172.21.15.121:0/1094537020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:29 smithi123 bash[19119]: audit 2024-02-12T16:11:29.218614+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.121:0/4115111106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:11:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:29 smithi121 bash[18489]: cluster 2024-02-12T16:11:27.728997+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.5 scrub starts 2024-02-12T16:11:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:29 smithi121 bash[18489]: cluster 2024-02-12T16:11:27.950675+0000 osd.0 (osd.0) 46 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:11:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:29 smithi121 bash[18489]: audit 2024-02-12T16:11:28.833420+0000 mon.b (mon.2) 110 : audit [DBG] from='client.? v1:172.21.15.121:0/1094537020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:29 smithi121 bash[18489]: audit 2024-02-12T16:11:29.218614+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.121:0/4115111106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:11:29.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:29.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:29.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:29.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:29 smithi169 bash[19604]: cluster 2024-02-12T16:11:27.728997+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.5 scrub starts 2024-02-12T16:11:29.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:29 smithi169 bash[19604]: cluster 2024-02-12T16:11:27.950675+0000 osd.0 (osd.0) 46 : cluster [DBG] 3.1 scrub ok 2024-02-12T16:11:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:29 smithi169 bash[19604]: audit 2024-02-12T16:11:28.833420+0000 mon.b (mon.2) 110 : audit [DBG] from='client.? v1:172.21.15.121:0/1094537020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:29 smithi169 bash[19604]: audit 2024-02-12T16:11:29.218614+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? v1:172.21.15.121:0/4115111106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:11:29.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:29.894 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:29.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:30.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:30.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:30.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:30.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:30.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:30.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:30 smithi121 bash[18489]: cluster 2024-02-12T16:11:29.347581+0000 mon.a (mon.0) 1249 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:11:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:30 smithi121 bash[18489]: cluster 2024-02-12T16:11:29.347620+0000 mon.a (mon.0) 1250 : cluster [INF] Cluster is now healthy 2024-02-12T16:11:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:30 smithi121 bash[18489]: audit 2024-02-12T16:11:29.354550+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? v1:172.21.15.121:0/4115111106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:11:30.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:30 smithi121 bash[18489]: cluster 2024-02-12T16:11:29.354615+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e303: 12 total, 12 up, 12 in 2024-02-12T16:11:30.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:30 smithi121 bash[18489]: cluster 2024-02-12T16:11:29.620651+0000 osd.9 (osd.9) 30 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:30.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:30 smithi121 bash[18489]: cluster 2024-02-12T16:11:29.620709+0000 osd.9 (osd.9) 31 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:30.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:30 smithi121 bash[18489]: cluster 2024-02-12T16:11:29.661523+0000 mgr.y (mgr.14146) 783 : cluster [DBG] pgmap v989: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 1394/230600 objects misplaced (0.605%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:11:30.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:30.437 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:30.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:30.566 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:30.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:30.679 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:30 smithi123 bash[19119]: cluster 2024-02-12T16:11:29.347581+0000 mon.a (mon.0) 1249 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:11:30.679 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:30 smithi123 bash[19119]: cluster 2024-02-12T16:11:29.347620+0000 mon.a (mon.0) 1250 : cluster [INF] Cluster is now healthy 2024-02-12T16:11:30.679 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:30 smithi123 bash[19119]: audit 2024-02-12T16:11:29.354550+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? v1:172.21.15.121:0/4115111106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:11:30.679 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:30 smithi123 bash[19119]: cluster 2024-02-12T16:11:29.354615+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e303: 12 total, 12 up, 12 in 2024-02-12T16:11:30.680 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:30 smithi123 bash[19119]: cluster 2024-02-12T16:11:29.620651+0000 osd.9 (osd.9) 30 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:30.680 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:30 smithi123 bash[19119]: cluster 2024-02-12T16:11:29.620709+0000 osd.9 (osd.9) 31 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:30.680 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:30 smithi123 bash[19119]: cluster 2024-02-12T16:11:29.661523+0000 mgr.y (mgr.14146) 783 : cluster [DBG] pgmap v989: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 1394/230600 objects misplaced (0.605%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:11:30.697 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:30.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:30.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-02-12T16:11:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:30 smithi169 bash[19604]: cluster 2024-02-12T16:11:29.347581+0000 mon.a (mon.0) 1249 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:11:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:30 smithi169 bash[19604]: cluster 2024-02-12T16:11:29.347620+0000 mon.a (mon.0) 1250 : cluster [INF] Cluster is now healthy 2024-02-12T16:11:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:30 smithi169 bash[19604]: audit 2024-02-12T16:11:29.354550+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? v1:172.21.15.121:0/4115111106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:11:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:30 smithi169 bash[19604]: cluster 2024-02-12T16:11:29.354615+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e303: 12 total, 12 up, 12 in 2024-02-12T16:11:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:30 smithi169 bash[19604]: cluster 2024-02-12T16:11:29.620651+0000 osd.9 (osd.9) 30 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:30 smithi169 bash[19604]: cluster 2024-02-12T16:11:29.620709+0000 osd.9 (osd.9) 31 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:30 smithi169 bash[19604]: cluster 2024-02-12T16:11:29.661523+0000 mgr.y (mgr.14146) 783 : cluster [DBG] pgmap v989: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 1394/230600 objects misplaced (0.605%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:11:30.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:30.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:30.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:30.978 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:30.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:31.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:31.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:31.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:31.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:31.256 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:31.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:31.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:31.382 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:31.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:31.394 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:31 smithi169 bash[19604]: cluster 2024-02-12T16:11:30.305411+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.5 scrub ok 2024-02-12T16:11:31.394 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:31 smithi169 bash[19604]: cluster 2024-02-12T16:11:30.390435+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e304: 12 total, 12 up, 12 in 2024-02-12T16:11:31.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:31.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:31.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:31.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:31.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:31 smithi123 bash[19119]: cluster 2024-02-12T16:11:30.305411+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.5 scrub ok 2024-02-12T16:11:31.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:31 smithi123 bash[19119]: cluster 2024-02-12T16:11:30.390435+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e304: 12 total, 12 up, 12 in 2024-02-12T16:11:31.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:31.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:31 smithi121 bash[18489]: cluster 2024-02-12T16:11:30.305411+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.5 scrub ok 2024-02-12T16:11:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:31 smithi121 bash[18489]: cluster 2024-02-12T16:11:30.390435+0000 mon.a (mon.0) 1253 : cluster [DBG] osdmap e304: 12 total, 12 up, 12 in 2024-02-12T16:11:31.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:31.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:31.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:31.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:31.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:31.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:32.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:32.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:32.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:32.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:32.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:32 smithi121 bash[18489]: cluster 2024-02-12T16:11:31.661876+0000 mgr.y (mgr.14146) 784 : cluster [DBG] pgmap v991: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s; 1144/226922 objects misplaced (0.504%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:11:32.415 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:32.442 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:32.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:32.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:32.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:32 smithi123 bash[19119]: cluster 2024-02-12T16:11:31.661876+0000 mgr.y (mgr.14146) 784 : cluster [DBG] pgmap v991: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s; 1144/226922 objects misplaced (0.504%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:11:32.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:32.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:32.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:32 smithi169 bash[19604]: cluster 2024-02-12T16:11:31.661876+0000 mgr.y (mgr.14146) 784 : cluster [DBG] pgmap v991: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s; 1144/226922 objects misplaced (0.504%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:11:32.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:32.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:32.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:32.968 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:32.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:33.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:33.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:33.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:33.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:33.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:33.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:33.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:33.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:33.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:33.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:11:33.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:11:33.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:33.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:33.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:33.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:33.693 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:33.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:33.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:33.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:33.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:33.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:33.965 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:33.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:34.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:34.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:34.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:34.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:34.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:34.230 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:34.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:34.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:34.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:34.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:34.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:34.497 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:34.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:34.602 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:34.636 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:34.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:34.725 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:34 smithi121 bash[18489]: cluster 2024-02-12T16:11:33.662340+0000 mgr.y (mgr.14146) 785 : cluster [DBG] pgmap v992: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 1144/223608 objects misplaced (0.512%); 2.1 MiB/s, 34 objects/s recovering 2024-02-12T16:11:34.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:34.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:34.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:34.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:34 smithi123 bash[19119]: cluster 2024-02-12T16:11:33.662340+0000 mgr.y (mgr.14146) 785 : cluster [DBG] pgmap v992: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 1144/223608 objects misplaced (0.512%); 2.1 MiB/s, 34 objects/s recovering 2024-02-12T16:11:34.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:34.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:34.978 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:35.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:35.035 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:34 smithi169 bash[19604]: cluster 2024-02-12T16:11:33.662340+0000 mgr.y (mgr.14146) 785 : cluster [DBG] pgmap v992: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 1144/223608 objects misplaced (0.512%); 2.1 MiB/s, 34 objects/s recovering 2024-02-12T16:11:35.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:35.164 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:35.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:35.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:35.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:35.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:35.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:35.583 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:35.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:35.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:35.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:35.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:35.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:35.990 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:35.991 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:35.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:36.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:36.127 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:36.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:36.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:36.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:36.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:36.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:36.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:36.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:36.518 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:36.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:36.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:36.627 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:36 smithi123 bash[19119]: cluster 2024-02-12T16:11:35.662733+0000 mgr.y (mgr.14146) 786 : cluster [DBG] pgmap v993: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 881/215994 objects misplaced (0.408%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:11:36.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:36.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:36 smithi121 bash[18489]: cluster 2024-02-12T16:11:35.662733+0000 mgr.y (mgr.14146) 786 : cluster [DBG] pgmap v993: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 881/215994 objects misplaced (0.408%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:11:36.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:36.759 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:36 smithi169 bash[19604]: cluster 2024-02-12T16:11:35.662733+0000 mgr.y (mgr.14146) 786 : cluster [DBG] pgmap v993: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 881/215994 objects misplaced (0.408%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:11:36.778 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:36.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:36.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:36.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:36.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:37.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:37.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:37.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:37.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:37.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:37.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:37.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:37.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:37.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:37.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:37.467 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:37.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:37.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:37.599 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:37.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:37.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:37.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:37.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:37.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:37.863 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:37.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:37.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:38.029 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:38.030 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:38.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:38.135 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:38.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:38.270 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:38.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:38.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:38.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:38.544 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:38.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:38.681 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:38.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:38.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:38 smithi121 bash[18489]: cluster 2024-02-12T16:11:37.663182+0000 mgr.y (mgr.14146) 787 : cluster [DBG] pgmap v994: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 881/214072 objects misplaced (0.412%); 3.9 MiB/s, 61 objects/s recovering 2024-02-12T16:11:38.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:38.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:38.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:39.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:38 smithi169 bash[19604]: cluster 2024-02-12T16:11:37.663182+0000 mgr.y (mgr.14146) 787 : cluster [DBG] pgmap v994: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 881/214072 objects misplaced (0.412%); 3.9 MiB/s, 61 objects/s recovering 2024-02-12T16:11:39.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:39.070 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:39.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:39.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:38 smithi123 bash[19119]: cluster 2024-02-12T16:11:37.663182+0000 mgr.y (mgr.14146) 787 : cluster [DBG] pgmap v994: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 881/214072 objects misplaced (0.412%); 3.9 MiB/s, 61 objects/s recovering 2024-02-12T16:11:39.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:39.204 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:39.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:39.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:39.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:39.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:39.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:39.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:39.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:39.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:39.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:39.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:39.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:39.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:39.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:39.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:39.876 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:39.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:39.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:39.988 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:39 smithi121 bash[18489]: cluster 2024-02-12T16:11:38.485280+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.3 scrub starts 2024-02-12T16:11:40.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:40.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:39 smithi169 bash[19604]: cluster 2024-02-12T16:11:38.485280+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.3 scrub starts 2024-02-12T16:11:40.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:40.120 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:39 smithi123 bash[19119]: cluster 2024-02-12T16:11:38.485280+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.3 scrub starts 2024-02-12T16:11:40.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:40.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:40.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:40.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:40.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:40.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:40.531 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:40.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:40.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:40.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:40 smithi169 bash[19604]: cluster 2024-02-12T16:11:39.663704+0000 mgr.y (mgr.14146) 788 : cluster [DBG] pgmap v995: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 15 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 881/209470 objects misplaced (0.421%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:11:40.805 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:40.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:40.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:40.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:41.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:41.067 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:41.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:41.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:40 smithi123 bash[19119]: cluster 2024-02-12T16:11:39.663704+0000 mgr.y (mgr.14146) 788 : cluster [DBG] pgmap v995: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 15 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 881/209470 objects misplaced (0.421%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:11:41.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:40 smithi121 bash[18489]: cluster 2024-02-12T16:11:39.663704+0000 mgr.y (mgr.14146) 788 : cluster [DBG] pgmap v995: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 15 active+clean; 6.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 881/209470 objects misplaced (0.421%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:11:41.173 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:41.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:41.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:41.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:41.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:41.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:41.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:41.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:41.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:41.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:41.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:41.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:41.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:41.728 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:11:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:11:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:11:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:11:41.728 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:11:41.746 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:41.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:41.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:41.862 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:41 smithi169 bash[19604]: cluster 2024-02-12T16:11:40.618207+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.3 scrub ok 2024-02-12T16:11:41.884 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:41.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:41.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:42.024 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:42.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:42.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:41 smithi123 bash[19119]: cluster 2024-02-12T16:11:40.618207+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.3 scrub ok 2024-02-12T16:11:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:41 smithi121 bash[18489]: cluster 2024-02-12T16:11:40.618207+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.3 scrub ok 2024-02-12T16:11:42.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:42.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:42.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:42.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:42.443 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:42.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:42.576 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:42.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:42.709 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:42.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:42.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:42 smithi123 bash[19119]: cluster 2024-02-12T16:11:41.664174+0000 mgr.y (mgr.14146) 789 : cluster [DBG] pgmap v996: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s; 610/205350 objects misplaced (0.297%); 3.0 MiB/s, 47 objects/s recovering 2024-02-12T16:11:42.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:42.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:42.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:42 smithi169 bash[19604]: cluster 2024-02-12T16:11:41.664174+0000 mgr.y (mgr.14146) 789 : cluster [DBG] pgmap v996: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s; 610/205350 objects misplaced (0.297%); 3.0 MiB/s, 47 objects/s recovering 2024-02-12T16:11:43.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:43.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:43.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:43.233 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:43.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:43.279 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:42 smithi121 bash[18489]: cluster 2024-02-12T16:11:41.664174+0000 mgr.y (mgr.14146) 789 : cluster [DBG] pgmap v996: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 6.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s; 610/205350 objects misplaced (0.297%); 3.0 MiB/s, 47 objects/s recovering 2024-02-12T16:11:43.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:43.366 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:43.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:43.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:43.512 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:43.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:43.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:43.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:43.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:43.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:11:43.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:43.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:43.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:43.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:43 smithi121 bash[18489]: cluster 2024-02-12T16:11:42.047423+0000 osd.3 (osd.3) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:43 smithi121 bash[18489]: cluster 2024-02-12T16:11:42.047466+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:43 smithi121 bash[18489]: cluster 2024-02-12T16:11:42.747893+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e305: 12 total, 12 up, 12 in 2024-02-12T16:11:43.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:43.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:43.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.3 reset_purged_snaps_last 2024-02-12T16:11:44.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:44.025 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:43 smithi169 bash[19604]: cluster 2024-02-12T16:11:42.047423+0000 osd.3 (osd.3) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:44.025 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:43 smithi169 bash[19604]: cluster 2024-02-12T16:11:42.047466+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:44.025 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:43 smithi169 bash[19604]: cluster 2024-02-12T16:11:42.747893+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e305: 12 total, 12 up, 12 in 2024-02-12T16:11:44.044 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:44.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:43 smithi123 bash[19119]: cluster 2024-02-12T16:11:42.047423+0000 osd.3 (osd.3) 33 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:44.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:43 smithi123 bash[19119]: cluster 2024-02-12T16:11:42.047466+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:44.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:43 smithi123 bash[19119]: cluster 2024-02-12T16:11:42.747893+0000 mon.a (mon.0) 1254 : cluster [DBG] osdmap e305: 12 total, 12 up, 12 in 2024-02-12T16:11:44.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:44.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:44.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:44.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:44.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:44.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:44.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:44.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:44.695 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:44.722 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:44.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:44.858 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:44 smithi121 bash[18489]: cluster 2024-02-12T16:11:43.664590+0000 mgr.y (mgr.14146) 790 : cluster [DBG] pgmap v998: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 610/201764 objects misplaced (0.302%); 3.3 MiB/s, 53 objects/s recovering 2024-02-12T16:11:44.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:44.969 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:44 smithi123 bash[19119]: cluster 2024-02-12T16:11:43.664590+0000 mgr.y (mgr.14146) 790 : cluster [DBG] pgmap v998: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 610/201764 objects misplaced (0.302%); 3.3 MiB/s, 53 objects/s recovering 2024-02-12T16:11:44.994 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:44.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:44 smithi169 bash[19604]: cluster 2024-02-12T16:11:43.664590+0000 mgr.y (mgr.14146) 790 : cluster [DBG] pgmap v998: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 610/201764 objects misplaced (0.302%); 3.3 MiB/s, 53 objects/s recovering 2024-02-12T16:11:45.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:45.128 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:45.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:45.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:45.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:45.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:45.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:45.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:45.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:45.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:45.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:45.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:45.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:45.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:45.675 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:45.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:45.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:45.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:45.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:45.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:45.952 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:45.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:46.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:46.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:46.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:46.094 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:11:46.152 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:11:46.182 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:11:46.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:46.217 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:11:46.220 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:46.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:46.242 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:11:46.275 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:11:46.304 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:11:46.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:46.334 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:11:46.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:46.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:46.366 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:11:46.367 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:11:46.394 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:11:46.423 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:11:46.448 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.451 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.451 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:11:46.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:46.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:46.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:46.512 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:46 smithi121 bash[18489]: cluster 2024-02-12T16:11:44.720877+0000 osd.5 (osd.5) 37 : cluster [DBG] 3.c scrub starts 2024-02-12T16:11:46.512 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:46 smithi121 bash[18489]: cluster 2024-02-12T16:11:45.664989+0000 mgr.y (mgr.14146) 791 : cluster [DBG] pgmap v999: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 303/193938 objects misplaced (0.156%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T16:11:46.513 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:46 smithi121 bash[18489]: audit 2024-02-12T16:11:46.050628+0000 mon.a (mon.0) 1255 : audit [DBG] from='client.? v1:172.21.15.121:0/3309217503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:46.513 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:46 smithi121 bash[18489]: audit 2024-02-12T16:11:46.199061+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? v1:172.21.15.121:0/2973649699' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:11:46.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:46.610 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:46 smithi169 bash[19604]: cluster 2024-02-12T16:11:44.720877+0000 osd.5 (osd.5) 37 : cluster [DBG] 3.c scrub starts 2024-02-12T16:11:46.610 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:46 smithi169 bash[19604]: cluster 2024-02-12T16:11:45.664989+0000 mgr.y (mgr.14146) 791 : cluster [DBG] pgmap v999: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 303/193938 objects misplaced (0.156%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T16:11:46.610 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:46 smithi169 bash[19604]: audit 2024-02-12T16:11:46.050628+0000 mon.a (mon.0) 1255 : audit [DBG] from='client.? v1:172.21.15.121:0/3309217503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:46.610 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:46 smithi169 bash[19604]: audit 2024-02-12T16:11:46.199061+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? v1:172.21.15.121:0/2973649699' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:11:46.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:46 smithi123 bash[19119]: cluster 2024-02-12T16:11:44.720877+0000 osd.5 (osd.5) 37 : cluster [DBG] 3.c scrub starts 2024-02-12T16:11:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:46 smithi123 bash[19119]: cluster 2024-02-12T16:11:45.664989+0000 mgr.y (mgr.14146) 791 : cluster [DBG] pgmap v999: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 303/193938 objects misplaced (0.156%); 3.6 MiB/s, 57 objects/s recovering 2024-02-12T16:11:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:46 smithi123 bash[19119]: audit 2024-02-12T16:11:46.050628+0000 mon.a (mon.0) 1255 : audit [DBG] from='client.? v1:172.21.15.121:0/3309217503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:11:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:46 smithi123 bash[19119]: audit 2024-02-12T16:11:46.199061+0000 mon.a (mon.0) 1256 : audit [INF] from='client.? v1:172.21.15.121:0/2973649699' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:11:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:46.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:46.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:46.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:46.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:46.874 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:46.900 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:46.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:47.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:47.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:47.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:47.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:47.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:47.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:47.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:47.443 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:47.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:47.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:47 smithi123 bash[19119]: cluster 2024-02-12T16:11:46.336157+0000 mon.a (mon.0) 1257 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:11:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:47 smithi123 bash[19119]: cluster 2024-02-12T16:11:46.336705+0000 osd.5 (osd.5) 38 : cluster [DBG] 3.c scrub ok 2024-02-12T16:11:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:47 smithi123 bash[19119]: audit 2024-02-12T16:11:46.347662+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? v1:172.21.15.121:0/2973649699' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:11:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:47 smithi123 bash[19119]: cluster 2024-02-12T16:11:46.347739+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e306: 12 total, 12 up, 12 in 2024-02-12T16:11:47.639 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:47 smithi121 bash[18489]: cluster 2024-02-12T16:11:46.336157+0000 mon.a (mon.0) 1257 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:11:47.639 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:47 smithi121 bash[18489]: cluster 2024-02-12T16:11:46.336705+0000 osd.5 (osd.5) 38 : cluster [DBG] 3.c scrub ok 2024-02-12T16:11:47.639 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:47 smithi121 bash[18489]: audit 2024-02-12T16:11:46.347662+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? v1:172.21.15.121:0/2973649699' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:11:47.639 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:47 smithi121 bash[18489]: cluster 2024-02-12T16:11:46.347739+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e306: 12 total, 12 up, 12 in 2024-02-12T16:11:47.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:47.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:47 smithi169 bash[19604]: cluster 2024-02-12T16:11:46.336157+0000 mon.a (mon.0) 1257 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:11:47.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:47 smithi169 bash[19604]: cluster 2024-02-12T16:11:46.336705+0000 osd.5 (osd.5) 38 : cluster [DBG] 3.c scrub ok 2024-02-12T16:11:47.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:47 smithi169 bash[19604]: audit 2024-02-12T16:11:46.347662+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? v1:172.21.15.121:0/2973649699' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:11:47.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:47 smithi169 bash[19604]: cluster 2024-02-12T16:11:46.347739+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e306: 12 total, 12 up, 12 in 2024-02-12T16:11:47.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:47.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:47.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:47.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:47.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:47.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:48.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:48.139 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:48.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:48.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:48.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:48.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:48.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:48.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:48.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:48.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:48.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:48.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:48.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:48 smithi169 bash[19604]: cluster 2024-02-12T16:11:47.385547+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:48.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:48 smithi169 bash[19604]: cluster 2024-02-12T16:11:47.385605+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:48.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:48 smithi169 bash[19604]: cluster 2024-02-12T16:11:47.665407+0000 mgr.y (mgr.14146) 792 : cluster [DBG] pgmap v1001: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 303/191758 objects misplaced (0.158%); 4.5 MiB/s, 72 objects/s recovering 2024-02-12T16:11:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:48 smithi123 bash[19119]: cluster 2024-02-12T16:11:47.385547+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:48 smithi123 bash[19119]: cluster 2024-02-12T16:11:47.385605+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:48 smithi123 bash[19119]: cluster 2024-02-12T16:11:47.665407+0000 mgr.y (mgr.14146) 792 : cluster [DBG] pgmap v1001: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 303/191758 objects misplaced (0.158%); 4.5 MiB/s, 72 objects/s recovering 2024-02-12T16:11:48.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:48 smithi121 bash[18489]: cluster 2024-02-12T16:11:47.385547+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:11:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:48 smithi121 bash[18489]: cluster 2024-02-12T16:11:47.385605+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:11:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:48 smithi121 bash[18489]: cluster 2024-02-12T16:11:47.665407+0000 mgr.y (mgr.14146) 792 : cluster [DBG] pgmap v1001: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 303/191758 objects misplaced (0.158%); 4.5 MiB/s, 72 objects/s recovering 2024-02-12T16:11:48.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:48.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:48.790 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:48.825 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:48.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:48.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:48.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:48.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:49.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:49.095 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:49.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:49.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:49.238 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:49.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:49.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:49.357 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:49 smithi121 bash[18489]: cluster 2024-02-12T16:11:48.352295+0000 mon.a (mon.0) 1260 : cluster [DBG] osdmap e307: 12 total, 12 up, 12 in 2024-02-12T16:11:49.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:49.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:49.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:49.520 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:49.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:49.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:11:49.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:49.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:49 smithi123 bash[19119]: cluster 2024-02-12T16:11:48.352295+0000 mon.a (mon.0) 1260 : cluster [DBG] osdmap e307: 12 total, 12 up, 12 in 2024-02-12T16:11:49.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:11:49.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:49.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:49.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:49.756 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:49 smithi169 bash[19604]: cluster 2024-02-12T16:11:48.352295+0000 mon.a (mon.0) 1260 : cluster [DBG] osdmap e307: 12 total, 12 up, 12 in 2024-02-12T16:11:49.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:49.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:49.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:49.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:49.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:50.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:50.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:50.132 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:50.132 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:50.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:50.162 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:50.265 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:50.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:50.435 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:50.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:50.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:50 smithi123 bash[19119]: cluster 2024-02-12T16:11:49.665842+0000 mgr.y (mgr.14146) 793 : cluster [DBG] pgmap v1003: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 303/188818 objects misplaced (0.160%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T16:11:50.573 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:50 smithi121 bash[18489]: cluster 2024-02-12T16:11:49.665842+0000 mgr.y (mgr.14146) 793 : cluster [DBG] pgmap v1003: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 303/188818 objects misplaced (0.160%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T16:11:50.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:50.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:50 smithi169 bash[19604]: cluster 2024-02-12T16:11:49.665842+0000 mgr.y (mgr.14146) 793 : cluster [DBG] pgmap v1003: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 303/188818 objects misplaced (0.160%); 2.8 MiB/s, 44 objects/s recovering 2024-02-12T16:11:50.808 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:50.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:50.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:50.965 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:11:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:50.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:11:51.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:51.102 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:51.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:51.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:51.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:51.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:51.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:51.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:51.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:51.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:51.486 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:51.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:51.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:51.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:51.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:51.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:51.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:51.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:51.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:51.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:51.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:51.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:52.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:52.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:52.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:52.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:52.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:52.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:52.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:52.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:52.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:52.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:52.680 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:52.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:52.794 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:52 smithi169 bash[19604]: cluster 2024-02-12T16:11:51.666248+0000 mgr.y (mgr.14146) 794 : cluster [DBG] pgmap v1004: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:11:52.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:52.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:52.947 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:52.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:53.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:53.059 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:52 smithi123 bash[19119]: cluster 2024-02-12T16:11:51.666248+0000 mgr.y (mgr.14146) 794 : cluster [DBG] pgmap v1004: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:11:53.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:53.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:52 smithi121 bash[18489]: cluster 2024-02-12T16:11:51.666248+0000 mgr.y (mgr.14146) 794 : cluster [DBG] pgmap v1004: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:11:53.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:53.206 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:53.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:53.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:53.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:53.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:53.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:53.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:53.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:53.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:53.605 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:53.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:53.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:53.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:53.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:53.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:53.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:53.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:53.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:54.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:54.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:54.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:54.180 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:54.181 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:11:54.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:54.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:54.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:54.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:54.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:54.541 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:54.644 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:54.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:54.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:54.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:54 smithi169 bash[19604]: cluster 2024-02-12T16:11:53.666717+0000 mgr.y (mgr.14146) 795 : cluster [DBG] pgmap v1005: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-02-12T16:11:54.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:54.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:54.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:54.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:55.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:55.052 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:54 smithi123 bash[19119]: cluster 2024-02-12T16:11:53.666717+0000 mgr.y (mgr.14146) 795 : cluster [DBG] pgmap v1005: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-02-12T16:11:55.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:55.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:54 smithi121 bash[18489]: cluster 2024-02-12T16:11:53.666717+0000 mgr.y (mgr.14146) 795 : cluster [DBG] pgmap v1005: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-02-12T16:11:55.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:55.224 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:55.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:55.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:55.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:11:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:55.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:11:55.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:55.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:11:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:55.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:11:55.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:55.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:55.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:55.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:55.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:55.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:55.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:55.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:55.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:55.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:56.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:56.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:56.152 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:56.257 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:56.286 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:56.370 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:11:56.370 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:11:56.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:56.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:56 smithi121 bash[18489]: cluster 2024-02-12T16:11:55.667178+0000 mgr.y (mgr.14146) 796 : cluster [DBG] pgmap v1006: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 2.0 MiB/s, 73 objects/s recovering 2024-02-12T16:11:56.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:56.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:56.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:56 smithi169 bash[19604]: cluster 2024-02-12T16:11:55.667178+0000 mgr.y (mgr.14146) 796 : cluster [DBG] pgmap v1006: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 2.0 MiB/s, 73 objects/s recovering 2024-02-12T16:11:56.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:11:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:11:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:56 smithi123 bash[19119]: cluster 2024-02-12T16:11:55.667178+0000 mgr.y (mgr.14146) 796 : cluster [DBG] pgmap v1006: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 2.0 MiB/s, 73 objects/s recovering 2024-02-12T16:11:56.662 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:56.689 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:56.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:56.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:56.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:56.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:56.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:57.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:57.084 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:57.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:57.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:57.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:57.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:57.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:57.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:57.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:57.492 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:57.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:57.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-02-12T16:11:57.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:57.621 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:57.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:57.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:57.751 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:57.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:57.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:57.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:11:57.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:11:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:57.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:11:58.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:58.032 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:58.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:58.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:58.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:58.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:58.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:58.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:58.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:58.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:58.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:11:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:11:58.731 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:58 smithi123 bash[19119]: cluster 2024-02-12T16:11:57.667532+0000 mgr.y (mgr.14146) 797 : cluster [DBG] pgmap v1007: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1.9 MiB/s, 68 objects/s recovering 2024-02-12T16:11:58.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:58.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:58.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:58.979 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:11:58.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:58 smithi169 bash[19604]: cluster 2024-02-12T16:11:57.667532+0000 mgr.y (mgr.14146) 797 : cluster [DBG] pgmap v1007: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1.9 MiB/s, 68 objects/s recovering 2024-02-12T16:11:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:58.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:11:59.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:59.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:58 smithi121 bash[18489]: cluster 2024-02-12T16:11:57.667532+0000 mgr.y (mgr.14146) 797 : cluster [DBG] pgmap v1007: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1.9 MiB/s, 68 objects/s recovering 2024-02-12T16:11:59.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:11:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:59.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:11:59.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:59.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:11:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:59.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:11:59.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:59.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:11:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:59.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:11:59.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:59.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:59.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:59.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:59.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:11:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:59.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:11:59.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:59.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:59.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:11:59.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:59 smithi121 bash[18489]: audit 2024-02-12T16:11:58.732693+0000 mon.a (mon.0) 1261 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:11:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:59 smithi121 bash[18489]: audit 2024-02-12T16:11:59.096777+0000 mon.a (mon.0) 1262 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:11:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:59 smithi121 bash[18489]: audit 2024-02-12T16:11:59.097805+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:11:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:11:59 smithi121 bash[18489]: audit 2024-02-12T16:11:59.103745+0000 mon.a (mon.0) 1264 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:11:59.926 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:11:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:11:59.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:00.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:00.040 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:59 smithi123 bash[19119]: audit 2024-02-12T16:11:58.732693+0000 mon.a (mon.0) 1261 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:12:00.040 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:59 smithi123 bash[19119]: audit 2024-02-12T16:11:59.096777+0000 mon.a (mon.0) 1262 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:12:00.040 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:59 smithi123 bash[19119]: audit 2024-02-12T16:11:59.097805+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:12:00.040 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:11:59 smithi123 bash[19119]: audit 2024-02-12T16:11:59.103745+0000 mon.a (mon.0) 1264 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:12:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:59 smithi169 bash[19604]: audit 2024-02-12T16:11:58.732693+0000 mon.a (mon.0) 1261 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:12:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:59 smithi169 bash[19604]: audit 2024-02-12T16:11:59.096777+0000 mon.a (mon.0) 1262 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:12:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:59 smithi169 bash[19604]: audit 2024-02-12T16:11:59.097805+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:12:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:11:59 smithi169 bash[19604]: audit 2024-02-12T16:11:59.103745+0000 mon.a (mon.0) 1264 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:12:00.059 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:00.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:00.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:00.191 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:00.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:00.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:00.324 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:00.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:00.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:00.471 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:00.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:00.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:00.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:00.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:00.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:00.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:00.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:00.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:00.872 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:00.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:00.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:00.986 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:12:01.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:01.031 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:12:01.060 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:12:01.093 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:12:01.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:01.130 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:12:01.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:01.163 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:12:01.189 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:12:01.225 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:12:01.254 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:12:01.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:01.269 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:01 smithi169 bash[19604]: cluster 2024-02-12T16:11:59.667906+0000 mgr.y (mgr.14146) 798 : cluster [DBG] pgmap v1008: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1.7 MiB/s, 60 objects/s recovering 2024-02-12T16:12:01.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:01 smithi169 bash[19604]: audit 2024-02-12T16:12:00.960196+0000 mon.b (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.121:0/1182399165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:01.281 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:12:01.296 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:01.303 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:12:01.333 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:12:01.342 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.342 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.342 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.342 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.342 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.342 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.343 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.343 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.343 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.343 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.343 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.343 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:12:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:01 smithi123 bash[19119]: cluster 2024-02-12T16:11:59.667906+0000 mgr.y (mgr.14146) 798 : cluster [DBG] pgmap v1008: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1.7 MiB/s, 60 objects/s recovering 2024-02-12T16:12:01.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:01 smithi123 bash[19119]: audit 2024-02-12T16:12:00.960196+0000 mon.b (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.121:0/1182399165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:01.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:01.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:01 smithi121 bash[18489]: cluster 2024-02-12T16:11:59.667906+0000 mgr.y (mgr.14146) 798 : cluster [DBG] pgmap v1008: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1.7 MiB/s, 60 objects/s recovering 2024-02-12T16:12:01.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:01 smithi121 bash[18489]: audit 2024-02-12T16:12:00.960196+0000 mon.b (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.121:0/1182399165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:01.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:01.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:01.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:01.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:01.691 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:01.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:01.829 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:01.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:01.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:02.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:02.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:02.126 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:12:02.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:02.214 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:02 smithi121 bash[18489]: cluster 2024-02-12T16:12:01.182604+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:02.214 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:02 smithi121 bash[18489]: cluster 2024-02-12T16:12:01.182660+0000 osd.4 (osd.4) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:02.214 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:02 smithi121 bash[18489]: audit 2024-02-12T16:12:01.230175+0000 mon.a (mon.0) 1265 : audit [INF] from='client.25123 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:02.214 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:02 smithi121 bash[18489]: audit 2024-02-12T16:12:01.232168+0000 mon.c (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.121:0/1257394474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:02.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:02.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:02.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:02.361 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:02.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:02 smithi123 bash[19119]: cluster 2024-02-12T16:12:01.182604+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:02 smithi123 bash[19119]: cluster 2024-02-12T16:12:01.182660+0000 osd.4 (osd.4) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:02 smithi123 bash[19119]: audit 2024-02-12T16:12:01.230175+0000 mon.a (mon.0) 1265 : audit [INF] from='client.25123 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:02.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:02 smithi123 bash[19119]: audit 2024-02-12T16:12:01.232168+0000 mon.c (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.121:0/1257394474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:02.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:02.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:02.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:02.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:02 smithi169 bash[19604]: cluster 2024-02-12T16:12:01.182604+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:02.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:02 smithi169 bash[19604]: cluster 2024-02-12T16:12:01.182660+0000 osd.4 (osd.4) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:02.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:02 smithi169 bash[19604]: audit 2024-02-12T16:12:01.230175+0000 mon.a (mon.0) 1265 : audit [INF] from='client.25123 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:02.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:02 smithi169 bash[19604]: audit 2024-02-12T16:12:01.232168+0000 mon.c (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.121:0/1257394474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:02.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:02.620 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:02.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:02.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:02.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:02.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:02.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:02.880 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:02.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:02.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:03.015 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:03.119 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:03 smithi121 bash[18489]: cluster 2024-02-12T16:12:01.668315+0000 mgr.y (mgr.14146) 799 : cluster [DBG] pgmap v1009: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 1.6 MiB/s, 91 objects/s recovering 2024-02-12T16:12:03.119 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:03 smithi121 bash[18489]: cluster 2024-02-12T16:12:02.104148+0000 mon.a (mon.0) 1266 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:03.119 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:03 smithi121 bash[18489]: audit 2024-02-12T16:12:02.116084+0000 mon.a (mon.0) 1267 : audit [INF] from='client.25123 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:12:03.119 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:03 smithi121 bash[18489]: cluster 2024-02-12T16:12:02.116169+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e308: 12 total, 12 up, 12 in 2024-02-12T16:12:03.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:03.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:03.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:03.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:03 smithi123 bash[19119]: cluster 2024-02-12T16:12:01.668315+0000 mgr.y (mgr.14146) 799 : cluster [DBG] pgmap v1009: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 1.6 MiB/s, 91 objects/s recovering 2024-02-12T16:12:03.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:03 smithi123 bash[19119]: cluster 2024-02-12T16:12:02.104148+0000 mon.a (mon.0) 1266 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:03.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:03 smithi123 bash[19119]: audit 2024-02-12T16:12:02.116084+0000 mon.a (mon.0) 1267 : audit [INF] from='client.25123 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:12:03.265 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:03 smithi123 bash[19119]: cluster 2024-02-12T16:12:02.116169+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e308: 12 total, 12 up, 12 in 2024-02-12T16:12:03.289 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:03.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:03.409 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:03.409 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:03.432 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:03.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:03.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:03 smithi169 bash[19604]: cluster 2024-02-12T16:12:01.668315+0000 mgr.y (mgr.14146) 799 : cluster [DBG] pgmap v1009: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 1.6 MiB/s, 91 objects/s recovering 2024-02-12T16:12:03.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:03 smithi169 bash[19604]: cluster 2024-02-12T16:12:02.104148+0000 mon.a (mon.0) 1266 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:03.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:03 smithi169 bash[19604]: audit 2024-02-12T16:12:02.116084+0000 mon.a (mon.0) 1267 : audit [INF] from='client.25123 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:12:03.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:03 smithi169 bash[19604]: cluster 2024-02-12T16:12:02.116169+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e308: 12 total, 12 up, 12 in 2024-02-12T16:12:03.568 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:03.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:03.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:03.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:03.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:03.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:03.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:04.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:04.099 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:04.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:04.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:04.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:12:04.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:04.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:04.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:04.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:04.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:04.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:04.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:04.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:04.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:04.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:04.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:04.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:04.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:04.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:04.876 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:04.903 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:04.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:04.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:12:05.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:05.033 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:05.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:05.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:05 smithi169 bash[19604]: cluster 2024-02-12T16:12:03.668735+0000 mgr.y (mgr.14146) 800 : cluster [DBG] pgmap v1011: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 0 B/s, 78 objects/s recovering 2024-02-12T16:12:05.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:05 smithi169 bash[19604]: cluster 2024-02-12T16:12:03.816002+0000 osd.7 (osd.7) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:05.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:05 smithi169 bash[19604]: cluster 2024-02-12T16:12:03.816054+0000 osd.7 (osd.7) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:05.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:05 smithi169 bash[19604]: cluster 2024-02-12T16:12:04.125311+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e309: 12 total, 12 up, 12 in 2024-02-12T16:12:05.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:05.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:05.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:05.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:05 smithi121 bash[18489]: cluster 2024-02-12T16:12:03.668735+0000 mgr.y (mgr.14146) 800 : cluster [DBG] pgmap v1011: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 0 B/s, 78 objects/s recovering 2024-02-12T16:12:05.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:05 smithi121 bash[18489]: cluster 2024-02-12T16:12:03.816002+0000 osd.7 (osd.7) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:05.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:05 smithi121 bash[18489]: cluster 2024-02-12T16:12:03.816054+0000 osd.7 (osd.7) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:05.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:05 smithi121 bash[18489]: cluster 2024-02-12T16:12:04.125311+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e309: 12 total, 12 up, 12 in 2024-02-12T16:12:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:05 smithi123 bash[19119]: cluster 2024-02-12T16:12:03.668735+0000 mgr.y (mgr.14146) 800 : cluster [DBG] pgmap v1011: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 0 B/s, 78 objects/s recovering 2024-02-12T16:12:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:05 smithi123 bash[19119]: cluster 2024-02-12T16:12:03.816002+0000 osd.7 (osd.7) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:05 smithi123 bash[19119]: cluster 2024-02-12T16:12:03.816054+0000 osd.7 (osd.7) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:05.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:05 smithi123 bash[19119]: cluster 2024-02-12T16:12:04.125311+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e309: 12 total, 12 up, 12 in 2024-02-12T16:12:05.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:05.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:05.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:05.545 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:05.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:05.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:05.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:05.820 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:05.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:05.959 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:05.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:06.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:06.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:06.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:06.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:06.212 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:06 smithi169 bash[19604]: cluster 2024-02-12T16:12:05.138004+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e310: 12 total, 12 up, 12 in 2024-02-12T16:12:06.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:06.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:06.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:06.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:06.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:06.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:06 smithi121 bash[18489]: cluster 2024-02-12T16:12:05.138004+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e310: 12 total, 12 up, 12 in 2024-02-12T16:12:06.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:06.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:06.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:06.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:06.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:06 smithi123 bash[19119]: cluster 2024-02-12T16:12:05.138004+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e310: 12 total, 12 up, 12 in 2024-02-12T16:12:06.646 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:06.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:06.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:06.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:06.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:06.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:06.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:06.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:07.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:07.033 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:07.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:07.164 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:07.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:07 smithi169 bash[19604]: cluster 2024-02-12T16:12:05.669353+0000 mgr.y (mgr.14146) 801 : cluster [DBG] pgmap v1014: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.43k op/s; 0 B/s, 68 objects/s recovering 2024-02-12T16:12:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:07 smithi169 bash[19604]: cluster 2024-02-12T16:12:06.137479+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e311: 12 total, 12 up, 12 in 2024-02-12T16:12:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:07 smithi169 bash[19604]: cluster 2024-02-12T16:12:06.618877+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:07.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:07 smithi169 bash[19604]: cluster 2024-02-12T16:12:06.618938+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:07.300 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:07.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:07 smithi121 bash[18489]: cluster 2024-02-12T16:12:05.669353+0000 mgr.y (mgr.14146) 801 : cluster [DBG] pgmap v1014: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.43k op/s; 0 B/s, 68 objects/s recovering 2024-02-12T16:12:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:07 smithi121 bash[18489]: cluster 2024-02-12T16:12:06.137479+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e311: 12 total, 12 up, 12 in 2024-02-12T16:12:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:07 smithi121 bash[18489]: cluster 2024-02-12T16:12:06.618877+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:07 smithi121 bash[18489]: cluster 2024-02-12T16:12:06.618938+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:07.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:07.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:07.568 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:07.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:07 smithi123 bash[19119]: cluster 2024-02-12T16:12:05.669353+0000 mgr.y (mgr.14146) 801 : cluster [DBG] pgmap v1014: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.43k op/s; 0 B/s, 68 objects/s recovering 2024-02-12T16:12:07.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:07 smithi123 bash[19119]: cluster 2024-02-12T16:12:06.137479+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e311: 12 total, 12 up, 12 in 2024-02-12T16:12:07.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:07 smithi123 bash[19119]: cluster 2024-02-12T16:12:06.618877+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:07.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:07 smithi123 bash[19119]: cluster 2024-02-12T16:12:06.618938+0000 osd.6 (osd.6) 33 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:07.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:07.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:07.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:07.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:07.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:07.978 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:07.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:08.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:08.109 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:08.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:08.203 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:12:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:12:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:12:08.203 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:12:08.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:08.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:08.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:08.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:08.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:08.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:08.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:08.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:08 smithi169 bash[19604]: cluster 2024-02-12T16:12:07.345113+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e312: 12 total, 12 up, 12 in 2024-02-12T16:12:08.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:08 smithi169 bash[19604]: cluster 2024-02-12T16:12:07.669782+0000 mgr.y (mgr.14146) 802 : cluster [DBG] pgmap v1017: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:12:08.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:08.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:08.602 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:08.609 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:08 smithi123 bash[19119]: cluster 2024-02-12T16:12:07.345113+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e312: 12 total, 12 up, 12 in 2024-02-12T16:12:08.609 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:08 smithi123 bash[19119]: cluster 2024-02-12T16:12:07.669782+0000 mgr.y (mgr.14146) 802 : cluster [DBG] pgmap v1017: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:12:08.628 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:08.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:08 smithi121 bash[18489]: cluster 2024-02-12T16:12:07.345113+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e312: 12 total, 12 up, 12 in 2024-02-12T16:12:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:08 smithi121 bash[18489]: cluster 2024-02-12T16:12:07.669782+0000 mgr.y (mgr.14146) 802 : cluster [DBG] pgmap v1017: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:12:08.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:08.756 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:08.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:08.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:08.885 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:08.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:08.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:09.022 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:09.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:09.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:09.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:09.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:09.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:09.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:09.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:09.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-02-12T16:12:09.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:12:09.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:09.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:09.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:09.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:09.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:09.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:09.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:09.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:09.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:09.781 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:09.781 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:09.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:09.873 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:09.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:09.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:10.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:10.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:10.132 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:10.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:10.287 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:10.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:10.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:10.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:10.557 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:10.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:10.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:10.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:10.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:10.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:10.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:10.960 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:10.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:10 smithi169 bash[19604]: cluster 2024-02-12T16:12:09.670207+0000 mgr.y (mgr.14146) 803 : cluster [DBG] pgmap v1018: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 0 B/s, 84 objects/s recovering 2024-02-12T16:12:11.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:11.088 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:11.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:10 smithi123 bash[19119]: cluster 2024-02-12T16:12:09.670207+0000 mgr.y (mgr.14146) 803 : cluster [DBG] pgmap v1018: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 0 B/s, 84 objects/s recovering 2024-02-12T16:12:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:10 smithi121 bash[18489]: cluster 2024-02-12T16:12:09.670207+0000 mgr.y (mgr.14146) 803 : cluster [DBG] pgmap v1018: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 0 B/s, 84 objects/s recovering 2024-02-12T16:12:11.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:11.213 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:11.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:11.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:11.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:11.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:11.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:11.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:11.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:11.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:11.611 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:11.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:11.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:11.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:11.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:11.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:11.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:11.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:11.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:12.010 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:12.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:12.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:12.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:12.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:12.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:12.411 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:12.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:12.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:12.665 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:12.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:12.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:12.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:12 smithi121 bash[18489]: cluster 2024-02-12T16:12:10.975264+0000 osd.8 (osd.8) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:12 smithi121 bash[18489]: cluster 2024-02-12T16:12:10.975331+0000 osd.8 (osd.8) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:12 smithi121 bash[18489]: cluster 2024-02-12T16:12:11.670658+0000 mgr.y (mgr.14146) 804 : cluster [DBG] pgmap v1019: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 0 B/s, 71 objects/s recovering 2024-02-12T16:12:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:12 smithi121 bash[18489]: cluster 2024-02-12T16:12:11.735494+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e313: 12 total, 12 up, 12 in 2024-02-12T16:12:12.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:12.928 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:12 smithi123 bash[19119]: cluster 2024-02-12T16:12:10.975264+0000 osd.8 (osd.8) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:12.928 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:12 smithi123 bash[19119]: cluster 2024-02-12T16:12:10.975331+0000 osd.8 (osd.8) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:12.928 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:12 smithi123 bash[19119]: cluster 2024-02-12T16:12:11.670658+0000 mgr.y (mgr.14146) 804 : cluster [DBG] pgmap v1019: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 0 B/s, 71 objects/s recovering 2024-02-12T16:12:12.928 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:12 smithi123 bash[19119]: cluster 2024-02-12T16:12:11.735494+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e313: 12 total, 12 up, 12 in 2024-02-12T16:12:12.947 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:12.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:13.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:12 smithi169 bash[19604]: cluster 2024-02-12T16:12:10.975264+0000 osd.8 (osd.8) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:13.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:12 smithi169 bash[19604]: cluster 2024-02-12T16:12:10.975331+0000 osd.8 (osd.8) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:12 smithi169 bash[19604]: cluster 2024-02-12T16:12:11.670658+0000 mgr.y (mgr.14146) 804 : cluster [DBG] pgmap v1019: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 0 B/s, 71 objects/s recovering 2024-02-12T16:12:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:12 smithi169 bash[19604]: cluster 2024-02-12T16:12:11.735494+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e313: 12 total, 12 up, 12 in 2024-02-12T16:12:13.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.080 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:13.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.210 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:13.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:13.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:13.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.472 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:13.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:13.605 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:13.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.742 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:13.844 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:13.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:13.872 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:13.872 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:13.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:13.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:14.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:14.128 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:14.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:14.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:14.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:14.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:14.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:14.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:14.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:14.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:14.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:14.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:14.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:14.675 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:14.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:14.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:14.807 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:14.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:14 smithi121 bash[18489]: cluster 2024-02-12T16:12:13.671021+0000 mgr.y (mgr.14146) 805 : cluster [DBG] pgmap v1021: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s; 0 B/s, 62 objects/s recovering 2024-02-12T16:12:14.911 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:14.935 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:14.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:15.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:14 smithi169 bash[19604]: cluster 2024-02-12T16:12:13.671021+0000 mgr.y (mgr.14146) 805 : cluster [DBG] pgmap v1021: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s; 0 B/s, 62 objects/s recovering 2024-02-12T16:12:15.066 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:15.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:15.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:14 smithi123 bash[19119]: cluster 2024-02-12T16:12:13.671021+0000 mgr.y (mgr.14146) 805 : cluster [DBG] pgmap v1021: 17 pgs: 17 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s; 0 B/s, 62 objects/s recovering 2024-02-12T16:12:15.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:15.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:15.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:15.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:15.324 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:15.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:15.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:15.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:15.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:15.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:15.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:15.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:15.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:15.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:15.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:15.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:15.800 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:12:15.837 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:12:15.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:15.879 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:12:15.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:15.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:15.918 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:12:15.954 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:12:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:12:15.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:16.018 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:12:16.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:16.047 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:12:16.075 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:12:16.119 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:12:16.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:16.149 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:12:16.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:16.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:16.175 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.179 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.179 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.179 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.179 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:12:16.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:16.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:16.359 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:16 smithi123 bash[19119]: cluster 2024-02-12T16:12:15.671603+0000 mgr.y (mgr.14146) 806 : cluster [DBG] pgmap v1022: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 0 B/s, 56 objects/s recovering 2024-02-12T16:12:16.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:16 smithi123 bash[19119]: audit 2024-02-12T16:12:15.707087+0000 mon.b (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.121:0/4010291819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:16.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:16 smithi123 bash[19119]: audit 2024-02-12T16:12:16.025171+0000 mon.a (mon.0) 1274 : audit [INF] from='client.25202 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:12:16.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:16 smithi123 bash[19119]: audit 2024-02-12T16:12:16.028194+0000 mon.b (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.121:0/2320264043' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:12:16.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:16.411 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:16.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:16.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:16 smithi169 bash[19604]: cluster 2024-02-12T16:12:15.671603+0000 mgr.y (mgr.14146) 806 : cluster [DBG] pgmap v1022: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 0 B/s, 56 objects/s recovering 2024-02-12T16:12:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:16 smithi169 bash[19604]: audit 2024-02-12T16:12:15.707087+0000 mon.b (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.121:0/4010291819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:16 smithi169 bash[19604]: audit 2024-02-12T16:12:16.025171+0000 mon.a (mon.0) 1274 : audit [INF] from='client.25202 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:12:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:16 smithi169 bash[19604]: audit 2024-02-12T16:12:16.028194+0000 mon.b (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.121:0/2320264043' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:12:16.607 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:12:16.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:16.650 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:16 smithi121 bash[18489]: cluster 2024-02-12T16:12:15.671603+0000 mgr.y (mgr.14146) 806 : cluster [DBG] pgmap v1022: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 0 B/s, 56 objects/s recovering 2024-02-12T16:12:16.650 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:16 smithi121 bash[18489]: audit 2024-02-12T16:12:15.707087+0000 mon.b (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.121:0/4010291819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:16.650 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:16 smithi121 bash[18489]: audit 2024-02-12T16:12:16.025171+0000 mon.a (mon.0) 1274 : audit [INF] from='client.25202 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:12:16.651 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:16 smithi121 bash[18489]: audit 2024-02-12T16:12:16.028194+0000 mon.b (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.121:0/2320264043' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:12:16.669 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:16.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:16.806 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:16.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:16.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:16.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:17.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:17.070 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:17.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:17.206 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:17.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:17.343 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:17.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:17.472 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:17.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:17.584 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:17 smithi169 bash[19604]: cluster 2024-02-12T16:12:16.344306+0000 mon.a (mon.0) 1275 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:17.584 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:17 smithi169 bash[19604]: audit 2024-02-12T16:12:16.596621+0000 mon.a (mon.0) 1276 : audit [INF] from='client.25202 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:12:17.584 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:17 smithi169 bash[19604]: cluster 2024-02-12T16:12:16.596697+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e314: 12 total, 12 up, 12 in 2024-02-12T16:12:17.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:17 smithi123 bash[19119]: cluster 2024-02-12T16:12:16.344306+0000 mon.a (mon.0) 1275 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:17 smithi123 bash[19119]: audit 2024-02-12T16:12:16.596621+0000 mon.a (mon.0) 1276 : audit [INF] from='client.25202 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:12:17.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:17 smithi123 bash[19119]: cluster 2024-02-12T16:12:16.596697+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e314: 12 total, 12 up, 12 in 2024-02-12T16:12:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:17 smithi121 bash[18489]: cluster 2024-02-12T16:12:16.344306+0000 mon.a (mon.0) 1275 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:17 smithi121 bash[18489]: audit 2024-02-12T16:12:16.596621+0000 mon.a (mon.0) 1276 : audit [INF] from='client.25202 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:12:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:17 smithi121 bash[18489]: cluster 2024-02-12T16:12:16.596697+0000 mon.a (mon.0) 1277 : cluster [DBG] osdmap e314: 12 total, 12 up, 12 in 2024-02-12T16:12:17.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:17.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:17.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:17.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.862 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:17.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:17.993 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:17.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:18.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:18.121 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:18.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:18.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:18.264 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:18.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:18.365 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:18 smithi169 bash[19604]: cluster 2024-02-12T16:12:17.671961+0000 mgr.y (mgr.14146) 807 : cluster [DBG] pgmap v1024: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-02-12T16:12:18.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:18.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:18.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:18.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:18.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:18.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:18.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:18.635 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:18 smithi123 bash[19119]: cluster 2024-02-12T16:12:17.671961+0000 mgr.y (mgr.14146) 807 : cluster [DBG] pgmap v1024: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-02-12T16:12:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:18 smithi121 bash[18489]: cluster 2024-02-12T16:12:17.671961+0000 mgr.y (mgr.14146) 807 : cluster [DBG] pgmap v1024: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-02-12T16:12:18.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:18.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:18.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:18.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:18.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:18.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:18.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:18.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:19.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:19.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:19.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:19.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:19.202 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:19.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:19.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:19.334 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:19.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:19.365 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:19 smithi121 bash[18489]: cluster 2024-02-12T16:12:17.455340+0000 osd.7 (osd.7) 49 : cluster [DBG] 3.f scrub starts 2024-02-12T16:12:19.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:19 smithi121 bash[18489]: audit 2024-02-12T16:12:19.057890+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:12:19.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:19.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:19.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:19.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:19.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:19.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:19 smithi123 bash[19119]: cluster 2024-02-12T16:12:17.455340+0000 osd.7 (osd.7) 49 : cluster [DBG] 3.f scrub starts 2024-02-12T16:12:19.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:19 smithi123 bash[19119]: audit 2024-02-12T16:12:19.057890+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:12:19.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:19.742 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:19.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:19.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:19 smithi169 bash[19604]: cluster 2024-02-12T16:12:17.455340+0000 osd.7 (osd.7) 49 : cluster [DBG] 3.f scrub starts 2024-02-12T16:12:19.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:19 smithi169 bash[19604]: audit 2024-02-12T16:12:19.057890+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:12:19.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:19.879 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:19.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:19.964 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:12:19.964 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:12:19.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:20.010 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:20.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:20.140 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:20.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:12:20.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:20.274 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:20.376 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:20 smithi123 bash[19119]: cluster 2024-02-12T16:12:18.964404+0000 osd.7 (osd.7) 50 : cluster [DBG] 3.f scrub ok 2024-02-12T16:12:20.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:20 smithi123 bash[19119]: cluster 2024-02-12T16:12:19.019397+0000 osd.2 (osd.2) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:20.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:20 smithi123 bash[19119]: cluster 2024-02-12T16:12:19.019452+0000 osd.2 (osd.2) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:20.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:20 smithi123 bash[19119]: cluster 2024-02-12T16:12:19.367598+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e315: 12 total, 12 up, 12 in 2024-02-12T16:12:20.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:20 smithi123 bash[19119]: cluster 2024-02-12T16:12:19.452131+0000 osd.3 (osd.3) 35 : cluster [DBG] 3.7 scrub starts 2024-02-12T16:12:20.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:20 smithi123 bash[19119]: cluster 2024-02-12T16:12:19.672320+0000 mgr.y (mgr.14146) 808 : cluster [DBG] pgmap v1026: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:12:20.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:20.405 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:20.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:20.545 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:20.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:20 smithi169 bash[19604]: cluster 2024-02-12T16:12:18.964404+0000 osd.7 (osd.7) 50 : cluster [DBG] 3.f scrub ok 2024-02-12T16:12:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:20 smithi169 bash[19604]: cluster 2024-02-12T16:12:19.019397+0000 osd.2 (osd.2) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:20.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:20 smithi169 bash[19604]: cluster 2024-02-12T16:12:19.019452+0000 osd.2 (osd.2) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:20 smithi169 bash[19604]: cluster 2024-02-12T16:12:19.367598+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e315: 12 total, 12 up, 12 in 2024-02-12T16:12:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:20 smithi169 bash[19604]: cluster 2024-02-12T16:12:19.452131+0000 osd.3 (osd.3) 35 : cluster [DBG] 3.7 scrub starts 2024-02-12T16:12:20.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:20 smithi169 bash[19604]: cluster 2024-02-12T16:12:19.672320+0000 mgr.y (mgr.14146) 808 : cluster [DBG] pgmap v1026: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:12:20.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:20 smithi121 bash[18489]: cluster 2024-02-12T16:12:18.964404+0000 osd.7 (osd.7) 50 : cluster [DBG] 3.f scrub ok 2024-02-12T16:12:20.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:20 smithi121 bash[18489]: cluster 2024-02-12T16:12:19.019397+0000 osd.2 (osd.2) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:20.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:20 smithi121 bash[18489]: cluster 2024-02-12T16:12:19.019452+0000 osd.2 (osd.2) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:20.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:20 smithi121 bash[18489]: cluster 2024-02-12T16:12:19.367598+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e315: 12 total, 12 up, 12 in 2024-02-12T16:12:20.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:20 smithi121 bash[18489]: cluster 2024-02-12T16:12:19.452131+0000 osd.3 (osd.3) 35 : cluster [DBG] 3.7 scrub starts 2024-02-12T16:12:20.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:20 smithi121 bash[18489]: cluster 2024-02-12T16:12:19.672320+0000 mgr.y (mgr.14146) 808 : cluster [DBG] pgmap v1026: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:12:20.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:20.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:12:20.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:20.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:20.943 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:20.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:20.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:21.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:21.111 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:21.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:21.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:21.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:21.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:21.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:21.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:21 smithi169 bash[19604]: cluster 2024-02-12T16:12:20.435528+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub starts 2024-02-12T16:12:21.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:21 smithi169 bash[19604]: cluster 2024-02-12T16:12:20.594923+0000 osd.3 (osd.3) 36 : cluster [DBG] 3.7 scrub ok 2024-02-12T16:12:21.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:12:21.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:21.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:21.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.6 reset_purged_snaps_last 2024-02-12T16:12:21.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:21.526 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:21.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:21.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:21.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:21 smithi123 bash[19119]: cluster 2024-02-12T16:12:20.435528+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub starts 2024-02-12T16:12:21.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:21 smithi123 bash[19119]: cluster 2024-02-12T16:12:20.594923+0000 osd.3 (osd.3) 36 : cluster [DBG] 3.7 scrub ok 2024-02-12T16:12:21.654 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:21 smithi121 bash[18489]: cluster 2024-02-12T16:12:20.435528+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub starts 2024-02-12T16:12:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:21 smithi121 bash[18489]: cluster 2024-02-12T16:12:20.594923+0000 osd.3 (osd.3) 36 : cluster [DBG] 3.7 scrub ok 2024-02-12T16:12:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:21.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:21.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:21.793 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:21.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:21.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:21.924 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:21.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:22.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:22.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:22.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:22.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:22.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:22.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:22.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:22.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:22.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:22.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:22.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:22 smithi169 bash[19604]: cluster 2024-02-12T16:12:20.480089+0000 osd.7 (osd.7) 51 : cluster [DBG] 3.d scrub starts 2024-02-12T16:12:22.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:22 smithi169 bash[19604]: cluster 2024-02-12T16:12:21.326458+0000 osd.6 (osd.6) 35 : cluster [DBG] 3.a scrub ok 2024-02-12T16:12:22.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:22 smithi169 bash[19604]: cluster 2024-02-12T16:12:21.672684+0000 mgr.y (mgr.14146) 809 : cluster [DBG] pgmap v1027: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-02-12T16:12:22.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:22.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:22.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:22.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:22.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:22 smithi123 bash[19119]: cluster 2024-02-12T16:12:20.480089+0000 osd.7 (osd.7) 51 : cluster [DBG] 3.d scrub starts 2024-02-12T16:12:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:22 smithi123 bash[19119]: cluster 2024-02-12T16:12:21.326458+0000 osd.6 (osd.6) 35 : cluster [DBG] 3.a scrub ok 2024-02-12T16:12:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:22 smithi123 bash[19119]: cluster 2024-02-12T16:12:21.672684+0000 mgr.y (mgr.14146) 809 : cluster [DBG] pgmap v1027: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-02-12T16:12:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:22 smithi121 bash[18489]: cluster 2024-02-12T16:12:20.480089+0000 osd.7 (osd.7) 51 : cluster [DBG] 3.d scrub starts 2024-02-12T16:12:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:22 smithi121 bash[18489]: cluster 2024-02-12T16:12:21.326458+0000 osd.6 (osd.6) 35 : cluster [DBG] 3.a scrub ok 2024-02-12T16:12:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:22 smithi121 bash[18489]: cluster 2024-02-12T16:12:21.672684+0000 mgr.y (mgr.14146) 809 : cluster [DBG] pgmap v1027: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-02-12T16:12:22.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:22.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:22.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:22.873 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:22.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:22.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:23.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:23.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:23.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:23.275 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:23.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:23.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:23.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:23.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:23.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:23.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:23 smithi123 bash[19119]: cluster 2024-02-12T16:12:21.669839+0000 osd.11 (osd.11) 33 : cluster [DBG] 3.e scrub starts 2024-02-12T16:12:23.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:23 smithi123 bash[19119]: cluster 2024-02-12T16:12:21.740712+0000 osd.7 (osd.7) 52 : cluster [DBG] 3.d scrub ok 2024-02-12T16:12:23.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:23 smithi123 bash[19119]: cluster 2024-02-12T16:12:22.373567+0000 osd.11 (osd.11) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:23.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:23 smithi123 bash[19119]: cluster 2024-02-12T16:12:22.373621+0000 osd.11 (osd.11) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:23.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:23 smithi123 bash[19119]: cluster 2024-02-12T16:12:22.428204+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e316: 12 total, 12 up, 12 in 2024-02-12T16:12:23.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:23.689 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:23.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:23.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:23 smithi169 bash[19604]: cluster 2024-02-12T16:12:21.669839+0000 osd.11 (osd.11) 33 : cluster [DBG] 3.e scrub starts 2024-02-12T16:12:23.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:23 smithi169 bash[19604]: cluster 2024-02-12T16:12:21.740712+0000 osd.7 (osd.7) 52 : cluster [DBG] 3.d scrub ok 2024-02-12T16:12:23.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:23 smithi169 bash[19604]: cluster 2024-02-12T16:12:22.373567+0000 osd.11 (osd.11) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:23.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:23 smithi169 bash[19604]: cluster 2024-02-12T16:12:22.373621+0000 osd.11 (osd.11) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:23.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:23 smithi169 bash[19604]: cluster 2024-02-12T16:12:22.428204+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e316: 12 total, 12 up, 12 in 2024-02-12T16:12:23.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:23 smithi121 bash[18489]: cluster 2024-02-12T16:12:21.669839+0000 osd.11 (osd.11) 33 : cluster [DBG] 3.e scrub starts 2024-02-12T16:12:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:23 smithi121 bash[18489]: cluster 2024-02-12T16:12:21.740712+0000 osd.7 (osd.7) 52 : cluster [DBG] 3.d scrub ok 2024-02-12T16:12:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:23 smithi121 bash[18489]: cluster 2024-02-12T16:12:22.373567+0000 osd.11 (osd.11) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:23 smithi121 bash[18489]: cluster 2024-02-12T16:12:22.373621+0000 osd.11 (osd.11) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:23 smithi121 bash[18489]: cluster 2024-02-12T16:12:22.428204+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e316: 12 total, 12 up, 12 in 2024-02-12T16:12:23.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:23.957 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:23.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:24.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:24.088 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:24.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:24.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:24.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:24.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:24.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:24.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:24.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:24.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:24.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:24.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:24.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:24.606 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:24 smithi169 bash[19604]: cluster 2024-02-12T16:12:22.911087+0000 osd.11 (osd.11) 36 : cluster [DBG] 3.e scrub ok 2024-02-12T16:12:24.606 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:24 smithi169 bash[19604]: cluster 2024-02-12T16:12:23.463262+0000 osd.3 (osd.3) 37 : cluster [DBG] 3.b scrub starts 2024-02-12T16:12:24.606 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:24 smithi169 bash[19604]: cluster 2024-02-12T16:12:23.635161+0000 osd.11 (osd.11) 37 : cluster [DBG] 3.8 scrub starts 2024-02-12T16:12:24.607 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:24 smithi169 bash[19604]: cluster 2024-02-12T16:12:23.673110+0000 mgr.y (mgr.14146) 810 : cluster [DBG] pgmap v1029: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:12:24.621 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:24.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:24.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:24 smithi121 bash[18489]: cluster 2024-02-12T16:12:22.911087+0000 osd.11 (osd.11) 36 : cluster [DBG] 3.e scrub ok 2024-02-12T16:12:24.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:24 smithi121 bash[18489]: cluster 2024-02-12T16:12:23.463262+0000 osd.3 (osd.3) 37 : cluster [DBG] 3.b scrub starts 2024-02-12T16:12:24.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:24 smithi121 bash[18489]: cluster 2024-02-12T16:12:23.635161+0000 osd.11 (osd.11) 37 : cluster [DBG] 3.8 scrub starts 2024-02-12T16:12:24.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:24 smithi121 bash[18489]: cluster 2024-02-12T16:12:23.673110+0000 mgr.y (mgr.14146) 810 : cluster [DBG] pgmap v1029: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:12:24.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:24.732 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:24 smithi123 bash[19119]: cluster 2024-02-12T16:12:22.911087+0000 osd.11 (osd.11) 36 : cluster [DBG] 3.e scrub ok 2024-02-12T16:12:24.732 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:24 smithi123 bash[19119]: cluster 2024-02-12T16:12:23.463262+0000 osd.3 (osd.3) 37 : cluster [DBG] 3.b scrub starts 2024-02-12T16:12:24.732 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:24 smithi123 bash[19119]: cluster 2024-02-12T16:12:23.635161+0000 osd.11 (osd.11) 37 : cluster [DBG] 3.8 scrub starts 2024-02-12T16:12:24.732 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:24 smithi123 bash[19119]: cluster 2024-02-12T16:12:23.673110+0000 mgr.y (mgr.14146) 810 : cluster [DBG] pgmap v1029: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:12:24.750 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:24.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:24.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:24.879 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:24.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:24.982 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:24.982 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:24.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:25.011 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:25.120 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:25.144 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:25.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:25.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:25.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:25.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:25.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:25.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:25.594 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:25 smithi121 bash[18489]: cluster 2024-02-12T16:12:23.472555+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:12:25.594 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:25 smithi121 bash[18489]: cluster 2024-02-12T16:12:23.505161+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:12:25.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:25.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:25.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:25 smithi169 bash[19604]: cluster 2024-02-12T16:12:23.472555+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:12:25.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:25 smithi169 bash[19604]: cluster 2024-02-12T16:12:23.505161+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:12:25.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:25 smithi169 bash[19604]: cluster 2024-02-12T16:12:24.604592+0000 osd.3 (osd.3) 38 : cluster [DBG] 3.b scrub ok 2024-02-12T16:12:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:25.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:25.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:25 smithi123 bash[19119]: cluster 2024-02-12T16:12:23.472555+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:12:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:25 smithi123 bash[19119]: cluster 2024-02-12T16:12:23.505161+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:12:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:25 smithi123 bash[19119]: cluster 2024-02-12T16:12:24.604592+0000 osd.3 (osd.3) 38 : cluster [DBG] 3.b scrub ok 2024-02-12T16:12:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:25 smithi121 bash[18489]: cluster 2024-02-12T16:12:24.604592+0000 osd.3 (osd.3) 38 : cluster [DBG] 3.b scrub ok 2024-02-12T16:12:25.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:25.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:25.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:25.997 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:25.998 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:26.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:26.114 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:26.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:26.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:26.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:26.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:26.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:26.389 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:26.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:26.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:26.527 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:26.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:26.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:26.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:26 smithi123 bash[19119]: cluster 2024-02-12T16:12:25.221662+0000 osd.11 (osd.11) 38 : cluster [DBG] 3.8 scrub ok 2024-02-12T16:12:26.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:26 smithi123 bash[19119]: cluster 2024-02-12T16:12:25.673508+0000 mgr.y (mgr.14146) 811 : cluster [DBG] pgmap v1030: 17 pgs: 17 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s 2024-02-12T16:12:26.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:26.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:26.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:26 smithi169 bash[19604]: cluster 2024-02-12T16:12:25.221662+0000 osd.11 (osd.11) 38 : cluster [DBG] 3.8 scrub ok 2024-02-12T16:12:26.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:26 smithi169 bash[19604]: cluster 2024-02-12T16:12:25.673508+0000 mgr.y (mgr.14146) 811 : cluster [DBG] pgmap v1030: 17 pgs: 17 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s 2024-02-12T16:12:26.795 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:26.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:26.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:26.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:26 smithi121 bash[18489]: cluster 2024-02-12T16:12:25.221662+0000 osd.11 (osd.11) 38 : cluster [DBG] 3.8 scrub ok 2024-02-12T16:12:26.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:26 smithi121 bash[18489]: cluster 2024-02-12T16:12:25.673508+0000 mgr.y (mgr.14146) 811 : cluster [DBG] pgmap v1030: 17 pgs: 17 active+clean; 3.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s 2024-02-12T16:12:26.928 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:26.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:27.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:27.062 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:27.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:27.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:27.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:27.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:27.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:27.325 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:27.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:27.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:27.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:27.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:27.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:27.570 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:27 smithi169 bash[19604]: cluster 2024-02-12T16:12:26.421095+0000 osd.3 (osd.3) 39 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:12:27.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:27.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:27.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:27.699 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:27 smithi123 bash[19119]: cluster 2024-02-12T16:12:26.421095+0000 osd.3 (osd.3) 39 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:12:27.722 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:27.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:27.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:27.854 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:27.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:27.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:27 smithi121 bash[18489]: cluster 2024-02-12T16:12:26.421095+0000 osd.3 (osd.3) 39 : cluster [DBG] 3.2 scrub starts 2024-02-12T16:12:27.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:28.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:28.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:28.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:28.284 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:28.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:28.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:28.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:28.543 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:28 smithi121 bash[18489]: cluster 2024-02-12T16:12:27.309004+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:12:28.543 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:28 smithi121 bash[18489]: cluster 2024-02-12T16:12:27.531668+0000 osd.1 (osd.1) 43 : cluster [DBG] 3.4 scrub starts 2024-02-12T16:12:28.543 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:28 smithi121 bash[18489]: cluster 2024-02-12T16:12:27.673882+0000 mgr.y (mgr.14146) 812 : cluster [DBG] pgmap v1031: 17 pgs: 17 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-02-12T16:12:28.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:28.650 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:28.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:28 smithi169 bash[19604]: cluster 2024-02-12T16:12:27.309004+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:12:28.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:28 smithi169 bash[19604]: cluster 2024-02-12T16:12:27.531668+0000 osd.1 (osd.1) 43 : cluster [DBG] 3.4 scrub starts 2024-02-12T16:12:28.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:28 smithi169 bash[19604]: cluster 2024-02-12T16:12:27.673882+0000 mgr.y (mgr.14146) 812 : cluster [DBG] pgmap v1031: 17 pgs: 17 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-02-12T16:12:28.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:28.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:28.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:28 smithi123 bash[19119]: cluster 2024-02-12T16:12:27.309004+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.2 scrub ok 2024-02-12T16:12:28.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:28 smithi123 bash[19119]: cluster 2024-02-12T16:12:27.531668+0000 osd.1 (osd.1) 43 : cluster [DBG] 3.4 scrub starts 2024-02-12T16:12:28.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:28 smithi123 bash[19119]: cluster 2024-02-12T16:12:27.673882+0000 mgr.y (mgr.14146) 812 : cluster [DBG] pgmap v1031: 17 pgs: 17 active+clean; 3.1 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-02-12T16:12:28.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:28.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:28.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:29.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:29.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:29.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:29.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:29.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:29.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:29.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:29.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:29.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:29.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:29.455 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:29 smithi121 bash[18489]: cluster 2024-02-12T16:12:28.396247+0000 osd.1 (osd.1) 44 : cluster [DBG] 3.4 scrub ok 2024-02-12T16:12:29.484 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:29.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:29.588 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:29.595 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:29 smithi123 bash[19119]: cluster 2024-02-12T16:12:28.396247+0000 osd.1 (osd.1) 44 : cluster [DBG] 3.4 scrub ok 2024-02-12T16:12:29.618 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:29.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:29.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:29 smithi169 bash[19604]: cluster 2024-02-12T16:12:28.396247+0000 osd.1 (osd.1) 44 : cluster [DBG] 3.4 scrub ok 2024-02-12T16:12:29.754 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:29.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:29.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:29.885 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:29.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:29.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:30.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:30.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:30.139 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:30.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:30.276 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:30.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:30.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:30.521 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:30.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:30 smithi121 bash[18489]: cluster 2024-02-12T16:12:29.674245+0000 mgr.y (mgr.14146) 813 : cluster [DBG] pgmap v1032: 17 pgs: 17 active+clean; 3.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-02-12T16:12:30.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:30.696 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:30.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:30.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:30.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:30.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:30 smithi123 bash[19119]: cluster 2024-02-12T16:12:29.674245+0000 mgr.y (mgr.14146) 813 : cluster [DBG] pgmap v1032: 17 pgs: 17 active+clean; 3.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-02-12T16:12:30.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:30.972 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:30.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:31.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:30 smithi169 bash[19604]: cluster 2024-02-12T16:12:29.674245+0000 mgr.y (mgr.14146) 813 : cluster [DBG] pgmap v1032: 17 pgs: 17 active+clean; 3.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-02-12T16:12:31.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:31.107 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:31.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:31.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:31.240 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:31.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:31.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:31.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:31.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:31.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:31 smithi169 bash[19604]: cluster 2024-02-12T16:12:30.246000+0000 osd.9 (osd.9) 32 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:31.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:31 smithi169 bash[19604]: cluster 2024-02-12T16:12:30.246072+0000 osd.9 (osd.9) 33 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:31.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:31 smithi169 bash[19604]: cluster 2024-02-12T16:12:30.462051+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e317: 12 total, 12 up, 12 in 2024-02-12T16:12:31.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:31.522 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:31.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:31.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:31.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:31 smithi121 bash[18489]: cluster 2024-02-12T16:12:30.246000+0000 osd.9 (osd.9) 32 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:31 smithi121 bash[18489]: cluster 2024-02-12T16:12:30.246072+0000 osd.9 (osd.9) 33 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:31 smithi121 bash[18489]: cluster 2024-02-12T16:12:30.462051+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e317: 12 total, 12 up, 12 in 2024-02-12T16:12:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:31.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:31.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:31.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:31 smithi123 bash[19119]: cluster 2024-02-12T16:12:30.246000+0000 osd.9 (osd.9) 32 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:31.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:31 smithi123 bash[19119]: cluster 2024-02-12T16:12:30.246072+0000 osd.9 (osd.9) 33 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:31.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:31 smithi123 bash[19119]: cluster 2024-02-12T16:12:30.462051+0000 mon.a (mon.0) 1281 : cluster [DBG] osdmap e317: 12 total, 12 up, 12 in 2024-02-12T16:12:31.787 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:31.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:31.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:31.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:32.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:32.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:32.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:32.211 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:32.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:32.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:32.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:32.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:32.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:32.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:32.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:32.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:32.610 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:32.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:32 smithi121 bash[18489]: cluster 2024-02-12T16:12:31.148072+0000 osd.11 (osd.11) 39 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:12:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:32 smithi121 bash[18489]: cluster 2024-02-12T16:12:31.674643+0000 mgr.y (mgr.14146) 814 : cluster [DBG] pgmap v1034: 17 pgs: 17 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-02-12T16:12:32.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:32.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:32 smithi123 bash[19119]: cluster 2024-02-12T16:12:31.148072+0000 osd.11 (osd.11) 39 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:12:32.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:32 smithi123 bash[19119]: cluster 2024-02-12T16:12:31.674643+0000 mgr.y (mgr.14146) 814 : cluster [DBG] pgmap v1034: 17 pgs: 17 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-02-12T16:12:32.776 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:32.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:32.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:32 smithi169 bash[19604]: cluster 2024-02-12T16:12:31.148072+0000 osd.11 (osd.11) 39 : cluster [DBG] 3.0 scrub starts 2024-02-12T16:12:32.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:32 smithi169 bash[19604]: cluster 2024-02-12T16:12:31.674643+0000 mgr.y (mgr.14146) 814 : cluster [DBG] pgmap v1034: 17 pgs: 17 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-02-12T16:12:32.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:32.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:32.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:33.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:33.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:33.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:33.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:33.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:33.218 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:12:33.248 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:12:33.284 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:33.293 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:12:33.309 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:33.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:33.318 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:12:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:12:33.376 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:12:33.407 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:12:33.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:33.449 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:33.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:33.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:12:33.482 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:12:33.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:33 smithi123 bash[19119]: cluster 2024-02-12T16:12:32.346168+0000 osd.11 (osd.11) 40 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:12:33.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:33 smithi123 bash[19119]: audit 2024-02-12T16:12:33.177914+0000 mon.c (mon.1) 32 : audit [DBG] from='client.? v1:172.21.15.121:0/2962036532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:33.523 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:12:33.549 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:12:33.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:33.556 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:33 smithi121 bash[18489]: cluster 2024-02-12T16:12:32.346168+0000 osd.11 (osd.11) 40 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:12:33.556 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:33 smithi121 bash[18489]: audit 2024-02-12T16:12:33.177914+0000 mon.c (mon.1) 32 : audit [DBG] from='client.? v1:172.21.15.121:0/2962036532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:33.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:33.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:33.581 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:12:33.587 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.587 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.587 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.587 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.587 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.588 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.588 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.588 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.588 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.588 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.588 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.588 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:12:33.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:33.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:33.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:33.775 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:155348 2.e 2:77010cd8:::benchmark_data_smithi169_42458_object155347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e173)", 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:05:32.818373+0000", 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 420.94259193300002, 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27370472200000001, 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.776 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 155348 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818373+0000", 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818373+0000", 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818371+0000", 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8388999999999997e-05 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818460+0000", 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1164e-05 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818471+0000", 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3570000000000003e-06 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818477+0000", 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5042e-05 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818522+0000", 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012639199999999999 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818649+0000", 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013983600000000001 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.818788+0000", 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27309921599999998 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.091888+0000", 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4579999999999994e-05 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.091962+0000", 2024-02-12T16:12:33.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4593999999999999e-05 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.092027+0000", 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1322000000000001e-05 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.092078+0000", 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27370472200000001 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:155351 2.1 2:8766990e:::benchmark_data_smithi169_42458_object155350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e173)", 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:05:32.821590+0000", 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 420.93937563399999, 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27131618699999999, 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 155351 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821590+0000", 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821590+0000", 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821588+0000", 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101474 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821689+0000", 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0772e-05 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821700+0000", 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3640000000000004e-06 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821706+0000", 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0037999999999994e-05 2024-02-12T16:12:33.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821796+0000", 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016310700000000001 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821959+0000", 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019034299999999999 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.822150+0000", 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27048931999999998 2024-02-12T16:12:33.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.092639+0000", 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3046000000000005e-05 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.092712+0000", 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116884 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.092829+0000", 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6883000000000002e-05 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.092906+0000", 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27131618699999999 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:155356 2.e 2:71cd6540:::benchmark_data_smithi169_42458_object155355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e173)", 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:05:32.821965+0000", 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 420.93900010999999, 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28154369600000001, 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.787 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 155356 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821965+0000", 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821965+0000", 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.821964+0000", 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013129099999999999 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.822095+0000", 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7260000000000002e-06 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.822103+0000", 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4410000000000001e-06 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.822108+0000", 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025084970000000002 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.824617+0000", 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013396 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.824751+0000", 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019750400000000001 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.824948+0000", 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.267001936 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.091950+0000", 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01147106 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.103421+0000", 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0083000000000001e-05 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.103471+0000", 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7625e-05 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.103509+0000", 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28154369600000001 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.792 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:155363 2.1 2:86b1f2cb:::benchmark_data_smithi169_42458_object155362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e173)", 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:05:32.826437+0000", 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 420.93452871800002, 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27535085199999998, 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 155363 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.826437+0000", 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.826437+0000", 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.826434+0000", 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012352399999999999 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:32.826558+0000", 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26023603099999998 2024-02-12T16:12:33.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.086794+0000", 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3006999999999998e-05 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.086827+0000", 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2081000000000001e-05 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.086919+0000", 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024308399999999999 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.087162+0000", 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050588100000000002 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.087668+0000", 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053531969999999996 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.093021+0000", 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086674560000000005 2024-02-12T16:12:33.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.101688+0000", 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7778000000000001e-05 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.101746+0000", 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1326999999999999e-05 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:05:33.101787+0000", 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27535085199999998 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333598 2.13 2:cbe7c818:::benchmark_data_smithi169_42458_object160564:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320134+0000", 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44083101299998, 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30529220600000001, 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333598 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320134+0000", 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320134+0000", 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320131+0000", 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9749999999999996e-06 2024-02-12T16:12:33.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320140+0000", 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.153e-05 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320152+0000", 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0083e-05 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320162+0000", 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4074999999999998e-05 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320236+0000", 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5379999999999997e-06 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320244+0000", 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25565338700000001 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.575897+0000", 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6471999999999994e-05 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.575993+0000", 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073284499999999998 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.576726+0000", 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092272810000000004 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.585954+0000", 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.038980288000000002 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.624934+0000", 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2793000000000002e-05 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.624977+0000", 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000223682 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625200+0000", 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6697000000000002e-05 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625267+0000", 2024-02-12T16:12:33.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159469 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625427+0000", 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30529220600000001 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333629 2.13 2:cd1f890f:::benchmark_data_smithi169_42458_object160595:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320178+0000", 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44078712499999, 2024-02-12T16:12:33.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30558835600000001, 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333629 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320178+0000", 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320178+0000", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320178+0000", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1640000000000001e-06 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320181+0000", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5250000000000005e-06 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320189+0000", 2024-02-12T16:12:33.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0819999999999998e-06 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320194+0000", 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6761999999999994e-05 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320281+0000", 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9079999999999995e-06 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320289+0000", 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25999638000000003 2024-02-12T16:12:33.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.580285+0000", 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014796899999999999 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.580433+0000", 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055269800000000001 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.580986+0000", 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011103887999999999 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.592090+0000", 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.032943795999999997 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625034+0000", 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5528999999999997e-05 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625079+0000", 2024-02-12T16:12:33.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057275000000000002 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625652+0000", 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6591000000000003e-05 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625689+0000", 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7997999999999994e-05 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625767+0000", 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30558835600000001 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333638 2.13 2:ca1302b1:::benchmark_data_smithi169_42458_object160604:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320200+0000", 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.440764989, 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30616000500000001, 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333638 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320200+0000", 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320200+0000", 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320200+0000", 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8029999999999998e-06 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320203+0000", 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6099999999999997e-06 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320209+0000", 2024-02-12T16:12:33.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4950000000000002e-06 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320214+0000", 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109201 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320323+0000", 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.618e-06 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320329+0000", 2024-02-12T16:12:33.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26315843500000002 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.583488+0000", 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013168699999999999 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.583619+0000", 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040328800000000002 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.584023+0000", 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013904888000000001 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.597928+0000", 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027197868 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625126+0000", 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069177900000000001 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625817+0000", 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033267399999999998 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626150+0000", 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8230999999999999e-05 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626208+0000", 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000152118 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626360+0000", 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30616000500000001 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333648 2.13 2:cf30b99f:::benchmark_data_smithi169_42458_object160614:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320240+0000", 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44072528800001, 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30633011599999999, 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.818 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333648 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320240+0000", 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320240+0000", 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320239+0000", 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3800000000000004e-06 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320243+0000", 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7839999999999992e-06 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320252+0000", 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6330000000000001e-06 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320258+0000", 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101399 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320359+0000", 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.832e-06 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320366+0000", 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.265619885 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.585986+0000", 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5099999999999995e-05 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.586071+0000", 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034106499999999999 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.586412+0000", 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011526362 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.597939+0000", 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027216223000000001 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625155+0000", 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070012999999999998 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625855+0000", 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00053268300000000005 2024-02-12T16:12:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626388+0000", 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3597000000000002e-05 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626441+0000", 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012897000000000001 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626570+0000", 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30633011599999999 2024-02-12T16:12:33.824 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333679 2.13 2:c8fb6db0:::benchmark_data_smithi169_42458_object160645:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320275+0000", 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44069033599999, 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.306845284, 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333679 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320275+0000", 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320275+0000", 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320274+0000", 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8654000000000001e-05 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320293+0000", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5066999999999999e-05 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320318+0000", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9519999999999997e-06 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320325+0000", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0159000000000002e-05 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320415+0000", 2024-02-12T16:12:33.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7759999999999997e-06 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320423+0000", 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.267924304 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.588347+0000", 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013517600000000001 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.588482+0000", 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00044461300000000001 2024-02-12T16:12:33.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.588927+0000", 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016130983000000002 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.605058+0000", 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020419722000000001 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625477+0000", 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042637300000000002 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625904+0000", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000947755 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626852+0000", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8516000000000005e-05 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626930+0000", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019013499999999999 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627120+0000", 2024-02-12T16:12:33.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.306845284 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333697 2.13 2:cbcf5cbf:::benchmark_data_smithi169_42458_object160663:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320332+0000", 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44063308699998, 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30712144200000002, 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333697 2024-02-12T16:12:33.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320332+0000", 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320332+0000", 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320331+0000", 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8559999999999998e-06 2024-02-12T16:12:33.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320336+0000", 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5320000000000001e-06 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320344+0000", 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.677e-06 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320349+0000", 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119193 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320468+0000", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0299999999999999e-06 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320475+0000", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27166524199999997 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.592140+0000", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105315 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.592245+0000", 2024-02-12T16:12:33.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033312100000000002 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.592578+0000", 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012491095000000001 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.605069+0000", 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020453510000000001 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625523+0000", 2024-02-12T16:12:33.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042186099999999999 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625945+0000", 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012327760000000001 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627177+0000", 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7760000000000002e-05 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627245+0000", 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020848899999999999 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627454+0000", 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30712144200000002 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333724 2.13 2:cad53bf2:::benchmark_data_smithi169_42458_object160690:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320355+0000", 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.440609903, 2024-02-12T16:12:33.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31093699699999999, 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333724 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320355+0000", 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320355+0000", 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320355+0000", 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.014e-06 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320359+0000", 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.182e-06 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320366+0000", 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4269999999999999e-06 2024-02-12T16:12:33.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320371+0000", 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136365 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320508+0000", 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9179999999999999e-06 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320514+0000", 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27395343599999999 2024-02-12T16:12:33.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.594467+0000", 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013067499999999999 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.594598+0000", 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039808200000000001 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.594996+0000", 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015882358999999999 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.610878+0000", 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014679938999999999 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625558+0000", 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043463200000000001 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625993+0000", 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001880816 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627873+0000", 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003233358 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631107+0000", 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018556700000000001 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631292+0000", 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31093699699999999 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333772 2.13 2:ce7cd7d4:::benchmark_data_smithi169_42458_object160738:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320406+0000", 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44055910100002, 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31110088200000002, 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333772 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320406+0000", 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320406+0000", 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320405+0000", 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5709999999999997e-06 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320410+0000", 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5500000000000002e-06 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320416+0000", 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8430000000000003e-06 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320420+0000", 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012219800000000001 2024-02-12T16:12:33.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320542+0000", 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5119999999999999e-06 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320547+0000", 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27743261600000002 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.597980+0000", 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6322999999999993e-05 2024-02-12T16:12:33.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.598066+0000", 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031108900000000002 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.598378+0000", 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012511471 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.610889+0000", 2024-02-12T16:12:33.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014713352000000001 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625602+0000", 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046900799999999998 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626071+0000", 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053054540000000002 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631377+0000", 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4897000000000001e-05 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631412+0000", 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5398999999999999e-05 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631507+0000", 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31110088200000002 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.850 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333803 2.13 2:ccff1cc5:::benchmark_data_smithi169_42458_object160769:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320423+0000", 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44054247700001, 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31139097100000002, 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333803 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320423+0000", 2024-02-12T16:12:33.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320423+0000", 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320422+0000", 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.317e-06 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320425+0000", 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2229999999999999e-06 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320428+0000", 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0080000000000004e-06 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320432+0000", 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013611600000000001 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320568+0000", 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7599999999999999e-06 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320574+0000", 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27957576000000001 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.600150+0000", 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133294 2024-02-12T16:12:33.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.600283+0000", 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052408299999999995 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.600807+0000", 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015812577000000001 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.616620+0000", 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094067179999999997 2024-02-12T16:12:33.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626026+0000", 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7647999999999999e-05 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626104+0000", 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056064019999999999 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631710+0000", 2024-02-12T16:12:33.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7885000000000002e-05 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631738+0000", 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5586000000000004e-05 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631814+0000", 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31139097100000002 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333827 2.13 2:c8828e7d:::benchmark_data_smithi169_42458_object160793:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320472+0000", 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44049290300001, 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.311474262, 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333827 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320472+0000", 2024-02-12T16:12:33.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320472+0000", 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320472+0000", 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2820000000000001e-06 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320475+0000", 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4280000000000001e-06 2024-02-12T16:12:33.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320481+0000", 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8570000000000003e-06 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320486+0000", 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110846 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320597+0000", 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4970000000000001e-06 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320603+0000", 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451615600000002 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.605119+0000", 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0580000000000004e-05 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.605199+0000", 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033599900000000002 2024-02-12T16:12:33.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.605535+0000", 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011095211000000001 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.616631+0000", 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010039596 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626670+0000", 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012561899999999999 2024-02-12T16:12:33.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626796+0000", 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050436760000000004 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631840+0000", 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6825000000000002e-05 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631866+0000", 2024-02-12T16:12:33.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0326000000000003e-05 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631947+0000", 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.311474262 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333834 2.13 2:ced0cf02:::benchmark_data_smithi169_42458_object160800:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320491+0000", 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44047452799998, 2024-02-12T16:12:33.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31823912300000001, 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333834 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320491+0000", 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320491+0000", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320490+0000", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.942e-06 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320493+0000", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2480000000000004e-06 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320498+0000", 2024-02-12T16:12:33.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8139999999999998e-06 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320502+0000", 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012472000000000001 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320627+0000", 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6570000000000002e-06 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320633+0000", 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28674535600000001 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.607378+0000", 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013286999999999999 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.607511+0000", 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039378599999999997 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.607905+0000", 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014547387 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.622452+0000", 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042865630000000002 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.626739+0000", 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083049999999999997 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627569+0000", 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010947468 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.638517+0000", 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1810999999999995e-05 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.638578+0000", 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015157500000000001 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.638730+0000", 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31823912300000001 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333859 2.13 2:ce0b7a8e:::benchmark_data_smithi169_42458_object160825:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320506+0000", 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.4404591, 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31858125199999998, 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333859 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320506+0000", 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320506+0000", 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320506+0000", 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9500000000000001e-06 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320509+0000", 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4280000000000004e-06 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320514+0000", 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8489999999999997e-06 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320519+0000", 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000138947 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320658+0000", 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.682e-06 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320664+0000", 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29026512300000001 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.610929+0000", 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114109 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.611043+0000", 2024-02-12T16:12:33.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031944100000000001 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.611362+0000", 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011101463000000001 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.622464+0000", 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050518009999999999 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627515+0000", 2024-02-12T16:12:33.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000161278 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627677+0000", 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011135258 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.638812+0000", 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102081 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.638914+0000", 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000173394 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639087+0000", 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31858125199999998 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333864 2.13 2:ccfb3805:::benchmark_data_smithi169_42458_object160830:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320550+0000", 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44041542999997, 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.318882583, 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333864 2024-02-12T16:12:33.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320550+0000", 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320550+0000", 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320549+0000", 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6849999999999999e-06 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320553+0000", 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0079999999999999e-06 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320559+0000", 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1779999999999996e-06 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320563+0000", 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120077 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320683+0000", 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5219999999999998e-06 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320689+0000", 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29247489100000001 2024-02-12T16:12:33.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.613164+0000", 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013211399999999999 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.613296+0000", 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039959799999999998 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.613695+0000", 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011042658 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.624738+0000", 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002890358 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627628+0000", 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000150672 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627779+0000", 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011486645 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639266+0000", 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1391999999999999e-05 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639317+0000", 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115462 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639432+0000", 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.318882583 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333898 2.13 2:ca52910f:::benchmark_data_smithi169_42458_object160864:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320568+0000", 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44039776800003, 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31900684000000001, 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333898 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320568+0000", 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320568+0000", 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320567+0000", 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7879999999999998e-06 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320570+0000", 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320575+0000", 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5619999999999999e-06 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320578+0000", 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000130309 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320708+0000", 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6779999999999997e-06 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320714+0000", 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.295955517 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.616670+0000", 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106944 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.616777+0000", 2024-02-12T16:12:33.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031791099999999999 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.617094+0000", 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0076541949999999999 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.624749+0000", 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002981044 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627730+0000", 2024-02-12T16:12:33.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038132040000000002 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631543+0000", 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007931858 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639475+0000", 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1053000000000002e-05 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639506+0000", 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8596000000000005e-05 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639574+0000", 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31900684000000001 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333902 2.13 2:ca8165b3:::benchmark_data_smithi169_42458_object160868:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.892 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320594+0000", 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.44037111400002, 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31917606900000001, 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333902 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320594+0000", 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320594+0000", 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320594+0000", 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7800000000000001e-06 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320596+0000", 2024-02-12T16:12:33.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.023e-06 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320601+0000", 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7840000000000001e-06 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320605+0000", 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012867 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320734+0000", 2024-02-12T16:12:33.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5969999999999999e-06 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320739+0000", 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29818779299999998 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.618927+0000", 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000146661 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.619074+0000", 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042625900000000001 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.619500+0000", 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054458909999999996 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.624946+0000", 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002880512 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.627827+0000", 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037383770000000002 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631565+0000", 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080498310000000003 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639615+0000", 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6906000000000001e-05 2024-02-12T16:12:33.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639662+0000", 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010862 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.639770+0000", 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31917606900000001 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.899 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.14811.0:333938 2.13 2:cee2fb01:::benchmark_data_smithi169_42458_object160904:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e228)", 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:08:10.320624+0000", 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 263.440341386, 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.320981193, 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.14811", 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 333938 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320624+0000", 2024-02-12T16:12:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320624+0000", 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320623+0000", 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.847e-06 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320626+0000", 2024-02-12T16:12:33.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.091e-06 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320631+0000", 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5769999999999998e-06 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320635+0000", 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012174 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320757+0000", 2024-02-12T16:12:33.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6779999999999997e-06 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.320762+0000", 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30173741199999998 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.622500+0000", 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6440999999999995e-05 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:12:33.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.622576+0000", 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033252500000000002 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,5,9", 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.622909+0000", 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021363760000000002 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.625045+0000", 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066293280000000003 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631674+0000", 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8342000000000002e-05 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.631693+0000", 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097050980000000005 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.641398+0000", 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3774000000000003e-05 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:12:33.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.641462+0000", 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014361199999999999 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:08:10.641605+0000", 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.320981193 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:12:33.906 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:12:33.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:33.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:33 smithi169 bash[19604]: cluster 2024-02-12T16:12:32.346168+0000 osd.11 (osd.11) 40 : cluster [DBG] 3.0 scrub ok 2024-02-12T16:12:33.908 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:33 smithi169 bash[19604]: audit 2024-02-12T16:12:33.177914+0000 mon.c (mon.1) 32 : audit [DBG] from='client.? v1:172.21.15.121:0/2962036532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:33.934 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:33.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:34.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:34.059 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:34.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:34.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:34.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:34.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:34.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:34.324 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:34.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:34.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:34.462 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:34.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:34.501 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:12:34.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:34 smithi169 bash[19604]: cluster 2024-02-12T16:12:33.017526+0000 osd.8 (osd.8) 33 : cluster [DBG] 3.9 scrub starts 2024-02-12T16:12:34.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:34 smithi169 bash[19604]: cluster 2024-02-12T16:12:33.508957+0000 osd.8 (osd.8) 34 : cluster [DBG] 3.9 scrub ok 2024-02-12T16:12:34.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:34 smithi169 bash[19604]: cluster 2024-02-12T16:12:33.675096+0000 mgr.y (mgr.14146) 815 : cluster [DBG] pgmap v1035: 17 pgs: 17 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:12:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:34 smithi169 bash[19604]: audit 2024-02-12T16:12:34.045821+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? v1:172.21.15.121:0/2361764759' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:34.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:34.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:34.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:34.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:34.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:34.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:34.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:34.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:34.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:34.966 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:34 smithi121 bash[18489]: cluster 2024-02-12T16:12:33.017526+0000 osd.8 (osd.8) 33 : cluster [DBG] 3.9 scrub starts 2024-02-12T16:12:34.966 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:34 smithi121 bash[18489]: cluster 2024-02-12T16:12:33.508957+0000 osd.8 (osd.8) 34 : cluster [DBG] 3.9 scrub ok 2024-02-12T16:12:34.966 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:34 smithi121 bash[18489]: cluster 2024-02-12T16:12:33.675096+0000 mgr.y (mgr.14146) 815 : cluster [DBG] pgmap v1035: 17 pgs: 17 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:12:34.966 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:34 smithi121 bash[18489]: audit 2024-02-12T16:12:34.045821+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? v1:172.21.15.121:0/2361764759' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:34.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:34.999 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:34 smithi123 bash[19119]: cluster 2024-02-12T16:12:33.017526+0000 osd.8 (osd.8) 33 : cluster [DBG] 3.9 scrub starts 2024-02-12T16:12:34.999 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:34 smithi123 bash[19119]: cluster 2024-02-12T16:12:33.508957+0000 osd.8 (osd.8) 34 : cluster [DBG] 3.9 scrub ok 2024-02-12T16:12:34.999 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:34 smithi123 bash[19119]: cluster 2024-02-12T16:12:33.675096+0000 mgr.y (mgr.14146) 815 : cluster [DBG] pgmap v1035: 17 pgs: 17 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:12:34.999 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:34 smithi123 bash[19119]: audit 2024-02-12T16:12:34.045821+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? v1:172.21.15.121:0/2361764759' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:12:35.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:35.128 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:35.157 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:35.261 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:35.290 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:35.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:35.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:35.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:35.535 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:35 smithi123 bash[19119]: cluster 2024-02-12T16:12:33.746865+0000 osd.0 (osd.0) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:35.535 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:35 smithi123 bash[19119]: cluster 2024-02-12T16:12:33.746915+0000 osd.0 (osd.0) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:35.536 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:35 smithi123 bash[19119]: cluster 2024-02-12T16:12:34.477470+0000 mon.a (mon.0) 1283 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:12:35.536 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:35 smithi123 bash[19119]: cluster 2024-02-12T16:12:34.477495+0000 mon.a (mon.0) 1284 : cluster [INF] Cluster is now healthy 2024-02-12T16:12:35.536 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:35 smithi123 bash[19119]: audit 2024-02-12T16:12:34.489381+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? v1:172.21.15.121:0/2361764759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:12:35.536 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:35 smithi123 bash[19119]: cluster 2024-02-12T16:12:34.489428+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e318: 12 total, 12 up, 12 in 2024-02-12T16:12:35.553 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:35.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:35 smithi121 bash[18489]: cluster 2024-02-12T16:12:33.746865+0000 osd.0 (osd.0) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:35 smithi121 bash[18489]: cluster 2024-02-12T16:12:33.746915+0000 osd.0 (osd.0) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:35 smithi121 bash[18489]: cluster 2024-02-12T16:12:34.477470+0000 mon.a (mon.0) 1283 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:12:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:35 smithi121 bash[18489]: cluster 2024-02-12T16:12:34.477495+0000 mon.a (mon.0) 1284 : cluster [INF] Cluster is now healthy 2024-02-12T16:12:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:35 smithi121 bash[18489]: audit 2024-02-12T16:12:34.489381+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? v1:172.21.15.121:0/2361764759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:12:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:35 smithi121 bash[18489]: cluster 2024-02-12T16:12:34.489428+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e318: 12 total, 12 up, 12 in 2024-02-12T16:12:35.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:35.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:12:35.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:35.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:35 smithi169 bash[19604]: cluster 2024-02-12T16:12:33.746865+0000 osd.0 (osd.0) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:35.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:35 smithi169 bash[19604]: cluster 2024-02-12T16:12:33.746915+0000 osd.0 (osd.0) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:35.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:35 smithi169 bash[19604]: cluster 2024-02-12T16:12:34.477470+0000 mon.a (mon.0) 1283 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:12:35.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:35 smithi169 bash[19604]: cluster 2024-02-12T16:12:34.477495+0000 mon.a (mon.0) 1284 : cluster [INF] Cluster is now healthy 2024-02-12T16:12:35.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:35 smithi169 bash[19604]: audit 2024-02-12T16:12:34.489381+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? v1:172.21.15.121:0/2361764759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:12:35.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:35 smithi169 bash[19604]: cluster 2024-02-12T16:12:34.489428+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e318: 12 total, 12 up, 12 in 2024-02-12T16:12:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:35.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-02-12T16:12:35.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:35.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:35.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:36.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:36.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:36.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:36.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:36.254 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:36.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:36.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:36.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:36.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:36.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:36 smithi169 bash[19604]: cluster 2024-02-12T16:12:35.675516+0000 mgr.y (mgr.14146) 816 : cluster [DBG] pgmap v1037: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-02-12T16:12:36.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:36.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:36.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:36.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:36.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:36 smithi121 bash[18489]: cluster 2024-02-12T16:12:35.675516+0000 mgr.y (mgr.14146) 816 : cluster [DBG] pgmap v1037: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-02-12T16:12:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:36.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:36.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:36.769 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:36 smithi123 bash[19119]: cluster 2024-02-12T16:12:35.675516+0000 mgr.y (mgr.14146) 816 : cluster [DBG] pgmap v1037: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-02-12T16:12:36.789 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:36.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:36.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:36.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:36.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:37.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:37.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:37.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:37.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:37.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.317 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.319 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:37.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:12:37.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.443 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:37.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:37.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:37 smithi121 bash[18489]: cluster 2024-02-12T16:12:36.944730+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1 deep-scrub starts 2024-02-12T16:12:37.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:37.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:37.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:37.823 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:37.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:12:37.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:37 smithi169 bash[19604]: cluster 2024-02-12T16:12:36.944730+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1 deep-scrub starts 2024-02-12T16:12:37.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:37.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:38.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:38.103 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:38.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:38.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:37 smithi123 bash[19119]: cluster 2024-02-12T16:12:36.944730+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1 deep-scrub starts 2024-02-12T16:12:38.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:38.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:38.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:38.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:38.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:38.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:38.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:38.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:38.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:38.618 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:38.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:38.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:38.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:38 smithi169 bash[19604]: cluster 2024-02-12T16:12:37.675937+0000 mgr.y (mgr.14146) 817 : cluster [DBG] pgmap v1038: 17 pgs: 17 active+clean; 2.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-02-12T16:12:38.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:38.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:38.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:38.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:38.887 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:38.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:38 smithi123 bash[19119]: cluster 2024-02-12T16:12:37.675937+0000 mgr.y (mgr.14146) 817 : cluster [DBG] pgmap v1038: 17 pgs: 17 active+clean; 2.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-02-12T16:12:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:38 smithi121 bash[18489]: cluster 2024-02-12T16:12:37.675937+0000 mgr.y (mgr.14146) 817 : cluster [DBG] pgmap v1038: 17 pgs: 17 active+clean; 2.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-02-12T16:12:38.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:38.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:39.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:39.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:39.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:39.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:39.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:39.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:39.580 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:39.683 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:39.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:39.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:39.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:39.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:40.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:40.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:40.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:40.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:40.247 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:40.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:40.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:40.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:40.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:40.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:40.524 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:40.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:40.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:40.629 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:40.631 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:40.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:40.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:40.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:40.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:40.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:40.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:40.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:40.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:41.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:41.032 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:40 smithi121 bash[18489]: cluster 2024-02-12T16:12:39.676385+0000 mgr.y (mgr.14146) 818 : cluster [DBG] pgmap v1039: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 770 op/s 2024-02-12T16:12:41.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:40 smithi169 bash[19604]: cluster 2024-02-12T16:12:39.676385+0000 mgr.y (mgr.14146) 818 : cluster [DBG] pgmap v1039: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 770 op/s 2024-02-12T16:12:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:41.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:40 smithi123 bash[19119]: cluster 2024-02-12T16:12:39.676385+0000 mgr.y (mgr.14146) 818 : cluster [DBG] pgmap v1039: 17 pgs: 17 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 770 op/s 2024-02-12T16:12:41.162 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:41.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:41.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:41.317 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:41.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:41.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:41.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:41.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:41.653 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:41.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:41.708 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:41.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:41.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:41.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:41.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:41.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:42.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:42.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:42.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:42.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:42.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:42.364 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:42.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:42.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:42.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:42.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:42.624 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:42.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:42.659 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:42.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.738 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:42 smithi169 bash[19604]: cluster 2024-02-12T16:12:41.676756+0000 mgr.y (mgr.14146) 819 : cluster [DBG] pgmap v1040: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-02-12T16:12:42.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:42.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:42.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:42 smithi123 bash[19119]: cluster 2024-02-12T16:12:41.676756+0000 mgr.y (mgr.14146) 819 : cluster [DBG] pgmap v1040: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-02-12T16:12:42.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:42.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:42.995 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:43.027 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:43.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:43.157 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:42 smithi121 bash[18489]: cluster 2024-02-12T16:12:41.676756+0000 mgr.y (mgr.14146) 819 : cluster [DBG] pgmap v1040: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-02-12T16:12:43.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:43.285 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:43.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:43.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:43.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:43.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:43.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:43.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:43.745 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:43 smithi123 bash[19119]: cluster 2024-02-12T16:12:42.098361+0000 osd.5 (osd.5) 41 : cluster [DBG] 3.5 deep-scrub starts 2024-02-12T16:12:43.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:43.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:43 smithi169 bash[19604]: cluster 2024-02-12T16:12:42.098361+0000 osd.5 (osd.5) 41 : cluster [DBG] 3.5 deep-scrub starts 2024-02-12T16:12:43.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:43 smithi121 bash[18489]: cluster 2024-02-12T16:12:42.098361+0000 osd.5 (osd.5) 41 : cluster [DBG] 3.5 deep-scrub starts 2024-02-12T16:12:43.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:43.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:43.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:44.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:44.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:44.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:44.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:44.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:44.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:44.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:44.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:44.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:44.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:44.468 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:44.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:44.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:44.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:44.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:44.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:44.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:44.745 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:44 smithi123 bash[19119]: cluster 2024-02-12T16:12:43.677123+0000 mgr.y (mgr.14146) 820 : cluster [DBG] pgmap v1041: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 977 op/s 2024-02-12T16:12:44.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:44.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:44.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:44 smithi121 bash[18489]: cluster 2024-02-12T16:12:43.677123+0000 mgr.y (mgr.14146) 820 : cluster [DBG] pgmap v1041: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 977 op/s 2024-02-12T16:12:44.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:45.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:44 smithi169 bash[19604]: cluster 2024-02-12T16:12:43.677123+0000 mgr.y (mgr.14146) 820 : cluster [DBG] pgmap v1041: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 977 op/s 2024-02-12T16:12:45.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:45.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:45.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:45.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:45.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:45.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:45.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:45.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:45.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:45.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:45.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:45.786 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:45.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:45.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:45.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:46.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:46.053 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:46.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:46.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:46.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:46.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:46.363 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:46 smithi123 bash[19119]: cluster 2024-02-12T16:12:45.677583+0000 mgr.y (mgr.14146) 821 : cluster [DBG] pgmap v1042: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-12T16:12:46.425 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:46.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:46.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:46.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:46 smithi121 bash[18489]: cluster 2024-02-12T16:12:45.677583+0000 mgr.y (mgr.14146) 821 : cluster [DBG] pgmap v1042: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-12T16:12:46.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:46.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:46.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:46.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:46 smithi169 bash[19604]: cluster 2024-02-12T16:12:45.677583+0000 mgr.y (mgr.14146) 821 : cluster [DBG] pgmap v1042: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.9 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-12T16:12:46.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:46.847 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:46.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:46.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:46.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:47.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:47.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:47.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:47.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:47.249 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:47.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:47.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:47.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:47.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:47.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:47.503 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:47.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:47.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:47.633 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:47.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:47.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:47.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:47.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:47.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:47.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:47.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:47.996 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.030 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:48.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:48.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:48.156 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:48.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:48.284 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:48.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:48.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:48.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:48.543 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:48.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:48.673 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:48.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:48.807 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:48 smithi123 bash[19119]: cluster 2024-02-12T16:12:47.678019+0000 mgr.y (mgr.14146) 822 : cluster [DBG] pgmap v1043: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-02-12T16:12:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:48 smithi123 bash[19119]: cluster 2024-02-12T16:12:48.674854+0000 osd.10 (osd.10) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:48 smithi123 bash[19119]: cluster 2024-02-12T16:12:48.674905+0000 osd.10 (osd.10) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:48.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:48.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:48 smithi169 bash[19604]: cluster 2024-02-12T16:12:47.678019+0000 mgr.y (mgr.14146) 822 : cluster [DBG] pgmap v1043: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-02-12T16:12:48.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:48 smithi169 bash[19604]: cluster 2024-02-12T16:12:48.674854+0000 osd.10 (osd.10) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:48.921 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:48 smithi169 bash[19604]: cluster 2024-02-12T16:12:48.674905+0000 osd.10 (osd.10) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:48.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:48.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:49.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:49.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:48 smithi121 bash[18489]: cluster 2024-02-12T16:12:47.678019+0000 mgr.y (mgr.14146) 822 : cluster [DBG] pgmap v1043: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s 2024-02-12T16:12:49.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:48 smithi121 bash[18489]: cluster 2024-02-12T16:12:48.674854+0000 osd.10 (osd.10) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:49.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:48 smithi121 bash[18489]: cluster 2024-02-12T16:12:48.674905+0000 osd.10 (osd.10) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:49.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:49.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:49.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:49.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:49.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:49.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:49.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:49.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:49.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:49.475 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:12:49.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:49.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:49.509 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:12:49.540 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:12:49.572 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:12:49.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:49.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:49.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:49.612 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:12:49.633 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:12:49.659 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:12:49.687 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:12:49.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:49.716 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:12:49.747 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:12:49.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:49.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:49.780 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:12:49.800 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:12:49.800 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:12:49.805 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:12:49.810 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.810 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.810 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.810 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.810 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.810 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:12:49.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:49.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:49.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:49 smithi123 bash[19119]: cluster 2024-02-12T16:12:48.734713+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e319: 12 total, 12 up, 12 in 2024-02-12T16:12:49.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:49 smithi123 bash[19119]: audit 2024-02-12T16:12:49.447637+0000 mon.a (mon.0) 1288 : audit [DBG] from='client.? v1:172.21.15.121:0/3809006593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:49.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:49.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:50.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:50.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:50.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:49 smithi169 bash[19604]: cluster 2024-02-12T16:12:48.734713+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e319: 12 total, 12 up, 12 in 2024-02-12T16:12:50.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:49 smithi169 bash[19604]: audit 2024-02-12T16:12:49.447637+0000 mon.a (mon.0) 1288 : audit [DBG] from='client.? v1:172.21.15.121:0/3809006593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:50.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:49 smithi121 bash[18489]: cluster 2024-02-12T16:12:48.734713+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e319: 12 total, 12 up, 12 in 2024-02-12T16:12:50.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:49 smithi121 bash[18489]: audit 2024-02-12T16:12:49.447637+0000 mon.a (mon.0) 1288 : audit [DBG] from='client.? v1:172.21.15.121:0/3809006593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:12:50.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:50.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:50.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:50.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:50.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:50.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:50.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:50.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:50.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:50.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:50.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:50.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:50.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:50.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:50.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:50.751 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:50 smithi121 bash[18489]: cluster 2024-02-12T16:12:49.678368+0000 mgr.y (mgr.14146) 823 : cluster [DBG] pgmap v1045: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-12T16:12:50.751 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:50 smithi121 bash[18489]: audit 2024-02-12T16:12:50.531102+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? v1:172.21.15.121:0/118015581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:12:50.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:50.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:50.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:50.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:50.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.9 reset_purged_snaps_last 2024-02-12T16:12:50.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:51.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:50 smithi169 bash[19604]: cluster 2024-02-12T16:12:49.678368+0000 mgr.y (mgr.14146) 823 : cluster [DBG] pgmap v1045: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-12T16:12:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:50 smithi169 bash[19604]: audit 2024-02-12T16:12:50.531102+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? v1:172.21.15.121:0/118015581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:12:51.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:51.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:50 smithi123 bash[19119]: cluster 2024-02-12T16:12:49.678368+0000 mgr.y (mgr.14146) 823 : cluster [DBG] pgmap v1045: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s 2024-02-12T16:12:51.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:50 smithi123 bash[19119]: audit 2024-02-12T16:12:50.531102+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? v1:172.21.15.121:0/118015581' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:12:51.158 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:51.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:51.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:51.358 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:12:51.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:51.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:51.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:51.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:12:51.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:51.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:51.752 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:51 smithi123 bash[19119]: cluster 2024-02-12T16:12:50.902726+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1 deep-scrub ok 2024-02-12T16:12:51.753 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:51 smithi123 bash[19119]: cluster 2024-02-12T16:12:51.346832+0000 mon.a (mon.0) 1290 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:51.753 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:51 smithi123 bash[19119]: audit 2024-02-12T16:12:51.350202+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? v1:172.21.15.121:0/118015581' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:12:51.753 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:51 smithi123 bash[19119]: cluster 2024-02-12T16:12:51.350238+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e320: 12 total, 12 up, 12 in 2024-02-12T16:12:51.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:51.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:51 smithi169 bash[19604]: cluster 2024-02-12T16:12:50.902726+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1 deep-scrub ok 2024-02-12T16:12:51.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:51 smithi169 bash[19604]: cluster 2024-02-12T16:12:51.346832+0000 mon.a (mon.0) 1290 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:51.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:51 smithi169 bash[19604]: audit 2024-02-12T16:12:51.350202+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? v1:172.21.15.121:0/118015581' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:12:51.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:51 smithi169 bash[19604]: cluster 2024-02-12T16:12:51.350238+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e320: 12 total, 12 up, 12 in 2024-02-12T16:12:51.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:51.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:51.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:51.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:51 smithi121 bash[18489]: cluster 2024-02-12T16:12:50.902726+0000 osd.0 (osd.0) 50 : cluster [DBG] 3.1 deep-scrub ok 2024-02-12T16:12:51.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:51 smithi121 bash[18489]: cluster 2024-02-12T16:12:51.346832+0000 mon.a (mon.0) 1290 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:12:51.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:51 smithi121 bash[18489]: audit 2024-02-12T16:12:51.350202+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? v1:172.21.15.121:0/118015581' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:12:51.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:51 smithi121 bash[18489]: cluster 2024-02-12T16:12:51.350238+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e320: 12 total, 12 up, 12 in 2024-02-12T16:12:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:51.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:52.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:52.094 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:52.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:52.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:52.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:52.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:52.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:52.344 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:52.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:52.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:52.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:52.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:52.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:52.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:52.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:52.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:52.732 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:52.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:52.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:52.845 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:52 smithi123 bash[19119]: cluster 2024-02-12T16:12:51.647785+0000 osd.1 (osd.1) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:52.845 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:52 smithi123 bash[19119]: cluster 2024-02-12T16:12:51.647847+0000 osd.1 (osd.1) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:52.845 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:52 smithi123 bash[19119]: cluster 2024-02-12T16:12:51.678791+0000 mgr.y (mgr.14146) 824 : cluster [DBG] pgmap v1047: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s 2024-02-12T16:12:52.845 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:52 smithi123 bash[19119]: cluster 2024-02-12T16:12:52.354510+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e321: 12 total, 12 up, 12 in 2024-02-12T16:12:52.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:52.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:52.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:52.979 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:52 smithi121 bash[18489]: cluster 2024-02-12T16:12:51.647785+0000 osd.1 (osd.1) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:52.979 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:52 smithi121 bash[18489]: cluster 2024-02-12T16:12:51.647847+0000 osd.1 (osd.1) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:52.979 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:52 smithi121 bash[18489]: cluster 2024-02-12T16:12:51.678791+0000 mgr.y (mgr.14146) 824 : cluster [DBG] pgmap v1047: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s 2024-02-12T16:12:52.980 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:52 smithi121 bash[18489]: cluster 2024-02-12T16:12:52.354510+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e321: 12 total, 12 up, 12 in 2024-02-12T16:12:53.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:52 smithi169 bash[19604]: cluster 2024-02-12T16:12:51.647785+0000 osd.1 (osd.1) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:12:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:52 smithi169 bash[19604]: cluster 2024-02-12T16:12:51.647847+0000 osd.1 (osd.1) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:12:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:52 smithi169 bash[19604]: cluster 2024-02-12T16:12:51.678791+0000 mgr.y (mgr.14146) 824 : cluster [DBG] pgmap v1047: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s 2024-02-12T16:12:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:52 smithi169 bash[19604]: cluster 2024-02-12T16:12:52.354510+0000 mon.a (mon.0) 1293 : cluster [DBG] osdmap e321: 12 total, 12 up, 12 in 2024-02-12T16:12:53.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:53.133 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:53.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:53.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:53.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:53.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:53.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:12:53.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:53.523 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:53.628 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:53.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:53.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:53.778 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:53.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:53.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:53.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:54.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:54.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:54.148 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:54.172 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:54.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:12:54.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:54.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:54.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:54.445 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:54.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:54.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:54.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:54.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:54.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:54.843 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:54.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:54.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:54.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:54 smithi169 bash[19604]: cluster 2024-02-12T16:12:53.679593+0000 mgr.y (mgr.14146) 825 : cluster [DBG] pgmap v1049: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s 2024-02-12T16:12:55.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:55.080 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:54 smithi123 bash[19119]: cluster 2024-02-12T16:12:53.679593+0000 mgr.y (mgr.14146) 825 : cluster [DBG] pgmap v1049: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s 2024-02-12T16:12:55.108 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:55.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:54 smithi121 bash[18489]: cluster 2024-02-12T16:12:53.679593+0000 mgr.y (mgr.14146) 825 : cluster [DBG] pgmap v1049: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s 2024-02-12T16:12:55.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:55.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:55.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:55.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:55.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:55.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:55.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:55.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:55.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:55.630 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:55.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:55.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:55.761 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:55.762 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:55.762 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:55.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:55.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:55.898 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:55.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:56.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:56.049 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:56.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:56.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:56.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:56.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:56.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:56 smithi121 bash[18489]: cluster 2024-02-12T16:12:55.679961+0000 mgr.y (mgr.14146) 826 : cluster [DBG] pgmap v1050: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s 2024-02-12T16:12:56.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:56 smithi121 bash[18489]: cluster 2024-02-12T16:12:55.744470+0000 osd.5 (osd.5) 42 : cluster [DBG] 3.5 deep-scrub ok 2024-02-12T16:12:56.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:56.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:56.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:56 smithi169 bash[19604]: cluster 2024-02-12T16:12:55.679961+0000 mgr.y (mgr.14146) 826 : cluster [DBG] pgmap v1050: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s 2024-02-12T16:12:56.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:56 smithi169 bash[19604]: cluster 2024-02-12T16:12:55.744470+0000 osd.5 (osd.5) 42 : cluster [DBG] 3.5 deep-scrub ok 2024-02-12T16:12:56.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:56.573 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:56 smithi123 bash[19119]: cluster 2024-02-12T16:12:55.679961+0000 mgr.y (mgr.14146) 826 : cluster [DBG] pgmap v1050: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s 2024-02-12T16:12:56.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:56 smithi123 bash[19119]: cluster 2024-02-12T16:12:55.744470+0000 osd.5 (osd.5) 42 : cluster [DBG] 3.5 deep-scrub ok 2024-02-12T16:12:56.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:56.704 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:56.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:56.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:56.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:56.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:56.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:57.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:57.110 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:57.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:57.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:57.238 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:57.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:57.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:57.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:57.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:57.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:57.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:57.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:57.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:57.630 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:12:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:57.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:12:57.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:57.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:57.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:57.898 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:57.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:57.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:58.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:58.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:12:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:12:58.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:58.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:58.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:58.302 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:12:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:12:58.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:58.428 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:58.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:58.556 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:58.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:58.686 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:12:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:12:58.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:58.798 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:12:58.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:12:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:12:58.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:58 smithi123 bash[19119]: cluster 2024-02-12T16:12:57.680360+0000 mgr.y (mgr.14146) 827 : cluster [DBG] pgmap v1051: 17 pgs: 17 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-02-12T16:12:58.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:58.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:12:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:58.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:12:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:58 smithi169 bash[19604]: cluster 2024-02-12T16:12:57.680360+0000 mgr.y (mgr.14146) 827 : cluster [DBG] pgmap v1051: 17 pgs: 17 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-02-12T16:12:59.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:59.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:59.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:58 smithi121 bash[18489]: cluster 2024-02-12T16:12:57.680360+0000 mgr.y (mgr.14146) 827 : cluster [DBG] pgmap v1051: 17 pgs: 17 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-02-12T16:12:59.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:12:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:59.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:12:59.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:59.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:12:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:59.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:12:59.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:59.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:12:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:59.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:12:59.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:59.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:12:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:59.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:12:59.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:12:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:59.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:12:59.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:59.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:59 smithi121 bash[18489]: audit 2024-02-12T16:12:59.107664+0000 mon.a (mon.0) 1294 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:12:59.862 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:59 smithi121 bash[18489]: audit 2024-02-12T16:12:59.471263+0000 mon.a (mon.0) 1295 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:12:59.862 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:59 smithi121 bash[18489]: audit 2024-02-12T16:12:59.472268+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:12:59.862 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:12:59 smithi121 bash[18489]: audit 2024-02-12T16:12:59.477290+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:12:59.878 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:12:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:12:59.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:12:59.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:00.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:00.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:59 smithi169 bash[19604]: audit 2024-02-12T16:12:59.107664+0000 mon.a (mon.0) 1294 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:13:00.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:59 smithi169 bash[19604]: audit 2024-02-12T16:12:59.471263+0000 mon.a (mon.0) 1295 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:13:00.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:59 smithi169 bash[19604]: audit 2024-02-12T16:12:59.472268+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:13:00.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:12:59 smithi169 bash[19604]: audit 2024-02-12T16:12:59.477290+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:13:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:00.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:00.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:59 smithi123 bash[19119]: audit 2024-02-12T16:12:59.107664+0000 mon.a (mon.0) 1294 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:13:00.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:59 smithi123 bash[19119]: audit 2024-02-12T16:12:59.471263+0000 mon.a (mon.0) 1295 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:13:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:59 smithi123 bash[19119]: audit 2024-02-12T16:12:59.472268+0000 mon.a (mon.0) 1296 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:13:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:12:59 smithi123 bash[19119]: audit 2024-02-12T16:12:59.477290+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:13:00.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:00.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:00.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:00.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:00.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:00.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:00.530 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:00.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.660 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:00.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:00.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:00.784 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:00.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:00.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:00 smithi169 bash[19604]: cluster 2024-02-12T16:12:59.680723+0000 mgr.y (mgr.14146) 828 : cluster [DBG] pgmap v1052: 17 pgs: 17 active+clean; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s 2024-02-12T16:13:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:00 smithi121 bash[18489]: cluster 2024-02-12T16:12:59.680723+0000 mgr.y (mgr.14146) 828 : cluster [DBG] pgmap v1052: 17 pgs: 17 active+clean; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s 2024-02-12T16:13:00.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:00.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:01.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:01.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:01.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:00 smithi123 bash[19119]: cluster 2024-02-12T16:12:59.680723+0000 mgr.y (mgr.14146) 828 : cluster [DBG] pgmap v1052: 17 pgs: 17 active+clean; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s 2024-02-12T16:13:01.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:01.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:01.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:01.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:01.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:01.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:01.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:01.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:01.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:01.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:01.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:01.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:01.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:01.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:01.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:01.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:01.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:01.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:02.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:02.037 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:02.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:02.170 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:02.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:02.296 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:02.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:02.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:02.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:02.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:02.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:02.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:02.705 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:02.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:02.830 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:02.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:02.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:02.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:03.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:02 smithi169 bash[19604]: cluster 2024-02-12T16:13:01.681089+0000 mgr.y (mgr.14146) 829 : cluster [DBG] pgmap v1053: 17 pgs: 17 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 975 op/s 2024-02-12T16:13:03.063 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:03.071 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:02 smithi123 bash[19119]: cluster 2024-02-12T16:13:01.681089+0000 mgr.y (mgr.14146) 829 : cluster [DBG] pgmap v1053: 17 pgs: 17 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 975 op/s 2024-02-12T16:13:03.098 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:03.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:02 smithi121 bash[18489]: cluster 2024-02-12T16:13:01.681089+0000 mgr.y (mgr.14146) 829 : cluster [DBG] pgmap v1053: 17 pgs: 17 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 975 op/s 2024-02-12T16:13:03.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:03.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:03.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:03.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:03.361 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:03.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:03.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:03.487 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:03.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:03.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:03.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:03.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:03.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:03.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:03.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:03.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:03.895 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:03.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:03.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:04.026 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:04.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:04.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:04.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:04.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:04.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:04.422 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:04.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:04.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:04.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:04.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:04.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:04.835 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:04 smithi121 bash[18489]: cluster 2024-02-12T16:13:03.681822+0000 mgr.y (mgr.14146) 830 : cluster [DBG] pgmap v1054: 17 pgs: 17 active+clean; 1012 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 981 op/s 2024-02-12T16:13:04.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:04.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:04.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:05.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:04 smithi169 bash[19604]: cluster 2024-02-12T16:13:03.681822+0000 mgr.y (mgr.14146) 830 : cluster [DBG] pgmap v1054: 17 pgs: 17 active+clean; 1012 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 981 op/s 2024-02-12T16:13:05.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:05.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:05.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:05.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:04 smithi123 bash[19119]: cluster 2024-02-12T16:13:03.681822+0000 mgr.y (mgr.14146) 830 : cluster [DBG] pgmap v1054: 17 pgs: 17 active+clean; 1012 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 981 op/s 2024-02-12T16:13:05.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:05.229 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:05.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:05.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:05.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:05.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:05.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:05.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:05.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:05.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:05.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:05.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:05.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:05.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:05.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:05.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:05.891 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:05.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:05.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:06.010 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:13:06.011 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:13:06.028 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:06.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:06.156 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:06.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:06.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:06.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:06.430 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:06.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:06.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:06 smithi169 bash[19604]: cluster 2024-02-12T16:13:05.682250+0000 mgr.y (mgr.14146) 831 : cluster [DBG] pgmap v1055: 17 pgs: 17 active+clean; 817 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:13:06.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:06.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:06 smithi123 bash[19119]: cluster 2024-02-12T16:13:05.682250+0000 mgr.y (mgr.14146) 831 : cluster [DBG] pgmap v1055: 17 pgs: 17 active+clean; 817 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:13:06.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:06 smithi121 bash[18489]: cluster 2024-02-12T16:13:05.682250+0000 mgr.y (mgr.14146) 831 : cluster [DBG] pgmap v1055: 17 pgs: 17 active+clean; 817 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:13:06.690 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:06.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:06.820 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:06.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:06.952 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:06.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:07.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:07.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:07.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:07.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:07.219 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:07.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:07.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:07.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:07.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:07.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:07.489 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:07.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:07.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:07.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:07.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:07.687 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:13:07.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.10 reset_purged_snaps_last 2024-02-12T16:13:07.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:07.736 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:13:07.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:07.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:07.771 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:13:07.810 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:13:07.842 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:13:07.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:07.882 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:13:07.888 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:07.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:07.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:13:07.916 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:13:07.952 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:13:07.991 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:13:07.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:08.036 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:13:08.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:08.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:08.072 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:13:08.114 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.122 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.122 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.122 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:13:08.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:08.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:08.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:08.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:08.332 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:08.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:08.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:08.463 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:08.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:08.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:08.588 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:08.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:08 smithi123 bash[19119]: cluster 2024-02-12T16:13:07.133150+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:08 smithi123 bash[19119]: cluster 2024-02-12T16:13:07.133198+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:08 smithi123 bash[19119]: cluster 2024-02-12T16:13:07.368250+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e322: 12 total, 12 up, 12 in 2024-02-12T16:13:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:08 smithi123 bash[19119]: audit 2024-02-12T16:13:07.652535+0000 mon.a (mon.0) 1299 : audit [DBG] from='client.? v1:172.21.15.121:0/3740596705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:08 smithi123 bash[19119]: cluster 2024-02-12T16:13:07.682580+0000 mgr.y (mgr.14146) 832 : cluster [DBG] pgmap v1057: 17 pgs: 17 active+clean; 754 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-12T16:13:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:08 smithi121 bash[18489]: cluster 2024-02-12T16:13:07.133150+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:08 smithi121 bash[18489]: cluster 2024-02-12T16:13:07.133198+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:08 smithi121 bash[18489]: cluster 2024-02-12T16:13:07.368250+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e322: 12 total, 12 up, 12 in 2024-02-12T16:13:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:08 smithi121 bash[18489]: audit 2024-02-12T16:13:07.652535+0000 mon.a (mon.0) 1299 : audit [DBG] from='client.? v1:172.21.15.121:0/3740596705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:08 smithi121 bash[18489]: cluster 2024-02-12T16:13:07.682580+0000 mgr.y (mgr.14146) 832 : cluster [DBG] pgmap v1057: 17 pgs: 17 active+clean; 754 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-12T16:13:08.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:08.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:08.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:08 smithi169 bash[19604]: cluster 2024-02-12T16:13:07.133150+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:08 smithi169 bash[19604]: cluster 2024-02-12T16:13:07.133198+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:08 smithi169 bash[19604]: cluster 2024-02-12T16:13:07.368250+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e322: 12 total, 12 up, 12 in 2024-02-12T16:13:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:08 smithi169 bash[19604]: audit 2024-02-12T16:13:07.652535+0000 mon.a (mon.0) 1299 : audit [DBG] from='client.? v1:172.21.15.121:0/3740596705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:08 smithi169 bash[19604]: cluster 2024-02-12T16:13:07.682580+0000 mgr.y (mgr.14146) 832 : cluster [DBG] pgmap v1057: 17 pgs: 17 active+clean; 754 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-12T16:13:08.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:08.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:08.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:08.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:09.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:09.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:09.141 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:09.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:09.271 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:09.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:09.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:13:09.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:09.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:09.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:09 smithi169 bash[19604]: cluster 2024-02-12T16:13:08.216176+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:09.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:09 smithi169 bash[19604]: cluster 2024-02-12T16:13:08.216234+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:09.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:09 smithi169 bash[19604]: cluster 2024-02-12T16:13:08.384842+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e323: 12 total, 12 up, 12 in 2024-02-12T16:13:09.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:09 smithi169 bash[19604]: audit 2024-02-12T16:13:08.444621+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? v1:172.21.15.121:0/955102136' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:09.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:09.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:09 smithi121 bash[18489]: cluster 2024-02-12T16:13:08.216176+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:09.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:09 smithi121 bash[18489]: cluster 2024-02-12T16:13:08.216234+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:09.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:09 smithi121 bash[18489]: cluster 2024-02-12T16:13:08.384842+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e323: 12 total, 12 up, 12 in 2024-02-12T16:13:09.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:09 smithi121 bash[18489]: audit 2024-02-12T16:13:08.444621+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? v1:172.21.15.121:0/955102136' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:09.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:09.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:09.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:09 smithi123 bash[19119]: cluster 2024-02-12T16:13:08.216176+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:09 smithi123 bash[19119]: cluster 2024-02-12T16:13:08.216234+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:09 smithi123 bash[19119]: cluster 2024-02-12T16:13:08.384842+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e323: 12 total, 12 up, 12 in 2024-02-12T16:13:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:09 smithi123 bash[19119]: audit 2024-02-12T16:13:08.444621+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? v1:172.21.15.121:0/955102136' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:09.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:09.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:09.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:10.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:10.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:10.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:10.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:10.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:10.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:10.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:10.333 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:10.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:10.435 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:10.443 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:10 smithi123 bash[19119]: cluster 2024-02-12T16:13:09.378202+0000 mon.a (mon.0) 1302 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:10.443 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:10 smithi123 bash[19119]: audit 2024-02-12T16:13:09.393000+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? v1:172.21.15.121:0/955102136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:13:10.443 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:10 smithi123 bash[19119]: cluster 2024-02-12T16:13:09.393061+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e324: 12 total, 12 up, 12 in 2024-02-12T16:13:10.443 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:10 smithi123 bash[19119]: cluster 2024-02-12T16:13:09.682966+0000 mgr.y (mgr.14146) 833 : cluster [DBG] pgmap v1060: 17 pgs: 17 active+clean; 645 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 977 op/s 2024-02-12T16:13:10.466 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:10.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:10.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:10.602 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:10.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:10 smithi121 bash[18489]: cluster 2024-02-12T16:13:09.378202+0000 mon.a (mon.0) 1302 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:10 smithi121 bash[18489]: audit 2024-02-12T16:13:09.393000+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? v1:172.21.15.121:0/955102136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:13:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:10 smithi121 bash[18489]: cluster 2024-02-12T16:13:09.393061+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e324: 12 total, 12 up, 12 in 2024-02-12T16:13:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:10 smithi121 bash[18489]: cluster 2024-02-12T16:13:09.682966+0000 mgr.y (mgr.14146) 833 : cluster [DBG] pgmap v1060: 17 pgs: 17 active+clean; 645 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 977 op/s 2024-02-12T16:13:10.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:10.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:10 smithi169 bash[19604]: cluster 2024-02-12T16:13:09.378202+0000 mon.a (mon.0) 1302 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:10.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:10 smithi169 bash[19604]: audit 2024-02-12T16:13:09.393000+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? v1:172.21.15.121:0/955102136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:13:10.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:10 smithi169 bash[19604]: cluster 2024-02-12T16:13:09.393061+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e324: 12 total, 12 up, 12 in 2024-02-12T16:13:10.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:10 smithi169 bash[19604]: cluster 2024-02-12T16:13:09.682966+0000 mgr.y (mgr.14146) 833 : cluster [DBG] pgmap v1060: 17 pgs: 17 active+clean; 645 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 977 op/s 2024-02-12T16:13:10.739 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:10.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:10.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:10.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:10.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:10.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:11.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:11.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:13:11.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:11.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:11.288 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:11.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:11.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:11.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:11.564 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:11.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:11.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:11.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:11.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:11.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:11.959 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:11.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:12.063 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:12.092 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:12.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:12.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:12.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:12.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:12.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:12.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:12.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:12.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:13:12.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:12.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:12.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:12.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:12.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:12.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:12.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:12.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:12 smithi169 bash[19604]: cluster 2024-02-12T16:13:11.683407+0000 mgr.y (mgr.14146) 834 : cluster [DBG] pgmap v1061: 17 pgs: 17 active+clean; 464 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s 2024-02-12T16:13:12.753 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:12.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:12.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:12.886 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:12.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:12.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:13.000 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:12 smithi121 bash[18489]: cluster 2024-02-12T16:13:11.683407+0000 mgr.y (mgr.14146) 834 : cluster [DBG] pgmap v1061: 17 pgs: 17 active+clean; 464 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s 2024-02-12T16:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:13.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:12 smithi123 bash[19119]: cluster 2024-02-12T16:13:11.683407+0000 mgr.y (mgr.14146) 834 : cluster [DBG] pgmap v1061: 17 pgs: 17 active+clean; 464 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s 2024-02-12T16:13:13.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:13.170 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:13.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:13.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:13.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:13.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:13.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:13.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:13.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:13.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:13.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:13.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:13.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:13.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:13.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:14.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:14.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:14.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:14.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:14.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:14.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:14.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:14.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:14.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:14.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:14.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:14.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:14.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:14.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:14.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:14.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:14.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:14.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:14.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:14.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:14.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:15.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:15.033 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:14 smithi123 bash[19119]: cluster 2024-02-12T16:13:13.683798+0000 mgr.y (mgr.14146) 835 : cluster [DBG] pgmap v1062: 17 pgs: 17 active+clean; 327 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-12T16:13:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:14 smithi169 bash[19604]: cluster 2024-02-12T16:13:13.683798+0000 mgr.y (mgr.14146) 835 : cluster [DBG] pgmap v1062: 17 pgs: 17 active+clean; 327 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-12T16:13:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:15.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:15.155 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:14 smithi121 bash[18489]: cluster 2024-02-12T16:13:13.683798+0000 mgr.y (mgr.14146) 835 : cluster [DBG] pgmap v1062: 17 pgs: 17 active+clean; 327 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-02-12T16:13:15.192 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:15.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:15.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:15.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:15.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:15.371 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 163781 objects 2024-02-12T16:13:15.371 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :155.399818 2024-02-12T16:13:15.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:15.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:15.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:15.597 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:15.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:15.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:15.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:15.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:15.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:15.878 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:15.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:15.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:16.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:16.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:16.155 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:16.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:16.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:16.297 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:16.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:16.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:16.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:16.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:16.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:16.580 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:16.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:16.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:16.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:16.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:16.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:16.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:16.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:16.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:16.983 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:16 smithi121 bash[18489]: cluster 2024-02-12T16:13:14.999037+0000 osd.8 (osd.8) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:16.983 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:16 smithi121 bash[18489]: cluster 2024-02-12T16:13:14.999103+0000 osd.8 (osd.8) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:16.983 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:16 smithi121 bash[18489]: cluster 2024-02-12T16:13:15.684414+0000 mgr.y (mgr.14146) 836 : cluster [DBG] pgmap v1063: 17 pgs: 17 active+clean; 91 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-02-12T16:13:16.984 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:16 smithi121 bash[18489]: cluster 2024-02-12T16:13:15.750295+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e325: 12 total, 12 up, 12 in 2024-02-12T16:13:17.007 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:17.009 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:16 smithi123 bash[19119]: cluster 2024-02-12T16:13:14.999037+0000 osd.8 (osd.8) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:17.009 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:16 smithi123 bash[19119]: cluster 2024-02-12T16:13:14.999103+0000 osd.8 (osd.8) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:17.009 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:16 smithi123 bash[19119]: cluster 2024-02-12T16:13:15.684414+0000 mgr.y (mgr.14146) 836 : cluster [DBG] pgmap v1063: 17 pgs: 17 active+clean; 91 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-02-12T16:13:17.009 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:16 smithi123 bash[19119]: cluster 2024-02-12T16:13:15.750295+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e325: 12 total, 12 up, 12 in 2024-02-12T16:13:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:16 smithi169 bash[19604]: cluster 2024-02-12T16:13:14.999037+0000 osd.8 (osd.8) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:16 smithi169 bash[19604]: cluster 2024-02-12T16:13:14.999103+0000 osd.8 (osd.8) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:16 smithi169 bash[19604]: cluster 2024-02-12T16:13:15.684414+0000 mgr.y (mgr.14146) 836 : cluster [DBG] pgmap v1063: 17 pgs: 17 active+clean; 91 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-02-12T16:13:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:16 smithi169 bash[19604]: cluster 2024-02-12T16:13:15.750295+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e325: 12 total, 12 up, 12 in 2024-02-12T16:13:17.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:17.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:17.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:17.286 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:17.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:17.420 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:17.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:17.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:17.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:17.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:17.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:17.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:17.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:17.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:17.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:17.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:18.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:18.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:18.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:18.230 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:18.267 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:18.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:18.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:18.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:18.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:18.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:18.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:18.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:18.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:18.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:18.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:18.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:18.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:18 smithi121 bash[18489]: cluster 2024-02-12T16:13:17.452949+0000 osd.5 (osd.5) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:18.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:18 smithi121 bash[18489]: cluster 2024-02-12T16:13:17.453058+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:18.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:18 smithi121 bash[18489]: cluster 2024-02-12T16:13:17.685145+0000 mgr.y (mgr.14146) 837 : cluster [DBG] pgmap v1065: 17 pgs: 17 active+clean; 45 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-02-12T16:13:18.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:18 smithi121 bash[18489]: cluster 2024-02-12T16:13:17.764321+0000 mon.a (mon.0) 1306 : cluster [DBG] osdmap e326: 12 total, 12 up, 12 in 2024-02-12T16:13:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:18.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:18.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:18.972 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:18.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:18 smithi169 bash[19604]: cluster 2024-02-12T16:13:17.452949+0000 osd.5 (osd.5) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:18 smithi169 bash[19604]: cluster 2024-02-12T16:13:17.453058+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:18 smithi169 bash[19604]: cluster 2024-02-12T16:13:17.685145+0000 mgr.y (mgr.14146) 837 : cluster [DBG] pgmap v1065: 17 pgs: 17 active+clean; 45 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-02-12T16:13:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:18 smithi169 bash[19604]: cluster 2024-02-12T16:13:17.764321+0000 mon.a (mon.0) 1306 : cluster [DBG] osdmap e326: 12 total, 12 up, 12 in 2024-02-12T16:13:19.067 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-02-12T16:13:19.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-02-12T16:13:19.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:19.087 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:18 smithi123 bash[19119]: cluster 2024-02-12T16:13:17.452949+0000 osd.5 (osd.5) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:19.087 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:18 smithi123 bash[19119]: cluster 2024-02-12T16:13:17.453058+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:19.087 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:18 smithi123 bash[19119]: cluster 2024-02-12T16:13:17.685145+0000 mgr.y (mgr.14146) 837 : cluster [DBG] pgmap v1065: 17 pgs: 17 active+clean; 45 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-02-12T16:13:19.088 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:18 smithi123 bash[19119]: cluster 2024-02-12T16:13:17.764321+0000 mon.a (mon.0) 1306 : cluster [DBG] osdmap e326: 12 total, 12 up, 12 in 2024-02-12T16:13:19.112 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:19.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:19.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:19.256 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:19.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:19.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:19.399 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:19.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:19.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:19.539 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:19.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:19.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:19.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:19.684 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:19.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:19.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:19.819 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:19.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:19.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:19.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:19.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:20.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:20.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:20.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:20.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:20.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:20.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:20.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:20.378 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:20.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:20.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:20.516 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:20.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:20.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:20.647 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:20.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:20.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:20.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:20.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:20.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:20.920 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:20.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:21.015 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:13:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:13:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:13:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:13:21.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51085 2.a e156/147 2:50205334:::benchmark_data_smithi169_42458_object51084:head v 156'3243, mlcod=156'3240)", 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.096434+0000", 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 529.91097824099995, 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26604782599999999, 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096434+0000", 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096434+0000", 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999814 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096416+0000", 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7355000000000003e-05 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096463+0000", 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0964e-05 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096474+0000", 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.959e-06 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096480+0000", 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3524999999999999e-05 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096543+0000", 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9534e-05 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096563+0000", 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26581924499999998 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362382+0000", 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100048 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362482+0000", 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26604782599999999 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51084 2.e e156/147 2:705710c5:::benchmark_data_smithi169_42458_object51083:head v 156'3209, mlcod=156'3207)", 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.096526+0000", 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 529.91088628, 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25884117499999998, 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096526+0000", 2024-02-12T16:13:21.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096526+0000", 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096525+0000", 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1891000000000001e-05 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096547+0000", 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6749999999999996e-06 2024-02-12T16:13:21.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096554+0000", 2024-02-12T16:13:21.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6019999999999996e-06 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096561+0000", 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4433e-05 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096605+0000", 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0227000000000001e-05 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.096625+0000", 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25866377600000001 2024-02-12T16:13:21.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355289+0000", 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8609000000000006e-05 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355368+0000", 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25884117499999998 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51088 2.a e156/147 2:522a68c4:::benchmark_data_smithi169_42458_object51087:head v 156'3244, mlcod=156'3240)", 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.097911+0000", 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 529.909501363, 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26463948399999998, 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097911+0000", 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097911+0000", 2024-02-12T16:13:21.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097910+0000", 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0844000000000002e-05 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097971+0000", 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0606e-05 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097981+0000", 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8599999999999998e-06 2024-02-12T16:13:21.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.097987+0000", 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015630200000000001 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.098143+0000", 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8615e-05 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.098162+0000", 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26432980900000003 2024-02-12T16:13:21.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362492+0000", 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8909000000000001e-05 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.362551+0000", 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26463948399999998 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.028 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.14811.0:51087 2.1 e156/147 2:85827259:::benchmark_data_smithi169_42458_object51086:head v 156'3177, mlcod=156'3175)", 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:04:31.099054+0000", 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 529.90835823099997, 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26445508099999998, 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099054+0000", 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099054+0000", 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099053+0000", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8713000000000001e-05 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.099081+0000", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25612307000000001 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355205+0000", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2753000000000001e-05 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355227+0000", 2024-02-12T16:13:21.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106338 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355334+0000", 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0373000000000001e-05 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.355374+0000", 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.008059933 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.363434+0000", 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5575999999999996e-05 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:04:31.363510+0000", 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26445508099999998 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:69301 3.b 3:d794b199:::benchmark_data_smithi169_51390_object69300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.918779+0000", 2024-02-12T16:13:21.032 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.088633937, 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34272655899999999, 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69301 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.918779+0000", 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.918779+0000", 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.918778+0000", 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3051000000000001e-05 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.918871+0000", 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5505000000000001e-05 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.918886+0000", 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5710000000000003e-06 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.918893+0000", 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087845860000000005 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.927678+0000", 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119677 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.927797+0000", 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015532600000000001 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.927953+0000", 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087242039999999993 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936677+0000", 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022813999999999998 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.938958+0000", 2024-02-12T16:13:21.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8781000000000003e-05 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.939047+0000", 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32245831400000002 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261505+0000", 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34272655899999999 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.037 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:69305 3.b 3:d342a0ee:::benchmark_data_smithi169_51390_object69304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.920987+0000", 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.08642541200001, 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346314224, 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69305 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.920987+0000", 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.920987+0000", 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.920986+0000", 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5406000000000004e-05 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.921051+0000", 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2215000000000001e-05 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.921063+0000", 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8750000000000002e-06 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.921070+0000", 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093098020000000007 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.930380+0000", 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012702199999999999 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.930507+0000", 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017628399999999999 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.930683+0000", 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006026656 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936710+0000", 2024-02-12T16:13:21.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33035618 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.267066+0000", 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013142199999999999 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.267197+0000", 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104036 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.267301+0000", 2024-02-12T16:13:21.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346314224 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:69333 3.b 3:da685c10:::benchmark_data_smithi169_51390_object69332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.935709+0000", 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.07170320099999, 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.335777669, 2024-02-12T16:13:21.043 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69333 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935709+0000", 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935709+0000", 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935707+0000", 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9995999999999999e-05 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935777+0000", 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1577999999999999e-05 2024-02-12T16:13:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935799+0000", 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5649999999999998e-06 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935805+0000", 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00092541200000000002 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936731+0000", 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6359999999999994e-05 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936807+0000", 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1878000000000004e-05 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936899+0000", 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.330423827 2024-02-12T16:13:21.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.267323+0000", 2024-02-12T16:13:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040866959999999999 2024-02-12T16:13:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.049 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:20 smithi121 bash[18489]: cluster 2024-02-12T16:13:19.685897+0000 mgr.y (mgr.14146) 838 : cluster [DBG] pgmap v1067: 17 pgs: 17 active+clean; 9.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-02-12T16:13:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271409+0000", 2024-02-12T16:13:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2658999999999999e-05 2024-02-12T16:13:21.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271452+0000", 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5128e-05 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271487+0000", 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.335777669 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:69334 3.7 3:e7bfd7a0:::benchmark_data_smithi169_51390_object69333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:13:21.052 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.935827+0000", 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.07158604399999, 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33446439500000003, 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69334 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935827+0000", 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935827+0000", 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935826+0000", 2024-02-12T16:13:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0377999999999999e-05 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935866+0000", 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3350000000000003e-06 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935872+0000", 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4680000000000003e-06 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935877+0000", 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7811000000000001e-05 2024-02-12T16:13:21.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935915+0000", 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9093999999999999e-05 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935994+0000", 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8431999999999996e-05 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936082+0000", 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32991647899999998 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.265999+0000", 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039464849999999996 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.269945+0000", 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000217011 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.270162+0000", 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000128992 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.270291+0000", 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33446439500000003 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:69336 3.2 3:466a69ae:::benchmark_data_smithi169_51390_object69335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:13:21.058 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.935888+0000", 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.071524294, 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32589513599999997, 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69336 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935888+0000", 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935888+0000", 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935888+0000", 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2949e-05 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935911+0000", 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6439999999999997e-06 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935916+0000", 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.817e-06 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935920+0000", 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5216000000000002e-05 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.935965+0000", 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010919199999999999 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936074+0000", 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013867600000000001 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936213+0000", 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32518143100000002 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261395+0000", 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000172067 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261567+0000", 2024-02-12T16:13:21.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127181 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261694+0000", 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9691000000000006e-05 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261784+0000", 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32589513599999997 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:69339 3.b 3:dc80fd2f:::benchmark_data_smithi169_51390_object69338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:13:21.063 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.936152+0000", 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.07126070699999, 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33544102799999997, 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69339 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936152+0000", 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936152+0000", 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936151+0000", 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5008000000000001e-05 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936186+0000", 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.486e-06 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936193+0000", 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.634e-06 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.936198+0000", 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32540633600000002 2024-02-12T16:13:21.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261604+0000", 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00033247700000000002 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261937+0000", 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031564000000000001 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.262253+0000", 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0090986469999999996 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271351+0000", 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019032200000000001 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271541+0000", 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9198000000000001e-05 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271571+0000", 2024-02-12T16:13:21.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2331e-05 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271593+0000", 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33544102799999997 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.15108.0:69336 3.2 e265/256 ondisk, result = 0)", 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.939622+0000", 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.06779082700001, 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.322178195, 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:13:21.069 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.939622+0000", 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.939622+0000", 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.939620+0000", 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3590000000000002e-06 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.939628+0000", 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32175258800000001 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261380+0000", 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8076999999999999e-05 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261408+0000", 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122028 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261530+0000", 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8525e-05 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261549+0000", 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025127 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261800+0000", 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.322178195 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.072 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:69348 3.b 3:d0c3ef7b:::benchmark_data_smithi169_51390_object69347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.941860+0000", 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.06555255800001, 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33221747800000001, 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69348 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.941860+0000", 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.941860+0000", 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.941858+0000", 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011279499999999999 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.941970+0000", 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31943234300000001 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261403+0000", 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9023e-05 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261432+0000", 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059605400000000003 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.267392+0000", 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028682399999999998 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.267679+0000", 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037929700000000001 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.268058+0000", 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003573367 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.271632+0000", 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002359908 2024-02-12T16:13:21.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.273992+0000", 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8795999999999998e-05 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.274041+0000", 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7073999999999998e-05 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.274078+0000", 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33221747800000001 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.077 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.15108.0:69366 3.c e265/256 3:3bd173c3:::benchmark_data_smithi169_51390_object69365:head v 265'4361, mlcod=265'4359)", 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:09:29.951108+0000", 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 231.05630486300001, 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.315068399, 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.951108+0000", 2024-02-12T16:13:21.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.951108+0000", 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.951106+0000", 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0183000000000001e-05 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:29.951186+0000", 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31022433199999999 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261410+0000", 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8176999999999999e-05 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261429+0000", 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9053000000000003e-05 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261518+0000", 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4817e-05 2024-02-12T16:13:21.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.261572+0000", 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045264809999999997 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.266099+0000", 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7311000000000006e-05 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:09:30.266176+0000", 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.315068399 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.081 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:222352 3.7 3:ec43ba0c:::benchmark_data_smithi169_51390_object58568:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:11:36.672271+0000", 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.33514148899999, 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31820906500000001, 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222352 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672271+0000", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672271+0000", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672270+0000", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.617e-06 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672275+0000", 2024-02-12T16:13:21.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2189999999999998e-06 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672282+0000", 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7720000000000002e-06 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672287+0000", 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081496940000000007 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.680436+0000", 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2107999999999996e-05 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.680498+0000", 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6681000000000007e-05 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.680575+0000", 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081832229999999999 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.688758+0000", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30155390599999998 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990312+0000", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4335000000000001e-05 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990367+0000", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011362399999999999 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990480+0000", 2024-02-12T16:13:21.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31820906500000001 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:222355 3.7 3:e8196c0f:::benchmark_data_smithi169_51390_object58571:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:11:36.672390+0000", 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.335023034, 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31825318200000002, 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222355 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672390+0000", 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672390+0000", 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672389+0000", 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9400000000000004e-06 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672393+0000", 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5220000000000004e-06 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672399+0000", 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0820000000000001e-06 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.672403+0000", 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011399495000000001 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.683803+0000", 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7734999999999999e-05 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.683900+0000", 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012907899999999999 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.684030+0000", 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0047337689999999997 2024-02-12T16:13:21.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.688763+0000", 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.301802283 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990566+0000", 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2920999999999998e-05 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990609+0000", 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4303e-05 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990643+0000", 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31825318200000002 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:222366 3.7 3:eaeea655:::benchmark_data_smithi169_51390_object58582:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:11:36.681382+0000", 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.32603025500001, 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31436898600000002, 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222366 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.681382+0000", 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.681382+0000", 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.681381+0000", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3609999999999997e-06 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.681387+0000", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4789999999999996e-06 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.681395+0000", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7899999999999996e-06 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.681401+0000", 2024-02-12T16:13:21.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30050012500000001 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.981901+0000", 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000369522 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.982271+0000", 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00029814199999999999 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 11", 2024-02-12T16:13:21.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.982569+0000", 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0081120419999999999 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990681+0000", 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050112569999999999 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.995692+0000", 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3810000000000003e-05 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.995726+0000", 2024-02-12T16:13:21.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.529e-05 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.995751+0000", 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31436898600000002 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.15108.0:222371 3.5 e304/256 3:a8591d55:::benchmark_data_smithi169_51390_object58587:head v 304'13806, mlcod=304'13804)", 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:11:36.686375+0000", 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.321037875, 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31185278, 2024-02-12T16:13:21.097 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686375+0000", 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686375+0000", 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686373+0000", 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.618e-06 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686380+0000", 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2010000000000002e-06 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686387+0000", 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5090000000000004e-06 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686392+0000", 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3723000000000001e-05 2024-02-12T16:13:21.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686435+0000", 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9409e-05 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.686455+0000", 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31173043499999997 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.998185+0000", 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2247999999999999e-05 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.998228+0000", 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31185278 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.15108.0:222375 3.b 3:d1fe5fca:::benchmark_data_smithi169_51390_object58591:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:11:36.688970+0000", 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.318443019, 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31763893799999998, 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.15108", 2024-02-12T16:13:21.101 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 222375 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.688970+0000", 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.688970+0000", 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.688968+0000", 2024-02-12T16:13:21.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9609999999999996e-06 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.688973+0000", 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29291681899999999 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.981890+0000", 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8742999999999999e-05 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.981919+0000", 2024-02-12T16:13:21.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013217799999999999 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.982051+0000", 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030694300000000001 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.982358+0000", 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001416531 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.983774+0000", 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022680432 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:37.006455+0000", 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5909000000000003e-05 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:37.006511+0000", 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4271999999999998e-05 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:37.006565+0000", 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3800000000000001e-05 2024-02-12T16:13:21.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:37.006609+0000", 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31763893799999998 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.15108.0:222373 3.c e304/256 3:37edf101:::benchmark_data_smithi169_51390_object58589:head v 304'13979, mlcod=304'13977)", 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:11:36.689229+0000", 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.31818395499999, 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.301199355, 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.689229+0000", 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.689229+0000", 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.689227+0000", 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0819999999999999e-06 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.689235+0000", 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29269367800000001 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.981929+0000", 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8756000000000001e-05 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.981948+0000", 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0804999999999998e-05 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.982009+0000", 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6118999999999998e-05 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.982045+0000", 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083498649999999997 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990395+0000", 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3581000000000003e-05 2024-02-12T16:13:21.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990428+0000", 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.301199355 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.15108.0:222352 3.7 e304/256 ondisk, result = 0)", 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:11:36.692138+0000", 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.31527478700001, 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29838351499999999, 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:13:21.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.692138+0000", 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.692138+0000", 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.692137+0000", 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0059999999999998e-06 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.692143+0000", 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28974543600000002 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.981888+0000", 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5387e-05 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.981903+0000", 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0083906710000000006 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990294+0000", 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5119999999999997e-06 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990304+0000", 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000217789 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:11:36.990521+0000", 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29838351499999999 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:13:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:13:21.114 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:13:21.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:21.114 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:20 smithi169 bash[19604]: cluster 2024-02-12T16:13:19.685897+0000 mgr.y (mgr.14146) 838 : cluster [DBG] pgmap v1067: 17 pgs: 17 active+clean; 9.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-02-12T16:13:21.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:20 smithi123 bash[19119]: cluster 2024-02-12T16:13:19.685897+0000 mgr.y (mgr.14146) 838 : cluster [DBG] pgmap v1067: 17 pgs: 17 active+clean; 9.1 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-02-12T16:13:21.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:21.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:21.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:13:21.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:21.278 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:13:21.296 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:21.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:21.303 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:13:21.333 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:13:21.360 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:13:21.390 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:13:21.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:21.414 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:13:21.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:13:21.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:21.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:21.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:21.464 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:13:21.486 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:13:21.510 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:13:21.535 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.538 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.539 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:13:21.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:21.576 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:21.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:21.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:21.716 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:21.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:21.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:21.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:21.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:21 smithi123 bash[19119]: audit 2024-02-12T16:13:21.226143+0000 mon.a (mon.0) 1307 : audit [DBG] from='client.? v1:172.21.15.121:0/2066889388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:21.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:21 smithi121 bash[18489]: audit 2024-02-12T16:13:21.226143+0000 mon.a (mon.0) 1307 : audit [DBG] from='client.? v1:172.21.15.121:0/2066889388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:21.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:22.004 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:22.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:22.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:21 smithi169 bash[19604]: audit 2024-02-12T16:13:21.226143+0000 mon.a (mon.0) 1307 : audit [DBG] from='client.? v1:172.21.15.121:0/2066889388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:22.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:22.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:22.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:22.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:22.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:22.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:22.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:22.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:22.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:22.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:22.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:22.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:22.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:22.710 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:22.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:22 smithi169 bash[19604]: cluster 2024-02-12T16:13:21.686628+0000 mgr.y (mgr.14146) 839 : cluster [DBG] pgmap v1068: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s 2024-02-12T16:13:22.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:22.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:22.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:22.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.11 reset_purged_snaps_last 2024-02-12T16:13:22.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:23.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:23.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:22 smithi123 bash[19119]: cluster 2024-02-12T16:13:21.686628+0000 mgr.y (mgr.14146) 839 : cluster [DBG] pgmap v1068: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s 2024-02-12T16:13:23.139 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:23.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:22 smithi121 bash[18489]: cluster 2024-02-12T16:13:21.686628+0000 mgr.y (mgr.14146) 839 : cluster [DBG] pgmap v1068: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s 2024-02-12T16:13:23.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:23.280 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:23.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:23.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:23.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:23.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:23.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:13:23.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:23.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:23.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:23.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:23.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:23.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:23.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:23.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:23.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:24.054 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:24.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:24.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:24.196 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:24.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:24.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:24.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:24.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:24.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:24.470 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:24.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:24.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:24.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:24.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:24.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:24.756 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:24.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:24.806 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:13:24.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:24.897 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:24.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:25.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:25.014 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:24 smithi169 bash[19604]: cluster 2024-02-12T16:13:23.687536+0000 mgr.y (mgr.14146) 840 : cluster [DBG] pgmap v1069: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s 2024-02-12T16:13:25.014 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:24 smithi169 bash[19604]: audit 2024-02-12T16:13:24.154637+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.121:0/631548079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:13:25.039 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:25.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:24 smithi123 bash[19119]: cluster 2024-02-12T16:13:23.687536+0000 mgr.y (mgr.14146) 840 : cluster [DBG] pgmap v1069: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s 2024-02-12T16:13:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:24 smithi123 bash[19119]: audit 2024-02-12T16:13:24.154637+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.121:0/631548079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:13:25.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:24 smithi121 bash[18489]: cluster 2024-02-12T16:13:23.687536+0000 mgr.y (mgr.14146) 840 : cluster [DBG] pgmap v1069: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s 2024-02-12T16:13:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:24 smithi121 bash[18489]: audit 2024-02-12T16:13:24.154637+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.121:0/631548079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:13:25.179 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:25.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:25.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:25.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:25.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:25.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:25.456 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:25.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:25.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:13:25.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:25.597 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:25.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:25.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:25.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:25.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:25.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:25.844 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:25 smithi169 bash[19604]: cluster 2024-02-12T16:13:24.780510+0000 mon.a (mon.0) 1309 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:25.844 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:25 smithi169 bash[19604]: audit 2024-02-12T16:13:24.795767+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? v1:172.21.15.121:0/631548079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:13:25.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:25 smithi169 bash[19604]: cluster 2024-02-12T16:13:24.795848+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e327: 12 total, 12 up, 12 in 2024-02-12T16:13:25.865 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:25.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:25 smithi121 bash[18489]: cluster 2024-02-12T16:13:24.780510+0000 mon.a (mon.0) 1309 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:25 smithi121 bash[18489]: audit 2024-02-12T16:13:24.795767+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? v1:172.21.15.121:0/631548079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:13:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:25 smithi121 bash[18489]: cluster 2024-02-12T16:13:24.795848+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e327: 12 total, 12 up, 12 in 2024-02-12T16:13:25.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:26.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:26.083 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:26.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:26.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:25 smithi123 bash[19119]: cluster 2024-02-12T16:13:24.780510+0000 mon.a (mon.0) 1309 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:26.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:25 smithi123 bash[19119]: audit 2024-02-12T16:13:24.795767+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? v1:172.21.15.121:0/631548079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:13:26.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:25 smithi123 bash[19119]: cluster 2024-02-12T16:13:24.795848+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e327: 12 total, 12 up, 12 in 2024-02-12T16:13:26.151 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:26.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:26.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:26.435 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:26.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:26.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:26.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:26.710 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:26.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:26.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:26.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:26.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:26 smithi121 bash[18489]: cluster 2024-02-12T16:13:25.410249+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:26.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:26 smithi121 bash[18489]: cluster 2024-02-12T16:13:25.410302+0000 osd.6 (osd.6) 37 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:26.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:26 smithi121 bash[18489]: cluster 2024-02-12T16:13:25.688276+0000 mgr.y (mgr.14146) 841 : cluster [DBG] pgmap v1071: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s 2024-02-12T16:13:26.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:26 smithi121 bash[18489]: cluster 2024-02-12T16:13:26.379654+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e328: 12 total, 12 up, 12 in 2024-02-12T16:13:26.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:26.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:26 smithi169 bash[19604]: cluster 2024-02-12T16:13:25.410249+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:26 smithi169 bash[19604]: cluster 2024-02-12T16:13:25.410302+0000 osd.6 (osd.6) 37 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:26 smithi169 bash[19604]: cluster 2024-02-12T16:13:25.688276+0000 mgr.y (mgr.14146) 841 : cluster [DBG] pgmap v1071: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s 2024-02-12T16:13:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:26 smithi169 bash[19604]: cluster 2024-02-12T16:13:26.379654+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e328: 12 total, 12 up, 12 in 2024-02-12T16:13:27.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:27.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:27.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:26 smithi123 bash[19119]: cluster 2024-02-12T16:13:25.410249+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:26 smithi123 bash[19119]: cluster 2024-02-12T16:13:25.410302+0000 osd.6 (osd.6) 37 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:26 smithi123 bash[19119]: cluster 2024-02-12T16:13:25.688276+0000 mgr.y (mgr.14146) 841 : cluster [DBG] pgmap v1071: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s 2024-02-12T16:13:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:26 smithi123 bash[19119]: cluster 2024-02-12T16:13:26.379654+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e328: 12 total, 12 up, 12 in 2024-02-12T16:13:27.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:27.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:27.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:27.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:27.416 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:27.419 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:27.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:27.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:27.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:27.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:13:27.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:27.697 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:27.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:27.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:27.829 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:27.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:27.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:27.970 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:27.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:28.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:28.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:27 smithi169 bash[19604]: cluster 2024-02-12T16:13:27.173937+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:13:28.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:27 smithi169 bash[19604]: cluster 2024-02-12T16:13:27.205917+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:13:28.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:28.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:28.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:27 smithi123 bash[19119]: cluster 2024-02-12T16:13:27.173937+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:13:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:27 smithi123 bash[19119]: cluster 2024-02-12T16:13:27.205917+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:13:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:27 smithi121 bash[18489]: cluster 2024-02-12T16:13:27.173937+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.6 scrub starts 2024-02-12T16:13:28.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:27 smithi121 bash[18489]: cluster 2024-02-12T16:13:27.205917+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.6 scrub ok 2024-02-12T16:13:28.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:28.251 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:28.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:28.286 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:13:28.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:28.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:28.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:28.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:28.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:28.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:28.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:28.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:28.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:28.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:28.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:28.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:28.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:28.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:28.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:29.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:29.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:28 smithi121 bash[18489]: cluster 2024-02-12T16:13:27.688821+0000 mgr.y (mgr.14146) 842 : cluster [DBG] pgmap v1073: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 18 op/s 2024-02-12T16:13:29.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:28 smithi121 bash[18489]: cluster 2024-02-12T16:13:27.905994+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-02-12T16:13:29.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:28 smithi121 bash[18489]: cluster 2024-02-12T16:13:27.928993+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-02-12T16:13:29.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:29.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:28 smithi123 bash[19119]: cluster 2024-02-12T16:13:27.688821+0000 mgr.y (mgr.14146) 842 : cluster [DBG] pgmap v1073: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 18 op/s 2024-02-12T16:13:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:28 smithi123 bash[19119]: cluster 2024-02-12T16:13:27.905994+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-02-12T16:13:29.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:28 smithi123 bash[19119]: cluster 2024-02-12T16:13:27.928993+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-02-12T16:13:29.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:29.244 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:29.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:28 smithi169 bash[19604]: cluster 2024-02-12T16:13:27.688821+0000 mgr.y (mgr.14146) 842 : cluster [DBG] pgmap v1073: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 18 op/s 2024-02-12T16:13:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:28 smithi169 bash[19604]: cluster 2024-02-12T16:13:27.905994+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-02-12T16:13:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:28 smithi169 bash[19604]: cluster 2024-02-12T16:13:27.928993+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-02-12T16:13:29.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:29.385 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:29.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:29.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:29.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:29.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:29.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:29.669 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:29.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:29.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:29.814 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:29.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:29.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:29.958 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:29.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:30.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:30.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:30.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:30.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:30.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:30.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:30.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:30.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:30.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:30.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:30.518 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:30.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:30.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:30.657 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:30.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:30.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:30.799 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:30.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:30.828 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_1' removed 2024-02-12T16:13:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:30 smithi123 bash[19119]: cluster 2024-02-12T16:13:29.496635+0000 osd.5 (osd.5) 45 : cluster [DBG] 3.c scrub starts 2024-02-12T16:13:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:30 smithi123 bash[19119]: cluster 2024-02-12T16:13:29.523958+0000 osd.5 (osd.5) 46 : cluster [DBG] 3.c scrub ok 2024-02-12T16:13:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:30 smithi123 bash[19119]: cluster 2024-02-12T16:13:29.689522+0000 mgr.y (mgr.14146) 843 : cluster [DBG] pgmap v1074: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:30 smithi123 bash[19119]: cluster 2024-02-12T16:13:29.860304+0000 osd.7 (osd.7) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:30 smithi123 bash[19119]: cluster 2024-02-12T16:13:29.860406+0000 osd.7 (osd.7) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:30 smithi123 bash[19119]: audit 2024-02-12T16:13:30.315221+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? v1:172.21.15.121:0/590477206' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:13:30.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:30.926 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:30.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:31.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:30 smithi169 bash[19604]: cluster 2024-02-12T16:13:29.496635+0000 osd.5 (osd.5) 45 : cluster [DBG] 3.c scrub starts 2024-02-12T16:13:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:30 smithi169 bash[19604]: cluster 2024-02-12T16:13:29.523958+0000 osd.5 (osd.5) 46 : cluster [DBG] 3.c scrub ok 2024-02-12T16:13:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:30 smithi169 bash[19604]: cluster 2024-02-12T16:13:29.689522+0000 mgr.y (mgr.14146) 843 : cluster [DBG] pgmap v1074: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:30 smithi169 bash[19604]: cluster 2024-02-12T16:13:29.860304+0000 osd.7 (osd.7) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:30 smithi169 bash[19604]: cluster 2024-02-12T16:13:29.860406+0000 osd.7 (osd.7) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:30 smithi169 bash[19604]: audit 2024-02-12T16:13:30.315221+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? v1:172.21.15.121:0/590477206' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:13:31.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:31.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:30 smithi121 bash[18489]: cluster 2024-02-12T16:13:29.496635+0000 osd.5 (osd.5) 45 : cluster [DBG] 3.c scrub starts 2024-02-12T16:13:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:30 smithi121 bash[18489]: cluster 2024-02-12T16:13:29.523958+0000 osd.5 (osd.5) 46 : cluster [DBG] 3.c scrub ok 2024-02-12T16:13:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:30 smithi121 bash[18489]: cluster 2024-02-12T16:13:29.689522+0000 mgr.y (mgr.14146) 843 : cluster [DBG] pgmap v1074: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:31.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:30 smithi121 bash[18489]: cluster 2024-02-12T16:13:29.860304+0000 osd.7 (osd.7) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:31.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:30 smithi121 bash[18489]: cluster 2024-02-12T16:13:29.860406+0000 osd.7 (osd.7) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:31.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:30 smithi121 bash[18489]: audit 2024-02-12T16:13:30.315221+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? v1:172.21.15.121:0/590477206' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:13:31.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:31.184 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:31.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:31.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:31.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:31.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:31.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:31.468 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:31.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:31.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:31.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:31.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:31.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:31.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:31.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:31.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:31.867 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:31 smithi121 bash[18489]: audit 2024-02-12T16:13:30.809587+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? v1:172.21.15.121:0/590477206' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:13:31.868 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:31 smithi121 bash[18489]: cluster 2024-02-12T16:13:30.809691+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e329: 12 total, 12 up, 12 in 2024-02-12T16:13:31.891 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:31.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:31.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:32.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:32.045 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:13:32.046 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:13:32.046 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-02-12T16:13:32.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-02-12T16:13:32.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:31 smithi123 bash[19119]: audit 2024-02-12T16:13:30.809587+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? v1:172.21.15.121:0/590477206' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:13:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:31 smithi123 bash[19119]: cluster 2024-02-12T16:13:30.809691+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e329: 12 total, 12 up, 12 in 2024-02-12T16:13:32.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:32.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:32.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:32.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:31 smithi169 bash[19604]: audit 2024-02-12T16:13:30.809587+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? v1:172.21.15.121:0/590477206' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:13:32.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:31 smithi169 bash[19604]: cluster 2024-02-12T16:13:30.809691+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e329: 12 total, 12 up, 12 in 2024-02-12T16:13:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:32.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:32.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:32.410 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:32.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:32.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:32.466 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:13:32.466 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:13:32.513 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:32.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:32.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:32.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:32.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:32.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:32.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:32.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:32.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:32.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:33.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:32 smithi169 bash[19604]: cluster 2024-02-12T16:13:31.690175+0000 mgr.y (mgr.14146) 844 : cluster [DBG] pgmap v1076: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:33.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:13:33.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:32 smithi123 bash[19119]: cluster 2024-02-12T16:13:31.690175+0000 mgr.y (mgr.14146) 844 : cluster [DBG] pgmap v1076: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:33.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:33.118 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:32 smithi121 bash[18489]: cluster 2024-02-12T16:13:31.690175+0000 mgr.y (mgr.14146) 844 : cluster [DBG] pgmap v1076: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:33.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:33.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:33.279 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:33.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:33.420 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:33.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:33.561 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:33.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:33.700 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:33.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:33.844 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:33.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:33.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:33.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:34.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:34.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:33 smithi123 bash[19119]: cluster 2024-02-12T16:13:32.282376+0000 osd.11 (osd.11) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:34.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:33 smithi123 bash[19119]: cluster 2024-02-12T16:13:32.282467+0000 osd.11 (osd.11) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:34.098 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:33 smithi123 bash[19119]: cluster 2024-02-12T16:13:32.838021+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e330: 12 total, 12 up, 12 in 2024-02-12T16:13:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:34.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:33 smithi121 bash[18489]: cluster 2024-02-12T16:13:32.282376+0000 osd.11 (osd.11) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:33 smithi121 bash[18489]: cluster 2024-02-12T16:13:32.282467+0000 osd.11 (osd.11) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:33 smithi121 bash[18489]: cluster 2024-02-12T16:13:32.838021+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e330: 12 total, 12 up, 12 in 2024-02-12T16:13:34.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.8 reset_purged_snaps_last 2024-02-12T16:13:34.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:34.261 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:34.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:33 smithi169 bash[19604]: cluster 2024-02-12T16:13:32.282376+0000 osd.11 (osd.11) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:33 smithi169 bash[19604]: cluster 2024-02-12T16:13:32.282467+0000 osd.11 (osd.11) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:33 smithi169 bash[19604]: cluster 2024-02-12T16:13:32.838021+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e330: 12 total, 12 up, 12 in 2024-02-12T16:13:34.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:34.400 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:34.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:34.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:34.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:34.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:34.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:34.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:34.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:34.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:34.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:34.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:34.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:34.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:34.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:35.059 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:35.096 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:35.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:35.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:34 smithi123 bash[19119]: cluster 2024-02-12T16:13:33.690793+0000 mgr.y (mgr.14146) 845 : cluster [DBG] pgmap v1078: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:34 smithi121 bash[18489]: cluster 2024-02-12T16:13:33.690793+0000 mgr.y (mgr.14146) 845 : cluster [DBG] pgmap v1078: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:35.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:35.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:35.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:34 smithi169 bash[19604]: cluster 2024-02-12T16:13:33.690793+0000 mgr.y (mgr.14146) 845 : cluster [DBG] pgmap v1078: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:35.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:35.370 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:35.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:35.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:35.516 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:35.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:35.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:35.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:35.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:35.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:35.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:35.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:35.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:35.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:35.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:35.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:36.038 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:36.073 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:36.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:36.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:36.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:36.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:36.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:36.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:36.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:36.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:36.490 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:36.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:36.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:36.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:36.606 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:36 smithi121 bash[18489]: cluster 2024-02-12T16:13:35.691424+0000 mgr.y (mgr.14146) 846 : cluster [DBG] pgmap v1079: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:36.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:36.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:36.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:36 smithi123 bash[19119]: cluster 2024-02-12T16:13:35.691424+0000 mgr.y (mgr.14146) 846 : cluster [DBG] pgmap v1079: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:36.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:36.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:36.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:36.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:36 smithi169 bash[19604]: cluster 2024-02-12T16:13:35.691424+0000 mgr.y (mgr.14146) 846 : cluster [DBG] pgmap v1079: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:36.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:36.900 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:36.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:36.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:37.005 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:37.044 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:37.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:37.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:37.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:37.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:37.296 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:37.333 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:37.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:37.424 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:13:37.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:37.445 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:13:37.470 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:13:37.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:37.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:37.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:13:37.514 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:13:37.539 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:13:37.564 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:13:37.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:37.591 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:13:37.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:13:37.614 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:37.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:37.636 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:13:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:37 smithi123 bash[19119]: cluster 2024-02-12T16:13:36.436387+0000 osd.9 (osd.9) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:37 smithi123 bash[19119]: cluster 2024-02-12T16:13:36.436478+0000 osd.9 (osd.9) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:37 smithi121 bash[18489]: cluster 2024-02-12T16:13:36.436387+0000 osd.9 (osd.9) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:37 smithi121 bash[18489]: cluster 2024-02-12T16:13:36.436478+0000 osd.9 (osd.9) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:37.660 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:13:37.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:13:37.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:37.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:37.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:37.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:37 smithi169 bash[19604]: cluster 2024-02-12T16:13:36.436387+0000 osd.9 (osd.9) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:37.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:37 smithi169 bash[19604]: cluster 2024-02-12T16:13:36.436478+0000 osd.9 (osd.9) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:37.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:37.867 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:13:37.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:37.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:38.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:38.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:38.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:38.139 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:38.176 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:38.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:38.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:38.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:38.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:38.410 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:13:38.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:38.432 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:38 smithi123 bash[19119]: cluster 2024-02-12T16:13:37.388323+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e331: 12 total, 12 up, 12 in 2024-02-12T16:13:38.432 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:38 smithi123 bash[19119]: audit 2024-02-12T16:13:37.397793+0000 mon.c (mon.1) 33 : audit [DBG] from='client.? v1:172.21.15.121:0/1552312176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:38.432 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:38 smithi123 bash[19119]: audit 2024-02-12T16:13:37.587007+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:38.432 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:38 smithi123 bash[19119]: audit 2024-02-12T16:13:37.589928+0000 mon.b (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.121:0/159009635' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:38.432 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:38 smithi123 bash[19119]: cluster 2024-02-12T16:13:37.692003+0000 mgr.y (mgr.14146) 847 : cluster [DBG] pgmap v1081: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:38.458 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:38.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:38.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:38.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:38.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:38 smithi121 bash[18489]: cluster 2024-02-12T16:13:37.388323+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e331: 12 total, 12 up, 12 in 2024-02-12T16:13:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:38 smithi121 bash[18489]: audit 2024-02-12T16:13:37.397793+0000 mon.c (mon.1) 33 : audit [DBG] from='client.? v1:172.21.15.121:0/1552312176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:38.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:38 smithi121 bash[18489]: audit 2024-02-12T16:13:37.587007+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:38.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:38 smithi121 bash[18489]: audit 2024-02-12T16:13:37.589928+0000 mon.b (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.121:0/159009635' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:38.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:38 smithi121 bash[18489]: cluster 2024-02-12T16:13:37.692003+0000 mgr.y (mgr.14146) 847 : cluster [DBG] pgmap v1081: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:38.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:38.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:38.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:38.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:38 smithi169 bash[19604]: cluster 2024-02-12T16:13:37.388323+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e331: 12 total, 12 up, 12 in 2024-02-12T16:13:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:38 smithi169 bash[19604]: audit 2024-02-12T16:13:37.397793+0000 mon.c (mon.1) 33 : audit [DBG] from='client.? v1:172.21.15.121:0/1552312176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:38 smithi169 bash[19604]: audit 2024-02-12T16:13:37.587007+0000 mon.a (mon.0) 1318 : audit [INF] from='client.25298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:38 smithi169 bash[19604]: audit 2024-02-12T16:13:37.589928+0000 mon.b (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.121:0/159009635' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:13:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:38 smithi169 bash[19604]: cluster 2024-02-12T16:13:37.692003+0000 mgr.y (mgr.14146) 847 : cluster [DBG] pgmap v1081: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:38.847 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:38.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:38.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:38.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:39.019 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:39.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:39.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:39.149 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:39.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:39.288 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:39.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:39.430 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:39.534 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:39.550 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:39.551 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:39.571 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:39.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:39 smithi121 bash[18489]: cluster 2024-02-12T16:13:38.385992+0000 mon.a (mon.0) 1319 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:13:39.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:39 smithi121 bash[18489]: cluster 2024-02-12T16:13:38.386025+0000 mon.a (mon.0) 1320 : cluster [INF] Cluster is now healthy 2024-02-12T16:13:39.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:39 smithi121 bash[18489]: audit 2024-02-12T16:13:38.398317+0000 mon.a (mon.0) 1321 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:13:39.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:39 smithi121 bash[18489]: cluster 2024-02-12T16:13:38.398427+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e332: 12 total, 12 up, 12 in 2024-02-12T16:13:39.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:39.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:39 smithi169 bash[19604]: cluster 2024-02-12T16:13:38.385992+0000 mon.a (mon.0) 1319 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:13:39.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:39 smithi169 bash[19604]: cluster 2024-02-12T16:13:38.386025+0000 mon.a (mon.0) 1320 : cluster [INF] Cluster is now healthy 2024-02-12T16:13:39.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:39 smithi169 bash[19604]: audit 2024-02-12T16:13:38.398317+0000 mon.a (mon.0) 1321 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:13:39.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:39 smithi169 bash[19604]: cluster 2024-02-12T16:13:38.398427+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e332: 12 total, 12 up, 12 in 2024-02-12T16:13:39.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:39.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:39.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:39 smithi123 bash[19119]: cluster 2024-02-12T16:13:38.385992+0000 mon.a (mon.0) 1319 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:13:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:39 smithi123 bash[19119]: cluster 2024-02-12T16:13:38.386025+0000 mon.a (mon.0) 1320 : cluster [INF] Cluster is now healthy 2024-02-12T16:13:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:39 smithi123 bash[19119]: audit 2024-02-12T16:13:38.398317+0000 mon.a (mon.0) 1321 : audit [INF] from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:13:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:39 smithi123 bash[19119]: cluster 2024-02-12T16:13:38.398427+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e332: 12 total, 12 up, 12 in 2024-02-12T16:13:39.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:39.994 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:39.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:40.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:40.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:40.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:40.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:40.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:40.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:40.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:40.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:40.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:40.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:13:40.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:40.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:40.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:40.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:40 smithi123 bash[19119]: cluster 2024-02-12T16:13:39.692686+0000 mgr.y (mgr.14146) 848 : cluster [DBG] pgmap v1083: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:40 smithi121 bash[18489]: cluster 2024-02-12T16:13:39.692686+0000 mgr.y (mgr.14146) 848 : cluster [DBG] pgmap v1083: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:40.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:40.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:40.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:40 smithi169 bash[19604]: cluster 2024-02-12T16:13:39.692686+0000 mgr.y (mgr.14146) 848 : cluster [DBG] pgmap v1083: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:40.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:40.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:40.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:40.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:40.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:40.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:41.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:41.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:13:41.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:41.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:41.154 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:13:41.154 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:13:41.155 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2024-02-12T16:13:41.155 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2024-02-12T16:13:41.155 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:13:41.240 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:41.272 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:41.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:41.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:41.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:41.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:41.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:41.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:41.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:41.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:41.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:41.701 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:41.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:41.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:41.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:13:41.847 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:41.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:41.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:41.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:41.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:42.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:42.129 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:42.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:42.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:42.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:42.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:42.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:42.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:42.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:42.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:42.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:42.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:42.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:42.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:42.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:42.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:42 smithi169 bash[19604]: cluster 2024-02-12T16:13:41.693336+0000 mgr.y (mgr.14146) 849 : cluster [DBG] pgmap v1084: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:42.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:42.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:42.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:42.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:42.981 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:42.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:43.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:43.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:43.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:43.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:42 smithi123 bash[19119]: cluster 2024-02-12T16:13:41.693336+0000 mgr.y (mgr.14146) 849 : cluster [DBG] pgmap v1084: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:42 smithi121 bash[18489]: cluster 2024-02-12T16:13:41.693336+0000 mgr.y (mgr.14146) 849 : cluster [DBG] pgmap v1084: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:43.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:43.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:43.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:43.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:43.403 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:43.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:43.508 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:43.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:43.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:43.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:43.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:43.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:43.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:43.829 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:43.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:43.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:43.962 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:43.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:43 smithi169 bash[19604]: cluster 2024-02-12T16:13:42.978654+0000 osd.7 (osd.7) 57 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:13:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:43 smithi169 bash[19604]: cluster 2024-02-12T16:13:43.044967+0000 osd.7 (osd.7) 58 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:13:44.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:44.075 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:43 smithi123 bash[19119]: cluster 2024-02-12T16:13:42.978654+0000 osd.7 (osd.7) 57 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:13:44.075 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:43 smithi123 bash[19119]: cluster 2024-02-12T16:13:43.044967+0000 osd.7 (osd.7) 58 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:13:44.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:43 smithi121 bash[18489]: cluster 2024-02-12T16:13:42.978654+0000 osd.7 (osd.7) 57 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:13:44.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:43 smithi121 bash[18489]: cluster 2024-02-12T16:13:43.044967+0000 osd.7 (osd.7) 58 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:13:44.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:44.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:44.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:44.238 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:44.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:44.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:44.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:44.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:44.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:44.526 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:44.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:44.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:44.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:44.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:44.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:44.788 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_2' created 2024-02-12T16:13:44.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:44.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:44.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:44 smithi121 bash[18489]: cluster 2024-02-12T16:13:42.975694+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:44 smithi121 bash[18489]: cluster 2024-02-12T16:13:42.975784+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:44 smithi121 bash[18489]: cluster 2024-02-12T16:13:43.693964+0000 mgr.y (mgr.14146) 850 : cluster [DBG] pgmap v1085: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:44 smithi121 bash[18489]: cluster 2024-02-12T16:13:43.763112+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e333: 12 total, 12 up, 12 in 2024-02-12T16:13:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:44 smithi121 bash[18489]: cluster 2024-02-12T16:13:43.974353+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:44 smithi121 bash[18489]: cluster 2024-02-12T16:13:43.974439+0000 osd.0 (osd.0) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:44 smithi121 bash[18489]: audit 2024-02-12T16:13:44.187293+0000 mon.a (mon.0) 1324 : audit [INF] from='client.? v1:172.21.15.121:0/2523782308' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-12T16:13:44.936 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:44.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:45.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:44 smithi169 bash[19604]: cluster 2024-02-12T16:13:42.975694+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:44 smithi169 bash[19604]: cluster 2024-02-12T16:13:42.975784+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:44 smithi169 bash[19604]: cluster 2024-02-12T16:13:43.693964+0000 mgr.y (mgr.14146) 850 : cluster [DBG] pgmap v1085: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:44 smithi169 bash[19604]: cluster 2024-02-12T16:13:43.763112+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e333: 12 total, 12 up, 12 in 2024-02-12T16:13:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:44 smithi169 bash[19604]: cluster 2024-02-12T16:13:43.974353+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:44 smithi169 bash[19604]: cluster 2024-02-12T16:13:43.974439+0000 osd.0 (osd.0) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:44 smithi169 bash[19604]: audit 2024-02-12T16:13:44.187293+0000 mon.a (mon.0) 1324 : audit [INF] from='client.? v1:172.21.15.121:0/2523782308' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-12T16:13:45.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:45.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:44 smithi123 bash[19119]: cluster 2024-02-12T16:13:42.975694+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:44 smithi123 bash[19119]: cluster 2024-02-12T16:13:42.975784+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:44 smithi123 bash[19119]: cluster 2024-02-12T16:13:43.693964+0000 mgr.y (mgr.14146) 850 : cluster [DBG] pgmap v1085: 1 pgs: 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:44 smithi123 bash[19119]: cluster 2024-02-12T16:13:43.763112+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e333: 12 total, 12 up, 12 in 2024-02-12T16:13:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:44 smithi123 bash[19119]: cluster 2024-02-12T16:13:43.974353+0000 osd.0 (osd.0) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:13:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:44 smithi123 bash[19119]: cluster 2024-02-12T16:13:43.974439+0000 osd.0 (osd.0) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:13:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:44 smithi123 bash[19119]: audit 2024-02-12T16:13:44.187293+0000 mon.a (mon.0) 1324 : audit [INF] from='client.? v1:172.21.15.121:0/2523782308' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-12T16:13:45.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:45.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:45.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:45.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:45.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:45.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:45.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:45.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:45.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-02-12T16:13:45.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:45.640 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:45.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:45.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:45.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:45.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:45.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:45.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:45 smithi123 bash[19119]: audit 2024-02-12T16:13:44.777191+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? v1:172.21.15.121:0/2523782308' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-12T16:13:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:45 smithi123 bash[19119]: cluster 2024-02-12T16:13:44.777357+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e334: 12 total, 12 up, 12 in 2024-02-12T16:13:45.920 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:45.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:45.950 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:45.950 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:46.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:45 smithi169 bash[19604]: audit 2024-02-12T16:13:44.777191+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? v1:172.21.15.121:0/2523782308' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-12T16:13:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:45 smithi169 bash[19604]: cluster 2024-02-12T16:13:44.777357+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e334: 12 total, 12 up, 12 in 2024-02-12T16:13:46.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:46.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:46.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:46.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:45 smithi121 bash[18489]: audit 2024-02-12T16:13:44.777191+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? v1:172.21.15.121:0/2523782308' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-12T16:13:46.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:45 smithi121 bash[18489]: cluster 2024-02-12T16:13:44.777357+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e334: 12 total, 12 up, 12 in 2024-02-12T16:13:46.198 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:46.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:46.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:46.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:46.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:46.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:46.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:46.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:46.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:46.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:46.613 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:13:46.624 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:46.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:46.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:46.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:46.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:46.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:46.909 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:46.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:47.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:47.026 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:46 smithi169 bash[19604]: cluster 2024-02-12T16:13:45.694472+0000 mgr.y (mgr.14146) 851 : cluster [DBG] pgmap v1088: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:47.026 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:46 smithi169 bash[19604]: cluster 2024-02-12T16:13:45.783012+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e335: 12 total, 12 up, 12 in 2024-02-12T16:13:47.052 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:47.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:47.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:46 smithi123 bash[19119]: cluster 2024-02-12T16:13:45.694472+0000 mgr.y (mgr.14146) 851 : cluster [DBG] pgmap v1088: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:47.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:46 smithi123 bash[19119]: cluster 2024-02-12T16:13:45.783012+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e335: 12 total, 12 up, 12 in 2024-02-12T16:13:47.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:47.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:46 smithi121 bash[18489]: cluster 2024-02-12T16:13:45.694472+0000 mgr.y (mgr.14146) 851 : cluster [DBG] pgmap v1088: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:47.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:46 smithi121 bash[18489]: cluster 2024-02-12T16:13:45.783012+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e335: 12 total, 12 up, 12 in 2024-02-12T16:13:47.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:47.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:47.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:47.333 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:47.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:47.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:47.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:47.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:47.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:47.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:47.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:47.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:47.746 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:47.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:47.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:47.885 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:47.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:47.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:47 smithi123 bash[19119]: cluster 2024-02-12T16:13:46.785574+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e336: 12 total, 12 up, 12 in 2024-02-12T16:13:47.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:48.025 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:48.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:48.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:47 smithi169 bash[19604]: cluster 2024-02-12T16:13:46.785574+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e336: 12 total, 12 up, 12 in 2024-02-12T16:13:48.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:48.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:47 smithi121 bash[18489]: cluster 2024-02-12T16:13:46.785574+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e336: 12 total, 12 up, 12 in 2024-02-12T16:13:48.168 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:48.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:48.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:48.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:48.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:48.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:48.452 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:48.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:48.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:48.593 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:48.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:48.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:48.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:48.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:48.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:48.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:48 smithi169 bash[19604]: cluster 2024-02-12T16:13:47.695208+0000 mgr.y (mgr.14146) 852 : cluster [DBG] pgmap v1091: 17 pgs: 1 creating+peering, 10 unknown, 6 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:48.877 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:48.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:48.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:49.020 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:49.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:49.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:49.136 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:48 smithi123 bash[19119]: cluster 2024-02-12T16:13:47.695208+0000 mgr.y (mgr.14146) 852 : cluster [DBG] pgmap v1091: 17 pgs: 1 creating+peering, 10 unknown, 6 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:48 smithi121 bash[18489]: cluster 2024-02-12T16:13:47.695208+0000 mgr.y (mgr.14146) 852 : cluster [DBG] pgmap v1091: 17 pgs: 1 creating+peering, 10 unknown, 6 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:49.161 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:49.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:49.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:49.302 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:49.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:49.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:49.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:49.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:49.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:49.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:49.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:49.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:49.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:153852 3.3 3:cab4ad1c:::benchmark_data_smithi169_51390_object153851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e285)", 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:10:29.041456+0000", 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 200.71951418200001, 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20223028800000001, 2024-02-12T16:13:49.766 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 153852 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041456+0000", 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041456+0000", 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041454+0000", 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2121999999999997e-05 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041536+0000", 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5543e-05 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041552+0000", 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.691e-06 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041558+0000", 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8700999999999999e-05 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041617+0000", 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000164802 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041782+0000", 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016885700000000001 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.041951+0000", 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041375240000000001 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046088+0000", 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.197459568 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.243548+0000", 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7566999999999998e-05 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.243625+0000", 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1284000000000005e-05 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.243687+0000", 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20223028800000001 2024-02-12T16:13:49.771 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:153857 3.3 3:cb150752:::benchmark_data_smithi169_51390_object153856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e285)", 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:10:29.046638+0000", 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 200.714333052, 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.206290955, 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.772 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 153857 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046638+0000", 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046638+0000", 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046635+0000", 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6035999999999998e-05 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046681+0000", 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2564e-05 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046694+0000", 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0690000000000001e-06 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046699+0000", 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7194000000000002e-05 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046746+0000", 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103669 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046850+0000", 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120698 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.046970+0000", 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003901524 2024-02-12T16:13:49.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.050872+0000", 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20194469800000001 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.252817+0000", 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8238999999999995e-05 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.252885+0000", 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3786999999999999e-05 2024-02-12T16:13:49.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.252929+0000", 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.206290955 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:153862 3.3 3:c2ce3406:::benchmark_data_smithi169_51390_object153861:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e285)", 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:10:29.054276+0000", 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 200.70669441199999, 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20903633299999999, 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 153862 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054276+0000", 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054276+0000", 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999919 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054268+0000", 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9390000000000001e-05 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054308+0000", 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3942e-05 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054322+0000", 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9510000000000003e-06 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054328+0000", 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8313999999999998e-05 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054386+0000", 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015320199999999999 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054539+0000", 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019427800000000001 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.054733+0000", 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043814290000000001 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.059115+0000", 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20408593999999999 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.263201+0000", 2024-02-12T16:13:49.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2260000000000004e-05 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.263263+0000", 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9632999999999997e-05 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:10:29.263313+0000", 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20903633299999999 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:219741 3.3 3:cc43c20c:::benchmark_data_smithi169_51390_object55957:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:34.083732+0000", 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 135.67723908599999, 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34937416399999999, 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 219741 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.083732+0000", 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.083732+0000", 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.083729+0000", 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4569999999999999e-06 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.083737+0000", 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0012e-05 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.083747+0000", 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9819999999999999e-06 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.083752+0000", 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34040612399999998 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424158+0000", 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030529299999999999 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424463+0000", 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018108999999999999 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424644+0000", 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005363245 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.430007+0000", 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030190210000000002 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433026+0000", 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1764999999999997e-05 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433068+0000", 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7521000000000002e-05 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433106+0000", 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34937416399999999 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:251985 3.3 3:cf743e5d:::benchmark_data_smithi169_51390_object88201:head [delete] snapc 0=[] ondisk+write+known_if_redirected e308)", 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:02.432510+0000", 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 107.328460521, 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21232525699999999, 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.788 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 251985 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432510+0000", 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432510+0000", 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432507+0000", 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1292e-05 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432519+0000", 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3961999999999999e-05 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432533+0000", 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3233e-05 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432546+0000", 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9534e-05 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432635+0000", 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010343799999999999 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432739+0000", 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011736299999999999 2024-02-12T16:13:49.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.432856+0000", 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007780459 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.440637+0000", 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.204079961 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.644717+0000", 2024-02-12T16:13:49.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9426999999999998e-05 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.644786+0000", 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9419999999999998e-05 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.644835+0000", 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21232525699999999 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:251988 3.3 3:cc84037b:::benchmark_data_smithi169_51390_object88204:head [delete] snapc 0=[] ondisk+write+known_if_redirected e308)", 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:02.436666+0000", 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 107.32430492, 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.208319908, 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 251988 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436666+0000", 2024-02-12T16:13:49.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436666+0000", 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436664+0000", 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0769999999999996e-06 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436670+0000", 2024-02-12T16:13:49.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1772e-05 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436682+0000", 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4190000000000004e-06 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436686+0000", 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5199e-05 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436721+0000", 2024-02-12T16:13:49.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9716999999999994e-05 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436821+0000", 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000132044 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.436953+0000", 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038453599999999999 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.440798+0000", 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20408604599999999 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.644884+0000", 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2114999999999998e-05 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.644937+0000", 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9039999999999998e-05 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.644986+0000", 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.208319908 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:252000 3.3 3:c5cdbeb1:::benchmark_data_smithi169_51390_object88216:head [delete] snapc 0=[] ondisk+write+known_if_redirected e308)", 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:02.446972+0000", 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 107.31399832300001, 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20104377200000001, 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.799 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 252000 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.446972+0000", 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.446972+0000", 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.446970+0000", 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3250000000000002e-06 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.446977+0000", 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6319000000000001e-05 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.446994+0000", 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9120000000000004e-06 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.447003+0000", 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5865999999999997e-05 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.447058+0000", 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8064000000000002e-05 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.447157+0000", 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7854000000000003e-05 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.447244+0000", 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044095749999999998 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.451654+0000", 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19626750400000001 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.647921+0000", 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2509999999999997e-05 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.647974+0000", 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2079999999999997e-05 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:02.648016+0000", 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20104377200000001 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287209 3.3 3:c52098c1:::benchmark_data_smithi169_51390_object123425:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.498762+0000", 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.262208673999993, 2024-02-12T16:13:49.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49929650599999997, 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287209 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498762+0000", 2024-02-12T16:13:49.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498762+0000", 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498759+0000", 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9199999999999993e-06 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498768+0000", 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3084999999999999e-05 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498781+0000", 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5809999999999999e-06 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498788+0000", 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2963999999999997e-05 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498831+0000", 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5323999999999997e-05 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498927+0000", 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010236 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.499029+0000", 2024-02-12T16:13:49.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0078967670000000007 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506926+0000", 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49075337699999999 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997679+0000", 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015290200000000001 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997832+0000", 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022633399999999999 2024-02-12T16:13:49.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998058+0000", 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49929650599999997 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287211 3.3 3:cff2d28c:::benchmark_data_smithi169_51390_object123427:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.502243+0000", 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.258727895000007, 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.495993411, 2024-02-12T16:13:49.812 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287211 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502243+0000", 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502243+0000", 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502240+0000", 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5879999999999998e-06 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502249+0000", 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2266000000000001e-05 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502261+0000", 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0820000000000001e-06 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502268+0000", 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058720500000000004 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502855+0000", 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010033499999999999 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.502955+0000", 2024-02-12T16:13:49.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010233100000000001 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503058+0000", 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040468750000000001 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507105+0000", 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.491011641 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998116+0000", 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2342999999999994e-05 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998199+0000", 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7469000000000003e-05 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998236+0000", 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.495993411 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:287215 3.a e318/256 3:55b1025c:::benchmark_data_smithi169_51390_object123431:head v 318'17891, mlcod=317'17889)", 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.506373+0000", 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.254597199000003, 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49454500600000001, 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506373+0000", 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506373+0000", 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506373+0000", 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7920000000000004e-06 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506380+0000", 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9451000000000001e-05 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506399+0000", 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7360000000000004e-06 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506406+0000", 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7453000000000001e-05 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506453+0000", 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0231999999999999e-05 2024-02-12T16:13:49.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506473+0000", 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49422145499999998 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.000695+0000", 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022355899999999999 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.000918+0000", 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49454500600000001 2024-02-12T16:13:49.821 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287221 3.3 3:cf584b35:::benchmark_data_smithi169_51390_object123437:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.508316+0000", 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.252654471, 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49003415900000002, 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287221 2024-02-12T16:13:49.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508316+0000", 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508316+0000", 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508314+0000", 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9030000000000003e-06 2024-02-12T16:13:49.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508319+0000", 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1430000000000003e-06 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508325+0000", 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6219999999999998e-06 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508329+0000", 2024-02-12T16:13:49.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6121000000000002e-05 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508365+0000", 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5730999999999997e-05 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508441+0000", 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0388000000000004e-05 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508521+0000", 2024-02-12T16:13:49.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038799329999999999 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512401+0000", 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48587478200000001 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998276+0000", 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2178000000000002e-05 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998318+0000", 2024-02-12T16:13:49.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2221999999999997e-05 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998350+0000", 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49003415900000002 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:287219 3.1 e318/256 3:89f2ebc9:::benchmark_data_smithi169_51390_object123435:head v 318'18105, mlcod=317'18103)", 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.510625+0000", 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.250345921999994, 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48802204399999999, 2024-02-12T16:13:49.827 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510625+0000", 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510625+0000", 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510623+0000", 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1273e-05 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510634+0000", 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0213e-05 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510644+0000", 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.728e-06 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510650+0000", 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9405e-05 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510690+0000", 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4265e-05 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510704+0000", 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0051727550000000002 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515877+0000", 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48277020799999998 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998647+0000", 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48802204399999999 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:287218 3.a e318/256 3:5667ef8a:::benchmark_data_smithi169_51390_object123434:head v 318'17892, mlcod=317'17889)", 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.510909+0000", 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.250061916000007, 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.496164826, 2024-02-12T16:13:49.831 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510909+0000", 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510909+0000", 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510907+0000", 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5279999999999992e-06 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510917+0000", 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5799999999999998e-06 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510926+0000", 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6380000000000001e-06 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510932+0000", 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000207667 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511140+0000", 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7283e-05 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511157+0000", 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49582724099999997 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006984+0000", 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9481999999999995e-05 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007074+0000", 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.496164826 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287223 3.3 3:c6e83110:::benchmark_data_smithi169_51390_object123439:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:13:49.835 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.512406+0000", 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.248564587000004, 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48604857000000001, 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287223 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512406+0000", 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512406+0000", 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512404+0000", 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9990000000000008e-06 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512412+0000", 2024-02-12T16:13:49.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1491999999999999e-05 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512423+0000", 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3419999999999996e-06 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512429+0000", 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2413000000000002e-05 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512472+0000", 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112315 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512584+0000", 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117563 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.512702+0000", 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48458367699999999 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997285+0000", 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001098882 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998384+0000", 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8485999999999997e-05 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998423+0000", 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1930000000000001e-05 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998455+0000", 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48604857000000001 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:287226 3.1 e318/256 3:8046e782:::benchmark_data_smithi169_51390_object123442:head v 318'18106, mlcod=317'18103)", 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.515869+0000", 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.245101890000001, 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48923048000000002, 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515869+0000", 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515869+0000", 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515867+0000", 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3638999999999999e-05 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515880+0000", 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47858906099999998 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994470+0000", 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6475999999999999e-05 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994486+0000", 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041998799999999996 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998686+0000", 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2708999999999999e-05 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998709+0000", 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063327339999999996 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.005041+0000", 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7871999999999997e-05 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.005099+0000", 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48923048000000002 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:287232 3.a e318/256 3:5fcb2087:::benchmark_data_smithi169_51390_object123448:head v 318'17893, mlcod=317'17889)", 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.524540+0000", 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.236430904000002, 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48256044999999997, 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524540+0000", 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524540+0000", 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524538+0000", 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.259e-06 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524546+0000", 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46988074299999999 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994427+0000", 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6650000000000001e-05 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994454+0000", 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011700000000001 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994634+0000", 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0941000000000001e-05 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994665+0000", 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012414786000000001 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007080+0000", 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0624000000000001e-05 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007100+0000", 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48256044999999997 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:287236 3.e e318/256 3:713cbbd1:::benchmark_data_smithi169_51390_object123452:head v 318'17918, mlcod=318'17916)", 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.530752+0000", 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.230218925000003, 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48181084400000002, 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.530752+0000", 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.530752+0000", 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.530750+0000", 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3885e-05 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.530764+0000", 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46691142899999999 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997675+0000", 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9572e-05 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997715+0000", 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000288757 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998003+0000", 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4160999999999998e-05 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998067+0000", 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014450954 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.012518+0000", 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4178999999999998e-05 2024-02-12T16:13:49.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.012563+0000", 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48181084400000002 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:293182 3.3 3:cc7a69b3:::benchmark_data_smithi169_51390_object129398:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:40.530916+0000", 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 69.230054589999995, 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22228504199999999, 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.853 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 293182 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.530916+0000", 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.530916+0000", 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.530914+0000", 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8449999999999997e-06 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.530921+0000", 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2184999999999999e-05 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.530933+0000", 2024-02-12T16:13:49.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2270000000000001e-06 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.530938+0000", 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5521e-05 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.530984+0000", 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5725999999999998e-05 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.531079+0000", 2024-02-12T16:13:49.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9088999999999999e-05 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.531178+0000", 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040283480000000002 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535207+0000", 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21790420099999999 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.753111+0000", 2024-02-12T16:13:49.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2818999999999997e-05 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.753164+0000", 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7264000000000001e-05 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.753201+0000", 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22228504199999999 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:293185 3.3 3:ccefaf2c:::benchmark_data_smithi169_51390_object129401:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:40.535183+0000", 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 69.225788031999997, 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.222730606, 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 293185 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535183+0000", 2024-02-12T16:13:49.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535183+0000", 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535181+0000", 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5540000000000003e-06 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535188+0000", 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4884e-05 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535202+0000", 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535209+0000", 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3015999999999997e-05 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535242+0000", 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0029000000000002e-05 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535322+0000", 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5116000000000002e-05 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.535417+0000", 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041657669999999999 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.539583+0000", 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21825456900000001 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.757838+0000", 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4537999999999999e-05 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.757882+0000", 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0988000000000002e-05 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.757913+0000", 2024-02-12T16:13:49.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.222730606 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:293211 3.3 3:c9245688:::benchmark_data_smithi169_51390_object129427:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:40.557224+0000", 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 69.203746792999993, 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20078874399999999, 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:13:49.864 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 293211 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557224+0000", 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557224+0000", 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557221+0000", 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5739999999999996e-06 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557230+0000", 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2081e-05 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557242+0000", 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6319999999999996e-06 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557248+0000", 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1560999999999997e-05 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557289+0000", 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8583000000000001e-05 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557388+0000", 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100351 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.557488+0000", 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004023263 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.561511+0000", 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.196444007 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.757955+0000", 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1798999999999998e-05 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.757987+0000", 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5401e-05 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:40.758013+0000", 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20078874399999999 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:13:49.869 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:13:49.870 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:13:49.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:13:49.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:49.905 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:49.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:50.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:50.042 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:50.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:50.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:50.152 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:13:50.183 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:50.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:50.187 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:13:50.211 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:13:50.232 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:13:50.254 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:13:50.278 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:13:50.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:50.300 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:13:50.325 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:13:50.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:50.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:50.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:13:50.370 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:13:50.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:13:50.393 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:13:50.420 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:13:50.421 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.421 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.421 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.422 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:13:50.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:50.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:50.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:50.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:50.609 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:50.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:50.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:50.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:50.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:50.828 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:13:50.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:50.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:50 smithi169 bash[19604]: cluster 2024-02-12T16:13:49.695919+0000 mgr.y (mgr.14146) 853 : cluster [DBG] pgmap v1092: 17 pgs: 1 creating+peering, 5 unknown, 11 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:50.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:50 smithi169 bash[19604]: audit 2024-02-12T16:13:50.122816+0000 mon.a (mon.0) 1329 : audit [DBG] from='client.? v1:172.21.15.121:0/2137637481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:50.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:50 smithi169 bash[19604]: audit 2024-02-12T16:13:50.352979+0000 mon.a (mon.0) 1330 : audit [INF] from='client.? v1:172.21.15.121:0/782013579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:13:50.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:50.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:50.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:51.025 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:51.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:51.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:51.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:50 smithi123 bash[19119]: cluster 2024-02-12T16:13:49.695919+0000 mgr.y (mgr.14146) 853 : cluster [DBG] pgmap v1092: 17 pgs: 1 creating+peering, 5 unknown, 11 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:51.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:50 smithi123 bash[19119]: audit 2024-02-12T16:13:50.122816+0000 mon.a (mon.0) 1329 : audit [DBG] from='client.? v1:172.21.15.121:0/2137637481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:51.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:50 smithi123 bash[19119]: audit 2024-02-12T16:13:50.352979+0000 mon.a (mon.0) 1330 : audit [INF] from='client.? v1:172.21.15.121:0/782013579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:13:51.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:50 smithi121 bash[18489]: cluster 2024-02-12T16:13:49.695919+0000 mgr.y (mgr.14146) 853 : cluster [DBG] pgmap v1092: 17 pgs: 1 creating+peering, 5 unknown, 11 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:51.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:50 smithi121 bash[18489]: audit 2024-02-12T16:13:50.122816+0000 mon.a (mon.0) 1329 : audit [DBG] from='client.? v1:172.21.15.121:0/2137637481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:13:51.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:50 smithi121 bash[18489]: audit 2024-02-12T16:13:50.352979+0000 mon.a (mon.0) 1330 : audit [INF] from='client.? v1:172.21.15.121:0/782013579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:13:51.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:51.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:51.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:51.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:51.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:51.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:51.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:51.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:51.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:51.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:51.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:51.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:51.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:51.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:51.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:51.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:51.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:51.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:51.985 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:51 smithi169 bash[19604]: cluster 2024-02-12T16:13:50.805222+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:51.985 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:51 smithi169 bash[19604]: audit 2024-02-12T16:13:50.813893+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? v1:172.21.15.121:0/782013579' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:13:51.985 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:51 smithi169 bash[19604]: cluster 2024-02-12T16:13:50.813941+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e337: 12 total, 12 up, 12 in 2024-02-12T16:13:52.010 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:52.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:52.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:51 smithi123 bash[19119]: cluster 2024-02-12T16:13:50.805222+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:52.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:51 smithi123 bash[19119]: audit 2024-02-12T16:13:50.813893+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? v1:172.21.15.121:0/782013579' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:13:52.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:51 smithi123 bash[19119]: cluster 2024-02-12T16:13:50.813941+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e337: 12 total, 12 up, 12 in 2024-02-12T16:13:52.149 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:51 smithi121 bash[18489]: cluster 2024-02-12T16:13:50.805222+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:13:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:51 smithi121 bash[18489]: audit 2024-02-12T16:13:50.813893+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? v1:172.21.15.121:0/782013579' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:13:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:51 smithi121 bash[18489]: cluster 2024-02-12T16:13:50.813941+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e337: 12 total, 12 up, 12 in 2024-02-12T16:13:52.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:52.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:52.431 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:52.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:52.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:52.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:52.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:52.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:52.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:52.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:52.992 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:52.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:52 smithi169 bash[19604]: cluster 2024-02-12T16:13:51.696435+0000 mgr.y (mgr.14146) 854 : cluster [DBG] pgmap v1094: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:53.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:53.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:53.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:53.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:52 smithi123 bash[19119]: cluster 2024-02-12T16:13:51.696435+0000 mgr.y (mgr.14146) 854 : cluster [DBG] pgmap v1094: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:52 smithi121 bash[18489]: cluster 2024-02-12T16:13:51.696435+0000 mgr.y (mgr.14146) 854 : cluster [DBG] pgmap v1094: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:53.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:13:53.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:53.272 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:53.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:53.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:53.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:53.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:53.560 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:53.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:53.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:53.704 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:53.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:53.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:13:53.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:53.841 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:53.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:53.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:53.984 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:53.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:54.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:54.124 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:54.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:54.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:54.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:54.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:54.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:54.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:54.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:54.515 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:54.550 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:54.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:54.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:54.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:54.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:54.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:54.957 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:54.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:54 smithi169 bash[19604]: cluster 2024-02-12T16:13:53.696866+0000 mgr.y (mgr.14146) 855 : cluster [DBG] pgmap v1095: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:55.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:55.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:55.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:54 smithi121 bash[18489]: cluster 2024-02-12T16:13:53.696866+0000 mgr.y (mgr.14146) 855 : cluster [DBG] pgmap v1095: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:55.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:54 smithi123 bash[19119]: cluster 2024-02-12T16:13:53.696866+0000 mgr.y (mgr.14146) 855 : cluster [DBG] pgmap v1095: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:55.179 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:13:55.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:55.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:55.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:55.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:55.369 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:55.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:55.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:55.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:55.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:55.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:55.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:55.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:55.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:55.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:55.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:55.930 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:55.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:56.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:56.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:56.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:56.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:56.210 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:56.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:56.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:56.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:56.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:56.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:56.391 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-02-12T16:13:56.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:56.488 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:56 smithi123 bash[19119]: audit 2024-02-12T16:13:55.613043+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? v1:172.21.15.121:0/4141501580' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:13:56.488 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:56 smithi123 bash[19119]: cluster 2024-02-12T16:13:55.697495+0000 mgr.y (mgr.14146) 856 : cluster [DBG] pgmap v1096: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:56.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:13:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:56.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:13:56.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:56.628 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:56.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:56.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:56 smithi121 bash[18489]: audit 2024-02-12T16:13:55.613043+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? v1:172.21.15.121:0/4141501580' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:13:56.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:56 smithi121 bash[18489]: cluster 2024-02-12T16:13:55.697495+0000 mgr.y (mgr.14146) 856 : cluster [DBG] pgmap v1096: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:56.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:56.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:56.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:56 smithi169 bash[19604]: audit 2024-02-12T16:13:55.613043+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? v1:172.21.15.121:0/4141501580' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:13:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:56 smithi169 bash[19604]: cluster 2024-02-12T16:13:55.697495+0000 mgr.y (mgr.14146) 856 : cluster [DBG] pgmap v1096: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:56.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:56.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:56.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:57.011 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:57.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:57.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:57.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:57.183 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:57.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:57.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:57.324 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:57.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:57.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:57.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:13:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:57.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:13:57.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:57.584 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:57 smithi123 bash[19119]: audit 2024-02-12T16:13:56.379851+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.121:0/4141501580' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:13:57.584 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:57 smithi123 bash[19119]: cluster 2024-02-12T16:13:56.379930+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e338: 12 total, 12 up, 12 in 2024-02-12T16:13:57.606 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:57.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:57 smithi121 bash[18489]: audit 2024-02-12T16:13:56.379851+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.121:0/4141501580' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:13:57.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:57 smithi121 bash[18489]: cluster 2024-02-12T16:13:56.379930+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e338: 12 total, 12 up, 12 in 2024-02-12T16:13:57.708 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:57.747 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:57.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:57.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:57.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:57 smithi169 bash[19604]: audit 2024-02-12T16:13:56.379851+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? v1:172.21.15.121:0/4141501580' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:13:57.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:57 smithi169 bash[19604]: cluster 2024-02-12T16:13:56.379930+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e338: 12 total, 12 up, 12 in 2024-02-12T16:13:57.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:57.888 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:13:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:57.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:13:57.992 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:58.029 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:58.113 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-02-12T16:13:58.113 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-02-12T16:13:58.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:13:58.116 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:13:58.116 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:13:58.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:58.155 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:13:58.155 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:13:58.155 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_60938 2024-02-12T16:13:58.155 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:13:58.155 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:13:58.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:13:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:13:58.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:58.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:58.342 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:13:58.343 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:13:58.415 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:58.439 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:58.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:58.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:58 smithi169 bash[19604]: cluster 2024-02-12T16:13:57.698097+0000 mgr.y (mgr.14146) 857 : cluster [DBG] pgmap v1098: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:58.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:58 smithi123 bash[19119]: cluster 2024-02-12T16:13:57.698097+0000 mgr.y (mgr.14146) 857 : cluster [DBG] pgmap v1098: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:58.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:13:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:13:58.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:58 smithi121 bash[18489]: cluster 2024-02-12T16:13:57.698097+0000 mgr.y (mgr.14146) 857 : cluster [DBG] pgmap v1098: 17 pgs: 17 active+clean; 0 B data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:13:58.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:58.702 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:13:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:13:58.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:58.831 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:13:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:13:58.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:58.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:59.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:59.089 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:59.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:59.155 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 16 6147 6131 383.168 383.188 0.00201955 0.00260518 2024-02-12T16:13:59.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:59.222 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:13:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:59.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:13:59.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:13:59.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:59.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:59.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:59.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:59.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:13:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:59.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:13:59.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:13:59 smithi169 bash[19604]: audit 2024-02-12T16:13:59.481620+0000 mon.a (mon.0) 1337 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:13:59.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:59.610 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:59.610 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:13:59.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:13:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:59.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:13:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:13:59 smithi121 bash[18489]: audit 2024-02-12T16:13:59.481620+0000 mon.a (mon.0) 1337 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:13:59.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:13:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:59.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:13:59.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:59.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:13:59 smithi123 bash[19119]: audit 2024-02-12T16:13:59.481620+0000 mon.a (mon.0) 1337 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:13:59.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:13:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:13:59.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:13:59.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:00.023 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:00.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:00.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 16 11571 11555 361.054 339 0.00337286 0.00276563 2024-02-12T16:14:00.156 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:00.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:00.285 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:00.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:00.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:00.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:00.550 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:00.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:00.661 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:00 smithi169 bash[19604]: cluster 2024-02-12T16:13:59.698607+0000 mgr.y (mgr.14146) 858 : cluster [DBG] pgmap v1099: 17 pgs: 17 active+clean; 136 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:00.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:00.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:00.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:00.814 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:00 smithi123 bash[19119]: cluster 2024-02-12T16:13:59.698607+0000 mgr.y (mgr.14146) 858 : cluster [DBG] pgmap v1099: 17 pgs: 17 active+clean; 136 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:00 smithi121 bash[18489]: cluster 2024-02-12T16:13:59.698607+0000 mgr.y (mgr.14146) 858 : cluster [DBG] pgmap v1099: 17 pgs: 17 active+clean; 136 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:00.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:00.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:00.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:01.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:01.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:01.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:01.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 16958 16942 352.922 336.688 0.00329589 0.0028308 2024-02-12T16:14:01.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:01.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:01.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:01.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:01.346 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:01.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:01.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:01.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:01.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:01.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:01.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:01.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:01.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:01.739 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:01.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:01.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:01.869 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:01.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:01.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:02.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:02.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:02.132 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:02.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 16 21798 21782 340.31 302.5 0.00353864 0.00293612 2024-02-12T16:14:02.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:02.261 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:02.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:02.392 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:02.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:02.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:02.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:02.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:02.659 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:02.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:02.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:02.786 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:02 smithi121 bash[18489]: cluster 2024-02-12T16:14:01.699050+0000 mgr.y (mgr.14146) 859 : cluster [DBG] pgmap v1100: 17 pgs: 17 active+clean; 513 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:02.887 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:02.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:02 smithi123 bash[19119]: cluster 2024-02-12T16:14:01.699050+0000 mgr.y (mgr.14146) 859 : cluster [DBG] pgmap v1100: 17 pgs: 17 active+clean; 513 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:02.913 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:02.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:03.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:03.041 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:03.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:02 smithi169 bash[19604]: cluster 2024-02-12T16:14:01.699050+0000 mgr.y (mgr.14146) 859 : cluster [DBG] pgmap v1100: 17 pgs: 17 active+clean; 513 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:03.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:03.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 26125 26109 326.331 270.438 0.00364759 0.00306262 2024-02-12T16:14:03.172 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:03.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:03.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:03.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:03.433 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:03.536 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:03.564 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:03.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:03.704 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:03.776 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:03 smithi169 bash[19604]: cluster 2024-02-12T16:14:02.717201+0000 osd.1 (osd.1) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:03.776 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:03 smithi169 bash[19604]: cluster 2024-02-12T16:14:02.717250+0000 osd.1 (osd.1) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:03.776 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:03 smithi169 bash[19604]: cluster 2024-02-12T16:14:02.763950+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e339: 12 total, 12 up, 12 in 2024-02-12T16:14:03.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:03.818 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:03 smithi123 bash[19119]: cluster 2024-02-12T16:14:02.717201+0000 osd.1 (osd.1) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:03.818 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:03 smithi123 bash[19119]: cluster 2024-02-12T16:14:02.717250+0000 osd.1 (osd.1) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:03.818 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:03 smithi123 bash[19119]: cluster 2024-02-12T16:14:02.763950+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e339: 12 total, 12 up, 12 in 2024-02-12T16:14:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:03.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:03.948 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:03 smithi121 bash[18489]: cluster 2024-02-12T16:14:02.717201+0000 osd.1 (osd.1) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:03.948 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:03 smithi121 bash[18489]: cluster 2024-02-12T16:14:02.717250+0000 osd.1 (osd.1) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:03.948 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:03 smithi121 bash[18489]: cluster 2024-02-12T16:14:02.763950+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e339: 12 total, 12 up, 12 in 2024-02-12T16:14:03.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:03.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:04.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.078 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:14:04.105 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:04.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:04.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 30224 30208 314.636 256.188 0.00186805 0.00317631 2024-02-12T16:14:04.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.227 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:04.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:04.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:04.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:04.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:04.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:04.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:04.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:04.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:04.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:04.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:04.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:04.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:04 smithi121 bash[18489]: cluster 2024-02-12T16:14:03.699712+0000 mgr.y (mgr.14146) 860 : cluster [DBG] pgmap v1102: 17 pgs: 17 active+clean; 758 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:04.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:05.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:05.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:04 smithi169 bash[19604]: cluster 2024-02-12T16:14:03.699712+0000 mgr.y (mgr.14146) 860 : cluster [DBG] pgmap v1102: 17 pgs: 17 active+clean; 758 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:05.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:05.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 16 31230 31214 278.67 62.875 0.00263288 0.00358613 2024-02-12T16:14:05.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:05.201 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:04 smithi123 bash[19119]: cluster 2024-02-12T16:14:03.699712+0000 mgr.y (mgr.14146) 860 : cluster [DBG] pgmap v1102: 17 pgs: 17 active+clean; 758 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:05.220 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:05.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:05.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:05.352 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:05.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:05.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:05.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:05.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:05.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:05.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:05.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:05.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:05.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:05.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:05.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:05.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:05.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:05.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:05.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:05.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:05 smithi169 bash[19604]: audit 2024-02-12T16:14:04.931301+0000 mon.a (mon.0) 1339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:14:05.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:05 smithi169 bash[19604]: audit 2024-02-12T16:14:05.289316+0000 mon.a (mon.0) 1340 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:14:05.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:05 smithi169 bash[19604]: audit 2024-02-12T16:14:05.290325+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:14:05.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:05 smithi169 bash[19604]: audit 2024-02-12T16:14:05.296257+0000 mon.a (mon.0) 1342 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:14:06.013 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:06.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:06.145 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:06.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 16 33991 33975 265.405 172.562 0.0028443 0.00376612 2024-02-12T16:14:06.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:06.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:06.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:05 smithi123 bash[19119]: audit 2024-02-12T16:14:04.931301+0000 mon.a (mon.0) 1339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:14:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:05 smithi123 bash[19119]: audit 2024-02-12T16:14:05.289316+0000 mon.a (mon.0) 1340 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:14:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:05 smithi123 bash[19119]: audit 2024-02-12T16:14:05.290325+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:14:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:05 smithi123 bash[19119]: audit 2024-02-12T16:14:05.296257+0000 mon.a (mon.0) 1342 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:14:06.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:05 smithi121 bash[18489]: audit 2024-02-12T16:14:04.931301+0000 mon.a (mon.0) 1339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:14:06.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:05 smithi121 bash[18489]: audit 2024-02-12T16:14:05.289316+0000 mon.a (mon.0) 1340 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:14:06.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:05 smithi121 bash[18489]: audit 2024-02-12T16:14:05.290325+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:14:06.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:05 smithi121 bash[18489]: audit 2024-02-12T16:14:05.296257+0000 mon.a (mon.0) 1342 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:14:06.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:06.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:06.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:06.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:06.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:06.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:06.830 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:06.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:06.962 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:06.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:06 smithi169 bash[19604]: cluster 2024-02-12T16:14:05.700155+0000 mgr.y (mgr.14146) 861 : cluster [DBG] pgmap v1103: 17 pgs: 17 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:06 smithi169 bash[19604]: cluster 2024-02-12T16:14:05.848851+0000 osd.10 (osd.10) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:06 smithi169 bash[19604]: cluster 2024-02-12T16:14:05.848972+0000 osd.10 (osd.10) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:06 smithi169 bash[19604]: cluster 2024-02-12T16:14:06.386116+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e340: 12 total, 12 up, 12 in 2024-02-12T16:14:07.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:07.092 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:07.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:07.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 35394 35378 245.658 87.6875 0.00665966 0.0040691 2024-02-12T16:14:07.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:07.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:07.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:07.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:06 smithi123 bash[19119]: cluster 2024-02-12T16:14:05.700155+0000 mgr.y (mgr.14146) 861 : cluster [DBG] pgmap v1103: 17 pgs: 17 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:07.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:06 smithi123 bash[19119]: cluster 2024-02-12T16:14:05.848851+0000 osd.10 (osd.10) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:07.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:06 smithi123 bash[19119]: cluster 2024-02-12T16:14:05.848972+0000 osd.10 (osd.10) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:07.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:06 smithi123 bash[19119]: cluster 2024-02-12T16:14:06.386116+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e340: 12 total, 12 up, 12 in 2024-02-12T16:14:07.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:06 smithi121 bash[18489]: cluster 2024-02-12T16:14:05.700155+0000 mgr.y (mgr.14146) 861 : cluster [DBG] pgmap v1103: 17 pgs: 17 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:14:07.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:06 smithi121 bash[18489]: cluster 2024-02-12T16:14:05.848851+0000 osd.10 (osd.10) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:07.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:06 smithi121 bash[18489]: cluster 2024-02-12T16:14:05.848972+0000 osd.10 (osd.10) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:07.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:06 smithi121 bash[18489]: cluster 2024-02-12T16:14:06.386116+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e340: 12 total, 12 up, 12 in 2024-02-12T16:14:07.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:07.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:07.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:07.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:07.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:07.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:07.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:07.616 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:07.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:07.719 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:07.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:07.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:07.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:07.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:08.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:08.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:08.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:08.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 15 38903 38888 243.028 219.375 0.00287251 0.0041118 2024-02-12T16:14:08.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:08.280 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:08.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:08.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:08.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:08.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:08.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:08.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:08.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:08.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:08.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:08.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:08.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:08.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:08 smithi169 bash[19604]: cluster 2024-02-12T16:14:07.700600+0000 mgr.y (mgr.14146) 862 : cluster [DBG] pgmap v1105: 17 pgs: 17 active+clean; 1.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-02-12T16:14:09.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:09.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:09.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:09.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 41757 41741 237.143 178.312 0.00218804 0.0042139 2024-02-12T16:14:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:08 smithi121 bash[18489]: cluster 2024-02-12T16:14:07.700600+0000 mgr.y (mgr.14146) 862 : cluster [DBG] pgmap v1105: 17 pgs: 17 active+clean; 1.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-02-12T16:14:09.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:09.199 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:08 smithi123 bash[19119]: cluster 2024-02-12T16:14:07.700600+0000 mgr.y (mgr.14146) 862 : cluster [DBG] pgmap v1105: 17 pgs: 17 active+clean; 1.9 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-02-12T16:14:09.218 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:09.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:09.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:09.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:09.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:09.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:09.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:09.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:09.608 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:09.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:09.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:09.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:09.745 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:14:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:09.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:09.773 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:14:09.788 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:14:09.788 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:14:09.792 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:14:09.828 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:14:09.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:09.855 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:14:09.877 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:09.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:09.885 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:14:09.917 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:14:09.945 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:14:09.950 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:14:09.972 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:14:09.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:09.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:09 smithi121 bash[18489]: audit 2024-02-12T16:14:09.427304+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.121:0/2828337324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:14:09.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:09 smithi121 bash[18489]: audit 2024-02-12T16:14:09.703965+0000 mon.a (mon.0) 1345 : audit [DBG] from='client.? v1:172.21.15.121:0/2831975700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:10.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:14:10.008 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:10.029 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:14:10.055 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:14:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.057 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.058 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:14:10.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:10.118 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:09 smithi123 bash[19119]: audit 2024-02-12T16:14:09.427304+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.121:0/2828337324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:14:10.118 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:09 smithi123 bash[19119]: audit 2024-02-12T16:14:09.703965+0000 mon.a (mon.0) 1345 : audit [DBG] from='client.? v1:172.21.15.121:0/2831975700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:10.135 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:10.156 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 44852 44836 233.5 193.438 0.00549414 0.00428075 2024-02-12T16:14:10.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:10.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:09 smithi169 bash[19604]: audit 2024-02-12T16:14:09.427304+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.121:0/2828337324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:14:10.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:09 smithi169 bash[19604]: audit 2024-02-12T16:14:09.703965+0000 mon.a (mon.0) 1345 : audit [DBG] from='client.? v1:172.21.15.121:0/2831975700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:10.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:10.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:10.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:10.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:10.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:10.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:10.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:10.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:14:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:10.772 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:10.799 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:10.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:10.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:10.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:11.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:11.074 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:11.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 47522 47506 228.374 166.875 0.00621058 0.00437703 2024-02-12T16:14:11.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:11.201 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:10 smithi121 bash[18489]: cluster 2024-02-12T16:14:09.701046+0000 mgr.y (mgr.14146) 863 : cluster [DBG] pgmap v1106: 17 pgs: 17 active+clean; 2.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 194 MiB/s wr, 3.11k op/s 2024-02-12T16:14:11.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:10 smithi121 bash[18489]: cluster 2024-02-12T16:14:09.932227+0000 mon.a (mon.0) 1346 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:14:11.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:10 smithi121 bash[18489]: audit 2024-02-12T16:14:09.941441+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? v1:172.21.15.121:0/2828337324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:14:11.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:10 smithi121 bash[18489]: cluster 2024-02-12T16:14:09.941586+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e341: 12 total, 12 up, 12 in 2024-02-12T16:14:11.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:11.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:11.310 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:10 smithi123 bash[19119]: cluster 2024-02-12T16:14:09.701046+0000 mgr.y (mgr.14146) 863 : cluster [DBG] pgmap v1106: 17 pgs: 17 active+clean; 2.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 194 MiB/s wr, 3.11k op/s 2024-02-12T16:14:11.311 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:10 smithi123 bash[19119]: cluster 2024-02-12T16:14:09.932227+0000 mon.a (mon.0) 1346 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:14:11.311 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:10 smithi123 bash[19119]: audit 2024-02-12T16:14:09.941441+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? v1:172.21.15.121:0/2828337324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:14:11.311 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:10 smithi123 bash[19119]: cluster 2024-02-12T16:14:09.941586+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e341: 12 total, 12 up, 12 in 2024-02-12T16:14:11.328 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:11.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:10 smithi169 bash[19604]: cluster 2024-02-12T16:14:09.701046+0000 mgr.y (mgr.14146) 863 : cluster [DBG] pgmap v1106: 17 pgs: 17 active+clean; 2.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 194 MiB/s wr, 3.11k op/s 2024-02-12T16:14:11.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:10 smithi169 bash[19604]: cluster 2024-02-12T16:14:09.932227+0000 mon.a (mon.0) 1346 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:14:11.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:10 smithi169 bash[19604]: audit 2024-02-12T16:14:09.941441+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? v1:172.21.15.121:0/2828337324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:14:11.353 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:10 smithi169 bash[19604]: cluster 2024-02-12T16:14:09.941586+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e341: 12 total, 12 up, 12 in 2024-02-12T16:14:11.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:11.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:11.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:11.591 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:11.695 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:11.722 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:11.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:11.851 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:11.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:11.988 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:11.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:12.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.124 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:12.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:12.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 50507 50491 225.386 186.562 0.00637148 0.00443487 2024-02-12T16:14:12.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:12.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:12.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:12.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:12.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:12.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:12.403 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:14:12.411 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:14:12.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:12.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:12.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:12.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:12.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:12.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:12.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.924 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:12.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:13.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:13.033 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:12 smithi123 bash[19119]: cluster 2024-02-12T16:14:11.701396+0000 mgr.y (mgr.14146) 864 : cluster [DBG] pgmap v1108: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 223 MiB/s wr, 3.56k op/s 2024-02-12T16:14:13.053 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:13.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 53401 53385 222.418 180.875 0.00327691 0.00449439 2024-02-12T16:14:13.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:13.182 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:13.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:12 smithi169 bash[19604]: cluster 2024-02-12T16:14:11.701396+0000 mgr.y (mgr.14146) 864 : cluster [DBG] pgmap v1108: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 223 MiB/s wr, 3.56k op/s 2024-02-12T16:14:13.308 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:12 smithi121 bash[18489]: cluster 2024-02-12T16:14:11.701396+0000 mgr.y (mgr.14146) 864 : cluster [DBG] pgmap v1108: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 223 MiB/s wr, 3.56k op/s 2024-02-12T16:14:13.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:13.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:13.553 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1108,"stamp":"2024-02-12T16:14:11.701229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2662858752,"num_objects":40632,"num_object_clones":0,"num_object_copies":81264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40632,"num_write_kb":2600448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40632,"ondisk_log_size":40632,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":8616832,"kb_used_data":4964128,"kb_used_omap":0,"kb_used_meta":3652608,"kb_avail":1116324992,"statfs":{"total":1151940427776,"available":1143116791808,"internally_reserved":0,"allocated":5083267072,"data_stored":5078509525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3740270592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1868496896,"num_objects":28511,"num_object_clones":0,"num_object_copies":57022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28511,"num_write_kb":1824704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001741"},"pg_stats":[{"pgid":"4.f","version":"340'2205","reported_seq":2218,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:06.814878+0000","last_change":"2024-02-12T16:13:45.788790+0000","last_active":"2024-02-12T16:14:06.814878+0000","last_peered":"2024-02-12T16:14:06.814878+0000","last_clean":"2024-02-12T16:14:06.814878+0000","last_became_active":"2024-02-12T16:13:45.788522+0000","last_became_peered":"2024-02-12T16:13:45.788522+0000","last_unstale":"2024-02-12T16:14:06.814878+0000","last_undegraded":"2024-02-12T16:14:06.814878+0000","last_fullsized":"2024-02-12T16:14:06.814878+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2205,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":144506880,"num_objects":2205,"num_object_clones":0,"num_object_copies":4410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2205,"num_write_kb":141120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"340'2771","reported_seq":2784,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.917843+0000","last_change":"2024-02-12T16:13:45.970411+0000","last_active":"2024-02-12T16:14:09.917843+0000","last_peered":"2024-02-12T16:14:09.917843+0000","last_clean":"2024-02-12T16:14:09.917843+0000","last_became_active":"2024-02-12T16:13:45.970126+0000","last_became_peered":"2024-02-12T16:13:45.970126+0000","last_unstale":"2024-02-12T16:14:09.917843+0000","last_undegraded":"2024-02-12T16:14:09.917843+0000","last_fullsized":"2024-02-12T16:14:09.917843+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2771,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":181600256,"num_objects":2771,"num_object_clones":0,"num_object_copies":5542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2771,"num_write_kb":177344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"340'2735","reported_seq":2748,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.922223+0000","last_change":"2024-02-12T16:13:45.794115+0000","last_active":"2024-02-12T16:14:09.922223+0000","last_peered":"2024-02-12T16:14:09.922223+0000","last_clean":"2024-02-12T16:14:09.922223+0000","last_became_active":"2024-02-12T16:13:45.793796+0000","last_became_peered":"2024-02-12T16:13:45.793796+0000","last_unstale":"2024-02-12T16:14:09.922223+0000","last_undegraded":"2024-02-12T16:14:09.922223+0000","last_fullsized":"2024-02-12T16:14:09.922223+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2735,"ondisk_log_size":2735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":5470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2735,"num_write_kb":175040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"340'2739","reported_seq":2751,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.923804+0000","last_change":"2024-02-12T16:13:45.791824+0000","last_active":"2024-02-12T16:14:09.923804+0000","last_peered":"2024-02-12T16:14:09.923804+0000","last_clean":"2024-02-12T16:14:09.923804+0000","last_became_active":"2024-02-12T16:13:45.791545+0000","last_became_peered":"2024-02-12T16:13:45.791545+0000","last_unstale":"2024-02-12T16:14:09.923804+0000","last_undegraded":"2024-02-12T16:14:09.923804+0000","last_fullsized":"2024-02-12T16:14:09.923804+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2739,"ondisk_log_size":2739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179503104,"num_objects":2739,"num_object_clones":0,"num_object_copies":5478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2739,"num_write_kb":175296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"341'2847","reported_seq":2861,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-02-12T16:14:10.442909+0000","last_change":"2024-02-12T16:13:45.790724+0000","last_active":"2024-02-12T16:14:10.442909+0000","last_peered":"2024-02-12T16:14:10.442909+0000","last_clean":"2024-02-12T16:14:10.442909+0000","last_became_active":"2024-02-12T16:13:45.790415+0000","last_became_peered":"2024-02-12T16:13:45.790415+0000","last_unstale":"2024-02-12T16:14:10.442909+0000","last_undegraded":"2024-02-12T16:14:10.442909+0000","last_fullsized":"2024-02-12T16:14:10.442909+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2847,"ondisk_log_size":2847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":186580992,"num_objects":2847,"num_object_clones":0,"num_object_copies":5694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2847,"num_write_kb":182208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"340'2618","reported_seq":2631,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:08.980183+0000","last_change":"2024-02-12T16:13:45.790198+0000","last_active":"2024-02-12T16:14:08.980183+0000","last_peered":"2024-02-12T16:14:08.980183+0000","last_clean":"2024-02-12T16:14:08.980183+0000","last_became_active":"2024-02-12T16:13:45.789678+0000","last_became_peered":"2024-02-12T16:13:45.789678+0000","last_unstale":"2024-02-12T16:14:08.980183+0000","last_undegraded":"2024-02-12T16:14:08.980183+0000","last_fullsized":"2024-02-12T16:14:08.980183+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2618,"ondisk_log_size":2618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":171573248,"num_objects":2618,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":167552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"340'2737","reported_seq":2750,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.799837+0000","last_change":"2024-02-12T16:13:45.795090+0000","last_active":"2024-02-12T16:14:09.799837+0000","last_peered":"2024-02-12T16:14:09.799837+0000","last_clean":"2024-02-12T16:14:09.799837+0000","last_became_active":"2024-02-12T16:13:45.794679+0000","last_became_peered":"2024-02-12T16:13:45.794679+0000","last_unstale":"2024-02-12T16:14:09.799837+0000","last_undegraded":"2024-02-12T16:14:09.799837+0000","last_fullsized":"2024-02-12T16:14:09.799837+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2737,"ondisk_log_size":2737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179372032,"num_objects":2737,"num_object_clones":0,"num_object_copies":5474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2737,"num_write_kb":175168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"340'2713","reported_seq":2726,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.800701+0000","last_change":"2024-02-12T16:13:45.793973+0000","last_active":"2024-02-12T16:14:09.800701+0000","last_peered":"2024-02-12T16:14:09.800701+0000","last_clean":"2024-02-12T16:14:09.800701+0000","last_became_active":"2024-02-12T16:13:45.793378+0000","last_became_peered":"2024-02-12T16:13:45.793378+0000","last_unstale":"2024-02-12T16:14:09.800701+0000","last_undegraded":"2024-02-12T16:14:09.800701+0000","last_fullsized":"2024-02-12T16:14:09.800701+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2713,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":177799168,"num_objects":2713,"num_object_clones":0,"num_object_copies":5426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2713,"num_write_kb":173632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"340'2098","reported_seq":2111,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:06.826882+0000","last_change":"2024-02-12T16:13:45.966753+0000","last_active":"2024-02-12T16:14:06.826882+0000","last_peered":"2024-02-12T16:14:06.826882+0000","last_clean":"2024-02-12T16:14:06.826882+0000","last_became_active":"2024-02-12T16:13:45.966345+0000","last_became_peered":"2024-02-12T16:13:45.966345+0000","last_unstale":"2024-02-12T16:14:06.826882+0000","last_undegraded":"2024-02-12T16:14:06.826882+0000","last_fullsized":"2024-02-12T16:14:06.826882+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2098,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":137494528,"num_objects":2098,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2098,"num_write_kb":134272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"339'2147","reported_seq":2158,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-02-12T16:14:06.163452+0000","last_change":"2024-02-12T16:13:45.793559+0000","last_active":"2024-02-12T16:14:06.163452+0000","last_peered":"2024-02-12T16:14:06.163452+0000","last_clean":"2024-02-12T16:14:06.163452+0000","last_became_active":"2024-02-12T16:13:45.793246+0000","last_became_peered":"2024-02-12T16:13:45.793246+0000","last_unstale":"2024-02-12T16:14:06.163452+0000","last_undegraded":"2024-02-12T16:14:06.163452+0000","last_fullsized":"2024-02-12T16:14:06.163452+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2147,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140705792,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":137408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"340'2187","reported_seq":2200,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:06.830579+0000","last_change":"2024-02-12T16:13:45.786420+0000","last_active":"2024-02-12T16:14:06.830579+0000","last_peered":"2024-02-12T16:14:06.830579+0000","last_clean":"2024-02-12T16:14:06.830579+0000","last_became_active":"2024-02-12T16:13:45.786192+0000","last_became_peered":"2024-02-12T16:13:45.786192+0000","last_unstale":"2024-02-12T16:14:06.830579+0000","last_undegraded":"2024-02-12T16:14:06.830579+0000","last_fullsized":"2024-02-12T16:14:06.830579+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2187,"ondisk_log_size":2187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":143327232,"num_objects":2187,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2187,"num_write_kb":139968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"340'2731","reported_seq":2743,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.785929+0000","last_change":"2024-02-12T16:13:45.795525+0000","last_active":"2024-02-12T16:14:09.785929+0000","last_peered":"2024-02-12T16:14:09.785929+0000","last_clean":"2024-02-12T16:14:09.785929+0000","last_became_active":"2024-02-12T16:13:45.794673+0000","last_became_peered":"2024-02-12T16:13:45.794673+0000","last_unstale":"2024-02-12T16:14:09.785929+0000","last_undegraded":"2024-02-12T16:14:09.785929+0000","last_fullsized":"2024-02-12T16:14:09.785929+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2731,"ondisk_log_size":2731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":5462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":174784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"339'2101","reported_seq":2114,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-02-12T16:14:06.162760+0000","last_change":"2024-02-12T16:13:45.969945+0000","last_active":"2024-02-12T16:14:06.162760+0000","last_peered":"2024-02-12T16:14:06.162760+0000","last_clean":"2024-02-12T16:14:06.162760+0000","last_became_active":"2024-02-12T16:13:45.969665+0000","last_became_peered":"2024-02-12T16:13:45.969665+0000","last_unstale":"2024-02-12T16:14:06.162760+0000","last_undegraded":"2024-02-12T16:14:06.162760+0000","last_fullsized":"2024-02-12T16:14:06.162760+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2101,"ondisk_log_size":2101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":137691136,"num_objects":2101,"num_object_clones":0,"num_object_copies":4202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":134464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"340'2711","reported_seq":2724,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.785083+0000","last_change":"2024-02-12T16:13:45.970010+0000","last_active":"2024-02-12T16:14:09.785083+0000","last_peered":"2024-02-12T16:14:09.785083+0000","last_clean":"2024-02-12T16:14:09.785083+0000","last_became_active":"2024-02-12T16:13:45.969700+0000","last_became_peered":"2024-02-12T16:13:45.969700+0000","last_unstale":"2024-02-12T16:14:09.785083+0000","last_undegraded":"2024-02-12T16:14:09.785083+0000","last_fullsized":"2024-02-12T16:14:09.785083+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2711,"ondisk_log_size":2711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":177668096,"num_objects":2711,"num_object_clones":0,"num_object_copies":5422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2711,"num_write_kb":173504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"340'2686","reported_seq":2699,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:09.751157+0000","last_change":"2024-02-12T16:13:45.788916+0000","last_active":"2024-02-12T16:14:09.751157+0000","last_peered":"2024-02-12T16:14:09.751157+0000","last_clean":"2024-02-12T16:14:09.751157+0000","last_became_active":"2024-02-12T16:13:45.788482+0000","last_became_peered":"2024-02-12T16:13:45.788482+0000","last_unstale":"2024-02-12T16:14:09.751157+0000","last_undegraded":"2024-02-12T16:14:09.751157+0000","last_fullsized":"2024-02-12T16:14:09.751157+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2686,"ondisk_log_size":2686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":176029696,"num_objects":2686,"num_object_clones":0,"num_object_copies":5372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2686,"num_write_kb":171904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"340'2606","reported_seq":2619,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-02-12T16:14:08.994511+0000","last_change":"2024-02-12T16:13:45.969991+0000","last_active":"2024-02-12T16:14:08.994511+0000","last_peered":"2024-02-12T16:14:08.994511+0000","last_clean":"2024-02-12T16:14:08.994511+0000","last_became_active":"2024-02-12T16:13:45.969639+0000","last_became_peered":"2024-02-12T16:13:45.969639+0000","last_unstale":"2024-02-12T16:14:08.994511+0000","last_undegraded":"2024-02-12T16:14:08.994511+0000","last_fullsized":"2024-02-12T16:14:08.994511+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":2606,"ondisk_log_size":2606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":5212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":166784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":371,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-02-12T16:14:02.766629+0000","last_change":"2024-02-12T16:13:43.045211+0000","last_active":"2024-02-12T16:14:02.766629+0000","last_peered":"2024-02-12T16:14:02.766629+0000","last_clean":"2024-02-12T16:14:02.766629+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:14:02.766629+0000","last_undegraded":"2024-02-12T16:14:02.766629+0000","last_fullsized":"2024-02-12T16:14:02.766629+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:43.044979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:13:43.044979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2662858752,"num_objects":40632,"num_object_clones":0,"num_object_copies":81264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40632,"num_write_kb":2600448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5182128128,"data_stored":5182128128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40632,"ondisk_log_size":40632,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958720,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965004,"kb_used_data":655492,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92780148,"statfs":{"total":95995035648,"available":95006871552,"internally_reserved":0,"allocated":671223808,"data_stored":670826546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.70999999999999996,"15min":0.82299999999999995},"min":{"1min":0.26800000000000002,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":1.554,"5min":3.6720000000000002,"15min":4.7850000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.35599999999999998,"5min":0.30599999999999999,"15min":0.25900000000000001},"max":{"1min":1.7170000000000001,"5min":4.992,"15min":4.992},"last":1.5720000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.85999999999999999,"15min":1.032},"min":{"1min":0.375,"5min":0.27900000000000003,"15min":0.25700000000000001},"max":{"1min":1.4470000000000001,"5min":4.851,"15min":6.7569999999999997},"last":2.96},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.95399999999999996,"15min":1.046},"min":{"1min":0.441,"5min":0.33600000000000002,"15min":0.27300000000000002},"max":{"1min":1.5649999999999999,"5min":5.0339999999999998,"15min":5.0339999999999998},"last":1.746}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0980000000000001,"15min":1.1259999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.7109999999999999,"5min":5.274,"15min":5.274},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":0.96999999999999997,"15min":1.1160000000000001},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":3.7360000000000002,"5min":5.375,"15min":5.375},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":3.056,"15min":1.7549999999999999},"min":{"1min":0.61799999999999999,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":1.6679999999999999,"5min":106.301,"15min":106.301},"last":1.9610000000000001},{"interface":"front","average":{"1min":1.252,"5min":3.121,"15min":1.8520000000000001},"min":{"1min":0.372,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.8610000000000002,"5min":106.437,"15min":106.437},"last":0.34599999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":4.702,"15min":2.3239999999999998},"min":{"1min":0.55900000000000005,"5min":0.39200000000000002,"15min":0.33100000000000002},"max":{"1min":2.052,"5min":368.48399999999998,"15min":368.48399999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.1279999999999999,"5min":4.7469999999999999,"15min":2.298},"min":{"1min":0.46400000000000002,"5min":0.37,"15min":0.36799999999999999},"max":{"1min":1.591,"5min":371.65100000000001,"15min":371.65100000000001},"last":0.78900000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":9.0960000000000001,"15min":3.8159999999999998},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.34399999999999997},"max":{"1min":1.752,"5min":407.07600000000002,"15min":407.07600000000002},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":9.0519999999999996,"15min":3.8679999999999999},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35399999999999998},"max":{"1min":1.9730000000000001,"5min":407.26299999999998,"15min":407.26299999999998},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.96599999999999997,"15min":1.1060000000000001},"min":{"1min":0.505,"5min":0.40899999999999997,"15min":0.34799999999999998},"max":{"1min":1.7170000000000001,"5min":3.722,"15min":4.4989999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.103,"15min":1.2150000000000001},"min":{"1min":0.54500000000000004,"5min":0.41099999999999998,"15min":0.40300000000000002},"max":{"1min":1.875,"5min":3.8039999999999998,"15min":3.8039999999999998},"last":0.755}]},{"osd":7,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.21,"15min":1.234},"min":{"1min":0.57199999999999995,"5min":0.435,"15min":0.39100000000000001},"max":{"1min":1.9359999999999999,"5min":5.6779999999999999,"15min":5.6779999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.369,"5min":1.2450000000000001,"15min":1.2849999999999999},"min":{"1min":0.56499999999999995,"5min":0.438,"15min":0.438},"max":{"1min":1.776,"5min":5.2510000000000003,"15min":5.2510000000000003},"last":1.6259999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.079,"15min":1.218},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.0760000000000001,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.1419999999999999,"15min":1.2589999999999999},"min":{"1min":0.49299999999999999,"5min":0.39000000000000001,"15min":0.375},"max":{"1min":2.0019999999999998,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":1.6659999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:14:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.0329999999999999,"15min":0.96499999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.0859999999999999,"5min":5.0039999999999996,"15min":5.0039999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.2350000000000001,"15min":1.0369999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.9079999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":1.778}]},{"osd":10,"last update":"Mon Feb 12 16:14:04 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.1359999999999999,"15min":1.2529999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.377},"max":{"1min":2.109,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.1910000000000001,"15min":1.3240000000000001},"min":{"1min":0.504,"5min":0.496,"15min":0.45200000000000001},"max":{"1min":2.3490000000000002,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":0.57099999999999995}]}]},{"osd":10,"up_from":83,"seq":356482285769,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790860,"kb_used_data":481348,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92954292,"statfs":{"total":95995035648,"available":95185195008,"internally_reserved":0,"allocated":492900352,"data_stored":492503090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.29,"15min":1.0489999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":1.212,"5min":49.802999999999997,"15min":49.802999999999997},"last":0.309},{"interface":"front","average":{"1min":0.53600000000000003,"5min":1.3149999999999999,"15min":1.101},"min":{"1min":0.32700000000000001,"5min":0.28899999999999998,"15min":0.28399999999999997},"max":{"1min":0.93899999999999995,"5min":50.863,"15min":50.863},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63200000000000001,"15min":0.83699999999999997},"min":{"1min":0.38,"5min":0.309,"15min":0.29299999999999998},"max":{"1min":1.2509999999999999,"5min":4.4240000000000004,"15min":4.8300000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.64400000000000002,"15min":0.89900000000000002},"min":{"1min":0.42599999999999999,"5min":0.34499999999999997,"15min":0.29699999999999999},"max":{"1min":1.0780000000000001,"5min":1.661,"15min":7.9829999999999997},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.97799999999999998,"15min":1.022},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.7229999999999999,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.96099999999999997},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.29599999999999999},"max":{"1min":2.3090000000000002,"5min":4.2149999999999999,"15min":5.4100000000000001},"last":0.85099999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":2.21,"15min":1.4510000000000001},"min":{"1min":0.48499999999999999,"5min":0.38400000000000001,"15min":0.34499999999999997},"max":{"1min":3.875,"5min":145.017,"15min":145.017},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":2.2109999999999999,"15min":1.4419999999999999},"min":{"1min":0.52100000000000002,"5min":0.36399999999999999,"15min":0.34200000000000003},"max":{"1min":1.304,"5min":144.99600000000001,"15min":144.99600000000001},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":2.222,"15min":3.5459999999999998},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":4.7160000000000002,"5min":151.25800000000001,"15min":602.05399999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.88400000000000001,"5min":2.3730000000000002,"15min":3.6619999999999999},"min":{"1min":0.501,"5min":0.439,"15min":0.33200000000000002},"max":{"1min":3.1749999999999998,"5min":151.416,"15min":598.875},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.83699999999999997,"15min":1.008},"min":{"1min":0.40400000000000003,"5min":0.39600000000000002,"15min":0.35399999999999998},"max":{"1min":3.8660000000000001,"5min":3.8660000000000001,"15min":5.4729999999999999},"last":0.378},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.91600000000000004,"15min":1.129},"min":{"1min":0.55500000000000005,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.4099999999999999,"5min":3.0369999999999999,"15min":4.6660000000000004},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.99199999999999999,"15min":3.238},"min":{"1min":0.47699999999999998,"5min":0.41299999999999998,"15min":0.376},"max":{"1min":1.375,"5min":5.3220000000000001,"15min":597.48299999999995},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.90500000000000003,"15min":3.234},"min":{"1min":0.53500000000000003,"5min":0.47199999999999998,"15min":0.41799999999999998},"max":{"1min":1.792,"5min":3.8149999999999999,"15min":600.23099999999999},"last":0.48799999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.93400000000000005,"15min":1.169},"min":{"1min":0.434,"5min":0.37,"15min":0.33400000000000002},"max":{"1min":3.4279999999999999,"5min":4.0640000000000001,"15min":7.2720000000000002},"last":0.745},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.95199999999999996,"15min":1.2030000000000001},"min":{"1min":0.60099999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.4850000000000001,"5min":3.278,"15min":7.3810000000000002},"last":0.59099999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.008,"15min":1.2290000000000001},"min":{"1min":0.66400000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6890000000000001,"5min":4.3369999999999997,"15min":4.7089999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.006,"15min":1.226},"min":{"1min":0.56299999999999994,"5min":0.53200000000000003,"15min":0.40000000000000002},"max":{"1min":1.837,"5min":4.3529999999999998,"15min":4.7789999999999999},"last":0.80400000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:14:03 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.623,"15min":1.47},"min":{"1min":0.79800000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.972,"5min":30.917999999999999,"15min":30.917999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.599,"15min":1.452},"min":{"1min":0.73599999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.0230000000000001,"5min":30.742000000000001,"15min":30.742000000000001},"last":0.69999999999999996}]}]},{"osd":9,"up_from":74,"seq":317827580113,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":702848,"kb_used_data":405624,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93042304,"statfs":{"total":95995035648,"available":95275319296,"internally_reserved":0,"allocated":415358976,"data_stored":414967773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.59499999999999997,"15min":0.78700000000000003},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.215},"max":{"1min":1.1830000000000001,"5min":3.702,"15min":10.788},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.91000000000000003,"15min":0.93400000000000005},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.27400000000000002},"max":{"1min":5.3550000000000004,"5min":6.9749999999999996,"15min":7.9630000000000001},"last":0.34499999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":2.1360000000000001,"15min":1.5489999999999999},"min":{"1min":0.30399999999999999,"5min":0.29399999999999998,"15min":0.25700000000000001},"max":{"1min":1.3520000000000001,"5min":175.33500000000001,"15min":175.33500000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.752,"5min":2.2919999999999998,"15min":1.5669999999999999},"min":{"1min":0.34699999999999998,"5min":0.33900000000000002,"15min":0.317},"max":{"1min":1.3620000000000001,"5min":175.24299999999999,"15min":175.24299999999999},"last":0.53300000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.90300000000000002,"15min":1.0349999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.27700000000000002},"max":{"1min":5.0430000000000001,"5min":7.04,"15min":7.8140000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.79600000000000004,"15min":1.004},"min":{"1min":0.38100000000000001,"5min":0.35199999999999998,"15min":0.28199999999999997},"max":{"1min":0.91000000000000003,"5min":4.6859999999999999,"15min":4.7240000000000002},"last":0.79400000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.155,"15min":1.0800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.308},"max":{"1min":3.1989999999999998,"5min":34.591000000000001,"15min":34.591000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.3200000000000001,"15min":1.2290000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.35199999999999998},"max":{"1min":3.2130000000000001,"5min":34.494,"15min":34.494},"last":0.63}]},{"osd":4,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.81000000000000005,"15min":1.097},"min":{"1min":0.371,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.298,"5min":4.391,"15min":25.552},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79000000000000004,"15min":1.026},"min":{"1min":0.44600000000000001,"5min":0.40699999999999997,"15min":0.35099999999999998},"max":{"1min":1.448,"5min":1.6839999999999999,"15min":25.847999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.82899999999999996,"15min":3.3380000000000001},"min":{"1min":0.41799999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3480000000000001,"5min":4.4630000000000001,"15min":617.95899999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0489999999999999,"15min":3.4460000000000002},"min":{"1min":0.53200000000000003,"5min":0.41599999999999998,"15min":0.375},"max":{"1min":2.4350000000000001,"5min":7.1580000000000004,"15min":617.88099999999997},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.98099999999999998,"15min":1.1359999999999999},"min":{"1min":0.441,"5min":0.434,"15min":0.40999999999999998},"max":{"1min":1.482,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.93999999999999995,"15min":1.0529999999999999},"min":{"1min":0.44700000000000001,"5min":0.44500000000000001,"15min":0.377},"max":{"1min":1.381,"5min":4.7910000000000004,"15min":4.7910000000000004},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99199999999999999,"15min":4.625},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.379},"max":{"1min":1.4279999999999999,"5min":4.5030000000000001,"15min":613.28200000000004},"last":0.873},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.1299999999999999,"15min":4.7169999999999996},"min":{"1min":0.59899999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5289999999999999,"5min":7.2590000000000003,"15min":613.25199999999995},"last":0.66600000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.94599999999999995,"15min":1.1899999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.4550000000000001,"5min":3.7799999999999998,"15min":4.96},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.016,"15min":1.135},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.5760000000000001,"5min":7.2270000000000003,"15min":7.2270000000000003},"last":0.745}]},{"osd":10,"last update":"Mon Feb 12 16:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.006,"15min":1.1559999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.38400000000000001},"max":{"1min":1.2909999999999999,"5min":4.4219999999999997,"15min":5.5},"last":1.048},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.88900000000000001,"15min":1.173},"min":{"1min":0.52100000000000002,"5min":0.434,"15min":0.434},"max":{"1min":1.1379999999999999,"5min":3.581,"15min":4.7619999999999996},"last":0.90200000000000002}]}]},{"osd":8,"up_from":65,"seq":279172874456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":310284,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434868,"statfs":{"total":95995035648,"available":95677304832,"internally_reserved":0,"allocated":790528,"data_stored":393286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73999999999999999,"15min":0.83799999999999997},"min":{"1min":0.33200000000000002,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":1.2390000000000001,"5min":4.827,"15min":5.383},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.79300000000000004,"15min":0.88500000000000001},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.28299999999999997},"max":{"1min":3.0249999999999999,"5min":7.319,"15min":7.319},"last":0.66200000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.76200000000000001,"15min":1.581},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.5619999999999998,"5min":3.8170000000000002,"15min":23.661000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.0860000000000001,"5min":0.78800000000000003,"15min":1.6759999999999999},"min":{"1min":0.38800000000000001,"5min":0.315,"15min":0.30299999999999999},"max":{"1min":3.5739999999999998,"5min":3.8330000000000002,"15min":23.672000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":12.574,"5min":3.3340000000000001,"15min":1.696},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":193.929,"5min":193.929,"15min":193.929},"last":0.52300000000000002},{"interface":"front","average":{"1min":12.651,"5min":3.484,"15min":1.8380000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":193.82499999999999,"5min":193.82499999999999,"15min":193.82499999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.052,"15min":1.073},"min":{"1min":0.39000000000000001,"5min":0.307,"15min":0.28000000000000003},"max":{"1min":1.6539999999999999,"5min":10.4,"15min":10.4},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.95899999999999996,"15min":1.0880000000000001},"min":{"1min":0.34399999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.2769999999999999,"5min":10.413,"15min":10.413},"last":0.61299999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":0.79800000000000004,"15min":0.997},"min":{"1min":0.46400000000000002,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":6.4619999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.71299999999999997,"15min":1.048},"min":{"1min":0.435,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4159999999999999,"5min":1.4610000000000001,"15min":5.71},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.80700000000000005,"15min":1.012},"min":{"1min":0.44600000000000001,"5min":0.37,"15min":0.32000000000000001},"max":{"1min":1.7749999999999999,"5min":1.7749999999999999,"15min":13.961},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.2390000000000001,"5min":0.99099999999999999,"15min":1.1379999999999999},"min":{"1min":0.55100000000000005,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.0649999999999999,"5min":4.8339999999999996,"15min":13.103},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.84499999999999997,"15min":1.0609999999999999},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.36299999999999999},"max":{"1min":1.4490000000000001,"5min":3.5209999999999999,"15min":4.3440000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":0.98499999999999999,"15min":1.1990000000000001},"min":{"1min":0.49299999999999999,"5min":0.39000000000000001,"15min":0.38300000000000001},"max":{"1min":2.0190000000000001,"5min":4.2160000000000002,"15min":4.4720000000000004},"last":0.86599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.0860000000000001,"15min":1.254},"min":{"1min":0.58299999999999996,"5min":0.45600000000000002,"15min":0.42099999999999999},"max":{"1min":3.452,"5min":4.617,"15min":13.242000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.28,"5min":0.95499999999999996,"15min":1.264},"min":{"1min":0.60499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.3809999999999998,"5min":4.6310000000000002,"15min":13.388},"last":0.78800000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:13:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.96699999999999997,"15min":1.2110000000000001},"min":{"1min":0.51600000000000001,"5min":0.35199999999999998,"15min":0.34699999999999998},"max":{"1min":1.9730000000000001,"5min":4.093,"15min":5.8449999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.198,"5min":0.97599999999999998,"15min":1.177},"min":{"1min":0.58199999999999996,"5min":0.45300000000000001,"15min":0.36799999999999999},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":6.2000000000000002},"last":1.288}]},{"osd":10,"last update":"Mon Feb 12 16:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.85499999999999998,"15min":1.1499999999999999},"min":{"1min":0.48399999999999999,"5min":0.38500000000000001,"15min":0.36399999999999999},"max":{"1min":1.7749999999999999,"5min":5.2670000000000003,"15min":5.4189999999999996},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.90300000000000002,"15min":1.151},"min":{"1min":0.503,"5min":0.39500000000000002,"15min":0.376},"max":{"1min":1.4470000000000001,"5min":3.4860000000000002,"15min":3.6779999999999999},"last":0.878}]},{"osd":11,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.0349999999999999,"15min":17.695},"min":{"1min":0.55000000000000004,"5min":0.441,"15min":0.441},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":635.66399999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.111,"5min":1.125,"15min":17.526},"min":{"1min":0.55600000000000005,"5min":0.46400000000000002,"15min":0.44600000000000001},"max":{"1min":4.6639999999999997,"5min":4.9489999999999998,"15min":636.28099999999995},"last":0.81799999999999995}]}]},{"osd":7,"up_from":56,"seq":240518168801,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761420,"kb_used_data":456004,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92983732,"statfs":{"total":95995035648,"available":95215341568,"internally_reserved":0,"allocated":466948096,"data_stored":466550854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.84199999999999997,"15min":1.129},"min":{"1min":0.26700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.0880000000000001,"5min":5.6669999999999998,"15min":60.152000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.0049999999999999,"15min":1.2350000000000001},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.5759999999999996,"5min":5.8479999999999999,"15min":62.020000000000003},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.81399999999999995,"15min":1.194},"min":{"1min":0.33900000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68500000000000005,"5min":6.1790000000000003,"15min":60.204999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.872,"15min":1.1910000000000001},"min":{"1min":0.35499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.1429999999999998,"5min":5.8040000000000003,"15min":59.365000000000002},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.1220000000000001,"15min":1.1080000000000001},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.27900000000000003},"max":{"1min":2.1829999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":1.1080000000000001,"15min":1.0840000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":1.139,"5min":8.2940000000000005,"15min":8.2940000000000005},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.998,"15min":0.91700000000000004},"min":{"1min":0.29499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.6299999999999999,"5min":6.048,"15min":6.2519999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.0760000000000001,"15min":0.996},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.28599999999999998},"max":{"1min":3.23,"5min":6.0919999999999996,"15min":6.0919999999999996},"last":0.40799999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0349999999999999,"15min":1.0129999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":4.5119999999999996,"5min":6.0739999999999998,"15min":6.0739999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.0429999999999999,"15min":1.0269999999999999},"min":{"1min":0.36099999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.1689999999999996,"5min":6.274,"15min":6.274},"last":0.55500000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:13:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.1519999999999999,"15min":1.1240000000000001},"min":{"1min":0.41799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.6070000000000002,"5min":6.0830000000000002,"15min":6.0830000000000002},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.1180000000000001,"15min":1.0860000000000001},"min":{"1min":0.45200000000000001,"5min":0.37,"15min":0.32200000000000001},"max":{"1min":2.9100000000000001,"5min":6.1349999999999998,"15min":6.1349999999999998},"last":0.876}]},{"osd":8,"last update":"Mon Feb 12 16:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1539999999999999,"15min":1.2410000000000001},"min":{"1min":0.45400000000000001,"5min":0.432,"15min":0.432},"max":{"1min":2.0579999999999998,"5min":6.0250000000000004,"15min":6.0250000000000004},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.04,"15min":1.2450000000000001},"min":{"1min":0.438,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.1829999999999998,"5min":6.6150000000000002,"15min":6.6150000000000002},"last":0.73599999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:13:33 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.222,"15min":2.3300000000000001},"min":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.441},"max":{"1min":1.929,"5min":6.9539999999999997,"15min":278.54599999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.069,"5min":1.079,"15min":2.2570000000000001},"min":{"1min":0.56699999999999995,"5min":0.42699999999999999,"15min":0.38200000000000001},"max":{"1min":2.1400000000000001,"5min":6.3479999999999999,"15min":281.09199999999998},"last":0.69299999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:13:09 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.1020000000000001,"15min":1.254},"min":{"1min":0.53300000000000003,"5min":0.49099999999999999,"15min":0.48599999999999999},"max":{"1min":4.2329999999999997,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.745},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.1850000000000001,"15min":1.298},"min":{"1min":0.58399999999999996,"5min":0.5,"15min":0.42099999999999999},"max":{"1min":4.2599999999999998,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":0.70499999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.5660000000000001,"15min":2.415},"min":{"1min":0.50700000000000001,"5min":0.499,"15min":0.498},"max":{"1min":1.885,"5min":31.542999999999999,"15min":280.83999999999997},"last":1.585},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.774,"15min":2.4510000000000001},"min":{"1min":0.57899999999999996,"5min":0.52300000000000002,"15min":0.51200000000000001},"max":{"1min":4.0229999999999997,"5min":31.988,"15min":278.63799999999998},"last":0.76300000000000001}]}]},{"osd":6,"up_from":48,"seq":206158430442,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":634572,"kb_used_data":333252,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93110580,"statfs":{"total":95995035648,"available":95345233920,"internally_reserved":0,"allocated":341250048,"data_stored":340852806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:13:34 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.76300000000000001,"15min":1.0620000000000001},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.26200000000000001},"max":{"1min":4.4199999999999999,"5min":4.4199999999999999,"15min":67.093000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.879,"5min":0.85799999999999998,"15min":1.1100000000000001},"min":{"1min":0.378,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.5350000000000001,"5min":4.4779999999999998,"15min":68.549000000000007},"last":0.499}]},{"osd":1,"last update":"Mon Feb 12 16:13:34 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.877,"15min":1.1819999999999999},"min":{"1min":0.36299999999999999,"5min":0.32200000000000001,"15min":0.308},"max":{"1min":4.4900000000000002,"5min":4.4900000000000002,"15min":66.936000000000007},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.82699999999999996,"15min":1.1639999999999999},"min":{"1min":0.34100000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.1200000000000001,"5min":4.9409999999999998,"15min":68.683000000000007},"last":0.438}]},{"osd":2,"last update":"Mon Feb 12 16:13:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.73499999999999999,"15min":1.4139999999999999},"min":{"1min":0.38100000000000001,"5min":0.317,"15min":0.28899999999999998},"max":{"1min":3.722,"5min":3.722,"15min":68.462000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.997,"5min":0.77600000000000002,"15min":1.4530000000000001},"min":{"1min":0.437,"5min":0.373,"15min":0.34599999999999997},"max":{"1min":4.7839999999999998,"5min":4.7839999999999998,"15min":67.322999999999993},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:13:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":1.0549999999999999,"15min":1.208},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":1.6850000000000001,"5min":5.3179999999999996,"15min":68.332999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.0189999999999999,"15min":1.216},"min":{"1min":0.47699999999999998,"5min":0.39800000000000002,"15min":0.32600000000000001},"max":{"1min":1.923,"5min":6,"15min":67.281999999999996},"last":1.99}]},{"osd":4,"last update":"Mon Feb 12 16:13:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":3.7250000000000001,"15min":2.0870000000000002},"min":{"1min":0.42699999999999999,"5min":0.32200000000000001,"15min":0.316},"max":{"1min":1.4490000000000001,"5min":284.05599999999998,"15min":284.05599999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":3.871,"15min":1.921},"min":{"1min":0.46200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.5169999999999999,"5min":284.041,"15min":284.041},"last":1.4299999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":3.7370000000000001,"15min":2.1789999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33500000000000002},"max":{"1min":4.1219999999999999,"5min":281.49700000000001,"15min":281.49700000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":3.8490000000000002,"15min":2.165},"min":{"1min":0.48499999999999999,"5min":0.371,"15min":0.29399999999999998},"max":{"1min":3.4900000000000002,"5min":281.61900000000003,"15min":281.61900000000003},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.97699999999999998,"15min":1.6799999999999999},"min":{"1min":0.5,"5min":0.438,"15min":0.29399999999999998},"max":{"1min":3.5030000000000001,"5min":3.6349999999999998,"15min":67.144999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.0940000000000001,"15min":1.698},"min":{"1min":0.54400000000000004,"5min":0.54300000000000004,"15min":0.38200000000000001},"max":{"1min":4.5449999999999999,"5min":4.5449999999999999,"15min":67.230999999999995},"last":0.53700000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":0.997,"15min":1.468},"min":{"1min":0.5,"5min":0.496,"15min":0.39800000000000002},"max":{"1min":1.802,"5min":3.516,"15min":68.653000000000006},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.006,"15min":1.4279999999999999},"min":{"1min":0.497,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.8620000000000001,"5min":5.016,"15min":68.620999999999995},"last":0.59099999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99199999999999999,"15min":1.472},"min":{"1min":0.58499999999999996,"5min":0.51000000000000001,"15min":0.40600000000000003},"max":{"1min":2.0110000000000001,"5min":2.4769999999999999,"15min":67.188000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.0640000000000001,"15min":1.4610000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.766,"5min":4.9930000000000003,"15min":68.506},"last":0.63600000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.0169999999999999,"15min":1.516},"min":{"1min":0.58699999999999997,"5min":0.58299999999999996,"15min":0.49099999999999999},"max":{"1min":1.7589999999999999,"5min":3.5470000000000002,"15min":68.722999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0720000000000001,"15min":1.4610000000000001},"min":{"1min":0.64100000000000001,"5min":0.60299999999999998,"15min":0.505},"max":{"1min":1.881,"5min":3.6389999999999998,"15min":68.573999999999998},"last":0.751}]}]},{"osd":1,"up_from":13,"seq":55834575120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267136,"kb_used_data":961720,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92478016,"statfs":{"total":95995035648,"available":94697488384,"internally_reserved":0,"allocated":984801280,"data_stored":984410077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":7.6349999999999998,"5min":2.5049999999999999,"15min":1.5329999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":114.40000000000001,"5min":114.40000000000001,"15min":114.40000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":8.2430000000000003,"5min":2.8439999999999999,"15min":1.679},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":115.09099999999999,"5min":115.09099999999999,"15min":115.09099999999999},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.71799999999999997,"15min":1.181},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":0.68999999999999995,"5min":4.4080000000000004,"15min":40.542999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.2989999999999999,"15min":1.351},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.24299999999999999},"max":{"1min":2.863,"5min":9.2119999999999997,"15min":40.811999999999998},"last":0.67300000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.169,"15min":1.004},"min":{"1min":0.28499999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.6900000000000004,"5min":6.0170000000000003,"15min":6.0170000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.962,"5min":1.405,"15min":1.1259999999999999},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":10.225,"5min":10.648999999999999,"15min":10.648999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":2.7210000000000001,"5min":1.2929999999999999,"15min":1.298},"min":{"1min":0.40799999999999997,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":33.125,"5min":33.125,"15min":33.125},"last":1.1539999999999999},{"interface":"front","average":{"1min":2.669,"5min":1.373,"15min":1.339},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":33.060000000000002,"5min":33.060000000000002,"15min":33.060000000000002},"last":0.95499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.91300000000000003,"15min":1.2350000000000001},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":1.0569999999999999,"5min":4.6829999999999998,"15min":9.8019999999999996},"last":0.749},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.22,"15min":1.288},"min":{"1min":0.40400000000000003,"5min":0.36199999999999999,"15min":0.34000000000000002},"max":{"1min":2.6509999999999998,"5min":4.9340000000000002,"15min":10.420999999999999},"last":0.624}]},{"osd":6,"last update":"Mon Feb 12 16:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.2589999999999999,"15min":1.278},"min":{"1min":0.37,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.617,"5min":5.0549999999999997,"15min":10.455},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.175,"5min":1.3400000000000001,"15min":1.3480000000000001},"min":{"1min":0.49199999999999999,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":2.6509999999999998,"5min":5.101,"15min":10.488},"last":0.77800000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.214,"15min":1.3560000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.766,"5min":5.0119999999999996,"15min":6.2869999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.1479999999999999,"15min":1.2569999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":2.6230000000000002,"5min":4.8840000000000003,"15min":5.1429999999999998},"last":0.73799999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:13:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.97799999999999998,"15min":1.3740000000000001},"min":{"1min":0.51700000000000002,"5min":0.45900000000000002,"15min":0.39400000000000002},"max":{"1min":0.95999999999999996,"5min":4.5449999999999999,"15min":9.9049999999999994},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.292,"15min":1.5349999999999999},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.41999999999999998},"max":{"1min":2.798,"5min":5.0099999999999998,"15min":10.313000000000001},"last":0.89400000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.244,"15min":1.4810000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.40699999999999997},"max":{"1min":1.9570000000000001,"5min":5.2169999999999996,"15min":10.352},"last":1.8340000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.3720000000000001,"15min":1.3919999999999999},"min":{"1min":0.502,"5min":0.443,"15min":0.40200000000000002},"max":{"1min":2.8050000000000002,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":1.4330000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.048,"15min":1.3999999999999999},"min":{"1min":0.55500000000000005,"5min":0.51000000000000001,"15min":0.48499999999999999},"max":{"1min":4.0179999999999998,"5min":4.5899999999999999,"15min":9.7629999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.093,"5min":1.2949999999999999,"15min":1.5369999999999999},"min":{"1min":0.57199999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"max":{"1min":2.5419999999999998,"5min":5.1390000000000002,"15min":10.387},"last":0.85799999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.534,"15min":1.3400000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.9039999999999999,"5min":13.555,"15min":13.555},"last":1.175},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.52,"15min":1.3120000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.7530000000000001,"5min":13.846,"15min":13.846},"last":0.92900000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738651,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":481560,"kb_used_data":180240,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93263592,"statfs":{"total":95995035648,"available":95501918208,"internally_reserved":0,"allocated":184565760,"data_stored":184162429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.147,"15min":1.4750000000000001},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.22500000000000001},"max":{"1min":3.6909999999999998,"5min":8.266,"15min":114.01600000000001},"last":0.251},{"interface":"front","average":{"1min":0.87,"5min":0.85799999999999998,"15min":1.4379999999999999},"min":{"1min":0.25600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.8090000000000002,"5min":4.5819999999999999,"15min":114.342},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:13:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.788,"15min":1.24},"min":{"1min":0.315,"5min":0.26300000000000001,"15min":0.215},"max":{"1min":3.7200000000000002,"5min":53.793999999999997,"15min":53.793999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.627,"15min":1.1859999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.7089999999999996,"5min":53.853000000000002,"15min":53.853000000000002},"last":0.77100000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":2.1339999999999999,"15min":1.3859999999999999},"min":{"1min":0.34399999999999997,"5min":0.312,"15min":0.29799999999999999},"max":{"1min":3.6640000000000001,"5min":116.187,"15min":116.187},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":2.0880000000000001,"15min":1.371},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.5030000000000001,"5min":116.163,"15min":116.163},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":2.3959999999999999,"15min":1.4750000000000001},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":3.5880000000000001,"5min":116.005,"15min":116.005},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.151,"5min":2.452,"15min":1.4990000000000001},"min":{"1min":0.51800000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.802,"5min":115.973,"15min":115.973},"last":0.35899999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:13:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.98299999999999998,"15min":1.0349999999999999},"min":{"1min":0.38700000000000001,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":3.1899999999999999,"5min":5.1310000000000002,"15min":5.4699999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.0169999999999999,"15min":1.0820000000000001},"min":{"1min":0.42499999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.2170000000000001,"5min":3.6019999999999999,"15min":5.5389999999999997},"last":0.40100000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":2.4809999999999999,"15min":1.617},"min":{"1min":0.39800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.6360000000000001,"5min":115.26000000000001,"15min":115.26000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.407,"5min":2.5739999999999998,"15min":1.7270000000000001},"min":{"1min":0.51500000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":3.7029999999999998,"5min":115.402,"15min":115.402},"last":0.66700000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.1479999999999999,"15min":1.278},"min":{"1min":0.59099999999999997,"5min":0.378,"15min":0.378},"max":{"1min":3.3439999999999999,"5min":3.6619999999999999,"15min":5.6059999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.599,"5min":1.1579999999999999,"15min":1.296},"min":{"1min":0.61199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":3.4049999999999998,"5min":3.7959999999999998,"15min":7.5599999999999996},"last":0.70399999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:13:38 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.2190000000000001,"15min":1.355},"min":{"1min":0.47699999999999998,"5min":0.46000000000000002,"15min":0.40600000000000003},"max":{"1min":3.6309999999999998,"5min":4.6310000000000002,"15min":5.7969999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.208,"15min":1.2949999999999999},"min":{"1min":0.47499999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2839999999999998,"5min":6.6070000000000002,"15min":6.6070000000000002},"last":0.80200000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.2,"15min":1.3129999999999999},"min":{"1min":0.499,"5min":0.432,"15min":0.432},"max":{"1min":3.6019999999999999,"5min":5.7869999999999999,"15min":5.8259999999999996},"last":0.496},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2150000000000001,"15min":1.3029999999999999},"min":{"1min":0.60099999999999998,"5min":0.44900000000000001,"15min":0.437},"max":{"1min":3.7229999999999999,"5min":6.2480000000000002,"15min":6.2480000000000002},"last":0.56399999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.3260000000000001,"15min":1.3979999999999999},"min":{"1min":0.67800000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":6.0149999999999997,"5min":6.0149999999999997,"15min":7.9660000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.216,"15min":1.383},"min":{"1min":0.46400000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.0739999999999998,"5min":8.4800000000000004,"15min":8.4800000000000004},"last":0.95199999999999996}]}]},{"osd":2,"up_from":21,"seq":90194313482,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501592,"kb_used_data":192080,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93243560,"statfs":{"total":95995035648,"available":95481405440,"internally_reserved":0,"allocated":196689920,"data_stored":196286589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":3.5099999999999998,"15min":1.8520000000000001},"min":{"1min":0.29699999999999999,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":5.173,"5min":177.68899999999999,"15min":177.68899999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.90700000000000003,"5min":3.4900000000000002,"15min":1.8979999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.216},"max":{"1min":2.4540000000000002,"5min":178.26900000000001,"15min":178.26900000000001},"last":0.79600000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.7370000000000001,"15min":1.3879999999999999},"min":{"1min":0.35499999999999998,"5min":0.23799999999999999,"15min":0.223},"max":{"1min":1.403,"5min":70.944000000000003,"15min":70.944000000000003},"last":1.145},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.77,"15min":1.4259999999999999},"min":{"1min":0.40899999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.4399999999999999,"5min":71.203999999999994,"15min":71.203999999999994},"last":1.248}]},{"osd":3,"last update":"Mon Feb 12 16:13:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":2.734,"15min":1.774},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.25600000000000001},"max":{"1min":1.411,"5min":80.436999999999998,"15min":80.436999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.877,"5min":2.6779999999999999,"15min":1.631},"min":{"1min":0.38200000000000001,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":2.0070000000000001,"5min":77.286000000000001,"15min":77.286000000000001},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:13:26 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.82899999999999996,"15min":1.131},"min":{"1min":0.40899999999999997,"5min":0.33300000000000002,"15min":0.311},"max":{"1min":2.3719999999999999,"5min":4.2789999999999999,"15min":9.657},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92700000000000005,"15min":1.0209999999999999},"min":{"1min":0.35899999999999999,"5min":0.31,"15min":0.31},"max":{"1min":1.8839999999999999,"5min":4.7969999999999997,"15min":9.5530000000000008},"last":0.82699999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":0.93700000000000006,"15min":3.2559999999999998},"min":{"1min":0.42899999999999999,"5min":0.33900000000000002,"15min":0.33400000000000002},"max":{"1min":1.913,"5min":5.1520000000000001,"15min":616.34199999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":0.92700000000000005,"15min":3.3530000000000002},"min":{"1min":0.50600000000000001,"5min":0.38400000000000001,"15min":0.33900000000000002},"max":{"1min":1.772,"5min":5.0910000000000002,"15min":615.18899999999996},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:13:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.82999999999999996,"15min":1.0629999999999999},"min":{"1min":0.55600000000000005,"5min":0.372,"15min":0.36799999999999999},"max":{"1min":1.589,"5min":3.597,"15min":9.6150000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.89200000000000002,"15min":1.165},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.35099999999999998},"max":{"1min":1.6180000000000001,"5min":5.4349999999999996,"15min":6.3979999999999997},"last":0.96199999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0289999999999999,"15min":3.464},"min":{"1min":0.54900000000000004,"5min":0.40400000000000003,"15min":0.377},"max":{"1min":2.532,"5min":4.032,"15min":611.79600000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.071,"15min":3.488},"min":{"1min":0.41999999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.8490000000000002,"5min":4.0369999999999999,"15min":611.83900000000006},"last":0.49199999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:13:47 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":0.91300000000000003,"15min":1.3109999999999999},"min":{"1min":0.497,"5min":0.38700000000000001,"15min":0.38200000000000001},"max":{"1min":2.1419999999999999,"5min":2.6560000000000001,"15min":6.4960000000000004},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.96599999999999997,"15min":1.403},"min":{"1min":0.45300000000000001,"5min":0.432,"15min":0.432},"max":{"1min":2.0910000000000002,"5min":2.1680000000000001,"15min":8.5920000000000005},"last":0.53400000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:13:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97399999999999998,"15min":1.2430000000000001},"min":{"1min":0.60899999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.0880000000000001,"5min":7.2830000000000004,"15min":9.6460000000000008},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.95499999999999996,"15min":1.3300000000000001},"min":{"1min":0.58399999999999996,"5min":0.47799999999999998,"15min":0.42999999999999999},"max":{"1min":2.3100000000000001,"5min":2.7690000000000001,"15min":6.4699999999999998},"last":0.85899999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":0.96799999999999997,"15min":1.284},"min":{"1min":0.54900000000000004,"5min":0.442,"15min":0.442},"max":{"1min":2.4460000000000002,"5min":3.8250000000000002,"15min":9.6760000000000002},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.95799999999999996,"15min":1.339},"min":{"1min":0.58999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.165,"5min":2.6520000000000001,"15min":6.4359999999999999},"last":0.92700000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149953,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":800392,"kb_used_data":499072,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92944760,"statfs":{"total":95995035648,"available":95175434240,"internally_reserved":0,"allocated":511049728,"data_stored":510655944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:05 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.0269999999999999,"15min":1.01},"min":{"1min":0.27100000000000002,"5min":0.246,"15min":0.246},"max":{"1min":2.7930000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":0.313},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":0.90200000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.5740000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":2.0659999999999998,"15min":1.2549999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":8.3680000000000003,"5min":70.031999999999996,"15min":70.031999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.621,"5min":2.25,"15min":1.339},"min":{"1min":0.317,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.3399999999999999,"5min":69.944999999999993,"15min":69.944999999999993},"last":0.95299999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.98899999999999999,"15min":1.2629999999999999},"min":{"1min":0.27400000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.9379999999999999,"5min":7.5780000000000003,"15min":68.405000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.0309999999999999,"15min":1.2869999999999999},"min":{"1min":0.32200000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.8779999999999999,"5min":9.5619999999999994,"15min":69.009},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82899999999999996,"15min":0.92300000000000004},"min":{"1min":0.315,"5min":0.30199999999999999,"15min":0.27800000000000002},"max":{"1min":2.4740000000000002,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.876,"5min":0.90500000000000003,"15min":0.97299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.8490000000000002,"5min":5.9870000000000001,"15min":5.9870000000000001},"last":0.38100000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":2.2879999999999998,"5min":1.4119999999999999,"15min":1.1930000000000001},"min":{"1min":0.36299999999999999,"5min":0.313,"15min":0.313},"max":{"1min":22.751000000000001,"5min":22.751000000000001,"15min":22.751000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":2.4209999999999998,"5min":1.2130000000000001,"15min":1.097},"min":{"1min":0.52000000000000002,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":22.527999999999999,"5min":22.527999999999999,"15min":22.527999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.85499999999999998,"15min":1.03},"min":{"1min":0.40699999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.9460000000000002,"5min":6.0289999999999999,"15min":6.0289999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.258,"15min":1.206},"min":{"1min":0.44500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.0710000000000002,"5min":9.5990000000000002,"15min":9.5990000000000002},"last":0.41699999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1399999999999999,"15min":1.482},"min":{"1min":0.42199999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":4.0140000000000002,"5min":7.5970000000000004,"15min":91.718999999999994},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.165,"5min":1.4690000000000001,"15min":1.623},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.0869999999999997,"5min":10.944000000000001,"15min":90.367999999999995},"last":0.442}]},{"osd":8,"last update":"Mon Feb 12 16:13:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.101,"15min":1.238},"min":{"1min":0.45900000000000002,"5min":0.41999999999999998,"15min":0.317},"max":{"1min":2.395,"5min":6.0970000000000004,"15min":6.5899999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.0860000000000001,"15min":1.2509999999999999},"min":{"1min":0.48999999999999999,"5min":0.47199999999999998,"15min":0.442},"max":{"1min":2.1429999999999998,"5min":6.2430000000000003,"15min":8.516},"last":0.495}]},{"osd":9,"last update":"Mon Feb 12 16:13:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.99399999999999999,"15min":1.254},"min":{"1min":0.52700000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.9049999999999998,"5min":6.2699999999999996,"15min":6.2699999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.268,"5min":1.26,"15min":1.365},"min":{"1min":0.53400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.0570000000000004,"5min":9.6509999999999998,"15min":9.6509999999999998},"last":0.74299999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.137,"15min":1.323},"min":{"1min":0.59499999999999997,"5min":0.42699999999999999,"15min":0.41099999999999998},"max":{"1min":2.9569999999999999,"5min":6.4199999999999999,"15min":6.4199999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.577,"5min":1.153,"15min":1.345},"min":{"1min":0.628,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":2.9820000000000002,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.63600000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:13:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.3480000000000001,"15min":1.554},"min":{"1min":0.53400000000000003,"5min":0.47699999999999998,"15min":0.39100000000000001},"max":{"1min":3.891,"5min":6.1390000000000002,"15min":49.167000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.357,"15min":1.5669999999999999},"min":{"1min":0.624,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.4809999999999999,"5min":6.1310000000000002,"15min":48.548999999999999},"last":0.51800000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953722,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":823308,"kb_used_data":526084,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92921844,"statfs":{"total":95995035648,"available":95151968256,"internally_reserved":0,"allocated":538710016,"data_stored":538312774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":2.556,"15min":1.4690000000000001},"min":{"1min":0.29799999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.6619999999999999,"5min":96.052000000000007,"15min":96.052000000000007},"last":0.442},{"interface":"front","average":{"1min":1.099,"5min":2.7149999999999999,"15min":1.595},"min":{"1min":0.39900000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.3470000000000004,"5min":96.307000000000002,"15min":96.307000000000002},"last":0.38}]},{"osd":1,"last update":"Mon Feb 12 16:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.2509999999999999,"15min":1.3180000000000001},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.0330000000000004,"5min":9.718,"15min":42.226999999999997},"last":2.4700000000000002},{"interface":"front","average":{"1min":1.623,"5min":1.3100000000000001,"15min":1.2490000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32200000000000001},"max":{"1min":4.3259999999999996,"5min":5.5119999999999996,"15min":43.689},"last":0.52800000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.046,"15min":1.1799999999999999},"min":{"1min":0.33100000000000002,"5min":0.28299999999999997,"15min":0.27700000000000002},"max":{"1min":4.048,"5min":6.3239999999999998,"15min":41.546999999999997},"last":0.432},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.1839999999999999,"15min":1.252},"min":{"1min":0.41599999999999998,"5min":0.34100000000000003,"15min":0.31},"max":{"1min":4.1619999999999999,"5min":5.5449999999999999,"15min":41.57},"last":1.2370000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":2.2429999999999999,"15min":1.3999999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":14.779999999999999,"5min":82.781000000000006,"15min":82.781000000000006},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.2639999999999998,"5min":2.1760000000000002,"15min":1.3839999999999999},"min":{"1min":0.46100000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":14.114000000000001,"5min":83.349000000000004,"15min":83.349000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:13:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.95099999999999996,"15min":0.95299999999999996},"min":{"1min":0.42699999999999999,"5min":0.308,"15min":0.25800000000000001},"max":{"1min":1.3120000000000001,"5min":6.0460000000000003,"15min":6.0460000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0820000000000001,"15min":1.052},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":1.421,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.46800000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:13:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.095,"15min":1.022},"min":{"1min":0.496,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.5589999999999999,"5min":5.2539999999999996,"15min":5.2539999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.089,"15min":1.006},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.236,"5min":6.2249999999999996,"15min":6.2249999999999996},"last":0.61599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.3089999999999999,"15min":1.556},"min":{"1min":0.443,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.3,"5min":9.0879999999999992,"15min":116.273},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2669999999999999,"15min":1.661},"min":{"1min":0.622,"5min":0.433,"15min":0.433},"max":{"1min":4.1929999999999996,"5min":5.2850000000000001,"15min":116.24299999999999},"last":0.55400000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.264,"15min":1.3200000000000001},"min":{"1min":0.70899999999999996,"5min":0.5,"15min":0.45400000000000001},"max":{"1min":1.734,"5min":5.4320000000000004,"15min":8.4190000000000005},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1579999999999999,"15min":1.2290000000000001},"min":{"1min":0.624,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.7050000000000001,"5min":7.0380000000000003,"15min":7.0380000000000003},"last":0.75600000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.2529999999999999,"15min":2.3820000000000001},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":1.7589999999999999,"5min":9.2910000000000004,"15min":297.55900000000003},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.357,"5min":1.3129999999999999,"15min":2.3730000000000002},"min":{"1min":0.64600000000000002,"5min":0.52600000000000002,"15min":0.497},"max":{"1min":4.2389999999999999,"5min":5.4640000000000004,"15min":297.61900000000003},"last":0.86799999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.2290000000000001,"15min":1.2809999999999999},"min":{"1min":0.56200000000000006,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.9259999999999999,"5min":6.9770000000000003,"15min":6.9770000000000003},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.88,"5min":1.4350000000000001,"15min":1.339},"min":{"1min":0.55700000000000005,"5min":0.51400000000000001,"15min":0.47599999999999998},"max":{"1min":1.585,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.77800000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724785,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577856,"kb_used_data":272440,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93167296,"statfs":{"total":95995035648,"available":95403311104,"internally_reserved":0,"allocated":278978560,"data_stored":278587357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.575,"15min":1.2070000000000001},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25},"max":{"1min":2.9900000000000002,"5min":75.396000000000001,"15min":75.396000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.6839999999999999,"15min":1.24},"min":{"1min":0.30399999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.5449999999999999,"5min":77.286000000000001,"15min":77.286000000000001},"last":0.79700000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.88,"15min":1.2070000000000001},"min":{"1min":0.40000000000000002,"5min":0.28999999999999998,"15min":0.26000000000000001},"max":{"1min":3.0350000000000001,"5min":6.3630000000000004,"15min":33.084000000000003},"last":1.163},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.93700000000000006,"15min":1.1000000000000001},"min":{"1min":0.373,"5min":0.29599999999999999,"15min":0.253},"max":{"1min":2.9409999999999998,"5min":5.5890000000000004,"15min":33.052999999999997},"last":1.0209999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.93300000000000005,"15min":1.2330000000000001},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.621,"5min":5.6580000000000004,"15min":26.797000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.91300000000000003,"15min":1.1279999999999999},"min":{"1min":0.38,"5min":0.35599999999999998,"15min":0.28599999999999998},"max":{"1min":3.0680000000000001,"5min":6.4610000000000003,"15min":26.905999999999999},"last":0.92900000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":5.7569999999999997,"15min":2.7650000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":252.57300000000001,"5min":252.57300000000001,"15min":252.57300000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":14.452,"5min":5.8220000000000001,"15min":2.7530000000000001},"min":{"1min":0.47499999999999998,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":252.24199999999999,"5min":252.24199999999999,"15min":252.24199999999999},"last":1.196}]},{"osd":4,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.004,"15min":0.99199999999999999},"min":{"1min":0.46999999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0939999999999999,"5min":6.492,"15min":14.259},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.095,"5min":1.1559999999999999,"15min":1.103},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.32600000000000001},"max":{"1min":4.9880000000000004,"5min":5.7300000000000004,"15min":13.268000000000001},"last":1.1000000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:13:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.016,"15min":1.1220000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.318},"max":{"1min":4.0209999999999999,"5min":4.0209999999999999,"15min":13.148},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.399,"5min":1.1120000000000001,"15min":1.1659999999999999},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":5.242,"5min":5.6749999999999998,"15min":13.214},"last":1.3140000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.2809999999999999,"15min":1.5960000000000001},"min":{"1min":0.50700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":4.907,"5min":6.9729999999999999,"15min":98.971000000000004},"last":1.002},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.232,"15min":1.6499999999999999},"min":{"1min":0.52100000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":5.9909999999999997,"5min":5.9909999999999997,"15min":99.218000000000004},"last":1.3879999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.175,"15min":1.3100000000000001},"min":{"1min":0.53200000000000003,"5min":0.44700000000000001,"15min":0.35099999999999998},"max":{"1min":3.117,"5min":6.5190000000000001,"15min":14.372999999999999},"last":1.593},{"interface":"front","average":{"1min":1.415,"5min":1.194,"15min":1.321},"min":{"1min":0.51500000000000001,"5min":0.42999999999999999,"15min":0.40500000000000003},"max":{"1min":2.714,"5min":5.6950000000000003,"15min":14.074999999999999},"last":1.4690000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:13:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.1899999999999999,"15min":1.345},"min":{"1min":0.498,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":13.375999999999999},"last":449.89499999999998},{"interface":"front","average":{"1min":1.744,"5min":1.3140000000000001,"15min":1.409},"min":{"1min":0.65700000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":5.4870000000000001,"5min":5.7160000000000002,"15min":14.132},"last":448.70800000000003}]},{"osd":10,"last update":"Mon Feb 12 16:13:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.23,"15min":1.347},"min":{"1min":0.64500000000000002,"5min":0.495,"15min":0.47499999999999998},"max":{"1min":5.4580000000000002,"5min":6.6849999999999996,"15min":14.214},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.347,"15min":1.421},"min":{"1min":0.58199999999999996,"5min":0.53900000000000003,"15min":0.52100000000000002},"max":{"1min":5.1890000000000001,"5min":5.1890000000000001,"15min":13.464},"last":1.2849999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186712064,"data_stored":186712064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":987693056,"data_stored":987693056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196345856,"data_stored":196345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":534577152,"data_stored":534577152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":540409856,"data_stored":540409856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":285999104,"data_stored":285999104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":342622208,"data_stored":342622208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":480903168,"data_stored":480903168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":416546816,"data_stored":416546816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":496500736,"data_stored":496500736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":713818112,"data_stored":713818112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:14:13.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-02-12T16:14:13.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:13.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:13.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:13.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:13.830 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:13.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:14:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:13.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:14:13.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:13.954 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:13 smithi123 bash[19119]: audit 2024-02-12T16:14:12.388670+0000 mgr.y (mgr.14146) 865 : audit [DBG] from='client.25349 v1:172.21.15.121:0/2779642471' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:14:13.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:13.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:14.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:14.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:14.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:14.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 15 56074 56059 218.961 167.125 0.00592359 0.00456551 2024-02-12T16:14:14.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:13 smithi121 bash[18489]: audit 2024-02-12T16:14:12.388670+0000 mgr.y (mgr.14146) 865 : audit [DBG] from='client.25349 v1:172.21.15.121:0/2779642471' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:14:14.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:14.227 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:14.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:14.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:13 smithi169 bash[19604]: audit 2024-02-12T16:14:12.388670+0000 mgr.y (mgr.14146) 865 : audit [DBG] from='client.25349 v1:172.21.15.121:0/2779642471' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:14:14.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:14.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:14.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:14.459 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:14.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:14.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:14.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:14.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:14.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:14.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:14.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:14.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:14.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:14.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:14.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:14.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:15.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:15.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:15.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:14 smithi121 bash[18489]: cluster 2024-02-12T16:14:13.701806+0000 mgr.y (mgr.14146) 866 : cluster [DBG] pgmap v1109: 17 pgs: 17 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.33k op/s 2024-02-12T16:14:15.147 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:15.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:15.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 58882 58866 216.4 175.438 0.0155199 0.00461954 2024-02-12T16:14:15.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:15.262 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:14 smithi169 bash[19604]: cluster 2024-02-12T16:14:13.701806+0000 mgr.y (mgr.14146) 866 : cluster [DBG] pgmap v1109: 17 pgs: 17 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.33k op/s 2024-02-12T16:14:15.279 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:15.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:15.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:14 smithi123 bash[19119]: cluster 2024-02-12T16:14:13.701806+0000 mgr.y (mgr.14146) 866 : cluster [DBG] pgmap v1109: 17 pgs: 17 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 146 MiB/s wr, 2.33k op/s 2024-02-12T16:14:15.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:15.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:15.530 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:14:15.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:15.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:15.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:15.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:15.806 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:15.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:15.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:15.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:16.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:16.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:16.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 61586 61570 213.766 169 0.00543171 0.0046766 2024-02-12T16:14:16.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:16.191 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:16.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:16.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:16.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:16.432 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:16 smithi169 bash[19604]: cluster 2024-02-12T16:14:15.702249+0000 mgr.y (mgr.14146) 867 : cluster [DBG] pgmap v1110: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 190 MiB/s wr, 3.03k op/s 2024-02-12T16:14:16.457 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:16.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.592 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:16.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:16.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:16 smithi123 bash[19119]: cluster 2024-02-12T16:14:15.702249+0000 mgr.y (mgr.14146) 867 : cluster [DBG] pgmap v1110: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 190 MiB/s wr, 3.03k op/s 2024-02-12T16:14:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:16 smithi121 bash[18489]: cluster 2024-02-12T16:14:15.702249+0000 mgr.y (mgr.14146) 867 : cluster [DBG] pgmap v1110: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 190 MiB/s wr, 3.03k op/s 2024-02-12T16:14:16.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:16.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:16.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:16.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:16.982 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:16.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:17.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:17.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:17.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:17.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 64144 64128 210.929 159.875 0.00557507 0.00473905 2024-02-12T16:14:17.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:17.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:17.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:17.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:17.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:17.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:17.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:17.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:17.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:17.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:17.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:17.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:17.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:17.774 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:17.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:17.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:17.909 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:17.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:18.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:18.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:18.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:18.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:14:18.155193 min lat: 0.0010598 max lat: 0.735335 avg lat: 0.00479868 2024-02-12T16:14:18.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:14:18.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 16 66687 66671 208.329 158.938 0.00857404 0.00479868 2024-02-12T16:14:18.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:18.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:18.307 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:18.407 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:18.408 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:18.409 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:18.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:18.436 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:18.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:18.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:18.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:18.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:18.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:18.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:18.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:18.933 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:18 smithi123 bash[19119]: cluster 2024-02-12T16:14:17.702660+0000 mgr.y (mgr.14146) 868 : cluster [DBG] pgmap v1111: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.79k op/s 2024-02-12T16:14:18.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:18.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:18 smithi169 bash[19604]: cluster 2024-02-12T16:14:17.702660+0000 mgr.y (mgr.14146) 868 : cluster [DBG] pgmap v1111: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.79k op/s 2024-02-12T16:14:19.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:19.087 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:19.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:19.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 68655 68639 204.265 123 0.00514981 0.00489398 2024-02-12T16:14:19.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:18 smithi121 bash[18489]: cluster 2024-02-12T16:14:17.702660+0000 mgr.y (mgr.14146) 868 : cluster [DBG] pgmap v1111: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.79k op/s 2024-02-12T16:14:19.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:19.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:19.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:19.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:19.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:19.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:19.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:19.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:19.481 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:19.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:19.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:19.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:19.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:19.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:19.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:19.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:19.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:19.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:19.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:20.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:20.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:20.146 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:20.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:20.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 71119 71103 201.98 154 0.00798106 0.00494917 2024-02-12T16:14:20.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:20.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:20.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:20.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:20.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:20.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:115388 2.2 2:434a7624:::benchmark_data_smithi169_42458_object115387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:03.677896+0000", 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 556.77694789899999, 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.70456666099999998, 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.463 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 115388 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.677896+0000", 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.677896+0000", 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999919 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.677888+0000", 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0956000000000003e-05 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.677959+0000", 2024-02-12T16:14:20.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.363e-06 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.677968+0000", 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8300000000000003e-06 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.677973+0000", 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5710000000000001e-05 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678019+0000", 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3391000000000001e-05 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678112+0000", 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115167 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678227+0000", 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.70400300299999996 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.382230+0000", 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1628e-05 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.382252+0000", 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012368599999999999 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.382375+0000", 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7223000000000002e-05 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.382463+0000", 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.70456666099999998 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:115393 2.3 2:c705edba:::benchmark_data_smithi169_42458_object115392:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:03.678455+0000", 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 556.77638910799999, 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.71344470299999996, 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 115393 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678455+0000", 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678455+0000", 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678454+0000", 2024-02-12T16:14:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9475000000000002e-05 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678513+0000", 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.883e-06 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678520+0000", 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.924e-06 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678524+0000", 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8243e-05 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678562+0000", 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1026999999999995e-05 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678643+0000", 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8132000000000002e-05 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2", 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.678741+0000", 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.70874897599999997 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.387490+0000", 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042896999999999996 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.391780+0000", 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6239000000000001e-05 2024-02-12T16:14:20.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.391846+0000", 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3402000000000001e-05 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.391900+0000", 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.71344470299999996 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.14811.0:115388 2.2 e167/147 ondisk, result = 0)", 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:03.682499+0000", 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 556.77234451899994, 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.699977285, 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682499+0000", 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682499+0000", 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682498+0000", 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8539999999999999e-06 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682505+0000", 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.69634078099999996 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.378845+0000", 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5007e-05 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.378870+0000", 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033272589999999999 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.382198+0000", 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5962000000000001e-05 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.382214+0000", 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000263176 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.382477+0000", 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.699977285 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.479 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.14811.0:115393 2.3 e167/147 ondisk, result = 0)", 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:03.682569+0000", 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 556.77227470900004, 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.70492300799999996, 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.480 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:14:20.483 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682569+0000", 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682569+0000", 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682569+0000", 2024-02-12T16:14:20.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9469999999999999e-06 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:03.682572+0000", 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.69799926599999995 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.380571+0000", 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8677000000000001e-05 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.380590+0000", 2024-02-12T16:14:20.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068843309999999996 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.387474+0000", 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2339999999999994e-06 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.387482+0000", 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0081000000000001e-05 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:04.387492+0000", 2024-02-12T16:14:20.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.70492300799999996 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:149162 2.13 2:c9245137:::benchmark_data_smithi169_42458_object149161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e171)", 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:28.245866+0000", 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 532.20897788900004, 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0483163600000001, 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:20.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:20.488 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 149162 2024-02-12T16:14:20.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.488 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245866+0000", 2024-02-12T16:14:20.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245866+0000", 2024-02-12T16:14:20.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:14:20.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245864+0000", 2024-02-12T16:14:20.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112972 2024-02-12T16:14:20.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245977+0000", 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3118999999999999e-05 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245990+0000", 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0609999999999996e-06 2024-02-12T16:14:20.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245998+0000", 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.16112722600000001 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.407125+0000", 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0274e-05 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.407135+0000", 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.88697637299999998 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.294112+0000", 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0612999999999996e-05 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.294182+0000", 2024-02-12T16:14:20.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0483163600000001 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:149169 2.12 2:4cbd32e3:::benchmark_data_smithi169_42458_object149168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e171)", 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:28.246011+0000", 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 532.20883325800003, 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0495018570000001, 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:20.495 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 149169 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246011+0000", 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246011+0000", 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246010+0000", 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000138379 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246148+0000", 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7579999999999998e-06 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246157+0000", 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9059999999999999e-06 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246163+0000", 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13304712499999999 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.379210+0000", 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5546999999999999e-05 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.379225+0000", 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.91619120899999995 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.295416+0000", 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6088000000000006e-05 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.295513+0000", 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0495018570000001 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:69366 3.c 3:3bd173c3:::benchmark_data_smithi169_51390_object69365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-12T16:14:20.499 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:09:29.954678+0000", 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 290.500166417, 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31605743800000002, 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 69366 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954678+0000", 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954678+0000", 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954675+0000", 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1056000000000006e-05 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954736+0000", 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3124e-05 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954749+0000", 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9680000000000006e-06 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954757+0000", 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0074999999999998e-05 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954797+0000", 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9043999999999996e-05 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.954896+0000", 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105684 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.955002+0000", 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00185086 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:29.956853+0000", 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.313717317 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:30.270570+0000", 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5620999999999998e-05 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:30.270666+0000", 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9074999999999998e-05 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:09:30.270735+0000", 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31605743800000002 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.505 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:219664 3.5 3:ac1b9cb3:::benchmark_data_smithi169_51390_object55880:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:33.997844+0000", 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 166.45700011100001, 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42621488499999999, 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 219664 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997844+0000", 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997844+0000", 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:14:20.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997842+0000", 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8320000000000006e-06 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997849+0000", 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1868999999999999e-05 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997861+0000", 2024-02-12T16:14:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2669999999999997e-06 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997867+0000", 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8968000000000003e-05 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997916+0000", 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6483000000000006e-05 2024-02-12T16:14:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997992+0000", 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8905000000000003e-05 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998081+0000", 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044546769999999998 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.002536+0000", 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8839e-05 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.002554+0000", 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4565000000000003e-05 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.002619+0000", 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42143968500000001 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424059+0000", 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42621488499999999 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:219665 3.c 3:3e688d3f:::benchmark_data_smithi169_51390_object55881:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:33.998798+0000", 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 166.456045907, 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43492321900000003, 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 219665 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998798+0000", 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998798+0000", 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998797+0000", 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0520000000000002e-06 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998805+0000", 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9300000000000003e-06 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998813+0000", 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3069999999999996e-06 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998817+0000", 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6118999999999997e-05 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998863+0000", 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105734 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998969+0000", 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011653400000000001 2024-02-12T16:14:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.999085+0000", 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42522143299999998 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424307+0000", 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092645780000000007 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433571+0000", 2024-02-12T16:14:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4145000000000002e-05 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433646+0000", 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5753999999999993e-05 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433721+0000", 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43492321900000003 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:219676 3.d e304/256 3:bd218378:::benchmark_data_smithi169_51390_object55892:head v 304'13548, mlcod=304'13546)", 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:34.005655+0000", 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 166.449189109, 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43757516699999999, 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005655+0000", 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005655+0000", 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005654+0000", 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8189999999999998e-06 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005662+0000", 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.417952029 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423614+0000", 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8082999999999999e-05 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423642+0000", 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043031599999999998 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424073+0000", 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8934999999999997e-05 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424132+0000", 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019061813 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.443194+0000", 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6468999999999999e-05 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.443230+0000", 2024-02-12T16:14:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43757516699999999 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:219684 3.c 3:39f55561:::benchmark_data_smithi169_51390_object55900:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:34.013105+0000", 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 166.441738743, 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42544842399999999, 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 219684 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013105+0000", 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013105+0000", 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013102+0000", 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4159999999999993e-06 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013112+0000", 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.41051737199999999 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423629+0000", 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0295999999999999e-05 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423659+0000", 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079160000000000005 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424451+0000", 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031754599999999998 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424769+0000", 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037539399999999998 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.425144+0000", 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008653193 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433797+0000", 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046760500000000002 2024-02-12T16:14:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.438473+0000", 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5593999999999998e-05 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.438509+0000", 2024-02-12T16:14:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4826000000000001e-05 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.438554+0000", 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42544842399999999 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:222371 3.5 3:a8591d55:::benchmark_data_smithi169_51390_object58587:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:36.689417+0000", 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 163.765426942, 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32083903699999999, 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 222371 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689417+0000", 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689417+0000", 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689415+0000", 2024-02-12T16:14:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8939999999999993e-06 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689423+0000", 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4545999999999999e-05 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689437+0000", 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1400000000000006e-06 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689446+0000", 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4110000000000002e-05 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689500+0000", 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001303 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689631+0000", 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014779999999999999 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689778+0000", 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055622780000000004 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.695341+0000", 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31480284800000002 2024-02-12T16:14:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010144+0000", 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4207000000000004e-05 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010208+0000", 2024-02-12T16:14:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8272999999999997e-05 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010256+0000", 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32083903699999999 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:222373 3.c 3:37edf101:::benchmark_data_smithi169_51390_object58589:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:36.692259+0000", 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 163.762585, 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.30243373299999998, 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 222373 2024-02-12T16:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692259+0000", 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692259+0000", 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692257+0000", 2024-02-12T16:14:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2659e-05 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692269+0000", 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6239000000000002e-05 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692286+0000", 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7729999999999999e-06 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692293+0000", 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7355999999999999e-05 2024-02-12T16:14:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692341+0000", 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012307099999999999 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692464+0000", 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014246300000000001 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692606+0000", 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046957240000000001 2024-02-12T16:14:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.697302+0000", 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.29709079199999999 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.994393+0000", 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016829900000000001 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.994561+0000", 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013167300000000001 2024-02-12T16:14:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.994693+0000", 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.30243373299999998 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:222385 3.5 3:ae306d2e:::benchmark_data_smithi169_51390_object58601:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:36.699414+0000", 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 163.75542974800001, 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31099299200000002, 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 222385 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699414+0000", 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699414+0000", 2024-02-12T16:14:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699413+0000", 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0499999999999992e-06 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699421+0000", 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0274e-05 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699431+0000", 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7480000000000001e-06 2024-02-12T16:14:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699436+0000", 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2242000000000001e-05 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699468+0000", 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2415000000000003e-05 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699550+0000", 2024-02-12T16:14:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5103000000000002e-05 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699636+0000", 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042469489999999999 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.703883+0000", 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.30642461300000001 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010307+0000", 2024-02-12T16:14:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3876999999999999e-05 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010361+0000", 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6155000000000001e-05 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010407+0000", 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31099299200000002 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287207 3.c 3:3621cc5d:::benchmark_data_smithi169_51390_object123423:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.493969+0000", 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 105.960874988, 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50046115199999996, 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287207 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493969+0000", 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493969+0000", 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493967+0000", 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3289999999999996e-06 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493975+0000", 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493986+0000", 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3589999999999996e-06 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493992+0000", 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2116000000000001e-05 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.494035+0000", 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111876 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.494146+0000", 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015048199999999999 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.494297+0000", 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043225190000000004 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498619+0000", 2024-02-12T16:14:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094397479999999995 2024-02-12T16:14:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508059+0000", 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1324e-05 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508091+0000", 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48633959500000001 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994430+0000", 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50046115199999996 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287212 3.5 3:a506526d:::benchmark_data_smithi169_51390_object123428:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.503073+0000", 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 105.951771111, 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49480824099999998, 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287212 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503073+0000", 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503073+0000", 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503071+0000", 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0090000000000001e-06 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503078+0000", 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8640000000000008e-06 2024-02-12T16:14:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503088+0000", 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7060000000000003e-06 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503092+0000", 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7908999999999999e-05 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503130+0000", 2024-02-12T16:14:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100688 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503231+0000", 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7761000000000003e-05 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503309+0000", 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043448489999999996 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507654+0000", 2024-02-12T16:14:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48952226900000001 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997176+0000", 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034127499999999999 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997517+0000", 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036395299999999998 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997881+0000", 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49480824099999998 2024-02-12T16:14:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287214 3.c 3:33c50097:::benchmark_data_smithi169_51390_object123430:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.503324+0000", 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 105.95151970400001, 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50303783700000004, 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287214 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503324+0000", 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503324+0000", 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503323+0000", 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9109999999999998e-06 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503329+0000", 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2590000000000006e-06 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503338+0000", 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7750000000000004e-06 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503345+0000", 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5648000000000001e-05 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503390+0000", 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9902999999999998e-05 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503460+0000", 2024-02-12T16:14:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4788000000000003e-05 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503535+0000", 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.490980374 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994515+0000", 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011741451999999999 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006257+0000", 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0683999999999997e-05 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006318+0000", 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4465000000000002e-05 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006362+0000", 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50303783700000004 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.15108.0:287207 3.c e318/256 ondisk, result = 0)", 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 105.948197738, 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48782185099999997, 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0639999999999998e-06 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506649+0000", 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5179999999999998e-06 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506653+0000", 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506658+0000", 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388727 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508047+0000", 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0949999999999998e-06 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508054+0000", 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48641379800000001 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994468+0000", 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48782185099999997 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287220 3.c 3:35aadc3b:::benchmark_data_smithi169_51390_object123436:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.508033+0000", 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 105.946810965, 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49976067499999999, 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287220 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508033+0000", 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508033+0000", 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508031+0000", 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6420000000000001e-06 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508038+0000", 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.089e-06 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508046+0000", 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7230000000000004e-06 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508050+0000", 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48656027800000001 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994611+0000", 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00045055900000000001 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.995061+0000", 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054616300000000001 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.995607+0000", 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010806431 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006414+0000", 2024-02-12T16:14:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8079999999999999e-06 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006421+0000", 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7545e-05 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006468+0000", 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013254810000000001 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007794+0000", 2024-02-12T16:14:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49976067499999999 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.15108.0:287212 3.5 e318/256 ondisk, result = 0)", 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.511973+0000", 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 105.94287068, 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.486612779, 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511973+0000", 2024-02-12T16:14:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511973+0000", 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511971+0000", 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0340000000000002e-06 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511980+0000", 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48277212000000003 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994752+0000", 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9496e-05 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994822+0000", 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022711879999999999 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997093+0000", 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4272e-05 2024-02-12T16:14:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997127+0000", 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001458593 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998586+0000", 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.486612779 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:20.581 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:14:20.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:20.612 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:20.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:20.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:20.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:20.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:20.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:20.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:20.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:20 smithi123 bash[19119]: cluster 2024-02-12T16:14:19.703179+0000 mgr.y (mgr.14146) 869 : cluster [DBG] pgmap v1112: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 175 MiB/s wr, 2.79k op/s 2024-02-12T16:14:20.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:21.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:20 smithi169 bash[19604]: cluster 2024-02-12T16:14:19.703179+0000 mgr.y (mgr.14146) 869 : cluster [DBG] pgmap v1112: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 175 MiB/s wr, 2.79k op/s 2024-02-12T16:14:21.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:21.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:20 smithi121 bash[18489]: cluster 2024-02-12T16:14:19.703179+0000 mgr.y (mgr.14146) 869 : cluster [DBG] pgmap v1112: 17 pgs: 17 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 175 MiB/s wr, 2.79k op/s 2024-02-12T16:14:21.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:21.157 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 73363 73347 199.295 140.25 0.00882419 0.00501561 2024-02-12T16:14:21.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:21.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:21.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:14:21.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:21.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:21.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:21.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:21.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:21.679 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:21.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:21.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:21.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:21.945 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:21.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:22.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:22.071 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:22.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:22.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 75619 75603 196.866 141 0.016667 0.00507753 2024-02-12T16:14:22.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:22.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:22.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:22.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:22.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:22.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:22.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:22.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:22.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:22.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:22.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:22.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:22.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:22.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:22.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:22.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:22.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:23.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:22 smithi169 bash[19604]: cluster 2024-02-12T16:14:21.189077+0000 osd.8 (osd.8) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:22 smithi169 bash[19604]: cluster 2024-02-12T16:14:21.189127+0000 osd.8 (osd.8) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:22 smithi169 bash[19604]: cluster 2024-02-12T16:14:21.703561+0000 mgr.y (mgr.14146) 870 : cluster [DBG] pgmap v1113: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s wr, 2.40k op/s 2024-02-12T16:14:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:22 smithi169 bash[19604]: cluster 2024-02-12T16:14:21.766029+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e342: 12 total, 12 up, 12 in 2024-02-12T16:14:23.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:23.113 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:22 smithi123 bash[19119]: cluster 2024-02-12T16:14:21.189077+0000 osd.8 (osd.8) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:23.113 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:22 smithi123 bash[19119]: cluster 2024-02-12T16:14:21.189127+0000 osd.8 (osd.8) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:23.113 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:22 smithi123 bash[19119]: cluster 2024-02-12T16:14:21.703561+0000 mgr.y (mgr.14146) 870 : cluster [DBG] pgmap v1113: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s wr, 2.40k op/s 2024-02-12T16:14:23.113 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:22 smithi123 bash[19119]: cluster 2024-02-12T16:14:21.766029+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e342: 12 total, 12 up, 12 in 2024-02-12T16:14:23.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:22 smithi121 bash[18489]: cluster 2024-02-12T16:14:21.189077+0000 osd.8 (osd.8) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:23.126 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:22 smithi121 bash[18489]: cluster 2024-02-12T16:14:21.189127+0000 osd.8 (osd.8) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:23.126 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:22 smithi121 bash[18489]: cluster 2024-02-12T16:14:21.703561+0000 mgr.y (mgr.14146) 870 : cluster [DBG] pgmap v1113: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s wr, 2.40k op/s 2024-02-12T16:14:23.126 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:22 smithi121 bash[18489]: cluster 2024-02-12T16:14:21.766029+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e342: 12 total, 12 up, 12 in 2024-02-12T16:14:23.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:23.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 77867 77851 194.611 140.5 0.0171852 0.00513704 2024-02-12T16:14:23.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:23.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:23.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:23.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:23.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:23.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:23.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:23.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:23.759 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:23.788 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:23.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:23.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:23.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:24.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:24.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:24.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 16 80089 80073 192.467 138.875 0.00660576 0.00519412 2024-02-12T16:14:24.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:24.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:24.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:24.337 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:24.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:24.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:24.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:24.605 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:24.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:24.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:24.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:24.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:24 smithi169 bash[19604]: cluster 2024-02-12T16:14:23.411750+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:24.845 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:24 smithi169 bash[19604]: cluster 2024-02-12T16:14:23.411803+0000 osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:24.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:24 smithi169 bash[19604]: cluster 2024-02-12T16:14:23.703985+0000 mgr.y (mgr.14146) 871 : cluster [DBG] pgmap v1115: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.61k op/s 2024-02-12T16:14:24.846 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:24 smithi169 bash[19604]: cluster 2024-02-12T16:14:23.770102+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e343: 12 total, 12 up, 12 in 2024-02-12T16:14:24.862 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:24.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:24.994 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:24.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:25.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:25.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:24 smithi121 bash[18489]: cluster 2024-02-12T16:14:23.411750+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:25.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:24 smithi121 bash[18489]: cluster 2024-02-12T16:14:23.411803+0000 osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:25.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:24 smithi121 bash[18489]: cluster 2024-02-12T16:14:23.703985+0000 mgr.y (mgr.14146) 871 : cluster [DBG] pgmap v1115: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.61k op/s 2024-02-12T16:14:25.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:24 smithi121 bash[18489]: cluster 2024-02-12T16:14:23.770102+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e343: 12 total, 12 up, 12 in 2024-02-12T16:14:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:25.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:24 smithi123 bash[19119]: cluster 2024-02-12T16:14:23.411750+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:24 smithi123 bash[19119]: cluster 2024-02-12T16:14:23.411803+0000 osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:24 smithi123 bash[19119]: cluster 2024-02-12T16:14:23.703985+0000 mgr.y (mgr.14146) 871 : cluster [DBG] pgmap v1115: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 MiB/s wr, 2.61k op/s 2024-02-12T16:14:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:24 smithi123 bash[19119]: cluster 2024-02-12T16:14:23.770102+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e343: 12 total, 12 up, 12 in 2024-02-12T16:14:25.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 82251 82235 190.342 135.125 0.00322537 0.00525146 2024-02-12T16:14:25.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:25.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:25.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:25.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:25.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:25.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:25.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:25.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:25.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:25.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:25.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:25.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:25.857 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:25.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:25 smithi123 bash[19119]: cluster 2024-02-12T16:14:24.063962+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:25 smithi123 bash[19119]: cluster 2024-02-12T16:14:24.064007+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:25 smithi123 bash[19119]: cluster 2024-02-12T16:14:24.780475+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e344: 12 total, 12 up, 12 in 2024-02-12T16:14:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:25 smithi123 bash[19119]: cluster 2024-02-12T16:14:25.378862+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e345: 12 total, 12 up, 12 in 2024-02-12T16:14:25.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:25.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:26.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:25 smithi169 bash[19604]: cluster 2024-02-12T16:14:24.063962+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:26.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:25 smithi169 bash[19604]: cluster 2024-02-12T16:14:24.064007+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:26.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:25 smithi169 bash[19604]: cluster 2024-02-12T16:14:24.780475+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e344: 12 total, 12 up, 12 in 2024-02-12T16:14:26.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:25 smithi169 bash[19604]: cluster 2024-02-12T16:14:25.378862+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e345: 12 total, 12 up, 12 in 2024-02-12T16:14:26.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:26.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:25 smithi121 bash[18489]: cluster 2024-02-12T16:14:24.063962+0000 osd.4 (osd.4) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:26.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:25 smithi121 bash[18489]: cluster 2024-02-12T16:14:24.064007+0000 osd.4 (osd.4) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:26.113 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:25 smithi121 bash[18489]: cluster 2024-02-12T16:14:24.780475+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e344: 12 total, 12 up, 12 in 2024-02-12T16:14:26.113 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:25 smithi121 bash[18489]: cluster 2024-02-12T16:14:25.378862+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e345: 12 total, 12 up, 12 in 2024-02-12T16:14:26.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:26.146 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:14:26.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 84325 84309 188.174 129.625 0.00774387 0.00531252 2024-02-12T16:14:26.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:26.266 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:26.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:26.395 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:26.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:26.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:26.763 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:26.790 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:26.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:26.904 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:26 smithi121 bash[18489]: cluster 2024-02-12T16:14:25.350667+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:26.904 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:26 smithi121 bash[18489]: cluster 2024-02-12T16:14:25.350725+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:26.904 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:26 smithi121 bash[18489]: cluster 2024-02-12T16:14:25.704390+0000 mgr.y (mgr.14146) 872 : cluster [DBG] pgmap v1119: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 177 MiB/s wr, 2.84k op/s 2024-02-12T16:14:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:26.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:27.039 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:27.046 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:26 smithi123 bash[19119]: cluster 2024-02-12T16:14:25.350667+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:27.047 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:26 smithi123 bash[19119]: cluster 2024-02-12T16:14:25.350725+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:27.047 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:26 smithi123 bash[19119]: cluster 2024-02-12T16:14:25.704390+0000 mgr.y (mgr.14146) 872 : cluster [DBG] pgmap v1119: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 177 MiB/s wr, 2.84k op/s 2024-02-12T16:14:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:26 smithi169 bash[19604]: cluster 2024-02-12T16:14:25.350667+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:26 smithi169 bash[19604]: cluster 2024-02-12T16:14:25.350725+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:26 smithi169 bash[19604]: cluster 2024-02-12T16:14:25.704390+0000 mgr.y (mgr.14146) 872 : cluster [DBG] pgmap v1119: 17 pgs: 17 active+clean; 4.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 177 MiB/s wr, 2.84k op/s 2024-02-12T16:14:27.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:27.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:27.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 86069 86053 185.443 109 0.00566649 0.00539046 2024-02-12T16:14:27.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:27.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:27.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:27.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:27.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:27.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:27.418 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:14:27.442 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:14:27.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:27.460 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:14:27.485 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:14:27.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:27.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:27.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:27.505 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:14:27.538 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:14:27.567 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:14:27.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:27.598 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:14:27.618 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:27.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:27.632 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:14:27.662 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:14:27.689 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:14:27.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:27.724 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:14:27.728 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.728 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.728 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.729 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.729 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.729 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.729 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.729 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.729 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.730 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.730 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.730 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:14:27.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:27.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:14:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 4 pgp_num to 16 2024-02-12T16:14:27.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:27.879 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:27.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:27.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:27 smithi123 bash[19119]: audit 2024-02-12T16:14:27.377838+0000 mon.a (mon.0) 1353 : audit [DBG] from='client.? v1:172.21.15.121:0/2320891356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:27.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:27 smithi123 bash[19119]: audit 2024-02-12T16:14:27.414683+0000 mon.a (mon.0) 1354 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:14:27.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:27 smithi123 bash[19119]: audit 2024-02-12T16:14:27.417908+0000 mon.b (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.121:0/1810768688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:14:27.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:27 smithi123 bash[19119]: audit 2024-02-12T16:14:27.491875+0000 mon.a (mon.0) 1355 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-12T16:14:27.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:27 smithi123 bash[19119]: audit 2024-02-12T16:14:27.493725+0000 mon.c (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.121:0/1048070338' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-12T16:14:27.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:28.016 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:27 smithi169 bash[19604]: audit 2024-02-12T16:14:27.377838+0000 mon.a (mon.0) 1353 : audit [DBG] from='client.? v1:172.21.15.121:0/2320891356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:27 smithi169 bash[19604]: audit 2024-02-12T16:14:27.414683+0000 mon.a (mon.0) 1354 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:14:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:27 smithi169 bash[19604]: audit 2024-02-12T16:14:27.417908+0000 mon.b (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.121:0/1810768688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:14:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:27 smithi169 bash[19604]: audit 2024-02-12T16:14:27.491875+0000 mon.a (mon.0) 1355 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-12T16:14:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:27 smithi169 bash[19604]: audit 2024-02-12T16:14:27.493725+0000 mon.c (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.121:0/1048070338' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-12T16:14:28.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:28.146 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:28.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 87702 87686 182.664 102.062 0.00360927 0.00547264 2024-02-12T16:14:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:27 smithi121 bash[18489]: audit 2024-02-12T16:14:27.377838+0000 mon.a (mon.0) 1353 : audit [DBG] from='client.? v1:172.21.15.121:0/2320891356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:27 smithi121 bash[18489]: audit 2024-02-12T16:14:27.414683+0000 mon.a (mon.0) 1354 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:14:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:27 smithi121 bash[18489]: audit 2024-02-12T16:14:27.417908+0000 mon.b (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.121:0/1810768688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:14:28.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:27 smithi121 bash[18489]: audit 2024-02-12T16:14:27.491875+0000 mon.a (mon.0) 1355 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-12T16:14:28.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:27 smithi121 bash[18489]: audit 2024-02-12T16:14:27.493725+0000 mon.c (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.121:0/1048070338' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-12T16:14:28.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:28.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:28.412 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:28.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:28.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:28.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:28.686 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:28.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:28.825 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:28.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:28.936 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:28 smithi123 bash[19119]: cluster 2024-02-12T16:14:27.704842+0000 mgr.y (mgr.14146) 873 : cluster [DBG] pgmap v1120: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-12T16:14:28.936 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:28 smithi123 bash[19119]: cluster 2024-02-12T16:14:27.782194+0000 mon.a (mon.0) 1356 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:14:28.936 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:28 smithi123 bash[19119]: audit 2024-02-12T16:14:27.791963+0000 mon.a (mon.0) 1357 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:14:28.936 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:28 smithi123 bash[19119]: audit 2024-02-12T16:14:27.792022+0000 mon.a (mon.0) 1358 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-02-12T16:14:28.936 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:28 smithi123 bash[19119]: cluster 2024-02-12T16:14:27.792071+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e346: 12 total, 12 up, 12 in 2024-02-12T16:14:28.953 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:28.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:28 smithi169 bash[19604]: cluster 2024-02-12T16:14:27.704842+0000 mgr.y (mgr.14146) 873 : cluster [DBG] pgmap v1120: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-12T16:14:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:28 smithi169 bash[19604]: cluster 2024-02-12T16:14:27.782194+0000 mon.a (mon.0) 1356 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:14:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:28 smithi169 bash[19604]: audit 2024-02-12T16:14:27.791963+0000 mon.a (mon.0) 1357 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:14:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:28 smithi169 bash[19604]: audit 2024-02-12T16:14:27.792022+0000 mon.a (mon.0) 1358 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-02-12T16:14:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:28 smithi169 bash[19604]: cluster 2024-02-12T16:14:27.792071+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e346: 12 total, 12 up, 12 in 2024-02-12T16:14:29.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:29.081 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:29.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 88931 88915 179.249 76.8125 0.0201173 0.00557688 2024-02-12T16:14:29.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:29.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.215 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:29.234 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:28 smithi121 bash[18489]: cluster 2024-02-12T16:14:27.704842+0000 mgr.y (mgr.14146) 873 : cluster [DBG] pgmap v1120: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 MiB/s wr, 2.46k op/s 2024-02-12T16:14:29.234 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:28 smithi121 bash[18489]: cluster 2024-02-12T16:14:27.782194+0000 mon.a (mon.0) 1356 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:14:29.234 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:28 smithi121 bash[18489]: audit 2024-02-12T16:14:27.791963+0000 mon.a (mon.0) 1357 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:14:29.234 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:28 smithi121 bash[18489]: audit 2024-02-12T16:14:27.792022+0000 mon.a (mon.0) 1358 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-02-12T16:14:29.234 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:28 smithi121 bash[18489]: cluster 2024-02-12T16:14:27.792071+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e346: 12 total, 12 up, 12 in 2024-02-12T16:14:29.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:29.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:29.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:29.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:29.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:29.605 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:29.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:29.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:29.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:29.858 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:29.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:29.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:29.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:29.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:30.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:30.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:30.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:30.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 90923 90907 177.538 124.5 0.00462587 0.0056312 2024-02-12T16:14:30.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:30.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:30.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:30.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:30.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:30.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:30.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:30.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:30.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:30.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:30.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:30.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:30.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:30.764 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:14:30.764 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:14:30.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:30.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:30 smithi123 bash[19119]: cluster 2024-02-12T16:14:29.705316+0000 mgr.y (mgr.14146) 874 : cluster [DBG] pgmap v1122: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-02-12T16:14:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:30 smithi123 bash[19119]: cluster 2024-02-12T16:14:30.377529+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e347: 12 total, 12 up, 12 in 2024-02-12T16:14:30.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:30.903 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:30.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:31.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:31.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:30 smithi169 bash[19604]: cluster 2024-02-12T16:14:29.705316+0000 mgr.y (mgr.14146) 874 : cluster [DBG] pgmap v1122: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-02-12T16:14:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:30 smithi169 bash[19604]: cluster 2024-02-12T16:14:30.377529+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e347: 12 total, 12 up, 12 in 2024-02-12T16:14:31.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:31.158 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 92770 92754 175.655 115.438 0.0105807 0.00569144 2024-02-12T16:14:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:30 smithi121 bash[18489]: cluster 2024-02-12T16:14:29.705316+0000 mgr.y (mgr.14146) 874 : cluster [DBG] pgmap v1122: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.05k op/s 2024-02-12T16:14:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:30 smithi121 bash[18489]: cluster 2024-02-12T16:14:30.377529+0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e347: 12 total, 12 up, 12 in 2024-02-12T16:14:31.161 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:31.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:31.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:31.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:31.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:31.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:14:31.526 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:31.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:14:31.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:14:31.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:31.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:31.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:31.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:31.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:31.947 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:31.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:32.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:32.078 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:32.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:32.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 94688 94672 174.015 119.875 0.0119802 0.00574531 2024-02-12T16:14:32.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:32.208 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:32.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:32.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:32.344 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:32.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:32.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:32.471 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:32.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:32.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:32.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:32.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:32.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:32.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:32.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:32.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:32.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:32.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:32.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:32.980 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:32 smithi169 bash[19604]: cluster 2024-02-12T16:14:31.705716+0000 mgr.y (mgr.14146) 875 : cluster [DBG] pgmap v1124: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-02-12T16:14:32.980 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:32 smithi169 bash[19604]: cluster 2024-02-12T16:14:31.944681+0000 osd.6 (osd.6) 38 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:32.980 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:32 smithi169 bash[19604]: cluster 2024-02-12T16:14:31.944722+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:33.008 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:33.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:33.120 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:32 smithi123 bash[19119]: cluster 2024-02-12T16:14:31.705716+0000 mgr.y (mgr.14146) 875 : cluster [DBG] pgmap v1124: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-02-12T16:14:33.120 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:32 smithi123 bash[19119]: cluster 2024-02-12T16:14:31.944681+0000 osd.6 (osd.6) 38 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:33.121 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:32 smithi123 bash[19119]: cluster 2024-02-12T16:14:31.944722+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:33.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 96613 96597 172.48 120.312 0.00393826 0.00579642 2024-02-12T16:14:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:32 smithi121 bash[18489]: cluster 2024-02-12T16:14:31.705716+0000 mgr.y (mgr.14146) 875 : cluster [DBG] pgmap v1124: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-02-12T16:14:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:32 smithi121 bash[18489]: cluster 2024-02-12T16:14:31.944681+0000 osd.6 (osd.6) 38 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:32 smithi121 bash[18489]: cluster 2024-02-12T16:14:31.944722+0000 osd.6 (osd.6) 39 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:33.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:33.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:33.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:33.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:33.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:33.539 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:33.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:33.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:33.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:33.801 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:33.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:33.932 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:33.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:34.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:34.070 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:34.071 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:33 smithi169 bash[19604]: cluster 2024-02-12T16:14:32.808179+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e348: 12 total, 12 up, 12 in 2024-02-12T16:14:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:34.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:34.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:33 smithi123 bash[19119]: cluster 2024-02-12T16:14:32.808179+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e348: 12 total, 12 up, 12 in 2024-02-12T16:14:34.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 15 98534 98519 171.025 120.125 0.00452116 0.00584521 2024-02-12T16:14:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:33 smithi121 bash[18489]: cluster 2024-02-12T16:14:32.808179+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e348: 12 total, 12 up, 12 in 2024-02-12T16:14:34.174 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:34.205 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:34.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:34.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:34.334 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:34.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:34.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:34.463 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:34.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:34.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:34.591 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:34.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:34.608 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:14:34.609 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:14:34.609 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-12T16:14:34.609 INFO:tasks.thrashosds.thrasher:Setting osd 10 primary_affinity to 1.000000 2024-02-12T16:14:34.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 10 1 2024-02-12T16:14:34.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:34.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:34.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:34.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:34.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:34.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:34 smithi121 bash[18489]: cluster 2024-02-12T16:14:33.706052+0000 mgr.y (mgr.14146) 876 : cluster [DBG] pgmap v1126: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:14:34.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:34.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:34 smithi169 bash[19604]: cluster 2024-02-12T16:14:33.706052+0000 mgr.y (mgr.14146) 876 : cluster [DBG] pgmap v1126: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:14:35.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:35.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:35.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:35.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:34 smithi123 bash[19119]: cluster 2024-02-12T16:14:33.706052+0000 mgr.y (mgr.14146) 876 : cluster [DBG] pgmap v1126: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:14:35.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 100359 100343 169.484 114 0.00364534 0.00589865 2024-02-12T16:14:35.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:35.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:35.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:35.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:35.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:35.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:35.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:35.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:35.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:35.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:35.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:35.925 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:35.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:36.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:36.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:36.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 15 102142 102127 167.958 111.5 0.0148586 0.00595218 2024-02-12T16:14:36.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:36.187 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:36.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:36.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:36.354 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:36.355 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:36.386 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:36 smithi121 bash[18489]: cluster 2024-02-12T16:14:35.378230+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e349: 12 total, 12 up, 12 in 2024-02-12T16:14:36.394 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:14:36.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:36.451 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:36.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:36.591 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:36.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:36 smithi123 bash[19119]: cluster 2024-02-12T16:14:35.378230+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e349: 12 total, 12 up, 12 in 2024-02-12T16:14:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:36 smithi123 bash[19119]: cluster 2024-02-12T16:14:35.706436+0000 mgr.y (mgr.14146) 877 : cluster [DBG] pgmap v1128: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s 2024-02-12T16:14:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:36 smithi121 bash[18489]: cluster 2024-02-12T16:14:35.706436+0000 mgr.y (mgr.14146) 877 : cluster [DBG] pgmap v1128: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s 2024-02-12T16:14:36.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:36.701 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:36 smithi169 bash[19604]: cluster 2024-02-12T16:14:35.378230+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e349: 12 total, 12 up, 12 in 2024-02-12T16:14:36.702 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:36 smithi169 bash[19604]: cluster 2024-02-12T16:14:35.706436+0000 mgr.y (mgr.14146) 877 : cluster [DBG] pgmap v1128: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 147 MiB/s wr, 2.35k op/s 2024-02-12T16:14:36.719 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:36.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:36.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:36.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:36.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:36.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:37.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:37.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:37.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:37.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 103909 103893 166.481 110.375 0.008534 0.00600523 2024-02-12T16:14:37.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:37.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:37.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:37.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:37.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:37.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:37.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:37.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:37.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:37.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:37.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:37.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:37.794 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:37.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:37.899 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:37.923 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:37.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:38.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:38.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:38.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:14:38.156832 min lat: 0.0010598 max lat: 0.735335 avg lat: 0.00606444 2024-02-12T16:14:38.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:14:38.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 105529 105513 164.85 101.25 0.00438883 0.00606444 2024-02-12T16:14:38.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:38.187 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:38.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:38.321 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:38.425 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:38.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:38.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:38.579 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:38.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:38.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:38.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:38.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:38 smithi121 bash[18489]: cluster 2024-02-12T16:14:37.706851+0000 mgr.y (mgr.14146) 878 : cluster [DBG] pgmap v1129: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-02-12T16:14:38.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:38.969 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:38.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:39.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:38 smithi169 bash[19604]: cluster 2024-02-12T16:14:37.706851+0000 mgr.y (mgr.14146) 878 : cluster [DBG] pgmap v1129: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-02-12T16:14:39.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:39.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:39.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:38 smithi123 bash[19119]: cluster 2024-02-12T16:14:37.706851+0000 mgr.y (mgr.14146) 878 : cluster [DBG] pgmap v1129: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-02-12T16:14:39.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 15 107289 107274 163.514 110.062 0.0101166 0.00611409 2024-02-12T16:14:39.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:39.224 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:39.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:39.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:39.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:39.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:39.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:39.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:39.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:39.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:39.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:39.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:39.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:39.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:39.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:39.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:39.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:39.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:39.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:40.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:40.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:40.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:40.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 109018 109002 162.192 108 0.00362249 0.006164 2024-02-12T16:14:40.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:40.285 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:40.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:40.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:40.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:40.545 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:40.651 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:40.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:40.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:40.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:40 smithi169 bash[19604]: cluster 2024-02-12T16:14:39.707355+0000 mgr.y (mgr.14146) 879 : cluster [DBG] pgmap v1130: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-02-12T16:14:40.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:40 smithi169 bash[19604]: cluster 2024-02-12T16:14:39.708001+0000 osd.7 (osd.7) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:40.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:40.844 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:14:40.844 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:14:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:40 smithi123 bash[19119]: cluster 2024-02-12T16:14:39.707355+0000 mgr.y (mgr.14146) 879 : cluster [DBG] pgmap v1130: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-02-12T16:14:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:40 smithi123 bash[19119]: cluster 2024-02-12T16:14:39.708001+0000 osd.7 (osd.7) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:40 smithi123 bash[19119]: cluster 2024-02-12T16:14:39.708055+0000 osd.7 (osd.7) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:40 smithi123 bash[19119]: cluster 2024-02-12T16:14:39.776392+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e350: 12 total, 12 up, 12 in 2024-02-12T16:14:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:40 smithi123 bash[19119]: cluster 2024-02-12T16:14:40.380119+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e351: 12 total, 12 up, 12 in 2024-02-12T16:14:40.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:40.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:40.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:41.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:40 smithi169 bash[19604]: cluster 2024-02-12T16:14:39.708055+0000 osd.7 (osd.7) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:40 smithi169 bash[19604]: cluster 2024-02-12T16:14:39.776392+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e350: 12 total, 12 up, 12 in 2024-02-12T16:14:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:40 smithi169 bash[19604]: cluster 2024-02-12T16:14:40.380119+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e351: 12 total, 12 up, 12 in 2024-02-12T16:14:41.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:40 smithi121 bash[18489]: cluster 2024-02-12T16:14:39.707355+0000 mgr.y (mgr.14146) 879 : cluster [DBG] pgmap v1130: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-02-12T16:14:41.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:40 smithi121 bash[18489]: cluster 2024-02-12T16:14:39.708001+0000 osd.7 (osd.7) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:41.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:40 smithi121 bash[18489]: cluster 2024-02-12T16:14:39.708055+0000 osd.7 (osd.7) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:41.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:40 smithi121 bash[18489]: cluster 2024-02-12T16:14:39.776392+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e350: 12 total, 12 up, 12 in 2024-02-12T16:14:41.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:40 smithi121 bash[18489]: cluster 2024-02-12T16:14:40.380119+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e351: 12 total, 12 up, 12 in 2024-02-12T16:14:41.073 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:41.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 16 110654 110638 160.797 102.25 0.00904551 0.00621747 2024-02-12T16:14:41.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:41.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:41.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:41.334 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:41.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:41.460 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:41.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:41.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:41.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:41.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:41.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:14:41.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:41.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:41.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:41.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:41.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:42.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:42.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:42.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:42.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 112253 112237 159.414 99.9375 0.00391564 0.00627102 2024-02-12T16:14:42.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:42.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:42.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:42.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:42.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:42.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:42.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:42.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:42.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:42.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:42.652 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:42.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:42.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:42.777 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:42.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:42.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:42.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:42.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:42.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:43.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:43.039 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:43.041 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:42 smithi169 bash[19604]: cluster 2024-02-12T16:14:41.707790+0000 mgr.y (mgr.14146) 880 : cluster [DBG] pgmap v1133: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.50k op/s 2024-02-12T16:14:43.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:43.122 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:42 smithi121 bash[18489]: cluster 2024-02-12T16:14:41.707790+0000 mgr.y (mgr.14146) 880 : cluster [DBG] pgmap v1133: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.50k op/s 2024-02-12T16:14:43.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:42 smithi123 bash[19119]: cluster 2024-02-12T16:14:41.707790+0000 mgr.y (mgr.14146) 880 : cluster [DBG] pgmap v1133: 17 pgs: 17 active+clean; 6.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.50k op/s 2024-02-12T16:14:43.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:43.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 113708 113692 157.892 90.9375 0.00359494 0.00633173 2024-02-12T16:14:43.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:43.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:43.302 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:43.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:43.431 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:43.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:43.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:43.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:43.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:43.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:43.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:43.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:43.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:43.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:44.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:44.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:44.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:44.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 16 115299 115283 156.621 99.4375 0.0127426 0.0063829 2024-02-12T16:14:44.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:44.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:44.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:44.377 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:44.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:44.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:44.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:44.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:44.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:44.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:44.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:44.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:44.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:44.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:44.846 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:44 smithi121 bash[18489]: cluster 2024-02-12T16:14:43.708211+0000 mgr.y (mgr.14146) 881 : cluster [DBG] pgmap v1134: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:14:44.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:44 smithi123 bash[19119]: cluster 2024-02-12T16:14:43.708211+0000 mgr.y (mgr.14146) 881 : cluster [DBG] pgmap v1134: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:14:44.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:44.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:45.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:45.032 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:14:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:44 smithi169 bash[19604]: cluster 2024-02-12T16:14:43.708211+0000 mgr.y (mgr.14146) 881 : cluster [DBG] pgmap v1134: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:14:45.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:45.077 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:14:45.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:14:45.135 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:14:45.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:45.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 15 116891 116876 155.407 99.5625 0.0261202 0.00643264 2024-02-12T16:14:45.164 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:14:45.178 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:45.192 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:14:45.222 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:14:45.252 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:14:45.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:45.291 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:14:45.309 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:45.323 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:14:45.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:14:45.374 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.381 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.382 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.382 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.382 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:14:45.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:45.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:45.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:45.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:45.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:45.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:45.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:45.814 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:14:45.816 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:45 smithi169 bash[19604]: audit 2024-02-12T16:14:44.996032+0000 mon.b (mon.2) 116 : audit [DBG] from='client.? v1:172.21.15.121:0/361502543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:45.816 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:45 smithi169 bash[19604]: audit 2024-02-12T16:14:45.292525+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? v1:172.21.15.121:0/2625479971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:14:45.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:45.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:45.967 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:45.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:46.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:46.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:46.113 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:45 smithi121 bash[18489]: audit 2024-02-12T16:14:44.996032+0000 mon.b (mon.2) 116 : audit [DBG] from='client.? v1:172.21.15.121:0/361502543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:46.113 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:45 smithi121 bash[18489]: audit 2024-02-12T16:14:45.292525+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? v1:172.21.15.121:0/2625479971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:14:46.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:45 smithi123 bash[19119]: audit 2024-02-12T16:14:44.996032+0000 mon.b (mon.2) 116 : audit [DBG] from='client.? v1:172.21.15.121:0/361502543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:14:46.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:45 smithi123 bash[19119]: audit 2024-02-12T16:14:45.292525+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? v1:172.21.15.121:0/2625479971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:14:46.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 118338 118322 154.052 90.375 0.00447058 0.00648959 2024-02-12T16:14:46.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:46.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:46.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:46.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:46.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:46.490 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:46.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:46.535 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:14:46.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:46.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:46.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:46.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.754 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:46.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:46.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:46.870 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:46 smithi169 bash[19604]: cluster 2024-02-12T16:14:45.708622+0000 mgr.y (mgr.14146) 882 : cluster [DBG] pgmap v1135: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.75k op/s 2024-02-12T16:14:46.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:46 smithi169 bash[19604]: cluster 2024-02-12T16:14:45.788561+0000 mon.a (mon.0) 1366 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:14:46.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:46 smithi169 bash[19604]: cluster 2024-02-12T16:14:45.788597+0000 mon.a (mon.0) 1367 : cluster [INF] Cluster is now healthy 2024-02-12T16:14:46.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:46 smithi169 bash[19604]: audit 2024-02-12T16:14:45.797689+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? v1:172.21.15.121:0/2625479971' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:14:46.871 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:46 smithi169 bash[19604]: cluster 2024-02-12T16:14:45.797746+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e352: 12 total, 12 up, 12 in 2024-02-12T16:14:46.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:46.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:46.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:47.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:47.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:46 smithi121 bash[18489]: cluster 2024-02-12T16:14:45.708622+0000 mgr.y (mgr.14146) 882 : cluster [DBG] pgmap v1135: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.75k op/s 2024-02-12T16:14:47.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:46 smithi121 bash[18489]: cluster 2024-02-12T16:14:45.788561+0000 mon.a (mon.0) 1366 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:14:47.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:46 smithi121 bash[18489]: cluster 2024-02-12T16:14:45.788597+0000 mon.a (mon.0) 1367 : cluster [INF] Cluster is now healthy 2024-02-12T16:14:47.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:46 smithi121 bash[18489]: audit 2024-02-12T16:14:45.797689+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? v1:172.21.15.121:0/2625479971' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:14:47.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:46 smithi121 bash[18489]: cluster 2024-02-12T16:14:45.797746+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e352: 12 total, 12 up, 12 in 2024-02-12T16:14:47.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:47.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:46 smithi123 bash[19119]: cluster 2024-02-12T16:14:45.708622+0000 mgr.y (mgr.14146) 882 : cluster [DBG] pgmap v1135: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.75k op/s 2024-02-12T16:14:47.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:46 smithi123 bash[19119]: cluster 2024-02-12T16:14:45.788561+0000 mon.a (mon.0) 1366 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:14:47.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:46 smithi123 bash[19119]: cluster 2024-02-12T16:14:45.788597+0000 mon.a (mon.0) 1367 : cluster [INF] Cluster is now healthy 2024-02-12T16:14:47.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:46 smithi123 bash[19119]: audit 2024-02-12T16:14:45.797689+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? v1:172.21.15.121:0/2625479971' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:14:47.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:46 smithi123 bash[19119]: cluster 2024-02-12T16:14:45.797746+0000 mon.a (mon.0) 1369 : cluster [DBG] osdmap e352: 12 total, 12 up, 12 in 2024-02-12T16:14:47.147 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:47.159 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 119927 119911 152.935 99.3125 0.0042331 0.00653737 2024-02-12T16:14:47.251 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:47.279 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:47.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:47.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:47.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:47.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:47.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:47.678 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:47.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:47.828 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-02-12T16:14:47.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:47.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:47.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:47.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:48.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:47 smithi121 bash[18489]: audit 2024-02-12T16:14:47.773063+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? v1:172.21.15.121:0/2549696352' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:14:48.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:48.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:48.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:47 smithi123 bash[19119]: audit 2024-02-12T16:14:47.773063+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? v1:172.21.15.121:0/2549696352' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:14:48.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 121438 121422 151.765 94.4375 0.00520758 0.00658756 2024-02-12T16:14:48.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:48.232 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:48.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:48.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:47 smithi169 bash[19604]: audit 2024-02-12T16:14:47.773063+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? v1:172.21.15.121:0/2549696352' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:14:48.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:48.369 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:48.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:48.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:48.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:48.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:48.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:48.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:48.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:48.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:48.753 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:48.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:48.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:48.886 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:48.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:48 smithi121 bash[18489]: cluster 2024-02-12T16:14:47.709119+0000 mgr.y (mgr.14146) 883 : cluster [DBG] pgmap v1137: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-02-12T16:14:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:48 smithi121 bash[18489]: audit 2024-02-12T16:14:47.802810+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.121:0/2549696352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:14:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:48 smithi121 bash[18489]: cluster 2024-02-12T16:14:47.802873+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e353: 12 total, 12 up, 12 in 2024-02-12T16:14:48.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:49.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:49.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:14:49.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:49.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:48 smithi123 bash[19119]: cluster 2024-02-12T16:14:47.709119+0000 mgr.y (mgr.14146) 883 : cluster [DBG] pgmap v1137: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-02-12T16:14:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:48 smithi123 bash[19119]: audit 2024-02-12T16:14:47.802810+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.121:0/2549696352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:14:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:48 smithi123 bash[19119]: cluster 2024-02-12T16:14:47.802873+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e353: 12 total, 12 up, 12 in 2024-02-12T16:14:49.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 122994 122978 150.696 97.25 0.0174553 0.0066343 2024-02-12T16:14:49.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:49.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:49.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:14:49.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:48 smithi169 bash[19604]: cluster 2024-02-12T16:14:47.709119+0000 mgr.y (mgr.14146) 883 : cluster [DBG] pgmap v1137: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-02-12T16:14:49.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:48 smithi169 bash[19604]: audit 2024-02-12T16:14:47.802810+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.121:0/2549696352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:14:49.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:48 smithi169 bash[19604]: cluster 2024-02-12T16:14:47.802873+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e353: 12 total, 12 up, 12 in 2024-02-12T16:14:49.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:49.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:49.436 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:49.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:49.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:49.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:49.702 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:49.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:49.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:49.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:49.965 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:49.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:50.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:50.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:50.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:50.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 15 124475 124460 149.579 92.625 0.00932424 0.00668385 2024-02-12T16:14:50.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:50.226 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:50.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:50.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:50.355 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:50.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:50.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:50.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:50.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:50.611 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:50.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:50.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:50.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:50.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:50.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:50.891 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:50.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:50.961 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:14:50.961 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:14:50.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:14:50.961 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:14:50.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.961 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161369 2.11 2.34686ef1 (undecoded) ondisk+write+known_if_redirected e177)", 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.981059+0000", 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 551.97543452000002, 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.28505741400000001, 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161369 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981059+0000", 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981059+0000", 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.99999 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981049+0000", 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0915999999999996e-05 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981130+0000", 2024-02-12T16:14:50.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8921999999999999e-05 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981148+0000", 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3240000000000003e-06 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981157+0000", 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8028000000000007e-05 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981225+0000", 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6989999999999992e-06 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.981233+0000", 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.28486699199999999 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266100+0000", 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5511000000000001e-05 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:50.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266116+0000", 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.28505741400000001 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161466 2.11 2.71897f1 (undecoded) ondisk+write+known_if_redirected e177)", 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:39.058989+0000", 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 551.89750364500003, 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.207153054, 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.966 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161466 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.058989+0000", 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.058989+0000", 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:14:50.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.058986+0000", 2024-02-12T16:14:50.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7766000000000002e-05 2024-02-12T16:14:50.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.059044+0000", 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8739000000000001e-05 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.059063+0000", 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1359999999999999e-06 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.059070+0000", 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3465000000000003e-05 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.059134+0000", 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.076e-05 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.059144+0000", 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20699057300000001 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266135+0000", 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5859999999999999e-06 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266142+0000", 2024-02-12T16:14:50.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.207153054 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161474 2.11 2.2aee9a91 (undecoded) ondisk+write+known_if_redirected e177)", 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:39.063385+0000", 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 551.89310842400005, 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20278849400000001, 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:50.972 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161474 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063385+0000", 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063385+0000", 2024-02-12T16:14:50.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.99999 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063374+0000", 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3324000000000004e-05 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063458+0000", 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8317000000000001e-05 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063476+0000", 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8590000000000005e-06 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063484+0000", 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9324000000000002e-05 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063553+0000", 2024-02-12T16:14:50.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0424e-05 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.063564+0000", 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.202597851 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266162+0000", 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1644999999999999e-05 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266173+0000", 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20278849400000001 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161559 2.11 2.fee6d871 (undecoded) ondisk+write+known_if_redirected e177)", 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:39.127183+0000", 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 551.82930994599997, 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.139012677, 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.977 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161559 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127183+0000", 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127183+0000", 2024-02-12T16:14:50.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127182+0000", 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0881999999999999e-05 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127223+0000", 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2815e-05 2024-02-12T16:14:50.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127236+0000", 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.203e-06 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127241+0000", 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0544999999999999e-05 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127282+0000", 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9989999999999999e-06 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.127287+0000", 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13890308100000001 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266190+0000", 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0669999999999997e-06 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266196+0000", 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.139012677 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161571 2.11 2.fd602f71 (undecoded) ondisk+write+known_if_redirected e177)", 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:39.135200+0000", 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 551.821292804, 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13102438599999999, 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:50.982 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161571 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135200+0000", 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135200+0000", 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:14:50.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135198+0000", 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6846000000000003e-05 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135275+0000", 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5345999999999998e-05 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135290+0000", 2024-02-12T16:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1540000000000002e-06 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135298+0000", 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4993999999999995e-05 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135363+0000", 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2360000000000004e-06 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.135371+0000", 2024-02-12T16:14:50.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13084341899999999 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266215+0000", 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7599999999999997e-06 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266225+0000", 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13102438599999999 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.986 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161584 2.11 2.89893031 (undecoded) ondisk+write+known_if_redirected e177)", 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:39.141967+0000", 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 551.81452644900003, 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.124273988, 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161584 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.141967+0000", 2024-02-12T16:14:50.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.141967+0000", 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.141965+0000", 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2654000000000002e-05 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.142007+0000", 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4072e-05 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.142022+0000", 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5690000000000001e-06 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.142028+0000", 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6814000000000001e-05 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.142095+0000", 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2609000000000001e-05 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.142108+0000", 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.124126695 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266234+0000", 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4180000000000002e-06 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:50.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266241+0000", 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.124273988 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161600 2.11 2.cf98ec71 (undecoded) ondisk+write+known_if_redirected e177)", 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:39.158453+0000", 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 551.79804019699998, 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.107813041, 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:14:50.991 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161600 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158453+0000", 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158453+0000", 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158451+0000", 2024-02-12T16:14:50.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108616 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158560+0000", 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0499000000000001e-05 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158570+0000", 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158575+0000", 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0621000000000002e-05 2024-02-12T16:14:50.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158626+0000", 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0420000000000002e-06 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.158631+0000", 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10762699000000001 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266258+0000", 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1529999999999997e-06 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.266266+0000", 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.107813041 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287215 3.a 3:55b1025c:::benchmark_data_smithi169_51390_object123431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.504457+0000", 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 136.45203623800001, 2024-02-12T16:14:50.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49660404400000002, 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287215 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504457+0000", 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504457+0000", 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504453+0000", 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2464e-05 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504465+0000", 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7924999999999999e-05 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:50.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504483+0000", 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2390000000000002e-06 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504492+0000", 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2698000000000002e-05 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504554+0000", 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9949999999999999e-06 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504561+0000", 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001445558 2024-02-12T16:14:50.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506007+0000", 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111267 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506118+0000", 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012511600000000001 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506243+0000", 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093499770000000006 2024-02-12T16:14:50.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515593+0000", 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48534981799999999 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.000943+0000", 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8232000000000007e-05 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.001011+0000", 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9607999999999997e-05 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.001061+0000", 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49660404400000002 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287218 3.a 3:5667ef8a:::benchmark_data_smithi169_51390_object123434:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.507798+0000", 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 136.44869460199999, 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49960972100000001, 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.001 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287218 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507798+0000", 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507798+0000", 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:51.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507796+0000", 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0628000000000001e-05 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507807+0000", 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0042e-05 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507817+0000", 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2530000000000004e-06 2024-02-12T16:14:51.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507822+0000", 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028043299999999998 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510627+0000", 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001171 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510744+0000", 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1235000000000003e-05 2024-02-12T16:14:51.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510835+0000", 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048881100000000002 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515723+0000", 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49154228500000002 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007265+0000", 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9278999999999998e-05 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007345+0000", 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3569000000000001e-05 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007408+0000", 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49960972100000001 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287232 3.a 3:5fcb2087:::benchmark_data_smithi169_51390_object123448:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:51.006 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.524268+0000", 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 136.43222548200001, 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48364025700000002, 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287232 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524268+0000", 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524268+0000", 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524265+0000", 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4519999999999997e-06 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524273+0000", 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0237999999999999e-05 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524284+0000", 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.524e-06 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524289+0000", 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5358999999999998e-05 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524324+0000", 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2391000000000005e-05 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524397+0000", 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7861000000000006e-05 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524475+0000", 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041166830000000003 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.528591+0000", 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47912998299999998 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007721+0000", 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8727999999999995e-05 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007810+0000", 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7752999999999997e-05 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007908+0000", 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48364025700000002 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.011 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287237 3.a 3:5246a6cc:::benchmark_data_smithi169_51390_object123453:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.531869+0000", 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 136.42462400900001, 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47620760400000001, 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287237 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531869+0000", 2024-02-12T16:14:51.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531869+0000", 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531867+0000", 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4760000000000001e-06 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531875+0000", 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5639999999999997e-06 2024-02-12T16:14:51.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531883+0000", 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0839999999999999e-06 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531888+0000", 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0645000000000001e-05 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531929+0000", 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2466999999999994e-05 2024-02-12T16:14:51.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.532021+0000", 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8151000000000003e-05 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.532119+0000", 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035702770000000002 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.535690+0000", 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.472285854 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007976+0000", 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8838000000000002e-05 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.008024+0000", 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2268999999999997e-05 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.008077+0000", 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47620760400000001 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.016 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:47446 4.a 4:5217d0cf:::benchmark_data_smithi169_60938_object47445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:11.027017+0000", 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.929476516999998, 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.114610826, 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 47446 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027017+0000", 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027017+0000", 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027015+0000", 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1649000000000006e-05 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027076+0000", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1199000000000001e-05 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027087+0000", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1940000000000001e-06 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027095+0000", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3074999999999997e-05 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027138+0000", 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012282999999999999 2024-02-12T16:14:51.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027260+0000", 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000126012 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.027386+0000", 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001570725 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.028957+0000", 2024-02-12T16:14:51.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.112576705 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.141534+0000", 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2936000000000003e-05 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.141587+0000", 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0540000000000001e-05 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.141627+0000", 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.114610826 2024-02-12T16:14:51.021 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:47452 4.a 4:5f5c39c9:::benchmark_data_smithi169_60938_object47451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:11.029437+0000", 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.927056114000003, 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.112653344, 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 47452 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.022 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029437+0000", 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029437+0000", 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029434+0000", 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000130567 2024-02-12T16:14:51.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029565+0000", 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1371e-05 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029576+0000", 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1130000000000003e-06 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029583+0000", 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2092999999999998e-05 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029625+0000", 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100806 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029726+0000", 2024-02-12T16:14:51.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105185 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.029831+0000", 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014514879999999999 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031283+0000", 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.110493408 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.141776+0000", 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016605 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.141942+0000", 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014798900000000001 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.142090+0000", 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.112653344 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.027 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:47456 4.a 4:5364572b:::benchmark_data_smithi169_60938_object47455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:11.031085+0000", 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.925408484999998, 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11137124299999999, 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.028 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 47456 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031085+0000", 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031085+0000", 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031083+0000", 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084725600000000001 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031930+0000", 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2721000000000001e-05 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031943+0000", 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5540000000000003e-06 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031949+0000", 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3242999999999999e-05 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.031992+0000", 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011629900000000001 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.032109+0000", 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021738399999999999 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.032326+0000", 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001587792 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.033914+0000", 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.108339251 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.142253+0000", 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129636 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.142383+0000", 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3022000000000007e-05 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:11.142456+0000", 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11137124299999999 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:67494 4.a 4:5f5ff3e7:::benchmark_data_smithi169_60938_object67493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:18.496666+0000", 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.459827463000003, 2024-02-12T16:14:51.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11417071299999999, 2024-02-12T16:14:51.034 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.034 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.034 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 67494 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496666+0000", 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496666+0000", 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496664+0000", 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3222e-05 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496717+0000", 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0835e-05 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496728+0000", 2024-02-12T16:14:51.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2490000000000004e-06 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496735+0000", 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2251999999999999e-05 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496807+0000", 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4464999999999998e-05 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.496902+0000", 2024-02-12T16:14:51.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103472 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.497005+0000", 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041274509999999999 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.501133+0000", 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.109546191 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.610679+0000", 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1601999999999995e-05 2024-02-12T16:14:51.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.610771+0000", 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5797999999999996e-05 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.610836+0000", 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11417071299999999 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.039 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:67499 4.a 4:5c75b864:::benchmark_data_smithi169_60938_object67498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:18.497122+0000", 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.459371091000001, 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.113956902, 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 67499 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.497122+0000", 2024-02-12T16:14:51.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.497122+0000", 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.497120+0000", 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000177376 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.497297+0000", 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0428000000000001e-05 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.497308+0000", 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7600000000000002e-06 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.497313+0000", 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018554660000000001 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499168+0000", 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014828000000000001 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499316+0000", 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113206 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499429+0000", 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036871479999999999 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.503117+0000", 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10780928300000001 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.610926+0000", 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9147999999999994e-05 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.611015+0000", 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3862000000000006e-05 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.611079+0000", 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.113956902 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:67506 4.a 4:5410c182:::benchmark_data_smithi169_60938_object67505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:18.499544+0000", 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.456948664000002, 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11172645, 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.045 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 67506 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499544+0000", 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499544+0000", 2024-02-12T16:14:51.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499543+0000", 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013067299999999999 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499674+0000", 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5381999999999999e-05 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499689+0000", 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3380000000000001e-06 2024-02-12T16:14:51.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.499695+0000", 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001606012 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.501301+0000", 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4560000000000004e-05 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.501386+0000", 2024-02-12T16:14:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101508 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.501487+0000", 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018322569999999999 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.503320+0000", 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10782594400000001 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.611146+0000", 2024-02-12T16:14:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9744000000000002e-05 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.611215+0000", 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5538999999999998e-05 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:18.611271+0000", 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11172645 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:68522 4.a 4:5e1c6883:::benchmark_data_smithi169_60938_object68521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e341)", 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:19.018487+0000", 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 31.938005786000002, 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.1043794, 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 68522 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.018487+0000", 2024-02-12T16:14:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.018487+0000", 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.018486+0000", 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0393000000000002e-05 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.018557+0000", 2024-02-12T16:14:51.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6539999999999999e-06 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.018564+0000", 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8180000000000003e-06 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.018569+0000", 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001477812 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.020047+0000", 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013868800000000001 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.020185+0000", 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105512 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.020291+0000", 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001796246 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.022087+0000", 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10067743799999999 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.122765+0000", 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5811999999999999e-05 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.122830+0000", 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6211999999999998e-05 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:19.122867+0000", 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.1043794 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:84587 4.5 4:a4899310:::benchmark_data_smithi169_60938_object84586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:26.276967+0000", 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.679525782999999, 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.137213996, 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 84587 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.276967+0000", 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.276967+0000", 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:14:51.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.276966+0000", 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2454999999999997e-05 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277028+0000", 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1540000000000002e-06 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277036+0000", 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8520000000000003e-06 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277041+0000", 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5503e-05 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277067+0000", 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3953000000000002e-05 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277141+0000", 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9172999999999995e-05 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277230+0000", 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13590312399999999 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.413133+0000", 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000907087 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.414040+0000", 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2632000000000005e-05 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.414113+0000", 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8708999999999996e-05 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.414181+0000", 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.137213996 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:84595 4.a 4:54b5e1de:::benchmark_data_smithi169_60938_object84594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:26.280150+0000", 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.676342868999999, 2024-02-12T16:14:51.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.135721392, 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 84595 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.280150+0000", 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.280150+0000", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.280148+0000", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012619399999999999 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.280274+0000", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.123746642 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.404021+0000", 2024-02-12T16:14:51.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4187000000000001e-05 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.404045+0000", 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100022 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.404145+0000", 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033653699999999998 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.404481+0000", 2024-02-12T16:14:51.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000404165 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.404886+0000", 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086284069999999994 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.413514+0000", 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022644509999999998 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:14:51.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.415778+0000", 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2518000000000001e-05 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.415831+0000", 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.07e-05 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.415872+0000", 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.135721392 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:14:51.066 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:14:51.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:51.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:50 smithi169 bash[19604]: cluster 2024-02-12T16:14:49.061245+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.e scrub starts 2024-02-12T16:14:51.067 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:50 smithi169 bash[19604]: cluster 2024-02-12T16:14:49.709499+0000 mgr.y (mgr.14146) 884 : cluster [DBG] pgmap v1139: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-12T16:14:51.074 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:50 smithi123 bash[19119]: cluster 2024-02-12T16:14:49.061245+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.e scrub starts 2024-02-12T16:14:51.074 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:50 smithi123 bash[19119]: cluster 2024-02-12T16:14:49.709499+0000 mgr.y (mgr.14146) 884 : cluster [DBG] pgmap v1139: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-12T16:14:51.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:51.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:51.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 125981 125965 148.531 94.0625 0.00536789 0.00673128 2024-02-12T16:14:51.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:50 smithi121 bash[18489]: cluster 2024-02-12T16:14:49.061245+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.e scrub starts 2024-02-12T16:14:51.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:50 smithi121 bash[18489]: cluster 2024-02-12T16:14:49.709499+0000 mgr.y (mgr.14146) 884 : cluster [DBG] pgmap v1139: 17 pgs: 17 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-12T16:14:51.200 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:51.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:51.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:51.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:51.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:51.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:51.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:51.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:51.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:51.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:51.612 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:51.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:51.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:51.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:51.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:51.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:51.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:14:51.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:51.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:51.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:52.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:52.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:52.149 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:52.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 127502 127486 147.541 95.0625 0.00989012 0.00677614 2024-02-12T16:14:52.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:52.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:52.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:52.412 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:52.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:52.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:52.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:52.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:52.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:52.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:52.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:52.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:52 smithi169 bash[19604]: cluster 2024-02-12T16:14:51.435187+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.e scrub ok 2024-02-12T16:14:52.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:52 smithi169 bash[19604]: cluster 2024-02-12T16:14:51.709939+0000 mgr.y (mgr.14146) 885 : cluster [DBG] pgmap v1140: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-02-12T16:14:52.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:14:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:52.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:14:53.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:53.074 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:53.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:53.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:52 smithi123 bash[19119]: cluster 2024-02-12T16:14:51.435187+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.e scrub ok 2024-02-12T16:14:53.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:52 smithi123 bash[19119]: cluster 2024-02-12T16:14:51.709939+0000 mgr.y (mgr.14146) 885 : cluster [DBG] pgmap v1140: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-02-12T16:14:53.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 129041 129025 146.607 96.1875 0.00507349 0.00681936 2024-02-12T16:14:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:52 smithi121 bash[18489]: cluster 2024-02-12T16:14:51.435187+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.e scrub ok 2024-02-12T16:14:53.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:52 smithi121 bash[18489]: cluster 2024-02-12T16:14:51.709939+0000 mgr.y (mgr.14146) 885 : cluster [DBG] pgmap v1140: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-02-12T16:14:53.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:53.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:53.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:53.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:53.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:53.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:53.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:53.480 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:53.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:53.585 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:53.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:53.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:53.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:53.752 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:53.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:53.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:14:53.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:14:53.793 INFO:tasks.thrashosds.thrasher:Growing pool device_health_metrics 2024-02-12T16:14:53.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:14:53.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:53.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:53.893 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:53.894 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:53.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:53.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:54.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:54.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:54.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 130519 130503 145.639 92.375 0.00886059 0.00686435 2024-02-12T16:14:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:54.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:54.309 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:54.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:54.451 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:54.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:54.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:54.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:54.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:54.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:54 smithi169 bash[19604]: cluster 2024-02-12T16:14:53.710418+0000 mgr.y (mgr.14146) 886 : cluster [DBG] pgmap v1141: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:14:54.847 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:54.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:54.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:54 smithi121 bash[18489]: cluster 2024-02-12T16:14:53.710418+0000 mgr.y (mgr.14146) 886 : cluster [DBG] pgmap v1141: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:14:54.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:54.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:55.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:55.089 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:54 smithi123 bash[19119]: cluster 2024-02-12T16:14:53.710418+0000 mgr.y (mgr.14146) 886 : cluster [DBG] pgmap v1141: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:14:55.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:55.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:55.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:55.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 132047 132031 144.759 95.5 0.00880163 0.00690676 2024-02-12T16:14:55.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:55.233 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:55.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:55.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:55.372 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:55.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:55.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:55.501 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:55.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:55.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:55.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:55.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:55.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:55.760 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:55.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:55.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:55.871 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:55 smithi121 bash[18489]: cluster 2024-02-12T16:14:54.640558+0000 osd.9 (osd.9) 36 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:55.871 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:55 smithi121 bash[18489]: cluster 2024-02-12T16:14:54.640614+0000 osd.9 (osd.9) 37 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:55.871 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:55 smithi121 bash[18489]: cluster 2024-02-12T16:14:54.814416+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e354: 12 total, 12 up, 12 in 2024-02-12T16:14:55.871 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:55 smithi121 bash[18489]: cluster 2024-02-12T16:14:55.383286+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e355: 12 total, 12 up, 12 in 2024-02-12T16:14:55.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:55.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:55.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:56.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:56.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:56.172 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 16 133465 133449 143.791 88.625 0.012059 0.00695276 2024-02-12T16:14:56.173 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:55 smithi123 bash[19119]: cluster 2024-02-12T16:14:54.640558+0000 osd.9 (osd.9) 36 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:56.173 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:55 smithi123 bash[19119]: cluster 2024-02-12T16:14:54.640614+0000 osd.9 (osd.9) 37 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:56.173 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:55 smithi123 bash[19119]: cluster 2024-02-12T16:14:54.814416+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e354: 12 total, 12 up, 12 in 2024-02-12T16:14:56.173 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:55 smithi123 bash[19119]: cluster 2024-02-12T16:14:55.383286+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e355: 12 total, 12 up, 12 in 2024-02-12T16:14:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:56.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:55 smithi169 bash[19604]: cluster 2024-02-12T16:14:54.640558+0000 osd.9 (osd.9) 36 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:14:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:55 smithi169 bash[19604]: cluster 2024-02-12T16:14:54.640614+0000 osd.9 (osd.9) 37 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:14:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:55 smithi169 bash[19604]: cluster 2024-02-12T16:14:54.814416+0000 mon.a (mon.0) 1373 : cluster [DBG] osdmap e354: 12 total, 12 up, 12 in 2024-02-12T16:14:56.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:55 smithi169 bash[19604]: cluster 2024-02-12T16:14:55.383286+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e355: 12 total, 12 up, 12 in 2024-02-12T16:14:56.303 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:56.329 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:56.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:56.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:56.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:56.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:56.680 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:14:56.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:56.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:56.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:56.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:56.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:56.959 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:56 smithi123 bash[19119]: cluster 2024-02-12T16:14:55.576840+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:14:56.959 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:56 smithi123 bash[19119]: cluster 2024-02-12T16:14:55.710869+0000 mgr.y (mgr.14146) 887 : cluster [DBG] pgmap v1144: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:14:56.978 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:14:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:56.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:14:57.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:57.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:56 smithi169 bash[19604]: cluster 2024-02-12T16:14:55.576840+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:14:57.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:56 smithi169 bash[19604]: cluster 2024-02-12T16:14:55.710869+0000 mgr.y (mgr.14146) 887 : cluster [DBG] pgmap v1144: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:14:57.114 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:57.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:57.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 134749 134733 142.714 80.25 0.0036957 0.00700374 2024-02-12T16:14:57.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:56 smithi121 bash[18489]: cluster 2024-02-12T16:14:55.576840+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:14:57.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:56 smithi121 bash[18489]: cluster 2024-02-12T16:14:55.710869+0000 mgr.y (mgr.14146) 887 : cluster [DBG] pgmap v1144: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:14:57.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:57.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:57.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:57.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:57.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:57.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:57.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:57.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:14:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:57.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:14:57.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:57.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:14:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:57.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:14:57.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:57.779 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:57.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:57.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:57.914 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:57.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:58.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:58.043 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:58.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:58.160 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:14:58.158421 min lat: 0.0010598 max lat: 0.735335 avg lat: 0.00705526 2024-02-12T16:14:58.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:14:58.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 136062 136046 141.703 82.0625 0.00540924 0.00705526 2024-02-12T16:14:58.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:58.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:58.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:14:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:14:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:14:58.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:58.451 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:58.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:58.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:58.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:14:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:14:58.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:58.829 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:58 smithi123 bash[19119]: cluster 2024-02-12T16:14:57.711335+0000 mgr.y (mgr.14146) 888 : cluster [DBG] pgmap v1145: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:14:58.851 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:58.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:58.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:14:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:58.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:14:59.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:59.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:58 smithi169 bash[19604]: cluster 2024-02-12T16:14:57.711335+0000 mgr.y (mgr.14146) 888 : cluster [DBG] pgmap v1145: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:14:59.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:59.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:14:59.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 137396 137380 140.747 83.375 0.007972 0.00710309 2024-02-12T16:14:59.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:59.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:58 smithi121 bash[18489]: cluster 2024-02-12T16:14:57.711335+0000 mgr.y (mgr.14146) 888 : cluster [DBG] pgmap v1145: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:14:59.243 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:14:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:59.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:14:59.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:59.371 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:14:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:59.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:14:59.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:59.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:59.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:59.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:59.647 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:14:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:59.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:14:59.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:59.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:14:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:59.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:14:59.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:59.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:14:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:14:59.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:00.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:00.046 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:00.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:59 smithi121 bash[18489]: cluster 2024-02-12T16:14:58.710812+0000 osd.11 (osd.11) 43 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:15:00.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:14:59 smithi121 bash[18489]: cluster 2024-02-12T16:14:58.901598+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:15:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:59 smithi123 bash[19119]: cluster 2024-02-12T16:14:58.710812+0000 osd.11 (osd.11) 43 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:15:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:14:59 smithi123 bash[19119]: cluster 2024-02-12T16:14:58.901598+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:15:00.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:00.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 138748 138732 139.839 84.5 0.0160471 0.00714928 2024-02-12T16:15:00.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:00.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:59 smithi169 bash[19604]: cluster 2024-02-12T16:14:58.710812+0000 osd.11 (osd.11) 43 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:15:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:14:59 smithi169 bash[19604]: cluster 2024-02-12T16:14:58.901598+0000 osd.1 (osd.1) 50 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:15:00.312 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:00.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:00.451 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:00.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:00.648 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:15:00.680 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:15:00.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:00.713 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:15:00.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:00.739 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:15:00.769 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:15:00.798 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:15:00.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:00.830 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:15:00.858 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:15:00.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:00.886 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:15:00.920 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:15:00.946 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:15:00.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:00.973 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: cluster 2024-02-12T16:14:59.254502+0000 osd.11 (osd.11) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:00.973 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: cluster 2024-02-12T16:14:59.254546+0000 osd.11 (osd.11) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:00.973 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: cluster 2024-02-12T16:14:59.438393+0000 osd.9 (osd.9) 38 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:15:00.974 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: cluster 2024-02-12T16:14:59.715250+0000 mgr.y (mgr.14146) 889 : cluster [DBG] pgmap v1146: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-12T16:15:00.974 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: cluster 2024-02-12T16:14:59.832292+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e356: 12 total, 12 up, 12 in 2024-02-12T16:15:00.974 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: cluster 2024-02-12T16:15:00.388428+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e357: 12 total, 12 up, 12 in 2024-02-12T16:15:00.974 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: audit 2024-02-12T16:15:00.618365+0000 mon.a (mon.0) 1377 : audit [DBG] from='client.? v1:172.21.15.121:0/2273852854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:00.974 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:00 smithi121 bash[18489]: audit 2024-02-12T16:15:00.661866+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? v1:172.21.15.121:0/1334262097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:15:00.980 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:15:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:15:00.994 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:00.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:01.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:01.107 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: cluster 2024-02-12T16:14:59.254502+0000 osd.11 (osd.11) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:01.107 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: cluster 2024-02-12T16:14:59.254546+0000 osd.11 (osd.11) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:01.107 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: cluster 2024-02-12T16:14:59.438393+0000 osd.9 (osd.9) 38 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:15:01.107 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: cluster 2024-02-12T16:14:59.715250+0000 mgr.y (mgr.14146) 889 : cluster [DBG] pgmap v1146: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-12T16:15:01.107 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: cluster 2024-02-12T16:14:59.832292+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e356: 12 total, 12 up, 12 in 2024-02-12T16:15:01.107 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: cluster 2024-02-12T16:15:00.388428+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e357: 12 total, 12 up, 12 in 2024-02-12T16:15:01.108 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: audit 2024-02-12T16:15:00.618365+0000 mon.a (mon.0) 1377 : audit [DBG] from='client.? v1:172.21.15.121:0/2273852854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:01.108 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:00 smithi169 bash[19604]: audit 2024-02-12T16:15:00.661866+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? v1:172.21.15.121:0/1334262097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:15:01.110 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:15:01.110 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:15:01.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:01.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: cluster 2024-02-12T16:14:59.254502+0000 osd.11 (osd.11) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: cluster 2024-02-12T16:14:59.254546+0000 osd.11 (osd.11) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: cluster 2024-02-12T16:14:59.438393+0000 osd.9 (osd.9) 38 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: cluster 2024-02-12T16:14:59.715250+0000 mgr.y (mgr.14146) 889 : cluster [DBG] pgmap v1146: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: cluster 2024-02-12T16:14:59.832292+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e356: 12 total, 12 up, 12 in 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: cluster 2024-02-12T16:15:00.388428+0000 mon.a (mon.0) 1376 : cluster [DBG] osdmap e357: 12 total, 12 up, 12 in 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: audit 2024-02-12T16:15:00.618365+0000 mon.a (mon.0) 1377 : audit [DBG] from='client.? v1:172.21.15.121:0/2273852854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:00 smithi123 bash[19119]: audit 2024-02-12T16:15:00.661866+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? v1:172.21.15.121:0/1334262097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:15:01.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 16 139938 139922 138.8 74.375 0.0150563 0.00720335 2024-02-12T16:15:01.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:01.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:01.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:01.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:01.403 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:01.404 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:15:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:01.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:01.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:01.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:01.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:01.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:01.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:01.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:01.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:01.815 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:01.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:01.838 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:01 smithi121 bash[18489]: cluster 2024-02-12T16:15:00.186634+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.a scrub starts 2024-02-12T16:15:01.838 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:01 smithi121 bash[18489]: cluster 2024-02-12T16:15:01.391953+0000 mon.a (mon.0) 1379 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:01.838 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:01 smithi121 bash[18489]: audit 2024-02-12T16:15:01.395918+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? v1:172.21.15.121:0/1334262097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:15:01.838 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:01 smithi121 bash[18489]: cluster 2024-02-12T16:15:01.395978+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e358: 12 total, 12 up, 12 in 2024-02-12T16:15:01.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:01.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:01.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:02.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:01 smithi169 bash[19604]: cluster 2024-02-12T16:15:00.186634+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.a scrub starts 2024-02-12T16:15:02.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:01 smithi169 bash[19604]: cluster 2024-02-12T16:15:01.391953+0000 mon.a (mon.0) 1379 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:02.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:01 smithi169 bash[19604]: audit 2024-02-12T16:15:01.395918+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? v1:172.21.15.121:0/1334262097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:15:02.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:01 smithi169 bash[19604]: cluster 2024-02-12T16:15:01.395978+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e358: 12 total, 12 up, 12 in 2024-02-12T16:15:02.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:02.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:01 smithi123 bash[19119]: cluster 2024-02-12T16:15:00.186634+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.a scrub starts 2024-02-12T16:15:02.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:01 smithi123 bash[19119]: cluster 2024-02-12T16:15:01.391953+0000 mon.a (mon.0) 1379 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:02.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:01 smithi123 bash[19119]: audit 2024-02-12T16:15:01.395918+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? v1:172.21.15.121:0/1334262097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:15:02.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:01 smithi123 bash[19119]: cluster 2024-02-12T16:15:01.395978+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e358: 12 total, 12 up, 12 in 2024-02-12T16:15:02.063 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:15:02.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:02.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:02.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 141292 141276 137.953 84.625 0.0165596 0.00724723 2024-02-12T16:15:02.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:02.209 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:02.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:02.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:02.347 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:02.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:02.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:02.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:02.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:02.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:02.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:02.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:02.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:02.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:02.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:02.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:02.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:02.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:02.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:03.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:03.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:03.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:02 smithi123 bash[19119]: cluster 2024-02-12T16:15:01.715629+0000 mgr.y (mgr.14146) 890 : cluster [DBG] pgmap v1150: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-02-12T16:15:03.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:02 smithi121 bash[18489]: cluster 2024-02-12T16:15:01.715629+0000 mgr.y (mgr.14146) 890 : cluster [DBG] pgmap v1150: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-02-12T16:15:03.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:03.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 142393 142377 136.89 68.8125 0.00710366 0.00730358 2024-02-12T16:15:03.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:03.261 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:03.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:02 smithi169 bash[19604]: cluster 2024-02-12T16:15:01.715629+0000 mgr.y (mgr.14146) 890 : cluster [DBG] pgmap v1150: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-02-12T16:15:03.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:03.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:03.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:03.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:03.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:03.688 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:03.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:03.816 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.575815+0000 mon.a (mon.0) 1382 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2723834333,config=2903391811,health=1637288787,kv=2039616995}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.576483+0000 mon.a (mon.0) 1383 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1194262349,logm=3719261711}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.577216+0000 mon.a (mon.0) 1384 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523030865}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.577923+0000 mon.a (mon.0) 1385 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1691322460}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.578580+0000 mon.a (mon.0) 1386 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698103351}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.579269+0000 mon.a (mon.0) 1387 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577732575}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.581808+0000 mon.a (mon.0) 1388 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1998685161}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.585440+0000 mon.a (mon.0) 1389 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2720487875}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.589146+0000 mon.a (mon.0) 1390 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125985662}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.595140+0000 mon.a (mon.0) 1391 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2687567935}) 2024-02-12T16:15:03.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.598968+0000 mon.a (mon.0) 1392 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2254891472}) 2024-02-12T16:15:03.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.601565+0000 mon.a (mon.0) 1393 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=20,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=40} crc {logm=993724039,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=507872979,mon_config_key=100881130}) 2024-02-12T16:15:03.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.602428+0000 mon.a (mon.0) 1394 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3708646246}) 2024-02-12T16:15:03.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.606563+0000 mon.a (mon.0) 1395 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2457933767}) 2024-02-12T16:15:03.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.610271+0000 mon.a (mon.0) 1396 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=6,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=76} crc {mon_config_key=1044455694,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=2624318390,osdmap=3365454856}) 2024-02-12T16:15:03.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:03 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.623152+0000 mon.a (mon.0) 1397 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=76} crc {osdmap=2567304947}) 2024-02-12T16:15:03.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:15:03.947 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:15:03.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:03.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:04.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:04.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.575815+0000 mon.a (mon.0) 1382 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2723834333,config=2903391811,health=1637288787,kv=2039616995}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.576483+0000 mon.a (mon.0) 1383 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1194262349,logm=3719261711}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.577216+0000 mon.a (mon.0) 1384 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523030865}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.577923+0000 mon.a (mon.0) 1385 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1691322460}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.578580+0000 mon.a (mon.0) 1386 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698103351}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.579269+0000 mon.a (mon.0) 1387 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577732575}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.581808+0000 mon.a (mon.0) 1388 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1998685161}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.585440+0000 mon.a (mon.0) 1389 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2720487875}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.589146+0000 mon.a (mon.0) 1390 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125985662}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.595140+0000 mon.a (mon.0) 1391 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2687567935}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.598968+0000 mon.a (mon.0) 1392 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2254891472}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.601565+0000 mon.a (mon.0) 1393 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=20,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=40} crc {logm=993724039,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=507872979,mon_config_key=100881130}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.602428+0000 mon.a (mon.0) 1394 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3708646246}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.606563+0000 mon.a (mon.0) 1395 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2457933767}) 2024-02-12T16:15:04.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.610271+0000 mon.a (mon.0) 1396 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=6,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=76} crc {mon_config_key=1044455694,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=2624318390,osdmap=3365454856}) 2024-02-12T16:15:04.063 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:03 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.623152+0000 mon.a (mon.0) 1397 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=76} crc {osdmap=2567304947}) 2024-02-12T16:15:04.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:04.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 143726 143710 136.078 83.3125 0.00958965 0.00734725 2024-02-12T16:15:04.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:04.191 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.575815+0000 mon.a (mon.0) 1382 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2723834333,config=2903391811,health=1637288787,kv=2039616995}) 2024-02-12T16:15:04.191 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.576483+0000 mon.a (mon.0) 1383 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1194262349,logm=3719261711}) 2024-02-12T16:15:04.191 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.577216+0000 mon.a (mon.0) 1384 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523030865}) 2024-02-12T16:15:04.191 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.577923+0000 mon.a (mon.0) 1385 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1691322460}) 2024-02-12T16:15:04.191 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.578580+0000 mon.a (mon.0) 1386 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1698103351}) 2024-02-12T16:15:04.191 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.579269+0000 mon.a (mon.0) 1387 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577732575}) 2024-02-12T16:15:04.191 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.581808+0000 mon.a (mon.0) 1388 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1998685161}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.585440+0000 mon.a (mon.0) 1389 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2720487875}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.589146+0000 mon.a (mon.0) 1390 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125985662}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.595140+0000 mon.a (mon.0) 1391 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2687567935}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.598968+0000 mon.a (mon.0) 1392 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2254891472}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.601565+0000 mon.a (mon.0) 1393 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=20,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=40} crc {logm=993724039,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=507872979,mon_config_key=100881130}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.602428+0000 mon.a (mon.0) 1394 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3708646246}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.606563+0000 mon.a (mon.0) 1395 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2457933767}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.610271+0000 mon.a (mon.0) 1396 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=6,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=76} crc {mon_config_key=1044455694,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=2624318390,osdmap=3365454856}) 2024-02-12T16:15:04.192 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:03 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.623152+0000 mon.a (mon.0) 1397 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=76} crc {osdmap=2567304947}) 2024-02-12T16:15:04.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:04.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:04.337 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:04.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:04.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:04.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:04.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:04.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:04.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:04.847 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:04 smithi123 bash[19119]: cluster 2024-02-12T16:15:03.716026+0000 mgr.y (mgr.14146) 891 : cluster [DBG] pgmap v1151: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-12T16:15:04.847 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:04 smithi123 bash[19119]: audit 2024-02-12T16:15:03.916245+0000 mgr.y (mgr.14146) 892 : audit [DBG] from='client.15498 v1:172.21.15.121:0/3902226235' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:15:04.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:04.912 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1151,"stamp":"2024-02-12T16:15:03.715789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9112518656,"num_objects":139046,"num_object_clones":0,"num_object_copies":278092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139046,"num_write_kb":8898944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139046,"ondisk_log_size":139046,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":21523948,"kb_used_data":17871244,"kb_used_omap":0,"kb_used_meta":3652608,"kb_avail":1103417876,"statfs":{"total":1151940427776,"available":1129899905024,"internally_reserved":0,"allocated":18300153856,"data_stored":18294181346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3740270592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,3,8,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":429824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004690"},"pg_stats":[{"pgid":"4.f","version":"358'8725","reported_seq":8756,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-12T16:15:01.862815+0000","last_change":"2024-02-12T16:13:45.788790+0000","last_active":"2024-02-12T16:15:01.862815+0000","last_peered":"2024-02-12T16:15:01.862815+0000","last_clean":"2024-02-12T16:15:01.862815+0000","last_became_active":"2024-02-12T16:13:45.788522+0000","last_became_peered":"2024-02-12T16:13:45.788522+0000","last_unstale":"2024-02-12T16:15:01.862815+0000","last_undegraded":"2024-02-12T16:15:01.862815+0000","last_fullsized":"2024-02-12T16:15:01.862815+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8725,"ondisk_log_size":8725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":571801600,"num_objects":8725,"num_object_clones":0,"num_object_copies":17450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8725,"num_write_kb":558400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"356'8658","reported_seq":8692,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.957173+0000","last_change":"2024-02-12T16:14:51.435279+0000","last_active":"2024-02-12T16:14:59.957173+0000","last_peered":"2024-02-12T16:14:59.957173+0000","last_clean":"2024-02-12T16:14:59.957173+0000","last_became_active":"2024-02-12T16:13:45.970126+0000","last_became_peered":"2024-02-12T16:13:45.970126+0000","last_unstale":"2024-02-12T16:14:59.957173+0000","last_undegraded":"2024-02-12T16:14:59.957173+0000","last_fullsized":"2024-02-12T16:14:59.957173+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'7891","last_scrub_stamp":"2024-02-12T16:14:51.435192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:14:51.435192+0000","log_size":8658,"ondisk_log_size":8658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":567410688,"num_objects":8658,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8658,"num_write_kb":554112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"356'8639","reported_seq":8667,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.944205+0000","last_change":"2024-02-12T16:13:45.794115+0000","last_active":"2024-02-12T16:14:59.944205+0000","last_peered":"2024-02-12T16:14:59.944205+0000","last_clean":"2024-02-12T16:14:59.944205+0000","last_became_active":"2024-02-12T16:13:45.793796+0000","last_became_peered":"2024-02-12T16:13:45.793796+0000","last_unstale":"2024-02-12T16:14:59.944205+0000","last_undegraded":"2024-02-12T16:14:59.944205+0000","last_fullsized":"2024-02-12T16:14:59.944205+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8639,"ondisk_log_size":8639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566165504,"num_objects":8639,"num_object_clones":0,"num_object_copies":17278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8639,"num_write_kb":552896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"356'8656","reported_seq":8685,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.951311+0000","last_change":"2024-02-12T16:13:45.791824+0000","last_active":"2024-02-12T16:14:59.951311+0000","last_peered":"2024-02-12T16:14:59.951311+0000","last_clean":"2024-02-12T16:14:59.951311+0000","last_became_active":"2024-02-12T16:13:45.791545+0000","last_became_peered":"2024-02-12T16:13:45.791545+0000","last_unstale":"2024-02-12T16:14:59.951311+0000","last_undegraded":"2024-02-12T16:14:59.951311+0000","last_fullsized":"2024-02-12T16:14:59.951311+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8656,"ondisk_log_size":8656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":567279616,"num_objects":8656,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8656,"num_write_kb":553984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"357'8774","reported_seq":8804,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-12T16:15:00.421531+0000","last_change":"2024-02-12T16:13:45.790724+0000","last_active":"2024-02-12T16:15:00.421531+0000","last_peered":"2024-02-12T16:15:00.421531+0000","last_clean":"2024-02-12T16:15:00.421531+0000","last_became_active":"2024-02-12T16:13:45.790415+0000","last_became_peered":"2024-02-12T16:13:45.790415+0000","last_unstale":"2024-02-12T16:15:00.421531+0000","last_undegraded":"2024-02-12T16:15:00.421531+0000","last_fullsized":"2024-02-12T16:15:00.421531+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8774,"ondisk_log_size":8774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575012864,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8774,"num_write_kb":561536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"355'8610","reported_seq":8642,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-12T16:14:58.979573+0000","last_change":"2024-02-12T16:14:57.227617+0000","last_active":"2024-02-12T16:14:58.979573+0000","last_peered":"2024-02-12T16:14:58.979573+0000","last_clean":"2024-02-12T16:14:58.979573+0000","last_became_active":"2024-02-12T16:13:45.789678+0000","last_became_peered":"2024-02-12T16:13:45.789678+0000","last_unstale":"2024-02-12T16:14:58.979573+0000","last_undegraded":"2024-02-12T16:14:58.979573+0000","last_fullsized":"2024-02-12T16:14:58.979573+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8610,"ondisk_log_size":8610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":564264960,"num_objects":8610,"num_object_clones":0,"num_object_copies":17220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8610,"num_write_kb":551040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"355'8702","reported_seq":8731,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.843981+0000","last_change":"2024-02-12T16:13:45.795090+0000","last_active":"2024-02-12T16:14:59.843981+0000","last_peered":"2024-02-12T16:14:59.843981+0000","last_clean":"2024-02-12T16:14:59.843981+0000","last_became_active":"2024-02-12T16:13:45.794679+0000","last_became_peered":"2024-02-12T16:13:45.794679+0000","last_unstale":"2024-02-12T16:14:59.843981+0000","last_undegraded":"2024-02-12T16:14:59.843981+0000","last_fullsized":"2024-02-12T16:14:59.843981+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8702,"ondisk_log_size":8702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":570294272,"num_objects":8702,"num_object_clones":0,"num_object_copies":17404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8702,"num_write_kb":556928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"356'8607","reported_seq":8637,"reported_epoch":356,"state":"active+clean+scrubbing","last_fresh":"2024-02-12T16:14:59.840805+0000","last_change":"2024-02-12T16:14:58.710181+0000","last_active":"2024-02-12T16:14:59.840805+0000","last_peered":"2024-02-12T16:14:59.840805+0000","last_clean":"2024-02-12T16:14:59.840805+0000","last_became_active":"2024-02-12T16:13:45.793378+0000","last_became_peered":"2024-02-12T16:13:45.793378+0000","last_unstale":"2024-02-12T16:14:59.840805+0000","last_undegraded":"2024-02-12T16:14:59.840805+0000","last_fullsized":"2024-02-12T16:14:59.840805+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8607,"ondisk_log_size":8607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":564068352,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":550848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"358'8869","reported_seq":8901,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-12T16:15:01.884927+0000","last_change":"2024-02-12T16:14:58.901712+0000","last_active":"2024-02-12T16:15:01.884927+0000","last_peered":"2024-02-12T16:15:01.884927+0000","last_clean":"2024-02-12T16:15:01.884927+0000","last_became_active":"2024-02-12T16:13:45.966345+0000","last_became_peered":"2024-02-12T16:13:45.966345+0000","last_unstale":"2024-02-12T16:15:01.884927+0000","last_undegraded":"2024-02-12T16:15:01.884927+0000","last_fullsized":"2024-02-12T16:15:01.884927+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'8622","last_scrub_stamp":"2024-02-12T16:14:58.901608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:14:58.901608+0000","log_size":8869,"ondisk_log_size":8869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":581238784,"num_objects":8869,"num_object_clones":0,"num_object_copies":17738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8869,"num_write_kb":567616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"358'8892","reported_seq":8923,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-12T16:15:01.801643+0000","last_change":"2024-02-12T16:13:45.793559+0000","last_active":"2024-02-12T16:15:01.801643+0000","last_peered":"2024-02-12T16:15:01.801643+0000","last_clean":"2024-02-12T16:15:01.801643+0000","last_became_active":"2024-02-12T16:13:45.793246+0000","last_became_peered":"2024-02-12T16:13:45.793246+0000","last_unstale":"2024-02-12T16:15:01.801643+0000","last_undegraded":"2024-02-12T16:15:01.801643+0000","last_fullsized":"2024-02-12T16:15:01.801643+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8892,"ondisk_log_size":8892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":582746112,"num_objects":8892,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8892,"num_write_kb":569088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"358'8877","reported_seq":8908,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-12T16:15:01.893676+0000","last_change":"2024-02-12T16:13:45.786420+0000","last_active":"2024-02-12T16:15:01.893676+0000","last_peered":"2024-02-12T16:15:01.893676+0000","last_clean":"2024-02-12T16:15:01.893676+0000","last_became_active":"2024-02-12T16:13:45.786192+0000","last_became_peered":"2024-02-12T16:13:45.786192+0000","last_unstale":"2024-02-12T16:15:01.893676+0000","last_undegraded":"2024-02-12T16:15:01.893676+0000","last_fullsized":"2024-02-12T16:15:01.893676+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8877,"ondisk_log_size":8877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":581763072,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":568128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"356'8577","reported_seq":8605,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.851681+0000","last_change":"2024-02-12T16:13:45.795525+0000","last_active":"2024-02-12T16:14:59.851681+0000","last_peered":"2024-02-12T16:14:59.851681+0000","last_clean":"2024-02-12T16:14:59.851681+0000","last_became_active":"2024-02-12T16:13:45.794673+0000","last_became_peered":"2024-02-12T16:13:45.794673+0000","last_unstale":"2024-02-12T16:14:59.851681+0000","last_undegraded":"2024-02-12T16:14:59.851681+0000","last_fullsized":"2024-02-12T16:14:59.851681+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8577,"ondisk_log_size":8577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":562102272,"num_objects":8577,"num_object_clones":0,"num_object_copies":17154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":548928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"358'8677","reported_seq":8711,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-12T16:15:01.801831+0000","last_change":"2024-02-12T16:15:01.405216+0000","last_active":"2024-02-12T16:15:01.801831+0000","last_peered":"2024-02-12T16:15:01.801831+0000","last_clean":"2024-02-12T16:15:01.801831+0000","last_became_active":"2024-02-12T16:13:45.969665+0000","last_became_peered":"2024-02-12T16:13:45.969665+0000","last_unstale":"2024-02-12T16:15:01.801831+0000","last_undegraded":"2024-02-12T16:15:01.801831+0000","last_fullsized":"2024-02-12T16:15:01.801831+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8677,"ondisk_log_size":8677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":568655872,"num_objects":8677,"num_object_clones":0,"num_object_copies":17354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8677,"num_write_kb":555328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"355'8578","reported_seq":8607,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.843771+0000","last_change":"2024-02-12T16:13:45.970010+0000","last_active":"2024-02-12T16:14:59.843771+0000","last_peered":"2024-02-12T16:14:59.843771+0000","last_clean":"2024-02-12T16:14:59.843771+0000","last_became_active":"2024-02-12T16:13:45.969700+0000","last_became_peered":"2024-02-12T16:13:45.969700+0000","last_unstale":"2024-02-12T16:14:59.843771+0000","last_undegraded":"2024-02-12T16:14:59.843771+0000","last_fullsized":"2024-02-12T16:14:59.843771+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8578,"ondisk_log_size":8578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":562167808,"num_objects":8578,"num_object_clones":0,"num_object_copies":17156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":548992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"355'8652","reported_seq":8680,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.744869+0000","last_change":"2024-02-12T16:13:45.788916+0000","last_active":"2024-02-12T16:14:59.744869+0000","last_peered":"2024-02-12T16:14:59.744869+0000","last_clean":"2024-02-12T16:14:59.744869+0000","last_became_active":"2024-02-12T16:13:45.788482+0000","last_became_peered":"2024-02-12T16:13:45.788482+0000","last_unstale":"2024-02-12T16:14:59.744869+0000","last_undegraded":"2024-02-12T16:14:59.744869+0000","last_fullsized":"2024-02-12T16:14:59.744869+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8652,"ondisk_log_size":8652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":567017472,"num_objects":8652,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8652,"num_write_kb":553728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"355'8553","reported_seq":8578,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-02-12T16:14:59.012165+0000","last_change":"2024-02-12T16:13:45.969991+0000","last_active":"2024-02-12T16:14:59.012165+0000","last_peered":"2024-02-12T16:14:59.012165+0000","last_clean":"2024-02-12T16:14:59.012165+0000","last_became_active":"2024-02-12T16:13:45.969639+0000","last_became_peered":"2024-02-12T16:13:45.969639+0000","last_unstale":"2024-02-12T16:14:59.012165+0000","last_undegraded":"2024-02-12T16:14:59.012165+0000","last_fullsized":"2024-02-12T16:14:59.012165+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:13:44.758147+0000","log_size":8553,"ondisk_log_size":8553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":560529408,"num_objects":8553,"num_object_clones":0,"num_object_copies":17106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8553,"num_write_kb":547392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":389,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-12T16:15:00.407388+0000","last_change":"2024-02-12T16:13:43.045211+0000","last_active":"2024-02-12T16:15:00.407388+0000","last_peered":"2024-02-12T16:15:00.407388+0000","last_clean":"2024-02-12T16:15:00.407388+0000","last_became_active":"2024-02-12T15:58:25.841208+0000","last_became_peered":"2024-02-12T15:58:25.841208+0000","last_unstale":"2024-02-12T16:15:00.407388+0000","last_undegraded":"2024-02-12T16:15:00.407388+0000","last_fullsized":"2024-02-12T16:15:00.407388+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:13:43.044979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:13:43.044979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":9112518656,"num_objects":139046,"num_object_clones":0,"num_object_copies":278092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139046,"num_write_kb":8898944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18371772416,"data_stored":18371772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139046,"ondisk_log_size":139046,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958730,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2508224,"kb_used_data":2202808,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91236928,"statfs":{"total":95995035648,"available":93426614272,"internally_reserved":0,"allocated":2255675392,"data_stored":2255187493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.70999999999999996,"15min":0.82299999999999995},"min":{"1min":0.26800000000000002,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":1.554,"5min":3.6720000000000002,"15min":4.7850000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.35599999999999998,"5min":0.30599999999999999,"15min":0.25900000000000001},"max":{"1min":1.7170000000000001,"5min":4.992,"15min":4.992},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.85999999999999999,"15min":1.032},"min":{"1min":0.375,"5min":0.27900000000000003,"15min":0.25700000000000001},"max":{"1min":1.4470000000000001,"5min":4.851,"15min":6.7569999999999997},"last":0.376},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.95399999999999996,"15min":1.046},"min":{"1min":0.441,"5min":0.33600000000000002,"15min":0.27300000000000002},"max":{"1min":1.5649999999999999,"5min":5.0339999999999998,"15min":5.0339999999999998},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0980000000000001,"15min":1.1259999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.7109999999999999,"5min":5.274,"15min":5.274},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":0.96999999999999997,"15min":1.1160000000000001},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":3.7360000000000002,"5min":5.375,"15min":5.375},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":3.056,"15min":1.7549999999999999},"min":{"1min":0.61799999999999999,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":1.6679999999999999,"5min":106.301,"15min":106.301},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.252,"5min":3.121,"15min":1.8520000000000001},"min":{"1min":0.372,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.8610000000000002,"5min":106.437,"15min":106.437},"last":0.496}]},{"osd":4,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":4.702,"15min":2.3239999999999998},"min":{"1min":0.55900000000000005,"5min":0.39200000000000002,"15min":0.33100000000000002},"max":{"1min":2.052,"5min":368.48399999999998,"15min":368.48399999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.1279999999999999,"5min":4.7469999999999999,"15min":2.298},"min":{"1min":0.46400000000000002,"5min":0.37,"15min":0.36799999999999999},"max":{"1min":1.591,"5min":371.65100000000001,"15min":371.65100000000001},"last":0.53200000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":9.0960000000000001,"15min":3.8159999999999998},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.34399999999999997},"max":{"1min":1.752,"5min":407.07600000000002,"15min":407.07600000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.2130000000000001,"5min":9.0519999999999996,"15min":3.8679999999999999},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35399999999999998},"max":{"1min":1.9730000000000001,"5min":407.26299999999998,"15min":407.26299999999998},"last":0.624}]},{"osd":6,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.96599999999999997,"15min":1.1060000000000001},"min":{"1min":0.505,"5min":0.40899999999999997,"15min":0.34799999999999998},"max":{"1min":1.7170000000000001,"5min":3.722,"15min":4.4989999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.103,"15min":1.2150000000000001},"min":{"1min":0.54500000000000004,"5min":0.41099999999999998,"15min":0.40300000000000002},"max":{"1min":1.875,"5min":3.8039999999999998,"15min":3.8039999999999998},"last":0.66400000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.21,"15min":1.234},"min":{"1min":0.57199999999999995,"5min":0.435,"15min":0.39100000000000001},"max":{"1min":1.9359999999999999,"5min":5.6779999999999999,"15min":5.6779999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.369,"5min":1.2450000000000001,"15min":1.2849999999999999},"min":{"1min":0.56499999999999995,"5min":0.438,"15min":0.438},"max":{"1min":1.776,"5min":5.2510000000000003,"15min":5.2510000000000003},"last":0.71299999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.079,"15min":1.218},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.0760000000000001,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.1419999999999999,"15min":1.2589999999999999},"min":{"1min":0.49299999999999999,"5min":0.39000000000000001,"15min":0.375},"max":{"1min":2.0019999999999998,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.63500000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:14:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.0329999999999999,"15min":0.96499999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.0859999999999999,"5min":5.0039999999999996,"15min":5.0039999999999996},"last":10.743},{"interface":"front","average":{"1min":1.478,"5min":1.2350000000000001,"15min":1.0369999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.9079999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":10.698}]},{"osd":10,"last update":"Mon Feb 12 16:14:04 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.1359999999999999,"15min":1.2529999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.377},"max":{"1min":2.109,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.1910000000000001,"15min":1.3240000000000001},"min":{"1min":0.504,"5min":0.496,"15min":0.45200000000000001},"max":{"1min":2.3490000000000002,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":0.754}]}]},{"osd":10,"up_from":83,"seq":356482285780,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2004708,"kb_used_data":1703388,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91740444,"statfs":{"total":95995035648,"available":93942214656,"internally_reserved":0,"allocated":1744269312,"data_stored":1743763196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":1,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.212,"15min":1.0309999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":1.075,"5min":49.802999999999997,"15min":49.802999999999997},"last":3.8450000000000002},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.292,"15min":1.0860000000000001},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.28399999999999997},"max":{"1min":1.665,"5min":50.863,"15min":50.863},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.63700000000000001,"15min":0.83099999999999996},"min":{"1min":0.311,"5min":0.309,"15min":0.29299999999999998},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":4.8300000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.624,"15min":0.88400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.29699999999999999},"max":{"1min":1.772,"5min":1.772,"15min":7.9829999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.98099999999999998,"15min":1.012},"min":{"1min":0.32900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.7629999999999999,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.89100000000000001,"15min":0.94099999999999995},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.29599999999999999},"max":{"1min":1.2470000000000001,"5min":4.2149999999999999,"15min":5.4100000000000001},"last":6.4500000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":35.899999999999999,"5min":9.1899999999999995,"15min":3.7719999999999998},"min":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.34499999999999997},"max":{"1min":628.77099999999996,"5min":628.77099999999996,"15min":628.77099999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":35.537999999999997,"5min":9.0980000000000008,"15min":3.7290000000000001},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.34200000000000003},"max":{"1min":624.19299999999998,"5min":624.19299999999998,"15min":624.19299999999998},"last":0.51100000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":2.1720000000000002,"15min":3.5179999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.33700000000000002},"max":{"1min":1.3520000000000001,"5min":151.25800000000001,"15min":602.05399999999997},"last":4.1369999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":2.3370000000000002,"15min":3.641},"min":{"1min":0.435,"5min":0.435,"15min":0.33200000000000002},"max":{"1min":1.8049999999999999,"5min":151.416,"15min":598.875},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:14:24 2024","interfaces":[{"interface":"back","average":{"1min":35.582999999999998,"5min":7.7300000000000004,"15min":3.298},"min":{"1min":0.378,"5min":0.378,"15min":0.35399999999999998},"max":{"1min":626.26599999999996,"5min":626.26599999999996,"15min":626.26599999999996},"last":4.1740000000000004},{"interface":"front","average":{"1min":35.962000000000003,"5min":7.883,"15min":3.4380000000000002},"min":{"1min":0.55100000000000005,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":625.90700000000004,"5min":625.90700000000004,"15min":625.90700000000004},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.93500000000000005,"15min":3.2240000000000002},"min":{"1min":0.46400000000000002,"5min":0.41299999999999998,"15min":0.376},"max":{"1min":2.1240000000000001,"5min":3.8359999999999999,"15min":597.48299999999995},"last":3.9950000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.873,"15min":3.2109999999999999},"min":{"1min":0.48799999999999999,"5min":0.47199999999999998,"15min":0.41799999999999998},"max":{"1min":2.3660000000000001,"5min":3.7080000000000002,"15min":600.23099999999999},"last":4.0949999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":0.90900000000000003,"15min":1.1559999999999999},"min":{"1min":0.51900000000000002,"5min":0.37,"15min":0.33400000000000002},"max":{"1min":1.998,"5min":3.4279999999999999,"15min":7.2720000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.111,"5min":0.89300000000000002,"15min":1.1910000000000001},"min":{"1min":0.47799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.732,"5min":1.7350000000000001,"15min":7.3810000000000002},"last":0.85699999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.99099999999999999,"15min":1.216},"min":{"1min":0.66500000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.893,"5min":4.3369999999999997,"15min":4.7089999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.2529999999999999,"5min":0.995,"15min":1.218},"min":{"1min":0.68899999999999995,"5min":0.53200000000000003,"15min":0.40000000000000002},"max":{"1min":2.0270000000000001,"5min":4.3529999999999998,"15min":4.7789999999999999},"last":0.60499999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:15:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.5429999999999999,"15min":1.429},"min":{"1min":0.58799999999999997,"5min":0.51000000000000001,"15min":0.45500000000000002},"max":{"1min":3.2320000000000002,"5min":30.917999999999999,"15min":30.917999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.5109999999999999,"15min":1.407},"min":{"1min":0.59999999999999998,"5min":0.51700000000000002,"15min":0.48499999999999999},"max":{"1min":2.7789999999999999,"5min":30.742000000000001,"15min":30.742000000000001},"last":0.61899999999999999}]}]},{"osd":9,"up_from":74,"seq":317827580124,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1991780,"kb_used_data":1682268,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91753372,"statfs":{"total":95995035648,"available":93955452928,"internally_reserved":0,"allocated":1722642432,"data_stored":1722136336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56899999999999995,"15min":0.76200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":1.2130000000000001,"5min":3.702,"15min":10.788},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.94199999999999995,"15min":0.93500000000000005},"min":{"1min":0.34499999999999997,"5min":0.30499999999999999,"15min":0.27400000000000002},"max":{"1min":7.4429999999999996,"5min":7.4429999999999996,"15min":7.9630000000000001},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":0.83699999999999997,"15min":1.5549999999999999},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.25700000000000001},"max":{"1min":4.851,"5min":5.0999999999999996,"15min":175.33500000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.482,"5min":1.0609999999999999,"15min":1.5960000000000001},"min":{"1min":0.53300000000000003,"5min":0.34599999999999997,"15min":0.317},"max":{"1min":7.5019999999999998,"5min":7.5019999999999998,"15min":175.24299999999999},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":0.97599999999999998,"15min":1.054},"min":{"1min":0.34300000000000003,"5min":0.32200000000000001,"15min":0.27700000000000002},"max":{"1min":7.2999999999999998,"5min":7.2999999999999998,"15min":7.8140000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.75800000000000001,"15min":0.98499999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.28199999999999997},"max":{"1min":1.915,"5min":4.6859999999999999,"15min":4.7240000000000002},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.86599999999999999,"15min":1.0620000000000001},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.308},"max":{"1min":1.8400000000000001,"5min":4.6520000000000001,"15min":34.591000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.095,"15min":1.2330000000000001},"min":{"1min":0.51300000000000001,"5min":0.38100000000000001,"15min":0.35199999999999998},"max":{"1min":7.4729999999999999,"5min":7.4729999999999999,"15min":34.494},"last":4.4489999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.74299999999999999,"15min":1.0680000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.3879999999999999,"5min":3.75,"15min":25.552},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.77000000000000002,"15min":1.006},"min":{"1min":0.439,"5min":0.42699999999999999,"15min":0.35099999999999998},"max":{"1min":1.841,"5min":1.841,"15min":25.847999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.76400000000000001,"15min":3.3119999999999998},"min":{"1min":0.39900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.778,"5min":2.5230000000000001,"15min":617.95899999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.236,"5min":1.0509999999999999,"15min":3.4409999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.375},"max":{"1min":7.3739999999999997,"5min":7.3739999999999997,"15min":617.88099999999997},"last":0.5}]},{"osd":6,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":0.98399999999999999,"15min":1.1279999999999999},"min":{"1min":0.49199999999999999,"5min":0.434,"15min":0.40999999999999998},"max":{"1min":7.3360000000000003,"5min":7.3360000000000003,"15min":7.3360000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.88,"15min":1.0189999999999999},"min":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.377},"max":{"1min":3.1800000000000002,"5min":4.7910000000000004,"15min":4.7910000000000004},"last":0.39500000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.93300000000000005,"15min":4.6040000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.379},"max":{"1min":2.5499999999999998,"5min":4.3209999999999997,"15min":613.28200000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.1299999999999999,"15min":4.7069999999999999},"min":{"1min":0.61099999999999999,"5min":0.52500000000000002,"15min":0.45300000000000001},"max":{"1min":7.4089999999999998,"5min":7.4089999999999998,"15min":613.25199999999995},"last":0.58999999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.86799999999999999,"15min":1.157},"min":{"1min":0.54200000000000004,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.6579999999999999,"5min":3.7799999999999998,"15min":4.96},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.032,"15min":1.1259999999999999},"min":{"1min":0.56899999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":7.25,"5min":7.25,"15min":7.25},"last":0.52700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":0.97099999999999997,"15min":1.137},"min":{"1min":0.52200000000000002,"5min":0.47599999999999998,"15min":0.38400000000000001},"max":{"1min":1.5509999999999999,"5min":4.0709999999999997,"15min":5.5},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.89700000000000002,"15min":1.163},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.595,"5min":3.581,"15min":4.7619999999999996},"last":1.405}]}]},{"osd":8,"up_from":65,"seq":279172874467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":310500,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434652,"statfs":{"total":95995035648,"available":95677083648,"internally_reserved":0,"allocated":1011712,"data_stored":505616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.69899999999999995,"15min":0.80000000000000004},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.22700000000000001},"max":{"1min":1.026,"5min":4.827,"15min":5.383},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.74199999999999999,"15min":0.85699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29999999999999999,"15min":0.28299999999999997},"max":{"1min":1.6080000000000001,"5min":7.319,"15min":7.319},"last":0.317}]},{"osd":1,"last update":"Mon Feb 12 16:14:52 2024","interfaces":[{"interface":"back","average":{"1min":3.665,"5min":1.3400000000000001,"15min":1.6859999999999999},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":54.975999999999999,"5min":54.975999999999999,"15min":54.975999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":3.911,"5min":1.401,"15min":1.7869999999999999},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.30299999999999999},"max":{"1min":55.005000000000003,"5min":55.005000000000003,"15min":55.005000000000003},"last":0.441}]},{"osd":2,"last update":"Mon Feb 12 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":12.574,"5min":3.3340000000000001,"15min":1.696},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":193.929,"5min":193.929,"15min":193.929},"last":0.52300000000000002},{"interface":"front","average":{"1min":12.651,"5min":3.484,"15min":1.8380000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":193.82499999999999,"5min":193.82499999999999,"15min":193.82499999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:14:52 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.0289999999999999,"15min":1.0529999999999999},"min":{"1min":0.377,"5min":0.34799999999999998,"15min":0.28000000000000003},"max":{"1min":2.5030000000000001,"5min":10.4,"15min":10.4},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.93600000000000005,"15min":1.052},"min":{"1min":0.311,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":2.5209999999999999,"5min":10.413,"15min":10.413},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:14:52 2024","interfaces":[{"interface":"back","average":{"1min":12.050000000000001,"5min":3.032,"15min":1.7090000000000001},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.307},"max":{"1min":225.33099999999999,"5min":225.33099999999999,"15min":225.33099999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":11.903,"5min":2.948,"15min":1.7490000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":224.71000000000001,"5min":224.71000000000001,"15min":224.71000000000001},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:14:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.75700000000000001,"15min":0.96099999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":1.377,"5min":1.7749999999999999,"15min":13.961},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.94199999999999995,"15min":1.0940000000000001},"min":{"1min":0.47499999999999998,"5min":0.39200000000000002,"15min":0.29999999999999999},"max":{"1min":1.986,"5min":4.8339999999999996,"15min":13.103},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:14:52 2024","interfaces":[{"interface":"back","average":{"1min":11.678000000000001,"5min":3.0070000000000001,"15min":1.7370000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.36299999999999999},"max":{"1min":221.36699999999999,"5min":221.36699999999999,"15min":221.36699999999999},"last":0.433},{"interface":"front","average":{"1min":11.816000000000001,"5min":3.1469999999999998,"15min":1.8859999999999999},"min":{"1min":0.54300000000000004,"5min":0.40300000000000002,"15min":0.38300000000000001},"max":{"1min":221.203,"5min":221.203,"15min":221.203},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:14:52 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.034,"15min":1.2090000000000001},"min":{"1min":0.60999999999999999,"5min":0.47499999999999998,"15min":0.42099999999999999},"max":{"1min":2.093,"5min":4.617,"15min":13.242000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.93100000000000005,"15min":1.2130000000000001},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.40400000000000003},"max":{"1min":1.6910000000000001,"5min":4.6310000000000002,"15min":13.388},"last":0.52600000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:14:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.95399999999999996,"15min":1.1799999999999999},"min":{"1min":0.57099999999999995,"5min":0.40200000000000002,"15min":0.34699999999999998},"max":{"1min":2.3650000000000002,"5min":4.093,"15min":5.8449999999999998},"last":4.5099999999999998},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.92500000000000004,"15min":1.1499999999999999},"min":{"1min":0.51000000000000001,"5min":0.45300000000000001,"15min":0.36799999999999999},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":6.2000000000000002},"last":0.66500000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:14:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":0.79700000000000004,"15min":1.1399999999999999},"min":{"1min":0.45300000000000001,"5min":0.38500000000000001,"15min":0.36399999999999999},"max":{"1min":1.794,"5min":1.794,"15min":5.4189999999999996},"last":3.0190000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":0.84499999999999997,"15min":1.135},"min":{"1min":0.48699999999999999,"5min":0.39500000000000002,"15min":0.376},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":3.6779999999999999},"last":0.64900000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:14:20 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.0409999999999999,"15min":15.347},"min":{"1min":0.50700000000000001,"5min":0.441,"15min":0.441},"max":{"1min":3.645,"5min":4.8170000000000002,"15min":635.66399999999999},"last":2.0219999999999998},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.1160000000000001,"15min":15.202999999999999},"min":{"1min":0.52300000000000002,"5min":0.46400000000000002,"15min":0.44600000000000001},"max":{"1min":3.5070000000000001,"5min":4.9489999999999998,"15min":636.28099999999995},"last":2.0139999999999998}]}]},{"osd":7,"up_from":56,"seq":240518168812,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2000612,"kb_used_data":1699292,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91744540,"statfs":{"total":95995035648,"available":93946408960,"internally_reserved":0,"allocated":1740075008,"data_stored":1739568912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 15:57:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.90200000000000002,"15min":0.89000000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96599999999999997,"15min":0.96499999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83799999999999997,"15min":1.1080000000000001},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.49,"5min":5.6669999999999998,"15min":60.152000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.99399999999999999,"15min":1.2330000000000001},"min":{"1min":0.375,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.425,"5min":5.8479999999999999,"15min":62.020000000000003},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.754,"15min":1.179},"min":{"1min":0.34200000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.2350000000000001,"5min":6.1790000000000003,"15min":60.204999999999998},"last":0.373},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83899999999999997,"15min":1.1739999999999999},"min":{"1min":0.42299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.1990000000000001,"5min":5.8040000000000003,"15min":59.365000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.0700000000000001,"15min":1.0880000000000001},"min":{"1min":0.46500000000000002,"5min":0.28599999999999998,"15min":0.27900000000000003},"max":{"1min":1.359,"5min":6.2439999999999998,"15min":6.2439999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.056,"5min":1.1200000000000001,"15min":1.0780000000000001},"min":{"1min":0.50800000000000001,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":1.4870000000000001,"5min":8.2940000000000005,"15min":8.2940000000000005},"last":0.437}]},{"osd":4,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.99199999999999999,"15min":0.90900000000000003},"min":{"1min":0.30299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.367,"5min":6.048,"15min":6.2519999999999998},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.024,"15min":0.98199999999999998},"min":{"1min":0.40300000000000002,"5min":0.29899999999999999,"15min":0.28599999999999998},"max":{"1min":1.6180000000000001,"5min":6.0919999999999996,"15min":6.0919999999999996},"last":0.28899999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.0229999999999999,"15min":0.999},"min":{"1min":0.36199999999999999,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":2.0699999999999998,"5min":6.0739999999999998,"15min":6.0739999999999998},"last":0.998},{"interface":"front","average":{"1min":1.054,"5min":1.0269999999999999,"15min":1.018},"min":{"1min":0.35999999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.2050000000000001,"5min":6.274,"15min":6.274},"last":0.41699999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1759999999999999,"15min":1.1240000000000001},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.6619999999999999,"5min":6.0830000000000002,"15min":6.0830000000000002},"last":0.318},{"interface":"front","average":{"1min":1.198,"5min":1.1180000000000001,"15min":1.0820000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.7869999999999999,"5min":6.1349999999999998,"15min":6.1349999999999998},"last":0.48199999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:14:46 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0720000000000001,"15min":1.2090000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.2229999999999999,"5min":6.0250000000000004,"15min":6.0250000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.133,"15min":1.2410000000000001},"min":{"1min":0.45500000000000002,"5min":0.433,"15min":0.42999999999999999},"max":{"1min":9.2140000000000004,"5min":9.2140000000000004,"15min":9.2140000000000004},"last":0.51200000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:14:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.1659999999999999,"15min":2.3029999999999999},"min":{"1min":0.48499999999999999,"5min":0.46100000000000002,"15min":0.441},"max":{"1min":1.998,"5min":6.9539999999999997,"15min":278.54599999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.048,"5min":1.032,"15min":2.2280000000000002},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.38200000000000001},"max":{"1min":1.786,"5min":6.3479999999999999,"15min":281.09199999999998},"last":0.52300000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.0940000000000001,"15min":1.24},"min":{"1min":0.54800000000000004,"5min":0.49099999999999999,"15min":0.48599999999999999},"max":{"1min":2.2890000000000001,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.335,"5min":1.1479999999999999,"15min":1.282},"min":{"1min":0.55500000000000005,"5min":0.5,"15min":0.42099999999999999},"max":{"1min":1.9370000000000001,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":0.54200000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.615,"15min":2.399},"min":{"1min":0.51700000000000002,"5min":0.499,"15min":0.498},"max":{"1min":9.2560000000000002,"5min":31.542999999999999,"15min":280.83999999999997},"last":0.625},{"interface":"front","average":{"1min":1.046,"5min":1.661,"15min":2.4199999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":3.5139999999999998,"5min":31.988,"15min":278.63799999999998},"last":0.61599999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430452,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1387776,"kb_used_data":1090552,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92357376,"statfs":{"total":95995035648,"available":94573953024,"internally_reserved":0,"allocated":1116725248,"data_stored":1116237369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.72599999999999998,"15min":1.052},"min":{"1min":0.30299999999999999,"5min":0.28999999999999998,"15min":0.26200000000000001},"max":{"1min":3.173,"5min":4.4199999999999999,"15min":67.093000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":1.1000000000000001},"min":{"1min":0.318,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.554,"5min":4.4779999999999998,"15min":68.549000000000007},"last":0.309}]},{"osd":1,"last update":"Mon Feb 12 16:14:36 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.84799999999999998,"15min":1.1799999999999999},"min":{"1min":0.34699999999999998,"5min":0.34200000000000003,"15min":0.308},"max":{"1min":2.032,"5min":4.4900000000000002,"15min":66.936000000000007},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.81299999999999994,"15min":1.1599999999999999},"min":{"1min":0.33400000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.5310000000000001,"5min":4.9409999999999998,"15min":68.683000000000007},"last":0.433}]},{"osd":2,"last update":"Mon Feb 12 16:14:36 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.73599999999999999,"15min":1.4099999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.28899999999999998},"max":{"1min":3.226,"5min":3.722,"15min":68.462000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.749,"15min":1.4370000000000001},"min":{"1min":0.38600000000000001,"5min":0.373,"15min":0.34599999999999997},"max":{"1min":1.6739999999999999,"5min":4.7839999999999998,"15min":67.322999999999993},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:14:36 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.0209999999999999,"15min":1.1910000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.5720000000000001,"5min":5.3179999999999996,"15min":68.332999999999998},"last":2.214},{"interface":"front","average":{"1min":1.137,"5min":1.0009999999999999,"15min":1.22},"min":{"1min":0.41299999999999998,"5min":0.39800000000000002,"15min":0.32600000000000001},"max":{"1min":5.1520000000000001,"5min":6,"15min":67.281999999999996},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:14:39 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":3.7879999999999998,"15min":2.1099999999999999},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.316},"max":{"1min":4.6859999999999999,"5min":284.05599999999998,"15min":284.05599999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.458,"5min":3.9569999999999999,"15min":1.948},"min":{"1min":0.41299999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.7140000000000004,"5min":284.041,"15min":284.041},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:14:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":3.7250000000000001,"15min":2.177},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33500000000000002},"max":{"1min":3.2989999999999999,"5min":281.49700000000001,"15min":281.49700000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":3.8279999999999998,"15min":2.153},"min":{"1min":0.39600000000000002,"5min":0.371,"15min":0.29399999999999998},"max":{"1min":3.6059999999999999,"5min":281.61900000000003,"15min":281.61900000000003},"last":1.2330000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:14:12 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.96699999999999997,"15min":1.6759999999999999},"min":{"1min":0.56799999999999995,"5min":0.438,"15min":0.29399999999999998},"max":{"1min":1.9219999999999999,"5min":3.6349999999999998,"15min":67.144999999999996},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.05,"15min":1.6859999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.38200000000000001},"max":{"1min":1.5600000000000001,"5min":4.5449999999999999,"15min":67.230999999999995},"last":0.58199999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.93999999999999995,"15min":1.4339999999999999},"min":{"1min":0.53600000000000003,"5min":0.496,"15min":0.39800000000000002},"max":{"1min":3.2490000000000001,"5min":3.516,"15min":68.653000000000006},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.98499999999999999,"15min":1.3899999999999999},"min":{"1min":0.505,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.7149999999999999,"5min":5.016,"15min":68.620999999999995},"last":0.67100000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:14:29 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":0.94299999999999995,"15min":1.4530000000000001},"min":{"1min":0.56699999999999995,"5min":0.51000000000000001,"15min":0.40600000000000003},"max":{"1min":1.9830000000000001,"5min":2.2799999999999998,"15min":67.188000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.036,"15min":1.4450000000000001},"min":{"1min":0.57599999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.0449999999999999,"5min":4.9930000000000003,"15min":68.506},"last":1.181}]},{"osd":10,"last update":"Mon Feb 12 16:14:21 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":0.98599999999999999,"15min":1.4930000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.49099999999999999},"max":{"1min":1.802,"5min":3.5470000000000002,"15min":68.722999999999999},"last":1.153},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.046,"15min":1.4470000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.505},"max":{"1min":1.881,"5min":3.6389999999999998,"15min":68.573999999999998},"last":1.0800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4261476,"kb_used_data":3956060,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":89483676,"statfs":{"total":95995035648,"available":91631284224,"internally_reserved":0,"allocated":4051005440,"data_stored":4050499344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":2.512,"15min":1.526},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":1.2050000000000001,"5min":114.40000000000001,"15min":114.40000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":2.8460000000000001,"15min":1.665},"min":{"1min":0.32200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.1230000000000002,"5min":115.09099999999999,"15min":115.09099999999999},"last":3.5089999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.69399999999999995,"15min":1.1739999999999999},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":1.2889999999999999,"5min":4.4080000000000004,"15min":40.542999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.127,"15min":1.347},"min":{"1min":0.36099999999999999,"5min":0.27000000000000002,"15min":0.24299999999999999},"max":{"1min":1.867,"5min":7.1120000000000001,"15min":40.811999999999998},"last":0.501}]},{"osd":3,"last update":"Mon Feb 12 16:14:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":3.0379999999999998},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.8220000000000001,"5min":5.8220000000000001,"15min":5.8220000000000001},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":2.7210000000000001,"5min":1.2929999999999999,"15min":1.298},"min":{"1min":0.40799999999999997,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":33.125,"5min":33.125,"15min":33.125},"last":1.1539999999999999},{"interface":"front","average":{"1min":2.669,"5min":1.373,"15min":1.339},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":33.060000000000002,"5min":33.060000000000002,"15min":33.060000000000002},"last":0.95499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.92400000000000004,"15min":1.2210000000000001},"min":{"1min":0.39600000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.4950000000000001,"5min":4.6829999999999998,"15min":9.8019999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2150000000000001,"15min":1.27},"min":{"1min":0.41599999999999998,"5min":0.36199999999999999,"15min":0.34000000000000002},"max":{"1min":2.359,"5min":4.9340000000000002,"15min":10.420999999999999},"last":3.0840000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":6.7610000000000001,"5min":2.375,"15min":1.6459999999999999},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":56.137,"5min":56.137,"15min":56.137},"last":3.1190000000000002},{"interface":"front","average":{"1min":6.8849999999999998,"5min":2.4550000000000001,"15min":1.7150000000000001},"min":{"1min":0.50800000000000001,"5min":0.39000000000000001,"15min":0.33700000000000002},"max":{"1min":56.195999999999998,"5min":56.195999999999998,"15min":56.195999999999998},"last":3.3610000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:14:26 2024","interfaces":[{"interface":"back","average":{"1min":6.9900000000000002,"5min":2.3900000000000001,"15min":1.722},"min":{"1min":0.377,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":55.869999999999997,"5min":55.869999999999997,"15min":55.869999999999997},"last":0.73899999999999999},{"interface":"front","average":{"1min":7.1859999999999999,"5min":2.3799999999999999,"15min":1.647},"min":{"1min":0.47499999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":55.892000000000003,"5min":55.892000000000003,"15min":55.892000000000003},"last":3.3839999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:14:14 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.0469999999999999,"15min":1.365},"min":{"1min":0.502,"5min":0.45900000000000002,"15min":0.39400000000000002},"max":{"1min":2.2959999999999998,"5min":4.5449999999999999,"15min":9.9049999999999994},"last":3.2909999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.3160000000000001,"15min":1.518},"min":{"1min":0.68600000000000005,"5min":0.44400000000000001,"15min":0.41999999999999998},"max":{"1min":2.4689999999999999,"5min":5.0099999999999998,"15min":10.313000000000001},"last":0.69299999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.2989999999999999,"15min":1.482},"min":{"1min":0.55300000000000005,"5min":0.45400000000000001,"15min":0.40699999999999997},"max":{"1min":7.0359999999999996,"5min":7.0359999999999996,"15min":10.352},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.417,"5min":1.4370000000000001,"15min":1.391},"min":{"1min":0.53800000000000003,"5min":0.443,"15min":0.40200000000000002},"max":{"1min":2.4710000000000001,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":3.1680000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.0489999999999999,"15min":1.375},"min":{"1min":0.52800000000000002,"5min":0.51000000000000001,"15min":0.48499999999999999},"max":{"1min":1.758,"5min":4.5899999999999999,"15min":9.7629999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.3129999999999999,"15min":1.514},"min":{"1min":0.56499999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"max":{"1min":2.3740000000000001,"5min":5.1390000000000002,"15min":10.387},"last":3.2450000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.611,"15min":1.365},"min":{"1min":0.72499999999999998,"5min":0.443,"15min":0.443},"max":{"1min":2.5089999999999999,"5min":13.555,"15min":13.555},"last":3.2130000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.599,"15min":1.339},"min":{"1min":0.78200000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.6869999999999998,"5min":13.846,"15min":13.846},"last":3.3290000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738661,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":867840,"kb_used_data":558328,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92877312,"statfs":{"total":95995035648,"available":95106367488,"internally_reserved":0,"allocated":571727872,"data_stored":571239993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.109,"15min":1.472},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.22500000000000001},"max":{"1min":1.845,"5min":8.266,"15min":114.01600000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.85099999999999998,"15min":1.4430000000000001},"min":{"1min":0.45800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.7869999999999999,"5min":4.5819999999999999,"15min":114.342},"last":2.444}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.6819999999999999,"15min":1.2370000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.215},"max":{"1min":3.1110000000000002,"5min":53.793999999999997,"15min":53.793999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.609,"15min":1.165},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.454,"5min":53.853000000000002,"15min":53.853000000000002},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:14:25 2024","interfaces":[{"interface":"back","average":{"1min":5.1559999999999997,"5min":2.9729999999999999,"15min":1.647},"min":{"1min":0.34499999999999997,"5min":0.316,"15min":0.29799999999999999},"max":{"1min":80.619,"5min":116.187,"15min":116.187},"last":0.37},{"interface":"front","average":{"1min":4.9779999999999998,"5min":2.8730000000000002,"15min":1.6259999999999999},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":80.123000000000005,"5min":116.163,"15min":116.163},"last":0.32300000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:14:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":2.3580000000000001,"15min":1.4570000000000001},"min":{"1min":0.39100000000000001,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":3.2400000000000002,"5min":116.005,"15min":116.005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":2.3540000000000001,"15min":1.46},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.506,"5min":115.973,"15min":115.973},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:14:42 2024","interfaces":[{"interface":"back","average":{"1min":4.7160000000000002,"5min":1.728,"15min":1.2709999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33000000000000002},"max":{"1min":81.147000000000006,"5min":81.147000000000006,"15min":81.147000000000006},"last":0.63800000000000001},{"interface":"front","average":{"1min":4.734,"5min":1.7509999999999999,"15min":1.3089999999999999},"min":{"1min":0.39200000000000002,"5min":0.375,"15min":0.34300000000000003},"max":{"1min":80.233000000000004,"5min":80.233000000000004,"15min":80.233000000000004},"last":0.71499999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":2.4239999999999999,"15min":1.5820000000000001},"min":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.54,"5min":115.26000000000001,"15min":115.26000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.0820000000000001,"5min":2.516,"15min":1.702},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.6000000000000001,"5min":115.402,"15min":115.402},"last":0.80200000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.0880000000000001,"15min":1.234},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.378},"max":{"1min":1.7090000000000001,"5min":3.6619999999999999,"15min":5.6059999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.0940000000000001,"15min":1.26},"min":{"1min":0.53900000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8100000000000001,"5min":3.7959999999999998,"15min":7.5599999999999996},"last":0.82999999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.2130000000000001,"15min":1.3280000000000001},"min":{"1min":0.46999999999999997,"5min":0.46000000000000002,"15min":0.40600000000000003},"max":{"1min":3.4649999999999999,"5min":4.6310000000000002,"15min":5.7969999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.004,"5min":1.1639999999999999,"15min":1.2569999999999999},"min":{"1min":0.47199999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.996,"5min":6.6070000000000002,"15min":6.6070000000000002},"last":0.92500000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:14:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.073,"15min":1.268},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.804,"5min":5.7869999999999999,"15min":5.8259999999999996},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.22,"15min":1.2969999999999999},"min":{"1min":0.56399999999999995,"5min":0.48099999999999998,"15min":0.437},"max":{"1min":2.952,"5min":6.2480000000000002,"15min":6.2480000000000002},"last":0.95699999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.3080000000000001,"15min":1.3740000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":4.1680000000000001,"5min":6.0149999999999997,"15min":7.9660000000000002},"last":0.438},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.1599999999999999,"15min":1.3560000000000001},"min":{"1min":0.54700000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.3919999999999999,"5min":8.4800000000000004,"15min":8.4800000000000004},"last":0.77400000000000002}]}]},{"osd":2,"up_from":21,"seq":90194313492,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":856396,"kb_used_data":559172,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92888756,"statfs":{"total":95995035648,"available":95118086144,"internally_reserved":0,"allocated":572592128,"data_stored":572098210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":3.5099999999999998,"15min":1.8520000000000001},"min":{"1min":0.29699999999999999,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":5.173,"5min":177.68899999999999,"15min":177.68899999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":3.4900000000000002,"15min":1.8979999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.216},"max":{"1min":2.4540000000000002,"5min":178.26900000000001,"15min":178.26900000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":1.7370000000000001,"15min":1.3879999999999999},"min":{"1min":0.35499999999999998,"5min":0.23799999999999999,"15min":0.223},"max":{"1min":1.403,"5min":70.944000000000003,"15min":70.944000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.77,"15min":1.4259999999999999},"min":{"1min":0.40899999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.4399999999999999,"5min":71.203999999999994,"15min":71.203999999999994},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:14:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":2.6800000000000002,"15min":1.7470000000000001},"min":{"1min":0.29999999999999999,"5min":0.28599999999999998,"15min":0.25600000000000001},"max":{"1min":1.617,"5min":80.436999999999998,"15min":80.436999999999998},"last":0.376},{"interface":"front","average":{"1min":0.65900000000000003,"5min":2.6389999999999998,"15min":1.6020000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26300000000000001},"max":{"1min":1.2070000000000001,"5min":77.286000000000001,"15min":77.286000000000001},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:14:30 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.83599999999999997,"15min":1.1120000000000001},"min":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.311},"max":{"1min":1.7809999999999999,"5min":4.2789999999999999,"15min":9.657},"last":1.778},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.875,"15min":1.008},"min":{"1min":0.33100000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.647,"5min":4.7969999999999997,"15min":9.5530000000000008},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":0.93700000000000006,"15min":3.2559999999999998},"min":{"1min":0.42899999999999999,"5min":0.33900000000000002,"15min":0.33400000000000002},"max":{"1min":1.913,"5min":5.1520000000000001,"15min":616.34199999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":0.92700000000000005,"15min":3.3530000000000002},"min":{"1min":0.50600000000000001,"5min":0.38400000000000001,"15min":0.33900000000000002},"max":{"1min":1.772,"5min":5.0910000000000002,"15min":615.18899999999996},"last":0.73999999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:14:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.81699999999999995,"15min":1.0489999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.1309999999999998,"5min":3.597,"15min":9.6150000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.84099999999999997,"15min":1.1399999999999999},"min":{"1min":0.48199999999999998,"5min":0.39600000000000002,"15min":0.35099999999999998},"max":{"1min":1.7589999999999999,"5min":5.4349999999999996,"15min":6.3979999999999997},"last":0.55300000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:14:30 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.008,"15min":3.4369999999999998},"min":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.377},"max":{"1min":1.986,"5min":4.032,"15min":611.79600000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.018,"5min":1.044,"15min":3.4710000000000001},"min":{"1min":0.47599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.843,"5min":4.0369999999999999,"15min":611.83900000000006},"last":0.71099999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.84299999999999997,"15min":1.26},"min":{"1min":0.51700000000000002,"5min":0.38700000000000001,"15min":0.38200000000000001},"max":{"1min":1.6379999999999999,"5min":2.5920000000000001,"15min":6.4960000000000004},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.88800000000000001,"15min":1.3560000000000001},"min":{"1min":0.52200000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.577,"5min":2.1680000000000001,"15min":8.5920000000000005},"last":0.79400000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:14:30 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.96199999999999997,"15min":1.226},"min":{"1min":0.47399999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.3199999999999998,"5min":7.2830000000000004,"15min":9.6460000000000008},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.085,"5min":0.90300000000000002,"15min":1.3049999999999999},"min":{"1min":0.54500000000000004,"5min":0.47799999999999998,"15min":0.42999999999999999},"max":{"1min":2.2589999999999999,"5min":2.6230000000000002,"15min":6.4699999999999998},"last":0.85099999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":0.96799999999999997,"15min":1.284},"min":{"1min":0.54900000000000004,"5min":0.442,"15min":0.442},"max":{"1min":2.4460000000000002,"5min":3.8250000000000002,"15min":9.6760000000000002},"last":0.621},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.95799999999999996,"15min":1.339},"min":{"1min":0.58999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.165,"5min":2.6520000000000001,"15min":6.4359999999999999},"last":0.88200000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149963,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1948604,"kb_used_data":1643188,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91796548,"statfs":{"total":95995035648,"available":93999665152,"internally_reserved":0,"allocated":1682624512,"data_stored":1682140091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:05 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.0269999999999999,"15min":1.01},"min":{"1min":0.27100000000000002,"5min":0.246,"15min":0.246},"max":{"1min":2.7930000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":0.496},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.96599999999999997,"15min":0.90200000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.5740000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:08:18 2024","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":2.0659999999999998,"15min":1.2549999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":8.3680000000000003,"5min":70.031999999999996,"15min":70.031999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.621,"5min":2.25,"15min":1.339},"min":{"1min":0.317,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.3399999999999999,"5min":69.944999999999993,"15min":69.944999999999993},"last":0.95299999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:14:35 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.96299999999999997,"15min":1.254},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.526,"5min":7.5780000000000003,"15min":68.405000000000001},"last":3.077},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.0449999999999999,"15min":1.2789999999999999},"min":{"1min":0.35099999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.5600000000000001,"5min":9.5619999999999994,"15min":69.009},"last":3.2120000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.86899999999999999,"15min":0.91800000000000004},"min":{"1min":0.34200000000000003,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":3.1899999999999999,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.90000000000000002,"15min":0.96199999999999997},"min":{"1min":0.38100000000000001,"5min":0.307,"15min":0.307},"max":{"1min":2.2839999999999998,"5min":5.9870000000000001,"15min":5.9870000000000001},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:14:55 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.444,"15min":1.1870000000000001},"min":{"1min":0.371,"5min":0.313,"15min":0.313},"max":{"1min":3.6309999999999998,"5min":22.751000000000001,"15min":22.751000000000001},"last":3.1160000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.1799999999999999,"15min":1.0900000000000001},"min":{"1min":0.38200000000000001,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":4.5970000000000004,"5min":22.527999999999999,"15min":22.527999999999999},"last":4.5970000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:14:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.89500000000000002,"15min":1.0229999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.0049999999999999,"5min":6.0289999999999999,"15min":6.0289999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.24,"15min":1.1919999999999999},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.6879999999999999,"5min":9.5990000000000002,"15min":9.5990000000000002},"last":3.1589999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:14:23 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.169,"15min":1.4730000000000001},"min":{"1min":0.45000000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.857,"5min":7.5970000000000004,"15min":91.718999999999994},"last":3.1899999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.4770000000000001,"15min":1.611},"min":{"1min":0.442,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.0640000000000001,"5min":10.944000000000001,"15min":90.367999999999995},"last":0.41499999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:14:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0640000000000001,"15min":1.222},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.317},"max":{"1min":3.1030000000000002,"5min":6.0970000000000004,"15min":6.5899999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.038,"5min":1.0509999999999999,"15min":1.2270000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.1629999999999998,"5min":6.2430000000000003,"15min":8.516},"last":0.60599999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:14:35 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.0029999999999999,"15min":1.232},"min":{"1min":0.48899999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.0219999999999998,"5min":6.2699999999999996,"15min":6.2699999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.486,"5min":1.296,"15min":1.3660000000000001},"min":{"1min":0.55900000000000005,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.0379999999999998,"5min":9.6509999999999998,"15min":9.6509999999999998},"last":0.51400000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.137,"15min":1.323},"min":{"1min":0.59499999999999997,"5min":0.42699999999999999,"15min":0.41099999999999998},"max":{"1min":2.9569999999999999,"5min":6.4199999999999999,"15min":6.4199999999999999},"last":0.751},{"interface":"front","average":{"1min":1.577,"5min":1.153,"15min":1.345},"min":{"1min":0.628,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":2.9820000000000002,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.81200000000000006}]},{"osd":11,"last update":"Mon Feb 12 16:14:55 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.2829999999999999,"15min":1.518},"min":{"1min":0.60699999999999998,"5min":0.47699999999999998,"15min":0.39100000000000001},"max":{"1min":2.0640000000000001,"5min":6.1390000000000002,"15min":49.167000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.278,"15min":1.5349999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":3.137,"5min":6.1310000000000002,"15min":48.548999999999999},"last":0.57799999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953732,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1962956,"kb_used_data":1661636,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91782196,"statfs":{"total":95995035648,"available":93984968704,"internally_reserved":0,"allocated":1701515264,"data_stored":1701021346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":2.4969999999999999,"15min":1.4419999999999999},"min":{"1min":0.29399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.1040000000000001,"5min":96.052000000000007,"15min":96.052000000000007},"last":0.376},{"interface":"front","average":{"1min":0.76900000000000002,"5min":2.6789999999999998,"15min":1.575},"min":{"1min":0.34499999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.6240000000000001,"5min":96.307000000000002,"15min":96.307000000000002},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.1930000000000001,"15min":1.304},"min":{"1min":0.38600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.4700000000000002,"5min":9.718,"15min":42.226999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.2789999999999999,"15min":1.234},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":1.486,"5min":5.5119999999999996,"15min":43.689},"last":4.883}]},{"osd":2,"last update":"Mon Feb 12 16:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.998,"15min":1.169},"min":{"1min":0.36199999999999999,"5min":0.28299999999999997,"15min":0.27700000000000002},"max":{"1min":2.3759999999999999,"5min":6.3239999999999998,"15min":41.546999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.113,"15min":1.2410000000000001},"min":{"1min":0.39400000000000002,"5min":0.38300000000000001,"15min":0.31},"max":{"1min":1.548,"5min":5.5449999999999999,"15min":41.57},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:14:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":2.226,"15min":1.381},"min":{"1min":0.34399999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.3359999999999999,"5min":82.781000000000006,"15min":82.781000000000006},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.91000000000000003,"5min":2.1240000000000001,"15min":1.3720000000000001},"min":{"1min":0.35599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4770000000000001,"5min":83.349000000000004,"15min":83.349000000000004},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.91400000000000003,"15min":0.92400000000000004},"min":{"1min":0.40100000000000002,"5min":0.315,"15min":0.25800000000000001},"max":{"1min":1.9910000000000001,"5min":6.0460000000000003,"15min":6.0460000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.044,"15min":1.032},"min":{"1min":0.35799999999999998,"5min":0.35099999999999998,"15min":0.31900000000000001},"max":{"1min":3.0979999999999999,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:14:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0649999999999999,"15min":1.018},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.0219999999999998,"5min":5.2539999999999996,"15min":5.2539999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0740000000000001,"15min":0.98499999999999999},"min":{"1min":0.36499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.9630000000000001,"5min":6.2249999999999996,"15min":6.2249999999999996},"last":0.67900000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:14:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.302,"15min":1.55},"min":{"1min":0.47799999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.8359999999999999,"5min":9.0879999999999992,"15min":116.273},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.157,"5min":1.222,"15min":1.643},"min":{"1min":0.50600000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.71,"5min":5.2850000000000001,"15min":116.24299999999999},"last":0.754}]},{"osd":8,"last update":"Mon Feb 12 16:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.2310000000000001,"15min":1.2909999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":3.2360000000000002,"5min":5.4320000000000004,"15min":8.4190000000000005},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.1579999999999999,"15min":1.2070000000000001},"min":{"1min":0.54700000000000004,"5min":0.502,"15min":0.42899999999999999},"max":{"1min":3.1680000000000001,"5min":7.0380000000000003,"15min":7.0380000000000003},"last":0.76500000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:14:27 2024","interfaces":[{"interface":"back","average":{"1min":2.4319999999999999,"5min":1.474,"15min":2.4369999999999998},"min":{"1min":0.51300000000000001,"5min":0.46300000000000002,"15min":0.45500000000000002},"max":{"1min":25.294,"5min":25.294,"15min":297.55900000000003},"last":0.73899999999999999},{"interface":"front","average":{"1min":2.395,"5min":1.4970000000000001,"15min":2.431},"min":{"1min":0.59899999999999998,"5min":0.52600000000000002,"15min":0.497},"max":{"1min":24.622,"5min":24.622,"15min":297.61900000000003},"last":0.86599999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:14:21 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.216,"15min":1.268},"min":{"1min":0.70499999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.7909999999999999,"5min":6.9770000000000003,"15min":6.9770000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3919999999999999,"15min":1.325},"min":{"1min":0.58699999999999997,"5min":0.51400000000000001,"15min":0.47599999999999998},"max":{"1min":2.4079999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.58899999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724796,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1423076,"kb_used_data":1113564,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92322076,"statfs":{"total":95995035648,"available":94537805824,"internally_reserved":0,"allocated":1140289536,"data_stored":1139783440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.575,"15min":1.2070000000000001},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25},"max":{"1min":2.9900000000000002,"5min":75.396000000000001,"15min":75.396000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.6839999999999999,"15min":1.24},"min":{"1min":0.30399999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.5449999999999999,"5min":77.286000000000001,"15min":77.286000000000001},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.88,"15min":1.2070000000000001},"min":{"1min":0.40000000000000002,"5min":0.28999999999999998,"15min":0.26000000000000001},"max":{"1min":3.0350000000000001,"5min":6.3630000000000004,"15min":33.084000000000003},"last":0.308},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.93700000000000006,"15min":1.1000000000000001},"min":{"1min":0.373,"5min":0.29599999999999999,"15min":0.253},"max":{"1min":2.9409999999999998,"5min":5.5890000000000004,"15min":33.052999999999997},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.93300000000000005,"15min":1.2330000000000001},"min":{"1min":0.38700000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.621,"5min":5.6580000000000004,"15min":26.797000000000001},"last":0.438},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.91300000000000003,"15min":1.1279999999999999},"min":{"1min":0.38,"5min":0.35599999999999998,"15min":0.28599999999999998},"max":{"1min":3.0680000000000001,"5min":6.4610000000000003,"15min":26.905999999999999},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":14.382,"5min":5.7569999999999997,"15min":2.7650000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":252.57300000000001,"5min":252.57300000000001,"15min":252.57300000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":14.452,"5min":5.8220000000000001,"15min":2.7530000000000001},"min":{"1min":0.47499999999999998,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":252.24199999999999,"5min":252.24199999999999,"15min":252.24199999999999},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.004,"15min":0.99199999999999999},"min":{"1min":0.46999999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0939999999999999,"5min":6.492,"15min":14.259},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.095,"5min":1.1559999999999999,"15min":1.103},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.32600000000000001},"max":{"1min":4.9880000000000004,"5min":5.7300000000000004,"15min":13.268000000000001},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:14:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6189999999999998,"5min":1.9390000000000001,"15min":1.419},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.318},"max":{"1min":74.105000000000004,"5min":74.105000000000004,"15min":74.105000000000004},"last":2.9620000000000002},{"interface":"front","average":{"1min":5.9279999999999999,"5min":2.032,"15min":1.4690000000000001},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":75.013999999999996,"5min":75.013999999999996,"15min":75.013999999999996},"last":0.626}]},{"osd":7,"last update":"Mon Feb 12 16:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.1539999999999999,"15min":1.577},"min":{"1min":0.42399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.466,"5min":6.9729999999999999,"15min":98.971000000000004},"last":0.376},{"interface":"front","average":{"1min":1.252,"5min":1.157,"15min":1.639},"min":{"1min":0.53300000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.925,"5min":5.9909999999999997,"15min":99.218000000000004},"last":0.73599999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.175,"15min":1.3100000000000001},"min":{"1min":0.53200000000000003,"5min":0.44700000000000001,"15min":0.35099999999999998},"max":{"1min":3.117,"5min":6.5190000000000001,"15min":14.372999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.415,"5min":1.194,"15min":1.321},"min":{"1min":0.51500000000000001,"5min":0.42999999999999999,"15min":0.40500000000000003},"max":{"1min":2.714,"5min":5.6950000000000003,"15min":14.074999999999999},"last":0.69299999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:14:40 2024","interfaces":[{"interface":"back","average":{"1min":30.861000000000001,"5min":7.1100000000000003,"15min":3.2959999999999998},"min":{"1min":0.46999999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":449.89499999999998,"5min":449.89499999999998,"15min":449.89499999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":30.834,"5min":7.1980000000000004,"15min":3.355},"min":{"1min":0.51600000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":448.70800000000003,"5min":448.70800000000003,"15min":448.70800000000003},"last":0.71999999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:14:09 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.2190000000000001,"15min":1.3300000000000001},"min":{"1min":0.57699999999999996,"5min":0.503,"15min":0.47499999999999998},"max":{"1min":1.724,"5min":6.6849999999999996,"15min":14.214},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.286,"15min":1.3939999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.52100000000000002},"max":{"1min":1.8879999999999999,"5min":5.1890000000000001,"15min":13.464},"last":3.1360000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":575012864,"data_stored":575012864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4062969856,"data_stored":4062969856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":577241088,"data_stored":577241088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1695678464,"data_stored":1695678464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1700855808,"data_stored":1700855808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1144258560,"data_stored":1144258560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1124794368,"data_stored":1124794368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1748303872,"data_stored":1748303872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1727070208,"data_stored":1727070208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1756823552,"data_stored":1756823552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2258763776,"data_stored":2258763776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:15:04.916 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T16:15:04.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set device_health_metrics pg_num 11 2024-02-12T16:15:04.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:04.992 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:04.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:05.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:05.103 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:04 smithi169 bash[19604]: cluster 2024-02-12T16:15:03.716026+0000 mgr.y (mgr.14146) 891 : cluster [DBG] pgmap v1151: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-12T16:15:05.103 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:04 smithi169 bash[19604]: audit 2024-02-12T16:15:03.916245+0000 mgr.y (mgr.14146) 892 : audit [DBG] from='client.15498 v1:172.21.15.121:0/3902226235' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:15:05.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:15:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:05.129 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:05.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:05.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:04 smithi121 bash[18489]: cluster 2024-02-12T16:15:03.716026+0000 mgr.y (mgr.14146) 891 : cluster [DBG] pgmap v1151: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-12T16:15:05.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:04 smithi121 bash[18489]: audit 2024-02-12T16:15:03.916245+0000 mgr.y (mgr.14146) 892 : audit [DBG] from='client.15498 v1:172.21.15.121:0/3902226235' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:15:05.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:15:05.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 145071 145055 135.301 84.0625 0.00541999 0.0073888 2024-02-12T16:15:05.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:05.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:05.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:05.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:05.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:05.386 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:05.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:05.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:05.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:05.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:05.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:05.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:05.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:05.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:05.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:05.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:05.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:05.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:06.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:06.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:05 smithi121 bash[18489]: audit 2024-02-12T16:15:05.299985+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:15:06.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:05 smithi121 bash[18489]: cluster 2024-02-12T16:15:05.388270+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e359: 12 total, 12 up, 12 in 2024-02-12T16:15:06.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:05 smithi121 bash[18489]: audit 2024-02-12T16:15:05.655215+0000 mon.a (mon.0) 1400 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:15:06.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:05 smithi121 bash[18489]: audit 2024-02-12T16:15:05.656111+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:15:06.030 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:05 smithi121 bash[18489]: audit 2024-02-12T16:15:05.661500+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:15:06.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:06.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:05 smithi123 bash[19119]: audit 2024-02-12T16:15:05.299985+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:15:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:05 smithi123 bash[19119]: cluster 2024-02-12T16:15:05.388270+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e359: 12 total, 12 up, 12 in 2024-02-12T16:15:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:05 smithi123 bash[19119]: audit 2024-02-12T16:15:05.655215+0000 mon.a (mon.0) 1400 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:15:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:05 smithi123 bash[19119]: audit 2024-02-12T16:15:05.656111+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:15:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:05 smithi123 bash[19119]: audit 2024-02-12T16:15:05.661500+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:15:06.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:06.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 16 146329 146313 134.468 78.625 0.012436 0.00743523 2024-02-12T16:15:06.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:06.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:05 smithi169 bash[19604]: audit 2024-02-12T16:15:05.299985+0000 mon.a (mon.0) 1398 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:15:06.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:05 smithi169 bash[19604]: cluster 2024-02-12T16:15:05.388270+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e359: 12 total, 12 up, 12 in 2024-02-12T16:15:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:05 smithi169 bash[19604]: audit 2024-02-12T16:15:05.655215+0000 mon.a (mon.0) 1400 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:15:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:05 smithi169 bash[19604]: audit 2024-02-12T16:15:05.656111+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:15:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:05 smithi169 bash[19604]: audit 2024-02-12T16:15:05.661500+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:15:06.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:06.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:06.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:06.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:06.584 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:06.687 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:06.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:06.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:15:06.841 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:06.945 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:06.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:06.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:06 smithi169 bash[19604]: cluster 2024-02-12T16:15:05.716430+0000 mgr.y (mgr.14146) 893 : cluster [DBG] pgmap v1153: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-12T16:15:07.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:06 smithi121 bash[18489]: cluster 2024-02-12T16:15:05.716430+0000 mgr.y (mgr.14146) 893 : cluster [DBG] pgmap v1153: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-12T16:15:07.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:07.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:07.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:06 smithi123 bash[19119]: cluster 2024-02-12T16:15:05.716430+0000 mgr.y (mgr.14146) 893 : cluster [DBG] pgmap v1153: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-02-12T16:15:07.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 147662 147646 133.726 83.3125 0.00479352 0.00747554 2024-02-12T16:15:07.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:07.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:07.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:07.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:07.370 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:07.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:07.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:07.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:07.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:07.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:07.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:07.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:07.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:07.758 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:07.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:07.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:07.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:07.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:07.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:08.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:08.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:08.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:08.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 16 149011 148995 133.02 84.3125 0.0162954 0.00751596 2024-02-12T16:15:08.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:08.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:08.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:08.418 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:08.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:08.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:08.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:08.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:08.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:08.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:08.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:08 smithi169 bash[19604]: cluster 2024-02-12T16:15:07.612316+0000 osd.1 (osd.1) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:08.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:08 smithi169 bash[19604]: cluster 2024-02-12T16:15:07.612362+0000 osd.1 (osd.1) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:08.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:08 smithi169 bash[19604]: cluster 2024-02-12T16:15:07.716848+0000 mgr.y (mgr.14146) 894 : cluster [DBG] pgmap v1154: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-02-12T16:15:08.937 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:08 smithi169 bash[19604]: cluster 2024-02-12T16:15:07.789217+0000 osd.10 (osd.10) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:08.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:08 smithi169 bash[19604]: cluster 2024-02-12T16:15:07.789280+0000 osd.10 (osd.10) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:08.938 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:08 smithi169 bash[19604]: cluster 2024-02-12T16:15:07.855436+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e360: 12 total, 12 up, 12 in 2024-02-12T16:15:08.958 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:08.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:09.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:09.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:09.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:09.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:08 smithi123 bash[19119]: cluster 2024-02-12T16:15:07.612316+0000 osd.1 (osd.1) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:08 smithi123 bash[19119]: cluster 2024-02-12T16:15:07.612362+0000 osd.1 (osd.1) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:08 smithi123 bash[19119]: cluster 2024-02-12T16:15:07.716848+0000 mgr.y (mgr.14146) 894 : cluster [DBG] pgmap v1154: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-02-12T16:15:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:08 smithi123 bash[19119]: cluster 2024-02-12T16:15:07.789217+0000 osd.10 (osd.10) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:08 smithi123 bash[19119]: cluster 2024-02-12T16:15:07.789280+0000 osd.10 (osd.10) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:08 smithi123 bash[19119]: cluster 2024-02-12T16:15:07.855436+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e360: 12 total, 12 up, 12 in 2024-02-12T16:15:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:08 smithi121 bash[18489]: cluster 2024-02-12T16:15:07.612316+0000 osd.1 (osd.1) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:08 smithi121 bash[18489]: cluster 2024-02-12T16:15:07.612362+0000 osd.1 (osd.1) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:08 smithi121 bash[18489]: cluster 2024-02-12T16:15:07.716848+0000 mgr.y (mgr.14146) 894 : cluster [DBG] pgmap v1154: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-02-12T16:15:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:08 smithi121 bash[18489]: cluster 2024-02-12T16:15:07.789217+0000 osd.10 (osd.10) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:08 smithi121 bash[18489]: cluster 2024-02-12T16:15:07.789280+0000 osd.10 (osd.10) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:08 smithi121 bash[18489]: cluster 2024-02-12T16:15:07.855436+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e360: 12 total, 12 up, 12 in 2024-02-12T16:15:09.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 150376 150360 132.348 85.3125 0.0182907 0.00755439 2024-02-12T16:15:09.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:09.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:09.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:09.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:09.347 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:09.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:09.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:09.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:09.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:09.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:09.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:09.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:09.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:09.752 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:09.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:09.777 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:09.777 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:09.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:09.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:09.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:09.994 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:10.020 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:10.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:10.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:10.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 151678 151662 131.64 81.375 0.0113936 0.00759509 2024-02-12T16:15:10.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:10.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:10.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:10.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:10.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:10.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:10.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:10.683 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:10.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:10.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:10.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:10.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:10 smithi123 bash[19119]: cluster 2024-02-12T16:15:09.717274+0000 mgr.y (mgr.14146) 895 : cluster [DBG] pgmap v1156: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-12T16:15:10.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:10 smithi123 bash[19119]: cluster 2024-02-12T16:15:09.926616+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:10.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:10 smithi123 bash[19119]: cluster 2024-02-12T16:15:09.926672+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:10.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:10 smithi123 bash[19119]: cluster 2024-02-12T16:15:10.387053+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e361: 12 total, 12 up, 12 in 2024-02-12T16:15:10.947 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:10.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:11.031 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:15:11.032 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:15:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:10 smithi169 bash[19604]: cluster 2024-02-12T16:15:09.717274+0000 mgr.y (mgr.14146) 895 : cluster [DBG] pgmap v1156: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-12T16:15:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:10 smithi169 bash[19604]: cluster 2024-02-12T16:15:09.926616+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:10 smithi169 bash[19604]: cluster 2024-02-12T16:15:09.926672+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:10 smithi169 bash[19604]: cluster 2024-02-12T16:15:10.387053+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e361: 12 total, 12 up, 12 in 2024-02-12T16:15:11.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:11.079 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:11.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:10 smithi121 bash[18489]: cluster 2024-02-12T16:15:09.717274+0000 mgr.y (mgr.14146) 895 : cluster [DBG] pgmap v1156: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-12T16:15:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:10 smithi121 bash[18489]: cluster 2024-02-12T16:15:09.926616+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:10 smithi121 bash[18489]: cluster 2024-02-12T16:15:09.926672+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:11.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:10 smithi121 bash[18489]: cluster 2024-02-12T16:15:10.387053+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e361: 12 total, 12 up, 12 in 2024-02-12T16:15:11.161 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 152974 152958 130.946 81 0.00428684 0.00763472 2024-02-12T16:15:11.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:11.207 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:11.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:11.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:11.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:11.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:11.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:11.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:11.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:11.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:11.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:11.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:11.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:11.746 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:11.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:11.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:11.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:15:11.872 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:11 smithi169 bash[19604]: cluster 2024-02-12T16:15:11.067654+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:11.875 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:11.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:11.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:12.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:12.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:12.118 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:11 smithi123 bash[19119]: cluster 2024-02-12T16:15:11.067654+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:12.118 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:11 smithi123 bash[19119]: cluster 2024-02-12T16:15:11.067707+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:12.145 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:12.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:11 smithi121 bash[18489]: cluster 2024-02-12T16:15:11.067654+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:12.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:11 smithi121 bash[18489]: cluster 2024-02-12T16:15:11.067707+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:12.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 154290 154274 130.288 82.25 0.0104298 0.00767387 2024-02-12T16:15:12.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:12.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:12.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:11 smithi169 bash[19604]: cluster 2024-02-12T16:15:11.067707+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:12.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:12.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:12.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:12.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:12.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:12.702 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:12.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:12.829 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:12.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:12.975 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:12.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:13.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:12 smithi169 bash[19604]: cluster 2024-02-12T16:15:11.717643+0000 mgr.y (mgr.14146) 896 : cluster [DBG] pgmap v1158: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-02-12T16:15:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:12 smithi169 bash[19604]: cluster 2024-02-12T16:15:11.869305+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e362: 12 total, 12 up, 12 in 2024-02-12T16:15:13.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:13.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:13.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:13.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:12 smithi123 bash[19119]: cluster 2024-02-12T16:15:11.717643+0000 mgr.y (mgr.14146) 896 : cluster [DBG] pgmap v1158: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-02-12T16:15:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:12 smithi123 bash[19119]: cluster 2024-02-12T16:15:11.869305+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e362: 12 total, 12 up, 12 in 2024-02-12T16:15:13.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:12 smithi121 bash[18489]: cluster 2024-02-12T16:15:11.717643+0000 mgr.y (mgr.14146) 896 : cluster [DBG] pgmap v1158: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-02-12T16:15:13.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:12 smithi121 bash[18489]: cluster 2024-02-12T16:15:11.869305+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e362: 12 total, 12 up, 12 in 2024-02-12T16:15:13.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 16 155497 155481 129.557 75.4375 0.0359821 0.0077159 2024-02-12T16:15:13.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:13.241 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:13.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:13.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:13.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:13.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:13.476 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:13.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:13.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:13.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:13.628 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:13.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:13.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:13.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:13.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:13.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:13.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:13.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:13.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:14.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:14.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:14.157 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:14.163 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 156625 156609 128.78 70.5 0.0164044 0.00776354 2024-02-12T16:15:14.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:14.285 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:14.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:14.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:14.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:14.547 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:14.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:14.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:14.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:14.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:14.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:14.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:14.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:14 smithi169 bash[19604]: cluster 2024-02-12T16:15:13.718082+0000 mgr.y (mgr.14146) 897 : cluster [DBG] pgmap v1160: 17 pgs: 17 active+clean; 9.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-12T16:15:15.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:15.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:15.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:15.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:14 smithi123 bash[19119]: cluster 2024-02-12T16:15:13.718082+0000 mgr.y (mgr.14146) 897 : cluster [DBG] pgmap v1160: 17 pgs: 17 active+clean; 9.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-12T16:15:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:14 smithi121 bash[18489]: cluster 2024-02-12T16:15:13.718082+0000 mgr.y (mgr.14146) 897 : cluster [DBG] pgmap v1160: 17 pgs: 17 active+clean; 9.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-02-12T16:15:15.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 16 157915 157899 128.154 80.625 0.00490783 0.00780134 2024-02-12T16:15:15.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:15.238 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:15.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:15.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:15.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:15.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:15.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:15.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:15.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:15.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:15.640 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:15.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:15.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:15.768 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:15.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:15.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:15.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:15.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:16.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:16.034 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:16.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:16.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:16.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 159201 159185 127.541 80.375 0.00712041 0.00783847 2024-02-12T16:15:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:16.271 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:16.296 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:16.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:16.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:16.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:16 smithi121 bash[18489]: cluster 2024-02-12T16:15:15.388359+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e363: 12 total, 12 up, 12 in 2024-02-12T16:15:16.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:16 smithi121 bash[18489]: cluster 2024-02-12T16:15:15.718430+0000 mgr.y (mgr.14146) 898 : cluster [DBG] pgmap v1162: 17 pgs: 17 active+clean; 9.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:15:16.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:16.531 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:16.538 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:16 smithi123 bash[19119]: cluster 2024-02-12T16:15:15.388359+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e363: 12 total, 12 up, 12 in 2024-02-12T16:15:16.538 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:16 smithi123 bash[19119]: cluster 2024-02-12T16:15:15.718430+0000 mgr.y (mgr.14146) 898 : cluster [DBG] pgmap v1162: 17 pgs: 17 active+clean; 9.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:15:16.562 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:16.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:16.667 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:15:16.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:16.746 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:15:16.780 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:15:16.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:16.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:16 smithi169 bash[19604]: cluster 2024-02-12T16:15:15.388359+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e363: 12 total, 12 up, 12 in 2024-02-12T16:15:16.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:16 smithi169 bash[19604]: cluster 2024-02-12T16:15:15.718430+0000 mgr.y (mgr.14146) 898 : cluster [DBG] pgmap v1162: 17 pgs: 17 active+clean; 9.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:15:16.827 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:15:16.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:16.863 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:15:16.894 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:15:16.922 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:15:16.952 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:15:16.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:16.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:16.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:16.992 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:15:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:15:17.057 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:15:17.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:17.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:15:17.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:15:17.111 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:17.113 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.113 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.113 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.113 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.113 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:15:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:17.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:17.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 160297 160281 126.794 68.5 0.00645434 0.0078853 2024-02-12T16:15:17.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:17.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:17.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:17.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:17.365 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:17.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:17.410 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 1 pg_num to 11 2024-02-12T16:15:17.416 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:15:17.457 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:17 smithi121 bash[18489]: audit 2024-02-12T16:15:16.706937+0000 mon.b (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.121:0/3934276585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:17.457 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:17 smithi121 bash[18489]: audit 2024-02-12T16:15:17.019937+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:17.457 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:17 smithi121 bash[18489]: audit 2024-02-12T16:15:17.022161+0000 mon.c (mon.1) 35 : audit [INF] from='client.? v1:172.21.15.121:0/23172251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:17.457 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:17 smithi121 bash[18489]: audit 2024-02-12T16:15:17.051904+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? v1:172.21.15.121:0/395030306' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2024-02-12T16:15:17.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:17.494 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:17.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:17.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:17.623 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:17.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:17 smithi123 bash[19119]: audit 2024-02-12T16:15:16.706937+0000 mon.b (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.121:0/3934276585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:17 smithi123 bash[19119]: audit 2024-02-12T16:15:17.019937+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:17.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:17 smithi123 bash[19119]: audit 2024-02-12T16:15:17.022161+0000 mon.c (mon.1) 35 : audit [INF] from='client.? v1:172.21.15.121:0/23172251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:17.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:17 smithi123 bash[19119]: audit 2024-02-12T16:15:17.051904+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? v1:172.21.15.121:0/395030306' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2024-02-12T16:15:17.729 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:17.761 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:17.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:17.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:17 smithi169 bash[19604]: audit 2024-02-12T16:15:16.706937+0000 mon.b (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.121:0/3934276585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:17 smithi169 bash[19604]: audit 2024-02-12T16:15:17.019937+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:17 smithi169 bash[19604]: audit 2024-02-12T16:15:17.022161+0000 mon.c (mon.1) 35 : audit [INF] from='client.? v1:172.21.15.121:0/23172251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:17 smithi169 bash[19604]: audit 2024-02-12T16:15:17.051904+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? v1:172.21.15.121:0/395030306' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2024-02-12T16:15:17.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:17.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:17.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:18.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:18.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:18.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:18.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:15:18.160087 min lat: 0.0010598 max lat: 0.735335 avg lat: 0.00792449 2024-02-12T16:15:18.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:15:18.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 161527 161511 126.17 76.875 0.0157245 0.00792449 2024-02-12T16:15:18.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:18.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:18.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:18.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:18.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:18.526 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:18.562 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:18.665 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:18.673 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:18 smithi123 bash[19119]: cluster 2024-02-12T16:15:17.391298+0000 mon.a (mon.0) 1409 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:18.673 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:18 smithi123 bash[19119]: audit 2024-02-12T16:15:17.398259+0000 mon.a (mon.0) 1410 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:15:18.673 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:18 smithi123 bash[19119]: audit 2024-02-12T16:15:17.398309+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? v1:172.21.15.121:0/395030306' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2024-02-12T16:15:18.673 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:18 smithi123 bash[19119]: cluster 2024-02-12T16:15:17.398338+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e364: 12 total, 12 up, 12 in 2024-02-12T16:15:18.673 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:18 smithi123 bash[19119]: cluster 2024-02-12T16:15:17.718758+0000 mgr.y (mgr.14146) 899 : cluster [DBG] pgmap v1164: 17 pgs: 17 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-12T16:15:18.673 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:18 smithi123 bash[19119]: audit 2024-02-12T16:15:17.719536+0000 mon.a (mon.0) 1413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-12T16:15:18.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:18.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:18 smithi169 bash[19604]: cluster 2024-02-12T16:15:17.391298+0000 mon.a (mon.0) 1409 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:18 smithi169 bash[19604]: audit 2024-02-12T16:15:17.398259+0000 mon.a (mon.0) 1410 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:15:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:18 smithi169 bash[19604]: audit 2024-02-12T16:15:17.398309+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? v1:172.21.15.121:0/395030306' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2024-02-12T16:15:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:18 smithi169 bash[19604]: cluster 2024-02-12T16:15:17.398338+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e364: 12 total, 12 up, 12 in 2024-02-12T16:15:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:18 smithi169 bash[19604]: cluster 2024-02-12T16:15:17.718758+0000 mgr.y (mgr.14146) 899 : cluster [DBG] pgmap v1164: 17 pgs: 17 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-12T16:15:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:18 smithi169 bash[19604]: audit 2024-02-12T16:15:17.719536+0000 mon.a (mon.0) 1413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-12T16:15:18.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:18.828 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:18 smithi121 bash[18489]: cluster 2024-02-12T16:15:17.391298+0000 mon.a (mon.0) 1409 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:18 smithi121 bash[18489]: audit 2024-02-12T16:15:17.398259+0000 mon.a (mon.0) 1410 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:15:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:18 smithi121 bash[18489]: audit 2024-02-12T16:15:17.398309+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? v1:172.21.15.121:0/395030306' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2024-02-12T16:15:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:18 smithi121 bash[18489]: cluster 2024-02-12T16:15:17.398338+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e364: 12 total, 12 up, 12 in 2024-02-12T16:15:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:18 smithi121 bash[18489]: cluster 2024-02-12T16:15:17.718758+0000 mgr.y (mgr.14146) 899 : cluster [DBG] pgmap v1164: 17 pgs: 17 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-12T16:15:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:18 smithi121 bash[18489]: audit 2024-02-12T16:15:17.719536+0000 mon.a (mon.0) 1413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-12T16:15:18.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:18.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:18.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:18.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:19.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:19.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:19.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:19.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 162622 162606 125.457 68.4375 0.0058854 0.00796929 2024-02-12T16:15:19.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:19.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:19.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:19.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:19.356 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:19.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:19.418 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:19 smithi169 bash[19604]: audit 2024-02-12T16:15:18.407546+0000 mon.a (mon.0) 1414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-12T16:15:19.418 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:19 smithi169 bash[19604]: cluster 2024-02-12T16:15:18.407716+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e365: 12 total, 12 up, 12 in 2024-02-12T16:15:19.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:19.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:19.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:19.496 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:19 smithi123 bash[19119]: audit 2024-02-12T16:15:18.407546+0000 mon.a (mon.0) 1414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-12T16:15:19.496 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:19 smithi123 bash[19119]: cluster 2024-02-12T16:15:18.407716+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e365: 12 total, 12 up, 12 in 2024-02-12T16:15:19.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:19.614 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:19.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:19.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:19.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:19 smithi121 bash[18489]: audit 2024-02-12T16:15:18.407546+0000 mon.a (mon.0) 1414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-12T16:15:19.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:19 smithi121 bash[18489]: cluster 2024-02-12T16:15:18.407716+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e365: 12 total, 12 up, 12 in 2024-02-12T16:15:19.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:19.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:19.855 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:19.878 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:19.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:19.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:20.032 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:20.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:20.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:20.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 163869 163853 124.878 77.9375 0.00574851 0.00800626 2024-02-12T16:15:20.169 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:20.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:20.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:15:20.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:20.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:20.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:20.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:20.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:20.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:20.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:20 smithi121 bash[18489]: cluster 2024-02-12T16:15:19.418114+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e366: 12 total, 12 up, 12 in 2024-02-12T16:15:20.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:20 smithi121 bash[18489]: cluster 2024-02-12T16:15:19.719208+0000 mgr.y (mgr.14146) 900 : cluster [DBG] pgmap v1167: 27 pgs: 10 unknown, 17 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 696 op/s 2024-02-12T16:15:20.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:20 smithi121 bash[18489]: cluster 2024-02-12T16:15:20.390218+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e367: 12 total, 12 up, 12 in 2024-02-12T16:15:20.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:20.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:20.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:20 smithi123 bash[19119]: cluster 2024-02-12T16:15:19.418114+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e366: 12 total, 12 up, 12 in 2024-02-12T16:15:20.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:20 smithi123 bash[19119]: cluster 2024-02-12T16:15:19.719208+0000 mgr.y (mgr.14146) 900 : cluster [DBG] pgmap v1167: 27 pgs: 10 unknown, 17 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 696 op/s 2024-02-12T16:15:20.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:20 smithi123 bash[19119]: cluster 2024-02-12T16:15:20.390218+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e367: 12 total, 12 up, 12 in 2024-02-12T16:15:20.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:20.690 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:20.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:20.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:20 smithi169 bash[19604]: cluster 2024-02-12T16:15:19.418114+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e366: 12 total, 12 up, 12 in 2024-02-12T16:15:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:20 smithi169 bash[19604]: cluster 2024-02-12T16:15:19.719208+0000 mgr.y (mgr.14146) 900 : cluster [DBG] pgmap v1167: 27 pgs: 10 unknown, 17 active+clean; 9.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 696 op/s 2024-02-12T16:15:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:20 smithi169 bash[19604]: cluster 2024-02-12T16:15:20.390218+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e367: 12 total, 12 up, 12 in 2024-02-12T16:15:20.826 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:20.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:20.876 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:15:20.876 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:15:20.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:15:20.876 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:15:20.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.876 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:149161 2.18 2:1f29d795:::benchmark_data_smithi169_42458_object149160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e171)", 2024-02-12T16:15:20.876 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:28.245893+0000", 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 592.62520745899997, 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0325912340000001, 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 149161 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245893+0000", 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245893+0000", 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245892+0000", 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0893e-05 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.245963+0000", 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8885999999999999e-05 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246012+0000", 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9469999999999997e-06 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246019+0000", 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.27310701300000001 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.519126+0000", 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5130000000000002e-06 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.519135+0000", 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.75923179399999996 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.278367+0000", 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011736800000000001 2024-02-12T16:15:20.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.278484+0000", 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0325912340000001 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:149163 2.18 2:18281d7a:::benchmark_data_smithi169_42458_object149162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e171)", 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:28.246025+0000", 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 592.62507541299999, 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0325493379999999, 2024-02-12T16:15:20.881 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 149163 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246025+0000", 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246025+0000", 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246024+0000", 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4032999999999997e-05 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246068+0000", 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5730000000000002e-06 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246076+0000", 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7519999999999996e-06 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.246084+0000", 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.27309360700000002 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.519177+0000", 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9850000000000001e-06 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:28.519187+0000", 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.75932597700000004 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.278513+0000", 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1433000000000006e-05 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:29.278575+0000", 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0325493379999999 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161147 2.10 2:08816aad:::benchmark_data_smithi169_42458_object161146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.384675+0000", 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48642598900005, 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.61898841000000004, 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161147 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384675+0000", 2024-02-12T16:15:20.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384675+0000", 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384674+0000", 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9805000000000003e-05 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384713+0000", 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2243e-05 2024-02-12T16:15:20.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384726+0000", 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9599999999999997e-06 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384732+0000", 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8500000000000002e-05 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384760+0000", 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.805e-06 2024-02-12T16:15:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384764+0000", 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.56615111799999995 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.950915+0000", 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7145000000000006e-05 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.951012+0000", 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113956 2024-02-12T16:15:20.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.951126+0000", 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040148220000000004 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.955141+0000", 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039097393000000001 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.994238+0000", 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076612859999999998 2024-02-12T16:15:20.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.001900+0000", 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7228000000000003e-05 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.001937+0000", 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017262550000000001 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.003663+0000", 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.61898841000000004 2024-02-12T16:15:20.891 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161183 2.10 2:0f370898:::benchmark_data_smithi169_42458_object161182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.384744+0000", 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48635614800003, 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.63015038000000001, 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161183 2024-02-12T16:15:20.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384744+0000", 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384744+0000", 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384743+0000", 2024-02-12T16:15:20.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1025000000000001e-05 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384784+0000", 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2534000000000001e-05 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384797+0000", 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0109999999999997e-06 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384803+0000", 2024-02-12T16:15:20.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9933999999999998e-05 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384833+0000", 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9559999999999999e-06 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384837+0000", 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.56750013899999996 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.952337+0000", 2024-02-12T16:15:20.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9244000000000001e-05 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.952436+0000", 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011724500000000001 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.952553+0000", 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040179459999999997 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.956571+0000", 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037693531000000002 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.994265+0000", 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016953314000000001 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.011218+0000", 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2361999999999996e-05 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.011291+0000", 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036042449999999998 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.014895+0000", 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.63015038000000001 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161209 2.10 2:0f501045:::benchmark_data_smithi169_42458_object161208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.384818+0000", 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48628308100001, 2024-02-12T16:15:20.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.63285047699999997, 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161209 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384818+0000", 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384818+0000", 2024-02-12T16:15:20.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384816+0000", 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8325000000000003e-05 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384855+0000", 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2459999999999999e-05 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384867+0000", 2024-02-12T16:15:20.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1650000000000003e-06 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384873+0000", 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8583999999999999e-05 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384902+0000", 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7220000000000001e-06 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384906+0000", 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.56883154700000005 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.953737+0000", 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2844999999999996e-05 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.953830+0000", 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129054 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.953959+0000", 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040155759999999999 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.957975+0000", 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.056993027000000002 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.014968+0000", 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7805e-05 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.015026+0000", 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1495000000000007e-05 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.015087+0000", 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002580981 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.017668+0000", 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.63285047699999997 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161213 2.10 2:0f4b22ec:::benchmark_data_smithi169_42458_object161212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.384890+0000", 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48621069199999, 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64908517300000002, 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161213 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384890+0000", 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384890+0000", 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384889+0000", 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1170000000000001e-05 2024-02-12T16:15:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384930+0000", 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2857000000000001e-05 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384943+0000", 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9460000000000003e-06 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384949+0000", 2024-02-12T16:15:20.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9689999999999999e-05 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384978+0000", 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.755e-06 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384982+0000", 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.57018207099999996 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.955164+0000", 2024-02-12T16:15:20.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2291000000000001e-05 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.955256+0000", 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118004 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.955374+0000", 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040403319999999998 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.959415+0000", 2024-02-12T16:15:20.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.058376210999999997 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.017791+0000", 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012218992 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.030010+0000", 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106704 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.030117+0000", 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038583950000000001 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.033975+0000", 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64908517300000002 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161281 2.10 2:08b5b4a9:::benchmark_data_smithi169_42458_object161280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.384963+0000", 2024-02-12T16:15:20.911 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48613724799998, 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65186390000000005, 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161281 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384963+0000", 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384963+0000", 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.384962+0000", 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2920999999999998e-05 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385005+0000", 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2877000000000001e-05 2024-02-12T16:15:20.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385018+0000", 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8969999999999996e-06 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385024+0000", 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0032e-05 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385054+0000", 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385058+0000", 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.57153626099999999 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.956594+0000", 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7608000000000005e-05 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.956692+0000", 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011448800000000001 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.956806+0000", 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040106129999999997 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.960817+0000", 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.064876754999999994 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.025693+0000", 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083945040000000005 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.034088+0000", 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107917 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.034196+0000", 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026314189999999999 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.036827+0000", 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65186390000000005 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161285 2.10 2:0d29ba27:::benchmark_data_smithi169_42458_object161284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.385039+0000", 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48606183000004, 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65491149599999998, 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161285 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385039+0000", 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385039+0000", 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385038+0000", 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5232999999999999e-05 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385083+0000", 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.256e-05 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385095+0000", 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9370000000000001e-06 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385101+0000", 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6704e-05 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385128+0000", 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7560000000000001e-06 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385132+0000", 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.57286443799999998 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.957996+0000", 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3863000000000006e-05 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.958090+0000", 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000136131 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.958226+0000", 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0054856460000000003 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.963712+0000", 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.062063083999999998 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.025775+0000", 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011402778000000001 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.037178+0000", 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5217999999999999e-05 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.037233+0000", 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002717279 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.039950+0000", 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65491149599999998 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161288 2.10 2:0f8c9615:::benchmark_data_smithi169_42458_object161287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.385116+0000", 2024-02-12T16:15:20.924 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48598460599999, 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65719401, 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161288 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385116+0000", 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385116+0000", 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385115+0000", 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4082000000000002e-05 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385159+0000", 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568000000000001e-05 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385172+0000", 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9370000000000001e-06 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385178+0000", 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9838000000000001e-05 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385207+0000", 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7270000000000001e-06 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385211+0000", 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.57422399999999996 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.959435+0000", 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7153999999999998e-05 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.959532+0000", 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011203499999999999 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.959644+0000", 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010117222 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.969762+0000", 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.056053560000000002 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.025815+0000", 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014248718 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.040064+0000", 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7253999999999999e-05 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.040111+0000", 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021989919999999999 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.042310+0000", 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65719401 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161324 2.10 2:0c9e0b13:::benchmark_data_smithi169_42458_object161323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.385192+0000", 2024-02-12T16:15:20.931 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48590831599995, 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65888611200000002, 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161324 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385192+0000", 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385192+0000", 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385191+0000", 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2760999999999999e-05 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385234+0000", 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2652e-05 2024-02-12T16:15:20.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385247+0000", 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0739999999999998e-06 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385253+0000", 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8240000000000001e-05 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385281+0000", 2024-02-12T16:15:20.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7730000000000001e-06 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385285+0000", 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.57558753900000004 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.960872+0000", 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000150097 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.961022+0000", 2024-02-12T16:15:20.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011684900000000001 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.961139+0000", 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086320489999999993 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.969771+0000", 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.067155803 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.036927+0000", 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0054698819999999997 2024-02-12T16:15:20.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.042397+0000", 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9522000000000003e-05 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.042466+0000", 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001611971 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.044078+0000", 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.65888611200000002 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:161361 2.10 2:090aea0e:::benchmark_data_smithi169_42458_object161360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e176)", 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:05:38.385266+0000", 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 582.48583504400005, 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.66020809899999999, 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.938 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161361 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385266+0000", 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385266+0000", 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385264+0000", 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7676000000000002e-05 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385312+0000", 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2952000000000001e-05 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385325+0000", 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1879999999999999e-06 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385331+0000", 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6633999999999999e-05 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385358+0000", 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8249999999999998e-06 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.385362+0000", 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.57692722699999999 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.962289+0000", 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4091000000000004e-05 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.962383+0000", 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116544 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.962500+0000", 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0073834699999999996 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:38.969883+0000", 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.067115912 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.036999+0000", 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007141672 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.044141+0000", 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1254000000000002e-05 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.044182+0000", 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012917829999999999 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:05:39.045474+0000", 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.66020809899999999 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.944 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:192863 2.10 2:0cd9162f:::benchmark_data_smithi169_42458_object19829:head [delete] snapc 0=[] ondisk+write+known_if_redirected e187)", 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:04.381058+0000", 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 556.49004288599997, 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.63660514800000001, 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 192863 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381058+0000", 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381058+0000", 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381055+0000", 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5559999999999996e-06 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381065+0000", 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3312999999999999e-05 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381078+0000", 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7259999999999996e-06 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381085+0000", 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6303999999999997e-05 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381121+0000", 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049562 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for degraded object", 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.381617+0000", 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024069113999999999 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.405686+0000", 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3207000000000001e-05 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.405739+0000", 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00044413500000000002 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,8", 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.406183+0000", 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.60907958100000004 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.015263+0000", 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4599999999999997e-06 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.015270+0000", 2024-02-12T16:15:20.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0241999999999999e-05 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.015300+0000", 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4056000000000001e-05 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.015334+0000", 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023284730000000002 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.017663+0000", 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.63660514800000001 2024-02-12T16:15:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.14811.0:192898 2.8 2:14c3870d:::benchmark_data_smithi169_42458_object19864:head [delete] snapc 0=[] ondisk+write+known_if_redirected e187)", 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:06:04.411565+0000", 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 556.45953546600003, 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.62301512699999995, 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.14811", 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:20.951 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 192898 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.411565+0000", 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.411565+0000", 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.411563+0000", 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3900000000000004e-06 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:04.411570+0000", 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.596082628 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.007653+0000", 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3312999999999999e-05 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.007676+0000", 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077909260000000001 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.015467+0000", 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011085400000000001 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.015578+0000", 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034203340000000001 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.018998+0000", 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012225061000000001 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.031223+0000", 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032977369999999998 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.034521+0000", 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2935999999999998e-05 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.034554+0000", 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6234999999999999e-05 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:06:05.034580+0000", 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.62301512699999995 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:161430 4.0 e364/334 4:0eaf0600:::benchmark_data_smithi169_60938_object161429:head v 364'10150, mlcod=364'10146)", 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.094872+0000", 2024-02-12T16:15:20.956 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 2.7762282819999999, 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977106, 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.094872+0000", 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.094872+0000", 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.094871+0000", 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8719000000000004e-05 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.094940+0000", 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2924e-05 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.094952+0000", 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.109e-06 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.094960+0000", 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6019999999999997e-05 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.094996+0000", 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1415000000000001e-05 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.095017+0000", 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021789329 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.116806+0000", 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3120000000000001e-05 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.116849+0000", 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977106 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:161437 4.8 e364/334 4:1dfee004:::benchmark_data_smithi169_60938_object161436:head v 364'10037, mlcod=364'10034)", 2024-02-12T16:15:20.960 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.096936+0000", 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 2.7741642459999998, 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019949354999999998, 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.096936+0000", 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.096936+0000", 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.096935+0000", 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4041000000000001e-05 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.096979+0000", 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4657999999999999e-05 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.096994+0000", 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.998e-06 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.097000+0000", 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026316299999999998 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.099631+0000", 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9561000000000001e-05 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.099651+0000", 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017200877 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.116852+0000", 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4073999999999999e-05 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.116886+0000", 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019949354999999998 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:162825 4.0 e365/334 4:0774136a:::benchmark_data_smithi169_60938_object162824:head v 365'10248, mlcod=365'10244)", 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:19.331128+0000", 2024-02-12T16:15:20.964 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.539972527, 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023564531999999999, 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331128+0000", 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331128+0000", 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:15:20.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331126+0000", 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4190000000000002e-05 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331170+0000", 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5585000000000001e-05 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331186+0000", 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9149999999999999e-06 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331193+0000", 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7530000000000001e-05 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331240+0000", 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0613000000000001e-05 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.331261+0000", 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023390945 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.354652+0000", 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0924999999999999e-05 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:19.354693+0000", 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023564531999999999 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:164011 4.0 e366/334 4:065e95e3:::benchmark_data_smithi169_60938_object164010:head v 366'10321, mlcod=366'10319)", 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:20.280849+0000", 2024-02-12T16:15:20.968 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.59025170800000004, 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021567732999999999, 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.280849+0000", 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.280849+0000", 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.99998 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.280829+0000", 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0386e-05 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.280899+0000", 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6125000000000001e-05 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.280926+0000", 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9880000000000004e-06 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.280931+0000", 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019611329999999999 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.282892+0000", 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0517999999999999e-05 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.282912+0000", 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019464555000000001 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.302377+0000", 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9889999999999999e-05 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.302417+0000", 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021567732999999999 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.972 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:164469 4.0 e367/334 4:06d07bca:::benchmark_data_smithi169_60938_object164468:head v 367'10352, mlcod=367'10349)", 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:20.668498+0000", 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.20260257600000001, 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025356930999999999, 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.668498+0000", 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.668498+0000", 2024-02-12T16:15:20.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.668494+0000", 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8258999999999996e-05 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.668582+0000", 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2167e-05 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.668604+0000", 2024-02-12T16:15:20.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0198e-05 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.668615+0000", 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025554509999999998 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.671170+0000", 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1878e-05 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.671192+0000", 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022587757 2024-02-12T16:15:20.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.693780+0000", 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5234999999999994e-05 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.693855+0000", 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025356930999999999 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.976 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:164470 4.0 e367/334 4:02c1f688:::benchmark_data_smithi169_60938_object164469:head v 367'10353, mlcod=367'10350)", 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:20.672117+0000", 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.19898400399999999, 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021837118999999999, 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.672117+0000", 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.672117+0000", 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:15:20.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.672115+0000", 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8890000000000002e-05 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.672154+0000", 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4815e-05 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.672169+0000", 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2729999999999999e-06 2024-02-12T16:15:20.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.672175+0000", 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084392449999999997 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.680614+0000", 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1577e-05 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.680636+0000", 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013223204000000001 2024-02-12T16:15:20.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.693859+0000", 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4940999999999998e-05 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.693954+0000", 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021837118999999999 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.980 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:164599 4.8 e367/334 4:1a1aeb50:::benchmark_data_smithi169_60938_object164598:head v 367'10230, mlcod=367'10228)", 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:20.785462+0000", 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.085638166000000002, 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020065868000000001, 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785462+0000", 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785462+0000", 2024-02-12T16:15:20.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785462+0000", 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2725999999999995e-05 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785524+0000", 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1489999999999999e-06 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785530+0000", 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3529999999999997e-06 2024-02-12T16:15:20.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785535+0000", 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7361000000000002e-05 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785592+0000", 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0302999999999999e-05 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.785613+0000", 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019871851999999999 2024-02-12T16:15:20.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.805484+0000", 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3985e-05 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.805528+0000", 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020065868000000001 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:15:20.984 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:15:20.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:21.008 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:21.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:15:21.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:21.135 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:21.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 165072 165056 124.279 75.1875 0.015428 0.00804487 2024-02-12T16:15:21.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:21.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:21.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:21.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:21.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:21.690 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:21.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-12T16:15:21.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:21.928 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:21.956 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:21.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:22.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:22.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:22.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:22.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 166283 166267 123.7 75.6875 0.0151706 0.00808268 2024-02-12T16:15:22.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:22.233 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:22.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:22.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:22.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:22.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:22.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:22.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:22.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:22.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:22.624 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:22.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:22.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:22.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:22.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:22.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:22.880 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:22.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:22.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:22.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:22 smithi121 bash[18489]: cluster 2024-02-12T16:15:21.719556+0000 mgr.y (mgr.14146) 901 : cluster [DBG] pgmap v1169: 27 pgs: 10 unknown, 17 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 953 op/s 2024-02-12T16:15:23.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:22 smithi169 bash[19604]: cluster 2024-02-12T16:15:21.719556+0000 mgr.y (mgr.14146) 901 : cluster [DBG] pgmap v1169: 27 pgs: 10 unknown, 17 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 953 op/s 2024-02-12T16:15:23.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:22 smithi123 bash[19119]: cluster 2024-02-12T16:15:21.719556+0000 mgr.y (mgr.14146) 901 : cluster [DBG] pgmap v1169: 27 pgs: 10 unknown, 17 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 953 op/s 2024-02-12T16:15:23.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:23.162 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 167549 167533 123.176 79.125 0.00482689 0.00811636 2024-02-12T16:15:23.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:23.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:23.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:23.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:23.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:23.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:23.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:23.679 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:23.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:23.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:23.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:23.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:23.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:23.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:24.038 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:15:24.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:15:24.039 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-12T16:15:24.039 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:15:24.039 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:15:24.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-12T16:15:24.046 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:23 smithi121 bash[18489]: audit 2024-02-12T16:15:23.720583+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-12T16:15:24.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:23 smithi169 bash[19604]: audit 2024-02-12T16:15:23.720583+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-12T16:15:24.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:24.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:23 smithi123 bash[19119]: audit 2024-02-12T16:15:23.720583+0000 mon.a (mon.0) 1418 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-12T16:15:24.106 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:24.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:24.163 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 168779 168763 122.637 76.875 0.0287224 0.00815237 2024-02-12T16:15:24.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:24.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:24.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:24.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:24.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:24.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:24.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:24.520 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:24.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:24.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:24.654 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:24.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:24.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:24.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:24 smithi123 bash[19119]: cluster 2024-02-12T16:15:23.719943+0000 mgr.y (mgr.14146) 902 : cluster [DBG] pgmap v1170: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 999 op/s 2024-02-12T16:15:24.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:24 smithi123 bash[19119]: audit 2024-02-12T16:15:23.783931+0000 mon.a (mon.0) 1419 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-12T16:15:24.797 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:24 smithi123 bash[19119]: cluster 2024-02-12T16:15:23.784012+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e368: 12 total, 12 up, 12 in 2024-02-12T16:15:24.803 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:24.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:24.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:24.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:24.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:25.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:24 smithi169 bash[19604]: cluster 2024-02-12T16:15:23.719943+0000 mgr.y (mgr.14146) 902 : cluster [DBG] pgmap v1170: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 999 op/s 2024-02-12T16:15:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:24 smithi169 bash[19604]: audit 2024-02-12T16:15:23.783931+0000 mon.a (mon.0) 1419 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-12T16:15:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:24 smithi169 bash[19604]: cluster 2024-02-12T16:15:23.784012+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e368: 12 total, 12 up, 12 in 2024-02-12T16:15:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:25.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:25.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:25.163 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 169918 169902 122.046 71.1875 0.00621397 0.00819183 2024-02-12T16:15:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:24 smithi121 bash[18489]: cluster 2024-02-12T16:15:23.719943+0000 mgr.y (mgr.14146) 902 : cluster [DBG] pgmap v1170: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 999 op/s 2024-02-12T16:15:25.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:24 smithi121 bash[18489]: audit 2024-02-12T16:15:23.783931+0000 mon.a (mon.0) 1419 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-12T16:15:25.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:24 smithi121 bash[18489]: cluster 2024-02-12T16:15:23.784012+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e368: 12 total, 12 up, 12 in 2024-02-12T16:15:25.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:25.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:25.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:25.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:25.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:25.358 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:25.459 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:25.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:25.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:25.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:25.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:25.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:25.722 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:25.748 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:25.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:25.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:25.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:25.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:25 smithi121 bash[18489]: cluster 2024-02-12T16:15:24.786616+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e369: 12 total, 12 up, 12 in 2024-02-12T16:15:25.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.019 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:26.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:26.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:26.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:25 smithi169 bash[19604]: cluster 2024-02-12T16:15:24.786616+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e369: 12 total, 12 up, 12 in 2024-02-12T16:15:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:25 smithi123 bash[19119]: cluster 2024-02-12T16:15:24.786616+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e369: 12 total, 12 up, 12 in 2024-02-12T16:15:26.150 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:26.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:26.163 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 171134 171118 121.523 76 0.00682925 0.00822737 2024-02-12T16:15:26.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:26.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:26.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.411 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:26.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:26.509 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:15:26.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.551 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:26.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:26.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:26.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:26.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:26.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:26.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:26.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:26.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:26 smithi169 bash[19604]: cluster 2024-02-12T16:15:25.720460+0000 mgr.y (mgr.14146) 903 : cluster [DBG] pgmap v1173: 27 pgs: 1 peering, 26 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:15:27.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:27.108 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:26 smithi123 bash[19119]: cluster 2024-02-12T16:15:25.720460+0000 mgr.y (mgr.14146) 903 : cluster [DBG] pgmap v1173: 27 pgs: 1 peering, 26 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:15:27.124 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:26 smithi121 bash[18489]: cluster 2024-02-12T16:15:25.720460+0000 mgr.y (mgr.14146) 903 : cluster [DBG] pgmap v1173: 27 pgs: 1 peering, 26 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:15:27.163 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 172100 172084 120.835 60.375 0.00883557 0.0082738 2024-02-12T16:15:27.232 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:27.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:27.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:27.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:27.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:27.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:27.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:27.663 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:27.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:27.791 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:27.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:27.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:27.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:28.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:28.054 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:28.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:28.108 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:28.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:28.163 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 90 16 173334 173318 120.35 77.125 0.022153 0.00830665 2024-02-12T16:15:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:28.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:28.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:28.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:28.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:28.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:28.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:28.560 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:28.597 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:28.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:28.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:28.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:28.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:28.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:28.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:28.873 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:28.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:28.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:29.015 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:29.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:29.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:28 smithi169 bash[19604]: cluster 2024-02-12T16:15:27.720892+0000 mgr.y (mgr.14146) 904 : cluster [DBG] pgmap v1174: 27 pgs: 1 peering, 26 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-12T16:15:29.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:28 smithi123 bash[19119]: cluster 2024-02-12T16:15:27.720892+0000 mgr.y (mgr.14146) 904 : cluster [DBG] pgmap v1174: 27 pgs: 1 peering, 26 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-12T16:15:29.152 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:29.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:28 smithi121 bash[18489]: cluster 2024-02-12T16:15:27.720892+0000 mgr.y (mgr.14146) 904 : cluster [DBG] pgmap v1174: 27 pgs: 1 peering, 26 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-02-12T16:15:29.163 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.012447 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 173334 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 120.354 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 63.5575 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 383.188 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 60.375 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 1925 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 1016 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 6131 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 966 2024-02-12T16:15:29.164 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.00830797 2024-02-12T16:15:29.165 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.0126091 2024-02-12T16:15:29.165 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 0.735335 2024-02-12T16:15:29.165 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.0010598 2024-02-12T16:15:29.175 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:15:29.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:29.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:29.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:29.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:29.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:29.555 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:29.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:29.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:29.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:29.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:29.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:29.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:29.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:29.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:30.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:30.059 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:29 smithi123 bash[19119]: cluster 2024-02-12T16:15:28.261082+0000 osd.8 (osd.8) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:30.059 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:29 smithi123 bash[19119]: cluster 2024-02-12T16:15:28.261162+0000 osd.8 (osd.8) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:30.059 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:29 smithi123 bash[19119]: cluster 2024-02-12T16:15:28.807738+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e370: 12 total, 12 up, 12 in 2024-02-12T16:15:30.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:30.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:29 smithi121 bash[18489]: cluster 2024-02-12T16:15:28.261082+0000 osd.8 (osd.8) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:29 smithi121 bash[18489]: cluster 2024-02-12T16:15:28.261162+0000 osd.8 (osd.8) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:29 smithi121 bash[18489]: cluster 2024-02-12T16:15:28.807738+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e370: 12 total, 12 up, 12 in 2024-02-12T16:15:30.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:30.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:30.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:29 smithi169 bash[19604]: cluster 2024-02-12T16:15:28.261082+0000 osd.8 (osd.8) 39 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:29 smithi169 bash[19604]: cluster 2024-02-12T16:15:28.261162+0000 osd.8 (osd.8) 40 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:29 smithi169 bash[19604]: cluster 2024-02-12T16:15:28.807738+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e370: 12 total, 12 up, 12 in 2024-02-12T16:15:30.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:30.356 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:30.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:30.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:30.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:30.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:30.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:30.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:30.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:30.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:30.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:30.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:30.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:30.896 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:30.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:30.987 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:30 smithi169 bash[19604]: cluster 2024-02-12T16:15:29.721241+0000 mgr.y (mgr.14146) 905 : cluster [DBG] pgmap v1176: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-02-12T16:15:30.987 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:30 smithi169 bash[19604]: cluster 2024-02-12T16:15:30.392475+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e371: 12 total, 12 up, 12 in 2024-02-12T16:15:31.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:31.029 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:31.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:31.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:30 smithi123 bash[19119]: cluster 2024-02-12T16:15:29.721241+0000 mgr.y (mgr.14146) 905 : cluster [DBG] pgmap v1176: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-02-12T16:15:31.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:30 smithi123 bash[19119]: cluster 2024-02-12T16:15:30.392475+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e371: 12 total, 12 up, 12 in 2024-02-12T16:15:31.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:30 smithi121 bash[18489]: cluster 2024-02-12T16:15:29.721241+0000 mgr.y (mgr.14146) 905 : cluster [DBG] pgmap v1176: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-02-12T16:15:31.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:30 smithi121 bash[18489]: cluster 2024-02-12T16:15:30.392475+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e371: 12 total, 12 up, 12 in 2024-02-12T16:15:31.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:31.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:31.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:31.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:31.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:31.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:31.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:31.690 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:15:31.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:31.716 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:15:31.738 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:15:31.763 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:15:31.790 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:15:31.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:31.824 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:15:31.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:31.852 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:15:31.881 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:15:31.908 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:15:31.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:31.940 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:15:31.941 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:31 smithi121 bash[18489]: audit 2024-02-12T16:15:31.651676+0000 mon.a (mon.0) 1424 : audit [DBG] from='client.? v1:172.21.15.121:0/201408943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:31.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:15:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:31.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:31.998 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:15:32.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:32.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:32.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:32.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:31 smithi123 bash[19119]: audit 2024-02-12T16:15:31.651676+0000 mon.a (mon.0) 1424 : audit [DBG] from='client.? v1:172.21.15.121:0/201408943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:32.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:32.257 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:32.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:32.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:31 smithi169 bash[19604]: audit 2024-02-12T16:15:31.651676+0000 mon.a (mon.0) 1424 : audit [DBG] from='client.? v1:172.21.15.121:0/201408943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:32.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:32.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:32.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:32.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:32.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:32.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:32.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:32.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:32.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:32.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:32.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:32.861 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:15:32.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:32.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:32.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:33.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:33.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:32 smithi169 bash[19604]: cluster 2024-02-12T16:15:31.721639+0000 mgr.y (mgr.14146) 906 : cluster [DBG] pgmap v1178: 27 pgs: 27 active+clean; 11 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s rd, 71 MiB/s wr, 1.21k op/s 2024-02-12T16:15:33.062 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:32 smithi169 bash[19604]: audit 2024-02-12T16:15:32.420221+0000 mon.a (mon.0) 1425 : audit [INF] from='client.? v1:172.21.15.121:0/2089465089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:15:33.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:33.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:33.109 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:15:33.109 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:15:33.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:32 smithi123 bash[19119]: cluster 2024-02-12T16:15:31.721639+0000 mgr.y (mgr.14146) 906 : cluster [DBG] pgmap v1178: 27 pgs: 27 active+clean; 11 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s rd, 71 MiB/s wr, 1.21k op/s 2024-02-12T16:15:33.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:32 smithi123 bash[19119]: audit 2024-02-12T16:15:32.420221+0000 mon.a (mon.0) 1425 : audit [INF] from='client.? v1:172.21.15.121:0/2089465089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:15:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:32 smithi121 bash[18489]: cluster 2024-02-12T16:15:31.721639+0000 mgr.y (mgr.14146) 906 : cluster [DBG] pgmap v1178: 27 pgs: 27 active+clean; 11 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s rd, 71 MiB/s wr, 1.21k op/s 2024-02-12T16:15:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:32 smithi121 bash[18489]: audit 2024-02-12T16:15:32.420221+0000 mon.a (mon.0) 1425 : audit [INF] from='client.? v1:172.21.15.121:0/2089465089' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:15:33.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:33.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:33.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:33.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:33.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:33.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:33.513 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:33.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:33.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:33.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:33.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:33.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:33.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:33.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:33.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:33.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:33.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:33.965 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:33 smithi169 bash[19604]: cluster 2024-02-12T16:15:32.829431+0000 mon.a (mon.0) 1426 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:33.966 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:33 smithi169 bash[19604]: audit 2024-02-12T16:15:32.845066+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? v1:172.21.15.121:0/2089465089' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:15:33.966 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:33 smithi169 bash[19604]: cluster 2024-02-12T16:15:32.845140+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e372: 12 total, 12 up, 12 in 2024-02-12T16:15:33.966 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:33 smithi169 bash[19604]: cluster 2024-02-12T16:15:32.979602+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.a scrub starts 2024-02-12T16:15:33.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-12T16:15:34.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:34.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:34.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:33 smithi123 bash[19119]: cluster 2024-02-12T16:15:32.829431+0000 mon.a (mon.0) 1426 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:33 smithi123 bash[19119]: audit 2024-02-12T16:15:32.845066+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? v1:172.21.15.121:0/2089465089' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:15:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:33 smithi123 bash[19119]: cluster 2024-02-12T16:15:32.845140+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e372: 12 total, 12 up, 12 in 2024-02-12T16:15:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:33 smithi123 bash[19119]: cluster 2024-02-12T16:15:32.979602+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.a scrub starts 2024-02-12T16:15:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:33 smithi121 bash[18489]: cluster 2024-02-12T16:15:32.829431+0000 mon.a (mon.0) 1426 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:15:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:33 smithi121 bash[18489]: audit 2024-02-12T16:15:32.845066+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? v1:172.21.15.121:0/2089465089' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:15:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:33 smithi121 bash[18489]: cluster 2024-02-12T16:15:32.845140+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e372: 12 total, 12 up, 12 in 2024-02-12T16:15:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:33 smithi121 bash[18489]: cluster 2024-02-12T16:15:32.979602+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.a scrub starts 2024-02-12T16:15:34.163 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:34.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:34.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:34.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:34.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:34.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:34.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:34.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:34.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:34.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:34.604 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:34.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:34.634 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:15:34.634 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "" 2024-02-12T16:15:34.634 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:15:34.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:34.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:34.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:34.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:34.871 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:34.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:34.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:35.014 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:35.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:35.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:34 smithi123 bash[19119]: cluster 2024-02-12T16:15:33.112186+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:15:35.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:34 smithi123 bash[19119]: cluster 2024-02-12T16:15:33.722021+0000 mgr.y (mgr.14146) 907 : cluster [DBG] pgmap v1180: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 32 MiB/s wr, 774 op/s 2024-02-12T16:15:35.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:34 smithi123 bash[19119]: cluster 2024-02-12T16:15:33.750200+0000 osd.10 (osd.10) 31 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:15:35.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:34 smithi123 bash[19119]: cluster 2024-02-12T16:15:33.775677+0000 osd.10 (osd.10) 32 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:15:35.148 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:34 smithi121 bash[18489]: cluster 2024-02-12T16:15:33.112186+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:15:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:34 smithi121 bash[18489]: cluster 2024-02-12T16:15:33.722021+0000 mgr.y (mgr.14146) 907 : cluster [DBG] pgmap v1180: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 32 MiB/s wr, 774 op/s 2024-02-12T16:15:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:34 smithi121 bash[18489]: cluster 2024-02-12T16:15:33.750200+0000 osd.10 (osd.10) 31 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:15:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:34 smithi121 bash[18489]: cluster 2024-02-12T16:15:33.775677+0000 osd.10 (osd.10) 32 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:15:35.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:35.287 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:34 smithi169 bash[19604]: cluster 2024-02-12T16:15:33.112186+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:15:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:34 smithi169 bash[19604]: cluster 2024-02-12T16:15:33.722021+0000 mgr.y (mgr.14146) 907 : cluster [DBG] pgmap v1180: 27 pgs: 27 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 32 MiB/s wr, 774 op/s 2024-02-12T16:15:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:34 smithi169 bash[19604]: cluster 2024-02-12T16:15:33.750200+0000 osd.10 (osd.10) 31 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:15:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:34 smithi169 bash[19604]: cluster 2024-02-12T16:15:33.775677+0000 osd.10 (osd.10) 32 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:15:35.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:15:35.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:35.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:35.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:35.560 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:35.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:35.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:35.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:35.820 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:35.923 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:35.952 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:35.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:36.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:35 smithi169 bash[19604]: cluster 2024-02-12T16:15:34.653306+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:35 smithi169 bash[19604]: cluster 2024-02-12T16:15:34.653368+0000 osd.5 (osd.5) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:35 smithi169 bash[19604]: cluster 2024-02-12T16:15:34.744546+0000 osd.10 (osd.10) 33 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:15:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:35 smithi169 bash[19604]: cluster 2024-02-12T16:15:34.766969+0000 osd.10 (osd.10) 34 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:15:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:35 smithi169 bash[19604]: cluster 2024-02-12T16:15:34.846662+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e373: 12 total, 12 up, 12 in 2024-02-12T16:15:36.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:36.085 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:36.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:36.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:35 smithi123 bash[19119]: cluster 2024-02-12T16:15:34.653306+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:35 smithi123 bash[19119]: cluster 2024-02-12T16:15:34.653368+0000 osd.5 (osd.5) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:35 smithi123 bash[19119]: cluster 2024-02-12T16:15:34.744546+0000 osd.10 (osd.10) 33 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:15:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:35 smithi123 bash[19119]: cluster 2024-02-12T16:15:34.766969+0000 osd.10 (osd.10) 34 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:15:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:35 smithi123 bash[19119]: cluster 2024-02-12T16:15:34.846662+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e373: 12 total, 12 up, 12 in 2024-02-12T16:15:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:35 smithi121 bash[18489]: cluster 2024-02-12T16:15:34.653306+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:35 smithi121 bash[18489]: cluster 2024-02-12T16:15:34.653368+0000 osd.5 (osd.5) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:35 smithi121 bash[18489]: cluster 2024-02-12T16:15:34.744546+0000 osd.10 (osd.10) 33 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:15:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:35 smithi121 bash[18489]: cluster 2024-02-12T16:15:34.766969+0000 osd.10 (osd.10) 34 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:15:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:35 smithi121 bash[18489]: cluster 2024-02-12T16:15:34.846662+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e373: 12 total, 12 up, 12 in 2024-02-12T16:15:36.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:36.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:15:36.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:36.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:36.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:36.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:36.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:36.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:36.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:36.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:36.648 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:36.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:36.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:36.784 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:36.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:36.859 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:35.423570+0000 osd.4 (osd.4) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:36.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:36.913 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:36.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:37.016 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:37.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:35.423570+0000 osd.4 (osd.4) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:37.024 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:35.423621+0000 osd.4 (osd.4) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:37.024 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:35.722454+0000 mgr.y (mgr.14146) 908 : cluster [DBG] pgmap v1182: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 26 MiB/s wr, 1.32k op/s 2024-02-12T16:15:37.024 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:35.786503+0000 osd.10 (osd.10) 35 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:15:37.024 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:35.815224+0000 osd.10 (osd.10) 36 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:15:37.024 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:35.950628+0000 osd.6 (osd.6) 42 : cluster [DBG] 4.a scrub ok 2024-02-12T16:15:37.024 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.363585+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.b scrub starts 2024-02-12T16:15:37.024 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:36 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.409363+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e374: 12 total, 12 up, 12 in 2024-02-12T16:15:37.040 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:37.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:35.423570+0000 osd.4 (osd.4) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:35.423621+0000 osd.4 (osd.4) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:35.722454+0000 mgr.y (mgr.14146) 908 : cluster [DBG] pgmap v1182: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 26 MiB/s wr, 1.32k op/s 2024-02-12T16:15:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:35.786503+0000 osd.10 (osd.10) 35 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:15:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:35.815224+0000 osd.10 (osd.10) 36 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:15:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:35.950628+0000 osd.6 (osd.6) 42 : cluster [DBG] 4.a scrub ok 2024-02-12T16:15:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.363585+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.b scrub starts 2024-02-12T16:15:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:36 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.409363+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e374: 12 total, 12 up, 12 in 2024-02-12T16:15:37.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:37.150 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:35.423621+0000 osd.4 (osd.4) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:37.151 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:35.722454+0000 mgr.y (mgr.14146) 908 : cluster [DBG] pgmap v1182: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 26 MiB/s wr, 1.32k op/s 2024-02-12T16:15:37.151 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:35.786503+0000 osd.10 (osd.10) 35 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:15:37.151 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:35.815224+0000 osd.10 (osd.10) 36 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:15:37.151 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:35.950628+0000 osd.6 (osd.6) 42 : cluster [DBG] 4.a scrub ok 2024-02-12T16:15:37.151 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.363585+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.b scrub starts 2024-02-12T16:15:37.151 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:36 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.409363+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e374: 12 total, 12 up, 12 in 2024-02-12T16:15:37.176 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:37.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:37.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:37.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:37.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:37.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:37.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:37.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:37.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:37.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:37.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:37.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:37.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:37.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:37.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:37.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:37.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:38.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:38.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:38.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:38.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.163976+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:15:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.300882+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.300945+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.575319+0000 osd.7 (osd.7) 61 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:15:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.644615+0000 osd.7 (osd.7) 62 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:15:38.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.658992+0000 osd.11 (osd.11) 46 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:15:38.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.732247+0000 osd.5 (osd.5) 51 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:15:38.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.779596+0000 osd.5 (osd.5) 52 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:15:38.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:38 smithi121 bash[18489]: cluster 2024-02-12T16:15:37.062929+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:15:38.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:38.279 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:38.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:38.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.163976+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.300882+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.300945+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.575319+0000 osd.7 (osd.7) 61 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.644615+0000 osd.7 (osd.7) 62 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.658992+0000 osd.11 (osd.11) 46 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.732247+0000 osd.5 (osd.5) 51 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.779596+0000 osd.5 (osd.5) 52 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:15:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:38 smithi123 bash[19119]: cluster 2024-02-12T16:15:37.062929+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:15:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:38.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.163976+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.300882+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.300945+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.575319+0000 osd.7 (osd.7) 61 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.644615+0000 osd.7 (osd.7) 62 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.658992+0000 osd.11 (osd.11) 46 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.732247+0000 osd.5 (osd.5) 51 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.779596+0000 osd.5 (osd.5) 52 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:15:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:38 smithi169 bash[19604]: cluster 2024-02-12T16:15:37.062929+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:15:38.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:38.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:38.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:38.778 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:15:38.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:38.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:38.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:38.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:38.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:39.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:39 smithi121 bash[18489]: cluster 2024-02-12T16:15:36.999447+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.c scrub starts 2024-02-12T16:15:39.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:39 smithi121 bash[18489]: cluster 2024-02-12T16:15:37.722910+0000 mgr.y (mgr.14146) 909 : cluster [DBG] pgmap v1184: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s wr, 1.19k op/s 2024-02-12T16:15:39.072 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:39.103 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:39.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:39.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:39.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:39.361 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:39 smithi169 bash[19604]: cluster 2024-02-12T16:15:36.999447+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.c scrub starts 2024-02-12T16:15:39.361 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:39 smithi169 bash[19604]: cluster 2024-02-12T16:15:37.722910+0000 mgr.y (mgr.14146) 909 : cluster [DBG] pgmap v1184: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s wr, 1.19k op/s 2024-02-12T16:15:39.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:39.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:39 smithi123 bash[19119]: cluster 2024-02-12T16:15:36.999447+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.c scrub starts 2024-02-12T16:15:39.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:39 smithi123 bash[19119]: cluster 2024-02-12T16:15:37.722910+0000 mgr.y (mgr.14146) 909 : cluster [DBG] pgmap v1184: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s wr, 1.19k op/s 2024-02-12T16:15:39.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:39.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:39.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:39.643 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:39.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:39.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:39.879 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:39.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:39.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:40.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:40.048 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:40.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:40.080 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:40 smithi123 bash[19119]: cluster 2024-02-12T16:15:38.879062+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.b scrub ok 2024-02-12T16:15:40.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:40.182 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:40.186 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:40.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:40.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:40.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:40.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:40 smithi169 bash[19604]: cluster 2024-02-12T16:15:38.879062+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.b scrub ok 2024-02-12T16:15:40.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:40 smithi169 bash[19604]: cluster 2024-02-12T16:15:39.781149+0000 osd.9 (osd.9) 39 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:15:40.331 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:40.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:40.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:40 smithi123 bash[19119]: cluster 2024-02-12T16:15:39.781149+0000 osd.9 (osd.9) 39 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:15:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:40 smithi121 bash[18489]: cluster 2024-02-12T16:15:38.879062+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.b scrub ok 2024-02-12T16:15:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:40 smithi121 bash[18489]: cluster 2024-02-12T16:15:39.781149+0000 osd.9 (osd.9) 39 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:15:40.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:40.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:40.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:40.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:40.593 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:40.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:40.669 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:15:40.669 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:15:40.669 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-12T16:15:40.669 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:15:40.669 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:15:40.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-12T16:15:40.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:40.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:40.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:40.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:40.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:40.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:40.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:41.001 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:41 smithi121 bash[18489]: cluster 2024-02-12T16:15:39.442728+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:15:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:41 smithi121 bash[18489]: cluster 2024-02-12T16:15:39.577594+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:15:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:41 smithi121 bash[18489]: cluster 2024-02-12T16:15:39.723336+0000 mgr.y (mgr.14146) 910 : cluster [DBG] pgmap v1185: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-12T16:15:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:41 smithi121 bash[18489]: cluster 2024-02-12T16:15:40.397932+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e375: 12 total, 12 up, 12 in 2024-02-12T16:15:41.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:41.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:41 smithi169 bash[19604]: cluster 2024-02-12T16:15:39.442728+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:15:41.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:41 smithi169 bash[19604]: cluster 2024-02-12T16:15:39.577594+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:15:41.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:41 smithi169 bash[19604]: cluster 2024-02-12T16:15:39.723336+0000 mgr.y (mgr.14146) 910 : cluster [DBG] pgmap v1185: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-12T16:15:41.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:41 smithi169 bash[19604]: cluster 2024-02-12T16:15:40.397932+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e375: 12 total, 12 up, 12 in 2024-02-12T16:15:41.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:41 smithi169 bash[19604]: cluster 2024-02-12T16:15:40.640274+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.c scrub ok 2024-02-12T16:15:41.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:41.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:41.254 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:41 smithi123 bash[19119]: cluster 2024-02-12T16:15:39.442728+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:15:41.254 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:41 smithi123 bash[19119]: cluster 2024-02-12T16:15:39.577594+0000 osd.1 (osd.1) 53 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:15:41.254 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:41 smithi123 bash[19119]: cluster 2024-02-12T16:15:39.723336+0000 mgr.y (mgr.14146) 910 : cluster [DBG] pgmap v1185: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-12T16:15:41.254 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:41 smithi123 bash[19119]: cluster 2024-02-12T16:15:40.397932+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e375: 12 total, 12 up, 12 in 2024-02-12T16:15:41.254 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:41 smithi123 bash[19119]: cluster 2024-02-12T16:15:40.640274+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.c scrub ok 2024-02-12T16:15:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:41.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:41.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:41.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:41.551 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:41.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:41.684 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:41.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:41.815 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:41 smithi121 bash[18489]: cluster 2024-02-12T16:15:40.640274+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.c scrub ok 2024-02-12T16:15:41.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:41.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:41.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:42.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:42.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:42.214 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:42.241 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:42.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:42.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:42.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:42 smithi123 bash[19119]: cluster 2024-02-12T16:15:40.578260+0000 osd.11 (osd.11) 47 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:15:42.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:42.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:42.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:42 smithi121 bash[18489]: cluster 2024-02-12T16:15:40.578260+0000 osd.11 (osd.11) 47 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:15:42.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:42.510 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:42.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:42 smithi169 bash[19604]: cluster 2024-02-12T16:15:40.578260+0000 osd.11 (osd.11) 47 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:15:42.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:42.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:42.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:42.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:42.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:42.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:42.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:42.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:42.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:43.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:43.056 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:43.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:43.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:43.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:43.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:43.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:43.330 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:43.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:43.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:43 smithi121 bash[18489]: cluster 2024-02-12T16:15:41.530174+0000 osd.7 (osd.7) 63 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:15:43.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:43 smithi121 bash[18489]: cluster 2024-02-12T16:15:41.581474+0000 osd.7 (osd.7) 64 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:15:43.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:43 smithi121 bash[18489]: cluster 2024-02-12T16:15:41.723723+0000 mgr.y (mgr.14146) 911 : cluster [DBG] pgmap v1187: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-02-12T16:15:43.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:43 smithi121 bash[18489]: cluster 2024-02-12T16:15:42.784517+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:15:43.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:43 smithi123 bash[19119]: cluster 2024-02-12T16:15:41.530174+0000 osd.7 (osd.7) 63 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:15:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:43 smithi123 bash[19119]: cluster 2024-02-12T16:15:41.581474+0000 osd.7 (osd.7) 64 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:15:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:43 smithi123 bash[19119]: cluster 2024-02-12T16:15:41.723723+0000 mgr.y (mgr.14146) 911 : cluster [DBG] pgmap v1187: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-02-12T16:15:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:43 smithi123 bash[19119]: cluster 2024-02-12T16:15:42.784517+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:15:43.435 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:43 smithi169 bash[19604]: cluster 2024-02-12T16:15:41.530174+0000 osd.7 (osd.7) 63 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:15:43.435 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:43 smithi169 bash[19604]: cluster 2024-02-12T16:15:41.581474+0000 osd.7 (osd.7) 64 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:15:43.435 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:43 smithi169 bash[19604]: cluster 2024-02-12T16:15:41.723723+0000 mgr.y (mgr.14146) 911 : cluster [DBG] pgmap v1187: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-02-12T16:15:43.435 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:43 smithi169 bash[19604]: cluster 2024-02-12T16:15:42.784517+0000 osd.1 (osd.1) 54 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:15:43.436 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:43.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:43.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:43.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:43.612 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:43.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:43.622 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:15:43.622 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:15:43.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:43.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:43.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:43.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:43.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:43.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:44.014 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:44.120 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:44.148 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:44.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:44.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:44 smithi121 bash[18489]: cluster 2024-02-12T16:15:42.968210+0000 osd.9 (osd.9) 40 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:15:44.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:44 smithi121 bash[18489]: cluster 2024-02-12T16:15:43.057632+0000 osd.6 (osd.6) 45 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:15:44.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:44 smithi121 bash[18489]: cluster 2024-02-12T16:15:43.084207+0000 osd.6 (osd.6) 46 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:15:44.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:44 smithi121 bash[18489]: cluster 2024-02-12T16:15:43.086818+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.d scrub starts 2024-02-12T16:15:44.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:44.276 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:44.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:44.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:44 smithi123 bash[19119]: cluster 2024-02-12T16:15:42.968210+0000 osd.9 (osd.9) 40 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:15:44.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:44 smithi123 bash[19119]: cluster 2024-02-12T16:15:43.057632+0000 osd.6 (osd.6) 45 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:15:44.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:44 smithi123 bash[19119]: cluster 2024-02-12T16:15:43.084207+0000 osd.6 (osd.6) 46 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:15:44.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:44 smithi123 bash[19119]: cluster 2024-02-12T16:15:43.086818+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.d scrub starts 2024-02-12T16:15:44.406 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:44.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:44 smithi169 bash[19604]: cluster 2024-02-12T16:15:42.968210+0000 osd.9 (osd.9) 40 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:15:44.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:44 smithi169 bash[19604]: cluster 2024-02-12T16:15:43.057632+0000 osd.6 (osd.6) 45 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:15:44.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:44 smithi169 bash[19604]: cluster 2024-02-12T16:15:43.084207+0000 osd.6 (osd.6) 46 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:15:44.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:44 smithi169 bash[19604]: cluster 2024-02-12T16:15:43.086818+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.d scrub starts 2024-02-12T16:15:44.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-12T16:15:44.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:44.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:44.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:44.673 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:44.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:44.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:44.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:44.927 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:44.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:45.031 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:45.052 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:45.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:45.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:45.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:45.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:45.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:45.322 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:45.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:45 smithi123 bash[19119]: audit 2024-02-12T16:15:44.256348+0000 mon.a (mon.0) 1432 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:15:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:45 smithi123 bash[19119]: audit 2024-02-12T16:15:44.256562+0000 mon.a (mon.0) 1433 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:15:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:45 smithi123 bash[19119]: audit 2024-02-12T16:15:44.256762+0000 mon.a (mon.0) 1434 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:15:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:45 smithi123 bash[19119]: audit 2024-02-12T16:15:44.284389+0000 mon.a (mon.0) 1435 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:15:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:45 smithi123 bash[19119]: audit 2024-02-12T16:15:44.286159+0000 mon.a (mon.0) 1436 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:15:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:45 smithi123 bash[19119]: audit 2024-02-12T16:15:44.286766+0000 mon.a (mon.0) 1437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:15:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:45 smithi123 bash[19119]: audit 2024-02-12T16:15:44.291272+0000 mon.a (mon.0) 1438 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:15:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:45 smithi121 bash[18489]: audit 2024-02-12T16:15:44.256348+0000 mon.a (mon.0) 1432 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:15:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:45 smithi121 bash[18489]: audit 2024-02-12T16:15:44.256562+0000 mon.a (mon.0) 1433 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:15:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:45 smithi121 bash[18489]: audit 2024-02-12T16:15:44.256762+0000 mon.a (mon.0) 1434 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:15:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:45 smithi121 bash[18489]: audit 2024-02-12T16:15:44.284389+0000 mon.a (mon.0) 1435 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:15:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:45 smithi121 bash[18489]: audit 2024-02-12T16:15:44.286159+0000 mon.a (mon.0) 1436 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:15:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:45 smithi121 bash[18489]: audit 2024-02-12T16:15:44.286766+0000 mon.a (mon.0) 1437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:15:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:45 smithi121 bash[18489]: audit 2024-02-12T16:15:44.291272+0000 mon.a (mon.0) 1438 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:15:45.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:45.433 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:15:45.460 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:45.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:45 smithi169 bash[19604]: audit 2024-02-12T16:15:44.256348+0000 mon.a (mon.0) 1432 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:15:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:45 smithi169 bash[19604]: audit 2024-02-12T16:15:44.256562+0000 mon.a (mon.0) 1433 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:15:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:45 smithi169 bash[19604]: audit 2024-02-12T16:15:44.256762+0000 mon.a (mon.0) 1434 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:15:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:45 smithi169 bash[19604]: audit 2024-02-12T16:15:44.284389+0000 mon.a (mon.0) 1435 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:15:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:45 smithi169 bash[19604]: audit 2024-02-12T16:15:44.286159+0000 mon.a (mon.0) 1436 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:15:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:45 smithi169 bash[19604]: audit 2024-02-12T16:15:44.286766+0000 mon.a (mon.0) 1437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:15:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:45 smithi169 bash[19604]: audit 2024-02-12T16:15:44.291272+0000 mon.a (mon.0) 1438 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:15:45.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:45.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:45.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:45.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:45.736 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:45.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:45.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:45.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:45.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:45.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:46.009 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:46.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:46.147 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:46.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:46.284 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:46.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:46.428 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:46.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:46.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:46.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:46.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:46 smithi123 bash[19119]: cluster 2024-02-12T16:15:45.702727+0000 osd.11 (osd.11) 48 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:15:46.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:46 smithi123 bash[19119]: cluster 2024-02-12T16:15:45.724579+0000 mgr.y (mgr.14146) 913 : cluster [DBG] pgmap v1189: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s 2024-02-12T16:15:46.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:46.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:46 smithi121 bash[18489]: cluster 2024-02-12T16:15:45.702727+0000 osd.11 (osd.11) 48 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:15:46.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:46 smithi121 bash[18489]: cluster 2024-02-12T16:15:45.724579+0000 mgr.y (mgr.14146) 913 : cluster [DBG] pgmap v1189: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s 2024-02-12T16:15:46.701 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:46.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:46 smithi169 bash[19604]: cluster 2024-02-12T16:15:45.702727+0000 osd.11 (osd.11) 48 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:15:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:46 smithi169 bash[19604]: cluster 2024-02-12T16:15:45.724579+0000 mgr.y (mgr.14146) 913 : cluster [DBG] pgmap v1189: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s 2024-02-12T16:15:46.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:46.834 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:46.884 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:15:46.929 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:15:46.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:46.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:46.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:46.976 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:15:47.017 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:15:47.048 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:15:47.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:47.087 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:15:47.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:47.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:47.120 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:15:47.156 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:15:47.188 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:15:47.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:47.214 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:15:47.230 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:47.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:47.236 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:15:47.271 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.273 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.274 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:15:47.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:47.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:47.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:15:47.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:47.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:47.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:47.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:47.616 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:47 smithi169 bash[19604]: cluster 2024-02-12T16:15:46.343097+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.d scrub ok 2024-02-12T16:15:47.616 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:47 smithi169 bash[19604]: cluster 2024-02-12T16:15:46.532175+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.f scrub starts 2024-02-12T16:15:47.616 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:47 smithi169 bash[19604]: audit 2024-02-12T16:15:46.846981+0000 mon.b (mon.2) 118 : audit [DBG] from='client.? v1:172.21.15.121:0/2926763477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:47.616 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:47 smithi169 bash[19604]: cluster 2024-02-12T16:15:46.940602+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:47.616 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:47 smithi169 bash[19604]: cluster 2024-02-12T16:15:46.940644+0000 osd.6 (osd.6) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:47.616 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:47 smithi169 bash[19604]: audit 2024-02-12T16:15:47.223769+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? v1:172.21.15.121:0/1955766751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:47.637 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:47 smithi123 bash[19119]: cluster 2024-02-12T16:15:46.343097+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.d scrub ok 2024-02-12T16:15:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:47 smithi123 bash[19119]: cluster 2024-02-12T16:15:46.532175+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.f scrub starts 2024-02-12T16:15:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:47 smithi123 bash[19119]: audit 2024-02-12T16:15:46.846981+0000 mon.b (mon.2) 118 : audit [DBG] from='client.? v1:172.21.15.121:0/2926763477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:47 smithi123 bash[19119]: cluster 2024-02-12T16:15:46.940602+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:47 smithi123 bash[19119]: cluster 2024-02-12T16:15:46.940644+0000 osd.6 (osd.6) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:47 smithi123 bash[19119]: audit 2024-02-12T16:15:47.223769+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? v1:172.21.15.121:0/1955766751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:47.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:47 smithi121 bash[18489]: cluster 2024-02-12T16:15:46.343097+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.d scrub ok 2024-02-12T16:15:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:47 smithi121 bash[18489]: cluster 2024-02-12T16:15:46.532175+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.f scrub starts 2024-02-12T16:15:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:47 smithi121 bash[18489]: audit 2024-02-12T16:15:46.846981+0000 mon.b (mon.2) 118 : audit [DBG] from='client.? v1:172.21.15.121:0/2926763477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:15:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:47 smithi121 bash[18489]: cluster 2024-02-12T16:15:46.940602+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:15:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:47 smithi121 bash[18489]: cluster 2024-02-12T16:15:46.940644+0000 osd.6 (osd.6) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:15:47.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:47 smithi121 bash[18489]: audit 2024-02-12T16:15:47.223769+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? v1:172.21.15.121:0/1955766751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:15:47.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:47.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:47.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:47.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:47.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:48.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:48.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:48.135 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:48.162 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:48.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:48.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:48.430 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:48 smithi123 bash[19119]: cluster 2024-02-12T16:15:46.567051+0000 osd.7 (osd.7) 65 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:15:48.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:48 smithi123 bash[19119]: cluster 2024-02-12T16:15:46.610390+0000 osd.7 (osd.7) 66 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:15:48.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:48 smithi123 bash[19119]: cluster 2024-02-12T16:15:47.408854+0000 mon.a (mon.0) 1440 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:15:48.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:48 smithi123 bash[19119]: cluster 2024-02-12T16:15:47.408893+0000 mon.a (mon.0) 1441 : cluster [INF] Cluster is now healthy 2024-02-12T16:15:48.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:48 smithi123 bash[19119]: audit 2024-02-12T16:15:47.415036+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.121:0/1955766751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:15:48.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:48 smithi123 bash[19119]: cluster 2024-02-12T16:15:47.415104+0000 mon.a (mon.0) 1443 : cluster [DBG] osdmap e376: 12 total, 12 up, 12 in 2024-02-12T16:15:48.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:48 smithi123 bash[19119]: cluster 2024-02-12T16:15:47.724980+0000 mgr.y (mgr.14146) 914 : cluster [DBG] pgmap v1191: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-02-12T16:15:48.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:48.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:48.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:48.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:48.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:48 smithi121 bash[18489]: cluster 2024-02-12T16:15:46.567051+0000 osd.7 (osd.7) 65 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:15:48.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:48 smithi121 bash[18489]: cluster 2024-02-12T16:15:46.610390+0000 osd.7 (osd.7) 66 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:15:48.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:48 smithi121 bash[18489]: cluster 2024-02-12T16:15:47.408854+0000 mon.a (mon.0) 1440 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:15:48.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:48 smithi121 bash[18489]: cluster 2024-02-12T16:15:47.408893+0000 mon.a (mon.0) 1441 : cluster [INF] Cluster is now healthy 2024-02-12T16:15:48.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:48 smithi121 bash[18489]: audit 2024-02-12T16:15:47.415036+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.121:0/1955766751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:15:48.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:48 smithi121 bash[18489]: cluster 2024-02-12T16:15:47.415104+0000 mon.a (mon.0) 1443 : cluster [DBG] osdmap e376: 12 total, 12 up, 12 in 2024-02-12T16:15:48.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:48 smithi121 bash[18489]: cluster 2024-02-12T16:15:47.724980+0000 mgr.y (mgr.14146) 914 : cluster [DBG] pgmap v1191: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-02-12T16:15:48.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:48 smithi169 bash[19604]: cluster 2024-02-12T16:15:46.567051+0000 osd.7 (osd.7) 65 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:15:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:48 smithi169 bash[19604]: cluster 2024-02-12T16:15:46.610390+0000 osd.7 (osd.7) 66 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:15:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:48 smithi169 bash[19604]: cluster 2024-02-12T16:15:47.408854+0000 mon.a (mon.0) 1440 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:15:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:48 smithi169 bash[19604]: cluster 2024-02-12T16:15:47.408893+0000 mon.a (mon.0) 1441 : cluster [INF] Cluster is now healthy 2024-02-12T16:15:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:48 smithi169 bash[19604]: audit 2024-02-12T16:15:47.415036+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.121:0/1955766751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:15:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:48 smithi169 bash[19604]: cluster 2024-02-12T16:15:47.415104+0000 mon.a (mon.0) 1443 : cluster [DBG] osdmap e376: 12 total, 12 up, 12 in 2024-02-12T16:15:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:48 smithi169 bash[19604]: cluster 2024-02-12T16:15:47.724980+0000 mgr.y (mgr.14146) 914 : cluster [DBG] pgmap v1191: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s 2024-02-12T16:15:48.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:48.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:48.985 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:48.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:49.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:49.111 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:49.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:49.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:49.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:49.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:49.302 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:15:49.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:49.396 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:49.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:49.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:49.510 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:49 smithi121 bash[18489]: cluster 2024-02-12T16:15:48.730478+0000 osd.11 (osd.11) 49 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:15:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:49.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:49.606 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:49 smithi123 bash[19119]: cluster 2024-02-12T16:15:48.730478+0000 osd.11 (osd.11) 49 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:15:49.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:49.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:49.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:49.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:49.786 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:49.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:49.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:49 smithi169 bash[19604]: cluster 2024-02-12T16:15:48.730478+0000 osd.11 (osd.11) 49 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:15:49.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:49.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:49.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:50.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:50.046 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:50.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:50.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:50.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:50.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:50.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:50.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:50.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:50.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:50.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:50.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:50.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:50.687 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:50.695 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:50 smithi123 bash[19119]: cluster 2024-02-12T16:15:49.725433+0000 mgr.y (mgr.14146) 915 : cluster [DBG] pgmap v1192: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s 2024-02-12T16:15:50.695 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:50 smithi123 bash[19119]: cluster 2024-02-12T16:15:49.878402+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.f scrub ok 2024-02-12T16:15:50.695 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:50 smithi123 bash[19119]: cluster 2024-02-12T16:15:50.399969+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e377: 12 total, 12 up, 12 in 2024-02-12T16:15:50.726 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:50.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:50.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:50 smithi169 bash[19604]: cluster 2024-02-12T16:15:49.725433+0000 mgr.y (mgr.14146) 915 : cluster [DBG] pgmap v1192: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s 2024-02-12T16:15:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:50 smithi169 bash[19604]: cluster 2024-02-12T16:15:49.878402+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.f scrub ok 2024-02-12T16:15:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:50 smithi169 bash[19604]: cluster 2024-02-12T16:15:50.399969+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e377: 12 total, 12 up, 12 in 2024-02-12T16:15:50.830 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:50.856 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:50.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:50 smithi121 bash[18489]: cluster 2024-02-12T16:15:49.725433+0000 mgr.y (mgr.14146) 915 : cluster [DBG] pgmap v1192: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s 2024-02-12T16:15:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:50 smithi121 bash[18489]: cluster 2024-02-12T16:15:49.878402+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.f scrub ok 2024-02-12T16:15:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:50 smithi121 bash[18489]: cluster 2024-02-12T16:15:50.399969+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e377: 12 total, 12 up, 12 in 2024-02-12T16:15:50.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:15:50.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:51.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:51.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:15:51.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:51.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:51.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:51.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:51.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:51.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:51.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:51.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:51 smithi123 bash[19119]: cluster 2024-02-12T16:15:50.582543+0000 osd.11 (osd.11) 50 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:15:51.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:51.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:51.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:51 smithi169 bash[19604]: cluster 2024-02-12T16:15:50.582543+0000 osd.11 (osd.11) 50 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:15:51.710 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:51.737 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:15:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T16:15:51.737 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:15:51.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:51.841 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:51 smithi121 bash[18489]: cluster 2024-02-12T16:15:50.582543+0000 osd.11 (osd.11) 50 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:15:51.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:51.984 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:51.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:52.090 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:52.115 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:52.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:52.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:52.252 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:52.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:52.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:52.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:52.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:52.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:52.494 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:52 smithi121 bash[18489]: cluster 2024-02-12T16:15:51.594775+0000 osd.1 (osd.1) 57 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:15:52.494 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:52 smithi121 bash[18489]: cluster 2024-02-12T16:15:51.659208+0000 osd.1 (osd.1) 58 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:15:52.495 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:52 smithi121 bash[18489]: cluster 2024-02-12T16:15:51.725779+0000 mgr.y (mgr.14146) 916 : cluster [DBG] pgmap v1194: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-02-12T16:15:52.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:52.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:52.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:52.635 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:52 smithi123 bash[19119]: cluster 2024-02-12T16:15:51.594775+0000 osd.1 (osd.1) 57 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:15:52.635 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:52 smithi123 bash[19119]: cluster 2024-02-12T16:15:51.659208+0000 osd.1 (osd.1) 58 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:15:52.635 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:52 smithi123 bash[19119]: cluster 2024-02-12T16:15:51.725779+0000 mgr.y (mgr.14146) 916 : cluster [DBG] pgmap v1194: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-02-12T16:15:52.658 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:52.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:52.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:52.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:52 smithi169 bash[19604]: cluster 2024-02-12T16:15:51.594775+0000 osd.1 (osd.1) 57 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:15:52.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:52 smithi169 bash[19604]: cluster 2024-02-12T16:15:51.659208+0000 osd.1 (osd.1) 58 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:15:52.768 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:52 smithi169 bash[19604]: cluster 2024-02-12T16:15:51.725779+0000 mgr.y (mgr.14146) 916 : cluster [DBG] pgmap v1194: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-02-12T16:15:52.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:52.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:52.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:52.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:52.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:53.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:53.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:53.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:53.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:53.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:53.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:53.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:53.450 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:53.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:53.583 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:53.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:53 smithi121 bash[18489]: cluster 2024-02-12T16:15:52.830840+0000 osd.11 (osd.11) 51 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:15:53.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:53.714 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:53.716 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:53 smithi123 bash[19119]: cluster 2024-02-12T16:15:52.830840+0000 osd.11 (osd.11) 51 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:15:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:53.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:53 smithi169 bash[19604]: cluster 2024-02-12T16:15:52.830840+0000 osd.11 (osd.11) 51 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:15:53.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:53.844 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:15:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:15:53.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:53.975 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:53.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:54.081 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:54.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:54.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:54.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:54.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:54.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:54.293 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:15:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:15:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:15:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:15:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.293 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:283933 2.5 2:a5b89332:::benchmark_data_smithi169_42458_object110899:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:19.363983+0000", 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 514.92559748999997, 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16776664199999999, 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 283933 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.363983+0000", 2024-02-12T16:15:54.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.363983+0000", 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.363981+0000", 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8420000000000005e-06 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.363989+0000", 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4413000000000001e-05 2024-02-12T16:15:54.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.364003+0000", 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2160000000000003e-06 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.364010+0000", 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9359e-05 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.364059+0000", 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116135 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.364175+0000", 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00071697199999999997 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0", 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.364892+0000", 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077110440000000002 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.372603+0000", 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.159016095 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.531619+0000", 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4107e-05 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.531693+0000", 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6598999999999998e-05 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.531750+0000", 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16776664199999999 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:15:54.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:283956 2.5 2:a08dcaa2:::benchmark_data_smithi169_42458_object110922:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:19.381854+0000", 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 514.90772720500001, 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16527502199999999, 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:15:54.299 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 283956 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.381854+0000", 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.381854+0000", 2024-02-12T16:15:54.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.381852+0000", 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9719999999999997e-06 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.381858+0000", 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6569999999999992e-06 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.381868+0000", 2024-02-12T16:15:54.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0519999999999998e-06 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.381874+0000", 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9293999999999999e-05 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.381923+0000", 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9888000000000003e-05 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.382023+0000", 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069245600000000004 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0", 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.382715+0000", 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0035727580000000001 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.386288+0000", 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.160678353 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.546967+0000", 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6988e-05 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.547064+0000", 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5075999999999998e-05 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.547129+0000", 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16527502199999999 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:15:54.304 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.14811.0:283960 2.5 2:a3f50859:::benchmark_data_smithi169_42458_object110926:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:07:19.386879+0000", 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 514.90270141300005, 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.163962477, 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.14811", 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/641713321", 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 283960 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:15:54.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.386879+0000", 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.386879+0000", 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.386878+0000", 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2190000000000003e-06 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:15:54.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.386884+0000", 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1980000000000004e-06 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.386891+0000", 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9919999999999998e-06 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.386896+0000", 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005376189 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.392272+0000", 2024-02-12T16:15:54.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100343 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.392372+0000", 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000547044 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0", 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.392919+0000", 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027722770000000001 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.395692+0000", 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15509719199999999 2024-02-12T16:15:54.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.550789+0000", 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0408000000000001e-05 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.550819+0000", 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2654e-05 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:07:19.550842+0000", 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.163962477 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:15:54.309 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:15:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:15:54.310 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:15:54.310 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:15:54.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:54.372 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:54.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:54.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:54.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:54 smithi121 bash[18489]: cluster 2024-02-12T16:15:53.726230+0000 mgr.y (mgr.14146) 917 : cluster [DBG] pgmap v1195: 27 pgs: 27 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-12T16:15:54.500 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:54.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:54.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:54.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:54.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:54.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:54.773 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:54.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:54.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:54 smithi169 bash[19604]: cluster 2024-02-12T16:15:53.726230+0000 mgr.y (mgr.14146) 917 : cluster [DBG] pgmap v1195: 27 pgs: 27 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-12T16:15:54.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:54.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:54 smithi123 bash[19119]: cluster 2024-02-12T16:15:53.726230+0000 mgr.y (mgr.14146) 917 : cluster [DBG] pgmap v1195: 27 pgs: 27 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 935 op/s 2024-02-12T16:15:54.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:54.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:54.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:55.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:55.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:55.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-12T16:15:55.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:55.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:55.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:55.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:55.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:55.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:55.431 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:55.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:55.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:55.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:55.669 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:55 smithi123 bash[19119]: cluster 2024-02-12T16:15:54.837766+0000 osd.9 (osd.9) 41 : cluster [DBG] 4.6 deep-scrub starts 2024-02-12T16:15:55.687 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:55.774 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:55.775 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:15:55.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:55.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:55 smithi169 bash[19604]: cluster 2024-02-12T16:15:54.837766+0000 osd.9 (osd.9) 41 : cluster [DBG] 4.6 deep-scrub starts 2024-02-12T16:15:55.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:55 smithi121 bash[18489]: cluster 2024-02-12T16:15:54.837766+0000 osd.9 (osd.9) 41 : cluster [DBG] 4.6 deep-scrub starts 2024-02-12T16:15:55.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:55.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:55.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:56.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:56.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:56.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:56.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:56.239 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:56.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:56.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:56.385 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:56.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:56.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:56.528 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:56.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:56.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:56.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:56.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:56.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:56.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:56 smithi123 bash[19119]: cluster 2024-02-12T16:15:55.726672+0000 mgr.y (mgr.14146) 918 : cluster [DBG] pgmap v1196: 27 pgs: 27 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-02-12T16:15:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:56 smithi169 bash[19604]: cluster 2024-02-12T16:15:55.726672+0000 mgr.y (mgr.14146) 918 : cluster [DBG] pgmap v1196: 27 pgs: 27 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-02-12T16:15:56.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:15:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:56.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:15:56.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:56.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:56 smithi121 bash[18489]: cluster 2024-02-12T16:15:55.726672+0000 mgr.y (mgr.14146) 918 : cluster [DBG] pgmap v1196: 27 pgs: 27 active+clean; 8.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-02-12T16:15:56.931 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:56.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:57.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:57.059 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:57.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:57.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:57.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:57.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:57.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:57.330 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:57.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:57.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:57.458 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:57.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:57.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:57.601 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:15:57.601 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:15:57.601 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-12T16:15:57.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:15:57.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:57.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:57.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:57.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:57.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:57.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:57.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:57.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:57.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:58.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:58.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:58.156 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:15:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:15:58.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:58.287 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:58.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:15:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:15:58.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:15:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:15:58.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:58.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:58.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:58.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:15:58 smithi121 bash[18489]: cluster 2024-02-12T16:15:57.727052+0000 mgr.y (mgr.14146) 919 : cluster [DBG] pgmap v1197: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-02-12T16:15:58.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:58.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:58.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:15:58 smithi169 bash[19604]: cluster 2024-02-12T16:15:57.727052+0000 mgr.y (mgr.14146) 919 : cluster [DBG] pgmap v1197: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-02-12T16:15:58.968 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:58.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:59.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:59.099 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:15:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:59.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:15:59.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:15:58 smithi123 bash[19119]: cluster 2024-02-12T16:15:57.727052+0000 mgr.y (mgr.14146) 919 : cluster [DBG] pgmap v1197: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-02-12T16:15:59.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:59.230 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:15:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:59.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:15:59.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:59.359 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:15:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:59.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:15:59.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:59.490 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:15:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:59.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:15:59.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:59.623 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:15:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:59.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:15:59.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:59.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:15:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:59.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:15:59.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:59.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:15:59.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:15:59.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:15:59.918 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:16:00.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:00.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:00.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:00.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:00.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:00.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:00.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:00.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:00.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:00.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:00.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:00.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:00.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:00 smithi169 bash[19604]: cluster 2024-02-12T16:15:59.727805+0000 mgr.y (mgr.14146) 920 : cluster [DBG] pgmap v1198: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-02-12T16:16:00.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:00.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:00.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:00.941 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:00 smithi121 bash[18489]: cluster 2024-02-12T16:15:59.727805+0000 mgr.y (mgr.14146) 920 : cluster [DBG] pgmap v1198: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-02-12T16:16:00.965 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:00.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:01.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:01.076 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:00 smithi123 bash[19119]: cluster 2024-02-12T16:15:59.727805+0000 mgr.y (mgr.14146) 920 : cluster [DBG] pgmap v1198: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-02-12T16:16:01.112 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:01.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:01.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:01.251 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:01.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:01.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:01.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:01.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:01.476 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:16:01.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:01.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:01.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:01.520 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:16:01.553 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:16:01.581 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:16:01.608 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:01.615 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:16:01.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:01.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:01.641 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:16:01.661 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:16:01.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:16:01.707 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:16:01.737 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:16:01.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:01.760 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:16:01.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:01.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:01.796 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:16:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:16:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:16:01.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:01.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:01.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:01.999 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:01 smithi169 bash[19604]: audit 2024-02-12T16:16:01.427978+0000 mon.a (mon.0) 1445 : audit [DBG] from='client.? v1:172.21.15.121:0/3194076361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:01.999 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:01 smithi169 bash[19604]: audit 2024-02-12T16:16:01.714338+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25445 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:01.999 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:01 smithi169 bash[19604]: audit 2024-02-12T16:16:01.715618+0000 mon.b (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.121:0/846973846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:01.999 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:01 smithi169 bash[19604]: audit 2024-02-12T16:16:01.716998+0000 mon.b (mon.2) 120 : audit [INF] from='client.25445 v1:172.21.15.121:0/846973846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:02.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:02.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:02.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:02.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:01 smithi123 bash[19119]: audit 2024-02-12T16:16:01.427978+0000 mon.a (mon.0) 1445 : audit [DBG] from='client.? v1:172.21.15.121:0/3194076361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:02.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:01 smithi123 bash[19119]: audit 2024-02-12T16:16:01.714338+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25445 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:02.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:01 smithi123 bash[19119]: audit 2024-02-12T16:16:01.715618+0000 mon.b (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.121:0/846973846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:02.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:01 smithi123 bash[19119]: audit 2024-02-12T16:16:01.716998+0000 mon.b (mon.2) 120 : audit [INF] from='client.25445 v1:172.21.15.121:0/846973846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:02.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:02.154 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:01 smithi121 bash[18489]: audit 2024-02-12T16:16:01.427978+0000 mon.a (mon.0) 1445 : audit [DBG] from='client.? v1:172.21.15.121:0/3194076361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:02.155 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:01 smithi121 bash[18489]: audit 2024-02-12T16:16:01.714338+0000 mon.a (mon.0) 1446 : audit [INF] from='client.25445 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:02.155 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:01 smithi121 bash[18489]: audit 2024-02-12T16:16:01.715618+0000 mon.b (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.121:0/846973846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:02.155 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:01 smithi121 bash[18489]: audit 2024-02-12T16:16:01.716998+0000 mon.b (mon.2) 120 : audit [INF] from='client.25445 v1:172.21.15.121:0/846973846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:02.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:02.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:02.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:02.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:02.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:02.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:02.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:02.572 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:02.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:02.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:02.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:02.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.862 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:02.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:02.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:02.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:02 smithi169 bash[19604]: cluster 2024-02-12T16:16:01.728211+0000 mgr.y (mgr.14146) 921 : cluster [DBG] pgmap v1199: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-02-12T16:16:02.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:02 smithi169 bash[19604]: cluster 2024-02-12T16:16:01.783219+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:02.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:02 smithi169 bash[19604]: audit 2024-02-12T16:16:01.793997+0000 mon.a (mon.0) 1448 : audit [INF] from='client.25445 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:16:02.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:02 smithi169 bash[19604]: cluster 2024-02-12T16:16:01.794704+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e378: 12 total, 12 up, 12 in 2024-02-12T16:16:03.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:03.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:02 smithi121 bash[18489]: cluster 2024-02-12T16:16:01.728211+0000 mgr.y (mgr.14146) 921 : cluster [DBG] pgmap v1199: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-02-12T16:16:03.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:02 smithi121 bash[18489]: cluster 2024-02-12T16:16:01.783219+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:03.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:02 smithi121 bash[18489]: audit 2024-02-12T16:16:01.793997+0000 mon.a (mon.0) 1448 : audit [INF] from='client.25445 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:16:03.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:02 smithi121 bash[18489]: cluster 2024-02-12T16:16:01.794704+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e378: 12 total, 12 up, 12 in 2024-02-12T16:16:03.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:03.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:03.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:02 smithi123 bash[19119]: cluster 2024-02-12T16:16:01.728211+0000 mgr.y (mgr.14146) 921 : cluster [DBG] pgmap v1199: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-02-12T16:16:03.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:02 smithi123 bash[19119]: cluster 2024-02-12T16:16:01.783219+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:03.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:02 smithi123 bash[19119]: audit 2024-02-12T16:16:01.793997+0000 mon.a (mon.0) 1448 : audit [INF] from='client.25445 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:16:03.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:02 smithi123 bash[19119]: cluster 2024-02-12T16:16:01.794704+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e378: 12 total, 12 up, 12 in 2024-02-12T16:16:03.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:03.258 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:03.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:03.386 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:03.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:03.527 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:03.600 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:03.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:03.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:03.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:03.786 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:03.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:03.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:03.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:04.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:04.055 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:04.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:04.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:04.273 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:16:04.273 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:16:04.286 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:04.313 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:04.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:04.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:04.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:04.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:04.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:04.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:04.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:04.837 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:04 smithi121 bash[18489]: cluster 2024-02-12T16:16:03.728663+0000 mgr.y (mgr.14146) 922 : cluster [DBG] pgmap v1201: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-12T16:16:04.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:04.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:04.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:05.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:04 smithi169 bash[19604]: cluster 2024-02-12T16:16:03.728663+0000 mgr.y (mgr.14146) 922 : cluster [DBG] pgmap v1201: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-12T16:16:05.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:16:05.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:05.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:16:05.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:04 smithi123 bash[19119]: cluster 2024-02-12T16:16:03.728663+0000 mgr.y (mgr.14146) 922 : cluster [DBG] pgmap v1201: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-02-12T16:16:05.159 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-12T16:16:05.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:05.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:05.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:05.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:05.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:05.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:05.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:05.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:05.544 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:05.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:05.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:05.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:05.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:05.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:05.807 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:05.833 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:05.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:05.939 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:05.960 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:05.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:06.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:06.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:06.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:06.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:06.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:06.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:06.374 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:06.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:06.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:06.506 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:06.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:06.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:06.616 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:06 smithi123 bash[19119]: cluster 2024-02-12T16:16:05.401441+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e379: 12 total, 12 up, 12 in 2024-02-12T16:16:06.616 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:06 smithi123 bash[19119]: cluster 2024-02-12T16:16:05.729004+0000 mgr.y (mgr.14146) 923 : cluster [DBG] pgmap v1203: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-02-12T16:16:06.639 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:06.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:06 smithi121 bash[18489]: cluster 2024-02-12T16:16:05.401441+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e379: 12 total, 12 up, 12 in 2024-02-12T16:16:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:06 smithi121 bash[18489]: cluster 2024-02-12T16:16:05.729004+0000 mgr.y (mgr.14146) 923 : cluster [DBG] pgmap v1203: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-02-12T16:16:06.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:06.751 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:06 smithi169 bash[19604]: cluster 2024-02-12T16:16:05.401441+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e379: 12 total, 12 up, 12 in 2024-02-12T16:16:06.751 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:06 smithi169 bash[19604]: cluster 2024-02-12T16:16:05.729004+0000 mgr.y (mgr.14146) 923 : cluster [DBG] pgmap v1203: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-02-12T16:16:06.770 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:06.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:06.873 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:06.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:06.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:07.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:07.041 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:07.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:07.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:07.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:07.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:07.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:07.419 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:07 smithi123 bash[19119]: cluster 2024-02-12T16:16:06.047126+0000 osd.7 (osd.7) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:07.434 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:07.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:07.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:07 smithi169 bash[19604]: cluster 2024-02-12T16:16:06.047126+0000 osd.7 (osd.7) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:07.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:07 smithi169 bash[19604]: cluster 2024-02-12T16:16:06.047186+0000 osd.7 (osd.7) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:07.546 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:07 smithi169 bash[19604]: cluster 2024-02-12T16:16:06.414260+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e380: 12 total, 12 up, 12 in 2024-02-12T16:16:07.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:07 smithi121 bash[18489]: cluster 2024-02-12T16:16:06.047126+0000 osd.7 (osd.7) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:07 smithi121 bash[18489]: cluster 2024-02-12T16:16:06.047186+0000 osd.7 (osd.7) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:07 smithi121 bash[18489]: cluster 2024-02-12T16:16:06.414260+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e380: 12 total, 12 up, 12 in 2024-02-12T16:16:07.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:07.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:07.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:07.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 381, 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-02-12T15:47:18.273813+0000", 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-02-12T16:16:07.412193+0000", 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "last_up_change": "2024-02-12T15:58:23.801839+0000", 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "last_in_change": "2024-02-12T16:10:58.738300+0000", 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_num": 5801984, 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_set": [ 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "noscrub", 2024-02-12T16:16:07.889 INFO:teuthology.orchestra.run.smithi121.stdout: "pglog_hardlimit", 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "purged_snapdirs", 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_deletes", 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "sortbitwise" 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_version": 27, 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_snapshot": "", 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_max": 4, 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd": 12, 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:16:07.890 INFO:teuthology.orchestra.run.smithi121.stdout: "min_compat_client": "jewel", 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "require_osd_release": "pacific", 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "pools": [ 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 1, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T15:51:35.886878+0000", 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:16:07.891 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 11, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 11, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 11, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 11, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 11, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:16:07.892 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "368", 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "365", 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:16:07.893 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:16:07.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "options": { 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_max": 32, 2024-02-12T16:16:07.895 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_min": 1 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_devicehealth": {} 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 4, 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "unique_pool_2", 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T16:13:44.637807+0000", 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:16:07.896 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 16, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 16, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 16, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 16, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 16, 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:16:07.897 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "346", 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:16:07.898 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:16:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:16:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "options": {}, 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "rados": {} 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0", 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.901 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 8, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 334, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.902 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6802", 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6804", 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:16:07.903 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6803", 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6801/1815232143", 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6802/1815232143", 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6804/1815232143", 2024-02-12T16:16:07.904 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6803/1815232143", 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "c6140309-5703-4538-9739-276fbbdaa880", 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.905 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 13, 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6805", 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.906 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6806", 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6808", 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6807", 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.908 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6805/2968433670", 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6806/2968433670", 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6808/2968433670", 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6807/2968433670", 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170", 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.909 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 21, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 234, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6809", 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.910 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6810", 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6812", 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:16:07.911 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6811", 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6809/2158187852", 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6810/2158187852", 2024-02-12T16:16:07.912 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6812/2158187852", 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6811/2158187852", 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd", 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.913 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 26, 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 334, 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6813", 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.914 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6814", 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6816", 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.915 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6815", 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6813/1415189366", 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6814/1415189366", 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6816/1415189366", 2024-02-12T16:16:07.916 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6815/1415189366", 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31", 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.917 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 32, 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 334, 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6800", 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6801", 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6803", 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6802", 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6800/3447700123", 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6801/3447700123", 2024-02-12T16:16:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6803/3447700123", 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6802/3447700123", 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602", 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.921 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 39, 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6804", 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.922 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6805", 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6807", 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:16:07.923 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6806", 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6804/961317306", 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6805/961317306", 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6807/961317306", 2024-02-12T16:16:07.924 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6806/961317306", 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0", 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.925 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 48, 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6808", 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6809", 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6811", 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.927 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6810", 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6808/1377261784", 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6809/1377261784", 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6811/1377261784", 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6810/1377261784", 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.928 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515", 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 56, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:16:07.929 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6812", 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6813", 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6815", 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.931 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6814", 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6812/1702613957", 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6813/1702613957", 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6815/1702613957", 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6814/1702613957", 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.932 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75", 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 65, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 310, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6800", 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6801", 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:16:07.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6803", 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6802", 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6800/2551075225", 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6801/2551075225", 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6803/2551075225", 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6802/2551075225", 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15", 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 74, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6804", 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6805", 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.938 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6807", 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6806", 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6804/2339555071", 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6805/2339555071", 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6807/2339555071", 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6806/2339555071", 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "91f74dca-503a-45d1-9647-df4804d58515", 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.940 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 83, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.941 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6808", 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6809", 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.942 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6811", 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6810", 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.943 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6808/391306610", 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6809/391306610", 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6811/391306610", 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6810/391306610", 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf", 2024-02-12T16:16:07.944 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 93, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6812", 2024-02-12T16:16:07.945 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6813", 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.946 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6815", 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6814", 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.947 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6812/3696812305", 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6813/3696812305", 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3696812305", 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6814/3696812305", 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_xinfo": [ 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:16:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:11.067709+0000", 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.949 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:07.612364+0000", 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.950 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:09.926674+0000", 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:36.300947+0000", 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:16:07.951 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:35.423624+0000", 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.952 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:34.653370+0000", 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:46.940646+0000", 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.953 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:16:06.047188+0000", 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.954 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:28.261167+0000", 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:14:54.640616+0000", 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:15:07.789283+0000", 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:16:07.956 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:16:07.210334+0000", 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap": [], 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap_items": [], 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_temp": [], 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_temp": [], 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklist": { 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/2474849789": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/253689008": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/3531151725": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/1778611055": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:16:07.957 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/28718860": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/445290372": "2024-02-13T15:47:47.232761+0000" 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "range_blocklist": {}, 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profiles": { 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "default": { 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-failure-domain": "osd", 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "k": "2", 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "m": "1", 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin": "jerasure", 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "technique": "reed_sol_van" 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps_queue": [], 2024-02-12T16:16:07.958 INFO:teuthology.orchestra.run.smithi121.stdout: "new_removed_snaps": [], 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "new_purged_snaps": [], 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_node_flags": {}, 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "device_class_flags": {}, 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": { 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:16:07.959 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:16:07.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:07.960 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:07 smithi123 bash[19119]: cluster 2024-02-12T16:16:06.047186+0000 osd.7 (osd.7) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:07.960 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:07 smithi123 bash[19119]: cluster 2024-02-12T16:16:06.414260+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e380: 12 total, 12 up, 12 in 2024-02-12T16:16:07.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:07.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:08.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:08.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:08.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:08.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:08.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:08.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:08.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:08.401 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:08.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:08.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:08.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:08.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:08 smithi123 bash[19119]: cluster 2024-02-12T16:16:07.210268+0000 osd.11 (osd.11) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:08.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:08 smithi123 bash[19119]: cluster 2024-02-12T16:16:07.210332+0000 osd.11 (osd.11) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:08 smithi123 bash[19119]: cluster 2024-02-12T16:16:07.424370+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e381: 12 total, 12 up, 12 in 2024-02-12T16:16:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:08 smithi123 bash[19119]: cluster 2024-02-12T16:16:07.729417+0000 mgr.y (mgr.14146) 924 : cluster [DBG] pgmap v1206: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:16:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:08 smithi123 bash[19119]: audit 2024-02-12T16:16:07.881784+0000 mon.b (mon.2) 121 : audit [DBG] from='client.? v1:172.21.15.121:0/3279972325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:16:08.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:08.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:08.734 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:08 smithi121 bash[18489]: cluster 2024-02-12T16:16:07.210268+0000 osd.11 (osd.11) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:08.734 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:08 smithi121 bash[18489]: cluster 2024-02-12T16:16:07.210332+0000 osd.11 (osd.11) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:08.735 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:08 smithi121 bash[18489]: cluster 2024-02-12T16:16:07.424370+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e381: 12 total, 12 up, 12 in 2024-02-12T16:16:08.735 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:08 smithi121 bash[18489]: cluster 2024-02-12T16:16:07.729417+0000 mgr.y (mgr.14146) 924 : cluster [DBG] pgmap v1206: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:16:08.735 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:08 smithi121 bash[18489]: audit 2024-02-12T16:16:07.881784+0000 mon.b (mon.2) 121 : audit [DBG] from='client.? v1:172.21.15.121:0/3279972325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:16:08.736 INFO:tasks.thrashosds.thrasher:j is {'epoch': 381, 'fsid': 'eb2349b2-c9bd-11ee-95b9-87774f69a715', 'created': '2024-02-12T15:47:18.273813+0000', 'modified': '2024-02-12T16:16:07.412193+0000', 'last_up_change': '2024-02-12T15:58:23.801839+0000', 'last_in_change': '2024-02-12T16:10:58.738300+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '368', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '365', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-12T16:13:44.637807+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '346', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 1815232143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 1815232143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 1815232143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 1815232143}]}, 'public_addr': '172.21.15.121:6801/1815232143', 'cluster_addr': '172.21.15.121:6802/1815232143', 'heartbeat_back_addr': '172.21.15.121:6804/1815232143', 'heartbeat_front_addr': '172.21.15.121:6803/1815232143', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c6140309-5703-4538-9739-276fbbdaa880', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2968433670}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2968433670}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 2968433670}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2968433670}]}, 'public_addr': '172.21.15.121:6805/2968433670', 'cluster_addr': '172.21.15.121:6806/2968433670', 'heartbeat_back_addr': '172.21.15.121:6808/2968433670', 'heartbeat_front_addr': '172.21.15.121:6807/2968433670', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 234, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 2158187852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 2158187852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 2158187852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 2158187852}]}, 'public_addr': '172.21.15.121:6809/2158187852', 'cluster_addr': '172.21.15.121:6810/2158187852', 'heartbeat_back_addr': '172.21.15.121:6812/2158187852', 'heartbeat_front_addr': '172.21.15.121:6811/2158187852', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5f824526-596d-42e1-8b7e-cd8676ded6dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 1415189366}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 1415189366}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6816', 'nonce': 1415189366}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 1415189366}]}, 'public_addr': '172.21.15.121:6813/1415189366', 'cluster_addr': '172.21.15.121:6814/1415189366', 'heartbeat_back_addr': '172.21.15.121:6816/1415189366', 'heartbeat_front_addr': '172.21.15.121:6815/1415189366', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '14c2f878-99ce-4b51-8d9c-131c1d883b31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6800', 'nonce': 3447700123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6801', 'nonce': 3447700123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6803', 'nonce': 3447700123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6802', 'nonce': 3447700123}]}, 'public_addr': '172.21.15.123:6800/3447700123', 'cluster_addr': '172.21.15.123:6801/3447700123', 'heartbeat_back_addr': '172.21.15.123:6803/3447700123', 'heartbeat_front_addr': '172.21.15.123:6802/3447700123', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6804', 'nonce': 961317306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6805', 'nonce': 961317306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6807', 'nonce': 961317306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6806', 'nonce': 961317306}]}, 'public_addr': '172.21.15.123:6804/961317306', 'cluster_addr': '172.21.15.123:6805/961317306', 'heartbeat_back_addr': '172.21.15.123:6807/961317306', 'heartbeat_front_addr': '172.21.15.123:6806/961317306', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '71e715e6-eac1-4386-927e-51df0cc419c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6808', 'nonce': 1377261784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6809', 'nonce': 1377261784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6811', 'nonce': 1377261784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6810', 'nonce': 1377261784}]}, 'public_addr': '172.21.15.123:6808/1377261784', 'cluster_addr': '172.21.15.123:6809/1377261784', 'heartbeat_back_addr': '172.21.15.123:6811/1377261784', 'heartbeat_front_addr': '172.21.15.123:6810/1377261784', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b419b04f-0c0c-495a-aa9b-25c334159515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6812', 'nonce': 1702613957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6813', 'nonce': 1702613957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6815', 'nonce': 1702613957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6814', 'nonce': 1702613957}]}, 'public_addr': '172.21.15.123:6812/1702613957', 'cluster_addr': '172.21.15.123:6813/1702613957', 'heartbeat_back_addr': '172.21.15.123:6815/1702613957', 'heartbeat_front_addr': '172.21.15.123:6814/1702613957', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1591e7bc-5b86-4401-803d-5325062ffb75', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 310, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6800', 'nonce': 2551075225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 2551075225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 2551075225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6802', 'nonce': 2551075225}]}, 'public_addr': '172.21.15.169:6800/2551075225', 'cluster_addr': '172.21.15.169:6801/2551075225', 'heartbeat_back_addr': '172.21.15.169:6803/2551075225', 'heartbeat_front_addr': '172.21.15.169:6802/2551075225', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '522d61df-8dc1-4915-83fd-f94d0d572c15', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6804', 'nonce': 2339555071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 2339555071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 2339555071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6806', 'nonce': 2339555071}]}, 'public_addr': '172.21.15.169:6804/2339555071', 'cluster_addr': '172.21.15.169:6805/2339555071', 'heartbeat_back_addr': '172.21.15.169:6807/2339555071', 'heartbeat_front_addr': '172.21.15.169:6806/2339555071', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '91f74dca-503a-45d1-9647-df4804d58515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6808', 'nonce': 391306610}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 391306610}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 391306610}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6810', 'nonce': 391306610}]}, 'public_addr': '172.21.15.169:6808/391306610', 'cluster_addr': '172.21.15.169:6809/391306610', 'heartbeat_back_addr': '172.21.15.169:6811/391306610', 'heartbeat_front_addr': '172.21.15.169:6810/391306610', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '788b4f47-eadf-4601-a5b4-51fb6f04fbdf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 93, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6812', 'nonce': 3696812305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3696812305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3696812305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6814', 'nonce': 3696812305}]}, 'public_addr': '172.21.15.169:6812/3696812305', 'cluster_addr': '172.21.15.169:6813/3696812305', 'heartbeat_back_addr': '172.21.15.169:6815/3696812305', 'heartbeat_front_addr': '172.21.15.169:6814/3696812305', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:11.067709+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:07.612364+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:09.926674+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:36.300947+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:35.423624+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:34.653370+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:46.940646+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:16:06.047188+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:28.261167+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:14:54.640616+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:15:07.789283+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:16:07.210334+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.121:0/2474849789': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/253689008': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:6800/3531151725': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:0/1778611055': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:6800/28718860': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/445290372': '2024-02-13T15:47:47.232761+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:16:08.736 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-02-12T16:16:08.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:08.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:08.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:08 smithi169 bash[19604]: cluster 2024-02-12T16:16:07.210268+0000 osd.11 (osd.11) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:08 smithi169 bash[19604]: cluster 2024-02-12T16:16:07.210332+0000 osd.11 (osd.11) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:08 smithi169 bash[19604]: cluster 2024-02-12T16:16:07.424370+0000 mon.a (mon.0) 1452 : cluster [DBG] osdmap e381: 12 total, 12 up, 12 in 2024-02-12T16:16:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:08 smithi169 bash[19604]: cluster 2024-02-12T16:16:07.729417+0000 mgr.y (mgr.14146) 924 : cluster [DBG] pgmap v1206: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:16:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:08 smithi169 bash[19604]: audit 2024-02-12T16:16:07.881784+0000 mon.b (mon.2) 121 : audit [DBG] from='client.? v1:172.21.15.121:0/3279972325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:16:08.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:08.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:08.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:09.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:09.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:09.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:09.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:09.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:09.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:09.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:09.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:09.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:09.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:09.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:09.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:09.611 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:09.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:09.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:09.738 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:09.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:09.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:09.868 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:09.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:09.956 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:16:09.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:10.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:10.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:10.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:10.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:10.263 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:10.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:10.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:10.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:10.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:10.651 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:10.683 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:10.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:10.814 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:10.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:10.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:10.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:10 smithi169 bash[19604]: cluster 2024-02-12T16:16:09.729874+0000 mgr.y (mgr.14146) 925 : cluster [DBG] pgmap v1207: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 917 op/s 2024-02-12T16:16:11.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:11.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:10 smithi121 bash[18489]: cluster 2024-02-12T16:16:09.729874+0000 mgr.y (mgr.14146) 925 : cluster [DBG] pgmap v1207: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 917 op/s 2024-02-12T16:16:11.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:11.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:11.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:10 smithi123 bash[19119]: cluster 2024-02-12T16:16:09.729874+0000 mgr.y (mgr.14146) 925 : cluster [DBG] pgmap v1207: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 917 op/s 2024-02-12T16:16:11.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.206 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:11.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:11.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:11.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:11.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:11.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:11.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:11.608 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:11.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:11.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.739 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:11.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:11.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:11.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.993156+0000 mon.a (mon.0) 1453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08745526901584448]}]: dispatch 2024-02-12T16:16:11.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.993514+0000 mon.a (mon.0) 1454 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08713182629648165]}]: dispatch 2024-02-12T16:16:11.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.993761+0000 mon.a (mon.0) 1455 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.993995+0000 mon.a (mon.0) 1456 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.994184+0000 mon.a (mon.0) 1457 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.994387+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.994558+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08719649589197342]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.994745+0000 mon.a (mon.0) 1460 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08716775267987953]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.994942+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08788689441573246]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.995151+0000 mon.a (mon.0) 1462 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08714722295988081]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.995338+0000 mon.a (mon.0) 1463 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08714722295988081]}]: dispatch 2024-02-12T16:16:11.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:11 smithi121 bash[18489]: audit 2024-02-12T16:16:10.995515+0000 mon.a (mon.0) 1464 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716775267987953]}]: dispatch 2024-02-12T16:16:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:11.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:11.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:12.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.993156+0000 mon.a (mon.0) 1453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08745526901584448]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.993514+0000 mon.a (mon.0) 1454 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08713182629648165]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.993761+0000 mon.a (mon.0) 1455 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.993995+0000 mon.a (mon.0) 1456 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.994184+0000 mon.a (mon.0) 1457 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.994387+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.994558+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08719649589197342]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.994745+0000 mon.a (mon.0) 1460 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08716775267987953]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.994942+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08788689441573246]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.995151+0000 mon.a (mon.0) 1462 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08714722295988081]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.995338+0000 mon.a (mon.0) 1463 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08714722295988081]}]: dispatch 2024-02-12T16:16:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:11 smithi169 bash[19604]: audit 2024-02-12T16:16:10.995515+0000 mon.a (mon.0) 1464 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716775267987953]}]: dispatch 2024-02-12T16:16:12.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:12.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.993156+0000 mon.a (mon.0) 1453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08745526901584448]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.993514+0000 mon.a (mon.0) 1454 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08713182629648165]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.993761+0000 mon.a (mon.0) 1455 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.993995+0000 mon.a (mon.0) 1456 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.994184+0000 mon.a (mon.0) 1457 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.994387+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08731148745479929]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.994558+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08719649589197342]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.994745+0000 mon.a (mon.0) 1460 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08716775267987953]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.994942+0000 mon.a (mon.0) 1461 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08788689441573246]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.995151+0000 mon.a (mon.0) 1462 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08714722295988081]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.995338+0000 mon.a (mon.0) 1463 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08714722295988081]}]: dispatch 2024-02-12T16:16:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:11 smithi123 bash[19119]: audit 2024-02-12T16:16:10.995515+0000 mon.a (mon.0) 1464 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716775267987953]}]: dispatch 2024-02-12T16:16:12.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:12.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:12.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:12.392 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:12.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:12.520 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:12.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:12.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:12.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:12.784 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:12.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: cluster 2024-02-12T16:16:11.730245+0000 mgr.y (mgr.14146) 926 : cluster [DBG] pgmap v1208: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-02-12T16:16:12.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.792149+0000 mon.a (mon.0) 1465 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08745526901584448]}]': finished 2024-02-12T16:16:12.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.792217+0000 mon.a (mon.0) 1466 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08713182629648165]}]': finished 2024-02-12T16:16:12.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.792267+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.792309+0000 mon.a (mon.0) 1468 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.792352+0000 mon.a (mon.0) 1469 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.792711+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.792859+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08719649589197342]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.793078+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08716775267987953]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.793145+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08788689441573246]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.793265+0000 mon.a (mon.0) 1474 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08714722295988081]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.793316+0000 mon.a (mon.0) 1475 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08714722295988081]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: audit 2024-02-12T16:16:11.793393+0000 mon.a (mon.0) 1476 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716775267987953]}]': finished 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: cluster 2024-02-12T16:16:11.793457+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e382: 12 total, 12 up, 12 in 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: cluster 2024-02-12T16:16:11.861312+0000 osd.1 (osd.1) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:12.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:12 smithi121 bash[18489]: cluster 2024-02-12T16:16:11.861374+0000 osd.1 (osd.1) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:12.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:12.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:12.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:13.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.049 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:13.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: cluster 2024-02-12T16:16:11.730245+0000 mgr.y (mgr.14146) 926 : cluster [DBG] pgmap v1208: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-02-12T16:16:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.792149+0000 mon.a (mon.0) 1465 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08745526901584448]}]': finished 2024-02-12T16:16:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.792217+0000 mon.a (mon.0) 1466 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08713182629648165]}]': finished 2024-02-12T16:16:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.792267+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.792309+0000 mon.a (mon.0) 1468 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.792352+0000 mon.a (mon.0) 1469 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.792711+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.792859+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08719649589197342]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.793078+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08716775267987953]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.793145+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08788689441573246]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.793265+0000 mon.a (mon.0) 1474 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08714722295988081]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.793316+0000 mon.a (mon.0) 1475 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08714722295988081]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: audit 2024-02-12T16:16:11.793393+0000 mon.a (mon.0) 1476 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716775267987953]}]': finished 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: cluster 2024-02-12T16:16:11.793457+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e382: 12 total, 12 up, 12 in 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: cluster 2024-02-12T16:16:11.861312+0000 osd.1 (osd.1) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:12 smithi123 bash[19119]: cluster 2024-02-12T16:16:11.861374+0000 osd.1 (osd.1) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:13.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: cluster 2024-02-12T16:16:11.730245+0000 mgr.y (mgr.14146) 926 : cluster [DBG] pgmap v1208: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-02-12T16:16:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.792149+0000 mon.a (mon.0) 1465 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08745526901584448]}]': finished 2024-02-12T16:16:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.792217+0000 mon.a (mon.0) 1466 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08713182629648165]}]': finished 2024-02-12T16:16:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.792267+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.792309+0000 mon.a (mon.0) 1468 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.792352+0000 mon.a (mon.0) 1469 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.159 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.792711+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08731148745479929]}]': finished 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.792859+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08719649589197342]}]': finished 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.793078+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08716775267987953]}]': finished 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.793145+0000 mon.a (mon.0) 1473 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08788689441573246]}]': finished 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.793265+0000 mon.a (mon.0) 1474 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08714722295988081]}]': finished 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.793316+0000 mon.a (mon.0) 1475 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08714722295988081]}]': finished 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: audit 2024-02-12T16:16:11.793393+0000 mon.a (mon.0) 1476 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08716775267987953]}]': finished 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: cluster 2024-02-12T16:16:11.793457+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e382: 12 total, 12 up, 12 in 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: cluster 2024-02-12T16:16:11.861312+0000 osd.1 (osd.1) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:13.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:12 smithi169 bash[19604]: cluster 2024-02-12T16:16:11.861374+0000 osd.1 (osd.1) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:13.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:13.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:13.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:13.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:13.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:13.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:13.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:13.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:13.581 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:13.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:13.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:13.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:13.737 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:16:13.737 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:16:13.737 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-12T16:16:13.738 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:16:13.738 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:16:13.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-12T16:16:13.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:13.871 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:13 smithi121 bash[18489]: cluster 2024-02-12T16:16:12.795854+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e383: 12 total, 12 up, 12 in 2024-02-12T16:16:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:13.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:13.973 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:14.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:14.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:13 smithi169 bash[19604]: cluster 2024-02-12T16:16:12.795854+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e383: 12 total, 12 up, 12 in 2024-02-12T16:16:14.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:14.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:13 smithi123 bash[19119]: cluster 2024-02-12T16:16:12.795854+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e383: 12 total, 12 up, 12 in 2024-02-12T16:16:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:14.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:14.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:14.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:14.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:14.427 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:16:14.427 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:16:14.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:14.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:14.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:14.658 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:14.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:14.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:14.863 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:16:14.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:14 smithi123 bash[19119]: cluster 2024-02-12T16:16:13.730661+0000 mgr.y (mgr.14146) 927 : cluster [DBG] pgmap v1211: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s 2024-02-12T16:16:14.889 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:16:14.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:14.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:14.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:14.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:14.918 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:16:14.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:16:14.998 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:16:15.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:15.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:14 smithi121 bash[18489]: cluster 2024-02-12T16:16:13.730661+0000 mgr.y (mgr.14146) 927 : cluster [DBG] pgmap v1211: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s 2024-02-12T16:16:15.038 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:16:15.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:15.075 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:16:15.110 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:16:15.143 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:16:15.163 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:15.170 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:16:15.186 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:15.198 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:16:15.230 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.232 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.233 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.233 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.233 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:16:15.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:14 smithi169 bash[19604]: cluster 2024-02-12T16:16:13.730661+0000 mgr.y (mgr.14146) 927 : cluster [DBG] pgmap v1211: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s 2024-02-12T16:16:15.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:15.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-12T16:16:15.417 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:16:15.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:15.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:15.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:15.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:15.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:15.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:15.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:15.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: cluster 2024-02-12T16:16:14.766776+0000 osd.9 (osd.9) 42 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: cluster 2024-02-12T16:16:14.766825+0000 osd.9 (osd.9) 43 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: cluster 2024-02-12T16:16:14.813808+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e384: 12 total, 12 up, 12 in 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: audit 2024-02-12T16:16:14.822771+0000 mon.a (mon.0) 1480 : audit [DBG] from='client.? v1:172.21.15.121:0/1909693652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: audit 2024-02-12T16:16:15.102254+0000 mon.a (mon.0) 1481 : audit [INF] from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: audit 2024-02-12T16:16:15.104853+0000 mon.b (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.121:0/2768025522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: cluster 2024-02-12T16:16:15.400337+0000 mon.a (mon.0) 1482 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: audit 2024-02-12T16:16:15.405020+0000 mon.a (mon.0) 1483 : audit [INF] from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:16:15.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:15 smithi169 bash[19604]: cluster 2024-02-12T16:16:15.405068+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e385: 12 total, 12 up, 12 in 2024-02-12T16:16:15.991 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:15.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:16.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:16.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: cluster 2024-02-12T16:16:14.766776+0000 osd.9 (osd.9) 42 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:16.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: cluster 2024-02-12T16:16:14.766825+0000 osd.9 (osd.9) 43 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:16.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: cluster 2024-02-12T16:16:14.813808+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e384: 12 total, 12 up, 12 in 2024-02-12T16:16:16.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: audit 2024-02-12T16:16:14.822771+0000 mon.a (mon.0) 1480 : audit [DBG] from='client.? v1:172.21.15.121:0/1909693652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:16.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: audit 2024-02-12T16:16:15.102254+0000 mon.a (mon.0) 1481 : audit [INF] from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:16.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: audit 2024-02-12T16:16:15.104853+0000 mon.b (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.121:0/2768025522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:16.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: cluster 2024-02-12T16:16:15.400337+0000 mon.a (mon.0) 1482 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:16.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: audit 2024-02-12T16:16:15.405020+0000 mon.a (mon.0) 1483 : audit [INF] from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:16:16.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:15 smithi121 bash[18489]: cluster 2024-02-12T16:16:15.405068+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e385: 12 total, 12 up, 12 in 2024-02-12T16:16:16.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:16.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: cluster 2024-02-12T16:16:14.766776+0000 osd.9 (osd.9) 42 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: cluster 2024-02-12T16:16:14.766825+0000 osd.9 (osd.9) 43 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: cluster 2024-02-12T16:16:14.813808+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e384: 12 total, 12 up, 12 in 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: audit 2024-02-12T16:16:14.822771+0000 mon.a (mon.0) 1480 : audit [DBG] from='client.? v1:172.21.15.121:0/1909693652' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: audit 2024-02-12T16:16:15.102254+0000 mon.a (mon.0) 1481 : audit [INF] from='client.25457 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: audit 2024-02-12T16:16:15.104853+0000 mon.b (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.121:0/2768025522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: cluster 2024-02-12T16:16:15.400337+0000 mon.a (mon.0) 1482 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: audit 2024-02-12T16:16:15.405020+0000 mon.a (mon.0) 1483 : audit [INF] from='client.25457 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:16:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:15 smithi123 bash[19119]: cluster 2024-02-12T16:16:15.405068+0000 mon.a (mon.0) 1484 : cluster [DBG] osdmap e385: 12 total, 12 up, 12 in 2024-02-12T16:16:16.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:16.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:16.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:16.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:16.377 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:16.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:16.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:16.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:16.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:16.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:16.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:16.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:16.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:16.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:16.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:16.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:16 smithi121 bash[18489]: cluster 2024-02-12T16:16:15.731138+0000 mgr.y (mgr.14146) 928 : cluster [DBG] pgmap v1214: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-02-12T16:16:16.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:16.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:17.018 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:17.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:16 smithi169 bash[19604]: cluster 2024-02-12T16:16:15.731138+0000 mgr.y (mgr.14146) 928 : cluster [DBG] pgmap v1214: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-02-12T16:16:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:16 smithi123 bash[19119]: cluster 2024-02-12T16:16:15.731138+0000 mgr.y (mgr.14146) 928 : cluster [DBG] pgmap v1214: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-02-12T16:16:17.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:17.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:17.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:17.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:17.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:17.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:17.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:17.576 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:17.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:17.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:17.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:17.842 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:17.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:17.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:17.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:18.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:18.110 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:18.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:18.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:18.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:18.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:18.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:18.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:18.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:18.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:18.520 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:18.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:18.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:18.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:16:18.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:18.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:18.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:16:18.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:18.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:18.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:18.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:18 smithi121 bash[18489]: cluster 2024-02-12T16:16:17.731593+0000 mgr.y (mgr.14146) 929 : cluster [DBG] pgmap v1215: 27 pgs: 27 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-02-12T16:16:18.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:18.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:18 smithi123 bash[19119]: cluster 2024-02-12T16:16:17.731593+0000 mgr.y (mgr.14146) 929 : cluster [DBG] pgmap v1215: 27 pgs: 27 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-02-12T16:16:18.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:18.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:19.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:19.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:18 smithi169 bash[19604]: cluster 2024-02-12T16:16:17.731593+0000 mgr.y (mgr.14146) 929 : cluster [DBG] pgmap v1215: 27 pgs: 27 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-02-12T16:16:19.055 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:19.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:19.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:19.180 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:19.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:19.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:19.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:19.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:19.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:19.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:19.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:19.482 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:19.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:19.588 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:19.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:19.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:19.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:19.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:19.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:19.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:19.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:19.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:20.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:20.041 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:20.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:20.157 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:16:20.167 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:20.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:20.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:20.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:20.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:20.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:20.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:20.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:20.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:20.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:20.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:20.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:20 smithi121 bash[18489]: cluster 2024-02-12T16:16:19.732052+0000 mgr.y (mgr.14146) 930 : cluster [DBG] pgmap v1216: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s 2024-02-12T16:16:20.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:20.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:20 smithi169 bash[19604]: cluster 2024-02-12T16:16:19.732052+0000 mgr.y (mgr.14146) 930 : cluster [DBG] pgmap v1216: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s 2024-02-12T16:16:21.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:21.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:20 smithi123 bash[19119]: cluster 2024-02-12T16:16:19.732052+0000 mgr.y (mgr.14146) 930 : cluster [DBG] pgmap v1216: 27 pgs: 27 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s 2024-02-12T16:16:21.124 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:21.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:21.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:21.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:21.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:21.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:21.389 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:21.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:21.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:21.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:21.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:21.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:21.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:21.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:21.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:21.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:21.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:21.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:21.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:21.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:22.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:22.070 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:22.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:22.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:22.207 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:22.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:22.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:22.343 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:22.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:22.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:22.474 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:22.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:22.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:22.605 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:22.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:22.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:22.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:22.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:22.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:22.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:22.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:22 smithi121 bash[18489]: cluster 2024-02-12T16:16:21.732539+0000 mgr.y (mgr.14146) 931 : cluster [DBG] pgmap v1217: 27 pgs: 27 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:16:22.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:23.015 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:23.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:23.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:23.181 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:22 smithi123 bash[19119]: cluster 2024-02-12T16:16:21.732539+0000 mgr.y (mgr.14146) 931 : cluster [DBG] pgmap v1217: 27 pgs: 27 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:16:23.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:23.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:23.230 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:23.231 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:23.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:22 smithi169 bash[19604]: cluster 2024-02-12T16:16:21.732539+0000 mgr.y (mgr.14146) 931 : cluster [DBG] pgmap v1217: 27 pgs: 27 active+clean; 7.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:16:23.319 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:16:23.320 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "" 2024-02-12T16:16:23.320 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:16:23.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:23.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:23.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:23.452 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:23.482 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:23.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:23.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:23.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:23.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:23.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:23.771 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:23.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:23.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:23.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:23.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:24.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:24.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:24.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:24.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:24.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:24.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:24.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:24.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:24.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15108.0:24083 3.0 e262/256 3:0f0d65c3:::benchmark_data_smithi169_51390_object24082:head v 262'1521, mlcod=262'1519)", 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:14.207659+0000", 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 430.12360158400003, 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24796887400000001, 2024-02-12T16:16:24.340 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207659+0000", 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207659+0000", 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207657+0000", 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5877999999999999e-05 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207683+0000", 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0091e-05 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207693+0000", 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2389999999999999e-06 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207699+0000", 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8883e-05 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207738+0000", 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1916000000000001e-05 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.207750+0000", 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.247786915 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.455537+0000", 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0620999999999999e-05 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.455627+0000", 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24796887400000001 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.15108.0:24091 3.2 e262/256 3:45d66431:::benchmark_data_smithi169_51390_object24090:head v 262'1466, mlcod=262'1464)", 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:09:14.209311+0000", 2024-02-12T16:16:24.344 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 430.12194870600001, 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24952506399999999, 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209311+0000", 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209311+0000", 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209310+0000", 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9494999999999998e-05 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.209380+0000", 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24128054600000001 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450660+0000", 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.385e-05 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450684+0000", 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012295700000000001 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450807+0000", 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0756000000000001e-05 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.450848+0000", 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079541049999999995 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.458802+0000", 2024-02-12T16:16:24.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4740000000000003e-05 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:09:14.458837+0000", 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24952506399999999 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86335 4.6 4:67d81edf:::benchmark_data_smithi169_60938_object86334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.312762+0000", 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 117.018498347, 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197120395, 2024-02-12T16:16:24.348 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86335 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312762+0000", 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312762+0000", 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312760+0000", 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0772999999999999e-05 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312781+0000", 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4519999999999997e-06 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312790+0000", 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9189999999999998e-06 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312794+0000", 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00063509799999999996 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.313430+0000", 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001055 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.313535+0000", 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013436500000000001 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.313669+0000", 2024-02-12T16:16:24.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003412689 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.317082+0000", 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.192655664 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509738+0000", 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0968000000000002e-05 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509809+0000", 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3480999999999997e-05 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509882+0000", 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197120395 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86337 4.6 4:6b8db7f7:::benchmark_data_smithi169_60938_object86336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.312806+0000", 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 117.018454495, 2024-02-12T16:16:24.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197206625, 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86337 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312806+0000", 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312806+0000", 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312805+0000", 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7529000000000001e-05 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312823+0000", 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2400000000000004e-06 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312829+0000", 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3109999999999999e-06 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312833+0000", 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029173200000000002 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.315750+0000", 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6951000000000003e-05 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.315817+0000", 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8974000000000006e-05 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.315906+0000", 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012923069999999999 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.317199+0000", 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.192730133 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509929+0000", 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3449999999999999e-05 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509972+0000", 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0093999999999999e-05 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510012+0000", 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197206625 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.358 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86348 4.6 4:690a34d8:::benchmark_data_smithi169_60938_object86347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.322481+0000", 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 117.008778842, 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.187767985, 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86348 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.359 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322481+0000", 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322481+0000", 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322480+0000", 2024-02-12T16:16:24.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4097999999999999e-05 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322504+0000", 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1962e-05 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322516+0000", 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-06 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322522+0000", 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6094999999999997e-05 2024-02-12T16:16:24.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322579+0000", 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1720999999999999e-05 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322660+0000", 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010755399999999999 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322768+0000", 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020081309999999998 2024-02-12T16:16:24.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.324776+0000", 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18531487399999999 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510091+0000", 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8924000000000004e-05 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510180+0000", 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9573000000000006e-05 2024-02-12T16:16:24.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510249+0000", 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.187767985 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86368 4.2 4:42a37998:::benchmark_data_smithi169_60938_object86367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.334423+0000", 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 116.996837236, 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18501995099999999, 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.364 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86368 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334423+0000", 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334423+0000", 2024-02-12T16:16:24.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334421+0000", 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5689999999999999e-05 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334457+0000", 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0488e-05 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334467+0000", 2024-02-12T16:16:24.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0270000000000001e-06 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334473+0000", 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17526112199999999 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509735+0000", 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00035024900000000001 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510085+0000", 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00067824599999999997 2024-02-12T16:16:24.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510763+0000", 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0085641840000000007 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519327+0000", 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6293000000000001e-05 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519344+0000", 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4197999999999998e-05 2024-02-12T16:16:24.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519398+0000", 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5169000000000001e-05 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519443+0000", 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18501995099999999 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.369 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171789 4.6 4:6ca60eb7:::benchmark_data_smithi169_60938_object171788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.709418+0000", 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 57.621842510999997, 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20106019, 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171789 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709418+0000", 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709418+0000", 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709415+0000", 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5406999999999999e-05 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709440+0000", 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4986e-05 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709455+0000", 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9490000000000001e-06 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709461+0000", 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5544999999999998e-05 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709517+0000", 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014442200000000001 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709661+0000", 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018882299999999999 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709850+0000", 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049930249999999999 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.714843+0000", 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.195279853 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.910123+0000", 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000177753 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.910301+0000", 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017720600000000001 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.910478+0000", 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20106019 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171799 4.6 4:6cb0dab3:::benchmark_data_smithi169_60938_object171798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.719308+0000", 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 57.611952119000001, 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22523388999999999, 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171799 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719308+0000", 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719308+0000", 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719305+0000", 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8177999999999998e-05 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719353+0000", 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5305e-05 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719369+0000", 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0270000000000001e-06 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719375+0000", 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6123000000000004e-05 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719441+0000", 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000177877 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719619+0000", 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000179975 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719799+0000", 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079059779999999993 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.727705+0000", 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21674727599999999 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944452+0000", 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6431000000000003e-05 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944498+0000", 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3581999999999997e-05 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944542+0000", 2024-02-12T16:16:24.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22523388999999999 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171800 4.6 4:631aee4e:::benchmark_data_smithi169_60938_object171799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.719430+0000", 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 57.611830636999997, 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.225286442, 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171800 2024-02-12T16:16:24.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719430+0000", 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719430+0000", 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719427+0000", 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7589999999999998e-05 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719455+0000", 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0733999999999999e-05 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719466+0000", 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0770000000000008e-06 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719474+0000", 2024-02-12T16:16:24.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045064959999999996 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.723980+0000", 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4834999999999996e-05 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.724075+0000", 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117805 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.724193+0000", 2024-02-12T16:16:24.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003681981 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.727875+0000", 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.216733237 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944608+0000", 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4753999999999997e-05 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944673+0000", 2024-02-12T16:16:24.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3019999999999998e-05 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944716+0000", 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.225286442 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171812 4.6 4:6e56c86e:::benchmark_data_smithi169_60938_object171811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.745502+0000", 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 57.585758597999998, 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19937315999999999, 2024-02-12T16:16:24.385 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171812 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745502+0000", 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745502+0000", 2024-02-12T16:16:24.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745500+0000", 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2982000000000001e-05 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745523+0000", 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3764000000000001e-05 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745536+0000", 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5720000000000002e-06 2024-02-12T16:16:24.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745542+0000", 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6146999999999998e-05 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745578+0000", 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4498000000000003e-05 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745663+0000", 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8611999999999997e-05 2024-02-12T16:16:24.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745761+0000", 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037280770000000002 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.749489+0000", 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19529492000000001 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944784+0000", 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0645e-05 2024-02-12T16:16:24.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944835+0000", 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9860000000000001e-05 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944875+0000", 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19937315999999999 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.390 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204304 4.6 4:6e94d6d4:::benchmark_data_smithi169_60938_object30967:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:55.988179+0000", 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.343081432000002, 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66636980899999998, 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204304 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988179+0000", 2024-02-12T16:16:24.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988179+0000", 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988177+0000", 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5560000000000002e-06 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988184+0000", 2024-02-12T16:16:24.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8339999999999994e-06 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988194+0000", 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5600000000000001e-06 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988199+0000", 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022521169000000001 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010721+0000", 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2280000000000005e-05 2024-02-12T16:16:24.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010803+0000", 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5997999999999999e-05 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010889+0000", 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014777161 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025666+0000", 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62878948999999995 2024-02-12T16:16:24.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654456+0000", 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1832000000000002e-05 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654497+0000", 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1159999999999998e-05 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654549+0000", 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66636980899999998 2024-02-12T16:16:24.395 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204305 4.6 4:66ec962d:::benchmark_data_smithi169_60938_object30968:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:55.988325+0000", 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.342935139000001, 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66632389999999997, 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204305 2024-02-12T16:16:24.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988325+0000", 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988325+0000", 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988324+0000", 2024-02-12T16:16:24.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5495999999999999e-05 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988339+0000", 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4069999999999993e-06 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988347+0000", 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6090000000000003e-06 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988352+0000", 2024-02-12T16:16:24.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.026163005999999999 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.014515+0000", 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011727600000000001 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.014632+0000", 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5241999999999994e-05 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.014728+0000", 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011066411 2024-02-12T16:16:24.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025794+0000", 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62879766199999998 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654592+0000", 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0914000000000003e-05 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654623+0000", 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6397000000000002e-05 2024-02-12T16:16:24.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654649+0000", 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66632389999999997 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204327 4.6 4:6a1a5157:::benchmark_data_smithi169_60938_object30990:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.010343+0000", 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.320917475000002, 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66292315300000004, 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.401 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204327 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010343+0000", 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010343+0000", 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010341+0000", 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6649999999999997e-06 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010348+0000", 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1185e-05 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010359+0000", 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5560000000000004e-06 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010365+0000", 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082932909999999995 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.018659+0000", 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7928000000000003e-05 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.018737+0000", 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6607000000000005e-05 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.018833+0000", 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069676490000000002 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025801+0000", 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.64736806700000005 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.673169+0000", 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1934000000000002e-05 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.673211+0000", 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5004999999999999e-05 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.673266+0000", 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66292315300000004 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204328 4.6 4:6bbed11b:::benchmark_data_smithi169_60938_object30991:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.010503+0000", 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.320757463, 2024-02-12T16:16:24.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66864712000000004, 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204328 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010503+0000", 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010503+0000", 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010501+0000", 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.674e-06 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010508+0000", 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0327e-05 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010518+0000", 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5882000000000001e-05 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010544+0000", 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011838392999999999 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.022382+0000", 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129247 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.022511+0000", 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0579000000000001e-05 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.022602+0000", 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032876149999999998 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025890+0000", 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65316824299999998 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679058+0000", 2024-02-12T16:16:24.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9817999999999997e-05 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679098+0000", 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2179000000000003e-05 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679150+0000", 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66864712000000004 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204339 4.6 4:6432c7bb:::benchmark_data_smithi169_60938_object31002:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.032281+0000", 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.298978781999999, 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.647598381, 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204339 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032281+0000", 2024-02-12T16:16:24.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032281+0000", 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032279+0000", 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2420000000000003e-06 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032287+0000", 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0118999999999999e-05 2024-02-12T16:16:24.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032297+0000", 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1210000000000001e-06 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032304+0000", 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033154349999999998 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.035619+0000", 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0241000000000006e-05 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.035699+0000", 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0716000000000004e-05 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.035790+0000", 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005813609 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.041604+0000", 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63818005300000002 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679784+0000", 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4894999999999997e-05 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679819+0000", 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1005999999999998e-05 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679880+0000", 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.647598381 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.416 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204341 4.6 4:63b4452c:::benchmark_data_smithi169_60938_object31004:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.038877+0000", 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.292383162, 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65003887000000005, 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204341 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.417 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038877+0000", 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038877+0000", 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038874+0000", 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5920000000000003e-06 2024-02-12T16:16:24.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038882+0000", 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2948e-05 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038895+0000", 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0270000000000001e-06 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038901+0000", 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00044145399999999998 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.039342+0000", 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5527999999999999e-05 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.039418+0000", 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2207999999999996e-05 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.039500+0000", 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022929280000000001 2024-02-12T16:16:24.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.041793+0000", 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.64697973399999997 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.688773+0000", 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4234999999999997e-05 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.688847+0000", 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9132000000000002e-05 2024-02-12T16:16:24.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.688916+0000", 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65003887000000005 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:205428 4.6 4:6d427ab9:::benchmark_data_smithi169_60938_object32091:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:57.662172+0000", 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.669087849, 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22755656099999999, 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.422 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 205428 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662172+0000", 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662172+0000", 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:16:24.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662170+0000", 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7810000000000005e-06 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662178+0000", 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2616000000000001e-05 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662190+0000", 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9349999999999997e-06 2024-02-12T16:16:24.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662197+0000", 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.078460746999999997 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.740658+0000", 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8795999999999995e-05 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.740747+0000", 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0286999999999998e-05 2024-02-12T16:16:24.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.740837+0000", 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012130958000000001 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.752968+0000", 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13665096099999999 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889619+0000", 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2207999999999998e-05 2024-02-12T16:16:24.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889681+0000", 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7828999999999999e-05 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889729+0000", 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22755656099999999 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.427 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:205451 4.6 4:6f821f24:::benchmark_data_smithi169_60938_object32114:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:57.683927+0000", 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.647332953999999, 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20594626199999999, 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 205451 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683927+0000", 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683927+0000", 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683925+0000", 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8709999999999997e-06 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683933+0000", 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3066e-05 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683946+0000", 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4609999999999996e-06 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683952+0000", 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060822810999999997 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.744775+0000", 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000128672 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.744904+0000", 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016349499999999999 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.745067+0000", 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011550878000000001 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.756618+0000", 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13317361799999999 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889792+0000", 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5293999999999998e-05 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889827+0000", 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6288999999999997e-05 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889873+0000", 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20594626199999999 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:207829 4.6 4:6d04706a:::benchmark_data_smithi169_60938_object34492:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.069986+0000", 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.261273997, 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.188431816, 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.433 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 207829 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069986+0000", 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069986+0000", 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069984+0000", 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3629999999999999e-06 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069991+0000", 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069999+0000", 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6959999999999996e-06 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.070004+0000", 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.091331159999999995 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.161335+0000", 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015471500000000001 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.161490+0000", 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017501499999999999 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.161665+0000", 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065059690000000003 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.168171+0000", 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.090183764 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.258355+0000", 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6961000000000001e-05 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.258391+0000", 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6519000000000001e-05 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.258418+0000", 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.188431816 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:208988 4.6 4:69363d5c:::benchmark_data_smithi169_60938_object35651:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:01.468089+0000", 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.863171433000002, 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19592237500000001, 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 208988 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:24.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468089+0000", 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468089+0000", 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468086+0000", 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2191000000000001e-05 2024-02-12T16:16:24.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468099+0000", 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5423000000000001e-05 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468114+0000", 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0219999999999999e-06 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468121+0000", 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.103600845 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.571722+0000", 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9457999999999997e-05 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.571801+0000", 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3430000000000006e-05 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.571885+0000", 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013916245000000001 2024-02-12T16:16:24.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.585801+0000", 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.078146683999999994 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.663948+0000", 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5927999999999999e-05 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.663984+0000", 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7572e-05 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:24.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.664011+0000", 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19592237500000001 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:24.444 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:16:24.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:24.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:24.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:24.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:24.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:24.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:24.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:24.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:24.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:24.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:24.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:24.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:25.006 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:25.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:24 smithi121 bash[18489]: cluster 2024-02-12T16:16:23.732940+0000 mgr.y (mgr.14146) 932 : cluster [DBG] pgmap v1218: 27 pgs: 27 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-02-12T16:16:25.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:25.111 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:24 smithi169 bash[19604]: cluster 2024-02-12T16:16:23.732940+0000 mgr.y (mgr.14146) 932 : cluster [DBG] pgmap v1218: 27 pgs: 27 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-02-12T16:16:25.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-12T16:16:25.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:24 smithi123 bash[19119]: cluster 2024-02-12T16:16:23.732940+0000 mgr.y (mgr.14146) 932 : cluster [DBG] pgmap v1218: 27 pgs: 27 active+clean; 7.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-02-12T16:16:25.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:25.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:25.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:25.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:25.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:25.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:25.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:25.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:25.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:25.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:25.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:25.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:25.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:26.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:26.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:26.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:26.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:26.225 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:26.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:26.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:26.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:26.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:26.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:26.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:26.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:26 smithi169 bash[19604]: cluster 2024-02-12T16:16:25.733383+0000 mgr.y (mgr.14146) 933 : cluster [DBG] pgmap v1219: 27 pgs: 27 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-02-12T16:16:26.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:26.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:26.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:26.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:26 smithi123 bash[19119]: cluster 2024-02-12T16:16:25.733383+0000 mgr.y (mgr.14146) 933 : cluster [DBG] pgmap v1219: 27 pgs: 27 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-02-12T16:16:26.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:26.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:26.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:26.873 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:26.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:26 smithi121 bash[18489]: cluster 2024-02-12T16:16:25.733383+0000 mgr.y (mgr.14146) 933 : cluster [DBG] pgmap v1219: 27 pgs: 27 active+clean; 7.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-02-12T16:16:26.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:26.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:27.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:27.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:27.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:27.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:27.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:27.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:27.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:27.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:27.670 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:27.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:27.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:27.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:27.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:27.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:28.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:28.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:28.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:28.231 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:28.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:28.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:28.365 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:28.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:28.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:28.507 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:16:28.508 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:28.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:28.532 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:16:28.567 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:16:28.599 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:16:28.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:28.629 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:16:28.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:28.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:28.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:16:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:16:28.725 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:16:28.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:28.757 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:16:28.788 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:16:28.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:28.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:28.826 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:16:28.858 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:16:28.860 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.860 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.860 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.860 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.860 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:16:28.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:28 smithi123 bash[19119]: cluster 2024-02-12T16:16:27.733746+0000 mgr.y (mgr.14146) 934 : cluster [DBG] pgmap v1220: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:16:28.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:28 smithi123 bash[19119]: audit 2024-02-12T16:16:28.471160+0000 mon.b (mon.2) 123 : audit [DBG] from='client.? v1:172.21.15.121:0/674750627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:28.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:28.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:28.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:29.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:29.031 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:28 smithi121 bash[18489]: cluster 2024-02-12T16:16:27.733746+0000 mgr.y (mgr.14146) 934 : cluster [DBG] pgmap v1220: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:16:29.031 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:28 smithi121 bash[18489]: audit 2024-02-12T16:16:28.471160+0000 mon.b (mon.2) 123 : audit [DBG] from='client.? v1:172.21.15.121:0/674750627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:28 smithi169 bash[19604]: cluster 2024-02-12T16:16:27.733746+0000 mgr.y (mgr.14146) 934 : cluster [DBG] pgmap v1220: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:16:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:28 smithi169 bash[19604]: audit 2024-02-12T16:16:28.471160+0000 mon.b (mon.2) 123 : audit [DBG] from='client.? v1:172.21.15.121:0/674750627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:29.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:29.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:29.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:29.200 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:29.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:29.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:29.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:29.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:29.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:29.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:29.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:29.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:29.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:29.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:29.715 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:29.739 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:29.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:29.795 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:16:29.795 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:16:29.795 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.10392998258262173 2024-02-12T16:16:29.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 1 0.10392998258262173 2024-02-12T16:16:29.806 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:16:29.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:29.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:29 smithi121 bash[18489]: audit 2024-02-12T16:16:28.856115+0000 mon.a (mon.0) 1485 : audit [INF] from='client.25463 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:16:29.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:29 smithi121 bash[18489]: audit 2024-02-12T16:16:28.858523+0000 mon.b (mon.2) 124 : audit [INF] from='client.? v1:172.21.15.121:0/1017153852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:16:29.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:29.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:29.921 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:16:29.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:29.995 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:29 smithi123 bash[19119]: audit 2024-02-12T16:16:28.856115+0000 mon.a (mon.0) 1485 : audit [INF] from='client.25463 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:16:29.995 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:29 smithi123 bash[19119]: audit 2024-02-12T16:16:28.858523+0000 mon.b (mon.2) 124 : audit [INF] from='client.? v1:172.21.15.121:0/1017153852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:16:30.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:30.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:29 smithi169 bash[19604]: audit 2024-02-12T16:16:28.856115+0000 mon.a (mon.0) 1485 : audit [INF] from='client.25463 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:16:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:29 smithi169 bash[19604]: audit 2024-02-12T16:16:28.858523+0000 mon.b (mon.2) 124 : audit [INF] from='client.? v1:172.21.15.121:0/1017153852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:16:30.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:30.150 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:30.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:30.278 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:30.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:30.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:30.517 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:30.549 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:30.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:30.692 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:30.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:30.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:30.824 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:30.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:30.962 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:30.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:30 smithi169 bash[19604]: cluster 2024-02-12T16:16:29.734150+0000 mgr.y (mgr.14146) 935 : cluster [DBG] pgmap v1221: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-12T16:16:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:30 smithi169 bash[19604]: cluster 2024-02-12T16:16:29.788027+0000 mon.a (mon.0) 1486 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:30 smithi169 bash[19604]: audit 2024-02-12T16:16:29.794997+0000 mon.a (mon.0) 1487 : audit [INF] from='client.25463 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:16:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:30 smithi169 bash[19604]: cluster 2024-02-12T16:16:29.795115+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e386: 12 total, 12 up, 12 in 2024-02-12T16:16:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:30 smithi169 bash[19604]: cluster 2024-02-12T16:16:30.407941+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e387: 12 total, 12 up, 12 in 2024-02-12T16:16:31.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:31.073 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:30 smithi123 bash[19119]: cluster 2024-02-12T16:16:29.734150+0000 mgr.y (mgr.14146) 935 : cluster [DBG] pgmap v1221: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-12T16:16:31.073 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:30 smithi123 bash[19119]: cluster 2024-02-12T16:16:29.788027+0000 mon.a (mon.0) 1486 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:31.073 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:30 smithi123 bash[19119]: audit 2024-02-12T16:16:29.794997+0000 mon.a (mon.0) 1487 : audit [INF] from='client.25463 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:16:31.073 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:30 smithi123 bash[19119]: cluster 2024-02-12T16:16:29.795115+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e386: 12 total, 12 up, 12 in 2024-02-12T16:16:31.074 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:30 smithi123 bash[19119]: cluster 2024-02-12T16:16:30.407941+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e387: 12 total, 12 up, 12 in 2024-02-12T16:16:31.090 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:31.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:30 smithi121 bash[18489]: cluster 2024-02-12T16:16:29.734150+0000 mgr.y (mgr.14146) 935 : cluster [DBG] pgmap v1221: 27 pgs: 27 active+clean; 6.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-12T16:16:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:30 smithi121 bash[18489]: cluster 2024-02-12T16:16:29.788027+0000 mon.a (mon.0) 1486 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:16:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:30 smithi121 bash[18489]: audit 2024-02-12T16:16:29.794997+0000 mon.a (mon.0) 1487 : audit [INF] from='client.25463 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:16:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:30 smithi121 bash[18489]: cluster 2024-02-12T16:16:29.795115+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e386: 12 total, 12 up, 12 in 2024-02-12T16:16:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:30 smithi121 bash[18489]: cluster 2024-02-12T16:16:30.407941+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e387: 12 total, 12 up, 12 in 2024-02-12T16:16:31.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:31.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:31.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:31.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:31.350 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:31.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:31.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:31.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:31.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:31.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:31.611 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:31.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:31.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:31.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:31.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:31.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:31.877 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:31.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:31.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:16:31.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:32.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:32.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:32.151 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:32.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:32.283 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:32.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:32.413 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:32.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:32.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:32.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:32.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:32.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:16:32.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:32.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:32.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:32.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:32 smithi169 bash[19604]: cluster 2024-02-12T16:16:31.734557+0000 mgr.y (mgr.14146) 936 : cluster [DBG] pgmap v1224: 27 pgs: 27 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-02-12T16:16:32.984 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:32.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:33.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:33.116 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:33.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:32 smithi123 bash[19119]: cluster 2024-02-12T16:16:31.734557+0000 mgr.y (mgr.14146) 936 : cluster [DBG] pgmap v1224: 27 pgs: 27 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-02-12T16:16:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:32 smithi121 bash[18489]: cluster 2024-02-12T16:16:31.734557+0000 mgr.y (mgr.14146) 936 : cluster [DBG] pgmap v1224: 27 pgs: 27 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-02-12T16:16:33.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:33.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:33.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:33.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:33.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:33.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:33.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:33.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:33.524 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:33.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:33.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:33.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:33.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:33.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:33.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:33.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:33 smithi123 bash[19119]: cluster 2024-02-12T16:16:31.962713+0000 osd.9 (osd.9) 44 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:16:33.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:33.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:33.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:34.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:34.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:33 smithi169 bash[19604]: cluster 2024-02-12T16:16:31.962713+0000 osd.9 (osd.9) 44 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:16:34.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:33 smithi121 bash[18489]: cluster 2024-02-12T16:16:31.962713+0000 osd.9 (osd.9) 44 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:16:34.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:34.185 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:34.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:34.299 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:16:34.299 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:16:34.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:34.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:34.463 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:34.569 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:34.623 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:34.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:34.808 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:34 smithi123 bash[19119]: cluster 2024-02-12T16:16:33.066545+0000 osd.2 (osd.2) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:34.808 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:34 smithi123 bash[19119]: cluster 2024-02-12T16:16:33.066599+0000 osd.2 (osd.2) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:34.808 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:34 smithi123 bash[19119]: cluster 2024-02-12T16:16:33.163417+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.e scrub starts 2024-02-12T16:16:34.808 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:34 smithi123 bash[19119]: cluster 2024-02-12T16:16:33.735024+0000 mgr.y (mgr.14146) 937 : cluster [DBG] pgmap v1225: 27 pgs: 27 active+clean; 6.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-02-12T16:16:34.808 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:34 smithi123 bash[19119]: cluster 2024-02-12T16:16:33.804953+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e388: 12 total, 12 up, 12 in 2024-02-12T16:16:34.808 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:34 smithi123 bash[19119]: cluster 2024-02-12T16:16:33.829054+0000 osd.9 (osd.9) 45 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:16:34.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:34.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:34.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:34.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:34.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:34 smithi169 bash[19604]: cluster 2024-02-12T16:16:33.066545+0000 osd.2 (osd.2) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:34 smithi169 bash[19604]: cluster 2024-02-12T16:16:33.066599+0000 osd.2 (osd.2) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:34 smithi169 bash[19604]: cluster 2024-02-12T16:16:33.163417+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.e scrub starts 2024-02-12T16:16:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:34 smithi169 bash[19604]: cluster 2024-02-12T16:16:33.735024+0000 mgr.y (mgr.14146) 937 : cluster [DBG] pgmap v1225: 27 pgs: 27 active+clean; 6.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-02-12T16:16:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:34 smithi169 bash[19604]: cluster 2024-02-12T16:16:33.804953+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e388: 12 total, 12 up, 12 in 2024-02-12T16:16:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:34 smithi169 bash[19604]: cluster 2024-02-12T16:16:33.829054+0000 osd.9 (osd.9) 45 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:16:35.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:35.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:34 smithi121 bash[18489]: cluster 2024-02-12T16:16:33.066545+0000 osd.2 (osd.2) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:35.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:34 smithi121 bash[18489]: cluster 2024-02-12T16:16:33.066599+0000 osd.2 (osd.2) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:35.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:34 smithi121 bash[18489]: cluster 2024-02-12T16:16:33.163417+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.e scrub starts 2024-02-12T16:16:35.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:34 smithi121 bash[18489]: cluster 2024-02-12T16:16:33.735024+0000 mgr.y (mgr.14146) 937 : cluster [DBG] pgmap v1225: 27 pgs: 27 active+clean; 6.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-02-12T16:16:35.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:34 smithi121 bash[18489]: cluster 2024-02-12T16:16:33.804953+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e388: 12 total, 12 up, 12 in 2024-02-12T16:16:35.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:34 smithi121 bash[18489]: cluster 2024-02-12T16:16:33.829054+0000 osd.9 (osd.9) 45 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:16:35.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:35.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:35.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-12T16:16:35.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:35.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:35.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:35.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:35.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:35.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:35.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:35.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:35.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:35.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:35.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:35.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:35.800 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:35.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:35.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:35.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:35 smithi121 bash[18489]: cluster 2024-02-12T16:16:34.741338+0000 osd.11 (osd.11) 54 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:16:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:35 smithi121 bash[18489]: cluster 2024-02-12T16:16:34.994933+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:35 smithi121 bash[18489]: cluster 2024-02-12T16:16:34.994990+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:35 smithi121 bash[18489]: cluster 2024-02-12T16:16:35.408837+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e389: 12 total, 12 up, 12 in 2024-02-12T16:16:35.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:35.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:36.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:36.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:35 smithi123 bash[19119]: cluster 2024-02-12T16:16:34.741338+0000 osd.11 (osd.11) 54 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:16:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:35 smithi123 bash[19119]: cluster 2024-02-12T16:16:34.994933+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:35 smithi123 bash[19119]: cluster 2024-02-12T16:16:34.994990+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:35 smithi123 bash[19119]: cluster 2024-02-12T16:16:35.408837+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e389: 12 total, 12 up, 12 in 2024-02-12T16:16:36.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:36.179 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:35 smithi169 bash[19604]: cluster 2024-02-12T16:16:34.741338+0000 osd.11 (osd.11) 54 : cluster [DBG] 4.9 scrub starts 2024-02-12T16:16:36.179 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:35 smithi169 bash[19604]: cluster 2024-02-12T16:16:34.994933+0000 osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:36.180 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:35 smithi169 bash[19604]: cluster 2024-02-12T16:16:34.994990+0000 osd.0 (osd.0) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:36.180 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:35 smithi169 bash[19604]: cluster 2024-02-12T16:16:35.408837+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e389: 12 total, 12 up, 12 in 2024-02-12T16:16:36.197 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:36.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:36.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:36.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.457 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:36.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:36.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:36.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:36.645 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:36.645 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:36.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:36.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:36.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:36.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:36.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:36.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:36.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:37.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:37.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:36 smithi169 bash[19604]: cluster 2024-02-12T16:16:35.098673+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.e scrub ok 2024-02-12T16:16:37.105 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:36 smithi169 bash[19604]: cluster 2024-02-12T16:16:35.735424+0000 mgr.y (mgr.14146) 938 : cluster [DBG] pgmap v1228: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.58k op/s 2024-02-12T16:16:37.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:37.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:37.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:36 smithi123 bash[19119]: cluster 2024-02-12T16:16:35.098673+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.e scrub ok 2024-02-12T16:16:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:36 smithi123 bash[19119]: cluster 2024-02-12T16:16:35.735424+0000 mgr.y (mgr.14146) 938 : cluster [DBG] pgmap v1228: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.58k op/s 2024-02-12T16:16:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:36 smithi121 bash[18489]: cluster 2024-02-12T16:16:35.098673+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.e scrub ok 2024-02-12T16:16:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:36 smithi121 bash[18489]: cluster 2024-02-12T16:16:35.735424+0000 mgr.y (mgr.14146) 938 : cluster [DBG] pgmap v1228: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.58k op/s 2024-02-12T16:16:37.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:37.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:37.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:37.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:37.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:37.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:37.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:37.527 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:37.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:37.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:37.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:37.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:37.799 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:37.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:37.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:37.934 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:37.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:38.036 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:38.076 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:38.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:38.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:38.208 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:38.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:38.311 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:38.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:38.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:38.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:38.470 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:38.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:38.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:38.644 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:38.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:38.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:38.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:38.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:38.912 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:38.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:39.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:39.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.730732+0000 osd.11 (osd.11) 55 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:16:39.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.735844+0000 mgr.y (mgr.14146) 939 : cluster [DBG] pgmap v1229: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:16:39.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.801351+0000 osd.10 (osd.10) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:39.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.801410+0000 osd.10 (osd.10) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:39.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.812555+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e390: 12 total, 12 up, 12 in 2024-02-12T16:16:39.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.900659+0000 osd.5 (osd.5) 53 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:16:39.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.938146+0000 osd.5 (osd.5) 54 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:16:39.021 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:38 smithi121 bash[18489]: cluster 2024-02-12T16:16:37.958241+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.a scrub starts 2024-02-12T16:16:39.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:39.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.730732+0000 osd.11 (osd.11) 55 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.735844+0000 mgr.y (mgr.14146) 939 : cluster [DBG] pgmap v1229: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.801351+0000 osd.10 (osd.10) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.801410+0000 osd.10 (osd.10) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.812555+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e390: 12 total, 12 up, 12 in 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.900659+0000 osd.5 (osd.5) 53 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.938146+0000 osd.5 (osd.5) 54 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:16:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:38 smithi123 bash[19119]: cluster 2024-02-12T16:16:37.958241+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.a scrub starts 2024-02-12T16:16:39.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.730732+0000 osd.11 (osd.11) 55 : cluster [DBG] 4.9 scrub ok 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.735844+0000 mgr.y (mgr.14146) 939 : cluster [DBG] pgmap v1229: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.801351+0000 osd.10 (osd.10) 37 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.801410+0000 osd.10 (osd.10) 38 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.812555+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e390: 12 total, 12 up, 12 in 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.900659+0000 osd.5 (osd.5) 53 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.938146+0000 osd.5 (osd.5) 54 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:16:39.157 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:38 smithi169 bash[19604]: cluster 2024-02-12T16:16:37.958241+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.a scrub starts 2024-02-12T16:16:39.182 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:39.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:39.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:39.318 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:39.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:39.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:39.469 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:39.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:39.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:39.600 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:39.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:39.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:39.736 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:39.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:39.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:39.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:39.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:39.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:40.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:40.009 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:16:40.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:40.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:40.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:40.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:40.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:40.393 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:40.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:40.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:40.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:40.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:40.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:40.791 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: cluster 2024-02-12T16:16:39.695561+0000 osd.11 (osd.11) 56 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: cluster 2024-02-12T16:16:39.721877+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.a scrub ok 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: cluster 2024-02-12T16:16:39.736304+0000 mgr.y (mgr.14146) 940 : cluster [DBG] pgmap v1231: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: cluster 2024-02-12T16:16:39.780852+0000 osd.9 (osd.9) 46 : cluster [DBG] 1.a scrub starts 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: cluster 2024-02-12T16:16:39.800002+0000 osd.11 (osd.11) 57 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: cluster 2024-02-12T16:16:39.823543+0000 osd.9 (osd.9) 47 : cluster [DBG] 1.a scrub ok 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: cluster 2024-02-12T16:16:40.410919+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e391: 12 total, 12 up, 12 in 2024-02-12T16:16:40.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:40 smithi123 bash[19119]: audit 2024-02-12T16:16:40.421407+0000 mon.a (mon.0) 1494 : audit [INF] from='client.? v1:172.21.15.121:0/3059210832' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.10392998258262173}]: dispatch 2024-02-12T16:16:40.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: cluster 2024-02-12T16:16:39.695561+0000 osd.11 (osd.11) 56 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: cluster 2024-02-12T16:16:39.721877+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.a scrub ok 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: cluster 2024-02-12T16:16:39.736304+0000 mgr.y (mgr.14146) 940 : cluster [DBG] pgmap v1231: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: cluster 2024-02-12T16:16:39.780852+0000 osd.9 (osd.9) 46 : cluster [DBG] 1.a scrub starts 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: cluster 2024-02-12T16:16:39.800002+0000 osd.11 (osd.11) 57 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: cluster 2024-02-12T16:16:39.823543+0000 osd.9 (osd.9) 47 : cluster [DBG] 1.a scrub ok 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: cluster 2024-02-12T16:16:40.410919+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e391: 12 total, 12 up, 12 in 2024-02-12T16:16:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:40 smithi121 bash[18489]: audit 2024-02-12T16:16:40.421407+0000 mon.a (mon.0) 1494 : audit [INF] from='client.? v1:172.21.15.121:0/3059210832' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.10392998258262173}]: dispatch 2024-02-12T16:16:40.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:40.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:41.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:41.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:41.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:41.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:41.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:41.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:41.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: cluster 2024-02-12T16:16:39.695561+0000 osd.11 (osd.11) 56 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:16:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: cluster 2024-02-12T16:16:39.721877+0000 osd.6 (osd.6) 50 : cluster [DBG] 4.a scrub ok 2024-02-12T16:16:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: cluster 2024-02-12T16:16:39.736304+0000 mgr.y (mgr.14146) 940 : cluster [DBG] pgmap v1231: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-02-12T16:16:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: cluster 2024-02-12T16:16:39.780852+0000 osd.9 (osd.9) 46 : cluster [DBG] 1.a scrub starts 2024-02-12T16:16:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: cluster 2024-02-12T16:16:39.800002+0000 osd.11 (osd.11) 57 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:16:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: cluster 2024-02-12T16:16:39.823543+0000 osd.9 (osd.9) 47 : cluster [DBG] 1.a scrub ok 2024-02-12T16:16:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: cluster 2024-02-12T16:16:40.410919+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e391: 12 total, 12 up, 12 in 2024-02-12T16:16:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:40 smithi169 bash[19604]: audit 2024-02-12T16:16:40.421407+0000 mon.a (mon.0) 1494 : audit [INF] from='client.? v1:172.21.15.121:0/3059210832' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 0.10392998258262173}]: dispatch 2024-02-12T16:16:41.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:41.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:41.419 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:41.431 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.1 to 0.10393 (1a9b) 2024-02-12T16:16:41.443 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:41.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:41.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:41.582 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:41.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:41.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:41.721 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:41.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:41.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:41.862 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:41.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:41.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:42.002 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:42.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:42.135 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:42.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:42.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:42.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:42.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:42.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:42.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:42.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:42 smithi169 bash[19604]: audit 2024-02-12T16:16:41.417295+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? v1:172.21.15.121:0/3059210832' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.10392998258262173}]': finished 2024-02-12T16:16:42.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:42 smithi169 bash[19604]: cluster 2024-02-12T16:16:41.417352+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e392: 12 total, 12 up, 12 in 2024-02-12T16:16:42.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:42 smithi169 bash[19604]: cluster 2024-02-12T16:16:41.736888+0000 mgr.y (mgr.14146) 941 : cluster [DBG] pgmap v1234: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s 2024-02-12T16:16:42.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:42.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:42 smithi121 bash[18489]: audit 2024-02-12T16:16:41.417295+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? v1:172.21.15.121:0/3059210832' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.10392998258262173}]': finished 2024-02-12T16:16:42.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:42 smithi121 bash[18489]: cluster 2024-02-12T16:16:41.417352+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e392: 12 total, 12 up, 12 in 2024-02-12T16:16:42.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:42 smithi121 bash[18489]: cluster 2024-02-12T16:16:41.736888+0000 mgr.y (mgr.14146) 941 : cluster [DBG] pgmap v1234: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s 2024-02-12T16:16:42.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:42.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:42.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:42 smithi123 bash[19119]: audit 2024-02-12T16:16:41.417295+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? v1:172.21.15.121:0/3059210832' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.10392998258262173}]': finished 2024-02-12T16:16:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:42 smithi123 bash[19119]: cluster 2024-02-12T16:16:41.417352+0000 mon.a (mon.0) 1496 : cluster [DBG] osdmap e392: 12 total, 12 up, 12 in 2024-02-12T16:16:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:42 smithi123 bash[19119]: cluster 2024-02-12T16:16:41.736888+0000 mgr.y (mgr.14146) 941 : cluster [DBG] pgmap v1234: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s 2024-02-12T16:16:42.817 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:42.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:42.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:42.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:42.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:43.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:43.095 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:43.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:43.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:43.217 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:16:43.235 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:43.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:43.243 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:16:43.268 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:16:43.290 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:16:43.319 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:16:43.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:43.357 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:16:43.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:43.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:43.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:16:43.425 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:16:43.455 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:16:43.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:43.478 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:16:43.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:43.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:43.513 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:16:43.546 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:16:43.548 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.548 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.548 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.548 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.549 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:16:43.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:43.604 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:43 smithi121 bash[18489]: cluster 2024-02-12T16:16:42.422047+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e393: 12 total, 12 up, 12 in 2024-02-12T16:16:43.604 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:43 smithi121 bash[18489]: audit 2024-02-12T16:16:43.185602+0000 mon.a (mon.0) 1498 : audit [DBG] from='client.? v1:172.21.15.121:0/2839708403' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:43.604 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:43 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.422248+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e394: 12 total, 12 up, 12 in 2024-02-12T16:16:43.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:43.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:43.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:43 smithi123 bash[19119]: cluster 2024-02-12T16:16:42.422047+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e393: 12 total, 12 up, 12 in 2024-02-12T16:16:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:43 smithi123 bash[19119]: audit 2024-02-12T16:16:43.185602+0000 mon.a (mon.0) 1498 : audit [DBG] from='client.? v1:172.21.15.121:0/2839708403' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:43 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.422248+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e394: 12 total, 12 up, 12 in 2024-02-12T16:16:43.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:43.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:43.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:43.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:43 smithi169 bash[19604]: cluster 2024-02-12T16:16:42.422047+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e393: 12 total, 12 up, 12 in 2024-02-12T16:16:43.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:43 smithi169 bash[19604]: audit 2024-02-12T16:16:43.185602+0000 mon.a (mon.0) 1498 : audit [DBG] from='client.? v1:172.21.15.121:0/2839708403' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:43 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.422248+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e394: 12 total, 12 up, 12 in 2024-02-12T16:16:43.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:43.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:43.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:44.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:44.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:44.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:44.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:44.166 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:16:44.166 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:16:44.166 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:16:44.166 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:16:44.166 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:16:44.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:44.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:44.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:44.329 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:44.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:44.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:44.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:44.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.421360+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.f starting backfill to osd.11 from (0'0,0'0] MAX to 391'15777 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.423474+0000 osd.2 (osd.2) 43 : cluster [DBG] 4.4 starting backfill to osd.3 from (0'0,0'0] MAX to 391'15980 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.426703+0000 osd.11 (osd.11) 58 : cluster [DBG] 4.9 starting backfill to osd.3 from (0'0,0'0] MAX to 391'15930 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.426846+0000 osd.6 (osd.6) 51 : cluster [DBG] 4.a starting backfill to osd.10 from (0'0,0'0] MAX to 391'16025 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.427889+0000 osd.10 (osd.10) 39 : cluster [DBG] 4.7 starting backfill to osd.0 from (0'0,0'0] MAX to 391'15942 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.428561+0000 osd.9 (osd.9) 48 : cluster [DBG] 4.6 starting backfill to osd.2 from (0'0,0'0] MAX to 391'18440 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: audit 2024-02-12T16:16:43.436456+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? v1:172.21.15.121:0/1771850104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: cluster 2024-02-12T16:16:43.737453+0000 mgr.y (mgr.14146) 942 : cluster [DBG] pgmap v1237: 27 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+clean+scrubbing, 24 active+clean; 5.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 5816/189972 objects misplaced (3.062%); 144 KiB/s, 2 objects/s recovering 2024-02-12T16:16:44.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:44 smithi169 bash[19604]: audit 2024-02-12T16:16:44.294587+0000 mon.a (mon.0) 1501 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:16:44.478 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:16:44.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.421360+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.f starting backfill to osd.11 from (0'0,0'0] MAX to 391'15777 2024-02-12T16:16:44.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.423474+0000 osd.2 (osd.2) 43 : cluster [DBG] 4.4 starting backfill to osd.3 from (0'0,0'0] MAX to 391'15980 2024-02-12T16:16:44.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.426703+0000 osd.11 (osd.11) 58 : cluster [DBG] 4.9 starting backfill to osd.3 from (0'0,0'0] MAX to 391'15930 2024-02-12T16:16:44.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.426846+0000 osd.6 (osd.6) 51 : cluster [DBG] 4.a starting backfill to osd.10 from (0'0,0'0] MAX to 391'16025 2024-02-12T16:16:44.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.427889+0000 osd.10 (osd.10) 39 : cluster [DBG] 4.7 starting backfill to osd.0 from (0'0,0'0] MAX to 391'15942 2024-02-12T16:16:44.549 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.428561+0000 osd.9 (osd.9) 48 : cluster [DBG] 4.6 starting backfill to osd.2 from (0'0,0'0] MAX to 391'18440 2024-02-12T16:16:44.549 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: audit 2024-02-12T16:16:43.436456+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? v1:172.21.15.121:0/1771850104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:44.549 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: cluster 2024-02-12T16:16:43.737453+0000 mgr.y (mgr.14146) 942 : cluster [DBG] pgmap v1237: 27 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+clean+scrubbing, 24 active+clean; 5.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 5816/189972 objects misplaced (3.062%); 144 KiB/s, 2 objects/s recovering 2024-02-12T16:16:44.549 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:44 smithi121 bash[18489]: audit 2024-02-12T16:16:44.294587+0000 mon.a (mon.0) 1501 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:16:44.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:44.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:44.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:44.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:44.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:44.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:44.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.421360+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.f starting backfill to osd.11 from (0'0,0'0] MAX to 391'15777 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.423474+0000 osd.2 (osd.2) 43 : cluster [DBG] 4.4 starting backfill to osd.3 from (0'0,0'0] MAX to 391'15980 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.426703+0000 osd.11 (osd.11) 58 : cluster [DBG] 4.9 starting backfill to osd.3 from (0'0,0'0] MAX to 391'15930 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.426846+0000 osd.6 (osd.6) 51 : cluster [DBG] 4.a starting backfill to osd.10 from (0'0,0'0] MAX to 391'16025 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.427889+0000 osd.10 (osd.10) 39 : cluster [DBG] 4.7 starting backfill to osd.0 from (0'0,0'0] MAX to 391'15942 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.428561+0000 osd.9 (osd.9) 48 : cluster [DBG] 4.6 starting backfill to osd.2 from (0'0,0'0] MAX to 391'18440 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: audit 2024-02-12T16:16:43.436456+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? v1:172.21.15.121:0/1771850104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:16:44.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: cluster 2024-02-12T16:16:43.737453+0000 mgr.y (mgr.14146) 942 : cluster [DBG] pgmap v1237: 27 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+clean+scrubbing, 24 active+clean; 5.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 5816/189972 objects misplaced (3.062%); 144 KiB/s, 2 objects/s recovering 2024-02-12T16:16:44.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:44 smithi123 bash[19119]: audit 2024-02-12T16:16:44.294587+0000 mon.a (mon.0) 1501 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:16:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:44.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:44.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:45.013 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:45.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:45.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:45.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:45.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-12T16:16:45.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:45.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:45.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:45.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:45.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:45.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:45.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:45.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:45.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:45 smithi121 bash[18489]: cluster 2024-02-12T16:16:44.427925+0000 mon.a (mon.0) 1502 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:16:45.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:45 smithi121 bash[18489]: cluster 2024-02-12T16:16:44.427960+0000 mon.a (mon.0) 1503 : cluster [INF] Cluster is now healthy 2024-02-12T16:16:45.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:45 smithi121 bash[18489]: audit 2024-02-12T16:16:44.468461+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? v1:172.21.15.121:0/1771850104' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:16:45.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:45 smithi121 bash[18489]: cluster 2024-02-12T16:16:44.468521+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e395: 12 total, 12 up, 12 in 2024-02-12T16:16:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:45.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:45.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:45.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:45 smithi169 bash[19604]: cluster 2024-02-12T16:16:44.427925+0000 mon.a (mon.0) 1502 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:16:45.715 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:45 smithi169 bash[19604]: cluster 2024-02-12T16:16:44.427960+0000 mon.a (mon.0) 1503 : cluster [INF] Cluster is now healthy 2024-02-12T16:16:45.715 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:45 smithi169 bash[19604]: audit 2024-02-12T16:16:44.468461+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? v1:172.21.15.121:0/1771850104' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:16:45.715 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:45 smithi169 bash[19604]: cluster 2024-02-12T16:16:44.468521+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e395: 12 total, 12 up, 12 in 2024-02-12T16:16:45.732 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:45.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:45.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:45.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:45.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:45 smithi123 bash[19119]: cluster 2024-02-12T16:16:44.427925+0000 mon.a (mon.0) 1502 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:16:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:45 smithi123 bash[19119]: cluster 2024-02-12T16:16:44.427960+0000 mon.a (mon.0) 1503 : cluster [INF] Cluster is now healthy 2024-02-12T16:16:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:45 smithi123 bash[19119]: audit 2024-02-12T16:16:44.468461+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? v1:172.21.15.121:0/1771850104' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:16:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:45 smithi123 bash[19119]: cluster 2024-02-12T16:16:44.468521+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e395: 12 total, 12 up, 12 in 2024-02-12T16:16:45.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:45.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:46.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:46.144 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:46.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:46.277 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:46.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:46.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:16:46.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:46.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:46.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:46.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:46 smithi123 bash[19119]: cluster 2024-02-12T16:16:45.737973+0000 mgr.y (mgr.14146) 943 : cluster [DBG] pgmap v1239: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 22 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 22949/185186 objects misplaced (12.392%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:16:46.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:46 smithi123 bash[19119]: cluster 2024-02-12T16:16:45.787309+0000 osd.9 (osd.9) 49 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:16:46.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:46.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:46 smithi121 bash[18489]: cluster 2024-02-12T16:16:45.737973+0000 mgr.y (mgr.14146) 943 : cluster [DBG] pgmap v1239: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 22 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 22949/185186 objects misplaced (12.392%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:16:46.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:46 smithi121 bash[18489]: cluster 2024-02-12T16:16:45.787309+0000 osd.9 (osd.9) 49 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:16:46.696 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:46.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:46 smithi169 bash[19604]: cluster 2024-02-12T16:16:45.737973+0000 mgr.y (mgr.14146) 943 : cluster [DBG] pgmap v1239: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 22 active+clean; 5.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 22949/185186 objects misplaced (12.392%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:16:46.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:46 smithi169 bash[19604]: cluster 2024-02-12T16:16:45.787309+0000 osd.9 (osd.9) 49 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:16:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:46.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:46.827 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:46.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:46.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:46.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:47.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:47.108 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:47.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:47.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:47.238 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:47.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:47.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:47.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:47.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:47.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:16:47.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:47.529 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:47.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:47.606 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:16:47.606 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:16:47.606 INFO:tasks.thrashosds.thrasher:Removing osd 8, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-12T16:16:47.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd out 8 2024-02-12T16:16:47.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:47 smithi123 bash[19119]: cluster 2024-02-12T16:16:46.431852+0000 mon.a (mon.0) 1506 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:16:47.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:47.724 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:47 smithi121 bash[18489]: cluster 2024-02-12T16:16:46.431852+0000 mon.a (mon.0) 1506 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:16:47.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:47.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:47 smithi169 bash[19604]: cluster 2024-02-12T16:16:46.431852+0000 mon.a (mon.0) 1506 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:16:47.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:47.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:47.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:47.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:48.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:48.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:48.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:48.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:48.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:48.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:48.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:48.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:48.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:48.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:48 smithi169 bash[19604]: cluster 2024-02-12T16:16:47.363799+0000 osd.9 (osd.9) 50 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:16:48.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:48 smithi169 bash[19604]: cluster 2024-02-12T16:16:47.738466+0000 mgr.y (mgr.14146) 944 : cluster [DBG] pgmap v1240: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 34360/183452 objects misplaced (18.730%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:16:48.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:48.652 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:48 smithi121 bash[18489]: cluster 2024-02-12T16:16:47.363799+0000 osd.9 (osd.9) 50 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:16:48.652 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:48 smithi121 bash[18489]: cluster 2024-02-12T16:16:47.738466+0000 mgr.y (mgr.14146) 944 : cluster [DBG] pgmap v1240: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 34360/183452 objects misplaced (18.730%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:16:48.679 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:48.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:48.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:48 smithi123 bash[19119]: cluster 2024-02-12T16:16:47.363799+0000 osd.9 (osd.9) 50 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:16:48.788 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:48 smithi123 bash[19119]: cluster 2024-02-12T16:16:47.738466+0000 mgr.y (mgr.14146) 944 : cluster [DBG] pgmap v1240: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 34360/183452 objects misplaced (18.730%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:16:48.807 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:48.911 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:48.934 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:48.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:49.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:49.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:49.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:49.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:49.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:49.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:49.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:49.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:49.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:49.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:49.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:49.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:49.646 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:49.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:49.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:49.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:49.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:49.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:49.949 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:16:49.949 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:49.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:50.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:50.085 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:50.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:50.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:50.211 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:50.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:50.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:50.356 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:50.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:50.469 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:50.495 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:50.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:50.600 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:50.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:50.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:50.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:50 smithi121 bash[18489]: audit 2024-02-12T16:16:49.647415+0000 mon.a (mon.0) 1507 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:16:50.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:50 smithi121 bash[18489]: cluster 2024-02-12T16:16:49.738912+0000 mgr.y (mgr.14146) 945 : cluster [DBG] pgmap v1241: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 33019/181574 objects misplaced (18.185%); 7.6 MiB/s, 122 objects/s recovering 2024-02-12T16:16:50.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:50 smithi121 bash[18489]: audit 2024-02-12T16:16:49.982281+0000 mon.a (mon.0) 1508 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:16:50.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:50 smithi121 bash[18489]: audit 2024-02-12T16:16:49.983677+0000 mon.a (mon.0) 1509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:16:50.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:50 smithi121 bash[18489]: audit 2024-02-12T16:16:49.996423+0000 mon.a (mon.0) 1510 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:16:50.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:50.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:50.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:50.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:50 smithi123 bash[19119]: audit 2024-02-12T16:16:49.647415+0000 mon.a (mon.0) 1507 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:16:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:50 smithi123 bash[19119]: cluster 2024-02-12T16:16:49.738912+0000 mgr.y (mgr.14146) 945 : cluster [DBG] pgmap v1241: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 33019/181574 objects misplaced (18.185%); 7.6 MiB/s, 122 objects/s recovering 2024-02-12T16:16:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:50 smithi123 bash[19119]: audit 2024-02-12T16:16:49.982281+0000 mon.a (mon.0) 1508 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:16:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:50 smithi123 bash[19119]: audit 2024-02-12T16:16:49.983677+0000 mon.a (mon.0) 1509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:16:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:50 smithi123 bash[19119]: audit 2024-02-12T16:16:49.996423+0000 mon.a (mon.0) 1510 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:16:50.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:50.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:51.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:51.043 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:50 smithi169 bash[19604]: audit 2024-02-12T16:16:49.647415+0000 mon.a (mon.0) 1507 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:16:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:50 smithi169 bash[19604]: cluster 2024-02-12T16:16:49.738912+0000 mgr.y (mgr.14146) 945 : cluster [DBG] pgmap v1241: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 33019/181574 objects misplaced (18.185%); 7.6 MiB/s, 122 objects/s recovering 2024-02-12T16:16:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:50 smithi169 bash[19604]: audit 2024-02-12T16:16:49.982281+0000 mon.a (mon.0) 1508 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:16:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:50 smithi169 bash[19604]: audit 2024-02-12T16:16:49.983677+0000 mon.a (mon.0) 1509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:16:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:50 smithi169 bash[19604]: audit 2024-02-12T16:16:49.996423+0000 mon.a (mon.0) 1510 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:16:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:51.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:51.176 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:51.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:51.307 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:51.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:51.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:51.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:51.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:51.695 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:51.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:51.832 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:51.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:51.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:52.068 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:52.094 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:52.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:52.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:52.218 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:52.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:52.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:52.327 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:16:52.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:52.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:52.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:52.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:52.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:52.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:52.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:52.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:52.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:52.796 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:52.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:52.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:52.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:52 smithi121 bash[18489]: cluster 2024-02-12T16:16:51.739644+0000 mgr.y (mgr.14146) 946 : cluster [DBG] pgmap v1242: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 31430/174720 objects misplaced (17.989%); 12 MiB/s, 192 objects/s recovering 2024-02-12T16:16:52.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:52.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:53.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:52 smithi169 bash[19604]: cluster 2024-02-12T16:16:51.739644+0000 mgr.y (mgr.14146) 946 : cluster [DBG] pgmap v1242: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 31430/174720 objects misplaced (17.989%); 12 MiB/s, 192 objects/s recovering 2024-02-12T16:16:53.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:16:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:53.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:16:53.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:52 smithi123 bash[19119]: cluster 2024-02-12T16:16:51.739644+0000 mgr.y (mgr.14146) 946 : cluster [DBG] pgmap v1242: 27 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 5.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 31430/174720 objects misplaced (17.989%); 12 MiB/s, 192 objects/s recovering 2024-02-12T16:16:53.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:53.240 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:53.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:53.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:53.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:53.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:53.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:53.506 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:53.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:53.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:53.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:53.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:53.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:53.767 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:53.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:53.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:53.927 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:53.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:54.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:54.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:54.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:54.169 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:54.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:54.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:54.299 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:54.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:54.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:54.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:54.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:54.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:54.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:54.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:54.612 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:54.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:54.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:54.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:54.741 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:54.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:54.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:54 smithi123 bash[19119]: cluster 2024-02-12T16:16:53.740125+0000 mgr.y (mgr.14146) 947 : cluster [DBG] pgmap v1243: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 30157/172944 objects misplaced (17.437%); 15 MiB/s, 232 objects/s recovering 2024-02-12T16:16:54.894 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:54.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:55.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:55.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:54 smithi121 bash[18489]: cluster 2024-02-12T16:16:53.740125+0000 mgr.y (mgr.14146) 947 : cluster [DBG] pgmap v1243: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 30157/172944 objects misplaced (17.437%); 15 MiB/s, 232 objects/s recovering 2024-02-12T16:16:55.029 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:124964 4.5 e353/334 4:a403e5c2:::benchmark_data_smithi169_60938_object124963:head v 353'7772, mlcod=353'7770)", 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:50.478314+0000", 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 124.57460834600001, 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.058873186000000001, 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.060 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478314+0000", 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478314+0000", 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478312+0000", 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2361999999999995e-05 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478394+0000", 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.045691896000000003 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524086+0000", 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4335000000000001e-05 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524101+0000", 2024-02-12T16:16:55.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4688999999999998e-05 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524155+0000", 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4323000000000001e-05 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524180+0000", 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012964513 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.537144+0000", 2024-02-12T16:16:55.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2988999999999998e-05 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.537187+0000", 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.058873186000000001 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207911 4.5 e377/334 4:a47a7480:::benchmark_data_smithi169_60938_object34574:head v 377'13033, mlcod=377'13031)", 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.202860+0000", 2024-02-12T16:16:55.064 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 54.850061963999998, 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13450072800000001, 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202860+0000", 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202860+0000", 2024-02-12T16:16:55.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202859+0000", 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9830000000000004e-06 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202864+0000", 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2080000000000002e-06 2024-02-12T16:16:55.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202869+0000", 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7690000000000001e-06 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202871+0000", 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1919000000000003e-05 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202913+0000", 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3874e-05 2024-02-12T16:16:55.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202927+0000", 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.134390395 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337318+0000", 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3404e-05 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337361+0000", 2024-02-12T16:16:55.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13450072800000001 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207916 4.5 e377/334 4:a7a2a20e:::benchmark_data_smithi169_60938_object34579:head v 377'13034, mlcod=377'13031)", 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.207129+0000", 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 54.845793399999998, 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130258391, 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207129+0000", 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207129+0000", 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207126+0000", 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4731e-05 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207141+0000", 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4761e-05 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207156+0000", 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4319999999999999e-06 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207164+0000", 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000294114 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207458+0000", 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1787999999999999e-05 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207480+0000", 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12988481499999999 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337365+0000", 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.215e-05 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337387+0000", 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130258391 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207919 4.e e377/334 4:736a4e29:::benchmark_data_smithi169_60938_object34582:head v 377'12960, mlcod=377'12958)", 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.208546+0000", 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 54.844375978000002, 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.132842772, 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.073 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208546+0000", 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208546+0000", 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208545+0000", 2024-02-12T16:16:55.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0985000000000001e-05 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208556+0000", 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1014e-05 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208567+0000", 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2559999999999999e-06 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208573+0000", 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.120351823 2024-02-12T16:16:55.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328925+0000", 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6460000000000001e-05 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328951+0000", 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012392357 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341343+0000", 2024-02-12T16:16:55.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5546000000000001e-05 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341389+0000", 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.132842772 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207918 4.7 e377/334 4:e0a92af0:::benchmark_data_smithi169_60938_object34581:head v 377'13007, mlcod=377'13004)", 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.210837+0000", 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 54.842084896000003, 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13963632000000001, 2024-02-12T16:16:55.077 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210837+0000", 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210837+0000", 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210836+0000", 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0129000000000001e-05 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210846+0000", 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.11802884399999999 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328875+0000", 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9264999999999999e-05 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328894+0000", 2024-02-12T16:16:55.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016667400000000001 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.329061+0000", 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1637999999999998e-05 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.329082+0000", 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021358701000000001 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.350441+0000", 2024-02-12T16:16:55.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2668e-05 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.350474+0000", 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13963632000000001 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207934 4.e e377/334 4:73357360:::benchmark_data_smithi169_60938_object34597:head v 377'12961, mlcod=377'12958)", 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.228289+0000", 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 54.824633382000002, 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.113124713, 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.081 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228289+0000", 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228289+0000", 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228287+0000", 2024-02-12T16:16:55.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2738000000000001e-05 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228300+0000", 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.100576519 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328876+0000", 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7234000000000001e-05 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328893+0000", 2024-02-12T16:16:55.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042875989999999996 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.333181+0000", 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3295999999999999e-05 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.333204+0000", 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081894759999999994 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341394+0000", 2024-02-12T16:16:55.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9811000000000001e-05 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341414+0000", 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.113124713 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:253678 4.7 4:ea133223:::benchmark_data_smithi169_60938_object80341:head [delete] snapc 0=[] ondisk+write+known_if_redirected e394)", 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:43.814440+0000", 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.238481912999999, 2024-02-12T16:16:55.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054126177999999997, 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 253678 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.814440+0000", 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.814440+0000", 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.814438+0000", 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6460000000000004e-06 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.814445+0000", 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3040999999999999e-05 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.814458+0000", 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0619999999999997e-06 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.814464+0000", 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046549720000000003 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.819119+0000", 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000133826 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.819253+0000", 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018117799999999999 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.819434+0000", 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052228409999999998 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.824657+0000", 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001036188 2024-02-12T16:16:55.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.825693+0000", 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.042660621000000003 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.868354+0000", 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3329000000000003e-05 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.868417+0000", 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000149314 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:43.868566+0000", 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054126177999999997 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:254376 4.7 4:eba33e6e:::benchmark_data_smithi169_60938_object81039:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:44.542477+0000", 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.510445456999999, 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.052478799999999999, 2024-02-12T16:16:55.091 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 254376 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542477+0000", 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542477+0000", 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542474+0000", 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6119999999999999e-06 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542483+0000", 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0779e-05 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542493+0000", 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.46e-06 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542500+0000", 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3659999999999999e-05 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542544+0000", 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000135396 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.542679+0000", 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073767600000000004 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.543417+0000", 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068459760000000001 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.550263+0000", 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012636282 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.562899+0000", 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031907191000000001 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.594806+0000", 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8943000000000003e-05 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.594845+0000", 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000110593 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:44.594956+0000", 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.052478799999999999 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:256743 4.7 4:e59c4d2a:::benchmark_data_smithi169_60938_object83406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:46.964215+0000", 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0887075609999997, 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054632014, 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.097 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 256743 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.964215+0000", 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.964215+0000", 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.964213+0000", 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.878e-06 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.964220+0000", 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1581e-05 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.964231+0000", 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1330000000000001e-06 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.964239+0000", 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025630760000000001 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.966802+0000", 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000180013 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.966982+0000", 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000132399 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.967114+0000", 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053347760000000003 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.972449+0000", 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012324139 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:46.984773+0000", 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033828121000000003 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:47.018601+0000", 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0350000000000002e-05 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:47.018671+0000", 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000175308 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:47.018847+0000", 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054632014 2024-02-12T16:16:55.102 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:259910 4.7 4:e1179fb6:::benchmark_data_smithi169_60938_object86573:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.089776+0000", 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.963145881, 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.079746256000000001, 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259910 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.103 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089776+0000", 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089776+0000", 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089774+0000", 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7060000000000004e-06 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089782+0000", 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1078e-05 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089793+0000", 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0090000000000004e-06 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089799+0000", 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0108000000000003e-05 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089839+0000", 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4909999999999999e-05 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089914+0000", 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7779000000000001e-05 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.090002+0000", 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047817750000000003 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.094783+0000", 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001908152 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.096691+0000", 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.072652089000000003 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169344+0000", 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1241999999999994e-05 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169405+0000", 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117841 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169523+0000", 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.079746256000000001 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:259922 4.7 4:e2a9c3b9:::benchmark_data_smithi169_60938_object86585:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.098594+0000", 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9543282800000004, 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.071203441000000006, 2024-02-12T16:16:55.108 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259922 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098594+0000", 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098594+0000", 2024-02-12T16:16:55.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098592+0000", 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0305999999999998e-05 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098612+0000", 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4029999999999996e-06 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098622+0000", 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9370000000000001e-06 2024-02-12T16:16:55.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098627+0000", 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4458e-05 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098672+0000", 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012025 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098792+0000", 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015108100000000001 2024-02-12T16:16:55.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098943+0000", 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043909810000000004 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.103334+0000", 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.066236931999999998 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169571+0000", 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1420000000000005e-05 2024-02-12T16:16:55.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169633+0000", 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2843999999999998e-05 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169685+0000", 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000111938 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169797+0000", 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.071203441000000006 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:259926 4.e e395/334 4:72322939:::benchmark_data_smithi169_60938_object86589:head v 395'16218, mlcod=395'16216)", 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.104248+0000", 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9486742150000005, 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068572121, 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104248+0000", 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104248+0000", 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104247+0000", 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.433e-05 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104261+0000", 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3900000000000004e-06 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104269+0000", 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5240000000000002e-06 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104273+0000", 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1297999999999997e-05 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104304+0000", 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8640000000000001e-05 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104323+0000", 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068398252000000007 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.172721+0000", 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8781000000000002e-05 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.172820+0000", 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068572121 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:259927 4.a e395/393 4:58da0553:::benchmark_data_smithi169_60938_object86590:head v 395'16447, mlcod=395'16444)", 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.105096+0000", 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9478258229999996, 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068729290999999998, 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105096+0000", 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105096+0000", 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105095+0000", 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.49e-06 2024-02-12T16:16:55.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105104+0000", 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060811997999999999 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.165916+0000", 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6510000000000007e-06 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.165924+0000", 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5616000000000001e-05 2024-02-12T16:16:55.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.166020+0000", 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1800000000000001e-05 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.166042+0000", 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077482810000000001 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.173790+0000", 2024-02-12T16:16:55.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5675e-05 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.173826+0000", 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068729290999999998 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.25264.0:259922 4.7 e395/393 ondisk, result = 0)", 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.107586+0000", 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9453359109999999, 2024-02-12T16:16:55.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062214634999999997, 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107586+0000", 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107586+0000", 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107585+0000", 2024-02-12T16:16:55.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0880000000000002e-06 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107590+0000", 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.056974006000000001 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164564+0000", 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0686999999999999e-05 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164575+0000", 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050434390000000003 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169619+0000", 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2110000000000001e-06 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169626+0000", 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017516499999999999 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.125 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169801+0000", 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062214634999999997 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.25264.0:259910 4.7 e395/393 ondisk, result = 0)", 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.109642+0000", 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9432797480000001, 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059889112000000001, 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109642+0000", 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109642+0000", 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109641+0000", 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8079999999999999e-06 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109648+0000", 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054774981 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164423+0000", 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8316999999999999e-05 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164461+0000", 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048644780000000002 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169325+0000", 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0338e-05 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169336+0000", 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000195959 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169532+0000", 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059889112000000001 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:260336 4.7 4:ebe3e51e:::benchmark_data_smithi169_60938_object86999:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.554100+0000", 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.498822315, 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.052226360999999999, 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 260336 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.554100+0000", 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.554100+0000", 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.554099+0000", 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9509999999999997e-06 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.554104+0000", 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8889999999999996e-06 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.554111+0000", 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6049999999999997e-06 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.554116+0000", 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003766293 2024-02-12T16:16:55.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.557882+0000", 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011875 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.558001+0000", 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014971599999999999 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.558151+0000", 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043710620000000002 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.562522+0000", 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047925509999999999 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.567314+0000", 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.038803405999999999 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.606118+0000", 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1693999999999995e-05 2024-02-12T16:16:55.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.606190+0000", 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000136701 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.606326+0000", 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.052226360999999999 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:263277 4.7 4:e6b63bf5:::benchmark_data_smithi169_60938_object89940:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:53.560510+0000", 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.492411846, 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062677158999999996, 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 263277 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560510+0000", 2024-02-12T16:16:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560510+0000", 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560509+0000", 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.539e-05 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560525+0000", 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2799999999999999e-05 2024-02-12T16:16:55.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560538+0000", 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9300000000000002e-06 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560542+0000", 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048533120000000003 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.565396+0000", 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5651999999999998e-05 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.565491+0000", 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000123387 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.565615+0000", 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011324532 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.576939+0000", 2024-02-12T16:16:55.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029273950000000002 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.579867+0000", 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.043138573999999999 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.623005+0000", 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4831000000000003e-05 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.623070+0000", 2024-02-12T16:16:55.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117427 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.623188+0000", 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062677158999999996 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:263279 4.7 4:e24fed27:::benchmark_data_smithi169_60938_object89942:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:53.560547+0000", 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.4923749589999999, 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070380943000000001, 2024-02-12T16:16:55.141 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 263279 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560547+0000", 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560547+0000", 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560546+0000", 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7069999999999998e-06 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560550+0000", 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9609999999999996e-06 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560555+0000", 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1960000000000001e-06 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560559+0000", 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086128639999999996 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.569172+0000", 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010993999999999999 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.144 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.569282+0000", 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104549 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.569387+0000", 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075612120000000003 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.576948+0000", 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029613119999999998 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.579909+0000", 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.050824741999999999 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.630734+0000", 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9920000000000002e-05 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.630794+0000", 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013435699999999999 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.630928+0000", 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070380943000000001 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:263282 4.7 4:eee7feaf:::benchmark_data_smithi169_60938_object89945:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:53.560699+0000", 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.4922233039999999, 2024-02-12T16:16:55.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070485122999999997, 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 263282 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560699+0000", 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560699+0000", 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560698+0000", 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0669999999999999e-06 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560703+0000", 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6640000000000001e-06 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560709+0000", 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6289999999999999e-06 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560713+0000", 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01241046 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.573123+0000", 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4479000000000002e-05 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.573208+0000", 2024-02-12T16:16:55.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8281000000000004e-05 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.573296+0000", 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046021719999999999 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.577898+0000", 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002060409 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.579959+0000", 2024-02-12T16:16:55.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.051058427000000003 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.631017+0000", 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7184e-05 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.631074+0000", 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109629 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.631184+0000", 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070485122999999997 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:264645 4.7 4:edf50b6e:::benchmark_data_smithi169_60938_object91308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:54.920927+0000", 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.13199555700000001, 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.072150912999999997, 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:16:55.153 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264645 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920927+0000", 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920927+0000", 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920924+0000", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4759999999999998e-06 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920933+0000", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8766999999999998e-05 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920951+0000", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9646e-05 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920981+0000", 2024-02-12T16:16:55.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00059622699999999995 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.921577+0000", 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013502100000000001 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.921712+0000", 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019237 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:16:55.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.921905+0000", 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003766049 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.925671+0000", 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037594800000000003 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.926047+0000", 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.066859926 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:16:55.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.992906+0000", 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3754000000000001e-05 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.992960+0000", 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117371 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.993078+0000", 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.072150912999999997 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.158 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:16:55.159 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:16:55.159 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:16:55.159 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:55.160 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:54 smithi169 bash[19604]: cluster 2024-02-12T16:16:53.740125+0000 mgr.y (mgr.14146) 947 : cluster [DBG] pgmap v1243: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 30157/172944 objects misplaced (17.437%); 15 MiB/s, 232 objects/s recovering 2024-02-12T16:16:55.193 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:55.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:55.326 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:55.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:55.457 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:55.560 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:55.597 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:55.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:55.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:55.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:55.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:55.863 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:55.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:55 smithi123 bash[19119]: cluster 2024-02-12T16:16:55.410435+0000 mon.a (mon.0) 1511 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:16:55.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-12T16:16:55.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:55.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:55.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:55 smithi169 bash[19604]: cluster 2024-02-12T16:16:55.410435+0000 mon.a (mon.0) 1511 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:16:56.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:56.132 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:56.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:56.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:55 smithi121 bash[18489]: cluster 2024-02-12T16:16:55.410435+0000 mon.a (mon.0) 1511 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:16:56.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:56.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:16:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:56.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:16:56.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:56.404 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:56.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:56.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:56.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:56.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:56.670 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:56.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:56.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:56.813 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:56 smithi123 bash[19119]: cluster 2024-02-12T16:16:55.480270+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:56.813 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:56 smithi123 bash[19119]: cluster 2024-02-12T16:16:55.480323+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:56.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:56.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:56.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:56.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:56 smithi169 bash[19604]: cluster 2024-02-12T16:16:55.480270+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:56.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:56 smithi169 bash[19604]: cluster 2024-02-12T16:16:55.480323+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:56.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:56 smithi169 bash[19604]: cluster 2024-02-12T16:16:55.640240+0000 osd.4 (osd.4) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:56.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:56 smithi169 bash[19604]: cluster 2024-02-12T16:16:55.640289+0000 osd.4 (osd.4) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:56.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:56 smithi169 bash[19604]: cluster 2024-02-12T16:16:55.740619+0000 mgr.y (mgr.14146) 948 : cluster [DBG] pgmap v1244: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 28610/165532 objects misplaced (17.284%); 16 MiB/s, 262 objects/s recovering 2024-02-12T16:16:56.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:56 smithi169 bash[19604]: cluster 2024-02-12T16:16:56.424079+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e396: 12 total, 12 up, 12 in 2024-02-12T16:16:56.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:56.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:57.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:57.085 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:57.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:56 smithi123 bash[19119]: cluster 2024-02-12T16:16:55.640240+0000 osd.4 (osd.4) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:57.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:56 smithi123 bash[19119]: cluster 2024-02-12T16:16:55.640289+0000 osd.4 (osd.4) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:57.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:56 smithi123 bash[19119]: cluster 2024-02-12T16:16:55.740619+0000 mgr.y (mgr.14146) 948 : cluster [DBG] pgmap v1244: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 28610/165532 objects misplaced (17.284%); 16 MiB/s, 262 objects/s recovering 2024-02-12T16:16:57.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:56 smithi123 bash[19119]: cluster 2024-02-12T16:16:56.424079+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e396: 12 total, 12 up, 12 in 2024-02-12T16:16:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:57.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:56 smithi121 bash[18489]: cluster 2024-02-12T16:16:55.480270+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:56 smithi121 bash[18489]: cluster 2024-02-12T16:16:55.480323+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:56 smithi121 bash[18489]: cluster 2024-02-12T16:16:55.640240+0000 osd.4 (osd.4) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:56 smithi121 bash[18489]: cluster 2024-02-12T16:16:55.640289+0000 osd.4 (osd.4) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:56 smithi121 bash[18489]: cluster 2024-02-12T16:16:55.740619+0000 mgr.y (mgr.14146) 948 : cluster [DBG] pgmap v1244: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 28610/165532 objects misplaced (17.284%); 16 MiB/s, 262 objects/s recovering 2024-02-12T16:16:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:56 smithi121 bash[18489]: cluster 2024-02-12T16:16:56.424079+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e396: 12 total, 12 up, 12 in 2024-02-12T16:16:57.192 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:57.213 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:57.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:57.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:57.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:57.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:57.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:16:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:57.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:16:57.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:57.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:57.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:57.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:57.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:16:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:57.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:16:57.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:57.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:57.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:57.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:57 smithi121 bash[18489]: cluster 2024-02-12T16:16:56.622367+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:57 smithi121 bash[18489]: cluster 2024-02-12T16:16:56.622405+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:57 smithi121 bash[18489]: cluster 2024-02-12T16:16:57.432649+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e397: 12 total, 12 up, 12 in 2024-02-12T16:16:57.992 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:58.016 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:58.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:58.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:57 smithi169 bash[19604]: cluster 2024-02-12T16:16:56.622367+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:57 smithi169 bash[19604]: cluster 2024-02-12T16:16:56.622405+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:58.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:57 smithi169 bash[19604]: cluster 2024-02-12T16:16:57.432649+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e397: 12 total, 12 up, 12 in 2024-02-12T16:16:58.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:58.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:57 smithi123 bash[19119]: cluster 2024-02-12T16:16:56.622367+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:58.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:57 smithi123 bash[19119]: cluster 2024-02-12T16:16:56.622405+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:58.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:57 smithi123 bash[19119]: cluster 2024-02-12T16:16:57.432649+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e397: 12 total, 12 up, 12 in 2024-02-12T16:16:58.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:58.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:58.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:58.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:58.304 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:58.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:58.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:58.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:58.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:58.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:58.604 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:58.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:58.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:16:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:58.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:16:58.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:58.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:16:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:58.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:16:58.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:58.991 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:58 smithi121 bash[18489]: cluster 2024-02-12T16:16:57.730801+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:58.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:58 smithi121 bash[18489]: cluster 2024-02-12T16:16:57.730861+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:58.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:58 smithi121 bash[18489]: cluster 2024-02-12T16:16:57.741061+0000 mgr.y (mgr.14146) 949 : cluster [DBG] pgmap v1247: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 27892/163636 objects misplaced (17.045%); 17 MiB/s, 266 objects/s recovering 2024-02-12T16:16:58.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:58 smithi121 bash[18489]: cluster 2024-02-12T16:16:57.761113+0000 osd.8 (osd.8) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:58.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:58 smithi121 bash[18489]: cluster 2024-02-12T16:16:57.761162+0000 osd.8 (osd.8) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:58.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:58 smithi121 bash[18489]: cluster 2024-02-12T16:16:58.435425+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e398: 12 total, 12 up, 12 in 2024-02-12T16:16:59.004 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:16:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:16:59.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:58 smithi123 bash[19119]: cluster 2024-02-12T16:16:57.730801+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:58 smithi123 bash[19119]: cluster 2024-02-12T16:16:57.730861+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:58 smithi123 bash[19119]: cluster 2024-02-12T16:16:57.741061+0000 mgr.y (mgr.14146) 949 : cluster [DBG] pgmap v1247: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 27892/163636 objects misplaced (17.045%); 17 MiB/s, 266 objects/s recovering 2024-02-12T16:16:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:58 smithi123 bash[19119]: cluster 2024-02-12T16:16:57.761113+0000 osd.8 (osd.8) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:58 smithi123 bash[19119]: cluster 2024-02-12T16:16:57.761162+0000 osd.8 (osd.8) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:58 smithi123 bash[19119]: cluster 2024-02-12T16:16:58.435425+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e398: 12 total, 12 up, 12 in 2024-02-12T16:16:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:16:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:16:59.177 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:16:59.235 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:16:59.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:59.270 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:16:59.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:16:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:16:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:58 smithi169 bash[19604]: cluster 2024-02-12T16:16:57.730801+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:58 smithi169 bash[19604]: cluster 2024-02-12T16:16:57.730861+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:58 smithi169 bash[19604]: cluster 2024-02-12T16:16:57.741061+0000 mgr.y (mgr.14146) 949 : cluster [DBG] pgmap v1247: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 27892/163636 objects misplaced (17.045%); 17 MiB/s, 266 objects/s recovering 2024-02-12T16:16:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:58 smithi169 bash[19604]: cluster 2024-02-12T16:16:57.761113+0000 osd.8 (osd.8) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:16:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:58 smithi169 bash[19604]: cluster 2024-02-12T16:16:57.761162+0000 osd.8 (osd.8) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:16:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:58 smithi169 bash[19604]: cluster 2024-02-12T16:16:58.435425+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e398: 12 total, 12 up, 12 in 2024-02-12T16:16:59.326 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:16:59.366 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:16:59.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:59.405 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:16:59.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:16:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:16:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:16:59.486 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:16:59.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:59.526 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:16:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:16:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:16:59.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:16:59.582 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:16:59.616 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.628 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.629 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:16:59.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:59.686 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:59.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:59.797 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:59 smithi121 bash[18489]: debug 2024-02-12T16:16:59.533+0000 7fbba85de700 -1 mon.a@0(leader).osd e398 definitely_dead 0 2024-02-12T16:16:59.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:59 smithi123 bash[19119]: audit 2024-02-12T16:16:59.121162+0000 mon.a (mon.0) 1515 : audit [DBG] from='client.? v1:172.21.15.121:0/1911841665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:16:59.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:59 smithi123 bash[19119]: audit 2024-02-12T16:16:59.475046+0000 mon.a (mon.0) 1516 : audit [INF] from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:59.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:59 smithi123 bash[19119]: audit 2024-02-12T16:16:59.476682+0000 mon.c (mon.1) 36 : audit [INF] from='client.? v1:172.21.15.121:0/1536947391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:16:59.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:59 smithi123 bash[19119]: audit 2024-02-12T16:16:59.536787+0000 mon.a (mon.0) 1517 : audit [INF] from='client.25496 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:16:59.840 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:16:59.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:59.841 INFO:teuthology.orchestra.run.smithi121.stderr:marked out osd.8. 2024-02-12T16:16:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:16:59.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:59.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:16:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:16:59.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:00.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:00.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:59 smithi123 bash[19119]: cluster 2024-02-12T16:16:59.536913+0000 mon.a (mon.0) 1518 : cluster [INF] Client client.admin marked osd.8 out, while it was still marked up 2024-02-12T16:17:00.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:16:59 smithi123 bash[19119]: audit 2024-02-12T16:16:59.539060+0000 mon.b (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.121:0/3157793494' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:17:00.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:00.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:59 smithi121 bash[18489]: audit 2024-02-12T16:16:59.121162+0000 mon.a (mon.0) 1515 : audit [DBG] from='client.? v1:172.21.15.121:0/1911841665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:59 smithi121 bash[18489]: audit 2024-02-12T16:16:59.475046+0000 mon.a (mon.0) 1516 : audit [INF] from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:17:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:59 smithi121 bash[18489]: audit 2024-02-12T16:16:59.476682+0000 mon.c (mon.1) 36 : audit [INF] from='client.? v1:172.21.15.121:0/1536947391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:17:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:59 smithi121 bash[18489]: audit 2024-02-12T16:16:59.536787+0000 mon.a (mon.0) 1517 : audit [INF] from='client.25496 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:17:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:59 smithi121 bash[18489]: cluster 2024-02-12T16:16:59.536913+0000 mon.a (mon.0) 1518 : cluster [INF] Client client.admin marked osd.8 out, while it was still marked up 2024-02-12T16:17:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:16:59 smithi121 bash[18489]: audit 2024-02-12T16:16:59.539060+0000 mon.b (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.121:0/3157793494' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:17:00.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:00.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:00.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:59 smithi169 bash[19604]: audit 2024-02-12T16:16:59.121162+0000 mon.a (mon.0) 1515 : audit [DBG] from='client.? v1:172.21.15.121:0/1911841665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:59 smithi169 bash[19604]: audit 2024-02-12T16:16:59.475046+0000 mon.a (mon.0) 1516 : audit [INF] from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:17:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:59 smithi169 bash[19604]: audit 2024-02-12T16:16:59.476682+0000 mon.c (mon.1) 36 : audit [INF] from='client.? v1:172.21.15.121:0/1536947391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:17:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:59 smithi169 bash[19604]: audit 2024-02-12T16:16:59.536787+0000 mon.a (mon.0) 1517 : audit [INF] from='client.25496 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:17:00.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:59 smithi169 bash[19604]: cluster 2024-02-12T16:16:59.536913+0000 mon.a (mon.0) 1518 : cluster [INF] Client client.admin marked osd.8 out, while it was still marked up 2024-02-12T16:17:00.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:16:59 smithi169 bash[19604]: audit 2024-02-12T16:16:59.539060+0000 mon.b (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.121:0/3157793494' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-02-12T16:17:00.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:00.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:00.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:00.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:00.514 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:00.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:00.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:00.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:00.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:00.738 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:17:00.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:00.772 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:00.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:00.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:00 smithi123 bash[19119]: cluster 2024-02-12T16:16:59.741546+0000 mgr.y (mgr.14146) 950 : cluster [DBG] pgmap v1249: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 26636/161792 objects misplaced (16.463%); 15 MiB/s, 241 objects/s recovering 2024-02-12T16:17:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:00 smithi123 bash[19119]: cluster 2024-02-12T16:16:59.810526+0000 mon.a (mon.0) 1519 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:00 smithi123 bash[19119]: audit 2024-02-12T16:16:59.826070+0000 mon.a (mon.0) 1520 : audit [INF] from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:17:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:00 smithi123 bash[19119]: audit 2024-02-12T16:16:59.826137+0000 mon.a (mon.0) 1521 : audit [INF] from='client.25496 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-02-12T16:17:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:00 smithi123 bash[19119]: cluster 2024-02-12T16:16:59.826182+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e399: 12 total, 12 up, 11 in 2024-02-12T16:17:00.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:00.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:01.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:01.062 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:01.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:01.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:00 smithi121 bash[18489]: cluster 2024-02-12T16:16:59.741546+0000 mgr.y (mgr.14146) 950 : cluster [DBG] pgmap v1249: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 26636/161792 objects misplaced (16.463%); 15 MiB/s, 241 objects/s recovering 2024-02-12T16:17:01.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:00 smithi121 bash[18489]: cluster 2024-02-12T16:16:59.810526+0000 mon.a (mon.0) 1519 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:01.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:00 smithi121 bash[18489]: audit 2024-02-12T16:16:59.826070+0000 mon.a (mon.0) 1520 : audit [INF] from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:17:01.169 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:00 smithi121 bash[18489]: audit 2024-02-12T16:16:59.826137+0000 mon.a (mon.0) 1521 : audit [INF] from='client.25496 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-02-12T16:17:01.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:00 smithi121 bash[18489]: cluster 2024-02-12T16:16:59.826182+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e399: 12 total, 12 up, 11 in 2024-02-12T16:17:01.176 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:01.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:01.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:01.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:00 smithi169 bash[19604]: cluster 2024-02-12T16:16:59.741546+0000 mgr.y (mgr.14146) 950 : cluster [DBG] pgmap v1249: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 26636/161792 objects misplaced (16.463%); 15 MiB/s, 241 objects/s recovering 2024-02-12T16:17:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:00 smithi169 bash[19604]: cluster 2024-02-12T16:16:59.810526+0000 mon.a (mon.0) 1519 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:00 smithi169 bash[19604]: audit 2024-02-12T16:16:59.826070+0000 mon.a (mon.0) 1520 : audit [INF] from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:17:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:00 smithi169 bash[19604]: audit 2024-02-12T16:16:59.826137+0000 mon.a (mon.0) 1521 : audit [INF] from='client.25496 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-02-12T16:17:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:00 smithi169 bash[19604]: cluster 2024-02-12T16:16:59.826182+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e399: 12 total, 12 up, 11 in 2024-02-12T16:17:01.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:01.356 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:01.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:01.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:01.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:01.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:01.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:01.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:01.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:01.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:01.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:01.868 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:01.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:01.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:01 smithi121 bash[18489]: cluster 2024-02-12T16:17:00.830919+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e400: 12 total, 12 up, 11 in 2024-02-12T16:17:02.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:02.006 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:01 smithi169 bash[19604]: cluster 2024-02-12T16:17:00.830919+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e400: 12 total, 12 up, 11 in 2024-02-12T16:17:02.026 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:02.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:02.137 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:01 smithi123 bash[19119]: cluster 2024-02-12T16:17:00.830919+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e400: 12 total, 12 up, 11 in 2024-02-12T16:17:02.159 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:02.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:02.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:02.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:02.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:02.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:02.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:02.583 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:02.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:02.715 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:02.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:02.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:02.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:02.972 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:02.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:03.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:03.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:03.126 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:02 smithi121 bash[18489]: cluster 2024-02-12T16:17:01.741998+0000 mgr.y (mgr.14146) 951 : cluster [DBG] pgmap v1252: 27 pgs: 1 peering, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 4.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 989 op/s; 25146/155106 objects misplaced (16.212%); 21 MiB/s, 335 objects/s recovering 2024-02-12T16:17:03.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:02 smithi123 bash[19119]: cluster 2024-02-12T16:17:01.741998+0000 mgr.y (mgr.14146) 951 : cluster [DBG] pgmap v1252: 27 pgs: 1 peering, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 4.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 989 op/s; 25146/155106 objects misplaced (16.212%); 21 MiB/s, 335 objects/s recovering 2024-02-12T16:17:03.139 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:02 smithi169 bash[19604]: cluster 2024-02-12T16:17:01.741998+0000 mgr.y (mgr.14146) 951 : cluster [DBG] pgmap v1252: 27 pgs: 1 peering, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 4.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 989 op/s; 25146/155106 objects misplaced (16.212%); 21 MiB/s, 335 objects/s recovering 2024-02-12T16:17:03.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:03.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:03.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:03.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:03.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:03.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:03.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:03.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:03.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:03.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:03.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:03.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:03.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:03.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:17:03.816 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:03.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:03.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:03.942 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:03.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:04.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:17:04.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:04.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:04.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:04.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:04.204 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:04.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:04.308 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:04.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:04.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:04.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:04.478 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:04.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:04.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:04.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:04.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:04.712 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:04.736 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:04.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:04.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:04.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:04.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:04.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:04.983 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:04 smithi123 bash[19119]: cluster 2024-02-12T16:17:03.742542+0000 mgr.y (mgr.14146) 952 : cluster [DBG] pgmap v1253: 27 pgs: 1 peering, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 4.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 861 op/s; 23973/153296 objects misplaced (15.638%); 22 MiB/s, 357 objects/s recovering 2024-02-12T16:17:05.003 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:05.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:05.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:05.114 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:04 smithi121 bash[18489]: cluster 2024-02-12T16:17:03.742542+0000 mgr.y (mgr.14146) 952 : cluster [DBG] pgmap v1253: 27 pgs: 1 peering, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 4.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 861 op/s; 23973/153296 objects misplaced (15.638%); 22 MiB/s, 357 objects/s recovering 2024-02-12T16:17:05.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:05.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:05.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:05.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:04 smithi169 bash[19604]: cluster 2024-02-12T16:17:03.742542+0000 mgr.y (mgr.14146) 952 : cluster [DBG] pgmap v1253: 27 pgs: 1 peering, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 20 active+clean; 4.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 861 op/s; 23973/153296 objects misplaced (15.638%); 22 MiB/s, 357 objects/s recovering 2024-02-12T16:17:05.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:05.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:05.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:05.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:05.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:05.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:05.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:05.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:05.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:05.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:05.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:05.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:05.849 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:05.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:05.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:05.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:06.046 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:17:06.046 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:17:06.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:06.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:06.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:06.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:06.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:06.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:06.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:06.519 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:06.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:06.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:06.689 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:06 smithi121 bash[18489]: cluster 2024-02-12T16:17:05.743065+0000 mgr.y (mgr.14146) 953 : cluster [DBG] pgmap v1254: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s; 22453/145796 objects misplaced (15.400%); 25 MiB/s, 401 objects/s recovering 2024-02-12T16:17:06.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:06.795 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:06 smithi169 bash[19604]: cluster 2024-02-12T16:17:05.743065+0000 mgr.y (mgr.14146) 953 : cluster [DBG] pgmap v1254: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s; 22453/145796 objects misplaced (15.400%); 25 MiB/s, 401 objects/s recovering 2024-02-12T16:17:06.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:06 smithi123 bash[19119]: cluster 2024-02-12T16:17:05.743065+0000 mgr.y (mgr.14146) 953 : cluster [DBG] pgmap v1254: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s; 22453/145796 objects misplaced (15.400%); 25 MiB/s, 401 objects/s recovering 2024-02-12T16:17:06.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:06.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:06.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:06.997 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:17:06.997 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:17:06.997 INFO:tasks.thrashosds.thrasher:Growing pool device_health_metrics 2024-02-12T16:17:06.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:17:07.005 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-12T16:17:07.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:07.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:07.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:07.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:07.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:07.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:07.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:07.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:07.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:07.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:07.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:07.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:07.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:07.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:07.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:07.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:07.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:07.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:07.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:07.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:07.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:08.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:08.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:08.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:08.200 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:08.222 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:08.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:08.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:08.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:08.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:08.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:08.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:08.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:08.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:08.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:08.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:08.654 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:08.658 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:08.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:08.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:08.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:08.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:08.902 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:08 smithi121 bash[18489]: cluster 2024-02-12T16:17:07.745029+0000 mgr.y (mgr.14146) 954 : cluster [DBG] pgmap v1255: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 21796/143930 objects misplaced (15.143%); 18 MiB/s, 280 objects/s recovering 2024-02-12T16:17:08.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:08.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:09.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:09.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:09.084 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:08 smithi123 bash[19119]: cluster 2024-02-12T16:17:07.745029+0000 mgr.y (mgr.14146) 954 : cluster [DBG] pgmap v1255: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 21796/143930 objects misplaced (15.143%); 18 MiB/s, 280 objects/s recovering 2024-02-12T16:17:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:09.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:09.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:09.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:09.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:08 smithi169 bash[19604]: cluster 2024-02-12T16:17:07.745029+0000 mgr.y (mgr.14146) 954 : cluster [DBG] pgmap v1255: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 21796/143930 objects misplaced (15.143%); 18 MiB/s, 280 objects/s recovering 2024-02-12T16:17:09.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:09.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:09.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:09.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:09.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:09.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:09.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:09.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:09.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:09.782 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:09.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:09.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:09 smithi123 bash[19119]: cluster 2024-02-12T16:17:08.311997+0000 osd.11 (osd.11) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:09.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:09 smithi123 bash[19119]: cluster 2024-02-12T16:17:08.312053+0000 osd.11 (osd.11) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:09.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:09 smithi123 bash[19119]: cluster 2024-02-12T16:17:08.801977+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e401: 12 total, 12 up, 11 in 2024-02-12T16:17:09.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:09 smithi121 bash[18489]: cluster 2024-02-12T16:17:08.311997+0000 osd.11 (osd.11) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:09 smithi121 bash[18489]: cluster 2024-02-12T16:17:08.312053+0000 osd.11 (osd.11) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:09 smithi121 bash[18489]: cluster 2024-02-12T16:17:08.801977+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e401: 12 total, 12 up, 11 in 2024-02-12T16:17:09.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:09.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:09.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:10.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:10.051 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:10.056 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:10.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:10.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:09 smithi169 bash[19604]: cluster 2024-02-12T16:17:08.311997+0000 osd.11 (osd.11) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:10.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:09 smithi169 bash[19604]: cluster 2024-02-12T16:17:08.312053+0000 osd.11 (osd.11) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:10.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:09 smithi169 bash[19604]: cluster 2024-02-12T16:17:08.801977+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e401: 12 total, 12 up, 11 in 2024-02-12T16:17:10.187 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:10.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:10.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:10.315 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:10.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:10.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:10.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:10.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:10.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:10.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:10.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:10.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:10.720 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:10.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:10.816 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:10 smithi121 bash[18489]: cluster 2024-02-12T16:17:09.745498+0000 mgr.y (mgr.14146) 955 : cluster [DBG] pgmap v1257: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 20654/141968 objects misplaced (14.548%); 16 MiB/s, 256 objects/s recovering 2024-02-12T16:17:10.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:10.831 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:10 smithi123 bash[19119]: cluster 2024-02-12T16:17:09.745498+0000 mgr.y (mgr.14146) 955 : cluster [DBG] pgmap v1257: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 20654/141968 objects misplaced (14.548%); 16 MiB/s, 256 objects/s recovering 2024-02-12T16:17:10.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:10.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:10.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:10.992 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:11.104 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:11.126 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:11.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:11.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:11.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:10 smithi169 bash[19604]: cluster 2024-02-12T16:17:09.745498+0000 mgr.y (mgr.14146) 955 : cluster [DBG] pgmap v1257: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 20654/141968 objects misplaced (14.548%); 16 MiB/s, 256 objects/s recovering 2024-02-12T16:17:11.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:11.368 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:11.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:11.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:11.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:11.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:11.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:11.696 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:11.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:11.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:11.791 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:17:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:11.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:11.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:11.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:11.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:12.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:12.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:11 smithi121 bash[18489]: cluster 2024-02-12T16:17:11.019279+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:12.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:11 smithi121 bash[18489]: cluster 2024-02-12T16:17:11.019330+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:12.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:11 smithi169 bash[19604]: cluster 2024-02-12T16:17:11.019279+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:12.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:11 smithi169 bash[19604]: cluster 2024-02-12T16:17:11.019330+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:12.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:12.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:11 smithi123 bash[19119]: cluster 2024-02-12T16:17:11.019279+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:11 smithi123 bash[19119]: cluster 2024-02-12T16:17:11.019330+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:12.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:12.210 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:12.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:12.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:12.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:12.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:12.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:12.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:12.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:12.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:12.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:12.656 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:12.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:12.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:12.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:12.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:12.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:12.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:12 smithi121 bash[18489]: cluster 2024-02-12T16:17:11.746016+0000 mgr.y (mgr.14146) 956 : cluster [DBG] pgmap v1258: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 982 op/s; 19255/135450 objects misplaced (14.216%); 18 MiB/s, 292 objects/s recovering 2024-02-12T16:17:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:12 smithi121 bash[18489]: cluster 2024-02-12T16:17:11.816960+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e402: 12 total, 12 up, 11 in 2024-02-12T16:17:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:12.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:13.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:13.043 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:12 smithi169 bash[19604]: cluster 2024-02-12T16:17:11.746016+0000 mgr.y (mgr.14146) 956 : cluster [DBG] pgmap v1258: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 982 op/s; 19255/135450 objects misplaced (14.216%); 18 MiB/s, 292 objects/s recovering 2024-02-12T16:17:13.044 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:12 smithi169 bash[19604]: cluster 2024-02-12T16:17:11.816960+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e402: 12 total, 12 up, 11 in 2024-02-12T16:17:13.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:13.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:12 smithi123 bash[19119]: cluster 2024-02-12T16:17:11.746016+0000 mgr.y (mgr.14146) 956 : cluster [DBG] pgmap v1258: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 982 op/s; 19255/135450 objects misplaced (14.216%); 18 MiB/s, 292 objects/s recovering 2024-02-12T16:17:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:12 smithi123 bash[19119]: cluster 2024-02-12T16:17:11.816960+0000 mon.a (mon.0) 1525 : cluster [DBG] osdmap e402: 12 total, 12 up, 11 in 2024-02-12T16:17:13.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:13.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:13.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:13.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:13.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:13.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:13.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:13.473 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:13.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:13.576 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:13.603 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:13.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:13.709 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:13.734 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:13.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:13.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:13.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:13.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:13.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:14.023 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:14.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:14.163 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:14.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:14.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:14.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:14.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:14.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:14.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:14.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:14.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:14.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:14.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:14.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:14.986 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:14.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:15.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:15.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:15.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:15.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:15.241 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:15 smithi169 bash[19604]: cluster 2024-02-12T16:17:13.746532+0000 mgr.y (mgr.14146) 957 : cluster [DBG] pgmap v1260: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 18160/133700 objects misplaced (13.583%); 17 MiB/s, 270 objects/s recovering 2024-02-12T16:17:15.241 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:15 smithi169 bash[19604]: audit 2024-02-12T16:17:14.180399+0000 mon.a (mon.0) 1526 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:17:15.257 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:15.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:15.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:15.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:15.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:15.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:15 smithi121 bash[18489]: cluster 2024-02-12T16:17:13.746532+0000 mgr.y (mgr.14146) 957 : cluster [DBG] pgmap v1260: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 18160/133700 objects misplaced (13.583%); 17 MiB/s, 270 objects/s recovering 2024-02-12T16:17:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:15 smithi121 bash[18489]: audit 2024-02-12T16:17:14.180399+0000 mon.a (mon.0) 1526 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:17:15.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:15.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:15.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:15.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:15.636 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:15 smithi123 bash[19119]: cluster 2024-02-12T16:17:13.746532+0000 mgr.y (mgr.14146) 957 : cluster [DBG] pgmap v1260: 27 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 21 active+clean; 4.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 18160/133700 objects misplaced (13.583%); 17 MiB/s, 270 objects/s recovering 2024-02-12T16:17:15.636 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:15 smithi123 bash[19119]: audit 2024-02-12T16:17:14.180399+0000 mon.a (mon.0) 1526 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:17:15.656 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:15.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:15.759 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:15.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:15.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:15.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:15.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:15.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:16.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:16.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:16.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:16.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:16.204 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:16.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:16.307 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:16.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:16.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:16.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:16.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:16.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:16.572 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:16.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:16.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:16 smithi123 bash[19119]: cluster 2024-02-12T16:17:15.747079+0000 mgr.y (mgr.14146) 958 : cluster [DBG] pgmap v1261: 27 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 21 active+clean; 3.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 16680/126114 objects misplaced (13.226%); 23 MiB/s, 361 objects/s recovering 2024-02-12T16:17:16.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:16 smithi121 bash[18489]: cluster 2024-02-12T16:17:15.747079+0000 mgr.y (mgr.14146) 958 : cluster [DBG] pgmap v1261: 27 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 21 active+clean; 3.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 16680/126114 objects misplaced (13.226%); 23 MiB/s, 361 objects/s recovering 2024-02-12T16:17:16.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:16 smithi169 bash[19604]: cluster 2024-02-12T16:17:15.747079+0000 mgr.y (mgr.14146) 958 : cluster [DBG] pgmap v1261: 27 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 21 active+clean; 3.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 16680/126114 objects misplaced (13.226%); 23 MiB/s, 361 objects/s recovering 2024-02-12T16:17:16.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:16.735 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:16.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:16.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:16.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:16.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:16.912 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:17:16.913 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:17:16.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:17.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:17.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:17.145 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:17.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:17.290 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:17.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:17.429 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:17.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:17.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:17.699 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:17.704 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:17 smithi121 bash[18489]: cluster 2024-02-12T16:17:16.416865+0000 mon.a (mon.0) 1527 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:17:17.704 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:17 smithi121 bash[18489]: cluster 2024-02-12T16:17:16.431316+0000 mon.a (mon.0) 1528 : cluster [DBG] osdmap e403: 12 total, 12 up, 11 in 2024-02-12T16:17:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:17.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:17 smithi169 bash[19604]: cluster 2024-02-12T16:17:16.416865+0000 mon.a (mon.0) 1527 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:17:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:17 smithi169 bash[19604]: cluster 2024-02-12T16:17:16.431316+0000 mon.a (mon.0) 1528 : cluster [DBG] osdmap e403: 12 total, 12 up, 11 in 2024-02-12T16:17:17.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:17.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:17.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-12T16:17:17.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:17 smithi123 bash[19119]: cluster 2024-02-12T16:17:16.416865+0000 mon.a (mon.0) 1527 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:17:17.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:17 smithi123 bash[19119]: cluster 2024-02-12T16:17:16.431316+0000 mon.a (mon.0) 1528 : cluster [DBG] osdmap e403: 12 total, 12 up, 11 in 2024-02-12T16:17:17.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:17.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:17.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:17.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:18.014 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:17:18.033 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:17:18.056 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:17:18.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:18.083 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:17:18.090 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:18.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:18.108 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:17:18.128 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:17:18.163 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:17:18.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:18.194 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:17:18.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:18.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:18.225 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:17:18.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:17:18.290 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:17:18.320 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.323 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:17:18.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:18.363 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:18.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:18.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:18.503 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:18.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:18.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:18.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:18.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:18 smithi123 bash[19119]: cluster 2024-02-12T16:17:17.429472+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e404: 12 total, 12 up, 11 in 2024-02-12T16:17:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:18 smithi123 bash[19119]: cluster 2024-02-12T16:17:17.747492+0000 mgr.y (mgr.14146) 959 : cluster [DBG] pgmap v1264: 27 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 21 active+clean; 3.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 925 op/s; 16012/124344 objects misplaced (12.877%); 16 MiB/s, 260 objects/s recovering 2024-02-12T16:17:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:18 smithi123 bash[19119]: audit 2024-02-12T16:17:17.987154+0000 mon.a (mon.0) 1530 : audit [DBG] from='client.? v1:172.21.15.121:0/2457098988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:18 smithi123 bash[19119]: cluster 2024-02-12T16:17:18.429757+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e405: 12 total, 12 up, 11 in 2024-02-12T16:17:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:18 smithi121 bash[18489]: cluster 2024-02-12T16:17:17.429472+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e404: 12 total, 12 up, 11 in 2024-02-12T16:17:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:18 smithi121 bash[18489]: cluster 2024-02-12T16:17:17.747492+0000 mgr.y (mgr.14146) 959 : cluster [DBG] pgmap v1264: 27 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 21 active+clean; 3.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 925 op/s; 16012/124344 objects misplaced (12.877%); 16 MiB/s, 260 objects/s recovering 2024-02-12T16:17:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:18 smithi121 bash[18489]: audit 2024-02-12T16:17:17.987154+0000 mon.a (mon.0) 1530 : audit [DBG] from='client.? v1:172.21.15.121:0/2457098988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:18 smithi121 bash[18489]: cluster 2024-02-12T16:17:18.429757+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e405: 12 total, 12 up, 11 in 2024-02-12T16:17:18.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:18.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:18.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:18.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:18 smithi169 bash[19604]: cluster 2024-02-12T16:17:17.429472+0000 mon.a (mon.0) 1529 : cluster [DBG] osdmap e404: 12 total, 12 up, 11 in 2024-02-12T16:17:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:18 smithi169 bash[19604]: cluster 2024-02-12T16:17:17.747492+0000 mgr.y (mgr.14146) 959 : cluster [DBG] pgmap v1264: 27 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 21 active+clean; 3.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 925 op/s; 16012/124344 objects misplaced (12.877%); 16 MiB/s, 260 objects/s recovering 2024-02-12T16:17:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:18 smithi169 bash[19604]: audit 2024-02-12T16:17:17.987154+0000 mon.a (mon.0) 1530 : audit [DBG] from='client.? v1:172.21.15.121:0/2457098988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:18 smithi169 bash[19604]: cluster 2024-02-12T16:17:18.429757+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e405: 12 total, 12 up, 11 in 2024-02-12T16:17:18.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:18.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:18.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:18.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:19.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:19.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:19.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:19.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:19.451 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:19.469 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:17:19.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:19.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:19.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:19.706 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:19 smithi123 bash[19119]: audit 2024-02-12T16:17:18.917759+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? v1:172.21.15.121:0/3456805901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:17:19.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:19.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:19 smithi169 bash[19604]: audit 2024-02-12T16:17:18.917759+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? v1:172.21.15.121:0/3456805901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:17:19.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:19.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:19.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:19 smithi121 bash[18489]: audit 2024-02-12T16:17:18.917759+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? v1:172.21.15.121:0/3456805901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:17:19.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:19.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:19.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:20.007 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:17:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:17:20.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:20.110 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:20.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:20.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:20.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:20.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:20.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:20.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:20.484 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:20.510 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:20.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:20.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:20.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:20 smithi169 bash[19604]: cluster 2024-02-12T16:17:19.441978+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:20.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:20 smithi169 bash[19604]: audit 2024-02-12T16:17:19.454793+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? v1:172.21.15.121:0/3456805901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:17:20.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:20 smithi169 bash[19604]: cluster 2024-02-12T16:17:19.454866+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e406: 12 total, 12 up, 11 in 2024-02-12T16:17:20.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:20 smithi169 bash[19604]: cluster 2024-02-12T16:17:19.747933+0000 mgr.y (mgr.14146) 960 : cluster [DBG] pgmap v1267: 27 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 22 active+clean; 3.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 15411/123120 objects misplaced (12.517%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T16:17:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:20 smithi169 bash[19604]: audit 2024-02-12T16:17:19.992239+0000 mgr.y (mgr.14146) 961 : audit [DBG] from='client.15612 v1:172.21.15.121:0/2271165794' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:17:20.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:20.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:20 smithi123 bash[19119]: cluster 2024-02-12T16:17:19.441978+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:20.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:20 smithi123 bash[19119]: audit 2024-02-12T16:17:19.454793+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? v1:172.21.15.121:0/3456805901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:17:20.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:20 smithi123 bash[19119]: cluster 2024-02-12T16:17:19.454866+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e406: 12 total, 12 up, 11 in 2024-02-12T16:17:20.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:20 smithi123 bash[19119]: cluster 2024-02-12T16:17:19.747933+0000 mgr.y (mgr.14146) 960 : cluster [DBG] pgmap v1267: 27 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 22 active+clean; 3.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 15411/123120 objects misplaced (12.517%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T16:17:20.623 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:20 smithi123 bash[19119]: audit 2024-02-12T16:17:19.992239+0000 mgr.y (mgr.14146) 961 : audit [DBG] from='client.15612 v1:172.21.15.121:0/2271165794' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:17:20.639 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:20.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:20.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:20.748 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:20 smithi121 bash[18489]: cluster 2024-02-12T16:17:19.441978+0000 mon.a (mon.0) 1533 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:20.748 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:20 smithi121 bash[18489]: audit 2024-02-12T16:17:19.454793+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? v1:172.21.15.121:0/3456805901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:17:20.748 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:20 smithi121 bash[18489]: cluster 2024-02-12T16:17:19.454866+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e406: 12 total, 12 up, 11 in 2024-02-12T16:17:20.749 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:20 smithi121 bash[18489]: cluster 2024-02-12T16:17:19.747933+0000 mgr.y (mgr.14146) 960 : cluster [DBG] pgmap v1267: 27 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 22 active+clean; 3.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 15411/123120 objects misplaced (12.517%); 7.0 MiB/s, 112 objects/s recovering 2024-02-12T16:17:20.749 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:20 smithi121 bash[18489]: audit 2024-02-12T16:17:19.992239+0000 mgr.y (mgr.14146) 961 : audit [DBG] from='client.15612 v1:172.21.15.121:0/2271165794' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:17:20.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:20.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:20.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:20.925 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:20.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:21.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:21.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:21.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:21.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:21.182 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1267,"stamp":"2024-02-12T16:17:19.747661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4034330648,"num_objects":61560,"num_object_clones":0,"num_object_copies":123120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15411,"num_objects_unfound":0,"num_objects_dirty":61560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285110,"num_write_kb":11093377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9365,"num_bytes_recovered":613679104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":11972220,"kb_used_data":8340580,"kb_used_omap":0,"kb_used_meta":3631488,"kb_avail":1019224452,"statfs":{"total":1055945392128,"available":1043685838848,"internally_reserved":0,"allocated":8540753920,"data_stored":8536704373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3718643712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-98107392,"num_objects":-1497,"num_object_clones":0,"num_object_copies":-2994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1269,"num_objects_unfound":0,"num_objects_dirty":-1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":29425664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000918"},"pg_stats":[{"pgid":"1.a","version":"0'0","reported_seq":48,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:11.823931+0000","last_change":"2024-02-12T16:16:39.823678+0000","last_active":"2024-02-12T16:17:11.823931+0000","last_peered":"2024-02-12T16:17:11.823931+0000","last_clean":"2024-02-12T16:17:11.823931+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:17:11.823931+0000","last_undegraded":"2024-02-12T16:17:11.823931+0000","last_fullsized":"2024-02-12T16:17:11.823931+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:16:39.823550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:16:39.823550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.f","version":"402'17640","reported_seq":26019,"reported_epoch":402,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:17:14.958978+0000","last_change":"2024-02-12T16:16:43.699751+0000","last_active":"2024-02-12T16:17:14.958978+0000","last_peered":"2024-02-12T16:17:14.958978+0000","last_clean":"2024-02-12T16:16:41.366124+0000","last_became_active":"2024-02-12T16:16:43.605294+0000","last_became_peered":"2024-02-12T16:16:43.605294+0000","last_unstale":"2024-02-12T16:17:14.958978+0000","last_undegraded":"2024-02-12T16:17:14.958978+0000","last_fullsized":"2024-02-12T16:17:14.958978+0000","mapping_epoch":393,"log_start":"353'7600","ondisk_log_start":"353'7600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'12227","last_scrub_stamp":"2024-02-12T16:15:49.878411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:49.878411+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":246939648,"num_objects":3768,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":390,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17640,"num_write_kb":685056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4145,"num_bytes_recovered":271646720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3768}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"402'17732","reported_seq":17817,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:14.997924+0000","last_change":"2024-02-12T16:16:35.098799+0000","last_active":"2024-02-12T16:17:14.997924+0000","last_peered":"2024-02-12T16:17:14.997924+0000","last_clean":"2024-02-12T16:17:14.997924+0000","last_became_active":"2024-02-12T16:13:45.970126+0000","last_became_peered":"2024-02-12T16:13:45.970126+0000","last_unstale":"2024-02-12T16:17:14.997924+0000","last_undegraded":"2024-02-12T16:17:14.997924+0000","last_fullsized":"2024-02-12T16:17:14.997924+0000","mapping_epoch":334,"log_start":"353'7700","ondisk_log_start":"353'7700","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15348","last_scrub_stamp":"2024-02-12T16:16:35.098683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:16:35.098683+0000","log_size":10032,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":254017536,"num_objects":3876,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17732,"num_write_kb":691456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":53,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:11.843721+0000","last_change":"2024-02-12T16:16:37.938267+0000","last_active":"2024-02-12T16:17:11.843721+0000","last_peered":"2024-02-12T16:17:11.843721+0000","last_clean":"2024-02-12T16:17:11.843721+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:17:11.843721+0000","last_undegraded":"2024-02-12T16:17:11.843721+0000","last_fullsized":"2024-02-12T16:17:11.843721+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:16:37.938157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:16:37.938157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"402'17632","reported_seq":17713,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:14.997254+0000","last_change":"2024-02-12T16:15:46.343230+0000","last_active":"2024-02-12T16:17:14.997254+0000","last_peered":"2024-02-12T16:17:14.997254+0000","last_clean":"2024-02-12T16:17:14.997254+0000","last_became_active":"2024-02-12T16:13:45.793796+0000","last_became_peered":"2024-02-12T16:13:45.793796+0000","last_unstale":"2024-02-12T16:17:14.997254+0000","last_undegraded":"2024-02-12T16:17:14.997254+0000","last_fullsized":"2024-02-12T16:17:14.997254+0000","mapping_epoch":334,"log_start":"353'7600","ondisk_log_start":"353'7600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'11975","last_scrub_stamp":"2024-02-12T16:15:46.343107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:46.343107+0000","log_size":10032,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256573464,"num_objects":3916,"num_object_clones":0,"num_object_copies":7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17632,"num_write_kb":689473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":459,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-02-12T16:17:16.448930+0000","last_change":"2024-02-12T16:15:46.610472+0000","last_active":"2024-02-12T16:17:16.448930+0000","last_peered":"2024-02-12T16:17:16.448930+0000","last_clean":"2024-02-12T16:17:16.448930+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:17:16.448930+0000","last_undegraded":"2024-02-12T16:17:16.448930+0000","last_fullsized":"2024-02-12T16:17:16.448930+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:46.610395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:46.610395+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"402'17778","reported_seq":17868,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:15.007294+0000","last_change":"2024-02-12T16:16:55.564378+0000","last_active":"2024-02-12T16:17:15.007294+0000","last_peered":"2024-02-12T16:17:15.007294+0000","last_clean":"2024-02-12T16:17:15.007294+0000","last_became_active":"2024-02-12T16:13:45.791545+0000","last_became_peered":"2024-02-12T16:13:45.791545+0000","last_unstale":"2024-02-12T16:17:15.007294+0000","last_undegraded":"2024-02-12T16:17:15.007294+0000","last_fullsized":"2024-02-12T16:17:15.007294+0000","mapping_epoch":334,"log_start":"353'7700","ondisk_log_start":"353'7700","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'11689","last_scrub_stamp":"2024-02-12T16:15:40.640284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:40.640284+0000","log_size":10078,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17778,"num_write_kb":695552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"402'18008","reported_seq":18090,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:15.482524+0000","last_change":"2024-02-12T16:15:38.879179+0000","last_active":"2024-02-12T16:17:15.482524+0000","last_peered":"2024-02-12T16:17:15.482524+0000","last_clean":"2024-02-12T16:17:15.482524+0000","last_became_active":"2024-02-12T16:13:45.790415+0000","last_became_peered":"2024-02-12T16:13:45.790415+0000","last_unstale":"2024-02-12T16:17:15.482524+0000","last_undegraded":"2024-02-12T16:17:15.482524+0000","last_fullsized":"2024-02-12T16:17:15.482524+0000","mapping_epoch":334,"log_start":"353'8000","ondisk_log_start":"353'8000","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'11619","last_scrub_stamp":"2024-02-12T16:15:38.879072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:38.879072+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":248643584,"num_objects":3794,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18008,"num_write_kb":697664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"405'18111","reported_seq":18223,"reported_epoch":405,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:17:19.037807+0000","last_change":"2024-02-12T16:17:13.633940+0000","last_active":"2024-02-12T16:17:19.037807+0000","last_peered":"2024-02-12T16:17:19.037807+0000","last_clean":"2024-02-12T16:16:41.417695+0000","last_became_active":"2024-02-12T16:16:43.579339+0000","last_became_peered":"2024-02-12T16:16:43.579339+0000","last_unstale":"2024-02-12T16:17:19.037807+0000","last_undegraded":"2024-02-12T16:17:19.037807+0000","last_fullsized":"2024-02-12T16:17:19.037807+0000","mapping_epoch":393,"log_start":"353'8100","ondisk_log_start":"353'8100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'15908","last_scrub_stamp":"2024-02-12T16:16:39.721881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:16:39.721881+0000","log_size":10011,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":255524864,"num_objects":3899,"num_object_clones":0,"num_object_copies":7798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3899,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18111,"num_write_kb":704320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":3899}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"402'17920","reported_seq":18287,"reported_epoch":402,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:17:15.051220+0000","last_change":"2024-02-12T16:17:13.706228+0000","last_active":"2024-02-12T16:17:15.051220+0000","last_peered":"2024-02-12T16:17:15.051220+0000","last_clean":"2024-02-12T16:16:41.415277+0000","last_became_active":"2024-02-12T16:16:43.614448+0000","last_became_peered":"2024-02-12T16:16:43.614448+0000","last_unstale":"2024-02-12T16:17:15.051220+0000","last_undegraded":"2024-02-12T16:17:15.051220+0000","last_fullsized":"2024-02-12T16:17:15.051220+0000","mapping_epoch":393,"log_start":"353'7900","ondisk_log_start":"353'7900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15658","last_scrub_stamp":"2024-02-12T16:16:37.730742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:16:37.730742+0000","log_size":10020,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":251658240,"num_objects":3840,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3709,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17920,"num_write_kb":696320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,1],"avail_no_missing":["11","1"],"object_location_counts":[{"shards":"1,11","objects":3840}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"402'17654","reported_seq":17743,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:15.012306+0000","last_change":"2024-02-12T16:15:40.578370+0000","last_active":"2024-02-12T16:17:15.012306+0000","last_peered":"2024-02-12T16:17:15.012306+0000","last_clean":"2024-02-12T16:17:15.012306+0000","last_became_active":"2024-02-12T16:13:45.793378+0000","last_became_peered":"2024-02-12T16:13:45.793378+0000","last_unstale":"2024-02-12T16:17:15.012306+0000","last_undegraded":"2024-02-12T16:17:15.012306+0000","last_fullsized":"2024-02-12T16:17:15.012306+0000","mapping_epoch":334,"log_start":"353'7600","ondisk_log_start":"353'7600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'11528","last_scrub_stamp":"2024-02-12T16:15:40.578269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:40.578269+0000","log_size":10054,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":251265024,"num_objects":3834,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17654,"num_write_kb":687616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":455,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-02-12T16:17:16.448937+0000","last_change":"2024-02-12T16:15:41.581586+0000","last_active":"2024-02-12T16:17:16.448937+0000","last_peered":"2024-02-12T16:17:16.448937+0000","last_clean":"2024-02-12T16:17:16.448937+0000","last_became_active":"2024-02-12T16:15:24.799547+0000","last_became_peered":"2024-02-12T16:15:24.799547+0000","last_unstale":"2024-02-12T16:17:16.448937+0000","last_undegraded":"2024-02-12T16:17:16.448937+0000","last_fullsized":"2024-02-12T16:17:16.448937+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:41.581483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:41.581483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"405'17982","reported_seq":27507,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-02-12T16:17:18.716154+0000","last_change":"2024-02-12T16:17:17.478472+0000","last_active":"2024-02-12T16:17:18.716154+0000","last_peered":"2024-02-12T16:17:18.716154+0000","last_clean":"2024-02-12T16:17:18.716154+0000","last_became_active":"2024-02-12T16:17:17.433883+0000","last_became_peered":"2024-02-12T16:17:17.433883+0000","last_unstale":"2024-02-12T16:17:18.716154+0000","last_undegraded":"2024-02-12T16:17:18.716154+0000","last_fullsized":"2024-02-12T16:17:18.716154+0000","mapping_epoch":403,"log_start":"353'7900","ondisk_log_start":"353'7900","created":334,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'8622","last_scrub_stamp":"2024-02-12T16:14:58.901608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:14:58.901608+0000","log_size":10082,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":248119296,"num_objects":3786,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17982,"num_write_kb":696576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4710,"num_bytes_recovered":308609024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":28,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-12T16:17:08.807091+0000","last_change":"2024-02-12T16:16:12.802443+0000","last_active":"2024-02-12T16:17:08.807091+0000","last_peered":"2024-02-12T16:17:08.807091+0000","last_clean":"2024-02-12T16:17:08.807091+0000","last_became_active":"2024-02-12T16:16:12.802102+0000","last_became_peered":"2024-02-12T16:16:12.802102+0000","last_unstale":"2024-02-12T16:17:08.807091+0000","last_undegraded":"2024-02-12T16:17:08.807091+0000","last_fullsized":"2024-02-12T16:17:08.807091+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:51.659214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:51.659214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"403'20540","reported_seq":20639,"reported_epoch":403,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:17:16.589999+0000","last_change":"2024-02-12T16:17:13.661453+0000","last_active":"2024-02-12T16:17:16.589999+0000","last_peered":"2024-02-12T16:17:16.589999+0000","last_clean":"2024-02-12T16:16:41.403174+0000","last_became_active":"2024-02-12T16:16:43.588061+0000","last_became_peered":"2024-02-12T16:16:43.588061+0000","last_unstale":"2024-02-12T16:17:16.589999+0000","last_undegraded":"2024-02-12T16:17:16.589999+0000","last_fullsized":"2024-02-12T16:17:16.589999+0000","mapping_epoch":393,"log_start":"367'10501","ondisk_log_start":"367'10501","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'17904","last_scrub_stamp":"2024-02-12T16:16:33.829061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:16:33.829061+0000","log_size":10039,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":249430016,"num_objects":3806,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3806,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18034,"num_write_kb":698880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,1],"avail_no_missing":["9","1"],"object_location_counts":[{"shards":"1,9","objects":3806}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":50,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-02-12T16:17:17.430520+0000","last_change":"2024-02-12T16:15:35.815326+0000","last_active":"2024-02-12T16:17:17.430520+0000","last_peered":"2024-02-12T16:17:17.430520+0000","last_clean":"2024-02-12T16:17:17.430520+0000","last_became_active":"2024-02-12T16:15:24.789635+0000","last_became_peered":"2024-02-12T16:15:24.789635+0000","last_unstale":"2024-02-12T16:17:17.430520+0000","last_undegraded":"2024-02-12T16:17:17.430520+0000","last_fullsized":"2024-02-12T16:17:17.430520+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:35.815231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:35.815231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"402'17970","reported_seq":18809,"reported_epoch":402,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:17:16.185399+0000","last_change":"2024-02-12T16:16:43.636676+0000","last_active":"2024-02-12T16:17:16.185399+0000","last_peered":"2024-02-12T16:17:16.185399+0000","last_clean":"2024-02-12T16:16:41.394949+0000","last_became_active":"2024-02-12T16:16:43.576069+0000","last_became_peered":"2024-02-12T16:16:43.576069+0000","last_unstale":"2024-02-12T16:17:16.185399+0000","last_undegraded":"2024-02-12T16:17:16.185399+0000","last_fullsized":"2024-02-12T16:17:16.185399+0000","mapping_epoch":393,"log_start":"353'7900","ondisk_log_start":"353'7900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'11977","last_scrub_stamp":"2024-02-12T16:15:42.784522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:42.784522+0000","log_size":10070,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256376832,"num_objects":3912,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3607,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17970,"num_write_kb":700224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":375,"num_bytes_recovered":24576000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":3912}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":50,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-02-12T16:17:17.430776+0000","last_change":"2024-02-12T16:15:34.767071+0000","last_active":"2024-02-12T16:17:17.430776+0000","last_peered":"2024-02-12T16:17:17.430776+0000","last_clean":"2024-02-12T16:17:17.430776+0000","last_became_active":"2024-02-12T16:15:24.789639+0000","last_became_peered":"2024-02-12T16:15:24.789639+0000","last_unstale":"2024-02-12T16:17:17.430776+0000","last_undegraded":"2024-02-12T16:17:17.430776+0000","last_fullsized":"2024-02-12T16:17:17.430776+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:34.766976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:34.766976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"402'17676","reported_seq":17756,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:15.037031+0000","last_change":"2024-02-12T16:15:52.830962+0000","last_active":"2024-02-12T16:17:15.037031+0000","last_peered":"2024-02-12T16:17:15.037031+0000","last_clean":"2024-02-12T16:17:15.037031+0000","last_became_active":"2024-02-12T16:13:45.794673+0000","last_became_peered":"2024-02-12T16:13:45.794673+0000","last_unstale":"2024-02-12T16:17:15.037031+0000","last_undegraded":"2024-02-12T16:17:15.037031+0000","last_fullsized":"2024-02-12T16:17:15.037031+0000","mapping_epoch":334,"log_start":"353'7600","ondisk_log_start":"353'7600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'12454","last_scrub_stamp":"2024-02-12T16:15:52.830853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:52.830853+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":250609664,"num_objects":3824,"num_object_clones":0,"num_object_copies":7648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17676,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":50,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-02-12T16:17:17.510110+0000","last_change":"2024-02-12T16:15:43.084332+0000","last_active":"2024-02-12T16:17:17.510110+0000","last_peered":"2024-02-12T16:17:17.510110+0000","last_clean":"2024-02-12T16:17:17.510110+0000","last_became_active":"2024-02-12T16:15:24.790390+0000","last_became_peered":"2024-02-12T16:15:24.790390+0000","last_unstale":"2024-02-12T16:17:17.510110+0000","last_undegraded":"2024-02-12T16:17:17.510110+0000","last_fullsized":"2024-02-12T16:17:17.510110+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:43.084215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:43.084215+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"403'17563","reported_seq":17652,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-02-12T16:17:16.609033+0000","last_change":"2024-02-12T16:16:47.363925+0000","last_active":"2024-02-12T16:17:16.609033+0000","last_peered":"2024-02-12T16:17:16.609033+0000","last_clean":"2024-02-12T16:17:16.609033+0000","last_became_active":"2024-02-12T16:13:45.969665+0000","last_became_peered":"2024-02-12T16:13:45.969665+0000","last_unstale":"2024-02-12T16:17:16.609033+0000","last_undegraded":"2024-02-12T16:17:16.609033+0000","last_fullsized":"2024-02-12T16:17:16.609033+0000","mapping_epoch":334,"log_start":"353'7500","ondisk_log_start":"353'7500","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'15837","last_scrub_stamp":"2024-02-12T16:16:47.363809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:16:47.363809+0000","log_size":10063,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":246087680,"num_objects":3755,"num_object_clones":0,"num_object_copies":7510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17563,"num_write_kb":682176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":453,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-12T16:17:08.804652+0000","last_change":"2024-02-12T16:17:00.843260+0000","last_active":"2024-02-12T16:17:08.804652+0000","last_peered":"2024-02-12T16:17:08.804652+0000","last_clean":"2024-02-12T16:17:08.804652+0000","last_became_active":"2024-02-12T16:17:00.842960+0000","last_became_peered":"2024-02-12T16:17:00.842960+0000","last_unstale":"2024-02-12T16:17:08.804652+0000","last_undegraded":"2024-02-12T16:17:08.804652+0000","last_fullsized":"2024-02-12T16:17:08.804652+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:16:39.800007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:16:39.800007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.1","version":"402'17622","reported_seq":17702,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:15.056868+0000","last_change":"2024-02-12T16:15:48.730580+0000","last_active":"2024-02-12T16:17:15.056868+0000","last_peered":"2024-02-12T16:17:15.056868+0000","last_clean":"2024-02-12T16:17:15.056868+0000","last_became_active":"2024-02-12T16:13:45.969700+0000","last_became_peered":"2024-02-12T16:13:45.969700+0000","last_unstale":"2024-02-12T16:17:15.056868+0000","last_undegraded":"2024-02-12T16:17:15.056868+0000","last_fullsized":"2024-02-12T16:17:15.056868+0000","mapping_epoch":334,"log_start":"353'7600","ondisk_log_start":"353'7600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'12139","last_scrub_stamp":"2024-02-12T16:15:48.730485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:48.730485+0000","log_size":10022,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":253624320,"num_objects":3870,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17622,"num_write_kb":687744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-02-12T16:17:17.430624+0000","last_change":"2024-02-12T16:15:33.775778+0000","last_active":"2024-02-12T16:17:17.430624+0000","last_peered":"2024-02-12T16:17:17.430624+0000","last_clean":"2024-02-12T16:17:17.430624+0000","last_became_active":"2024-02-12T16:15:24.789535+0000","last_became_peered":"2024-02-12T16:15:24.789535+0000","last_unstale":"2024-02-12T16:17:17.430624+0000","last_undegraded":"2024-02-12T16:17:17.430624+0000","last_fullsized":"2024-02-12T16:17:17.430624+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:33.775683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:33.775683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.0","version":"402'17848","reported_seq":17928,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-12T16:17:14.932770+0000","last_change":"2024-02-12T16:15:36.164097+0000","last_active":"2024-02-12T16:17:14.932770+0000","last_peered":"2024-02-12T16:17:14.932770+0000","last_clean":"2024-02-12T16:17:14.932770+0000","last_became_active":"2024-02-12T16:13:45.788482+0000","last_became_peered":"2024-02-12T16:13:45.788482+0000","last_unstale":"2024-02-12T16:17:14.932770+0000","last_undegraded":"2024-02-12T16:17:14.932770+0000","last_fullsized":"2024-02-12T16:17:14.932770+0000","mapping_epoch":334,"log_start":"353'7800","ondisk_log_start":"353'7800","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'11421","last_scrub_stamp":"2024-02-12T16:15:36.163985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:36.163985+0000","log_size":10048,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260964352,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17848,"num_write_kb":698560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"405'17940","reported_seq":18037,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-02-12T16:17:19.032678+0000","last_change":"2024-02-12T16:16:58.718632+0000","last_active":"2024-02-12T16:17:19.032678+0000","last_peered":"2024-02-12T16:17:19.032678+0000","last_clean":"2024-02-12T16:17:19.032678+0000","last_became_active":"2024-02-12T16:13:45.969639+0000","last_became_peered":"2024-02-12T16:13:45.969639+0000","last_unstale":"2024-02-12T16:17:19.032678+0000","last_undegraded":"2024-02-12T16:17:19.032678+0000","last_fullsized":"2024-02-12T16:17:19.032678+0000","mapping_epoch":334,"log_start":"353'7900","ondisk_log_start":"353'7900","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'11605","last_scrub_stamp":"2024-02-12T16:15:39.442737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:39.442737+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":245104640,"num_objects":3740,"num_object_clones":0,"num_object_copies":7480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17940,"num_write_kb":693760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":466,"reported_epoch":403,"state":"active+clean","last_fresh":"2024-02-12T16:17:16.448984+0000","last_change":"2024-02-12T16:16:59.404094+0000","last_active":"2024-02-12T16:17:16.448984+0000","last_peered":"2024-02-12T16:17:16.448984+0000","last_clean":"2024-02-12T16:17:16.448984+0000","last_became_active":"2024-02-12T16:15:19.436534+0000","last_became_peered":"2024-02-12T16:15:19.436534+0000","last_unstale":"2024-02-12T16:17:16.448984+0000","last_undegraded":"2024-02-12T16:17:16.448984+0000","last_fullsized":"2024-02-12T16:17:16.448984+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:36.644619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:36.644619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4034330648,"num_objects":61560,"num_object_clones":0,"num_object_copies":123120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15411,"num_objects_unfound":0,"num_objects_dirty":61560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285110,"num_write_kb":11093377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9365,"num_bytes_recovered":613679104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8512086016,"data_stored":8512077872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958757,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1507152,"kb_used_data":1205832,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92238000,"statfs":{"total":95995035648,"available":94451712000,"internally_reserved":0,"allocated":1234771968,"data_stored":1234410491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.64400000000000002,"15min":0.73999999999999999},"min":{"1min":0.27700000000000002,"5min":0.26800000000000002,"15min":0.246},"max":{"1min":3.0880000000000001,"5min":5.649,"15min":5.649},"last":4.6829999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.81999999999999995,"15min":0.90900000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.25900000000000001},"max":{"1min":3.5430000000000001,"5min":4.992,"15min":4.992},"last":0.57499999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.0740000000000001,"15min":1.0640000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.25700000000000001},"max":{"1min":7.0110000000000001,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.123,"15min":1.0700000000000001},"min":{"1min":0.39300000000000002,"5min":0.28799999999999998,"15min":0.27300000000000002},"max":{"1min":3.2330000000000001,"5min":5.6920000000000002,"15min":5.6920000000000002},"last":0.628}]},{"osd":2,"last update":"Mon Feb 12 16:08:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0980000000000001,"15min":1.1259999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.7109999999999999,"5min":5.274,"15min":5.274},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":0.96999999999999997,"15min":1.1160000000000001},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":3.7360000000000002,"5min":5.375,"15min":5.375},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":8.3100000000000005,"5min":3.2749999999999999,"15min":2.2090000000000001},"min":{"1min":0.36499999999999999,"5min":0.313,"15min":0.28999999999999998},"max":{"1min":128.93899999999999,"5min":128.93899999999999,"15min":128.93899999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":7.6340000000000003,"5min":3.1909999999999998,"15min":2.2519999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.30299999999999999},"max":{"1min":128.87100000000001,"5min":128.87100000000001,"15min":128.87100000000001},"last":0.91800000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":4.7030000000000003,"15min":2.282},"min":{"1min":0.42599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.5499999999999998,"5min":368.48399999999998,"15min":368.48399999999998},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.94399999999999995,"5min":4.556,"15min":2.1930000000000001},"min":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.5499999999999998,"5min":371.65100000000001,"15min":371.65100000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":4.7519999999999998,"15min":3.722},"min":{"1min":0.41099999999999998,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":3.1840000000000002,"5min":368.51400000000001,"15min":407.07600000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.0569999999999999,"5min":4.7220000000000004,"15min":3.7789999999999999},"min":{"1min":0.42899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.8220000000000001,"5min":368.70499999999998,"15min":407.26299999999998},"last":0.88900000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.84499999999999997,"15min":0.97599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.34799999999999998},"max":{"1min":1.419,"5min":4.8330000000000002,"15min":4.8330000000000002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.94999999999999996,"15min":1.0820000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":1.4890000000000001,"5min":3.3889999999999998,"15min":3.8039999999999998},"last":0.50600000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.0720000000000001,"15min":1.1259999999999999},"min":{"1min":0.46100000000000002,"5min":0.437,"15min":0.39100000000000001},"max":{"1min":3.1320000000000001,"5min":5.766,"15min":5.766},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.159,"15min":1.175},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.438},"max":{"1min":4.9939999999999998,"5min":5.1189999999999998,"15min":5.2510000000000003},"last":0.79600000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.873,"15min":1.0640000000000001},"min":{"1min":0.443,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3120000000000001,"5min":3.5910000000000002,"15min":4.5940000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.095,"5min":1.1180000000000001,"15min":1.1279999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.375},"max":{"1min":3.069,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":0.46500000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:17:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2150000000000001,"15min":1.0309999999999999},"min":{"1min":0.53000000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.2890000000000001,"5min":10.743,"15min":10.743},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.3700000000000001,"15min":1.1200000000000001},"min":{"1min":0.59999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.5060000000000002,"5min":10.698,"15min":10.698},"last":0.55900000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.1120000000000001,"15min":1.1479999999999999},"min":{"1min":0.50600000000000001,"5min":0.433,"15min":0.377},"max":{"1min":2.5030000000000001,"5min":6.7160000000000002,"15min":6.7160000000000002},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0109999999999999,"15min":1.1679999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.45200000000000001},"max":{"1min":4.6210000000000004,"5min":4.6210000000000004,"15min":4.8630000000000004},"last":0.76500000000000001}]}]},{"osd":10,"up_from":83,"seq":356482285807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1030260,"kb_used_data":724844,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92714892,"statfs":{"total":95995035648,"available":94940049408,"internally_reserved":0,"allocated":742240256,"data_stored":741860963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.8999999999999999,"5min":0.94399999999999995,"15min":1.095},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":19.866,"5min":19.866,"15min":49.802999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":2.4279999999999999,"5min":1.1439999999999999,"15min":1.181},"min":{"1min":0.317,"5min":0.317,"15min":0.28399999999999997},"max":{"1min":20.277999999999999,"5min":20.277999999999999,"15min":50.863},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.69999999999999996,"15min":0.78900000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":3.8439999999999999,"5min":5.3040000000000003,"15min":5.3040000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.65200000000000002,"15min":0.83799999999999997},"min":{"1min":0.32000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.73,"5min":3.73,"15min":7.9829999999999997},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":7.8730000000000002,"15min":3.7320000000000002},"min":{"1min":0.42699999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.6130000000000004,"5min":628.77099999999996,"15min":628.77099999999996},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":7.734,"15min":3.665},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":3.6899999999999999,"5min":624.19299999999998,"15min":624.19299999999998},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.84799999999999998,"15min":3.4860000000000002},"min":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.33700000000000002},"max":{"1min":1.982,"5min":4.8200000000000003,"15min":602.05399999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.91000000000000003,"15min":3.5859999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.33200000000000002},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":598.875},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":7.8220000000000001,"15min":3.266},"min":{"1min":0.44900000000000001,"5min":0.378,"15min":0.35399999999999998},"max":{"1min":3.2120000000000002,"5min":626.26599999999996,"15min":626.26599999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":7.9119999999999999,"15min":3.3650000000000002},"min":{"1min":0.41099999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.8279999999999998,"5min":625.90700000000004,"15min":625.90700000000004},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.96799999999999997,"15min":3.1709999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.376},"max":{"1min":3.758,"5min":3.9950000000000001,"15min":597.48299999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96599999999999997,"15min":3.1669999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.41799999999999998},"max":{"1min":3.2469999999999999,"5min":4.0949999999999998,"15min":600.23099999999999},"last":0.60499999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.92300000000000004,"15min":1.0820000000000001},"min":{"1min":0.45700000000000002,"5min":0.434,"15min":0.33400000000000002},"max":{"1min":3.895,"5min":3.895,"15min":7.2720000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.91500000000000004,"15min":1.103},"min":{"1min":0.53300000000000003,"5min":0.42299999999999999,"15min":0.38200000000000001},"max":{"1min":3.9009999999999998,"5min":3.9009999999999998,"15min":7.3810000000000002},"last":0.50900000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.1060000000000001,"15min":1.1659999999999999},"min":{"1min":0.60699999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":4.7009999999999996,"5min":4.7009999999999996,"15min":4.7089999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.135,"15min":1.179},"min":{"1min":0.71799999999999997,"5min":0.49299999999999999,"15min":0.40000000000000002},"max":{"1min":4.766,"5min":4.766,"15min":4.7789999999999999},"last":0.61699999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:17:05 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.333,"15min":1.4099999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":5.056,"5min":10.218999999999999,"15min":30.917999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.357,"5min":1.2010000000000001,"15min":1.355},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.48499999999999999},"max":{"1min":4.9219999999999997,"5min":4.9219999999999997,"15min":30.742000000000001},"last":0.53500000000000003}]}]},{"osd":9,"up_from":74,"seq":317827580151,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1027168,"kb_used_data":729944,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92717984,"statfs":{"total":95995035648,"available":94943215616,"internally_reserved":0,"allocated":747462656,"data_stored":747091200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56899999999999995,"15min":0.76200000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":1.2130000000000001,"5min":3.702,"15min":10.788},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.94199999999999995,"15min":0.93500000000000005},"min":{"1min":0.34499999999999997,"5min":0.30499999999999999,"15min":0.27400000000000002},"max":{"1min":7.4429999999999996,"5min":7.4429999999999996,"15min":7.9630000000000001},"last":3.4300000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.83999999999999997,"15min":1.526},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.843,"5min":5.843,"15min":175.33500000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.181,"15min":1.6200000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.8600000000000003,"5min":7.5019999999999998,"15min":175.24299999999999},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.997,"15min":1.0329999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":2.4420000000000002,"5min":7.2999999999999998,"15min":7.8140000000000001},"last":0.433},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.68799999999999994,"15min":0.96199999999999997},"min":{"1min":0.34200000000000003,"5min":0.317,"15min":0.28199999999999997},"max":{"1min":0.90700000000000003,"5min":5.7469999999999999,"15min":5.7469999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.2310000000000001,"15min":1.1599999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.5750000000000002,"5min":34.344000000000001,"15min":34.591000000000001},"last":10.206},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.577,"15min":1.3680000000000001},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":2.597,"5min":34.389000000000003,"15min":34.494},"last":4.3259999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.80600000000000005,"15min":1.032},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":25.552},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92800000000000005,"15min":1},"min":{"1min":0.44800000000000001,"5min":0.435,"15min":0.35099999999999998},"max":{"1min":1.0169999999999999,"5min":5.8010000000000002,"15min":25.847999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.81399999999999995,"15min":3.2650000000000001},"min":{"1min":0.42199999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98399999999999999,"5min":5.899,"15min":617.95899999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.075,"15min":3.4060000000000001},"min":{"1min":0.53700000000000003,"5min":0.40999999999999998,"15min":0.375},"max":{"1min":2.4940000000000002,"5min":7.3739999999999997,"15min":617.88099999999997},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.073,"15min":1.103},"min":{"1min":0.49099999999999999,"5min":0.441,"15min":0.40999999999999998},"max":{"1min":2.395,"5min":7.3360000000000003,"15min":7.3360000000000003},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88600000000000001,"15min":0.97099999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.377},"max":{"1min":3.6240000000000001,"5min":3.6240000000000001,"15min":4.7910000000000004},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.014,"15min":4.5970000000000004},"min":{"1min":0.46600000000000003,"5min":0.442,"15min":0.379},"max":{"1min":1.085,"5min":5.8470000000000004,"15min":613.28200000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.101,"15min":4.6539999999999999},"min":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"max":{"1min":2.5310000000000001,"5min":7.4089999999999998,"15min":613.25199999999995},"last":0.56899999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.90700000000000003,"15min":1.095},"min":{"1min":0.43099999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":3.722,"5min":3.722,"15min":4.96},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":1.0900000000000001,"15min":1.0760000000000001},"min":{"1min":0.59899999999999998,"5min":0.46800000000000003,"15min":0.41199999999999998},"max":{"1min":2.5600000000000001,"5min":7.25,"15min":7.25},"last":0.627}]},{"osd":10,"last update":"Mon Feb 12 16:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.0309999999999999,"15min":1.0720000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.38400000000000001},"max":{"1min":3.6920000000000002,"5min":4.0709999999999997,"15min":5.5},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.0149999999999999,"15min":1.1359999999999999},"min":{"1min":0.503,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.8700000000000001,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":0.68600000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:16:25 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.747},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":5.2750000000000004,"5min":5.2750000000000004,"15min":5.2750000000000004},"last":0.66600000000000004}]}]},{"osd":8,"up_from":0,"seq":279172874495,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":56,"seq":240518168839,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1033512,"kb_used_data":724000,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92711640,"statfs":{"total":95995035648,"available":94936719360,"internally_reserved":0,"allocated":741376000,"data_stored":741002680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:16:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.9220000000000002,"5min":3.9220000000000002,"15min":3.9220000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.8570000000000002,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:15:16 2024","interfaces":[{"interface":"back","average":{"1min":2.9889999999999999,"5min":1.3109999999999999,"15min":1.2390000000000001},"min":{"1min":0.29699999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":48.941000000000003,"5min":48.941000000000003,"15min":60.152000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.5760000000000001,"5min":1.3120000000000001,"15min":1.343},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":48.338000000000001,"5min":48.338000000000001,"15min":62.020000000000003},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70799999999999996,"15min":1.155},"min":{"1min":0.32100000000000001,"5min":0.318,"15min":0.29799999999999999},"max":{"1min":4.5899999999999999,"5min":4.5899999999999999,"15min":60.204999999999998},"last":1.8120000000000001},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.878,"15min":1.1699999999999999},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.28799999999999998},"max":{"1min":3.9540000000000002,"5min":9.1240000000000006,"15min":59.365000000000002},"last":0.499}]},{"osd":3,"last update":"Mon Feb 12 16:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.86199999999999999,"15min":1.0609999999999999},"min":{"1min":0.33200000000000002,"5min":0.28599999999999998,"15min":0.27900000000000003},"max":{"1min":4.6369999999999996,"5min":4.6369999999999996,"15min":6.2439999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.94199999999999995,"15min":1.0640000000000001},"min":{"1min":0.434,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":4.75,"5min":4.8949999999999996,"15min":8.2940000000000005},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.85899999999999999,"15min":0.88},"min":{"1min":0.34699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.7919999999999998,"5min":4.6299999999999999,"15min":6.2519999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.0349999999999999,"15min":0.97999999999999998},"min":{"1min":0.36799999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.5030000000000001,"5min":6.0090000000000003,"15min":6.0919999999999996},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.85399999999999998,"15min":0.95099999999999996},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.28999999999999998},"max":{"1min":2.883,"5min":4.5119999999999996,"15min":6.0739999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.079,"15min":1.0209999999999999},"min":{"1min":0.40500000000000003,"5min":0.35999999999999999,"15min":0.33600000000000002},"max":{"1min":3.9180000000000001,"5min":9.3640000000000008,"15min":9.3640000000000008},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:16:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.0009999999999999,"15min":1.083},"min":{"1min":0.40300000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.4510000000000001,"5min":6.0830000000000002,"15min":6.0830000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.0880000000000001,"15min":1.075},"min":{"1min":0.40799999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.5449999999999999,"5min":6.0430000000000001,"15min":6.1349999999999998},"last":0.58999999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.0209999999999999,"15min":1.1499999999999999},"min":{"1min":0.44800000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":5.6130000000000004,"5min":5.6130000000000004,"15min":6.0250000000000004},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.105,"15min":1.1890000000000001},"min":{"1min":0.45200000000000001,"5min":0.433,"15min":0.42999999999999999},"max":{"1min":5.5860000000000003,"5min":9.2140000000000004,"15min":9.2140000000000004},"last":0.70499999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:16:41 2024","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":1.2490000000000001,"15min":2.3109999999999999},"min":{"1min":0.54900000000000004,"5min":0.46100000000000002,"15min":0.441},"max":{"1min":12.907999999999999,"5min":12.907999999999999,"15min":278.54599999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.137,"15min":2.2149999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.38200000000000001},"max":{"1min":4.6719999999999997,"5min":4.6719999999999997,"15min":281.09199999999998},"last":4.109}]},{"osd":10,"last update":"Mon Feb 12 16:16:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.115,"15min":1.1919999999999999},"min":{"1min":0.53900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.7110000000000003,"5min":4.9349999999999996,"15min":5.9489999999999998},"last":4.0279999999999996},{"interface":"front","average":{"1min":1.157,"5min":1.1259999999999999,"15min":1.2390000000000001},"min":{"1min":0.54500000000000004,"5min":0.52800000000000002,"15min":0.42099999999999999},"max":{"1min":4.0069999999999997,"5min":9.1690000000000005,"15min":9.1690000000000005},"last":0.85199999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2270000000000001,"15min":2.3540000000000001},"min":{"1min":0.60299999999999998,"5min":0.499,"15min":0.498},"max":{"1min":5.0220000000000002,"5min":9.2560000000000002,"15min":280.83999999999997},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.2330000000000001,"15min":2.3690000000000002},"min":{"1min":0.52900000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":6.2930000000000001,"5min":6.2930000000000001,"15min":278.63799999999998},"last":0.71999999999999997}]}]},{"osd":6,"up_from":48,"seq":206158430480,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802804,"kb_used_data":493292,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92942348,"statfs":{"total":95995035648,"available":95172964352,"internally_reserved":0,"allocated":505131008,"data_stored":504751723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:14:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.72599999999999998,"15min":1.052},"min":{"1min":0.30299999999999999,"5min":0.28999999999999998,"15min":0.26200000000000001},"max":{"1min":3.173,"5min":4.4199999999999999,"15min":67.093000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":1.1000000000000001},"min":{"1min":0.318,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.554,"5min":4.4779999999999998,"15min":68.549000000000007},"last":0.34799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.89500000000000002,"15min":1.1659999999999999},"min":{"1min":0.33700000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.5999999999999996,"5min":5.5999999999999996,"15min":66.936000000000007},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.89400000000000002,"15min":1.161},"min":{"1min":0.318,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":12.436,"5min":12.436,"15min":68.683000000000007},"last":0.68000000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.74199999999999999,"15min":1.3480000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":3.3279999999999998,"5min":3.722,"15min":68.462000000000003},"last":0.37},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.77800000000000002,"15min":1.377},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.875,"5min":5.0629999999999997,"15min":67.322999999999993},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":0.621},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":3.694,"15min":2.0510000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.6120000000000001,"5min":284.05599999999998,"15min":284.05599999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":3.8540000000000001,"15min":1.901},"min":{"1min":0.40500000000000003,"5min":0.38500000000000001,"15min":0.27000000000000002},"max":{"1min":2.0230000000000001,"5min":284.041,"15min":284.041},"last":0.50700000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":3.6600000000000001,"15min":2.0990000000000002},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.33500000000000002},"max":{"1min":3.3650000000000002,"5min":281.49700000000001,"15min":281.49700000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.88800000000000001,"5min":3.806,"15min":2.1000000000000001},"min":{"1min":0.39300000000000002,"5min":0.371,"15min":0.29399999999999998},"max":{"1min":4.3579999999999997,"5min":281.61900000000003,"15min":281.61900000000003},"last":0.34699999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.92400000000000004,"15min":1.6120000000000001},"min":{"1min":0.40799999999999997,"5min":0.38600000000000001,"15min":0.29399999999999998},"max":{"1min":1.607,"5min":5.0099999999999998,"15min":67.144999999999996},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.077,"15min":1.657},"min":{"1min":0.49199999999999999,"5min":0.42499999999999999,"15min":0.38200000000000001},"max":{"1min":5.0190000000000001,"5min":6.6449999999999996,"15min":67.230999999999995},"last":0.74199999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:16:57 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.90700000000000003,"15min":1.339},"min":{"1min":0.497,"5min":0.497,"15min":0.39800000000000002},"max":{"1min":3.8959999999999999,"5min":3.8959999999999999,"15min":68.653000000000006},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.026,"15min":1.3109999999999999},"min":{"1min":0.48599999999999999,"5min":0.44,"15min":0.40400000000000003},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":68.620999999999995},"last":0.72599999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:16:33 2024","interfaces":[{"interface":"back","average":{"1min":1.881,"5min":1.262,"15min":1.456},"min":{"1min":0.54400000000000004,"5min":0.44600000000000001,"15min":0.40600000000000003},"max":{"1min":18.059999999999999,"5min":18.059999999999999,"15min":67.188000000000002},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.2370000000000001,"15min":1.4359999999999999},"min":{"1min":0.623,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":17.425000000000001,"5min":17.425000000000001,"15min":68.506},"last":0.79300000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.99199999999999999,"15min":1.4059999999999999},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.49099999999999999},"max":{"1min":3.4129999999999998,"5min":3.4129999999999998,"15min":68.722999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.077,"5min":1.0489999999999999,"15min":1.381},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.505},"max":{"1min":3.4630000000000001,"5min":3.4630000000000001,"15min":68.573999999999998},"last":0.65800000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":7.0270000000000001,"5min":7.0270000000000001,"15min":7.0270000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":139.727,"5min":139.727,"15min":139.727},"last":2.081},{"interface":"front","average":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":6.6769999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":139.24199999999999,"5min":139.24199999999999,"15min":139.24199999999999},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2299356,"kb_used_data":1705556,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":91445796,"statfs":{"total":95995035648,"available":93640495104,"internally_reserved":0,"allocated":1746489344,"data_stored":1746121960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:16:31 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":2.218,"15min":1.5369999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":3.399,"5min":114.40000000000001,"15min":114.40000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.073,"5min":2.4449999999999998,"15min":1.6839999999999999},"min":{"1min":0.23899999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.8499999999999996,"5min":115.09099999999999,"15min":115.09099999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.7689999999999999,"5min":1.038,"15min":1.248},"min":{"1min":0.312,"5min":0.248,"15min":0.248},"max":{"1min":9.8599999999999994,"5min":9.8599999999999994,"15min":40.542999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.206,"5min":0.998,"15min":1.379},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.24299999999999999},"max":{"1min":4.9580000000000002,"5min":6.7999999999999998,"15min":40.811999999999998},"last":0.54400000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:16:51 2024","interfaces":[{"interface":"back","average":{"1min":6.9550000000000001,"5min":2.2480000000000002,"15min":1.3380000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":142.78100000000001,"5min":142.78100000000001,"15min":142.78100000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":7.0170000000000003,"5min":2.3719999999999999,"15min":1.5269999999999999},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":142.32499999999999,"5min":142.32499999999999,"15min":142.32499999999999},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:13:23 2024","interfaces":[{"interface":"back","average":{"1min":2.7210000000000001,"5min":1.2929999999999999,"15min":1.298},"min":{"1min":0.40799999999999997,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":33.125,"5min":33.125,"15min":33.125},"last":1.1539999999999999},{"interface":"front","average":{"1min":2.669,"5min":1.373,"15min":1.339},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":33.060000000000002,"5min":33.060000000000002,"15min":33.060000000000002},"last":0.95499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.1259999999999999,"15min":1.22},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.313},"max":{"1min":4.5049999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.157,"15min":1.236},"min":{"1min":0.374,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":7.0110000000000001,"5min":7.0110000000000001,"15min":10.420999999999999},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:16:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":2.456,"15min":1.675},"min":{"1min":0.46400000000000002,"5min":0.37,"15min":0.31900000000000001},"max":{"1min":6.7869999999999999,"5min":56.137,"15min":56.137},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.5269999999999999,"5min":2.536,"15min":1.73},"min":{"1min":0.49099999999999999,"5min":0.47599999999999998,"15min":0.33700000000000002},"max":{"1min":6.6289999999999996,"5min":56.195999999999998,"15min":56.195999999999998},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:16:31 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":2.379,"15min":1.7010000000000001},"min":{"1min":0.43099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.609,"5min":55.869999999999997,"15min":55.869999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.5069999999999999,"5min":2.496,"15min":1.679},"min":{"1min":0.51600000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.7140000000000004,"5min":55.892000000000003,"15min":55.892000000000003},"last":0.55800000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.26,"15min":1.371},"min":{"1min":0.51500000000000001,"5min":0.502,"15min":0.39400000000000002},"max":{"1min":6.9160000000000004,"5min":6.9160000000000004,"15min":9.9049999999999994},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.3340000000000001,"15min":1.486},"min":{"1min":0.60399999999999998,"5min":0.44400000000000001,"15min":0.41999999999999998},"max":{"1min":3.9990000000000001,"5min":10.119999999999999,"15min":10.313000000000001},"last":0.70799999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:16:35 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3049999999999999,"15min":1.4419999999999999},"min":{"1min":0.52500000000000002,"5min":0.45400000000000001,"15min":0.40699999999999997},"max":{"1min":4.5590000000000002,"5min":10.07,"15min":10.352},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.5580000000000001,"15min":1.3999999999999999},"min":{"1min":0.54600000000000004,"5min":0.45000000000000001,"15min":0.40200000000000002},"max":{"1min":6.7510000000000003,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.64100000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:16:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2829999999999999,"15min":1.3799999999999999},"min":{"1min":0.54500000000000004,"5min":0.52800000000000002,"15min":0.48499999999999999},"max":{"1min":4.6219999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.514,"5min":1.377,"15min":1.496},"min":{"1min":0.59499999999999997,"5min":0.53700000000000003,"15min":0.51800000000000002},"max":{"1min":6.8200000000000003,"5min":6.8200000000000003,"15min":10.387},"last":0.66600000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.5169999999999999,"15min":1.4399999999999999},"min":{"1min":0.58399999999999996,"5min":0.443,"15min":0.443},"max":{"1min":7.1600000000000001,"5min":7.1600000000000001,"15min":13.555},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.768,"5min":1.552,"15min":1.4350000000000001},"min":{"1min":0.59599999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":7.2489999999999997,"5min":7.2489999999999997,"15min":13.846},"last":1.583}]}]},{"osd":0,"up_from":8,"seq":34359738688,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785104,"kb_used_data":483784,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92960048,"statfs":{"total":95995035648,"available":95191089152,"internally_reserved":0,"allocated":495394816,"data_stored":495033347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.038,"15min":1.49},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":4.4359999999999999,"5min":8.1289999999999996,"15min":114.01600000000001},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.27,"5min":1.1399999999999999,"15min":1.5249999999999999},"min":{"1min":0.27000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.4939999999999998,"5min":5.4939999999999998,"15min":114.342},"last":6.1230000000000002}]},{"osd":2,"last update":"Mon Feb 12 15:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.76600000000000001,"15min":0.65000000000000002},"min":{"1min":0.68300000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.67100000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.80900000000000005,"15min":1.2},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.215},"max":{"1min":1.48,"5min":5.4269999999999996,"15min":53.793999999999997},"last":1.899},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.80700000000000005,"15min":1.137},"min":{"1min":0.29199999999999998,"5min":0.252,"15min":0.252},"max":{"1min":4.2199999999999998,"5min":6.7089999999999996,"15min":53.853000000000002},"last":0.80300000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":2.9279999999999999,"15min":1.5860000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29799999999999999},"max":{"1min":4.4850000000000003,"5min":116.187,"15min":116.187},"last":5.9969999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":2.984,"15min":1.6200000000000001},"min":{"1min":0.32800000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.516,"5min":116.163,"15min":116.163},"last":4.7190000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.86399999999999999,"15min":1.407},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":4.0250000000000004,"5min":4.2409999999999997,"15min":116.005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.3899999999999999},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":115.973},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":1.6859999999999999,"15min":1.216},"min":{"1min":0.36199999999999999,"5min":0.33600000000000002,"15min":0.33000000000000002},"max":{"1min":4.0540000000000003,"5min":81.147000000000006,"15min":81.147000000000006},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.7330000000000001,"15min":1.2589999999999999},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":4.1470000000000002,"5min":80.233000000000004,"15min":80.233000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":2.3370000000000002,"15min":1.522},"min":{"1min":0.42699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":4.1139999999999999,"5min":115.26000000000001,"15min":115.26000000000001},"last":0.503},{"interface":"front","average":{"1min":1.0049999999999999,"5min":2.4279999999999999,"15min":1.635},"min":{"1min":0.55200000000000005,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":4.1639999999999997,"5min":115.402,"15min":115.402},"last":0.70699999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.0469999999999999,"15min":1.165},"min":{"1min":0.503,"5min":0.42899999999999999,"15min":0.378},"max":{"1min":4.1310000000000002,"5min":4.1310000000000002,"15min":5.6059999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.119,"15min":1.208},"min":{"1min":0.53800000000000003,"5min":0.51300000000000001,"15min":0.41299999999999998},"max":{"1min":4.1870000000000003,"5min":4.1870000000000003,"15min":7.5599999999999996},"last":0.77100000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.157,"15min":1.2729999999999999},"min":{"1min":0.50800000000000001,"5min":0.42699999999999999,"15min":0.40600000000000003},"max":{"1min":3.1299999999999999,"5min":5.4329999999999998,"15min":5.7969999999999997},"last":4.6689999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0309999999999999,"15min":1.1930000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.39500000000000002},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":6.6070000000000002},"last":0.67600000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:16:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1950000000000001,"15min":1.2490000000000001},"min":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":4.5460000000000003,"5min":5.7869999999999999,"15min":5.8259999999999996},"last":6.0979999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.218,"15min":1.238},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.437},"max":{"1min":4.6070000000000002,"5min":6.2480000000000002,"15min":6.2480000000000002},"last":0.64000000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":1.468,"15min":1.367},"min":{"1min":0.435,"5min":0.434,"15min":0.434},"max":{"1min":6.032,"5min":6.032,"15min":7.9660000000000002},"last":6.0629999999999997},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.208,"15min":1.3480000000000001},"min":{"1min":0.56000000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.8780000000000001,"5min":6.0099999999999998,"15min":8.4800000000000004},"last":0.60899999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313519,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564304,"kb_used_data":254792,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93180848,"statfs":{"total":95995035648,"available":95417188352,"internally_reserved":0,"allocated":260907008,"data_stored":260545539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.82499999999999996,"15min":1.8260000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.222},"max":{"1min":6.1859999999999999,"5min":6.1859999999999999,"15min":177.68899999999999},"last":3.5219999999999998},{"interface":"front","average":{"1min":1.161,"5min":0.78900000000000003,"15min":1.8740000000000001},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.216},"max":{"1min":6.3099999999999996,"5min":6.3099999999999996,"15min":178.26900000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.0109999999999999,"15min":1.401},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.223},"max":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":70.944000000000003},"last":4.4729999999999999},{"interface":"front","average":{"1min":1.601,"5min":1.0189999999999999,"15min":1.427},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.22600000000000001},"max":{"1min":6.806,"5min":6.806,"15min":71.203999999999994},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":2.4969999999999999,"15min":1.7010000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":2.8519999999999999,"5min":80.436999999999998,"15min":80.436999999999998},"last":3.6059999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":2.6110000000000002,"15min":1.5900000000000001},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":3.7919999999999998,"5min":77.286000000000001,"15min":77.286000000000001},"last":0.315}]},{"osd":4,"last update":"Mon Feb 12 16:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.871,"15min":1.0640000000000001},"min":{"1min":0.37,"5min":0.36099999999999999,"15min":0.311},"max":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":9.657},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.76200000000000001,"15min":0.97499999999999998},"min":{"1min":0.35099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":3.9289999999999998,"5min":3.9289999999999998,"15min":9.5530000000000008},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.90600000000000003,"15min":3.1579999999999999},"min":{"1min":0.35299999999999998,"5min":0.315,"15min":0.315},"max":{"1min":5.048,"5min":5.048,"15min":616.34199999999998},"last":3.6480000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.90500000000000003,"15min":3.2770000000000001},"min":{"1min":0.38500000000000001,"5min":0.373,"15min":0.33900000000000002},"max":{"1min":4.3070000000000004,"5min":4.6820000000000004,"15min":615.18899999999996},"last":0.39000000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.81699999999999995,"15min":1.0069999999999999},"min":{"1min":0.40000000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76000000000000001,"5min":4.907,"15min":9.6150000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80600000000000005,"15min":1.0720000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.35099999999999998},"max":{"1min":4.0270000000000001,"5min":4.8739999999999997,"15min":6.3979999999999997},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.95499999999999996,"15min":3.375},"min":{"1min":0.378,"5min":0.378,"15min":0.377},"max":{"1min":1.107,"5min":5.2960000000000003,"15min":611.79600000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.89000000000000001,"15min":3.3849999999999998},"min":{"1min":0.439,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85699999999999998,"5min":4.0369999999999999,"15min":611.83900000000006},"last":3.6949999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.80200000000000005,"15min":1.1499999999999999},"min":{"1min":0.5,"5min":0.38700000000000001,"15min":0.38200000000000001},"max":{"1min":1.784,"5min":2.1419999999999999,"15min":6.4960000000000004},"last":3.73},{"interface":"front","average":{"1min":0.752,"5min":0.88500000000000001,"15min":1.2689999999999999},"min":{"1min":0.49399999999999999,"5min":0.432,"15min":0.432},"max":{"1min":2.3290000000000002,"5min":3.9849999999999999,"15min":8.5920000000000005},"last":0.55300000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.95299999999999996,"15min":1.1699999999999999},"min":{"1min":0.48799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.5110000000000001,"5min":4.4809999999999999,"15min":9.6460000000000008},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.93600000000000005,"15min":1.22},"min":{"1min":0.52600000000000002,"5min":0.47499999999999998,"15min":0.42999999999999999},"max":{"1min":3.9609999999999999,"5min":3.9609999999999999,"15min":6.4699999999999998},"last":0.77100000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.0249999999999999,"15min":1.1830000000000001},"min":{"1min":0.504,"5min":0.5,"15min":0.442},"max":{"1min":5.1840000000000002,"5min":5.1840000000000002,"15min":9.6760000000000002},"last":3.3100000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.0489999999999999,"15min":1.2370000000000001},"min":{"1min":0.497,"5min":0.46500000000000002,"15min":0.45400000000000001},"max":{"1min":4.3739999999999997,"5min":7.2699999999999996,"15min":7.2699999999999996},"last":0.627}]}]},{"osd":3,"up_from":26,"seq":111669149990,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1077328,"kb_used_data":780104,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92667824,"statfs":{"total":95995035648,"available":94891851776,"internally_reserved":0,"allocated":798826496,"data_stored":798465027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:16:08 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.86499999999999999,"15min":0.94999999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":2.7429999999999999,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.92900000000000005,"15min":0.879},"min":{"1min":0.27300000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.4569999999999999,"5min":6.6680000000000001,"15min":6.6680000000000001},"last":0.755}]},{"osd":1,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.7220000000000004,"5min":6.7220000000000004,"15min":6.7220000000000004},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"last":1.6890000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.94099999999999995,"15min":1.2370000000000001},"min":{"1min":0.26700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.927,"5min":5.851,"15min":68.405000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":1.0289999999999999,"15min":1.252},"min":{"1min":0.28999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8999999999999999,"5min":5.8979999999999997,"15min":69.009},"last":1.425}]},{"osd":4,"last update":"Mon Feb 12 16:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.81499999999999995,"15min":0.879},"min":{"1min":0.29599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.089,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.88100000000000001,"15min":0.90400000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76200000000000001,"5min":5.9870000000000001,"15min":5.9870000000000001},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.4470000000000001,"15min":1.1459999999999999},"min":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.313},"max":{"1min":4.0670000000000002,"5min":22.751000000000001,"15min":22.751000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.198,"5min":1.2,"15min":1.0720000000000001},"min":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32200000000000001},"max":{"1min":4.8040000000000003,"5min":22.527999999999999,"15min":22.527999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.93200000000000005,"15min":0.96599999999999997},"min":{"1min":0.39000000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.9540000000000002,"5min":6.0289999999999999,"15min":6.0289999999999999},"last":1.45},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.1299999999999999,"15min":1.1439999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.992,"5min":6.1100000000000003,"15min":9.5990000000000002},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.321,"15min":1.472},"min":{"1min":0.53100000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.02,"5min":10.061,"15min":91.718999999999994},"last":0.629},{"interface":"front","average":{"1min":0.69399999999999995,"5min":1.2549999999999999,"15min":1.538},"min":{"1min":0.36899999999999999,"5min":0.35199999999999998,"15min":0.34799999999999998},"max":{"1min":1.7749999999999999,"5min":10.944000000000001,"15min":90.367999999999995},"last":1.5529999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.1319999999999999,"15min":1.1990000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.317},"max":{"1min":5.1500000000000004,"5min":6.0970000000000004,"15min":6.5899999999999999},"last":1.474},{"interface":"front","average":{"1min":1.143,"5min":1.097,"15min":1.1559999999999999},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.7039999999999997,"5min":6.2430000000000003,"15min":8.516},"last":1.4950000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:16:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.0349999999999999,"15min":1.1499999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.6509999999999998,"5min":6.2699999999999996,"15min":6.2699999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.127,"15min":1.268},"min":{"1min":0.55000000000000004,"5min":0.42099999999999999,"15min":0.38200000000000001},"max":{"1min":1.345,"5min":6.1660000000000004,"15min":9.6509999999999998},"last":1.534}]},{"osd":10,"last update":"Mon Feb 12 16:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.044,"15min":1.212},"min":{"1min":0.42899999999999999,"5min":0.42699999999999999,"15min":0.41099999999999998},"max":{"1min":4.8399999999999999,"5min":6.4199999999999999,"15min":6.4199999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.101,"15min":1.22},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":4.9630000000000001,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.71899999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.5680000000000001,"15min":1.5860000000000001},"min":{"1min":0.505,"5min":0.47699999999999998,"15min":0.39100000000000001},"max":{"1min":6.0110000000000001,"5min":26.190000000000001,"15min":49.167000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.607,"15min":1.5960000000000001},"min":{"1min":0.59299999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":7.4059999999999997,"5min":26.202999999999999,"15min":48.548999999999999},"last":1.5149999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066004,"kb_used_data":756428,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92679148,"statfs":{"total":95995035648,"available":94903447552,"internally_reserved":0,"allocated":774582272,"data_stored":774216731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.70899999999999996,"15min":1.391},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":1.254,"5min":4.5339999999999998,"15min":96.052000000000007},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.83499999999999996,"15min":1.508},"min":{"1min":0.30299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.75900000000000001,"5min":4.5629999999999997,"15min":96.307000000000002},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":2.004,"15min":1.581},"min":{"1min":0.311,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.589,"5min":90.597999999999999,"15min":90.597999999999999},"last":4.7770000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":2.274,"15min":1.5660000000000001},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":5.7930000000000001,"5min":90.676000000000002,"15min":90.676000000000002},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.80500000000000005,"15min":1.1020000000000001},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.27700000000000002},"max":{"1min":2.298,"5min":6.3239999999999998,"15min":41.546999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.008,"15min":1.202},"min":{"1min":0.40999999999999998,"5min":0.33000000000000002,"15min":0.31},"max":{"1min":2.1739999999999999,"5min":5.5449999999999999,"15min":41.57},"last":0.35099999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":1.4570000000000001,"15min":1.423},"min":{"1min":0.34399999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.3580000000000001,"5min":28.434000000000001,"15min":82.781000000000006},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":1.641,"15min":1.4530000000000001},"min":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.28599999999999998},"max":{"1min":3.3769999999999998,"5min":27.887,"15min":83.349000000000004},"last":3.0569999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:16:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.82599999999999996,"15min":0.89800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.25800000000000001},"max":{"1min":4.3719999999999999,"5min":4.3719999999999999,"15min":6.0460000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.111,"5min":0.93899999999999995,"15min":1.012},"min":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.31900000000000001},"max":{"1min":4.4109999999999996,"5min":4.4109999999999996,"15min":7.1070000000000002},"last":0.44900000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:16:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.054,"15min":0.98799999999999999},"min":{"1min":0.41799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.9370000000000003,"5min":5.2539999999999996,"15min":5.2539999999999996},"last":0.505},{"interface":"front","average":{"1min":0.879,"5min":0.98299999999999998,"15min":0.94499999999999995},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.27000000000000002},"max":{"1min":4.7809999999999997,"5min":6.2249999999999996,"15min":6.2249999999999996},"last":0.57599999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":1.0669999999999999,"15min":1.502},"min":{"1min":0.40500000000000003,"5min":0.36599999999999999,"15min":0.35699999999999998},"max":{"1min":3.5009999999999999,"5min":9.0879999999999992,"15min":116.273},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.1779999999999999,"15min":1.5920000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.433},"max":{"1min":3.48,"5min":5.2850000000000001,"15min":116.24299999999999},"last":0.53900000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:16:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1160000000000001,"15min":1.206},"min":{"1min":0.55500000000000005,"5min":0.442,"15min":0.442},"max":{"1min":5.0659999999999998,"5min":5.0659999999999998,"15min":8.4190000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.0289999999999999,"15min":1.147},"min":{"1min":0.443,"5min":0.443,"15min":0.42899999999999999},"max":{"1min":4.5090000000000003,"5min":4.6369999999999996,"15min":7.0380000000000003},"last":0.58999999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:16:29 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.4059999999999999,"15min":2.375},"min":{"1min":0.53600000000000003,"5min":0.51300000000000001,"15min":0.45500000000000002},"max":{"1min":4.8239999999999998,"5min":25.294,"15min":297.55900000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.9490000000000001,"5min":1.6519999999999999,"15min":2.4249999999999998},"min":{"1min":0.58499999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":8.2469999999999999,"5min":24.622,"15min":297.61900000000003},"last":0.60799999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:16:29 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.143,"15min":1.2170000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48699999999999999},"max":{"1min":6.0899999999999999,"5min":6.0899999999999999,"15min":6.9770000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.1240000000000001,"15min":1.2370000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.47599999999999998},"max":{"1min":2.4129999999999998,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.65100000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779228,"kb_used_data":482004,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92965924,"statfs":{"total":95995035648,"available":95197106176,"internally_reserved":0,"allocated":493572096,"data_stored":493204712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.82599999999999996,"15min":1.1739999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.25},"max":{"1min":2.5019999999999998,"5min":6.5960000000000001,"15min":75.396000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.879,"15min":1.214},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.27700000000000002},"max":{"1min":3.0329999999999999,"5min":5.6360000000000001,"15min":77.286000000000001},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":6.3840000000000003,"15min":2.9580000000000002},"min":{"1min":0.27900000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.516,"5min":565.34500000000003,"15min":565.34500000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.262,"5min":6.4039999999999999,"15min":2.8860000000000001},"min":{"1min":0.29199999999999998,"5min":0.28000000000000003,"15min":0.253},"max":{"1min":4.3250000000000002,"5min":565.98400000000004,"15min":565.98400000000004},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":0.97299999999999998,"15min":1.175},"min":{"1min":0.33800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":26.797000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.89200000000000002,"15min":1.071},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.28599999999999998},"max":{"1min":3.101,"5min":4.9509999999999996,"15min":26.905999999999999},"last":10.140000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":3.7029999999999998,"15min":2.7360000000000002},"min":{"1min":0.30499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.5830000000000002,"5min":252.57300000000001,"15min":252.57300000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.345,"5min":3.972,"15min":2.7480000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.548,"5min":252.24199999999999,"15min":252.24199999999999},"last":4.2469999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.8280000000000001,"15min":1.258},"min":{"1min":0.35199999999999998,"5min":0.31900000000000001,"15min":0.27300000000000002},"max":{"1min":5.6779999999999999,"5min":75.257000000000005,"15min":75.257000000000005},"last":0.434},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.8109999999999999,"15min":1.3009999999999999},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.32600000000000001},"max":{"1min":5.5949999999999998,"5min":74.427000000000007,"15min":74.427000000000007},"last":0.442}]},{"osd":6,"last update":"Mon Feb 12 16:16:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.8129999999999999,"15min":1.3660000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.318},"max":{"1min":2.847,"5min":74.105000000000004,"15min":74.105000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":2.0539999999999998,"15min":1.409},"min":{"1min":0.45100000000000001,"5min":0.371,"15min":0.35399999999999998},"max":{"1min":2.8450000000000002,"5min":75.013999999999996,"15min":75.013999999999996},"last":0.84399999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.89800000000000002,"15min":1.47},"min":{"1min":0.41999999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.8660000000000001,"5min":6.6630000000000003,"15min":98.971000000000004},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.86899999999999999,"5min":1.095,"15min":1.524},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40600000000000003},"max":{"1min":4.6379999999999999,"5min":5.9909999999999997,"15min":99.218000000000004},"last":0.69499999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.099,"15min":1.1779999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.35099999999999998},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":14.372999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.1279999999999999,"15min":1.21},"min":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.40500000000000003},"max":{"1min":2.871,"5min":5.6950000000000003,"15min":14.074999999999999},"last":0.74099999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:16:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":7.0259999999999998,"15min":3.2120000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.42999999999999999},"max":{"1min":4.4009999999999998,"5min":449.89499999999998,"15min":449.89499999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.93799999999999994,"5min":7.1920000000000002,"15min":3.2799999999999998},"min":{"1min":0.56200000000000006,"5min":0.51600000000000001,"15min":0.45300000000000001},"max":{"1min":2.899,"5min":448.70800000000003,"15min":448.70800000000003},"last":0.80700000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.2130000000000001,"15min":1.248},"min":{"1min":0.57399999999999995,"5min":0.55500000000000005,"15min":0.47499999999999998},"max":{"1min":5.9210000000000003,"5min":5.9210000000000003,"15min":14.214},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.115,"15min":1.264},"min":{"1min":0.51700000000000002,"5min":0.5,"15min":0.5},"max":{"1min":2.7719999999999998,"5min":6.1379999999999999,"15min":13.464},"last":0.626}]},{"osd":11,"last update":"Mon Feb 12 16:16:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":4.5129999999999999,"5min":4.5129999999999999,"15min":4.5129999999999999},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":495779840,"data_stored":495779840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1745223680,"data_stored":1745223680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256376832,"data_stored":256376832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":794886144,"data_stored":794886144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":769986560,"data_stored":769982488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":492830720,"data_stored":492830720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":500563968,"data_stored":500563968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":740622336,"data_stored":740622336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":746721280,"data_stored":746717208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":740950016,"data_stored":740950016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1228144640,"data_stored":1228144640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:17:21.186 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T16:17:21.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set device_health_metrics pg_num 21 2024-02-12T16:17:21.198 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:21.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:21.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:21.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:21.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:21.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:21.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:21.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:17:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:21.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:21.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:21.597 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:21.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:21.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:21.727 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:21.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:21.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:21.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:21.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:21.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:22.003 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:22.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:22.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:22.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:22.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:22.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:22.275 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:22.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:22.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:22.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:22.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:22.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:22.544 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:22.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:22.644 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:17:22.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:22.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:22.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:22.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:22.833 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:22.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:22 smithi121 bash[18489]: cluster 2024-02-12T16:17:21.748427+0000 mgr.y (mgr.14146) 962 : cluster [DBG] pgmap v1268: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 13443/118418 objects misplaced (11.352%); 23 MiB/s, 369 objects/s recovering 2024-02-12T16:17:22.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:17:22.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:23.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:23.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:23.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:22 smithi123 bash[19119]: cluster 2024-02-12T16:17:21.748427+0000 mgr.y (mgr.14146) 962 : cluster [DBG] pgmap v1268: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 13443/118418 objects misplaced (11.352%); 23 MiB/s, 369 objects/s recovering 2024-02-12T16:17:23.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:23.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:23.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:23.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:22 smithi169 bash[19604]: cluster 2024-02-12T16:17:21.748427+0000 mgr.y (mgr.14146) 962 : cluster [DBG] pgmap v1268: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 13443/118418 objects misplaced (11.352%); 23 MiB/s, 369 objects/s recovering 2024-02-12T16:17:23.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:23.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:23.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:23.436 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:23.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:23.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:23.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:23.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:23.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:23.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:23.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:23.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:23.866 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:23.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:23.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:23.998 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:24.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:24.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:24.143 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:24.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:24.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:24.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:24.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:24.376 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:24.401 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:24.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:24.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:24.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:24.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:24.755 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:24.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:24.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:24.892 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:24.930 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:24.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:25.023 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:24 smithi169 bash[19604]: cluster 2024-02-12T16:17:23.748927+0000 mgr.y (mgr.14146) 963 : cluster [DBG] pgmap v1269: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 13191/116702 objects misplaced (11.303%); 19 MiB/s, 311 objects/s recovering 2024-02-12T16:17:25.032 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:25.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:25.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:24 smithi123 bash[19119]: cluster 2024-02-12T16:17:23.748927+0000 mgr.y (mgr.14146) 963 : cluster [DBG] pgmap v1269: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 13191/116702 objects misplaced (11.303%); 19 MiB/s, 311 objects/s recovering 2024-02-12T16:17:25.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:24 smithi121 bash[18489]: cluster 2024-02-12T16:17:23.748927+0000 mgr.y (mgr.14146) 963 : cluster [DBG] pgmap v1269: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 13191/116702 objects misplaced (11.303%); 19 MiB/s, 311 objects/s recovering 2024-02-12T16:17:25.180 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:25.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:25.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:25.339 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:25.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:25.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:25.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:25.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:25.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:25.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:25.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:25.723 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:25.747 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:25.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:25.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:25.878 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:25.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:25.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:26.001 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:26.004 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:26.022 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:26.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:26.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:26.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:26.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:26.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:26.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:26.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:26.539 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:26 smithi123 bash[19119]: cluster 2024-02-12T16:17:25.749429+0000 mgr.y (mgr.14146) 964 : cluster [DBG] pgmap v1270: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 992 op/s; 12146/108462 objects misplaced (11.198%); 19 MiB/s, 309 objects/s recovering 2024-02-12T16:17:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:26 smithi121 bash[18489]: cluster 2024-02-12T16:17:25.749429+0000 mgr.y (mgr.14146) 964 : cluster [DBG] pgmap v1270: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 992 op/s; 12146/108462 objects misplaced (11.198%); 19 MiB/s, 309 objects/s recovering 2024-02-12T16:17:26.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:26.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:26.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:26.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:26 smithi169 bash[19604]: cluster 2024-02-12T16:17:25.749429+0000 mgr.y (mgr.14146) 964 : cluster [DBG] pgmap v1270: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 992 op/s; 12146/108462 objects misplaced (11.198%); 19 MiB/s, 309 objects/s recovering 2024-02-12T16:17:26.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:26.926 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:26.957 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:26.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:27.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:27.087 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:27.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:27.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:27.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:27.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:27.378 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:27.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:27.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:27.531 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:27.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:27.562 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:17:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:17:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:17:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:17:27.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183447 3.9 3:9417c732:::benchmark_data_smithi169_51390_object19663:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795664+0000", 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75411032199997, 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41474414300000001, 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183447 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795664+0000", 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795664+0000", 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795663+0000", 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2169999999999996e-06 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795667+0000", 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3999999999999992e-06 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795675+0000", 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8860000000000001e-06 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795681+0000", 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1872000000000004e-05 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795773+0000", 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8400000000000007e-06 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795783+0000", 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34675683699999998 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.142540+0000", 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014538100000000001 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for missing object", 2024-02-12T16:17:27.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.142685+0000", 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.067633352999999993 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.210319+0000", 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9127000000000002e-05 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.210408+0000", 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41474414300000001 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.567 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183455 3.9 3:9becb5fa:::benchmark_data_smithi169_51390_object19671:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795687+0000", 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.754086841, 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42402842499999999, 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183455 2024-02-12T16:17:27.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795687+0000", 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795687+0000", 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795686+0000", 2024-02-12T16:17:27.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7100000000000001e-06 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795690+0000", 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6619999999999999e-06 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795697+0000", 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4149999999999998e-06 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795702+0000", 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121103 2024-02-12T16:17:27.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795823+0000", 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3260000000000003e-06 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795829+0000", 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.348107324 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.143936+0000", 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000108733 2024-02-12T16:17:27.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for missing object", 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.144045+0000", 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.075565666000000004 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.219610+0000", 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010521 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.219715+0000", 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42402842499999999 2024-02-12T16:17:27.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183464 3.9 3:9ff6ff72:::benchmark_data_smithi169_51390_object19680:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795707+0000", 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75406644600002, 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42830448399999999, 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183464 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.573 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795707+0000", 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795707+0000", 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795707+0000", 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3869999999999999e-06 2024-02-12T16:17:27.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795710+0000", 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2419999999999997e-06 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795716+0000", 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.345e-06 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795724+0000", 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013304899999999999 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795857+0000", 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7900000000000001e-06 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795861+0000", 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34821656600000001 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.144077+0000", 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9828999999999998e-05 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for missing object", 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.144157+0000", 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.079739352999999999 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.223896+0000", 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011560799999999999 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.224012+0000", 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42830448399999999 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183478 3.9 3:9ce73675:::benchmark_data_smithi169_51390_object19694:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795729+0000", 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75404490599999, 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41175289900000001, 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183478 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795729+0000", 2024-02-12T16:17:27.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795729+0000", 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795728+0000", 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1509999999999997e-06 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795732+0000", 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9709999999999998e-06 2024-02-12T16:17:27.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795739+0000", 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.186e-06 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795744+0000", 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000137923 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795882+0000", 2024-02-12T16:17:27.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.692e-06 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795886+0000", 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34831118 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.144197+0000", 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119405 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.144317+0000", 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019422800000000001 2024-02-12T16:17:27.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.144511+0000", 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.017811381000000001 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.162322+0000", 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.043690526 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.206013+0000", 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9494999999999999e-05 2024-02-12T16:17:27.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.206072+0000", 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014095449999999999 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.207482+0000", 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41175289900000001 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.583 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183481 3.9 3:91244778:::benchmark_data_smithi169_51390_object19697:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795764+0000", 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.754009698, 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41293761099999998, 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183481 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795764+0000", 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795764+0000", 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795763+0000", 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5369999999999999e-06 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795768+0000", 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4370000000000001e-06 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795775+0000", 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2439999999999998e-06 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795782+0000", 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129789 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795911+0000", 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.76e-06 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795915+0000", 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35446942799999998 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.150385+0000", 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1971999999999994e-05 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.150466+0000", 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118816 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.150585+0000", 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012650414 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.163236+0000", 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.044347857999999997 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.207584+0000", 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9676999999999998e-05 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.207643+0000", 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001058509 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.208702+0000", 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41293761099999998 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.589 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183487 3.9 3:94fdd777:::benchmark_data_smithi169_51390_object19703:head [delete] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795787+0000", 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.753986709, 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42484203199999998, 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183487 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795787+0000", 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795787+0000", 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795786+0000", 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3470000000000001e-06 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795790+0000", 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7700000000000004e-06 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795797+0000", 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.947e-06 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795803+0000", 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013408100000000001 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795937+0000", 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8290000000000001e-06 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795941+0000", 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35869803 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.154639+0000", 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000171075 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.154810+0000", 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019204000000000001 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.155002+0000", 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082433720000000005 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.163246+0000", 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.056495861000000001 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.219741+0000", 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6378999999999997e-05 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.219798+0000", 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00083136500000000003 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.220629+0000", 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42484203199999998 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183529 3.9 3:993691a1:::benchmark_data_smithi169_51390_object19745:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795822+0000", 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75395214399998, 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42603143199999999, 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183529 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795822+0000", 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795822+0000", 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795821+0000", 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3529999999999997e-06 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795825+0000", 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7700000000000007e-06 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795834+0000", 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8239999999999996e-06 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795840+0000", 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012489399999999999 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795965+0000", 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6059999999999999e-06 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795968+0000", 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36732483100000002 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.163293+0000", 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9402999999999997e-05 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.163393+0000", 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118818 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.163511+0000", 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.017840106000000001 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.181351+0000", 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.039355523000000003 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.220707+0000", 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3565000000000006e-05 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.220771+0000", 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010825730000000001 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.221853+0000", 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42603143199999999 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.601 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183539 3.9 3:91b264e0:::benchmark_data_smithi169_51390_object19755:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795857+0000", 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75391646700001, 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.429459849, 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183539 2024-02-12T16:17:27.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795857+0000", 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795857+0000", 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795857+0000", 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1339999999999997e-06 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795861+0000", 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8550000000000008e-06 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795870+0000", 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4319999999999998e-06 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795875+0000", 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011498299999999999 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795990+0000", 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4300000000000002e-06 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795993+0000", 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.371638951 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.167632+0000", 2024-02-12T16:17:27.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010769599999999999 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.167740+0000", 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102939 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.167843+0000", 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02618593 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.194029+0000", 2024-02-12T16:17:27.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030036462999999999 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.224065+0000", 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7388000000000001e-05 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.224113+0000", 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012044549999999999 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.225317+0000", 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.429459849 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183547 3.9 3:97b8ee86:::benchmark_data_smithi169_51390_object19763:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795881+0000", 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75389313300002, 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.43544423300000001, 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.608 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183547 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795881+0000", 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795881+0000", 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795880+0000", 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8850000000000001e-06 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795884+0000", 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0790000000000003e-06 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795891+0000", 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9940000000000003e-06 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795898+0000", 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011741800000000001 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796015+0000", 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6619999999999998e-06 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796019+0000", 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37573135800000002 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.171750+0000", 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013568200000000001 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.171886+0000", 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018400099999999999 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.172070+0000", 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021971154 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.194041+0000", 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.035807559000000003 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.229849+0000", 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1123000000000003e-05 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.229920+0000", 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014050740000000001 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.231325+0000", 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.43544423300000001 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183581 3.9 3:97e9eded:::benchmark_data_smithi169_51390_object19797:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795911+0000", 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75386249600001, 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.43682122200000001, 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.614 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183581 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795911+0000", 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795911+0000", 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795911+0000", 2024-02-12T16:17:27.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.481e-06 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795915+0000", 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3470000000000001e-06 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795922+0000", 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9490000000000001e-06 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795928+0000", 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112268 2024-02-12T16:17:27.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796041+0000", 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5449999999999999e-06 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796044+0000", 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38103440100000002 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.177079+0000", 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000149782 2024-02-12T16:17:27.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.177228+0000", 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131555 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.177360+0000", 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028576475 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.205936+0000", 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025463201000000001 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.231400+0000", 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1853000000000006e-05 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.231462+0000", 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012710659999999999 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.232733+0000", 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.43682122200000001 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183643 3.9 3:92ec8a8d:::benchmark_data_smithi169_51390_object19859:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795935+0000", 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.753838755, 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44951025700000002, 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183643 2024-02-12T16:17:27.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795935+0000", 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795935+0000", 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795934+0000", 2024-02-12T16:17:27.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6850000000000007e-06 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795943+0000", 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.354e-06 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795951+0000", 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.913e-06 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795957+0000", 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010887600000000001 2024-02-12T16:17:27.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796066+0000", 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4479999999999999e-06 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796070+0000", 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.385335804 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.181405+0000", 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016144400000000001 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.181567+0000", 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019390299999999999 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.181761+0000", 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024189698999999999 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.205950+0000", 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.037808586999999998 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.243759+0000", 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5086000000000001e-05 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.243814+0000", 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016312729999999999 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.245445+0000", 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44951025700000002 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183650 3.9 3:907e4148:::benchmark_data_smithi169_51390_object19866:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795963+0000", 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75381123199998, 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.45714031399999999, 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183650 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795963+0000", 2024-02-12T16:17:27.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795963+0000", 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795962+0000", 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8039999999999999e-06 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795966+0000", 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.861e-06 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795972+0000", 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9089999999999999e-06 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795977+0000", 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113243 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796091+0000", 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6069999999999999e-06 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796094+0000", 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38958622500000001 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.185680+0000", 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015958199999999999 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.185840+0000", 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022317100000000001 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.186063+0000", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.019898862999999999 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.205962+0000", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.045279881000000001 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.251242+0000", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2704000000000001e-05 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.251305+0000", 2024-02-12T16:17:27.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017983039999999999 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.253103+0000", 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.45714031399999999 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183668 3.9 3:9c1de7b6:::benchmark_data_smithi169_51390_object19884:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.795996+0000", 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.753777872, 2024-02-12T16:17:27.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.45865203900000001, 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183668 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795996+0000", 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795996+0000", 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.795995+0000", 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5739999999999997e-06 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796000+0000", 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3370000000000002e-06 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796007+0000", 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8839999999999999e-06 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796013+0000", 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010307299999999999 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796116+0000", 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4039999999999999e-06 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796119+0000", 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39395813699999999 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.190077+0000", 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014114100000000001 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.190219+0000", 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014768600000000001 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.190366+0000", 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.017132606000000002 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.207499+0000", 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.045711148 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.253210+0000", 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5519999999999995e-05 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.253286+0000", 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001362474 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.254648+0000", 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.45865203900000001 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.637 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:183687 3.9 3:91951e80:::benchmark_data_smithi169_51390_object19903:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e294)", 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:11:00.796018+0000", 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 386.75375585500001, 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.47430728599999999, 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 183687 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796018+0000", 2024-02-12T16:17:27.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796018+0000", 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796017+0000", 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9439999999999998e-06 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796021+0000", 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6229999999999997e-06 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796029+0000", 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6509999999999998e-06 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796034+0000", 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100078 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796135+0000", 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.597e-06 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:00.796138+0000", 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39795026500000003 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.194088+0000", 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012164599999999999 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.194210+0000", 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015576500000000001 2024-02-12T16:17:27.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.194366+0000", 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013143026 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.207509+0000", 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060964698999999997 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.268474+0000", 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3621999999999995e-05 2024-02-12T16:17:27.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.268537+0000", 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00178816 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:11:01.270325+0000", 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.47430728599999999 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.15108.0:287236 3.e 3:713cbbd1:::benchmark_data_smithi169_51390_object123452:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:17:27.643 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:12:34.529411+0000", 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 293.02036264200001, 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.48329744899999999, 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.15108", 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 287236 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529411+0000", 2024-02-12T16:17:27.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529411+0000", 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529410+0000", 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2480000000000004e-06 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529415+0000", 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7609999999999996e-06 2024-02-12T16:17:27.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529424+0000", 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2729999999999996e-06 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529428+0000", 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1230000000000002e-05 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529479+0000", 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3788000000000004e-05 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529573+0000", 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3323000000000001e-05 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 4", 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.529666+0000", 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040671129999999998 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:34.533733+0000", 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.47876343700000001 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:35.012497+0000", 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2271000000000001e-05 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:35.012549+0000", 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015959899999999999 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:12:35.012709+0000", 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.48329744899999999 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204321 4.9 4:96e11958:::benchmark_data_smithi169_60938_object30984:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.009483+0000", 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 91.540290830000004, 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62252338500000004, 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204321 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009483+0000", 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009483+0000", 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009480+0000", 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1177e-05 2024-02-12T16:17:27.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009491+0000", 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8924999999999999e-05 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009510+0000", 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7619999999999995e-06 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009518+0000", 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0454000000000002e-05 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009568+0000", 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015020100000000001 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009719+0000", 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020923099999999999 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009928+0000", 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060745019999999999 2024-02-12T16:17:27.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.016002+0000", 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.61570960299999999 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.631712+0000", 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015550000000000001 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.631867+0000", 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013907700000000001 2024-02-12T16:17:27.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.632006+0000", 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62252338500000004 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228647 4.8 4:15c660ab:::benchmark_data_smithi169_60938_object55310:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.305729+0000", 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 67.244044540000004, 2024-02-12T16:17:27.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38848608099999998, 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228647 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305729+0000", 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305729+0000", 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305728+0000", 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8180000000000003e-06 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305733+0000", 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3279999999999996e-06 2024-02-12T16:17:27.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305738+0000", 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2540000000000001e-06 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305741+0000", 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044469720000000004 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310188+0000", 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102354 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310291+0000", 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119107 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310410+0000", 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049471979999999999 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.315357+0000", 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37877825500000001 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.694135+0000", 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6576999999999998e-05 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.694182+0000", 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3447e-05 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.694215+0000", 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38848608099999998 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228649 4.1 4:83f34d8e:::benchmark_data_smithi169_60938_object55312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.307435+0000", 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 67.242339341000005, 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38982620600000001, 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228649 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307435+0000", 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307435+0000", 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307433+0000", 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9450000000000001e-06 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307438+0000", 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5780000000000001e-06 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307446+0000", 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3259999999999997e-06 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307450+0000", 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8389000000000001e-05 2024-02-12T16:17:27.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307489+0000", 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6730000000000005e-05 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307575+0000", 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8439999999999999e-05 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:17:27.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307674+0000", 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37720748199999998 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.684881+0000", 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012317112 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697198+0000", 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5228999999999998e-05 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697234+0000", 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7214999999999999e-05 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697261+0000", 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38982620600000001 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228650 4.1 4:8a3347e3:::benchmark_data_smithi169_60938_object55313:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:17:27.665 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.307642+0000", 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 67.242131365000006, 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38971022799999999, 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228650 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307642+0000", 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307642+0000", 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307641+0000", 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6260000000000003e-06 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307646+0000", 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2560000000000002e-06 2024-02-12T16:17:27.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307651+0000", 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0900000000000001e-06 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307654+0000", 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046890480000000003 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.312343+0000", 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121348 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.312465+0000", 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012537599999999999 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.312590+0000", 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37316567299999998 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.685756+0000", 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011512557 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697268+0000", 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7462000000000001e-05 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697316+0000", 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6844000000000002e-05 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697353+0000", 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38971022799999999 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228653 4.9 4:96aa022a:::benchmark_data_smithi169_60938_object55316:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.310158+0000", 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 67.239616101999999, 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39785818499999998, 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228653 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310158+0000", 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:17:27.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310158+0000", 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310156+0000", 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5910000000000003e-06 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310162+0000", 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1329999999999999e-06 2024-02-12T16:17:27.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310170+0000", 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2270000000000004e-06 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310174+0000", 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37473862699999999 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.684913+0000", 2024-02-12T16:17:27.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000136811 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.685049+0000", 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017495299999999999 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.685224+0000", 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022617696 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.707842+0000", 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8616000000000003e-05 2024-02-12T16:17:27.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.707901+0000", 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8312e-05 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.707959+0000", 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7003999999999998e-05 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.708016+0000", 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39785818499999998 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.675 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.676 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:17:27.676 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:17:27.676 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:17:27.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:27.706 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:27.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:27.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:27.833 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:27.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:27.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:27.968 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:27.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:28.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:28.099 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:28.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:28.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:28.251 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:28.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:28.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:28.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:28.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:28.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:17:28.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:28.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:28.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:28.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:28.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:28.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:28.840 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:28.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:28.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:28.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:28.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:29.065 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:28 smithi121 bash[18489]: cluster 2024-02-12T16:17:27.749828+0000 mgr.y (mgr.14146) 965 : cluster [DBG] pgmap v1271: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 11447/106398 objects misplaced (10.759%); 17 MiB/s, 271 objects/s recovering 2024-02-12T16:17:29.065 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:28 smithi121 bash[18489]: cluster 2024-02-12T16:17:27.857555+0000 osd.1 (osd.1) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:29.065 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:28 smithi121 bash[18489]: cluster 2024-02-12T16:17:27.857606+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:29.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:29.103 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:28 smithi169 bash[19604]: cluster 2024-02-12T16:17:27.749828+0000 mgr.y (mgr.14146) 965 : cluster [DBG] pgmap v1271: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 11447/106398 objects misplaced (10.759%); 17 MiB/s, 271 objects/s recovering 2024-02-12T16:17:29.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:28 smithi169 bash[19604]: cluster 2024-02-12T16:17:27.857555+0000 osd.1 (osd.1) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:29.104 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:28 smithi169 bash[19604]: cluster 2024-02-12T16:17:27.857606+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:29.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:29.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:29.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:28 smithi123 bash[19119]: cluster 2024-02-12T16:17:27.749828+0000 mgr.y (mgr.14146) 965 : cluster [DBG] pgmap v1271: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 11447/106398 objects misplaced (10.759%); 17 MiB/s, 271 objects/s recovering 2024-02-12T16:17:29.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:28 smithi123 bash[19119]: cluster 2024-02-12T16:17:27.857555+0000 osd.1 (osd.1) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:29.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:28 smithi123 bash[19119]: cluster 2024-02-12T16:17:27.857606+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:29.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:29.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:29.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:29.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:29.411 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:29.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:29.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:29.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:29.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:29.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:29.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:29.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:29.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:29.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:29.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:29.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:29.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:29.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:30.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:30.077 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:29 smithi169 bash[19604]: cluster 2024-02-12T16:17:28.812477+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e407: 12 total, 12 up, 11 in 2024-02-12T16:17:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:30.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:30.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:29 smithi123 bash[19119]: cluster 2024-02-12T16:17:28.812477+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e407: 12 total, 12 up, 11 in 2024-02-12T16:17:30.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:29 smithi121 bash[18489]: cluster 2024-02-12T16:17:28.812477+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e407: 12 total, 12 up, 11 in 2024-02-12T16:17:30.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:30.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:30.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:30.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:30.363 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:30.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:30.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:30.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:30.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:30.631 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:30.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:30.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:30.765 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:30.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:30.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:30.894 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:30.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:30.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:31.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:31.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:31.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:30 smithi123 bash[19119]: cluster 2024-02-12T16:17:29.750277+0000 mgr.y (mgr.14146) 966 : cluster [DBG] pgmap v1273: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 935 op/s; 11111/104404 objects misplaced (10.642%); 13 MiB/s, 214 objects/s recovering 2024-02-12T16:17:31.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:30 smithi121 bash[18489]: cluster 2024-02-12T16:17:29.750277+0000 mgr.y (mgr.14146) 966 : cluster [DBG] pgmap v1273: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 935 op/s; 11111/104404 objects misplaced (10.642%); 13 MiB/s, 214 objects/s recovering 2024-02-12T16:17:31.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:31.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:30 smithi169 bash[19604]: cluster 2024-02-12T16:17:29.750277+0000 mgr.y (mgr.14146) 966 : cluster [DBG] pgmap v1273: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 935 op/s; 11111/104404 objects misplaced (10.642%); 13 MiB/s, 214 objects/s recovering 2024-02-12T16:17:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:31.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:31.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:31.536 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:31.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:31.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:31.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:31.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:31.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:31.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:31.981 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:31.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:32.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:32.113 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:32.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:32.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:32.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:32.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:32.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:32.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:32.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:32.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:32.535 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:32.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:32.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:32.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:32.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:32.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:32.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:32.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:32.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:32.978 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:32.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:33.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:33.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:32 smithi121 bash[18489]: cluster 2024-02-12T16:17:31.750731+0000 mgr.y (mgr.14146) 967 : cluster [DBG] pgmap v1274: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 10000/96862 objects misplaced (10.324%); 7.7 MiB/s, 123 objects/s recovering 2024-02-12T16:17:33.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:33.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:32 smithi123 bash[19119]: cluster 2024-02-12T16:17:31.750731+0000 mgr.y (mgr.14146) 967 : cluster [DBG] pgmap v1274: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 10000/96862 objects misplaced (10.324%); 7.7 MiB/s, 123 objects/s recovering 2024-02-12T16:17:33.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:33.248 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:33.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:32 smithi169 bash[19604]: cluster 2024-02-12T16:17:31.750731+0000 mgr.y (mgr.14146) 967 : cluster [DBG] pgmap v1274: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 10000/96862 objects misplaced (10.324%); 7.7 MiB/s, 123 objects/s recovering 2024-02-12T16:17:33.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:33.380 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:33.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:33.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:33.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:33.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:33.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:33.639 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:33.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:33.745 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:33.768 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:33.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:33.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:33.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:17:33.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:33.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:33.971 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:17:34.035 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:17:34.047 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:34.054 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:33 smithi123 bash[19119]: cluster 2024-02-12T16:17:32.579732+0000 osd.9 (osd.9) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:34.054 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:33 smithi123 bash[19119]: cluster 2024-02-12T16:17:32.579775+0000 osd.9 (osd.9) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:34.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:33 smithi123 bash[19119]: cluster 2024-02-12T16:17:32.838085+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e408: 12 total, 12 up, 11 in 2024-02-12T16:17:34.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:17:34.073 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:34.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:34.105 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:17:34.135 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:17:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:33 smithi121 bash[18489]: cluster 2024-02-12T16:17:32.579732+0000 osd.9 (osd.9) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:33 smithi121 bash[18489]: cluster 2024-02-12T16:17:32.579775+0000 osd.9 (osd.9) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:33 smithi121 bash[18489]: cluster 2024-02-12T16:17:32.838085+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e408: 12 total, 12 up, 11 in 2024-02-12T16:17:34.166 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:17:34.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:34.183 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:33 smithi169 bash[19604]: cluster 2024-02-12T16:17:32.579732+0000 osd.9 (osd.9) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:34.183 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:33 smithi169 bash[19604]: cluster 2024-02-12T16:17:32.579775+0000 osd.9 (osd.9) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:34.183 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:33 smithi169 bash[19604]: cluster 2024-02-12T16:17:32.838085+0000 mon.a (mon.0) 1537 : cluster [DBG] osdmap e408: 12 total, 12 up, 11 in 2024-02-12T16:17:34.196 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:17:34.206 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:34.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:34.231 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:17:34.271 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:17:34.299 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:17:34.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:34.329 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.342 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:17:34.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:34.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:34.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:34.474 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:34.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:34.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:34.605 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:34.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:34.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:34.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:34.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:34.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:34.858 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:17:34.864 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 1 pg_num to 21 2024-02-12T16:17:34.870 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:34.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:34.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:35.005 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:34 smithi169 bash[19604]: cluster 2024-02-12T16:17:33.751233+0000 mgr.y (mgr.14146) 968 : cluster [DBG] pgmap v1276: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 2.9 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 9636/94742 objects misplaced (10.171%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T16:17:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:34 smithi169 bash[19604]: audit 2024-02-12T16:17:33.851402+0000 mon.b (mon.2) 126 : audit [DBG] from='client.15627 v1:172.21.15.121:0/2463949620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:34 smithi169 bash[19604]: audit 2024-02-12T16:17:33.851670+0000 mon.b (mon.2) 127 : audit [DBG] from='client.15627 v1:172.21.15.121:0/2463949620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:34 smithi169 bash[19604]: audit 2024-02-12T16:17:33.999280+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? v1:172.21.15.121:0/1507916925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "21"}]: dispatch 2024-02-12T16:17:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:34 smithi169 bash[19604]: audit 2024-02-12T16:17:34.070541+0000 mon.a (mon.0) 1539 : audit [INF] from='client.25520 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:17:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:34 smithi169 bash[19604]: audit 2024-02-12T16:17:34.072601+0000 mon.b (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.121:0/1691597515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:17:35.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:35.116 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:34 smithi123 bash[19119]: cluster 2024-02-12T16:17:33.751233+0000 mgr.y (mgr.14146) 968 : cluster [DBG] pgmap v1276: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 2.9 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 9636/94742 objects misplaced (10.171%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T16:17:35.116 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:34 smithi123 bash[19119]: audit 2024-02-12T16:17:33.851402+0000 mon.b (mon.2) 126 : audit [DBG] from='client.15627 v1:172.21.15.121:0/2463949620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:35.116 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:34 smithi123 bash[19119]: audit 2024-02-12T16:17:33.851670+0000 mon.b (mon.2) 127 : audit [DBG] from='client.15627 v1:172.21.15.121:0/2463949620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:35.116 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:34 smithi123 bash[19119]: audit 2024-02-12T16:17:33.999280+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? v1:172.21.15.121:0/1507916925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "21"}]: dispatch 2024-02-12T16:17:35.116 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:34 smithi123 bash[19119]: audit 2024-02-12T16:17:34.070541+0000 mon.a (mon.0) 1539 : audit [INF] from='client.25520 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:17:35.116 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:34 smithi123 bash[19119]: audit 2024-02-12T16:17:34.072601+0000 mon.b (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.121:0/1691597515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:17:35.133 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:35.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:34 smithi121 bash[18489]: cluster 2024-02-12T16:17:33.751233+0000 mgr.y (mgr.14146) 968 : cluster [DBG] pgmap v1276: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 2.9 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 9636/94742 objects misplaced (10.171%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T16:17:35.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:34 smithi121 bash[18489]: audit 2024-02-12T16:17:33.851402+0000 mon.b (mon.2) 126 : audit [DBG] from='client.15627 v1:172.21.15.121:0/2463949620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:34 smithi121 bash[18489]: audit 2024-02-12T16:17:33.851670+0000 mon.b (mon.2) 127 : audit [DBG] from='client.15627 v1:172.21.15.121:0/2463949620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:34 smithi121 bash[18489]: audit 2024-02-12T16:17:33.999280+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? v1:172.21.15.121:0/1507916925' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "21"}]: dispatch 2024-02-12T16:17:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:34 smithi121 bash[18489]: audit 2024-02-12T16:17:34.070541+0000 mon.a (mon.0) 1539 : audit [INF] from='client.25520 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:17:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:34 smithi121 bash[18489]: audit 2024-02-12T16:17:34.072601+0000 mon.b (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.121:0/1691597515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:17:35.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:35.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:35.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:35.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:35.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:35.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.564 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:35.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:35.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:35.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:35.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:35.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:35.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:35.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:36.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:36.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:36.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:36.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:35 smithi121 bash[18489]: cluster 2024-02-12T16:17:34.838020+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:36.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:35 smithi121 bash[18489]: audit 2024-02-12T16:17:34.849419+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? v1:172.21.15.121:0/1507916925' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "21"}]': finished 2024-02-12T16:17:36.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:35 smithi121 bash[18489]: audit 2024-02-12T16:17:34.849498+0000 mon.a (mon.0) 1542 : audit [INF] from='client.25520 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:17:36.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:35 smithi121 bash[18489]: cluster 2024-02-12T16:17:34.849565+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e409: 12 total, 12 up, 11 in 2024-02-12T16:17:36.118 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:35 smithi121 bash[18489]: cluster 2024-02-12T16:17:35.596400+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.b scrub starts 2024-02-12T16:17:36.118 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:35 smithi121 bash[18489]: audit 2024-02-12T16:17:35.752701+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T16:17:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:35 smithi123 bash[19119]: cluster 2024-02-12T16:17:34.838020+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:35 smithi123 bash[19119]: audit 2024-02-12T16:17:34.849419+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? v1:172.21.15.121:0/1507916925' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "21"}]': finished 2024-02-12T16:17:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:35 smithi123 bash[19119]: audit 2024-02-12T16:17:34.849498+0000 mon.a (mon.0) 1542 : audit [INF] from='client.25520 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:17:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:35 smithi123 bash[19119]: cluster 2024-02-12T16:17:34.849565+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e409: 12 total, 12 up, 11 in 2024-02-12T16:17:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:35 smithi123 bash[19119]: cluster 2024-02-12T16:17:35.596400+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.b scrub starts 2024-02-12T16:17:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:35 smithi123 bash[19119]: audit 2024-02-12T16:17:35.752701+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T16:17:36.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:36.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:36.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:35 smithi169 bash[19604]: cluster 2024-02-12T16:17:34.838020+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:17:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:35 smithi169 bash[19604]: audit 2024-02-12T16:17:34.849419+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? v1:172.21.15.121:0/1507916925' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "21"}]': finished 2024-02-12T16:17:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:35 smithi169 bash[19604]: audit 2024-02-12T16:17:34.849498+0000 mon.a (mon.0) 1542 : audit [INF] from='client.25520 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:17:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:35 smithi169 bash[19604]: cluster 2024-02-12T16:17:34.849565+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e409: 12 total, 12 up, 11 in 2024-02-12T16:17:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:35 smithi169 bash[19604]: cluster 2024-02-12T16:17:35.596400+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.b scrub starts 2024-02-12T16:17:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:35 smithi169 bash[19604]: audit 2024-02-12T16:17:35.752701+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-12T16:17:36.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:36.371 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:36.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:36.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:36.510 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:36.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:36.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:36.634 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:36.642 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:36.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:36.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:36.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:36.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:36.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:36.920 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:36.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:37.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:37.034 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:36 smithi121 bash[18489]: cluster 2024-02-12T16:17:35.751767+0000 mgr.y (mgr.14146) 969 : cluster [DBG] pgmap v1278: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 2.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s; 8683/86466 objects misplaced (10.042%); 11 MiB/s, 170 objects/s recovering 2024-02-12T16:17:37.034 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:36 smithi121 bash[18489]: cluster 2024-02-12T16:17:36.328867+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.b scrub ok 2024-02-12T16:17:37.034 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:36 smithi121 bash[18489]: audit 2024-02-12T16:17:36.429585+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-12T16:17:37.034 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:36 smithi121 bash[18489]: cluster 2024-02-12T16:17:36.429642+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e410: 12 total, 12 up, 11 in 2024-02-12T16:17:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:36 smithi169 bash[19604]: cluster 2024-02-12T16:17:35.751767+0000 mgr.y (mgr.14146) 969 : cluster [DBG] pgmap v1278: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 2.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s; 8683/86466 objects misplaced (10.042%); 11 MiB/s, 170 objects/s recovering 2024-02-12T16:17:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:36 smithi169 bash[19604]: cluster 2024-02-12T16:17:36.328867+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.b scrub ok 2024-02-12T16:17:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:36 smithi169 bash[19604]: audit 2024-02-12T16:17:36.429585+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-12T16:17:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:36 smithi169 bash[19604]: cluster 2024-02-12T16:17:36.429642+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e410: 12 total, 12 up, 11 in 2024-02-12T16:17:37.066 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:37.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:36 smithi123 bash[19119]: cluster 2024-02-12T16:17:35.751767+0000 mgr.y (mgr.14146) 969 : cluster [DBG] pgmap v1278: 27 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 23 active+clean; 2.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s; 8683/86466 objects misplaced (10.042%); 11 MiB/s, 170 objects/s recovering 2024-02-12T16:17:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:36 smithi123 bash[19119]: cluster 2024-02-12T16:17:36.328867+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.b scrub ok 2024-02-12T16:17:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:36 smithi123 bash[19119]: audit 2024-02-12T16:17:36.429585+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-12T16:17:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:36 smithi123 bash[19119]: cluster 2024-02-12T16:17:36.429642+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e410: 12 total, 12 up, 11 in 2024-02-12T16:17:37.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:37.196 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:37.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:37.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:37.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:37.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:37.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:37.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:17:37.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:37.652 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:37.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:37.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:37.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:37.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:37.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:37.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:37.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:37.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:38.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:38.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:38.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:38.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:38.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:17:38.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:38.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:38.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:38.347 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:38.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:38.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:38 smithi121 bash[18489]: cluster 2024-02-12T16:17:36.810564+0000 osd.10 (osd.10) 40 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:17:38.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:38 smithi121 bash[18489]: cluster 2024-02-12T16:17:37.398534+0000 osd.10 (osd.10) 41 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:17:38.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:38 smithi121 bash[18489]: cluster 2024-02-12T16:17:37.444172+0000 mon.a (mon.0) 1547 : cluster [DBG] osdmap e411: 12 total, 12 up, 11 in 2024-02-12T16:17:38.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:38 smithi121 bash[18489]: cluster 2024-02-12T16:17:37.752339+0000 mgr.y (mgr.14146) 970 : cluster [DBG] pgmap v1281: 37 pgs: 2 active+remapped+backfill_toofull, 10 unknown, 2 active+remapped+backfilling, 23 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 7974/84328 objects misplaced (9.456%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T16:17:38.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:38.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:38.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:38.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:38.601 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:38 smithi169 bash[19604]: cluster 2024-02-12T16:17:36.810564+0000 osd.10 (osd.10) 40 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:17:38.601 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:38 smithi169 bash[19604]: cluster 2024-02-12T16:17:37.398534+0000 osd.10 (osd.10) 41 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:17:38.601 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:38 smithi169 bash[19604]: cluster 2024-02-12T16:17:37.444172+0000 mon.a (mon.0) 1547 : cluster [DBG] osdmap e411: 12 total, 12 up, 11 in 2024-02-12T16:17:38.601 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:38 smithi169 bash[19604]: cluster 2024-02-12T16:17:37.752339+0000 mgr.y (mgr.14146) 970 : cluster [DBG] pgmap v1281: 37 pgs: 2 active+remapped+backfill_toofull, 10 unknown, 2 active+remapped+backfilling, 23 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 7974/84328 objects misplaced (9.456%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T16:17:38.616 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:38.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:38.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:38.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:38.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:38.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:38.878 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:38.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:38 smithi123 bash[19119]: cluster 2024-02-12T16:17:36.810564+0000 osd.10 (osd.10) 40 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:17:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:38 smithi123 bash[19119]: cluster 2024-02-12T16:17:37.398534+0000 osd.10 (osd.10) 41 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:17:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:38 smithi123 bash[19119]: cluster 2024-02-12T16:17:37.444172+0000 mon.a (mon.0) 1547 : cluster [DBG] osdmap e411: 12 total, 12 up, 11 in 2024-02-12T16:17:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:38 smithi123 bash[19119]: cluster 2024-02-12T16:17:37.752339+0000 mgr.y (mgr.14146) 970 : cluster [DBG] pgmap v1281: 37 pgs: 2 active+remapped+backfill_toofull, 10 unknown, 2 active+remapped+backfilling, 23 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s; 7974/84328 objects misplaced (9.456%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T16:17:38.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:39.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:39.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:39.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:39.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:39.296 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:39.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:39.436 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:39.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:39.548 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:39 smithi123 bash[19119]: cluster 2024-02-12T16:17:37.509713+0000 osd.7 (osd.7) 71 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:17:39.548 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:39 smithi123 bash[19119]: cluster 2024-02-12T16:17:37.559306+0000 osd.7 (osd.7) 72 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:17:39.548 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:39 smithi123 bash[19119]: cluster 2024-02-12T16:17:38.448634+0000 mon.a (mon.0) 1548 : cluster [DBG] osdmap e412: 12 total, 12 up, 11 in 2024-02-12T16:17:39.548 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:39 smithi123 bash[19119]: cluster 2024-02-12T16:17:38.685692+0000 osd.11 (osd.11) 61 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:17:39.548 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:39 smithi123 bash[19119]: cluster 2024-02-12T16:17:39.166454+0000 osd.11 (osd.11) 62 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:17:39.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:39.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:39.696 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:39.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:39 smithi169 bash[19604]: cluster 2024-02-12T16:17:37.509713+0000 osd.7 (osd.7) 71 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:17:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:39 smithi169 bash[19604]: cluster 2024-02-12T16:17:37.559306+0000 osd.7 (osd.7) 72 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:17:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:39 smithi169 bash[19604]: cluster 2024-02-12T16:17:38.448634+0000 mon.a (mon.0) 1548 : cluster [DBG] osdmap e412: 12 total, 12 up, 11 in 2024-02-12T16:17:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:39 smithi169 bash[19604]: cluster 2024-02-12T16:17:38.685692+0000 osd.11 (osd.11) 61 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:17:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:39 smithi169 bash[19604]: cluster 2024-02-12T16:17:39.166454+0000 osd.11 (osd.11) 62 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:17:39.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:39.830 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:39 smithi121 bash[18489]: cluster 2024-02-12T16:17:37.509713+0000 osd.7 (osd.7) 71 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:17:39.941 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:39 smithi121 bash[18489]: cluster 2024-02-12T16:17:37.559306+0000 osd.7 (osd.7) 72 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:17:39.941 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:39 smithi121 bash[18489]: cluster 2024-02-12T16:17:38.448634+0000 mon.a (mon.0) 1548 : cluster [DBG] osdmap e412: 12 total, 12 up, 11 in 2024-02-12T16:17:39.942 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:39 smithi121 bash[18489]: cluster 2024-02-12T16:17:38.685692+0000 osd.11 (osd.11) 61 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:17:39.942 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:39 smithi121 bash[18489]: cluster 2024-02-12T16:17:39.166454+0000 osd.11 (osd.11) 62 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:17:39.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:39.970 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:39.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:40.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:40.096 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:40.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:40.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:40.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:40.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:40.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:40.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:40.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:40.476 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:40.504 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:40.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:40.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:40.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:40 smithi123 bash[19119]: cluster 2024-02-12T16:17:39.459074+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e413: 12 total, 12 up, 11 in 2024-02-12T16:17:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:40 smithi123 bash[19119]: cluster 2024-02-12T16:17:39.752787+0000 mgr.y (mgr.14146) 971 : cluster [DBG] pgmap v1284: 37 pgs: 2 active+remapped+backfill_toofull, 4 unknown, 2 active+remapped+backfilling, 29 active+clean; 2.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 7652/82426 objects misplaced (9.283%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:17:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:40.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:40.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:40.778 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:40.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:40 smithi169 bash[19604]: cluster 2024-02-12T16:17:39.459074+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e413: 12 total, 12 up, 11 in 2024-02-12T16:17:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:40 smithi169 bash[19604]: cluster 2024-02-12T16:17:39.752787+0000 mgr.y (mgr.14146) 971 : cluster [DBG] pgmap v1284: 37 pgs: 2 active+remapped+backfill_toofull, 4 unknown, 2 active+remapped+backfilling, 29 active+clean; 2.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 7652/82426 objects misplaced (9.283%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:17:40.831 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:17:40.831 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:17:40.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:40.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:40 smithi121 bash[18489]: cluster 2024-02-12T16:17:39.459074+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e413: 12 total, 12 up, 11 in 2024-02-12T16:17:40.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:40 smithi121 bash[18489]: cluster 2024-02-12T16:17:39.752787+0000 mgr.y (mgr.14146) 971 : cluster [DBG] pgmap v1284: 37 pgs: 2 active+remapped+backfill_toofull, 4 unknown, 2 active+remapped+backfilling, 29 active+clean; 2.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 7652/82426 objects misplaced (9.283%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:17:40.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:40.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:17:41.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:41.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:41.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:41.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:41.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:41.241 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:17:41.241 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:17:41.242 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:17:41.242 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-12T16:17:41.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.3 2024-02-12T16:17:41.303 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:41.327 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:41.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:41.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:41.457 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:41.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:41.499 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[18489]: cluster 2024-02-12T16:17:39.508407+0000 osd.7 (osd.7) 73 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:17:41.499 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[18489]: cluster 2024-02-12T16:17:39.538739+0000 osd.7 (osd.7) 74 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:17:41.499 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[18489]: cluster 2024-02-12T16:17:39.968773+0000 osd.0 (osd.0) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:41.499 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[18489]: cluster 2024-02-12T16:17:39.968812+0000 osd.0 (osd.0) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:41.499 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[18489]: cluster 2024-02-12T16:17:40.495097+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e414: 12 total, 12 up, 11 in 2024-02-12T16:17:41.500 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:17:41 smithi121 systemd[1]: Stopping Ceph osd.3 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:17:41.500 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[130662]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:17:41.560 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:41.567 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:41 smithi123 bash[19119]: cluster 2024-02-12T16:17:39.508407+0000 osd.7 (osd.7) 73 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:17:41.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:41 smithi123 bash[19119]: cluster 2024-02-12T16:17:39.538739+0000 osd.7 (osd.7) 74 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:17:41.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:41 smithi123 bash[19119]: cluster 2024-02-12T16:17:39.968773+0000 osd.0 (osd.0) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:41.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:41 smithi123 bash[19119]: cluster 2024-02-12T16:17:39.968812+0000 osd.0 (osd.0) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:41.568 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:41 smithi123 bash[19119]: cluster 2024-02-12T16:17:40.495097+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e414: 12 total, 12 up, 11 in 2024-02-12T16:17:41.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:41.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:41.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:41.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:41.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:41.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:41 smithi169 bash[19604]: cluster 2024-02-12T16:17:39.508407+0000 osd.7 (osd.7) 73 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:17:41.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:41 smithi169 bash[19604]: cluster 2024-02-12T16:17:39.538739+0000 osd.7 (osd.7) 74 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:17:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:41 smithi169 bash[19604]: cluster 2024-02-12T16:17:39.968773+0000 osd.0 (osd.0) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:41 smithi169 bash[19604]: cluster 2024-02-12T16:17:39.968812+0000 osd.0 (osd.0) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:41 smithi169 bash[19604]: cluster 2024-02-12T16:17:40.495097+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e414: 12 total, 12 up, 11 in 2024-02-12T16:17:41.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:17:41.809 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[44443]: debug 2024-02-12T16:17:41.660+0000 7fdc97767700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:17:41.809 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[44443]: debug 2024-02-12T16:17:41.660+0000 7fdc97767700 -1 osd.3 414 *** Got signal Terminated *** 2024-02-12T16:17:41.809 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:17:41 smithi121 bash[44443]: debug 2024-02-12T16:17:41.660+0000 7fdc97767700 -1 osd.3 414 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:17:41.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:41.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:41.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:42.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:42.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:42.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:42.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:42.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:42.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:42.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:42.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:42.361 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:42.362 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:42.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:42.466 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:42.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:42.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:42.577 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:42 smithi123 bash[19119]: cluster 2024-02-12T16:17:41.492054+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:17:42.578 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:42 smithi123 bash[19119]: cluster 2024-02-12T16:17:41.533535+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:17:42.578 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:42 smithi123 bash[19119]: cluster 2024-02-12T16:17:41.665431+0000 mon.a (mon.0) 1551 : cluster [INF] osd.3 marked itself down and dead 2024-02-12T16:17:42.578 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:42 smithi123 bash[19119]: cluster 2024-02-12T16:17:41.753253+0000 mgr.y (mgr.14146) 972 : cluster [DBG] pgmap v1286: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 2.3 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 927 op/s; 6983/76326 objects misplaced (9.149%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:17:42.578 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:42 smithi123 bash[19119]: cluster 2024-02-12T16:17:41.862853+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:42.578 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:42 smithi123 bash[19119]: cluster 2024-02-12T16:17:41.862916+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:42.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:42.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:42.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:42.735 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:42.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:42 smithi169 bash[19604]: cluster 2024-02-12T16:17:41.492054+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:17:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:42 smithi169 bash[19604]: cluster 2024-02-12T16:17:41.533535+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:17:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:42 smithi169 bash[19604]: cluster 2024-02-12T16:17:41.665431+0000 mon.a (mon.0) 1551 : cluster [INF] osd.3 marked itself down and dead 2024-02-12T16:17:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:42 smithi169 bash[19604]: cluster 2024-02-12T16:17:41.753253+0000 mgr.y (mgr.14146) 972 : cluster [DBG] pgmap v1286: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 2.3 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 927 op/s; 6983/76326 objects misplaced (9.149%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:17:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:42 smithi169 bash[19604]: cluster 2024-02-12T16:17:41.862853+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:42 smithi169 bash[19604]: cluster 2024-02-12T16:17:41.862916+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:42.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:42.850 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:42 smithi121 bash[18489]: cluster 2024-02-12T16:17:41.492054+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:42 smithi121 bash[18489]: cluster 2024-02-12T16:17:41.533535+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:42 smithi121 bash[18489]: cluster 2024-02-12T16:17:41.665431+0000 mon.a (mon.0) 1551 : cluster [INF] osd.3 marked itself down and dead 2024-02-12T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:42 smithi121 bash[18489]: cluster 2024-02-12T16:17:41.753253+0000 mgr.y (mgr.14146) 972 : cluster [DBG] pgmap v1286: 37 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 34 active+clean; 2.3 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 927 op/s; 6983/76326 objects misplaced (9.149%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:42 smithi121 bash[18489]: cluster 2024-02-12T16:17:41.862853+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:42.851 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:42 smithi121 bash[18489]: cluster 2024-02-12T16:17:41.862916+0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:42.877 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:42.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:42.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:43.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:43.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:43.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:43.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.438 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:43.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:43.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:43.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:43.695 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:43.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:43 smithi169 bash[19604]: cluster 2024-02-12T16:17:42.493886+0000 mon.a (mon.0) 1552 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T16:17:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:43 smithi169 bash[19604]: cluster 2024-02-12T16:17:42.503138+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e415: 12 total, 11 up, 11 in 2024-02-12T16:17:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:43 smithi169 bash[19604]: cluster 2024-02-12T16:17:42.523203+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.d scrub starts 2024-02-12T16:17:43.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:43 smithi169 bash[19604]: cluster 2024-02-12T16:17:42.558792+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:17:43.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:43 smithi169 bash[19604]: cluster 2024-02-12T16:17:42.905479+0000 osd.6 (osd.6) 55 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:17:43.828 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:43.882 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:43 smithi121 bash[18489]: cluster 2024-02-12T16:17:42.493886+0000 mon.a (mon.0) 1552 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T16:17:43.882 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:43 smithi121 bash[18489]: cluster 2024-02-12T16:17:42.503138+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e415: 12 total, 11 up, 11 in 2024-02-12T16:17:43.882 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:43 smithi121 bash[18489]: cluster 2024-02-12T16:17:42.523203+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.d scrub starts 2024-02-12T16:17:43.882 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:43 smithi121 bash[18489]: cluster 2024-02-12T16:17:42.558792+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:17:43.882 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:43 smithi121 bash[18489]: cluster 2024-02-12T16:17:42.905479+0000 osd.6 (osd.6) 55 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:17:43.884 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:17:43 smithi121 bash[130662]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-3 2024-02-12T16:17:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:43 smithi123 bash[19119]: cluster 2024-02-12T16:17:42.493886+0000 mon.a (mon.0) 1552 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T16:17:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:43 smithi123 bash[19119]: cluster 2024-02-12T16:17:42.503138+0000 mon.a (mon.0) 1553 : cluster [DBG] osdmap e415: 12 total, 11 up, 11 in 2024-02-12T16:17:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:43 smithi123 bash[19119]: cluster 2024-02-12T16:17:42.523203+0000 osd.4 (osd.4) 53 : cluster [DBG] 4.d scrub starts 2024-02-12T16:17:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:43 smithi123 bash[19119]: cluster 2024-02-12T16:17:42.558792+0000 osd.6 (osd.6) 54 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:17:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:43 smithi123 bash[19119]: cluster 2024-02-12T16:17:42.905479+0000 osd.6 (osd.6) 55 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:17:43.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:43.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:44.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:44.086 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:44.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:44.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:44.220 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:44.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:44.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:44.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:44.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:44.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:44.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:44.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:44.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:44.608 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:44.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:44.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:44.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:44 smithi169 bash[19604]: cluster 2024-02-12T16:17:43.005867+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.d scrub ok 2024-02-12T16:17:44.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:44 smithi169 bash[19604]: cluster 2024-02-12T16:17:43.480647+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:17:44.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:44 smithi169 bash[19604]: cluster 2024-02-12T16:17:43.508479+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in 2024-02-12T16:17:44.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:44 smithi169 bash[19604]: cluster 2024-02-12T16:17:43.513119+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:17:44.719 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:44 smithi169 bash[19604]: cluster 2024-02-12T16:17:43.753633+0000 mgr.y (mgr.14146) 973 : cluster [DBG] pgmap v1289: 37 pgs: 1 peering, 1 active+clean+scrubbing, 4 stale+active+clean, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 28 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 937 op/s; 6688/74366 objects misplaced (8.993%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:17:44.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:44.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:44.753 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:44 smithi121 bash[18489]: cluster 2024-02-12T16:17:43.005867+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.d scrub ok 2024-02-12T16:17:44.753 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:44 smithi121 bash[18489]: cluster 2024-02-12T16:17:43.480647+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:17:44.753 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:44 smithi121 bash[18489]: cluster 2024-02-12T16:17:43.508479+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in 2024-02-12T16:17:44.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:44 smithi121 bash[18489]: cluster 2024-02-12T16:17:43.513119+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:17:44.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:44 smithi121 bash[18489]: cluster 2024-02-12T16:17:43.753633+0000 mgr.y (mgr.14146) 973 : cluster [DBG] pgmap v1289: 37 pgs: 1 peering, 1 active+clean+scrubbing, 4 stale+active+clean, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 28 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 937 op/s; 6688/74366 objects misplaced (8.993%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:17:44.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:44.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:44 smithi123 bash[19119]: cluster 2024-02-12T16:17:43.005867+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.d scrub ok 2024-02-12T16:17:44.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:44 smithi123 bash[19119]: cluster 2024-02-12T16:17:43.480647+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:17:44.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:44 smithi123 bash[19119]: cluster 2024-02-12T16:17:43.508479+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e416: 12 total, 11 up, 11 in 2024-02-12T16:17:44.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:44 smithi123 bash[19119]: cluster 2024-02-12T16:17:43.513119+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:17:44.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:44 smithi123 bash[19119]: cluster 2024-02-12T16:17:43.753633+0000 mgr.y (mgr.14146) 973 : cluster [DBG] pgmap v1289: 37 pgs: 1 peering, 1 active+clean+scrubbing, 4 stale+active+clean, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 28 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 937 op/s; 6688/74366 objects misplaced (8.993%); 6.0 MiB/s, 96 objects/s recovering 2024-02-12T16:17:44.875 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:44.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:44.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:44.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:45.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:45.133 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:45.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:45.359 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:45.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:45.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:45.515 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:45.618 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:45.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:45 smithi123 bash[19119]: cluster 2024-02-12T16:17:44.685048+0000 osd.11 (osd.11) 63 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:17:45.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:45 smithi123 bash[19119]: cluster 2024-02-12T16:17:44.996997+0000 osd.11 (osd.11) 64 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:17:45.652 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:45.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:45.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:45.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:45.783 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:45 smithi169 bash[19604]: cluster 2024-02-12T16:17:44.685048+0000 osd.11 (osd.11) 63 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:17:45.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:45 smithi169 bash[19604]: cluster 2024-02-12T16:17:44.996997+0000 osd.11 (osd.11) 64 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:17:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:45.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:45.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:45 smithi121 bash[18489]: cluster 2024-02-12T16:17:44.685048+0000 osd.11 (osd.11) 63 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:17:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:45 smithi121 bash[18489]: cluster 2024-02-12T16:17:44.996997+0000 osd.11 (osd.11) 64 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:17:45.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:45.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:46.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:46.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:46.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:46.168 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:46.271 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:46.308 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:46.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:46.446 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:46.535 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: cluster 2024-02-12T16:17:45.655996+0000 osd.11 (osd.11) 65 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: audit 2024-02-12T16:17:45.797683+0000 mon.a (mon.0) 1555 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: audit 2024-02-12T16:17:45.797827+0000 mon.a (mon.0) 1556 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: audit 2024-02-12T16:17:45.797973+0000 mon.a (mon.0) 1557 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: audit 2024-02-12T16:17:45.835577+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: cluster 2024-02-12T16:17:45.985200+0000 osd.11 (osd.11) 66 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: audit 2024-02-12T16:17:46.159676+0000 mon.a (mon.0) 1559 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: audit 2024-02-12T16:17:46.160357+0000 mon.a (mon.0) 1560 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:17:46.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:46 smithi169 bash[19604]: audit 2024-02-12T16:17:46.165022+0000 mon.a (mon.0) 1561 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:17:46.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:46.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:46.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:46.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:46.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:46.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:46.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: cluster 2024-02-12T16:17:45.655996+0000 osd.11 (osd.11) 65 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:17:46.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: audit 2024-02-12T16:17:45.797683+0000 mon.a (mon.0) 1555 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:17:46.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: audit 2024-02-12T16:17:45.797827+0000 mon.a (mon.0) 1556 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:17:46.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: audit 2024-02-12T16:17:45.797973+0000 mon.a (mon.0) 1557 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:17:46.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: audit 2024-02-12T16:17:45.835577+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:17:46.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: cluster 2024-02-12T16:17:45.985200+0000 osd.11 (osd.11) 66 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:17:46.829 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: audit 2024-02-12T16:17:46.159676+0000 mon.a (mon.0) 1559 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:17:46.829 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: audit 2024-02-12T16:17:46.160357+0000 mon.a (mon.0) 1560 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:17:46.829 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:46 smithi123 bash[19119]: audit 2024-02-12T16:17:46.165022+0000 mon.a (mon.0) 1561 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:17:46.852 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:46.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: cluster 2024-02-12T16:17:45.655996+0000 osd.11 (osd.11) 65 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: audit 2024-02-12T16:17:45.797683+0000 mon.a (mon.0) 1555 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: audit 2024-02-12T16:17:45.797827+0000 mon.a (mon.0) 1556 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: audit 2024-02-12T16:17:45.797973+0000 mon.a (mon.0) 1557 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: audit 2024-02-12T16:17:45.835577+0000 mon.a (mon.0) 1558 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: cluster 2024-02-12T16:17:45.985200+0000 osd.11 (osd.11) 66 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: audit 2024-02-12T16:17:46.159676+0000 mon.a (mon.0) 1559 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: audit 2024-02-12T16:17:46.160357+0000 mon.a (mon.0) 1560 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:17:46.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:46 smithi121 bash[18489]: audit 2024-02-12T16:17:46.165022+0000 mon.a (mon.0) 1561 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:17:46.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:46.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:46.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:47.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:47.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:47.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:47.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:47.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:47.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:47.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:47.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:47.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:47.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:47.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:47.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:47.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:47.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:47 smithi169 bash[19604]: cluster 2024-02-12T16:17:46.520314+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:17:47.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:47 smithi169 bash[19604]: cluster 2024-02-12T16:17:46.520352+0000 mon.a (mon.0) 1563 : cluster [WRN] Health check failed: Degraded data redundancy: 6534/70400 objects degraded (9.281%), 3 pgs degraded (PG_DEGRADED) 2024-02-12T16:17:47.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:47.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:47.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:47.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:47.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:47.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:47 smithi123 bash[19119]: cluster 2024-02-12T16:17:46.520314+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:17:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:47 smithi123 bash[19119]: cluster 2024-02-12T16:17:46.520352+0000 mon.a (mon.0) 1563 : cluster [WRN] Health check failed: Degraded data redundancy: 6534/70400 objects degraded (9.281%), 3 pgs degraded (PG_DEGRADED) 2024-02-12T16:17:47.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:47.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:47.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:47 smithi121 bash[18489]: cluster 2024-02-12T16:17:46.520314+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:17:47.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:47 smithi121 bash[18489]: cluster 2024-02-12T16:17:46.520352+0000 mon.a (mon.0) 1563 : cluster [WRN] Health check failed: Degraded data redundancy: 6534/70400 objects degraded (9.281%), 3 pgs degraded (PG_DEGRADED) 2024-02-12T16:17:47.994 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:48.026 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:48.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:48.168 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:48.271 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:48.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:48.431 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:48.534 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:48.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:48.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.3.service' 2024-02-12T16:17:48.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:17:48.663 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-12T16:17:48.663 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-02-12T16:17:48.663 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi121.front.sepia.ceph.com 2024-02-12T16:17:48.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.3 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-02-12T16:17:48.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:48.680 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:48 smithi169 bash[19604]: cluster 2024-02-12T16:17:46.787513+0000 osd.10 (osd.10) 42 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:48.680 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:48 smithi169 bash[19604]: cluster 2024-02-12T16:17:46.787572+0000 osd.10 (osd.10) 43 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:48.680 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:48 smithi169 bash[19604]: cluster 2024-02-12T16:17:47.539616+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e417: 12 total, 11 up, 11 in 2024-02-12T16:17:48.680 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:48 smithi169 bash[19604]: cluster 2024-02-12T16:17:47.754480+0000 mgr.y (mgr.14146) 975 : cluster [DBG] pgmap v1292: 37 pgs: 1 active+clean+remapped, 3 undersized+degraded+peered, 3 undersized+peered, 26 active+clean, 2 active+remapped+backfilling, 1 active+clean+scrubbing, 1 peering; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 6534/70012 objects degraded (9.333%); 4891/70012 objects misplaced (6.986%); 19 MiB/s, 309 objects/s recovering 2024-02-12T16:17:48.696 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:48.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:48.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:48.850 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.3/config 2024-02-12T16:17:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:48 smithi123 bash[19119]: cluster 2024-02-12T16:17:46.787513+0000 osd.10 (osd.10) 42 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:48 smithi123 bash[19119]: cluster 2024-02-12T16:17:46.787572+0000 osd.10 (osd.10) 43 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:48 smithi123 bash[19119]: cluster 2024-02-12T16:17:47.539616+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e417: 12 total, 11 up, 11 in 2024-02-12T16:17:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:48 smithi123 bash[19119]: cluster 2024-02-12T16:17:47.754480+0000 mgr.y (mgr.14146) 975 : cluster [DBG] pgmap v1292: 37 pgs: 1 active+clean+remapped, 3 undersized+degraded+peered, 3 undersized+peered, 26 active+clean, 2 active+remapped+backfilling, 1 active+clean+scrubbing, 1 peering; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 6534/70012 objects degraded (9.333%); 4891/70012 objects misplaced (6.986%); 19 MiB/s, 309 objects/s recovering 2024-02-12T16:17:48.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:48.948 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:48.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:49.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:49.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:48 smithi121 bash[18489]: cluster 2024-02-12T16:17:46.787513+0000 osd.10 (osd.10) 42 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:49.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:48 smithi121 bash[18489]: cluster 2024-02-12T16:17:46.787572+0000 osd.10 (osd.10) 43 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:49.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:48 smithi121 bash[18489]: cluster 2024-02-12T16:17:47.539616+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e417: 12 total, 11 up, 11 in 2024-02-12T16:17:49.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:48 smithi121 bash[18489]: cluster 2024-02-12T16:17:47.754480+0000 mgr.y (mgr.14146) 975 : cluster [DBG] pgmap v1292: 37 pgs: 1 active+clean+remapped, 3 undersized+degraded+peered, 3 undersized+peered, 26 active+clean, 2 active+remapped+backfilling, 1 active+clean+scrubbing, 1 peering; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 6534/70012 objects degraded (9.333%); 4891/70012 objects misplaced (6.986%); 19 MiB/s, 309 objects/s recovering 2024-02-12T16:17:49.074 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:49.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:49.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:49.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:49.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:49.347 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:49.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:49.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:49.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:49.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:49.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:49.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:49.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:49.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:49.729 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.072174+0000 osd.9 (osd.9) 53 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:17:49.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.389731+0000 osd.9 (osd.9) 54 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:17:49.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.538011+0000 mon.a (mon.0) 1565 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:17:49.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.540929+0000 osd.7 (osd.7) 79 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:17:49.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.551661+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e418: 12 total, 11 up, 11 in 2024-02-12T16:17:49.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.599455+0000 osd.7 (osd.7) 80 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:17:49.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.639582+0000 osd.11 (osd.11) 67 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:17:49.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:49 smithi123 bash[19119]: cluster 2024-02-12T16:17:48.697001+0000 osd.11 (osd.11) 68 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:17:49.756 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:49.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:49.853 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.072174+0000 osd.9 (osd.9) 53 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:17:49.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.389731+0000 osd.9 (osd.9) 54 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:17:49.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.538011+0000 mon.a (mon.0) 1565 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:17:49.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.540929+0000 osd.7 (osd.7) 79 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:17:49.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.551661+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e418: 12 total, 11 up, 11 in 2024-02-12T16:17:49.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.599455+0000 osd.7 (osd.7) 80 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:17:49.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.639582+0000 osd.11 (osd.11) 67 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:17:49.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:49 smithi121 bash[18489]: cluster 2024-02-12T16:17:48.697001+0000 osd.11 (osd.11) 68 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:17:49.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:49.867 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.072174+0000 osd.9 (osd.9) 53 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:17:49.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.389731+0000 osd.9 (osd.9) 54 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:17:49.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.538011+0000 mon.a (mon.0) 1565 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:17:49.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.540929+0000 osd.7 (osd.7) 79 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:17:49.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.551661+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e418: 12 total, 11 up, 11 in 2024-02-12T16:17:49.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.599455+0000 osd.7 (osd.7) 80 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:17:49.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.639582+0000 osd.11 (osd.11) 67 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:17:49.868 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:49 smithi169 bash[19604]: cluster 2024-02-12T16:17:48.697001+0000 osd.11 (osd.11) 68 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:17:49.883 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:49.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:49.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:50.020 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:50.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:50.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:50.160 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:50.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:50.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:50.302 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:50.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:50.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:50.443 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:17:50.445 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:50.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:50.540 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:17:50.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:50.560 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:17:50.582 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:17:50.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:50.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:50.603 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:17:50.626 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:17:50.649 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:17:50.673 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:17:50.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:50.698 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:17:50.720 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:17:50.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:50.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:50.749 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:17:50.775 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:17:50.775 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.775 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.775 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:17:50.775 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.776 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.776 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.776 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.776 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.776 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.776 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.776 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:17:50.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: cluster 2024-02-12T16:17:49.073665+0000 osd.9 (osd.9) 55 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: cluster 2024-02-12T16:17:49.482158+0000 osd.9 (osd.9) 56 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: cluster 2024-02-12T16:17:49.558321+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e419: 12 total, 11 up, 11 in 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: cluster 2024-02-12T16:17:49.755214+0000 mgr.y (mgr.14146) 976 : cluster [DBG] pgmap v1295: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 27 active+clean, 2 active+remapped+backfilling; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 362 op/s; 8797/70012 objects degraded (12.565%); 3095/70012 objects misplaced (4.421%); 38 MiB/s, 608 objects/s recovering 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: audit 2024-02-12T16:17:49.908701+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? v1:172.21.15.121:0/170149098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: cluster 2024-02-12T16:17:50.428039+0000 mon.a (mon.0) 1569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: audit 2024-02-12T16:17:50.435403+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? v1:172.21.15.121:0/170149098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: cluster 2024-02-12T16:17:50.435513+0000 mon.a (mon.0) 1571 : cluster [DBG] osdmap e420: 12 total, 11 up, 11 in 2024-02-12T16:17:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:50 smithi169 bash[19604]: audit 2024-02-12T16:17:50.508098+0000 mon.a (mon.0) 1572 : audit [DBG] from='client.? v1:172.21.15.121:0/3621121543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: cluster 2024-02-12T16:17:49.073665+0000 osd.9 (osd.9) 55 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: cluster 2024-02-12T16:17:49.482158+0000 osd.9 (osd.9) 56 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: cluster 2024-02-12T16:17:49.558321+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e419: 12 total, 11 up, 11 in 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: cluster 2024-02-12T16:17:49.755214+0000 mgr.y (mgr.14146) 976 : cluster [DBG] pgmap v1295: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 27 active+clean, 2 active+remapped+backfilling; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 362 op/s; 8797/70012 objects degraded (12.565%); 3095/70012 objects misplaced (4.421%); 38 MiB/s, 608 objects/s recovering 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: audit 2024-02-12T16:17:49.908701+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? v1:172.21.15.121:0/170149098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: cluster 2024-02-12T16:17:50.428039+0000 mon.a (mon.0) 1569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: audit 2024-02-12T16:17:50.435403+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? v1:172.21.15.121:0/170149098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: cluster 2024-02-12T16:17:50.435513+0000 mon.a (mon.0) 1571 : cluster [DBG] osdmap e420: 12 total, 11 up, 11 in 2024-02-12T16:17:50.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:50 smithi121 bash[18489]: audit 2024-02-12T16:17:50.508098+0000 mon.a (mon.0) 1572 : audit [DBG] from='client.? v1:172.21.15.121:0/3621121543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:50.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: cluster 2024-02-12T16:17:49.073665+0000 osd.9 (osd.9) 55 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: cluster 2024-02-12T16:17:49.482158+0000 osd.9 (osd.9) 56 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: cluster 2024-02-12T16:17:49.558321+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e419: 12 total, 11 up, 11 in 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: cluster 2024-02-12T16:17:49.755214+0000 mgr.y (mgr.14146) 976 : cluster [DBG] pgmap v1295: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 27 active+clean, 2 active+remapped+backfilling; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 362 op/s; 8797/70012 objects degraded (12.565%); 3095/70012 objects misplaced (4.421%); 38 MiB/s, 608 objects/s recovering 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: audit 2024-02-12T16:17:49.908701+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? v1:172.21.15.121:0/170149098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: cluster 2024-02-12T16:17:50.428039+0000 mon.a (mon.0) 1569 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: audit 2024-02-12T16:17:50.435403+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? v1:172.21.15.121:0/170149098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: cluster 2024-02-12T16:17:50.435513+0000 mon.a (mon.0) 1571 : cluster [DBG] osdmap e420: 12 total, 11 up, 11 in 2024-02-12T16:17:50.840 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:50 smithi123 bash[19119]: audit 2024-02-12T16:17:50.508098+0000 mon.a (mon.0) 1572 : audit [DBG] from='client.? v1:172.21.15.121:0/3621121543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:17:50.859 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:50.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:50.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:51.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:51.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:51.143 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:51.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:51.282 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:51.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:51.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:51.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:51.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:51.677 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:51 smithi123 bash[19119]: cluster 2024-02-12T16:17:50.382840+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.e scrub starts 2024-02-12T16:17:51.677 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:51 smithi123 bash[19119]: cluster 2024-02-12T16:17:50.872993+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.e scrub ok 2024-02-12T16:17:51.677 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:51 smithi123 bash[19119]: cluster 2024-02-12T16:17:51.448941+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e421: 12 total, 11 up, 11 in 2024-02-12T16:17:51.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:51.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:51.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:51 smithi121 bash[18489]: cluster 2024-02-12T16:17:50.382840+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.e scrub starts 2024-02-12T16:17:51.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:51 smithi121 bash[18489]: cluster 2024-02-12T16:17:50.872993+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.e scrub ok 2024-02-12T16:17:51.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:51 smithi121 bash[18489]: cluster 2024-02-12T16:17:51.448941+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e421: 12 total, 11 up, 11 in 2024-02-12T16:17:51.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:51.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:51.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:51 smithi169 bash[19604]: cluster 2024-02-12T16:17:50.382840+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.e scrub starts 2024-02-12T16:17:51.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:51 smithi169 bash[19604]: cluster 2024-02-12T16:17:50.872993+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.e scrub ok 2024-02-12T16:17:51.954 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:51 smithi169 bash[19604]: cluster 2024-02-12T16:17:51.448941+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e421: 12 total, 11 up, 11 in 2024-02-12T16:17:51.976 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:51.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:52.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:52.111 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:52.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:52.215 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:52.254 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:52.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:17:52.337 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:17:52.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:52.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:52.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:52.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:52.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:52.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:52.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:52 smithi121 bash[18489]: cluster 2024-02-12T16:17:51.678292+0000 osd.11 (osd.11) 69 : cluster [DBG] 1.c scrub starts 2024-02-12T16:17:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:52 smithi121 bash[18489]: cluster 2024-02-12T16:17:51.755641+0000 osd.11 (osd.11) 70 : cluster [DBG] 1.c scrub ok 2024-02-12T16:17:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:52 smithi121 bash[18489]: cluster 2024-02-12T16:17:51.756015+0000 mgr.y (mgr.14146) 977 : cluster [DBG] pgmap v1298: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 27 active+clean, 2 active+remapped+backfilling; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 3095/70012 objects misplaced (4.421%); 28 MiB/s, 448 objects/s recovering 2024-02-12T16:17:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:52.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:52.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:52.815 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:52.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:52 smithi123 bash[19119]: cluster 2024-02-12T16:17:51.678292+0000 osd.11 (osd.11) 69 : cluster [DBG] 1.c scrub starts 2024-02-12T16:17:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:52 smithi123 bash[19119]: cluster 2024-02-12T16:17:51.755641+0000 osd.11 (osd.11) 70 : cluster [DBG] 1.c scrub ok 2024-02-12T16:17:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:52 smithi123 bash[19119]: cluster 2024-02-12T16:17:51.756015+0000 mgr.y (mgr.14146) 977 : cluster [DBG] pgmap v1298: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 27 active+clean, 2 active+remapped+backfilling; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 3095/70012 objects misplaced (4.421%); 28 MiB/s, 448 objects/s recovering 2024-02-12T16:17:52.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:52.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:52 smithi169 bash[19604]: cluster 2024-02-12T16:17:51.678292+0000 osd.11 (osd.11) 69 : cluster [DBG] 1.c scrub starts 2024-02-12T16:17:52.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:52 smithi169 bash[19604]: cluster 2024-02-12T16:17:51.755641+0000 osd.11 (osd.11) 70 : cluster [DBG] 1.c scrub ok 2024-02-12T16:17:52.932 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:52 smithi169 bash[19604]: cluster 2024-02-12T16:17:51.756015+0000 mgr.y (mgr.14146) 977 : cluster [DBG] pgmap v1298: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 27 active+clean, 2 active+remapped+backfilling; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 3095/70012 objects misplaced (4.421%); 28 MiB/s, 448 objects/s recovering 2024-02-12T16:17:52.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:52.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:53.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:53.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:53.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:53.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:53.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:53.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:53.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:53.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:53.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:17:53.520 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:53.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:53.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:53 smithi123 bash[19119]: cluster 2024-02-12T16:17:52.502214+0000 osd.6 (osd.6) 56 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:17:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:53 smithi123 bash[19119]: cluster 2024-02-12T16:17:52.527484+0000 osd.6 (osd.6) 57 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:17:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:53 smithi123 bash[19119]: cluster 2024-02-12T16:17:52.551847+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:17:53.637 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:53 smithi123 bash[19119]: cluster 2024-02-12T16:17:52.992882+0000 osd.2 (osd.2) 47 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:17:53.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:53.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:53.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:53.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:53 smithi121 bash[18489]: cluster 2024-02-12T16:17:52.502214+0000 osd.6 (osd.6) 56 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:17:53.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:53 smithi121 bash[18489]: cluster 2024-02-12T16:17:52.527484+0000 osd.6 (osd.6) 57 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:17:53.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:53 smithi121 bash[18489]: cluster 2024-02-12T16:17:52.551847+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:17:53.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:53 smithi121 bash[18489]: cluster 2024-02-12T16:17:52.992882+0000 osd.2 (osd.2) 47 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:17:53.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:53.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:53.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:53.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:53.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:53 smithi169 bash[19604]: cluster 2024-02-12T16:17:52.502214+0000 osd.6 (osd.6) 56 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:17:53.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:53 smithi169 bash[19604]: cluster 2024-02-12T16:17:52.527484+0000 osd.6 (osd.6) 57 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:17:53.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:53 smithi169 bash[19604]: cluster 2024-02-12T16:17:52.551847+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:17:53.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:53 smithi169 bash[19604]: cluster 2024-02-12T16:17:52.992882+0000 osd.2 (osd.2) 47 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:17:53.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:53.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:54.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:54.077 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:54.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:54.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:54.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:54.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:54.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:54.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:17:54.360 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:54.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:54.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:54.502 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:54.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:54.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:54.644 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:54.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:54.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:54.672 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:17:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:17:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:17:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:17:54.672 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:17:54.716 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:54 smithi121 bash[18489]: cluster 2024-02-12T16:17:53.503770+0000 osd.6 (osd.6) 58 : cluster [DBG] 1.f scrub starts 2024-02-12T16:17:54.716 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:54 smithi121 bash[18489]: cluster 2024-02-12T16:17:53.531196+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.f scrub ok 2024-02-12T16:17:54.716 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:54 smithi121 bash[18489]: cluster 2024-02-12T16:17:53.756818+0000 mgr.y (mgr.14146) 978 : cluster [DBG] pgmap v1299: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean, 1 active+remapped+backfilling; 2.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2619/70012 objects misplaced (3.741%); 27 MiB/s, 435 objects/s recovering 2024-02-12T16:17:54.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:54.784 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:54.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:54.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:54 smithi123 bash[19119]: cluster 2024-02-12T16:17:53.503770+0000 osd.6 (osd.6) 58 : cluster [DBG] 1.f scrub starts 2024-02-12T16:17:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:54 smithi123 bash[19119]: cluster 2024-02-12T16:17:53.531196+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.f scrub ok 2024-02-12T16:17:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:54 smithi123 bash[19119]: cluster 2024-02-12T16:17:53.756818+0000 mgr.y (mgr.14146) 978 : cluster [DBG] pgmap v1299: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean, 1 active+remapped+backfilling; 2.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2619/70012 objects misplaced (3.741%); 27 MiB/s, 435 objects/s recovering 2024-02-12T16:17:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:54 smithi169 bash[19604]: cluster 2024-02-12T16:17:53.503770+0000 osd.6 (osd.6) 58 : cluster [DBG] 1.f scrub starts 2024-02-12T16:17:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:54 smithi169 bash[19604]: cluster 2024-02-12T16:17:53.531196+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.f scrub ok 2024-02-12T16:17:54.901 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:54 smithi169 bash[19604]: cluster 2024-02-12T16:17:53.756818+0000 mgr.y (mgr.14146) 978 : cluster [DBG] pgmap v1299: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean, 1 active+remapped+backfilling; 2.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2619/70012 objects misplaced (3.741%); 27 MiB/s, 435 objects/s recovering 2024-02-12T16:17:54.928 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:54.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:55.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:55.068 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:55.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:55.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:55.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:55.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:55.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:55.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:55.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:55.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:17:55.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:55.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:55.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:55.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:55.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:55.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:55.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:55.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:55.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:55.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:55.870 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:55 smithi123 bash[19119]: cluster 2024-02-12T16:17:54.820876+0000 osd.5 (osd.5) 57 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:17:55.870 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:55 smithi123 bash[19119]: cluster 2024-02-12T16:17:54.863743+0000 osd.5 (osd.5) 58 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:17:55.870 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:55 smithi123 bash[19119]: cluster 2024-02-12T16:17:55.429682+0000 mon.a (mon.0) 1574 : cluster [WRN] Health check update: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded (PG_DEGRADED) 2024-02-12T16:17:55.870 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:55 smithi123 bash[19119]: cluster 2024-02-12T16:17:55.440440+0000 mon.a (mon.0) 1575 : cluster [DBG] osdmap e422: 12 total, 11 up, 11 in 2024-02-12T16:17:55.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:55.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:55.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:55 smithi121 bash[18489]: cluster 2024-02-12T16:17:54.820876+0000 osd.5 (osd.5) 57 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:17:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:55 smithi121 bash[18489]: cluster 2024-02-12T16:17:54.863743+0000 osd.5 (osd.5) 58 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:17:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:55 smithi121 bash[18489]: cluster 2024-02-12T16:17:55.429682+0000 mon.a (mon.0) 1574 : cluster [WRN] Health check update: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded (PG_DEGRADED) 2024-02-12T16:17:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:55 smithi121 bash[18489]: cluster 2024-02-12T16:17:55.440440+0000 mon.a (mon.0) 1575 : cluster [DBG] osdmap e422: 12 total, 11 up, 11 in 2024-02-12T16:17:55.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:56.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:55 smithi169 bash[19604]: cluster 2024-02-12T16:17:54.820876+0000 osd.5 (osd.5) 57 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:17:56.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:55 smithi169 bash[19604]: cluster 2024-02-12T16:17:54.863743+0000 osd.5 (osd.5) 58 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:17:56.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:55 smithi169 bash[19604]: cluster 2024-02-12T16:17:55.429682+0000 mon.a (mon.0) 1574 : cluster [WRN] Health check update: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded (PG_DEGRADED) 2024-02-12T16:17:56.009 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:55 smithi169 bash[19604]: cluster 2024-02-12T16:17:55.440440+0000 mon.a (mon.0) 1575 : cluster [DBG] osdmap e422: 12 total, 11 up, 11 in 2024-02-12T16:17:56.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:56.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:56.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:56.175 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:56.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:56.279 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:56.315 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:56.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:56.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:56.454 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:17:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:56.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:17:56.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:56.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:56.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:56.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:56.733 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:56.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:56.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:56 smithi169 bash[19604]: cluster 2024-02-12T16:17:55.757556+0000 mgr.y (mgr.14146) 979 : cluster [DBG] pgmap v1301: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 8.8 MiB/s, 141 objects/s recovering 2024-02-12T16:17:56.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:56.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:56.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:56 smithi123 bash[19119]: cluster 2024-02-12T16:17:55.757556+0000 mgr.y (mgr.14146) 979 : cluster [DBG] pgmap v1301: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 8.8 MiB/s, 141 objects/s recovering 2024-02-12T16:17:56.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:56 smithi121 bash[18489]: cluster 2024-02-12T16:17:55.757556+0000 mgr.y (mgr.14146) 979 : cluster [DBG] pgmap v1301: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 8.8 MiB/s, 141 objects/s recovering 2024-02-12T16:17:56.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:57.014 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:57.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:57.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:57.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:57.288 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:57.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:57.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:57.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:57.667 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:57.693 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:17:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:17:57.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:57 smithi169 bash[19604]: cluster 2024-02-12T16:17:56.028090+0000 osd.10 (osd.10) 44 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:17:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:57 smithi169 bash[19604]: cluster 2024-02-12T16:17:56.042420+0000 osd.10 (osd.10) 45 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:17:57.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:57 smithi169 bash[19604]: cluster 2024-02-12T16:17:56.650322+0000 osd.11 (osd.11) 71 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:17:57.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:57 smithi169 bash[19604]: cluster 2024-02-12T16:17:56.726731+0000 osd.11 (osd.11) 72 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:17:57.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.840 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:57 smithi123 bash[19119]: cluster 2024-02-12T16:17:56.028090+0000 osd.10 (osd.10) 44 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:17:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:57 smithi123 bash[19119]: cluster 2024-02-12T16:17:56.042420+0000 osd.10 (osd.10) 45 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:17:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:57 smithi123 bash[19119]: cluster 2024-02-12T16:17:56.650322+0000 osd.11 (osd.11) 71 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:17:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:57 smithi123 bash[19119]: cluster 2024-02-12T16:17:56.726731+0000 osd.11 (osd.11) 72 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:17:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:57 smithi121 bash[18489]: cluster 2024-02-12T16:17:56.028090+0000 osd.10 (osd.10) 44 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:17:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:57 smithi121 bash[18489]: cluster 2024-02-12T16:17:56.042420+0000 osd.10 (osd.10) 45 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:17:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:57 smithi121 bash[18489]: cluster 2024-02-12T16:17:56.650322+0000 osd.11 (osd.11) 71 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:17:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:57 smithi121 bash[18489]: cluster 2024-02-12T16:17:56.726731+0000 osd.11 (osd.11) 72 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:17:57.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:57.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:57.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:58.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:58.116 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:58.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:58.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:58.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:17:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:58.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:17:58.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:58.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:58.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:58.530 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:17:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:58.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:17:58.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:58.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:58 smithi121 bash[18489]: cluster 2024-02-12T16:17:57.663407+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:58.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:58 smithi121 bash[18489]: cluster 2024-02-12T16:17:57.663495+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:58.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:58 smithi121 bash[18489]: cluster 2024-02-12T16:17:57.758091+0000 mgr.y (mgr.14146) 980 : cluster [DBG] pgmap v1302: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 7.2 MiB/s, 115 objects/s recovering 2024-02-12T16:17:58.672 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:17:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:58.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:17:58.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:58.806 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:17:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:58.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:17:58.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:58 smithi123 bash[19119]: cluster 2024-02-12T16:17:57.663407+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:58.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:58 smithi123 bash[19119]: cluster 2024-02-12T16:17:57.663495+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:58.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:58 smithi123 bash[19119]: cluster 2024-02-12T16:17:57.758091+0000 mgr.y (mgr.14146) 980 : cluster [DBG] pgmap v1302: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 7.2 MiB/s, 115 objects/s recovering 2024-02-12T16:17:58.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:58.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:58 smithi169 bash[19604]: cluster 2024-02-12T16:17:57.663407+0000 osd.4 (osd.4) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:17:58.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:58 smithi169 bash[19604]: cluster 2024-02-12T16:17:57.663495+0000 osd.4 (osd.4) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:17:58.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:58 smithi169 bash[19604]: cluster 2024-02-12T16:17:57.758091+0000 mgr.y (mgr.14146) 980 : cluster [DBG] pgmap v1302: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 7.2 MiB/s, 115 objects/s recovering 2024-02-12T16:17:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:58.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:59.016 INFO:teuthology.orchestra.run.smithi121.stdout:4.f 2024-02-12T16:17:59.016 INFO:teuthology.orchestra.run.smithi121.stdout:4.c 2024-02-12T16:17:59.017 INFO:teuthology.orchestra.run.smithi121.stdout:1.3 2024-02-12T16:17:59.017 INFO:teuthology.orchestra.run.smithi121.stdout:1.13 2024-02-12T16:17:59.017 INFO:teuthology.orchestra.run.smithi121.stdout:1.b 2024-02-12T16:17:59.017 INFO:teuthology.orchestra.run.smithi121.stdout:4.0 2024-02-12T16:17:59.017 INFO:teuthology.orchestra.run.smithi121.stdout:4.4 2024-02-12T16:17:59.017 INFO:teuthology.orchestra.run.smithi121.stdout:4.9 2024-02-12T16:17:59.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:59.088 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:59.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:59.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:59.226 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:59.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:59.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:59.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:17:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:59.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:17:59.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:59.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:17:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:59.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:17:59.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:59.645 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:17:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:59.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:17:59.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:59.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:17:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:59.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:17:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:17:59 smithi123 bash[19119]: cluster 2024-02-12T16:17:58.608108+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e423: 12 total, 11 up, 11 in 2024-02-12T16:17:59.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:59.902 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:17:59 smithi121 bash[18489]: cluster 2024-02-12T16:17:58.608108+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e423: 12 total, 11 up, 11 in 2024-02-12T16:17:59.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:17:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:17:59.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:17:59.977 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:59.978 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:17:59.978 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:00.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:17:59 smithi169 bash[19604]: cluster 2024-02-12T16:17:58.608108+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e423: 12 total, 11 up, 11 in 2024-02-12T16:18:00.067 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:00.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:00.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:00.208 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:00.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:00.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:00.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:00.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:00.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:00.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:00.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:00.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:00.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:00.628 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:00.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:00.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:00.772 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:00.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:00.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:00 smithi121 bash[18489]: cluster 2024-02-12T16:17:59.083900+0000 osd.10 (osd.10) 46 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:18:00.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:00 smithi121 bash[18489]: cluster 2024-02-12T16:17:59.106908+0000 osd.10 (osd.10) 47 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:18:00.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:00 smithi121 bash[18489]: cluster 2024-02-12T16:17:59.758891+0000 mgr.y (mgr.14146) 981 : cluster [DBG] pgmap v1304: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 6.6 MiB/s, 105 objects/s recovering 2024-02-12T16:18:00.823 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:00 smithi121 bash[18489]: cluster 2024-02-12T16:18:00.442425+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e424: 12 total, 11 up, 11 in 2024-02-12T16:18:00.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.3 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 4.f --file /var/log/ceph/exp.4.f.3 2024-02-12T16:18:00.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:00.888 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:00 smithi169 bash[19604]: cluster 2024-02-12T16:17:59.083900+0000 osd.10 (osd.10) 46 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:18:00.888 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:00 smithi169 bash[19604]: cluster 2024-02-12T16:17:59.106908+0000 osd.10 (osd.10) 47 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:18:00.888 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:00 smithi169 bash[19604]: cluster 2024-02-12T16:17:59.758891+0000 mgr.y (mgr.14146) 981 : cluster [DBG] pgmap v1304: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 6.6 MiB/s, 105 objects/s recovering 2024-02-12T16:18:00.888 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:00 smithi169 bash[19604]: cluster 2024-02-12T16:18:00.442425+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e424: 12 total, 11 up, 11 in 2024-02-12T16:18:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:00 smithi123 bash[19119]: cluster 2024-02-12T16:17:59.083900+0000 osd.10 (osd.10) 46 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:18:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:00 smithi123 bash[19119]: cluster 2024-02-12T16:17:59.106908+0000 osd.10 (osd.10) 47 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:18:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:00 smithi123 bash[19119]: cluster 2024-02-12T16:17:59.758891+0000 mgr.y (mgr.14146) 981 : cluster [DBG] pgmap v1304: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%); 6.6 MiB/s, 105 objects/s recovering 2024-02-12T16:18:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:00 smithi123 bash[19119]: cluster 2024-02-12T16:18:00.442425+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e424: 12 total, 11 up, 11 in 2024-02-12T16:18:00.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:00.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.3/config 2024-02-12T16:18:01.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:01.053 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:01.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:01.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:01.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:01.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:01.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:01.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:01.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:01.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:01.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:01.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:01.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:01.626 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:01.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:01.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:01.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:01.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:01.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:01.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:01 smithi169 bash[19604]: cluster 2024-02-12T16:18:00.109328+0000 osd.10 (osd.10) 48 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:18:01.877 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:01 smithi169 bash[19604]: cluster 2024-02-12T16:18:00.122689+0000 osd.10 (osd.10) 49 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:18:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:01 smithi123 bash[19119]: cluster 2024-02-12T16:18:00.109328+0000 osd.10 (osd.10) 48 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:18:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:01 smithi123 bash[19119]: cluster 2024-02-12T16:18:00.122689+0000 osd.10 (osd.10) 49 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:18:01.903 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:01.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:01.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:01 smithi121 bash[18489]: cluster 2024-02-12T16:18:00.109328+0000 osd.10 (osd.10) 48 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:18:01.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:01 smithi121 bash[18489]: cluster 2024-02-12T16:18:00.122689+0000 osd.10 (osd.10) 49 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:18:02.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:02.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:02.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:02.190 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:02.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:02.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:02.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:02.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:02.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:02.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:02.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:02.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:02.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:02.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:02.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:02.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:02.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:02.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:02.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:02 smithi123 bash[19119]: cluster 2024-02-12T16:18:01.759528+0000 mgr.y (mgr.14146) 982 : cluster [DBG] pgmap v1306: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:02.872 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:02.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:02 smithi121 bash[18489]: cluster 2024-02-12T16:18:01.759528+0000 mgr.y (mgr.14146) 982 : cluster [DBG] pgmap v1306: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:02.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:03.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:03.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:03.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:02 smithi169 bash[19604]: cluster 2024-02-12T16:18:01.759528+0000 mgr.y (mgr.14146) 982 : cluster [DBG] pgmap v1306: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:03.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:03.153 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:03.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:03.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:03.296 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:03.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:03.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:03.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:03.439 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:03.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:03.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:03.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:03.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:03.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:03.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:03.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:03.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:03.835 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:03 smithi123 bash[19119]: cluster 2024-02-12T16:18:02.543329+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.a scrub starts 2024-02-12T16:18:03.835 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:03 smithi123 bash[19119]: cluster 2024-02-12T16:18:02.941987+0000 osd.6 (osd.6) 61 : cluster [DBG] 4.a scrub ok 2024-02-12T16:18:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:03.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:03.922 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:03 smithi121 bash[18489]: cluster 2024-02-12T16:18:02.543329+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.a scrub starts 2024-02-12T16:18:03.923 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:03 smithi121 bash[18489]: cluster 2024-02-12T16:18:02.941987+0000 osd.6 (osd.6) 61 : cluster [DBG] 4.a scrub ok 2024-02-12T16:18:03.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:04.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:04.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:03 smithi169 bash[19604]: cluster 2024-02-12T16:18:02.543329+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.a scrub starts 2024-02-12T16:18:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:03 smithi169 bash[19604]: cluster 2024-02-12T16:18:02.941987+0000 osd.6 (osd.6) 61 : cluster [DBG] 4.a scrub ok 2024-02-12T16:18:04.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:04.147 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:04.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:04.291 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:04.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:04.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:04.537 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:04.576 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:04 smithi121 bash[18489]: cluster 2024-02-12T16:18:03.105901+0000 osd.10 (osd.10) 50 : cluster [DBG] 1.d scrub starts 2024-02-12T16:18:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:04 smithi121 bash[18489]: cluster 2024-02-12T16:18:03.127727+0000 osd.10 (osd.10) 51 : cluster [DBG] 1.d scrub ok 2024-02-12T16:18:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:04 smithi121 bash[18489]: cluster 2024-02-12T16:18:03.504541+0000 osd.8 (osd.8) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:04 smithi121 bash[18489]: cluster 2024-02-12T16:18:03.504633+0000 osd.8 (osd.8) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:04 smithi121 bash[18489]: cluster 2024-02-12T16:18:03.633707+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e425: 12 total, 11 up, 11 in 2024-02-12T16:18:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:04 smithi121 bash[18489]: cluster 2024-02-12T16:18:03.760329+0000 mgr.y (mgr.14146) 983 : cluster [DBG] pgmap v1308: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:04.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:04.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:04.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:04.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:04 smithi123 bash[19119]: cluster 2024-02-12T16:18:03.105901+0000 osd.10 (osd.10) 50 : cluster [DBG] 1.d scrub starts 2024-02-12T16:18:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:04 smithi123 bash[19119]: cluster 2024-02-12T16:18:03.127727+0000 osd.10 (osd.10) 51 : cluster [DBG] 1.d scrub ok 2024-02-12T16:18:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:04 smithi123 bash[19119]: cluster 2024-02-12T16:18:03.504541+0000 osd.8 (osd.8) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:04 smithi123 bash[19119]: cluster 2024-02-12T16:18:03.504633+0000 osd.8 (osd.8) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:04 smithi123 bash[19119]: cluster 2024-02-12T16:18:03.633707+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e425: 12 total, 11 up, 11 in 2024-02-12T16:18:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:04 smithi123 bash[19119]: cluster 2024-02-12T16:18:03.760329+0000 mgr.y (mgr.14146) 983 : cluster [DBG] pgmap v1308: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:04.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:04.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:04.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:05.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:04 smithi169 bash[19604]: cluster 2024-02-12T16:18:03.105901+0000 osd.10 (osd.10) 50 : cluster [DBG] 1.d scrub starts 2024-02-12T16:18:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:04 smithi169 bash[19604]: cluster 2024-02-12T16:18:03.127727+0000 osd.10 (osd.10) 51 : cluster [DBG] 1.d scrub ok 2024-02-12T16:18:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:04 smithi169 bash[19604]: cluster 2024-02-12T16:18:03.504541+0000 osd.8 (osd.8) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:04 smithi169 bash[19604]: cluster 2024-02-12T16:18:03.504633+0000 osd.8 (osd.8) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:04 smithi169 bash[19604]: cluster 2024-02-12T16:18:03.633707+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e425: 12 total, 11 up, 11 in 2024-02-12T16:18:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:04 smithi169 bash[19604]: cluster 2024-02-12T16:18:03.760329+0000 mgr.y (mgr.14146) 983 : cluster [DBG] pgmap v1308: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:05.095 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:05.128 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:05.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:05.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:05.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:05.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:05.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:05.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:05.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:05.413 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:05.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:05.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:05.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:05.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:05.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:05.692 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:05.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:05.795 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:05.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:05.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:05 smithi121 bash[18489]: cluster 2024-02-12T16:18:04.083672+0000 osd.10 (osd.10) 52 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:18:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:05 smithi121 bash[18489]: cluster 2024-02-12T16:18:04.114023+0000 osd.10 (osd.10) 53 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:18:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:05 smithi121 bash[18489]: cluster 2024-02-12T16:18:05.441259+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e426: 12 total, 11 up, 11 in 2024-02-12T16:18:05.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:05.977 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:05.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:06.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:05 smithi169 bash[19604]: cluster 2024-02-12T16:18:04.083672+0000 osd.10 (osd.10) 52 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:18:06.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:05 smithi169 bash[19604]: cluster 2024-02-12T16:18:04.114023+0000 osd.10 (osd.10) 53 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:18:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:05 smithi169 bash[19604]: cluster 2024-02-12T16:18:05.441259+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e426: 12 total, 11 up, 11 in 2024-02-12T16:18:06.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:06.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:06.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:05 smithi123 bash[19119]: cluster 2024-02-12T16:18:04.083672+0000 osd.10 (osd.10) 52 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:18:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:05 smithi123 bash[19119]: cluster 2024-02-12T16:18:04.114023+0000 osd.10 (osd.10) 53 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:18:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:05 smithi123 bash[19119]: cluster 2024-02-12T16:18:05.441259+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e426: 12 total, 11 up, 11 in 2024-02-12T16:18:06.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:06.255 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:06.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:06.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:06.396 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:06.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:06.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:06.539 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:06.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:06.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:06.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:06.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:06.779 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:06.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:06.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:06.874 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:18:06.896 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:18:06.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:06 smithi121 bash[18489]: cluster 2024-02-12T16:18:05.073943+0000 osd.9 (osd.9) 57 : cluster [DBG] 1.a scrub starts 2024-02-12T16:18:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:06 smithi121 bash[18489]: cluster 2024-02-12T16:18:05.090499+0000 osd.9 (osd.9) 58 : cluster [DBG] 1.a scrub ok 2024-02-12T16:18:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:06 smithi121 bash[18489]: cluster 2024-02-12T16:18:05.761177+0000 mgr.y (mgr.14146) 984 : cluster [DBG] pgmap v1310: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:06.916 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:18:06.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:06.939 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:18:06.954 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:06.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:06.964 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:18:06.987 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:18:07.010 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:18:07.039 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:18:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:06 smithi169 bash[19604]: cluster 2024-02-12T16:18:05.073943+0000 osd.9 (osd.9) 57 : cluster [DBG] 1.a scrub starts 2024-02-12T16:18:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:06 smithi169 bash[19604]: cluster 2024-02-12T16:18:05.090499+0000 osd.9 (osd.9) 58 : cluster [DBG] 1.a scrub ok 2024-02-12T16:18:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:06 smithi169 bash[19604]: cluster 2024-02-12T16:18:05.761177+0000 mgr.y (mgr.14146) 984 : cluster [DBG] pgmap v1310: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:07.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:07.062 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:18:07.090 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:18:07.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:07.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:07.114 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:18:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.114 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.115 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:18:07.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:06 smithi123 bash[19119]: cluster 2024-02-12T16:18:05.073943+0000 osd.9 (osd.9) 57 : cluster [DBG] 1.a scrub starts 2024-02-12T16:18:07.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:06 smithi123 bash[19119]: cluster 2024-02-12T16:18:05.090499+0000 osd.9 (osd.9) 58 : cluster [DBG] 1.a scrub ok 2024-02-12T16:18:07.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:06 smithi123 bash[19119]: cluster 2024-02-12T16:18:05.761177+0000 mgr.y (mgr.14146) 984 : cluster [DBG] pgmap v1310: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:07.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:07.235 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:07.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:07.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:07.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:07.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:07.403 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:18:07.404 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:18:07.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:07.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:07.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:07.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:07.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:07.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:07.776 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:07.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:07.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:07.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:07 smithi121 bash[18489]: cluster 2024-02-12T16:18:06.126229+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.e scrub starts 2024-02-12T16:18:07.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:07 smithi121 bash[18489]: cluster 2024-02-12T16:18:06.137330+0000 osd.10 (osd.10) 55 : cluster [DBG] 1.e scrub ok 2024-02-12T16:18:07.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:07 smithi121 bash[18489]: audit 2024-02-12T16:18:06.844719+0000 mon.a (mon.0) 1580 : audit [DBG] from='client.? v1:172.21.15.121:0/3005298046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:07.916 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:07.919 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:07 smithi123 bash[19119]: cluster 2024-02-12T16:18:06.126229+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.e scrub starts 2024-02-12T16:18:07.919 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:07 smithi123 bash[19119]: cluster 2024-02-12T16:18:06.137330+0000 osd.10 (osd.10) 55 : cluster [DBG] 1.e scrub ok 2024-02-12T16:18:07.919 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:07 smithi123 bash[19119]: audit 2024-02-12T16:18:06.844719+0000 mon.a (mon.0) 1580 : audit [DBG] from='client.? v1:172.21.15.121:0/3005298046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:07.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:08.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:08.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:07 smithi169 bash[19604]: cluster 2024-02-12T16:18:06.126229+0000 osd.10 (osd.10) 54 : cluster [DBG] 1.e scrub starts 2024-02-12T16:18:08.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:07 smithi169 bash[19604]: cluster 2024-02-12T16:18:06.137330+0000 osd.10 (osd.10) 55 : cluster [DBG] 1.e scrub ok 2024-02-12T16:18:08.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:07 smithi169 bash[19604]: audit 2024-02-12T16:18:06.844719+0000 mon.a (mon.0) 1580 : audit [DBG] from='client.? v1:172.21.15.121:0/3005298046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:08.057 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:08.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:08.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:08.198 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:08.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:08.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:08.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:08.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:08.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:08.481 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:08.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:08.505 INFO:teuthology.orchestra.run.smithi121.stderr:Exporting 4.f info 4.f( v 414'19284 (360'9200,414'19284] local-lis/les=404/405 n=2124 ec=334/334 lis/c=404/404 les/c/f=405/405/0 sis=404) 2024-02-12T16:18:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:18:08.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:18:08.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f003003b:::benchmark_data_smithi169_60938_object153827:head# 2024-02-12T16:18:08.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0044b9b:::benchmark_data_smithi169_60938_object155923:head# 2024-02-12T16:18:08.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0068bc1:::benchmark_data_smithi169_60938_object152279:head# 2024-02-12T16:18:08.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0068e0b:::benchmark_data_smithi169_60938_object151323:head# 2024-02-12T16:18:08.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0083b71:::benchmark_data_smithi169_60938_object155111:head# 2024-02-12T16:18:08.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f008be54:::benchmark_data_smithi169_60938_object147409:head# 2024-02-12T16:18:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f009df6b:::benchmark_data_smithi169_60938_object147882:head# 2024-02-12T16:18:08.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f00a816e:::benchmark_data_smithi169_60938_object152564:head# 2024-02-12T16:18:08.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f00ca4cf:::benchmark_data_smithi169_60938_object162655:head# 2024-02-12T16:18:08.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f010295a:::benchmark_data_smithi169_60938_object165661:head# 2024-02-12T16:18:08.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0129d45:::benchmark_data_smithi169_60938_object145851:head# 2024-02-12T16:18:08.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f018f139:::benchmark_data_smithi169_60938_object142957:head# 2024-02-12T16:18:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f01a5829:::benchmark_data_smithi169_60938_object147023:head# 2024-02-12T16:18:08.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f01d36ce:::benchmark_data_smithi169_60938_object140404:head# 2024-02-12T16:18:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f01fa2e7:::benchmark_data_smithi169_60938_object143275:head# 2024-02-12T16:18:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0201f7d:::benchmark_data_smithi169_60938_object158786:head# 2024-02-12T16:18:08.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f021b06e:::benchmark_data_smithi169_60938_object168589:head# 2024-02-12T16:18:08.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f024bd70:::benchmark_data_smithi169_60938_object156768:head# 2024-02-12T16:18:08.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f024f4f9:::benchmark_data_smithi169_60938_object148207:head# 2024-02-12T16:18:08.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f026d980:::benchmark_data_smithi169_60938_object154657:head# 2024-02-12T16:18:08.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f02e6e58:::benchmark_data_smithi169_60938_object148079:head# 2024-02-12T16:18:08.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f02ecca3:::benchmark_data_smithi169_60938_object165266:head# 2024-02-12T16:18:08.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0301c50:::benchmark_data_smithi169_60938_object171622:head# 2024-02-12T16:18:08.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f032f862:::benchmark_data_smithi169_60938_object163958:head# 2024-02-12T16:18:08.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0346170:::benchmark_data_smithi169_60938_object143545:head# 2024-02-12T16:18:08.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0367c8d:::benchmark_data_smithi169_60938_object162104:head# 2024-02-12T16:18:08.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f039c058:::benchmark_data_smithi169_60938_object154386:head# 2024-02-12T16:18:08.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f039d353:::benchmark_data_smithi169_60938_object163634:head# 2024-02-12T16:18:08.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f039e488:::benchmark_data_smithi169_60938_object144596:head# 2024-02-12T16:18:08.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f041378b:::benchmark_data_smithi169_60938_object163331:head# 2024-02-12T16:18:08.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0415d7d:::benchmark_data_smithi169_60938_object172830:head# 2024-02-12T16:18:08.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f041c8b5:::benchmark_data_smithi169_60938_object163464:head# 2024-02-12T16:18:08.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0443db3:::benchmark_data_smithi169_60938_object161483:head# 2024-02-12T16:18:08.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f044de42:::benchmark_data_smithi169_60938_object140312:head# 2024-02-12T16:18:08.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f046acbb:::benchmark_data_smithi169_60938_object165288:head# 2024-02-12T16:18:08.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f04d0aa7:::benchmark_data_smithi169_60938_object155364:head# 2024-02-12T16:18:08.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f04f4541:::benchmark_data_smithi169_60938_object172233:head# 2024-02-12T16:18:08.584 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:08.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f050406c:::benchmark_data_smithi169_60938_object138280:head# 2024-02-12T16:18:08.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f05197f3:::benchmark_data_smithi169_60938_object168928:head# 2024-02-12T16:18:08.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f05198a2:::benchmark_data_smithi169_60938_object141643:head# 2024-02-12T16:18:08.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0527369:::benchmark_data_smithi169_60938_object163356:head# 2024-02-12T16:18:08.586 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f053e107:::benchmark_data_smithi169_60938_object158346:head# 2024-02-12T16:18:08.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f053f03f:::benchmark_data_smithi169_60938_object142666:head# 2024-02-12T16:18:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0566f69:::benchmark_data_smithi169_60938_object157921:head# 2024-02-12T16:18:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0575171:::benchmark_data_smithi169_60938_object160344:head# 2024-02-12T16:18:08.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f05a276e:::benchmark_data_smithi169_60938_object141577:head# 2024-02-12T16:18:08.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f05cb2f7:::benchmark_data_smithi169_60938_object143821:head# 2024-02-12T16:18:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f05ce178:::benchmark_data_smithi169_60938_object158565:head# 2024-02-12T16:18:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f05e372c:::benchmark_data_smithi169_60938_object159360:head# 2024-02-12T16:18:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f05e6c57:::benchmark_data_smithi169_60938_object169966:head# 2024-02-12T16:18:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0602705:::benchmark_data_smithi169_60938_object156283:head# 2024-02-12T16:18:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f06178ae:::benchmark_data_smithi169_60938_object154870:head# 2024-02-12T16:18:08.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0618ec6:::benchmark_data_smithi169_60938_object154048:head# 2024-02-12T16:18:08.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f062be84:::benchmark_data_smithi169_60938_object146679:head# 2024-02-12T16:18:08.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f064c4ce:::benchmark_data_smithi169_60938_object165992:head# 2024-02-12T16:18:08.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f068f934:::benchmark_data_smithi169_60938_object164932:head# 2024-02-12T16:18:08.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f06b35c8:::benchmark_data_smithi169_60938_object164407:head# 2024-02-12T16:18:08.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0762ef8:::benchmark_data_smithi169_60938_object163482:head# 2024-02-12T16:18:08.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f07899e2:::benchmark_data_smithi169_60938_object138350:head# 2024-02-12T16:18:08.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f07ac115:::benchmark_data_smithi169_60938_object171883:head# 2024-02-12T16:18:08.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f07c8c74:::benchmark_data_smithi169_60938_object154180:head# 2024-02-12T16:18:08.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f07d54aa:::benchmark_data_smithi169_60938_object160232:head# 2024-02-12T16:18:08.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f07e0c30:::benchmark_data_smithi169_60938_object159238:head# 2024-02-12T16:18:08.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f07e49cc:::benchmark_data_smithi169_60938_object164042:head# 2024-02-12T16:18:08.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f07e6d7e:::benchmark_data_smithi169_60938_object170388:head# 2024-02-12T16:18:08.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f080c861:::benchmark_data_smithi169_60938_object171850:head# 2024-02-12T16:18:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f08160fa:::benchmark_data_smithi169_60938_object167769:head# 2024-02-12T16:18:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f083d7ed:::benchmark_data_smithi169_60938_object145607:head# 2024-02-12T16:18:08.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f084cd89:::benchmark_data_smithi169_60938_object170229:head# 2024-02-12T16:18:08.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f088b0a7:::benchmark_data_smithi169_60938_object172464:head# 2024-02-12T16:18:08.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f08b0d6b:::benchmark_data_smithi169_60938_object150853:head# 2024-02-12T16:18:08.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0909054:::benchmark_data_smithi169_60938_object171324:head# 2024-02-12T16:18:08.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0913974:::benchmark_data_smithi169_60938_object148111:head# 2024-02-12T16:18:08.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f09777de:::benchmark_data_smithi169_60938_object158383:head# 2024-02-12T16:18:08.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f098d2fa:::benchmark_data_smithi169_60938_object162607:head# 2024-02-12T16:18:08.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f09a43b1:::benchmark_data_smithi169_60938_object168114:head# 2024-02-12T16:18:08.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f09a89e5:::benchmark_data_smithi169_60938_object158204:head# 2024-02-12T16:18:08.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f09dc608:::benchmark_data_smithi169_60938_object148173:head# 2024-02-12T16:18:08.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f09e4cd6:::benchmark_data_smithi169_60938_object150794:head# 2024-02-12T16:18:08.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f09f686b:::benchmark_data_smithi169_60938_object141531:head# 2024-02-12T16:18:08.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0a13c66:::benchmark_data_smithi169_60938_object142625:head# 2024-02-12T16:18:08.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0a1aeaf:::benchmark_data_smithi169_60938_object172359:head# 2024-02-12T16:18:08.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0a3029c:::benchmark_data_smithi169_60938_object149518:head# 2024-02-12T16:18:08.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0a66b8b:::benchmark_data_smithi169_60938_object163727:head# 2024-02-12T16:18:08.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0a74dd6:::benchmark_data_smithi169_60938_object153460:head# 2024-02-12T16:18:08.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0a8c752:::benchmark_data_smithi169_60938_object156457:head# 2024-02-12T16:18:08.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0a9d431:::benchmark_data_smithi169_60938_object155142:head# 2024-02-12T16:18:08.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0ab57f4:::benchmark_data_smithi169_60938_object156516:head# 2024-02-12T16:18:08.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0ab86cb:::benchmark_data_smithi169_60938_object157074:head# 2024-02-12T16:18:08.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0aee27a:::benchmark_data_smithi169_60938_object169835:head# 2024-02-12T16:18:08.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0af8853:::benchmark_data_smithi169_60938_object160700:head# 2024-02-12T16:18:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0b028e5:::benchmark_data_smithi169_60938_object147810:head# 2024-02-12T16:18:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0b2a66d:::benchmark_data_smithi169_60938_object139209:head# 2024-02-12T16:18:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0b2bf8b:::benchmark_data_smithi169_60938_object151348:head# 2024-02-12T16:18:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0b95afd:::benchmark_data_smithi169_60938_object149022:head# 2024-02-12T16:18:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0bb725c:::benchmark_data_smithi169_60938_object171415:head# 2024-02-12T16:18:08.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0bba991:::benchmark_data_smithi169_60938_object138361:head# 2024-02-12T16:18:08.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0bc445d:::benchmark_data_smithi169_60938_object139070:head# 2024-02-12T16:18:08.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0bf2445:::benchmark_data_smithi169_60938_object147670:head# 2024-02-12T16:18:08.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:08.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0c25909:::benchmark_data_smithi169_60938_object141178:head# 2024-02-12T16:18:08.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0c54279:::benchmark_data_smithi169_60938_object168458:head# 2024-02-12T16:18:08.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0c72fc8:::benchmark_data_smithi169_60938_object153134:head# 2024-02-12T16:18:08.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0ca7bdf:::benchmark_data_smithi169_60938_object172721:head# 2024-02-12T16:18:08.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0ce91bb:::benchmark_data_smithi169_60938_object145350:head# 2024-02-12T16:18:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:08.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:08.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0cedbee:::benchmark_data_smithi169_60938_object139499:head# 2024-02-12T16:18:08.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0d09071:::benchmark_data_smithi169_60938_object140579:head# 2024-02-12T16:18:08.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0d0c24a:::benchmark_data_smithi169_60938_object167494:head# 2024-02-12T16:18:08.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0d3ff5d:::benchmark_data_smithi169_60938_object165626:head# 2024-02-12T16:18:08.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0d6c0fd:::benchmark_data_smithi169_60938_object152917:head# 2024-02-12T16:18:08.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0d83a0b:::benchmark_data_smithi169_60938_object169238:head# 2024-02-12T16:18:08.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0d90f1c:::benchmark_data_smithi169_60938_object140992:head# 2024-02-12T16:18:08.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0da4e9e:::benchmark_data_smithi169_60938_object139001:head# 2024-02-12T16:18:08.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0db052a:::benchmark_data_smithi169_60938_object156744:head# 2024-02-12T16:18:08.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0dbaeef:::benchmark_data_smithi169_60938_object143692:head# 2024-02-12T16:18:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0de002a:::benchmark_data_smithi169_60938_object158909:head# 2024-02-12T16:18:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0de05ab:::benchmark_data_smithi169_60938_object156045:head# 2024-02-12T16:18:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0e074ed:::benchmark_data_smithi169_60938_object146723:head# 2024-02-12T16:18:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0e0e1b9:::benchmark_data_smithi169_60938_object142031:head# 2024-02-12T16:18:08.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0e29d34:::benchmark_data_smithi169_60938_object147943:head# 2024-02-12T16:18:08.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0eb8e06:::benchmark_data_smithi169_60938_object142711:head# 2024-02-12T16:18:08.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0ebbe36:::benchmark_data_smithi169_60938_object172933:head# 2024-02-12T16:18:08.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0ed221b:::benchmark_data_smithi169_60938_object171421:head# 2024-02-12T16:18:08.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0f361d6:::benchmark_data_smithi169_60938_object173129:head# 2024-02-12T16:18:08.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0f4f253:::benchmark_data_smithi169_60938_object140735:head# 2024-02-12T16:18:08.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0f6a32c:::benchmark_data_smithi169_60938_object157411:head# 2024-02-12T16:18:08.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0fc5741:::benchmark_data_smithi169_60938_object145879:head# 2024-02-12T16:18:08.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0fe8aff:::benchmark_data_smithi169_60938_object161123:head# 2024-02-12T16:18:08.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f0ff02f2:::benchmark_data_smithi169_60938_object160754:head# 2024-02-12T16:18:08.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f10444c1:::benchmark_data_smithi169_60938_object171082:head# 2024-02-12T16:18:08.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f104e67c:::benchmark_data_smithi169_60938_object153195:head# 2024-02-12T16:18:08.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f10524eb:::benchmark_data_smithi169_60938_object164515:head# 2024-02-12T16:18:08.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f108112f:::benchmark_data_smithi169_60938_object171389:head# 2024-02-12T16:18:08.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f108788b:::benchmark_data_smithi169_60938_object171620:head# 2024-02-12T16:18:08.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f109587b:::benchmark_data_smithi169_60938_object145145:head# 2024-02-12T16:18:08.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f10977d5:::benchmark_data_smithi169_60938_object149301:head# 2024-02-12T16:18:08.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f10a44f2:::benchmark_data_smithi169_60938_object143085:head# 2024-02-12T16:18:08.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f10ef2dd:::benchmark_data_smithi169_60938_object139151:head# 2024-02-12T16:18:08.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f10fb6f9:::benchmark_data_smithi169_60938_object148251:head# 2024-02-12T16:18:08.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f111d2d1:::benchmark_data_smithi169_60938_object143290:head# 2024-02-12T16:18:08.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1244cbf:::benchmark_data_smithi169_60938_object165191:head# 2024-02-12T16:18:08.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12524ba:::benchmark_data_smithi169_60938_object165607:head# 2024-02-12T16:18:08.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1265b60:::benchmark_data_smithi169_60938_object150584:head# 2024-02-12T16:18:08.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12667d5:::benchmark_data_smithi169_60938_object143485:head# 2024-02-12T16:18:08.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f127ab1d:::benchmark_data_smithi169_60938_object163498:head# 2024-02-12T16:18:08.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12922f2:::benchmark_data_smithi169_60938_object139932:head# 2024-02-12T16:18:08.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12b5377:::benchmark_data_smithi169_60938_object146871:head# 2024-02-12T16:18:08.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12b9d4e:::benchmark_data_smithi169_60938_object150680:head# 2024-02-12T16:18:08.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12be857:::benchmark_data_smithi169_60938_object159951:head# 2024-02-12T16:18:08.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12c1b8e:::benchmark_data_smithi169_60938_object145999:head# 2024-02-12T16:18:08.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12c1e00:::benchmark_data_smithi169_60938_object147604:head# 2024-02-12T16:18:08.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12c7822:::benchmark_data_smithi169_60938_object160727:head# 2024-02-12T16:18:08.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12cfc51:::benchmark_data_smithi169_60938_object145327:head# 2024-02-12T16:18:08.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12d0982:::benchmark_data_smithi169_60938_object140400:head# 2024-02-12T16:18:08.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12e3010:::benchmark_data_smithi169_60938_object144615:head# 2024-02-12T16:18:08.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f12e8132:::benchmark_data_smithi169_60938_object145225:head# 2024-02-12T16:18:08.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1333a5a:::benchmark_data_smithi169_60938_object140035:head# 2024-02-12T16:18:08.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f133c38d:::benchmark_data_smithi169_60938_object160531:head# 2024-02-12T16:18:08.656 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f13438e5:::benchmark_data_smithi169_60938_object149402:head# 2024-02-12T16:18:08.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f138a806:::benchmark_data_smithi169_60938_object167796:head# 2024-02-12T16:18:08.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f139fbc5:::benchmark_data_smithi169_60938_object165640:head# 2024-02-12T16:18:08.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f13b9199:::benchmark_data_smithi169_60938_object156635:head# 2024-02-12T16:18:08.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f13d48d9:::benchmark_data_smithi169_60938_object163472:head# 2024-02-12T16:18:08.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f13e6adb:::benchmark_data_smithi169_60938_object159658:head# 2024-02-12T16:18:08.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f13f295d:::benchmark_data_smithi169_60938_object152228:head# 2024-02-12T16:18:08.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f13fe004:::benchmark_data_smithi169_60938_object160786:head# 2024-02-12T16:18:08.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f140ef75:::benchmark_data_smithi169_60938_object168448:head# 2024-02-12T16:18:08.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f143a9b5:::benchmark_data_smithi169_60938_object150821:head# 2024-02-12T16:18:08.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f145b713:::benchmark_data_smithi169_60938_object155172:head# 2024-02-12T16:18:08.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1477b49:::benchmark_data_smithi169_60938_object149219:head# 2024-02-12T16:18:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1479725:::benchmark_data_smithi169_60938_object139494:head# 2024-02-12T16:18:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f14c3325:::benchmark_data_smithi169_60938_object166966:head# 2024-02-12T16:18:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f150f81a:::benchmark_data_smithi169_60938_object168167:head# 2024-02-12T16:18:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1510436:::benchmark_data_smithi169_60938_object155676:head# 2024-02-12T16:18:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1535578:::benchmark_data_smithi169_60938_object140345:head# 2024-02-12T16:18:08.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:18:08.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1539eb1:::benchmark_data_smithi169_60938_object148598:head# 2024-02-12T16:18:08.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f157566f:::benchmark_data_smithi169_60938_object171425:head# 2024-02-12T16:18:08.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15874ac:::benchmark_data_smithi169_60938_object162391:head# 2024-02-12T16:18:08.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15a3db8:::benchmark_data_smithi169_60938_object163355:head# 2024-02-12T16:18:08.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15aa612:::benchmark_data_smithi169_60938_object140391:head# 2024-02-12T16:18:08.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15b488a:::benchmark_data_smithi169_60938_object155732:head# 2024-02-12T16:18:08.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15b6c50:::benchmark_data_smithi169_60938_object143281:head# 2024-02-12T16:18:08.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15c1f2e:::benchmark_data_smithi169_60938_object166781:head# 2024-02-12T16:18:08.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15d002f:::benchmark_data_smithi169_60938_object149037:head# 2024-02-12T16:18:08.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15dcb90:::benchmark_data_smithi169_60938_object155410:head# 2024-02-12T16:18:08.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f15dd091:::benchmark_data_smithi169_60938_object172174:head# 2024-02-12T16:18:08.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1612ca8:::benchmark_data_smithi169_60938_object159943:head# 2024-02-12T16:18:08.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f164c96d:::benchmark_data_smithi169_60938_object138584:head# 2024-02-12T16:18:08.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f165159f:::benchmark_data_smithi169_60938_object152162:head# 2024-02-12T16:18:08.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1676f47:::benchmark_data_smithi169_60938_object162988:head# 2024-02-12T16:18:08.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1693485:::benchmark_data_smithi169_60938_object150180:head# 2024-02-12T16:18:08.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f169f74d:::benchmark_data_smithi169_60938_object157365:head# 2024-02-12T16:18:08.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f16ac276:::benchmark_data_smithi169_60938_object150994:head# 2024-02-12T16:18:08.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f16c66a9:::benchmark_data_smithi169_60938_object146337:head# 2024-02-12T16:18:08.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f16f3089:::benchmark_data_smithi169_60938_object172305:head# 2024-02-12T16:18:08.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f17332fc:::benchmark_data_smithi169_60938_object160885:head# 2024-02-12T16:18:08.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1747a0f:::benchmark_data_smithi169_60938_object155728:head# 2024-02-12T16:18:08.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f17ae632:::benchmark_data_smithi169_60938_object170878:head# 2024-02-12T16:18:08.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f17e3448:::benchmark_data_smithi169_60938_object151511:head# 2024-02-12T16:18:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f17e36b8:::benchmark_data_smithi169_60938_object163899:head# 2024-02-12T16:18:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f18241ab:::benchmark_data_smithi169_60938_object144353:head# 2024-02-12T16:18:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:18:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f182ce2f:::benchmark_data_smithi169_60938_object167488:head# 2024-02-12T16:18:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f182f68c:::benchmark_data_smithi169_60938_object164459:head# 2024-02-12T16:18:08.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f18546b5:::benchmark_data_smithi169_60938_object150341:head# 2024-02-12T16:18:08.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f187f0e1:::benchmark_data_smithi169_60938_object154860:head# 2024-02-12T16:18:08.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1898af1:::benchmark_data_smithi169_60938_object171615:head# 2024-02-12T16:18:08.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f189d3cb:::benchmark_data_smithi169_60938_object147480:head# 2024-02-12T16:18:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1905efc:::benchmark_data_smithi169_60938_object162321:head# 2024-02-12T16:18:08.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1946394:::benchmark_data_smithi169_60938_object154891:head# 2024-02-12T16:18:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1952806:::benchmark_data_smithi169_60938_object154334:head# 2024-02-12T16:18:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f196cae3:::benchmark_data_smithi169_60938_object147906:head# 2024-02-12T16:18:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f19bde44:::benchmark_data_smithi169_60938_object158046:head# 2024-02-12T16:18:08.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f19be320:::benchmark_data_smithi169_60938_object146005:head# 2024-02-12T16:18:08.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f19cc637:::benchmark_data_smithi169_60938_object158747:head# 2024-02-12T16:18:08.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f19e79bd:::benchmark_data_smithi169_60938_object156845:head# 2024-02-12T16:18:08.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f19fedc2:::benchmark_data_smithi169_60938_object142747:head# 2024-02-12T16:18:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a20d5e:::benchmark_data_smithi169_60938_object147074:head# 2024-02-12T16:18:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a28f39:::benchmark_data_smithi169_60938_object153974:head# 2024-02-12T16:18:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a4d770:::benchmark_data_smithi169_60938_object171450:head# 2024-02-12T16:18:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a5577b:::benchmark_data_smithi169_60938_object153673:head# 2024-02-12T16:18:08.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a8bed8:::benchmark_data_smithi169_60938_object166929:head# 2024-02-12T16:18:08.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a8c406:::benchmark_data_smithi169_60938_object147720:head# 2024-02-12T16:18:08.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a8dab3:::benchmark_data_smithi169_60938_object165491:head# 2024-02-12T16:18:08.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a8dc99:::benchmark_data_smithi169_60938_object151159:head# 2024-02-12T16:18:08.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1a94a1c:::benchmark_data_smithi169_60938_object152762:head# 2024-02-12T16:18:08.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ab45b1:::benchmark_data_smithi169_60938_object145805:head# 2024-02-12T16:18:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1aba13f:::benchmark_data_smithi169_60938_object142915:head# 2024-02-12T16:18:08.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ac4933:::benchmark_data_smithi169_60938_object152410:head# 2024-02-12T16:18:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1aca719:::benchmark_data_smithi169_60938_object153155:head# 2024-02-12T16:18:08.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ad5f79:::benchmark_data_smithi169_60938_object146781:head# 2024-02-12T16:18:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ae17ea:::benchmark_data_smithi169_60938_object169789:head# 2024-02-12T16:18:08.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ae371a:::benchmark_data_smithi169_60938_object163328:head# 2024-02-12T16:18:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1afd0be:::benchmark_data_smithi169_60938_object170945:head# 2024-02-12T16:18:08.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1b1b708:::benchmark_data_smithi169_60938_object165209:head# 2024-02-12T16:18:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1b1b93d:::benchmark_data_smithi169_60938_object147607:head# 2024-02-12T16:18:08.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1b88e0b:::benchmark_data_smithi169_60938_object153112:head# 2024-02-12T16:18:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1b97206:::benchmark_data_smithi169_60938_object156475:head# 2024-02-12T16:18:08.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1bcc8c8:::benchmark_data_smithi169_60938_object159770:head# 2024-02-12T16:18:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1bcff11:::benchmark_data_smithi169_60938_object171360:head# 2024-02-12T16:18:08.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1bd78da:::benchmark_data_smithi169_60938_object156171:head# 2024-02-12T16:18:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1bde816:::benchmark_data_smithi169_60938_object146833:head# 2024-02-12T16:18:08.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1bf2437:::benchmark_data_smithi169_60938_object155681:head# 2024-02-12T16:18:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c00d31:::benchmark_data_smithi169_60938_object147725:head# 2024-02-12T16:18:08.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c2a2f1:::benchmark_data_smithi169_60938_object146945:head# 2024-02-12T16:18:08.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c60956:::benchmark_data_smithi169_60938_object165537:head# 2024-02-12T16:18:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c7ecac:::benchmark_data_smithi169_60938_object155264:head# 2024-02-12T16:18:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c815c1:::benchmark_data_smithi169_60938_object156627:head# 2024-02-12T16:18:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c826b6:::benchmark_data_smithi169_60938_object162051:head# 2024-02-12T16:18:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c855a0:::benchmark_data_smithi169_60938_object141604:head# 2024-02-12T16:18:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c8d3fa:::benchmark_data_smithi169_60938_object154263:head# 2024-02-12T16:18:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1c94512:::benchmark_data_smithi169_60938_object156494:head# 2024-02-12T16:18:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1cb2068:::benchmark_data_smithi169_60938_object154527:head# 2024-02-12T16:18:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1d052ae:::benchmark_data_smithi169_60938_object143825:head# 2024-02-12T16:18:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1d3bd66:::benchmark_data_smithi169_60938_object162823:head# 2024-02-12T16:18:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1d57cee:::benchmark_data_smithi169_60938_object160772:head# 2024-02-12T16:18:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1d58c26:::benchmark_data_smithi169_60938_object138675:head# 2024-02-12T16:18:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1d6af79:::benchmark_data_smithi169_60938_object149532:head# 2024-02-12T16:18:08.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1d93fa3:::benchmark_data_smithi169_60938_object169219:head# 2024-02-12T16:18:08.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ddc9a7:::benchmark_data_smithi169_60938_object161948:head# 2024-02-12T16:18:08.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ddee7e:::benchmark_data_smithi169_60938_object150405:head# 2024-02-12T16:18:08.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1dffa58:::benchmark_data_smithi169_60938_object171120:head# 2024-02-12T16:18:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1e4e9f4:::benchmark_data_smithi169_60938_object141470:head# 2024-02-12T16:18:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1e55d06:::benchmark_data_smithi169_60938_object156798:head# 2024-02-12T16:18:08.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1e5c085:::benchmark_data_smithi169_60938_object157835:head# 2024-02-12T16:18:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1e6c281:::benchmark_data_smithi169_60938_object142530:head# 2024-02-12T16:18:08.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1e79e8f:::benchmark_data_smithi169_60938_object142941:head# 2024-02-12T16:18:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1e7ee4b:::benchmark_data_smithi169_60938_object153670:head# 2024-02-12T16:18:08.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1eae26b:::benchmark_data_smithi169_60938_object150705:head# 2024-02-12T16:18:08.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1eb36b4:::benchmark_data_smithi169_60938_object162102:head# 2024-02-12T16:18:08.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ee19ca:::benchmark_data_smithi169_60938_object139959:head# 2024-02-12T16:18:08.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ee254e:::benchmark_data_smithi169_60938_object163610:head# 2024-02-12T16:18:08.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ee3039:::benchmark_data_smithi169_60938_object168243:head# 2024-02-12T16:18:08.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1ee8ef8:::benchmark_data_smithi169_60938_object153220:head# 2024-02-12T16:18:08.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1f1fb9e:::benchmark_data_smithi169_60938_object155970:head# 2024-02-12T16:18:08.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1f5c080:::benchmark_data_smithi169_60938_object141029:head# 2024-02-12T16:18:08.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1f783a7:::benchmark_data_smithi169_60938_object163861:head# 2024-02-12T16:18:08.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1fb4e24:::benchmark_data_smithi169_60938_object157468:head# 2024-02-12T16:18:08.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f1fdcfbe:::benchmark_data_smithi169_60938_object157267:head# 2024-02-12T16:18:08.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2033ee2:::benchmark_data_smithi169_60938_object160221:head# 2024-02-12T16:18:08.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f20580f9:::benchmark_data_smithi169_60938_object144826:head# 2024-02-12T16:18:08.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f20894db:::benchmark_data_smithi169_60938_object169530:head# 2024-02-12T16:18:08.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f20d056e:::benchmark_data_smithi169_60938_object146259:head# 2024-02-12T16:18:08.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f210565d:::benchmark_data_smithi169_60938_object147595:head# 2024-02-12T16:18:08.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2112e86:::benchmark_data_smithi169_60938_object163010:head# 2024-02-12T16:18:08.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f211a858:::benchmark_data_smithi169_60938_object144523:head# 2024-02-12T16:18:08.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f21985a3:::benchmark_data_smithi169_60938_object161245:head# 2024-02-12T16:18:08.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:08.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f21da37e:::benchmark_data_smithi169_60938_object153884:head# 2024-02-12T16:18:08.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f22145ac:::benchmark_data_smithi169_60938_object141158:head# 2024-02-12T16:18:08.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f222860f:::benchmark_data_smithi169_60938_object172222:head# 2024-02-12T16:18:08.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f223677d:::benchmark_data_smithi169_60938_object148116:head# 2024-02-12T16:18:08.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2237f2e:::benchmark_data_smithi169_60938_object151243:head# 2024-02-12T16:18:08.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f223ce4e:::benchmark_data_smithi169_60938_object165631:head# 2024-02-12T16:18:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2289156:::benchmark_data_smithi169_60938_object159812:head# 2024-02-12T16:18:08.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f22b76d8:::benchmark_data_smithi169_60938_object164377:head# 2024-02-12T16:18:08.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f22b9d45:::benchmark_data_smithi169_60938_object141016:head# 2024-02-12T16:18:08.733 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f22c532a:::benchmark_data_smithi169_60938_object139500:head# 2024-02-12T16:18:08.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2357fdf:::benchmark_data_smithi169_60938_object172746:head# 2024-02-12T16:18:08.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2380690:::benchmark_data_smithi169_60938_object159038:head# 2024-02-12T16:18:08.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f238c546:::benchmark_data_smithi169_60938_object163934:head# 2024-02-12T16:18:08.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f239b718:::benchmark_data_smithi169_60938_object170869:head# 2024-02-12T16:18:08.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f23b09b7:::benchmark_data_smithi169_60938_object139119:head# 2024-02-12T16:18:08.739 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:08 smithi169 bash[19604]: cluster 2024-02-12T16:18:07.761941+0000 mgr.y (mgr.14146) 985 : cluster [DBG] pgmap v1311: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:08.739 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:08 smithi169 bash[19604]: audit 2024-02-12T16:18:08.182982+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? v1:172.21.15.121:0/1806690187' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:18:08.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f23b5962:::benchmark_data_smithi169_60938_object168808:head# 2024-02-12T16:18:08.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f23ce405:::benchmark_data_smithi169_60938_object151394:head# 2024-02-12T16:18:08.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f23e5347:::benchmark_data_smithi169_60938_object166609:head# 2024-02-12T16:18:08.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f23e56f3:::benchmark_data_smithi169_60938_object146063:head# 2024-02-12T16:18:08.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2402ad1:::benchmark_data_smithi169_60938_object165099:head# 2024-02-12T16:18:08.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f246a7b9:::benchmark_data_smithi169_60938_object167214:head# 2024-02-12T16:18:08.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f247184e:::benchmark_data_smithi169_60938_object168756:head# 2024-02-12T16:18:08.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f24a14d7:::benchmark_data_smithi169_60938_object142034:head# 2024-02-12T16:18:08.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f24c6b92:::benchmark_data_smithi169_60938_object169875:head# 2024-02-12T16:18:08.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f24da4e9:::benchmark_data_smithi169_60938_object145181:head# 2024-02-12T16:18:08.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f24e7c5f:::benchmark_data_smithi169_60938_object167500:head# 2024-02-12T16:18:08.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2548089:::benchmark_data_smithi169_60938_object156613:head# 2024-02-12T16:18:08.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2555bb5:::benchmark_data_smithi169_60938_object173265:head# 2024-02-12T16:18:08.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2563c87:::benchmark_data_smithi169_60938_object148768:head# 2024-02-12T16:18:08.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2571e11:::benchmark_data_smithi169_60938_object146217:head# 2024-02-12T16:18:08.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f257d1bd:::benchmark_data_smithi169_60938_object142410:head# 2024-02-12T16:18:08.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f257e96d:::benchmark_data_smithi169_60938_object144381:head# 2024-02-12T16:18:08.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f259a919:::benchmark_data_smithi169_60938_object140925:head# 2024-02-12T16:18:08.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f25ae07b:::benchmark_data_smithi169_60938_object151834:head# 2024-02-12T16:18:08.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f25c20b1:::benchmark_data_smithi169_60938_object172130:head# 2024-02-12T16:18:08.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f25f9584:::benchmark_data_smithi169_60938_object159884:head# 2024-02-12T16:18:08.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f25fe59d:::benchmark_data_smithi169_60938_object162261:head# 2024-02-12T16:18:08.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f264fc85:::benchmark_data_smithi169_60938_object170978:head# 2024-02-12T16:18:08.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2672730:::benchmark_data_smithi169_60938_object141295:head# 2024-02-12T16:18:08.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2689f86:::benchmark_data_smithi169_60938_object169131:head# 2024-02-12T16:18:08.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f26943d2:::benchmark_data_smithi169_60938_object165933:head# 2024-02-12T16:18:08.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f26fb372:::benchmark_data_smithi169_60938_object167366:head# 2024-02-12T16:18:08.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f278ae8e:::benchmark_data_smithi169_60938_object144369:head# 2024-02-12T16:18:08.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f27cf9f4:::benchmark_data_smithi169_60938_object164370:head# 2024-02-12T16:18:08.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f27f2d61:::benchmark_data_smithi169_60938_object169856:head# 2024-02-12T16:18:08.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f27ff23b:::benchmark_data_smithi169_60938_object158131:head# 2024-02-12T16:18:08.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2803edd:::benchmark_data_smithi169_60938_object152160:head# 2024-02-12T16:18:08.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2810357:::benchmark_data_smithi169_60938_object161156:head# 2024-02-12T16:18:08.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f286da53:::benchmark_data_smithi169_60938_object152411:head# 2024-02-12T16:18:08.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f287caa8:::benchmark_data_smithi169_60938_object141200:head# 2024-02-12T16:18:08.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28861b1:::benchmark_data_smithi169_60938_object167478:head# 2024-02-12T16:18:08.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28895a2:::benchmark_data_smithi169_60938_object141307:head# 2024-02-12T16:18:08.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f288d721:::benchmark_data_smithi169_60938_object140540:head# 2024-02-12T16:18:08.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28a70ba:::benchmark_data_smithi169_60938_object141566:head# 2024-02-12T16:18:08.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28cdf40:::benchmark_data_smithi169_60938_object156187:head# 2024-02-12T16:18:08.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28deca8:::benchmark_data_smithi169_60938_object150460:head# 2024-02-12T16:18:08.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28ed3ff:::benchmark_data_smithi169_60938_object144959:head# 2024-02-12T16:18:08.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28fab29:::benchmark_data_smithi169_60938_object171048:head# 2024-02-12T16:18:08.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f28fb105:::benchmark_data_smithi169_60938_object152082:head# 2024-02-12T16:18:08.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f291b42a:::benchmark_data_smithi169_60938_object155608:head# 2024-02-12T16:18:08.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f29277ca:::benchmark_data_smithi169_60938_object169568:head# 2024-02-12T16:18:08.762 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:08.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f292a1ff:::benchmark_data_smithi169_60938_object166705:head# 2024-02-12T16:18:08.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f294368d:::benchmark_data_smithi169_60938_object142965:head# 2024-02-12T16:18:08.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2964850:::benchmark_data_smithi169_60938_object172273:head# 2024-02-12T16:18:08.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2973656:::benchmark_data_smithi169_60938_object142614:head# 2024-02-12T16:18:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:08.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:08.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2977286:::benchmark_data_smithi169_60938_object143612:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f29816b0:::benchmark_data_smithi169_60938_object154452:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f29b66be:::benchmark_data_smithi169_60938_object164488:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f29bbc95:::benchmark_data_smithi169_60938_object139774:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f29c6d49:::benchmark_data_smithi169_60938_object146030:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f29cb813:::benchmark_data_smithi169_60938_object166023:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f29cc230:::benchmark_data_smithi169_60938_object154274:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2a0f12c:::benchmark_data_smithi169_60938_object142487:head# 2024-02-12T16:18:08.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2a43112:::benchmark_data_smithi169_60938_object144780:head# 2024-02-12T16:18:08.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2a576fb:::benchmark_data_smithi169_60938_object173175:head# 2024-02-12T16:18:08.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2a5a7dc:::benchmark_data_smithi169_60938_object152034:head# 2024-02-12T16:18:08.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2a6c631:::benchmark_data_smithi169_60938_object158507:head# 2024-02-12T16:18:08.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2a70595:::benchmark_data_smithi169_60938_object171802:head# 2024-02-12T16:18:08.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2a84242:::benchmark_data_smithi169_60938_object152379:head# 2024-02-12T16:18:08.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2ab315b:::benchmark_data_smithi169_60938_object151273:head# 2024-02-12T16:18:08.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2acc79a:::benchmark_data_smithi169_60938_object144668:head# 2024-02-12T16:18:08.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2b2d8b5:::benchmark_data_smithi169_60938_object150979:head# 2024-02-12T16:18:08.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2b38274:::benchmark_data_smithi169_60938_object167383:head# 2024-02-12T16:18:08.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2b4687a:::benchmark_data_smithi169_60938_object142954:head# 2024-02-12T16:18:08.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2b6c3c9:::benchmark_data_smithi169_60938_object157040:head# 2024-02-12T16:18:08.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2b7d9b9:::benchmark_data_smithi169_60938_object145061:head# 2024-02-12T16:18:08.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2b93958:::benchmark_data_smithi169_60938_object168484:head# 2024-02-12T16:18:08.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2b9ab52:::benchmark_data_smithi169_60938_object141990:head# 2024-02-12T16:18:08.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2bd32e9:::benchmark_data_smithi169_60938_object144246:head# 2024-02-12T16:18:08.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2c0dd91:::benchmark_data_smithi169_60938_object151380:head# 2024-02-12T16:18:08.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2c1e78c:::benchmark_data_smithi169_60938_object160779:head# 2024-02-12T16:18:08.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2c3b378:::benchmark_data_smithi169_60938_object154803:head# 2024-02-12T16:18:08.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2c4059f:::benchmark_data_smithi169_60938_object168527:head# 2024-02-12T16:18:08.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2c557e3:::benchmark_data_smithi169_60938_object138572:head# 2024-02-12T16:18:08.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2c62d21:::benchmark_data_smithi169_60938_object143838:head# 2024-02-12T16:18:08.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2cb318c:::benchmark_data_smithi169_60938_object139185:head# 2024-02-12T16:18:08.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2cc8902:::benchmark_data_smithi169_60938_object160747:head# 2024-02-12T16:18:08.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d1fe54:::benchmark_data_smithi169_60938_object149160:head# 2024-02-12T16:18:08.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d34d16:::benchmark_data_smithi169_60938_object148970:head# 2024-02-12T16:18:08.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d3799b:::benchmark_data_smithi169_60938_object166880:head# 2024-02-12T16:18:08.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d3fb57:::benchmark_data_smithi169_60938_object149141:head# 2024-02-12T16:18:08.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d7542b:::benchmark_data_smithi169_60938_object164730:head# 2024-02-12T16:18:08.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d7600e:::benchmark_data_smithi169_60938_object159750:head# 2024-02-12T16:18:08.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d7833f:::benchmark_data_smithi169_60938_object152722:head# 2024-02-12T16:18:08.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d79580:::benchmark_data_smithi169_60938_object168990:head# 2024-02-12T16:18:08.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2d7dae7:::benchmark_data_smithi169_60938_object162534:head# 2024-02-12T16:18:08.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2dadd7a:::benchmark_data_smithi169_60938_object157788:head# 2024-02-12T16:18:08.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2db355b:::benchmark_data_smithi169_60938_object165852:head# 2024-02-12T16:18:08.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2dc4c5b:::benchmark_data_smithi169_60938_object163553:head# 2024-02-12T16:18:08.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2de4980:::benchmark_data_smithi169_60938_object151141:head# 2024-02-12T16:18:08.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2df3190:::benchmark_data_smithi169_60938_object163342:head# 2024-02-12T16:18:08.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2dff076:::benchmark_data_smithi169_60938_object149240:head# 2024-02-12T16:18:08.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e09675:::benchmark_data_smithi169_60938_object168557:head# 2024-02-12T16:18:08.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e25c43:::benchmark_data_smithi169_60938_object166262:head# 2024-02-12T16:18:08.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e27f70:::benchmark_data_smithi169_60938_object162525:head# 2024-02-12T16:18:08.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e2937a:::benchmark_data_smithi169_60938_object160636:head# 2024-02-12T16:18:08.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e2c8aa:::benchmark_data_smithi169_60938_object166135:head# 2024-02-12T16:18:08.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e48b4d:::benchmark_data_smithi169_60938_object142046:head# 2024-02-12T16:18:08.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e498d2:::benchmark_data_smithi169_60938_object144633:head# 2024-02-12T16:18:08.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e688e7:::benchmark_data_smithi169_60938_object161323:head# 2024-02-12T16:18:08.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e884db:::benchmark_data_smithi169_60938_object161412:head# 2024-02-12T16:18:08.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e9655b:::benchmark_data_smithi169_60938_object166975:head# 2024-02-12T16:18:08.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2e9c263:::benchmark_data_smithi169_60938_object144697:head# 2024-02-12T16:18:08.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2ed0e09:::benchmark_data_smithi169_60938_object160207:head# 2024-02-12T16:18:08.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2ed15be:::benchmark_data_smithi169_60938_object149277:head# 2024-02-12T16:18:08.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2edef31:::benchmark_data_smithi169_60938_object166473:head# 2024-02-12T16:18:08.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2eed258:::benchmark_data_smithi169_60938_object144023:head# 2024-02-12T16:18:08.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2f36a79:::benchmark_data_smithi169_60938_object152520:head# 2024-02-12T16:18:08.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2f4ac33:::benchmark_data_smithi169_60938_object150583:head# 2024-02-12T16:18:08.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2f50199:::benchmark_data_smithi169_60938_object164817:head# 2024-02-12T16:18:08.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2fbb8ba:::benchmark_data_smithi169_60938_object171109:head# 2024-02-12T16:18:08.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2fd51bc:::benchmark_data_smithi169_60938_object168865:head# 2024-02-12T16:18:08.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f2fd8062:::benchmark_data_smithi169_60938_object147174:head# 2024-02-12T16:18:08.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3018822:::benchmark_data_smithi169_60938_object168888:head# 2024-02-12T16:18:08.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3024779:::benchmark_data_smithi169_60938_object154620:head# 2024-02-12T16:18:08.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f305559e:::benchmark_data_smithi169_60938_object141471:head# 2024-02-12T16:18:08.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f305e90c:::benchmark_data_smithi169_60938_object149832:head# 2024-02-12T16:18:08.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f30604ec:::benchmark_data_smithi169_60938_object172249:head# 2024-02-12T16:18:08.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f30803bb:::benchmark_data_smithi169_60938_object147254:head# 2024-02-12T16:18:08.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f30a3bbd:::benchmark_data_smithi169_60938_object156295:head# 2024-02-12T16:18:08.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f30bc51a:::benchmark_data_smithi169_60938_object150570:head# 2024-02-12T16:18:08.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f30c5afb:::benchmark_data_smithi169_60938_object155351:head# 2024-02-12T16:18:08.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f30d7932:::benchmark_data_smithi169_60938_object154317:head# 2024-02-12T16:18:08.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f30f4abc:::benchmark_data_smithi169_60938_object150533:head# 2024-02-12T16:18:08.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3142bd1:::benchmark_data_smithi169_60938_object147333:head# 2024-02-12T16:18:08.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3152974:::benchmark_data_smithi169_60938_object138976:head# 2024-02-12T16:18:08.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f31bafdf:::benchmark_data_smithi169_60938_object141058:head# 2024-02-12T16:18:08.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f31bb724:::benchmark_data_smithi169_60938_object140630:head# 2024-02-12T16:18:08.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3248efd:::benchmark_data_smithi169_60938_object138699:head# 2024-02-12T16:18:08.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32574d5:::benchmark_data_smithi169_60938_object157229:head# 2024-02-12T16:18:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32579c5:::benchmark_data_smithi169_60938_object164132:head# 2024-02-12T16:18:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3280ab9:::benchmark_data_smithi169_60938_object167973:head# 2024-02-12T16:18:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32a2c44:::benchmark_data_smithi169_60938_object151500:head# 2024-02-12T16:18:08.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32b9eaf:::benchmark_data_smithi169_60938_object156821:head# 2024-02-12T16:18:08.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32c739c:::benchmark_data_smithi169_60938_object167623:head# 2024-02-12T16:18:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32d03e0:::benchmark_data_smithi169_60938_object159270:head# 2024-02-12T16:18:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32d09d3:::benchmark_data_smithi169_60938_object161787:head# 2024-02-12T16:18:08.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f32ed07f:::benchmark_data_smithi169_60938_object140141:head# 2024-02-12T16:18:08.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3356df0:::benchmark_data_smithi169_60938_object146824:head# 2024-02-12T16:18:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3371382:::benchmark_data_smithi169_60938_object165665:head# 2024-02-12T16:18:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f337a896:::benchmark_data_smithi169_60938_object172655:head# 2024-02-12T16:18:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3384ef1:::benchmark_data_smithi169_60938_object173006:head# 2024-02-12T16:18:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3386d16:::benchmark_data_smithi169_60938_object159124:head# 2024-02-12T16:18:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3390f7f:::benchmark_data_smithi169_60938_object168763:head# 2024-02-12T16:18:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f33b361a:::benchmark_data_smithi169_60938_object149129:head# 2024-02-12T16:18:08.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f33ceac8:::benchmark_data_smithi169_60938_object159280:head# 2024-02-12T16:18:08.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f34294b1:::benchmark_data_smithi169_60938_object152681:head# 2024-02-12T16:18:08.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f346e988:::benchmark_data_smithi169_60938_object147120:head# 2024-02-12T16:18:08.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3472cc2:::benchmark_data_smithi169_60938_object144904:head# 2024-02-12T16:18:08.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3479ccc:::benchmark_data_smithi169_60938_object141944:head# 2024-02-12T16:18:08.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f347aeb0:::benchmark_data_smithi169_60938_object170569:head# 2024-02-12T16:18:08.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3481484:::benchmark_data_smithi169_60938_object146325:head# 2024-02-12T16:18:08.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f34bffb7:::benchmark_data_smithi169_60938_object138627:head# 2024-02-12T16:18:08.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f35174f9:::benchmark_data_smithi169_60938_object164091:head# 2024-02-12T16:18:08.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f351ffba:::benchmark_data_smithi169_60938_object150898:head# 2024-02-12T16:18:08.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f35251aa:::benchmark_data_smithi169_60938_object140102:head# 2024-02-12T16:18:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f352ab27:::benchmark_data_smithi169_60938_object164233:head# 2024-02-12T16:18:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f35329e2:::benchmark_data_smithi169_60938_object151613:head# 2024-02-12T16:18:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3583a0b:::benchmark_data_smithi169_60938_object139794:head# 2024-02-12T16:18:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f359cf14:::benchmark_data_smithi169_60938_object146715:head# 2024-02-12T16:18:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f35a013a:::benchmark_data_smithi169_60938_object149868:head# 2024-02-12T16:18:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f35bd8ee:::benchmark_data_smithi169_60938_object159836:head# 2024-02-12T16:18:08.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f35f20e8:::benchmark_data_smithi169_60938_object171333:head# 2024-02-12T16:18:08.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3666f59:::benchmark_data_smithi169_60938_object147944:head# 2024-02-12T16:18:08.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f366865b:::benchmark_data_smithi169_60938_object139479:head# 2024-02-12T16:18:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f367dfe3:::benchmark_data_smithi169_60938_object170009:head# 2024-02-12T16:18:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3694fe0:::benchmark_data_smithi169_60938_object146796:head# 2024-02-12T16:18:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3697299:::benchmark_data_smithi169_60938_object164293:head# 2024-02-12T16:18:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f36a900e:::benchmark_data_smithi169_60938_object172672:head# 2024-02-12T16:18:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f36c4bc2:::benchmark_data_smithi169_60938_object163842:head# 2024-02-12T16:18:08.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f37350d9:::benchmark_data_smithi169_60938_object148029:head# 2024-02-12T16:18:08.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3747660:::benchmark_data_smithi169_60938_object158571:head# 2024-02-12T16:18:08.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3748581:::benchmark_data_smithi169_60938_object153815:head# 2024-02-12T16:18:08.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f374e625:::benchmark_data_smithi169_60938_object142488:head# 2024-02-12T16:18:08.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3793802:::benchmark_data_smithi169_60938_object162050:head# 2024-02-12T16:18:08.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f37a0e9b:::benchmark_data_smithi169_60938_object167540:head# 2024-02-12T16:18:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f37c3a14:::benchmark_data_smithi169_60938_object160060:head# 2024-02-12T16:18:08.833 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3829845:::benchmark_data_smithi169_60938_object150394:head# 2024-02-12T16:18:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3837373:::benchmark_data_smithi169_60938_object143974:head# 2024-02-12T16:18:08.834 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f38534e9:::benchmark_data_smithi169_60938_object170646:head# 2024-02-12T16:18:08.835 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3854b2c:::benchmark_data_smithi169_60938_object146231:head# 2024-02-12T16:18:08.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3876d6b:::benchmark_data_smithi169_60938_object150553:head# 2024-02-12T16:18:08.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f387b801:::benchmark_data_smithi169_60938_object158575:head# 2024-02-12T16:18:08.836 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3882510:::benchmark_data_smithi169_60938_object169085:head# 2024-02-12T16:18:08.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f38996d6:::benchmark_data_smithi169_60938_object163173:head# 2024-02-12T16:18:08.837 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f38f578e:::benchmark_data_smithi169_60938_object161309:head# 2024-02-12T16:18:08.838 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f38fb032:::benchmark_data_smithi169_60938_object146489:head# 2024-02-12T16:18:08.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3912270:::benchmark_data_smithi169_60938_object145803:head# 2024-02-12T16:18:08.839 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3920ce4:::benchmark_data_smithi169_60938_object143108:head# 2024-02-12T16:18:08.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f394da28:::benchmark_data_smithi169_60938_object151748:head# 2024-02-12T16:18:08.840 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f397f1f6:::benchmark_data_smithi169_60938_object159897:head# 2024-02-12T16:18:08.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f398eca0:::benchmark_data_smithi169_60938_object172053:head# 2024-02-12T16:18:08.841 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3996862:::benchmark_data_smithi169_60938_object147372:head# 2024-02-12T16:18:08.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f39ab24e:::benchmark_data_smithi169_60938_object164183:head# 2024-02-12T16:18:08.842 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f39c9db2:::benchmark_data_smithi169_60938_object158776:head# 2024-02-12T16:18:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f39df2ae:::benchmark_data_smithi169_60938_object166056:head# 2024-02-12T16:18:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3a29290:::benchmark_data_smithi169_60938_object150632:head# 2024-02-12T16:18:08.844 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3a5e222:::benchmark_data_smithi169_60938_object162994:head# 2024-02-12T16:18:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3a5f6bb:::benchmark_data_smithi169_60938_object162953:head# 2024-02-12T16:18:08.845 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3a6ba14:::benchmark_data_smithi169_60938_object158851:head# 2024-02-12T16:18:08.846 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3adeac6:::benchmark_data_smithi169_60938_object169342:head# 2024-02-12T16:18:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3af78ce:::benchmark_data_smithi169_60938_object164256:head# 2024-02-12T16:18:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3b25cc5:::benchmark_data_smithi169_60938_object154738:head# 2024-02-12T16:18:08.847 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3b53710:::benchmark_data_smithi169_60938_object159422:head# 2024-02-12T16:18:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3b54187:::benchmark_data_smithi169_60938_object166146:head# 2024-02-12T16:18:08.848 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3b55793:::benchmark_data_smithi169_60938_object138818:head# 2024-02-12T16:18:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3b7227e:::benchmark_data_smithi169_60938_object160483:head# 2024-02-12T16:18:08.849 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3b7985a:::benchmark_data_smithi169_60938_object146481:head# 2024-02-12T16:18:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3bc4758:::benchmark_data_smithi169_60938_object166060:head# 2024-02-12T16:18:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3c0ed1f:::benchmark_data_smithi169_60938_object170837:head# 2024-02-12T16:18:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3c0fceb:::benchmark_data_smithi169_60938_object161302:head# 2024-02-12T16:18:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3c2ac2f:::benchmark_data_smithi169_60938_object150565:head# 2024-02-12T16:18:08.852 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3c3750d:::benchmark_data_smithi169_60938_object159113:head# 2024-02-12T16:18:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3c47fad:::benchmark_data_smithi169_60938_object140740:head# 2024-02-12T16:18:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3cf8b83:::benchmark_data_smithi169_60938_object163995:head# 2024-02-12T16:18:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3d00316:::benchmark_data_smithi169_60938_object144050:head# 2024-02-12T16:18:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3d094be:::benchmark_data_smithi169_60938_object159276:head# 2024-02-12T16:18:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3d4742a:::benchmark_data_smithi169_60938_object146682:head# 2024-02-12T16:18:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3d73804:::benchmark_data_smithi169_60938_object148865:head# 2024-02-12T16:18:08.856 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3d9a01e:::benchmark_data_smithi169_60938_object150547:head# 2024-02-12T16:18:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3daaa6c:::benchmark_data_smithi169_60938_object170275:head# 2024-02-12T16:18:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3dd46f8:::benchmark_data_smithi169_60938_object158303:head# 2024-02-12T16:18:08.857 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3e011a9:::benchmark_data_smithi169_60938_object160155:head# 2024-02-12T16:18:08.858 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3e1d9c0:::benchmark_data_smithi169_60938_object172473:head# 2024-02-12T16:18:08.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3e609f9:::benchmark_data_smithi169_60938_object155960:head# 2024-02-12T16:18:08.859 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3e91e5a:::benchmark_data_smithi169_60938_object149389:head# 2024-02-12T16:18:08.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3eace7d:::benchmark_data_smithi169_60938_object164041:head# 2024-02-12T16:18:08.860 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3eb0a9a:::benchmark_data_smithi169_60938_object139668:head# 2024-02-12T16:18:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3eb6c1d:::benchmark_data_smithi169_60938_object161471:head# 2024-02-12T16:18:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3efde77:::benchmark_data_smithi169_60938_object164100:head# 2024-02-12T16:18:08.861 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3f2ca3c:::benchmark_data_smithi169_60938_object146376:head# 2024-02-12T16:18:08.862 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3f798e2:::benchmark_data_smithi169_60938_object160787:head# 2024-02-12T16:18:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3f89c2d:::benchmark_data_smithi169_60938_object145288:head# 2024-02-12T16:18:08.863 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3fa329c:::benchmark_data_smithi169_60938_object140281:head# 2024-02-12T16:18:08.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3faeaaa:::benchmark_data_smithi169_60938_object165672:head# 2024-02-12T16:18:08.864 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3fdcfff:::benchmark_data_smithi169_60938_object157029:head# 2024-02-12T16:18:08.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:08.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f3ffb30b:::benchmark_data_smithi169_60938_object159605:head# 2024-02-12T16:18:08.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f400c90c:::benchmark_data_smithi169_60938_object153034:head# 2024-02-12T16:18:08.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4020561:::benchmark_data_smithi169_60938_object164113:head# 2024-02-12T16:18:08.867 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4030595:::benchmark_data_smithi169_60938_object161252:head# 2024-02-12T16:18:08.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f40354bc:::benchmark_data_smithi169_60938_object151281:head# 2024-02-12T16:18:08.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4075981:::benchmark_data_smithi169_60938_object161386:head# 2024-02-12T16:18:08.868 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f40aa79a:::benchmark_data_smithi169_60938_object160361:head# 2024-02-12T16:18:08.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f40ce395:::benchmark_data_smithi169_60938_object138957:head# 2024-02-12T16:18:08.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f40e08e3:::benchmark_data_smithi169_60938_object142633:head# 2024-02-12T16:18:08.869 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f40e8375:::benchmark_data_smithi169_60938_object142657:head# 2024-02-12T16:18:08.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f41185e1:::benchmark_data_smithi169_60938_object166931:head# 2024-02-12T16:18:08.870 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f413dedd:::benchmark_data_smithi169_60938_object173238:head# 2024-02-12T16:18:08.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4143b41:::benchmark_data_smithi169_60938_object146374:head# 2024-02-12T16:18:08.871 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f417b486:::benchmark_data_smithi169_60938_object159569:head# 2024-02-12T16:18:08.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f41af79d:::benchmark_data_smithi169_60938_object153823:head# 2024-02-12T16:18:08.872 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f41b2467:::benchmark_data_smithi169_60938_object139714:head# 2024-02-12T16:18:08.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f41cedac:::benchmark_data_smithi169_60938_object159392:head# 2024-02-12T16:18:08.873 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4206bbd:::benchmark_data_smithi169_60938_object161371:head# 2024-02-12T16:18:08.874 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4220f51:::benchmark_data_smithi169_60938_object170578:head# 2024-02-12T16:18:08.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4256df4:::benchmark_data_smithi169_60938_object154313:head# 2024-02-12T16:18:08.875 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f42759ee:::benchmark_data_smithi169_60938_object142043:head# 2024-02-12T16:18:08.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4281318:::benchmark_data_smithi169_60938_object170487:head# 2024-02-12T16:18:08.876 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4282369:::benchmark_data_smithi169_60938_object164684:head# 2024-02-12T16:18:08.877 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f42831ee:::benchmark_data_smithi169_60938_object150710:head# 2024-02-12T16:18:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f429c1c4:::benchmark_data_smithi169_60938_object163513:head# 2024-02-12T16:18:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f429ece8:::benchmark_data_smithi169_60938_object154827:head# 2024-02-12T16:18:08.878 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f42e1ac8:::benchmark_data_smithi169_60938_object170244:head# 2024-02-12T16:18:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f430ebc7:::benchmark_data_smithi169_60938_object146565:head# 2024-02-12T16:18:08.879 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4366cca:::benchmark_data_smithi169_60938_object140539:head# 2024-02-12T16:18:08.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4369742:::benchmark_data_smithi169_60938_object171969:head# 2024-02-12T16:18:08.880 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f437813e:::benchmark_data_smithi169_60938_object164299:head# 2024-02-12T16:18:08.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4381f3a:::benchmark_data_smithi169_60938_object151973:head# 2024-02-12T16:18:08.881 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f43867b2:::benchmark_data_smithi169_60938_object152168:head# 2024-02-12T16:18:08.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f43a9730:::benchmark_data_smithi169_60938_object172395:head# 2024-02-12T16:18:08.882 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f43d3557:::benchmark_data_smithi169_60938_object142272:head# 2024-02-12T16:18:08.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f43d5029:::benchmark_data_smithi169_60938_object165887:head# 2024-02-12T16:18:08.883 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f43e625a:::benchmark_data_smithi169_60938_object139238:head# 2024-02-12T16:18:08.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f43ea917:::benchmark_data_smithi169_60938_object144999:head# 2024-02-12T16:18:08.884 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f443ac87:::benchmark_data_smithi169_60938_object153919:head# 2024-02-12T16:18:08.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4472e35:::benchmark_data_smithi169_60938_object140765:head# 2024-02-12T16:18:08.885 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4485acf:::benchmark_data_smithi169_60938_object141045:head# 2024-02-12T16:18:08.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f44b5491:::benchmark_data_smithi169_60938_object143301:head# 2024-02-12T16:18:08.886 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f44e613d:::benchmark_data_smithi169_60938_object162303:head# 2024-02-12T16:18:08.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f44f1a8e:::benchmark_data_smithi169_60938_object139541:head# 2024-02-12T16:18:08.887 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4500d70:::benchmark_data_smithi169_60938_object160562:head# 2024-02-12T16:18:08.888 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4523265:::benchmark_data_smithi169_60938_object146307:head# 2024-02-12T16:18:08.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f45346b3:::benchmark_data_smithi169_60938_object164529:head# 2024-02-12T16:18:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:08 smithi123 bash[19119]: cluster 2024-02-12T16:18:07.761941+0000 mgr.y (mgr.14146) 985 : cluster [DBG] pgmap v1311: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:08 smithi123 bash[19119]: audit 2024-02-12T16:18:08.182982+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? v1:172.21.15.121:0/1806690187' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:18:08.889 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4570097:::benchmark_data_smithi169_60938_object151739:head# 2024-02-12T16:18:08.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f45a864c:::benchmark_data_smithi169_60938_object161116:head# 2024-02-12T16:18:08.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f45e4f0a:::benchmark_data_smithi169_60938_object168547:head# 2024-02-12T16:18:08.890 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f45f8482:::benchmark_data_smithi169_60938_object165512:head# 2024-02-12T16:18:08.891 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f463d9bc:::benchmark_data_smithi169_60938_object146696:head# 2024-02-12T16:18:08.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4646349:::benchmark_data_smithi169_60938_object165509:head# 2024-02-12T16:18:08.892 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4663a4b:::benchmark_data_smithi169_60938_object153767:head# 2024-02-12T16:18:08.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4673c19:::benchmark_data_smithi169_60938_object157208:head# 2024-02-12T16:18:08.893 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4679ea1:::benchmark_data_smithi169_60938_object152773:head# 2024-02-12T16:18:08.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f46bc74a:::benchmark_data_smithi169_60938_object149866:head# 2024-02-12T16:18:08.894 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f46c8a17:::benchmark_data_smithi169_60938_object153786:head# 2024-02-12T16:18:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:08.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f46f6d14:::benchmark_data_smithi169_60938_object172827:head# 2024-02-12T16:18:08.896 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f47087db:::benchmark_data_smithi169_60938_object141698:head# 2024-02-12T16:18:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:08.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:08.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f47110df:::benchmark_data_smithi169_60938_object145715:head# 2024-02-12T16:18:08.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f471d60a:::benchmark_data_smithi169_60938_object155334:head# 2024-02-12T16:18:08.898 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f47349f1:::benchmark_data_smithi169_60938_object172405:head# 2024-02-12T16:18:08.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f473bed6:::benchmark_data_smithi169_60938_object168485:head# 2024-02-12T16:18:08.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4760d0c:::benchmark_data_smithi169_60938_object156160:head# 2024-02-12T16:18:08.899 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4794d9f:::benchmark_data_smithi169_60938_object166443:head# 2024-02-12T16:18:08.900 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4795ca8:::benchmark_data_smithi169_60938_object153763:head# 2024-02-12T16:18:08.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f47e8b2c:::benchmark_data_smithi169_60938_object155715:head# 2024-02-12T16:18:08.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f47f94fb:::benchmark_data_smithi169_60938_object139320:head# 2024-02-12T16:18:08.901 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f481bf0c:::benchmark_data_smithi169_60938_object152588:head# 2024-02-12T16:18:08.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f481e5d1:::benchmark_data_smithi169_60938_object144182:head# 2024-02-12T16:18:08.902 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4856e9a:::benchmark_data_smithi169_60938_object163425:head# 2024-02-12T16:18:08.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f486cb75:::benchmark_data_smithi169_60938_object151223:head# 2024-02-12T16:18:08.903 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4879593:::benchmark_data_smithi169_60938_object158480:head# 2024-02-12T16:18:08.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f48fa2cf:::benchmark_data_smithi169_60938_object168726:head# 2024-02-12T16:18:08.904 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4944516:::benchmark_data_smithi169_60938_object168360:head# 2024-02-12T16:18:08.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f49b12ba:::benchmark_data_smithi169_60938_object148776:head# 2024-02-12T16:18:08.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f49d4011:::benchmark_data_smithi169_60938_object142467:head# 2024-02-12T16:18:08.905 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f49f718f:::benchmark_data_smithi169_60938_object145455:head# 2024-02-12T16:18:08.906 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f49f92b1:::benchmark_data_smithi169_60938_object163045:head# 2024-02-12T16:18:08.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4a03130:::benchmark_data_smithi169_60938_object150089:head# 2024-02-12T16:18:08.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4a1ca0b:::benchmark_data_smithi169_60938_object159970:head# 2024-02-12T16:18:08.907 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4a3268b:::benchmark_data_smithi169_60938_object156575:head# 2024-02-12T16:18:08.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4a52049:::benchmark_data_smithi169_60938_object163158:head# 2024-02-12T16:18:08.908 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4a5b40e:::benchmark_data_smithi169_60938_object164616:head# 2024-02-12T16:18:08.909 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4a65154:::benchmark_data_smithi169_60938_object156958:head# 2024-02-12T16:18:08.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4a655a3:::benchmark_data_smithi169_60938_object159211:head# 2024-02-12T16:18:08.910 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4abc208:::benchmark_data_smithi169_60938_object169955:head# 2024-02-12T16:18:08.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ac544b:::benchmark_data_smithi169_60938_object150481:head# 2024-02-12T16:18:08.911 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ad0ebc:::benchmark_data_smithi169_60938_object166865:head# 2024-02-12T16:18:08.912 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ae7629:::benchmark_data_smithi169_60938_object164303:head# 2024-02-12T16:18:08.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ae883f:::benchmark_data_smithi169_60938_object170790:head# 2024-02-12T16:18:08.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4b073d4:::benchmark_data_smithi169_60938_object141118:head# 2024-02-12T16:18:08.913 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4b0a8c6:::benchmark_data_smithi169_60938_object141832:head# 2024-02-12T16:18:08.914 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4b25e41:::benchmark_data_smithi169_60938_object172140:head# 2024-02-12T16:18:08.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4b2c166:::benchmark_data_smithi169_60938_object164844:head# 2024-02-12T16:18:08.915 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4b3a4ed:::benchmark_data_smithi169_60938_object147826:head# 2024-02-12T16:18:08.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4b4d3ff:::benchmark_data_smithi169_60938_object172431:head# 2024-02-12T16:18:08.916 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ba5953:::benchmark_data_smithi169_60938_object144346:head# 2024-02-12T16:18:08.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4bca98d:::benchmark_data_smithi169_60938_object154520:head# 2024-02-12T16:18:08.917 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4bd4846:::benchmark_data_smithi169_60938_object166434:head# 2024-02-12T16:18:08.918 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4c01238:::benchmark_data_smithi169_60938_object146021:head# 2024-02-12T16:18:08.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4c1e00e:::benchmark_data_smithi169_60938_object167303:head# 2024-02-12T16:18:08.919 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4c21107:::benchmark_data_smithi169_60938_object158402:head# 2024-02-12T16:18:08.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4c7bb50:::benchmark_data_smithi169_60938_object159808:head# 2024-02-12T16:18:08.920 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4c80198:::benchmark_data_smithi169_60938_object166053:head# 2024-02-12T16:18:08.921 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4c84d1e:::benchmark_data_smithi169_60938_object141176:head# 2024-02-12T16:18:08.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4c8d18e:::benchmark_data_smithi169_60938_object160178:head# 2024-02-12T16:18:08.922 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4cc4c1a:::benchmark_data_smithi169_60938_object152283:head# 2024-02-12T16:18:08.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4cd7b97:::benchmark_data_smithi169_60938_object155176:head# 2024-02-12T16:18:08.923 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d0166c:::benchmark_data_smithi169_60938_object160981:head# 2024-02-12T16:18:08.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d07341:::benchmark_data_smithi169_60938_object145864:head# 2024-02-12T16:18:08.924 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d1d9fe:::benchmark_data_smithi169_60938_object141548:head# 2024-02-12T16:18:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d1e590:::benchmark_data_smithi169_60938_object173282:head# 2024-02-12T16:18:08.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d20d54:::benchmark_data_smithi169_60938_object150240:head# 2024-02-12T16:18:08.926 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d53409:::benchmark_data_smithi169_60938_object161605:head# 2024-02-12T16:18:08.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d5ed0c:::benchmark_data_smithi169_60938_object156061:head# 2024-02-12T16:18:08.927 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d65aff:::benchmark_data_smithi169_60938_object150384:head# 2024-02-12T16:18:08.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4d7b794:::benchmark_data_smithi169_60938_object162314:head# 2024-02-12T16:18:08.928 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4dc67cb:::benchmark_data_smithi169_60938_object154506:head# 2024-02-12T16:18:08.929 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4dec3f1:::benchmark_data_smithi169_60938_object166202:head# 2024-02-12T16:18:08.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4e0153d:::benchmark_data_smithi169_60938_object156376:head# 2024-02-12T16:18:08.930 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4e3c1d2:::benchmark_data_smithi169_60938_object171154:head# 2024-02-12T16:18:08.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4e3fef3:::benchmark_data_smithi169_60938_object139612:head# 2024-02-12T16:18:08.931 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4e7840b:::benchmark_data_smithi169_60938_object171581:head# 2024-02-12T16:18:08.935 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4eacb68:::benchmark_data_smithi169_60938_object156325:head# 2024-02-12T16:18:08.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4eb365d:::benchmark_data_smithi169_60938_object173036:head# 2024-02-12T16:18:08.936 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ed9c21:::benchmark_data_smithi169_60938_object148964:head# 2024-02-12T16:18:08.937 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ef88a6:::benchmark_data_smithi169_60938_object166859:head# 2024-02-12T16:18:08.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4f21fd7:::benchmark_data_smithi169_60938_object148519:head# 2024-02-12T16:18:08.938 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4f331ee:::benchmark_data_smithi169_60938_object143313:head# 2024-02-12T16:18:08.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4f3b112:::benchmark_data_smithi169_60938_object148069:head# 2024-02-12T16:18:08.939 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4f8742a:::benchmark_data_smithi169_60938_object143512:head# 2024-02-12T16:18:08.940 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f4ffb5c4:::benchmark_data_smithi169_60938_object142615:head# 2024-02-12T16:18:08.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f500feb1:::benchmark_data_smithi169_60938_object158842:head# 2024-02-12T16:18:08.941 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5011ff9:::benchmark_data_smithi169_60938_object149304:head# 2024-02-12T16:18:08.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5014274:::benchmark_data_smithi169_60938_object157420:head# 2024-02-12T16:18:08.942 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f502ec26:::benchmark_data_smithi169_60938_object142775:head# 2024-02-12T16:18:08.943 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f50c02dd:::benchmark_data_smithi169_60938_object163086:head# 2024-02-12T16:18:08.944 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f50cabee:::benchmark_data_smithi169_60938_object169716:head# 2024-02-12T16:18:08.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f50ccc05:::benchmark_data_smithi169_60938_object149244:head# 2024-02-12T16:18:08.945 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f50e72dc:::benchmark_data_smithi169_60938_object155684:head# 2024-02-12T16:18:08.946 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5115f9a:::benchmark_data_smithi169_60938_object142975:head# 2024-02-12T16:18:08.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f515e8a8:::benchmark_data_smithi169_60938_object146490:head# 2024-02-12T16:18:08.947 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5171023:::benchmark_data_smithi169_60938_object154449:head# 2024-02-12T16:18:08.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f51a0238:::benchmark_data_smithi169_60938_object150232:head# 2024-02-12T16:18:08.948 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f51a7be9:::benchmark_data_smithi169_60938_object151659:head# 2024-02-12T16:18:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f51c4364:::benchmark_data_smithi169_60938_object173205:head# 2024-02-12T16:18:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f51cc232:::benchmark_data_smithi169_60938_object168520:head# 2024-02-12T16:18:08.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f51e8bd5:::benchmark_data_smithi169_60938_object169224:head# 2024-02-12T16:18:08.950 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5224c35:::benchmark_data_smithi169_60938_object159128:head# 2024-02-12T16:18:08.951 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5279278:::benchmark_data_smithi169_60938_object149412:head# 2024-02-12T16:18:08.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f52960b2:::benchmark_data_smithi169_60938_object148654:head# 2024-02-12T16:18:08.952 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f52bd7fb:::benchmark_data_smithi169_60938_object150351:head# 2024-02-12T16:18:08.953 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f52c9de4:::benchmark_data_smithi169_60938_object160229:head# 2024-02-12T16:18:08.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f52d81ec:::benchmark_data_smithi169_60938_object171032:head# 2024-02-12T16:18:08.954 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f52ed210:::benchmark_data_smithi169_60938_object166233:head# 2024-02-12T16:18:08.955 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f531ffea:::benchmark_data_smithi169_60938_object148490:head# 2024-02-12T16:18:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5329472:::benchmark_data_smithi169_60938_object141270:head# 2024-02-12T16:18:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53638c0:::benchmark_data_smithi169_60938_object155568:head# 2024-02-12T16:18:08.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5370bbd:::benchmark_data_smithi169_60938_object164061:head# 2024-02-12T16:18:08.957 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f537460c:::benchmark_data_smithi169_60938_object153940:head# 2024-02-12T16:18:08.958 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53a5890:::benchmark_data_smithi169_60938_object141191:head# 2024-02-12T16:18:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53abd23:::benchmark_data_smithi169_60938_object145946:head# 2024-02-12T16:18:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53b1f6f:::benchmark_data_smithi169_60938_object170309:head# 2024-02-12T16:18:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53bc8ed:::benchmark_data_smithi169_60938_object138832:head# 2024-02-12T16:18:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53d214a:::benchmark_data_smithi169_60938_object143623:head# 2024-02-12T16:18:08.961 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53df32c:::benchmark_data_smithi169_60938_object164553:head# 2024-02-12T16:18:08.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f53f98c7:::benchmark_data_smithi169_60938_object145483:head# 2024-02-12T16:18:08.962 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5423fd4:::benchmark_data_smithi169_60938_object145086:head# 2024-02-12T16:18:08.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f548c706:::benchmark_data_smithi169_60938_object172636:head# 2024-02-12T16:18:08.963 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f548feb1:::benchmark_data_smithi169_60938_object170616:head# 2024-02-12T16:18:08.964 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5496328:::benchmark_data_smithi169_60938_object166858:head# 2024-02-12T16:18:08.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f54b3b05:::benchmark_data_smithi169_60938_object162120:head# 2024-02-12T16:18:08.965 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f54bd80a:::benchmark_data_smithi169_60938_object145100:head# 2024-02-12T16:18:08.966 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f551e4de:::benchmark_data_smithi169_60938_object168995:head# 2024-02-12T16:18:08.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5547b8f:::benchmark_data_smithi169_60938_object152952:head# 2024-02-12T16:18:08.967 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f557dcf3:::benchmark_data_smithi169_60938_object151357:head# 2024-02-12T16:18:08.968 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5581bb8:::benchmark_data_smithi169_60938_object153526:head# 2024-02-12T16:18:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f558214a:::benchmark_data_smithi169_60938_object164145:head# 2024-02-12T16:18:08.969 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f55c198a:::benchmark_data_smithi169_60938_object138750:head# 2024-02-12T16:18:08.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f55e1e2e:::benchmark_data_smithi169_60938_object158882:head# 2024-02-12T16:18:08.970 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5605616:::benchmark_data_smithi169_60938_object169718:head# 2024-02-12T16:18:08.971 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f56077cb:::benchmark_data_smithi169_60938_object153570:head# 2024-02-12T16:18:08.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5645432:::benchmark_data_smithi169_60938_object157799:head# 2024-02-12T16:18:08.972 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5646459:::benchmark_data_smithi169_60938_object162497:head# 2024-02-12T16:18:08.973 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f564f03f:::benchmark_data_smithi169_60938_object168684:head# 2024-02-12T16:18:08.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5654721:::benchmark_data_smithi169_60938_object171446:head# 2024-02-12T16:18:08.974 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f565cd0d:::benchmark_data_smithi169_60938_object168565:head# 2024-02-12T16:18:08.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f566579c:::benchmark_data_smithi169_60938_object169518:head# 2024-02-12T16:18:08.975 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5677941:::benchmark_data_smithi169_60938_object142224:head# 2024-02-12T16:18:08.976 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5688135:::benchmark_data_smithi169_60938_object163066:head# 2024-02-12T16:18:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f569a1b2:::benchmark_data_smithi169_60938_object160904:head# 2024-02-12T16:18:08.977 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f569b77a:::benchmark_data_smithi169_60938_object153832:head# 2024-02-12T16:18:08.978 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f56c7ba7:::benchmark_data_smithi169_60938_object159983:head# 2024-02-12T16:18:08.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f571f903:::benchmark_data_smithi169_60938_object139087:head# 2024-02-12T16:18:08.979 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f574298e:::benchmark_data_smithi169_60938_object141661:head# 2024-02-12T16:18:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f57441d1:::benchmark_data_smithi169_60938_object164309:head# 2024-02-12T16:18:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f574e562:::benchmark_data_smithi169_60938_object154712:head# 2024-02-12T16:18:08.981 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5757225:::benchmark_data_smithi169_60938_object170136:head# 2024-02-12T16:18:08.982 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5759ccd:::benchmark_data_smithi169_60938_object153958:head# 2024-02-12T16:18:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f575f94d:::benchmark_data_smithi169_60938_object161515:head# 2024-02-12T16:18:08.983 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f577fd19:::benchmark_data_smithi169_60938_object169889:head# 2024-02-12T16:18:08.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5795845:::benchmark_data_smithi169_60938_object163555:head# 2024-02-12T16:18:08.984 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f57aa24a:::benchmark_data_smithi169_60938_object167805:head# 2024-02-12T16:18:08.985 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f580220d:::benchmark_data_smithi169_60938_object164721:head# 2024-02-12T16:18:08.986 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f581c458:::benchmark_data_smithi169_60938_object141512:head# 2024-02-12T16:18:08.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f582c5c5:::benchmark_data_smithi169_60938_object156510:head# 2024-02-12T16:18:08.987 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5852675:::benchmark_data_smithi169_60938_object151451:head# 2024-02-12T16:18:08.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5860e5e:::benchmark_data_smithi169_60938_object160541:head# 2024-02-12T16:18:08.988 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5869c29:::benchmark_data_smithi169_60938_object170160:head# 2024-02-12T16:18:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f58a9f70:::benchmark_data_smithi169_60938_object139906:head# 2024-02-12T16:18:08.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f58cab5f:::benchmark_data_smithi169_60938_object167535:head# 2024-02-12T16:18:08.990 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f58d2bb0:::benchmark_data_smithi169_60938_object150367:head# 2024-02-12T16:18:08.991 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5912d33:::benchmark_data_smithi169_60938_object151917:head# 2024-02-12T16:18:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5927674:::benchmark_data_smithi169_60938_object157847:head# 2024-02-12T16:18:08.992 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f592a884:::benchmark_data_smithi169_60938_object139955:head# 2024-02-12T16:18:08.993 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5939d09:::benchmark_data_smithi169_60938_object146488:head# 2024-02-12T16:18:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f59bd3c4:::benchmark_data_smithi169_60938_object145244:head# 2024-02-12T16:18:08.994 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f59c7d61:::benchmark_data_smithi169_60938_object141578:head# 2024-02-12T16:18:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f59dac21:::benchmark_data_smithi169_60938_object167970:head# 2024-02-12T16:18:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5a00380:::benchmark_data_smithi169_60938_object169305:head# 2024-02-12T16:18:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5a34988:::benchmark_data_smithi169_60938_object142524:head# 2024-02-12T16:18:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5a66a4a:::benchmark_data_smithi169_60938_object163299:head# 2024-02-12T16:18:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5a87381:::benchmark_data_smithi169_60938_object150265:head# 2024-02-12T16:18:08.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5ab7078:::benchmark_data_smithi169_60938_object144913:head# 2024-02-12T16:18:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5ac4dac:::benchmark_data_smithi169_60938_object142309:head# 2024-02-12T16:18:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b013d4:::benchmark_data_smithi169_60938_object141001:head# 2024-02-12T16:18:09.000 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b128f4:::benchmark_data_smithi169_60938_object168027:head# 2024-02-12T16:18:09.001 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b28659:::benchmark_data_smithi169_60938_object150824:head# 2024-02-12T16:18:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b363e9:::benchmark_data_smithi169_60938_object155912:head# 2024-02-12T16:18:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b4c566:::benchmark_data_smithi169_60938_object161696:head# 2024-02-12T16:18:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b53e2b:::benchmark_data_smithi169_60938_object142862:head# 2024-02-12T16:18:09.003 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b5d7e6:::benchmark_data_smithi169_60938_object169448:head# 2024-02-12T16:18:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5b9c2ee:::benchmark_data_smithi169_60938_object163156:head# 2024-02-12T16:18:09.004 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5bf558a:::benchmark_data_smithi169_60938_object143078:head# 2024-02-12T16:18:09.005 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c0dc73:::benchmark_data_smithi169_60938_object149192:head# 2024-02-12T16:18:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c37d9c:::benchmark_data_smithi169_60938_object138561:head# 2024-02-12T16:18:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c3f5f5:::benchmark_data_smithi169_60938_object172023:head# 2024-02-12T16:18:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c6b087:::benchmark_data_smithi169_60938_object163614:head# 2024-02-12T16:18:09.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c70e17:::benchmark_data_smithi169_60938_object155947:head# 2024-02-12T16:18:09.008 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c72795:::benchmark_data_smithi169_60938_object162341:head# 2024-02-12T16:18:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c73f1f:::benchmark_data_smithi169_60938_object148838:head# 2024-02-12T16:18:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5c96368:::benchmark_data_smithi169_60938_object149566:head# 2024-02-12T16:18:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5cb4f44:::benchmark_data_smithi169_60938_object153181:head# 2024-02-12T16:18:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5cba01f:::benchmark_data_smithi169_60938_object169601:head# 2024-02-12T16:18:09.011 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5d04315:::benchmark_data_smithi169_60938_object139019:head# 2024-02-12T16:18:09.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5d04fa8:::benchmark_data_smithi169_60938_object150901:head# 2024-02-12T16:18:09.012 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5d0ba9f:::benchmark_data_smithi169_60938_object150156:head# 2024-02-12T16:18:09.013 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5d3dac7:::benchmark_data_smithi169_60938_object147859:head# 2024-02-12T16:18:09.014 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5d5cb62:::benchmark_data_smithi169_60938_object165991:head# 2024-02-12T16:18:09.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5db3562:::benchmark_data_smithi169_60938_object139746:head# 2024-02-12T16:18:09.015 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5dbd196:::benchmark_data_smithi169_60938_object141881:head# 2024-02-12T16:18:09.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5dd0628:::benchmark_data_smithi169_60938_object154174:head# 2024-02-12T16:18:09.016 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5df8183:::benchmark_data_smithi169_60938_object143917:head# 2024-02-12T16:18:09.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5e3ad3b:::benchmark_data_smithi169_60938_object167508:head# 2024-02-12T16:18:09.017 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5e4cea9:::benchmark_data_smithi169_60938_object159874:head# 2024-02-12T16:18:09.018 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5e6cbc5:::benchmark_data_smithi169_60938_object141125:head# 2024-02-12T16:18:09.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5e7f370:::benchmark_data_smithi169_60938_object150381:head# 2024-02-12T16:18:09.019 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5ea9280:::benchmark_data_smithi169_60938_object161638:head# 2024-02-12T16:18:09.020 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5ebce47:::benchmark_data_smithi169_60938_object143970:head# 2024-02-12T16:18:09.021 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5ec829a:::benchmark_data_smithi169_60938_object152687:head# 2024-02-12T16:18:09.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5f03136:::benchmark_data_smithi169_60938_object172688:head# 2024-02-12T16:18:09.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5f0a061:::benchmark_data_smithi169_60938_object139487:head# 2024-02-12T16:18:09.022 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5f0c9de:::benchmark_data_smithi169_60938_object139705:head# 2024-02-12T16:18:09.023 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5f18c3f:::benchmark_data_smithi169_60938_object164194:head# 2024-02-12T16:18:09.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5f1d1ca:::benchmark_data_smithi169_60938_object169144:head# 2024-02-12T16:18:09.024 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5f3991d:::benchmark_data_smithi169_60938_object150248:head# 2024-02-12T16:18:09.025 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5f8e88b:::benchmark_data_smithi169_60938_object166464:head# 2024-02-12T16:18:09.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f5fa8a4d:::benchmark_data_smithi169_60938_object156411:head# 2024-02-12T16:18:09.026 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f600da82:::benchmark_data_smithi169_60938_object170341:head# 2024-02-12T16:18:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f602b62a:::benchmark_data_smithi169_60938_object142670:head# 2024-02-12T16:18:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f604e917:::benchmark_data_smithi169_60938_object159149:head# 2024-02-12T16:18:09.028 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6069959:::benchmark_data_smithi169_60938_object163516:head# 2024-02-12T16:18:09.029 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6077d8e:::benchmark_data_smithi169_60938_object154331:head# 2024-02-12T16:18:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f61396c4:::benchmark_data_smithi169_60938_object155387:head# 2024-02-12T16:18:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6161947:::benchmark_data_smithi169_60938_object147619:head# 2024-02-12T16:18:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f61952a5:::benchmark_data_smithi169_60938_object163326:head# 2024-02-12T16:18:09.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f61beb97:::benchmark_data_smithi169_60938_object156515:head# 2024-02-12T16:18:09.032 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f61ec495:::benchmark_data_smithi169_60938_object140483:head# 2024-02-12T16:18:09.033 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6217ae6:::benchmark_data_smithi169_60938_object140838:head# 2024-02-12T16:18:09.034 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6240fc9:::benchmark_data_smithi169_60938_object154542:head# 2024-02-12T16:18:09.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f625ee10:::benchmark_data_smithi169_60938_object157910:head# 2024-02-12T16:18:09.035 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f62c52af:::benchmark_data_smithi169_60938_object147302:head# 2024-02-12T16:18:09.036 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f62f8ad5:::benchmark_data_smithi169_60938_object154533:head# 2024-02-12T16:18:09.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:09.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6354c37:::benchmark_data_smithi169_60938_object168429:head# 2024-02-12T16:18:09.037 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f63bddf0:::benchmark_data_smithi169_60938_object172666:head# 2024-02-12T16:18:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f63cdf21:::benchmark_data_smithi169_60938_object164296:head# 2024-02-12T16:18:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6410648:::benchmark_data_smithi169_60938_object145480:head# 2024-02-12T16:18:09.039 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6439a5f:::benchmark_data_smithi169_60938_object164554:head# 2024-02-12T16:18:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:09.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:09.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f643a657:::benchmark_data_smithi169_60938_object161946:head# 2024-02-12T16:18:09.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f645b4b5:::benchmark_data_smithi169_60938_object152524:head# 2024-02-12T16:18:09.041 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f648197c:::benchmark_data_smithi169_60938_object142647:head# 2024-02-12T16:18:09.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f648fbb6:::benchmark_data_smithi169_60938_object164454:head# 2024-02-12T16:18:09.042 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f64f75a9:::benchmark_data_smithi169_60938_object142991:head# 2024-02-12T16:18:09.043 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f65785c1:::benchmark_data_smithi169_60938_object153639:head# 2024-02-12T16:18:09.044 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f657ec26:::benchmark_data_smithi169_60938_object166906:head# 2024-02-12T16:18:09.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6580b9f:::benchmark_data_smithi169_60938_object139077:head# 2024-02-12T16:18:09.045 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f65af97a:::benchmark_data_smithi169_60938_object146295:head# 2024-02-12T16:18:09.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f662c160:::benchmark_data_smithi169_60938_object163278:head# 2024-02-12T16:18:09.046 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6653465:::benchmark_data_smithi169_60938_object147609:head# 2024-02-12T16:18:09.047 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f665e3a5:::benchmark_data_smithi169_60938_object172917:head# 2024-02-12T16:18:09.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6674aa1:::benchmark_data_smithi169_60938_object152672:head# 2024-02-12T16:18:09.048 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f668fb3a:::benchmark_data_smithi169_60938_object161223:head# 2024-02-12T16:18:09.049 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f66d021b:::benchmark_data_smithi169_60938_object141179:head# 2024-02-12T16:18:09.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f66fe986:::benchmark_data_smithi169_60938_object154094:head# 2024-02-12T16:18:09.050 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f66ffb86:::benchmark_data_smithi169_60938_object165227:head# 2024-02-12T16:18:09.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6705adb:::benchmark_data_smithi169_60938_object155102:head# 2024-02-12T16:18:09.051 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f67163a0:::benchmark_data_smithi169_60938_object142228:head# 2024-02-12T16:18:09.052 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f673764d:::benchmark_data_smithi169_60938_object140379:head# 2024-02-12T16:18:09.053 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6740edf:::benchmark_data_smithi169_60938_object142129:head# 2024-02-12T16:18:09.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6795921:::benchmark_data_smithi169_60938_object168843:head# 2024-02-12T16:18:09.054 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f67b76c4:::benchmark_data_smithi169_60938_object149730:head# 2024-02-12T16:18:09.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f67df4b2:::benchmark_data_smithi169_60938_object146450:head# 2024-02-12T16:18:09.055 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f67ed959:::benchmark_data_smithi169_60938_object148822:head# 2024-02-12T16:18:09.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f67f731e:::benchmark_data_smithi169_60938_object155762:head# 2024-02-12T16:18:09.056 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f681de7e:::benchmark_data_smithi169_60938_object168089:head# 2024-02-12T16:18:09.057 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f682e400:::benchmark_data_smithi169_60938_object161401:head# 2024-02-12T16:18:09.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f68414b9:::benchmark_data_smithi169_60938_object160528:head# 2024-02-12T16:18:09.058 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6849ee1:::benchmark_data_smithi169_60938_object170960:head# 2024-02-12T16:18:09.059 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f684de00:::benchmark_data_smithi169_60938_object172936:head# 2024-02-12T16:18:09.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f686d9ab:::benchmark_data_smithi169_60938_object141831:head# 2024-02-12T16:18:09.060 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6882766:::benchmark_data_smithi169_60938_object158613:head# 2024-02-12T16:18:09.061 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f688349f:::benchmark_data_smithi169_60938_object160297:head# 2024-02-12T16:18:09.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f68bf701:::benchmark_data_smithi169_60938_object141795:head# 2024-02-12T16:18:09.062 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f68cf5cd:::benchmark_data_smithi169_60938_object153232:head# 2024-02-12T16:18:09.063 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6900296:::benchmark_data_smithi169_60938_object145617:head# 2024-02-12T16:18:09.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f693cdcf:::benchmark_data_smithi169_60938_object140380:head# 2024-02-12T16:18:09.064 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6964ebf:::benchmark_data_smithi169_60938_object168354:head# 2024-02-12T16:18:09.065 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f69845d6:::benchmark_data_smithi169_60938_object160302:head# 2024-02-12T16:18:09.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f698fcce:::benchmark_data_smithi169_60938_object140670:head# 2024-02-12T16:18:09.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6997ded:::benchmark_data_smithi169_60938_object149589:head# 2024-02-12T16:18:09.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f69ad12a:::benchmark_data_smithi169_60938_object160916:head# 2024-02-12T16:18:09.067 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f69e2b21:::benchmark_data_smithi169_60938_object149551:head# 2024-02-12T16:18:09.068 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6a0308d:::benchmark_data_smithi169_60938_object166325:head# 2024-02-12T16:18:09.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6a11ada:::benchmark_data_smithi169_60938_object150463:head# 2024-02-12T16:18:09.069 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6a1c736:::benchmark_data_smithi169_60938_object161527:head# 2024-02-12T16:18:09.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6a27fb2:::benchmark_data_smithi169_60938_object144793:head# 2024-02-12T16:18:09.070 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6a4365c:::benchmark_data_smithi169_60938_object172112:head# 2024-02-12T16:18:09.071 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6a7eaf7:::benchmark_data_smithi169_60938_object152789:head# 2024-02-12T16:18:09.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ab68ce:::benchmark_data_smithi169_60938_object139530:head# 2024-02-12T16:18:09.072 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ac0753:::benchmark_data_smithi169_60938_object160369:head# 2024-02-12T16:18:09.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ac92a4:::benchmark_data_smithi169_60938_object155228:head# 2024-02-12T16:18:09.073 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ad8495:::benchmark_data_smithi169_60938_object150143:head# 2024-02-12T16:18:09.074 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6af4a2b:::benchmark_data_smithi169_60938_object147915:head# 2024-02-12T16:18:09.075 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6b0220f:::benchmark_data_smithi169_60938_object151495:head# 2024-02-12T16:18:09.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6b26675:::benchmark_data_smithi169_60938_object152367:head# 2024-02-12T16:18:09.076 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6b28aa9:::benchmark_data_smithi169_60938_object170267:head# 2024-02-12T16:18:09.077 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6b3955d:::benchmark_data_smithi169_60938_object141695:head# 2024-02-12T16:18:09.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6b41200:::benchmark_data_smithi169_60938_object155574:head# 2024-02-12T16:18:09.078 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6b7d581:::benchmark_data_smithi169_60938_object157637:head# 2024-02-12T16:18:09.079 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6bab837:::benchmark_data_smithi169_60938_object161351:head# 2024-02-12T16:18:09.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6c1dc44:::benchmark_data_smithi169_60938_object157210:head# 2024-02-12T16:18:09.080 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6c9a017:::benchmark_data_smithi169_60938_object160939:head# 2024-02-12T16:18:09.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6cffa30:::benchmark_data_smithi169_60938_object171144:head# 2024-02-12T16:18:09.081 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6d319b4:::benchmark_data_smithi169_60938_object141417:head# 2024-02-12T16:18:09.082 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6d76e2b:::benchmark_data_smithi169_60938_object165203:head# 2024-02-12T16:18:09.083 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6d99c69:::benchmark_data_smithi169_60938_object171254:head# 2024-02-12T16:18:09.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6da9d0d:::benchmark_data_smithi169_60938_object165816:head# 2024-02-12T16:18:09.085 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6dd7292:::benchmark_data_smithi169_60938_object146049:head# 2024-02-12T16:18:09.086 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ddf645:::benchmark_data_smithi169_60938_object169121:head# 2024-02-12T16:18:09.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6de93c5:::benchmark_data_smithi169_60938_object161299:head# 2024-02-12T16:18:09.087 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6dfb0d7:::benchmark_data_smithi169_60938_object148511:head# 2024-02-12T16:18:09.088 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6e2c851:::benchmark_data_smithi169_60938_object169533:head# 2024-02-12T16:18:09.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6e3450a:::benchmark_data_smithi169_60938_object164245:head# 2024-02-12T16:18:09.089 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6e42d31:::benchmark_data_smithi169_60938_object146299:head# 2024-02-12T16:18:09.090 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6e626ae:::benchmark_data_smithi169_60938_object148825:head# 2024-02-12T16:18:09.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6e64c61:::benchmark_data_smithi169_60938_object140181:head# 2024-02-12T16:18:09.091 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ea2a88:::benchmark_data_smithi169_60938_object169595:head# 2024-02-12T16:18:09.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6eb50cf:::benchmark_data_smithi169_60938_object138254:head# 2024-02-12T16:18:09.092 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6eb5d7d:::benchmark_data_smithi169_60938_object169366:head# 2024-02-12T16:18:09.093 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ebe209:::benchmark_data_smithi169_60938_object153932:head# 2024-02-12T16:18:09.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6ebeb67:::benchmark_data_smithi169_60938_object162167:head# 2024-02-12T16:18:09.094 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f0ab55:::benchmark_data_smithi169_60938_object157703:head# 2024-02-12T16:18:09.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f108f1:::benchmark_data_smithi169_60938_object146467:head# 2024-02-12T16:18:09.095 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f17431:::benchmark_data_smithi169_60938_object158742:head# 2024-02-12T16:18:09.096 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f1b23a:::benchmark_data_smithi169_60938_object165688:head# 2024-02-12T16:18:09.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f1e67c:::benchmark_data_smithi169_60938_object164887:head# 2024-02-12T16:18:09.097 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f30c85:::benchmark_data_smithi169_60938_object143892:head# 2024-02-12T16:18:09.098 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f4c340:::benchmark_data_smithi169_60938_object166535:head# 2024-02-12T16:18:09.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f7ae9b:::benchmark_data_smithi169_60938_object172584:head# 2024-02-12T16:18:09.099 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f84a1f:::benchmark_data_smithi169_60938_object164354:head# 2024-02-12T16:18:09.100 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6f8cf34:::benchmark_data_smithi169_60938_object152846:head# 2024-02-12T16:18:09.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6fa2668:::benchmark_data_smithi169_60938_object156226:head# 2024-02-12T16:18:09.101 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6fcfabd:::benchmark_data_smithi169_60938_object149900:head# 2024-02-12T16:18:09.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f6fdc19e:::benchmark_data_smithi169_60938_object148090:head# 2024-02-12T16:18:09.102 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f700a68c:::benchmark_data_smithi169_60938_object152849:head# 2024-02-12T16:18:09.103 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f700f132:::benchmark_data_smithi169_60938_object155226:head# 2024-02-12T16:18:09.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f70746a6:::benchmark_data_smithi169_60938_object152972:head# 2024-02-12T16:18:09.104 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f70a9243:::benchmark_data_smithi169_60938_object143629:head# 2024-02-12T16:18:09.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f70b2182:::benchmark_data_smithi169_60938_object144970:head# 2024-02-12T16:18:09.105 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f70c0c33:::benchmark_data_smithi169_60938_object156396:head# 2024-02-12T16:18:09.106 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f70ffcca:::benchmark_data_smithi169_60938_object165578:head# 2024-02-12T16:18:09.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f710fd49:::benchmark_data_smithi169_60938_object152737:head# 2024-02-12T16:18:09.107 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7128243:::benchmark_data_smithi169_60938_object156914:head# 2024-02-12T16:18:09.108 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f712a073:::benchmark_data_smithi169_60938_object155031:head# 2024-02-12T16:18:09.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f714bf9e:::benchmark_data_smithi169_60938_object165632:head# 2024-02-12T16:18:09.109 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f716b978:::benchmark_data_smithi169_60938_object155617:head# 2024-02-12T16:18:09.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f71c48c1:::benchmark_data_smithi169_60938_object145635:head# 2024-02-12T16:18:09.110 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7219477:::benchmark_data_smithi169_60938_object154026:head# 2024-02-12T16:18:09.111 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7227f88:::benchmark_data_smithi169_60938_object166749:head# 2024-02-12T16:18:09.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f723213b:::benchmark_data_smithi169_60938_object148191:head# 2024-02-12T16:18:09.112 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7236604:::benchmark_data_smithi169_60938_object147422:head# 2024-02-12T16:18:09.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f725bcfe:::benchmark_data_smithi169_60938_object172741:head# 2024-02-12T16:18:09.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f726a40d:::benchmark_data_smithi169_60938_object141874:head# 2024-02-12T16:18:09.114 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f726e68a:::benchmark_data_smithi169_60938_object162962:head# 2024-02-12T16:18:09.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7277359:::benchmark_data_smithi169_60938_object152458:head# 2024-02-12T16:18:09.115 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f72b5f2d:::benchmark_data_smithi169_60938_object155662:head# 2024-02-12T16:18:09.116 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f72ddad5:::benchmark_data_smithi169_60938_object155241:head# 2024-02-12T16:18:09.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f72f0993:::benchmark_data_smithi169_60938_object156944:head# 2024-02-12T16:18:09.117 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f730e857:::benchmark_data_smithi169_60938_object172662:head# 2024-02-12T16:18:09.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7329014:::benchmark_data_smithi169_60938_object162022:head# 2024-02-12T16:18:09.118 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f737c2c4:::benchmark_data_smithi169_60938_object148315:head# 2024-02-12T16:18:09.119 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f737c40a:::benchmark_data_smithi169_60938_object156074:head# 2024-02-12T16:18:09.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7393928:::benchmark_data_smithi169_60938_object147840:head# 2024-02-12T16:18:09.120 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f739821a:::benchmark_data_smithi169_60938_object164360:head# 2024-02-12T16:18:09.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f73a8b4b:::benchmark_data_smithi169_60938_object140967:head# 2024-02-12T16:18:09.121 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f73b9571:::benchmark_data_smithi169_60938_object154193:head# 2024-02-12T16:18:09.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f73d029c:::benchmark_data_smithi169_60938_object168850:head# 2024-02-12T16:18:09.122 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f73fcf2e:::benchmark_data_smithi169_60938_object159809:head# 2024-02-12T16:18:09.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f741dd7d:::benchmark_data_smithi169_60938_object170885:head# 2024-02-12T16:18:09.123 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7421713:::benchmark_data_smithi169_60938_object163046:head# 2024-02-12T16:18:09.124 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7423de4:::benchmark_data_smithi169_60938_object146393:head# 2024-02-12T16:18:09.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f746091e:::benchmark_data_smithi169_60938_object165990:head# 2024-02-12T16:18:09.125 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7465d0f:::benchmark_data_smithi169_60938_object151045:head# 2024-02-12T16:18:09.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7473f9e:::benchmark_data_smithi169_60938_object161298:head# 2024-02-12T16:18:09.126 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f74769b7:::benchmark_data_smithi169_60938_object154951:head# 2024-02-12T16:18:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f747ed1d:::benchmark_data_smithi169_60938_object171962:head# 2024-02-12T16:18:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f74991bd:::benchmark_data_smithi169_60938_object159107:head# 2024-02-12T16:18:09.128 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f750e25b:::benchmark_data_smithi169_60938_object139715:head# 2024-02-12T16:18:09.129 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f753408c:::benchmark_data_smithi169_60938_object143555:head# 2024-02-12T16:18:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f753711c:::benchmark_data_smithi169_60938_object147367:head# 2024-02-12T16:18:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f75713e5:::benchmark_data_smithi169_60938_object153572:head# 2024-02-12T16:18:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f760c720:::benchmark_data_smithi169_60938_object149479:head# 2024-02-12T16:18:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7616853:::benchmark_data_smithi169_60938_object155408:head# 2024-02-12T16:18:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7624c44:::benchmark_data_smithi169_60938_object155307:head# 2024-02-12T16:18:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7631ab2:::benchmark_data_smithi169_60938_object138258:head# 2024-02-12T16:18:09.133 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f76635ab:::benchmark_data_smithi169_60938_object148797:head# 2024-02-12T16:18:09.134 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7699bf3:::benchmark_data_smithi169_60938_object173192:head# 2024-02-12T16:18:09.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f76a537b:::benchmark_data_smithi169_60938_object158974:head# 2024-02-12T16:18:09.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f76b5e5b:::benchmark_data_smithi169_60938_object154159:head# 2024-02-12T16:18:09.135 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f76becd2:::benchmark_data_smithi169_60938_object164199:head# 2024-02-12T16:18:09.136 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f76d2d84:::benchmark_data_smithi169_60938_object171826:head# 2024-02-12T16:18:09.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f76d885a:::benchmark_data_smithi169_60938_object159077:head# 2024-02-12T16:18:09.137 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f775b87c:::benchmark_data_smithi169_60938_object140165:head# 2024-02-12T16:18:09.138 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f777b22a:::benchmark_data_smithi169_60938_object154192:head# 2024-02-12T16:18:09.139 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7781277:::benchmark_data_smithi169_60938_object142403:head# 2024-02-12T16:18:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f778570b:::benchmark_data_smithi169_60938_object146287:head# 2024-02-12T16:18:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f778c7cb:::benchmark_data_smithi169_60938_object163068:head# 2024-02-12T16:18:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7791073:::benchmark_data_smithi169_60938_object155857:head# 2024-02-12T16:18:09.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:09.141 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f77a0901:::benchmark_data_smithi169_60938_object167593:head# 2024-02-12T16:18:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f77c0236:::benchmark_data_smithi169_60938_object149113:head# 2024-02-12T16:18:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f77cfc1d:::benchmark_data_smithi169_60938_object152244:head# 2024-02-12T16:18:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f77dadf4:::benchmark_data_smithi169_60938_object151075:head# 2024-02-12T16:18:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f77e68c8:::benchmark_data_smithi169_60938_object166298:head# 2024-02-12T16:18:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f77ee9ce:::benchmark_data_smithi169_60938_object163317:head# 2024-02-12T16:18:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f784528e:::benchmark_data_smithi169_60938_object148745:head# 2024-02-12T16:18:09.145 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7848a17:::benchmark_data_smithi169_60938_object150042:head# 2024-02-12T16:18:09.146 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f784c6e9:::benchmark_data_smithi169_60938_object163461:head# 2024-02-12T16:18:09.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:08 smithi121 bash[18489]: cluster 2024-02-12T16:18:07.761941+0000 mgr.y (mgr.14146) 985 : cluster [DBG] pgmap v1311: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:09.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:08 smithi121 bash[18489]: audit 2024-02-12T16:18:08.182982+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? v1:172.21.15.121:0/1806690187' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:18:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f78912d6:::benchmark_data_smithi169_60938_object154423:head# 2024-02-12T16:18:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f78a29e4:::benchmark_data_smithi169_60938_object156364:head# 2024-02-12T16:18:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f78b2a43:::benchmark_data_smithi169_60938_object169204:head# 2024-02-12T16:18:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f78e0f1f:::benchmark_data_smithi169_60938_object156587:head# 2024-02-12T16:18:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7939553:::benchmark_data_smithi169_60938_object155885:head# 2024-02-12T16:18:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7944a47:::benchmark_data_smithi169_60938_object164401:head# 2024-02-12T16:18:09.150 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f795dd3a:::benchmark_data_smithi169_60938_object171331:head# 2024-02-12T16:18:09.151 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f79a3e44:::benchmark_data_smithi169_60938_object153033:head# 2024-02-12T16:18:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f79a7148:::benchmark_data_smithi169_60938_object138961:head# 2024-02-12T16:18:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f79cb411:::benchmark_data_smithi169_60938_object140860:head# 2024-02-12T16:18:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f79d15b5:::benchmark_data_smithi169_60938_object163757:head# 2024-02-12T16:18:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f79f862d:::benchmark_data_smithi169_60938_object155934:head# 2024-02-12T16:18:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f79f9a4a:::benchmark_data_smithi169_60938_object141927:head# 2024-02-12T16:18:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7a0671d:::benchmark_data_smithi169_60938_object171157:head# 2024-02-12T16:18:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7a377ed:::benchmark_data_smithi169_60938_object144818:head# 2024-02-12T16:18:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7a4a547:::benchmark_data_smithi169_60938_object165404:head# 2024-02-12T16:18:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7b40a89:::benchmark_data_smithi169_60938_object148128:head# 2024-02-12T16:18:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7b42897:::benchmark_data_smithi169_60938_object161456:head# 2024-02-12T16:18:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7b580fc:::benchmark_data_smithi169_60938_object139144:head# 2024-02-12T16:18:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7b7c258:::benchmark_data_smithi169_60938_object171617:head# 2024-02-12T16:18:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7b8d972:::benchmark_data_smithi169_60938_object160431:head# 2024-02-12T16:18:09.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7bb5fee:::benchmark_data_smithi169_60938_object144070:head# 2024-02-12T16:18:09.161 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7bb9f7a:::benchmark_data_smithi169_60938_object146218:head# 2024-02-12T16:18:09.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7bc52f8:::benchmark_data_smithi169_60938_object156388:head# 2024-02-12T16:18:09.162 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7bd56a5:::benchmark_data_smithi169_60938_object168772:head# 2024-02-12T16:18:09.163 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7bdf8b0:::benchmark_data_smithi169_60938_object165733:head# 2024-02-12T16:18:09.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7c0b5bf:::benchmark_data_smithi169_60938_object167335:head# 2024-02-12T16:18:09.164 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7c16fbf:::benchmark_data_smithi169_60938_object171337:head# 2024-02-12T16:18:09.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7c58c18:::benchmark_data_smithi169_60938_object172712:head# 2024-02-12T16:18:09.165 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7ca09e9:::benchmark_data_smithi169_60938_object170854:head# 2024-02-12T16:18:09.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7ca5966:::benchmark_data_smithi169_60938_object162415:head# 2024-02-12T16:18:09.166 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7cb03a1:::benchmark_data_smithi169_60938_object166875:head# 2024-02-12T16:18:09.167 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7cd1403:::benchmark_data_smithi169_60938_object139876:head# 2024-02-12T16:18:09.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7d2a78c:::benchmark_data_smithi169_60938_object166720:head# 2024-02-12T16:18:09.168 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7d42fa9:::benchmark_data_smithi169_60938_object147218:head# 2024-02-12T16:18:09.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:09.169 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7d637fc:::benchmark_data_smithi169_60938_object147773:head# 2024-02-12T16:18:09.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7d6c5e2:::benchmark_data_smithi169_60938_object156204:head# 2024-02-12T16:18:09.170 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7d83990:::benchmark_data_smithi169_60938_object149472:head# 2024-02-12T16:18:09.171 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7d85a23:::benchmark_data_smithi169_60938_object166377:head# 2024-02-12T16:18:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:09.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:09.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7da6be5:::benchmark_data_smithi169_60938_object161105:head# 2024-02-12T16:18:09.172 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7df1f08:::benchmark_data_smithi169_60938_object160925:head# 2024-02-12T16:18:09.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7e6f3d4:::benchmark_data_smithi169_60938_object157167:head# 2024-02-12T16:18:09.173 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7ee213c:::benchmark_data_smithi169_60938_object144748:head# 2024-02-12T16:18:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7ee96d2:::benchmark_data_smithi169_60938_object146148:head# 2024-02-12T16:18:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7f06d3d:::benchmark_data_smithi169_60938_object167328:head# 2024-02-12T16:18:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7f0735c:::benchmark_data_smithi169_60938_object162072:head# 2024-02-12T16:18:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7f1e75d:::benchmark_data_smithi169_60938_object167551:head# 2024-02-12T16:18:09.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7f35c7a:::benchmark_data_smithi169_60938_object163497:head# 2024-02-12T16:18:09.176 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7fab6fb:::benchmark_data_smithi169_60938_object143253:head# 2024-02-12T16:18:09.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7fb11f7:::benchmark_data_smithi169_60938_object162066:head# 2024-02-12T16:18:09.177 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f7ff5bb1:::benchmark_data_smithi169_60938_object165786:head# 2024-02-12T16:18:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8004384:::benchmark_data_smithi169_60938_object143697:head# 2024-02-12T16:18:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f800d4ca:::benchmark_data_smithi169_60938_object152853:head# 2024-02-12T16:18:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f801f6c9:::benchmark_data_smithi169_60938_object141905:head# 2024-02-12T16:18:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f802e144:::benchmark_data_smithi169_60938_object158295:head# 2024-02-12T16:18:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8030e3b:::benchmark_data_smithi169_60938_object139214:head# 2024-02-12T16:18:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8036a54:::benchmark_data_smithi169_60938_object147568:head# 2024-02-12T16:18:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f804639a:::benchmark_data_smithi169_60938_object162183:head# 2024-02-12T16:18:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8049831:::benchmark_data_smithi169_60938_object165406:head# 2024-02-12T16:18:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f80796bc:::benchmark_data_smithi169_60938_object169015:head# 2024-02-12T16:18:09.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f808a242:::benchmark_data_smithi169_60938_object150526:head# 2024-02-12T16:18:09.183 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f80ac997:::benchmark_data_smithi169_60938_object163349:head# 2024-02-12T16:18:09.184 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f80ae78a:::benchmark_data_smithi169_60938_object143269:head# 2024-02-12T16:18:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f80b17a0:::benchmark_data_smithi169_60938_object151326:head# 2024-02-12T16:18:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f80c9b3a:::benchmark_data_smithi169_60938_object164378:head# 2024-02-12T16:18:09.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f80cd8c2:::benchmark_data_smithi169_60938_object144175:head# 2024-02-12T16:18:09.186 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f810f68f:::benchmark_data_smithi169_60938_object153248:head# 2024-02-12T16:18:09.187 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f812d6ae:::benchmark_data_smithi169_60938_object167301:head# 2024-02-12T16:18:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8138e69:::benchmark_data_smithi169_60938_object160296:head# 2024-02-12T16:18:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8188f0d:::benchmark_data_smithi169_60938_object171311:head# 2024-02-12T16:18:09.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f81b3962:::benchmark_data_smithi169_60938_object163018:head# 2024-02-12T16:18:09.189 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f81eec35:::benchmark_data_smithi169_60938_object169371:head# 2024-02-12T16:18:09.190 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f821b6bc:::benchmark_data_smithi169_60938_object166896:head# 2024-02-12T16:18:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8220d24:::benchmark_data_smithi169_60938_object161968:head# 2024-02-12T16:18:09.191 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f823e6ff:::benchmark_data_smithi169_60938_object156638:head# 2024-02-12T16:18:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f824f17d:::benchmark_data_smithi169_60938_object159661:head# 2024-02-12T16:18:09.192 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8260777:::benchmark_data_smithi169_60938_object145001:head# 2024-02-12T16:18:09.193 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f82bdb57:::benchmark_data_smithi169_60938_object150980:head# 2024-02-12T16:18:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f830e2e8:::benchmark_data_smithi169_60938_object158130:head# 2024-02-12T16:18:09.194 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f831d352:::benchmark_data_smithi169_60938_object144229:head# 2024-02-12T16:18:09.195 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8333a7f:::benchmark_data_smithi169_60938_object146932:head# 2024-02-12T16:18:09.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f83373d1:::benchmark_data_smithi169_60938_object173107:head# 2024-02-12T16:18:09.196 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f834f0f3:::benchmark_data_smithi169_60938_object164055:head# 2024-02-12T16:18:09.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f834f99f:::benchmark_data_smithi169_60938_object140019:head# 2024-02-12T16:18:09.197 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f835bfdc:::benchmark_data_smithi169_60938_object148132:head# 2024-02-12T16:18:09.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8373f82:::benchmark_data_smithi169_60938_object157975:head# 2024-02-12T16:18:09.198 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f83bd51e:::benchmark_data_smithi169_60938_object147862:head# 2024-02-12T16:18:09.199 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f83d3877:::benchmark_data_smithi169_60938_object166693:head# 2024-02-12T16:18:09.200 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f83e5b4a:::benchmark_data_smithi169_60938_object146220:head# 2024-02-12T16:18:09.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8428537:::benchmark_data_smithi169_60938_object145853:head# 2024-02-12T16:18:09.201 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f843fa92:::benchmark_data_smithi169_60938_object139966:head# 2024-02-12T16:18:09.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f844e169:::benchmark_data_smithi169_60938_object147385:head# 2024-02-12T16:18:09.202 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8476887:::benchmark_data_smithi169_60938_object161746:head# 2024-02-12T16:18:09.203 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f84774be:::benchmark_data_smithi169_60938_object171010:head# 2024-02-12T16:18:09.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8492d8e:::benchmark_data_smithi169_60938_object170882:head# 2024-02-12T16:18:09.204 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f84a8776:::benchmark_data_smithi169_60938_object144609:head# 2024-02-12T16:18:09.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f84da7f2:::benchmark_data_smithi169_60938_object154255:head# 2024-02-12T16:18:09.205 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f853e241:::benchmark_data_smithi169_60938_object168318:head# 2024-02-12T16:18:09.206 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8561352:::benchmark_data_smithi169_60938_object173246:head# 2024-02-12T16:18:09.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f856f7a1:::benchmark_data_smithi169_60938_object160478:head# 2024-02-12T16:18:09.207 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8585575:::benchmark_data_smithi169_60938_object146646:head# 2024-02-12T16:18:09.208 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8588023:::benchmark_data_smithi169_60938_object153703:head# 2024-02-12T16:18:09.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f85b569a:::benchmark_data_smithi169_60938_object162062:head# 2024-02-12T16:18:09.209 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f85c97f3:::benchmark_data_smithi169_60938_object139498:head# 2024-02-12T16:18:09.210 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f85ca5e9:::benchmark_data_smithi169_60938_object166043:head# 2024-02-12T16:18:09.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f85cb916:::benchmark_data_smithi169_60938_object172463:head# 2024-02-12T16:18:09.211 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f85d1989:::benchmark_data_smithi169_60938_object162604:head# 2024-02-12T16:18:09.212 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f85d43dc:::benchmark_data_smithi169_60938_object139474:head# 2024-02-12T16:18:09.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8600c87:::benchmark_data_smithi169_60938_object154539:head# 2024-02-12T16:18:09.213 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f865be85:::benchmark_data_smithi169_60938_object154114:head# 2024-02-12T16:18:09.214 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8671b45:::benchmark_data_smithi169_60938_object144200:head# 2024-02-12T16:18:09.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f867b735:::benchmark_data_smithi169_60938_object147182:head# 2024-02-12T16:18:09.215 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8692ec5:::benchmark_data_smithi169_60938_object160771:head# 2024-02-12T16:18:09.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f86bb070:::benchmark_data_smithi169_60938_object164246:head# 2024-02-12T16:18:09.216 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f86d4f46:::benchmark_data_smithi169_60938_object144848:head# 2024-02-12T16:18:09.217 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f86d9ce5:::benchmark_data_smithi169_60938_object166240:head# 2024-02-12T16:18:09.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f871ee0d:::benchmark_data_smithi169_60938_object162451:head# 2024-02-12T16:18:09.218 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8770db8:::benchmark_data_smithi169_60938_object167528:head# 2024-02-12T16:18:09.219 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f877dca9:::benchmark_data_smithi169_60938_object145995:head# 2024-02-12T16:18:09.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f878770d:::benchmark_data_smithi169_60938_object149744:head# 2024-02-12T16:18:09.220 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8789529:::benchmark_data_smithi169_60938_object173316:head# 2024-02-12T16:18:09.221 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f87c019e:::benchmark_data_smithi169_60938_object167905:head# 2024-02-12T16:18:09.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8801c28:::benchmark_data_smithi169_60938_object161030:head# 2024-02-12T16:18:09.222 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8881db1:::benchmark_data_smithi169_60938_object166463:head# 2024-02-12T16:18:09.223 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f88c2c73:::benchmark_data_smithi169_60938_object140724:head# 2024-02-12T16:18:09.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f88c36d0:::benchmark_data_smithi169_60938_object146390:head# 2024-02-12T16:18:09.224 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f88c962f:::benchmark_data_smithi169_60938_object153226:head# 2024-02-12T16:18:09.225 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8934441:::benchmark_data_smithi169_60938_object139621:head# 2024-02-12T16:18:09.226 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f89370c2:::benchmark_data_smithi169_60938_object144047:head# 2024-02-12T16:18:09.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f893c388:::benchmark_data_smithi169_60938_object149394:head# 2024-02-12T16:18:09.227 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8943742:::benchmark_data_smithi169_60938_object146258:head# 2024-02-12T16:18:09.228 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8956256:::benchmark_data_smithi169_60938_object156873:head# 2024-02-12T16:18:09.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f897b8f2:::benchmark_data_smithi169_60938_object171293:head# 2024-02-12T16:18:09.229 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f89bdc50:::benchmark_data_smithi169_60938_object171432:head# 2024-02-12T16:18:09.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8a4c6b6:::benchmark_data_smithi169_60938_object159973:head# 2024-02-12T16:18:09.230 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8a993b1:::benchmark_data_smithi169_60938_object152600:head# 2024-02-12T16:18:09.231 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8afe4cb:::benchmark_data_smithi169_60938_object154339:head# 2024-02-12T16:18:09.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8b269af:::benchmark_data_smithi169_60938_object172488:head# 2024-02-12T16:18:09.232 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8b6c6ae:::benchmark_data_smithi169_60938_object148095:head# 2024-02-12T16:18:09.233 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8b92c47:::benchmark_data_smithi169_60938_object152164:head# 2024-02-12T16:18:09.234 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8b9fd31:::benchmark_data_smithi169_60938_object145479:head# 2024-02-12T16:18:09.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8bb455f:::benchmark_data_smithi169_60938_object151898:head# 2024-02-12T16:18:09.235 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8bfc65d:::benchmark_data_smithi169_60938_object150355:head# 2024-02-12T16:18:09.236 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8c23b3d:::benchmark_data_smithi169_60938_object143363:head# 2024-02-12T16:18:09.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8c5187c:::benchmark_data_smithi169_60938_object141732:head# 2024-02-12T16:18:09.237 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8c76810:::benchmark_data_smithi169_60938_object151204:head# 2024-02-12T16:18:09.238 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8c8a13e:::benchmark_data_smithi169_60938_object163907:head# 2024-02-12T16:18:09.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8c9e9b2:::benchmark_data_smithi169_60938_object165019:head# 2024-02-12T16:18:09.239 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8cb04d5:::benchmark_data_smithi169_60938_object167833:head# 2024-02-12T16:18:09.240 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8cb0a4f:::benchmark_data_smithi169_60938_object162114:head# 2024-02-12T16:18:09.241 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8cd03e7:::benchmark_data_smithi169_60938_object138612:head# 2024-02-12T16:18:09.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8cd7679:::benchmark_data_smithi169_60938_object166008:head# 2024-02-12T16:18:09.242 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8ce0fac:::benchmark_data_smithi169_60938_object143353:head# 2024-02-12T16:18:09.243 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8cff757:::benchmark_data_smithi169_60938_object154303:head# 2024-02-12T16:18:09.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8d4c305:::benchmark_data_smithi169_60938_object149569:head# 2024-02-12T16:18:09.244 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8d59450:::benchmark_data_smithi169_60938_object140733:head# 2024-02-12T16:18:09.245 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8d6ea09:::benchmark_data_smithi169_60938_object153953:head# 2024-02-12T16:18:09.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8d7d62c:::benchmark_data_smithi169_60938_object167133:head# 2024-02-12T16:18:09.246 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8ddd0a6:::benchmark_data_smithi169_60938_object158025:head# 2024-02-12T16:18:09.247 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8dec458:::benchmark_data_smithi169_60938_object165254:head# 2024-02-12T16:18:09.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8e31481:::benchmark_data_smithi169_60938_object139167:head# 2024-02-12T16:18:09.248 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8e3a58d:::benchmark_data_smithi169_60938_object156194:head# 2024-02-12T16:18:09.249 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8e5ac56:::benchmark_data_smithi169_60938_object157439:head# 2024-02-12T16:18:09.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8e6a24c:::benchmark_data_smithi169_60938_object152356:head# 2024-02-12T16:18:09.250 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8e7416d:::benchmark_data_smithi169_60938_object159485:head# 2024-02-12T16:18:09.251 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8e94658:::benchmark_data_smithi169_60938_object149091:head# 2024-02-12T16:18:09.252 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8ea14dc:::benchmark_data_smithi169_60938_object153666:head# 2024-02-12T16:18:09.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8ed0268:::benchmark_data_smithi169_60938_object161378:head# 2024-02-12T16:18:09.253 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8edbf77:::benchmark_data_smithi169_60938_object163209:head# 2024-02-12T16:18:09.254 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8ef8636:::benchmark_data_smithi169_60938_object152964:head# 2024-02-12T16:18:09.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8efbba6:::benchmark_data_smithi169_60938_object140538:head# 2024-02-12T16:18:09.255 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8efdf8e:::benchmark_data_smithi169_60938_object140795:head# 2024-02-12T16:18:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8f13cb3:::benchmark_data_smithi169_60938_object144973:head# 2024-02-12T16:18:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8f276f0:::benchmark_data_smithi169_60938_object158249:head# 2024-02-12T16:18:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8f2868e:::benchmark_data_smithi169_60938_object173201:head# 2024-02-12T16:18:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8f4dd1a:::benchmark_data_smithi169_60938_object162357:head# 2024-02-12T16:18:09.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8f80474:::benchmark_data_smithi169_60938_object149259:head# 2024-02-12T16:18:09.259 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8f8c9b5:::benchmark_data_smithi169_60938_object146672:head# 2024-02-12T16:18:09.260 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8fa5ba9:::benchmark_data_smithi169_60938_object169030:head# 2024-02-12T16:18:09.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8fb9d2f:::benchmark_data_smithi169_60938_object153003:head# 2024-02-12T16:18:09.261 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8fefbe5:::benchmark_data_smithi169_60938_object161853:head# 2024-02-12T16:18:09.262 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f8ffb017:::benchmark_data_smithi169_60938_object148578:head# 2024-02-12T16:18:09.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f900e3b7:::benchmark_data_smithi169_60938_object140468:head# 2024-02-12T16:18:09.263 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9014bef:::benchmark_data_smithi169_60938_object141738:head# 2024-02-12T16:18:09.264 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f90512d6:::benchmark_data_smithi169_60938_object147351:head# 2024-02-12T16:18:09.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9066663:::benchmark_data_smithi169_60938_object141241:head# 2024-02-12T16:18:09.265 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f90b8e13:::benchmark_data_smithi169_60938_object169687:head# 2024-02-12T16:18:09.266 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f90c4363:::benchmark_data_smithi169_60938_object154977:head# 2024-02-12T16:18:09.267 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f90e41e9:::benchmark_data_smithi169_60938_object150517:head# 2024-02-12T16:18:09.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f90f0c02:::benchmark_data_smithi169_60938_object160138:head# 2024-02-12T16:18:09.268 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f91b0f80:::benchmark_data_smithi169_60938_object144928:head# 2024-02-12T16:18:09.269 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f91c4015:::benchmark_data_smithi169_60938_object150470:head# 2024-02-12T16:18:09.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f91d8e39:::benchmark_data_smithi169_60938_object158422:head# 2024-02-12T16:18:09.270 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9239b8d:::benchmark_data_smithi169_60938_object165543:head# 2024-02-12T16:18:09.271 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9245cbf:::benchmark_data_smithi169_60938_object164712:head# 2024-02-12T16:18:09.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f924f9b7:::benchmark_data_smithi169_60938_object154064:head# 2024-02-12T16:18:09.272 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f925759f:::benchmark_data_smithi169_60938_object170590:head# 2024-02-12T16:18:09.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:09.273 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f925febb:::benchmark_data_smithi169_60938_object157181:head# 2024-02-12T16:18:09.274 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f92614b9:::benchmark_data_smithi169_60938_object170718:head# 2024-02-12T16:18:09.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f92a40b1:::benchmark_data_smithi169_60938_object160711:head# 2024-02-12T16:18:09.277 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f92d504e:::benchmark_data_smithi169_60938_object172539:head# 2024-02-12T16:18:09.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f92dd80b:::benchmark_data_smithi169_60938_object145860:head# 2024-02-12T16:18:09.278 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f92e4b00:::benchmark_data_smithi169_60938_object139638:head# 2024-02-12T16:18:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9301934:::benchmark_data_smithi169_60938_object141945:head# 2024-02-12T16:18:09.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9303f0a:::benchmark_data_smithi169_60938_object153908:head# 2024-02-12T16:18:09.280 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f931a089:::benchmark_data_smithi169_60938_object146308:head# 2024-02-12T16:18:09.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f93cc3f0:::benchmark_data_smithi169_60938_object165501:head# 2024-02-12T16:18:09.281 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f93df261:::benchmark_data_smithi169_60938_object140588:head# 2024-02-12T16:18:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f93f9182:::benchmark_data_smithi169_60938_object139183:head# 2024-02-12T16:18:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9408bfe:::benchmark_data_smithi169_60938_object164639:head# 2024-02-12T16:18:09.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f944a043:::benchmark_data_smithi169_60938_object162001:head# 2024-02-12T16:18:09.284 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9456525:::benchmark_data_smithi169_60938_object145461:head# 2024-02-12T16:18:09.285 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f948cfaf:::benchmark_data_smithi169_60938_object163340:head# 2024-02-12T16:18:09.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f948db39:::benchmark_data_smithi169_60938_object142828:head# 2024-02-12T16:18:09.286 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f949f758:::benchmark_data_smithi169_60938_object160329:head# 2024-02-12T16:18:09.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f94a10f2:::benchmark_data_smithi169_60938_object141629:head# 2024-02-12T16:18:09.287 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f94a3d25:::benchmark_data_smithi169_60938_object167754:head# 2024-02-12T16:18:09.288 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f94c7524:::benchmark_data_smithi169_60938_object151970:head# 2024-02-12T16:18:09.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f94cd6c5:::benchmark_data_smithi169_60938_object159715:head# 2024-02-12T16:18:09.289 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f94e9dbc:::benchmark_data_smithi169_60938_object153227:head# 2024-02-12T16:18:09.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9576aff:::benchmark_data_smithi169_60938_object143572:head# 2024-02-12T16:18:09.290 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f95a392c:::benchmark_data_smithi169_60938_object163314:head# 2024-02-12T16:18:09.291 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f95c2959:::benchmark_data_smithi169_60938_object151583:head# 2024-02-12T16:18:09.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f95fa43d:::benchmark_data_smithi169_60938_object145794:head# 2024-02-12T16:18:09.292 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9639753:::benchmark_data_smithi169_60938_object142945:head# 2024-02-12T16:18:09.293 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f963df3e:::benchmark_data_smithi169_60938_object143856:head# 2024-02-12T16:18:09.294 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f964580b:::benchmark_data_smithi169_60938_object143687:head# 2024-02-12T16:18:09.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9651b30:::benchmark_data_smithi169_60938_object167140:head# 2024-02-12T16:18:09.295 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f965d9e9:::benchmark_data_smithi169_60938_object143435:head# 2024-02-12T16:18:09.296 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9664bc4:::benchmark_data_smithi169_60938_object138938:head# 2024-02-12T16:18:09.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f96b3f25:::benchmark_data_smithi169_60938_object156189:head# 2024-02-12T16:18:09.297 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f96cf022:::benchmark_data_smithi169_60938_object143482:head# 2024-02-12T16:18:09.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f972fffd:::benchmark_data_smithi169_60938_object149185:head# 2024-02-12T16:18:09.298 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f975a755:::benchmark_data_smithi169_60938_object151915:head# 2024-02-12T16:18:09.299 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97acf2a:::benchmark_data_smithi169_60938_object154758:head# 2024-02-12T16:18:09.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97bcabf:::benchmark_data_smithi169_60938_object166940:head# 2024-02-12T16:18:09.300 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97bcda5:::benchmark_data_smithi169_60938_object148118:head# 2024-02-12T16:18:09.301 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97c60f4:::benchmark_data_smithi169_60938_object141370:head# 2024-02-12T16:18:09.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97c6404:::benchmark_data_smithi169_60938_object156005:head# 2024-02-12T16:18:09.302 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97d9ece:::benchmark_data_smithi169_60938_object142788:head# 2024-02-12T16:18:09.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97d9ff4:::benchmark_data_smithi169_60938_object151888:head# 2024-02-12T16:18:09.303 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97f9945:::benchmark_data_smithi169_60938_object150174:head# 2024-02-12T16:18:09.304 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f97fd27d:::benchmark_data_smithi169_60938_object147237:head# 2024-02-12T16:18:09.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f983ecf7:::benchmark_data_smithi169_60938_object143330:head# 2024-02-12T16:18:09.305 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f983fde8:::benchmark_data_smithi169_60938_object154465:head# 2024-02-12T16:18:09.306 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f984b692:::benchmark_data_smithi169_60938_object159654:head# 2024-02-12T16:18:09.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f98576c8:::benchmark_data_smithi169_60938_object153912:head# 2024-02-12T16:18:09.307 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f986e386:::benchmark_data_smithi169_60938_object164352:head# 2024-02-12T16:18:09.308 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f987e110:::benchmark_data_smithi169_60938_object172279:head# 2024-02-12T16:18:09.309 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f98a36f0:::benchmark_data_smithi169_60938_object166198:head# 2024-02-12T16:18:09.309 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:09.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f98a8bcf:::benchmark_data_smithi169_60938_object144645:head# 2024-02-12T16:18:09.310 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f98bdb84:::benchmark_data_smithi169_60938_object147871:head# 2024-02-12T16:18:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9921757:::benchmark_data_smithi169_60938_object142444:head# 2024-02-12T16:18:09.311 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f995c925:::benchmark_data_smithi169_60938_object160573:head# 2024-02-12T16:18:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:09.312 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f998b4ce:::benchmark_data_smithi169_60938_object161103:head# 2024-02-12T16:18:09.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:09.313 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f99a8757:::benchmark_data_smithi169_60938_object145034:head# 2024-02-12T16:18:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f99bb8db:::benchmark_data_smithi169_60938_object151623:head# 2024-02-12T16:18:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f99c6a8c:::benchmark_data_smithi169_60938_object144086:head# 2024-02-12T16:18:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f99cd928:::benchmark_data_smithi169_60938_object139174:head# 2024-02-12T16:18:09.315 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f99db328:::benchmark_data_smithi169_60938_object152404:head# 2024-02-12T16:18:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f99e4c93:::benchmark_data_smithi169_60938_object159232:head# 2024-02-12T16:18:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f99e9d58:::benchmark_data_smithi169_60938_object156538:head# 2024-02-12T16:18:09.317 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9a2163d:::benchmark_data_smithi169_60938_object151958:head# 2024-02-12T16:18:09.318 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9a252b8:::benchmark_data_smithi169_60938_object165832:head# 2024-02-12T16:18:09.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9a702ea:::benchmark_data_smithi169_60938_object169899:head# 2024-02-12T16:18:09.319 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9a7338a:::benchmark_data_smithi169_60938_object156465:head# 2024-02-12T16:18:09.320 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9aac5b5:::benchmark_data_smithi169_60938_object170016:head# 2024-02-12T16:18:09.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9abb8e8:::benchmark_data_smithi169_60938_object157826:head# 2024-02-12T16:18:09.321 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9aea148:::benchmark_data_smithi169_60938_object167036:head# 2024-02-12T16:18:09.322 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9afe00b:::benchmark_data_smithi169_60938_object167351:head# 2024-02-12T16:18:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9b0b28b:::benchmark_data_smithi169_60938_object166168:head# 2024-02-12T16:18:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9b25392:::benchmark_data_smithi169_60938_object145908:head# 2024-02-12T16:18:09.324 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9baae24:::benchmark_data_smithi169_60938_object170721:head# 2024-02-12T16:18:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9bc5e68:::benchmark_data_smithi169_60938_object146061:head# 2024-02-12T16:18:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9bd434d:::benchmark_data_smithi169_60938_object156772:head# 2024-02-12T16:18:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9be078d:::benchmark_data_smithi169_60938_object142194:head# 2024-02-12T16:18:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9c5933b:::benchmark_data_smithi169_60938_object145598:head# 2024-02-12T16:18:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9c59a3b:::benchmark_data_smithi169_60938_object160974:head# 2024-02-12T16:18:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9c7b40e:::benchmark_data_smithi169_60938_object149997:head# 2024-02-12T16:18:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9c92a26:::benchmark_data_smithi169_60938_object140325:head# 2024-02-12T16:18:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9cb56f5:::benchmark_data_smithi169_60938_object160188:head# 2024-02-12T16:18:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9cee56f:::benchmark_data_smithi169_60938_object138891:head# 2024-02-12T16:18:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d022bd:::benchmark_data_smithi169_60938_object160358:head# 2024-02-12T16:18:09.332 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d1bae4:::benchmark_data_smithi169_60938_object145863:head# 2024-02-12T16:18:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d242cd:::benchmark_data_smithi169_60938_object172387:head# 2024-02-12T16:18:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d299d8:::benchmark_data_smithi169_60938_object147466:head# 2024-02-12T16:18:09.334 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d3d62d:::benchmark_data_smithi169_60938_object158013:head# 2024-02-12T16:18:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d4e9db:::benchmark_data_smithi169_60938_object144190:head# 2024-02-12T16:18:09.335 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d6017d:::benchmark_data_smithi169_60938_object148591:head# 2024-02-12T16:18:09.336 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d7349e:::benchmark_data_smithi169_60938_object161660:head# 2024-02-12T16:18:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9d986fc:::benchmark_data_smithi169_60938_object142908:head# 2024-02-12T16:18:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9dbd291:::benchmark_data_smithi169_60938_object158968:head# 2024-02-12T16:18:09.338 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9dbe31a:::benchmark_data_smithi169_60938_object155692:head# 2024-02-12T16:18:09.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9dd3808:::benchmark_data_smithi169_60938_object164509:head# 2024-02-12T16:18:09.339 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9deab88:::benchmark_data_smithi169_60938_object159088:head# 2024-02-12T16:18:09.340 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9e1cc2b:::benchmark_data_smithi169_60938_object155145:head# 2024-02-12T16:18:09.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9e1f4eb:::benchmark_data_smithi169_60938_object155899:head# 2024-02-12T16:18:09.341 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9e60bd6:::benchmark_data_smithi169_60938_object170265:head# 2024-02-12T16:18:09.342 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9e69114:::benchmark_data_smithi169_60938_object151020:head# 2024-02-12T16:18:09.343 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9e902bd:::benchmark_data_smithi169_60938_object158215:head# 2024-02-12T16:18:09.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9ed51b7:::benchmark_data_smithi169_60938_object149461:head# 2024-02-12T16:18:09.344 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9eebc7f:::benchmark_data_smithi169_60938_object147485:head# 2024-02-12T16:18:09.345 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9f01a2f:::benchmark_data_smithi169_60938_object165555:head# 2024-02-12T16:18:09.346 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9f8efec:::benchmark_data_smithi169_60938_object165135:head# 2024-02-12T16:18:09.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9f9e2f0:::benchmark_data_smithi169_60938_object153845:head# 2024-02-12T16:18:09.347 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:f9fee51f:::benchmark_data_smithi169_60938_object154168:head# 2024-02-12T16:18:09.348 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa0146eb:::benchmark_data_smithi169_60938_object160506:head# 2024-02-12T16:18:09.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa04ac84:::benchmark_data_smithi169_60938_object147895:head# 2024-02-12T16:18:09.349 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa05d54d:::benchmark_data_smithi169_60938_object157516:head# 2024-02-12T16:18:09.350 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa07e77a:::benchmark_data_smithi169_60938_object163877:head# 2024-02-12T16:18:09.351 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa0adf4f:::benchmark_data_smithi169_60938_object141562:head# 2024-02-12T16:18:09.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa0d11c1:::benchmark_data_smithi169_60938_object153885:head# 2024-02-12T16:18:09.352 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa0e2426:::benchmark_data_smithi169_60938_object147235:head# 2024-02-12T16:18:09.353 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa0e3b5c:::benchmark_data_smithi169_60938_object144547:head# 2024-02-12T16:18:09.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa0fbeb3:::benchmark_data_smithi169_60938_object171439:head# 2024-02-12T16:18:09.354 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa1445f6:::benchmark_data_smithi169_60938_object171803:head# 2024-02-12T16:18:09.355 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa14cea4:::benchmark_data_smithi169_60938_object146580:head# 2024-02-12T16:18:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa1afc99:::benchmark_data_smithi169_60938_object141160:head# 2024-02-12T16:18:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa1cbd99:::benchmark_data_smithi169_60938_object168940:head# 2024-02-12T16:18:09.357 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa20f994:::benchmark_data_smithi169_60938_object152699:head# 2024-02-12T16:18:09.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa210ab4:::benchmark_data_smithi169_60938_object157281:head# 2024-02-12T16:18:09.358 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa21b276:::benchmark_data_smithi169_60938_object152474:head# 2024-02-12T16:18:09.359 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa21dff8:::benchmark_data_smithi169_60938_object149133:head# 2024-02-12T16:18:09.360 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2252c7:::benchmark_data_smithi169_60938_object163611:head# 2024-02-12T16:18:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2572fd:::benchmark_data_smithi169_60938_object160094:head# 2024-02-12T16:18:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa27341d:::benchmark_data_smithi169_60938_object157389:head# 2024-02-12T16:18:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2807fd:::benchmark_data_smithi169_60938_object169316:head# 2024-02-12T16:18:09.363 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2a1890:::benchmark_data_smithi169_60938_object168076:head# 2024-02-12T16:18:09.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2a7cbb:::benchmark_data_smithi169_60938_object152677:head# 2024-02-12T16:18:09.364 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2b0159:::benchmark_data_smithi169_60938_object145321:head# 2024-02-12T16:18:09.365 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2b7d48:::benchmark_data_smithi169_60938_object166588:head# 2024-02-12T16:18:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2b9201:::benchmark_data_smithi169_60938_object160947:head# 2024-02-12T16:18:09.366 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2c0794:::benchmark_data_smithi169_60938_object167654:head# 2024-02-12T16:18:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2cd662:::benchmark_data_smithi169_60938_object151438:head# 2024-02-12T16:18:09.368 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa2fa11d:::benchmark_data_smithi169_60938_object161157:head# 2024-02-12T16:18:09.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa3212b5:::benchmark_data_smithi169_60938_object146196:head# 2024-02-12T16:18:09.369 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa348da4:::benchmark_data_smithi169_60938_object145614:head# 2024-02-12T16:18:09.370 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa3ca4fe:::benchmark_data_smithi169_60938_object165475:head# 2024-02-12T16:18:09.371 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa3cc27b:::benchmark_data_smithi169_60938_object140531:head# 2024-02-12T16:18:09.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa3dab40:::benchmark_data_smithi169_60938_object140520:head# 2024-02-12T16:18:09.372 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa40519f:::benchmark_data_smithi169_60938_object166838:head# 2024-02-12T16:18:09.373 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa414988:::benchmark_data_smithi169_60938_object169506:head# 2024-02-12T16:18:09.374 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa424616:::benchmark_data_smithi169_60938_object150691:head# 2024-02-12T16:18:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa4783ef:::benchmark_data_smithi169_60938_object138369:head# 2024-02-12T16:18:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa4dbca5:::benchmark_data_smithi169_60938_object145408:head# 2024-02-12T16:18:09.376 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa4ff3ee:::benchmark_data_smithi169_60938_object152464:head# 2024-02-12T16:18:09.377 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa501a63:::benchmark_data_smithi169_60938_object173235:head# 2024-02-12T16:18:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5092c9:::benchmark_data_smithi169_60938_object143597:head# 2024-02-12T16:18:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa50a6f7:::benchmark_data_smithi169_60938_object167480:head# 2024-02-12T16:18:09.379 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa51c366:::benchmark_data_smithi169_60938_object161478:head# 2024-02-12T16:18:09.380 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5308d3:::benchmark_data_smithi169_60938_object156297:head# 2024-02-12T16:18:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa53f089:::benchmark_data_smithi169_60938_object163670:head# 2024-02-12T16:18:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa54bfba:::benchmark_data_smithi169_60938_object166761:head# 2024-02-12T16:18:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa569522:::benchmark_data_smithi169_60938_object145533:head# 2024-02-12T16:18:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5699e0:::benchmark_data_smithi169_60938_object149836:head# 2024-02-12T16:18:09.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5ce45b:::benchmark_data_smithi169_60938_object142481:head# 2024-02-12T16:18:09.384 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5d4565:::benchmark_data_smithi169_60938_object156754:head# 2024-02-12T16:18:09.385 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5d47d2:::benchmark_data_smithi169_60938_object147577:head# 2024-02-12T16:18:09.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5d7e52:::benchmark_data_smithi169_60938_object148103:head# 2024-02-12T16:18:09.386 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa5db063:::benchmark_data_smithi169_60938_object159274:head# 2024-02-12T16:18:09.387 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa6490e7:::benchmark_data_smithi169_60938_object155337:head# 2024-02-12T16:18:09.388 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa661b50:::benchmark_data_smithi169_60938_object161270:head# 2024-02-12T16:18:09.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa6676df:::benchmark_data_smithi169_60938_object139651:head# 2024-02-12T16:18:09.389 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa673de6:::benchmark_data_smithi169_60938_object146555:head# 2024-02-12T16:18:09.390 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa682511:::benchmark_data_smithi169_60938_object144880:head# 2024-02-12T16:18:09.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa690acf:::benchmark_data_smithi169_60938_object156421:head# 2024-02-12T16:18:09.391 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa6aa164:::benchmark_data_smithi169_60938_object163908:head# 2024-02-12T16:18:09.392 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa6be2d9:::benchmark_data_smithi169_60938_object157188:head# 2024-02-12T16:18:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa71f2f2:::benchmark_data_smithi169_60938_object155254:head# 2024-02-12T16:18:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa74feea:::benchmark_data_smithi169_60938_object151291:head# 2024-02-12T16:18:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa758b84:::benchmark_data_smithi169_60938_object162574:head# 2024-02-12T16:18:09.395 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa75b8a0:::benchmark_data_smithi169_60938_object152065:head# 2024-02-12T16:18:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa7893f5:::benchmark_data_smithi169_60938_object163319:head# 2024-02-12T16:18:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa793ef0:::benchmark_data_smithi169_60938_object161034:head# 2024-02-12T16:18:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa796f81:::benchmark_data_smithi169_60938_object166897:head# 2024-02-12T16:18:09.398 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa7afe0b:::benchmark_data_smithi169_60938_object142321:head# 2024-02-12T16:18:09.399 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa7b07b5:::benchmark_data_smithi169_60938_object140076:head# 2024-02-12T16:18:09.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa7cb8b7:::benchmark_data_smithi169_60938_object139534:head# 2024-02-12T16:18:09.400 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa7ddc31:::benchmark_data_smithi169_60938_object153693:head# 2024-02-12T16:18:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa7ec0b5:::benchmark_data_smithi169_60938_object163208:head# 2024-02-12T16:18:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa7edf45:::benchmark_data_smithi169_60938_object162542:head# 2024-02-12T16:18:09.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa815051:::benchmark_data_smithi169_60938_object149514:head# 2024-02-12T16:18:09.403 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa8279df:::benchmark_data_smithi169_60938_object139639:head# 2024-02-12T16:18:09.404 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa87bc6f:::benchmark_data_smithi169_60938_object154136:head# 2024-02-12T16:18:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa898ed6:::benchmark_data_smithi169_60938_object143191:head# 2024-02-12T16:18:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa8a2e39:::benchmark_data_smithi169_60938_object163914:head# 2024-02-12T16:18:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa8f9c53:::benchmark_data_smithi169_60938_object144582:head# 2024-02-12T16:18:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa956c2e:::benchmark_data_smithi169_60938_object162464:head# 2024-02-12T16:18:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa96cbdf:::benchmark_data_smithi169_60938_object166746:head# 2024-02-12T16:18:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa9a021f:::benchmark_data_smithi169_60938_object158520:head# 2024-02-12T16:18:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fa9a7d08:::benchmark_data_smithi169_60938_object155507:head# 2024-02-12T16:18:09.410 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faa136a9:::benchmark_data_smithi169_60938_object160882:head# 2024-02-12T16:18:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faa6f5fc:::benchmark_data_smithi169_60938_object147918:head# 2024-02-12T16:18:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faa72e9a:::benchmark_data_smithi169_60938_object147697:head# 2024-02-12T16:18:09.412 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faa7b429:::benchmark_data_smithi169_60938_object152801:head# 2024-02-12T16:18:09.413 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faaab9eb:::benchmark_data_smithi169_60938_object152880:head# 2024-02-12T16:18:09.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:09.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faac2e91:::benchmark_data_smithi169_60938_object161642:head# 2024-02-12T16:18:09.414 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faad4128:::benchmark_data_smithi169_60938_object163274:head# 2024-02-12T16:18:09.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fab0dbf4:::benchmark_data_smithi169_60938_object162951:head# 2024-02-12T16:18:09.415 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fab2b31f:::benchmark_data_smithi169_60938_object163765:head# 2024-02-12T16:18:09.416 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fab389d8:::benchmark_data_smithi169_60938_object170342:head# 2024-02-12T16:18:09.417 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fab3ae54:::benchmark_data_smithi169_60938_object165895:head# 2024-02-12T16:18:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fabc4c29:::benchmark_data_smithi169_60938_object161786:head# 2024-02-12T16:18:09.418 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fabd543a:::benchmark_data_smithi169_60938_object163410:head# 2024-02-12T16:18:09.419 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac17480:::benchmark_data_smithi169_60938_object160867:head# 2024-02-12T16:18:09.420 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac28432:::benchmark_data_smithi169_60938_object171377:head# 2024-02-12T16:18:09.421 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac2a447:::benchmark_data_smithi169_60938_object157925:head# 2024-02-12T16:18:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac44938:::benchmark_data_smithi169_60938_object169504:head# 2024-02-12T16:18:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac6d020:::benchmark_data_smithi169_60938_object158492:head# 2024-02-12T16:18:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac70fec:::benchmark_data_smithi169_60938_object160751:head# 2024-02-12T16:18:09.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac922f3:::benchmark_data_smithi169_60938_object160070:head# 2024-02-12T16:18:09.424 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fac9aded:::benchmark_data_smithi169_60938_object164062:head# 2024-02-12T16:18:09.425 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:facb60a9:::benchmark_data_smithi169_60938_object155367:head# 2024-02-12T16:18:09.426 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:facbc4cb:::benchmark_data_smithi169_60938_object139432:head# 2024-02-12T16:18:09.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faccf70b:::benchmark_data_smithi169_60938_object145437:head# 2024-02-12T16:18:09.427 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fad30437:::benchmark_data_smithi169_60938_object143517:head# 2024-02-12T16:18:09.428 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fad3ae21:::benchmark_data_smithi169_60938_object155594:head# 2024-02-12T16:18:09.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fad4444c:::benchmark_data_smithi169_60938_object150878:head# 2024-02-12T16:18:09.429 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fad5d228:::benchmark_data_smithi169_60938_object146686:head# 2024-02-12T16:18:09.430 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fad7ae4c:::benchmark_data_smithi169_60938_object141012:head# 2024-02-12T16:18:09.431 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fad82dc3:::benchmark_data_smithi169_60938_object141190:head# 2024-02-12T16:18:09.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fad9fe3e:::benchmark_data_smithi169_60938_object156460:head# 2024-02-12T16:18:09.432 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fada41bf:::benchmark_data_smithi169_60938_object142751:head# 2024-02-12T16:18:09.433 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fade200f:::benchmark_data_smithi169_60938_object168492:head# 2024-02-12T16:18:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fadee1ed:::benchmark_data_smithi169_60938_object160798:head# 2024-02-12T16:18:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fae18eb3:::benchmark_data_smithi169_60938_object156543:head# 2024-02-12T16:18:09.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fae7230f:::benchmark_data_smithi169_60938_object144856:head# 2024-02-12T16:18:09.436 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fae74df6:::benchmark_data_smithi169_60938_object160706:head# 2024-02-12T16:18:09.437 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faea9937:::benchmark_data_smithi169_60938_object170660:head# 2024-02-12T16:18:09.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faecdf65:::benchmark_data_smithi169_60938_object165629:head# 2024-02-12T16:18:09.438 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faf026e5:::benchmark_data_smithi169_60938_object154733:head# 2024-02-12T16:18:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faf1d4d3:::benchmark_data_smithi169_60938_object149570:head# 2024-02-12T16:18:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faf34e4f:::benchmark_data_smithi169_60938_object167124:head# 2024-02-12T16:18:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faf3cdb6:::benchmark_data_smithi169_60938_object167779:head# 2024-02-12T16:18:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faf45515:::benchmark_data_smithi169_60938_object145571:head# 2024-02-12T16:18:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faf5041d:::benchmark_data_smithi169_60938_object148593:head# 2024-02-12T16:18:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:faf6a386:::benchmark_data_smithi169_60938_object167125:head# 2024-02-12T16:18:09.443 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fafceab7:::benchmark_data_smithi169_60938_object170171:head# 2024-02-12T16:18:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb0397e5:::benchmark_data_smithi169_60938_object158016:head# 2024-02-12T16:18:09.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb06903c:::benchmark_data_smithi169_60938_object153218:head# 2024-02-12T16:18:09.445 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb08c652:::benchmark_data_smithi169_60938_object138608:head# 2024-02-12T16:18:09.446 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb090aef:::benchmark_data_smithi169_60938_object161276:head# 2024-02-12T16:18:09.447 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb0e6360:::benchmark_data_smithi169_60938_object154852:head# 2024-02-12T16:18:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb0f7eb9:::benchmark_data_smithi169_60938_object162544:head# 2024-02-12T16:18:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb1030a9:::benchmark_data_smithi169_60938_object145688:head# 2024-02-12T16:18:09.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb112154:::benchmark_data_smithi169_60938_object169323:head# 2024-02-12T16:18:09.449 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb1a065d:::benchmark_data_smithi169_60938_object151366:head# 2024-02-12T16:18:09.450 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb1cc389:::benchmark_data_smithi169_60938_object151909:head# 2024-02-12T16:18:09.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb219f76:::benchmark_data_smithi169_60938_object141839:head# 2024-02-12T16:18:09.451 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb22b6b0:::benchmark_data_smithi169_60938_object160661:head# 2024-02-12T16:18:09.452 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb244a6f:::benchmark_data_smithi169_60938_object169164:head# 2024-02-12T16:18:09.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:09.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb2939aa:::benchmark_data_smithi169_60938_object140193:head# 2024-02-12T16:18:09.453 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb2bcf9a:::benchmark_data_smithi169_60938_object156474:head# 2024-02-12T16:18:09.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb2ed3a7:::benchmark_data_smithi169_60938_object148929:head# 2024-02-12T16:18:09.454 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb30df7a:::benchmark_data_smithi169_60938_object166380:head# 2024-02-12T16:18:09.455 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb31e4aa:::benchmark_data_smithi169_60938_object162854:head# 2024-02-12T16:18:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:09.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:09.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb326883:::benchmark_data_smithi169_60938_object158701:head# 2024-02-12T16:18:09.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb32748e:::benchmark_data_smithi169_60938_object158921:head# 2024-02-12T16:18:09.457 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb373c88:::benchmark_data_smithi169_60938_object141883:head# 2024-02-12T16:18:09.458 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb380831:::benchmark_data_smithi169_60938_object141460:head# 2024-02-12T16:18:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb38d750:::benchmark_data_smithi169_60938_object150113:head# 2024-02-12T16:18:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb3990be:::benchmark_data_smithi169_60938_object160260:head# 2024-02-12T16:18:09.460 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb428a13:::benchmark_data_smithi169_60938_object162328:head# 2024-02-12T16:18:09.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb491dfa:::benchmark_data_smithi169_60938_object138615:head# 2024-02-12T16:18:09.461 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb499fb1:::benchmark_data_smithi169_60938_object149417:head# 2024-02-12T16:18:09.462 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb4d1374:::benchmark_data_smithi169_60938_object139544:head# 2024-02-12T16:18:09.463 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb4fbd33:::benchmark_data_smithi169_60938_object138985:head# 2024-02-12T16:18:09.464 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb51a3d3:::benchmark_data_smithi169_60938_object143959:head# 2024-02-12T16:18:09.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb524d8d:::benchmark_data_smithi169_60938_object140730:head# 2024-02-12T16:18:09.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb530ec7:::benchmark_data_smithi169_60938_object158739:head# 2024-02-12T16:18:09.465 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb536a4b:::benchmark_data_smithi169_60938_object166003:head# 2024-02-12T16:18:09.466 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb54150d:::benchmark_data_smithi169_60938_object172314:head# 2024-02-12T16:18:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb54f3a2:::benchmark_data_smithi169_60938_object157938:head# 2024-02-12T16:18:09.468 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb550832:::benchmark_data_smithi169_60938_object168723:head# 2024-02-12T16:18:09.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb5544cd:::benchmark_data_smithi169_60938_object156808:head# 2024-02-12T16:18:09.469 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb55ba92:::benchmark_data_smithi169_60938_object165056:head# 2024-02-12T16:18:09.470 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb56761c:::benchmark_data_smithi169_60938_object148031:head# 2024-02-12T16:18:09.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb57aaed:::benchmark_data_smithi169_60938_object159706:head# 2024-02-12T16:18:09.471 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb585a91:::benchmark_data_smithi169_60938_object170266:head# 2024-02-12T16:18:09.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb59c9c5:::benchmark_data_smithi169_60938_object145690:head# 2024-02-12T16:18:09.472 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb5cd2da:::benchmark_data_smithi169_60938_object165233:head# 2024-02-12T16:18:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb5dfe9f:::benchmark_data_smithi169_60938_object166215:head# 2024-02-12T16:18:09.474 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb5e19ef:::benchmark_data_smithi169_60938_object143595:head# 2024-02-12T16:18:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb5e2ddb:::benchmark_data_smithi169_60938_object154362:head# 2024-02-12T16:18:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb5f5363:::benchmark_data_smithi169_60938_object154593:head# 2024-02-12T16:18:09.476 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb603eec:::benchmark_data_smithi169_60938_object172340:head# 2024-02-12T16:18:09.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb610285:::benchmark_data_smithi169_60938_object140983:head# 2024-02-12T16:18:09.477 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb61f8a9:::benchmark_data_smithi169_60938_object150617:head# 2024-02-12T16:18:09.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb651930:::benchmark_data_smithi169_60938_object169851:head# 2024-02-12T16:18:09.478 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb65ff55:::benchmark_data_smithi169_60938_object165411:head# 2024-02-12T16:18:09.479 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb66ce57:::benchmark_data_smithi169_60938_object149767:head# 2024-02-12T16:18:09.480 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb6f1358:::benchmark_data_smithi169_60938_object144623:head# 2024-02-12T16:18:09.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb712276:::benchmark_data_smithi169_60938_object168962:head# 2024-02-12T16:18:09.481 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb73b306:::benchmark_data_smithi169_60938_object173287:head# 2024-02-12T16:18:09.482 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb74029a:::benchmark_data_smithi169_60938_object156572:head# 2024-02-12T16:18:09.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb7517f0:::benchmark_data_smithi169_60938_object160061:head# 2024-02-12T16:18:09.483 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb753c1d:::benchmark_data_smithi169_60938_object148946:head# 2024-02-12T16:18:09.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb79d706:::benchmark_data_smithi169_60938_object161107:head# 2024-02-12T16:18:09.484 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb7c7cac:::benchmark_data_smithi169_60938_object150511:head# 2024-02-12T16:18:09.487 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb7d5a96:::benchmark_data_smithi169_60938_object170651:head# 2024-02-12T16:18:09.488 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb7d7017:::benchmark_data_smithi169_60938_object170207:head# 2024-02-12T16:18:09.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb834f26:::benchmark_data_smithi169_60938_object157969:head# 2024-02-12T16:18:09.489 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb843298:::benchmark_data_smithi169_60938_object142513:head# 2024-02-12T16:18:09.490 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb85dd84:::benchmark_data_smithi169_60938_object159567:head# 2024-02-12T16:18:09.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb862128:::benchmark_data_smithi169_60938_object167498:head# 2024-02-12T16:18:09.491 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb86f95a:::benchmark_data_smithi169_60938_object161403:head# 2024-02-12T16:18:09.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb89d6e5:::benchmark_data_smithi169_60938_object160111:head# 2024-02-12T16:18:09.492 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb8b66ee:::benchmark_data_smithi169_60938_object152276:head# 2024-02-12T16:18:09.493 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb8e84ca:::benchmark_data_smithi169_60938_object169656:head# 2024-02-12T16:18:09.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb908373:::benchmark_data_smithi169_60938_object158847:head# 2024-02-12T16:18:09.494 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb91b131:::benchmark_data_smithi169_60938_object150206:head# 2024-02-12T16:18:09.495 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb94e013:::benchmark_data_smithi169_60938_object146410:head# 2024-02-12T16:18:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb95ab05:::benchmark_data_smithi169_60938_object157746:head# 2024-02-12T16:18:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb95dea3:::benchmark_data_smithi169_60938_object157024:head# 2024-02-12T16:18:09.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb9c1dc0:::benchmark_data_smithi169_60938_object149136:head# 2024-02-12T16:18:09.497 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb9cc303:::benchmark_data_smithi169_60938_object168699:head# 2024-02-12T16:18:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fb9ccbef:::benchmark_data_smithi169_60938_object164704:head# 2024-02-12T16:18:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fba28dc5:::benchmark_data_smithi169_60938_object171833:head# 2024-02-12T16:18:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fba2fa71:::benchmark_data_smithi169_60938_object146624:head# 2024-02-12T16:18:09.500 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fba7f1e0:::benchmark_data_smithi169_60938_object144367:head# 2024-02-12T16:18:09.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fba8d522:::benchmark_data_smithi169_60938_object151439:head# 2024-02-12T16:18:09.501 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbaa6e25:::benchmark_data_smithi169_60938_object147585:head# 2024-02-12T16:18:09.502 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbab9a2c:::benchmark_data_smithi169_60938_object152682:head# 2024-02-12T16:18:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbabdc59:::benchmark_data_smithi169_60938_object161325:head# 2024-02-12T16:18:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbad8f6b:::benchmark_data_smithi169_60938_object161567:head# 2024-02-12T16:18:09.504 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbb245c0:::benchmark_data_smithi169_60938_object148226:head# 2024-02-12T16:18:09.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbb84a12:::benchmark_data_smithi169_60938_object161648:head# 2024-02-12T16:18:09.505 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbb918bf:::benchmark_data_smithi169_60938_object143132:head# 2024-02-12T16:18:09.506 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbb92b51:::benchmark_data_smithi169_60938_object168199:head# 2024-02-12T16:18:09.507 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbbae976:::benchmark_data_smithi169_60938_object148143:head# 2024-02-12T16:18:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbbbc134:::benchmark_data_smithi169_60938_object138913:head# 2024-02-12T16:18:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbbbf7fa:::benchmark_data_smithi169_60938_object155750:head# 2024-02-12T16:18:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbbd466e:::benchmark_data_smithi169_60938_object142842:head# 2024-02-12T16:18:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbbe407c:::benchmark_data_smithi169_60938_object147287:head# 2024-02-12T16:18:09.510 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbbed521:::benchmark_data_smithi169_60938_object151813:head# 2024-02-12T16:18:09.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbc0a5cf:::benchmark_data_smithi169_60938_object156262:head# 2024-02-12T16:18:09.511 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbc2953c:::benchmark_data_smithi169_60938_object150373:head# 2024-02-12T16:18:09.512 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbc564ab:::benchmark_data_smithi169_60938_object157665:head# 2024-02-12T16:18:09.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbc5ae6b:::benchmark_data_smithi169_60938_object168254:head# 2024-02-12T16:18:09.513 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbc677d0:::benchmark_data_smithi169_60938_object160935:head# 2024-02-12T16:18:09.514 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbd0d135:::benchmark_data_smithi169_60938_object154887:head# 2024-02-12T16:18:09.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbd4dd50:::benchmark_data_smithi169_60938_object158656:head# 2024-02-12T16:18:09.515 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbd59772:::benchmark_data_smithi169_60938_object171312:head# 2024-02-12T16:18:09.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbd691a0:::benchmark_data_smithi169_60938_object163446:head# 2024-02-12T16:18:09.516 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbd6a56f:::benchmark_data_smithi169_60938_object167602:head# 2024-02-12T16:18:09.517 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbd7d522:::benchmark_data_smithi169_60938_object145833:head# 2024-02-12T16:18:09.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbd93872:::benchmark_data_smithi169_60938_object165498:head# 2024-02-12T16:18:09.518 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbdc6ac2:::benchmark_data_smithi169_60938_object138496:head# 2024-02-12T16:18:09.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbe0f5f5:::benchmark_data_smithi169_60938_object157860:head# 2024-02-12T16:18:09.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbe59b0f:::benchmark_data_smithi169_60938_object151079:head# 2024-02-12T16:18:09.519 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbe727c0:::benchmark_data_smithi169_60938_object160533:head# 2024-02-12T16:18:09.520 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbe925a9:::benchmark_data_smithi169_60938_object165156:head# 2024-02-12T16:18:09.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbe9b191:::benchmark_data_smithi169_60938_object146168:head# 2024-02-12T16:18:09.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbec62f1:::benchmark_data_smithi169_60938_object140681:head# 2024-02-12T16:18:09.521 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbf047ce:::benchmark_data_smithi169_60938_object148129:head# 2024-02-12T16:18:09.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbf200a6:::benchmark_data_smithi169_60938_object164500:head# 2024-02-12T16:18:09.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbf2e15f:::benchmark_data_smithi169_60938_object169873:head# 2024-02-12T16:18:09.523 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbfa903d:::benchmark_data_smithi169_60938_object157789:head# 2024-02-12T16:18:09.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbfb653d:::benchmark_data_smithi169_60938_object158741:head# 2024-02-12T16:18:09.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbfbda38:::benchmark_data_smithi169_60938_object163160:head# 2024-02-12T16:18:09.524 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbfdcd95:::benchmark_data_smithi169_60938_object156305:head# 2024-02-12T16:18:09.525 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbfe6333:::benchmark_data_smithi169_60938_object172632:head# 2024-02-12T16:18:09.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fbffb42e:::benchmark_data_smithi169_60938_object147994:head# 2024-02-12T16:18:09.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc00f223:::benchmark_data_smithi169_60938_object155787:head# 2024-02-12T16:18:09.526 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc020934:::benchmark_data_smithi169_60938_object159785:head# 2024-02-12T16:18:09.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc0258e0:::benchmark_data_smithi169_60938_object164516:head# 2024-02-12T16:18:09.527 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc035753:::benchmark_data_smithi169_60938_object164103:head# 2024-02-12T16:18:09.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc05e220:::benchmark_data_smithi169_60938_object167915:head# 2024-02-12T16:18:09.528 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc06ab93:::benchmark_data_smithi169_60938_object140800:head# 2024-02-12T16:18:09.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc0735b5:::benchmark_data_smithi169_60938_object171105:head# 2024-02-12T16:18:09.529 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc07b6fd:::benchmark_data_smithi169_60938_object156890:head# 2024-02-12T16:18:09.530 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc08d6c4:::benchmark_data_smithi169_60938_object156089:head# 2024-02-12T16:18:09.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc08dee2:::benchmark_data_smithi169_60938_object150727:head# 2024-02-12T16:18:09.531 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc095eb8:::benchmark_data_smithi169_60938_object170194:head# 2024-02-12T16:18:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc0a1f70:::benchmark_data_smithi169_60938_object170479:head# 2024-02-12T16:18:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc0bb874:::benchmark_data_smithi169_60938_object149151:head# 2024-02-12T16:18:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc1064df:::benchmark_data_smithi169_60938_object154322:head# 2024-02-12T16:18:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc10f7d4:::benchmark_data_smithi169_60938_object144939:head# 2024-02-12T16:18:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc128907:::benchmark_data_smithi169_60938_object173295:head# 2024-02-12T16:18:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc12b4da:::benchmark_data_smithi169_60938_object146192:head# 2024-02-12T16:18:09.536 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc12bee6:::benchmark_data_smithi169_60938_object162950:head# 2024-02-12T16:18:09.537 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc15695e:::benchmark_data_smithi169_60938_object140585:head# 2024-02-12T16:18:09.538 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc15b2b9:::benchmark_data_smithi169_60938_object166632:head# 2024-02-12T16:18:09.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc1693a1:::benchmark_data_smithi169_60938_object156315:head# 2024-02-12T16:18:09.539 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc17b7ae:::benchmark_data_smithi169_60938_object155155:head# 2024-02-12T16:18:09.540 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc18482b:::benchmark_data_smithi169_60938_object141323:head# 2024-02-12T16:18:09.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc194f8a:::benchmark_data_smithi169_60938_object166845:head# 2024-02-12T16:18:09.541 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc1b7a88:::benchmark_data_smithi169_60938_object168426:head# 2024-02-12T16:18:09.542 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc1e3034:::benchmark_data_smithi169_60938_object166050:head# 2024-02-12T16:18:09.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc1e6545:::benchmark_data_smithi169_60938_object138497:head# 2024-02-12T16:18:09.543 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc1e80b7:::benchmark_data_smithi169_60938_object152887:head# 2024-02-12T16:18:09.544 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc217403:::benchmark_data_smithi169_60938_object153640:head# 2024-02-12T16:18:09.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc28379c:::benchmark_data_smithi169_60938_object169818:head# 2024-02-12T16:18:09.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc2a3cc3:::benchmark_data_smithi169_60938_object157932:head# 2024-02-12T16:18:09.545 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc2c2ccf:::benchmark_data_smithi169_60938_object159885:head# 2024-02-12T16:18:09.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc2dd090:::benchmark_data_smithi169_60938_object167765:head# 2024-02-12T16:18:09.546 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc30ce26:::benchmark_data_smithi169_60938_object151804:head# 2024-02-12T16:18:09.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc320e66:::benchmark_data_smithi169_60938_object152822:head# 2024-02-12T16:18:09.547 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc329f5a:::benchmark_data_smithi169_60938_object167712:head# 2024-02-12T16:18:09.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc337af7:::benchmark_data_smithi169_60938_object147371:head# 2024-02-12T16:18:09.548 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc35854e:::benchmark_data_smithi169_60938_object167081:head# 2024-02-12T16:18:09.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc363888:::benchmark_data_smithi169_60938_object143907:head# 2024-02-12T16:18:09.549 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc36ddf7:::benchmark_data_smithi169_60938_object143225:head# 2024-02-12T16:18:09.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc37eed4:::benchmark_data_smithi169_60938_object144159:head# 2024-02-12T16:18:09.550 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc3ac158:::benchmark_data_smithi169_60938_object156987:head# 2024-02-12T16:18:09.551 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc3ac6ff:::benchmark_data_smithi169_60938_object149292:head# 2024-02-12T16:18:09.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc3c4c38:::benchmark_data_smithi169_60938_object146354:head# 2024-02-12T16:18:09.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc412acd:::benchmark_data_smithi169_60938_object151018:head# 2024-02-12T16:18:09.552 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc422dd1:::benchmark_data_smithi169_60938_object163741:head# 2024-02-12T16:18:09.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc423e17:::benchmark_data_smithi169_60938_object139464:head# 2024-02-12T16:18:09.553 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc470d88:::benchmark_data_smithi169_60938_object164437:head# 2024-02-12T16:18:09.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc472f7c:::benchmark_data_smithi169_60938_object171118:head# 2024-02-12T16:18:09.554 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc4884e6:::benchmark_data_smithi169_60938_object147424:head# 2024-02-12T16:18:09.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc49798f:::benchmark_data_smithi169_60938_object168362:head# 2024-02-12T16:18:09.555 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc497ad9:::benchmark_data_smithi169_60938_object145411:head# 2024-02-12T16:18:09.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc4d0c10:::benchmark_data_smithi169_60938_object162755:head# 2024-02-12T16:18:09.556 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc4e4c04:::benchmark_data_smithi169_60938_object168859:head# 2024-02-12T16:18:09.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:09.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc4e5849:::benchmark_data_smithi169_60938_object147718:head# 2024-02-12T16:18:09.557 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc50b1d3:::benchmark_data_smithi169_60938_object165017:head# 2024-02-12T16:18:09.558 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc544a3a:::benchmark_data_smithi169_60938_object162889:head# 2024-02-12T16:18:09.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc5b839b:::benchmark_data_smithi169_60938_object142912:head# 2024-02-12T16:18:09.559 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc5e8f95:::benchmark_data_smithi169_60938_object168848:head# 2024-02-12T16:18:09.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc64f2d5:::benchmark_data_smithi169_60938_object148905:head# 2024-02-12T16:18:09.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc660fa8:::benchmark_data_smithi169_60938_object165799:head# 2024-02-12T16:18:09.560 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc6768fb:::benchmark_data_smithi169_60938_object163532:head# 2024-02-12T16:18:09.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc6874c4:::benchmark_data_smithi169_60938_object141810:head# 2024-02-12T16:18:09.561 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc687555:::benchmark_data_smithi169_60938_object162142:head# 2024-02-12T16:18:09.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc69ef0a:::benchmark_data_smithi169_60938_object165696:head# 2024-02-12T16:18:09.562 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc6b500b:::benchmark_data_smithi169_60938_object144728:head# 2024-02-12T16:18:09.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc6c1fd5:::benchmark_data_smithi169_60938_object143043:head# 2024-02-12T16:18:09.563 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc6d227e:::benchmark_data_smithi169_60938_object146303:head# 2024-02-12T16:18:09.564 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc7021bc:::benchmark_data_smithi169_60938_object163283:head# 2024-02-12T16:18:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc70613c:::benchmark_data_smithi169_60938_object153863:head# 2024-02-12T16:18:09.565 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc70a99e:::benchmark_data_smithi169_60938_object142073:head# 2024-02-12T16:18:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc70cfce:::benchmark_data_smithi169_60938_object167355:head# 2024-02-12T16:18:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc713583:::benchmark_data_smithi169_60938_object147173:head# 2024-02-12T16:18:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc72cfb2:::benchmark_data_smithi169_60938_object162054:head# 2024-02-12T16:18:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc733604:::benchmark_data_smithi169_60938_object144218:head# 2024-02-12T16:18:09.567 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc73e559:::benchmark_data_smithi169_60938_object141940:head# 2024-02-12T16:18:09.568 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc75536f:::benchmark_data_smithi169_60938_object154706:head# 2024-02-12T16:18:09.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc76666c:::benchmark_data_smithi169_60938_object161509:head# 2024-02-12T16:18:09.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc7818e6:::benchmark_data_smithi169_60938_object143074:head# 2024-02-12T16:18:09.569 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc7977c6:::benchmark_data_smithi169_60938_object139054:head# 2024-02-12T16:18:09.570 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc7a41ad:::benchmark_data_smithi169_60938_object171149:head# 2024-02-12T16:18:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc7fffd6:::benchmark_data_smithi169_60938_object145574:head# 2024-02-12T16:18:09.571 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc82f8c1:::benchmark_data_smithi169_60938_object156196:head# 2024-02-12T16:18:09.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc84b211:::benchmark_data_smithi169_60938_object172172:head# 2024-02-12T16:18:09.572 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc84b91a:::benchmark_data_smithi169_60938_object150084:head# 2024-02-12T16:18:09.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc87932f:::benchmark_data_smithi169_60938_object157876:head# 2024-02-12T16:18:09.573 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc886308:::benchmark_data_smithi169_60938_object153616:head# 2024-02-12T16:18:09.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc8894bb:::benchmark_data_smithi169_60938_object139430:head# 2024-02-12T16:18:09.574 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc8966f2:::benchmark_data_smithi169_60938_object166912:head# 2024-02-12T16:18:09.575 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc8d0a0f:::benchmark_data_smithi169_60938_object158914:head# 2024-02-12T16:18:09.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc8d190a:::benchmark_data_smithi169_60938_object168781:head# 2024-02-12T16:18:09.576 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc91d39c:::benchmark_data_smithi169_60938_object140389:head# 2024-02-12T16:18:09.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc921815:::benchmark_data_smithi169_60938_object161814:head# 2024-02-12T16:18:09.577 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc92c755:::benchmark_data_smithi169_60938_object157892:head# 2024-02-12T16:18:09.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc92d094:::benchmark_data_smithi169_60938_object144081:head# 2024-02-12T16:18:09.578 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc93acf4:::benchmark_data_smithi169_60938_object162371:head# 2024-02-12T16:18:09.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc946cc3:::benchmark_data_smithi169_60938_object163814:head# 2024-02-12T16:18:09.579 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc955e62:::benchmark_data_smithi169_60938_object153412:head# 2024-02-12T16:18:09.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc975430:::benchmark_data_smithi169_60938_object167059:head# 2024-02-12T16:18:09.580 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc985a1f:::benchmark_data_smithi169_60938_object143805:head# 2024-02-12T16:18:09.581 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc988bef:::benchmark_data_smithi169_60938_object147982:head# 2024-02-12T16:18:09.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc99b62a:::benchmark_data_smithi169_60938_object169195:head# 2024-02-12T16:18:09.582 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc9b305c:::benchmark_data_smithi169_60938_object163953:head# 2024-02-12T16:18:09.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc9b4ae4:::benchmark_data_smithi169_60938_object152081:head# 2024-02-12T16:18:09.583 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc9e5381:::benchmark_data_smithi169_60938_object162197:head# 2024-02-12T16:18:09.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc9ea6ac:::benchmark_data_smithi169_60938_object159579:head# 2024-02-12T16:18:09.584 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fc9fb1b6:::benchmark_data_smithi169_60938_object149766:head# 2024-02-12T16:18:09.585 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fca1910e:::benchmark_data_smithi169_60938_object143001:head# 2024-02-12T16:18:09.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fca3744a:::benchmark_data_smithi169_60938_object171148:head# 2024-02-12T16:18:09.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fca76ec7:::benchmark_data_smithi169_60938_object166849:head# 2024-02-12T16:18:09.587 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fca9d479:::benchmark_data_smithi169_60938_object142739:head# 2024-02-12T16:18:09.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcab132e:::benchmark_data_smithi169_60938_object150595:head# 2024-02-12T16:18:09.588 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcac6518:::benchmark_data_smithi169_60938_object150002:head# 2024-02-12T16:18:09.589 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcadcd23:::benchmark_data_smithi169_60938_object172701:head# 2024-02-12T16:18:09.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcb0fc31:::benchmark_data_smithi169_60938_object155201:head# 2024-02-12T16:18:09.590 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcb1d7f9:::benchmark_data_smithi169_60938_object160088:head# 2024-02-12T16:18:09.591 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcb234a1:::benchmark_data_smithi169_60938_object151744:head# 2024-02-12T16:18:09.592 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcb3a1b7:::benchmark_data_smithi169_60938_object150751:head# 2024-02-12T16:18:09.592 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:09.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcb79864:::benchmark_data_smithi169_60938_object167964:head# 2024-02-12T16:18:09.593 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcbe4746:::benchmark_data_smithi169_60938_object163041:head# 2024-02-12T16:18:09.594 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcbe8245:::benchmark_data_smithi169_60938_object162313:head# 2024-02-12T16:18:09.595 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc1b6e7:::benchmark_data_smithi169_60938_object147068:head# 2024-02-12T16:18:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:09.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc281b7:::benchmark_data_smithi169_60938_object158828:head# 2024-02-12T16:18:09.596 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc2b2b1:::benchmark_data_smithi169_60938_object143937:head# 2024-02-12T16:18:09.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:09.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc36645:::benchmark_data_smithi169_60938_object138944:head# 2024-02-12T16:18:09.597 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc38193:::benchmark_data_smithi169_60938_object140047:head# 2024-02-12T16:18:09.598 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc3c163:::benchmark_data_smithi169_60938_object170808:head# 2024-02-12T16:18:09.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc6c8ee:::benchmark_data_smithi169_60938_object143402:head# 2024-02-12T16:18:09.599 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc6e9e6:::benchmark_data_smithi169_60938_object150612:head# 2024-02-12T16:18:09.600 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcc94f4f:::benchmark_data_smithi169_60938_object146353:head# 2024-02-12T16:18:09.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fccac44c:::benchmark_data_smithi169_60938_object168240:head# 2024-02-12T16:18:09.601 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fccc5146:::benchmark_data_smithi169_60938_object172566:head# 2024-02-12T16:18:09.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fccd1fe2:::benchmark_data_smithi169_60938_object141671:head# 2024-02-12T16:18:09.602 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fccfed03:::benchmark_data_smithi169_60938_object172381:head# 2024-02-12T16:18:09.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcd0e623:::benchmark_data_smithi169_60938_object146498:head# 2024-02-12T16:18:09.603 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcd53a0a:::benchmark_data_smithi169_60938_object155834:head# 2024-02-12T16:18:09.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcd6d267:::benchmark_data_smithi169_60938_object155512:head# 2024-02-12T16:18:09.604 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcd98e4f:::benchmark_data_smithi169_60938_object166336:head# 2024-02-12T16:18:09.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcd9c3d4:::benchmark_data_smithi169_60938_object151080:head# 2024-02-12T16:18:09.605 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcda39db:::benchmark_data_smithi169_60938_object144391:head# 2024-02-12T16:18:09.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcda4529:::benchmark_data_smithi169_60938_object153000:head# 2024-02-12T16:18:09.606 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcdaac06:::benchmark_data_smithi169_60938_object167268:head# 2024-02-12T16:18:09.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcdadf3a:::benchmark_data_smithi169_60938_object169315:head# 2024-02-12T16:18:09.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcdbac86:::benchmark_data_smithi169_60938_object146548:head# 2024-02-12T16:18:09.607 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcdc6f66:::benchmark_data_smithi169_60938_object167446:head# 2024-02-12T16:18:09.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcdd5365:::benchmark_data_smithi169_60938_object138462:head# 2024-02-12T16:18:09.608 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce0263d:::benchmark_data_smithi169_60938_object152423:head# 2024-02-12T16:18:09.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce0a8b2:::benchmark_data_smithi169_60938_object170332:head# 2024-02-12T16:18:09.609 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce19471:::benchmark_data_smithi169_60938_object151835:head# 2024-02-12T16:18:09.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce24ed7:::benchmark_data_smithi169_60938_object145253:head# 2024-02-12T16:18:09.610 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce2db9a:::benchmark_data_smithi169_60938_object140877:head# 2024-02-12T16:18:09.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce43118:::benchmark_data_smithi169_60938_object153456:head# 2024-02-12T16:18:09.611 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce439ea:::benchmark_data_smithi169_60938_object148930:head# 2024-02-12T16:18:09.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce48217:::benchmark_data_smithi169_60938_object169823:head# 2024-02-12T16:18:09.612 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce48cc1:::benchmark_data_smithi169_60938_object140861:head# 2024-02-12T16:18:09.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce4a535:::benchmark_data_smithi169_60938_object148192:head# 2024-02-12T16:18:09.613 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce776c8:::benchmark_data_smithi169_60938_object156852:head# 2024-02-12T16:18:09.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fce7a3f5:::benchmark_data_smithi169_60938_object143228:head# 2024-02-12T16:18:09.614 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcec4b10:::benchmark_data_smithi169_60938_object171704:head# 2024-02-12T16:18:09.615 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcee2528:::benchmark_data_smithi169_60938_object155922:head# 2024-02-12T16:18:09.616 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcf0deda:::benchmark_data_smithi169_60938_object164986:head# 2024-02-12T16:18:09.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcf1b6e8:::benchmark_data_smithi169_60938_object158061:head# 2024-02-12T16:18:09.617 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcf8f6ee:::benchmark_data_smithi169_60938_object160567:head# 2024-02-12T16:18:09.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcfa28d7:::benchmark_data_smithi169_60938_object170288:head# 2024-02-12T16:18:09.618 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcfd6251:::benchmark_data_smithi169_60938_object168408:head# 2024-02-12T16:18:09.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcfd937a:::benchmark_data_smithi169_60938_object138511:head# 2024-02-12T16:18:09.619 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fcfee552:::benchmark_data_smithi169_60938_object144945:head# 2024-02-12T16:18:09.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd02d92d:::benchmark_data_smithi169_60938_object160873:head# 2024-02-12T16:18:09.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd043864:::benchmark_data_smithi169_60938_object144563:head# 2024-02-12T16:18:09.620 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd06933a:::benchmark_data_smithi169_60938_object141061:head# 2024-02-12T16:18:09.621 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd0a24b0:::benchmark_data_smithi169_60938_object165684:head# 2024-02-12T16:18:09.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd0c6d98:::benchmark_data_smithi169_60938_object171578:head# 2024-02-12T16:18:09.622 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd0c92a4:::benchmark_data_smithi169_60938_object171024:head# 2024-02-12T16:18:09.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd0ca6f1:::benchmark_data_smithi169_60938_object139983:head# 2024-02-12T16:18:09.623 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd0f993c:::benchmark_data_smithi169_60938_object143850:head# 2024-02-12T16:18:09.624 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd11381a:::benchmark_data_smithi169_60938_object146962:head# 2024-02-12T16:18:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd136ef6:::benchmark_data_smithi169_60938_object138767:head# 2024-02-12T16:18:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd1489be:::benchmark_data_smithi169_60938_object154984:head# 2024-02-12T16:18:09.626 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd168186:::benchmark_data_smithi169_60938_object154428:head# 2024-02-12T16:18:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd190649:::benchmark_data_smithi169_60938_object161546:head# 2024-02-12T16:18:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd19dd06:::benchmark_data_smithi169_60938_object167957:head# 2024-02-12T16:18:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd1a716f:::benchmark_data_smithi169_60938_object149505:head# 2024-02-12T16:18:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd1dcc38:::benchmark_data_smithi169_60938_object151721:head# 2024-02-12T16:18:09.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd1e0acd:::benchmark_data_smithi169_60938_object158248:head# 2024-02-12T16:18:09.629 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd1e9d31:::benchmark_data_smithi169_60938_object156841:head# 2024-02-12T16:18:09.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd1f20bd:::benchmark_data_smithi169_60938_object171240:head# 2024-02-12T16:18:09.630 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd253c08:::benchmark_data_smithi169_60938_object155599:head# 2024-02-12T16:18:09.631 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd264f1d:::benchmark_data_smithi169_60938_object164029:head# 2024-02-12T16:18:09.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd26a58f:::benchmark_data_smithi169_60938_object156163:head# 2024-02-12T16:18:09.632 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd28d93b:::benchmark_data_smithi169_60938_object160150:head# 2024-02-12T16:18:09.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd29fc8f:::benchmark_data_smithi169_60938_object140935:head# 2024-02-12T16:18:09.633 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd2b9541:::benchmark_data_smithi169_60938_object159730:head# 2024-02-12T16:18:09.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd2c80ba:::benchmark_data_smithi169_60938_object158940:head# 2024-02-12T16:18:09.634 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd2cd37d:::benchmark_data_smithi169_60938_object158441:head# 2024-02-12T16:18:09.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd2d6eb4:::benchmark_data_smithi169_60938_object168079:head# 2024-02-12T16:18:09.635 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd2dde3c:::benchmark_data_smithi169_60938_object171274:head# 2024-02-12T16:18:09.636 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd325aa6:::benchmark_data_smithi169_60938_object151986:head# 2024-02-12T16:18:09.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd329de2:::benchmark_data_smithi169_60938_object157951:head# 2024-02-12T16:18:09.637 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd332276:::benchmark_data_smithi169_60938_object143831:head# 2024-02-12T16:18:09.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd33e10d:::benchmark_data_smithi169_60938_object149257:head# 2024-02-12T16:18:09.638 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd38c69f:::benchmark_data_smithi169_60938_object149592:head# 2024-02-12T16:18:09.639 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd39bbf5:::benchmark_data_smithi169_60938_object160107:head# 2024-02-12T16:18:09.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd3e63cd:::benchmark_data_smithi169_60938_object161524:head# 2024-02-12T16:18:09.640 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd4035f0:::benchmark_data_smithi169_60938_object151540:head# 2024-02-12T16:18:09.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd40cab7:::benchmark_data_smithi169_60938_object169347:head# 2024-02-12T16:18:09.641 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd437324:::benchmark_data_smithi169_60938_object172567:head# 2024-02-12T16:18:09.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd45a017:::benchmark_data_smithi169_60938_object169498:head# 2024-02-12T16:18:09.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd46048d:::benchmark_data_smithi169_60938_object166971:head# 2024-02-12T16:18:09.642 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd4a7bb8:::benchmark_data_smithi169_60938_object150415:head# 2024-02-12T16:18:09.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd4b13ad:::benchmark_data_smithi169_60938_object151971:head# 2024-02-12T16:18:09.643 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd54a07e:::benchmark_data_smithi169_60938_object150224:head# 2024-02-12T16:18:09.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd57c839:::benchmark_data_smithi169_60938_object140563:head# 2024-02-12T16:18:09.644 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd5813c9:::benchmark_data_smithi169_60938_object141793:head# 2024-02-12T16:18:09.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd5b773f:::benchmark_data_smithi169_60938_object141550:head# 2024-02-12T16:18:09.645 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd5c8359:::benchmark_data_smithi169_60938_object159552:head# 2024-02-12T16:18:09.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd5d5c5e:::benchmark_data_smithi169_60938_object164738:head# 2024-02-12T16:18:09.646 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd5e4413:::benchmark_data_smithi169_60938_object143780:head# 2024-02-12T16:18:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd623fb7:::benchmark_data_smithi169_60938_object168725:head# 2024-02-12T16:18:09.647 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd63e08e:::benchmark_data_smithi169_60938_object170540:head# 2024-02-12T16:18:09.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd659e5b:::benchmark_data_smithi169_60938_object151440:head# 2024-02-12T16:18:09.648 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd66884f:::benchmark_data_smithi169_60938_object162885:head# 2024-02-12T16:18:09.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd66abf8:::benchmark_data_smithi169_60938_object157652:head# 2024-02-12T16:18:09.649 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd67c9ae:::benchmark_data_smithi169_60938_object160328:head# 2024-02-12T16:18:09.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd69c896:::benchmark_data_smithi169_60938_object146836:head# 2024-02-12T16:18:09.650 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd6b2c04:::benchmark_data_smithi169_60938_object155173:head# 2024-02-12T16:18:09.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd6b9fcf:::benchmark_data_smithi169_60938_object147410:head# 2024-02-12T16:18:09.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd717d31:::benchmark_data_smithi169_60938_object166255:head# 2024-02-12T16:18:09.651 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd71bf89:::benchmark_data_smithi169_60938_object139817:head# 2024-02-12T16:18:09.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd721579:::benchmark_data_smithi169_60938_object163710:head# 2024-02-12T16:18:09.652 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd73dc5e:::benchmark_data_smithi169_60938_object141618:head# 2024-02-12T16:18:09.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd78a347:::benchmark_data_smithi169_60938_object164892:head# 2024-02-12T16:18:09.653 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd79a239:::benchmark_data_smithi169_60938_object155974:head# 2024-02-12T16:18:09.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd7a57c4:::benchmark_data_smithi169_60938_object148339:head# 2024-02-12T16:18:09.654 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd7f5474:::benchmark_data_smithi169_60938_object166589:head# 2024-02-12T16:18:09.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd7faaed:::benchmark_data_smithi169_60938_object162595:head# 2024-02-12T16:18:09.655 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd8111d5:::benchmark_data_smithi169_60938_object141346:head# 2024-02-12T16:18:09.657 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd817ab8:::benchmark_data_smithi169_60938_object155382:head# 2024-02-12T16:18:09.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd881bd8:::benchmark_data_smithi169_60938_object141463:head# 2024-02-12T16:18:09.658 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd8a930d:::benchmark_data_smithi169_60938_object155281:head# 2024-02-12T16:18:09.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd8e217b:::benchmark_data_smithi169_60938_object154340:head# 2024-02-12T16:18:09.659 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd8ffa41:::benchmark_data_smithi169_60938_object153515:head# 2024-02-12T16:18:09.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd93ffb1:::benchmark_data_smithi169_60938_object165584:head# 2024-02-12T16:18:09.660 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd970925:::benchmark_data_smithi169_60938_object170811:head# 2024-02-12T16:18:09.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd99ed87:::benchmark_data_smithi169_60938_object159334:head# 2024-02-12T16:18:09.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd9a517b:::benchmark_data_smithi169_60938_object158035:head# 2024-02-12T16:18:09.661 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fd9c5fec:::benchmark_data_smithi169_60938_object168631:head# 2024-02-12T16:18:09.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fda11442:::benchmark_data_smithi169_60938_object154031:head# 2024-02-12T16:18:09.662 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fda1539e:::benchmark_data_smithi169_60938_object161540:head# 2024-02-12T16:18:09.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fda1bafe:::benchmark_data_smithi169_60938_object159629:head# 2024-02-12T16:18:09.663 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fda2d58f:::benchmark_data_smithi169_60938_object151307:head# 2024-02-12T16:18:09.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fda4dd1a:::benchmark_data_smithi169_60938_object156871:head# 2024-02-12T16:18:09.664 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fda92bb3:::benchmark_data_smithi169_60938_object167668:head# 2024-02-12T16:18:09.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb3ca44:::benchmark_data_smithi169_60938_object162172:head# 2024-02-12T16:18:09.665 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb401b8:::benchmark_data_smithi169_60938_object153665:head# 2024-02-12T16:18:09.666 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb47391:::benchmark_data_smithi169_60938_object164190:head# 2024-02-12T16:18:09.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb57069:::benchmark_data_smithi169_60938_object139438:head# 2024-02-12T16:18:09.667 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb5b63c:::benchmark_data_smithi169_60938_object165372:head# 2024-02-12T16:18:09.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb66ad0:::benchmark_data_smithi169_60938_object165351:head# 2024-02-12T16:18:09.668 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb6d825:::benchmark_data_smithi169_60938_object171546:head# 2024-02-12T16:18:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb73ce1:::benchmark_data_smithi169_60938_object168013:head# 2024-02-12T16:18:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdb9f554:::benchmark_data_smithi169_60938_object146461:head# 2024-02-12T16:18:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdbe1d00:::benchmark_data_smithi169_60938_object158351:head# 2024-02-12T16:18:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdc15f2f:::benchmark_data_smithi169_60938_object162869:head# 2024-02-12T16:18:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdc24d67:::benchmark_data_smithi169_60938_object138598:head# 2024-02-12T16:18:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdc395e0:::benchmark_data_smithi169_60938_object142960:head# 2024-02-12T16:18:09.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdc6e2de:::benchmark_data_smithi169_60938_object159263:head# 2024-02-12T16:18:09.672 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdd02033:::benchmark_data_smithi169_60938_object161187:head# 2024-02-12T16:18:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdd070e0:::benchmark_data_smithi169_60938_object155566:head# 2024-02-12T16:18:09.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdd19a60:::benchmark_data_smithi169_60938_object148041:head# 2024-02-12T16:18:09.674 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdd64c40:::benchmark_data_smithi169_60938_object170634:head# 2024-02-12T16:18:09.675 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdd749c4:::benchmark_data_smithi169_60938_object162912:head# 2024-02-12T16:18:09.676 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdd7f9a2:::benchmark_data_smithi169_60938_object160489:head# 2024-02-12T16:18:09.677 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdd99329:::benchmark_data_smithi169_60938_object165294:head# 2024-02-12T16:18:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fddb323c:::benchmark_data_smithi169_60938_object162019:head# 2024-02-12T16:18:09.678 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fddbe5a8:::benchmark_data_smithi169_60938_object173199:head# 2024-02-12T16:18:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fddbffaf:::benchmark_data_smithi169_60938_object170108:head# 2024-02-12T16:18:09.679 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdde86f0:::benchmark_data_smithi169_60938_object168315:head# 2024-02-12T16:18:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fde1a0f7:::benchmark_data_smithi169_60938_object148615:head# 2024-02-12T16:18:09.680 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fde3d548:::benchmark_data_smithi169_60938_object144412:head# 2024-02-12T16:18:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fde4dfc7:::benchmark_data_smithi169_60938_object166311:head# 2024-02-12T16:18:09.681 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fde63687:::benchmark_data_smithi169_60938_object147229:head# 2024-02-12T16:18:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdeaf520:::benchmark_data_smithi169_60938_object147388:head# 2024-02-12T16:18:09.682 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fded0ee7:::benchmark_data_smithi169_60938_object148711:head# 2024-02-12T16:18:09.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdee221e:::benchmark_data_smithi169_60938_object138871:head# 2024-02-12T16:18:09.683 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf038cc:::benchmark_data_smithi169_60938_object139460:head# 2024-02-12T16:18:09.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf0fbf4:::benchmark_data_smithi169_60938_object141148:head# 2024-02-12T16:18:09.684 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf48c61:::benchmark_data_smithi169_60938_object141675:head# 2024-02-12T16:18:09.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf5008c:::benchmark_data_smithi169_60938_object151044:head# 2024-02-12T16:18:09.685 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf7daa6:::benchmark_data_smithi169_60938_object159161:head# 2024-02-12T16:18:09.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf83826:::benchmark_data_smithi169_60938_object149007:head# 2024-02-12T16:18:09.686 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf838e2:::benchmark_data_smithi169_60938_object151602:head# 2024-02-12T16:18:09.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdf9ec26:::benchmark_data_smithi169_60938_object158794:head# 2024-02-12T16:18:09.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdfb406e:::benchmark_data_smithi169_60938_object148351:head# 2024-02-12T16:18:09.687 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdfbe114:::benchmark_data_smithi169_60938_object168546:head# 2024-02-12T16:18:09.688 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdfbf5b9:::benchmark_data_smithi169_60938_object164269:head# 2024-02-12T16:18:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdfd8bcd:::benchmark_data_smithi169_60938_object154550:head# 2024-02-12T16:18:09.689 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdfe76b6:::benchmark_data_smithi169_60938_object156115:head# 2024-02-12T16:18:09.690 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fdff0fd7:::benchmark_data_smithi169_60938_object168187:head# 2024-02-12T16:18:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe03c379:::benchmark_data_smithi169_60938_object153804:head# 2024-02-12T16:18:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe072a45:::benchmark_data_smithi169_60938_object146556:head# 2024-02-12T16:18:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe08bb2e:::benchmark_data_smithi169_60938_object161561:head# 2024-02-12T16:18:09.692 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe0b2729:::benchmark_data_smithi169_60938_object168224:head# 2024-02-12T16:18:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe0b8867:::benchmark_data_smithi169_60938_object163629:head# 2024-02-12T16:18:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe0ca3db:::benchmark_data_smithi169_60938_object153026:head# 2024-02-12T16:18:09.693 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe0f38f4:::benchmark_data_smithi169_60938_object141806:head# 2024-02-12T16:18:09.694 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe1128b7:::benchmark_data_smithi169_60938_object143386:head# 2024-02-12T16:18:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe15491b:::benchmark_data_smithi169_60938_object151315:head# 2024-02-12T16:18:09.695 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe174750:::benchmark_data_smithi169_60938_object163977:head# 2024-02-12T16:18:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe19be1e:::benchmark_data_smithi169_60938_object173309:head# 2024-02-12T16:18:09.696 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe1a16ff:::benchmark_data_smithi169_60938_object155100:head# 2024-02-12T16:18:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe1b1b95:::benchmark_data_smithi169_60938_object173004:head# 2024-02-12T16:18:09.697 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe1ce967:::benchmark_data_smithi169_60938_object154525:head# 2024-02-12T16:18:09.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe201ab4:::benchmark_data_smithi169_60938_object169929:head# 2024-02-12T16:18:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe20a69a:::benchmark_data_smithi169_60938_object149964:head# 2024-02-12T16:18:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe20e9ad:::benchmark_data_smithi169_60938_object143006:head# 2024-02-12T16:18:09.699 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2296f1:::benchmark_data_smithi169_60938_object155317:head# 2024-02-12T16:18:09.700 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe23d8d2:::benchmark_data_smithi169_60938_object163730:head# 2024-02-12T16:18:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2a2e49:::benchmark_data_smithi169_60938_object143469:head# 2024-02-12T16:18:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2cc982:::benchmark_data_smithi169_60938_object150679:head# 2024-02-12T16:18:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2cd6e1:::benchmark_data_smithi169_60938_object158920:head# 2024-02-12T16:18:09.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2dc29a:::benchmark_data_smithi169_60938_object140943:head# 2024-02-12T16:18:09.702 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2ddf6b:::benchmark_data_smithi169_60938_object162134:head# 2024-02-12T16:18:09.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2e4ea6:::benchmark_data_smithi169_60938_object151517:head# 2024-02-12T16:18:09.703 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe2ed7d2:::benchmark_data_smithi169_60938_object164743:head# 2024-02-12T16:18:09.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe36c68a:::benchmark_data_smithi169_60938_object165586:head# 2024-02-12T16:18:09.704 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe36e922:::benchmark_data_smithi169_60938_object163779:head# 2024-02-12T16:18:09.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe3a2f36:::benchmark_data_smithi169_60938_object141616:head# 2024-02-12T16:18:09.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe3c9df3:::benchmark_data_smithi169_60938_object151823:head# 2024-02-12T16:18:09.705 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe43e359:::benchmark_data_smithi169_60938_object156279:head# 2024-02-12T16:18:09.706 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe4704ff:::benchmark_data_smithi169_60938_object161303:head# 2024-02-12T16:18:09.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe481b36:::benchmark_data_smithi169_60938_object159097:head# 2024-02-12T16:18:09.707 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe49ff2e:::benchmark_data_smithi169_60938_object166681:head# 2024-02-12T16:18:09.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe515027:::benchmark_data_smithi169_60938_object172467:head# 2024-02-12T16:18:09.708 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe52b4db:::benchmark_data_smithi169_60938_object143979:head# 2024-02-12T16:18:09.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe565e49:::benchmark_data_smithi169_60938_object139338:head# 2024-02-12T16:18:09.709 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe575ccc:::benchmark_data_smithi169_60938_object150416:head# 2024-02-12T16:18:09.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe57ab38:::benchmark_data_smithi169_60938_object151535:head# 2024-02-12T16:18:09.710 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe58ea04:::benchmark_data_smithi169_60938_object148157:head# 2024-02-12T16:18:09.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe594b6b:::benchmark_data_smithi169_60938_object169972:head# 2024-02-12T16:18:09.711 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe5cc310:::benchmark_data_smithi169_60938_object159109:head# 2024-02-12T16:18:09.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe5ced71:::benchmark_data_smithi169_60938_object169247:head# 2024-02-12T16:18:09.712 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe5d0c9d:::benchmark_data_smithi169_60938_object170949:head# 2024-02-12T16:18:09.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe62a33e:::benchmark_data_smithi169_60938_object161113:head# 2024-02-12T16:18:09.713 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe63360d:::benchmark_data_smithi169_60938_object158118:head# 2024-02-12T16:18:09.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe644aa3:::benchmark_data_smithi169_60938_object151135:head# 2024-02-12T16:18:09.714 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe64c4ab:::benchmark_data_smithi169_60938_object148784:head# 2024-02-12T16:18:09.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe64cb02:::benchmark_data_smithi169_60938_object154606:head# 2024-02-12T16:18:09.715 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe674ad9:::benchmark_data_smithi169_60938_object143781:head# 2024-02-12T16:18:09.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe67f6b7:::benchmark_data_smithi169_60938_object155345:head# 2024-02-12T16:18:09.716 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe684e8e:::benchmark_data_smithi169_60938_object170020:head# 2024-02-12T16:18:09.717 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe6abc00:::benchmark_data_smithi169_60938_object154571:head# 2024-02-12T16:18:09.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe6c8fb0:::benchmark_data_smithi169_60938_object145543:head# 2024-02-12T16:18:09.718 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe6df594:::benchmark_data_smithi169_60938_object163521:head# 2024-02-12T16:18:09.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe6e221d:::benchmark_data_smithi169_60938_object172938:head# 2024-02-12T16:18:09.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe6ecb7a:::benchmark_data_smithi169_60938_object165823:head# 2024-02-12T16:18:09.719 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe73407b:::benchmark_data_smithi169_60938_object146533:head# 2024-02-12T16:18:09.720 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe738e85:::benchmark_data_smithi169_60938_object138287:head# 2024-02-12T16:18:09.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe74ba54:::benchmark_data_smithi169_60938_object168015:head# 2024-02-12T16:18:09.721 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe761009:::benchmark_data_smithi169_60938_object139451:head# 2024-02-12T16:18:09.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe7736b8:::benchmark_data_smithi169_60938_object170738:head# 2024-02-12T16:18:09.722 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe774337:::benchmark_data_smithi169_60938_object163824:head# 2024-02-12T16:18:09.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe787815:::benchmark_data_smithi169_60938_object169463:head# 2024-02-12T16:18:09.723 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe7ef450:::benchmark_data_smithi169_60938_object170318:head# 2024-02-12T16:18:09.724 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe8002cc:::benchmark_data_smithi169_60938_object166408:head# 2024-02-12T16:18:09.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe80712b:::benchmark_data_smithi169_60938_object160140:head# 2024-02-12T16:18:09.725 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe84e00f:::benchmark_data_smithi169_60938_object173078:head# 2024-02-12T16:18:09.726 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe882ae6:::benchmark_data_smithi169_60938_object143124:head# 2024-02-12T16:18:09.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe887ab2:::benchmark_data_smithi169_60938_object155001:head# 2024-02-12T16:18:09.727 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe88c499:::benchmark_data_smithi169_60938_object155434:head# 2024-02-12T16:18:09.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe8c0921:::benchmark_data_smithi169_60938_object157421:head# 2024-02-12T16:18:09.728 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe8d78cd:::benchmark_data_smithi169_60938_object163665:head# 2024-02-12T16:18:09.729 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe931705:::benchmark_data_smithi169_60938_object167590:head# 2024-02-12T16:18:09.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe94f5d1:::benchmark_data_smithi169_60938_object167563:head# 2024-02-12T16:18:09.730 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe94fb10:::benchmark_data_smithi169_60938_object142052:head# 2024-02-12T16:18:09.731 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:09.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe95bc62:::benchmark_data_smithi169_60938_object151458:head# 2024-02-12T16:18:09.731 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe988cd8:::benchmark_data_smithi169_60938_object163354:head# 2024-02-12T16:18:09.732 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe997c0a:::benchmark_data_smithi169_60938_object160797:head# 2024-02-12T16:18:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:09.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:09.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe99b11c:::benchmark_data_smithi169_60938_object157103:head# 2024-02-12T16:18:09.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe99b410:::benchmark_data_smithi169_60938_object158901:head# 2024-02-12T16:18:09.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe9b0bcc:::benchmark_data_smithi169_60938_object149294:head# 2024-02-12T16:18:09.734 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe9bae8a:::benchmark_data_smithi169_60938_object155911:head# 2024-02-12T16:18:09.735 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe9c527b:::benchmark_data_smithi169_60938_object149583:head# 2024-02-12T16:18:09.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fe9cd352:::benchmark_data_smithi169_60938_object169384:head# 2024-02-12T16:18:09.736 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fea5747e:::benchmark_data_smithi169_60938_object167022:head# 2024-02-12T16:18:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fea6fd31:::benchmark_data_smithi169_60938_object171231:head# 2024-02-12T16:18:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fea9d61b:::benchmark_data_smithi169_60938_object145811:head# 2024-02-12T16:18:09.738 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fead43c2:::benchmark_data_smithi169_60938_object164600:head# 2024-02-12T16:18:09.739 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feaeb1b1:::benchmark_data_smithi169_60938_object139318:head# 2024-02-12T16:18:09.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feb1e8af:::benchmark_data_smithi169_60938_object142146:head# 2024-02-12T16:18:09.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feb3dbfa:::benchmark_data_smithi169_60938_object164369:head# 2024-02-12T16:18:09.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feb550ee:::benchmark_data_smithi169_60938_object153452:head# 2024-02-12T16:18:09.740 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feb5ba19:::benchmark_data_smithi169_60938_object170157:head# 2024-02-12T16:18:09.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feb5f47d:::benchmark_data_smithi169_60938_object169027:head# 2024-02-12T16:18:09.741 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feb885c7:::benchmark_data_smithi169_60938_object170964:head# 2024-02-12T16:18:09.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:febb3e19:::benchmark_data_smithi169_60938_object147440:head# 2024-02-12T16:18:09.742 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:febbc125:::benchmark_data_smithi169_60938_object147972:head# 2024-02-12T16:18:09.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:febbec3a:::benchmark_data_smithi169_60938_object148881:head# 2024-02-12T16:18:09.743 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:febbfc1d:::benchmark_data_smithi169_60938_object162445:head# 2024-02-12T16:18:09.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:febd626e:::benchmark_data_smithi169_60938_object165159:head# 2024-02-12T16:18:09.744 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:febf49fc:::benchmark_data_smithi169_60938_object169447:head# 2024-02-12T16:18:09.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fec16b8c:::benchmark_data_smithi169_60938_object160133:head# 2024-02-12T16:18:09.745 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fec17549:::benchmark_data_smithi169_60938_object153998:head# 2024-02-12T16:18:09.746 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fec2b25a:::benchmark_data_smithi169_60938_object158199:head# 2024-02-12T16:18:09.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fec6c631:::benchmark_data_smithi169_60938_object154575:head# 2024-02-12T16:18:09.747 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fec9ae98:::benchmark_data_smithi169_60938_object149418:head# 2024-02-12T16:18:09.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feca662c:::benchmark_data_smithi169_60938_object166683:head# 2024-02-12T16:18:09.748 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fecb1be3:::benchmark_data_smithi169_60938_object149429:head# 2024-02-12T16:18:09.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fece3af1:::benchmark_data_smithi169_60938_object151025:head# 2024-02-12T16:18:09.749 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fecf4e07:::benchmark_data_smithi169_60938_object140523:head# 2024-02-12T16:18:09.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fed22703:::benchmark_data_smithi169_60938_object150655:head# 2024-02-12T16:18:09.750 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fed43ec6:::benchmark_data_smithi169_60938_object164116:head# 2024-02-12T16:18:09.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fed467e0:::benchmark_data_smithi169_60938_object144025:head# 2024-02-12T16:18:09.751 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fed4d477:::benchmark_data_smithi169_60938_object152250:head# 2024-02-12T16:18:09.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fed78bff:::benchmark_data_smithi169_60938_object167579:head# 2024-02-12T16:18:09.752 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fed804cc:::benchmark_data_smithi169_60938_object146519:head# 2024-02-12T16:18:09.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feda3ca2:::benchmark_data_smithi169_60938_object162710:head# 2024-02-12T16:18:09.753 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fedbabac:::benchmark_data_smithi169_60938_object138993:head# 2024-02-12T16:18:09.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fedf51ef:::benchmark_data_smithi169_60938_object158335:head# 2024-02-12T16:18:09.754 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fedf6f33:::benchmark_data_smithi169_60938_object140117:head# 2024-02-12T16:18:09.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fee2d41a:::benchmark_data_smithi169_60938_object166378:head# 2024-02-12T16:18:09.755 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fee3efb2:::benchmark_data_smithi169_60938_object166270:head# 2024-02-12T16:18:09.756 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fee63525:::benchmark_data_smithi169_60938_object171480:head# 2024-02-12T16:18:09.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fee64c65:::benchmark_data_smithi169_60938_object157427:head# 2024-02-12T16:18:09.757 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fee68685:::benchmark_data_smithi169_60938_object161294:head# 2024-02-12T16:18:09.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feeca0a5:::benchmark_data_smithi169_60938_object165723:head# 2024-02-12T16:18:09.758 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feee27c3:::benchmark_data_smithi169_60938_object156418:head# 2024-02-12T16:18:09.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fef04cc5:::benchmark_data_smithi169_60938_object157494:head# 2024-02-12T16:18:09.759 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fef29978:::benchmark_data_smithi169_60938_object146535:head# 2024-02-12T16:18:09.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fef7a14b:::benchmark_data_smithi169_60938_object164727:head# 2024-02-12T16:18:09.760 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fef800d5:::benchmark_data_smithi169_60938_object145754:head# 2024-02-12T16:18:09.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fef996e1:::benchmark_data_smithi169_60938_object151485:head# 2024-02-12T16:18:09.761 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fef99994:::benchmark_data_smithi169_60938_object159003:head# 2024-02-12T16:18:09.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fefba263:::benchmark_data_smithi169_60938_object147879:head# 2024-02-12T16:18:09.762 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fefdfd6e:::benchmark_data_smithi169_60938_object145726:head# 2024-02-12T16:18:09.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:feff6933:::benchmark_data_smithi169_60938_object157264:head# 2024-02-12T16:18:09.763 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff019c16:::benchmark_data_smithi169_60938_object169544:head# 2024-02-12T16:18:09.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff02baea:::benchmark_data_smithi169_60938_object148225:head# 2024-02-12T16:18:09.764 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff03fa79:::benchmark_data_smithi169_60938_object160242:head# 2024-02-12T16:18:09.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff07ee47:::benchmark_data_smithi169_60938_object169295:head# 2024-02-12T16:18:09.765 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff090a50:::benchmark_data_smithi169_60938_object144658:head# 2024-02-12T16:18:09.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff093519:::benchmark_data_smithi169_60938_object172168:head# 2024-02-12T16:18:09.766 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff09aabf:::benchmark_data_smithi169_60938_object169999:head# 2024-02-12T16:18:09.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff09ac7e:::benchmark_data_smithi169_60938_object170846:head# 2024-02-12T16:18:09.767 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff09c0ad:::benchmark_data_smithi169_60938_object157446:head# 2024-02-12T16:18:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff09ffe6:::benchmark_data_smithi169_60938_object150763:head# 2024-02-12T16:18:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff0a780e:::benchmark_data_smithi169_60938_object161548:head# 2024-02-12T16:18:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff0c1e3d:::benchmark_data_smithi169_60938_object168810:head# 2024-02-12T16:18:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff0ddcc4:::benchmark_data_smithi169_60938_object141920:head# 2024-02-12T16:18:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff139aa0:::benchmark_data_smithi169_60938_object171227:head# 2024-02-12T16:18:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff156182:::benchmark_data_smithi169_60938_object156868:head# 2024-02-12T16:18:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff167ce4:::benchmark_data_smithi169_60938_object170047:head# 2024-02-12T16:18:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff1e7297:::benchmark_data_smithi169_60938_object166585:head# 2024-02-12T16:18:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff1ee92e:::benchmark_data_smithi169_60938_object143455:head# 2024-02-12T16:18:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff1fe42c:::benchmark_data_smithi169_60938_object139957:head# 2024-02-12T16:18:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff218956:::benchmark_data_smithi169_60938_object144769:head# 2024-02-12T16:18:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff22dcbb:::benchmark_data_smithi169_60938_object154844:head# 2024-02-12T16:18:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff250980:::benchmark_data_smithi169_60938_object156704:head# 2024-02-12T16:18:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff25b2be:::benchmark_data_smithi169_60938_object146995:head# 2024-02-12T16:18:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff262112:::benchmark_data_smithi169_60938_object139504:head# 2024-02-12T16:18:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff29b138:::benchmark_data_smithi169_60938_object171675:head# 2024-02-12T16:18:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff2afa78:::benchmark_data_smithi169_60938_object150961:head# 2024-02-12T16:18:09.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff2b757c:::benchmark_data_smithi169_60938_object171375:head# 2024-02-12T16:18:09.777 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff2c5618:::benchmark_data_smithi169_60938_object145556:head# 2024-02-12T16:18:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff2d2d30:::benchmark_data_smithi169_60938_object162269:head# 2024-02-12T16:18:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff39d6d0:::benchmark_data_smithi169_60938_object151891:head# 2024-02-12T16:18:09.779 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff3f142f:::benchmark_data_smithi169_60938_object160811:head# 2024-02-12T16:18:09.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff3f84fb:::benchmark_data_smithi169_60938_object152733:head# 2024-02-12T16:18:09.780 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff4003a5:::benchmark_data_smithi169_60938_object165014:head# 2024-02-12T16:18:09.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff4ed405:::benchmark_data_smithi169_60938_object168347:head# 2024-02-12T16:18:09.781 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff503ee7:::benchmark_data_smithi169_60938_object156988:head# 2024-02-12T16:18:09.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff509ea6:::benchmark_data_smithi169_60938_object144747:head# 2024-02-12T16:18:09.782 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff5310de:::benchmark_data_smithi169_60938_object148648:head# 2024-02-12T16:18:09.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff53f4f5:::benchmark_data_smithi169_60938_object156046:head# 2024-02-12T16:18:09.783 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff55b24f:::benchmark_data_smithi169_60938_object156760:head# 2024-02-12T16:18:09.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff575b98:::benchmark_data_smithi169_60938_object163905:head# 2024-02-12T16:18:09.784 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff5a897d:::benchmark_data_smithi169_60938_object156108:head# 2024-02-12T16:18:09.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff5c2209:::benchmark_data_smithi169_60938_object146562:head# 2024-02-12T16:18:09.785 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff5c7a0a:::benchmark_data_smithi169_60938_object144947:head# 2024-02-12T16:18:09.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff5f2240:::benchmark_data_smithi169_60938_object146136:head# 2024-02-12T16:18:09.786 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff60cdce:::benchmark_data_smithi169_60938_object155657:head# 2024-02-12T16:18:09.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff610490:::benchmark_data_smithi169_60938_object147651:head# 2024-02-12T16:18:09.787 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff6119fe:::benchmark_data_smithi169_60938_object144985:head# 2024-02-12T16:18:09.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff64c2c9:::benchmark_data_smithi169_60938_object170971:head# 2024-02-12T16:18:09.788 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff6683e9:::benchmark_data_smithi169_60938_object147858:head# 2024-02-12T16:18:09.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff66f419:::benchmark_data_smithi169_60938_object151391:head# 2024-02-12T16:18:09.789 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff6a6739:::benchmark_data_smithi169_60938_object171264:head# 2024-02-12T16:18:09.790 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff6d74b6:::benchmark_data_smithi169_60938_object150869:head# 2024-02-12T16:18:09.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff718128:::benchmark_data_smithi169_60938_object156752:head# 2024-02-12T16:18:09.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff71da30:::benchmark_data_smithi169_60938_object145140:head# 2024-02-12T16:18:09.791 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff721dec:::benchmark_data_smithi169_60938_object167373:head# 2024-02-12T16:18:09.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff752995:::benchmark_data_smithi169_60938_object165061:head# 2024-02-12T16:18:09.792 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff768241:::benchmark_data_smithi169_60938_object156677:head# 2024-02-12T16:18:09.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff7763e4:::benchmark_data_smithi169_60938_object171569:head# 2024-02-12T16:18:09.793 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff7b82fe:::benchmark_data_smithi169_60938_object164176:head# 2024-02-12T16:18:09.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff7e1b07:::benchmark_data_smithi169_60938_object146409:head# 2024-02-12T16:18:09.794 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff7efae0:::benchmark_data_smithi169_60938_object165521:head# 2024-02-12T16:18:09.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff81323d:::benchmark_data_smithi169_60938_object152728:head# 2024-02-12T16:18:09.795 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff814094:::benchmark_data_smithi169_60938_object163511:head# 2024-02-12T16:18:09.796 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff8176a6:::benchmark_data_smithi169_60938_object142617:head# 2024-02-12T16:18:09.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff8ab3ef:::benchmark_data_smithi169_60938_object154403:head# 2024-02-12T16:18:09.797 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff8b4ec0:::benchmark_data_smithi169_60938_object162490:head# 2024-02-12T16:18:09.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff90f733:::benchmark_data_smithi169_60938_object155452:head# 2024-02-12T16:18:09.798 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff91086a:::benchmark_data_smithi169_60938_object152287:head# 2024-02-12T16:18:09.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff932c03:::benchmark_data_smithi169_60938_object166383:head# 2024-02-12T16:18:09.799 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff940597:::benchmark_data_smithi169_60938_object151090:head# 2024-02-12T16:18:09.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff95ac3a:::benchmark_data_smithi169_60938_object170075:head# 2024-02-12T16:18:09.800 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff96b71c:::benchmark_data_smithi169_60938_object159275:head# 2024-02-12T16:18:09.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff9911ca:::benchmark_data_smithi169_60938_object159709:head# 2024-02-12T16:18:09.801 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff99535b:::benchmark_data_smithi169_60938_object152555:head# 2024-02-12T16:18:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff9a3b7f:::benchmark_data_smithi169_60938_object156549:head# 2024-02-12T16:18:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff9c08e3:::benchmark_data_smithi169_60938_object149665:head# 2024-02-12T16:18:09.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff9c599b:::benchmark_data_smithi169_60938_object140401:head# 2024-02-12T16:18:09.803 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff9c8270:::benchmark_data_smithi169_60938_object172407:head# 2024-02-12T16:18:09.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ff9e6ce0:::benchmark_data_smithi169_60938_object145604:head# 2024-02-12T16:18:09.804 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffa2338f:::benchmark_data_smithi169_60938_object157126:head# 2024-02-12T16:18:09.805 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffa23b04:::benchmark_data_smithi169_60938_object161350:head# 2024-02-12T16:18:09.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffa909a7:::benchmark_data_smithi169_60938_object145393:head# 2024-02-12T16:18:09.806 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffaa4766:::benchmark_data_smithi169_60938_object167514:head# 2024-02-12T16:18:09.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffad0ecb:::benchmark_data_smithi169_60938_object146129:head# 2024-02-12T16:18:09.807 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffad6f72:::benchmark_data_smithi169_60938_object149309:head# 2024-02-12T16:18:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffb10c52:::benchmark_data_smithi169_60938_object169107:head# 2024-02-12T16:18:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffb3c4e3:::benchmark_data_smithi169_60938_object171861:head# 2024-02-12T16:18:09.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffb606b7:::benchmark_data_smithi169_60938_object146753:head# 2024-02-12T16:18:09.809 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffb6f315:::benchmark_data_smithi169_60938_object138509:head# 2024-02-12T16:18:09.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffb6f913:::benchmark_data_smithi169_60938_object164655:head# 2024-02-12T16:18:09.810 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffb8afcb:::benchmark_data_smithi169_60938_object157736:head# 2024-02-12T16:18:09.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffbadf60:::benchmark_data_smithi169_60938_object169755:head# 2024-02-12T16:18:09.811 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc00ba6:::benchmark_data_smithi169_60938_object164789:head# 2024-02-12T16:18:09.812 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc29121:::benchmark_data_smithi169_60938_object159200:head# 2024-02-12T16:18:09.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc29c23:::benchmark_data_smithi169_60938_object165951:head# 2024-02-12T16:18:09.813 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc45f7c:::benchmark_data_smithi169_60938_object162547:head# 2024-02-12T16:18:09.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc5c354:::benchmark_data_smithi169_60938_object166204:head# 2024-02-12T16:18:09.814 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc66b72:::benchmark_data_smithi169_60938_object163142:head# 2024-02-12T16:18:09.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc6f171:::benchmark_data_smithi169_60938_object142029:head# 2024-02-12T16:18:09.815 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc7078f:::benchmark_data_smithi169_60938_object173150:head# 2024-02-12T16:18:09.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffc99f61:::benchmark_data_smithi169_60938_object167617:head# 2024-02-12T16:18:09.816 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffcb74a3:::benchmark_data_smithi169_60938_object166706:head# 2024-02-12T16:18:09.817 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffcdcba4:::benchmark_data_smithi169_60938_object140031:head# 2024-02-12T16:18:09.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffcdd95b:::benchmark_data_smithi169_60938_object165552:head# 2024-02-12T16:18:09.818 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffceaf83:::benchmark_data_smithi169_60938_object160516:head# 2024-02-12T16:18:09.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffd675c8:::benchmark_data_smithi169_60938_object171253:head# 2024-02-12T16:18:09.819 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffd74667:::benchmark_data_smithi169_60938_object156891:head# 2024-02-12T16:18:09.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffd90856:::benchmark_data_smithi169_60938_object160239:head# 2024-02-12T16:18:09.820 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffded052:::benchmark_data_smithi169_60938_object152036:head# 2024-02-12T16:18:09.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffe01d4b:::benchmark_data_smithi169_60938_object141581:head# 2024-02-12T16:18:09.821 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffe11e3f:::benchmark_data_smithi169_60938_object162685:head# 2024-02-12T16:18:09.822 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffe455e5:::benchmark_data_smithi169_60938_object156022:head# 2024-02-12T16:18:09.823 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffe45f92:::benchmark_data_smithi169_60938_object151979:head# 2024-02-12T16:18:09.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffe8eea8:::benchmark_data_smithi169_60938_object159218:head# 2024-02-12T16:18:09.824 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffe9d2f0:::benchmark_data_smithi169_60938_object164821:head# 2024-02-12T16:18:09.825 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffeb371c:::benchmark_data_smithi169_60938_object148230:head# 2024-02-12T16:18:09.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffedac52:::benchmark_data_smithi169_60938_object172531:head# 2024-02-12T16:18:09.826 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffee3a07:::benchmark_data_smithi169_60938_object140024:head# 2024-02-12T16:18:09.827 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:ffee8215:::benchmark_data_smithi169_60938_object156153:head# 2024-02-12T16:18:09.828 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fff035e2:::benchmark_data_smithi169_60938_object146825:head# 2024-02-12T16:18:09.829 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fff03f4e:::benchmark_data_smithi169_60938_object150917:head# 2024-02-12T16:18:09.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fff99d7f:::benchmark_data_smithi169_60938_object156078:head# 2024-02-12T16:18:09.830 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fff9e285:::benchmark_data_smithi169_60938_object171846:head# 2024-02-12T16:18:09.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fffa9af9:::benchmark_data_smithi169_60938_object139816:head# 2024-02-12T16:18:09.831 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fffbea3f:::benchmark_data_smithi169_60938_object152674:head# 2024-02-12T16:18:09.832 INFO:teuthology.orchestra.run.smithi121.stderr:Read #4:fffeb1b6:::benchmark_data_smithi169_60938_object166285:head# 2024-02-12T16:18:09.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:09.839 INFO:teuthology.orchestra.run.smithi121.stderr:Export successful 2024-02-12T16:18:09.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:09.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:09 smithi123 bash[19119]: cluster 2024-02-12T16:18:08.660691+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:09 smithi123 bash[19119]: audit 2024-02-12T16:18:08.673262+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? v1:172.21.15.121:0/1806690187' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:18:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:09 smithi123 bash[19119]: cluster 2024-02-12T16:18:08.673298+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e427: 12 total, 11 up, 11 in 2024-02-12T16:18:09.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:09.988 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:09 smithi169 bash[19604]: cluster 2024-02-12T16:18:08.660691+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:09.988 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:09 smithi169 bash[19604]: audit 2024-02-12T16:18:08.673262+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? v1:172.21.15.121:0/1806690187' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:18:09.988 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:09 smithi169 bash[19604]: cluster 2024-02-12T16:18:08.673298+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e427: 12 total, 11 up, 11 in 2024-02-12T16:18:10.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:10.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:10.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:10.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:10.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:09 smithi121 bash[18489]: cluster 2024-02-12T16:18:08.660691+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:10.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:09 smithi121 bash[18489]: audit 2024-02-12T16:18:08.673262+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? v1:172.21.15.121:0/1806690187' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:18:10.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:09 smithi121 bash[18489]: cluster 2024-02-12T16:18:08.673298+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e427: 12 total, 11 up, 11 in 2024-02-12T16:18:10.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:10.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:10.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:10.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:10.437 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:10.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:10.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:18:10.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:10.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:10.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:10.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:10.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:10.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:10.820 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:10.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:10.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:10.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:10.974 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:10 smithi123 bash[19119]: cluster 2024-02-12T16:18:09.762789+0000 mgr.y (mgr.14146) 986 : cluster [DBG] pgmap v1313: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:11.001 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:10 smithi169 bash[19604]: cluster 2024-02-12T16:18:09.762789+0000 mgr.y (mgr.14146) 986 : cluster [DBG] pgmap v1313: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:11.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:11.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:11.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:10 smithi121 bash[18489]: cluster 2024-02-12T16:18:09.762789+0000 mgr.y (mgr.14146) 986 : cluster [DBG] pgmap v1313: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:11.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:11.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:11.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:11.420 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:11.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:11.561 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.564 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:11.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:11.701 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:11.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.3 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 4.f 2024-02-12T16:18:11.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:18:11.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:11.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.3/config 2024-02-12T16:18:11.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:11.993 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:11.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:12.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:12.133 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:12.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:12.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:12.276 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:12.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:12.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:12.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:12.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:12.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:12.561 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:12.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:12.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:12.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:12.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:12.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:12.845 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:12.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:12.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:12.988 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:12.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:12.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:12 smithi121 bash[18489]: cluster 2024-02-12T16:18:11.763550+0000 mgr.y (mgr.14146) 987 : cluster [DBG] pgmap v1314: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:13.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:13.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:13.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:12 smithi123 bash[19119]: cluster 2024-02-12T16:18:11.763550+0000 mgr.y (mgr.14146) 987 : cluster [DBG] pgmap v1314: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:13.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:13.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:12 smithi169 bash[19604]: cluster 2024-02-12T16:18:11.763550+0000 mgr.y (mgr.14146) 987 : cluster [DBG] pgmap v1314: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:13.270 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:13.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:13.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:13.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:13.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:13.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:13.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:13.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:13.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:13.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:13.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:13.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:13.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:13.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:13.937 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:13.974 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:13.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:14.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:14.116 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:14.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:14.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:14.258 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:14.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:14.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:14.403 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:14.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:14.449 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:14.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:14.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:14.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:14.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:14.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:14.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:14.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:14.832 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:14.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:14.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:14.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:14.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:14.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:15.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:15.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:14 smithi121 bash[18489]: cluster 2024-02-12T16:18:13.418241+0000 osd.6 (osd.6) 62 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:15.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:14 smithi121 bash[18489]: cluster 2024-02-12T16:18:13.418330+0000 osd.6 (osd.6) 63 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:15.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:14 smithi121 bash[18489]: cluster 2024-02-12T16:18:13.764515+0000 mgr.y (mgr.14146) 988 : cluster [DBG] pgmap v1315: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:15.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:14 smithi121 bash[18489]: cluster 2024-02-12T16:18:13.835905+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e428: 12 total, 11 up, 11 in 2024-02-12T16:18:15.113 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:15.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:14 smithi123 bash[19119]: cluster 2024-02-12T16:18:13.418241+0000 osd.6 (osd.6) 62 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:14 smithi123 bash[19119]: cluster 2024-02-12T16:18:13.418330+0000 osd.6 (osd.6) 63 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:14 smithi123 bash[19119]: cluster 2024-02-12T16:18:13.764515+0000 mgr.y (mgr.14146) 988 : cluster [DBG] pgmap v1315: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:14 smithi123 bash[19119]: cluster 2024-02-12T16:18:13.835905+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e428: 12 total, 11 up, 11 in 2024-02-12T16:18:15.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:15.229 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:14 smithi169 bash[19604]: cluster 2024-02-12T16:18:13.418241+0000 osd.6 (osd.6) 62 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:15.229 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:14 smithi169 bash[19604]: cluster 2024-02-12T16:18:13.418330+0000 osd.6 (osd.6) 63 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:15.229 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:14 smithi169 bash[19604]: cluster 2024-02-12T16:18:13.764515+0000 mgr.y (mgr.14146) 988 : cluster [DBG] pgmap v1315: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:15.229 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:14 smithi169 bash[19604]: cluster 2024-02-12T16:18:13.835905+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e428: 12 total, 11 up, 11 in 2024-02-12T16:18:15.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:15.254 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:15.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:15.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:15.400 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:15.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:15.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:15.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:15.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:15.678 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:15.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:15.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:15.823 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:15.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:15.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:15.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:15.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:16.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:16.109 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:16.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:16.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:16.250 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:16.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:16.267 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:16.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:16.389 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:16.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:16.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:16.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:16.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:16.606 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:16 smithi121 bash[18489]: cluster 2024-02-12T16:18:15.448343+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e429: 12 total, 11 up, 11 in 2024-02-12T16:18:16.607 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:16 smithi121 bash[18489]: cluster 2024-02-12T16:18:15.765230+0000 mgr.y (mgr.14146) 989 : cluster [DBG] pgmap v1318: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:16.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:16.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:16 smithi123 bash[19119]: cluster 2024-02-12T16:18:15.448343+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e429: 12 total, 11 up, 11 in 2024-02-12T16:18:16.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:16 smithi123 bash[19119]: cluster 2024-02-12T16:18:15.765230+0000 mgr.y (mgr.14146) 989 : cluster [DBG] pgmap v1318: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:16.669 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:16.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:16.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:16.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:16 smithi169 bash[19604]: cluster 2024-02-12T16:18:15.448343+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e429: 12 total, 11 up, 11 in 2024-02-12T16:18:16.784 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:16 smithi169 bash[19604]: cluster 2024-02-12T16:18:15.765230+0000 mgr.y (mgr.14146) 989 : cluster [DBG] pgmap v1318: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:16.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:16.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:16.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:16.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:16.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: marking collection for removal 2024-02-12T16:18:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:setting '_remove' omap key 2024-02-12T16:18:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:finish_remove_pgs 4.f_head removing 4.f 2024-02-12T16:18:17.033 INFO:teuthology.orchestra.run.smithi121.stdout:Remove successful 2024-02-12T16:18:17.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:17.089 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:17.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:17.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:17.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:17.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:17.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:17.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:17.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:17.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:17.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:17.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:17.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:17.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:17.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:17.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:17.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:17.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:17.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:17.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:17.934 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:17.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:18.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:18.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:18.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:18.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:18.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:18.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:18.311 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:18.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:18.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:18.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:18.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:18.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:18.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:18.635 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:18.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:18.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:18.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:18.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:18.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:18.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:18 smithi123 bash[19119]: cluster 2024-02-12T16:18:17.765993+0000 mgr.y (mgr.14146) 990 : cluster [DBG] pgmap v1319: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:18.916 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:18.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:19.008 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:18 smithi121 bash[18489]: cluster 2024-02-12T16:18:17.765993+0000 mgr.y (mgr.14146) 990 : cluster [DBG] pgmap v1319: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:19.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:18 smithi169 bash[19604]: cluster 2024-02-12T16:18:17.765993+0000 mgr.y (mgr.14146) 990 : cluster [DBG] pgmap v1319: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:19.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:19.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:19.159 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:19.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:19.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:19.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:19.335 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:19.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:19.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:19.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:19.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:19.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:19.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:19.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:19.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.3 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.f.3 2024-02-12T16:18:19.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:19.755 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:19.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:19.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:19.895 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.3/config 2024-02-12T16:18:19.896 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:19.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:20.000 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:20.039 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:20.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:20.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:20.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:20.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:20.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:20.322 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:20.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:20.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:20.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:20.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:20.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:20.605 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:20.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:20.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:20.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:20.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:20.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:20.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:20.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:20.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:21.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:21.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:21.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:21.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:20 smithi123 bash[19119]: cluster 2024-02-12T16:18:19.766593+0000 mgr.y (mgr.14146) 991 : cluster [DBG] pgmap v1320: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:21.148 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:20 smithi169 bash[19604]: cluster 2024-02-12T16:18:19.766593+0000 mgr.y (mgr.14146) 991 : cluster [DBG] pgmap v1320: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:21.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:20 smithi121 bash[18489]: cluster 2024-02-12T16:18:19.766593+0000 mgr.y (mgr.14146) 991 : cluster [DBG] pgmap v1320: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:21.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:21.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:21.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:21.317 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:21.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:21.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:21.460 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:21.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:21.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:21.599 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:21.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:21.681 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:18:21.681 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:18:21.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:21.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:21.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:21.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:21.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:21.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:21.981 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:22.020 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:22.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:22.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:22.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:21 smithi121 bash[18489]: cluster 2024-02-12T16:18:20.915497+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:22.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:21 smithi121 bash[18489]: cluster 2024-02-12T16:18:20.915583+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:22.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:21 smithi123 bash[19119]: cluster 2024-02-12T16:18:20.915497+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:22.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:21 smithi123 bash[19119]: cluster 2024-02-12T16:18:20.915583+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:22.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:22.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:22.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:22.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:21 smithi169 bash[19604]: cluster 2024-02-12T16:18:20.915497+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:21 smithi169 bash[19604]: cluster 2024-02-12T16:18:20.915583+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:22.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:22.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:22.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:22.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:22.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:22.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:22.592 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:22.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:22.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:22.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:22.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:22.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:22.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:22.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:22.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:23.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:22 smithi169 bash[19604]: cluster 2024-02-12T16:18:21.767120+0000 mgr.y (mgr.14146) 992 : cluster [DBG] pgmap v1321: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:22 smithi169 bash[19604]: cluster 2024-02-12T16:18:21.841169+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e430: 12 total, 11 up, 11 in 2024-02-12T16:18:23.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:23.127 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:22 smithi121 bash[18489]: cluster 2024-02-12T16:18:21.767120+0000 mgr.y (mgr.14146) 992 : cluster [DBG] pgmap v1321: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:23.127 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:22 smithi121 bash[18489]: cluster 2024-02-12T16:18:21.841169+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e430: 12 total, 11 up, 11 in 2024-02-12T16:18:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:22 smithi123 bash[19119]: cluster 2024-02-12T16:18:21.767120+0000 mgr.y (mgr.14146) 992 : cluster [DBG] pgmap v1321: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:22 smithi123 bash[19119]: cluster 2024-02-12T16:18:21.841169+0000 mon.a (mon.0) 1587 : cluster [DBG] osdmap e430: 12 total, 11 up, 11 in 2024-02-12T16:18:23.150 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:23.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.296 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:23.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:23.387 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:18:23.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:23.417 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:18:23.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:23.441 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:18:23.470 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:18:23.494 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:18:23.517 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:18:23.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:23.545 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:18:23.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:23.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:18:23.581 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:18:23.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:18:23.631 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:18:23.669 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:18:23.669 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.669 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.669 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.669 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.670 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:18:23.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:23.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:23.843 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:23.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:23.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:23.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:24.081 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:24.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:24.118 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:23 smithi123 bash[19119]: cluster 2024-02-12T16:18:22.842599+0000 osd.5 (osd.5) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:24.118 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:23 smithi123 bash[19119]: cluster 2024-02-12T16:18:22.842692+0000 osd.5 (osd.5) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:24.119 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:23 smithi123 bash[19119]: audit 2024-02-12T16:18:23.361021+0000 mon.b (mon.2) 129 : audit [DBG] from='client.? v1:172.21.15.121:0/429214411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:24.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:24.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:23 smithi121 bash[18489]: cluster 2024-02-12T16:18:22.842599+0000 osd.5 (osd.5) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:24.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:23 smithi121 bash[18489]: cluster 2024-02-12T16:18:22.842692+0000 osd.5 (osd.5) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:24.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:23 smithi121 bash[18489]: audit 2024-02-12T16:18:23.361021+0000 mon.b (mon.2) 129 : audit [DBG] from='client.? v1:172.21.15.121:0/429214411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:24.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:24.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:24.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:24.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:23 smithi169 bash[19604]: cluster 2024-02-12T16:18:22.842599+0000 osd.5 (osd.5) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:24.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:23 smithi169 bash[19604]: cluster 2024-02-12T16:18:22.842692+0000 osd.5 (osd.5) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:23 smithi169 bash[19604]: audit 2024-02-12T16:18:23.361021+0000 mon.b (mon.2) 129 : audit [DBG] from='client.? v1:172.21.15.121:0/429214411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:24.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:24.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:24.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:24.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:24.541 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:24.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:24.645 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:24.681 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:24.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:24.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:24.814 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:24.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:18:24.835 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:18:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:18:24.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:24.948 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:24.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:25.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:25.090 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:25.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:24 smithi123 bash[19119]: cluster 2024-02-12T16:18:23.767727+0000 mgr.y (mgr.14146) 993 : cluster [DBG] pgmap v1323: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:25.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:24 smithi123 bash[19119]: cluster 2024-02-12T16:18:23.852263+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e431: 12 total, 11 up, 11 in 2024-02-12T16:18:25.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:24 smithi123 bash[19119]: audit 2024-02-12T16:18:23.959981+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? v1:172.21.15.121:0/1860578835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:18:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:24 smithi121 bash[18489]: cluster 2024-02-12T16:18:23.767727+0000 mgr.y (mgr.14146) 993 : cluster [DBG] pgmap v1323: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:24 smithi121 bash[18489]: cluster 2024-02-12T16:18:23.852263+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e431: 12 total, 11 up, 11 in 2024-02-12T16:18:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:24 smithi121 bash[18489]: audit 2024-02-12T16:18:23.959981+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? v1:172.21.15.121:0/1860578835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:18:25.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:25.230 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:25.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:24 smithi169 bash[19604]: cluster 2024-02-12T16:18:23.767727+0000 mgr.y (mgr.14146) 993 : cluster [DBG] pgmap v1323: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:24 smithi169 bash[19604]: cluster 2024-02-12T16:18:23.852263+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e431: 12 total, 11 up, 11 in 2024-02-12T16:18:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:24 smithi169 bash[19604]: audit 2024-02-12T16:18:23.959981+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? v1:172.21.15.121:0/1860578835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:18:25.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:25.373 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:25.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:25.476 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:25.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:25.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:25.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:25.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:25.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:25.759 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:25.792 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:25.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:25.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:25.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:25.937 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:25.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:26.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:26.079 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:26.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:25 smithi123 bash[19119]: cluster 2024-02-12T16:18:24.280047+0000 osd.11 (osd.11) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:25 smithi123 bash[19119]: cluster 2024-02-12T16:18:24.280136+0000 osd.11 (osd.11) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:25 smithi123 bash[19119]: cluster 2024-02-12T16:18:24.849495+0000 mon.a (mon.0) 1590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:26.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:25 smithi123 bash[19119]: audit 2024-02-12T16:18:24.862161+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? v1:172.21.15.121:0/1860578835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:18:26.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:25 smithi123 bash[19119]: cluster 2024-02-12T16:18:24.862265+0000 mon.a (mon.0) 1592 : cluster [DBG] osdmap e432: 12 total, 11 up, 11 in 2024-02-12T16:18:26.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:25 smithi123 bash[19119]: cluster 2024-02-12T16:18:25.453520+0000 mon.a (mon.0) 1593 : cluster [DBG] osdmap e433: 12 total, 11 up, 11 in 2024-02-12T16:18:26.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:25 smithi121 bash[18489]: cluster 2024-02-12T16:18:24.280047+0000 osd.11 (osd.11) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:26.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:25 smithi121 bash[18489]: cluster 2024-02-12T16:18:24.280136+0000 osd.11 (osd.11) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:26.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:25 smithi121 bash[18489]: cluster 2024-02-12T16:18:24.849495+0000 mon.a (mon.0) 1590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:26.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:25 smithi121 bash[18489]: audit 2024-02-12T16:18:24.862161+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? v1:172.21.15.121:0/1860578835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:18:26.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:25 smithi121 bash[18489]: cluster 2024-02-12T16:18:24.862265+0000 mon.a (mon.0) 1592 : cluster [DBG] osdmap e432: 12 total, 11 up, 11 in 2024-02-12T16:18:26.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:25 smithi121 bash[18489]: cluster 2024-02-12T16:18:25.453520+0000 mon.a (mon.0) 1593 : cluster [DBG] osdmap e433: 12 total, 11 up, 11 in 2024-02-12T16:18:26.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:26.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:26.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:26.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:25 smithi169 bash[19604]: cluster 2024-02-12T16:18:24.280047+0000 osd.11 (osd.11) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:25 smithi169 bash[19604]: cluster 2024-02-12T16:18:24.280136+0000 osd.11 (osd.11) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:25 smithi169 bash[19604]: cluster 2024-02-12T16:18:24.849495+0000 mon.a (mon.0) 1590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:25 smithi169 bash[19604]: audit 2024-02-12T16:18:24.862161+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? v1:172.21.15.121:0/1860578835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:18:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:25 smithi169 bash[19604]: cluster 2024-02-12T16:18:24.862265+0000 mon.a (mon.0) 1592 : cluster [DBG] osdmap e432: 12 total, 11 up, 11 in 2024-02-12T16:18:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:25 smithi169 bash[19604]: cluster 2024-02-12T16:18:25.453520+0000 mon.a (mon.0) 1593 : cluster [DBG] osdmap e433: 12 total, 11 up, 11 in 2024-02-12T16:18:26.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:26.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:26.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:26.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:26.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:26.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:26.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:26.632 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:26.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:26.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:26.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:26.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:26.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:18:26.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:26.917 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:26.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:27.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:27.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:26 smithi169 bash[19604]: cluster 2024-02-12T16:18:25.768374+0000 mgr.y (mgr.14146) 994 : cluster [DBG] pgmap v1327: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:27.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:27.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:26 smithi123 bash[19119]: cluster 2024-02-12T16:18:25.768374+0000 mgr.y (mgr.14146) 994 : cluster [DBG] pgmap v1327: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:27.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:26 smithi121 bash[18489]: cluster 2024-02-12T16:18:25.768374+0000 mgr.y (mgr.14146) 994 : cluster [DBG] pgmap v1327: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:27.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:27.197 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:27.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:27.301 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:27.340 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:27.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:27.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:27.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:27.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:27.562 INFO:teuthology.orchestra.run.smithi121.stdout:get_pg_num_history pg_num_history pg_num_history(e414 pg_nums {1={14=1,365=11,410=21},4={334=16}} deleted_pools ) 2024-02-12T16:18:27.563 INFO:teuthology.orchestra.run.smithi121.stdout:Importing pgid 4.f 2024-02-12T16:18:27.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f003003b:::benchmark_data_smithi169_60938_object153827:head# 2024-02-12T16:18:27.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.572 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0044b9b:::benchmark_data_smithi169_60938_object155923:head# 2024-02-12T16:18:27.572 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.573 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0068bc1:::benchmark_data_smithi169_60938_object152279:head# 2024-02-12T16:18:27.573 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0068e0b:::benchmark_data_smithi169_60938_object151323:head# 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0083b71:::benchmark_data_smithi169_60938_object155111:head# 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f008be54:::benchmark_data_smithi169_60938_object147409:head# 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f009df6b:::benchmark_data_smithi169_60938_object147882:head# 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f00a816e:::benchmark_data_smithi169_60938_object152564:head# 2024-02-12T16:18:27.574 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f00ca4cf:::benchmark_data_smithi169_60938_object162655:head# 2024-02-12T16:18:27.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f010295a:::benchmark_data_smithi169_60938_object165661:head# 2024-02-12T16:18:27.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.576 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0129d45:::benchmark_data_smithi169_60938_object145851:head# 2024-02-12T16:18:27.576 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f018f139:::benchmark_data_smithi169_60938_object142957:head# 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f01a5829:::benchmark_data_smithi169_60938_object147023:head# 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f01d36ce:::benchmark_data_smithi169_60938_object140404:head# 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f01fa2e7:::benchmark_data_smithi169_60938_object143275:head# 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0201f7d:::benchmark_data_smithi169_60938_object158786:head# 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f021b06e:::benchmark_data_smithi169_60938_object168589:head# 2024-02-12T16:18:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.579 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f024bd70:::benchmark_data_smithi169_60938_object156768:head# 2024-02-12T16:18:27.579 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f024f4f9:::benchmark_data_smithi169_60938_object148207:head# 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f026d980:::benchmark_data_smithi169_60938_object154657:head# 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f02e6e58:::benchmark_data_smithi169_60938_object148079:head# 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f02ecca3:::benchmark_data_smithi169_60938_object165266:head# 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0301c50:::benchmark_data_smithi169_60938_object171622:head# 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f032f862:::benchmark_data_smithi169_60938_object163958:head# 2024-02-12T16:18:27.580 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0346170:::benchmark_data_smithi169_60938_object143545:head# 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0367c8d:::benchmark_data_smithi169_60938_object162104:head# 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f039c058:::benchmark_data_smithi169_60938_object154386:head# 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f039d353:::benchmark_data_smithi169_60938_object163634:head# 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f039e488:::benchmark_data_smithi169_60938_object144596:head# 2024-02-12T16:18:27.581 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.584 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f041378b:::benchmark_data_smithi169_60938_object163331:head# 2024-02-12T16:18:27.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0415d7d:::benchmark_data_smithi169_60938_object172830:head# 2024-02-12T16:18:27.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f041c8b5:::benchmark_data_smithi169_60938_object163464:head# 2024-02-12T16:18:27.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.587 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0443db3:::benchmark_data_smithi169_60938_object161483:head# 2024-02-12T16:18:27.587 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f044de42:::benchmark_data_smithi169_60938_object140312:head# 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f046acbb:::benchmark_data_smithi169_60938_object165288:head# 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f04d0aa7:::benchmark_data_smithi169_60938_object155364:head# 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f04f4541:::benchmark_data_smithi169_60938_object172233:head# 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f050406c:::benchmark_data_smithi169_60938_object138280:head# 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f05197f3:::benchmark_data_smithi169_60938_object168928:head# 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f05198a2:::benchmark_data_smithi169_60938_object141643:head# 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0527369:::benchmark_data_smithi169_60938_object163356:head# 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f053e107:::benchmark_data_smithi169_60938_object158346:head# 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f053f03f:::benchmark_data_smithi169_60938_object142666:head# 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0566f69:::benchmark_data_smithi169_60938_object157921:head# 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0575171:::benchmark_data_smithi169_60938_object160344:head# 2024-02-12T16:18:27.589 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.590 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f05a276e:::benchmark_data_smithi169_60938_object141577:head# 2024-02-12T16:18:27.590 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:27.599 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f05cb2f7:::benchmark_data_smithi169_60938_object143821:head# 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f05ce178:::benchmark_data_smithi169_60938_object158565:head# 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f05e372c:::benchmark_data_smithi169_60938_object159360:head# 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f05e6c57:::benchmark_data_smithi169_60938_object169966:head# 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0602705:::benchmark_data_smithi169_60938_object156283:head# 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f06178ae:::benchmark_data_smithi169_60938_object154870:head# 2024-02-12T16:18:27.600 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0618ec6:::benchmark_data_smithi169_60938_object154048:head# 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f062be84:::benchmark_data_smithi169_60938_object146679:head# 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f064c4ce:::benchmark_data_smithi169_60938_object165992:head# 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f068f934:::benchmark_data_smithi169_60938_object164932:head# 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f06b35c8:::benchmark_data_smithi169_60938_object164407:head# 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0762ef8:::benchmark_data_smithi169_60938_object163482:head# 2024-02-12T16:18:27.601 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f07899e2:::benchmark_data_smithi169_60938_object138350:head# 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f07ac115:::benchmark_data_smithi169_60938_object171883:head# 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f07c8c74:::benchmark_data_smithi169_60938_object154180:head# 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f07d54aa:::benchmark_data_smithi169_60938_object160232:head# 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f07e0c30:::benchmark_data_smithi169_60938_object159238:head# 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f07e49cc:::benchmark_data_smithi169_60938_object164042:head# 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.602 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f07e6d7e:::benchmark_data_smithi169_60938_object170388:head# 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f080c861:::benchmark_data_smithi169_60938_object171850:head# 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f08160fa:::benchmark_data_smithi169_60938_object167769:head# 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f083d7ed:::benchmark_data_smithi169_60938_object145607:head# 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f084cd89:::benchmark_data_smithi169_60938_object170229:head# 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f088b0a7:::benchmark_data_smithi169_60938_object172464:head# 2024-02-12T16:18:27.603 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f08b0d6b:::benchmark_data_smithi169_60938_object150853:head# 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0909054:::benchmark_data_smithi169_60938_object171324:head# 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0913974:::benchmark_data_smithi169_60938_object148111:head# 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f09777de:::benchmark_data_smithi169_60938_object158383:head# 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.604 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f098d2fa:::benchmark_data_smithi169_60938_object162607:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f09a43b1:::benchmark_data_smithi169_60938_object168114:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f09a89e5:::benchmark_data_smithi169_60938_object158204:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f09dc608:::benchmark_data_smithi169_60938_object148173:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f09e4cd6:::benchmark_data_smithi169_60938_object150794:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f09f686b:::benchmark_data_smithi169_60938_object141531:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0a13c66:::benchmark_data_smithi169_60938_object142625:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0a1aeaf:::benchmark_data_smithi169_60938_object172359:head# 2024-02-12T16:18:27.605 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0a3029c:::benchmark_data_smithi169_60938_object149518:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0a66b8b:::benchmark_data_smithi169_60938_object163727:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0a74dd6:::benchmark_data_smithi169_60938_object153460:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0a8c752:::benchmark_data_smithi169_60938_object156457:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0a9d431:::benchmark_data_smithi169_60938_object155142:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0ab57f4:::benchmark_data_smithi169_60938_object156516:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0ab86cb:::benchmark_data_smithi169_60938_object157074:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0aee27a:::benchmark_data_smithi169_60938_object169835:head# 2024-02-12T16:18:27.606 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0af8853:::benchmark_data_smithi169_60938_object160700:head# 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0b028e5:::benchmark_data_smithi169_60938_object147810:head# 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0b2a66d:::benchmark_data_smithi169_60938_object139209:head# 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0b2bf8b:::benchmark_data_smithi169_60938_object151348:head# 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0b95afd:::benchmark_data_smithi169_60938_object149022:head# 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0bb725c:::benchmark_data_smithi169_60938_object171415:head# 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0bba991:::benchmark_data_smithi169_60938_object138361:head# 2024-02-12T16:18:27.607 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0bc445d:::benchmark_data_smithi169_60938_object139070:head# 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0bf2445:::benchmark_data_smithi169_60938_object147670:head# 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0c25909:::benchmark_data_smithi169_60938_object141178:head# 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0c54279:::benchmark_data_smithi169_60938_object168458:head# 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0c72fc8:::benchmark_data_smithi169_60938_object153134:head# 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0ca7bdf:::benchmark_data_smithi169_60938_object172721:head# 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0ce91bb:::benchmark_data_smithi169_60938_object145350:head# 2024-02-12T16:18:27.608 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0cedbee:::benchmark_data_smithi169_60938_object139499:head# 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0d09071:::benchmark_data_smithi169_60938_object140579:head# 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0d0c24a:::benchmark_data_smithi169_60938_object167494:head# 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0d3ff5d:::benchmark_data_smithi169_60938_object165626:head# 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0d6c0fd:::benchmark_data_smithi169_60938_object152917:head# 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0d83a0b:::benchmark_data_smithi169_60938_object169238:head# 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.618 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0d90f1c:::benchmark_data_smithi169_60938_object140992:head# 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0da4e9e:::benchmark_data_smithi169_60938_object139001:head# 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0db052a:::benchmark_data_smithi169_60938_object156744:head# 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0dbaeef:::benchmark_data_smithi169_60938_object143692:head# 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0de002a:::benchmark_data_smithi169_60938_object158909:head# 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0de05ab:::benchmark_data_smithi169_60938_object156045:head# 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0e074ed:::benchmark_data_smithi169_60938_object146723:head# 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.619 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0e0e1b9:::benchmark_data_smithi169_60938_object142031:head# 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0e29d34:::benchmark_data_smithi169_60938_object147943:head# 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0eb8e06:::benchmark_data_smithi169_60938_object142711:head# 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0ebbe36:::benchmark_data_smithi169_60938_object172933:head# 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0ed221b:::benchmark_data_smithi169_60938_object171421:head# 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0f361d6:::benchmark_data_smithi169_60938_object173129:head# 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0f4f253:::benchmark_data_smithi169_60938_object140735:head# 2024-02-12T16:18:27.620 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0f6a32c:::benchmark_data_smithi169_60938_object157411:head# 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0fc5741:::benchmark_data_smithi169_60938_object145879:head# 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0fe8aff:::benchmark_data_smithi169_60938_object161123:head# 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f0ff02f2:::benchmark_data_smithi169_60938_object160754:head# 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f10444c1:::benchmark_data_smithi169_60938_object171082:head# 2024-02-12T16:18:27.621 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f104e67c:::benchmark_data_smithi169_60938_object153195:head# 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f10524eb:::benchmark_data_smithi169_60938_object164515:head# 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f108112f:::benchmark_data_smithi169_60938_object171389:head# 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f108788b:::benchmark_data_smithi169_60938_object171620:head# 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f109587b:::benchmark_data_smithi169_60938_object145145:head# 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f10977d5:::benchmark_data_smithi169_60938_object149301:head# 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f10a44f2:::benchmark_data_smithi169_60938_object143085:head# 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.622 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f10ef2dd:::benchmark_data_smithi169_60938_object139151:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f10fb6f9:::benchmark_data_smithi169_60938_object148251:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f111d2d1:::benchmark_data_smithi169_60938_object143290:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1244cbf:::benchmark_data_smithi169_60938_object165191:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12524ba:::benchmark_data_smithi169_60938_object165607:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1265b60:::benchmark_data_smithi169_60938_object150584:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12667d5:::benchmark_data_smithi169_60938_object143485:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f127ab1d:::benchmark_data_smithi169_60938_object163498:head# 2024-02-12T16:18:27.623 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12922f2:::benchmark_data_smithi169_60938_object139932:head# 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12b5377:::benchmark_data_smithi169_60938_object146871:head# 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12b9d4e:::benchmark_data_smithi169_60938_object150680:head# 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12be857:::benchmark_data_smithi169_60938_object159951:head# 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12c1b8e:::benchmark_data_smithi169_60938_object145999:head# 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12c1e00:::benchmark_data_smithi169_60938_object147604:head# 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12c7822:::benchmark_data_smithi169_60938_object160727:head# 2024-02-12T16:18:27.624 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12cfc51:::benchmark_data_smithi169_60938_object145327:head# 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12d0982:::benchmark_data_smithi169_60938_object140400:head# 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12e3010:::benchmark_data_smithi169_60938_object144615:head# 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f12e8132:::benchmark_data_smithi169_60938_object145225:head# 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1333a5a:::benchmark_data_smithi169_60938_object140035:head# 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f133c38d:::benchmark_data_smithi169_60938_object160531:head# 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f13438e5:::benchmark_data_smithi169_60938_object149402:head# 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.625 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f138a806:::benchmark_data_smithi169_60938_object167796:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f139fbc5:::benchmark_data_smithi169_60938_object165640:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f13b9199:::benchmark_data_smithi169_60938_object156635:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f13d48d9:::benchmark_data_smithi169_60938_object163472:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f13e6adb:::benchmark_data_smithi169_60938_object159658:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f13f295d:::benchmark_data_smithi169_60938_object152228:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f13fe004:::benchmark_data_smithi169_60938_object160786:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f140ef75:::benchmark_data_smithi169_60938_object168448:head# 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.626 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f143a9b5:::benchmark_data_smithi169_60938_object150821:head# 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f145b713:::benchmark_data_smithi169_60938_object155172:head# 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1477b49:::benchmark_data_smithi169_60938_object149219:head# 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1479725:::benchmark_data_smithi169_60938_object139494:head# 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f14c3325:::benchmark_data_smithi169_60938_object166966:head# 2024-02-12T16:18:27.627 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:27.641 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f150f81a:::benchmark_data_smithi169_60938_object168167:head# 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1510436:::benchmark_data_smithi169_60938_object155676:head# 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1535578:::benchmark_data_smithi169_60938_object140345:head# 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1539eb1:::benchmark_data_smithi169_60938_object148598:head# 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f157566f:::benchmark_data_smithi169_60938_object171425:head# 2024-02-12T16:18:27.642 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15874ac:::benchmark_data_smithi169_60938_object162391:head# 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15a3db8:::benchmark_data_smithi169_60938_object163355:head# 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15aa612:::benchmark_data_smithi169_60938_object140391:head# 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15b488a:::benchmark_data_smithi169_60938_object155732:head# 2024-02-12T16:18:27.643 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15b6c50:::benchmark_data_smithi169_60938_object143281:head# 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15c1f2e:::benchmark_data_smithi169_60938_object166781:head# 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15d002f:::benchmark_data_smithi169_60938_object149037:head# 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15dcb90:::benchmark_data_smithi169_60938_object155410:head# 2024-02-12T16:18:27.644 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f15dd091:::benchmark_data_smithi169_60938_object172174:head# 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1612ca8:::benchmark_data_smithi169_60938_object159943:head# 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f164c96d:::benchmark_data_smithi169_60938_object138584:head# 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f165159f:::benchmark_data_smithi169_60938_object152162:head# 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.645 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1676f47:::benchmark_data_smithi169_60938_object162988:head# 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1693485:::benchmark_data_smithi169_60938_object150180:head# 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f169f74d:::benchmark_data_smithi169_60938_object157365:head# 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f16ac276:::benchmark_data_smithi169_60938_object150994:head# 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f16c66a9:::benchmark_data_smithi169_60938_object146337:head# 2024-02-12T16:18:27.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f16f3089:::benchmark_data_smithi169_60938_object172305:head# 2024-02-12T16:18:27.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f17332fc:::benchmark_data_smithi169_60938_object160885:head# 2024-02-12T16:18:27.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.647 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1747a0f:::benchmark_data_smithi169_60938_object155728:head# 2024-02-12T16:18:27.647 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f17ae632:::benchmark_data_smithi169_60938_object170878:head# 2024-02-12T16:18:27.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f17e3448:::benchmark_data_smithi169_60938_object151511:head# 2024-02-12T16:18:27.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f17e36b8:::benchmark_data_smithi169_60938_object163899:head# 2024-02-12T16:18:27.648 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.648 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f18241ab:::benchmark_data_smithi169_60938_object144353:head# 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f182ce2f:::benchmark_data_smithi169_60938_object167488:head# 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f182f68c:::benchmark_data_smithi169_60938_object164459:head# 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f18546b5:::benchmark_data_smithi169_60938_object150341:head# 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f187f0e1:::benchmark_data_smithi169_60938_object154860:head# 2024-02-12T16:18:27.649 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1898af1:::benchmark_data_smithi169_60938_object171615:head# 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f189d3cb:::benchmark_data_smithi169_60938_object147480:head# 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1905efc:::benchmark_data_smithi169_60938_object162321:head# 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1946394:::benchmark_data_smithi169_60938_object154891:head# 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1952806:::benchmark_data_smithi169_60938_object154334:head# 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f196cae3:::benchmark_data_smithi169_60938_object147906:head# 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f19bde44:::benchmark_data_smithi169_60938_object158046:head# 2024-02-12T16:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f19be320:::benchmark_data_smithi169_60938_object146005:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f19cc637:::benchmark_data_smithi169_60938_object158747:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f19e79bd:::benchmark_data_smithi169_60938_object156845:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f19fedc2:::benchmark_data_smithi169_60938_object142747:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a20d5e:::benchmark_data_smithi169_60938_object147074:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a28f39:::benchmark_data_smithi169_60938_object153974:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a4d770:::benchmark_data_smithi169_60938_object171450:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a5577b:::benchmark_data_smithi169_60938_object153673:head# 2024-02-12T16:18:27.651 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a8bed8:::benchmark_data_smithi169_60938_object166929:head# 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a8c406:::benchmark_data_smithi169_60938_object147720:head# 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a8dab3:::benchmark_data_smithi169_60938_object165491:head# 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a8dc99:::benchmark_data_smithi169_60938_object151159:head# 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1a94a1c:::benchmark_data_smithi169_60938_object152762:head# 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ab45b1:::benchmark_data_smithi169_60938_object145805:head# 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1aba13f:::benchmark_data_smithi169_60938_object142915:head# 2024-02-12T16:18:27.652 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ac4933:::benchmark_data_smithi169_60938_object152410:head# 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1aca719:::benchmark_data_smithi169_60938_object153155:head# 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ad5f79:::benchmark_data_smithi169_60938_object146781:head# 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ae17ea:::benchmark_data_smithi169_60938_object169789:head# 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ae371a:::benchmark_data_smithi169_60938_object163328:head# 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1afd0be:::benchmark_data_smithi169_60938_object170945:head# 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1b1b708:::benchmark_data_smithi169_60938_object165209:head# 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1b1b93d:::benchmark_data_smithi169_60938_object147607:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1b88e0b:::benchmark_data_smithi169_60938_object153112:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1b97206:::benchmark_data_smithi169_60938_object156475:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1bcc8c8:::benchmark_data_smithi169_60938_object159770:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1bcff11:::benchmark_data_smithi169_60938_object171360:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1bd78da:::benchmark_data_smithi169_60938_object156171:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1bde816:::benchmark_data_smithi169_60938_object146833:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1bf2437:::benchmark_data_smithi169_60938_object155681:head# 2024-02-12T16:18:27.654 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c00d31:::benchmark_data_smithi169_60938_object147725:head# 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c2a2f1:::benchmark_data_smithi169_60938_object146945:head# 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c60956:::benchmark_data_smithi169_60938_object165537:head# 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c7ecac:::benchmark_data_smithi169_60938_object155264:head# 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c815c1:::benchmark_data_smithi169_60938_object156627:head# 2024-02-12T16:18:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c826b6:::benchmark_data_smithi169_60938_object162051:head# 2024-02-12T16:18:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c855a0:::benchmark_data_smithi169_60938_object141604:head# 2024-02-12T16:18:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c8d3fa:::benchmark_data_smithi169_60938_object154263:head# 2024-02-12T16:18:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1c94512:::benchmark_data_smithi169_60938_object156494:head# 2024-02-12T16:18:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1cb2068:::benchmark_data_smithi169_60938_object154527:head# 2024-02-12T16:18:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1d052ae:::benchmark_data_smithi169_60938_object143825:head# 2024-02-12T16:18:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1d3bd66:::benchmark_data_smithi169_60938_object162823:head# 2024-02-12T16:18:27.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1d57cee:::benchmark_data_smithi169_60938_object160772:head# 2024-02-12T16:18:27.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1d58c26:::benchmark_data_smithi169_60938_object138675:head# 2024-02-12T16:18:27.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.658 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1d6af79:::benchmark_data_smithi169_60938_object149532:head# 2024-02-12T16:18:27.658 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1d93fa3:::benchmark_data_smithi169_60938_object169219:head# 2024-02-12T16:18:27.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ddc9a7:::benchmark_data_smithi169_60938_object161948:head# 2024-02-12T16:18:27.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.659 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ddee7e:::benchmark_data_smithi169_60938_object150405:head# 2024-02-12T16:18:27.659 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.660 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1dffa58:::benchmark_data_smithi169_60938_object171120:head# 2024-02-12T16:18:27.660 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:27.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:27.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1e4e9f4:::benchmark_data_smithi169_60938_object141470:head# 2024-02-12T16:18:27.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1e55d06:::benchmark_data_smithi169_60938_object156798:head# 2024-02-12T16:18:27.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1e5c085:::benchmark_data_smithi169_60938_object157835:head# 2024-02-12T16:18:27.697 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.697 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1e6c281:::benchmark_data_smithi169_60938_object142530:head# 2024-02-12T16:18:27.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1e79e8f:::benchmark_data_smithi169_60938_object142941:head# 2024-02-12T16:18:27.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1e7ee4b:::benchmark_data_smithi169_60938_object153670:head# 2024-02-12T16:18:27.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.698 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1eae26b:::benchmark_data_smithi169_60938_object150705:head# 2024-02-12T16:18:27.698 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1eb36b4:::benchmark_data_smithi169_60938_object162102:head# 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ee19ca:::benchmark_data_smithi169_60938_object139959:head# 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ee254e:::benchmark_data_smithi169_60938_object163610:head# 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ee3039:::benchmark_data_smithi169_60938_object168243:head# 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1ee8ef8:::benchmark_data_smithi169_60938_object153220:head# 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1f1fb9e:::benchmark_data_smithi169_60938_object155970:head# 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1f5c080:::benchmark_data_smithi169_60938_object141029:head# 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.699 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1f783a7:::benchmark_data_smithi169_60938_object163861:head# 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1fb4e24:::benchmark_data_smithi169_60938_object157468:head# 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f1fdcfbe:::benchmark_data_smithi169_60938_object157267:head# 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2033ee2:::benchmark_data_smithi169_60938_object160221:head# 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f20580f9:::benchmark_data_smithi169_60938_object144826:head# 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f20894db:::benchmark_data_smithi169_60938_object169530:head# 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f20d056e:::benchmark_data_smithi169_60938_object146259:head# 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.700 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f210565d:::benchmark_data_smithi169_60938_object147595:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2112e86:::benchmark_data_smithi169_60938_object163010:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f211a858:::benchmark_data_smithi169_60938_object144523:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f21985a3:::benchmark_data_smithi169_60938_object161245:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f21da37e:::benchmark_data_smithi169_60938_object153884:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f22145ac:::benchmark_data_smithi169_60938_object141158:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f222860f:::benchmark_data_smithi169_60938_object172222:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f223677d:::benchmark_data_smithi169_60938_object148116:head# 2024-02-12T16:18:27.701 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2237f2e:::benchmark_data_smithi169_60938_object151243:head# 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f223ce4e:::benchmark_data_smithi169_60938_object165631:head# 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2289156:::benchmark_data_smithi169_60938_object159812:head# 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f22b76d8:::benchmark_data_smithi169_60938_object164377:head# 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f22b9d45:::benchmark_data_smithi169_60938_object141016:head# 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f22c532a:::benchmark_data_smithi169_60938_object139500:head# 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2357fdf:::benchmark_data_smithi169_60938_object172746:head# 2024-02-12T16:18:27.702 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2380690:::benchmark_data_smithi169_60938_object159038:head# 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f238c546:::benchmark_data_smithi169_60938_object163934:head# 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f239b718:::benchmark_data_smithi169_60938_object170869:head# 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f23b09b7:::benchmark_data_smithi169_60938_object139119:head# 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f23b5962:::benchmark_data_smithi169_60938_object168808:head# 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f23ce405:::benchmark_data_smithi169_60938_object151394:head# 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f23e5347:::benchmark_data_smithi169_60938_object166609:head# 2024-02-12T16:18:27.703 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f23e56f3:::benchmark_data_smithi169_60938_object146063:head# 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2402ad1:::benchmark_data_smithi169_60938_object165099:head# 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f246a7b9:::benchmark_data_smithi169_60938_object167214:head# 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f247184e:::benchmark_data_smithi169_60938_object168756:head# 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f24a14d7:::benchmark_data_smithi169_60938_object142034:head# 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f24c6b92:::benchmark_data_smithi169_60938_object169875:head# 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f24da4e9:::benchmark_data_smithi169_60938_object145181:head# 2024-02-12T16:18:27.704 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f24e7c5f:::benchmark_data_smithi169_60938_object167500:head# 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2548089:::benchmark_data_smithi169_60938_object156613:head# 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2555bb5:::benchmark_data_smithi169_60938_object173265:head# 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2563c87:::benchmark_data_smithi169_60938_object148768:head# 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2571e11:::benchmark_data_smithi169_60938_object146217:head# 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f257d1bd:::benchmark_data_smithi169_60938_object142410:head# 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f257e96d:::benchmark_data_smithi169_60938_object144381:head# 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.705 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f259a919:::benchmark_data_smithi169_60938_object140925:head# 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f25ae07b:::benchmark_data_smithi169_60938_object151834:head# 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f25c20b1:::benchmark_data_smithi169_60938_object172130:head# 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f25f9584:::benchmark_data_smithi169_60938_object159884:head# 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f25fe59d:::benchmark_data_smithi169_60938_object162261:head# 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f264fc85:::benchmark_data_smithi169_60938_object170978:head# 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2672730:::benchmark_data_smithi169_60938_object141295:head# 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.706 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2689f86:::benchmark_data_smithi169_60938_object169131:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f26943d2:::benchmark_data_smithi169_60938_object165933:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f26fb372:::benchmark_data_smithi169_60938_object167366:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f278ae8e:::benchmark_data_smithi169_60938_object144369:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f27cf9f4:::benchmark_data_smithi169_60938_object164370:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f27f2d61:::benchmark_data_smithi169_60938_object169856:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f27ff23b:::benchmark_data_smithi169_60938_object158131:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2803edd:::benchmark_data_smithi169_60938_object152160:head# 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.707 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2810357:::benchmark_data_smithi169_60938_object161156:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f286da53:::benchmark_data_smithi169_60938_object152411:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f287caa8:::benchmark_data_smithi169_60938_object141200:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28861b1:::benchmark_data_smithi169_60938_object167478:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28895a2:::benchmark_data_smithi169_60938_object141307:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f288d721:::benchmark_data_smithi169_60938_object140540:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28a70ba:::benchmark_data_smithi169_60938_object141566:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28cdf40:::benchmark_data_smithi169_60938_object156187:head# 2024-02-12T16:18:27.708 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28deca8:::benchmark_data_smithi169_60938_object150460:head# 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28ed3ff:::benchmark_data_smithi169_60938_object144959:head# 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28fab29:::benchmark_data_smithi169_60938_object171048:head# 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f28fb105:::benchmark_data_smithi169_60938_object152082:head# 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f291b42a:::benchmark_data_smithi169_60938_object155608:head# 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f29277ca:::benchmark_data_smithi169_60938_object169568:head# 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f292a1ff:::benchmark_data_smithi169_60938_object166705:head# 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f294368d:::benchmark_data_smithi169_60938_object142965:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2964850:::benchmark_data_smithi169_60938_object172273:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2973656:::benchmark_data_smithi169_60938_object142614:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2977286:::benchmark_data_smithi169_60938_object143612:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f29816b0:::benchmark_data_smithi169_60938_object154452:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f29b66be:::benchmark_data_smithi169_60938_object164488:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f29bbc95:::benchmark_data_smithi169_60938_object139774:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f29c6d49:::benchmark_data_smithi169_60938_object146030:head# 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.710 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f29cb813:::benchmark_data_smithi169_60938_object166023:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f29cc230:::benchmark_data_smithi169_60938_object154274:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2a0f12c:::benchmark_data_smithi169_60938_object142487:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2a43112:::benchmark_data_smithi169_60938_object144780:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2a576fb:::benchmark_data_smithi169_60938_object173175:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2a5a7dc:::benchmark_data_smithi169_60938_object152034:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2a6c631:::benchmark_data_smithi169_60938_object158507:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2a70595:::benchmark_data_smithi169_60938_object171802:head# 2024-02-12T16:18:27.711 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2a84242:::benchmark_data_smithi169_60938_object152379:head# 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2ab315b:::benchmark_data_smithi169_60938_object151273:head# 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2acc79a:::benchmark_data_smithi169_60938_object144668:head# 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2b2d8b5:::benchmark_data_smithi169_60938_object150979:head# 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2b38274:::benchmark_data_smithi169_60938_object167383:head# 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2b4687a:::benchmark_data_smithi169_60938_object142954:head# 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2b6c3c9:::benchmark_data_smithi169_60938_object157040:head# 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.712 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2b7d9b9:::benchmark_data_smithi169_60938_object145061:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2b93958:::benchmark_data_smithi169_60938_object168484:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2b9ab52:::benchmark_data_smithi169_60938_object141990:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2bd32e9:::benchmark_data_smithi169_60938_object144246:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2c0dd91:::benchmark_data_smithi169_60938_object151380:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2c1e78c:::benchmark_data_smithi169_60938_object160779:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2c3b378:::benchmark_data_smithi169_60938_object154803:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2c4059f:::benchmark_data_smithi169_60938_object168527:head# 2024-02-12T16:18:27.713 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2c557e3:::benchmark_data_smithi169_60938_object138572:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2c62d21:::benchmark_data_smithi169_60938_object143838:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2cb318c:::benchmark_data_smithi169_60938_object139185:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2cc8902:::benchmark_data_smithi169_60938_object160747:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d1fe54:::benchmark_data_smithi169_60938_object149160:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d34d16:::benchmark_data_smithi169_60938_object148970:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d3799b:::benchmark_data_smithi169_60938_object166880:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d3fb57:::benchmark_data_smithi169_60938_object149141:head# 2024-02-12T16:18:27.714 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d7542b:::benchmark_data_smithi169_60938_object164730:head# 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d7600e:::benchmark_data_smithi169_60938_object159750:head# 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d7833f:::benchmark_data_smithi169_60938_object152722:head# 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d79580:::benchmark_data_smithi169_60938_object168990:head# 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2d7dae7:::benchmark_data_smithi169_60938_object162534:head# 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2dadd7a:::benchmark_data_smithi169_60938_object157788:head# 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2db355b:::benchmark_data_smithi169_60938_object165852:head# 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.715 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2dc4c5b:::benchmark_data_smithi169_60938_object163553:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2de4980:::benchmark_data_smithi169_60938_object151141:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2df3190:::benchmark_data_smithi169_60938_object163342:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2dff076:::benchmark_data_smithi169_60938_object149240:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e09675:::benchmark_data_smithi169_60938_object168557:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e25c43:::benchmark_data_smithi169_60938_object166262:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e27f70:::benchmark_data_smithi169_60938_object162525:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e2937a:::benchmark_data_smithi169_60938_object160636:head# 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.716 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e2c8aa:::benchmark_data_smithi169_60938_object166135:head# 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e48b4d:::benchmark_data_smithi169_60938_object142046:head# 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e498d2:::benchmark_data_smithi169_60938_object144633:head# 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e688e7:::benchmark_data_smithi169_60938_object161323:head# 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e884db:::benchmark_data_smithi169_60938_object161412:head# 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e9655b:::benchmark_data_smithi169_60938_object166975:head# 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2e9c263:::benchmark_data_smithi169_60938_object144697:head# 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.717 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2ed0e09:::benchmark_data_smithi169_60938_object160207:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2ed15be:::benchmark_data_smithi169_60938_object149277:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2edef31:::benchmark_data_smithi169_60938_object166473:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2eed258:::benchmark_data_smithi169_60938_object144023:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2f36a79:::benchmark_data_smithi169_60938_object152520:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2f4ac33:::benchmark_data_smithi169_60938_object150583:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2f50199:::benchmark_data_smithi169_60938_object164817:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2fbb8ba:::benchmark_data_smithi169_60938_object171109:head# 2024-02-12T16:18:27.718 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2fd51bc:::benchmark_data_smithi169_60938_object168865:head# 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f2fd8062:::benchmark_data_smithi169_60938_object147174:head# 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3018822:::benchmark_data_smithi169_60938_object168888:head# 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3024779:::benchmark_data_smithi169_60938_object154620:head# 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f305559e:::benchmark_data_smithi169_60938_object141471:head# 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f305e90c:::benchmark_data_smithi169_60938_object149832:head# 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f30604ec:::benchmark_data_smithi169_60938_object172249:head# 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.719 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f30803bb:::benchmark_data_smithi169_60938_object147254:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f30a3bbd:::benchmark_data_smithi169_60938_object156295:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f30bc51a:::benchmark_data_smithi169_60938_object150570:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f30c5afb:::benchmark_data_smithi169_60938_object155351:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f30d7932:::benchmark_data_smithi169_60938_object154317:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f30f4abc:::benchmark_data_smithi169_60938_object150533:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3142bd1:::benchmark_data_smithi169_60938_object147333:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3152974:::benchmark_data_smithi169_60938_object138976:head# 2024-02-12T16:18:27.720 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f31bafdf:::benchmark_data_smithi169_60938_object141058:head# 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f31bb724:::benchmark_data_smithi169_60938_object140630:head# 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3248efd:::benchmark_data_smithi169_60938_object138699:head# 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32574d5:::benchmark_data_smithi169_60938_object157229:head# 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32579c5:::benchmark_data_smithi169_60938_object164132:head# 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3280ab9:::benchmark_data_smithi169_60938_object167973:head# 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32a2c44:::benchmark_data_smithi169_60938_object151500:head# 2024-02-12T16:18:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32b9eaf:::benchmark_data_smithi169_60938_object156821:head# 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32c739c:::benchmark_data_smithi169_60938_object167623:head# 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32d03e0:::benchmark_data_smithi169_60938_object159270:head# 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32d09d3:::benchmark_data_smithi169_60938_object161787:head# 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f32ed07f:::benchmark_data_smithi169_60938_object140141:head# 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3356df0:::benchmark_data_smithi169_60938_object146824:head# 2024-02-12T16:18:27.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3371382:::benchmark_data_smithi169_60938_object165665:head# 2024-02-12T16:18:27.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f337a896:::benchmark_data_smithi169_60938_object172655:head# 2024-02-12T16:18:27.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.723 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3384ef1:::benchmark_data_smithi169_60938_object173006:head# 2024-02-12T16:18:27.723 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3386d16:::benchmark_data_smithi169_60938_object159124:head# 2024-02-12T16:18:27.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3390f7f:::benchmark_data_smithi169_60938_object168763:head# 2024-02-12T16:18:27.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.724 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f33b361a:::benchmark_data_smithi169_60938_object149129:head# 2024-02-12T16:18:27.724 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f33ceac8:::benchmark_data_smithi169_60938_object159280:head# 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f34294b1:::benchmark_data_smithi169_60938_object152681:head# 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f346e988:::benchmark_data_smithi169_60938_object147120:head# 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3472cc2:::benchmark_data_smithi169_60938_object144904:head# 2024-02-12T16:18:27.725 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3479ccc:::benchmark_data_smithi169_60938_object141944:head# 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f347aeb0:::benchmark_data_smithi169_60938_object170569:head# 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3481484:::benchmark_data_smithi169_60938_object146325:head# 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f34bffb7:::benchmark_data_smithi169_60938_object138627:head# 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.726 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f35174f9:::benchmark_data_smithi169_60938_object164091:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f351ffba:::benchmark_data_smithi169_60938_object150898:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f35251aa:::benchmark_data_smithi169_60938_object140102:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f352ab27:::benchmark_data_smithi169_60938_object164233:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f35329e2:::benchmark_data_smithi169_60938_object151613:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3583a0b:::benchmark_data_smithi169_60938_object139794:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f359cf14:::benchmark_data_smithi169_60938_object146715:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f35a013a:::benchmark_data_smithi169_60938_object149868:head# 2024-02-12T16:18:27.727 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.757 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f35bd8ee:::benchmark_data_smithi169_60938_object159836:head# 2024-02-12T16:18:27.757 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f35f20e8:::benchmark_data_smithi169_60938_object171333:head# 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3666f59:::benchmark_data_smithi169_60938_object147944:head# 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f366865b:::benchmark_data_smithi169_60938_object139479:head# 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f367dfe3:::benchmark_data_smithi169_60938_object170009:head# 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3694fe0:::benchmark_data_smithi169_60938_object146796:head# 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3697299:::benchmark_data_smithi169_60938_object164293:head# 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f36a900e:::benchmark_data_smithi169_60938_object172672:head# 2024-02-12T16:18:27.758 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f36c4bc2:::benchmark_data_smithi169_60938_object163842:head# 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f37350d9:::benchmark_data_smithi169_60938_object148029:head# 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3747660:::benchmark_data_smithi169_60938_object158571:head# 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3748581:::benchmark_data_smithi169_60938_object153815:head# 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f374e625:::benchmark_data_smithi169_60938_object142488:head# 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3793802:::benchmark_data_smithi169_60938_object162050:head# 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f37a0e9b:::benchmark_data_smithi169_60938_object167540:head# 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.759 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f37c3a14:::benchmark_data_smithi169_60938_object160060:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3829845:::benchmark_data_smithi169_60938_object150394:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3837373:::benchmark_data_smithi169_60938_object143974:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f38534e9:::benchmark_data_smithi169_60938_object170646:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3854b2c:::benchmark_data_smithi169_60938_object146231:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3876d6b:::benchmark_data_smithi169_60938_object150553:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f387b801:::benchmark_data_smithi169_60938_object158575:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3882510:::benchmark_data_smithi169_60938_object169085:head# 2024-02-12T16:18:27.760 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f38996d6:::benchmark_data_smithi169_60938_object163173:head# 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f38f578e:::benchmark_data_smithi169_60938_object161309:head# 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f38fb032:::benchmark_data_smithi169_60938_object146489:head# 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3912270:::benchmark_data_smithi169_60938_object145803:head# 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3920ce4:::benchmark_data_smithi169_60938_object143108:head# 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f394da28:::benchmark_data_smithi169_60938_object151748:head# 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f397f1f6:::benchmark_data_smithi169_60938_object159897:head# 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.761 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f398eca0:::benchmark_data_smithi169_60938_object172053:head# 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3996862:::benchmark_data_smithi169_60938_object147372:head# 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f39ab24e:::benchmark_data_smithi169_60938_object164183:head# 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f39c9db2:::benchmark_data_smithi169_60938_object158776:head# 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f39df2ae:::benchmark_data_smithi169_60938_object166056:head# 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3a29290:::benchmark_data_smithi169_60938_object150632:head# 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3a5e222:::benchmark_data_smithi169_60938_object162994:head# 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3a5f6bb:::benchmark_data_smithi169_60938_object162953:head# 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3a6ba14:::benchmark_data_smithi169_60938_object158851:head# 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3adeac6:::benchmark_data_smithi169_60938_object169342:head# 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3af78ce:::benchmark_data_smithi169_60938_object164256:head# 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3b25cc5:::benchmark_data_smithi169_60938_object154738:head# 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3b53710:::benchmark_data_smithi169_60938_object159422:head# 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.763 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3b54187:::benchmark_data_smithi169_60938_object166146:head# 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3b55793:::benchmark_data_smithi169_60938_object138818:head# 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3b7227e:::benchmark_data_smithi169_60938_object160483:head# 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3b7985a:::benchmark_data_smithi169_60938_object146481:head# 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3bc4758:::benchmark_data_smithi169_60938_object166060:head# 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3c0ed1f:::benchmark_data_smithi169_60938_object170837:head# 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3c0fceb:::benchmark_data_smithi169_60938_object161302:head# 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.764 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3c2ac2f:::benchmark_data_smithi169_60938_object150565:head# 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3c3750d:::benchmark_data_smithi169_60938_object159113:head# 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3c47fad:::benchmark_data_smithi169_60938_object140740:head# 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3cf8b83:::benchmark_data_smithi169_60938_object163995:head# 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3d00316:::benchmark_data_smithi169_60938_object144050:head# 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3d094be:::benchmark_data_smithi169_60938_object159276:head# 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.765 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3d4742a:::benchmark_data_smithi169_60938_object146682:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3d73804:::benchmark_data_smithi169_60938_object148865:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3d9a01e:::benchmark_data_smithi169_60938_object150547:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3daaa6c:::benchmark_data_smithi169_60938_object170275:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3dd46f8:::benchmark_data_smithi169_60938_object158303:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3e011a9:::benchmark_data_smithi169_60938_object160155:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3e1d9c0:::benchmark_data_smithi169_60938_object172473:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3e609f9:::benchmark_data_smithi169_60938_object155960:head# 2024-02-12T16:18:27.766 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3e91e5a:::benchmark_data_smithi169_60938_object149389:head# 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3eace7d:::benchmark_data_smithi169_60938_object164041:head# 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3eb0a9a:::benchmark_data_smithi169_60938_object139668:head# 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3eb6c1d:::benchmark_data_smithi169_60938_object161471:head# 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3efde77:::benchmark_data_smithi169_60938_object164100:head# 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3f2ca3c:::benchmark_data_smithi169_60938_object146376:head# 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3f798e2:::benchmark_data_smithi169_60938_object160787:head# 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.767 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3f89c2d:::benchmark_data_smithi169_60938_object145288:head# 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3fa329c:::benchmark_data_smithi169_60938_object140281:head# 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3faeaaa:::benchmark_data_smithi169_60938_object165672:head# 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3fdcfff:::benchmark_data_smithi169_60938_object157029:head# 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f3ffb30b:::benchmark_data_smithi169_60938_object159605:head# 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f400c90c:::benchmark_data_smithi169_60938_object153034:head# 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4020561:::benchmark_data_smithi169_60938_object164113:head# 2024-02-12T16:18:27.768 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4030595:::benchmark_data_smithi169_60938_object161252:head# 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f40354bc:::benchmark_data_smithi169_60938_object151281:head# 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4075981:::benchmark_data_smithi169_60938_object161386:head# 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f40aa79a:::benchmark_data_smithi169_60938_object160361:head# 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f40ce395:::benchmark_data_smithi169_60938_object138957:head# 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f40e08e3:::benchmark_data_smithi169_60938_object142633:head# 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f40e8375:::benchmark_data_smithi169_60938_object142657:head# 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.769 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f41185e1:::benchmark_data_smithi169_60938_object166931:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f413dedd:::benchmark_data_smithi169_60938_object173238:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4143b41:::benchmark_data_smithi169_60938_object146374:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f417b486:::benchmark_data_smithi169_60938_object159569:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f41af79d:::benchmark_data_smithi169_60938_object153823:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f41b2467:::benchmark_data_smithi169_60938_object139714:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f41cedac:::benchmark_data_smithi169_60938_object159392:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4206bbd:::benchmark_data_smithi169_60938_object161371:head# 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.770 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4220f51:::benchmark_data_smithi169_60938_object170578:head# 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4256df4:::benchmark_data_smithi169_60938_object154313:head# 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f42759ee:::benchmark_data_smithi169_60938_object142043:head# 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4281318:::benchmark_data_smithi169_60938_object170487:head# 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4282369:::benchmark_data_smithi169_60938_object164684:head# 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f42831ee:::benchmark_data_smithi169_60938_object150710:head# 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.771 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f429c1c4:::benchmark_data_smithi169_60938_object163513:head# 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f429ece8:::benchmark_data_smithi169_60938_object154827:head# 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f42e1ac8:::benchmark_data_smithi169_60938_object170244:head# 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f430ebc7:::benchmark_data_smithi169_60938_object146565:head# 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4366cca:::benchmark_data_smithi169_60938_object140539:head# 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4369742:::benchmark_data_smithi169_60938_object171969:head# 2024-02-12T16:18:27.772 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f437813e:::benchmark_data_smithi169_60938_object164299:head# 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4381f3a:::benchmark_data_smithi169_60938_object151973:head# 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f43867b2:::benchmark_data_smithi169_60938_object152168:head# 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f43a9730:::benchmark_data_smithi169_60938_object172395:head# 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f43d3557:::benchmark_data_smithi169_60938_object142272:head# 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f43d5029:::benchmark_data_smithi169_60938_object165887:head# 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f43e625a:::benchmark_data_smithi169_60938_object139238:head# 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.773 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f43ea917:::benchmark_data_smithi169_60938_object144999:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f443ac87:::benchmark_data_smithi169_60938_object153919:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4472e35:::benchmark_data_smithi169_60938_object140765:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4485acf:::benchmark_data_smithi169_60938_object141045:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f44b5491:::benchmark_data_smithi169_60938_object143301:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f44e613d:::benchmark_data_smithi169_60938_object162303:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f44f1a8e:::benchmark_data_smithi169_60938_object139541:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4500d70:::benchmark_data_smithi169_60938_object160562:head# 2024-02-12T16:18:27.774 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4523265:::benchmark_data_smithi169_60938_object146307:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f45346b3:::benchmark_data_smithi169_60938_object164529:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4570097:::benchmark_data_smithi169_60938_object151739:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f45a864c:::benchmark_data_smithi169_60938_object161116:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f45e4f0a:::benchmark_data_smithi169_60938_object168547:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f45f8482:::benchmark_data_smithi169_60938_object165512:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f463d9bc:::benchmark_data_smithi169_60938_object146696:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4646349:::benchmark_data_smithi169_60938_object165509:head# 2024-02-12T16:18:27.775 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4663a4b:::benchmark_data_smithi169_60938_object153767:head# 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4673c19:::benchmark_data_smithi169_60938_object157208:head# 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4679ea1:::benchmark_data_smithi169_60938_object152773:head# 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f46bc74a:::benchmark_data_smithi169_60938_object149866:head# 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f46c8a17:::benchmark_data_smithi169_60938_object153786:head# 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f46f6d14:::benchmark_data_smithi169_60938_object172827:head# 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f47087db:::benchmark_data_smithi169_60938_object141698:head# 2024-02-12T16:18:27.776 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f47110df:::benchmark_data_smithi169_60938_object145715:head# 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f471d60a:::benchmark_data_smithi169_60938_object155334:head# 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f47349f1:::benchmark_data_smithi169_60938_object172405:head# 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f473bed6:::benchmark_data_smithi169_60938_object168485:head# 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4760d0c:::benchmark_data_smithi169_60938_object156160:head# 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4794d9f:::benchmark_data_smithi169_60938_object166443:head# 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4795ca8:::benchmark_data_smithi169_60938_object153763:head# 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.777 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f47e8b2c:::benchmark_data_smithi169_60938_object155715:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f47f94fb:::benchmark_data_smithi169_60938_object139320:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f481bf0c:::benchmark_data_smithi169_60938_object152588:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f481e5d1:::benchmark_data_smithi169_60938_object144182:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4856e9a:::benchmark_data_smithi169_60938_object163425:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f486cb75:::benchmark_data_smithi169_60938_object151223:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4879593:::benchmark_data_smithi169_60938_object158480:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f48fa2cf:::benchmark_data_smithi169_60938_object168726:head# 2024-02-12T16:18:27.778 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4944516:::benchmark_data_smithi169_60938_object168360:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f49b12ba:::benchmark_data_smithi169_60938_object148776:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f49d4011:::benchmark_data_smithi169_60938_object142467:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f49f718f:::benchmark_data_smithi169_60938_object145455:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f49f92b1:::benchmark_data_smithi169_60938_object163045:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4a03130:::benchmark_data_smithi169_60938_object150089:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4a1ca0b:::benchmark_data_smithi169_60938_object159970:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4a3268b:::benchmark_data_smithi169_60938_object156575:head# 2024-02-12T16:18:27.779 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4a52049:::benchmark_data_smithi169_60938_object163158:head# 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4a5b40e:::benchmark_data_smithi169_60938_object164616:head# 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4a65154:::benchmark_data_smithi169_60938_object156958:head# 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4a655a3:::benchmark_data_smithi169_60938_object159211:head# 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4abc208:::benchmark_data_smithi169_60938_object169955:head# 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ac544b:::benchmark_data_smithi169_60938_object150481:head# 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ad0ebc:::benchmark_data_smithi169_60938_object166865:head# 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.780 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ae7629:::benchmark_data_smithi169_60938_object164303:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ae883f:::benchmark_data_smithi169_60938_object170790:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4b073d4:::benchmark_data_smithi169_60938_object141118:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4b0a8c6:::benchmark_data_smithi169_60938_object141832:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4b25e41:::benchmark_data_smithi169_60938_object172140:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4b2c166:::benchmark_data_smithi169_60938_object164844:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4b3a4ed:::benchmark_data_smithi169_60938_object147826:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4b4d3ff:::benchmark_data_smithi169_60938_object172431:head# 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.781 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ba5953:::benchmark_data_smithi169_60938_object144346:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4bca98d:::benchmark_data_smithi169_60938_object154520:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4bd4846:::benchmark_data_smithi169_60938_object166434:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4c01238:::benchmark_data_smithi169_60938_object146021:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4c1e00e:::benchmark_data_smithi169_60938_object167303:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4c21107:::benchmark_data_smithi169_60938_object158402:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4c7bb50:::benchmark_data_smithi169_60938_object159808:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4c80198:::benchmark_data_smithi169_60938_object166053:head# 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.782 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4c84d1e:::benchmark_data_smithi169_60938_object141176:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4c8d18e:::benchmark_data_smithi169_60938_object160178:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4cc4c1a:::benchmark_data_smithi169_60938_object152283:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4cd7b97:::benchmark_data_smithi169_60938_object155176:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d0166c:::benchmark_data_smithi169_60938_object160981:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d07341:::benchmark_data_smithi169_60938_object145864:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d1d9fe:::benchmark_data_smithi169_60938_object141548:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d1e590:::benchmark_data_smithi169_60938_object173282:head# 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d20d54:::benchmark_data_smithi169_60938_object150240:head# 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d53409:::benchmark_data_smithi169_60938_object161605:head# 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d5ed0c:::benchmark_data_smithi169_60938_object156061:head# 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d65aff:::benchmark_data_smithi169_60938_object150384:head# 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4d7b794:::benchmark_data_smithi169_60938_object162314:head# 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4dc67cb:::benchmark_data_smithi169_60938_object154506:head# 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4dec3f1:::benchmark_data_smithi169_60938_object166202:head# 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.784 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4e0153d:::benchmark_data_smithi169_60938_object156376:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4e3c1d2:::benchmark_data_smithi169_60938_object171154:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4e3fef3:::benchmark_data_smithi169_60938_object139612:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4e7840b:::benchmark_data_smithi169_60938_object171581:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4eacb68:::benchmark_data_smithi169_60938_object156325:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4eb365d:::benchmark_data_smithi169_60938_object173036:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ed9c21:::benchmark_data_smithi169_60938_object148964:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ef88a6:::benchmark_data_smithi169_60938_object166859:head# 2024-02-12T16:18:27.785 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4f21fd7:::benchmark_data_smithi169_60938_object148519:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4f331ee:::benchmark_data_smithi169_60938_object143313:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4f3b112:::benchmark_data_smithi169_60938_object148069:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4f8742a:::benchmark_data_smithi169_60938_object143512:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f4ffb5c4:::benchmark_data_smithi169_60938_object142615:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f500feb1:::benchmark_data_smithi169_60938_object158842:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5011ff9:::benchmark_data_smithi169_60938_object149304:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5014274:::benchmark_data_smithi169_60938_object157420:head# 2024-02-12T16:18:27.786 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f502ec26:::benchmark_data_smithi169_60938_object142775:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f50c02dd:::benchmark_data_smithi169_60938_object163086:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f50cabee:::benchmark_data_smithi169_60938_object169716:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f50ccc05:::benchmark_data_smithi169_60938_object149244:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f50e72dc:::benchmark_data_smithi169_60938_object155684:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5115f9a:::benchmark_data_smithi169_60938_object142975:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f515e8a8:::benchmark_data_smithi169_60938_object146490:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5171023:::benchmark_data_smithi169_60938_object154449:head# 2024-02-12T16:18:27.787 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f51a0238:::benchmark_data_smithi169_60938_object150232:head# 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f51a7be9:::benchmark_data_smithi169_60938_object151659:head# 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f51c4364:::benchmark_data_smithi169_60938_object173205:head# 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f51cc232:::benchmark_data_smithi169_60938_object168520:head# 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f51e8bd5:::benchmark_data_smithi169_60938_object169224:head# 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5224c35:::benchmark_data_smithi169_60938_object159128:head# 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5279278:::benchmark_data_smithi169_60938_object149412:head# 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.788 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f52960b2:::benchmark_data_smithi169_60938_object148654:head# 2024-02-12T16:18:27.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.789 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f52bd7fb:::benchmark_data_smithi169_60938_object150351:head# 2024-02-12T16:18:27.789 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f52c9de4:::benchmark_data_smithi169_60938_object160229:head# 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f52d81ec:::benchmark_data_smithi169_60938_object171032:head# 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f52ed210:::benchmark_data_smithi169_60938_object166233:head# 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f531ffea:::benchmark_data_smithi169_60938_object148490:head# 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.813 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5329472:::benchmark_data_smithi169_60938_object141270:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53638c0:::benchmark_data_smithi169_60938_object155568:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5370bbd:::benchmark_data_smithi169_60938_object164061:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f537460c:::benchmark_data_smithi169_60938_object153940:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53a5890:::benchmark_data_smithi169_60938_object141191:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53abd23:::benchmark_data_smithi169_60938_object145946:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53b1f6f:::benchmark_data_smithi169_60938_object170309:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53bc8ed:::benchmark_data_smithi169_60938_object138832:head# 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.814 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53d214a:::benchmark_data_smithi169_60938_object143623:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53df32c:::benchmark_data_smithi169_60938_object164553:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f53f98c7:::benchmark_data_smithi169_60938_object145483:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5423fd4:::benchmark_data_smithi169_60938_object145086:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f548c706:::benchmark_data_smithi169_60938_object172636:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f548feb1:::benchmark_data_smithi169_60938_object170616:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5496328:::benchmark_data_smithi169_60938_object166858:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f54b3b05:::benchmark_data_smithi169_60938_object162120:head# 2024-02-12T16:18:27.815 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f54bd80a:::benchmark_data_smithi169_60938_object145100:head# 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f551e4de:::benchmark_data_smithi169_60938_object168995:head# 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5547b8f:::benchmark_data_smithi169_60938_object152952:head# 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f557dcf3:::benchmark_data_smithi169_60938_object151357:head# 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5581bb8:::benchmark_data_smithi169_60938_object153526:head# 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f558214a:::benchmark_data_smithi169_60938_object164145:head# 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f55c198a:::benchmark_data_smithi169_60938_object138750:head# 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f55e1e2e:::benchmark_data_smithi169_60938_object158882:head# 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5605616:::benchmark_data_smithi169_60938_object169718:head# 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f56077cb:::benchmark_data_smithi169_60938_object153570:head# 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5645432:::benchmark_data_smithi169_60938_object157799:head# 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5646459:::benchmark_data_smithi169_60938_object162497:head# 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f564f03f:::benchmark_data_smithi169_60938_object168684:head# 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5654721:::benchmark_data_smithi169_60938_object171446:head# 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.817 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f565cd0d:::benchmark_data_smithi169_60938_object168565:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f566579c:::benchmark_data_smithi169_60938_object169518:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5677941:::benchmark_data_smithi169_60938_object142224:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5688135:::benchmark_data_smithi169_60938_object163066:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f569a1b2:::benchmark_data_smithi169_60938_object160904:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f569b77a:::benchmark_data_smithi169_60938_object153832:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f56c7ba7:::benchmark_data_smithi169_60938_object159983:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f571f903:::benchmark_data_smithi169_60938_object139087:head# 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.818 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f574298e:::benchmark_data_smithi169_60938_object141661:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f57441d1:::benchmark_data_smithi169_60938_object164309:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f574e562:::benchmark_data_smithi169_60938_object154712:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5757225:::benchmark_data_smithi169_60938_object170136:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5759ccd:::benchmark_data_smithi169_60938_object153958:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f575f94d:::benchmark_data_smithi169_60938_object161515:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f577fd19:::benchmark_data_smithi169_60938_object169889:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5795845:::benchmark_data_smithi169_60938_object163555:head# 2024-02-12T16:18:27.819 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f57aa24a:::benchmark_data_smithi169_60938_object167805:head# 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f580220d:::benchmark_data_smithi169_60938_object164721:head# 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f581c458:::benchmark_data_smithi169_60938_object141512:head# 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f582c5c5:::benchmark_data_smithi169_60938_object156510:head# 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5852675:::benchmark_data_smithi169_60938_object151451:head# 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5860e5e:::benchmark_data_smithi169_60938_object160541:head# 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5869c29:::benchmark_data_smithi169_60938_object170160:head# 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.820 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f58a9f70:::benchmark_data_smithi169_60938_object139906:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f58cab5f:::benchmark_data_smithi169_60938_object167535:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f58d2bb0:::benchmark_data_smithi169_60938_object150367:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5912d33:::benchmark_data_smithi169_60938_object151917:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5927674:::benchmark_data_smithi169_60938_object157847:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f592a884:::benchmark_data_smithi169_60938_object139955:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5939d09:::benchmark_data_smithi169_60938_object146488:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f59bd3c4:::benchmark_data_smithi169_60938_object145244:head# 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.821 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f59c7d61:::benchmark_data_smithi169_60938_object141578:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f59dac21:::benchmark_data_smithi169_60938_object167970:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5a00380:::benchmark_data_smithi169_60938_object169305:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5a34988:::benchmark_data_smithi169_60938_object142524:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5a66a4a:::benchmark_data_smithi169_60938_object163299:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5a87381:::benchmark_data_smithi169_60938_object150265:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5ab7078:::benchmark_data_smithi169_60938_object144913:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5ac4dac:::benchmark_data_smithi169_60938_object142309:head# 2024-02-12T16:18:27.822 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b013d4:::benchmark_data_smithi169_60938_object141001:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b128f4:::benchmark_data_smithi169_60938_object168027:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b28659:::benchmark_data_smithi169_60938_object150824:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b363e9:::benchmark_data_smithi169_60938_object155912:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b4c566:::benchmark_data_smithi169_60938_object161696:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b53e2b:::benchmark_data_smithi169_60938_object142862:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b5d7e6:::benchmark_data_smithi169_60938_object169448:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5b9c2ee:::benchmark_data_smithi169_60938_object163156:head# 2024-02-12T16:18:27.823 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5bf558a:::benchmark_data_smithi169_60938_object143078:head# 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c0dc73:::benchmark_data_smithi169_60938_object149192:head# 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c37d9c:::benchmark_data_smithi169_60938_object138561:head# 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c3f5f5:::benchmark_data_smithi169_60938_object172023:head# 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c6b087:::benchmark_data_smithi169_60938_object163614:head# 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c70e17:::benchmark_data_smithi169_60938_object155947:head# 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c72795:::benchmark_data_smithi169_60938_object162341:head# 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.824 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c73f1f:::benchmark_data_smithi169_60938_object148838:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5c96368:::benchmark_data_smithi169_60938_object149566:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5cb4f44:::benchmark_data_smithi169_60938_object153181:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5cba01f:::benchmark_data_smithi169_60938_object169601:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5d04315:::benchmark_data_smithi169_60938_object139019:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5d04fa8:::benchmark_data_smithi169_60938_object150901:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5d0ba9f:::benchmark_data_smithi169_60938_object150156:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5d3dac7:::benchmark_data_smithi169_60938_object147859:head# 2024-02-12T16:18:27.825 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5d5cb62:::benchmark_data_smithi169_60938_object165991:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5db3562:::benchmark_data_smithi169_60938_object139746:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5dbd196:::benchmark_data_smithi169_60938_object141881:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5dd0628:::benchmark_data_smithi169_60938_object154174:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5df8183:::benchmark_data_smithi169_60938_object143917:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5e3ad3b:::benchmark_data_smithi169_60938_object167508:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5e4cea9:::benchmark_data_smithi169_60938_object159874:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5e6cbc5:::benchmark_data_smithi169_60938_object141125:head# 2024-02-12T16:18:27.826 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5e7f370:::benchmark_data_smithi169_60938_object150381:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5ea9280:::benchmark_data_smithi169_60938_object161638:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5ebce47:::benchmark_data_smithi169_60938_object143970:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5ec829a:::benchmark_data_smithi169_60938_object152687:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5f03136:::benchmark_data_smithi169_60938_object172688:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5f0a061:::benchmark_data_smithi169_60938_object139487:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5f0c9de:::benchmark_data_smithi169_60938_object139705:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5f18c3f:::benchmark_data_smithi169_60938_object164194:head# 2024-02-12T16:18:27.827 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5f1d1ca:::benchmark_data_smithi169_60938_object169144:head# 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5f3991d:::benchmark_data_smithi169_60938_object150248:head# 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5f8e88b:::benchmark_data_smithi169_60938_object166464:head# 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f5fa8a4d:::benchmark_data_smithi169_60938_object156411:head# 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f600da82:::benchmark_data_smithi169_60938_object170341:head# 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f602b62a:::benchmark_data_smithi169_60938_object142670:head# 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f604e917:::benchmark_data_smithi169_60938_object159149:head# 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.828 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6069959:::benchmark_data_smithi169_60938_object163516:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6077d8e:::benchmark_data_smithi169_60938_object154331:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f61396c4:::benchmark_data_smithi169_60938_object155387:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6161947:::benchmark_data_smithi169_60938_object147619:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f61952a5:::benchmark_data_smithi169_60938_object163326:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f61beb97:::benchmark_data_smithi169_60938_object156515:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f61ec495:::benchmark_data_smithi169_60938_object140483:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6217ae6:::benchmark_data_smithi169_60938_object140838:head# 2024-02-12T16:18:27.829 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6240fc9:::benchmark_data_smithi169_60938_object154542:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f625ee10:::benchmark_data_smithi169_60938_object157910:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f62c52af:::benchmark_data_smithi169_60938_object147302:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f62f8ad5:::benchmark_data_smithi169_60938_object154533:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6354c37:::benchmark_data_smithi169_60938_object168429:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f63bddf0:::benchmark_data_smithi169_60938_object172666:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f63cdf21:::benchmark_data_smithi169_60938_object164296:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6410648:::benchmark_data_smithi169_60938_object145480:head# 2024-02-12T16:18:27.830 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6439a5f:::benchmark_data_smithi169_60938_object164554:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f643a657:::benchmark_data_smithi169_60938_object161946:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f645b4b5:::benchmark_data_smithi169_60938_object152524:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f648197c:::benchmark_data_smithi169_60938_object142647:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f648fbb6:::benchmark_data_smithi169_60938_object164454:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f64f75a9:::benchmark_data_smithi169_60938_object142991:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f65785c1:::benchmark_data_smithi169_60938_object153639:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f657ec26:::benchmark_data_smithi169_60938_object166906:head# 2024-02-12T16:18:27.831 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6580b9f:::benchmark_data_smithi169_60938_object139077:head# 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f65af97a:::benchmark_data_smithi169_60938_object146295:head# 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f662c160:::benchmark_data_smithi169_60938_object163278:head# 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6653465:::benchmark_data_smithi169_60938_object147609:head# 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f665e3a5:::benchmark_data_smithi169_60938_object172917:head# 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6674aa1:::benchmark_data_smithi169_60938_object152672:head# 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f668fb3a:::benchmark_data_smithi169_60938_object161223:head# 2024-02-12T16:18:27.832 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f66d021b:::benchmark_data_smithi169_60938_object141179:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f66fe986:::benchmark_data_smithi169_60938_object154094:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f66ffb86:::benchmark_data_smithi169_60938_object165227:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6705adb:::benchmark_data_smithi169_60938_object155102:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f67163a0:::benchmark_data_smithi169_60938_object142228:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f673764d:::benchmark_data_smithi169_60938_object140379:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6740edf:::benchmark_data_smithi169_60938_object142129:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6795921:::benchmark_data_smithi169_60938_object168843:head# 2024-02-12T16:18:27.833 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f67b76c4:::benchmark_data_smithi169_60938_object149730:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f67df4b2:::benchmark_data_smithi169_60938_object146450:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f67ed959:::benchmark_data_smithi169_60938_object148822:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f67f731e:::benchmark_data_smithi169_60938_object155762:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f681de7e:::benchmark_data_smithi169_60938_object168089:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f682e400:::benchmark_data_smithi169_60938_object161401:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f68414b9:::benchmark_data_smithi169_60938_object160528:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6849ee1:::benchmark_data_smithi169_60938_object170960:head# 2024-02-12T16:18:27.834 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f684de00:::benchmark_data_smithi169_60938_object172936:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f686d9ab:::benchmark_data_smithi169_60938_object141831:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6882766:::benchmark_data_smithi169_60938_object158613:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f688349f:::benchmark_data_smithi169_60938_object160297:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f68bf701:::benchmark_data_smithi169_60938_object141795:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f68cf5cd:::benchmark_data_smithi169_60938_object153232:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6900296:::benchmark_data_smithi169_60938_object145617:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f693cdcf:::benchmark_data_smithi169_60938_object140380:head# 2024-02-12T16:18:27.835 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6964ebf:::benchmark_data_smithi169_60938_object168354:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f69845d6:::benchmark_data_smithi169_60938_object160302:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f698fcce:::benchmark_data_smithi169_60938_object140670:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6997ded:::benchmark_data_smithi169_60938_object149589:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f69ad12a:::benchmark_data_smithi169_60938_object160916:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f69e2b21:::benchmark_data_smithi169_60938_object149551:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6a0308d:::benchmark_data_smithi169_60938_object166325:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6a11ada:::benchmark_data_smithi169_60938_object150463:head# 2024-02-12T16:18:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.837 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6a1c736:::benchmark_data_smithi169_60938_object161527:head# 2024-02-12T16:18:27.837 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.844 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6a27fb2:::benchmark_data_smithi169_60938_object144793:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6a4365c:::benchmark_data_smithi169_60938_object172112:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6a7eaf7:::benchmark_data_smithi169_60938_object152789:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ab68ce:::benchmark_data_smithi169_60938_object139530:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ac0753:::benchmark_data_smithi169_60938_object160369:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ac92a4:::benchmark_data_smithi169_60938_object155228:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ad8495:::benchmark_data_smithi169_60938_object150143:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6af4a2b:::benchmark_data_smithi169_60938_object147915:head# 2024-02-12T16:18:27.845 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6b0220f:::benchmark_data_smithi169_60938_object151495:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6b26675:::benchmark_data_smithi169_60938_object152367:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6b28aa9:::benchmark_data_smithi169_60938_object170267:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6b3955d:::benchmark_data_smithi169_60938_object141695:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6b41200:::benchmark_data_smithi169_60938_object155574:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6b7d581:::benchmark_data_smithi169_60938_object157637:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6bab837:::benchmark_data_smithi169_60938_object161351:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6c1dc44:::benchmark_data_smithi169_60938_object157210:head# 2024-02-12T16:18:27.846 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6c9a017:::benchmark_data_smithi169_60938_object160939:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6cffa30:::benchmark_data_smithi169_60938_object171144:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6d319b4:::benchmark_data_smithi169_60938_object141417:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6d76e2b:::benchmark_data_smithi169_60938_object165203:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6d99c69:::benchmark_data_smithi169_60938_object171254:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6da9d0d:::benchmark_data_smithi169_60938_object165816:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6dd7292:::benchmark_data_smithi169_60938_object146049:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ddf645:::benchmark_data_smithi169_60938_object169121:head# 2024-02-12T16:18:27.847 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6de93c5:::benchmark_data_smithi169_60938_object161299:head# 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6dfb0d7:::benchmark_data_smithi169_60938_object148511:head# 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6e2c851:::benchmark_data_smithi169_60938_object169533:head# 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6e3450a:::benchmark_data_smithi169_60938_object164245:head# 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6e42d31:::benchmark_data_smithi169_60938_object146299:head# 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6e626ae:::benchmark_data_smithi169_60938_object148825:head# 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6e64c61:::benchmark_data_smithi169_60938_object140181:head# 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ea2a88:::benchmark_data_smithi169_60938_object169595:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6eb50cf:::benchmark_data_smithi169_60938_object138254:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6eb5d7d:::benchmark_data_smithi169_60938_object169366:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ebe209:::benchmark_data_smithi169_60938_object153932:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6ebeb67:::benchmark_data_smithi169_60938_object162167:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f0ab55:::benchmark_data_smithi169_60938_object157703:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f108f1:::benchmark_data_smithi169_60938_object146467:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f17431:::benchmark_data_smithi169_60938_object158742:head# 2024-02-12T16:18:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f1b23a:::benchmark_data_smithi169_60938_object165688:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f1e67c:::benchmark_data_smithi169_60938_object164887:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f30c85:::benchmark_data_smithi169_60938_object143892:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f4c340:::benchmark_data_smithi169_60938_object166535:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f7ae9b:::benchmark_data_smithi169_60938_object172584:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f84a1f:::benchmark_data_smithi169_60938_object164354:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6f8cf34:::benchmark_data_smithi169_60938_object152846:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6fa2668:::benchmark_data_smithi169_60938_object156226:head# 2024-02-12T16:18:27.850 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6fcfabd:::benchmark_data_smithi169_60938_object149900:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f6fdc19e:::benchmark_data_smithi169_60938_object148090:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f700a68c:::benchmark_data_smithi169_60938_object152849:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f700f132:::benchmark_data_smithi169_60938_object155226:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f70746a6:::benchmark_data_smithi169_60938_object152972:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f70a9243:::benchmark_data_smithi169_60938_object143629:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f70b2182:::benchmark_data_smithi169_60938_object144970:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f70c0c33:::benchmark_data_smithi169_60938_object156396:head# 2024-02-12T16:18:27.851 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f70ffcca:::benchmark_data_smithi169_60938_object165578:head# 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f710fd49:::benchmark_data_smithi169_60938_object152737:head# 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7128243:::benchmark_data_smithi169_60938_object156914:head# 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f712a073:::benchmark_data_smithi169_60938_object155031:head# 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f714bf9e:::benchmark_data_smithi169_60938_object165632:head# 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f716b978:::benchmark_data_smithi169_60938_object155617:head# 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f71c48c1:::benchmark_data_smithi169_60938_object145635:head# 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.852 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7219477:::benchmark_data_smithi169_60938_object154026:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7227f88:::benchmark_data_smithi169_60938_object166749:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f723213b:::benchmark_data_smithi169_60938_object148191:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7236604:::benchmark_data_smithi169_60938_object147422:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f725bcfe:::benchmark_data_smithi169_60938_object172741:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f726a40d:::benchmark_data_smithi169_60938_object141874:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f726e68a:::benchmark_data_smithi169_60938_object162962:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7277359:::benchmark_data_smithi169_60938_object152458:head# 2024-02-12T16:18:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f72b5f2d:::benchmark_data_smithi169_60938_object155662:head# 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f72ddad5:::benchmark_data_smithi169_60938_object155241:head# 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f72f0993:::benchmark_data_smithi169_60938_object156944:head# 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f730e857:::benchmark_data_smithi169_60938_object172662:head# 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7329014:::benchmark_data_smithi169_60938_object162022:head# 2024-02-12T16:18:27.855 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:27.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:27.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f737c2c4:::benchmark_data_smithi169_60938_object148315:head# 2024-02-12T16:18:27.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.880 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f737c40a:::benchmark_data_smithi169_60938_object156074:head# 2024-02-12T16:18:27.880 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7393928:::benchmark_data_smithi169_60938_object147840:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f739821a:::benchmark_data_smithi169_60938_object164360:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f73a8b4b:::benchmark_data_smithi169_60938_object140967:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f73b9571:::benchmark_data_smithi169_60938_object154193:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f73d029c:::benchmark_data_smithi169_60938_object168850:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f73fcf2e:::benchmark_data_smithi169_60938_object159809:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f741dd7d:::benchmark_data_smithi169_60938_object170885:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7421713:::benchmark_data_smithi169_60938_object163046:head# 2024-02-12T16:18:27.881 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7423de4:::benchmark_data_smithi169_60938_object146393:head# 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f746091e:::benchmark_data_smithi169_60938_object165990:head# 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7465d0f:::benchmark_data_smithi169_60938_object151045:head# 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7473f9e:::benchmark_data_smithi169_60938_object161298:head# 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f74769b7:::benchmark_data_smithi169_60938_object154951:head# 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f747ed1d:::benchmark_data_smithi169_60938_object171962:head# 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f74991bd:::benchmark_data_smithi169_60938_object159107:head# 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.882 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f750e25b:::benchmark_data_smithi169_60938_object139715:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f753408c:::benchmark_data_smithi169_60938_object143555:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f753711c:::benchmark_data_smithi169_60938_object147367:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f75713e5:::benchmark_data_smithi169_60938_object153572:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f760c720:::benchmark_data_smithi169_60938_object149479:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7616853:::benchmark_data_smithi169_60938_object155408:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7624c44:::benchmark_data_smithi169_60938_object155307:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7631ab2:::benchmark_data_smithi169_60938_object138258:head# 2024-02-12T16:18:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f76635ab:::benchmark_data_smithi169_60938_object148797:head# 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7699bf3:::benchmark_data_smithi169_60938_object173192:head# 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f76a537b:::benchmark_data_smithi169_60938_object158974:head# 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f76b5e5b:::benchmark_data_smithi169_60938_object154159:head# 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f76becd2:::benchmark_data_smithi169_60938_object164199:head# 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f76d2d84:::benchmark_data_smithi169_60938_object171826:head# 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f76d885a:::benchmark_data_smithi169_60938_object159077:head# 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.884 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f775b87c:::benchmark_data_smithi169_60938_object140165:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f777b22a:::benchmark_data_smithi169_60938_object154192:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7781277:::benchmark_data_smithi169_60938_object142403:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f778570b:::benchmark_data_smithi169_60938_object146287:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f778c7cb:::benchmark_data_smithi169_60938_object163068:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7791073:::benchmark_data_smithi169_60938_object155857:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f77a0901:::benchmark_data_smithi169_60938_object167593:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f77c0236:::benchmark_data_smithi169_60938_object149113:head# 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.885 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f77cfc1d:::benchmark_data_smithi169_60938_object152244:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f77dadf4:::benchmark_data_smithi169_60938_object151075:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f77e68c8:::benchmark_data_smithi169_60938_object166298:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f77ee9ce:::benchmark_data_smithi169_60938_object163317:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f784528e:::benchmark_data_smithi169_60938_object148745:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7848a17:::benchmark_data_smithi169_60938_object150042:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f784c6e9:::benchmark_data_smithi169_60938_object163461:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f78912d6:::benchmark_data_smithi169_60938_object154423:head# 2024-02-12T16:18:27.886 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f78a29e4:::benchmark_data_smithi169_60938_object156364:head# 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f78b2a43:::benchmark_data_smithi169_60938_object169204:head# 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f78e0f1f:::benchmark_data_smithi169_60938_object156587:head# 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7939553:::benchmark_data_smithi169_60938_object155885:head# 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7944a47:::benchmark_data_smithi169_60938_object164401:head# 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f795dd3a:::benchmark_data_smithi169_60938_object171331:head# 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f79a3e44:::benchmark_data_smithi169_60938_object153033:head# 2024-02-12T16:18:27.887 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f79a7148:::benchmark_data_smithi169_60938_object138961:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f79cb411:::benchmark_data_smithi169_60938_object140860:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f79d15b5:::benchmark_data_smithi169_60938_object163757:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f79f862d:::benchmark_data_smithi169_60938_object155934:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f79f9a4a:::benchmark_data_smithi169_60938_object141927:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7a0671d:::benchmark_data_smithi169_60938_object171157:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7a377ed:::benchmark_data_smithi169_60938_object144818:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7a4a547:::benchmark_data_smithi169_60938_object165404:head# 2024-02-12T16:18:27.888 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7b40a89:::benchmark_data_smithi169_60938_object148128:head# 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7b42897:::benchmark_data_smithi169_60938_object161456:head# 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7b580fc:::benchmark_data_smithi169_60938_object139144:head# 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7b7c258:::benchmark_data_smithi169_60938_object171617:head# 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7b8d972:::benchmark_data_smithi169_60938_object160431:head# 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7bb5fee:::benchmark_data_smithi169_60938_object144070:head# 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7bb9f7a:::benchmark_data_smithi169_60938_object146218:head# 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.889 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7bc52f8:::benchmark_data_smithi169_60938_object156388:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7bd56a5:::benchmark_data_smithi169_60938_object168772:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7bdf8b0:::benchmark_data_smithi169_60938_object165733:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7c0b5bf:::benchmark_data_smithi169_60938_object167335:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7c16fbf:::benchmark_data_smithi169_60938_object171337:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7c58c18:::benchmark_data_smithi169_60938_object172712:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7ca09e9:::benchmark_data_smithi169_60938_object170854:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7ca5966:::benchmark_data_smithi169_60938_object162415:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7cb03a1:::benchmark_data_smithi169_60938_object166875:head# 2024-02-12T16:18:27.890 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7cd1403:::benchmark_data_smithi169_60938_object139876:head# 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7d2a78c:::benchmark_data_smithi169_60938_object166720:head# 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7d42fa9:::benchmark_data_smithi169_60938_object147218:head# 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7d637fc:::benchmark_data_smithi169_60938_object147773:head# 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7d6c5e2:::benchmark_data_smithi169_60938_object156204:head# 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7d83990:::benchmark_data_smithi169_60938_object149472:head# 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7d85a23:::benchmark_data_smithi169_60938_object166377:head# 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7da6be5:::benchmark_data_smithi169_60938_object161105:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7df1f08:::benchmark_data_smithi169_60938_object160925:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7e6f3d4:::benchmark_data_smithi169_60938_object157167:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7ee213c:::benchmark_data_smithi169_60938_object144748:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7ee96d2:::benchmark_data_smithi169_60938_object146148:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7f06d3d:::benchmark_data_smithi169_60938_object167328:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7f0735c:::benchmark_data_smithi169_60938_object162072:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7f1e75d:::benchmark_data_smithi169_60938_object167551:head# 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.892 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7f35c7a:::benchmark_data_smithi169_60938_object163497:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7fab6fb:::benchmark_data_smithi169_60938_object143253:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7fb11f7:::benchmark_data_smithi169_60938_object162066:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f7ff5bb1:::benchmark_data_smithi169_60938_object165786:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8004384:::benchmark_data_smithi169_60938_object143697:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f800d4ca:::benchmark_data_smithi169_60938_object152853:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f801f6c9:::benchmark_data_smithi169_60938_object141905:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f802e144:::benchmark_data_smithi169_60938_object158295:head# 2024-02-12T16:18:27.893 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8030e3b:::benchmark_data_smithi169_60938_object139214:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8036a54:::benchmark_data_smithi169_60938_object147568:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f804639a:::benchmark_data_smithi169_60938_object162183:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8049831:::benchmark_data_smithi169_60938_object165406:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f80796bc:::benchmark_data_smithi169_60938_object169015:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f808a242:::benchmark_data_smithi169_60938_object150526:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f80ac997:::benchmark_data_smithi169_60938_object163349:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f80ae78a:::benchmark_data_smithi169_60938_object143269:head# 2024-02-12T16:18:27.894 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f80b17a0:::benchmark_data_smithi169_60938_object151326:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f80c9b3a:::benchmark_data_smithi169_60938_object164378:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f80cd8c2:::benchmark_data_smithi169_60938_object144175:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f810f68f:::benchmark_data_smithi169_60938_object153248:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f812d6ae:::benchmark_data_smithi169_60938_object167301:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8138e69:::benchmark_data_smithi169_60938_object160296:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8188f0d:::benchmark_data_smithi169_60938_object171311:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f81b3962:::benchmark_data_smithi169_60938_object163018:head# 2024-02-12T16:18:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f81eec35:::benchmark_data_smithi169_60938_object169371:head# 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f821b6bc:::benchmark_data_smithi169_60938_object166896:head# 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8220d24:::benchmark_data_smithi169_60938_object161968:head# 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f823e6ff:::benchmark_data_smithi169_60938_object156638:head# 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f824f17d:::benchmark_data_smithi169_60938_object159661:head# 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8260777:::benchmark_data_smithi169_60938_object145001:head# 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f82bdb57:::benchmark_data_smithi169_60938_object150980:head# 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.896 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f830e2e8:::benchmark_data_smithi169_60938_object158130:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f831d352:::benchmark_data_smithi169_60938_object144229:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8333a7f:::benchmark_data_smithi169_60938_object146932:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f83373d1:::benchmark_data_smithi169_60938_object173107:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f834f0f3:::benchmark_data_smithi169_60938_object164055:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f834f99f:::benchmark_data_smithi169_60938_object140019:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f835bfdc:::benchmark_data_smithi169_60938_object148132:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8373f82:::benchmark_data_smithi169_60938_object157975:head# 2024-02-12T16:18:27.897 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f83bd51e:::benchmark_data_smithi169_60938_object147862:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f83d3877:::benchmark_data_smithi169_60938_object166693:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f83e5b4a:::benchmark_data_smithi169_60938_object146220:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8428537:::benchmark_data_smithi169_60938_object145853:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f843fa92:::benchmark_data_smithi169_60938_object139966:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f844e169:::benchmark_data_smithi169_60938_object147385:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8476887:::benchmark_data_smithi169_60938_object161746:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f84774be:::benchmark_data_smithi169_60938_object171010:head# 2024-02-12T16:18:27.898 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8492d8e:::benchmark_data_smithi169_60938_object170882:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f84a8776:::benchmark_data_smithi169_60938_object144609:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f84da7f2:::benchmark_data_smithi169_60938_object154255:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f853e241:::benchmark_data_smithi169_60938_object168318:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8561352:::benchmark_data_smithi169_60938_object173246:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f856f7a1:::benchmark_data_smithi169_60938_object160478:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8585575:::benchmark_data_smithi169_60938_object146646:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8588023:::benchmark_data_smithi169_60938_object153703:head# 2024-02-12T16:18:27.899 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f85b569a:::benchmark_data_smithi169_60938_object162062:head# 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f85c97f3:::benchmark_data_smithi169_60938_object139498:head# 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f85ca5e9:::benchmark_data_smithi169_60938_object166043:head# 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f85cb916:::benchmark_data_smithi169_60938_object172463:head# 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f85d1989:::benchmark_data_smithi169_60938_object162604:head# 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f85d43dc:::benchmark_data_smithi169_60938_object139474:head# 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8600c87:::benchmark_data_smithi169_60938_object154539:head# 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.900 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f865be85:::benchmark_data_smithi169_60938_object154114:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8671b45:::benchmark_data_smithi169_60938_object144200:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f867b735:::benchmark_data_smithi169_60938_object147182:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8692ec5:::benchmark_data_smithi169_60938_object160771:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f86bb070:::benchmark_data_smithi169_60938_object164246:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f86d4f46:::benchmark_data_smithi169_60938_object144848:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f86d9ce5:::benchmark_data_smithi169_60938_object166240:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f871ee0d:::benchmark_data_smithi169_60938_object162451:head# 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.901 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8770db8:::benchmark_data_smithi169_60938_object167528:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f877dca9:::benchmark_data_smithi169_60938_object145995:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f878770d:::benchmark_data_smithi169_60938_object149744:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8789529:::benchmark_data_smithi169_60938_object173316:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f87c019e:::benchmark_data_smithi169_60938_object167905:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8801c28:::benchmark_data_smithi169_60938_object161030:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8881db1:::benchmark_data_smithi169_60938_object166463:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f88c2c73:::benchmark_data_smithi169_60938_object140724:head# 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.902 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f88c36d0:::benchmark_data_smithi169_60938_object146390:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f88c962f:::benchmark_data_smithi169_60938_object153226:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8934441:::benchmark_data_smithi169_60938_object139621:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f89370c2:::benchmark_data_smithi169_60938_object144047:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f893c388:::benchmark_data_smithi169_60938_object149394:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8943742:::benchmark_data_smithi169_60938_object146258:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8956256:::benchmark_data_smithi169_60938_object156873:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f897b8f2:::benchmark_data_smithi169_60938_object171293:head# 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.903 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f89bdc50:::benchmark_data_smithi169_60938_object171432:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8a4c6b6:::benchmark_data_smithi169_60938_object159973:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8a993b1:::benchmark_data_smithi169_60938_object152600:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8afe4cb:::benchmark_data_smithi169_60938_object154339:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8b269af:::benchmark_data_smithi169_60938_object172488:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8b6c6ae:::benchmark_data_smithi169_60938_object148095:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8b92c47:::benchmark_data_smithi169_60938_object152164:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8b9fd31:::benchmark_data_smithi169_60938_object145479:head# 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.904 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8bb455f:::benchmark_data_smithi169_60938_object151898:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8bfc65d:::benchmark_data_smithi169_60938_object150355:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8c23b3d:::benchmark_data_smithi169_60938_object143363:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8c5187c:::benchmark_data_smithi169_60938_object141732:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8c76810:::benchmark_data_smithi169_60938_object151204:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8c8a13e:::benchmark_data_smithi169_60938_object163907:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8c9e9b2:::benchmark_data_smithi169_60938_object165019:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8cb04d5:::benchmark_data_smithi169_60938_object167833:head# 2024-02-12T16:18:27.905 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8cb0a4f:::benchmark_data_smithi169_60938_object162114:head# 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8cd03e7:::benchmark_data_smithi169_60938_object138612:head# 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8cd7679:::benchmark_data_smithi169_60938_object166008:head# 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8ce0fac:::benchmark_data_smithi169_60938_object143353:head# 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8cff757:::benchmark_data_smithi169_60938_object154303:head# 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8d4c305:::benchmark_data_smithi169_60938_object149569:head# 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8d59450:::benchmark_data_smithi169_60938_object140733:head# 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8d6ea09:::benchmark_data_smithi169_60938_object153953:head# 2024-02-12T16:18:27.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.907 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8d7d62c:::benchmark_data_smithi169_60938_object167133:head# 2024-02-12T16:18:27.907 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.927 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8ddd0a6:::benchmark_data_smithi169_60938_object158025:head# 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8dec458:::benchmark_data_smithi169_60938_object165254:head# 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8e31481:::benchmark_data_smithi169_60938_object139167:head# 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8e3a58d:::benchmark_data_smithi169_60938_object156194:head# 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8e5ac56:::benchmark_data_smithi169_60938_object157439:head# 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8e6a24c:::benchmark_data_smithi169_60938_object152356:head# 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.928 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8e7416d:::benchmark_data_smithi169_60938_object159485:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8e94658:::benchmark_data_smithi169_60938_object149091:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8ea14dc:::benchmark_data_smithi169_60938_object153666:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8ed0268:::benchmark_data_smithi169_60938_object161378:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8edbf77:::benchmark_data_smithi169_60938_object163209:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8ef8636:::benchmark_data_smithi169_60938_object152964:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8efbba6:::benchmark_data_smithi169_60938_object140538:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8efdf8e:::benchmark_data_smithi169_60938_object140795:head# 2024-02-12T16:18:27.929 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8f13cb3:::benchmark_data_smithi169_60938_object144973:head# 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8f276f0:::benchmark_data_smithi169_60938_object158249:head# 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8f2868e:::benchmark_data_smithi169_60938_object173201:head# 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8f4dd1a:::benchmark_data_smithi169_60938_object162357:head# 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8f80474:::benchmark_data_smithi169_60938_object149259:head# 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8f8c9b5:::benchmark_data_smithi169_60938_object146672:head# 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8fa5ba9:::benchmark_data_smithi169_60938_object169030:head# 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.930 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8fb9d2f:::benchmark_data_smithi169_60938_object153003:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8fefbe5:::benchmark_data_smithi169_60938_object161853:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f8ffb017:::benchmark_data_smithi169_60938_object148578:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f900e3b7:::benchmark_data_smithi169_60938_object140468:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9014bef:::benchmark_data_smithi169_60938_object141738:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f90512d6:::benchmark_data_smithi169_60938_object147351:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9066663:::benchmark_data_smithi169_60938_object141241:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f90b8e13:::benchmark_data_smithi169_60938_object169687:head# 2024-02-12T16:18:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f90c4363:::benchmark_data_smithi169_60938_object154977:head# 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f90e41e9:::benchmark_data_smithi169_60938_object150517:head# 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f90f0c02:::benchmark_data_smithi169_60938_object160138:head# 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f91b0f80:::benchmark_data_smithi169_60938_object144928:head# 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f91c4015:::benchmark_data_smithi169_60938_object150470:head# 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f91d8e39:::benchmark_data_smithi169_60938_object158422:head# 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9239b8d:::benchmark_data_smithi169_60938_object165543:head# 2024-02-12T16:18:27.932 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9245cbf:::benchmark_data_smithi169_60938_object164712:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f924f9b7:::benchmark_data_smithi169_60938_object154064:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f925759f:::benchmark_data_smithi169_60938_object170590:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f925febb:::benchmark_data_smithi169_60938_object157181:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f92614b9:::benchmark_data_smithi169_60938_object170718:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f92a40b1:::benchmark_data_smithi169_60938_object160711:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f92d504e:::benchmark_data_smithi169_60938_object172539:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f92dd80b:::benchmark_data_smithi169_60938_object145860:head# 2024-02-12T16:18:27.933 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f92e4b00:::benchmark_data_smithi169_60938_object139638:head# 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9301934:::benchmark_data_smithi169_60938_object141945:head# 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9303f0a:::benchmark_data_smithi169_60938_object153908:head# 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f931a089:::benchmark_data_smithi169_60938_object146308:head# 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f93cc3f0:::benchmark_data_smithi169_60938_object165501:head# 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f93df261:::benchmark_data_smithi169_60938_object140588:head# 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f93f9182:::benchmark_data_smithi169_60938_object139183:head# 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.934 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9408bfe:::benchmark_data_smithi169_60938_object164639:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f944a043:::benchmark_data_smithi169_60938_object162001:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9456525:::benchmark_data_smithi169_60938_object145461:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f948cfaf:::benchmark_data_smithi169_60938_object163340:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f948db39:::benchmark_data_smithi169_60938_object142828:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f949f758:::benchmark_data_smithi169_60938_object160329:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f94a10f2:::benchmark_data_smithi169_60938_object141629:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f94a3d25:::benchmark_data_smithi169_60938_object167754:head# 2024-02-12T16:18:27.935 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f94c7524:::benchmark_data_smithi169_60938_object151970:head# 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f94cd6c5:::benchmark_data_smithi169_60938_object159715:head# 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f94e9dbc:::benchmark_data_smithi169_60938_object153227:head# 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9576aff:::benchmark_data_smithi169_60938_object143572:head# 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f95a392c:::benchmark_data_smithi169_60938_object163314:head# 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f95c2959:::benchmark_data_smithi169_60938_object151583:head# 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f95fa43d:::benchmark_data_smithi169_60938_object145794:head# 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.936 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9639753:::benchmark_data_smithi169_60938_object142945:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f963df3e:::benchmark_data_smithi169_60938_object143856:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f964580b:::benchmark_data_smithi169_60938_object143687:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9651b30:::benchmark_data_smithi169_60938_object167140:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f965d9e9:::benchmark_data_smithi169_60938_object143435:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9664bc4:::benchmark_data_smithi169_60938_object138938:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f96b3f25:::benchmark_data_smithi169_60938_object156189:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f96cf022:::benchmark_data_smithi169_60938_object143482:head# 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.937 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f972fffd:::benchmark_data_smithi169_60938_object149185:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f975a755:::benchmark_data_smithi169_60938_object151915:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97acf2a:::benchmark_data_smithi169_60938_object154758:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97bcabf:::benchmark_data_smithi169_60938_object166940:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97bcda5:::benchmark_data_smithi169_60938_object148118:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97c60f4:::benchmark_data_smithi169_60938_object141370:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97c6404:::benchmark_data_smithi169_60938_object156005:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97d9ece:::benchmark_data_smithi169_60938_object142788:head# 2024-02-12T16:18:27.938 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97d9ff4:::benchmark_data_smithi169_60938_object151888:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97f9945:::benchmark_data_smithi169_60938_object150174:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f97fd27d:::benchmark_data_smithi169_60938_object147237:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f983ecf7:::benchmark_data_smithi169_60938_object143330:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f983fde8:::benchmark_data_smithi169_60938_object154465:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f984b692:::benchmark_data_smithi169_60938_object159654:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f98576c8:::benchmark_data_smithi169_60938_object153912:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f986e386:::benchmark_data_smithi169_60938_object164352:head# 2024-02-12T16:18:27.939 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f987e110:::benchmark_data_smithi169_60938_object172279:head# 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f98a36f0:::benchmark_data_smithi169_60938_object166198:head# 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f98a8bcf:::benchmark_data_smithi169_60938_object144645:head# 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f98bdb84:::benchmark_data_smithi169_60938_object147871:head# 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9921757:::benchmark_data_smithi169_60938_object142444:head# 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f995c925:::benchmark_data_smithi169_60938_object160573:head# 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f998b4ce:::benchmark_data_smithi169_60938_object161103:head# 2024-02-12T16:18:27.940 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f99a8757:::benchmark_data_smithi169_60938_object145034:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f99bb8db:::benchmark_data_smithi169_60938_object151623:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f99c6a8c:::benchmark_data_smithi169_60938_object144086:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f99cd928:::benchmark_data_smithi169_60938_object139174:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f99db328:::benchmark_data_smithi169_60938_object152404:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f99e4c93:::benchmark_data_smithi169_60938_object159232:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f99e9d58:::benchmark_data_smithi169_60938_object156538:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9a2163d:::benchmark_data_smithi169_60938_object151958:head# 2024-02-12T16:18:27.941 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9a252b8:::benchmark_data_smithi169_60938_object165832:head# 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9a702ea:::benchmark_data_smithi169_60938_object169899:head# 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9a7338a:::benchmark_data_smithi169_60938_object156465:head# 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9aac5b5:::benchmark_data_smithi169_60938_object170016:head# 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9abb8e8:::benchmark_data_smithi169_60938_object157826:head# 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9aea148:::benchmark_data_smithi169_60938_object167036:head# 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9afe00b:::benchmark_data_smithi169_60938_object167351:head# 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.942 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9b0b28b:::benchmark_data_smithi169_60938_object166168:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9b25392:::benchmark_data_smithi169_60938_object145908:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9baae24:::benchmark_data_smithi169_60938_object170721:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9bc5e68:::benchmark_data_smithi169_60938_object146061:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9bd434d:::benchmark_data_smithi169_60938_object156772:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9be078d:::benchmark_data_smithi169_60938_object142194:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9c5933b:::benchmark_data_smithi169_60938_object145598:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9c59a3b:::benchmark_data_smithi169_60938_object160974:head# 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.943 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9c7b40e:::benchmark_data_smithi169_60938_object149997:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9c92a26:::benchmark_data_smithi169_60938_object140325:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9cb56f5:::benchmark_data_smithi169_60938_object160188:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9cee56f:::benchmark_data_smithi169_60938_object138891:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d022bd:::benchmark_data_smithi169_60938_object160358:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d1bae4:::benchmark_data_smithi169_60938_object145863:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d242cd:::benchmark_data_smithi169_60938_object172387:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d299d8:::benchmark_data_smithi169_60938_object147466:head# 2024-02-12T16:18:27.944 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d3d62d:::benchmark_data_smithi169_60938_object158013:head# 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d4e9db:::benchmark_data_smithi169_60938_object144190:head# 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d6017d:::benchmark_data_smithi169_60938_object148591:head# 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d7349e:::benchmark_data_smithi169_60938_object161660:head# 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9d986fc:::benchmark_data_smithi169_60938_object142908:head# 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9dbd291:::benchmark_data_smithi169_60938_object158968:head# 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9dbe31a:::benchmark_data_smithi169_60938_object155692:head# 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.945 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9dd3808:::benchmark_data_smithi169_60938_object164509:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9deab88:::benchmark_data_smithi169_60938_object159088:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9e1cc2b:::benchmark_data_smithi169_60938_object155145:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9e1f4eb:::benchmark_data_smithi169_60938_object155899:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9e60bd6:::benchmark_data_smithi169_60938_object170265:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9e69114:::benchmark_data_smithi169_60938_object151020:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9e902bd:::benchmark_data_smithi169_60938_object158215:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9ed51b7:::benchmark_data_smithi169_60938_object149461:head# 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.946 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9eebc7f:::benchmark_data_smithi169_60938_object147485:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9f01a2f:::benchmark_data_smithi169_60938_object165555:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9f8efec:::benchmark_data_smithi169_60938_object165135:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9f9e2f0:::benchmark_data_smithi169_60938_object153845:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:f9fee51f:::benchmark_data_smithi169_60938_object154168:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa0146eb:::benchmark_data_smithi169_60938_object160506:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa04ac84:::benchmark_data_smithi169_60938_object147895:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa05d54d:::benchmark_data_smithi169_60938_object157516:head# 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.947 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa07e77a:::benchmark_data_smithi169_60938_object163877:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa0adf4f:::benchmark_data_smithi169_60938_object141562:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa0d11c1:::benchmark_data_smithi169_60938_object153885:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa0e2426:::benchmark_data_smithi169_60938_object147235:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa0e3b5c:::benchmark_data_smithi169_60938_object144547:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa0fbeb3:::benchmark_data_smithi169_60938_object171439:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa1445f6:::benchmark_data_smithi169_60938_object171803:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa14cea4:::benchmark_data_smithi169_60938_object146580:head# 2024-02-12T16:18:27.948 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa1afc99:::benchmark_data_smithi169_60938_object141160:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa1cbd99:::benchmark_data_smithi169_60938_object168940:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa20f994:::benchmark_data_smithi169_60938_object152699:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa210ab4:::benchmark_data_smithi169_60938_object157281:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa21b276:::benchmark_data_smithi169_60938_object152474:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa21dff8:::benchmark_data_smithi169_60938_object149133:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2252c7:::benchmark_data_smithi169_60938_object163611:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2572fd:::benchmark_data_smithi169_60938_object160094:head# 2024-02-12T16:18:27.949 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa27341d:::benchmark_data_smithi169_60938_object157389:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2807fd:::benchmark_data_smithi169_60938_object169316:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2a1890:::benchmark_data_smithi169_60938_object168076:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2a7cbb:::benchmark_data_smithi169_60938_object152677:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2b0159:::benchmark_data_smithi169_60938_object145321:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2b7d48:::benchmark_data_smithi169_60938_object166588:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2b9201:::benchmark_data_smithi169_60938_object160947:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2c0794:::benchmark_data_smithi169_60938_object167654:head# 2024-02-12T16:18:27.950 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2cd662:::benchmark_data_smithi169_60938_object151438:head# 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa2fa11d:::benchmark_data_smithi169_60938_object161157:head# 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa3212b5:::benchmark_data_smithi169_60938_object146196:head# 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa348da4:::benchmark_data_smithi169_60938_object145614:head# 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa3ca4fe:::benchmark_data_smithi169_60938_object165475:head# 2024-02-12T16:18:27.951 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:27.970 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa3cc27b:::benchmark_data_smithi169_60938_object140531:head# 2024-02-12T16:18:27.970 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa3dab40:::benchmark_data_smithi169_60938_object140520:head# 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa40519f:::benchmark_data_smithi169_60938_object166838:head# 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa414988:::benchmark_data_smithi169_60938_object169506:head# 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa424616:::benchmark_data_smithi169_60938_object150691:head# 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa4783ef:::benchmark_data_smithi169_60938_object138369:head# 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa4dbca5:::benchmark_data_smithi169_60938_object145408:head# 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa4ff3ee:::benchmark_data_smithi169_60938_object152464:head# 2024-02-12T16:18:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa501a63:::benchmark_data_smithi169_60938_object173235:head# 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5092c9:::benchmark_data_smithi169_60938_object143597:head# 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa50a6f7:::benchmark_data_smithi169_60938_object167480:head# 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa51c366:::benchmark_data_smithi169_60938_object161478:head# 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5308d3:::benchmark_data_smithi169_60938_object156297:head# 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa53f089:::benchmark_data_smithi169_60938_object163670:head# 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa54bfba:::benchmark_data_smithi169_60938_object166761:head# 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.972 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa569522:::benchmark_data_smithi169_60938_object145533:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5699e0:::benchmark_data_smithi169_60938_object149836:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5ce45b:::benchmark_data_smithi169_60938_object142481:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5d4565:::benchmark_data_smithi169_60938_object156754:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5d47d2:::benchmark_data_smithi169_60938_object147577:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5d7e52:::benchmark_data_smithi169_60938_object148103:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa5db063:::benchmark_data_smithi169_60938_object159274:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa6490e7:::benchmark_data_smithi169_60938_object155337:head# 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.973 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa661b50:::benchmark_data_smithi169_60938_object161270:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa6676df:::benchmark_data_smithi169_60938_object139651:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa673de6:::benchmark_data_smithi169_60938_object146555:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa682511:::benchmark_data_smithi169_60938_object144880:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa690acf:::benchmark_data_smithi169_60938_object156421:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa6aa164:::benchmark_data_smithi169_60938_object163908:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa6be2d9:::benchmark_data_smithi169_60938_object157188:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa71f2f2:::benchmark_data_smithi169_60938_object155254:head# 2024-02-12T16:18:27.974 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa74feea:::benchmark_data_smithi169_60938_object151291:head# 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa758b84:::benchmark_data_smithi169_60938_object162574:head# 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa75b8a0:::benchmark_data_smithi169_60938_object152065:head# 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa7893f5:::benchmark_data_smithi169_60938_object163319:head# 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa793ef0:::benchmark_data_smithi169_60938_object161034:head# 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa796f81:::benchmark_data_smithi169_60938_object166897:head# 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa7afe0b:::benchmark_data_smithi169_60938_object142321:head# 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.975 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa7b07b5:::benchmark_data_smithi169_60938_object140076:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa7cb8b7:::benchmark_data_smithi169_60938_object139534:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa7ddc31:::benchmark_data_smithi169_60938_object153693:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa7ec0b5:::benchmark_data_smithi169_60938_object163208:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa7edf45:::benchmark_data_smithi169_60938_object162542:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa815051:::benchmark_data_smithi169_60938_object149514:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa8279df:::benchmark_data_smithi169_60938_object139639:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa87bc6f:::benchmark_data_smithi169_60938_object154136:head# 2024-02-12T16:18:27.976 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa898ed6:::benchmark_data_smithi169_60938_object143191:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa8a2e39:::benchmark_data_smithi169_60938_object163914:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa8f9c53:::benchmark_data_smithi169_60938_object144582:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa956c2e:::benchmark_data_smithi169_60938_object162464:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa96cbdf:::benchmark_data_smithi169_60938_object166746:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa9a021f:::benchmark_data_smithi169_60938_object158520:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fa9a7d08:::benchmark_data_smithi169_60938_object155507:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faa136a9:::benchmark_data_smithi169_60938_object160882:head# 2024-02-12T16:18:27.977 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faa6f5fc:::benchmark_data_smithi169_60938_object147918:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faa72e9a:::benchmark_data_smithi169_60938_object147697:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faa7b429:::benchmark_data_smithi169_60938_object152801:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faaab9eb:::benchmark_data_smithi169_60938_object152880:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faac2e91:::benchmark_data_smithi169_60938_object161642:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faad4128:::benchmark_data_smithi169_60938_object163274:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fab0dbf4:::benchmark_data_smithi169_60938_object162951:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fab2b31f:::benchmark_data_smithi169_60938_object163765:head# 2024-02-12T16:18:27.978 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fab389d8:::benchmark_data_smithi169_60938_object170342:head# 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fab3ae54:::benchmark_data_smithi169_60938_object165895:head# 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fabc4c29:::benchmark_data_smithi169_60938_object161786:head# 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fabd543a:::benchmark_data_smithi169_60938_object163410:head# 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac17480:::benchmark_data_smithi169_60938_object160867:head# 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac28432:::benchmark_data_smithi169_60938_object171377:head# 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac2a447:::benchmark_data_smithi169_60938_object157925:head# 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.979 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac44938:::benchmark_data_smithi169_60938_object169504:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac6d020:::benchmark_data_smithi169_60938_object158492:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac70fec:::benchmark_data_smithi169_60938_object160751:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac922f3:::benchmark_data_smithi169_60938_object160070:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fac9aded:::benchmark_data_smithi169_60938_object164062:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:facb60a9:::benchmark_data_smithi169_60938_object155367:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:facbc4cb:::benchmark_data_smithi169_60938_object139432:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faccf70b:::benchmark_data_smithi169_60938_object145437:head# 2024-02-12T16:18:27.980 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fad30437:::benchmark_data_smithi169_60938_object143517:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fad3ae21:::benchmark_data_smithi169_60938_object155594:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fad4444c:::benchmark_data_smithi169_60938_object150878:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fad5d228:::benchmark_data_smithi169_60938_object146686:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fad7ae4c:::benchmark_data_smithi169_60938_object141012:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fad82dc3:::benchmark_data_smithi169_60938_object141190:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fad9fe3e:::benchmark_data_smithi169_60938_object156460:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fada41bf:::benchmark_data_smithi169_60938_object142751:head# 2024-02-12T16:18:27.981 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fade200f:::benchmark_data_smithi169_60938_object168492:head# 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fadee1ed:::benchmark_data_smithi169_60938_object160798:head# 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fae18eb3:::benchmark_data_smithi169_60938_object156543:head# 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fae7230f:::benchmark_data_smithi169_60938_object144856:head# 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fae74df6:::benchmark_data_smithi169_60938_object160706:head# 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faea9937:::benchmark_data_smithi169_60938_object170660:head# 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faecdf65:::benchmark_data_smithi169_60938_object165629:head# 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faf026e5:::benchmark_data_smithi169_60938_object154733:head# 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faf1d4d3:::benchmark_data_smithi169_60938_object149570:head# 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faf34e4f:::benchmark_data_smithi169_60938_object167124:head# 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faf3cdb6:::benchmark_data_smithi169_60938_object167779:head# 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faf45515:::benchmark_data_smithi169_60938_object145571:head# 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faf5041d:::benchmark_data_smithi169_60938_object148593:head# 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.983 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:faf6a386:::benchmark_data_smithi169_60938_object167125:head# 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fafceab7:::benchmark_data_smithi169_60938_object170171:head# 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb0397e5:::benchmark_data_smithi169_60938_object158016:head# 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb06903c:::benchmark_data_smithi169_60938_object153218:head# 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb08c652:::benchmark_data_smithi169_60938_object138608:head# 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb090aef:::benchmark_data_smithi169_60938_object161276:head# 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb0e6360:::benchmark_data_smithi169_60938_object154852:head# 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.984 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb0f7eb9:::benchmark_data_smithi169_60938_object162544:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb1030a9:::benchmark_data_smithi169_60938_object145688:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb112154:::benchmark_data_smithi169_60938_object169323:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb1a065d:::benchmark_data_smithi169_60938_object151366:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb1cc389:::benchmark_data_smithi169_60938_object151909:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb219f76:::benchmark_data_smithi169_60938_object141839:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb22b6b0:::benchmark_data_smithi169_60938_object160661:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb244a6f:::benchmark_data_smithi169_60938_object169164:head# 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.985 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb2939aa:::benchmark_data_smithi169_60938_object140193:head# 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb2bcf9a:::benchmark_data_smithi169_60938_object156474:head# 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb2ed3a7:::benchmark_data_smithi169_60938_object148929:head# 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb30df7a:::benchmark_data_smithi169_60938_object166380:head# 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb31e4aa:::benchmark_data_smithi169_60938_object162854:head# 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb326883:::benchmark_data_smithi169_60938_object158701:head# 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb32748e:::benchmark_data_smithi169_60938_object158921:head# 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.986 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb373c88:::benchmark_data_smithi169_60938_object141883:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb380831:::benchmark_data_smithi169_60938_object141460:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb38d750:::benchmark_data_smithi169_60938_object150113:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb3990be:::benchmark_data_smithi169_60938_object160260:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb428a13:::benchmark_data_smithi169_60938_object162328:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb491dfa:::benchmark_data_smithi169_60938_object138615:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb499fb1:::benchmark_data_smithi169_60938_object149417:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb4d1374:::benchmark_data_smithi169_60938_object139544:head# 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.987 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb4fbd33:::benchmark_data_smithi169_60938_object138985:head# 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb51a3d3:::benchmark_data_smithi169_60938_object143959:head# 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb524d8d:::benchmark_data_smithi169_60938_object140730:head# 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb530ec7:::benchmark_data_smithi169_60938_object158739:head# 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb536a4b:::benchmark_data_smithi169_60938_object166003:head# 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb54150d:::benchmark_data_smithi169_60938_object172314:head# 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb54f3a2:::benchmark_data_smithi169_60938_object157938:head# 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.988 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb550832:::benchmark_data_smithi169_60938_object168723:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb5544cd:::benchmark_data_smithi169_60938_object156808:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb55ba92:::benchmark_data_smithi169_60938_object165056:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb56761c:::benchmark_data_smithi169_60938_object148031:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb57aaed:::benchmark_data_smithi169_60938_object159706:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb585a91:::benchmark_data_smithi169_60938_object170266:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb59c9c5:::benchmark_data_smithi169_60938_object145690:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb5cd2da:::benchmark_data_smithi169_60938_object165233:head# 2024-02-12T16:18:27.989 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb5dfe9f:::benchmark_data_smithi169_60938_object166215:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb5e19ef:::benchmark_data_smithi169_60938_object143595:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb5e2ddb:::benchmark_data_smithi169_60938_object154362:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb5f5363:::benchmark_data_smithi169_60938_object154593:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb603eec:::benchmark_data_smithi169_60938_object172340:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb610285:::benchmark_data_smithi169_60938_object140983:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb61f8a9:::benchmark_data_smithi169_60938_object150617:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb651930:::benchmark_data_smithi169_60938_object169851:head# 2024-02-12T16:18:27.990 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb65ff55:::benchmark_data_smithi169_60938_object165411:head# 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb66ce57:::benchmark_data_smithi169_60938_object149767:head# 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb6f1358:::benchmark_data_smithi169_60938_object144623:head# 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb712276:::benchmark_data_smithi169_60938_object168962:head# 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb73b306:::benchmark_data_smithi169_60938_object173287:head# 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb74029a:::benchmark_data_smithi169_60938_object156572:head# 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb7517f0:::benchmark_data_smithi169_60938_object160061:head# 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.991 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb753c1d:::benchmark_data_smithi169_60938_object148946:head# 2024-02-12T16:18:27.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb79d706:::benchmark_data_smithi169_60938_object161107:head# 2024-02-12T16:18:27.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb7c7cac:::benchmark_data_smithi169_60938_object150511:head# 2024-02-12T16:18:27.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:27.992 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb7d5a96:::benchmark_data_smithi169_60938_object170651:head# 2024-02-12T16:18:27.992 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:28.005 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:28.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:28.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb7d7017:::benchmark_data_smithi169_60938_object170207:head# 2024-02-12T16:18:28.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb834f26:::benchmark_data_smithi169_60938_object157969:head# 2024-02-12T16:18:28.013 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.013 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb843298:::benchmark_data_smithi169_60938_object142513:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb85dd84:::benchmark_data_smithi169_60938_object159567:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb862128:::benchmark_data_smithi169_60938_object167498:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb86f95a:::benchmark_data_smithi169_60938_object161403:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb89d6e5:::benchmark_data_smithi169_60938_object160111:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb8b66ee:::benchmark_data_smithi169_60938_object152276:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb8e84ca:::benchmark_data_smithi169_60938_object169656:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb908373:::benchmark_data_smithi169_60938_object158847:head# 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.014 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb91b131:::benchmark_data_smithi169_60938_object150206:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb94e013:::benchmark_data_smithi169_60938_object146410:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb95ab05:::benchmark_data_smithi169_60938_object157746:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb95dea3:::benchmark_data_smithi169_60938_object157024:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb9c1dc0:::benchmark_data_smithi169_60938_object149136:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb9cc303:::benchmark_data_smithi169_60938_object168699:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fb9ccbef:::benchmark_data_smithi169_60938_object164704:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fba28dc5:::benchmark_data_smithi169_60938_object171833:head# 2024-02-12T16:18:28.015 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fba2fa71:::benchmark_data_smithi169_60938_object146624:head# 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fba7f1e0:::benchmark_data_smithi169_60938_object144367:head# 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fba8d522:::benchmark_data_smithi169_60938_object151439:head# 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbaa6e25:::benchmark_data_smithi169_60938_object147585:head# 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbab9a2c:::benchmark_data_smithi169_60938_object152682:head# 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbabdc59:::benchmark_data_smithi169_60938_object161325:head# 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbad8f6b:::benchmark_data_smithi169_60938_object161567:head# 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.016 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbb245c0:::benchmark_data_smithi169_60938_object148226:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbb84a12:::benchmark_data_smithi169_60938_object161648:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbb918bf:::benchmark_data_smithi169_60938_object143132:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbb92b51:::benchmark_data_smithi169_60938_object168199:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbbae976:::benchmark_data_smithi169_60938_object148143:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbbbc134:::benchmark_data_smithi169_60938_object138913:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbbbf7fa:::benchmark_data_smithi169_60938_object155750:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbbd466e:::benchmark_data_smithi169_60938_object142842:head# 2024-02-12T16:18:28.017 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbbe407c:::benchmark_data_smithi169_60938_object147287:head# 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbbed521:::benchmark_data_smithi169_60938_object151813:head# 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbc0a5cf:::benchmark_data_smithi169_60938_object156262:head# 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbc2953c:::benchmark_data_smithi169_60938_object150373:head# 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbc564ab:::benchmark_data_smithi169_60938_object157665:head# 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbc5ae6b:::benchmark_data_smithi169_60938_object168254:head# 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbc677d0:::benchmark_data_smithi169_60938_object160935:head# 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.018 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbd0d135:::benchmark_data_smithi169_60938_object154887:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbd4dd50:::benchmark_data_smithi169_60938_object158656:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbd59772:::benchmark_data_smithi169_60938_object171312:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbd691a0:::benchmark_data_smithi169_60938_object163446:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbd6a56f:::benchmark_data_smithi169_60938_object167602:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbd7d522:::benchmark_data_smithi169_60938_object145833:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbd93872:::benchmark_data_smithi169_60938_object165498:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbdc6ac2:::benchmark_data_smithi169_60938_object138496:head# 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.019 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbe0f5f5:::benchmark_data_smithi169_60938_object157860:head# 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbe59b0f:::benchmark_data_smithi169_60938_object151079:head# 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbe727c0:::benchmark_data_smithi169_60938_object160533:head# 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbe925a9:::benchmark_data_smithi169_60938_object165156:head# 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbe9b191:::benchmark_data_smithi169_60938_object146168:head# 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbec62f1:::benchmark_data_smithi169_60938_object140681:head# 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbf047ce:::benchmark_data_smithi169_60938_object148129:head# 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbf200a6:::benchmark_data_smithi169_60938_object164500:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbf2e15f:::benchmark_data_smithi169_60938_object169873:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbfa903d:::benchmark_data_smithi169_60938_object157789:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbfb653d:::benchmark_data_smithi169_60938_object158741:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbfbda38:::benchmark_data_smithi169_60938_object163160:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbfdcd95:::benchmark_data_smithi169_60938_object156305:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbfe6333:::benchmark_data_smithi169_60938_object172632:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fbffb42e:::benchmark_data_smithi169_60938_object147994:head# 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.021 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc00f223:::benchmark_data_smithi169_60938_object155787:head# 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc020934:::benchmark_data_smithi169_60938_object159785:head# 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc0258e0:::benchmark_data_smithi169_60938_object164516:head# 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc035753:::benchmark_data_smithi169_60938_object164103:head# 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc05e220:::benchmark_data_smithi169_60938_object167915:head# 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc06ab93:::benchmark_data_smithi169_60938_object140800:head# 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc0735b5:::benchmark_data_smithi169_60938_object171105:head# 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.022 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc07b6fd:::benchmark_data_smithi169_60938_object156890:head# 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc08d6c4:::benchmark_data_smithi169_60938_object156089:head# 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc08dee2:::benchmark_data_smithi169_60938_object150727:head# 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc095eb8:::benchmark_data_smithi169_60938_object170194:head# 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc0a1f70:::benchmark_data_smithi169_60938_object170479:head# 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc0bb874:::benchmark_data_smithi169_60938_object149151:head# 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc1064df:::benchmark_data_smithi169_60938_object154322:head# 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.023 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc10f7d4:::benchmark_data_smithi169_60938_object144939:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc128907:::benchmark_data_smithi169_60938_object173295:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc12b4da:::benchmark_data_smithi169_60938_object146192:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc12bee6:::benchmark_data_smithi169_60938_object162950:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc15695e:::benchmark_data_smithi169_60938_object140585:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc15b2b9:::benchmark_data_smithi169_60938_object166632:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc1693a1:::benchmark_data_smithi169_60938_object156315:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc17b7ae:::benchmark_data_smithi169_60938_object155155:head# 2024-02-12T16:18:28.024 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc18482b:::benchmark_data_smithi169_60938_object141323:head# 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc194f8a:::benchmark_data_smithi169_60938_object166845:head# 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc1b7a88:::benchmark_data_smithi169_60938_object168426:head# 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc1e3034:::benchmark_data_smithi169_60938_object166050:head# 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc1e6545:::benchmark_data_smithi169_60938_object138497:head# 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc1e80b7:::benchmark_data_smithi169_60938_object152887:head# 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc217403:::benchmark_data_smithi169_60938_object153640:head# 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.025 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc28379c:::benchmark_data_smithi169_60938_object169818:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc2a3cc3:::benchmark_data_smithi169_60938_object157932:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc2c2ccf:::benchmark_data_smithi169_60938_object159885:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc2dd090:::benchmark_data_smithi169_60938_object167765:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc30ce26:::benchmark_data_smithi169_60938_object151804:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc320e66:::benchmark_data_smithi169_60938_object152822:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc329f5a:::benchmark_data_smithi169_60938_object167712:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc337af7:::benchmark_data_smithi169_60938_object147371:head# 2024-02-12T16:18:28.026 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc35854e:::benchmark_data_smithi169_60938_object167081:head# 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc363888:::benchmark_data_smithi169_60938_object143907:head# 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc36ddf7:::benchmark_data_smithi169_60938_object143225:head# 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc37eed4:::benchmark_data_smithi169_60938_object144159:head# 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc3ac158:::benchmark_data_smithi169_60938_object156987:head# 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc3ac6ff:::benchmark_data_smithi169_60938_object149292:head# 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc3c4c38:::benchmark_data_smithi169_60938_object146354:head# 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.027 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc412acd:::benchmark_data_smithi169_60938_object151018:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc422dd1:::benchmark_data_smithi169_60938_object163741:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc423e17:::benchmark_data_smithi169_60938_object139464:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc470d88:::benchmark_data_smithi169_60938_object164437:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc472f7c:::benchmark_data_smithi169_60938_object171118:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc4884e6:::benchmark_data_smithi169_60938_object147424:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc49798f:::benchmark_data_smithi169_60938_object168362:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc497ad9:::benchmark_data_smithi169_60938_object145411:head# 2024-02-12T16:18:28.028 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc4d0c10:::benchmark_data_smithi169_60938_object162755:head# 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc4e4c04:::benchmark_data_smithi169_60938_object168859:head# 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc4e5849:::benchmark_data_smithi169_60938_object147718:head# 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc50b1d3:::benchmark_data_smithi169_60938_object165017:head# 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc544a3a:::benchmark_data_smithi169_60938_object162889:head# 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc5b839b:::benchmark_data_smithi169_60938_object142912:head# 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc5e8f95:::benchmark_data_smithi169_60938_object168848:head# 2024-02-12T16:18:28.029 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc64f2d5:::benchmark_data_smithi169_60938_object148905:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc660fa8:::benchmark_data_smithi169_60938_object165799:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc6768fb:::benchmark_data_smithi169_60938_object163532:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc6874c4:::benchmark_data_smithi169_60938_object141810:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc687555:::benchmark_data_smithi169_60938_object162142:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc69ef0a:::benchmark_data_smithi169_60938_object165696:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc6b500b:::benchmark_data_smithi169_60938_object144728:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc6c1fd5:::benchmark_data_smithi169_60938_object143043:head# 2024-02-12T16:18:28.030 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc6d227e:::benchmark_data_smithi169_60938_object146303:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc7021bc:::benchmark_data_smithi169_60938_object163283:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc70613c:::benchmark_data_smithi169_60938_object153863:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc70a99e:::benchmark_data_smithi169_60938_object142073:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc70cfce:::benchmark_data_smithi169_60938_object167355:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc713583:::benchmark_data_smithi169_60938_object147173:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc72cfb2:::benchmark_data_smithi169_60938_object162054:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc733604:::benchmark_data_smithi169_60938_object144218:head# 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.031 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc73e559:::benchmark_data_smithi169_60938_object141940:head# 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc75536f:::benchmark_data_smithi169_60938_object154706:head# 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc76666c:::benchmark_data_smithi169_60938_object161509:head# 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc7818e6:::benchmark_data_smithi169_60938_object143074:head# 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc7977c6:::benchmark_data_smithi169_60938_object139054:head# 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc7a41ad:::benchmark_data_smithi169_60938_object171149:head# 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc7fffd6:::benchmark_data_smithi169_60938_object145574:head# 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.032 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc82f8c1:::benchmark_data_smithi169_60938_object156196:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc84b211:::benchmark_data_smithi169_60938_object172172:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc84b91a:::benchmark_data_smithi169_60938_object150084:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc87932f:::benchmark_data_smithi169_60938_object157876:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc886308:::benchmark_data_smithi169_60938_object153616:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc8894bb:::benchmark_data_smithi169_60938_object139430:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc8966f2:::benchmark_data_smithi169_60938_object166912:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc8d0a0f:::benchmark_data_smithi169_60938_object158914:head# 2024-02-12T16:18:28.033 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc8d190a:::benchmark_data_smithi169_60938_object168781:head# 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc91d39c:::benchmark_data_smithi169_60938_object140389:head# 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc921815:::benchmark_data_smithi169_60938_object161814:head# 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc92c755:::benchmark_data_smithi169_60938_object157892:head# 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc92d094:::benchmark_data_smithi169_60938_object144081:head# 2024-02-12T16:18:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc93acf4:::benchmark_data_smithi169_60938_object162371:head# 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc946cc3:::benchmark_data_smithi169_60938_object163814:head# 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc955e62:::benchmark_data_smithi169_60938_object153412:head# 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc975430:::benchmark_data_smithi169_60938_object167059:head# 2024-02-12T16:18:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc985a1f:::benchmark_data_smithi169_60938_object143805:head# 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc988bef:::benchmark_data_smithi169_60938_object147982:head# 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc99b62a:::benchmark_data_smithi169_60938_object169195:head# 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc9b305c:::benchmark_data_smithi169_60938_object163953:head# 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc9b4ae4:::benchmark_data_smithi169_60938_object152081:head# 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc9e5381:::benchmark_data_smithi169_60938_object162197:head# 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc9ea6ac:::benchmark_data_smithi169_60938_object159579:head# 2024-02-12T16:18:28.052 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fc9fb1b6:::benchmark_data_smithi169_60938_object149766:head# 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fca1910e:::benchmark_data_smithi169_60938_object143001:head# 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fca3744a:::benchmark_data_smithi169_60938_object171148:head# 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fca76ec7:::benchmark_data_smithi169_60938_object166849:head# 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fca9d479:::benchmark_data_smithi169_60938_object142739:head# 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcab132e:::benchmark_data_smithi169_60938_object150595:head# 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcac6518:::benchmark_data_smithi169_60938_object150002:head# 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.053 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcadcd23:::benchmark_data_smithi169_60938_object172701:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcb0fc31:::benchmark_data_smithi169_60938_object155201:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcb1d7f9:::benchmark_data_smithi169_60938_object160088:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcb234a1:::benchmark_data_smithi169_60938_object151744:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcb3a1b7:::benchmark_data_smithi169_60938_object150751:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcb79864:::benchmark_data_smithi169_60938_object167964:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcbe4746:::benchmark_data_smithi169_60938_object163041:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcbe8245:::benchmark_data_smithi169_60938_object162313:head# 2024-02-12T16:18:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc1b6e7:::benchmark_data_smithi169_60938_object147068:head# 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc281b7:::benchmark_data_smithi169_60938_object158828:head# 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc2b2b1:::benchmark_data_smithi169_60938_object143937:head# 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc36645:::benchmark_data_smithi169_60938_object138944:head# 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc38193:::benchmark_data_smithi169_60938_object140047:head# 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc3c163:::benchmark_data_smithi169_60938_object170808:head# 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc6c8ee:::benchmark_data_smithi169_60938_object143402:head# 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc6e9e6:::benchmark_data_smithi169_60938_object150612:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcc94f4f:::benchmark_data_smithi169_60938_object146353:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fccac44c:::benchmark_data_smithi169_60938_object168240:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fccc5146:::benchmark_data_smithi169_60938_object172566:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fccd1fe2:::benchmark_data_smithi169_60938_object141671:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fccfed03:::benchmark_data_smithi169_60938_object172381:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcd0e623:::benchmark_data_smithi169_60938_object146498:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcd53a0a:::benchmark_data_smithi169_60938_object155834:head# 2024-02-12T16:18:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcd6d267:::benchmark_data_smithi169_60938_object155512:head# 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcd98e4f:::benchmark_data_smithi169_60938_object166336:head# 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcd9c3d4:::benchmark_data_smithi169_60938_object151080:head# 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcda39db:::benchmark_data_smithi169_60938_object144391:head# 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcda4529:::benchmark_data_smithi169_60938_object153000:head# 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcdaac06:::benchmark_data_smithi169_60938_object167268:head# 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcdadf3a:::benchmark_data_smithi169_60938_object169315:head# 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcdbac86:::benchmark_data_smithi169_60938_object146548:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcdc6f66:::benchmark_data_smithi169_60938_object167446:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcdd5365:::benchmark_data_smithi169_60938_object138462:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce0263d:::benchmark_data_smithi169_60938_object152423:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce0a8b2:::benchmark_data_smithi169_60938_object170332:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce19471:::benchmark_data_smithi169_60938_object151835:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce24ed7:::benchmark_data_smithi169_60938_object145253:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce2db9a:::benchmark_data_smithi169_60938_object140877:head# 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.058 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce43118:::benchmark_data_smithi169_60938_object153456:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce439ea:::benchmark_data_smithi169_60938_object148930:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce48217:::benchmark_data_smithi169_60938_object169823:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce48cc1:::benchmark_data_smithi169_60938_object140861:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce4a535:::benchmark_data_smithi169_60938_object148192:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce776c8:::benchmark_data_smithi169_60938_object156852:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fce7a3f5:::benchmark_data_smithi169_60938_object143228:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcec4b10:::benchmark_data_smithi169_60938_object171704:head# 2024-02-12T16:18:28.059 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcee2528:::benchmark_data_smithi169_60938_object155922:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcf0deda:::benchmark_data_smithi169_60938_object164986:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcf1b6e8:::benchmark_data_smithi169_60938_object158061:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcf8f6ee:::benchmark_data_smithi169_60938_object160567:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcfa28d7:::benchmark_data_smithi169_60938_object170288:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcfd6251:::benchmark_data_smithi169_60938_object168408:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcfd937a:::benchmark_data_smithi169_60938_object138511:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fcfee552:::benchmark_data_smithi169_60938_object144945:head# 2024-02-12T16:18:28.060 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd02d92d:::benchmark_data_smithi169_60938_object160873:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd043864:::benchmark_data_smithi169_60938_object144563:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd06933a:::benchmark_data_smithi169_60938_object141061:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd0a24b0:::benchmark_data_smithi169_60938_object165684:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd0c6d98:::benchmark_data_smithi169_60938_object171578:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd0c92a4:::benchmark_data_smithi169_60938_object171024:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd0ca6f1:::benchmark_data_smithi169_60938_object139983:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd0f993c:::benchmark_data_smithi169_60938_object143850:head# 2024-02-12T16:18:28.061 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd11381a:::benchmark_data_smithi169_60938_object146962:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd136ef6:::benchmark_data_smithi169_60938_object138767:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd1489be:::benchmark_data_smithi169_60938_object154984:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd168186:::benchmark_data_smithi169_60938_object154428:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd190649:::benchmark_data_smithi169_60938_object161546:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd19dd06:::benchmark_data_smithi169_60938_object167957:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd1a716f:::benchmark_data_smithi169_60938_object149505:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd1dcc38:::benchmark_data_smithi169_60938_object151721:head# 2024-02-12T16:18:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd1e0acd:::benchmark_data_smithi169_60938_object158248:head# 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd1e9d31:::benchmark_data_smithi169_60938_object156841:head# 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd1f20bd:::benchmark_data_smithi169_60938_object171240:head# 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd253c08:::benchmark_data_smithi169_60938_object155599:head# 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd264f1d:::benchmark_data_smithi169_60938_object164029:head# 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd26a58f:::benchmark_data_smithi169_60938_object156163:head# 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd28d93b:::benchmark_data_smithi169_60938_object160150:head# 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd29fc8f:::benchmark_data_smithi169_60938_object140935:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd2b9541:::benchmark_data_smithi169_60938_object159730:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd2c80ba:::benchmark_data_smithi169_60938_object158940:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd2cd37d:::benchmark_data_smithi169_60938_object158441:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd2d6eb4:::benchmark_data_smithi169_60938_object168079:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd2dde3c:::benchmark_data_smithi169_60938_object171274:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd325aa6:::benchmark_data_smithi169_60938_object151986:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd329de2:::benchmark_data_smithi169_60938_object157951:head# 2024-02-12T16:18:28.064 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd332276:::benchmark_data_smithi169_60938_object143831:head# 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd33e10d:::benchmark_data_smithi169_60938_object149257:head# 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd38c69f:::benchmark_data_smithi169_60938_object149592:head# 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd39bbf5:::benchmark_data_smithi169_60938_object160107:head# 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd3e63cd:::benchmark_data_smithi169_60938_object161524:head# 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd4035f0:::benchmark_data_smithi169_60938_object151540:head# 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd40cab7:::benchmark_data_smithi169_60938_object169347:head# 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.065 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd437324:::benchmark_data_smithi169_60938_object172567:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd45a017:::benchmark_data_smithi169_60938_object169498:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd46048d:::benchmark_data_smithi169_60938_object166971:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd4a7bb8:::benchmark_data_smithi169_60938_object150415:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd4b13ad:::benchmark_data_smithi169_60938_object151971:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd54a07e:::benchmark_data_smithi169_60938_object150224:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd57c839:::benchmark_data_smithi169_60938_object140563:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd5813c9:::benchmark_data_smithi169_60938_object141793:head# 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.066 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd5b773f:::benchmark_data_smithi169_60938_object141550:head# 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd5c8359:::benchmark_data_smithi169_60938_object159552:head# 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd5d5c5e:::benchmark_data_smithi169_60938_object164738:head# 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd5e4413:::benchmark_data_smithi169_60938_object143780:head# 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd623fb7:::benchmark_data_smithi169_60938_object168725:head# 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd63e08e:::benchmark_data_smithi169_60938_object170540:head# 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd659e5b:::benchmark_data_smithi169_60938_object151440:head# 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.067 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd66884f:::benchmark_data_smithi169_60938_object162885:head# 2024-02-12T16:18:28.068 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.083 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd66abf8:::benchmark_data_smithi169_60938_object157652:head# 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd67c9ae:::benchmark_data_smithi169_60938_object160328:head# 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd69c896:::benchmark_data_smithi169_60938_object146836:head# 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd6b2c04:::benchmark_data_smithi169_60938_object155173:head# 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd6b9fcf:::benchmark_data_smithi169_60938_object147410:head# 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd717d31:::benchmark_data_smithi169_60938_object166255:head# 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd71bf89:::benchmark_data_smithi169_60938_object139817:head# 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.084 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd721579:::benchmark_data_smithi169_60938_object163710:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd73dc5e:::benchmark_data_smithi169_60938_object141618:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd78a347:::benchmark_data_smithi169_60938_object164892:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd79a239:::benchmark_data_smithi169_60938_object155974:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd7a57c4:::benchmark_data_smithi169_60938_object148339:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd7f5474:::benchmark_data_smithi169_60938_object166589:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd7faaed:::benchmark_data_smithi169_60938_object162595:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd8111d5:::benchmark_data_smithi169_60938_object141346:head# 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.085 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd817ab8:::benchmark_data_smithi169_60938_object155382:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd881bd8:::benchmark_data_smithi169_60938_object141463:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd8a930d:::benchmark_data_smithi169_60938_object155281:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd8e217b:::benchmark_data_smithi169_60938_object154340:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd8ffa41:::benchmark_data_smithi169_60938_object153515:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd93ffb1:::benchmark_data_smithi169_60938_object165584:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd970925:::benchmark_data_smithi169_60938_object170811:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd99ed87:::benchmark_data_smithi169_60938_object159334:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd9a517b:::benchmark_data_smithi169_60938_object158035:head# 2024-02-12T16:18:28.086 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fd9c5fec:::benchmark_data_smithi169_60938_object168631:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fda11442:::benchmark_data_smithi169_60938_object154031:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fda1539e:::benchmark_data_smithi169_60938_object161540:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fda1bafe:::benchmark_data_smithi169_60938_object159629:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fda2d58f:::benchmark_data_smithi169_60938_object151307:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fda4dd1a:::benchmark_data_smithi169_60938_object156871:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fda92bb3:::benchmark_data_smithi169_60938_object167668:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb3ca44:::benchmark_data_smithi169_60938_object162172:head# 2024-02-12T16:18:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb401b8:::benchmark_data_smithi169_60938_object153665:head# 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb47391:::benchmark_data_smithi169_60938_object164190:head# 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb57069:::benchmark_data_smithi169_60938_object139438:head# 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb5b63c:::benchmark_data_smithi169_60938_object165372:head# 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb66ad0:::benchmark_data_smithi169_60938_object165351:head# 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb6d825:::benchmark_data_smithi169_60938_object171546:head# 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb73ce1:::benchmark_data_smithi169_60938_object168013:head# 2024-02-12T16:18:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdb9f554:::benchmark_data_smithi169_60938_object146461:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdbe1d00:::benchmark_data_smithi169_60938_object158351:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdc15f2f:::benchmark_data_smithi169_60938_object162869:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdc24d67:::benchmark_data_smithi169_60938_object138598:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdc395e0:::benchmark_data_smithi169_60938_object142960:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdc6e2de:::benchmark_data_smithi169_60938_object159263:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdd02033:::benchmark_data_smithi169_60938_object161187:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdd070e0:::benchmark_data_smithi169_60938_object155566:head# 2024-02-12T16:18:28.089 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdd19a60:::benchmark_data_smithi169_60938_object148041:head# 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdd64c40:::benchmark_data_smithi169_60938_object170634:head# 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdd749c4:::benchmark_data_smithi169_60938_object162912:head# 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdd7f9a2:::benchmark_data_smithi169_60938_object160489:head# 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdd99329:::benchmark_data_smithi169_60938_object165294:head# 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fddb323c:::benchmark_data_smithi169_60938_object162019:head# 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fddbe5a8:::benchmark_data_smithi169_60938_object173199:head# 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.090 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fddbffaf:::benchmark_data_smithi169_60938_object170108:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdde86f0:::benchmark_data_smithi169_60938_object168315:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fde1a0f7:::benchmark_data_smithi169_60938_object148615:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fde3d548:::benchmark_data_smithi169_60938_object144412:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fde4dfc7:::benchmark_data_smithi169_60938_object166311:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fde63687:::benchmark_data_smithi169_60938_object147229:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdeaf520:::benchmark_data_smithi169_60938_object147388:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fded0ee7:::benchmark_data_smithi169_60938_object148711:head# 2024-02-12T16:18:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdee221e:::benchmark_data_smithi169_60938_object138871:head# 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf038cc:::benchmark_data_smithi169_60938_object139460:head# 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf0fbf4:::benchmark_data_smithi169_60938_object141148:head# 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf48c61:::benchmark_data_smithi169_60938_object141675:head# 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf5008c:::benchmark_data_smithi169_60938_object151044:head# 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf7daa6:::benchmark_data_smithi169_60938_object159161:head# 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf83826:::benchmark_data_smithi169_60938_object149007:head# 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.092 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf838e2:::benchmark_data_smithi169_60938_object151602:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdf9ec26:::benchmark_data_smithi169_60938_object158794:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdfb406e:::benchmark_data_smithi169_60938_object148351:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdfbe114:::benchmark_data_smithi169_60938_object168546:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdfbf5b9:::benchmark_data_smithi169_60938_object164269:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdfd8bcd:::benchmark_data_smithi169_60938_object154550:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdfe76b6:::benchmark_data_smithi169_60938_object156115:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fdff0fd7:::benchmark_data_smithi169_60938_object168187:head# 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.093 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe03c379:::benchmark_data_smithi169_60938_object153804:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe072a45:::benchmark_data_smithi169_60938_object146556:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe08bb2e:::benchmark_data_smithi169_60938_object161561:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe0b2729:::benchmark_data_smithi169_60938_object168224:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe0b8867:::benchmark_data_smithi169_60938_object163629:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe0ca3db:::benchmark_data_smithi169_60938_object153026:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe0f38f4:::benchmark_data_smithi169_60938_object141806:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe1128b7:::benchmark_data_smithi169_60938_object143386:head# 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.094 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe15491b:::benchmark_data_smithi169_60938_object151315:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe174750:::benchmark_data_smithi169_60938_object163977:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe19be1e:::benchmark_data_smithi169_60938_object173309:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe1a16ff:::benchmark_data_smithi169_60938_object155100:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe1b1b95:::benchmark_data_smithi169_60938_object173004:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe1ce967:::benchmark_data_smithi169_60938_object154525:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe201ab4:::benchmark_data_smithi169_60938_object169929:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe20a69a:::benchmark_data_smithi169_60938_object149964:head# 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.095 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe20e9ad:::benchmark_data_smithi169_60938_object143006:head# 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2296f1:::benchmark_data_smithi169_60938_object155317:head# 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe23d8d2:::benchmark_data_smithi169_60938_object163730:head# 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2a2e49:::benchmark_data_smithi169_60938_object143469:head# 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2cc982:::benchmark_data_smithi169_60938_object150679:head# 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2cd6e1:::benchmark_data_smithi169_60938_object158920:head# 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2dc29a:::benchmark_data_smithi169_60938_object140943:head# 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.096 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2ddf6b:::benchmark_data_smithi169_60938_object162134:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2e4ea6:::benchmark_data_smithi169_60938_object151517:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe2ed7d2:::benchmark_data_smithi169_60938_object164743:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe36c68a:::benchmark_data_smithi169_60938_object165586:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe36e922:::benchmark_data_smithi169_60938_object163779:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe3a2f36:::benchmark_data_smithi169_60938_object141616:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe3c9df3:::benchmark_data_smithi169_60938_object151823:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe43e359:::benchmark_data_smithi169_60938_object156279:head# 2024-02-12T16:18:28.097 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe4704ff:::benchmark_data_smithi169_60938_object161303:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe481b36:::benchmark_data_smithi169_60938_object159097:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe49ff2e:::benchmark_data_smithi169_60938_object166681:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe515027:::benchmark_data_smithi169_60938_object172467:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe52b4db:::benchmark_data_smithi169_60938_object143979:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe565e49:::benchmark_data_smithi169_60938_object139338:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe575ccc:::benchmark_data_smithi169_60938_object150416:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe57ab38:::benchmark_data_smithi169_60938_object151535:head# 2024-02-12T16:18:28.098 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe58ea04:::benchmark_data_smithi169_60938_object148157:head# 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe594b6b:::benchmark_data_smithi169_60938_object169972:head# 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe5cc310:::benchmark_data_smithi169_60938_object159109:head# 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe5ced71:::benchmark_data_smithi169_60938_object169247:head# 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe5d0c9d:::benchmark_data_smithi169_60938_object170949:head# 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe62a33e:::benchmark_data_smithi169_60938_object161113:head# 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe63360d:::benchmark_data_smithi169_60938_object158118:head# 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.099 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe644aa3:::benchmark_data_smithi169_60938_object151135:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe64c4ab:::benchmark_data_smithi169_60938_object148784:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe64cb02:::benchmark_data_smithi169_60938_object154606:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe674ad9:::benchmark_data_smithi169_60938_object143781:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe67f6b7:::benchmark_data_smithi169_60938_object155345:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe684e8e:::benchmark_data_smithi169_60938_object170020:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe6abc00:::benchmark_data_smithi169_60938_object154571:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe6c8fb0:::benchmark_data_smithi169_60938_object145543:head# 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.100 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe6df594:::benchmark_data_smithi169_60938_object163521:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe6e221d:::benchmark_data_smithi169_60938_object172938:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe6ecb7a:::benchmark_data_smithi169_60938_object165823:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe73407b:::benchmark_data_smithi169_60938_object146533:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe738e85:::benchmark_data_smithi169_60938_object138287:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe74ba54:::benchmark_data_smithi169_60938_object168015:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe761009:::benchmark_data_smithi169_60938_object139451:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe7736b8:::benchmark_data_smithi169_60938_object170738:head# 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.101 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe774337:::benchmark_data_smithi169_60938_object163824:head# 2024-02-12T16:18:28.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe787815:::benchmark_data_smithi169_60938_object169463:head# 2024-02-12T16:18:28.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe7ef450:::benchmark_data_smithi169_60938_object170318:head# 2024-02-12T16:18:28.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.102 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe8002cc:::benchmark_data_smithi169_60938_object166408:head# 2024-02-12T16:18:28.102 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:28.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe80712b:::benchmark_data_smithi169_60938_object160140:head# 2024-02-12T16:18:28.119 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.119 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe84e00f:::benchmark_data_smithi169_60938_object173078:head# 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe882ae6:::benchmark_data_smithi169_60938_object143124:head# 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe887ab2:::benchmark_data_smithi169_60938_object155001:head# 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe88c499:::benchmark_data_smithi169_60938_object155434:head# 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe8c0921:::benchmark_data_smithi169_60938_object157421:head# 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe8d78cd:::benchmark_data_smithi169_60938_object163665:head# 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe931705:::benchmark_data_smithi169_60938_object167590:head# 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.120 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe94f5d1:::benchmark_data_smithi169_60938_object167563:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe94fb10:::benchmark_data_smithi169_60938_object142052:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe95bc62:::benchmark_data_smithi169_60938_object151458:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe988cd8:::benchmark_data_smithi169_60938_object163354:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe997c0a:::benchmark_data_smithi169_60938_object160797:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe99b11c:::benchmark_data_smithi169_60938_object157103:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe99b410:::benchmark_data_smithi169_60938_object158901:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe9b0bcc:::benchmark_data_smithi169_60938_object149294:head# 2024-02-12T16:18:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe9bae8a:::benchmark_data_smithi169_60938_object155911:head# 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe9c527b:::benchmark_data_smithi169_60938_object149583:head# 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fe9cd352:::benchmark_data_smithi169_60938_object169384:head# 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fea5747e:::benchmark_data_smithi169_60938_object167022:head# 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fea6fd31:::benchmark_data_smithi169_60938_object171231:head# 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fea9d61b:::benchmark_data_smithi169_60938_object145811:head# 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fead43c2:::benchmark_data_smithi169_60938_object164600:head# 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.122 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feaeb1b1:::benchmark_data_smithi169_60938_object139318:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feb1e8af:::benchmark_data_smithi169_60938_object142146:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feb3dbfa:::benchmark_data_smithi169_60938_object164369:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feb550ee:::benchmark_data_smithi169_60938_object153452:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feb5ba19:::benchmark_data_smithi169_60938_object170157:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feb5f47d:::benchmark_data_smithi169_60938_object169027:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feb885c7:::benchmark_data_smithi169_60938_object170964:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:febb3e19:::benchmark_data_smithi169_60938_object147440:head# 2024-02-12T16:18:28.123 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:febbc125:::benchmark_data_smithi169_60938_object147972:head# 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:febbec3a:::benchmark_data_smithi169_60938_object148881:head# 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:febbfc1d:::benchmark_data_smithi169_60938_object162445:head# 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:febd626e:::benchmark_data_smithi169_60938_object165159:head# 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:febf49fc:::benchmark_data_smithi169_60938_object169447:head# 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fec16b8c:::benchmark_data_smithi169_60938_object160133:head# 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fec17549:::benchmark_data_smithi169_60938_object153998:head# 2024-02-12T16:18:28.124 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fec2b25a:::benchmark_data_smithi169_60938_object158199:head# 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fec6c631:::benchmark_data_smithi169_60938_object154575:head# 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fec9ae98:::benchmark_data_smithi169_60938_object149418:head# 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feca662c:::benchmark_data_smithi169_60938_object166683:head# 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fecb1be3:::benchmark_data_smithi169_60938_object149429:head# 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fece3af1:::benchmark_data_smithi169_60938_object151025:head# 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fecf4e07:::benchmark_data_smithi169_60938_object140523:head# 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.125 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fed22703:::benchmark_data_smithi169_60938_object150655:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fed43ec6:::benchmark_data_smithi169_60938_object164116:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fed467e0:::benchmark_data_smithi169_60938_object144025:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fed4d477:::benchmark_data_smithi169_60938_object152250:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fed78bff:::benchmark_data_smithi169_60938_object167579:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fed804cc:::benchmark_data_smithi169_60938_object146519:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feda3ca2:::benchmark_data_smithi169_60938_object162710:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fedbabac:::benchmark_data_smithi169_60938_object138993:head# 2024-02-12T16:18:28.126 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fedf51ef:::benchmark_data_smithi169_60938_object158335:head# 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fedf6f33:::benchmark_data_smithi169_60938_object140117:head# 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fee2d41a:::benchmark_data_smithi169_60938_object166378:head# 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fee3efb2:::benchmark_data_smithi169_60938_object166270:head# 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fee63525:::benchmark_data_smithi169_60938_object171480:head# 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fee64c65:::benchmark_data_smithi169_60938_object157427:head# 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fee68685:::benchmark_data_smithi169_60938_object161294:head# 2024-02-12T16:18:28.127 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feeca0a5:::benchmark_data_smithi169_60938_object165723:head# 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feee27c3:::benchmark_data_smithi169_60938_object156418:head# 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fef04cc5:::benchmark_data_smithi169_60938_object157494:head# 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fef29978:::benchmark_data_smithi169_60938_object146535:head# 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fef7a14b:::benchmark_data_smithi169_60938_object164727:head# 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fef800d5:::benchmark_data_smithi169_60938_object145754:head# 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fef996e1:::benchmark_data_smithi169_60938_object151485:head# 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.128 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fef99994:::benchmark_data_smithi169_60938_object159003:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fefba263:::benchmark_data_smithi169_60938_object147879:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fefdfd6e:::benchmark_data_smithi169_60938_object145726:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:feff6933:::benchmark_data_smithi169_60938_object157264:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff019c16:::benchmark_data_smithi169_60938_object169544:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff02baea:::benchmark_data_smithi169_60938_object148225:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff03fa79:::benchmark_data_smithi169_60938_object160242:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff07ee47:::benchmark_data_smithi169_60938_object169295:head# 2024-02-12T16:18:28.129 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff090a50:::benchmark_data_smithi169_60938_object144658:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff093519:::benchmark_data_smithi169_60938_object172168:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff09aabf:::benchmark_data_smithi169_60938_object169999:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff09ac7e:::benchmark_data_smithi169_60938_object170846:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff09c0ad:::benchmark_data_smithi169_60938_object157446:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff09ffe6:::benchmark_data_smithi169_60938_object150763:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff0a780e:::benchmark_data_smithi169_60938_object161548:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff0c1e3d:::benchmark_data_smithi169_60938_object168810:head# 2024-02-12T16:18:28.130 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff0ddcc4:::benchmark_data_smithi169_60938_object141920:head# 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff139aa0:::benchmark_data_smithi169_60938_object171227:head# 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff156182:::benchmark_data_smithi169_60938_object156868:head# 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff167ce4:::benchmark_data_smithi169_60938_object170047:head# 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff1e7297:::benchmark_data_smithi169_60938_object166585:head# 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff1ee92e:::benchmark_data_smithi169_60938_object143455:head# 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff1fe42c:::benchmark_data_smithi169_60938_object139957:head# 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff218956:::benchmark_data_smithi169_60938_object144769:head# 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff22dcbb:::benchmark_data_smithi169_60938_object154844:head# 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff250980:::benchmark_data_smithi169_60938_object156704:head# 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff25b2be:::benchmark_data_smithi169_60938_object146995:head# 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff262112:::benchmark_data_smithi169_60938_object139504:head# 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff29b138:::benchmark_data_smithi169_60938_object171675:head# 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff2afa78:::benchmark_data_smithi169_60938_object150961:head# 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.132 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff2b757c:::benchmark_data_smithi169_60938_object171375:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff2c5618:::benchmark_data_smithi169_60938_object145556:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff2d2d30:::benchmark_data_smithi169_60938_object162269:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff39d6d0:::benchmark_data_smithi169_60938_object151891:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff3f142f:::benchmark_data_smithi169_60938_object160811:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff3f84fb:::benchmark_data_smithi169_60938_object152733:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff4003a5:::benchmark_data_smithi169_60938_object165014:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff4ed405:::benchmark_data_smithi169_60938_object168347:head# 2024-02-12T16:18:28.133 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff503ee7:::benchmark_data_smithi169_60938_object156988:head# 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff509ea6:::benchmark_data_smithi169_60938_object144747:head# 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff5310de:::benchmark_data_smithi169_60938_object148648:head# 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff53f4f5:::benchmark_data_smithi169_60938_object156046:head# 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff55b24f:::benchmark_data_smithi169_60938_object156760:head# 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff575b98:::benchmark_data_smithi169_60938_object163905:head# 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff5a897d:::benchmark_data_smithi169_60938_object156108:head# 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.134 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff5c2209:::benchmark_data_smithi169_60938_object146562:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff5c7a0a:::benchmark_data_smithi169_60938_object144947:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff5f2240:::benchmark_data_smithi169_60938_object146136:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff60cdce:::benchmark_data_smithi169_60938_object155657:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff610490:::benchmark_data_smithi169_60938_object147651:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff6119fe:::benchmark_data_smithi169_60938_object144985:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff64c2c9:::benchmark_data_smithi169_60938_object170971:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff6683e9:::benchmark_data_smithi169_60938_object147858:head# 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.135 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff66f419:::benchmark_data_smithi169_60938_object151391:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff6a6739:::benchmark_data_smithi169_60938_object171264:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff6d74b6:::benchmark_data_smithi169_60938_object150869:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff718128:::benchmark_data_smithi169_60938_object156752:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff71da30:::benchmark_data_smithi169_60938_object145140:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff721dec:::benchmark_data_smithi169_60938_object167373:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff752995:::benchmark_data_smithi169_60938_object165061:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff768241:::benchmark_data_smithi169_60938_object156677:head# 2024-02-12T16:18:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff7763e4:::benchmark_data_smithi169_60938_object171569:head# 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff7b82fe:::benchmark_data_smithi169_60938_object164176:head# 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff7e1b07:::benchmark_data_smithi169_60938_object146409:head# 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff7efae0:::benchmark_data_smithi169_60938_object165521:head# 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff81323d:::benchmark_data_smithi169_60938_object152728:head# 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff814094:::benchmark_data_smithi169_60938_object163511:head# 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff8176a6:::benchmark_data_smithi169_60938_object142617:head# 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff8ab3ef:::benchmark_data_smithi169_60938_object154403:head# 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff8b4ec0:::benchmark_data_smithi169_60938_object162490:head# 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff90f733:::benchmark_data_smithi169_60938_object155452:head# 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff91086a:::benchmark_data_smithi169_60938_object152287:head# 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff932c03:::benchmark_data_smithi169_60938_object166383:head# 2024-02-12T16:18:28.138 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff940597:::benchmark_data_smithi169_60938_object151090:head# 2024-02-12T16:18:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff95ac3a:::benchmark_data_smithi169_60938_object170075:head# 2024-02-12T16:18:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff96b71c:::benchmark_data_smithi169_60938_object159275:head# 2024-02-12T16:18:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff9911ca:::benchmark_data_smithi169_60938_object159709:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff99535b:::benchmark_data_smithi169_60938_object152555:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff9a3b7f:::benchmark_data_smithi169_60938_object156549:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff9c08e3:::benchmark_data_smithi169_60938_object149665:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff9c599b:::benchmark_data_smithi169_60938_object140401:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff9c8270:::benchmark_data_smithi169_60938_object172407:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ff9e6ce0:::benchmark_data_smithi169_60938_object145604:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffa2338f:::benchmark_data_smithi169_60938_object157126:head# 2024-02-12T16:18:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffa23b04:::benchmark_data_smithi169_60938_object161350:head# 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffa909a7:::benchmark_data_smithi169_60938_object145393:head# 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffaa4766:::benchmark_data_smithi169_60938_object167514:head# 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffad0ecb:::benchmark_data_smithi169_60938_object146129:head# 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffad6f72:::benchmark_data_smithi169_60938_object149309:head# 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffb10c52:::benchmark_data_smithi169_60938_object169107:head# 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffb3c4e3:::benchmark_data_smithi169_60938_object171861:head# 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffb606b7:::benchmark_data_smithi169_60938_object146753:head# 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffb6f315:::benchmark_data_smithi169_60938_object138509:head# 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffb6f913:::benchmark_data_smithi169_60938_object164655:head# 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffb8afcb:::benchmark_data_smithi169_60938_object157736:head# 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffbadf60:::benchmark_data_smithi169_60938_object169755:head# 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc00ba6:::benchmark_data_smithi169_60938_object164789:head# 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc29121:::benchmark_data_smithi169_60938_object159200:head# 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.148 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc29c23:::benchmark_data_smithi169_60938_object165951:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc45f7c:::benchmark_data_smithi169_60938_object162547:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc5c354:::benchmark_data_smithi169_60938_object166204:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc66b72:::benchmark_data_smithi169_60938_object163142:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc6f171:::benchmark_data_smithi169_60938_object142029:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc7078f:::benchmark_data_smithi169_60938_object173150:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffc99f61:::benchmark_data_smithi169_60938_object167617:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffcb74a3:::benchmark_data_smithi169_60938_object166706:head# 2024-02-12T16:18:28.149 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffcdcba4:::benchmark_data_smithi169_60938_object140031:head# 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffcdd95b:::benchmark_data_smithi169_60938_object165552:head# 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffceaf83:::benchmark_data_smithi169_60938_object160516:head# 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffd675c8:::benchmark_data_smithi169_60938_object171253:head# 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffd74667:::benchmark_data_smithi169_60938_object156891:head# 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffd90856:::benchmark_data_smithi169_60938_object160239:head# 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffded052:::benchmark_data_smithi169_60938_object152036:head# 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.150 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffe01d4b:::benchmark_data_smithi169_60938_object141581:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffe11e3f:::benchmark_data_smithi169_60938_object162685:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffe455e5:::benchmark_data_smithi169_60938_object156022:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffe45f92:::benchmark_data_smithi169_60938_object151979:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffe8eea8:::benchmark_data_smithi169_60938_object159218:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffe9d2f0:::benchmark_data_smithi169_60938_object164821:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffeb371c:::benchmark_data_smithi169_60938_object148230:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffedac52:::benchmark_data_smithi169_60938_object172531:head# 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffee3a07:::benchmark_data_smithi169_60938_object140024:head# 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:ffee8215:::benchmark_data_smithi169_60938_object156153:head# 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fff035e2:::benchmark_data_smithi169_60938_object146825:head# 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fff03f4e:::benchmark_data_smithi169_60938_object150917:head# 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fff99d7f:::benchmark_data_smithi169_60938_object156078:head# 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fff9e285:::benchmark_data_smithi169_60938_object171846:head# 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fffa9af9:::benchmark_data_smithi169_60938_object139816:head# 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.152 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fffbea3f:::benchmark_data_smithi169_60938_object152674:head# 2024-02-12T16:18:28.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.153 INFO:teuthology.orchestra.run.smithi121.stdout:Write #4:fffeb1b6:::benchmark_data_smithi169_60938_object166285:head# 2024-02-12T16:18:28.153 INFO:teuthology.orchestra.run.smithi121.stdout:snapset 0=[]:{} 2024-02-12T16:18:28.153 INFO:teuthology.orchestra.run.smithi121.stdout:write_pg epoch 414 info 4.f( v 414'19284 (360'9200,414'19284] local-lis/les=404/405 n=2124 ec=334/334 lis/c=404/404 les/c/f=405/405/0 sis=404) 2024-02-12T16:18:28.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:28.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:28.205 INFO:teuthology.orchestra.run.smithi121.stdout:Import successful 2024-02-12T16:18:28.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:28.295 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:28.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:28.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:28.436 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:28.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:28.521 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:28.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:28.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:28.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:28.683 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:28.717 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:28.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:28.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:28.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:28.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:28.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:28.975 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:28 smithi123 bash[19119]: cluster 2024-02-12T16:18:27.768831+0000 mgr.y (mgr.14146) 995 : cluster [DBG] pgmap v1328: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:29.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:29.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:29.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:28 smithi169 bash[19604]: cluster 2024-02-12T16:18:27.768831+0000 mgr.y (mgr.14146) 995 : cluster [DBG] pgmap v1328: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:29.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:29.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:28 smithi121 bash[18489]: cluster 2024-02-12T16:18:27.768831+0000 mgr.y (mgr.14146) 995 : cluster [DBG] pgmap v1328: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:29.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:29.283 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:29.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:29.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:29.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:29.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:29.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:29.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:29.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:29.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/exp.4.f.3 2024-02-12T16:18:29.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:29.965 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:29 smithi123 bash[19119]: cluster 2024-02-12T16:18:29.311287+0000 osd.1 (osd.1) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:29.965 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:29 smithi123 bash[19119]: cluster 2024-02-12T16:18:29.311376+0000 osd.1 (osd.1) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:29.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:29.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:30.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:30.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:30.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:30.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:29 smithi121 bash[18489]: cluster 2024-02-12T16:18:29.311287+0000 osd.1 (osd.1) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:30.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:29 smithi121 bash[18489]: cluster 2024-02-12T16:18:29.311376+0000 osd.1 (osd.1) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:30.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:30.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:30.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:29 smithi169 bash[19604]: cluster 2024-02-12T16:18:29.311287+0000 osd.1 (osd.1) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:29 smithi169 bash[19604]: cluster 2024-02-12T16:18:29.311376+0000 osd.1 (osd.1) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:30.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:30.402 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:30.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:30.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:30.541 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:30.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:30.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:30.684 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:30.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:30.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:30.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:30.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:30.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:30.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:30.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:31.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:31.108 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:31.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:31.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:30 smithi123 bash[19119]: cluster 2024-02-12T16:18:29.769725+0000 mgr.y (mgr.14146) 996 : cluster [DBG] pgmap v1329: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:31.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:30 smithi123 bash[19119]: cluster 2024-02-12T16:18:29.871244+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e434: 12 total, 11 up, 11 in 2024-02-12T16:18:31.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:30 smithi123 bash[19119]: cluster 2024-02-12T16:18:30.455500+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e435: 12 total, 11 up, 11 in 2024-02-12T16:18:31.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:30 smithi121 bash[18489]: cluster 2024-02-12T16:18:29.769725+0000 mgr.y (mgr.14146) 996 : cluster [DBG] pgmap v1329: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:30 smithi121 bash[18489]: cluster 2024-02-12T16:18:29.871244+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e434: 12 total, 11 up, 11 in 2024-02-12T16:18:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:30 smithi121 bash[18489]: cluster 2024-02-12T16:18:30.455500+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e435: 12 total, 11 up, 11 in 2024-02-12T16:18:31.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:31.251 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:31.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:30 smithi169 bash[19604]: cluster 2024-02-12T16:18:29.769725+0000 mgr.y (mgr.14146) 996 : cluster [DBG] pgmap v1329: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:30 smithi169 bash[19604]: cluster 2024-02-12T16:18:29.871244+0000 mon.a (mon.0) 1594 : cluster [DBG] osdmap e434: 12 total, 11 up, 11 in 2024-02-12T16:18:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:30 smithi169 bash[19604]: cluster 2024-02-12T16:18:30.455500+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e435: 12 total, 11 up, 11 in 2024-02-12T16:18:31.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:31.391 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:31.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:31.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:31.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:31.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:31.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:31.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:31.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:31.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:31.819 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:31.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:31.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:31.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:31.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:32.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:32.077 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:31 smithi123 bash[19119]: audit 2024-02-12T16:18:30.879623+0000 mon.a (mon.0) 1596 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:18:32.077 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:31 smithi123 bash[19119]: audit 2024-02-12T16:18:30.880105+0000 mon.a (mon.0) 1597 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:18:32.077 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:31 smithi123 bash[19119]: audit 2024-02-12T16:18:30.880480+0000 mon.a (mon.0) 1598 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:18:32.078 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:31 smithi123 bash[19119]: audit 2024-02-12T16:18:30.929683+0000 mon.a (mon.0) 1599 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:18:32.078 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:31 smithi123 bash[19119]: audit 2024-02-12T16:18:31.222223+0000 mon.a (mon.0) 1600 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:18:32.078 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:31 smithi123 bash[19119]: audit 2024-02-12T16:18:31.223839+0000 mon.a (mon.0) 1601 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:18:32.078 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:31 smithi123 bash[19119]: audit 2024-02-12T16:18:31.231466+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:18:32.101 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:32.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:31 smithi121 bash[18489]: audit 2024-02-12T16:18:30.879623+0000 mon.a (mon.0) 1596 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:18:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:31 smithi121 bash[18489]: audit 2024-02-12T16:18:30.880105+0000 mon.a (mon.0) 1597 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:18:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:31 smithi121 bash[18489]: audit 2024-02-12T16:18:30.880480+0000 mon.a (mon.0) 1598 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:18:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:31 smithi121 bash[18489]: audit 2024-02-12T16:18:30.929683+0000 mon.a (mon.0) 1599 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:18:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:31 smithi121 bash[18489]: audit 2024-02-12T16:18:31.222223+0000 mon.a (mon.0) 1600 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:18:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:31 smithi121 bash[18489]: audit 2024-02-12T16:18:31.223839+0000 mon.a (mon.0) 1601 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:18:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:31 smithi121 bash[18489]: audit 2024-02-12T16:18:31.231466+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:18:32.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:32.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:32.281 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:18:32.281 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:18:32.281 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:18:32.281 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:18:32.281 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:18:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:31 smithi169 bash[19604]: audit 2024-02-12T16:18:30.879623+0000 mon.a (mon.0) 1596 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:18:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:31 smithi169 bash[19604]: audit 2024-02-12T16:18:30.880105+0000 mon.a (mon.0) 1597 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:18:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:31 smithi169 bash[19604]: audit 2024-02-12T16:18:30.880480+0000 mon.a (mon.0) 1598 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:18:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:31 smithi169 bash[19604]: audit 2024-02-12T16:18:30.929683+0000 mon.a (mon.0) 1599 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:18:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:31 smithi169 bash[19604]: audit 2024-02-12T16:18:31.222223+0000 mon.a (mon.0) 1600 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:18:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:31 smithi169 bash[19604]: audit 2024-02-12T16:18:31.223839+0000 mon.a (mon.0) 1601 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:18:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:31 smithi169 bash[19604]: audit 2024-02-12T16:18:31.231466+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:18:32.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.368 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:32.375 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:32.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:32.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.515 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:32.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:32.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:32.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:32.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:32.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.798 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:32.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:32.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:32.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:32.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:33.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:33.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:33.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:33.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:33.220 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:33.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:33.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:33.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:33 smithi121 bash[18489]: cluster 2024-02-12T16:18:31.770551+0000 mgr.y (mgr.14146) 997 : cluster [DBG] pgmap v1332: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:33.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:33.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:33.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:33.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:33.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:33.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:33 smithi169 bash[19604]: cluster 2024-02-12T16:18:31.770551+0000 mgr.y (mgr.14146) 997 : cluster [DBG] pgmap v1332: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:33.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:33.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:33 smithi123 bash[19119]: cluster 2024-02-12T16:18:31.770551+0000 mgr.y (mgr.14146) 997 : cluster [DBG] pgmap v1332: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:33.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:33.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:18:33.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:33.800 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:33.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:33.903 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:33.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:33.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:34.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:34.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:34.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:34.152 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:34.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:34.222 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:34.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:34.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:34.364 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:34.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:34.469 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:34.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:34.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:34.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:34.648 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:34.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:34.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:34.790 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:34.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:34.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:34.899 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] out_osds: [8] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:18:34.899 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:18:34.899 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-12T16:18:34.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:18:34.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:34.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:35.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:35.076 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:35.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:35.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:35.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:35.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:35.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:35.356 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:35.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:35.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:35.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:35 smithi121 bash[18489]: cluster 2024-02-12T16:18:33.771551+0000 mgr.y (mgr.14146) 998 : cluster [DBG] pgmap v1333: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:35.494 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:35.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:35 smithi169 bash[19604]: cluster 2024-02-12T16:18:33.771551+0000 mgr.y (mgr.14146) 998 : cluster [DBG] pgmap v1333: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:35.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:35.611 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:35 smithi123 bash[19119]: cluster 2024-02-12T16:18:33.771551+0000 mgr.y (mgr.14146) 998 : cluster [DBG] pgmap v1333: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:35.634 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:35.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:35.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:35.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:35.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:35.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:35.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:35.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:36.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:36.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:36.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:36.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:36.201 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:36.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:36.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:36.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:36.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:36.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:36.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:36.558 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:36 smithi121 bash[18489]: cluster 2024-02-12T16:18:35.772458+0000 mgr.y (mgr.14146) 999 : cluster [DBG] pgmap v1334: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:36.571 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:36.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:36.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:36.687 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:36 smithi169 bash[19604]: cluster 2024-02-12T16:18:35.772458+0000 mgr.y (mgr.14146) 999 : cluster [DBG] pgmap v1334: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:36.713 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:36.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:36.806 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:18:36.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:36.828 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:36 smithi123 bash[19119]: cluster 2024-02-12T16:18:35.772458+0000 mgr.y (mgr.14146) 999 : cluster [DBG] pgmap v1334: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:36.829 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:18:36.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:36.852 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:18:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:36.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:36.878 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:18:36.910 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:18:36.935 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:18:36.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:36.960 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:18:36.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:36.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:36.989 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:18:37.014 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:18:37.046 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:18:37.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:18:37.071 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.071 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.071 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.072 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:18:37.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:37.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:37.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:37.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:37.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:37.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:37.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:37.401 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:37.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:37.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:37.543 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:37.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:37.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:37.659 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:37 smithi169 bash[19604]: audit 2024-02-12T16:18:36.773970+0000 mon.a (mon.0) 1603 : audit [DBG] from='client.? v1:172.21.15.121:0/2532830414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:37.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:37 smithi121 bash[18489]: audit 2024-02-12T16:18:36.773970+0000 mon.a (mon.0) 1603 : audit [DBG] from='client.? v1:172.21.15.121:0/2532830414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:37.684 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:37.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:37.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:37 smithi123 bash[19119]: audit 2024-02-12T16:18:36.773970+0000 mon.a (mon.0) 1603 : audit [DBG] from='client.? v1:172.21.15.121:0/2532830414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:37.823 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:37.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:37.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:37.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:37.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:38.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:38.101 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:18:38.101 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:18:38.103 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:38.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:38.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:38.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:38.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:38.331 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:38.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:38.388 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:38.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:38.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:38.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:38.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:38.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:38.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:38.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:38.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:38.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:38 smithi169 bash[19604]: cluster 2024-02-12T16:18:37.773402+0000 mgr.y (mgr.14146) 1000 : cluster [DBG] pgmap v1335: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:38.808 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:38.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:38.819 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:38 smithi121 bash[18489]: cluster 2024-02-12T16:18:37.773402+0000 mgr.y (mgr.14146) 1000 : cluster [DBG] pgmap v1335: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:38 smithi123 bash[19119]: cluster 2024-02-12T16:18:37.773402+0000 mgr.y (mgr.14146) 1000 : cluster [DBG] pgmap v1335: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:38.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:38.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:38.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:39.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:39.078 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:39.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:39.218 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:39.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:39.357 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:39.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:39.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:39.498 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:18:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:39.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:39.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:39.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:39.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:39 smithi169 bash[19604]: cluster 2024-02-12T16:18:38.463899+0000 mon.a (mon.0) 1604 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:18:39.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:39 smithi169 bash[19604]: audit 2024-02-12T16:18:39.250965+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? v1:172.21.15.121:0/1922774607' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:18:39.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:39.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:39 smithi123 bash[19119]: cluster 2024-02-12T16:18:38.463899+0000 mon.a (mon.0) 1604 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:18:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:39 smithi123 bash[19119]: audit 2024-02-12T16:18:39.250965+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? v1:172.21.15.121:0/1922774607' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:18:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:39 smithi121 bash[18489]: cluster 2024-02-12T16:18:38.463899+0000 mon.a (mon.0) 1604 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:18:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:39 smithi121 bash[18489]: audit 2024-02-12T16:18:39.250965+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? v1:172.21.15.121:0/1922774607' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:18:39.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:39.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:40.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:40.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:18:40.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:40.200 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:40.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:40.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:40.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:40.485 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:40.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:40.627 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:40 smithi121 bash[18489]: cluster 2024-02-12T16:18:39.474055+0000 mon.a (mon.0) 1606 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:40 smithi121 bash[18489]: audit 2024-02-12T16:18:39.487451+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? v1:172.21.15.121:0/1922774607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:18:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:40 smithi121 bash[18489]: cluster 2024-02-12T16:18:39.487597+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e436: 12 total, 11 up, 11 in 2024-02-12T16:18:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:40 smithi121 bash[18489]: cluster 2024-02-12T16:18:39.774090+0000 mgr.y (mgr.14146) 1001 : cluster [DBG] pgmap v1337: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:40 smithi121 bash[18489]: cluster 2024-02-12T16:18:40.460391+0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e437: 12 total, 11 up, 11 in 2024-02-12T16:18:40.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:40.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:40.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:40 smithi169 bash[19604]: cluster 2024-02-12T16:18:39.474055+0000 mon.a (mon.0) 1606 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:40 smithi169 bash[19604]: audit 2024-02-12T16:18:39.487451+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? v1:172.21.15.121:0/1922774607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:18:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:40 smithi169 bash[19604]: cluster 2024-02-12T16:18:39.487597+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e436: 12 total, 11 up, 11 in 2024-02-12T16:18:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:40 smithi169 bash[19604]: cluster 2024-02-12T16:18:39.774090+0000 mgr.y (mgr.14146) 1001 : cluster [DBG] pgmap v1337: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:40 smithi169 bash[19604]: cluster 2024-02-12T16:18:40.460391+0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e437: 12 total, 11 up, 11 in 2024-02-12T16:18:40.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:40.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:40 smithi123 bash[19119]: cluster 2024-02-12T16:18:39.474055+0000 mon.a (mon.0) 1606 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:18:40.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:40 smithi123 bash[19119]: audit 2024-02-12T16:18:39.487451+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? v1:172.21.15.121:0/1922774607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:18:40.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:40 smithi123 bash[19119]: cluster 2024-02-12T16:18:39.487597+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e436: 12 total, 11 up, 11 in 2024-02-12T16:18:40.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:40 smithi123 bash[19119]: cluster 2024-02-12T16:18:39.774090+0000 mgr.y (mgr.14146) 1001 : cluster [DBG] pgmap v1337: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:40.882 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:40 smithi123 bash[19119]: cluster 2024-02-12T16:18:40.460391+0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e437: 12 total, 11 up, 11 in 2024-02-12T16:18:40.911 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:40.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:41.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:41.051 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:41.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:41.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:41.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:41.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:41.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:41.320 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:41.327 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:41.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:41.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:41.466 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:41.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:41.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:41.606 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:41.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:41.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:41.748 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:41.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:41.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:41.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:41.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:42.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:42.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:42.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:42.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:42.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:42.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:42.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:42.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:42.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:18:42.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:42.451 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:42.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:42.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:42.590 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:42.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:42.620 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:18:42.620 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:18:42.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:42.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:42.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:42.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:42.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:42 smithi121 bash[18489]: cluster 2024-02-12T16:18:41.569643+0000 osd.7 (osd.7) 83 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:18:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:42 smithi121 bash[18489]: cluster 2024-02-12T16:18:41.616879+0000 osd.7 (osd.7) 84 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:18:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:42 smithi121 bash[18489]: cluster 2024-02-12T16:18:41.774993+0000 mgr.y (mgr.14146) 1002 : cluster [DBG] pgmap v1339: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:42.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:43.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:43.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:43.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:43.129 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:42 smithi169 bash[19604]: cluster 2024-02-12T16:18:41.569643+0000 osd.7 (osd.7) 83 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:18:43.129 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:42 smithi169 bash[19604]: cluster 2024-02-12T16:18:41.616879+0000 osd.7 (osd.7) 84 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:18:43.129 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:42 smithi169 bash[19604]: cluster 2024-02-12T16:18:41.774993+0000 mgr.y (mgr.14146) 1002 : cluster [DBG] pgmap v1339: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:43.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:42 smithi123 bash[19119]: cluster 2024-02-12T16:18:41.569643+0000 osd.7 (osd.7) 83 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:18:43.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:42 smithi123 bash[19119]: cluster 2024-02-12T16:18:41.616879+0000 osd.7 (osd.7) 84 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:18:43.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:42 smithi123 bash[19119]: cluster 2024-02-12T16:18:41.774993+0000 mgr.y (mgr.14146) 1002 : cluster [DBG] pgmap v1339: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:43.155 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:43.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:43.257 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:43.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:43.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:43.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:43.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:18:43.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:43.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:43.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:43.581 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:43.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:43.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:43.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:43.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:43.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:43.865 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:43.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:43.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:43.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:43 smithi169 bash[19604]: cluster 2024-02-12T16:18:43.579889+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-02-12T16:18:44.004 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:44.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:44.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:44.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:43 smithi123 bash[19119]: cluster 2024-02-12T16:18:43.579889+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-02-12T16:18:44.145 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:44.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:44.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:43 smithi121 bash[18489]: cluster 2024-02-12T16:18:43.579889+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-02-12T16:18:44.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:44.288 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:44.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:44.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:44.428 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:44.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:44.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:44.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:44.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:44.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:44.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:44.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:44.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:44.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:44.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:45.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:44 smithi169 bash[19604]: cluster 2024-02-12T16:18:43.775899+0000 mgr.y (mgr.14146) 1003 : cluster [DBG] pgmap v1340: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:44 smithi169 bash[19604]: cluster 2024-02-12T16:18:43.831676+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:18:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:44 smithi169 bash[19604]: cluster 2024-02-12T16:18:43.831776+0000 mon.a (mon.0) 1611 : cluster [WRN] Health check update: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-12T16:18:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:44 smithi169 bash[19604]: cluster 2024-02-12T16:18:43.881569+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-02-12T16:18:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:44 smithi169 bash[19604]: cluster 2024-02-12T16:18:43.883206+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:44 smithi169 bash[19604]: cluster 2024-02-12T16:18:43.883275+0000 osd.2 (osd.2) 49 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:45.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:44 smithi123 bash[19119]: cluster 2024-02-12T16:18:43.775899+0000 mgr.y (mgr.14146) 1003 : cluster [DBG] pgmap v1340: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:44 smithi123 bash[19119]: cluster 2024-02-12T16:18:43.831676+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:18:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:44 smithi123 bash[19119]: cluster 2024-02-12T16:18:43.831776+0000 mon.a (mon.0) 1611 : cluster [WRN] Health check update: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-12T16:18:45.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:44 smithi123 bash[19119]: cluster 2024-02-12T16:18:43.881569+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-02-12T16:18:45.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:44 smithi123 bash[19119]: cluster 2024-02-12T16:18:43.883206+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:45.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:44 smithi123 bash[19119]: cluster 2024-02-12T16:18:43.883275+0000 osd.2 (osd.2) 49 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:45.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:44 smithi121 bash[18489]: cluster 2024-02-12T16:18:43.775899+0000 mgr.y (mgr.14146) 1003 : cluster [DBG] pgmap v1340: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:44 smithi121 bash[18489]: cluster 2024-02-12T16:18:43.831676+0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:18:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:44 smithi121 bash[18489]: cluster 2024-02-12T16:18:43.831776+0000 mon.a (mon.0) 1611 : cluster [WRN] Health check update: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-02-12T16:18:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:44 smithi121 bash[18489]: cluster 2024-02-12T16:18:43.881569+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-02-12T16:18:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:44 smithi121 bash[18489]: cluster 2024-02-12T16:18:43.883206+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:44 smithi121 bash[18489]: cluster 2024-02-12T16:18:43.883275+0000 osd.2 (osd.2) 49 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:45.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:45.280 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:45.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:45.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:45.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:45.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:45.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:45.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:45.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:45.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:45.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:45.962 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:45.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:45.997 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 439, 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-02-12T15:47:18.273813+0000", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-02-12T16:18:45.455765+0000", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "last_up_change": "2024-02-12T16:17:42.493645+0000", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "last_in_change": "2024-02-12T16:16:59.810319+0000", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_num": 5804032, 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_set": [ 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "nodeep-scrub", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "pglog_hardlimit", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "purged_snapdirs", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_deletes", 2024-02-12T16:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout: "sortbitwise" 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_version": 28, 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_snapshot": "", 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_max": 4, 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd": 12, 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "min_compat_client": "jewel", 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "require_osd_release": "pacific", 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "pools": [ 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 1, 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T15:51:35.886878+0000", 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 21, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 11, 2024-02-12T16:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 21, 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 21, 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 21, 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "410", 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "410", 2024-02-12T16:18:46.001 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:18:46.002 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:18:46.003 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "options": { 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_max": 32, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_min": 1 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_devicehealth": {} 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 4, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "unique_pool_2", 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T16:13:44.637807+0000", 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:18:46.005 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 16, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 16, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 16, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 16, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 16, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.006 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "346", 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:18:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:18:46.008 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "options": {}, 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "rados": {} 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:18:46.009 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0", 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 8, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 334, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6802", 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6804", 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.012 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6803", 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6801/1815232143", 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6802/1815232143", 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6804/1815232143", 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6803/1815232143", 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:18:46.013 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "c6140309-5703-4538-9739-276fbbdaa880", 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.1039276123046875, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 13, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.014 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6805", 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6806", 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.015 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6808", 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6807", 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:18:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6805/2968433670", 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6806/2968433670", 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6808/2968433670", 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6807/2968433670", 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170", 2024-02-12T16:18:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 21, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6809", 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:18:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6810", 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6812", 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6811", 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6809/2158187852", 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6810/2158187852", 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6812/2158187852", 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6811/2158187852", 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd", 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 0, 2024-02-12T16:18:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 26, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 410, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 415, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6813", 2024-02-12T16:18:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6814", 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6816", 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6815", 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6813/1415189366", 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6814/1415189366", 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6816/1415189366", 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6815/1415189366", 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "exists" 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31", 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 32, 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6800", 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6801", 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6803", 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6802", 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6800/3447700123", 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6801/3447700123", 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6803/3447700123", 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6802/3447700123", 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602", 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 39, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6804", 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6805", 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:18:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6807", 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6806", 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6804/961317306", 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6805/961317306", 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6807/961317306", 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6806/961317306", 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0", 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 48, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 420, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6808", 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6809", 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6811", 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6810", 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6808/1377261784", 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6809/1377261784", 2024-02-12T16:18:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6811/1377261784", 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6810/1377261784", 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515", 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 56, 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6812", 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6813", 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6815", 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6814", 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6812/1702613957", 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6813/1702613957", 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6815/1702613957", 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6814/1702613957", 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75", 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 0, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 65, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 310, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6800", 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6801", 2024-02-12T16:18:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6803", 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6802", 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6800/2551075225", 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6801/2551075225", 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6803/2551075225", 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6802/2551075225", 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:18:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15", 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 74, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 418, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6804", 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6805", 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6807", 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6806", 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6804/2339555071", 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6805/2339555071", 2024-02-12T16:18:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6807/2339555071", 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6806/2339555071", 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "91f74dca-503a-45d1-9647-df4804d58515", 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 83, 2024-02-12T16:18:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 410, 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6808", 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6809", 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6811", 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6810", 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6808/391306610", 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6809/391306610", 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6811/391306610", 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6810/391306610", 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf", 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 93, 2024-02-12T16:18:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6812", 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6813", 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6815", 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:18:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6814", 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6812/3696812305", 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6813/3696812305", 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3696812305", 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6814/3696812305", 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_xinfo": [ 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:17:39.968814+0000", 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:29.311380+0000", 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:43.883278+0000", 2024-02-12T16:18:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:16:56.622408+0000", 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 414 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:17:57.663503+0000", 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:22.842696+0000", 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:13.418335+0000", 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:20.915588+0000", 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 65536, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:03.504638+0000", 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:18:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:17:32.579777+0000", 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:17:46.787578+0000", 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:24.280143+0000", 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap": [], 2024-02-12T16:18:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap_items": [], 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_temp": [ 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "4.4", 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_temp": [], 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklist": { 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/2474849789": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/253689008": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/3531151725": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/1778611055": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/28718860": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:18:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/445290372": "2024-02-13T15:47:47.232761+0000" 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "range_blocklist": {}, 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profiles": { 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "default": { 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-failure-domain": "osd", 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "k": "2", 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "m": "1", 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin": "jerasure", 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "technique": "reed_sol_van" 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps_queue": [], 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "new_removed_snaps": [], 2024-02-12T16:18:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "new_purged_snaps": [], 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_node_flags": {}, 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "device_class_flags": {}, 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": { 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:18:46.065 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:18:46.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:46.066 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:45 smithi121 bash[18489]: cluster 2024-02-12T16:18:44.851113+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e438: 12 total, 11 up, 11 in 2024-02-12T16:18:46.066 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:45 smithi121 bash[18489]: cluster 2024-02-12T16:18:45.134008+0000 osd.10 (osd.10) 56 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:18:46.066 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:45 smithi121 bash[18489]: cluster 2024-02-12T16:18:45.461826+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e439: 12 total, 11 up, 11 in 2024-02-12T16:18:46.066 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:45 smithi121 bash[18489]: cluster 2024-02-12T16:18:45.485995+0000 osd.10 (osd.10) 57 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:18:46.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:45 smithi169 bash[19604]: cluster 2024-02-12T16:18:44.851113+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e438: 12 total, 11 up, 11 in 2024-02-12T16:18:46.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:45 smithi169 bash[19604]: cluster 2024-02-12T16:18:45.134008+0000 osd.10 (osd.10) 56 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:18:46.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:45 smithi169 bash[19604]: cluster 2024-02-12T16:18:45.461826+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e439: 12 total, 11 up, 11 in 2024-02-12T16:18:46.078 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:45 smithi169 bash[19604]: cluster 2024-02-12T16:18:45.485995+0000 osd.10 (osd.10) 57 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:18:46.103 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:46.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:45 smithi123 bash[19119]: cluster 2024-02-12T16:18:44.851113+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e438: 12 total, 11 up, 11 in 2024-02-12T16:18:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:45 smithi123 bash[19119]: cluster 2024-02-12T16:18:45.134008+0000 osd.10 (osd.10) 56 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:18:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:45 smithi123 bash[19119]: cluster 2024-02-12T16:18:45.461826+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e439: 12 total, 11 up, 11 in 2024-02-12T16:18:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:45 smithi123 bash[19119]: cluster 2024-02-12T16:18:45.485995+0000 osd.10 (osd.10) 57 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:18:46.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:46.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:46.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:46.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:46.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:46.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:46.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:46.526 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:46.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:46.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:46.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:46.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:46.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:46.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:46.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:46.831 INFO:tasks.thrashosds.thrasher:j is {'epoch': 439, 'fsid': 'eb2349b2-c9bd-11ee-95b9-87774f69a715', 'created': '2024-02-12T15:47:18.273813+0000', 'modified': '2024-02-12T16:18:45.455765+0000', 'last_up_change': '2024-02-12T16:17:42.493645+0000', 'last_in_change': '2024-02-12T16:16:59.810319+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 11, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '410', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '410', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-12T16:13:44.637807+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '346', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 1815232143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 1815232143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 1815232143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 1815232143}]}, 'public_addr': '172.21.15.121:6801/1815232143', 'cluster_addr': '172.21.15.121:6802/1815232143', 'heartbeat_back_addr': '172.21.15.121:6804/1815232143', 'heartbeat_front_addr': '172.21.15.121:6803/1815232143', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c6140309-5703-4538-9739-276fbbdaa880', 'up': 1, 'in': 1, 'weight': 0.1039276123046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2968433670}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2968433670}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 2968433670}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2968433670}]}, 'public_addr': '172.21.15.121:6805/2968433670', 'cluster_addr': '172.21.15.121:6806/2968433670', 'heartbeat_back_addr': '172.21.15.121:6808/2968433670', 'heartbeat_front_addr': '172.21.15.121:6807/2968433670', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 2158187852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 2158187852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 2158187852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 2158187852}]}, 'public_addr': '172.21.15.121:6809/2158187852', 'cluster_addr': '172.21.15.121:6810/2158187852', 'heartbeat_back_addr': '172.21.15.121:6812/2158187852', 'heartbeat_front_addr': '172.21.15.121:6811/2158187852', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5f824526-596d-42e1-8b7e-cd8676ded6dd', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 410, 'down_at': 415, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 1415189366}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 1415189366}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6816', 'nonce': 1415189366}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 1415189366}]}, 'public_addr': '172.21.15.121:6813/1415189366', 'cluster_addr': '172.21.15.121:6814/1415189366', 'heartbeat_back_addr': '172.21.15.121:6816/1415189366', 'heartbeat_front_addr': '172.21.15.121:6815/1415189366', 'state': ['exists']}, {'osd': 4, 'uuid': '14c2f878-99ce-4b51-8d9c-131c1d883b31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6800', 'nonce': 3447700123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6801', 'nonce': 3447700123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6803', 'nonce': 3447700123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6802', 'nonce': 3447700123}]}, 'public_addr': '172.21.15.123:6800/3447700123', 'cluster_addr': '172.21.15.123:6801/3447700123', 'heartbeat_back_addr': '172.21.15.123:6803/3447700123', 'heartbeat_front_addr': '172.21.15.123:6802/3447700123', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6804', 'nonce': 961317306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6805', 'nonce': 961317306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6807', 'nonce': 961317306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6806', 'nonce': 961317306}]}, 'public_addr': '172.21.15.123:6804/961317306', 'cluster_addr': '172.21.15.123:6805/961317306', 'heartbeat_back_addr': '172.21.15.123:6807/961317306', 'heartbeat_front_addr': '172.21.15.123:6806/961317306', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '71e715e6-eac1-4386-927e-51df0cc419c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6808', 'nonce': 1377261784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6809', 'nonce': 1377261784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6811', 'nonce': 1377261784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6810', 'nonce': 1377261784}]}, 'public_addr': '172.21.15.123:6808/1377261784', 'cluster_addr': '172.21.15.123:6809/1377261784', 'heartbeat_back_addr': '172.21.15.123:6811/1377261784', 'heartbeat_front_addr': '172.21.15.123:6810/1377261784', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b419b04f-0c0c-495a-aa9b-25c334159515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6812', 'nonce': 1702613957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6813', 'nonce': 1702613957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6815', 'nonce': 1702613957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6814', 'nonce': 1702613957}]}, 'public_addr': '172.21.15.123:6812/1702613957', 'cluster_addr': '172.21.15.123:6813/1702613957', 'heartbeat_back_addr': '172.21.15.123:6815/1702613957', 'heartbeat_front_addr': '172.21.15.123:6814/1702613957', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1591e7bc-5b86-4401-803d-5325062ffb75', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 310, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6800', 'nonce': 2551075225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 2551075225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 2551075225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6802', 'nonce': 2551075225}]}, 'public_addr': '172.21.15.169:6800/2551075225', 'cluster_addr': '172.21.15.169:6801/2551075225', 'heartbeat_back_addr': '172.21.15.169:6803/2551075225', 'heartbeat_front_addr': '172.21.15.169:6802/2551075225', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '522d61df-8dc1-4915-83fd-f94d0d572c15', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6804', 'nonce': 2339555071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 2339555071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 2339555071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6806', 'nonce': 2339555071}]}, 'public_addr': '172.21.15.169:6804/2339555071', 'cluster_addr': '172.21.15.169:6805/2339555071', 'heartbeat_back_addr': '172.21.15.169:6807/2339555071', 'heartbeat_front_addr': '172.21.15.169:6806/2339555071', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '91f74dca-503a-45d1-9647-df4804d58515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 410, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6808', 'nonce': 391306610}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 391306610}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 391306610}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6810', 'nonce': 391306610}]}, 'public_addr': '172.21.15.169:6808/391306610', 'cluster_addr': '172.21.15.169:6809/391306610', 'heartbeat_back_addr': '172.21.15.169:6811/391306610', 'heartbeat_front_addr': '172.21.15.169:6810/391306610', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '788b4f47-eadf-4601-a5b4-51fb6f04fbdf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 93, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6812', 'nonce': 3696812305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3696812305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3696812305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6814', 'nonce': 3696812305}]}, 'public_addr': '172.21.15.169:6812/3696812305', 'cluster_addr': '172.21.15.169:6813/3696812305', 'heartbeat_back_addr': '172.21.15.169:6815/3696812305', 'heartbeat_front_addr': '172.21.15.169:6814/3696812305', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:17:39.968814+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:29.311380+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:43.883278+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:16:56.622408+0000', 'dead_epoch': 414}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:17:57.663503+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:22.842696+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:13.418335+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:20.915588+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T16:18:03.504638+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:17:32.579777+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:17:46.787578+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:24.280143+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '4.4', 'osds': [2, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.121:0/2474849789': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/253689008': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:6800/3531151725': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:0/1778611055': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:6800/28718860': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/445290372': '2024-02-13T15:47:47.232761+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:18:46.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:18:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:46 smithi121 bash[18489]: cluster 2024-02-12T16:18:45.776565+0000 mgr.y (mgr.14146) 1004 : cluster [DBG] pgmap v1343: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:46 smithi121 bash[18489]: audit 2024-02-12T16:18:45.990493+0000 mon.a (mon.0) 1614 : audit [DBG] from='client.? v1:172.21.15.121:0/3546560371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:18:46.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:46.929 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:46 smithi123 bash[19119]: cluster 2024-02-12T16:18:45.776565+0000 mgr.y (mgr.14146) 1004 : cluster [DBG] pgmap v1343: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:46.929 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:46 smithi123 bash[19119]: audit 2024-02-12T16:18:45.990493+0000 mon.a (mon.0) 1614 : audit [DBG] from='client.? v1:172.21.15.121:0/3546560371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:18:46.951 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:46.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:46 smithi169 bash[19604]: cluster 2024-02-12T16:18:45.776565+0000 mgr.y (mgr.14146) 1004 : cluster [DBG] pgmap v1343: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:47.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:46 smithi169 bash[19604]: audit 2024-02-12T16:18:45.990493+0000 mon.a (mon.0) 1614 : audit [DBG] from='client.? v1:172.21.15.121:0/3546560371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:18:47.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:47.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:47.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:47.227 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:47.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:47.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:47.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:47.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:47.504 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:47.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:47.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:47.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:47.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:47.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:47.792 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:47.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:47.815 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:47.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:47.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:47.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:48.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:48.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:48.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:48.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:48.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:48.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:48.314 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:48.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:48.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:48.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:48.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:48.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:48.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:48.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:48.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:48.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:48.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:48.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:48.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:48.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:48.926 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:48.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:49.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:49.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:48 smithi169 bash[19604]: cluster 2024-02-12T16:18:47.777540+0000 mgr.y (mgr.14146) 1005 : cluster [DBG] pgmap v1344: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:49.068 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:49.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:48 smithi123 bash[19119]: cluster 2024-02-12T16:18:47.777540+0000 mgr.y (mgr.14146) 1005 : cluster [DBG] pgmap v1344: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:48 smithi121 bash[18489]: cluster 2024-02-12T16:18:47.777540+0000 mgr.y (mgr.14146) 1005 : cluster [DBG] pgmap v1344: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:49.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:49.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:49.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:49.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:49.349 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:49.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:49.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:49.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:49.492 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:49.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:49.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:49.614 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:18:49.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:49.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:49.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:18:49.653 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:18:49.676 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:18:49.708 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:18:49.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:49.733 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:18:49.755 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:18:49.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:49.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:49.784 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:18:49.812 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:18:49.837 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.863 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:18:49.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:49.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:49.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:50.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:50.028 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:49 smithi169 bash[19604]: audit 2024-02-12T16:18:49.584491+0000 mon.a (mon.0) 1615 : audit [DBG] from='client.? v1:172.21.15.121:0/745215528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:50.052 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:50.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:50.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:49 smithi123 bash[19119]: audit 2024-02-12T16:18:49.584491+0000 mon.a (mon.0) 1615 : audit [DBG] from='client.? v1:172.21.15.121:0/745215528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:50.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:50.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:49 smithi121 bash[18489]: audit 2024-02-12T16:18:49.584491+0000 mon.a (mon.0) 1615 : audit [DBG] from='client.? v1:172.21.15.121:0/745215528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:18:50.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:50.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:50.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:50.337 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:50.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:50.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:50.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:50.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:50.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:50.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:50.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:50.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:50.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:50.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:50.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:50.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:50.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:50.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:51.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:51.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:51.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:50 smithi123 bash[19119]: cluster 2024-02-12T16:18:49.778260+0000 mgr.y (mgr.14146) 1006 : cluster [DBG] pgmap v1345: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:51.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:51.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:50 smithi121 bash[18489]: cluster 2024-02-12T16:18:49.778260+0000 mgr.y (mgr.14146) 1006 : cluster [DBG] pgmap v1345: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:18:51.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:18:51.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:51.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:51.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:51.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:50 smithi169 bash[19604]: cluster 2024-02-12T16:18:49.778260+0000 mgr.y (mgr.14146) 1006 : cluster [DBG] pgmap v1345: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:51.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:51.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:51.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:51.456 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:51.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:51.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:51.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:51.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:51.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:51.737 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:51.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:51.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:51.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:51.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:51.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:52.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:52.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:52.152 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:52.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:52.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:52.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:52.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:52.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:52.564 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:52.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:52.708 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:52.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:52.850 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:52.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:52.965 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:52 smithi123 bash[19119]: cluster 2024-02-12T16:18:51.179886+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.d scrub starts 2024-02-12T16:18:52.965 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:52 smithi123 bash[19119]: cluster 2024-02-12T16:18:51.665977+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.d scrub ok 2024-02-12T16:18:52.965 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:52 smithi123 bash[19119]: cluster 2024-02-12T16:18:51.779329+0000 mgr.y (mgr.14146) 1007 : cluster [DBG] pgmap v1346: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:52.965 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:52 smithi123 bash[19119]: cluster 2024-02-12T16:18:52.156570+0000 osd.11 (osd.11) 75 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:18:52.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:52.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:53.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:53.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:52 smithi121 bash[18489]: cluster 2024-02-12T16:18:51.179886+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.d scrub starts 2024-02-12T16:18:53.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:52 smithi121 bash[18489]: cluster 2024-02-12T16:18:51.665977+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.d scrub ok 2024-02-12T16:18:53.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:52 smithi121 bash[18489]: cluster 2024-02-12T16:18:51.779329+0000 mgr.y (mgr.14146) 1007 : cluster [DBG] pgmap v1346: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:53.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:52 smithi121 bash[18489]: cluster 2024-02-12T16:18:52.156570+0000 osd.11 (osd.11) 75 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:18:53.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:53.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:53.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:18:53.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:53.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:53.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:53.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:52 smithi169 bash[19604]: cluster 2024-02-12T16:18:51.179886+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.d scrub starts 2024-02-12T16:18:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:52 smithi169 bash[19604]: cluster 2024-02-12T16:18:51.665977+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.d scrub ok 2024-02-12T16:18:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:52 smithi169 bash[19604]: cluster 2024-02-12T16:18:51.779329+0000 mgr.y (mgr.14146) 1007 : cluster [DBG] pgmap v1346: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:52 smithi169 bash[19604]: cluster 2024-02-12T16:18:52.156570+0000 osd.11 (osd.11) 75 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:18:53.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:53.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:53.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:53.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:53.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:53.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:53.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:53.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:53.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:53.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:53.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:53.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:53.905 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:18:53.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:53.958 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:18:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:53.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:18:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:53 smithi169 bash[19604]: cluster 2024-02-12T16:18:52.108183+0000 osd.9 (osd.9) 59 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:18:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:53 smithi169 bash[19604]: cluster 2024-02-12T16:18:52.433646+0000 osd.9 (osd.9) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:54.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:53 smithi169 bash[19604]: cluster 2024-02-12T16:18:52.433691+0000 osd.9 (osd.9) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:54.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:53 smithi169 bash[19604]: cluster 2024-02-12T16:18:52.483978+0000 osd.9 (osd.9) 62 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:18:54.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:53 smithi169 bash[19604]: cluster 2024-02-12T16:18:52.625011+0000 osd.11 (osd.11) 76 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:18:54.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:53 smithi169 bash[19604]: cluster 2024-02-12T16:18:52.883889+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e440: 12 total, 11 up, 11 in 2024-02-12T16:18:54.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:53 smithi169 bash[19604]: audit 2024-02-12T16:18:53.863966+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? v1:172.21.15.121:0/252760988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:18:54.063 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:54.100 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:54.101 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:53 smithi121 bash[18489]: cluster 2024-02-12T16:18:52.108183+0000 osd.9 (osd.9) 59 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:18:54.101 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:53 smithi121 bash[18489]: cluster 2024-02-12T16:18:52.433646+0000 osd.9 (osd.9) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:54.101 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:53 smithi121 bash[18489]: cluster 2024-02-12T16:18:52.433691+0000 osd.9 (osd.9) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:54.101 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:53 smithi121 bash[18489]: cluster 2024-02-12T16:18:52.483978+0000 osd.9 (osd.9) 62 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:18:54.101 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:53 smithi121 bash[18489]: cluster 2024-02-12T16:18:52.625011+0000 osd.11 (osd.11) 76 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:18:54.101 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:53 smithi121 bash[18489]: cluster 2024-02-12T16:18:52.883889+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e440: 12 total, 11 up, 11 in 2024-02-12T16:18:54.101 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:53 smithi121 bash[18489]: audit 2024-02-12T16:18:53.863966+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? v1:172.21.15.121:0/252760988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:18:54.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:54.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:54.129 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:18:54.129 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:18:54.129 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:18:54.129 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:18:54.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:54.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:54.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:54.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:54.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:53 smithi123 bash[19119]: cluster 2024-02-12T16:18:52.108183+0000 osd.9 (osd.9) 59 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:18:54.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:53 smithi123 bash[19119]: cluster 2024-02-12T16:18:52.433646+0000 osd.9 (osd.9) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:18:54.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:53 smithi123 bash[19119]: cluster 2024-02-12T16:18:52.433691+0000 osd.9 (osd.9) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:18:54.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:53 smithi123 bash[19119]: cluster 2024-02-12T16:18:52.483978+0000 osd.9 (osd.9) 62 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:18:54.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:53 smithi123 bash[19119]: cluster 2024-02-12T16:18:52.625011+0000 osd.11 (osd.11) 76 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:18:54.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:53 smithi123 bash[19119]: cluster 2024-02-12T16:18:52.883889+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e440: 12 total, 11 up, 11 in 2024-02-12T16:18:54.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:53 smithi123 bash[19119]: audit 2024-02-12T16:18:53.863966+0000 mon.a (mon.0) 1617 : audit [INF] from='client.? v1:172.21.15.121:0/252760988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:18:54.377 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:54.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:54.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:54.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:54.517 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:54.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:54.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:54.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:54.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:54.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:54.798 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:54.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:54.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:54.939 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:54.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:55.043 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:55.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:54 smithi123 bash[19119]: cluster 2024-02-12T16:18:53.780337+0000 mgr.y (mgr.14146) 1008 : cluster [DBG] pgmap v1348: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:55.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:54 smithi123 bash[19119]: cluster 2024-02-12T16:18:53.886707+0000 mon.a (mon.0) 1618 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:18:55.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:54 smithi123 bash[19119]: audit 2024-02-12T16:18:53.897815+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? v1:172.21.15.121:0/252760988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:18:55.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:54 smithi123 bash[19119]: cluster 2024-02-12T16:18:53.897893+0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e441: 12 total, 11 up, 11 in 2024-02-12T16:18:55.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:54 smithi123 bash[19119]: cluster 2024-02-12T16:18:54.090440+0000 osd.11 (osd.11) 77 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:18:55.080 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:55.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:54 smithi121 bash[18489]: cluster 2024-02-12T16:18:53.780337+0000 mgr.y (mgr.14146) 1008 : cluster [DBG] pgmap v1348: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:54 smithi121 bash[18489]: cluster 2024-02-12T16:18:53.886707+0000 mon.a (mon.0) 1618 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:18:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:54 smithi121 bash[18489]: audit 2024-02-12T16:18:53.897815+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? v1:172.21.15.121:0/252760988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:18:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:54 smithi121 bash[18489]: cluster 2024-02-12T16:18:53.897893+0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e441: 12 total, 11 up, 11 in 2024-02-12T16:18:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:54 smithi121 bash[18489]: cluster 2024-02-12T16:18:54.090440+0000 osd.11 (osd.11) 77 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:18:55.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:55.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:54 smithi169 bash[19604]: cluster 2024-02-12T16:18:53.780337+0000 mgr.y (mgr.14146) 1008 : cluster [DBG] pgmap v1348: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:55.197 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:54 smithi169 bash[19604]: cluster 2024-02-12T16:18:53.886707+0000 mon.a (mon.0) 1618 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:18:55.197 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:54 smithi169 bash[19604]: audit 2024-02-12T16:18:53.897815+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? v1:172.21.15.121:0/252760988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:18:55.197 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:54 smithi169 bash[19604]: cluster 2024-02-12T16:18:53.897893+0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e441: 12 total, 11 up, 11 in 2024-02-12T16:18:55.197 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:54 smithi169 bash[19604]: cluster 2024-02-12T16:18:54.090440+0000 osd.11 (osd.11) 77 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:18:55.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:55.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:55.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:55.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:55.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:55.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:55.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:55.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:55.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:55.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:55.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:55.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:55.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:55.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:55.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:55.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:55.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:18:56.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:56.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:55 smithi121 bash[18489]: cluster 2024-02-12T16:18:54.501885+0000 osd.11 (osd.11) 78 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:18:56.056 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:56.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:56.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:56.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:55 smithi169 bash[19604]: cluster 2024-02-12T16:18:54.501885+0000 osd.11 (osd.11) 78 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:18:56.198 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:56.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:56.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:56.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:56.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:56.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:55 smithi123 bash[19119]: cluster 2024-02-12T16:18:54.501885+0000 osd.11 (osd.11) 78 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:18:56.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:56.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:56.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:56.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:56.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:56.625 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:56.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:56.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:56.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:56.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:56.901 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:56.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:57.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:57.016 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:56 smithi121 bash[18489]: cluster 2024-02-12T16:18:55.781039+0000 mgr.y (mgr.14146) 1009 : cluster [DBG] pgmap v1350: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:57.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:18:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:57.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:18:57.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:57.177 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:57.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:18:57.180 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:56 smithi123 bash[19119]: cluster 2024-02-12T16:18:55.781039+0000 mgr.y (mgr.14146) 1009 : cluster [DBG] pgmap v1350: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:57.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:57.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:57.294 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:56 smithi169 bash[19604]: cluster 2024-02-12T16:18:55.781039+0000 mgr.y (mgr.14146) 1009 : cluster [DBG] pgmap v1350: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:57.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:57.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:57.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:57.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:18:57.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:57.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:18:57.530 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:18:57.532 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:18:57.559 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:57.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:57.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:57.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:57.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:57.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:57.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:57.875 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:18:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:57.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:18:57.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:58.016 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:18:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:58.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:18:58.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:58.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:57 smithi123 bash[19119]: cluster 2024-02-12T16:18:56.703433+0000 osd.7 (osd.7) 85 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:18:58.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:57 smithi123 bash[19119]: cluster 2024-02-12T16:18:56.754762+0000 osd.7 (osd.7) 86 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:18:58.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:58.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:57 smithi121 bash[18489]: cluster 2024-02-12T16:18:56.703433+0000 osd.7 (osd.7) 85 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:18:58.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:57 smithi121 bash[18489]: cluster 2024-02-12T16:18:56.754762+0000 osd.7 (osd.7) 86 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:18:58.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:57 smithi169 bash[19604]: cluster 2024-02-12T16:18:56.703433+0000 osd.7 (osd.7) 85 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:18:58.162 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:57 smithi169 bash[19604]: cluster 2024-02-12T16:18:56.754762+0000 osd.7 (osd.7) 86 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:18:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:58.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:58.236 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1350,"stamp":"2024-02-12T16:18:55.780619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2294087704,"num_objects":35006,"num_object_clones":0,"num_object_copies":70012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8797,"num_objects_misplaced":2246,"num_objects_unfound":0,"num_objects_dirty":35006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":311664,"num_write_kb":11093377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17666,"num_bytes_recovered":1157627904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":66,"acting":67,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":8637504,"kb_used_data":5022312,"kb_used_omap":0,"kb_used_meta":3615104,"kb_avail":1022559168,"statfs":{"total":1055945392128,"available":1047100588032,"internally_reserved":0,"allocated":5142847488,"data_stored":5137305949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3701866496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003777"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":500,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.466288+0000","last_change":"2024-02-12T16:17:56.726824+0000","last_active":"2024-02-12T16:18:45.466288+0000","last_peered":"2024-02-12T16:18:45.466288+0000","last_clean":"2024-02-12T16:18:45.466288+0000","last_became_active":"2024-02-12T16:17:37.447290+0000","last_became_peered":"2024-02-12T16:17:37.447290+0000","last_unstale":"2024-02-12T16:18:45.466288+0000","last_undegraded":"2024-02-12T16:18:45.466288+0000","last_fullsized":"2024-02-12T16:18:45.466288+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:56.726735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:56.726735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":513,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:40.461767+0000","last_change":"2024-02-12T16:17:39.538923+0000","last_active":"2024-02-12T16:18:40.461767+0000","last_peered":"2024-02-12T16:18:40.461767+0000","last_clean":"2024-02-12T16:18:40.461767+0000","last_became_active":"2024-02-12T16:17:37.450871+0000","last_became_peered":"2024-02-12T16:17:37.450871+0000","last_unstale":"2024-02-12T16:18:40.461767+0000","last_undegraded":"2024-02-12T16:18:40.461767+0000","last_fullsized":"2024-02-12T16:18:40.461767+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:39.538747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:39.538747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"416'19460","reported_seq":19594,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.923706+0000","last_change":"2024-02-12T16:17:42.905634+0000","last_active":"2024-02-12T16:18:45.923706+0000","last_peered":"2024-02-12T16:18:45.923706+0000","last_clean":"2024-02-12T16:18:45.923706+0000","last_became_active":"2024-02-12T16:13:45.969639+0000","last_became_peered":"2024-02-12T16:13:45.969639+0000","last_unstale":"2024-02-12T16:18:45.923706+0000","last_undegraded":"2024-02-12T16:18:45.923706+0000","last_fullsized":"2024-02-12T16:18:45.923706+0000","mapping_epoch":334,"log_start":"360'9400","ondisk_log_start":"360'9400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19459","last_scrub_stamp":"2024-02-12T16:17:42.905491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:17:42.905491+0000","log_size":10060,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":145489920,"num_objects":2220,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19460,"num_write_kb":693760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"414'19567","reported_seq":19687,"reported_epoch":437,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:18:40.461900+0000","last_change":"2024-02-12T16:17:43.520446+0000","last_active":"2024-02-12T16:17:42.522571+0000","last_peered":"2024-02-12T16:18:40.461900+0000","last_clean":"2024-02-12T16:17:42.468741+0000","last_became_active":"2024-02-12T16:13:45.788482+0000","last_became_peered":"2024-02-12T16:17:43.520446+0000","last_unstale":"2024-02-12T16:18:40.461900+0000","last_undegraded":"2024-02-12T16:17:43.518138+0000","last_fullsized":"2024-02-12T16:17:43.518031+0000","mapping_epoch":415,"log_start":"360'9500","ondisk_log_start":"360'9500","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'11421","last_scrub_stamp":"2024-02-12T16:15:36.163985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:36.163985+0000","log_size":10067,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":148307968,"num_objects":2263,"num_object_clones":0,"num_object_copies":4526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19567,"num_write_kb":698560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":2263}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":97,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.282053+0000","last_change":"2024-02-12T16:17:56.042539+0000","last_active":"2024-02-12T16:18:41.282053+0000","last_peered":"2024-02-12T16:18:41.282053+0000","last_clean":"2024-02-12T16:18:41.282053+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:18:41.282053+0000","last_undegraded":"2024-02-12T16:18:41.282053+0000","last_fullsized":"2024-02-12T16:18:41.282053+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:56.042425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:56.042425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1","version":"415'19324","reported_seq":19444,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.466364+0000","last_change":"2024-02-12T16:17:45.985288+0000","last_active":"2024-02-12T16:18:45.466364+0000","last_peered":"2024-02-12T16:18:45.466364+0000","last_clean":"2024-02-12T16:18:45.466364+0000","last_became_active":"2024-02-12T16:13:45.969700+0000","last_became_peered":"2024-02-12T16:13:45.969700+0000","last_unstale":"2024-02-12T16:18:45.466364+0000","last_undegraded":"2024-02-12T16:18:45.466364+0000","last_fullsized":"2024-02-12T16:18:45.466364+0000","mapping_epoch":334,"log_start":"360'9300","ondisk_log_start":"360'9300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19324","last_scrub_stamp":"2024-02-12T16:17:45.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:17:45.985205+0000","log_size":10024,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142082048,"num_objects":2168,"num_object_clones":0,"num_object_copies":4336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19324,"num_write_kb":687744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":501,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.466470+0000","last_change":"2024-02-12T16:17:48.697084+0000","last_active":"2024-02-12T16:18:45.466470+0000","last_peered":"2024-02-12T16:18:45.466470+0000","last_clean":"2024-02-12T16:18:45.466470+0000","last_became_active":"2024-02-12T16:17:37.446399+0000","last_became_peered":"2024-02-12T16:17:37.446399+0000","last_unstale":"2024-02-12T16:18:45.466470+0000","last_undegraded":"2024-02-12T16:18:45.466470+0000","last_fullsized":"2024-02-12T16:18:45.466470+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:48.697006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:48.697006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"416'19181","reported_seq":19308,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.281981+0000","last_change":"2024-02-12T16:17:49.482254+0000","last_active":"2024-02-12T16:18:41.281981+0000","last_peered":"2024-02-12T16:18:41.281981+0000","last_clean":"2024-02-12T16:18:41.281981+0000","last_became_active":"2024-02-12T16:13:45.969665+0000","last_became_peered":"2024-02-12T16:13:45.969665+0000","last_unstale":"2024-02-12T16:18:41.281981+0000","last_undegraded":"2024-02-12T16:18:41.281981+0000","last_fullsized":"2024-02-12T16:18:41.281981+0000","mapping_epoch":334,"log_start":"359'9100","ondisk_log_start":"359'9100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19181","last_scrub_stamp":"2024-02-12T16:17:49.482163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:17:49.482163+0000","log_size":10081,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140050432,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19181,"num_write_kb":682176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":97,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.923830+0000","last_change":"2024-02-12T16:17:52.527597+0000","last_active":"2024-02-12T16:18:45.923830+0000","last_peered":"2024-02-12T16:18:45.923830+0000","last_clean":"2024-02-12T16:18:45.923830+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:18:45.923830+0000","last_undegraded":"2024-02-12T16:18:45.923830+0000","last_fullsized":"2024-02-12T16:18:45.923830+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:52.527491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:52.527491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"416'19327","reported_seq":19452,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-02-12T16:18:54.502051+0000","last_change":"2024-02-12T16:18:54.502051+0000","last_active":"2024-02-12T16:18:54.502051+0000","last_peered":"2024-02-12T16:18:54.502051+0000","last_clean":"2024-02-12T16:18:54.502051+0000","last_became_active":"2024-02-12T16:13:45.794673+0000","last_became_peered":"2024-02-12T16:13:45.794673+0000","last_unstale":"2024-02-12T16:18:54.502051+0000","last_undegraded":"2024-02-12T16:18:54.502051+0000","last_fullsized":"2024-02-12T16:18:54.502051+0000","mapping_epoch":334,"log_start":"360'9300","ondisk_log_start":"360'9300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19327","last_scrub_stamp":"2024-02-12T16:18:54.501897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:18:54.501897+0000","log_size":10027,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142409728,"num_objects":2173,"num_object_clones":0,"num_object_copies":4346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19327,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":93,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.282111+0000","last_change":"2024-02-12T16:18:00.122902+0000","last_active":"2024-02-12T16:18:41.282111+0000","last_peered":"2024-02-12T16:18:41.282111+0000","last_clean":"2024-02-12T16:18:41.282111+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:18:41.282111+0000","last_undegraded":"2024-02-12T16:18:41.282111+0000","last_fullsized":"2024-02-12T16:18:41.282111+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:18:00.122706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:18:00.122706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"416'19636","reported_seq":21132,"reported_epoch":438,"state":"active+clean+remapped","last_fresh":"2024-02-12T16:18:44.848522+0000","last_change":"2024-02-12T16:17:52.993027+0000","last_active":"2024-02-12T16:18:44.848522+0000","last_peered":"2024-02-12T16:18:44.848522+0000","last_clean":"2024-02-12T16:18:44.848522+0000","last_became_active":"2024-02-12T16:17:43.511096+0000","last_became_peered":"2024-02-12T16:17:43.511096+0000","last_unstale":"2024-02-12T16:18:44.848522+0000","last_undegraded":"2024-02-12T16:18:44.848522+0000","last_fullsized":"2024-02-12T16:18:44.848522+0000","mapping_epoch":415,"log_start":"361'9600","ondisk_log_start":"361'9600","created":334,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19636","last_scrub_stamp":"2024-02-12T16:17:52.992887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:17:52.992887+0000","log_size":10036,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":147193856,"num_objects":2246,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2246,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19636,"num_write_kb":700224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":44498944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":2246}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":93,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.281903+0000","last_change":"2024-02-12T16:17:59.107023+0000","last_active":"2024-02-12T16:18:41.281903+0000","last_peered":"2024-02-12T16:18:41.281903+0000","last_clean":"2024-02-12T16:18:41.281903+0000","last_became_active":"2024-02-12T16:17:37.450884+0000","last_became_peered":"2024-02-12T16:17:37.450884+0000","last_unstale":"2024-02-12T16:18:41.281903+0000","last_undegraded":"2024-02-12T16:18:41.281903+0000","last_fullsized":"2024-02-12T16:18:41.281903+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:59.106914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:59.106914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"416'22178","reported_seq":26665,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.281990+0000","last_change":"2024-02-12T16:17:49.613301+0000","last_active":"2024-02-12T16:18:41.281990+0000","last_peered":"2024-02-12T16:18:41.281990+0000","last_clean":"2024-02-12T16:18:41.281990+0000","last_became_active":"2024-02-12T16:17:49.564271+0000","last_became_peered":"2024-02-12T16:17:49.564271+0000","last_unstale":"2024-02-12T16:18:41.281990+0000","last_undegraded":"2024-02-12T16:18:41.281990+0000","last_fullsized":"2024-02-12T16:18:41.281990+0000","mapping_epoch":418,"log_start":"375'12101","ondisk_log_start":"375'12101","created":334,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'22178","last_scrub_stamp":"2024-02-12T16:17:48.389736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:17:48.389736+0000","log_size":10077,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142082048,"num_objects":2168,"num_object_clones":0,"num_object_copies":4336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19672,"num_write_kb":698880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2168,"num_bytes_recovered":142082048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":66,"reported_epoch":439,"state":"undersized+peered","last_fresh":"2024-02-12T16:18:45.923523+0000","last_change":"2024-02-12T16:17:43.517378+0000","last_active":"2024-02-12T16:17:36.432068+0000","last_peered":"2024-02-12T16:18:45.923523+0000","last_clean":"2024-02-12T16:17:34.859280+0000","last_became_active":"2024-02-12T16:16:12.802102+0000","last_became_peered":"2024-02-12T16:17:43.517378+0000","last_unstale":"2024-02-12T16:18:45.923523+0000","last_undegraded":"2024-02-12T16:18:45.923523+0000","last_fullsized":"2024-02-12T16:17:43.515557+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:51.659214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:51.659214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"415'19599","reported_seq":29164,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.486252+0000","last_change":"2024-02-12T16:18:45.486177+0000","last_active":"2024-02-12T16:18:45.486252+0000","last_peered":"2024-02-12T16:18:45.486252+0000","last_clean":"2024-02-12T16:18:45.486252+0000","last_became_active":"2024-02-12T16:17:17.433883+0000","last_became_peered":"2024-02-12T16:17:17.433883+0000","last_unstale":"2024-02-12T16:18:45.486252+0000","last_undegraded":"2024-02-12T16:18:45.486252+0000","last_fullsized":"2024-02-12T16:18:45.486252+0000","mapping_epoch":403,"log_start":"360'9500","ondisk_log_start":"360'9500","created":334,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19599","last_scrub_stamp":"2024-02-12T16:18:45.486007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:18:45.486007+0000","log_size":10099,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142147584,"num_objects":2169,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19599,"num_write_kb":696576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4710,"num_bytes_recovered":308609024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":502,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:40.461835+0000","last_change":"2024-02-12T16:17:41.533646+0000","last_active":"2024-02-12T16:18:40.461835+0000","last_peered":"2024-02-12T16:18:40.461835+0000","last_clean":"2024-02-12T16:18:40.461835+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:18:40.461835+0000","last_undegraded":"2024-02-12T16:18:40.461835+0000","last_fullsized":"2024-02-12T16:18:40.461835+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:41.533543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:41.533543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"416'19351","reported_seq":19485,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:52.625182+0000","last_change":"2024-02-12T16:18:52.625147+0000","last_active":"2024-02-12T16:18:52.625182+0000","last_peered":"2024-02-12T16:18:52.625182+0000","last_clean":"2024-02-12T16:18:52.625182+0000","last_became_active":"2024-02-12T16:13:45.793378+0000","last_became_peered":"2024-02-12T16:13:45.793378+0000","last_unstale":"2024-02-12T16:18:52.625182+0000","last_undegraded":"2024-02-12T16:18:52.625182+0000","last_fullsized":"2024-02-12T16:18:52.625182+0000","mapping_epoch":334,"log_start":"360'9300","ondisk_log_start":"360'9300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19351","last_scrub_stamp":"2024-02-12T16:18:52.625021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:18:52.625021+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140050432,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19351,"num_write_kb":687616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":92,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.281946+0000","last_change":"2024-02-12T16:18:03.127832+0000","last_active":"2024-02-12T16:18:41.281946+0000","last_peered":"2024-02-12T16:18:41.281946+0000","last_clean":"2024-02-12T16:18:41.281946+0000","last_became_active":"2024-02-12T16:17:37.453541+0000","last_became_peered":"2024-02-12T16:17:37.453541+0000","last_unstale":"2024-02-12T16:18:41.281946+0000","last_undegraded":"2024-02-12T16:18:41.281946+0000","last_fullsized":"2024-02-12T16:18:41.281946+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:18:03.127732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:18:03.127732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.9","version":"414'19577","reported_seq":26258,"reported_epoch":439,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:18:45.466088+0000","last_change":"2024-02-12T16:17:43.513141+0000","last_active":"2024-02-12T16:17:42.510526+0000","last_peered":"2024-02-12T16:18:45.466088+0000","last_clean":"2024-02-12T16:17:42.490992+0000","last_became_active":"2024-02-12T16:17:39.479526+0000","last_became_peered":"2024-02-12T16:17:43.513141+0000","last_unstale":"2024-02-12T16:18:45.466088+0000","last_undegraded":"2024-02-12T16:17:43.512450+0000","last_fullsized":"2024-02-12T16:17:43.512256+0000","mapping_epoch":415,"log_start":"360'9500","ondisk_log_start":"360'9500","created":334,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15658","last_scrub_stamp":"2024-02-12T16:16:37.730742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:16:37.730742+0000","log_size":10077,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":143065088,"num_objects":2183,"num_object_clones":0,"num_object_copies":4366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19577,"num_write_kb":696320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3267,"num_bytes_recovered":214040576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11","1"],"object_location_counts":[{"shards":"11","objects":2183}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":500,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.466427+0000","last_change":"2024-02-12T16:17:51.755743+0000","last_active":"2024-02-12T16:18:45.466427+0000","last_peered":"2024-02-12T16:18:45.466427+0000","last_clean":"2024-02-12T16:18:45.466427+0000","last_became_active":"2024-02-12T16:17:37.446899+0000","last_became_peered":"2024-02-12T16:17:37.446899+0000","last_unstale":"2024-02-12T16:18:45.466427+0000","last_undegraded":"2024-02-12T16:18:45.466427+0000","last_fullsized":"2024-02-12T16:18:45.466427+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:51.755645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:51.755645+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.a","version":"415'19694","reported_seq":24489,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.923704+0000","last_change":"2024-02-12T16:18:02.942235+0000","last_active":"2024-02-12T16:18:45.923704+0000","last_peered":"2024-02-12T16:18:45.923704+0000","last_clean":"2024-02-12T16:18:45.923704+0000","last_became_active":"2024-02-12T16:17:51.453423+0000","last_became_peered":"2024-02-12T16:17:51.453423+0000","last_unstale":"2024-02-12T16:18:45.923704+0000","last_undegraded":"2024-02-12T16:18:45.923704+0000","last_fullsized":"2024-02-12T16:18:45.923704+0000","mapping_epoch":420,"log_start":"360'9600","ondisk_log_start":"360'9600","created":334,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19694","last_scrub_stamp":"2024-02-12T16:18:02.942003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:18:02.942003+0000","log_size":10094,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":151781376,"num_objects":2316,"num_object_clones":0,"num_object_copies":4632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19694,"num_write_kb":704320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":151781376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":94,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.923782+0000","last_change":"2024-02-12T16:17:53.531332+0000","last_active":"2024-02-12T16:18:45.923782+0000","last_peered":"2024-02-12T16:18:45.923782+0000","last_clean":"2024-02-12T16:18:45.923782+0000","last_became_active":"2024-02-12T16:17:37.452550+0000","last_became_peered":"2024-02-12T16:17:37.452550+0000","last_unstale":"2024-02-12T16:18:45.923782+0000","last_undegraded":"2024-02-12T16:18:45.923782+0000","last_fullsized":"2024-02-12T16:18:45.923782+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:53.531203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:53.531203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"416'19630","reported_seq":19758,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-02-12T16:18:52.882534+0000","last_change":"2024-02-12T16:18:43.881713+0000","last_active":"2024-02-12T16:18:52.882534+0000","last_peered":"2024-02-12T16:18:52.882534+0000","last_clean":"2024-02-12T16:18:52.882534+0000","last_became_active":"2024-02-12T16:13:45.790415+0000","last_became_peered":"2024-02-12T16:13:45.790415+0000","last_unstale":"2024-02-12T16:18:52.882534+0000","last_undegraded":"2024-02-12T16:18:52.882534+0000","last_fullsized":"2024-02-12T16:18:52.882534+0000","mapping_epoch":334,"log_start":"360'9600","ondisk_log_start":"360'9600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19630","last_scrub_stamp":"2024-02-12T16:18:43.881579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:18:43.881579+0000","log_size":10030,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142344192,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19630,"num_write_kb":697664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":92,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.282082+0000","last_change":"2024-02-12T16:18:06.137534+0000","last_active":"2024-02-12T16:18:41.282082+0000","last_peered":"2024-02-12T16:18:41.282082+0000","last_clean":"2024-02-12T16:18:41.282082+0000","last_became_active":"2024-02-12T16:17:37.456743+0000","last_became_peered":"2024-02-12T16:17:37.456743+0000","last_unstale":"2024-02-12T16:18:41.282082+0000","last_undegraded":"2024-02-12T16:18:41.282082+0000","last_fullsized":"2024-02-12T16:18:41.282082+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:18:06.137346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:18:06.137346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.c","version":"414'19508","reported_seq":19636,"reported_epoch":439,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:18:45.923122+0000","last_change":"2024-02-12T16:17:43.516008+0000","last_active":"2024-02-12T16:17:42.505394+0000","last_peered":"2024-02-12T16:18:45.923122+0000","last_clean":"2024-02-12T16:17:42.470323+0000","last_became_active":"2024-02-12T16:13:45.791545+0000","last_became_peered":"2024-02-12T16:17:43.516008+0000","last_unstale":"2024-02-12T16:18:45.923122+0000","last_undegraded":"2024-02-12T16:17:43.515377+0000","last_fullsized":"2024-02-12T16:17:43.515057+0000","mapping_epoch":415,"log_start":"361'9500","ondisk_log_start":"361'9500","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'11689","last_scrub_stamp":"2024-02-12T16:15:40.640284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:40.640284+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":146014208,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19508,"num_write_kb":695552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":2228}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":503,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.617081+0000","last_change":"2024-02-12T16:18:41.617036+0000","last_active":"2024-02-12T16:18:41.617081+0000","last_peered":"2024-02-12T16:18:41.617081+0000","last_clean":"2024-02-12T16:18:41.617081+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:18:41.617081+0000","last_undegraded":"2024-02-12T16:18:41.617081+0000","last_fullsized":"2024-02-12T16:18:41.617081+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:18:41.616891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:18:41.616891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"416'19402","reported_seq":19526,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:51.666164+0000","last_change":"2024-02-12T16:18:51.666118+0000","last_active":"2024-02-12T16:18:51.666164+0000","last_peered":"2024-02-12T16:18:51.666164+0000","last_clean":"2024-02-12T16:18:51.666164+0000","last_became_active":"2024-02-12T16:13:45.793796+0000","last_became_peered":"2024-02-12T16:13:45.793796+0000","last_unstale":"2024-02-12T16:18:51.666164+0000","last_undegraded":"2024-02-12T16:18:51.666164+0000","last_fullsized":"2024-02-12T16:18:51.666164+0000","mapping_epoch":334,"log_start":"360'9400","ondisk_log_start":"360'9400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19402","last_scrub_stamp":"2024-02-12T16:18:51.665982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:18:51.665982+0000","log_size":10002,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140574744,"num_objects":2146,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19402,"num_write_kb":689473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":92,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.609026+0000","last_change":"2024-02-12T16:17:54.863865+0000","last_active":"2024-02-12T16:18:45.609026+0000","last_peered":"2024-02-12T16:18:45.609026+0000","last_clean":"2024-02-12T16:18:45.609026+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:18:45.609026+0000","last_undegraded":"2024-02-12T16:18:45.609026+0000","last_fullsized":"2024-02-12T16:18:45.609026+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:54.863750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:54.863750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"416'19451","reported_seq":19576,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-02-12T16:18:45.923223+0000","last_change":"2024-02-12T16:17:50.873120+0000","last_active":"2024-02-12T16:18:45.923223+0000","last_peered":"2024-02-12T16:18:45.923223+0000","last_clean":"2024-02-12T16:18:45.923223+0000","last_became_active":"2024-02-12T16:13:45.970126+0000","last_became_peered":"2024-02-12T16:13:45.970126+0000","last_unstale":"2024-02-12T16:18:45.923223+0000","last_undegraded":"2024-02-12T16:18:45.923223+0000","last_fullsized":"2024-02-12T16:18:45.923223+0000","mapping_epoch":334,"log_start":"360'9400","ondisk_log_start":"360'9400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19451","last_scrub_stamp":"2024-02-12T16:17:50.872999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:17:50.872999+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":141361152,"num_objects":2157,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19451,"num_write_kb":691456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":65,"reported_epoch":439,"state":"undersized+peered","last_fresh":"2024-02-12T16:18:45.923726+0000","last_change":"2024-02-12T16:17:43.520897+0000","last_active":"2024-02-12T16:17:36.432068+0000","last_peered":"2024-02-12T16:18:45.923726+0000","last_clean":"2024-02-12T16:17:34.859280+0000","last_became_active":"2024-02-12T16:16:12.802102+0000","last_became_peered":"2024-02-12T16:17:43.520897+0000","last_unstale":"2024-02-12T16:18:45.923726+0000","last_undegraded":"2024-02-12T16:18:45.923726+0000","last_fullsized":"2024-02-12T16:17:43.515790+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:51.659214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:51.659214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"414'19285","reported_seq":28476,"reported_epoch":439,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:18:45.466140+0000","last_change":"2024-02-12T16:17:43.516082+0000","last_active":"2024-02-12T16:17:42.510665+0000","last_peered":"2024-02-12T16:18:45.466140+0000","last_clean":"2024-02-12T16:17:42.498681+0000","last_became_active":"2024-02-12T16:17:18.432226+0000","last_became_peered":"2024-02-12T16:17:43.516082+0000","last_unstale":"2024-02-12T16:18:45.466140+0000","last_undegraded":"2024-02-12T16:17:43.512779+0000","last_fullsized":"2024-02-12T16:17:43.512406+0000","mapping_epoch":415,"log_start":"360'9200","ondisk_log_start":"360'9200","created":334,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'12227","last_scrub_stamp":"2024-02-12T16:15:49.878411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:49.878411+0000","log_size":10085,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139132928,"num_objects":2123,"num_object_clones":0,"num_object_copies":4246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19285,"num_write_kb":685056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4526,"num_bytes_recovered":296615936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11","1"],"object_location_counts":[{"shards":"11","objects":2123}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":86,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.282235+0000","last_change":"2024-02-12T16:18:05.090725+0000","last_active":"2024-02-12T16:18:41.282235+0000","last_peered":"2024-02-12T16:18:41.282235+0000","last_clean":"2024-02-12T16:18:41.282235+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:18:41.282235+0000","last_undegraded":"2024-02-12T16:18:41.282235+0000","last_fullsized":"2024-02-12T16:18:41.282235+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:18:05.090514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:18:05.090514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":512,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:40.461332+0000","last_change":"2024-02-12T16:17:43.513233+0000","last_active":"2024-02-12T16:18:40.461332+0000","last_peered":"2024-02-12T16:18:40.461332+0000","last_clean":"2024-02-12T16:18:40.461332+0000","last_became_active":"2024-02-12T16:17:37.450842+0000","last_became_peered":"2024-02-12T16:17:37.450842+0000","last_unstale":"2024-02-12T16:18:40.461332+0000","last_undegraded":"2024-02-12T16:18:40.461332+0000","last_fullsized":"2024-02-12T16:18:40.461332+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:43.513125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:43.513125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":92,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:41.281802+0000","last_change":"2024-02-12T16:18:04.114123+0000","last_active":"2024-02-12T16:18:41.281802+0000","last_peered":"2024-02-12T16:18:41.281802+0000","last_clean":"2024-02-12T16:18:41.281802+0000","last_became_active":"2024-02-12T16:17:37.451769+0000","last_became_peered":"2024-02-12T16:17:37.451769+0000","last_unstale":"2024-02-12T16:18:41.281802+0000","last_undegraded":"2024-02-12T16:18:41.281802+0000","last_fullsized":"2024-02-12T16:18:41.281802+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:18:04.114027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:18:04.114027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":501,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-02-12T16:18:40.461679+0000","last_change":"2024-02-12T16:17:48.599553+0000","last_active":"2024-02-12T16:18:40.461679+0000","last_peered":"2024-02-12T16:18:40.461679+0000","last_clean":"2024-02-12T16:18:40.461679+0000","last_became_active":"2024-02-12T16:17:37.459545+0000","last_became_peered":"2024-02-12T16:17:37.459545+0000","last_unstale":"2024-02-12T16:18:40.461679+0000","last_undegraded":"2024-02-12T16:18:40.461679+0000","last_fullsized":"2024-02-12T16:18:40.461679+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:17:48.599461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:17:48.599461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":65,"reported_epoch":439,"state":"undersized+peered","last_fresh":"2024-02-12T16:18:45.923505+0000","last_change":"2024-02-12T16:17:43.520569+0000","last_active":"2024-02-12T16:17:36.432068+0000","last_peered":"2024-02-12T16:18:45.923505+0000","last_clean":"2024-02-12T16:17:34.859280+0000","last_became_active":"2024-02-12T16:16:12.802102+0000","last_became_peered":"2024-02-12T16:17:43.520569+0000","last_unstale":"2024-02-12T16:18:45.923505+0000","last_undegraded":"2024-02-12T16:18:45.923505+0000","last_fullsized":"2024-02-12T16:17:43.515534+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:51.659214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:51.659214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2294087704,"num_objects":35006,"num_object_clones":0,"num_object_copies":70012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8797,"num_objects_misplaced":2246,"num_objects_unfound":0,"num_objects_dirty":35006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":311664,"num_write_kb":11093377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17666,"num_bytes_recovered":1157627904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5121384448,"data_stored":5121376304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":27,"acting":28,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":39,"acting":39,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958777,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":996640,"kb_used_data":691224,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92748512,"statfs":{"total":95995035648,"available":94974476288,"internally_reserved":0,"allocated":707813376,"data_stored":707285144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:06 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":0.77500000000000002,"15min":0.752},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":5.6870000000000003},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.76800000000000002,"15min":0.89800000000000002},"min":{"1min":0.313,"5min":0.29299999999999998,"15min":0.25900000000000001},"max":{"1min":4.5010000000000003,"5min":4.5010000000000003,"15min":4.992},"last":0.82899999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:18:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.1539999999999999,"15min":1.083},"min":{"1min":0.309,"5min":0.27100000000000002,"15min":0.25700000000000001},"max":{"1min":6.1479999999999997,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.647,"5min":1.2390000000000001,"15min":1.113},"min":{"1min":0.55400000000000005,"5min":0.28799999999999998,"15min":0.27300000000000002},"max":{"1min":5.5590000000000002,"5min":5.6920000000000002,"15min":5.6920000000000002},"last":1.2210000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:18:45 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":1.075}]},{"osd":3,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":8.3100000000000005,"5min":3.2749999999999999,"15min":2.2090000000000001},"min":{"1min":0.36499999999999999,"5min":0.313,"15min":0.28999999999999998},"max":{"1min":128.93899999999999,"5min":128.93899999999999,"15min":128.93899999999999},"last":0.751},{"interface":"front","average":{"1min":7.6340000000000003,"5min":3.1909999999999998,"15min":2.2519999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.30299999999999999},"max":{"1min":128.87100000000001,"5min":128.87100000000001,"15min":128.87100000000001},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1,"15min":2.2570000000000001},"min":{"1min":0.36799999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.6910000000000001,"5min":6.5970000000000004,"15min":368.48399999999998},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.03,"5min":0.91800000000000004,"15min":2.1760000000000002},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.6859999999999999,"5min":4.6859999999999999,"15min":371.65100000000001},"last":1.119}]},{"osd":5,"last update":"Mon Feb 12 16:18:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.0189999999999999,"15min":3.7050000000000001},"min":{"1min":0.40400000000000003,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":5.5039999999999996,"5min":5.8789999999999996,"15min":407.07600000000002},"last":1.325},{"interface":"front","average":{"1min":1.054,"5min":1.046,"15min":3.7650000000000001},"min":{"1min":0.38,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.9220000000000002,"5min":4.9909999999999997,"15min":407.26299999999998},"last":1.2929999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:18:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":0.91600000000000004,"15min":0.95799999999999996},"min":{"1min":0.40699999999999997,"5min":0.373,"15min":0.34799999999999998},"max":{"1min":4.5979999999999999,"5min":4.8330000000000002,"15min":4.8330000000000002},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":0.996,"15min":1.0589999999999999},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":4.6539999999999999,"5min":4.6539999999999999,"15min":4.6539999999999999},"last":1.387}]},{"osd":7,"last update":"Mon Feb 12 16:18:06 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.1519999999999999,"15min":1.133},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1499999999999999,"15min":1.153},"min":{"1min":0.48299999999999998,"5min":0.47899999999999998,"15min":0.438},"max":{"1min":4.5679999999999996,"5min":4.9939999999999998,"15min":5.2510000000000003},"last":1.575}]},{"osd":8,"last update":"Mon Feb 12 16:18:06 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.94299999999999995,"15min":1.0389999999999999},"min":{"1min":0.45500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.5339999999999998,"5min":4.5339999999999998,"15min":4.5940000000000003},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.1419999999999999,"15min":1.115},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.7679999999999998,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":1.6759999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:18:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.2949999999999999,"15min":1.0629999999999999},"min":{"1min":0.47299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":5.843,"5min":10.743,"15min":10.743},"last":1.506},{"interface":"front","average":{"1min":1.964,"5min":1.498,"15min":1.1879999999999999},"min":{"1min":0.54700000000000004,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":6.8869999999999996,"5min":10.698,"15min":10.698},"last":1.6060000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:18:09 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.101,"15min":1.1259999999999999},"min":{"1min":0.53400000000000003,"5min":0.433,"15min":0.377},"max":{"1min":1.831,"5min":6.7160000000000002,"15min":6.7160000000000002},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0569999999999999,"15min":1.1299999999999999},"min":{"1min":0.53600000000000003,"5min":0.46500000000000002,"15min":0.45200000000000001},"max":{"1min":1.9399999999999999,"5min":4.6210000000000004,"15min":4.8630000000000004},"last":1.639}]}]},{"osd":10,"up_from":83,"seq":356482285826,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865416,"kb_used_data":568192,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92879736,"statfs":{"total":95995035648,"available":95108849664,"internally_reserved":0,"allocated":581828608,"data_stored":581312446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.999,"15min":1.022},"min":{"1min":0.53700000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2629999999999999,"5min":4.4669999999999996,"15min":4.4669999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1040000000000001,"15min":1.1140000000000001},"min":{"1min":0.56499999999999995,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5209999999999999,"5min":4.726,"15min":4.726},"last":0.75900000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.097,"15min":1.115},"min":{"1min":0.38500000000000001,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":1.5029999999999999,"5min":19.866,"15min":49.802999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.387,"15min":1.222},"min":{"1min":0.309,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1699999999999999,"5min":20.277999999999999,"15min":50.863},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.90300000000000002,"15min":0.80400000000000005},"min":{"1min":0.35199999999999998,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":2.8650000000000002,"5min":5.3040000000000003,"15min":5.3040000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.1539999999999999,"5min":0.85799999999999998,"15min":0.85099999999999998},"min":{"1min":0.46700000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.3260000000000001,"5min":4.8070000000000004,"15min":7.9829999999999997},"last":0.86099999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":8.0109999999999992,"15min":3.714},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.9359999999999999,"5min":628.77099999999996,"15min":628.77099999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.21,"5min":7.8810000000000002,"15min":3.6589999999999998},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.7809999999999999,"5min":624.19299999999998,"15min":624.19299999999998},"last":1.2989999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0189999999999999,"15min":3.4950000000000001},"min":{"1min":0.44600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.5629999999999999,"5min":5.5739999999999998,"15min":602.05399999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.0640000000000001,"15min":3.577},"min":{"1min":0.41799999999999998,"5min":0.35799999999999998,"15min":0.33200000000000002},"max":{"1min":1.9710000000000001,"5min":4.8769999999999998,"15min":598.875},"last":1.218}]},{"osd":6,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":7.9980000000000002,"15min":3.2719999999999998},"min":{"1min":0.41599999999999998,"5min":0.378,"15min":0.35399999999999998},"max":{"1min":1.7290000000000001,"5min":626.26599999999996,"15min":626.26599999999996},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.333,"5min":8.1059999999999999,"15min":3.3559999999999999},"min":{"1min":0.54500000000000004,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.9079999999999999,"5min":625.90700000000004,"15min":625.90700000000004},"last":1.383}]},{"osd":7,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.105,"15min":3.1480000000000001},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.376},"max":{"1min":2.9750000000000001,"5min":5.9210000000000003,"15min":597.48299999999995},"last":1.083},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.1990000000000001,"15min":3.1749999999999998},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.41799999999999998},"max":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":600.23099999999999},"last":0.95199999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.0629999999999999,"15min":1.0489999999999999},"min":{"1min":0.48299999999999998,"5min":0.372,"15min":0.33400000000000002},"max":{"1min":2.9089999999999998,"5min":4.827,"15min":7.2720000000000002},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.1140000000000001,"15min":1.0820000000000001},"min":{"1min":0.51300000000000001,"5min":0.42299999999999999,"15min":0.38200000000000001},"max":{"1min":2.081,"5min":4.8330000000000002,"15min":7.3810000000000002},"last":1.4830000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.2669999999999999,"15min":1.1379999999999999},"min":{"1min":0.48699999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.9339999999999999,"5min":4.9829999999999997,"15min":4.9829999999999997},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.409,"5min":1.284,"15min":1.143},"min":{"1min":0.66500000000000004,"5min":0.41399999999999998,"15min":0.40000000000000002},"max":{"1min":2.956,"5min":4.952,"15min":4.952},"last":1.1160000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:18:07 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.302,"15min":1.383},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.9900000000000002,"5min":10.218999999999999,"15min":30.917999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.613,"5min":1.2949999999999999,"15min":1.3580000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48499999999999999},"max":{"1min":7.3250000000000002,"5min":7.3250000000000002,"15min":30.742000000000001},"last":1.4390000000000001}]}]},{"osd":9,"up_from":74,"seq":317827580170,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":715148,"kb_used_data":413828,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93030004,"statfs":{"total":95995035648,"available":95262724096,"internally_reserved":0,"allocated":423759872,"data_stored":423239646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:50 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.81299999999999994}]},{"osd":1,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.871,"15min":1.4890000000000001},"min":{"1min":0.32200000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.649,"5min":5.843,"15min":175.33500000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.1970000000000001,"15min":1.615},"min":{"1min":0.31,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.458,"5min":7.9550000000000001,"15min":175.24299999999999},"last":1.014}]},{"osd":2,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":2.0819999999999999,"15min":1.4059999999999999},"min":{"1min":0.43099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.6390000000000002,"5min":97.275000000000006,"15min":97.275000000000006},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.876,"5min":1.881,"15min":1.3089999999999999},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":1.587,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.89800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.84199999999999997,"15min":0.98499999999999999},"min":{"1min":0.51900000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.556,"5min":3.6629999999999998,"15min":25.552},"last":1.401},{"interface":"front","average":{"1min":1.327,"5min":1.03,"15min":0.97799999999999998},"min":{"1min":0.499,"5min":0.435,"15min":0.35099999999999998},"max":{"1min":2.02,"5min":5.8010000000000002,"15min":25.847999999999999},"last":1.5289999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":0.90700000000000003,"15min":3.23},"min":{"1min":0.46800000000000003,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":1.901,"5min":5.899,"15min":617.95899999999995},"last":1.427},{"interface":"front","average":{"1min":1.214,"5min":1.056,"15min":3.355},"min":{"1min":0.69699999999999995,"5min":0.40999999999999998,"15min":0.375},"max":{"1min":1.9319999999999999,"5min":7.3739999999999997,"15min":617.88099999999997},"last":1.446}]},{"osd":6,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.101,"15min":1.0620000000000001},"min":{"1min":0.68700000000000006,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.746,"5min":7.3360000000000003,"15min":7.3360000000000003},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.161,"5min":0.94499999999999995,"15min":0.92300000000000004},"min":{"1min":0.46200000000000002,"5min":0.39300000000000002,"15min":0.377},"max":{"1min":2.25,"5min":3.6240000000000001,"15min":4.7910000000000004},"last":1.6319999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.1439999999999999,"15min":4.5780000000000003},"min":{"1min":0.65000000000000002,"5min":0.42799999999999999,"15min":0.379},"max":{"1min":2.1150000000000002,"5min":5.8470000000000004,"15min":613.28200000000004},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.1679999999999999,"15min":4.6280000000000001},"min":{"1min":0.63100000000000001,"5min":0.438,"15min":0.438},"max":{"1min":4.2110000000000003,"5min":7.4089999999999998,"15min":613.25199999999995},"last":1.357}]},{"osd":8,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":0.97999999999999998,"15min":1.052},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.40500000000000003},"max":{"1min":2.0499999999999998,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":1.508},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.1719999999999999,"15min":1.0669999999999999},"min":{"1min":0.78200000000000003,"5min":0.46800000000000003,"15min":0.41199999999999998},"max":{"1min":1.944,"5min":7.25,"15min":7.25},"last":1.5600000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:18:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.012,"15min":1.016},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.0190000000000001,"5min":3.6920000000000002,"15min":5.5},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.147,"15min":1.0900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.671,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":1.734}]},{"osd":11,"last update":"Mon Feb 12 16:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.3,"15min":1.1839999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.5609999999999999,"5min":5.3840000000000003,"15min":5.3840000000000003},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.3009999999999999,"15min":1.2529999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":3.0630000000000002,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":1.6970000000000001}]}]},{"osd":8,"up_from":0,"seq":279172874514,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":56,"seq":240518168858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718920,"kb_used_data":421696,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93026232,"statfs":{"total":95995035648,"available":95258861568,"internally_reserved":0,"allocated":431816704,"data_stored":431300550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.88400000000000001,"15min":0.87},"min":{"1min":0.32600000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.7389999999999999,"5min":5.5149999999999997,"15min":5.5149999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.88800000000000001,"15min":0.84299999999999997},"min":{"1min":0.35499999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.7639999999999998,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":1.1759999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:18:46 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.1359999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.83999999999999997,"15min":1.1459999999999999},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.456,"5min":4.5899999999999999,"15min":60.204999999999998},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.373,"5min":0.97999999999999998,"15min":1.1679999999999999},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.28799999999999998},"max":{"1min":7.2320000000000002,"5min":9.1240000000000006,"15min":59.365000000000002},"last":0.88400000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:17:20 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.86399999999999999,"15min":1.0620000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27900000000000003},"max":{"1min":4.1040000000000001,"5min":4.6369999999999996,"15min":6.2439999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":0.93600000000000005,"15min":1.0740000000000001},"min":{"1min":0.39200000000000002,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":4.1539999999999999,"5min":4.75,"15min":8.2940000000000005},"last":0.46100000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.83099999999999996,"15min":0.876},"min":{"1min":0.29699999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.3169999999999999,"5min":4.7930000000000001,"15min":6.2519999999999998},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98199999999999998,"15min":0.96799999999999997},"min":{"1min":0.39100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.732,"5min":4.5030000000000001,"15min":6.0919999999999996},"last":0.85299999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.88200000000000001,"15min":0.94199999999999995},"min":{"1min":0.44500000000000001,"5min":0.33600000000000002,"15min":0.28999999999999998},"max":{"1min":1.4279999999999999,"5min":4.9050000000000002,"15min":6.0739999999999998},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.98799999999999999,"15min":0.97599999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.2050000000000001,"5min":9.3640000000000008,"15min":9.3640000000000008},"last":1.034}]},{"osd":6,"last update":"Mon Feb 12 16:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.032,"15min":1.0780000000000001},"min":{"1min":0.439,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.786,"5min":4.9359999999999999,"15min":6.0830000000000002},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.181,"15min":1.0840000000000001},"min":{"1min":0.48699999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.7970000000000002,"5min":4.633,"15min":6.1349999999999998},"last":1.3919999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.0349999999999999,"15min":1.1200000000000001},"min":{"1min":1.0469999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.8069999999999999,"5min":5.6130000000000004,"15min":6.0250000000000004},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.1359999999999999,"15min":1.147},"min":{"1min":1.028,"5min":0.45000000000000001,"15min":0.42999999999999999},"max":{"1min":1.984,"5min":9.2140000000000004,"15min":9.2140000000000004},"last":1.226}]},{"osd":9,"last update":"Mon Feb 12 16:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2769999999999999,"15min":2.274},"min":{"1min":0.65300000000000002,"5min":0.47099999999999997,"15min":0.441},"max":{"1min":1.948,"5min":12.907999999999999,"15min":278.54599999999999},"last":1.9330000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.268,"15min":2.2040000000000002},"min":{"1min":0.55400000000000005,"5min":0.42699999999999999,"15min":0.38200000000000001},"max":{"1min":2.5499999999999998,"5min":4.6719999999999997,"15min":281.09199999999998},"last":1.502}]},{"osd":10,"last update":"Mon Feb 12 16:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.335,"15min":1.1970000000000001},"min":{"1min":0.59599999999999997,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.149,"5min":5.8689999999999998,"15min":5.9489999999999998},"last":1.669},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.105,"15min":1.1659999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.42099999999999999},"max":{"1min":1.8460000000000001,"5min":9.1690000000000005,"15min":9.1690000000000005},"last":1.321}]},{"osd":11,"last update":"Mon Feb 12 16:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":2.343,"15min":2.681},"min":{"1min":1.1639999999999999,"5min":0.51700000000000002,"15min":0.498},"max":{"1min":1.883,"5min":120.943,"15min":280.83999999999997},"last":1.855},{"interface":"front","average":{"1min":1.766,"5min":2.4199999999999999,"15min":2.7469999999999999},"min":{"1min":1.407,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":2.6680000000000001,"5min":120.661,"15min":278.63799999999998},"last":1.587}]}]},{"osd":6,"up_from":48,"seq":206158430499,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596744,"kb_used_data":291328,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93148408,"statfs":{"total":95995035648,"available":95383969792,"internally_reserved":0,"allocated":298319872,"data_stored":297803718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.145}]},{"osd":1,"last update":"Mon Feb 12 16:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":3.7200000000000002,"15min":2.0880000000000001},"min":{"1min":0.66100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.5509999999999999,"5min":252.22800000000001,"15min":252.22800000000001},"last":1.026},{"interface":"front","average":{"1min":0.92900000000000005,"5min":3.9350000000000001,"15min":2.1320000000000001},"min":{"1min":0.67500000000000004,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.3360000000000001,"5min":252.61199999999999,"15min":252.61199999999999},"last":1.3580000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.75700000000000001,"15min":1.3149999999999999},"min":{"1min":0.66300000000000003,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":1.2,"5min":3.3279999999999998,"15min":68.462000000000003},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.86799999999999999,"15min":1.3660000000000001},"min":{"1min":0.78000000000000003,"5min":0.311,"15min":0.311},"max":{"1min":1.2909999999999999,"5min":5.0629999999999997,"15min":67.322999999999993},"last":1.444}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.871,"15min":2.0230000000000001},"min":{"1min":0.81699999999999995,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.48,"5min":6.6120000000000001,"15min":284.05599999999998},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.073,"15min":1.907},"min":{"1min":0.73699999999999999,"5min":0.38500000000000001,"15min":0.27000000000000002},"max":{"1min":1.9059999999999999,"5min":5.6429999999999998,"15min":284.041},"last":1.4159999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":0.88200000000000001,"15min":2.0710000000000002},"min":{"1min":0.79500000000000004,"5min":0.312,"15min":0.312},"max":{"1min":1.405,"5min":4.0519999999999996,"15min":281.49700000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.90000000000000002,"15min":2.0590000000000002},"min":{"1min":0.80300000000000005,"5min":0.33500000000000002,"15min":0.29399999999999998},"max":{"1min":1.651,"5min":4.3579999999999997,"15min":281.61900000000003},"last":1.327}]},{"osd":7,"last update":"Mon Feb 12 16:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.02,"15min":1.5800000000000001},"min":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.29399999999999998},"max":{"1min":5.343,"5min":5.343,"15min":67.144999999999996},"last":1.232},{"interface":"front","average":{"1min":1.032,"5min":1.0489999999999999,"15min":1.6020000000000001},"min":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.38200000000000001},"max":{"1min":1.6719999999999999,"5min":6.6449999999999996,"15min":67.230999999999995},"last":1.554}]},{"osd":8,"last update":"Mon Feb 12 16:17:58 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":0.98799999999999999,"15min":1.3089999999999999},"min":{"1min":0.53900000000000003,"5min":0.497,"15min":0.39800000000000002},"max":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":68.653000000000006},"last":1.74},{"interface":"front","average":{"1min":1.502,"5min":1.1200000000000001,"15min":1.3080000000000001},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.40400000000000003},"max":{"1min":5.4379999999999997,"5min":5.4379999999999997,"15min":68.620999999999995},"last":1.504}]},{"osd":9,"last update":"Mon Feb 12 16:18:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.349,"15min":1.385},"min":{"1min":0.78400000000000003,"5min":0.44600000000000001,"15min":0.40600000000000003},"max":{"1min":1.792,"5min":18.059999999999999,"15min":67.188000000000002},"last":1.718},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.3440000000000001,"15min":1.399},"min":{"1min":0.73199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":2.633,"5min":17.425000000000001,"15min":68.506},"last":1.6559999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.22,"15min":1.371},"min":{"1min":0.76200000000000001,"5min":0.54300000000000004,"15min":0.49099999999999999},"max":{"1min":4.1600000000000001,"5min":6.5300000000000002,"15min":68.722999999999999},"last":1.833},{"interface":"front","average":{"1min":1.762,"5min":1.3009999999999999,"15min":1.3640000000000001},"min":{"1min":0.81999999999999995,"5min":0.58399999999999996,"15min":0.505},"max":{"1min":4.2069999999999999,"5min":6.5730000000000004,"15min":68.573999999999998},"last":1.893}]},{"osd":11,"last update":"Mon Feb 12 16:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":4.7859999999999996,"15min":6.2800000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.837,"5min":139.727,"15min":139.727},"last":1.619},{"interface":"front","average":{"1min":1.7130000000000001,"5min":4.5890000000000004,"15min":5.9809999999999999},"min":{"1min":0.64400000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.5499999999999998,"5min":139.24199999999999,"15min":139.24199999999999},"last":1.772}]}]},{"osd":1,"up_from":13,"seq":55834575177,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329288,"kb_used_data":735488,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92415864,"statfs":{"total":95995035648,"available":94633844736,"internally_reserved":0,"allocated":753139712,"data_stored":752623558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.88,"15min":1.5409999999999999},"min":{"1min":0.25600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.0720000000000001,"5min":9.8629999999999995,"15min":114.40000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.091,"15min":1.726},"min":{"1min":0.32700000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.306,"5min":11.433,"15min":115.09099999999999},"last":0.85399999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:18:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.212,"15min":1.27},"min":{"1min":0.30099999999999999,"5min":0.248,"15min":0.248},"max":{"1min":6.6139999999999999,"5min":9.8599999999999994,"15min":40.542999999999999},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.091,"15min":1.391},"min":{"1min":0.38500000000000001,"5min":0.26100000000000001,"15min":0.24299999999999999},"max":{"1min":3.133,"5min":6.7999999999999998,"15min":40.811999999999998},"last":1.9690000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:16:51 2024","interfaces":[{"interface":"back","average":{"1min":6.9550000000000001,"5min":2.2480000000000002,"15min":1.3380000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":142.78100000000001,"5min":142.78100000000001,"15min":142.78100000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":7.0170000000000003,"5min":2.3719999999999999,"15min":1.5269999999999999},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":142.32499999999999,"5min":142.32499999999999,"15min":142.32499999999999},"last":4.2370000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:18:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":2.141},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.91800000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:18:13 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.2090000000000001,"15min":1.1990000000000001},"min":{"1min":0.54900000000000004,"5min":0.35599999999999998,"15min":0.313},"max":{"1min":1.825,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.349,"5min":1.244,"15min":1.2230000000000001},"min":{"1min":0.47599999999999998,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":4.4900000000000002,"5min":7.0110000000000001,"15min":10.420999999999999},"last":1.1899999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":2.4950000000000001,"15min":1.663},"min":{"1min":0.69899999999999995,"5min":0.312,"15min":0.312},"max":{"1min":1.9410000000000001,"5min":56.137,"15min":56.137},"last":1.409},{"interface":"front","average":{"1min":1.2290000000000001,"5min":2.5870000000000002,"15min":1.7230000000000001},"min":{"1min":0.747,"5min":0.44,"15min":0.33700000000000002},"max":{"1min":1.865,"5min":56.195999999999998,"15min":56.195999999999998},"last":1.7030000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":2.5249999999999999,"15min":1.6990000000000001},"min":{"1min":0.504,"5min":0.36399999999999999,"15min":0.34699999999999998},"max":{"1min":4.8369999999999997,"5min":55.869999999999997,"15min":55.869999999999997},"last":2.2029999999999998},{"interface":"front","average":{"1min":1.292,"5min":2.669,"15min":1.6899999999999999},"min":{"1min":0.54300000000000004,"5min":0.42999999999999999,"15min":0.36799999999999999},"max":{"1min":4.3730000000000002,"5min":55.892000000000003,"15min":55.892000000000003},"last":1.2989999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:18:22 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4330000000000001,"15min":1.3560000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.39400000000000002},"max":{"1min":4.5380000000000003,"5min":7.2190000000000003,"15min":9.9049999999999994},"last":1.605},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.3939999999999999,"15min":1.4510000000000001},"min":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.41999999999999998},"max":{"1min":1.9019999999999999,"5min":10.119999999999999,"15min":10.313000000000001},"last":1.0489999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.409,"15min":1.3919999999999999},"min":{"1min":0.68799999999999994,"5min":0.52100000000000002,"15min":0.40699999999999997},"max":{"1min":1.899,"5min":10.07,"15min":10.352},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.5529999999999999,"15min":1.4059999999999999},"min":{"1min":0.54700000000000004,"5min":0.496,"15min":0.40200000000000002},"max":{"1min":2.1869999999999998,"5min":7.2789999999999999,"15min":7.6059999999999999},"last":1.4990000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.4019999999999999,"15min":1.3520000000000001},"min":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.48499999999999999},"max":{"1min":2.0049999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5740000000000001,"15min":1.4910000000000001},"min":{"1min":0.71799999999999997,"5min":0.53700000000000003,"15min":0.51800000000000002},"max":{"1min":4.6299999999999999,"5min":8.7720000000000002,"15min":10.387},"last":1.8089999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:18:13 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.6479999999999999,"15min":1.4750000000000001},"min":{"1min":0.69299999999999995,"5min":0.55800000000000005,"15min":0.443},"max":{"1min":4.7320000000000002,"5min":7.1600000000000001,"15min":13.555},"last":1.901},{"interface":"front","average":{"1min":2.0129999999999999,"5min":1.738,"15min":1.4890000000000001},"min":{"1min":0.80600000000000005,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":4.8200000000000003,"5min":7.2489999999999997,"15min":13.846},"last":1.9890000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738708,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580180,"kb_used_data":278860,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93164972,"statfs":{"total":95995035648,"available":95400931328,"internally_reserved":0,"allocated":285552640,"data_stored":285030476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":1.296,"15min":1.635},"min":{"1min":0.24199999999999999,"5min":0.215,"15min":0.215},"max":{"1min":11.324,"5min":11.324,"15min":114.01600000000001},"last":0.748},{"interface":"front","average":{"1min":0.879,"5min":1.294,"15min":1.589},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.22800000000000001},"max":{"1min":3.5720000000000001,"5min":9.8249999999999993,"15min":114.342},"last":0.78600000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:18:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.85399999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.80900000000000005,"15min":1.2},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.215},"max":{"1min":1.48,"5min":5.4269999999999996,"15min":53.793999999999997},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.80700000000000005,"15min":1.137},"min":{"1min":0.29199999999999998,"5min":0.252,"15min":0.252},"max":{"1min":4.2199999999999998,"5min":6.7089999999999996,"15min":53.853000000000002},"last":1.4390000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:18:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.728,"15min":1.5760000000000001},"min":{"1min":0.318,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.5350000000000001,"5min":80.619,"15min":116.187},"last":1.028},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.9359999999999999,"15min":1.667},"min":{"1min":0.379,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.9749999999999996,"5min":80.123000000000005,"15min":116.163},"last":1.012}]},{"osd":5,"last update":"Mon Feb 12 16:18:02 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.84899999999999998,"15min":1.3959999999999999},"min":{"1min":0.39300000000000002,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":3.4510000000000001,"5min":4.0250000000000004,"15min":116.005},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.81000000000000005,"15min":1.3740000000000001},"min":{"1min":0.436,"5min":0.31,"15min":0.31},"max":{"1min":3.403,"5min":4.2060000000000004,"15min":115.973},"last":1.1659999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.625,"15min":1.181},"min":{"1min":0.68899999999999995,"5min":0.33600000000000002,"15min":0.33000000000000002},"max":{"1min":1.4990000000000001,"5min":81.147000000000006,"15min":81.147000000000006},"last":1.083},{"interface":"front","average":{"1min":1.177,"5min":1.7090000000000001,"15min":1.2330000000000001},"min":{"1min":0.67600000000000005,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.681,"5min":80.233000000000004,"15min":80.233000000000004},"last":1.244}]},{"osd":7,"last update":"Mon Feb 12 16:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":0.95899999999999996,"15min":1.486},"min":{"1min":0.439,"5min":0.39200000000000002,"15min":0.36699999999999999},"max":{"1min":3.4300000000000002,"5min":4.1139999999999999,"15min":115.26000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.0549999999999999,"15min":1.583},"min":{"1min":0.60399999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":3.4729999999999999,"5min":4.1639999999999997,"15min":115.402},"last":1.6319999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:18:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.1080000000000001,"15min":1.1459999999999999},"min":{"1min":0.495,"5min":0.42899999999999999,"15min":0.378},"max":{"1min":4.806,"5min":4.806,"15min":5.6059999999999999},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.64400000000000002,"5min":0.51300000000000001,"15min":0.41299999999999998},"max":{"1min":4.8380000000000001,"5min":4.8380000000000001,"15min":7.5599999999999996},"last":1.72}]},{"osd":9,"last update":"Mon Feb 12 16:18:48 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.6499999999999999,"15min":1.391},"min":{"1min":0.59199999999999997,"5min":0.42699999999999999,"15min":0.40600000000000003},"max":{"1min":1.722,"5min":46.246000000000002,"15min":46.246000000000002},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.411,"15min":1.2749999999999999},"min":{"1min":0.63500000000000001,"5min":0.45800000000000002,"15min":0.39500000000000002},"max":{"1min":1.845,"5min":46.167999999999999,"15min":46.167999999999999},"last":1.7769999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:18:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.2809999999999999,"15min":1.2370000000000001},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":5.5529999999999999,"5min":6.0979999999999999,"15min":6.0979999999999999},"last":1.444},{"interface":"front","average":{"1min":1.407,"5min":1.1499999999999999,"15min":1.224},"min":{"1min":0.58899999999999997,"5min":0.47999999999999998,"15min":0.437},"max":{"1min":5.6269999999999998,"5min":5.6269999999999998,"15min":6.2480000000000002},"last":1.5169999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:18:02 2024","interfaces":[{"interface":"back","average":{"1min":3.4580000000000002,"5min":1.9430000000000001,"15min":1.4850000000000001},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.434},"max":{"1min":46.064,"5min":46.064,"15min":46.064},"last":1.3460000000000001},{"interface":"front","average":{"1min":3.3210000000000002,"5min":1.667,"15min":1.4730000000000001},"min":{"1min":0.57599999999999996,"5min":0.46400000000000002,"15min":0.38300000000000001},"max":{"1min":45.642000000000003,"5min":45.642000000000003,"15min":45.642000000000003},"last":1.8400000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313538,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593032,"kb_used_data":283520,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93152120,"statfs":{"total":95995035648,"available":95387770880,"internally_reserved":0,"allocated":290324480,"data_stored":289808326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.82899999999999996,"15min":1.8300000000000001},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.222},"max":{"1min":3.3410000000000002,"5min":6.1859999999999999,"15min":177.68899999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.5740000000000001,"5min":0.96099999999999997,"15min":1.923},"min":{"1min":0.33800000000000002,"5min":0.246,"15min":0.216},"max":{"1min":6.2800000000000002,"5min":6.3099999999999996,"15min":178.26900000000001},"last":0.85799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.0449999999999999,"15min":1.419},"min":{"1min":0.373,"5min":0.27200000000000002,"15min":0.223},"max":{"1min":3.4849999999999999,"5min":6.7839999999999998,"15min":70.944000000000003},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.159,"15min":1.466},"min":{"1min":0.41799999999999998,"5min":0.27000000000000002,"15min":0.22600000000000001},"max":{"1min":4.9569999999999999,"5min":6.806,"15min":71.203999999999994},"last":1.0860000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":2.4969999999999999,"15min":1.7010000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":2.8519999999999999,"5min":80.436999999999998,"15min":80.436999999999998},"last":3.4279999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":2.6110000000000002,"15min":1.5900000000000001},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":3.7919999999999998,"5min":77.286000000000001,"15min":77.286000000000001},"last":0.90500000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.879,"15min":1.0249999999999999},"min":{"1min":0.38600000000000001,"5min":0.36199999999999999,"15min":0.311},"max":{"1min":1.1799999999999999,"5min":7.2859999999999996,"15min":9.657},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":0.94099999999999995,"15min":1.012},"min":{"1min":0.437,"5min":0.317,"15min":0.31},"max":{"1min":3.3159999999999998,"5min":5.0140000000000002,"15min":9.5530000000000008},"last":1.3380000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.91900000000000004,"15min":3.1680000000000001},"min":{"1min":0.36899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":3.4750000000000001,"5min":5.048,"15min":616.34199999999998},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.262,"5min":0.92800000000000005,"15min":3.2810000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33900000000000002},"max":{"1min":6.5730000000000004,"5min":6.5730000000000004,"15min":615.18899999999996},"last":1.2589999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:18:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.05,"15min":1.0309999999999999},"min":{"1min":0.56200000000000006,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.3820000000000001,"5min":6.5049999999999999,"15min":9.6150000000000002},"last":1.226},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.048,"15min":1.0800000000000001},"min":{"1min":0.504,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":3.5099999999999998,"5min":5.1529999999999996,"15min":6.3979999999999997},"last":1.4330000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.98599999999999999,"15min":3.3319999999999999},"min":{"1min":0.442,"5min":0.378,"15min":0.377},"max":{"1min":1.29,"5min":5.2960000000000003,"15min":611.79600000000005},"last":1.137},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.05,"15min":3.4060000000000001},"min":{"1min":0.53600000000000003,"5min":0.439,"15min":0.38700000000000001},"max":{"1min":3.5419999999999998,"5min":5.0739999999999998,"15min":611.83900000000006},"last":1.5569999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:17:53 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":0.99099999999999999,"15min":1.161},"min":{"1min":0.48699999999999999,"5min":0.42099999999999999,"15min":0.38200000000000001},"max":{"1min":5.101,"5min":5.101,"15min":6.4960000000000004},"last":1.819},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.075,"15min":1.2829999999999999},"min":{"1min":0.47799999999999998,"5min":0.45300000000000001,"15min":0.432},"max":{"1min":5.1180000000000003,"5min":5.1180000000000003,"15min":8.5920000000000005},"last":1.696}]},{"osd":9,"last update":"Mon Feb 12 16:18:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1559999999999999,"15min":1.1659999999999999},"min":{"1min":0.51400000000000001,"5min":0.45700000000000002,"15min":0.40600000000000003},"max":{"1min":2.3290000000000002,"5min":5.2329999999999997,"15min":9.6460000000000008},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.147,"15min":1.2130000000000001},"min":{"1min":0.59499999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":3.4940000000000002,"5min":5.085,"15min":6.4699999999999998},"last":1.4830000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:18:15 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.2050000000000001,"15min":1.206},"min":{"1min":0.50600000000000001,"5min":0.5,"15min":0.442},"max":{"1min":5.1890000000000001,"5min":5.1890000000000001,"15min":9.6760000000000002},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.718,"5min":1.2010000000000001,"15min":1.25},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":5.0519999999999996,"5min":7.2699999999999996,"15min":7.2699999999999996},"last":1.792}]},{"osd":11,"last update":"Mon Feb 12 16:18:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"min":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"max":{"1min":1.921,"5min":1.921,"15min":1.921},"last":1.8959999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.7090000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149995,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939908,"kb_used_data":642684,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92805244,"statfs":{"total":95995035648,"available":95032569856,"internally_reserved":0,"allocated":658108416,"data_stored":657754657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:16:08 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.86499999999999999,"15min":0.94999999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":2.7429999999999999,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.92900000000000005,"15min":0.879},"min":{"1min":0.27300000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.4569999999999999,"5min":6.6680000000000001,"15min":6.6680000000000001},"last":0.755}]},{"osd":1,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.7220000000000004,"5min":6.7220000000000004,"15min":6.7220000000000004},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:17:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":0.93700000000000006,"15min":1.28},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23200000000000001},"max":{"1min":5.4550000000000001,"5min":5.7320000000000002,"15min":68.405000000000001},"last":1.004},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.0169999999999999,"15min":1.29},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.9649999999999999,"5min":7.9649999999999999,"15min":69.009},"last":0.83999999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:17:31 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":0.98399999999999999,"15min":0.93700000000000006},"min":{"1min":0.308,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.5609999999999999,"5min":6.5609999999999999,"15min":6.5609999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.88700000000000001,"15min":0.90400000000000003},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":4.7619999999999996,"5min":4.7619999999999996,"15min":5.9870000000000001},"last":1.0649999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.4470000000000001,"15min":1.1459999999999999},"min":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.313},"max":{"1min":4.0670000000000002,"5min":22.751000000000001,"15min":22.751000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.2,"15min":1.0720000000000001},"min":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32200000000000001},"max":{"1min":4.8040000000000003,"5min":22.527999999999999,"15min":22.527999999999999},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:17:39 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.0660000000000001,"15min":1.0009999999999999},"min":{"1min":0.36599999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6260000000000003,"5min":4.6260000000000003,"15min":6.0289999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0309999999999999,"15min":1.141},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.5549999999999997,"5min":4.5549999999999997,"15min":9.5990000000000002},"last":1.2}]},{"osd":7,"last update":"Mon Feb 12 16:17:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2290000000000001,"15min":1.47},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.4119999999999999,"5min":10.061,"15min":91.718999999999994},"last":1.254},{"interface":"front","average":{"1min":1.571,"5min":1.1180000000000001,"15min":1.554},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.5949999999999998,"5min":5.0869999999999997,"15min":90.367999999999995},"last":0.88800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.1319999999999999,"15min":1.1990000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.317},"max":{"1min":5.1500000000000004,"5min":6.0970000000000004,"15min":6.5899999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.097,"15min":1.1559999999999999},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.7039999999999997,"5min":6.2430000000000003,"15min":8.516},"last":1.089}]},{"osd":9,"last update":"Mon Feb 12 16:17:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.107,"15min":1.131},"min":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.8739999999999997,"5min":4.8739999999999997,"15min":6.2699999999999996},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.2130000000000001,"15min":1.2869999999999999},"min":{"1min":0.45800000000000002,"5min":0.45300000000000001,"15min":0.38200000000000001},"max":{"1min":4.6619999999999999,"5min":4.6619999999999999,"15min":9.6509999999999998},"last":1.0489999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.044,"15min":1.212},"min":{"1min":0.42899999999999999,"5min":0.42699999999999999,"15min":0.41099999999999998},"max":{"1min":4.8399999999999999,"5min":6.4199999999999999,"15min":6.4199999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.101,"15min":1.22},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":4.9630000000000001,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":1.389}]},{"osd":11,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.5680000000000001,"15min":1.5860000000000001},"min":{"1min":0.505,"5min":0.47699999999999998,"15min":0.39100000000000001},"max":{"1min":6.0110000000000001,"5min":26.190000000000001,"15min":49.167000000000002},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.607,"15min":1.5960000000000001},"min":{"1min":0.59299999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":7.4059999999999997,"5min":26.202999999999999,"15min":48.548999999999999},"last":1.163}]}]},{"osd":4,"up_from":32,"seq":137438953779,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716172,"kb_used_data":418948,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93028980,"statfs":{"total":95995035648,"available":95261675520,"internally_reserved":0,"allocated":429002752,"data_stored":428482526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.749,"15min":1.361},"min":{"1min":0.504,"5min":0.26500000000000001,"15min":0.25700000000000001},"max":{"1min":3.992,"5min":4.1740000000000004,"15min":96.052000000000007},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.872,"15min":1.532},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.871,"5min":4.8979999999999997,"15min":96.307000000000002},"last":1.042}]},{"osd":1,"last update":"Mon Feb 12 16:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.976,"15min":1.623},"min":{"1min":0.74099999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.6319999999999997,"5min":90.597999999999999,"15min":90.597999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":2.2799999999999998,"15min":1.6379999999999999},"min":{"1min":0.41699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.3899999999999997,"5min":90.676000000000002,"15min":90.676000000000002},"last":1.288}]},{"osd":2,"last update":"Mon Feb 12 16:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":0.91500000000000004,"15min":1.135},"min":{"1min":0.38800000000000001,"5min":0.29399999999999998,"15min":0.27700000000000002},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":41.546999999999997},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.742,"5min":1.101,"15min":1.278},"min":{"1min":0.497,"5min":0.33000000000000002,"15min":0.31},"max":{"1min":7.452,"5min":7.452,"15min":41.57},"last":1.1899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.2969999999999999,"15min":1.4219999999999999},"min":{"1min":0.36799999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.7949999999999999,"5min":28.434000000000001,"15min":82.781000000000006},"last":4.3259999999999996},{"interface":"front","average":{"1min":1.103,"5min":1.5469999999999999,"15min":1.456},"min":{"1min":0.45700000000000002,"5min":0.35599999999999998,"15min":0.28599999999999998},"max":{"1min":4.7240000000000002,"5min":27.887,"15min":83.349000000000004},"last":3.3999999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.86099999999999999,"15min":0.88700000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.25800000000000001},"max":{"1min":4.0339999999999998,"5min":4.3719999999999999,"15min":6.0460000000000003},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.98999999999999999,"15min":1.0029999999999999},"min":{"1min":0.44900000000000001,"5min":0.35799999999999998,"15min":0.31900000000000001},"max":{"1min":4.0629999999999997,"5min":4.4109999999999996,"15min":7.1070000000000002},"last":1.071}]},{"osd":6,"last update":"Mon Feb 12 16:18:44 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.0529999999999999,"15min":0.98899999999999999},"min":{"1min":0.88800000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.637,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.095,"5min":1.042,"15min":0.94999999999999996},"min":{"1min":0.85399999999999998,"5min":0.35199999999999998,"15min":0.27000000000000002},"max":{"1min":1.766,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":0.75700000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.1519999999999999,"15min":1.5489999999999999},"min":{"1min":0.69899999999999995,"5min":0.36599999999999999,"15min":0.35699999999999998},"max":{"1min":8.2070000000000007,"5min":8.2070000000000007,"15min":116.273},"last":1.02},{"interface":"front","average":{"1min":2.0299999999999998,"5min":1.274,"15min":1.6180000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.433},"max":{"1min":7.6689999999999996,"5min":7.6689999999999996,"15min":116.24299999999999},"last":1.367}]},{"osd":8,"last update":"Mon Feb 12 16:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.1279999999999999,"15min":1.2010000000000001},"min":{"1min":0.49299999999999999,"5min":0.442,"15min":0.442},"max":{"1min":7.8650000000000002,"5min":7.8650000000000002,"15min":8.4190000000000005},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.1200000000000001,"15min":1.133},"min":{"1min":0.57099999999999995,"5min":0.443,"15min":0.42899999999999999},"max":{"1min":4.2649999999999997,"5min":4.6369999999999996,"15min":7.0380000000000003},"last":1.2330000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.776,"5min":1.8540000000000001,"15min":2.4620000000000002},"min":{"1min":1.246,"5min":0.504,"15min":0.45500000000000002},"max":{"1min":4.3070000000000004,"5min":34.112000000000002,"15min":297.55900000000003},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.9399999999999999,"5min":2.073,"15min":2.548},"min":{"1min":0.53900000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":7.9180000000000001,"5min":34.590000000000003,"15min":297.61900000000003},"last":1.6439999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:18:35 2024","interfaces":[{"interface":"back","average":{"1min":2.0190000000000001,"5min":1.3979999999999999,"15min":1.23},"min":{"1min":1.3,"5min":0.51000000000000001,"15min":0.48699999999999999},"max":{"1min":4.5609999999999999,"5min":6.0899999999999999,"15min":6.9770000000000003},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.264,"15min":1.2470000000000001},"min":{"1min":0.50900000000000001,"5min":0.50700000000000001,"15min":0.47599999999999998},"max":{"1min":8.5299999999999994,"5min":8.5299999999999994,"15min":8.5299999999999994},"last":1.1679999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:18:44 2024","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.706},{"interface":"front","average":{"1min":1.552,"5min":1.552,"15min":1.552},"min":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.345}]}]},{"osd":5,"up_from":39,"seq":167503724842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586056,"kb_used_data":276544,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93159096,"statfs":{"total":95995035648,"available":95394914304,"internally_reserved":0,"allocated":283181056,"data_stored":282664902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":1.1000000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:18:10 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":6.3220000000000001,"15min":2.9550000000000001},"min":{"1min":0.28399999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.7879999999999998,"5min":565.34500000000003,"15min":565.34500000000003},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.091,"5min":6.4359999999999999,"15min":2.8889999999999998},"min":{"1min":0.30399999999999999,"5min":0.28000000000000003,"15min":0.253},"max":{"1min":4.0750000000000002,"5min":565.98400000000004,"15min":565.98400000000004},"last":1.359}]},{"osd":2,"last update":"Mon Feb 12 16:18:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.89400000000000002,"15min":1.161},"min":{"1min":0.32200000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.6539999999999999,"5min":5.7080000000000002,"15min":26.797000000000001},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.925,"5min":1.0669999999999999,"15min":1.121},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.28599999999999998},"max":{"1min":10.140000000000001,"5min":10.140000000000001,"15min":26.905999999999999},"last":0.85799999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":3.7029999999999998,"15min":2.7360000000000002},"min":{"1min":0.30499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.5830000000000002,"5min":252.57300000000001,"15min":252.57300000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.345,"5min":3.972,"15min":2.7480000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.548,"5min":252.24199999999999,"15min":252.24199999999999},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:18:10 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.8149999999999999,"15min":1.264},"min":{"1min":0.42699999999999999,"5min":0.31900000000000001,"15min":0.27300000000000002},"max":{"1min":2.9140000000000001,"5min":75.257000000000005,"15min":75.257000000000005},"last":1.454},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.7130000000000001,"15min":1.282},"min":{"1min":0.442,"5min":0.35199999999999998,"15min":0.32600000000000001},"max":{"1min":2.7290000000000001,"5min":74.427000000000007,"15min":74.427000000000007},"last":0.95399999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.823,"15min":1.3280000000000001},"min":{"1min":0.86799999999999999,"5min":0.35299999999999998,"15min":0.318},"max":{"1min":1.7829999999999999,"5min":74.105000000000004,"15min":74.105000000000004},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.946,"15min":1.399},"min":{"1min":0.52800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.2999999999999998,"5min":75.013999999999996,"15min":75.013999999999996},"last":1.3}]},{"osd":7,"last update":"Mon Feb 12 16:18:16 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":0.89000000000000001,"15min":1.4650000000000001},"min":{"1min":0.499,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.7759999999999998,"5min":6.6630000000000003,"15min":98.971000000000004},"last":0.997},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.0409999999999999,"15min":1.5209999999999999},"min":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40600000000000003},"max":{"1min":3.0430000000000001,"5min":4.6379999999999999,"15min":99.218000000000004},"last":1.2430000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:18:10 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.087,"15min":1.179},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.35099999999999998},"max":{"1min":4.2220000000000004,"5min":5.9509999999999996,"15min":14.372999999999999},"last":1.55},{"interface":"front","average":{"1min":1.468,"5min":1.0960000000000001,"15min":1.2010000000000001},"min":{"1min":0.57999999999999996,"5min":0.51500000000000001,"15min":0.40500000000000003},"max":{"1min":4.2130000000000001,"5min":4.2130000000000001,"15min":14.074999999999999},"last":1.4199999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":7.1310000000000002,"15min":3.1989999999999998},"min":{"1min":1.167,"5min":0.439,"15min":0.42999999999999999},"max":{"1min":1.9530000000000001,"5min":449.89499999999998,"15min":449.89499999999998},"last":1.504},{"interface":"front","average":{"1min":1.6379999999999999,"5min":7.109,"15min":3.23},"min":{"1min":0.72799999999999998,"5min":0.51600000000000001,"15min":0.45300000000000001},"max":{"1min":2.8500000000000001,"5min":448.70800000000003,"15min":448.70800000000003},"last":1.5209999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:18:16 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.208,"15min":1.2569999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.47499999999999998},"max":{"1min":4.8620000000000001,"5min":5.9210000000000003,"15min":14.214},"last":1.714},{"interface":"front","average":{"1min":1.343,"5min":1.1000000000000001,"15min":1.2490000000000001},"min":{"1min":0.65300000000000002,"5min":0.5,"15min":0.5},"max":{"1min":3.069,"5min":6.1379999999999999,"15min":13.464},"last":1.4319999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.1140000000000001,"15min":0.92000000000000004},"min":{"1min":0.622,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.9399999999999999,"5min":5.976,"15min":5.976},"last":1.637},{"interface":"front","average":{"1min":1.619,"5min":1.1990000000000001,"15min":1.111},"min":{"1min":0.69699999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":4.4779999999999998,"5min":4.5129999999999999,"15min":4.5129999999999999},"last":1.3440000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284491776,"data_stored":284491776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":750125056,"data_stored":750125056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289275904,"data_stored":289275904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":649134080,"data_stored":649134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":427954176,"data_stored":427950104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":282132480,"data_stored":282132480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":297271296,"data_stored":297271296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":430768128,"data_stored":430768128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":422711296,"data_stored":422707224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":580780032,"data_stored":580780032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":706740224,"data_stored":706740224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:18:58.243 INFO:tasks.thrashosds.thrasher:Setting 1.d to [6, 0, 9, 10] 2024-02-12T16:18:58.243 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.d', '6', '0', '9', '10'] 2024-02-12T16:18:58.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pg-upmap-items 1.d 6 0 9 10 2024-02-12T16:18:58.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:58.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:58.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:58.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:58.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:58.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:58.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:58.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:58.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:58.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:58.734 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:58.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:58.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:58.878 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:18:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:58.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:18:58.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:59.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:18:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:59.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:18:59.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:58 smithi123 bash[19119]: audit 2024-02-12T16:18:57.506549+0000 mgr.y (mgr.14146) 1010 : audit [DBG] from='client.15702 v1:172.21.15.121:0/425832298' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:18:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:58 smithi123 bash[19119]: cluster 2024-02-12T16:18:57.748196+0000 osd.7 (osd.7) 87 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:18:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:58 smithi123 bash[19119]: cluster 2024-02-12T16:18:57.780543+0000 osd.7 (osd.7) 88 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:18:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:18:58 smithi123 bash[19119]: cluster 2024-02-12T16:18:57.781989+0000 mgr.y (mgr.14146) 1011 : cluster [DBG] pgmap v1351: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:59.161 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:59.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:59.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:59.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:58 smithi169 bash[19604]: audit 2024-02-12T16:18:57.506549+0000 mgr.y (mgr.14146) 1010 : audit [DBG] from='client.15702 v1:172.21.15.121:0/425832298' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:18:59.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:58 smithi169 bash[19604]: cluster 2024-02-12T16:18:57.748196+0000 osd.7 (osd.7) 87 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:18:59.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:58 smithi169 bash[19604]: cluster 2024-02-12T16:18:57.780543+0000 osd.7 (osd.7) 88 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:18:59.278 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:18:58 smithi169 bash[19604]: cluster 2024-02-12T16:18:57.781989+0000 mgr.y (mgr.14146) 1011 : cluster [DBG] pgmap v1351: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:59.303 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:18:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:59.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:18:59.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:59.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:58 smithi121 bash[18489]: audit 2024-02-12T16:18:57.506549+0000 mgr.y (mgr.14146) 1010 : audit [DBG] from='client.15702 v1:172.21.15.121:0/425832298' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:18:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:58 smithi121 bash[18489]: cluster 2024-02-12T16:18:57.748196+0000 osd.7 (osd.7) 87 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:18:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:58 smithi121 bash[18489]: cluster 2024-02-12T16:18:57.780543+0000 osd.7 (osd.7) 88 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:18:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:18:58 smithi121 bash[18489]: cluster 2024-02-12T16:18:57.781989+0000 mgr.y (mgr.14146) 1011 : cluster [DBG] pgmap v1351: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:18:59.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:59.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:59.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:59.588 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:18:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:59.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:18:59.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:59.726 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:18:59.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:18:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:59.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:18:59.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:59.868 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:18:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:18:59.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:18:59.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:00.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:00.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:00.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:00.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:00.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:00.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:00.741 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:00.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:00.846 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:00.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:00.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:00.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:01.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:00 smithi121 bash[18489]: cluster 2024-02-12T16:18:59.782985+0000 mgr.y (mgr.14146) 1012 : cluster [DBG] pgmap v1352: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:01.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:00 smithi121 bash[18489]: cluster 2024-02-12T16:18:59.801519+0000 osd.0 (osd.0) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:01.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:00 smithi121 bash[18489]: cluster 2024-02-12T16:18:59.801619+0000 osd.0 (osd.0) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:01.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:00 smithi121 bash[18489]: cluster 2024-02-12T16:18:59.926685+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e442: 12 total, 11 up, 11 in 2024-02-12T16:19:01.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:00 smithi121 bash[18489]: cluster 2024-02-12T16:19:00.467694+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e443: 12 total, 11 up, 11 in 2024-02-12T16:19:01.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:01.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:00 smithi169 bash[19604]: cluster 2024-02-12T16:18:59.782985+0000 mgr.y (mgr.14146) 1012 : cluster [DBG] pgmap v1352: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:01.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:00 smithi169 bash[19604]: cluster 2024-02-12T16:18:59.801519+0000 osd.0 (osd.0) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:01.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:00 smithi169 bash[19604]: cluster 2024-02-12T16:18:59.801619+0000 osd.0 (osd.0) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:01.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:00 smithi169 bash[19604]: cluster 2024-02-12T16:18:59.926685+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e442: 12 total, 11 up, 11 in 2024-02-12T16:19:01.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:00 smithi169 bash[19604]: cluster 2024-02-12T16:19:00.467694+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e443: 12 total, 11 up, 11 in 2024-02-12T16:19:01.116 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:01.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:01.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:01.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:01.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:00 smithi123 bash[19119]: cluster 2024-02-12T16:18:59.782985+0000 mgr.y (mgr.14146) 1012 : cluster [DBG] pgmap v1352: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:00 smithi123 bash[19119]: cluster 2024-02-12T16:18:59.801519+0000 osd.0 (osd.0) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:00 smithi123 bash[19119]: cluster 2024-02-12T16:18:59.801619+0000 osd.0 (osd.0) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:00 smithi123 bash[19119]: cluster 2024-02-12T16:18:59.926685+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e442: 12 total, 11 up, 11 in 2024-02-12T16:19:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:00 smithi123 bash[19119]: cluster 2024-02-12T16:19:00.467694+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e443: 12 total, 11 up, 11 in 2024-02-12T16:19:01.401 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:01.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:01.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:01.538 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:01.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:01.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:01.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:01.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:01.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:01.824 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:01.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:01.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:01.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:01.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:02.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:02.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:02.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:01 smithi123 bash[19119]: cluster 2024-02-12T16:19:00.380626+0000 osd.6 (osd.6) 64 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:19:02.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:01 smithi123 bash[19119]: cluster 2024-02-12T16:19:00.407525+0000 osd.6 (osd.6) 65 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:19:02.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:01 smithi123 bash[19119]: cluster 2024-02-12T16:19:00.727855+0000 osd.7 (osd.7) 89 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:19:02.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:01 smithi123 bash[19119]: cluster 2024-02-12T16:19:00.782132+0000 osd.7 (osd.7) 90 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:19:02.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:02.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:01 smithi169 bash[19604]: cluster 2024-02-12T16:19:00.380626+0000 osd.6 (osd.6) 64 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:19:02.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:01 smithi169 bash[19604]: cluster 2024-02-12T16:19:00.407525+0000 osd.6 (osd.6) 65 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:19:02.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:01 smithi169 bash[19604]: cluster 2024-02-12T16:19:00.727855+0000 osd.7 (osd.7) 89 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:19:02.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:01 smithi169 bash[19604]: cluster 2024-02-12T16:19:00.782132+0000 osd.7 (osd.7) 90 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:19:02.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:02.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:02.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:02.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:01 smithi121 bash[18489]: cluster 2024-02-12T16:19:00.380626+0000 osd.6 (osd.6) 64 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:19:02.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:01 smithi121 bash[18489]: cluster 2024-02-12T16:19:00.407525+0000 osd.6 (osd.6) 65 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:19:02.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:01 smithi121 bash[18489]: cluster 2024-02-12T16:19:00.727855+0000 osd.7 (osd.7) 89 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:19:02.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:01 smithi121 bash[18489]: cluster 2024-02-12T16:19:00.782132+0000 osd.7 (osd.7) 90 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:19:02.385 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:02.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:02.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:02.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:02.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:02.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:02.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:02.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:02.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:02.810 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:02.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:02.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:02.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:02.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:03.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:02 smithi169 bash[19604]: cluster 2024-02-12T16:19:01.362230+0000 osd.6 (osd.6) 66 : cluster [DBG] 1.f scrub starts 2024-02-12T16:19:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:02 smithi169 bash[19604]: cluster 2024-02-12T16:19:01.385190+0000 osd.6 (osd.6) 67 : cluster [DBG] 1.f scrub ok 2024-02-12T16:19:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:02 smithi169 bash[19604]: cluster 2024-02-12T16:19:01.783638+0000 mgr.y (mgr.14146) 1013 : cluster [DBG] pgmap v1355: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:03.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:03.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:03.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:03.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:03.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:02 smithi121 bash[18489]: cluster 2024-02-12T16:19:01.362230+0000 osd.6 (osd.6) 66 : cluster [DBG] 1.f scrub starts 2024-02-12T16:19:03.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:02 smithi121 bash[18489]: cluster 2024-02-12T16:19:01.385190+0000 osd.6 (osd.6) 67 : cluster [DBG] 1.f scrub ok 2024-02-12T16:19:03.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:02 smithi121 bash[18489]: cluster 2024-02-12T16:19:01.783638+0000 mgr.y (mgr.14146) 1013 : cluster [DBG] pgmap v1355: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:03.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:03.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:03.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:03.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:02 smithi123 bash[19119]: cluster 2024-02-12T16:19:01.362230+0000 osd.6 (osd.6) 66 : cluster [DBG] 1.f scrub starts 2024-02-12T16:19:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:02 smithi123 bash[19119]: cluster 2024-02-12T16:19:01.385190+0000 osd.6 (osd.6) 67 : cluster [DBG] 1.f scrub ok 2024-02-12T16:19:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:02 smithi123 bash[19119]: cluster 2024-02-12T16:19:01.783638+0000 mgr.y (mgr.14146) 1013 : cluster [DBG] pgmap v1355: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:03.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:03.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:03.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:03.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:03.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:03.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:03.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:03.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:03.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:03.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:03.925 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:03.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:04.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:04.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:04.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:03 smithi123 bash[19119]: cluster 2024-02-12T16:19:02.990634+0000 osd.11 (osd.11) 79 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:19:04.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:04.204 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:04.206 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:03 smithi169 bash[19604]: cluster 2024-02-12T16:19:02.990634+0000 osd.11 (osd.11) 79 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:19:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:04.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:04.308 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:04.321 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:03 smithi121 bash[18489]: cluster 2024-02-12T16:19:02.990634+0000 osd.11 (osd.11) 79 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:19:04.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:04.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:04.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:04.485 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:04.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:04.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:04.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:04.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:04.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:04.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:04.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:04.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:04.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:04.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:05.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:05.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:05.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:05.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:05.155 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:04 smithi121 bash[18489]: cluster 2024-02-12T16:19:03.367418+0000 osd.11 (osd.11) 80 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:19:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:04 smithi121 bash[18489]: cluster 2024-02-12T16:19:03.784695+0000 mgr.y (mgr.14146) 1014 : cluster [DBG] pgmap v1356: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:04 smithi121 bash[18489]: cluster 2024-02-12T16:19:03.793576+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:19:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:04 smithi121 bash[18489]: cluster 2024-02-12T16:19:03.823148+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:19:05.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:05.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:05.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:04 smithi169 bash[19604]: cluster 2024-02-12T16:19:03.367418+0000 osd.11 (osd.11) 80 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:19:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:04 smithi169 bash[19604]: cluster 2024-02-12T16:19:03.784695+0000 mgr.y (mgr.14146) 1014 : cluster [DBG] pgmap v1356: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:04 smithi169 bash[19604]: cluster 2024-02-12T16:19:03.793576+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:19:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:04 smithi169 bash[19604]: cluster 2024-02-12T16:19:03.823148+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:19:05.330 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:05.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:04 smithi123 bash[19119]: cluster 2024-02-12T16:19:03.367418+0000 osd.11 (osd.11) 80 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:19:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:04 smithi123 bash[19119]: cluster 2024-02-12T16:19:03.784695+0000 mgr.y (mgr.14146) 1014 : cluster [DBG] pgmap v1356: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:04 smithi123 bash[19119]: cluster 2024-02-12T16:19:03.793576+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:19:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:04 smithi123 bash[19119]: cluster 2024-02-12T16:19:03.823148+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:19:05.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:05.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:05.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:05.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:05.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:05.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:05.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:05.748 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:05.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:05.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:05.889 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:05.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:05.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:06.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:06.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:06.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:06.172 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:06.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:06.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:06.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:06.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:06.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:06.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:06.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:06.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:06 smithi121 bash[18489]: cluster 2024-02-12T16:19:05.112640+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:19:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:06 smithi121 bash[18489]: cluster 2024-02-12T16:19:05.133263+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:19:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:06 smithi121 bash[18489]: cluster 2024-02-12T16:19:05.612619+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:06 smithi121 bash[18489]: cluster 2024-02-12T16:19:05.612712+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:06 smithi121 bash[18489]: cluster 2024-02-12T16:19:05.785768+0000 mgr.y (mgr.14146) 1015 : cluster [DBG] pgmap v1357: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:06.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:06.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:06.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:06.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:06.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:06.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:06 smithi169 bash[19604]: cluster 2024-02-12T16:19:05.112640+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:19:06.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:06 smithi169 bash[19604]: cluster 2024-02-12T16:19:05.133263+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:19:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:06 smithi169 bash[19604]: cluster 2024-02-12T16:19:05.612619+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:06 smithi169 bash[19604]: cluster 2024-02-12T16:19:05.612712+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:06 smithi169 bash[19604]: cluster 2024-02-12T16:19:05.785768+0000 mgr.y (mgr.14146) 1015 : cluster [DBG] pgmap v1357: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:06.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:06.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:06.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:06 smithi123 bash[19119]: cluster 2024-02-12T16:19:05.112640+0000 osd.10 (osd.10) 58 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:19:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:06 smithi123 bash[19119]: cluster 2024-02-12T16:19:05.133263+0000 osd.10 (osd.10) 59 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:19:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:06 smithi123 bash[19119]: cluster 2024-02-12T16:19:05.612619+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:06 smithi123 bash[19119]: cluster 2024-02-12T16:19:05.612712+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:06.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:06 smithi123 bash[19119]: cluster 2024-02-12T16:19:05.785768+0000 mgr.y (mgr.14146) 1015 : cluster [DBG] pgmap v1357: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:06.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:07.011 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:07.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:07.153 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:07.173 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:19:07.194 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:19:07.214 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:19:07.237 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:19:07.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:07.261 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:19:07.287 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:19:07.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:07.309 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:19:07.341 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:19:07.364 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:19:07.387 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:19:07.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:07.414 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:19:07.436 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:07.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:07.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:07.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:07 smithi123 bash[19119]: cluster 2024-02-12T16:19:05.875332+0000 osd.10 (osd.10) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:07.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:07 smithi123 bash[19119]: cluster 2024-02-12T16:19:05.875426+0000 osd.10 (osd.10) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:07.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:07 smithi123 bash[19119]: cluster 2024-02-12T16:19:06.488223+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e444: 12 total, 11 up, 11 in 2024-02-12T16:19:07.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:07 smithi123 bash[19119]: cluster 2024-02-12T16:19:06.987149+0000 osd.11 (osd.11) 81 : cluster [DBG] 1.c scrub starts 2024-02-12T16:19:07.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:07 smithi123 bash[19119]: cluster 2024-02-12T16:19:07.060174+0000 osd.11 (osd.11) 82 : cluster [DBG] 1.c scrub ok 2024-02-12T16:19:07.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:07 smithi123 bash[19119]: audit 2024-02-12T16:19:07.144298+0000 mon.a (mon.0) 1624 : audit [DBG] from='client.? v1:172.21.15.121:0/3527974398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:07.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:07.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:07 smithi169 bash[19604]: cluster 2024-02-12T16:19:05.875332+0000 osd.10 (osd.10) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:07 smithi169 bash[19604]: cluster 2024-02-12T16:19:05.875426+0000 osd.10 (osd.10) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:07 smithi169 bash[19604]: cluster 2024-02-12T16:19:06.488223+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e444: 12 total, 11 up, 11 in 2024-02-12T16:19:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:07 smithi169 bash[19604]: cluster 2024-02-12T16:19:06.987149+0000 osd.11 (osd.11) 81 : cluster [DBG] 1.c scrub starts 2024-02-12T16:19:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:07 smithi169 bash[19604]: cluster 2024-02-12T16:19:07.060174+0000 osd.11 (osd.11) 82 : cluster [DBG] 1.c scrub ok 2024-02-12T16:19:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:07 smithi169 bash[19604]: audit 2024-02-12T16:19:07.144298+0000 mon.a (mon.0) 1624 : audit [DBG] from='client.? v1:172.21.15.121:0/3527974398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:07.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:07.848 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:19:07.848 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:19:07.852 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:07 smithi121 bash[18489]: cluster 2024-02-12T16:19:05.875332+0000 osd.10 (osd.10) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:07 smithi121 bash[18489]: cluster 2024-02-12T16:19:05.875426+0000 osd.10 (osd.10) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:07 smithi121 bash[18489]: cluster 2024-02-12T16:19:06.488223+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e444: 12 total, 11 up, 11 in 2024-02-12T16:19:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:07 smithi121 bash[18489]: cluster 2024-02-12T16:19:06.987149+0000 osd.11 (osd.11) 81 : cluster [DBG] 1.c scrub starts 2024-02-12T16:19:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:07 smithi121 bash[18489]: cluster 2024-02-12T16:19:07.060174+0000 osd.11 (osd.11) 82 : cluster [DBG] 1.c scrub ok 2024-02-12T16:19:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:07 smithi121 bash[18489]: audit 2024-02-12T16:19:07.144298+0000 mon.a (mon.0) 1624 : audit [DBG] from='client.? v1:172.21.15.121:0/3527974398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:07.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:07.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:07.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:08.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:08.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:08.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:08.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:08.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:08.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:08.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:08.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:08.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:08.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:08.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:08.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:08.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:08.692 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:08.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:08.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:08 smithi169 bash[19604]: cluster 2024-02-12T16:19:07.786436+0000 mgr.y (mgr.14146) 1016 : cluster [DBG] pgmap v1359: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:08.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:08 smithi123 bash[19119]: cluster 2024-02-12T16:19:07.786436+0000 mgr.y (mgr.14146) 1016 : cluster [DBG] pgmap v1359: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:08.834 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:08.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:08.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:08 smithi121 bash[18489]: cluster 2024-02-12T16:19:07.786436+0000 mgr.y (mgr.14146) 1016 : cluster [DBG] pgmap v1359: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:08.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:08.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:08.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:19:09.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:19:09.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:09.114 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:09.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:09.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:09.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:09.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:09.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:09.393 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:09.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:09.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:09.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:09.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:09.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:09.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:19:09.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:09.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:09 smithi169 bash[19604]: cluster 2024-02-12T16:19:08.118247+0000 osd.10 (osd.10) 62 : cluster [DBG] 1.e scrub starts 2024-02-12T16:19:09.789 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:09 smithi169 bash[19604]: cluster 2024-02-12T16:19:08.131077+0000 osd.10 (osd.10) 63 : cluster [DBG] 1.e scrub ok 2024-02-12T16:19:09.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:09.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:09.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:09 smithi123 bash[19119]: cluster 2024-02-12T16:19:08.118247+0000 osd.10 (osd.10) 62 : cluster [DBG] 1.e scrub starts 2024-02-12T16:19:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:09 smithi123 bash[19119]: cluster 2024-02-12T16:19:08.131077+0000 osd.10 (osd.10) 63 : cluster [DBG] 1.e scrub ok 2024-02-12T16:19:09.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:09 smithi121 bash[18489]: cluster 2024-02-12T16:19:08.118247+0000 osd.10 (osd.10) 62 : cluster [DBG] 1.e scrub starts 2024-02-12T16:19:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:09 smithi121 bash[18489]: cluster 2024-02-12T16:19:08.131077+0000 osd.10 (osd.10) 63 : cluster [DBG] 1.e scrub ok 2024-02-12T16:19:09.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:09.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:09.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:10.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:10.092 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:10.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:10.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:10.233 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:10.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:10.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:10.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:10.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:10.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:10.486 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:19:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:10.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:10.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:10 smithi123 bash[19119]: cluster 2024-02-12T16:19:09.787418+0000 mgr.y (mgr.14146) 1017 : cluster [DBG] pgmap v1360: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:10 smithi123 bash[19119]: audit 2024-02-12T16:19:10.016078+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? v1:172.21.15.121:0/1551576846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:19:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:10 smithi123 bash[19119]: cluster 2024-02-12T16:19:10.466728+0000 mon.a (mon.0) 1626 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:10 smithi123 bash[19119]: audit 2024-02-12T16:19:10.472447+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? v1:172.21.15.121:0/1551576846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:19:10.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:10 smithi123 bash[19119]: cluster 2024-02-12T16:19:10.472512+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e445: 12 total, 11 up, 11 in 2024-02-12T16:19:10.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:10.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:10.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:10.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:10.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:10 smithi169 bash[19604]: cluster 2024-02-12T16:19:09.787418+0000 mgr.y (mgr.14146) 1017 : cluster [DBG] pgmap v1360: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:10 smithi169 bash[19604]: audit 2024-02-12T16:19:10.016078+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? v1:172.21.15.121:0/1551576846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:19:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:10 smithi169 bash[19604]: cluster 2024-02-12T16:19:10.466728+0000 mon.a (mon.0) 1626 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:10 smithi169 bash[19604]: audit 2024-02-12T16:19:10.472447+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? v1:172.21.15.121:0/1551576846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:19:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:10 smithi169 bash[19604]: cluster 2024-02-12T16:19:10.472512+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e445: 12 total, 11 up, 11 in 2024-02-12T16:19:10.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:10 smithi121 bash[18489]: cluster 2024-02-12T16:19:09.787418+0000 mgr.y (mgr.14146) 1017 : cluster [DBG] pgmap v1360: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:10 smithi121 bash[18489]: audit 2024-02-12T16:19:10.016078+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? v1:172.21.15.121:0/1551576846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:19:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:10 smithi121 bash[18489]: cluster 2024-02-12T16:19:10.466728+0000 mon.a (mon.0) 1626 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:10 smithi121 bash[18489]: audit 2024-02-12T16:19:10.472447+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? v1:172.21.15.121:0/1551576846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:19:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:10 smithi121 bash[18489]: cluster 2024-02-12T16:19:10.472512+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e445: 12 total, 11 up, 11 in 2024-02-12T16:19:10.928 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:10.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:11.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:11.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:19:11.071 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:11.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:11.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:11.206 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:11.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:11.310 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:11.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:11.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:11.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:11.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:11.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:11.530 INFO:teuthology.orchestra.run.smithi121.stderr:set 1.d pg_upmap_items mapping to [6->0,9->10] 2024-02-12T16:19:11.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:11.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:11.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:11 smithi123 bash[19119]: audit 2024-02-12T16:19:10.841060+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? v1:172.21.15.121:0/4051407687' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.d", "id": [6, 0, 9, 10]}]: dispatch 2024-02-12T16:19:11.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:11.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:11 smithi169 bash[19604]: audit 2024-02-12T16:19:10.841060+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? v1:172.21.15.121:0/4051407687' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.d", "id": [6, 0, 9, 10]}]: dispatch 2024-02-12T16:19:11.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:11.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:11.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:11.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:11 smithi121 bash[18489]: audit 2024-02-12T16:19:10.841060+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? v1:172.21.15.121:0/4051407687' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.d", "id": [6, 0, 9, 10]}]: dispatch 2024-02-12T16:19:11.919 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:11.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:12.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:12.060 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:12.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:12.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:12.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:12.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:12.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:12.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:12.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:12.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:12.493 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:12.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:12.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:12.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:12.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:12.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:12.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:12.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:12 smithi169 bash[19604]: audit 2024-02-12T16:19:11.520536+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? v1:172.21.15.121:0/4051407687' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.d", "id": [6, 0, 9, 10]}]': finished 2024-02-12T16:19:12.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:12 smithi169 bash[19604]: cluster 2024-02-12T16:19:11.520659+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e446: 12 total, 11 up, 11 in 2024-02-12T16:19:12.782 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:12 smithi169 bash[19604]: cluster 2024-02-12T16:19:11.788253+0000 mgr.y (mgr.14146) 1018 : cluster [DBG] pgmap v1363: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:12.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:12.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:12 smithi123 bash[19119]: audit 2024-02-12T16:19:11.520536+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? v1:172.21.15.121:0/4051407687' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.d", "id": [6, 0, 9, 10]}]': finished 2024-02-12T16:19:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:12 smithi123 bash[19119]: cluster 2024-02-12T16:19:11.520659+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e446: 12 total, 11 up, 11 in 2024-02-12T16:19:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:12 smithi123 bash[19119]: cluster 2024-02-12T16:19:11.788253+0000 mgr.y (mgr.14146) 1018 : cluster [DBG] pgmap v1363: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:12.896 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:12 smithi121 bash[18489]: audit 2024-02-12T16:19:11.520536+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? v1:172.21.15.121:0/4051407687' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.d", "id": [6, 0, 9, 10]}]': finished 2024-02-12T16:19:12.897 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:12 smithi121 bash[18489]: cluster 2024-02-12T16:19:11.520659+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e446: 12 total, 11 up, 11 in 2024-02-12T16:19:12.897 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:12 smithi121 bash[18489]: cluster 2024-02-12T16:19:11.788253+0000 mgr.y (mgr.14146) 1018 : cluster [DBG] pgmap v1363: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:12.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:13.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:13.064 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:13.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:13.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:13.205 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:13.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:13.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:19:13.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:13.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:13.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:13.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:13.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:13.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:13.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:13.626 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:13.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:13.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:13.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:13.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:13.874 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:13.909 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:13.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:14.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:14.052 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:14.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:14.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:14.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:14.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:14.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:14.329 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:14.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:14.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:14.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:14.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:14.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:14.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:14.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:14.755 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:14.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:14.824 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:14.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:14.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:14.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:15.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:15.042 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:15.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:14 smithi123 bash[19119]: cluster 2024-02-12T16:19:13.789274+0000 mgr.y (mgr.14146) 1019 : cluster [DBG] pgmap v1364: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:15.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:15.156 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:14 smithi121 bash[18489]: cluster 2024-02-12T16:19:13.789274+0000 mgr.y (mgr.14146) 1019 : cluster [DBG] pgmap v1364: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:15.181 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:15.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:15.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:14 smithi169 bash[19604]: cluster 2024-02-12T16:19:13.789274+0000 mgr.y (mgr.14146) 1019 : cluster [DBG] pgmap v1364: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:15.324 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:15.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:15.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:15.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:15.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:15.604 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:15.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:15.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:15.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:15.746 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:15.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:15.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:15.840 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:15.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:15.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:15.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:15.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:16.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:16.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:16.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:16.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:16.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:16.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:16.456 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:16.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:16.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:16.598 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:16.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:16.692 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:16 smithi121 bash[18489]: cluster 2024-02-12T16:19:15.475374+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e447: 12 total, 11 up, 11 in 2024-02-12T16:19:16.692 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:16 smithi121 bash[18489]: cluster 2024-02-12T16:19:15.790243+0000 mgr.y (mgr.14146) 1020 : cluster [DBG] pgmap v1366: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:16.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:16.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:16 smithi169 bash[19604]: cluster 2024-02-12T16:19:15.475374+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e447: 12 total, 11 up, 11 in 2024-02-12T16:19:16.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:16 smithi169 bash[19604]: cluster 2024-02-12T16:19:15.790243+0000 mgr.y (mgr.14146) 1020 : cluster [DBG] pgmap v1366: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:16.740 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:16.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:16.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:16.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:16.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:16.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:16 smithi123 bash[19119]: cluster 2024-02-12T16:19:15.475374+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e447: 12 total, 11 up, 11 in 2024-02-12T16:19:16.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:16 smithi123 bash[19119]: cluster 2024-02-12T16:19:15.790243+0000 mgr.y (mgr.14146) 1020 : cluster [DBG] pgmap v1366: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:16.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:17.022 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:17.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:17.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:17.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:17.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:17.271 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:17.308 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:17.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:17.327 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11] out_osds: [8] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:19:17.327 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:19:17.328 INFO:tasks.thrashosds.thrasher:Adding osd 8 2024-02-12T16:19:17.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd in 8 2024-02-12T16:19:17.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:17.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:17.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:17.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:17.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:17.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:17.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:17.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:17.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:17.825 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:17.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:17.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:18.011 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:18.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:18.109 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:18.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:18.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:18.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:18.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:18.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:18.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:18.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:18.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:18.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:18.578 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:18.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:18.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:18.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:18.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:18.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:18.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:18.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:18.977 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:18 smithi121 bash[18489]: cluster 2024-02-12T16:19:17.791305+0000 mgr.y (mgr.14146) 1021 : cluster [DBG] pgmap v1367: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:19.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:19.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:19.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:18 smithi123 bash[19119]: cluster 2024-02-12T16:19:17.791305+0000 mgr.y (mgr.14146) 1021 : cluster [DBG] pgmap v1367: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:19.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:19.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:19.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:19.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:18 smithi169 bash[19604]: cluster 2024-02-12T16:19:17.791305+0000 mgr.y (mgr.14146) 1021 : cluster [DBG] pgmap v1367: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:19.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:19.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:19.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:19.565 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:19.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:19.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:19.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:19.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:19.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:19.983 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:19.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:20.086 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:20.120 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:20.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:20.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:20.263 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:20.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:20.270 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:19:20.270 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:19:20.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:20.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:20.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:20.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:20.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:20.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:20.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:20.676 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:20.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:20.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:20.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:20.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:20.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:20.935 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:20 smithi123 bash[19119]: cluster 2024-02-12T16:19:19.529069+0000 osd.8 (osd.8) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:20.935 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:20 smithi123 bash[19119]: cluster 2024-02-12T16:19:19.529176+0000 osd.8 (osd.8) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:20.935 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:20 smithi123 bash[19119]: cluster 2024-02-12T16:19:19.792384+0000 mgr.y (mgr.14146) 1022 : cluster [DBG] pgmap v1368: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:20.935 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:20 smithi123 bash[19119]: cluster 2024-02-12T16:19:19.862821+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e448: 12 total, 11 up, 11 in 2024-02-12T16:19:20.935 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:20 smithi123 bash[19119]: cluster 2024-02-12T16:19:20.482147+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e449: 12 total, 11 up, 11 in 2024-02-12T16:19:20.958 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:20.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:21.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:21.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:21.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:20 smithi121 bash[18489]: cluster 2024-02-12T16:19:19.529069+0000 osd.8 (osd.8) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:20 smithi121 bash[18489]: cluster 2024-02-12T16:19:19.529176+0000 osd.8 (osd.8) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:20 smithi121 bash[18489]: cluster 2024-02-12T16:19:19.792384+0000 mgr.y (mgr.14146) 1022 : cluster [DBG] pgmap v1368: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:20 smithi121 bash[18489]: cluster 2024-02-12T16:19:19.862821+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e448: 12 total, 11 up, 11 in 2024-02-12T16:19:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:20 smithi121 bash[18489]: cluster 2024-02-12T16:19:20.482147+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e449: 12 total, 11 up, 11 in 2024-02-12T16:19:21.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:21.217 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:20 smithi169 bash[19604]: cluster 2024-02-12T16:19:19.529069+0000 osd.8 (osd.8) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:21.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:20 smithi169 bash[19604]: cluster 2024-02-12T16:19:19.529176+0000 osd.8 (osd.8) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:21.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:20 smithi169 bash[19604]: cluster 2024-02-12T16:19:19.792384+0000 mgr.y (mgr.14146) 1022 : cluster [DBG] pgmap v1368: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:21.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:20 smithi169 bash[19604]: cluster 2024-02-12T16:19:19.862821+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e448: 12 total, 11 up, 11 in 2024-02-12T16:19:21.218 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:20 smithi169 bash[19604]: cluster 2024-02-12T16:19:20.482147+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e449: 12 total, 11 up, 11 in 2024-02-12T16:19:21.245 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:21.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:21.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:21.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:21.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:21.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:21.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:21.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:21.588 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:19:21.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:21.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:21.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:21.768 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:21.796 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:21.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:21.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:21.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:21.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:22.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:22.081 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:22.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:22.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:22.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:22.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:22.276 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:19:22.296 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:19:22.318 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:19:22.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:22.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:22.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:22.358 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:19:22.384 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:19:22.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:19:22.446 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:19:22.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:22.480 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:19:22.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:22.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:22.506 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:19:22.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:19:22.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:19:22.555 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.555 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.555 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:19:22.555 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.556 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.556 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.556 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.556 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.556 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.556 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.556 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:19:22.600 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:22.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:22.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:22.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:22.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:22.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:22.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:22.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:22.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:23.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:23.061 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:23.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:23.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:22 smithi123 bash[19119]: cluster 2024-02-12T16:19:21.793505+0000 mgr.y (mgr.14146) 1023 : cluster [DBG] pgmap v1371: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:23.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:22 smithi123 bash[19119]: audit 2024-02-12T16:19:22.246301+0000 mon.a (mon.0) 1635 : audit [DBG] from='client.? v1:172.21.15.121:0/1301972450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:22 smithi121 bash[18489]: cluster 2024-02-12T16:19:21.793505+0000 mgr.y (mgr.14146) 1023 : cluster [DBG] pgmap v1371: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:22 smithi121 bash[18489]: audit 2024-02-12T16:19:22.246301+0000 mon.a (mon.0) 1635 : audit [DBG] from='client.? v1:172.21.15.121:0/1301972450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:23.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:23.178 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:22 smithi169 bash[19604]: cluster 2024-02-12T16:19:21.793505+0000 mgr.y (mgr.14146) 1023 : cluster [DBG] pgmap v1371: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:23.178 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:22 smithi169 bash[19604]: audit 2024-02-12T16:19:22.246301+0000 mon.a (mon.0) 1635 : audit [DBG] from='client.? v1:172.21.15.121:0/1301972450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:23.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:23.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:23.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:23.345 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:23.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:23.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:23.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:23.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:19:23.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:19:23.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:23.626 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:23.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:23.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:23.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:23.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:23.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:23.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:23.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:24.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:24.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:24.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:24.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:24.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:24.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:24.328 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:24.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:24.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:24.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:24.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:24.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:24.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:24.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:24.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:24.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:24.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:24.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:24.887 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:19:24.888 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:24.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:24.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:25.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:25.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:24 smithi121 bash[18489]: cluster 2024-02-12T16:19:23.794516+0000 mgr.y (mgr.14146) 1024 : cluster [DBG] pgmap v1372: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:25.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:24 smithi121 bash[18489]: audit 2024-02-12T16:19:24.129126+0000 mon.a (mon.0) 1636 : audit [INF] from='client.25631 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:25.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:24 smithi121 bash[18489]: audit 2024-02-12T16:19:24.130558+0000 mon.b (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.121:0/3420083926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:25.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:24 smithi123 bash[19119]: cluster 2024-02-12T16:19:23.794516+0000 mgr.y (mgr.14146) 1024 : cluster [DBG] pgmap v1372: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:25.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:24 smithi123 bash[19119]: audit 2024-02-12T16:19:24.129126+0000 mon.a (mon.0) 1636 : audit [INF] from='client.25631 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:25.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:24 smithi123 bash[19119]: audit 2024-02-12T16:19:24.130558+0000 mon.b (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.121:0/3420083926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:25.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:25.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:24 smithi169 bash[19604]: cluster 2024-02-12T16:19:23.794516+0000 mgr.y (mgr.14146) 1024 : cluster [DBG] pgmap v1372: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:24 smithi169 bash[19604]: audit 2024-02-12T16:19:24.129126+0000 mon.a (mon.0) 1636 : audit [INF] from='client.25631 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:24 smithi169 bash[19604]: audit 2024-02-12T16:19:24.130558+0000 mon.b (mon.2) 130 : audit [INF] from='client.? v1:172.21.15.121:0/3420083926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:25.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:25.433 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:25.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:25.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:19:25.572 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:25.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:25.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:25.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:25.887 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:25 smithi121 bash[18489]: cluster 2024-02-12T16:19:24.867297+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:25.887 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:25 smithi121 bash[18489]: audit 2024-02-12T16:19:24.874828+0000 mon.a (mon.0) 1638 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:19:25.887 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:25 smithi121 bash[18489]: cluster 2024-02-12T16:19:24.875014+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e450: 12 total, 11 up, 11 in 2024-02-12T16:19:25.887 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:25 smithi121 bash[18489]: cluster 2024-02-12T16:19:25.483635+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e451: 12 total, 11 up, 11 in 2024-02-12T16:19:25.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:25.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:25.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:26.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:25 smithi123 bash[19119]: cluster 2024-02-12T16:19:24.867297+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:25 smithi123 bash[19119]: audit 2024-02-12T16:19:24.874828+0000 mon.a (mon.0) 1638 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:19:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:25 smithi123 bash[19119]: cluster 2024-02-12T16:19:24.875014+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e450: 12 total, 11 up, 11 in 2024-02-12T16:19:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:25 smithi123 bash[19119]: cluster 2024-02-12T16:19:25.483635+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e451: 12 total, 11 up, 11 in 2024-02-12T16:19:26.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:26.142 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:25 smithi169 bash[19604]: cluster 2024-02-12T16:19:24.867297+0000 mon.a (mon.0) 1637 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:26.142 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:25 smithi169 bash[19604]: audit 2024-02-12T16:19:24.874828+0000 mon.a (mon.0) 1638 : audit [INF] from='client.25631 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:19:26.142 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:25 smithi169 bash[19604]: cluster 2024-02-12T16:19:24.875014+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e450: 12 total, 11 up, 11 in 2024-02-12T16:19:26.142 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:25 smithi169 bash[19604]: cluster 2024-02-12T16:19:25.483635+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e451: 12 total, 11 up, 11 in 2024-02-12T16:19:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:26.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:26.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:26.283 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:26.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:26.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:26.389 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:19:26.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:26.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:26.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:26.566 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:26.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:26.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:26.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:26.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:26.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:26.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:26.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:26.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:26.971 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:26 smithi123 bash[19119]: cluster 2024-02-12T16:19:25.795388+0000 mgr.y (mgr.14146) 1025 : cluster [DBG] pgmap v1375: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:26.971 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:26 smithi123 bash[19119]: cluster 2024-02-12T16:19:26.267615+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:26.972 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:26 smithi123 bash[19119]: cluster 2024-02-12T16:19:26.267709+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:26.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:27.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:27.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:27.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:26 smithi121 bash[18489]: cluster 2024-02-12T16:19:25.795388+0000 mgr.y (mgr.14146) 1025 : cluster [DBG] pgmap v1375: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:26 smithi121 bash[18489]: cluster 2024-02-12T16:19:26.267615+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:26 smithi121 bash[18489]: cluster 2024-02-12T16:19:26.267709+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:27.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:27.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:26 smithi169 bash[19604]: cluster 2024-02-12T16:19:25.795388+0000 mgr.y (mgr.14146) 1025 : cluster [DBG] pgmap v1375: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:27.253 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:26 smithi169 bash[19604]: cluster 2024-02-12T16:19:26.267615+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:27.253 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:26 smithi169 bash[19604]: cluster 2024-02-12T16:19:26.267709+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:27.277 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:27.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:27.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:27.415 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:27.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:27.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:27.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:27.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:19:27.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:27.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:27.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.801 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:27 smithi121 bash[18489]: debug 2024-02-12T16:19:27.667+0000 7fbba85de700 -1 mon.a@0(leader).osd e452 definitely_dead 0 2024-02-12T16:19:27.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:27.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:27.911 INFO:teuthology.orchestra.run.smithi121.stderr:marked in osd.8. 2024-02-12T16:19:27.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:27.956 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:27.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:27.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:28.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:28.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:27 smithi123 bash[19119]: cluster 2024-02-12T16:19:26.894736+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e452: 12 total, 11 up, 11 in 2024-02-12T16:19:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:27 smithi123 bash[19119]: audit 2024-02-12T16:19:27.673854+0000 mon.a (mon.0) 1642 : audit [DBG] from='client.? v1:172.21.15.121:0/2744448598' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-02-12T16:19:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:27 smithi121 bash[18489]: cluster 2024-02-12T16:19:26.894736+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e452: 12 total, 11 up, 11 in 2024-02-12T16:19:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:27 smithi121 bash[18489]: audit 2024-02-12T16:19:27.673854+0000 mon.a (mon.0) 1642 : audit [DBG] from='client.? v1:172.21.15.121:0/2744448598' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-02-12T16:19:28.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:28.217 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:27 smithi169 bash[19604]: cluster 2024-02-12T16:19:26.894736+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e452: 12 total, 11 up, 11 in 2024-02-12T16:19:28.217 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:27 smithi169 bash[19604]: audit 2024-02-12T16:19:27.673854+0000 mon.a (mon.0) 1642 : audit [DBG] from='client.? v1:172.21.15.121:0/2744448598' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-02-12T16:19:28.242 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:28.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:28.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:28.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:28.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:28.520 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:28.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:28.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:28.661 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:28.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:28.729 INFO:tasks.thrashosds.thrasher:Added osd 8 2024-02-12T16:19:28.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:28.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:28.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:28.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:28.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:28 smithi169 bash[19604]: cluster 2024-02-12T16:19:27.796071+0000 mgr.y (mgr.14146) 1026 : cluster [DBG] pgmap v1377: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:28.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:28 smithi169 bash[19604]: audit 2024-02-12T16:19:27.903403+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.121:0/2744448598' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-02-12T16:19:28.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:28 smithi169 bash[19604]: cluster 2024-02-12T16:19:27.903472+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e453: 12 total, 11 up, 12 in 2024-02-12T16:19:28.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:28.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:29.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:29.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:29.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:29.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:28 smithi121 bash[18489]: cluster 2024-02-12T16:19:27.796071+0000 mgr.y (mgr.14146) 1026 : cluster [DBG] pgmap v1377: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:29.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:28 smithi121 bash[18489]: audit 2024-02-12T16:19:27.903403+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.121:0/2744448598' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-02-12T16:19:29.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:28 smithi121 bash[18489]: cluster 2024-02-12T16:19:27.903472+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e453: 12 total, 11 up, 12 in 2024-02-12T16:19:29.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:29.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:29.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:29.241 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:28 smithi123 bash[19119]: cluster 2024-02-12T16:19:27.796071+0000 mgr.y (mgr.14146) 1026 : cluster [DBG] pgmap v1377: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:29.241 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:28 smithi123 bash[19119]: audit 2024-02-12T16:19:27.903403+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.121:0/2744448598' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-02-12T16:19:29.241 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:28 smithi123 bash[19119]: cluster 2024-02-12T16:19:27.903472+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e453: 12 total, 11 up, 12 in 2024-02-12T16:19:29.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:29.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:29.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:29.478 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:19:29.478 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:19:29.478 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:19:29.478 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:19:29.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:29.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:29.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:29.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:29.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:29.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:29.776 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:29.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:29.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:29.920 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:29.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:30.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:30.036 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:29 smithi123 bash[19119]: cluster 2024-02-12T16:19:28.141985+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:30.036 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:29 smithi123 bash[19119]: cluster 2024-02-12T16:19:28.142039+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:30.036 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:29 smithi123 bash[19119]: cluster 2024-02-12T16:19:28.911929+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e454: 12 total, 11 up, 12 in 2024-02-12T16:19:30.036 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:29 smithi123 bash[19119]: cluster 2024-02-12T16:19:29.180540+0000 osd.11 (osd.11) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:30.037 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:29 smithi123 bash[19119]: cluster 2024-02-12T16:19:29.180629+0000 osd.11 (osd.11) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:30.060 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:30.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:30.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:30.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:29 smithi121 bash[18489]: cluster 2024-02-12T16:19:28.141985+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:30.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:29 smithi121 bash[18489]: cluster 2024-02-12T16:19:28.142039+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:30.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:29 smithi121 bash[18489]: cluster 2024-02-12T16:19:28.911929+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e454: 12 total, 11 up, 12 in 2024-02-12T16:19:30.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:29 smithi121 bash[18489]: cluster 2024-02-12T16:19:29.180540+0000 osd.11 (osd.11) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:30.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:29 smithi121 bash[18489]: cluster 2024-02-12T16:19:29.180629+0000 osd.11 (osd.11) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:30.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:30.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:19:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:30.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:29 smithi169 bash[19604]: cluster 2024-02-12T16:19:28.141985+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:29 smithi169 bash[19604]: cluster 2024-02-12T16:19:28.142039+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:29 smithi169 bash[19604]: cluster 2024-02-12T16:19:28.911929+0000 mon.a (mon.0) 1645 : cluster [DBG] osdmap e454: 12 total, 11 up, 12 in 2024-02-12T16:19:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:29 smithi169 bash[19604]: cluster 2024-02-12T16:19:29.180540+0000 osd.11 (osd.11) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:29 smithi169 bash[19604]: cluster 2024-02-12T16:19:29.180629+0000 osd.11 (osd.11) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:30.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:30.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:30.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:30.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:30.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:30.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:30.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:30.622 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:30.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:30.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:30.762 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:30.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:30.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:30.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:30.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:31.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:31.046 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:31.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:31.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:31.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:30 smithi121 bash[18489]: cluster 2024-02-12T16:19:29.797010+0000 mgr.y (mgr.14146) 1027 : cluster [DBG] pgmap v1380: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:30 smithi121 bash[18489]: cluster 2024-02-12T16:19:29.924707+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e455: 12 total, 11 up, 12 in 2024-02-12T16:19:31.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:31.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:31.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:30 smithi169 bash[19604]: cluster 2024-02-12T16:19:29.797010+0000 mgr.y (mgr.14146) 1027 : cluster [DBG] pgmap v1380: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:30 smithi169 bash[19604]: cluster 2024-02-12T16:19:29.924707+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e455: 12 total, 11 up, 12 in 2024-02-12T16:19:31.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:31.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:31.343 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:30 smithi123 bash[19119]: cluster 2024-02-12T16:19:29.797010+0000 mgr.y (mgr.14146) 1027 : cluster [DBG] pgmap v1380: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:30 smithi123 bash[19119]: cluster 2024-02-12T16:19:29.924707+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e455: 12 total, 11 up, 12 in 2024-02-12T16:19:31.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:31.470 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:31.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:31.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:31.612 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:31.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:31.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:31.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:31.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:31.860 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:31.896 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:31.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:32.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:32.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:31 smithi123 bash[19119]: audit 2024-02-12T16:19:31.238848+0000 mon.a (mon.0) 1647 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:19:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:31 smithi123 bash[19119]: audit 2024-02-12T16:19:31.546029+0000 mon.a (mon.0) 1648 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:32.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:32.183 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:32.185 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:31 smithi169 bash[19604]: audit 2024-02-12T16:19:31.238848+0000 mon.a (mon.0) 1647 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:19:32.185 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:31 smithi169 bash[19604]: audit 2024-02-12T16:19:31.546029+0000 mon.a (mon.0) 1648 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:32.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:32.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:31 smithi121 bash[18489]: audit 2024-02-12T16:19:31.238848+0000 mon.a (mon.0) 1647 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:19:32.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:31 smithi121 bash[18489]: audit 2024-02-12T16:19:31.546029+0000 mon.a (mon.0) 1648 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:32.324 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:32.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:32.356 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:32.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:32.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:32.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:32.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:32.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:32.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.732 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:32.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:32.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:32.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:32.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:32.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:33.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:33.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:33.118 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:32 smithi121 bash[18489]: cluster 2024-02-12T16:19:31.797663+0000 mgr.y (mgr.14146) 1028 : cluster [DBG] pgmap v1382: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:33.119 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:32 smithi121 bash[18489]: cluster 2024-02-12T16:19:32.232194+0000 osd.1 (osd.1) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:33.119 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:32 smithi121 bash[18489]: cluster 2024-02-12T16:19:32.232287+0000 osd.1 (osd.1) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:33.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:33.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:33.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:32 smithi123 bash[19119]: cluster 2024-02-12T16:19:31.797663+0000 mgr.y (mgr.14146) 1028 : cluster [DBG] pgmap v1382: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:33.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:32 smithi123 bash[19119]: cluster 2024-02-12T16:19:32.232194+0000 osd.1 (osd.1) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:33.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:32 smithi123 bash[19119]: cluster 2024-02-12T16:19:32.232287+0000 osd.1 (osd.1) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:33.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:32 smithi169 bash[19604]: cluster 2024-02-12T16:19:31.797663+0000 mgr.y (mgr.14146) 1028 : cluster [DBG] pgmap v1382: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:32 smithi169 bash[19604]: cluster 2024-02-12T16:19:32.232194+0000 osd.1 (osd.1) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:32 smithi169 bash[19604]: cluster 2024-02-12T16:19:32.232287+0000 osd.1 (osd.1) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:33.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:33.424 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:33.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:33.564 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:33.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:33.705 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:33.730 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:19:33.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:19:33.731 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-12T16:19:33.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:19:33.808 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:33.847 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:33.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:33.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:33.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:34.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:34.105 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:33 smithi121 bash[18489]: cluster 2024-02-12T16:19:32.944415+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e456: 12 total, 11 up, 12 in 2024-02-12T16:19:34.128 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:34.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:33 smithi123 bash[19119]: cluster 2024-02-12T16:19:32.944415+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e456: 12 total, 11 up, 12 in 2024-02-12T16:19:34.232 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:34.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:33 smithi169 bash[19604]: cluster 2024-02-12T16:19:32.944415+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e456: 12 total, 11 up, 12 in 2024-02-12T16:19:34.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:34.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:34.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:34.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:34.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:34.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:34.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:34.696 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:34.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:34.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:34.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:34.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:34.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:34.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:34.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:34.994 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:19:35.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:35.118 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:35.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:35.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:34 smithi123 bash[19119]: cluster 2024-02-12T16:19:33.798377+0000 mgr.y (mgr.14146) 1029 : cluster [DBG] pgmap v1384: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:35.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:35.251 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:34 smithi121 bash[18489]: cluster 2024-02-12T16:19:33.798377+0000 mgr.y (mgr.14146) 1029 : cluster [DBG] pgmap v1384: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:35.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:35.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:34 smithi169 bash[19604]: cluster 2024-02-12T16:19:33.798377+0000 mgr.y (mgr.14146) 1029 : cluster [DBG] pgmap v1384: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:35.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:35.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:35.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:35.536 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:35.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:35.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:35.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:35.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:35.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:35.814 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:35.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:35.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:35.954 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:35.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:36.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:36.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:36.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:36.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:36.226 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:36.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:36.317 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:19:36.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:36.335 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:19:36.358 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:19:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:36.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:36.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:36.379 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:19:36.403 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:19:36.429 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:19:36.449 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:19:36.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:36.472 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:19:36.501 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:36.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:36.504 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:19:36.527 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:19:36.549 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.550 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:19:36.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:36 smithi121 bash[18489]: cluster 2024-02-12T16:19:35.486916+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e457: 12 total, 11 up, 12 in 2024-02-12T16:19:36.592 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:36 smithi121 bash[18489]: cluster 2024-02-12T16:19:35.799366+0000 mgr.y (mgr.14146) 1030 : cluster [DBG] pgmap v1386: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:36.592 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:36 smithi121 bash[18489]: audit 2024-02-12T16:19:36.292289+0000 mon.a (mon.0) 1651 : audit [DBG] from='client.? v1:172.21.15.121:0/1254346093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:36.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:36.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:36.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:36.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:36.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:36.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:36.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:36 smithi169 bash[19604]: cluster 2024-02-12T16:19:35.486916+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e457: 12 total, 11 up, 12 in 2024-02-12T16:19:36.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:36 smithi169 bash[19604]: cluster 2024-02-12T16:19:35.799366+0000 mgr.y (mgr.14146) 1030 : cluster [DBG] pgmap v1386: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:36.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:36 smithi169 bash[19604]: audit 2024-02-12T16:19:36.292289+0000 mon.a (mon.0) 1651 : audit [DBG] from='client.? v1:172.21.15.121:0/1254346093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:36.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:36 smithi123 bash[19119]: cluster 2024-02-12T16:19:35.486916+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e457: 12 total, 11 up, 12 in 2024-02-12T16:19:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:36 smithi123 bash[19119]: cluster 2024-02-12T16:19:35.799366+0000 mgr.y (mgr.14146) 1030 : cluster [DBG] pgmap v1386: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:36 smithi123 bash[19119]: audit 2024-02-12T16:19:36.292289+0000 mon.a (mon.0) 1651 : audit [DBG] from='client.? v1:172.21.15.121:0/1254346093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:36.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:36.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:37.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:37.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:37.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:37.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:37.201 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:37.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:37.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:37.343 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:37.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:19:37.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:19:37.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:37.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:37.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:37.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:37.623 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:37.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:37 smithi123 bash[19119]: audit 2024-02-12T16:19:36.533436+0000 mon.a (mon.0) 1652 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:37 smithi123 bash[19119]: audit 2024-02-12T16:19:36.823070+0000 mon.a (mon.0) 1653 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:19:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:37 smithi123 bash[19119]: audit 2024-02-12T16:19:36.824775+0000 mon.a (mon.0) 1654 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:19:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:37 smithi123 bash[19119]: audit 2024-02-12T16:19:36.833566+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:37.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:37.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:37.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:37.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:37 smithi169 bash[19604]: audit 2024-02-12T16:19:36.533436+0000 mon.a (mon.0) 1652 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:37.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:37 smithi169 bash[19604]: audit 2024-02-12T16:19:36.823070+0000 mon.a (mon.0) 1653 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:19:37.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:37 smithi169 bash[19604]: audit 2024-02-12T16:19:36.824775+0000 mon.a (mon.0) 1654 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:19:37.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:37 smithi169 bash[19604]: audit 2024-02-12T16:19:36.833566+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:37.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:37.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:37.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:37 smithi121 bash[18489]: audit 2024-02-12T16:19:36.533436+0000 mon.a (mon.0) 1652 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:37 smithi121 bash[18489]: audit 2024-02-12T16:19:36.823070+0000 mon.a (mon.0) 1653 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:19:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:37 smithi121 bash[18489]: audit 2024-02-12T16:19:36.824775+0000 mon.a (mon.0) 1654 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:19:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:37 smithi121 bash[18489]: audit 2024-02-12T16:19:36.833566+0000 mon.a (mon.0) 1655 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:38.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:38.042 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:38.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:38.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:38.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:38.201 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:19:38.202 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:19:38.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:38.320 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:38.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:38.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:38.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:38.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:38.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:38.560 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:19:38.586 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:38.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:38.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:38.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:38.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:38 smithi169 bash[19604]: cluster 2024-02-12T16:19:37.800348+0000 mgr.y (mgr.14146) 1031 : cluster [DBG] pgmap v1387: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:38 smithi169 bash[19604]: audit 2024-02-12T16:19:37.884838+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? v1:172.21.15.121:0/4270734458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:19:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:38 smithi169 bash[19604]: cluster 2024-02-12T16:19:38.010524+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:38 smithi169 bash[19604]: cluster 2024-02-12T16:19:38.010628+0000 osd.6 (osd.6) 69 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:38.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:38.868 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:38.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:38 smithi123 bash[19119]: cluster 2024-02-12T16:19:37.800348+0000 mgr.y (mgr.14146) 1031 : cluster [DBG] pgmap v1387: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:38 smithi123 bash[19119]: audit 2024-02-12T16:19:37.884838+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? v1:172.21.15.121:0/4270734458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:19:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:38 smithi123 bash[19119]: cluster 2024-02-12T16:19:38.010524+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:38 smithi123 bash[19119]: cluster 2024-02-12T16:19:38.010628+0000 osd.6 (osd.6) 69 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:38.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:38 smithi121 bash[18489]: cluster 2024-02-12T16:19:37.800348+0000 mgr.y (mgr.14146) 1031 : cluster [DBG] pgmap v1387: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:38 smithi121 bash[18489]: audit 2024-02-12T16:19:37.884838+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? v1:172.21.15.121:0/4270734458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:19:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:38 smithi121 bash[18489]: cluster 2024-02-12T16:19:38.010524+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:38 smithi121 bash[18489]: cluster 2024-02-12T16:19:38.010628+0000 osd.6 (osd.6) 69 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:38.911 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:19:38.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:39.011 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:39.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:39.148 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:39.251 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:39.288 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:39.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:39.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:39.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:19:39.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:39.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:39.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:39 smithi121 bash[18489]: cluster 2024-02-12T16:19:38.533573+0000 mon.a (mon.0) 1657 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:39 smithi121 bash[18489]: audit 2024-02-12T16:19:38.550546+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? v1:172.21.15.121:0/4270734458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:19:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:39 smithi121 bash[18489]: cluster 2024-02-12T16:19:38.550617+0000 mon.a (mon.0) 1659 : cluster [DBG] osdmap e458: 12 total, 11 up, 12 in 2024-02-12T16:19:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:39 smithi121 bash[18489]: cluster 2024-02-12T16:19:38.883767+0000 osd.5 (osd.5) 63 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:19:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:39 smithi121 bash[18489]: cluster 2024-02-12T16:19:38.929711+0000 osd.5 (osd.5) 64 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:19:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:39 smithi121 bash[18489]: audit 2024-02-12T16:19:39.266926+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:39.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:39.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:39 smithi169 bash[19604]: cluster 2024-02-12T16:19:38.533573+0000 mon.a (mon.0) 1657 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:39.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:39 smithi169 bash[19604]: audit 2024-02-12T16:19:38.550546+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? v1:172.21.15.121:0/4270734458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:19:39.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:39 smithi169 bash[19604]: cluster 2024-02-12T16:19:38.550617+0000 mon.a (mon.0) 1659 : cluster [DBG] osdmap e458: 12 total, 11 up, 12 in 2024-02-12T16:19:39.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:39 smithi169 bash[19604]: cluster 2024-02-12T16:19:38.883767+0000 osd.5 (osd.5) 63 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:19:39.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:39 smithi169 bash[19604]: cluster 2024-02-12T16:19:38.929711+0000 osd.5 (osd.5) 64 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:19:39.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:39 smithi169 bash[19604]: audit 2024-02-12T16:19:39.266926+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:39.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:39.777 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:39.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:39.843 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:39 smithi123 bash[19119]: cluster 2024-02-12T16:19:38.533573+0000 mon.a (mon.0) 1657 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:19:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:39 smithi123 bash[19119]: audit 2024-02-12T16:19:38.550546+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? v1:172.21.15.121:0/4270734458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:19:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:39 smithi123 bash[19119]: cluster 2024-02-12T16:19:38.550617+0000 mon.a (mon.0) 1659 : cluster [DBG] osdmap e458: 12 total, 11 up, 12 in 2024-02-12T16:19:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:39 smithi123 bash[19119]: cluster 2024-02-12T16:19:38.883767+0000 osd.5 (osd.5) 63 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:19:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:39 smithi123 bash[19119]: cluster 2024-02-12T16:19:38.929711+0000 osd.5 (osd.5) 64 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:19:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:39 smithi123 bash[19119]: audit 2024-02-12T16:19:39.266926+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:39.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:39.983 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:39.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:40.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:40.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:40.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:40.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:40.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:40.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:40.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:40.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:40.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:40.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:40.553 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:40.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:40.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:40.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:40.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:40.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:40 smithi169 bash[19604]: cluster 2024-02-12T16:19:39.801079+0000 mgr.y (mgr.14146) 1032 : cluster [DBG] pgmap v1389: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:40 smithi169 bash[19604]: cluster 2024-02-12T16:19:40.489031+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e459: 12 total, 11 up, 12 in 2024-02-12T16:19:40.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:40 smithi123 bash[19119]: cluster 2024-02-12T16:19:39.801079+0000 mgr.y (mgr.14146) 1032 : cluster [DBG] pgmap v1389: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:40.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:40 smithi123 bash[19119]: cluster 2024-02-12T16:19:40.489031+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e459: 12 total, 11 up, 12 in 2024-02-12T16:19:40.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:40.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:40 smithi121 bash[18489]: cluster 2024-02-12T16:19:39.801079+0000 mgr.y (mgr.14146) 1032 : cluster [DBG] pgmap v1389: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:40 smithi121 bash[18489]: cluster 2024-02-12T16:19:40.489031+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e459: 12 total, 11 up, 12 in 2024-02-12T16:19:40.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:40.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:41.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:41.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:41.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:41.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:41.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:41.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:41.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:41.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:19:41.388 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:41.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:41.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:41.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:41.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:41.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:41.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:41.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:41.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:41.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:41 smithi121 bash[18489]: cluster 2024-02-12T16:19:40.102301+0000 osd.2 (osd.2) 50 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:19:41.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:41 smithi121 bash[18489]: cluster 2024-02-12T16:19:40.117564+0000 osd.9 (osd.9) 63 : cluster [DBG] 1.a scrub starts 2024-02-12T16:19:41.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:41 smithi121 bash[18489]: cluster 2024-02-12T16:19:40.134173+0000 osd.9 (osd.9) 64 : cluster [DBG] 1.a scrub ok 2024-02-12T16:19:41.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:41 smithi121 bash[18489]: cluster 2024-02-12T16:19:40.530121+0000 osd.2 (osd.2) 51 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:19:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:41.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:41 smithi123 bash[19119]: cluster 2024-02-12T16:19:40.102301+0000 osd.2 (osd.2) 50 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:19:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:41 smithi123 bash[19119]: cluster 2024-02-12T16:19:40.117564+0000 osd.9 (osd.9) 63 : cluster [DBG] 1.a scrub starts 2024-02-12T16:19:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:41 smithi123 bash[19119]: cluster 2024-02-12T16:19:40.134173+0000 osd.9 (osd.9) 64 : cluster [DBG] 1.a scrub ok 2024-02-12T16:19:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:41 smithi123 bash[19119]: cluster 2024-02-12T16:19:40.530121+0000 osd.2 (osd.2) 51 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:19:41.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:41.956 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:41.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:41 smithi169 bash[19604]: cluster 2024-02-12T16:19:40.102301+0000 osd.2 (osd.2) 50 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:19:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:41 smithi169 bash[19604]: cluster 2024-02-12T16:19:40.117564+0000 osd.9 (osd.9) 63 : cluster [DBG] 1.a scrub starts 2024-02-12T16:19:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:41 smithi169 bash[19604]: cluster 2024-02-12T16:19:40.134173+0000 osd.9 (osd.9) 64 : cluster [DBG] 1.a scrub ok 2024-02-12T16:19:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:41 smithi169 bash[19604]: cluster 2024-02-12T16:19:40.530121+0000 osd.2 (osd.2) 51 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:19:42.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:42.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:42.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:42.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:42.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:42.234 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:42.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:42.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:42.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:42.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:42.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:42.516 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:42.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:42.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:42.654 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:42.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:42.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:42.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:42 smithi169 bash[19604]: cluster 2024-02-12T16:19:41.122305+0000 osd.10 (osd.10) 64 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:19:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:42 smithi169 bash[19604]: cluster 2024-02-12T16:19:41.144495+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:19:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:42 smithi169 bash[19604]: cluster 2024-02-12T16:19:41.280237+0000 osd.11 (osd.11) 85 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:19:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:42 smithi169 bash[19604]: cluster 2024-02-12T16:19:41.336441+0000 osd.11 (osd.11) 86 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:19:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:42 smithi169 bash[19604]: cluster 2024-02-12T16:19:41.801754+0000 mgr.y (mgr.14146) 1033 : cluster [DBG] pgmap v1391: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:42 smithi123 bash[19119]: cluster 2024-02-12T16:19:41.122305+0000 osd.10 (osd.10) 64 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:19:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:42 smithi123 bash[19119]: cluster 2024-02-12T16:19:41.144495+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:19:42.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:42 smithi123 bash[19119]: cluster 2024-02-12T16:19:41.280237+0000 osd.11 (osd.11) 85 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:19:42.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:42 smithi123 bash[19119]: cluster 2024-02-12T16:19:41.336441+0000 osd.11 (osd.11) 86 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:19:42.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:42 smithi123 bash[19119]: cluster 2024-02-12T16:19:41.801754+0000 mgr.y (mgr.14146) 1033 : cluster [DBG] pgmap v1391: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:42.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:42.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:42 smithi121 bash[18489]: cluster 2024-02-12T16:19:41.122305+0000 osd.10 (osd.10) 64 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:19:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:42 smithi121 bash[18489]: cluster 2024-02-12T16:19:41.144495+0000 osd.10 (osd.10) 65 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:19:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:42 smithi121 bash[18489]: cluster 2024-02-12T16:19:41.280237+0000 osd.11 (osd.11) 85 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:19:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:42 smithi121 bash[18489]: cluster 2024-02-12T16:19:41.336441+0000 osd.11 (osd.11) 86 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:19:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:42 smithi121 bash[18489]: cluster 2024-02-12T16:19:41.801754+0000 mgr.y (mgr.14146) 1033 : cluster [DBG] pgmap v1391: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:42.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:42.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:43.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:43.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:43.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:43.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:43.216 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:43.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:43.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:43.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:43.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:43.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:43.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:43.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:43.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:43.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:43.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 459, 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-02-12T15:47:18.273813+0000", 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-02-12T16:19:40.482005+0000", 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "last_up_change": "2024-02-12T16:17:42.493645+0000", 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "last_in_change": "2024-02-12T16:19:27.892497+0000", 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_num": 5804032, 2024-02-12T16:19:43.659 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_set": [ 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "nodeep-scrub", 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "pglog_hardlimit", 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "purged_snapdirs", 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_deletes", 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "sortbitwise" 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_version": 28, 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_snapshot": "", 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_max": 4, 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd": 12, 2024-02-12T16:19:43.660 INFO:teuthology.orchestra.run.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "min_compat_client": "jewel", 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "require_osd_release": "pacific", 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "pools": [ 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 1, 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T15:51:35.886878+0000", 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:19:43.661 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:19:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:19:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:19:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:19:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:19:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:19:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 21, 2024-02-12T16:19:43.662 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 11, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 21, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 21, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 21, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.663 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "410", 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "410", 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:19:43.664 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.665 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "options": { 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_max": 32, 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_min": 1 2024-02-12T16:19:43.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_devicehealth": {} 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 4, 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "unique_pool_2", 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T16:13:44.637807+0000", 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:19:43.667 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 16, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 16, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 16, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 16, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 16, 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:19:43.668 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "346", 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:19:43.669 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:19:43.670 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:19:43.671 INFO:teuthology.orchestra.run.smithi121.stdout: "options": {}, 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "rados": {} 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0", 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.672 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 8, 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 334, 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6802", 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6804", 2024-02-12T16:19:43.674 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6803", 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6801/1815232143", 2024-02-12T16:19:43.675 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6802/1815232143", 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6804/1815232143", 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6803/1815232143", 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "c6140309-5703-4538-9739-276fbbdaa880", 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.1039276123046875, 2024-02-12T16:19:43.676 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 13, 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6805", 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.677 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6806", 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.678 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6808", 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6807", 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6805/2968433670", 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6806/2968433670", 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6808/2968433670", 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6807/2968433670", 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170", 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 21, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6809", 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:19:43.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6810", 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6812", 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6811", 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6809/2158187852", 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6810/2158187852", 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6812/2158187852", 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6811/2158187852", 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd", 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 0, 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.684 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 26, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 410, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 415, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6813", 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.685 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6814", 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.686 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6816", 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6815", 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6813/1415189366", 2024-02-12T16:19:43.687 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6814/1415189366", 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6816/1415189366", 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6815/1415189366", 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "exists" 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31", 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.688 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 32, 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6800", 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.689 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6801", 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6803", 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.690 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6802", 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6800/3447700123", 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6801/3447700123", 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6803/3447700123", 2024-02-12T16:19:43.691 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6802/3447700123", 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602", 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.692 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 39, 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6804", 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6805", 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6807", 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6806", 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6804/961317306", 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6805/961317306", 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6807/961317306", 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6806/961317306", 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.695 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0", 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 48, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 420, 2024-02-12T16:19:43.696 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6808", 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6809", 2024-02-12T16:19:43.697 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6811", 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6810", 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6808/1377261784", 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6809/1377261784", 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6811/1377261784", 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6810/1377261784", 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515", 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 56, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.700 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6812", 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6813", 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:19:43.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6815", 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.702 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6814", 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6812/1702613957", 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6813/1702613957", 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6815/1702613957", 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6814/1702613957", 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75", 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 65, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 310, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6800", 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6801", 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6803", 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6802", 2024-02-12T16:19:43.706 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6800/2551075225", 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6801/2551075225", 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6803/2551075225", 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6802/2551075225", 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:19:43.707 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15", 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 74, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 418, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.708 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6804", 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6805", 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.709 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6807", 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6806", 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6804/2339555071", 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6805/2339555071", 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6807/2339555071", 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6806/2339555071", 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "91f74dca-503a-45d1-9647-df4804d58515", 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.711 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 83, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 410, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6808", 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:19:43.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6809", 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.713 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6811", 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6810", 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6808/391306610", 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6809/391306610", 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6811/391306610", 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6810/391306610", 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf", 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:19:43.715 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 93, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 453, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6812", 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6813", 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.717 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6815", 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6814", 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6812/3696812305", 2024-02-12T16:19:43.718 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6813/3696812305", 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3696812305", 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6814/3696812305", 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_xinfo": [ 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.719 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:59.801623+0000", 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:32.232292+0000", 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:43.883278+0000", 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.721 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:16:56.622408+0000", 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 414 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:05.612718+0000", 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:28.142045+0000", 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.723 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:38.010636+0000", 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:26.267714+0000", 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.724 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:19.529182+0000", 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.725 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:18:52.433694+0000", 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:05.875431+0000", 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:19:43.726 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:29.180634+0000", 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap": [], 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap_items": [], 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_temp": [ 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "4.4", 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-02-12T16:19:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_temp": [], 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklist": { 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/2474849789": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/253689008": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/3531151725": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/1778611055": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/28718860": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/445290372": "2024-02-13T15:47:47.232761+0000" 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "range_blocklist": {}, 2024-02-12T16:19:43.728 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profiles": { 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "default": { 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-failure-domain": "osd", 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "k": "2", 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "m": "1", 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin": "jerasure", 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "technique": "reed_sol_van" 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps_queue": [], 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "new_removed_snaps": [], 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "new_purged_snaps": [], 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_node_flags": {}, 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "device_class_flags": {}, 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": { 2024-02-12T16:19:43.729 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-12T16:19:43.730 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-12T16:19:43.730 INFO:teuthology.orchestra.run.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:19:43.730 INFO:teuthology.orchestra.run.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:19:43.730 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-12T16:19:43.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:19:43.730 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:19:43.730 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:19:43.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:43.775 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:43.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:43.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:43 smithi123 bash[19119]: cluster 2024-02-12T16:19:42.128956+0000 osd.10 (osd.10) 66 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:19:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:43 smithi123 bash[19119]: cluster 2024-02-12T16:19:42.158235+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:19:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:43 smithi123 bash[19119]: cluster 2024-02-12T16:19:42.299126+0000 osd.11 (osd.11) 87 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:19:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:43 smithi123 bash[19119]: cluster 2024-02-12T16:19:42.380202+0000 osd.11 (osd.11) 88 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:19:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:43 smithi123 bash[19119]: cluster 2024-02-12T16:19:42.457559+0000 osd.6 (osd.6) 70 : cluster [DBG] 4.a scrub starts 2024-02-12T16:19:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:43 smithi123 bash[19119]: cluster 2024-02-12T16:19:42.861183+0000 osd.6 (osd.6) 71 : cluster [DBG] 4.a scrub ok 2024-02-12T16:19:43.890 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:43 smithi121 bash[18489]: cluster 2024-02-12T16:19:42.128956+0000 osd.10 (osd.10) 66 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:19:43.890 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:43 smithi121 bash[18489]: cluster 2024-02-12T16:19:42.158235+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:19:43.890 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:43 smithi121 bash[18489]: cluster 2024-02-12T16:19:42.299126+0000 osd.11 (osd.11) 87 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:19:43.890 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:43 smithi121 bash[18489]: cluster 2024-02-12T16:19:42.380202+0000 osd.11 (osd.11) 88 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:19:43.890 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:43 smithi121 bash[18489]: cluster 2024-02-12T16:19:42.457559+0000 osd.6 (osd.6) 70 : cluster [DBG] 4.a scrub starts 2024-02-12T16:19:43.890 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:43 smithi121 bash[18489]: cluster 2024-02-12T16:19:42.861183+0000 osd.6 (osd.6) 71 : cluster [DBG] 4.a scrub ok 2024-02-12T16:19:43.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:43.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:44.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:43 smithi169 bash[19604]: cluster 2024-02-12T16:19:42.128956+0000 osd.10 (osd.10) 66 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:19:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:43 smithi169 bash[19604]: cluster 2024-02-12T16:19:42.158235+0000 osd.10 (osd.10) 67 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:19:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:43 smithi169 bash[19604]: cluster 2024-02-12T16:19:42.299126+0000 osd.11 (osd.11) 87 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:19:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:43 smithi169 bash[19604]: cluster 2024-02-12T16:19:42.380202+0000 osd.11 (osd.11) 88 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:19:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:43 smithi169 bash[19604]: cluster 2024-02-12T16:19:42.457559+0000 osd.6 (osd.6) 70 : cluster [DBG] 4.a scrub starts 2024-02-12T16:19:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:43 smithi169 bash[19604]: cluster 2024-02-12T16:19:42.861183+0000 osd.6 (osd.6) 71 : cluster [DBG] 4.a scrub ok 2024-02-12T16:19:44.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:44.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:44.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:44.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:44.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:44.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:44.343 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:44.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:44.370 INFO:tasks.thrashosds.thrasher:j is {'epoch': 459, 'fsid': 'eb2349b2-c9bd-11ee-95b9-87774f69a715', 'created': '2024-02-12T15:47:18.273813+0000', 'modified': '2024-02-12T16:19:40.482005+0000', 'last_up_change': '2024-02-12T16:17:42.493645+0000', 'last_in_change': '2024-02-12T16:19:27.892497+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 11, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '410', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '410', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-12T16:13:44.637807+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '346', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 1815232143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 1815232143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 1815232143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 1815232143}]}, 'public_addr': '172.21.15.121:6801/1815232143', 'cluster_addr': '172.21.15.121:6802/1815232143', 'heartbeat_back_addr': '172.21.15.121:6804/1815232143', 'heartbeat_front_addr': '172.21.15.121:6803/1815232143', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c6140309-5703-4538-9739-276fbbdaa880', 'up': 1, 'in': 1, 'weight': 0.1039276123046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2968433670}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2968433670}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 2968433670}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2968433670}]}, 'public_addr': '172.21.15.121:6805/2968433670', 'cluster_addr': '172.21.15.121:6806/2968433670', 'heartbeat_back_addr': '172.21.15.121:6808/2968433670', 'heartbeat_front_addr': '172.21.15.121:6807/2968433670', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 2158187852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 2158187852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 2158187852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 2158187852}]}, 'public_addr': '172.21.15.121:6809/2158187852', 'cluster_addr': '172.21.15.121:6810/2158187852', 'heartbeat_back_addr': '172.21.15.121:6812/2158187852', 'heartbeat_front_addr': '172.21.15.121:6811/2158187852', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5f824526-596d-42e1-8b7e-cd8676ded6dd', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 410, 'down_at': 415, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 1415189366}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 1415189366}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6816', 'nonce': 1415189366}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 1415189366}]}, 'public_addr': '172.21.15.121:6813/1415189366', 'cluster_addr': '172.21.15.121:6814/1415189366', 'heartbeat_back_addr': '172.21.15.121:6816/1415189366', 'heartbeat_front_addr': '172.21.15.121:6815/1415189366', 'state': ['exists']}, {'osd': 4, 'uuid': '14c2f878-99ce-4b51-8d9c-131c1d883b31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6800', 'nonce': 3447700123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6801', 'nonce': 3447700123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6803', 'nonce': 3447700123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6802', 'nonce': 3447700123}]}, 'public_addr': '172.21.15.123:6800/3447700123', 'cluster_addr': '172.21.15.123:6801/3447700123', 'heartbeat_back_addr': '172.21.15.123:6803/3447700123', 'heartbeat_front_addr': '172.21.15.123:6802/3447700123', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6804', 'nonce': 961317306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6805', 'nonce': 961317306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6807', 'nonce': 961317306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6806', 'nonce': 961317306}]}, 'public_addr': '172.21.15.123:6804/961317306', 'cluster_addr': '172.21.15.123:6805/961317306', 'heartbeat_back_addr': '172.21.15.123:6807/961317306', 'heartbeat_front_addr': '172.21.15.123:6806/961317306', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '71e715e6-eac1-4386-927e-51df0cc419c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6808', 'nonce': 1377261784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6809', 'nonce': 1377261784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6811', 'nonce': 1377261784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6810', 'nonce': 1377261784}]}, 'public_addr': '172.21.15.123:6808/1377261784', 'cluster_addr': '172.21.15.123:6809/1377261784', 'heartbeat_back_addr': '172.21.15.123:6811/1377261784', 'heartbeat_front_addr': '172.21.15.123:6810/1377261784', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b419b04f-0c0c-495a-aa9b-25c334159515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6812', 'nonce': 1702613957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6813', 'nonce': 1702613957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6815', 'nonce': 1702613957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6814', 'nonce': 1702613957}]}, 'public_addr': '172.21.15.123:6812/1702613957', 'cluster_addr': '172.21.15.123:6813/1702613957', 'heartbeat_back_addr': '172.21.15.123:6815/1702613957', 'heartbeat_front_addr': '172.21.15.123:6814/1702613957', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1591e7bc-5b86-4401-803d-5325062ffb75', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 310, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6800', 'nonce': 2551075225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 2551075225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 2551075225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6802', 'nonce': 2551075225}]}, 'public_addr': '172.21.15.169:6800/2551075225', 'cluster_addr': '172.21.15.169:6801/2551075225', 'heartbeat_back_addr': '172.21.15.169:6803/2551075225', 'heartbeat_front_addr': '172.21.15.169:6802/2551075225', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '522d61df-8dc1-4915-83fd-f94d0d572c15', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6804', 'nonce': 2339555071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 2339555071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 2339555071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6806', 'nonce': 2339555071}]}, 'public_addr': '172.21.15.169:6804/2339555071', 'cluster_addr': '172.21.15.169:6805/2339555071', 'heartbeat_back_addr': '172.21.15.169:6807/2339555071', 'heartbeat_front_addr': '172.21.15.169:6806/2339555071', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '91f74dca-503a-45d1-9647-df4804d58515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 410, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6808', 'nonce': 391306610}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 391306610}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 391306610}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6810', 'nonce': 391306610}]}, 'public_addr': '172.21.15.169:6808/391306610', 'cluster_addr': '172.21.15.169:6809/391306610', 'heartbeat_back_addr': '172.21.15.169:6811/391306610', 'heartbeat_front_addr': '172.21.15.169:6810/391306610', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '788b4f47-eadf-4601-a5b4-51fb6f04fbdf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 93, 'up_thru': 453, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6812', 'nonce': 3696812305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3696812305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3696812305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6814', 'nonce': 3696812305}]}, 'public_addr': '172.21.15.169:6812/3696812305', 'cluster_addr': '172.21.15.169:6813/3696812305', 'heartbeat_back_addr': '172.21.15.169:6815/3696812305', 'heartbeat_front_addr': '172.21.15.169:6814/3696812305', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:59.801623+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:32.232292+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:43.883278+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:16:56.622408+0000', 'dead_epoch': 414}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:05.612718+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:28.142045+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:38.010636+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:26.267714+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:19.529182+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:18:52.433694+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:05.875431+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:29.180634+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '4.4', 'osds': [2, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.121:0/2474849789': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/253689008': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:6800/3531151725': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:0/1778611055': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:6800/28718860': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/445290372': '2024-02-13T15:47:47.232761+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:19:44.371 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-02-12T16:19:44.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:44.480 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:44.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:44.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:44.621 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:44.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:44.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:44.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:44.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:44.847 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:44.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:44.882 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:44 smithi169 bash[19604]: cluster 2024-02-12T16:19:43.424745+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.e scrub starts 2024-02-12T16:19:44.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:44 smithi169 bash[19604]: audit 2024-02-12T16:19:43.650981+0000 mon.a (mon.0) 1662 : audit [DBG] from='client.? v1:172.21.15.121:0/1860502338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:19:44.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:44 smithi169 bash[19604]: cluster 2024-02-12T16:19:43.802790+0000 mgr.y (mgr.14146) 1034 : cluster [DBG] pgmap v1392: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:44.883 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:44 smithi169 bash[19604]: cluster 2024-02-12T16:19:43.863286+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.e scrub ok 2024-02-12T16:19:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:44 smithi123 bash[19119]: cluster 2024-02-12T16:19:43.424745+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.e scrub starts 2024-02-12T16:19:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:44 smithi123 bash[19119]: audit 2024-02-12T16:19:43.650981+0000 mon.a (mon.0) 1662 : audit [DBG] from='client.? v1:172.21.15.121:0/1860502338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:19:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:44 smithi123 bash[19119]: cluster 2024-02-12T16:19:43.802790+0000 mgr.y (mgr.14146) 1034 : cluster [DBG] pgmap v1392: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:44 smithi123 bash[19119]: cluster 2024-02-12T16:19:43.863286+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.e scrub ok 2024-02-12T16:19:44.908 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:44.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:44 smithi121 bash[18489]: cluster 2024-02-12T16:19:43.424745+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.e scrub starts 2024-02-12T16:19:44.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:44 smithi121 bash[18489]: audit 2024-02-12T16:19:43.650981+0000 mon.a (mon.0) 1662 : audit [DBG] from='client.? v1:172.21.15.121:0/1860502338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:19:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:44 smithi121 bash[18489]: cluster 2024-02-12T16:19:43.802790+0000 mgr.y (mgr.14146) 1034 : cluster [DBG] pgmap v1392: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:44 smithi121 bash[18489]: cluster 2024-02-12T16:19:43.863286+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.e scrub ok 2024-02-12T16:19:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:44.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:45.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:45.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:45.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:45.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:45.187 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:45.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:45.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:45.326 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:45.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:45.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:45.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:45.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:45.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:45.598 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:45.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:45.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:45.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:45.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:45.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:45.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:45 smithi169 bash[19604]: cluster 2024-02-12T16:19:44.024423+0000 osd.9 (osd.9) 65 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:19:45.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:45 smithi169 bash[19604]: cluster 2024-02-12T16:19:44.446967+0000 osd.6 (osd.6) 72 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:19:45.855 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:45 smithi169 bash[19604]: cluster 2024-02-12T16:19:44.463584+0000 osd.9 (osd.9) 66 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:19:45.855 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:45 smithi169 bash[19604]: cluster 2024-02-12T16:19:44.825101+0000 osd.6 (osd.6) 73 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:19:45.862 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:45.881 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:45.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:45 smithi123 bash[19119]: cluster 2024-02-12T16:19:44.024423+0000 osd.9 (osd.9) 65 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:19:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:45 smithi123 bash[19119]: cluster 2024-02-12T16:19:44.446967+0000 osd.6 (osd.6) 72 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:19:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:45 smithi123 bash[19119]: cluster 2024-02-12T16:19:44.463584+0000 osd.9 (osd.9) 66 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:19:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:45 smithi123 bash[19119]: cluster 2024-02-12T16:19:44.825101+0000 osd.6 (osd.6) 73 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:19:45.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:45 smithi121 bash[18489]: cluster 2024-02-12T16:19:44.024423+0000 osd.9 (osd.9) 65 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:19:45.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:45 smithi121 bash[18489]: cluster 2024-02-12T16:19:44.446967+0000 osd.6 (osd.6) 72 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:19:45.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:45 smithi121 bash[18489]: cluster 2024-02-12T16:19:44.463584+0000 osd.9 (osd.9) 66 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:19:45.904 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:45 smithi121 bash[18489]: cluster 2024-02-12T16:19:44.825101+0000 osd.6 (osd.6) 73 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:19:45.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:46.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:46.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:46.126 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:46.164 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:46.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:46.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:46.306 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:46.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:46.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:46.449 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:46.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:46.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:46.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:46.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:46 smithi121 bash[18489]: cluster 2024-02-12T16:19:45.156965+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:19:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:46 smithi121 bash[18489]: cluster 2024-02-12T16:19:45.168801+0000 osd.10 (osd.10) 69 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:19:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:46 smithi121 bash[18489]: cluster 2024-02-12T16:19:45.249251+0000 osd.2 (osd.2) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:46 smithi121 bash[18489]: cluster 2024-02-12T16:19:45.249359+0000 osd.2 (osd.2) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:46 smithi121 bash[18489]: cluster 2024-02-12T16:19:45.577260+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e460: 12 total, 11 up, 12 in 2024-02-12T16:19:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:46 smithi121 bash[18489]: cluster 2024-02-12T16:19:45.803758+0000 mgr.y (mgr.14146) 1035 : cluster [DBG] pgmap v1394: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:46.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:46.739 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:46.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:46.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:46.856 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:46 smithi123 bash[19119]: cluster 2024-02-12T16:19:45.156965+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:19:46.856 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:46 smithi123 bash[19119]: cluster 2024-02-12T16:19:45.168801+0000 osd.10 (osd.10) 69 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:19:46.856 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:46 smithi123 bash[19119]: cluster 2024-02-12T16:19:45.249251+0000 osd.2 (osd.2) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:46.856 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:46 smithi123 bash[19119]: cluster 2024-02-12T16:19:45.249359+0000 osd.2 (osd.2) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:46.856 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:46 smithi123 bash[19119]: cluster 2024-02-12T16:19:45.577260+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e460: 12 total, 11 up, 12 in 2024-02-12T16:19:46.856 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:46 smithi123 bash[19119]: cluster 2024-02-12T16:19:45.803758+0000 mgr.y (mgr.14146) 1035 : cluster [DBG] pgmap v1394: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:46.880 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:46.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:46.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:46.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:46 smithi169 bash[19604]: cluster 2024-02-12T16:19:45.156965+0000 osd.10 (osd.10) 68 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:19:46.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:46 smithi169 bash[19604]: cluster 2024-02-12T16:19:45.168801+0000 osd.10 (osd.10) 69 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:19:46.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:46 smithi169 bash[19604]: cluster 2024-02-12T16:19:45.249251+0000 osd.2 (osd.2) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:19:46.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:46 smithi169 bash[19604]: cluster 2024-02-12T16:19:45.249359+0000 osd.2 (osd.2) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:19:46.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:46 smithi169 bash[19604]: cluster 2024-02-12T16:19:45.577260+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e460: 12 total, 11 up, 12 in 2024-02-12T16:19:46.997 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:46 smithi169 bash[19604]: cluster 2024-02-12T16:19:45.803758+0000 mgr.y (mgr.14146) 1035 : cluster [DBG] pgmap v1394: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:47.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:47.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:47.122 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:47.157 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:47.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:47.195 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:19:47.195 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:19:47.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:47.302 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:47.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:47.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:47.450 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:47.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:47.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:47.592 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:47.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:47.670 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:47 smithi121 bash[18489]: cluster 2024-02-12T16:19:46.137719+0000 osd.10 (osd.10) 70 : cluster [DBG] 1.d scrub starts 2024-02-12T16:19:47.670 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:47 smithi121 bash[18489]: cluster 2024-02-12T16:19:46.160466+0000 osd.10 (osd.10) 71 : cluster [DBG] 1.d scrub ok 2024-02-12T16:19:47.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:47.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:47.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:47.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:47.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:47 smithi169 bash[19604]: cluster 2024-02-12T16:19:46.137719+0000 osd.10 (osd.10) 70 : cluster [DBG] 1.d scrub starts 2024-02-12T16:19:47.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:47 smithi169 bash[19604]: cluster 2024-02-12T16:19:46.160466+0000 osd.10 (osd.10) 71 : cluster [DBG] 1.d scrub ok 2024-02-12T16:19:47.876 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:47.877 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:47 smithi123 bash[19119]: cluster 2024-02-12T16:19:46.137719+0000 osd.10 (osd.10) 70 : cluster [DBG] 1.d scrub starts 2024-02-12T16:19:47.877 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:47 smithi123 bash[19119]: cluster 2024-02-12T16:19:46.160466+0000 osd.10 (osd.10) 71 : cluster [DBG] 1.d scrub ok 2024-02-12T16:19:47.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:19:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:47.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:47.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:48.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:48.122 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:48.154 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:48.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:48.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:48.289 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:48.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:48.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:48.430 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:48.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:48.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:48.572 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:48.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:48.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:48.713 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:48.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:48.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:48 smithi169 bash[19604]: cluster 2024-02-12T16:19:47.804742+0000 mgr.y (mgr.14146) 1036 : cluster [DBG] pgmap v1395: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:48.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:48.831 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:48 smithi123 bash[19119]: cluster 2024-02-12T16:19:47.804742+0000 mgr.y (mgr.14146) 1036 : cluster [DBG] pgmap v1395: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:48.858 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:48.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:48 smithi121 bash[18489]: cluster 2024-02-12T16:19:47.804742+0000 mgr.y (mgr.14146) 1036 : cluster [DBG] pgmap v1395: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:48.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:49.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:49.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:49.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:49.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:49.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:49.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:49.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:19:49.283 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:49.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:49.300 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:19:49.326 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:19:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:19:49.372 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:19:49.372 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:19:49.372 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-12T16:19:49.372 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:19:49.373 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:19:49.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-12T16:19:49.375 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:19:49.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:49.404 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:19:49.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:49.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:49.431 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:19:49.462 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:19:49.490 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:19:49.515 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:19:49.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:49.541 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:19:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:19:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:19:49.567 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:49.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:49.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:49 smithi123 bash[19119]: cluster 2024-02-12T16:19:49.152136+0000 osd.7 (osd.7) 95 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:19:49.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:49 smithi123 bash[19119]: cluster 2024-02-12T16:19:49.201772+0000 osd.7 (osd.7) 96 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:19:49.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:49 smithi123 bash[19119]: audit 2024-02-12T16:19:49.252581+0000 mon.a (mon.0) 1664 : audit [DBG] from='client.? v1:172.21.15.121:0/1953114065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:49.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:49.713 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:49.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:49.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:49 smithi169 bash[19604]: cluster 2024-02-12T16:19:49.152136+0000 osd.7 (osd.7) 95 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:19:49.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:49 smithi169 bash[19604]: cluster 2024-02-12T16:19:49.201772+0000 osd.7 (osd.7) 96 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:19:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:49 smithi169 bash[19604]: audit 2024-02-12T16:19:49.252581+0000 mon.a (mon.0) 1664 : audit [DBG] from='client.? v1:172.21.15.121:0/1953114065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:49.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:49.857 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:49 smithi121 bash[18489]: cluster 2024-02-12T16:19:49.152136+0000 osd.7 (osd.7) 95 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:19:49.857 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:49 smithi121 bash[18489]: cluster 2024-02-12T16:19:49.201772+0000 osd.7 (osd.7) 96 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:19:49.857 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:49 smithi121 bash[18489]: audit 2024-02-12T16:19:49.252581+0000 mon.a (mon.0) 1664 : audit [DBG] from='client.? v1:172.21.15.121:0/1953114065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:19:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:49.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:49.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:49.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:50.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:50.141 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:50.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:50.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:19:50.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:19:50.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:50.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:50.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:50.553 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:50.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:50.694 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:50.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:50.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:50 smithi123 bash[19119]: cluster 2024-02-12T16:19:49.805826+0000 mgr.y (mgr.14146) 1037 : cluster [DBG] pgmap v1396: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:50.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:50 smithi123 bash[19119]: cluster 2024-02-12T16:19:50.492808+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e461: 12 total, 11 up, 12 in 2024-02-12T16:19:50.833 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:50 smithi121 bash[18489]: cluster 2024-02-12T16:19:49.805826+0000 mgr.y (mgr.14146) 1037 : cluster [DBG] pgmap v1396: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:50 smithi121 bash[18489]: cluster 2024-02-12T16:19:50.492808+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e461: 12 total, 11 up, 12 in 2024-02-12T16:19:50.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:50.949 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:50 smithi169 bash[19604]: cluster 2024-02-12T16:19:49.805826+0000 mgr.y (mgr.14146) 1037 : cluster [DBG] pgmap v1396: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:50.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:50 smithi169 bash[19604]: cluster 2024-02-12T16:19:50.492808+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e461: 12 total, 11 up, 12 in 2024-02-12T16:19:50.975 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:50.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:51.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:51.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:51.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:51.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:51.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:51.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:51.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:51.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:51.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:51.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:51.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:51.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:51.637 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:19:51.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:51.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:51 smithi121 bash[18489]: audit 2024-02-12T16:19:50.819911+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.121:0/2378262799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:51.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:51.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:51.791 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:51 smithi123 bash[19119]: audit 2024-02-12T16:19:50.819911+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.121:0/2378262799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:51.816 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:51.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:51.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:51.958 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:51.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:51 smithi169 bash[19604]: audit 2024-02-12T16:19:50.819911+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.121:0/2378262799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:19:52.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:52.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:52.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:52.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:52.241 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:52.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:52.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:52.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:19:52.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:52.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:52.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:52.517 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:52.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:52.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:52.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:52.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:52.723 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:19:52.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:52.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: cluster 2024-02-12T16:19:51.613913+0000 mon.a (mon.0) 1667 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:19:52.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:51.628263+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.121:0/2378262799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:19:52.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: cluster 2024-02-12T16:19:51.628389+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e462: 12 total, 11 up, 12 in 2024-02-12T16:19:52.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:52.312908+0000 mon.a (mon.0) 1670 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:19:52.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:52.313412+0000 mon.a (mon.0) 1671 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:19:52.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:52.313907+0000 mon.a (mon.0) 1672 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:19:52.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:52.336265+0000 mon.a (mon.0) 1673 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:19:52.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:52.339291+0000 mon.a (mon.0) 1674 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:19:52.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:52.340625+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:19:52.773 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:52 smithi121 bash[18489]: audit 2024-02-12T16:19:52.348851+0000 mon.a (mon.0) 1676 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:52.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:52.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: cluster 2024-02-12T16:19:51.613913+0000 mon.a (mon.0) 1667 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:51.628263+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.121:0/2378262799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: cluster 2024-02-12T16:19:51.628389+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e462: 12 total, 11 up, 12 in 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:52.312908+0000 mon.a (mon.0) 1670 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:52.313412+0000 mon.a (mon.0) 1671 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:52.313907+0000 mon.a (mon.0) 1672 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:52.336265+0000 mon.a (mon.0) 1673 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:52.339291+0000 mon.a (mon.0) 1674 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:52.340625+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:19:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:52 smithi123 bash[19119]: audit 2024-02-12T16:19:52.348851+0000 mon.a (mon.0) 1676 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:52.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:52.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: cluster 2024-02-12T16:19:51.613913+0000 mon.a (mon.0) 1667 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:51.628263+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.121:0/2378262799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: cluster 2024-02-12T16:19:51.628389+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e462: 12 total, 11 up, 12 in 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:52.312908+0000 mon.a (mon.0) 1670 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:52.313412+0000 mon.a (mon.0) 1671 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:52.313907+0000 mon.a (mon.0) 1672 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:52.336265+0000 mon.a (mon.0) 1673 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:52.339291+0000 mon.a (mon.0) 1674 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:52.340625+0000 mon.a (mon.0) 1675 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:19:52.912 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:52 smithi169 bash[19604]: audit 2024-02-12T16:19:52.348851+0000 mon.a (mon.0) 1676 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:19:52.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:52.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:53.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:53.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:53.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:53.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:53.217 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:53.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:53.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:53.362 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:53.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:53.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:53.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:53.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:53.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:53.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:53.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:53.737 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:19:53.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:53.776 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:53.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:53.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:53.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:53.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:54.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:54.060 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:54.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:54.164 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:54.201 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:54.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:54.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:54.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:54.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:19:54.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:54.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:54.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:54.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:54.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:54.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:54.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:54.769 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:54.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:54.873 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:54.909 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:54.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:55.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:55.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:54 smithi123 bash[19119]: cluster 2024-02-12T16:19:53.807880+0000 mgr.y (mgr.14146) 1039 : cluster [DBG] pgmap v1400: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:55.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:55.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:55.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:54 smithi121 bash[18489]: cluster 2024-02-12T16:19:53.807880+0000 mgr.y (mgr.14146) 1039 : cluster [DBG] pgmap v1400: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:55.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:55.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:55.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:55.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:54 smithi169 bash[19604]: cluster 2024-02-12T16:19:53.807880+0000 mgr.y (mgr.14146) 1039 : cluster [DBG] pgmap v1400: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:55.329 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:55.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:55.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:55.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:55.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:55.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:55.602 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:55.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:55.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:55.745 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:55.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:55.848 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:55.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:55.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:55.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:56.029 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:56.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:56.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:56.172 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:56.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:56.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:56.313 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:56.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:56.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:56.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:56.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:56.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:56 smithi169 bash[19604]: cluster 2024-02-12T16:19:55.496908+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e463: 12 total, 11 up, 12 in 2024-02-12T16:19:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:56 smithi169 bash[19604]: cluster 2024-02-12T16:19:55.808939+0000 mgr.y (mgr.14146) 1040 : cluster [DBG] pgmap v1402: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:56.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:56.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:56.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:56.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:56.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:56 smithi121 bash[18489]: cluster 2024-02-12T16:19:55.496908+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e463: 12 total, 11 up, 12 in 2024-02-12T16:19:56.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:56 smithi121 bash[18489]: cluster 2024-02-12T16:19:55.808939+0000 mgr.y (mgr.14146) 1040 : cluster [DBG] pgmap v1402: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:56.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:56.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:56.761 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:56 smithi123 bash[19119]: cluster 2024-02-12T16:19:55.496908+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e463: 12 total, 11 up, 12 in 2024-02-12T16:19:56.761 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:56 smithi123 bash[19119]: cluster 2024-02-12T16:19:55.808939+0000 mgr.y (mgr.14146) 1040 : cluster [DBG] pgmap v1402: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:56.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:56.880 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:56.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:56.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:57.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:57.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:57.152 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:57.162 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:19:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:57.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:19:57.180 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:19:57.180 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:19:57.180 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:19:57.180 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:19:57.265 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:19:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:57.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:19:57.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:57.448 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:57.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:57.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:57.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:57.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:57.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:57.732 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:57.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:57.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:57.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:19:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:57.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:19:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:57 smithi123 bash[19119]: cluster 2024-02-12T16:19:57.570043+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.b scrub starts 2024-02-12T16:19:57.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:57.985 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:57 smithi169 bash[19604]: cluster 2024-02-12T16:19:57.570043+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.b scrub starts 2024-02-12T16:19:58.008 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:58.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:19:58.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:58.148 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:19:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:19:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:57 smithi121 bash[18489]: cluster 2024-02-12T16:19:57.570043+0000 osd.0 (osd.0) 69 : cluster [DBG] 4.b scrub starts 2024-02-12T16:19:58.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:58.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:58.366 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:19:58.366 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "" 2024-02-12T16:19:58.366 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:19:58.391 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:58.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:58.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:58.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:58.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:58.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:58.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:19:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:19:58.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:58.972 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:58 smithi169 bash[19604]: cluster 2024-02-12T16:19:57.809671+0000 mgr.y (mgr.14146) 1041 : cluster [DBG] pgmap v1403: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:58.972 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:19:58 smithi169 bash[19604]: cluster 2024-02-12T16:19:57.919836+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.b scrub ok 2024-02-12T16:19:58.991 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:19:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:58.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:19:59.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:58 smithi123 bash[19119]: cluster 2024-02-12T16:19:57.809671+0000 mgr.y (mgr.14146) 1041 : cluster [DBG] pgmap v1403: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:59.023 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:19:58 smithi123 bash[19119]: cluster 2024-02-12T16:19:57.919836+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.b scrub ok 2024-02-12T16:19:59.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:59.132 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:59.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:58 smithi121 bash[18489]: cluster 2024-02-12T16:19:57.809671+0000 mgr.y (mgr.14146) 1041 : cluster [DBG] pgmap v1403: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:19:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:19:58 smithi121 bash[18489]: cluster 2024-02-12T16:19:57.919836+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.b scrub ok 2024-02-12T16:19:59.179 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:19:59.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:59.275 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:19:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:59.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:19:59.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:59.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:19:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:59.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:19:59.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:59.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:19:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:59.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:19:59.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:59.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:59.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:19:59.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:59.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:19:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:59.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:19:59.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:59.981 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:19:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:19:59.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:00.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:00.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:00.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:00.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:00.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:00.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:00.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:00.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:00.533 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:00.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:00.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:00.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:00.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:00.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:00.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:00.953 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:00.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:00.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:19:59.810726+0000 mgr.y (mgr.14146) 1042 : cluster [DBG] pgmap v1404: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:00.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000110+0000 mon.a (mon.0) 1678 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 7 pgs inactive; Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000165+0000 mon.a (mon.0) 1679 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000191+0000 mon.a (mon.0) 1680 : cluster [WRN] osd.3 (root=default,host=smithi121) is down 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000215+0000 mon.a (mon.0) 1681 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 7 pgs inactive 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000238+0000 mon.a (mon.0) 1682 : cluster [WRN] pg 1.3 is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000259+0000 mon.a (mon.0) 1683 : cluster [WRN] pg 1.b is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000285+0000 mon.a (mon.0) 1684 : cluster [WRN] pg 1.13 is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000305+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 4.0 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000326+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 4.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000354+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 4.c is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000376+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 4.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000404+0000 mon.a (mon.0) 1689 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized 2024-02-12T16:20:00.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000426+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 1.3 is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:00.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000451+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 1.b is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:00.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000473+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 1.13 is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:00.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000494+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 4.0 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [7] 2024-02-12T16:20:00.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000521+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 4.9 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:00.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000548+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 4.c is stuck undersized for 2m, current state undersized+degraded+peered, last acting [4] 2024-02-12T16:20:00.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:00.000577+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:19:59.810726+0000 mgr.y (mgr.14146) 1042 : cluster [DBG] pgmap v1404: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000110+0000 mon.a (mon.0) 1678 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 7 pgs inactive; Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000165+0000 mon.a (mon.0) 1679 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000191+0000 mon.a (mon.0) 1680 : cluster [WRN] osd.3 (root=default,host=smithi121) is down 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000215+0000 mon.a (mon.0) 1681 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 7 pgs inactive 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000238+0000 mon.a (mon.0) 1682 : cluster [WRN] pg 1.3 is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000259+0000 mon.a (mon.0) 1683 : cluster [WRN] pg 1.b is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000285+0000 mon.a (mon.0) 1684 : cluster [WRN] pg 1.13 is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000305+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 4.0 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000326+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 4.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000354+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 4.c is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000376+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 4.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000404+0000 mon.a (mon.0) 1689 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000426+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 1.3 is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000451+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 1.b is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000473+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 1.13 is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000494+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 4.0 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [7] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000521+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 4.9 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000548+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 4.c is stuck undersized for 2m, current state undersized+degraded+peered, last acting [4] 2024-02-12T16:20:01.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:00.000577+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:01.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:01.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:01.099 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:20:01.120 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:19:59.810726+0000 mgr.y (mgr.14146) 1042 : cluster [DBG] pgmap v1404: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000110+0000 mon.a (mon.0) 1678 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 7 pgs inactive; Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000165+0000 mon.a (mon.0) 1679 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000191+0000 mon.a (mon.0) 1680 : cluster [WRN] osd.3 (root=default,host=smithi121) is down 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000215+0000 mon.a (mon.0) 1681 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 7 pgs inactive 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000238+0000 mon.a (mon.0) 1682 : cluster [WRN] pg 1.3 is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000259+0000 mon.a (mon.0) 1683 : cluster [WRN] pg 1.b is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000285+0000 mon.a (mon.0) 1684 : cluster [WRN] pg 1.13 is stuck inactive for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000305+0000 mon.a (mon.0) 1685 : cluster [WRN] pg 4.0 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000326+0000 mon.a (mon.0) 1686 : cluster [WRN] pg 4.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000354+0000 mon.a (mon.0) 1687 : cluster [WRN] pg 4.c is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2024-02-12T16:20:01.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000376+0000 mon.a (mon.0) 1688 : cluster [WRN] pg 4.f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000404+0000 mon.a (mon.0) 1689 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 8797/70012 objects degraded (12.565%), 4 pgs degraded, 7 pgs undersized 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000426+0000 mon.a (mon.0) 1690 : cluster [WRN] pg 1.3 is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000451+0000 mon.a (mon.0) 1691 : cluster [WRN] pg 1.b is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000473+0000 mon.a (mon.0) 1692 : cluster [WRN] pg 1.13 is stuck undersized for 2m, current state undersized+peered, last acting [6] 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000494+0000 mon.a (mon.0) 1693 : cluster [WRN] pg 4.0 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [7] 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000521+0000 mon.a (mon.0) 1694 : cluster [WRN] pg 4.9 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000548+0000 mon.a (mon.0) 1695 : cluster [WRN] pg 4.c is stuck undersized for 2m, current state undersized+degraded+peered, last acting [4] 2024-02-12T16:20:01.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:00.000577+0000 mon.a (mon.0) 1696 : cluster [WRN] pg 4.f is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2024-02-12T16:20:01.143 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:20:01.168 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:20:01.192 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:20:01.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:01.217 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:20:01.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:01.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:01.240 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:20:01.271 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:20:01.302 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:20:01.327 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:20:01.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:01.352 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:20:01.352 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.352 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.352 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.353 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:20:01.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:01.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:01.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:01.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:01.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:01.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:01.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:01.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:01.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:01.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:01.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:01.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:01 smithi123 bash[19119]: audit 2024-02-12T16:20:01.068135+0000 mon.a (mon.0) 1697 : audit [DBG] from='client.? v1:172.21.15.121:0/210140327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:01.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:01.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:02.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:02.064 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:02.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:02.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:01 smithi121 bash[18489]: audit 2024-02-12T16:20:01.068135+0000 mon.a (mon.0) 1697 : audit [DBG] from='client.? v1:172.21.15.121:0/210140327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:02.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:02.181 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:01 smithi169 bash[19604]: audit 2024-02-12T16:20:01.068135+0000 mon.a (mon.0) 1697 : audit [DBG] from='client.? v1:172.21.15.121:0/210140327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:02.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:02.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:02.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:02.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:02.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:02.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:02.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:02.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:02.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:02.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:02.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:20:02.641 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:20:02.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:02.771 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:02.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:02.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:02.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:02 smithi121 bash[18489]: cluster 2024-02-12T16:20:01.811796+0000 mgr.y (mgr.14146) 1043 : cluster [DBG] pgmap v1405: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:02.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:02.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:03.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:03.046 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:03.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:02 smithi169 bash[19604]: cluster 2024-02-12T16:20:01.811796+0000 mgr.y (mgr.14146) 1043 : cluster [DBG] pgmap v1405: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:03.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:03.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:02 smithi123 bash[19119]: cluster 2024-02-12T16:20:01.811796+0000 mgr.y (mgr.14146) 1043 : cluster [DBG] pgmap v1405: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:03.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:03.186 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:03.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:03.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:03.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:03.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:03.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:03.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:03.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:03.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:03.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:03.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:03.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:03.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:03.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:03.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:03.907 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:20:03.910 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:03.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:04.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:04.024 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:20:04.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:20:04.024 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-12T16:20:04.024 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.907595 2024-02-12T16:20:04.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 1 0.9075945314633669 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: audit 2024-02-12T16:20:03.457728+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? v1:172.21.15.121:0/1903475749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.625199+0000 mon.a (mon.0) 1699 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1898568996,config=2903391811,health=2716334176,kv=806410826}) 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.626667+0000 mon.a (mon.0) 1700 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=1770631240,logm=2986005325}) 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.628141+0000 mon.a (mon.0) 1701 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586190312}) 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.629563+0000 mon.a (mon.0) 1702 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962682514}) 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.631258+0000 mon.a (mon.0) 1703 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3377327364}) 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.632668+0000 mon.a (mon.0) 1704 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577732575}) 2024-02-12T16:20:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.638732+0000 mon.a (mon.0) 1705 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=201363643}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.644049+0000 mon.a (mon.0) 1706 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572844}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.649641+0000 mon.a (mon.0) 1707 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1823400655}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.653849+0000 mon.a (mon.0) 1708 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3602140642}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.658058+0000 mon.a (mon.0) 1709 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767848225}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.660077+0000 mon.a (mon.0) 1710 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=16,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=44} crc {logm=2251661134,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2700914397,mon_config_key=1081218343}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.660841+0000 mon.a (mon.0) 1711 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=630933330}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.662698+0000 mon.a (mon.0) 1712 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3253129989}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.664376+0000 mon.a (mon.0) 1713 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=1647634727,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=564181195,osdmap=2741461713}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.667135+0000 mon.a (mon.0) 1714 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=670530120}) 2024-02-12T16:20:04.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:03 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.679892+0000 mon.a (mon.0) 1715 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=12} crc {osdmap=3969250624}) 2024-02-12T16:20:04.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:04.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:04.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: audit 2024-02-12T16:20:03.457728+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? v1:172.21.15.121:0/1903475749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.625199+0000 mon.a (mon.0) 1699 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1898568996,config=2903391811,health=2716334176,kv=806410826}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.626667+0000 mon.a (mon.0) 1700 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=1770631240,logm=2986005325}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.628141+0000 mon.a (mon.0) 1701 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586190312}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.629563+0000 mon.a (mon.0) 1702 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962682514}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.631258+0000 mon.a (mon.0) 1703 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3377327364}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.632668+0000 mon.a (mon.0) 1704 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577732575}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.638732+0000 mon.a (mon.0) 1705 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=201363643}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.644049+0000 mon.a (mon.0) 1706 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572844}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.649641+0000 mon.a (mon.0) 1707 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1823400655}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.653849+0000 mon.a (mon.0) 1708 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3602140642}) 2024-02-12T16:20:04.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.658058+0000 mon.a (mon.0) 1709 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767848225}) 2024-02-12T16:20:04.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.660077+0000 mon.a (mon.0) 1710 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=16,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=44} crc {logm=2251661134,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2700914397,mon_config_key=1081218343}) 2024-02-12T16:20:04.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.660841+0000 mon.a (mon.0) 1711 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=630933330}) 2024-02-12T16:20:04.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.662698+0000 mon.a (mon.0) 1712 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3253129989}) 2024-02-12T16:20:04.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.664376+0000 mon.a (mon.0) 1713 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=1647634727,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=564181195,osdmap=2741461713}) 2024-02-12T16:20:04.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.667135+0000 mon.a (mon.0) 1714 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=670530120}) 2024-02-12T16:20:04.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:03 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.679892+0000 mon.a (mon.0) 1715 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=12} crc {osdmap=3969250624}) 2024-02-12T16:20:04.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: audit 2024-02-12T16:20:03.457728+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? v1:172.21.15.121:0/1903475749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.625199+0000 mon.a (mon.0) 1699 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1898568996,config=2903391811,health=2716334176,kv=806410826}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.626667+0000 mon.a (mon.0) 1700 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=1770631240,logm=2986005325}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.628141+0000 mon.a (mon.0) 1701 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586190312}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.629563+0000 mon.a (mon.0) 1702 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962682514}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.631258+0000 mon.a (mon.0) 1703 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3377327364}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.632668+0000 mon.a (mon.0) 1704 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577732575}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.638732+0000 mon.a (mon.0) 1705 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=201363643}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.644049+0000 mon.a (mon.0) 1706 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989572844}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.649641+0000 mon.a (mon.0) 1707 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1823400655}) 2024-02-12T16:20:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.653849+0000 mon.a (mon.0) 1708 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3602140642}) 2024-02-12T16:20:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.658058+0000 mon.a (mon.0) 1709 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767848225}) 2024-02-12T16:20:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.660077+0000 mon.a (mon.0) 1710 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=16,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=44} crc {logm=2251661134,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2700914397,mon_config_key=1081218343}) 2024-02-12T16:20:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.660841+0000 mon.a (mon.0) 1711 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=630933330}) 2024-02-12T16:20:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.662698+0000 mon.a (mon.0) 1712 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3253129989}) 2024-02-12T16:20:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.664376+0000 mon.a (mon.0) 1713 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=1647634727,monmap=2720272532,osd_metadata=281555209,osd_pg_creating=564181195,osdmap=2741461713}) 2024-02-12T16:20:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.667135+0000 mon.a (mon.0) 1714 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=670530120}) 2024-02-12T16:20:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:03 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.679892+0000 mon.a (mon.0) 1715 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=12} crc {osdmap=3969250624}) 2024-02-12T16:20:04.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:04.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:04.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:04.328 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:04.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:04.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:04.468 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:04.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:04.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:04.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:04.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:04.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:20:04.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:04.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:04.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:04.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:04.874 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:04.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:04.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:05.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:05.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:05.132 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:04 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.259059+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.d scrub starts 2024-02-12T16:20:05.132 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:04 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.747124+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.d scrub ok 2024-02-12T16:20:05.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:04 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.812872+0000 mgr.y (mgr.14146) 1044 : cluster [DBG] pgmap v1406: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:05.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:04 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.880971+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:05.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:04 smithi121 bash[18489]: audit 2024-02-12T16:20:03.898399+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? v1:172.21.15.121:0/1903475749' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:20:05.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:04 smithi121 bash[18489]: cluster 2024-02-12T16:20:03.898446+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e464: 12 total, 11 up, 12 in 2024-02-12T16:20:05.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:05.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:05.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:04 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.259059+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.d scrub starts 2024-02-12T16:20:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:04 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.747124+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.d scrub ok 2024-02-12T16:20:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:04 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.812872+0000 mgr.y (mgr.14146) 1044 : cluster [DBG] pgmap v1406: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:04 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.880971+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:04 smithi169 bash[19604]: audit 2024-02-12T16:20:03.898399+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? v1:172.21.15.121:0/1903475749' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:20:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:04 smithi169 bash[19604]: cluster 2024-02-12T16:20:03.898446+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e464: 12 total, 11 up, 12 in 2024-02-12T16:20:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:04 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.259059+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.d scrub starts 2024-02-12T16:20:05.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:04 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.747124+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.d scrub ok 2024-02-12T16:20:05.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:04 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.812872+0000 mgr.y (mgr.14146) 1044 : cluster [DBG] pgmap v1406: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:05.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:04 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.880971+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:05.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:04 smithi123 bash[19119]: audit 2024-02-12T16:20:03.898399+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? v1:172.21.15.121:0/1903475749' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:20:05.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:04 smithi123 bash[19119]: cluster 2024-02-12T16:20:03.898446+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e464: 12 total, 11 up, 12 in 2024-02-12T16:20:05.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:05.433 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:05.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:05.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:05.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:05.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:05.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:05.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:05.986 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:05.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:06.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:06.096 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:20:06.096 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:20:06.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:06.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:06.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:06.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:06.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:06.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:06.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:06.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:06.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:06.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:06.542 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:06.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:06.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:20:06.635 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:06 smithi121 bash[18489]: cluster 2024-02-12T16:20:05.502424+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e465: 12 total, 11 up, 12 in 2024-02-12T16:20:06.635 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:06 smithi121 bash[18489]: cluster 2024-02-12T16:20:05.813838+0000 mgr.y (mgr.14146) 1045 : cluster [DBG] pgmap v1409: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:06.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:06.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:06.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:06.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:06 smithi123 bash[19119]: cluster 2024-02-12T16:20:05.502424+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e465: 12 total, 11 up, 12 in 2024-02-12T16:20:06.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:06 smithi123 bash[19119]: cluster 2024-02-12T16:20:05.813838+0000 mgr.y (mgr.14146) 1045 : cluster [DBG] pgmap v1409: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:06.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:20:06.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:06.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:06 smithi169 bash[19604]: cluster 2024-02-12T16:20:05.502424+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e465: 12 total, 11 up, 12 in 2024-02-12T16:20:06.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:06 smithi169 bash[19604]: cluster 2024-02-12T16:20:05.813838+0000 mgr.y (mgr.14146) 1045 : cluster [DBG] pgmap v1409: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:06.824 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:06.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:06.928 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:06.960 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:06.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:07.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:07.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:07.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:07.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:07.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:07.240 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:07.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:07.344 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:07.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:07.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:07.525 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:07.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:07.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:07.669 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:07.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:07.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:07.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:07.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:07.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:07.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:07.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:08.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:08.090 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:08.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:08.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:08.233 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:08.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:08.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:08.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:08.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:08.476 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:08.512 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:08.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:08.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:08.652 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:08.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:08.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:08.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:08.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:08.819 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:08.819 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:08.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:08.935 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:08.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:09.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:09.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:08 smithi169 bash[19604]: cluster 2024-02-12T16:20:07.814948+0000 mgr.y (mgr.14146) 1046 : cluster [DBG] pgmap v1410: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:09.077 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:09.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:08 smithi123 bash[19119]: cluster 2024-02-12T16:20:07.814948+0000 mgr.y (mgr.14146) 1046 : cluster [DBG] pgmap v1410: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:08 smithi121 bash[18489]: cluster 2024-02-12T16:20:07.814948+0000 mgr.y (mgr.14146) 1046 : cluster [DBG] pgmap v1410: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:09.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:09.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:09.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:09.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:09.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:09.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:09.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:09.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:09.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:09.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:09.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:09.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:09.744 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:09.783 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:09.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:09.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:09.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:09.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:10.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:10.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:10.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:10.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:10.205 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:10.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:10.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:10.347 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:10.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:10.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:10.485 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:10.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:10.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:10.628 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:10.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:10.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:10.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:10.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:10.845 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:10.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:10.906 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:10.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:11.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:11.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:11.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:11.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:10 smithi123 bash[19119]: cluster 2024-02-12T16:20:09.816047+0000 mgr.y (mgr.14146) 1047 : cluster [DBG] pgmap v1411: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:11.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:11.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:10 smithi121 bash[18489]: cluster 2024-02-12T16:20:09.816047+0000 mgr.y (mgr.14146) 1047 : cluster [DBG] pgmap v1411: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:11.164 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:10 smithi169 bash[19604]: cluster 2024-02-12T16:20:09.816047+0000 mgr.y (mgr.14146) 1047 : cluster [DBG] pgmap v1411: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:11.190 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:11.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:11.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:11.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:11.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:11.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:11.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:11.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:11.578 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:11.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:11.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:11.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:11.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:11.753 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:11.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:11.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:11.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:11.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:12.030 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:12.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:12.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:12.172 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:12.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:12.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:12.312 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:12.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:12.415 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:12.453 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:12.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:12.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:12.590 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:12.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:12.595 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:12 smithi123 bash[19119]: cluster 2024-02-12T16:20:10.770594+0000 osd.10 (osd.10) 72 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:12.595 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:12 smithi123 bash[19119]: cluster 2024-02-12T16:20:10.770662+0000 osd.10 (osd.10) 73 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:12.595 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:12 smithi123 bash[19119]: cluster 2024-02-12T16:20:11.370989+0000 osd.9 (osd.9) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:12.595 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:12 smithi123 bash[19119]: cluster 2024-02-12T16:20:11.371083+0000 osd.9 (osd.9) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:12.595 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:12 smithi123 bash[19119]: cluster 2024-02-12T16:20:11.505533+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e466: 12 total, 11 up, 12 in 2024-02-12T16:20:12.595 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:12 smithi123 bash[19119]: cluster 2024-02-12T16:20:11.818046+0000 mgr.y (mgr.14146) 1048 : cluster [DBG] pgmap v1413: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:12.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:12.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:12 smithi169 bash[19604]: cluster 2024-02-12T16:20:10.770594+0000 osd.10 (osd.10) 72 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:12 smithi169 bash[19604]: cluster 2024-02-12T16:20:10.770662+0000 osd.10 (osd.10) 73 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:12 smithi169 bash[19604]: cluster 2024-02-12T16:20:11.370989+0000 osd.9 (osd.9) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:12 smithi169 bash[19604]: cluster 2024-02-12T16:20:11.371083+0000 osd.9 (osd.9) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:12 smithi169 bash[19604]: cluster 2024-02-12T16:20:11.505533+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e466: 12 total, 11 up, 12 in 2024-02-12T16:20:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:12 smithi169 bash[19604]: cluster 2024-02-12T16:20:11.818046+0000 mgr.y (mgr.14146) 1048 : cluster [DBG] pgmap v1413: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:12.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:12.865 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:12.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:12 smithi121 bash[18489]: cluster 2024-02-12T16:20:10.770594+0000 osd.10 (osd.10) 72 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:12 smithi121 bash[18489]: cluster 2024-02-12T16:20:10.770662+0000 osd.10 (osd.10) 73 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:12 smithi121 bash[18489]: cluster 2024-02-12T16:20:11.370989+0000 osd.9 (osd.9) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:12 smithi121 bash[18489]: cluster 2024-02-12T16:20:11.371083+0000 osd.9 (osd.9) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:12 smithi121 bash[18489]: cluster 2024-02-12T16:20:11.505533+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e466: 12 total, 11 up, 12 in 2024-02-12T16:20:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:12 smithi121 bash[18489]: cluster 2024-02-12T16:20:11.818046+0000 mgr.y (mgr.14146) 1048 : cluster [DBG] pgmap v1413: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:12.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:13.005 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:13.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:13.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:13.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:13.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:13.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:13.417 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:13.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:13.555 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:13.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:13.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:13.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:13.841 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:13.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:13.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:13.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:14.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:14.111 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:14.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:14.165 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:20:14.188 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:20:14.209 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:20:14.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:14.240 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:20:14.249 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:14.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:14.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:14.262 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:20:14.285 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:20:14.314 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:20:14.333 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:20:14.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:14.356 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:20:14.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:14.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:14.393 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:20:14.431 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:20:14.431 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.431 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.431 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.432 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:20:14.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:14.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:14.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:14.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:14.603 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:20:14.603 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:219664 3.5 3:ac1b9cb3:::benchmark_data_smithi169_51390_object55880:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:33.997844+0000", 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 520.59973853999998, 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42621488499999999, 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 219664 2024-02-12T16:20:14.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997844+0000", 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997844+0000", 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997842+0000", 2024-02-12T16:20:14.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8320000000000006e-06 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997849+0000", 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1868999999999999e-05 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997861+0000", 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2669999999999997e-06 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997867+0000", 2024-02-12T16:20:14.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8968000000000003e-05 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997916+0000", 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6483000000000006e-05 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.997992+0000", 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8905000000000003e-05 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998081+0000", 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044546769999999998 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.002536+0000", 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8839e-05 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.002554+0000", 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4565000000000003e-05 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.002619+0000", 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42143968500000001 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424059+0000", 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42621488499999999 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:219665 3.c 3:3e688d3f:::benchmark_data_smithi169_51390_object55881:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:20:14.609 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:33.998798+0000", 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 520.59878433599999, 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43492321900000003, 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 219665 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998798+0000", 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998798+0000", 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998797+0000", 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0520000000000002e-06 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998805+0000", 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9300000000000003e-06 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998813+0000", 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3069999999999996e-06 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998817+0000", 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6118999999999997e-05 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998863+0000", 2024-02-12T16:20:14.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105734 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.998969+0000", 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011653400000000001 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:33.999085+0000", 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42522143299999998 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424307+0000", 2024-02-12T16:20:14.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092645780000000007 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433571+0000", 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4145000000000002e-05 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433646+0000", 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5753999999999993e-05 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433721+0000", 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43492321900000003 2024-02-12T16:20:14.614 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.15108.0:219676 3.d e304/256 3:bd218378:::benchmark_data_smithi169_51390_object55892:head v 304'13548, mlcod=304'13546)", 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:34.005655+0000", 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 520.59192753800005, 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43757516699999999, 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005655+0000", 2024-02-12T16:20:14.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005655+0000", 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005654+0000", 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8189999999999998e-06 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.005662+0000", 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.417952029 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423614+0000", 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8082999999999999e-05 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423642+0000", 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043031599999999998 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424073+0000", 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8934999999999997e-05 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424132+0000", 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019061813 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.443194+0000", 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6468999999999999e-05 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.443230+0000", 2024-02-12T16:20:14.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43757516699999999 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:219684 3.c 3:39f55561:::benchmark_data_smithi169_51390_object55900:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:34.013105+0000", 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 520.58447717199999, 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42544842399999999, 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.619 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.620 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.620 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 219684 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013105+0000", 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013105+0000", 2024-02-12T16:20:14.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:14.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013102+0000", 2024-02-12T16:20:14.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4159999999999993e-06 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.013112+0000", 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.41051737199999999 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423629+0000", 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0295999999999999e-05 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.423659+0000", 2024-02-12T16:20:14.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079160000000000005 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424451+0000", 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031754599999999998 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.424769+0000", 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037539399999999998 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.425144+0000", 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008653193 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.433797+0000", 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046760500000000002 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.438473+0000", 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5593999999999998e-05 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.438509+0000", 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4826000000000001e-05 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:34.438554+0000", 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42544842399999999 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:222371 3.5 3:a8591d55:::benchmark_data_smithi169_51390_object58587:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:36.689417+0000", 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 517.908165371, 2024-02-12T16:20:14.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32083903699999999, 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 222371 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689417+0000", 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689417+0000", 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689415+0000", 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8939999999999993e-06 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689423+0000", 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4545999999999999e-05 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689437+0000", 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1400000000000006e-06 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689446+0000", 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4110000000000002e-05 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689500+0000", 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001303 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689631+0000", 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014779999999999999 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.689778+0000", 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055622780000000004 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.695341+0000", 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31480284800000002 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010144+0000", 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4207000000000004e-05 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010208+0000", 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8272999999999997e-05 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010256+0000", 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32083903699999999 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.631 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:222373 3.c 3:37edf101:::benchmark_data_smithi169_51390_object58589:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:36.692259+0000", 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 517.90532342899996, 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.30243373299999998, 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 222373 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.632 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692259+0000", 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692259+0000", 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692257+0000", 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2659e-05 2024-02-12T16:20:14.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692269+0000", 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6239000000000002e-05 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692286+0000", 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7729999999999999e-06 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692293+0000", 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7355999999999999e-05 2024-02-12T16:20:14.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692341+0000", 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012307099999999999 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692464+0000", 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014246300000000001 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.692606+0000", 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046957240000000001 2024-02-12T16:20:14.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.697302+0000", 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.29709079199999999 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.994393+0000", 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016829900000000001 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.994561+0000", 2024-02-12T16:20:14.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013167300000000001 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.994693+0000", 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.30243373299999998 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:222385 3.5 3:ae306d2e:::benchmark_data_smithi169_51390_object58601:head [delete] snapc 0=[] ondisk+write+known_if_redirected e304)", 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:11:36.699414+0000", 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 517.898168177, 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31099299200000002, 2024-02-12T16:20:14.637 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 222385 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699414+0000", 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699414+0000", 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699413+0000", 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0499999999999992e-06 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699421+0000", 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0274e-05 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699431+0000", 2024-02-12T16:20:14.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7480000000000001e-06 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699436+0000", 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2242000000000001e-05 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699468+0000", 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2415000000000003e-05 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699550+0000", 2024-02-12T16:20:14.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5103000000000002e-05 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.699636+0000", 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042469489999999999 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:36.703883+0000", 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.30642461300000001 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010307+0000", 2024-02-12T16:20:14.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3876999999999999e-05 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010361+0000", 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6155000000000001e-05 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:11:37.010407+0000", 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31099299200000002 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287207 3.c 3:3621cc5d:::benchmark_data_smithi169_51390_object123423:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.493969+0000", 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 460.10361341700002, 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50046115199999996, 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.643 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287207 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493969+0000", 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493969+0000", 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493967+0000", 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3289999999999996e-06 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493975+0000", 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493986+0000", 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3589999999999996e-06 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.493992+0000", 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2116000000000001e-05 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.494035+0000", 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111876 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.494146+0000", 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015048199999999999 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.494297+0000", 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043225190000000004 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.498619+0000", 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094397479999999995 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508059+0000", 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1324e-05 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508091+0000", 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48633959500000001 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994430+0000", 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50046115199999996 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287212 3.5 3:a506526d:::benchmark_data_smithi169_51390_object123428:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:14.648 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.503073+0000", 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 460.09450953999999, 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49480824099999998, 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287212 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503073+0000", 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503073+0000", 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503071+0000", 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0090000000000001e-06 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503078+0000", 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8640000000000008e-06 2024-02-12T16:20:14.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503088+0000", 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7060000000000003e-06 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503092+0000", 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7908999999999999e-05 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503130+0000", 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100688 2024-02-12T16:20:14.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503231+0000", 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7761000000000003e-05 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503309+0000", 2024-02-12T16:20:14.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043448489999999996 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507654+0000", 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48952226900000001 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997176+0000", 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034127499999999999 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997517+0000", 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036395299999999998 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997881+0000", 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49480824099999998 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.656 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287214 3.c 3:33c50097:::benchmark_data_smithi169_51390_object123430:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.503324+0000", 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 460.09425813299998, 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50303783700000004, 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287214 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.657 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503324+0000", 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503324+0000", 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503323+0000", 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9109999999999998e-06 2024-02-12T16:20:14.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503329+0000", 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2590000000000006e-06 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503338+0000", 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7750000000000004e-06 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503345+0000", 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5648000000000001e-05 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503390+0000", 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9902999999999998e-05 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503460+0000", 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4788000000000003e-05 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.503535+0000", 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.490980374 2024-02-12T16:20:14.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994515+0000", 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011741451999999999 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006257+0000", 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0683999999999997e-05 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006318+0000", 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4465000000000002e-05 2024-02-12T16:20:14.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006362+0000", 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.50303783700000004 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.15108.0:287207 3.c e318/256 ondisk, result = 0)", 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 460.090936167, 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48782185099999997, 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.662 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506646+0000", 2024-02-12T16:20:14.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0639999999999998e-06 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506649+0000", 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5179999999999998e-06 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506653+0000", 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506658+0000", 2024-02-12T16:20:14.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388727 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508047+0000", 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0949999999999998e-06 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508054+0000", 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48641379800000001 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994468+0000", 2024-02-12T16:20:14.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48782185099999997 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287220 3.c 3:35aadc3b:::benchmark_data_smithi169_51390_object123436:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.508033+0000", 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 460.08954939400002, 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49976067499999999, 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:14.666 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287220 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508033+0000", 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508033+0000", 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508031+0000", 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6420000000000001e-06 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508038+0000", 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.089e-06 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508046+0000", 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7230000000000004e-06 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.508050+0000", 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48656027800000001 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994611+0000", 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00045055900000000001 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.995061+0000", 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054616300000000001 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:20:14.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.995607+0000", 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010806431 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006414+0000", 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8079999999999999e-06 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006421+0000", 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7545e-05 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.006468+0000", 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013254810000000001 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007794+0000", 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49976067499999999 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.15108.0:287212 3.5 e318/256 ondisk, result = 0)", 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.511973+0000", 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 460.08560910900002, 2024-02-12T16:20:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.486612779, 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511973+0000", 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511973+0000", 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511971+0000", 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0340000000000002e-06 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.511980+0000", 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48277212000000003 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994752+0000", 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9496e-05 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.994822+0000", 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022711879999999999 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997093+0000", 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4272e-05 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.997127+0000", 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001458593 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.998586+0000", 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.486612779 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:164500 4.1 e367/334 4:81c5aec5:::benchmark_data_smithi169_60938_object164499:head v 367'10235, mlcod=367'10231)", 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:20.705218+0000", 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 293.89236466599999, 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022438948, 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.675 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.705218+0000", 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.705218+0000", 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.705215+0000", 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102761 2024-02-12T16:20:14.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.705318+0000", 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6206999999999999e-05 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.705334+0000", 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2370000000000003e-06 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.705341+0000", 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006060518 2024-02-12T16:20:14.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.711402+0000", 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8385e-05 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.711430+0000", 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016184417 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.727615+0000", 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1928000000000002e-05 2024-02-12T16:20:14.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.727657+0000", 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022438948 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:170214 4.2 e369/334 4:4f25c8ea:::benchmark_data_smithi169_60938_object170213:head v 369'10487, mlcod=369'10485)", 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:25.416698+0000", 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 289.18088453899998, 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025058187999999999, 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.679 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416698+0000", 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416698+0000", 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416696+0000", 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6552000000000003e-05 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416752+0000", 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5187999999999999e-05 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416768+0000", 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7989999999999996e-06 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416774+0000", 2024-02-12T16:20:14.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4601000000000002e-05 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416819+0000", 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6120999999999999e-05 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.416845+0000", 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024858188999999999 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.441703+0000", 2024-02-12T16:20:14.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2683999999999999e-05 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:25.441756+0000", 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025058187999999999 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259208 4.1 e395/334 4:8edb39d9:::benchmark_data_smithi169_60938_object85871:head v 395'16079, mlcod=395'16076)", 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.408334+0000", 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 205.18924791200001, 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021771950000000002, 2024-02-12T16:20:14.683 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.408334+0000", 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.408334+0000", 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.408332+0000", 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5180999999999999e-05 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.408347+0000", 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0102000000000002e-05 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.408367+0000", 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8380000000000002e-06 2024-02-12T16:20:14.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.408375+0000", 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000637153 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.409012+0000", 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4087000000000002e-05 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.409036+0000", 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021029104999999999 2024-02-12T16:20:14.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.430065+0000", 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1294000000000001e-05 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.430106+0000", 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021771950000000002 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.687 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:288280 4.2 e406/334 4:41ec2e61:::benchmark_data_smithi169_60938_object114943:head v 406'17699, mlcod=406'17697)", 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:17:20.445462+0000", 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 174.15211998300001, 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022019459000000002, 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445462+0000", 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445462+0000", 2024-02-12T16:20:14.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445460+0000", 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8464000000000001e-05 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445478+0000", 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8337000000000001e-05 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445497+0000", 2024-02-12T16:20:14.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.61e-06 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445504+0000", 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8131000000000002e-05 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445572+0000", 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4776e-05 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.445607+0000", 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021822894999999998 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.467430+0000", 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1897999999999997e-05 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:20.467482+0000", 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022019459000000002 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.691 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:294982 4.2 e406/334 4:4a28840a:::benchmark_data_smithi169_60938_object121645:head v 406'18124, mlcod=406'18122)", 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:17:26.611213+0000", 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 167.98636920300001, 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023823855000000001, 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611213+0000", 2024-02-12T16:20:14.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611213+0000", 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611211+0000", 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1533e-05 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611222+0000", 2024-02-12T16:20:14.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6804999999999999e-05 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611239+0000", 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5339999999999996e-06 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611246+0000", 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7521999999999997e-05 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611293+0000", 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7155000000000001e-05 2024-02-12T16:20:14.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.611310+0000", 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023696037999999999 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.635006+0000", 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0812000000000002e-05 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:26.635037+0000", 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023823855000000001 2024-02-12T16:20:14.695 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:301308 4.1 e407/334 4:81e8d6b1:::benchmark_data_smithi169_60938_object127971:head v 407'18680, mlcod=407'18675)", 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:17:32.228065+0000", 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 162.36951695100001, 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022719539, 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.228065+0000", 2024-02-12T16:20:14.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.228065+0000", 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.228063+0000", 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8961999999999999e-05 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.228082+0000", 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4722000000000001e-05 2024-02-12T16:20:14.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.228096+0000", 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5250000000000005e-06 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.228105+0000", 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063524540000000004 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.234457+0000", 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8246999999999999e-05 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.234486+0000", 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016262961999999999 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.250749+0000", 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6424000000000002e-05 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:17:32.250785+0000", 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022719539 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.699 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:14.700 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:14.700 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:20:14.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:14.735 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:14.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:14.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:14.874 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:14.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:14.893 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.1 primary-affinity to 0.907595 (8594802) 2024-02-12T16:20:14.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:15.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:15.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:14 smithi123 bash[19119]: cluster 2024-02-12T16:20:13.819126+0000 mgr.y (mgr.14146) 1049 : cluster [DBG] pgmap v1414: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:14 smithi123 bash[19119]: audit 2024-02-12T16:20:14.135856+0000 mon.a (mon.0) 1721 : audit [DBG] from='client.? v1:172.21.15.121:0/2410581026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:14 smithi123 bash[19119]: audit 2024-02-12T16:20:14.433085+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.121:0/644313768' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9075945314633669}]: dispatch 2024-02-12T16:20:15.150 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:15.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:14 smithi121 bash[18489]: cluster 2024-02-12T16:20:13.819126+0000 mgr.y (mgr.14146) 1049 : cluster [DBG] pgmap v1414: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:15.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:14 smithi121 bash[18489]: audit 2024-02-12T16:20:14.135856+0000 mon.a (mon.0) 1721 : audit [DBG] from='client.? v1:172.21.15.121:0/2410581026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:15.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:14 smithi121 bash[18489]: audit 2024-02-12T16:20:14.433085+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.121:0/644313768' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9075945314633669}]: dispatch 2024-02-12T16:20:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:15.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:15.267 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:14 smithi169 bash[19604]: cluster 2024-02-12T16:20:13.819126+0000 mgr.y (mgr.14146) 1049 : cluster [DBG] pgmap v1414: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:15.267 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:14 smithi169 bash[19604]: audit 2024-02-12T16:20:14.135856+0000 mon.a (mon.0) 1721 : audit [DBG] from='client.? v1:172.21.15.121:0/2410581026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:15.267 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:14 smithi169 bash[19604]: audit 2024-02-12T16:20:14.433085+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.121:0/644313768' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9075945314633669}]: dispatch 2024-02-12T16:20:15.271 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:20:15.289 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:15.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:15.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:15.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:15.566 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:15.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:15.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:15.852 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:15.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:15.968 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:15 smithi123 bash[19119]: audit 2024-02-12T16:20:14.881352+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? v1:172.21.15.121:0/644313768' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9075945314633669}]': finished 2024-02-12T16:20:15.969 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:15 smithi123 bash[19119]: cluster 2024-02-12T16:20:14.881433+0000 mon.a (mon.0) 1724 : cluster [DBG] osdmap e467: 12 total, 11 up, 12 in 2024-02-12T16:20:15.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:15.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:20:16.060 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:20:16.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:16.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:16.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:16.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:15 smithi121 bash[18489]: audit 2024-02-12T16:20:14.881352+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? v1:172.21.15.121:0/644313768' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9075945314633669}]': finished 2024-02-12T16:20:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:15 smithi121 bash[18489]: cluster 2024-02-12T16:20:14.881433+0000 mon.a (mon.0) 1724 : cluster [DBG] osdmap e467: 12 total, 11 up, 12 in 2024-02-12T16:20:16.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:16.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:15 smithi169 bash[19604]: audit 2024-02-12T16:20:14.881352+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? v1:172.21.15.121:0/644313768' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9075945314633669}]': finished 2024-02-12T16:20:16.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:15 smithi169 bash[19604]: cluster 2024-02-12T16:20:14.881433+0000 mon.a (mon.0) 1724 : cluster [DBG] osdmap e467: 12 total, 11 up, 12 in 2024-02-12T16:20:16.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:16.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:16.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:16.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:16.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:16.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:16.554 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:16.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:16.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:16.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:16.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:16.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:16.833 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:16.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:16.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:16.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:16 smithi121 bash[18489]: cluster 2024-02-12T16:20:15.820270+0000 mgr.y (mgr.14146) 1050 : cluster [DBG] pgmap v1416: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:16.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:16 smithi121 bash[18489]: audit 2024-02-12T16:20:16.745892+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? v1:172.21.15.121:0/2059213026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:20:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:16.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:17.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:17.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:17.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:17.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:16 smithi123 bash[19119]: cluster 2024-02-12T16:20:15.820270+0000 mgr.y (mgr.14146) 1050 : cluster [DBG] pgmap v1416: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:17.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:16 smithi123 bash[19119]: audit 2024-02-12T16:20:16.745892+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? v1:172.21.15.121:0/2059213026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:20:17.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:17.257 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:17.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:16 smithi169 bash[19604]: cluster 2024-02-12T16:20:15.820270+0000 mgr.y (mgr.14146) 1050 : cluster [DBG] pgmap v1416: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:16 smithi169 bash[19604]: audit 2024-02-12T16:20:16.745892+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? v1:172.21.15.121:0/2059213026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:20:17.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:17.397 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:17.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:17.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:17.531 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:20:17.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:17.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:17.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:17.681 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:17.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:17.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:17.822 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:17.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:17.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:17.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:17.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:18.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:20:18.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:18.076 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:17 smithi123 bash[19119]: cluster 2024-02-12T16:20:17.509630+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:18.076 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:17 smithi123 bash[19119]: audit 2024-02-12T16:20:17.521585+0000 mon.a (mon.0) 1727 : audit [INF] from='client.? v1:172.21.15.121:0/2059213026' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:20:18.076 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:17 smithi123 bash[19119]: cluster 2024-02-12T16:20:17.521675+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e468: 12 total, 11 up, 12 in 2024-02-12T16:20:18.096 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:18.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:17 smithi121 bash[18489]: cluster 2024-02-12T16:20:17.509630+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:17 smithi121 bash[18489]: audit 2024-02-12T16:20:17.521585+0000 mon.a (mon.0) 1727 : audit [INF] from='client.? v1:172.21.15.121:0/2059213026' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:20:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:17 smithi121 bash[18489]: cluster 2024-02-12T16:20:17.521675+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e468: 12 total, 11 up, 12 in 2024-02-12T16:20:18.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:18.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:18.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:18.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:17 smithi169 bash[19604]: cluster 2024-02-12T16:20:17.509630+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:17 smithi169 bash[19604]: audit 2024-02-12T16:20:17.521585+0000 mon.a (mon.0) 1727 : audit [INF] from='client.? v1:172.21.15.121:0/2059213026' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:20:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:17 smithi169 bash[19604]: cluster 2024-02-12T16:20:17.521675+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e468: 12 total, 11 up, 12 in 2024-02-12T16:20:18.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:18.374 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:18.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:18.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:18.516 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:18.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:18.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:18.655 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:18.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:18.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:18.795 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:18.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:18.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:18.937 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:18.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:19.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:19.079 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:19.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:19.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:18 smithi123 bash[19119]: cluster 2024-02-12T16:20:17.821264+0000 mgr.y (mgr.14146) 1051 : cluster [DBG] pgmap v1418: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:19.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:18 smithi121 bash[18489]: cluster 2024-02-12T16:20:17.821264+0000 mgr.y (mgr.14146) 1051 : cluster [DBG] pgmap v1418: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:19.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:19.220 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:19.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:19.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:18 smithi169 bash[19604]: cluster 2024-02-12T16:20:17.821264+0000 mgr.y (mgr.14146) 1051 : cluster [DBG] pgmap v1418: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:19.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:19.360 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:19.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:19.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:19.503 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:19.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:19.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:19.645 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:19.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:19.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:19.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:19.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:19.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:19.929 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:19.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:20.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:20.069 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:20.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:20.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:20.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:20.215 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:20.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:20.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:20:20.319 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:20.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:20.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:20.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:20.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:20.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:20.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:20.633 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:20.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:20.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:20.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:20.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:20.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:20.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:20.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:21.024 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:21.036 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:20 smithi169 bash[19604]: cluster 2024-02-12T16:20:19.822345+0000 mgr.y (mgr.14146) 1052 : cluster [DBG] pgmap v1419: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:21.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:20 smithi169 bash[19604]: cluster 2024-02-12T16:20:20.506336+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e469: 12 total, 11 up, 12 in 2024-02-12T16:20:21.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:20 smithi169 bash[19604]: cluster 2024-02-12T16:20:20.521503+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:21.037 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:20 smithi169 bash[19604]: cluster 2024-02-12T16:20:20.521584+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:21.061 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:21.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:20 smithi121 bash[18489]: cluster 2024-02-12T16:20:19.822345+0000 mgr.y (mgr.14146) 1052 : cluster [DBG] pgmap v1419: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:20 smithi121 bash[18489]: cluster 2024-02-12T16:20:20.506336+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e469: 12 total, 11 up, 12 in 2024-02-12T16:20:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:20 smithi121 bash[18489]: cluster 2024-02-12T16:20:20.521503+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:20 smithi121 bash[18489]: cluster 2024-02-12T16:20:20.521584+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:21.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:21.201 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:21.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:21.213 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:20 smithi123 bash[19119]: cluster 2024-02-12T16:20:19.822345+0000 mgr.y (mgr.14146) 1052 : cluster [DBG] pgmap v1419: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:21.213 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:20 smithi123 bash[19119]: cluster 2024-02-12T16:20:20.506336+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e469: 12 total, 11 up, 12 in 2024-02-12T16:20:21.213 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:20 smithi123 bash[19119]: cluster 2024-02-12T16:20:20.521503+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:21.214 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:20 smithi123 bash[19119]: cluster 2024-02-12T16:20:20.521584+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:21.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:21.333 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:20:21.333 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:20:21.333 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-12T16:20:21.333 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:20:21.333 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:20:21.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-12T16:20:21.346 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:21.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:21.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:21.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:21.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:21.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:21.620 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:21.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:21.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:21.754 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:21.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:21.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:21.896 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:21.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:22.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:22.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:22.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:22.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:22.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:22.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:22.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:22.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:22.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:22.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:22.464 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:22.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:22.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:22 smithi169 bash[19604]: cluster 2024-02-12T16:20:21.520177+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e470: 12 total, 11 up, 12 in 2024-02-12T16:20:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:22 smithi169 bash[19604]: cluster 2024-02-12T16:20:21.823422+0000 mgr.y (mgr.14146) 1053 : cluster [DBG] pgmap v1422: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:22.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:22.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:22.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:22.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:22.734 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:22.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:22.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:22.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:22 smithi123 bash[19119]: cluster 2024-02-12T16:20:21.520177+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e470: 12 total, 11 up, 12 in 2024-02-12T16:20:22.851 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:22 smithi123 bash[19119]: cluster 2024-02-12T16:20:21.823422+0000 mgr.y (mgr.14146) 1053 : cluster [DBG] pgmap v1422: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:22.855 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:22.872 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:22.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:22.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:22 smithi121 bash[18489]: cluster 2024-02-12T16:20:21.520177+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e470: 12 total, 11 up, 12 in 2024-02-12T16:20:22.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:22 smithi121 bash[18489]: cluster 2024-02-12T16:20:21.823422+0000 mgr.y (mgr.14146) 1053 : cluster [DBG] pgmap v1422: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:22.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:23.009 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:23.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:23.123 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:20:23.123 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:20:23.123 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:20:23.123 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:20:23.150 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:23.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:23.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:23.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:23.428 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:23.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:23.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:23.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:23.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:20:23.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:23.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:23.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:23.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:23.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:24.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:24.129 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:24.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:24.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:24.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:24.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:24.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:24.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:24.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:24.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:24.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:24.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:24.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:24.688 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:24.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:24.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:24.826 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:24.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:24.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:24.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:24.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:24.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:25.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:25.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:24 smithi169 bash[19604]: cluster 2024-02-12T16:20:23.824553+0000 mgr.y (mgr.14146) 1054 : cluster [DBG] pgmap v1423: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:25.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:25.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:24 smithi123 bash[19119]: cluster 2024-02-12T16:20:23.824553+0000 mgr.y (mgr.14146) 1054 : cluster [DBG] pgmap v1423: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:25.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:24 smithi121 bash[18489]: cluster 2024-02-12T16:20:23.824553+0000 mgr.y (mgr.14146) 1054 : cluster [DBG] pgmap v1423: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:25.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:25.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:25.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:25.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:25.388 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:25.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:25.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:25.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:25.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:25.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:25.679 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:25.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:25.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:25.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:25.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:25.953 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:25.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:26.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:26.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:26.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:26.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:26.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:26.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:26.303 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:26.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:26.365 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:26.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:26.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:26.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:26.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:26.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:26 smithi123 bash[19119]: cluster 2024-02-12T16:20:25.511269+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e471: 12 total, 11 up, 12 in 2024-02-12T16:20:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:26 smithi123 bash[19119]: cluster 2024-02-12T16:20:25.825575+0000 mgr.y (mgr.14146) 1055 : cluster [DBG] pgmap v1425: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:26.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:26.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:26.705 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:20:26.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:20:26.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:26.747 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:20:26.753 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:26 smithi169 bash[19604]: cluster 2024-02-12T16:20:25.511269+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e471: 12 total, 11 up, 12 in 2024-02-12T16:20:26.753 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:26 smithi169 bash[19604]: cluster 2024-02-12T16:20:25.825575+0000 mgr.y (mgr.14146) 1055 : cluster [DBG] pgmap v1425: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:26.772 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:20:26.775 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:26.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:26.797 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:20:26.819 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:20:26.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:26 smithi121 bash[18489]: cluster 2024-02-12T16:20:25.511269+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e471: 12 total, 11 up, 12 in 2024-02-12T16:20:26.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:26 smithi121 bash[18489]: cluster 2024-02-12T16:20:25.825575+0000 mgr.y (mgr.14146) 1055 : cluster [DBG] pgmap v1425: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:26.840 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:20:26.871 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:20:26.879 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:26.896 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:20:26.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:26.921 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:20:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:26.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:26.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:20:26.944 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:20:26.945 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:20:27.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:27.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:27.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:27.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:27.200 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:27.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:27.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:27.340 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:27.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:27.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:27.484 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:27.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:27.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:27.594 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:27 smithi123 bash[19119]: audit 2024-02-12T16:20:26.674746+0000 mon.a (mon.0) 1732 : audit [DBG] from='client.? v1:172.21.15.121:0/309481126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:27.624 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:27.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:27.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:27.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:27 smithi169 bash[19604]: audit 2024-02-12T16:20:26.674746+0000 mon.a (mon.0) 1732 : audit [DBG] from='client.? v1:172.21.15.121:0/309481126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:27.767 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:27.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:27.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:27.909 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:27.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:27 smithi121 bash[18489]: audit 2024-02-12T16:20:26.674746+0000 mon.a (mon.0) 1732 : audit [DBG] from='client.? v1:172.21.15.121:0/309481126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:27.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:28.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:28.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:28.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:28.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:28.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:28.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:28.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:28.326 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:28.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:28.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:28.471 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:28.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:20:28.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:20:28.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:28.610 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:28.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:28.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:28.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:28.753 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:28 smithi121 bash[18489]: cluster 2024-02-12T16:20:27.826598+0000 mgr.y (mgr.14146) 1056 : cluster [DBG] pgmap v1426: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:28.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:28.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:28 smithi169 bash[19604]: cluster 2024-02-12T16:20:27.826598+0000 mgr.y (mgr.14146) 1056 : cluster [DBG] pgmap v1426: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:28.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:28.863 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:28.868 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:28 smithi123 bash[19119]: cluster 2024-02-12T16:20:27.826598+0000 mgr.y (mgr.14146) 1056 : cluster [DBG] pgmap v1426: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:28.890 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:28.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:28.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:29.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:29.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:29.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:29.182 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:29.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:29.285 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:29.322 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:29.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:29.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:29.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:29.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:29.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:29.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:29.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:29.661 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:20:29.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:29.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:29.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:29.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:29.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:29 smithi123 bash[19119]: audit 2024-02-12T16:20:29.594512+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? v1:172.21.15.121:0/606958894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:20:29.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:29 smithi123 bash[19119]: cluster 2024-02-12T16:20:29.645166+0000 mon.a (mon.0) 1734 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:29.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:29.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:29 smithi121 bash[18489]: audit 2024-02-12T16:20:29.594512+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? v1:172.21.15.121:0/606958894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:20:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:29 smithi121 bash[18489]: cluster 2024-02-12T16:20:29.645166+0000 mon.a (mon.0) 1734 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:29.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:29.995 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:29 smithi169 bash[19604]: audit 2024-02-12T16:20:29.594512+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? v1:172.21.15.121:0/606958894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:20:29.996 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:29 smithi169 bash[19604]: cluster 2024-02-12T16:20:29.645166+0000 mon.a (mon.0) 1734 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:30.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:30.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:30.159 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:30.237 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:20:30.237 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "" 2024-02-12T16:20:30.237 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:20:30.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:30.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:30.404 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:30.434 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:30.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:20:30.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:30.566 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:30.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:30.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:30.705 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:30.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:30.840 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:30.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:30.958 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:30 smithi123 bash[19119]: audit 2024-02-12T16:20:29.653002+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? v1:172.21.15.121:0/606958894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:20:30.958 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:30 smithi123 bash[19119]: cluster 2024-02-12T16:20:29.653186+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e472: 12 total, 11 up, 12 in 2024-02-12T16:20:30.958 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:30 smithi123 bash[19119]: cluster 2024-02-12T16:20:29.827565+0000 mgr.y (mgr.14146) 1057 : cluster [DBG] pgmap v1428: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:30.958 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:30 smithi123 bash[19119]: cluster 2024-02-12T16:20:30.512282+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e473: 12 total, 11 up, 12 in 2024-02-12T16:20:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:30.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:30 smithi169 bash[19604]: audit 2024-02-12T16:20:29.653002+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? v1:172.21.15.121:0/606958894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:20:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:30 smithi169 bash[19604]: cluster 2024-02-12T16:20:29.653186+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e472: 12 total, 11 up, 12 in 2024-02-12T16:20:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:30 smithi169 bash[19604]: cluster 2024-02-12T16:20:29.827565+0000 mgr.y (mgr.14146) 1057 : cluster [DBG] pgmap v1428: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:30 smithi169 bash[19604]: cluster 2024-02-12T16:20:30.512282+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e473: 12 total, 11 up, 12 in 2024-02-12T16:20:31.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:31.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:31.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:30 smithi121 bash[18489]: audit 2024-02-12T16:20:29.653002+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? v1:172.21.15.121:0/606958894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:20:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:30 smithi121 bash[18489]: cluster 2024-02-12T16:20:29.653186+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e472: 12 total, 11 up, 12 in 2024-02-12T16:20:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:30 smithi121 bash[18489]: cluster 2024-02-12T16:20:29.827565+0000 mgr.y (mgr.14146) 1057 : cluster [DBG] pgmap v1428: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:30 smithi121 bash[18489]: cluster 2024-02-12T16:20:30.512282+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e473: 12 total, 11 up, 12 in 2024-02-12T16:20:31.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:31.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:31.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:31.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:31.404 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:31.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:31.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:31.539 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:31.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:31.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:31.679 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:31.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:31.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:31.821 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:31.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:31 smithi123 bash[19119]: cluster 2024-02-12T16:20:30.722364+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:31 smithi123 bash[19119]: cluster 2024-02-12T16:20:30.722456+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:31.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:31 smithi123 bash[19119]: cluster 2024-02-12T16:20:31.297856+0000 osd.7 (osd.7) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:31.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:31 smithi123 bash[19119]: cluster 2024-02-12T16:20:31.297985+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:31.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:31 smithi123 bash[19119]: cluster 2024-02-12T16:20:31.433731+0000 osd.7 (osd.7) 99 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:20:31.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:31 smithi123 bash[19119]: cluster 2024-02-12T16:20:31.483195+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:20:31.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:31.961 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:31.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:31 smithi169 bash[19604]: cluster 2024-02-12T16:20:30.722364+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:31 smithi169 bash[19604]: cluster 2024-02-12T16:20:30.722456+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:31 smithi169 bash[19604]: cluster 2024-02-12T16:20:31.297856+0000 osd.7 (osd.7) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:31 smithi169 bash[19604]: cluster 2024-02-12T16:20:31.297985+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:31 smithi169 bash[19604]: cluster 2024-02-12T16:20:31.433731+0000 osd.7 (osd.7) 99 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:20:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:31 smithi169 bash[19604]: cluster 2024-02-12T16:20:31.483195+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:20:32.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:32.106 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:32.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:31 smithi121 bash[18489]: cluster 2024-02-12T16:20:30.722364+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:31 smithi121 bash[18489]: cluster 2024-02-12T16:20:30.722456+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:31 smithi121 bash[18489]: cluster 2024-02-12T16:20:31.297856+0000 osd.7 (osd.7) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:31 smithi121 bash[18489]: cluster 2024-02-12T16:20:31.297985+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:31 smithi121 bash[18489]: cluster 2024-02-12T16:20:31.433731+0000 osd.7 (osd.7) 99 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:20:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:31 smithi121 bash[18489]: cluster 2024-02-12T16:20:31.483195+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:20:32.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:32.241 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:32.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:32.344 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:32.380 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:32.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:32.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:32.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:20:32.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:32.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:32.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:32.662 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:32.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:32.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:32.793 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:20:32.793 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:20:32.795 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:32.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:32.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:32.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:32 smithi121 bash[18489]: cluster 2024-02-12T16:20:31.673119+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e474: 12 total, 11 up, 12 in 2024-02-12T16:20:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:32 smithi121 bash[18489]: cluster 2024-02-12T16:20:31.828687+0000 mgr.y (mgr.14146) 1058 : cluster [DBG] pgmap v1431: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:32.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:32.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:33.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:33.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:32 smithi169 bash[19604]: cluster 2024-02-12T16:20:31.673119+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e474: 12 total, 11 up, 12 in 2024-02-12T16:20:33.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:32 smithi169 bash[19604]: cluster 2024-02-12T16:20:31.828687+0000 mgr.y (mgr.14146) 1058 : cluster [DBG] pgmap v1431: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:33.068 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:33.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:32 smithi123 bash[19119]: cluster 2024-02-12T16:20:31.673119+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e474: 12 total, 11 up, 12 in 2024-02-12T16:20:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:32 smithi123 bash[19119]: cluster 2024-02-12T16:20:31.828687+0000 mgr.y (mgr.14146) 1058 : cluster [DBG] pgmap v1431: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:33.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:33.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:33.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:33.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:33.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:33.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:33.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:33.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:20:33.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:33.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:33.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:33.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:33.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:33.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:33.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:33.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:33.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:33 smithi169 bash[19604]: cluster 2024-02-12T16:20:32.212220+0000 osd.6 (osd.6) 74 : cluster [DBG] 1.f scrub starts 2024-02-12T16:20:33.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:33 smithi169 bash[19604]: cluster 2024-02-12T16:20:32.237047+0000 osd.6 (osd.6) 75 : cluster [DBG] 1.f scrub ok 2024-02-12T16:20:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:33 smithi169 bash[19604]: cluster 2024-02-12T16:20:32.607643+0000 osd.10 (osd.10) 74 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:20:33.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:33.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:33.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:34.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:34.034 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:33 smithi123 bash[19119]: cluster 2024-02-12T16:20:32.212220+0000 osd.6 (osd.6) 74 : cluster [DBG] 1.f scrub starts 2024-02-12T16:20:34.034 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:33 smithi123 bash[19119]: cluster 2024-02-12T16:20:32.237047+0000 osd.6 (osd.6) 75 : cluster [DBG] 1.f scrub ok 2024-02-12T16:20:34.034 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:33 smithi123 bash[19119]: cluster 2024-02-12T16:20:32.607643+0000 osd.10 (osd.10) 74 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:20:34.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:34.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:33 smithi121 bash[18489]: cluster 2024-02-12T16:20:32.212220+0000 osd.6 (osd.6) 74 : cluster [DBG] 1.f scrub starts 2024-02-12T16:20:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:33 smithi121 bash[18489]: cluster 2024-02-12T16:20:32.237047+0000 osd.6 (osd.6) 75 : cluster [DBG] 1.f scrub ok 2024-02-12T16:20:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:33 smithi121 bash[18489]: cluster 2024-02-12T16:20:32.607643+0000 osd.10 (osd.10) 74 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:20:34.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:34.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:34.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:34.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:34.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:34.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:34.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:34.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:34.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:34.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:34.703 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:34 smithi123 bash[19119]: cluster 2024-02-12T16:20:32.960157+0000 osd.10 (osd.10) 75 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:20:34.704 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:34 smithi123 bash[19119]: cluster 2024-02-12T16:20:33.011822+0000 osd.9 (osd.9) 69 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:20:34.704 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:34 smithi123 bash[19119]: cluster 2024-02-12T16:20:33.365823+0000 osd.9 (osd.9) 70 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:20:34.704 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:34 smithi123 bash[19119]: cluster 2024-02-12T16:20:33.428137+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:34.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:34.749 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:34.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:34.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:34.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:34.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:34.995 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:35.006 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:34 smithi123 bash[19119]: cluster 2024-02-12T16:20:33.428283+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:35.006 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:34 smithi123 bash[19119]: cluster 2024-02-12T16:20:33.700232+0000 mon.a (mon.0) 1739 : cluster [DBG] osdmap e475: 12 total, 11 up, 12 in 2024-02-12T16:20:35.006 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:34 smithi123 bash[19119]: cluster 2024-02-12T16:20:33.829660+0000 mgr.y (mgr.14146) 1059 : cluster [DBG] pgmap v1433: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:35.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:34 smithi169 bash[19604]: cluster 2024-02-12T16:20:32.960157+0000 osd.10 (osd.10) 75 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:20:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:34 smithi169 bash[19604]: cluster 2024-02-12T16:20:33.011822+0000 osd.9 (osd.9) 69 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:20:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:34 smithi169 bash[19604]: cluster 2024-02-12T16:20:33.365823+0000 osd.9 (osd.9) 70 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:20:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:34 smithi169 bash[19604]: cluster 2024-02-12T16:20:33.428137+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:34 smithi169 bash[19604]: cluster 2024-02-12T16:20:33.428283+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:34 smithi169 bash[19604]: cluster 2024-02-12T16:20:33.700232+0000 mon.a (mon.0) 1739 : cluster [DBG] osdmap e475: 12 total, 11 up, 12 in 2024-02-12T16:20:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:34 smithi169 bash[19604]: cluster 2024-02-12T16:20:33.829660+0000 mgr.y (mgr.14146) 1059 : cluster [DBG] pgmap v1433: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:35.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:35.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:34 smithi121 bash[18489]: cluster 2024-02-12T16:20:32.960157+0000 osd.10 (osd.10) 75 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:20:35.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:34 smithi121 bash[18489]: cluster 2024-02-12T16:20:33.011822+0000 osd.9 (osd.9) 69 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:20:35.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:34 smithi121 bash[18489]: cluster 2024-02-12T16:20:33.365823+0000 osd.9 (osd.9) 70 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:20:35.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:34 smithi121 bash[18489]: cluster 2024-02-12T16:20:33.428137+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:35.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:34 smithi121 bash[18489]: cluster 2024-02-12T16:20:33.428283+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:35.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:34 smithi121 bash[18489]: cluster 2024-02-12T16:20:33.700232+0000 mon.a (mon.0) 1739 : cluster [DBG] osdmap e475: 12 total, 11 up, 12 in 2024-02-12T16:20:35.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:34 smithi121 bash[18489]: cluster 2024-02-12T16:20:33.829660+0000 mgr.y (mgr.14146) 1059 : cluster [DBG] pgmap v1433: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:35.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:35.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:35.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:35.281 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:35.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:35.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:35.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:35.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:35.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:35.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:35.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:35.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:35.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:35.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:35 smithi169 bash[19604]: cluster 2024-02-12T16:20:34.326926+0000 osd.11 (osd.11) 89 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:20:35.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:35 smithi169 bash[19604]: cluster 2024-02-12T16:20:34.698600+0000 osd.11 (osd.11) 90 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:20:35.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:35.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:35.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:35.872 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:35.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:35.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:35 smithi121 bash[18489]: cluster 2024-02-12T16:20:34.326926+0000 osd.11 (osd.11) 89 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:20:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:35 smithi121 bash[18489]: cluster 2024-02-12T16:20:34.698600+0000 osd.11 (osd.11) 90 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:20:35.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:35.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:35 smithi123 bash[19119]: cluster 2024-02-12T16:20:34.326926+0000 osd.11 (osd.11) 89 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:20:35.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:35 smithi123 bash[19119]: cluster 2024-02-12T16:20:34.698600+0000 osd.11 (osd.11) 90 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:20:36.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:36.044 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:20:36.045 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:20:36.045 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T16:20:36.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:20:36.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:36.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:36.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:36.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:36.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:36.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:36.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:36.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:36.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:36.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:36.705 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:36.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:36.848 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:36.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:36 smithi121 bash[18489]: cluster 2024-02-12T16:20:35.212625+0000 osd.6 (osd.6) 76 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:20:36.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:36 smithi121 bash[18489]: cluster 2024-02-12T16:20:35.237902+0000 osd.6 (osd.6) 77 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:20:36.859 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:36 smithi121 bash[18489]: cluster 2024-02-12T16:20:35.304093+0000 osd.11 (osd.11) 91 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:20:36.859 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:36 smithi121 bash[18489]: cluster 2024-02-12T16:20:35.769037+0000 osd.11 (osd.11) 92 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:20:36.859 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:36 smithi121 bash[18489]: cluster 2024-02-12T16:20:35.830778+0000 mgr.y (mgr.14146) 1060 : cluster [DBG] pgmap v1434: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:36.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:36.963 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:36 smithi123 bash[19119]: cluster 2024-02-12T16:20:35.212625+0000 osd.6 (osd.6) 76 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:20:36.963 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:36 smithi123 bash[19119]: cluster 2024-02-12T16:20:35.237902+0000 osd.6 (osd.6) 77 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:20:36.963 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:36 smithi123 bash[19119]: cluster 2024-02-12T16:20:35.304093+0000 osd.11 (osd.11) 91 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:20:36.963 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:36 smithi123 bash[19119]: cluster 2024-02-12T16:20:35.769037+0000 osd.11 (osd.11) 92 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:20:36.963 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:36 smithi123 bash[19119]: cluster 2024-02-12T16:20:35.830778+0000 mgr.y (mgr.14146) 1060 : cluster [DBG] pgmap v1434: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:36.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:36.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:36 smithi169 bash[19604]: cluster 2024-02-12T16:20:35.212625+0000 osd.6 (osd.6) 76 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:20:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:36 smithi169 bash[19604]: cluster 2024-02-12T16:20:35.237902+0000 osd.6 (osd.6) 77 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:20:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:36 smithi169 bash[19604]: cluster 2024-02-12T16:20:35.304093+0000 osd.11 (osd.11) 91 : cluster [DBG] 4.8 scrub starts 2024-02-12T16:20:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:36 smithi169 bash[19604]: cluster 2024-02-12T16:20:35.769037+0000 osd.11 (osd.11) 92 : cluster [DBG] 4.8 scrub ok 2024-02-12T16:20:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:36 smithi169 bash[19604]: cluster 2024-02-12T16:20:35.830778+0000 mgr.y (mgr.14146) 1060 : cluster [DBG] pgmap v1434: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:37.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:37.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:37.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:37.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:37.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:37.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:37.311 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:37.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:37.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:37.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:37.517 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:37.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:37.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:37.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:37.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:37.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:37.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:37.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:37.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:37.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:37.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:38.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:38.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:38.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:38.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:38.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:38.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:38.286 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:38.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:38.408 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:38.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:38.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:38.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:38.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:38.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:38.689 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:38.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:38.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:38.826 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:38.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:38.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:38.964 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:38.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:39.068 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:39.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:38 smithi121 bash[18489]: cluster 2024-02-12T16:20:37.831832+0000 mgr.y (mgr.14146) 1061 : cluster [DBG] pgmap v1435: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:39.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:38 smithi121 bash[18489]: cluster 2024-02-12T16:20:37.967141+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:39.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:38 smithi121 bash[18489]: cluster 2024-02-12T16:20:37.967240+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:39.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:38 smithi121 bash[18489]: cluster 2024-02-12T16:20:38.447152+0000 osd.7 (osd.7) 101 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:20:39.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:38 smithi121 bash[18489]: cluster 2024-02-12T16:20:38.480935+0000 osd.7 (osd.7) 102 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:20:39.104 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:39.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:38 smithi123 bash[19119]: cluster 2024-02-12T16:20:37.831832+0000 mgr.y (mgr.14146) 1061 : cluster [DBG] pgmap v1435: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:38 smithi123 bash[19119]: cluster 2024-02-12T16:20:37.967141+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:38 smithi123 bash[19119]: cluster 2024-02-12T16:20:37.967240+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:38 smithi123 bash[19119]: cluster 2024-02-12T16:20:38.447152+0000 osd.7 (osd.7) 101 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:20:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:38 smithi123 bash[19119]: cluster 2024-02-12T16:20:38.480935+0000 osd.7 (osd.7) 102 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:20:39.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:38 smithi169 bash[19604]: cluster 2024-02-12T16:20:37.831832+0000 mgr.y (mgr.14146) 1061 : cluster [DBG] pgmap v1435: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:38 smithi169 bash[19604]: cluster 2024-02-12T16:20:37.967141+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:38 smithi169 bash[19604]: cluster 2024-02-12T16:20:37.967240+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:38 smithi169 bash[19604]: cluster 2024-02-12T16:20:38.447152+0000 osd.7 (osd.7) 101 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:20:39.222 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:38 smithi169 bash[19604]: cluster 2024-02-12T16:20:38.480935+0000 osd.7 (osd.7) 102 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:20:39.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:39.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:39.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:39.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:39.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:39.392 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:20:39.412 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:20:39.434 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:20:39.457 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:20:39.479 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:20:39.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:39.499 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:20:39.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:39.522 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:20:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:39.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:39.545 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:20:39.569 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:20:39.594 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:20:39.620 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:20:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.621 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:20:39.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:39.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:39.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:39.801 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:39.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:39.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:39.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:39.941 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:39.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:40.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:39 smithi169 bash[19604]: cluster 2024-02-12T16:20:38.695829+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:39 smithi169 bash[19604]: cluster 2024-02-12T16:20:38.695911+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:39 smithi169 bash[19604]: cluster 2024-02-12T16:20:38.894016+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e476: 12 total, 11 up, 12 in 2024-02-12T16:20:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:39 smithi169 bash[19604]: audit 2024-02-12T16:20:39.360277+0000 mon.a (mon.0) 1741 : audit [DBG] from='client.? v1:172.21.15.121:0/1805766844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:39 smithi169 bash[19604]: cluster 2024-02-12T16:20:39.420310+0000 osd.7 (osd.7) 103 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:20:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:39 smithi169 bash[19604]: cluster 2024-02-12T16:20:39.470991+0000 osd.7 (osd.7) 104 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:20:40.056 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:39 smithi123 bash[19119]: cluster 2024-02-12T16:20:38.695829+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:40.056 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:39 smithi123 bash[19119]: cluster 2024-02-12T16:20:38.695911+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:40.056 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:39 smithi123 bash[19119]: cluster 2024-02-12T16:20:38.894016+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e476: 12 total, 11 up, 12 in 2024-02-12T16:20:40.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:39 smithi123 bash[19119]: audit 2024-02-12T16:20:39.360277+0000 mon.a (mon.0) 1741 : audit [DBG] from='client.? v1:172.21.15.121:0/1805766844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:40.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:39 smithi123 bash[19119]: cluster 2024-02-12T16:20:39.420310+0000 osd.7 (osd.7) 103 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:20:40.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:39 smithi123 bash[19119]: cluster 2024-02-12T16:20:39.470991+0000 osd.7 (osd.7) 104 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:20:40.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:40.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:39 smithi121 bash[18489]: cluster 2024-02-12T16:20:38.695829+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:39 smithi121 bash[18489]: cluster 2024-02-12T16:20:38.695911+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:39 smithi121 bash[18489]: cluster 2024-02-12T16:20:38.894016+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e476: 12 total, 11 up, 12 in 2024-02-12T16:20:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:39 smithi121 bash[18489]: audit 2024-02-12T16:20:39.360277+0000 mon.a (mon.0) 1741 : audit [DBG] from='client.? v1:172.21.15.121:0/1805766844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:39 smithi121 bash[18489]: cluster 2024-02-12T16:20:39.420310+0000 osd.7 (osd.7) 103 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:20:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:39 smithi121 bash[18489]: cluster 2024-02-12T16:20:39.470991+0000 osd.7 (osd.7) 104 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:20:40.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:40.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:40.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:40.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:40.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:40.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:40.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:40.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:40.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:20:40.631 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:20:40.639 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:40.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:40.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:40.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:40.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:40.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:40.924 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:40.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:41.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:41.068 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:41.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:41.113 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:41.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:41.184 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:40 smithi123 bash[19119]: cluster 2024-02-12T16:20:39.832807+0000 mgr.y (mgr.14146) 1062 : cluster [DBG] pgmap v1437: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:41.184 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:40 smithi123 bash[19119]: cluster 2024-02-12T16:20:40.518323+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e477: 12 total, 11 up, 12 in 2024-02-12T16:20:41.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:41.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:40 smithi169 bash[19604]: cluster 2024-02-12T16:20:39.832807+0000 mgr.y (mgr.14146) 1062 : cluster [DBG] pgmap v1437: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:40 smithi169 bash[19604]: cluster 2024-02-12T16:20:40.518323+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e477: 12 total, 11 up, 12 in 2024-02-12T16:20:41.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:41.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:40 smithi121 bash[18489]: cluster 2024-02-12T16:20:39.832807+0000 mgr.y (mgr.14146) 1062 : cluster [DBG] pgmap v1437: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:41.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:40 smithi121 bash[18489]: cluster 2024-02-12T16:20:40.518323+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e477: 12 total, 11 up, 12 in 2024-02-12T16:20:41.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:41.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:41.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:41.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:41.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:41.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287215 3.a 3:55b1025c:::benchmark_data_smithi169_51390_object123431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.504457+0000", 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 487.09430306199999, 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49660404400000002, 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.607 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287215 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504457+0000", 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504457+0000", 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504453+0000", 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2464e-05 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504465+0000", 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7924999999999999e-05 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504483+0000", 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2390000000000002e-06 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504492+0000", 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2698000000000002e-05 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504554+0000", 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9949999999999999e-06 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.504561+0000", 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001445558 2024-02-12T16:20:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506007+0000", 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111267 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506118+0000", 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012511600000000001 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:20:41.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.506243+0000", 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093499770000000006 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515593+0000", 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48534981799999999 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.000943+0000", 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8232000000000007e-05 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.001011+0000", 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9607999999999997e-05 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.001061+0000", 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49660404400000002 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287218 3.a 3:5667ef8a:::benchmark_data_smithi169_51390_object123434:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.507798+0000", 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 487.09096142599998, 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49960972100000001, 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287218 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507798+0000", 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507798+0000", 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507796+0000", 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0628000000000001e-05 2024-02-12T16:20:41.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507807+0000", 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0042e-05 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507817+0000", 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2530000000000004e-06 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.507822+0000", 2024-02-12T16:20:41.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028043299999999998 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510627+0000", 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001171 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510744+0000", 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1235000000000003e-05 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:20:41.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.510835+0000", 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048881100000000002 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.515723+0000", 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49154228500000002 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007265+0000", 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9278999999999998e-05 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007345+0000", 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3569000000000001e-05 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007408+0000", 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49960972100000001 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287232 3.a 3:5fcb2087:::benchmark_data_smithi169_51390_object123448:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.524268+0000", 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 487.07449230600002, 2024-02-12T16:20:41.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48364025700000002, 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287232 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524268+0000", 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524268+0000", 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524265+0000", 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4519999999999997e-06 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524273+0000", 2024-02-12T16:20:41.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0237999999999999e-05 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524284+0000", 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.524e-06 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524289+0000", 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5358999999999998e-05 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524324+0000", 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2391000000000005e-05 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524397+0000", 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7861000000000006e-05 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.524475+0000", 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041166830000000003 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.528591+0000", 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47912998299999998 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007721+0000", 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8727999999999995e-05 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007810+0000", 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7752999999999997e-05 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007908+0000", 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48364025700000002 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.15108.0:287237 3.a 3:5246a6cc:::benchmark_data_smithi169_51390_object123453:head [delete] snapc 0=[] ondisk+write+known_if_redirected e318)", 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:12:34.531869+0000", 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 487.066890833, 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47620760400000001, 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.625 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.15108", 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2390889375", 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 287237 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531869+0000", 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531869+0000", 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531867+0000", 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4760000000000001e-06 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531875+0000", 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5639999999999997e-06 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531883+0000", 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0839999999999999e-06 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531888+0000", 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0645000000000001e-05 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.531929+0000", 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2466999999999994e-05 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.532021+0000", 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8151000000000003e-05 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.532119+0000", 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035702770000000002 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:34.535690+0000", 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.472285854 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.007976+0000", 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8838000000000002e-05 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.008024+0000", 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2268999999999997e-05 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:12:35.008077+0000", 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47620760400000001 2024-02-12T16:20:41.630 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:84587 4.5 4:a4899310:::benchmark_data_smithi169_60938_object84586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:14:26.276967+0000", 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 375.32179260700002, 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.137213996, 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 84587 2024-02-12T16:20:41.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.276967+0000", 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.276967+0000", 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.276966+0000", 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2454999999999997e-05 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277028+0000", 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1540000000000002e-06 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277036+0000", 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8520000000000003e-06 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277041+0000", 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5503e-05 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277067+0000", 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3953000000000002e-05 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277141+0000", 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9172999999999995e-05 2024-02-12T16:20:41.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.277230+0000", 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13590312399999999 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.413133+0000", 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000907087 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.414040+0000", 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2632000000000005e-05 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.414113+0000", 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8708999999999996e-05 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:14:26.414181+0000", 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.137213996 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161566 4.a 4:514202fa:::benchmark_data_smithi169_60938_object161565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.193705+0000", 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.40505508699999, 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.227013722, 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161566 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.637 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193705+0000", 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193705+0000", 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193703+0000", 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9545999999999997e-05 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193742+0000", 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4466e-05 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193757+0000", 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4130000000000003e-06 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193763+0000", 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3257999999999997e-05 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193816+0000", 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014184899999999999 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.193958+0000", 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018540200000000001 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.194143+0000", 2024-02-12T16:20:41.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018365227000000001 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.212509+0000", 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20806068899999999 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.420569+0000", 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4877999999999996e-05 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.420654+0000", 2024-02-12T16:20:41.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4305000000000002e-05 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.420719+0000", 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.227013722 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161571 4.a 4:529281aa:::benchmark_data_smithi169_60938_object161570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.194202+0000", 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.40455776900001, 2024-02-12T16:20:41.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22670487, 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161571 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.194202+0000", 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.194202+0000", 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.194201+0000", 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3200000000000001e-05 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.194234+0000", 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6299999999999998e-06 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.194242+0000", 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2177999999999999e-05 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.194254+0000", 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049326229999999997 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.199187+0000", 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016370499999999999 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.199350+0000", 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000218338 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.199569+0000", 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01294474 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.212513+0000", 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20827880700000001 2024-02-12T16:20:41.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.420792+0000", 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2639999999999997e-05 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.420855+0000", 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2256999999999998e-05 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.420907+0000", 2024-02-12T16:20:41.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22670487 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161574 4.a 4:5a7cd897:::benchmark_data_smithi169_60938_object161573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.198970+0000", 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.39979007300002, 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.226482188, 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.648 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161574 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.198970+0000", 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.198970+0000", 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.198967+0000", 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7061999999999998e-05 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.199014+0000", 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3397999999999999e-05 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.199028+0000", 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1940000000000004e-06 2024-02-12T16:20:41.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.199037+0000", 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0051176829999999996 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.204155+0000", 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016298899999999999 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.204318+0000", 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017988100000000001 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.204497+0000", 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008019132 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.212517+0000", 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.212860667 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.425377+0000", 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7253999999999999e-05 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.425415+0000", 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7508000000000001e-05 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.425452+0000", 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.226482188 2024-02-12T16:20:41.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161580 4.a 4:5ef8a2bb:::benchmark_data_smithi169_60938_object161579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.201520+0000", 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.39724025300001, 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22408313899999999, 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161580 2024-02-12T16:20:41.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.201520+0000", 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.201520+0000", 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999909 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.201511+0000", 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3850000000000001e-05 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.201565+0000", 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6260000000000007e-06 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.201574+0000", 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7090000000000001e-06 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.201579+0000", 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072341059999999997 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.208813+0000", 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012950700000000001 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.208942+0000", 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102335 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.209045+0000", 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035780959999999998 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.212623+0000", 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21288628000000001 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.425509+0000", 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2657000000000001e-05 2024-02-12T16:20:41.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.425562+0000", 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0951000000000001e-05 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.425603+0000", 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22408313899999999 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.659 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161593 4.a 4:56da7513:::benchmark_data_smithi169_60938_object161592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.213772+0000", 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.38498833400001, 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22234388299999999, 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161593 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213772+0000", 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213772+0000", 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213771+0000", 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9852999999999997e-05 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213840+0000", 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.242e-06 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213846+0000", 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.151e-06 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213849+0000", 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4224999999999998e-05 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213883+0000", 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0729999999999994e-05 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.213974+0000", 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9010000000000002e-05 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.214073+0000", 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049029989999999999 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.218976+0000", 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21703070999999999 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436006+0000", 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2487e-05 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436069+0000", 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6479e-05 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436115+0000", 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22234388299999999 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161603 4.a 4:52ec8e6e:::benchmark_data_smithi169_60938_object161602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.222039+0000", 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.37672108100003, 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.214235071, 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.665 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161603 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222039+0000", 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222039+0000", 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:20:41.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222037+0000", 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4573000000000002e-05 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222081+0000", 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5150000000000001e-05 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222097+0000", 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9699999999999996e-06 2024-02-12T16:20:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222102+0000", 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1875000000000001e-05 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222154+0000", 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014412 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222298+0000", 2024-02-12T16:20:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018792900000000001 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.222486+0000", 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015571471 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.238058+0000", 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.198109849 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436168+0000", 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5957e-05 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436224+0000", 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0220999999999999e-05 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436274+0000", 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.214235071 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161608 4.a 4:50c586f4:::benchmark_data_smithi169_60938_object161607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.225938+0000", 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.37282156700002, 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21071010000000001, 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161608 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.671 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.225938+0000", 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.225938+0000", 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.225936+0000", 2024-02-12T16:20:41.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2527000000000006e-05 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.225999+0000", 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5860000000000001e-05 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.226014+0000", 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8329999999999996e-06 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.226021+0000", 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077013299999999999 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.226791+0000", 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1599000000000003e-05 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.226873+0000", 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013929200000000001 2024-02-12T16:20:41.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.227012+0000", 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011053593 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.238066+0000", 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.198478765 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436545+0000", 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7926000000000002e-05 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436593+0000", 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5883e-05 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436648+0000", 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21071010000000001 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161609 4.a 4:5e9e66e1:::benchmark_data_smithi169_60938_object161608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.676 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.228820+0000", 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.36993955000003, 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20796085, 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161609 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228820+0000", 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228820+0000", 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228818+0000", 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9642000000000003e-05 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228877+0000", 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4557e-05 2024-02-12T16:20:41.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228892+0000", 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9059999999999996e-06 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228899+0000", 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018768039999999999 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.230776+0000", 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015592899999999999 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.230932+0000", 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000196663 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.231128+0000", 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0069444019999999997 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.238073+0000", 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19862465400000001 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436697+0000", 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0404000000000001e-05 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436748+0000", 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3513000000000003e-05 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.436781+0000", 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20796085 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161610 4.a 4:5c628cce:::benchmark_data_smithi169_60938_object161609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.228983+0000", 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.36977722300003, 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22653116600000001, 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161610 2024-02-12T16:20:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228983+0000", 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228983+0000", 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.228982+0000", 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7576000000000003e-05 2024-02-12T16:20:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.229069+0000", 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.092e-05 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.229080+0000", 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8559999999999998e-06 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.229085+0000", 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055185110000000002 2024-02-12T16:20:41.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.234603+0000", 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012498000000000001 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.234728+0000", 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018683299999999999 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.234915+0000", 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032771179999999999 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.238192+0000", 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21726493399999999 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455457+0000", 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1114000000000001e-05 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455488+0000", 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5375e-05 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455514+0000", 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22653116600000001 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161625 4.a 4:54ad26f7:::benchmark_data_smithi169_60938_object161624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.245941+0000", 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.35281885500001, 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20972849900000001, 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.687 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161625 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.245941+0000", 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.245941+0000", 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2024-02-12T16:20:41.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.245937+0000", 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8463e-05 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.245986+0000", 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2798000000000001e-05 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.246009+0000", 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0895e-05 2024-02-12T16:20:41.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.246020+0000", 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8044999999999997e-05 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.246078+0000", 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014814400000000001 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.246226+0000", 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016131399999999999 2024-02-12T16:20:41.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.246387+0000", 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040120640000000001 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.250399+0000", 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.205168194 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455567+0000", 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5188000000000001e-05 2024-02-12T16:20:41.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455622+0000", 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7101000000000002e-05 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455670+0000", 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20972849900000001 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.692 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:161634 4.a 4:53cb7377:::benchmark_data_smithi169_60938_object161633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e364)", 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:18.256142+0000", 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 323.34261775200002, 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.199686844, 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161634 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256142+0000", 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256142+0000", 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256140+0000", 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013021800000000001 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256270+0000", 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5727999999999999e-05 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256286+0000", 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8109999999999992e-06 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256294+0000", 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1804999999999995e-05 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256356+0000", 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015846 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256515+0000", 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019817700000000001 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.256713+0000", 2024-02-12T16:20:41.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041612760000000002 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.260874+0000", 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19487117200000001 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455745+0000", 2024-02-12T16:20:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6748000000000001e-05 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455782+0000", 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6986e-05 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:18.455829+0000", 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.199686844 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:207911 4.5 4:a47a7480:::benchmark_data_smithi169_60938_object34574:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:00.203356+0000", 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 281.39540378599997, 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13542780400000001, 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207911 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203356+0000", 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203356+0000", 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203353+0000", 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7749999999999998e-06 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203362+0000", 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5296000000000001e-05 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203377+0000", 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2480000000000001e-06 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203384+0000", 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6693999999999997e-05 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203430+0000", 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6540999999999996e-05 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203527+0000", 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7645999999999994e-05 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.203614+0000", 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008788232 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.212403+0000", 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12619939999999999 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.338602+0000", 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9683000000000001e-05 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.338702+0000", 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2138000000000005e-05 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.338784+0000", 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13542780400000001 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:207916 4.5 4:a7a2a20e:::benchmark_data_smithi169_60938_object34579:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:00.206535+0000", 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 281.39222512600003, 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.132420117, 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.706 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207916 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.206535+0000", 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.206535+0000", 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:20:41.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.206532+0000", 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0189999999999995e-06 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.206541+0000", 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5871000000000001e-05 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.206557+0000", 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5089999999999996e-06 2024-02-12T16:20:41.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.206564+0000", 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010441770000000001 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.207609+0000", 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014610400000000001 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.207755+0000", 2024-02-12T16:20:41.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016096200000000001 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.207916+0000", 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046469069999999996 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.212563+0000", 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12627939199999999 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.338842+0000", 2024-02-12T16:20:41.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0649000000000002e-05 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.338903+0000", 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2240000000000001e-05 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:00.338955+0000", 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.132420117 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:253521 4.a 4:5b299f8f:::benchmark_data_smithi169_60938_object80184:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e394)", 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:43.451157+0000", 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 238.14760248600001, 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.17873059499999999, 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 253521 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451157+0000", 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451157+0000", 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999943 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451151+0000", 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0443000000000001e-05 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451172+0000", 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9879999999999999e-05 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451192+0000", 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3239e-05 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451205+0000", 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4799000000000002e-05 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451260+0000", 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.081e-06 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451269+0000", 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.16988471899999999 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.621154+0000", 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000167214 2024-02-12T16:20:41.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.621321+0000", 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021294099999999999 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,10", 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.621534+0000", 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082046170000000009 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.629738+0000", 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3318000000000002e-05 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.629772+0000", 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6699999999999994e-06 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.629779+0000", 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6477000000000003e-05 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.629816+0000", 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2119999999999994e-05 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.629888+0000", 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.17873059499999999 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:253522 4.a 4:559fffa2:::benchmark_data_smithi169_60938_object80185:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e394)", 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:43.451220+0000", 2024-02-12T16:20:41.718 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 238.14753950100001, 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.18395086499999999, 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 253522 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451220+0000", 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451220+0000", 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451218+0000", 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.96e-06 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451226+0000", 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6959999999999994e-06 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451235+0000", 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6529999999999997e-06 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451241+0000", 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3207000000000001e-05 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451295+0000", 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0674e-05 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.451305+0000", 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.17485284600000001 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.626158+0000", 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011791800000000001 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.626276+0000", 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7794000000000006e-05 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,10", 2024-02-12T16:20:41.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.626374+0000", 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038449560000000001 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.630219+0000", 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2038999999999999e-05 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.630231+0000", 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046925339999999999 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:20:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.634923+0000", 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9188000000000001e-05 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.635003+0000", 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168686 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:43.635171+0000", 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.18395086499999999 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.724 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:20:41.725 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:20:41.725 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:20:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:41.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:41.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:41.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:41.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:41.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:41.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:42.008 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:42.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:42.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:42.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:42.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:42.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:42.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:20:42.279 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:42.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:42.382 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:42.416 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:42.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:42.519 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:42.555 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:42.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:42.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:20:42.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:42.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:42.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:42.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:42.857 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:42.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:42.938 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:20:42.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:43.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:43.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:43.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:42 smithi169 bash[19604]: cluster 2024-02-12T16:20:41.833814+0000 mgr.y (mgr.14146) 1063 : cluster [DBG] pgmap v1439: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:43.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:42 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.362530+0000 osd.7 (osd.7) 105 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:20:43.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:42 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.391401+0000 osd.7 (osd.7) 106 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:20:43.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:42 smithi169 bash[19604]: audit 2024-02-12T16:20:42.596262+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? v1:172.21.15.121:0/1204791276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:20:43.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:42 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.665702+0000 osd.10 (osd.10) 76 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:20:43.115 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:42 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.686484+0000 osd.10 (osd.10) 77 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:20:43.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:43.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:43.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:43.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:42 smithi123 bash[19119]: cluster 2024-02-12T16:20:41.833814+0000 mgr.y (mgr.14146) 1063 : cluster [DBG] pgmap v1439: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:42 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.362530+0000 osd.7 (osd.7) 105 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:20:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:42 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.391401+0000 osd.7 (osd.7) 106 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:20:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:42 smithi123 bash[19119]: audit 2024-02-12T16:20:42.596262+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? v1:172.21.15.121:0/1204791276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:20:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:42 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.665702+0000 osd.10 (osd.10) 76 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:20:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:42 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.686484+0000 osd.10 (osd.10) 77 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:20:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:42 smithi121 bash[18489]: cluster 2024-02-12T16:20:41.833814+0000 mgr.y (mgr.14146) 1063 : cluster [DBG] pgmap v1439: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:42 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.362530+0000 osd.7 (osd.7) 105 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:20:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:42 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.391401+0000 osd.7 (osd.7) 106 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:20:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:42 smithi121 bash[18489]: audit 2024-02-12T16:20:42.596262+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? v1:172.21.15.121:0/1204791276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:20:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:42 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.665702+0000 osd.10 (osd.10) 76 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:20:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:42 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.686484+0000 osd.10 (osd.10) 77 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:20:43.416 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:43.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:43.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:43.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:43.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:43.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:43.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:43.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:43.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:43.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:43 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.105702+0000 osd.9 (osd.9) 71 : cluster [DBG] 1.a scrub starts 2024-02-12T16:20:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:43 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.121357+0000 osd.9 (osd.9) 72 : cluster [DBG] 1.a scrub ok 2024-02-12T16:20:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:43 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.910351+0000 mon.a (mon.0) 1744 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:20:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:43 smithi169 bash[19604]: audit 2024-02-12T16:20:42.928584+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? v1:172.21.15.121:0/1204791276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:20:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:43 smithi169 bash[19604]: cluster 2024-02-12T16:20:42.928827+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e478: 12 total, 11 up, 12 in 2024-02-12T16:20:44.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:44.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:44.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:43 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.105702+0000 osd.9 (osd.9) 71 : cluster [DBG] 1.a scrub starts 2024-02-12T16:20:44.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:43 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.121357+0000 osd.9 (osd.9) 72 : cluster [DBG] 1.a scrub ok 2024-02-12T16:20:44.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:43 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.910351+0000 mon.a (mon.0) 1744 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:20:44.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:43 smithi123 bash[19119]: audit 2024-02-12T16:20:42.928584+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? v1:172.21.15.121:0/1204791276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:20:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:43 smithi123 bash[19119]: cluster 2024-02-12T16:20:42.928827+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e478: 12 total, 11 up, 12 in 2024-02-12T16:20:44.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:44.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:44.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:44.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:43 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.105702+0000 osd.9 (osd.9) 71 : cluster [DBG] 1.a scrub starts 2024-02-12T16:20:44.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:43 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.121357+0000 osd.9 (osd.9) 72 : cluster [DBG] 1.a scrub ok 2024-02-12T16:20:44.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:43 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.910351+0000 mon.a (mon.0) 1744 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:20:44.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:43 smithi121 bash[18489]: audit 2024-02-12T16:20:42.928584+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? v1:172.21.15.121:0/1204791276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:20:44.292 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:43 smithi121 bash[18489]: cluster 2024-02-12T16:20:42.928827+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e478: 12 total, 11 up, 12 in 2024-02-12T16:20:44.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:44.387 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:44.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:44.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:44.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:44.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:44.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:44.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:44.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:44.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:44.765 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:44.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:44.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:44.889 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:44.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:44.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:45.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:44 smithi169 bash[19604]: cluster 2024-02-12T16:20:43.259430+0000 osd.11 (osd.11) 93 : cluster [DBG] 1.c scrub starts 2024-02-12T16:20:45.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:44 smithi169 bash[19604]: cluster 2024-02-12T16:20:43.335575+0000 osd.11 (osd.11) 94 : cluster [DBG] 1.c scrub ok 2024-02-12T16:20:45.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:44 smithi169 bash[19604]: cluster 2024-02-12T16:20:43.834796+0000 mgr.y (mgr.14146) 1064 : cluster [DBG] pgmap v1441: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:45.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:44 smithi169 bash[19604]: cluster 2024-02-12T16:20:44.170115+0000 osd.11 (osd.11) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:45.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:44 smithi169 bash[19604]: cluster 2024-02-12T16:20:44.170212+0000 osd.11 (osd.11) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:45.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:45.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:45.147 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:45.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:45.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:45.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:44 smithi121 bash[18489]: cluster 2024-02-12T16:20:43.259430+0000 osd.11 (osd.11) 93 : cluster [DBG] 1.c scrub starts 2024-02-12T16:20:45.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:44 smithi121 bash[18489]: cluster 2024-02-12T16:20:43.335575+0000 osd.11 (osd.11) 94 : cluster [DBG] 1.c scrub ok 2024-02-12T16:20:45.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:44 smithi121 bash[18489]: cluster 2024-02-12T16:20:43.834796+0000 mgr.y (mgr.14146) 1064 : cluster [DBG] pgmap v1441: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:45.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:44 smithi121 bash[18489]: cluster 2024-02-12T16:20:44.170115+0000 osd.11 (osd.11) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:45.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:44 smithi121 bash[18489]: cluster 2024-02-12T16:20:44.170212+0000 osd.11 (osd.11) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:20:45.359 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:20:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 479, 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-02-12T15:47:18.273813+0000", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-02-12T16:20:44.924036+0000", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "last_up_change": "2024-02-12T16:17:42.493645+0000", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "last_in_change": "2024-02-12T16:19:27.892497+0000", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_num": 5799936, 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_set": [ 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "pglog_hardlimit", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "purged_snapdirs", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_deletes", 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "sortbitwise" 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:20:45.360 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_version": 28, 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_snapshot": "", 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_max": 4, 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd": 12, 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "min_compat_client": "jewel", 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "require_osd_release": "pacific", 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "pools": [ 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 1, 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T15:51:35.886878+0000", 2024-02-12T16:20:45.361 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 21, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 11, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 21, 2024-02-12T16:20:45.362 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 21, 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 21, 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "410", 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "410", 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:20:45.363 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:20:45.364 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:20:45.365 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "options": { 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_max": 32, 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_min": 1 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_devicehealth": {} 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 4, 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "unique_pool_2", 2024-02-12T16:20:45.366 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T16:13:44.637807+0000", 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 16, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 16, 2024-02-12T16:20:45.367 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 16, 2024-02-12T16:20:45.368 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 16, 2024-02-12T16:20:45.368 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 16, 2024-02-12T16:20:45.368 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "346", 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:20:45.369 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:20:45.370 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:20:45.371 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "options": {}, 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "rados": {} 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0", 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.372 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 8, 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 334, 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6802", 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6804", 2024-02-12T16:20:45.374 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6803", 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6801/1815232143", 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6802/1815232143", 2024-02-12T16:20:45.375 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6804/1815232143", 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6803/1815232143", 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "c6140309-5703-4538-9739-276fbbdaa880", 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.1039276123046875, 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 0.9075927734375, 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.376 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 13, 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6805", 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.377 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6806", 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6808", 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.378 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6807", 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6805/2968433670", 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6806/2968433670", 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6808/2968433670", 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6807/2968433670", 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170", 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 21, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:20:45.380 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6809", 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.381 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6810", 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6812", 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.382 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6811", 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6809/2158187852", 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6810/2158187852", 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6812/2158187852", 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6811/2158187852", 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.383 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd", 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 0, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 26, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 410, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 415, 2024-02-12T16:20:45.384 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6813", 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.385 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6814", 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6816", 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6815", 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1415189366 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6813/1415189366", 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6814/1415189366", 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6816/1415189366", 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6815/1415189366", 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: "exists" 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31", 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 32, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.388 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6800", 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6801", 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6803", 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6802", 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:20:45.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6800/3447700123", 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6801/3447700123", 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6803/3447700123", 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6802/3447700123", 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602", 2024-02-12T16:20:45.391 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 39, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 368, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6804", 2024-02-12T16:20:45.392 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6805", 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6807", 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6806", 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.394 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6804/961317306", 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6805/961317306", 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6807/961317306", 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6806/961317306", 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0", 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.395 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 48, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 420, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6808", 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:20:45.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6809", 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.397 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6811", 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6810", 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6808/1377261784", 2024-02-12T16:20:45.398 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6809/1377261784", 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6811/1377261784", 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6810/1377261784", 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515", 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.399 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 56, 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 415, 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6812", 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.400 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6813", 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6815", 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:20:45.401 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6814", 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6812/1702613957", 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6813/1702613957", 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6815/1702613957", 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6814/1702613957", 2024-02-12T16:20:45.402 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75", 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 65, 2024-02-12T16:20:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 310, 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6800", 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.404 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6801", 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6803", 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.405 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6802", 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6800/2551075225", 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6801/2551075225", 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6803/2551075225", 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6802/2551075225", 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.406 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15", 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 74, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 418, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.407 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6804", 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6805", 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.408 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6807", 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6806", 2024-02-12T16:20:45.409 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6804/2339555071", 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6805/2339555071", 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6807/2339555071", 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6806/2339555071", 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:20:45.410 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "91f74dca-503a-45d1-9647-df4804d58515", 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 83, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 410, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.411 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6808", 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6809", 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.412 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6811", 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6810", 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.413 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6808/391306610", 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6809/391306610", 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6811/391306610", 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6810/391306610", 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf", 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:20:45.414 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 93, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 453, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6812", 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:20:45.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6813", 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6815", 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6814", 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6812/3696812305", 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6813/3696812305", 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3696812305", 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6814/3696812305", 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_xinfo": [ 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.418 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:20.521588+0000", 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:37.967244+0000", 2024-02-12T16:20:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:19:45.249366+0000", 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.420 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:16:56.622408+0000", 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 414 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:30.722460+0000", 2024-02-12T16:20:45.421 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:33.428291+0000", 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.422 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:38.695914+0000", 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:31.297999+0000", 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:44.573271+0000", 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.424 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:11.371088+0000", 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:10.770665+0000", 2024-02-12T16:20:45.425 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:20:44.170217+0000", 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap": [], 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap_items": [], 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_temp": [ 2024-02-12T16:20:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "4.4", 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: 1 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_temp": [], 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklist": { 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/2474849789": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/253689008": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/3531151725": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/1778611055": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/28718860": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/445290372": "2024-02-13T15:47:47.232761+0000" 2024-02-12T16:20:45.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "range_blocklist": {}, 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profiles": { 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "default": { 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-failure-domain": "osd", 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "k": "2", 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "m": "1", 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin": "jerasure", 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "technique": "reed_sol_van" 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps_queue": [], 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "new_removed_snaps": [], 2024-02-12T16:20:45.428 INFO:teuthology.orchestra.run.smithi121.stdout: "new_purged_snaps": [], 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_node_flags": {}, 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "device_class_flags": {}, 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": { 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:20:45.429 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:20:45.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:45.430 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:44 smithi123 bash[19119]: cluster 2024-02-12T16:20:43.259430+0000 osd.11 (osd.11) 93 : cluster [DBG] 1.c scrub starts 2024-02-12T16:20:45.430 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:44 smithi123 bash[19119]: cluster 2024-02-12T16:20:43.335575+0000 osd.11 (osd.11) 94 : cluster [DBG] 1.c scrub ok 2024-02-12T16:20:45.430 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:44 smithi123 bash[19119]: cluster 2024-02-12T16:20:43.834796+0000 mgr.y (mgr.14146) 1064 : cluster [DBG] pgmap v1441: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:45.430 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:44 smithi123 bash[19119]: cluster 2024-02-12T16:20:44.170115+0000 osd.11 (osd.11) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:45.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:44 smithi123 bash[19119]: cluster 2024-02-12T16:20:44.170212+0000 osd.11 (osd.11) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:45.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:45.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:45.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:45.723 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:45.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:45.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:45.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:45.973 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:45.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:46.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:46.084 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:45 smithi169 bash[19604]: cluster 2024-02-12T16:20:44.241970+0000 osd.11 (osd.11) 97 : cluster [DBG] 4.1 deep-scrub starts 2024-02-12T16:20:46.084 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:45 smithi169 bash[19604]: cluster 2024-02-12T16:20:44.573220+0000 osd.8 (osd.8) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:46.085 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:45 smithi169 bash[19604]: cluster 2024-02-12T16:20:44.573269+0000 osd.8 (osd.8) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:46.085 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:45 smithi169 bash[19604]: cluster 2024-02-12T16:20:44.937484+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e479: 12 total, 11 up, 12 in 2024-02-12T16:20:46.085 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:45 smithi169 bash[19604]: audit 2024-02-12T16:20:45.352131+0000 mon.a (mon.0) 1748 : audit [DBG] from='client.? v1:172.21.15.121:0/527251062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:20:46.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:46.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:46.122 INFO:tasks.thrashosds.thrasher:j is {'epoch': 479, 'fsid': 'eb2349b2-c9bd-11ee-95b9-87774f69a715', 'created': '2024-02-12T15:47:18.273813+0000', 'modified': '2024-02-12T16:20:44.924036+0000', 'last_up_change': '2024-02-12T16:17:42.493645+0000', 'last_in_change': '2024-02-12T16:19:27.892497+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 11, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '410', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '410', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-12T16:13:44.637807+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '346', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 1815232143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 1815232143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 1815232143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 1815232143}]}, 'public_addr': '172.21.15.121:6801/1815232143', 'cluster_addr': '172.21.15.121:6802/1815232143', 'heartbeat_back_addr': '172.21.15.121:6804/1815232143', 'heartbeat_front_addr': '172.21.15.121:6803/1815232143', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c6140309-5703-4538-9739-276fbbdaa880', 'up': 1, 'in': 1, 'weight': 0.1039276123046875, 'primary_affinity': 0.9075927734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2968433670}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2968433670}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 2968433670}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2968433670}]}, 'public_addr': '172.21.15.121:6805/2968433670', 'cluster_addr': '172.21.15.121:6806/2968433670', 'heartbeat_back_addr': '172.21.15.121:6808/2968433670', 'heartbeat_front_addr': '172.21.15.121:6807/2968433670', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 2158187852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 2158187852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 2158187852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 2158187852}]}, 'public_addr': '172.21.15.121:6809/2158187852', 'cluster_addr': '172.21.15.121:6810/2158187852', 'heartbeat_back_addr': '172.21.15.121:6812/2158187852', 'heartbeat_front_addr': '172.21.15.121:6811/2158187852', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5f824526-596d-42e1-8b7e-cd8676ded6dd', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 410, 'down_at': 415, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 1415189366}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 1415189366}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6816', 'nonce': 1415189366}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 1415189366}]}, 'public_addr': '172.21.15.121:6813/1415189366', 'cluster_addr': '172.21.15.121:6814/1415189366', 'heartbeat_back_addr': '172.21.15.121:6816/1415189366', 'heartbeat_front_addr': '172.21.15.121:6815/1415189366', 'state': ['exists']}, {'osd': 4, 'uuid': '14c2f878-99ce-4b51-8d9c-131c1d883b31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6800', 'nonce': 3447700123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6801', 'nonce': 3447700123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6803', 'nonce': 3447700123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6802', 'nonce': 3447700123}]}, 'public_addr': '172.21.15.123:6800/3447700123', 'cluster_addr': '172.21.15.123:6801/3447700123', 'heartbeat_back_addr': '172.21.15.123:6803/3447700123', 'heartbeat_front_addr': '172.21.15.123:6802/3447700123', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6804', 'nonce': 961317306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6805', 'nonce': 961317306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6807', 'nonce': 961317306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6806', 'nonce': 961317306}]}, 'public_addr': '172.21.15.123:6804/961317306', 'cluster_addr': '172.21.15.123:6805/961317306', 'heartbeat_back_addr': '172.21.15.123:6807/961317306', 'heartbeat_front_addr': '172.21.15.123:6806/961317306', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '71e715e6-eac1-4386-927e-51df0cc419c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6808', 'nonce': 1377261784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6809', 'nonce': 1377261784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6811', 'nonce': 1377261784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6810', 'nonce': 1377261784}]}, 'public_addr': '172.21.15.123:6808/1377261784', 'cluster_addr': '172.21.15.123:6809/1377261784', 'heartbeat_back_addr': '172.21.15.123:6811/1377261784', 'heartbeat_front_addr': '172.21.15.123:6810/1377261784', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b419b04f-0c0c-495a-aa9b-25c334159515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6812', 'nonce': 1702613957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6813', 'nonce': 1702613957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6815', 'nonce': 1702613957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6814', 'nonce': 1702613957}]}, 'public_addr': '172.21.15.123:6812/1702613957', 'cluster_addr': '172.21.15.123:6813/1702613957', 'heartbeat_back_addr': '172.21.15.123:6815/1702613957', 'heartbeat_front_addr': '172.21.15.123:6814/1702613957', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1591e7bc-5b86-4401-803d-5325062ffb75', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 310, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6800', 'nonce': 2551075225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 2551075225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 2551075225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6802', 'nonce': 2551075225}]}, 'public_addr': '172.21.15.169:6800/2551075225', 'cluster_addr': '172.21.15.169:6801/2551075225', 'heartbeat_back_addr': '172.21.15.169:6803/2551075225', 'heartbeat_front_addr': '172.21.15.169:6802/2551075225', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '522d61df-8dc1-4915-83fd-f94d0d572c15', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6804', 'nonce': 2339555071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 2339555071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 2339555071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6806', 'nonce': 2339555071}]}, 'public_addr': '172.21.15.169:6804/2339555071', 'cluster_addr': '172.21.15.169:6805/2339555071', 'heartbeat_back_addr': '172.21.15.169:6807/2339555071', 'heartbeat_front_addr': '172.21.15.169:6806/2339555071', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '91f74dca-503a-45d1-9647-df4804d58515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 410, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6808', 'nonce': 391306610}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 391306610}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 391306610}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6810', 'nonce': 391306610}]}, 'public_addr': '172.21.15.169:6808/391306610', 'cluster_addr': '172.21.15.169:6809/391306610', 'heartbeat_back_addr': '172.21.15.169:6811/391306610', 'heartbeat_front_addr': '172.21.15.169:6810/391306610', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '788b4f47-eadf-4601-a5b4-51fb6f04fbdf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 93, 'up_thru': 453, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6812', 'nonce': 3696812305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3696812305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3696812305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6814', 'nonce': 3696812305}]}, 'public_addr': '172.21.15.169:6812/3696812305', 'cluster_addr': '172.21.15.169:6813/3696812305', 'heartbeat_back_addr': '172.21.15.169:6815/3696812305', 'heartbeat_front_addr': '172.21.15.169:6814/3696812305', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:20.521588+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:37.967244+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:19:45.249366+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:16:56.622408+0000', 'dead_epoch': 414}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:30.722460+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:33.428291+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:38.695914+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:31.297999+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:44.573271+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:11.371088+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:10.770665+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:20:44.170217+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '4.4', 'osds': [2, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.121:0/2474849789': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/253689008': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:6800/3531151725': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:0/1778611055': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:6800/28718860': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/445290372': '2024-02-13T15:47:47.232761+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:20:46.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:20:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:45 smithi123 bash[19119]: cluster 2024-02-12T16:20:44.241970+0000 osd.11 (osd.11) 97 : cluster [DBG] 4.1 deep-scrub starts 2024-02-12T16:20:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:45 smithi123 bash[19119]: cluster 2024-02-12T16:20:44.573220+0000 osd.8 (osd.8) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:45 smithi123 bash[19119]: cluster 2024-02-12T16:20:44.573269+0000 osd.8 (osd.8) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:45 smithi123 bash[19119]: cluster 2024-02-12T16:20:44.937484+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e479: 12 total, 11 up, 12 in 2024-02-12T16:20:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:45 smithi123 bash[19119]: audit 2024-02-12T16:20:45.352131+0000 mon.a (mon.0) 1748 : audit [DBG] from='client.? v1:172.21.15.121:0/527251062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:20:46.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:46.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:46.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:46.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:20:46.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:45 smithi121 bash[18489]: cluster 2024-02-12T16:20:44.241970+0000 osd.11 (osd.11) 97 : cluster [DBG] 4.1 deep-scrub starts 2024-02-12T16:20:46.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:45 smithi121 bash[18489]: cluster 2024-02-12T16:20:44.573220+0000 osd.8 (osd.8) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:20:46.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:45 smithi121 bash[18489]: cluster 2024-02-12T16:20:44.573269+0000 osd.8 (osd.8) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:20:46.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:45 smithi121 bash[18489]: cluster 2024-02-12T16:20:44.937484+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e479: 12 total, 11 up, 12 in 2024-02-12T16:20:46.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:45 smithi121 bash[18489]: audit 2024-02-12T16:20:45.352131+0000 mon.a (mon.0) 1748 : audit [DBG] from='client.? v1:172.21.15.121:0/527251062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:20:46.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:46.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:46.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:46.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:46.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:46.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:46.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:46.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:46.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:46.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:46.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:46.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:46.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:46.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:47.017 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:47.038 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:47.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:47.128 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:46 smithi169 bash[19604]: cluster 2024-02-12T16:20:45.835837+0000 mgr.y (mgr.14146) 1065 : cluster [DBG] pgmap v1443: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:47.128 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:46 smithi169 bash[19604]: cluster 2024-02-12T16:20:46.632699+0000 osd.10 (osd.10) 78 : cluster [DBG] 1.e scrub starts 2024-02-12T16:20:47.128 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:46 smithi169 bash[19604]: cluster 2024-02-12T16:20:46.640800+0000 osd.10 (osd.10) 79 : cluster [DBG] 1.e scrub ok 2024-02-12T16:20:47.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:47.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:47.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:47.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:46 smithi123 bash[19119]: cluster 2024-02-12T16:20:45.835837+0000 mgr.y (mgr.14146) 1065 : cluster [DBG] pgmap v1443: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:46 smithi123 bash[19119]: cluster 2024-02-12T16:20:46.632699+0000 osd.10 (osd.10) 78 : cluster [DBG] 1.e scrub starts 2024-02-12T16:20:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:46 smithi123 bash[19119]: cluster 2024-02-12T16:20:46.640800+0000 osd.10 (osd.10) 79 : cluster [DBG] 1.e scrub ok 2024-02-12T16:20:47.400 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:46 smithi121 bash[18489]: cluster 2024-02-12T16:20:45.835837+0000 mgr.y (mgr.14146) 1065 : cluster [DBG] pgmap v1443: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:46 smithi121 bash[18489]: cluster 2024-02-12T16:20:46.632699+0000 osd.10 (osd.10) 78 : cluster [DBG] 1.e scrub starts 2024-02-12T16:20:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:46 smithi121 bash[18489]: cluster 2024-02-12T16:20:46.640800+0000 osd.10 (osd.10) 79 : cluster [DBG] 1.e scrub ok 2024-02-12T16:20:47.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:47.522 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:47.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:47.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:47.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:47.779 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:47.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:47.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:47.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:48.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:48.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:48.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:48.192 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:48.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:48.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:48.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:48.451 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:48.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:48.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:48.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:48.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:48.845 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:48.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:48.974 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:48.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:49.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:49.089 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:48 smithi123 bash[19119]: cluster 2024-02-12T16:20:47.836873+0000 mgr.y (mgr.14146) 1066 : cluster [DBG] pgmap v1444: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:49.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:49.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:49.217 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:48 smithi169 bash[19604]: cluster 2024-02-12T16:20:47.836873+0000 mgr.y (mgr.14146) 1066 : cluster [DBG] pgmap v1444: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:49.234 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:49.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:49.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:49.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:49.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:48 smithi121 bash[18489]: cluster 2024-02-12T16:20:47.836873+0000 mgr.y (mgr.14146) 1066 : cluster [DBG] pgmap v1444: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:49.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:49.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:49.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:49.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:49.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:49.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:49.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:49.759 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:49.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:49.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:49.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:49.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:49.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:50.017 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:50.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:50.144 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:50.234 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:20:50.234 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:20:50.234 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:20:50.234 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:20:50.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:50.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:50.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:50.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.681 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:50.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:50.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:50.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:20:50.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:50.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:51.060 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:51.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:50 smithi123 bash[19119]: cluster 2024-02-12T16:20:49.837959+0000 mgr.y (mgr.14146) 1067 : cluster [DBG] pgmap v1445: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:51.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:50 smithi123 bash[19119]: cluster 2024-02-12T16:20:50.619237+0000 osd.10 (osd.10) 80 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:20:51.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:50 smithi123 bash[19119]: cluster 2024-02-12T16:20:50.627076+0000 osd.10 (osd.10) 81 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:20:51.085 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:51.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:51.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:51.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:50 smithi121 bash[18489]: cluster 2024-02-12T16:20:49.837959+0000 mgr.y (mgr.14146) 1067 : cluster [DBG] pgmap v1445: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:51.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:50 smithi121 bash[18489]: cluster 2024-02-12T16:20:50.619237+0000 osd.10 (osd.10) 80 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:20:51.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:50 smithi121 bash[18489]: cluster 2024-02-12T16:20:50.627076+0000 osd.10 (osd.10) 81 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:20:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:51.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:51.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:50 smithi169 bash[19604]: cluster 2024-02-12T16:20:49.837959+0000 mgr.y (mgr.14146) 1067 : cluster [DBG] pgmap v1445: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:51.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:50 smithi169 bash[19604]: cluster 2024-02-12T16:20:50.619237+0000 osd.10 (osd.10) 80 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:20:51.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:50 smithi169 bash[19604]: cluster 2024-02-12T16:20:50.627076+0000 osd.10 (osd.10) 81 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:20:51.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:51.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:51.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:51.444 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:51.444 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:20:51.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:51.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:51.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:51.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:51.640 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:51.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:51.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:51.779 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:51.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:51.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:51.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:51.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:52.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:52.062 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:52.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:52.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:52.204 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:52.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:52.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:52.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:52.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:52.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:52.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:52.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:52.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:52.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:52.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:52.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:52.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:52.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:52.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:52.897 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:52.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:53.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:53.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:53.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:53.032 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:20:53.050 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:20:53.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:20:53.072 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: cluster 2024-02-12T16:20:51.243481+0000 osd.11 (osd.11) 98 : cluster [DBG] 4.1 deep-scrub ok 2024-02-12T16:20:53.072 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: cluster 2024-02-12T16:20:51.297385+0000 osd.6 (osd.6) 80 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:20:53.072 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: cluster 2024-02-12T16:20:51.298955+0000 osd.5 (osd.5) 67 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:20:53.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: cluster 2024-02-12T16:20:51.342958+0000 osd.5 (osd.5) 68 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:20:53.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: cluster 2024-02-12T16:20:51.676714+0000 osd.6 (osd.6) 81 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:20:53.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: cluster 2024-02-12T16:20:51.839053+0000 mgr.y (mgr.14146) 1068 : cluster [DBG] pgmap v1446: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:53.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: audit 2024-02-12T16:20:52.354959+0000 mon.a (mon.0) 1749 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:20:53.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:52 smithi121 bash[18489]: audit 2024-02-12T16:20:52.666099+0000 mon.a (mon.0) 1750 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:20:53.099 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:20:53.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:20:53.131 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:53.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: cluster 2024-02-12T16:20:51.243481+0000 osd.11 (osd.11) 98 : cluster [DBG] 4.1 deep-scrub ok 2024-02-12T16:20:53.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: cluster 2024-02-12T16:20:51.297385+0000 osd.6 (osd.6) 80 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:20:53.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: cluster 2024-02-12T16:20:51.298955+0000 osd.5 (osd.5) 67 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:20:53.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: cluster 2024-02-12T16:20:51.342958+0000 osd.5 (osd.5) 68 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:20:53.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: cluster 2024-02-12T16:20:51.676714+0000 osd.6 (osd.6) 81 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:20:53.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: cluster 2024-02-12T16:20:51.839053+0000 mgr.y (mgr.14146) 1068 : cluster [DBG] pgmap v1446: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:53.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: audit 2024-02-12T16:20:52.354959+0000 mon.a (mon.0) 1749 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:20:53.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:52 smithi123 bash[19119]: audit 2024-02-12T16:20:52.666099+0000 mon.a (mon.0) 1750 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:20:53.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:20:53.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:53.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:53.176 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:20:53.200 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:20:53.228 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:20:53.253 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:20:53.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:53.278 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.279 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: cluster 2024-02-12T16:20:51.243481+0000 osd.11 (osd.11) 98 : cluster [DBG] 4.1 deep-scrub ok 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: cluster 2024-02-12T16:20:51.297385+0000 osd.6 (osd.6) 80 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: cluster 2024-02-12T16:20:51.298955+0000 osd.5 (osd.5) 67 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: cluster 2024-02-12T16:20:51.342958+0000 osd.5 (osd.5) 68 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: cluster 2024-02-12T16:20:51.676714+0000 osd.6 (osd.6) 81 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: cluster 2024-02-12T16:20:51.839053+0000 mgr.y (mgr.14146) 1068 : cluster [DBG] pgmap v1446: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: audit 2024-02-12T16:20:52.354959+0000 mon.a (mon.0) 1749 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:20:53.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:52 smithi169 bash[19604]: audit 2024-02-12T16:20:52.666099+0000 mon.a (mon.0) 1750 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:20:53.314 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:53.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:53.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:53.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:53.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:53.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:53.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:53.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:53.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:53.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:53.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:53.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:53.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:53.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:53.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:54.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:54.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:54.148 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:54.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: cluster 2024-02-12T16:20:52.339084+0000 osd.4 (osd.4) 69 : cluster [DBG] 4.e scrub starts 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: cluster 2024-02-12T16:20:52.788294+0000 osd.4 (osd.4) 70 : cluster [DBG] 4.e scrub ok 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: audit 2024-02-12T16:20:52.954181+0000 mon.a (mon.0) 1751 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: audit 2024-02-12T16:20:52.955388+0000 mon.a (mon.0) 1752 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: audit 2024-02-12T16:20:52.962894+0000 mon.a (mon.0) 1753 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: audit 2024-02-12T16:20:53.000700+0000 mon.a (mon.0) 1754 : audit [DBG] from='client.? v1:172.21.15.121:0/2161229762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: cluster 2024-02-12T16:20:53.584454+0000 osd.10 (osd.10) 82 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:20:54.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:53 smithi121 bash[18489]: cluster 2024-02-12T16:20:53.615301+0000 osd.10 (osd.10) 83 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:20:54.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:54.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: cluster 2024-02-12T16:20:52.339084+0000 osd.4 (osd.4) 69 : cluster [DBG] 4.e scrub starts 2024-02-12T16:20:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: cluster 2024-02-12T16:20:52.788294+0000 osd.4 (osd.4) 70 : cluster [DBG] 4.e scrub ok 2024-02-12T16:20:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: audit 2024-02-12T16:20:52.954181+0000 mon.a (mon.0) 1751 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:20:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: audit 2024-02-12T16:20:52.955388+0000 mon.a (mon.0) 1752 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:20:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: audit 2024-02-12T16:20:52.962894+0000 mon.a (mon.0) 1753 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:20:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: audit 2024-02-12T16:20:53.000700+0000 mon.a (mon.0) 1754 : audit [DBG] from='client.? v1:172.21.15.121:0/2161229762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: cluster 2024-02-12T16:20:53.584454+0000 osd.10 (osd.10) 82 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:20:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:53 smithi169 bash[19604]: cluster 2024-02-12T16:20:53.615301+0000 osd.10 (osd.10) 83 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:20:54.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: cluster 2024-02-12T16:20:52.339084+0000 osd.4 (osd.4) 69 : cluster [DBG] 4.e scrub starts 2024-02-12T16:20:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: cluster 2024-02-12T16:20:52.788294+0000 osd.4 (osd.4) 70 : cluster [DBG] 4.e scrub ok 2024-02-12T16:20:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: audit 2024-02-12T16:20:52.954181+0000 mon.a (mon.0) 1751 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:20:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: audit 2024-02-12T16:20:52.955388+0000 mon.a (mon.0) 1752 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:20:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: audit 2024-02-12T16:20:52.962894+0000 mon.a (mon.0) 1753 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:20:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: audit 2024-02-12T16:20:53.000700+0000 mon.a (mon.0) 1754 : audit [DBG] from='client.? v1:172.21.15.121:0/2161229762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:20:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: cluster 2024-02-12T16:20:53.584454+0000 osd.10 (osd.10) 82 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:20:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:53 smithi123 bash[19119]: cluster 2024-02-12T16:20:53.615301+0000 osd.10 (osd.10) 83 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:20:54.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:54.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:54.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:54.565 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:20:54.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:20:54.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:54.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:54.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:54.848 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:54.952 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:54.988 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:54.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:54.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:55.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:55.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:54 smithi123 bash[19119]: cluster 2024-02-12T16:20:53.840171+0000 mgr.y (mgr.14146) 1069 : cluster [DBG] pgmap v1447: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:55.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:54 smithi123 bash[19119]: cluster 2024-02-12T16:20:53.968793+0000 osd.9 (osd.9) 73 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:20:55.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:54 smithi123 bash[19119]: cluster 2024-02-12T16:20:54.133994+0000 osd.11 (osd.11) 99 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:20:55.128 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:55.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:55.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:55.245 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:54 smithi169 bash[19604]: cluster 2024-02-12T16:20:53.840171+0000 mgr.y (mgr.14146) 1069 : cluster [DBG] pgmap v1447: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:55.245 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:54 smithi169 bash[19604]: cluster 2024-02-12T16:20:53.968793+0000 osd.9 (osd.9) 73 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:20:55.245 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:54 smithi169 bash[19604]: cluster 2024-02-12T16:20:54.133994+0000 osd.11 (osd.11) 99 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:20:55.268 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:55.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:55.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:55.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:55.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:54 smithi121 bash[18489]: cluster 2024-02-12T16:20:53.840171+0000 mgr.y (mgr.14146) 1069 : cluster [DBG] pgmap v1447: 37 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 4 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:54 smithi121 bash[18489]: cluster 2024-02-12T16:20:53.968793+0000 osd.9 (osd.9) 73 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:20:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:54 smithi121 bash[18489]: cluster 2024-02-12T16:20:54.133994+0000 osd.11 (osd.11) 99 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:20:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:55.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:55.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:55.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:55.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:55.642 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:20:55.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:55.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:55.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:55.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:55.832 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:55.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:55.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:55.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:55.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:56.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:56.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:55 smithi123 bash[19119]: cluster 2024-02-12T16:20:54.207879+0000 osd.11 (osd.11) 100 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:20:56.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:55 smithi123 bash[19119]: cluster 2024-02-12T16:20:54.399629+0000 osd.9 (osd.9) 74 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:20:56.115 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:56.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:56.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:56.256 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:56.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:56.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:55 smithi169 bash[19604]: cluster 2024-02-12T16:20:54.207879+0000 osd.11 (osd.11) 100 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:20:56.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:55 smithi169 bash[19604]: cluster 2024-02-12T16:20:54.399629+0000 osd.9 (osd.9) 74 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:20:56.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:56.392 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:55 smithi121 bash[18489]: cluster 2024-02-12T16:20:54.207879+0000 osd.11 (osd.11) 100 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:20:56.392 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:55 smithi121 bash[18489]: cluster 2024-02-12T16:20:54.399629+0000 osd.9 (osd.9) 74 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:20:56.397 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:56.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:56.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:56.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:20:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:56.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:20:56.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:20:56.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:56.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:56.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:56.806 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:56.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:56.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:56.914 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:20:56.916 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:20:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:56.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:57.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:57.093 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:57.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:57.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:57.207 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:56 smithi123 bash[19119]: cluster 2024-02-12T16:20:55.841245+0000 mgr.y (mgr.14146) 1070 : cluster [DBG] pgmap v1448: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:57.207 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:56 smithi123 bash[19119]: audit 2024-02-12T16:20:56.893929+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? v1:172.21.15.121:0/4236567214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:20:57.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:57.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:56 smithi169 bash[19604]: cluster 2024-02-12T16:20:55.841245+0000 mgr.y (mgr.14146) 1070 : cluster [DBG] pgmap v1448: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:56 smithi169 bash[19604]: audit 2024-02-12T16:20:56.893929+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? v1:172.21.15.121:0/4236567214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:20:57.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:57.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:57.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:56 smithi121 bash[18489]: cluster 2024-02-12T16:20:55.841245+0000 mgr.y (mgr.14146) 1070 : cluster [DBG] pgmap v1448: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:56 smithi121 bash[18489]: audit 2024-02-12T16:20:56.893929+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? v1:172.21.15.121:0/4236567214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:20:57.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:57.512 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:57.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:57.540 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:20:57.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:57.654 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:57.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:57.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:57.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:57.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:57.931 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:20:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:57.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:20:57.983 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1448,"stamp":"2024-02-12T16:20:55.840499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2294087704,"num_objects":35006,"num_object_clones":0,"num_object_copies":70012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8797,"num_objects_misplaced":2246,"num_objects_unfound":0,"num_objects_dirty":35006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":311664,"num_write_kb":11093377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17666,"num_bytes_recovered":1157627904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160937,"ondisk_log_size":160937,"up":66,"acting":67,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9118916,"kb_used_data":4914020,"kb_used_omap":0,"kb_used_meta":4204800,"kb_avail":1115822908,"statfs":{"total":1151940427776,"available":1142602657792,"internally_reserved":0,"allocated":5031956480,"data_stored":5023263711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4305715200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.907144"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":556,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-02-12T16:20:54.208020+0000","last_change":"2024-02-12T16:20:54.208020+0000","last_active":"2024-02-12T16:20:54.208020+0000","last_peered":"2024-02-12T16:20:54.208020+0000","last_clean":"2024-02-12T16:20:54.208020+0000","last_became_active":"2024-02-12T16:19:29.356492+0000","last_became_peered":"2024-02-12T16:19:29.356492+0000","last_unstale":"2024-02-12T16:20:54.208020+0000","last_undegraded":"2024-02-12T16:20:54.208020+0000","last_fullsized":"2024-02-12T16:20:54.208020+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:54.207884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:54.207884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":560,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041511+0000","last_change":"2024-02-12T16:20:38.481065+0000","last_active":"2024-02-12T16:20:43.041511+0000","last_peered":"2024-02-12T16:20:43.041511+0000","last_clean":"2024-02-12T16:20:43.041511+0000","last_became_active":"2024-02-12T16:17:37.450871+0000","last_became_peered":"2024-02-12T16:17:37.450871+0000","last_unstale":"2024-02-12T16:20:43.041511+0000","last_undegraded":"2024-02-12T16:20:43.041511+0000","last_fullsized":"2024-02-12T16:20:43.041511+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:38.480941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:38.480941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"416'19460","reported_seq":19643,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-02-12T16:20:51.676914+0000","last_change":"2024-02-12T16:20:51.676914+0000","last_active":"2024-02-12T16:20:51.676914+0000","last_peered":"2024-02-12T16:20:51.676914+0000","last_clean":"2024-02-12T16:20:51.676914+0000","last_became_active":"2024-02-12T16:13:45.969639+0000","last_became_peered":"2024-02-12T16:13:45.969639+0000","last_unstale":"2024-02-12T16:20:51.676914+0000","last_undegraded":"2024-02-12T16:20:51.676914+0000","last_fullsized":"2024-02-12T16:20:51.676914+0000","mapping_epoch":334,"log_start":"360'9400","ondisk_log_start":"360'9400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19460","last_scrub_stamp":"2024-02-12T16:20:51.676728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:20:51.676728+0000","log_size":10060,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":145489920,"num_objects":2220,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19460,"num_write_kb":693760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"414'19567","reported_seq":19724,"reported_epoch":478,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:20:43.041379+0000","last_change":"2024-02-12T16:17:43.520446+0000","last_active":"2024-02-12T16:17:42.522571+0000","last_peered":"2024-02-12T16:20:43.041379+0000","last_clean":"2024-02-12T16:17:42.468741+0000","last_became_active":"2024-02-12T16:13:45.788482+0000","last_became_peered":"2024-02-12T16:17:43.520446+0000","last_unstale":"2024-02-12T16:20:43.041379+0000","last_undegraded":"2024-02-12T16:17:43.518138+0000","last_fullsized":"2024-02-12T16:17:43.518031+0000","mapping_epoch":415,"log_start":"360'9500","ondisk_log_start":"360'9500","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'11421","last_scrub_stamp":"2024-02-12T16:15:36.163985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:36.163985+0000","log_size":10067,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":148307968,"num_objects":2263,"num_object_clones":0,"num_object_copies":4526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19567,"num_write_kb":698560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":2263}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":146,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.926554+0000","last_change":"2024-02-12T16:20:42.686622+0000","last_active":"2024-02-12T16:20:42.926554+0000","last_peered":"2024-02-12T16:20:42.926554+0000","last_clean":"2024-02-12T16:20:42.926554+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:20:42.926554+0000","last_undegraded":"2024-02-12T16:20:42.926554+0000","last_fullsized":"2024-02-12T16:20:42.926554+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:42.686491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:42.686491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1","version":"479'21492","reported_seq":21660,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-02-12T16:20:51.243559+0000","last_change":"2024-02-12T16:20:51.243559+0000","last_active":"2024-02-12T16:20:51.243559+0000","last_peered":"2024-02-12T16:20:51.243559+0000","last_clean":"2024-02-12T16:20:51.243559+0000","last_became_active":"2024-02-12T16:13:45.969700+0000","last_became_peered":"2024-02-12T16:13:45.969700+0000","last_unstale":"2024-02-12T16:20:51.243559+0000","last_undegraded":"2024-02-12T16:20:51.243559+0000","last_fullsized":"2024-02-12T16:20:51.243559+0000","mapping_epoch":334,"log_start":"374'11400","ondisk_log_start":"374'11400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21492","last_scrub_stamp":"2024-02-12T16:20:51.243488+0000","last_deep_scrub":"479'21492","last_deep_scrub_stamp":"2024-02-12T16:20:51.243488+0000","last_clean_scrub_stamp":"2024-02-12T16:20:51.243488+0000","log_size":10092,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142082048,"num_objects":2168,"num_object_clones":0,"num_object_copies":4336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19324,"num_write_kb":687744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":552,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.929145+0000","last_change":"2024-02-12T16:19:41.336558+0000","last_active":"2024-02-12T16:20:42.929145+0000","last_peered":"2024-02-12T16:20:42.929145+0000","last_clean":"2024-02-12T16:20:42.929145+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:20:42.929145+0000","last_undegraded":"2024-02-12T16:20:42.929145+0000","last_fullsized":"2024-02-12T16:20:42.929145+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:19:41.336446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:19:41.336446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"416'19181","reported_seq":19351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.040903+0000","last_change":"2024-02-12T16:19:44.463759+0000","last_active":"2024-02-12T16:20:43.040903+0000","last_peered":"2024-02-12T16:20:43.040903+0000","last_clean":"2024-02-12T16:20:43.040903+0000","last_became_active":"2024-02-12T16:13:45.969665+0000","last_became_peered":"2024-02-12T16:13:45.969665+0000","last_unstale":"2024-02-12T16:20:43.040903+0000","last_undegraded":"2024-02-12T16:20:43.040903+0000","last_fullsized":"2024-02-12T16:20:43.040903+0000","mapping_epoch":334,"log_start":"359'9100","ondisk_log_start":"359'9100","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19181","last_scrub_stamp":"2024-02-12T16:19:44.463591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:19:44.463591+0000","log_size":10081,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140050432,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19181,"num_write_kb":682176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":146,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041655+0000","last_change":"2024-02-12T16:20:35.238003+0000","last_active":"2024-02-12T16:20:43.041655+0000","last_peered":"2024-02-12T16:20:43.041655+0000","last_clean":"2024-02-12T16:20:43.041655+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:20:43.041655+0000","last_undegraded":"2024-02-12T16:20:43.041655+0000","last_fullsized":"2024-02-12T16:20:43.041655+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:35.237907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:35.237907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"416'19327","reported_seq":19495,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.929142+0000","last_change":"2024-02-12T16:20:34.698738+0000","last_active":"2024-02-12T16:20:42.929142+0000","last_peered":"2024-02-12T16:20:42.929142+0000","last_clean":"2024-02-12T16:20:42.929142+0000","last_became_active":"2024-02-12T16:13:45.794673+0000","last_became_peered":"2024-02-12T16:13:45.794673+0000","last_unstale":"2024-02-12T16:20:42.929142+0000","last_undegraded":"2024-02-12T16:20:42.929142+0000","last_fullsized":"2024-02-12T16:20:42.929142+0000","mapping_epoch":334,"log_start":"360'9300","ondisk_log_start":"360'9300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19327","last_scrub_stamp":"2024-02-12T16:20:34.698607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:20:34.698607+0000","log_size":10027,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142409728,"num_objects":2173,"num_object_clones":0,"num_object_copies":4346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19327,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":142,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-02-12T16:20:50.627208+0000","last_change":"2024-02-12T16:20:50.627208+0000","last_active":"2024-02-12T16:20:50.627208+0000","last_peered":"2024-02-12T16:20:50.627208+0000","last_clean":"2024-02-12T16:20:50.627208+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:20:50.627208+0000","last_undegraded":"2024-02-12T16:20:50.627208+0000","last_fullsized":"2024-02-12T16:20:50.627208+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:50.627086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:50.627086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"416'19636","reported_seq":21174,"reported_epoch":478,"state":"active+clean+remapped","last_fresh":"2024-02-12T16:20:42.924188+0000","last_change":"2024-02-12T16:19:40.530372+0000","last_active":"2024-02-12T16:20:42.924188+0000","last_peered":"2024-02-12T16:20:42.924188+0000","last_clean":"2024-02-12T16:20:42.924188+0000","last_became_active":"2024-02-12T16:17:43.511096+0000","last_became_peered":"2024-02-12T16:17:43.511096+0000","last_unstale":"2024-02-12T16:20:42.924188+0000","last_undegraded":"2024-02-12T16:20:42.924188+0000","last_fullsized":"2024-02-12T16:20:42.924188+0000","mapping_epoch":415,"log_start":"361'9600","ondisk_log_start":"361'9600","created":334,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19636","last_scrub_stamp":"2024-02-12T16:19:40.530134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:19:40.530134+0000","log_size":10036,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":147193856,"num_objects":2246,"num_object_clones":0,"num_object_copies":4492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2246,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19636,"num_write_kb":700224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":44498944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":2246}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":137,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.926290+0000","last_change":"2024-02-12T16:19:41.144652+0000","last_active":"2024-02-12T16:20:42.926290+0000","last_peered":"2024-02-12T16:20:42.926290+0000","last_clean":"2024-02-12T16:20:42.926290+0000","last_became_active":"2024-02-12T16:17:37.450884+0000","last_became_peered":"2024-02-12T16:17:37.450884+0000","last_unstale":"2024-02-12T16:20:42.926290+0000","last_undegraded":"2024-02-12T16:20:42.926290+0000","last_fullsized":"2024-02-12T16:20:42.926290+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:19:41.144504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:19:41.144504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"416'22178","reported_seq":26713,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.040808+0000","last_change":"2024-02-12T16:20:33.365956+0000","last_active":"2024-02-12T16:20:43.040808+0000","last_peered":"2024-02-12T16:20:43.040808+0000","last_clean":"2024-02-12T16:20:43.040808+0000","last_became_active":"2024-02-12T16:17:49.564271+0000","last_became_peered":"2024-02-12T16:17:49.564271+0000","last_unstale":"2024-02-12T16:20:43.040808+0000","last_undegraded":"2024-02-12T16:20:43.040808+0000","last_fullsized":"2024-02-12T16:20:43.040808+0000","mapping_epoch":418,"log_start":"375'12101","ondisk_log_start":"375'12101","created":334,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'22178","last_scrub_stamp":"2024-02-12T16:20:33.365829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:20:33.365829+0000","log_size":10077,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142082048,"num_objects":2168,"num_object_clones":0,"num_object_copies":4336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19672,"num_write_kb":698880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2168,"num_bytes_recovered":142082048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":105,"reported_epoch":478,"state":"undersized+peered","last_fresh":"2024-02-12T16:20:43.041176+0000","last_change":"2024-02-12T16:17:43.517378+0000","last_active":"2024-02-12T16:17:36.432068+0000","last_peered":"2024-02-12T16:20:43.041176+0000","last_clean":"2024-02-12T16:17:34.859280+0000","last_became_active":"2024-02-12T16:16:12.802102+0000","last_became_peered":"2024-02-12T16:17:43.517378+0000","last_unstale":"2024-02-12T16:20:43.041176+0000","last_undegraded":"2024-02-12T16:20:43.041176+0000","last_fullsized":"2024-02-12T16:17:43.515557+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:51.659214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:51.659214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"415'19599","reported_seq":29208,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.926703+0000","last_change":"2024-02-12T16:20:32.960286+0000","last_active":"2024-02-12T16:20:42.926703+0000","last_peered":"2024-02-12T16:20:42.926703+0000","last_clean":"2024-02-12T16:20:42.926703+0000","last_became_active":"2024-02-12T16:17:17.433883+0000","last_became_peered":"2024-02-12T16:17:17.433883+0000","last_unstale":"2024-02-12T16:20:42.926703+0000","last_undegraded":"2024-02-12T16:20:42.926703+0000","last_fullsized":"2024-02-12T16:20:42.926703+0000","mapping_epoch":403,"log_start":"360'9500","ondisk_log_start":"360'9500","created":334,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19599","last_scrub_stamp":"2024-02-12T16:20:32.960163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:20:32.960163+0000","log_size":10099,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142147584,"num_objects":2169,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19599,"num_write_kb":696576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4710,"num_bytes_recovered":308609024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":549,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041176+0000","last_change":"2024-02-12T16:20:31.483380+0000","last_active":"2024-02-12T16:20:43.041176+0000","last_peered":"2024-02-12T16:20:43.041176+0000","last_clean":"2024-02-12T16:20:43.041176+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:20:43.041176+0000","last_undegraded":"2024-02-12T16:20:43.041176+0000","last_fullsized":"2024-02-12T16:20:43.041176+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:31.483207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:31.483207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"416'19351","reported_seq":19528,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.929004+0000","last_change":"2024-02-12T16:20:35.769163+0000","last_active":"2024-02-12T16:20:42.929004+0000","last_peered":"2024-02-12T16:20:42.929004+0000","last_clean":"2024-02-12T16:20:42.929004+0000","last_became_active":"2024-02-12T16:13:45.793378+0000","last_became_peered":"2024-02-12T16:13:45.793378+0000","last_unstale":"2024-02-12T16:20:42.929004+0000","last_undegraded":"2024-02-12T16:20:42.929004+0000","last_fullsized":"2024-02-12T16:20:42.929004+0000","mapping_epoch":334,"log_start":"360'9300","ondisk_log_start":"360'9300","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19351","last_scrub_stamp":"2024-02-12T16:20:35.769042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:20:35.769042+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140050432,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19351,"num_write_kb":687616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":136,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.926399+0000","last_change":"2024-02-12T16:19:46.160601+0000","last_active":"2024-02-12T16:20:42.926399+0000","last_peered":"2024-02-12T16:20:42.926399+0000","last_clean":"2024-02-12T16:20:42.926399+0000","last_became_active":"2024-02-12T16:17:37.453541+0000","last_became_peered":"2024-02-12T16:17:37.453541+0000","last_unstale":"2024-02-12T16:20:42.926399+0000","last_undegraded":"2024-02-12T16:20:42.926399+0000","last_fullsized":"2024-02-12T16:20:42.926399+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:19:46.160476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:19:46.160476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.9","version":"414'19577","reported_seq":26296,"reported_epoch":478,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:20:42.928799+0000","last_change":"2024-02-12T16:17:43.513141+0000","last_active":"2024-02-12T16:17:42.510526+0000","last_peered":"2024-02-12T16:20:42.928799+0000","last_clean":"2024-02-12T16:17:42.490992+0000","last_became_active":"2024-02-12T16:17:39.479526+0000","last_became_peered":"2024-02-12T16:17:43.513141+0000","last_unstale":"2024-02-12T16:20:42.928799+0000","last_undegraded":"2024-02-12T16:17:43.512450+0000","last_fullsized":"2024-02-12T16:17:43.512256+0000","mapping_epoch":415,"log_start":"360'9500","ondisk_log_start":"360'9500","created":334,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'15658","last_scrub_stamp":"2024-02-12T16:16:37.730742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:16:37.730742+0000","log_size":10077,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":143065088,"num_objects":2183,"num_object_clones":0,"num_object_copies":4366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19577,"num_write_kb":696320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3267,"num_bytes_recovered":214040576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11","1"],"object_location_counts":[{"shards":"11","objects":2183}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":556,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.335761+0000","last_change":"2024-02-12T16:20:43.335715+0000","last_active":"2024-02-12T16:20:43.335761+0000","last_peered":"2024-02-12T16:20:43.335761+0000","last_clean":"2024-02-12T16:20:43.335761+0000","last_became_active":"2024-02-12T16:19:29.356482+0000","last_became_peered":"2024-02-12T16:19:29.356482+0000","last_unstale":"2024-02-12T16:20:43.335761+0000","last_undegraded":"2024-02-12T16:20:43.335761+0000","last_fullsized":"2024-02-12T16:20:43.335761+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:43.335582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:43.335582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.a","version":"415'19694","reported_seq":24533,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041513+0000","last_change":"2024-02-12T16:19:42.861389+0000","last_active":"2024-02-12T16:20:43.041513+0000","last_peered":"2024-02-12T16:20:43.041513+0000","last_clean":"2024-02-12T16:20:43.041513+0000","last_became_active":"2024-02-12T16:17:51.453423+0000","last_became_peered":"2024-02-12T16:17:51.453423+0000","last_unstale":"2024-02-12T16:20:43.041513+0000","last_undegraded":"2024-02-12T16:20:43.041513+0000","last_fullsized":"2024-02-12T16:20:43.041513+0000","mapping_epoch":420,"log_start":"360'9600","ondisk_log_start":"360'9600","created":334,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'19694","last_scrub_stamp":"2024-02-12T16:19:42.861197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:19:42.861197+0000","log_size":10094,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":151781376,"num_objects":2316,"num_object_clones":0,"num_object_copies":4632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19694,"num_write_kb":704320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":151781376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":143,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041602+0000","last_change":"2024-02-12T16:20:32.237171+0000","last_active":"2024-02-12T16:20:43.041602+0000","last_peered":"2024-02-12T16:20:43.041602+0000","last_clean":"2024-02-12T16:20:43.041602+0000","last_became_active":"2024-02-12T16:17:37.452550+0000","last_became_peered":"2024-02-12T16:17:37.452550+0000","last_unstale":"2024-02-12T16:20:43.041602+0000","last_undegraded":"2024-02-12T16:20:43.041602+0000","last_fullsized":"2024-02-12T16:20:43.041602+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:32.237053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:32.237053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"416'19630","reported_seq":19799,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.038396+0000","last_change":"2024-02-12T16:19:57.920004+0000","last_active":"2024-02-12T16:20:43.038396+0000","last_peered":"2024-02-12T16:20:43.038396+0000","last_clean":"2024-02-12T16:20:43.038396+0000","last_became_active":"2024-02-12T16:13:45.790415+0000","last_became_peered":"2024-02-12T16:13:45.790415+0000","last_unstale":"2024-02-12T16:20:43.038396+0000","last_undegraded":"2024-02-12T16:20:43.038396+0000","last_fullsized":"2024-02-12T16:20:43.038396+0000","mapping_epoch":334,"log_start":"360'9600","ondisk_log_start":"360'9600","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19630","last_scrub_stamp":"2024-02-12T16:19:57.919847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:19:57.919847+0000","log_size":10030,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142344192,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19630,"num_write_kb":697664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":141,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-02-12T16:20:46.640892+0000","last_change":"2024-02-12T16:20:46.640892+0000","last_active":"2024-02-12T16:20:46.640892+0000","last_peered":"2024-02-12T16:20:46.640892+0000","last_clean":"2024-02-12T16:20:46.640892+0000","last_became_active":"2024-02-12T16:17:37.456743+0000","last_became_peered":"2024-02-12T16:17:37.456743+0000","last_unstale":"2024-02-12T16:20:46.640892+0000","last_undegraded":"2024-02-12T16:20:46.640892+0000","last_fullsized":"2024-02-12T16:20:46.640892+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:46.640804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:46.640804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.c","version":"414'19508","reported_seq":19674,"reported_epoch":478,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:20:43.040946+0000","last_change":"2024-02-12T16:17:43.516008+0000","last_active":"2024-02-12T16:17:42.505394+0000","last_peered":"2024-02-12T16:20:43.040946+0000","last_clean":"2024-02-12T16:17:42.470323+0000","last_became_active":"2024-02-12T16:13:45.791545+0000","last_became_peered":"2024-02-12T16:17:43.516008+0000","last_unstale":"2024-02-12T16:20:43.040946+0000","last_undegraded":"2024-02-12T16:17:43.515377+0000","last_fullsized":"2024-02-12T16:17:43.515057+0000","mapping_epoch":415,"log_start":"361'9500","ondisk_log_start":"361'9500","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'11689","last_scrub_stamp":"2024-02-12T16:15:40.640284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:40.640284+0000","log_size":10008,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":146014208,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19508,"num_write_kb":695552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":2228}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":545,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041127+0000","last_change":"2024-02-12T16:19:49.201888+0000","last_active":"2024-02-12T16:20:43.041127+0000","last_peered":"2024-02-12T16:20:43.041127+0000","last_clean":"2024-02-12T16:20:43.041127+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:20:43.041127+0000","last_undegraded":"2024-02-12T16:20:43.041127+0000","last_fullsized":"2024-02-12T16:20:43.041127+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:19:49.201777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:19:49.201777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"416'19402","reported_seq":19569,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.040935+0000","last_change":"2024-02-12T16:20:03.747232+0000","last_active":"2024-02-12T16:20:43.040935+0000","last_peered":"2024-02-12T16:20:43.040935+0000","last_clean":"2024-02-12T16:20:43.040935+0000","last_became_active":"2024-02-12T16:13:45.793796+0000","last_became_peered":"2024-02-12T16:13:45.793796+0000","last_unstale":"2024-02-12T16:20:43.040935+0000","last_undegraded":"2024-02-12T16:20:43.040935+0000","last_fullsized":"2024-02-12T16:20:43.040935+0000","mapping_epoch":334,"log_start":"360'9400","ondisk_log_start":"360'9400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19402","last_scrub_stamp":"2024-02-12T16:20:03.747129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:20:03.747129+0000","log_size":10002,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140574744,"num_objects":2146,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19402,"num_write_kb":689473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":140,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-02-12T16:20:51.343128+0000","last_change":"2024-02-12T16:20:51.343128+0000","last_active":"2024-02-12T16:20:51.343128+0000","last_peered":"2024-02-12T16:20:51.343128+0000","last_clean":"2024-02-12T16:20:51.343128+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:20:51.343128+0000","last_undegraded":"2024-02-12T16:20:51.343128+0000","last_fullsized":"2024-02-12T16:20:51.343128+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:51.342966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:51.342966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"416'19451","reported_seq":19624,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-02-12T16:20:52.788388+0000","last_change":"2024-02-12T16:20:52.788388+0000","last_active":"2024-02-12T16:20:52.788388+0000","last_peered":"2024-02-12T16:20:52.788388+0000","last_clean":"2024-02-12T16:20:52.788388+0000","last_became_active":"2024-02-12T16:13:45.970126+0000","last_became_peered":"2024-02-12T16:13:45.970126+0000","last_unstale":"2024-02-12T16:20:52.788388+0000","last_undegraded":"2024-02-12T16:20:52.788388+0000","last_fullsized":"2024-02-12T16:20:52.788388+0000","mapping_epoch":334,"log_start":"360'9400","ondisk_log_start":"360'9400","created":334,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'19451","last_scrub_stamp":"2024-02-12T16:20:52.788299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:20:52.788299+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":141361152,"num_objects":2157,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19451,"num_write_kb":691456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":104,"reported_epoch":478,"state":"undersized+peered","last_fresh":"2024-02-12T16:20:43.041269+0000","last_change":"2024-02-12T16:17:43.520897+0000","last_active":"2024-02-12T16:17:36.432068+0000","last_peered":"2024-02-12T16:20:43.041269+0000","last_clean":"2024-02-12T16:17:34.859280+0000","last_became_active":"2024-02-12T16:16:12.802102+0000","last_became_peered":"2024-02-12T16:17:43.520897+0000","last_unstale":"2024-02-12T16:20:43.041269+0000","last_undegraded":"2024-02-12T16:20:43.041269+0000","last_fullsized":"2024-02-12T16:17:43.515790+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:51.659214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:51.659214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"414'19285","reported_seq":28514,"reported_epoch":478,"state":"undersized+degraded+peered","last_fresh":"2024-02-12T16:20:42.928785+0000","last_change":"2024-02-12T16:17:43.516082+0000","last_active":"2024-02-12T16:17:42.510665+0000","last_peered":"2024-02-12T16:20:42.928785+0000","last_clean":"2024-02-12T16:17:42.498681+0000","last_became_active":"2024-02-12T16:17:18.432226+0000","last_became_peered":"2024-02-12T16:17:43.516082+0000","last_unstale":"2024-02-12T16:20:42.928785+0000","last_undegraded":"2024-02-12T16:17:43.512779+0000","last_fullsized":"2024-02-12T16:17:43.512406+0000","mapping_epoch":415,"log_start":"360'9200","ondisk_log_start":"360'9200","created":334,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'12227","last_scrub_stamp":"2024-02-12T16:15:49.878411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:13:44.758147+0000","last_clean_scrub_stamp":"2024-02-12T16:15:49.878411+0000","log_size":10085,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139132928,"num_objects":2123,"num_object_clones":0,"num_object_copies":4246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19285,"num_write_kb":685056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4526,"num_bytes_recovered":296615936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11","1"],"object_location_counts":[{"shards":"11","objects":2123}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":134,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.040825+0000","last_change":"2024-02-12T16:20:42.121478+0000","last_active":"2024-02-12T16:20:43.040825+0000","last_peered":"2024-02-12T16:20:43.040825+0000","last_clean":"2024-02-12T16:20:43.040825+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:20:43.040825+0000","last_undegraded":"2024-02-12T16:20:43.040825+0000","last_fullsized":"2024-02-12T16:20:43.040825+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:42.121370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:42.121370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":559,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041153+0000","last_change":"2024-02-12T16:20:42.391514+0000","last_active":"2024-02-12T16:20:43.041153+0000","last_peered":"2024-02-12T16:20:43.041153+0000","last_clean":"2024-02-12T16:20:43.041153+0000","last_became_active":"2024-02-12T16:17:37.450842+0000","last_became_peered":"2024-02-12T16:17:37.450842+0000","last_unstale":"2024-02-12T16:20:43.041153+0000","last_undegraded":"2024-02-12T16:20:43.041153+0000","last_fullsized":"2024-02-12T16:20:43.041153+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:42.391405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:42.391405+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":136,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:42.926286+0000","last_change":"2024-02-12T16:19:42.158482+0000","last_active":"2024-02-12T16:20:42.926286+0000","last_peered":"2024-02-12T16:20:42.926286+0000","last_clean":"2024-02-12T16:20:42.926286+0000","last_became_active":"2024-02-12T16:17:37.451769+0000","last_became_peered":"2024-02-12T16:17:37.451769+0000","last_unstale":"2024-02-12T16:20:42.926286+0000","last_undegraded":"2024-02-12T16:20:42.926286+0000","last_fullsized":"2024-02-12T16:20:42.926286+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:19:42.158251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:19:42.158251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":548,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-12T16:20:43.041069+0000","last_change":"2024-02-12T16:20:39.471143+0000","last_active":"2024-02-12T16:20:43.041069+0000","last_peered":"2024-02-12T16:20:43.041069+0000","last_clean":"2024-02-12T16:20:43.041069+0000","last_became_active":"2024-02-12T16:17:37.459545+0000","last_became_peered":"2024-02-12T16:17:37.459545+0000","last_unstale":"2024-02-12T16:20:43.041069+0000","last_undegraded":"2024-02-12T16:20:43.041069+0000","last_fullsized":"2024-02-12T16:20:43.041069+0000","mapping_epoch":410,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:20:39.471003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:20:39.471003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":104,"reported_epoch":478,"state":"undersized+peered","last_fresh":"2024-02-12T16:20:43.041049+0000","last_change":"2024-02-12T16:17:43.520569+0000","last_active":"2024-02-12T16:17:36.432068+0000","last_peered":"2024-02-12T16:20:43.041049+0000","last_clean":"2024-02-12T16:17:34.859280+0000","last_became_active":"2024-02-12T16:16:12.802102+0000","last_became_peered":"2024-02-12T16:17:43.520569+0000","last_unstale":"2024-02-12T16:20:43.041049+0000","last_undegraded":"2024-02-12T16:20:43.041049+0000","last_fullsized":"2024-02-12T16:17:43.515534+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:15:51.659214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:15:51.659214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2294087704,"num_objects":35006,"num_object_clones":0,"num_object_copies":70012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8797,"num_objects_misplaced":2246,"num_objects_unfound":0,"num_objects_dirty":35006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":311664,"num_write_kb":11093377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17666,"num_bytes_recovered":1157627904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5006041088,"data_stored":5006032944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160937,"ondisk_log_size":160937,"up":27,"acting":28,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":39,"acting":39,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958801,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1277288,"kb_used_data":691680,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92467864,"statfs":{"total":95995035648,"available":94687092736,"internally_reserved":0,"allocated":708280320,"data_stored":707522962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.96499999999999997,"15min":0.77100000000000002},"min":{"1min":0.63900000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":1.4650000000000001,"5min":5.6870000000000003,"15min":5.6870000000000003},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.82799999999999996,"15min":0.88500000000000001},"min":{"1min":0.626,"5min":0.29299999999999998,"15min":0.25900000000000001},"max":{"1min":1.502,"5min":4.5010000000000003,"15min":4.992},"last":0.82099999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.1519999999999999,"15min":1.1040000000000001},"min":{"1min":0.66000000000000003,"5min":0.27100000000000002,"15min":0.25700000000000001},"max":{"1min":1.698,"5min":7.0110000000000001,"15min":7.0110000000000001},"last":1.071},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.371,"15min":1.123},"min":{"1min":0.69399999999999995,"5min":0.36299999999999999,"15min":0.27300000000000002},"max":{"1min":1.5529999999999999,"5min":5.6920000000000002,"15min":5.6920000000000002},"last":1.573}]},{"osd":2,"last update":"Mon Feb 12 16:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.0409999999999999,"15min":1.0389999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.323,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.073,"15min":1.008},"min":{"1min":0.65100000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.1890000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:17:06 2024","interfaces":[{"interface":"back","average":{"1min":8.3100000000000005,"5min":3.2749999999999999,"15min":2.2090000000000001},"min":{"1min":0.36499999999999999,"5min":0.313,"15min":0.28999999999999998},"max":{"1min":128.93899999999999,"5min":128.93899999999999,"15min":128.93899999999999},"last":0.751},{"interface":"front","average":{"1min":7.6340000000000003,"5min":3.1909999999999998,"15min":2.2519999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.30299999999999999},"max":{"1min":128.87100000000001,"5min":128.87100000000001,"15min":128.87100000000001},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.0329999999999999,"15min":2.286},"min":{"1min":0.66300000000000003,"5min":0.36799999999999999,"15min":0.29499999999999998},"max":{"1min":1.6799999999999999,"5min":4.9580000000000002,"15min":368.48399999999998},"last":1.403},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.97099999999999997,"15min":2.169},"min":{"1min":0.70099999999999996,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.6859999999999999,"5min":4.6859999999999999,"15min":371.65100000000001},"last":0.89100000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.1699999999999999,"15min":3.7160000000000002},"min":{"1min":0.89200000000000002,"5min":0.35099999999999998,"15min":0.34399999999999997},"max":{"1min":1.758,"5min":5.8789999999999996,"15min":407.07600000000002},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.1839999999999999,"15min":3.7949999999999999},"min":{"1min":0.997,"5min":0.38,"15min":0.35399999999999998},"max":{"1min":1.772,"5min":4.9909999999999997,"15min":407.26299999999998},"last":1.542}]},{"osd":6,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.0009999999999999,"15min":0.95999999999999996},"min":{"1min":0.93200000000000005,"5min":0.373,"15min":0.34799999999999998},"max":{"1min":1.575,"5min":4.5979999999999999,"15min":4.8330000000000002},"last":1.427},{"interface":"front","average":{"1min":1.345,"5min":1.1100000000000001,"15min":1.0720000000000001},"min":{"1min":0.95599999999999996,"5min":0.40999999999999998,"15min":0.40300000000000002},"max":{"1min":1.6539999999999999,"5min":4.6539999999999999,"15min":4.6539999999999999},"last":1.6799999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.3580000000000001,"15min":1.1579999999999999},"min":{"1min":0.97399999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.774,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.425,"5min":1.3140000000000001,"15min":1.165},"min":{"1min":1.044,"5min":0.48299999999999998,"15min":0.438},"max":{"1min":1.8220000000000001,"5min":4.9939999999999998,"15min":5.2510000000000003},"last":1.331}]},{"osd":8,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.117,"15min":1.048},"min":{"1min":0.92000000000000004,"5min":0.443,"15min":0.38200000000000001},"max":{"1min":1.8009999999999999,"5min":4.5339999999999998,"15min":4.5940000000000003},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.569,"5min":1.327,"15min":1.131},"min":{"1min":0.93200000000000005,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.0009999999999999,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":1.651}]},{"osd":9,"last update":"Mon Feb 12 16:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.427,"15min":1.1599999999999999},"min":{"1min":0.69899999999999995,"5min":0.45900000000000002,"15min":0.40400000000000003},"max":{"1min":2.3610000000000002,"5min":5.843,"15min":10.743},"last":1.742},{"interface":"front","average":{"1min":1.704,"5min":1.605,"15min":1.2909999999999999},"min":{"1min":0.98499999999999999,"5min":0.53000000000000003,"15min":0.39500000000000002},"max":{"1min":2.4180000000000001,"5min":6.8869999999999996,"15min":10.698},"last":1.8140000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.278,"15min":1.1599999999999999},"min":{"1min":0.94699999999999995,"5min":0.47399999999999998,"15min":0.377},"max":{"1min":2.2210000000000001,"5min":6.7160000000000002,"15min":6.7160000000000002},"last":1.635},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.274,"15min":1.1539999999999999},"min":{"1min":1.024,"5min":0.46500000000000002,"15min":0.45200000000000001},"max":{"1min":2.117,"5min":4.6210000000000004,"15min":4.8630000000000004},"last":1.7629999999999999}]}]},{"osd":10,"up_from":83,"seq":356482285850,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869992,"kb_used_data":568672,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92875160,"statfs":{"total":95995035648,"available":95104163840,"internally_reserved":0,"allocated":582320128,"data_stored":581562770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95499999999999996,"15min":1.0069999999999999},"min":{"1min":0.61899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.3819999999999999,"5min":4.4669999999999996,"15min":4.4669999999999996},"last":1.042},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.0489999999999999,"15min":1.095},"min":{"1min":0.59899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.2569999999999999,"5min":4.726,"15min":4.726},"last":1.0820000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.177,"15min":1.1180000000000001},"min":{"1min":0.72499999999999998,"5min":0.29599999999999999,"15min":0.26100000000000001},"max":{"1min":1.3109999999999999,"5min":19.866,"15min":49.802999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.464,"15min":1.246},"min":{"1min":0.80300000000000005,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.484,"5min":20.277999999999999,"15min":50.863},"last":1.145}]},{"osd":2,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.94899999999999995,"15min":0.80100000000000005},"min":{"1min":0.60799999999999998,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":1.516,"5min":4.7889999999999997,"15min":5.3040000000000003},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.22,"5min":1.0469999999999999,"15min":0.88900000000000001},"min":{"1min":0.81599999999999995,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":1.7789999999999999,"5min":4.8070000000000004,"15min":7.9829999999999997},"last":0.80600000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.2190000000000001,"15min":3.746},"min":{"1min":0.66000000000000003,"5min":0.34100000000000003,"15min":0.32300000000000001},"max":{"1min":1.964,"5min":5.6420000000000003,"15min":628.77099999999996},"last":0.999},{"interface":"front","average":{"1min":1.177,"5min":1.151,"15min":3.7000000000000002},"min":{"1min":0.78000000000000003,"5min":0.38800000000000001,"15min":0.33800000000000002},"max":{"1min":1.6930000000000001,"5min":5.0149999999999997,"15min":624.19299999999998},"last":0.96599999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1100000000000001,"15min":3.5139999999999998},"min":{"1min":0.68799999999999994,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.9399999999999999,"5min":5.5739999999999998,"15min":602.05399999999997},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.2769999999999999,"15min":3.6240000000000001},"min":{"1min":0.88800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.885,"5min":4.8769999999999998,"15min":598.875},"last":1.2070000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.1830000000000001,"15min":3.3010000000000002},"min":{"1min":0.874,"5min":0.41199999999999998,"15min":0.35399999999999998},"max":{"1min":2.0299999999999998,"5min":4.6210000000000004,"15min":626.26599999999996},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.3169999999999999,"15min":3.4009999999999998},"min":{"1min":0.98799999999999999,"5min":0.41099999999999998,"15min":0.40000000000000002},"max":{"1min":2.04,"5min":4.9580000000000002,"15min":625.90700000000004},"last":1.325}]},{"osd":7,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.202,"15min":3.1669999999999998},"min":{"1min":0.72599999999999998,"5min":0.39800000000000002,"15min":0.376},"max":{"1min":1.754,"5min":5.9210000000000003,"15min":597.48299999999995},"last":1.107},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.3280000000000001,"15min":3.1960000000000002},"min":{"1min":0.99399999999999999,"5min":0.438,"15min":0.438},"max":{"1min":2.161,"5min":5.3300000000000001,"15min":600.23099999999999},"last":1.4219999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.3009999999999999,"15min":1.1100000000000001},"min":{"1min":1.1339999999999999,"5min":0.372,"15min":0.33400000000000002},"max":{"1min":2.226,"5min":4.827,"15min":7.2720000000000002},"last":1.258},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.304,"15min":1.093},"min":{"1min":1.0449999999999999,"5min":0.42699999999999999,"15min":0.38200000000000001},"max":{"1min":1.931,"5min":4.8330000000000002,"15min":5.4550000000000001},"last":1.3959999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.393,"15min":1.1599999999999999},"min":{"1min":1.139,"5min":0.372,"15min":0.372},"max":{"1min":2.2549999999999999,"5min":4.9829999999999997,"15min":4.9829999999999997},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.51,"5min":1.399,"15min":1.1679999999999999},"min":{"1min":1.1180000000000001,"5min":0.41399999999999998,"15min":0.40000000000000002},"max":{"1min":2.1600000000000001,"5min":4.952,"15min":4.952},"last":1.2729999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.4430000000000001,"15min":1.411},"min":{"1min":1.079,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.347,"5min":10.218999999999999,"15min":30.917999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.4570000000000001,"15min":1.3759999999999999},"min":{"1min":0.94199999999999995,"5min":0.52300000000000002,"15min":0.48499999999999999},"max":{"1min":2.0550000000000002,"5min":7.3250000000000002,"15min":30.742000000000001},"last":1.363}]}]},{"osd":9,"up_from":74,"seq":317827580194,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":715628,"kb_used_data":414308,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93029524,"statfs":{"total":95995035648,"available":95262232576,"internally_reserved":0,"allocated":424251392,"data_stored":423489970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.0740000000000001,"15min":1.077},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.4119999999999999,"5min":1.615,"15min":1.615},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.89300000000000002,"15min":0.88700000000000001},"min":{"1min":0.66700000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.76800000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.95299999999999996,"15min":1.498},"min":{"1min":0.84699999999999998,"5min":0.253,"15min":0.253},"max":{"1min":1.615,"5min":5.843,"15min":175.33500000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.05,"15min":1.6279999999999999},"min":{"1min":0.77500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.212,"5min":7.9550000000000001,"15min":175.24299999999999},"last":0.96599999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":2.0739999999999998,"15min":1.4379999999999999},"min":{"1min":0.79400000000000004,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.827,"5min":97.275000000000006,"15min":97.275000000000006},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.169,"5min":1.9370000000000001,"15min":1.3200000000000001},"min":{"1min":0.76700000000000002,"5min":0.312,"15min":0.28199999999999997},"max":{"1min":2.6000000000000001,"5min":97.417000000000002,"15min":97.417000000000002},"last":0.78000000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.93600000000000005,"15min":1.0129999999999999},"min":{"1min":0.78900000000000003,"5min":0.35099999999999998,"15min":0.32200000000000001},"max":{"1min":1.577,"5min":3.6629999999999998,"15min":25.552},"last":0.873},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.1240000000000001,"15min":1.0429999999999999},"min":{"1min":1.107,"5min":0.442,"15min":0.35399999999999998},"max":{"1min":1.788,"5min":2.125,"15min":25.847999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.0780000000000001,"15min":0.98799999999999999},"min":{"1min":0.82599999999999996,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":1.6799999999999999,"5min":2.0270000000000001,"15min":5.899},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.101,"15min":1.1040000000000001},"min":{"1min":0.751,"5min":0.47999999999999998,"15min":0.375},"max":{"1min":1.7529999999999999,"5min":2.726,"15min":7.3739999999999997},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.1140000000000001,"15min":1.105},"min":{"1min":0.93200000000000005,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.581,"5min":2.7559999999999998,"15min":7.3360000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.226,"5min":1.044,"15min":0.94599999999999995},"min":{"1min":0.95399999999999996,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.9730000000000001,"5min":3.6240000000000001,"15min":4.7910000000000004},"last":0.69999999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.218,"15min":2.3620000000000001},"min":{"1min":1.298,"5min":0.42799999999999999,"15min":0.379},"max":{"1min":1.7789999999999999,"5min":2.867,"15min":429.14699999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.2350000000000001,"15min":2.4039999999999999},"min":{"1min":1.02,"5min":0.438,"15min":0.438},"max":{"1min":1.726,"5min":4.2110000000000003,"15min":429.73700000000002},"last":0.72499999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.149,"15min":1.079},"min":{"1min":0.85099999999999998,"5min":0.41299999999999998,"15min":0.40500000000000003},"max":{"1min":2.1640000000000001,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.518,"5min":1.3009999999999999,"15min":1.117},"min":{"1min":1.1120000000000001,"5min":0.52500000000000002,"15min":0.41199999999999998},"max":{"1min":1.905,"5min":5.4340000000000002,"15min":7.25},"last":0.89700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:20:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.274,"15min":1.069},"min":{"1min":1.0620000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.2639999999999998,"5min":3.6920000000000002,"15min":5.5},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.482,"5min":1.3400000000000001,"15min":1.133},"min":{"1min":0.93100000000000005,"5min":0.38,"15min":0.38},"max":{"1min":2.153,"5min":5.8700000000000001,"15min":5.8700000000000001},"last":0.83599999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.6990000000000001,"5min":1.554,"15min":1.2689999999999999},"min":{"1min":1.266,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.95,"5min":5.3840000000000003,"15min":5.3840000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.4690000000000001,"15min":1.3089999999999999},"min":{"1min":1.177,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.0179999999999998,"5min":6.3639999999999999,"15min":6.3639999999999999},"last":0.81200000000000006}]}]},{"osd":8,"up_from":65,"seq":279172874539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311016,"kb_used_data":1504,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434136,"statfs":{"total":95995035648,"available":95676555264,"internally_reserved":0,"allocated":1540096,"data_stored":782746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.90200000000000002,"15min":0.79500000000000004},"min":{"1min":0.66200000000000003,"5min":0.29699999999999999,"15min":0.22700000000000001},"max":{"1min":1.7569999999999999,"5min":5.2290000000000001,"15min":5.383},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.0649999999999999,"5min":0.754,"15min":0.79200000000000004},"min":{"1min":0.64600000000000002,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":1.7410000000000001,"5min":2.9420000000000002,"15min":7.319},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.1359999999999999,"15min":1.369},"min":{"1min":0.72899999999999998,"5min":0.35299999999999998,"15min":0.30499999999999999},"max":{"1min":1.663,"5min":6.3940000000000001,"15min":54.975999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.242,"5min":1.2549999999999999,"15min":1.458},"min":{"1min":0.81299999999999994,"5min":0.35499999999999998,"15min":0.30299999999999999},"max":{"1min":1.706,"5min":6.4039999999999999,"15min":55.005000000000003},"last":0.69099999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.84199999999999997,"15min":0.83399999999999996},"min":{"1min":0.69899999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.069,"5min":1.0289999999999999,"15min":1.022},"min":{"1min":0.69999999999999996,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.014,"15min":1.6459999999999999},"min":{"1min":0.89400000000000002,"5min":0.376,"15min":0.32400000000000001},"max":{"1min":1.895,"5min":2.109,"15min":225.33099999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.304,"5min":0.92700000000000005,"15min":1.619},"min":{"1min":1.04,"5min":0.378,"15min":0.32400000000000001},"max":{"1min":1.677,"5min":2.0459999999999998,"15min":224.71000000000001},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":0.94199999999999995,"15min":0.90400000000000003},"min":{"1min":1.0660000000000001,"5min":0.42999999999999999,"15min":0.32000000000000001},"max":{"1min":1.9670000000000001,"5min":4.8689999999999998,"15min":13.961},"last":0.438},{"interface":"front","average":{"1min":1.399,"5min":1.0129999999999999,"15min":0.998},"min":{"1min":0.76500000000000001,"5min":0.439,"15min":0.29999999999999999},"max":{"1min":1.9570000000000001,"5min":2.6709999999999998,"15min":13.103},"last":0.52400000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.94799999999999995,"15min":1.6379999999999999},"min":{"1min":0.80600000000000005,"5min":0.377,"15min":0.36299999999999999},"max":{"1min":1.631,"5min":4.056,"15min":221.36699999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.573,"5min":1.1439999999999999,"15min":1.8049999999999999},"min":{"1min":0.89100000000000001,"5min":0.46500000000000002,"15min":0.38300000000000001},"max":{"1min":2.0699999999999998,"5min":4.7110000000000003,"15min":221.203},"last":0.42199999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.1679999999999999,"15min":1.133},"min":{"1min":1.0860000000000001,"5min":0.42199999999999999,"15min":0.42099999999999999},"max":{"1min":1.635,"5min":7.9160000000000004,"15min":13.242000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.3200000000000001,"15min":1.159},"min":{"1min":1.3460000000000001,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":2.2469999999999999,"5min":7.899,"15min":13.388},"last":0.61699999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:19:52 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.1890000000000001,"15min":1.069},"min":{"1min":0.98599999999999999,"5min":0.41799999999999998,"15min":0.34699999999999998},"max":{"1min":2.1789999999999998,"5min":4.5099999999999998,"15min":5.8449999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.746,"5min":1.2330000000000001,"15min":1.121},"min":{"1min":1.04,"5min":0.46200000000000002,"15min":0.36799999999999999},"max":{"1min":2.222,"5min":2.222,"15min":6.2000000000000002},"last":0.47099999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:20:23 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":2.2440000000000002,"15min":1.3939999999999999},"min":{"1min":1.091,"5min":0.44400000000000001,"15min":0.36399999999999999},"max":{"1min":1.8779999999999999,"5min":109.92,"15min":109.92},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.444,"5min":2.363,"15min":1.397},"min":{"1min":1.179,"5min":0.46899999999999997,"15min":0.39200000000000002},"max":{"1min":2.1139999999999999,"5min":110.249,"15min":110.249},"last":0.49099999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.2989999999999999,"15min":1.214},"min":{"1min":1.141,"5min":0.51200000000000001,"15min":0.441},"max":{"1min":2.5569999999999999,"5min":5.0670000000000002,"15min":6.1509999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.3160000000000001,"15min":1.208},"min":{"1min":1.3100000000000001,"5min":0.53100000000000003,"15min":0.46100000000000002},"max":{"1min":2.5899999999999999,"5min":3.23,"15min":8.25},"last":0.53400000000000003}]}]},{"osd":7,"up_from":56,"seq":240518168882,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719400,"kb_used_data":422176,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93025752,"statfs":{"total":95995035648,"available":95258370048,"internally_reserved":0,"allocated":432308224,"data_stored":431550874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.94299999999999995,"15min":0.89000000000000001},"min":{"1min":0.57899999999999996,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.508,"5min":5.5149999999999997,"15min":5.5149999999999997},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.087,"5min":0.99299999999999999,"15min":0.878},"min":{"1min":0.68300000000000005,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.7909999999999999,"5min":4.8570000000000002,"15min":4.8570000000000002},"last":0.65600000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.97199999999999998,"15min":0.97799999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.123,"5min":1.153,"15min":1.155},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:20:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.97999999999999998,"15min":1.1819999999999999},"min":{"1min":0.79300000000000004,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.1640000000000001,"5min":4.5899999999999999,"15min":60.204999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.054,"5min":1.0229999999999999,"15min":1.175},"min":{"1min":0.71399999999999997,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":1.367,"5min":7.2320000000000002,"15min":59.365000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:17:20 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.86399999999999999,"15min":1.0620000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27900000000000003},"max":{"1min":4.1040000000000001,"5min":4.6369999999999996,"15min":6.2439999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":0.93600000000000005,"15min":1.0740000000000001},"min":{"1min":0.39200000000000002,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":4.1539999999999999,"5min":4.75,"15min":8.2940000000000005},"last":0.46100000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.89100000000000001,"15min":0.89200000000000002},"min":{"1min":0.60199999999999998,"5min":0.29699999999999999,"15min":0.28399999999999997},"max":{"1min":1.4339999999999999,"5min":4.7930000000000001,"15min":6.2519999999999998},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.143,"5min":1.079,"15min":0.997},"min":{"1min":0.60599999999999998,"5min":0.36799999999999999,"15min":0.25600000000000001},"max":{"1min":1.524,"5min":4.5030000000000001,"15min":6.0919999999999996},"last":1.0429999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:20:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0129999999999999,"15min":0.95199999999999996},"min":{"1min":0.67900000000000005,"5min":0.38800000000000001,"15min":0.29099999999999998},"max":{"1min":1.579,"5min":4.9050000000000002,"15min":6.0739999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.21,"5min":1.022,"15min":0.998},"min":{"1min":0.66700000000000004,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.659,"5min":3.9180000000000001,"15min":9.3640000000000008},"last":0.95799999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:20:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1459999999999999,"15min":1.1140000000000001},"min":{"1min":0.84899999999999998,"5min":0.36699999999999999,"15min":0.29499999999999998},"max":{"1min":2.327,"5min":4.9359999999999999,"15min":6.0830000000000002},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.304,"15min":1.1259999999999999},"min":{"1min":0.93799999999999994,"5min":0.40799999999999997,"15min":0.29999999999999999},"max":{"1min":2.532,"5min":4.633,"15min":6.1349999999999998},"last":0.98299999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.1180000000000001,"15min":1.1299999999999999},"min":{"1min":0.91800000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.891,"5min":5.6130000000000004,"15min":6.0250000000000004},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.165,"15min":1.1639999999999999},"min":{"1min":1.3169999999999999,"5min":0.45000000000000001,"15min":0.42999999999999999},"max":{"1min":1.8620000000000001,"5min":5.5860000000000003,"15min":9.2140000000000004},"last":1.0309999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:20:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.4530000000000001,"15min":2.3100000000000001},"min":{"1min":0.56000000000000005,"5min":0.47599999999999998,"15min":0.441},"max":{"1min":1.7509999999999999,"5min":12.907999999999999,"15min":278.54599999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.466,"15min":2.258},"min":{"1min":0.54300000000000004,"5min":0.42699999999999999,"15min":0.38200000000000001},"max":{"1min":2.2250000000000001,"5min":4.6719999999999997,"15min":281.09199999999998},"last":1.1479999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:20:21 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.524,"15min":1.2410000000000001},"min":{"1min":1.1839999999999999,"5min":0.53900000000000003,"15min":0.47899999999999998},"max":{"1min":2.3730000000000002,"5min":5.8689999999999998,"15min":5.9489999999999998},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.238,"15min":1.1859999999999999},"min":{"1min":0.82999999999999996,"5min":0.47899999999999998,"15min":0.42099999999999999},"max":{"1min":1.9330000000000001,"5min":4.0069999999999997,"15min":9.1690000000000005},"last":1.246}]},{"osd":11,"last update":"Mon Feb 12 16:19:53 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":2.3929999999999998,"15min":2.6949999999999998},"min":{"1min":1.1060000000000001,"5min":0.57599999999999996,"15min":0.498},"max":{"1min":2.3969999999999998,"5min":120.943,"15min":280.83999999999997},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.7430000000000001,"5min":2.5600000000000001,"15min":2.7709999999999999},"min":{"1min":1.2809999999999999,"5min":0.503,"15min":0.45300000000000001},"max":{"1min":2.452,"5min":120.661,"15min":278.63799999999998},"last":1.103}]}]},{"osd":6,"up_from":48,"seq":206158430523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597224,"kb_used_data":291808,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93147928,"statfs":{"total":95995035648,"available":95383478272,"internally_reserved":0,"allocated":298811392,"data_stored":298054042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.022,"15min":1.0349999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.415,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.1000000000000001,"15min":1.111},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":1.123}]},{"osd":1,"last update":"Mon Feb 12 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":3.7610000000000001,"15min":2.0950000000000002},"min":{"1min":0.67900000000000005,"5min":0.33700000000000002,"15min":0.27200000000000002},"max":{"1min":1.5269999999999999,"5min":252.22800000000001,"15min":252.22800000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":3.9350000000000001,"15min":2.1179999999999999},"min":{"1min":0.61799999999999999,"5min":0.30399999999999999,"15min":0.28799999999999998},"max":{"1min":1.081,"5min":252.61199999999999,"15min":252.61199999999999},"last":1.081}]},{"osd":2,"last update":"Mon Feb 12 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.86899999999999999,"15min":1.1180000000000001},"min":{"1min":0.58299999999999996,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":1.6319999999999999,"5min":3.3279999999999998,"15min":68.462000000000003},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.90800000000000003,"15min":1.169},"min":{"1min":0.505,"5min":0.311,"15min":0.311},"max":{"1min":1.232,"5min":4.5810000000000004,"15min":67.322999999999993},"last":0.85699999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.93300000000000005,"15min":2.0510000000000002},"min":{"1min":0.42899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.3680000000000001,"5min":6.6120000000000001,"15min":284.05599999999998},"last":1.224},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.0640000000000001,"15min":1.9450000000000001},"min":{"1min":0.53400000000000003,"5min":0.40500000000000003,"15min":0.27000000000000002},"max":{"1min":2.2200000000000002,"5min":5.6429999999999998,"15min":284.041},"last":1.423}]},{"osd":5,"last update":"Mon Feb 12 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.97299999999999998,"15min":2.081},"min":{"1min":0.34899999999999998,"5min":0.312,"15min":0.312},"max":{"1min":1.4750000000000001,"5min":4.0519999999999996,"15min":281.49700000000001},"last":1.149},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.0429999999999999,"15min":2.0990000000000002},"min":{"1min":0.48599999999999999,"5min":0.33500000000000002,"15min":0.29399999999999998},"max":{"1min":2.069,"5min":4.3579999999999997,"15min":281.61900000000003},"last":1.546}]},{"osd":7,"last update":"Mon Feb 12 16:20:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.1160000000000001,"15min":1.5960000000000001},"min":{"1min":0.51400000000000001,"5min":0.39500000000000002,"15min":0.29399999999999998},"max":{"1min":2.4889999999999999,"5min":5.343,"15min":67.144999999999996},"last":1.484},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.018,"15min":1.5980000000000001},"min":{"1min":0.67100000000000004,"5min":0.436,"15min":0.38200000000000001},"max":{"1min":1.383,"5min":5.0190000000000001,"15min":67.230999999999995},"last":1.1990000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:19:59 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.1539999999999999,"15min":1.3089999999999999},"min":{"1min":0.99399999999999999,"5min":0.497,"15min":0.39800000000000002},"max":{"1min":2.1349999999999998,"5min":4.2380000000000004,"15min":68.653000000000006},"last":1.29},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.2949999999999999,"15min":1.3240000000000001},"min":{"1min":1.125,"5min":0.44,"15min":0.40400000000000003},"max":{"1min":2.23,"5min":5.4379999999999997,"15min":68.620999999999995},"last":1.611}]},{"osd":9,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.3859999999999999,"15min":1.3879999999999999},"min":{"1min":0.84699999999999998,"5min":0.52600000000000002,"15min":0.40600000000000003},"max":{"1min":2.2400000000000002,"5min":18.059999999999999,"15min":67.188000000000002},"last":1.345},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.478,"15min":1.446},"min":{"1min":0.81000000000000005,"5min":0.442,"15min":0.442},"max":{"1min":2.8460000000000001,"5min":17.425000000000001,"15min":68.506},"last":1.6639999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:20:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.399,"15min":1.3799999999999999},"min":{"1min":0.85199999999999998,"5min":0.54300000000000004,"15min":0.49099999999999999},"max":{"1min":2.323,"5min":6.5300000000000002,"15min":68.722999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.569,"5min":1.5049999999999999,"15min":1.407},"min":{"1min":1.0189999999999999,"5min":0.58399999999999996,"15min":0.505},"max":{"1min":2.448,"5min":6.5730000000000004,"15min":68.573999999999998},"last":1.698}]},{"osd":11,"last update":"Mon Feb 12 16:20:35 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":2.6059999999999999,"15min":5.5529999999999999},"min":{"1min":0.73799999999999999,"5min":0.622,"15min":0.622},"max":{"1min":2.1240000000000001,"5min":139.727,"15min":139.727},"last":1.454},{"interface":"front","average":{"1min":1.6220000000000001,"5min":2.5790000000000002,"15min":5.3109999999999999},"min":{"1min":0.84399999999999997,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.9489999999999998,"5min":139.24199999999999,"15min":139.24199999999999},"last":1.7370000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575201,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1215208,"kb_used_data":621408,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92529944,"statfs":{"total":95995035648,"available":94750662656,"internally_reserved":0,"allocated":636321792,"data_stored":635564442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81999999999999995,"15min":1.4910000000000001},"min":{"1min":0.51600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.1639999999999999,"5min":4.0880000000000001,"15min":114.40000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.0429999999999999,"15min":1.698},"min":{"1min":0.47999999999999998,"5min":0.23899999999999999,"15min":0.23699999999999999},"max":{"1min":1.083,"5min":11.433,"15min":115.09099999999999},"last":1.042}]},{"osd":2,"last update":"Mon Feb 12 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.1899999999999999,"15min":1.24},"min":{"1min":0.55800000000000005,"5min":0.248,"15min":0.248},"max":{"1min":1.028,"5min":9.8599999999999994,"15min":40.542999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.032,"15min":1.389},"min":{"1min":0.56399999999999995,"5min":0.26100000000000001,"15min":0.24299999999999999},"max":{"1min":1.401,"5min":5.1689999999999996,"15min":40.811999999999998},"last":1.2949999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:51 2024","interfaces":[{"interface":"back","average":{"1min":6.9550000000000001,"5min":2.2480000000000002,"15min":1.3380000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":142.78100000000001,"5min":142.78100000000001,"15min":142.78100000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":7.0170000000000003,"5min":2.3719999999999999,"15min":1.5269999999999999},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":142.32499999999999,"5min":142.32499999999999,"15min":142.32499999999999},"last":4.2370000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:19:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.998,"15min":1.0009999999999999},"min":{"1min":0.72599999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.206,"5min":2.141,"15min":2.141},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1100000000000001,"15min":1.1180000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.264,"5min":1.494,"15min":1.494},"last":1.0129999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1839999999999999,"15min":1.169},"min":{"1min":0.76600000000000001,"5min":0.35599999999999998,"15min":0.313},"max":{"1min":1.641,"5min":9.9359999999999999,"15min":9.9359999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.262,"15min":1.206},"min":{"1min":0.79300000000000004,"5min":0.35899999999999999,"15min":0.34000000000000002},"max":{"1min":1.8520000000000001,"5min":7.0110000000000001,"15min":10.420999999999999},"last":0.94399999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.3140000000000001,"15min":1.6739999999999999},"min":{"1min":0.73199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":2.1659999999999999,"5min":7.0940000000000003,"15min":56.137},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.421,"15min":1.76},"min":{"1min":0.88700000000000001,"5min":0.44,"15min":0.39000000000000001},"max":{"1min":2.4569999999999999,"5min":7.125,"15min":56.195999999999998},"last":1.27}]},{"osd":7,"last update":"Mon Feb 12 16:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.3220000000000001,"15min":1.6830000000000001},"min":{"1min":0.71599999999999997,"5min":0.39300000000000002,"15min":0.34699999999999998},"max":{"1min":1.486,"5min":6.609,"15min":55.869999999999997},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.4239999999999999,"15min":1.7010000000000001},"min":{"1min":0.84499999999999997,"5min":0.42999999999999999,"15min":0.36799999999999999},"max":{"1min":2.024,"5min":9.7420000000000009,"15min":55.892000000000003},"last":1.337}]},{"osd":8,"last update":"Mon Feb 12 16:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.4450000000000001,"15min":1.353},"min":{"1min":0.94799999999999995,"5min":0.436,"15min":0.39400000000000002},"max":{"1min":2.3319999999999999,"5min":7.2190000000000003,"15min":9.9049999999999994},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.325,"15min":1.4350000000000001},"min":{"1min":1.028,"5min":0.56999999999999995,"15min":0.41999999999999998},"max":{"1min":2.0670000000000002,"5min":4.5910000000000002,"15min":10.313000000000001},"last":1.385}]},{"osd":9,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.3959999999999999,"15min":1.399},"min":{"1min":0.98099999999999998,"5min":0.52500000000000002,"15min":0.40699999999999997},"max":{"1min":1.9770000000000001,"5min":4.7450000000000001,"15min":10.352},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5960000000000001,"15min":1.45},"min":{"1min":1.071,"5min":0.52700000000000002,"15min":0.435},"max":{"1min":2.6909999999999998,"5min":7.2789999999999999,"15min":7.6059999999999999},"last":1.544}]},{"osd":10,"last update":"Mon Feb 12 16:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.401,"15min":1.3520000000000001},"min":{"1min":1.1619999999999999,"5min":0.54500000000000004,"15min":0.48499999999999999},"max":{"1min":2.081,"5min":4.6219999999999999,"15min":10.157999999999999},"last":1.448},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.633,"15min":1.5049999999999999},"min":{"1min":1.1259999999999999,"5min":0.54100000000000004,"15min":0.52700000000000002},"max":{"1min":2.573,"5min":8.7720000000000002,"15min":10.387},"last":1.6140000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.7130000000000001,"15min":1.5289999999999999},"min":{"1min":1.135,"5min":0.55800000000000005,"15min":0.443},"max":{"1min":2.823,"5min":7.1600000000000001,"15min":13.555},"last":1.488},{"interface":"front","average":{"1min":1.649,"5min":1.8340000000000001,"15min":1.5509999999999999},"min":{"1min":1.2,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":3.0030000000000001,"5min":7.2489999999999997,"15min":13.846},"last":1.4990000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738732,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580648,"kb_used_data":279328,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93164504,"statfs":{"total":95995035648,"available":95400452096,"internally_reserved":0,"allocated":286031872,"data_stored":285274522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.2729999999999999,"15min":1.639},"min":{"1min":0.55700000000000005,"5min":0.215,"15min":0.215},"max":{"1min":1.2410000000000001,"5min":11.324,"15min":114.01600000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.1240000000000001,"15min":1.534},"min":{"1min":0.53600000000000003,"5min":0.23699999999999999,"15min":0.22800000000000001},"max":{"1min":1.333,"5min":9.8249999999999993,"15min":114.342},"last":0.96399999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.83099999999999996,"15min":0.82099999999999995},"min":{"1min":0.60599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":1.329,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.996,"5min":0.89300000000000002,"15min":0.877},"min":{"1min":0.58399999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.397,"5min":1.647,"15min":1.647},"last":1.121}]},{"osd":3,"last update":"Mon Feb 12 16:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.80900000000000005,"15min":1.2},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.215},"max":{"1min":1.48,"5min":5.4269999999999996,"15min":53.793999999999997},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.80700000000000005,"15min":1.137},"min":{"1min":0.29199999999999998,"5min":0.252,"15min":0.252},"max":{"1min":4.2199999999999998,"5min":6.7089999999999996,"15min":53.853000000000002},"last":1.4390000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:20:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0109999999999999,"15min":1.583},"min":{"1min":0.64900000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.4970000000000001,"5min":5.9969999999999999,"15min":116.187},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.1479999999999999,"15min":1.6759999999999999},"min":{"1min":0.71099999999999997,"5min":0.32800000000000001,"15min":0.28299999999999997},"max":{"1min":1.321,"5min":6.9749999999999996,"15min":116.163},"last":1.3069999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:20:05 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.92300000000000004,"15min":1.3959999999999999},"min":{"1min":0.59199999999999997,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":1.6419999999999999,"5min":4.0250000000000004,"15min":116.005},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":0.94499999999999995,"15min":1.3959999999999999},"min":{"1min":0.83599999999999997,"5min":0.35799999999999998,"15min":0.31},"max":{"1min":1.8,"5min":4.2060000000000004,"15min":115.973},"last":1.363}]},{"osd":6,"last update":"Mon Feb 12 16:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.0249999999999999,"15min":1.2170000000000001},"min":{"1min":0.89600000000000002,"5min":0.35499999999999998,"15min":0.33500000000000002},"max":{"1min":1.544,"5min":4.0540000000000003,"15min":81.147000000000006},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.1339999999999999,"15min":1.2749999999999999},"min":{"1min":1.024,"5min":0.42799999999999999,"15min":0.34300000000000003},"max":{"1min":1.8720000000000001,"5min":4.1470000000000002,"15min":80.233000000000004},"last":1.536}]},{"osd":7,"last update":"Mon Feb 12 16:20:40 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.133,"15min":1.5069999999999999},"min":{"1min":0.95799999999999996,"5min":0.42699999999999999,"15min":0.36699999999999999},"max":{"1min":1.921,"5min":4.1139999999999999,"15min":115.26000000000001},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.2529999999999999,"15min":1.6080000000000001},"min":{"1min":1.036,"5min":0.51600000000000001,"15min":0.44800000000000001},"max":{"1min":1.988,"5min":4.1639999999999997,"15min":115.402},"last":1.794}]},{"osd":8,"last update":"Mon Feb 12 16:20:05 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.258,"15min":1.165},"min":{"1min":1.097,"5min":0.44800000000000001,"15min":0.378},"max":{"1min":2.0960000000000001,"5min":4.806,"15min":5.6059999999999999},"last":1.855},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.343,"15min":1.228},"min":{"1min":1.23,"5min":0.51300000000000001,"15min":0.41299999999999998},"max":{"1min":2.2490000000000001,"5min":4.8380000000000001,"15min":7.5599999999999996},"last":1.9079999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.7050000000000001,"15min":1.3919999999999999},"min":{"1min":0.997,"5min":0.436,"15min":0.40600000000000003},"max":{"1min":1.915,"5min":46.246000000000002,"15min":46.246000000000002},"last":1.649},{"interface":"front","average":{"1min":1.679,"5min":1.6870000000000001,"15min":1.323},"min":{"1min":1.153,"5min":0.45800000000000002,"15min":0.39500000000000002},"max":{"1min":2.024,"5min":46.167999999999999,"15min":46.167999999999999},"last":1.6140000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:20:40 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.3600000000000001,"15min":1.2430000000000001},"min":{"1min":0.85899999999999999,"5min":0.41999999999999998,"15min":0.41399999999999998},"max":{"1min":1.8049999999999999,"5min":6.0979999999999999,"15min":6.0979999999999999},"last":1.456},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.3360000000000001,"15min":1.256},"min":{"1min":0.99099999999999999,"5min":0.47999999999999998,"15min":0.437},"max":{"1min":1.998,"5min":5.6269999999999998,"15min":6.2480000000000002},"last":1.625}]},{"osd":11,"last update":"Mon Feb 12 16:20:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.891,"15min":1.478},"min":{"1min":0.91500000000000004,"5min":0.434,"15min":0.434},"max":{"1min":2.036,"5min":46.064,"15min":46.064},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.8720000000000001,"15min":1.5129999999999999},"min":{"1min":1.175,"5min":0.56000000000000005,"15min":0.38300000000000001},"max":{"1min":2.3570000000000002,"5min":45.642000000000003,"15min":45.642000000000003},"last":1.6719999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593512,"kb_used_data":284000,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93151640,"statfs":{"total":95995035648,"available":95387279360,"internally_reserved":0,"allocated":290816000,"data_stored":290058650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.876,"15min":1.829},"min":{"1min":0.47899999999999998,"5min":0.255,"15min":0.222},"max":{"1min":2.153,"5min":6.1859999999999999,"15min":177.68899999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.875,"5min":1.008,"15min":1.925},"min":{"1min":0.42699999999999999,"5min":0.26100000000000001,"15min":0.216},"max":{"1min":1.831,"5min":6.3099999999999996,"15min":178.26900000000001},"last":1.0009999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.139,"15min":1.4119999999999999},"min":{"1min":0.49399999999999999,"5min":0.27200000000000002,"15min":0.223},"max":{"1min":2.2360000000000002,"5min":6.7839999999999998,"15min":70.944000000000003},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.022,"5min":1.256,"15min":1.468},"min":{"1min":0.47299999999999998,"5min":0.27000000000000002,"15min":0.22600000000000001},"max":{"1min":2.532,"5min":6.806,"15min":71.203999999999994},"last":1.5509999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":2.4969999999999999,"15min":1.7010000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":2.8519999999999999,"5min":80.436999999999998,"15min":80.436999999999998},"last":3.4279999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":2.6110000000000002,"15min":1.5900000000000001},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":3.7919999999999998,"5min":77.286000000000001,"15min":77.286000000000001},"last":0.90500000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90700000000000003,"15min":1.0169999999999999},"min":{"1min":0.64600000000000002,"5min":0.37,"15min":0.316},"max":{"1min":1.9990000000000001,"5min":7.2859999999999996,"15min":9.657},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.234,"5min":1.153,"15min":1.0529999999999999},"min":{"1min":0.78400000000000003,"5min":0.317,"15min":0.31},"max":{"1min":2.4780000000000002,"5min":5.0140000000000002,"15min":9.5530000000000008},"last":1.302}]},{"osd":5,"last update":"Mon Feb 12 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.0940000000000001,"15min":3.2080000000000002},"min":{"1min":0.72599999999999998,"5min":0.315,"15min":0.315},"max":{"1min":2.903,"5min":5.048,"15min":616.34199999999998},"last":1.333},{"interface":"front","average":{"1min":1.131,"5min":0.98899999999999999,"15min":3.2719999999999998},"min":{"1min":0.503,"5min":0.35599999999999998,"15min":0.33900000000000002},"max":{"1min":3.3239999999999998,"5min":6.5730000000000004,"15min":615.18899999999996},"last":1.244}]},{"osd":6,"last update":"Mon Feb 12 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.127,"15min":1.0569999999999999},"min":{"1min":0.74299999999999999,"5min":0.36499999999999999,"15min":0.35299999999999998},"max":{"1min":2.6459999999999999,"5min":6.5049999999999999,"15min":9.6150000000000002},"last":1.46},{"interface":"front","average":{"1min":1.349,"5min":1.22,"15min":1.0900000000000001},"min":{"1min":0.72899999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.9700000000000002,"5min":5.1529999999999996,"15min":6.3979999999999997},"last":1.581}]},{"osd":7,"last update":"Mon Feb 12 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.98799999999999999,"15min":1.1799999999999999},"min":{"1min":0.85199999999999998,"5min":0.378,"15min":0.377},"max":{"1min":3.5880000000000001,"5min":4.3369999999999997,"15min":18.405999999999999},"last":1.371},{"interface":"front","average":{"1min":1.423,"5min":1.2569999999999999,"15min":1.2929999999999999},"min":{"1min":0.77700000000000002,"5min":0.439,"15min":0.38700000000000001},"max":{"1min":3.0169999999999999,"5min":5.0739999999999998,"15min":18.388000000000002},"last":1.486}]},{"osd":8,"last update":"Mon Feb 12 16:19:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.212,"15min":1.1830000000000001},"min":{"1min":1.02,"5min":0.42099999999999999,"15min":0.38200000000000001},"max":{"1min":3.0630000000000002,"5min":5.101,"15min":6.4960000000000004},"last":1.875},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.2589999999999999,"15min":1.2929999999999999},"min":{"1min":0.60399999999999998,"5min":0.47799999999999998,"15min":0.432},"max":{"1min":2.7480000000000002,"5min":5.1180000000000003,"15min":8.5920000000000005},"last":1.657}]},{"osd":9,"last update":"Mon Feb 12 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.2649999999999999,"15min":1.177},"min":{"1min":0.88900000000000001,"5min":0.47099999999999997,"15min":0.40600000000000003},"max":{"1min":3.7400000000000002,"5min":5.2329999999999997,"15min":9.6460000000000008},"last":1.446},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.401,"15min":1.2250000000000001},"min":{"1min":1.0349999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":3.2679999999999998,"5min":5.085,"15min":6.4699999999999998},"last":1.8520000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.337,"15min":1.2370000000000001},"min":{"1min":0.81000000000000005,"5min":0.504,"15min":0.442},"max":{"1min":2.5870000000000002,"5min":5.1890000000000001,"15min":9.6760000000000002},"last":1.756},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.4390000000000001,"15min":1.2629999999999999},"min":{"1min":0.85699999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.7069999999999999,"5min":7.2699999999999996,"15min":7.2699999999999996},"last":1.9370000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:20:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.579,"15min":1.571},"min":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"max":{"1min":2.161,"5min":3.2010000000000001,"15min":3.2010000000000001},"last":1.677},{"interface":"front","average":{"1min":1.601,"5min":1.631,"15min":1.635},"min":{"1min":1.054,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":2.371,"5min":3.117,"15min":3.117},"last":1.776}]}]},{"osd":3,"up_from":26,"seq":111669149995,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":939908,"kb_used_data":642684,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92805244,"statfs":{"total":95995035648,"available":95032569856,"internally_reserved":0,"allocated":658108416,"data_stored":657754657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:16:08 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.86499999999999999,"15min":0.94999999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":2.7429999999999999,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.92900000000000005,"15min":0.879},"min":{"1min":0.27300000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.4569999999999999,"5min":6.6680000000000001,"15min":6.6680000000000001},"last":0.755}]},{"osd":1,"last update":"Mon Feb 12 16:17:13 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.7220000000000004,"5min":6.7220000000000004,"15min":6.7220000000000004},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":8.0030000000000001,"5min":8.0030000000000001,"15min":8.0030000000000001},"last":0.93200000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:17:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":0.93700000000000006,"15min":1.28},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23200000000000001},"max":{"1min":5.4550000000000001,"5min":5.7320000000000002,"15min":68.405000000000001},"last":1.004},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.0169999999999999,"15min":1.29},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.9649999999999999,"5min":7.9649999999999999,"15min":69.009},"last":0.83999999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:17:31 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":0.98399999999999999,"15min":0.93700000000000006},"min":{"1min":0.308,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.5609999999999999,"5min":6.5609999999999999,"15min":6.5609999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.88700000000000001,"15min":0.90400000000000003},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":4.7619999999999996,"5min":4.7619999999999996,"15min":5.9870000000000001},"last":1.0649999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.4470000000000001,"15min":1.1459999999999999},"min":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.313},"max":{"1min":4.0670000000000002,"5min":22.751000000000001,"15min":22.751000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.2,"15min":1.0720000000000001},"min":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32200000000000001},"max":{"1min":4.8040000000000003,"5min":22.527999999999999,"15min":22.527999999999999},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:17:39 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.0660000000000001,"15min":1.0009999999999999},"min":{"1min":0.36599999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6260000000000003,"5min":4.6260000000000003,"15min":6.0289999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0309999999999999,"15min":1.141},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.5549999999999997,"5min":4.5549999999999997,"15min":9.5990000000000002},"last":1.2}]},{"osd":7,"last update":"Mon Feb 12 16:17:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2290000000000001,"15min":1.47},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.4119999999999999,"5min":10.061,"15min":91.718999999999994},"last":1.254},{"interface":"front","average":{"1min":1.571,"5min":1.1180000000000001,"15min":1.554},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.5949999999999998,"5min":5.0869999999999997,"15min":90.367999999999995},"last":0.88800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.1319999999999999,"15min":1.1990000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.317},"max":{"1min":5.1500000000000004,"5min":6.0970000000000004,"15min":6.5899999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.097,"15min":1.1559999999999999},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.7039999999999997,"5min":6.2430000000000003,"15min":8.516},"last":1.089}]},{"osd":9,"last update":"Mon Feb 12 16:17:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.107,"15min":1.131},"min":{"1min":0.40799999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.8739999999999997,"5min":4.8739999999999997,"15min":6.2699999999999996},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.2130000000000001,"15min":1.2869999999999999},"min":{"1min":0.45800000000000002,"5min":0.45300000000000001,"15min":0.38200000000000001},"max":{"1min":4.6619999999999999,"5min":4.6619999999999999,"15min":9.6509999999999998},"last":1.0489999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:17:11 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.044,"15min":1.212},"min":{"1min":0.42899999999999999,"5min":0.42699999999999999,"15min":0.41099999999999998},"max":{"1min":4.8399999999999999,"5min":6.4199999999999999,"15min":6.4199999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.101,"15min":1.22},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":4.9630000000000001,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":1.389}]},{"osd":11,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.5680000000000001,"15min":1.5860000000000001},"min":{"1min":0.505,"5min":0.47699999999999998,"15min":0.39100000000000001},"max":{"1min":6.0110000000000001,"5min":26.190000000000001,"15min":49.167000000000002},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.607,"15min":1.5960000000000001},"min":{"1min":0.59299999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":7.4059999999999997,"5min":26.202999999999999,"15min":48.548999999999999},"last":1.163}]}]},{"osd":4,"up_from":32,"seq":137438953803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716652,"kb_used_data":419428,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93028500,"statfs":{"total":95995035648,"available":95261184000,"internally_reserved":0,"allocated":429494272,"data_stored":428732850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.84299999999999997,"15min":1.3879999999999999},"min":{"1min":0.55900000000000005,"5min":0.26500000000000001,"15min":0.25700000000000001},"max":{"1min":3.7850000000000001,"5min":3.992,"15min":96.052000000000007},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.042,"15min":1.5249999999999999},"min":{"1min":0.53000000000000003,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":3.7400000000000002,"5min":4.8979999999999997,"15min":96.307000000000002},"last":0.81999999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1080000000000001,"15min":1.631},"min":{"1min":0.68000000000000005,"5min":0.311,"15min":0.29199999999999998},"max":{"1min":4.1029999999999998,"5min":7.6319999999999997,"15min":90.597999999999999},"last":0.996},{"interface":"front","average":{"1min":1.278,"5min":1.3260000000000001,"15min":1.677},"min":{"1min":0.66000000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.3079999999999998,"5min":8.4299999999999997,"15min":90.676000000000002},"last":1.397}]},{"osd":2,"last update":"Mon Feb 12 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.038,"15min":1.1160000000000001},"min":{"1min":0.60499999999999998,"5min":0.311,"15min":0.27700000000000002},"max":{"1min":3.8380000000000001,"5min":8.0530000000000008,"15min":41.546999999999997},"last":1.091},{"interface":"front","average":{"1min":1.23,"5min":1.2330000000000001,"15min":1.2829999999999999},"min":{"1min":0.47299999999999998,"5min":0.35099999999999998,"15min":0.33000000000000002},"max":{"1min":4.2169999999999996,"5min":7.452,"15min":41.57},"last":1.2909999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.2969999999999999,"15min":1.4219999999999999},"min":{"1min":0.36799999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.7949999999999999,"5min":28.434000000000001,"15min":82.781000000000006},"last":4.3259999999999996},{"interface":"front","average":{"1min":1.103,"5min":1.5469999999999999,"15min":1.456},"min":{"1min":0.45700000000000002,"5min":0.35599999999999998,"15min":0.28599999999999998},"max":{"1min":4.7240000000000002,"5min":27.887,"15min":83.349000000000004},"last":3.3999999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":0.996,"15min":0.90000000000000002},"min":{"1min":0.60899999999999999,"5min":0.35199999999999998,"15min":0.308},"max":{"1min":4.1580000000000004,"5min":4.3719999999999999,"15min":6.0460000000000003},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.1299999999999999,"15min":1.0229999999999999},"min":{"1min":0.70499999999999996,"5min":0.40300000000000002,"15min":0.31900000000000001},"max":{"1min":4.2439999999999998,"5min":4.4109999999999996,"15min":7.1070000000000002},"last":1.325}]},{"osd":6,"last update":"Mon Feb 12 16:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.1639999999999999,"15min":1.03},"min":{"1min":0.67900000000000005,"5min":0.41799999999999998,"15min":0.29899999999999999},"max":{"1min":4.3799999999999999,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":1.204},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1080000000000001,"15min":0.99099999999999999},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.27000000000000002},"max":{"1min":3.988,"5min":8.0139999999999993,"15min":8.0139999999999993},"last":1.2250000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2829999999999999,"15min":1.579},"min":{"1min":0.57299999999999995,"5min":0.40500000000000003,"15min":0.35699999999999998},"max":{"1min":4.0469999999999997,"5min":8.2070000000000007,"15min":116.273},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.3919999999999999,"15min":1.6579999999999999},"min":{"1min":0.77800000000000002,"5min":0.47199999999999998,"15min":0.433},"max":{"1min":4.4580000000000002,"5min":7.6689999999999996,"15min":116.24299999999999},"last":1.482}]},{"osd":8,"last update":"Mon Feb 12 16:20:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6359999999999999,"5min":1.3049999999999999,"15min":1.228},"min":{"1min":0.97899999999999998,"5min":0.49299999999999999,"15min":0.442},"max":{"1min":4.5279999999999996,"5min":7.8650000000000002,"15min":8.4190000000000005},"last":1.571},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.2809999999999999,"15min":1.1579999999999999},"min":{"1min":0.80700000000000005,"5min":0.443,"15min":0.42899999999999999},"max":{"1min":4.6299999999999999,"5min":4.6369999999999996,"15min":7.0380000000000003},"last":1.661}]},{"osd":9,"last update":"Mon Feb 12 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.7370000000000001,"15min":2.492},"min":{"1min":0.76100000000000001,"5min":0.504,"15min":0.45500000000000002},"max":{"1min":4.5709999999999997,"5min":34.112000000000002,"15min":297.55900000000003},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.696,"5min":2.0419999999999998,"15min":2.5990000000000002},"min":{"1min":0.96099999999999997,"5min":0.53900000000000003,"15min":0.49299999999999999},"max":{"1min":4.5899999999999999,"5min":34.590000000000003,"15min":297.61900000000003},"last":1.645}]},{"osd":10,"last update":"Mon Feb 12 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.5329999999999999,"15min":1.272},"min":{"1min":0.73499999999999999,"5min":0.51000000000000001,"15min":0.48699999999999999},"max":{"1min":4.4859999999999998,"5min":6.0899999999999999,"15min":6.9770000000000003},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.4019999999999999,"15min":1.282},"min":{"1min":0.745,"5min":0.50700000000000001,"15min":0.47599999999999998},"max":{"1min":4.3550000000000004,"5min":8.5299999999999994,"15min":8.5299999999999994},"last":1.3740000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:20:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":1.6910000000000001,"15min":1.6830000000000001},"min":{"1min":1.0760000000000001,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"last":1.706},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.6020000000000001,"15min":1.5680000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":4.4089999999999998,"5min":4.4089999999999998,"15min":4.4089999999999998},"last":1.724}]}]},{"osd":5,"up_from":39,"seq":167503724866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582440,"kb_used_data":277024,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93162712,"statfs":{"total":95995035648,"available":95398617088,"internally_reserved":0,"allocated":283672576,"data_stored":282915226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.78100000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.2769999999999999,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.996,"5min":0.97299999999999998,"15min":0.96899999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.415,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.72899999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":4.8899999999999997,"15min":2.9510000000000001},"min":{"1min":0.72699999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.9279999999999999,"5min":565.34500000000003,"15min":565.34500000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.1619999999999999,"5min":4.9630000000000001,"15min":2.911},"min":{"1min":0.83399999999999996,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":1.595,"5min":565.98400000000004,"15min":565.98400000000004},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":0.99399999999999999,"15min":1.1830000000000001},"min":{"1min":0.69499999999999995,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.788,"5min":5.7080000000000002,"15min":26.797000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.1200000000000001,"15min":1.1319999999999999},"min":{"1min":0.48799999999999999,"5min":0.33500000000000002,"15min":0.28599999999999998},"max":{"1min":1.599,"5min":10.140000000000001,"15min":26.905999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":3.7029999999999998,"15min":2.7360000000000002},"min":{"1min":0.30499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.5830000000000002,"5min":252.57300000000001,"15min":252.57300000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.345,"5min":3.972,"15min":2.7480000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.548,"5min":252.24199999999999,"15min":252.24199999999999},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.0669999999999999,"15min":1.2989999999999999},"min":{"1min":0.56299999999999994,"5min":0.31900000000000001,"15min":0.27300000000000002},"max":{"1min":1.571,"5min":5.6779999999999999,"15min":75.257000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.1379999999999999,"5min":0.96499999999999997,"15min":1.3009999999999999},"min":{"1min":0.66700000000000004,"5min":0.35199999999999998,"15min":0.32600000000000001},"max":{"1min":1.823,"5min":5.5949999999999998,"15min":74.427000000000007},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:19:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.93700000000000006,"15min":1.3280000000000001},"min":{"1min":0.61799999999999999,"5min":0.35299999999999998,"15min":0.318},"max":{"1min":1.7889999999999999,"5min":3.1840000000000002,"15min":74.105000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.004,"15min":1.3879999999999999},"min":{"1min":0.76300000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.8939999999999999,"5min":3.1030000000000002,"15min":75.013999999999996},"last":0.56299999999999994}]},{"osd":7,"last update":"Mon Feb 12 16:20:22 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.091,"15min":1.4930000000000001},"min":{"1min":0.82399999999999995,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.728,"5min":6.6630000000000003,"15min":98.971000000000004},"last":0.751},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.1220000000000001,"15min":1.5289999999999999},"min":{"1min":0.52100000000000002,"5min":0.40799999999999997,"15min":0.40600000000000003},"max":{"1min":1.744,"5min":4.6379999999999999,"15min":99.218000000000004},"last":0.63800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.2889999999999999,"15min":1.214},"min":{"1min":0.96799999999999997,"5min":0.40200000000000002,"15min":0.35099999999999998},"max":{"1min":2.5590000000000002,"5min":5.9509999999999996,"15min":14.372999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.254,"15min":1.212},"min":{"1min":0.84799999999999998,"5min":0.54900000000000004,"15min":0.40500000000000003},"max":{"1min":2.1230000000000002,"5min":4.2130000000000001,"15min":14.074999999999999},"last":0.75900000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:19:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.272,"15min":3.2010000000000001},"min":{"1min":1.0640000000000001,"5min":0.439,"15min":0.42999999999999999},"max":{"1min":2.331,"5min":5.4909999999999997,"15min":449.89499999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.577,"5min":1.258,"15min":3.2170000000000001},"min":{"1min":0.95599999999999996,"5min":0.53100000000000003,"15min":0.45300000000000001},"max":{"1min":2.3940000000000001,"5min":9.3659999999999997,"15min":448.70800000000003},"last":0.67100000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:20:22 2024","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.4179999999999999,"15min":1.296},"min":{"1min":1.228,"5min":0.52000000000000002,"15min":0.47499999999999998},"max":{"1min":2.2690000000000001,"5min":5.9210000000000003,"15min":14.214},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.593,"5min":1.2949999999999999,"15min":1.288},"min":{"1min":1.0620000000000001,"5min":0.51000000000000001,"15min":0.5},"max":{"1min":2.1970000000000001,"5min":3.069,"15min":13.464},"last":0.874}]},{"osd":11,"last update":"Mon Feb 12 16:20:32 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.4419999999999999,"15min":1.0289999999999999},"min":{"1min":1.147,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.944,"5min":5.976,"15min":5.976},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.536,"5min":1.421,"15min":1.1850000000000001},"min":{"1min":0.998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.0920000000000001,"5min":4.5129999999999999,"15min":4.5129999999999999},"last":0.91100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284491776,"data_stored":284491776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":634781696,"data_stored":634781696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289275904,"data_stored":289275904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":649134080,"data_stored":649134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":427954176,"data_stored":427950104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":282132480,"data_stored":282132480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":297271296,"data_stored":297271296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":430768128,"data_stored":430768128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":422711296,"data_stored":422707224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":580780032,"data_stored":580780032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":706740224,"data_stored":706740224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:20:57.988 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [9, 0] 2024-02-12T16:20:57.989 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '9', '0'] 2024-02-12T16:20:57.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 1.0 9 0 2024-02-12T16:20:58.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:58.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:58.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:58.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:58.210 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:20:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:58.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:20:58.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:57 smithi169 bash[19604]: cluster 2024-02-12T16:20:56.407315+0000 osd.2 (osd.2) 54 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:20:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:57 smithi169 bash[19604]: cluster 2024-02-12T16:20:56.896056+0000 osd.2 (osd.2) 55 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:20:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:57 smithi169 bash[19604]: audit 2024-02-12T16:20:56.899041+0000 mgr.y (mgr.14146) 1071 : audit [DBG] from='client.15843 v1:172.21.15.121:0/2199125460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:20:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:57 smithi169 bash[19604]: cluster 2024-02-12T16:20:57.522145+0000 mon.a (mon.0) 1756 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:57 smithi169 bash[19604]: audit 2024-02-12T16:20:57.530122+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? v1:172.21.15.121:0/4236567214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:20:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:57 smithi169 bash[19604]: cluster 2024-02-12T16:20:57.530460+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e480: 12 total, 11 up, 12 in 2024-02-12T16:20:58.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:58.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:57 smithi123 bash[19119]: cluster 2024-02-12T16:20:56.407315+0000 osd.2 (osd.2) 54 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:20:58.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:57 smithi123 bash[19119]: cluster 2024-02-12T16:20:56.896056+0000 osd.2 (osd.2) 55 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:20:58.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:57 smithi123 bash[19119]: audit 2024-02-12T16:20:56.899041+0000 mgr.y (mgr.14146) 1071 : audit [DBG] from='client.15843 v1:172.21.15.121:0/2199125460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:20:58.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:57 smithi123 bash[19119]: cluster 2024-02-12T16:20:57.522145+0000 mon.a (mon.0) 1756 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:58.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:57 smithi123 bash[19119]: audit 2024-02-12T16:20:57.530122+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? v1:172.21.15.121:0/4236567214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:20:58.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:57 smithi123 bash[19119]: cluster 2024-02-12T16:20:57.530460+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e480: 12 total, 11 up, 12 in 2024-02-12T16:20:58.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:58.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:57 smithi121 bash[18489]: cluster 2024-02-12T16:20:56.407315+0000 osd.2 (osd.2) 54 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:20:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:57 smithi121 bash[18489]: cluster 2024-02-12T16:20:56.896056+0000 osd.2 (osd.2) 55 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:20:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:57 smithi121 bash[18489]: audit 2024-02-12T16:20:56.899041+0000 mgr.y (mgr.14146) 1071 : audit [DBG] from='client.15843 v1:172.21.15.121:0/2199125460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:20:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:57 smithi121 bash[18489]: cluster 2024-02-12T16:20:57.522145+0000 mon.a (mon.0) 1756 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:20:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:57 smithi121 bash[18489]: audit 2024-02-12T16:20:57.530122+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? v1:172.21.15.121:0/4236567214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:20:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:57 smithi121 bash[18489]: cluster 2024-02-12T16:20:57.530460+0000 mon.a (mon.0) 1758 : cluster [DBG] osdmap e480: 12 total, 11 up, 12 in 2024-02-12T16:20:58.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:58.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:58.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:58.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:58.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:58.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:58.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:58.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:58.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:20:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:58.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:20:58.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:20:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:58.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:20:59.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:59.028 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:20:58 smithi123 bash[19119]: cluster 2024-02-12T16:20:57.842288+0000 mgr.y (mgr.14146) 1072 : cluster [DBG] pgmap v1450: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:59.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:20:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:59.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:20:59.127 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:20:59.127 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:20:59.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:20:58 smithi121 bash[18489]: cluster 2024-02-12T16:20:57.842288+0000 mgr.y (mgr.14146) 1072 : cluster [DBG] pgmap v1450: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:59.192 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:20:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:59.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:20:59.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:20:58 smithi169 bash[19604]: cluster 2024-02-12T16:20:57.842288+0000 mgr.y (mgr.14146) 1072 : cluster [DBG] pgmap v1450: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:20:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:20:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:59.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:20:59.431 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:59.466 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:20:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:59.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:20:59.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:20:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:59.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:20:59.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:59.749 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:59.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:59.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:20:59.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:59.889 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:20:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:20:59.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:20:59.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:00.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:00.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:00.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:00.169 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:00.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:00.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:00.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:00.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:00.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:00.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:21:00.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:00.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:00.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:00.595 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:00.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:00.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:00.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:00.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:00.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:00.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:00.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:00.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:01.018 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:01.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:01.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:00 smithi123 bash[19119]: cluster 2024-02-12T16:20:59.843451+0000 mgr.y (mgr.14146) 1073 : cluster [DBG] pgmap v1451: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:01.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:00 smithi123 bash[19119]: cluster 2024-02-12T16:21:00.526802+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e481: 12 total, 11 up, 12 in 2024-02-12T16:21:01.158 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:01.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:01.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:01.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:00 smithi169 bash[19604]: cluster 2024-02-12T16:20:59.843451+0000 mgr.y (mgr.14146) 1073 : cluster [DBG] pgmap v1451: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:01.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:00 smithi169 bash[19604]: cluster 2024-02-12T16:21:00.526802+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e481: 12 total, 11 up, 12 in 2024-02-12T16:21:01.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:01.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:01.404 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:00 smithi121 bash[18489]: cluster 2024-02-12T16:20:59.843451+0000 mgr.y (mgr.14146) 1073 : cluster [DBG] pgmap v1451: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:00 smithi121 bash[18489]: cluster 2024-02-12T16:21:00.526802+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e481: 12 total, 11 up, 12 in 2024-02-12T16:21:01.442 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:01.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:01.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:01.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:01.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:01.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:01.728 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:01.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:01.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:01.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:01.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:01.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:02.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:02.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:02.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:02.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:02.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:02.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:02.282 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:02.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:02.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:02.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:02.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:02.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:02.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:02.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:02.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:02.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:02.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:02.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:02.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:02.853 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:02.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:02.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:03.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:03.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:03.255 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:02 smithi121 bash[18489]: cluster 2024-02-12T16:21:01.844415+0000 mgr.y (mgr.14146) 1074 : cluster [DBG] pgmap v1453: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:03.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:02 smithi169 bash[19604]: cluster 2024-02-12T16:21:01.844415+0000 mgr.y (mgr.14146) 1074 : cluster [DBG] pgmap v1453: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:03.382 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:03.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:02 smithi123 bash[19119]: cluster 2024-02-12T16:21:01.844415+0000 mgr.y (mgr.14146) 1074 : cluster [DBG] pgmap v1453: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:03.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:03.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:03.566 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:03.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:03.708 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:03.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:03.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:03.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:03.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:03.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:04.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:04.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:04.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:04.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:04.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:04.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:04.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:04.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:04.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:04.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:04.596 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:21:04.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:04.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:04.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:04.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:04.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:04.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:04.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:05.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:05.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:05.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:05.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:04 smithi169 bash[19604]: cluster 2024-02-12T16:21:03.845465+0000 mgr.y (mgr.14146) 1075 : cluster [DBG] pgmap v1454: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:05.155 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:05.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:05.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:05.271 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:04 smithi123 bash[19119]: cluster 2024-02-12T16:21:03.845465+0000 mgr.y (mgr.14146) 1075 : cluster [DBG] pgmap v1454: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:05.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:05.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:05.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:04 smithi121 bash[18489]: cluster 2024-02-12T16:21:03.845465+0000 mgr.y (mgr.14146) 1075 : cluster [DBG] pgmap v1454: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:05.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:05.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:05.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:05.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:05.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:05.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:05.713 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:05.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:05.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:05.853 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:05.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:05.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:05.998 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:06.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:06.134 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:06.136 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:06.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:06.277 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:06.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:06.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:06.522 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:06.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:06.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:06.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:06.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:06 smithi169 bash[19604]: cluster 2024-02-12T16:21:05.846587+0000 mgr.y (mgr.14146) 1076 : cluster [DBG] pgmap v1455: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:06.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:06.814 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:06 smithi123 bash[19119]: cluster 2024-02-12T16:21:05.846587+0000 mgr.y (mgr.14146) 1076 : cluster [DBG] pgmap v1455: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:06.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:06.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:06 smithi121 bash[18489]: cluster 2024-02-12T16:21:05.846587+0000 mgr.y (mgr.14146) 1076 : cluster [DBG] pgmap v1455: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:06.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:06.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:06.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:06.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:07.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:07.118 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:07.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:07.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:07.258 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:07.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:07.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:07.400 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:07.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:07.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:07.539 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:07.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:07.562 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:21:07.581 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:21:07.602 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:21:07.625 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:21:07.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:07.646 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:21:07.655 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:07 smithi169 bash[19604]: audit 2024-02-12T16:21:07.535806+0000 mon.b (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.121:0/3025620948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:07.668 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:21:07.680 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:07.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:07.699 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:21:07.721 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:21:07.747 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:21:07.774 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:21:07.784 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:07.800 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:21:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:21:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:164679 4.0 e367/334 4:0decb776:::benchmark_data_smithi169_60938_object164678:head v 367'10371, mlcod=367'10368)", 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:20.847931+0000", 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 346.942779814, 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030348832999999999, 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.847931+0000", 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.847931+0000", 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.847929+0000", 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3544000000000001e-05 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.847983+0000", 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0471e-05 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.847993+0000", 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9889999999999997e-06 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.847999+0000", 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8075000000000003e-05 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.848047+0000", 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7560000000000001e-05 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.848065+0000", 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030173258000000001 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.878238+0000", 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1638999999999999e-05 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:20.878280+0000", 2024-02-12T16:21:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030348832999999999 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:169559 4.0 e369/334 4:0419674c:::benchmark_data_smithi169_60938_object169558:head v 369'10691, mlcod=368'10689)", 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:24.797536+0000", 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 342.99317502700001, 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.042635737, 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797536+0000", 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797536+0000", 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797534+0000", 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5090000000000001e-05 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797609+0000", 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3182e-05 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797622+0000", 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6089999999999997e-06 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797629+0000", 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8887000000000001e-05 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797678+0000", 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4433999999999999e-05 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.797692+0000", 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.042437760999999997 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.840130+0000", 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1556e-05 2024-02-12T16:21:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.840172+0000", 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.042635737 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:169558 4.8 e369/334 4:15b51327:::benchmark_data_smithi169_60938_object169557:head v 369'10515, mlcod=368'10512)", 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:24.801051+0000", 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 342.98965976400001, 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.056834783, 2024-02-12T16:21:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801051+0000", 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801051+0000", 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801050+0000", 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3900000000000003e-05 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801094+0000", 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8009999999999993e-06 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801103+0000", 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3759999999999996e-06 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801109+0000", 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6235999999999997e-05 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801145+0000", 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1919999999999999e-05 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.801157+0000", 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.056677204000000002 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.857834+0000", 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1542999999999998e-05 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.857886+0000", 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.056834783 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:169560 4.0 e369/334 4:0b835461:::benchmark_data_smithi169_60938_object169559:head v 369'10692, mlcod=368'10689)", 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:24.806848+0000", 2024-02-12T16:21:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 342.98386306200001, 2024-02-12T16:21:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.033346404000000003, 2024-02-12T16:21:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.806848+0000", 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.806848+0000", 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:21:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.806845+0000", 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9922000000000001e-05 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.806895+0000", 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9970999999999999e-05 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.806915+0000", 2024-02-12T16:21:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4679999999999992e-06 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.806924+0000", 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0516999999999998e-05 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.806974+0000", 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6225999999999999e-05 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.807000+0000", 2024-02-12T16:21:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.033176337 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.840177+0000", 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7557000000000001e-05 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.840194+0000", 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.033346404000000003 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:169572 4.0 e369/334 4:09aa58a5:::benchmark_data_smithi169_60938_object169571:head v 369'10693, mlcod=368'10689)", 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:24.814327+0000", 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 342.976383666, 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.051193874, 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.814327+0000", 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.814327+0000", 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.814325+0000", 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2273000000000003e-05 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.814408+0000", 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5628e-05 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.814423+0000", 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0990000000000001e-06 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.814430+0000", 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030444970000000002 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.817475+0000", 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5403000000000001e-05 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.817500+0000", 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.047978789000000001 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.865479+0000", 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1972999999999999e-05 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.865521+0000", 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.051193874 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:169583 4.3 e369/334 4:cf135ef2:::benchmark_data_smithi169_60938_object169582:head v 369'10541, mlcod=369'10539)", 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:24.826453+0000", 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 342.96425785600002, 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031430804, 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826453+0000", 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826453+0000", 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999914 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826444+0000", 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5395999999999997e-05 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826490+0000", 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7436999999999999e-05 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826507+0000", 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.516e-06 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826514+0000", 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7259999999999999e-05 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826551+0000", 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7229000000000001e-05 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.826568+0000", 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031266065000000003 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.857834+0000", 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9486e-05 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:24.857884+0000", 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031430804 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:197166 4.0 e376/334 4:0135a872:::benchmark_data_smithi169_60938_object23829:head v 376'12386, mlcod=376'12381)", 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:49.974002+0000", 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 317.816708656, 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034700896000000002, 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.974002+0000", 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.974002+0000", 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.974000+0000", 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1070000000000001e-05 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.974012+0000", 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7109999999999995e-06 2024-02-12T16:21:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.974019+0000", 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8420000000000004e-06 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.974024+0000", 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034162400000000001 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.977440+0000", 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3133999999999999e-05 2024-02-12T16:21:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.977463+0000", 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031199042999999999 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:50.008663+0000", 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0531000000000002e-05 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:50.008703+0000", 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034700896000000002 2024-02-12T16:21:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:197167 4.0 e376/334 4:02940e09:::benchmark_data_smithi169_60938_object23830:head v 376'12387, mlcod=376'12382)", 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:49.978255+0000", 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 317.81245602000001, 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030471422000000001, 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.978255+0000", 2024-02-12T16:21:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.978255+0000", 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.978253+0000", 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0454e-05 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.978264+0000", 2024-02-12T16:21:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5163e-05 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.978279+0000", 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5690000000000001e-06 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.978286+0000", 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046833639999999998 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.982969+0000", 2024-02-12T16:21:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1622000000000001e-05 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:49.982991+0000", 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025716276 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:50.008707+0000", 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9388000000000001e-05 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:50.008726+0000", 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030471422000000001 2024-02-12T16:21:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:199104 4.0 e377/334 4:0c270b08:::benchmark_data_smithi169_60938_object25767:head v 377'12512, mlcod=377'12510)", 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:15:51.591287+0000", 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 316.19942409999999, 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032999081, 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591287+0000", 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591287+0000", 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591285+0000", 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5609999999999999e-06 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591294+0000", 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3959999999999995e-06 2024-02-12T16:21:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591303+0000", 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5892999999999998e-05 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591319+0000", 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7576000000000001e-05 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591356+0000", 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6125000000000001e-05 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.591383+0000", 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032871003000000003 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.624254+0000", 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2175999999999998e-05 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:15:51.624286+0000", 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032999081 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:232141 4.0 e385/334 4:0042ef30:::benchmark_data_smithi169_60938_object58804:head v 385'14564, mlcod=385'14558)", 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:23.487108+0000", 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 284.30360325100003, 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030559622000000002, 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.487108+0000", 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.487108+0000", 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.487105+0000", 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.099e-05 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.487116+0000", 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3365e-05 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.487129+0000", 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9379999999999997e-06 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.487135+0000", 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082802750000000001 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.495416+0000", 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3659000000000002e-05 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.495439+0000", 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022202926000000001 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.517642+0000", 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4913999999999999e-05 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:23.517667+0000", 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030559622000000002 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:257879 4.0 e395/334 4:02c31f83:::benchmark_data_smithi169_60938_object84542:head v 395'16165, mlcod=395'16161)", 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:48.139448+0000", 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 259.65126239199998, 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032284475999999999, 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.139448+0000", 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.139448+0000", 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.139447+0000", 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0672999999999999e-05 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.139457+0000", 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1653e-05 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.139469+0000", 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4779999999999999e-06 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.139475+0000", 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062553699999999997 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.145730+0000", 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4292000000000001e-05 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.145754+0000", 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025952066999999999 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.171706+0000", 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6551999999999999e-05 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:48.171733+0000", 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032284475999999999 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259321 4.8 e395/334 4:15cdad88:::benchmark_data_smithi169_60938_object85984:head v 395'16052, mlcod=395'16046)", 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.521483+0000", 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.26922737699999, 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037912745999999997, 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.521483+0000", 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.521483+0000", 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.521482+0000", 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0591000000000001e-05 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.521493+0000", 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1579e-05 2024-02-12T16:21:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.521504+0000", 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9199999999999998e-06 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.521511+0000", 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044436570000000002 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525955+0000", 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7438000000000001e-05 2024-02-12T16:21:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525982+0000", 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.033373933000000001 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.559356+0000", 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0074999999999998e-05 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.559396+0000", 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037912745999999997 2024-02-12T16:21:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259327 4.8 e395/334 4:1e347a32:::benchmark_data_smithi169_60938_object85990:head v 395'16053, mlcod=395'16046)", 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.525177+0000", 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.26553384200002, 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039741809000000003, 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525177+0000", 2024-02-12T16:21:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525177+0000", 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525175+0000", 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2031999999999999e-05 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525187+0000", 2024-02-12T16:21:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5056e-05 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525202+0000", 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6599999999999995e-06 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.525210+0000", 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052739249999999996 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530484+0000", 2024-02-12T16:21:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0814e-05 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530515+0000", 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034362140999999999 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.564877+0000", 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2002000000000002e-05 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.564919+0000", 2024-02-12T16:21:07.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039741809000000003 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259340 4.0 e395/334 4:04e74f39:::benchmark_data_smithi169_60938_object86003:head v 395'16265, mlcod=395'16261)", 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.529656+0000", 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.26105501900003, 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030316773000000002, 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.529656+0000", 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.529656+0000", 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.529654+0000", 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3488e-05 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.529667+0000", 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8022000000000001e-05 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.529685+0000", 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0800000000000002e-06 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.529691+0000", 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077657500000000005 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530468+0000", 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5398000000000001e-05 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530483+0000", 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029446837 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.559930+0000", 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2406e-05 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.559973+0000", 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030316773000000002 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259335 4.8 e395/334 4:15fc1b91:::benchmark_data_smithi169_60938_object85998:head v 395'16054, mlcod=395'16047)", 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.530612+0000", 2024-02-12T16:21:07.859 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.26009920500002, 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034339873999999999, 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530612+0000", 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530612+0000", 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530610+0000", 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.007e-06 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530617+0000", 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0029e-05 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530627+0000", 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5669999999999996e-06 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.530632+0000", 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049087970000000003 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.535541+0000", 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3096e-05 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.535564+0000", 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029361569000000001 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.564925+0000", 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6261000000000001e-05 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.564951+0000", 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034339873999999999 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259336 4.8 e395/334 4:1a127b33:::benchmark_data_smithi169_60938_object85999:head v 395'16055, mlcod=395'16047)", 2024-02-12T16:21:07.863 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.534769+0000", 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.25594228099999, 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030201215, 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.534769+0000", 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.534769+0000", 2024-02-12T16:21:07.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.534767+0000", 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1260000000000001e-05 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.534778+0000", 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3284e-05 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.534791+0000", 2024-02-12T16:21:07.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9660000000000004e-06 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.534796+0000", 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049500459999999996 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539746+0000", 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9259999999999999e-05 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539765+0000", 2024-02-12T16:21:07.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025191582000000001 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.564957+0000", 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2833999999999999e-05 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.564970+0000", 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030201215 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259343 4.8 e395/334 4:1f7d7091:::benchmark_data_smithi169_60938_object86006:head v 395'16056, mlcod=395'16048)", 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.539124+0000", 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.25158651300001, 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.035605278999999997, 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539124+0000", 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539124+0000", 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539122+0000", 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2758000000000001e-05 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539135+0000", 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8362000000000001e-05 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539153+0000", 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8710000000000003e-06 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.539162+0000", 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010646452000000001 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.549808+0000", 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9429999999999999e-05 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.549828+0000", 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024859128000000001 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.574687+0000", 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2577999999999998e-05 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.574730+0000", 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.035605278999999997 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.871 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259344 4.8 e395/334 4:19d086cd:::benchmark_data_smithi169_60938_object86007:head v 395'16057, mlcod=395'16048)", 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.543771+0000", 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.24693957699998, 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030986149000000001, 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.543771+0000", 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.543771+0000", 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.543770+0000", 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0779e-05 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.543780+0000", 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5393e-05 2024-02-12T16:21:07.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.543796+0000", 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7449999999999998e-06 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.543802+0000", 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010537209000000001 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.554340+0000", 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2631999999999999e-05 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.554362+0000", 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020372652000000002 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.574735+0000", 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2405e-05 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.574757+0000", 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030986149000000001 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.875 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259354 4.0 e395/334 4:0d4d2a6e:::benchmark_data_smithi169_60938_object86017:head v 395'16267, mlcod=395'16262)", 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.547942+0000", 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.24276842900002, 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032717481999999999, 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.547942+0000", 2024-02-12T16:21:07.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.547942+0000", 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.547940+0000", 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2476999999999999e-05 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.547952+0000", 2024-02-12T16:21:07.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8844000000000001e-05 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.547971+0000", 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7979999999999995e-06 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.547980+0000", 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063438369999999997 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.554324+0000", 2024-02-12T16:21:07.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3640000000000001e-05 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.554347+0000", 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026273622999999999 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.580621+0000", 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8800999999999997e-05 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.580660+0000", 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032717481999999999 2024-02-12T16:21:07.879 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.25264.0:259350 4.8 e395/334 4:1c8f7479:::benchmark_data_smithi169_60938_object86013:head v 395'16058, mlcod=395'16050)", 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:16:49.548538+0000", 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 258.24217289199999, 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036302951, 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:21:07.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.548538+0000", 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.548538+0000", 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.548536+0000", 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1622e-05 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.548548+0000", 2024-02-12T16:21:07.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1484999999999999e-05 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.548559+0000", 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1350000000000003e-06 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.548567+0000", 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011310420999999999 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.559877+0000", 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3382000000000001e-05 2024-02-12T16:21:07.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.559900+0000", 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024897967999999999 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.584798+0000", 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2481000000000002e-05 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:16:49.584841+0000", 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036302951 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.883 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.884 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.884 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:21:07.884 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:21:07.884 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:21:07.884 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:21:07.885 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:07.885 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.885 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.885 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.885 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:21:07.885 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:21:07.886 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:07 smithi123 bash[19119]: audit 2024-02-12T16:21:07.535806+0000 mon.b (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.121:0/3025620948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:07.886 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:07 smithi121 bash[18489]: audit 2024-02-12T16:21:07.535806+0000 mon.b (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.121:0/3025620948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:07.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:07.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:08.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:08.158 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:08.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:08.298 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:08.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:08.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:08.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-12T16:21:08.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:08.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:08 smithi123 bash[19119]: cluster 2024-02-12T16:21:07.847327+0000 mgr.y (mgr.14146) 1077 : cluster [DBG] pgmap v1456: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:08.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:08 smithi123 bash[19119]: audit 2024-02-12T16:21:08.106830+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? v1:172.21.15.121:0/573693036' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [9, 0]}]: dispatch 2024-02-12T16:21:08.564 INFO:teuthology.orchestra.run.smithi121.stderr:set 1.0 pg_upmap mapping to [9,0] 2024-02-12T16:21:08.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:21:08.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:21:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:08 smithi121 bash[18489]: cluster 2024-02-12T16:21:07.847327+0000 mgr.y (mgr.14146) 1077 : cluster [DBG] pgmap v1456: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:08 smithi121 bash[18489]: audit 2024-02-12T16:21:08.106830+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? v1:172.21.15.121:0/573693036' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [9, 0]}]: dispatch 2024-02-12T16:21:08.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:08.714 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:08 smithi169 bash[19604]: cluster 2024-02-12T16:21:07.847327+0000 mgr.y (mgr.14146) 1077 : cluster [DBG] pgmap v1456: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:08.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:08 smithi169 bash[19604]: audit 2024-02-12T16:21:08.106830+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? v1:172.21.15.121:0/573693036' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [9, 0]}]: dispatch 2024-02-12T16:21:08.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:08.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:08.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:08.994 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:08.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:09.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:09.134 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:09.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:09.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:09.276 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:09.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:09.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:09.418 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:09.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:09.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:09.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:09.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:09.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:09.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:09.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:09.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:09 smithi169 bash[19604]: cluster 2024-02-12T16:21:08.510119+0000 osd.2 (osd.2) 56 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:09 smithi169 bash[19604]: cluster 2024-02-12T16:21:08.510213+0000 osd.2 (osd.2) 57 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:09 smithi169 bash[19604]: audit 2024-02-12T16:21:08.555637+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? v1:172.21.15.121:0/573693036' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [9, 0]}]': finished 2024-02-12T16:21:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:09 smithi169 bash[19604]: cluster 2024-02-12T16:21:08.555717+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e482: 12 total, 11 up, 12 in 2024-02-12T16:21:09.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:09.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:09.839 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:09 smithi121 bash[18489]: cluster 2024-02-12T16:21:08.510119+0000 osd.2 (osd.2) 56 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:09.840 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:09 smithi121 bash[18489]: cluster 2024-02-12T16:21:08.510213+0000 osd.2 (osd.2) 57 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:09.840 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:09 smithi121 bash[18489]: audit 2024-02-12T16:21:08.555637+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? v1:172.21.15.121:0/573693036' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [9, 0]}]': finished 2024-02-12T16:21:09.840 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:09 smithi121 bash[18489]: cluster 2024-02-12T16:21:08.555717+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e482: 12 total, 11 up, 12 in 2024-02-12T16:21:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:09.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:09 smithi123 bash[19119]: cluster 2024-02-12T16:21:08.510119+0000 osd.2 (osd.2) 56 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:09 smithi123 bash[19119]: cluster 2024-02-12T16:21:08.510213+0000 osd.2 (osd.2) 57 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:09.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:09 smithi123 bash[19119]: audit 2024-02-12T16:21:08.555637+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? v1:172.21.15.121:0/573693036' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [9, 0]}]': finished 2024-02-12T16:21:09.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:09 smithi123 bash[19119]: cluster 2024-02-12T16:21:08.555717+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e482: 12 total, 11 up, 12 in 2024-02-12T16:21:09.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:09.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:09.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:10.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:10.118 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:10.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:10.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:10.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:10.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:10.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:10.402 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:10.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:10.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:10.548 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:10.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:10.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:21:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:10 smithi123 bash[19119]: cluster 2024-02-12T16:21:09.559085+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e483: 12 total, 11 up, 12 in 2024-02-12T16:21:10.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:10 smithi123 bash[19119]: cluster 2024-02-12T16:21:09.848047+0000 mgr.y (mgr.14146) 1078 : cluster [DBG] pgmap v1459: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:10.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:10.692 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:10.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:10.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:10.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:10.834 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:10 smithi169 bash[19604]: cluster 2024-02-12T16:21:09.559085+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e483: 12 total, 11 up, 12 in 2024-02-12T16:21:10.835 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:10 smithi169 bash[19604]: cluster 2024-02-12T16:21:09.848047+0000 mgr.y (mgr.14146) 1078 : cluster [DBG] pgmap v1459: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:10.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:10.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:10 smithi121 bash[18489]: cluster 2024-02-12T16:21:09.559085+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e483: 12 total, 11 up, 12 in 2024-02-12T16:21:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:10 smithi121 bash[18489]: cluster 2024-02-12T16:21:09.848047+0000 mgr.y (mgr.14146) 1078 : cluster [DBG] pgmap v1459: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:10.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:10.976 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:10.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:11.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:11.116 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:11.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:11.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:11.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:11.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:11.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:11.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:11.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:11.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:11.539 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:11.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:11.599 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:21:11.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:11.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:11 smithi121 bash[18489]: audit 2024-02-12T16:21:11.012401+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? v1:172.21.15.121:0/2916571835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:21:11.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:11.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:11.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:11.828 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:11.830 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:11 smithi123 bash[19119]: audit 2024-02-12T16:21:11.012401+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? v1:172.21.15.121:0/2916571835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:21:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:11.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:11.932 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:11.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:11.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:12.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:11 smithi169 bash[19604]: audit 2024-02-12T16:21:11.012401+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? v1:172.21.15.121:0/2916571835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:21:12.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:12.110 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:12.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:12.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:12.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:12.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:12.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:12.392 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:12.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:12.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:12.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:12.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:12.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:12.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:12 smithi121 bash[18489]: cluster 2024-02-12T16:21:11.572532+0000 mon.a (mon.0) 1765 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:12.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:12 smithi121 bash[18489]: audit 2024-02-12T16:21:11.588958+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? v1:172.21.15.121:0/2916571835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:21:12.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:12 smithi121 bash[18489]: cluster 2024-02-12T16:21:11.589064+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e484: 12 total, 11 up, 12 in 2024-02-12T16:21:12.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:12 smithi121 bash[18489]: cluster 2024-02-12T16:21:11.848921+0000 mgr.y (mgr.14146) 1079 : cluster [DBG] pgmap v1461: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:12.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:12.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:12.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:12.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:12 smithi123 bash[19119]: cluster 2024-02-12T16:21:11.572532+0000 mon.a (mon.0) 1765 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:12.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:12 smithi123 bash[19119]: audit 2024-02-12T16:21:11.588958+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? v1:172.21.15.121:0/2916571835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:21:12.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:12 smithi123 bash[19119]: cluster 2024-02-12T16:21:11.589064+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e484: 12 total, 11 up, 12 in 2024-02-12T16:21:12.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:12 smithi123 bash[19119]: cluster 2024-02-12T16:21:11.848921+0000 mgr.y (mgr.14146) 1079 : cluster [DBG] pgmap v1461: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:12 smithi169 bash[19604]: cluster 2024-02-12T16:21:11.572532+0000 mon.a (mon.0) 1765 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:12 smithi169 bash[19604]: audit 2024-02-12T16:21:11.588958+0000 mon.a (mon.0) 1766 : audit [INF] from='client.? v1:172.21.15.121:0/2916571835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:21:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:12 smithi169 bash[19604]: cluster 2024-02-12T16:21:11.589064+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e484: 12 total, 11 up, 12 in 2024-02-12T16:21:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:12 smithi169 bash[19604]: cluster 2024-02-12T16:21:11.848921+0000 mgr.y (mgr.14146) 1079 : cluster [DBG] pgmap v1461: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:12.813 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:12.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:12.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:12.956 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:12.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:13.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:13.100 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:13.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:13.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:13.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:13.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:13.294 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:21:13.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:13.377 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:13.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:13.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:13.515 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:13.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:13.619 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:13.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:13.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:13.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:13.790 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:13.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:13.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:13.928 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:13.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:14.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:14.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:14.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:14.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:14.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:14.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:14.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:14.354 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:14.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:14.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:21:14.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:14.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:14.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:14.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:14.639 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:14.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:14.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:14.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:14.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:14.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:14.911 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:21:14.911 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:21:14.911 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-02-12T16:21:14.911 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:21:14.911 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:21:14.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-02-12T16:21:14.917 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:14.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:15.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:15.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:14 smithi121 bash[18489]: cluster 2024-02-12T16:21:13.849977+0000 mgr.y (mgr.14146) 1080 : cluster [DBG] pgmap v1462: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:15.060 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:15.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:15.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:15.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:15.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:14 smithi169 bash[19604]: cluster 2024-02-12T16:21:13.849977+0000 mgr.y (mgr.14146) 1080 : cluster [DBG] pgmap v1462: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:15.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:15.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:15.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:15.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:14 smithi123 bash[19119]: cluster 2024-02-12T16:21:13.849977+0000 mgr.y (mgr.14146) 1080 : cluster [DBG] pgmap v1462: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:15.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:15.480 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:15.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:15.584 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:15.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:15.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:15.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:15.762 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:15.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:15.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:15.903 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:15.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:15.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:16.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:16.041 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:16.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:16.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:16.181 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:16.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:16.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:16.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:16.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:16.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:16.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:16.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:16.569 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:16.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:16 smithi123 bash[19119]: cluster 2024-02-12T16:21:15.535229+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e485: 12 total, 11 up, 12 in 2024-02-12T16:21:16.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:16 smithi123 bash[19119]: cluster 2024-02-12T16:21:15.850930+0000 mgr.y (mgr.14146) 1081 : cluster [DBG] pgmap v1464: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:16.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:16.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:16.744 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:16.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:16.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:16 smithi169 bash[19604]: cluster 2024-02-12T16:21:15.535229+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e485: 12 total, 11 up, 12 in 2024-02-12T16:21:16.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:16 smithi169 bash[19604]: cluster 2024-02-12T16:21:15.850930+0000 mgr.y (mgr.14146) 1081 : cluster [DBG] pgmap v1464: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:16.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:16.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:16.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:16 smithi121 bash[18489]: cluster 2024-02-12T16:21:15.535229+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e485: 12 total, 11 up, 12 in 2024-02-12T16:21:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:16 smithi121 bash[18489]: cluster 2024-02-12T16:21:15.850930+0000 mgr.y (mgr.14146) 1081 : cluster [DBG] pgmap v1464: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:16.910 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:21:16.910 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:21:16.910 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-02-12T16:21:16.910 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:21:16.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:17.025 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:17.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:17.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:17.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:17.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:17.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:17.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:17.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:17.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:17.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:17.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:17.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:17.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-12T16:21:17.575 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:17.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:17.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:17.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:17.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:17.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:17.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:17.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:17.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:18.004 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:18.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:18.145 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:18.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:18.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:18.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:18.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:18.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:18 smithi121 bash[18489]: cluster 2024-02-12T16:21:17.222248+0000 osd.10 (osd.10) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:18 smithi121 bash[18489]: cluster 2024-02-12T16:21:17.222339+0000 osd.10 (osd.10) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:18 smithi121 bash[18489]: cluster 2024-02-12T16:21:17.537311+0000 osd.9 (osd.9) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:18 smithi121 bash[18489]: cluster 2024-02-12T16:21:17.537403+0000 osd.9 (osd.9) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:18 smithi121 bash[18489]: cluster 2024-02-12T16:21:17.550683+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e486: 12 total, 11 up, 12 in 2024-02-12T16:21:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:18 smithi121 bash[18489]: cluster 2024-02-12T16:21:17.851922+0000 mgr.y (mgr.14146) 1082 : cluster [DBG] pgmap v1466: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:18.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:18.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:18.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:18.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:18 smithi123 bash[19119]: cluster 2024-02-12T16:21:17.222248+0000 osd.10 (osd.10) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:18.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:18 smithi123 bash[19119]: cluster 2024-02-12T16:21:17.222339+0000 osd.10 (osd.10) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:18.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:18 smithi123 bash[19119]: cluster 2024-02-12T16:21:17.537311+0000 osd.9 (osd.9) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:18.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:18 smithi123 bash[19119]: cluster 2024-02-12T16:21:17.537403+0000 osd.9 (osd.9) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:18.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:18 smithi123 bash[19119]: cluster 2024-02-12T16:21:17.550683+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e486: 12 total, 11 up, 12 in 2024-02-12T16:21:18.827 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:18 smithi123 bash[19119]: cluster 2024-02-12T16:21:17.851922+0000 mgr.y (mgr.14146) 1082 : cluster [DBG] pgmap v1466: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:18.852 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:18.947 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:18.947 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:18.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:18.996 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:18.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:18 smithi169 bash[19604]: cluster 2024-02-12T16:21:17.222248+0000 osd.10 (osd.10) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:18 smithi169 bash[19604]: cluster 2024-02-12T16:21:17.222339+0000 osd.10 (osd.10) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:18 smithi169 bash[19604]: cluster 2024-02-12T16:21:17.537311+0000 osd.9 (osd.9) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:18 smithi169 bash[19604]: cluster 2024-02-12T16:21:17.537403+0000 osd.9 (osd.9) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:18 smithi169 bash[19604]: cluster 2024-02-12T16:21:17.550683+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e486: 12 total, 11 up, 12 in 2024-02-12T16:21:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:18 smithi169 bash[19604]: cluster 2024-02-12T16:21:17.851922+0000 mgr.y (mgr.14146) 1082 : cluster [DBG] pgmap v1466: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:19.055 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:21:19.077 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:21:19.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:19.106 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:21:19.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:19.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:19.138 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:21:19.168 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:21:19.193 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:21:19.222 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:21:19.232 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:19.254 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:21:19.269 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:19.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:19.276 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:21:19.298 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:21:19.321 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.321 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.322 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:21:19.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:19.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:19.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:19.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:19.553 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:19.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:19.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:19.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:19.712 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:19 smithi121 bash[18489]: audit 2024-02-12T16:21:19.023099+0000 mon.a (mon.0) 1770 : audit [DBG] from='client.? v1:172.21.15.121:0/335924302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:19.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:19 smithi169 bash[19604]: audit 2024-02-12T16:21:19.023099+0000 mon.a (mon.0) 1770 : audit [DBG] from='client.? v1:172.21.15.121:0/335924302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:19.809 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:19 smithi123 bash[19119]: audit 2024-02-12T16:21:19.023099+0000 mon.a (mon.0) 1770 : audit [DBG] from='client.? v1:172.21.15.121:0/335924302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:19.834 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:19.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:19.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:19.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:19.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:20.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:20.127 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:20.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:20.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:20.269 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:20.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:20.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:21:20.393 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:21:20.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:20.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:20.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:20.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:20.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:20 smithi123 bash[19119]: cluster 2024-02-12T16:21:19.852942+0000 mgr.y (mgr.14146) 1083 : cluster [DBG] pgmap v1467: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:20 smithi123 bash[19119]: cluster 2024-02-12T16:21:20.534522+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e487: 12 total, 11 up, 12 in 2024-02-12T16:21:20.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:20.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:20 smithi121 bash[18489]: cluster 2024-02-12T16:21:19.852942+0000 mgr.y (mgr.14146) 1083 : cluster [DBG] pgmap v1467: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:20.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:20 smithi121 bash[18489]: cluster 2024-02-12T16:21:20.534522+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e487: 12 total, 11 up, 12 in 2024-02-12T16:21:20.693 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:20.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:20.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:20.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:20.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:20.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:20.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:20 smithi169 bash[19604]: cluster 2024-02-12T16:21:19.852942+0000 mgr.y (mgr.14146) 1083 : cluster [DBG] pgmap v1467: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:20.950 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:20 smithi169 bash[19604]: cluster 2024-02-12T16:21:20.534522+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e487: 12 total, 11 up, 12 in 2024-02-12T16:21:20.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:20.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:21.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:21.116 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:21.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:21.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:21.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:21.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:21.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:21.396 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:21.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:21.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:21.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:21.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:21.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:21.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:21.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:21.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:21.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:21.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:21.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:21.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:21.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:22.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:22.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:22.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:22.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:22.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:22.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:22.348 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:21:22.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:22.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:22.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:22.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:21:22.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:22.524 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:22.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:22.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:22.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:22.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:22.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:22.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:22.802 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:22.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:22.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:22.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:22.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:23.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:23.062 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:22 smithi123 bash[19119]: cluster 2024-02-12T16:21:21.853803+0000 mgr.y (mgr.14146) 1084 : cluster [DBG] pgmap v1469: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:23.084 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:23.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:23.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:23.223 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:23.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:23.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:22 smithi169 bash[19604]: cluster 2024-02-12T16:21:21.853803+0000 mgr.y (mgr.14146) 1084 : cluster [DBG] pgmap v1469: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:23.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:23.339 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:22 smithi121 bash[18489]: cluster 2024-02-12T16:21:21.853803+0000 mgr.y (mgr.14146) 1084 : cluster [DBG] pgmap v1469: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:23.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:23.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:23.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:23.502 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:23.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:23.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:23.640 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:21:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-12T16:21:23.640 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:21:23.644 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:23.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:23.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:23.788 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:23.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:23.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:23.926 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:23.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:23.934 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:21:23.993 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:23 smithi121 bash[18489]: audit 2024-02-12T16:21:23.862969+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? v1:172.21.15.121:0/2383890244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:21:24.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:24.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:24.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:24.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:24.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:24.302 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:23 smithi169 bash[19604]: audit 2024-02-12T16:21:23.862969+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? v1:172.21.15.121:0/2383890244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:21:24.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:24.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:24.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:24.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:23 smithi123 bash[19119]: audit 2024-02-12T16:21:23.862969+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? v1:172.21.15.121:0/2383890244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:21:24.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:24.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:24.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:24.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:24.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:24.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:24.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:24.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:24.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:24.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:24.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:24.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:25.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:25.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:24 smithi169 bash[19604]: cluster 2024-02-12T16:21:23.854823+0000 mgr.y (mgr.14146) 1085 : cluster [DBG] pgmap v1470: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:24 smithi169 bash[19604]: cluster 2024-02-12T16:21:23.911525+0000 mon.a (mon.0) 1773 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:24 smithi169 bash[19604]: audit 2024-02-12T16:21:23.924652+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? v1:172.21.15.121:0/2383890244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:21:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:24 smithi169 bash[19604]: cluster 2024-02-12T16:21:23.924761+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e488: 12 total, 11 up, 12 in 2024-02-12T16:21:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:24 smithi169 bash[19604]: cluster 2024-02-12T16:21:24.368553+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:21:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:24 smithi169 bash[19604]: cluster 2024-02-12T16:21:24.417538+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:21:25.063 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:25.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:25.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:25.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:25.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:25.326 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:24 smithi123 bash[19119]: cluster 2024-02-12T16:21:23.854823+0000 mgr.y (mgr.14146) 1085 : cluster [DBG] pgmap v1470: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:25.326 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:24 smithi123 bash[19119]: cluster 2024-02-12T16:21:23.911525+0000 mon.a (mon.0) 1773 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:25.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:24 smithi123 bash[19119]: audit 2024-02-12T16:21:23.924652+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? v1:172.21.15.121:0/2383890244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:21:25.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:24 smithi123 bash[19119]: cluster 2024-02-12T16:21:23.924761+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e488: 12 total, 11 up, 12 in 2024-02-12T16:21:25.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:24 smithi123 bash[19119]: cluster 2024-02-12T16:21:24.368553+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:21:25.327 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:24 smithi123 bash[19119]: cluster 2024-02-12T16:21:24.417538+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:21:25.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:25.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:25.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:24 smithi121 bash[18489]: cluster 2024-02-12T16:21:23.854823+0000 mgr.y (mgr.14146) 1085 : cluster [DBG] pgmap v1470: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:24 smithi121 bash[18489]: cluster 2024-02-12T16:21:23.911525+0000 mon.a (mon.0) 1773 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:24 smithi121 bash[18489]: audit 2024-02-12T16:21:23.924652+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? v1:172.21.15.121:0/2383890244' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:21:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:24 smithi121 bash[18489]: cluster 2024-02-12T16:21:23.924761+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e488: 12 total, 11 up, 12 in 2024-02-12T16:21:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:24 smithi121 bash[18489]: cluster 2024-02-12T16:21:24.368553+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:21:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:24 smithi121 bash[18489]: cluster 2024-02-12T16:21:24.417538+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:21:25.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:25.488 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:25.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:25.554 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:21:25.554 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:21:25.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:25.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:25.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:25.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:25.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:25.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:25.898 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:25.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:26.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:26.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:26.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:26.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:26.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:26.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:25 smithi123 bash[19119]: cluster 2024-02-12T16:21:24.378529+0000 osd.6 (osd.6) 82 : cluster [DBG] 4.a scrub starts 2024-02-12T16:21:26.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:25 smithi123 bash[19119]: cluster 2024-02-12T16:21:24.771802+0000 osd.6 (osd.6) 83 : cluster [DBG] 4.a scrub ok 2024-02-12T16:21:26.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:25 smithi123 bash[19119]: cluster 2024-02-12T16:21:25.276696+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.b scrub starts 2024-02-12T16:21:26.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:25 smithi123 bash[19119]: cluster 2024-02-12T16:21:25.537866+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e489: 12 total, 11 up, 12 in 2024-02-12T16:21:26.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:25 smithi123 bash[19119]: cluster 2024-02-12T16:21:25.599359+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.b scrub ok 2024-02-12T16:21:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:26.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:26.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:25 smithi169 bash[19604]: cluster 2024-02-12T16:21:24.378529+0000 osd.6 (osd.6) 82 : cluster [DBG] 4.a scrub starts 2024-02-12T16:21:26.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:25 smithi169 bash[19604]: cluster 2024-02-12T16:21:24.771802+0000 osd.6 (osd.6) 83 : cluster [DBG] 4.a scrub ok 2024-02-12T16:21:26.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:25 smithi169 bash[19604]: cluster 2024-02-12T16:21:25.276696+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.b scrub starts 2024-02-12T16:21:26.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:25 smithi169 bash[19604]: cluster 2024-02-12T16:21:25.537866+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e489: 12 total, 11 up, 12 in 2024-02-12T16:21:26.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:25 smithi169 bash[19604]: cluster 2024-02-12T16:21:25.599359+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.b scrub ok 2024-02-12T16:21:26.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-12T16:21:26.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:26.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:26.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:25 smithi121 bash[18489]: cluster 2024-02-12T16:21:24.378529+0000 osd.6 (osd.6) 82 : cluster [DBG] 4.a scrub starts 2024-02-12T16:21:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:25 smithi121 bash[18489]: cluster 2024-02-12T16:21:24.771802+0000 osd.6 (osd.6) 83 : cluster [DBG] 4.a scrub ok 2024-02-12T16:21:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:25 smithi121 bash[18489]: cluster 2024-02-12T16:21:25.276696+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.b scrub starts 2024-02-12T16:21:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:25 smithi121 bash[18489]: cluster 2024-02-12T16:21:25.537866+0000 mon.a (mon.0) 1776 : cluster [DBG] osdmap e489: 12 total, 11 up, 12 in 2024-02-12T16:21:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:25 smithi121 bash[18489]: cluster 2024-02-12T16:21:25.599359+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.b scrub ok 2024-02-12T16:21:26.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:26.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:26.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:26.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:26.632 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:26.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:26.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:26.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:26.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:26.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:26.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:26.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:26.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:21:27.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:27.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:27.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:27.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:26 smithi121 bash[18489]: cluster 2024-02-12T16:21:25.855847+0000 mgr.y (mgr.14146) 1086 : cluster [DBG] pgmap v1473: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:27.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:27.179 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:26 smithi123 bash[19119]: cluster 2024-02-12T16:21:25.855847+0000 mgr.y (mgr.14146) 1086 : cluster [DBG] pgmap v1473: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:27.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:27.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:26 smithi169 bash[19604]: cluster 2024-02-12T16:21:25.855847+0000 mgr.y (mgr.14146) 1086 : cluster [DBG] pgmap v1473: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:27.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:27.354 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:27.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:27.414 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:27.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:27.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:27.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:27.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:27.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:27.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:27.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:27.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:27.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:27.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:27.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:27.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:28.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:27 smithi169 bash[19604]: cluster 2024-02-12T16:21:27.178689+0000 osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:27 smithi169 bash[19604]: cluster 2024-02-12T16:21:27.178787+0000 osd.0 (osd.0) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:28.060 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:28.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:28.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:27 smithi121 bash[18489]: cluster 2024-02-12T16:21:27.178689+0000 osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:28.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:27 smithi121 bash[18489]: cluster 2024-02-12T16:21:27.178787+0000 osd.0 (osd.0) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:28.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:28.201 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:28.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:28.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:28.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:28.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:28.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:27 smithi123 bash[19119]: cluster 2024-02-12T16:21:27.178689+0000 osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:28.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:27 smithi123 bash[19119]: cluster 2024-02-12T16:21:27.178787+0000 osd.0 (osd.0) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:28.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:28.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:28.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:28.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:28.629 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:28.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:28.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:28.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:28.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:28.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:28.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:28.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:28.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:29.001 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:28 smithi121 bash[18489]: cluster 2024-02-12T16:21:27.856879+0000 mgr.y (mgr.14146) 1087 : cluster [DBG] pgmap v1474: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:29.001 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:28 smithi121 bash[18489]: cluster 2024-02-12T16:21:27.933118+0000 mon.a (mon.0) 1777 : cluster [DBG] osdmap e490: 12 total, 11 up, 12 in 2024-02-12T16:21:29.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:29.047 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:29.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:29.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:29.190 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:29.192 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:28 smithi123 bash[19119]: cluster 2024-02-12T16:21:27.856879+0000 mgr.y (mgr.14146) 1087 : cluster [DBG] pgmap v1474: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:29.192 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:28 smithi123 bash[19119]: cluster 2024-02-12T16:21:27.933118+0000 mon.a (mon.0) 1777 : cluster [DBG] osdmap e490: 12 total, 11 up, 12 in 2024-02-12T16:21:29.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:29.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:29.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:28 smithi169 bash[19604]: cluster 2024-02-12T16:21:27.856879+0000 mgr.y (mgr.14146) 1087 : cluster [DBG] pgmap v1474: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:28 smithi169 bash[19604]: cluster 2024-02-12T16:21:27.933118+0000 mon.a (mon.0) 1777 : cluster [DBG] osdmap e490: 12 total, 11 up, 12 in 2024-02-12T16:21:29.333 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:29.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:29.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:29.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:29.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:29.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:29.621 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:29.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:29.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:21:29.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:21:29.687 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-12T16:21:29.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd test-reweight-by-utilization 110 3.0 2024-02-12T16:21:29.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:29.761 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:29.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:29.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:29.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:29.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:30.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:30.041 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:30.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:30.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:30.184 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:30.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:30.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:30.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:30.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:30.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:30.466 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:30.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:30.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:30.600 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:30.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:30.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:30.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:30.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:30.844 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:30.881 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:30.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:30.890 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:21:30.912 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:21:30.935 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:21:30.963 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:21:30.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:30.990 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:21:30.996 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:30 smithi123 bash[19119]: cluster 2024-02-12T16:21:29.765002+0000 osd.10 (osd.10) 86 : cluster [DBG] 1.d scrub starts 2024-02-12T16:21:30.996 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:30 smithi123 bash[19119]: cluster 2024-02-12T16:21:29.787604+0000 osd.10 (osd.10) 87 : cluster [DBG] 1.d scrub ok 2024-02-12T16:21:30.996 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:30 smithi123 bash[19119]: cluster 2024-02-12T16:21:29.857860+0000 mgr.y (mgr.14146) 1088 : cluster [DBG] pgmap v1476: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:30.996 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:30 smithi123 bash[19119]: cluster 2024-02-12T16:21:30.541204+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e491: 12 total, 11 up, 12 in 2024-02-12T16:21:30.996 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:30 smithi123 bash[19119]: audit 2024-02-12T16:21:30.858190+0000 mon.a (mon.0) 1779 : audit [DBG] from='client.? v1:172.21.15.121:0/989466022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:31.013 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:31.017 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:21:31.040 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:21:31.063 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:21:31.065 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:21:31.095 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:21:31.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:31.125 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:21:31.148 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.149 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.150 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:21:31.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:31.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:30 smithi121 bash[18489]: cluster 2024-02-12T16:21:29.765002+0000 osd.10 (osd.10) 86 : cluster [DBG] 1.d scrub starts 2024-02-12T16:21:31.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:30 smithi121 bash[18489]: cluster 2024-02-12T16:21:29.787604+0000 osd.10 (osd.10) 87 : cluster [DBG] 1.d scrub ok 2024-02-12T16:21:31.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:30 smithi121 bash[18489]: cluster 2024-02-12T16:21:29.857860+0000 mgr.y (mgr.14146) 1088 : cluster [DBG] pgmap v1476: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:31.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:30 smithi121 bash[18489]: cluster 2024-02-12T16:21:30.541204+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e491: 12 total, 11 up, 12 in 2024-02-12T16:21:31.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:30 smithi121 bash[18489]: audit 2024-02-12T16:21:30.858190+0000 mon.a (mon.0) 1779 : audit [DBG] from='client.? v1:172.21.15.121:0/989466022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:31.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:30 smithi169 bash[19604]: cluster 2024-02-12T16:21:29.765002+0000 osd.10 (osd.10) 86 : cluster [DBG] 1.d scrub starts 2024-02-12T16:21:31.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:30 smithi169 bash[19604]: cluster 2024-02-12T16:21:29.787604+0000 osd.10 (osd.10) 87 : cluster [DBG] 1.d scrub ok 2024-02-12T16:21:31.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:30 smithi169 bash[19604]: cluster 2024-02-12T16:21:29.857860+0000 mgr.y (mgr.14146) 1088 : cluster [DBG] pgmap v1476: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:31.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:30 smithi169 bash[19604]: cluster 2024-02-12T16:21:30.541204+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e491: 12 total, 11 up, 12 in 2024-02-12T16:21:31.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:30 smithi169 bash[19604]: audit 2024-02-12T16:21:30.858190+0000 mon.a (mon.0) 1779 : audit [DBG] from='client.? v1:172.21.15.121:0/989466022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:31.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:31.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:31.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:31.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:31.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.710 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:31.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:31.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:21:31.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:21:31.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.987 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:31.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:32.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:32.126 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:32.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:32.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:32.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:32.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:32.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:32.281 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:32.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:32.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:32.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:32.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:32.553 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:32.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:32.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:32.691 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:32.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:32.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:32.835 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:32.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:32.939 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:32.976 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:32.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:33.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:33.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:33.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:33.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:33.234 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:32 smithi123 bash[19119]: cluster 2024-02-12T16:21:31.767740+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.d scrub starts 2024-02-12T16:21:33.234 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:32 smithi123 bash[19119]: cluster 2024-02-12T16:21:31.858515+0000 mgr.y (mgr.14146) 1089 : cluster [DBG] pgmap v1478: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:33.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:32 smithi169 bash[19604]: cluster 2024-02-12T16:21:31.767740+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.d scrub starts 2024-02-12T16:21:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:32 smithi169 bash[19604]: cluster 2024-02-12T16:21:31.858515+0000 mgr.y (mgr.14146) 1089 : cluster [DBG] pgmap v1478: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:33.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:32 smithi121 bash[18489]: cluster 2024-02-12T16:21:31.767740+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.d scrub starts 2024-02-12T16:21:33.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:32 smithi121 bash[18489]: cluster 2024-02-12T16:21:31.858515+0000 mgr.y (mgr.14146) 1089 : cluster [DBG] pgmap v1478: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:33.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:33.400 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:33.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:33.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:33.542 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:33.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:33.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:33.683 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:33.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:33.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:33.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:33.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:21:33.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:33.962 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:33.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:34.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:34.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:34.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:33 smithi123 bash[19119]: cluster 2024-02-12T16:21:32.187686+0000 osd.4 (osd.4) 72 : cluster [DBG] 4.d scrub ok 2024-02-12T16:21:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:33 smithi123 bash[19119]: cluster 2024-02-12T16:21:32.888495+0000 osd.11 (osd.11) 101 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:21:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:33 smithi123 bash[19119]: cluster 2024-02-12T16:21:32.944885+0000 osd.11 (osd.11) 102 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:21:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:33 smithi121 bash[18489]: cluster 2024-02-12T16:21:32.187686+0000 osd.4 (osd.4) 72 : cluster [DBG] 4.d scrub ok 2024-02-12T16:21:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:33 smithi121 bash[18489]: cluster 2024-02-12T16:21:32.888495+0000 osd.11 (osd.11) 101 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:21:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:33 smithi121 bash[18489]: cluster 2024-02-12T16:21:32.944885+0000 osd.11 (osd.11) 102 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:21:34.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:34.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:34.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:34.245 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:33 smithi169 bash[19604]: cluster 2024-02-12T16:21:32.187686+0000 osd.4 (osd.4) 72 : cluster [DBG] 4.d scrub ok 2024-02-12T16:21:34.245 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:33 smithi169 bash[19604]: cluster 2024-02-12T16:21:32.888495+0000 osd.11 (osd.11) 101 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:21:34.245 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:33 smithi169 bash[19604]: cluster 2024-02-12T16:21:32.944885+0000 osd.11 (osd.11) 102 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:21:34.336 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:34.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:34.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:34.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:34.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:34.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:34.516 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:21:34.516 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:21:34.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:21:34.516 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.c REQUEST e478)", 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:20:43.258952+0000", 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.253884855000003, 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027116800000000002, 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.258952+0000", 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:21:34.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.258952+0000", 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.258950+0000", 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.503e-06 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.258958+0000", 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4953e-05 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.258983+0000", 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4239999999999999e-05 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.258997+0000", 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013145999999999999 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.259129+0000", 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7132999999999998e-05 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.259166+0000", 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7111999999999998e-05 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.259223+0000", 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027116800000000002 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.520 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "replica_scrub(pg: 1.c,from:0'0,to:0'0,epoch:478/453,start:1:30000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:20:43.298854+0000", 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.213982518000002, 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014622499999999999, 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298854+0000", 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298854+0000", 2024-02-12T16:21:34.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999962 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298850+0000", 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4484999999999999e-05 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298865+0000", 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8912e-05 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298894+0000", 2024-02-12T16:21:34.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3526999999999999e-05 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298907+0000", 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8199000000000003e-05 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298965+0000", 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0039e-05 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.298976+0000", 2024-02-12T16:21:34.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4729e-05 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.299000+0000", 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014622499999999999 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.c RELEASE e478)", 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:20:43.335877+0000", 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.176959873000001, 2024-02-12T16:21:34.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000232172, 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.335877+0000", 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.335877+0000", 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999948 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.335871+0000", 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0883e-05 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.335892+0000", 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7282e-05 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.335930+0000", 2024-02-12T16:21:34.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5236999999999999e-05 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.335955+0000", 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2871e-05 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.336038+0000", 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1372000000000001e-05 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.336059+0000", 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9857000000000001e-05 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:43.336109+0000", 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000232172 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.4 REQUEST e491)", 2024-02-12T16:21:34.528 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:21:32.887891+0000", 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.6249454079999999, 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024369800000000001, 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.887891+0000", 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.887891+0000", 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999943 2024-02-12T16:21:34.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.887886+0000", 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0737000000000002e-05 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.887906+0000", 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3801999999999999e-05 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.887940+0000", 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5087e-05 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.887955+0000", 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7948000000000006e-05 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.888043+0000", 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3336000000000001e-05 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.888076+0000", 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8368000000000001e-05 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.888135+0000", 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024369800000000001 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "replica_scrub(pg: 1.4,from:0'0,to:0'0,epoch:491/453,start:1:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:21:32.918517+0000", 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.5943191969999999, 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026467400000000001, 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:21:34.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918517+0000", 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918517+0000", 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999948 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918512+0000", 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1206e-05 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918533+0000", 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0036000000000001e-05 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918574+0000", 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5363000000000001e-05 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918599+0000", 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116538 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918715+0000", 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7079999999999999e-05 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918732+0000", 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9536999999999997e-05 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.918782+0000", 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026467400000000001 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:21:34.535 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.4 RELEASE e491)", 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:21:32.945090+0000", 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.5677463600000001, 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109981, 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945090+0000", 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945090+0000", 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945087+0000", 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1185999999999999e-05 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945098+0000", 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7197000000000002e-05 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945116+0000", 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.464e-06 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945123+0000", 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2658999999999999e-05 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945166+0000", 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3062e-05 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:21:34.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945179+0000", 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1418000000000001e-05 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.945200+0000", 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109981 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:21:34.540 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:21:34.540 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:21:34.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:34.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:34.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:34.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:34.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:34.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:34.850 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:34.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:34.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:35.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:35.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:35.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:35.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:34 smithi121 bash[18489]: cluster 2024-02-12T16:21:33.859567+0000 mgr.y (mgr.14146) 1090 : cluster [DBG] pgmap v1479: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:35.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:35.208 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:35.210 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:34 smithi123 bash[19119]: cluster 2024-02-12T16:21:33.859567+0000 mgr.y (mgr.14146) 1090 : cluster [DBG] pgmap v1479: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:35.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:35.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:34 smithi169 bash[19604]: cluster 2024-02-12T16:21:33.859567+0000 mgr.y (mgr.14146) 1090 : cluster [DBG] pgmap v1479: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:35.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-12T16:21:35.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:35.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:35.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:35.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:35.495 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:35.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:35.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:35.636 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:35.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:35.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:35.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:35.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:35.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:35.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:35.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:35.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:36.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:36.032 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:35 smithi121 bash[18489]: cluster 2024-02-12T16:21:34.797874+0000 osd.10 (osd.10) 88 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:21:36.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:35 smithi121 bash[18489]: cluster 2024-02-12T16:21:34.817695+0000 osd.10 (osd.10) 89 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:21:36.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:35 smithi121 bash[18489]: cluster 2024-02-12T16:21:34.952658+0000 osd.4 (osd.4) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:36.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:35 smithi121 bash[18489]: cluster 2024-02-12T16:21:34.952751+0000 osd.4 (osd.4) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:36.049 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:36.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:36.154 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:36.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:35 smithi123 bash[19119]: cluster 2024-02-12T16:21:34.797874+0000 osd.10 (osd.10) 88 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:21:36.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:35 smithi123 bash[19119]: cluster 2024-02-12T16:21:34.817695+0000 osd.10 (osd.10) 89 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:21:36.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:35 smithi123 bash[19119]: cluster 2024-02-12T16:21:34.952658+0000 osd.4 (osd.4) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:36.167 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:35 smithi123 bash[19119]: cluster 2024-02-12T16:21:34.952751+0000 osd.4 (osd.4) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:36.192 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:36.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:36.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:35 smithi169 bash[19604]: cluster 2024-02-12T16:21:34.797874+0000 osd.10 (osd.10) 88 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:21:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:35 smithi169 bash[19604]: cluster 2024-02-12T16:21:34.817695+0000 osd.10 (osd.10) 89 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:21:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:35 smithi169 bash[19604]: cluster 2024-02-12T16:21:34.952658+0000 osd.4 (osd.4) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:35 smithi169 bash[19604]: cluster 2024-02-12T16:21:34.952751+0000 osd.4 (osd.4) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:36.334 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:36.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:36.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:36.469 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:36.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:36.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:36.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:36.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:36.752 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:36.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:36.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:36.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:36.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:36.981 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:21:37.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:37.032 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:37.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:37.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:37.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:37.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:37.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:37.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:36 smithi169 bash[19604]: cluster 2024-02-12T16:21:35.860952+0000 mgr.y (mgr.14146) 1091 : cluster [DBG] pgmap v1480: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:37.288 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:36 smithi169 bash[19604]: cluster 2024-02-12T16:21:35.958355+0000 mon.a (mon.0) 1780 : cluster [DBG] osdmap e492: 12 total, 11 up, 12 in 2024-02-12T16:21:37.288 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:36 smithi169 bash[19604]: audit 2024-02-12T16:21:36.306767+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.121:0/1093357287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:21:37.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:37.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:36 smithi123 bash[19119]: cluster 2024-02-12T16:21:35.860952+0000 mgr.y (mgr.14146) 1091 : cluster [DBG] pgmap v1480: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:36 smithi123 bash[19119]: cluster 2024-02-12T16:21:35.958355+0000 mon.a (mon.0) 1780 : cluster [DBG] osdmap e492: 12 total, 11 up, 12 in 2024-02-12T16:21:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:36 smithi123 bash[19119]: audit 2024-02-12T16:21:36.306767+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.121:0/1093357287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:21:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:36 smithi121 bash[18489]: cluster 2024-02-12T16:21:35.860952+0000 mgr.y (mgr.14146) 1091 : cluster [DBG] pgmap v1480: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:36 smithi121 bash[18489]: cluster 2024-02-12T16:21:35.958355+0000 mon.a (mon.0) 1780 : cluster [DBG] osdmap e492: 12 total, 11 up, 12 in 2024-02-12T16:21:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:36 smithi121 bash[18489]: audit 2024-02-12T16:21:36.306767+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.121:0/1093357287' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:21:37.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:37.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:37.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:37.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:37.595 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:37.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:37.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:37.738 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:37.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:37.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:37.880 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:37.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:37.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:38.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:38.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:38.151 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:38.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:38.267 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:37 smithi123 bash[19119]: cluster 2024-02-12T16:21:36.961879+0000 mon.a (mon.0) 1782 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:21:38.268 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:37 smithi123 bash[19119]: audit 2024-02-12T16:21:36.969510+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? v1:172.21.15.121:0/1093357287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:21:38.268 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:37 smithi123 bash[19119]: cluster 2024-02-12T16:21:36.969743+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e493: 12 total, 11 up, 12 in 2024-02-12T16:21:38.271 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:37 smithi121 bash[18489]: cluster 2024-02-12T16:21:36.961879+0000 mon.a (mon.0) 1782 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:21:38.271 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:37 smithi121 bash[18489]: audit 2024-02-12T16:21:36.969510+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? v1:172.21.15.121:0/1093357287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:21:38.271 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:37 smithi121 bash[18489]: cluster 2024-02-12T16:21:36.969743+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e493: 12 total, 11 up, 12 in 2024-02-12T16:21:38.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:37 smithi169 bash[19604]: cluster 2024-02-12T16:21:36.961879+0000 mon.a (mon.0) 1782 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:21:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:37 smithi169 bash[19604]: audit 2024-02-12T16:21:36.969510+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? v1:172.21.15.121:0/1093357287' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:21:38.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:37 smithi169 bash[19604]: cluster 2024-02-12T16:21:36.969743+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e493: 12 total, 11 up, 12 in 2024-02-12T16:21:38.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:38.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:38.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:38.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:38.644 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:38.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:38.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:38.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:38.844 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:38.887 INFO:teuthology.orchestra.run.smithi121.stdout:moved 4 / 74 (5.40541%) 2024-02-12T16:21:38.887 INFO:teuthology.orchestra.run.smithi121.stdout:avg 6.72727 2024-02-12T16:21:38.887 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 3.38201 -> 3.44976 (expected baseline 2.47299) 2024-02-12T16:21:38.887 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.1 with 1 -> 0 pgs (0.148649 -> 0 * mean) 2024-02-12T16:21:38.887 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.10 with 12 -> 12 pgs (1.78378 -> 1.78378 * mean) 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:oload 110 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 3 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0081 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.0089 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11 weight 1.0000 -> 0.5950 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 weight 0.1039 -> 0.0650 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 weight 1.0000 -> 0.8086 2024-02-12T16:21:38.888 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10 weight 1.0000 -> 0.8735 2024-02-12T16:21:38.891 INFO:teuthology.orchestra.run.smithi121.stderr:no change 2024-02-12T16:21:38.941 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 16:21:38 smithi121 bash[18805]: debug 2024-02-12T16:21:38.877+0000 7fb564e0b700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-02-12T16:21:38.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:38.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:38.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:39.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:39.125 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:39.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:39.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:39.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:38 smithi169 bash[19604]: cluster 2024-02-12T16:21:37.861977+0000 mgr.y (mgr.14146) 1092 : cluster [DBG] pgmap v1483: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:39.267 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:39.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:39.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:39.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:38 smithi123 bash[19119]: cluster 2024-02-12T16:21:37.861977+0000 mgr.y (mgr.14146) 1092 : cluster [DBG] pgmap v1483: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:39.406 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:39.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:39.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:38 smithi121 bash[18489]: cluster 2024-02-12T16:21:37.861977+0000 mgr.y (mgr.14146) 1092 : cluster [DBG] pgmap v1483: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:39.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:39.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:39.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:39.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:39.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:39.700 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:39.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:39.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:39.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:39.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:39.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:39.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:39.992 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-12T16:21:39.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight-by-utilization 1000 3.0 2024-02-12T16:21:40.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:40.091 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:39 smithi121 bash[18489]: audit 2024-02-12T16:21:38.882140+0000 mgr.y (mgr.14146) 1093 : audit [DBG] from='client.15891 v1:172.21.15.121:0/2019940481' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:21:40.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:40.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:40.122 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:21:40.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:40.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:40.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:40.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:21:40.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:40 smithi169 bash[19604]: audit 2024-02-12T16:21:38.882140+0000 mgr.y (mgr.14146) 1093 : audit [DBG] from='client.15891 v1:172.21.15.121:0/2019940481' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:21:40.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:40.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:40 smithi123 bash[19119]: audit 2024-02-12T16:21:38.882140+0000 mgr.y (mgr.14146) 1093 : audit [DBG] from='client.15891 v1:172.21.15.121:0/2019940481' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:21:40.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:40.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:40.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:40.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:40.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:40.621 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:40.658 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:40.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:40.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:40.800 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:40.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:40.904 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:40.941 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:40.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:41.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:41.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:41.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:41.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:41.195 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:41 smithi123 bash[19119]: cluster 2024-02-12T16:21:39.863005+0000 mgr.y (mgr.14146) 1094 : cluster [DBG] pgmap v1484: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:41.216 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:41.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:41 smithi169 bash[19604]: cluster 2024-02-12T16:21:39.863005+0000 mgr.y (mgr.14146) 1094 : cluster [DBG] pgmap v1484: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:41.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:41.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:41 smithi121 bash[18489]: cluster 2024-02-12T16:21:39.863005+0000 mgr.y (mgr.14146) 1094 : cluster [DBG] pgmap v1484: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:41.357 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:41.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:41.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:41.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:41.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:41.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:41.635 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:41.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:41.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:41.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:41.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:41.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:41.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:41.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:42.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:42.052 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:42.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:42.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:42.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:42.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:42.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:42 smithi169 bash[19604]: cluster 2024-02-12T16:21:41.115678+0000 osd.6 (osd.6) 84 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:21:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:42 smithi169 bash[19604]: cluster 2024-02-12T16:21:41.141130+0000 osd.6 (osd.6) 85 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:21:42.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:42.307 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:42 smithi121 bash[18489]: cluster 2024-02-12T16:21:41.115678+0000 osd.6 (osd.6) 84 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:21:42.307 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:42 smithi121 bash[18489]: cluster 2024-02-12T16:21:41.141130+0000 osd.6 (osd.6) 85 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:21:42.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:42.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:42.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:42 smithi123 bash[19119]: cluster 2024-02-12T16:21:41.115678+0000 osd.6 (osd.6) 84 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:21:42.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:42 smithi123 bash[19119]: cluster 2024-02-12T16:21:41.141130+0000 osd.6 (osd.6) 85 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:21:42.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:42.468 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:21:42.480 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:42.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:42.493 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:21:42.518 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:21:42.541 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:21:42.567 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:21:42.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:42.595 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:21:42.619 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:42.620 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:21:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:42.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:42.642 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:21:42.671 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:21:42.695 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:21:42.723 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:21:42.762 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:42.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:42.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:42.900 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:42.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:43.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:43.044 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:43.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:43.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:43 smithi121 bash[18489]: cluster 2024-02-12T16:21:41.864188+0000 mgr.y (mgr.14146) 1095 : cluster [DBG] pgmap v1485: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:43 smithi121 bash[18489]: audit 2024-02-12T16:21:42.439889+0000 mon.a (mon.0) 1785 : audit [DBG] from='client.? v1:172.21.15.121:0/3831285626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:43.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:43.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:43.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:43 smithi169 bash[19604]: cluster 2024-02-12T16:21:41.864188+0000 mgr.y (mgr.14146) 1095 : cluster [DBG] pgmap v1485: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:43.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:43 smithi169 bash[19604]: audit 2024-02-12T16:21:42.439889+0000 mon.a (mon.0) 1785 : audit [DBG] from='client.? v1:172.21.15.121:0/3831285626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:43.324 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:43.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:43 smithi123 bash[19119]: cluster 2024-02-12T16:21:41.864188+0000 mgr.y (mgr.14146) 1095 : cluster [DBG] pgmap v1485: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:43 smithi123 bash[19119]: audit 2024-02-12T16:21:42.439889+0000 mon.a (mon.0) 1785 : audit [DBG] from='client.? v1:172.21.15.121:0/3831285626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:21:43.413 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:21:43.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:21:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:21:43.462 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:43.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:43.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:43.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:43.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:43.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:43.741 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:43.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:43.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:43.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:43.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:43.987 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:44.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:44.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:44.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-12T16:21:44.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:44.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:44.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:44.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:44.306 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:44.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:44.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:44.448 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:44.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:44.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:44.590 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:44.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:44.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:44.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:44.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:44.783 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:44.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:44.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:44.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:44.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:45.015 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:45.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:45.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:45.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:45.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:45.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:45.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:45 smithi169 bash[19604]: cluster 2024-02-12T16:21:43.865263+0000 mgr.y (mgr.14146) 1096 : cluster [DBG] pgmap v1486: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:45.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:45.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:45 smithi123 bash[19119]: cluster 2024-02-12T16:21:43.865263+0000 mgr.y (mgr.14146) 1096 : cluster [DBG] pgmap v1486: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:45.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:45.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:45 smithi121 bash[18489]: cluster 2024-02-12T16:21:43.865263+0000 mgr.y (mgr.14146) 1096 : cluster [DBG] pgmap v1486: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:45.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:21:45.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:45.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:45.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:45.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:45.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:45.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:45.727 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:45.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:45.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:45.866 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:45.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:45.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:46.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:46.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:46.148 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:46.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:46.291 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:46.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:46.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:46.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:46.578 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:46.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:46 smithi121 bash[18489]: cluster 2024-02-12T16:21:45.865945+0000 mgr.y (mgr.14146) 1097 : cluster [DBG] pgmap v1487: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:46.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:46.692 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:46 smithi169 bash[19604]: cluster 2024-02-12T16:21:45.865945+0000 mgr.y (mgr.14146) 1097 : cluster [DBG] pgmap v1487: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:46.717 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:46.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:46.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:46 smithi123 bash[19119]: cluster 2024-02-12T16:21:45.865945+0000 mgr.y (mgr.14146) 1097 : cluster [DBG] pgmap v1487: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:46.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:46.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:46.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:46.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:47.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:47.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:47.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:47.279 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:47.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:47.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:47.420 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:47.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:47.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:47.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:47.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:47.693 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:47.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:47.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:47.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:47 smithi169 bash[19604]: cluster 2024-02-12T16:21:47.317204+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:47.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:47 smithi169 bash[19604]: cluster 2024-02-12T16:21:47.317293+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:47.834 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:47.837 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:47 smithi123 bash[19119]: cluster 2024-02-12T16:21:47.317204+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:47.837 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:47 smithi123 bash[19119]: cluster 2024-02-12T16:21:47.317293+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:47.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:47.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:47.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:47 smithi121 bash[18489]: cluster 2024-02-12T16:21:47.317204+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:47.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:47 smithi121 bash[18489]: cluster 2024-02-12T16:21:47.317293+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:47.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:47.977 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:47.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:48.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:48.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:48.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:48.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:48.264 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:48.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:48.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:48.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:48.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:48.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:48.545 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:48.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:48.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:48.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:48 smithi121 bash[18489]: cluster 2024-02-12T16:21:47.571229+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e494: 12 total, 11 up, 12 in 2024-02-12T16:21:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:48 smithi121 bash[18489]: cluster 2024-02-12T16:21:47.866856+0000 mgr.y (mgr.14146) 1098 : cluster [DBG] pgmap v1489: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:48.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:48.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:48.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:48.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:48.858 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:21:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:48 smithi123 bash[19119]: cluster 2024-02-12T16:21:47.571229+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e494: 12 total, 11 up, 12 in 2024-02-12T16:21:48.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:48 smithi123 bash[19119]: cluster 2024-02-12T16:21:47.866856+0000 mgr.y (mgr.14146) 1098 : cluster [DBG] pgmap v1489: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:48.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:48.943 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:48 smithi169 bash[19604]: cluster 2024-02-12T16:21:47.571229+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e494: 12 total, 11 up, 12 in 2024-02-12T16:21:48.943 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:48 smithi169 bash[19604]: cluster 2024-02-12T16:21:47.866856+0000 mgr.y (mgr.14146) 1098 : cluster [DBG] pgmap v1489: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:48.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:48.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:49.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:49.100 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:49.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:49.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:49.240 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:49.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:49.323 INFO:teuthology.orchestra.run.smithi121.stdout:moved 0 / 74 (0%) 2024-02-12T16:21:49.323 INFO:teuthology.orchestra.run.smithi121.stdout:avg 6.72727 2024-02-12T16:21:49.323 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 3.38201 -> 3.38201 (expected baseline 2.47299) 2024-02-12T16:21:49.323 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.1 with 1 -> 1 pgs (0.148649 -> 0.148649 * mean) 2024-02-12T16:21:49.323 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.10 with 12 -> 12 pgs (1.78378 -> 1.78378 * mean) 2024-02-12T16:21:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:21:49.324 INFO:teuthology.orchestra.run.smithi121.stdout:oload 1000 2024-02-12T16:21:49.324 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 3 2024-02-12T16:21:49.324 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:21:49.324 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0081 2024-02-12T16:21:49.324 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.0811 2024-02-12T16:21:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:no change 2024-02-12T16:21:49.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:49.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:49.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:49.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:49.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:49.607 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:21:49.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:49 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.169638+0000 osd.11 (osd.11) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:49 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.169726+0000 osd.11 (osd.11) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:49.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:49 smithi121 bash[18489]: audit 2024-02-12T16:21:49.298044+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? v1:172.21.15.121:0/1134400339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:21:49.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:49 smithi121 bash[18489]: audit 2024-02-12T16:21:49.318157+0000 mgr.y (mgr.14146) 1099 : audit [DBG] from='client.15909 v1:172.21.15.121:0/3246182416' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:21:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:49.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:49.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:49.798 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:49.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:49 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.169638+0000 osd.11 (osd.11) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:49 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.169726+0000 osd.11 (osd.11) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:49.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:49 smithi123 bash[19119]: audit 2024-02-12T16:21:49.298044+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? v1:172.21.15.121:0/1134400339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:21:49.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:49 smithi123 bash[19119]: audit 2024-02-12T16:21:49.318157+0000 mgr.y (mgr.14146) 1099 : audit [DBG] from='client.15909 v1:172.21.15.121:0/3246182416' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:21:49.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:49.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:49 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.169638+0000 osd.11 (osd.11) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:49.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:49 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.169726+0000 osd.11 (osd.11) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:49.914 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:49 smithi169 bash[19604]: audit 2024-02-12T16:21:49.298044+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? v1:172.21.15.121:0/1134400339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:21:49.915 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:49 smithi169 bash[19604]: audit 2024-02-12T16:21:49.318157+0000 mgr.y (mgr.14146) 1099 : audit [DBG] from='client.15909 v1:172.21.15.121:0/3246182416' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:21:49.940 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:49.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:50.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:50.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:50.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:50.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:50.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:50.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:50.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:50.366 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:50.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:50.442 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-12T16:21:50.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-02-12T16:21:50.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:50.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:50.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:50.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:50.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:50.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:50.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:50.786 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:50.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.375885+0000 osd.5 (osd.5) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.375972+0000 osd.5 (osd.5) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.579140+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: audit 2024-02-12T16:21:49.596275+0000 mon.a (mon.0) 1789 : audit [INF] from='client.? v1:172.21.15.121:0/1134400339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.596449+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e495: 12 total, 11 up, 12 in 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.656267+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.656341+0000 osd.6 (osd.6) 87 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.766498+0000 osd.10 (osd.10) 90 : cluster [DBG] 1.5 deep-scrub starts 2024-02-12T16:21:50.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:50 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.867834+0000 mgr.y (mgr.14146) 1100 : cluster [DBG] pgmap v1491: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:50.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.375885+0000 osd.5 (osd.5) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.375972+0000 osd.5 (osd.5) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.579140+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: audit 2024-02-12T16:21:49.596275+0000 mon.a (mon.0) 1789 : audit [INF] from='client.? v1:172.21.15.121:0/1134400339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.596449+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e495: 12 total, 11 up, 12 in 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.656267+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.656341+0000 osd.6 (osd.6) 87 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.766498+0000 osd.10 (osd.10) 90 : cluster [DBG] 1.5 deep-scrub starts 2024-02-12T16:21:50.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:50 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.867834+0000 mgr.y (mgr.14146) 1100 : cluster [DBG] pgmap v1491: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:50.939 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:50.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:51.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.375885+0000 osd.5 (osd.5) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.375972+0000 osd.5 (osd.5) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.579140+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:21:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: audit 2024-02-12T16:21:49.596275+0000 mon.a (mon.0) 1789 : audit [INF] from='client.? v1:172.21.15.121:0/1134400339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:21:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.596449+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e495: 12 total, 11 up, 12 in 2024-02-12T16:21:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.656267+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.656341+0000 osd.6 (osd.6) 87 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.766498+0000 osd.10 (osd.10) 90 : cluster [DBG] 1.5 deep-scrub starts 2024-02-12T16:21:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:50 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.867834+0000 mgr.y (mgr.14146) 1100 : cluster [DBG] pgmap v1491: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:51.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:51.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:51.094 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:51.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:51.224 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:51.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:51.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:51.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:51.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:51.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:51.509 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:51.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:51.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:51.652 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:51.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:51.756 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:51.793 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:51.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:51 smithi123 bash[19119]: cluster 2024-02-12T16:21:49.784899+0000 osd.10 (osd.10) 91 : cluster [DBG] 1.5 deep-scrub ok 2024-02-12T16:21:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:51 smithi123 bash[19119]: cluster 2024-02-12T16:21:50.600485+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e496: 12 total, 11 up, 12 in 2024-02-12T16:21:51.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:51.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:51 smithi121 bash[18489]: cluster 2024-02-12T16:21:49.784899+0000 osd.10 (osd.10) 91 : cluster [DBG] 1.5 deep-scrub ok 2024-02-12T16:21:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:51 smithi121 bash[18489]: cluster 2024-02-12T16:21:50.600485+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e496: 12 total, 11 up, 12 in 2024-02-12T16:21:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:51.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:52.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:51 smithi169 bash[19604]: cluster 2024-02-12T16:21:49.784899+0000 osd.10 (osd.10) 91 : cluster [DBG] 1.5 deep-scrub ok 2024-02-12T16:21:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:51 smithi169 bash[19604]: cluster 2024-02-12T16:21:50.600485+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e496: 12 total, 11 up, 12 in 2024-02-12T16:21:52.076 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:52.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:52.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:52.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:52.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:52.317 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:21:52.317 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:21:52.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:52.360 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:52.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:52.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:52.502 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:52.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:52.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:52.642 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:52.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:52.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:52.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:21:52.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:52.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:52.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:52.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:52 smithi123 bash[19119]: cluster 2024-02-12T16:21:51.868822+0000 mgr.y (mgr.14146) 1101 : cluster [DBG] pgmap v1493: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:52.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:52 smithi121 bash[18489]: cluster 2024-02-12T16:21:51.868822+0000 mgr.y (mgr.14146) 1101 : cluster [DBG] pgmap v1493: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:52.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:52.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:52.988 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:52 smithi169 bash[19604]: cluster 2024-02-12T16:21:51.868822+0000 mgr.y (mgr.14146) 1101 : cluster [DBG] pgmap v1493: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:52.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-12T16:21:53.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:53.064 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:53.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:53.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:53.196 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:53.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:53.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:53.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:53.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:53.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:53.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:53.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:53.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:53.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:53.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:53.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:53.752 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:53.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:53 smithi169 bash[19604]: audit 2024-02-12T16:21:52.967330+0000 mon.a (mon.0) 1792 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:21:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:53 smithi169 bash[19604]: audit 2024-02-12T16:21:53.269895+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:21:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:53 smithi169 bash[19604]: audit 2024-02-12T16:21:53.572327+0000 mon.a (mon.0) 1794 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:21:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:53 smithi169 bash[19604]: audit 2024-02-12T16:21:53.573888+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:21:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:53 smithi169 bash[19604]: audit 2024-02-12T16:21:53.584008+0000 mon.a (mon.0) 1796 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:21:53.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:53.869 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:53 smithi123 bash[19119]: audit 2024-02-12T16:21:52.967330+0000 mon.a (mon.0) 1792 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:21:53.869 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:53 smithi123 bash[19119]: audit 2024-02-12T16:21:53.269895+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:21:53.869 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:53 smithi123 bash[19119]: audit 2024-02-12T16:21:53.572327+0000 mon.a (mon.0) 1794 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:21:53.869 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:53 smithi123 bash[19119]: audit 2024-02-12T16:21:53.573888+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:21:53.869 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:53 smithi123 bash[19119]: audit 2024-02-12T16:21:53.584008+0000 mon.a (mon.0) 1796 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:21:53.893 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:53.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:53.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:53 smithi121 bash[18489]: audit 2024-02-12T16:21:52.967330+0000 mon.a (mon.0) 1792 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:21:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:53 smithi121 bash[18489]: audit 2024-02-12T16:21:53.269895+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:21:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:53 smithi121 bash[18489]: audit 2024-02-12T16:21:53.572327+0000 mon.a (mon.0) 1794 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:21:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:53 smithi121 bash[18489]: audit 2024-02-12T16:21:53.573888+0000 mon.a (mon.0) 1795 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:21:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:53 smithi121 bash[18489]: audit 2024-02-12T16:21:53.584008+0000 mon.a (mon.0) 1796 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:21:53.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:54.039 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:54.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:54.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:21:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:54.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:21:54.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:54.327 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:54.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:54.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:54.462 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:54.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:54.539 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:21:54.558 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:21:54.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:54.579 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:21:54.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:54.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:54.609 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:21:54.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:21:54.656 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:21:54.686 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:21:54.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:54.715 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:21:54.739 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:21:54.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:54.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:54.764 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.790 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.791 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.791 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:21:54.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:54 smithi121 bash[18489]: cluster 2024-02-12T16:21:53.869781+0000 mgr.y (mgr.14146) 1102 : cluster [DBG] pgmap v1494: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:54.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:54 smithi121 bash[18489]: audit 2024-02-12T16:21:54.507918+0000 mon.a (mon.0) 1797 : audit [DBG] from='client.? v1:172.21.15.121:0/2604683831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:54.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:54.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:54.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:54 smithi123 bash[19119]: cluster 2024-02-12T16:21:53.869781+0000 mgr.y (mgr.14146) 1102 : cluster [DBG] pgmap v1494: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:54 smithi123 bash[19119]: audit 2024-02-12T16:21:54.507918+0000 mon.a (mon.0) 1797 : audit [DBG] from='client.? v1:172.21.15.121:0/2604683831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:54.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:55.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:55.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:54 smithi169 bash[19604]: cluster 2024-02-12T16:21:53.869781+0000 mgr.y (mgr.14146) 1102 : cluster [DBG] pgmap v1494: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:54 smithi169 bash[19604]: audit 2024-02-12T16:21:54.507918+0000 mon.a (mon.0) 1797 : audit [DBG] from='client.? v1:172.21.15.121:0/2604683831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:21:55.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:55.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:55.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:55.260 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:55.294 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:55.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:55.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:55.434 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:55.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:55.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:55.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:21:55.563 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:21:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:55.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:55.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:55.690 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:55 smithi169 bash[19604]: cluster 2024-02-12T16:21:53.713944+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:55.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:55 smithi169 bash[19604]: cluster 2024-02-12T16:21:53.714047+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:55.691 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:55 smithi169 bash[19604]: cluster 2024-02-12T16:21:54.627362+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e497: 12 total, 11 up, 12 in 2024-02-12T16:21:55.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:55.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:55.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:55.855 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:55.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:55 smithi123 bash[19119]: cluster 2024-02-12T16:21:53.713944+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:55 smithi123 bash[19119]: cluster 2024-02-12T16:21:53.714047+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:55.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:55 smithi123 bash[19119]: cluster 2024-02-12T16:21:54.627362+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e497: 12 total, 11 up, 12 in 2024-02-12T16:21:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:55 smithi121 bash[18489]: cluster 2024-02-12T16:21:53.713944+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:21:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:55 smithi121 bash[18489]: cluster 2024-02-12T16:21:53.714047+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:21:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:55 smithi121 bash[18489]: cluster 2024-02-12T16:21:54.627362+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e497: 12 total, 11 up, 12 in 2024-02-12T16:21:55.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:55.997 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:56.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:56.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:56.224 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:56.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:56.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:56.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:56.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:56.410 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:56.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:56.555 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:56.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:56.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:56.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:56.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:56 smithi123 bash[19119]: cluster 2024-02-12T16:21:55.870890+0000 mgr.y (mgr.14146) 1103 : cluster [DBG] pgmap v1496: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:56.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:56 smithi121 bash[18489]: cluster 2024-02-12T16:21:55.870890+0000 mgr.y (mgr.14146) 1103 : cluster [DBG] pgmap v1496: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:56.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:56.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:21:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:56.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:21:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:56 smithi169 bash[19604]: cluster 2024-02-12T16:21:55.870890+0000 mgr.y (mgr.14146) 1103 : cluster [DBG] pgmap v1496: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:57.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:57.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:57.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:57.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:57.267 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:57.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:57.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:57.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:57.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:57.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:57.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:57.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:57.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:21:57.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:57.692 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:57.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:57.767 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:21:57.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:57.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:21:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:57.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:21:57.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:57.962 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:57.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:58.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:21:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:58.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:21:58.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:58.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:21:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:58.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:21:58.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:58.389 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:58.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:58.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:58.534 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:58.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:58.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:58.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:21:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:58.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:21:58.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:58.816 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:21:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:58.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:21:58.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:58.932 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:21:58.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:58.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:21:58 smithi169 bash[19604]: cluster 2024-02-12T16:21:57.871566+0000 mgr.y (mgr.14146) 1104 : cluster [DBG] pgmap v1497: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:59.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:59.084 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:21:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:59.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:21:59.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:59.201 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:21:58 smithi123 bash[19119]: cluster 2024-02-12T16:21:57.871566+0000 mgr.y (mgr.14146) 1104 : cluster [DBG] pgmap v1497: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:59.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:59.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:59.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:59.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:59.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:21:59.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:21:58 smithi121 bash[18489]: cluster 2024-02-12T16:21:57.871566+0000 mgr.y (mgr.14146) 1104 : cluster [DBG] pgmap v1497: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:21:59.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:59.512 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:59.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:59.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:59.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:21:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:59.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:21:59.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:59.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:21:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:59.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:21:59.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:59.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:21:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:21:59.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:00.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:00.067 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:00.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:00.170 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:00.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:00.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:00.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:00.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:00.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:00.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:00.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:00.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:00.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:00.645 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:00.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:00.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:00.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:00.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:00.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:00.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:00.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:01.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:01.063 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:01.066 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:22:01.066 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:22:01.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:22:01.066 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:22:01.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.066 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86335 4.6 4:67d81edf:::benchmark_data_smithi169_60938_object86334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:22:01.066 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.312762+0000", 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 453.74661597699998, 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197120395, 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86335 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312762+0000", 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312762+0000", 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312760+0000", 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0772999999999999e-05 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312781+0000", 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4519999999999997e-06 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312790+0000", 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9189999999999998e-06 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312794+0000", 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00063509799999999996 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.313430+0000", 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001055 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.313535+0000", 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013436500000000001 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.313669+0000", 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003412689 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.317082+0000", 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.192655664 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509738+0000", 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0968000000000002e-05 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509809+0000", 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3480999999999997e-05 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509882+0000", 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197120395 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86337 4.6 4:6b8db7f7:::benchmark_data_smithi169_60938_object86336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.312806+0000", 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 453.746572125, 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197206625, 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.072 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86337 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312806+0000", 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312806+0000", 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312805+0000", 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7529000000000001e-05 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312823+0000", 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2400000000000004e-06 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312829+0000", 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3109999999999999e-06 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.312833+0000", 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029173200000000002 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.315750+0000", 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6951000000000003e-05 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.315817+0000", 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8974000000000006e-05 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.315906+0000", 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012923069999999999 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.317199+0000", 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.192730133 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509929+0000", 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3449999999999999e-05 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509972+0000", 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0093999999999999e-05 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510012+0000", 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197206625 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86348 4.6 4:690a34d8:::benchmark_data_smithi169_60938_object86347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.322481+0000", 2024-02-12T16:22:01.077 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 453.73689647200001, 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.187767985, 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86348 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322481+0000", 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322481+0000", 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322480+0000", 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4097999999999999e-05 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322504+0000", 2024-02-12T16:22:01.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1962e-05 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322516+0000", 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-06 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322522+0000", 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6094999999999997e-05 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322579+0000", 2024-02-12T16:22:01.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1720999999999999e-05 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322660+0000", 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010755399999999999 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.322768+0000", 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020081309999999998 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.324776+0000", 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18531487399999999 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510091+0000", 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8924000000000004e-05 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510180+0000", 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9573000000000006e-05 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510249+0000", 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.187767985 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:86368 4.2 4:42a37998:::benchmark_data_smithi169_60938_object86367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:27.334423+0000", 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 453.72495486600002, 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18501995099999999, 2024-02-12T16:22:01.083 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 86368 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334423+0000", 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334423+0000", 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334421+0000", 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5689999999999999e-05 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334457+0000", 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0488e-05 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334467+0000", 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0270000000000001e-06 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.334473+0000", 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17526112199999999 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.509735+0000", 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00035024900000000001 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510085+0000", 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00067824599999999997 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.510763+0000", 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0085641840000000007 2024-02-12T16:22:01.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519327+0000", 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6293000000000001e-05 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519344+0000", 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4197999999999998e-05 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519398+0000", 2024-02-12T16:22:01.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5169000000000001e-05 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:27.519443+0000", 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18501995099999999 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171789 4.6 4:6ca60eb7:::benchmark_data_smithi169_60938_object171788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.709418+0000", 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.349960141, 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20106019, 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.089 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171789 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709418+0000", 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709418+0000", 2024-02-12T16:22:01.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709415+0000", 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5406999999999999e-05 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709440+0000", 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4986e-05 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709455+0000", 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9490000000000001e-06 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709461+0000", 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5544999999999998e-05 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709517+0000", 2024-02-12T16:22:01.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014442200000000001 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709661+0000", 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018882299999999999 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.709850+0000", 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049930249999999999 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.714843+0000", 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.195279853 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.910123+0000", 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000177753 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.910301+0000", 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017720600000000001 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.910478+0000", 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20106019 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171799 4.6 4:6cb0dab3:::benchmark_data_smithi169_60938_object171798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.719308+0000", 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.34006974900001, 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22523388999999999, 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.095 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171799 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719308+0000", 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719308+0000", 2024-02-12T16:22:01.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719305+0000", 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8177999999999998e-05 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719353+0000", 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5305e-05 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719369+0000", 2024-02-12T16:22:01.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0270000000000001e-06 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719375+0000", 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6123000000000004e-05 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719441+0000", 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000177877 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719619+0000", 2024-02-12T16:22:01.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000179975 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719799+0000", 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079059779999999993 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.727705+0000", 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21674727599999999 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944452+0000", 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6431000000000003e-05 2024-02-12T16:22:01.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944498+0000", 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3581999999999997e-05 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944542+0000", 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22523388999999999 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171800 4.6 4:631aee4e:::benchmark_data_smithi169_60938_object171799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.719430+0000", 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.33994826700001, 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.225286442, 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171800 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719430+0000", 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719430+0000", 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719427+0000", 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7589999999999998e-05 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719455+0000", 2024-02-12T16:22:01.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0733999999999999e-05 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719466+0000", 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0770000000000008e-06 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.719474+0000", 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045064959999999996 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.723980+0000", 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4834999999999996e-05 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.724075+0000", 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117805 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.724193+0000", 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003681981 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.727875+0000", 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.216733237 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944608+0000", 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4753999999999997e-05 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944673+0000", 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3019999999999998e-05 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944716+0000", 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.225286442 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:171812 4.6 4:6e56c86e:::benchmark_data_smithi169_60938_object171811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e369)", 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:26.745502+0000", 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.31387622800003, 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19937315999999999, 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.106 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 171812 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745502+0000", 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745502+0000", 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:01.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745500+0000", 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2982000000000001e-05 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745523+0000", 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3764000000000001e-05 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745536+0000", 2024-02-12T16:22:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5720000000000002e-06 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745542+0000", 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6146999999999998e-05 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745578+0000", 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4498000000000003e-05 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745663+0000", 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8611999999999997e-05 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.745761+0000", 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037280770000000002 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.749489+0000", 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19529492000000001 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944784+0000", 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0645e-05 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944835+0000", 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9860000000000001e-05 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:26.944875+0000", 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19937315999999999 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204304 4.6 4:6e94d6d4:::benchmark_data_smithi169_60938_object30967:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:55.988179+0000", 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 365.07119906200001, 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66636980899999998, 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204304 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988179+0000", 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988179+0000", 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988177+0000", 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5560000000000002e-06 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988184+0000", 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8339999999999994e-06 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988194+0000", 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5600000000000001e-06 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988199+0000", 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022521169000000001 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010721+0000", 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2280000000000005e-05 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010803+0000", 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5997999999999999e-05 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010889+0000", 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014777161 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025666+0000", 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62878948999999995 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654456+0000", 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1832000000000002e-05 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654497+0000", 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1159999999999998e-05 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654549+0000", 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66636980899999998 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204305 4.6 4:66ec962d:::benchmark_data_smithi169_60938_object30968:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:55.988325+0000", 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 365.071052769, 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66632389999999997, 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.117 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204305 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988325+0000", 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988325+0000", 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988324+0000", 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5495999999999999e-05 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988339+0000", 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4069999999999993e-06 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988347+0000", 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6090000000000003e-06 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.119 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:55.988352+0000", 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.026163005999999999 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.014515+0000", 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011727600000000001 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.014632+0000", 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5241999999999994e-05 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.014728+0000", 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011066411 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025794+0000", 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62879766199999998 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654592+0000", 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0914000000000003e-05 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654623+0000", 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6397000000000002e-05 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.654649+0000", 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66632389999999997 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204327 4.6 4:6a1a5157:::benchmark_data_smithi169_60938_object30990:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.010343+0000", 2024-02-12T16:22:01.122 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 365.04903510499997, 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66292315300000004, 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204327 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010343+0000", 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010343+0000", 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010341+0000", 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6649999999999997e-06 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010348+0000", 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1185e-05 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010359+0000", 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5560000000000004e-06 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010365+0000", 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082932909999999995 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.018659+0000", 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7928000000000003e-05 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.018737+0000", 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6607000000000005e-05 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.018833+0000", 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069676490000000002 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025801+0000", 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.64736806700000005 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.673169+0000", 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1934000000000002e-05 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.673211+0000", 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5004999999999999e-05 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.673266+0000", 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66292315300000004 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.127 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204328 4.6 4:6bbed11b:::benchmark_data_smithi169_60938_object30991:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.010503+0000", 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 365.04887509299999, 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66864712000000004, 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204328 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010503+0000", 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010503+0000", 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010501+0000", 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.674e-06 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010508+0000", 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0327e-05 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010518+0000", 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5882000000000001e-05 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.010544+0000", 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011838392999999999 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.022382+0000", 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129247 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.022511+0000", 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0579000000000001e-05 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.022602+0000", 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032876149999999998 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.025890+0000", 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65316824299999998 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679058+0000", 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9817999999999997e-05 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679098+0000", 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2179000000000003e-05 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679150+0000", 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.66864712000000004 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204339 4.6 4:6432c7bb:::benchmark_data_smithi169_60938_object31002:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.032281+0000", 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 365.02709641199999, 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.647598381, 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.133 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204339 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032281+0000", 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032281+0000", 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032279+0000", 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2420000000000003e-06 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032287+0000", 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0118999999999999e-05 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032297+0000", 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1210000000000001e-06 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.032304+0000", 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033154349999999998 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.035619+0000", 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0241000000000006e-05 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.035699+0000", 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0716000000000004e-05 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.035790+0000", 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005813609 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.041604+0000", 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63818005300000002 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679784+0000", 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4894999999999997e-05 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679819+0000", 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1005999999999998e-05 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.679880+0000", 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.647598381 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.138 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204341 4.6 4:63b4452c:::benchmark_data_smithi169_60938_object31004:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.038877+0000", 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 365.02050079200001, 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65003887000000005, 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204341 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038877+0000", 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038877+0000", 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038874+0000", 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5920000000000003e-06 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038882+0000", 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2948e-05 2024-02-12T16:22:01.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038895+0000", 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0270000000000001e-06 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.038901+0000", 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00044145399999999998 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.039342+0000", 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5527999999999999e-05 2024-02-12T16:22:01.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.039418+0000", 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2207999999999996e-05 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.039500+0000", 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022929280000000001 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.041793+0000", 2024-02-12T16:22:01.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.64697973399999997 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.688773+0000", 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4234999999999997e-05 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.688847+0000", 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9132000000000002e-05 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.688916+0000", 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65003887000000005 2024-02-12T16:22:01.143 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:205428 4.6 4:6d427ab9:::benchmark_data_smithi169_60938_object32091:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:57.662172+0000", 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 363.39720547899998, 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22755656099999999, 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.144 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 205428 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662172+0000", 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662172+0000", 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662170+0000", 2024-02-12T16:22:01.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7810000000000005e-06 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662178+0000", 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2616000000000001e-05 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662190+0000", 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9349999999999997e-06 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.662197+0000", 2024-02-12T16:22:01.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.078460746999999997 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.740658+0000", 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8795999999999995e-05 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.740747+0000", 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0286999999999998e-05 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.740837+0000", 2024-02-12T16:22:01.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012130958000000001 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.752968+0000", 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13665096099999999 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889619+0000", 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2207999999999998e-05 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889681+0000", 2024-02-12T16:22:01.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7828999999999999e-05 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889729+0000", 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22755656099999999 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:205451 4.6 4:6f821f24:::benchmark_data_smithi169_60938_object32114:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:57.683927+0000", 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 363.37545058400002, 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20594626199999999, 2024-02-12T16:22:01.149 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 205451 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683927+0000", 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683927+0000", 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683925+0000", 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8709999999999997e-06 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683933+0000", 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3066e-05 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683946+0000", 2024-02-12T16:22:01.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4609999999999996e-06 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.683952+0000", 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060822810999999997 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.744775+0000", 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000128672 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.744904+0000", 2024-02-12T16:22:01.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016349499999999999 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.745067+0000", 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011550878000000001 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.756618+0000", 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13317361799999999 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889792+0000", 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5293999999999998e-05 2024-02-12T16:22:01.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889827+0000", 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6288999999999997e-05 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:57.889873+0000", 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20594626199999999 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:207829 4.6 4:6d04706a:::benchmark_data_smithi169_60938_object34492:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.154 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.069986+0000", 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 360.98939162699997, 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.188431816, 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 207829 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069986+0000", 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069986+0000", 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069984+0000", 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3629999999999999e-06 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069991+0000", 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-02-12T16:22:01.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.069999+0000", 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6959999999999996e-06 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.070004+0000", 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.091331159999999995 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.161335+0000", 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015471500000000001 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.161490+0000", 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017501499999999999 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.161665+0000", 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065059690000000003 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.168171+0000", 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.090183764 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.258355+0000", 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6961000000000001e-05 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.258391+0000", 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6519000000000001e-05 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.258418+0000", 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.188431816 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.159 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:208988 4.6 4:69363d5c:::benchmark_data_smithi169_60938_object35651:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:01.468089+0000", 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 359.59128906299998, 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19592237500000001, 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 208988 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.160 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468089+0000", 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468089+0000", 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468086+0000", 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2191000000000001e-05 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468099+0000", 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5423000000000001e-05 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468114+0000", 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0219999999999999e-06 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.468121+0000", 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.103600845 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.571722+0000", 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9457999999999997e-05 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.571801+0000", 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3430000000000006e-05 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.571885+0000", 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013916245000000001 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.585801+0000", 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.078146683999999994 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.663948+0000", 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5927999999999999e-05 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.663984+0000", 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7572e-05 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.164 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:01.664011+0000", 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19592237500000001 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "rep_scrubmap(1.a e368 from shard 6)", 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:20:42.118952+0000", 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.940425382000001, 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012213059999999999, 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.118952+0000", 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.118952+0000", 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999957 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.118948+0000", 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7520000000000002e-05 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.118966+0000", 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1402999999999997e-05 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.118997+0000", 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8882e-05 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.119016+0000", 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001057103 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.167 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.120073+0000", 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3853e-05 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.120097+0000", 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6688999999999997e-05 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.168 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:20:42.120174+0000", 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012213059999999999 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "replica_scrub(pg: 4.d,from:0'0,to:414'19318,epoch:491/334,start:4:bd14bc95:::benchmark_data_smithi169_60938_object139795:0,end:4:bd529c26:::benchmark_data_smithi169_60938_object161689:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:21:32.117514+0000", 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.941863875999999, 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038290299999999998, 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:01.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117514+0000", 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117514+0000", 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117511+0000", 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4970999999999999e-05 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117526+0000", 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7135000000000003e-05 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117563+0000", 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9726999999999999e-05 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117583+0000", 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018554500000000001 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.171 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117768+0000", 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0329999999999999e-05 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117798+0000", 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8413999999999997e-05 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.117897+0000", 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038290299999999998 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.172 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.173 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:01.173 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:01.173 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:22:01.173 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:00 smithi123 bash[19119]: cluster 2024-02-12T16:21:59.872567+0000 mgr.y (mgr.14146) 1105 : cluster [DBG] pgmap v1498: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:22:01.174 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:00 smithi123 bash[19119]: audit 2024-02-12T16:22:00.609738+0000 mon.a (mon.0) 1799 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'4259','3':'52999','10':'57244','11':'38993'}"}]: dispatch 2024-02-12T16:22:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:01.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:01.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:01.287 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:00 smithi169 bash[19604]: cluster 2024-02-12T16:21:59.872567+0000 mgr.y (mgr.14146) 1105 : cluster [DBG] pgmap v1498: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:22:01.288 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:00 smithi169 bash[19604]: audit 2024-02-12T16:22:00.609738+0000 mon.a (mon.0) 1799 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'4259','3':'52999','10':'57244','11':'38993'}"}]: dispatch 2024-02-12T16:22:01.312 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:01.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:00 smithi121 bash[18489]: cluster 2024-02-12T16:21:59.872567+0000 mgr.y (mgr.14146) 1105 : cluster [DBG] pgmap v1498: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:22:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:00 smithi121 bash[18489]: audit 2024-02-12T16:22:00.609738+0000 mon.a (mon.0) 1799 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'4259','3':'52999','10':'57244','11':'38993'}"}]: dispatch 2024-02-12T16:22:01.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:01.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:01.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:01.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:01.568 INFO:teuthology.orchestra.run.smithi121.stdout:moved 4 / 74 (5.40541%) 2024-02-12T16:22:01.568 INFO:teuthology.orchestra.run.smithi121.stdout:avg 6.72727 2024-02-12T16:22:01.568 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 3.38201 -> 3.44976 (expected baseline 2.47299) 2024-02-12T16:22:01.568 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.1 with 1 -> 0 pgs (0.148649 -> 0 * mean) 2024-02-12T16:22:01.568 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.10 with 12 -> 12 pgs (1.78378 -> 1.78378 * mean) 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:oload 110 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 3 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0081 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.0089 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11 weight 1.0000 -> 0.5950 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 weight 0.1039 -> 0.0650 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 weight 1.0000 -> 0.8087 2024-02-12T16:22:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10 weight 1.0000 -> 0.8735 2024-02-12T16:22:01.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:01.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:01.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:01.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:01.722 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:01.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:01.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-12T16:22:01.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:01.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:01.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:01.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:01 smithi169 bash[19604]: audit 2024-02-12T16:22:00.609280+0000 mgr.y (mgr.14146) 1106 : audit [DBG] from='client.15921 v1:172.21.15.121:0/1812134529' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:01.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:01 smithi169 bash[19604]: audit 2024-02-12T16:22:01.562968+0000 mon.a (mon.0) 1800 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'4259','3':'52999','10':'57244','11':'38993'}"}]': finished 2024-02-12T16:22:01.977 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:01 smithi169 bash[19604]: cluster 2024-02-12T16:22:01.563099+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e498: 12 total, 11 up, 12 in 2024-02-12T16:22:02.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:02.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:02.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:02.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:02.279 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:02.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:02.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:01 smithi123 bash[19119]: audit 2024-02-12T16:22:00.609280+0000 mgr.y (mgr.14146) 1106 : audit [DBG] from='client.15921 v1:172.21.15.121:0/1812134529' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:02.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:01 smithi123 bash[19119]: audit 2024-02-12T16:22:01.562968+0000 mon.a (mon.0) 1800 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'4259','3':'52999','10':'57244','11':'38993'}"}]': finished 2024-02-12T16:22:02.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:01 smithi123 bash[19119]: cluster 2024-02-12T16:22:01.563099+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e498: 12 total, 11 up, 12 in 2024-02-12T16:22:02.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:01 smithi121 bash[18489]: audit 2024-02-12T16:22:00.609280+0000 mgr.y (mgr.14146) 1106 : audit [DBG] from='client.15921 v1:172.21.15.121:0/1812134529' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:02.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:01 smithi121 bash[18489]: audit 2024-02-12T16:22:01.562968+0000 mon.a (mon.0) 1800 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'4259','3':'52999','10':'57244','11':'38993'}"}]': finished 2024-02-12T16:22:02.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:01 smithi121 bash[18489]: cluster 2024-02-12T16:22:01.563099+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e498: 12 total, 11 up, 12 in 2024-02-12T16:22:02.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:02.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:02.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:02.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:02.708 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:02.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:02.847 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:02.857 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-12T16:22:02.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-02-12T16:22:02.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:02.988 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:02.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:03.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:03.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:02 smithi121 bash[18489]: cluster 2024-02-12T16:22:01.873589+0000 mgr.y (mgr.14146) 1107 : cluster [DBG] pgmap v1500: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:22:03.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:02 smithi121 bash[18489]: cluster 2024-02-12T16:22:02.563193+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 12 total, 11 up, 12 in 2024-02-12T16:22:03.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:03.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:03.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:03.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:03.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:02 smithi169 bash[19604]: cluster 2024-02-12T16:22:01.873589+0000 mgr.y (mgr.14146) 1107 : cluster [DBG] pgmap v1500: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:22:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:02 smithi169 bash[19604]: cluster 2024-02-12T16:22:02.563193+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 12 total, 11 up, 12 in 2024-02-12T16:22:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:03.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:02 smithi123 bash[19119]: cluster 2024-02-12T16:22:01.873589+0000 mgr.y (mgr.14146) 1107 : cluster [DBG] pgmap v1500: 37 pgs: 1 active+clean+remapped, 4 undersized+degraded+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8797/70012 objects degraded (12.565%); 2246/70012 objects misplaced (3.208%) 2024-02-12T16:22:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:02 smithi123 bash[19119]: cluster 2024-02-12T16:22:02.563193+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e499: 12 total, 11 up, 12 in 2024-02-12T16:22:03.401 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:03.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:03.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:03.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:03.542 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:03.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:03.589 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:22:03.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:03.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:03.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:03.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:03.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:03.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:03.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:04.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:04.070 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:04.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:04.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:04.201 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:04.203 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:03 smithi169 bash[19604]: cluster 2024-02-12T16:22:02.842448+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:04.203 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:03 smithi169 bash[19604]: cluster 2024-02-12T16:22:02.842540+0000 osd.8 (osd.8) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:04.203 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:03 smithi169 bash[19604]: audit 2024-02-12T16:22:03.290446+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? v1:172.21.15.121:0/962546229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:04.203 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:03 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.569791+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:04.203 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:03 smithi169 bash[19604]: audit 2024-02-12T16:22:03.578132+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.121:0/962546229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:04.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:03 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.578359+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 12 total, 11 up, 12 in 2024-02-12T16:22:04.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:03 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.580453+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b starting backfill to osd.4 from (0'0,0'0] MAX to 416'19630 2024-02-12T16:22:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:04.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:04.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:04.313 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:03 smithi121 bash[18489]: cluster 2024-02-12T16:22:02.842448+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:04.313 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:03 smithi121 bash[18489]: cluster 2024-02-12T16:22:02.842540+0000 osd.8 (osd.8) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:04.313 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:03 smithi121 bash[18489]: audit 2024-02-12T16:22:03.290446+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? v1:172.21.15.121:0/962546229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:04.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:03 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.569791+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:04.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:03 smithi121 bash[18489]: audit 2024-02-12T16:22:03.578132+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.121:0/962546229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:04.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:03 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.578359+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 12 total, 11 up, 12 in 2024-02-12T16:22:04.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:03 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.580453+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b starting backfill to osd.4 from (0'0,0'0] MAX to 416'19630 2024-02-12T16:22:04.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:04.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:03 smithi123 bash[19119]: cluster 2024-02-12T16:22:02.842448+0000 osd.8 (osd.8) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:03 smithi123 bash[19119]: cluster 2024-02-12T16:22:02.842540+0000 osd.8 (osd.8) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:03 smithi123 bash[19119]: audit 2024-02-12T16:22:03.290446+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? v1:172.21.15.121:0/962546229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:03 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.569791+0000 mon.a (mon.0) 1804 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:03 smithi123 bash[19119]: audit 2024-02-12T16:22:03.578132+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? v1:172.21.15.121:0/962546229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:03 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.578359+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e500: 12 total, 11 up, 12 in 2024-02-12T16:22:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:03 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.580453+0000 osd.0 (osd.0) 77 : cluster [DBG] 4.b starting backfill to osd.4 from (0'0,0'0] MAX to 416'19630 2024-02-12T16:22:04.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:04.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:04.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:04.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:04.600 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:04.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:04.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:04.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:04.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:04.869 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:04.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:04.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:05.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:05.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:05.136 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.579065+0000 osd.11 (osd.11) 105 : cluster [DBG] 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 414'19577 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.579377+0000 osd.11 (osd.11) 106 : cluster [DBG] 4.f starting backfill to osd.9 from (0'0,0'0] MAX to 414'19285 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.581711+0000 osd.7 (osd.7) 111 : cluster [DBG] 4.0 starting backfill to osd.0 from (0'0,0'0] MAX to 414'19567 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.582064+0000 osd.7 (osd.7) 112 : cluster [DBG] 4.3 starting backfill to osd.0 from (0'0,0'0] MAX to 416'19327 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.615922+0000 osd.11 (osd.11) 107 : cluster [DBG] 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 414'19577 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:03.874188+0000 mgr.y (mgr.14146) 1108 : cluster [DBG] pgmap v1503: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4411/65766 objects degraded (6.707%); 2246/65766 objects misplaced (3.415%) 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:04.572024+0000 mon.a (mon.0) 1807 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:22:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:04 smithi121 bash[18489]: cluster 2024-02-12T16:22:04.572061+0000 mon.a (mon.0) 1808 : cluster [WRN] Health check update: Degraded data redundancy: 4411/65766 objects degraded (6.707%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:05.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:05.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.579065+0000 osd.11 (osd.11) 105 : cluster [DBG] 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 414'19577 2024-02-12T16:22:05.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.579377+0000 osd.11 (osd.11) 106 : cluster [DBG] 4.f starting backfill to osd.9 from (0'0,0'0] MAX to 414'19285 2024-02-12T16:22:05.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.581711+0000 osd.7 (osd.7) 111 : cluster [DBG] 4.0 starting backfill to osd.0 from (0'0,0'0] MAX to 414'19567 2024-02-12T16:22:05.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.582064+0000 osd.7 (osd.7) 112 : cluster [DBG] 4.3 starting backfill to osd.0 from (0'0,0'0] MAX to 416'19327 2024-02-12T16:22:05.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.615922+0000 osd.11 (osd.11) 107 : cluster [DBG] 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 414'19577 2024-02-12T16:22:05.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:03.874188+0000 mgr.y (mgr.14146) 1108 : cluster [DBG] pgmap v1503: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4411/65766 objects degraded (6.707%); 2246/65766 objects misplaced (3.415%) 2024-02-12T16:22:05.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:04.572024+0000 mon.a (mon.0) 1807 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:22:05.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:04 smithi123 bash[19119]: cluster 2024-02-12T16:22:04.572061+0000 mon.a (mon.0) 1808 : cluster [WRN] Health check update: Degraded data redundancy: 4411/65766 objects degraded (6.707%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:05.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.579065+0000 osd.11 (osd.11) 105 : cluster [DBG] 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 414'19577 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.579377+0000 osd.11 (osd.11) 106 : cluster [DBG] 4.f starting backfill to osd.9 from (0'0,0'0] MAX to 414'19285 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.581711+0000 osd.7 (osd.7) 111 : cluster [DBG] 4.0 starting backfill to osd.0 from (0'0,0'0] MAX to 414'19567 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.582064+0000 osd.7 (osd.7) 112 : cluster [DBG] 4.3 starting backfill to osd.0 from (0'0,0'0] MAX to 416'19327 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.615922+0000 osd.11 (osd.11) 107 : cluster [DBG] 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 414'19577 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:03.874188+0000 mgr.y (mgr.14146) 1108 : cluster [DBG] pgmap v1503: 37 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 undersized+degraded+peered, 3 undersized+peered, 28 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4411/65766 objects degraded (6.707%); 2246/65766 objects misplaced (3.415%) 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:04.572024+0000 mon.a (mon.0) 1807 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:22:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:04 smithi169 bash[19604]: cluster 2024-02-12T16:22:04.572061+0000 mon.a (mon.0) 1808 : cluster [WRN] Health check update: Degraded data redundancy: 4411/65766 objects degraded (6.707%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:05.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:05.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:05.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:05.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:05.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:05.650 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:05.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:05.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:05.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:05.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:05.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:06.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:06.057 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:06.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:06.200 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:06.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:06.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:06.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:06.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:06.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:06.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:06.644 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:22:06.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:06.705 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:06 smithi123 bash[19119]: cluster 2024-02-12T16:22:05.552497+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e501: 12 total, 11 up, 12 in 2024-02-12T16:22:06.705 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:06 smithi123 bash[19119]: cluster 2024-02-12T16:22:05.874956+0000 mgr.y (mgr.14146) 1109 : cluster [DBG] pgmap v1505: 37 pgs: 1 undersized+degraded+peered, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 remapped+peering, 1 active+clean+remapped, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6534/70012 objects degraded (9.333%); 8267/70012 objects misplaced (11.808%); 6.6 MiB/s, 106 objects/s recovering 2024-02-12T16:22:06.720 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:06.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:22:06.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:06.847 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:06.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:06 smithi121 bash[18489]: cluster 2024-02-12T16:22:05.552497+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e501: 12 total, 11 up, 12 in 2024-02-12T16:22:06.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:06 smithi121 bash[18489]: cluster 2024-02-12T16:22:05.874956+0000 mgr.y (mgr.14146) 1109 : cluster [DBG] pgmap v1505: 37 pgs: 1 undersized+degraded+peered, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 remapped+peering, 1 active+clean+remapped, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6534/70012 objects degraded (9.333%); 8267/70012 objects misplaced (11.808%); 6.6 MiB/s, 106 objects/s recovering 2024-02-12T16:22:06.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:06.963 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:06 smithi169 bash[19604]: cluster 2024-02-12T16:22:05.552497+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e501: 12 total, 11 up, 12 in 2024-02-12T16:22:06.963 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:06 smithi169 bash[19604]: cluster 2024-02-12T16:22:05.874956+0000 mgr.y (mgr.14146) 1109 : cluster [DBG] pgmap v1505: 37 pgs: 1 undersized+degraded+peered, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 remapped+peering, 1 active+clean+remapped, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6534/70012 objects degraded (9.333%); 8267/70012 objects misplaced (11.808%); 6.6 MiB/s, 106 objects/s recovering 2024-02-12T16:22:06.988 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:06.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:07.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:07.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:07.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:07.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:07.254 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:07.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:07.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:07.396 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:07.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:07.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:07.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:07.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:07.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:07.643 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:07 smithi169 bash[19604]: cluster 2024-02-12T16:22:06.556234+0000 mon.a (mon.0) 1810 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:22:07.668 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:07.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:07.771 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:07.793 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:07.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:07.840 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:22:07.861 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:22:07.886 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:22:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:07 smithi123 bash[19119]: cluster 2024-02-12T16:22:06.556234+0000 mon.a (mon.0) 1810 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:22:07.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:07 smithi121 bash[18489]: cluster 2024-02-12T16:22:06.556234+0000 mon.a (mon.0) 1810 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:22:07.912 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:22:07.932 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:07.933 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:22:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:07.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:07.954 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:22:07.974 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:22:07.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:22:08.023 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:22:08.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:08.045 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:22:08.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:08.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:08.076 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:22:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:22:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.077 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.077 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.077 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.077 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.077 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.077 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.077 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:22:08.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:08.196 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:08.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:08.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:08.334 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:08.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:08.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:08.462 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:08.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:08.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:08.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:08.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:08.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:08.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:08.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:08.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:08.850 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:08 smithi121 bash[18489]: audit 2024-02-12T16:22:07.812858+0000 mon.b (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.121:0/2166083560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:08.850 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:08 smithi121 bash[18489]: cluster 2024-02-12T16:22:07.875554+0000 mgr.y (mgr.14146) 1110 : cluster [DBG] pgmap v1506: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+peered, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 remapped+peering, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6534/70012 objects degraded (9.333%); 8267/70012 objects misplaced (11.808%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:22:08.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:08.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:08 smithi123 bash[19119]: audit 2024-02-12T16:22:07.812858+0000 mon.b (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.121:0/2166083560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:08 smithi123 bash[19119]: cluster 2024-02-12T16:22:07.875554+0000 mgr.y (mgr.14146) 1110 : cluster [DBG] pgmap v1506: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+peered, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 remapped+peering, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6534/70012 objects degraded (9.333%); 8267/70012 objects misplaced (11.808%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:22:08.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:08.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:08 smithi169 bash[19604]: audit 2024-02-12T16:22:07.812858+0000 mon.b (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.121:0/2166083560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:08.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:08 smithi169 bash[19604]: cluster 2024-02-12T16:22:07.875554+0000 mgr.y (mgr.14146) 1110 : cluster [DBG] pgmap v1506: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+peered, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 2 remapped+peering, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6534/70012 objects degraded (9.333%); 8267/70012 objects misplaced (11.808%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:22:09.006 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:22:09.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:22:09.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:09.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:09.242 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:09.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:09.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:09.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:09.502 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:09.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:09.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:09.651 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:09.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:09 smithi169 bash[19604]: cluster 2024-02-12T16:22:08.568952+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e502: 12 total, 11 up, 12 in 2024-02-12T16:22:09.684 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:09.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:09.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:09 smithi123 bash[19119]: cluster 2024-02-12T16:22:08.568952+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e502: 12 total, 11 up, 12 in 2024-02-12T16:22:09.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:09 smithi121 bash[18489]: cluster 2024-02-12T16:22:08.568952+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e502: 12 total, 11 up, 12 in 2024-02-12T16:22:09.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:09.951 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:09.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:10.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:10.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:10.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:10.112 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:22:10.112 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:22:10.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:10.225 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:10.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:10.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:10.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:10.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:10.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:10.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:10.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:10.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:10.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:10.744 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:10.768 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:10.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:10.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:10 smithi169 bash[19604]: cluster 2024-02-12T16:22:09.571574+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e503: 12 total, 11 up, 12 in 2024-02-12T16:22:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:10 smithi169 bash[19604]: cluster 2024-02-12T16:22:09.876624+0000 mgr.y (mgr.14146) 1111 : cluster [DBG] pgmap v1509: 37 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 active+clean+remapped, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 3.21k op/s; 8748/70012 objects degraded (12.495%); 8267/70012 objects misplaced (11.808%); 75 MiB/s, 1.20k objects/s recovering 2024-02-12T16:22:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:10 smithi169 bash[19604]: cluster 2024-02-12T16:22:10.548651+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:22:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:10 smithi169 bash[19604]: cluster 2024-02-12T16:22:10.548721+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Degraded data redundancy: 6534/70012 objects degraded (9.333%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:10.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-12T16:22:10.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:10.879 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:10 smithi121 bash[18489]: cluster 2024-02-12T16:22:09.571574+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e503: 12 total, 11 up, 12 in 2024-02-12T16:22:10.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:10 smithi121 bash[18489]: cluster 2024-02-12T16:22:09.876624+0000 mgr.y (mgr.14146) 1111 : cluster [DBG] pgmap v1509: 37 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 active+clean+remapped, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 3.21k op/s; 8748/70012 objects degraded (12.495%); 8267/70012 objects misplaced (11.808%); 75 MiB/s, 1.20k objects/s recovering 2024-02-12T16:22:10.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:10 smithi121 bash[18489]: cluster 2024-02-12T16:22:10.548651+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:22:10.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:10 smithi121 bash[18489]: cluster 2024-02-12T16:22:10.548721+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Degraded data redundancy: 6534/70012 objects degraded (9.333%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:10.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:10 smithi123 bash[19119]: cluster 2024-02-12T16:22:09.571574+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e503: 12 total, 11 up, 12 in 2024-02-12T16:22:10.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:10 smithi123 bash[19119]: cluster 2024-02-12T16:22:09.876624+0000 mgr.y (mgr.14146) 1111 : cluster [DBG] pgmap v1509: 37 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 active+clean+remapped, 1 active+remapped+backfilling, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 3.21k op/s; 8748/70012 objects degraded (12.495%); 8267/70012 objects misplaced (11.808%); 75 MiB/s, 1.20k objects/s recovering 2024-02-12T16:22:10.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:10 smithi123 bash[19119]: cluster 2024-02-12T16:22:10.548651+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:22:10.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:10 smithi123 bash[19119]: cluster 2024-02-12T16:22:10.548721+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check update: Degraded data redundancy: 6534/70012 objects degraded (9.333%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:10.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:10.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:11.009 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:11.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:11.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:11.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:22:11.148 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:11.180 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:11.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:11.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:11.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:11.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:11.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:11.458 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:11.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:11.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:11.580 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:11.603 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:11.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:11.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:11.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:11.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:11.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:11.858 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:11 smithi123 bash[19119]: cluster 2024-02-12T16:22:10.573240+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e504: 12 total, 11 up, 12 in 2024-02-12T16:22:11.880 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:11.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:11.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:11 smithi121 bash[18489]: cluster 2024-02-12T16:22:10.573240+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e504: 12 total, 11 up, 12 in 2024-02-12T16:22:11.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:12.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:12.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:11 smithi169 bash[19604]: cluster 2024-02-12T16:22:10.573240+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e504: 12 total, 11 up, 12 in 2024-02-12T16:22:12.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:12.157 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:12.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:12.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:12.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:12.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:12.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:12.574 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:12.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:12.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:12.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:12 smithi169 bash[19604]: cluster 2024-02-12T16:22:11.877239+0000 mgr.y (mgr.14146) 1112 : cluster [DBG] pgmap v1511: 37 pgs: 1 peering, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8748/70012 objects degraded (12.495%); 6552/70012 objects misplaced (9.358%); 41 MiB/s, 655 objects/s recovering 2024-02-12T16:22:12.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:12.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:12 smithi123 bash[19119]: cluster 2024-02-12T16:22:11.877239+0000 mgr.y (mgr.14146) 1112 : cluster [DBG] pgmap v1511: 37 pgs: 1 peering, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8748/70012 objects degraded (12.495%); 6552/70012 objects misplaced (9.358%); 41 MiB/s, 655 objects/s recovering 2024-02-12T16:22:12.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:12.995 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:12.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:13.067 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:12 smithi121 bash[18489]: cluster 2024-02-12T16:22:11.877239+0000 mgr.y (mgr.14146) 1112 : cluster [DBG] pgmap v1511: 37 pgs: 1 peering, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 27 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8748/70012 objects degraded (12.495%); 6552/70012 objects misplaced (9.358%); 41 MiB/s, 655 objects/s recovering 2024-02-12T16:22:13.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:13.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:13.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:13.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:13.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:13.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:13.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:13.415 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:13.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:13.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:13.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:13.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:13.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:13.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:13.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:13.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:13.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:13.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:13.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:13.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:14.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:14.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:14.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:14.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:14.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:14.243 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:14.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:14.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:14.387 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:14.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:14.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:14.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:14.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:14.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:14.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:14.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:14.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:14.799 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:14.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:14.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:14.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:14.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:14.949 INFO:teuthology.orchestra.run.smithi121.stdout:moved 2 / 74 (2.7027%) 2024-02-12T16:22:14.949 INFO:teuthology.orchestra.run.smithi121.stdout:avg 6.72727 2024-02-12T16:22:14.949 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 3.44976 -> 3.52793 (expected baseline 2.47299) 2024-02-12T16:22:14.949 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.1 with 0 -> 0 pgs (0 -> 0 * mean) 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.10 with 12 -> 11 pgs (1.78378 -> 1.63514 * mean) 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:oload 110 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 3 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0084 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.0092 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11 weight 0.5950 -> 0.3657 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 weight 0.0650 -> 0.0420 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 weight 0.8087 -> 0.6746 2024-02-12T16:22:14.950 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10 weight 0.8735 -> 0.7869 2024-02-12T16:22:15.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:14 smithi169 bash[19604]: cluster 2024-02-12T16:22:13.877861+0000 mgr.y (mgr.14146) 1113 : cluster [DBG] pgmap v1512: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 peering, 1 undersized+degraded+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 42 MiB/s, 665 objects/s recovering 2024-02-12T16:22:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:14 smithi169 bash[19604]: audit 2024-02-12T16:22:14.054759+0000 mgr.y (mgr.14146) 1114 : audit [DBG] from='client.15939 v1:172.21.15.121:0/934379808' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:14 smithi169 bash[19604]: audit 2024-02-12T16:22:14.055748+0000 mon.a (mon.0) 1816 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'2751','3':'44209','10':'51570','11':'23964'}"}]: dispatch 2024-02-12T16:22:15.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:15.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:15.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:15.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:14 smithi123 bash[19119]: cluster 2024-02-12T16:22:13.877861+0000 mgr.y (mgr.14146) 1113 : cluster [DBG] pgmap v1512: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 peering, 1 undersized+degraded+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 42 MiB/s, 665 objects/s recovering 2024-02-12T16:22:15.192 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:14 smithi123 bash[19119]: audit 2024-02-12T16:22:14.054759+0000 mgr.y (mgr.14146) 1114 : audit [DBG] from='client.15939 v1:172.21.15.121:0/934379808' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:15.192 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:14 smithi123 bash[19119]: audit 2024-02-12T16:22:14.055748+0000 mon.a (mon.0) 1816 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'2751','3':'44209','10':'51570','11':'23964'}"}]: dispatch 2024-02-12T16:22:15.213 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:15.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:15.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:15.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:15.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:14 smithi121 bash[18489]: cluster 2024-02-12T16:22:13.877861+0000 mgr.y (mgr.14146) 1113 : cluster [DBG] pgmap v1512: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 peering, 1 undersized+degraded+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 42 MiB/s, 665 objects/s recovering 2024-02-12T16:22:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:14 smithi121 bash[18489]: audit 2024-02-12T16:22:14.054759+0000 mgr.y (mgr.14146) 1114 : audit [DBG] from='client.15939 v1:172.21.15.121:0/934379808' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:14 smithi121 bash[18489]: audit 2024-02-12T16:22:14.055748+0000 mon.a (mon.0) 1816 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'2751','3':'44209','10':'51570','11':'23964'}"}]: dispatch 2024-02-12T16:22:15.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:15.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:15.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:15.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:15.622 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:22:15.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:15.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:15.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:15.779 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:15.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:15.784 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-02-12T16:22:15.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-02-12T16:22:15.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:15.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:15.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:16.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:16.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:16.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:16.159 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:15 smithi121 bash[18489]: audit 2024-02-12T16:22:14.944419+0000 mon.a (mon.0) 1817 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'2751','3':'44209','10':'51570','11':'23964'}"}]': finished 2024-02-12T16:22:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:15 smithi121 bash[18489]: cluster 2024-02-12T16:22:14.944481+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e505: 12 total, 11 up, 12 in 2024-02-12T16:22:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:15 smithi121 bash[18489]: cluster 2024-02-12T16:22:15.550498+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:22:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:15 smithi121 bash[18489]: cluster 2024-02-12T16:22:15.550555+0000 mon.a (mon.0) 1820 : cluster [WRN] Health check update: Degraded data redundancy: 8693/70012 objects degraded (12.416%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:15 smithi121 bash[18489]: cluster 2024-02-12T16:22:15.561912+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e506: 12 total, 11 up, 12 in 2024-02-12T16:22:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:15 smithi121 bash[18489]: audit 2024-02-12T16:22:15.912844+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.121:0/4137383793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:22:16.170 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:15 smithi123 bash[19119]: audit 2024-02-12T16:22:14.944419+0000 mon.a (mon.0) 1817 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'2751','3':'44209','10':'51570','11':'23964'}"}]': finished 2024-02-12T16:22:16.170 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:15 smithi123 bash[19119]: cluster 2024-02-12T16:22:14.944481+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e505: 12 total, 11 up, 12 in 2024-02-12T16:22:16.170 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:15 smithi123 bash[19119]: cluster 2024-02-12T16:22:15.550498+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:22:16.170 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:15 smithi123 bash[19119]: cluster 2024-02-12T16:22:15.550555+0000 mon.a (mon.0) 1820 : cluster [WRN] Health check update: Degraded data redundancy: 8693/70012 objects degraded (12.416%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:16.171 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:15 smithi123 bash[19119]: cluster 2024-02-12T16:22:15.561912+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e506: 12 total, 11 up, 12 in 2024-02-12T16:22:16.171 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:15 smithi123 bash[19119]: audit 2024-02-12T16:22:15.912844+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.121:0/4137383793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:22:16.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:16.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:16.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:15 smithi169 bash[19604]: audit 2024-02-12T16:22:14.944419+0000 mon.a (mon.0) 1817 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'2751','3':'44209','10':'51570','11':'23964'}"}]': finished 2024-02-12T16:22:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:15 smithi169 bash[19604]: cluster 2024-02-12T16:22:14.944481+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e505: 12 total, 11 up, 12 in 2024-02-12T16:22:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:15 smithi169 bash[19604]: cluster 2024-02-12T16:22:15.550498+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:22:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:15 smithi169 bash[19604]: cluster 2024-02-12T16:22:15.550555+0000 mon.a (mon.0) 1820 : cluster [WRN] Health check update: Degraded data redundancy: 8693/70012 objects degraded (12.416%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:15 smithi169 bash[19604]: cluster 2024-02-12T16:22:15.561912+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e506: 12 total, 11 up, 12 in 2024-02-12T16:22:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:15 smithi169 bash[19604]: audit 2024-02-12T16:22:15.912844+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.121:0/4137383793' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:22:16.318 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:16.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:16.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:16.456 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:16.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:16.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:16.573 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:22:16.586 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:16.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:16.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:16.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:16.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:16.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:16.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:16.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:16.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:17.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:17.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:17.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:17.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:17.259 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:16 smithi123 bash[19119]: cluster 2024-02-12T16:22:15.878761+0000 mgr.y (mgr.14146) 1115 : cluster [DBG] pgmap v1515: 37 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 18 MiB/s, 294 objects/s recovering 2024-02-12T16:22:17.260 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:16 smithi123 bash[19119]: cluster 2024-02-12T16:22:16.554937+0000 mon.a (mon.0) 1823 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:17.260 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:16 smithi123 bash[19119]: audit 2024-02-12T16:22:16.562819+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? v1:172.21.15.121:0/4137383793' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:22:17.260 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:16 smithi123 bash[19119]: cluster 2024-02-12T16:22:16.562933+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e507: 12 total, 11 up, 12 in 2024-02-12T16:22:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:17.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:16 smithi169 bash[19604]: cluster 2024-02-12T16:22:15.878761+0000 mgr.y (mgr.14146) 1115 : cluster [DBG] pgmap v1515: 37 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 18 MiB/s, 294 objects/s recovering 2024-02-12T16:22:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:16 smithi169 bash[19604]: cluster 2024-02-12T16:22:16.554937+0000 mon.a (mon.0) 1823 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:16 smithi169 bash[19604]: audit 2024-02-12T16:22:16.562819+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? v1:172.21.15.121:0/4137383793' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:22:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:16 smithi169 bash[19604]: cluster 2024-02-12T16:22:16.562933+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e507: 12 total, 11 up, 12 in 2024-02-12T16:22:17.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:17.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:16 smithi121 bash[18489]: cluster 2024-02-12T16:22:15.878761+0000 mgr.y (mgr.14146) 1115 : cluster [DBG] pgmap v1515: 37 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 18 MiB/s, 294 objects/s recovering 2024-02-12T16:22:17.389 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:16 smithi121 bash[18489]: cluster 2024-02-12T16:22:16.554937+0000 mon.a (mon.0) 1823 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:17.389 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:16 smithi121 bash[18489]: audit 2024-02-12T16:22:16.562819+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? v1:172.21.15.121:0/4137383793' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:22:17.389 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:16 smithi121 bash[18489]: cluster 2024-02-12T16:22:16.562933+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e507: 12 total, 11 up, 12 in 2024-02-12T16:22:17.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:17.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:17.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:17.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:17.690 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:17.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:17.825 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:17.834 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:17.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:17.961 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:17.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:18.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:18.103 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:18.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:17 smithi123 bash[19119]: cluster 2024-02-12T16:22:16.566883+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 416'19451 2024-02-12T16:22:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:17 smithi123 bash[19119]: cluster 2024-02-12T16:22:16.744792+0000 osd.6 (osd.6) 88 : cluster [DBG] 1.f scrub starts 2024-02-12T16:22:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:17 smithi123 bash[19119]: cluster 2024-02-12T16:22:16.766309+0000 osd.6 (osd.6) 89 : cluster [DBG] 1.f scrub ok 2024-02-12T16:22:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:17 smithi123 bash[19119]: cluster 2024-02-12T16:22:16.956785+0000 osd.5 (osd.5) 71 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:22:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:17 smithi123 bash[19119]: cluster 2024-02-12T16:22:16.999818+0000 osd.5 (osd.5) 72 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:22:18.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:18.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:18.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:17 smithi169 bash[19604]: cluster 2024-02-12T16:22:16.566883+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 416'19451 2024-02-12T16:22:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:17 smithi169 bash[19604]: cluster 2024-02-12T16:22:16.744792+0000 osd.6 (osd.6) 88 : cluster [DBG] 1.f scrub starts 2024-02-12T16:22:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:17 smithi169 bash[19604]: cluster 2024-02-12T16:22:16.766309+0000 osd.6 (osd.6) 89 : cluster [DBG] 1.f scrub ok 2024-02-12T16:22:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:17 smithi169 bash[19604]: cluster 2024-02-12T16:22:16.956785+0000 osd.5 (osd.5) 71 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:22:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:17 smithi169 bash[19604]: cluster 2024-02-12T16:22:16.999818+0000 osd.5 (osd.5) 72 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:22:18.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:18.378 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:18.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:17 smithi121 bash[18489]: cluster 2024-02-12T16:22:16.566883+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 416'19451 2024-02-12T16:22:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:17 smithi121 bash[18489]: cluster 2024-02-12T16:22:16.744792+0000 osd.6 (osd.6) 88 : cluster [DBG] 1.f scrub starts 2024-02-12T16:22:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:17 smithi121 bash[18489]: cluster 2024-02-12T16:22:16.766309+0000 osd.6 (osd.6) 89 : cluster [DBG] 1.f scrub ok 2024-02-12T16:22:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:17 smithi121 bash[18489]: cluster 2024-02-12T16:22:16.956785+0000 osd.5 (osd.5) 71 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:22:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:17 smithi121 bash[18489]: cluster 2024-02-12T16:22:16.999818+0000 osd.5 (osd.5) 72 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:22:18.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:18.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:18.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:18.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:18.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:18.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:18.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:18.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:18.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:18.896 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:18.924 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:18.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:19.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:19.036 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:22:19.036 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:22:19.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:19.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:19.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:19.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:18 smithi121 bash[18489]: cluster 2024-02-12T16:22:17.805130+0000 osd.2 (osd.2) 58 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:22:19.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:18 smithi121 bash[18489]: cluster 2024-02-12T16:22:17.879593+0000 mgr.y (mgr.14146) 1116 : cluster [DBG] pgmap v1517: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 586 KiB/s, 9 objects/s recovering 2024-02-12T16:22:19.163 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:19.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:19.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:19.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:19.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:19.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:18 smithi169 bash[19604]: cluster 2024-02-12T16:22:17.805130+0000 osd.2 (osd.2) 58 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:22:19.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:18 smithi169 bash[19604]: cluster 2024-02-12T16:22:17.879593+0000 mgr.y (mgr.14146) 1116 : cluster [DBG] pgmap v1517: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 586 KiB/s, 9 objects/s recovering 2024-02-12T16:22:19.328 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:19.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:18 smithi123 bash[19119]: cluster 2024-02-12T16:22:17.805130+0000 osd.2 (osd.2) 58 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:22:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:18 smithi123 bash[19119]: cluster 2024-02-12T16:22:17.879593+0000 mgr.y (mgr.14146) 1116 : cluster [DBG] pgmap v1517: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 29 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8693/70012 objects degraded (12.416%); 6552/70012 objects misplaced (9.358%); 586 KiB/s, 9 objects/s recovering 2024-02-12T16:22:19.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:22:19.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:19.472 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:19.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:19.576 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:19.600 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:19.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:19.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:19.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:19.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:19.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-12T16:22:19.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:19.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:19.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:20.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:20.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:20.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:19 smithi169 bash[19604]: cluster 2024-02-12T16:22:18.224255+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:22:20.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:19 smithi169 bash[19604]: cluster 2024-02-12T16:22:18.295898+0000 osd.9 (osd.9) 77 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:22:20.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:19 smithi169 bash[19604]: cluster 2024-02-12T16:22:18.642594+0000 osd.9 (osd.9) 78 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:22:20.144 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:20.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:20.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:19 smithi121 bash[18489]: cluster 2024-02-12T16:22:18.224255+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:22:20.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:19 smithi121 bash[18489]: cluster 2024-02-12T16:22:18.295898+0000 osd.9 (osd.9) 77 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:22:20.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:19 smithi121 bash[18489]: cluster 2024-02-12T16:22:18.642594+0000 osd.9 (osd.9) 78 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:22:20.284 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:20.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:20.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:19 smithi123 bash[19119]: cluster 2024-02-12T16:22:18.224255+0000 osd.2 (osd.2) 59 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:22:20.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:19 smithi123 bash[19119]: cluster 2024-02-12T16:22:18.295898+0000 osd.9 (osd.9) 77 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:22:20.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:19 smithi123 bash[19119]: cluster 2024-02-12T16:22:18.642594+0000 osd.9 (osd.9) 78 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:22:20.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:20.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:20.555 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:20.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:20.684 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:20.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:20.825 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:20.930 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:20.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:20.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:20.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:21.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:21.096 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:21.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:21.200 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:21.228 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:21.229 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: cluster 2024-02-12T16:22:19.720557+0000 osd.9 (osd.9) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:21.229 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: cluster 2024-02-12T16:22:19.720642+0000 osd.9 (osd.9) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:21.229 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: cluster 2024-02-12T16:22:19.880365+0000 mgr.y (mgr.14146) 1117 : cluster [DBG] pgmap v1518: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8637/70012 objects degraded (12.336%); 8684/70012 objects misplaced (12.404%); 874 KiB/s, 13 objects/s recovering 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: cluster 2024-02-12T16:22:19.966623+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e508: 12 total, 11 up, 12 in 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: cluster 2024-02-12T16:22:20.553780+0000 mon.a (mon.0) 1827 : cluster [WRN] Health check update: Degraded data redundancy: 8637/70012 objects degraded (12.336%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: cluster 2024-02-12T16:22:20.561577+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e509: 12 total, 11 up, 12 in 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: audit 2024-02-12T16:22:20.815666+0000 mon.a (mon.0) 1829 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: audit 2024-02-12T16:22:20.815952+0000 mon.a (mon.0) 1830 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: audit 2024-02-12T16:22:20.816233+0000 mon.a (mon.0) 1831 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: audit 2024-02-12T16:22:20.866005+0000 mon.a (mon.0) 1832 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: audit 2024-02-12T16:22:20.868494+0000 mon.a (mon.0) 1833 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:22:21.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: audit 2024-02-12T16:22:20.869582+0000 mon.a (mon.0) 1834 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:22:21.231 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:20 smithi123 bash[19119]: audit 2024-02-12T16:22:20.876624+0000 mon.a (mon.0) 1835 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:22:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:21.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: cluster 2024-02-12T16:22:19.720557+0000 osd.9 (osd.9) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: cluster 2024-02-12T16:22:19.720642+0000 osd.9 (osd.9) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: cluster 2024-02-12T16:22:19.880365+0000 mgr.y (mgr.14146) 1117 : cluster [DBG] pgmap v1518: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8637/70012 objects degraded (12.336%); 8684/70012 objects misplaced (12.404%); 874 KiB/s, 13 objects/s recovering 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: cluster 2024-02-12T16:22:19.966623+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e508: 12 total, 11 up, 12 in 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: cluster 2024-02-12T16:22:20.553780+0000 mon.a (mon.0) 1827 : cluster [WRN] Health check update: Degraded data redundancy: 8637/70012 objects degraded (12.336%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: cluster 2024-02-12T16:22:20.561577+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e509: 12 total, 11 up, 12 in 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: audit 2024-02-12T16:22:20.815666+0000 mon.a (mon.0) 1829 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:22:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: audit 2024-02-12T16:22:20.815952+0000 mon.a (mon.0) 1830 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:22:21.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: audit 2024-02-12T16:22:20.816233+0000 mon.a (mon.0) 1831 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:22:21.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: audit 2024-02-12T16:22:20.866005+0000 mon.a (mon.0) 1832 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:22:21.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: audit 2024-02-12T16:22:20.868494+0000 mon.a (mon.0) 1833 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:22:21.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: audit 2024-02-12T16:22:20.869582+0000 mon.a (mon.0) 1834 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:22:21.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:20 smithi169 bash[19604]: audit 2024-02-12T16:22:20.876624+0000 mon.a (mon.0) 1835 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:22:21.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: cluster 2024-02-12T16:22:19.720557+0000 osd.9 (osd.9) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: cluster 2024-02-12T16:22:19.720642+0000 osd.9 (osd.9) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: cluster 2024-02-12T16:22:19.880365+0000 mgr.y (mgr.14146) 1117 : cluster [DBG] pgmap v1518: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8637/70012 objects degraded (12.336%); 8684/70012 objects misplaced (12.404%); 874 KiB/s, 13 objects/s recovering 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: cluster 2024-02-12T16:22:19.966623+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e508: 12 total, 11 up, 12 in 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: cluster 2024-02-12T16:22:20.553780+0000 mon.a (mon.0) 1827 : cluster [WRN] Health check update: Degraded data redundancy: 8637/70012 objects degraded (12.336%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: cluster 2024-02-12T16:22:20.561577+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e509: 12 total, 11 up, 12 in 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: audit 2024-02-12T16:22:20.815666+0000 mon.a (mon.0) 1829 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:22:21.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: audit 2024-02-12T16:22:20.815952+0000 mon.a (mon.0) 1830 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:22:21.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: audit 2024-02-12T16:22:20.816233+0000 mon.a (mon.0) 1831 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:22:21.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: audit 2024-02-12T16:22:20.866005+0000 mon.a (mon.0) 1832 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:22:21.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: audit 2024-02-12T16:22:20.868494+0000 mon.a (mon.0) 1833 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:22:21.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: audit 2024-02-12T16:22:20.869582+0000 mon.a (mon.0) 1834 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:22:21.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:20 smithi121 bash[18489]: audit 2024-02-12T16:22:20.876624+0000 mon.a (mon.0) 1835 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:22:21.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:21.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:21.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:21.503 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:21.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:21.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:21.632 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:21.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:21.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:21.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:21.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:21.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:21.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:21.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:22.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:22.055 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:22.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:22.181 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:22:22.195 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:22.208 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:22:22.232 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:22:22.253 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:22:22.278 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:22:22.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:22.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:22:22.330 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:22:22.330 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:22.354 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:22:22.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:22:22.401 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:22:22.426 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:22:22.426 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.426 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:22:22.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:22.454 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:22.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:22.583 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:22.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:22.687 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:22.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:22.816 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:22.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:22.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:22.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:22.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:23.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:23.121 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:23.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:22 smithi123 bash[19119]: cluster 2024-02-12T16:22:21.881322+0000 mgr.y (mgr.14146) 1118 : cluster [DBG] pgmap v1521: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8637/70012 objects degraded (12.336%); 8684/70012 objects misplaced (12.404%); 874 KiB/s, 13 objects/s recovering 2024-02-12T16:22:23.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:22 smithi123 bash[19119]: audit 2024-02-12T16:22:22.152650+0000 mon.a (mon.0) 1836 : audit [DBG] from='client.? v1:172.21.15.121:0/4232668622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:23.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:23.235 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:22 smithi121 bash[18489]: cluster 2024-02-12T16:22:21.881322+0000 mgr.y (mgr.14146) 1118 : cluster [DBG] pgmap v1521: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8637/70012 objects degraded (12.336%); 8684/70012 objects misplaced (12.404%); 874 KiB/s, 13 objects/s recovering 2024-02-12T16:22:23.235 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:22 smithi121 bash[18489]: audit 2024-02-12T16:22:22.152650+0000 mon.a (mon.0) 1836 : audit [DBG] from='client.? v1:172.21.15.121:0/4232668622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:23.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:23.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:23.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:22 smithi169 bash[19604]: cluster 2024-02-12T16:22:21.881322+0000 mgr.y (mgr.14146) 1118 : cluster [DBG] pgmap v1521: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8637/70012 objects degraded (12.336%); 8684/70012 objects misplaced (12.404%); 874 KiB/s, 13 objects/s recovering 2024-02-12T16:22:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:22 smithi169 bash[19604]: audit 2024-02-12T16:22:22.152650+0000 mon.a (mon.0) 1836 : audit [DBG] from='client.? v1:172.21.15.121:0/4232668622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:22:23.325 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:22:23.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:23.404 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:23.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:23.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:23.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:23.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:23.672 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:23.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:23.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:23.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:23.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:23.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:23.958 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:23.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:23.974 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:24.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:24.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:24.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:24.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:24.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:24.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:24.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:24.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:24.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:24.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:24.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:24.595 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:22:24.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:24.646 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:24.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:24.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:24.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:24.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:24.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:24.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:24.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:25.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:25.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:24 smithi169 bash[19604]: cluster 2024-02-12T16:22:23.882434+0000 mgr.y (mgr.14146) 1119 : cluster [DBG] pgmap v1522: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8684/70012 objects misplaced (12.404%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:22:25.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:25.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:25.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:25.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:25.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:25.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:24 smithi121 bash[18489]: cluster 2024-02-12T16:22:23.882434+0000 mgr.y (mgr.14146) 1119 : cluster [DBG] pgmap v1522: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8684/70012 objects misplaced (12.404%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:22:25.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:25.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:24 smithi123 bash[19119]: cluster 2024-02-12T16:22:23.882434+0000 mgr.y (mgr.14146) 1119 : cluster [DBG] pgmap v1522: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8684/70012 objects misplaced (12.404%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:22:25.319 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:25.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:25.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:22:25.400 INFO:teuthology.orchestra.run.smithi121.stdout:moved 0 / 74 (0%) 2024-02-12T16:22:25.400 INFO:teuthology.orchestra.run.smithi121.stdout:avg 6.72727 2024-02-12T16:22:25.400 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 3.52793 -> 3.52793 (expected baseline 2.47299) 2024-02-12T16:22:25.400 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.1 with 0 -> 0 pgs (0 -> 0 * mean) 2024-02-12T16:22:25.401 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.6 with 11 -> 11 pgs (1.63514 -> 1.63514 * mean) 2024-02-12T16:22:25.401 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:22:25.401 INFO:teuthology.orchestra.run.smithi121.stdout:oload 1000 2024-02-12T16:22:25.401 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 1 2024-02-12T16:22:25.401 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:22:25.401 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0083 2024-02-12T16:22:25.401 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.0834 2024-02-12T16:22:25.403 INFO:teuthology.orchestra.run.smithi121.stderr:no change 2024-02-12T16:22:25.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:25.460 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:25.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:25.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:25.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:25.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:25.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:25.736 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:25.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:25.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:25.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:25.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:25.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:25.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:26.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:26.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:26.135 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:26.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:25 smithi121 bash[18489]: cluster 2024-02-12T16:22:25.556344+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check update: Degraded data redundancy: 8583/70012 objects degraded (12.259%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:26.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:26.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:25 smithi169 bash[19604]: cluster 2024-02-12T16:22:25.556344+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check update: Degraded data redundancy: 8583/70012 objects degraded (12.259%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:26.277 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:26.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:26.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:25 smithi123 bash[19119]: cluster 2024-02-12T16:22:25.556344+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check update: Degraded data redundancy: 8583/70012 objects degraded (12.259%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:26.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:26.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:26.692 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:26.795 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:26.827 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:26.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:26.965 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:26.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:27.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:27.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:27.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:26 smithi123 bash[19119]: audit 2024-02-12T16:22:25.395765+0000 mgr.y (mgr.14146) 1120 : audit [DBG] from='client.25853 v1:172.21.15.121:0/2245573207' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:26 smithi123 bash[19119]: cluster 2024-02-12T16:22:25.883568+0000 mgr.y (mgr.14146) 1121 : cluster [DBG] pgmap v1523: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8626/70012 objects misplaced (12.321%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:22:27.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:27.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:27.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:26 smithi169 bash[19604]: audit 2024-02-12T16:22:25.395765+0000 mgr.y (mgr.14146) 1120 : audit [DBG] from='client.25853 v1:172.21.15.121:0/2245573207' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:26 smithi169 bash[19604]: cluster 2024-02-12T16:22:25.883568+0000 mgr.y (mgr.14146) 1121 : cluster [DBG] pgmap v1523: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8626/70012 objects misplaced (12.321%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:22:27.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:27.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:26 smithi121 bash[18489]: audit 2024-02-12T16:22:25.395765+0000 mgr.y (mgr.14146) 1120 : audit [DBG] from='client.25853 v1:172.21.15.121:0/2245573207' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:22:27.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:26 smithi121 bash[18489]: cluster 2024-02-12T16:22:25.883568+0000 mgr.y (mgr.14146) 1121 : cluster [DBG] pgmap v1523: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8626/70012 objects misplaced (12.321%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:22:27.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:27.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:27.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:27.495 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:27.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:27.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:27.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:27.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:27.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:27.767 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:27.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:27.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:27.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:27.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:28.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:28.043 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:22:28.043 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:22:28.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:22:28.043 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:22:28.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:124964 4.5 e353/334 4:a403e5c2:::benchmark_data_smithi169_60938_object124963:head v 353'7772, mlcod=353'7770)", 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:14:50.478314+0000", 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 457.55684767100001, 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.058873186000000001, 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478314+0000", 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478314+0000", 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478312+0000", 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2361999999999995e-05 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.478394+0000", 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.045691896000000003 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524086+0000", 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4335000000000001e-05 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524101+0000", 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4688999999999998e-05 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524155+0000", 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4323000000000001e-05 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.524180+0000", 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012964513 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.537144+0000", 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2988999999999998e-05 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:14:50.537187+0000", 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.058873186000000001 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207911 4.5 e377/334 4:a47a7480:::benchmark_data_smithi169_60938_object34574:head v 377'13033, mlcod=377'13031)", 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.202860+0000", 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 387.83230128899999, 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13450072800000001, 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202860+0000", 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202860+0000", 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202859+0000", 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9830000000000004e-06 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202864+0000", 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2080000000000002e-06 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202869+0000", 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7690000000000001e-06 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202871+0000", 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1919000000000003e-05 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202913+0000", 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3874e-05 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.202927+0000", 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.134390395 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337318+0000", 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3404e-05 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337361+0000", 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13450072800000001 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.051 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207916 4.5 e377/334 4:a7a2a20e:::benchmark_data_smithi169_60938_object34579:head v 377'13034, mlcod=377'13031)", 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.207129+0000", 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 387.82803272500001, 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130258391, 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207129+0000", 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207129+0000", 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207126+0000", 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4731e-05 2024-02-12T16:22:28.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207141+0000", 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4761e-05 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207156+0000", 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4319999999999999e-06 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207164+0000", 2024-02-12T16:22:28.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000294114 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207458+0000", 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1787999999999999e-05 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.207480+0000", 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.12988481499999999 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337365+0000", 2024-02-12T16:22:28.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.215e-05 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.337387+0000", 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.130258391 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207919 4.e e377/334 4:736a4e29:::benchmark_data_smithi169_60938_object34582:head v 377'12960, mlcod=377'12958)", 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.208546+0000", 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 387.82661530299998, 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.132842772, 2024-02-12T16:22:28.056 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208546+0000", 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208546+0000", 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208545+0000", 2024-02-12T16:22:28.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0985000000000001e-05 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208556+0000", 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1014e-05 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208567+0000", 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2559999999999999e-06 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.208573+0000", 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.120351823 2024-02-12T16:22:28.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328925+0000", 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6460000000000001e-05 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328951+0000", 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012392357 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341343+0000", 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5546000000000001e-05 2024-02-12T16:22:28.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341389+0000", 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.132842772 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207918 4.7 e377/334 4:e0a92af0:::benchmark_data_smithi169_60938_object34581:head v 377'13007, mlcod=377'13004)", 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.210837+0000", 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 387.82432422099998, 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13963632000000001, 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.060 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210837+0000", 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210837+0000", 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210836+0000", 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0129000000000001e-05 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.210846+0000", 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.11802884399999999 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328875+0000", 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9264999999999999e-05 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328894+0000", 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016667400000000001 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.329061+0000", 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1637999999999998e-05 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.329082+0000", 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021358701000000001 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.350441+0000", 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2668e-05 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.350474+0000", 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13963632000000001 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.064 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:207934 4.e e377/334 4:73357360:::benchmark_data_smithi169_60938_object34597:head v 377'12961, mlcod=377'12958)", 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:00.228289+0000", 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 387.80687270700003, 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.113124713, 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228289+0000", 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228289+0000", 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228287+0000", 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2738000000000001e-05 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.228300+0000", 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.100576519 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328876+0000", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7234000000000001e-05 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.328893+0000", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042875989999999996 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.333181+0000", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3295999999999999e-05 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.333204+0000", 2024-02-12T16:22:28.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081894759999999994 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341394+0000", 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9811000000000001e-05 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:00.341414+0000", 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.113124713 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:259910 4.7 4:e1179fb6:::benchmark_data_smithi169_60938_object86573:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.089776+0000", 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 337.94538520600003, 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.079746256000000001, 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.069 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.070 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.070 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259910 2024-02-12T16:22:28.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.070 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089776+0000", 2024-02-12T16:22:28.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089776+0000", 2024-02-12T16:22:28.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:22:28.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089774+0000", 2024-02-12T16:22:28.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7060000000000004e-06 2024-02-12T16:22:28.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089782+0000", 2024-02-12T16:22:28.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1078e-05 2024-02-12T16:22:28.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089793+0000", 2024-02-12T16:22:28.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0090000000000004e-06 2024-02-12T16:22:28.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089799+0000", 2024-02-12T16:22:28.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0108000000000003e-05 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089839+0000", 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4909999999999999e-05 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.089914+0000", 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7779000000000001e-05 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.090002+0000", 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047817750000000003 2024-02-12T16:22:28.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.094783+0000", 2024-02-12T16:22:28.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001908152 2024-02-12T16:22:28.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.096691+0000", 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.072652089000000003 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169344+0000", 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1241999999999994e-05 2024-02-12T16:22:28.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169405+0000", 2024-02-12T16:22:28.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117841 2024-02-12T16:22:28.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169523+0000", 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.079746256000000001 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:259922 4.7 4:e2a9c3b9:::benchmark_data_smithi169_60938_object86585:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:22:28.080 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.098594+0000", 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 337.93656760499999, 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.071203441000000006, 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 259922 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098594+0000", 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098594+0000", 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098592+0000", 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0305999999999998e-05 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098612+0000", 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4029999999999996e-06 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098622+0000", 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9370000000000001e-06 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098627+0000", 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4458e-05 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098672+0000", 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012025 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098792+0000", 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015108100000000001 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.098943+0000", 2024-02-12T16:22:28.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043909810000000004 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.103334+0000", 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.066236931999999998 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169571+0000", 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1420000000000005e-05 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169633+0000", 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2843999999999998e-05 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169685+0000", 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000111938 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169797+0000", 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.071203441000000006 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.086 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:259926 4.e e395/334 4:72322939:::benchmark_data_smithi169_60938_object86589:head v 395'16218, mlcod=395'16216)", 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.104248+0000", 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 337.93091354000001, 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068572121, 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104248+0000", 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104248+0000", 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104247+0000", 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.433e-05 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104261+0000", 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3900000000000004e-06 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104269+0000", 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5240000000000002e-06 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104273+0000", 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1297999999999997e-05 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104304+0000", 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8640000000000001e-05 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.104323+0000", 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068398252000000007 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.172721+0000", 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8781000000000002e-05 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.172820+0000", 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068572121 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.25264.0:259927 4.a e395/393 4:58da0553:::benchmark_data_smithi169_60938_object86590:head v 395'16447, mlcod=395'16444)", 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.105096+0000", 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 337.93006514799998, 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068729290999999998, 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105096+0000", 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105096+0000", 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105095+0000", 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.49e-06 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.105104+0000", 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.060811997999999999 2024-02-12T16:22:28.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.165916+0000", 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6510000000000007e-06 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.165924+0000", 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5616000000000001e-05 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.166020+0000", 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1800000000000001e-05 2024-02-12T16:22:28.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.166042+0000", 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077482810000000001 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.173790+0000", 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5675e-05 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.173826+0000", 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.068729290999999998 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.25264.0:259922 4.7 e395/393 ondisk, result = 0)", 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.107586+0000", 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 337.927575236, 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062214634999999997, 2024-02-12T16:22:28.096 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107586+0000", 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107586+0000", 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107585+0000", 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0880000000000002e-06 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.107590+0000", 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.056974006000000001 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164564+0000", 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0686999999999999e-05 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164575+0000", 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050434390000000003 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169619+0000", 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2110000000000001e-06 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169626+0000", 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017516499999999999 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169801+0000", 2024-02-12T16:22:28.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062214634999999997 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.25264.0:259910 4.7 e395/393 ondisk, result = 0)", 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:50.109642+0000", 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 337.92551907299998, 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059889112000000001, 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109642+0000", 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109642+0000", 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109641+0000", 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8079999999999999e-06 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.109648+0000", 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.054774981 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164423+0000", 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8316999999999999e-05 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.164461+0000", 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048644780000000002 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169325+0000", 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0338e-05 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169336+0000", 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000195959 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:50.169532+0000", 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.059889112000000001 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:263277 4.7 4:e6b63bf5:::benchmark_data_smithi169_60938_object89940:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:53.560510+0000", 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 334.474651171, 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062677158999999996, 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 263277 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560510+0000", 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560510+0000", 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560509+0000", 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.539e-05 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560525+0000", 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2799999999999999e-05 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560538+0000", 2024-02-12T16:22:28.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9300000000000002e-06 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560542+0000", 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048533120000000003 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.565396+0000", 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5651999999999998e-05 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.565491+0000", 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000123387 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.565615+0000", 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011324532 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.576939+0000", 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029273950000000002 2024-02-12T16:22:28.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.579867+0000", 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.043138573999999999 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.623005+0000", 2024-02-12T16:22:28.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4831000000000003e-05 2024-02-12T16:22:28.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.623070+0000", 2024-02-12T16:22:28.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117427 2024-02-12T16:22:28.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.623188+0000", 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.062677158999999996 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:263279 4.7 4:e24fed27:::benchmark_data_smithi169_60938_object89942:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:53.560547+0000", 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 334.47461428399998, 2024-02-12T16:22:28.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070380943000000001, 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 263279 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560547+0000", 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560547+0000", 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560546+0000", 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7069999999999998e-06 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560550+0000", 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9609999999999996e-06 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560555+0000", 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1960000000000001e-06 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560559+0000", 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086128639999999996 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.569172+0000", 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010993999999999999 2024-02-12T16:22:28.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.569282+0000", 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104549 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.569387+0000", 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075612120000000003 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.576948+0000", 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029613119999999998 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.579909+0000", 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.050824741999999999 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.630734+0000", 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9920000000000002e-05 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.630794+0000", 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013435699999999999 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.630928+0000", 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070380943000000001 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:263282 4.7 4:eee7feaf:::benchmark_data_smithi169_60938_object89945:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:53.560699+0000", 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 334.47446262900002, 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070485122999999997, 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 263282 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.119 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560699+0000", 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560699+0000", 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560698+0000", 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0669999999999999e-06 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560703+0000", 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6640000000000001e-06 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560709+0000", 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6289999999999999e-06 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.560713+0000", 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01241046 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.573123+0000", 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4479000000000002e-05 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.573208+0000", 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8281000000000004e-05 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.573296+0000", 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046021719999999999 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.577898+0000", 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002060409 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.579959+0000", 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.051058427000000003 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.631017+0000", 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7184e-05 2024-02-12T16:22:28.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.631074+0000", 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109629 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:53.631184+0000", 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.070485122999999997 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:264645 4.7 4:edf50b6e:::benchmark_data_smithi169_60938_object91308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:54.920927+0000", 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 333.11423488200001, 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.072150912999999997, 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264645 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920927+0000", 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920927+0000", 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920924+0000", 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4759999999999998e-06 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920933+0000", 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8766999999999998e-05 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920951+0000", 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9646e-05 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.920981+0000", 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00059622699999999995 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.921577+0000", 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013502100000000001 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.921712+0000", 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019237 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.921905+0000", 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003766049 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.925671+0000", 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037594800000000003 2024-02-12T16:22:28.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.926047+0000", 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.066859926 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.992906+0000", 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3754000000000001e-05 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.130 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.992960+0000", 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117371 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:54.993078+0000", 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.072150912999999997 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:266565 4.7 4:e219d14d:::benchmark_data_smithi169_60938_object93228:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:56.880585+0000", 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 331.15457673399999, 2024-02-12T16:22:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.061889736000000001, 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266565 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.880585+0000", 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.880585+0000", 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.880584+0000", 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4419999999999997e-06 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.880589+0000", 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0355e-05 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.133 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.880599+0000", 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1899999999999995e-06 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.880607+0000", 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004206291 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.884814+0000", 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118072 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.884932+0000", 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100579 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.885032+0000", 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003498087 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.888531+0000", 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068669830000000001 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.895398+0000", 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.046921108000000003 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.942319+0000", 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3995000000000001e-05 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.942363+0000", 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011195400000000001 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.942475+0000", 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.061889736000000001 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:266575 4.7 4:eaf8f119:::benchmark_data_smithi169_60938_object93238:head [delete] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:56.890130+0000", 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 331.14503155199998, 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.061769371000000003, 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.138 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266575 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.890130+0000", 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.890130+0000", 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:28.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.890128+0000", 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5129999999999999e-06 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.890135+0000", 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0818999999999999e-05 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.890146+0000", 2024-02-12T16:22:28.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2140000000000001e-06 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.890152+0000", 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086349500000000004 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.891016+0000", 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118362 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.891134+0000", 2024-02-12T16:22:28.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011416 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.891249+0000", 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042885529999999996 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.895537+0000", 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042199270000000001 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.899757+0000", 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.051957087999999998 2024-02-12T16:22:28.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.951714+0000", 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4412999999999995e-05 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.951778+0000", 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012089 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:56.951899+0000", 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.061769371000000003 2024-02-12T16:22:28.143 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:268793 4.7 4:e23a651f:::benchmark_data_smithi169_60938_object95456:head [delete] snapc 0=[] ondisk+write+known_if_redirected e398)", 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:59.083116+0000", 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 328.95204551099999, 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.064578369999999996, 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 268793 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083116+0000", 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083116+0000", 2024-02-12T16:22:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083113+0000", 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0883999999999999e-05 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083124+0000", 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2449000000000001e-05 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083146+0000", 2024-02-12T16:22:28.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9760000000000007e-06 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083156+0000", 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016127700000000001 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083317+0000", 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026702799999999999 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083584+0000", 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021248099999999999 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.083797+0000", 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010373791 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.094171+0000", 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042933399999999997 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.098464+0000", 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.049105626999999999 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.147570+0000", 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2800999999999999e-05 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.147613+0000", 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1883000000000002e-05 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.147694+0000", 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.064578369999999996 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.150 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:269029 4.7 4:e4285b5a:::benchmark_data_smithi169_60938_object95692:head [delete] snapc 0=[] ondisk+write+known_if_redirected e398)", 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:59.386122+0000", 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 328.64903971000001, 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.071476321999999995, 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 269029 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:28.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.386122+0000", 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.386122+0000", 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.386119+0000", 2024-02-12T16:22:28.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3659999999999992e-06 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.386128+0000", 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1586999999999999e-05 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.386139+0000", 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0210000000000002e-06 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.386146+0000", 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037242709999999999 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.389871+0000", 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000145334 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.390016+0000", 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025998290000000001 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.392616+0000", 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048577250000000002 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.397473+0000", 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022997488999999999 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.420471+0000", 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036939184 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.457410+0000", 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2992999999999998e-05 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.457473+0000", 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012511099999999999 2024-02-12T16:22:28.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:59.457598+0000", 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.071476321999999995 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:28.157 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:22:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:28.159 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:28.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:28.260 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:28.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:28.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:28.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:28.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:28.436 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:28.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:28.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:28.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:28.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:28.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:28.720 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:28.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:28.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-12T16:22:28.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:28.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:28.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:28.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:28.990 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:28.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:29.004 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:22:29.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:29.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:28 smithi123 bash[19119]: cluster 2024-02-12T16:22:27.884650+0000 mgr.y (mgr.14146) 1122 : cluster [DBG] pgmap v1524: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8626/70012 objects misplaced (12.321%); 896 KiB/s, 13 objects/s recovering 2024-02-12T16:22:29.101 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:28 smithi123 bash[19119]: audit 2024-02-12T16:22:28.222817+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? v1:172.21.15.121:0/3629323354' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:29.120 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:29.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:29.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:29.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:29.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:28 smithi169 bash[19604]: cluster 2024-02-12T16:22:27.884650+0000 mgr.y (mgr.14146) 1122 : cluster [DBG] pgmap v1524: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8626/70012 objects misplaced (12.321%); 896 KiB/s, 13 objects/s recovering 2024-02-12T16:22:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:28 smithi169 bash[19604]: audit 2024-02-12T16:22:28.222817+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? v1:172.21.15.121:0/3629323354' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:29.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:29.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:28 smithi121 bash[18489]: cluster 2024-02-12T16:22:27.884650+0000 mgr.y (mgr.14146) 1122 : cluster [DBG] pgmap v1524: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8583/70012 objects degraded (12.259%); 8626/70012 objects misplaced (12.321%); 896 KiB/s, 13 objects/s recovering 2024-02-12T16:22:29.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:28 smithi121 bash[18489]: audit 2024-02-12T16:22:28.222817+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? v1:172.21.15.121:0/3629323354' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:29.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:29.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:29.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:29.526 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:29.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:29.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:29.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:29.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:29.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:29.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:29.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:29.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:29.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:29.923 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:29.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:30.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:30.055 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:30.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:30.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:30.195 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:30.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:29 smithi169 bash[19604]: cluster 2024-02-12T16:22:28.981594+0000 mon.a (mon.0) 1839 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:22:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:29 smithi169 bash[19604]: audit 2024-02-12T16:22:28.994721+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? v1:172.21.15.121:0/3629323354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:29 smithi169 bash[19604]: cluster 2024-02-12T16:22:28.994857+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e510: 12 total, 11 up, 12 in 2024-02-12T16:22:30.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:30.309 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:29 smithi123 bash[19119]: cluster 2024-02-12T16:22:28.981594+0000 mon.a (mon.0) 1839 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:22:30.309 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:29 smithi123 bash[19119]: audit 2024-02-12T16:22:28.994721+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? v1:172.21.15.121:0/3629323354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:30.309 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:29 smithi123 bash[19119]: cluster 2024-02-12T16:22:28.994857+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e510: 12 total, 11 up, 12 in 2024-02-12T16:22:30.329 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:29 smithi121 bash[18489]: cluster 2024-02-12T16:22:28.981594+0000 mon.a (mon.0) 1839 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:22:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:29 smithi121 bash[18489]: audit 2024-02-12T16:22:28.994721+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? v1:172.21.15.121:0/3629323354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:29 smithi121 bash[18489]: cluster 2024-02-12T16:22:28.994857+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e510: 12 total, 11 up, 12 in 2024-02-12T16:22:30.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:30.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:30.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:30.594 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:30.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:30.727 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:30.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:30.859 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:30.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:30.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:30.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:30.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:31.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:31.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:31.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:31.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:30 smithi123 bash[19119]: cluster 2024-02-12T16:22:29.885740+0000 mgr.y (mgr.14146) 1123 : cluster [DBG] pgmap v1526: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8528/70012 objects degraded (12.181%); 8568/70012 objects misplaced (12.238%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:22:31.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:30 smithi123 bash[19119]: cluster 2024-02-12T16:22:30.557791+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 8528/70012 objects degraded (12.181%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:31.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:30 smithi123 bash[19119]: cluster 2024-02-12T16:22:30.566810+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e511: 12 total, 11 up, 12 in 2024-02-12T16:22:31.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:31.258 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:31.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:30 smithi169 bash[19604]: cluster 2024-02-12T16:22:29.885740+0000 mgr.y (mgr.14146) 1123 : cluster [DBG] pgmap v1526: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8528/70012 objects degraded (12.181%); 8568/70012 objects misplaced (12.238%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:22:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:30 smithi169 bash[19604]: cluster 2024-02-12T16:22:30.557791+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 8528/70012 objects degraded (12.181%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:30 smithi169 bash[19604]: cluster 2024-02-12T16:22:30.566810+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e511: 12 total, 11 up, 12 in 2024-02-12T16:22:31.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:31.388 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:31.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:31.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:30 smithi121 bash[18489]: cluster 2024-02-12T16:22:29.885740+0000 mgr.y (mgr.14146) 1123 : cluster [DBG] pgmap v1526: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8528/70012 objects degraded (12.181%); 8568/70012 objects misplaced (12.238%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:22:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:30 smithi121 bash[18489]: cluster 2024-02-12T16:22:30.557791+0000 mon.a (mon.0) 1842 : cluster [WRN] Health check update: Degraded data redundancy: 8528/70012 objects degraded (12.181%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:30 smithi121 bash[18489]: cluster 2024-02-12T16:22:30.566810+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e511: 12 total, 11 up, 12 in 2024-02-12T16:22:31.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:31.497 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:22:31.498 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:22:31.498 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-12T16:22:31.498 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:22:31.498 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:22:31.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-12T16:22:31.526 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:31.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:31.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:31.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:31.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:31.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:22:31.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:31.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:31.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:31.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:31.941 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:31.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:32.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:32.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:32.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:32.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:32.218 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:32.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:32.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:32.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:32.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:32.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:32.481 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:32.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:32.584 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:32.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:32.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:32.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:32.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:32.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:32.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:32.885 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:32.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:32.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:33.025 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:33.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:33.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:33 smithi121 bash[18489]: cluster 2024-02-12T16:22:31.886468+0000 mgr.y (mgr.14146) 1124 : cluster [DBG] pgmap v1528: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8528/70012 objects degraded (12.181%); 8568/70012 objects misplaced (12.238%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:33.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:33.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:33.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:33 smithi169 bash[19604]: cluster 2024-02-12T16:22:31.886468+0000 mgr.y (mgr.14146) 1124 : cluster [DBG] pgmap v1528: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8528/70012 objects degraded (12.181%); 8568/70012 objects misplaced (12.238%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:33.297 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:33.374 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:22:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:33 smithi123 bash[19119]: cluster 2024-02-12T16:22:31.886468+0000 mgr.y (mgr.14146) 1124 : cluster [DBG] pgmap v1528: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8528/70012 objects degraded (12.181%); 8568/70012 objects misplaced (12.238%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:33.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:22:33.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:33.414 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:22:33.435 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:22:33.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:33.456 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:22:33.478 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:22:33.503 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:22:33.528 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:22:33.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:33.559 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:22:33.575 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:33.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:33.580 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:22:33.583 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:22:33.614 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:22:33.614 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.615 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.616 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.616 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.616 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:22:33.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:33.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:33.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:33.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:33.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:33.981 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:33.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:34.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:34.094 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:34 smithi169 bash[19604]: audit 2024-02-12T16:22:33.344909+0000 mon.a (mon.0) 1844 : audit [DBG] from='client.? v1:172.21.15.121:0/4209164371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:34.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:34.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:34.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:34.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:34 smithi121 bash[18489]: audit 2024-02-12T16:22:33.344909+0000 mon.a (mon.0) 1844 : audit [DBG] from='client.? v1:172.21.15.121:0/4209164371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:34.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:34.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:34.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:34.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:34.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:22:34.386 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:22:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:34 smithi123 bash[19119]: audit 2024-02-12T16:22:33.344909+0000 mon.a (mon.0) 1844 : audit [DBG] from='client.? v1:172.21.15.121:0/4209164371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:34.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:34.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:34.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:34.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:34.654 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:34.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:34.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:34.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:34.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:34.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:34.932 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:34.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:35.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:35.062 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:35.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:35.197 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:35 smithi169 bash[19604]: cluster 2024-02-12T16:22:33.887574+0000 mgr.y (mgr.14146) 1125 : cluster [DBG] pgmap v1529: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8472/70012 objects degraded (12.101%); 8568/70012 objects misplaced (12.238%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:35.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:35.307 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:35 smithi123 bash[19119]: cluster 2024-02-12T16:22:33.887574+0000 mgr.y (mgr.14146) 1125 : cluster [DBG] pgmap v1529: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8472/70012 objects degraded (12.101%); 8568/70012 objects misplaced (12.238%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:35.327 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:35.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:35 smithi121 bash[18489]: cluster 2024-02-12T16:22:33.887574+0000 mgr.y (mgr.14146) 1125 : cluster [DBG] pgmap v1529: 37 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8472/70012 objects degraded (12.101%); 8568/70012 objects misplaced (12.238%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:35.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:35.459 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:35.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:35.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:35.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:35.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:35.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:35.854 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:35.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:35.984 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:35.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:36.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:36.121 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:36.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:36.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:36.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:36.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:36.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:36 smithi169 bash[19604]: cluster 2024-02-12T16:22:34.225231+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:36 smithi169 bash[19604]: cluster 2024-02-12T16:22:34.225320+0000 osd.2 (osd.2) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:36 smithi169 bash[19604]: cluster 2024-02-12T16:22:35.015689+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e512: 12 total, 11 up, 12 in 2024-02-12T16:22:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:36 smithi169 bash[19604]: cluster 2024-02-12T16:22:35.560713+0000 mon.a (mon.0) 1846 : cluster [WRN] Health check update: Degraded data redundancy: 8472/70012 objects degraded (12.101%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:36 smithi169 bash[19604]: cluster 2024-02-12T16:22:35.568772+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e513: 12 total, 11 up, 12 in 2024-02-12T16:22:36.359 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:36.364 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:36 smithi123 bash[19119]: cluster 2024-02-12T16:22:34.225231+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:36.364 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:36 smithi123 bash[19119]: cluster 2024-02-12T16:22:34.225320+0000 osd.2 (osd.2) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:36.364 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:36 smithi123 bash[19119]: cluster 2024-02-12T16:22:35.015689+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e512: 12 total, 11 up, 12 in 2024-02-12T16:22:36.364 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:36 smithi123 bash[19119]: cluster 2024-02-12T16:22:35.560713+0000 mon.a (mon.0) 1846 : cluster [WRN] Health check update: Degraded data redundancy: 8472/70012 objects degraded (12.101%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:36.365 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:36 smithi123 bash[19119]: cluster 2024-02-12T16:22:35.568772+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e513: 12 total, 11 up, 12 in 2024-02-12T16:22:36.380 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:36.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:36.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:22:36.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:36 smithi121 bash[18489]: cluster 2024-02-12T16:22:34.225231+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:36.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:36 smithi121 bash[18489]: cluster 2024-02-12T16:22:34.225320+0000 osd.2 (osd.2) 61 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:36.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:36 smithi121 bash[18489]: cluster 2024-02-12T16:22:35.015689+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e512: 12 total, 11 up, 12 in 2024-02-12T16:22:36.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:36 smithi121 bash[18489]: cluster 2024-02-12T16:22:35.560713+0000 mon.a (mon.0) 1846 : cluster [WRN] Health check update: Degraded data redundancy: 8472/70012 objects degraded (12.101%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:36.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:36 smithi121 bash[18489]: cluster 2024-02-12T16:22:35.568772+0000 mon.a (mon.0) 1847 : cluster [DBG] osdmap e513: 12 total, 11 up, 12 in 2024-02-12T16:22:36.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:36.512 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:36.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:36.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:36.647 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:36.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:36.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:36.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:36.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:36.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:36.915 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:36.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:37.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:37.031 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:22:37.031 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:22:37.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:37.155 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:37.180 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:37.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:37.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:37 smithi123 bash[19119]: cluster 2024-02-12T16:22:35.888563+0000 mgr.y (mgr.14146) 1126 : cluster [DBG] pgmap v1532: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8458/70012 objects degraded (12.081%); 8510/70012 objects misplaced (12.155%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:37.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:37 smithi123 bash[19119]: cluster 2024-02-12T16:22:36.570261+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:22:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:37 smithi169 bash[19604]: cluster 2024-02-12T16:22:35.888563+0000 mgr.y (mgr.14146) 1126 : cluster [DBG] pgmap v1532: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8458/70012 objects degraded (12.081%); 8510/70012 objects misplaced (12.155%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:37 smithi169 bash[19604]: cluster 2024-02-12T16:22:36.570261+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:22:37.316 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:37.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:37 smithi121 bash[18489]: cluster 2024-02-12T16:22:35.888563+0000 mgr.y (mgr.14146) 1126 : cluster [DBG] pgmap v1532: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8458/70012 objects degraded (12.081%); 8510/70012 objects misplaced (12.155%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:22:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:37 smithi121 bash[18489]: cluster 2024-02-12T16:22:36.570261+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:22:37.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:37.560 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:37.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:37.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:37.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:37.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-12T16:22:37.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:37.853 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:37.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:37.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:37.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:38.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:38.114 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:38.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:38.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:38.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:38.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:38.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:38.378 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:38.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:38.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:38.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:38.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:38.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:38.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:38.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:38.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:38.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:38.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:38.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:38.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:39.019 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:39.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:39.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:39 smithi121 bash[18489]: cluster 2024-02-12T16:22:37.889660+0000 mgr.y (mgr.14146) 1127 : cluster [DBG] pgmap v1533: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8458/70012 objects degraded (12.081%); 8510/70012 objects misplaced (12.155%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:22:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:39 smithi121 bash[18489]: cluster 2024-02-12T16:22:38.096248+0000 osd.10 (osd.10) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:39 smithi121 bash[18489]: cluster 2024-02-12T16:22:38.096324+0000 osd.10 (osd.10) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:39.168 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:39.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:39.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:39 smithi169 bash[19604]: cluster 2024-02-12T16:22:37.889660+0000 mgr.y (mgr.14146) 1127 : cluster [DBG] pgmap v1533: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8458/70012 objects degraded (12.081%); 8510/70012 objects misplaced (12.155%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:22:39.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:39 smithi169 bash[19604]: cluster 2024-02-12T16:22:38.096248+0000 osd.10 (osd.10) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:39.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:39 smithi169 bash[19604]: cluster 2024-02-12T16:22:38.096324+0000 osd.10 (osd.10) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:39.303 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:39.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:39 smithi123 bash[19119]: cluster 2024-02-12T16:22:37.889660+0000 mgr.y (mgr.14146) 1127 : cluster [DBG] pgmap v1533: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8458/70012 objects degraded (12.081%); 8510/70012 objects misplaced (12.155%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:22:39.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:39 smithi123 bash[19119]: cluster 2024-02-12T16:22:38.096248+0000 osd.10 (osd.10) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:39.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:39 smithi123 bash[19119]: cluster 2024-02-12T16:22:38.096324+0000 osd.10 (osd.10) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:39.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:39.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:39.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:39.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:39.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:39.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:39.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:39.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:39.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:39.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:39.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:40.070 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:40.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:40.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:40.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:40.212 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:40 smithi169 bash[19604]: cluster 2024-02-12T16:22:39.031455+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e514: 12 total, 11 up, 12 in 2024-02-12T16:22:40.235 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:40.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:40.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:40.345 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:40 smithi123 bash[19119]: cluster 2024-02-12T16:22:39.031455+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e514: 12 total, 11 up, 12 in 2024-02-12T16:22:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:40.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:40 smithi121 bash[18489]: cluster 2024-02-12T16:22:39.031455+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e514: 12 total, 11 up, 12 in 2024-02-12T16:22:40.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:40.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:40.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:40.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:40.647 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:40.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:40.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:40.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:40.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:40.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:40.908 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:40.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:41.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:41.056 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:41.159 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:41.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:41 smithi121 bash[18489]: cluster 2024-02-12T16:22:39.890687+0000 mgr.y (mgr.14146) 1128 : cluster [DBG] pgmap v1535: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8404/70012 objects degraded (12.004%); 8452/70012 objects misplaced (12.072%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:22:41.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:41 smithi121 bash[18489]: cluster 2024-02-12T16:22:40.563125+0000 mon.a (mon.0) 1850 : cluster [WRN] Health check update: Degraded data redundancy: 8404/70012 objects degraded (12.004%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:41.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:41 smithi121 bash[18489]: cluster 2024-02-12T16:22:40.573125+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e515: 12 total, 11 up, 12 in 2024-02-12T16:22:41.192 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:41.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:41 smithi169 bash[19604]: cluster 2024-02-12T16:22:39.890687+0000 mgr.y (mgr.14146) 1128 : cluster [DBG] pgmap v1535: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8404/70012 objects degraded (12.004%); 8452/70012 objects misplaced (12.072%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:22:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:41 smithi169 bash[19604]: cluster 2024-02-12T16:22:40.563125+0000 mon.a (mon.0) 1850 : cluster [WRN] Health check update: Degraded data redundancy: 8404/70012 objects degraded (12.004%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:41 smithi169 bash[19604]: cluster 2024-02-12T16:22:40.573125+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e515: 12 total, 11 up, 12 in 2024-02-12T16:22:41.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:41 smithi123 bash[19119]: cluster 2024-02-12T16:22:39.890687+0000 mgr.y (mgr.14146) 1128 : cluster [DBG] pgmap v1535: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8404/70012 objects degraded (12.004%); 8452/70012 objects misplaced (12.072%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:22:41.302 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:41 smithi123 bash[19119]: cluster 2024-02-12T16:22:40.563125+0000 mon.a (mon.0) 1850 : cluster [WRN] Health check update: Degraded data redundancy: 8404/70012 objects degraded (12.004%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:41.302 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:41 smithi123 bash[19119]: cluster 2024-02-12T16:22:40.573125+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e515: 12 total, 11 up, 12 in 2024-02-12T16:22:41.324 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:41.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:41.461 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:41.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:41.590 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:41.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:41.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:41.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:41.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:41.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:41.984 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:41.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:41.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:42.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:42.121 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:42.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:42.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:42.248 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:42.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:42.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:42.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:42.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:42.483 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:22:42.483 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "" 2024-02-12T16:22:42.483 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:22:42.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:42.495 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:22:42.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:42.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:42.608 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:42.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:42.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:42.761 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:42.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:42.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:42.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:42.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:42.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:43.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:43.072 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:22:43.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:43.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:43.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:43.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:43 smithi169 bash[19604]: cluster 2024-02-12T16:22:41.891431+0000 mgr.y (mgr.14146) 1129 : cluster [DBG] pgmap v1537: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8346/70012 objects degraded (11.921%); 8452/70012 objects misplaced (12.072%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:22:43.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:43 smithi169 bash[19604]: cluster 2024-02-12T16:22:42.013971+0000 osd.4 (osd.4) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:43.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:43 smithi169 bash[19604]: cluster 2024-02-12T16:22:42.014051+0000 osd.4 (osd.4) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:43.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:43 smithi169 bash[19604]: cluster 2024-02-12T16:22:42.058229+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e516: 12 total, 11 up, 12 in 2024-02-12T16:22:43.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:43 smithi169 bash[19604]: audit 2024-02-12T16:22:42.999441+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? v1:172.21.15.121:0/694025030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:22:43.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:43.345 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:43 smithi121 bash[18489]: cluster 2024-02-12T16:22:41.891431+0000 mgr.y (mgr.14146) 1129 : cluster [DBG] pgmap v1537: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8346/70012 objects degraded (11.921%); 8452/70012 objects misplaced (12.072%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:22:43.345 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:43 smithi121 bash[18489]: cluster 2024-02-12T16:22:42.013971+0000 osd.4 (osd.4) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:43.345 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:43 smithi121 bash[18489]: cluster 2024-02-12T16:22:42.014051+0000 osd.4 (osd.4) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:43.345 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:43 smithi121 bash[18489]: cluster 2024-02-12T16:22:42.058229+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e516: 12 total, 11 up, 12 in 2024-02-12T16:22:43.345 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:43 smithi121 bash[18489]: audit 2024-02-12T16:22:42.999441+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? v1:172.21.15.121:0/694025030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:22:43.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:43 smithi123 bash[19119]: cluster 2024-02-12T16:22:41.891431+0000 mgr.y (mgr.14146) 1129 : cluster [DBG] pgmap v1537: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8346/70012 objects degraded (11.921%); 8452/70012 objects misplaced (12.072%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:22:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:43 smithi123 bash[19119]: cluster 2024-02-12T16:22:42.013971+0000 osd.4 (osd.4) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:43 smithi123 bash[19119]: cluster 2024-02-12T16:22:42.014051+0000 osd.4 (osd.4) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:43 smithi123 bash[19119]: cluster 2024-02-12T16:22:42.058229+0000 mon.a (mon.0) 1852 : cluster [DBG] osdmap e516: 12 total, 11 up, 12 in 2024-02-12T16:22:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:43 smithi123 bash[19119]: audit 2024-02-12T16:22:42.999441+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? v1:172.21.15.121:0/694025030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:22:43.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:43.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:43.535 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:43.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:43.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:43.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:43.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:43.820 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:43.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:43.945 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:43.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:44.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:44.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:44.180 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:44.187 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:44 smithi123 bash[19119]: cluster 2024-02-12T16:22:43.053462+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:44.187 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:44 smithi123 bash[19119]: audit 2024-02-12T16:22:43.063262+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.121:0/694025030' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:22:44.187 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:44 smithi123 bash[19119]: cluster 2024-02-12T16:22:43.063324+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e517: 12 total, 11 up, 12 in 2024-02-12T16:22:44.206 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:44 smithi169 bash[19604]: cluster 2024-02-12T16:22:43.053462+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:44 smithi169 bash[19604]: audit 2024-02-12T16:22:43.063262+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.121:0/694025030' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:22:44.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:44 smithi169 bash[19604]: cluster 2024-02-12T16:22:43.063324+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e517: 12 total, 11 up, 12 in 2024-02-12T16:22:44.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:44.332 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:44 smithi121 bash[18489]: cluster 2024-02-12T16:22:43.053462+0000 mon.a (mon.0) 1854 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:44 smithi121 bash[18489]: audit 2024-02-12T16:22:43.063262+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.121:0/694025030' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:22:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:44 smithi121 bash[18489]: cluster 2024-02-12T16:22:43.063324+0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e517: 12 total, 11 up, 12 in 2024-02-12T16:22:44.435 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:44.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:44.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:44.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:44.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:44.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:44.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:44.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:44.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:44.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:44.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:45.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:45.099 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:45.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:45.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:45.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:45.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:45.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:45.346 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:45 smithi123 bash[19119]: cluster 2024-02-12T16:22:43.892499+0000 mgr.y (mgr.14146) 1130 : cluster [DBG] pgmap v1540: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8292/70012 objects degraded (11.844%); 8452/70012 objects misplaced (12.072%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:22:45.366 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:45.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:45.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:45 smithi121 bash[18489]: cluster 2024-02-12T16:22:43.892499+0000 mgr.y (mgr.14146) 1130 : cluster [DBG] pgmap v1540: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8292/70012 objects degraded (11.844%); 8452/70012 objects misplaced (12.072%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:22:45.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:45.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:45 smithi169 bash[19604]: cluster 2024-02-12T16:22:43.892499+0000 mgr.y (mgr.14146) 1130 : cluster [DBG] pgmap v1540: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8292/70012 objects degraded (11.844%); 8452/70012 objects misplaced (12.072%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:22:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:45.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:45.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:45.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:45.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:45.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:45.759 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:45.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:45.779 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:22:45.800 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:22:45.819 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:22:45.838 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:22:45.858 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:22:45.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:45.879 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:22:45.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:45.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:45.899 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:22:45.920 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:22:45.940 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:22:45.960 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:22:45.988 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:22:45.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:46.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:46.059 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:22:46.059 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:22:46.059 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:22:46.059 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:22:46.059 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:22:46.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:46 smithi169 bash[19604]: cluster 2024-02-12T16:22:45.565358+0000 mon.a (mon.0) 1857 : cluster [WRN] Health check update: Degraded data redundancy: 8292/70012 objects degraded (11.844%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:46.099 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:46 smithi169 bash[19604]: audit 2024-02-12T16:22:45.751281+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.121:0/1700620888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:46.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:46.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:46.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:46 smithi123 bash[19119]: cluster 2024-02-12T16:22:45.565358+0000 mon.a (mon.0) 1857 : cluster [WRN] Health check update: Degraded data redundancy: 8292/70012 objects degraded (11.844%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:46 smithi123 bash[19119]: audit 2024-02-12T16:22:45.751281+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.121:0/1700620888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:46.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:46 smithi121 bash[18489]: cluster 2024-02-12T16:22:45.565358+0000 mon.a (mon.0) 1857 : cluster [WRN] Health check update: Degraded data redundancy: 8292/70012 objects degraded (11.844%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:46 smithi121 bash[18489]: audit 2024-02-12T16:22:45.751281+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.121:0/1700620888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:22:46.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:22:46.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:46.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:46.555 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:46.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:46.684 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:46.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-12T16:22:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:22:46.741 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:22:46.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:46.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:46.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:46.948 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:46.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:47.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:47.078 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:47.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:47.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:47.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:47.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:47.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:47.328 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:47 smithi169 bash[19604]: cluster 2024-02-12T16:22:45.893088+0000 mgr.y (mgr.14146) 1131 : cluster [DBG] pgmap v1541: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8234/70012 objects degraded (11.761%); 8393/70012 objects misplaced (11.988%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:22:47.347 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:47.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:47 smithi123 bash[19119]: cluster 2024-02-12T16:22:45.893088+0000 mgr.y (mgr.14146) 1131 : cluster [DBG] pgmap v1541: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8234/70012 objects degraded (11.761%); 8393/70012 objects misplaced (11.988%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:22:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:47 smithi121 bash[18489]: cluster 2024-02-12T16:22:45.893088+0000 mgr.y (mgr.14146) 1131 : cluster [DBG] pgmap v1541: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8234/70012 objects degraded (11.761%); 8393/70012 objects misplaced (11.988%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:22:47.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:47.481 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:47.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:47.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:47.612 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:47.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:47.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:47.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:47.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:47.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:47.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:47.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:47.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:48.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:48.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:48.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:48.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:48.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:48.289 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:48.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:48.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:48.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:48.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:48.505 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-02-12T16:22:48.505 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-02-12T16:22:48.505 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-02-12T16:22:48.506 DEBUG:teuthology.orchestra.run.smithi121:osd.3> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.3.service 2024-02-12T16:22:48.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl start ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.3 2024-02-12T16:22:48.524 INFO:journalctl@ceph.osd.3.smithi121.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T16:22:48.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:48.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:22:48.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:48.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:48.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:48.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:22:48.753 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:48.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:48.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:48.868 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:48 smithi121 systemd[1]: Started Ceph osd.3 for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T16:22:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:48.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:48.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:49.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:49.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:49.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:49 smithi121 bash[18489]: cluster 2024-02-12T16:22:47.894070+0000 mgr.y (mgr.14146) 1132 : cluster [DBG] pgmap v1542: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8234/70012 objects degraded (11.761%); 8393/70012 objects misplaced (11.988%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:22:49.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:49 smithi121 bash[18489]: cluster 2024-02-12T16:22:48.487434+0000 osd.0 (osd.0) 78 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:49.142 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:49 smithi121 bash[18489]: cluster 2024-02-12T16:22:48.487510+0000 osd.0 (osd.0) 79 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:49.162 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:49.264 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:49.287 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:49 smithi123 bash[19119]: cluster 2024-02-12T16:22:47.894070+0000 mgr.y (mgr.14146) 1132 : cluster [DBG] pgmap v1542: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8234/70012 objects degraded (11.761%); 8393/70012 objects misplaced (11.988%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:22:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:49 smithi123 bash[19119]: cluster 2024-02-12T16:22:48.487434+0000 osd.0 (osd.0) 78 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:49 smithi123 bash[19119]: cluster 2024-02-12T16:22:48.487510+0000 osd.0 (osd.0) 79 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:49.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:49.399 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:49 smithi169 bash[19604]: cluster 2024-02-12T16:22:47.894070+0000 mgr.y (mgr.14146) 1132 : cluster [DBG] pgmap v1542: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8234/70012 objects degraded (11.761%); 8393/70012 objects misplaced (11.988%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:22:49.399 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:49 smithi169 bash[19604]: cluster 2024-02-12T16:22:48.487434+0000 osd.0 (osd.0) 78 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:22:49.399 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:49 smithi169 bash[19604]: cluster 2024-02-12T16:22:48.487510+0000 osd.0 (osd.0) 79 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:22:49.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:49.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:49.551 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:49.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:49.683 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:49.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:49.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:49.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:49.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:49.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:50.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:50.071 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:50.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:50.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:50.311 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:50.321 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:50 smithi123 bash[19119]: cluster 2024-02-12T16:22:49.085440+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e518: 12 total, 11 up, 12 in 2024-02-12T16:22:50.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:50.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[18489]: cluster 2024-02-12T16:22:49.085440+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e518: 12 total, 11 up, 12 in 2024-02-12T16:22:50.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:50.466 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:50 smithi169 bash[19604]: cluster 2024-02-12T16:22:49.085440+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e518: 12 total, 11 up, 12 in 2024-02-12T16:22:50.569 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:50.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:50.724 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:50.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:50.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:50.912 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-12T16:22:50.912 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-12T16:22:50.912 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-02-12T16:22:50.912 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-12T16:22:50.912 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-02-12T16:22:50.913 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-12T16:22:50.913 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-12T16:22:50.913 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:50 smithi121 bash[159491]: --> ceph-volume lvm activate successful for osd ID: 3 2024-02-12T16:22:50.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:50.990 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:50.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:51.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:51.121 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:51.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:51.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:51.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:51.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:51 smithi169 bash[19604]: cluster 2024-02-12T16:22:49.894749+0000 mgr.y (mgr.14146) 1133 : cluster [DBG] pgmap v1544: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8178/70012 objects degraded (11.681%); 8335/70012 objects misplaced (11.905%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:22:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:51 smithi169 bash[19604]: cluster 2024-02-12T16:22:50.567614+0000 mon.a (mon.0) 1859 : cluster [WRN] Health check update: Degraded data redundancy: 8178/70012 objects degraded (11.681%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:51 smithi169 bash[19604]: cluster 2024-02-12T16:22:50.573995+0000 mon.a (mon.0) 1860 : cluster [DBG] osdmap e519: 12 total, 11 up, 12 in 2024-02-12T16:22:51.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:51.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:51 smithi123 bash[19119]: cluster 2024-02-12T16:22:49.894749+0000 mgr.y (mgr.14146) 1133 : cluster [DBG] pgmap v1544: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8178/70012 objects degraded (11.681%); 8335/70012 objects misplaced (11.905%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:22:51.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:51 smithi123 bash[19119]: cluster 2024-02-12T16:22:50.567614+0000 mon.a (mon.0) 1859 : cluster [WRN] Health check update: Degraded data redundancy: 8178/70012 objects degraded (11.681%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:51.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:51 smithi123 bash[19119]: cluster 2024-02-12T16:22:50.573995+0000 mon.a (mon.0) 1860 : cluster [DBG] osdmap e519: 12 total, 11 up, 12 in 2024-02-12T16:22:51.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:51.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:51 smithi121 bash[18489]: cluster 2024-02-12T16:22:49.894749+0000 mgr.y (mgr.14146) 1133 : cluster [DBG] pgmap v1544: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8178/70012 objects degraded (11.681%); 8335/70012 objects misplaced (11.905%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:22:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:51 smithi121 bash[18489]: cluster 2024-02-12T16:22:50.567614+0000 mon.a (mon.0) 1859 : cluster [WRN] Health check update: Degraded data redundancy: 8178/70012 objects degraded (11.681%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:51 smithi121 bash[18489]: cluster 2024-02-12T16:22:50.573995+0000 mon.a (mon.0) 1860 : cluster [DBG] osdmap e519: 12 total, 11 up, 12 in 2024-02-12T16:22:51.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:51.500 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:51.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:51.504 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:22:51.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:51.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:51.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:51.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:51.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:51.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:51.876 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:51.901 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:51.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:52.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:52.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:52.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:52.166 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:52.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:52.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:52.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:52.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:52.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:52.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:52.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:52.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:52.834 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:52.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:52.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:52.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:53.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:53.103 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:53.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:53.206 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:53.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:53.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:53.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:53.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:53.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:53 smithi121 bash[18489]: cluster 2024-02-12T16:22:51.895715+0000 mgr.y (mgr.14146) 1134 : cluster [DBG] pgmap v1546: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8119/70012 objects degraded (11.597%); 8335/70012 objects misplaced (11.905%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:22:53.368 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:53 smithi121 bash[159491]: debug 2024-02-12T16:22:53.140+0000 7f6ad3373700 1 -- 172.21.15.121:0/3100075137 <== mon.0 v1:172.21.15.121:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 2234802219 0 0) 0x55b6173249c0 con 0x55b617396c00 2024-02-12T16:22:53.369 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:53 smithi123 bash[19119]: cluster 2024-02-12T16:22:51.895715+0000 mgr.y (mgr.14146) 1134 : cluster [DBG] pgmap v1546: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8119/70012 objects degraded (11.597%); 8335/70012 objects misplaced (11.905%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:22:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:53.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:53.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:53.502 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:53.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:53.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:53 smithi169 bash[19604]: cluster 2024-02-12T16:22:51.895715+0000 mgr.y (mgr.14146) 1134 : cluster [DBG] pgmap v1546: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8119/70012 objects degraded (11.597%); 8335/70012 objects misplaced (11.905%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:22:53.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:53.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:53.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:53.676 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:53.677 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:22:53.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:53.772 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:53.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:53.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:53.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:53.888 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:53 smithi121 bash[159491]: debug 2024-02-12T16:22:53.688+0000 7f6adae01380 -1 Falling back to public interface 2024-02-12T16:22:53.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:53.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:54.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:54.039 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:54.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:54.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:54.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:22:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:54.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:22:54.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:54.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:54.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:54.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:54.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:54.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:54.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:54.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:54.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:54.653 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:22:54.653 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:22:54.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:204321 4.9 4:96e11958:::benchmark_data_smithi169_60938_object30984:head [delete] snapc 0=[] ondisk+write+known_if_redirected e377)", 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:15:56.009483+0000", 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 418.63793758999998, 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62252338500000004, 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 204321 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009483+0000", 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009483+0000", 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009480+0000", 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1177e-05 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009491+0000", 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8924999999999999e-05 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009510+0000", 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7619999999999995e-06 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009518+0000", 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0454000000000002e-05 2024-02-12T16:22:54.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009568+0000", 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015020100000000001 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009719+0000", 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020923099999999999 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.009928+0000", 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060745019999999999 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.016002+0000", 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.61570960299999999 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.631712+0000", 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015550000000000001 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.631867+0000", 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013907700000000001 2024-02-12T16:22:54.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:15:56.632006+0000", 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62252338500000004 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228647 4.8 4:15c660ab:::benchmark_data_smithi169_60938_object55310:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.305729+0000", 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.34169129999998, 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38848608099999998, 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.659 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228647 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305729+0000", 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305729+0000", 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305728+0000", 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8180000000000003e-06 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305733+0000", 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3279999999999996e-06 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305738+0000", 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2540000000000001e-06 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.305741+0000", 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044469720000000004 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310188+0000", 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000102354 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:54.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310291+0000", 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000119107 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 7", 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310410+0000", 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049471979999999999 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.315357+0000", 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37877825500000001 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.694135+0000", 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6576999999999998e-05 2024-02-12T16:22:54.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.694182+0000", 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3447e-05 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.694215+0000", 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38848608099999998 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.664 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228649 4.1 4:83f34d8e:::benchmark_data_smithi169_60938_object55312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.307435+0000", 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.33998610100002, 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38982620600000001, 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228649 2024-02-12T16:22:54.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307435+0000", 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307435+0000", 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307433+0000", 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9450000000000001e-06 2024-02-12T16:22:54.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307438+0000", 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5780000000000001e-06 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307446+0000", 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3259999999999997e-06 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307450+0000", 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8389000000000001e-05 2024-02-12T16:22:54.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307489+0000", 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6730000000000005e-05 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307575+0000", 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8439999999999999e-05 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:22:54.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307674+0000", 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37720748199999998 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.684881+0000", 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012317112 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697198+0000", 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5228999999999998e-05 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:54.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697234+0000", 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7214999999999999e-05 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697261+0000", 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38982620600000001 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.670 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228650 4.1 4:8a3347e3:::benchmark_data_smithi169_60938_object55313:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.307642+0000", 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.33977812500001, 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38971022799999999, 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:54.671 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228650 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307642+0000", 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307642+0000", 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307641+0000", 2024-02-12T16:22:54.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6260000000000003e-06 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307646+0000", 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2560000000000002e-06 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307651+0000", 2024-02-12T16:22:54.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0900000000000001e-06 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.307654+0000", 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046890480000000003 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.312343+0000", 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121348 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.312465+0000", 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012537599999999999 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.312590+0000", 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37316567299999998 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.685756+0000", 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011512557 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697268+0000", 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7462000000000001e-05 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697316+0000", 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6844000000000002e-05 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.697353+0000", 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38971022799999999 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.677 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:228653 4.9 4:96aa022a:::benchmark_data_smithi169_60938_object55316:head [delete] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:16:20.310158+0000", 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 394.33726286199999, 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39785818499999998, 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 228653 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310158+0000", 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310158+0000", 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310156+0000", 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5910000000000003e-06 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310162+0000", 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1329999999999999e-06 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310170+0000", 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2270000000000004e-06 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.310174+0000", 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.37473862699999999 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.684913+0000", 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000136811 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.685049+0000", 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017495299999999999 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 1", 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.685224+0000", 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022617696 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.707842+0000", 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8616000000000003e-05 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.707901+0000", 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8312e-05 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.707959+0000", 2024-02-12T16:22:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7003999999999998e-05 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:16:20.708016+0000", 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39785818499999998 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "rep_scrubmap(1.4 e453 from shard 8)", 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:21:32.936982+0000", 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 81.710438350000004, 2024-02-12T16:22:54.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067265470000000003, 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.936982+0000", 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.936982+0000", 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999962 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.936978+0000", 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7047000000000001e-05 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.936996+0000", 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9805000000000001e-05 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.937015+0000", 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9850000000000001e-06 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.937025+0000", 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066179230000000004 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.943643+0000", 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4399999999999999e-05 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.943658+0000", 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1193000000000003e-05 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:21:32.943709+0000", 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067265470000000003 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311590 4.9 4.ac7aa2c9 (undecoded) ondisk+retry+write+known_if_redirected e499)", 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:02.566334+0000", 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.081086947999999, 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024918140000000002, 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.687 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311590 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566334+0000", 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566334+0000", 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999948 2024-02-12T16:22:54.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566328+0000", 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7842999999999998e-05 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566346+0000", 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3615999999999998e-05 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566380+0000", 2024-02-12T16:22:54.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5613000000000001e-05 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566396+0000", 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000154251 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566550+0000", 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6585e-05 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566576+0000", 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021546109999999999 2024-02-12T16:22:54.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.568731+0000", 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4507999999999997e-05 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.568825+0000", 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024918140000000002 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311591 4.f 4.f30ad76f (undecoded) ondisk+retry+write+known_if_redirected e499)", 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:02.566421+0000", 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.080999308999999, 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024359120000000001, 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311591 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.693 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566421+0000", 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566421+0000", 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566419+0000", 2024-02-12T16:22:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6900000000000004e-06 2024-02-12T16:22:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566429+0000", 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6350000000000001e-05 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566445+0000", 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4579999999999999e-05 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566460+0000", 2024-02-12T16:22:54.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015402699999999999 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566614+0000", 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8581999999999998e-05 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.566643+0000", 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021858440000000002 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.568828+0000", 2024-02-12T16:22:54.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8782999999999999e-05 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:02.568857+0000", 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024359120000000001 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311591 4.f 4.f30ad76f (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.613128+0000", 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.034292192000002, 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18704958599999999, 2024-02-12T16:22:54.698 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311591 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613128+0000", 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613128+0000", 2024-02-12T16:22:54.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999943 2024-02-12T16:22:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613123+0000", 2024-02-12T16:22:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2515999999999999e-05 2024-02-12T16:22:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613145+0000", 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5456000000000002e-05 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613171+0000", 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2178999999999999e-05 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613193+0000", 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6036000000000002e-05 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613269+0000", 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2050999999999999e-05 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613291+0000", 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.186829037 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800120+0000", 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8e-05 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800178+0000", 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18704958599999999 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311595 4.f 4.4d58c6ef (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.613221+0000", 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.03419959, 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.186977435, 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.703 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311595 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613221+0000", 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613221+0000", 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613219+0000", 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1230000000000002e-06 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613226+0000", 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1080000000000001e-05 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613237+0000", 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3930000000000002e-06 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613245+0000", 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8444999999999999e-05 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613323+0000", 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.653e-06 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613332+0000", 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.186858201 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800190+0000", 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2870000000000004e-06 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800198+0000", 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.186977435 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311617 4.f 4.36020a0f (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.613275+0000", 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.034145987999999, 2024-02-12T16:22:54.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18696912099999999, 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311617 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613275+0000", 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613275+0000", 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613273+0000", 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5160000000000002e-06 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613279+0000", 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8730000000000002e-06 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613289+0000", 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2740000000000004e-06 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613295+0000", 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8696000000000002e-05 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613354+0000", 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6359999999999994e-06 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613361+0000", 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.186873337 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.711 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800235+0000", 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1419999999999999e-06 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800244+0000", 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18696912099999999 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311624 4.f 4.a171ce7f (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.613323+0000", 2024-02-12T16:22:54.712 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.034097361999997, 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.186970574, 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311624 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613323+0000", 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613323+0000", 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613322+0000", 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.13e-06 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613327+0000", 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7699999999999996e-06 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613337+0000", 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1500000000000004e-06 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613343+0000", 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2255000000000001e-05 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613385+0000", 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0891e-05 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.613396+0000", 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18688690599999999 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800283+0000", 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0659000000000001e-05 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.800294+0000", 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.186970574 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311590 4.9 4.ac7aa2c9 (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.633499+0000", 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.013921453999998, 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50320433399999998, 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311590 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633499+0000", 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633499+0000", 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633496+0000", 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1883e-05 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633508+0000", 2024-02-12T16:22:54.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3159999999999999e-05 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633521+0000", 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3689999999999996e-06 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633530+0000", 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.901e-05 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633579+0000", 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2359e-05 2024-02-12T16:22:54.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633591+0000", 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50303827499999998 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.136630+0000", 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3795999999999996e-05 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.136703+0000", 2024-02-12T16:22:54.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50320433399999998 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311596 4.9 4.96417339 (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.633538+0000", 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.013882195999997, 2024-02-12T16:22:54.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50335439500000001, 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311596 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633538+0000", 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633538+0000", 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633538+0000", 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2919999999999997e-06 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633542+0000", 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.604e-06 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633547+0000", 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9979999999999999e-06 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633550+0000", 2024-02-12T16:22:54.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5365999999999995e-05 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633636+0000", 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0559999999999999e-06 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633643+0000", 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50322945299999999 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.136872+0000", 2024-02-12T16:22:54.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0491000000000001e-05 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.136893+0000", 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50335439500000001 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311601 4.9 4.d22cf519 (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.633553+0000", 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.013867206999997, 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.5033936, 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.726 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311601 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633553+0000", 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633553+0000", 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:22:54.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633553+0000", 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7530000000000002e-06 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633556+0000", 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5160000000000002e-06 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633561+0000", 2024-02-12T16:22:54.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5009999999999999e-06 2024-02-12T16:22:54.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633564+0000", 2024-02-12T16:22:54.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012950099999999999 2024-02-12T16:22:54.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633693+0000", 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0002e-05 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633703+0000", 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50323537100000004 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.136939+0000", 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4349999999999997e-06 2024-02-12T16:22:54.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.136947+0000", 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.5033936 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311618 4.9 4.fc1317e9 (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.633567+0000", 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.01385338, 2024-02-12T16:22:54.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50343327100000002, 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311618 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.732 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633567+0000", 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633567+0000", 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633567+0000", 2024-02-12T16:22:54.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.526e-06 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633569+0000", 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.195e-06 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633575+0000", 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0840000000000002e-06 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633580+0000", 2024-02-12T16:22:54.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016176999999999999 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633741+0000", 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7180000000000008e-06 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633751+0000", 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50323879699999996 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.136990+0000", 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.066e-05 2024-02-12T16:22:54.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.137001+0000", 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50343327100000002 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311641 4.9 4.e4b95d29 (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.633586+0000", 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.013834938999999, 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50358474399999997, 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.736 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311641 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633586+0000", 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633586+0000", 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:22:54.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633585+0000", 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2799999999999999e-06 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633588+0000", 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1939999999999998e-06 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633597+0000", 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3590000000000001e-06 2024-02-12T16:22:54.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633601+0000", 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019555899999999999 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633796+0000", 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8759999999999998e-06 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633800+0000", 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50336207099999997 2024-02-12T16:22:54.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.137162+0000", 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0010000000000001e-06 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.137170+0000", 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50358474399999997 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311661 4.9 4.ecf65959 (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.633632+0000", 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.013788722999998, 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50356813199999995, 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311661 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633632+0000", 2024-02-12T16:22:54.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633632+0000", 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633631+0000", 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.566e-06 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633635+0000", 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6690000000000003e-06 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633641+0000", 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1699999999999999e-06 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633645+0000", 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022578999999999999 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633871+0000", 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.631e-06 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633877+0000", 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.503316549 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.137193+0000", 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7819999999999996e-06 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.137200+0000", 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50356813199999995 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.25264.0:311668 4.9 4.dd1b0cf9 (undecoded) ondisk+retry+write+known_if_redirected e500)", 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:03.633662+0000", 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.013758178000003, 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50356624500000002, 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.745 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.25264", 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 311668 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633662+0000", 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633662+0000", 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:22:54.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633662+0000", 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4630000000000001e-06 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633665+0000", 2024-02-12T16:22:54.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0641e-05 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633676+0000", 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7389999999999996e-06 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633682+0000", 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00021767499999999999 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633899+0000", 2024-02-12T16:22:54.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4900000000000006e-06 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:03.633909+0000", 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50331264099999995 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.137222+0000", 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1579999999999999e-06 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:04.137229+0000", 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50356624500000002 2024-02-12T16:22:54.749 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPushReply(4.f 517/499 [PushReplyOp(4:f13438e5:::benchmark_data_smithi169_60938_object149402:head)])", 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:22:47.383520+0000", 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.2639007380000002, 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00096201800000000003, 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.383520+0000", 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:22:54.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.383520+0000", 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.383519+0000", 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5219999999999999e-06 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.383523+0000", 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0980000000000001e-06 2024-02-12T16:22:54.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.383528+0000", 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.763e-06 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.383533+0000", 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000120257 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.383653+0000", 2024-02-12T16:22:54.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00082881899999999995 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:22:47.384482+0000", 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00096201800000000003 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:22:54.753 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:22:54.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:54.778 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:54.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:54.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:54.910 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:54.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:55.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:55.041 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:55.105 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:55 smithi123 bash[19119]: cluster 2024-02-12T16:22:53.896797+0000 mgr.y (mgr.14146) 1135 : cluster [DBG] pgmap v1547: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8065/70012 objects degraded (11.519%); 8335/70012 objects misplaced (11.905%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:22:55.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:55.175 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:55.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:55.290 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:55 smithi169 bash[19604]: cluster 2024-02-12T16:22:53.896797+0000 mgr.y (mgr.14146) 1135 : cluster [DBG] pgmap v1547: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8065/70012 objects degraded (11.519%); 8335/70012 objects misplaced (11.905%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:22:55.309 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:55.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:22:55.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:55 smithi121 bash[18489]: cluster 2024-02-12T16:22:53.896797+0000 mgr.y (mgr.14146) 1135 : cluster [DBG] pgmap v1547: 37 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8065/70012 objects degraded (11.519%); 8335/70012 objects misplaced (11.905%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:22:55.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:55.451 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:55.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:55.589 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:55.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:55.717 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:55.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:55.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:55.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:55.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:55.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:56.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:56.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:56.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:56.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:56 smithi121 bash[18489]: cluster 2024-02-12T16:22:55.570200+0000 mon.a (mon.0) 1861 : cluster [WRN] Health check update: Degraded data redundancy: 8065/70012 objects degraded (11.519%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:56.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:56.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:56.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:56.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:56.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:56.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:56 smithi123 bash[19119]: cluster 2024-02-12T16:22:55.570200+0000 mon.a (mon.0) 1861 : cluster [WRN] Health check update: Degraded data redundancy: 8065/70012 objects degraded (11.519%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:56.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:56.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:56 smithi169 bash[19604]: cluster 2024-02-12T16:22:55.570200+0000 mon.a (mon.0) 1861 : cluster [WRN] Health check update: Degraded data redundancy: 8065/70012 objects degraded (11.519%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:22:56.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:56.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:56.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:56.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:56.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:56.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:56.776 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:56.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:56.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:56.910 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:22:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:56.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:22:57.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:57.041 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:57.114 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:22:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:57 smithi123 bash[19119]: cluster 2024-02-12T16:22:55.897921+0000 mgr.y (mgr.14146) 1136 : cluster [DBG] pgmap v1548: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8007/70012 objects degraded (11.437%); 8277/70012 objects misplaced (11.822%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:22:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:57 smithi123 bash[19119]: audit 2024-02-12T16:22:56.413759+0000 mon.a (mon.0) 1862 : audit [INF] from='client.25886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:57 smithi123 bash[19119]: audit 2024-02-12T16:22:56.414934+0000 mon.b (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.121:0/3874731705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:57.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:57.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:57.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:57.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:57 smithi169 bash[19604]: cluster 2024-02-12T16:22:55.897921+0000 mgr.y (mgr.14146) 1136 : cluster [DBG] pgmap v1548: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8007/70012 objects degraded (11.437%); 8277/70012 objects misplaced (11.822%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:22:57.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:57 smithi169 bash[19604]: audit 2024-02-12T16:22:56.413759+0000 mon.a (mon.0) 1862 : audit [INF] from='client.25886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:57.280 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:57 smithi169 bash[19604]: audit 2024-02-12T16:22:56.414934+0000 mon.b (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.121:0/3874731705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:57.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:57.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:57.434 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:57.438 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:57 smithi121 bash[18489]: cluster 2024-02-12T16:22:55.897921+0000 mgr.y (mgr.14146) 1136 : cluster [DBG] pgmap v1548: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8007/70012 objects degraded (11.437%); 8277/70012 objects misplaced (11.822%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:22:57.438 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:57 smithi121 bash[18489]: audit 2024-02-12T16:22:56.413759+0000 mon.a (mon.0) 1862 : audit [INF] from='client.25886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:57.438 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:57 smithi121 bash[18489]: audit 2024-02-12T16:22:56.414934+0000 mon.b (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.121:0/3874731705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:22:57.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:22:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:22:57.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:57.708 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:57.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:57.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:57.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:57.971 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:57.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:58.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:58.103 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:58.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:58.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:22:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:58.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:22:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:58 smithi169 bash[19604]: cluster 2024-02-12T16:22:57.101888+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:58 smithi169 bash[19604]: audit 2024-02-12T16:22:57.106010+0000 mon.a (mon.0) 1864 : audit [INF] from='client.25886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:58 smithi169 bash[19604]: cluster 2024-02-12T16:22:57.106078+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e520: 12 total, 11 up, 12 in 2024-02-12T16:22:58.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:58.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:58.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:58 smithi123 bash[19119]: cluster 2024-02-12T16:22:57.101888+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:58 smithi123 bash[19119]: audit 2024-02-12T16:22:57.106010+0000 mon.a (mon.0) 1864 : audit [INF] from='client.25886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:58 smithi123 bash[19119]: cluster 2024-02-12T16:22:57.106078+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e520: 12 total, 11 up, 12 in 2024-02-12T16:22:58.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:58 smithi121 bash[18489]: cluster 2024-02-12T16:22:57.101888+0000 mon.a (mon.0) 1863 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:22:58.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:58 smithi121 bash[18489]: audit 2024-02-12T16:22:57.106010+0000 mon.a (mon.0) 1864 : audit [INF] from='client.25886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:22:58.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:58 smithi121 bash[18489]: cluster 2024-02-12T16:22:57.106078+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e520: 12 total, 11 up, 12 in 2024-02-12T16:22:58.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:58.503 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:22:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:58.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:22:58.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:58.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:22:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:58.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:22:58.745 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:58.773 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:58.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:58.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:58.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:58.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:59.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:59.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:22:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:22:59.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:59.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:59.255 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:22:59 smithi121 bash[18489]: cluster 2024-02-12T16:22:57.898888+0000 mgr.y (mgr.14146) 1137 : cluster [DBG] pgmap v1550: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8007/70012 objects degraded (11.437%); 8277/70012 objects misplaced (11.822%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:22:59.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:22:59 smithi123 bash[19119]: cluster 2024-02-12T16:22:57.898888+0000 mgr.y (mgr.14146) 1137 : cluster [DBG] pgmap v1550: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8007/70012 objects degraded (11.437%); 8277/70012 objects misplaced (11.822%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:22:59.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:59.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:22:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:22:59.466 INFO:teuthology.orchestra.run.smithi121.stderr:no valid command found; 10 closest matches: 2024-02-12T16:22:59.466 INFO:teuthology.orchestra.run.smithi121.stderr:0 2024-02-12T16:22:59.466 INFO:teuthology.orchestra.run.smithi121.stderr:1 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:2 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:abort 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:assert 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:bluefs debug_inject_read_zeros 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:bluefs files list 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:bluefs stats 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:bluestore allocator dump block 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:bluestore allocator fragmentation block 2024-02-12T16:22:59.467 INFO:teuthology.orchestra.run.smithi121.stderr:admin_socket: invalid command 2024-02-12T16:22:59.506 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:22:59 smithi121 bash[159491]: debug 2024-02-12T16:22:59.352+0000 7f6adae01380 -1 osd.3 414 log_to_monitors {default=true} 2024-02-12T16:22:59.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:59.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:22:59 smithi169 bash[19604]: cluster 2024-02-12T16:22:57.898888+0000 mgr.y (mgr.14146) 1137 : cluster [DBG] pgmap v1550: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8007/70012 objects degraded (11.437%); 8277/70012 objects misplaced (11.822%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:22:59.551 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:22:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:22:59.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:59.680 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:22:59.694 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:22:59.714 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:22:59.735 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:22:59.763 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:22:59.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:59.789 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:22:59.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:22:59.817 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:22:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:22:59.830 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:22:59.850 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:22:59.872 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:22:59.895 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:22:59.916 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:22:59.916 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.916 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.916 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:22:59.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:59.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:22:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:22:59.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:00.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:00.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:00.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:00.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:00.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:00.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:00.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:00.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:00.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:00.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:00 smithi123 bash[19119]: audit 2024-02-12T16:22:59.360209+0000 mon.a (mon.0) 1866 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:23:00.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:00 smithi123 bash[19119]: audit 2024-02-12T16:22:59.361619+0000 mon.c (mon.1) 38 : audit [INF] from='osd.3 v1:172.21.15.121:6813/2551349828' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:23:00.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:00 smithi123 bash[19119]: audit 2024-02-12T16:22:59.668032+0000 mon.b (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.121:0/217519153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:00.410 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:23:00 smithi121 bash[159491]: debug 2024-02-12T16:23:00.152+0000 7f6acdaeb700 -1 osd.3 521 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:23:00.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:00 smithi121 bash[18489]: audit 2024-02-12T16:22:59.360209+0000 mon.a (mon.0) 1866 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:23:00.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:00 smithi121 bash[18489]: audit 2024-02-12T16:22:59.361619+0000 mon.c (mon.1) 38 : audit [INF] from='osd.3 v1:172.21.15.121:6813/2551349828' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:23:00.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:00 smithi121 bash[18489]: audit 2024-02-12T16:22:59.668032+0000 mon.b (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.121:0/217519153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:00.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:00.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:00.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:00.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:00 smithi169 bash[19604]: audit 2024-02-12T16:22:59.360209+0000 mon.a (mon.0) 1866 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:23:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:00 smithi169 bash[19604]: audit 2024-02-12T16:22:59.361619+0000 mon.c (mon.1) 38 : audit [INF] from='osd.3 v1:172.21.15.121:6813/2551349828' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-12T16:23:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:00 smithi169 bash[19604]: audit 2024-02-12T16:22:59.668032+0000 mon.b (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.121:0/217519153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:00.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:00.620 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:00.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:00.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:00.756 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:00.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-12T16:23:00.797 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-02-12T16:23:00.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:00.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:00.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:23:01.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:01.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:23:01.039 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:23:01.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:01.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: cluster 2024-02-12T16:22:59.896039+0000 osd.11 (osd.11) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: cluster 2024-02-12T16:22:59.896079+0000 osd.11 (osd.11) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: cluster 2024-02-12T16:22:59.899978+0000 mgr.y (mgr.14146) 1138 : cluster [DBG] pgmap v1551: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7952/70012 objects degraded (11.358%); 8219/70012 objects misplaced (11.739%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:23:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: audit 2024-02-12T16:23:00.128174+0000 mon.a (mon.0) 1867 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T16:23:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: cluster 2024-02-12T16:23:00.128319+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e521: 12 total, 11 up, 12 in 2024-02-12T16:23:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: audit 2024-02-12T16:23:00.129980+0000 mon.a (mon.0) 1869 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T16:23:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: audit 2024-02-12T16:23:00.131129+0000 mon.c (mon.1) 39 : audit [INF] from='osd.3 v1:172.21.15.121:6813/2551349828' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T16:23:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:01 smithi169 bash[19604]: cluster 2024-02-12T16:23:00.572028+0000 mon.a (mon.0) 1870 : cluster [WRN] Health check update: Degraded data redundancy: 7952/70012 objects degraded (11.358%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:23:01.301 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:01.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: cluster 2024-02-12T16:22:59.896039+0000 osd.11 (osd.11) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: cluster 2024-02-12T16:22:59.896079+0000 osd.11 (osd.11) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: cluster 2024-02-12T16:22:59.899978+0000 mgr.y (mgr.14146) 1138 : cluster [DBG] pgmap v1551: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7952/70012 objects degraded (11.358%); 8219/70012 objects misplaced (11.739%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: audit 2024-02-12T16:23:00.128174+0000 mon.a (mon.0) 1867 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: cluster 2024-02-12T16:23:00.128319+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e521: 12 total, 11 up, 12 in 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: audit 2024-02-12T16:23:00.129980+0000 mon.a (mon.0) 1869 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: audit 2024-02-12T16:23:00.131129+0000 mon.c (mon.1) 39 : audit [INF] from='osd.3 v1:172.21.15.121:6813/2551349828' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T16:23:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:01 smithi121 bash[18489]: cluster 2024-02-12T16:23:00.572028+0000 mon.a (mon.0) 1870 : cluster [WRN] Health check update: Degraded data redundancy: 7952/70012 objects degraded (11.358%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:23:01.444 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:01.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: cluster 2024-02-12T16:22:59.896039+0000 osd.11 (osd.11) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: cluster 2024-02-12T16:22:59.896079+0000 osd.11 (osd.11) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: cluster 2024-02-12T16:22:59.899978+0000 mgr.y (mgr.14146) 1138 : cluster [DBG] pgmap v1551: 37 pgs: 1 active+remapped+backfilling, 1 undersized+degraded+peered, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7952/70012 objects degraded (11.358%); 8219/70012 objects misplaced (11.739%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: audit 2024-02-12T16:23:00.128174+0000 mon.a (mon.0) 1867 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: cluster 2024-02-12T16:23:00.128319+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e521: 12 total, 11 up, 12 in 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: audit 2024-02-12T16:23:00.129980+0000 mon.a (mon.0) 1869 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: audit 2024-02-12T16:23:00.131129+0000 mon.c (mon.1) 39 : audit [INF] from='osd.3 v1:172.21.15.121:6813/2551349828' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-12T16:23:01.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:01 smithi123 bash[19119]: cluster 2024-02-12T16:23:00.572028+0000 mon.a (mon.0) 1870 : cluster [WRN] Health check update: Degraded data redundancy: 7952/70012 objects degraded (11.358%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-12T16:23:01.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:01.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:01.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:01.812 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:01.842 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:01.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:01.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:01.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:02.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:02.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:02.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:02.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:02.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:02.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:02.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:02.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:02.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:02 smithi121 bash[18489]: cluster 2024-02-12T16:23:01.124228+0000 mon.a (mon.0) 1871 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T16:23:02.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:02 smithi121 bash[18489]: cluster 2024-02-12T16:23:01.132637+0000 mon.a (mon.0) 1872 : cluster [INF] osd.3 v1:172.21.15.121:6813/2551349828 boot 2024-02-12T16:23:02.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:02 smithi121 bash[18489]: cluster 2024-02-12T16:23:01.132756+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e522: 12 total, 12 up, 12 in 2024-02-12T16:23:02.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:02 smithi121 bash[18489]: audit 2024-02-12T16:23:01.133522+0000 mon.a (mon.0) 1874 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:23:02.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:02 smithi121 bash[18489]: cluster 2024-02-12T16:23:02.133562+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e523: 12 total, 12 up, 12 in 2024-02-12T16:23:02.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:02.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:02.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:02.497 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:02 smithi123 bash[19119]: cluster 2024-02-12T16:23:01.124228+0000 mon.a (mon.0) 1871 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T16:23:02.497 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:02 smithi123 bash[19119]: cluster 2024-02-12T16:23:01.132637+0000 mon.a (mon.0) 1872 : cluster [INF] osd.3 v1:172.21.15.121:6813/2551349828 boot 2024-02-12T16:23:02.497 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:02 smithi123 bash[19119]: cluster 2024-02-12T16:23:01.132756+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e522: 12 total, 12 up, 12 in 2024-02-12T16:23:02.497 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:02 smithi123 bash[19119]: audit 2024-02-12T16:23:01.133522+0000 mon.a (mon.0) 1874 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:23:02.497 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:02 smithi123 bash[19119]: cluster 2024-02-12T16:23:02.133562+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e523: 12 total, 12 up, 12 in 2024-02-12T16:23:02.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:02.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:02.525 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:02 smithi169 bash[19604]: cluster 2024-02-12T16:23:01.124228+0000 mon.a (mon.0) 1871 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T16:23:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:02 smithi169 bash[19604]: cluster 2024-02-12T16:23:01.132637+0000 mon.a (mon.0) 1872 : cluster [INF] osd.3 v1:172.21.15.121:6813/2551349828 boot 2024-02-12T16:23:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:02 smithi169 bash[19604]: cluster 2024-02-12T16:23:01.132756+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e522: 12 total, 12 up, 12 in 2024-02-12T16:23:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:02 smithi169 bash[19604]: audit 2024-02-12T16:23:01.133522+0000 mon.a (mon.0) 1874 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-12T16:23:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:02 smithi169 bash[19604]: cluster 2024-02-12T16:23:02.133562+0000 mon.a (mon.0) 1875 : cluster [DBG] osdmap e523: 12 total, 12 up, 12 in 2024-02-12T16:23:02.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:02.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:02.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:02.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:02.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:02.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:02.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:02.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:02.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:03.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:03.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:23:03.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:03.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:03.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:03.173 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:03.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:03.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:03.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:03.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:03 smithi121 bash[18489]: cluster 2024-02-12T16:23:00.394556+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:03.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:03 smithi121 bash[18489]: cluster 2024-02-12T16:23:00.394703+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:03.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:03 smithi121 bash[18489]: cluster 2024-02-12T16:23:01.900985+0000 mgr.y (mgr.14146) 1139 : cluster [DBG] pgmap v1554: 37 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7894/70012 objects degraded (11.275%); 8219/70012 objects misplaced (11.739%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:23:03.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:03 smithi121 bash[18489]: cluster 2024-02-12T16:23:03.144304+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e524: 12 total, 12 up, 12 in 2024-02-12T16:23:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:03.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:03.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:03.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:03.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:03 smithi169 bash[19604]: cluster 2024-02-12T16:23:00.394556+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:03 smithi169 bash[19604]: cluster 2024-02-12T16:23:00.394703+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:03.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:03 smithi169 bash[19604]: cluster 2024-02-12T16:23:01.900985+0000 mgr.y (mgr.14146) 1139 : cluster [DBG] pgmap v1554: 37 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7894/70012 objects degraded (11.275%); 8219/70012 objects misplaced (11.739%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:23:03.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:03 smithi169 bash[19604]: cluster 2024-02-12T16:23:03.144304+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e524: 12 total, 12 up, 12 in 2024-02-12T16:23:03.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:03.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:03.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:03 smithi123 bash[19119]: cluster 2024-02-12T16:23:00.394556+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:03 smithi123 bash[19119]: cluster 2024-02-12T16:23:00.394703+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:03 smithi123 bash[19119]: cluster 2024-02-12T16:23:01.900985+0000 mgr.y (mgr.14146) 1139 : cluster [DBG] pgmap v1554: 37 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+peered, 2 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7894/70012 objects degraded (11.275%); 8219/70012 objects misplaced (11.739%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:23:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:03 smithi123 bash[19119]: cluster 2024-02-12T16:23:03.144304+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e524: 12 total, 12 up, 12 in 2024-02-12T16:23:03.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:03.731 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:03.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:03.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:03.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:03.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:03.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:04.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:04.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:04.137 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:04.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:04.269 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:04.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:04.386 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:04 smithi123 bash[19119]: cluster 2024-02-12T16:23:03.144296+0000 osd.3 (osd.3) 3 : cluster [INF] 4.f continuing backfill to osd.9 from (360'9285,414'19285] 4:f257d1bd:::benchmark_data_smithi169_60938_object142410:head to 414'19285 2024-02-12T16:23:04.386 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:04 smithi123 bash[19119]: cluster 2024-02-12T16:23:03.147438+0000 osd.7 (osd.7) 113 : cluster [INF] 4.0 continuing backfill to osd.0 from (361'9567,414'19567] 4:0493ec76:::benchmark_data_smithi169_60938_object154053:head to 414'19567 2024-02-12T16:23:04.386 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:04 smithi123 bash[19119]: cluster 2024-02-12T16:23:03.151147+0000 osd.11 (osd.11) 110 : cluster [INF] 4.9 continuing backfill to osd.4 from (361'9577,414'19577] MIN to 414'19577 2024-02-12T16:23:04.386 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:04 smithi123 bash[19119]: cluster 2024-02-12T16:23:03.151170+0000 osd.11 (osd.11) 111 : cluster [INF] 4.9 continuing backfill to osd.6 from (361'9577,414'19577] MIN to 414'19577 2024-02-12T16:23:04.404 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:04.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:04 smithi121 bash[18489]: cluster 2024-02-12T16:23:03.144296+0000 osd.3 (osd.3) 3 : cluster [INF] 4.f continuing backfill to osd.9 from (360'9285,414'19285] 4:f257d1bd:::benchmark_data_smithi169_60938_object142410:head to 414'19285 2024-02-12T16:23:04.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:04 smithi121 bash[18489]: cluster 2024-02-12T16:23:03.147438+0000 osd.7 (osd.7) 113 : cluster [INF] 4.0 continuing backfill to osd.0 from (361'9567,414'19567] 4:0493ec76:::benchmark_data_smithi169_60938_object154053:head to 414'19567 2024-02-12T16:23:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:04 smithi121 bash[18489]: cluster 2024-02-12T16:23:03.151147+0000 osd.11 (osd.11) 110 : cluster [INF] 4.9 continuing backfill to osd.4 from (361'9577,414'19577] MIN to 414'19577 2024-02-12T16:23:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:04 smithi121 bash[18489]: cluster 2024-02-12T16:23:03.151170+0000 osd.11 (osd.11) 111 : cluster [INF] 4.9 continuing backfill to osd.6 from (361'9577,414'19577] MIN to 414'19577 2024-02-12T16:23:04.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:04.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:04.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:04 smithi169 bash[19604]: cluster 2024-02-12T16:23:03.144296+0000 osd.3 (osd.3) 3 : cluster [INF] 4.f continuing backfill to osd.9 from (360'9285,414'19285] 4:f257d1bd:::benchmark_data_smithi169_60938_object142410:head to 414'19285 2024-02-12T16:23:04.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:04 smithi169 bash[19604]: cluster 2024-02-12T16:23:03.147438+0000 osd.7 (osd.7) 113 : cluster [INF] 4.0 continuing backfill to osd.0 from (361'9567,414'19567] 4:0493ec76:::benchmark_data_smithi169_60938_object154053:head to 414'19567 2024-02-12T16:23:04.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:04 smithi169 bash[19604]: cluster 2024-02-12T16:23:03.151147+0000 osd.11 (osd.11) 110 : cluster [INF] 4.9 continuing backfill to osd.4 from (361'9577,414'19577] MIN to 414'19577 2024-02-12T16:23:04.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:04 smithi169 bash[19604]: cluster 2024-02-12T16:23:03.151170+0000 osd.11 (osd.11) 111 : cluster [INF] 4.9 continuing backfill to osd.6 from (361'9577,414'19577] MIN to 414'19577 2024-02-12T16:23:04.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:04.680 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:04.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:04.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:04.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:04.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:04.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:05.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:05.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:05.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:05.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:05.211 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:05.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:05.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:05.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:05.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:05 smithi121 bash[18489]: cluster 2024-02-12T16:23:03.901696+0000 mgr.y (mgr.14146) 1140 : cluster [DBG] pgmap v1557: 37 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 11 op/s; 6229/69920 objects degraded (8.909%); 9830/69920 objects misplaced (14.059%); 1.6 MiB/s, 24 objects/s recovering 2024-02-12T16:23:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:05 smithi121 bash[18489]: cluster 2024-02-12T16:23:04.149427+0000 mon.a (mon.0) 1877 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:23:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:05 smithi121 bash[18489]: cluster 2024-02-12T16:23:04.626970+0000 osd.6 (osd.6) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:05 smithi121 bash[18489]: cluster 2024-02-12T16:23:04.627026+0000 osd.6 (osd.6) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:05.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:05.465 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:05 smithi123 bash[19119]: cluster 2024-02-12T16:23:03.901696+0000 mgr.y (mgr.14146) 1140 : cluster [DBG] pgmap v1557: 37 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 11 op/s; 6229/69920 objects degraded (8.909%); 9830/69920 objects misplaced (14.059%); 1.6 MiB/s, 24 objects/s recovering 2024-02-12T16:23:05.465 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:05 smithi123 bash[19119]: cluster 2024-02-12T16:23:04.149427+0000 mon.a (mon.0) 1877 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:23:05.465 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:05 smithi123 bash[19119]: cluster 2024-02-12T16:23:04.626970+0000 osd.6 (osd.6) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:05.465 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:05 smithi123 bash[19119]: cluster 2024-02-12T16:23:04.627026+0000 osd.6 (osd.6) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:05.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:05.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:05.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:05 smithi169 bash[19604]: cluster 2024-02-12T16:23:03.901696+0000 mgr.y (mgr.14146) 1140 : cluster [DBG] pgmap v1557: 37 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 3 undersized+peered, 28 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 11 op/s; 6229/69920 objects degraded (8.909%); 9830/69920 objects misplaced (14.059%); 1.6 MiB/s, 24 objects/s recovering 2024-02-12T16:23:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:05 smithi169 bash[19604]: cluster 2024-02-12T16:23:04.149427+0000 mon.a (mon.0) 1877 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:23:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:05 smithi169 bash[19604]: cluster 2024-02-12T16:23:04.626970+0000 osd.6 (osd.6) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:05 smithi169 bash[19604]: cluster 2024-02-12T16:23:04.627026+0000 osd.6 (osd.6) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:05.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:05.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:05.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:05.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:05.733 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:05.755 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:05.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:05.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:23:05.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:05.889 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:05.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:05.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:06.019 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:06.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:06.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:06.159 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:06.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:06.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:06.295 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:06.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:06.398 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:06.440 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:06.445 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:06 smithi123 bash[19119]: cluster 2024-02-12T16:23:05.170132+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e525: 12 total, 12 up, 12 in 2024-02-12T16:23:06.446 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:06 smithi123 bash[19119]: cluster 2024-02-12T16:23:05.574471+0000 mon.a (mon.0) 1879 : cluster [WRN] Health check update: Degraded data redundancy: 6229/69920 objects degraded (8.909%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:23:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:06.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:06.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:06 smithi169 bash[19604]: cluster 2024-02-12T16:23:05.170132+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e525: 12 total, 12 up, 12 in 2024-02-12T16:23:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:06 smithi169 bash[19604]: cluster 2024-02-12T16:23:05.574471+0000 mon.a (mon.0) 1879 : cluster [WRN] Health check update: Degraded data redundancy: 6229/69920 objects degraded (8.909%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:23:06.569 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:06 smithi121 bash[18489]: cluster 2024-02-12T16:23:05.170132+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e525: 12 total, 12 up, 12 in 2024-02-12T16:23:06.569 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:06 smithi121 bash[18489]: cluster 2024-02-12T16:23:05.574471+0000 mon.a (mon.0) 1879 : cluster [WRN] Health check update: Degraded data redundancy: 6229/69920 objects degraded (8.909%), 3 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:23:06.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:06.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:06.684 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:06.730 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:06.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:06.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:06.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:06.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:06.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:07.013 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:07.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:07.143 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:07.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:07.271 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:07.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:07.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:07 smithi123 bash[19119]: cluster 2024-02-12T16:23:05.903393+0000 mgr.y (mgr.14146) 1141 : cluster [DBG] pgmap v1559: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 32 active+clean; 2.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 218 op/s; 10809/67922 objects misplaced (15.914%); 7.7 MiB/s, 122 objects/s recovering 2024-02-12T16:23:07.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:07 smithi123 bash[19119]: cluster 2024-02-12T16:23:06.583214+0000 mon.a (mon.0) 1880 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive) 2024-02-12T16:23:07.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:07 smithi123 bash[19119]: cluster 2024-02-12T16:23:06.583249+0000 mon.a (mon.0) 1881 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6229/69920 objects degraded (8.909%), 3 pgs degraded, 6 pgs undersized) 2024-02-12T16:23:07.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:07.447 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:07 smithi121 bash[18489]: cluster 2024-02-12T16:23:05.903393+0000 mgr.y (mgr.14146) 1141 : cluster [DBG] pgmap v1559: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 32 active+clean; 2.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 218 op/s; 10809/67922 objects misplaced (15.914%); 7.7 MiB/s, 122 objects/s recovering 2024-02-12T16:23:07.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:07 smithi121 bash[18489]: cluster 2024-02-12T16:23:06.583214+0000 mon.a (mon.0) 1880 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive) 2024-02-12T16:23:07.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:07 smithi121 bash[18489]: cluster 2024-02-12T16:23:06.583249+0000 mon.a (mon.0) 1881 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6229/69920 objects degraded (8.909%), 3 pgs degraded, 6 pgs undersized) 2024-02-12T16:23:07.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:07.539 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:07.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:07 smithi169 bash[19604]: cluster 2024-02-12T16:23:05.903393+0000 mgr.y (mgr.14146) 1141 : cluster [DBG] pgmap v1559: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 3 active+remapped+backfilling, 32 active+clean; 2.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 218 op/s; 10809/67922 objects misplaced (15.914%); 7.7 MiB/s, 122 objects/s recovering 2024-02-12T16:23:07.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:07 smithi169 bash[19604]: cluster 2024-02-12T16:23:06.583214+0000 mon.a (mon.0) 1880 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive) 2024-02-12T16:23:07.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:07 smithi169 bash[19604]: cluster 2024-02-12T16:23:06.583249+0000 mon.a (mon.0) 1881 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6229/69920 objects degraded (8.909%), 3 pgs degraded, 6 pgs undersized) 2024-02-12T16:23:07.590 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:23:07.590 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:23:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:07.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:07.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:07.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:07.830 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:07.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:07.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:07.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:08.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:08.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:08.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:08.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:08.236 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:08.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:08.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:08.371 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:08.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:08.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:08.507 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:08.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:08.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:08.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:08.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:08.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:08.773 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:08.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:08.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:08.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:08.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:09.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:09.042 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:09.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:09.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:09.171 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:09.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:09.304 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:09.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:09.436 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:09 smithi121 bash[18489]: cluster 2024-02-12T16:23:07.903869+0000 mgr.y (mgr.14146) 1142 : cluster [DBG] pgmap v1560: 37 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 32 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 291 op/s; 10203/66508 objects misplaced (15.341%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:23:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:09.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:09.545 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:09 smithi123 bash[19119]: cluster 2024-02-12T16:23:07.903869+0000 mgr.y (mgr.14146) 1142 : cluster [DBG] pgmap v1560: 37 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 32 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 291 op/s; 10203/66508 objects misplaced (15.341%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:23:09.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:09 smithi169 bash[19604]: cluster 2024-02-12T16:23:07.903869+0000 mgr.y (mgr.14146) 1142 : cluster [DBG] pgmap v1560: 37 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 32 active+clean; 2.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 291 op/s; 10203/66508 objects misplaced (15.341%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:23:09.581 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:09.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:09.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:09.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:23:09.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:09.851 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:09.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:09.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:09.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:10.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:10.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:10.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:10.230 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:10.257 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:10.272 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:10.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:10.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:10 smithi169 bash[19604]: cluster 2024-02-12T16:23:09.792259+0000 osd.8 (osd.8) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:10.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:10 smithi169 bash[19604]: cluster 2024-02-12T16:23:09.792324+0000 osd.8 (osd.8) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:10.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:10.385 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:10 smithi121 bash[18489]: cluster 2024-02-12T16:23:09.792259+0000 osd.8 (osd.8) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:10.385 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:10 smithi121 bash[18489]: cluster 2024-02-12T16:23:09.792324+0000 osd.8 (osd.8) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:10.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:10.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:10.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:10.534 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:10.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:10.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:10.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:10 smithi123 bash[19119]: cluster 2024-02-12T16:23:09.792259+0000 osd.8 (osd.8) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:10.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:10 smithi123 bash[19119]: cluster 2024-02-12T16:23:09.792324+0000 osd.8 (osd.8) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:10.665 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:10.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:10.768 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:10.790 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:10.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:10.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:10.926 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:10.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:11.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:11.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:11.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:11.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:11.222 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:11.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:11.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:11.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:11.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:11.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:11.460 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:11 smithi169 bash[19604]: cluster 2024-02-12T16:23:09.904387+0000 mgr.y (mgr.14146) 1143 : cluster [DBG] pgmap v1561: 37 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 32 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 9590/62286 objects misplaced (15.397%); 5.9 MiB/s, 95 objects/s recovering 2024-02-12T16:23:11.460 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:11 smithi169 bash[19604]: cluster 2024-02-12T16:23:10.188495+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e526: 12 total, 12 up, 12 in 2024-02-12T16:23:11.460 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:11 smithi169 bash[19604]: cluster 2024-02-12T16:23:10.583140+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e527: 12 total, 12 up, 12 in 2024-02-12T16:23:11.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:11.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:11.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:11.588 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:11 smithi121 bash[18489]: cluster 2024-02-12T16:23:09.904387+0000 mgr.y (mgr.14146) 1143 : cluster [DBG] pgmap v1561: 37 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 32 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 9590/62286 objects misplaced (15.397%); 5.9 MiB/s, 95 objects/s recovering 2024-02-12T16:23:11.589 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:11 smithi121 bash[18489]: cluster 2024-02-12T16:23:10.188495+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e526: 12 total, 12 up, 12 in 2024-02-12T16:23:11.589 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:11 smithi121 bash[18489]: cluster 2024-02-12T16:23:10.583140+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e527: 12 total, 12 up, 12 in 2024-02-12T16:23:11.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:11.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:11 smithi123 bash[19119]: cluster 2024-02-12T16:23:09.904387+0000 mgr.y (mgr.14146) 1143 : cluster [DBG] pgmap v1561: 37 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 32 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 9590/62286 objects misplaced (15.397%); 5.9 MiB/s, 95 objects/s recovering 2024-02-12T16:23:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:11 smithi123 bash[19119]: cluster 2024-02-12T16:23:10.188495+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e526: 12 total, 12 up, 12 in 2024-02-12T16:23:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:11 smithi123 bash[19119]: cluster 2024-02-12T16:23:10.583140+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e527: 12 total, 12 up, 12 in 2024-02-12T16:23:11.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:11.742 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:11.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:11.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:11.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:11.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:11.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:12.027 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:12.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:12.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:12.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:12.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:12.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:12.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:12.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:12.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:12.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:12.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:12.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:12.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:12.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:12.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:12.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:12.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:12.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:12.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:12.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:12.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:12.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:13.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:13.056 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:13.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:13.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:13.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:13 smithi169 bash[19604]: cluster 2024-02-12T16:23:11.904843+0000 mgr.y (mgr.14146) 1144 : cluster [DBG] pgmap v1564: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 32 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 7313/56704 objects misplaced (12.897%); 13 MiB/s, 202 objects/s recovering 2024-02-12T16:23:13.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:13.343 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:13.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:13 smithi121 bash[18489]: cluster 2024-02-12T16:23:11.904843+0000 mgr.y (mgr.14146) 1144 : cluster [DBG] pgmap v1564: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 32 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 7313/56704 objects misplaced (12.897%); 13 MiB/s, 202 objects/s recovering 2024-02-12T16:23:13.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:13.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:13.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:13.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:13.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:13.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:13.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:13 smithi123 bash[19119]: cluster 2024-02-12T16:23:11.904843+0000 mgr.y (mgr.14146) 1144 : cluster [DBG] pgmap v1564: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 32 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 7313/56704 objects misplaced (12.897%); 13 MiB/s, 202 objects/s recovering 2024-02-12T16:23:13.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:13.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:13.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:13.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:13.893 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:13.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:14.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:14.046 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:14.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:14.171 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:14.210 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:23:14.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:14.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:14.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:14 smithi123 bash[19119]: audit 2024-02-12T16:23:13.374652+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? v1:172.21.15.121:0/2432394398' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:23:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:14 smithi123 bash[19119]: cluster 2024-02-12T16:23:13.418916+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:14 smithi123 bash[19119]: cluster 2024-02-12T16:23:13.418975+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:14.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:14.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:14.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:14.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:14 smithi169 bash[19604]: audit 2024-02-12T16:23:13.374652+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? v1:172.21.15.121:0/2432394398' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:23:14.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:14 smithi169 bash[19604]: cluster 2024-02-12T16:23:13.418916+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:14.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:14 smithi169 bash[19604]: cluster 2024-02-12T16:23:13.418975+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:14.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:14.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:14 smithi121 bash[18489]: audit 2024-02-12T16:23:13.374652+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? v1:172.21.15.121:0/2432394398' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:23:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:14 smithi121 bash[18489]: cluster 2024-02-12T16:23:13.418916+0000 osd.7 (osd.7) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:14 smithi121 bash[18489]: cluster 2024-02-12T16:23:13.418975+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:14.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:14.684 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:14.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:14.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:14.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:14.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:14.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:15.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:15.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:15.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:15.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:15.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:15.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:15.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:15.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:15.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:15.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:15.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:15 smithi169 bash[19604]: cluster 2024-02-12T16:23:13.905363+0000 mgr.y (mgr.14146) 1145 : cluster [DBG] pgmap v1565: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 32 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 901 op/s; 7028/53496 objects misplaced (13.137%); 11 MiB/s, 176 objects/s recovering 2024-02-12T16:23:15.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:15 smithi169 bash[19604]: cluster 2024-02-12T16:23:14.189750+0000 mon.a (mon.0) 1885 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:23:15.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:15 smithi169 bash[19604]: audit 2024-02-12T16:23:14.202831+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.121:0/2432394398' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:23:15.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:15 smithi169 bash[19604]: cluster 2024-02-12T16:23:14.202878+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e528: 12 total, 12 up, 12 in 2024-02-12T16:23:15.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:15 smithi169 bash[19604]: cluster 2024-02-12T16:23:14.985197+0000 osd.5 (osd.5) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:15.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:15 smithi169 bash[19604]: cluster 2024-02-12T16:23:14.985262+0000 osd.5 (osd.5) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:15.495 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:15.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:15.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:15 smithi121 bash[18489]: cluster 2024-02-12T16:23:13.905363+0000 mgr.y (mgr.14146) 1145 : cluster [DBG] pgmap v1565: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 32 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 901 op/s; 7028/53496 objects misplaced (13.137%); 11 MiB/s, 176 objects/s recovering 2024-02-12T16:23:15.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:15 smithi121 bash[18489]: cluster 2024-02-12T16:23:14.189750+0000 mon.a (mon.0) 1885 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:23:15.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:15 smithi121 bash[18489]: audit 2024-02-12T16:23:14.202831+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.121:0/2432394398' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:23:15.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:15 smithi121 bash[18489]: cluster 2024-02-12T16:23:14.202878+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e528: 12 total, 12 up, 12 in 2024-02-12T16:23:15.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:15 smithi121 bash[18489]: cluster 2024-02-12T16:23:14.985197+0000 osd.5 (osd.5) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:15.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:15 smithi121 bash[18489]: cluster 2024-02-12T16:23:14.985262+0000 osd.5 (osd.5) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:15.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:15.607 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:15 smithi123 bash[19119]: cluster 2024-02-12T16:23:13.905363+0000 mgr.y (mgr.14146) 1145 : cluster [DBG] pgmap v1565: 37 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 32 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 901 op/s; 7028/53496 objects misplaced (13.137%); 11 MiB/s, 176 objects/s recovering 2024-02-12T16:23:15.607 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:15 smithi123 bash[19119]: cluster 2024-02-12T16:23:14.189750+0000 mon.a (mon.0) 1885 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:23:15.607 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:15 smithi123 bash[19119]: audit 2024-02-12T16:23:14.202831+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.121:0/2432394398' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:23:15.607 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:15 smithi123 bash[19119]: cluster 2024-02-12T16:23:14.202878+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e528: 12 total, 12 up, 12 in 2024-02-12T16:23:15.607 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:15 smithi123 bash[19119]: cluster 2024-02-12T16:23:14.985197+0000 osd.5 (osd.5) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:15.607 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:15 smithi123 bash[19119]: cluster 2024-02-12T16:23:14.985262+0000 osd.5 (osd.5) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:15.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:15.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:15.733 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:23:15.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:15.758 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:15.758 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:23:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:15.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:15.781 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:23:15.806 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:23:15.830 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:23:15.858 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:23:15.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:15.891 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:15.895 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:23:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:15.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:15.924 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:23:15.953 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:23:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:23:15.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:16.021 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:23:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:16.047 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:23:16.051 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.052 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:23:16.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:16.163 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:16.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:16.299 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:16.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:16.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:16 smithi121 bash[18489]: cluster 2024-02-12T16:23:15.206592+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e529: 12 total, 12 up, 12 in 2024-02-12T16:23:16.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:16 smithi121 bash[18489]: audit 2024-02-12T16:23:15.695903+0000 mon.b (mon.2) 135 : audit [DBG] from='client.? v1:172.21.15.121:0/295395167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:16.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:16.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:16 smithi169 bash[19604]: cluster 2024-02-12T16:23:15.206592+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e529: 12 total, 12 up, 12 in 2024-02-12T16:23:16.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:16 smithi169 bash[19604]: audit 2024-02-12T16:23:15.695903+0000 mon.b (mon.2) 135 : audit [DBG] from='client.? v1:172.21.15.121:0/295395167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:16.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:16.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:16 smithi123 bash[19119]: cluster 2024-02-12T16:23:15.206592+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e529: 12 total, 12 up, 12 in 2024-02-12T16:23:16.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:16 smithi123 bash[19119]: audit 2024-02-12T16:23:15.695903+0000 mon.b (mon.2) 135 : audit [DBG] from='client.? v1:172.21.15.121:0/295395167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:16.665 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:16.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:16.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:16.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:16.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:16.958 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:16.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:17.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:17.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:17.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:17.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:17.227 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:17.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:17.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:17.338 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:17 smithi169 bash[19604]: cluster 2024-02-12T16:23:15.905834+0000 mgr.y (mgr.14146) 1146 : cluster [DBG] pgmap v1568: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 6061/46562 objects misplaced (13.017%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:23:17.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:17.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:17.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:17.473 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:17 smithi123 bash[19119]: cluster 2024-02-12T16:23:15.905834+0000 mgr.y (mgr.14146) 1146 : cluster [DBG] pgmap v1568: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 6061/46562 objects misplaced (13.017%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:23:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:17.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:17.553 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:17 smithi121 bash[18489]: cluster 2024-02-12T16:23:15.905834+0000 mgr.y (mgr.14146) 1146 : cluster [DBG] pgmap v1568: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 6061/46562 objects misplaced (13.017%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:23:17.600 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:17.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:17.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:17.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:17.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:17.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:17.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:17.910 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:17.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:17.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:23:17.987 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:23:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25264.0:327859 4.9 e529/523 4:9d31ae9f:::benchmark_data_smithi169_60938_object154522:head v 529'20571, mlcod=529'20569)", 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.980123+0000", 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00022550900000000001, 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025305800000000002, 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980123+0000", 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980123+0000", 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980121+0000", 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1929999999999993e-06 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980129+0000", 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9119999999999998e-06 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980137+0000", 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4299999999999999e-06 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980141+0000", 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.926e-05 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980181+0000", 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2634e-05 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980203+0000", 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0714e-05 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25264.0:327860 4.4 e529/522)", 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.980166+0000", 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.000182244, 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000279058, 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980166+0000", 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980166+0000", 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980165+0000", 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5309999999999998e-06 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980171+0000", 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6019999999999999e-06 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980177+0000", 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6600000000000001e-06 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980181+0000", 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4846e-05 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.25264.0:327857 4.f e529/526)", 2024-02-12T16:23:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.980201+0000", 2024-02-12T16:23:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.000146973, 2024-02-12T16:23:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00029373499999999999, 2024-02-12T16:23:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:23:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980201+0000", 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980201+0000", 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980200+0000", 2024-02-12T16:23:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3089999999999997e-06 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980205+0000", 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4999999999999999e-06 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980210+0000", 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3229999999999999e-06 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980214+0000", 2024-02-12T16:23:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3303e-05 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:327864 4.f 4.1604d38f (undecoded) ondisk+write+known_if_redirected e529)", 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.980228+0000", 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.000120104, 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032146099999999999, 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:23:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 327864 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980228+0000", 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980228+0000", 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980227+0000", 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1270000000000002e-06 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980232+0000", 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0530000000000001e-06 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980239+0000", 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8740000000000003e-06 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980244+0000", 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5613000000000001e-05 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25264.0:327846 4.0 e529/523 4:01fe841c:::benchmark_data_smithi169_60938_object154509:head v 529'20618, mlcod=529'20614)", 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.969440+0000", 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.010907791999999999, 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011172243, 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969440+0000", 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969440+0000", 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969439+0000", 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0340000000000002e-06 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969448+0000", 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0468e-05 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969458+0000", 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2890000000000003e-06 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969464+0000", 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2156000000000005e-05 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969537+0000", 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0037999999999999e-05 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969567+0000", 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126334 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25264.0:327845 4.c e529/522 4:3f67686e:::benchmark_data_smithi169_60938_object154508:head v 529'20509, mlcod=529'20507)", 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.969925+0000", 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.010423170000000001, 2024-02-12T16:23:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010753054, 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969925+0000", 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969925+0000", 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969924+0000", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9750000000000007e-06 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969932+0000", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2560000000000008e-06 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969941+0000", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1449999999999999e-06 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.969946+0000", 2024-02-12T16:23:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001155592 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.971102+0000", 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1223e-05 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.971123+0000", 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011978340000000001 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.25264.0:327852 4.0 e529/523 4:0a15615a:::benchmark_data_smithi169_60938_object154515:head v 529'20619, mlcod=529'20615)", 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.973975+0000", 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0063726299999999998, 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067633809999999997, 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.973975+0000", 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.973975+0000", 2024-02-12T16:23:18.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.973974+0000", 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6919999999999997e-06 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.973983+0000", 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4562000000000001e-05 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.973998+0000", 2024-02-12T16:23:18.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5429999999999996e-06 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.974005+0000", 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00123036 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975236+0000", 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001293556 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.976529+0000", 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025537950000000002 2024-02-12T16:23:18.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:327857 4.f 4:f4bca98d:::benchmark_data_smithi169_60938_object154520:head [delete] snapc 0=[] ondisk+write+known_if_redirected e529)", 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.975088+0000", 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0052603989999999998, 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005712862, 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 327857 2024-02-12T16:23:18.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975088+0000", 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975088+0000", 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975085+0000", 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6649999999999998e-06 2024-02-12T16:23:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975095+0000", 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0831e-05 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975106+0000", 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6579999999999996e-06 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975113+0000", 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.033e-05 2024-02-12T16:23:18.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975173+0000", 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9123999999999996e-05 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975262+0000", 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7608000000000006e-05 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.975350+0000", 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029959399999999999 2024-02-12T16:23:18.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.978346+0000", 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0032579079999999999 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:327862 4.f 4:fe1ce967:::benchmark_data_smithi169_60938_object154525:head [delete] snapc 0=[] ondisk+write+known_if_redirected e529)", 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:17.978643+0000", 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0017055760000000001, 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002241636, 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:23:18.014 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 327862 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.978643+0000", 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.978643+0000", 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.978640+0000", 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.712e-06 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.978648+0000", 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1878999999999999e-05 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.978660+0000", 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9820000000000002e-06 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:23:18.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.978667+0000", 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0014760159999999999 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980143+0000", 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108495 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980251+0000", 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119722 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:17.980371+0000", 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017282059999999999 2024-02-12T16:23:18.017 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.018 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:23:18.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:23:18.018 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:23:18.018 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 9 2024-02-12T16:23:18.018 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:23:18.018 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:18.052 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:18.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:18.194 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:18.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:18.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:18.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:18.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:18.562 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:18.591 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:18.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:18.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:18.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:18.862 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:18.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:18.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:18.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:19.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:19.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:19.216 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:19 smithi169 bash[19604]: cluster 2024-02-12T16:23:17.906297+0000 mgr.y (mgr.14146) 1147 : cluster [DBG] pgmap v1569: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s; 5725/44398 objects misplaced (12.895%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:23:19.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:19.264 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:19.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:19.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T16:23:19.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:19 smithi123 bash[19119]: cluster 2024-02-12T16:23:17.906297+0000 mgr.y (mgr.14146) 1147 : cluster [DBG] pgmap v1569: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s; 5725/44398 objects misplaced (12.895%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:23:19.395 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:19.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:19.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:19.539 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:19.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:19.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:23:19.561 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:19 smithi121 bash[18489]: cluster 2024-02-12T16:23:17.906297+0000 mgr.y (mgr.14146) 1147 : cluster [DBG] pgmap v1569: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s; 5725/44398 objects misplaced (12.895%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:23:19.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:19.681 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:19.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:19.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:19.808 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:19.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:19.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:19.944 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:19.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:20.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:20.081 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:20.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:20.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:20.220 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:20.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:20.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:20.357 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:20.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:20.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:20.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:20.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:20.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:20.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:20.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:20.778 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:23:20.778 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:23:20.778 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:23:20.778 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:23:20.778 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:23:20.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:20.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:20.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:20.906 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:20.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:21.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:21.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:21.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:21.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:21.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:21.301 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:21.404 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:21 smithi121 bash[18489]: cluster 2024-02-12T16:23:19.812552+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:21 smithi121 bash[18489]: cluster 2024-02-12T16:23:19.812591+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:21 smithi121 bash[18489]: cluster 2024-02-12T16:23:19.906721+0000 mgr.y (mgr.14146) 1148 : cluster [DBG] pgmap v1570: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 5155/40224 objects misplaced (12.816%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:21 smithi121 bash[18489]: cluster 2024-02-12T16:23:20.225272+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e530: 12 total, 12 up, 12 in 2024-02-12T16:23:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:21 smithi121 bash[18489]: cluster 2024-02-12T16:23:20.584696+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e531: 12 total, 12 up, 12 in 2024-02-12T16:23:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:21 smithi121 bash[18489]: audit 2024-02-12T16:23:20.880576+0000 mon.a (mon.0) 1891 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:23:21.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:21.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:21.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:21.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:21 smithi169 bash[19604]: cluster 2024-02-12T16:23:19.812552+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:21.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:21 smithi169 bash[19604]: cluster 2024-02-12T16:23:19.812591+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:21 smithi169 bash[19604]: cluster 2024-02-12T16:23:19.906721+0000 mgr.y (mgr.14146) 1148 : cluster [DBG] pgmap v1570: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 5155/40224 objects misplaced (12.816%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:21 smithi169 bash[19604]: cluster 2024-02-12T16:23:20.225272+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e530: 12 total, 12 up, 12 in 2024-02-12T16:23:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:21 smithi169 bash[19604]: cluster 2024-02-12T16:23:20.584696+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e531: 12 total, 12 up, 12 in 2024-02-12T16:23:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:21 smithi169 bash[19604]: audit 2024-02-12T16:23:20.880576+0000 mon.a (mon.0) 1891 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:23:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:21.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:23:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:21 smithi123 bash[19119]: cluster 2024-02-12T16:23:19.812552+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:21 smithi123 bash[19119]: cluster 2024-02-12T16:23:19.812591+0000 osd.1 (osd.1) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:21 smithi123 bash[19119]: cluster 2024-02-12T16:23:19.906721+0000 mgr.y (mgr.14146) 1148 : cluster [DBG] pgmap v1570: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.2 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 5155/40224 objects misplaced (12.816%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:21 smithi123 bash[19119]: cluster 2024-02-12T16:23:20.225272+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e530: 12 total, 12 up, 12 in 2024-02-12T16:23:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:21 smithi123 bash[19119]: cluster 2024-02-12T16:23:20.584696+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e531: 12 total, 12 up, 12 in 2024-02-12T16:23:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:21 smithi123 bash[19119]: audit 2024-02-12T16:23:20.880576+0000 mon.a (mon.0) 1891 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:23:21.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:21.695 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:21.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:21.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:21.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:21.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:21.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:22.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:22.100 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:22.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:22.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:22.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:22.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:22.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:22.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:22.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:22.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:22.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:22.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:22.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:22.634 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:22.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:22.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:22.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:22.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:22.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:22.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:23.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:23.057 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:23.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:23.124 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:23.164 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:23.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:23.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:23.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:23.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:23.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:23 smithi123 bash[19119]: cluster 2024-02-12T16:23:21.907202+0000 mgr.y (mgr.14146) 1149 : cluster [DBG] pgmap v1573: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 4198/34734 objects misplaced (12.086%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:23:23.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:23.471 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:23.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:23.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:23 smithi169 bash[19604]: cluster 2024-02-12T16:23:21.907202+0000 mgr.y (mgr.14146) 1149 : cluster [DBG] pgmap v1573: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 4198/34734 objects misplaced (12.086%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:23:23.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:23 smithi121 bash[18489]: cluster 2024-02-12T16:23:21.907202+0000 mgr.y (mgr.14146) 1149 : cluster [DBG] pgmap v1573: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 4198/34734 objects misplaced (12.086%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:23:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:23.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:23.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:23.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:23.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:23.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:23.889 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:23.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:23.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:24.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:24.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:24.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:24.139 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:24.143 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:24.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:24.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:24.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:24.337 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:24.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:24.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:24.471 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:24.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:24.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:24.618 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:24.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:24.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:24.752 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:24.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:24.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:24.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:24.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:25.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:25.026 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:25.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:25.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:25.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:25.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:25.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:25 smithi169 bash[19604]: cluster 2024-02-12T16:23:23.907641+0000 mgr.y (mgr.14146) 1150 : cluster [DBG] pgmap v1574: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 975 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 3933/31186 objects misplaced (12.611%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:25.309 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:25.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:25 smithi123 bash[19119]: cluster 2024-02-12T16:23:23.907641+0000 mgr.y (mgr.14146) 1150 : cluster [DBG] pgmap v1574: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 975 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 3933/31186 objects misplaced (12.611%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:25.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:25.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:25.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:25.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:25.603 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:25.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:25.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:25 smithi121 bash[18489]: cluster 2024-02-12T16:23:23.907641+0000 mgr.y (mgr.14146) 1150 : cluster [DBG] pgmap v1574: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 975 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 3933/31186 objects misplaced (12.611%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:25.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:25.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:25.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:25.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:25.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:25.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:25.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:25.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:26.102 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:26.143 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:26.170 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:26.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:26.290 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:26.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:26.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:26.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:26.561 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:26.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:26 smithi121 bash[18489]: cluster 2024-02-12T16:23:25.908057+0000 mgr.y (mgr.14146) 1151 : cluster [DBG] pgmap v1575: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 753 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 2955/24082 objects misplaced (12.271%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:26.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:26.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:26.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:26.856 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:26 smithi169 bash[19604]: cluster 2024-02-12T16:23:25.908057+0000 mgr.y (mgr.14146) 1151 : cluster [DBG] pgmap v1575: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 753 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 2955/24082 objects misplaced (12.271%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:26.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:26 smithi123 bash[19119]: cluster 2024-02-12T16:23:25.908057+0000 mgr.y (mgr.14146) 1151 : cluster [DBG] pgmap v1575: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 753 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 2955/24082 objects misplaced (12.271%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:26.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:26.985 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:26.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:27.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:27.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:27.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:27.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:27.264 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:27.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:27.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:27.404 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:27.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:27.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:27.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:27.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:27.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:27.675 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:27.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:27.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:27.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:27.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:27.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:27.941 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:27.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:28.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:28.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:28.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:28.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:28.208 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:28.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:28.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:28.347 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:28.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:28.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:28.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:28.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:28.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:28.629 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:28.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:28.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:28.771 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:28.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:28.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:28.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:28.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:29.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:29.043 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:28 smithi169 bash[19604]: cluster 2024-02-12T16:23:27.908528+0000 mgr.y (mgr.14146) 1152 : cluster [DBG] pgmap v1576: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 687 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 2656/21980 objects misplaced (12.084%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:23:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:29.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:29.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:29.175 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:29.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:29.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:29.285 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:28 smithi121 bash[18489]: cluster 2024-02-12T16:23:27.908528+0000 mgr.y (mgr.14146) 1152 : cluster [DBG] pgmap v1576: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 687 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 2656/21980 objects misplaced (12.084%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:23:29.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:29.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:28 smithi123 bash[19119]: cluster 2024-02-12T16:23:27.908528+0000 mgr.y (mgr.14146) 1152 : cluster [DBG] pgmap v1576: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 687 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 2656/21980 objects misplaced (12.084%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:23:29.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:29.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:29.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:29.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:29.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:29.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:29.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:29.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:29.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:29.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:29.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:29.871 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:29.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:29.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:30.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:30.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:30.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:30.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.399 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:30.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:30.534 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:30.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:30.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:30.820 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:30.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:30.966 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:30.969 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:30 smithi169 bash[19604]: cluster 2024-02-12T16:23:29.908937+0000 mgr.y (mgr.14146) 1153 : cluster [DBG] pgmap v1577: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 553 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s; 2137/17700 objects misplaced (12.073%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:30.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:31.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:31.078 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:30 smithi123 bash[19119]: cluster 2024-02-12T16:23:29.908937+0000 mgr.y (mgr.14146) 1153 : cluster [DBG] pgmap v1577: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 553 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s; 2137/17700 objects misplaced (12.073%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:31.106 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:31.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:31.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:31.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:31.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:31.299 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:30 smithi121 bash[18489]: cluster 2024-02-12T16:23:29.908937+0000 mgr.y (mgr.14146) 1153 : cluster [DBG] pgmap v1577: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 553 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s; 2137/17700 objects misplaced (12.073%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:31.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:31.375 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:31.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:31.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:31.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:31.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:31.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:31.633 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:31.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:31.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:31.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:31.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:31.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:31.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:31.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:32.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:32.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:32.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:32.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:32.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:32.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:32.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:32.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:32.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:32.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:32.457 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:32.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:32.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:32.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:32.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:32.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:32.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:32.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:32.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:32.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:32.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:32.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:33.016 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:33.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:33.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:32 smithi121 bash[18489]: cluster 2024-02-12T16:23:31.909361+0000 mgr.y (mgr.14146) 1154 : cluster [DBG] pgmap v1578: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 373 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 1128/11938 objects misplaced (9.449%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:33.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:33.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:33.268 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:32 smithi169 bash[19604]: cluster 2024-02-12T16:23:31.909361+0000 mgr.y (mgr.14146) 1154 : cluster [DBG] pgmap v1578: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 373 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 1128/11938 objects misplaced (9.449%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:33.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:32 smithi123 bash[19119]: cluster 2024-02-12T16:23:31.909361+0000 mgr.y (mgr.14146) 1154 : cluster [DBG] pgmap v1578: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 373 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 1128/11938 objects misplaced (9.449%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:33.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:33.431 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:33.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:33.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:33.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:33.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:33.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:33.824 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:33.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:33.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:33.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:34.000 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:23:34.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:34.086 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:34.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:34.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:34.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:34.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:33 smithi169 bash[19604]: audit 2024-02-12T16:23:33.650141+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? v1:172.21.15.121:0/1472838670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:23:34.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:34.325 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:33 smithi123 bash[19119]: audit 2024-02-12T16:23:33.650141+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? v1:172.21.15.121:0/1472838670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:23:34.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:34.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:33 smithi121 bash[18489]: audit 2024-02-12T16:23:33.650141+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? v1:172.21.15.121:0/1472838670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:23:34.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:34.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:34.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:34.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:34.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:34.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:34.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:34.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:34.746 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 173334 objects 2024-02-12T16:23:34.746 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :485.570499 2024-02-12T16:23:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:34.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:34.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:34.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:34.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:34.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:35.019 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:35.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:35.135 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:34 smithi121 bash[18489]: cluster 2024-02-12T16:23:33.909860+0000 mgr.y (mgr.14146) 1155 : cluster [DBG] pgmap v1579: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 264 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 948/8446 objects misplaced (11.224%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T16:23:35.135 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:34 smithi121 bash[18489]: cluster 2024-02-12T16:23:33.970716+0000 mon.a (mon.0) 1893 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:23:35.135 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:34 smithi121 bash[18489]: audit 2024-02-12T16:23:33.991298+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? v1:172.21.15.121:0/1472838670' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:23:35.135 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:34 smithi121 bash[18489]: cluster 2024-02-12T16:23:33.991464+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e532: 12 total, 12 up, 12 in 2024-02-12T16:23:35.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:34 smithi123 bash[19119]: cluster 2024-02-12T16:23:33.909860+0000 mgr.y (mgr.14146) 1155 : cluster [DBG] pgmap v1579: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 264 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 948/8446 objects misplaced (11.224%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T16:23:35.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:34 smithi123 bash[19119]: cluster 2024-02-12T16:23:33.970716+0000 mon.a (mon.0) 1893 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:23:35.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:34 smithi123 bash[19119]: audit 2024-02-12T16:23:33.991298+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? v1:172.21.15.121:0/1472838670' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:23:35.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:34 smithi123 bash[19119]: cluster 2024-02-12T16:23:33.991464+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e532: 12 total, 12 up, 12 in 2024-02-12T16:23:35.162 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:35.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:34 smithi169 bash[19604]: cluster 2024-02-12T16:23:33.909860+0000 mgr.y (mgr.14146) 1155 : cluster [DBG] pgmap v1579: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 264 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 948/8446 objects misplaced (11.224%); 2.0 MiB/s, 32 objects/s recovering 2024-02-12T16:23:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:34 smithi169 bash[19604]: cluster 2024-02-12T16:23:33.970716+0000 mon.a (mon.0) 1893 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:23:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:34 smithi169 bash[19604]: audit 2024-02-12T16:23:33.991298+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? v1:172.21.15.121:0/1472838670' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:23:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:34 smithi169 bash[19604]: cluster 2024-02-12T16:23:33.991464+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e532: 12 total, 12 up, 12 in 2024-02-12T16:23:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:35.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:35.444 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:35.539 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:23:35.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:35.558 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:23:35.577 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:23:35.579 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:35.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:23:35.629 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:23:35.652 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:23:35.672 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:23:35.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:35.697 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:23:35.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:35.724 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:23:35.751 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:23:35.790 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:23:35.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:35.824 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.826 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:23:35.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:35.886 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:23:35.886 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:23:35.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:35.976 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:35.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:36.020 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T16:23:36.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:35 smithi121 bash[18489]: cluster 2024-02-12T16:23:34.650338+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:23:36.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:35 smithi121 bash[18489]: cluster 2024-02-12T16:23:34.745153+0000 osd.6 (osd.6) 93 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:23:36.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:35 smithi121 bash[18489]: cluster 2024-02-12T16:23:34.999221+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e533: 12 total, 12 up, 12 in 2024-02-12T16:23:36.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:35 smithi121 bash[18489]: cluster 2024-02-12T16:23:35.167592+0000 osd.7 (osd.7) 116 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:23:36.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:35 smithi121 bash[18489]: cluster 2024-02-12T16:23:35.183304+0000 osd.7 (osd.7) 117 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:23:36.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:35 smithi121 bash[18489]: audit 2024-02-12T16:23:35.514091+0000 mon.a (mon.0) 1897 : audit [DBG] from='client.? v1:172.21.15.121:0/2706236515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:36.075 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:35 smithi121 bash[18489]: audit 2024-02-12T16:23:35.758406+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? v1:172.21.15.121:0/4163758908' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:23:36.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:36.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:36.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:36.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:36.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:36.303 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:36 smithi169 bash[19604]: cluster 2024-02-12T16:23:34.650338+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:23:36.303 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:36 smithi169 bash[19604]: cluster 2024-02-12T16:23:34.745153+0000 osd.6 (osd.6) 93 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:23:36.303 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:36 smithi169 bash[19604]: cluster 2024-02-12T16:23:34.999221+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e533: 12 total, 12 up, 12 in 2024-02-12T16:23:36.303 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:36 smithi169 bash[19604]: cluster 2024-02-12T16:23:35.167592+0000 osd.7 (osd.7) 116 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:23:36.303 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:36 smithi169 bash[19604]: cluster 2024-02-12T16:23:35.183304+0000 osd.7 (osd.7) 117 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:23:36.303 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:36 smithi169 bash[19604]: audit 2024-02-12T16:23:35.514091+0000 mon.a (mon.0) 1897 : audit [DBG] from='client.? v1:172.21.15.121:0/2706236515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:36.303 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:36 smithi169 bash[19604]: audit 2024-02-12T16:23:35.758406+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? v1:172.21.15.121:0/4163758908' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:23:36.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:36.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:36 smithi123 bash[19119]: cluster 2024-02-12T16:23:34.650338+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.5 scrub starts 2024-02-12T16:23:36.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:36 smithi123 bash[19119]: cluster 2024-02-12T16:23:34.745153+0000 osd.6 (osd.6) 93 : cluster [DBG] 4.5 scrub ok 2024-02-12T16:23:36.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:36 smithi123 bash[19119]: cluster 2024-02-12T16:23:34.999221+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e533: 12 total, 12 up, 12 in 2024-02-12T16:23:36.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:36 smithi123 bash[19119]: cluster 2024-02-12T16:23:35.167592+0000 osd.7 (osd.7) 116 : cluster [DBG] 4.3 scrub starts 2024-02-12T16:23:36.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:36 smithi123 bash[19119]: cluster 2024-02-12T16:23:35.183304+0000 osd.7 (osd.7) 117 : cluster [DBG] 4.3 scrub ok 2024-02-12T16:23:36.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:36 smithi123 bash[19119]: audit 2024-02-12T16:23:35.514091+0000 mon.a (mon.0) 1897 : audit [DBG] from='client.? v1:172.21.15.121:0/2706236515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:36.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:36 smithi123 bash[19119]: audit 2024-02-12T16:23:35.758406+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? v1:172.21.15.121:0/4163758908' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:23:36.402 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:36.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:36.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:36.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:36.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:36.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:36.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:36.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:36.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:36.822 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:36.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:36.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:36.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:36.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:37.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:37.100 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:37.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:37.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:37 smithi121 bash[18489]: cluster 2024-02-12T16:23:35.399110+0000 osd.9 (osd.9) 81 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:23:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:37 smithi121 bash[18489]: cluster 2024-02-12T16:23:35.439862+0000 osd.9 (osd.9) 82 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:23:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:37 smithi121 bash[18489]: cluster 2024-02-12T16:23:35.839978+0000 osd.11 (osd.11) 112 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T16:23:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:37 smithi121 bash[18489]: cluster 2024-02-12T16:23:35.872474+0000 osd.11 (osd.11) 113 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T16:23:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:37 smithi121 bash[18489]: cluster 2024-02-12T16:23:35.910617+0000 mgr.y (mgr.14146) 1156 : cluster [DBG] pgmap v1582: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 62 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 141/1970 objects misplaced (7.157%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:37 smithi121 bash[18489]: audit 2024-02-12T16:23:36.011092+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.121:0/4163758908' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:23:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:37 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.011176+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e534: 12 total, 12 up, 12 in 2024-02-12T16:23:37.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:37.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:37.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:37.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:37 smithi169 bash[19604]: cluster 2024-02-12T16:23:35.399110+0000 osd.9 (osd.9) 81 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:23:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:37 smithi169 bash[19604]: cluster 2024-02-12T16:23:35.439862+0000 osd.9 (osd.9) 82 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:23:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:37 smithi169 bash[19604]: cluster 2024-02-12T16:23:35.839978+0000 osd.11 (osd.11) 112 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T16:23:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:37 smithi169 bash[19604]: cluster 2024-02-12T16:23:35.872474+0000 osd.11 (osd.11) 113 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T16:23:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:37 smithi169 bash[19604]: cluster 2024-02-12T16:23:35.910617+0000 mgr.y (mgr.14146) 1156 : cluster [DBG] pgmap v1582: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 62 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 141/1970 objects misplaced (7.157%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:37 smithi169 bash[19604]: audit 2024-02-12T16:23:36.011092+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.121:0/4163758908' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:23:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:37 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.011176+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e534: 12 total, 12 up, 12 in 2024-02-12T16:23:37.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:37.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:37.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:37 smithi123 bash[19119]: cluster 2024-02-12T16:23:35.399110+0000 osd.9 (osd.9) 81 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:23:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:37 smithi123 bash[19119]: cluster 2024-02-12T16:23:35.439862+0000 osd.9 (osd.9) 82 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:23:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:37 smithi123 bash[19119]: cluster 2024-02-12T16:23:35.839978+0000 osd.11 (osd.11) 112 : cluster [DBG] 4.8 deep-scrub starts 2024-02-12T16:23:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:37 smithi123 bash[19119]: cluster 2024-02-12T16:23:35.872474+0000 osd.11 (osd.11) 113 : cluster [DBG] 4.8 deep-scrub ok 2024-02-12T16:23:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:37 smithi123 bash[19119]: cluster 2024-02-12T16:23:35.910617+0000 mgr.y (mgr.14146) 1156 : cluster [DBG] pgmap v1582: 37 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 33 active+clean; 62 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 141/1970 objects misplaced (7.157%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:23:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:37 smithi123 bash[19119]: audit 2024-02-12T16:23:36.011092+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.121:0/4163758908' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:23:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:37 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.011176+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e534: 12 total, 12 up, 12 in 2024-02-12T16:23:37.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:37.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:37.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:37.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:37.661 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:37.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:37.746 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-02-12T16:23:37.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-02-12T16:23:37.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:37.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:23:37.801 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:37.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:37.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:37.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:23:37.939 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:37.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:38.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.258522+0000 osd.5 (osd.5) 75 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:23:38.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.307663+0000 osd.5 (osd.5) 76 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:23:38.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.413892+0000 osd.9 (osd.9) 83 : cluster [DBG] 1.a scrub starts 2024-02-12T16:23:38.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.433269+0000 osd.9 (osd.9) 84 : cluster [DBG] 1.a scrub ok 2024-02-12T16:23:38.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.441922+0000 osd.3 (osd.3) 4 : cluster [DBG] 4.f scrub starts 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.449404+0000 osd.3 (osd.3) 5 : cluster [DBG] 4.f scrub ok 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.791543+0000 osd.11 (osd.11) 114 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.820886+0000 osd.11 (osd.11) 115 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:36.998897+0000 osd.10 (osd.10) 94 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.006966+0000 osd.10 (osd.10) 95 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.139307+0000 osd.7 (osd.7) 118 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.159795+0000 osd.7 (osd.7) 119 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:23:38.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:38 smithi121 bash[18489]: audit 2024-02-12T16:23:37.912329+0000 mon.a (mon.0) 1901 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:23:38.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:38.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:38.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:38.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T16:23:38.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.258522+0000 osd.5 (osd.5) 75 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.307663+0000 osd.5 (osd.5) 76 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.413892+0000 osd.9 (osd.9) 83 : cluster [DBG] 1.a scrub starts 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.433269+0000 osd.9 (osd.9) 84 : cluster [DBG] 1.a scrub ok 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.441922+0000 osd.3 (osd.3) 4 : cluster [DBG] 4.f scrub starts 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.449404+0000 osd.3 (osd.3) 5 : cluster [DBG] 4.f scrub ok 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.791543+0000 osd.11 (osd.11) 114 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.820886+0000 osd.11 (osd.11) 115 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:36.998897+0000 osd.10 (osd.10) 94 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:23:38.187 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.006966+0000 osd.10 (osd.10) 95 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:23:38.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.139307+0000 osd.7 (osd.7) 118 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:23:38.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.159795+0000 osd.7 (osd.7) 119 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:23:38.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:38 smithi169 bash[19604]: audit 2024-02-12T16:23:37.912329+0000 mon.a (mon.0) 1901 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:23:38.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:38.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:38.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:38.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.258522+0000 osd.5 (osd.5) 75 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.307663+0000 osd.5 (osd.5) 76 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.413892+0000 osd.9 (osd.9) 83 : cluster [DBG] 1.a scrub starts 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.433269+0000 osd.9 (osd.9) 84 : cluster [DBG] 1.a scrub ok 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.441922+0000 osd.3 (osd.3) 4 : cluster [DBG] 4.f scrub starts 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.449404+0000 osd.3 (osd.3) 5 : cluster [DBG] 4.f scrub ok 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.791543+0000 osd.11 (osd.11) 114 : cluster [DBG] 4.1 scrub starts 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.820886+0000 osd.11 (osd.11) 115 : cluster [DBG] 4.1 scrub ok 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:36.998897+0000 osd.10 (osd.10) 94 : cluster [DBG] 4.7 scrub starts 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.006966+0000 osd.10 (osd.10) 95 : cluster [DBG] 4.7 scrub ok 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.139307+0000 osd.7 (osd.7) 118 : cluster [DBG] 4.0 scrub starts 2024-02-12T16:23:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.159795+0000 osd.7 (osd.7) 119 : cluster [DBG] 4.0 scrub ok 2024-02-12T16:23:38.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:38 smithi123 bash[19119]: audit 2024-02-12T16:23:37.912329+0000 mon.a (mon.0) 1901 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:23:38.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:38.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:38.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:38.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:38.624 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:38.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:38.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:38.767 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:38.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:38.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:38.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:38.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:39.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.445361+0000 osd.3 (osd.3) 6 : cluster [DBG] 1.b scrub starts 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.477953+0000 osd.3 (osd.3) 7 : cluster [DBG] 1.b scrub ok 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.683071+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.a scrub starts 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.706454+0000 osd.6 (osd.6) 95 : cluster [DBG] 4.a scrub ok 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.809501+0000 osd.11 (osd.11) 116 : cluster [DBG] 1.c scrub starts 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.904546+0000 osd.11 (osd.11) 117 : cluster [DBG] 1.c scrub ok 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:37.911386+0000 mgr.y (mgr.14146) 1157 : cluster [DBG] pgmap v1584: 37 pgs: 1 active, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 33 active+clean; 23 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 28/736 objects misplaced (3.804%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: audit 2024-02-12T16:23:38.014545+0000 mon.a (mon.0) 1902 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:23:39.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:38.014639+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e535: 12 total, 12 up, 12 in 2024-02-12T16:23:39.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: audit 2024-02-12T16:23:38.458723+0000 mon.a (mon.0) 1904 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:23:39.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:38.469846+0000 osd.9 (osd.9) 85 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:23:39.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: cluster 2024-02-12T16:23:38.495566+0000 osd.9 (osd.9) 86 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:23:39.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: audit 2024-02-12T16:23:38.756599+0000 mon.a (mon.0) 1905 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:23:39.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: audit 2024-02-12T16:23:38.758262+0000 mon.a (mon.0) 1906 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:23:39.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:39 smithi169 bash[19604]: audit 2024-02-12T16:23:38.767579+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:23:39.044 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:39.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:39.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:39.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.445361+0000 osd.3 (osd.3) 6 : cluster [DBG] 1.b scrub starts 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.477953+0000 osd.3 (osd.3) 7 : cluster [DBG] 1.b scrub ok 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.683071+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.a scrub starts 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.706454+0000 osd.6 (osd.6) 95 : cluster [DBG] 4.a scrub ok 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.809501+0000 osd.11 (osd.11) 116 : cluster [DBG] 1.c scrub starts 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.904546+0000 osd.11 (osd.11) 117 : cluster [DBG] 1.c scrub ok 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:37.911386+0000 mgr.y (mgr.14146) 1157 : cluster [DBG] pgmap v1584: 37 pgs: 1 active, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 33 active+clean; 23 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 28/736 objects misplaced (3.804%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: audit 2024-02-12T16:23:38.014545+0000 mon.a (mon.0) 1902 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:38.014639+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e535: 12 total, 12 up, 12 in 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: audit 2024-02-12T16:23:38.458723+0000 mon.a (mon.0) 1904 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:38.469846+0000 osd.9 (osd.9) 85 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: cluster 2024-02-12T16:23:38.495566+0000 osd.9 (osd.9) 86 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: audit 2024-02-12T16:23:38.756599+0000 mon.a (mon.0) 1905 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:23:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: audit 2024-02-12T16:23:38.758262+0000 mon.a (mon.0) 1906 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:23:39.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:39 smithi121 bash[18489]: audit 2024-02-12T16:23:38.767579+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:23:39.173 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:39.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:39.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:39.308 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:39.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:39.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:39.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:39.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:39.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:39.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:39.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:39.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:39.732 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:39.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:39.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:39.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:39.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.445361+0000 osd.3 (osd.3) 6 : cluster [DBG] 1.b scrub starts 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.477953+0000 osd.3 (osd.3) 7 : cluster [DBG] 1.b scrub ok 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.683071+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.a scrub starts 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.706454+0000 osd.6 (osd.6) 95 : cluster [DBG] 4.a scrub ok 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.809501+0000 osd.11 (osd.11) 116 : cluster [DBG] 1.c scrub starts 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.904546+0000 osd.11 (osd.11) 117 : cluster [DBG] 1.c scrub ok 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:37.911386+0000 mgr.y (mgr.14146) 1157 : cluster [DBG] pgmap v1584: 37 pgs: 1 active, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 33 active+clean; 23 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 28/736 objects misplaced (3.804%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: audit 2024-02-12T16:23:38.014545+0000 mon.a (mon.0) 1902 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:38.014639+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e535: 12 total, 12 up, 12 in 2024-02-12T16:23:39.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: audit 2024-02-12T16:23:38.458723+0000 mon.a (mon.0) 1904 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:23:39.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:38.469846+0000 osd.9 (osd.9) 85 : cluster [DBG] 4.2 scrub starts 2024-02-12T16:23:39.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: cluster 2024-02-12T16:23:38.495566+0000 osd.9 (osd.9) 86 : cluster [DBG] 4.2 scrub ok 2024-02-12T16:23:39.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: audit 2024-02-12T16:23:38.756599+0000 mon.a (mon.0) 1905 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:23:39.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: audit 2024-02-12T16:23:38.758262+0000 mon.a (mon.0) 1906 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:23:39.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:39 smithi123 bash[19119]: audit 2024-02-12T16:23:38.767579+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:23:39.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:23:39.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:40.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:40.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:40.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:40.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:40.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:40.274 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:40 smithi123 bash[19119]: cluster 2024-02-12T16:23:38.783894+0000 osd.11 (osd.11) 118 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:23:40.274 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:40 smithi123 bash[19119]: cluster 2024-02-12T16:23:38.850167+0000 osd.11 (osd.11) 119 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:23:40.274 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:40 smithi123 bash[19119]: cluster 2024-02-12T16:23:38.978086+0000 osd.10 (osd.10) 96 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:23:40.274 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:40 smithi123 bash[19119]: cluster 2024-02-12T16:23:38.993683+0000 osd.10 (osd.10) 97 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:23:40.274 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:40 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.019329+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e536: 12 total, 12 up, 12 in 2024-02-12T16:23:40.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:40 smithi169 bash[19604]: cluster 2024-02-12T16:23:38.783894+0000 osd.11 (osd.11) 118 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:23:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:40 smithi169 bash[19604]: cluster 2024-02-12T16:23:38.850167+0000 osd.11 (osd.11) 119 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:23:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:40 smithi169 bash[19604]: cluster 2024-02-12T16:23:38.978086+0000 osd.10 (osd.10) 96 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:23:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:40 smithi169 bash[19604]: cluster 2024-02-12T16:23:38.993683+0000 osd.10 (osd.10) 97 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:23:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:40 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.019329+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e536: 12 total, 12 up, 12 in 2024-02-12T16:23:40.299 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:40.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:40.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:40.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:40 smithi121 bash[18489]: cluster 2024-02-12T16:23:38.783894+0000 osd.11 (osd.11) 118 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:23:40.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:40 smithi121 bash[18489]: cluster 2024-02-12T16:23:38.850167+0000 osd.11 (osd.11) 119 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:23:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:40 smithi121 bash[18489]: cluster 2024-02-12T16:23:38.978086+0000 osd.10 (osd.10) 96 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:23:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:40 smithi121 bash[18489]: cluster 2024-02-12T16:23:38.993683+0000 osd.10 (osd.10) 97 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:23:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:40 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.019329+0000 mon.a (mon.0) 1908 : cluster [DBG] osdmap e536: 12 total, 12 up, 12 in 2024-02-12T16:23:40.443 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:40.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:40.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:40.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:40.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:40.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:40.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:40.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:40.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:40.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:40.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:40.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:41.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:41.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:41.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.424026+0000 osd.2 (osd.2) 62 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:23:41.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.455562+0000 osd.2 (osd.2) 63 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:23:41.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.666333+0000 osd.6 (osd.6) 96 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:23:41.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.706005+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:23:41.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.776089+0000 osd.11 (osd.11) 120 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:23:41.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.867492+0000 osd.11 (osd.11) 121 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:23:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:39.912363+0000 mgr.y (mgr.14146) 1158 : cluster [DBG] pgmap v1587: 37 pgs: 1 peering, 1 active, 1 active+remapped+backfill_toofull, 34 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 651 KiB/s, 10 objects/s recovering 2024-02-12T16:23:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.115721+0000 osd.7 (osd.7) 120 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:23:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.145829+0000 osd.7 (osd.7) 121 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:23:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.405531+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub starts 2024-02-12T16:23:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.417657+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.b scrub ok 2024-02-12T16:23:41.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:41 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.589859+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e537: 12 total, 12 up, 12 in 2024-02-12T16:23:41.149 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:41.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:41.292 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.424026+0000 osd.2 (osd.2) 62 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.455562+0000 osd.2 (osd.2) 63 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.666333+0000 osd.6 (osd.6) 96 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.706005+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.776089+0000 osd.11 (osd.11) 120 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.867492+0000 osd.11 (osd.11) 121 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:39.912363+0000 mgr.y (mgr.14146) 1158 : cluster [DBG] pgmap v1587: 37 pgs: 1 peering, 1 active, 1 active+remapped+backfill_toofull, 34 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 651 KiB/s, 10 objects/s recovering 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.115721+0000 osd.7 (osd.7) 120 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:23:41.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.145829+0000 osd.7 (osd.7) 121 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:23:41.295 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.405531+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub starts 2024-02-12T16:23:41.295 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.417657+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.b scrub ok 2024-02-12T16:23:41.295 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:41 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.589859+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e537: 12 total, 12 up, 12 in 2024-02-12T16:23:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:41.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:41.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.424026+0000 osd.2 (osd.2) 62 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.455562+0000 osd.2 (osd.2) 63 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.666333+0000 osd.6 (osd.6) 96 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.706005+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.776089+0000 osd.11 (osd.11) 120 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.867492+0000 osd.11 (osd.11) 121 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:39.912363+0000 mgr.y (mgr.14146) 1158 : cluster [DBG] pgmap v1587: 37 pgs: 1 peering, 1 active, 1 active+remapped+backfill_toofull, 34 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 651 KiB/s, 10 objects/s recovering 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.115721+0000 osd.7 (osd.7) 120 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.145829+0000 osd.7 (osd.7) 121 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.405531+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub starts 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.417657+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.b scrub ok 2024-02-12T16:23:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:41 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.589859+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e537: 12 total, 12 up, 12 in 2024-02-12T16:23:41.431 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:41.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:41.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:41.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:41.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:41.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:41.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:41.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:41.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:41.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:42.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:42.125 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:42.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:42.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.653263+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:23:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.694319+0000 osd.6 (osd.6) 99 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:23:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.959859+0000 osd.10 (osd.10) 98 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:23:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:40.989374+0000 osd.10 (osd.10) 99 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:23:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:41.375838+0000 osd.9 (osd.9) 87 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:23:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:41.393606+0000 osd.9 (osd.9) 88 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:23:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:41.687364+0000 osd.4 (osd.4) 78 : cluster [DBG] 4.c deep-scrub starts 2024-02-12T16:23:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:42 smithi121 bash[18489]: cluster 2024-02-12T16:23:41.712062+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.c deep-scrub ok 2024-02-12T16:23:42.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:42.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:42.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.653263+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.694319+0000 osd.6 (osd.6) 99 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.959859+0000 osd.10 (osd.10) 98 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:40.989374+0000 osd.10 (osd.10) 99 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:41.375838+0000 osd.9 (osd.9) 87 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:41.393606+0000 osd.9 (osd.9) 88 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:41.687364+0000 osd.4 (osd.4) 78 : cluster [DBG] 4.c deep-scrub starts 2024-02-12T16:23:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:42 smithi169 bash[19604]: cluster 2024-02-12T16:23:41.712062+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.c deep-scrub ok 2024-02-12T16:23:42.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:42.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.653263+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:23:42.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.694319+0000 osd.6 (osd.6) 99 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:23:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.959859+0000 osd.10 (osd.10) 98 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:23:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:40.989374+0000 osd.10 (osd.10) 99 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:23:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:41.375838+0000 osd.9 (osd.9) 87 : cluster [DBG] 4.6 scrub starts 2024-02-12T16:23:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:41.393606+0000 osd.9 (osd.9) 88 : cluster [DBG] 4.6 scrub ok 2024-02-12T16:23:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:41.687364+0000 osd.4 (osd.4) 78 : cluster [DBG] 4.c deep-scrub starts 2024-02-12T16:23:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:42 smithi123 bash[19119]: cluster 2024-02-12T16:23:41.712062+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.c deep-scrub ok 2024-02-12T16:23:42.409 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:42.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:42.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:42.549 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:42.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:42.558 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:42.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:42.693 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:42.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:42.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:42.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:42.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:42.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:42.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:43.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:43.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:43 smithi121 bash[18489]: cluster 2024-02-12T16:23:41.913228+0000 mgr.y (mgr.14146) 1159 : cluster [DBG] pgmap v1589: 37 pgs: 1 active+remapped+backfill_toofull, 1 peering, 35 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 542 KiB/s, 8 objects/s recovering 2024-02-12T16:23:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:43 smithi121 bash[18489]: cluster 2024-02-12T16:23:42.721494+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.e scrub starts 2024-02-12T16:23:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:43 smithi121 bash[18489]: cluster 2024-02-12T16:23:42.738744+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.e scrub ok 2024-02-12T16:23:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:43 smithi121 bash[18489]: cluster 2024-02-12T16:23:42.826471+0000 osd.9 (osd.9) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:43 smithi121 bash[18489]: cluster 2024-02-12T16:23:42.826555+0000 osd.9 (osd.9) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:43.214 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:43.245 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:43.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:43.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:43.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:43 smithi169 bash[19604]: cluster 2024-02-12T16:23:41.913228+0000 mgr.y (mgr.14146) 1159 : cluster [DBG] pgmap v1589: 37 pgs: 1 active+remapped+backfill_toofull, 1 peering, 35 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 542 KiB/s, 8 objects/s recovering 2024-02-12T16:23:43.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:43 smithi169 bash[19604]: cluster 2024-02-12T16:23:42.721494+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.e scrub starts 2024-02-12T16:23:43.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:43 smithi169 bash[19604]: cluster 2024-02-12T16:23:42.738744+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.e scrub ok 2024-02-12T16:23:43.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:43 smithi169 bash[19604]: cluster 2024-02-12T16:23:42.826471+0000 osd.9 (osd.9) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:43.360 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:43 smithi169 bash[19604]: cluster 2024-02-12T16:23:42.826555+0000 osd.9 (osd.9) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:43.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:43.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:43 smithi123 bash[19119]: cluster 2024-02-12T16:23:41.913228+0000 mgr.y (mgr.14146) 1159 : cluster [DBG] pgmap v1589: 37 pgs: 1 active+remapped+backfill_toofull, 1 peering, 35 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 542 KiB/s, 8 objects/s recovering 2024-02-12T16:23:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:43 smithi123 bash[19119]: cluster 2024-02-12T16:23:42.721494+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.e scrub starts 2024-02-12T16:23:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:43 smithi123 bash[19119]: cluster 2024-02-12T16:23:42.738744+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.e scrub ok 2024-02-12T16:23:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:43 smithi123 bash[19119]: cluster 2024-02-12T16:23:42.826471+0000 osd.9 (osd.9) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:43 smithi123 bash[19119]: cluster 2024-02-12T16:23:42.826555+0000 osd.9 (osd.9) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:43.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:43.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:43.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:43.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:43.662 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:43.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:43.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:43.800 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:43.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:43.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:43.942 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:43.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:44.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:44.084 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:44.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:44.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:44.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:44.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:44.341 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:44 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.033718+0000 osd.10 (osd.10) 100 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:23:44.341 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:44 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.057473+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e538: 12 total, 12 up, 12 in 2024-02-12T16:23:44.341 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:44 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.068322+0000 osd.10 (osd.10) 101 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:23:44.365 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:44.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:44 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.033718+0000 osd.10 (osd.10) 100 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:23:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:44 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.057473+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e538: 12 total, 12 up, 12 in 2024-02-12T16:23:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:44 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.068322+0000 osd.10 (osd.10) 101 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:23:44.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:44.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:44 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.033718+0000 osd.10 (osd.10) 100 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:23:44.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:44 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.057473+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e538: 12 total, 12 up, 12 in 2024-02-12T16:23:44.482 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:44 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.068322+0000 osd.10 (osd.10) 101 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:23:44.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:44.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:44.586 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:44.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:44.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:44.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:44.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:44.787 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:44.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:44.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:44.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:45.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:45.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:45.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:45 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.175852+0000 osd.5 (osd.5) 77 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:23:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:45 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.186346+0000 osd.5 (osd.5) 78 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:23:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:45 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.488480+0000 osd.3 (osd.3) 8 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:23:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:45 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.525380+0000 osd.3 (osd.3) 9 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:23:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:45 smithi121 bash[18489]: cluster 2024-02-12T16:23:43.914022+0000 mgr.y (mgr.14146) 1160 : cluster [DBG] pgmap v1591: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 62 op/s; 130 KiB/s, 2 objects/s recovering 2024-02-12T16:23:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:45 smithi121 bash[18489]: cluster 2024-02-12T16:23:44.094308+0000 osd.7 (osd.7) 122 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:23:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:45 smithi121 bash[18489]: cluster 2024-02-12T16:23:44.153687+0000 osd.7 (osd.7) 123 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:23:45.174 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:45.215 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:45.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:45.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:45.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:45 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.175852+0000 osd.5 (osd.5) 77 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:23:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:45 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.186346+0000 osd.5 (osd.5) 78 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:23:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:45 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.488480+0000 osd.3 (osd.3) 8 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:23:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:45 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.525380+0000 osd.3 (osd.3) 9 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:23:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:45 smithi123 bash[19119]: cluster 2024-02-12T16:23:43.914022+0000 mgr.y (mgr.14146) 1160 : cluster [DBG] pgmap v1591: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 62 op/s; 130 KiB/s, 2 objects/s recovering 2024-02-12T16:23:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:45 smithi123 bash[19119]: cluster 2024-02-12T16:23:44.094308+0000 osd.7 (osd.7) 122 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:23:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:45 smithi123 bash[19119]: cluster 2024-02-12T16:23:44.153687+0000 osd.7 (osd.7) 123 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:23:45.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:45.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:45 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.175852+0000 osd.5 (osd.5) 77 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:23:45.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:45 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.186346+0000 osd.5 (osd.5) 78 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:23:45.471 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:45 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.488480+0000 osd.3 (osd.3) 8 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:23:45.472 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:45 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.525380+0000 osd.3 (osd.3) 9 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:23:45.472 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:45 smithi169 bash[19604]: cluster 2024-02-12T16:23:43.914022+0000 mgr.y (mgr.14146) 1160 : cluster [DBG] pgmap v1591: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 62 op/s; 130 KiB/s, 2 objects/s recovering 2024-02-12T16:23:45.472 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:45 smithi169 bash[19604]: cluster 2024-02-12T16:23:44.094308+0000 osd.7 (osd.7) 122 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:23:45.472 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:45 smithi169 bash[19604]: cluster 2024-02-12T16:23:44.153687+0000 osd.7 (osd.7) 123 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:23:45.494 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:45.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:45.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:45.626 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:45.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:45.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:45.766 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:45.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:45.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:45.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:45.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:46.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:46.056 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:46.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:46.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:46.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:46.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:46.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:46 smithi169 bash[19604]: cluster 2024-02-12T16:23:44.151254+0000 osd.5 (osd.5) 79 : cluster [DBG] 1.12 deep-scrub starts 2024-02-12T16:23:46.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:46 smithi169 bash[19604]: cluster 2024-02-12T16:23:44.185843+0000 osd.5 (osd.5) 80 : cluster [DBG] 1.12 deep-scrub ok 2024-02-12T16:23:46.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:46 smithi169 bash[19604]: cluster 2024-02-12T16:23:45.596120+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e539: 12 total, 12 up, 12 in 2024-02-12T16:23:46.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:46.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:46.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:46 smithi123 bash[19119]: cluster 2024-02-12T16:23:44.151254+0000 osd.5 (osd.5) 79 : cluster [DBG] 1.12 deep-scrub starts 2024-02-12T16:23:46.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:46 smithi123 bash[19119]: cluster 2024-02-12T16:23:44.185843+0000 osd.5 (osd.5) 80 : cluster [DBG] 1.12 deep-scrub ok 2024-02-12T16:23:46.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:46 smithi123 bash[19119]: cluster 2024-02-12T16:23:45.596120+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e539: 12 total, 12 up, 12 in 2024-02-12T16:23:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:46.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:46 smithi121 bash[18489]: cluster 2024-02-12T16:23:44.151254+0000 osd.5 (osd.5) 79 : cluster [DBG] 1.12 deep-scrub starts 2024-02-12T16:23:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:46 smithi121 bash[18489]: cluster 2024-02-12T16:23:44.185843+0000 osd.5 (osd.5) 80 : cluster [DBG] 1.12 deep-scrub ok 2024-02-12T16:23:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:46 smithi121 bash[18489]: cluster 2024-02-12T16:23:45.596120+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e539: 12 total, 12 up, 12 in 2024-02-12T16:23:46.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:46.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:46.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:46.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:46.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:46.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:46.726 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:46.763 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:46.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:46.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:46.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:46.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:47.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:47.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:47.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:47.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:47.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:47.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:47.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:47.306 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:47 smithi123 bash[19119]: cluster 2024-02-12T16:23:45.914743+0000 mgr.y (mgr.14146) 1161 : cluster [DBG] pgmap v1593: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:47.306 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:47 smithi123 bash[19119]: cluster 2024-02-12T16:23:46.462678+0000 osd.0 (osd.0) 82 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:23:47.306 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:47 smithi123 bash[19119]: cluster 2024-02-12T16:23:46.525646+0000 osd.0 (osd.0) 83 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:23:47.332 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:47.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:47 smithi121 bash[18489]: cluster 2024-02-12T16:23:45.914743+0000 mgr.y (mgr.14146) 1161 : cluster [DBG] pgmap v1593: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:47 smithi121 bash[18489]: cluster 2024-02-12T16:23:46.462678+0000 osd.0 (osd.0) 82 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:23:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:47 smithi121 bash[18489]: cluster 2024-02-12T16:23:46.525646+0000 osd.0 (osd.0) 83 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:23:47.436 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:47.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:47.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:47.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:47 smithi169 bash[19604]: cluster 2024-02-12T16:23:45.914743+0000 mgr.y (mgr.14146) 1161 : cluster [DBG] pgmap v1593: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:47 smithi169 bash[19604]: cluster 2024-02-12T16:23:46.462678+0000 osd.0 (osd.0) 82 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:23:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:47 smithi169 bash[19604]: cluster 2024-02-12T16:23:46.525646+0000 osd.0 (osd.0) 83 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:23:47.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:47.616 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:47.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:47.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:47.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:47.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:47.902 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:47.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:48.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:48.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:48.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:48.179 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:48.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:48.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:48.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:48 smithi169 bash[19604]: cluster 2024-02-12T16:23:46.330100+0000 osd.2 (osd.2) 64 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:23:48.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:48 smithi169 bash[19604]: cluster 2024-02-12T16:23:46.349267+0000 osd.2 (osd.2) 65 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:23:48.320 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:48.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:48 smithi123 bash[19119]: cluster 2024-02-12T16:23:46.330100+0000 osd.2 (osd.2) 64 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:23:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:48 smithi123 bash[19119]: cluster 2024-02-12T16:23:46.349267+0000 osd.2 (osd.2) 65 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:23:48.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:48 smithi121 bash[18489]: cluster 2024-02-12T16:23:46.330100+0000 osd.2 (osd.2) 64 : cluster [DBG] 4.4 scrub starts 2024-02-12T16:23:48.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:48 smithi121 bash[18489]: cluster 2024-02-12T16:23:46.349267+0000 osd.2 (osd.2) 65 : cluster [DBG] 4.4 scrub ok 2024-02-12T16:23:48.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:48.459 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:48.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:48.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:48.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:48.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:48.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:48.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:48.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:48.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:48.885 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:48.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:48.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:49.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:49.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:49.131 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:49.166 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:49.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:49.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:49.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:49.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:49 smithi123 bash[19119]: cluster 2024-02-12T16:23:47.915586+0000 mgr.y (mgr.14146) 1162 : cluster [DBG] pgmap v1594: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:49.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:49 smithi121 bash[18489]: cluster 2024-02-12T16:23:47.915586+0000 mgr.y (mgr.14146) 1162 : cluster [DBG] pgmap v1594: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:49.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:49 smithi169 bash[19604]: cluster 2024-02-12T16:23:47.915586+0000 mgr.y (mgr.14146) 1162 : cluster [DBG] pgmap v1594: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:49.445 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:49.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:49.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:49.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:49.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:49.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:49.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:49.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:49.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:49.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:49.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:49.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:50.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:50.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:50.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:50.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:50.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:50 smithi121 bash[18489]: cluster 2024-02-12T16:23:49.266405+0000 osd.7 (osd.7) 124 : cluster [DBG] 1.d scrub starts 2024-02-12T16:23:50.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:50 smithi121 bash[18489]: cluster 2024-02-12T16:23:49.328304+0000 osd.7 (osd.7) 125 : cluster [DBG] 1.d scrub ok 2024-02-12T16:23:50.289 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:50 smithi123 bash[19119]: cluster 2024-02-12T16:23:49.266405+0000 osd.7 (osd.7) 124 : cluster [DBG] 1.d scrub starts 2024-02-12T16:23:50.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:50 smithi123 bash[19119]: cluster 2024-02-12T16:23:49.328304+0000 osd.7 (osd.7) 125 : cluster [DBG] 1.d scrub ok 2024-02-12T16:23:50.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:50.405 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:50 smithi169 bash[19604]: cluster 2024-02-12T16:23:49.266405+0000 osd.7 (osd.7) 124 : cluster [DBG] 1.d scrub starts 2024-02-12T16:23:50.406 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:50 smithi169 bash[19604]: cluster 2024-02-12T16:23:49.328304+0000 osd.7 (osd.7) 125 : cluster [DBG] 1.d scrub ok 2024-02-12T16:23:50.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:50.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:50.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:50.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:50.713 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:50.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:50.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:50.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:50.994 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:50.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:51.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:51.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:51.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:51.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:51.272 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:51.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:51.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:51 smithi121 bash[18489]: cluster 2024-02-12T16:23:49.916356+0000 mgr.y (mgr.14146) 1163 : cluster [DBG] pgmap v1595: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:51.376 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:51.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:51 smithi169 bash[19604]: cluster 2024-02-12T16:23:49.916356+0000 mgr.y (mgr.14146) 1163 : cluster [DBG] pgmap v1595: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:51 smithi123 bash[19119]: cluster 2024-02-12T16:23:49.916356+0000 mgr.y (mgr.14146) 1163 : cluster [DBG] pgmap v1595: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:51.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:51.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:51.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:51.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:51.592 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:23:51.618 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:23:51.643 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:23:51.648 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:23:51.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:23:51.648 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:23:51.648 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:23:51.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:23:51.695 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:23:51.696 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:51.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:51.720 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:23:51.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:23:51.772 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:23:51.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:51.800 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:23:51.832 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:23:51.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:51.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:51.856 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:23:51.880 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:23:51.882 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.882 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.882 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:23:51.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:51.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:51.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:51.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:52.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:52.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:52.123 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:23:52.123 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T16:23:52.124 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_2' removed 2024-02-12T16:23:52.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:52.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:52.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:52.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:52 smithi121 bash[18489]: audit 2024-02-12T16:23:51.234352+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? v1:172.21.15.121:0/2689586471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:23:52.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:52 smithi121 bash[18489]: audit 2024-02-12T16:23:51.249322+0000 mon.a (mon.0) 1913 : audit [INF] from='client.25934 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:23:52.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:52 smithi121 bash[18489]: audit 2024-02-12T16:23:51.250694+0000 mon.b (mon.2) 136 : audit [INF] from='client.? v1:172.21.15.121:0/3111968931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:23:52.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:52 smithi121 bash[18489]: audit 2024-02-12T16:23:51.447302+0000 mon.a (mon.0) 1914 : audit [INF] from='client.? v1:172.21.15.121:0/3304782990' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:23:52.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:52 smithi121 bash[18489]: audit 2024-02-12T16:23:51.562670+0000 mon.a (mon.0) 1915 : audit [DBG] from='client.? v1:172.21.15.121:0/2883271321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:52.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:52 smithi121 bash[18489]: cluster 2024-02-12T16:23:51.606356+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub starts 2024-02-12T16:23:52.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:52 smithi121 bash[18489]: cluster 2024-02-12T16:23:51.623198+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.d scrub ok 2024-02-12T16:23:52.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:52.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:52.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:52 smithi123 bash[19119]: audit 2024-02-12T16:23:51.234352+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? v1:172.21.15.121:0/2689586471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:23:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:52 smithi123 bash[19119]: audit 2024-02-12T16:23:51.249322+0000 mon.a (mon.0) 1913 : audit [INF] from='client.25934 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:23:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:52 smithi123 bash[19119]: audit 2024-02-12T16:23:51.250694+0000 mon.b (mon.2) 136 : audit [INF] from='client.? v1:172.21.15.121:0/3111968931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:23:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:52 smithi123 bash[19119]: audit 2024-02-12T16:23:51.447302+0000 mon.a (mon.0) 1914 : audit [INF] from='client.? v1:172.21.15.121:0/3304782990' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:23:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:52 smithi123 bash[19119]: audit 2024-02-12T16:23:51.562670+0000 mon.a (mon.0) 1915 : audit [DBG] from='client.? v1:172.21.15.121:0/2883271321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:52 smithi123 bash[19119]: cluster 2024-02-12T16:23:51.606356+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub starts 2024-02-12T16:23:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:52 smithi123 bash[19119]: cluster 2024-02-12T16:23:51.623198+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.d scrub ok 2024-02-12T16:23:52.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:52.520 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:52.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:52.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:52 smithi169 bash[19604]: audit 2024-02-12T16:23:51.234352+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? v1:172.21.15.121:0/2689586471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:23:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:52 smithi169 bash[19604]: audit 2024-02-12T16:23:51.249322+0000 mon.a (mon.0) 1913 : audit [INF] from='client.25934 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:23:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:52 smithi169 bash[19604]: audit 2024-02-12T16:23:51.250694+0000 mon.b (mon.2) 136 : audit [INF] from='client.? v1:172.21.15.121:0/3111968931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:23:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:52 smithi169 bash[19604]: audit 2024-02-12T16:23:51.447302+0000 mon.a (mon.0) 1914 : audit [INF] from='client.? v1:172.21.15.121:0/3304782990' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:23:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:52 smithi169 bash[19604]: audit 2024-02-12T16:23:51.562670+0000 mon.a (mon.0) 1915 : audit [DBG] from='client.? v1:172.21.15.121:0/2883271321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:23:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:52 smithi169 bash[19604]: cluster 2024-02-12T16:23:51.606356+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d scrub starts 2024-02-12T16:23:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:52 smithi169 bash[19604]: cluster 2024-02-12T16:23:51.623198+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.d scrub ok 2024-02-12T16:23:52.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:52.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:23:52.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:52.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:52.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.797 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:52.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:52.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:52.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:53.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:53.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:53.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:53.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:53.194 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: cluster 2024-02-12T16:23:51.917190+0000 mgr.y (mgr.14146) 1164 : cluster [DBG] pgmap v1596: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:53.195 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: cluster 2024-02-12T16:23:52.095979+0000 mon.a (mon.0) 1916 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:23:53.195 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: audit 2024-02-12T16:23:52.110266+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? v1:172.21.15.121:0/2689586471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:23:53.195 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: audit 2024-02-12T16:23:52.110393+0000 mon.a (mon.0) 1918 : audit [INF] from='client.25934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:23:53.195 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: audit 2024-02-12T16:23:52.110521+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? v1:172.21.15.121:0/3304782990' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:23:53.195 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: cluster 2024-02-12T16:23:52.110586+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e540: 12 total, 12 up, 12 in 2024-02-12T16:23:53.195 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: cluster 2024-02-12T16:23:52.928889+0000 osd.2 (osd.2) 66 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:53.195 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:53 smithi121 bash[18489]: cluster 2024-02-12T16:23:52.928984+0000 osd.2 (osd.2) 67 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:53.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:53.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:53.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:53.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:53.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: cluster 2024-02-12T16:23:51.917190+0000 mgr.y (mgr.14146) 1164 : cluster [DBG] pgmap v1596: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: cluster 2024-02-12T16:23:52.095979+0000 mon.a (mon.0) 1916 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: audit 2024-02-12T16:23:52.110266+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? v1:172.21.15.121:0/2689586471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: audit 2024-02-12T16:23:52.110393+0000 mon.a (mon.0) 1918 : audit [INF] from='client.25934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: audit 2024-02-12T16:23:52.110521+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? v1:172.21.15.121:0/3304782990' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: cluster 2024-02-12T16:23:52.110586+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e540: 12 total, 12 up, 12 in 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: cluster 2024-02-12T16:23:52.928889+0000 osd.2 (osd.2) 66 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:53 smithi123 bash[19119]: cluster 2024-02-12T16:23:52.928984+0000 osd.2 (osd.2) 67 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:53.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:53.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:23:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:53.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: cluster 2024-02-12T16:23:51.917190+0000 mgr.y (mgr.14146) 1164 : cluster [DBG] pgmap v1596: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: cluster 2024-02-12T16:23:52.095979+0000 mon.a (mon.0) 1916 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: audit 2024-02-12T16:23:52.110266+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? v1:172.21.15.121:0/2689586471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: audit 2024-02-12T16:23:52.110393+0000 mon.a (mon.0) 1918 : audit [INF] from='client.25934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: audit 2024-02-12T16:23:52.110521+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? v1:172.21.15.121:0/3304782990' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: cluster 2024-02-12T16:23:52.110586+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e540: 12 total, 12 up, 12 in 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: cluster 2024-02-12T16:23:52.928889+0000 osd.2 (osd.2) 66 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:53 smithi169 bash[19604]: cluster 2024-02-12T16:23:52.928984+0000 osd.2 (osd.2) 67 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:53.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:53.642 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:53.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:53.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:53.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T16:23:53.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:53.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:53.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:53.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:53.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:54.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:54.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:54.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:54.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:54.178 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:54 smithi169 bash[19604]: cluster 2024-02-12T16:23:53.118775+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e541: 12 total, 12 up, 12 in 2024-02-12T16:23:54.204 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:54.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:54.277 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:23:54.278 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:23:54.278 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-02-12T16:23:54.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-02-12T16:23:54.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:54.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:54.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:54.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:54 smithi123 bash[19119]: cluster 2024-02-12T16:23:53.118775+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e541: 12 total, 12 up, 12 in 2024-02-12T16:23:54.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:54 smithi121 bash[18489]: cluster 2024-02-12T16:23:53.118775+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e541: 12 total, 12 up, 12 in 2024-02-12T16:23:54.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:54.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:54.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:54.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:54.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:54.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:54.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:54.770 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:54.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:54.875 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:54.913 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:54.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:55.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:55.054 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:55.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:55.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:55.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:55.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:55.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:55.307 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:55 smithi123 bash[19119]: cluster 2024-02-12T16:23:53.917819+0000 mgr.y (mgr.14146) 1165 : cluster [DBG] pgmap v1599: 21 pgs: 1 peering, 20 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:55.307 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:55 smithi123 bash[19119]: cluster 2024-02-12T16:23:54.113969+0000 mon.a (mon.0) 1922 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:23:55.332 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:55.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:55 smithi121 bash[18489]: cluster 2024-02-12T16:23:53.917819+0000 mgr.y (mgr.14146) 1165 : cluster [DBG] pgmap v1599: 21 pgs: 1 peering, 20 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:55 smithi121 bash[18489]: cluster 2024-02-12T16:23:54.113969+0000 mon.a (mon.0) 1922 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:23:55.436 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:55.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:55.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:55.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:55.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:55 smithi169 bash[19604]: cluster 2024-02-12T16:23:53.917819+0000 mgr.y (mgr.14146) 1165 : cluster [DBG] pgmap v1599: 21 pgs: 1 peering, 20 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:55 smithi169 bash[19604]: cluster 2024-02-12T16:23:54.113969+0000 mon.a (mon.0) 1922 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:23:55.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:55.614 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:55.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:55.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:55.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:23:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:23:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:55.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:23:55.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:55.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:55.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:56.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:56.044 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:56.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:56.148 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:56.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:23:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:56.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:23:56.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:56.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:23:56.329 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:56.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:56.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:56.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:56.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:56.569 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:56.608 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:56.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:56.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:56.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:56.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:56.860 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:56.872 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:56 smithi123 bash[19119]: cluster 2024-02-12T16:23:55.918375+0000 mgr.y (mgr.14146) 1166 : cluster [DBG] pgmap v1600: 21 pgs: 1 peering, 20 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:56.900 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:56.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:56.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:56 smithi121 bash[18489]: cluster 2024-02-12T16:23:55.918375+0000 mgr.y (mgr.14146) 1166 : cluster [DBG] pgmap v1600: 21 pgs: 1 peering, 20 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:57.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:57.041 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:57.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:56 smithi169 bash[19604]: cluster 2024-02-12T16:23:55.918375+0000 mgr.y (mgr.14146) 1166 : cluster [DBG] pgmap v1600: 21 pgs: 1 peering, 20 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:57.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:57.184 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:23:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:57.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:23:57.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:57.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:57.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:57.423 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:57.425 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:57.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:23:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:57.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:23:57.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:57.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:57.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:57.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:57.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:57.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:57.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:57.879 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:57.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:57.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:58.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:58.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:58.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:58.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:58.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:58.296 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:58.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:58.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:58.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:58.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:23:58.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:58.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:58.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:58.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:58.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:23:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:58.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:23:58.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:59.039 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:58 smithi121 bash[18489]: cluster 2024-02-12T16:23:57.796995+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:59.039 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:58 smithi121 bash[18489]: cluster 2024-02-12T16:23:57.797087+0000 osd.4 (osd.4) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:59.039 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:58 smithi121 bash[18489]: cluster 2024-02-12T16:23:57.855864+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e542: 12 total, 12 up, 12 in 2024-02-12T16:23:59.039 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:23:58 smithi121 bash[18489]: cluster 2024-02-12T16:23:57.919441+0000 mgr.y (mgr.14146) 1167 : cluster [DBG] pgmap v1602: 21 pgs: 21 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:59.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:59.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:23:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:58 smithi123 bash[19119]: cluster 2024-02-12T16:23:57.796995+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:58 smithi123 bash[19119]: cluster 2024-02-12T16:23:57.797087+0000 osd.4 (osd.4) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:58 smithi123 bash[19119]: cluster 2024-02-12T16:23:57.855864+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e542: 12 total, 12 up, 12 in 2024-02-12T16:23:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:23:58 smithi123 bash[19119]: cluster 2024-02-12T16:23:57.919441+0000 mgr.y (mgr.14146) 1167 : cluster [DBG] pgmap v1602: 21 pgs: 21 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:59.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:59.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:58 smithi169 bash[19604]: cluster 2024-02-12T16:23:57.796995+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:23:59.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:58 smithi169 bash[19604]: cluster 2024-02-12T16:23:57.797087+0000 osd.4 (osd.4) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:23:59.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:58 smithi169 bash[19604]: cluster 2024-02-12T16:23:57.855864+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e542: 12 total, 12 up, 12 in 2024-02-12T16:23:59.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:23:58 smithi169 bash[19604]: cluster 2024-02-12T16:23:57.919441+0000 mgr.y (mgr.14146) 1167 : cluster [DBG] pgmap v1602: 21 pgs: 21 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:23:59.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:23:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:59.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:23:59.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:59.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:23:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:59.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:23:59.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:59.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:23:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:59.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:23:59.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:59.639 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:23:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:59.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:23:59.743 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:59.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:23:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:59.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:23:59.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:59.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:23:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:23:59.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:00.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:00.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:00.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:00.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:00.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:00.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:00.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:00.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:00.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:00.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:00.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:00.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:00.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:00.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:00.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:00.759 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:00.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:00.807 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:00.810 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:00.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:00.894 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:00.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:00.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:01.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:01.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:01.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:00 smithi169 bash[19604]: cluster 2024-02-12T16:23:59.920385+0000 mgr.y (mgr.14146) 1168 : cluster [DBG] pgmap v1603: 21 pgs: 21 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:01.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:00 smithi169 bash[19604]: cluster 2024-02-12T16:24:00.599599+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e543: 12 total, 12 up, 12 in 2024-02-12T16:24:01.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:01.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:00 smithi121 bash[18489]: cluster 2024-02-12T16:23:59.920385+0000 mgr.y (mgr.14146) 1168 : cluster [DBG] pgmap v1603: 21 pgs: 21 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:01.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:00 smithi121 bash[18489]: cluster 2024-02-12T16:24:00.599599+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e543: 12 total, 12 up, 12 in 2024-02-12T16:24:01.174 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:01.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:01.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:01.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:00 smithi123 bash[19119]: cluster 2024-02-12T16:23:59.920385+0000 mgr.y (mgr.14146) 1168 : cluster [DBG] pgmap v1603: 21 pgs: 21 active+clean; 0 B data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:00 smithi123 bash[19119]: cluster 2024-02-12T16:24:00.599599+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e543: 12 total, 12 up, 12 in 2024-02-12T16:24:01.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:01.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:01.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:01.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:01.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:01.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:01.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:01.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:01.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:01.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:01.868 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:01.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:01.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:02.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:02.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:02.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:02.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:02.290 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:02.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:02.428 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:02.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:02.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:02.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:02.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:02.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:02.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:02.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:02.850 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:02.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:02.991 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:02.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:03.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:03.108 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:02 smithi121 bash[18489]: cluster 2024-02-12T16:24:01.921113+0000 mgr.y (mgr.14146) 1169 : cluster [DBG] pgmap v1605: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:03.108 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:02 smithi121 bash[18489]: cluster 2024-02-12T16:24:02.305619+0000 osd.10 (osd.10) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:03.108 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:02 smithi121 bash[18489]: cluster 2024-02-12T16:24:02.305702+0000 osd.10 (osd.10) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:03.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:03.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:03.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:03.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:02 smithi169 bash[19604]: cluster 2024-02-12T16:24:01.921113+0000 mgr.y (mgr.14146) 1169 : cluster [DBG] pgmap v1605: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:03.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:02 smithi169 bash[19604]: cluster 2024-02-12T16:24:02.305619+0000 osd.10 (osd.10) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:03.252 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:02 smithi169 bash[19604]: cluster 2024-02-12T16:24:02.305702+0000 osd.10 (osd.10) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:03.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:03.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:03.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:02 smithi123 bash[19119]: cluster 2024-02-12T16:24:01.921113+0000 mgr.y (mgr.14146) 1169 : cluster [DBG] pgmap v1605: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:02 smithi123 bash[19119]: cluster 2024-02-12T16:24:02.305619+0000 osd.10 (osd.10) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:02 smithi123 bash[19119]: cluster 2024-02-12T16:24:02.305702+0000 osd.10 (osd.10) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:03.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:03.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:03.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:03.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:03.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:03.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:03.695 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:03.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:03.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:03.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:03.840 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:03.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:03.982 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:03.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:04.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:04.128 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:04.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:04.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:04.270 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:04 smithi121 bash[18489]: cluster 2024-02-12T16:24:02.988924+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e544: 12 total, 12 up, 12 in 2024-02-12T16:24:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:04.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:04.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:04 smithi169 bash[19604]: cluster 2024-02-12T16:24:02.988924+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e544: 12 total, 12 up, 12 in 2024-02-12T16:24:04.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:04.386 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:04 smithi123 bash[19119]: cluster 2024-02-12T16:24:02.988924+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e544: 12 total, 12 up, 12 in 2024-02-12T16:24:04.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:04.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:04.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:04.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:04.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:04.568 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:24:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:24:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:24:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:24:04.568 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:24:04.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:04.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:04.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:04.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:04.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:04.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:04.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:04.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:04.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:05.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:05.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:05.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:05.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:05 smithi121 bash[18489]: cluster 2024-02-12T16:24:03.921779+0000 mgr.y (mgr.14146) 1170 : cluster [DBG] pgmap v1607: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:05.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:05 smithi121 bash[18489]: audit 2024-02-12T16:24:04.439982+0000 mon.a (mon.0) 1926 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:05.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:05.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:05.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:05 smithi169 bash[19604]: cluster 2024-02-12T16:24:03.921779+0000 mgr.y (mgr.14146) 1170 : cluster [DBG] pgmap v1607: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:05 smithi169 bash[19604]: audit 2024-02-12T16:24:04.439982+0000 mon.a (mon.0) 1926 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:05.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:05 smithi123 bash[19119]: cluster 2024-02-12T16:24:03.921779+0000 mgr.y (mgr.14146) 1170 : cluster [DBG] pgmap v1607: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:05.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:05 smithi123 bash[19119]: audit 2024-02-12T16:24:04.439982+0000 mon.a (mon.0) 1926 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:05.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:05.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:05.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:05.551 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:05.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:05.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:05.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:05.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:05.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:05.830 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:05.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:05.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:05.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:06.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:06.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:06.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:06.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:06.256 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:06.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:06.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:24:06.359 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:06.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:06.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:06.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:06.536 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:06.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:06.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:06.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:06.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:06.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:06.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:06.820 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:06.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:06 smithi123 bash[19119]: cluster 2024-02-12T16:24:05.603773+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e545: 12 total, 12 up, 12 in 2024-02-12T16:24:06.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:06 smithi123 bash[19119]: cluster 2024-02-12T16:24:05.922229+0000 mgr.y (mgr.14146) 1171 : cluster [DBG] pgmap v1609: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:06.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:06 smithi121 bash[18489]: cluster 2024-02-12T16:24:05.603773+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e545: 12 total, 12 up, 12 in 2024-02-12T16:24:06.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:06 smithi121 bash[18489]: cluster 2024-02-12T16:24:05.922229+0000 mgr.y (mgr.14146) 1171 : cluster [DBG] pgmap v1609: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:06.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:06.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:06 smithi169 bash[19604]: cluster 2024-02-12T16:24:05.603773+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e545: 12 total, 12 up, 12 in 2024-02-12T16:24:06.935 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:06 smithi169 bash[19604]: cluster 2024-02-12T16:24:05.922229+0000 mgr.y (mgr.14146) 1171 : cluster [DBG] pgmap v1609: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:06.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:06.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:07.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:07.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:07.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:07.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:07.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:07.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:07.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:07.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:07.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:07.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:07.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:07.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:07.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:07.634 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T16:24:07.657 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:07.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:07.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:07.798 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:07.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:07 smithi123 bash[19119]: cluster 2024-02-12T16:24:06.741840+0000 osd.11 (osd.11) 122 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:07.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:07 smithi123 bash[19119]: cluster 2024-02-12T16:24:06.741950+0000 osd.11 (osd.11) 123 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:07.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:07 smithi123 bash[19119]: audit 2024-02-12T16:24:07.046541+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.121:0/3980428745' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:24:07.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:07 smithi121 bash[18489]: cluster 2024-02-12T16:24:06.741840+0000 osd.11 (osd.11) 122 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:07 smithi121 bash[18489]: cluster 2024-02-12T16:24:06.741950+0000 osd.11 (osd.11) 123 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:07 smithi121 bash[18489]: audit 2024-02-12T16:24:07.046541+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.121:0/3980428745' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:24:07.939 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:07.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:08.043 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:07 smithi169 bash[19604]: cluster 2024-02-12T16:24:06.741840+0000 osd.11 (osd.11) 122 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:07 smithi169 bash[19604]: cluster 2024-02-12T16:24:06.741950+0000 osd.11 (osd.11) 123 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:07 smithi169 bash[19604]: audit 2024-02-12T16:24:07.046541+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.121:0/3980428745' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:24:08.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:08.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:08.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:08.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:08.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:08.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:08.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:08.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:08.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:08.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:08.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:08.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:08.640 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_3' created 2024-02-12T16:24:08.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:08.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:08.744 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:08.756 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:08 smithi123 bash[19119]: audit 2024-02-12T16:24:07.625165+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? v1:172.21.15.121:0/3980428745' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:24:08.756 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:08 smithi123 bash[19119]: cluster 2024-02-12T16:24:07.625273+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e546: 12 total, 12 up, 12 in 2024-02-12T16:24:08.756 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:08 smithi123 bash[19119]: cluster 2024-02-12T16:24:07.922654+0000 mgr.y (mgr.14146) 1172 : cluster [DBG] pgmap v1611: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:08.756 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:08 smithi123 bash[19119]: audit 2024-02-12T16:24:08.041542+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? v1:172.21.15.121:0/278290782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-12T16:24:08.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:08.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:08.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:08.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:08 smithi121 bash[18489]: audit 2024-02-12T16:24:07.625165+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? v1:172.21.15.121:0/3980428745' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:24:08.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:08 smithi121 bash[18489]: cluster 2024-02-12T16:24:07.625273+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e546: 12 total, 12 up, 12 in 2024-02-12T16:24:08.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:08 smithi121 bash[18489]: cluster 2024-02-12T16:24:07.922654+0000 mgr.y (mgr.14146) 1172 : cluster [DBG] pgmap v1611: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:08.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:08 smithi121 bash[18489]: audit 2024-02-12T16:24:08.041542+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? v1:172.21.15.121:0/278290782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-12T16:24:08.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:08.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:09.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:09.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:08 smithi169 bash[19604]: audit 2024-02-12T16:24:07.625165+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? v1:172.21.15.121:0/3980428745' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:24:09.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:08 smithi169 bash[19604]: cluster 2024-02-12T16:24:07.625273+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e546: 12 total, 12 up, 12 in 2024-02-12T16:24:09.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:08 smithi169 bash[19604]: cluster 2024-02-12T16:24:07.922654+0000 mgr.y (mgr.14146) 1172 : cluster [DBG] pgmap v1611: 21 pgs: 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:09.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:08 smithi169 bash[19604]: audit 2024-02-12T16:24:08.041542+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? v1:172.21.15.121:0/278290782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-12T16:24:09.053 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:09.056 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:09.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:09.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:09.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:09.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:09.327 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:09.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:09.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:09.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:09.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-12T16:24:09.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:09.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:09.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:09.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:09 smithi123 bash[19119]: audit 2024-02-12T16:24:08.632659+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? v1:172.21.15.121:0/278290782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-12T16:24:09.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:09 smithi123 bash[19119]: cluster 2024-02-12T16:24:08.632716+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e547: 12 total, 12 up, 12 in 2024-02-12T16:24:09.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:09.752 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:09.756 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:09.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:09.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:09 smithi121 bash[18489]: audit 2024-02-12T16:24:08.632659+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? v1:172.21.15.121:0/278290782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-12T16:24:09.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:09 smithi121 bash[18489]: cluster 2024-02-12T16:24:08.632716+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e547: 12 total, 12 up, 12 in 2024-02-12T16:24:09.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:09.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:09.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-02-12T16:24:10.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:10.040 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:10.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:10.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:09 smithi169 bash[19604]: audit 2024-02-12T16:24:08.632659+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? v1:172.21.15.121:0/278290782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-12T16:24:10.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:09 smithi169 bash[19604]: cluster 2024-02-12T16:24:08.632716+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e547: 12 total, 12 up, 12 in 2024-02-12T16:24:10.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:10.184 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:10.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:10.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:10.328 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:10.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:10.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:10.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:10.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:24:10.455 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:24:10.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:10.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:10.478 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:24:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:24:10.523 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:24:10.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:24:10.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:10.581 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:24:10.605 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:24:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:10.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:10.651 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:24:10.682 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:24:10.705 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:24:10.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:10.732 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:24:10.734 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.735 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:24:10.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:10.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:10.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:10 smithi169 bash[19604]: cluster 2024-02-12T16:24:09.642525+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e548: 12 total, 12 up, 12 in 2024-02-12T16:24:10.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:10 smithi169 bash[19604]: cluster 2024-02-12T16:24:09.923171+0000 mgr.y (mgr.14146) 1173 : cluster [DBG] pgmap v1614: 37 pgs: 1 creating+peering, 15 unknown, 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:10.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:10 smithi169 bash[19604]: audit 2024-02-12T16:24:10.412276+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.121:0/3972276988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:10.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:10 smithi169 bash[19604]: cluster 2024-02-12T16:24:10.605742+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e549: 12 total, 12 up, 12 in 2024-02-12T16:24:10.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:10.884 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:10.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:10.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:11.000 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:10 smithi121 bash[18489]: cluster 2024-02-12T16:24:09.642525+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e548: 12 total, 12 up, 12 in 2024-02-12T16:24:11.000 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:10 smithi121 bash[18489]: cluster 2024-02-12T16:24:09.923171+0000 mgr.y (mgr.14146) 1173 : cluster [DBG] pgmap v1614: 37 pgs: 1 creating+peering, 15 unknown, 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:11.000 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:10 smithi121 bash[18489]: audit 2024-02-12T16:24:10.412276+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.121:0/3972276988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:11.000 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:10 smithi121 bash[18489]: cluster 2024-02-12T16:24:10.605742+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e549: 12 total, 12 up, 12 in 2024-02-12T16:24:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:11.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:11.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:10 smithi123 bash[19119]: cluster 2024-02-12T16:24:09.642525+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e548: 12 total, 12 up, 12 in 2024-02-12T16:24:11.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:10 smithi123 bash[19119]: cluster 2024-02-12T16:24:09.923171+0000 mgr.y (mgr.14146) 1173 : cluster [DBG] pgmap v1614: 37 pgs: 1 creating+peering, 15 unknown, 21 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:11.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:10 smithi123 bash[19119]: audit 2024-02-12T16:24:10.412276+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.121:0/3972276988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:10 smithi123 bash[19119]: cluster 2024-02-12T16:24:10.605742+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e549: 12 total, 12 up, 12 in 2024-02-12T16:24:11.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:11.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:11.298 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:11.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:11.441 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:11.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:11.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:11.626 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:24:11.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:11.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:11.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:11.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:11 smithi121 bash[18489]: audit 2024-02-12T16:24:10.652840+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? v1:172.21.15.121:0/740331370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:11.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:11 smithi121 bash[18489]: cluster 2024-02-12T16:24:11.112916+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:11.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:11 smithi121 bash[18489]: cluster 2024-02-12T16:24:11.113006+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:11.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:11 smithi121 bash[18489]: cluster 2024-02-12T16:24:11.608088+0000 mon.a (mon.0) 1937 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:24:11.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:11 smithi121 bash[18489]: audit 2024-02-12T16:24:11.616414+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.121:0/740331370' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:24:11.827 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:11 smithi121 bash[18489]: cluster 2024-02-12T16:24:11.616538+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e550: 12 total, 12 up, 12 in 2024-02-12T16:24:11.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:11.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:11.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:11.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:11 smithi169 bash[19604]: audit 2024-02-12T16:24:10.652840+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? v1:172.21.15.121:0/740331370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:11 smithi169 bash[19604]: cluster 2024-02-12T16:24:11.112916+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:11 smithi169 bash[19604]: cluster 2024-02-12T16:24:11.113006+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:11 smithi169 bash[19604]: cluster 2024-02-12T16:24:11.608088+0000 mon.a (mon.0) 1937 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:24:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:11 smithi169 bash[19604]: audit 2024-02-12T16:24:11.616414+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.121:0/740331370' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:24:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:11 smithi169 bash[19604]: cluster 2024-02-12T16:24:11.616538+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e550: 12 total, 12 up, 12 in 2024-02-12T16:24:12.067 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:12.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:12.107 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:11 smithi123 bash[19119]: audit 2024-02-12T16:24:10.652840+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? v1:172.21.15.121:0/740331370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:12.107 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:11 smithi123 bash[19119]: cluster 2024-02-12T16:24:11.112916+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:12.107 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:11 smithi123 bash[19119]: cluster 2024-02-12T16:24:11.113006+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:12.107 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:11 smithi123 bash[19119]: cluster 2024-02-12T16:24:11.608088+0000 mon.a (mon.0) 1937 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:24:12.107 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:11 smithi123 bash[19119]: audit 2024-02-12T16:24:11.616414+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? v1:172.21.15.121:0/740331370' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:24:12.107 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:11 smithi123 bash[19119]: cluster 2024-02-12T16:24:11.616538+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e550: 12 total, 12 up, 12 in 2024-02-12T16:24:12.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:12.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:12.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:12.269 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:12.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:12.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:12.410 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:12.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:12.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:12.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:12.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:12.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:12.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:12.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:12.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:12.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:12 smithi123 bash[19119]: cluster 2024-02-12T16:24:11.923777+0000 mgr.y (mgr.14146) 1174 : cluster [DBG] pgmap v1617: 37 pgs: 1 creating+peering, 6 unknown, 30 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:12.836 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:12.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:12.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:12.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:12 smithi169 bash[19604]: cluster 2024-02-12T16:24:11.923777+0000 mgr.y (mgr.14146) 1174 : cluster [DBG] pgmap v1617: 37 pgs: 1 creating+peering, 6 unknown, 30 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:12.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:12.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:12.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:13.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:13.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:13.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:13.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:12 smithi121 bash[18489]: cluster 2024-02-12T16:24:11.923777+0000 mgr.y (mgr.14146) 1174 : cluster [DBG] pgmap v1617: 37 pgs: 1 creating+peering, 6 unknown, 30 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:13.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:13.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:13.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:13.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:13.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:13.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:13.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:13.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:24:13.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:13.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:13.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:13.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:13.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:13.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:13 smithi121 bash[18489]: cluster 2024-02-12T16:24:12.354860+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:13 smithi121 bash[18489]: cluster 2024-02-12T16:24:12.354939+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:13 smithi121 bash[18489]: cluster 2024-02-12T16:24:12.696610+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e551: 12 total, 12 up, 12 in 2024-02-12T16:24:13.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:13.952 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:13.955 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:13 smithi169 bash[19604]: cluster 2024-02-12T16:24:12.354860+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:13.955 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:13 smithi169 bash[19604]: cluster 2024-02-12T16:24:12.354939+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:13.955 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:13 smithi169 bash[19604]: cluster 2024-02-12T16:24:12.696610+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e551: 12 total, 12 up, 12 in 2024-02-12T16:24:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:13.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:14.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:14.069 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:13 smithi123 bash[19119]: cluster 2024-02-12T16:24:12.354860+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:14.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:13 smithi123 bash[19119]: cluster 2024-02-12T16:24:12.354939+0000 osd.0 (osd.0) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:14.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:13 smithi123 bash[19119]: cluster 2024-02-12T16:24:12.696610+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e551: 12 total, 12 up, 12 in 2024-02-12T16:24:14.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:14.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:14.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:14.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:14.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:14.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:14.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:14.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:14.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:14.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:14.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:14.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:14.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:14.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:14.709 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:14.709 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:14.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:24:14.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:14.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:14 smithi169 bash[19604]: cluster 2024-02-12T16:24:13.924571+0000 mgr.y (mgr.14146) 1175 : cluster [DBG] pgmap v1619: 37 pgs: 1 creating+peering, 36 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:14.798 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:14.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:14.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:14.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:14 smithi121 bash[18489]: cluster 2024-02-12T16:24:13.924571+0000 mgr.y (mgr.14146) 1175 : cluster [DBG] pgmap v1619: 37 pgs: 1 creating+peering, 36 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:14.937 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:14.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:15.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:15.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:15.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:14 smithi123 bash[19119]: cluster 2024-02-12T16:24:13.924571+0000 mgr.y (mgr.14146) 1175 : cluster [DBG] pgmap v1619: 37 pgs: 1 creating+peering, 36 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:15.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:15.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:15.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:15.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:15.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:15.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:15.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:15.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:15.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:15.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:15.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.784 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:15.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:15.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:15.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:15.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:16.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:16.063 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:16.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:16.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:16.205 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:16.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:16.310 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:16.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:16.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:16.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:16.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:16.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:16.591 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:24:16.591 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:24:16.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:16.636 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:16.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:16.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:16.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:16 smithi169 bash[19604]: cluster 2024-02-12T16:24:15.925379+0000 mgr.y (mgr.14146) 1176 : cluster [DBG] pgmap v1620: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:16.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:16.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:16.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:16.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:16 smithi123 bash[19119]: cluster 2024-02-12T16:24:15.925379+0000 mgr.y (mgr.14146) 1176 : cluster [DBG] pgmap v1620: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:16.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:16 smithi121 bash[18489]: cluster 2024-02-12T16:24:15.925379+0000 mgr.y (mgr.14146) 1176 : cluster [DBG] pgmap v1620: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:16.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:16.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:17.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:17.064 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:17.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:17.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:17.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:17.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:17.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:17.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:17.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:17.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:17.481 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:17.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:17.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:24:17.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:17.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:17.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:17.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:17.761 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:17.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:17.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:17.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:17.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:18.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:18.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:18.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:18.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:18.192 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:18.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:18.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:18.333 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:18.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:18.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:18.473 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:18.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:18.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:18.609 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:18.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:18.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:18.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:18.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:18.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:18.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:18.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:18.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:19.008 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:19.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:19.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:19.150 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:18 smithi169 bash[19604]: cluster 2024-02-12T16:24:17.926149+0000 mgr.y (mgr.14146) 1177 : cluster [DBG] pgmap v1621: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:19.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:19.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:19.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:19.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:18 smithi123 bash[19119]: cluster 2024-02-12T16:24:17.926149+0000 mgr.y (mgr.14146) 1177 : cluster [DBG] pgmap v1621: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:19.319 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:19.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:19.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:18 smithi121 bash[18489]: cluster 2024-02-12T16:24:17.926149+0000 mgr.y (mgr.14146) 1177 : cluster [DBG] pgmap v1621: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:19.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:19.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:19.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:19.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:19.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:19.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:19.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:19.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:19.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:19.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:19.875 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:19.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:19.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:20.017 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:20.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:20.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:20.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:20.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:20.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:20.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:20.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:20.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:20.436 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:20.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:20.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:20.580 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:20.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:20.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:20.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:20.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:20.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:20.857 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:20.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:20.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:21.001 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:21.036 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:21.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:21.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:21.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:21.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:21.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:20 smithi169 bash[19604]: cluster 2024-02-12T16:24:19.926886+0000 mgr.y (mgr.14146) 1178 : cluster [DBG] pgmap v1622: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:21.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:20 smithi169 bash[19604]: audit 2024-02-12T16:24:20.814824+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.121:0/2054347530' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:24:21.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:20 smithi169 bash[19604]: audit 2024-02-12T16:24:20.884340+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.121:0/2140394761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:24:21.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:21.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:20 smithi123 bash[19119]: cluster 2024-02-12T16:24:19.926886+0000 mgr.y (mgr.14146) 1178 : cluster [DBG] pgmap v1622: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:20 smithi123 bash[19119]: audit 2024-02-12T16:24:20.814824+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.121:0/2054347530' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:24:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:20 smithi123 bash[19119]: audit 2024-02-12T16:24:20.884340+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.121:0/2140394761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:24:21.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:21.463 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:20 smithi121 bash[18489]: cluster 2024-02-12T16:24:19.926886+0000 mgr.y (mgr.14146) 1178 : cluster [DBG] pgmap v1622: 37 pgs: 37 active+clean; 0 B data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:21.463 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:20 smithi121 bash[18489]: audit 2024-02-12T16:24:20.814824+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? v1:172.21.15.121:0/2054347530' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:24:21.463 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:20 smithi121 bash[18489]: audit 2024-02-12T16:24:20.884340+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.121:0/2140394761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:24:21.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:21.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:21.625 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-02-12T16:24:21.625 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.3 to 1 (10000) 2024-02-12T16:24:21.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:21.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:21.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:21.837 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:21.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:21.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:21.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:22.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:22.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:22.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:22.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:22.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:22.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:22.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:22.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:22.530 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:22.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:22.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:22.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:22.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:22.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:22.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:22.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:22.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:22 smithi123 bash[19119]: audit 2024-02-12T16:24:21.614375+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.121:0/2054347530' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:24:22.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:22 smithi123 bash[19119]: audit 2024-02-12T16:24:21.614557+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? v1:172.21.15.121:0/2140394761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:24:22.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:22 smithi123 bash[19119]: cluster 2024-02-12T16:24:21.614717+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e552: 12 total, 12 up, 12 in 2024-02-12T16:24:22.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:22 smithi123 bash[19119]: cluster 2024-02-12T16:24:21.927552+0000 mgr.y (mgr.14146) 1179 : cluster [DBG] pgmap v1624: 37 pgs: 37 active+clean; 0 B data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:22 smithi121 bash[18489]: audit 2024-02-12T16:24:21.614375+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.121:0/2054347530' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:24:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:22 smithi121 bash[18489]: audit 2024-02-12T16:24:21.614557+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? v1:172.21.15.121:0/2140394761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:24:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:22 smithi121 bash[18489]: cluster 2024-02-12T16:24:21.614717+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e552: 12 total, 12 up, 12 in 2024-02-12T16:24:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:22 smithi121 bash[18489]: cluster 2024-02-12T16:24:21.927552+0000 mgr.y (mgr.14146) 1179 : cluster [DBG] pgmap v1624: 37 pgs: 37 active+clean; 0 B data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:22.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:22.956 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:22.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:22.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:22 smithi169 bash[19604]: audit 2024-02-12T16:24:21.614375+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? v1:172.21.15.121:0/2054347530' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:24:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:22 smithi169 bash[19604]: audit 2024-02-12T16:24:21.614557+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? v1:172.21.15.121:0/2140394761' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:24:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:22 smithi169 bash[19604]: cluster 2024-02-12T16:24:21.614717+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e552: 12 total, 12 up, 12 in 2024-02-12T16:24:23.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:22 smithi169 bash[19604]: cluster 2024-02-12T16:24:21.927552+0000 mgr.y (mgr.14146) 1179 : cluster [DBG] pgmap v1624: 37 pgs: 37 active+clean; 0 B data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:23.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:23.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:23.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:23.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:23.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:23.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:23.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:23.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:23.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:23.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:23.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:23.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:23.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:23.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:23.797 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:23.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:23.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T16:24:23.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:23.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:23.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:24.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:24.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:24.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:24.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:24.227 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:24.256 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:24.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:24.311 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:24:24.332 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:24:24.354 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:24:24.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:24.381 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:24:24.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:24.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:24.411 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:24:24.437 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:24:24.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:24:24.493 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:24:24.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:24.518 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:24:24.538 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:24:24.539 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:24.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:24.560 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:24:24.588 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:24:24.591 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:24:24.644 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:24.681 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:24.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:24.741 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-02-12T16:24:24.744 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:24:24.744 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:24:24.785 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:24:24.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:24:24.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_84712 2024-02-12T16:24:24.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:24:24.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:24:24.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:24.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:24.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:24.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:24.959 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:24.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:24.992 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:24 smithi123 bash[19119]: cluster 2024-02-12T16:24:23.928171+0000 mgr.y (mgr.14146) 1180 : cluster [DBG] pgmap v1625: 37 pgs: 37 active+clean; 0 B data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:24.992 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:24 smithi123 bash[19119]: audit 2024-02-12T16:24:24.280247+0000 mon.a (mon.0) 1946 : audit [DBG] from='client.? v1:172.21.15.121:0/3770210730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:25.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:25.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:25.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:25.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:25.206 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:24 smithi121 bash[18489]: cluster 2024-02-12T16:24:23.928171+0000 mgr.y (mgr.14146) 1180 : cluster [DBG] pgmap v1625: 37 pgs: 37 active+clean; 0 B data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:25.206 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:24 smithi121 bash[18489]: audit 2024-02-12T16:24:24.280247+0000 mon.a (mon.0) 1946 : audit [DBG] from='client.? v1:172.21.15.121:0/3770210730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:25.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:25.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:24 smithi169 bash[19604]: cluster 2024-02-12T16:24:23.928171+0000 mgr.y (mgr.14146) 1180 : cluster [DBG] pgmap v1625: 37 pgs: 37 active+clean; 0 B data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:25.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:24 smithi169 bash[19604]: audit 2024-02-12T16:24:24.280247+0000 mon.a (mon.0) 1946 : audit [DBG] from='client.? v1:172.21.15.121:0/3770210730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:25.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:25.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:25.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:25.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:25.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:25.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:25.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:25.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:25.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:25.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:25.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:25.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:25.785 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 16 5414 5398 337.358 337.375 0.00195214 0.00295981 2024-02-12T16:24:25.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:25.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:25.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:26.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:26.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:26.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:26.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:26.162 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:26.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:26.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:26.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:26.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:26.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:26.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:26.665 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:26 smithi169 bash[19604]: cluster 2024-02-12T16:24:25.611201+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e553: 12 total, 12 up, 12 in 2024-02-12T16:24:26.666 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:26 smithi169 bash[19604]: cluster 2024-02-12T16:24:25.928823+0000 mgr.y (mgr.14146) 1181 : cluster [DBG] pgmap v1627: 37 pgs: 37 active+clean; 29 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:26.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:26.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:26.785 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 16 10890 10874 339.786 342.25 0.00189475 0.00294036 2024-02-12T16:24:26.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:26.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:26 smithi123 bash[19119]: cluster 2024-02-12T16:24:25.611201+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e553: 12 total, 12 up, 12 in 2024-02-12T16:24:26.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:26 smithi123 bash[19119]: cluster 2024-02-12T16:24:25.928823+0000 mgr.y (mgr.14146) 1181 : cluster [DBG] pgmap v1627: 37 pgs: 37 active+clean; 29 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:26.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:26 smithi121 bash[18489]: cluster 2024-02-12T16:24:25.611201+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e553: 12 total, 12 up, 12 in 2024-02-12T16:24:26.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:26 smithi121 bash[18489]: cluster 2024-02-12T16:24:25.928823+0000 mgr.y (mgr.14146) 1181 : cluster [DBG] pgmap v1627: 37 pgs: 37 active+clean; 29 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:26.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:26.944 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:26.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:27.047 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:27.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:27.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:27.208 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:27.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:27.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:27.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:27.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:27.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:27.598 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:27.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:27.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:27.785 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 16087 16071 334.785 324.812 0.00207652 0.00298461 2024-02-12T16:24:27.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:27.859 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:27.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:27.988 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:27.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:28.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:28.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:28.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:28.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:28.248 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:28.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:28.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:28.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:28.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:28.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:28.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:28.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:28.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:28.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:28.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:28.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:24:28.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:28.785 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 15 20822 20807 325.083 296 0.00290015 0.00307349 2024-02-12T16:24:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:28.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:28.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:28.918 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:28.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:29.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:29.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:29.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:29.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:29 smithi121 bash[18489]: cluster 2024-02-12T16:24:27.929269+0000 mgr.y (mgr.14146) 1182 : cluster [DBG] pgmap v1628: 37 pgs: 37 active+clean; 349 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:29.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:28 smithi169 bash[19604]: cluster 2024-02-12T16:24:27.929269+0000 mgr.y (mgr.14146) 1182 : cluster [DBG] pgmap v1628: 37 pgs: 37 active+clean; 349 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:29.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:28 smithi123 bash[19119]: cluster 2024-02-12T16:24:27.929269+0000 mgr.y (mgr.14146) 1182 : cluster [DBG] pgmap v1628: 37 pgs: 37 active+clean; 349 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:29.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:29.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:29.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:29.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:29.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:29.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:29.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 25126 25110 313.849 268.938 0.00158051 0.00318412 2024-02-12T16:24:29.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:29.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:29.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:29.982 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:29.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:30.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:30.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:30.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:30.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:30.248 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:29 smithi121 bash[18489]: cluster 2024-02-12T16:24:28.139620+0000 osd.8 (osd.8) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:30.248 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:29 smithi121 bash[18489]: cluster 2024-02-12T16:24:28.139679+0000 osd.8 (osd.8) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:30.248 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:29 smithi121 bash[18489]: cluster 2024-02-12T16:24:28.944488+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:30.248 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:29 smithi121 bash[18489]: cluster 2024-02-12T16:24:28.944576+0000 osd.3 (osd.3) 11 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:30.248 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:29 smithi121 bash[18489]: cluster 2024-02-12T16:24:28.990502+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e554: 12 total, 12 up, 12 in 2024-02-12T16:24:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:30.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:29 smithi169 bash[19604]: cluster 2024-02-12T16:24:28.139620+0000 osd.8 (osd.8) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:29 smithi169 bash[19604]: cluster 2024-02-12T16:24:28.139679+0000 osd.8 (osd.8) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:30.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:29 smithi169 bash[19604]: cluster 2024-02-12T16:24:28.944488+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:29 smithi169 bash[19604]: cluster 2024-02-12T16:24:28.944576+0000 osd.3 (osd.3) 11 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:29 smithi169 bash[19604]: cluster 2024-02-12T16:24:28.990502+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e554: 12 total, 12 up, 12 in 2024-02-12T16:24:30.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:30.373 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:30.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:29 smithi123 bash[19119]: cluster 2024-02-12T16:24:28.139620+0000 osd.8 (osd.8) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:29 smithi123 bash[19119]: cluster 2024-02-12T16:24:28.139679+0000 osd.8 (osd.8) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:29 smithi123 bash[19119]: cluster 2024-02-12T16:24:28.944488+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:29 smithi123 bash[19119]: cluster 2024-02-12T16:24:28.944576+0000 osd.3 (osd.3) 11 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:29 smithi123 bash[19119]: cluster 2024-02-12T16:24:28.990502+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e554: 12 total, 12 up, 12 in 2024-02-12T16:24:30.476 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:30.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:30.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:30.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:30.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:30.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:30.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:30.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:30.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 29004 28988 301.934 242.375 0.00460531 0.00331006 2024-02-12T16:24:30.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:30.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:30.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:31.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:31.034 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:31.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:31.146 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:30 smithi169 bash[19604]: cluster 2024-02-12T16:24:29.929670+0000 mgr.y (mgr.14146) 1183 : cluster [DBG] pgmap v1630: 37 pgs: 37 active+clean; 597 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:31.146 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:30 smithi169 bash[19604]: cluster 2024-02-12T16:24:30.613327+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e555: 12 total, 12 up, 12 in 2024-02-12T16:24:31.146 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:30 smithi169 bash[19604]: audit 2024-02-12T16:24:30.687611+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? v1:172.21.15.121:0/2509738933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:24:31.162 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:31.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:31.274 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:30 smithi121 bash[18489]: cluster 2024-02-12T16:24:29.929670+0000 mgr.y (mgr.14146) 1183 : cluster [DBG] pgmap v1630: 37 pgs: 37 active+clean; 597 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:31.274 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:30 smithi121 bash[18489]: cluster 2024-02-12T16:24:30.613327+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e555: 12 total, 12 up, 12 in 2024-02-12T16:24:31.274 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:30 smithi121 bash[18489]: audit 2024-02-12T16:24:30.687611+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? v1:172.21.15.121:0/2509738933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:24:31.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:30 smithi123 bash[19119]: cluster 2024-02-12T16:24:29.929670+0000 mgr.y (mgr.14146) 1183 : cluster [DBG] pgmap v1630: 37 pgs: 37 active+clean; 597 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:24:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:30 smithi123 bash[19119]: cluster 2024-02-12T16:24:30.613327+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e555: 12 total, 12 up, 12 in 2024-02-12T16:24:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:30 smithi123 bash[19119]: audit 2024-02-12T16:24:30.687611+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? v1:172.21.15.121:0/2509738933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:24:31.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:31.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:31.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:31.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:31.635 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:24:31.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:31.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:31.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 16 32838 32822 293.03 239.625 0.00522765 0.00341098 2024-02-12T16:24:31.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:31.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:31.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:31.943 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:31.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:32.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:32.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:32.184 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:32 smithi169 bash[19604]: cluster 2024-02-12T16:24:31.618593+0000 mon.a (mon.0) 1951 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:24:32.184 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:32 smithi169 bash[19604]: audit 2024-02-12T16:24:31.624828+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.121:0/2509738933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:24:32.185 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:32 smithi169 bash[19604]: cluster 2024-02-12T16:24:31.624876+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e556: 12 total, 12 up, 12 in 2024-02-12T16:24:32.204 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:32.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:32.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:32.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:32.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:32 smithi123 bash[19119]: cluster 2024-02-12T16:24:31.618593+0000 mon.a (mon.0) 1951 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:24:32.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:32 smithi123 bash[19119]: audit 2024-02-12T16:24:31.624828+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.121:0/2509738933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:24:32.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:32 smithi123 bash[19119]: cluster 2024-02-12T16:24:31.624876+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e556: 12 total, 12 up, 12 in 2024-02-12T16:24:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:31 smithi121 bash[18489]: cluster 2024-02-12T16:24:31.618593+0000 mon.a (mon.0) 1951 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:24:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:31 smithi121 bash[18489]: audit 2024-02-12T16:24:31.624828+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? v1:172.21.15.121:0/2509738933' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:24:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:31 smithi121 bash[18489]: cluster 2024-02-12T16:24:31.624876+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e556: 12 total, 12 up, 12 in 2024-02-12T16:24:32.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:32.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:32.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:32.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:32.731 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:32.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 16 36396 36380 284.196 222.375 0.00579039 0.00351716 2024-02-12T16:24:32.836 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:32.865 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:32.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:32.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:32.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:33.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:33.106 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:33 smithi123 bash[19119]: cluster 2024-02-12T16:24:31.710565+0000 osd.4 (osd.4) 86 : cluster [DBG] 1.e scrub starts 2024-02-12T16:24:33.106 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:33 smithi123 bash[19119]: cluster 2024-02-12T16:24:31.781411+0000 osd.4 (osd.4) 87 : cluster [DBG] 1.e scrub ok 2024-02-12T16:24:33.106 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:33 smithi123 bash[19119]: cluster 2024-02-12T16:24:31.930138+0000 mgr.y (mgr.14146) 1184 : cluster [DBG] pgmap v1633: 37 pgs: 37 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 262 MiB/s wr, 4.18k op/s 2024-02-12T16:24:33.125 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:33.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:33.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:33.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:33.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:32 smithi121 bash[18489]: cluster 2024-02-12T16:24:31.710565+0000 osd.4 (osd.4) 86 : cluster [DBG] 1.e scrub starts 2024-02-12T16:24:33.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:32 smithi121 bash[18489]: cluster 2024-02-12T16:24:31.781411+0000 osd.4 (osd.4) 87 : cluster [DBG] 1.e scrub ok 2024-02-12T16:24:33.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:32 smithi121 bash[18489]: cluster 2024-02-12T16:24:31.930138+0000 mgr.y (mgr.14146) 1184 : cluster [DBG] pgmap v1633: 37 pgs: 37 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 262 MiB/s wr, 4.18k op/s 2024-02-12T16:24:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:33.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:33 smithi169 bash[19604]: cluster 2024-02-12T16:24:31.710565+0000 osd.4 (osd.4) 86 : cluster [DBG] 1.e scrub starts 2024-02-12T16:24:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:33 smithi169 bash[19604]: cluster 2024-02-12T16:24:31.781411+0000 osd.4 (osd.4) 87 : cluster [DBG] 1.e scrub ok 2024-02-12T16:24:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:33 smithi169 bash[19604]: cluster 2024-02-12T16:24:31.930138+0000 mgr.y (mgr.14146) 1184 : cluster [DBG] pgmap v1633: 37 pgs: 37 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 262 MiB/s wr, 4.18k op/s 2024-02-12T16:24:33.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:33.390 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:33.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:33.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:33.516 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:24:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:24:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:24:33.516 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:24:33.519 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:33.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:33.614 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:33.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:33.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:33.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:33.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:33.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:33.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:33.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 39613 39597 274.957 201.062 0.00429458 0.00363466 2024-02-12T16:24:33.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:33.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:33.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:34.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:34.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:34.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:34.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:34.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:34.282 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:34 smithi123 bash[19119]: cluster 2024-02-12T16:24:32.473712+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:34.282 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:34 smithi123 bash[19119]: cluster 2024-02-12T16:24:32.473759+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:34.282 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:34 smithi123 bash[19119]: cluster 2024-02-12T16:24:33.008735+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e557: 12 total, 12 up, 12 in 2024-02-12T16:24:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:34 smithi169 bash[19604]: cluster 2024-02-12T16:24:32.473712+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:34 smithi169 bash[19604]: cluster 2024-02-12T16:24:32.473759+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:34 smithi169 bash[19604]: cluster 2024-02-12T16:24:33.008735+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e557: 12 total, 12 up, 12 in 2024-02-12T16:24:34.300 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:34.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:34 smithi121 bash[18489]: cluster 2024-02-12T16:24:32.473712+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:34 smithi121 bash[18489]: cluster 2024-02-12T16:24:32.473759+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:34 smithi121 bash[18489]: cluster 2024-02-12T16:24:33.008735+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e557: 12 total, 12 up, 12 in 2024-02-12T16:24:34.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:34.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:34.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:34.564 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:34.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:34.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:34.693 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:34.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 16 42785 42769 267.285 198.25 0.00500947 0.00373916 2024-02-12T16:24:34.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:34.820 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:34.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:34.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:34.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:35.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:35.080 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:35.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:35.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:35.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:35.222 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:35.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:35.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:24:35.274 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:35 smithi121 bash[18489]: cluster 2024-02-12T16:24:33.930568+0000 mgr.y (mgr.14146) 1185 : cluster [DBG] pgmap v1635: 37 pgs: 37 active+clean; 1.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 MiB/s wr, 4.56k op/s 2024-02-12T16:24:35.274 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:35 smithi121 bash[18489]: cluster 2024-02-12T16:24:34.325288+0000 osd.1 (osd.1) 73 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:24:35.274 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:35 smithi121 bash[18489]: cluster 2024-02-12T16:24:34.437963+0000 osd.1 (osd.1) 74 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:24:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:35 smithi169 bash[19604]: cluster 2024-02-12T16:24:33.930568+0000 mgr.y (mgr.14146) 1185 : cluster [DBG] pgmap v1635: 37 pgs: 37 active+clean; 1.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 MiB/s wr, 4.56k op/s 2024-02-12T16:24:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:35 smithi169 bash[19604]: cluster 2024-02-12T16:24:34.325288+0000 osd.1 (osd.1) 73 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:24:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:35 smithi169 bash[19604]: cluster 2024-02-12T16:24:34.437963+0000 osd.1 (osd.1) 74 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:24:35.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:35.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:35.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:35 smithi123 bash[19119]: cluster 2024-02-12T16:24:33.930568+0000 mgr.y (mgr.14146) 1185 : cluster [DBG] pgmap v1635: 37 pgs: 37 active+clean; 1.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 285 MiB/s wr, 4.56k op/s 2024-02-12T16:24:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:35 smithi123 bash[19119]: cluster 2024-02-12T16:24:34.325288+0000 osd.1 (osd.1) 73 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:24:35.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:35 smithi123 bash[19119]: cluster 2024-02-12T16:24:34.437963+0000 osd.1 (osd.1) 74 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:24:35.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:35.494 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:35.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:35.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:35.625 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:35.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:35.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:35.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:35.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:35.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 45846 45830 260.377 191.312 0.0042386 0.00383855 2024-02-12T16:24:35.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:35.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:35.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:35.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:24:35.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:36.023 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:36.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:36.153 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:36.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:36.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:36.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:36.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:36.442 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:36.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:36.543 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:36.634 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:36 smithi121 bash[18489]: cluster 2024-02-12T16:24:35.931096+0000 mgr.y (mgr.14146) 1186 : cluster [DBG] pgmap v1636: 37 pgs: 37 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 288 MiB/s wr, 4.62k op/s 2024-02-12T16:24:36.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:36.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:36.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:36.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 48733 48717 253.714 180.438 0.00273105 0.00393933 2024-02-12T16:24:36.805 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:36 smithi123 bash[19119]: cluster 2024-02-12T16:24:35.931096+0000 mgr.y (mgr.14146) 1186 : cluster [DBG] pgmap v1636: 37 pgs: 37 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 288 MiB/s wr, 4.62k op/s 2024-02-12T16:24:36.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:36.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:36.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:37.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:36 smithi169 bash[19604]: cluster 2024-02-12T16:24:35.931096+0000 mgr.y (mgr.14146) 1186 : cluster [DBG] pgmap v1636: 37 pgs: 37 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 288 MiB/s wr, 4.62k op/s 2024-02-12T16:24:37.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:37.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:37.200 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:37.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:37.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:37.473 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:37.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:37.604 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:37.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:37.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:37.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 51682 51666 248.374 184.312 0.00734393 0.00402455 2024-02-12T16:24:37.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:37.865 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:37.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:37.994 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:37.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:38.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:38.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:38.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:38.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:38.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:38.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:38.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:38.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:38.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:38.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:38.519 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:38.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:38.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:38.645 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:38.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:38.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:38.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:38.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:38.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 54338 54322 242.49 166 0.00702067 0.00412175 2024-02-12T16:24:38.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:38.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:38.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:39.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:39.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:39.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:38 smithi123 bash[19119]: cluster 2024-02-12T16:24:37.932562+0000 mgr.y (mgr.14146) 1187 : cluster [DBG] pgmap v1637: 37 pgs: 37 active+clean; 2.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 240 MiB/s wr, 3.83k op/s 2024-02-12T16:24:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:38 smithi123 bash[19119]: audit 2024-02-12T16:24:38.771367+0000 mon.a (mon.0) 1955 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:24:39.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:39 smithi169 bash[19604]: cluster 2024-02-12T16:24:37.932562+0000 mgr.y (mgr.14146) 1187 : cluster [DBG] pgmap v1637: 37 pgs: 37 active+clean; 2.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 240 MiB/s wr, 3.83k op/s 2024-02-12T16:24:39.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:39 smithi169 bash[19604]: audit 2024-02-12T16:24:38.771367+0000 mon.a (mon.0) 1955 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:24:39.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:39.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:39.275 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:38 smithi121 bash[18489]: cluster 2024-02-12T16:24:37.932562+0000 mgr.y (mgr.14146) 1187 : cluster [DBG] pgmap v1637: 37 pgs: 37 active+clean; 2.8 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 240 MiB/s wr, 3.83k op/s 2024-02-12T16:24:39.275 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:38 smithi121 bash[18489]: audit 2024-02-12T16:24:38.771367+0000 mon.a (mon.0) 1955 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:24:39.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:39.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:39.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:39.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:39.550 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:39.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:39.679 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:39.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:39.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 56895 56879 236.977 159.812 0.00529328 0.0042179 2024-02-12T16:24:39.806 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:39.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:39.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:39.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:40.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:40.060 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:40.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:40.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:40.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:40.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:40.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:40.446 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:40.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:40.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:40.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:40.786 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 16 59301 59285 231.564 150.375 0.00348948 0.00431694 2024-02-12T16:24:40.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:40.844 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:40.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:40.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:41.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:41.098 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:41.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:41.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:41.208 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:41 smithi123 bash[19119]: cluster 2024-02-12T16:24:39.933101+0000 mgr.y (mgr.14146) 1188 : cluster [DBG] pgmap v1638: 37 pgs: 37 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:24:41.208 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:41 smithi123 bash[19119]: cluster 2024-02-12T16:24:40.211503+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:41.208 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:41 smithi123 bash[19119]: cluster 2024-02-12T16:24:40.211573+0000 osd.7 (osd.7) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:41.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:41.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:41.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:41 smithi169 bash[19604]: cluster 2024-02-12T16:24:39.933101+0000 mgr.y (mgr.14146) 1188 : cluster [DBG] pgmap v1638: 37 pgs: 37 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:24:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:41 smithi169 bash[19604]: cluster 2024-02-12T16:24:40.211503+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:41 smithi169 bash[19604]: cluster 2024-02-12T16:24:40.211573+0000 osd.7 (osd.7) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:41.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:41.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:40 smithi121 bash[18489]: cluster 2024-02-12T16:24:39.933101+0000 mgr.y (mgr.14146) 1188 : cluster [DBG] pgmap v1638: 37 pgs: 37 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:24:41.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:40 smithi121 bash[18489]: cluster 2024-02-12T16:24:40.211503+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:41.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:40 smithi121 bash[18489]: cluster 2024-02-12T16:24:40.211573+0000 osd.7 (osd.7) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:41.357 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:41.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:41.454 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:41.454 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:41.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:41.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:41.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:41.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:41.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:41.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:41.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:41.753 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:41.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:41.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 61682 61666 226.695 148.812 0.00373428 0.00440933 2024-02-12T16:24:41.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:41.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:41.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:42.020 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T16:24:42.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:42.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:42.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:42 smithi121 bash[18489]: cluster 2024-02-12T16:24:40.257003+0000 osd.9 (osd.9) 91 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:24:42.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:42 smithi121 bash[18489]: cluster 2024-02-12T16:24:40.282656+0000 osd.9 (osd.9) 92 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:24:42.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:42 smithi121 bash[18489]: cluster 2024-02-12T16:24:40.997118+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e558: 12 total, 12 up, 12 in 2024-02-12T16:24:42.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:42 smithi121 bash[18489]: audit 2024-02-12T16:24:41.530740+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.121:0/1773276873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:24:42.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:42.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:42.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:42.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:42 smithi169 bash[19604]: cluster 2024-02-12T16:24:40.257003+0000 osd.9 (osd.9) 91 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:24:42.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:42 smithi169 bash[19604]: cluster 2024-02-12T16:24:40.282656+0000 osd.9 (osd.9) 92 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:24:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:42 smithi169 bash[19604]: cluster 2024-02-12T16:24:40.997118+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e558: 12 total, 12 up, 12 in 2024-02-12T16:24:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:42 smithi169 bash[19604]: audit 2024-02-12T16:24:41.530740+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.121:0/1773276873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:24:42.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:42 smithi123 bash[19119]: cluster 2024-02-12T16:24:40.257003+0000 osd.9 (osd.9) 91 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:24:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:42 smithi123 bash[19119]: cluster 2024-02-12T16:24:40.282656+0000 osd.9 (osd.9) 92 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:24:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:42 smithi123 bash[19119]: cluster 2024-02-12T16:24:40.997118+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e558: 12 total, 12 up, 12 in 2024-02-12T16:24:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:42 smithi123 bash[19119]: audit 2024-02-12T16:24:41.530740+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.121:0/1773276873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:24:42.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:42.426 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:42.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:42.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:42.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:42.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:42.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 64170 64154 222.739 155.5 0.00498949 0.00448801 2024-02-12T16:24:42.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:42.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:42.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:42.950 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:42.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:43.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:43.060 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:43 smithi121 bash[18489]: cluster 2024-02-12T16:24:41.933494+0000 mgr.y (mgr.14146) 1189 : cluster [DBG] pgmap v1640: 37 pgs: 37 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.23k op/s 2024-02-12T16:24:43.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:43 smithi121 bash[18489]: audit 2024-02-12T16:24:42.003224+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? v1:172.21.15.121:0/1773276873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:24:43.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:43 smithi121 bash[18489]: cluster 2024-02-12T16:24:42.003282+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e559: 12 total, 12 up, 12 in 2024-02-12T16:24:43.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:43.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:43.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:43.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:43.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:43.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:43.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:43 smithi169 bash[19604]: cluster 2024-02-12T16:24:41.933494+0000 mgr.y (mgr.14146) 1189 : cluster [DBG] pgmap v1640: 37 pgs: 37 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.23k op/s 2024-02-12T16:24:43.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:43 smithi169 bash[19604]: audit 2024-02-12T16:24:42.003224+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? v1:172.21.15.121:0/1773276873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:24:43.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:43 smithi169 bash[19604]: cluster 2024-02-12T16:24:42.003282+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e559: 12 total, 12 up, 12 in 2024-02-12T16:24:43.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:43.329 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:43 smithi123 bash[19119]: cluster 2024-02-12T16:24:41.933494+0000 mgr.y (mgr.14146) 1189 : cluster [DBG] pgmap v1640: 37 pgs: 37 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.23k op/s 2024-02-12T16:24:43.330 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:43 smithi123 bash[19119]: audit 2024-02-12T16:24:42.003224+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? v1:172.21.15.121:0/1773276873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:24:43.330 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:43 smithi123 bash[19119]: cluster 2024-02-12T16:24:42.003282+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e559: 12 total, 12 up, 12 in 2024-02-12T16:24:43.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:43.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:43.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:43.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:43.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:43.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:43.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:43.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:43.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:43.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:43.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:43.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:43.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 66606 66590 219.029 152.25 0.00749772 0.00456384 2024-02-12T16:24:43.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:43.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:43.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:43.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:43.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:44.016 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:44.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:44.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T16:24:44.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:44.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:44.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:44.312 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:44.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:44.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:44.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:44.566 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:44.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:44.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:44.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:24:44.784733 min lat: 0.000938475 max lat: 0.0854576 avg lat: 0.00464742 2024-02-12T16:24:44.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:24:44.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 15 68846 68831 215.08 140.062 0.00354814 0.00464742 2024-02-12T16:24:44.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:44.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:44.853 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:24:44.890 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:24:44.923 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:24:44.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:44.951 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:24:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:44.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:44.983 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:24:45.015 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:24:45.017 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:45 smithi123 bash[19119]: cluster 2024-02-12T16:24:43.933919+0000 mgr.y (mgr.14146) 1190 : cluster [DBG] pgmap v1642: 37 pgs: 37 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.89k op/s 2024-02-12T16:24:45.017 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:45 smithi123 bash[19119]: audit 2024-02-12T16:24:44.811068+0000 mon.c (mon.1) 41 : audit [DBG] from='client.? v1:172.21.15.121:0/3532156185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:45.047 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:24:45.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:45.083 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:24:45.094 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:45.113 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:24:45.143 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:24:45.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:45 smithi121 bash[18489]: cluster 2024-02-12T16:24:43.933919+0000 mgr.y (mgr.14146) 1190 : cluster [DBG] pgmap v1642: 37 pgs: 37 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.89k op/s 2024-02-12T16:24:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:45 smithi121 bash[18489]: audit 2024-02-12T16:24:44.811068+0000 mon.c (mon.1) 41 : audit [DBG] from='client.? v1:172.21.15.121:0/3532156185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:45.175 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:24:45.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:45.203 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:24:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:24:45.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:45 smithi169 bash[19604]: cluster 2024-02-12T16:24:43.933919+0000 mgr.y (mgr.14146) 1190 : cluster [DBG] pgmap v1642: 37 pgs: 37 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.89k op/s 2024-02-12T16:24:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:45 smithi169 bash[19604]: audit 2024-02-12T16:24:44.811068+0000 mon.c (mon.1) 41 : audit [DBG] from='client.? v1:172.21.15.121:0/3532156185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:24:45.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:45.350 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:45.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:45.476 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:45.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:45.603 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:45.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.605 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:45.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:45.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:45.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 70992 70976 211.221 134.062 0.00343158 0.00473287 2024-02-12T16:24:45.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:45.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:45.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:45.997 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:45.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:46.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:46.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:46.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:46.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:46.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:46.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:46.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:46.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:46.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:46.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:46.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:46.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:46 smithi121 bash[18489]: cluster 2024-02-12T16:24:45.923522+0000 osd.6 (osd.6) 102 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:24:46.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:46 smithi121 bash[18489]: cluster 2024-02-12T16:24:45.934361+0000 mgr.y (mgr.14146) 1191 : cluster [DBG] pgmap v1643: 37 pgs: 37 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.45k op/s 2024-02-12T16:24:46.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:46.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:46.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:46.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:46.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 73103 73087 207.617 131.938 0.00688736 0.00481503 2024-02-12T16:24:46.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:46.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:46 smithi123 bash[19119]: cluster 2024-02-12T16:24:45.923522+0000 osd.6 (osd.6) 102 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:24:46.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:46 smithi123 bash[19119]: cluster 2024-02-12T16:24:45.934361+0000 mgr.y (mgr.14146) 1191 : cluster [DBG] pgmap v1643: 37 pgs: 37 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.45k op/s 2024-02-12T16:24:46.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:46 smithi123 bash[19119]: cluster 2024-02-12T16:24:45.967335+0000 osd.6 (osd.6) 103 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:24:46.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:46.908 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:46 smithi121 bash[18489]: cluster 2024-02-12T16:24:45.967335+0000 osd.6 (osd.6) 103 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:24:46.927 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:46.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:47.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:46 smithi169 bash[19604]: cluster 2024-02-12T16:24:45.923522+0000 osd.6 (osd.6) 102 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:24:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:46 smithi169 bash[19604]: cluster 2024-02-12T16:24:45.934361+0000 mgr.y (mgr.14146) 1191 : cluster [DBG] pgmap v1643: 37 pgs: 37 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s wr, 2.45k op/s 2024-02-12T16:24:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:46 smithi169 bash[19604]: cluster 2024-02-12T16:24:45.967335+0000 osd.6 (osd.6) 103 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:24:47.057 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:47.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:47.184 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:47.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:47.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:47.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:47.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:47.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:47.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:47.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:47.714 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:47.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 75309 75293 204.584 137.875 0.00626648 0.00488593 2024-02-12T16:24:47.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:47.848 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:47.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:47.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:47.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:48.090 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:48.121 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:48.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:48.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:48.254 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:48.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:48.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:48.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:48.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:48.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:48.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:48.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:48.623 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:48.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:48.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:48.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 77451 77435 201.638 133.875 0.0144302 0.00495739 2024-02-12T16:24:48.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:48.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:48 smithi169 bash[19604]: cluster 2024-02-12T16:24:46.942932+0000 osd.1 (osd.1) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:48 smithi169 bash[19604]: cluster 2024-02-12T16:24:46.942982+0000 osd.1 (osd.1) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:48 smithi169 bash[19604]: cluster 2024-02-12T16:24:47.357097+0000 osd.9 (osd.9) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:48 smithi169 bash[19604]: cluster 2024-02-12T16:24:47.357175+0000 osd.9 (osd.9) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:48 smithi169 bash[19604]: cluster 2024-02-12T16:24:47.630969+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e560: 12 total, 12 up, 12 in 2024-02-12T16:24:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:48 smithi169 bash[19604]: cluster 2024-02-12T16:24:47.934693+0000 mgr.y (mgr.14146) 1192 : cluster [DBG] pgmap v1645: 37 pgs: 37 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.63k op/s 2024-02-12T16:24:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:48 smithi123 bash[19119]: cluster 2024-02-12T16:24:46.942932+0000 osd.1 (osd.1) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:48.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:48 smithi123 bash[19119]: cluster 2024-02-12T16:24:46.942982+0000 osd.1 (osd.1) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:48.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:48 smithi123 bash[19119]: cluster 2024-02-12T16:24:47.357097+0000 osd.9 (osd.9) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:48.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:48 smithi123 bash[19119]: cluster 2024-02-12T16:24:47.357175+0000 osd.9 (osd.9) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:48.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:48 smithi123 bash[19119]: cluster 2024-02-12T16:24:47.630969+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e560: 12 total, 12 up, 12 in 2024-02-12T16:24:48.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:48 smithi123 bash[19119]: cluster 2024-02-12T16:24:47.934693+0000 mgr.y (mgr.14146) 1192 : cluster [DBG] pgmap v1645: 37 pgs: 37 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.63k op/s 2024-02-12T16:24:48.891 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:48.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:48 smithi121 bash[18489]: cluster 2024-02-12T16:24:46.942932+0000 osd.1 (osd.1) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:48 smithi121 bash[18489]: cluster 2024-02-12T16:24:46.942982+0000 osd.1 (osd.1) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:48 smithi121 bash[18489]: cluster 2024-02-12T16:24:47.357097+0000 osd.9 (osd.9) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:24:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:48 smithi121 bash[18489]: cluster 2024-02-12T16:24:47.357175+0000 osd.9 (osd.9) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:24:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:48 smithi121 bash[18489]: cluster 2024-02-12T16:24:47.630969+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e560: 12 total, 12 up, 12 in 2024-02-12T16:24:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:48 smithi121 bash[18489]: cluster 2024-02-12T16:24:47.934693+0000 mgr.y (mgr.14146) 1192 : cluster [DBG] pgmap v1645: 37 pgs: 37 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 165 MiB/s wr, 2.63k op/s 2024-02-12T16:24:48.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:48.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:49.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:49.038 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:49.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:49.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:49.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:49.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:49.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:49.418 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:49.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:49.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:49.544 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:49.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:49.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:49.683 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:49 smithi121 bash[18489]: cluster 2024-02-12T16:24:47.824475+0000 osd.10 (osd.10) 104 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:24:49.683 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:49 smithi121 bash[18489]: cluster 2024-02-12T16:24:47.872492+0000 osd.10 (osd.10) 105 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:24:49.683 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:49 smithi121 bash[18489]: audit 2024-02-12T16:24:49.413204+0000 mon.a (mon.0) 1961 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:49.725 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:24:49.725 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:24:49.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:49.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 79328 79312 198.264 117.312 0.0148396 0.00504192 2024-02-12T16:24:49.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:49 smithi169 bash[19604]: cluster 2024-02-12T16:24:47.824475+0000 osd.10 (osd.10) 104 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:24:49.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:49 smithi169 bash[19604]: cluster 2024-02-12T16:24:47.872492+0000 osd.10 (osd.10) 105 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:24:49.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:49 smithi169 bash[19604]: audit 2024-02-12T16:24:49.413204+0000 mon.a (mon.0) 1961 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:49.802 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:49.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:49.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:49 smithi123 bash[19119]: cluster 2024-02-12T16:24:47.824475+0000 osd.10 (osd.10) 104 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:24:49.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:49 smithi123 bash[19119]: cluster 2024-02-12T16:24:47.872492+0000 osd.10 (osd.10) 105 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:24:49.915 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:49 smithi123 bash[19119]: audit 2024-02-12T16:24:49.413204+0000 mon.a (mon.0) 1961 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:49.941 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:49.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:50.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:50.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:50.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:50.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:50.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:50.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:50.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:50.360 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:50.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:50.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:24:50.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:50.487 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:50.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:50.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:50.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:50.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:50.628 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:24:50.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: audit 2024-02-12T16:24:49.781465+0000 mon.a (mon.0) 1962 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:24:50.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: audit 2024-02-12T16:24:49.782396+0000 mon.a (mon.0) 1963 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:24:50.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: audit 2024-02-12T16:24:49.788698+0000 mon.a (mon.0) 1964 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:50.708 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: cluster 2024-02-12T16:24:49.935154+0000 mgr.y (mgr.14146) 1193 : cluster [DBG] pgmap v1646: 37 pgs: 37 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-12T16:24:50.709 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: audit 2024-02-12T16:24:50.177315+0000 mon.a (mon.0) 1965 : audit [INF] from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:50.709 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: audit 2024-02-12T16:24:50.178701+0000 mon.c (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.121:0/4293933177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:50.709 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: cluster 2024-02-12T16:24:50.611565+0000 mon.a (mon.0) 1966 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:24:50.709 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: cluster 2024-02-12T16:24:50.611605+0000 mon.a (mon.0) 1967 : cluster [INF] Cluster is now healthy 2024-02-12T16:24:50.709 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: audit 2024-02-12T16:24:50.615629+0000 mon.a (mon.0) 1968 : audit [INF] from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:24:50.709 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:50 smithi121 bash[18489]: cluster 2024-02-12T16:24:50.615714+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e561: 12 total, 12 up, 12 in 2024-02-12T16:24:50.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:50.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:50.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:50.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 16 81264 81248 195.292 121 0.00517009 0.00511889 2024-02-12T16:24:50.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:50.854 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: audit 2024-02-12T16:24:49.781465+0000 mon.a (mon.0) 1962 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: audit 2024-02-12T16:24:49.782396+0000 mon.a (mon.0) 1963 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: audit 2024-02-12T16:24:49.788698+0000 mon.a (mon.0) 1964 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: cluster 2024-02-12T16:24:49.935154+0000 mgr.y (mgr.14146) 1193 : cluster [DBG] pgmap v1646: 37 pgs: 37 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: audit 2024-02-12T16:24:50.177315+0000 mon.a (mon.0) 1965 : audit [INF] from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: audit 2024-02-12T16:24:50.178701+0000 mon.c (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.121:0/4293933177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: cluster 2024-02-12T16:24:50.611565+0000 mon.a (mon.0) 1966 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: cluster 2024-02-12T16:24:50.611605+0000 mon.a (mon.0) 1967 : cluster [INF] Cluster is now healthy 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: audit 2024-02-12T16:24:50.615629+0000 mon.a (mon.0) 1968 : audit [INF] from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:24:50.855 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:50 smithi123 bash[19119]: cluster 2024-02-12T16:24:50.615714+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e561: 12 total, 12 up, 12 in 2024-02-12T16:24:50.871 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:50.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:50.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:24:50.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:50.980 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: audit 2024-02-12T16:24:49.781465+0000 mon.a (mon.0) 1962 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: audit 2024-02-12T16:24:49.782396+0000 mon.a (mon.0) 1963 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: audit 2024-02-12T16:24:49.788698+0000 mon.a (mon.0) 1964 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: cluster 2024-02-12T16:24:49.935154+0000 mgr.y (mgr.14146) 1193 : cluster [DBG] pgmap v1646: 37 pgs: 37 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: audit 2024-02-12T16:24:50.177315+0000 mon.a (mon.0) 1965 : audit [INF] from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: audit 2024-02-12T16:24:50.178701+0000 mon.c (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.121:0/4293933177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: cluster 2024-02-12T16:24:50.611565+0000 mon.a (mon.0) 1966 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: cluster 2024-02-12T16:24:50.611605+0000 mon.a (mon.0) 1967 : cluster [INF] Cluster is now healthy 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: audit 2024-02-12T16:24:50.615629+0000 mon.a (mon.0) 1968 : audit [INF] from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:24:50.981 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:50 smithi169 bash[19604]: cluster 2024-02-12T16:24:50.615714+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e561: 12 total, 12 up, 12 in 2024-02-12T16:24:50.998 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:51.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:51.127 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:51.230 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:51.261 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:51.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:51.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:51.520 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:51.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:51.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:51.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:51.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 83267 83251 192.695 125.188 0.00574211 0.00518784 2024-02-12T16:24:51.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:51 smithi169 bash[19604]: cluster 2024-02-12T16:24:50.554106+0000 osd.5 (osd.5) 83 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:24:51.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:51 smithi169 bash[19604]: cluster 2024-02-12T16:24:50.575283+0000 osd.5 (osd.5) 84 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:24:51.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:51 smithi169 bash[19604]: cluster 2024-02-12T16:24:50.873459+0000 osd.7 (osd.7) 128 : cluster [DBG] 1.d scrub starts 2024-02-12T16:24:51.796 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:51 smithi169 bash[19604]: cluster 2024-02-12T16:24:50.938707+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.d scrub ok 2024-02-12T16:24:51.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:51.893 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:51 smithi121 bash[18489]: cluster 2024-02-12T16:24:50.554106+0000 osd.5 (osd.5) 83 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:24:51.893 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:51 smithi121 bash[18489]: cluster 2024-02-12T16:24:50.575283+0000 osd.5 (osd.5) 84 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:24:51.893 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:51 smithi121 bash[18489]: cluster 2024-02-12T16:24:50.873459+0000 osd.7 (osd.7) 128 : cluster [DBG] 1.d scrub starts 2024-02-12T16:24:51.893 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:51 smithi121 bash[18489]: cluster 2024-02-12T16:24:50.938707+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.d scrub ok 2024-02-12T16:24:51.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:51.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:52.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:52.033 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:51 smithi123 bash[19119]: cluster 2024-02-12T16:24:50.554106+0000 osd.5 (osd.5) 83 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:24:52.033 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:51 smithi123 bash[19119]: cluster 2024-02-12T16:24:50.575283+0000 osd.5 (osd.5) 84 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:24:52.033 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:51 smithi123 bash[19119]: cluster 2024-02-12T16:24:50.873459+0000 osd.7 (osd.7) 128 : cluster [DBG] 1.d scrub starts 2024-02-12T16:24:52.033 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:51 smithi123 bash[19119]: cluster 2024-02-12T16:24:50.938707+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.d scrub ok 2024-02-12T16:24:52.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:52.084 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:52.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:52.178 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:52.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:52.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:52.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:52.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:52.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:52.566 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:52.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:52.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:52.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 85254 85238 190.248 124.188 0.00669796 0.0052544 2024-02-12T16:24:52.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:52.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:52.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:52 smithi169 bash[19604]: cluster 2024-02-12T16:24:51.935622+0000 mgr.y (mgr.14146) 1194 : cluster [DBG] pgmap v1648: 37 pgs: 37 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s wr, 2.40k op/s 2024-02-12T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:52 smithi169 bash[19604]: cluster 2024-02-12T16:24:52.502604+0000 osd.0 (osd.0) 86 : cluster [DBG] 1.b scrub starts 2024-02-12T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:52 smithi169 bash[19604]: cluster 2024-02-12T16:24:52.579059+0000 osd.0 (osd.0) 87 : cluster [DBG] 1.b scrub ok 2024-02-12T16:24:52.969 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:52.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:53.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:52 smithi121 bash[18489]: cluster 2024-02-12T16:24:51.935622+0000 mgr.y (mgr.14146) 1194 : cluster [DBG] pgmap v1648: 37 pgs: 37 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s wr, 2.40k op/s 2024-02-12T16:24:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:52 smithi121 bash[18489]: cluster 2024-02-12T16:24:52.502604+0000 osd.0 (osd.0) 86 : cluster [DBG] 1.b scrub starts 2024-02-12T16:24:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:52 smithi121 bash[18489]: cluster 2024-02-12T16:24:52.579059+0000 osd.0 (osd.0) 87 : cluster [DBG] 1.b scrub ok 2024-02-12T16:24:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:53.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:53.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:52 smithi123 bash[19119]: cluster 2024-02-12T16:24:51.935622+0000 mgr.y (mgr.14146) 1194 : cluster [DBG] pgmap v1648: 37 pgs: 37 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s wr, 2.40k op/s 2024-02-12T16:24:53.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:52 smithi123 bash[19119]: cluster 2024-02-12T16:24:52.502604+0000 osd.0 (osd.0) 86 : cluster [DBG] 1.b scrub starts 2024-02-12T16:24:53.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:52 smithi123 bash[19119]: cluster 2024-02-12T16:24:52.579059+0000 osd.0 (osd.0) 87 : cluster [DBG] 1.b scrub ok 2024-02-12T16:24:53.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:53.236 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:53.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:53.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:53.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:53.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:53.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:53.513 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:53.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:53.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:53.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:53.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:53.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:24:53.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:53.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:53.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:53.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 87223 87207 187.931 123.062 0.0100038 0.00531977 2024-02-12T16:24:53.813 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:53 smithi123 bash[19119]: cluster 2024-02-12T16:24:52.862652+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:24:53.813 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:53 smithi123 bash[19119]: cluster 2024-02-12T16:24:52.902188+0000 osd.7 (osd.7) 131 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:24:53.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:53.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:53 smithi121 bash[18489]: cluster 2024-02-12T16:24:52.862652+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:24:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:53 smithi121 bash[18489]: cluster 2024-02-12T16:24:52.902188+0000 osd.7 (osd.7) 131 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:24:53.913 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:53.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:54.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:54.047 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:53 smithi169 bash[19604]: cluster 2024-02-12T16:24:52.862652+0000 osd.7 (osd.7) 130 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:24:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:53 smithi169 bash[19604]: cluster 2024-02-12T16:24:52.902188+0000 osd.7 (osd.7) 131 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:24:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:54.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:54.184 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:54.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:54.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:54.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:54.451 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:54.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:54.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:54.709 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:54.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 88671 88655 184.683 90.5 0.00478785 0.00541286 2024-02-12T16:24:54.812 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:54.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:54.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:54.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:54.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:55.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:55.094 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:54 smithi169 bash[19604]: cluster 2024-02-12T16:24:53.936071+0000 mgr.y (mgr.14146) 1195 : cluster [DBG] pgmap v1649: 37 pgs: 37 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-02-12T16:24:55.094 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:54 smithi169 bash[19604]: cluster 2024-02-12T16:24:54.464041+0000 osd.0 (osd.0) 88 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:24:55.094 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:54 smithi169 bash[19604]: cluster 2024-02-12T16:24:54.540851+0000 osd.0 (osd.0) 89 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:24:55.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:55.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:55.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:54 smithi123 bash[19119]: cluster 2024-02-12T16:24:53.936071+0000 mgr.y (mgr.14146) 1195 : cluster [DBG] pgmap v1649: 37 pgs: 37 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-02-12T16:24:55.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:54 smithi123 bash[19119]: cluster 2024-02-12T16:24:54.464041+0000 osd.0 (osd.0) 88 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:24:55.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:54 smithi123 bash[19119]: cluster 2024-02-12T16:24:54.540851+0000 osd.0 (osd.0) 89 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:24:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:54 smithi121 bash[18489]: cluster 2024-02-12T16:24:53.936071+0000 mgr.y (mgr.14146) 1195 : cluster [DBG] pgmap v1649: 37 pgs: 37 active+clean; 5.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-02-12T16:24:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:54 smithi121 bash[18489]: cluster 2024-02-12T16:24:54.464041+0000 osd.0 (osd.0) 88 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:24:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:54 smithi121 bash[18489]: cluster 2024-02-12T16:24:54.540851+0000 osd.0 (osd.0) 89 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:24:55.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:55.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:24:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:55.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:24:55.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:55.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:55.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:55.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:55.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:55.544 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:55.547 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:24:55.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:55.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:55.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:55.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:55.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:55.787 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 90462 90446 182.336 111.938 0.00393159 0.00548224 2024-02-12T16:24:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:55.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:55.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:55.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:55.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:56.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:56.042 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:56.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:56.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:56.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:56.316 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:56.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:56.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:56.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:56.573 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:56.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:56.683 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:56 smithi121 bash[18489]: cluster 2024-02-12T16:24:55.936528+0000 mgr.y (mgr.14146) 1196 : cluster [DBG] pgmap v1650: 37 pgs: 37 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-02-12T16:24:56.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:56.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 92330 92314 180.286 116.75 0.00449663 0.00554508 2024-02-12T16:24:56.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:56.844 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:56 smithi123 bash[19119]: cluster 2024-02-12T16:24:55.936528+0000 mgr.y (mgr.14146) 1196 : cluster [DBG] pgmap v1650: 37 pgs: 37 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-02-12T16:24:56.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:56.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:24:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:56.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:24:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:56 smithi169 bash[19604]: cluster 2024-02-12T16:24:55.936528+0000 mgr.y (mgr.14146) 1196 : cluster [DBG] pgmap v1650: 37 pgs: 37 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.82k op/s 2024-02-12T16:24:57.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:57.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:57.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:57.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:57.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:57.254 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:57.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:57.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:57.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:57.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:57.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:57.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:57.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:57.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:57.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:57.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:57.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:57.766 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:57.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:57.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 94167 94151 178.302 114.812 0.00576028 0.00560614 2024-02-12T16:24:57.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:57.896 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:57.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:58.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:58.027 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:24:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:24:58.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:58.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:24:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:24:58.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:58.308 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:24:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:24:58.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:58.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:24:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:24:58.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:58.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:24:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:24:58.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 95787 95771 176.035 101.25 0.00978407 0.00567907 2024-02-12T16:24:58.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:58.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:58.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:58.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:58.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:59.009 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T16:24:59.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:59.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:59.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:59 smithi123 bash[19119]: cluster 2024-02-12T16:24:57.937007+0000 mgr.y (mgr.14146) 1197 : cluster [DBG] pgmap v1651: 37 pgs: 37 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.96k op/s 2024-02-12T16:24:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:59 smithi123 bash[19119]: audit 2024-02-12T16:24:58.272585+0000 mon.a (mon.0) 1970 : audit [INF] from='client.25951 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:24:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:24:59 smithi123 bash[19119]: audit 2024-02-12T16:24:58.273883+0000 mon.c (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.121:0/110873593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:24:59.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:59.240 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:59.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:59 smithi169 bash[19604]: cluster 2024-02-12T16:24:57.937007+0000 mgr.y (mgr.14146) 1197 : cluster [DBG] pgmap v1651: 37 pgs: 37 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.96k op/s 2024-02-12T16:24:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:59 smithi169 bash[19604]: audit 2024-02-12T16:24:58.272585+0000 mon.a (mon.0) 1970 : audit [INF] from='client.25951 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:24:59.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:24:59 smithi169 bash[19604]: audit 2024-02-12T16:24:58.273883+0000 mon.c (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.121:0/110873593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:24:59.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:59.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:24:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:59.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:24:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:58 smithi121 bash[18489]: cluster 2024-02-12T16:24:57.937007+0000 mgr.y (mgr.14146) 1197 : cluster [DBG] pgmap v1651: 37 pgs: 37 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.96k op/s 2024-02-12T16:24:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:58 smithi121 bash[18489]: audit 2024-02-12T16:24:58.272585+0000 mon.a (mon.0) 1970 : audit [INF] from='client.25951 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:24:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:24:58 smithi121 bash[18489]: audit 2024-02-12T16:24:58.273883+0000 mon.c (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.121:0/110873593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:24:59.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:59.499 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:24:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:59.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:24:59.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:59.626 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:24:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:59.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:24:59.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:59.754 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:24:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:59.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:24:59.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 97573 97557 174.195 111.625 0.00410748 0.00573905 2024-02-12T16:24:59.857 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:59.890 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:24:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:24:59.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:24:59.999 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:00.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:00.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:00 smithi123 bash[19119]: audit 2024-02-12T16:24:58.994608+0000 mon.a (mon.0) 1971 : audit [INF] from='client.25951 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:25:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:00 smithi123 bash[19119]: cluster 2024-02-12T16:24:58.994713+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e562: 12 total, 12 up, 12 in 2024-02-12T16:25:00.163 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:00.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:00.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:00 smithi169 bash[19604]: audit 2024-02-12T16:24:58.994608+0000 mon.a (mon.0) 1971 : audit [INF] from='client.25951 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:25:00.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:00 smithi169 bash[19604]: cluster 2024-02-12T16:24:58.994713+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e562: 12 total, 12 up, 12 in 2024-02-12T16:25:00.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:00 smithi121 bash[18489]: audit 2024-02-12T16:24:58.994608+0000 mon.a (mon.0) 1971 : audit [INF] from='client.25951 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:25:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:00 smithi121 bash[18489]: cluster 2024-02-12T16:24:58.994713+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e562: 12 total, 12 up, 12 in 2024-02-12T16:25:00.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:00.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:00.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:00.553 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:00.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:00.688 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:00.731 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:00.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 16 99310 99294 172.372 108.562 0.00668486 0.00580001 2024-02-12T16:25:00.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:00.831 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:00.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:00.958 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:00.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:00.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:01.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:01.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:01 smithi123 bash[19119]: cluster 2024-02-12T16:24:59.937461+0000 mgr.y (mgr.14146) 1198 : cluster [DBG] pgmap v1653: 37 pgs: 37 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-02-12T16:25:01.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:01 smithi123 bash[19119]: cluster 2024-02-12T16:25:00.618440+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e563: 12 total, 12 up, 12 in 2024-02-12T16:25:01.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:01.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:01.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:01.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:01.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:01 smithi169 bash[19604]: cluster 2024-02-12T16:24:59.937461+0000 mgr.y (mgr.14146) 1198 : cluster [DBG] pgmap v1653: 37 pgs: 37 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-02-12T16:25:01.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:01 smithi169 bash[19604]: cluster 2024-02-12T16:25:00.618440+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e563: 12 total, 12 up, 12 in 2024-02-12T16:25:01.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:01.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:01.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:01.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:01 smithi121 bash[18489]: cluster 2024-02-12T16:24:59.937461+0000 mgr.y (mgr.14146) 1198 : cluster [DBG] pgmap v1653: 37 pgs: 37 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-02-12T16:25:01.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:01 smithi121 bash[18489]: cluster 2024-02-12T16:25:00.618440+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e563: 12 total, 12 up, 12 in 2024-02-12T16:25:01.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T16:25:01.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:01.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:01.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:01.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:01.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:01.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:01.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:01.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:01.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 101026 101010 170.611 107.25 0.0043579 0.00585945 2024-02-12T16:25:01.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:01.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:01.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:01.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:02.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:02.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:02.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:02.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:02.285 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:02.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:02.417 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:02.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:02.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:02.544 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:02.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:02.671 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:02.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:02.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 16 102720 102704 168.907 105.875 0.00404641 0.0059186 2024-02-12T16:25:02.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:02.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:02.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:02.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:03.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:03.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:03.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:03.146 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:03 smithi121 bash[18489]: cluster 2024-02-12T16:25:01.937901+0000 mgr.y (mgr.14146) 1199 : cluster [DBG] pgmap v1655: 37 pgs: 37 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-12T16:25:03.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:03.173 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:03 smithi169 bash[19604]: cluster 2024-02-12T16:25:01.937901+0000 mgr.y (mgr.14146) 1199 : cluster [DBG] pgmap v1655: 37 pgs: 37 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-12T16:25:03.189 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:03.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:03.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:03.324 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:03.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:03 smithi123 bash[19119]: cluster 2024-02-12T16:25:01.937901+0000 mgr.y (mgr.14146) 1199 : cluster [DBG] pgmap v1655: 37 pgs: 37 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-02-12T16:25:03.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:03.466 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:25:03.466 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:25:03.475 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:03.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:03.566 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:25:03.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:03.591 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:25:03.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:03.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:03.630 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:25:03.666 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:25:03.702 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:25:03.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:03.732 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:25:03.738 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:03.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:03.763 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:25:03.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 104305 104289 167.116 99.0625 0.009707 0.00598222 2024-02-12T16:25:03.790 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:25:03.821 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:25:03.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:03.849 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:25:03.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:03.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:03.881 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:25:03.914 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:25:03.920 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:25:03.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:04.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:04.139 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:04.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:02.477687+0000 osd.2 (osd.2) 68 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:04.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:02.477728+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.020904+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e564: 12 total, 12 up, 12 in 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: audit 2024-02-12T16:25:03.519637+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? v1:172.21.15.121:0/2648723791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.681289+0000 mon.a (mon.0) 1975 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=145534078,config=2903391811,health=3183869367,kv=1316805454}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.682269+0000 mon.a (mon.0) 1976 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=1249794974,logm=2611551194}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.686390+0000 mon.a (mon.0) 1977 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2372303730}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.687456+0000 mon.a (mon.0) 1978 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1381319640}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.688412+0000 mon.a (mon.0) 1979 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459661659}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.689322+0000 mon.a (mon.0) 1980 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128739226}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.692443+0000 mon.a (mon.0) 1981 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923283477}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.696629+0000 mon.a (mon.0) 1982 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092585448}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.700621+0000 mon.a (mon.0) 1983 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260407651}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.704549+0000 mon.a (mon.0) 1984 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4289448973}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.708393+0000 mon.a (mon.0) 1985 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021340921}) 2024-02-12T16:25:04.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.710379+0000 mon.a (mon.0) 1986 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=21,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=39} crc {logm=1376165264,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2845909620,mon_config_key=182807460}) 2024-02-12T16:25:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.711174+0000 mon.a (mon.0) 1987 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2993318715}) 2024-02-12T16:25:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.714714+0000 mon.a (mon.0) 1988 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=555211379}) 2024-02-12T16:25:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.716908+0000 mon.a (mon.0) 1989 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=2807001462,monmap=2720272532,osd_metadata=1720028586,osd_pg_creating=573451927,osdmap=406672819}) 2024-02-12T16:25:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.720286+0000 mon.a (mon.0) 1990 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2252709668}) 2024-02-12T16:25:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.724234+0000 mon.a (mon.0) 1991 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3336193467}) 2024-02-12T16:25:04.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:04 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.738085+0000 mon.a (mon.0) 1992 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=69} crc {osdmap=1897974408}) 2024-02-12T16:25:04.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:04.269 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:02.477687+0000 osd.2 (osd.2) 68 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:02.477728+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.020904+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e564: 12 total, 12 up, 12 in 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: audit 2024-02-12T16:25:03.519637+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? v1:172.21.15.121:0/2648723791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.681289+0000 mon.a (mon.0) 1975 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=145534078,config=2903391811,health=3183869367,kv=1316805454}) 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.682269+0000 mon.a (mon.0) 1976 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=1249794974,logm=2611551194}) 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.686390+0000 mon.a (mon.0) 1977 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2372303730}) 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.687456+0000 mon.a (mon.0) 1978 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1381319640}) 2024-02-12T16:25:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.688412+0000 mon.a (mon.0) 1979 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459661659}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.689322+0000 mon.a (mon.0) 1980 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128739226}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.692443+0000 mon.a (mon.0) 1981 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923283477}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.696629+0000 mon.a (mon.0) 1982 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092585448}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.700621+0000 mon.a (mon.0) 1983 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260407651}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.704549+0000 mon.a (mon.0) 1984 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4289448973}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.708393+0000 mon.a (mon.0) 1985 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021340921}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.710379+0000 mon.a (mon.0) 1986 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=21,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=39} crc {logm=1376165264,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2845909620,mon_config_key=182807460}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.711174+0000 mon.a (mon.0) 1987 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2993318715}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.714714+0000 mon.a (mon.0) 1988 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=555211379}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.716908+0000 mon.a (mon.0) 1989 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=2807001462,monmap=2720272532,osd_metadata=1720028586,osd_pg_creating=573451927,osdmap=406672819}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.720286+0000 mon.a (mon.0) 1990 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2252709668}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.724234+0000 mon.a (mon.0) 1991 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3336193467}) 2024-02-12T16:25:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:04 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.738085+0000 mon.a (mon.0) 1992 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=69} crc {osdmap=1897974408}) 2024-02-12T16:25:04.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:02.477687+0000 osd.2 (osd.2) 68 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:02.477728+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.020904+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e564: 12 total, 12 up, 12 in 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: audit 2024-02-12T16:25:03.519637+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? v1:172.21.15.121:0/2648723791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.681289+0000 mon.a (mon.0) 1975 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=145534078,config=2903391811,health=3183869367,kv=1316805454}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.682269+0000 mon.a (mon.0) 1976 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=1249794974,logm=2611551194}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.686390+0000 mon.a (mon.0) 1977 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2372303730}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.687456+0000 mon.a (mon.0) 1978 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1381319640}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.688412+0000 mon.a (mon.0) 1979 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459661659}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.689322+0000 mon.a (mon.0) 1980 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128739226}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.692443+0000 mon.a (mon.0) 1981 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923283477}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.696629+0000 mon.a (mon.0) 1982 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092585448}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.700621+0000 mon.a (mon.0) 1983 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260407651}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.704549+0000 mon.a (mon.0) 1984 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4289448973}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.708393+0000 mon.a (mon.0) 1985 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021340921}) 2024-02-12T16:25:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.710379+0000 mon.a (mon.0) 1986 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=21,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=39} crc {logm=1376165264,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2845909620,mon_config_key=182807460}) 2024-02-12T16:25:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.711174+0000 mon.a (mon.0) 1987 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2993318715}) 2024-02-12T16:25:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.714714+0000 mon.a (mon.0) 1988 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=555211379}) 2024-02-12T16:25:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.716908+0000 mon.a (mon.0) 1989 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=2807001462,monmap=2720272532,osd_metadata=1720028586,osd_pg_creating=573451927,osdmap=406672819}) 2024-02-12T16:25:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.720286+0000 mon.a (mon.0) 1990 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2252709668}) 2024-02-12T16:25:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.724234+0000 mon.a (mon.0) 1991 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3336193467}) 2024-02-12T16:25:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:04 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.738085+0000 mon.a (mon.0) 1992 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=69} crc {osdmap=1897974408}) 2024-02-12T16:25:04.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:04.508 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:04.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:04.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:04.671 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:04.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:04.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:25:04.786355 min lat: 0.000938475 max lat: 0.17982 avg lat: 0.00603775 2024-02-12T16:25:04.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:25:04.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 105988 105972 165.568 105.188 0.00790889 0.00603775 2024-02-12T16:25:04.802 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:04.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:04.931 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:04.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:05.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:05.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:05 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.314250+0000 osd.9 (osd.9) 95 : cluster [DBG] 1.a deep-scrub starts 2024-02-12T16:25:05.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:05 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.361746+0000 osd.9 (osd.9) 96 : cluster [DBG] 1.a deep-scrub ok 2024-02-12T16:25:05.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:05 smithi123 bash[19119]: cluster 2024-02-12T16:25:03.938330+0000 mgr.y (mgr.14146) 1200 : cluster [DBG] pgmap v1657: 37 pgs: 37 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-02-12T16:25:05.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:05.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:05 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.314250+0000 osd.9 (osd.9) 95 : cluster [DBG] 1.a deep-scrub starts 2024-02-12T16:25:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:05 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.361746+0000 osd.9 (osd.9) 96 : cluster [DBG] 1.a deep-scrub ok 2024-02-12T16:25:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:05 smithi169 bash[19604]: cluster 2024-02-12T16:25:03.938330+0000 mgr.y (mgr.14146) 1200 : cluster [DBG] pgmap v1657: 37 pgs: 37 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-02-12T16:25:05.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:05.331 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:05 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.314250+0000 osd.9 (osd.9) 95 : cluster [DBG] 1.a deep-scrub starts 2024-02-12T16:25:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:05 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.361746+0000 osd.9 (osd.9) 96 : cluster [DBG] 1.a deep-scrub ok 2024-02-12T16:25:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:05 smithi121 bash[18489]: cluster 2024-02-12T16:25:03.938330+0000 mgr.y (mgr.14146) 1200 : cluster [DBG] pgmap v1657: 37 pgs: 37 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-02-12T16:25:05.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:05.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:05.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:05.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:05.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:25:05.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:05.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:05.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 16 107593 107577 163.976 100.312 0.00984483 0.00609649 2024-02-12T16:25:05.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:05.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:05.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:05.985 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:05.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:06.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:06.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:06.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:06.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:06.254 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:06.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:06.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:06.384 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:06.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:06.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:06.518 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:06.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:06.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:06 smithi123 bash[19119]: cluster 2024-02-12T16:25:05.055200+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:06 smithi123 bash[19119]: cluster 2024-02-12T16:25:05.055249+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:06 smithi123 bash[19119]: cluster 2024-02-12T16:25:05.482582+0000 osd.2 (osd.2) 70 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:06 smithi123 bash[19119]: cluster 2024-02-12T16:25:05.516157+0000 osd.2 (osd.2) 71 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:06 smithi123 bash[19119]: cluster 2024-02-12T16:25:05.619937+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e565: 12 total, 12 up, 12 in 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:06 smithi123 bash[19119]: cluster 2024-02-12T16:25:05.938732+0000 mgr.y (mgr.14146) 1201 : cluster [DBG] pgmap v1659: 37 pgs: 37 active+clean; 6.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:06 smithi169 bash[19604]: cluster 2024-02-12T16:25:05.055200+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:06.628 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:06 smithi169 bash[19604]: cluster 2024-02-12T16:25:05.055249+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:06.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:06 smithi169 bash[19604]: cluster 2024-02-12T16:25:05.482582+0000 osd.2 (osd.2) 70 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:25:06.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:06 smithi169 bash[19604]: cluster 2024-02-12T16:25:05.516157+0000 osd.2 (osd.2) 71 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:25:06.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:06 smithi169 bash[19604]: cluster 2024-02-12T16:25:05.619937+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e565: 12 total, 12 up, 12 in 2024-02-12T16:25:06.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:06 smithi169 bash[19604]: cluster 2024-02-12T16:25:05.938732+0000 mgr.y (mgr.14146) 1201 : cluster [DBG] pgmap v1659: 37 pgs: 37 active+clean; 6.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:25:06.644 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:06.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:06.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:06.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:06.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:06.788 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 109213 109197 162.482 101.25 0.00632317 0.00615294 2024-02-12T16:25:06.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:06.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:06 smithi121 bash[18489]: cluster 2024-02-12T16:25:05.055200+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:06.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:06 smithi121 bash[18489]: cluster 2024-02-12T16:25:05.055249+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:06 smithi121 bash[18489]: cluster 2024-02-12T16:25:05.482582+0000 osd.2 (osd.2) 70 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:25:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:06 smithi121 bash[18489]: cluster 2024-02-12T16:25:05.516157+0000 osd.2 (osd.2) 71 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:25:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:06 smithi121 bash[18489]: cluster 2024-02-12T16:25:05.619937+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e565: 12 total, 12 up, 12 in 2024-02-12T16:25:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:06 smithi121 bash[18489]: cluster 2024-02-12T16:25:05.938732+0000 mgr.y (mgr.14146) 1201 : cluster [DBG] pgmap v1659: 37 pgs: 37 active+clean; 6.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:25:06.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:06.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:07.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:07.057 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:07.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:07.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:07.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:07.318 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:07.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:07.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:07.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:07.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:07.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:07 smithi123 bash[19119]: cluster 2024-02-12T16:25:07.140515+0000 osd.11 (osd.11) 124 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:25:07.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:07.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:07.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 15 110779 110764 160.981 97.9375 0.0187267 0.00621001 2024-02-12T16:25:07.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:25:07.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:07.824 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:07 smithi169 bash[19604]: cluster 2024-02-12T16:25:07.140515+0000 osd.11 (osd.11) 124 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:25:07.839 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:07 smithi121 bash[18489]: cluster 2024-02-12T16:25:07.140515+0000 osd.11 (osd.11) 124 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:25:07.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:07.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:07.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:08.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:08.115 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:08.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:08.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:08.254 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:08.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:08.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:08.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:08.386 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:08.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:08.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:08.524 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:08.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:08.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:08.667 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:25:08.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:08.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:08.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:08.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:08 smithi123 bash[19119]: cluster 2024-02-12T16:25:07.263683+0000 osd.11 (osd.11) 125 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:25:08.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:08 smithi123 bash[19119]: audit 2024-02-12T16:25:07.666843+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? v1:172.21.15.121:0/4242313767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:25:08.776 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:08 smithi123 bash[19119]: cluster 2024-02-12T16:25:07.939204+0000 mgr.y (mgr.14146) 1202 : cluster [DBG] pgmap v1660: 37 pgs: 37 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:25:08.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 112307 112291 159.491 95.4375 0.00471087 0.00626822 2024-02-12T16:25:08.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:08.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:08.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:08 smithi121 bash[18489]: cluster 2024-02-12T16:25:07.263683+0000 osd.11 (osd.11) 125 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:25:08.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:08 smithi121 bash[18489]: audit 2024-02-12T16:25:07.666843+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? v1:172.21.15.121:0/4242313767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:25:08.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:08 smithi121 bash[18489]: cluster 2024-02-12T16:25:07.939204+0000 mgr.y (mgr.14146) 1202 : cluster [DBG] pgmap v1660: 37 pgs: 37 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:25:08.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:08 smithi169 bash[19604]: cluster 2024-02-12T16:25:07.263683+0000 osd.11 (osd.11) 125 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:25:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:08 smithi169 bash[19604]: audit 2024-02-12T16:25:07.666843+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? v1:172.21.15.121:0/4242313767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:25:08.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:08 smithi169 bash[19604]: cluster 2024-02-12T16:25:07.939204+0000 mgr.y (mgr.14146) 1202 : cluster [DBG] pgmap v1660: 37 pgs: 37 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:25:08.940 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:08.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:09.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:09.069 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:09.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:09.173 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:09.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:09.212 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:09.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:09.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:09.435 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:09.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:09.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:09.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:09.604 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:09.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:09.636 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:09 smithi121 bash[18489]: cluster 2024-02-12T16:25:07.913865+0000 osd.10 (osd.10) 106 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:25:09.636 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:09 smithi121 bash[18489]: cluster 2024-02-12T16:25:07.944447+0000 osd.10 (osd.10) 107 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:25:09.636 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:09 smithi121 bash[18489]: cluster 2024-02-12T16:25:08.628227+0000 mon.a (mon.0) 1995 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:09.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:09.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:09.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:09.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 113937 113921 158.211 101.875 0.0067001 0.006319 2024-02-12T16:25:09.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:09.853 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:09 smithi123 bash[19119]: cluster 2024-02-12T16:25:07.913865+0000 osd.10 (osd.10) 106 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:25:09.853 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:09 smithi123 bash[19119]: cluster 2024-02-12T16:25:07.944447+0000 osd.10 (osd.10) 107 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:25:09.853 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:09 smithi123 bash[19119]: cluster 2024-02-12T16:25:08.628227+0000 mon.a (mon.0) 1995 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:09.853 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:09 smithi123 bash[19119]: audit 2024-02-12T16:25:08.638052+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.121:0/4242313767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:25:09.854 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:09 smithi123 bash[19119]: cluster 2024-02-12T16:25:08.638145+0000 mon.a (mon.0) 1997 : cluster [DBG] osdmap e566: 12 total, 12 up, 12 in 2024-02-12T16:25:09.873 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:09.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:09.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:09 smithi121 bash[18489]: audit 2024-02-12T16:25:08.638052+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.121:0/4242313767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:25:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:09 smithi121 bash[18489]: cluster 2024-02-12T16:25:08.638145+0000 mon.a (mon.0) 1997 : cluster [DBG] osdmap e566: 12 total, 12 up, 12 in 2024-02-12T16:25:09.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:09.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:09 smithi169 bash[19604]: cluster 2024-02-12T16:25:07.913865+0000 osd.10 (osd.10) 106 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:25:09.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:09 smithi169 bash[19604]: cluster 2024-02-12T16:25:07.944447+0000 osd.10 (osd.10) 107 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:25:09.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:09 smithi169 bash[19604]: cluster 2024-02-12T16:25:08.628227+0000 mon.a (mon.0) 1995 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:09.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:09 smithi169 bash[19604]: audit 2024-02-12T16:25:08.638052+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.121:0/4242313767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:25:09.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:09 smithi169 bash[19604]: cluster 2024-02-12T16:25:08.638145+0000 mon.a (mon.0) 1997 : cluster [DBG] osdmap e566: 12 total, 12 up, 12 in 2024-02-12T16:25:10.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:10.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:10.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:10.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:10.359 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:10.389 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:10.474 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:25:10.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:10.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.664 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:10.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:10.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 16 115483 115467 156.872 96.625 0.00461643 0.00637287 2024-02-12T16:25:10.802 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:10.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:10 smithi121 bash[18489]: cluster 2024-02-12T16:25:09.939700+0000 mgr.y (mgr.14146) 1203 : cluster [DBG] pgmap v1662: 37 pgs: 37 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-12T16:25:10.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:10 smithi121 bash[18489]: cluster 2024-02-12T16:25:10.622895+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e567: 12 total, 12 up, 12 in 2024-02-12T16:25:10.914 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:10 smithi123 bash[19119]: cluster 2024-02-12T16:25:09.939700+0000 mgr.y (mgr.14146) 1203 : cluster [DBG] pgmap v1662: 37 pgs: 37 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-12T16:25:10.914 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:10 smithi123 bash[19119]: cluster 2024-02-12T16:25:10.622895+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e567: 12 total, 12 up, 12 in 2024-02-12T16:25:10.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:10.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:11.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:10 smithi169 bash[19604]: cluster 2024-02-12T16:25:09.939700+0000 mgr.y (mgr.14146) 1203 : cluster [DBG] pgmap v1662: 37 pgs: 37 active+clean; 6.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-02-12T16:25:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:10 smithi169 bash[19604]: cluster 2024-02-12T16:25:10.622895+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e567: 12 total, 12 up, 12 in 2024-02-12T16:25:11.071 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:11.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:11.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:11.196 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:11.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:11.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:11.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:11.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:11.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:11.460 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:11.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:11.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:11.588 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:11.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:11.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:11.718 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:11.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:11.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:11.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 16 116993 116977 155.542 94.375 0.0134255 0.00642738 2024-02-12T16:25:11.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:11.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:11.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:11.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:12.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:12.028 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.955129+0000 mon.a (mon.0) 1999 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.955685+0000 mon.a (mon.0) 2000 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.955968+0000 mon.a (mon.0) 2001 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.956225+0000 mon.a (mon.0) 2002 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.956509+0000 mon.a (mon.0) 2003 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.956996+0000 mon.a (mon.0) 2004 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.957272+0000 mon.a (mon.0) 2005 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.957526+0000 mon.a (mon.0) 2006 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.957782+0000 mon.a (mon.0) 2007 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.958045+0000 mon.a (mon.0) 2008 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.958360+0000 mon.a (mon.0) 2009 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.958579+0000 mon.a (mon.0) 2010 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]: dispatch 2024-02-12T16:25:12.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:12 smithi169 bash[19604]: audit 2024-02-12T16:25:11.958896+0000 mon.a (mon.0) 2011 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]: dispatch 2024-02-12T16:25:12.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:25:12.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:12.139 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.955129+0000 mon.a (mon.0) 1999 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]: dispatch 2024-02-12T16:25:12.139 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.955685+0000 mon.a (mon.0) 2000 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]: dispatch 2024-02-12T16:25:12.139 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.955968+0000 mon.a (mon.0) 2001 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:12.139 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.956225+0000 mon.a (mon.0) 2002 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]: dispatch 2024-02-12T16:25:12.139 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.956509+0000 mon.a (mon.0) 2003 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.956996+0000 mon.a (mon.0) 2004 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.957272+0000 mon.a (mon.0) 2005 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.957526+0000 mon.a (mon.0) 2006 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.957782+0000 mon.a (mon.0) 2007 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.958045+0000 mon.a (mon.0) 2008 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.958360+0000 mon.a (mon.0) 2009 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.958579+0000 mon.a (mon.0) 2010 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]: dispatch 2024-02-12T16:25:12.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:12 smithi121 bash[18489]: audit 2024-02-12T16:25:11.958896+0000 mon.a (mon.0) 2011 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]: dispatch 2024-02-12T16:25:12.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:12.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:12.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:12.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.955129+0000 mon.a (mon.0) 1999 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]: dispatch 2024-02-12T16:25:12.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.955685+0000 mon.a (mon.0) 2000 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]: dispatch 2024-02-12T16:25:12.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.955968+0000 mon.a (mon.0) 2001 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:12.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.956225+0000 mon.a (mon.0) 2002 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.956509+0000 mon.a (mon.0) 2003 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.956996+0000 mon.a (mon.0) 2004 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.957272+0000 mon.a (mon.0) 2005 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.957526+0000 mon.a (mon.0) 2006 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.957782+0000 mon.a (mon.0) 2007 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.958045+0000 mon.a (mon.0) 2008 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.958360+0000 mon.a (mon.0) 2009 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.958579+0000 mon.a (mon.0) 2010 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]: dispatch 2024-02-12T16:25:12.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:12 smithi123 bash[19119]: audit 2024-02-12T16:25:11.958896+0000 mon.a (mon.0) 2011 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]: dispatch 2024-02-12T16:25:12.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:12.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:12.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:12.597 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:12.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:12.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:12.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:12.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 118455 118439 154.205 91.375 0.0101268 0.00648333 2024-02-12T16:25:12.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:12.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:12.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:13.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011434+0000 mon.a (mon.0) 2012 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011503+0000 mon.a (mon.0) 2013 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011548+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011592+0000 mon.a (mon.0) 2015 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011635+0000 mon.a (mon.0) 2016 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011677+0000 mon.a (mon.0) 2017 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011719+0000 mon.a (mon.0) 2018 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011759+0000 mon.a (mon.0) 2019 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011800+0000 mon.a (mon.0) 2020 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]': finished 2024-02-12T16:25:13.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011840+0000 mon.a (mon.0) 2021 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]': finished 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011882+0000 mon.a (mon.0) 2022 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]': finished 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011921+0000 mon.a (mon.0) 2023 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]': finished 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.011959+0000 mon.a (mon.0) 2024 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]': finished 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: cluster 2024-02-12T16:25:12.011990+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e568: 12 total, 12 up, 12 in 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.017756+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.018341+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.018621+0000 mon.a (mon.0) 2028 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.018752+0000 mon.a (mon.0) 2029 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.018949+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.019159+0000 mon.a (mon.0) 2031 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.019530+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.019773+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]: dispatch 2024-02-12T16:25:13.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.020027+0000 mon.a (mon.0) 2034 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.020285+0000 mon.a (mon.0) 2035 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.020435+0000 mon.a (mon.0) 2036 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.020632+0000 mon.a (mon.0) 2037 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.020844+0000 mon.a (mon.0) 2038 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.021047+0000 mon.a (mon.0) 2039 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.021255+0000 mon.a (mon.0) 2040 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.021467+0000 mon.a (mon.0) 2041 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.046256+0000 mon.a (mon.0) 2042 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.051552+0000 mon.a (mon.0) 2043 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.052617+0000 mon.a (mon.0) 2044 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:12.058480+0000 mon.a (mon.0) 2045 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.014683+0000 mon.a (mon.0) 2046 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]': finished 2024-02-12T16:25:13.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.014751+0000 mon.a (mon.0) 2047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]': finished 2024-02-12T16:25:13.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.014806+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.014854+0000 mon.a (mon.0) 2049 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]': finished 2024-02-12T16:25:13.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.014897+0000 mon.a (mon.0) 2050 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.014940+0000 mon.a (mon.0) 2051 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]': finished 2024-02-12T16:25:13.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.014982+0000 mon.a (mon.0) 2052 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]': finished 2024-02-12T16:25:13.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.015024+0000 mon.a (mon.0) 2053 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]': finished 2024-02-12T16:25:13.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.015294+0000 mon.a (mon.0) 2054 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]': finished 2024-02-12T16:25:13.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:13.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:13.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:13.199 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:13.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.015390+0000 mon.a (mon.0) 2055 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]': finished 2024-02-12T16:25:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.015532+0000 mon.a (mon.0) 2056 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]': finished 2024-02-12T16:25:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.015624+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]': finished 2024-02-12T16:25:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: audit 2024-02-12T16:25:13.015714+0000 mon.a (mon.0) 2058 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]': finished 2024-02-12T16:25:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:13 smithi169 bash[19604]: cluster 2024-02-12T16:25:13.015763+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e569: 12 total, 12 up, 12 in 2024-02-12T16:25:13.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:13.329 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:13.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011434+0000 mon.a (mon.0) 2012 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]': finished 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011503+0000 mon.a (mon.0) 2013 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]': finished 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011548+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011592+0000 mon.a (mon.0) 2015 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]': finished 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011635+0000 mon.a (mon.0) 2016 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011677+0000 mon.a (mon.0) 2017 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]': finished 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011719+0000 mon.a (mon.0) 2018 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]': finished 2024-02-12T16:25:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011759+0000 mon.a (mon.0) 2019 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]': finished 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011800+0000 mon.a (mon.0) 2020 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]': finished 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011840+0000 mon.a (mon.0) 2021 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]': finished 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011882+0000 mon.a (mon.0) 2022 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]': finished 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011921+0000 mon.a (mon.0) 2023 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]': finished 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.011959+0000 mon.a (mon.0) 2024 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]': finished 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: cluster 2024-02-12T16:25:12.011990+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e568: 12 total, 12 up, 12 in 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.017756+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.018341+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.018621+0000 mon.a (mon.0) 2028 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.018752+0000 mon.a (mon.0) 2029 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.018949+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.019159+0000 mon.a (mon.0) 2031 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.019530+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.019773+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]: dispatch 2024-02-12T16:25:13.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.020027+0000 mon.a (mon.0) 2034 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.020285+0000 mon.a (mon.0) 2035 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.020435+0000 mon.a (mon.0) 2036 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.020632+0000 mon.a (mon.0) 2037 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.020844+0000 mon.a (mon.0) 2038 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.021047+0000 mon.a (mon.0) 2039 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.021255+0000 mon.a (mon.0) 2040 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.021467+0000 mon.a (mon.0) 2041 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.046256+0000 mon.a (mon.0) 2042 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.051552+0000 mon.a (mon.0) 2043 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.052617+0000 mon.a (mon.0) 2044 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:12.058480+0000 mon.a (mon.0) 2045 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.014683+0000 mon.a (mon.0) 2046 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]': finished 2024-02-12T16:25:13.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.014751+0000 mon.a (mon.0) 2047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.014806+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.014854+0000 mon.a (mon.0) 2049 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.014897+0000 mon.a (mon.0) 2050 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.014940+0000 mon.a (mon.0) 2051 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.014982+0000 mon.a (mon.0) 2052 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.015024+0000 mon.a (mon.0) 2053 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.015294+0000 mon.a (mon.0) 2054 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.015390+0000 mon.a (mon.0) 2055 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.015532+0000 mon.a (mon.0) 2056 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.015624+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: audit 2024-02-12T16:25:13.015714+0000 mon.a (mon.0) 2058 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]': finished 2024-02-12T16:25:13.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:13 smithi123 bash[19119]: cluster 2024-02-12T16:25:13.015763+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e569: 12 total, 12 up, 12 in 2024-02-12T16:25:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011434+0000 mon.a (mon.0) 2012 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]': finished 2024-02-12T16:25:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011503+0000 mon.a (mon.0) 2013 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]': finished 2024-02-12T16:25:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011548+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011592+0000 mon.a (mon.0) 2015 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]': finished 2024-02-12T16:25:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011635+0000 mon.a (mon.0) 2016 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011677+0000 mon.a (mon.0) 2017 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]': finished 2024-02-12T16:25:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011719+0000 mon.a (mon.0) 2018 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]': finished 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011759+0000 mon.a (mon.0) 2019 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]': finished 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011800+0000 mon.a (mon.0) 2020 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]': finished 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011840+0000 mon.a (mon.0) 2021 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]': finished 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011882+0000 mon.a (mon.0) 2022 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]': finished 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011921+0000 mon.a (mon.0) 2023 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]': finished 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.011959+0000 mon.a (mon.0) 2024 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]': finished 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: cluster 2024-02-12T16:25:12.011990+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e568: 12 total, 12 up, 12 in 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.017756+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]: dispatch 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.018341+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]: dispatch 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.018621+0000 mon.a (mon.0) 2028 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.018752+0000 mon.a (mon.0) 2029 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]: dispatch 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.018949+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]: dispatch 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.019159+0000 mon.a (mon.0) 2031 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]: dispatch 2024-02-12T16:25:13.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.019530+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.019773+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.020027+0000 mon.a (mon.0) 2034 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.020285+0000 mon.a (mon.0) 2035 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.020435+0000 mon.a (mon.0) 2036 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.020632+0000 mon.a (mon.0) 2037 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.020844+0000 mon.a (mon.0) 2038 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.021047+0000 mon.a (mon.0) 2039 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.021255+0000 mon.a (mon.0) 2040 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.021467+0000 mon.a (mon.0) 2041 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.046256+0000 mon.a (mon.0) 2042 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.051552+0000 mon.a (mon.0) 2043 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.052617+0000 mon.a (mon.0) 2044 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:12.058480+0000 mon.a (mon.0) 2045 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:25:13.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.014683+0000 mon.a (mon.0) 2046 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08532095511831929]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.014751+0000 mon.a (mon.0) 2047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08834231359287897]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.014806+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.014854+0000 mon.a (mon.0) 2049 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09269547692484358]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.014897+0000 mon.a (mon.0) 2050 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08780937127119486]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.014940+0000 mon.a (mon.0) 2051 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08562084626157476]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.014982+0000 mon.a (mon.0) 2052 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08674607294000826]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.015024+0000 mon.a (mon.0) 2053 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08671571029634707]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.015294+0000 mon.a (mon.0) 2054 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09174161860994628]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.015390+0000 mon.a (mon.0) 2055 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08670052897451645]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.015532+0000 mon.a (mon.0) 2056 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08514782158550842]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.015624+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0828962998724172]}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: audit 2024-02-12T16:25:13.015714+0000 mon.a (mon.0) 2058 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"10\": \"51897\", \"11\": \"24291\"}"}]': finished 2024-02-12T16:25:13.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:13 smithi121 bash[18489]: cluster 2024-02-12T16:25:13.015763+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e569: 12 total, 12 up, 12 in 2024-02-12T16:25:13.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:13.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:13.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:13.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:13.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:13.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:13.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:13.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:13.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 119912 119896 152.916 91.0625 0.0106901 0.00653786 2024-02-12T16:25:13.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:13.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:13.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:13.996 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:14.032 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:14.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:14.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:14.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:14.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:14.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:14.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:14.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:14.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:14.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:14.699 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:14.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 121348 121332 151.653 89.75 0.0179107 0.00659213 2024-02-12T16:25:14.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:14.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:14.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:14.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:15.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:15.118 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:15.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:15.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:15 smithi121 bash[18489]: cluster 2024-02-12T16:25:13.942632+0000 mgr.y (mgr.14146) 1205 : cluster [DBG] pgmap v1667: 37 pgs: 37 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-12T16:25:15.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:15.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:15.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:15 smithi169 bash[19604]: cluster 2024-02-12T16:25:13.942632+0000 mgr.y (mgr.14146) 1205 : cluster [DBG] pgmap v1667: 37 pgs: 37 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-12T16:25:15.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:15.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:15.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:15 smithi123 bash[19119]: cluster 2024-02-12T16:25:13.942632+0000 mgr.y (mgr.14146) 1205 : cluster [DBG] pgmap v1667: 37 pgs: 37 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-02-12T16:25:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:15.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:15.494 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:15.520 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:15.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:15.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:15.699 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:15.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:15.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 122776 122760 150.429 89.25 0.0135513 0.00664618 2024-02-12T16:25:15.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:15.827 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:15.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:15.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:15.952 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:15.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:16.048 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T16:25:16.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:16.084 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:16.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:16.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:16.195 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:16 smithi123 bash[19119]: audit 2024-02-12T16:25:15.137377+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? v1:172.21.15.121:0/2455641543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:25:16.211 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:16.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:16 smithi169 bash[19604]: audit 2024-02-12T16:25:15.137377+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? v1:172.21.15.121:0/2455641543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:25:16.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:16.321 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:16 smithi121 bash[18489]: audit 2024-02-12T16:25:15.137377+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? v1:172.21.15.121:0/2455641543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:25:16.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:16.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:16.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:16.475 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:16.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:16.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:16.597 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:25:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:25:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.16095.0:123910 5.0 e570/547 5:0bfa07de:::benchmark_data_smithi169_84712_object123909:head v 570'7750, mlcod=570'7745)", 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:16.589595+0000", 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0049744300000000002, 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050048660000000002, 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589595+0000", 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589595+0000", 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589594+0000", 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9051000000000003e-05 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589633+0000", 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5384999999999999e-05 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589648+0000", 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7429999999999996e-06 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589655+0000", 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6947999999999999e-05 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589692+0000", 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0639e-05 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589712+0000", 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117039 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.16095.0:123924 5.f e570/547 5:fb243766:::benchmark_data_smithi169_84712_object123923:head v 570'7635, mlcod=570'7633)", 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:16.589792+0000", 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0047778289999999999, 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048786910000000001, 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:25:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589792+0000", 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589792+0000", 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589790+0000", 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4916999999999999e-05 2024-02-12T16:25:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589815+0000", 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0131e-05 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589825+0000", 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4520000000000002e-06 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589832+0000", 2024-02-12T16:25:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8708000000000001e-05 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589890+0000", 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7594000000000001e-05 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:16.589908+0000", 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011607099999999999 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:25:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 2 2024-02-12T16:25:16.605 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:25:16.621 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:16.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:16.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:16.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:16.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:16.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:16.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 16 124192 124176 149.238 88.5 0.00805553 0.00669933 2024-02-12T16:25:16.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:16.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:16.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:16.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:17.022 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:17.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:17.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:17.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:17.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:17 smithi169 bash[19604]: cluster 2024-02-12T16:25:15.943095+0000 mgr.y (mgr.14146) 1206 : cluster [DBG] pgmap v1668: 37 pgs: 37 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:25:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:17 smithi169 bash[19604]: audit 2024-02-12T16:25:16.038828+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? v1:172.21.15.121:0/2455641543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:25:17.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:17 smithi169 bash[19604]: cluster 2024-02-12T16:25:16.038900+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e570: 12 total, 12 up, 12 in 2024-02-12T16:25:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.301 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:17 smithi121 bash[18489]: cluster 2024-02-12T16:25:15.943095+0000 mgr.y (mgr.14146) 1206 : cluster [DBG] pgmap v1668: 37 pgs: 37 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:25:17.301 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:17 smithi121 bash[18489]: audit 2024-02-12T16:25:16.038828+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? v1:172.21.15.121:0/2455641543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:25:17.301 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:17 smithi121 bash[18489]: cluster 2024-02-12T16:25:16.038900+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e570: 12 total, 12 up, 12 in 2024-02-12T16:25:17.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:17.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T16:25:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:17 smithi123 bash[19119]: cluster 2024-02-12T16:25:15.943095+0000 mgr.y (mgr.14146) 1206 : cluster [DBG] pgmap v1668: 37 pgs: 37 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-02-12T16:25:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:17 smithi123 bash[19119]: audit 2024-02-12T16:25:16.038828+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? v1:172.21.15.121:0/2455641543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:25:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:17 smithi123 bash[19119]: cluster 2024-02-12T16:25:16.038900+0000 mon.a (mon.0) 2062 : cluster [DBG] osdmap e570: 12 total, 12 up, 12 in 2024-02-12T16:25:17.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:17.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:25:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:17.536 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:17.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:17.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:17.785 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:25:17.789 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 125597 125581 148.079 87.8125 0.00855225 0.00675161 2024-02-12T16:25:17.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:17.814 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:25:17.814 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:17.829 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:17.851 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:25:17.886 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:25:17.916 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:25:17.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:17.943 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:25:17.957 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:17.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:17.976 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:25:18.006 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:25:18.038 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:25:18.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:18.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:25:18.085 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:18.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:18.103 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:25:18.129 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:25:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.133 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:25:18.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:18.196 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:18 smithi121 bash[18489]: audit 2024-02-12T16:25:17.745999+0000 mon.a (mon.0) 2063 : audit [DBG] from='client.? v1:172.21.15.121:0/2574240727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:18.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:18.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:18.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:18.325 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:18 smithi123 bash[19119]: audit 2024-02-12T16:25:17.745999+0000 mon.a (mon.0) 2063 : audit [DBG] from='client.? v1:172.21.15.121:0/2574240727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:18.343 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:18.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:18.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:18.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:18.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:18.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:18 smithi169 bash[19604]: audit 2024-02-12T16:25:17.745999+0000 mon.a (mon.0) 2063 : audit [DBG] from='client.? v1:172.21.15.121:0/2574240727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:18.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:18.596 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:18.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:18.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:18.738 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:18.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:18.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 126880 126864 146.821 80.1875 0.00943297 0.0068098 2024-02-12T16:25:18.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:18.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:18.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:18.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:19.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:19.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:19.135 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:19.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:19 smithi121 bash[18489]: cluster 2024-02-12T16:25:17.947478+0000 mgr.y (mgr.14146) 1207 : cluster [DBG] pgmap v1670: 37 pgs: 37 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:25:19.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:19.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:19.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:19.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:19 smithi123 bash[19119]: cluster 2024-02-12T16:25:17.947478+0000 mgr.y (mgr.14146) 1207 : cluster [DBG] pgmap v1670: 37 pgs: 37 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:25:19.394 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:19.501 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:19.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:19 smithi169 bash[19604]: cluster 2024-02-12T16:25:17.947478+0000 mgr.y (mgr.14146) 1207 : cluster [DBG] pgmap v1670: 37 pgs: 37 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:25:19.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:19.661 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:19.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:19.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 128324 128308 145.793 90.25 0.00687818 0.00685752 2024-02-12T16:25:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:19.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:19.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:19.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:20.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:20.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:20.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:20.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:20.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:20 smithi123 bash[19119]: cluster 2024-02-12T16:25:18.572923+0000 osd.11 (osd.11) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:20.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:20 smithi123 bash[19119]: cluster 2024-02-12T16:25:18.572988+0000 osd.11 (osd.11) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:20.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:20 smithi123 bash[19119]: cluster 2024-02-12T16:25:19.054173+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e571: 12 total, 12 up, 12 in 2024-02-12T16:25:20.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:20.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:20.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:20.320 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:20 smithi169 bash[19604]: cluster 2024-02-12T16:25:18.572923+0000 osd.11 (osd.11) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:20.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:20 smithi169 bash[19604]: cluster 2024-02-12T16:25:18.572988+0000 osd.11 (osd.11) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:20.321 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:20 smithi169 bash[19604]: cluster 2024-02-12T16:25:19.054173+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e571: 12 total, 12 up, 12 in 2024-02-12T16:25:20.339 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:20.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:20.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:20 smithi121 bash[18489]: cluster 2024-02-12T16:25:18.572923+0000 osd.11 (osd.11) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:20 smithi121 bash[18489]: cluster 2024-02-12T16:25:18.572988+0000 osd.11 (osd.11) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:20 smithi121 bash[18489]: cluster 2024-02-12T16:25:19.054173+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e571: 12 total, 12 up, 12 in 2024-02-12T16:25:20.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:20.464 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:20.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:20.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:20.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:20.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:20.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:20.739 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:20.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:20.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 129745 129729 144.775 88.8125 0.00926776 0.00690529 2024-02-12T16:25:20.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:20.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:20.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:20.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:21.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:21.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:21.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:21.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:21.240 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:21 smithi169 bash[19604]: cluster 2024-02-12T16:25:19.947913+0000 mgr.y (mgr.14146) 1208 : cluster [DBG] pgmap v1672: 37 pgs: 37 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-12T16:25:21.240 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:21 smithi169 bash[19604]: cluster 2024-02-12T16:25:20.298005+0000 osd.10 (osd.10) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:21.241 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:21 smithi169 bash[19604]: cluster 2024-02-12T16:25:20.298061+0000 osd.10 (osd.10) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:21.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:21.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:21.372 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:21 smithi121 bash[18489]: cluster 2024-02-12T16:25:19.947913+0000 mgr.y (mgr.14146) 1208 : cluster [DBG] pgmap v1672: 37 pgs: 37 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-12T16:25:21.373 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:21 smithi121 bash[18489]: cluster 2024-02-12T16:25:20.298005+0000 osd.10 (osd.10) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:21.373 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:21 smithi121 bash[18489]: cluster 2024-02-12T16:25:20.298061+0000 osd.10 (osd.10) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:21 smithi123 bash[19119]: cluster 2024-02-12T16:25:19.947913+0000 mgr.y (mgr.14146) 1208 : cluster [DBG] pgmap v1672: 37 pgs: 37 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-12T16:25:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:21 smithi123 bash[19119]: cluster 2024-02-12T16:25:20.298005+0000 osd.10 (osd.10) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:21 smithi123 bash[19119]: cluster 2024-02-12T16:25:20.298061+0000 osd.10 (osd.10) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:21.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:21.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:21.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:21.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:25:21.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:21.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:21.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:21.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 131153 131137 143.779 88 0.0109102 0.00695372 2024-02-12T16:25:21.808 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:21.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:21.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:21.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:22.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:22.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:22.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:22.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:22.206 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:22.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:22.279 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:25:22.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:22.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:22 smithi121 bash[18489]: cluster 2024-02-12T16:25:21.067604+0000 mon.a (mon.0) 2065 : cluster [DBG] osdmap e572: 12 total, 12 up, 12 in 2024-02-12T16:25:22.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:22.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:22.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:22.451 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:22 smithi123 bash[19119]: cluster 2024-02-12T16:25:21.067604+0000 mon.a (mon.0) 2065 : cluster [DBG] osdmap e572: 12 total, 12 up, 12 in 2024-02-12T16:25:22.499 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:22.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:22.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:22 smithi169 bash[19604]: cluster 2024-02-12T16:25:21.067604+0000 mon.a (mon.0) 2065 : cluster [DBG] osdmap e572: 12 total, 12 up, 12 in 2024-02-12T16:25:22.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:22.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:22.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:22.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:22.770 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:22.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:22.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 16 132479 132463 142.729 82.875 0.0134019 0.0070045 2024-02-12T16:25:22.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:22.895 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:22.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:22.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:23.019 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:23.095 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:25:23.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:23.148 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:23.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:23.258 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:23 smithi169 bash[19604]: cluster 2024-02-12T16:25:21.948365+0000 mgr.y (mgr.14146) 1209 : cluster [DBG] pgmap v1674: 37 pgs: 37 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:25:23.259 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:23 smithi169 bash[19604]: audit 2024-02-12T16:25:22.850515+0000 mon.a (mon.0) 2066 : audit [INF] from='client.26045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:23.259 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:23 smithi169 bash[19604]: audit 2024-02-12T16:25:22.851860+0000 mon.b (mon.2) 137 : audit [INF] from='client.? v1:172.21.15.121:0/4055599583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:23.278 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:23.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:23 smithi123 bash[19119]: cluster 2024-02-12T16:25:21.948365+0000 mgr.y (mgr.14146) 1209 : cluster [DBG] pgmap v1674: 37 pgs: 37 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:25:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:23 smithi123 bash[19119]: audit 2024-02-12T16:25:22.850515+0000 mon.a (mon.0) 2066 : audit [INF] from='client.26045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:23 smithi123 bash[19119]: audit 2024-02-12T16:25:22.851860+0000 mon.b (mon.2) 137 : audit [INF] from='client.? v1:172.21.15.121:0/4055599583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:23.407 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:23 smithi121 bash[18489]: cluster 2024-02-12T16:25:21.948365+0000 mgr.y (mgr.14146) 1209 : cluster [DBG] pgmap v1674: 37 pgs: 37 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:25:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:23 smithi121 bash[18489]: audit 2024-02-12T16:25:22.850515+0000 mon.a (mon.0) 2066 : audit [INF] from='client.26045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:23 smithi121 bash[18489]: audit 2024-02-12T16:25:22.851860+0000 mon.b (mon.2) 137 : audit [INF] from='client.? v1:172.21.15.121:0/4055599583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:23.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:23.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:23.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:23.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:23.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:23.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:23.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 133748 133732 141.654 79.3125 0.00443078 0.00705786 2024-02-12T16:25:23.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:23.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:23.950 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:23.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:24.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:24.084 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:24.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:24.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:24.217 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:24.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:24.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:24.328 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:24 smithi123 bash[19119]: cluster 2024-02-12T16:25:22.758863+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:24.328 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:24 smithi123 bash[19119]: cluster 2024-02-12T16:25:22.758910+0000 osd.6 (osd.6) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:24.328 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:24 smithi123 bash[19119]: cluster 2024-02-12T16:25:23.078860+0000 mon.a (mon.0) 2067 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:24.328 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:24 smithi123 bash[19119]: audit 2024-02-12T16:25:23.086668+0000 mon.a (mon.0) 2068 : audit [INF] from='client.26045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:25:24.328 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:24 smithi123 bash[19119]: cluster 2024-02-12T16:25:23.086717+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e573: 12 total, 12 up, 12 in 2024-02-12T16:25:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:24.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:24.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:24 smithi121 bash[18489]: cluster 2024-02-12T16:25:22.758863+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:24.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:24 smithi121 bash[18489]: cluster 2024-02-12T16:25:22.758910+0000 osd.6 (osd.6) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:24.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:24 smithi121 bash[18489]: cluster 2024-02-12T16:25:23.078860+0000 mon.a (mon.0) 2067 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:24.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:24 smithi121 bash[18489]: audit 2024-02-12T16:25:23.086668+0000 mon.a (mon.0) 2068 : audit [INF] from='client.26045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:25:24.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:24 smithi121 bash[18489]: cluster 2024-02-12T16:25:23.086717+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e573: 12 total, 12 up, 12 in 2024-02-12T16:25:24.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:24.468 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:24 smithi169 bash[19604]: cluster 2024-02-12T16:25:22.758863+0000 osd.6 (osd.6) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:24.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:24 smithi169 bash[19604]: cluster 2024-02-12T16:25:22.758910+0000 osd.6 (osd.6) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:24.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:24 smithi169 bash[19604]: cluster 2024-02-12T16:25:23.078860+0000 mon.a (mon.0) 2067 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:24.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:24 smithi169 bash[19604]: audit 2024-02-12T16:25:23.086668+0000 mon.a (mon.0) 2068 : audit [INF] from='client.26045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:25:24.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:24 smithi169 bash[19604]: cluster 2024-02-12T16:25:23.086717+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e573: 12 total, 12 up, 12 in 2024-02-12T16:25:24.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:24.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:24.531 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:24.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:24.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:24.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:24.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:24.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:24.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:24.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:25:24.787983 min lat: 0.000938475 max lat: 0.227554 avg lat: 0.00710418 2024-02-12T16:25:24.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:25:24.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 135132 135116 140.734 86.5 0.0152002 0.00710418 2024-02-12T16:25:24.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:24.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:24.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:24.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:25.013 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:25.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:25 smithi121 bash[18489]: cluster 2024-02-12T16:25:23.948798+0000 mgr.y (mgr.14146) 1210 : cluster [DBG] pgmap v1676: 37 pgs: 37 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:25:25.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:25.123 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:25 smithi169 bash[19604]: cluster 2024-02-12T16:25:23.948798+0000 mgr.y (mgr.14146) 1210 : cluster [DBG] pgmap v1676: 37 pgs: 37 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:25:25.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:25.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:25.271 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:25.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:25 smithi123 bash[19119]: cluster 2024-02-12T16:25:23.948798+0000 mgr.y (mgr.14146) 1210 : cluster [DBG] pgmap v1676: 37 pgs: 37 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:25:25.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:25.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:25.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:25.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:25.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:25.771 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:25.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 136468 136452 139.796 83.5 0.0172381 0.00715091 2024-02-12T16:25:25.797 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:25.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:25.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:25.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:25.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:26.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:26.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:26.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:26.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:25:26.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:26.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:26.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:26.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:26.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:26.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:26.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:26.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:26.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:26.625 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:26.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:26.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:26.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:26.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:26.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 137785 137769 138.869 82.3125 0.0205084 0.00719972 2024-02-12T16:25:26.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:26.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:26.850 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:25:26.850 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:25:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:26 smithi123 bash[19119]: cluster 2024-02-12T16:25:25.949264+0000 mgr.y (mgr.14146) 1211 : cluster [DBG] pgmap v1677: 37 pgs: 37 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:25:26.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:26.902 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:26 smithi121 bash[18489]: cluster 2024-02-12T16:25:25.949264+0000 mgr.y (mgr.14146) 1211 : cluster [DBG] pgmap v1677: 37 pgs: 37 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:25:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:26.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:27.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:26 smithi169 bash[19604]: cluster 2024-02-12T16:25:25.949264+0000 mgr.y (mgr.14146) 1211 : cluster [DBG] pgmap v1677: 37 pgs: 37 active+clean; 8.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:25:27.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:27.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:27.180 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:27.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:27.318 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:27.425 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:27.452 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:27.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:27.578 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:27.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:27.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:27.722 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:25:27.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 16 139138 139122 138.007 84.5625 0.0203856 0.0072446 2024-02-12T16:25:27.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:27.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:27.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:27.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:27.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:28.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:28.107 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:28.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:28.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:28.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:28.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:28.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:28.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:28.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:28.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:28.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:28.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:28.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:28.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:28.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:28.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:28.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:28.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 140466 140450 137.147 83 0.00912491 0.00728962 2024-02-12T16:25:28.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:28.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:28.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:28.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:28.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:29.033 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:29.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:29.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:29.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:29 smithi123 bash[19119]: cluster 2024-02-12T16:25:27.949726+0000 mgr.y (mgr.14146) 1212 : cluster [DBG] pgmap v1678: 37 pgs: 37 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:25:29.167 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:29.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:29.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:29 smithi121 bash[18489]: cluster 2024-02-12T16:25:27.949726+0000 mgr.y (mgr.14146) 1212 : cluster [DBG] pgmap v1678: 37 pgs: 37 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:25:29.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:29 smithi169 bash[19604]: cluster 2024-02-12T16:25:27.949726+0000 mgr.y (mgr.14146) 1212 : cluster [DBG] pgmap v1678: 37 pgs: 37 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-02-12T16:25:29.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:29.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:29.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:29.443 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:29.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:29.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:29.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:29.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:29.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:29.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:29.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:29.790 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 141719 141703 136.242 78.3125 0.0050128 0.00733806 2024-02-12T16:25:29.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:29.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:29.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:29.973 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:30.019 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T16:25:30.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:30.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:30 smithi169 bash[19604]: audit 2024-02-12T16:25:29.733427+0000 mon.a (mon.0) 2070 : audit [INF] from='client.25978 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:25:30.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:30 smithi169 bash[19604]: audit 2024-02-12T16:25:29.734894+0000 mon.c (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.121:0/2603990261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:25:30.132 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:30.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:30.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:30.276 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:30 smithi121 bash[18489]: audit 2024-02-12T16:25:29.733427+0000 mon.a (mon.0) 2070 : audit [INF] from='client.25978 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:25:30.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:30 smithi121 bash[18489]: audit 2024-02-12T16:25:29.734894+0000 mon.c (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.121:0/2603990261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:25:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:30.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:30 smithi123 bash[19119]: audit 2024-02-12T16:25:29.733427+0000 mon.a (mon.0) 2070 : audit [INF] from='client.25978 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:25:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:30 smithi123 bash[19119]: audit 2024-02-12T16:25:29.734894+0000 mon.c (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.121:0/2603990261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:25:30.404 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:30.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:30.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:30.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:30.668 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:30.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:30.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 143037 143021 135.426 82.375 0.0106859 0.00738225 2024-02-12T16:25:30.803 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:30.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:30.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:30.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:31.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:31.068 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:31.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:31.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:31 smithi121 bash[18489]: cluster 2024-02-12T16:25:29.950168+0000 mgr.y (mgr.14146) 1213 : cluster [DBG] pgmap v1679: 37 pgs: 37 active+clean; 8.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-12T16:25:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:31 smithi121 bash[18489]: audit 2024-02-12T16:25:30.011411+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25978 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:25:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:31 smithi121 bash[18489]: cluster 2024-02-12T16:25:30.011478+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e574: 12 total, 12 up, 12 in 2024-02-12T16:25:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:31 smithi121 bash[18489]: cluster 2024-02-12T16:25:30.204012+0000 osd.0 (osd.0) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:31 smithi121 bash[18489]: cluster 2024-02-12T16:25:30.204071+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:31.173 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:31.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:31.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:31 smithi169 bash[19604]: cluster 2024-02-12T16:25:29.950168+0000 mgr.y (mgr.14146) 1213 : cluster [DBG] pgmap v1679: 37 pgs: 37 active+clean; 8.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-12T16:25:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:31 smithi169 bash[19604]: audit 2024-02-12T16:25:30.011411+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25978 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:25:31.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:31 smithi169 bash[19604]: cluster 2024-02-12T16:25:30.011478+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e574: 12 total, 12 up, 12 in 2024-02-12T16:25:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:31 smithi169 bash[19604]: cluster 2024-02-12T16:25:30.204012+0000 osd.0 (osd.0) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:31 smithi169 bash[19604]: cluster 2024-02-12T16:25:30.204071+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:31.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:31.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 9 1 2024-02-12T16:25:31.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:31.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:31 smithi123 bash[19119]: cluster 2024-02-12T16:25:29.950168+0000 mgr.y (mgr.14146) 1213 : cluster [DBG] pgmap v1679: 37 pgs: 37 active+clean; 8.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-02-12T16:25:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:31 smithi123 bash[19119]: audit 2024-02-12T16:25:30.011411+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25978 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:25:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:31 smithi123 bash[19119]: cluster 2024-02-12T16:25:30.011478+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e574: 12 total, 12 up, 12 in 2024-02-12T16:25:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:31 smithi123 bash[19119]: cluster 2024-02-12T16:25:30.204012+0000 osd.0 (osd.0) 90 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:31 smithi123 bash[19119]: cluster 2024-02-12T16:25:30.204071+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:31.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:31.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:31.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:31.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:31.613 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:31.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:31.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:31.746 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:31.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:31.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 144305 144289 134.587 79.25 0.0106627 0.00742859 2024-02-12T16:25:31.814 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:31.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:31.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:31.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:31.959 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:25:31.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:31.994 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:25:32.007 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:32.019 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:25:32.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:32 smithi169 bash[19604]: cluster 2024-02-12T16:25:31.025253+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e575: 12 total, 12 up, 12 in 2024-02-12T16:25:32.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:32 smithi169 bash[19604]: audit 2024-02-12T16:25:31.910310+0000 mon.a (mon.0) 2074 : audit [DBG] from='client.? v1:172.21.15.121:0/1027621749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:32.055 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:25:32.083 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:25:32.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:32.115 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:25:32.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:32 smithi123 bash[19119]: cluster 2024-02-12T16:25:31.025253+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e575: 12 total, 12 up, 12 in 2024-02-12T16:25:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:32 smithi123 bash[19119]: audit 2024-02-12T16:25:31.910310+0000 mon.a (mon.0) 2074 : audit [DBG] from='client.? v1:172.21.15.121:0/1027621749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:32.146 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:25:32.178 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:25:32.207 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:25:32.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:25:32.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:32.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:32.279 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:25:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:32.304 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:25:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:25:32.374 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:32 smithi121 bash[18489]: cluster 2024-02-12T16:25:31.025253+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e575: 12 total, 12 up, 12 in 2024-02-12T16:25:32.374 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:32 smithi121 bash[18489]: audit 2024-02-12T16:25:31.910310+0000 mon.a (mon.0) 2074 : audit [DBG] from='client.? v1:172.21.15.121:0/1027621749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:32.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:32.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:32.513 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:25:32.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:32.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:32.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:32.699 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:32.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 16 145566 145550 133.767 78.8125 0.00516318 0.00747376 2024-02-12T16:25:32.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:32.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:32.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:32.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:32.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:33.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:33.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:33 smithi121 bash[18489]: cluster 2024-02-12T16:25:31.950614+0000 mgr.y (mgr.14146) 1214 : cluster [DBG] pgmap v1682: 37 pgs: 37 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-02-12T16:25:33.113 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:33.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:33.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:33.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:33.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:33 smithi169 bash[19604]: cluster 2024-02-12T16:25:31.950614+0000 mgr.y (mgr.14146) 1214 : cluster [DBG] pgmap v1682: 37 pgs: 37 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-02-12T16:25:33.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:33 smithi123 bash[19119]: cluster 2024-02-12T16:25:31.950614+0000 mgr.y (mgr.14146) 1214 : cluster [DBG] pgmap v1682: 37 pgs: 37 active+clean; 8.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-02-12T16:25:33.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:33.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:33.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:33.527 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:33.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:33.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:33.666 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:33.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:33.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:33.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 146865 146849 133.005 81.1875 0.00812181 0.00751672 2024-02-12T16:25:33.803 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:33.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:33.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:33.934 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:33.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:34.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:34.066 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:34.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:34.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:34.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:34.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:34.303 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:34.327 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:34.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:34.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:34.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:34.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:34.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:34.600 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:34.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:34.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:34.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:34.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:34.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 16 148128 148112 132.232 78.9375 0.0140764 0.00756099 2024-02-12T16:25:34.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:34.868 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:34.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:34.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:35.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:35.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:35.122 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:35 smithi169 bash[19604]: cluster 2024-02-12T16:25:33.951105+0000 mgr.y (mgr.14146) 1215 : cluster [DBG] pgmap v1683: 37 pgs: 37 active+clean; 8.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-12T16:25:35.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:35.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:35.248 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:35 smithi121 bash[18489]: cluster 2024-02-12T16:25:33.951105+0000 mgr.y (mgr.14146) 1215 : cluster [DBG] pgmap v1683: 37 pgs: 37 active+clean; 8.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-12T16:25:35.278 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:35.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:35.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:25:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:35 smithi123 bash[19119]: cluster 2024-02-12T16:25:33.951105+0000 mgr.y (mgr.14146) 1215 : cluster [DBG] pgmap v1683: 37 pgs: 37 active+clean; 8.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-02-12T16:25:35.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:35.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:35.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:35.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:35.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:35.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:35.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 149395 149379 131.485 79.1875 0.00984862 0.00760388 2024-02-12T16:25:35.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:35.868 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:35.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:35.957 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:35.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:36.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:36.128 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:36.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:36.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:36.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:36.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:36.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:36.391 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:36.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:36.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:36.523 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:36.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:36.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:36.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:36.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:36.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:36.771 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:36 smithi169 bash[19604]: cluster 2024-02-12T16:25:35.951586+0000 mgr.y (mgr.14146) 1216 : cluster [DBG] pgmap v1684: 37 pgs: 37 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-12T16:25:36.788 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:36.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 150657 150641 130.754 78.875 0.00582376 0.00764668 2024-02-12T16:25:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:36.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:36 smithi123 bash[19119]: cluster 2024-02-12T16:25:35.951586+0000 mgr.y (mgr.14146) 1216 : cluster [DBG] pgmap v1684: 37 pgs: 37 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-12T16:25:36.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:36.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:36 smithi121 bash[18489]: cluster 2024-02-12T16:25:35.951586+0000 mgr.y (mgr.14146) 1216 : cluster [DBG] pgmap v1684: 37 pgs: 37 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:327436 4.c 4:3f027441:::benchmark_data_smithi169_60938_object154099:head [delete] snapc 0=[] ondisk+write+known_if_redirected e529)", 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:17.599273+0000", 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 139.32072173500001, 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.051880887000000001, 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.930 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 327436 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.599273+0000", 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.599273+0000", 2024-02-12T16:25:36.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.599271+0000", 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3499999999999999e-06 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.599278+0000", 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0013999999999999e-05 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.599288+0000", 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9819999999999996e-06 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.599294+0000", 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021996544999999999 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.621291+0000", 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102981 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.621394+0000", 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013472399999999999 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.621528+0000", 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014991852 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.636520+0000", 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014560358000000001 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.651081+0000", 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4408e-05 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.651115+0000", 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8504000000000003e-05 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:17.651154+0000", 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.051880887000000001 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.935 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:328729 4.c 4:3a04c3c6:::benchmark_data_smithi169_60938_object155392:head [delete] snapc 0=[] ondisk+write+known_if_redirected e529)", 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:18.748878+0000", 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 138.17111609700001, 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.069702781000000005, 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 328729 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.936 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.748878+0000", 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.748878+0000", 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.748877+0000", 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8320000000000002e-06 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.748883+0000", 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1647999999999999e-05 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.748894+0000", 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3549999999999998e-06 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.748899+0000", 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2324999999999999e-05 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.748941+0000", 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9827999999999995e-05 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.749021+0000", 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349000000000002e-05 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.749105+0000", 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041461919999999999 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.753251+0000", 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.065245639999999994 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.818497+0000", 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5331999999999999e-05 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.818542+0000", 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8872999999999999e-05 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:18.818581+0000", 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.069702781000000005 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:328974 4.e 4:7002f6a6:::benchmark_data_smithi169_60938_object155637:head [delete] snapc 0=[] ondisk+write+known_if_redirected e529)", 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:19.026205+0000", 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 137.893789766, 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.052057600000000002, 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.941 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 328974 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.026205+0000", 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.026205+0000", 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.026204+0000", 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.026207+0000", 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0209999999999999e-06 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.026212+0000", 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2359999999999998e-06 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.026217+0000", 2024-02-12T16:25:36.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024926784 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.051143+0000", 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142075 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.051286+0000", 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015975700000000001 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6,10", 2024-02-12T16:25:36.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.051445+0000", 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010562568 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.062008+0000", 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022561510000000001 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.064264+0000", 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013838715 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.078103+0000", 2024-02-12T16:25:36.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7589999999999997e-05 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.078150+0000", 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111871 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:19.078262+0000", 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.052057600000000002 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.946 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:330742 4.c 4:3f54ef90:::benchmark_data_smithi169_60938_object157405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e530)", 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:20.573285+0000", 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 136.34670920600001, 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.05259105, 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 330742 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.573285+0000", 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.573285+0000", 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.573283+0000", 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6050000000000004e-06 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.573293+0000", 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1813e-05 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.573304+0000", 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0369999999999997e-06 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.573312+0000", 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031330186000000003 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.604642+0000", 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013509700000000001 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.604777+0000", 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129486 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.604906+0000", 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011008578 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.615915+0000", 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098703799999999998 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.625785+0000", 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0686000000000002e-05 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.625836+0000", 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0266999999999999e-05 2024-02-12T16:25:36.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.625876+0000", 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.05259105 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:330753 4.c 4:3cda8e77:::benchmark_data_smithi169_60938_object157416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e530)", 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:20.581980+0000", 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 136.33801453500001, 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044007782000000002, 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.952 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 330753 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.581980+0000", 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.581980+0000", 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:25:36.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.581977+0000", 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7509999999999997e-06 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.581986+0000", 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6688999999999999e-05 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.582003+0000", 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4159999999999998e-06 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.582010+0000", 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028180501 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.610191+0000", 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117594 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.610308+0000", 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000131848 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.610440+0000", 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055536989999999996 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.615994+0000", 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099158630000000005 2024-02-12T16:25:36.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.625910+0000", 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1751000000000001e-05 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.625951+0000", 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6168999999999999e-05 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.625988+0000", 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044007782000000002 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.958 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:330937 4.c 4:3ca92ca0:::benchmark_data_smithi169_60938_object157600:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:20.744887+0000", 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 136.175107121, 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.059673497999999998, 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 330937 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.959 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.744887+0000", 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.744887+0000", 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.744886+0000", 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5310000000000003e-06 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.744891+0000", 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8460000000000008e-06 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.744898+0000", 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2680000000000002e-06 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.744904+0000", 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014314759 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.759218+0000", 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012489200000000001 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.759343+0000", 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015694600000000001 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.759500+0000", 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004073587 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.763574+0000", 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.040893095999999997 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.804467+0000", 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2905999999999998e-05 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.804520+0000", 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0828999999999998e-05 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:20.804561+0000", 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.059673497999999998 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:336457 4.e 4:7d3a9593:::benchmark_data_smithi169_60938_object163120:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:25.592988+0000", 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 131.32700663400001, 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046528132, 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.964 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 336457 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.592988+0000", 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.592988+0000", 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.592985+0000", 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2199999999999992e-06 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.592994+0000", 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7850999999999999e-05 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593011+0000", 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9020000000000002e-06 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593018+0000", 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010209594000000001 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.603228+0000", 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000128102 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.603356+0000", 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017084700000000001 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6,10", 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.603527+0000", 2024-02-12T16:25:36.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013300584000000001 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.616827+0000", 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038764399999999998 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.617215+0000", 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.02218032 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.639395+0000", 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5954000000000002e-05 2024-02-12T16:25:36.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.639431+0000", 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4546e-05 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.639516+0000", 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046528132 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.969 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:336461 4.e 4:70cbcff8:::benchmark_data_smithi169_60938_object163124:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:25.593038+0000", 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 131.32695620499999, 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046681999000000002, 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 336461 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593038+0000", 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593038+0000", 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593037+0000", 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6209999999999993e-06 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593046+0000", 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2249999999999997e-06 2024-02-12T16:25:36.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593052+0000", 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3830000000000003e-06 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.593059+0000", 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015715459000000001 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.608775+0000", 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016150600000000001 2024-02-12T16:25:36.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.608936+0000", 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203155 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6,10", 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.609139+0000", 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080820909999999996 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.617222+0000", 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013024437999999999 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.630246+0000", 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093187849999999996 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.639565+0000", 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8778e-05 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.639614+0000", 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010665799999999999 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:25.639720+0000", 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046681999000000002 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:337316 4.c 4:3fa5d93b:::benchmark_data_smithi169_60938_object163979:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:26.388390+0000", 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 130.531604466, 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044931598000000003, 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.975 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 337316 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388390+0000", 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388390+0000", 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388388+0000", 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2899999999999999e-06 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388395+0000", 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2265e-05 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388407+0000", 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6878999999999999e-05 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388424+0000", 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014704269000000001 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.403128+0000", 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105751 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.403234+0000", 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115419 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.403349+0000", 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020205170000000001 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.423554+0000", 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096787260000000003 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.433233+0000", 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0210999999999997e-05 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.433283+0000", 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8244000000000002e-05 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.433321+0000", 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044931598000000003 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:337317 4.c 4:3d0eb85d:::benchmark_data_smithi169_60938_object163980:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:26.388485+0000", 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 130.53150986, 2024-02-12T16:25:36.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044978150000000001, 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 337317 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388485+0000", 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388485+0000", 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388483+0000", 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3059999999999999e-06 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388488+0000", 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5210000000000002e-06 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388496+0000", 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4360000000000004e-06 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.388501+0000", 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020245916999999999 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.408747+0000", 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125888 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.408873+0000", 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129107 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.409002+0000", 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014555976 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.423558+0000", 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098191089999999995 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.433377+0000", 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8353000000000003e-05 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.433425+0000", 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7611000000000003e-05 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:26.433463+0000", 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044978150000000001 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:338215 4.c 4:36fcaf7d:::benchmark_data_smithi169_60938_object164878:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:27.158139+0000", 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 129.761854954, 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044575166999999999, 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 338215 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158139+0000", 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158139+0000", 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158138+0000", 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7220000000000002e-06 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158144+0000", 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2799999999999992e-06 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158154+0000", 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8390000000000003e-06 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158159+0000", 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020295757000000001 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.178455+0000", 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111805 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.178567+0000", 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023013999999999999 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.178797+0000", 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085717220000000004 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.187369+0000", 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015267855 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.202637+0000", 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3161000000000002e-05 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.202680+0000", 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4672999999999998e-05 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.202715+0000", 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044575166999999999 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:338217 4.c 4:35c96da8:::benchmark_data_smithi169_60938_object164880:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:27.158410+0000", 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 129.76158479, 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044377666000000003, 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.991 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 338217 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158410+0000", 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158410+0000", 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158408+0000", 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.263e-06 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158414+0000", 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0487e-05 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158425+0000", 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8189999999999997e-06 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.158431+0000", 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025212946999999999 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.183644+0000", 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114122 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.183758+0000", 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011199199999999999 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.183870+0000", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018674560999999999 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.202544+0000", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017833899999999999 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.202723+0000", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4820999999999997e-05 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.202757+0000", 2024-02-12T16:25:36.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9879e-05 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:27.202787+0000", 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044377666000000003 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:339307 4.d 4:bb1cdc39:::benchmark_data_smithi169_60938_object165970:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:28.108566+0000", 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 128.811428038, 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.047269354999999999, 2024-02-12T16:25:36.996 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 339307 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108566+0000", 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108566+0000", 2024-02-12T16:25:36.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108565+0000", 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8039999999999998e-06 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108571+0000", 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2298e-05 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:36.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108583+0000", 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0920000000000003e-06 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108589+0000", 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020079591000000001 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.128669+0000", 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102272 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:36.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.128771+0000", 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119287 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.128890+0000", 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011156931 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.140047+0000", 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015682211000000001 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.155729+0000", 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2706000000000006e-05 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.155792+0000", 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3577e-05 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.155836+0000", 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.047269354999999999 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:339308 4.d 4:b97b84ae:::benchmark_data_smithi169_60938_object165971:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:28.108601+0000", 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 128.81139295400001, 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.047329805000000003, 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 339308 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:37.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108601+0000", 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108601+0000", 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108600+0000", 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0029999999999996e-06 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108604+0000", 2024-02-12T16:25:37.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0129000000000001e-05 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108615+0000", 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0500000000000002e-06 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.108619+0000", 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025359033 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.133978+0000", 2024-02-12T16:25:37.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102089 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.134080+0000", 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121727 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.134201+0000", 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012349791000000001 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.146551+0000", 2024-02-12T16:25:37.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092955969999999992 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.155847+0000", 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6180000000000002e-05 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.155893+0000", 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8155000000000003e-05 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.155931+0000", 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.047329805000000003 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:339418 4.e 4:7adfc96d:::benchmark_data_smithi169_60938_object166081:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:28.202123+0000", 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 128.71787178899999, 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.051908628999999998, 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:37.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 339418 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202123+0000", 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202123+0000", 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:25:37.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202122+0000", 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.985e-06 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202126+0000", 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5939999999999999e-06 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202132+0000", 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5959999999999997e-06 2024-02-12T16:25:37.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202137+0000", 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014978620999999999 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.217116+0000", 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000148181 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.217264+0000", 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182053 2024-02-12T16:25:37.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6,10", 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.217446+0000", 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019792607 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.237239+0000", 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7628999999999998e-05 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.237316+0000", 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016570120000000001 2024-02-12T16:25:37.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.253886+0000", 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.49e-05 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.253931+0000", 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100048 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.254031+0000", 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.051908628999999998 2024-02-12T16:25:37.012 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:339424 4.e 4:732b63e4:::benchmark_data_smithi169_60938_object166087:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:28.202142+0000", 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 128.71785270399999, 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.052112700999999997, 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 339424 2024-02-12T16:25:37.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202142+0000", 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202142+0000", 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202141+0000", 2024-02-12T16:25:37.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2569999999999999e-06 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202144+0000", 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0159999999999999e-06 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202149+0000", 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5549999999999998e-06 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.202153+0000", 2024-02-12T16:25:37.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020311124 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.222464+0000", 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012485099999999999 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.222589+0000", 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014671000000000001 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6,10", 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.222736+0000", 2024-02-12T16:25:37.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014538413 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.237274+0000", 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016840009 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.254114+0000", 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999000000000002e-05 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.254161+0000", 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4377e-05 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.254195+0000", 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9063e-05 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:28.254254+0000", 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.052112700999999997 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:340950 4.c 4:3220bc21:::benchmark_data_smithi169_60938_object167613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:29.514584+0000", 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 127.405410195, 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046243149999999997, 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 340950 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.514584+0000", 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.514584+0000", 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.514583+0000", 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.198e-06 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.514588+0000", 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0593e-05 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.514599+0000", 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9529999999999999e-06 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.514607+0000", 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010962563 2024-02-12T16:25:37.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.525569+0000", 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010840699999999999 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.525678+0000", 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4312000000000002e-05 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.525762+0000", 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046750409999999996 2024-02-12T16:25:37.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.530437+0000", 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.030304096999999999 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.560741+0000", 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6856e-05 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.560788+0000", 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9384000000000001e-05 2024-02-12T16:25:37.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:29.560827+0000", 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046243149999999997 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:342401 4.e 4:71aadd2d:::benchmark_data_smithi169_60938_object169064:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:30.931679+0000", 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 125.98831493, 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044853015000000003, 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:37.024 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 342401 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.931679+0000", 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.931679+0000", 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.931677+0000", 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.422e-06 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.931685+0000", 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3456e-05 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.931698+0000", 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7630000000000003e-06 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.931705+0000", 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013722003 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.945427+0000", 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120015 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.945547+0000", 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013798099999999999 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6,10", 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.945685+0000", 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020478783 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.966164+0000", 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2140999999999999e-05 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.966176+0000", 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010180612 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.976357+0000", 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8838999999999996e-05 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.976426+0000", 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106882 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:30.976532+0000", 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044853015000000003 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:345217 4.c 4:3a285768:::benchmark_data_smithi169_60938_object171880:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:33.398476+0000", 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 123.521518042, 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.047114350999999999, 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 345217 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:37.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.398476+0000", 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.398476+0000", 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.398475+0000", 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1180000000000002e-06 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.398480+0000", 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2240000000000001e-06 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.398489+0000", 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5070000000000002e-06 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.398493+0000", 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014554236999999999 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.413048+0000", 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124108 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.413172+0000", 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013260100000000001 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.413304+0000", 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015937322 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.429242+0000", 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016259207000000001 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.445501+0000", 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.88e-05 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.445550+0000", 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1127000000000001e-05 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.445591+0000", 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.047114350999999999 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:345484 4.d 4:b240d70d:::benchmark_data_smithi169_60938_object172147:head [delete] snapc 0=[] ondisk+write+known_if_redirected e531)", 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:23:33.649319+0000", 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 123.27067540500001, 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.050642906000000001, 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:25:37.035 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 345484 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.649319+0000", 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.649319+0000", 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.649318+0000", 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1710000000000004e-06 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.649322+0000", 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2570000000000001e-06 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.649329+0000", 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9069999999999998e-06 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.649334+0000", 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022103243000000002 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.671438+0000", 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106555 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.671544+0000", 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3311000000000003e-05 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.671617+0000", 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090580469999999996 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.680675+0000", 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019176266000000001 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.699852+0000", 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5469999999999995e-05 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.699917+0000", 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4700000000000002e-05 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:23:33.699962+0000", 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.050642906000000001 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:37.040 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:25:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:37.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:37.167 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:37.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:37.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:37.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:37.420 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:37.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:37.556 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:37.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:25:37.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:37.687 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:37.712 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:37 smithi121 bash[18489]: audit 2024-02-12T16:25:37.564588+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? v1:172.21.15.121:0/4011669505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:25:37.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 151842 151826 129.977 74.0625 0.00532868 0.00769209 2024-02-12T16:25:37.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:37.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:25:37.798 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:37 smithi123 bash[19119]: audit 2024-02-12T16:25:37.564588+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? v1:172.21.15.121:0/4011669505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:25:37.825 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:37.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:37.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:37 smithi169 bash[19604]: audit 2024-02-12T16:25:37.564588+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? v1:172.21.15.121:0/4011669505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:25:37.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:37.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:38.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:38.088 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:38.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:38.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:38.220 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:38.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:38.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:38.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:38.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:38.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:38.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:38.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:38.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:38.646 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:38.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:38.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:38.779 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:38.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:38.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 153040 153024 129.233 74.875 0.00607246 0.00773634 2024-02-12T16:25:38.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:38.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:38 smithi169 bash[19604]: cluster 2024-02-12T16:25:37.635220+0000 mon.a (mon.0) 2076 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:38.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:38 smithi169 bash[19604]: audit 2024-02-12T16:25:37.645721+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? v1:172.21.15.121:0/4011669505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:25:38.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:38 smithi169 bash[19604]: cluster 2024-02-12T16:25:37.645770+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e576: 12 total, 12 up, 12 in 2024-02-12T16:25:38.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:38 smithi169 bash[19604]: cluster 2024-02-12T16:25:37.940043+0000 osd.4 (osd.4) 90 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:25:38.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:38 smithi169 bash[19604]: cluster 2024-02-12T16:25:37.951990+0000 mgr.y (mgr.14146) 1217 : cluster [DBG] pgmap v1686: 37 pgs: 37 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-12T16:25:38.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:38 smithi169 bash[19604]: cluster 2024-02-12T16:25:38.233167+0000 osd.11 (osd.11) 128 : cluster [DBG] 5.d scrub starts 2024-02-12T16:25:38.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:38 smithi121 bash[18489]: cluster 2024-02-12T16:25:37.635220+0000 mon.a (mon.0) 2076 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:38.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:38 smithi121 bash[18489]: audit 2024-02-12T16:25:37.645721+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? v1:172.21.15.121:0/4011669505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:25:38.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:38 smithi121 bash[18489]: cluster 2024-02-12T16:25:37.645770+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e576: 12 total, 12 up, 12 in 2024-02-12T16:25:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:38 smithi121 bash[18489]: cluster 2024-02-12T16:25:37.940043+0000 osd.4 (osd.4) 90 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:25:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:38 smithi121 bash[18489]: cluster 2024-02-12T16:25:37.951990+0000 mgr.y (mgr.14146) 1217 : cluster [DBG] pgmap v1686: 37 pgs: 37 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-12T16:25:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:38 smithi121 bash[18489]: cluster 2024-02-12T16:25:38.233167+0000 osd.11 (osd.11) 128 : cluster [DBG] 5.d scrub starts 2024-02-12T16:25:38.910 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:38.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:39.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:39.037 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:38 smithi123 bash[19119]: cluster 2024-02-12T16:25:37.635220+0000 mon.a (mon.0) 2076 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:25:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:38 smithi123 bash[19119]: audit 2024-02-12T16:25:37.645721+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? v1:172.21.15.121:0/4011669505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:25:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:38 smithi123 bash[19119]: cluster 2024-02-12T16:25:37.645770+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e576: 12 total, 12 up, 12 in 2024-02-12T16:25:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:38 smithi123 bash[19119]: cluster 2024-02-12T16:25:37.940043+0000 osd.4 (osd.4) 90 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:25:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:38 smithi123 bash[19119]: cluster 2024-02-12T16:25:37.951990+0000 mgr.y (mgr.14146) 1217 : cluster [DBG] pgmap v1686: 37 pgs: 37 active+clean; 9.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-12T16:25:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:38 smithi123 bash[19119]: cluster 2024-02-12T16:25:38.233167+0000 osd.11 (osd.11) 128 : cluster [DBG] 5.d scrub starts 2024-02-12T16:25:39.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:39.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:39.315 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:39.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:39.459 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:39.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:39.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:39.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:39.720 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:39.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 16 154086 154070 128.381 65.375 0.0213838 0.00778749 2024-02-12T16:25:39.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:39.847 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:39.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:39.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:39 smithi121 bash[18489]: cluster 2024-02-12T16:25:37.842192+0000 osd.10 (osd.10) 110 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:25:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:39 smithi121 bash[18489]: cluster 2024-02-12T16:25:37.899162+0000 osd.10 (osd.10) 111 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:25:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:39 smithi121 bash[18489]: cluster 2024-02-12T16:25:38.258455+0000 osd.0 (osd.0) 92 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:25:39.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:39.957 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:39 smithi169 bash[19604]: cluster 2024-02-12T16:25:37.842192+0000 osd.10 (osd.10) 110 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:25:39.957 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:39 smithi169 bash[19604]: cluster 2024-02-12T16:25:37.899162+0000 osd.10 (osd.10) 111 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:25:39.957 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:39 smithi169 bash[19604]: cluster 2024-02-12T16:25:38.258455+0000 osd.0 (osd.0) 92 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:25:39.975 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:39.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:40.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:40.108 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:40.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:39 smithi123 bash[19119]: cluster 2024-02-12T16:25:37.842192+0000 osd.10 (osd.10) 110 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:25:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:39 smithi123 bash[19119]: cluster 2024-02-12T16:25:37.899162+0000 osd.10 (osd.10) 111 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:25:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:39 smithi123 bash[19119]: cluster 2024-02-12T16:25:38.258455+0000 osd.0 (osd.0) 92 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:25:40.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:40.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:40.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:40.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:40.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:40.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:40.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:40.516 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:40.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:40.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:40.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:40.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:40.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:40.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:40.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:40.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 155161 155145 127.576 67.1875 0.00642015 0.00783652 2024-02-12T16:25:40.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:40.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:40 smithi123 bash[19119]: cluster 2024-02-12T16:25:39.952446+0000 mgr.y (mgr.14146) 1218 : cluster [DBG] pgmap v1687: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:25:40.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:40 smithi123 bash[19119]: cluster 2024-02-12T16:25:40.058549+0000 osd.7 (osd.7) 132 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:25:40.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:40 smithi123 bash[19119]: cluster 2024-02-12T16:25:40.144856+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:25:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:40 smithi121 bash[18489]: cluster 2024-02-12T16:25:39.952446+0000 mgr.y (mgr.14146) 1218 : cluster [DBG] pgmap v1687: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:25:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:40 smithi121 bash[18489]: cluster 2024-02-12T16:25:40.058549+0000 osd.7 (osd.7) 132 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:25:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:40 smithi121 bash[18489]: cluster 2024-02-12T16:25:40.144856+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:25:40.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:40.923 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:40 smithi169 bash[19604]: cluster 2024-02-12T16:25:39.952446+0000 mgr.y (mgr.14146) 1218 : cluster [DBG] pgmap v1687: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 9.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:25:40.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:40 smithi169 bash[19604]: cluster 2024-02-12T16:25:40.058549+0000 osd.7 (osd.7) 132 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:25:40.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:40 smithi169 bash[19604]: cluster 2024-02-12T16:25:40.144856+0000 osd.7 (osd.7) 133 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:25:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:40.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:40.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:25:41.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:41.048 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:41.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:41.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:41.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:41.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:41.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:41.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:41.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:41.475 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:41.504 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:41.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:41.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:41.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:41.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:41.683 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.9 to 1 (10000) 2024-02-12T16:25:41.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:41.742 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:41 smithi121 bash[18489]: cluster 2024-02-12T16:25:40.938524+0000 osd.11 (osd.11) 129 : cluster [DBG] 5.d scrub ok 2024-02-12T16:25:41.742 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:41 smithi121 bash[18489]: audit 2024-02-12T16:25:41.011362+0000 mon.a (mon.0) 2079 : audit [INF] from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:25:41.742 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:41 smithi121 bash[18489]: audit 2024-02-12T16:25:41.012800+0000 mon.c (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.121:0/286719316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:25:41.779 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:41.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:41.791 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 16 156284 156268 126.831 70.1875 0.00558648 0.007883 2024-02-12T16:25:41.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:41.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:41.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:41.921 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:41 smithi123 bash[19119]: cluster 2024-02-12T16:25:40.938524+0000 osd.11 (osd.11) 129 : cluster [DBG] 5.d scrub ok 2024-02-12T16:25:41.921 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:41 smithi123 bash[19119]: audit 2024-02-12T16:25:41.011362+0000 mon.a (mon.0) 2079 : audit [INF] from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:25:41.921 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:41 smithi123 bash[19119]: audit 2024-02-12T16:25:41.012800+0000 mon.c (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.121:0/286719316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:25:42.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:42.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:41 smithi169 bash[19604]: cluster 2024-02-12T16:25:40.938524+0000 osd.11 (osd.11) 129 : cluster [DBG] 5.d scrub ok 2024-02-12T16:25:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:41 smithi169 bash[19604]: audit 2024-02-12T16:25:41.011362+0000 mon.a (mon.0) 2079 : audit [INF] from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:25:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:41 smithi169 bash[19604]: audit 2024-02-12T16:25:41.012800+0000 mon.c (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.121:0/286719316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:25:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:42.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:42.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:42.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:42.308 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:42.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:42.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:42.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:42.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:42.631 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:25:42.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:42 smithi169 bash[19604]: cluster 2024-02-12T16:25:41.242212+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:25:42.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:42 smithi169 bash[19604]: cluster 2024-02-12T16:25:41.538888+0000 osd.1 (osd.1) 77 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:25:42.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:42 smithi169 bash[19604]: audit 2024-02-12T16:25:41.669768+0000 mon.a (mon.0) 2080 : audit [INF] from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:25:42.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:42 smithi169 bash[19604]: cluster 2024-02-12T16:25:41.669858+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e577: 12 total, 12 up, 12 in 2024-02-12T16:25:42.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:42 smithi169 bash[19604]: cluster 2024-02-12T16:25:41.952877+0000 mgr.y (mgr.14146) 1219 : cluster [DBG] pgmap v1689: 37 pgs: 4 active+clean+scrubbing, 33 active+clean; 9.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-12T16:25:42.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:42 smithi169 bash[19604]: cluster 2024-02-12T16:25:42.122885+0000 osd.7 (osd.7) 134 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:25:42.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:42.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:42.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 157420 157404 126.115 71 0.00640127 0.00792659 2024-02-12T16:25:42.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:42.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:42.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:42 smithi121 bash[18489]: cluster 2024-02-12T16:25:41.242212+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:25:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:42 smithi121 bash[18489]: cluster 2024-02-12T16:25:41.538888+0000 osd.1 (osd.1) 77 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:25:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:42 smithi121 bash[18489]: audit 2024-02-12T16:25:41.669768+0000 mon.a (mon.0) 2080 : audit [INF] from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:25:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:42 smithi121 bash[18489]: cluster 2024-02-12T16:25:41.669858+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e577: 12 total, 12 up, 12 in 2024-02-12T16:25:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:42 smithi121 bash[18489]: cluster 2024-02-12T16:25:41.952877+0000 mgr.y (mgr.14146) 1219 : cluster [DBG] pgmap v1689: 37 pgs: 4 active+clean+scrubbing, 33 active+clean; 9.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-12T16:25:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:42 smithi121 bash[18489]: cluster 2024-02-12T16:25:42.122885+0000 osd.7 (osd.7) 134 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:25:42.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:42.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 10 1 2024-02-12T16:25:42.966 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:42 smithi123 bash[19119]: cluster 2024-02-12T16:25:41.242212+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:25:42.966 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:42 smithi123 bash[19119]: cluster 2024-02-12T16:25:41.538888+0000 osd.1 (osd.1) 77 : cluster [DBG] 5.0 scrub starts 2024-02-12T16:25:42.966 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:42 smithi123 bash[19119]: audit 2024-02-12T16:25:41.669768+0000 mon.a (mon.0) 2080 : audit [INF] from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:25:42.967 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:42 smithi123 bash[19119]: cluster 2024-02-12T16:25:41.669858+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e577: 12 total, 12 up, 12 in 2024-02-12T16:25:42.967 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:42 smithi123 bash[19119]: cluster 2024-02-12T16:25:41.952877+0000 mgr.y (mgr.14146) 1219 : cluster [DBG] pgmap v1689: 37 pgs: 4 active+clean+scrubbing, 33 active+clean; 9.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-02-12T16:25:42.967 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:42 smithi123 bash[19119]: cluster 2024-02-12T16:25:42.122885+0000 osd.7 (osd.7) 134 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:25:42.984 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:42.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:43.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:43.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:43.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:43.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:43.252 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:43.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:43.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:43.384 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:43.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:43.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:43.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:43.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:43.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:43.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:43.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:43.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:43.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:43.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 158435 158419 125.321 63.4375 0.0150187 0.0079783 2024-02-12T16:25:43.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:43 smithi169 bash[19604]: cluster 2024-02-12T16:25:42.181946+0000 osd.0 (osd.0) 93 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:25:43.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:43 smithi169 bash[19604]: cluster 2024-02-12T16:25:42.330167+0000 osd.9 (osd.9) 97 : cluster [DBG] 5.b scrub starts 2024-02-12T16:25:43.809 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:43.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:43.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:43 smithi121 bash[18489]: cluster 2024-02-12T16:25:42.181946+0000 osd.0 (osd.0) 93 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:25:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:43 smithi121 bash[18489]: cluster 2024-02-12T16:25:42.330167+0000 osd.9 (osd.9) 97 : cluster [DBG] 5.b scrub starts 2024-02-12T16:25:43.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:43.942 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:43.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:44.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:44.071 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:44.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:44.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:43 smithi123 bash[19119]: cluster 2024-02-12T16:25:42.181946+0000 osd.0 (osd.0) 93 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:25:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:43 smithi123 bash[19119]: cluster 2024-02-12T16:25:42.330167+0000 osd.9 (osd.9) 97 : cluster [DBG] 5.b scrub starts 2024-02-12T16:25:44.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:44.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:44.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:44.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:44.362 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:44.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:44.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:44.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:44.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:44.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:44.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:44.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:44.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:44.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:44.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:44.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:44.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:25:44.789652 min lat: 0.000938475 max lat: 0.227554 avg lat: 0.00802359 2024-02-12T16:25:44.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:25:44.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 159526 159510 124.607 68.1875 0.0162695 0.00802359 2024-02-12T16:25:44.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:44 smithi169 bash[19604]: cluster 2024-02-12T16:25:43.811028+0000 osd.10 (osd.10) 112 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:25:44.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:44 smithi169 bash[19604]: cluster 2024-02-12T16:25:43.953288+0000 mgr.y (mgr.14146) 1220 : cluster [DBG] pgmap v1690: 37 pgs: 5 active+clean+scrubbing, 32 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-02-12T16:25:44.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:44 smithi123 bash[19119]: cluster 2024-02-12T16:25:43.811028+0000 osd.10 (osd.10) 112 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:25:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:44 smithi123 bash[19119]: cluster 2024-02-12T16:25:43.953288+0000 mgr.y (mgr.14146) 1220 : cluster [DBG] pgmap v1690: 37 pgs: 5 active+clean+scrubbing, 32 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-02-12T16:25:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:44.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:45.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:45.022 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:44 smithi121 bash[18489]: cluster 2024-02-12T16:25:43.811028+0000 osd.10 (osd.10) 112 : cluster [DBG] 5.4 scrub starts 2024-02-12T16:25:45.022 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:44 smithi121 bash[18489]: cluster 2024-02-12T16:25:43.953288+0000 mgr.y (mgr.14146) 1220 : cluster [DBG] pgmap v1690: 37 pgs: 5 active+clean+scrubbing, 32 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-02-12T16:25:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:45.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:45.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:45.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:45.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:45.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:45.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:45.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:45.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:45.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:45.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:45.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:45.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:45.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:45.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:45 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.294877+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:25:45.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:45.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:45.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 160595 160579 123.893 66.8125 0.0141118 0.00806999 2024-02-12T16:25:45.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:45.908 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:45.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:45 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.294877+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:25:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:45.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:46.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:46.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:46.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:46.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:45 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.294877+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:25:46.162 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:46.207 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:46.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:46.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:46.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:46.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:46.499 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:46.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:46.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:46.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:46.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:46.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:46 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.135731+0000 osd.8 (osd.8) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:46.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:46 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.135778+0000 osd.8 (osd.8) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:46.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:46 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.702683+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e578: 12 total, 12 up, 12 in 2024-02-12T16:25:46.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:46 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.893902+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.f scrub starts 2024-02-12T16:25:46.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:46 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.955329+0000 mgr.y (mgr.14146) 1221 : cluster [DBG] pgmap v1692: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-12T16:25:46.741 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:46 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.967722+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.f scrub ok 2024-02-12T16:25:46.771 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:46.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:46.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 161734 161718 123.251 71.1875 0.0202488 0.00811134 2024-02-12T16:25:46.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:46.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:46.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:46.914 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:47.009 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:47.016 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:46 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.135731+0000 osd.8 (osd.8) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:47.016 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:46 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.135778+0000 osd.8 (osd.8) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:47.016 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:46 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.702683+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e578: 12 total, 12 up, 12 in 2024-02-12T16:25:47.016 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:46 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.893902+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.f scrub starts 2024-02-12T16:25:47.016 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:46 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.955329+0000 mgr.y (mgr.14146) 1221 : cluster [DBG] pgmap v1692: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-12T16:25:47.016 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:46 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.967722+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.f scrub ok 2024-02-12T16:25:47.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:46 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.135731+0000 osd.8 (osd.8) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:46 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.135778+0000 osd.8 (osd.8) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:46 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.702683+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e578: 12 total, 12 up, 12 in 2024-02-12T16:25:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:46 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.893902+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.f scrub starts 2024-02-12T16:25:47.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:46 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.955329+0000 mgr.y (mgr.14146) 1221 : cluster [DBG] pgmap v1692: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-12T16:25:47.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:46 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.967722+0000 osd.6 (osd.6) 107 : cluster [DBG] 1.f scrub ok 2024-02-12T16:25:47.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:47.162 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:47.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:47.297 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:47.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:47.442 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:47.497 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:25:47.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:47.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:47.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:47.696 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:47.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 162675 162659 122.474 58.8125 0.0125658 0.00816324 2024-02-12T16:25:47.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:47.834 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:47.844 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:25:47.844 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:25:47.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.844 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:162705 5.f 5:f4ff1024:::benchmark_data_smithi169_84712_object162704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:25:47.844 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.816777+0000", 2024-02-12T16:25:47.844 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.022881103999999999, 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022916410000000002, 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 162705 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.816777+0000", 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:47.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.816777+0000", 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.816775+0000", 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2197999999999998e-05 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.816817+0000", 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5920999999999999e-05 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.816833+0000", 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7410000000000003e-06 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.816840+0000", 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9877999999999999e-05 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.816889+0000", 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012034 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.817010+0000", 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000165246 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.817175+0000", 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039754900000000003 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:47.848 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:162710 5.f 5:f792387f:::benchmark_data_smithi169_84712_object162709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.819258+0000", 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.020400491999999999, 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020535543999999999, 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 162710 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:47.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.819258+0000", 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.819258+0000", 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.819256+0000", 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9762999999999997e-05 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.819316+0000", 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9381e-05 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.819335+0000", 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0230000000000004e-06 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.819342+0000", 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032417359999999998 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822584+0000", 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119292 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822703+0000", 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114181 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822818+0000", 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035594160000000001 2024-02-12T16:25:47.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:162714 5.f 5:f887f7ce:::benchmark_data_smithi169_84712_object162713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.822920+0000", 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.016738370999999998, 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016954832, 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 162714 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.853 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822920+0000", 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822920+0000", 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822918+0000", 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8488e-05 2024-02-12T16:25:47.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822977+0000", 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2995e-05 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822990+0000", 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6820000000000003e-06 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.822997+0000", 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049836619999999998 2024-02-12T16:25:47.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.827981+0000", 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013546200000000001 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.828116+0000", 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010032685 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.838149+0000", 2024-02-12T16:25:47.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015228790000000001 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.16095.0:162705 5.f e578/547)", 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.828369+0000", 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.011290059999999999, 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011577929000000001, 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "queued for pg", 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.828369+0000", 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.828369+0000", 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.828367+0000", 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9179999999999994e-06 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:47.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.828375+0000", 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3298e-05 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.828388+0000", 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3540000000000002e-06 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.828395+0000", 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6737000000000002e-05 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.16095.0:162710 5.f e578/547)", 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.832338+0000", 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0073209579999999998, 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007661747, 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "queued for pg", 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.832338+0000", 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.832338+0000", 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.832335+0000", 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9699999999999999e-06 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.832343+0000", 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5375000000000001e-05 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.832359+0000", 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9219999999999997e-06 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.832367+0000", 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9051e-05 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.16095.0:162723 5.6 e578/547 5:60a70a82:::benchmark_data_smithi169_84712_object162722:head v 578'10107, mlcod=578'10105)", 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.836355+0000", 2024-02-12T16:25:47.862 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0033039380000000002, 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003687947, 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836355+0000", 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836355+0000", 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836353+0000", 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4057999999999997e-05 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836397+0000", 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3354e-05 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836410+0000", 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9999999999999999e-06 2024-02-12T16:25:47.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836417+0000", 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0719000000000001e-05 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836458+0000", 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9572000000000002e-05 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.836477+0000", 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000122757 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.865 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:25:47.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:25:47.866 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:25:47.866 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 6 2024-02-12T16:25:47.866 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:25:47.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:47.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:47.976 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:47 smithi121 bash[18489]: cluster 2024-02-12T16:25:45.765328+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:25:47.976 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:47 smithi121 bash[18489]: cluster 2024-02-12T16:25:46.454036+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.c scrub starts 2024-02-12T16:25:47.976 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:47 smithi121 bash[18489]: cluster 2024-02-12T16:25:46.665046+0000 osd.10 (osd.10) 113 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:25:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:47.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:48.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:47 smithi169 bash[19604]: cluster 2024-02-12T16:25:45.765328+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:25:48.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:47 smithi169 bash[19604]: cluster 2024-02-12T16:25:46.454036+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.c scrub starts 2024-02-12T16:25:48.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:47 smithi169 bash[19604]: cluster 2024-02-12T16:25:46.665046+0000 osd.10 (osd.10) 113 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:25:48.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:48.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:48.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:48.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:47 smithi123 bash[19119]: cluster 2024-02-12T16:25:45.765328+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.0 scrub ok 2024-02-12T16:25:48.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:47 smithi123 bash[19119]: cluster 2024-02-12T16:25:46.454036+0000 osd.1 (osd.1) 79 : cluster [DBG] 5.c scrub starts 2024-02-12T16:25:48.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:47 smithi123 bash[19119]: cluster 2024-02-12T16:25:46.665046+0000 osd.10 (osd.10) 113 : cluster [DBG] 5.4 scrub ok 2024-02-12T16:25:48.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:48.230 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:48.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:48.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:48.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:48.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:48.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:48.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:48.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:48.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:48.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:48.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:48.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:48.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:25:48.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:48.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:48.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 163407 163391 121.561 45.75 0.010641 0.00822464 2024-02-12T16:25:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:48 smithi169 bash[19604]: cluster 2024-02-12T16:25:47.403720+0000 osd.9 (osd.9) 98 : cluster [DBG] 5.b scrub ok 2024-02-12T16:25:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:48 smithi169 bash[19604]: cluster 2024-02-12T16:25:47.890996+0000 osd.5 (osd.5) 85 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:25:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:48 smithi169 bash[19604]: cluster 2024-02-12T16:25:47.933334+0000 osd.5 (osd.5) 86 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:25:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:48 smithi169 bash[19604]: cluster 2024-02-12T16:25:47.939474+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:25:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:48 smithi169 bash[19604]: cluster 2024-02-12T16:25:47.955783+0000 mgr.y (mgr.14146) 1222 : cluster [DBG] pgmap v1693: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 9.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:25:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:48 smithi169 bash[19604]: cluster 2024-02-12T16:25:48.102619+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:25:48.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:48.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:48.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:48.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:48.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:49.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:49.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:48 smithi123 bash[19119]: cluster 2024-02-12T16:25:47.403720+0000 osd.9 (osd.9) 98 : cluster [DBG] 5.b scrub ok 2024-02-12T16:25:49.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:48 smithi123 bash[19119]: cluster 2024-02-12T16:25:47.890996+0000 osd.5 (osd.5) 85 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:25:49.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:48 smithi123 bash[19119]: cluster 2024-02-12T16:25:47.933334+0000 osd.5 (osd.5) 86 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:25:49.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:48 smithi123 bash[19119]: cluster 2024-02-12T16:25:47.939474+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:25:49.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:48 smithi123 bash[19119]: cluster 2024-02-12T16:25:47.955783+0000 mgr.y (mgr.14146) 1222 : cluster [DBG] pgmap v1693: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 9.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:25:49.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:48 smithi123 bash[19119]: cluster 2024-02-12T16:25:48.102619+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:25:49.153 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:48 smithi121 bash[18489]: cluster 2024-02-12T16:25:47.403720+0000 osd.9 (osd.9) 98 : cluster [DBG] 5.b scrub ok 2024-02-12T16:25:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:48 smithi121 bash[18489]: cluster 2024-02-12T16:25:47.890996+0000 osd.5 (osd.5) 85 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:25:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:48 smithi121 bash[18489]: cluster 2024-02-12T16:25:47.933334+0000 osd.5 (osd.5) 86 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:25:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:48 smithi121 bash[18489]: cluster 2024-02-12T16:25:47.939474+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub starts 2024-02-12T16:25:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:48 smithi121 bash[18489]: cluster 2024-02-12T16:25:47.955783+0000 mgr.y (mgr.14146) 1222 : cluster [DBG] pgmap v1693: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 9.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-02-12T16:25:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:48 smithi121 bash[18489]: cluster 2024-02-12T16:25:48.102619+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:25:49.257 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:49.291 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:49.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:49.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:49.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:49.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:49.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:49.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:49.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 164411 164395 120.869 62.75 0.00614197 0.00827169 2024-02-12T16:25:49.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:49.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:49.897 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:49 smithi121 bash[18489]: cluster 2024-02-12T16:25:48.161424+0000 osd.9 (osd.9) 99 : cluster [DBG] 5.e scrub starts 2024-02-12T16:25:49.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:49.949 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:49 smithi123 bash[19119]: cluster 2024-02-12T16:25:48.161424+0000 osd.9 (osd.9) 99 : cluster [DBG] 5.e scrub starts 2024-02-12T16:25:49.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:49.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:50.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:49 smithi169 bash[19604]: cluster 2024-02-12T16:25:48.161424+0000 osd.9 (osd.9) 99 : cluster [DBG] 5.e scrub starts 2024-02-12T16:25:50.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:50.095 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:50.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:50.194 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:25:50.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:50.235 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:25:50.252 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:50.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:50.260 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:25:50.292 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:25:50.326 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:25:50.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:50.363 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:25:50.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:50.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:50.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:50.396 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:25:50.430 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:25:50.457 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:25:50.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:50.490 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:25:50.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:50.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:50.516 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:25:50.550 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.557 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.558 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.558 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.558 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:25:50.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:50.639 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:50.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:50.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:50.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:50.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 165413 165397 120.191 62.625 0.00652561 0.00831866 2024-02-12T16:25:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:50.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:50.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:50.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:50 smithi121 bash[18489]: cluster 2024-02-12T16:25:49.810194+0000 osd.9 (osd.9) 100 : cluster [DBG] 5.e scrub ok 2024-02-12T16:25:50.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:50 smithi121 bash[18489]: cluster 2024-02-12T16:25:49.956244+0000 mgr.y (mgr.14146) 1223 : cluster [DBG] pgmap v1694: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 9.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 943 op/s 2024-02-12T16:25:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:50 smithi121 bash[18489]: audit 2024-02-12T16:25:50.153995+0000 mon.a (mon.0) 2083 : audit [DBG] from='client.? v1:172.21.15.121:0/756139879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:50.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:51.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:51.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:50 smithi169 bash[19604]: cluster 2024-02-12T16:25:49.810194+0000 osd.9 (osd.9) 100 : cluster [DBG] 5.e scrub ok 2024-02-12T16:25:51.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:50 smithi169 bash[19604]: cluster 2024-02-12T16:25:49.956244+0000 mgr.y (mgr.14146) 1223 : cluster [DBG] pgmap v1694: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 9.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 943 op/s 2024-02-12T16:25:51.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:50 smithi169 bash[19604]: audit 2024-02-12T16:25:50.153995+0000 mon.a (mon.0) 2083 : audit [DBG] from='client.? v1:172.21.15.121:0/756139879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:51.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:51.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:50 smithi123 bash[19119]: cluster 2024-02-12T16:25:49.810194+0000 osd.9 (osd.9) 100 : cluster [DBG] 5.e scrub ok 2024-02-12T16:25:51.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:50 smithi123 bash[19119]: cluster 2024-02-12T16:25:49.956244+0000 mgr.y (mgr.14146) 1223 : cluster [DBG] pgmap v1694: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 9.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 943 op/s 2024-02-12T16:25:51.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:50 smithi123 bash[19119]: audit 2024-02-12T16:25:50.153995+0000 mon.a (mon.0) 2083 : audit [DBG] from='client.? v1:172.21.15.121:0/756139879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:25:51.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:51.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:51.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:51.326 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:51.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:51.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:51.596 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:51.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:51.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:51.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 166532 166516 119.614 69.9375 0.00594678 0.00835892 2024-02-12T16:25:51.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:51.862 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:51 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.062517+0000 osd.1 (osd.1) 80 : cluster [DBG] 5.c scrub ok 2024-02-12T16:25:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:51 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.698575+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:51 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.698651+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:51 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.727602+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:25:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:51 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.828025+0000 osd.4 (osd.4) 92 : cluster [DBG] 1.e scrub starts 2024-02-12T16:25:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:51 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.900651+0000 osd.4 (osd.4) 93 : cluster [DBG] 1.e scrub ok 2024-02-12T16:25:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:51 smithi123 bash[19119]: cluster 2024-02-12T16:25:51.642241+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e579: 12 total, 12 up, 12 in 2024-02-12T16:25:51.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:51.994 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:51.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:51 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.062517+0000 osd.1 (osd.1) 80 : cluster [DBG] 5.c scrub ok 2024-02-12T16:25:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:51 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.698575+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:51 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.698651+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:51 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.727602+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:25:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:51 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.828025+0000 osd.4 (osd.4) 92 : cluster [DBG] 1.e scrub starts 2024-02-12T16:25:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:51 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.900651+0000 osd.4 (osd.4) 93 : cluster [DBG] 1.e scrub ok 2024-02-12T16:25:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:51 smithi169 bash[19604]: cluster 2024-02-12T16:25:51.642241+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e579: 12 total, 12 up, 12 in 2024-02-12T16:25:52.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:52.124 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:52.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:52.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:51 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.062517+0000 osd.1 (osd.1) 80 : cluster [DBG] 5.c scrub ok 2024-02-12T16:25:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:51 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.698575+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:51 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.698651+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:51 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.727602+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.2 scrub ok 2024-02-12T16:25:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:51 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.828025+0000 osd.4 (osd.4) 92 : cluster [DBG] 1.e scrub starts 2024-02-12T16:25:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:51 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.900651+0000 osd.4 (osd.4) 93 : cluster [DBG] 1.e scrub ok 2024-02-12T16:25:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:51 smithi121 bash[18489]: cluster 2024-02-12T16:25:51.642241+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e579: 12 total, 12 up, 12 in 2024-02-12T16:25:52.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:52.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:52.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:52.374 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:52.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:52.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:52.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:52.546 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:52.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:52.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:52.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:52.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:52.743 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:52 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.769546+0000 osd.1 (osd.1) 81 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:25:52.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:52 smithi169 bash[19604]: cluster 2024-02-12T16:25:50.908686+0000 osd.1 (osd.1) 82 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:25:52.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:52 smithi169 bash[19604]: cluster 2024-02-12T16:25:51.956689+0000 mgr.y (mgr.14146) 1224 : cluster [DBG] pgmap v1696: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-12T16:25:52.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:52 smithi169 bash[19604]: cluster 2024-02-12T16:25:52.053564+0000 osd.2 (osd.2) 73 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:25:52.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:52.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 167704 167688 119.087 73.25 0.00669485 0.00839561 2024-02-12T16:25:52.803 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:52.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:52 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.769546+0000 osd.1 (osd.1) 81 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:25:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:52 smithi123 bash[19119]: cluster 2024-02-12T16:25:50.908686+0000 osd.1 (osd.1) 82 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:25:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:52 smithi123 bash[19119]: cluster 2024-02-12T16:25:51.956689+0000 mgr.y (mgr.14146) 1224 : cluster [DBG] pgmap v1696: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-12T16:25:52.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:52 smithi123 bash[19119]: cluster 2024-02-12T16:25:52.053564+0000 osd.2 (osd.2) 73 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:25:52.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:52.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:52 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.769546+0000 osd.1 (osd.1) 81 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:25:52.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:52 smithi121 bash[18489]: cluster 2024-02-12T16:25:50.908686+0000 osd.1 (osd.1) 82 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:25:52.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:52 smithi121 bash[18489]: cluster 2024-02-12T16:25:51.956689+0000 mgr.y (mgr.14146) 1224 : cluster [DBG] pgmap v1696: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-02-12T16:25:52.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:52 smithi121 bash[18489]: cluster 2024-02-12T16:25:52.053564+0000 osd.2 (osd.2) 73 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:25:52.928 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:52.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:52.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:53.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:53.059 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:53.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:53.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:53.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:53.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:53.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:53.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:53.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:53.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:53.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:53.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:53.575 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:25:53.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:53.611 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:53.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:53.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:53.742 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:53.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:53.792 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 168898 168882 118.587 74.625 0.00532026 0.00843104 2024-02-12T16:25:53.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:53.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:53.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:53.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:53.994 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:53 smithi123 bash[19119]: cluster 2024-02-12T16:25:52.591503+0000 osd.3 (osd.3) 14 : cluster [DBG] 1.c scrub starts 2024-02-12T16:25:53.994 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:53 smithi123 bash[19119]: cluster 2024-02-12T16:25:52.631302+0000 osd.3 (osd.3) 15 : cluster [DBG] 1.c scrub ok 2024-02-12T16:25:53.994 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:53 smithi123 bash[19119]: cluster 2024-02-12T16:25:53.151224+0000 osd.2 (osd.2) 74 : cluster [DBG] 5.a scrub starts 2024-02-12T16:25:54.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:53 smithi169 bash[19604]: cluster 2024-02-12T16:25:52.591503+0000 osd.3 (osd.3) 14 : cluster [DBG] 1.c scrub starts 2024-02-12T16:25:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:53 smithi169 bash[19604]: cluster 2024-02-12T16:25:52.631302+0000 osd.3 (osd.3) 15 : cluster [DBG] 1.c scrub ok 2024-02-12T16:25:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:53 smithi169 bash[19604]: cluster 2024-02-12T16:25:53.151224+0000 osd.2 (osd.2) 74 : cluster [DBG] 5.a scrub starts 2024-02-12T16:25:54.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:54.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:54.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:53 smithi121 bash[18489]: cluster 2024-02-12T16:25:52.591503+0000 osd.3 (osd.3) 14 : cluster [DBG] 1.c scrub starts 2024-02-12T16:25:54.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:53 smithi121 bash[18489]: cluster 2024-02-12T16:25:52.631302+0000 osd.3 (osd.3) 15 : cluster [DBG] 1.c scrub ok 2024-02-12T16:25:54.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:53 smithi121 bash[18489]: cluster 2024-02-12T16:25:53.151224+0000 osd.2 (osd.2) 74 : cluster [DBG] 5.a scrub starts 2024-02-12T16:25:54.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:54.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:54.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:54.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:54.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:54.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:54.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:25:54.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:54.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:54.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:54 smithi121 bash[18489]: cluster 2024-02-12T16:25:53.957100+0000 mgr.y (mgr.14146) 1225 : cluster [DBG] pgmap v1697: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-02-12T16:25:54.772 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:54 smithi169 bash[19604]: cluster 2024-02-12T16:25:53.957100+0000 mgr.y (mgr.14146) 1225 : cluster [DBG] pgmap v1697: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-02-12T16:25:54.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 90 16 169387 169371 117.609 30.5625 0.604731 0.00844869 2024-02-12T16:25:54.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:54.863 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:54.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:54.995 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:54.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:55.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:55.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:55.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:55.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:54 smithi123 bash[19119]: cluster 2024-02-12T16:25:53.957100+0000 mgr.y (mgr.14146) 1225 : cluster [DBG] pgmap v1697: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-02-12T16:25:55.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:55.266 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:55.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:55.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:55.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:55.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:55.515 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:55.548 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:25:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:55.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:25:55.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:55.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:55.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:55.771 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:25:55.772 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.10 to 1 (10000) 2024-02-12T16:25:55.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.016626 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 169387 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 117.608 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 63.6559 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 342.25 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 30.5625 2024-02-12T16:25:55.793 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 1881 2024-02-12T16:25:55.794 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 1018 2024-02-12T16:25:55.794 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 5476 2024-02-12T16:25:55.794 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 489 2024-02-12T16:25:55.794 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.00850196 2024-02-12T16:25:55.794 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.00999841 2024-02-12T16:25:55.794 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 0.661687 2024-02-12T16:25:55.794 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.000938475 2024-02-12T16:25:55.808 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:25:55.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:55.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:55.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:55.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:55.946 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:55 smithi169 bash[19604]: cluster 2024-02-12T16:25:54.547342+0000 osd.3 (osd.3) 16 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:25:55.946 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:55 smithi169 bash[19604]: cluster 2024-02-12T16:25:54.579244+0000 osd.3 (osd.3) 17 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:25:55.946 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:55 smithi169 bash[19604]: audit 2024-02-12T16:25:55.053606+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.121:0/337673756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:55.946 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:55 smithi169 bash[19604]: audit 2024-02-12T16:25:55.112719+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.121:0/3921102359' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:25:55.964 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:55.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:56.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:56.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:55 smithi121 bash[18489]: cluster 2024-02-12T16:25:54.547342+0000 osd.3 (osd.3) 16 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:25:56.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:55 smithi121 bash[18489]: cluster 2024-02-12T16:25:54.579244+0000 osd.3 (osd.3) 17 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:25:56.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:55 smithi121 bash[18489]: audit 2024-02-12T16:25:55.053606+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.121:0/337673756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:56.074 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:55 smithi121 bash[18489]: audit 2024-02-12T16:25:55.112719+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.121:0/3921102359' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:25:56.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:56.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:56.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:55 smithi123 bash[19119]: cluster 2024-02-12T16:25:54.547342+0000 osd.3 (osd.3) 16 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:25:56.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:55 smithi123 bash[19119]: cluster 2024-02-12T16:25:54.579244+0000 osd.3 (osd.3) 17 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:25:56.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:55 smithi123 bash[19119]: audit 2024-02-12T16:25:55.053606+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.121:0/337673756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:25:56.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:55 smithi123 bash[19119]: audit 2024-02-12T16:25:55.112719+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.121:0/3921102359' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:25:56.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:56.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:56.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:56.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:56.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:56.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:56.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:56.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:56.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:56.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:56.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:56.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:56.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:56.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:56.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:56.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:56.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:56.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:25:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:56.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:25:57.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:57.038 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:56 smithi123 bash[19119]: cluster 2024-02-12T16:25:55.748475+0000 mon.a (mon.0) 2087 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:25:57.038 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:56 smithi123 bash[19119]: cluster 2024-02-12T16:25:55.748527+0000 mon.a (mon.0) 2088 : cluster [INF] Cluster is now healthy 2024-02-12T16:25:57.038 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:56 smithi123 bash[19119]: audit 2024-02-12T16:25:55.762928+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? v1:172.21.15.121:0/337673756' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:25:57.038 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:56 smithi123 bash[19119]: audit 2024-02-12T16:25:55.763015+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? v1:172.21.15.121:0/3921102359' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:25:57.038 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:56 smithi123 bash[19119]: cluster 2024-02-12T16:25:55.763090+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e580: 12 total, 12 up, 12 in 2024-02-12T16:25:57.038 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:56 smithi123 bash[19119]: cluster 2024-02-12T16:25:55.957482+0000 mgr.y (mgr.14146) 1226 : cluster [DBG] pgmap v1699: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s 2024-02-12T16:25:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:56 smithi169 bash[19604]: cluster 2024-02-12T16:25:55.748475+0000 mon.a (mon.0) 2087 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:25:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:56 smithi169 bash[19604]: cluster 2024-02-12T16:25:55.748527+0000 mon.a (mon.0) 2088 : cluster [INF] Cluster is now healthy 2024-02-12T16:25:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:56 smithi169 bash[19604]: audit 2024-02-12T16:25:55.762928+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? v1:172.21.15.121:0/337673756' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:25:57.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:56 smithi169 bash[19604]: audit 2024-02-12T16:25:55.763015+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? v1:172.21.15.121:0/3921102359' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:25:57.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:56 smithi169 bash[19604]: cluster 2024-02-12T16:25:55.763090+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e580: 12 total, 12 up, 12 in 2024-02-12T16:25:57.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:56 smithi169 bash[19604]: cluster 2024-02-12T16:25:55.957482+0000 mgr.y (mgr.14146) 1226 : cluster [DBG] pgmap v1699: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s 2024-02-12T16:25:57.062 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:57.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:56 smithi121 bash[18489]: cluster 2024-02-12T16:25:55.748475+0000 mon.a (mon.0) 2087 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:25:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:56 smithi121 bash[18489]: cluster 2024-02-12T16:25:55.748527+0000 mon.a (mon.0) 2088 : cluster [INF] Cluster is now healthy 2024-02-12T16:25:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:56 smithi121 bash[18489]: audit 2024-02-12T16:25:55.762928+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? v1:172.21.15.121:0/337673756' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:25:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:56 smithi121 bash[18489]: audit 2024-02-12T16:25:55.763015+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? v1:172.21.15.121:0/3921102359' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:25:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:56 smithi121 bash[18489]: cluster 2024-02-12T16:25:55.763090+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e580: 12 total, 12 up, 12 in 2024-02-12T16:25:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:56 smithi121 bash[18489]: cluster 2024-02-12T16:25:55.957482+0000 mgr.y (mgr.14146) 1226 : cluster [DBG] pgmap v1699: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s 2024-02-12T16:25:57.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:57.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:25:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:57.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:25:57.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 11 1 2024-02-12T16:25:57.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:57.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:57.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:57.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:57.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:25:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:57.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:25:57.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:57.645 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:57.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:25:57.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:57.786 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:57.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:57.806 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:25:57.806 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:25:57.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:57 smithi123 bash[19119]: cluster 2024-02-12T16:25:56.326335+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.a scrub ok 2024-02-12T16:25:57.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:57 smithi123 bash[19119]: cluster 2024-02-12T16:25:56.639187+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:57.849 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:57 smithi123 bash[19119]: cluster 2024-02-12T16:25:56.639287+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:57.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:57 smithi123 bash[19119]: cluster 2024-02-12T16:25:56.770021+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e581: 12 total, 12 up, 12 in 2024-02-12T16:25:57.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:57 smithi121 bash[18489]: cluster 2024-02-12T16:25:56.326335+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.a scrub ok 2024-02-12T16:25:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:57 smithi121 bash[18489]: cluster 2024-02-12T16:25:56.639187+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:57 smithi121 bash[18489]: cluster 2024-02-12T16:25:56.639287+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:57 smithi121 bash[18489]: cluster 2024-02-12T16:25:56.770021+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e581: 12 total, 12 up, 12 in 2024-02-12T16:25:57.928 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:25:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:57.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:25:58.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:57 smithi169 bash[19604]: cluster 2024-02-12T16:25:56.326335+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.a scrub ok 2024-02-12T16:25:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:57 smithi169 bash[19604]: cluster 2024-02-12T16:25:56.639187+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:25:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:57 smithi169 bash[19604]: cluster 2024-02-12T16:25:56.639287+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:25:58.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:57 smithi169 bash[19604]: cluster 2024-02-12T16:25:56.770021+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e581: 12 total, 12 up, 12 in 2024-02-12T16:25:58.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:58.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:58.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:58.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:58.329 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:58.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:58.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:58.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:58.594 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:58.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:25:58.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:58.722 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:58.787 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:58 smithi121 bash[18489]: cluster 2024-02-12T16:25:57.772398+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e582: 12 total, 12 up, 12 in 2024-02-12T16:25:58.787 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:58 smithi121 bash[18489]: cluster 2024-02-12T16:25:57.774003+0000 osd.1 (osd.1) 83 : cluster [DBG] 5.0 starting backfill to osd.10 from (0'0,0'0] MAX to 579'10548 2024-02-12T16:25:58.787 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:25:58 smithi121 bash[18489]: cluster 2024-02-12T16:25:57.957912+0000 mgr.y (mgr.14146) 1227 : cluster [DBG] pgmap v1702: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 744 op/s 2024-02-12T16:25:58.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:58.853 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:25:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:25:58.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:58 smithi123 bash[19119]: cluster 2024-02-12T16:25:57.772398+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e582: 12 total, 12 up, 12 in 2024-02-12T16:25:58.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:58 smithi123 bash[19119]: cluster 2024-02-12T16:25:57.774003+0000 osd.1 (osd.1) 83 : cluster [DBG] 5.0 starting backfill to osd.10 from (0'0,0'0] MAX to 579'10548 2024-02-12T16:25:58.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:25:58 smithi123 bash[19119]: cluster 2024-02-12T16:25:57.957912+0000 mgr.y (mgr.14146) 1227 : cluster [DBG] pgmap v1702: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 744 op/s 2024-02-12T16:25:58.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:58.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:59.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:58 smithi169 bash[19604]: cluster 2024-02-12T16:25:57.772398+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e582: 12 total, 12 up, 12 in 2024-02-12T16:25:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:58 smithi169 bash[19604]: cluster 2024-02-12T16:25:57.774003+0000 osd.1 (osd.1) 83 : cluster [DBG] 5.0 starting backfill to osd.10 from (0'0,0'0] MAX to 579'10548 2024-02-12T16:25:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:25:58 smithi169 bash[19604]: cluster 2024-02-12T16:25:57.957912+0000 mgr.y (mgr.14146) 1227 : cluster [DBG] pgmap v1702: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 744 op/s 2024-02-12T16:25:59.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:59.118 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:59.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:59.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:59.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:25:59.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:25:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:59.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:25:59.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:59.380 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:25:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:59.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:25:59.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:59.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:25:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:59.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:25:59.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:59.637 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:25:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:59.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:25:59.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:59.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:25:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:59.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:25:59.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:59.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:25:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:25:59.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:00.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:00.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:00.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:00.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:00.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:00.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:00.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:00.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:00.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:00.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:00.481 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:00.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:00.585 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:00.612 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:00.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:00.717 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:00.754 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:00.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:00.764 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:00.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:00.885 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:00.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:00.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:01.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:01 smithi123 bash[19119]: cluster 2024-02-12T16:25:59.770865+0000 osd.5 (osd.5) 89 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:26:01.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:01 smithi123 bash[19119]: cluster 2024-02-12T16:25:59.785486+0000 osd.5 (osd.5) 90 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:26:01.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:01 smithi123 bash[19119]: cluster 2024-02-12T16:25:59.958362+0000 mgr.y (mgr.14146) 1228 : cluster [DBG] pgmap v1703: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 598 op/s 2024-02-12T16:26:01.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:01.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:01.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:01.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:01.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:01.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:01 smithi169 bash[19604]: cluster 2024-02-12T16:25:59.770865+0000 osd.5 (osd.5) 89 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:26:01.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:01 smithi169 bash[19604]: cluster 2024-02-12T16:25:59.785486+0000 osd.5 (osd.5) 90 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:26:01.277 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:01 smithi169 bash[19604]: cluster 2024-02-12T16:25:59.958362+0000 mgr.y (mgr.14146) 1228 : cluster [DBG] pgmap v1703: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 598 op/s 2024-02-12T16:26:01.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:01.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:01.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:01 smithi121 bash[18489]: cluster 2024-02-12T16:25:59.770865+0000 osd.5 (osd.5) 89 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:26:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:01 smithi121 bash[18489]: cluster 2024-02-12T16:25:59.785486+0000 osd.5 (osd.5) 90 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:26:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:01 smithi121 bash[18489]: cluster 2024-02-12T16:25:59.958362+0000 mgr.y (mgr.14146) 1228 : cluster [DBG] pgmap v1703: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 598 op/s 2024-02-12T16:26:01.422 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:01.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:01.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:01.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:01.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:01.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:01.679 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:01.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:01.785 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:01.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:01.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:01.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:01.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:02.013 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:02.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:02.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:02.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:02.292 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:02.398 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:02.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:02.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:02.562 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:02.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:02.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:02.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:02.816 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:02.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:02.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:02.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:03.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:03 smithi169 bash[19604]: cluster 2024-02-12T16:26:01.958809+0000 mgr.y (mgr.14146) 1229 : cluster [DBG] pgmap v1704: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 165 B/s rd, 17 MiB/s wr, 617 op/s; 10238/334418 objects misplaced (3.061%); 423 KiB/s, 6 objects/s recovering 2024-02-12T16:26:03.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:03.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:03.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:03 smithi123 bash[19119]: cluster 2024-02-12T16:26:01.958809+0000 mgr.y (mgr.14146) 1229 : cluster [DBG] pgmap v1704: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 165 B/s rd, 17 MiB/s wr, 617 op/s; 10238/334418 objects misplaced (3.061%); 423 KiB/s, 6 objects/s recovering 2024-02-12T16:26:03.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:03.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:03.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:03.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:03 smithi121 bash[18489]: cluster 2024-02-12T16:26:01.958809+0000 mgr.y (mgr.14146) 1229 : cluster [DBG] pgmap v1704: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 165 B/s rd, 17 MiB/s wr, 617 op/s; 10238/334418 objects misplaced (3.061%); 423 KiB/s, 6 objects/s recovering 2024-02-12T16:26:03.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:03.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:03.414 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:26:03.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:03.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:03.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:03.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:03.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:03.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:03.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:03.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:03.809 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:03.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:03.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:03.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:04.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:04.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:04.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:04.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:04.221 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:04.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:04.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:04.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:04.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:04.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:04.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:04.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:04.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:04.624 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:04.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:04.729 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:04.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:04.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:04.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:04.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:04.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:04.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:05.024 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:05.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:05.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:05.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:05 smithi121 bash[18489]: cluster 2024-02-12T16:26:03.959304+0000 mgr.y (mgr.14146) 1230 : cluster [DBG] pgmap v1705: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 13 MiB/s wr, 715 op/s; 10238/330626 objects misplaced (3.097%); 328 KiB/s, 5 objects/s recovering 2024-02-12T16:26:05.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:05.283 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:05.284 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:05 smithi169 bash[19604]: cluster 2024-02-12T16:26:03.959304+0000 mgr.y (mgr.14146) 1230 : cluster [DBG] pgmap v1705: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 13 MiB/s wr, 715 op/s; 10238/330626 objects misplaced (3.097%); 328 KiB/s, 5 objects/s recovering 2024-02-12T16:26:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:05.315 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:05 smithi123 bash[19119]: cluster 2024-02-12T16:26:03.959304+0000 mgr.y (mgr.14146) 1230 : cluster [DBG] pgmap v1705: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 13 MiB/s wr, 715 op/s; 10238/330626 objects misplaced (3.097%); 328 KiB/s, 5 objects/s recovering 2024-02-12T16:26:05.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:05.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:05.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:05.555 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:05.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:05.685 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:05.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:05.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:05.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:05.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:05.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:06.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:06 smithi169 bash[19604]: cluster 2024-02-12T16:26:04.717053+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.f scrub starts 2024-02-12T16:26:06.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:06 smithi169 bash[19604]: cluster 2024-02-12T16:26:04.877266+0000 osd.7 (osd.7) 136 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:26:06.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:06 smithi169 bash[19604]: cluster 2024-02-12T16:26:04.914296+0000 osd.7 (osd.7) 137 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:26:06.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:06 smithi169 bash[19604]: cluster 2024-02-12T16:26:05.034965+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:06.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:06 smithi169 bash[19604]: cluster 2024-02-12T16:26:05.035021+0000 osd.7 (osd.7) 139 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:06.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:06.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:06 smithi121 bash[18489]: cluster 2024-02-12T16:26:04.717053+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.f scrub starts 2024-02-12T16:26:06.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:06 smithi121 bash[18489]: cluster 2024-02-12T16:26:04.877266+0000 osd.7 (osd.7) 136 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:26:06.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:06 smithi121 bash[18489]: cluster 2024-02-12T16:26:04.914296+0000 osd.7 (osd.7) 137 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:26:06.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:06 smithi121 bash[18489]: cluster 2024-02-12T16:26:05.034965+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:06.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:06 smithi121 bash[18489]: cluster 2024-02-12T16:26:05.035021+0000 osd.7 (osd.7) 139 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:06.107 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:06.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:06.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:06.246 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:06.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:06.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:06.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:06 smithi123 bash[19119]: cluster 2024-02-12T16:26:04.717053+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.f scrub starts 2024-02-12T16:26:06.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:06 smithi123 bash[19119]: cluster 2024-02-12T16:26:04.877266+0000 osd.7 (osd.7) 136 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:26:06.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:06 smithi123 bash[19119]: cluster 2024-02-12T16:26:04.914296+0000 osd.7 (osd.7) 137 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:26:06.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:06 smithi123 bash[19119]: cluster 2024-02-12T16:26:05.034965+0000 osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:06.358 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:06 smithi123 bash[19119]: cluster 2024-02-12T16:26:05.035021+0000 osd.7 (osd.7) 139 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:06.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:06.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:06.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:06.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:06.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:06.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:06.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:06.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:06.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:06.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:06.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:06.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:06.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:06.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:07.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:07.051 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:07.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:07.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:07.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:07 smithi123 bash[19119]: cluster 2024-02-12T16:26:05.959981+0000 mgr.y (mgr.14146) 1231 : cluster [DBG] pgmap v1706: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 111 B/s rd, 6.7 MiB/s wr, 777 op/s; 10238/326346 objects misplaced (3.137%); 285 KiB/s, 4 objects/s recovering 2024-02-12T16:26:07.165 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:07 smithi123 bash[19119]: cluster 2024-02-12T16:26:06.025519+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e583: 12 total, 12 up, 12 in 2024-02-12T16:26:07.192 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:07.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:07.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:07 smithi169 bash[19604]: cluster 2024-02-12T16:26:05.959981+0000 mgr.y (mgr.14146) 1231 : cluster [DBG] pgmap v1706: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 111 B/s rd, 6.7 MiB/s wr, 777 op/s; 10238/326346 objects misplaced (3.137%); 285 KiB/s, 4 objects/s recovering 2024-02-12T16:26:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:07 smithi169 bash[19604]: cluster 2024-02-12T16:26:06.025519+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e583: 12 total, 12 up, 12 in 2024-02-12T16:26:07.305 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:07 smithi121 bash[18489]: cluster 2024-02-12T16:26:05.959981+0000 mgr.y (mgr.14146) 1231 : cluster [DBG] pgmap v1706: 37 pgs: 1 active+remapped+backfilling, 36 active+clean; 10 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 111 B/s rd, 6.7 MiB/s wr, 777 op/s; 10238/326346 objects misplaced (3.137%); 285 KiB/s, 4 objects/s recovering 2024-02-12T16:26:07.305 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:07 smithi121 bash[18489]: cluster 2024-02-12T16:26:06.025519+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e583: 12 total, 12 up, 12 in 2024-02-12T16:26:07.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:07.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:07.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:07.472 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:07.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:07.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:07.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:07.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:07.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:07.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:07.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:07.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:07.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:07.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:07.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:07.986 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:26:08.004 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:26:08.005 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:26:08.022 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:26:08.022 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:08.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:08.054 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:26:08.083 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:26:08.122 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:26:08.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:08.150 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:26:08.159 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:08.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:08.182 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:26:08.217 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:26:08.247 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:26:08.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:08.282 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:26:08.290 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:08.292 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:08 smithi123 bash[19119]: audit 2024-02-12T16:26:07.882441+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? v1:172.21.15.121:0/3067817185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:08.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:08.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:08 smithi169 bash[19604]: audit 2024-02-12T16:26:07.882441+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? v1:172.21.15.121:0/3067817185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:08.317 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:26:08.351 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.373 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.373 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.373 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:26:08.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:08 smithi121 bash[18489]: audit 2024-02-12T16:26:07.882441+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? v1:172.21.15.121:0/3067817185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:08.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:08.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:08.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:08.573 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:08.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:08.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:08.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:08.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:08.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:08.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:08.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:08.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:26:08.945 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:08.997 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:09.057 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.11 to 1 (10000) 2024-02-12T16:26:09.104 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:09.113 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:09 smithi121 bash[18489]: cluster 2024-02-12T16:26:07.960574+0000 mgr.y (mgr.14146) 1232 : cluster [DBG] pgmap v1708: 37 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 35 active+clean; 9.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 1.9 MiB/s wr, 919 op/s; 9812/320744 objects misplaced (3.059%); 633 KiB/s, 9 objects/s recovering 2024-02-12T16:26:09.113 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:09 smithi121 bash[18489]: audit 2024-02-12T16:26:08.295230+0000 mon.a (mon.0) 2095 : audit [INF] from='client.? v1:172.21.15.121:0/3134742146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:26:09.138 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:09.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:09.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:09 smithi169 bash[19604]: cluster 2024-02-12T16:26:07.960574+0000 mgr.y (mgr.14146) 1232 : cluster [DBG] pgmap v1708: 37 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 35 active+clean; 9.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 1.9 MiB/s wr, 919 op/s; 9812/320744 objects misplaced (3.059%); 633 KiB/s, 9 objects/s recovering 2024-02-12T16:26:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:09 smithi169 bash[19604]: audit 2024-02-12T16:26:08.295230+0000 mon.a (mon.0) 2095 : audit [INF] from='client.? v1:172.21.15.121:0/3134742146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:26:09.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:09.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:09 smithi123 bash[19119]: cluster 2024-02-12T16:26:07.960574+0000 mgr.y (mgr.14146) 1232 : cluster [DBG] pgmap v1708: 37 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 35 active+clean; 9.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 1.9 MiB/s wr, 919 op/s; 9812/320744 objects misplaced (3.059%); 633 KiB/s, 9 objects/s recovering 2024-02-12T16:26:09.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:09 smithi123 bash[19119]: audit 2024-02-12T16:26:08.295230+0000 mon.a (mon.0) 2095 : audit [INF] from='client.? v1:172.21.15.121:0/3134742146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:26:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:09.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:09.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:09.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:09.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:09.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:09.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:09.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:09.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:09.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:10.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:10.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:10.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:10.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:10.176 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:10 smithi123 bash[19119]: audit 2024-02-12T16:26:09.047932+0000 mon.a (mon.0) 2096 : audit [INF] from='client.? v1:172.21.15.121:0/3134742146' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:26:10.176 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:10 smithi123 bash[19119]: cluster 2024-02-12T16:26:09.048051+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e584: 12 total, 12 up, 12 in 2024-02-12T16:26:10.195 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:10.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:10.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:10.307 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:10 smithi121 bash[18489]: audit 2024-02-12T16:26:09.047932+0000 mon.a (mon.0) 2096 : audit [INF] from='client.? v1:172.21.15.121:0/3134742146' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:26:10.307 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:10 smithi121 bash[18489]: cluster 2024-02-12T16:26:09.048051+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e584: 12 total, 12 up, 12 in 2024-02-12T16:26:10.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:10.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:10.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 8 1 2024-02-12T16:26:10.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:10.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:10 smithi169 bash[19604]: audit 2024-02-12T16:26:09.047932+0000 mon.a (mon.0) 2096 : audit [INF] from='client.? v1:172.21.15.121:0/3134742146' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:26:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:10 smithi169 bash[19604]: cluster 2024-02-12T16:26:09.048051+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e584: 12 total, 12 up, 12 in 2024-02-12T16:26:10.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:10.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:10.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:10.742 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:10.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:10.846 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:10.881 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:10.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:10.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:11.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:11.075 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:26:11.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:11.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:11.276 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:11.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:11 smithi123 bash[19119]: cluster 2024-02-12T16:26:09.202906+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.f scrub ok 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:11 smithi123 bash[19119]: cluster 2024-02-12T16:26:09.961045+0000 mgr.y (mgr.14146) 1233 : cluster [DBG] pgmap v1710: 37 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 35 active+clean; 9.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 9812/318642 objects misplaced (3.079%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:11 smithi123 bash[19119]: cluster 2024-02-12T16:26:10.056688+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e585: 12 total, 12 up, 12 in 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:11 smithi123 bash[19119]: audit 2024-02-12T16:26:10.917746+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? v1:172.21.15.121:0/3967723729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:11 smithi121 bash[18489]: cluster 2024-02-12T16:26:09.202906+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.f scrub ok 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:11 smithi121 bash[18489]: cluster 2024-02-12T16:26:09.961045+0000 mgr.y (mgr.14146) 1233 : cluster [DBG] pgmap v1710: 37 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 35 active+clean; 9.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 9812/318642 objects misplaced (3.079%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:11 smithi121 bash[18489]: cluster 2024-02-12T16:26:10.056688+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e585: 12 total, 12 up, 12 in 2024-02-12T16:26:11.392 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:11 smithi121 bash[18489]: audit 2024-02-12T16:26:10.917746+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? v1:172.21.15.121:0/3967723729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:26:11.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:11.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:11.546 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:11.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:11 smithi169 bash[19604]: cluster 2024-02-12T16:26:09.202906+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.f scrub ok 2024-02-12T16:26:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:11 smithi169 bash[19604]: cluster 2024-02-12T16:26:09.961045+0000 mgr.y (mgr.14146) 1233 : cluster [DBG] pgmap v1710: 37 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 35 active+clean; 9.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 9812/318642 objects misplaced (3.079%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:26:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:11 smithi169 bash[19604]: cluster 2024-02-12T16:26:10.056688+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e585: 12 total, 12 up, 12 in 2024-02-12T16:26:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:11 smithi169 bash[19604]: audit 2024-02-12T16:26:10.917746+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? v1:172.21.15.121:0/3967723729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:26:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:11.650 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:11.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:11.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:11.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:11.810 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:11.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:11.943 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:11.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:12.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:26:12.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:12.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:12.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:12.217 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: cluster 2024-02-12T16:26:10.285056+0000 osd.11 (osd.11) 130 : cluster [DBG] 1.b scrub starts 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: cluster 2024-02-12T16:26:10.388500+0000 osd.11 (osd.11) 131 : cluster [DBG] 1.b scrub ok 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: cluster 2024-02-12T16:26:11.054812+0000 mon.a (mon.0) 2100 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: audit 2024-02-12T16:26:11.067605+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.121:0/3967723729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: cluster 2024-02-12T16:26:11.067667+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e586: 12 total, 12 up, 12 in 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: cluster 2024-02-12T16:26:11.067695+0000 osd.0 (osd.0) 94 : cluster [DBG] 5.8 starting backfill to osd.11 from (0'0,0'0] MAX to 583'11398 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: cluster 2024-02-12T16:26:11.780904+0000 osd.1 (osd.1) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:12.225 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:12 smithi123 bash[19119]: cluster 2024-02-12T16:26:11.780955+0000 osd.1 (osd.1) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:12.270 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:12.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:12.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:12.388 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: cluster 2024-02-12T16:26:10.285056+0000 osd.11 (osd.11) 130 : cluster [DBG] 1.b scrub starts 2024-02-12T16:26:12.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: cluster 2024-02-12T16:26:10.388500+0000 osd.11 (osd.11) 131 : cluster [DBG] 1.b scrub ok 2024-02-12T16:26:12.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: cluster 2024-02-12T16:26:11.054812+0000 mon.a (mon.0) 2100 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:12.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: audit 2024-02-12T16:26:11.067605+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.121:0/3967723729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:26:12.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: cluster 2024-02-12T16:26:11.067667+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e586: 12 total, 12 up, 12 in 2024-02-12T16:26:12.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: cluster 2024-02-12T16:26:11.067695+0000 osd.0 (osd.0) 94 : cluster [DBG] 5.8 starting backfill to osd.11 from (0'0,0'0] MAX to 583'11398 2024-02-12T16:26:12.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: cluster 2024-02-12T16:26:11.780904+0000 osd.1 (osd.1) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:12.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:12 smithi169 bash[19604]: cluster 2024-02-12T16:26:11.780955+0000 osd.1 (osd.1) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:12.409 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: cluster 2024-02-12T16:26:10.285056+0000 osd.11 (osd.11) 130 : cluster [DBG] 1.b scrub starts 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: cluster 2024-02-12T16:26:10.388500+0000 osd.11 (osd.11) 131 : cluster [DBG] 1.b scrub ok 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: cluster 2024-02-12T16:26:11.054812+0000 mon.a (mon.0) 2100 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: audit 2024-02-12T16:26:11.067605+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? v1:172.21.15.121:0/3967723729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: cluster 2024-02-12T16:26:11.067667+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e586: 12 total, 12 up, 12 in 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: cluster 2024-02-12T16:26:11.067695+0000 osd.0 (osd.0) 94 : cluster [DBG] 5.8 starting backfill to osd.11 from (0'0,0'0] MAX to 583'11398 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: cluster 2024-02-12T16:26:11.780904+0000 osd.1 (osd.1) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:12 smithi121 bash[18489]: cluster 2024-02-12T16:26:11.780955+0000 osd.1 (osd.1) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:12.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:12.512 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:12.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:12.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:12.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:12.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:12.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:12.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:12.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:12.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:12.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:12.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:13.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:13.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:13.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:13.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:13.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:13.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:13.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:13 smithi169 bash[19604]: cluster 2024-02-12T16:26:11.961488+0000 mgr.y (mgr.14146) 1234 : cluster [DBG] pgmap v1713: 37 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 1 active+clean+scrubbing, 34 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 9539/314196 objects misplaced (3.036%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:26:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:13 smithi169 bash[19604]: audit 2024-02-12T16:26:12.064844+0000 mon.a (mon.0) 2103 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:26:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:13 smithi169 bash[19604]: cluster 2024-02-12T16:26:12.078892+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e587: 12 total, 12 up, 12 in 2024-02-12T16:26:13.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:13.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:13 smithi121 bash[18489]: cluster 2024-02-12T16:26:11.961488+0000 mgr.y (mgr.14146) 1234 : cluster [DBG] pgmap v1713: 37 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 1 active+clean+scrubbing, 34 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 9539/314196 objects misplaced (3.036%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:26:13.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:13 smithi121 bash[18489]: audit 2024-02-12T16:26:12.064844+0000 mon.a (mon.0) 2103 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:26:13.348 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:13 smithi121 bash[18489]: cluster 2024-02-12T16:26:12.078892+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e587: 12 total, 12 up, 12 in 2024-02-12T16:26:13.384 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:13.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:13.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:13 smithi123 bash[19119]: cluster 2024-02-12T16:26:11.961488+0000 mgr.y (mgr.14146) 1234 : cluster [DBG] pgmap v1713: 37 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 1 active+clean+scrubbing, 34 active+clean; 9.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 9539/314196 objects misplaced (3.036%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:26:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:13 smithi123 bash[19119]: audit 2024-02-12T16:26:12.064844+0000 mon.a (mon.0) 2103 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:26:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:13 smithi123 bash[19119]: cluster 2024-02-12T16:26:12.078892+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e587: 12 total, 12 up, 12 in 2024-02-12T16:26:13.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:13.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:13.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:13.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:13.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:13.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:13.733 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:26:13.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:13.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:13.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:13.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:13.902 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:13.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:13.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:26:14.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:14.039 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:14.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:14.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:14.166 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:14.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:14.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:14.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:14.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:14.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:14.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:14.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:14.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:14.584 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:14.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:14.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:14.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:14.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:14.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:14.878 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:14.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:14.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:15.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:15.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:15.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:15 smithi123 bash[19119]: cluster 2024-02-12T16:26:13.961891+0000 mgr.y (mgr.14146) 1235 : cluster [DBG] pgmap v1715: 37 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 35 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s; 9539/311756 objects misplaced (3.060%); 755 KiB/s, 11 objects/s recovering 2024-02-12T16:26:15.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:15.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:15.253 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:15 smithi121 bash[18489]: cluster 2024-02-12T16:26:13.961891+0000 mgr.y (mgr.14146) 1235 : cluster [DBG] pgmap v1715: 37 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 35 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s; 9539/311756 objects misplaced (3.060%); 755 KiB/s, 11 objects/s recovering 2024-02-12T16:26:15.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:15.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:15.404 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:15.423 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:15.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:15.520 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:15 smithi169 bash[19604]: cluster 2024-02-12T16:26:13.961891+0000 mgr.y (mgr.14146) 1235 : cluster [DBG] pgmap v1715: 37 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 35 active+clean; 9.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s; 9539/311756 objects misplaced (3.060%); 755 KiB/s, 11 objects/s recovering 2024-02-12T16:26:15.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:15.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:15.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:15.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:15.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:15.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:15.970 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:15.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:16.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:16.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:16.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:16.227 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:16.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:16.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:16.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:16.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:16.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:16.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:16.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:16.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:16.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:16.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:16.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:16.770 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:16.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:16.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:16 smithi169 bash[19604]: cluster 2024-02-12T16:26:15.415324+0000 osd.9 (osd.9) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:16.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:16 smithi169 bash[19604]: cluster 2024-02-12T16:26:15.415379+0000 osd.9 (osd.9) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:16.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:16 smithi169 bash[19604]: cluster 2024-02-12T16:26:15.962424+0000 mgr.y (mgr.14146) 1236 : cluster [DBG] pgmap v1716: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 9.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 19049/308504 objects misplaced (6.175%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:26:16.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:16 smithi169 bash[19604]: cluster 2024-02-12T16:26:16.070575+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e588: 12 total, 12 up, 12 in 2024-02-12T16:26:16.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:16 smithi121 bash[18489]: cluster 2024-02-12T16:26:15.415324+0000 osd.9 (osd.9) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:16 smithi121 bash[18489]: cluster 2024-02-12T16:26:15.415379+0000 osd.9 (osd.9) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:16 smithi121 bash[18489]: cluster 2024-02-12T16:26:15.962424+0000 mgr.y (mgr.14146) 1236 : cluster [DBG] pgmap v1716: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 9.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 19049/308504 objects misplaced (6.175%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:26:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:16 smithi121 bash[18489]: cluster 2024-02-12T16:26:16.070575+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e588: 12 total, 12 up, 12 in 2024-02-12T16:26:16.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:16.920 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:16 smithi123 bash[19119]: cluster 2024-02-12T16:26:15.415324+0000 osd.9 (osd.9) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:16.920 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:16 smithi123 bash[19119]: cluster 2024-02-12T16:26:15.415379+0000 osd.9 (osd.9) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:16.920 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:16 smithi123 bash[19119]: cluster 2024-02-12T16:26:15.962424+0000 mgr.y (mgr.14146) 1236 : cluster [DBG] pgmap v1716: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 9.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 19049/308504 objects misplaced (6.175%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:26:16.921 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:16 smithi123 bash[19119]: cluster 2024-02-12T16:26:16.070575+0000 mon.a (mon.0) 2105 : cluster [DBG] osdmap e588: 12 total, 12 up, 12 in 2024-02-12T16:26:16.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:17.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:17.075 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:17.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:17.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:17.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:17.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:17.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:17.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:17.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:17.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:17.495 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:17.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:17.602 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:17.627 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:17.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:17.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:17 smithi121 bash[18489]: cluster 2024-02-12T16:26:16.476159+0000 osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:17.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:17 smithi121 bash[18489]: cluster 2024-02-12T16:26:16.476211+0000 osd.4 (osd.4) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:17.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:17 smithi121 bash[18489]: cluster 2024-02-12T16:26:16.637303+0000 mon.a (mon.0) 2106 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:26:17.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:17 smithi121 bash[18489]: cluster 2024-02-12T16:26:17.071510+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e589: 12 total, 12 up, 12 in 2024-02-12T16:26:17.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:17.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:17 smithi169 bash[19604]: cluster 2024-02-12T16:26:16.476159+0000 osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:17.741 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:17 smithi169 bash[19604]: cluster 2024-02-12T16:26:16.476211+0000 osd.4 (osd.4) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:17.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:17 smithi169 bash[19604]: cluster 2024-02-12T16:26:16.637303+0000 mon.a (mon.0) 2106 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:26:17.742 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:17 smithi169 bash[19604]: cluster 2024-02-12T16:26:17.071510+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e589: 12 total, 12 up, 12 in 2024-02-12T16:26:17.759 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:17.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:17.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:17.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:17 smithi123 bash[19119]: cluster 2024-02-12T16:26:16.476159+0000 osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:17.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:17 smithi123 bash[19119]: cluster 2024-02-12T16:26:16.476211+0000 osd.4 (osd.4) 95 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:17.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:17 smithi123 bash[19119]: cluster 2024-02-12T16:26:16.637303+0000 mon.a (mon.0) 2106 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:26:17.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:17 smithi123 bash[19119]: cluster 2024-02-12T16:26:17.071510+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e589: 12 total, 12 up, 12 in 2024-02-12T16:26:17.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:17.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:17.994 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:18.019 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:18.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:18.156 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:18.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:18.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:18.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:18.473 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:18.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:18.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:18.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:18.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.16095.0:190701 5.9 e589/547 5:92714269:::benchmark_data_smithi169_84712_object21311:head v 589'12144, mlcod=589'12142)", 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:18.701113+0000", 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0047382479999999996, 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047729510000000001, 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:18.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701113+0000", 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701113+0000", 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:18.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701111+0000", 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3091000000000001e-05 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701124+0000", 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1080000000000001e-05 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701145+0000", 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5830000000000007e-06 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701153+0000", 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6857999999999998e-05 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701200+0000", 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2637e-05 2024-02-12T16:26:18.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:18.701223+0000", 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109528 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 1 2024-02-12T16:26:18.715 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:26:18.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:18.808 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:18 smithi121 bash[18489]: cluster 2024-02-12T16:26:17.962867+0000 mgr.y (mgr.14146) 1237 : cluster [DBG] pgmap v1719: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s; 18626/302802 objects misplaced (6.151%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:26:18.837 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:18.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:18.949 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:18 smithi169 bash[19604]: cluster 2024-02-12T16:26:17.962867+0000 mgr.y (mgr.14146) 1237 : cluster [DBG] pgmap v1719: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s; 18626/302802 objects misplaced (6.151%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:26:18.953 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:18 smithi123 bash[19119]: cluster 2024-02-12T16:26:17.962867+0000 mgr.y (mgr.14146) 1237 : cluster [DBG] pgmap v1719: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s; 18626/302802 objects misplaced (6.151%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:26:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:18.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:19.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:19.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:19.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:19.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:19.243 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:19.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:19.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:19.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:19.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:19.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:19.526 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:19.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:19.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:19.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:19.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:19.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:19.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:19.782 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:19.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:19.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:19.932 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:19.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:20.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:20.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:26:20.079 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:20.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:20.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:20.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:20.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:20.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:20.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:20.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:20.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:20.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:20.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:20.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:20.600 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:20.632 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:20.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:20.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:20.770 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:20.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:20.873 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:20.902 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:20.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:21.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:21.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:21.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:21 smithi123 bash[19119]: cluster 2024-02-12T16:26:19.963385+0000 mgr.y (mgr.14146) 1238 : cluster [DBG] pgmap v1720: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 18626/300690 objects misplaced (6.194%); 471 KiB/s, 7 objects/s recovering 2024-02-12T16:26:21.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:21.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:21.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:21.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:21 smithi169 bash[19604]: cluster 2024-02-12T16:26:19.963385+0000 mgr.y (mgr.14146) 1238 : cluster [DBG] pgmap v1720: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 18626/300690 objects misplaced (6.194%); 471 KiB/s, 7 objects/s recovering 2024-02-12T16:26:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:21.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:21.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:21 smithi121 bash[18489]: cluster 2024-02-12T16:26:19.963385+0000 mgr.y (mgr.14146) 1238 : cluster [DBG] pgmap v1720: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 18626/300690 objects misplaced (6.194%); 471 KiB/s, 7 objects/s recovering 2024-02-12T16:26:21.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:21.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:21.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:21.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:21.704 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:21.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:21.832 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:21.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:21.968 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:21.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:22.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:22.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:22.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:22.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:22.234 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:22.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:22.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:22.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:22.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:22.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:22.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:22.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:22.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:22.631 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:22.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:22.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:22.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:22.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:22.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:22.921 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:22.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:23.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:23.054 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:23.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:23.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:23 smithi121 bash[18489]: cluster 2024-02-12T16:26:21.963900+0000 mgr.y (mgr.14146) 1239 : cluster [DBG] pgmap v1721: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 9.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 17857/294390 objects misplaced (6.066%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:23.176 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:23.231 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:26:23.279 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:23.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:26:23.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:23 smithi123 bash[19119]: cluster 2024-02-12T16:26:21.963900+0000 mgr.y (mgr.14146) 1239 : cluster [DBG] pgmap v1721: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 9.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 17857/294390 objects misplaced (6.066%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:23.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:23 smithi169 bash[19604]: cluster 2024-02-12T16:26:21.963900+0000 mgr.y (mgr.14146) 1239 : cluster [DBG] pgmap v1721: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 9.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 17857/294390 objects misplaced (6.066%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:23.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:23.325 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:26:23.363 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:26:23.396 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:26:23.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:23.429 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:26:23.443 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:23.456 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:26:23.486 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:26:23.517 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:26:23.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:23.563 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:26:23.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:23.594 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:26:23.627 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:26:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.640 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.641 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:26:23.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:23.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:23.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:23.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:23.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:23.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:23.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:24.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:24 smithi121 bash[18489]: audit 2024-02-12T16:26:23.191614+0000 mon.a (mon.0) 2108 : audit [DBG] from='client.? v1:172.21.15.121:0/3117377585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:24.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:24 smithi121 bash[18489]: audit 2024-02-12T16:26:23.697819+0000 mon.a (mon.0) 2109 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:26:24.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:24 smithi121 bash[18489]: audit 2024-02-12T16:26:23.720758+0000 mon.a (mon.0) 2110 : audit [INF] from='client.26038 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:26:24.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:24 smithi121 bash[18489]: audit 2024-02-12T16:26:23.722280+0000 mon.c (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.121:0/2256359390' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:26:24.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:24.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:24.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:24.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:24.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:24.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:24.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:24 smithi169 bash[19604]: audit 2024-02-12T16:26:23.191614+0000 mon.a (mon.0) 2108 : audit [DBG] from='client.? v1:172.21.15.121:0/3117377585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:24.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:24 smithi169 bash[19604]: audit 2024-02-12T16:26:23.697819+0000 mon.a (mon.0) 2109 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:26:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:24 smithi169 bash[19604]: audit 2024-02-12T16:26:23.720758+0000 mon.a (mon.0) 2110 : audit [INF] from='client.26038 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:26:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:24 smithi169 bash[19604]: audit 2024-02-12T16:26:23.722280+0000 mon.c (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.121:0/2256359390' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:26:24.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:24.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:24 smithi123 bash[19119]: audit 2024-02-12T16:26:23.191614+0000 mon.a (mon.0) 2108 : audit [DBG] from='client.? v1:172.21.15.121:0/3117377585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:24.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:24 smithi123 bash[19119]: audit 2024-02-12T16:26:23.697819+0000 mon.a (mon.0) 2109 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:26:24.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:24 smithi123 bash[19119]: audit 2024-02-12T16:26:23.720758+0000 mon.a (mon.0) 2110 : audit [INF] from='client.26038 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:26:24.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:24 smithi123 bash[19119]: audit 2024-02-12T16:26:23.722280+0000 mon.c (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.121:0/2256359390' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:26:24.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:24.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:24.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:24.533 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:24.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:24.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:24.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:24.724 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.8 to 1 (10000) 2024-02-12T16:26:24.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:24.792 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:24.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:24.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:24.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:24.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:25.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:25 smithi169 bash[19604]: cluster 2024-02-12T16:26:23.964308+0000 mgr.y (mgr.14146) 1240 : cluster [DBG] pgmap v1722: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 17857/290190 objects misplaced (6.154%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:25.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:25 smithi169 bash[19604]: audit 2024-02-12T16:26:24.049376+0000 mon.a (mon.0) 2111 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:26:25.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:25 smithi169 bash[19604]: audit 2024-02-12T16:26:24.050159+0000 mon.a (mon.0) 2112 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:26:25.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:25 smithi169 bash[19604]: audit 2024-02-12T16:26:24.056006+0000 mon.a (mon.0) 2113 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:26:25.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:25 smithi169 bash[19604]: audit 2024-02-12T16:26:24.699135+0000 mon.a (mon.0) 2114 : audit [INF] from='client.26038 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:26:25.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:25 smithi169 bash[19604]: cluster 2024-02-12T16:26:24.699330+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e590: 12 total, 12 up, 12 in 2024-02-12T16:26:25.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:25.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:25.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:25.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:25.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:25.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:25 smithi121 bash[18489]: cluster 2024-02-12T16:26:23.964308+0000 mgr.y (mgr.14146) 1240 : cluster [DBG] pgmap v1722: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 17857/290190 objects misplaced (6.154%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:25.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:25 smithi121 bash[18489]: audit 2024-02-12T16:26:24.049376+0000 mon.a (mon.0) 2111 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:26:25.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:25 smithi121 bash[18489]: audit 2024-02-12T16:26:24.050159+0000 mon.a (mon.0) 2112 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:26:25.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:25 smithi121 bash[18489]: audit 2024-02-12T16:26:24.056006+0000 mon.a (mon.0) 2113 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:26:25.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:25 smithi121 bash[18489]: audit 2024-02-12T16:26:24.699135+0000 mon.a (mon.0) 2114 : audit [INF] from='client.26038 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:26:25.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:25 smithi121 bash[18489]: cluster 2024-02-12T16:26:24.699330+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e590: 12 total, 12 up, 12 in 2024-02-12T16:26:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:25.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:25.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:25.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:25.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:25 smithi123 bash[19119]: cluster 2024-02-12T16:26:23.964308+0000 mgr.y (mgr.14146) 1240 : cluster [DBG] pgmap v1722: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 17857/290190 objects misplaced (6.154%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:25 smithi123 bash[19119]: audit 2024-02-12T16:26:24.049376+0000 mon.a (mon.0) 2111 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:26:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:25 smithi123 bash[19119]: audit 2024-02-12T16:26:24.050159+0000 mon.a (mon.0) 2112 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:26:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:25 smithi123 bash[19119]: audit 2024-02-12T16:26:24.056006+0000 mon.a (mon.0) 2113 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:26:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:25 smithi123 bash[19119]: audit 2024-02-12T16:26:24.699135+0000 mon.a (mon.0) 2114 : audit [INF] from='client.26038 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:26:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:25 smithi123 bash[19119]: cluster 2024-02-12T16:26:24.699330+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e590: 12 total, 12 up, 12 in 2024-02-12T16:26:25.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:25.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:25.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:25.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:25.631 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:25.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:25.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:25.755 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T16:26:25.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:26:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:25.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:25.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:25.915 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:25.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:26.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:26.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:26.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:26.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:26.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:26.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:26.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:26.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:26.434 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:26:26.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:26.496 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:26.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:26.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:26.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:26.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:26.660 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:26:26.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:26.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:26 smithi169 bash[19604]: cluster 2024-02-12T16:26:25.964779+0000 mgr.y (mgr.14146) 1241 : cluster [DBG] pgmap v1724: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s; 17540/285964 objects misplaced (6.134%); 417 KiB/s, 6 objects/s recovering 2024-02-12T16:26:26.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:26 smithi169 bash[19604]: audit 2024-02-12T16:26:25.985154+0000 mon.a (mon.0) 2116 : audit [INF] from='client.26044 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:26.748 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:26 smithi169 bash[19604]: audit 2024-02-12T16:26:25.986593+0000 mon.c (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.121:0/3290495774' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:26.765 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:26.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:26.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:26:26.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:26.876 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:26 smithi121 bash[18489]: cluster 2024-02-12T16:26:25.964779+0000 mgr.y (mgr.14146) 1241 : cluster [DBG] pgmap v1724: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s; 17540/285964 objects misplaced (6.134%); 417 KiB/s, 6 objects/s recovering 2024-02-12T16:26:26.877 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:26 smithi121 bash[18489]: audit 2024-02-12T16:26:25.985154+0000 mon.a (mon.0) 2116 : audit [INF] from='client.26044 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:26.877 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:26 smithi121 bash[18489]: audit 2024-02-12T16:26:25.986593+0000 mon.c (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.121:0/3290495774' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:26.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:26.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:27.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:27.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:26 smithi123 bash[19119]: cluster 2024-02-12T16:26:25.964779+0000 mgr.y (mgr.14146) 1241 : cluster [DBG] pgmap v1724: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s; 17540/285964 objects misplaced (6.134%); 417 KiB/s, 6 objects/s recovering 2024-02-12T16:26:27.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:26 smithi123 bash[19119]: audit 2024-02-12T16:26:25.985154+0000 mon.a (mon.0) 2116 : audit [INF] from='client.26044 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:27.022 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:26 smithi123 bash[19119]: audit 2024-02-12T16:26:25.986593+0000 mon.c (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.121:0/3290495774' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:27.039 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:27.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:27.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:27.170 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:27.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:27.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:27.301 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:27.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:27.404 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:27.428 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:27.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:27.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:27.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:27.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:27.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:27.676 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:27 smithi123 bash[19119]: cluster 2024-02-12T16:26:26.644117+0000 mon.a (mon.0) 2117 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:27.676 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:27 smithi123 bash[19119]: audit 2024-02-12T16:26:26.651544+0000 mon.a (mon.0) 2118 : audit [INF] from='client.26044 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:26:27.676 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:27 smithi123 bash[19119]: cluster 2024-02-12T16:26:26.651620+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e591: 12 total, 12 up, 12 in 2024-02-12T16:26:27.695 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:27.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:27.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:27 smithi169 bash[19604]: cluster 2024-02-12T16:26:26.644117+0000 mon.a (mon.0) 2117 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:27.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:27 smithi169 bash[19604]: audit 2024-02-12T16:26:26.651544+0000 mon.a (mon.0) 2118 : audit [INF] from='client.26044 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:26:27.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:27 smithi169 bash[19604]: cluster 2024-02-12T16:26:26.651620+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e591: 12 total, 12 up, 12 in 2024-02-12T16:26:27.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:27.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:27.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:27.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:27 smithi121 bash[18489]: cluster 2024-02-12T16:26:26.644117+0000 mon.a (mon.0) 2117 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:27.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:27 smithi121 bash[18489]: audit 2024-02-12T16:26:26.651544+0000 mon.a (mon.0) 2118 : audit [INF] from='client.26044 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:26:27.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:27 smithi121 bash[18489]: cluster 2024-02-12T16:26:26.651620+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e591: 12 total, 12 up, 12 in 2024-02-12T16:26:27.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:28.017 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:28.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:28.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:28.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:28.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:28.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:28.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:28.420 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:28.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:28.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:28.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:28.695 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:28.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:28.808 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:28 smithi123 bash[19119]: cluster 2024-02-12T16:26:27.965251+0000 mgr.y (mgr.14146) 1242 : cluster [DBG] pgmap v1726: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 17171/280360 objects misplaced (6.125%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:28.834 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:28.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:28 smithi121 bash[18489]: cluster 2024-02-12T16:26:27.965251+0000 mgr.y (mgr.14146) 1242 : cluster [DBG] pgmap v1726: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 17171/280360 objects misplaced (6.125%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:28.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:28.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:28.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:28 smithi169 bash[19604]: cluster 2024-02-12T16:26:27.965251+0000 mgr.y (mgr.14146) 1242 : cluster [DBG] pgmap v1726: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 8.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 17171/280360 objects misplaced (6.125%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:29.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:29.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:29.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:29.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:29.249 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:29.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:29.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:29.387 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:29.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:29.494 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:29.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:29.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:29.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:29.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:29.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:29.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:29.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:29.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:29.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:26:29.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:29.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:29.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:30.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:30.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:30.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:30.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:30.316 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:30.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:30.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:30.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:30.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:30.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:30.707 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:30.723 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:26:30.723 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:26:30.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:30.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:30.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:30.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:31.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:31 smithi169 bash[19604]: cluster 2024-02-12T16:26:29.965759+0000 mgr.y (mgr.14146) 1243 : cluster [DBG] pgmap v1727: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 17171/278112 objects misplaced (6.174%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:26:31.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:31.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:31.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:31 smithi123 bash[19119]: cluster 2024-02-12T16:26:29.965759+0000 mgr.y (mgr.14146) 1243 : cluster [DBG] pgmap v1727: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 17171/278112 objects misplaced (6.174%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:26:31.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:31 smithi121 bash[18489]: cluster 2024-02-12T16:26:29.965759+0000 mgr.y (mgr.14146) 1243 : cluster [DBG] pgmap v1727: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 8.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 17171/278112 objects misplaced (6.174%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:26:31.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:31.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:31.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:31.329 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:31.329 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:31.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:31.370 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:31.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:31.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:31.500 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:31.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:31.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:26:31.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:31.631 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:31.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:31.734 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:31.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:31.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:31.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:31.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:31.914 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:32.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:32.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:32.155 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:32.178 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:32.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:32.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:32.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:32.443 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:32.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:32.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:32.720 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:32.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:32.857 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:32.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:32.995 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:32.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:33.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:33.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:33.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:33 smithi121 bash[18489]: cluster 2024-02-12T16:26:31.522308+0000 osd.2 (osd.2) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:33 smithi121 bash[18489]: cluster 2024-02-12T16:26:31.522364+0000 osd.2 (osd.2) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:33 smithi121 bash[18489]: cluster 2024-02-12T16:26:31.966529+0000 mgr.y (mgr.14146) 1244 : cluster [DBG] pgmap v1728: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 16402/271632 objects misplaced (6.038%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:26:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:33 smithi121 bash[18489]: cluster 2024-02-12T16:26:32.035057+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e592: 12 total, 12 up, 12 in 2024-02-12T16:26:33.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:33.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:33.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:33 smithi169 bash[19604]: cluster 2024-02-12T16:26:31.522308+0000 osd.2 (osd.2) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:33 smithi169 bash[19604]: cluster 2024-02-12T16:26:31.522364+0000 osd.2 (osd.2) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:33 smithi169 bash[19604]: cluster 2024-02-12T16:26:31.966529+0000 mgr.y (mgr.14146) 1244 : cluster [DBG] pgmap v1728: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 16402/271632 objects misplaced (6.038%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:26:33.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:33 smithi169 bash[19604]: cluster 2024-02-12T16:26:32.035057+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e592: 12 total, 12 up, 12 in 2024-02-12T16:26:33.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:33.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:33 smithi123 bash[19119]: cluster 2024-02-12T16:26:31.522308+0000 osd.2 (osd.2) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:33.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:33 smithi123 bash[19119]: cluster 2024-02-12T16:26:31.522364+0000 osd.2 (osd.2) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:33.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:33 smithi123 bash[19119]: cluster 2024-02-12T16:26:31.966529+0000 mgr.y (mgr.14146) 1244 : cluster [DBG] pgmap v1728: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 16402/271632 objects misplaced (6.038%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:26:33.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:33 smithi123 bash[19119]: cluster 2024-02-12T16:26:32.035057+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e592: 12 total, 12 up, 12 in 2024-02-12T16:26:33.382 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:33.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:33.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:33.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:33.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:33.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:33.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:33.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:33.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:33.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:33.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:33.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:33.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:33.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:34.005 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:34.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:34.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:34 smithi123 bash[19119]: cluster 2024-02-12T16:26:33.765632+0000 osd.10 (osd.10) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:34.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:34 smithi123 bash[19119]: cluster 2024-02-12T16:26:33.765698+0000 osd.10 (osd.10) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:34.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:34.188 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:34 smithi121 bash[18489]: cluster 2024-02-12T16:26:33.765632+0000 osd.10 (osd.10) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:34.188 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:34 smithi121 bash[18489]: cluster 2024-02-12T16:26:33.765698+0000 osd.10 (osd.10) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:34.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:34.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:34 smithi169 bash[19604]: cluster 2024-02-12T16:26:33.765632+0000 osd.10 (osd.10) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:34 smithi169 bash[19604]: cluster 2024-02-12T16:26:33.765698+0000 osd.10 (osd.10) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:34.382 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:34.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:34.461 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:34.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:34.593 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:34.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:34.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:34.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:34.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:34.984 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:34.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:35.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:35.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:35.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:35.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:35.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:35.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:35.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:35 smithi169 bash[19604]: cluster 2024-02-12T16:26:33.967075+0000 mgr.y (mgr.14146) 1245 : cluster [DBG] pgmap v1730: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 16402/267374 objects misplaced (6.134%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:26:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:35 smithi169 bash[19604]: cluster 2024-02-12T16:26:34.044121+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e593: 12 total, 12 up, 12 in 2024-02-12T16:26:35.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:35.362 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:35 smithi123 bash[19119]: cluster 2024-02-12T16:26:33.967075+0000 mgr.y (mgr.14146) 1245 : cluster [DBG] pgmap v1730: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 16402/267374 objects misplaced (6.134%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:26:35.362 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:35 smithi123 bash[19119]: cluster 2024-02-12T16:26:34.044121+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e593: 12 total, 12 up, 12 in 2024-02-12T16:26:35.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:35.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:35 smithi121 bash[18489]: cluster 2024-02-12T16:26:33.967075+0000 mgr.y (mgr.14146) 1245 : cluster [DBG] pgmap v1730: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 16402/267374 objects misplaced (6.134%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:26:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:35 smithi121 bash[18489]: cluster 2024-02-12T16:26:34.044121+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e593: 12 total, 12 up, 12 in 2024-02-12T16:26:35.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:35.533 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:35.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:35.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:35.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:35.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:35.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:35.828 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:35.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:35.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:35.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:35.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:36.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:36.096 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:36.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:36.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:36.231 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:36.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:36.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:36.339 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:26:36.378 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:36.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:36.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:36.517 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:36.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:36.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:36.652 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:36.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:36.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:36.779 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:36.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:36.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:36.893 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:36 smithi169 bash[19604]: cluster 2024-02-12T16:26:35.967541+0000 mgr.y (mgr.14146) 1246 : cluster [DBG] pgmap v1732: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 16053/263170 objects misplaced (6.100%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:26:36.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:36.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:36 smithi121 bash[18489]: cluster 2024-02-12T16:26:35.967541+0000 mgr.y (mgr.14146) 1246 : cluster [DBG] pgmap v1732: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 16053/263170 objects misplaced (6.100%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:26:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:36.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:37.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:37.037 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:37.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:36 smithi123 bash[19119]: cluster 2024-02-12T16:26:35.967541+0000 mgr.y (mgr.14146) 1246 : cluster [DBG] pgmap v1732: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 8.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 16053/263170 objects misplaced (6.100%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:26:37.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:37.164 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:37.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:37.306 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:37.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:37.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:37.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:37.577 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:37.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:37.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:37.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:37.836 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:37.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:37 smithi123 bash[19119]: cluster 2024-02-12T16:26:36.715747+0000 osd.11 (osd.11) 132 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:37.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:37 smithi123 bash[19119]: cluster 2024-02-12T16:26:36.715805+0000 osd.11 (osd.11) 133 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:37.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:37.983 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:37.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:38.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:37 smithi169 bash[19604]: cluster 2024-02-12T16:26:36.715747+0000 osd.11 (osd.11) 132 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:38.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:37 smithi169 bash[19604]: cluster 2024-02-12T16:26:36.715805+0000 osd.11 (osd.11) 133 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:38.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:38.121 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:38.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:37 smithi121 bash[18489]: cluster 2024-02-12T16:26:36.715747+0000 osd.11 (osd.11) 132 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:37 smithi121 bash[18489]: cluster 2024-02-12T16:26:36.715805+0000 osd.11 (osd.11) 133 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:38.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:38.249 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:38.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:38.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:38.378 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:38.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:38.448 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:26:38.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:38.487 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:26:38.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:38.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:38.520 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:26:38.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:26:38.602 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:26:38.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:38.642 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:26:38.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:38.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:38.674 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:26:38.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:38 smithi169 bash[19604]: cluster 2024-02-12T16:26:37.672649+0000 mon.a (mon.0) 2122 : cluster [DBG] osdmap e594: 12 total, 12 up, 12 in 2024-02-12T16:26:38.686 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:38 smithi169 bash[19604]: cluster 2024-02-12T16:26:37.967972+0000 mgr.y (mgr.14146) 1247 : cluster [DBG] pgmap v1734: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 15686/257542 objects misplaced (6.091%); 608 KiB/s, 9 objects/s recovering 2024-02-12T16:26:38.686 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:38 smithi169 bash[19604]: audit 2024-02-12T16:26:38.406070+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? v1:172.21.15.121:0/883916181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:38.711 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:26:38.731 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:26:38.758 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:26:38.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:38.798 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:26:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:38.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:38 smithi121 bash[18489]: cluster 2024-02-12T16:26:37.672649+0000 mon.a (mon.0) 2122 : cluster [DBG] osdmap e594: 12 total, 12 up, 12 in 2024-02-12T16:26:38.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:38 smithi121 bash[18489]: cluster 2024-02-12T16:26:37.967972+0000 mgr.y (mgr.14146) 1247 : cluster [DBG] pgmap v1734: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 15686/257542 objects misplaced (6.091%); 608 KiB/s, 9 objects/s recovering 2024-02-12T16:26:38.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:38 smithi121 bash[18489]: audit 2024-02-12T16:26:38.406070+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? v1:172.21.15.121:0/883916181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:38.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:38.830 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.833 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:26:38.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:38.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:38.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:38 smithi123 bash[19119]: cluster 2024-02-12T16:26:37.672649+0000 mon.a (mon.0) 2122 : cluster [DBG] osdmap e594: 12 total, 12 up, 12 in 2024-02-12T16:26:38.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:38 smithi123 bash[19119]: cluster 2024-02-12T16:26:37.967972+0000 mgr.y (mgr.14146) 1247 : cluster [DBG] pgmap v1734: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 7.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 15686/257542 objects misplaced (6.091%); 608 KiB/s, 9 objects/s recovering 2024-02-12T16:26:38.952 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:38 smithi123 bash[19119]: audit 2024-02-12T16:26:38.406070+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? v1:172.21.15.121:0/883916181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:38.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:39.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:39.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:39.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:39.135 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:26:39.140 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:26:39.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:39.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:39.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:39.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:39.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:39.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:39.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:39.484 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:39.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:39.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:39.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:39.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:39.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:39.745 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:39 smithi169 bash[19604]: audit 2024-02-12T16:26:39.111488+0000 mgr.y (mgr.14146) 1248 : audit [DBG] from='client.26123 v1:172.21.15.121:0/3481439618' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:26:39.768 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:39.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:39.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:39.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:39.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:39.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:39 smithi121 bash[18489]: audit 2024-02-12T16:26:39.111488+0000 mgr.y (mgr.14146) 1248 : audit [DBG] from='client.26123 v1:172.21.15.121:0/3481439618' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:26:40.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:40.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:40.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:39 smithi123 bash[19119]: audit 2024-02-12T16:26:39.111488+0000 mgr.y (mgr.14146) 1248 : audit [DBG] from='client.26123 v1:172.21.15.121:0/3481439618' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:26:40.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:40.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:40.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:40.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:40.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:40.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:40.419 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:40.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:40.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:40.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:40.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:40.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:40.726 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:26:40.726 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:26:40.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:311590 4.9 4.ac7aa2c9 (undecoded) ondisk+retry+write+known_if_redirected e498)", 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:22:01.668824+0000", 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 279.05183729399999, 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89642557700000003, 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 311590 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668824+0000", 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668824+0000", 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999905 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668815+0000", 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1140000000000001e-05 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668836+0000", 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4327999999999999e-05 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668890+0000", 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7877000000000001e-05 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668918+0000", 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2594000000000002e-05 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669001+0000", 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0480000000000001e-06 2024-02-12T16:26:40.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669008+0000", 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.896229414 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565237+0000", 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2639e-05 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565250+0000", 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89642557700000003 2024-02-12T16:26:40.730 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:311596 4.9 4.96417339 (undecoded) ondisk+retry+write+known_if_redirected e498)", 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:22:01.668956+0000", 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 279.05170528999997, 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89632843500000003, 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:26:40.731 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 311596 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668956+0000", 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668956+0000", 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668953+0000", 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0766e-05 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668964+0000", 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9689e-05 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.668994+0000", 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1314e-05 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669005+0000", 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3774000000000003e-05 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669069+0000", 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.27e-06 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669072+0000", 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89620568099999998 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565278+0000", 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1040000000000001e-06 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565285+0000", 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89632843500000003 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:311601 4.9 4.d22cf519 (undecoded) ondisk+retry+write+known_if_redirected e498)", 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:22:01.669040+0000", 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 279.051622065, 2024-02-12T16:26:40.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89627433499999998, 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 311601 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669040+0000", 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669040+0000", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669037+0000", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5839999999999995e-06 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669046+0000", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6392000000000002e-05 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669062+0000", 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5126999999999999e-05 2024-02-12T16:26:40.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669077+0000", 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1424000000000001e-05 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669129+0000", 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8119999999999999e-06 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669132+0000", 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89617532799999999 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565307+0000", 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1219999999999997e-06 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565314+0000", 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89627433499999998 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:311618 4.9 4.fc1317e9 (undecoded) ondisk+retry+write+known_if_redirected e498)", 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:22:01.669144+0000", 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 279.05151822800002, 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89619968800000005, 2024-02-12T16:26:40.739 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 311618 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669144+0000", 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669144+0000", 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999905 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669134+0000", 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7218999999999999e-05 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669151+0000", 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.487e-05 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669166+0000", 2024-02-12T16:26:40.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1824000000000001e-05 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669178+0000", 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3000999999999998e-05 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669211+0000", 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8540000000000001e-06 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669214+0000", 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89612219599999998 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565336+0000", 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2570000000000001e-06 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565343+0000", 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89619968800000005 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:311641 4.9 4.e4b95d29 (undecoded) ondisk+retry+write+known_if_redirected e498)", 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:22:01.669195+0000", 2024-02-12T16:26:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 279.051466973, 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89617834200000002, 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 311641 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669195+0000", 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669195+0000", 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669193+0000", 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8270000000000004e-06 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669201+0000", 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8182999999999999e-05 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669220+0000", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4046000000000001e-05 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669234+0000", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0886000000000002e-05 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669265+0000", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9330000000000001e-06 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669267+0000", 2024-02-12T16:26:40.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89609852999999995 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565366+0000", 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1500000000000002e-06 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565373+0000", 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89617834200000002 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.747 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:311661 4.9 4.ecf65959 (undecoded) ondisk+retry+write+known_if_redirected e498)", 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:22:01.669249+0000", 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 279.05141239699998, 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89615275599999999, 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 311661 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669249+0000", 2024-02-12T16:26:40.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669249+0000", 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669247+0000", 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6229999999999997e-06 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669255+0000", 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5823000000000001e-05 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669271+0000", 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7102999999999999e-05 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669288+0000", 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1636e-05 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669320+0000", 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2050000000000004e-06 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:26:40.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669324+0000", 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.896070747 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565395+0000", 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6420000000000007e-06 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565402+0000", 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89615275599999999 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.25264.0:311668 4.9 4.dd1b0cf9 (undecoded) ondisk+retry+write+known_if_redirected e498)", 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:22:01.669304+0000", 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 279.05135751500001, 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89633492199999998, 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.25264", 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 311668 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.752 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669304+0000", 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669304+0000", 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669302+0000", 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5730000000000002e-06 2024-02-12T16:26:40.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669310+0000", 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7895999999999999e-05 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669328+0000", 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.394e-05 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669342+0000", 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1609999999999997e-05 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669373+0000", 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9809999999999999e-06 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:01.669376+0000", 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89621697600000005 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565593+0000", 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6019000000000001e-05 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:22:02.565639+0000", 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.89633492199999998 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161965 5.2 5:4ccda5b4:::benchmark_data_smithi169_84712_object161964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:46.997872+0000", 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.722790039000003, 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.212414571, 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.756 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161965 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997872+0000", 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997872+0000", 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997869+0000", 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7025000000000001e-05 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997956+0000", 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4990999999999999e-05 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997971+0000", 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3990000000000001e-06 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997978+0000", 2024-02-12T16:26:40.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023957980000000002 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.000374+0000", 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000151667 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.000525+0000", 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000183151 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.000708+0000", 2024-02-12T16:26:40.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010313325 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011022+0000", 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.198999169 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.210021+0000", 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000177367 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.210198+0000", 2024-02-12T16:26:40.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8038e-05 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.210286+0000", 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.212414571 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161971 5.2 5:41152504:::benchmark_data_smithi169_84712_object161970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.006154+0000", 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.714507607000002, 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23144814499999999, 2024-02-12T16:26:40.761 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161971 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006154+0000", 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006154+0000", 2024-02-12T16:26:40.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006152+0000", 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8842999999999999e-05 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006210+0000", 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3706999999999999e-05 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006224+0000", 2024-02-12T16:26:40.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.241e-06 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006230+0000", 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6778000000000001e-05 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006327+0000", 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015125800000000001 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006478+0000", 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018717199999999999 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006666+0000", 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044819100000000004 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011147+0000", 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.226312452 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237460+0000", 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8177999999999995e-05 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237548+0000", 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4280000000000002e-05 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237602+0000", 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23144814499999999 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161977 5.2 5:4a9c9bc2:::benchmark_data_smithi169_84712_object161976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.011338+0000", 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.709324037000002, 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22639947999999999, 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161977 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011338+0000", 2024-02-12T16:26:40.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011338+0000", 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011335+0000", 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5386e-05 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011401+0000", 2024-02-12T16:26:40.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7649999999999999e-05 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011418+0000", 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.605e-06 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011426+0000", 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2783e-05 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011469+0000", 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013387800000000001 2024-02-12T16:26:40.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011603+0000", 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017509700000000001 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011778+0000", 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047544500000000003 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.016532+0000", 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.221132512 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237665+0000", 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2134000000000001e-05 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237697+0000", 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0297999999999999e-05 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237737+0000", 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22639947999999999 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161982 5.2 5:43ce8bef:::benchmark_data_smithi169_84712_object161981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.017144+0000", 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.703517837, 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22072623399999999, 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161982 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017144+0000", 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017144+0000", 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017141+0000", 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012736299999999999 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017269+0000", 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5359999999999999e-05 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017284+0000", 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7969999999999997e-05 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017322+0000", 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7315000000000002e-05 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017389+0000", 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014794 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017537+0000", 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017065400000000001 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017708+0000", 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01007368 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.027782+0000", 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.210005576 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237787+0000", 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6298000000000003e-05 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237833+0000", 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6692e-05 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237870+0000", 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22072623399999999 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161984 5.2 5:490f6ed5:::benchmark_data_smithi169_84712_object161983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.017334+0000", 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.70332775, 2024-02-12T16:26:40.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22064978299999999, 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161984 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017334+0000", 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017334+0000", 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017333+0000", 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7771000000000001e-05 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017371+0000", 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7780000000000007e-06 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017378+0000", 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6789999999999999e-06 2024-02-12T16:26:40.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017385+0000", 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052991150000000001 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.022684+0000", 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000157882 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.022842+0000", 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000183788 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.023026+0000", 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096565990000000001 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.032682+0000", 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20524700800000001 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237930+0000", 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9995e-05 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237960+0000", 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4318999999999999e-05 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237984+0000", 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22064978299999999 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161989 5.2 5:4ce3b2a3:::benchmark_data_smithi169_84712_object161988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.024035+0000", 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.696626706000004, 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21474912700000001, 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.782 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161989 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024035+0000", 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024035+0000", 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:26:40.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024032+0000", 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8520999999999998e-05 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024091+0000", 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7348000000000001e-05 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024108+0000", 2024-02-12T16:26:40.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8700000000000003e-06 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024115+0000", 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037011940000000001 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.027816+0000", 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016248799999999999 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.027979+0000", 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019099899999999999 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.028170+0000", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046490619999999998 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.032819+0000", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20586957 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.238688+0000", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6029000000000002e-05 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.238744+0000", 2024-02-12T16:26:40.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9734000000000003e-05 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.238784+0000", 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21474912700000001 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:189003 5.2 5:42e47263:::benchmark_data_smithi169_84712_object19613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e589)", 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:17.089539+0000", 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.63112258, 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13290511499999999, 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.787 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 189003 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089539+0000", 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089539+0000", 2024-02-12T16:26:40.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089536+0000", 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6569999999999992e-06 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089546+0000", 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5468000000000002e-05 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089561+0000", 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2650000000000002e-06 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089567+0000", 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.011e-05 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089618+0000", 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102673 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089720+0000", 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9585000000000003e-05 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.089820+0000", 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052304320000000001 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.095050+0000", 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.127231176 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.222281+0000", 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4665000000000003e-05 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.222376+0000", 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8227999999999998e-05 2024-02-12T16:26:40.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.222444+0000", 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13290511499999999 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:189007 5.2 5:4ae22b20:::benchmark_data_smithi169_84712_object19617:head [delete] snapc 0=[] ondisk+write+known_if_redirected e589)", 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:17.097802+0000", 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.62285937, 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13045895099999999, 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.792 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 189007 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.097802+0000", 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.097802+0000", 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.097800+0000", 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4500000000000004e-06 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.097809+0000", 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3614e-05 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.097822+0000", 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9950000000000002e-06 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.097828+0000", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3841999999999997e-05 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.097882+0000", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000137727 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.098020+0000", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000166971 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.098187+0000", 2024-02-12T16:26:40.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044690930000000004 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.102656+0000", 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12531083900000001 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.227967+0000", 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011442099999999999 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.228081+0000", 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018027900000000001 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:17.228261+0000", 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13045895099999999 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:213879 5.2 5:457142c1:::benchmark_data_smithi169_84712_object44489:head [delete] snapc 0=[] ondisk+write+known_if_redirected e594)", 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:39.140072+0000", 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.5805895890000001, 2024-02-12T16:26:40.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15980639399999999, 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 213879 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140072+0000", 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140072+0000", 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:26:40.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140071+0000", 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9540000000000003e-06 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140076+0000", 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3259999999999998e-06 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140083+0000", 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9320000000000004e-06 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140088+0000", 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098086890000000006 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.149897+0000", 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000128794 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.150026+0000", 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000161223 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.150187+0000", 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011590903 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.161778+0000", 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13804018900000001 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299818+0000", 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5116000000000001e-05 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299853+0000", 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5465000000000001e-05 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299879+0000", 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15980639399999999 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:213882 5.2 5:4b0a2b37:::benchmark_data_smithi169_84712_object44492:head [delete] snapc 0=[] ondisk+write+known_if_redirected e594)", 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:39.143029+0000", 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.577632833, 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15696279499999999, 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.802 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 213882 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143029+0000", 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143029+0000", 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143027+0000", 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0739999999999993e-06 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143035+0000", 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0524e-05 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143045+0000", 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0979999999999999e-06 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143051+0000", 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010504138999999999 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.153556+0000", 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8573000000000006e-05 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.153654+0000", 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9264000000000005e-05 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.153743+0000", 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012003722 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.165747+0000", 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.134178724 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299926+0000", 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9660999999999999e-05 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299956+0000", 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6199999999999999e-05 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299992+0000", 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15696279499999999 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:213886 5.2 5:463687fa:::benchmark_data_smithi169_84712_object44496:head [delete] snapc 0=[] ondisk+write+known_if_redirected e594)", 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:39.145292+0000", 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.575370181, 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.154780837, 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.807 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 213886 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.145292+0000", 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.145292+0000", 2024-02-12T16:26:40.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.145290+0000", 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6449999999999999e-06 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.145296+0000", 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1513e-05 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.145308+0000", 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0587e-05 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.145318+0000", 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011828636 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.157147+0000", 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117607 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.157265+0000", 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013354799999999999 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.157398+0000", 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083561309999999993 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.165754+0000", 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13427065999999999 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300025+0000", 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4793000000000001e-05 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300050+0000", 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2585000000000001e-05 2024-02-12T16:26:40.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300072+0000", 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.154780837 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:213891 5.2 5:49271382:::benchmark_data_smithi169_84712_object44501:head [delete] snapc 0=[] ondisk+write+known_if_redirected e594)", 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:39.151131+0000", 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.569530764, 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.149032412, 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.812 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 213891 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.151131+0000", 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.151131+0000", 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.151129+0000", 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4520000000000003e-06 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.151139+0000", 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9170000000000001e-06 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.151148+0000", 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6009999999999997e-06 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.151153+0000", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010648853 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.161802+0000", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4524000000000006e-05 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.161886+0000", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011295599999999999 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.161999+0000", 2024-02-12T16:26:40.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038979420000000002 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.165897+0000", 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.134207672 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300105+0000", 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6216000000000001e-05 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300131+0000", 2024-02-12T16:26:40.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2203000000000003e-05 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300163+0000", 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.149032412 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:213910 5.2 5:4951d692:::benchmark_data_smithi169_84712_object44520:head [delete] snapc 0=[] ondisk+write+known_if_redirected e594)", 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:39.171863+0000", 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.5487984030000002, 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12840215899999999, 2024-02-12T16:26:40.817 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 213910 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.171863+0000", 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.171863+0000", 2024-02-12T16:26:40.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.171861+0000", 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4829999999999999e-06 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.171870+0000", 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0988000000000001e-05 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.171881+0000", 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4569999999999996e-06 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.171886+0000", 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8854000000000002e-05 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.171955+0000", 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8102999999999994e-05 2024-02-12T16:26:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.172033+0000", 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4190000000000005e-05 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.172117+0000", 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039406170000000004 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.176058+0000", 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.124142048 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300200+0000", 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7841999999999999e-05 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300228+0000", 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.773e-05 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.300266+0000", 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12840215899999999 2024-02-12T16:26:40.822 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.823 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.823 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.823 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:40.823 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:40.823 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:26:40.824 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1734,"stamp":"2024-02-12T16:26:37.967720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8439070744,"num_objects":128771,"num_object_clones":0,"num_object_copies":257542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15686,"num_objects_unfound":0,"num_objects_dirty":128771,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210005,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":29294592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160905,"ondisk_log_size":160905,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":20911240,"kb_used_data":16626280,"kb_used_omap":91,"kb_used_meta":4284772,"kb_avail":1104030584,"statfs":{"total":1151940427776,"available":1130527318016,"internally_reserved":0,"allocated":17025310720,"data_stored":17020649560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":4387606544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-461701120,"num_objects":-7045,"num_object_clones":0,"num_object_copies":-14090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-716,"num_objects_unfound":0,"num_objects_dirty":-7045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":3735552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001692"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":76,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.040617+0000","last_change":"2024-02-12T16:25:05.516315+0000","last_active":"2024-02-12T16:26:32.040617+0000","last_peered":"2024-02-12T16:26:32.040617+0000","last_clean":"2024-02-12T16:26:32.040617+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:26:32.040617+0000","last_undegraded":"2024-02-12T16:26:32.040617+0000","last_fullsized":"2024-02-12T16:26:32.040617+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:25:05.516165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"592'12829","reported_seq":12886,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:33.627762+0000","last_change":"2024-02-12T16:25:46.665174+0000","last_active":"2024-02-12T16:26:33.627762+0000","last_peered":"2024-02-12T16:26:33.627762+0000","last_clean":"2024-02-12T16:26:33.627762+0000","last_became_active":"2024-02-12T16:24:09.648523+0000","last_became_peered":"2024-02-12T16:24:09.648523+0000","last_unstale":"2024-02-12T16:26:33.627762+0000","last_undegraded":"2024-02-12T16:26:33.627762+0000","last_fullsized":"2024-02-12T16:26:33.627762+0000","mapping_epoch":547,"log_start":"557'2800","ondisk_log_start":"557'2800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10057","last_scrub_stamp":"2024-02-12T16:25:46.665055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:46.665055+0000","log_size":10029,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":537067520,"num_objects":8195,"num_object_clones":0,"num_object_copies":16390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12829,"num_write_kb":672768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":160,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.047456+0000","last_change":"2024-02-12T16:24:40.282763+0000","last_active":"2024-02-12T16:26:32.047456+0000","last_peered":"2024-02-12T16:26:32.047456+0000","last_clean":"2024-02-12T16:26:32.047456+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:26:32.047456+0000","last_undegraded":"2024-02-12T16:26:32.047456+0000","last_fullsized":"2024-02-12T16:26:32.047456+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:40.282662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:40.282662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"593'13217","reported_seq":13275,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:34.250237+0000","last_change":"2024-02-12T16:25:41.242294+0000","last_active":"2024-02-12T16:26:34.250237+0000","last_peered":"2024-02-12T16:26:34.250237+0000","last_clean":"2024-02-12T16:26:34.250237+0000","last_became_active":"2024-02-12T16:24:10.018691+0000","last_became_peered":"2024-02-12T16:24:10.018691+0000","last_unstale":"2024-02-12T16:26:34.250237+0000","last_undegraded":"2024-02-12T16:26:34.250237+0000","last_fullsized":"2024-02-12T16:26:34.250237+0000","mapping_epoch":547,"log_start":"557'3200","ondisk_log_start":"557'3200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9796","last_scrub_stamp":"2024-02-12T16:25:41.242217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:41.242217+0000","log_size":10017,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":534052864,"num_objects":8149,"num_object_clones":0,"num_object_copies":16298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13217,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":274,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-02-12T16:26:26.656374+0000","last_change":"2024-02-12T16:24:47.872602+0000","last_active":"2024-02-12T16:26:26.656374+0000","last_peered":"2024-02-12T16:26:26.656374+0000","last_clean":"2024-02-12T16:26:26.656374+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:26:26.656374+0000","last_undegraded":"2024-02-12T16:26:26.656374+0000","last_fullsized":"2024-02-12T16:26:26.656374+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:47.872500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:24:47.872500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"593'13151","reported_seq":14115,"reported_epoch":593,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:26:36.569371+0000","last_change":"2024-02-12T16:25:58.008531+0000","last_active":"2024-02-12T16:26:36.569371+0000","last_peered":"2024-02-12T16:26:36.569371+0000","last_clean":"2024-02-12T16:25:54.194068+0000","last_became_active":"2024-02-12T16:25:57.932697+0000","last_became_peered":"2024-02-12T16:25:57.932697+0000","last_unstale":"2024-02-12T16:26:36.569371+0000","last_undegraded":"2024-02-12T16:26:36.569371+0000","last_fullsized":"2024-02-12T16:26:36.569371+0000","mapping_epoch":581,"log_start":"557'3100","ondisk_log_start":"557'3100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10027","last_scrub_stamp":"2024-02-12T16:25:45.765333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.765333+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":520683520,"num_objects":7945,"num_object_clones":0,"num_object_copies":15890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7565,"num_objects_unfound":0,"num_objects_dirty":7945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13151,"num_write_kb":675072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":29294592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7945}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":680,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.041216+0000","last_change":"2024-02-12T16:25:07.263767+0000","last_active":"2024-02-12T16:26:32.041216+0000","last_peered":"2024-02-12T16:26:32.041216+0000","last_clean":"2024-02-12T16:26:32.041216+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:26:32.041216+0000","last_undegraded":"2024-02-12T16:26:32.041216+0000","last_fullsized":"2024-02-12T16:26:32.041216+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.263688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.263688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.3","version":"593'13303","reported_seq":13356,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:35.200048+0000","last_change":"2024-02-12T16:24:09.648426+0000","last_active":"2024-02-12T16:26:35.200048+0000","last_peered":"2024-02-12T16:26:35.200048+0000","last_clean":"2024-02-12T16:26:35.200048+0000","last_became_active":"2024-02-12T16:24:09.648139+0000","last_became_peered":"2024-02-12T16:24:09.648139+0000","last_unstale":"2024-02-12T16:26:35.200048+0000","last_undegraded":"2024-02-12T16:26:35.200048+0000","last_fullsized":"2024-02-12T16:26:35.200048+0000","mapping_epoch":547,"log_start":"557'3300","ondisk_log_start":"557'3300","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10003,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":533397504,"num_objects":8139,"num_object_clones":0,"num_object_copies":16278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13303,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":274,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.048687+0000","last_change":"2024-02-12T16:24:45.967418+0000","last_active":"2024-02-12T16:26:32.048687+0000","last_peered":"2024-02-12T16:26:32.048687+0000","last_clean":"2024-02-12T16:26:32.048687+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:26:32.048687+0000","last_undegraded":"2024-02-12T16:26:32.048687+0000","last_fullsized":"2024-02-12T16:26:32.048687+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:45.967340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:45.967340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"593'12850","reported_seq":12908,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:35.214736+0000","last_change":"2024-02-12T16:25:50.727730+0000","last_active":"2024-02-12T16:26:35.214736+0000","last_peered":"2024-02-12T16:26:35.214736+0000","last_clean":"2024-02-12T16:26:35.214736+0000","last_became_active":"2024-02-12T16:24:09.649607+0000","last_became_peered":"2024-02-12T16:24:09.649607+0000","last_unstale":"2024-02-12T16:26:35.214736+0000","last_undegraded":"2024-02-12T16:26:35.214736+0000","last_fullsized":"2024-02-12T16:26:35.214736+0000","mapping_epoch":547,"log_start":"557'2800","ondisk_log_start":"557'2800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10162","last_scrub_stamp":"2024-02-12T16:25:50.727611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.727611+0000","log_size":10050,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":522059776,"num_objects":7966,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12850,"num_write_kb":666112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":265,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-02-12T16:26:26.656355+0000","last_change":"2024-02-12T16:25:07.944563+0000","last_active":"2024-02-12T16:26:26.656355+0000","last_peered":"2024-02-12T16:26:26.656355+0000","last_clean":"2024-02-12T16:26:26.656355+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:26:26.656355+0000","last_undegraded":"2024-02-12T16:26:26.656355+0000","last_fullsized":"2024-02-12T16:26:26.656355+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.944451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.944451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"593'12651","reported_seq":12714,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:35.211396+0000","last_change":"2024-02-12T16:26:10.447550+0000","last_active":"2024-02-12T16:26:35.211396+0000","last_peered":"2024-02-12T16:26:35.211396+0000","last_clean":"2024-02-12T16:26:35.211396+0000","last_became_active":"2024-02-12T16:24:09.647095+0000","last_became_peered":"2024-02-12T16:24:09.647095+0000","last_unstale":"2024-02-12T16:26:35.211396+0000","last_undegraded":"2024-02-12T16:26:35.211396+0000","last_fullsized":"2024-02-12T16:26:35.211396+0000","mapping_epoch":547,"log_start":"557'2600","ondisk_log_start":"557'2600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":523567104,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12651,"num_write_kb":660480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":267,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-02-12T16:26:26.656320+0000","last_change":"2024-02-12T16:25:37.899287+0000","last_active":"2024-02-12T16:26:26.656320+0000","last_peered":"2024-02-12T16:26:26.656320+0000","last_clean":"2024-02-12T16:26:26.656320+0000","last_became_active":"2024-02-12T16:17:37.450884+0000","last_became_peered":"2024-02-12T16:17:37.450884+0000","last_unstale":"2024-02-12T16:26:26.656320+0000","last_undegraded":"2024-02-12T16:26:26.656320+0000","last_fullsized":"2024-02-12T16:26:26.656320+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:37.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:25:37.899168+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"593'13362","reported_seq":13414,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:36.997392+0000","last_change":"2024-02-12T16:24:09.651585+0000","last_active":"2024-02-12T16:26:36.997392+0000","last_peered":"2024-02-12T16:26:36.997392+0000","last_clean":"2024-02-12T16:26:36.997392+0000","last_became_active":"2024-02-12T16:24:09.651253+0000","last_became_peered":"2024-02-12T16:24:09.651253+0000","last_unstale":"2024-02-12T16:26:36.997392+0000","last_undegraded":"2024-02-12T16:26:36.997392+0000","last_fullsized":"2024-02-12T16:26:36.997392+0000","mapping_epoch":547,"log_start":"557'3300","ondisk_log_start":"557'3300","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10062,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":518520832,"num_objects":7912,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13362,"num_write_kb":680768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":135,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.037030+0000","last_change":"2024-02-12T16:25:54.579343+0000","last_active":"2024-02-12T16:26:32.037030+0000","last_peered":"2024-02-12T16:26:32.037030+0000","last_clean":"2024-02-12T16:26:32.037030+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:26:32.037030+0000","last_undegraded":"2024-02-12T16:26:32.037030+0000","last_fullsized":"2024-02-12T16:26:32.037030+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:54.579248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:54.579248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"593'13051","reported_seq":13109,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:36.217441+0000","last_change":"2024-02-12T16:25:52.053658+0000","last_active":"2024-02-12T16:26:36.217441+0000","last_peered":"2024-02-12T16:26:36.217441+0000","last_clean":"2024-02-12T16:26:36.217441+0000","last_became_active":"2024-02-12T16:24:09.647187+0000","last_became_peered":"2024-02-12T16:24:09.647187+0000","last_unstale":"2024-02-12T16:26:36.217441+0000","last_undegraded":"2024-02-12T16:26:36.217441+0000","last_fullsized":"2024-02-12T16:26:36.217441+0000","mapping_epoch":547,"log_start":"557'3000","ondisk_log_start":"557'3000","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'10370","last_scrub_stamp":"2024-02-12T16:25:52.053572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.053572+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":524877824,"num_objects":8009,"num_object_clones":0,"num_object_copies":16018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13051,"num_write_kb":673920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":677,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-02-12T16:26:26.655175+0000","last_change":"2024-02-12T16:26:04.914413+0000","last_active":"2024-02-12T16:26:26.655175+0000","last_peered":"2024-02-12T16:26:26.655175+0000","last_clean":"2024-02-12T16:26:26.655175+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:26:26.655175+0000","last_undegraded":"2024-02-12T16:26:26.655175+0000","last_fullsized":"2024-02-12T16:26:26.655175+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:04.914304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:04.914304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"592'13194","reported_seq":13251,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:33.271215+0000","last_change":"2024-02-12T16:25:45.294991+0000","last_active":"2024-02-12T16:26:33.271215+0000","last_peered":"2024-02-12T16:26:33.271215+0000","last_clean":"2024-02-12T16:26:33.271215+0000","last_became_active":"2024-02-12T16:24:09.650120+0000","last_became_peered":"2024-02-12T16:24:09.650120+0000","last_unstale":"2024-02-12T16:26:33.271215+0000","last_undegraded":"2024-02-12T16:26:33.271215+0000","last_fullsized":"2024-02-12T16:26:33.271215+0000","mapping_epoch":547,"log_start":"557'3100","ondisk_log_start":"557'3100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'10193","last_scrub_stamp":"2024-02-12T16:25:45.294886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.294886+0000","log_size":10094,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":544342016,"num_objects":8306,"num_object_clones":0,"num_object_copies":16612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13194,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":75,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-02-12T16:26:26.655109+0000","last_change":"2024-02-12T16:24:50.938805+0000","last_active":"2024-02-12T16:26:26.655109+0000","last_peered":"2024-02-12T16:26:26.655109+0000","last_clean":"2024-02-12T16:26:26.655109+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:26:26.655109+0000","last_undegraded":"2024-02-12T16:26:26.655109+0000","last_fullsized":"2024-02-12T16:26:26.655109+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.938713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.938713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"593'13085","reported_seq":13152,"reported_epoch":593,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:26:35.190818+0000","last_change":"2024-02-12T16:26:11.304004+0000","last_active":"2024-02-12T16:26:35.190818+0000","last_peered":"2024-02-12T16:26:35.190818+0000","last_clean":"2024-02-12T16:26:09.048189+0000","last_became_active":"2024-02-12T16:26:11.222461+0000","last_became_peered":"2024-02-12T16:26:11.222461+0000","last_unstale":"2024-02-12T16:26:35.190818+0000","last_undegraded":"2024-02-12T16:26:35.190818+0000","last_fullsized":"2024-02-12T16:26:35.190818+0000","mapping_epoch":585,"log_start":"557'3000","ondisk_log_start":"557'3000","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'9817","last_scrub_stamp":"2024-02-12T16:25:42.181950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:42.181950+0000","log_size":10085,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":532217856,"num_objects":8121,"num_object_clones":0,"num_object_copies":16242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8121,"num_objects_unfound":0,"num_objects_dirty":8121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13085,"num_write_kb":678592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[0,10],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,10","objects":8121}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":75,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.036955+0000","last_change":"2024-02-12T16:25:52.631444+0000","last_active":"2024-02-12T16:26:32.036955+0000","last_peered":"2024-02-12T16:26:32.036955+0000","last_clean":"2024-02-12T16:26:32.036955+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:26:32.036955+0000","last_undegraded":"2024-02-12T16:26:32.036955+0000","last_fullsized":"2024-02-12T16:26:32.036955+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:52.631321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.631321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"593'13487","reported_seq":13549,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:37.060130+0000","last_change":"2024-02-12T16:25:47.403836+0000","last_active":"2024-02-12T16:26:37.060130+0000","last_peered":"2024-02-12T16:26:37.060130+0000","last_clean":"2024-02-12T16:26:37.060130+0000","last_became_active":"2024-02-12T16:24:09.649590+0000","last_became_peered":"2024-02-12T16:24:09.649590+0000","last_unstale":"2024-02-12T16:26:37.060130+0000","last_undegraded":"2024-02-12T16:26:37.060130+0000","last_fullsized":"2024-02-12T16:26:37.060130+0000","mapping_epoch":547,"log_start":"557'3400","ondisk_log_start":"557'3400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10313","last_scrub_stamp":"2024-02-12T16:25:47.403730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.403730+0000","log_size":10087,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":525926400,"num_objects":8025,"num_object_clones":0,"num_object_copies":16050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13487,"num_write_kb":688384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":279,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.048683+0000","last_change":"2024-02-12T16:25:45.967785+0000","last_active":"2024-02-12T16:26:32.048683+0000","last_peered":"2024-02-12T16:26:32.048683+0000","last_clean":"2024-02-12T16:26:32.048683+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:26:32.048683+0000","last_undegraded":"2024-02-12T16:26:32.048683+0000","last_fullsized":"2024-02-12T16:26:32.048683+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:45.967726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.967726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"593'12899","reported_seq":12957,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:36.212933+0000","last_change":"2024-02-12T16:25:56.326421+0000","last_active":"2024-02-12T16:26:36.212933+0000","last_peered":"2024-02-12T16:26:36.212933+0000","last_clean":"2024-02-12T16:26:36.212933+0000","last_became_active":"2024-02-12T16:24:10.017647+0000","last_became_peered":"2024-02-12T16:24:10.017647+0000","last_unstale":"2024-02-12T16:26:36.212933+0000","last_undegraded":"2024-02-12T16:26:36.212933+0000","last_fullsized":"2024-02-12T16:26:36.212933+0000","mapping_epoch":547,"log_start":"557'2800","ondisk_log_start":"557'2800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'10379","last_scrub_stamp":"2024-02-12T16:25:56.326340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:56.326340+0000","log_size":10099,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512425984,"num_objects":7819,"num_object_clones":0,"num_object_copies":15638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12899,"num_write_kb":662976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":76,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.044269+0000","last_change":"2024-02-12T16:25:50.900847+0000","last_active":"2024-02-12T16:26:32.044269+0000","last_peered":"2024-02-12T16:26:32.044269+0000","last_clean":"2024-02-12T16:26:32.044269+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:26:32.044269+0000","last_undegraded":"2024-02-12T16:26:32.044269+0000","last_fullsized":"2024-02-12T16:26:32.044269+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.900660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.900660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"593'13117","reported_seq":13176,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:35.161976+0000","last_change":"2024-02-12T16:25:40.938617+0000","last_active":"2024-02-12T16:26:35.161976+0000","last_peered":"2024-02-12T16:26:35.161976+0000","last_clean":"2024-02-12T16:26:35.161976+0000","last_became_active":"2024-02-12T16:24:09.649655+0000","last_became_peered":"2024-02-12T16:24:09.649655+0000","last_unstale":"2024-02-12T16:26:35.161976+0000","last_undegraded":"2024-02-12T16:26:35.161976+0000","last_fullsized":"2024-02-12T16:26:35.161976+0000","mapping_epoch":547,"log_start":"557'3100","ondisk_log_start":"557'3100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9693","last_scrub_stamp":"2024-02-12T16:25:40.938531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:40.938531+0000","log_size":10017,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":534773784,"num_objects":8161,"num_object_clones":0,"num_object_copies":16322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13117,"num_write_kb":680833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":675,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-02-12T16:26:26.655092+0000","last_change":"2024-02-12T16:25:40.144984+0000","last_active":"2024-02-12T16:26:26.655092+0000","last_peered":"2024-02-12T16:26:26.655092+0000","last_clean":"2024-02-12T16:26:26.655092+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:26:26.655092+0000","last_undegraded":"2024-02-12T16:26:26.655092+0000","last_fullsized":"2024-02-12T16:26:26.655092+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:40.144865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:40.144865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"593'13399","reported_seq":13457,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:36.548292+0000","last_change":"2024-02-12T16:25:50.062621+0000","last_active":"2024-02-12T16:26:36.548292+0000","last_peered":"2024-02-12T16:26:36.548292+0000","last_clean":"2024-02-12T16:26:36.548292+0000","last_became_active":"2024-02-12T16:24:09.647617+0000","last_became_peered":"2024-02-12T16:24:09.647617+0000","last_unstale":"2024-02-12T16:26:36.548292+0000","last_undegraded":"2024-02-12T16:26:36.548292+0000","last_fullsized":"2024-02-12T16:26:36.548292+0000","mapping_epoch":547,"log_start":"557'3300","ondisk_log_start":"557'3300","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10397","last_scrub_stamp":"2024-02-12T16:25:50.062524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.062524+0000","log_size":10099,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":527761408,"num_objects":8053,"num_object_clones":0,"num_object_copies":16106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13399,"num_write_kb":686464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":269,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.045716+0000","last_change":"2024-02-12T16:24:50.575354+0000","last_active":"2024-02-12T16:26:32.045716+0000","last_peered":"2024-02-12T16:26:32.045716+0000","last_clean":"2024-02-12T16:26:32.045716+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:26:32.045716+0000","last_undegraded":"2024-02-12T16:26:32.045716+0000","last_fullsized":"2024-02-12T16:26:32.045716+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.575287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.575287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"593'13223","reported_seq":13279,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:36.987908+0000","last_change":"2024-02-12T16:26:09.203006+0000","last_active":"2024-02-12T16:26:36.987908+0000","last_peered":"2024-02-12T16:26:36.987908+0000","last_clean":"2024-02-12T16:26:36.987908+0000","last_became_active":"2024-02-12T16:24:09.650861+0000","last_became_peered":"2024-02-12T16:24:09.650861+0000","last_unstale":"2024-02-12T16:26:36.987908+0000","last_undegraded":"2024-02-12T16:26:36.987908+0000","last_fullsized":"2024-02-12T16:26:36.987908+0000","mapping_epoch":547,"log_start":"557'3200","ondisk_log_start":"557'3200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'11415","last_scrub_stamp":"2024-02-12T16:26:09.202913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:09.202913+0000","log_size":10023,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":526319616,"num_objects":8031,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13223,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":22,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.041254+0000","last_change":"2024-02-12T16:26:10.388710+0000","last_active":"2024-02-12T16:26:32.041254+0000","last_peered":"2024-02-12T16:26:32.041254+0000","last_clean":"2024-02-12T16:26:32.041254+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:26:32.041254+0000","last_undegraded":"2024-02-12T16:26:32.041254+0000","last_fullsized":"2024-02-12T16:26:32.041254+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:10.388515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:10.388515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"593'13187","reported_seq":13245,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-02-12T16:26:37.069450+0000","last_change":"2024-02-12T16:25:49.810317+0000","last_active":"2024-02-12T16:26:37.069450+0000","last_peered":"2024-02-12T16:26:37.069450+0000","last_clean":"2024-02-12T16:26:37.069450+0000","last_became_active":"2024-02-12T16:24:10.019266+0000","last_became_peered":"2024-02-12T16:24:10.019266+0000","last_unstale":"2024-02-12T16:26:37.069450+0000","last_undegraded":"2024-02-12T16:26:37.069450+0000","last_fullsized":"2024-02-12T16:26:37.069450+0000","mapping_epoch":547,"log_start":"557'3100","ondisk_log_start":"557'3100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10246","last_scrub_stamp":"2024-02-12T16:25:49.810203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:49.810203+0000","log_size":10087,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":521076736,"num_objects":7951,"num_object_clones":0,"num_object_copies":15902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13187,"num_write_kb":676416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":257,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.047499+0000","last_change":"2024-02-12T16:25:03.361871+0000","last_active":"2024-02-12T16:26:32.047499+0000","last_peered":"2024-02-12T16:26:32.047499+0000","last_clean":"2024-02-12T16:26:32.047499+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:26:32.047499+0000","last_undegraded":"2024-02-12T16:26:32.047499+0000","last_fullsized":"2024-02-12T16:26:32.047499+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:25:03.361757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":78,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.045758+0000","last_change":"2024-02-12T16:25:59.785619+0000","last_active":"2024-02-12T16:26:32.045758+0000","last_peered":"2024-02-12T16:26:32.045758+0000","last_clean":"2024-02-12T16:26:32.045758+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:26:32.045758+0000","last_undegraded":"2024-02-12T16:26:32.045758+0000","last_fullsized":"2024-02-12T16:26:32.045758+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:59.785496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:25:59.785496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":76,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.043726+0000","last_change":"2024-02-12T16:24:54.540925+0000","last_active":"2024-02-12T16:26:32.043726+0000","last_peered":"2024-02-12T16:26:32.043726+0000","last_clean":"2024-02-12T16:26:32.043726+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:26:32.043726+0000","last_undegraded":"2024-02-12T16:26:32.043726+0000","last_fullsized":"2024-02-12T16:26:32.043726+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:54.540855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:54.540855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":92,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.045808+0000","last_change":"2024-02-12T16:25:47.933452+0000","last_active":"2024-02-12T16:26:32.045808+0000","last_peered":"2024-02-12T16:26:32.045808+0000","last_clean":"2024-02-12T16:26:32.045808+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:26:32.045808+0000","last_undegraded":"2024-02-12T16:26:32.045808+0000","last_fullsized":"2024-02-12T16:26:32.045808+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:47.933341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.933341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":71,"reported_epoch":592,"state":"active+clean","last_fresh":"2024-02-12T16:26:32.040161+0000","last_change":"2024-02-12T16:25:50.908791+0000","last_active":"2024-02-12T16:26:32.040161+0000","last_peered":"2024-02-12T16:26:32.040161+0000","last_clean":"2024-02-12T16:26:32.040161+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:26:32.040161+0000","last_undegraded":"2024-02-12T16:26:32.040161+0000","last_fullsized":"2024-02-12T16:26:32.040161+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.908692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.908692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":8439070744,"num_objects":128771,"num_object_clones":0,"num_object_copies":257542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15686,"num_objects_unfound":0,"num_objects_dirty":128771,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210005,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":29294592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16935100416,"data_stored":16935092272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160905,"ondisk_log_size":160905,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958869,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121096,"kb_used_data":527296,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92624056,"statfs":{"total":95995035648,"available":94847033344,"internally_reserved":0,"allocated":539951104,"data_stored":539557290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.85499999999999998},"min":{"1min":0.25800000000000001,"5min":0.246,"15min":0.246},"max":{"1min":3.6560000000000001,"5min":4.2389999999999999,"15min":5.6870000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.89500000000000002,"15min":0.90100000000000002},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.5819999999999999,"5min":4.5819999999999999,"15min":4.992},"last":0.36799999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.93400000000000005,"15min":1.006},"min":{"1min":0.316,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":1.984,"5min":4.6609999999999996,"15min":7.0110000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.90600000000000003,"15min":1.075},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28799999999999998},"max":{"1min":3.444,"5min":4.2039999999999997,"15min":5.6920000000000002},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.1259999999999999,"15min":1.341},"min":{"1min":0.432,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1910000000000001,"5min":5.343,"15min":5.343},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":1.222,"15min":1.4350000000000001},"min":{"1min":0.39700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.0899999999999999,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.94099999999999995,"15min":2.2679999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.29499999999999998},"max":{"1min":3.6019999999999999,"5min":4.4400000000000004,"15min":368.48399999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.88600000000000001,"15min":2.1850000000000001},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.6110000000000002,"5min":4.915,"15min":371.65100000000001},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1870000000000001,"15min":3.8380000000000001},"min":{"1min":0.39400000000000002,"5min":0.35299999999999998,"15min":0.35099999999999998},"max":{"1min":4.8339999999999996,"5min":5.04,"15min":407.07600000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.159,"15min":3.7999999999999998},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":407.26299999999998},"last":0.745}]},{"osd":6,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.97799999999999998,"15min":0.996},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.373},"max":{"1min":4.7119999999999997,"5min":5.141,"15min":5.141},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.1579999999999999,"15min":1.1319999999999999},"min":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.7450000000000001,"5min":8.5709999999999997,"15min":8.5709999999999997},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.1890000000000001,"15min":1.242},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.524,"5min":5.2569999999999997,"15min":6.4370000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.0289999999999999,"15min":1.1779999999999999},"min":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.131,"5min":5.0430000000000001,"15min":5.1189999999999998},"last":0.61099999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":27.638000000000002,"5min":6.3220000000000001,"15min":2.835},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":537.54399999999998,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":27.850000000000001,"5min":6.5190000000000001,"15min":3.0209999999999999},"min":{"1min":0.496,"5min":0.40500000000000003,"15min":0.34699999999999998},"max":{"1min":539.86099999999999,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.64100000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.171,"15min":1.262},"min":{"1min":0.46400000000000002,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":1.504,"5min":5.7430000000000003,"15min":10.743},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.21,"15min":1.4079999999999999},"min":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.39500000000000002},"max":{"1min":1.5660000000000001,"5min":5.359,"15min":10.698},"last":0.69999999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":27.643000000000001,"5min":6.4820000000000002,"15min":2.9700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":536.245,"5min":536.245,"15min":536.245},"last":0.77200000000000002},{"interface":"front","average":{"1min":27.702999999999999,"5min":6.4950000000000001,"15min":2.984},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.443},"max":{"1min":536.64300000000003,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.65500000000000003}]}]},{"osd":10,"up_from":83,"seq":356482285918,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1385272,"kb_used_data":1079856,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92359880,"statfs":{"total":95995035648,"available":94576517120,"internally_reserved":0,"allocated":1105772544,"data_stored":1105388754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:01 2024","interfaces":[{"interface":"back","average":{"1min":9.9930000000000003,"5min":2.6509999999999998,"15min":1.546},"min":{"1min":0.33700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":162.38399999999999,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":9.8620000000000001,"5min":2.6760000000000002,"15min":1.615},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":162.01300000000001,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.36099999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":0.91100000000000003,"15min":0.96399999999999997},"min":{"1min":0.315,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":4.4059999999999997,"5min":4.4059999999999997,"15min":19.866},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.89700000000000002,"15min":1.0509999999999999},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.27700000000000002},"max":{"1min":1.52,"5min":2.641,"15min":20.277999999999999},"last":2.0270000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":0.94399999999999995,"15min":0.85699999999999998},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":5.3040000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.119,"5min":1.05,"15min":0.91100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":4.8070000000000004},"last":3.2160000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.98199999999999998,"15min":3.3399999999999999},"min":{"1min":0.44400000000000001,"5min":0.437,"15min":0.32300000000000001},"max":{"1min":3.2639999999999998,"5min":3.2639999999999998,"15min":628.77099999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.93899999999999995,"15min":3.2679999999999998},"min":{"1min":0.36299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.524,"5min":4.524,"15min":624.19299999999998},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.0629999999999999,"15min":1.008},"min":{"1min":0.44800000000000001,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":5.0679999999999996,"5min":5.0679999999999996,"15min":5.5739999999999998},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.321,"5min":1.198,"15min":1.1220000000000001},"min":{"1min":0.52700000000000002,"5min":0.48999999999999999,"15min":0.35799999999999998},"max":{"1min":3.2909999999999999,"5min":3.2949999999999999,"15min":4.8769999999999998},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.052,"15min":3.3410000000000002},"min":{"1min":0.45100000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.2640000000000002,"5min":4.2640000000000002,"15min":626.26599999999996},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.095,"15min":3.4470000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":625.90700000000004},"last":0.48699999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.98099999999999998,"15min":1.05},"min":{"1min":0.44400000000000001,"5min":0.40400000000000003,"15min":0.39800000000000002},"max":{"1min":1.246,"5min":1.994,"15min":5.9210000000000003},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.1890000000000001,"15min":1.149},"min":{"1min":0.53500000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":5.3300000000000001},"last":0.83099999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.091,"15min":1.093},"min":{"1min":0.44400000000000001,"5min":0.432,"15min":0.372},"max":{"1min":1.5589999999999999,"5min":1.9830000000000001,"15min":4.827},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":1.0940000000000001,"15min":1.0960000000000001},"min":{"1min":0.442,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4550000000000001,"5min":3.859,"15min":4.8330000000000002},"last":0.65800000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1970000000000001,"15min":1.2110000000000001},"min":{"1min":0.48499999999999999,"5min":0.44900000000000001,"15min":0.372},"max":{"1min":4.782,"5min":4.782,"15min":4.9829999999999997},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.139,"15min":1.1990000000000001},"min":{"1min":0.51700000000000002,"5min":0.47299999999999998,"15min":0.41399999999999998},"max":{"1min":1.4039999999999999,"5min":2.9169999999999998,"15min":4.952},"last":0.68400000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.1319999999999999,"15min":1.3999999999999999},"min":{"1min":0.55100000000000005,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":1.4550000000000001,"5min":2.0249999999999999,"15min":30.917999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.878,"5min":1.157,"15min":1.4119999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.617,"5min":2.2400000000000002,"15min":30.742000000000001},"last":0.63100000000000001}]}]},{"osd":9,"up_from":74,"seq":317827580263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1339588,"kb_used_data":1030076,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92405564,"statfs":{"total":95995035648,"available":94623297536,"internally_reserved":0,"allocated":1054797824,"data_stored":1054408090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":0.98399999999999999,"15min":1.042},"min":{"1min":0.35699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.8330000000000002,"5min":5.8330000000000002,"15min":5.8330000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.85799999999999998,"15min":0.879},"min":{"1min":0.36299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.3029999999999999,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.98799999999999999,"15min":0.91900000000000004},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.253},"max":{"1min":5.8819999999999997,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":3.5329999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.91600000000000004,"15min":1.0409999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.29299999999999998},"max":{"1min":2.323,"5min":5.4560000000000004,"15min":7.9550000000000001},"last":3.544}]},{"osd":2,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.0569999999999999,"15min":1.391},"min":{"1min":0.45600000000000002,"5min":0.371,"15min":0.27600000000000002},"max":{"1min":5.915,"5min":5.915,"15min":97.275000000000006},"last":0.505},{"interface":"front","average":{"1min":1.109,"5min":0.99399999999999999,"15min":1.2609999999999999},"min":{"1min":0.36099999999999999,"5min":0.314,"15min":0.312},"max":{"1min":6.1779999999999999,"5min":6.1779999999999999,"15min":97.417000000000002},"last":0.496}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.036,"15min":0.93400000000000005},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.32200000000000001},"max":{"1min":3.528,"5min":4.569,"15min":4.569},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.1990000000000001,"15min":1.089},"min":{"1min":0.5,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.5609999999999999,"5min":4.8840000000000003,"15min":5.8010000000000002},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.2350000000000001,"15min":1.0469999999999999},"min":{"1min":0.52500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":5.899},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1679999999999999,"15min":1.131},"min":{"1min":0.46100000000000002,"5min":0.44700000000000001,"15min":0.40999999999999998},"max":{"1min":4.5540000000000003,"5min":5.4290000000000003,"15min":7.3739999999999997},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.0600000000000001,"15min":1.0940000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":3.3250000000000002,"5min":5.5250000000000004,"15min":7.3360000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.1200000000000001,"15min":1.0309999999999999},"min":{"1min":0.47099999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.0279999999999996,"5min":7.0609999999999999,"15min":7.0609999999999999},"last":0.61599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.3140000000000001,"15min":1.212},"min":{"1min":0.59299999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.8010000000000002,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":3.8420000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.2150000000000001,"15min":1.218},"min":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.438},"max":{"1min":2.9550000000000001,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.55100000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.145,"15min":1.0649999999999999},"min":{"1min":0.45700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.069,"5min":5.3620000000000001,"15min":5.4059999999999997},"last":0.623},{"interface":"front","average":{"1min":0.80500000000000005,"5min":1.155,"15min":1.1930000000000001},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.778,"5min":4.0419999999999998,"15min":7.25},"last":0.58599999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.304,"15min":1.2310000000000001},"min":{"1min":0.53400000000000003,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":3.8029999999999999,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.101,"5min":1.1890000000000001,"15min":1.1719999999999999},"min":{"1min":0.53900000000000003,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":4.423,"5min":4.4509999999999996,"15min":5.8700000000000001},"last":3.8319999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:25:42 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.3779999999999999,"15min":1.353},"min":{"1min":0.53100000000000003,"5min":0.48999999999999999,"15min":0.45600000000000002},"max":{"1min":5.7690000000000001,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.2729999999999999,"15min":1.323},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.823,"5min":5.492,"15min":6.3639999999999999},"last":0.65900000000000003}]}]},{"osd":8,"up_from":65,"seq":279172874608,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1338936,"kb_used_data":1041712,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92406216,"statfs":{"total":95995035648,"available":94623965184,"internally_reserved":0,"allocated":1066713088,"data_stored":1066329306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":1.946,"5min":0.98599999999999999,"15min":0.85899999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":19.196999999999999,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":2.2810000000000001,"5min":1.1319999999999999,"15min":0.88200000000000001},"min":{"1min":0.40699999999999997,"5min":0.29599999999999999,"15min":0.29399999999999998},"max":{"1min":18.584,"5min":18.584,"15min":18.584},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":5.4279999999999999,"5min":1.7230000000000001,"15min":1.4099999999999999},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":69.956000000000003,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.437},{"interface":"front","average":{"1min":5.0999999999999996,"5min":1.7669999999999999,"15min":1.498},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.315},"max":{"1min":71.010999999999996,"5min":71.010999999999996,"15min":71.010999999999996},"last":1.2649999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.79000000000000004,"15min":0.82499999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.7919999999999998,"5min":2.806,"15min":2.806},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.90500000000000003,"15min":0.98499999999999999},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.1539999999999999,"5min":2.278,"15min":2.278},"last":1.9930000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.96699999999999997,"15min":1.72},"min":{"1min":0.35199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.0490000000000004,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.97799999999999998,"15min":1.6599999999999999},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":5.0380000000000003,"5min":5.0380000000000003,"15min":224.71000000000001},"last":0.51900000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.82499999999999996,"15min":0.86899999999999999},"min":{"1min":0.373,"5min":0.37,"15min":0.32200000000000001},"max":{"1min":1.204,"5min":1.8200000000000001,"15min":4.8689999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0589999999999999,"15min":1.0229999999999999},"min":{"1min":0.38600000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":4.8339999999999996},"last":0.46899999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.131,"15min":1.718},"min":{"1min":0.44400000000000001,"5min":0.42399999999999999,"15min":0.37},"max":{"1min":3.903,"5min":5.29,"15min":221.36699999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.163,"15min":1.8600000000000001},"min":{"1min":0.5,"5min":0.437,"15min":0.42199999999999999},"max":{"1min":4.4930000000000003,"5min":5.556,"15min":221.203},"last":0.56000000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.0760000000000001,"15min":1.1080000000000001},"min":{"1min":0.51500000000000001,"5min":0.45400000000000001,"15min":0.42199999999999999},"max":{"1min":2.7109999999999999,"5min":3.4279999999999999,"15min":7.9160000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.2250000000000001,"15min":1.198},"min":{"1min":0.52700000000000002,"5min":0.50900000000000001,"15min":0.41599999999999998},"max":{"1min":2.444,"5min":3.419,"15min":7.899},"last":0.59699999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.173,"15min":1.1539999999999999},"min":{"1min":0.46800000000000003,"5min":0.375,"15min":0.35599999999999998},"max":{"1min":1.716,"5min":2.9430000000000001,"15min":4.5099999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.298,"15min":1.2090000000000001},"min":{"1min":0.60299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.8240000000000001,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.61699999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.01,"15min":1.3979999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":109.92},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.05,"5min":1.115,"15min":1.4650000000000001},"min":{"1min":0.48299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0250000000000004,"5min":5.0250000000000004,"15min":110.249},"last":0.54400000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.159,"15min":1.1839999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.441},"max":{"1min":1.0509999999999999,"5min":2.9159999999999999,"15min":5.0670000000000002},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.085,"15min":1.151},"min":{"1min":0.497,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.80600000000000005,"5min":2.1259999999999999,"15min":4.6639999999999997},"last":0.746}]}]},{"osd":7,"up_from":56,"seq":240518168950,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":847224,"kb_used_data":533616,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":92897928,"statfs":{"total":95995035648,"available":95127478272,"internally_reserved":0,"allocated":546422784,"data_stored":546039002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.93899999999999995,"15min":0.91500000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.952,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.311},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.99399999999999999,"15min":0.93600000000000005},"min":{"1min":0.28299999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.7730000000000001,"5min":7.7699999999999996,"15min":7.7699999999999996},"last":1.0720000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.2769999999999999,"15min":1.008},"min":{"1min":0.313,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":4.5179999999999998,"5min":35.655000000000001,"15min":35.655000000000001},"last":2.0409999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.6339999999999999,"15min":1.1859999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.6719999999999997,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.1259999999999999,"15min":1.274},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.5140000000000002,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.90900000000000003,"15min":0.98899999999999999},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.4699999999999998,"5min":4.4699999999999998,"15min":4.4699999999999998},"last":1.369}]},{"osd":4,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.0600000000000001,"15min":0.93799999999999994},"min":{"1min":0.34100000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.4420000000000002,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.90300000000000002,"15min":1.0089999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":4.7389999999999999,"5min":4.7389999999999999,"15min":6.0090000000000003},"last":1.3160000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.98199999999999998,"15min":0.96499999999999997},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.5620000000000003,"5min":4.7220000000000004,"15min":4.9050000000000002},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.024,"15min":1.036},"min":{"1min":0.38300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":5.766,"5min":7.5640000000000001,"15min":9.3640000000000008},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.125,"15min":1.101},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":6.319,"5min":6.319,"15min":6.319},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.141,"15min":1.1799999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":1.3400000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":28.620999999999999,"5min":6.6399999999999997,"15min":2.976},"min":{"1min":0.44600000000000001,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":498.68299999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":28.295999999999999,"5min":6.6840000000000002,"15min":3.028},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":498.69999999999999,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.52000000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":1.087,"15min":1.2130000000000001},"min":{"1min":0.442,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.6299999999999999,"5min":2.9860000000000002,"15min":12.907999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.2010000000000001,"15min":1.2390000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.42699999999999999},"max":{"1min":4.665,"5min":4.665,"15min":6.0199999999999996},"last":1.415}]},{"osd":10,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":34.110999999999997,"5min":7.742,"15min":3.4820000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":497.52699999999999,"5min":497.52699999999999,"15min":497.52699999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":33.834000000000003,"5min":7.7649999999999997,"15min":3.3940000000000001},"min":{"1min":0.45300000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":496.29599999999999,"5min":496.29599999999999,"15min":496.29599999999999},"last":0.55700000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.236,"15min":1.6659999999999999},"min":{"1min":0.53100000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":4.6890000000000001,"5min":6.0490000000000004,"15min":120.943},"last":0.495},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.2370000000000001,"15min":1.738},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.45300000000000001},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":120.661},"last":1.4379999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430592,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1857668,"kb_used_data":1556348,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91887484,"statfs":{"total":95995035648,"available":94092783616,"internally_reserved":0,"allocated":1593700352,"data_stored":1593310618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.1659999999999999,"15min":1.077},"min":{"1min":0.26200000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.88,"5min":30.454999999999998,"15min":30.454999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.274,"15min":1.163},"min":{"1min":0.33300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.6930000000000001,"5min":30.573,"15min":30.573},"last":3.5459999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.97999999999999998,"15min":1.887},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":3.5129999999999999,"5min":3.605,"15min":252.22800000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.93100000000000005,"15min":1.8859999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.702,"5min":4.4880000000000004,"15min":252.61199999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":1.2490000000000001,"15min":0.94899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":28.739999999999998,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":3.0169999999999999,"5min":1.4530000000000001,"15min":1.0529999999999999},"min":{"1min":0.32300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":32.328000000000003,"5min":32.328000000000003,"15min":32.328000000000003},"last":2.0979999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.0569999999999999,"15min":1.9079999999999999},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27600000000000002},"max":{"1min":2.4020000000000001,"5min":7.7679999999999998,"15min":284.05599999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":1.0800000000000001,"15min":2.036},"min":{"1min":0.33200000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.7250000000000001,"5min":7.5979999999999999,"15min":284.041},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.96999999999999997,"15min":1.871},"min":{"1min":0.379,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.6019999999999999,"5min":3.3809999999999998,"15min":281.49700000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.99399999999999999,"15min":1.9470000000000001},"min":{"1min":0.374,"5min":0.37,"15min":0.33500000000000002},"max":{"1min":1.8959999999999999,"5min":4.4749999999999996,"15min":281.61900000000003},"last":0.66100000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.1830000000000001,"15min":1.097},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.7210000000000001,"5min":7.641,"15min":7.641},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.1519999999999999,"15min":1.093},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.641,"5min":7.8380000000000001,"15min":7.8380000000000001},"last":0.51800000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.0660000000000001,"15min":1.077},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.4910000000000001,"5min":3.351,"15min":4.2380000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.169,"15min":1.1859999999999999},"min":{"1min":0.40799999999999997,"5min":0.371,"15min":0.371},"max":{"1min":1.7649999999999999,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":0.48599999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":1.163,"15min":1.206},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.0039999999999996,"5min":4.5330000000000004,"15min":18.059999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.2310000000000001,"15min":1.288},"min":{"1min":0.51100000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.6910000000000001,"5min":4.6890000000000001,"15min":17.425000000000001},"last":0.54300000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.242,"15min":1.216},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.4580000000000002,"5min":3.3140000000000001,"15min":6.5300000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.002,"5min":1.331,"15min":1.3009999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.5779999999999998,"5min":3.4460000000000002,"15min":6.5730000000000004},"last":0.57899999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:25:45 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.278,"15min":3.637},"min":{"1min":0.53500000000000003,"5min":0.499,"15min":0.499},"max":{"1min":3.3809999999999998,"5min":3.4129999999999998,"15min":139.727},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.746,"5min":1.2629999999999999,"15min":3.5059999999999998},"min":{"1min":0.57399999999999995,"5min":0.504,"15min":0.504},"max":{"1min":1.01,"5min":4.5899999999999999,"15min":139.24199999999999},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575270,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2650692,"kb_used_data":2048700,"kb_used_omap":0,"kb_used_meta":601984,"kb_avail":91094460,"statfs":{"total":95995035648,"available":93280727040,"internally_reserved":0,"allocated":2097868800,"data_stored":2097479066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":616431616},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.877,"15min":1.353},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":2.581,"5min":3.9359999999999999,"15min":114.40000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.883,"5min":1.1419999999999999,"15min":1.577},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":7.7249999999999996,"5min":7.7249999999999996,"15min":115.09099999999999},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.0049999999999999,"15min":0.98599999999999999},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":4.2720000000000002,"5min":6.5389999999999997,"15min":9.8599999999999994},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.74,"5min":1.198,"15min":1.119},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":9.6120000000000001,"5min":9.6120000000000001,"15min":9.6120000000000001},"last":0.28899999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0840000000000001,"15min":1.1180000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":4.0430000000000001,"5min":4.0430000000000001,"15min":4.0430000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.393,"5min":1.012,"15min":0.89500000000000002},"min":{"1min":0.29499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.306,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.39100000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:15 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.216,"15min":1.103},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.7479999999999993,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.962,"5min":1.492,"15min":1.4139999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":7.1710000000000003,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":1.351,"15min":1.1819999999999999},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":10.109999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.627},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.2130000000000001,"15min":1.1899999999999999},"min":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.35899999999999999},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":7.0110000000000001},"last":0.46899999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:25:42 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.3089999999999999,"15min":1.679},"min":{"1min":0.45200000000000001,"5min":0.38500000000000001,"15min":0.34699999999999998},"max":{"1min":6.6310000000000002,"5min":6.6310000000000002,"15min":55.869999999999997},"last":0.439},{"interface":"front","average":{"1min":1.327,"5min":1.3169999999999999,"15min":1.73},"min":{"1min":0.47399999999999998,"5min":0.46600000000000003,"15min":0.36799999999999999},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":55.892000000000003},"last":0.54500000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:25:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4319999999999999,"15min":1.357},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.436},"max":{"1min":4.4900000000000002,"5min":4.4900000000000002,"15min":7.2190000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.357,"15min":1.371},"min":{"1min":0.499,"5min":0.499,"15min":0.44400000000000001},"max":{"1min":5.6639999999999997,"5min":5.6639999999999997,"15min":10.119999999999999},"last":0.59099999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.464,"15min":1.3919999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.45400000000000001},"max":{"1min":5.7240000000000002,"5min":5.7240000000000002,"15min":10.07},"last":1.361},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.4019999999999999,"15min":1.506},"min":{"1min":0.55400000000000005,"5min":0.46700000000000003,"15min":0.45000000000000001},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":7.6059999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:25:42 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.4950000000000001,"15min":1.3740000000000001},"min":{"1min":0.55300000000000005,"5min":0.499,"15min":0.499},"max":{"1min":5.694,"5min":5.694,"15min":10.157999999999999},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.8340000000000001,"5min":1.5549999999999999,"15min":1.516},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.53700000000000003},"max":{"1min":6.4690000000000003,"5min":6.4690000000000003,"15min":8.7720000000000002},"last":1.1759999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.421,"15min":1.534},"min":{"1min":0.60199999999999998,"5min":0.58099999999999996,"15min":0.443},"max":{"1min":3.996,"5min":4.5609999999999999,"15min":7.1600000000000001},"last":4.3040000000000003},{"interface":"front","average":{"1min":1.659,"5min":1.609,"15min":1.6180000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.47599999999999998},"max":{"1min":5.2949999999999999,"5min":5.7549999999999999,"15min":7.2489999999999997},"last":0.72699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738801,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2902980,"kb_used_data":2601660,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":90842172,"statfs":{"total":95995035648,"available":93022384128,"internally_reserved":0,"allocated":2664099840,"data_stored":2663710106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.88,"15min":1.085},"min":{"1min":0.245,"5min":0.245,"15min":0.215},"max":{"1min":10.754,"5min":10.754,"15min":11.324},"last":4.4260000000000002},{"interface":"front","average":{"1min":1.0820000000000001,"5min":0.94999999999999996,"15min":1.044},"min":{"1min":0.34200000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.617,"5min":4.1550000000000002,"15min":9.8249999999999993},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:25:41 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.085,"15min":1.0649999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.6059999999999999,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.318},{"interface":"front","average":{"1min":1.032,"5min":0.94499999999999995,"15min":0.93100000000000005},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.1499999999999999,"15min":1.698},"min":{"1min":0.32900000000000001,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":2.7429999999999999,"5min":7.6219999999999999,"15min":116.187},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.3380000000000001,"15min":1.8200000000000001},"min":{"1min":0.34399999999999997,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":7.5599999999999996,"5min":7.5599999999999996,"15min":116.163},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":0.99199999999999999,"15min":1.4199999999999999},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.1539999999999999,"5min":4.1539999999999999,"15min":116.005},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.1120000000000001,"15min":1.5009999999999999},"min":{"1min":0.40100000000000002,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":115.973},"last":1.8120000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:26:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.2430000000000001,"15min":1.327},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.4249999999999998,"5min":4.9119999999999999,"15min":81.147000000000006},"last":2.089},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.347,"15min":1.405},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":8.8789999999999996,"5min":8.8789999999999996,"15min":80.233000000000004},"last":1.853}]},{"osd":7,"last update":"Mon Feb 12 16:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.262,"15min":1.5800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.36699999999999999},"max":{"1min":4.194,"5min":4.194,"15min":115.26000000000001},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.3700000000000001,"15min":1.6859999999999999},"min":{"1min":0.52900000000000003,"5min":0.51300000000000001,"15min":0.44800000000000001},"max":{"1min":4.2279999999999998,"5min":4.2279999999999998,"15min":115.402},"last":2.0550000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.2609999999999999,"15min":1.2370000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":4.806},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.363,"15min":1.333},"min":{"1min":0.55900000000000005,"5min":0.51800000000000002,"15min":0.41299999999999998},"max":{"1min":4.3869999999999996,"5min":4.3869999999999996,"15min":4.8380000000000001},"last":1.992}]},{"osd":9,"last update":"Mon Feb 12 16:26:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.179,"15min":1.3500000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.42699999999999999},"max":{"1min":3.165,"5min":7.6539999999999999,"15min":46.246000000000002},"last":4.3920000000000003},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.401,"15min":1.401},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45800000000000002},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":46.167999999999999},"last":1.9239999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.274,"15min":1.2609999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":2.8500000000000001,"5min":4.2670000000000003,"15min":6.0979999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.077,"5min":1.3560000000000001,"15min":1.296},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.47999999999999998},"max":{"1min":2.875,"5min":4.5570000000000004,"15min":6.2480000000000002},"last":0.78700000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.25,"15min":1.5209999999999999},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.434},"max":{"1min":2.8079999999999998,"5min":4.5220000000000002,"15min":46.064},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.3580000000000001,"15min":1.48},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.38300000000000001},"max":{"1min":4.2750000000000004,"5min":4.2750000000000004,"15min":45.642000000000003},"last":1.958}]}]},{"osd":2,"up_from":21,"seq":90194313631,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1847172,"kb_used_data":1537660,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91897980,"statfs":{"total":95995035648,"available":94103531520,"internally_reserved":0,"allocated":1574563840,"data_stored":1574174106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.034,"15min":1.0509999999999999},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":8.5399999999999991},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.8220000000000001,"5min":1.2470000000000001,"15min":1.1579999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":6.484,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":1.222}]},{"osd":3,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.7689999999999999,"15min":2.0070000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.8460000000000001,"5min":21.407,"15min":21.407},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.169,"5min":0.86599999999999999,"15min":0.84199999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.887,"5min":3.887,"15min":3.887},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.0580000000000001,"15min":0.93500000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":4.5350000000000001,"5min":6.8090000000000002,"15min":7.2859999999999996},"last":0.441},{"interface":"front","average":{"1min":1.222,"5min":1.2509999999999999,"15min":1.071},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.7000000000000002,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.34300000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.226,"15min":1.095},"min":{"1min":0.39400000000000002,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":5.9489999999999998,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.482,"5min":1.175,"15min":1.0509999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":6.5730000000000004},"last":0.496}]},{"osd":6,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.472,"15min":1.157},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.6890000000000001,"5min":21.527000000000001,"15min":21.527000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.1839999999999999,"15min":1.089},"min":{"1min":0.40500000000000003,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":3.98,"5min":3.98,"15min":5.4349999999999996},"last":0.51500000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.1399999999999999,"15min":1.0549999999999999},"min":{"1min":0.497,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.8570000000000002,"5min":4.8570000000000002,"15min":5.2960000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.206,"5min":1.2230000000000001,"15min":1.147},"min":{"1min":0.443,"5min":0.443,"15min":0.38700000000000001},"max":{"1min":3.6619999999999999,"5min":3.8759999999999999,"15min":5.0739999999999998},"last":0.67000000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.2649999999999999,"15min":1.1519999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.38700000000000001},"max":{"1min":2.6960000000000002,"5min":4.1539999999999999,"15min":5.101},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.383,"15min":1.22},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":5.9269999999999996,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":0.42499999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.272,"15min":1.1839999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.40600000000000003},"max":{"1min":4.8040000000000003,"5min":4.8040000000000003,"15min":7.2830000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.401,"15min":1.232},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.42399999999999999},"max":{"1min":4.9089999999999998,"5min":4.9089999999999998,"15min":5.085},"last":0.55900000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.4339999999999999,"15min":1.3029999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6609999999999996,"5min":11.454000000000001,"15min":11.454000000000001},"last":1.194},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.484,"15min":1.3220000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":6.6340000000000003,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":1.1559999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3140000000000001,"15min":1.4870000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":3.859,"5min":4.0590000000000002,"15min":4.0590000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4279999999999999,"15min":1.5649999999999999},"min":{"1min":0.51800000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.8250000000000002,"5min":3.9910000000000001,"15min":3.9910000000000001},"last":0.70699999999999996}]}]},{"osd":3,"up_from":522,"seq":2241972928556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853956,"kb_used_data":521724,"kb_used_omap":91,"kb_used_meta":332132,"kb_avail":92891196,"statfs":{"total":95995035648,"available":95120584704,"internally_reserved":0,"allocated":534245376,"data_stored":533855642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":340103184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.77600000000000002,"15min":0.82299999999999995},"min":{"1min":0.28599999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":3.4750000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.83299999999999996,"15min":0.84599999999999997},"min":{"1min":0.307,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":3.6840000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.92300000000000004,"15min":0.88700000000000001},"min":{"1min":0.32800000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.5279999999999996},"last":3.5880000000000001},{"interface":"front","average":{"1min":1.593,"5min":1.1399999999999999,"15min":1.083},"min":{"1min":0.313,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":6.5030000000000001,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":3.4980000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.111,"15min":1.321},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.8959999999999999,"5min":10.004,"15min":10.004},"last":4.4210000000000003},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0369999999999999,"15min":1.1000000000000001},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.3029999999999999,"5min":4.3029999999999999,"15min":4.3029999999999999},"last":3.6339999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2729999999999999,"15min":1.389},"min":{"1min":0.35099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.4189999999999996,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":3.7530000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.88800000000000001,"15min":0.92800000000000005},"min":{"1min":0.34899999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.3639999999999999,"5min":3.593,"15min":3.593},"last":4.5339999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1060000000000001,"15min":1.141},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4779999999999998,"5min":4.4779999999999998,"15min":4.4779999999999998},"last":3.794},{"interface":"front","average":{"1min":1.1200000000000001,"5min":0.98699999999999999,"15min":1.0069999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":3.448,"5min":3.7530000000000001,"15min":3.7530000000000001},"last":6.1920000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.2709999999999999,"15min":1.4630000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":3.2839999999999998,"5min":6.851,"15min":6.851},"last":4.6289999999999996},{"interface":"front","average":{"1min":1.171,"5min":1.2170000000000001,"15min":1.2769999999999999},"min":{"1min":0.51000000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.3529999999999998,"5min":4.3529999999999998,"15min":4.3529999999999998},"last":3.7010000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.137,"15min":1.204},"min":{"1min":0.54400000000000004,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":3.899},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.018,"15min":1.1060000000000001},"min":{"1min":0.42699999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.9990000000000001,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":4.6589999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3420000000000001,"15min":1.3480000000000001},"min":{"1min":0.54800000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":5.0830000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":3.7389999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.3140000000000001,"15min":1.373},"min":{"1min":0.502,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":4.5800000000000001},"last":3.8319999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.409,"15min":1.54},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":3.867},{"interface":"front","average":{"1min":1.097,"5min":1.274,"15min":1.3140000000000001},"min":{"1min":0.56100000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":4.4969999999999999,"5min":4.4969999999999999,"15min":4.4969999999999999},"last":3.7759999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.234,"15min":1.2809999999999999},"min":{"1min":0.63900000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4349999999999996,"5min":4.4349999999999996,"15min":4.4349999999999996},"last":3.8500000000000001},{"interface":"front","average":{"1min":1.05,"5min":1.399,"15min":1.5329999999999999},"min":{"1min":0.60399999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.633,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":3.9359999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953871,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2411704,"kb_used_data":2102192,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91333448,"statfs":{"total":95995035648,"available":93525450752,"internally_reserved":0,"allocated":2152644608,"data_stored":2152260826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.0409999999999999,"15min":1.397},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":90.597999999999999},"last":1.994},{"interface":"front","average":{"1min":1.048,"5min":1.0169999999999999,"15min":1.5289999999999999},"min":{"1min":0.35799999999999998,"5min":0.30599999999999999,"15min":0.28399999999999997},"max":{"1min":5.056,"5min":5.056,"15min":90.676000000000002},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.88500000000000001,"15min":0.92700000000000005},"min":{"1min":0.36099999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.504,"5min":3.3519999999999999,"15min":8.0530000000000008},"last":0.31},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.012,"15min":1.101},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":4.4980000000000002,"5min":5.5709999999999997,"15min":7.452},"last":4.1109999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.0840000000000001,"15min":1.0880000000000001},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":8.3040000000000003,"5min":8.3040000000000003,"15min":8.3040000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.034,"15min":1.0509999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.5410000000000004,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:26:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.98299999999999998,"15min":0.95799999999999996},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.7009999999999996,"5min":5.5999999999999996,"15min":6.0460000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.51,"5min":1.177,"15min":1.1060000000000001},"min":{"1min":0.34599999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.7160000000000002,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.92800000000000005,"15min":1.046},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.673,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":2.0960000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.077,"15min":1.0580000000000001},"min":{"1min":0.35299999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.2040000000000002,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":0.435}]},{"osd":7,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.052,"15min":1.163},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":6.242,"5min":6.242,"15min":9.0879999999999992},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.0229999999999999,"15min":1.2110000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.5449999999999999,"5min":3.3799999999999999,"15min":7.6689999999999996},"last":0.35499999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1120000000000001,"15min":1.232},"min":{"1min":0.51900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.6099999999999999,"5min":3.3079999999999998,"15min":7.8650000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.331,"15min":1.2589999999999999},"min":{"1min":0.42799999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.4489999999999998,"5min":5.6699999999999999,"15min":5.6699999999999999},"last":0.63100000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.198,"15min":1.4790000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.73,"5min":3.2650000000000001,"15min":34.112000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.3700000000000001,"15min":1.625},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":5.569,"5min":5.6420000000000003,"15min":34.590000000000003},"last":0.59599999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:25:46 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.3620000000000001,"15min":1.383},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.54,"5min":5.6970000000000001,"15min":6.9770000000000003},"last":2.044},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.216,"15min":1.2909999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.9350000000000001,"5min":4.46,"15min":8.5299999999999994},"last":0.58199999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.266,"15min":1.5449999999999999},"min":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.7010000000000001,"5min":3.2320000000000002,"15min":4.6500000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.421,"15min":1.5249999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":5.5060000000000002,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.66300000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724935,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2354952,"kb_used_data":2045440,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91390200,"statfs":{"total":95995035648,"available":93583564800,"internally_reserved":0,"allocated":2094530560,"data_stored":2094136754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80200000000000005,"15min":2.702},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.25800000000000001},"max":{"1min":3.331,"5min":3.8690000000000002,"15min":565.34500000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.91600000000000004,"15min":2.7829999999999999},"min":{"1min":0.39300000000000002,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":3.3660000000000001,"5min":3.9399999999999999,"15min":565.98400000000004},"last":0.503}]},{"osd":2,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.3440000000000001,"15min":1.097},"min":{"1min":0.36099999999999999,"5min":0.33200000000000002,"15min":0.30499999999999999},"max":{"1min":3.9289999999999998,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.361,"15min":1.129},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.194,"5min":40.225000000000001,"15min":40.225000000000001},"last":0.438}]},{"osd":3,"last update":"Mon Feb 12 16:26:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.4169999999999998,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":2.5150000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.387,"15min":1.4299999999999999},"min":{"1min":0.317,"5min":0.246,"15min":0.246},"max":{"1min":41.908000000000001,"5min":41.908000000000001,"15min":75.257000000000005},"last":1.3680000000000001},{"interface":"front","average":{"1min":3.4350000000000001,"5min":1.4370000000000001,"15min":1.4259999999999999},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":42.430999999999997,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:26:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0580000000000001,"15min":1.2949999999999999},"min":{"1min":0.34200000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":74.105000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.135,"15min":1.413},"min":{"1min":0.47999999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":75.013999999999996},"last":3.3839999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.012,"15min":1.026},"min":{"1min":0.374,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1120000000000001,"5min":6.1120000000000001,"15min":6.6630000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.0349999999999999,"15min":1.107},"min":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":5.6900000000000004,"5min":5.6900000000000004,"15min":5.9909999999999997},"last":0.496}]},{"osd":8,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.097,"15min":1.1930000000000001},"min":{"1min":0.47499999999999998,"5min":0.41699999999999998,"15min":0.40200000000000002},"max":{"1min":2.6859999999999999,"5min":4.0069999999999997,"15min":5.9509999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1619999999999999,"15min":1.222},"min":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":6.0739999999999998,"5min":6.0739999999999998,"15min":6.0739999999999998},"last":0.70899999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:26:05 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.359,"15min":3.2469999999999999},"min":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.42999999999999999},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":449.89499999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.167,"15min":3.2429999999999999},"min":{"1min":0.55100000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.052,"5min":3.0840000000000001,"15min":448.70800000000003},"last":0.65700000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.1220000000000001,"15min":1.2589999999999999},"min":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.7639999999999998,"5min":4.0410000000000004,"15min":5.9210000000000003},"last":0.747},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.304,"15min":1.3009999999999999},"min":{"1min":0.56999999999999995,"5min":0.52400000000000002,"15min":0.5},"max":{"1min":5.7949999999999999,"5min":5.7949999999999999,"15min":6.1379999999999999},"last":0.64600000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2070000000000001,"15min":1.204},"min":{"1min":0.503,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.806,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.367,"15min":1.3120000000000001},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":5.8230000000000004,"5min":5.8230000000000004,"15min":5.8230000000000004},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2646474752,"data_stored":2646474752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2087059456,"data_stored":2087059456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1561919488,"data_stored":1561919488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":529465344,"data_stored":529465344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2142896128,"data_stored":2142896128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2092830720,"data_stored":2092826648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1590886400,"data_stored":1590886400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":544342016,"data_stored":544342016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1058078720,"data_stored":1058078720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1047003136,"data_stored":1047003136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1099563008,"data_stored":1099563008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":534581248,"data_stored":534577176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:26:40.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:26:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:40.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:40.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:40 smithi123 bash[19119]: cluster 2024-02-12T16:26:39.968428+0000 mgr.y (mgr.14146) 1249 : cluster [DBG] pgmap v1735: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 7.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 15686/255482 objects misplaced (6.140%); 460 KiB/s, 7 objects/s recovering 2024-02-12T16:26:40.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:40.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:40.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:40.969 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:40 smithi169 bash[19604]: cluster 2024-02-12T16:26:39.968428+0000 mgr.y (mgr.14146) 1249 : cluster [DBG] pgmap v1735: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 7.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 15686/255482 objects misplaced (6.140%); 460 KiB/s, 7 objects/s recovering 2024-02-12T16:26:41.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:40 smithi121 bash[18489]: cluster 2024-02-12T16:26:39.968428+0000 mgr.y (mgr.14146) 1249 : cluster [DBG] pgmap v1735: 37 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 35 active+clean; 7.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 15686/255482 objects misplaced (6.140%); 460 KiB/s, 7 objects/s recovering 2024-02-12T16:26:41.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:41.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:41.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:41.255 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:41.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:41.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:41.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:41.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:41.503 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:41.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:41.560 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:26:41.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:41.660 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:41.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:26:41.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:41.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:41.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:41 smithi123 bash[19119]: audit 2024-02-12T16:26:41.537265+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? v1:172.21.15.121:0/1583945821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:26:41.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:41.924 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:41.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:42.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:42.034 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:41 smithi121 bash[18489]: audit 2024-02-12T16:26:41.537265+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? v1:172.21.15.121:0/1583945821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:26:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:41 smithi169 bash[19604]: audit 2024-02-12T16:26:41.537265+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? v1:172.21.15.121:0/1583945821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:26:42.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:42.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:42.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:42.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:42.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:42.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:42.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:42.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:42.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:42.478 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:42.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:42.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:42.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:42.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:42.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:42.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:42.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:42.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:26:42.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:42 smithi123 bash[19119]: cluster 2024-02-12T16:26:41.687184+0000 mon.a (mon.0) 2124 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:42 smithi123 bash[19119]: audit 2024-02-12T16:26:41.694604+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? v1:172.21.15.121:0/1583945821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:26:42.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:42 smithi123 bash[19119]: cluster 2024-02-12T16:26:41.694656+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e595: 12 total, 12 up, 12 in 2024-02-12T16:26:42.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:42 smithi123 bash[19119]: cluster 2024-02-12T16:26:41.968866+0000 mgr.y (mgr.14146) 1250 : cluster [DBG] pgmap v1737: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 14936/249152 objects misplaced (5.995%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:42.931 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:42.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:43.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:42 smithi169 bash[19604]: cluster 2024-02-12T16:26:41.687184+0000 mon.a (mon.0) 2124 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:42 smithi169 bash[19604]: audit 2024-02-12T16:26:41.694604+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? v1:172.21.15.121:0/1583945821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:26:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:42 smithi169 bash[19604]: cluster 2024-02-12T16:26:41.694656+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e595: 12 total, 12 up, 12 in 2024-02-12T16:26:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:42 smithi169 bash[19604]: cluster 2024-02-12T16:26:41.968866+0000 mgr.y (mgr.14146) 1250 : cluster [DBG] pgmap v1737: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 14936/249152 objects misplaced (5.995%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:43.050 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:42 smithi121 bash[18489]: cluster 2024-02-12T16:26:41.687184+0000 mon.a (mon.0) 2124 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:26:43.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:42 smithi121 bash[18489]: audit 2024-02-12T16:26:41.694604+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? v1:172.21.15.121:0/1583945821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:26:43.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:42 smithi121 bash[18489]: cluster 2024-02-12T16:26:41.694656+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e595: 12 total, 12 up, 12 in 2024-02-12T16:26:43.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:42 smithi121 bash[18489]: cluster 2024-02-12T16:26:41.968866+0000 mgr.y (mgr.14146) 1250 : cluster [DBG] pgmap v1737: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 7.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 14936/249152 objects misplaced (5.995%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:26:43.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:43.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:43.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:43.234 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:43.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:43.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:43.391 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:43.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:43.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:43.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:43.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:43.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:43.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:43.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:43.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:43.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:43.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:43.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:43 smithi121 bash[18489]: cluster 2024-02-12T16:26:42.225737+0000 osd.2 (osd.2) 78 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:26:43.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:43 smithi121 bash[18489]: cluster 2024-02-12T16:26:42.266454+0000 osd.2 (osd.2) 79 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:26:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:43 smithi121 bash[18489]: cluster 2024-02-12T16:26:42.961886+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.d scrub starts 2024-02-12T16:26:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:43 smithi121 bash[18489]: cluster 2024-02-12T16:26:43.025863+0000 osd.7 (osd.7) 141 : cluster [DBG] 1.d scrub ok 2024-02-12T16:26:43.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:43.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:43.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:43 smithi169 bash[19604]: cluster 2024-02-12T16:26:42.225737+0000 osd.2 (osd.2) 78 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:26:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:43 smithi169 bash[19604]: cluster 2024-02-12T16:26:42.266454+0000 osd.2 (osd.2) 79 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:26:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:43 smithi169 bash[19604]: cluster 2024-02-12T16:26:42.961886+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.d scrub starts 2024-02-12T16:26:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:43 smithi169 bash[19604]: cluster 2024-02-12T16:26:43.025863+0000 osd.7 (osd.7) 141 : cluster [DBG] 1.d scrub ok 2024-02-12T16:26:44.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:44.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:43 smithi123 bash[19119]: cluster 2024-02-12T16:26:42.225737+0000 osd.2 (osd.2) 78 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:26:44.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:43 smithi123 bash[19119]: cluster 2024-02-12T16:26:42.266454+0000 osd.2 (osd.2) 79 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:26:44.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:43 smithi123 bash[19119]: cluster 2024-02-12T16:26:42.961886+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.d scrub starts 2024-02-12T16:26:44.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:43 smithi123 bash[19119]: cluster 2024-02-12T16:26:43.025863+0000 osd.7 (osd.7) 141 : cluster [DBG] 1.d scrub ok 2024-02-12T16:26:44.099 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:44.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:44.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:44.234 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:44.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:44.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:44.378 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:44.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:44.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:44.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:44.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:44.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:44.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:44.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:44.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:44.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:44.763 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:44.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:44.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:44 smithi169 bash[19604]: cluster 2024-02-12T16:26:43.969364+0000 mgr.y (mgr.14146) 1251 : cluster [DBG] pgmap v1738: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 14936/245020 objects misplaced (6.096%); 920 KiB/s, 14 objects/s recovering 2024-02-12T16:26:44.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:44.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:26:44.898 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:44.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:45.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:45.029 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:45.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:45.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:44 smithi123 bash[19119]: cluster 2024-02-12T16:26:43.969364+0000 mgr.y (mgr.14146) 1251 : cluster [DBG] pgmap v1738: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 14936/245020 objects misplaced (6.096%); 920 KiB/s, 14 objects/s recovering 2024-02-12T16:26:45.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:44 smithi121 bash[18489]: cluster 2024-02-12T16:26:43.969364+0000 mgr.y (mgr.14146) 1251 : cluster [DBG] pgmap v1738: 37 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 35 active+clean; 7.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 14936/245020 objects misplaced (6.096%); 920 KiB/s, 14 objects/s recovering 2024-02-12T16:26:45.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:45.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:45.295 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:45.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:45.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:45.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:45.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:45.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:45.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:45.848 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:45.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:45.961 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:45 smithi123 bash[19119]: cluster 2024-02-12T16:26:44.420489+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:45.961 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:45 smithi123 bash[19119]: cluster 2024-02-12T16:26:44.420533+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:45.961 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:45 smithi123 bash[19119]: cluster 2024-02-12T16:26:44.705166+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e596: 12 total, 12 up, 12 in 2024-02-12T16:26:45.983 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:45.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:45 smithi169 bash[19604]: cluster 2024-02-12T16:26:44.420489+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:45 smithi169 bash[19604]: cluster 2024-02-12T16:26:44.420533+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:45 smithi169 bash[19604]: cluster 2024-02-12T16:26:44.705166+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e596: 12 total, 12 up, 12 in 2024-02-12T16:26:46.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:46.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:45 smithi121 bash[18489]: cluster 2024-02-12T16:26:44.420489+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:46.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:45 smithi121 bash[18489]: cluster 2024-02-12T16:26:44.420533+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:46.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:45 smithi121 bash[18489]: cluster 2024-02-12T16:26:44.705166+0000 mon.a (mon.0) 2127 : cluster [DBG] osdmap e596: 12 total, 12 up, 12 in 2024-02-12T16:26:46.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:46.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:46.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:46.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:46.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:46.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:46.396 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:46.399 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:46.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:46.400 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:26:46.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:46.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:46.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:46.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:46.683 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:46.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:46.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:46.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:46.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:46.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:46 smithi121 bash[18489]: cluster 2024-02-12T16:26:45.969805+0000 mgr.y (mgr.14146) 1252 : cluster [DBG] pgmap v1740: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 14535/240824 objects misplaced (6.036%); 824 KiB/s, 12 objects/s recovering 2024-02-12T16:26:46.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:46.918 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:46 smithi123 bash[19119]: cluster 2024-02-12T16:26:45.969805+0000 mgr.y (mgr.14146) 1252 : cluster [DBG] pgmap v1740: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 14535/240824 objects misplaced (6.036%); 824 KiB/s, 12 objects/s recovering 2024-02-12T16:26:46.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:46.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:47.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:46 smithi169 bash[19604]: cluster 2024-02-12T16:26:45.969805+0000 mgr.y (mgr.14146) 1252 : cluster [DBG] pgmap v1740: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 14535/240824 objects misplaced (6.036%); 824 KiB/s, 12 objects/s recovering 2024-02-12T16:26:47.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:47.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:47.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:47.219 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:47.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:47.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:47.350 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:47.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:47.416 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:47.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:47.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:47.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:47.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:47.616 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:47.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:47.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:47.765 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:47.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:47.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:47.893 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:47.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:47.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:48.005 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:47 smithi169 bash[19604]: cluster 2024-02-12T16:26:46.705233+0000 mon.a (mon.0) 2128 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:26:48.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:48.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:48.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:47 smithi123 bash[19119]: cluster 2024-02-12T16:26:46.705233+0000 mon.a (mon.0) 2128 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:26:48.152 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:48.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:47 smithi121 bash[18489]: cluster 2024-02-12T16:26:46.705233+0000 mon.a (mon.0) 2128 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:26:48.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:48.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:48.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:48.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:48.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:48.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:48.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:48.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:48.793 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:48.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:48.904 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:48 smithi169 bash[19604]: cluster 2024-02-12T16:26:47.970252+0000 mgr.y (mgr.14146) 1253 : cluster [DBG] pgmap v1741: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 14143/235116 objects misplaced (6.015%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:26:48.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:48.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:49.031 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:48 smithi121 bash[18489]: cluster 2024-02-12T16:26:47.970252+0000 mgr.y (mgr.14146) 1253 : cluster [DBG] pgmap v1741: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 14143/235116 objects misplaced (6.015%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:26:49.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:49.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:49.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:48 smithi123 bash[19119]: cluster 2024-02-12T16:26:47.970252+0000 mgr.y (mgr.14146) 1253 : cluster [DBG] pgmap v1741: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 14143/235116 objects misplaced (6.015%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:26:49.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:49.204 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:49.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:49.307 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:49.329 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:49.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:49.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:49.438 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:26:49.458 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:49.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:49.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:49.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:49.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:49.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:49.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:49.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:49.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:49.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:49.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:49 smithi121 bash[18489]: cluster 2024-02-12T16:26:48.874994+0000 osd.7 (osd.7) 142 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:26:49.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:49 smithi121 bash[18489]: cluster 2024-02-12T16:26:48.943781+0000 osd.7 (osd.7) 143 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:26:49.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:50.000 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:49 smithi123 bash[19119]: cluster 2024-02-12T16:26:48.874994+0000 osd.7 (osd.7) 142 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:26:50.000 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:49 smithi123 bash[19119]: cluster 2024-02-12T16:26:48.943781+0000 osd.7 (osd.7) 143 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:26:50.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:50.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:49 smithi169 bash[19604]: cluster 2024-02-12T16:26:48.874994+0000 osd.7 (osd.7) 142 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:26:50.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:49 smithi169 bash[19604]: cluster 2024-02-12T16:26:48.943781+0000 osd.7 (osd.7) 143 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:26:50.127 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:50.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:50.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:50.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:50.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:50.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:50.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:50.577 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:50.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:50.738 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:50 smithi123 bash[19119]: cluster 2024-02-12T16:26:49.803795+0000 osd.6 (osd.6) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:50.738 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:50 smithi123 bash[19119]: cluster 2024-02-12T16:26:49.803846+0000 osd.6 (osd.6) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:50.738 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:50 smithi123 bash[19119]: cluster 2024-02-12T16:26:49.970691+0000 mgr.y (mgr.14146) 1254 : cluster [DBG] pgmap v1742: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s; 14143/233068 objects misplaced (6.068%); 796 KiB/s, 12 objects/s recovering 2024-02-12T16:26:50.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:50.826 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:26:50.826 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:226963 5.9 5:93735148:::benchmark_data_smithi169_84712_object57573:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:50.815795+0000", 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.0077066369999999997, 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007735123, 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 226963 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:26:50.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.815795+0000", 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.815795+0000", 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.815792+0000", 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7499999999999992e-06 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:26:50.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.815801+0000", 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7881e-05 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.815819+0000", 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6290000000000001e-06 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.815826+0000", 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00253857 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.818365+0000", 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108896 2024-02-12T16:26:50.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.818474+0000", 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000132008 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:50.818606+0000", 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028109609999999998 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:26:50.830 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-02-12T16:26:50.831 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 1 2024-02-12T16:26:50.831 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:26:50.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:50.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:50.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:50 smithi121 bash[18489]: cluster 2024-02-12T16:26:49.803795+0000 osd.6 (osd.6) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:50.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:50 smithi121 bash[18489]: cluster 2024-02-12T16:26:49.803846+0000 osd.6 (osd.6) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:50.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:50 smithi121 bash[18489]: cluster 2024-02-12T16:26:49.970691+0000 mgr.y (mgr.14146) 1254 : cluster [DBG] pgmap v1742: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s; 14143/233068 objects misplaced (6.068%); 796 KiB/s, 12 objects/s recovering 2024-02-12T16:26:50.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:50.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:50 smithi169 bash[19604]: cluster 2024-02-12T16:26:49.803795+0000 osd.6 (osd.6) 110 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:51.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:50 smithi169 bash[19604]: cluster 2024-02-12T16:26:49.803846+0000 osd.6 (osd.6) 111 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:51.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:50 smithi169 bash[19604]: cluster 2024-02-12T16:26:49.970691+0000 mgr.y (mgr.14146) 1254 : cluster [DBG] pgmap v1742: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 7.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s; 14143/233068 objects misplaced (6.068%); 796 KiB/s, 12 objects/s recovering 2024-02-12T16:26:51.088 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:51.124 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:51.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:51.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:51.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:51.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:51.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:51.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:51.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:51.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:51.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:51.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:51.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:51.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:51.655 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:51.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:51.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:51.769 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:51 smithi169 bash[19604]: cluster 2024-02-12T16:26:50.620695+0000 osd.4 (osd.4) 96 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:26:51.769 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:51 smithi169 bash[19604]: cluster 2024-02-12T16:26:50.734002+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e597: 12 total, 12 up, 12 in 2024-02-12T16:26:51.770 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:51 smithi169 bash[19604]: cluster 2024-02-12T16:26:50.737814+0000 osd.8 (osd.8) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:51.770 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:51 smithi169 bash[19604]: cluster 2024-02-12T16:26:50.737883+0000 osd.8 (osd.8) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:51.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:26:51.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:51.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:51.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:51.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:51 smithi121 bash[18489]: cluster 2024-02-12T16:26:50.620695+0000 osd.4 (osd.4) 96 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:26:51.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:51 smithi121 bash[18489]: cluster 2024-02-12T16:26:50.734002+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e597: 12 total, 12 up, 12 in 2024-02-12T16:26:51.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:51 smithi121 bash[18489]: cluster 2024-02-12T16:26:50.737814+0000 osd.8 (osd.8) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:51.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:51 smithi121 bash[18489]: cluster 2024-02-12T16:26:50.737883+0000 osd.8 (osd.8) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:51.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:51.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:52.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:52.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:51 smithi123 bash[19119]: cluster 2024-02-12T16:26:50.620695+0000 osd.4 (osd.4) 96 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:26:52.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:51 smithi123 bash[19119]: cluster 2024-02-12T16:26:50.734002+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e597: 12 total, 12 up, 12 in 2024-02-12T16:26:52.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:51 smithi123 bash[19119]: cluster 2024-02-12T16:26:50.737814+0000 osd.8 (osd.8) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:52.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:51 smithi123 bash[19119]: cluster 2024-02-12T16:26:50.737883+0000 osd.8 (osd.8) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:52.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:52.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:52.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:52.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:52.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:52.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:52.350 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:52.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:52.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:52.482 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:52.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:52.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:52.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:52.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:52.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:52.747 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:52 smithi169 bash[19604]: cluster 2024-02-12T16:26:51.733474+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e598: 12 total, 12 up, 12 in 2024-02-12T16:26:52.747 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:52 smithi169 bash[19604]: cluster 2024-02-12T16:26:51.971103+0000 mgr.y (mgr.14146) 1255 : cluster [DBG] pgmap v1745: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 13371/226746 objects misplaced (5.897%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:26:52.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:52.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:52.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:52.995 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:52 smithi121 bash[18489]: cluster 2024-02-12T16:26:51.733474+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e598: 12 total, 12 up, 12 in 2024-02-12T16:26:52.995 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:52 smithi121 bash[18489]: cluster 2024-02-12T16:26:51.971103+0000 mgr.y (mgr.14146) 1255 : cluster [DBG] pgmap v1745: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 13371/226746 objects misplaced (5.897%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:26:53.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:53.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:53.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:52 smithi123 bash[19119]: cluster 2024-02-12T16:26:51.733474+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e598: 12 total, 12 up, 12 in 2024-02-12T16:26:53.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:52 smithi123 bash[19119]: cluster 2024-02-12T16:26:51.971103+0000 mgr.y (mgr.14146) 1255 : cluster [DBG] pgmap v1745: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 13371/226746 objects misplaced (5.897%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:26:53.168 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:53.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:53.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:53.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:53.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:53.566 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:53.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:53.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:53 smithi169 bash[19604]: cluster 2024-02-12T16:26:52.958242+0000 osd.4 (osd.4) 97 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:26:53.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:53.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:26:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:26:53.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:53.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:53 smithi121 bash[18489]: cluster 2024-02-12T16:26:52.958242+0000 osd.4 (osd.4) 97 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:26:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:53.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:54.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:54.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:54.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:54.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:53 smithi123 bash[19119]: cluster 2024-02-12T16:26:52.958242+0000 osd.4 (osd.4) 97 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:26:54.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:54.249 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:54.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:54.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:54.386 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:54.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:54.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:54.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:54.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:54.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:54.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:54.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:54.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:54.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:54.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:26:54.871 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:26:54.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:54 smithi121 bash[18489]: cluster 2024-02-12T16:26:53.974802+0000 mgr.y (mgr.14146) 1256 : cluster [DBG] pgmap v1746: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 13371/222510 objects misplaced (6.009%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:26:54.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:54.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:54.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:54 smithi169 bash[19604]: cluster 2024-02-12T16:26:53.974802+0000 mgr.y (mgr.14146) 1256 : cluster [DBG] pgmap v1746: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 13371/222510 objects misplaced (6.009%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:26:55.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:55.074 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:54 smithi123 bash[19119]: cluster 2024-02-12T16:26:53.974802+0000 mgr.y (mgr.14146) 1256 : cluster [DBG] pgmap v1746: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 13371/222510 objects misplaced (6.009%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:26:55.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:55.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:55.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:55.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:55.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:55.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:55.377 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:55.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:55.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:55.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:55.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:55.557 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:26:55.590 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:26:55.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:55.620 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:26:55.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:55.651 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:26:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:55.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:55.677 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:26:55.713 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:26:55.741 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:26:55.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:55.757 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:55 smithi123 bash[19119]: audit 2024-02-12T16:26:54.850510+0000 mgr.y (mgr.14146) 1257 : audit [DBG] from='client.16242 v1:172.21.15.121:0/3997481618' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:26:55.757 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:55 smithi123 bash[19119]: audit 2024-02-12T16:26:55.523222+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? v1:172.21.15.121:0/596849189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:55.757 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:55 smithi169 bash[19604]: audit 2024-02-12T16:26:54.850510+0000 mgr.y (mgr.14146) 1257 : audit [DBG] from='client.16242 v1:172.21.15.121:0/3997481618' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:26:55.758 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:55 smithi169 bash[19604]: audit 2024-02-12T16:26:55.523222+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? v1:172.21.15.121:0/596849189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:55.773 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:26:55.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:55.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:55.805 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:26:55.839 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:26:55.866 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:26:55.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:55.901 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:26:55.909 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.909 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.909 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.909 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:26:55.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:55.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:56.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:56.025 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:55 smithi121 bash[18489]: audit 2024-02-12T16:26:54.850510+0000 mgr.y (mgr.14146) 1257 : audit [DBG] from='client.16242 v1:172.21.15.121:0/3997481618' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:26:56.025 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:55 smithi121 bash[18489]: audit 2024-02-12T16:26:55.523222+0000 mon.c (mon.1) 51 : audit [DBG] from='client.? v1:172.21.15.121:0/596849189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:26:56.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:56.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:56.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:56.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:56.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:56.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:56.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:56.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:56.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:56.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:56.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:56.580 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:26:56.614 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:56.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:56.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:56.753 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1746,"stamp":"2024-02-12T16:26:53.974475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7291142168,"num_objects":111255,"num_object_clones":0,"num_object_copies":222510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13371,"num_objects_unfound":0,"num_objects_dirty":111255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227521,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":724,"num_bytes_recovered":47448064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18630140,"kb_used_data":14369692,"kb_used_omap":91,"kb_used_meta":4260196,"kb_avail":1106311684,"statfs":{"total":1151940427776,"available":1132863164416,"internally_reserved":0,"allocated":14714564608,"data_stored":14710398796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":4362440720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,8,11,7,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-600113152,"num_objects":-9157,"num_object_clones":0,"num_object_copies":-18314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1164,"num_objects_unfound":0,"num_objects_dirty":-9157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":11403264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004953"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":85,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:44.709588+0000","last_change":"2024-02-12T16:26:42.266638+0000","last_active":"2024-02-12T16:26:44.709588+0000","last_peered":"2024-02-12T16:26:44.709588+0000","last_clean":"2024-02-12T16:26:44.709588+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:26:44.709588+0000","last_undegraded":"2024-02-12T16:26:44.709588+0000","last_fullsized":"2024-02-12T16:26:44.709588+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:42.266460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:26:42.266460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"598'14205","reported_seq":14268,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-12T16:26:53.325057+0000","last_change":"2024-02-12T16:25:46.665174+0000","last_active":"2024-02-12T16:26:53.325057+0000","last_peered":"2024-02-12T16:26:53.325057+0000","last_clean":"2024-02-12T16:26:53.325057+0000","last_became_active":"2024-02-12T16:24:09.648523+0000","last_became_peered":"2024-02-12T16:24:09.648523+0000","last_unstale":"2024-02-12T16:26:53.325057+0000","last_undegraded":"2024-02-12T16:26:53.325057+0000","last_fullsized":"2024-02-12T16:26:53.325057+0000","mapping_epoch":547,"log_start":"559'4200","ondisk_log_start":"559'4200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10057","last_scrub_stamp":"2024-02-12T16:25:46.665055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:46.665055+0000","log_size":10005,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":446889984,"num_objects":6819,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14205,"num_write_kb":672768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":169,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.753754+0000","last_change":"2024-02-12T16:26:45.245630+0000","last_active":"2024-02-12T16:26:50.753754+0000","last_peered":"2024-02-12T16:26:50.753754+0000","last_clean":"2024-02-12T16:26:50.753754+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:26:50.753754+0000","last_undegraded":"2024-02-12T16:26:50.753754+0000","last_fullsized":"2024-02-12T16:26:50.753754+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:40.282662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:40.282662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"596'14243","reported_seq":14304,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:49.238169+0000","last_change":"2024-02-12T16:25:41.242294+0000","last_active":"2024-02-12T16:26:49.238169+0000","last_peered":"2024-02-12T16:26:49.238169+0000","last_clean":"2024-02-12T16:26:49.238169+0000","last_became_active":"2024-02-12T16:24:10.018691+0000","last_became_peered":"2024-02-12T16:24:10.018691+0000","last_unstale":"2024-02-12T16:26:49.238169+0000","last_undegraded":"2024-02-12T16:26:49.238169+0000","last_fullsized":"2024-02-12T16:26:49.238169+0000","mapping_epoch":547,"log_start":"559'4200","ondisk_log_start":"559'4200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9796","last_scrub_stamp":"2024-02-12T16:25:41.242217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:41.242217+0000","log_size":10043,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466812928,"num_objects":7123,"num_object_clones":0,"num_object_copies":14246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14243,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":280,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.738096+0000","last_change":"2024-02-12T16:24:47.872602+0000","last_active":"2024-02-12T16:26:50.738096+0000","last_peered":"2024-02-12T16:26:50.738096+0000","last_clean":"2024-02-12T16:26:50.738096+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:26:50.738096+0000","last_undegraded":"2024-02-12T16:26:50.738096+0000","last_fullsized":"2024-02-12T16:26:50.738096+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:47.872500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:24:47.872500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"597'14193","reported_seq":15509,"reported_epoch":597,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:26:51.585091+0000","last_change":"2024-02-12T16:25:58.008531+0000","last_active":"2024-02-12T16:26:51.585091+0000","last_peered":"2024-02-12T16:26:51.585091+0000","last_clean":"2024-02-12T16:25:54.194068+0000","last_became_active":"2024-02-12T16:25:57.932697+0000","last_became_peered":"2024-02-12T16:25:57.932697+0000","last_unstale":"2024-02-12T16:26:51.585091+0000","last_undegraded":"2024-02-12T16:26:51.585091+0000","last_fullsized":"2024-02-12T16:26:51.585091+0000","mapping_epoch":581,"log_start":"559'4100","ondisk_log_start":"559'4100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10027","last_scrub_stamp":"2024-02-12T16:25:45.765333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.765333+0000","log_size":10093,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":452395008,"num_objects":6903,"num_object_clones":0,"num_object_copies":13806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6412,"num_objects_unfound":0,"num_objects_dirty":6903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14193,"num_write_kb":675072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":40697856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6903}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":683,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-02-12T16:26:41.700455+0000","last_change":"2024-02-12T16:25:07.263767+0000","last_active":"2024-02-12T16:26:41.700455+0000","last_peered":"2024-02-12T16:26:41.700455+0000","last_clean":"2024-02-12T16:26:41.700455+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:26:41.700455+0000","last_undegraded":"2024-02-12T16:26:41.700455+0000","last_fullsized":"2024-02-12T16:26:41.700455+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.263688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.263688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.3","version":"596'14389","reported_seq":14445,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.228718+0000","last_change":"2024-02-12T16:24:09.648426+0000","last_active":"2024-02-12T16:26:50.228718+0000","last_peered":"2024-02-12T16:26:50.228718+0000","last_clean":"2024-02-12T16:26:50.228718+0000","last_became_active":"2024-02-12T16:24:09.648139+0000","last_became_peered":"2024-02-12T16:24:09.648139+0000","last_unstale":"2024-02-12T16:26:50.228718+0000","last_undegraded":"2024-02-12T16:26:50.228718+0000","last_fullsized":"2024-02-12T16:26:50.228718+0000","mapping_epoch":547,"log_start":"559'4300","ondisk_log_start":"559'4300","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10089,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":462225408,"num_objects":7053,"num_object_clones":0,"num_object_copies":14106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14389,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":281,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:46.160209+0000","last_change":"2024-02-12T16:26:46.160671+0000","last_active":"2024-02-12T16:26:46.160209+0000","last_peered":"2024-02-12T16:26:46.160209+0000","last_clean":"2024-02-12T16:26:46.160209+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:26:46.160209+0000","last_undegraded":"2024-02-12T16:26:46.160209+0000","last_fullsized":"2024-02-12T16:26:46.160209+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:45.967340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:45.967340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"596'13904","reported_seq":13965,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.214274+0000","last_change":"2024-02-12T16:25:50.727730+0000","last_active":"2024-02-12T16:26:50.214274+0000","last_peered":"2024-02-12T16:26:50.214274+0000","last_clean":"2024-02-12T16:26:50.214274+0000","last_became_active":"2024-02-12T16:24:09.649607+0000","last_became_peered":"2024-02-12T16:24:09.649607+0000","last_unstale":"2024-02-12T16:26:50.214274+0000","last_undegraded":"2024-02-12T16:26:50.214274+0000","last_fullsized":"2024-02-12T16:26:50.214274+0000","mapping_epoch":547,"log_start":"559'3900","ondisk_log_start":"559'3900","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10162","last_scrub_stamp":"2024-02-12T16:25:50.727611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.727611+0000","log_size":10004,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":452984832,"num_objects":6912,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13904,"num_write_kb":666112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":271,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.738198+0000","last_change":"2024-02-12T16:25:07.944563+0000","last_active":"2024-02-12T16:26:50.738198+0000","last_peered":"2024-02-12T16:26:50.738198+0000","last_clean":"2024-02-12T16:26:50.738198+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:26:50.738198+0000","last_undegraded":"2024-02-12T16:26:50.738198+0000","last_fullsized":"2024-02-12T16:26:50.738198+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.944451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.944451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"596'13657","reported_seq":13722,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.249510+0000","last_change":"2024-02-12T16:26:10.447550+0000","last_active":"2024-02-12T16:26:50.249510+0000","last_peered":"2024-02-12T16:26:50.249510+0000","last_clean":"2024-02-12T16:26:50.249510+0000","last_became_active":"2024-02-12T16:24:09.647095+0000","last_became_peered":"2024-02-12T16:24:09.647095+0000","last_unstale":"2024-02-12T16:26:50.249510+0000","last_undegraded":"2024-02-12T16:26:50.249510+0000","last_fullsized":"2024-02-12T16:26:50.249510+0000","mapping_epoch":547,"log_start":"558'3600","ondisk_log_start":"558'3600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10057,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":457637888,"num_objects":6983,"num_object_clones":0,"num_object_copies":13966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13657,"num_write_kb":660480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":273,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.738032+0000","last_change":"2024-02-12T16:25:37.899287+0000","last_active":"2024-02-12T16:26:50.738032+0000","last_peered":"2024-02-12T16:26:50.738032+0000","last_clean":"2024-02-12T16:26:50.738032+0000","last_became_active":"2024-02-12T16:17:37.450884+0000","last_became_peered":"2024-02-12T16:17:37.450884+0000","last_unstale":"2024-02-12T16:26:50.738032+0000","last_undegraded":"2024-02-12T16:26:50.738032+0000","last_fullsized":"2024-02-12T16:26:50.738032+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:37.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:25:37.899168+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"598'14410","reported_seq":14476,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-12T16:26:52.001226+0000","last_change":"2024-02-12T16:26:45.898974+0000","last_active":"2024-02-12T16:26:52.001226+0000","last_peered":"2024-02-12T16:26:52.001226+0000","last_clean":"2024-02-12T16:26:52.001226+0000","last_became_active":"2024-02-12T16:24:09.651253+0000","last_became_peered":"2024-02-12T16:24:09.651253+0000","last_unstale":"2024-02-12T16:26:52.001226+0000","last_undegraded":"2024-02-12T16:26:52.001226+0000","last_fullsized":"2024-02-12T16:26:52.001226+0000","mapping_epoch":547,"log_start":"559'4400","ondisk_log_start":"559'4400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10010,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":449839104,"num_objects":6864,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14410,"num_write_kb":680768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":138,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-02-12T16:26:41.731474+0000","last_change":"2024-02-12T16:25:54.579343+0000","last_active":"2024-02-12T16:26:41.731474+0000","last_peered":"2024-02-12T16:26:41.731474+0000","last_clean":"2024-02-12T16:26:41.731474+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:26:41.731474+0000","last_undegraded":"2024-02-12T16:26:41.731474+0000","last_fullsized":"2024-02-12T16:26:41.731474+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:54.579248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:54.579248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"597'14098","reported_seq":14160,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:51.219114+0000","last_change":"2024-02-12T16:25:52.053658+0000","last_active":"2024-02-12T16:26:51.219114+0000","last_peered":"2024-02-12T16:26:51.219114+0000","last_clean":"2024-02-12T16:26:51.219114+0000","last_became_active":"2024-02-12T16:24:09.647187+0000","last_became_peered":"2024-02-12T16:24:09.647187+0000","last_unstale":"2024-02-12T16:26:51.219114+0000","last_undegraded":"2024-02-12T16:26:51.219114+0000","last_fullsized":"2024-02-12T16:26:51.219114+0000","mapping_epoch":547,"log_start":"559'4000","ondisk_log_start":"559'4000","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'10370","last_scrub_stamp":"2024-02-12T16:25:52.053572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.053572+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":456261632,"num_objects":6962,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14098,"num_write_kb":673920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":683,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.738543+0000","last_change":"2024-02-12T16:26:04.914413+0000","last_active":"2024-02-12T16:26:50.738543+0000","last_peered":"2024-02-12T16:26:50.738543+0000","last_clean":"2024-02-12T16:26:50.738543+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:26:50.738543+0000","last_undegraded":"2024-02-12T16:26:50.738543+0000","last_fullsized":"2024-02-12T16:26:50.738543+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:04.914304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:04.914304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"598'14600","reported_seq":14663,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-12T16:26:53.274536+0000","last_change":"2024-02-12T16:25:45.294991+0000","last_active":"2024-02-12T16:26:53.274536+0000","last_peered":"2024-02-12T16:26:53.274536+0000","last_clean":"2024-02-12T16:26:53.274536+0000","last_became_active":"2024-02-12T16:24:09.650120+0000","last_became_peered":"2024-02-12T16:24:09.650120+0000","last_unstale":"2024-02-12T16:26:53.274536+0000","last_undegraded":"2024-02-12T16:26:53.274536+0000","last_fullsized":"2024-02-12T16:26:53.274536+0000","mapping_epoch":547,"log_start":"559'4500","ondisk_log_start":"559'4500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'10193","last_scrub_stamp":"2024-02-12T16:25:45.294886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.294886+0000","log_size":10100,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":452198400,"num_objects":6900,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14600,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":86,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.738687+0000","last_change":"2024-02-12T16:26:43.025974+0000","last_active":"2024-02-12T16:26:50.738687+0000","last_peered":"2024-02-12T16:26:50.738687+0000","last_clean":"2024-02-12T16:26:50.738687+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:26:50.738687+0000","last_undegraded":"2024-02-12T16:26:50.738687+0000","last_fullsized":"2024-02-12T16:26:50.738687+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:43.025867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:43.025867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"596'14150","reported_seq":14429,"reported_epoch":596,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:26:50.225673+0000","last_change":"2024-02-12T16:26:41.328312+0000","last_active":"2024-02-12T16:26:50.225673+0000","last_peered":"2024-02-12T16:26:50.225673+0000","last_clean":"2024-02-12T16:26:09.048189+0000","last_became_active":"2024-02-12T16:26:11.222461+0000","last_became_peered":"2024-02-12T16:26:11.222461+0000","last_unstale":"2024-02-12T16:26:50.225673+0000","last_undegraded":"2024-02-12T16:26:50.225673+0000","last_fullsized":"2024-02-12T16:26:50.225673+0000","mapping_epoch":585,"log_start":"559'4100","ondisk_log_start":"559'4100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'9817","last_scrub_stamp":"2024-02-12T16:25:42.181950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:42.181950+0000","log_size":10050,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":462422016,"num_objects":7056,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6959,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14150,"num_write_kb":678592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":6750208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[0,10],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,10","objects":7056}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":78,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-02-12T16:26:41.731529+0000","last_change":"2024-02-12T16:25:52.631444+0000","last_active":"2024-02-12T16:26:41.731529+0000","last_peered":"2024-02-12T16:26:41.731529+0000","last_clean":"2024-02-12T16:26:41.731529+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:26:41.731529+0000","last_undegraded":"2024-02-12T16:26:41.731529+0000","last_fullsized":"2024-02-12T16:26:41.731529+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:52.631321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.631321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"598'14573","reported_seq":14638,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-12T16:26:52.075845+0000","last_change":"2024-02-12T16:25:47.403836+0000","last_active":"2024-02-12T16:26:52.075845+0000","last_peered":"2024-02-12T16:26:52.075845+0000","last_clean":"2024-02-12T16:26:52.075845+0000","last_became_active":"2024-02-12T16:24:09.649590+0000","last_became_peered":"2024-02-12T16:24:09.649590+0000","last_unstale":"2024-02-12T16:26:52.075845+0000","last_undegraded":"2024-02-12T16:26:52.075845+0000","last_fullsized":"2024-02-12T16:26:52.075845+0000","mapping_epoch":547,"log_start":"559'4500","ondisk_log_start":"559'4500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10313","last_scrub_stamp":"2024-02-12T16:25:47.403730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.403730+0000","log_size":10073,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":454754304,"num_objects":6939,"num_object_clones":0,"num_object_copies":13878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14573,"num_write_kb":688384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":282,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-02-12T16:26:41.715356+0000","last_change":"2024-02-12T16:25:45.967785+0000","last_active":"2024-02-12T16:26:41.715356+0000","last_peered":"2024-02-12T16:26:41.715356+0000","last_clean":"2024-02-12T16:26:41.715356+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:26:41.715356+0000","last_undegraded":"2024-02-12T16:26:41.715356+0000","last_fullsized":"2024-02-12T16:26:41.715356+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:45.967726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.967726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"597'13920","reported_seq":13982,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:51.250305+0000","last_change":"2024-02-12T16:25:56.326421+0000","last_active":"2024-02-12T16:26:51.250305+0000","last_peered":"2024-02-12T16:26:51.250305+0000","last_clean":"2024-02-12T16:26:51.250305+0000","last_became_active":"2024-02-12T16:24:10.017647+0000","last_became_peered":"2024-02-12T16:24:10.017647+0000","last_unstale":"2024-02-12T16:26:51.250305+0000","last_undegraded":"2024-02-12T16:26:51.250305+0000","last_fullsized":"2024-02-12T16:26:51.250305+0000","mapping_epoch":547,"log_start":"559'3900","ondisk_log_start":"559'3900","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'10379","last_scrub_stamp":"2024-02-12T16:25:56.326340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:56.326340+0000","log_size":10020,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":445513728,"num_objects":6798,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13920,"num_write_kb":662976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":79,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-02-12T16:26:41.700174+0000","last_change":"2024-02-12T16:25:50.900847+0000","last_active":"2024-02-12T16:26:41.700174+0000","last_peered":"2024-02-12T16:26:41.700174+0000","last_clean":"2024-02-12T16:26:41.700174+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:26:41.700174+0000","last_undegraded":"2024-02-12T16:26:41.700174+0000","last_fullsized":"2024-02-12T16:26:41.700174+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.900660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.900660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"596'14196","reported_seq":14258,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.091761+0000","last_change":"2024-02-12T16:25:40.938617+0000","last_active":"2024-02-12T16:26:50.091761+0000","last_peered":"2024-02-12T16:26:50.091761+0000","last_clean":"2024-02-12T16:26:50.091761+0000","last_became_active":"2024-02-12T16:24:09.649655+0000","last_became_peered":"2024-02-12T16:24:09.649655+0000","last_unstale":"2024-02-12T16:26:50.091761+0000","last_undegraded":"2024-02-12T16:26:50.091761+0000","last_fullsized":"2024-02-12T16:26:50.091761+0000","mapping_epoch":547,"log_start":"559'4100","ondisk_log_start":"559'4100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9693","last_scrub_stamp":"2024-02-12T16:25:40.938531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:40.938531+0000","log_size":10096,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":464060440,"num_objects":7082,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14196,"num_write_kb":680833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":686,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.738562+0000","last_change":"2024-02-12T16:26:48.943869+0000","last_active":"2024-02-12T16:26:50.738562+0000","last_peered":"2024-02-12T16:26:50.738562+0000","last_clean":"2024-02-12T16:26:50.738562+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:26:50.738562+0000","last_undegraded":"2024-02-12T16:26:50.738562+0000","last_fullsized":"2024-02-12T16:26:50.738562+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:48.943785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:48.943785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"597'14481","reported_seq":14543,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:51.583114+0000","last_change":"2024-02-12T16:25:50.062621+0000","last_active":"2024-02-12T16:26:51.583114+0000","last_peered":"2024-02-12T16:26:51.583114+0000","last_clean":"2024-02-12T16:26:51.583114+0000","last_became_active":"2024-02-12T16:24:09.647617+0000","last_became_peered":"2024-02-12T16:24:09.647617+0000","last_unstale":"2024-02-12T16:26:51.583114+0000","last_undegraded":"2024-02-12T16:26:51.583114+0000","last_fullsized":"2024-02-12T16:26:51.583114+0000","mapping_epoch":547,"log_start":"559'4400","ondisk_log_start":"559'4400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10397","last_scrub_stamp":"2024-02-12T16:25:50.062524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.062524+0000","log_size":10081,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":456851456,"num_objects":6971,"num_object_clones":0,"num_object_copies":13942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14481,"num_write_kb":686464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":274,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.744131+0000","last_change":"2024-02-12T16:24:50.575354+0000","last_active":"2024-02-12T16:26:50.744131+0000","last_peered":"2024-02-12T16:26:50.744131+0000","last_clean":"2024-02-12T16:26:50.744131+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:26:50.744131+0000","last_undegraded":"2024-02-12T16:26:50.744131+0000","last_fullsized":"2024-02-12T16:26:50.744131+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.575287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.575287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"598'14211","reported_seq":14274,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-12T16:26:51.992026+0000","last_change":"2024-02-12T16:26:09.203006+0000","last_active":"2024-02-12T16:26:51.992026+0000","last_peered":"2024-02-12T16:26:51.992026+0000","last_clean":"2024-02-12T16:26:51.992026+0000","last_became_active":"2024-02-12T16:24:09.650861+0000","last_became_peered":"2024-02-12T16:24:09.650861+0000","last_unstale":"2024-02-12T16:26:51.992026+0000","last_undegraded":"2024-02-12T16:26:51.992026+0000","last_fullsized":"2024-02-12T16:26:51.992026+0000","mapping_epoch":547,"log_start":"559'4200","ondisk_log_start":"559'4200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'11415","last_scrub_stamp":"2024-02-12T16:26:09.202913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:09.202913+0000","log_size":10011,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":461570048,"num_objects":7043,"num_object_clones":0,"num_object_copies":14086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14211,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":25,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-02-12T16:26:41.700363+0000","last_change":"2024-02-12T16:26:10.388710+0000","last_active":"2024-02-12T16:26:41.700363+0000","last_peered":"2024-02-12T16:26:41.700363+0000","last_clean":"2024-02-12T16:26:41.700363+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:26:41.700363+0000","last_undegraded":"2024-02-12T16:26:41.700363+0000","last_fullsized":"2024-02-12T16:26:41.700363+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:10.388515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:10.388515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"598'14291","reported_seq":14354,"reported_epoch":598,"state":"active+clean","last_fresh":"2024-02-12T16:26:52.038811+0000","last_change":"2024-02-12T16:25:49.810317+0000","last_active":"2024-02-12T16:26:52.038811+0000","last_peered":"2024-02-12T16:26:52.038811+0000","last_clean":"2024-02-12T16:26:52.038811+0000","last_became_active":"2024-02-12T16:24:10.019266+0000","last_became_peered":"2024-02-12T16:24:10.019266+0000","last_unstale":"2024-02-12T16:26:52.038811+0000","last_undegraded":"2024-02-12T16:26:52.038811+0000","last_fullsized":"2024-02-12T16:26:52.038811+0000","mapping_epoch":547,"log_start":"559'4200","ondisk_log_start":"559'4200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10246","last_scrub_stamp":"2024-02-12T16:25:49.810203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:49.810203+0000","log_size":10091,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":448724992,"num_objects":6847,"num_object_clones":0,"num_object_copies":13694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14291,"num_write_kb":676416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":262,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.753811+0000","last_change":"2024-02-12T16:25:03.361871+0000","last_active":"2024-02-12T16:26:50.753811+0000","last_peered":"2024-02-12T16:26:50.753811+0000","last_clean":"2024-02-12T16:26:50.753811+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:26:50.753811+0000","last_undegraded":"2024-02-12T16:26:50.753811+0000","last_fullsized":"2024-02-12T16:26:50.753811+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:25:03.361757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":83,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.744053+0000","last_change":"2024-02-12T16:25:59.785619+0000","last_active":"2024-02-12T16:26:50.744053+0000","last_peered":"2024-02-12T16:26:50.744053+0000","last_clean":"2024-02-12T16:26:50.744053+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:26:50.744053+0000","last_undegraded":"2024-02-12T16:26:50.744053+0000","last_fullsized":"2024-02-12T16:26:50.744053+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:59.785496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:25:59.785496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":79,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-02-12T16:26:41.702719+0000","last_change":"2024-02-12T16:24:54.540925+0000","last_active":"2024-02-12T16:26:41.702719+0000","last_peered":"2024-02-12T16:26:41.702719+0000","last_clean":"2024-02-12T16:26:41.702719+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:26:41.702719+0000","last_undegraded":"2024-02-12T16:26:41.702719+0000","last_fullsized":"2024-02-12T16:26:41.702719+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:54.540855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:54.540855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":97,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-02-12T16:26:50.744110+0000","last_change":"2024-02-12T16:25:47.933452+0000","last_active":"2024-02-12T16:26:50.744110+0000","last_peered":"2024-02-12T16:26:50.744110+0000","last_clean":"2024-02-12T16:26:50.744110+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:26:50.744110+0000","last_undegraded":"2024-02-12T16:26:50.744110+0000","last_fullsized":"2024-02-12T16:26:50.744110+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:47.933341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.933341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":75,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-02-12T16:26:44.711038+0000","last_change":"2024-02-12T16:25:50.908791+0000","last_active":"2024-02-12T16:26:44.711038+0000","last_peered":"2024-02-12T16:26:44.711038+0000","last_clean":"2024-02-12T16:26:44.711038+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:26:44.711038+0000","last_undegraded":"2024-02-12T16:26:44.711038+0000","last_fullsized":"2024-02-12T16:26:44.711038+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.908692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.908692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":7291142168,"num_objects":111255,"num_object_clones":0,"num_object_copies":222510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13371,"num_objects_unfound":0,"num_objects_dirty":111255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227521,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":724,"num_bytes_recovered":47448064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14637801472,"data_stored":14637793328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958872,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049064,"kb_used_data":463456,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92696088,"statfs":{"total":95995035648,"available":94920794112,"internally_reserved":0,"allocated":474578944,"data_stored":474233849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.85499999999999998},"min":{"1min":0.25800000000000001,"5min":0.246,"15min":0.246},"max":{"1min":3.6560000000000001,"5min":4.2389999999999999,"15min":5.6870000000000003},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.89500000000000002,"15min":0.90100000000000002},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.5819999999999999,"5min":4.5819999999999999,"15min":4.992},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.93400000000000005,"15min":1.006},"min":{"1min":0.316,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":1.984,"5min":4.6609999999999996,"15min":7.0110000000000001},"last":0.505},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.90600000000000003,"15min":1.075},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28799999999999998},"max":{"1min":3.444,"5min":4.2039999999999997,"15min":5.6920000000000002},"last":0.87}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.1259999999999999,"15min":1.341},"min":{"1min":0.432,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1910000000000001,"5min":5.343,"15min":5.343},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":1.222,"15min":1.4350000000000001},"min":{"1min":0.39700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.0899999999999999,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":0.58799999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.94099999999999995,"15min":2.2679999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.29499999999999998},"max":{"1min":3.6019999999999999,"5min":4.4400000000000004,"15min":368.48399999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.88600000000000001,"15min":2.1850000000000001},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.6110000000000002,"5min":4.915,"15min":371.65100000000001},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1870000000000001,"15min":3.8380000000000001},"min":{"1min":0.39400000000000002,"5min":0.35299999999999998,"15min":0.35099999999999998},"max":{"1min":4.8339999999999996,"5min":5.04,"15min":407.07600000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.472,"5min":1.159,"15min":3.7999999999999998},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":407.26299999999998},"last":0.62}]},{"osd":6,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.97799999999999998,"15min":0.996},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.373},"max":{"1min":4.7119999999999997,"5min":5.141,"15min":5.141},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.1579999999999999,"15min":1.1319999999999999},"min":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.7450000000000001,"5min":8.5709999999999997,"15min":8.5709999999999997},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.1890000000000001,"15min":1.242},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.524,"5min":5.2569999999999997,"15min":6.4370000000000003},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.0289999999999999,"15min":1.1779999999999999},"min":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.131,"5min":5.0430000000000001,"15min":5.1189999999999998},"last":0.59899999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":27.638000000000002,"5min":6.3220000000000001,"15min":2.835},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":537.54399999999998,"5min":537.54399999999998,"15min":537.54399999999998},"last":1.206},{"interface":"front","average":{"1min":27.850000000000001,"5min":6.5190000000000001,"15min":3.0209999999999999},"min":{"1min":0.496,"5min":0.40500000000000003,"15min":0.34699999999999998},"max":{"1min":539.86099999999999,"5min":539.86099999999999,"15min":539.86099999999999},"last":1.165}]},{"osd":9,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.171,"15min":1.262},"min":{"1min":0.46400000000000002,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":1.504,"5min":5.7430000000000003,"15min":10.743},"last":1.196},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.21,"15min":1.4079999999999999},"min":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.39500000000000002},"max":{"1min":1.5660000000000001,"5min":5.359,"15min":10.698},"last":1.2230000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":27.643000000000001,"5min":6.4820000000000002,"15min":2.9700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":536.245,"5min":536.245,"15min":536.245},"last":0.65700000000000003},{"interface":"front","average":{"1min":27.702999999999999,"5min":6.4950000000000001,"15min":2.984},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.443},"max":{"1min":536.64300000000003,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.67200000000000004}]}]},{"osd":10,"up_from":83,"seq":356482285922,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216508,"kb_used_data":911092,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92528644,"statfs":{"total":95995035648,"available":94749331456,"internally_reserved":0,"allocated":932958208,"data_stored":932605281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:01 2024","interfaces":[{"interface":"back","average":{"1min":9.9930000000000003,"5min":2.6509999999999998,"15min":1.546},"min":{"1min":0.33700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":162.38399999999999,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.443},{"interface":"front","average":{"1min":9.8620000000000001,"5min":2.6760000000000002,"15min":1.615},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":162.01300000000001,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":0.91100000000000003,"15min":0.96399999999999997},"min":{"1min":0.315,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":4.4059999999999997,"5min":4.4059999999999997,"15min":19.866},"last":2.5179999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.89700000000000002,"15min":1.0509999999999999},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.27700000000000002},"max":{"1min":1.52,"5min":2.641,"15min":20.277999999999999},"last":2.4119999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":0.94399999999999995,"15min":0.85699999999999998},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.9039999999999999,"5min":4.9039999999999999,"15min":5.3040000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.119,"5min":1.05,"15min":0.91100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":4.8070000000000004},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.98199999999999998,"15min":3.3399999999999999},"min":{"1min":0.44400000000000001,"5min":0.437,"15min":0.32300000000000001},"max":{"1min":3.2639999999999998,"5min":3.2639999999999998,"15min":628.77099999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.93899999999999995,"15min":3.2679999999999998},"min":{"1min":0.36299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.524,"5min":4.524,"15min":624.19299999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.0629999999999999,"15min":1.008},"min":{"1min":0.44800000000000001,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":5.0679999999999996,"5min":5.0679999999999996,"15min":5.5739999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.321,"5min":1.198,"15min":1.1220000000000001},"min":{"1min":0.52700000000000002,"5min":0.48999999999999999,"15min":0.35799999999999998},"max":{"1min":3.2909999999999999,"5min":3.2949999999999999,"15min":4.8769999999999998},"last":0.66700000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.052,"15min":3.3410000000000002},"min":{"1min":0.45100000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.2640000000000002,"5min":4.2640000000000002,"15min":626.26599999999996},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.095,"15min":3.4470000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.9390000000000001,"5min":2.9390000000000001,"15min":625.90700000000004},"last":0.70999999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.98099999999999998,"15min":1.05},"min":{"1min":0.44400000000000001,"5min":0.40400000000000003,"15min":0.39800000000000002},"max":{"1min":1.246,"5min":1.994,"15min":5.9210000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.1890000000000001,"15min":1.149},"min":{"1min":0.53500000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":5.3300000000000001},"last":0.878}]},{"osd":8,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.091,"15min":1.093},"min":{"1min":0.44400000000000001,"5min":0.432,"15min":0.372},"max":{"1min":1.5589999999999999,"5min":1.9830000000000001,"15min":4.827},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":1.0940000000000001,"15min":1.0960000000000001},"min":{"1min":0.442,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4550000000000001,"5min":3.859,"15min":4.8330000000000002},"last":0.68200000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1970000000000001,"15min":1.2110000000000001},"min":{"1min":0.48499999999999999,"5min":0.44900000000000001,"15min":0.372},"max":{"1min":4.782,"5min":4.782,"15min":4.9829999999999997},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.139,"15min":1.1990000000000001},"min":{"1min":0.51700000000000002,"5min":0.47299999999999998,"15min":0.41399999999999998},"max":{"1min":1.4039999999999999,"5min":2.9169999999999998,"15min":4.952},"last":0.71799999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.1319999999999999,"15min":1.3999999999999999},"min":{"1min":0.55100000000000005,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":1.4550000000000001,"5min":2.0249999999999999,"15min":30.917999999999999},"last":0.754},{"interface":"front","average":{"1min":0.878,"5min":1.157,"15min":1.4119999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.617,"5min":2.2400000000000002,"15min":30.742000000000001},"last":0.73999999999999999}]}]},{"osd":9,"up_from":74,"seq":317827580266,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1190960,"kb_used_data":889640,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92554192,"statfs":{"total":95995035648,"available":94775492608,"internally_reserved":0,"allocated":910991360,"data_stored":910644393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":0.98399999999999999,"15min":1.042},"min":{"1min":0.35699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.8330000000000002,"5min":5.8330000000000002,"15min":5.8330000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.85799999999999998,"15min":0.879},"min":{"1min":0.36299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.3029999999999999,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.98799999999999999,"15min":0.91900000000000004},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.253},"max":{"1min":5.8819999999999997,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":3.9620000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.91600000000000004,"15min":1.0409999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.29299999999999998},"max":{"1min":2.323,"5min":5.4560000000000004,"15min":7.9550000000000001},"last":3.976}]},{"osd":2,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.0569999999999999,"15min":1.391},"min":{"1min":0.45600000000000002,"5min":0.371,"15min":0.27600000000000002},"max":{"1min":5.915,"5min":5.915,"15min":97.275000000000006},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.109,"5min":0.99399999999999999,"15min":1.2609999999999999},"min":{"1min":0.36099999999999999,"5min":0.314,"15min":0.312},"max":{"1min":6.1779999999999999,"5min":6.1779999999999999,"15min":97.417000000000002},"last":3.0459999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.036,"15min":0.93400000000000005},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.32200000000000001},"max":{"1min":3.528,"5min":4.569,"15min":4.569},"last":0.623},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.1990000000000001,"15min":1.089},"min":{"1min":0.5,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.5609999999999999,"5min":4.8840000000000003,"15min":5.8010000000000002},"last":0.65900000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.2350000000000001,"15min":1.0469999999999999},"min":{"1min":0.52500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.7430000000000003,"5min":5.7430000000000003,"15min":5.899},"last":0.5},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1679999999999999,"15min":1.131},"min":{"1min":0.46100000000000002,"5min":0.44700000000000001,"15min":0.40999999999999998},"max":{"1min":4.5540000000000003,"5min":5.4290000000000003,"15min":7.3739999999999997},"last":0.441}]},{"osd":6,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.0600000000000001,"15min":1.0940000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":3.3250000000000002,"5min":5.5250000000000004,"15min":7.3360000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.1200000000000001,"15min":1.0309999999999999},"min":{"1min":0.47099999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.0279999999999996,"5min":7.0609999999999999,"15min":7.0609999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.3140000000000001,"15min":1.212},"min":{"1min":0.59299999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.8010000000000002,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.2150000000000001,"15min":1.218},"min":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.438},"max":{"1min":2.9550000000000001,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.47399999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.145,"15min":1.0649999999999999},"min":{"1min":0.45700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.069,"5min":5.3620000000000001,"15min":5.4059999999999997},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.80500000000000005,"5min":1.155,"15min":1.1930000000000001},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.778,"5min":4.0419999999999998,"15min":7.25},"last":0.85499999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.304,"15min":1.2310000000000001},"min":{"1min":0.53400000000000003,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":3.8029999999999999,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.1890000000000001,"15min":1.1719999999999999},"min":{"1min":0.53900000000000003,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":4.423,"5min":4.4509999999999996,"15min":5.8700000000000001},"last":0.93600000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:26:44 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.2949999999999999,"15min":1.349},"min":{"1min":0.55000000000000004,"5min":0.48999999999999999,"15min":0.45600000000000002},"max":{"1min":3.7370000000000001,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.026,"5min":1.1950000000000001,"15min":1.3100000000000001},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.0169999999999999,"5min":5.492,"15min":6.3639999999999999},"last":0.88400000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874612,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1170364,"kb_used_data":860852,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92574788,"statfs":{"total":95995035648,"available":94796582912,"internally_reserved":0,"allocated":881512448,"data_stored":881159529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":1.946,"5min":0.98599999999999999,"15min":0.85899999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":19.196999999999999,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.497},{"interface":"front","average":{"1min":2.2810000000000001,"5min":1.1319999999999999,"15min":0.88200000000000001},"min":{"1min":0.40699999999999997,"5min":0.29599999999999999,"15min":0.29399999999999998},"max":{"1min":18.584,"5min":18.584,"15min":18.584},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":5.4279999999999999,"5min":1.7230000000000001,"15min":1.4099999999999999},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":69.956000000000003,"5min":69.956000000000003,"15min":69.956000000000003},"last":3.9300000000000002},{"interface":"front","average":{"1min":5.0999999999999996,"5min":1.7669999999999999,"15min":1.498},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.315},"max":{"1min":71.010999999999996,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.79000000000000004,"15min":0.82499999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.7919999999999998,"5min":2.806,"15min":2.806},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.90500000000000003,"15min":0.98499999999999999},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.1539999999999999,"5min":2.278,"15min":2.278},"last":3.984}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.96699999999999997,"15min":1.72},"min":{"1min":0.35199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.0490000000000004,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.97799999999999998,"15min":1.6599999999999999},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":5.0380000000000003,"5min":5.0380000000000003,"15min":224.71000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.82499999999999996,"15min":0.86899999999999999},"min":{"1min":0.373,"5min":0.37,"15min":0.32200000000000001},"max":{"1min":1.204,"5min":1.8200000000000001,"15min":4.8689999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0589999999999999,"15min":1.0229999999999999},"min":{"1min":0.38600000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":4.8339999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.131,"15min":1.718},"min":{"1min":0.44400000000000001,"5min":0.42399999999999999,"15min":0.37},"max":{"1min":3.903,"5min":5.29,"15min":221.36699999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.163,"15min":1.8600000000000001},"min":{"1min":0.5,"5min":0.437,"15min":0.42199999999999999},"max":{"1min":4.4930000000000003,"5min":5.556,"15min":221.203},"last":2.2080000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:26:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.0760000000000001,"15min":1.1080000000000001},"min":{"1min":0.51500000000000001,"5min":0.45400000000000001,"15min":0.42199999999999999},"max":{"1min":2.7109999999999999,"5min":3.4279999999999999,"15min":7.9160000000000004},"last":3.2490000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.2250000000000001,"15min":1.198},"min":{"1min":0.52700000000000002,"5min":0.50900000000000001,"15min":0.41599999999999998},"max":{"1min":2.444,"5min":3.419,"15min":7.899},"last":3.238}]},{"osd":9,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.173,"15min":1.1539999999999999},"min":{"1min":0.46800000000000003,"5min":0.375,"15min":0.35599999999999998},"max":{"1min":1.716,"5min":2.9430000000000001,"15min":4.5099999999999998},"last":0.624},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.298,"15min":1.2090000000000001},"min":{"1min":0.60299999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.8240000000000001,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":0.68000000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.01,"15min":1.3979999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":109.92},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.115,"15min":1.4650000000000001},"min":{"1min":0.48299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0250000000000004,"5min":5.0250000000000004,"15min":110.249},"last":2.6469999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.0660000000000001,"15min":1.1899999999999999},"min":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":2.7869999999999999,"5min":2.9159999999999999,"15min":5.0670000000000002},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.97199999999999998,"15min":1.1399999999999999},"min":{"1min":0.56000000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":2.1259999999999999,"15min":4.6639999999999997},"last":0.73799999999999999}]}]},{"osd":7,"up_from":56,"seq":240518168954,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745276,"kb_used_data":443956,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92999876,"statfs":{"total":95995035648,"available":95231873024,"internally_reserved":0,"allocated":454610944,"data_stored":454258025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.88400000000000001,"15min":0.90000000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":3.1890000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.93500000000000005,"15min":0.93000000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.75,"5min":7.7699999999999996,"15min":7.7699999999999996},"last":0.375}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.2769999999999999,"15min":1.008},"min":{"1min":0.313,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":4.5179999999999998,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.5},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.6339999999999999,"15min":1.1859999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.6719999999999997,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.1259999999999999,"15min":1.274},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.5140000000000002,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":4.1669999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.90900000000000003,"15min":0.98899999999999999},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.4699999999999998,"5min":4.4699999999999998,"15min":4.4699999999999998},"last":0.60099999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.0600000000000001,"15min":0.93799999999999994},"min":{"1min":0.34100000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.4420000000000002,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.90300000000000002,"15min":1.0089999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":4.7389999999999999,"5min":4.7389999999999999,"15min":6.0090000000000003},"last":0.55800000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.98199999999999998,"15min":0.96499999999999997},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.5620000000000003,"5min":4.7220000000000004,"15min":4.9050000000000002},"last":3.5499999999999998},{"interface":"front","average":{"1min":1.089,"5min":1.024,"15min":1.036},"min":{"1min":0.38300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":5.766,"5min":7.5640000000000001,"15min":9.3640000000000008},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.125,"15min":1.101},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":6.319,"5min":6.319,"15min":6.319},"last":3.5819999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.141,"15min":1.1799999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":3.5670000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":28.620999999999999,"5min":6.6399999999999997,"15min":2.976},"min":{"1min":0.44600000000000001,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":498.68299999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":28.295999999999999,"5min":6.6840000000000002,"15min":3.028},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":498.69999999999999,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.74099999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:25:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":1.087,"15min":1.2130000000000001},"min":{"1min":0.442,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.6299999999999999,"5min":2.9860000000000002,"15min":12.907999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.2010000000000001,"15min":1.2390000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.42699999999999999},"max":{"1min":4.665,"5min":4.665,"15min":6.0199999999999996},"last":0.64700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":34.110999999999997,"5min":7.742,"15min":3.4820000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":497.52699999999999,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":33.834000000000003,"5min":7.7649999999999997,"15min":3.3940000000000001},"min":{"1min":0.45300000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":496.29599999999999,"5min":496.29599999999999,"15min":496.29599999999999},"last":0.77200000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.236,"15min":1.6659999999999999},"min":{"1min":0.53100000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":4.6890000000000001,"5min":6.0490000000000004,"15min":120.943},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.2370000000000001,"15min":1.738},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.45300000000000001},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":120.661},"last":3.4279999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430595,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648804,"kb_used_data":1351580,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92096348,"statfs":{"total":95995035648,"available":94306660352,"internally_reserved":0,"allocated":1384017920,"data_stored":1383676905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.1659999999999999,"15min":1.077},"min":{"1min":0.26200000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.88,"5min":30.454999999999998,"15min":30.454999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.274,"15min":1.163},"min":{"1min":0.33300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.6930000000000001,"5min":30.573,"15min":30.573},"last":0.502}]},{"osd":1,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.97999999999999998,"15min":1.887},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":3.5129999999999999,"5min":3.605,"15min":252.22800000000001},"last":0.629},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.93100000000000005,"15min":1.8859999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.702,"5min":4.4880000000000004,"15min":252.61199999999999},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":1.2490000000000001,"15min":0.94899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":28.739999999999998,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.99199999999999999},{"interface":"front","average":{"1min":3.0169999999999999,"5min":1.4530000000000001,"15min":1.0529999999999999},"min":{"1min":0.32300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":32.328000000000003,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.55300000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.0569999999999999,"15min":1.9079999999999999},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27600000000000002},"max":{"1min":2.4020000000000001,"5min":7.7679999999999998,"15min":284.05599999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":1.0800000000000001,"15min":2.036},"min":{"1min":0.33200000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.7250000000000001,"5min":7.5979999999999999,"15min":284.041},"last":0.82799999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.96999999999999997,"15min":1.871},"min":{"1min":0.379,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.6019999999999999,"5min":3.3809999999999998,"15min":281.49700000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.99399999999999999,"15min":1.9470000000000001},"min":{"1min":0.374,"5min":0.37,"15min":0.33500000000000002},"max":{"1min":1.8959999999999999,"5min":4.4749999999999996,"15min":281.61900000000003},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.097,"15min":1.0780000000000001},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.79,"5min":7.641,"15min":7.641},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.1200000000000001,"15min":1.071},"min":{"1min":0.34899999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.8210000000000002,"5min":7.8380000000000001,"15min":7.8380000000000001},"last":0.54100000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.0660000000000001,"15min":1.077},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.4910000000000001,"5min":3.351,"15min":4.2380000000000004},"last":0.878},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.169,"15min":1.1859999999999999},"min":{"1min":0.40799999999999997,"5min":0.371,"15min":0.371},"max":{"1min":1.7649999999999999,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":0.80700000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":1.163,"15min":1.206},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.0039999999999996,"5min":4.5330000000000004,"15min":18.059999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.2310000000000001,"15min":1.288},"min":{"1min":0.51100000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.6910000000000001,"5min":4.6890000000000001,"15min":17.425000000000001},"last":0.72699999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:26:45 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.202,"15min":1.224},"min":{"1min":0.505,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":6.5300000000000002},"last":1.012},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.276,"15min":1.3080000000000001},"min":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":6.5730000000000004},"last":1.056}]},{"osd":11,"last update":"Mon Feb 12 16:26:45 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.242,"15min":3.2519999999999998},"min":{"1min":0.53700000000000003,"5min":0.499,"15min":0.499},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":139.727},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.151,"15min":3.125},"min":{"1min":0.51500000000000001,"5min":0.504,"15min":0.504},"max":{"1min":2.706,"5min":4.5899999999999999,"15min":139.24199999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2379120,"kb_used_data":1781224,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":91366032,"statfs":{"total":95995035648,"available":93558816768,"internally_reserved":0,"allocated":1823973376,"data_stored":1823626409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.90600000000000003,"15min":1.329},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":7.8710000000000004,"5min":7.8710000000000004,"15min":114.40000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.589,"5min":1.2969999999999999,"15min":1.5780000000000001},"min":{"1min":0.30299999999999999,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":5.2690000000000001,"5min":7.7249999999999996,"15min":115.09099999999999},"last":6.5759999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.0049999999999999,"15min":0.98599999999999999},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":4.2720000000000002,"5min":6.5389999999999997,"15min":9.8599999999999994},"last":6.5119999999999996},{"interface":"front","average":{"1min":1.74,"5min":1.198,"15min":1.119},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":9.6120000000000001,"5min":9.6120000000000001,"15min":9.6120000000000001},"last":0.433}]},{"osd":3,"last update":"Mon Feb 12 16:26:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0840000000000001,"15min":1.1180000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":4.0430000000000001,"5min":4.0430000000000001,"15min":4.0430000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.393,"5min":1.012,"15min":0.89500000000000002},"min":{"1min":0.29499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.306,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":6.5490000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:26:15 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.216,"15min":1.103},"min":{"1min":0.41299999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.7479999999999993,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":6.6749999999999998},{"interface":"front","average":{"1min":1.962,"5min":1.492,"15min":1.4139999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":7.1710000000000003,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.54200000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":1.351,"15min":1.1819999999999999},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":10.109999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.2130000000000001,"15min":1.1899999999999999},"min":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.35899999999999999},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":7.0110000000000001},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.216,"15min":1.6379999999999999},"min":{"1min":0.439,"5min":0.38500000000000001,"15min":0.34699999999999998},"max":{"1min":2.4020000000000001,"5min":6.6310000000000002,"15min":55.869999999999997},"last":6.6219999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2110000000000001,"15min":1.6970000000000001},"min":{"1min":0.48399999999999999,"5min":0.46600000000000003,"15min":0.36799999999999999},"max":{"1min":2.5219999999999998,"5min":4.4470000000000001,"15min":55.892000000000003},"last":0.47899999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:26:39 2024","interfaces":[{"interface":"back","average":{"1min":31.306999999999999,"5min":7.3899999999999997,"15min":3.371},"min":{"1min":0.53700000000000003,"5min":0.51000000000000001,"15min":0.436},"max":{"1min":548.91600000000005,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.63},{"interface":"front","average":{"1min":32.816000000000003,"5min":7.6150000000000002,"15min":3.4550000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":552.42100000000005,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.66800000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.464,"15min":1.3919999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.45400000000000001},"max":{"1min":5.7240000000000002,"5min":5.7240000000000002,"15min":10.07},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.4019999999999999,"15min":1.506},"min":{"1min":0.55400000000000005,"5min":0.46700000000000003,"15min":0.45000000000000001},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":7.6059999999999999},"last":0.81299999999999994}]},{"osd":10,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":36.500999999999998,"5min":8.5050000000000008,"15min":3.7400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":548.93499999999995,"5min":548.93499999999995,"15min":548.93499999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":35.311,"5min":8.3140000000000001,"15min":3.774},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.53700000000000003},"max":{"1min":549.12099999999998,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.86199999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.421,"15min":1.534},"min":{"1min":0.60199999999999998,"5min":0.58099999999999996,"15min":0.443},"max":{"1min":3.996,"5min":4.5609999999999999,"15min":7.1600000000000001},"last":0.753},{"interface":"front","average":{"1min":1.659,"5min":1.609,"15min":1.6180000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.47599999999999998},"max":{"1min":5.2949999999999999,"5min":5.7549999999999999,"15min":7.2489999999999997},"last":0.91000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2565604,"kb_used_data":2264284,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91179548,"statfs":{"total":95995035648,"available":93367857152,"internally_reserved":0,"allocated":2318626816,"data_stored":2318285801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":1.0620000000000001},"min":{"1min":0.254,"5min":0.245,"15min":0.215},"max":{"1min":4.4260000000000002,"5min":10.754,"15min":11.324},"last":0.745},{"interface":"front","average":{"1min":1.069,"5min":1.0129999999999999,"15min":1.0580000000000001},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.6859999999999999,"5min":4.6859999999999999,"15min":9.8249999999999993},"last":4.6859999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.98099999999999998,"15min":1.03},"min":{"1min":0.308,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.0469999999999999,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.874,"15min":0.90700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.1970000000000001,"5min":3.637,"15min":3.637},"last":4.8220000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.167,"15min":1.7130000000000001},"min":{"1min":0.38900000000000001,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":4.8600000000000003,"5min":7.6219999999999999,"15min":116.187},"last":4.8600000000000003},{"interface":"front","average":{"1min":0.87,"5min":1.3080000000000001,"15min":1.829},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":3.1030000000000002,"5min":7.5599999999999996,"15min":116.163},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":0.99199999999999999,"15min":1.4199999999999999},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.1539999999999999,"5min":4.1539999999999999,"15min":116.005},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.1120000000000001,"15min":1.5009999999999999},"min":{"1min":0.40100000000000002,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":115.973},"last":0.35199999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:26:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.2430000000000001,"15min":1.327},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.4249999999999998,"5min":4.9119999999999999,"15min":81.147000000000006},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.347,"15min":1.405},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":8.8789999999999996,"5min":8.8789999999999996,"15min":80.233000000000004},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.2170000000000001,"15min":1.5880000000000001},"min":{"1min":0.40899999999999997,"5min":0.374,"15min":0.36699999999999999},"max":{"1min":4.4219999999999997,"5min":4.4219999999999997,"15min":115.26000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.3080000000000001,"15min":1.6919999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.4889999999999999,"5min":4.4889999999999999,"15min":115.402},"last":0.48799999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.2609999999999999,"15min":1.2370000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":4.806},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.363,"15min":1.333},"min":{"1min":0.55900000000000005,"5min":0.51800000000000002,"15min":0.41299999999999998},"max":{"1min":4.3869999999999996,"5min":4.3869999999999996,"15min":4.8380000000000001},"last":0.51400000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:26:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.179,"15min":1.3500000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.42699999999999999},"max":{"1min":3.165,"5min":7.6539999999999999,"15min":46.246000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.401,"15min":1.401},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45800000000000002},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":46.167999999999999},"last":0.58599999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2589999999999999,"15min":1.276},"min":{"1min":0.54400000000000004,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":4.944,"5min":4.944,"15min":6.0979999999999999},"last":4.944},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.339,"15min":1.3160000000000001},"min":{"1min":0.57399999999999995,"5min":0.52000000000000002,"15min":0.47999999999999998},"max":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":6.2480000000000002},"last":5.0049999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:26:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.25,"15min":1.5209999999999999},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.434},"max":{"1min":2.8079999999999998,"5min":4.5220000000000002,"15min":46.064},"last":4.976},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.3580000000000001,"15min":1.48},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.38300000000000001},"max":{"1min":4.2750000000000004,"5min":4.2750000000000004,"15min":45.642000000000003},"last":0.60999999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313634,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1633584,"kb_used_data":1328168,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92111568,"statfs":{"total":95995035648,"available":94322245632,"internally_reserved":0,"allocated":1360044032,"data_stored":1359697065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.034,"15min":1.0509999999999999},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":8.5399999999999991},"last":5.8650000000000002},{"interface":"front","average":{"1min":1.8220000000000001,"5min":1.2470000000000001,"15min":1.1579999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":6.484,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":1.8360000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.7689999999999999,"15min":2.0070000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.8460000000000001,"5min":21.407,"15min":21.407},"last":1.873},{"interface":"front","average":{"1min":1.169,"5min":0.86599999999999999,"15min":0.84199999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.887,"5min":3.887,"15min":3.887},"last":5.7830000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0780000000000001,"15min":0.95599999999999996},"min":{"1min":0.34000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":3.9079999999999999,"5min":6.8090000000000002,"15min":7.2859999999999996},"last":10.890000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.2989999999999999,"15min":1.093},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.601,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":5.9560000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.226,"15min":1.095},"min":{"1min":0.39400000000000002,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":5.9489999999999998,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":6.0300000000000002},{"interface":"front","average":{"1min":1.482,"5min":1.175,"15min":1.0509999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":6.5730000000000004},"last":11.081}]},{"osd":6,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.472,"15min":1.157},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.6890000000000001,"5min":21.527000000000001,"15min":21.527000000000001},"last":5.8739999999999997},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.1839999999999999,"15min":1.089},"min":{"1min":0.40500000000000003,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":3.98,"5min":3.98,"15min":5.4349999999999996},"last":5.968}]},{"osd":7,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1499999999999999,"15min":1.0649999999999999},"min":{"1min":0.41899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.9489999999999998,"5min":4.8570000000000002,"15min":5.2960000000000003},"last":11.044},{"interface":"front","average":{"1min":1.571,"5min":1.2569999999999999,"15min":1.1870000000000001},"min":{"1min":0.47999999999999998,"5min":0.443,"15min":0.38700000000000001},"max":{"1min":5.5099999999999998,"5min":5.5099999999999998,"15min":5.5099999999999998},"last":5.9980000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.2649999999999999,"15min":1.1519999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.38700000000000001},"max":{"1min":2.6960000000000002,"5min":4.1539999999999999,"15min":5.101},"last":5.923},{"interface":"front","average":{"1min":1.236,"5min":1.383,"15min":1.22},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":5.9269999999999996,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":6.0490000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.25,"15min":1.181},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.40600000000000003},"max":{"1min":4.0119999999999996,"5min":4.8040000000000003,"15min":5.2329999999999997},"last":10.972},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3380000000000001,"15min":1.252},"min":{"1min":0.50600000000000001,"5min":0.45400000000000001,"15min":0.42399999999999999},"max":{"1min":3.9809999999999999,"5min":4.9089999999999998,"15min":5.085},"last":11.006}]},{"osd":10,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.4339999999999999,"15min":1.3029999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6609999999999996,"5min":11.454000000000001,"15min":11.454000000000001},"last":6.0099999999999998},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.484,"15min":1.3220000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":6.6340000000000003,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":6.0919999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:25:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3140000000000001,"15min":1.4870000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":3.859,"5min":4.0590000000000002,"15min":4.0590000000000002},"last":5.9050000000000002},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4279999999999999,"15min":1.5649999999999999},"min":{"1min":0.51800000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.8250000000000002,"5min":3.9910000000000001,"15min":3.9910000000000001},"last":5.9359999999999999}]}]},{"osd":3,"up_from":522,"seq":2241972928559,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":791716,"kb_used_data":455388,"kb_used_omap":91,"kb_used_meta":336228,"kb_avail":92953436,"statfs":{"total":95995035648,"available":95184318464,"internally_reserved":0,"allocated":466317312,"data_stored":465976297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":344297488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":1.2490000000000001,"15min":0.98099999999999998},"min":{"1min":0.32500000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":34.055999999999997,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":3.4910000000000001,"5min":1.361,"15min":1.022},"min":{"1min":0.33800000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":27.600000000000001,"5min":27.600000000000001,"15min":27.600000000000001},"last":1.099}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.92300000000000004,"15min":0.88700000000000001},"min":{"1min":0.32800000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.5279999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.593,"5min":1.1399999999999999,"15min":1.083},"min":{"1min":0.313,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":6.5030000000000001,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":0.89000000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.111,"15min":1.321},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.8959999999999999,"5min":10.004,"15min":10.004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0369999999999999,"15min":1.1000000000000001},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.3029999999999999,"5min":4.3029999999999999,"15min":4.3029999999999999},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2729999999999999,"15min":1.389},"min":{"1min":0.35099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.4189999999999996,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.88800000000000001,"15min":0.92800000000000005},"min":{"1min":0.34899999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.3639999999999999,"5min":3.593,"15min":3.593},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.1060000000000001,"15min":1.141},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4779999999999998,"5min":4.4779999999999998,"15min":4.4779999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.1200000000000001,"5min":0.98699999999999999,"15min":1.0069999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":3.448,"5min":3.7530000000000001,"15min":3.7530000000000001},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.2709999999999999,"15min":1.4630000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":3.2839999999999998,"5min":6.851,"15min":6.851},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.171,"5min":1.2170000000000001,"15min":1.2769999999999999},"min":{"1min":0.51000000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.3529999999999998,"5min":4.3529999999999998,"15min":4.3529999999999998},"last":0.77800000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.137,"15min":1.204},"min":{"1min":0.54400000000000004,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.018,"15min":1.1060000000000001},"min":{"1min":0.42699999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.9990000000000001,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.69999999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3420000000000001,"15min":1.3480000000000001},"min":{"1min":0.54800000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":5.0830000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.3140000000000001,"15min":1.373},"min":{"1min":0.502,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":4.5800000000000001},"last":0.92700000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.409,"15min":1.54},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.097,"5min":1.274,"15min":1.3140000000000001},"min":{"1min":0.56100000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":4.4969999999999999,"5min":4.4969999999999999,"15min":4.4969999999999999},"last":0.95199999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.234,"15min":1.2809999999999999},"min":{"1min":0.63900000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.4349999999999996,"5min":4.4349999999999996,"15min":4.4349999999999996},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.05,"5min":1.399,"15min":1.5329999999999999},"min":{"1min":0.60399999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.633,"5min":4.7530000000000001,"15min":4.7530000000000001},"last":1.069}]}]},{"osd":4,"up_from":32,"seq":137438953874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2146852,"kb_used_data":1837276,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91598300,"statfs":{"total":95995035648,"available":93796659200,"internally_reserved":0,"allocated":1881370624,"data_stored":1881029609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:25:52 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.0409999999999999,"15min":1.397},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":90.597999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.0169999999999999,"15min":1.5289999999999999},"min":{"1min":0.35799999999999998,"5min":0.30599999999999999,"15min":0.28399999999999997},"max":{"1min":5.056,"5min":5.056,"15min":90.676000000000002},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":0.91300000000000003,"15min":0.94799999999999995},"min":{"1min":0.31,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":8.0530000000000008},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.0289999999999999,"15min":1.1240000000000001},"min":{"1min":0.29399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":4.1109999999999998,"5min":5.5709999999999997,"15min":7.452},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.0840000000000001,"15min":1.0880000000000001},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":8.3040000000000003,"5min":8.3040000000000003,"15min":8.3040000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.034,"15min":1.0509999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.5410000000000004,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.38400000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:26:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.98299999999999998,"15min":0.95799999999999996},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.7009999999999996,"5min":5.5999999999999996,"15min":6.0460000000000003},"last":0.313},{"interface":"front","average":{"1min":1.51,"5min":1.177,"15min":1.1060000000000001},"min":{"1min":0.34599999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.7160000000000002,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.92800000000000005,"15min":1.046},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.673,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.077,"15min":1.0580000000000001},"min":{"1min":0.35299999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.2040000000000002,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":3.746}]},{"osd":7,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.071,"15min":1.1539999999999999},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":9.0879999999999992},"last":3.782},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.89700000000000002,"15min":1.1890000000000001},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.296,"5min":3.3799999999999999,"15min":7.6689999999999996},"last":0.63200000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1120000000000001,"15min":1.232},"min":{"1min":0.51900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.6099999999999999,"5min":3.3079999999999998,"15min":7.8650000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.698,"5min":1.331,"15min":1.2589999999999999},"min":{"1min":0.42799999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.4489999999999998,"5min":5.6699999999999999,"15min":5.6699999999999999},"last":0.66200000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.054,"15min":1.429},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.417,"5min":3.2650000000000001,"15min":34.112000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.361,"15min":1.6639999999999999},"min":{"1min":0.502,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.9740000000000002,"5min":5.6420000000000003,"15min":34.590000000000003},"last":0.76700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.3740000000000001,"15min":1.3759999999999999},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.0949999999999998,"5min":5.6970000000000001,"15min":6.0899999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.163,"15min":1.262},"min":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.8300000000000001,"5min":4.46,"15min":8.5299999999999994},"last":3.8300000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.266,"15min":1.5449999999999999},"min":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.7010000000000001,"5min":3.2320000000000002,"15min":4.6500000000000004},"last":0.749},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.421,"15min":1.5249999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":5.5060000000000002,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.69699999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092288,"kb_used_data":1782776,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91652864,"statfs":{"total":95995035648,"available":93852532736,"internally_reserved":0,"allocated":1825562624,"data_stored":1825205633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80200000000000005,"15min":2.702},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.25800000000000001},"max":{"1min":3.331,"5min":3.8690000000000002,"15min":565.34500000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.91600000000000004,"15min":2.7829999999999999},"min":{"1min":0.39300000000000002,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":3.3660000000000001,"5min":3.9399999999999999,"15min":565.98400000000004},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.3440000000000001,"15min":1.097},"min":{"1min":0.36099999999999999,"5min":0.33200000000000002,"15min":0.30499999999999999},"max":{"1min":3.9289999999999998,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.361,"15min":1.129},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.194,"5min":40.225000000000001,"15min":40.225000000000001},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:26:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.4169999999999998,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":4.0990000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.387,"15min":1.4299999999999999},"min":{"1min":0.317,"5min":0.246,"15min":0.246},"max":{"1min":41.908000000000001,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":3.4350000000000001,"5min":1.4370000000000001,"15min":1.4259999999999999},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":42.430999999999997,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.90500000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:26:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.0580000000000001,"15min":1.2949999999999999},"min":{"1min":0.34200000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":5.6669999999999998,"5min":5.6669999999999998,"15min":74.105000000000004},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.135,"15min":1.413},"min":{"1min":0.47999999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":75.013999999999996},"last":0.46200000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.012,"15min":1.026},"min":{"1min":0.374,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1120000000000001,"5min":6.1120000000000001,"15min":6.6630000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.0349999999999999,"15min":1.107},"min":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":5.6900000000000004,"5min":5.6900000000000004,"15min":5.9909999999999997},"last":0.88400000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.097,"15min":1.1930000000000001},"min":{"1min":0.47499999999999998,"5min":0.41699999999999998,"15min":0.40200000000000002},"max":{"1min":2.6859999999999999,"5min":4.0069999999999997,"15min":5.9509999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1619999999999999,"15min":1.222},"min":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":6.0739999999999998,"5min":6.0739999999999998,"15min":6.0739999999999998},"last":0.623}]},{"osd":9,"last update":"Mon Feb 12 16:26:05 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.359,"15min":3.2469999999999999},"min":{"1min":0.49399999999999999,"5min":0.46899999999999997,"15min":0.42999999999999999},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":449.89499999999998},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.167,"15min":3.2429999999999999},"min":{"1min":0.55100000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.052,"5min":3.0840000000000001,"15min":448.70800000000003},"last":0.58499999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.1220000000000001,"15min":1.2589999999999999},"min":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.7639999999999998,"5min":4.0410000000000004,"15min":5.9210000000000003},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.304,"15min":1.3009999999999999},"min":{"1min":0.56999999999999995,"5min":0.52400000000000002,"15min":0.5},"max":{"1min":5.7949999999999999,"5min":5.7949999999999999,"15min":6.1379999999999999},"last":1.091}]},{"osd":11,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2070000000000001,"15min":1.204},"min":{"1min":0.503,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.806,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.367,"15min":1.3120000000000001},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":5.8230000000000004,"5min":5.8230000000000004,"15min":5.8230000000000004},"last":1.129}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2297888768,"data_stored":2297888768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1811742720,"data_stored":1811742720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1357905920,"data_stored":1357905920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":462553088,"data_stored":462553088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1870921728,"data_stored":1870921728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1821446144,"data_stored":1821442072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1381695488,"data_stored":1381695488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":452198400,"data_stored":452198400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":876740608,"data_stored":876740608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":903479296,"data_stored":903479296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":930676736,"data_stored":930676736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":470552576,"data_stored":470548504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:26:56.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T16:26:56.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:56.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:56.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:56.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:56 smithi121 bash[18489]: cluster 2024-02-12T16:26:55.858639+0000 osd.7 (osd.7) 144 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:26:56.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:56 smithi121 bash[18489]: cluster 2024-02-12T16:26:55.975293+0000 mgr.y (mgr.14146) 1258 : cluster [DBG] pgmap v1747: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 13023/218632 objects misplaced (5.957%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:26:56.910 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:56.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:57.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:57.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:56 smithi123 bash[19119]: cluster 2024-02-12T16:26:55.858639+0000 osd.7 (osd.7) 144 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:26:57.021 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:56 smithi123 bash[19119]: cluster 2024-02-12T16:26:55.975293+0000 mgr.y (mgr.14146) 1258 : cluster [DBG] pgmap v1747: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 13023/218632 objects misplaced (5.957%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:26:57.040 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:26:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:57.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:26:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:56 smithi169 bash[19604]: cluster 2024-02-12T16:26:55.858639+0000 osd.7 (osd.7) 144 : cluster [DBG] 5.9 scrub starts 2024-02-12T16:26:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:56 smithi169 bash[19604]: cluster 2024-02-12T16:26:55.975293+0000 mgr.y (mgr.14146) 1258 : cluster [DBG] pgmap v1747: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 13023/218632 objects misplaced (5.957%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:26:57.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:57.178 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:57.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:57.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:26:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:57.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:26:57.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:57.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:57.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:57.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:57.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:26:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:57.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:26:57.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:57.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:57.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:57.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:57 smithi123 bash[19119]: cluster 2024-02-12T16:26:56.566396+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:57 smithi123 bash[19119]: cluster 2024-02-12T16:26:56.566480+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:57.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:57 smithi123 bash[19119]: cluster 2024-02-12T16:26:56.767810+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e599: 12 total, 12 up, 12 in 2024-02-12T16:26:57.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:57.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:57.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:57 smithi169 bash[19604]: cluster 2024-02-12T16:26:56.566396+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:57 smithi169 bash[19604]: cluster 2024-02-12T16:26:56.566480+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:57 smithi169 bash[19604]: cluster 2024-02-12T16:26:56.767810+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e599: 12 total, 12 up, 12 in 2024-02-12T16:26:58.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:26:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:58.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:26:58.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:57 smithi121 bash[18489]: cluster 2024-02-12T16:26:56.566396+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:26:58.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:57 smithi121 bash[18489]: cluster 2024-02-12T16:26:56.566480+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:26:58.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:57 smithi121 bash[18489]: cluster 2024-02-12T16:26:56.767810+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e599: 12 total, 12 up, 12 in 2024-02-12T16:26:58.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:58.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:58.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:58.306 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:58.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:58.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:58.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:58.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:26:58.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:58.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:26:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:58.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:26:58.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:58.732 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:58.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:58.789 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:26:58.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:58.868 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:58.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:58.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:58.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:58 smithi169 bash[19604]: cluster 2024-02-12T16:26:57.827276+0000 osd.7 (osd.7) 145 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:26:58.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:58 smithi169 bash[19604]: cluster 2024-02-12T16:26:57.975763+0000 mgr.y (mgr.14146) 1259 : cluster [DBG] pgmap v1749: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 12669/213498 objects misplaced (5.934%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:26:58.979 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:58 smithi169 bash[19604]: audit 2024-02-12T16:26:58.087296+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? v1:172.21.15.121:0/3708389526' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:59.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:59.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:59.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:26:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:26:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:58 smithi123 bash[19119]: cluster 2024-02-12T16:26:57.827276+0000 osd.7 (osd.7) 145 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:26:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:58 smithi123 bash[19119]: cluster 2024-02-12T16:26:57.975763+0000 mgr.y (mgr.14146) 1259 : cluster [DBG] pgmap v1749: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 12669/213498 objects misplaced (5.934%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:26:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:58 smithi123 bash[19119]: audit 2024-02-12T16:26:58.087296+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? v1:172.21.15.121:0/3708389526' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:58 smithi121 bash[18489]: cluster 2024-02-12T16:26:57.827276+0000 osd.7 (osd.7) 145 : cluster [DBG] 5.9 scrub ok 2024-02-12T16:26:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:58 smithi121 bash[18489]: cluster 2024-02-12T16:26:57.975763+0000 mgr.y (mgr.14146) 1259 : cluster [DBG] pgmap v1749: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 12669/213498 objects misplaced (5.934%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:26:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:58 smithi121 bash[18489]: audit 2024-02-12T16:26:58.087296+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? v1:172.21.15.121:0/3708389526' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:26:59.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:26:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:26:59.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:26:59.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:59.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:26:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:26:59.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:59.544 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:59.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:59.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:59.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:26:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:26:59.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:59.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:26:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:26:59.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:59.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:26:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:26:59.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:59 smithi169 bash[19604]: cluster 2024-02-12T16:26:58.765546+0000 mon.a (mon.0) 2133 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:27:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:59 smithi169 bash[19604]: cluster 2024-02-12T16:26:58.765576+0000 mon.a (mon.0) 2134 : cluster [INF] Cluster is now healthy 2024-02-12T16:27:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:59 smithi169 bash[19604]: audit 2024-02-12T16:26:58.780021+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? v1:172.21.15.121:0/3708389526' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:27:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:26:59 smithi169 bash[19604]: cluster 2024-02-12T16:26:58.780101+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e600: 12 total, 12 up, 12 in 2024-02-12T16:27:00.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:59 smithi121 bash[18489]: cluster 2024-02-12T16:26:58.765546+0000 mon.a (mon.0) 2133 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:27:00.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:59 smithi121 bash[18489]: cluster 2024-02-12T16:26:58.765576+0000 mon.a (mon.0) 2134 : cluster [INF] Cluster is now healthy 2024-02-12T16:27:00.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:59 smithi121 bash[18489]: audit 2024-02-12T16:26:58.780021+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? v1:172.21.15.121:0/3708389526' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:27:00.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:26:59 smithi121 bash[18489]: cluster 2024-02-12T16:26:58.780101+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e600: 12 total, 12 up, 12 in 2024-02-12T16:27:00.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:00.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:00.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:59 smithi123 bash[19119]: cluster 2024-02-12T16:26:58.765546+0000 mon.a (mon.0) 2133 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:27:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:59 smithi123 bash[19119]: cluster 2024-02-12T16:26:58.765576+0000 mon.a (mon.0) 2134 : cluster [INF] Cluster is now healthy 2024-02-12T16:27:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:59 smithi123 bash[19119]: audit 2024-02-12T16:26:58.780021+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? v1:172.21.15.121:0/3708389526' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:27:00.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:26:59 smithi123 bash[19119]: cluster 2024-02-12T16:26:58.780101+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e600: 12 total, 12 up, 12 in 2024-02-12T16:27:00.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:00.246 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:00.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:00.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:00.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:00.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:00.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:00.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:00.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:00.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:00.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:00.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:00.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:00 smithi169 bash[19604]: cluster 2024-02-12T16:26:59.706117+0000 osd.4 (osd.4) 98 : cluster [DBG] 1.e scrub starts 2024-02-12T16:27:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:00 smithi169 bash[19604]: cluster 2024-02-12T16:26:59.766734+0000 osd.4 (osd.4) 99 : cluster [DBG] 1.e scrub ok 2024-02-12T16:27:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:00 smithi169 bash[19604]: cluster 2024-02-12T16:26:59.976185+0000 mgr.y (mgr.14146) 1260 : cluster [DBG] pgmap v1751: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 12669/211522 objects misplaced (5.989%); 919 KiB/s, 14 objects/s recovering 2024-02-12T16:27:00.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:00.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:00.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:00 smithi123 bash[19119]: cluster 2024-02-12T16:26:59.706117+0000 osd.4 (osd.4) 98 : cluster [DBG] 1.e scrub starts 2024-02-12T16:27:00.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:00 smithi123 bash[19119]: cluster 2024-02-12T16:26:59.766734+0000 osd.4 (osd.4) 99 : cluster [DBG] 1.e scrub ok 2024-02-12T16:27:00.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:00 smithi123 bash[19119]: cluster 2024-02-12T16:26:59.976185+0000 mgr.y (mgr.14146) 1260 : cluster [DBG] pgmap v1751: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 12669/211522 objects misplaced (5.989%); 919 KiB/s, 14 objects/s recovering 2024-02-12T16:27:00.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:00.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:00.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:01.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:01.081 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:01.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:01.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:00 smithi121 bash[18489]: cluster 2024-02-12T16:26:59.706117+0000 osd.4 (osd.4) 98 : cluster [DBG] 1.e scrub starts 2024-02-12T16:27:01.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:00 smithi121 bash[18489]: cluster 2024-02-12T16:26:59.766734+0000 osd.4 (osd.4) 99 : cluster [DBG] 1.e scrub ok 2024-02-12T16:27:01.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:00 smithi121 bash[18489]: cluster 2024-02-12T16:26:59.976185+0000 mgr.y (mgr.14146) 1260 : cluster [DBG] pgmap v1751: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 12669/211522 objects misplaced (5.989%); 919 KiB/s, 14 objects/s recovering 2024-02-12T16:27:01.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:01.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:01.264 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:27:01.264 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:27:01.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:01.362 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:01.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:01.469 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:01.495 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:01.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:01.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:01.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:01.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:01.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:01.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:01.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:01.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:01.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:02.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:02.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:27:02.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:02.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:02.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:27:02.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:02.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:02.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:02.307 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:02.331 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:02.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:02.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:02.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:02.546 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:02.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:02.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:02.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:02.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:02.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:02.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:02.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:02.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:02.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:03.015 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:03.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:03.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:03 smithi169 bash[19604]: cluster 2024-02-12T16:27:01.976685+0000 mgr.y (mgr.14146) 1261 : cluster [DBG] pgmap v1752: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 11857/204274 objects misplaced (5.804%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:27:03.144 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:03.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:03.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:03.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:03.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:03 smithi123 bash[19119]: cluster 2024-02-12T16:27:01.976685+0000 mgr.y (mgr.14146) 1261 : cluster [DBG] pgmap v1752: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 11857/204274 objects misplaced (5.804%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:27:03.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:03.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:03 smithi121 bash[18489]: cluster 2024-02-12T16:27:01.976685+0000 mgr.y (mgr.14146) 1261 : cluster [DBG] pgmap v1752: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 11857/204274 objects misplaced (5.804%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:27:03.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:03.534 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:03.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:03.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:03.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:03.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:03.952 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:03.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:04.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:04.093 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:04.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:04.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:04.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:04.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:04.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:04.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:04.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:04.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:04.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:04.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:04.578 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:04.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:04.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:04.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:04.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:04.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:04.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:04.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:04.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:04.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:05.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:05.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:05.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:05.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:05.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:05.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:05 smithi169 bash[19604]: cluster 2024-02-12T16:27:03.977156+0000 mgr.y (mgr.14146) 1262 : cluster [DBG] pgmap v1753: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 11857/201384 objects misplaced (5.888%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:05.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:05.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:05.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:05 smithi123 bash[19119]: cluster 2024-02-12T16:27:03.977156+0000 mgr.y (mgr.14146) 1262 : cluster [DBG] pgmap v1753: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 11857/201384 objects misplaced (5.888%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:05 smithi121 bash[18489]: cluster 2024-02-12T16:27:03.977156+0000 mgr.y (mgr.14146) 1262 : cluster [DBG] pgmap v1753: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 11857/201384 objects misplaced (5.888%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:05.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:05.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:05.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:05.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:05.594 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:05.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:05.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:05.722 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:05.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:05.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:05.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:05.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:05.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:06.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:06.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:06.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:06.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:06.284 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:06.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:06.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:06.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:06.573 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:06.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:06.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:06.704 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:06.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:06.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:06.855 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:27:06.880 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:06 smithi123 bash[19119]: cluster 2024-02-12T16:27:05.977622+0000 mgr.y (mgr.14146) 1263 : cluster [DBG] pgmap v1754: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 11470/197172 objects misplaced (5.817%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:06.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:06.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:06.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:06 smithi169 bash[19604]: cluster 2024-02-12T16:27:05.977622+0000 mgr.y (mgr.14146) 1263 : cluster [DBG] pgmap v1754: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 11470/197172 objects misplaced (5.817%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:07.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:07.099 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:07.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:07.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:06 smithi121 bash[18489]: cluster 2024-02-12T16:27:05.977622+0000 mgr.y (mgr.14146) 1263 : cluster [DBG] pgmap v1754: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 6.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 11470/197172 objects misplaced (5.817%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:07.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:07.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:07.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:07.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:07.362 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:07.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:07.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:07.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:07.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:07.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:07.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:07.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:07.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:07.748 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:07.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:07.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:07 smithi123 bash[19119]: cluster 2024-02-12T16:27:07.112496+0000 osd.2 (osd.2) 80 : cluster [DBG] 5.a scrub starts 2024-02-12T16:27:07.891 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:07.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:08.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:08.009 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:07 smithi121 bash[18489]: cluster 2024-02-12T16:27:07.112496+0000 osd.2 (osd.2) 80 : cluster [DBG] 5.a scrub starts 2024-02-12T16:27:08.043 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:07 smithi169 bash[19604]: cluster 2024-02-12T16:27:07.112496+0000 osd.2 (osd.2) 80 : cluster [DBG] 5.a scrub starts 2024-02-12T16:27:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:08.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:08.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:08.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:08.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:08.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:08.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:08.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:08.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:08.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:08.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:08.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:08.610 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:08.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:08.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:08.750 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:08.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:08.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:08 smithi169 bash[19604]: cluster 2024-02-12T16:27:07.978008+0000 mgr.y (mgr.14146) 1264 : cluster [DBG] pgmap v1755: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 11105/191658 objects misplaced (5.794%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:27:08.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:08.878 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:08.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:08 smithi123 bash[19119]: cluster 2024-02-12T16:27:07.978008+0000 mgr.y (mgr.14146) 1264 : cluster [DBG] pgmap v1755: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 11105/191658 objects misplaced (5.794%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:27:08.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:09.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:09.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:09.128 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:08 smithi121 bash[18489]: cluster 2024-02-12T16:27:07.978008+0000 mgr.y (mgr.14146) 1264 : cluster [DBG] pgmap v1755: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 11105/191658 objects misplaced (5.794%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:27:09.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:09.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:09.300 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:09.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:09.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:09.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:09.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:09.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:09.704 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:09.807 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:09.830 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:09.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:09.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:09.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:10.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:10.092 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:10.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:10.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:10.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:10.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:10.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:10.355 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:10.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:10.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:10.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:10.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:10.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:10.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:10.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:10.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:10.761 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:10.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:10.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:10.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:10.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:10.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:11.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:11.042 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:27:11.042 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:27:11.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:11.169 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:11.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:11.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:11 smithi169 bash[19604]: cluster 2024-02-12T16:27:09.548936+0000 osd.2 (osd.2) 81 : cluster [DBG] 5.a scrub ok 2024-02-12T16:27:11.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:11 smithi169 bash[19604]: cluster 2024-02-12T16:27:09.978455+0000 mgr.y (mgr.14146) 1265 : cluster [DBG] pgmap v1756: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 11105/189546 objects misplaced (5.859%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:27:11.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:11 smithi169 bash[19604]: cluster 2024-02-12T16:27:10.042680+0000 osd.2 (osd.2) 82 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:27:11.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:11.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:11 smithi121 bash[18489]: cluster 2024-02-12T16:27:09.548936+0000 osd.2 (osd.2) 81 : cluster [DBG] 5.a scrub ok 2024-02-12T16:27:11.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:11 smithi121 bash[18489]: cluster 2024-02-12T16:27:09.978455+0000 mgr.y (mgr.14146) 1265 : cluster [DBG] pgmap v1756: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 11105/189546 objects misplaced (5.859%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:27:11.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:11 smithi121 bash[18489]: cluster 2024-02-12T16:27:10.042680+0000 osd.2 (osd.2) 82 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:27:11.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:11 smithi123 bash[19119]: cluster 2024-02-12T16:27:09.548936+0000 osd.2 (osd.2) 81 : cluster [DBG] 5.a scrub ok 2024-02-12T16:27:11.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:11 smithi123 bash[19119]: cluster 2024-02-12T16:27:09.978455+0000 mgr.y (mgr.14146) 1265 : cluster [DBG] pgmap v1756: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 11105/189546 objects misplaced (5.859%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:27:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:11 smithi123 bash[19119]: cluster 2024-02-12T16:27:10.042680+0000 osd.2 (osd.2) 82 : cluster [DBG] 5.6 scrub starts 2024-02-12T16:27:11.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:11.459 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:11.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:11.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:11.597 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:27:11.597 INFO:teuthology.orchestra.run.smithi121.stdout:{"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2228,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":600,"num_osds":12,"num_up_osds":12,"osd_up_since":1707754981,"num_in_osds":12,"osd_in_since":1707754767,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":35},{"state_name":"active+remapped+backfilling","count":2}],"num_pgs":37,"num_pools":2,"num_objects":94773,"data_bytes":6210977816,"bytes_used":16888422400,"bytes_avail":1135052005376,"bytes_total":1151940427776,"misplaced_objects":11105,"misplaced_total":189546,"misplaced_ratio":0.058587361379295794,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":1363246,"recovering_keys_per_sec":0,"num_objects_recovered":233,"num_bytes_recovered":15269888,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1012},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.121:8443/"}},"servicemap":{"epoch":52,"modified":"2024-02-12T16:26:53.974601+0000","services":{}},"progress_events":{}} 2024-02-12T16:27:11.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:11.719 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:11.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:11.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:11.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:11.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-12T16:27:11.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:11.994 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:11.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:12.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:12.126 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:12 smithi121 bash[18489]: audit 2024-02-12T16:27:11.594612+0000 mon.b (mon.2) 138 : audit [DBG] from='client.? v1:172.21.15.121:0/1350248991' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:27:12.127 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:12.133 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:12 smithi169 bash[19604]: audit 2024-02-12T16:27:11.594612+0000 mon.b (mon.2) 138 : audit [DBG] from='client.? v1:172.21.15.121:0/1350248991' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:27:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:12.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:12 smithi123 bash[19119]: audit 2024-02-12T16:27:11.594612+0000 mon.b (mon.2) 138 : audit [DBG] from='client.? v1:172.21.15.121:0/1350248991' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:27:12.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:12.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:12.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:12.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:12.390 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:12.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:12.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:12.521 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:12.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:12.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:12.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:12.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:12.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:12.800 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:12.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:12.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:12.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:12.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:13.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:13.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:13.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:13 smithi121 bash[18489]: cluster 2024-02-12T16:27:11.687764+0000 osd.5 (osd.5) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:13.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:13.189 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:13.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:13.215 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:27:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:27:13.286 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:27:13.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:13.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:13 smithi123 bash[19119]: cluster 2024-02-12T16:27:11.687764+0000 osd.5 (osd.5) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:13.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:13 smithi123 bash[19119]: cluster 2024-02-12T16:27:11.687800+0000 osd.5 (osd.5) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:13.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:13 smithi123 bash[19119]: cluster 2024-02-12T16:27:11.978947+0000 mgr.y (mgr.14146) 1266 : cluster [DBG] pgmap v1757: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 10349/182260 objects misplaced (5.678%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:13.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:13 smithi123 bash[19119]: cluster 2024-02-12T16:27:12.046325+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e601: 12 total, 12 up, 12 in 2024-02-12T16:27:13.320 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:13.326 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:27:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:13.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:13.368 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:27:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:13 smithi121 bash[18489]: cluster 2024-02-12T16:27:11.687800+0000 osd.5 (osd.5) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:13 smithi121 bash[18489]: cluster 2024-02-12T16:27:11.978947+0000 mgr.y (mgr.14146) 1266 : cluster [DBG] pgmap v1757: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 10349/182260 objects misplaced (5.678%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:13 smithi121 bash[18489]: cluster 2024-02-12T16:27:12.046325+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e601: 12 total, 12 up, 12 in 2024-02-12T16:27:13.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:27:13.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:13.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:27:13.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:27:13.495 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-02-12T16:27:13.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:27:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:27:13.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:13 smithi169 bash[19604]: cluster 2024-02-12T16:27:11.687764+0000 osd.5 (osd.5) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:13.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:13 smithi169 bash[19604]: cluster 2024-02-12T16:27:11.687800+0000 osd.5 (osd.5) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:13 smithi169 bash[19604]: cluster 2024-02-12T16:27:11.978947+0000 mgr.y (mgr.14146) 1266 : cluster [DBG] pgmap v1757: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 10349/182260 objects misplaced (5.678%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:13 smithi169 bash[19604]: cluster 2024-02-12T16:27:12.046325+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e601: 12 total, 12 up, 12 in 2024-02-12T16:27:13.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:13.567 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:27:13.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:27:13.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:13.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:13.644 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.651 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.652 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.652 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:27:13.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:13.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:13.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:13.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:13.900 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:13.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:14.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:14.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:14.057 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:14 smithi121 bash[18489]: cluster 2024-02-12T16:27:12.609490+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:27:14.057 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:14 smithi121 bash[18489]: audit 2024-02-12T16:27:13.169167+0000 mon.b (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.121:0/3022283560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:14.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:14 smithi123 bash[19119]: cluster 2024-02-12T16:27:12.609490+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:27:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:14 smithi123 bash[19119]: audit 2024-02-12T16:27:13.169167+0000 mon.b (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.121:0/3022283560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:14.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:14.268 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:14.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:14.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:14.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:14.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:14.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:14 smithi169 bash[19604]: cluster 2024-02-12T16:27:12.609490+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.6 scrub ok 2024-02-12T16:27:14.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:14 smithi169 bash[19604]: audit 2024-02-12T16:27:13.169167+0000 mon.b (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.121:0/3022283560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:14.553 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:14.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:14.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:14.790 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:14.815 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:14.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:14.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:14.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:15.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:15.060 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:15 smithi123 bash[19119]: cluster 2024-02-12T16:27:13.979350+0000 mgr.y (mgr.14146) 1267 : cluster [DBG] pgmap v1759: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 10349/179668 objects misplaced (5.760%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:15.060 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:15 smithi123 bash[19119]: cluster 2024-02-12T16:27:14.050377+0000 osd.7 (osd.7) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:15.060 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:15 smithi123 bash[19119]: cluster 2024-02-12T16:27:14.050429+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:15.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:15.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:15.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:15.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:15.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:15.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:15 smithi169 bash[19604]: cluster 2024-02-12T16:27:13.979350+0000 mgr.y (mgr.14146) 1267 : cluster [DBG] pgmap v1759: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 10349/179668 objects misplaced (5.760%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:15 smithi169 bash[19604]: cluster 2024-02-12T16:27:14.050377+0000 osd.7 (osd.7) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:15.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:15 smithi169 bash[19604]: cluster 2024-02-12T16:27:14.050429+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:15.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:15.349 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:15.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:15.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:15 smithi121 bash[18489]: cluster 2024-02-12T16:27:13.979350+0000 mgr.y (mgr.14146) 1267 : cluster [DBG] pgmap v1759: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 10349/179668 objects misplaced (5.760%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:15.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:15 smithi121 bash[18489]: cluster 2024-02-12T16:27:14.050377+0000 osd.7 (osd.7) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:15 smithi121 bash[18489]: cluster 2024-02-12T16:27:14.050429+0000 osd.7 (osd.7) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:15.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:15.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:15.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:15.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:15.604 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:15.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:15.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:15.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:15.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:15.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:15.900 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:15.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:16.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:16.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:16.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:16.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:16.182 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:16.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:16.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:16.293 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:16 smithi121 bash[18489]: cluster 2024-02-12T16:27:15.051585+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e602: 12 total, 12 up, 12 in 2024-02-12T16:27:16.294 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:16 smithi121 bash[18489]: audit 2024-02-12T16:27:15.794550+0000 mon.a (mon.0) 2139 : audit [INF] from='client.26095 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:27:16.294 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:16 smithi121 bash[18489]: audit 2024-02-12T16:27:15.796012+0000 mon.c (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.121:0/4217370709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:27:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:16.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:16.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:16 smithi123 bash[19119]: cluster 2024-02-12T16:27:15.051585+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e602: 12 total, 12 up, 12 in 2024-02-12T16:27:16.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:16 smithi123 bash[19119]: audit 2024-02-12T16:27:15.794550+0000 mon.a (mon.0) 2139 : audit [INF] from='client.26095 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:27:16.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:16 smithi123 bash[19119]: audit 2024-02-12T16:27:15.796012+0000 mon.c (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.121:0/4217370709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:27:16.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:16.436 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:16 smithi169 bash[19604]: cluster 2024-02-12T16:27:15.051585+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e602: 12 total, 12 up, 12 in 2024-02-12T16:27:16.436 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:16 smithi169 bash[19604]: audit 2024-02-12T16:27:15.794550+0000 mon.a (mon.0) 2139 : audit [INF] from='client.26095 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:27:16.436 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:16 smithi169 bash[19604]: audit 2024-02-12T16:27:15.796012+0000 mon.c (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.121:0/4217370709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:27:16.464 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:16.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:16.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:16.594 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:16.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:16.658 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:27:16.692 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:27:16.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:16.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:16.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:16.892 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:16.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:16.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:16.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:17.033 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:17.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:17.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:17 smithi123 bash[19119]: cluster 2024-02-12T16:27:15.979839+0000 mgr.y (mgr.14146) 1268 : cluster [DBG] pgmap v1761: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 10001/175790 objects misplaced (5.689%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:17.061 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:17 smithi123 bash[19119]: cluster 2024-02-12T16:27:16.659542+0000 mon.a (mon.0) 2140 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:17.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:17.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:27:17.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:17.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:17.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:17 smithi121 bash[18489]: cluster 2024-02-12T16:27:15.979839+0000 mgr.y (mgr.14146) 1268 : cluster [DBG] pgmap v1761: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 10001/175790 objects misplaced (5.689%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:17.299 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:17 smithi121 bash[18489]: cluster 2024-02-12T16:27:16.659542+0000 mon.a (mon.0) 2140 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:17.299 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:17 smithi121 bash[18489]: audit 2024-02-12T16:27:16.675346+0000 mon.a (mon.0) 2141 : audit [INF] from='client.26095 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:27:17.299 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:17 smithi121 bash[18489]: cluster 2024-02-12T16:27:16.675566+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e603: 12 total, 12 up, 12 in 2024-02-12T16:27:17.334 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:17.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:17 smithi123 bash[19119]: audit 2024-02-12T16:27:16.675346+0000 mon.a (mon.0) 2141 : audit [INF] from='client.26095 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:27:17.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:17 smithi123 bash[19119]: cluster 2024-02-12T16:27:16.675566+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e603: 12 total, 12 up, 12 in 2024-02-12T16:27:17.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:17.450 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:17 smithi169 bash[19604]: cluster 2024-02-12T16:27:15.979839+0000 mgr.y (mgr.14146) 1268 : cluster [DBG] pgmap v1761: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 34 active+clean; 5.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 10001/175790 objects misplaced (5.689%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:17.450 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:17 smithi169 bash[19604]: cluster 2024-02-12T16:27:16.659542+0000 mon.a (mon.0) 2140 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:17.450 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:17 smithi169 bash[19604]: audit 2024-02-12T16:27:16.675346+0000 mon.a (mon.0) 2141 : audit [INF] from='client.26095 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:27:17.450 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:17 smithi169 bash[19604]: cluster 2024-02-12T16:27:16.675566+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e603: 12 total, 12 up, 12 in 2024-02-12T16:27:17.465 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:17.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:17.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:17.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:17.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:17.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:17.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:17.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:17.987 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:17.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:18.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:18.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:18.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:18.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:18.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:18.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:18.382 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:18.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:18.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:18.510 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:18.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:18.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:18.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:18.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:18.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:18.786 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:18.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:18.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:18.918 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:18.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:19.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:19.056 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:19.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:19 smithi121 bash[18489]: cluster 2024-02-12T16:27:17.980264+0000 mgr.y (mgr.14146) 1269 : cluster [DBG] pgmap v1763: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s; 9660/170588 objects misplaced (5.663%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:19.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:19.186 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:19.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:19.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:19 smithi123 bash[19119]: cluster 2024-02-12T16:27:17.980264+0000 mgr.y (mgr.14146) 1269 : cluster [DBG] pgmap v1763: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s; 9660/170588 objects misplaced (5.663%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:19.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:19.444 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:19.523 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:19 smithi169 bash[19604]: cluster 2024-02-12T16:27:17.980264+0000 mgr.y (mgr.14146) 1269 : cluster [DBG] pgmap v1763: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s; 9660/170588 objects misplaced (5.663%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:19.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:19.575 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:19.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:19.706 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:19.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:19.836 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:19.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:27:19.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:19.961 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:19.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:20.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:20.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:20.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:20.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:20.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:20.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:20.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:20.347 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:20.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:20.376 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:20.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:20.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:20.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:20.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:20.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:20.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:20.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:20.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:20.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:20.930 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:20.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:21.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:21.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:21.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:21.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:21.197 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:21.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:21.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:21 smithi169 bash[19604]: cluster 2024-02-12T16:27:19.980726+0000 mgr.y (mgr.14146) 1270 : cluster [DBG] pgmap v1764: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 9660/168536 objects misplaced (5.732%); 944 KiB/s, 14 objects/s recovering 2024-02-12T16:27:21.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:21 smithi169 bash[19604]: cluster 2024-02-12T16:27:20.098914+0000 osd.1 (osd.1) 86 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:21.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:21 smithi169 bash[19604]: cluster 2024-02-12T16:27:20.098966+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:21.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:21.309 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:21 smithi123 bash[19119]: cluster 2024-02-12T16:27:19.980726+0000 mgr.y (mgr.14146) 1270 : cluster [DBG] pgmap v1764: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 9660/168536 objects misplaced (5.732%); 944 KiB/s, 14 objects/s recovering 2024-02-12T16:27:21.309 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:21 smithi123 bash[19119]: cluster 2024-02-12T16:27:20.098914+0000 osd.1 (osd.1) 86 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:21.309 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:21 smithi123 bash[19119]: cluster 2024-02-12T16:27:20.098966+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:21.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:21.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:21 smithi121 bash[18489]: cluster 2024-02-12T16:27:19.980726+0000 mgr.y (mgr.14146) 1270 : cluster [DBG] pgmap v1764: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 5.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 9660/168536 objects misplaced (5.732%); 944 KiB/s, 14 objects/s recovering 2024-02-12T16:27:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:21 smithi121 bash[18489]: cluster 2024-02-12T16:27:20.098914+0000 osd.1 (osd.1) 86 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:21 smithi121 bash[18489]: cluster 2024-02-12T16:27:20.098966+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:21.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:21.473 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:21.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:21.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:21.601 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:21.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:21.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:21.739 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:21.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:21.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:21.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:21.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:21.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:21.998 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:22.077 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:22 smithi169 bash[19604]: cluster 2024-02-12T16:27:21.064329+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e604: 12 total, 12 up, 12 in 2024-02-12T16:27:22.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:22.104 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:27:22.104 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:27:22.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.104 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.16095.0:260492 5.e e604/547 5:7b4e370a:::benchmark_data_smithi169_84712_object91102:head v 604'16344, mlcod=604'16342)", 2024-02-12T16:27:22.104 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:27:22.097816+0000", 2024-02-12T16:27:22.104 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0033473890000000001, 2024-02-12T16:27:22.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033729369999999999, 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097816+0000", 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097816+0000", 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097814+0000", 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1148999999999999e-05 2024-02-12T16:27:22.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097825+0000", 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5421999999999999e-05 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097840+0000", 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9284e-05 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097859+0000", 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7476e-05 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097957+0000", 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7667000000000001e-05 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:22.097985+0000", 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016903299999999999 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 1 2024-02-12T16:27:22.107 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:27:22.126 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:22.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:22.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:22.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:22.375 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:22 smithi121 bash[18489]: cluster 2024-02-12T16:27:21.064329+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e604: 12 total, 12 up, 12 in 2024-02-12T16:27:22.376 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:22 smithi123 bash[19119]: cluster 2024-02-12T16:27:21.064329+0000 mon.a (mon.0) 2143 : cluster [DBG] osdmap e604: 12 total, 12 up, 12 in 2024-02-12T16:27:22.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:22.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:22.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:22.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:22.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:22.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:22.783 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:22.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:22.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-12T16:27:22.920 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:22.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:23.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:23.057 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:23.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:23.077 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:23 smithi123 bash[19119]: cluster 2024-02-12T16:27:21.981173+0000 mgr.y (mgr.14146) 1271 : cluster [DBG] pgmap v1766: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s; 8962/161182 objects misplaced (5.560%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:27:23.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:23.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:23.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:23 smithi169 bash[19604]: cluster 2024-02-12T16:27:21.981173+0000 mgr.y (mgr.14146) 1271 : cluster [DBG] pgmap v1766: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s; 8962/161182 objects misplaced (5.560%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:27:23.305 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:23 smithi121 bash[18489]: cluster 2024-02-12T16:27:21.981173+0000 mgr.y (mgr.14146) 1271 : cluster [DBG] pgmap v1766: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s; 8962/161182 objects misplaced (5.560%); 2.1 MiB/s, 33 objects/s recovering 2024-02-12T16:27:23.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:23.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:23.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:23.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:23.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:23.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:23.632 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:23.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:23.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:23.755 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:23.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:23.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:23.883 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:23.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:23.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:24.015 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:24.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:24.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:24 smithi169 bash[19604]: audit 2024-02-12T16:27:24.061890+0000 mon.a (mon.0) 2144 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:27:24.146 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:24.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:24.290 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:24.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:24 smithi123 bash[19119]: audit 2024-02-12T16:27:24.061890+0000 mon.a (mon.0) 2144 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:27:24.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:24.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:24 smithi121 bash[18489]: audit 2024-02-12T16:27:24.061890+0000 mon.a (mon.0) 2144 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:27:24.427 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:24.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:24.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:24.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:24.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:24.762 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:24.790 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:24.824 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:24.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:24.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:25.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:25 smithi123 bash[19119]: cluster 2024-02-12T16:27:23.981610+0000 mgr.y (mgr.14146) 1272 : cluster [DBG] pgmap v1767: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 8962/158544 objects misplaced (5.653%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:25.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:25.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:25.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:25.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:25.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:25.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:25.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:25 smithi121 bash[18489]: cluster 2024-02-12T16:27:23.981610+0000 mgr.y (mgr.14146) 1272 : cluster [DBG] pgmap v1767: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 8962/158544 objects misplaced (5.653%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:25.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:25.352 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:25 smithi169 bash[19604]: cluster 2024-02-12T16:27:23.981610+0000 mgr.y (mgr.14146) 1272 : cluster [DBG] pgmap v1767: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 8962/158544 objects misplaced (5.653%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:25.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:25.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:25.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:25.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:25.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:25.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:25.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:25.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:25.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:25.765 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:25.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:25.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:25.896 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:25.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:26.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:26.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:26.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:26.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:26.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:26.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:26.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:26.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:26.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:26.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:26.692 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:26 smithi123 bash[19119]: cluster 2024-02-12T16:27:25.982038+0000 mgr.y (mgr.14146) 1273 : cluster [DBG] pgmap v1768: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 937 op/s; 8609/154450 objects misplaced (5.574%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:27:26.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:26.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:26.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:26.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:26 smithi121 bash[18489]: cluster 2024-02-12T16:27:25.982038+0000 mgr.y (mgr.14146) 1273 : cluster [DBG] pgmap v1768: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 937 op/s; 8609/154450 objects misplaced (5.574%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:27:26.987 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:26.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:26 smithi169 bash[19604]: cluster 2024-02-12T16:27:25.982038+0000 mgr.y (mgr.14146) 1273 : cluster [DBG] pgmap v1768: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 937 op/s; 8609/154450 objects misplaced (5.574%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:27:27.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:27.121 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:27.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:27.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:27.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:27.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:27.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:27.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:27.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:27.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:27.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:27.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:27.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:27.671 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:27.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:27.726 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:27:27.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:27.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:27.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:27.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:27.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:27.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:28.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:28.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:28.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:28.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:28.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:28.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:28.342 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:27:28.350 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:27:28.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:28.398 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:28.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:28.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:28.538 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:28.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:28.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:28.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:28.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:28.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:28.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:28.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:28.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:28.931 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:28.932 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:28 smithi123 bash[19119]: cluster 2024-02-12T16:27:27.416300+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:28.932 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:28 smithi123 bash[19119]: cluster 2024-02-12T16:27:27.416354+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:28.932 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:28 smithi123 bash[19119]: cluster 2024-02-12T16:27:27.671523+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e605: 12 total, 12 up, 12 in 2024-02-12T16:27:28.932 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:28 smithi123 bash[19119]: cluster 2024-02-12T16:27:27.982464+0000 mgr.y (mgr.14146) 1274 : cluster [DBG] pgmap v1770: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 8253/148924 objects misplaced (5.542%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:27:28.932 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:28 smithi123 bash[19119]: audit 2024-02-12T16:27:28.323348+0000 mgr.y (mgr.14146) 1275 : audit [DBG] from='client.16272 v1:172.21.15.121:0/1787566481' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:27:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:28.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:29.031 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:27:29.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:29.041 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:28 smithi121 bash[18489]: cluster 2024-02-12T16:27:27.416300+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:29.041 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:28 smithi121 bash[18489]: cluster 2024-02-12T16:27:27.416354+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:29.041 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:28 smithi121 bash[18489]: cluster 2024-02-12T16:27:27.671523+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e605: 12 total, 12 up, 12 in 2024-02-12T16:27:29.041 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:28 smithi121 bash[18489]: cluster 2024-02-12T16:27:27.982464+0000 mgr.y (mgr.14146) 1274 : cluster [DBG] pgmap v1770: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 8253/148924 objects misplaced (5.542%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:27:29.041 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:28 smithi121 bash[18489]: audit 2024-02-12T16:27:28.323348+0000 mgr.y (mgr.14146) 1275 : audit [DBG] from='client.16272 v1:172.21.15.121:0/1787566481' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:27:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:28 smithi169 bash[19604]: cluster 2024-02-12T16:27:27.416300+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:28 smithi169 bash[19604]: cluster 2024-02-12T16:27:27.416354+0000 osd.4 (osd.4) 101 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:28 smithi169 bash[19604]: cluster 2024-02-12T16:27:27.671523+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e605: 12 total, 12 up, 12 in 2024-02-12T16:27:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:28 smithi169 bash[19604]: cluster 2024-02-12T16:27:27.982464+0000 mgr.y (mgr.14146) 1274 : cluster [DBG] pgmap v1770: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s; 8253/148924 objects misplaced (5.542%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:27:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:28 smithi169 bash[19604]: audit 2024-02-12T16:27:28.323348+0000 mgr.y (mgr.14146) 1275 : audit [DBG] from='client.16272 v1:172.21.15.121:0/1787566481' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:27:29.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:29.074 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:27:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:29.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:29.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:27:29.142 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:27:29.180 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:27:29.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:29.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:29.211 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:27:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:29.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:29.249 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:27:29.276 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:27:29.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:27:29.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:29.340 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:27:29.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:29.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:29.367 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:27:29.392 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.394 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:27:29.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:29.470 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:29.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:29.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:29.598 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:29.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:29.681 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:29 smithi169 bash[19604]: audit 2024-02-12T16:27:28.985616+0000 mon.a (mon.0) 2146 : audit [DBG] from='client.? v1:172.21.15.121:0/2826634895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:29.681 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:29 smithi169 bash[19604]: audit 2024-02-12T16:27:29.352720+0000 mon.a (mon.0) 2147 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:27:29.681 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:29 smithi169 bash[19604]: audit 2024-02-12T16:27:29.665762+0000 mon.a (mon.0) 2148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:27:29.681 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:29 smithi169 bash[19604]: audit 2024-02-12T16:27:29.666669+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:27:29.681 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:29 smithi169 bash[19604]: audit 2024-02-12T16:27:29.671959+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:27:29.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:29.719 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:29 smithi121 bash[18489]: audit 2024-02-12T16:27:28.985616+0000 mon.a (mon.0) 2146 : audit [DBG] from='client.? v1:172.21.15.121:0/2826634895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:29.719 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:29 smithi121 bash[18489]: audit 2024-02-12T16:27:29.352720+0000 mon.a (mon.0) 2147 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:27:29.720 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:29 smithi121 bash[18489]: audit 2024-02-12T16:27:29.665762+0000 mon.a (mon.0) 2148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:27:29.720 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:29 smithi121 bash[18489]: audit 2024-02-12T16:27:29.666669+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:27:29.720 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:29 smithi121 bash[18489]: audit 2024-02-12T16:27:29.671959+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:27:29.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:29.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:29.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:29.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:29.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:29 smithi123 bash[19119]: audit 2024-02-12T16:27:28.985616+0000 mon.a (mon.0) 2146 : audit [DBG] from='client.? v1:172.21.15.121:0/2826634895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:29.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:29 smithi123 bash[19119]: audit 2024-02-12T16:27:29.352720+0000 mon.a (mon.0) 2147 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:27:29.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:29 smithi123 bash[19119]: audit 2024-02-12T16:27:29.665762+0000 mon.a (mon.0) 2148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:27:29.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:29 smithi123 bash[19119]: audit 2024-02-12T16:27:29.666669+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:27:29.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:29 smithi123 bash[19119]: audit 2024-02-12T16:27:29.671959+0000 mon.a (mon.0) 2150 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:27:29.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:30.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:30.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:30.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:30.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:30.315 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:30.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:30.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1770,"stamp":"2024-02-12T16:27:27.982222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4879876120,"num_objects":74462,"num_object_clones":0,"num_object_copies":148924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8253,"num_objects_unfound":0,"num_objects_dirty":74462,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":264314,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":100728832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14240072,"kb_used_data":9687096,"kb_used_omap":91,"kb_used_meta":4552420,"kb_avail":1110701752,"statfs":{"total":1151940427776,"available":1137358594048,"internally_reserved":0,"allocated":9919586304,"data_stored":9915643092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":4661678096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,8,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-642646016,"num_objects":-9806,"num_object_clones":0,"num_object_copies":-19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1407,"num_objects_unfound":0,"num_objects_dirty":-9806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9806,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":15204352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001769"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":92,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.683440+0000","last_change":"2024-02-12T16:26:42.266638+0000","last_active":"2024-02-12T16:27:16.683440+0000","last_peered":"2024-02-12T16:27:16.683440+0000","last_clean":"2024-02-12T16:27:16.683440+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:27:16.683440+0000","last_undegraded":"2024-02-12T16:27:16.683440+0000","last_fullsized":"2024-02-12T16:27:16.683440+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:42.266460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:26:42.266460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"604'16296","reported_seq":16365,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:23.573823+0000","last_change":"2024-02-12T16:25:46.665174+0000","last_active":"2024-02-12T16:27:23.573823+0000","last_peered":"2024-02-12T16:27:23.573823+0000","last_clean":"2024-02-12T16:27:23.573823+0000","last_became_active":"2024-02-12T16:24:09.648523+0000","last_became_peered":"2024-02-12T16:24:09.648523+0000","last_unstale":"2024-02-12T16:27:23.573823+0000","last_undegraded":"2024-02-12T16:27:23.573823+0000","last_fullsized":"2024-02-12T16:27:23.573823+0000","mapping_epoch":547,"log_start":"563'6200","ondisk_log_start":"563'6200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10057","last_scrub_stamp":"2024-02-12T16:25:46.665055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:46.665055+0000","log_size":10096,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":309854208,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16296,"num_write_kb":672768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":195,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.693871+0000","last_change":"2024-02-12T16:27:16.104691+0000","last_active":"2024-02-12T16:27:16.693871+0000","last_peered":"2024-02-12T16:27:16.693871+0000","last_clean":"2024-02-12T16:27:16.693871+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:27:16.693871+0000","last_undegraded":"2024-02-12T16:27:16.693871+0000","last_fullsized":"2024-02-12T16:27:16.693871+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:40.282662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:40.282662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"604'16604","reported_seq":16678,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:24.256071+0000","last_change":"2024-02-12T16:26:52.958356+0000","last_active":"2024-02-12T16:27:24.256071+0000","last_peered":"2024-02-12T16:27:24.256071+0000","last_clean":"2024-02-12T16:27:24.256071+0000","last_became_active":"2024-02-12T16:24:10.018691+0000","last_became_peered":"2024-02-12T16:24:10.018691+0000","last_unstale":"2024-02-12T16:27:24.256071+0000","last_undegraded":"2024-02-12T16:27:24.256071+0000","last_fullsized":"2024-02-12T16:27:24.256071+0000","mapping_epoch":547,"log_start":"564'6600","ondisk_log_start":"564'6600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"598'14495","last_scrub_stamp":"2024-02-12T16:26:52.958251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:52.958251+0000","log_size":10004,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312082432,"num_objects":4762,"num_object_clones":0,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16604,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":286,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.698010+0000","last_change":"2024-02-12T16:24:47.872602+0000","last_active":"2024-02-12T16:27:16.698010+0000","last_peered":"2024-02-12T16:27:16.698010+0000","last_clean":"2024-02-12T16:27:16.698010+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:27:16.698010+0000","last_undegraded":"2024-02-12T16:27:16.698010+0000","last_fullsized":"2024-02-12T16:27:16.698010+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:47.872500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:24:47.872500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"604'16547","reported_seq":18686,"reported_epoch":604,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:27:26.645313+0000","last_change":"2024-02-12T16:25:58.008531+0000","last_active":"2024-02-12T16:27:26.645313+0000","last_peered":"2024-02-12T16:27:26.645313+0000","last_clean":"2024-02-12T16:25:54.194068+0000","last_became_active":"2024-02-12T16:25:57.932697+0000","last_became_peered":"2024-02-12T16:25:57.932697+0000","last_unstale":"2024-02-12T16:27:26.645313+0000","last_undegraded":"2024-02-12T16:27:26.645313+0000","last_fullsized":"2024-02-12T16:27:26.645313+0000","mapping_epoch":581,"log_start":"564'6500","ondisk_log_start":"564'6500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10027","last_scrub_stamp":"2024-02-12T16:25:45.765333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.765333+0000","log_size":10047,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":298123264,"num_objects":4549,"num_object_clones":0,"num_object_copies":9098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3891,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16547,"num_write_kb":675072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1029,"num_bytes_recovered":67436544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4549}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":691,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.678959+0000","last_change":"2024-02-12T16:25:07.263767+0000","last_active":"2024-02-12T16:27:16.678959+0000","last_peered":"2024-02-12T16:27:16.678959+0000","last_clean":"2024-02-12T16:27:16.678959+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:27:16.678959+0000","last_undegraded":"2024-02-12T16:27:16.678959+0000","last_fullsized":"2024-02-12T16:27:16.678959+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.263688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.263688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.3","version":"604'16722","reported_seq":16786,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:25.259511+0000","last_change":"2024-02-12T16:24:09.648426+0000","last_active":"2024-02-12T16:27:25.259511+0000","last_peered":"2024-02-12T16:27:25.259511+0000","last_clean":"2024-02-12T16:27:25.259511+0000","last_became_active":"2024-02-12T16:24:09.648139+0000","last_became_peered":"2024-02-12T16:24:09.648139+0000","last_unstale":"2024-02-12T16:27:25.259511+0000","last_undegraded":"2024-02-12T16:27:25.259511+0000","last_fullsized":"2024-02-12T16:27:25.259511+0000","mapping_epoch":547,"log_start":"564'6700","ondisk_log_start":"564'6700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10022,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":309329920,"num_objects":4720,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16722,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":307,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.695378+0000","last_change":"2024-02-12T16:27:16.201789+0000","last_active":"2024-02-12T16:27:16.695378+0000","last_peered":"2024-02-12T16:27:16.695378+0000","last_clean":"2024-02-12T16:27:16.695378+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:27:16.695378+0000","last_undegraded":"2024-02-12T16:27:16.695378+0000","last_fullsized":"2024-02-12T16:27:16.695378+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:45.967340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:45.967340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"604'16193","reported_seq":16262,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:25.173146+0000","last_change":"2024-02-12T16:25:50.727730+0000","last_active":"2024-02-12T16:27:25.173146+0000","last_peered":"2024-02-12T16:27:25.173146+0000","last_clean":"2024-02-12T16:27:25.173146+0000","last_became_active":"2024-02-12T16:24:09.649607+0000","last_became_peered":"2024-02-12T16:24:09.649607+0000","last_unstale":"2024-02-12T16:27:25.173146+0000","last_undegraded":"2024-02-12T16:27:25.173146+0000","last_fullsized":"2024-02-12T16:27:25.173146+0000","mapping_epoch":547,"log_start":"563'6100","ondisk_log_start":"563'6100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10162","last_scrub_stamp":"2024-02-12T16:25:50.727611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.727611+0000","log_size":10093,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":302972928,"num_objects":4623,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":666112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":277,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.698095+0000","last_change":"2024-02-12T16:25:07.944563+0000","last_active":"2024-02-12T16:27:16.698095+0000","last_peered":"2024-02-12T16:27:16.698095+0000","last_clean":"2024-02-12T16:27:16.698095+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:27:16.698095+0000","last_undegraded":"2024-02-12T16:27:16.698095+0000","last_fullsized":"2024-02-12T16:27:16.698095+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.944451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.944451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"604'16039","reported_seq":16113,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:25.232214+0000","last_change":"2024-02-12T16:26:10.447550+0000","last_active":"2024-02-12T16:27:25.232214+0000","last_peered":"2024-02-12T16:27:25.232214+0000","last_clean":"2024-02-12T16:27:25.232214+0000","last_became_active":"2024-02-12T16:24:09.647095+0000","last_became_peered":"2024-02-12T16:24:09.647095+0000","last_unstale":"2024-02-12T16:27:25.232214+0000","last_undegraded":"2024-02-12T16:27:25.232214+0000","last_fullsized":"2024-02-12T16:27:25.232214+0000","mapping_epoch":547,"log_start":"562'6000","ondisk_log_start":"562'6000","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10039,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":301531136,"num_objects":4601,"num_object_clones":0,"num_object_copies":9202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16039,"num_write_kb":660480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":279,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.698046+0000","last_change":"2024-02-12T16:25:37.899287+0000","last_active":"2024-02-12T16:27:16.698046+0000","last_peered":"2024-02-12T16:27:16.698046+0000","last_clean":"2024-02-12T16:27:16.698046+0000","last_became_active":"2024-02-12T16:17:37.450884+0000","last_became_peered":"2024-02-12T16:17:37.450884+0000","last_unstale":"2024-02-12T16:27:16.698046+0000","last_undegraded":"2024-02-12T16:27:16.698046+0000","last_fullsized":"2024-02-12T16:27:16.698046+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:37.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:25:37.899168+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"604'16718","reported_seq":16810,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:26.903533+0000","last_change":"2024-02-12T16:27:15.923522+0000","last_active":"2024-02-12T16:27:26.903533+0000","last_peered":"2024-02-12T16:27:26.903533+0000","last_clean":"2024-02-12T16:27:26.903533+0000","last_became_active":"2024-02-12T16:24:09.651253+0000","last_became_peered":"2024-02-12T16:24:09.651253+0000","last_unstale":"2024-02-12T16:27:26.903533+0000","last_undegraded":"2024-02-12T16:27:26.903533+0000","last_fullsized":"2024-02-12T16:27:26.903533+0000","mapping_epoch":547,"log_start":"564'6700","ondisk_log_start":"564'6700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10018,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":298582016,"num_objects":4556,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16718,"num_write_kb":680768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":146,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.676613+0000","last_change":"2024-02-12T16:25:54.579343+0000","last_active":"2024-02-12T16:27:16.676613+0000","last_peered":"2024-02-12T16:27:16.676613+0000","last_clean":"2024-02-12T16:27:16.676613+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:27:16.676613+0000","last_undegraded":"2024-02-12T16:27:16.676613+0000","last_fullsized":"2024-02-12T16:27:16.676613+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:54.579248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:54.579248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"604'16459","reported_seq":16533,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:26.299723+0000","last_change":"2024-02-12T16:27:12.609604+0000","last_active":"2024-02-12T16:27:26.299723+0000","last_peered":"2024-02-12T16:27:26.299723+0000","last_clean":"2024-02-12T16:27:26.299723+0000","last_became_active":"2024-02-12T16:24:09.647187+0000","last_became_peered":"2024-02-12T16:24:09.647187+0000","last_unstale":"2024-02-12T16:27:26.299723+0000","last_undegraded":"2024-02-12T16:27:26.299723+0000","last_fullsized":"2024-02-12T16:27:26.299723+0000","mapping_epoch":547,"log_start":"564'6400","ondisk_log_start":"564'6400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'15509","last_scrub_stamp":"2024-02-12T16:27:12.609496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:12.609496+0000","log_size":10059,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":301531136,"num_objects":4601,"num_object_clones":0,"num_object_copies":9202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16459,"num_write_kb":673920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":689,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.691735+0000","last_change":"2024-02-12T16:26:04.914413+0000","last_active":"2024-02-12T16:27:16.691735+0000","last_peered":"2024-02-12T16:27:16.691735+0000","last_clean":"2024-02-12T16:27:16.691735+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:27:16.691735+0000","last_undegraded":"2024-02-12T16:27:16.691735+0000","last_fullsized":"2024-02-12T16:27:16.691735+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:04.914304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:04.914304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"604'16602","reported_seq":16676,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:23.270936+0000","last_change":"2024-02-12T16:26:57.827400+0000","last_active":"2024-02-12T16:27:23.270936+0000","last_peered":"2024-02-12T16:27:23.270936+0000","last_clean":"2024-02-12T16:27:23.270936+0000","last_became_active":"2024-02-12T16:24:09.650120+0000","last_became_peered":"2024-02-12T16:24:09.650120+0000","last_unstale":"2024-02-12T16:27:23.270936+0000","last_undegraded":"2024-02-12T16:27:23.270936+0000","last_fullsized":"2024-02-12T16:27:23.270936+0000","mapping_epoch":547,"log_start":"564'6600","ondisk_log_start":"564'6600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'14868","last_scrub_stamp":"2024-02-12T16:26:57.827287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:57.827287+0000","log_size":10002,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":320995328,"num_objects":4898,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16602,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":92,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.691765+0000","last_change":"2024-02-12T16:26:43.025974+0000","last_active":"2024-02-12T16:27:16.691765+0000","last_peered":"2024-02-12T16:27:16.691765+0000","last_clean":"2024-02-12T16:27:16.691765+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:27:16.691765+0000","last_undegraded":"2024-02-12T16:27:16.691765+0000","last_fullsized":"2024-02-12T16:27:16.691765+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:43.025867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:43.025867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"604'16456","reported_seq":17553,"reported_epoch":604,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:27:25.293802+0000","last_change":"2024-02-12T16:26:41.328312+0000","last_active":"2024-02-12T16:27:25.293802+0000","last_peered":"2024-02-12T16:27:25.293802+0000","last_clean":"2024-02-12T16:26:09.048189+0000","last_became_active":"2024-02-12T16:26:11.222461+0000","last_became_peered":"2024-02-12T16:26:11.222461+0000","last_unstale":"2024-02-12T16:27:25.293802+0000","last_undegraded":"2024-02-12T16:27:25.293802+0000","last_fullsized":"2024-02-12T16:27:25.293802+0000","mapping_epoch":585,"log_start":"563'6400","ondisk_log_start":"563'6400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'9817","last_scrub_stamp":"2024-02-12T16:25:42.181950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:42.181950+0000","log_size":10056,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":311296000,"num_objects":4750,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4362,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16456,"num_write_kb":678592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":508,"num_bytes_recovered":33292288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[0,10],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,10","objects":4750}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":94,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.676564+0000","last_change":"2024-02-12T16:27:14.400780+0000","last_active":"2024-02-12T16:27:16.676564+0000","last_peered":"2024-02-12T16:27:16.676564+0000","last_clean":"2024-02-12T16:27:16.676564+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:27:16.676564+0000","last_undegraded":"2024-02-12T16:27:16.676564+0000","last_fullsized":"2024-02-12T16:27:16.676564+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:52.631321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.631321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"604'16941","reported_seq":17014,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:26.240526+0000","last_change":"2024-02-12T16:25:47.403836+0000","last_active":"2024-02-12T16:27:26.240526+0000","last_peered":"2024-02-12T16:27:26.240526+0000","last_clean":"2024-02-12T16:27:26.240526+0000","last_became_active":"2024-02-12T16:24:09.649590+0000","last_became_peered":"2024-02-12T16:24:09.649590+0000","last_unstale":"2024-02-12T16:27:26.240526+0000","last_undegraded":"2024-02-12T16:27:26.240526+0000","last_fullsized":"2024-02-12T16:27:26.240526+0000","mapping_epoch":547,"log_start":"564'6900","ondisk_log_start":"564'6900","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10313","last_scrub_stamp":"2024-02-12T16:25:47.403730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.403730+0000","log_size":10041,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":299565056,"num_objects":4571,"num_object_clones":0,"num_object_copies":9142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16941,"num_write_kb":688384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":290,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.695304+0000","last_change":"2024-02-12T16:25:45.967785+0000","last_active":"2024-02-12T16:27:16.695304+0000","last_peered":"2024-02-12T16:27:16.695304+0000","last_clean":"2024-02-12T16:27:16.695304+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:27:16.695304+0000","last_undegraded":"2024-02-12T16:27:16.695304+0000","last_fullsized":"2024-02-12T16:27:16.695304+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:45.967726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.967726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"604'16260","reported_seq":16334,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:26.336168+0000","last_change":"2024-02-12T16:27:09.549045+0000","last_active":"2024-02-12T16:27:26.336168+0000","last_peered":"2024-02-12T16:27:26.336168+0000","last_clean":"2024-02-12T16:27:26.336168+0000","last_became_active":"2024-02-12T16:24:10.017647+0000","last_became_peered":"2024-02-12T16:24:10.017647+0000","last_unstale":"2024-02-12T16:27:26.336168+0000","last_undegraded":"2024-02-12T16:27:26.336168+0000","last_fullsized":"2024-02-12T16:27:26.336168+0000","mapping_epoch":547,"log_start":"563'6200","ondisk_log_start":"563'6200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'15168","last_scrub_stamp":"2024-02-12T16:27:09.548944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:09.548944+0000","log_size":10060,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":292159488,"num_objects":4458,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16260,"num_write_kb":662976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":94,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.699723+0000","last_change":"2024-02-12T16:26:59.766851+0000","last_active":"2024-02-12T16:27:16.699723+0000","last_peered":"2024-02-12T16:27:16.699723+0000","last_clean":"2024-02-12T16:27:16.699723+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:27:16.699723+0000","last_undegraded":"2024-02-12T16:27:16.699723+0000","last_fullsized":"2024-02-12T16:27:16.699723+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:59.766744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:59.766744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"604'16548","reported_seq":16618,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:25.202869+0000","last_change":"2024-02-12T16:25:40.938617+0000","last_active":"2024-02-12T16:27:25.202869+0000","last_peered":"2024-02-12T16:27:25.202869+0000","last_clean":"2024-02-12T16:27:25.202869+0000","last_became_active":"2024-02-12T16:24:09.649655+0000","last_became_peered":"2024-02-12T16:24:09.649655+0000","last_unstale":"2024-02-12T16:27:25.202869+0000","last_undegraded":"2024-02-12T16:27:25.202869+0000","last_fullsized":"2024-02-12T16:27:25.202869+0000","mapping_epoch":547,"log_start":"564'6500","ondisk_log_start":"564'6500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9693","last_scrub_stamp":"2024-02-12T16:25:40.938531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:40.938531+0000","log_size":10048,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":309919768,"num_objects":4730,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16548,"num_write_kb":680833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":692,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.691716+0000","last_change":"2024-02-12T16:26:48.943869+0000","last_active":"2024-02-12T16:27:16.691716+0000","last_peered":"2024-02-12T16:27:16.691716+0000","last_clean":"2024-02-12T16:27:16.691716+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:27:16.691716+0000","last_undegraded":"2024-02-12T16:27:16.691716+0000","last_fullsized":"2024-02-12T16:27:16.691716+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:48.943785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:48.943785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"604'16760","reported_seq":16828,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:26.616662+0000","last_change":"2024-02-12T16:25:50.062621+0000","last_active":"2024-02-12T16:27:26.616662+0000","last_peered":"2024-02-12T16:27:26.616662+0000","last_clean":"2024-02-12T16:27:26.616662+0000","last_became_active":"2024-02-12T16:24:09.647617+0000","last_became_peered":"2024-02-12T16:24:09.647617+0000","last_unstale":"2024-02-12T16:27:26.616662+0000","last_undegraded":"2024-02-12T16:27:26.616662+0000","last_fullsized":"2024-02-12T16:27:26.616662+0000","mapping_epoch":547,"log_start":"564'6700","ondisk_log_start":"564'6700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10397","last_scrub_stamp":"2024-02-12T16:25:50.062524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.062524+0000","log_size":10060,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":307494912,"num_objects":4692,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16760,"num_write_kb":686464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":280,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.690346+0000","last_change":"2024-02-12T16:24:50.575354+0000","last_active":"2024-02-12T16:27:16.690346+0000","last_peered":"2024-02-12T16:27:16.690346+0000","last_clean":"2024-02-12T16:27:16.690346+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:27:16.690346+0000","last_undegraded":"2024-02-12T16:27:16.690346+0000","last_fullsized":"2024-02-12T16:27:16.690346+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.575287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.575287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"604'16544","reported_seq":16613,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:26.839737+0000","last_change":"2024-02-12T16:26:09.203006+0000","last_active":"2024-02-12T16:27:26.839737+0000","last_peered":"2024-02-12T16:27:26.839737+0000","last_clean":"2024-02-12T16:27:26.839737+0000","last_became_active":"2024-02-12T16:24:09.650861+0000","last_became_peered":"2024-02-12T16:24:09.650861+0000","last_unstale":"2024-02-12T16:27:26.839737+0000","last_undegraded":"2024-02-12T16:27:26.839737+0000","last_fullsized":"2024-02-12T16:27:26.839737+0000","mapping_epoch":547,"log_start":"564'6500","ondisk_log_start":"564'6500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'11415","last_scrub_stamp":"2024-02-12T16:26:09.202913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:09.202913+0000","log_size":10044,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":308674560,"num_objects":4710,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16544,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":33,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.678991+0000","last_change":"2024-02-12T16:26:10.388710+0000","last_active":"2024-02-12T16:27:16.678991+0000","last_peered":"2024-02-12T16:27:16.678991+0000","last_clean":"2024-02-12T16:27:16.678991+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:27:16.678991+0000","last_undegraded":"2024-02-12T16:27:16.678991+0000","last_fullsized":"2024-02-12T16:27:16.678991+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:10.388515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:10.388515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"604'16625","reported_seq":16694,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-02-12T16:27:26.240559+0000","last_change":"2024-02-12T16:25:49.810317+0000","last_active":"2024-02-12T16:27:26.240559+0000","last_peered":"2024-02-12T16:27:26.240559+0000","last_clean":"2024-02-12T16:27:26.240559+0000","last_became_active":"2024-02-12T16:24:10.019266+0000","last_became_peered":"2024-02-12T16:24:10.019266+0000","last_unstale":"2024-02-12T16:27:26.240559+0000","last_undegraded":"2024-02-12T16:27:26.240559+0000","last_fullsized":"2024-02-12T16:27:26.240559+0000","mapping_epoch":547,"log_start":"564'6600","ondisk_log_start":"564'6600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10246","last_scrub_stamp":"2024-02-12T16:25:49.810203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:49.810203+0000","log_size":10025,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":295763968,"num_objects":4513,"num_object_clones":0,"num_object_copies":9026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16625,"num_write_kb":676416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":268,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.693976+0000","last_change":"2024-02-12T16:25:03.361871+0000","last_active":"2024-02-12T16:27:16.693976+0000","last_peered":"2024-02-12T16:27:16.693976+0000","last_clean":"2024-02-12T16:27:16.693976+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:27:16.693976+0000","last_undegraded":"2024-02-12T16:27:16.693976+0000","last_fullsized":"2024-02-12T16:27:16.693976+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:25:03.361757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":89,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.690346+0000","last_change":"2024-02-12T16:25:59.785619+0000","last_active":"2024-02-12T16:27:16.690346+0000","last_peered":"2024-02-12T16:27:16.690346+0000","last_clean":"2024-02-12T16:27:16.690346+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:27:16.690346+0000","last_undegraded":"2024-02-12T16:27:16.690346+0000","last_fullsized":"2024-02-12T16:27:16.690346+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:59.785496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:25:59.785496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":87,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.698335+0000","last_change":"2024-02-12T16:24:54.540925+0000","last_active":"2024-02-12T16:27:16.698335+0000","last_peered":"2024-02-12T16:27:16.698335+0000","last_clean":"2024-02-12T16:27:16.698335+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:27:16.698335+0000","last_undegraded":"2024-02-12T16:27:16.698335+0000","last_fullsized":"2024-02-12T16:27:16.698335+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:54.540855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:54.540855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":103,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.690353+0000","last_change":"2024-02-12T16:25:47.933452+0000","last_active":"2024-02-12T16:27:16.690353+0000","last_peered":"2024-02-12T16:27:16.690353+0000","last_clean":"2024-02-12T16:27:16.690353+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:27:16.690353+0000","last_undegraded":"2024-02-12T16:27:16.690353+0000","last_fullsized":"2024-02-12T16:27:16.690353+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:47.933341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.933341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":82,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-12T16:27:16.690973+0000","last_change":"2024-02-12T16:25:50.908791+0000","last_active":"2024-02-12T16:27:16.690973+0000","last_peered":"2024-02-12T16:27:16.690973+0000","last_clean":"2024-02-12T16:27:16.690973+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:27:16.690973+0000","last_undegraded":"2024-02-12T16:27:16.690973+0000","last_fullsized":"2024-02-12T16:27:16.690973+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.908692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.908692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4879876120,"num_objects":74462,"num_object_clones":0,"num_object_copies":148924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8253,"num_objects_unfound":0,"num_objects_dirty":74462,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":264314,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":100728832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9853476864,"data_stored":9853468720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958879,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":921552,"kb_used_data":331848,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":92823600,"statfs":{"total":95995035648,"available":95051366400,"internally_reserved":0,"allocated":339812352,"data_stored":339480351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.85499999999999998},"min":{"1min":0.25800000000000001,"5min":0.246,"15min":0.246},"max":{"1min":3.6560000000000001,"5min":4.2389999999999999,"15min":5.6870000000000003},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.89500000000000002,"15min":0.90100000000000002},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.5819999999999999,"5min":4.5819999999999999,"15min":4.992},"last":0.70399999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.93400000000000005,"15min":1.006},"min":{"1min":0.316,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":1.984,"5min":4.6609999999999996,"15min":7.0110000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.90600000000000003,"15min":1.075},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28799999999999998},"max":{"1min":3.444,"5min":4.2039999999999997,"15min":5.6920000000000002},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:27:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.96299999999999997,"15min":1.2869999999999999},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3420000000000001,"5min":5.343,"15min":5.343},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.075,"5min":1.129,"15min":1.4039999999999999},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.4340000000000002,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":2.617}]},{"osd":4,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.94099999999999995,"15min":2.2679999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.29499999999999998},"max":{"1min":3.6019999999999999,"5min":4.4400000000000004,"15min":368.48399999999998},"last":0.437},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.88600000000000001,"15min":2.1850000000000001},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.6110000000000002,"5min":4.915,"15min":371.65100000000001},"last":0.63200000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1870000000000001,"15min":3.8380000000000001},"min":{"1min":0.39400000000000002,"5min":0.35299999999999998,"15min":0.35099999999999998},"max":{"1min":4.8339999999999996,"5min":5.04,"15min":407.07600000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.159,"15min":3.7999999999999998},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":407.26299999999998},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.97799999999999998,"15min":0.996},"min":{"1min":0.433,"5min":0.38500000000000001,"15min":0.373},"max":{"1min":4.7119999999999997,"5min":5.141,"15min":5.141},"last":4.7370000000000001},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.1579999999999999,"15min":1.1319999999999999},"min":{"1min":0.40400000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.7450000000000001,"5min":8.5709999999999997,"15min":8.5709999999999997},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.1890000000000001,"15min":1.242},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.524,"5min":5.2569999999999997,"15min":6.4370000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.0289999999999999,"15min":1.1779999999999999},"min":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.131,"5min":5.0430000000000001,"15min":5.1189999999999998},"last":0.52800000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":27.638000000000002,"5min":6.3220000000000001,"15min":2.835},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":537.54399999999998,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":27.850000000000001,"5min":6.5190000000000001,"15min":3.0209999999999999},"min":{"1min":0.496,"5min":0.40500000000000003,"15min":0.34699999999999998},"max":{"1min":539.86099999999999,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.51100000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.171,"15min":1.262},"min":{"1min":0.46400000000000002,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":1.504,"5min":5.7430000000000003,"15min":10.743},"last":1.9139999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.21,"15min":1.4079999999999999},"min":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.39500000000000002},"max":{"1min":1.5660000000000001,"5min":5.359,"15min":10.698},"last":1.861}]},{"osd":10,"last update":"Mon Feb 12 16:26:32 2024","interfaces":[{"interface":"back","average":{"1min":27.643000000000001,"5min":6.4820000000000002,"15min":2.9700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":536.245,"5min":536.245,"15min":536.245},"last":0.76800000000000002},{"interface":"front","average":{"1min":27.702999999999999,"5min":6.4950000000000001,"15min":2.984},"min":{"1min":0.55300000000000005,"5min":0.443,"15min":0.443},"max":{"1min":536.64300000000003,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.70799999999999996}]}]},{"osd":10,"up_from":83,"seq":356482285928,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960140,"kb_used_data":658820,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92785012,"statfs":{"total":95995035648,"available":95011852288,"internally_reserved":0,"allocated":674631680,"data_stored":674303751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":2.5899999999999999,"15min":1.518},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.245},"max":{"1min":1.97,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":2.6419999999999999,"15min":1.5960000000000001},"min":{"1min":0.30499999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.4829999999999997,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.82399999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90600000000000003,"15min":0.94099999999999995},"min":{"1min":0.40000000000000002,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":2.5179999999999998,"5min":4.4059999999999997,"15min":19.866},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.86599999999999999,"15min":1.0680000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":20.277999999999999},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95499999999999996,"15min":0.878},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.3200000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.123,"15min":0.95899999999999996},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":5.3680000000000003,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":0.64100000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.879,"15min":3.3210000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.32300000000000001},"max":{"1min":1.139,"5min":3.2639999999999998,"15min":628.77099999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.1240000000000001,"5min":0.96199999999999997,"15min":3.29},"min":{"1min":0.38800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":624.19299999999998},"last":0.77800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.03,"15min":1.024},"min":{"1min":0.47399999999999998,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":5.0910000000000002,"5min":5.0910000000000002,"15min":5.5739999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.085,"15min":1.109},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":1.26,"5min":3.2949999999999999,"15min":4.8769999999999998},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.004,"15min":3.3540000000000001},"min":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.1339999999999999,"5min":4.2640000000000002,"15min":626.26599999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.008,"15min":3.4390000000000001},"min":{"1min":0.438,"5min":0.376,"15min":0.376},"max":{"1min":4.1829999999999998,"5min":4.1829999999999998,"15min":625.90700000000004},"last":0.66800000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.95799999999999996,"15min":1.05},"min":{"1min":0.46100000000000002,"5min":0.40400000000000003,"15min":0.39800000000000002},"max":{"1min":4.1369999999999996,"5min":4.1369999999999996,"15min":5.9210000000000003},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.119,"15min":1.155},"min":{"1min":0.60299999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.1629999999999998,"5min":4.3150000000000004,"15min":5.3300000000000001},"last":0.61399999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.0649999999999999,"15min":1.101},"min":{"1min":0.51400000000000001,"5min":0.436,"15min":0.372},"max":{"1min":3.2210000000000001,"5min":3.2210000000000001,"15min":4.827},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.052,"15min":1.1120000000000001},"min":{"1min":0.46400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.2359999999999998,"5min":4.2359999999999998,"15min":4.8330000000000002},"last":0.71199999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.119,"15min":1.2010000000000001},"min":{"1min":0.57399999999999995,"5min":0.47599999999999998,"15min":0.372},"max":{"1min":2.105,"5min":4.782,"15min":4.9829999999999997},"last":0.874},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.101,"15min":1.2010000000000001},"min":{"1min":0.54700000000000004,"5min":0.47299999999999998,"15min":0.41399999999999998},"max":{"1min":4.1699999999999999,"5min":4.1699999999999999,"15min":4.952},"last":1.121}]},{"osd":11,"last update":"Mon Feb 12 16:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.1319999999999999,"15min":1.3999999999999999},"min":{"1min":0.55100000000000005,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":1.4550000000000001,"5min":2.0249999999999999,"15min":30.917999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.878,"5min":1.157,"15min":1.4119999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.617,"5min":2.2400000000000002,"15min":30.742000000000001},"last":0.73099999999999998}]}]},{"osd":9,"up_from":74,"seq":317827580273,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888396,"kb_used_data":582980,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92856756,"statfs":{"total":95995035648,"available":95085318144,"internally_reserved":0,"allocated":596971520,"data_stored":596643599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.006,"15min":1.042},"min":{"1min":0.30199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.4450000000000003,"5min":5.8330000000000002,"15min":5.8330000000000002},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":0.93600000000000005,"15min":0.90500000000000003},"min":{"1min":0.35999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.4619999999999997,"5min":5.4619999999999997,"15min":5.4619999999999997},"last":0.379}]},{"osd":1,"last update":"Mon Feb 12 16:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.0680000000000001,"15min":0.94899999999999995},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.253},"max":{"1min":3.9620000000000002,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.1879999999999999,"5min":0.96299999999999997,"15min":1.046},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":3.976,"5min":5.4560000000000004,"15min":7.9550000000000001},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0720000000000001,"15min":1.409},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.27600000000000002},"max":{"1min":3.7709999999999999,"5min":5.915,"15min":97.275000000000006},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.0549999999999999,"15min":1.258},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.312},"max":{"1min":4.0780000000000003,"5min":6.1779999999999999,"15min":97.417000000000002},"last":0.755}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.044,"15min":0.95299999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":4.173,"5min":4.569,"15min":4.569},"last":0.496},{"interface":"front","average":{"1min":1.278,"5min":1.1919999999999999,"15min":1.123},"min":{"1min":0.38500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.2510000000000003,"5min":4.8840000000000003,"15min":5.8010000000000002},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.3140000000000001,"15min":1.107},"min":{"1min":0.41999999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.726,"5min":6.726,"15min":6.726},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.1819999999999999,"15min":1.1519999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":6.7350000000000003,"5min":6.7350000000000003,"15min":7.3739999999999997},"last":0.46400000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.0189999999999999,"15min":1.089},"min":{"1min":0.39400000000000002,"5min":0.379,"15min":0.379},"max":{"1min":2.9750000000000001,"5min":5.5250000000000004,"15min":7.3360000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.1659999999999999,"15min":1.04},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.21,"5min":7.0609999999999999,"15min":7.0609999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.3280000000000001,"15min":1.2270000000000001},"min":{"1min":0.50800000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.8420000000000001,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.212,"15min":1.2130000000000001},"min":{"1min":0.47399999999999998,"5min":0.44600000000000001,"15min":0.438},"max":{"1min":3.6629999999999998,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.58799999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0880000000000001,"15min":1.0680000000000001},"min":{"1min":0.437,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.9460000000000002,"5min":5.3620000000000001,"15min":5.4059999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.1160000000000001,"15min":1.1950000000000001},"min":{"1min":0.48699999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":4.3310000000000004,"5min":4.3310000000000004,"15min":7.25},"last":0.71799999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.304,"15min":1.2310000000000001},"min":{"1min":0.53400000000000003,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":3.8029999999999999,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.101,"5min":1.1890000000000001,"15min":1.1719999999999999},"min":{"1min":0.53900000000000003,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":4.423,"5min":4.4509999999999996,"15min":5.8700000000000001},"last":0.746}]},{"osd":11,"last update":"Mon Feb 12 16:26:44 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.2949999999999999,"15min":1.349},"min":{"1min":0.55000000000000004,"5min":0.48999999999999999,"15min":0.45600000000000002},"max":{"1min":3.7370000000000001,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.1950000000000001,"15min":1.3100000000000001},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.0169999999999999,"5min":5.492,"15min":6.3639999999999999},"last":0.68999999999999995}]}]},{"osd":8,"up_from":65,"seq":279172874618,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911044,"kb_used_data":601412,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92834108,"statfs":{"total":95995035648,"available":95062126592,"internally_reserved":0,"allocated":615845888,"data_stored":615517967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.006,"15min":0.88600000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":4.7270000000000003,"5min":19.196999999999999,"15min":19.196999999999999},"last":2.4279999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.093,"15min":0.88900000000000001},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29399999999999998},"max":{"1min":3.331,"5min":18.584,"15min":18.584},"last":0.34699999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.7669999999999999,"15min":1.4530000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.4889999999999999,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.7529999999999999,"15min":1.53},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.1260000000000003,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.45000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4160000000000004,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":0.93999999999999995,"15min":0.998},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":0.498}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.82899999999999996,"15min":1.7150000000000001},"min":{"1min":0.33700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.7250000000000001,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.44},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.875,"15min":1.6519999999999999},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.95299999999999996,"5min":5.0380000000000003,"15min":224.71000000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.85199999999999998,"15min":0.90200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":5.2480000000000002,"5min":5.2480000000000002,"15min":5.2480000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.95199999999999996,"15min":1.0229999999999999},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.5600000000000001,"5min":3.5600000000000001,"15min":4.8339999999999996},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.155,"15min":1.7410000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.37},"max":{"1min":5.3639999999999999,"5min":5.3639999999999999,"15min":221.36699999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.099,"15min":1.8600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.2869999999999999,"5min":5.556,"15min":221.203},"last":0.68500000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.1319999999999999,"15min":1.1459999999999999},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.42199999999999999},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.1890000000000001,"15min":1.23},"min":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.41599999999999998},"max":{"1min":4.2240000000000002,"5min":4.2240000000000002,"15min":7.899},"last":0.64500000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:27:05 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.081,"15min":1.1679999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35599999999999998},"max":{"1min":3.8610000000000002,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1850000000000001,"15min":1.226},"min":{"1min":0.53600000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":3.8940000000000001,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":0.71499999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.01,"15min":1.3979999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.7530000000000001,"5min":2.7530000000000001,"15min":109.92},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.115,"15min":1.4650000000000001},"min":{"1min":0.48299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.0250000000000004,"5min":5.0250000000000004,"15min":110.249},"last":0.51300000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.0660000000000001,"15min":1.1899999999999999},"min":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":2.7869999999999999,"5min":2.9159999999999999,"15min":5.0670000000000002},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.97199999999999998,"15min":1.1399999999999999},"min":{"1min":0.56000000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1859999999999999,"5min":2.1259999999999999,"15min":4.6639999999999997},"last":0.76700000000000002}]}]},{"osd":7,"up_from":56,"seq":240518168961,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626116,"kb_used_data":316484,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93119036,"statfs":{"total":95995035648,"available":95353892864,"internally_reserved":0,"allocated":324079616,"data_stored":323751695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.88400000000000001,"15min":0.90000000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":3.1890000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.93500000000000005,"15min":0.93000000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.75,"5min":7.7699999999999996,"15min":7.7699999999999996},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.2769999999999999,"15min":1.008},"min":{"1min":0.313,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":4.5179999999999998,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.6339999999999999,"15min":1.1859999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.6719999999999997,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:27:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.034,"15min":1.2430000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.1669999999999998,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.88200000000000001,"15min":0.97999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.4340000000000002,"5min":4.4699999999999998,"15min":4.4699999999999998},"last":0.379}]},{"osd":4,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.0600000000000001,"15min":0.93799999999999994},"min":{"1min":0.34100000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.4420000000000002,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.90300000000000002,"15min":1.0089999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":4.7389999999999999,"5min":4.7389999999999999,"15min":6.0090000000000003},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.98199999999999998,"15min":0.96499999999999997},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.5620000000000003,"5min":4.7220000000000004,"15min":4.9050000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.089,"5min":1.024,"15min":1.036},"min":{"1min":0.38300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":5.766,"5min":7.5640000000000001,"15min":9.3640000000000008},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.125,"15min":1.101},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":6.319,"5min":6.319,"15min":6.319},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.141,"15min":1.1799999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.33600000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":6.5179999999999998,"15min":2.9369999999999998},"min":{"1min":0.46300000000000002,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.855,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":6.5220000000000002,"15min":2.9830000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.9119999999999999,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.68899999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:26:57 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.03,"15min":1.1890000000000001},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.6539999999999999,"5min":4.6539999999999999,"15min":12.907999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.133,"15min":1.2390000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.71,"5min":4.665,"15min":4.6719999999999997},"last":0.53800000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:26:29 2024","interfaces":[{"interface":"back","average":{"1min":34.110999999999997,"5min":7.742,"15min":3.4820000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":497.52699999999999,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":33.834000000000003,"5min":7.7649999999999997,"15min":3.3940000000000001},"min":{"1min":0.45300000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":496.29599999999999,"5min":496.29599999999999,"15min":496.29599999999999},"last":0.72799999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1160000000000001,"15min":1.617},"min":{"1min":0.495,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.27,"5min":6.0490000000000004,"15min":120.943},"last":0.63},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.21,"15min":1.7150000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":4.5,"5min":4.7110000000000003,"15min":120.661},"last":0.57899999999999996}]}]},{"osd":6,"up_from":48,"seq":206158430602,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212748,"kb_used_data":903236,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92532404,"statfs":{"total":95995035648,"available":94753181696,"internally_reserved":0,"allocated":924913664,"data_stored":924585743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.111,"15min":1.0640000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.464,"5min":30.454999999999998,"15min":30.454999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.246,"15min":1.1519999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.9769999999999999,"5min":30.573,"15min":30.573},"last":0.435}]},{"osd":1,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.028,"15min":1.921},"min":{"1min":0.29599999999999999,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":5.5179999999999998,"5min":5.5179999999999998,"15min":252.22800000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.298,"5min":0.98099999999999998,"15min":1.9259999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.6959999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.3520000000000001,"15min":1.004},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":5.9500000000000002,"5min":28.739999999999998,"15min":28.739999999999998},"last":2.6179999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.5820000000000001,"15min":1.1200000000000001},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.7939999999999996,"5min":32.328000000000003,"15min":32.328000000000003},"last":2.7229999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.2230000000000001,"15min":1.968},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.2260000000000009,"5min":8.2260000000000009,"15min":284.05599999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.268,"5min":1.1020000000000001,"15min":2.0390000000000001},"min":{"1min":0.28799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.0259999999999998,"5min":7.5979999999999999,"15min":284.041},"last":0.34999999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.0409999999999999,"15min":1.9139999999999999},"min":{"1min":0.34100000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.4249999999999998,"5min":5.4249999999999998,"15min":281.49700000000001},"last":0.371},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.0309999999999999,"15min":1.9890000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.52,"5min":10.52,"15min":281.61900000000003},"last":0.496}]},{"osd":7,"last update":"Mon Feb 12 16:26:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.097,"15min":1.0780000000000001},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.79,"5min":7.641,"15min":7.641},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.1200000000000001,"15min":1.071},"min":{"1min":0.34899999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.8210000000000002,"5min":7.8380000000000001,"15min":7.8380000000000001},"last":0.52200000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.0800000000000001,"15min":1.1080000000000001},"min":{"1min":0.42099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":5.6310000000000002,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.0169999999999999,"15min":1.1890000000000001},"min":{"1min":0.39600000000000002,"5min":0.371,"15min":0.371},"max":{"1min":2.6440000000000001,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":0.65200000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.121,"15min":1.2090000000000001},"min":{"1min":0.42099999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.044,"5min":4.5330000000000004,"15min":18.059999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.115,"15min":1.27},"min":{"1min":0.47499999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.673,"5min":4.6890000000000001,"15min":17.425000000000001},"last":1.24}]},{"osd":10,"last update":"Mon Feb 12 16:26:45 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.202,"15min":1.224},"min":{"1min":0.505,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":6.5300000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.276,"15min":1.3080000000000001},"min":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":6.5730000000000004},"last":0.54200000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:26:45 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.242,"15min":3.2519999999999998},"min":{"1min":0.53700000000000003,"5min":0.499,"15min":0.499},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":139.727},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.151,"15min":3.125},"min":{"1min":0.51500000000000001,"5min":0.504,"15min":0.504},"max":{"1min":2.706,"5min":4.5899999999999999,"15min":139.24199999999999},"last":0.70199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2068812,"kb_used_data":1178564,"kb_used_omap":0,"kb_used_meta":890112,"kb_avail":91676340,"statfs":{"total":95995035648,"available":93876572160,"internally_reserved":0,"allocated":1206849536,"data_stored":1206521615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911474688},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.90600000000000003,"15min":1.329},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":7.8710000000000004,"5min":7.8710000000000004,"15min":114.40000000000001},"last":0.623},{"interface":"front","average":{"1min":1.589,"5min":1.2969999999999999,"15min":1.5780000000000001},"min":{"1min":0.30299999999999999,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":5.2690000000000001,"5min":7.7249999999999996,"15min":115.09099999999999},"last":1}]},{"osd":2,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.0049999999999999,"15min":0.98599999999999999},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":4.2720000000000002,"5min":6.5389999999999997,"15min":9.8599999999999994},"last":1.028},{"interface":"front","average":{"1min":1.74,"5min":1.198,"15min":1.119},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":9.6120000000000001,"5min":9.6120000000000001,"15min":9.6120000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.147,"15min":1.139},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":9.9529999999999994,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.0860000000000001,"15min":0.92000000000000004},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.5490000000000004,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.92700000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.2829999999999999,"15min":1.125},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.6749999999999998,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.4650000000000001,"15min":1.405},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.633,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":1.351,"15min":1.1819999999999999},"min":{"1min":0.433,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":10.109999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.2130000000000001,"15min":1.1899999999999999},"min":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.35899999999999999},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":7.0110000000000001},"last":0.71999999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.216,"15min":1.6379999999999999},"min":{"1min":0.439,"5min":0.38500000000000001,"15min":0.34699999999999998},"max":{"1min":2.4020000000000001,"5min":6.6310000000000002,"15min":55.869999999999997},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2110000000000001,"15min":1.6970000000000001},"min":{"1min":0.48399999999999999,"5min":0.46600000000000003,"15min":0.36799999999999999},"max":{"1min":2.5219999999999998,"5min":4.4470000000000001,"15min":55.892000000000003},"last":0.76800000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:39 2024","interfaces":[{"interface":"back","average":{"1min":31.306999999999999,"5min":7.3899999999999997,"15min":3.371},"min":{"1min":0.53700000000000003,"5min":0.51000000000000001,"15min":0.436},"max":{"1min":548.91600000000005,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.877},{"interface":"front","average":{"1min":32.816000000000003,"5min":7.6150000000000002,"15min":3.4550000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":552.42100000000005,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.79000000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:26:52 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.4590000000000001,"15min":1.3919999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":4.4720000000000004,"5min":5.7240000000000002,"15min":10.07},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.214,"5min":1.3360000000000001,"15min":1.492},"min":{"1min":0.60399999999999998,"5min":0.46700000000000003,"15min":0.45000000000000001},"max":{"1min":6.1559999999999997,"5min":6.1559999999999997,"15min":7.6059999999999999},"last":0.83099999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":36.500999999999998,"5min":8.5050000000000008,"15min":3.7400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":548.93499999999995,"5min":548.93499999999995,"15min":548.93499999999995},"last":1.0960000000000001},{"interface":"front","average":{"1min":35.311,"5min":8.3140000000000001,"15min":3.774},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.53700000000000003},"max":{"1min":549.12099999999998,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.79800000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:26:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.421,"15min":1.534},"min":{"1min":0.60199999999999998,"5min":0.58099999999999996,"15min":0.443},"max":{"1min":3.996,"5min":4.5609999999999999,"15min":7.1600000000000001},"last":4.5510000000000002},{"interface":"front","average":{"1min":1.659,"5min":1.609,"15min":1.6180000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.47599999999999998},"max":{"1min":5.2949999999999999,"5min":5.7549999999999999,"15min":7.2489999999999997},"last":0.81999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738811,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1809676,"kb_used_data":1512452,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":91935476,"statfs":{"total":95995035648,"available":94141927424,"internally_reserved":0,"allocated":1548750848,"data_stored":1548422927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":1.0620000000000001},"min":{"1min":0.254,"5min":0.245,"15min":0.215},"max":{"1min":4.4260000000000002,"5min":10.754,"15min":11.324},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.0129999999999999,"15min":1.0580000000000001},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.6859999999999999,"5min":4.6859999999999999,"15min":9.8249999999999993},"last":4.7699999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:26:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.98099999999999998,"15min":1.03},"min":{"1min":0.308,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.0469999999999999,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.874,"15min":0.90700000000000003},"min":{"1min":0.27600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.1970000000000001,"5min":3.637,"15min":3.637},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.167,"15min":1.7130000000000001},"min":{"1min":0.38900000000000001,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":4.8600000000000003,"5min":7.6219999999999999,"15min":116.187},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.87,"5min":1.3080000000000001,"15min":1.829},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":3.1030000000000002,"5min":7.5599999999999996,"15min":116.163},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.1020000000000001,"15min":1.016},"min":{"1min":0.33600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.2690000000000001,"5min":5.2690000000000001,"15min":5.2690000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1659999999999999,"15min":1.052},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.72799999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:27:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.2989999999999999,"15min":1.369},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":81.147000000000006},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.339,"15min":1.425},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.0640000000000001,"5min":8.8789999999999996,"15min":80.233000000000004},"last":4.7919999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.2170000000000001,"15min":1.5880000000000001},"min":{"1min":0.40899999999999997,"5min":0.374,"15min":0.36699999999999999},"max":{"1min":4.4219999999999997,"5min":4.4219999999999997,"15min":115.26000000000001},"last":2.4129999999999998},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.3080000000000001,"15min":1.6919999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.4889999999999999,"5min":4.4889999999999999,"15min":115.402},"last":2.4430000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.276,"15min":1.2689999999999999},"min":{"1min":0.48399999999999999,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":5.1070000000000002,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.552,"5min":1.369,"15min":1.357},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":5.1749999999999998,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":0.83499999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.1160000000000001,"15min":1.3440000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.3920000000000003,"5min":7.6539999999999999,"15min":46.246000000000002},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3779999999999999,"15min":1.395},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.2080000000000002,"5min":5.2080000000000002,"15min":46.167999999999999},"last":0.80400000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2589999999999999,"15min":1.276},"min":{"1min":0.54400000000000004,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":4.944,"5min":4.944,"15min":6.0979999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.339,"15min":1.3160000000000001},"min":{"1min":0.57399999999999995,"5min":0.52000000000000002,"15min":0.47999999999999998},"max":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":6.2480000000000002},"last":0.95099999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":1.452,"15min":1.575},"min":{"1min":0.497,"5min":0.46700000000000003,"15min":0.434},"max":{"1min":5.1050000000000004,"5min":5.1050000000000004,"15min":46.064},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.355,"15min":1.5049999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.38300000000000001},"max":{"1min":5.1390000000000002,"5min":5.1390000000000002,"15min":45.642000000000003},"last":0.86599999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313641,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1177868,"kb_used_data":876548,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92567284,"statfs":{"total":95995035648,"available":94788898816,"internally_reserved":0,"allocated":897585152,"data_stored":897257231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.034,"15min":1.0509999999999999},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":8.5399999999999991},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.8220000000000001,"5min":1.2470000000000001,"15min":1.1579999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":6.484,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":1.113}]},{"osd":3,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6899999999999999,"15min":1.98},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":6.4509999999999996,"5min":21.407,"15min":21.407},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.698,"5min":1.04,"15min":0.90000000000000002},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.40500000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0780000000000001,"15min":0.95599999999999996},"min":{"1min":0.34000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":3.9079999999999999,"5min":6.8090000000000002,"15min":7.2859999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.2989999999999999,"15min":1.093},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.601,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.226,"15min":1.095},"min":{"1min":0.39400000000000002,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":5.9489999999999998,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.175,"15min":1.0509999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":6.5730000000000004},"last":0.58299999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:26:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5309999999999999,"15min":1.2010000000000001},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.5279999999999996,"5min":21.527000000000001,"15min":21.527000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.2969999999999999,"15min":1.137},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":6.2850000000000001,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1499999999999999,"15min":1.0649999999999999},"min":{"1min":0.41899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.9489999999999998,"5min":4.8570000000000002,"15min":5.2960000000000003},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.571,"5min":1.2569999999999999,"15min":1.1870000000000001},"min":{"1min":0.47999999999999998,"5min":0.443,"15min":0.38700000000000001},"max":{"1min":5.5099999999999998,"5min":5.5099999999999998,"15min":5.5099999999999998},"last":0.51200000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.883,"5min":1.3320000000000001,"15min":1.2250000000000001},"min":{"1min":0.499,"5min":0.442,"15min":0.38700000000000001},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.71,"5min":1.429,"15min":1.2729999999999999},"min":{"1min":0.42499999999999999,"5min":0.38,"15min":0.38},"max":{"1min":6.5659999999999998,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":0.77900000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.25,"15min":1.181},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.40600000000000003},"max":{"1min":4.0119999999999996,"5min":4.8040000000000003,"15min":5.2329999999999997},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3380000000000001,"15min":1.252},"min":{"1min":0.50600000000000001,"5min":0.45400000000000001,"15min":0.42399999999999999},"max":{"1min":3.9809999999999999,"5min":4.9089999999999998,"15min":5.085},"last":0.89700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.4339999999999999,"15min":1.3029999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6609999999999996,"5min":11.454000000000001,"15min":11.454000000000001},"last":0.751},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.484,"15min":1.3220000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":6.6340000000000003,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":0.72499999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:26:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.3300000000000001,"15min":1.49},"min":{"1min":0.53300000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":5.9050000000000002,"5min":5.9050000000000002,"15min":5.9050000000000002},"last":0.628},{"interface":"front","average":{"1min":1.7270000000000001,"5min":1.4530000000000001,"15min":1.571},"min":{"1min":0.59699999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":5.9359999999999999,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":0.59499999999999997}]}]},{"osd":3,"up_from":522,"seq":2241972928566,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":649420,"kb_used_data":304900,"kb_used_omap":91,"kb_used_meta":344420,"kb_avail":93095732,"statfs":{"total":95995035648,"available":95330029568,"internally_reserved":0,"allocated":312217600,"data_stored":311889679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":352686096},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":1.2490000000000001,"15min":0.98099999999999998},"min":{"1min":0.32500000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":34.055999999999997,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":3.4910000000000001,"5min":1.361,"15min":1.022},"min":{"1min":0.33800000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":27.600000000000001,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.1200000000000001,"15min":0.95199999999999996},"min":{"1min":0.317,"5min":0.314,"15min":0.314},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":2.2069999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.1970000000000001,"15min":1.1020000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.746,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":2.617}]},{"osd":4,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.0589999999999999,"15min":1.304},"min":{"1min":0.38400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.7080000000000002,"5min":10.004,"15min":10.004},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.085,"15min":1.1160000000000001},"min":{"1min":0.40000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.4560000000000004,"5min":4.4560000000000004,"15min":4.4560000000000004},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1870000000000001,"15min":1.3600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":3.7530000000000001,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.93999999999999995,"15min":0.94599999999999995},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.60299999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.1699999999999999,"15min":1.1619999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":8.3320000000000007,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.501},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.0449999999999999,"15min":1.0269999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.1920000000000002,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.52200000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.258,"15min":1.458},"min":{"1min":0.48499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.8090000000000002,"5min":6.851,"15min":6.851},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.319,"15min":1.3109999999999999},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.8890000000000002,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":0.44400000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.137,"15min":1.204},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.1459999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.276,"5min":1.0429999999999999,"15min":1.1140000000000001},"min":{"1min":0.47099999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.64400000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.3640000000000001,"15min":1.3560000000000001},"min":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.5810000000000004,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.823,"5min":1.3979999999999999,"15min":1.401},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.60999999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3480000000000001,"15min":1.52},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":4.8499999999999996,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.77,"5min":1.361,"15min":1.343},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.9320000000000004,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":1.524}]},{"osd":11,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.2789999999999999,"15min":1.296},"min":{"1min":0.58599999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.6399999999999997,"5min":4.6399999999999997,"15min":4.6399999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.3899999999999999,"15min":1.53},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":8.4540000000000006,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.65700000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1534412,"kb_used_data":1237188,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92210740,"statfs":{"total":95995035648,"available":94423797760,"internally_reserved":0,"allocated":1266880512,"data_stored":1266552591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.97699999999999998,"15min":1.387},"min":{"1min":0.33100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.3530000000000002,"5min":5.2539999999999996,"15min":90.597999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.085,"15min":1.577},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":12.236000000000001,"5min":12.236000000000001,"15min":90.676000000000002},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":0.91300000000000003,"15min":0.94799999999999995},"min":{"1min":0.31,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":8.0530000000000008},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.0289999999999999,"15min":1.1240000000000001},"min":{"1min":0.29399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":4.1109999999999998,"5min":5.5709999999999997,"15min":7.452},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.2090000000000001,"15min":1.129},"min":{"1min":0.38100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":12.409000000000001,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.089,"15min":1.069},"min":{"1min":0.38400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.5629999999999997,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:26:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.98299999999999998,"15min":0.95799999999999996},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.7009999999999996,"5min":5.5999999999999996,"15min":6.0460000000000003},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.51,"5min":1.177,"15min":1.1060000000000001},"min":{"1min":0.34599999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.7160000000000002,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83599999999999997,"15min":1.0429999999999999},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0960000000000001,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.333,"5min":1.1339999999999999,"15min":1.0860000000000001},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.0289999999999999,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":0.443}]},{"osd":7,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.071,"15min":1.1539999999999999},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":9.0879999999999992},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.89700000000000002,"15min":1.1890000000000001},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.296,"5min":3.3799999999999999,"15min":7.6689999999999996},"last":0.61199999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1120000000000001,"15min":1.232},"min":{"1min":0.51900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.6099999999999999,"5min":3.3079999999999998,"15min":7.8650000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.698,"5min":1.331,"15min":1.2589999999999999},"min":{"1min":0.42799999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.4489999999999998,"5min":5.6699999999999999,"15min":5.6699999999999999},"last":0.755}]},{"osd":9,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.054,"15min":1.429},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.417,"5min":3.2650000000000001,"15min":34.112000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.361,"15min":1.6639999999999999},"min":{"1min":0.502,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.9740000000000002,"5min":5.6420000000000003,"15min":34.590000000000003},"last":0.68300000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.3740000000000001,"15min":1.3759999999999999},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.0949999999999998,"5min":5.6970000000000001,"15min":6.0899999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.073,"5min":1.163,"15min":1.262},"min":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.8300000000000001,"5min":4.46,"15min":8.5299999999999994},"last":0.53700000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.097,"15min":1.4910000000000001},"min":{"1min":0.52800000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.1640000000000001,"5min":3.2320000000000002,"15min":4.6500000000000004},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.512},"min":{"1min":0.59499999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.0570000000000004,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.71399999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1479888,"kb_used_data":1182664,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92265264,"statfs":{"total":95995035648,"available":94479630336,"internally_reserved":0,"allocated":1211047936,"data_stored":1210715943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80200000000000005,"15min":2.702},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.25800000000000001},"max":{"1min":3.331,"5min":3.8690000000000002,"15min":565.34500000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.91600000000000004,"15min":2.7829999999999999},"min":{"1min":0.39300000000000002,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":3.3660000000000001,"5min":3.9399999999999999,"15min":565.98400000000004},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.3440000000000001,"15min":1.097},"min":{"1min":0.36099999999999999,"5min":0.33200000000000002,"15min":0.30499999999999999},"max":{"1min":3.9289999999999998,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.361,"15min":1.129},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.194,"5min":40.225000000000001,"15min":40.225000000000001},"last":2.5070000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.3420000000000001,"15min":1.3129999999999999},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.5289999999999999,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.80600000000000005,"15min":0.754},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.1619999999999999,"5min":5.1619999999999999,"15min":5.1619999999999999},"last":2.5819999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.387,"15min":1.4299999999999999},"min":{"1min":0.317,"5min":0.246,"15min":0.246},"max":{"1min":41.908000000000001,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":3.4350000000000001,"5min":1.4370000000000001,"15min":1.4259999999999999},"min":{"1min":0.34999999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":42.430999999999997,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.60499999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.038,"15min":1.306},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":74.105000000000004},"last":2.9460000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.0580000000000001,"15min":1.423},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":75.013999999999996},"last":2.4620000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.012,"15min":1.026},"min":{"1min":0.374,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1120000000000001,"5min":6.1120000000000001,"15min":6.6630000000000003},"last":2.5409999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.0349999999999999,"15min":1.107},"min":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":5.6900000000000004,"5min":5.6900000000000004,"15min":5.9909999999999997},"last":1.0549999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:26:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.097,"15min":1.1930000000000001},"min":{"1min":0.47499999999999998,"5min":0.41699999999999998,"15min":0.40200000000000002},"max":{"1min":2.6859999999999999,"5min":4.0069999999999997,"15min":5.9509999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1619999999999999,"15min":1.222},"min":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":6.0739999999999998,"5min":6.0739999999999998,"15min":6.0739999999999998},"last":2.6699999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.373,"15min":3.2930000000000001},"min":{"1min":0.48399999999999999,"5min":0.46899999999999997,"15min":0.439},"max":{"1min":6.2939999999999996,"5min":6.2939999999999996,"15min":449.89499999999998},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.538,"5min":1.161,"15min":3.2599999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":9.6649999999999991,"5min":9.6649999999999991,"15min":448.70800000000003},"last":2.6269999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:26:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.1220000000000001,"15min":1.2589999999999999},"min":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.7639999999999998,"5min":4.0410000000000004,"15min":5.9210000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.304,"15min":1.3009999999999999},"min":{"1min":0.56999999999999995,"5min":0.52400000000000002,"15min":0.5},"max":{"1min":5.7949999999999999,"5min":5.7949999999999999,"15min":6.1379999999999999},"last":1.0169999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2070000000000001,"15min":1.204},"min":{"1min":0.503,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.806,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.367,"15min":1.3120000000000001},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":5.8230000000000004,"5min":5.8230000000000004,"15min":5.8230000000000004},"last":0.98299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1529806848,"data_stored":1529806848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1200029696,"data_stored":1200029696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":892600320,"data_stored":892600320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":308477952,"data_stored":308477952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1257504768,"data_stored":1257504768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1209602048,"data_stored":1209597976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":923598848,"data_stored":923598848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":320995328,"data_stored":320995328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":610205696,"data_stored":610205696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":595263488,"data_stored":595263488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":669974528,"data_stored":669974528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":335417344,"data_stored":335413272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:27:30.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:30.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:30.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:30.715 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:30.718 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:30 smithi121 bash[18489]: cluster 2024-02-12T16:27:29.983191+0000 mgr.y (mgr.14146) 1276 : cluster [DBG] pgmap v1771: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 8253/147064 objects misplaced (5.612%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:30.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:30.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:30.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:30.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:30.974 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:30 smithi123 bash[19119]: cluster 2024-02-12T16:27:29.983191+0000 mgr.y (mgr.14146) 1276 : cluster [DBG] pgmap v1771: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 8253/147064 objects misplaced (5.612%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:30.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:31.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:30 smithi169 bash[19604]: cluster 2024-02-12T16:27:29.983191+0000 mgr.y (mgr.14146) 1276 : cluster [DBG] pgmap v1771: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 8253/147064 objects misplaced (5.612%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:27:31.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:31.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:31.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:31.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:31.235 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:31.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:31.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:31.375 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:31.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:31.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:31.538 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:31.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:31.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:31.670 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:31.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:31.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:31.801 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:31.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:31.887 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:27:31.887 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:27:31.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:31.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:31.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:32.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:32.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:32.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:32.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:32.251 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:32.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:32.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:32.390 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:32.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:32.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:32.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:32.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:32.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:32.665 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:32.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:32.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-12T16:27:32.736 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:27:32.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:32.806 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:32.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:32.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:32 smithi123 bash[19119]: cluster 2024-02-12T16:27:31.539981+0000 osd.9 (osd.9) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:32.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:32 smithi123 bash[19119]: cluster 2024-02-12T16:27:31.540041+0000 osd.9 (osd.9) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:32.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:32 smithi123 bash[19119]: cluster 2024-02-12T16:27:31.719683+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e606: 12 total, 12 up, 12 in 2024-02-12T16:27:32.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:32 smithi123 bash[19119]: cluster 2024-02-12T16:27:31.983647+0000 mgr.y (mgr.14146) 1277 : cluster [DBG] pgmap v1773: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 7607/140408 objects misplaced (5.418%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:32.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:32 smithi123 bash[19119]: audit 2024-02-12T16:27:32.010211+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? v1:172.21.15.121:0/2211068532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:27:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:32 smithi121 bash[18489]: cluster 2024-02-12T16:27:31.539981+0000 osd.9 (osd.9) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:32 smithi121 bash[18489]: cluster 2024-02-12T16:27:31.540041+0000 osd.9 (osd.9) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:32 smithi121 bash[18489]: cluster 2024-02-12T16:27:31.719683+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e606: 12 total, 12 up, 12 in 2024-02-12T16:27:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:32 smithi121 bash[18489]: cluster 2024-02-12T16:27:31.983647+0000 mgr.y (mgr.14146) 1277 : cluster [DBG] pgmap v1773: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 7607/140408 objects misplaced (5.418%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:32 smithi121 bash[18489]: audit 2024-02-12T16:27:32.010211+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? v1:172.21.15.121:0/2211068532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:27:32.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:32.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:32.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:32.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:32.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:27:33.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:33.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:32 smithi169 bash[19604]: cluster 2024-02-12T16:27:31.539981+0000 osd.9 (osd.9) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:33.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:32 smithi169 bash[19604]: cluster 2024-02-12T16:27:31.540041+0000 osd.9 (osd.9) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:32 smithi169 bash[19604]: cluster 2024-02-12T16:27:31.719683+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e606: 12 total, 12 up, 12 in 2024-02-12T16:27:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:32 smithi169 bash[19604]: cluster 2024-02-12T16:27:31.983647+0000 mgr.y (mgr.14146) 1277 : cluster [DBG] pgmap v1773: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 7607/140408 objects misplaced (5.418%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:32 smithi169 bash[19604]: audit 2024-02-12T16:27:32.010211+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? v1:172.21.15.121:0/2211068532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:27:33.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:33.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:33.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:33.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:33.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:33.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:33.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:33.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:33.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:27:33.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:33.532 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:33.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:33.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:33.666 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:33.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:33.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:33.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:33 smithi169 bash[19604]: cluster 2024-02-12T16:27:32.714643+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:33 smithi169 bash[19604]: audit 2024-02-12T16:27:32.725275+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? v1:172.21.15.121:0/2211068532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:27:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:33 smithi169 bash[19604]: cluster 2024-02-12T16:27:32.725328+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e607: 12 total, 12 up, 12 in 2024-02-12T16:27:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:33.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:33.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:33.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:33.924 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:33 smithi123 bash[19119]: cluster 2024-02-12T16:27:32.714643+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:33.924 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:33 smithi123 bash[19119]: audit 2024-02-12T16:27:32.725275+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? v1:172.21.15.121:0/2211068532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:27:33.924 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:33 smithi123 bash[19119]: cluster 2024-02-12T16:27:32.725328+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e607: 12 total, 12 up, 12 in 2024-02-12T16:27:33.941 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:33.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:34.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:34.053 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:33 smithi121 bash[18489]: cluster 2024-02-12T16:27:32.714643+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:34.053 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:33 smithi121 bash[18489]: audit 2024-02-12T16:27:32.725275+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? v1:172.21.15.121:0/2211068532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:27:34.053 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:33 smithi121 bash[18489]: cluster 2024-02-12T16:27:32.725328+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e607: 12 total, 12 up, 12 in 2024-02-12T16:27:34.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:34.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:34.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:34.211 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:34.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:34.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:34.344 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:34.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:34.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:34.469 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:34.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:34.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:34.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:34.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:34.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:34.746 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:34.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:34.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:34.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:34.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:34.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:34.997 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:34 smithi121 bash[18489]: cluster 2024-02-12T16:27:33.984086+0000 mgr.y (mgr.14146) 1278 : cluster [DBG] pgmap v1775: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 7607/137876 objects misplaced (5.517%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:27:35.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:35.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:34 smithi169 bash[19604]: cluster 2024-02-12T16:27:33.984086+0000 mgr.y (mgr.14146) 1278 : cluster [DBG] pgmap v1775: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 7607/137876 objects misplaced (5.517%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:27:35.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:35.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:34 smithi123 bash[19119]: cluster 2024-02-12T16:27:33.984086+0000 mgr.y (mgr.14146) 1278 : cluster [DBG] pgmap v1775: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 7607/137876 objects misplaced (5.517%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:27:35.149 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:35.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:35.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:35.276 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:35.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:35.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:35.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:35.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:35.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:35.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:35.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:35.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:35.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:35.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:35.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:35.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:35.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:27:35.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:35.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:35.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:36.052 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:36.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:36.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:36.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:36.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:36.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:36.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:36.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:36.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:36.471 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:36.500 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:36.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:36.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:36.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:36.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:36.677 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:36 smithi169 bash[19604]: cluster 2024-02-12T16:27:35.984568+0000 mgr.y (mgr.14146) 1279 : cluster [DBG] pgmap v1776: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 7238/133762 objects misplaced (5.411%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:36.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:36.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:36.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:36.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:36 smithi123 bash[19119]: cluster 2024-02-12T16:27:35.984568+0000 mgr.y (mgr.14146) 1279 : cluster [DBG] pgmap v1776: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 7238/133762 objects misplaced (5.411%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:36.895 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:36.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:36.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:36 smithi121 bash[18489]: cluster 2024-02-12T16:27:35.984568+0000 mgr.y (mgr.14146) 1279 : cluster [DBG] pgmap v1776: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 7238/133762 objects misplaced (5.411%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:27:36.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:37.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:37.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:37.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:37.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:37.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:37.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:37.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:37.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:37.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:37.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:37.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:27:37.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:37.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:37.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:37.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:37.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:37.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:37.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:37.847 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:37.848 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:37.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:37.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:37.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:38.010 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:38.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:38.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:38.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:38.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:38.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:38.312 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:38.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:38.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:38.446 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:38.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:38.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:38.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:38.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:38.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:38.732 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:38.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:38.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:38.873 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:38.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:38.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:39.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:39.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:39.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:39 smithi121 bash[18489]: cluster 2024-02-12T16:27:37.985025+0000 mgr.y (mgr.14146) 1280 : cluster [DBG] pgmap v1777: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 6894/128352 objects misplaced (5.371%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:39.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:39.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:39.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:39.267 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:39 smithi169 bash[19604]: cluster 2024-02-12T16:27:37.985025+0000 mgr.y (mgr.14146) 1280 : cluster [DBG] pgmap v1777: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 6894/128352 objects misplaced (5.371%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:39.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:39.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:39.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:39.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:39 smithi123 bash[19119]: cluster 2024-02-12T16:27:37.985025+0000 mgr.y (mgr.14146) 1280 : cluster [DBG] pgmap v1777: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 6894/128352 objects misplaced (5.371%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:27:39.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:39.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:39.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:39.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:39.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:39.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:39.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:39.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:39.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:39.866 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:39.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:39.871 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:39.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:39.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:40.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.139 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:40.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:40.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:40 smithi169 bash[19604]: cluster 2024-02-12T16:27:38.281823+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:40.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:40 smithi169 bash[19604]: cluster 2024-02-12T16:27:38.281876+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:40 smithi169 bash[19604]: cluster 2024-02-12T16:27:39.052832+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e608: 12 total, 12 up, 12 in 2024-02-12T16:27:40.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:40 smithi123 bash[19119]: cluster 2024-02-12T16:27:38.281823+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:40.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:40 smithi123 bash[19119]: cluster 2024-02-12T16:27:38.281876+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:40.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:40 smithi123 bash[19119]: cluster 2024-02-12T16:27:39.052832+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e608: 12 total, 12 up, 12 in 2024-02-12T16:27:40.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:40.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:40 smithi121 bash[18489]: cluster 2024-02-12T16:27:38.281823+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:40.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:40 smithi121 bash[18489]: cluster 2024-02-12T16:27:38.281876+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:40.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:40 smithi121 bash[18489]: cluster 2024-02-12T16:27:39.052832+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e608: 12 total, 12 up, 12 in 2024-02-12T16:27:40.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:40.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:40.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:40.781 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:40.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:40.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:40.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:41.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:41.066 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:41.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:41.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:41 smithi121 bash[18489]: cluster 2024-02-12T16:27:39.985442+0000 mgr.y (mgr.14146) 1281 : cluster [DBG] pgmap v1779: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 6894/126308 objects misplaced (5.458%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:27:41.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:41.199 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:41.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:41.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:41.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:41.328 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:41 smithi169 bash[19604]: cluster 2024-02-12T16:27:39.985442+0000 mgr.y (mgr.14146) 1281 : cluster [DBG] pgmap v1779: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 6894/126308 objects misplaced (5.458%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:27:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:41.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:41.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:41 smithi123 bash[19119]: cluster 2024-02-12T16:27:39.985442+0000 mgr.y (mgr.14146) 1281 : cluster [DBG] pgmap v1779: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 881 op/s; 6894/126308 objects misplaced (5.458%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:27:41.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:41.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:41.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:41.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:41.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:41.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:41.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:41.750 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:27:41.750 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:27:41.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:41.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:41.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:41.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:41.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:41.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:42.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:42.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:42.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:42.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:42.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:42.261 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:42.287 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:42.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:42.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:42.428 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:42.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:42.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:42.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:42.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-12T16:27:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:42.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:42.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:42.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:42.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:42.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:42.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:42.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:42.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:43.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:43.129 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:43.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:43.240 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:43 smithi169 bash[19604]: cluster 2024-02-12T16:27:41.986172+0000 mgr.y (mgr.14146) 1282 : cluster [DBG] pgmap v1780: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 6196/118888 objects misplaced (5.212%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:27:43.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:43.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:43.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:43 smithi123 bash[19119]: cluster 2024-02-12T16:27:41.986172+0000 mgr.y (mgr.14146) 1282 : cluster [DBG] pgmap v1780: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 6196/118888 objects misplaced (5.212%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:27:43.382 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:43 smithi121 bash[18489]: cluster 2024-02-12T16:27:41.986172+0000 mgr.y (mgr.14146) 1282 : cluster [DBG] pgmap v1780: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 6196/118888 objects misplaced (5.212%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:27:43.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:43.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:43.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:43.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:43.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:43.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:43.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:43.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:43.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:44.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:44.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:44.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:44.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:44.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:44.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:44.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:44.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:44.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:44.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:44.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:44.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:44.829 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:44.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:44.958 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:44.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:45.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:45 smithi123 bash[19119]: cluster 2024-02-12T16:27:43.986677+0000 mgr.y (mgr.14146) 1283 : cluster [DBG] pgmap v1781: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 6196/116278 objects misplaced (5.329%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:45.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:45.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:45.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:45.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:45 smithi169 bash[19604]: cluster 2024-02-12T16:27:43.986677+0000 mgr.y (mgr.14146) 1283 : cluster [DBG] pgmap v1781: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 6196/116278 objects misplaced (5.329%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:45.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:45.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:45 smithi121 bash[18489]: cluster 2024-02-12T16:27:43.986677+0000 mgr.y (mgr.14146) 1283 : cluster [DBG] pgmap v1781: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 6196/116278 objects misplaced (5.329%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:45.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:45.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:45.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:45.488 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:45.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:45.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:45.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:45.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:45.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:45.749 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:45.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:45.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:45.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:45.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:45.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:46.053 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:46.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:46.182 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:46.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:46.306 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:46.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:46.435 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:46.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:46.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:46.643 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:27:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:27:46.672 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:27:46.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:46.676 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:27:46.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:46.720 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:27:46.761 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:27:46.796 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:27:46.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:46.825 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:46.851 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:27:46.884 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:27:46.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:46 smithi121 bash[18489]: cluster 2024-02-12T16:27:45.987158+0000 mgr.y (mgr.14146) 1284 : cluster [DBG] pgmap v1782: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5861/112704 objects misplaced (5.200%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:46.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:46 smithi121 bash[18489]: audit 2024-02-12T16:27:46.570106+0000 mon.a (mon.0) 2157 : audit [DBG] from='client.? v1:172.21.15.121:0/3187328419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:27:46.932 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:46.948 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:27:46.961 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:46.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:46.970 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:46 smithi123 bash[19119]: cluster 2024-02-12T16:27:45.987158+0000 mgr.y (mgr.14146) 1284 : cluster [DBG] pgmap v1782: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5861/112704 objects misplaced (5.200%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:46.970 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:46 smithi123 bash[19119]: audit 2024-02-12T16:27:46.570106+0000 mon.a (mon.0) 2157 : audit [DBG] from='client.? v1:172.21.15.121:0/3187328419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:46.983 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:27:47.019 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:27:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:46 smithi169 bash[19604]: cluster 2024-02-12T16:27:45.987158+0000 mgr.y (mgr.14146) 1284 : cluster [DBG] pgmap v1782: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5861/112704 objects misplaced (5.200%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:46 smithi169 bash[19604]: audit 2024-02-12T16:27:46.570106+0000 mon.a (mon.0) 2157 : audit [DBG] from='client.? v1:172.21.15.121:0/3187328419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:27:47.057 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:27:47.064 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.064 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.064 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.064 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.064 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.064 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.064 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.065 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.065 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.065 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.065 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.065 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:27:47.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:47.103 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:47.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:47.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:47.252 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:47.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:47.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:47.384 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:27:47.387 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:47.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:47.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:47.526 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:47.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:47.628 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:47.657 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:47.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:47.690 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:47 smithi169 bash[19604]: audit 2024-02-12T16:27:46.643421+0000 mgr.y (mgr.14146) 1285 : audit [DBG] from='client.16290 v1:172.21.15.121:0/3528376996' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:27:47.690 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:47 smithi169 bash[19604]: audit 2024-02-12T16:27:47.220508+0000 mon.a (mon.0) 2158 : audit [INF] from='client.26131 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:27:47.690 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:47 smithi169 bash[19604]: audit 2024-02-12T16:27:47.221869+0000 mon.c (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.121:0/2024144798' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:27:47.711 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:27:47.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:47.768 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:47 smithi123 bash[19119]: audit 2024-02-12T16:27:46.643421+0000 mgr.y (mgr.14146) 1285 : audit [DBG] from='client.16290 v1:172.21.15.121:0/3528376996' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:27:47.768 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:47 smithi123 bash[19119]: audit 2024-02-12T16:27:47.220508+0000 mon.a (mon.0) 2158 : audit [INF] from='client.26131 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:27:47.768 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:47 smithi123 bash[19119]: audit 2024-02-12T16:27:47.221869+0000 mon.c (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.121:0/2024144798' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:27:47.791 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:47.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:47.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:47.919 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:47.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:48.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:48.046 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:48.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:48.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:47 smithi121 bash[18489]: audit 2024-02-12T16:27:46.643421+0000 mgr.y (mgr.14146) 1285 : audit [DBG] from='client.16290 v1:172.21.15.121:0/3528376996' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:27:48.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:47 smithi121 bash[18489]: audit 2024-02-12T16:27:47.220508+0000 mon.a (mon.0) 2158 : audit [INF] from='client.26131 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:27:48.158 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:47 smithi121 bash[18489]: audit 2024-02-12T16:27:47.221869+0000 mon.c (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.121:0/2024144798' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:27:48.164 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1782,"stamp":"2024-02-12T16:27:45.986858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3693019160,"num_objects":56352,"num_object_clones":0,"num_object_copies":112704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5861,"num_objects_unfound":0,"num_objects_dirty":56352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282424,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":127205376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12168984,"kb_used_data":7315576,"kb_used_omap":91,"kb_used_meta":4853092,"kb_avail":1112772840,"statfs":{"total":1151940427776,"available":1139479388160,"internally_reserved":0,"allocated":7491149824,"data_stored":7486920316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":4969566224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-690028544,"num_objects":-10529,"num_object_clones":0,"num_object_copies":-21058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1377,"num_objects_unfound":0,"num_objects_dirty":-10529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10529,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":15007744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002575"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":96,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.727322+0000","last_change":"2024-02-12T16:26:42.266638+0000","last_active":"2024-02-12T16:27:32.727322+0000","last_peered":"2024-02-12T16:27:32.727322+0000","last_clean":"2024-02-12T16:27:32.727322+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:27:32.727322+0000","last_undegraded":"2024-02-12T16:27:32.727322+0000","last_fullsized":"2024-02-12T16:27:32.727322+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:42.266460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:26:42.266460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"608'17565","reported_seq":17638,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:43.658355+0000","last_change":"2024-02-12T16:25:46.665174+0000","last_active":"2024-02-12T16:27:43.658355+0000","last_peered":"2024-02-12T16:27:43.658355+0000","last_clean":"2024-02-12T16:27:43.658355+0000","last_became_active":"2024-02-12T16:24:09.648523+0000","last_became_peered":"2024-02-12T16:24:09.648523+0000","last_unstale":"2024-02-12T16:27:43.658355+0000","last_undegraded":"2024-02-12T16:27:43.658355+0000","last_fullsized":"2024-02-12T16:27:43.658355+0000","mapping_epoch":547,"log_start":"569'7500","ondisk_log_start":"569'7500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10057","last_scrub_stamp":"2024-02-12T16:25:46.665055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:46.665055+0000","log_size":10065,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":226689024,"num_objects":3459,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17565,"num_write_kb":672768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":199,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.732181+0000","last_change":"2024-02-12T16:27:16.104691+0000","last_active":"2024-02-12T16:27:32.732181+0000","last_peered":"2024-02-12T16:27:32.732181+0000","last_clean":"2024-02-12T16:27:32.732181+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:27:32.732181+0000","last_undegraded":"2024-02-12T16:27:32.732181+0000","last_fullsized":"2024-02-12T16:27:32.732181+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:40.282662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:40.282662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"608'17887","reported_seq":17965,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:44.291068+0000","last_change":"2024-02-12T16:26:52.958356+0000","last_active":"2024-02-12T16:27:44.291068+0000","last_peered":"2024-02-12T16:27:44.291068+0000","last_clean":"2024-02-12T16:27:44.291068+0000","last_became_active":"2024-02-12T16:24:10.018691+0000","last_became_peered":"2024-02-12T16:24:10.018691+0000","last_unstale":"2024-02-12T16:27:44.291068+0000","last_undegraded":"2024-02-12T16:27:44.291068+0000","last_fullsized":"2024-02-12T16:27:44.291068+0000","mapping_epoch":547,"log_start":"570'7800","ondisk_log_start":"570'7800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"598'14495","last_scrub_stamp":"2024-02-12T16:26:52.958251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:52.958251+0000","log_size":10087,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":227999744,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17887,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":290,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.729997+0000","last_change":"2024-02-12T16:24:47.872602+0000","last_active":"2024-02-12T16:27:32.729997+0000","last_peered":"2024-02-12T16:27:32.729997+0000","last_clean":"2024-02-12T16:27:32.729997+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:27:32.729997+0000","last_undegraded":"2024-02-12T16:27:32.729997+0000","last_fullsized":"2024-02-12T16:27:32.729997+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:47.872500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:24:47.872500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"608'17550","reported_seq":20035,"reported_epoch":608,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:27:41.657197+0000","last_change":"2024-02-12T16:25:58.008531+0000","last_active":"2024-02-12T16:27:41.657197+0000","last_peered":"2024-02-12T16:27:41.657197+0000","last_clean":"2024-02-12T16:25:54.194068+0000","last_became_active":"2024-02-12T16:25:57.932697+0000","last_became_peered":"2024-02-12T16:25:57.932697+0000","last_unstale":"2024-02-12T16:27:41.657197+0000","last_undegraded":"2024-02-12T16:27:41.657197+0000","last_fullsized":"2024-02-12T16:27:41.657197+0000","mapping_epoch":581,"log_start":"569'7500","ondisk_log_start":"569'7500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10027","last_scrub_stamp":"2024-02-12T16:25:45.765333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.765333+0000","log_size":10050,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":232390656,"num_objects":3546,"num_object_clones":0,"num_object_copies":7092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2904,"num_objects_unfound":0,"num_objects_dirty":3546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17550,"num_write_kb":675072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":78643200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":3546}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":695,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.727527+0000","last_change":"2024-02-12T16:25:07.263767+0000","last_active":"2024-02-12T16:27:32.727527+0000","last_peered":"2024-02-12T16:27:32.727527+0000","last_clean":"2024-02-12T16:27:32.727527+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:27:32.727527+0000","last_undegraded":"2024-02-12T16:27:32.727527+0000","last_fullsized":"2024-02-12T16:27:32.727527+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.263688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.263688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.3","version":"608'18042","reported_seq":18110,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:45.323151+0000","last_change":"2024-02-12T16:24:09.648426+0000","last_active":"2024-02-12T16:27:45.323151+0000","last_peered":"2024-02-12T16:27:45.323151+0000","last_clean":"2024-02-12T16:27:45.323151+0000","last_became_active":"2024-02-12T16:24:09.648139+0000","last_became_peered":"2024-02-12T16:24:09.648139+0000","last_unstale":"2024-02-12T16:27:45.323151+0000","last_undegraded":"2024-02-12T16:27:45.323151+0000","last_fullsized":"2024-02-12T16:27:45.323151+0000","mapping_epoch":547,"log_start":"570'8000","ondisk_log_start":"570'8000","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10042,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":222822400,"num_objects":3400,"num_object_clones":0,"num_object_copies":6800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18042,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":311,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.736893+0000","last_change":"2024-02-12T16:27:16.201789+0000","last_active":"2024-02-12T16:27:32.736893+0000","last_peered":"2024-02-12T16:27:32.736893+0000","last_clean":"2024-02-12T16:27:32.736893+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:27:32.736893+0000","last_undegraded":"2024-02-12T16:27:32.736893+0000","last_fullsized":"2024-02-12T16:27:32.736893+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:45.967340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:45.967340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"608'17449","reported_seq":17522,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:45.251578+0000","last_change":"2024-02-12T16:25:50.727730+0000","last_active":"2024-02-12T16:27:45.251578+0000","last_peered":"2024-02-12T16:27:45.251578+0000","last_clean":"2024-02-12T16:27:45.251578+0000","last_became_active":"2024-02-12T16:24:09.649607+0000","last_became_peered":"2024-02-12T16:24:09.649607+0000","last_unstale":"2024-02-12T16:27:45.251578+0000","last_undegraded":"2024-02-12T16:27:45.251578+0000","last_fullsized":"2024-02-12T16:27:45.251578+0000","mapping_epoch":547,"log_start":"569'7400","ondisk_log_start":"569'7400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10162","last_scrub_stamp":"2024-02-12T16:25:50.727611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.727611+0000","log_size":10049,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220659712,"num_objects":3367,"num_object_clones":0,"num_object_copies":6734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17449,"num_write_kb":666112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":281,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730015+0000","last_change":"2024-02-12T16:25:07.944563+0000","last_active":"2024-02-12T16:27:32.730015+0000","last_peered":"2024-02-12T16:27:32.730015+0000","last_clean":"2024-02-12T16:27:32.730015+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:27:32.730015+0000","last_undegraded":"2024-02-12T16:27:32.730015+0000","last_fullsized":"2024-02-12T16:27:32.730015+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.944451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.944451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"608'17285","reported_seq":17363,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:45.267563+0000","last_change":"2024-02-12T16:26:10.447550+0000","last_active":"2024-02-12T16:27:45.267563+0000","last_peered":"2024-02-12T16:27:45.267563+0000","last_clean":"2024-02-12T16:27:45.267563+0000","last_became_active":"2024-02-12T16:24:09.647095+0000","last_became_peered":"2024-02-12T16:24:09.647095+0000","last_unstale":"2024-02-12T16:27:45.267563+0000","last_undegraded":"2024-02-12T16:27:45.267563+0000","last_fullsized":"2024-02-12T16:27:45.267563+0000","mapping_epoch":547,"log_start":"569'7200","ondisk_log_start":"569'7200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10085,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":219873280,"num_objects":3355,"num_object_clones":0,"num_object_copies":6710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17285,"num_write_kb":660480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":283,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.729987+0000","last_change":"2024-02-12T16:25:37.899287+0000","last_active":"2024-02-12T16:27:32.729987+0000","last_peered":"2024-02-12T16:27:32.729987+0000","last_clean":"2024-02-12T16:27:32.729987+0000","last_became_active":"2024-02-12T16:17:37.450884+0000","last_became_peered":"2024-02-12T16:17:37.450884+0000","last_unstale":"2024-02-12T16:27:32.729987+0000","last_undegraded":"2024-02-12T16:27:32.729987+0000","last_fullsized":"2024-02-12T16:27:32.729987+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:37.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:25:37.899168+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"608'17709","reported_seq":17805,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:41.780377+0000","last_change":"2024-02-12T16:27:15.923522+0000","last_active":"2024-02-12T16:27:41.780377+0000","last_peered":"2024-02-12T16:27:41.780377+0000","last_clean":"2024-02-12T16:27:41.780377+0000","last_became_active":"2024-02-12T16:24:09.651253+0000","last_became_peered":"2024-02-12T16:24:09.651253+0000","last_unstale":"2024-02-12T16:27:41.780377+0000","last_undegraded":"2024-02-12T16:27:41.780377+0000","last_fullsized":"2024-02-12T16:27:41.780377+0000","mapping_epoch":547,"log_start":"569'7700","ondisk_log_start":"569'7700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10009,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":233635840,"num_objects":3565,"num_object_clones":0,"num_object_copies":7130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17709,"num_write_kb":680768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":150,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.724208+0000","last_change":"2024-02-12T16:25:54.579343+0000","last_active":"2024-02-12T16:27:32.724208+0000","last_peered":"2024-02-12T16:27:32.724208+0000","last_clean":"2024-02-12T16:27:32.724208+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:27:32.724208+0000","last_undegraded":"2024-02-12T16:27:32.724208+0000","last_fullsized":"2024-02-12T16:27:32.724208+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:54.579248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:54.579248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"608'17398","reported_seq":17476,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:41.342560+0000","last_change":"2024-02-12T16:27:12.609604+0000","last_active":"2024-02-12T16:27:41.342560+0000","last_peered":"2024-02-12T16:27:41.342560+0000","last_clean":"2024-02-12T16:27:41.342560+0000","last_became_active":"2024-02-12T16:24:09.647187+0000","last_became_peered":"2024-02-12T16:24:09.647187+0000","last_unstale":"2024-02-12T16:27:41.342560+0000","last_undegraded":"2024-02-12T16:27:41.342560+0000","last_fullsized":"2024-02-12T16:27:41.342560+0000","mapping_epoch":547,"log_start":"568'7300","ondisk_log_start":"568'7300","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'15509","last_scrub_stamp":"2024-02-12T16:27:12.609496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:12.609496+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":239992832,"num_objects":3662,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17398,"num_write_kb":673920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":693,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730148+0000","last_change":"2024-02-12T16:26:04.914413+0000","last_active":"2024-02-12T16:27:32.730148+0000","last_peered":"2024-02-12T16:27:32.730148+0000","last_clean":"2024-02-12T16:27:32.730148+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:27:32.730148+0000","last_undegraded":"2024-02-12T16:27:32.730148+0000","last_fullsized":"2024-02-12T16:27:32.730148+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:04.914304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:04.914304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"608'17952","reported_seq":18030,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:43.253262+0000","last_change":"2024-02-12T16:26:57.827400+0000","last_active":"2024-02-12T16:27:43.253262+0000","last_peered":"2024-02-12T16:27:43.253262+0000","last_clean":"2024-02-12T16:27:43.253262+0000","last_became_active":"2024-02-12T16:24:09.650120+0000","last_became_peered":"2024-02-12T16:24:09.650120+0000","last_unstale":"2024-02-12T16:27:43.253262+0000","last_undegraded":"2024-02-12T16:27:43.253262+0000","last_fullsized":"2024-02-12T16:27:43.253262+0000","mapping_epoch":547,"log_start":"570'7900","ondisk_log_start":"570'7900","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'14868","last_scrub_stamp":"2024-02-12T16:26:57.827287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:57.827287+0000","log_size":10052,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":232521728,"num_objects":3548,"num_object_clones":0,"num_object_copies":7096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17952,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":96,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730310+0000","last_change":"2024-02-12T16:26:43.025974+0000","last_active":"2024-02-12T16:27:32.730310+0000","last_peered":"2024-02-12T16:27:32.730310+0000","last_clean":"2024-02-12T16:27:32.730310+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:27:32.730310+0000","last_undegraded":"2024-02-12T16:27:32.730310+0000","last_fullsized":"2024-02-12T16:27:32.730310+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:43.025867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:43.025867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"608'17777","reported_seq":19344,"reported_epoch":608,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:27:45.315806+0000","last_change":"2024-02-12T16:26:41.328312+0000","last_active":"2024-02-12T16:27:45.315806+0000","last_peered":"2024-02-12T16:27:45.315806+0000","last_clean":"2024-02-12T16:26:09.048189+0000","last_became_active":"2024-02-12T16:26:11.222461+0000","last_became_peered":"2024-02-12T16:26:11.222461+0000","last_unstale":"2024-02-12T16:27:45.315806+0000","last_undegraded":"2024-02-12T16:27:45.315806+0000","last_fullsized":"2024-02-12T16:27:45.315806+0000","mapping_epoch":585,"log_start":"569'7700","ondisk_log_start":"569'7700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'9817","last_scrub_stamp":"2024-02-12T16:25:42.181950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:42.181950+0000","log_size":10077,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":224722944,"num_objects":3429,"num_object_clones":0,"num_object_copies":6858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2957,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17777,"num_write_kb":678592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":741,"num_bytes_recovered":48562176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[0,10],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,10","objects":3429}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":98,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.724226+0000","last_change":"2024-02-12T16:27:14.400780+0000","last_active":"2024-02-12T16:27:32.724226+0000","last_peered":"2024-02-12T16:27:32.724226+0000","last_clean":"2024-02-12T16:27:32.724226+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:27:32.724226+0000","last_undegraded":"2024-02-12T16:27:32.724226+0000","last_fullsized":"2024-02-12T16:27:32.724226+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:52.631321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.631321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"608'17895","reported_seq":17972,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:41.213010+0000","last_change":"2024-02-12T16:25:47.403836+0000","last_active":"2024-02-12T16:27:41.213010+0000","last_peered":"2024-02-12T16:27:41.213010+0000","last_clean":"2024-02-12T16:27:41.213010+0000","last_became_active":"2024-02-12T16:24:09.649590+0000","last_became_peered":"2024-02-12T16:24:09.649590+0000","last_unstale":"2024-02-12T16:27:41.213010+0000","last_undegraded":"2024-02-12T16:27:41.213010+0000","last_fullsized":"2024-02-12T16:27:41.213010+0000","mapping_epoch":547,"log_start":"569'7800","ondisk_log_start":"569'7800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10313","last_scrub_stamp":"2024-02-12T16:25:47.403730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.403730+0000","log_size":10095,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":237043712,"num_objects":3617,"num_object_clones":0,"num_object_copies":7234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17895,"num_write_kb":688384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":294,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.736843+0000","last_change":"2024-02-12T16:25:45.967785+0000","last_active":"2024-02-12T16:27:32.736843+0000","last_peered":"2024-02-12T16:27:32.736843+0000","last_clean":"2024-02-12T16:27:32.736843+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:27:32.736843+0000","last_undegraded":"2024-02-12T16:27:32.736843+0000","last_fullsized":"2024-02-12T16:27:32.736843+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:45.967726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.967726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"608'17276","reported_seq":17354,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:41.295938+0000","last_change":"2024-02-12T16:27:09.549045+0000","last_active":"2024-02-12T16:27:41.295938+0000","last_peered":"2024-02-12T16:27:41.295938+0000","last_clean":"2024-02-12T16:27:41.295938+0000","last_became_active":"2024-02-12T16:24:10.017647+0000","last_became_peered":"2024-02-12T16:24:10.017647+0000","last_unstale":"2024-02-12T16:27:41.295938+0000","last_undegraded":"2024-02-12T16:27:41.295938+0000","last_fullsized":"2024-02-12T16:27:41.295938+0000","mapping_epoch":547,"log_start":"567'7200","ondisk_log_start":"567'7200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'15168","last_scrub_stamp":"2024-02-12T16:27:09.548944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:09.548944+0000","log_size":10076,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":225574912,"num_objects":3442,"num_object_clones":0,"num_object_copies":6884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17276,"num_write_kb":662976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":98,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.729444+0000","last_change":"2024-02-12T16:26:59.766851+0000","last_active":"2024-02-12T16:27:32.729444+0000","last_peered":"2024-02-12T16:27:32.729444+0000","last_clean":"2024-02-12T16:27:32.729444+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:27:32.729444+0000","last_undegraded":"2024-02-12T16:27:32.729444+0000","last_fullsized":"2024-02-12T16:27:32.729444+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:59.766744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:59.766744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"608'17813","reported_seq":17887,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:45.223624+0000","last_change":"2024-02-12T16:25:40.938617+0000","last_active":"2024-02-12T16:27:45.223624+0000","last_peered":"2024-02-12T16:27:45.223624+0000","last_clean":"2024-02-12T16:27:45.223624+0000","last_became_active":"2024-02-12T16:24:09.649655+0000","last_became_peered":"2024-02-12T16:24:09.649655+0000","last_unstale":"2024-02-12T16:27:45.223624+0000","last_undegraded":"2024-02-12T16:27:45.223624+0000","last_fullsized":"2024-02-12T16:27:45.223624+0000","mapping_epoch":547,"log_start":"570'7800","ondisk_log_start":"570'7800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9693","last_scrub_stamp":"2024-02-12T16:25:40.938531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:40.938531+0000","log_size":10013,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":227016728,"num_objects":3465,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17813,"num_write_kb":680833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":696,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730104+0000","last_change":"2024-02-12T16:26:48.943869+0000","last_active":"2024-02-12T16:27:32.730104+0000","last_peered":"2024-02-12T16:27:32.730104+0000","last_clean":"2024-02-12T16:27:32.730104+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:27:32.730104+0000","last_undegraded":"2024-02-12T16:27:32.730104+0000","last_fullsized":"2024-02-12T16:27:32.730104+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:48.943785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:48.943785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"608'17751","reported_seq":17824,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:41.639493+0000","last_change":"2024-02-12T16:25:50.062621+0000","last_active":"2024-02-12T16:27:41.639493+0000","last_peered":"2024-02-12T16:27:41.639493+0000","last_clean":"2024-02-12T16:27:41.639493+0000","last_became_active":"2024-02-12T16:24:09.647617+0000","last_became_peered":"2024-02-12T16:24:09.647617+0000","last_unstale":"2024-02-12T16:27:41.639493+0000","last_undegraded":"2024-02-12T16:27:41.639493+0000","last_fullsized":"2024-02-12T16:27:41.639493+0000","mapping_epoch":547,"log_start":"569'7700","ondisk_log_start":"569'7700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10397","last_scrub_stamp":"2024-02-12T16:25:50.062524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.062524+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":242548736,"num_objects":3701,"num_object_clones":0,"num_object_copies":7402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17751,"num_write_kb":686464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":284,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730293+0000","last_change":"2024-02-12T16:24:50.575354+0000","last_active":"2024-02-12T16:27:32.730293+0000","last_peered":"2024-02-12T16:27:32.730293+0000","last_clean":"2024-02-12T16:27:32.730293+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:27:32.730293+0000","last_undegraded":"2024-02-12T16:27:32.730293+0000","last_fullsized":"2024-02-12T16:27:32.730293+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.575287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.575287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"608'17525","reported_seq":17598,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:41.848081+0000","last_change":"2024-02-12T16:26:09.203006+0000","last_active":"2024-02-12T16:27:41.848081+0000","last_peered":"2024-02-12T16:27:41.848081+0000","last_clean":"2024-02-12T16:27:41.848081+0000","last_became_active":"2024-02-12T16:24:09.650861+0000","last_became_peered":"2024-02-12T16:24:09.650861+0000","last_unstale":"2024-02-12T16:27:41.848081+0000","last_undegraded":"2024-02-12T16:27:41.848081+0000","last_fullsized":"2024-02-12T16:27:41.848081+0000","mapping_epoch":547,"log_start":"569'7500","ondisk_log_start":"569'7500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'11415","last_scrub_stamp":"2024-02-12T16:26:09.202913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:09.202913+0000","log_size":10025,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":244383744,"num_objects":3729,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17525,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":37,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.727378+0000","last_change":"2024-02-12T16:26:10.388710+0000","last_active":"2024-02-12T16:27:32.727378+0000","last_peered":"2024-02-12T16:27:32.727378+0000","last_clean":"2024-02-12T16:27:32.727378+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:27:32.727378+0000","last_undegraded":"2024-02-12T16:27:32.727378+0000","last_fullsized":"2024-02-12T16:27:32.727378+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:10.388515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:10.388515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"608'17550","reported_seq":17623,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-02-12T16:27:41.268634+0000","last_change":"2024-02-12T16:25:49.810317+0000","last_active":"2024-02-12T16:27:41.268634+0000","last_peered":"2024-02-12T16:27:41.268634+0000","last_clean":"2024-02-12T16:27:41.268634+0000","last_became_active":"2024-02-12T16:24:10.019266+0000","last_became_peered":"2024-02-12T16:24:10.019266+0000","last_unstale":"2024-02-12T16:27:41.268634+0000","last_undegraded":"2024-02-12T16:27:41.268634+0000","last_fullsized":"2024-02-12T16:27:41.268634+0000","mapping_epoch":547,"log_start":"569'7500","ondisk_log_start":"569'7500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10246","last_scrub_stamp":"2024-02-12T16:25:49.810203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:49.810203+0000","log_size":10050,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":235143168,"num_objects":3588,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17550,"num_write_kb":676416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":272,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.732243+0000","last_change":"2024-02-12T16:25:03.361871+0000","last_active":"2024-02-12T16:27:32.732243+0000","last_peered":"2024-02-12T16:27:32.732243+0000","last_clean":"2024-02-12T16:27:32.732243+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:27:32.732243+0000","last_undegraded":"2024-02-12T16:27:32.732243+0000","last_fullsized":"2024-02-12T16:27:32.732243+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:25:03.361757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":93,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730222+0000","last_change":"2024-02-12T16:25:59.785619+0000","last_active":"2024-02-12T16:27:32.730222+0000","last_peered":"2024-02-12T16:27:32.730222+0000","last_clean":"2024-02-12T16:27:32.730222+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:27:32.730222+0000","last_undegraded":"2024-02-12T16:27:32.730222+0000","last_fullsized":"2024-02-12T16:27:32.730222+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:59.785496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:25:59.785496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":91,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.739352+0000","last_change":"2024-02-12T16:24:54.540925+0000","last_active":"2024-02-12T16:27:32.739352+0000","last_peered":"2024-02-12T16:27:32.739352+0000","last_clean":"2024-02-12T16:27:32.739352+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:27:32.739352+0000","last_undegraded":"2024-02-12T16:27:32.739352+0000","last_fullsized":"2024-02-12T16:27:32.739352+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:54.540855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:54.540855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":107,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730376+0000","last_change":"2024-02-12T16:25:47.933452+0000","last_active":"2024-02-12T16:27:32.730376+0000","last_peered":"2024-02-12T16:27:32.730376+0000","last_clean":"2024-02-12T16:27:32.730376+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:27:32.730376+0000","last_undegraded":"2024-02-12T16:27:32.730376+0000","last_fullsized":"2024-02-12T16:27:32.730376+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:47.933341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.933341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":86,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-02-12T16:27:32.730036+0000","last_change":"2024-02-12T16:25:50.908791+0000","last_active":"2024-02-12T16:27:32.730036+0000","last_peered":"2024-02-12T16:27:32.730036+0000","last_clean":"2024-02-12T16:27:32.730036+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:27:32.730036+0000","last_undegraded":"2024-02-12T16:27:32.730036+0000","last_fullsized":"2024-02-12T16:27:32.730036+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.908692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.908692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3693019160,"num_objects":56352,"num_object_clones":0,"num_object_copies":112704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5861,"num_objects_unfound":0,"num_objects_dirty":56352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282424,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":127205376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7424122880,"data_stored":7424114736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958883,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":841152,"kb_used_data":255544,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92904000,"statfs":{"total":95995035648,"available":95133696000,"internally_reserved":0,"allocated":261677056,"data_stored":261321197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.82799999999999996,"15min":0.83899999999999997},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":5.6600000000000001,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.88600000000000001,"15min":0.89200000000000002},"min":{"1min":0.29599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.53100000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.89900000000000002,"15min":1.0309999999999999},"min":{"1min":0.36399999999999999,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":4.1520000000000001,"5min":4.6609999999999996,"15min":7.0110000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.875,"15min":1.0600000000000001},"min":{"1min":0.371,"5min":0.33900000000000002,"15min":0.28799999999999998},"max":{"1min":1.9670000000000001,"5min":4.2039999999999997,"15min":5.6920000000000002},"last":3.6859999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:27:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.96299999999999997,"15min":1.2869999999999999},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3420000000000001,"5min":5.343,"15min":5.343},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.075,"5min":1.129,"15min":1.4039999999999999},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.4340000000000002,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":0.58199999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.86599999999999999,"15min":2.2450000000000001},"min":{"1min":0.39400000000000002,"5min":0.34699999999999998,"15min":0.29499999999999998},"max":{"1min":2.831,"5min":4.4400000000000004,"15min":368.48399999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.998,"5min":0.89500000000000002,"15min":2.1699999999999999},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.101,"5min":4.915,"15min":371.65100000000001},"last":0.64300000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":1.1020000000000001,"15min":2.3860000000000001},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35099999999999998},"max":{"1min":4.6159999999999997,"5min":5.04,"15min":368.51400000000001},"last":0.5},{"interface":"front","average":{"1min":0.81100000000000005,"5min":1.04,"15min":2.3639999999999999},"min":{"1min":0.46999999999999997,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":4.1929999999999996,"5min":5.1509999999999998,"15min":368.70499999999998},"last":0.67500000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.93600000000000005,"15min":0.97399999999999998},"min":{"1min":0.39200000000000002,"5min":0.38500000000000001,"15min":0.373},"max":{"1min":4.7370000000000001,"5min":5.141,"15min":5.141},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.0880000000000001,"15min":1.103},"min":{"1min":0.47699999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.5800000000000001,"5min":8.5709999999999997,"15min":8.5709999999999997},"last":0.629}]},{"osd":7,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.0700000000000001,"15min":1.1950000000000001},"min":{"1min":0.502,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.1279999999999999,"5min":5.2569999999999997,"15min":6.4370000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.93700000000000006,"15min":1.153},"min":{"1min":0.46400000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.2350000000000003,"5min":5.0430000000000001,"15min":5.1189999999999998},"last":0.71299999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":6.2320000000000002,"15min":2.7999999999999998},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":2.3679999999999999,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":6.3879999999999999,"15min":2.9940000000000002},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.34699999999999998},"max":{"1min":1.165,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.76000000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.071,"15min":1.268},"min":{"1min":0.58399999999999996,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":1.9139999999999999,"5min":5.7430000000000003,"15min":10.743},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0900000000000001,"15min":1.381},"min":{"1min":0.60799999999999998,"5min":0.46800000000000003,"15min":0.39500000000000002},"max":{"1min":1.861,"5min":5.359,"15min":10.698},"last":1.288}]},{"osd":10,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":6.3710000000000004,"15min":2.9670000000000001},"min":{"1min":0.48899999999999999,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":4.0599999999999996,"5min":536.245,"15min":536.245},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":6.4009999999999998,"15min":2.948},"min":{"1min":0.54700000000000004,"5min":0.443,"15min":0.443},"max":{"1min":2.544,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.89300000000000002}]}]},{"osd":10,"up_from":83,"seq":356482285932,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":797436,"kb_used_data":496116,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92947716,"statfs":{"total":95995035648,"available":95178461184,"internally_reserved":0,"allocated":508022784,"data_stored":507670997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":2.5899999999999999,"15min":1.518},"min":{"1min":0.33800000000000002,"5min":0.245,"15min":0.245},"max":{"1min":1.97,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.84299999999999997,"5min":2.6419999999999999,"15min":1.5960000000000001},"min":{"1min":0.30499999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.4829999999999997,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.90600000000000003,"15min":0.94099999999999995},"min":{"1min":0.40000000000000002,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":2.5179999999999998,"5min":4.4059999999999997,"15min":19.866},"last":2.371},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.86599999999999999,"15min":1.0680000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":20.277999999999999},"last":1.2649999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95499999999999996,"15min":0.878},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.3200000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":1.76},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.123,"15min":0.95899999999999996},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":5.3680000000000003,"5min":5.3680000000000003,"15min":5.3680000000000003},"last":1.421}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.879,"15min":3.3210000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.32300000000000001},"max":{"1min":1.139,"5min":3.2639999999999998,"15min":628.77099999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.1240000000000001,"5min":0.96199999999999997,"15min":3.29},"min":{"1min":0.38800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":624.19299999999998},"last":1.216}]},{"osd":5,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.03,"15min":1.024},"min":{"1min":0.47399999999999998,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":5.0910000000000002,"5min":5.0910000000000002,"15min":5.5739999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.085,"15min":1.109},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":1.26,"5min":3.2949999999999999,"15min":4.8769999999999998},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.004,"15min":3.3540000000000001},"min":{"1min":0.53500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.1339999999999999,"5min":4.2640000000000002,"15min":626.26599999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.008,"15min":3.4390000000000001},"min":{"1min":0.438,"5min":0.376,"15min":0.376},"max":{"1min":4.1829999999999998,"5min":4.1829999999999998,"15min":625.90700000000004},"last":0.752}]},{"osd":7,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.95799999999999996,"15min":1.05},"min":{"1min":0.46100000000000002,"5min":0.40400000000000003,"15min":0.39800000000000002},"max":{"1min":4.1369999999999996,"5min":4.1369999999999996,"15min":5.9210000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":1.119,"15min":1.155},"min":{"1min":0.60299999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.1629999999999998,"5min":4.3150000000000004,"15min":5.3300000000000001},"last":1.3899999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.0649999999999999,"15min":1.101},"min":{"1min":0.51400000000000001,"5min":0.436,"15min":0.372},"max":{"1min":3.2210000000000001,"5min":3.2210000000000001,"15min":4.827},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.052,"15min":1.1120000000000001},"min":{"1min":0.46400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.2359999999999998,"5min":4.2359999999999998,"15min":4.8330000000000002},"last":0.67400000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.119,"15min":1.2010000000000001},"min":{"1min":0.57399999999999995,"5min":0.47599999999999998,"15min":0.372},"max":{"1min":2.105,"5min":4.782,"15min":4.9829999999999997},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.101,"15min":1.2010000000000001},"min":{"1min":0.54700000000000004,"5min":0.47299999999999998,"15min":0.41399999999999998},"max":{"1min":4.1699999999999999,"5min":4.1699999999999999,"15min":4.952},"last":0.64300000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.159,"15min":1.3100000000000001},"min":{"1min":0.59299999999999997,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":7.3689999999999998,"5min":7.3689999999999998,"15min":10.218999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.875,"5min":1.016,"15min":1.268},"min":{"1min":0.622,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.21,"5min":2.21,"15min":7.3250000000000002},"last":0.81999999999999995}]}]},{"osd":9,"up_from":74,"seq":317827580276,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":771900,"kb_used_data":462388,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92973252,"statfs":{"total":95995035648,"available":95204610048,"internally_reserved":0,"allocated":473485312,"data_stored":473133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.006,"15min":1.042},"min":{"1min":0.30199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.4450000000000003,"5min":5.8330000000000002,"15min":5.8330000000000002},"last":1.167},{"interface":"front","average":{"1min":1.2689999999999999,"5min":0.93600000000000005,"15min":0.90500000000000003},"min":{"1min":0.35999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.4619999999999997,"5min":5.4619999999999997,"15min":5.4619999999999997},"last":2.581}]},{"osd":1,"last update":"Mon Feb 12 16:27:01 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.0680000000000001,"15min":0.94899999999999995},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.253},"max":{"1min":3.9620000000000002,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":0.96299999999999997,"15min":1.046},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":3.976,"5min":5.4560000000000004,"15min":7.9550000000000001},"last":2.7599999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0720000000000001,"15min":1.409},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.27600000000000002},"max":{"1min":3.7709999999999999,"5min":5.915,"15min":97.275000000000006},"last":1.349},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.0549999999999999,"15min":1.258},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.312},"max":{"1min":4.0780000000000003,"5min":6.1779999999999999,"15min":97.417000000000002},"last":2.7989999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.044,"15min":0.95299999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":4.173,"5min":4.569,"15min":4.569},"last":2.6240000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.1919999999999999,"15min":1.123},"min":{"1min":0.38500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.2510000000000003,"5min":4.8840000000000003,"15min":5.8010000000000002},"last":2.8119999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.3140000000000001,"15min":1.107},"min":{"1min":0.41999999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.726,"5min":6.726,"15min":6.726},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.1819999999999999,"15min":1.1519999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":6.7350000000000003,"5min":6.7350000000000003,"15min":7.3739999999999997},"last":2.5449999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.0189999999999999,"15min":1.089},"min":{"1min":0.39400000000000002,"5min":0.379,"15min":0.379},"max":{"1min":2.9750000000000001,"5min":5.5250000000000004,"15min":7.3360000000000003},"last":2.6459999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.1659999999999999,"15min":1.04},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.21,"5min":7.0609999999999999,"15min":7.0609999999999999},"last":2.7770000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.3280000000000001,"15min":1.2270000000000001},"min":{"1min":0.50800000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.8420000000000001,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":1.413},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.212,"15min":1.2130000000000001},"min":{"1min":0.47399999999999998,"5min":0.44600000000000001,"15min":0.438},"max":{"1min":3.6629999999999998,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":2.8239999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0880000000000001,"15min":1.0680000000000001},"min":{"1min":0.437,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.9460000000000002,"5min":5.3620000000000001,"15min":5.4059999999999997},"last":2.8420000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.1160000000000001,"15min":1.1950000000000001},"min":{"1min":0.48699999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":4.3310000000000004,"5min":4.3310000000000004,"15min":7.25},"last":2.7269999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.2,"15min":1.2070000000000001},"min":{"1min":0.54800000000000004,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":3.0550000000000002,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.1619999999999999,"15min":1.2030000000000001},"min":{"1min":0.49199999999999999,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":3.8319999999999999,"5min":4.4509999999999996,"15min":5.8700000000000001},"last":2.835}]},{"osd":11,"last update":"Mon Feb 12 16:26:44 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.2949999999999999,"15min":1.349},"min":{"1min":0.55000000000000004,"5min":0.48999999999999999,"15min":0.45600000000000002},"max":{"1min":3.7370000000000001,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.1950000000000001,"15min":1.3100000000000001},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.0169999999999999,"5min":5.492,"15min":6.3639999999999999},"last":2.8929999999999998}]}]},{"osd":8,"up_from":65,"seq":279172874622,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739324,"kb_used_data":438004,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93005828,"statfs":{"total":95995035648,"available":95237967872,"internally_reserved":0,"allocated":448516096,"data_stored":448164317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.006,"15min":0.88600000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":4.7270000000000003,"5min":19.196999999999999,"15min":19.196999999999999},"last":2.2040000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.093,"15min":0.88900000000000001},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29399999999999998},"max":{"1min":3.331,"5min":18.584,"15min":18.584},"last":2.294}]},{"osd":1,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.7669999999999999,"15min":1.4530000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.4889999999999999,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.7529999999999999,"15min":1.53},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.1260000000000003,"5min":71.010999999999996,"15min":71.010999999999996},"last":2.7450000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4160000000000004,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.2170000000000001,"5min":0.93999999999999995,"15min":0.998},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.7240000000000002,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":0.36699999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.82899999999999996,"15min":1.7150000000000001},"min":{"1min":0.33700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.7250000000000001,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.875,"15min":1.6519999999999999},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.95299999999999996,"5min":5.0380000000000003,"15min":224.71000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.85199999999999998,"15min":0.90200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":5.2480000000000002,"5min":5.2480000000000002,"15min":5.2480000000000002},"last":0.443},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.95199999999999996,"15min":1.0229999999999999},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.5600000000000001,"5min":3.5600000000000001,"15min":4.8339999999999996},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.155,"15min":1.7410000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.37},"max":{"1min":5.3639999999999999,"5min":5.3639999999999999,"15min":221.36699999999999},"last":0.745},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.099,"15min":1.8600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.2869999999999999,"5min":5.556,"15min":221.203},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.1319999999999999,"15min":1.1459999999999999},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.42199999999999999},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.1890000000000001,"15min":1.23},"min":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.41599999999999998},"max":{"1min":4.2240000000000002,"5min":4.2240000000000002,"15min":7.899},"last":0.60699999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:27:05 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.081,"15min":1.1679999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35599999999999998},"max":{"1min":3.8610000000000002,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1850000000000001,"15min":1.226},"min":{"1min":0.53600000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":3.8940000000000001,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":0.65500000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.996,"15min":1.4299999999999999},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":109.92},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.032,"15min":1.474},"min":{"1min":0.504,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6469999999999998,"5min":5.0250000000000004,"15min":110.249},"last":0.61899999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":1.1819999999999999},"min":{"1min":0.627,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":3.3109999999999999,"5min":3.3109999999999999,"15min":5.0670000000000002},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":1.141},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.327,"5min":3.327,"15min":4.6639999999999997},"last":0.58999999999999997}]}]},{"osd":7,"up_from":56,"seq":240518168965,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":528060,"kb_used_data":230836,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93217092,"statfs":{"total":95995035648,"available":95454302208,"internally_reserved":0,"allocated":236376064,"data_stored":236024285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":11.994,"5min":3.085,"15min":1.6419999999999999},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":187.33699999999999,"5min":187.33699999999999,"15min":187.33699999999999},"last":0.748},{"interface":"front","average":{"1min":11.662000000000001,"5min":3.0800000000000001,"15min":1.653},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":187.48400000000001,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.77700000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.3169999999999999,"15min":1.044},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.4009999999999998,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.591,"15min":1.1859999999999999},"min":{"1min":0.34699999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.4670000000000001,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:27:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.034,"15min":1.2430000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.1669999999999998,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.88200000000000001,"15min":0.97999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.4340000000000002,"5min":4.4699999999999998,"15min":4.4699999999999998},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.131,"15min":0.93999999999999995},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.899,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.92600000000000005,"15min":0.998},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":4.3170000000000002,"5min":4.7389999999999999,"15min":4.7389999999999999},"last":0.32700000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0029999999999999,"15min":0.96799999999999997},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.5499999999999998,"5min":4.7220000000000004,"15min":4.9050000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.999,"15min":1.018},"min":{"1min":0.40500000000000003,"5min":0.314,"15min":0.314},"max":{"1min":2.964,"5min":7.5640000000000001,"15min":9.3640000000000008},"last":0.71599999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.085,"15min":1.1000000000000001},"min":{"1min":0.34599999999999997,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":3.5819999999999999,"5min":6.319,"15min":6.319},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1240000000000001,"15min":1.179},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":4.3609999999999998,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.45100000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":6.5179999999999998,"15min":2.9369999999999998},"min":{"1min":0.46300000000000002,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.855,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":6.5220000000000002,"15min":2.9830000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.9119999999999999,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.97999999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:26:57 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.03,"15min":1.1890000000000001},"min":{"1min":0.42899999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.6539999999999999,"5min":4.6539999999999999,"15min":12.907999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.133,"15min":1.2390000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.71,"5min":4.665,"15min":4.6719999999999997},"last":2.552}]},{"osd":10,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":7.7400000000000002,"15min":3.508},"min":{"1min":0.46500000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.5309999999999997,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":7.6219999999999999,"15min":3.375},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.946,"5min":496.29599999999999,"15min":496.29599999999999},"last":1.486}]},{"osd":11,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1160000000000001,"15min":1.617},"min":{"1min":0.495,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":2.27,"5min":6.0490000000000004,"15min":120.943},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.21,"15min":1.7150000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":4.5,"5min":4.7110000000000003,"15min":120.661},"last":0.59799999999999998}]}]},{"osd":6,"up_from":48,"seq":206158430606,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964668,"kb_used_data":655156,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92780484,"statfs":{"total":95995035648,"available":95007215616,"internally_reserved":0,"allocated":670879744,"data_stored":670527965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.111,"15min":1.0640000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.464,"5min":30.454999999999998,"15min":30.454999999999998},"last":2.1739999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.246,"15min":1.1519999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.9769999999999999,"5min":30.573,"15min":30.573},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.028,"15min":1.921},"min":{"1min":0.29599999999999999,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":5.5179999999999998,"5min":5.5179999999999998,"15min":252.22800000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.298,"5min":0.98099999999999998,"15min":1.9259999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.6959999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.3520000000000001,"15min":1.004},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":5.9500000000000002,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.5820000000000001,"15min":1.1200000000000001},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.7939999999999996,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.2230000000000001,"15min":1.968},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.2260000000000009,"5min":8.2260000000000009,"15min":284.05599999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.1020000000000001,"15min":2.0390000000000001},"min":{"1min":0.28799999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.0259999999999998,"5min":7.5979999999999999,"15min":284.041},"last":0.497}]},{"osd":5,"last update":"Mon Feb 12 16:27:08 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.0409999999999999,"15min":1.9139999999999999},"min":{"1min":0.34100000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.4249999999999998,"5min":5.4249999999999998,"15min":281.49700000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.0309999999999999,"15min":1.9890000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.52,"5min":10.52,"15min":281.61900000000003},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0720000000000001,"15min":1.105},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.0970000000000004,"5min":7.641,"15min":7.641},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.0700000000000001,"15min":1.071},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.7759999999999998,"5min":7.8380000000000001,"15min":7.8380000000000001},"last":0.59299999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.0800000000000001,"15min":1.1080000000000001},"min":{"1min":0.42099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":5.6310000000000002,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.0169999999999999,"15min":1.1890000000000001},"min":{"1min":0.39600000000000002,"5min":0.371,"15min":0.371},"max":{"1min":2.6440000000000001,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":0.78600000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.121,"15min":1.2090000000000001},"min":{"1min":0.42099999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":3.044,"5min":4.5330000000000004,"15min":18.059999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.115,"15min":1.27},"min":{"1min":0.47499999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.673,"5min":4.6890000000000001,"15min":17.425000000000001},"last":0.85499999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:26:45 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.202,"15min":1.224},"min":{"1min":0.505,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":6.5300000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.276,"15min":1.3080000000000001},"min":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.1660000000000004,"5min":5.1660000000000004,"15min":6.5730000000000004},"last":0.68200000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:26:45 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.242,"15min":3.2519999999999998},"min":{"1min":0.53700000000000003,"5min":0.499,"15min":0.499},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":139.727},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.151,"15min":3.125},"min":{"1min":0.51500000000000001,"5min":0.504,"15min":0.504},"max":{"1min":2.706,"5min":4.5899999999999999,"15min":139.24199999999999},"last":0.82799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1816252,"kb_used_data":926004,"kb_used_omap":0,"kb_used_meta":890112,"kb_avail":91928900,"statfs":{"total":95995035648,"available":94135193600,"internally_reserved":0,"allocated":948228096,"data_stored":947876317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911474688},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.90600000000000003,"15min":1.329},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":7.8710000000000004,"5min":7.8710000000000004,"15min":114.40000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.589,"5min":1.2969999999999999,"15min":1.5780000000000001},"min":{"1min":0.30299999999999999,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":5.2690000000000001,"5min":7.7249999999999996,"15min":115.09099999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.1639999999999999,"15min":1.04},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":6.5119999999999996,"5min":6.5389999999999997,"15min":9.8599999999999994},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.76,"5min":1.367,"15min":1.1180000000000001},"min":{"1min":0.28899999999999998,"5min":0.246,"15min":0.246},"max":{"1min":9.9689999999999994,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.147,"15min":1.139},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":9.9529999999999994,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.441},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.0860000000000001,"15min":0.92000000000000004},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.5490000000000004,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.78700000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.2829999999999999,"15min":1.125},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.6749999999999998,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.4650000000000001,"15min":1.405},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.633,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.399,"15min":1.2190000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.6740000000000004,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2629999999999999,"15min":1.196},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":10.067,"5min":10.067,"15min":10.067},"last":0.72999999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.216,"15min":1.6379999999999999},"min":{"1min":0.439,"5min":0.38500000000000001,"15min":0.34699999999999998},"max":{"1min":2.4020000000000001,"5min":6.6310000000000002,"15min":55.869999999999997},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2110000000000001,"15min":1.6970000000000001},"min":{"1min":0.48399999999999999,"5min":0.46600000000000003,"15min":0.36799999999999999},"max":{"1min":2.5219999999999998,"5min":4.4470000000000001,"15min":55.892000000000003},"last":0.76700000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:26:39 2024","interfaces":[{"interface":"back","average":{"1min":31.306999999999999,"5min":7.3899999999999997,"15min":3.371},"min":{"1min":0.53700000000000003,"5min":0.51000000000000001,"15min":0.436},"max":{"1min":548.91600000000005,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.82299999999999995},{"interface":"front","average":{"1min":32.816000000000003,"5min":7.6150000000000002,"15min":3.4550000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":552.42100000000005,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.59699999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:26:52 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.4590000000000001,"15min":1.3919999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":4.4720000000000004,"5min":5.7240000000000002,"15min":10.07},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.214,"5min":1.3360000000000001,"15min":1.492},"min":{"1min":0.60399999999999998,"5min":0.46700000000000003,"15min":0.45000000000000001},"max":{"1min":6.1559999999999997,"5min":6.1559999999999997,"15min":7.6059999999999999},"last":0.79600000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:26:42 2024","interfaces":[{"interface":"back","average":{"1min":36.500999999999998,"5min":8.5050000000000008,"15min":3.7400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":548.93499999999995,"5min":548.93499999999995,"15min":548.93499999999995},"last":3.718},{"interface":"front","average":{"1min":35.311,"5min":8.3140000000000001,"15min":3.774},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.53700000000000003},"max":{"1min":549.12099999999998,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.89000000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.526,"15min":1.5680000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.443},"max":{"1min":10.202,"5min":10.202,"15min":10.202},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.649,"5min":1.6419999999999999,"15min":1.627},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":5.8300000000000001,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":0.65100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689724,"kb_used_data":1095924,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92055428,"statfs":{"total":95995035648,"available":94264758272,"internally_reserved":0,"allocated":1122226176,"data_stored":1121874397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.89800000000000002,"15min":1.0620000000000001},"min":{"1min":0.254,"5min":0.245,"15min":0.215},"max":{"1min":4.4260000000000002,"5min":10.754,"15min":11.324},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.0129999999999999,"15min":1.0580000000000001},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.6859999999999999,"5min":4.6859999999999999,"15min":9.8249999999999993},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.0649999999999999,"15min":1.0580000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.0279999999999996,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.0259999999999998,"5min":1.0940000000000001,"15min":0.97999999999999998},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.5510000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.167,"15min":1.7130000000000001},"min":{"1min":0.38900000000000001,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":4.8600000000000003,"5min":7.6219999999999999,"15min":116.187},"last":0.62},{"interface":"front","average":{"1min":0.87,"5min":1.3080000000000001,"15min":1.829},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.28299999999999997},"max":{"1min":3.1030000000000002,"5min":7.5599999999999996,"15min":116.163},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.1020000000000001,"15min":1.016},"min":{"1min":0.33600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.2690000000000001,"5min":5.2690000000000001,"15min":5.2690000000000001},"last":1.8700000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1659999999999999,"15min":1.052},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":2.0190000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:27:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.2989999999999999,"15min":1.369},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":81.147000000000006},"last":1.903},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.339,"15min":1.425},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.0640000000000001,"5min":8.8789999999999996,"15min":80.233000000000004},"last":1.9330000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.2170000000000001,"15min":1.5880000000000001},"min":{"1min":0.40899999999999997,"5min":0.374,"15min":0.36699999999999999},"max":{"1min":4.4219999999999997,"5min":4.4219999999999997,"15min":115.26000000000001},"last":1.9950000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.3080000000000001,"15min":1.6919999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.4889999999999999,"5min":4.4889999999999999,"15min":115.402},"last":2.0419999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.276,"15min":1.2689999999999999},"min":{"1min":0.48399999999999999,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":5.1070000000000002,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":2.089},{"interface":"front","average":{"1min":1.552,"5min":1.369,"15min":1.357},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":5.1749999999999998,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":2.1110000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.1160000000000001,"15min":1.3440000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.3920000000000003,"5min":7.6539999999999999,"15min":46.246000000000002},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3779999999999999,"15min":1.395},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.2080000000000002,"5min":5.2080000000000002,"15min":46.167999999999999},"last":1.9650000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2589999999999999,"15min":1.276},"min":{"1min":0.54400000000000004,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":4.944,"5min":4.944,"15min":6.0979999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.339,"15min":1.3160000000000001},"min":{"1min":0.57399999999999995,"5min":0.52000000000000002,"15min":0.47999999999999998},"max":{"1min":5.0049999999999999,"5min":5.0049999999999999,"15min":6.2480000000000002},"last":0.84199999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":1.452,"15min":1.575},"min":{"1min":0.497,"5min":0.46700000000000003,"15min":0.434},"max":{"1min":5.1050000000000004,"5min":5.1050000000000004,"15min":46.064},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.355,"15min":1.5049999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.38300000000000001},"max":{"1min":5.1390000000000002,"5min":5.1390000000000002,"15min":45.642000000000003},"last":2.0649999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313644,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1005564,"kb_used_data":691956,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":92739588,"statfs":{"total":95995035648,"available":94965338112,"internally_reserved":0,"allocated":708562944,"data_stored":708211165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":2.371,"5min":1.3200000000000001,"15min":1.1180000000000001},"min":{"1min":0.312,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":1.6459999999999999},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.446,"15min":1.208},"min":{"1min":0.32600000000000001,"5min":0.247,"15min":0.247},"max":{"1min":6.4180000000000001,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":1.6639999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.6899999999999999,"15min":1.98},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":6.4509999999999996,"5min":21.407,"15min":21.407},"last":0.501},{"interface":"front","average":{"1min":1.698,"5min":1.04,"15min":0.90000000000000002},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0780000000000001,"15min":0.95599999999999996},"min":{"1min":0.34000000000000002,"5min":0.313,"15min":0.313},"max":{"1min":3.9079999999999999,"5min":6.8090000000000002,"15min":7.2859999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.2989999999999999,"15min":1.093},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.601,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.54900000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":1.3420000000000001,"15min":1.147},"min":{"1min":0.44800000000000001,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":6.133,"5min":6.133,"15min":6.133},"last":2.5},{"interface":"front","average":{"1min":2.8140000000000001,"5min":1.5289999999999999,"15min":1.1850000000000001},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.315},"max":{"1min":11.081,"5min":11.081,"15min":11.081},"last":6.8170000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:26:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5309999999999999,"15min":1.2010000000000001},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.5279999999999996,"5min":21.527000000000001,"15min":21.527000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.2969999999999999,"15min":1.137},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":6.2850000000000001,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1499999999999999,"15min":1.0649999999999999},"min":{"1min":0.41899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.9489999999999998,"5min":4.8570000000000002,"15min":5.2960000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.571,"5min":1.2569999999999999,"15min":1.1870000000000001},"min":{"1min":0.47999999999999998,"5min":0.443,"15min":0.38700000000000001},"max":{"1min":5.5099999999999998,"5min":5.5099999999999998,"15min":5.5099999999999998},"last":0.53800000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.883,"5min":1.3320000000000001,"15min":1.2250000000000001},"min":{"1min":0.499,"5min":0.442,"15min":0.38700000000000001},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.71,"5min":1.429,"15min":1.2729999999999999},"min":{"1min":0.42499999999999999,"5min":0.38,"15min":0.38},"max":{"1min":6.5659999999999998,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":0.63600000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:26:47 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.25,"15min":1.181},"min":{"1min":0.47099999999999997,"5min":0.41899999999999998,"15min":0.40600000000000003},"max":{"1min":4.0119999999999996,"5min":4.8040000000000003,"15min":5.2329999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3380000000000001,"15min":1.252},"min":{"1min":0.50600000000000001,"5min":0.45400000000000001,"15min":0.42399999999999999},"max":{"1min":3.9809999999999999,"5min":4.9089999999999998,"15min":5.085},"last":0.51800000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.5629999999999999,"15min":1.385},"min":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6319999999999997,"5min":11.454000000000001,"15min":11.454000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":2.032,"5min":1.5800000000000001,"15min":1.411},"min":{"1min":0.501,"5min":0.441,"15min":0.441},"max":{"1min":6.8010000000000002,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":0.71999999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:26:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.3300000000000001,"15min":1.49},"min":{"1min":0.53300000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":5.9050000000000002,"5min":5.9050000000000002,"15min":5.9050000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.7270000000000001,"5min":1.4530000000000001,"15min":1.571},"min":{"1min":0.59699999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":5.9359999999999999,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":0.71299999999999997}]}]},{"osd":3,"up_from":522,"seq":2241972928570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":559164,"kb_used_data":222836,"kb_used_omap":91,"kb_used_meta":336228,"kb_avail":93185988,"statfs":{"total":95995035648,"available":95422451712,"internally_reserved":0,"allocated":228184064,"data_stored":227832285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":344297488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":3.21,"5min":1.2490000000000001,"15min":0.98099999999999998},"min":{"1min":0.32500000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":34.055999999999997,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":3.4910000000000001,"5min":1.361,"15min":1.022},"min":{"1min":0.33800000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":27.600000000000001,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.1200000000000001,"15min":0.95199999999999996},"min":{"1min":0.317,"5min":0.314,"15min":0.314},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.341,"5min":1.1970000000000001,"15min":1.1020000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.746,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":0.371}]},{"osd":4,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.0589999999999999,"15min":1.304},"min":{"1min":0.38400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.7080000000000002,"5min":10.004,"15min":10.004},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.085,"15min":1.1160000000000001},"min":{"1min":0.40000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.4560000000000004,"5min":4.4560000000000004,"15min":4.4560000000000004},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1870000000000001,"15min":1.3600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":3.7530000000000001,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.93999999999999995,"15min":0.94599999999999995},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.33300000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.1699999999999999,"15min":1.1619999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":8.3320000000000007,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.0449999999999999,"15min":1.0269999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.1920000000000002,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.38300000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.258,"15min":1.458},"min":{"1min":0.48499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.8090000000000002,"5min":6.851,"15min":6.851},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.319,"15min":1.3109999999999999},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.8890000000000002,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":0.60999999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.137,"15min":1.204},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.1459999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.276,"5min":1.0429999999999999,"15min":1.1140000000000001},"min":{"1min":0.47099999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.44700000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.3640000000000001,"15min":1.3560000000000001},"min":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.5810000000000004,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.823,"5min":1.3979999999999999,"15min":1.401},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":3.863}]},{"osd":10,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3480000000000001,"15min":1.52},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":4.8499999999999996,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.77,"5min":1.361,"15min":1.343},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.9320000000000004,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":0.70599999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.2789999999999999,"15min":1.296},"min":{"1min":0.58599999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.6399999999999997,"5min":4.6399999999999997,"15min":4.6399999999999997},"last":0.746},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.3899999999999999,"15min":1.53},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":8.4540000000000006,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.75800000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208380,"kb_used_data":907060,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92536772,"statfs":{"total":95995035648,"available":94757654528,"internally_reserved":0,"allocated":928829440,"data_stored":928477661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.97699999999999998,"15min":1.387},"min":{"1min":0.33100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.3530000000000002,"5min":5.2539999999999996,"15min":90.597999999999999},"last":1.506},{"interface":"front","average":{"1min":1.498,"5min":1.085,"15min":1.577},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":12.236000000000001,"5min":12.236000000000001,"15min":90.676000000000002},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":0.91300000000000003,"15min":0.94799999999999995},"min":{"1min":0.31,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":8.0530000000000008},"last":5.577},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.0289999999999999,"15min":1.1240000000000001},"min":{"1min":0.29399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":4.1109999999999998,"5min":5.5709999999999997,"15min":7.452},"last":0.95999999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":1.714,"5min":1.2090000000000001,"15min":1.129},"min":{"1min":0.38100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":12.409000000000001,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.434},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.089,"15min":1.069},"min":{"1min":0.38400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.5629999999999997,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.98399999999999999,"15min":0.93500000000000005},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.9950000000000001,"5min":5.5999999999999996,"15min":5.5999999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1419999999999999,"15min":1.0920000000000001},"min":{"1min":0.41099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.0309999999999997,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83599999999999997,"15min":1.0429999999999999},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0960000000000001,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.333,"5min":1.1339999999999999,"15min":1.0860000000000001},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.0289999999999999,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":0.31900000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.071,"15min":1.1539999999999999},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":9.0879999999999992},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.89700000000000002,"15min":1.1890000000000001},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.296,"5min":3.3799999999999999,"15min":7.6689999999999996},"last":0.45800000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.97699999999999998,"15min":1.204},"min":{"1min":0.44700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.0439999999999996,"5min":4.0439999999999996,"15min":7.8650000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.2370000000000001,"15min":1.2310000000000001},"min":{"1min":0.53700000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.7250000000000001,"5min":5.6699999999999999,"15min":5.6699999999999999},"last":0.79200000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.054,"15min":1.429},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.417,"5min":3.2650000000000001,"15min":34.112000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.361,"15min":1.6639999999999999},"min":{"1min":0.502,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":3.9740000000000002,"5min":5.6420000000000003,"15min":34.590000000000003},"last":0.83299999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.3740000000000001,"15min":1.3759999999999999},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.0949999999999998,"5min":5.6970000000000001,"15min":6.0899999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.073,"5min":1.163,"15min":1.262},"min":{"1min":0.45500000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.8300000000000001,"5min":4.46,"15min":8.5299999999999994},"last":0.41199999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.097,"15min":1.4910000000000001},"min":{"1min":0.52800000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.1640000000000001,"5min":3.2320000000000002,"15min":4.6500000000000004},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.512},"min":{"1min":0.59499999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":4.0570000000000004,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.86599999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1247360,"kb_used_data":933752,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":92497792,"statfs":{"total":95995035648,"available":94717739008,"internally_reserved":0,"allocated":956162048,"data_stored":955806197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.83799999999999997,"15min":2.7349999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":565.34500000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.113,"5min":0.93799999999999994,"15min":2.8140000000000001},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":4.5830000000000002,"5min":4.5830000000000002,"15min":565.98400000000004},"last":0.53200000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3839999999999999,"15min":1.143},"min":{"1min":0.314,"5min":0.314,"15min":0.30499999999999999},"max":{"1min":6.0709999999999997,"5min":40.021999999999998,"15min":40.021999999999998},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.478,"15min":1.1859999999999999},"min":{"1min":0.28599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.5069999999999997,"5min":40.225000000000001,"15min":40.225000000000001},"last":4.5750000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.3420000000000001,"15min":1.3129999999999999},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.5289999999999999,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":4.1429999999999998},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.80600000000000005,"15min":0.754},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.1619999999999999,"5min":5.1619999999999999,"15min":5.1619999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.331,"15min":1.4239999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.661,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.5329999999999999,"15min":1.46},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":6.4610000000000003,"5min":42.430999999999997,"15min":74.427000000000007},"last":5.0369999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.038,"15min":1.306},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":74.105000000000004},"last":4.3159999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.0580000000000001,"15min":1.423},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":75.013999999999996},"last":0.45700000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.113,"15min":1.0669999999999999},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":9.7710000000000008,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.1779999999999999,"15min":1.171},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.3979999999999997,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":4.1790000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.95799999999999996,"15min":1.181},"min":{"1min":0.432,"5min":0.41699999999999998,"15min":0.40200000000000002},"max":{"1min":2.524,"5min":4.0069999999999997,"15min":5.9509999999999996},"last":0.62},{"interface":"front","average":{"1min":1.202,"5min":1.119,"15min":1.2430000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.6159999999999997,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":0.72899999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.373,"15min":3.2930000000000001},"min":{"1min":0.48399999999999999,"5min":0.46899999999999997,"15min":0.439},"max":{"1min":6.2939999999999996,"5min":6.2939999999999996,"15min":449.89499999999998},"last":4.2240000000000002},{"interface":"front","average":{"1min":1.538,"5min":1.161,"15min":3.2599999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":9.6649999999999991,"5min":9.6649999999999991,"15min":448.70800000000003},"last":0.77700000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.016,"15min":1.258},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.6120000000000001,"5min":4.0410000000000004,"15min":5.9210000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":2.016,"5min":1.431,"15min":1.3540000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":15.006,"5min":15.006,"15min":15.006},"last":4.2549999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.131,"15min":1.21},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.7789999999999999,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.712,"5min":1.4419999999999999,"15min":1.355},"min":{"1min":0.57699999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":4.2850000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1108475904,"data_stored":1108475904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":941686784,"data_stored":941686784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":703135744,"data_stored":703135744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225116160,"data_stored":225116160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":914554880,"data_stored":914554880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":955453440,"data_stored":955449368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":668270592,"data_stored":668270592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":232521728,"data_stored":232521728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":443088896,"data_stored":443088896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":472121344,"data_stored":472121344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":501743616,"data_stored":501743616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":257953792,"data_stored":257949720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:27:48.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T16:27:48.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:48.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:48.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:48.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:48.465 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:48.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:48.592 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:48.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:48.723 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:48 smithi169 bash[19604]: cluster 2024-02-12T16:27:47.683870+0000 mon.a (mon.0) 2159 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:48 smithi169 bash[19604]: audit 2024-02-12T16:27:47.696939+0000 mon.a (mon.0) 2160 : audit [INF] from='client.26131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:27:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:48 smithi169 bash[19604]: cluster 2024-02-12T16:27:47.697016+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e609: 12 total, 12 up, 12 in 2024-02-12T16:27:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:48 smithi169 bash[19604]: cluster 2024-02-12T16:27:47.987590+0000 mgr.y (mgr.14146) 1286 : cluster [DBG] pgmap v1784: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 5546/107654 objects misplaced (5.152%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:27:48.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:48.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:48.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:48.981 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:48.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:49.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:49.091 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:48 smithi123 bash[19119]: cluster 2024-02-12T16:27:47.683870+0000 mon.a (mon.0) 2159 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:49.091 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:48 smithi123 bash[19119]: audit 2024-02-12T16:27:47.696939+0000 mon.a (mon.0) 2160 : audit [INF] from='client.26131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:27:49.092 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:48 smithi123 bash[19119]: cluster 2024-02-12T16:27:47.697016+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e609: 12 total, 12 up, 12 in 2024-02-12T16:27:49.092 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:48 smithi123 bash[19119]: cluster 2024-02-12T16:27:47.987590+0000 mgr.y (mgr.14146) 1286 : cluster [DBG] pgmap v1784: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 5546/107654 objects misplaced (5.152%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:27:49.108 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:49.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:49.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:48 smithi121 bash[18489]: cluster 2024-02-12T16:27:47.683870+0000 mon.a (mon.0) 2159 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:27:49.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:48 smithi121 bash[18489]: audit 2024-02-12T16:27:47.696939+0000 mon.a (mon.0) 2160 : audit [INF] from='client.26131 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:27:49.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:48 smithi121 bash[18489]: cluster 2024-02-12T16:27:47.697016+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e609: 12 total, 12 up, 12 in 2024-02-12T16:27:49.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:48 smithi121 bash[18489]: cluster 2024-02-12T16:27:47.987590+0000 mgr.y (mgr.14146) 1286 : cluster [DBG] pgmap v1784: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 5546/107654 objects misplaced (5.152%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:27:49.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:49.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:49.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:49.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:49.403 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:49.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:49.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:49.532 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:49.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:49.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:49.663 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:49.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:49.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:49.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:49.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:49.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:49.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:49.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:50.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:50.049 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:50.152 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:50.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:50.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:50.308 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:50.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:50.443 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:50.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:50.578 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:50.683 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:50.715 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:50.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:50.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:50.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:27:50.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:50.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:51.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:51.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:51 smithi121 bash[18489]: cluster 2024-02-12T16:27:49.988071+0000 mgr.y (mgr.14146) 1287 : cluster [DBG] pgmap v1785: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 5546/105780 objects misplaced (5.243%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:51.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:27:51.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:51.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:51.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:51.278 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:51 smithi123 bash[19119]: cluster 2024-02-12T16:27:49.988071+0000 mgr.y (mgr.14146) 1287 : cluster [DBG] pgmap v1785: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 5546/105780 objects misplaced (5.243%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:51.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:51.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:51.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:51.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:51 smithi169 bash[19604]: cluster 2024-02-12T16:27:49.988071+0000 mgr.y (mgr.14146) 1287 : cluster [DBG] pgmap v1785: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 5546/105780 objects misplaced (5.243%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:51.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:51.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:51.568 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:27:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:27:51.569 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-02-12T16:27:51.569 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:27:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:51.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:51.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:51.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:51.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:51.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:51.841 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:51.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:51.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:51.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:51.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:52.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:52.104 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:52.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:52.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:52.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:52.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:52.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:52.374 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:52.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:52.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-12T16:27:52.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:52.516 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:52.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:52.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:52.664 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:52.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:52.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:52.799 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:52.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:52.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:52.926 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:52.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:53.016 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:53.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:53.056 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:53.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:53.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:53.193 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:53.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:53.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:53 smithi169 bash[19604]: cluster 2024-02-12T16:27:51.988561+0000 mgr.y (mgr.14146) 1288 : cluster [DBG] pgmap v1786: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 4861/98358 objects misplaced (4.942%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:53.299 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:53.325 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:53.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:53 smithi123 bash[19119]: cluster 2024-02-12T16:27:51.988561+0000 mgr.y (mgr.14146) 1288 : cluster [DBG] pgmap v1786: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 4861/98358 objects misplaced (4.942%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:53 smithi121 bash[18489]: cluster 2024-02-12T16:27:51.988561+0000 mgr.y (mgr.14146) 1288 : cluster [DBG] pgmap v1786: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 4861/98358 objects misplaced (4.942%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:53.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:53.475 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:53.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:53.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:53.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:53.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:53.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:53.740 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:53.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:53.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:53.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:53.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:53.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:54.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:54.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:54.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:54.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:54.264 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:54.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:54.374 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:54 smithi121 bash[18489]: cluster 2024-02-12T16:27:53.829466+0000 osd.10 (osd.10) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:54.374 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:54 smithi121 bash[18489]: cluster 2024-02-12T16:27:53.829520+0000 osd.10 (osd.10) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:54.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:54 smithi123 bash[19119]: cluster 2024-02-12T16:27:53.829466+0000 osd.10 (osd.10) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:54.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:54 smithi123 bash[19119]: cluster 2024-02-12T16:27:53.829520+0000 osd.10 (osd.10) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:54.401 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:54.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:54.513 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:54 smithi169 bash[19604]: cluster 2024-02-12T16:27:53.829466+0000 osd.10 (osd.10) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:27:54.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:54 smithi169 bash[19604]: cluster 2024-02-12T16:27:53.829520+0000 osd.10 (osd.10) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:27:54.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:54.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:54.657 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:54.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:54.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:54.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:54.924 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:54.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:55.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:55.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:55.050 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:55.050 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:27:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:55.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:55.161 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:55 smithi123 bash[19119]: cluster 2024-02-12T16:27:53.990774+0000 mgr.y (mgr.14146) 1289 : cluster [DBG] pgmap v1787: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 4861/95638 objects misplaced (5.083%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:55.161 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:55 smithi123 bash[19119]: cluster 2024-02-12T16:27:54.060007+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e610: 12 total, 12 up, 12 in 2024-02-12T16:27:55.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:27:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:27:55.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:55.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:55.375 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:55 smithi121 bash[18489]: cluster 2024-02-12T16:27:53.990774+0000 mgr.y (mgr.14146) 1289 : cluster [DBG] pgmap v1787: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 4861/95638 objects misplaced (5.083%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:55.376 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:55 smithi121 bash[18489]: cluster 2024-02-12T16:27:54.060007+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e610: 12 total, 12 up, 12 in 2024-02-12T16:27:55.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:55.459 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:55 smithi169 bash[19604]: cluster 2024-02-12T16:27:53.990774+0000 mgr.y (mgr.14146) 1289 : cluster [DBG] pgmap v1787: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 4861/95638 objects misplaced (5.083%); 1.4 MiB/s, 23 objects/s recovering 2024-02-12T16:27:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:55 smithi169 bash[19604]: cluster 2024-02-12T16:27:54.060007+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e610: 12 total, 12 up, 12 in 2024-02-12T16:27:55.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:55.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:55.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:55.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:55.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:55.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:55.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:55.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:27:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:55.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:27:56.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:56.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:56.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:56.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:56.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:27:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:56.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:27:56.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:56.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:56.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:56.501 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:56.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:56.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:56.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:56.675 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:56.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:56.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:56.822 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:56.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:56.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:56.933 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:56 smithi121 bash[18489]: cluster 2024-02-12T16:27:55.991205+0000 mgr.y (mgr.14146) 1290 : cluster [DBG] pgmap v1789: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 975 op/s; 4518/91470 objects misplaced (4.939%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:27:56.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:56.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:56 smithi169 bash[19604]: cluster 2024-02-12T16:27:55.991205+0000 mgr.y (mgr.14146) 1290 : cluster [DBG] pgmap v1789: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 975 op/s; 4518/91470 objects misplaced (4.939%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:27:57.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:57.073 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:56 smithi123 bash[19119]: cluster 2024-02-12T16:27:55.991205+0000 mgr.y (mgr.14146) 1290 : cluster [DBG] pgmap v1789: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 975 op/s; 4518/91470 objects misplaced (4.939%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:27:57.098 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:57.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:57.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:57.226 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:57.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:57.247 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:27:57.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:57.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:57.361 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:57.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:57.490 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:27:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:57.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:27:57.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:57.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:57.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:57.719 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:57.743 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:57.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:57.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:57.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:57.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:57.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:58.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:58.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:58.128 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:58.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:58.255 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:58.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:58.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:58.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:58.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:58.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:58.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:58.778 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:27:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:27:58.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:27:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:58.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:27:59.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:59.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:27:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:59.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:27:59.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:27:59 smithi121 bash[18489]: cluster 2024-02-12T16:27:57.991672+0000 mgr.y (mgr.14146) 1291 : cluster [DBG] pgmap v1790: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 4226/86376 objects misplaced (4.893%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:59.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:59.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:27:59 smithi169 bash[19604]: cluster 2024-02-12T16:27:57.991672+0000 mgr.y (mgr.14146) 1291 : cluster [DBG] pgmap v1790: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 4226/86376 objects misplaced (4.893%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:59.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:59.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:59.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:59.343 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:27:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:59.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:27:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:27:59 smithi123 bash[19119]: cluster 2024-02-12T16:27:57.991672+0000 mgr.y (mgr.14146) 1291 : cluster [DBG] pgmap v1790: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 4226/86376 objects misplaced (4.893%); 1.4 MiB/s, 22 objects/s recovering 2024-02-12T16:27:59.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:59.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:27:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:59.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:27:59.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:59.608 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:27:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:59.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:27:59.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:59.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:27:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:59.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:27:59.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:59.880 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:27:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:27:59.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:27:59.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:00.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:00.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:00.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:00 smithi123 bash[19119]: cluster 2024-02-12T16:27:58.531416+0000 osd.6 (osd.6) 112 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:00 smithi123 bash[19119]: cluster 2024-02-12T16:27:58.531471+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:00.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:00 smithi123 bash[19119]: cluster 2024-02-12T16:27:59.060344+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e611: 12 total, 12 up, 12 in 2024-02-12T16:28:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:00.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:00.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:00.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:00 smithi121 bash[18489]: cluster 2024-02-12T16:27:58.531416+0000 osd.6 (osd.6) 112 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:00 smithi121 bash[18489]: cluster 2024-02-12T16:27:58.531471+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:00 smithi121 bash[18489]: cluster 2024-02-12T16:27:59.060344+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e611: 12 total, 12 up, 12 in 2024-02-12T16:28:00.415 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:00.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:00.526 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:00 smithi169 bash[19604]: cluster 2024-02-12T16:27:58.531416+0000 osd.6 (osd.6) 112 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:00.526 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:00 smithi169 bash[19604]: cluster 2024-02-12T16:27:58.531471+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:00.526 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:00 smithi169 bash[19604]: cluster 2024-02-12T16:27:59.060344+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e611: 12 total, 12 up, 12 in 2024-02-12T16:28:00.555 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:00.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:00.689 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:00.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:00.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:00.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:00.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:00.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:01.002 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:28:01.002 INFO:teuthology.orchestra.run.smithi121.stdout:{"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2277,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":611,"num_osds":12,"num_up_osds":12,"osd_up_since":1707754981,"num_in_osds":12,"osd_in_since":1707754767,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":35},{"state_name":"active+remapped+backfilling","count":2}],"num_pgs":37,"num_pools":2,"num_objects":42250,"data_bytes":2768830488,"bytes_used":10605092864,"bytes_avail":1141335334912,"bytes_total":1151940427776,"misplaced_objects":4226,"misplaced_total":84500,"misplaced_ratio":0.050011834319526625,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":917089,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":7340032,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":865},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.121:8443/"}},"servicemap":{"epoch":52,"modified":"2024-02-12T16:26:53.974601+0000","services":{}},"progress_events":{}} 2024-02-12T16:28:01.073 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:01 smithi169 bash[19604]: cluster 2024-02-12T16:27:59.630392+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:28:01.074 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:01 smithi169 bash[19604]: cluster 2024-02-12T16:27:59.992117+0000 mgr.y (mgr.14146) 1292 : cluster [DBG] pgmap v1792: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.6 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 4226/84500 objects misplaced (5.001%); 896 KiB/s, 13 objects/s recovering 2024-02-12T16:28:01.074 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:01 smithi169 bash[19604]: audit 2024-02-12T16:28:00.997553+0000 mon.a (mon.0) 2164 : audit [DBG] from='client.? v1:172.21.15.121:0/304229217' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:28:01.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:01.111 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:01.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:01.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:01.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:01 smithi121 bash[18489]: cluster 2024-02-12T16:27:59.630392+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:28:01.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:01 smithi121 bash[18489]: cluster 2024-02-12T16:27:59.992117+0000 mgr.y (mgr.14146) 1292 : cluster [DBG] pgmap v1792: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.6 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 4226/84500 objects misplaced (5.001%); 896 KiB/s, 13 objects/s recovering 2024-02-12T16:28:01.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:01 smithi121 bash[18489]: audit 2024-02-12T16:28:00.997553+0000 mon.a (mon.0) 2164 : audit [DBG] from='client.? v1:172.21.15.121:0/304229217' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:28:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:01.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:01.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:01.373 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:01 smithi123 bash[19119]: cluster 2024-02-12T16:27:59.630392+0000 osd.4 (osd.4) 102 : cluster [DBG] 5.1 scrub starts 2024-02-12T16:28:01.373 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:01 smithi123 bash[19119]: cluster 2024-02-12T16:27:59.992117+0000 mgr.y (mgr.14146) 1292 : cluster [DBG] pgmap v1792: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.6 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 4226/84500 objects misplaced (5.001%); 896 KiB/s, 13 objects/s recovering 2024-02-12T16:28:01.374 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:01 smithi123 bash[19119]: audit 2024-02-12T16:28:00.997553+0000 mon.a (mon.0) 2164 : audit [DBG] from='client.? v1:172.21.15.121:0/304229217' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:28:01.394 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:01.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:01.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:01.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:01.573 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:28:01.573 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:28:01.573 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:28:01.573 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:28:01.573 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:28:01.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:01.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:01.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:01.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:01.801 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:01.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:01.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:01.930 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:01.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:02.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:02.070 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:02.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:02.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:02.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:02.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:02.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:02 smithi169 bash[19604]: cluster 2024-02-12T16:28:00.438834+0000 osd.4 (osd.4) 103 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:28:02.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:02.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:02.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:02.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:02 smithi123 bash[19119]: cluster 2024-02-12T16:28:00.438834+0000 osd.4 (osd.4) 103 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:28:02.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:02 smithi121 bash[18489]: cluster 2024-02-12T16:28:00.438834+0000 osd.4 (osd.4) 103 : cluster [DBG] 5.1 scrub ok 2024-02-12T16:28:02.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-12T16:28:02.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:02.490 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:02.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:02.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:02.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:02.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:02.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:02.754 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:02.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:02.834 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-02-12T16:28:02.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:28:02.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:02.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:02.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:02.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:03.017 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:03.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:03.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:03.151 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:03.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:03.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:03 smithi121 bash[18489]: cluster 2024-02-12T16:28:01.992638+0000 mgr.y (mgr.14146) 1293 : cluster [DBG] pgmap v1793: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3612/77596 objects misplaced (4.655%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:28:03.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:03 smithi121 bash[18489]: cluster 2024-02-12T16:28:02.060146+0000 osd.11 (osd.11) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:03 smithi121 bash[18489]: cluster 2024-02-12T16:28:02.060208+0000 osd.11 (osd.11) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:03 smithi121 bash[18489]: cluster 2024-02-12T16:28:02.075576+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e612: 12 total, 12 up, 12 in 2024-02-12T16:28:03.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:03.287 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:03.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:03 smithi169 bash[19604]: cluster 2024-02-12T16:28:01.992638+0000 mgr.y (mgr.14146) 1293 : cluster [DBG] pgmap v1793: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3612/77596 objects misplaced (4.655%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:28:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:03 smithi169 bash[19604]: cluster 2024-02-12T16:28:02.060146+0000 osd.11 (osd.11) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:03 smithi169 bash[19604]: cluster 2024-02-12T16:28:02.060208+0000 osd.11 (osd.11) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:03 smithi169 bash[19604]: cluster 2024-02-12T16:28:02.075576+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e612: 12 total, 12 up, 12 in 2024-02-12T16:28:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:03 smithi123 bash[19119]: cluster 2024-02-12T16:28:01.992638+0000 mgr.y (mgr.14146) 1293 : cluster [DBG] pgmap v1793: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3612/77596 objects misplaced (4.655%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:28:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:03 smithi123 bash[19119]: cluster 2024-02-12T16:28:02.060146+0000 osd.11 (osd.11) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:03 smithi123 bash[19119]: cluster 2024-02-12T16:28:02.060208+0000 osd.11 (osd.11) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:03 smithi123 bash[19119]: cluster 2024-02-12T16:28:02.075576+0000 mon.a (mon.0) 2165 : cluster [DBG] osdmap e612: 12 total, 12 up, 12 in 2024-02-12T16:28:03.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:03.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:03.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:03.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:03.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:03.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:03.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:03.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:03.758 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:28:03.815 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:28:03.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:03.871 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:28:03.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:03.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:03.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:03.916 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:28:03.955 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:28:03.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:03.989 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:28:04.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:04.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:04.023 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:28:04.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:28:04.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:04.106 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:28:04.143 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:28:04.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:04.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:04.174 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:28:04.210 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.220 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.221 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.221 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.221 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.221 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:28:04.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:04.265 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:04 smithi169 bash[19604]: audit 2024-02-12T16:28:03.716508+0000 mon.b (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.121:0/253200606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:04.284 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:04.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:04 smithi123 bash[19119]: audit 2024-02-12T16:28:03.716508+0000 mon.b (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.121:0/253200606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:04.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:04.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:04 smithi121 bash[18489]: audit 2024-02-12T16:28:03.716508+0000 mon.b (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.121:0/253200606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:04.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:04.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:04.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:04.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:04.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:04.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:04.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:04.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:04.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:04.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:04.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:05.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:05.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:05 smithi121 bash[18489]: cluster 2024-02-12T16:28:03.993094+0000 mgr.y (mgr.14146) 1294 : cluster [DBG] pgmap v1795: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 3612/74784 objects misplaced (4.830%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:28:05.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:05 smithi121 bash[18489]: audit 2024-02-12T16:28:04.094557+0000 mon.a (mon.0) 2166 : audit [INF] from='client.26149 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:05.089 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:05 smithi123 bash[19119]: cluster 2024-02-12T16:28:03.993094+0000 mgr.y (mgr.14146) 1294 : cluster [DBG] pgmap v1795: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 3612/74784 objects misplaced (4.830%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:28:05.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:05 smithi123 bash[19119]: audit 2024-02-12T16:28:04.094557+0000 mon.a (mon.0) 2166 : audit [INF] from='client.26149 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:05.090 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:05 smithi123 bash[19119]: audit 2024-02-12T16:28:04.095929+0000 mon.c (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.121:0/1994154406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:05.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:28:05.143 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:05.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:05.270 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:05.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:05.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:05 smithi121 bash[18489]: audit 2024-02-12T16:28:04.095929+0000 mon.c (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.121:0/1994154406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:05.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:05.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:05.546 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:05 smithi169 bash[19604]: cluster 2024-02-12T16:28:03.993094+0000 mgr.y (mgr.14146) 1294 : cluster [DBG] pgmap v1795: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 3612/74784 objects misplaced (4.830%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:28:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:05 smithi169 bash[19604]: audit 2024-02-12T16:28:04.094557+0000 mon.a (mon.0) 2166 : audit [INF] from='client.26149 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:05 smithi169 bash[19604]: audit 2024-02-12T16:28:04.095929+0000 mon.c (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.121:0/1994154406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:05.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:05.650 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:05.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:05.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:05.806 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:05.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:05.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:06.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:06.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:06.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:06.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:06.227 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:06.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:06.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:06.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:06 smithi123 bash[19119]: cluster 2024-02-12T16:28:05.082909+0000 mon.a (mon.0) 2167 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:28:06.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:06 smithi123 bash[19119]: cluster 2024-02-12T16:28:05.082937+0000 mon.a (mon.0) 2168 : cluster [INF] Cluster is now healthy 2024-02-12T16:28:06.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:06 smithi123 bash[19119]: audit 2024-02-12T16:28:05.095174+0000 mon.a (mon.0) 2169 : audit [INF] from='client.26149 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:28:06.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:06 smithi123 bash[19119]: cluster 2024-02-12T16:28:05.095271+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e613: 12 total, 12 up, 12 in 2024-02-12T16:28:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:06.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:06 smithi121 bash[18489]: cluster 2024-02-12T16:28:05.082909+0000 mon.a (mon.0) 2167 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:28:06.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:06 smithi121 bash[18489]: cluster 2024-02-12T16:28:05.082937+0000 mon.a (mon.0) 2168 : cluster [INF] Cluster is now healthy 2024-02-12T16:28:06.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:06 smithi121 bash[18489]: audit 2024-02-12T16:28:05.095174+0000 mon.a (mon.0) 2169 : audit [INF] from='client.26149 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:28:06.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:06 smithi121 bash[18489]: cluster 2024-02-12T16:28:05.095271+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e613: 12 total, 12 up, 12 in 2024-02-12T16:28:06.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:06.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:06.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:06.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:06 smithi169 bash[19604]: cluster 2024-02-12T16:28:05.082909+0000 mon.a (mon.0) 2167 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:28:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:06 smithi169 bash[19604]: cluster 2024-02-12T16:28:05.082937+0000 mon.a (mon.0) 2168 : cluster [INF] Cluster is now healthy 2024-02-12T16:28:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:06 smithi169 bash[19604]: audit 2024-02-12T16:28:05.095174+0000 mon.a (mon.0) 2169 : audit [INF] from='client.26149 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:28:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:06 smithi169 bash[19604]: cluster 2024-02-12T16:28:05.095271+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e613: 12 total, 12 up, 12 in 2024-02-12T16:28:06.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:06.640 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:06.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:06.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:06.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:06.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:06.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:06.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:07.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:07.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:07.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:07.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:07 smithi123 bash[19119]: cluster 2024-02-12T16:28:05.993540+0000 mgr.y (mgr.14146) 1295 : cluster [DBG] pgmap v1797: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 3288/70800 objects misplaced (4.644%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:28:07.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:07.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:07.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:07.254 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:28:07.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:07.331 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:07.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:07 smithi121 bash[18489]: cluster 2024-02-12T16:28:05.993540+0000 mgr.y (mgr.14146) 1295 : cluster [DBG] pgmap v1797: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 3288/70800 objects misplaced (4.644%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:28:07.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:07.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:07.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:07.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:28:07.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:07 smithi169 bash[19604]: cluster 2024-02-12T16:28:05.993540+0000 mgr.y (mgr.14146) 1295 : cluster [DBG] pgmap v1797: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 3288/70800 objects misplaced (4.644%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:28:07.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:07.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:07.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:07.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:07.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:07.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:07.843 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:07.877 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:07.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:07.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:08.006 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:08.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:08.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:08 smithi123 bash[19119]: cluster 2024-02-12T16:28:06.627985+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:08.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:08 smithi123 bash[19119]: cluster 2024-02-12T16:28:06.628059+0000 osd.3 (osd.3) 21 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:08.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:08 smithi123 bash[19119]: cluster 2024-02-12T16:28:06.931899+0000 osd.2 (osd.2) 86 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:28:08.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:08 smithi123 bash[19119]: cluster 2024-02-12T16:28:06.991141+0000 osd.2 (osd.2) 87 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:28:08.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:08 smithi123 bash[19119]: cluster 2024-02-12T16:28:07.096783+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e614: 12 total, 12 up, 12 in 2024-02-12T16:28:08.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:08.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:28:08.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:08.283 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:08.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:08 smithi121 bash[18489]: cluster 2024-02-12T16:28:06.627985+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:08 smithi121 bash[18489]: cluster 2024-02-12T16:28:06.628059+0000 osd.3 (osd.3) 21 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:08 smithi121 bash[18489]: cluster 2024-02-12T16:28:06.931899+0000 osd.2 (osd.2) 86 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:28:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:08 smithi121 bash[18489]: cluster 2024-02-12T16:28:06.991141+0000 osd.2 (osd.2) 87 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:28:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:08 smithi121 bash[18489]: cluster 2024-02-12T16:28:07.096783+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e614: 12 total, 12 up, 12 in 2024-02-12T16:28:08.412 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:08.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:08.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:08 smithi169 bash[19604]: cluster 2024-02-12T16:28:06.627985+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:08.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:08 smithi169 bash[19604]: cluster 2024-02-12T16:28:06.628059+0000 osd.3 (osd.3) 21 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:08.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:08 smithi169 bash[19604]: cluster 2024-02-12T16:28:06.931899+0000 osd.2 (osd.2) 86 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:28:08.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:08 smithi169 bash[19604]: cluster 2024-02-12T16:28:06.991141+0000 osd.2 (osd.2) 87 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:28:08.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:08 smithi169 bash[19604]: cluster 2024-02-12T16:28:07.096783+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e614: 12 total, 12 up, 12 in 2024-02-12T16:28:08.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:08.654 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:08.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:08.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:08.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:08.810 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:08.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:08.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:08.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:09.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:09.073 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:09.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:09.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:09.201 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:09.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:09.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:09 smithi169 bash[19604]: cluster 2024-02-12T16:28:07.993875+0000 mgr.y (mgr.14146) 1296 : cluster [DBG] pgmap v1799: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 3006/65362 objects misplaced (4.599%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:28:09.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:09.331 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:09.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:09 smithi123 bash[19119]: cluster 2024-02-12T16:28:07.993875+0000 mgr.y (mgr.14146) 1296 : cluster [DBG] pgmap v1799: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 3006/65362 objects misplaced (4.599%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:28:09.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:09 smithi121 bash[18489]: cluster 2024-02-12T16:28:07.993875+0000 mgr.y (mgr.14146) 1296 : cluster [DBG] pgmap v1799: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 2.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 3006/65362 objects misplaced (4.599%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:28:09.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:09.479 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:09.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:09.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:09.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:09.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:09.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:09.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:09.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:09.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:09.930 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:09.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:10.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:10.057 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:10.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:10.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:10.184 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:10.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:10.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:10.356 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:10.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:10.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:10.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:10.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:10.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:10.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:10.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:10.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:10.773 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:10.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:10.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:10.905 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:10.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:11.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:11.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:11.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:11 smithi121 bash[18489]: cluster 2024-02-12T16:28:09.994312+0000 mgr.y (mgr.14146) 1297 : cluster [DBG] pgmap v1800: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3006/63366 objects misplaced (4.744%); 945 KiB/s, 14 objects/s recovering 2024-02-12T16:28:11.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:11.192 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:11.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:11.303 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:11 smithi123 bash[19119]: cluster 2024-02-12T16:28:09.994312+0000 mgr.y (mgr.14146) 1297 : cluster [DBG] pgmap v1800: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3006/63366 objects misplaced (4.744%); 945 KiB/s, 14 objects/s recovering 2024-02-12T16:28:11.321 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:11.362 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:28:11.362 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:28:11.416 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:11 smithi169 bash[19604]: cluster 2024-02-12T16:28:09.994312+0000 mgr.y (mgr.14146) 1297 : cluster [DBG] pgmap v1800: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 3006/63366 objects misplaced (4.744%); 945 KiB/s, 14 objects/s recovering 2024-02-12T16:28:11.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:11.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:11.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:11.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:11.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:11.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:11.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:11.842 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:11.949 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:11.974 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:11.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:12.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:12.107 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:12.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:12.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:12.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-12T16:28:12.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:12.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:12.252 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:12.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:12.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:12 smithi123 bash[19119]: cluster 2024-02-12T16:28:11.859073+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:12.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:12 smithi123 bash[19119]: cluster 2024-02-12T16:28:11.859128+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:12.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:12.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:12 smithi121 bash[18489]: cluster 2024-02-12T16:28:11.859073+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:12.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:12 smithi121 bash[18489]: cluster 2024-02-12T16:28:11.859128+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:12.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:12.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:12 smithi169 bash[19604]: cluster 2024-02-12T16:28:11.859073+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:12.499 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:12 smithi169 bash[19604]: cluster 2024-02-12T16:28:11.859128+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:12.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:12.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:12.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:12.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:12.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:12.722 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:12.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:12.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:12.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:12.905 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:12.936 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:12.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:13.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:13.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:13.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:13.194 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:13.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: cluster 2024-02-12T16:28:11.994797+0000 mgr.y (mgr.14146) 1298 : cluster [DBG] pgmap v1801: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 2383/55858 objects misplaced (4.266%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: cluster 2024-02-12T16:28:12.144068+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e615: 12 total, 12 up, 12 in 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.066159+0000 mon.a (mon.0) 2173 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08470490850563105]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.066486+0000 mon.a (mon.0) 2174 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0885548160725244]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.066743+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08773774383329531]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.067141+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09336049366898387]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.067431+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08801941815189245]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.067612+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0852139185604031]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.067918+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08647162638424509]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.068185+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08645641321985659]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.068457+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0924075218863897]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.068720+0000 mon.a (mon.0) 2182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08644120005546808]}]: dispatch 2024-02-12T16:28:13.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.068940+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08508456909798191]}]: dispatch 2024-02-12T16:28:13.306 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:13 smithi169 bash[19604]: audit 2024-02-12T16:28:13.069163+0000 mon.a (mon.0) 2184 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08309375728207852]}]: dispatch 2024-02-12T16:28:13.322 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:13.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:13.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:13.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:13.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:13.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: cluster 2024-02-12T16:28:11.994797+0000 mgr.y (mgr.14146) 1298 : cluster [DBG] pgmap v1801: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 2383/55858 objects misplaced (4.266%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: cluster 2024-02-12T16:28:12.144068+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e615: 12 total, 12 up, 12 in 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.066159+0000 mon.a (mon.0) 2173 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08470490850563105]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.066486+0000 mon.a (mon.0) 2174 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0885548160725244]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.066743+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08773774383329531]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.067141+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09336049366898387]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.067431+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08801941815189245]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.067612+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0852139185604031]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.067918+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08647162638424509]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.068185+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08645641321985659]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.068457+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0924075218863897]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.068720+0000 mon.a (mon.0) 2182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08644120005546808]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.068940+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08508456909798191]}]: dispatch 2024-02-12T16:28:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:13 smithi123 bash[19119]: audit 2024-02-12T16:28:13.069163+0000 mon.a (mon.0) 2184 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08309375728207852]}]: dispatch 2024-02-12T16:28:13.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:13.695 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: cluster 2024-02-12T16:28:11.994797+0000 mgr.y (mgr.14146) 1298 : cluster [DBG] pgmap v1801: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 2383/55858 objects misplaced (4.266%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:28:13.695 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: cluster 2024-02-12T16:28:12.144068+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e615: 12 total, 12 up, 12 in 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.066159+0000 mon.a (mon.0) 2173 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08470490850563105]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.066486+0000 mon.a (mon.0) 2174 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0885548160725244]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.066743+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08773774383329531]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.067141+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09336049366898387]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.067431+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08801941815189245]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.067612+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0852139185604031]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.067918+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08647162638424509]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.068185+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08645641321985659]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.068457+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0924075218863897]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.068720+0000 mon.a (mon.0) 2182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08644120005546808]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.068940+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08508456909798191]}]: dispatch 2024-02-12T16:28:13.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:13 smithi121 bash[18489]: audit 2024-02-12T16:28:13.069163+0000 mon.a (mon.0) 2184 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08309375728207852]}]: dispatch 2024-02-12T16:28:13.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:13.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:13.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:13.859 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:13.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:13.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:13.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:14.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:14.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:14.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:14.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:14.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145024+0000 mon.a (mon.0) 2185 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08470490850563105]}]': finished 2024-02-12T16:28:14.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145078+0000 mon.a (mon.0) 2186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0885548160725244]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145165+0000 mon.a (mon.0) 2187 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08773774383329531]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145269+0000 mon.a (mon.0) 2188 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09336049366898387]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145347+0000 mon.a (mon.0) 2189 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08801941815189245]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145447+0000 mon.a (mon.0) 2190 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0852139185604031]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145533+0000 mon.a (mon.0) 2191 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08647162638424509]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145630+0000 mon.a (mon.0) 2192 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08645641321985659]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145722+0000 mon.a (mon.0) 2193 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0924075218863897]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145793+0000 mon.a (mon.0) 2194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08644120005546808]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145869+0000 mon.a (mon.0) 2195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08508456909798191]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: audit 2024-02-12T16:28:13.145942+0000 mon.a (mon.0) 2196 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08309375728207852]}]': finished 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: cluster 2024-02-12T16:28:13.145992+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e616: 12 total, 12 up, 12 in 2024-02-12T16:28:14.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:14 smithi169 bash[19604]: cluster 2024-02-12T16:28:14.144021+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e617: 12 total, 12 up, 12 in 2024-02-12T16:28:14.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:14.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:14.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:14.395 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:14.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:14.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:14.533 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:14.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145024+0000 mon.a (mon.0) 2185 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08470490850563105]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145078+0000 mon.a (mon.0) 2186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0885548160725244]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145165+0000 mon.a (mon.0) 2187 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08773774383329531]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145269+0000 mon.a (mon.0) 2188 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09336049366898387]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145347+0000 mon.a (mon.0) 2189 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08801941815189245]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145447+0000 mon.a (mon.0) 2190 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0852139185604031]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145533+0000 mon.a (mon.0) 2191 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08647162638424509]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145630+0000 mon.a (mon.0) 2192 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08645641321985659]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145722+0000 mon.a (mon.0) 2193 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0924075218863897]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145793+0000 mon.a (mon.0) 2194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08644120005546808]}]': finished 2024-02-12T16:28:14.554 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145869+0000 mon.a (mon.0) 2195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08508456909798191]}]': finished 2024-02-12T16:28:14.555 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: audit 2024-02-12T16:28:13.145942+0000 mon.a (mon.0) 2196 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08309375728207852]}]': finished 2024-02-12T16:28:14.555 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: cluster 2024-02-12T16:28:13.145992+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e616: 12 total, 12 up, 12 in 2024-02-12T16:28:14.555 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:14 smithi121 bash[18489]: cluster 2024-02-12T16:28:14.144021+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e617: 12 total, 12 up, 12 in 2024-02-12T16:28:14.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145024+0000 mon.a (mon.0) 2185 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08470490850563105]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145078+0000 mon.a (mon.0) 2186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0885548160725244]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145165+0000 mon.a (mon.0) 2187 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08773774383329531]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145269+0000 mon.a (mon.0) 2188 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09336049366898387]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145347+0000 mon.a (mon.0) 2189 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08801941815189245]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145447+0000 mon.a (mon.0) 2190 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0852139185604031]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145533+0000 mon.a (mon.0) 2191 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08647162638424509]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145630+0000 mon.a (mon.0) 2192 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08645641321985659]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145722+0000 mon.a (mon.0) 2193 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0924075218863897]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145793+0000 mon.a (mon.0) 2194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08644120005546808]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145869+0000 mon.a (mon.0) 2195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08508456909798191]}]': finished 2024-02-12T16:28:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: audit 2024-02-12T16:28:13.145942+0000 mon.a (mon.0) 2196 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08309375728207852]}]': finished 2024-02-12T16:28:14.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: cluster 2024-02-12T16:28:13.145992+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e616: 12 total, 12 up, 12 in 2024-02-12T16:28:14.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:14 smithi123 bash[19119]: cluster 2024-02-12T16:28:14.144021+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e617: 12 total, 12 up, 12 in 2024-02-12T16:28:14.664 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:14.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:14.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:14.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:14.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:14.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:14.960 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:14.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:15.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:15.114 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:15.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:15.125 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:28:15.127 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:28:15.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:15.237 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:15 smithi121 bash[18489]: cluster 2024-02-12T16:28:13.995250+0000 mgr.y (mgr.14146) 1299 : cluster [DBG] pgmap v1804: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 2383/52992 objects misplaced (4.497%); 1.1 MiB/s, 16 objects/s recovering 2024-02-12T16:28:15.237 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:15 smithi121 bash[18489]: cluster 2024-02-12T16:28:14.685650+0000 osd.5 (osd.5) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:15.237 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:15 smithi121 bash[18489]: cluster 2024-02-12T16:28:14.685700+0000 osd.5 (osd.5) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:15.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:15.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:15.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:15.356 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:15 smithi123 bash[19119]: cluster 2024-02-12T16:28:13.995250+0000 mgr.y (mgr.14146) 1299 : cluster [DBG] pgmap v1804: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 2383/52992 objects misplaced (4.497%); 1.1 MiB/s, 16 objects/s recovering 2024-02-12T16:28:15.356 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:15 smithi123 bash[19119]: cluster 2024-02-12T16:28:14.685650+0000 osd.5 (osd.5) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:15.356 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:15 smithi123 bash[19119]: cluster 2024-02-12T16:28:14.685700+0000 osd.5 (osd.5) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:15.373 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:15.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:15.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:15.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:15.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:15.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:15 smithi169 bash[19604]: cluster 2024-02-12T16:28:13.995250+0000 mgr.y (mgr.14146) 1299 : cluster [DBG] pgmap v1804: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 2383/52992 objects misplaced (4.497%); 1.1 MiB/s, 16 objects/s recovering 2024-02-12T16:28:15.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:15 smithi169 bash[19604]: cluster 2024-02-12T16:28:14.685650+0000 osd.5 (osd.5) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:15 smithi169 bash[19604]: cluster 2024-02-12T16:28:14.685700+0000 osd.5 (osd.5) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:15.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:15.635 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:15.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:15.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:15.767 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:15.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:15.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:15.897 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:15.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:16.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:16.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:16.135 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:16.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:16 smithi169 bash[19604]: audit 2024-02-12T16:28:15.102522+0000 mgr.y (mgr.14146) 1300 : audit [DBG] from='client.16320 v1:172.21.15.121:0/3382697360' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:16.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:16 smithi169 bash[19604]: cluster 2024-02-12T16:28:15.166922+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e618: 12 total, 12 up, 12 in 2024-02-12T16:28:16.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:16 smithi169 bash[19604]: cluster 2024-02-12T16:28:15.670934+0000 mon.a (mon.0) 2200 : cluster [DBG] osdmap e619: 12 total, 12 up, 12 in 2024-02-12T16:28:16.298 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:16.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:16.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:16.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:16.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:16.539 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:16 smithi121 bash[18489]: audit 2024-02-12T16:28:15.102522+0000 mgr.y (mgr.14146) 1300 : audit [DBG] from='client.16320 v1:172.21.15.121:0/3382697360' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:16.539 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:16 smithi121 bash[18489]: cluster 2024-02-12T16:28:15.166922+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e618: 12 total, 12 up, 12 in 2024-02-12T16:28:16.539 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:16 smithi121 bash[18489]: cluster 2024-02-12T16:28:15.670934+0000 mon.a (mon.0) 2200 : cluster [DBG] osdmap e619: 12 total, 12 up, 12 in 2024-02-12T16:28:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:16 smithi123 bash[19119]: audit 2024-02-12T16:28:15.102522+0000 mgr.y (mgr.14146) 1300 : audit [DBG] from='client.16320 v1:172.21.15.121:0/3382697360' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:16 smithi123 bash[19119]: cluster 2024-02-12T16:28:15.166922+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e618: 12 total, 12 up, 12 in 2024-02-12T16:28:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:16 smithi123 bash[19119]: cluster 2024-02-12T16:28:15.670934+0000 mon.a (mon.0) 2200 : cluster [DBG] osdmap e619: 12 total, 12 up, 12 in 2024-02-12T16:28:16.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:16.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:16.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:16.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:16.960 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1805,"stamp":"2024-02-12T16:28:14.141846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1736376344,"num_objects":26496,"num_object_clones":0,"num_object_copies":52992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2383,"num_objects_unfound":0,"num_objects_dirty":26496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":312280,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2576,"num_bytes_recovered":168820736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8385844,"kb_used_data":3536596,"kb_used_omap":91,"kb_used_meta":4848996,"kb_avail":1116555980,"statfs":{"total":1151940427776,"available":1143353323520,"internally_reserved":0,"allocated":3621474304,"data_stored":3616788150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":4965371920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-405340160,"num_objects":-6185,"num_object_clones":0,"num_object_copies":-12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-623,"num_objects_unfound":0,"num_objects_dirty":-6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":7602176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.148138"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":107,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:06.991326+0000","last_change":"2024-02-12T16:28:06.991266+0000","last_active":"2024-02-12T16:28:06.991326+0000","last_peered":"2024-02-12T16:28:06.991326+0000","last_clean":"2024-02-12T16:28:06.991326+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:28:06.991326+0000","last_undegraded":"2024-02-12T16:28:06.991326+0000","last_fullsized":"2024-02-12T16:28:06.991326+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:06.991147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:28:06.991147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"616'19541","reported_seq":19622,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-02-12T16:28:13.675288+0000","last_change":"2024-02-12T16:25:46.665174+0000","last_active":"2024-02-12T16:28:13.675288+0000","last_peered":"2024-02-12T16:28:13.675288+0000","last_clean":"2024-02-12T16:28:13.675288+0000","last_became_active":"2024-02-12T16:24:09.648523+0000","last_became_peered":"2024-02-12T16:24:09.648523+0000","last_unstale":"2024-02-12T16:28:13.675288+0000","last_undegraded":"2024-02-12T16:28:13.675288+0000","last_fullsized":"2024-02-12T16:28:13.675288+0000","mapping_epoch":547,"log_start":"576'9500","ondisk_log_start":"576'9500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10057","last_scrub_stamp":"2024-02-12T16:25:46.665055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:46.665055+0000","log_size":10041,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97189888,"num_objects":1483,"num_object_clones":0,"num_object_copies":2966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19541,"num_write_kb":672768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":213,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.099044+0000","last_change":"2024-02-12T16:27:58.125119+0000","last_active":"2024-02-12T16:28:05.099044+0000","last_peered":"2024-02-12T16:28:05.099044+0000","last_clean":"2024-02-12T16:28:05.099044+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:28:05.099044+0000","last_undegraded":"2024-02-12T16:28:05.099044+0000","last_fullsized":"2024-02-12T16:28:05.099044+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:40.282662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:40.282662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"614'19529","reported_seq":19618,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:09.328325+0000","last_change":"2024-02-12T16:28:00.438914+0000","last_active":"2024-02-12T16:28:09.328325+0000","last_peered":"2024-02-12T16:28:09.328325+0000","last_clean":"2024-02-12T16:28:09.328325+0000","last_became_active":"2024-02-12T16:24:10.018691+0000","last_became_peered":"2024-02-12T16:24:10.018691+0000","last_unstale":"2024-02-12T16:28:09.328325+0000","last_undegraded":"2024-02-12T16:28:09.328325+0000","last_fullsized":"2024-02-12T16:28:09.328325+0000","mapping_epoch":547,"log_start":"575'9500","ondisk_log_start":"575'9500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"611'18916","last_scrub_stamp":"2024-02-12T16:28:00.438841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:28:00.438841+0000","log_size":10029,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":120389632,"num_objects":1837,"num_object_clones":0,"num_object_copies":3674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19529,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":298,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-02-12T16:28:12.153411+0000","last_change":"2024-02-12T16:24:47.872602+0000","last_active":"2024-02-12T16:28:12.153411+0000","last_peered":"2024-02-12T16:28:12.153411+0000","last_clean":"2024-02-12T16:28:12.153411+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:28:12.153411+0000","last_undegraded":"2024-02-12T16:28:12.153411+0000","last_fullsized":"2024-02-12T16:28:12.153411+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:47.872500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:24:47.872500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"614'19471","reported_seq":22650,"reported_epoch":614,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:28:11.699558+0000","last_change":"2024-02-12T16:25:58.008531+0000","last_active":"2024-02-12T16:28:11.699558+0000","last_peered":"2024-02-12T16:28:11.699558+0000","last_clean":"2024-02-12T16:25:54.194068+0000","last_became_active":"2024-02-12T16:25:57.932697+0000","last_became_peered":"2024-02-12T16:25:57.932697+0000","last_unstale":"2024-02-12T16:28:11.699558+0000","last_undegraded":"2024-02-12T16:28:11.699558+0000","last_fullsized":"2024-02-12T16:28:11.699558+0000","mapping_epoch":581,"log_start":"575'9400","ondisk_log_start":"575'9400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10027","last_scrub_stamp":"2024-02-12T16:25:45.765333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.765333+0000","log_size":10071,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106496000,"num_objects":1625,"num_object_clones":0,"num_object_copies":3250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1093,"num_objects_unfound":0,"num_objects_dirty":1625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19471,"num_write_kb":675072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1625}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":701,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.095274+0000","last_change":"2024-02-12T16:25:07.263767+0000","last_active":"2024-02-12T16:28:05.095274+0000","last_peered":"2024-02-12T16:28:05.095274+0000","last_clean":"2024-02-12T16:28:05.095274+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:28:05.095274+0000","last_undegraded":"2024-02-12T16:28:05.095274+0000","last_fullsized":"2024-02-12T16:28:05.095274+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.263688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.263688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.3","version":"614'19681","reported_seq":19755,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:10.309910+0000","last_change":"2024-02-12T16:24:09.648426+0000","last_active":"2024-02-12T16:28:10.309910+0000","last_peered":"2024-02-12T16:28:10.309910+0000","last_clean":"2024-02-12T16:28:10.309910+0000","last_became_active":"2024-02-12T16:24:09.648139+0000","last_became_peered":"2024-02-12T16:24:09.648139+0000","last_unstale":"2024-02-12T16:28:10.309910+0000","last_undegraded":"2024-02-12T16:28:10.309910+0000","last_fullsized":"2024-02-12T16:28:10.309910+0000","mapping_epoch":547,"log_start":"576'9600","ondisk_log_start":"576'9600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10081,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115408896,"num_objects":1761,"num_object_clones":0,"num_object_copies":3522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19681,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":333,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:10.158538+0000","last_change":"2024-02-12T16:28:10.158935+0000","last_active":"2024-02-12T16:28:10.158538+0000","last_peered":"2024-02-12T16:28:10.158538+0000","last_clean":"2024-02-12T16:28:10.158538+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:28:10.158538+0000","last_undegraded":"2024-02-12T16:28:10.158538+0000","last_fullsized":"2024-02-12T16:28:10.158538+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:45.967340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:45.967340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"614'19138","reported_seq":19217,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:10.232645+0000","last_change":"2024-02-12T16:25:50.727730+0000","last_active":"2024-02-12T16:28:10.232645+0000","last_peered":"2024-02-12T16:28:10.232645+0000","last_clean":"2024-02-12T16:28:10.232645+0000","last_became_active":"2024-02-12T16:24:09.649607+0000","last_became_peered":"2024-02-12T16:24:09.649607+0000","last_unstale":"2024-02-12T16:28:10.232645+0000","last_undegraded":"2024-02-12T16:28:10.232645+0000","last_fullsized":"2024-02-12T16:28:10.232645+0000","mapping_epoch":547,"log_start":"575'9100","ondisk_log_start":"575'9100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10162","last_scrub_stamp":"2024-02-12T16:25:50.727611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.727611+0000","log_size":10038,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109969408,"num_objects":1678,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19138,"num_write_kb":666112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":289,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-02-12T16:28:12.153506+0000","last_change":"2024-02-12T16:25:07.944563+0000","last_active":"2024-02-12T16:28:12.153506+0000","last_peered":"2024-02-12T16:28:12.153506+0000","last_clean":"2024-02-12T16:28:12.153506+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:28:12.153506+0000","last_undegraded":"2024-02-12T16:28:12.153506+0000","last_fullsized":"2024-02-12T16:28:12.153506+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.944451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.944451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"614'18917","reported_seq":19001,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:10.347988+0000","last_change":"2024-02-12T16:26:10.447550+0000","last_active":"2024-02-12T16:28:10.347988+0000","last_peered":"2024-02-12T16:28:10.347988+0000","last_clean":"2024-02-12T16:28:10.347988+0000","last_became_active":"2024-02-12T16:24:09.647095+0000","last_became_peered":"2024-02-12T16:24:09.647095+0000","last_unstale":"2024-02-12T16:28:10.347988+0000","last_undegraded":"2024-02-12T16:28:10.347988+0000","last_fullsized":"2024-02-12T16:28:10.347988+0000","mapping_epoch":547,"log_start":"575'8900","ondisk_log_start":"575'8900","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10017,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112918528,"num_objects":1723,"num_object_clones":0,"num_object_copies":3446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18917,"num_write_kb":660480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":292,"reported_epoch":616,"state":"peering","last_fresh":"2024-02-12T16:28:13.145086+0000","last_change":"2024-02-12T16:28:13.145333+0000","last_active":"2024-02-12T16:28:13.145086+0000","last_peered":"2024-02-12T16:28:12.153308+0000","last_clean":"2024-02-12T16:28:12.153308+0000","last_became_active":"2024-02-12T16:17:37.450884+0000","last_became_peered":"2024-02-12T16:17:37.450884+0000","last_unstale":"2024-02-12T16:28:13.145086+0000","last_undegraded":"2024-02-12T16:28:13.145086+0000","last_fullsized":"2024-02-12T16:28:13.145086+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:37.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:25:37.899168+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[8,9],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"614'19703","reported_seq":19822,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:12.067912+0000","last_change":"2024-02-12T16:28:11.213732+0000","last_active":"2024-02-12T16:28:12.067912+0000","last_peered":"2024-02-12T16:28:12.067912+0000","last_clean":"2024-02-12T16:28:12.067912+0000","last_became_active":"2024-02-12T16:24:09.651253+0000","last_became_peered":"2024-02-12T16:24:09.651253+0000","last_unstale":"2024-02-12T16:28:12.067912+0000","last_undegraded":"2024-02-12T16:28:12.067912+0000","last_fullsized":"2024-02-12T16:28:12.067912+0000","mapping_epoch":547,"log_start":"576'9700","ondisk_log_start":"576'9700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10003,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19703,"num_write_kb":680768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":156,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.102740+0000","last_change":"2024-02-12T16:25:54.579343+0000","last_active":"2024-02-12T16:28:05.102740+0000","last_peered":"2024-02-12T16:28:05.102740+0000","last_clean":"2024-02-12T16:28:05.102740+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:28:05.102740+0000","last_undegraded":"2024-02-12T16:28:05.102740+0000","last_fullsized":"2024-02-12T16:28:05.102740+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:54.579248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:54.579248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"614'19440","reported_seq":19524,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:11.384548+0000","last_change":"2024-02-12T16:27:12.609604+0000","last_active":"2024-02-12T16:28:11.384548+0000","last_peered":"2024-02-12T16:28:11.384548+0000","last_clean":"2024-02-12T16:28:11.384548+0000","last_became_active":"2024-02-12T16:24:09.647187+0000","last_became_peered":"2024-02-12T16:24:09.647187+0000","last_unstale":"2024-02-12T16:28:11.384548+0000","last_undegraded":"2024-02-12T16:28:11.384548+0000","last_fullsized":"2024-02-12T16:28:11.384548+0000","mapping_epoch":547,"log_start":"575'9400","ondisk_log_start":"575'9400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'15509","last_scrub_stamp":"2024-02-12T16:27:12.609496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:12.609496+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19440,"num_write_kb":673920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":721,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-02-12T16:28:12.161012+0000","last_change":"2024-02-12T16:28:11.617580+0000","last_active":"2024-02-12T16:28:12.161012+0000","last_peered":"2024-02-12T16:28:12.161012+0000","last_clean":"2024-02-12T16:28:12.161012+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:28:12.161012+0000","last_undegraded":"2024-02-12T16:28:12.161012+0000","last_fullsized":"2024-02-12T16:28:12.161012+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:04.914304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:04.914304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"616'20000","reported_seq":20086,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-02-12T16:28:13.300895+0000","last_change":"2024-02-12T16:26:57.827400+0000","last_active":"2024-02-12T16:28:13.300895+0000","last_peered":"2024-02-12T16:28:13.300895+0000","last_clean":"2024-02-12T16:28:13.300895+0000","last_became_active":"2024-02-12T16:24:09.650120+0000","last_became_peered":"2024-02-12T16:24:09.650120+0000","last_unstale":"2024-02-12T16:28:13.300895+0000","last_undegraded":"2024-02-12T16:28:13.300895+0000","last_fullsized":"2024-02-12T16:28:13.300895+0000","mapping_epoch":547,"log_start":"576'9900","ondisk_log_start":"576'9900","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'14868","last_scrub_stamp":"2024-02-12T16:26:57.827287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:57.827287+0000","log_size":10100,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98304000,"num_objects":1500,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20000,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":104,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-02-12T16:28:12.161028+0000","last_change":"2024-02-12T16:26:43.025974+0000","last_active":"2024-02-12T16:28:12.161028+0000","last_peered":"2024-02-12T16:28:12.161028+0000","last_clean":"2024-02-12T16:28:12.161028+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:28:12.161028+0000","last_undegraded":"2024-02-12T16:28:12.161028+0000","last_fullsized":"2024-02-12T16:28:12.161028+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:43.025867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:43.025867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"614'19509","reported_seq":21664,"reported_epoch":614,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:28:10.347799+0000","last_change":"2024-02-12T16:26:41.328312+0000","last_active":"2024-02-12T16:28:10.347799+0000","last_peered":"2024-02-12T16:28:10.347799+0000","last_clean":"2024-02-12T16:26:09.048189+0000","last_became_active":"2024-02-12T16:26:11.222461+0000","last_became_peered":"2024-02-12T16:26:11.222461+0000","last_unstale":"2024-02-12T16:28:10.347799+0000","last_undegraded":"2024-02-12T16:28:10.347799+0000","last_fullsized":"2024-02-12T16:28:10.347799+0000","mapping_epoch":585,"log_start":"575'9500","ondisk_log_start":"575'9500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'9817","last_scrub_stamp":"2024-02-12T16:25:42.181950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:42.181950+0000","log_size":10009,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":3394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1290,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":678592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":67633152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[0,10],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,10","objects":1697}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":120,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:09.504789+0000","last_change":"2024-02-12T16:28:09.507154+0000","last_active":"2024-02-12T16:28:09.504789+0000","last_peered":"2024-02-12T16:28:09.504789+0000","last_clean":"2024-02-12T16:28:09.504789+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:28:09.504789+0000","last_undegraded":"2024-02-12T16:28:09.504789+0000","last_fullsized":"2024-02-12T16:28:09.504789+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:52.631321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.631321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"614'19822","reported_seq":19905,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:11.272124+0000","last_change":"2024-02-12T16:25:47.403836+0000","last_active":"2024-02-12T16:28:11.272124+0000","last_peered":"2024-02-12T16:28:11.272124+0000","last_clean":"2024-02-12T16:28:11.272124+0000","last_became_active":"2024-02-12T16:24:09.649590+0000","last_became_peered":"2024-02-12T16:24:09.649590+0000","last_unstale":"2024-02-12T16:28:11.272124+0000","last_undegraded":"2024-02-12T16:28:11.272124+0000","last_fullsized":"2024-02-12T16:28:11.272124+0000","mapping_epoch":547,"log_start":"576'9800","ondisk_log_start":"576'9800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10313","last_scrub_stamp":"2024-02-12T16:25:47.403730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.403730+0000","log_size":10022,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110755840,"num_objects":1690,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19822,"num_write_kb":688384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":300,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.100423+0000","last_change":"2024-02-12T16:25:45.967785+0000","last_active":"2024-02-12T16:28:05.100423+0000","last_peered":"2024-02-12T16:28:05.100423+0000","last_clean":"2024-02-12T16:28:05.100423+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:28:05.100423+0000","last_undegraded":"2024-02-12T16:28:05.100423+0000","last_fullsized":"2024-02-12T16:28:05.100423+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:45.967726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.967726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"614'19192","reported_seq":19276,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:11.370524+0000","last_change":"2024-02-12T16:27:09.549045+0000","last_active":"2024-02-12T16:28:11.370524+0000","last_peered":"2024-02-12T16:28:11.370524+0000","last_clean":"2024-02-12T16:28:11.370524+0000","last_became_active":"2024-02-12T16:24:10.017647+0000","last_became_peered":"2024-02-12T16:24:10.017647+0000","last_unstale":"2024-02-12T16:28:11.370524+0000","last_undegraded":"2024-02-12T16:28:11.370524+0000","last_fullsized":"2024-02-12T16:28:11.370524+0000","mapping_epoch":547,"log_start":"575'9100","ondisk_log_start":"575'9100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'15168","last_scrub_stamp":"2024-02-12T16:27:09.548944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:09.548944+0000","log_size":10092,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100007936,"num_objects":1526,"num_object_clones":0,"num_object_copies":3052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19192,"num_write_kb":662976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":104,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.095747+0000","last_change":"2024-02-12T16:26:59.766851+0000","last_active":"2024-02-12T16:28:05.095747+0000","last_peered":"2024-02-12T16:28:05.095747+0000","last_clean":"2024-02-12T16:28:05.095747+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:28:05.095747+0000","last_undegraded":"2024-02-12T16:28:05.095747+0000","last_fullsized":"2024-02-12T16:28:05.095747+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:59.766744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:59.766744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"614'19489","reported_seq":19569,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:10.257113+0000","last_change":"2024-02-12T16:25:40.938617+0000","last_active":"2024-02-12T16:28:10.257113+0000","last_peered":"2024-02-12T16:28:10.257113+0000","last_clean":"2024-02-12T16:28:10.257113+0000","last_became_active":"2024-02-12T16:24:09.649655+0000","last_became_peered":"2024-02-12T16:24:09.649655+0000","last_unstale":"2024-02-12T16:28:10.257113+0000","last_undegraded":"2024-02-12T16:28:10.257113+0000","last_fullsized":"2024-02-12T16:28:10.257113+0000","mapping_epoch":547,"log_start":"575'9400","ondisk_log_start":"575'9400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9693","last_scrub_stamp":"2024-02-12T16:25:40.938531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:40.938531+0000","log_size":10089,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":117178392,"num_objects":1789,"num_object_clones":0,"num_object_copies":3578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1789,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19489,"num_write_kb":680833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":704,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-02-12T16:28:12.164345+0000","last_change":"2024-02-12T16:26:48.943869+0000","last_active":"2024-02-12T16:28:12.164345+0000","last_peered":"2024-02-12T16:28:12.164345+0000","last_clean":"2024-02-12T16:28:12.164345+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:28:12.164345+0000","last_undegraded":"2024-02-12T16:28:12.164345+0000","last_fullsized":"2024-02-12T16:28:12.164345+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:48.943785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:48.943785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"614'19774","reported_seq":19853,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:11.731197+0000","last_change":"2024-02-12T16:25:50.062621+0000","last_active":"2024-02-12T16:28:11.731197+0000","last_peered":"2024-02-12T16:28:11.731197+0000","last_clean":"2024-02-12T16:28:11.731197+0000","last_became_active":"2024-02-12T16:24:09.647617+0000","last_became_peered":"2024-02-12T16:24:09.647617+0000","last_unstale":"2024-02-12T16:28:11.731197+0000","last_undegraded":"2024-02-12T16:28:11.731197+0000","last_fullsized":"2024-02-12T16:28:11.731197+0000","mapping_epoch":547,"log_start":"576'9700","ondisk_log_start":"576'9700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10397","last_scrub_stamp":"2024-02-12T16:25:50.062524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.062524+0000","log_size":10074,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109969408,"num_objects":1678,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19774,"num_write_kb":686464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":290,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.096662+0000","last_change":"2024-02-12T16:24:50.575354+0000","last_active":"2024-02-12T16:28:05.096662+0000","last_peered":"2024-02-12T16:28:05.096662+0000","last_clean":"2024-02-12T16:28:05.096662+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:28:05.096662+0000","last_undegraded":"2024-02-12T16:28:05.096662+0000","last_fullsized":"2024-02-12T16:28:05.096662+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.575287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.575287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"614'19587","reported_seq":19666,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:12.053245+0000","last_change":"2024-02-12T16:26:09.203006+0000","last_active":"2024-02-12T16:28:12.053245+0000","last_peered":"2024-02-12T16:28:12.053245+0000","last_clean":"2024-02-12T16:28:12.053245+0000","last_became_active":"2024-02-12T16:24:09.650861+0000","last_became_peered":"2024-02-12T16:24:09.650861+0000","last_unstale":"2024-02-12T16:28:12.053245+0000","last_undegraded":"2024-02-12T16:28:12.053245+0000","last_fullsized":"2024-02-12T16:28:12.053245+0000","mapping_epoch":547,"log_start":"575'9500","ondisk_log_start":"575'9500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'11415","last_scrub_stamp":"2024-02-12T16:26:09.202913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:09.202913+0000","log_size":10087,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109248512,"num_objects":1667,"num_object_clones":0,"num_object_copies":3334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19587,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":43,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.095224+0000","last_change":"2024-02-12T16:26:10.388710+0000","last_active":"2024-02-12T16:28:05.095224+0000","last_peered":"2024-02-12T16:28:05.095224+0000","last_clean":"2024-02-12T16:28:05.095224+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:28:05.095224+0000","last_undegraded":"2024-02-12T16:28:05.095224+0000","last_fullsized":"2024-02-12T16:28:05.095224+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:10.388515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:10.388515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"614'19487","reported_seq":19566,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-02-12T16:28:11.275291+0000","last_change":"2024-02-12T16:25:49.810317+0000","last_active":"2024-02-12T16:28:11.275291+0000","last_peered":"2024-02-12T16:28:11.275291+0000","last_clean":"2024-02-12T16:28:11.275291+0000","last_became_active":"2024-02-12T16:24:10.019266+0000","last_became_peered":"2024-02-12T16:24:10.019266+0000","last_unstale":"2024-02-12T16:28:11.275291+0000","last_undegraded":"2024-02-12T16:28:11.275291+0000","last_fullsized":"2024-02-12T16:28:11.275291+0000","mapping_epoch":547,"log_start":"575'9400","ondisk_log_start":"575'9400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10246","last_scrub_stamp":"2024-02-12T16:25:49.810203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:49.810203+0000","log_size":10087,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":108199936,"num_objects":1651,"num_object_clones":0,"num_object_copies":3302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19487,"num_write_kb":676416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":278,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.099104+0000","last_change":"2024-02-12T16:25:03.361871+0000","last_active":"2024-02-12T16:28:05.099104+0000","last_peered":"2024-02-12T16:28:05.099104+0000","last_clean":"2024-02-12T16:28:05.099104+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:28:05.099104+0000","last_undegraded":"2024-02-12T16:28:05.099104+0000","last_fullsized":"2024-02-12T16:28:05.099104+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:25:03.361757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":99,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.096631+0000","last_change":"2024-02-12T16:25:59.785619+0000","last_active":"2024-02-12T16:28:05.096631+0000","last_peered":"2024-02-12T16:28:05.096631+0000","last_clean":"2024-02-12T16:28:05.096631+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:28:05.096631+0000","last_undegraded":"2024-02-12T16:28:05.096631+0000","last_fullsized":"2024-02-12T16:28:05.096631+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:59.785496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:25:59.785496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":97,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.096445+0000","last_change":"2024-02-12T16:24:54.540925+0000","last_active":"2024-02-12T16:28:05.096445+0000","last_peered":"2024-02-12T16:28:05.096445+0000","last_clean":"2024-02-12T16:28:05.096445+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:28:05.096445+0000","last_undegraded":"2024-02-12T16:28:05.096445+0000","last_fullsized":"2024-02-12T16:28:05.096445+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:54.540855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:54.540855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":113,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.096717+0000","last_change":"2024-02-12T16:25:47.933452+0000","last_active":"2024-02-12T16:28:05.096717+0000","last_peered":"2024-02-12T16:28:05.096717+0000","last_clean":"2024-02-12T16:28:05.096717+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:28:05.096717+0000","last_undegraded":"2024-02-12T16:28:05.096717+0000","last_fullsized":"2024-02-12T16:28:05.096717+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:47.933341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.933341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":92,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-02-12T16:28:05.106679+0000","last_change":"2024-02-12T16:25:50.908791+0000","last_active":"2024-02-12T16:28:05.106679+0000","last_peered":"2024-02-12T16:28:05.106679+0000","last_clean":"2024-02-12T16:28:05.106679+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:28:05.106679+0000","last_undegraded":"2024-02-12T16:28:05.106679+0000","last_fullsized":"2024-02-12T16:28:05.106679+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.908692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.908692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1736376344,"num_objects":26496,"num_object_clones":0,"num_object_copies":52992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2383,"num_objects_unfound":0,"num_objects_dirty":26496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":312280,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2576,"num_bytes_recovered":168820736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3539935232,"data_stored":3539927088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":729224,"kb_used_data":143616,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":93015928,"statfs":{"total":95995035648,"available":95248310272,"internally_reserved":0,"allocated":147062784,"data_stored":146671113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.82799999999999996,"15min":0.83899999999999997},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":5.6600000000000001,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.88600000000000001,"15min":0.89200000000000002},"min":{"1min":0.29599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":3.8210000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.89900000000000002,"15min":1.0309999999999999},"min":{"1min":0.36399999999999999,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":4.1520000000000001,"5min":4.6609999999999996,"15min":7.0110000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.875,"15min":1.0600000000000001},"min":{"1min":0.371,"5min":0.33900000000000002,"15min":0.28799999999999998},"max":{"1min":1.9670000000000001,"5min":4.2039999999999997,"15min":5.6920000000000002},"last":1.073}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:27:10 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.96299999999999997,"15min":1.2869999999999999},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3420000000000001,"5min":5.343,"15min":5.343},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.129,"15min":1.4039999999999999},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.4340000000000002,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":1.242}]},{"osd":4,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.86599999999999999,"15min":2.2450000000000001},"min":{"1min":0.39400000000000002,"5min":0.34699999999999998,"15min":0.29499999999999998},"max":{"1min":2.831,"5min":4.4400000000000004,"15min":368.48399999999998},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.998,"5min":0.89500000000000002,"15min":2.1699999999999999},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.101,"5min":4.915,"15min":371.65100000000001},"last":0.80900000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":1.1020000000000001,"15min":2.3860000000000001},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35099999999999998},"max":{"1min":4.6159999999999997,"5min":5.04,"15min":368.51400000000001},"last":1.026},{"interface":"front","average":{"1min":0.81100000000000005,"5min":1.04,"15min":2.3639999999999999},"min":{"1min":0.46999999999999997,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":4.1929999999999996,"5min":5.1509999999999998,"15min":368.70499999999998},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.93600000000000005,"15min":0.97399999999999998},"min":{"1min":0.39200000000000002,"5min":0.38500000000000001,"15min":0.373},"max":{"1min":4.7370000000000001,"5min":5.141,"15min":5.141},"last":1.3049999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.0880000000000001,"15min":1.103},"min":{"1min":0.47699999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.5800000000000001,"5min":8.5709999999999997,"15min":8.5709999999999997},"last":1.218}]},{"osd":7,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.0700000000000001,"15min":1.1950000000000001},"min":{"1min":0.502,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.1279999999999999,"5min":5.2569999999999997,"15min":6.4370000000000003},"last":1.135},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.93700000000000006,"15min":1.153},"min":{"1min":0.46400000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.2350000000000003,"5min":5.0430000000000001,"15min":5.1189999999999998},"last":1.1000000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":6.2320000000000002,"15min":2.7999999999999998},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":2.3679999999999999,"5min":537.54399999999998,"15min":537.54399999999998},"last":1.254},{"interface":"front","average":{"1min":0.72099999999999997,"5min":6.3879999999999999,"15min":2.9940000000000002},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.34699999999999998},"max":{"1min":1.165,"5min":539.86099999999999,"15min":539.86099999999999},"last":1.107}]},{"osd":9,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.071,"15min":1.268},"min":{"1min":0.58399999999999996,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":1.9139999999999999,"5min":5.7430000000000003,"15min":10.743},"last":3.2400000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0900000000000001,"15min":1.381},"min":{"1min":0.60799999999999998,"5min":0.46800000000000003,"15min":0.39500000000000002},"max":{"1min":1.861,"5min":5.359,"15min":10.698},"last":1.268}]},{"osd":10,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":6.3710000000000004,"15min":2.9670000000000001},"min":{"1min":0.48899999999999999,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":4.0599999999999996,"5min":536.245,"15min":536.245},"last":1.3340000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":6.4009999999999998,"15min":2.948},"min":{"1min":0.54700000000000004,"5min":0.443,"15min":0.443},"max":{"1min":2.544,"5min":536.64300000000003,"15min":536.64300000000003},"last":1.1279999999999999}]}]},{"osd":10,"up_from":83,"seq":356482285938,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535324,"kb_used_data":225812,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93209828,"statfs":{"total":95995035648,"available":95446863872,"internally_reserved":0,"allocated":231231488,"data_stored":230833414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.6070000000000002,"15min":1.5169999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":5.867,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.996,"5min":2.6499999999999999,"15min":1.5880000000000001},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.5060000000000002,"5min":162.01300000000001,"15min":162.01300000000001},"last":4.1799999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.028,"15min":0.97999999999999998},"min":{"1min":0.314,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":5.9000000000000004,"5min":5.9000000000000004,"15min":19.866},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.431,"5min":0.94399999999999995,"15min":1.1000000000000001},"min":{"1min":0.33000000000000002,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":20.277999999999999},"last":3.9980000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":2.077,"5min":1.1419999999999999,"15min":0.98199999999999998},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.2430000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":3.8580000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.1339999999999999,"15min":1.0009999999999999},"min":{"1min":0.40200000000000002,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85699999999999998,"15min":3.3359999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.32300000000000001},"max":{"1min":5.8109999999999999,"5min":5.8109999999999999,"15min":628.77099999999996},"last":2.6669999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.98699999999999999,"15min":3.3100000000000001},"min":{"1min":0.34100000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.3070000000000004,"5min":7.3070000000000004,"15min":624.19299999999998},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":6.343,"5min":6.343,"15min":6.343},"last":4.8109999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.099,"15min":1.1379999999999999},"min":{"1min":0.48199999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":7.0339999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":0.80300000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.028,"15min":3.3879999999999999},"min":{"1min":0.42099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":626.26599999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0049999999999999,"15min":3.4609999999999999},"min":{"1min":0.55900000000000005,"5min":0.376,"15min":0.376},"max":{"1min":6.8200000000000003,"5min":6.8200000000000003,"15min":625.90700000000004},"last":2.581}]},{"osd":7,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":0.94899999999999995,"15min":1.0700000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":7.3280000000000003,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.1579999999999999,"15min":1.1870000000000001},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":6.4539999999999997,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":3.3519999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.97799999999999998,"15min":1.1240000000000001},"min":{"1min":0.44400000000000001,"5min":0.436,"15min":0.372},"max":{"1min":5.758,"5min":5.758,"15min":5.758},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.135,"5min":0.98899999999999999,"15min":1.133},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.282,"5min":7.282,"15min":7.282},"last":0.50800000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1100000000000001,"15min":1.2250000000000001},"min":{"1min":0.505,"5min":0.47599999999999998,"15min":0.372},"max":{"1min":5.7809999999999997,"5min":5.7809999999999997,"15min":5.7809999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.048,"5min":1.0089999999999999,"15min":1.212},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":7.3479999999999999,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.53600000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.159,"15min":1.3100000000000001},"min":{"1min":0.59299999999999997,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":7.3689999999999998,"5min":7.3689999999999998,"15min":10.218999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.875,"5min":1.016,"15min":1.268},"min":{"1min":0.622,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.21,"5min":2.21,"15min":7.3250000000000002},"last":0.74199999999999999}]}]},{"osd":9,"up_from":74,"seq":317827580282,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":524932,"kb_used_data":215420,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93220220,"statfs":{"total":95995035648,"available":95457505280,"internally_reserved":0,"allocated":220590080,"data_stored":220202489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.1930000000000001,"15min":1.1020000000000001},"min":{"1min":0.30599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.9560000000000004,"5min":7.9560000000000004,"15min":7.9560000000000004},"last":1.409},{"interface":"front","average":{"1min":2.419,"5min":1.2370000000000001,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:28:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.1499999999999999,"15min":0.995},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.253},"max":{"1min":5.7750000000000004,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.0780000000000001,"15min":1.0669999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":5.7910000000000004,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.956,"5min":1.268,"15min":1.4730000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":12.441000000000001,"5min":12.441000000000001,"15min":97.275000000000006},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.1279999999999999,"15min":1.3129999999999999},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.312},"max":{"1min":4.6239999999999997,"5min":6.1779999999999999,"15min":97.417000000000002},"last":1.6899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.028,"15min":0.96899999999999997},"min":{"1min":0.35999999999999999,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":4.3940000000000001,"5min":4.569,"15min":4.569},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.1539999999999999,"15min":1.145},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.6589999999999998,"5min":4.8840000000000003,"15min":5.8010000000000002},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.304,"15min":1.1279999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":4.8780000000000001,"5min":6.726,"15min":6.726},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.149},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.9059999999999997,"5min":6.7350000000000003,"15min":7.3739999999999997},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0189999999999999,"15min":1.0880000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":4.6749999999999998,"5min":5.5250000000000004,"15min":7.3360000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1699999999999999,"15min":1.0660000000000001},"min":{"1min":0.51200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.4610000000000003,"5min":7.0609999999999999,"15min":7.0609999999999999},"last":0.747}]},{"osd":7,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.26,"15min":1.238},"min":{"1min":0.433,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":4.6399999999999997,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":1.5},{"interface":"front","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.216},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.7000000000000002,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.57199999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0449999999999999,"15min":1.077},"min":{"1min":0.42499999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.006,"5min":1.0680000000000001,"15min":1.1839999999999999},"min":{"1min":0.56100000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":7.25},"last":0.78200000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.2,"15min":1.2070000000000001},"min":{"1min":0.54800000000000004,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":3.0550000000000002,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":3.9820000000000002},{"interface":"front","average":{"1min":1.228,"5min":1.1619999999999999,"15min":1.2030000000000001},"min":{"1min":0.49199999999999999,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":3.8319999999999999,"5min":4.4509999999999996,"15min":5.8700000000000001},"last":0.84199999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.141,"15min":1.3260000000000001},"min":{"1min":0.60499999999999998,"5min":0.48999999999999999,"15min":0.45600000000000002},"max":{"1min":1.4379999999999999,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":1.556},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.0700000000000001,"15min":1.2829999999999999},"min":{"1min":0.52500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.8929999999999998,"5min":5.492,"15min":6.3639999999999999},"last":0.63600000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874628,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":488860,"kb_used_data":187540,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93256292,"statfs":{"total":95995035648,"available":95494443008,"internally_reserved":0,"allocated":192040960,"data_stored":191642894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.006,"15min":0.88600000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":4.7270000000000003,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":1.093,"15min":0.88900000000000001},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29399999999999998},"max":{"1min":3.331,"5min":18.584,"15min":18.584},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.7669999999999999,"15min":1.4530000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.4889999999999999,"5min":69.956000000000003,"15min":69.956000000000003},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.7529999999999999,"15min":1.53},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.1260000000000003,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.83499999999999996},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0510000000000002,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91600000000000004,"15min":0.99299999999999999},"min":{"1min":0.36699999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.5569999999999999,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":1.4059999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.82899999999999996,"15min":1.7150000000000001},"min":{"1min":0.33700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.7250000000000001,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.875,"15min":1.6519999999999999},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":0.95299999999999996,"5min":5.0380000000000003,"15min":224.71000000000001},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.85199999999999998,"15min":0.90200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":5.2480000000000002,"5min":5.2480000000000002,"15min":5.2480000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.95199999999999996,"15min":1.0229999999999999},"min":{"1min":0.40799999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.5600000000000001,"5min":3.5600000000000001,"15min":4.8339999999999996},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.155,"15min":1.7410000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.37},"max":{"1min":5.3639999999999999,"5min":5.3639999999999999,"15min":221.36699999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.099,"15min":1.8600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.2869999999999999,"5min":5.556,"15min":221.203},"last":0.60999999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.1319999999999999,"15min":1.1459999999999999},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.42199999999999999},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":2.5619999999999998},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.1890000000000001,"15min":1.23},"min":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.41599999999999998},"max":{"1min":4.2240000000000002,"5min":4.2240000000000002,"15min":7.899},"last":2.5779999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.028,"15min":1.165},"min":{"1min":0.55300000000000005,"5min":0.371,"15min":0.35599999999999998},"max":{"1min":2.6419999999999999,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":2.6419999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.117,"15min":1.2390000000000001},"min":{"1min":0.60299999999999998,"5min":0.47599999999999998,"15min":0.44700000000000001},"max":{"1min":2.6709999999999998,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":2.6709999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.996,"15min":1.4299999999999999},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":109.92},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.032,"15min":1.474},"min":{"1min":0.504,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6469999999999998,"5min":5.0250000000000004,"15min":110.249},"last":0.63100000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":1.1819999999999999},"min":{"1min":0.627,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":3.3109999999999999,"5min":3.3109999999999999,"15min":5.0670000000000002},"last":0.755},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":1.141},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.327,"5min":3.327,"15min":4.6639999999999997},"last":0.65400000000000003}]}]},{"osd":7,"up_from":56,"seq":240518168971,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405904,"kb_used_data":100488,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93339248,"statfs":{"total":95995035648,"available":95579389952,"internally_reserved":0,"allocated":102899712,"data_stored":102506169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":11.994,"5min":3.085,"15min":1.6419999999999999},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":187.33699999999999,"5min":187.33699999999999,"15min":187.33699999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":11.662000000000001,"5min":3.0800000000000001,"15min":1.653},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":187.48400000000001,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.39800000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.3169999999999999,"15min":1.044},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.4009999999999998,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.591,"15min":1.1859999999999999},"min":{"1min":0.34699999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.4670000000000001,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:27:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.034,"15min":1.2430000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.1669999999999998,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.88200000000000001,"15min":0.97999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":4.4340000000000002,"5min":4.4699999999999998,"15min":4.4699999999999998},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.131,"15min":0.93999999999999995},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.899,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.92600000000000005,"15min":0.998},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":4.3170000000000002,"5min":4.7389999999999999,"15min":4.7389999999999999},"last":0.36099999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0029999999999999,"15min":0.96799999999999997},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.5499999999999998,"5min":4.7220000000000004,"15min":4.9050000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.999,"15min":1.018},"min":{"1min":0.40500000000000003,"5min":0.314,"15min":0.314},"max":{"1min":2.964,"5min":7.5640000000000001,"15min":9.3640000000000008},"last":0.90000000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.085,"15min":1.1000000000000001},"min":{"1min":0.34599999999999997,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":3.5819999999999999,"5min":6.319,"15min":6.319},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1240000000000001,"15min":1.179},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":4.3609999999999998,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.39300000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":6.5750000000000002,"15min":2.9449999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.1619999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.635,"5min":6.6379999999999999,"15min":3.0310000000000001},"min":{"1min":0.50800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":9.9060000000000006,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.434}]},{"osd":9,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.97899999999999998,"15min":1.1579999999999999},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.8029999999999999,"5min":4.6539999999999999,"15min":12.907999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.381,"5min":1.171,"15min":1.2669999999999999},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.7800000000000002,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":0.69799999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":7.7400000000000002,"15min":3.508},"min":{"1min":0.46500000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.5309999999999997,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":7.6219999999999999,"15min":3.375},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.946,"5min":496.29599999999999,"15min":496.29599999999999},"last":0.50600000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.1060000000000001,"15min":1.601},"min":{"1min":0.495,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":4.9039999999999999,"5min":6.0490000000000004,"15min":120.943},"last":0.626},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.252,"15min":1.714},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":120.661},"last":0.73199999999999998}]}]},{"osd":6,"up_from":48,"seq":206158430611,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":630468,"kb_used_data":333244,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93114684,"statfs":{"total":95995035648,"available":95349436416,"internally_reserved":0,"allocated":341241856,"data_stored":340854265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":7.04,"5min":2.367,"15min":1.464},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":107.723,"5min":107.723,"15min":107.723},"last":1.179},{"interface":"front","average":{"1min":7.1260000000000003,"5min":2.5139999999999998,"15min":1.552},"min":{"1min":0.38,"5min":0.252,"15min":0.252},"max":{"1min":107.114,"5min":107.114,"15min":107.114},"last":1.0349999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0449999999999999,"15min":1.923},"min":{"1min":0.36499999999999999,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":3.5870000000000002,"5min":5.5179999999999998,"15min":252.22800000000001},"last":1.071},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.05,"15min":1.946},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.0549999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5009999999999999,"15min":1.0649999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.1369999999999996,"5min":28.739999999999998,"15min":28.739999999999998},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.7090000000000001,"15min":1.1719999999999999},"min":{"1min":0.33700000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3769999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.437}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.2529999999999999,"15min":1.024},"min":{"1min":0.29099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.4509999999999996,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":0.376},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.079,"15min":1.0620000000000001},"min":{"1min":0.34999999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.242,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.97699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.117,"15min":0.99099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":4.4779999999999998,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.99299999999999999,"15min":1.0109999999999999},"min":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1080000000000001,"5min":10.52,"15min":10.52},"last":1.1080000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0720000000000001,"15min":1.105},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.0970000000000004,"5min":7.641,"15min":7.641},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.0700000000000001,"15min":1.071},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.7759999999999998,"5min":7.8380000000000001,"15min":7.8380000000000001},"last":1.141}]},{"osd":8,"last update":"Mon Feb 12 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.0800000000000001,"15min":1.1080000000000001},"min":{"1min":0.42099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":5.6310000000000002,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.0169999999999999,"15min":1.1890000000000001},"min":{"1min":0.39600000000000002,"5min":0.371,"15min":0.371},"max":{"1min":2.6440000000000001,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":0.55100000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.1839999999999999,"15min":1.254},"min":{"1min":0.45600000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.1680000000000001,"5min":4.5330000000000004,"15min":18.059999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.0229999999999999,"15min":1.246},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2430000000000001,"5min":4.6890000000000001,"15min":17.425000000000001},"last":1.2110000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.222,"15min":1.27},"min":{"1min":0.45500000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":5.726,"5min":5.726,"15min":6.5300000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.542,"5min":1.294,"15min":1.3540000000000001},"min":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.7590000000000003,"5min":5.7590000000000003,"15min":6.5730000000000004},"last":0.57699999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.228,"15min":2.8719999999999999},"min":{"1min":0.52100000000000002,"5min":0.499,"15min":0.499},"max":{"1min":5.6859999999999999,"5min":5.6859999999999999,"15min":139.727},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.028,"15min":2.7349999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7310000000000001,"5min":4.5899999999999999,"15min":139.24199999999999},"last":1.248}]}]},{"osd":1,"up_from":13,"seq":55834575289,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1323204,"kb_used_data":420796,"kb_used_omap":0,"kb_used_meta":902400,"kb_avail":92421948,"statfs":{"total":95995035648,"available":94640074752,"internally_reserved":0,"allocated":430895104,"data_stored":430507513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":924057600},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0299999999999998,"5min":1.1599999999999999,"15min":1.405},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":16.632999999999999,"5min":16.632999999999999,"15min":114.40000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.29,"5min":1.5920000000000001,"15min":1.6599999999999999},"min":{"1min":0.32300000000000001,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":6.5759999999999996,"5min":7.7249999999999996,"15min":115.09099999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.1639999999999999,"15min":1.04},"min":{"1min":0.33400000000000002,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":6.5119999999999996,"5min":6.5389999999999997,"15min":9.8599999999999994},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.76,"5min":1.367,"15min":1.1180000000000001},"min":{"1min":0.28899999999999998,"5min":0.246,"15min":0.246},"max":{"1min":9.9689999999999994,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.55000000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.0920000000000001,"15min":1.1200000000000001},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.625,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.1910000000000001,"15min":0.95499999999999996},"min":{"1min":0.38300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.1639999999999997,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.2829999999999999,"15min":1.125},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.6749999999999998,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.441},{"interface":"front","average":{"1min":1.24,"5min":1.4650000000000001,"15min":1.405},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.633,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.399,"15min":1.2190000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.6740000000000004,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.2629999999999999,"15min":1.196},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":10.067,"5min":10.067,"15min":10.067},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":3.2530000000000001,"5min":1.661,"15min":1.7609999999999999},"min":{"1min":0.51700000000000002,"5min":0.38500000000000001,"15min":0.34699999999999998},"max":{"1min":30.437000000000001,"5min":30.437000000000001,"15min":55.869999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":3.2029999999999998,"5min":1.611,"15min":1.833},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36799999999999999},"max":{"1min":29.382000000000001,"5min":29.382000000000001,"15min":55.892000000000003},"last":0.623}]},{"osd":8,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":7.4260000000000002,"15min":3.399},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.436},"max":{"1min":10.138999999999999,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.1040000000000001,"5min":7.5540000000000003,"15min":3.4279999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.7160000000000002,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.68899999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:27:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.4450000000000001,"15min":1.3959999999999999},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":5.7590000000000003,"5min":5.7590000000000003,"15min":10.07},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.323,"15min":1.46},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":10.106,"5min":10.106,"15min":10.106},"last":0.70499999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":8.4920000000000009,"15min":3.7589999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.7949999999999999,"5min":548.93499999999995,"15min":548.93499999999995},"last":0.753},{"interface":"front","average":{"1min":1.7609999999999999,"5min":8.3650000000000002,"15min":3.806},"min":{"1min":0.57299999999999995,"5min":0.55100000000000005,"15min":0.53700000000000003},"max":{"1min":10.170999999999999,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.74399999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.526,"15min":1.5680000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.443},"max":{"1min":10.202,"5min":10.202,"15min":10.202},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.649,"5min":1.6419999999999999,"15min":1.627},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":5.8300000000000001,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":0.71899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738820,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151940,"kb_used_data":562236,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":92593212,"statfs":{"total":95995035648,"available":94815449088,"internally_reserved":0,"allocated":575729664,"data_stored":575342073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.0509999999999999,"15min":1.054},"min":{"1min":0.26500000000000001,"5min":0.245,"15min":0.215},"max":{"1min":5.5869999999999997,"5min":10.754,"15min":11.324},"last":0.27400000000000002},{"interface":"front","average":{"1min":2.3679999999999999,"5min":1.3140000000000001,"15min":1.1519999999999999},"min":{"1min":0.24399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":5.1020000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.0649999999999999,"15min":1.0580000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.0279999999999996,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":5.032},{"interface":"front","average":{"1min":2.0259999999999998,"5min":1.0940000000000001,"15min":0.97999999999999998},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.5510000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.24099999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":13.318,"5min":3.6579999999999999,"15min":2.1150000000000002},"min":{"1min":0.42799999999999999,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":193.77000000000001,"5min":193.77000000000001,"15min":193.77000000000001},"last":5.4729999999999999},{"interface":"front","average":{"1min":11.885999999999999,"5min":3.5009999999999999,"15min":2.1440000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":189.625,"5min":189.625,"15min":189.625},"last":5.141}]},{"osd":5,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.1020000000000001,"15min":1.016},"min":{"1min":0.33600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.2690000000000001,"5min":5.2690000000000001,"15min":5.2690000000000001},"last":5.4349999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.1659999999999999,"15min":1.052},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":5.4619999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":10.422000000000001,"5min":3.161,"15min":1.9890000000000001},"min":{"1min":0.371,"5min":0.311,"15min":0.311},"max":{"1min":189.79400000000001,"5min":189.79400000000001,"15min":189.79400000000001},"last":5.0919999999999996},{"interface":"front","average":{"1min":10.676,"5min":3.238,"15min":2.0680000000000001},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":194.41200000000001,"5min":194.41200000000001,"15min":194.41200000000001},"last":5.2279999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":12.795999999999999,"5min":3.5070000000000001,"15min":1.9350000000000001},"min":{"1min":0.39100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":191.47200000000001,"5min":191.47200000000001,"15min":191.47200000000001},"last":5.274},{"interface":"front","average":{"1min":12.872,"5min":3.5950000000000002,"15min":2.0369999999999999},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.44},"max":{"1min":191.50299999999999,"5min":191.50299999999999,"15min":191.50299999999999},"last":5.3769999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.276,"15min":1.2689999999999999},"min":{"1min":0.48399999999999999,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":5.1070000000000002,"5min":5.1070000000000002,"15min":5.1070000000000002},"last":5.1689999999999996},{"interface":"front","average":{"1min":1.552,"5min":1.369,"15min":1.357},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":5.1749999999999998,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":5.4029999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.2010000000000001,"15min":1.361},"min":{"1min":0.46899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.6550000000000002,"5min":7.6539999999999999,"15min":46.246000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.345,"15min":1.4219999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.5330000000000004,"5min":5.5330000000000004,"15min":46.167999999999999},"last":5.3399999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1099999999999999,"5min":1.4159999999999999,"15min":1.3320000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.41399999999999998},"max":{"1min":5.1390000000000002,"5min":5.1390000000000002,"15min":6.0979999999999999},"last":5.5279999999999996},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.4890000000000001,"15min":1.3580000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":5.6269999999999998},"last":5.5529999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":2.2679999999999998,"5min":1.452,"15min":1.575},"min":{"1min":0.497,"5min":0.46700000000000003,"15min":0.434},"max":{"1min":5.1050000000000004,"5min":5.1050000000000004,"15min":46.064},"last":5.5019999999999998},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.355,"15min":1.5049999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.38300000000000001},"max":{"1min":5.1390000000000002,"5min":5.1390000000000002,"15min":45.642000000000003},"last":5.3090000000000002}]}]},{"osd":2,"up_from":21,"seq":90194313650,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615492,"kb_used_data":314108,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93129660,"statfs":{"total":95995035648,"available":95364771840,"internally_reserved":0,"allocated":321646592,"data_stored":321259001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":2.371,"5min":1.3200000000000001,"15min":1.1180000000000001},"min":{"1min":0.312,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.378},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.446,"15min":1.208},"min":{"1min":0.32600000000000001,"5min":0.247,"15min":0.247},"max":{"1min":6.4180000000000001,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.52,"15min":1.9239999999999999},"min":{"1min":0.29599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":6.2910000000000004,"5min":21.407,"15min":21.407},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1020000000000001,"15min":0.92000000000000004},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.8899999999999997,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.54200000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:27:47 2024","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":1.4359999999999999,"15min":1.0800000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.502},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.526,"15min":1.194},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.5279999999999996,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":1.3420000000000001,"15min":1.147},"min":{"1min":0.44800000000000001,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":6.133,"5min":6.133,"15min":6.133},"last":0.623},{"interface":"front","average":{"1min":2.8140000000000001,"5min":1.5289999999999999,"15min":1.1850000000000001},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.315},"max":{"1min":11.081,"5min":11.081,"15min":11.081},"last":0.63}]},{"osd":6,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.653,"15min":1.278},"min":{"1min":0.45500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.4980000000000002,"5min":21.527000000000001,"15min":21.527000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.605,"5min":1.3759999999999999,"15min":1.2050000000000001},"min":{"1min":0.41999999999999998,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":6.1710000000000003,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:27:47 2024","interfaces":[{"interface":"back","average":{"1min":2.9609999999999999,"5min":1.522,"15min":1.2030000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":11.044,"5min":11.044,"15min":11.044},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.373,"15min":1.2490000000000001},"min":{"1min":0.51200000000000001,"5min":0.443,"15min":0.41999999999999998},"max":{"1min":6.2050000000000001,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":0.64400000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3919999999999999,"15min":1.2909999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42099999999999999},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.504,"5min":1.462,"15min":1.327},"min":{"1min":0.46000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":6.593,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":0.73799999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:27:47 2024","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":1.5489999999999999,"15min":1.323},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":10.972,"5min":10.972,"15min":10.972},"last":1.0900000000000001},{"interface":"front","average":{"1min":2.9089999999999998,"5min":1.627,"15min":1.395},"min":{"1min":0.442,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":11.006,"5min":11.006,"15min":11.006},"last":0.746}]},{"osd":10,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.5629999999999999,"15min":1.385},"min":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6319999999999997,"5min":11.454000000000001,"15min":11.454000000000001},"last":1.24},{"interface":"front","average":{"1min":2.032,"5min":1.5800000000000001,"15min":1.411},"min":{"1min":0.501,"5min":0.441,"15min":0.441},"max":{"1min":6.8010000000000002,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":1.216}]},{"osd":11,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.389,"15min":1.502},"min":{"1min":0.60999999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":6.2599999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.718,"5min":1.5069999999999999,"15min":1.5760000000000001},"min":{"1min":0.59499999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.76600000000000001}]}]},{"osd":3,"up_from":522,"seq":2241972928575,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":457924,"kb_used_data":117500,"kb_used_omap":91,"kb_used_meta":340324,"kb_avail":93287228,"statfs":{"total":95995035648,"available":95526121472,"internally_reserved":0,"allocated":120320000,"data_stored":119932409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":348491792},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3560000000000001,"15min":1.016},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.6630000000000003,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.309},{"interface":"front","average":{"1min":1.359,"5min":1.462,"15min":1.056},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.7850000000000001,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.1200000000000001,"15min":0.95199999999999996},"min":{"1min":0.317,"5min":0.314,"15min":0.314},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.341,"5min":1.1970000000000001,"15min":1.1020000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.746,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":0.77600000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.0589999999999999,"15min":1.304},"min":{"1min":0.38400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.7080000000000002,"5min":10.004,"15min":10.004},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.085,"15min":1.1160000000000001},"min":{"1min":0.40000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.4560000000000004,"5min":4.4560000000000004,"15min":4.4560000000000004},"last":0.40400000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1870000000000001,"15min":1.3600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":3.7530000000000001,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.93999999999999995,"15min":0.94599999999999995},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.55100000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.1699999999999999,"15min":1.1619999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":8.3320000000000007,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.0449999999999999,"15min":1.0269999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.1920000000000002,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.258,"15min":1.458},"min":{"1min":0.48499999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.8090000000000002,"5min":6.851,"15min":6.851},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.319,"15min":1.3109999999999999},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.8890000000000002,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":0.67900000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:27:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.137,"15min":1.204},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.1459999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.748},{"interface":"front","average":{"1min":1.276,"5min":1.0429999999999999,"15min":1.1140000000000001},"min":{"1min":0.47099999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.8460000000000001,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.81399999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.3640000000000001,"15min":1.3560000000000001},"min":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.5810000000000004,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.823,"5min":1.3979999999999999,"15min":1.401},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":5.0780000000000003,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.86399999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3480000000000001,"15min":1.52},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":4.8499999999999996,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.77,"5min":1.361,"15min":1.343},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.9320000000000004,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":0.56699999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:27:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.2789999999999999,"15min":1.296},"min":{"1min":0.58599999999999997,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":4.6399999999999997,"5min":4.6399999999999997,"15min":4.6399999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.3899999999999999,"15min":1.53},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":8.4540000000000006,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.71199999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953890,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781828,"kb_used_data":484604,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92963324,"statfs":{"total":95995035648,"available":95194443776,"internally_reserved":0,"allocated":496234496,"data_stored":495846905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:27:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.093,"15min":1.363},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.4710000000000001,"5min":5.4710000000000001,"15min":90.597999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.143,"5min":1.157,"15min":1.5409999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.5389999999999997,"5min":12.236000000000001,"15min":90.676000000000002},"last":3.4169999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.022,"15min":0.95999999999999996},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.577,"5min":5.577,"15min":8.0530000000000008},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.1619999999999999,"15min":1.1279999999999999},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":6.718,"5min":6.718,"15min":7.452},"last":0.34200000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":1.1259999999999999,"15min":1.1020000000000001},"min":{"1min":0.33300000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.2930000000000001,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.018,"15min":1.046},"min":{"1min":0.35499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.657,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":1.022}]},{"osd":5,"last update":"Mon Feb 12 16:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.98399999999999999,"15min":0.93500000000000005},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.9950000000000001,"5min":5.5999999999999996,"15min":5.5999999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1419999999999999,"15min":1.0920000000000001},"min":{"1min":0.41099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.0309999999999997,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.83299999999999996,"15min":0.97799999999999998},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.8660000000000001,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.018,"5min":1.218,"15min":1.071},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.5190000000000001,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":0.313}]},{"osd":7,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1379999999999999,"15min":1.101},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":5.4870000000000001,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":0.378},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.88100000000000001,"15min":1.143},"min":{"1min":0.36099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6459999999999999,"5min":3.6459999999999999,"15min":7.6689999999999996},"last":0.36299999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:27:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.97699999999999998,"15min":1.204},"min":{"1min":0.44700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.0439999999999996,"5min":4.0439999999999996,"15min":7.8650000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.2370000000000001,"15min":1.2310000000000001},"min":{"1min":0.53700000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.7250000000000001,"5min":5.6699999999999999,"15min":5.6699999999999999},"last":0.441}]},{"osd":9,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.998,"15min":1.399},"min":{"1min":0.53300000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.9750000000000001,"5min":3.2650000000000001,"15min":34.112000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.079,"5min":1.3440000000000001,"15min":1.637},"min":{"1min":0.45600000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.754,"5min":5.6420000000000003,"15min":34.590000000000003},"last":0.73399999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.373,"15min":1.381},"min":{"1min":0.503,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.48,"5min":5.6970000000000001,"15min":6.0899999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.177,"15min":1.2350000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.4269999999999996,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":0.52700000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0800000000000001,"15min":1.4470000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6500000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.357,"15min":1.482},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":3.52,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.56399999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740744,"kb_used_data":431232,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93004408,"statfs":{"total":95995035648,"available":95236513792,"internally_reserved":0,"allocated":441581568,"data_stored":441189905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.83799999999999997,"15min":2.7349999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":565.34500000000003},"last":3.3780000000000001},{"interface":"front","average":{"1min":1.113,"5min":0.93799999999999994,"15min":2.8140000000000001},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":4.5830000000000002,"5min":4.5830000000000002,"15min":565.98400000000004},"last":3.5430000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3839999999999999,"15min":1.143},"min":{"1min":0.314,"5min":0.314,"15min":0.30499999999999999},"max":{"1min":6.0709999999999997,"5min":40.021999999999998,"15min":40.021999999999998},"last":3.4860000000000002},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.478,"15min":1.1859999999999999},"min":{"1min":0.28599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.5069999999999997,"5min":40.225000000000001,"15min":40.225000000000001},"last":3.6539999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:27:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.3420000000000001,"15min":1.3129999999999999},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.5289999999999999,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.80600000000000005,"15min":0.754},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.1619999999999999,"5min":5.1619999999999999,"15min":5.1619999999999999},"last":3.7170000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.331,"15min":1.4239999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.661,"5min":41.908000000000001,"15min":75.257000000000005},"last":4.9509999999999996},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.5329999999999999,"15min":1.46},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":6.4610000000000003,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.33600000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1739999999999999,"15min":1.3320000000000001},"min":{"1min":0.34200000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.3490000000000002,"5min":6.4459999999999997,"15min":74.105000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.2330000000000001,"15min":1.4390000000000001},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":9.5830000000000002,"5min":9.5830000000000002,"15min":75.013999999999996},"last":3.6850000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.113,"15min":1.0669999999999999},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":9.7710000000000008,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":3.7429999999999999},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.1779999999999999,"15min":1.171},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.3979999999999997,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.36599999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.95799999999999996,"15min":1.181},"min":{"1min":0.432,"5min":0.41699999999999998,"15min":0.40200000000000002},"max":{"1min":2.524,"5min":4.0069999999999997,"15min":5.9509999999999996},"last":3.512},{"interface":"front","average":{"1min":1.202,"5min":1.119,"15min":1.2430000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.6159999999999997,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":3.7749999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4550000000000001,"15min":3.327},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":4.54,"5min":6.2939999999999996,"15min":449.89499999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.829,"5min":1.321,"15min":3.3010000000000002},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":7.5789999999999997,"5min":9.6649999999999991,"15min":448.70800000000003},"last":3.7999999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.016,"15min":1.258},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.6120000000000001,"5min":4.0410000000000004,"15min":5.9210000000000003},"last":3.5859999999999999},{"interface":"front","average":{"1min":2.016,"5min":1.431,"15min":1.3540000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":15.006,"5min":15.006,"15min":15.006},"last":0.58999999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.131,"15min":1.21},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.7789999999999999,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":3.6230000000000002},{"interface":"front","average":{"1min":1.712,"5min":1.4419999999999999,"15min":1.355},"min":{"1min":0.57699999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":3.9830000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":563609600,"data_stored":563609600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":428146688,"data_stored":428146688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313524224,"data_stored":313524224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115736576,"data_stored":115736576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":480903168,"data_stored":480903168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":423825408,"data_stored":423821336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":338952192,"data_stored":338952192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304000,"data_stored":98304000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":186908672,"data_stored":186908672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":218890240,"data_stored":218890240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":227016704,"data_stored":227016704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":144117760,"data_stored":144113688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:28:16.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:16.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:16.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:17.045 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:28:17.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:17.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:17.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:17.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:17.228 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:17 smithi123 bash[19119]: cluster 2024-02-12T16:28:15.995643+0000 mgr.y (mgr.14146) 1301 : cluster [DBG] pgmap v1808: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 2060/48828 objects misplaced (4.219%); 980 KiB/s, 15 objects/s recovering 2024-02-12T16:28:17.243 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:17.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:17.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:17.386 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:17.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:17.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:17 smithi121 bash[18489]: cluster 2024-02-12T16:28:15.995643+0000 mgr.y (mgr.14146) 1301 : cluster [DBG] pgmap v1808: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 2060/48828 objects misplaced (4.219%); 980 KiB/s, 15 objects/s recovering 2024-02-12T16:28:17.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:17.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:17 smithi169 bash[19604]: cluster 2024-02-12T16:28:15.995643+0000 mgr.y (mgr.14146) 1301 : cluster [DBG] pgmap v1808: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 2060/48828 objects misplaced (4.219%); 980 KiB/s, 15 objects/s recovering 2024-02-12T16:28:17.523 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:17.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:17.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:17.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:17.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:17.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:17.790 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:17.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:17.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:17.918 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:17.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:18.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:18.051 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:18.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:18.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:18.177 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:18.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:18.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:18.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:18.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:18.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:18.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:18.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:18.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:18.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:18.721 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:18.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:18.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:18.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:18.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:18.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:18.998 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:19.011 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:28:19.055 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:28:19.096 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:28:19.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:19.127 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:19.128 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:28:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:19.166 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:28:19.199 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:28:19.230 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:19.232 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:28:19.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:19.262 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:28:19.301 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:28:19.326 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:28:19.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:19.361 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:28:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:19 smithi123 bash[19119]: cluster 2024-02-12T16:28:17.996113+0000 mgr.y (mgr.14146) 1302 : cluster [DBG] pgmap v1809: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 1784/43304 objects misplaced (4.120%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:28:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:19 smithi123 bash[19119]: audit 2024-02-12T16:28:18.958217+0000 mon.c (mon.1) 55 : audit [DBG] from='client.? v1:172.21.15.121:0/1027798525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:19.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:19.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:28:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:28:19.458 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:19 smithi121 bash[18489]: cluster 2024-02-12T16:28:17.996113+0000 mgr.y (mgr.14146) 1302 : cluster [DBG] pgmap v1809: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 1784/43304 objects misplaced (4.120%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:28:19.458 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:19 smithi121 bash[18489]: audit 2024-02-12T16:28:18.958217+0000 mon.c (mon.1) 55 : audit [DBG] from='client.? v1:172.21.15.121:0/1027798525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:19.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:19.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:19 smithi169 bash[19604]: cluster 2024-02-12T16:28:17.996113+0000 mgr.y (mgr.14146) 1302 : cluster [DBG] pgmap v1809: 37 pgs: 1 peering, 2 active+remapped+backfilling, 34 active+clean; 1.3 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 1784/43304 objects misplaced (4.120%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:28:19.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:19 smithi169 bash[19604]: audit 2024-02-12T16:28:18.958217+0000 mon.c (mon.1) 55 : audit [DBG] from='client.? v1:172.21.15.121:0/1027798525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:19.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:19.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:19.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:19.810 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:19.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:19.957 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:19.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:19.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:28:20.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:20.088 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:20.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:20.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:28:20.217 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:20.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:20.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:20 smithi121 bash[18489]: audit 2024-02-12T16:28:19.290348+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.121:0/1817439051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:28:20.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:20 smithi121 bash[18489]: cluster 2024-02-12T16:28:19.370772+0000 osd.8 (osd.8) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:20.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:20 smithi121 bash[18489]: cluster 2024-02-12T16:28:19.370822+0000 osd.8 (osd.8) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:20.319 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:20.364 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:20.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:20.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:20.474 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:20 smithi123 bash[19119]: audit 2024-02-12T16:28:19.290348+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.121:0/1817439051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:28:20.474 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:20 smithi123 bash[19119]: cluster 2024-02-12T16:28:19.370772+0000 osd.8 (osd.8) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:20.474 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:20 smithi123 bash[19119]: cluster 2024-02-12T16:28:19.370822+0000 osd.8 (osd.8) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:20.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:20.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:20 smithi169 bash[19604]: audit 2024-02-12T16:28:19.290348+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.121:0/1817439051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:28:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:20 smithi169 bash[19604]: cluster 2024-02-12T16:28:19.370772+0000 osd.8 (osd.8) 59 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:20 smithi169 bash[19604]: cluster 2024-02-12T16:28:19.370822+0000 osd.8 (osd.8) 60 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:20.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:20.628 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:20.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:20.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:20.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:20.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:20.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:20.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:20.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:20.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:20.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:21.030 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:21.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:21.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:21.171 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:21.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:21.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:21.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:21 smithi169 bash[19604]: cluster 2024-02-12T16:28:19.996564+0000 mgr.y (mgr.14146) 1303 : cluster [DBG] pgmap v1810: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s; 1784/41316 objects misplaced (4.318%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:28:21.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:21 smithi169 bash[19604]: cluster 2024-02-12T16:28:20.169491+0000 mon.a (mon.0) 2202 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:21.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:21 smithi169 bash[19604]: audit 2024-02-12T16:28:20.179768+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? v1:172.21.15.121:0/1817439051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:28:21.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:21 smithi169 bash[19604]: cluster 2024-02-12T16:28:20.180170+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e620: 12 total, 12 up, 12 in 2024-02-12T16:28:21.285 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:21 smithi169 bash[19604]: cluster 2024-02-12T16:28:20.672738+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e621: 12 total, 12 up, 12 in 2024-02-12T16:28:21.328 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:28:21.328 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:28:21.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.328 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.16095.0:323161 5.0 e621/581 5:0088b45a:::benchmark_data_smithi169_84712_object153771:head v 621'20150, mlcod=621'20148)", 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:28:21.308901+0000", 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.016083124000000001, 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016107314000000001, 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.308901+0000", 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:28:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.308901+0000", 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.308899+0000", 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8970000000000003e-06 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.308909+0000", 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9860000000000006e-06 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.308919+0000", 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0750000000000003e-06 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.308925+0000", 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3396000000000001e-05 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.308988+0000", 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8192e-05 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.309016+0000", 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011549599999999999 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.16095.0:323170 5.0 e621/581 5:0cd4583e:::benchmark_data_smithi169_84712_object153780:head v 621'20151, mlcod=621'20148)", 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:28:21.315979+0000", 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0090043550000000003, 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091055430000000007, 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:28:21.332 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.315979+0000", 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.315979+0000", 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.315978+0000", 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2201999999999999e-05 2024-02-12T16:28:21.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.315990+0000", 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.43e-05 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.316004+0000", 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9759999999999998e-06 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.316011+0000", 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6702000000000001e-05 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.316058+0000", 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9424000000000002e-05 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.316087+0000", 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107771 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.16095.0:323175 5.0 e621/581 5:0f8d3877:::benchmark_data_smithi169_84712_object153785:head v 621'20152, mlcod=621'20148)", 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:28:21.321545+0000", 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0034391220000000002, 2024-02-12T16:28:21.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036117900000000001, 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321545+0000", 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321545+0000", 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321543+0000", 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1450999999999999e-05 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321554+0000", 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6314999999999999e-05 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321571+0000", 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4469999999999995e-06 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:28:21.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321580+0000", 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000122378 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321702+0000", 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7322e-05 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:28:21.321730+0000", 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018501699999999999 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-02-12T16:28:21.338 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 3 2024-02-12T16:28:21.339 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:28:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:21.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:21.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:21.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:21.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:21.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:21.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:21.597 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:21.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:21 smithi123 bash[19119]: cluster 2024-02-12T16:28:19.996564+0000 mgr.y (mgr.14146) 1303 : cluster [DBG] pgmap v1810: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s; 1784/41316 objects misplaced (4.318%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:28:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:21 smithi123 bash[19119]: cluster 2024-02-12T16:28:20.169491+0000 mon.a (mon.0) 2202 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:21 smithi123 bash[19119]: audit 2024-02-12T16:28:20.179768+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? v1:172.21.15.121:0/1817439051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:28:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:21 smithi123 bash[19119]: cluster 2024-02-12T16:28:20.180170+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e620: 12 total, 12 up, 12 in 2024-02-12T16:28:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:21 smithi123 bash[19119]: cluster 2024-02-12T16:28:20.672738+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e621: 12 total, 12 up, 12 in 2024-02-12T16:28:21.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:21 smithi121 bash[18489]: cluster 2024-02-12T16:28:19.996564+0000 mgr.y (mgr.14146) 1303 : cluster [DBG] pgmap v1810: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 972 op/s; 1784/41316 objects misplaced (4.318%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:28:21.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:21 smithi121 bash[18489]: cluster 2024-02-12T16:28:20.169491+0000 mon.a (mon.0) 2202 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:21 smithi121 bash[18489]: audit 2024-02-12T16:28:20.179768+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? v1:172.21.15.121:0/1817439051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:28:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:21 smithi121 bash[18489]: cluster 2024-02-12T16:28:20.180170+0000 mon.a (mon.0) 2204 : cluster [DBG] osdmap e620: 12 total, 12 up, 12 in 2024-02-12T16:28:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:21 smithi121 bash[18489]: cluster 2024-02-12T16:28:20.672738+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e621: 12 total, 12 up, 12 in 2024-02-12T16:28:21.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:21.728 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:21.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:21.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:21.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:21.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:21.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:21.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:22.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:22.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:22.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-12T16:28:22.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:22.284 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:22.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:22.420 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:22.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:22.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:28:22.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:22.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:22.683 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:22.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:22.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:22.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:22.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:22.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:23.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:23.101 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:23.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:23.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:23.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:28:23.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:23 smithi121 bash[18489]: cluster 2024-02-12T16:28:21.997014+0000 mgr.y (mgr.14146) 1304 : cluster [DBG] pgmap v1813: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 1211/33944 objects misplaced (3.568%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:28:23.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:23.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:23.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:23.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:23.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:23 smithi123 bash[19119]: cluster 2024-02-12T16:28:21.997014+0000 mgr.y (mgr.14146) 1304 : cluster [DBG] pgmap v1813: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 1211/33944 objects misplaced (3.568%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:28:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:23.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:23.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:23.544 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:23.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:23 smithi169 bash[19604]: cluster 2024-02-12T16:28:21.997014+0000 mgr.y (mgr.14146) 1304 : cluster [DBG] pgmap v1813: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 1211/33944 objects misplaced (3.568%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:28:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:23.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:23.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:23.683 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:23.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:23.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:23.826 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:23.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:23.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:23.959 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:23.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:24.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:24.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:24.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:24.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:24.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:24.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:24.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:24.351 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:24.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:24.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:24.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:24.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:24.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:24.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:24.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:24.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:24.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:24.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:24.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:24.815 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:24.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:24.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:24.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:24.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:25.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:25.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:25.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:25.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:25.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:25.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:25.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:25 smithi121 bash[18489]: cluster 2024-02-12T16:28:23.997410+0000 mgr.y (mgr.14146) 1305 : cluster [DBG] pgmap v1814: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 972 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1211/31104 objects misplaced (3.893%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:28:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:25.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:25.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:25.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:25.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:25.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:25 smithi169 bash[19604]: cluster 2024-02-12T16:28:23.997410+0000 mgr.y (mgr.14146) 1305 : cluster [DBG] pgmap v1814: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 972 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1211/31104 objects misplaced (3.893%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:28:25.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:25.573 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:25 smithi123 bash[19119]: cluster 2024-02-12T16:28:23.997410+0000 mgr.y (mgr.14146) 1305 : cluster [DBG] pgmap v1814: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 972 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1211/31104 objects misplaced (3.893%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:28:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:25.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:25.701 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:25.725 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:25.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:25.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:25.863 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:25.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:25.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:25.999 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:26.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:26.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:26.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:26.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:26.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:26.302 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:26.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:26.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:26.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:26.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:26.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:26.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:26.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:26.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:26.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:26.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:26.837 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:26.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:26.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:26 smithi123 bash[19119]: cluster 2024-02-12T16:28:25.675704+0000 mon.a (mon.0) 2206 : cluster [DBG] osdmap e622: 12 total, 12 up, 12 in 2024-02-12T16:28:26.850 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:26 smithi123 bash[19119]: cluster 2024-02-12T16:28:25.997802+0000 mgr.y (mgr.14146) 1306 : cluster [DBG] pgmap v1816: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 847 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 940/27104 objects misplaced (3.468%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:28:26.867 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:26.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:26.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:26.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:26 smithi121 bash[18489]: cluster 2024-02-12T16:28:25.675704+0000 mon.a (mon.0) 2206 : cluster [DBG] osdmap e622: 12 total, 12 up, 12 in 2024-02-12T16:28:26.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:26 smithi121 bash[18489]: cluster 2024-02-12T16:28:25.997802+0000 mgr.y (mgr.14146) 1306 : cluster [DBG] pgmap v1816: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 847 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 940/27104 objects misplaced (3.468%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:28:26.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:27.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:27.013 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:28:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:26 smithi169 bash[19604]: cluster 2024-02-12T16:28:25.675704+0000 mon.a (mon.0) 2206 : cluster [DBG] osdmap e622: 12 total, 12 up, 12 in 2024-02-12T16:28:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:26 smithi169 bash[19604]: cluster 2024-02-12T16:28:25.997802+0000 mgr.y (mgr.14146) 1306 : cluster [DBG] pgmap v1816: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 847 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 940/27104 objects misplaced (3.468%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:28:27.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:27.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:27.296 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:27.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:27.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:27.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:27.571 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:27.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:27.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:27.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:27.829 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:27.866 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:27.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:27.962 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:27.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:28.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:28.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:28.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:28.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:28.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:28.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:28.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:28.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:28.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:28.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:28.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:28.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:28.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:28.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:28.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:28.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:28.845 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:28.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:28.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:28.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:28.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:29.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:29.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:29.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:29.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:29.250 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:29.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:29.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:29 smithi169 bash[19604]: cluster 2024-02-12T16:28:27.998278+0000 mgr.y (mgr.14146) 1307 : cluster [DBG] pgmap v1817: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 681 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 696/21806 objects misplaced (3.192%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:28:29.302 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:29 smithi121 bash[18489]: cluster 2024-02-12T16:28:27.998278+0000 mgr.y (mgr.14146) 1307 : cluster [DBG] pgmap v1817: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 681 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 696/21806 objects misplaced (3.192%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:28:29.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:29.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:29 smithi123 bash[19119]: cluster 2024-02-12T16:28:27.998278+0000 mgr.y (mgr.14146) 1307 : cluster [DBG] pgmap v1817: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 681 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 696/21806 objects misplaced (3.192%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:28:29.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:29.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:29.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:29.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:29.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:29.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:29.651 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:29.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:29.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:29.804 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:29.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:29.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:29.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:29.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:30.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:30.100 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:30.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:30.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:30.204 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:30.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:30.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:30.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:30.361 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:30 smithi169 bash[19604]: audit 2024-02-12T16:28:29.675434+0000 mon.a (mon.0) 2207 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:30.361 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:30 smithi169 bash[19604]: audit 2024-02-12T16:28:30.037617+0000 mon.a (mon.0) 2208 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:28:30.361 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:30 smithi169 bash[19604]: audit 2024-02-12T16:28:30.043194+0000 mon.a (mon.0) 2209 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:28:30.361 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:30 smithi169 bash[19604]: audit 2024-02-12T16:28:30.049475+0000 mon.a (mon.0) 2210 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:28:30.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:30 smithi121 bash[18489]: audit 2024-02-12T16:28:29.675434+0000 mon.a (mon.0) 2207 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:30.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:30 smithi121 bash[18489]: audit 2024-02-12T16:28:30.037617+0000 mon.a (mon.0) 2208 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:28:30.364 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:30 smithi121 bash[18489]: audit 2024-02-12T16:28:30.043194+0000 mon.a (mon.0) 2209 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:28:30.364 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:30 smithi121 bash[18489]: audit 2024-02-12T16:28:30.049475+0000 mon.a (mon.0) 2210 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:28:30.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:30.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:30 smithi123 bash[19119]: audit 2024-02-12T16:28:29.675434+0000 mon.a (mon.0) 2207 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:28:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:30 smithi123 bash[19119]: audit 2024-02-12T16:28:30.037617+0000 mon.a (mon.0) 2208 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:28:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:30 smithi123 bash[19119]: audit 2024-02-12T16:28:30.043194+0000 mon.a (mon.0) 2209 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:28:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:30 smithi123 bash[19119]: audit 2024-02-12T16:28:30.049475+0000 mon.a (mon.0) 2210 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:28:30.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:30.513 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:30.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:30.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:30.642 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:30.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:30.749 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:30.773 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:30.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:30.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:30.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:30.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:31.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:31.038 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:31.096 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:28:31.096 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:28:31.096 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:28:31.096 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:28:31.096 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:28:31.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:31.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:31 smithi169 bash[19604]: cluster 2024-02-12T16:28:29.998691+0000 mgr.y (mgr.14146) 1308 : cluster [DBG] pgmap v1818: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 593 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 696/18982 objects misplaced (3.667%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:28:31.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:31 smithi169 bash[19604]: cluster 2024-02-12T16:28:30.202196+0000 osd.4 (osd.4) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:31.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:31 smithi169 bash[19604]: cluster 2024-02-12T16:28:30.202236+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:31.156 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:31 smithi169 bash[19604]: cluster 2024-02-12T16:28:30.676278+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e623: 12 total, 12 up, 12 in 2024-02-12T16:28:31.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:31.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:31.316 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:31 smithi123 bash[19119]: cluster 2024-02-12T16:28:29.998691+0000 mgr.y (mgr.14146) 1308 : cluster [DBG] pgmap v1818: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 593 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 696/18982 objects misplaced (3.667%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:28:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:31 smithi123 bash[19119]: cluster 2024-02-12T16:28:30.202196+0000 osd.4 (osd.4) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:31 smithi123 bash[19119]: cluster 2024-02-12T16:28:30.202236+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:31 smithi123 bash[19119]: cluster 2024-02-12T16:28:30.676278+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e623: 12 total, 12 up, 12 in 2024-02-12T16:28:31.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:31 smithi121 bash[18489]: cluster 2024-02-12T16:28:29.998691+0000 mgr.y (mgr.14146) 1308 : cluster [DBG] pgmap v1818: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 593 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 696/18982 objects misplaced (3.667%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:28:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:31 smithi121 bash[18489]: cluster 2024-02-12T16:28:30.202196+0000 osd.4 (osd.4) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:31 smithi121 bash[18489]: cluster 2024-02-12T16:28:30.202236+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:31 smithi121 bash[18489]: cluster 2024-02-12T16:28:30.676278+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e623: 12 total, 12 up, 12 in 2024-02-12T16:28:31.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:31.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:31.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:31.571 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:31.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:31.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:31.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:31.829 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:31.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-12T16:28:31.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:31.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:31.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:32.070 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:32.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:32.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:32.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:32.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:32.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:32.338 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:32.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:32.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:32.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:32.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:32.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:32.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:32.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:32.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:32.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:32.763 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:32.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:32.868 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:32.890 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:32.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:32.992 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:33.029 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:33.102 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:28:33.106 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:28:33.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:33 smithi123 bash[19119]: cluster 2024-02-12T16:28:31.999170+0000 mgr.y (mgr.14146) 1309 : cluster [DBG] pgmap v1820: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 391 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 279/12500 objects misplaced (2.232%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:28:33.158 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:33.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:33.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:33.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:33.426 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:33 smithi169 bash[19604]: cluster 2024-02-12T16:28:31.999170+0000 mgr.y (mgr.14146) 1309 : cluster [DBG] pgmap v1820: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 391 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 279/12500 objects misplaced (2.232%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:28:33.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:33.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:33 smithi121 bash[18489]: cluster 2024-02-12T16:28:31.999170+0000 mgr.y (mgr.14146) 1309 : cluster [DBG] pgmap v1820: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 391 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 279/12500 objects misplaced (2.232%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:28:33.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:33.590 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:33.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:33.719 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:33.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:33.854 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:33.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:33.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:33.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:34.086 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:34 smithi123 bash[19119]: audit 2024-02-12T16:28:33.083235+0000 mgr.y (mgr.14146) 1310 : audit [DBG] from='client.26173 v1:172.21.15.121:0/2591016438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:34.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:34.135 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:34.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:34.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:34.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:34.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:34.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:34 smithi121 bash[18489]: audit 2024-02-12T16:28:33.083235+0000 mgr.y (mgr.14146) 1310 : audit [DBG] from='client.26173 v1:172.21.15.121:0/2591016438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:34.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:34.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:34.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:34.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:34 smithi169 bash[19604]: audit 2024-02-12T16:28:33.083235+0000 mgr.y (mgr.14146) 1310 : audit [DBG] from='client.26173 v1:172.21.15.121:0/2591016438' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:34.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:34.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:34.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:34.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:34.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:34.741 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1820,"stamp":"2024-02-12T16:28:31.998870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409534488,"num_objects":6250,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":279,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":332526,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3041,"num_bytes_recovered":199294976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5817308,"kb_used_data":935356,"kb_used_omap":91,"kb_used_meta":4881764,"kb_avail":1119124516,"statfs":{"total":1151940427776,"available":1145983504384,"internally_reserved":0,"allocated":957804544,"data_stored":952570888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":4998926352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,7,2,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-609615872,"num_objects":-9302,"num_object_clones":0,"num_object_copies":-18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-932,"num_objects_unfound":0,"num_objects_dirty":-9302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9302,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":15138816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001678"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":116,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:25.677522+0000","last_change":"2024-02-12T16:28:06.991266+0000","last_active":"2024-02-12T16:28:25.677522+0000","last_peered":"2024-02-12T16:28:25.677522+0000","last_clean":"2024-02-12T16:28:25.677522+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:28:25.677522+0000","last_undegraded":"2024-02-12T16:28:25.677522+0000","last_fullsized":"2024-02-12T16:28:25.677522+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:06.991147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:28:06.991147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.4","version":"622'20587","reported_seq":20674,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:28.677721+0000","last_change":"2024-02-12T16:25:46.665174+0000","last_active":"2024-02-12T16:28:28.677721+0000","last_peered":"2024-02-12T16:28:28.677721+0000","last_clean":"2024-02-12T16:28:28.677721+0000","last_became_active":"2024-02-12T16:24:09.648523+0000","last_became_peered":"2024-02-12T16:24:09.648523+0000","last_unstale":"2024-02-12T16:28:28.677721+0000","last_undegraded":"2024-02-12T16:28:28.677721+0000","last_fullsized":"2024-02-12T16:28:28.677721+0000","mapping_epoch":547,"log_start":"579'10500","ondisk_log_start":"579'10500","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10057","last_scrub_stamp":"2024-02-12T16:25:46.665055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:46.665055+0000","log_size":10087,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28639232,"num_objects":437,"num_object_clones":0,"num_object_copies":874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20587,"num_write_kb":672768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":228,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:25.679726+0000","last_change":"2024-02-12T16:28:20.038276+0000","last_active":"2024-02-12T16:28:25.679726+0000","last_peered":"2024-02-12T16:28:25.679726+0000","last_clean":"2024-02-12T16:28:25.679726+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:28:25.679726+0000","last_undegraded":"2024-02-12T16:28:25.679726+0000","last_fullsized":"2024-02-12T16:28:25.679726+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:40.282662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:40.282662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.1","version":"622'20915","reported_seq":21012,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:29.323218+0000","last_change":"2024-02-12T16:28:00.438914+0000","last_active":"2024-02-12T16:28:29.323218+0000","last_peered":"2024-02-12T16:28:29.323218+0000","last_clean":"2024-02-12T16:28:29.323218+0000","last_became_active":"2024-02-12T16:24:10.018691+0000","last_became_peered":"2024-02-12T16:24:10.018691+0000","last_unstale":"2024-02-12T16:28:29.323218+0000","last_undegraded":"2024-02-12T16:28:29.323218+0000","last_fullsized":"2024-02-12T16:28:29.323218+0000","mapping_epoch":547,"log_start":"582'10900","ondisk_log_start":"582'10900","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"611'18916","last_scrub_stamp":"2024-02-12T16:28:00.438841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:28:00.438841+0000","log_size":10015,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29556736,"num_objects":451,"num_object_clones":0,"num_object_copies":902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20915,"num_write_kb":683712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":304,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.674844+0000","last_change":"2024-02-12T16:24:47.872602+0000","last_active":"2024-02-12T16:28:20.674844+0000","last_peered":"2024-02-12T16:28:20.674844+0000","last_clean":"2024-02-12T16:28:20.674844+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:28:20.674844+0000","last_undegraded":"2024-02-12T16:28:20.674844+0000","last_fullsized":"2024-02-12T16:28:20.674844+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:47.872500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:24:47.872500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"623'20817","reported_seq":24467,"reported_epoch":623,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:28:31.746135+0000","last_change":"2024-02-12T16:25:58.008531+0000","last_active":"2024-02-12T16:28:31.746135+0000","last_peered":"2024-02-12T16:28:31.746135+0000","last_clean":"2024-02-12T16:25:54.194068+0000","last_became_active":"2024-02-12T16:25:57.932697+0000","last_became_peered":"2024-02-12T16:25:57.932697+0000","last_unstale":"2024-02-12T16:28:31.746135+0000","last_undegraded":"2024-02-12T16:28:31.746135+0000","last_fullsized":"2024-02-12T16:28:31.746135+0000","mapping_epoch":581,"log_start":"582'10800","ondisk_log_start":"582'10800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10027","last_scrub_stamp":"2024-02-12T16:25:45.765333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.765333+0000","log_size":10017,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18284544,"num_objects":279,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":107,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20817,"num_write_kb":675072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1775,"num_bytes_recovered":116326400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":279}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":709,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.672934+0000","last_change":"2024-02-12T16:25:07.263767+0000","last_active":"2024-02-12T16:28:20.672934+0000","last_peered":"2024-02-12T16:28:20.672934+0000","last_clean":"2024-02-12T16:28:20.672934+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:28:20.672934+0000","last_undegraded":"2024-02-12T16:28:20.672934+0000","last_fullsized":"2024-02-12T16:28:20.672934+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.263688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.263688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.3","version":"622'21058","reported_seq":21140,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:30.426112+0000","last_change":"2024-02-12T16:24:09.648426+0000","last_active":"2024-02-12T16:28:30.426112+0000","last_peered":"2024-02-12T16:28:30.426112+0000","last_clean":"2024-02-12T16:28:30.426112+0000","last_became_active":"2024-02-12T16:24:09.648139+0000","last_became_peered":"2024-02-12T16:24:09.648139+0000","last_unstale":"2024-02-12T16:28:30.426112+0000","last_undegraded":"2024-02-12T16:28:30.426112+0000","last_fullsized":"2024-02-12T16:28:30.426112+0000","mapping_epoch":547,"log_start":"582'11000","ondisk_log_start":"582'11000","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165824,"num_objects":384,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21058,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":347,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.679890+0000","last_change":"2024-02-12T16:28:19.039256+0000","last_active":"2024-02-12T16:28:20.679890+0000","last_peered":"2024-02-12T16:28:20.679890+0000","last_clean":"2024-02-12T16:28:20.679890+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:28:20.679890+0000","last_undegraded":"2024-02-12T16:28:20.679890+0000","last_fullsized":"2024-02-12T16:28:20.679890+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:45.967340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:45.967340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"622'20453","reported_seq":20540,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:30.283782+0000","last_change":"2024-02-12T16:25:50.727730+0000","last_active":"2024-02-12T16:28:30.283782+0000","last_peered":"2024-02-12T16:28:30.283782+0000","last_clean":"2024-02-12T16:28:30.283782+0000","last_became_active":"2024-02-12T16:24:09.649607+0000","last_became_peered":"2024-02-12T16:24:09.649607+0000","last_unstale":"2024-02-12T16:28:30.283782+0000","last_undegraded":"2024-02-12T16:28:30.283782+0000","last_fullsized":"2024-02-12T16:28:30.283782+0000","mapping_epoch":547,"log_start":"579'10400","ondisk_log_start":"579'10400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10162","last_scrub_stamp":"2024-02-12T16:25:50.727611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.727611+0000","log_size":10053,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23789568,"num_objects":363,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20453,"num_write_kb":666112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":295,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.674993+0000","last_change":"2024-02-12T16:25:07.944563+0000","last_active":"2024-02-12T16:28:20.674993+0000","last_peered":"2024-02-12T16:28:20.674993+0000","last_clean":"2024-02-12T16:28:20.674993+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:28:20.674993+0000","last_undegraded":"2024-02-12T16:28:20.674993+0000","last_fullsized":"2024-02-12T16:28:20.674993+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.944451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.944451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"622'20270","reported_seq":20361,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:30.446577+0000","last_change":"2024-02-12T16:26:10.447550+0000","last_active":"2024-02-12T16:28:30.446577+0000","last_peered":"2024-02-12T16:28:30.446577+0000","last_clean":"2024-02-12T16:28:30.446577+0000","last_became_active":"2024-02-12T16:24:09.647095+0000","last_became_peered":"2024-02-12T16:24:09.647095+0000","last_unstale":"2024-02-12T16:28:30.446577+0000","last_undegraded":"2024-02-12T16:28:30.446577+0000","last_fullsized":"2024-02-12T16:28:30.446577+0000","mapping_epoch":547,"log_start":"579'10200","ondisk_log_start":"579'10200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10070,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20270,"num_write_kb":660480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":305,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.674870+0000","last_change":"2024-02-12T16:28:14.154056+0000","last_active":"2024-02-12T16:28:20.674870+0000","last_peered":"2024-02-12T16:28:20.674870+0000","last_clean":"2024-02-12T16:28:20.674870+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:28:20.674870+0000","last_undegraded":"2024-02-12T16:28:20.674870+0000","last_fullsized":"2024-02-12T16:28:20.674870+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:37.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:25:37.899168+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"622'20670","reported_seq":20806,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:27.068945+0000","last_change":"2024-02-12T16:28:17.149755+0000","last_active":"2024-02-12T16:28:27.068945+0000","last_peered":"2024-02-12T16:28:27.068945+0000","last_clean":"2024-02-12T16:28:27.068945+0000","last_became_active":"2024-02-12T16:24:09.651253+0000","last_became_peered":"2024-02-12T16:24:09.651253+0000","last_unstale":"2024-02-12T16:28:27.068945+0000","last_undegraded":"2024-02-12T16:28:27.068945+0000","last_fullsized":"2024-02-12T16:28:27.068945+0000","mapping_epoch":547,"log_start":"579'10600","ondisk_log_start":"579'10600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:24:08.619649+0000","log_size":10070,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39583744,"num_objects":604,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20670,"num_write_kb":680768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":164,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.671127+0000","last_change":"2024-02-12T16:25:54.579343+0000","last_active":"2024-02-12T16:28:20.671127+0000","last_peered":"2024-02-12T16:28:20.671127+0000","last_clean":"2024-02-12T16:28:20.671127+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:28:20.671127+0000","last_undegraded":"2024-02-12T16:28:20.671127+0000","last_fullsized":"2024-02-12T16:28:20.671127+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:54.579248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:54.579248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"623'20748","reported_seq":20841,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-02-12T16:28:31.381148+0000","last_change":"2024-02-12T16:27:12.609604+0000","last_active":"2024-02-12T16:28:31.381148+0000","last_peered":"2024-02-12T16:28:31.381148+0000","last_clean":"2024-02-12T16:28:31.381148+0000","last_became_active":"2024-02-12T16:24:09.647187+0000","last_became_peered":"2024-02-12T16:24:09.647187+0000","last_unstale":"2024-02-12T16:28:31.381148+0000","last_undegraded":"2024-02-12T16:28:31.381148+0000","last_fullsized":"2024-02-12T16:28:31.381148+0000","mapping_epoch":547,"log_start":"582'10700","ondisk_log_start":"582'10700","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'15509","last_scrub_stamp":"2024-02-12T16:27:12.609496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:12.609496+0000","log_size":10048,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20447232,"num_objects":312,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20748,"num_write_kb":673920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":735,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.705855+0000","last_change":"2024-02-12T16:28:19.557009+0000","last_active":"2024-02-12T16:28:20.705855+0000","last_peered":"2024-02-12T16:28:20.705855+0000","last_clean":"2024-02-12T16:28:20.705855+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:28:20.705855+0000","last_undegraded":"2024-02-12T16:28:20.705855+0000","last_fullsized":"2024-02-12T16:28:20.705855+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:04.914304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:04.914304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"622'21032","reported_seq":21124,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:28.280758+0000","last_change":"2024-02-12T16:26:57.827400+0000","last_active":"2024-02-12T16:28:28.280758+0000","last_peered":"2024-02-12T16:28:28.280758+0000","last_clean":"2024-02-12T16:28:28.280758+0000","last_became_active":"2024-02-12T16:24:09.650120+0000","last_became_peered":"2024-02-12T16:24:09.650120+0000","last_unstale":"2024-02-12T16:28:28.280758+0000","last_undegraded":"2024-02-12T16:28:28.280758+0000","last_fullsized":"2024-02-12T16:28:28.280758+0000","mapping_epoch":547,"log_start":"582'11000","ondisk_log_start":"582'11000","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'14868","last_scrub_stamp":"2024-02-12T16:26:57.827287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:57.827287+0000","log_size":10032,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30670848,"num_objects":468,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21032,"num_write_kb":688000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":110,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.705877+0000","last_change":"2024-02-12T16:26:43.025974+0000","last_active":"2024-02-12T16:28:20.705877+0000","last_peered":"2024-02-12T16:28:20.705877+0000","last_clean":"2024-02-12T16:28:20.705877+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:28:20.705877+0000","last_undegraded":"2024-02-12T16:28:20.705877+0000","last_fullsized":"2024-02-12T16:28:20.705877+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:43.025867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:43.025867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"622'20846","reported_seq":23477,"reported_epoch":622,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:28:30.439419+0000","last_change":"2024-02-12T16:26:41.328312+0000","last_active":"2024-02-12T16:28:30.439419+0000","last_peered":"2024-02-12T16:28:30.439419+0000","last_clean":"2024-02-12T16:26:09.048189+0000","last_became_active":"2024-02-12T16:26:11.222461+0000","last_became_peered":"2024-02-12T16:26:11.222461+0000","last_unstale":"2024-02-12T16:28:30.439419+0000","last_undegraded":"2024-02-12T16:28:30.439419+0000","last_fullsized":"2024-02-12T16:28:30.439419+0000","mapping_epoch":585,"log_start":"582'10800","ondisk_log_start":"582'10800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'9817","last_scrub_stamp":"2024-02-12T16:25:42.181950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:42.181950+0000","log_size":10046,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23592960,"num_objects":360,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":172,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20846,"num_write_kb":678592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1266,"num_bytes_recovered":82968576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[0,10],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,10","objects":360}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":136,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.671203+0000","last_change":"2024-02-12T16:28:17.599311+0000","last_active":"2024-02-12T16:28:20.671203+0000","last_peered":"2024-02-12T16:28:20.671203+0000","last_clean":"2024-02-12T16:28:20.671203+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:28:20.671203+0000","last_undegraded":"2024-02-12T16:28:20.671203+0000","last_fullsized":"2024-02-12T16:28:20.671203+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:52.631321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:52.631321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"623'21215","reported_seq":21307,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-02-12T16:28:31.297083+0000","last_change":"2024-02-12T16:25:47.403836+0000","last_active":"2024-02-12T16:28:31.297083+0000","last_peered":"2024-02-12T16:28:31.297083+0000","last_clean":"2024-02-12T16:28:31.297083+0000","last_became_active":"2024-02-12T16:24:09.649590+0000","last_became_peered":"2024-02-12T16:24:09.649590+0000","last_unstale":"2024-02-12T16:28:31.297083+0000","last_undegraded":"2024-02-12T16:28:31.297083+0000","last_fullsized":"2024-02-12T16:28:31.297083+0000","mapping_epoch":547,"log_start":"582'11200","ondisk_log_start":"582'11200","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10313","last_scrub_stamp":"2024-02-12T16:25:47.403730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.403730+0000","log_size":10015,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19464192,"num_objects":297,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21215,"num_write_kb":688384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":308,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.679815+0000","last_change":"2024-02-12T16:25:45.967785+0000","last_active":"2024-02-12T16:28:20.679815+0000","last_peered":"2024-02-12T16:28:20.679815+0000","last_clean":"2024-02-12T16:28:20.679815+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:28:20.679815+0000","last_undegraded":"2024-02-12T16:28:20.679815+0000","last_fullsized":"2024-02-12T16:28:20.679815+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:45.967726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.967726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"623'20436","reported_seq":20529,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-02-12T16:28:31.388515+0000","last_change":"2024-02-12T16:27:09.549045+0000","last_active":"2024-02-12T16:28:31.388515+0000","last_peered":"2024-02-12T16:28:31.388515+0000","last_clean":"2024-02-12T16:28:31.388515+0000","last_became_active":"2024-02-12T16:24:10.017647+0000","last_became_peered":"2024-02-12T16:24:10.017647+0000","last_unstale":"2024-02-12T16:28:31.388515+0000","last_undegraded":"2024-02-12T16:28:31.388515+0000","last_fullsized":"2024-02-12T16:28:31.388515+0000","mapping_epoch":547,"log_start":"582'10400","ondisk_log_start":"582'10400","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'15168","last_scrub_stamp":"2024-02-12T16:27:09.548944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:27:09.548944+0000","log_size":10036,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18481152,"num_objects":282,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20436,"num_write_kb":662976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":112,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.679360+0000","last_change":"2024-02-12T16:26:59.766851+0000","last_active":"2024-02-12T16:28:20.679360+0000","last_peered":"2024-02-12T16:28:20.679360+0000","last_clean":"2024-02-12T16:28:20.679360+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:28:20.679360+0000","last_undegraded":"2024-02-12T16:28:20.679360+0000","last_fullsized":"2024-02-12T16:28:20.679360+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:59.766744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:59.766744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"622'20858","reported_seq":20946,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:29.786926+0000","last_change":"2024-02-12T16:25:40.938617+0000","last_active":"2024-02-12T16:28:29.786926+0000","last_peered":"2024-02-12T16:28:29.786926+0000","last_clean":"2024-02-12T16:28:29.786926+0000","last_became_active":"2024-02-12T16:24:09.649655+0000","last_became_peered":"2024-02-12T16:24:09.649655+0000","last_unstale":"2024-02-12T16:28:29.786926+0000","last_undegraded":"2024-02-12T16:28:29.786926+0000","last_fullsized":"2024-02-12T16:28:29.786926+0000","mapping_epoch":547,"log_start":"582'10800","ondisk_log_start":"582'10800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'9693","last_scrub_stamp":"2024-02-12T16:25:40.938531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:40.938531+0000","log_size":10058,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27459608,"num_objects":420,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20858,"num_write_kb":680833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":710,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.705836+0000","last_change":"2024-02-12T16:26:48.943869+0000","last_active":"2024-02-12T16:28:20.705836+0000","last_peered":"2024-02-12T16:28:20.705836+0000","last_clean":"2024-02-12T16:28:20.705836+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:28:20.705836+0000","last_undegraded":"2024-02-12T16:28:20.705836+0000","last_fullsized":"2024-02-12T16:28:20.705836+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:48.943785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:48.943785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"623'21144","reported_seq":21230,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-02-12T16:28:31.764799+0000","last_change":"2024-02-12T16:25:50.062621+0000","last_active":"2024-02-12T16:28:31.764799+0000","last_peered":"2024-02-12T16:28:31.764799+0000","last_clean":"2024-02-12T16:28:31.764799+0000","last_became_active":"2024-02-12T16:24:09.647617+0000","last_became_peered":"2024-02-12T16:24:09.647617+0000","last_unstale":"2024-02-12T16:28:31.764799+0000","last_undegraded":"2024-02-12T16:28:31.764799+0000","last_fullsized":"2024-02-12T16:28:31.764799+0000","mapping_epoch":547,"log_start":"582'11100","ondisk_log_start":"582'11100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10397","last_scrub_stamp":"2024-02-12T16:25:50.062524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.062524+0000","log_size":10044,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21144,"num_write_kb":686464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":298,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.676256+0000","last_change":"2024-02-12T16:24:50.575354+0000","last_active":"2024-02-12T16:28:20.676256+0000","last_peered":"2024-02-12T16:28:20.676256+0000","last_clean":"2024-02-12T16:28:20.676256+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:28:20.676256+0000","last_undegraded":"2024-02-12T16:28:20.676256+0000","last_fullsized":"2024-02-12T16:28:20.676256+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.575287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.575287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"622'20656","reported_seq":20743,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:27.065772+0000","last_change":"2024-02-12T16:26:09.203006+0000","last_active":"2024-02-12T16:28:27.065772+0000","last_peered":"2024-02-12T16:28:27.065772+0000","last_clean":"2024-02-12T16:28:27.065772+0000","last_became_active":"2024-02-12T16:24:09.650861+0000","last_became_peered":"2024-02-12T16:24:09.650861+0000","last_unstale":"2024-02-12T16:28:27.065772+0000","last_undegraded":"2024-02-12T16:28:27.065772+0000","last_fullsized":"2024-02-12T16:28:27.065772+0000","mapping_epoch":547,"log_start":"579'10600","ondisk_log_start":"579'10600","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'11415","last_scrub_stamp":"2024-02-12T16:26:09.202913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:26:09.202913+0000","log_size":10056,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20656,"num_write_kb":680128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":51,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.672951+0000","last_change":"2024-02-12T16:26:10.388710+0000","last_active":"2024-02-12T16:28:20.672951+0000","last_peered":"2024-02-12T16:28:20.672951+0000","last_clean":"2024-02-12T16:28:20.672951+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:28:20.672951+0000","last_undegraded":"2024-02-12T16:28:20.672951+0000","last_fullsized":"2024-02-12T16:28:20.672951+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:10.388515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:10.388515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"623'20821","reported_seq":20909,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-02-12T16:28:31.321408+0000","last_change":"2024-02-12T16:25:49.810317+0000","last_active":"2024-02-12T16:28:31.321408+0000","last_peered":"2024-02-12T16:28:31.321408+0000","last_clean":"2024-02-12T16:28:31.321408+0000","last_became_active":"2024-02-12T16:24:10.019266+0000","last_became_peered":"2024-02-12T16:24:10.019266+0000","last_unstale":"2024-02-12T16:28:31.321408+0000","last_undegraded":"2024-02-12T16:28:31.321408+0000","last_fullsized":"2024-02-12T16:28:31.321408+0000","mapping_epoch":547,"log_start":"582'10800","ondisk_log_start":"582'10800","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'10246","last_scrub_stamp":"2024-02-12T16:25:49.810203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:24:08.619649+0000","last_clean_scrub_stamp":"2024-02-12T16:25:49.810203+0000","log_size":10021,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20774912,"num_objects":317,"num_object_clones":0,"num_object_copies":634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20821,"num_write_kb":676416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":287,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:25.679778+0000","last_change":"2024-02-12T16:25:03.361871+0000","last_active":"2024-02-12T16:28:25.679778+0000","last_peered":"2024-02-12T16:28:25.679778+0000","last_clean":"2024-02-12T16:28:25.679778+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:28:25.679778+0000","last_undegraded":"2024-02-12T16:28:25.679778+0000","last_fullsized":"2024-02-12T16:28:25.679778+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:25:03.361757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":107,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.676306+0000","last_change":"2024-02-12T16:25:59.785619+0000","last_active":"2024-02-12T16:28:20.676306+0000","last_peered":"2024-02-12T16:28:20.676306+0000","last_clean":"2024-02-12T16:28:20.676306+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:28:20.676306+0000","last_undegraded":"2024-02-12T16:28:20.676306+0000","last_fullsized":"2024-02-12T16:28:20.676306+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:59.785496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:25:59.785496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":105,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.672846+0000","last_change":"2024-02-12T16:24:54.540925+0000","last_active":"2024-02-12T16:28:20.672846+0000","last_peered":"2024-02-12T16:28:20.672846+0000","last_clean":"2024-02-12T16:28:20.672846+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:28:20.672846+0000","last_undegraded":"2024-02-12T16:28:20.672846+0000","last_fullsized":"2024-02-12T16:28:20.672846+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:54.540855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:54.540855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":121,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-12T16:28:20.676273+0000","last_change":"2024-02-12T16:25:47.933452+0000","last_active":"2024-02-12T16:28:20.676273+0000","last_peered":"2024-02-12T16:28:20.676273+0000","last_clean":"2024-02-12T16:28:20.676273+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:28:20.676273+0000","last_undegraded":"2024-02-12T16:28:20.676273+0000","last_fullsized":"2024-02-12T16:28:20.676273+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:47.933341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.933341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":101,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-02-12T16:28:25.688846+0000","last_change":"2024-02-12T16:25:50.908791+0000","last_active":"2024-02-12T16:28:25.688846+0000","last_peered":"2024-02-12T16:28:25.688846+0000","last_clean":"2024-02-12T16:28:25.688846+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:28:25.688846+0000","last_undegraded":"2024-02-12T16:28:25.688846+0000","last_fullsized":"2024-02-12T16:28:25.688846+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.908692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.908692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":409534488,"num_objects":6250,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":279,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":332526,"num_write_kb":10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3041,"num_bytes_recovered":199294976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":869081088,"data_stored":869072944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958892,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639400,"kb_used_data":41504,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":93105752,"statfs":{"total":95995035648,"available":95340290048,"internally_reserved":0,"allocated":42500096,"data_stored":42062046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.82799999999999996,"15min":0.83899999999999997},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":5.6600000000000001,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.88600000000000001,"15min":0.89200000000000002},"min":{"1min":0.29599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.54100000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.89900000000000002,"15min":1.0309999999999999},"min":{"1min":0.36399999999999999,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":4.1520000000000001,"5min":4.6609999999999996,"15min":7.0110000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.875,"15min":1.0600000000000001},"min":{"1min":0.371,"5min":0.33900000000000002,"15min":0.28799999999999998},"max":{"1min":1.9670000000000001,"5min":4.2039999999999997,"15min":5.6920000000000002},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":0.92600000000000005,"15min":1.274},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.1639999999999997,"5min":5.343,"15min":5.343},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.94399999999999995,"15min":1.3420000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.617,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.86599999999999999,"15min":2.2450000000000001},"min":{"1min":0.39400000000000002,"5min":0.34699999999999998,"15min":0.29499999999999998},"max":{"1min":2.831,"5min":4.4400000000000004,"15min":368.48399999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.998,"5min":0.89500000000000002,"15min":2.1699999999999999},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.101,"5min":4.915,"15min":371.65100000000001},"last":0.623}]},{"osd":5,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":1.1020000000000001,"15min":2.3860000000000001},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35099999999999998},"max":{"1min":4.6159999999999997,"5min":5.04,"15min":368.51400000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":1.04,"15min":2.3639999999999999},"min":{"1min":0.46999999999999997,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":4.1929999999999996,"5min":5.1509999999999998,"15min":368.70499999999998},"last":4.0099999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.93600000000000005,"15min":0.97399999999999998},"min":{"1min":0.39200000000000002,"5min":0.38500000000000001,"15min":0.373},"max":{"1min":4.7370000000000001,"5min":5.141,"15min":5.141},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.0880000000000001,"15min":1.103},"min":{"1min":0.47699999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.5800000000000001,"5min":8.5709999999999997,"15min":8.5709999999999997},"last":1.931}]},{"osd":7,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.0700000000000001,"15min":1.1950000000000001},"min":{"1min":0.502,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.1279999999999999,"5min":5.2569999999999997,"15min":6.4370000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.93700000000000006,"15min":1.153},"min":{"1min":0.46400000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.2350000000000003,"5min":5.0430000000000001,"15min":5.1189999999999998},"last":0.57699999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":6.2320000000000002,"15min":2.7999999999999998},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":2.3679999999999999,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":6.3879999999999999,"15min":2.9940000000000002},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.34699999999999998},"max":{"1min":1.165,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.63600000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.071,"15min":1.268},"min":{"1min":0.58399999999999996,"5min":0.432,"15min":0.40400000000000003},"max":{"1min":1.9139999999999999,"5min":5.7430000000000003,"15min":10.743},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0900000000000001,"15min":1.381},"min":{"1min":0.60799999999999998,"5min":0.46800000000000003,"15min":0.39500000000000002},"max":{"1min":1.861,"5min":5.359,"15min":10.698},"last":0.68999999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:27:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":6.3710000000000004,"15min":2.9670000000000001},"min":{"1min":0.48899999999999999,"5min":0.46200000000000002,"15min":0.433},"max":{"1min":4.0599999999999996,"5min":536.245,"15min":536.245},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":6.4009999999999998,"15min":2.948},"min":{"1min":0.54700000000000004,"5min":0.443,"15min":0.443},"max":{"1min":2.544,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.65700000000000003}]}]},{"osd":10,"up_from":83,"seq":356482285941,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":387300,"kb_used_data":77788,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93357852,"statfs":{"total":95995035648,"available":95598440448,"internally_reserved":0,"allocated":79654912,"data_stored":79220934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.6070000000000002,"15min":1.5169999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":5.867,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.996,"5min":2.6499999999999999,"15min":1.5880000000000001},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.5060000000000002,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.44500000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.028,"15min":0.97999999999999998},"min":{"1min":0.314,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":5.9000000000000004,"5min":5.9000000000000004,"15min":19.866},"last":0.5},{"interface":"front","average":{"1min":1.431,"5min":0.94399999999999995,"15min":1.1000000000000001},"min":{"1min":0.33000000000000002,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":20.277999999999999},"last":0.623}]},{"osd":2,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":2.077,"5min":1.1419999999999999,"15min":0.98199999999999998},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.2430000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.1339999999999999,"15min":1.0009999999999999},"min":{"1min":0.40200000000000002,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85699999999999998,"15min":3.3359999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.32300000000000001},"max":{"1min":5.8109999999999999,"5min":5.8109999999999999,"15min":628.77099999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.98699999999999999,"15min":3.3100000000000001},"min":{"1min":0.34100000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.3070000000000004,"5min":7.3070000000000004,"15min":624.19299999999998},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":6.343,"5min":6.343,"15min":6.343},"last":3.625},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.099,"15min":1.1379999999999999},"min":{"1min":0.48199999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":7.0339999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":2.8929999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.028,"15min":3.3879999999999999},"min":{"1min":0.42099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":626.26599999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0049999999999999,"15min":3.4609999999999999},"min":{"1min":0.55900000000000005,"5min":0.376,"15min":0.376},"max":{"1min":6.8200000000000003,"5min":6.8200000000000003,"15min":625.90700000000004},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":0.94899999999999995,"15min":1.0700000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":7.3280000000000003,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.1579999999999999,"15min":1.1870000000000001},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":6.4539999999999997,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.65100000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.97799999999999998,"15min":1.1240000000000001},"min":{"1min":0.44400000000000001,"5min":0.436,"15min":0.372},"max":{"1min":5.758,"5min":5.758,"15min":5.758},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.135,"5min":0.98899999999999999,"15min":1.133},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.282,"5min":7.282,"15min":7.282},"last":0.73199999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1100000000000001,"15min":1.2250000000000001},"min":{"1min":0.505,"5min":0.47599999999999998,"15min":0.372},"max":{"1min":5.7809999999999997,"5min":5.7809999999999997,"15min":5.7809999999999997},"last":0.755},{"interface":"front","average":{"1min":1.048,"5min":1.0089999999999999,"15min":1.212},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":7.3479999999999999,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.76900000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:27:27 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.159,"15min":1.3100000000000001},"min":{"1min":0.59299999999999997,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":7.3689999999999998,"5min":7.3689999999999998,"15min":10.218999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.875,"5min":1.016,"15min":1.268},"min":{"1min":0.622,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.21,"5min":2.21,"15min":7.3250000000000002},"last":0.78600000000000003}]}]},{"osd":9,"up_from":74,"seq":317827580286,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347248,"kb_used_data":41832,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93397904,"statfs":{"total":95995035648,"available":95639453696,"internally_reserved":0,"allocated":42835968,"data_stored":42396046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.1930000000000001,"15min":1.1020000000000001},"min":{"1min":0.30599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.9560000000000004,"5min":7.9560000000000004,"15min":7.9560000000000004},"last":3.452},{"interface":"front","average":{"1min":2.419,"5min":1.2370000000000001,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":3.3679999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:28:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.1499999999999999,"15min":0.995},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.253},"max":{"1min":5.7750000000000004,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.0780000000000001,"15min":1.0669999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":5.7910000000000004,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":0.434}]},{"osd":2,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.956,"5min":1.268,"15min":1.4730000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":12.441000000000001,"5min":12.441000000000001,"15min":97.275000000000006},"last":1.488},{"interface":"front","average":{"1min":1.327,"5min":1.1279999999999999,"15min":1.3129999999999999},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.312},"max":{"1min":4.6239999999999997,"5min":6.1779999999999999,"15min":97.417000000000002},"last":2.8700000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.028,"15min":0.96899999999999997},"min":{"1min":0.35999999999999999,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":4.3940000000000001,"5min":4.569,"15min":4.569},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.1539999999999999,"15min":1.145},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.6589999999999998,"5min":4.8840000000000003,"15min":5.8010000000000002},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.304,"15min":1.1279999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":4.8780000000000001,"5min":6.726,"15min":6.726},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.149},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.9059999999999997,"5min":6.7350000000000003,"15min":7.3739999999999997},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0189999999999999,"15min":1.0880000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":4.6749999999999998,"5min":5.5250000000000004,"15min":7.3360000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1699999999999999,"15min":1.0660000000000001},"min":{"1min":0.51200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.4610000000000003,"5min":7.0609999999999999,"15min":7.0609999999999999},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.26,"15min":1.238},"min":{"1min":0.433,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":4.6399999999999997,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.216},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.7000000000000002,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.47199999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0449999999999999,"15min":1.077},"min":{"1min":0.42499999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.006,"5min":1.0680000000000001,"15min":1.1839999999999999},"min":{"1min":0.56100000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":7.25},"last":0.79500000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.2,"15min":1.2070000000000001},"min":{"1min":0.54800000000000004,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":3.0550000000000002,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.1619999999999999,"15min":1.2030000000000001},"min":{"1min":0.49199999999999999,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":3.8319999999999999,"5min":4.4509999999999996,"15min":5.8700000000000001},"last":0.83399999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.141,"15min":1.3260000000000001},"min":{"1min":0.60499999999999998,"5min":0.48999999999999999,"15min":0.45600000000000002},"max":{"1min":1.4379999999999999,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.0700000000000001,"15min":1.2829999999999999},"min":{"1min":0.52500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.8929999999999998,"5min":5.492,"15min":6.3639999999999999},"last":0.52100000000000002}]}]},{"osd":8,"up_from":65,"seq":279172874631,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372900,"kb_used_data":63388,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93372252,"statfs":{"total":95995035648,"available":95613186048,"internally_reserved":0,"allocated":64909312,"data_stored":64475342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.0289999999999999,"15min":0.88300000000000001},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":2.4279999999999999,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92400000000000004},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":7.1840000000000002,"5min":18.584,"15min":18.584},"last":0.63400000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":1.968,"15min":1.5329999999999999},"min":{"1min":0.33300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":15.981999999999999,"5min":69.956000000000003,"15min":69.956000000000003},"last":1.056},{"interface":"front","average":{"1min":1.988,"5min":1.97,"15min":1.625},"min":{"1min":0.44,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":15.817,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.83499999999999996},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0510000000000002,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91600000000000004,"15min":0.99299999999999999},"min":{"1min":0.36699999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.5569999999999999,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":4.4619999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.77500000000000002,"15min":1.71},"min":{"1min":0.378,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.89300000000000002,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.81299999999999994,"15min":1.6559999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":1.1519999999999999,"5min":5.0380000000000003,"15min":224.71000000000001},"last":1.1519999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.872,"15min":0.91100000000000003},"min":{"1min":0.39600000000000002,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":5.4059999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":5.4059999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.93799999999999994,"15min":1.016},"min":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.8170000000000002,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.226}]},{"osd":6,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.071,"15min":1.742},"min":{"1min":0.373,"5min":0.373,"15min":0.37},"max":{"1min":3.8050000000000002,"5min":5.3639999999999999,"15min":221.36699999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.99299999999999999,"15min":1.843},"min":{"1min":0.497,"5min":0.38,"15min":0.38},"max":{"1min":1.1919999999999999,"5min":4.4930000000000003,"15min":221.203},"last":1.1919999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.052,"15min":1.131},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":2.5619999999999998,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.1279999999999999,"15min":1.2310000000000001},"min":{"1min":0.54000000000000004,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":2.5779999999999998,"5min":4.2240000000000002,"15min":7.899},"last":1.256}]},{"osd":9,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.028,"15min":1.165},"min":{"1min":0.55300000000000005,"5min":0.371,"15min":0.35599999999999998},"max":{"1min":2.6419999999999999,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":1.284},{"interface":"front","average":{"1min":1.006,"5min":1.117,"15min":1.2390000000000001},"min":{"1min":0.60299999999999998,"5min":0.47599999999999998,"15min":0.44700000000000001},"max":{"1min":2.6709999999999998,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":1.306}]},{"osd":10,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.996,"15min":1.4299999999999999},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.2560000000000002,"5min":5.2560000000000002,"15min":109.92},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.032,"15min":1.474},"min":{"1min":0.504,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.6469999999999998,"5min":5.0250000000000004,"15min":110.249},"last":1.0429999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":1.1819999999999999},"min":{"1min":0.627,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":3.3109999999999999,"5min":3.3109999999999999,"15min":5.0670000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.88400000000000001,"15min":1.141},"min":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.327,"5min":3.327,"15min":4.6639999999999997},"last":1.0169999999999999}]}]},{"osd":7,"up_from":56,"seq":240518168974,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":343524,"kb_used_data":34012,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93401628,"statfs":{"total":95995035648,"available":95643267072,"internally_reserved":0,"allocated":34828288,"data_stored":34394318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":11.994,"5min":3.085,"15min":1.6419999999999999},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":187.33699999999999,"5min":187.33699999999999,"15min":187.33699999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":11.662000000000001,"5min":3.0800000000000001,"15min":1.653},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":187.48400000000001,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.3169999999999999,"15min":1.044},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.4009999999999998,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.591,"15min":1.1859999999999999},"min":{"1min":0.34699999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.4670000000000001,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90000000000000002,"15min":1.198},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.7869999999999999,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":0.379},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.89600000000000002,"15min":0.98499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.7549999999999999,"5min":6.7549999999999999,"15min":6.7549999999999999},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.131,"15min":0.93999999999999995},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.899,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.92600000000000005,"15min":0.998},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":4.3170000000000002,"5min":4.7389999999999999,"15min":4.7389999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0029999999999999,"15min":0.96799999999999997},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.5499999999999998,"5min":4.7220000000000004,"15min":4.9050000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.999,"15min":1.018},"min":{"1min":0.40500000000000003,"5min":0.314,"15min":0.314},"max":{"1min":2.964,"5min":7.5640000000000001,"15min":9.3640000000000008},"last":0.55800000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.085,"15min":1.1000000000000001},"min":{"1min":0.34599999999999997,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":3.5819999999999999,"5min":6.319,"15min":6.319},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.1240000000000001,"15min":1.179},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":4.3609999999999998,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.74299999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":6.5750000000000002,"15min":2.9449999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.1619999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.754},{"interface":"front","average":{"1min":1.635,"5min":6.6379999999999999,"15min":3.0310000000000001},"min":{"1min":0.50800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":9.9060000000000006,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.61399999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.97899999999999998,"15min":1.1579999999999999},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.8029999999999999,"5min":4.6539999999999999,"15min":12.907999999999999},"last":4.7549999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.171,"15min":1.2669999999999999},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.7800000000000002,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":4.8360000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":7.7400000000000002,"15min":3.508},"min":{"1min":0.46500000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.5309999999999997,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":7.6219999999999999,"15min":3.375},"min":{"1min":0.48399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.946,"5min":496.29599999999999,"15min":496.29599999999999},"last":4.5129999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.1060000000000001,"15min":1.601},"min":{"1min":0.495,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":4.9039999999999999,"5min":6.0490000000000004,"15min":120.943},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.252,"15min":1.714},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":120.661},"last":4.181}]}]},{"osd":6,"up_from":48,"seq":206158430615,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372516,"kb_used_data":75292,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93372636,"statfs":{"total":95995035648,"available":95613579264,"internally_reserved":0,"allocated":77099008,"data_stored":76665038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":7.04,"5min":2.367,"15min":1.464},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":107.723,"5min":107.723,"15min":107.723},"last":5.7889999999999997},{"interface":"front","average":{"1min":7.1260000000000003,"5min":2.5139999999999998,"15min":1.552},"min":{"1min":0.38,"5min":0.252,"15min":0.252},"max":{"1min":107.114,"5min":107.114,"15min":107.114},"last":0.27700000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0449999999999999,"15min":1.923},"min":{"1min":0.36499999999999999,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":3.5870000000000002,"5min":5.5179999999999998,"15min":252.22800000000001},"last":1.7},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.05,"15min":1.946},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.0549999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5009999999999999,"15min":1.0649999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.1369999999999996,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.393,"5min":1.7090000000000001,"15min":1.1719999999999999},"min":{"1min":0.33700000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3769999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":2.2919999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.2529999999999999,"15min":1.024},"min":{"1min":0.29099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.4509999999999996,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.079,"15min":1.0620000000000001},"min":{"1min":0.34999999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.242,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.34399999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.117,"15min":0.99099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":4.4779999999999998,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.99299999999999999,"15min":1.0109999999999999},"min":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1080000000000001,"5min":10.52,"15min":10.52},"last":1.23}]},{"osd":7,"last update":"Mon Feb 12 16:27:38 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0720000000000001,"15min":1.105},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.0970000000000004,"5min":7.641,"15min":7.641},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.0700000000000001,"15min":1.071},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.7759999999999998,"5min":7.8380000000000001,"15min":7.8380000000000001},"last":0.39900000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1200000000000001,"15min":1.1259999999999999},"min":{"1min":0.48899999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.5119999999999996,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":0.436},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":1.175},"min":{"1min":0.55100000000000005,"5min":0.371,"15min":0.371},"max":{"1min":2.7679999999999998,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":0.71399999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.1839999999999999,"15min":1.254},"min":{"1min":0.45600000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.1680000000000001,"5min":4.5330000000000004,"15min":18.059999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.0229999999999999,"15min":1.246},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2430000000000001,"5min":4.6890000000000001,"15min":17.425000000000001},"last":0.501}]},{"osd":10,"last update":"Mon Feb 12 16:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.222,"15min":1.27},"min":{"1min":0.45500000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":5.726,"5min":5.726,"15min":6.5300000000000002},"last":0.753},{"interface":"front","average":{"1min":1.542,"5min":1.294,"15min":1.3540000000000001},"min":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.7590000000000003,"5min":5.7590000000000003,"15min":6.5730000000000004},"last":0.80900000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:27:46 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.228,"15min":2.8719999999999999},"min":{"1min":0.52100000000000002,"5min":0.499,"15min":0.499},"max":{"1min":5.6859999999999999,"5min":5.6859999999999999,"15min":139.727},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.028,"15min":2.7349999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7310000000000001,"5min":4.5899999999999999,"15min":139.24199999999999},"last":1.351}]}]},{"osd":1,"up_from":13,"seq":55834575293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978672,"kb_used_data":88552,"kb_used_omap":0,"kb_used_meta":890112,"kb_avail":92766480,"statfs":{"total":95995035648,"available":94992875520,"internally_reserved":0,"allocated":90677248,"data_stored":90237326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911474688},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0299999999999998,"5min":1.1599999999999999,"15min":1.405},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":16.632999999999999,"5min":16.632999999999999,"15min":114.40000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":2.29,"5min":1.5920000000000001,"15min":1.6599999999999999},"min":{"1min":0.32300000000000001,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":6.5759999999999996,"5min":7.7249999999999996,"15min":115.09099999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.345,"15min":1.1240000000000001},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":12.778,"5min":12.778,"15min":12.778},"last":0.67800000000000005},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.5569999999999999,"15min":1.216},"min":{"1min":0.28299999999999997,"5min":0.246,"15min":0.246},"max":{"1min":8.9329999999999998,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.377}]},{"osd":3,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.0920000000000001,"15min":1.1200000000000001},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.625,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.1910000000000001,"15min":0.95499999999999996},"min":{"1min":0.38300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.1639999999999997,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.89500000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.274,"15min":1.1220000000000001},"min":{"1min":0.42499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":3.536,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.417,"15min":1.389},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.6859999999999999,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.436}]},{"osd":5,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.482,"15min":1.27},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.7789999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.504},{"interface":"front","average":{"1min":1.268,"5min":1.347,"15min":1.2190000000000001},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":4.1340000000000003,"5min":10.067,"15min":10.067},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":3.2530000000000001,"5min":1.661,"15min":1.7609999999999999},"min":{"1min":0.51700000000000002,"5min":0.38500000000000001,"15min":0.34699999999999998},"max":{"1min":30.437000000000001,"5min":30.437000000000001,"15min":55.869999999999997},"last":0.95899999999999996},{"interface":"front","average":{"1min":3.2029999999999998,"5min":1.611,"15min":1.833},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36799999999999999},"max":{"1min":29.382000000000001,"5min":29.382000000000001,"15min":55.892000000000003},"last":0.70199999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":7.4260000000000002,"15min":3.399},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.436},"max":{"1min":10.138999999999999,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1040000000000001,"5min":7.5540000000000003,"15min":3.4279999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.7160000000000002,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.46999999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:27:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.4450000000000001,"15min":1.3959999999999999},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":5.7590000000000003,"5min":5.7590000000000003,"15min":10.07},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.323,"15min":1.46},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":10.106,"5min":10.106,"15min":10.106},"last":0.86199999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":8.4920000000000009,"15min":3.7589999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.7949999999999999,"5min":548.93499999999995,"15min":548.93499999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.7609999999999999,"5min":8.3650000000000002,"15min":3.806},"min":{"1min":0.57299999999999995,"5min":0.55100000000000005,"15min":0.53700000000000003},"max":{"1min":10.170999999999999,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.746}]},{"osd":11,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.536,"15min":1.5880000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":4.2619999999999996,"5min":10.202,"15min":10.202},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.212,"5min":1.552,"15min":1.6359999999999999},"min":{"1min":0.52200000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.1710000000000003,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":0.71599999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738824,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731940,"kb_used_data":129948,"kb_used_omap":0,"kb_used_meta":601984,"kb_avail":93013212,"statfs":{"total":95995035648,"available":95245529088,"internally_reserved":0,"allocated":133066752,"data_stored":132632782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":616431616},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.0509999999999999,"15min":1.054},"min":{"1min":0.26500000000000001,"5min":0.245,"15min":0.215},"max":{"1min":5.5869999999999997,"5min":10.754,"15min":11.324},"last":0.53100000000000003},{"interface":"front","average":{"1min":2.3679999999999999,"5min":1.3140000000000001,"15min":1.1519999999999999},"min":{"1min":0.24399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.5810000000000004,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.0649999999999999,"15min":1.0580000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.0279999999999996,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":2.0259999999999998,"5min":1.0940000000000001,"15min":0.97999999999999998},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.5510000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":13.318,"5min":3.6579999999999999,"15min":2.1150000000000002},"min":{"1min":0.42799999999999999,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":193.77000000000001,"5min":193.77000000000001,"15min":193.77000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":11.885999999999999,"5min":3.5009999999999999,"15min":2.1440000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":189.625,"5min":189.625,"15min":189.625},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9350000000000001,"5min":1.3320000000000001,"15min":1.0780000000000001},"min":{"1min":0.29799999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.49,"5min":10.49,"15min":10.49},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.772,"5min":1.3200000000000001,"15min":1.0940000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":7.3250000000000002,"5min":7.3250000000000002,"15min":7.3250000000000002},"last":0.34200000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":10.422000000000001,"5min":3.161,"15min":1.9890000000000001},"min":{"1min":0.371,"5min":0.311,"15min":0.311},"max":{"1min":189.79400000000001,"5min":189.79400000000001,"15min":189.79400000000001},"last":0.374},{"interface":"front","average":{"1min":10.676,"5min":3.238,"15min":2.0680000000000001},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":194.41200000000001,"5min":194.41200000000001,"15min":194.41200000000001},"last":0.40500000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":12.795999999999999,"5min":3.5070000000000001,"15min":1.9350000000000001},"min":{"1min":0.39100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":191.47200000000001,"5min":191.47200000000001,"15min":191.47200000000001},"last":0.441},{"interface":"front","average":{"1min":12.872,"5min":3.5950000000000002,"15min":2.0369999999999999},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.44},"max":{"1min":191.50299999999999,"5min":191.50299999999999,"15min":191.50299999999999},"last":0.47399999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.3480000000000001,"15min":1.3160000000000001},"min":{"1min":0.45900000000000002,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":7.3920000000000003,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.749},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.4399999999999999,"15min":1.4059999999999999},"min":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":7.5060000000000002,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.78100000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.2010000000000001,"15min":1.361},"min":{"1min":0.46899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.6550000000000002,"5min":7.6539999999999999,"15min":46.246000000000002},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.345,"15min":1.4219999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.5330000000000004,"5min":5.5330000000000004,"15min":46.167999999999999},"last":0.71599999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:27:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1099999999999999,"5min":1.4159999999999999,"15min":1.3320000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.41399999999999998},"max":{"1min":5.1390000000000002,"5min":5.1390000000000002,"15min":6.0979999999999999},"last":2.1440000000000001},{"interface":"front","average":{"1min":2.1680000000000001,"5min":1.4890000000000001,"15min":1.3580000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":5.6269999999999998},"last":2.1829999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.556,"15min":1.617},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":46.064},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.895,"5min":1.458,"15min":1.5629999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":7.4210000000000003,"5min":7.4210000000000003,"15min":45.642000000000003},"last":0.85099999999999998}]}]},{"osd":2,"up_from":21,"seq":90194313654,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364080,"kb_used_data":62760,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93381072,"statfs":{"total":95995035648,"available":95622217728,"internally_reserved":0,"allocated":64266240,"data_stored":63826318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":2.371,"5min":1.3200000000000001,"15min":1.1180000000000001},"min":{"1min":0.312,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.446,"15min":1.208},"min":{"1min":0.32600000000000001,"5min":0.247,"15min":0.247},"max":{"1min":6.4180000000000001,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":2.7509999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.52,"15min":1.9239999999999999},"min":{"1min":0.29599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":6.2910000000000004,"5min":21.407,"15min":21.407},"last":2.7709999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1020000000000001,"15min":0.92000000000000004},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.8899999999999997,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.753}]},{"osd":4,"last update":"Mon Feb 12 16:27:47 2024","interfaces":[{"interface":"back","average":{"1min":2.625,"5min":1.4359999999999999,"15min":1.0800000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":10.890000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.526,"15min":1.194},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.5279999999999996,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":1.3420000000000001,"15min":1.147},"min":{"1min":0.44800000000000001,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":6.133,"5min":6.133,"15min":6.133},"last":0.72699999999999998},{"interface":"front","average":{"1min":2.8140000000000001,"5min":1.5289999999999999,"15min":1.1850000000000001},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.315},"max":{"1min":11.081,"5min":11.081,"15min":11.081},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.653,"15min":1.278},"min":{"1min":0.45500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.4980000000000002,"5min":21.527000000000001,"15min":21.527000000000001},"last":4.4160000000000004},{"interface":"front","average":{"1min":1.605,"5min":1.3759999999999999,"15min":1.2050000000000001},"min":{"1min":0.41999999999999998,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":6.1710000000000003,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:27:47 2024","interfaces":[{"interface":"back","average":{"1min":2.9609999999999999,"5min":1.522,"15min":1.2030000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":11.044,"5min":11.044,"15min":11.044},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.373,"15min":1.2490000000000001},"min":{"1min":0.51200000000000001,"5min":0.443,"15min":0.41999999999999998},"max":{"1min":6.2050000000000001,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":0.67700000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3919999999999999,"15min":1.2909999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42099999999999999},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":2.7029999999999998},{"interface":"front","average":{"1min":1.504,"5min":1.462,"15min":1.327},"min":{"1min":0.46000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":6.593,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":2.6469999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:27:47 2024","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":1.5489999999999999,"15min":1.323},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":10.972,"5min":10.972,"15min":10.972},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.9089999999999998,"5min":1.627,"15min":1.395},"min":{"1min":0.442,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":11.006,"5min":11.006,"15min":11.006},"last":2.0830000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:27:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.5629999999999999,"15min":1.385},"min":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":6.6319999999999997,"5min":11.454000000000001,"15min":11.454000000000001},"last":2.6779999999999999},{"interface":"front","average":{"1min":2.032,"5min":1.5800000000000001,"15min":1.411},"min":{"1min":0.501,"5min":0.441,"15min":0.441},"max":{"1min":6.8010000000000002,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":2.73}]},{"osd":11,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.389,"15min":1.502},"min":{"1min":0.60999999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":6.2599999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.718,"5min":1.5069999999999999,"15min":1.5760000000000001},"min":{"1min":0.59499999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":0.61099999999999999}]}]},{"osd":3,"up_from":522,"seq":2241972928579,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":377124,"kb_used_data":28508,"kb_used_omap":91,"kb_used_meta":348516,"kb_avail":93368028,"statfs":{"total":95995035648,"available":95608860672,"internally_reserved":0,"allocated":29192192,"data_stored":28758222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":356880400},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:27:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.3560000000000001,"15min":1.016},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.6630000000000003,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.624},{"interface":"front","average":{"1min":1.359,"5min":1.462,"15min":1.056},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.7850000000000001,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.2270000000000001,"15min":0.98799999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5410000000000004,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.196,"15min":1.1020000000000001},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.1040000000000001,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":0.82099999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.89800000000000002,"15min":1.25},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":4.0899999999999999,"5min":10.004,"15min":10.004},"last":0.31},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.032,"15min":1.0980000000000001},"min":{"1min":0.35999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.806,"5min":6.806,"15min":6.806},"last":0.88}]},{"osd":5,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.0820000000000001,"15min":1.325},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.316},"max":{"1min":4.7530000000000001,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.318,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.1340000000000003,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.34599999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.0780000000000001,"15min":1.1319999999999999},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.7450000000000001,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.002,"15min":1.012},"min":{"1min":0.38300000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":4.1689999999999996,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.38300000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.1140000000000001,"15min":1.4099999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.21,"5min":6.851,"15min":6.851},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.218,"15min":1.2769999999999999},"min":{"1min":0.41799999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.8479999999999999,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":0.85499999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.0449999999999999,"15min":1.173},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.8109999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.99399999999999999,"15min":1.0980000000000001},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.2430000000000003,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.44500000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.2909999999999999,"15min":1.331},"min":{"1min":0.45500000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.1539999999999999,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.335,"15min":1.3799999999999999},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.863,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.70699999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.2130000000000001,"15min":1.4750000000000001},"min":{"1min":0.54600000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.8700000000000001,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.3089999999999999,"15min":1.3260000000000001},"min":{"1min":0.501,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.8180000000000001,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":0.93400000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.242,"15min":1.284},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.9870000000000001,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.2629999999999999,"15min":1.4870000000000001},"min":{"1min":0.63300000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.8399999999999999,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.77400000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953894,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425188,"kb_used_data":123868,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93319964,"statfs":{"total":95995035648,"available":95559643136,"internally_reserved":0,"allocated":126840832,"data_stored":126406862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:27:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.093,"15min":1.363},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.4710000000000001,"5min":5.4710000000000001,"15min":90.597999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.143,"5min":1.157,"15min":1.5409999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.5389999999999997,"5min":12.236000000000001,"15min":90.676000000000002},"last":1.742}]},{"osd":2,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.022,"15min":0.95999999999999996},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.577,"5min":5.577,"15min":8.0530000000000008},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.1619999999999999,"15min":1.1279999999999999},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":6.718,"5min":6.718,"15min":7.452},"last":0.502}]},{"osd":3,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":1.1259999999999999,"15min":1.1020000000000001},"min":{"1min":0.33300000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.2930000000000001,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.018,"15min":1.046},"min":{"1min":0.35499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.657,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.96299999999999997,"15min":0.93100000000000005},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.0529999999999999,"5min":4.7009999999999996,"15min":5.5999999999999996},"last":0.372},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.41299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.081,"5min":4.7160000000000002,"15min":5.7450000000000001},"last":0.45900000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.83299999999999996,"15min":0.97799999999999998},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.8660000000000001,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.018,"5min":1.218,"15min":1.071},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.5190000000000001,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1379999999999999,"15min":1.101},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":5.4870000000000001,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.88100000000000001,"15min":1.143},"min":{"1min":0.36099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6459999999999999,"5min":3.6459999999999999,"15min":7.6689999999999996},"last":0.81000000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.002,"15min":1.179},"min":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.6819999999999999,"5min":4.0439999999999996,"15min":7.8650000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.143,"15min":1.2230000000000001},"min":{"1min":0.42399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3200000000000001,"5min":5.4489999999999998,"15min":5.6699999999999999},"last":0.66900000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.998,"15min":1.399},"min":{"1min":0.53300000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.9750000000000001,"5min":3.2650000000000001,"15min":34.112000000000002},"last":4.0430000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.3440000000000001,"15min":1.637},"min":{"1min":0.45600000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.754,"5min":5.6420000000000003,"15min":34.590000000000003},"last":0.60599999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.373,"15min":1.381},"min":{"1min":0.503,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.48,"5min":5.6970000000000001,"15min":6.0899999999999999},"last":3.0310000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.177,"15min":1.2350000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.4269999999999996,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":0.66100000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0800000000000001,"15min":1.4470000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6500000000000004},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.357,"15min":1.482},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":3.52,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":0.72899999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477416,"kb_used_data":167904,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93267736,"statfs":{"total":95995035648,"available":95506161664,"internally_reserved":0,"allocated":171933696,"data_stored":171495654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.83799999999999997,"15min":2.7349999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":565.34500000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.113,"5min":0.93799999999999994,"15min":2.8140000000000001},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":4.5830000000000002,"5min":4.5830000000000002,"15min":565.98400000000004},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3839999999999999,"15min":1.143},"min":{"1min":0.314,"5min":0.314,"15min":0.30499999999999999},"max":{"1min":6.0709999999999997,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.478,"15min":1.1859999999999999},"min":{"1min":0.28599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.5069999999999997,"5min":40.225000000000001,"15min":40.225000000000001},"last":0.49299999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":1.4570000000000001,"15min":1.351},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":16.300999999999998,"5min":16.300999999999998,"15min":16.300999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.498,"5min":0.95999999999999996,"15min":0.80600000000000005},"min":{"1min":0.33700000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.5540000000000003,"5min":6.5540000000000003,"15min":6.5540000000000003},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.331,"15min":1.4239999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.661,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.5329999999999999,"15min":1.46},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":6.4610000000000003,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1739999999999999,"15min":1.3320000000000001},"min":{"1min":0.34200000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.3490000000000002,"5min":6.4459999999999997,"15min":74.105000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.2330000000000001,"15min":1.4390000000000001},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":9.5830000000000002,"5min":9.5830000000000002,"15min":75.013999999999996},"last":0.61599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.113,"15min":1.0669999999999999},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":9.7710000000000008,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.1779999999999999,"15min":1.171},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.3979999999999997,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.435}]},{"osd":8,"last update":"Mon Feb 12 16:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.95799999999999996,"15min":1.181},"min":{"1min":0.432,"5min":0.41699999999999998,"15min":0.40200000000000002},"max":{"1min":2.524,"5min":4.0069999999999997,"15min":5.9509999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.202,"5min":1.119,"15min":1.2430000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.6159999999999997,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":0.58899999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4550000000000001,"15min":3.327},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":4.54,"5min":6.2939999999999996,"15min":449.89499999999998},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.829,"5min":1.321,"15min":3.3010000000000002},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":7.5789999999999997,"5min":9.6649999999999991,"15min":448.70800000000003},"last":0.81000000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.016,"15min":1.258},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.6120000000000001,"5min":4.0410000000000004,"15min":5.9210000000000003},"last":0.63700000000000001},{"interface":"front","average":{"1min":2.016,"5min":1.431,"15min":1.3540000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":15.006,"5min":15.006,"15min":15.006},"last":0.64600000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:27:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.131,"15min":1.21},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.7789999999999999,"5min":6.0990000000000002,"15min":6.0990000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.712,"5min":1.4419999999999999,"15min":1.355},"min":{"1min":0.57699999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119668736,"data_stored":119668736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74579968,"data_stored":74579968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":56295424,"data_stored":56295424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26607616,"data_stored":26607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":116195328,"data_stored":116195328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155979776,"data_stored":155975704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":72417280,"data_stored":72417280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":30670848,"data_stored":30670848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":60489728,"data_stored":60489728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":40173568,"data_stored":40173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":75563008,"data_stored":75563008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":40439808,"data_stored":40435736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:28:34.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T16:28:34.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:34.858 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:34.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:34.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:34.987 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:34.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:35.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:35.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:35 smithi169 bash[19604]: cluster 2024-02-12T16:28:33.999629+0000 mgr.y (mgr.14146) 1311 : cluster [DBG] pgmap v1821: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 313 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 279/10032 objects misplaced (2.781%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:28:35.101 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:35 smithi169 bash[19604]: audit 2024-02-12T16:28:35.069284+0000 mon.a (mon.0) 2212 : audit [DBG] from='client.? v1:172.21.15.121:0/1786838146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:35.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:35.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:35.139 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:28:35.187 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:28:35.224 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:28:35.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:35.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:35.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:35.267 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:28:35.310 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:28:35.355 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:28:35.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:35.367 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:35 smithi121 bash[18489]: cluster 2024-02-12T16:28:33.999629+0000 mgr.y (mgr.14146) 1311 : cluster [DBG] pgmap v1821: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 313 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 279/10032 objects misplaced (2.781%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:28:35.367 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:35 smithi121 bash[18489]: audit 2024-02-12T16:28:35.069284+0000 mon.a (mon.0) 2212 : audit [DBG] from='client.? v1:172.21.15.121:0/1786838146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:35 smithi123 bash[19119]: cluster 2024-02-12T16:28:33.999629+0000 mgr.y (mgr.14146) 1311 : cluster [DBG] pgmap v1821: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 313 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 279/10032 objects misplaced (2.781%); 1.3 MiB/s, 20 objects/s recovering 2024-02-12T16:28:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:35 smithi123 bash[19119]: audit 2024-02-12T16:28:35.069284+0000 mon.a (mon.0) 2212 : audit [DBG] from='client.? v1:172.21.15.121:0/1786838146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:35.393 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:28:35.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:35.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:35.432 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:28:35.477 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:28:35.513 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:28:35.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:35.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:35.553 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:28:35.594 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:28:35.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:35.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:35.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:35.699 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:28:35.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:35.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:35.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:35.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:35.958 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:35.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:36.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:36.088 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:36.089 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:36 smithi169 bash[19604]: audit 2024-02-12T16:28:35.586731+0000 mon.a (mon.0) 2213 : audit [INF] from='client.26182 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:36.089 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:36 smithi169 bash[19604]: audit 2024-02-12T16:28:35.587984+0000 mon.c (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.121:0/229236661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:36.089 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:36 smithi169 bash[19604]: cluster 2024-02-12T16:28:35.672559+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:36.089 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:36 smithi169 bash[19604]: audit 2024-02-12T16:28:35.677470+0000 mon.a (mon.0) 2215 : audit [INF] from='client.26182 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:28:36.089 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:36 smithi169 bash[19604]: cluster 2024-02-12T16:28:35.677594+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e624: 12 total, 12 up, 12 in 2024-02-12T16:28:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:36.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:36.191 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:36.227 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:36.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:36.255 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 169387 objects 2024-02-12T16:28:36.255 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :160.446783 2024-02-12T16:28:36.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:36.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:36 smithi121 bash[18489]: audit 2024-02-12T16:28:35.586731+0000 mon.a (mon.0) 2213 : audit [INF] from='client.26182 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:36.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:36 smithi121 bash[18489]: audit 2024-02-12T16:28:35.587984+0000 mon.c (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.121:0/229236661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:36.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:36 smithi121 bash[18489]: cluster 2024-02-12T16:28:35.672559+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:36.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:36 smithi121 bash[18489]: audit 2024-02-12T16:28:35.677470+0000 mon.a (mon.0) 2215 : audit [INF] from='client.26182 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:28:36.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:36 smithi121 bash[18489]: cluster 2024-02-12T16:28:35.677594+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e624: 12 total, 12 up, 12 in 2024-02-12T16:28:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:36.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:36.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:36 smithi123 bash[19119]: audit 2024-02-12T16:28:35.586731+0000 mon.a (mon.0) 2213 : audit [INF] from='client.26182 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:36.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:36 smithi123 bash[19119]: audit 2024-02-12T16:28:35.587984+0000 mon.c (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.121:0/229236661' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:28:36.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:36 smithi123 bash[19119]: cluster 2024-02-12T16:28:35.672559+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:36.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:36 smithi123 bash[19119]: audit 2024-02-12T16:28:35.677470+0000 mon.a (mon.0) 2215 : audit [INF] from='client.26182 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:28:36.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:36 smithi123 bash[19119]: cluster 2024-02-12T16:28:35.677594+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e624: 12 total, 12 up, 12 in 2024-02-12T16:28:36.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:36.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:36.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:36.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:36.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:36.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:36.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:36.717 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:28:36.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:36.779 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:36.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:36.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:36.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:36.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:37.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:37.062 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:37.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:37 smithi121 bash[18489]: cluster 2024-02-12T16:28:36.000041+0000 mgr.y (mgr.14146) 1312 : cluster [DBG] pgmap v1823: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 196 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s; 110/6258 objects misplaced (1.758%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:28:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:37 smithi121 bash[18489]: cluster 2024-02-12T16:28:36.695054+0000 mon.a (mon.0) 2217 : cluster [DBG] osdmap e625: 12 total, 12 up, 12 in 2024-02-12T16:28:37.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:37.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:37.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:37.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:37.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:37.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:37 smithi123 bash[19119]: cluster 2024-02-12T16:28:36.000041+0000 mgr.y (mgr.14146) 1312 : cluster [DBG] pgmap v1823: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 196 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s; 110/6258 objects misplaced (1.758%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:28:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:37 smithi123 bash[19119]: cluster 2024-02-12T16:28:36.695054+0000 mon.a (mon.0) 2217 : cluster [DBG] osdmap e625: 12 total, 12 up, 12 in 2024-02-12T16:28:37.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:37.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:37.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:37 smithi169 bash[19604]: cluster 2024-02-12T16:28:36.000041+0000 mgr.y (mgr.14146) 1312 : cluster [DBG] pgmap v1823: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 196 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 971 op/s; 110/6258 objects misplaced (1.758%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:28:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:37 smithi169 bash[19604]: cluster 2024-02-12T16:28:36.695054+0000 mon.a (mon.0) 2217 : cluster [DBG] osdmap e625: 12 total, 12 up, 12 in 2024-02-12T16:28:37.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:37.648 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:37.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:37.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:37.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:37.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:37.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:37.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:37.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:38.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:38.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:38.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:38.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:38.204 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:38.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:38.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:38.345 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:38.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:38.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:38.487 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:38.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:38.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:38.625 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:38.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:38.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:38.768 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:38.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:38.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:38.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:38 smithi123 bash[19119]: cluster 2024-02-12T16:28:37.697699+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in 2024-02-12T16:28:38.885 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:38 smithi123 bash[19119]: cluster 2024-02-12T16:28:38.000714+0000 mgr.y (mgr.14146) 1313 : cluster [DBG] pgmap v1826: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 66 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 110/2124 objects misplaced (5.179%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:28:38.909 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:38 smithi121 bash[18489]: cluster 2024-02-12T16:28:37.697699+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in 2024-02-12T16:28:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:38 smithi121 bash[18489]: cluster 2024-02-12T16:28:38.000714+0000 mgr.y (mgr.14146) 1313 : cluster [DBG] pgmap v1826: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 66 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 110/2124 objects misplaced (5.179%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:28:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:38.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:39.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:39.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:38 smithi169 bash[19604]: cluster 2024-02-12T16:28:37.697699+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e626: 12 total, 12 up, 12 in 2024-02-12T16:28:39.024 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:38 smithi169 bash[19604]: cluster 2024-02-12T16:28:38.000714+0000 mgr.y (mgr.14146) 1313 : cluster [DBG] pgmap v1826: 37 pgs: 2 active+remapped+backfilling, 35 active+clean; 66 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 110/2124 objects misplaced (5.179%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:28:39.040 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:39.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:39.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:39.180 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:39.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:39.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:28:39.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:39.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:28:39.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:39.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:39.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:39.494 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:39.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:39.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:39.634 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:39.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:39.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:39.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:39 smithi169 bash[19604]: cluster 2024-02-12T16:28:39.528568+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:39.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:39 smithi169 bash[19604]: cluster 2024-02-12T16:28:39.528657+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:39.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:39.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:39.872 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:39.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:39 smithi123 bash[19119]: cluster 2024-02-12T16:28:39.528568+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:39.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:39 smithi123 bash[19119]: cluster 2024-02-12T16:28:39.528657+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:39.909 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:39.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:40.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:40.052 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:40.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:40.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:39 smithi121 bash[18489]: cluster 2024-02-12T16:28:39.528568+0000 osd.7 (osd.7) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:39 smithi121 bash[18489]: cluster 2024-02-12T16:28:39.528657+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:40.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:40.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:40.330 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:40.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:40.360 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:28:40.360 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:28:40.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:40.470 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:40.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:40.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:40.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:40.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:40.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:40.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:40.823 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:40.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:40.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:40.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:40.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:40 smithi123 bash[19119]: cluster 2024-02-12T16:28:39.709061+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e627: 12 total, 12 up, 12 in 2024-02-12T16:28:40.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:40 smithi123 bash[19119]: cluster 2024-02-12T16:28:40.001437+0000 mgr.y (mgr.14146) 1314 : cluster [DBG] pgmap v1828: 37 pgs: 37 active+clean; 11 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 710 KiB/s, 11 objects/s recovering 2024-02-12T16:28:40.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:40 smithi123 bash[19119]: cluster 2024-02-12T16:28:40.681558+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e628: 12 total, 12 up, 12 in 2024-02-12T16:28:40.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:40.999 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-02-12T16:28:40.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-02-12T16:28:41.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:40 smithi169 bash[19604]: cluster 2024-02-12T16:28:39.709061+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e627: 12 total, 12 up, 12 in 2024-02-12T16:28:41.001 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:40 smithi169 bash[19604]: cluster 2024-02-12T16:28:40.001437+0000 mgr.y (mgr.14146) 1314 : cluster [DBG] pgmap v1828: 37 pgs: 37 active+clean; 11 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 710 KiB/s, 11 objects/s recovering 2024-02-12T16:28:41.001 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:40 smithi169 bash[19604]: cluster 2024-02-12T16:28:40.681558+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e628: 12 total, 12 up, 12 in 2024-02-12T16:28:41.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:40 smithi121 bash[18489]: cluster 2024-02-12T16:28:39.709061+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e627: 12 total, 12 up, 12 in 2024-02-12T16:28:41.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:40 smithi121 bash[18489]: cluster 2024-02-12T16:28:40.001437+0000 mgr.y (mgr.14146) 1314 : cluster [DBG] pgmap v1828: 37 pgs: 37 active+clean; 11 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 710 KiB/s, 11 objects/s recovering 2024-02-12T16:28:41.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:40 smithi121 bash[18489]: cluster 2024-02-12T16:28:40.681558+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e628: 12 total, 12 up, 12 in 2024-02-12T16:28:41.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-12T16:28:41.024 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:41.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:41.128 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:41.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:41.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:41.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:41.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:41.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:41.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:41.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:41.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:41.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:41.544 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:41.581 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:41.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:41.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:41.724 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:41.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:41.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:41.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:41.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:41.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:42.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:42.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:42.139 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:42.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:42.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:42.416 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:42.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:42.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:42.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:42.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:42.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:42.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:42.841 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:42.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:42.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:43.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:43.124 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:43.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:43.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:43.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:43.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:43.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:43.378 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:43 smithi121 bash[18489]: cluster 2024-02-12T16:28:42.002205+0000 mgr.y (mgr.14146) 1315 : cluster [DBG] pgmap v1830: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 361 op/s; 578 KiB/s, 9 objects/s recovering 2024-02-12T16:28:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:43 smithi123 bash[19119]: cluster 2024-02-12T16:28:42.002205+0000 mgr.y (mgr.14146) 1315 : cluster [DBG] pgmap v1830: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 361 op/s; 578 KiB/s, 9 objects/s recovering 2024-02-12T16:28:43.402 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:43.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:43.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:43.538 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:43.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:43 smithi169 bash[19604]: cluster 2024-02-12T16:28:42.002205+0000 mgr.y (mgr.14146) 1315 : cluster [DBG] pgmap v1830: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 361 op/s; 578 KiB/s, 9 objects/s recovering 2024-02-12T16:28:43.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:43.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:43.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:43.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:43.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:43.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:43.874 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:43.876 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:43.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:43.957 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:43.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:44.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:44.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:44.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:44.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:44.236 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:44.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:44.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:44.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:44.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:44.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:44.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:44.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:44.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:44.659 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:44.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:44.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:44.801 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:44.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:44.904 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:44.939 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:44.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:44.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:45.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:45.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:45.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:45.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:45.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:45.267 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:28:45.267 INFO:teuthology.orchestra.run.smithi121.stdout:{"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2321,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":628,"num_osds":12,"num_up_osds":12,"osd_up_since":1707754981,"num_in_osds":12,"osd_in_since":1707754767,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":37}],"num_pgs":37,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":4947509248,"bytes_avail":1146992918528,"bytes_total":1151940427776,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":498604,"recovering_keys_per_sec":0,"num_objects_recovered":48,"num_bytes_recovered":3145728,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":168},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.121:8443/"}},"servicemap":{"epoch":55,"modified":"2024-02-12T16:28:40.001125+0000","services":{"osd":{"daemons":{"summary":"","11":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-12T16:28:45.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:45 smithi121 bash[18489]: cluster 2024-02-12T16:28:44.003009+0000 mgr.y (mgr.14146) 1316 : cluster [DBG] pgmap v1831: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 487 KiB/s, 7 objects/s recovering 2024-02-12T16:28:45.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:45.328 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:45 smithi169 bash[19604]: cluster 2024-02-12T16:28:44.003009+0000 mgr.y (mgr.14146) 1316 : cluster [DBG] pgmap v1831: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 487 KiB/s, 7 objects/s recovering 2024-02-12T16:28:45.352 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:45.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:45 smithi123 bash[19119]: cluster 2024-02-12T16:28:44.003009+0000 mgr.y (mgr.14146) 1316 : cluster [DBG] pgmap v1831: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 487 KiB/s, 7 objects/s recovering 2024-02-12T16:28:45.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:45.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:45.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:45.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:45.633 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:45.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:45.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:45.772 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:45.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:45.811 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:28:45.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:45.913 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:45.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:46.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:46.053 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:46.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:46.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:46.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:46 smithi121 bash[18489]: audit 2024-02-12T16:28:45.261880+0000 mon.a (mon.0) 2221 : audit [DBG] from='client.? v1:172.21.15.121:0/2122248088' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:28:46.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:46 smithi121 bash[18489]: cluster 2024-02-12T16:28:45.458318+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:46.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:46 smithi121 bash[18489]: cluster 2024-02-12T16:28:45.458410+0000 osd.1 (osd.1) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:46.172 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-02-12T16:28:46.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:28:46.191 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:46.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:46.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:46.330 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:46.333 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:46 smithi123 bash[19119]: audit 2024-02-12T16:28:45.261880+0000 mon.a (mon.0) 2221 : audit [DBG] from='client.? v1:172.21.15.121:0/2122248088' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:28:46.333 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:46 smithi123 bash[19119]: cluster 2024-02-12T16:28:45.458318+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:46.333 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:46 smithi123 bash[19119]: cluster 2024-02-12T16:28:45.458410+0000 osd.1 (osd.1) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:46.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:46.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:46.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:46.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:46 smithi169 bash[19604]: audit 2024-02-12T16:28:45.261880+0000 mon.a (mon.0) 2221 : audit [DBG] from='client.? v1:172.21.15.121:0/2122248088' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:28:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:46 smithi169 bash[19604]: cluster 2024-02-12T16:28:45.458318+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:46 smithi169 bash[19604]: cluster 2024-02-12T16:28:45.458410+0000 osd.1 (osd.1) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:46.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:46.606 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:46.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:46.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:46.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:46.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:46.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:46.881 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:46.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:46.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:47.016 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:47.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:47.120 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:47.157 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:47.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:47.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:47 smithi169 bash[19604]: cluster 2024-02-12T16:28:46.003753+0000 mgr.y (mgr.14146) 1317 : cluster [DBG] pgmap v1832: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 384 KiB/s, 5 objects/s recovering 2024-02-12T16:28:47.174 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:47 smithi169 bash[19604]: cluster 2024-02-12T16:28:46.068599+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e629: 12 total, 12 up, 12 in 2024-02-12T16:28:47.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:47.296 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:47.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:47 smithi123 bash[19119]: cluster 2024-02-12T16:28:46.003753+0000 mgr.y (mgr.14146) 1317 : cluster [DBG] pgmap v1832: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 384 KiB/s, 5 objects/s recovering 2024-02-12T16:28:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:47 smithi123 bash[19119]: cluster 2024-02-12T16:28:46.068599+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e629: 12 total, 12 up, 12 in 2024-02-12T16:28:47.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:47 smithi121 bash[18489]: cluster 2024-02-12T16:28:46.003753+0000 mgr.y (mgr.14146) 1317 : cluster [DBG] pgmap v1832: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 384 KiB/s, 5 objects/s recovering 2024-02-12T16:28:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:47 smithi121 bash[18489]: cluster 2024-02-12T16:28:46.068599+0000 mon.a (mon.0) 2222 : cluster [DBG] osdmap e629: 12 total, 12 up, 12 in 2024-02-12T16:28:47.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:47.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:47.454 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:47.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:47.572 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:47.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:47.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:47.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:47.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:47.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:47.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:47.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:47.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:47.997 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:48.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:48.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:48.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:48.278 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:48.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:48.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:48.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:48.558 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:48.662 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:48.697 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:48.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:48.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:48.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:48.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:48.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:49.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:49.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:49.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:49.148 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:28:49.148 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:28:49.199 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:49 smithi169 bash[19604]: cluster 2024-02-12T16:28:48.004491+0000 mgr.y (mgr.14146) 1318 : cluster [DBG] pgmap v1834: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s 2024-02-12T16:28:49.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:49.263 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:49.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:49.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:49.374 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:49 smithi121 bash[18489]: cluster 2024-02-12T16:28:48.004491+0000 mgr.y (mgr.14146) 1318 : cluster [DBG] pgmap v1834: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s 2024-02-12T16:28:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:49 smithi123 bash[19119]: cluster 2024-02-12T16:28:48.004491+0000 mgr.y (mgr.14146) 1318 : cluster [DBG] pgmap v1834: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s 2024-02-12T16:28:49.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:49.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:49.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:49.512 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:28:49.537 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:49.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:49.550 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:28:49.583 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:28:49.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:28:49.630 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:28:49.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:49.655 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:28:49.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:49.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:49.681 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:28:49.704 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:28:49.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:28:49.756 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:28:49.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:49.783 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:28:49.805 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:28:49.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:49.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:49.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-12T16:28:49.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:49.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:49.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:50.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:50.090 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:50.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:50.100 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:28:50.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:50 smithi121 bash[18489]: audit 2024-02-12T16:28:49.470420+0000 mon.a (mon.0) 2223 : audit [DBG] from='client.? v1:172.21.15.121:0/885732003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:50.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:50 smithi121 bash[18489]: audit 2024-02-12T16:28:49.682005+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? v1:172.21.15.121:0/4258378071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:28:50.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:50.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:50.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:50.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:50.370 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:50.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:50 smithi123 bash[19119]: audit 2024-02-12T16:28:49.470420+0000 mon.a (mon.0) 2223 : audit [DBG] from='client.? v1:172.21.15.121:0/885732003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:50.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:50 smithi123 bash[19119]: audit 2024-02-12T16:28:49.682005+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? v1:172.21.15.121:0/4258378071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:28:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:50.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:50.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:50.515 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:50.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:50.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:50 smithi169 bash[19604]: audit 2024-02-12T16:28:49.470420+0000 mon.a (mon.0) 2223 : audit [DBG] from='client.? v1:172.21.15.121:0/885732003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:28:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:50 smithi169 bash[19604]: audit 2024-02-12T16:28:49.682005+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? v1:172.21.15.121:0/4258378071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:28:50.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:50.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:50.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:50.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:50.805 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:50.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:50.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:50.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:51.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:51.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:51.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:51.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:51.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:51.368 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:51.370 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:51 smithi123 bash[19119]: cluster 2024-02-12T16:28:50.005211+0000 mgr.y (mgr.14146) 1319 : cluster [DBG] pgmap v1835: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:51.370 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:51 smithi123 bash[19119]: cluster 2024-02-12T16:28:50.082540+0000 mon.a (mon.0) 2225 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:51.370 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:51 smithi123 bash[19119]: audit 2024-02-12T16:28:50.090104+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? v1:172.21.15.121:0/4258378071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:28:51.371 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:51 smithi123 bash[19119]: cluster 2024-02-12T16:28:50.090249+0000 mon.a (mon.0) 2227 : cluster [DBG] osdmap e630: 12 total, 12 up, 12 in 2024-02-12T16:28:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:51.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:51.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:51 smithi121 bash[18489]: cluster 2024-02-12T16:28:50.005211+0000 mgr.y (mgr.14146) 1319 : cluster [DBG] pgmap v1835: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:51 smithi121 bash[18489]: cluster 2024-02-12T16:28:50.082540+0000 mon.a (mon.0) 2225 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:51 smithi121 bash[18489]: audit 2024-02-12T16:28:50.090104+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? v1:172.21.15.121:0/4258378071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:28:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:51 smithi121 bash[18489]: cluster 2024-02-12T16:28:50.090249+0000 mon.a (mon.0) 2227 : cluster [DBG] osdmap e630: 12 total, 12 up, 12 in 2024-02-12T16:28:51.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:51.510 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:51.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:51 smithi169 bash[19604]: cluster 2024-02-12T16:28:50.005211+0000 mgr.y (mgr.14146) 1319 : cluster [DBG] pgmap v1835: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:51 smithi169 bash[19604]: cluster 2024-02-12T16:28:50.082540+0000 mon.a (mon.0) 2225 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:28:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:51 smithi169 bash[19604]: audit 2024-02-12T16:28:50.090104+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? v1:172.21.15.121:0/4258378071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:28:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:51 smithi169 bash[19604]: cluster 2024-02-12T16:28:50.090249+0000 mon.a (mon.0) 2227 : cluster [DBG] osdmap e630: 12 total, 12 up, 12 in 2024-02-12T16:28:51.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:51.656 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:51.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:51.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:51.797 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:51.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:51.901 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:51.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:52.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:52.082 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:52.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:52.110 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:52.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:52.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:52 smithi121 bash[18489]: cluster 2024-02-12T16:28:50.412704+0000 osd.11 (osd.11) 136 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:28:52.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:52 smithi121 bash[18489]: cluster 2024-02-12T16:28:50.493991+0000 osd.11 (osd.11) 137 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:28:52.230 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:52.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:52.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:52.371 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:52.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:52 smithi123 bash[19119]: cluster 2024-02-12T16:28:50.412704+0000 osd.11 (osd.11) 136 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:28:52.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:52 smithi123 bash[19119]: cluster 2024-02-12T16:28:50.493991+0000 osd.11 (osd.11) 137 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:28:52.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:52.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:52 smithi169 bash[19604]: cluster 2024-02-12T16:28:50.412704+0000 osd.11 (osd.11) 136 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:28:52.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:52 smithi169 bash[19604]: cluster 2024-02-12T16:28:50.493991+0000 osd.11 (osd.11) 137 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:28:52.512 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:52.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:52.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:52.654 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:52.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:52.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:52.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:52.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:52.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:52.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:52.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:53.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:53.073 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:53.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:53.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:53.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:53.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:51.278559+0000 osd.5 (osd.5) 97 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:51.296109+0000 osd.5 (osd.5) 98 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:51.433639+0000 osd.11 (osd.11) 138 : cluster [DBG] 5.d scrub starts 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:51.457523+0000 osd.11 (osd.11) 139 : cluster [DBG] 5.d scrub ok 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:51.664552+0000 osd.3 (osd.3) 22 : cluster [DBG] 1.c scrub starts 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:51.703286+0000 osd.3 (osd.3) 23 : cluster [DBG] 1.c scrub ok 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:52.006027+0000 mgr.y (mgr.14146) 1320 : cluster [DBG] pgmap v1837: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:52.052363+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:28:53.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:52.076535+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:28:53.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:52.809045+0000 osd.9 (osd.9) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:53.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:52.809139+0000 osd.9 (osd.9) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:53.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:52.983631+0000 osd.9 (osd.9) 107 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:28:53.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:53 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.018676+0000 osd.9 (osd.9) 108 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:28:53.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:53.352 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:53.356 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:53.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:51.278559+0000 osd.5 (osd.5) 97 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:28:53.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:51.296109+0000 osd.5 (osd.5) 98 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:28:53.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:51.433639+0000 osd.11 (osd.11) 138 : cluster [DBG] 5.d scrub starts 2024-02-12T16:28:53.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:51.457523+0000 osd.11 (osd.11) 139 : cluster [DBG] 5.d scrub ok 2024-02-12T16:28:53.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:51.664552+0000 osd.3 (osd.3) 22 : cluster [DBG] 1.c scrub starts 2024-02-12T16:28:53.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:51.703286+0000 osd.3 (osd.3) 23 : cluster [DBG] 1.c scrub ok 2024-02-12T16:28:53.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:52.006027+0000 mgr.y (mgr.14146) 1320 : cluster [DBG] pgmap v1837: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:53.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:52.052363+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:28:53.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:52.076535+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:28:53.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:52.809045+0000 osd.9 (osd.9) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:53.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:52.809139+0000 osd.9 (osd.9) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:53.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:52.983631+0000 osd.9 (osd.9) 107 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:28:53.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:53 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.018676+0000 osd.9 (osd.9) 108 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:51.278559+0000 osd.5 (osd.5) 97 : cluster [DBG] 5.7 scrub starts 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:51.296109+0000 osd.5 (osd.5) 98 : cluster [DBG] 5.7 scrub ok 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:51.433639+0000 osd.11 (osd.11) 138 : cluster [DBG] 5.d scrub starts 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:51.457523+0000 osd.11 (osd.11) 139 : cluster [DBG] 5.d scrub ok 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:51.664552+0000 osd.3 (osd.3) 22 : cluster [DBG] 1.c scrub starts 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:51.703286+0000 osd.3 (osd.3) 23 : cluster [DBG] 1.c scrub ok 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:52.006027+0000 mgr.y (mgr.14146) 1320 : cluster [DBG] pgmap v1837: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:52.052363+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.5 scrub starts 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:52.076535+0000 osd.0 (osd.0) 100 : cluster [DBG] 5.5 scrub ok 2024-02-12T16:28:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:52.809045+0000 osd.9 (osd.9) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:53.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:52.809139+0000 osd.9 (osd.9) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:53.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:52.983631+0000 osd.9 (osd.9) 107 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:28:53.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:53 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.018676+0000 osd.9 (osd.9) 108 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:28:53.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:53.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:28:53.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:53.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:53.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:53.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:53.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:53.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:53.777 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:53.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:53.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:53.918 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:53.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:53.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:28:54.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:54.053 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:54.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:54.137 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_3' removed 2024-02-12T16:28:54.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:54.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:54 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.118889+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e631: 12 total, 12 up, 12 in 2024-02-12T16:28:54.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:54 smithi121 bash[18489]: audit 2024-02-12T16:28:53.509340+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? v1:172.21.15.121:0/1997101686' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:28:54.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:54.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:54.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:54 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.118889+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e631: 12 total, 12 up, 12 in 2024-02-12T16:28:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:54 smithi169 bash[19604]: audit 2024-02-12T16:28:53.509340+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? v1:172.21.15.121:0/1997101686' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:28:54.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:54.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:54 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.118889+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e631: 12 total, 12 up, 12 in 2024-02-12T16:28:54.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:54 smithi123 bash[19119]: audit 2024-02-12T16:28:53.509340+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? v1:172.21.15.121:0/1997101686' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:28:54.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:54.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:54.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:54.549 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:54.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:54.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:54.661 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:28:54.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:54.719 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:54.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:54.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:54.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:54.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:54.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:55.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:28:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:28:55.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.316867+0000 osd.5 (osd.5) 99 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:28:55.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.361605+0000 osd.5 (osd.5) 100 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.488166+0000 osd.11 (osd.11) 140 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.509062+0000 osd.11 (osd.11) 141 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.605485+0000 osd.7 (osd.7) 150 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.646012+0000 osd.7 (osd.7) 151 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.669550+0000 osd.3 (osd.3) 24 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.712417+0000 osd.3 (osd.3) 25 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.885814+0000 osd.6 (osd.6) 114 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:53.942652+0000 osd.6 (osd.6) 115 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.006760+0000 mgr.y (mgr.14146) 1321 : cluster [DBG] pgmap v1839: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: audit 2024-02-12T16:28:54.124957+0000 mon.a (mon.0) 2230 : audit [INF] from='client.? v1:172.21.15.121:0/1997101686' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.125029+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.253089+0000 osd.10 (osd.10) 118 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.310443+0000 osd.10 (osd.10) 119 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.922491+0000 osd.9 (osd.9) 109 : cluster [DBG] 1.a scrub starts 2024-02-12T16:28:55.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:55 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.979206+0000 osd.9 (osd.9) 110 : cluster [DBG] 1.a scrub ok 2024-02-12T16:28:55.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:55.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:55.262 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:28:55.263 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:28:55.263 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-02-12T16:28:55.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-02-12T16:28:55.279 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:55.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.316867+0000 osd.5 (osd.5) 99 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.361605+0000 osd.5 (osd.5) 100 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.488166+0000 osd.11 (osd.11) 140 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.509062+0000 osd.11 (osd.11) 141 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.605485+0000 osd.7 (osd.7) 150 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.646012+0000 osd.7 (osd.7) 151 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.669550+0000 osd.3 (osd.3) 24 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:28:55.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.712417+0000 osd.3 (osd.3) 25 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.885814+0000 osd.6 (osd.6) 114 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:53.942652+0000 osd.6 (osd.6) 115 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.006760+0000 mgr.y (mgr.14146) 1321 : cluster [DBG] pgmap v1839: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: audit 2024-02-12T16:28:54.124957+0000 mon.a (mon.0) 2230 : audit [INF] from='client.? v1:172.21.15.121:0/1997101686' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.125029+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.253089+0000 osd.10 (osd.10) 118 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.310443+0000 osd.10 (osd.10) 119 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.922491+0000 osd.9 (osd.9) 109 : cluster [DBG] 1.a scrub starts 2024-02-12T16:28:55.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:55 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.979206+0000 osd.9 (osd.9) 110 : cluster [DBG] 1.a scrub ok 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.316867+0000 osd.5 (osd.5) 99 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.361605+0000 osd.5 (osd.5) 100 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.488166+0000 osd.11 (osd.11) 140 : cluster [DBG] 5.8 scrub starts 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.509062+0000 osd.11 (osd.11) 141 : cluster [DBG] 5.8 scrub ok 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.605485+0000 osd.7 (osd.7) 150 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.646012+0000 osd.7 (osd.7) 151 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.669550+0000 osd.3 (osd.3) 24 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.712417+0000 osd.3 (osd.3) 25 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.885814+0000 osd.6 (osd.6) 114 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:53.942652+0000 osd.6 (osd.6) 115 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.006760+0000 mgr.y (mgr.14146) 1321 : cluster [DBG] pgmap v1839: 37 pgs: 37 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: audit 2024-02-12T16:28:54.124957+0000 mon.a (mon.0) 2230 : audit [INF] from='client.? v1:172.21.15.121:0/1997101686' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.125029+0000 mon.a (mon.0) 2231 : cluster [DBG] osdmap e632: 12 total, 12 up, 12 in 2024-02-12T16:28:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.253089+0000 osd.10 (osd.10) 118 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:28:55.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.310443+0000 osd.10 (osd.10) 119 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:28:55.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.922491+0000 osd.9 (osd.9) 109 : cluster [DBG] 1.a scrub starts 2024-02-12T16:28:55.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:55 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.979206+0000 osd.9 (osd.9) 110 : cluster [DBG] 1.a scrub ok 2024-02-12T16:28:55.418 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:55.522 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:55.560 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:55.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:55.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:55.803 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:55.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:55.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:55.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:55.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:56.050 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:28:56.052 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:28:56.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:56.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:56.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:56.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.534846+0000 osd.4 (osd.4) 106 : cluster [DBG] 1.e scrub starts 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:54.613488+0000 osd.4 (osd.4) 107 : cluster [DBG] 1.e scrub ok 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.088220+0000 osd.10 (osd.10) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.088308+0000 osd.10 (osd.10) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.119640+0000 osd.0 (osd.0) 101 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.142203+0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e633: 12 total, 12 up, 12 in 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.212562+0000 osd.0 (osd.0) 102 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:28:56.234 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:56 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.685354+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e634: 12 total, 12 up, 12 in 2024-02-12T16:28:56.255 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:56.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:56.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:56.389 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.534846+0000 osd.4 (osd.4) 106 : cluster [DBG] 1.e scrub starts 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:54.613488+0000 osd.4 (osd.4) 107 : cluster [DBG] 1.e scrub ok 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.088220+0000 osd.10 (osd.10) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.088308+0000 osd.10 (osd.10) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.119640+0000 osd.0 (osd.0) 101 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.142203+0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e633: 12 total, 12 up, 12 in 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.212562+0000 osd.0 (osd.0) 102 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:28:56.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:56 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.685354+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e634: 12 total, 12 up, 12 in 2024-02-12T16:28:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:56.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.534846+0000 osd.4 (osd.4) 106 : cluster [DBG] 1.e scrub starts 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:54.613488+0000 osd.4 (osd.4) 107 : cluster [DBG] 1.e scrub ok 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.088220+0000 osd.10 (osd.10) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.088308+0000 osd.10 (osd.10) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.119640+0000 osd.0 (osd.0) 101 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.142203+0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e633: 12 total, 12 up, 12 in 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.212562+0000 osd.0 (osd.0) 102 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:28:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:56 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.685354+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e634: 12 total, 12 up, 12 in 2024-02-12T16:28:56.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:56.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:56.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:56.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:56.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:28:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:56.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:28:56.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:56.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:56.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:56.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:56.924 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1843,"stamp":"2024-02-12T16:28:56.007062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4832084,"kb_used_data":11572,"kb_used_omap":91,"kb_used_meta":4820324,"kb_avail":1120109740,"statfs":{"total":1151940427776,"available":1146992373760,"internally_reserved":0,"allocated":11849728,"data_stored":6005822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":4936011792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-338776,"num_write_kb":-10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3147,"num_bytes_recovered":-206241792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.117313"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":123,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-02-12T16:28:47.381508+0000","last_change":"2024-02-12T16:28:06.991266+0000","last_active":"2024-02-12T16:28:47.381508+0000","last_peered":"2024-02-12T16:28:47.381508+0000","last_clean":"2024-02-12T16:28:47.381508+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:28:47.381508+0000","last_undegraded":"2024-02-12T16:28:47.381508+0000","last_fullsized":"2024-02-12T16:28:47.381508+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:06.991147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:28:06.991147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":235,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-02-12T16:28:46.072076+0000","last_change":"2024-02-12T16:28:20.038276+0000","last_active":"2024-02-12T16:28:46.072076+0000","last_peered":"2024-02-12T16:28:46.072076+0000","last_clean":"2024-02-12T16:28:46.072076+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:28:46.072076+0000","last_undegraded":"2024-02-12T16:28:46.072076+0000","last_fullsized":"2024-02-12T16:28:46.072076+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:40.282662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:40.282662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":313,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-02-12T16:28:51.112621+0000","last_change":"2024-02-12T16:24:47.872602+0000","last_active":"2024-02-12T16:28:51.112621+0000","last_peered":"2024-02-12T16:28:51.112621+0000","last_clean":"2024-02-12T16:28:51.112621+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:28:51.112621+0000","last_undegraded":"2024-02-12T16:28:51.112621+0000","last_fullsized":"2024-02-12T16:28:51.112621+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:47.872500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:24:47.872500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":723,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-02-12T16:28:53.164276+0000","last_change":"2024-02-12T16:28:50.494186+0000","last_active":"2024-02-12T16:28:53.164276+0000","last_peered":"2024-02-12T16:28:53.164276+0000","last_clean":"2024-02-12T16:28:53.164276+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:28:53.164276+0000","last_undegraded":"2024-02-12T16:28:53.164276+0000","last_fullsized":"2024-02-12T16:28:53.164276+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:50.494005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:50.494005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":363,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-02-12T16:28:54.513502+0000","last_change":"2024-02-12T16:28:53.942753+0000","last_active":"2024-02-12T16:28:54.513502+0000","last_peered":"2024-02-12T16:28:54.513502+0000","last_clean":"2024-02-12T16:28:54.513502+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:28:54.513502+0000","last_undegraded":"2024-02-12T16:28:54.513502+0000","last_fullsized":"2024-02-12T16:28:54.513502+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.942656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.942656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":304,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-02-12T16:28:51.112597+0000","last_change":"2024-02-12T16:25:07.944563+0000","last_active":"2024-02-12T16:28:51.112597+0000","last_peered":"2024-02-12T16:28:51.112597+0000","last_clean":"2024-02-12T16:28:51.112597+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:28:51.112597+0000","last_undegraded":"2024-02-12T16:28:51.112597+0000","last_fullsized":"2024-02-12T16:28:51.112597+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:07.944451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:07.944451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":314,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-02-12T16:28:51.112596+0000","last_change":"2024-02-12T16:28:14.154056+0000","last_active":"2024-02-12T16:28:51.112596+0000","last_peered":"2024-02-12T16:28:51.112596+0000","last_clean":"2024-02-12T16:28:51.112596+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:28:51.112596+0000","last_undegraded":"2024-02-12T16:28:51.112596+0000","last_fullsized":"2024-02-12T16:28:51.112596+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:37.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:25:37.899168+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":179,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-02-12T16:28:53.712550+0000","last_change":"2024-02-12T16:28:53.712516+0000","last_active":"2024-02-12T16:28:53.712550+0000","last_peered":"2024-02-12T16:28:53.712550+0000","last_clean":"2024-02-12T16:28:53.712550+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:28:53.712550+0000","last_undegraded":"2024-02-12T16:28:53.712550+0000","last_fullsized":"2024-02-12T16:28:53.712550+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.712421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.712421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":744,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-02-12T16:28:51.108612+0000","last_change":"2024-02-12T16:28:19.557009+0000","last_active":"2024-02-12T16:28:51.108612+0000","last_peered":"2024-02-12T16:28:51.108612+0000","last_clean":"2024-02-12T16:28:51.108612+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:28:51.108612+0000","last_undegraded":"2024-02-12T16:28:51.108612+0000","last_fullsized":"2024-02-12T16:28:51.108612+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:04.914304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:04.914304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":119,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-02-12T16:28:51.109412+0000","last_change":"2024-02-12T16:26:43.025974+0000","last_active":"2024-02-12T16:28:51.109412+0000","last_peered":"2024-02-12T16:28:51.109412+0000","last_clean":"2024-02-12T16:28:51.109412+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:28:51.109412+0000","last_undegraded":"2024-02-12T16:28:51.109412+0000","last_fullsized":"2024-02-12T16:28:51.109412+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:43.025867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:43.025867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":151,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-02-12T16:28:53.122344+0000","last_change":"2024-02-12T16:28:51.703479+0000","last_active":"2024-02-12T16:28:53.122344+0000","last_peered":"2024-02-12T16:28:53.122344+0000","last_clean":"2024-02-12T16:28:53.122344+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:28:53.122344+0000","last_undegraded":"2024-02-12T16:28:53.122344+0000","last_fullsized":"2024-02-12T16:28:53.122344+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:51.703298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:51.703298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":319,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-02-12T16:28:54.513375+0000","last_change":"2024-02-12T16:25:45.967785+0000","last_active":"2024-02-12T16:28:54.513375+0000","last_peered":"2024-02-12T16:28:54.513375+0000","last_clean":"2024-02-12T16:28:54.513375+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:28:54.513375+0000","last_undegraded":"2024-02-12T16:28:54.513375+0000","last_fullsized":"2024-02-12T16:28:54.513375+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:45.967726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:25:45.967726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":121,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-02-12T16:28:51.108713+0000","last_change":"2024-02-12T16:26:59.766851+0000","last_active":"2024-02-12T16:28:51.108713+0000","last_peered":"2024-02-12T16:28:51.108713+0000","last_clean":"2024-02-12T16:28:51.108713+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:28:51.108713+0000","last_undegraded":"2024-02-12T16:28:51.108713+0000","last_fullsized":"2024-02-12T16:28:51.108713+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:59.766744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:59.766744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":719,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-02-12T16:28:51.109442+0000","last_change":"2024-02-12T16:26:48.943869+0000","last_active":"2024-02-12T16:28:51.109442+0000","last_peered":"2024-02-12T16:28:51.109442+0000","last_clean":"2024-02-12T16:28:51.109442+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:28:51.109442+0000","last_undegraded":"2024-02-12T16:28:51.109442+0000","last_fullsized":"2024-02-12T16:28:51.109442+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:48.943785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:48.943785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":305,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-02-12T16:28:47.052449+0000","last_change":"2024-02-12T16:24:50.575354+0000","last_active":"2024-02-12T16:28:47.052449+0000","last_peered":"2024-02-12T16:28:47.052449+0000","last_clean":"2024-02-12T16:28:47.052449+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:28:47.052449+0000","last_undegraded":"2024-02-12T16:28:47.052449+0000","last_fullsized":"2024-02-12T16:28:47.052449+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:24:50.575287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:24:50.575287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":60,"reported_epoch":631,"state":"active+clean","last_fresh":"2024-02-12T16:28:53.164171+0000","last_change":"2024-02-12T16:26:10.388710+0000","last_active":"2024-02-12T16:28:53.164171+0000","last_peered":"2024-02-12T16:28:53.164171+0000","last_clean":"2024-02-12T16:28:53.164171+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:28:53.164171+0000","last_undegraded":"2024-02-12T16:28:53.164171+0000","last_fullsized":"2024-02-12T16:28:53.164171+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:26:10.388515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:26:10.388515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":294,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-02-12T16:28:46.072189+0000","last_change":"2024-02-12T16:25:03.361871+0000","last_active":"2024-02-12T16:28:46.072189+0000","last_peered":"2024-02-12T16:28:46.072189+0000","last_clean":"2024-02-12T16:28:46.072189+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:28:46.072189+0000","last_undegraded":"2024-02-12T16:28:46.072189+0000","last_fullsized":"2024-02-12T16:28:46.072189+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:25:03.361757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":114,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-02-12T16:28:47.052387+0000","last_change":"2024-02-12T16:25:59.785619+0000","last_active":"2024-02-12T16:28:47.052387+0000","last_peered":"2024-02-12T16:28:47.052387+0000","last_clean":"2024-02-12T16:28:47.052387+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:28:47.052387+0000","last_undegraded":"2024-02-12T16:28:47.052387+0000","last_fullsized":"2024-02-12T16:28:47.052387+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:59.785496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:25:59.785496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":121,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.212658+0000","last_change":"2024-02-12T16:28:55.212658+0000","last_active":"2024-02-12T16:28:55.212658+0000","last_peered":"2024-02-12T16:28:55.212658+0000","last_clean":"2024-02-12T16:28:55.212658+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:28:55.212658+0000","last_undegraded":"2024-02-12T16:28:55.212658+0000","last_fullsized":"2024-02-12T16:28:55.212658+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:55.212566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:55.212566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":128,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-02-12T16:28:47.052368+0000","last_change":"2024-02-12T16:25:47.933452+0000","last_active":"2024-02-12T16:28:47.052368+0000","last_peered":"2024-02-12T16:28:47.052368+0000","last_clean":"2024-02-12T16:28:47.052368+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:28:47.052368+0000","last_undegraded":"2024-02-12T16:28:47.052368+0000","last_fullsized":"2024-02-12T16:28:47.052368+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:47.933341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:25:47.933341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":108,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-02-12T16:28:47.361540+0000","last_change":"2024-02-12T16:25:50.908791+0000","last_active":"2024-02-12T16:28:47.361540+0000","last_peered":"2024-02-12T16:28:47.361540+0000","last_clean":"2024-02-12T16:28:47.361540+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:28:47.361540+0000","last_undegraded":"2024-02-12T16:28:47.361540+0000","last_fullsized":"2024-02-12T16:28:47.361540+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:25:50.908692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:25:50.908692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586592,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":93158560,"statfs":{"total":95995035648,"available":95394365440,"internally_reserved":0,"allocated":1007616,"data_stored":509328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.77000000000000002,"15min":0.84199999999999997},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.3420000000000001,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":1.046},{"interface":"front","average":{"1min":1.391,"5min":0.92700000000000005,"15min":0.90600000000000003},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.2279999999999998,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":1.129}]},{"osd":1,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":0.91200000000000003,"15min":1.056},"min":{"1min":0.39300000000000002,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":5.5430000000000001,"5min":5.5430000000000001,"15min":7.0110000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.089,"5min":0.89600000000000002,"15min":1.0609999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":3.6859999999999999,"5min":3.6859999999999999,"15min":5.6920000000000002},"last":0.69699999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":0.92600000000000005,"15min":1.274},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.1639999999999997,"5min":5.343,"15min":5.343},"last":1.4379999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.94399999999999995,"15min":1.3420000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.617,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":0.99099999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":0.86599999999999999,"15min":1.0349999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.29499999999999998},"max":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":6.5970000000000004},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.84799999999999998,"15min":0.93100000000000005},"min":{"1min":0.434,"5min":0.36799999999999999,"15min":0.34499999999999997},"max":{"1min":1.165,"5min":4.101,"15min":4.915},"last":1.2450000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.101,"15min":1.1419999999999999},"min":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"max":{"1min":3.7719999999999998,"5min":5.04,"15min":5.8789999999999996},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.98299999999999998,"15min":1.127},"min":{"1min":0.47799999999999998,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":4.0099999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.504}]},{"osd":6,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.875,"15min":0.99299999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.373},"max":{"1min":3.0819999999999999,"5min":4.7370000000000001,"15min":5.141},"last":1.472},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.014,"15min":1.1279999999999999},"min":{"1min":0.46400000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.1630000000000003,"5min":4.7450000000000001,"15min":8.5709999999999997},"last":1.5660000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.93100000000000005,"15min":1.1910000000000001},"min":{"1min":0.49399999999999999,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":3.9089999999999998,"5min":3.9089999999999998,"15min":6.4370000000000003},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.82499999999999996,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.5469999999999999,"5min":4.2350000000000003,"15min":5.0430000000000001},"last":1.3560000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":6.1779999999999999,"15min":2.8039999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.0329999999999999,"5min":537.54399999999998,"15min":537.54399999999998},"last":1.155},{"interface":"front","average":{"1min":1.0009999999999999,"5min":6.3239999999999998,"15min":2.98},"min":{"1min":0.51500000000000001,"5min":0.47399999999999998,"15min":0.34699999999999998},"max":{"1min":3.8719999999999999,"5min":539.86099999999999,"15min":539.86099999999999},"last":1.524}]},{"osd":9,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1,"15min":1.2749999999999999},"min":{"1min":0.56899999999999995,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":10.743},"last":1.288},{"interface":"front","average":{"1min":1.05,"5min":0.998,"15min":1.363},"min":{"1min":0.55300000000000005,"5min":0.48799999999999999,"15min":0.39500000000000002},"max":{"1min":3.8399999999999999,"5min":3.8399999999999999,"15min":10.698},"last":1.6379999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":6.3019999999999996,"15min":2.96},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.433},"max":{"1min":3.1880000000000002,"5min":536.245,"15min":536.245},"last":1.3200000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":6.3170000000000002,"15min":2.9460000000000002},"min":{"1min":0.53200000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.6899999999999999,"5min":536.64300000000003,"15min":536.64300000000003},"last":1.6080000000000001}]}]},{"osd":10,"up_from":83,"seq":356482285946,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":978944,"data_stored":496826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":2.6070000000000002,"15min":1.5169999999999999},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":5.867,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.751},{"interface":"front","average":{"1min":0.996,"5min":2.6499999999999999,"15min":1.5880000000000001},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.5060000000000002,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.80600000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.028,"15min":0.97999999999999998},"min":{"1min":0.314,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":5.9000000000000004,"5min":5.9000000000000004,"15min":19.866},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.431,"5min":0.94399999999999995,"15min":1.1000000000000001},"min":{"1min":0.33000000000000002,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":20.277999999999999},"last":1.115}]},{"osd":2,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":2.077,"5min":1.1419999999999999,"15min":0.98199999999999998},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.2430000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.1339999999999999,"15min":1.0009999999999999},"min":{"1min":0.40200000000000002,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.96599999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85699999999999998,"15min":3.3359999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.32300000000000001},"max":{"1min":5.8109999999999999,"5min":5.8109999999999999,"15min":628.77099999999996},"last":1.335},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.98699999999999999,"15min":3.3100000000000001},"min":{"1min":0.34100000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.3070000000000004,"5min":7.3070000000000004,"15min":624.19299999999998},"last":1.226}]},{"osd":5,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.41099999999999998,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":6.343,"5min":6.343,"15min":6.343},"last":1.292},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.099,"15min":1.1379999999999999},"min":{"1min":0.48199999999999998,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":7.0339999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":1.2589999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.028,"15min":3.3879999999999999},"min":{"1min":0.42099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":626.26599999999996},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0049999999999999,"15min":3.4609999999999999},"min":{"1min":0.55900000000000005,"5min":0.376,"15min":0.376},"max":{"1min":6.8200000000000003,"5min":6.8200000000000003,"15min":625.90700000000004},"last":1.3999999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":0.94899999999999995,"15min":1.0700000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":7.3280000000000003,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.1579999999999999,"15min":1.1870000000000001},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":6.4539999999999997,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":1.4510000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.97799999999999998,"15min":1.1240000000000001},"min":{"1min":0.44400000000000001,"5min":0.436,"15min":0.372},"max":{"1min":5.758,"5min":5.758,"15min":5.758},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.135,"5min":0.98899999999999999,"15min":1.133},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.282,"5min":7.282,"15min":7.282},"last":1.5109999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1100000000000001,"15min":1.2250000000000001},"min":{"1min":0.505,"5min":0.47599999999999998,"15min":0.372},"max":{"1min":5.7809999999999997,"5min":5.7809999999999997,"15min":5.7809999999999997},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.0089999999999999,"15min":1.212},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":7.3479999999999999,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":1.476}]},{"osd":11,"last update":"Mon Feb 12 16:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.115,"15min":1.288},"min":{"1min":0.40699999999999997,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":3.2280000000000002,"5min":7.3689999999999998,"15min":10.218999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.087,"15min":1.2889999999999999},"min":{"1min":0.63400000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":6.5860000000000003,"5min":6.5860000000000003,"15min":7.3250000000000002},"last":1.484}]}]},{"osd":9,"up_from":74,"seq":317827580290,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306360,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438792,"statfs":{"total":95995035648,"available":95681323008,"internally_reserved":0,"allocated":966656,"data_stored":490606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.1930000000000001,"15min":1.1020000000000001},"min":{"1min":0.30599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.9560000000000004,"5min":7.9560000000000004,"15min":7.9560000000000004},"last":0.95099999999999996},{"interface":"front","average":{"1min":2.419,"5min":1.2370000000000001,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":1.1000000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:28:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.1499999999999999,"15min":0.995},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.253},"max":{"1min":5.7750000000000004,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.482,"5min":1.0780000000000001,"15min":1.0669999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":5.7910000000000004,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":1.0309999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.956,"5min":1.268,"15min":1.4730000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":12.441000000000001,"5min":12.441000000000001,"15min":97.275000000000006},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.1279999999999999,"15min":1.3129999999999999},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.312},"max":{"1min":4.6239999999999997,"5min":6.1779999999999999,"15min":97.417000000000002},"last":0.80700000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.028,"15min":0.96899999999999997},"min":{"1min":0.35999999999999999,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":4.3940000000000001,"5min":4.569,"15min":4.569},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.1539999999999999,"15min":1.145},"min":{"1min":0.38,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.6589999999999998,"5min":4.8840000000000003,"15min":5.8010000000000002},"last":1.2130000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.304,"15min":1.1279999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":4.8780000000000001,"5min":6.726,"15min":6.726},"last":0.999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.149},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.9059999999999997,"5min":6.7350000000000003,"15min":7.3739999999999997},"last":1.254}]},{"osd":6,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0189999999999999,"15min":1.0880000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":4.6749999999999998,"5min":5.5250000000000004,"15min":7.3360000000000003},"last":1.159},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1699999999999999,"15min":1.0660000000000001},"min":{"1min":0.51200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.4610000000000003,"5min":7.0609999999999999,"15min":7.0609999999999999},"last":1.3089999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.26,"15min":1.238},"min":{"1min":0.433,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":4.6399999999999997,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.216},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.7000000000000002,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":1.3400000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0449999999999999,"15min":1.077},"min":{"1min":0.42499999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":1.423},{"interface":"front","average":{"1min":1.006,"5min":1.0680000000000001,"15min":1.1839999999999999},"min":{"1min":0.56100000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":7.25},"last":1.399}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.2,"15min":1.2549999999999999},"min":{"1min":0.57099999999999995,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":4.6900000000000004,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.244,"15min":1.266},"min":{"1min":0.55800000000000005,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":1.4450000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.141,"15min":1.3260000000000001},"min":{"1min":0.60499999999999998,"5min":0.48999999999999999,"15min":0.45600000000000002},"max":{"1min":1.4379999999999999,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.0700000000000001,"15min":1.2829999999999999},"min":{"1min":0.52500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.8929999999999998,"5min":5.492,"15min":6.3639999999999999},"last":1.4890000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874636,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":978944,"data_stored":496834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.0289999999999999,"15min":0.88300000000000001},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":2.4279999999999999,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92400000000000004},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":7.1840000000000002,"5min":18.584,"15min":18.584},"last":0.78000000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":1.968,"15min":1.5329999999999999},"min":{"1min":0.33300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":15.981999999999999,"5min":69.956000000000003,"15min":69.956000000000003},"last":1.181},{"interface":"front","average":{"1min":1.988,"5min":1.97,"15min":1.625},"min":{"1min":0.44,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":15.817,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.91900000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.82499999999999996,"15min":0.83499999999999996},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0510000000000002,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91600000000000004,"15min":0.99299999999999999},"min":{"1min":0.36699999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.5569999999999999,"5min":4.7240000000000002,"15min":4.7240000000000002},"last":1.129}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.77500000000000002,"15min":1.71},"min":{"1min":0.378,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.89300000000000002,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.81299999999999994,"15min":1.6559999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":1.1519999999999999,"5min":5.0380000000000003,"15min":224.71000000000001},"last":1.3500000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.872,"15min":0.91100000000000003},"min":{"1min":0.39600000000000002,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":5.4059999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":1.103},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.93799999999999994,"15min":1.016},"min":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.8170000000000002,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.2330000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.071,"15min":1.742},"min":{"1min":0.373,"5min":0.373,"15min":0.37},"max":{"1min":3.8050000000000002,"5min":5.3639999999999999,"15min":221.36699999999999},"last":1.167},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.99299999999999999,"15min":1.843},"min":{"1min":0.497,"5min":0.38,"15min":0.38},"max":{"1min":1.1919999999999999,"5min":4.4930000000000003,"15min":221.203},"last":1.506}]},{"osd":7,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.052,"15min":1.131},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":2.5619999999999998,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":1.3879999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.1279999999999999,"15min":1.2310000000000001},"min":{"1min":0.54000000000000004,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":2.5779999999999998,"5min":4.2240000000000002,"15min":7.899},"last":1.5700000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.028,"15min":1.165},"min":{"1min":0.55300000000000005,"5min":0.371,"15min":0.35599999999999998},"max":{"1min":2.6419999999999999,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.006,"5min":1.117,"15min":1.2390000000000001},"min":{"1min":0.60299999999999998,"5min":0.47599999999999998,"15min":0.44700000000000001},"max":{"1min":2.6709999999999998,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":1.6140000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.97199999999999998,"15min":1.4399999999999999},"min":{"1min":0.437,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.6120000000000001,"5min":5.2560000000000002,"15min":109.92},"last":1.081},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.999,"15min":1.484},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.38500000000000001},"max":{"1min":1.786,"5min":5.0250000000000004,"15min":110.249},"last":1.3169999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.028,"15min":1.1830000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":5.2530000000000001,"5min":5.2530000000000001,"15min":5.2530000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.135,"5min":0.93300000000000005,"15min":1.143},"min":{"1min":0.50700000000000001,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.4099999999999999}]}]},{"osd":7,"up_from":56,"seq":240518168979,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298180,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446972,"statfs":{"total":95995035648,"available":95689699328,"internally_reserved":0,"allocated":978944,"data_stored":496834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":3.1120000000000001,"15min":1.673},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.8310000000000004,"5min":187.33699999999999,"15min":187.33699999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":3.0539999999999998,"15min":1.6619999999999999},"min":{"1min":0.32100000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.4580000000000002,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.85099999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.302,"15min":1.056},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4729999999999999,"5min":35.655000000000001,"15min":35.655000000000001},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.6060000000000001,"15min":1.2130000000000001},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.6840000000000002,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90000000000000002,"15min":1.198},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.7869999999999999,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.89600000000000002,"15min":0.98499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.7549999999999999,"5min":6.7549999999999999,"15min":6.7549999999999999},"last":1.355}]},{"osd":4,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.145,"15min":0.95699999999999996},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.0049999999999999,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.95999999999999996,"15min":1.0029999999999999},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":6.9470000000000001,"5min":6.9470000000000001,"15min":6.9470000000000001},"last":1.5209999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.97799999999999998,"15min":0.95099999999999996},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9379999999999997,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.0600000000000001,"15min":1.048},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.8840000000000003,"5min":5.992,"15min":9.3640000000000008},"last":1.054}]},{"osd":6,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0449999999999999,"15min":1.081},"min":{"1min":0.38800000000000001,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":4.7409999999999997,"5min":6.319,"15min":6.319},"last":1.079},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.173,"15min":1.1890000000000001},"min":{"1min":0.39300000000000002,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":1.6639999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":6.5750000000000002,"15min":2.9449999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.1619999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.635,"5min":6.6379999999999999,"15min":3.0310000000000001},"min":{"1min":0.50800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":9.9060000000000006,"5min":498.69999999999999,"15min":498.69999999999999},"last":1.27}]},{"osd":9,"last update":"Mon Feb 12 16:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.97899999999999998,"15min":1.1579999999999999},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.8029999999999999,"5min":4.6539999999999999,"15min":12.907999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.171,"15min":1.2669999999999999},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.7800000000000002,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":1.762}]},{"osd":10,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":7.7859999999999996,"15min":3.5179999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":7.0339999999999998,"5min":497.52699999999999,"15min":497.52699999999999},"last":1.8560000000000001},{"interface":"front","average":{"1min":1.6990000000000001,"5min":7.6740000000000004,"15min":3.423},"min":{"1min":0.50600000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.5890000000000004,"5min":496.29599999999999,"15min":496.29599999999999},"last":1.4630000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.1060000000000001,"15min":1.601},"min":{"1min":0.495,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":4.9039999999999999,"5min":6.0490000000000004,"15min":120.943},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.252,"15min":1.714},"min":{"1min":0.48699999999999999,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":120.661},"last":1.9650000000000001}]}]},{"osd":6,"up_from":48,"seq":206158430620,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302304,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442848,"statfs":{"total":95995035648,"available":95685476352,"internally_reserved":0,"allocated":1007616,"data_stored":509336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":7.04,"5min":2.367,"15min":1.464},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":107.723,"5min":107.723,"15min":107.723},"last":0.81200000000000006},{"interface":"front","average":{"1min":7.1260000000000003,"5min":2.5139999999999998,"15min":1.552},"min":{"1min":0.38,"5min":0.252,"15min":0.252},"max":{"1min":107.114,"5min":107.114,"15min":107.114},"last":0.73299999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0449999999999999,"15min":1.923},"min":{"1min":0.36499999999999999,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":3.5870000000000002,"5min":5.5179999999999998,"15min":252.22800000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.05,"15min":1.946},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.0549999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5009999999999999,"15min":1.0649999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.1369999999999996,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.393,"5min":1.7090000000000001,"15min":1.1719999999999999},"min":{"1min":0.33700000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3769999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.67700000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.2529999999999999,"15min":1.024},"min":{"1min":0.29099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.4509999999999996,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.079,"15min":1.0620000000000001},"min":{"1min":0.34999999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.242,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.117,"15min":0.99099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":4.4779999999999998,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.99299999999999999,"15min":1.0109999999999999},"min":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1080000000000001,"5min":10.52,"15min":10.52},"last":0.47599999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.97299999999999998,"15min":1.089},"min":{"1min":0.34200000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.399,"5min":4.7149999999999999,"15min":7.641},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.81000000000000005,"15min":1.034},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.679,"5min":4.5609999999999999,"15min":7.8380000000000001},"last":0.56699999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1200000000000001,"15min":1.1259999999999999},"min":{"1min":0.48899999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.5119999999999996,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":1.071},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":1.175},"min":{"1min":0.55100000000000005,"5min":0.371,"15min":0.371},"max":{"1min":2.7679999999999998,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":1.115}]},{"osd":9,"last update":"Mon Feb 12 16:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.246},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.8089999999999999,"5min":4.1680000000000001,"15min":18.059999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.96099999999999997,"15min":1.2470000000000001},"min":{"1min":0.45300000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9970000000000001,"5min":2.673,"15min":17.425000000000001},"last":0.89800000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.24,"15min":1.2909999999999999},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.7400000000000002,"5min":5.726,"15min":6.5300000000000002},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.337,"15min":1.3839999999999999},"min":{"1min":0.57699999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.2080000000000002,"5min":5.7590000000000003,"15min":6.5730000000000004},"last":0.98299999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2290000000000001,"15min":2.48},"min":{"1min":0.54600000000000004,"5min":0.52100000000000002,"15min":0.499},"max":{"1min":2.7970000000000002,"5min":5.6859999999999999,"15min":139.727},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.995,"5min":0.98899999999999999,"15min":2.3559999999999999},"min":{"1min":0.55400000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.173,"5min":2.706,"15min":139.24199999999999},"last":1.0209999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895160,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":894208,"kb_avail":92849992,"statfs":{"total":95995035648,"available":95078391808,"internally_reserved":0,"allocated":966656,"data_stored":490606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915668992},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.129,"15min":0.96099999999999997},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":4.4610000000000003,"5min":16.632999999999999,"15min":16.632999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.645,"15min":1.212},"min":{"1min":0.33200000000000002,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":6.4939999999999998,"5min":7.7249999999999996,"15min":11.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.345,"15min":1.1240000000000001},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":12.778,"5min":12.778,"15min":12.778},"last":0.57599999999999996},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.5569999999999999,"15min":1.216},"min":{"1min":0.28299999999999997,"5min":0.246,"15min":0.246},"max":{"1min":8.9329999999999998,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.69999999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.0920000000000001,"15min":1.1200000000000001},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.625,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.1910000000000001,"15min":0.95499999999999996},"min":{"1min":0.38300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.1639999999999997,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.274,"15min":1.1220000000000001},"min":{"1min":0.42499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":3.536,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.417,"15min":1.389},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.6859999999999999,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.77200000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.482,"15min":1.27},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.7789999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.268,"5min":1.347,"15min":1.2190000000000001},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":4.1340000000000003,"5min":10.067,"15min":10.067},"last":1.05}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.6799999999999999,"15min":1.7849999999999999},"min":{"1min":0.503,"5min":0.439,"15min":0.36399999999999999},"max":{"1min":3.5,"5min":30.437000000000001,"15min":55.869999999999997},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.599,"15min":1.8540000000000001},"min":{"1min":0.47299999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":4.1680000000000001,"5min":29.382000000000001,"15min":55.892000000000003},"last":1.1599999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":7.4459999999999997,"15min":3.4529999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.2300000000000004,"5min":548.91600000000005,"15min":548.91600000000005},"last":1.319},{"interface":"front","average":{"1min":1.0680000000000001,"5min":7.5579999999999998,"15min":3.4359999999999999},"min":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.3769999999999998,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.95099999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:27:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.4450000000000001,"15min":1.3959999999999999},"min":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":5.7590000000000003,"5min":5.7590000000000003,"15min":10.07},"last":1.077},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.323,"15min":1.46},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":10.106,"5min":10.106,"15min":10.106},"last":1.1950000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":8.4459999999999997,"15min":3.7690000000000001},"min":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.105,"5min":548.93499999999995,"15min":548.93499999999995},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":8.4209999999999994,"15min":3.8460000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1989999999999998,"5min":549.12099999999998,"15min":549.12099999999998},"last":1.2450000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.536,"15min":1.5880000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":4.2619999999999996,"5min":10.202,"15min":10.202},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.552,"15min":1.6359999999999999},"min":{"1min":0.52200000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.1710000000000003,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":1.141}]}]},{"osd":0,"up_from":8,"seq":34359738829,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590688,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":93154464,"statfs":{"total":95995035648,"available":95390171136,"internally_reserved":0,"allocated":1007616,"data_stored":509336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.1830000000000001,"15min":1.0800000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":3.6890000000000001,"5min":10.754,"15min":11.324},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.27,"5min":1.401,"15min":1.179},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1020000000000003,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0600000000000001,"15min":1.056},"min":{"1min":0.26500000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.032,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.1579999999999999,"15min":1.002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":3.8050000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.74399999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":3.5310000000000001,"15min":2.1269999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":5.4729999999999999,"5min":193.77000000000001,"15min":193.77000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.5469999999999999,"5min":3.4790000000000001,"15min":2.1909999999999998},"min":{"1min":0.29499999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":5.141,"5min":189.625,"15min":189.625},"last":1.022}]},{"osd":5,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9350000000000001,"5min":1.3320000000000001,"15min":1.0780000000000001},"min":{"1min":0.29799999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.49,"5min":10.49,"15min":10.49},"last":1.236},{"interface":"front","average":{"1min":1.772,"5min":1.3200000000000001,"15min":1.0940000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":7.3250000000000002,"5min":7.3250000000000002,"15min":7.3250000000000002},"last":0.96999999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":10.422000000000001,"5min":3.161,"15min":1.9890000000000001},"min":{"1min":0.371,"5min":0.311,"15min":0.311},"max":{"1min":189.79400000000001,"5min":189.79400000000001,"15min":189.79400000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":10.676,"5min":3.238,"15min":2.0680000000000001},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":194.41200000000001,"5min":194.41200000000001,"15min":194.41200000000001},"last":1.3169999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":3.5750000000000002,"15min":1.9450000000000001},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":7.4509999999999996,"5min":191.47200000000001,"15min":191.47200000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.6319999999999999,"5min":3.6760000000000002,"15min":2.052},"min":{"1min":0.47399999999999998,"5min":0.44,"15min":0.44},"max":{"1min":7.4779999999999998,"5min":191.50299999999999,"15min":191.50299999999999},"last":1.52}]},{"osd":8,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.3480000000000001,"15min":1.3160000000000001},"min":{"1min":0.45900000000000002,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":7.3920000000000003,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.4399999999999999,"15min":1.4059999999999999},"min":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":7.5060000000000002,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":1.655}]},{"osd":9,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.2010000000000001,"15min":1.361},"min":{"1min":0.46899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.6550000000000002,"5min":7.6539999999999999,"15min":46.246000000000002},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.345,"15min":1.4219999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.5330000000000004,"5min":5.5330000000000004,"15min":46.167999999999999},"last":1.595}]},{"osd":10,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.52,"15min":1.377},"min":{"1min":0.45100000000000001,"5min":0.442,"15min":0.41399999999999998},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.6459999999999999,"15min":1.4079999999999999},"min":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":9.3490000000000002,"5min":9.3490000000000002,"15min":9.3490000000000002},"last":1.5489999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.556,"15min":1.617},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":46.064},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.895,"5min":1.458,"15min":1.5629999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":7.4210000000000003,"5min":7.4210000000000003,"15min":45.642000000000003},"last":1.4470000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313658,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306372,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438780,"statfs":{"total":95995035648,"available":95681310720,"internally_reserved":0,"allocated":978944,"data_stored":496834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.3560000000000001,"15min":1.1080000000000001},"min":{"1min":0.26000000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.6440000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.4970000000000001,"15min":1.2170000000000001},"min":{"1min":0.33900000000000002,"5min":0.247,"15min":0.247},"max":{"1min":4.5300000000000002,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":0.99299999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.52,"15min":1.9239999999999999},"min":{"1min":0.29599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":6.2910000000000004,"5min":21.407,"15min":21.407},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1020000000000001,"15min":0.92000000000000004},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.8899999999999997,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.71199999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.393,"15min":1.079},"min":{"1min":0.33500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0019999999999998,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.419,"15min":1.1839999999999999},"min":{"1min":0.33600000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.6060000000000001,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":1.222}]},{"osd":5,"last update":"Mon Feb 12 16:28:33 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.365,"15min":1.1539999999999999},"min":{"1min":0.38200000000000001,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":4.8040000000000003,"5min":6.133,"15min":6.133},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6930000000000001,"15min":1.22},"min":{"1min":0.437,"5min":0.315,"15min":0.315},"max":{"1min":6.8170000000000002,"5min":11.081,"15min":11.081},"last":0.84699999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.653,"15min":1.278},"min":{"1min":0.45500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.4980000000000002,"5min":21.527000000000001,"15min":21.527000000000001},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.3759999999999999,"15min":1.2050000000000001},"min":{"1min":0.41999999999999998,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":6.1710000000000003,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":1.1519999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.6359999999999999,"15min":1.234},"min":{"1min":0.39500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.1260000000000003,"5min":11.044,"15min":11.044},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.2629999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.8739999999999997,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":1.034}]},{"osd":8,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3919999999999999,"15min":1.2909999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42099999999999999},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":4.8360000000000003},{"interface":"front","average":{"1min":1.504,"5min":1.462,"15min":1.327},"min":{"1min":0.46000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":6.593,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":4.7389999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.607,"15min":1.3500000000000001},"min":{"1min":0.51500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":6.0910000000000002,"5min":10.972,"15min":10.972},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.71,"15min":1.429},"min":{"1min":0.45600000000000002,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":6.1600000000000001,"5min":11.006,"15min":11.006},"last":1.286}]},{"osd":10,"last update":"Mon Feb 12 16:28:33 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":1.702,"15min":1.454},"min":{"1min":0.52200000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":7.7539999999999996,"5min":11.454000000000001,"15min":11.454000000000001},"last":1.843},{"interface":"front","average":{"1min":1.9299999999999999,"5min":1.7210000000000001,"15min":1.476},"min":{"1min":0.55300000000000005,"5min":0.441,"15min":0.441},"max":{"1min":7.7140000000000004,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":1.988}]},{"osd":11,"last update":"Mon Feb 12 16:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.389,"15min":1.502},"min":{"1min":0.60999999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":6.2599999999999998,"5min":6.2599999999999998,"15min":6.2599999999999998},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.718,"5min":1.5069999999999999,"15min":1.5760000000000001},"min":{"1min":0.59499999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":6.2329999999999997,"5min":6.2329999999999997,"15min":6.2329999999999997},"last":4.7699999999999996}]}]},{"osd":3,"up_from":522,"seq":2241972928584,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333216,"kb_used_data":984,"kb_used_omap":91,"kb_used_meta":332132,"kb_avail":93411936,"statfs":{"total":95995035648,"available":95653822464,"internally_reserved":0,"allocated":1007616,"data_stored":509336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":340103184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.3280000000000001,"15min":1.0069999999999999},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.5529999999999999,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.4570000000000001,"15min":1.054},"min":{"1min":0.373,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.536,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.2270000000000001,"15min":0.98799999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5410000000000004,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.196,"15min":1.1020000000000001},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.1040000000000001,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":0.94899999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.89800000000000002,"15min":1.25},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":4.0899999999999999,"5min":10.004,"15min":10.004},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.032,"15min":1.0980000000000001},"min":{"1min":0.35999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.806,"5min":6.806,"15min":6.806},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.0820000000000001,"15min":1.325},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.316},"max":{"1min":4.7530000000000001,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.318,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.1340000000000003,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":1.044}]},{"osd":6,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.0780000000000001,"15min":1.1319999999999999},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.7450000000000001,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.002,"15min":1.012},"min":{"1min":0.38300000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":4.1689999999999996,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":1.113}]},{"osd":7,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.1140000000000001,"15min":1.4099999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.21,"5min":6.851,"15min":6.851},"last":1.149},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.218,"15min":1.2769999999999999},"min":{"1min":0.41799999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.8479999999999999,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":0.85599999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.0449999999999999,"15min":1.173},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.8109999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.99399999999999999,"15min":1.0980000000000001},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.2430000000000003,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.999}]},{"osd":9,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.2909999999999999,"15min":1.331},"min":{"1min":0.45500000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.1539999999999999,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.087,"5min":1.335,"15min":1.3799999999999999},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.863,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.753}]},{"osd":10,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.2130000000000001,"15min":1.4750000000000001},"min":{"1min":0.54600000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.8700000000000001,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.3089999999999999,"15min":1.3260000000000001},"min":{"1min":0.501,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.8180000000000001,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":0.79100000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.242,"15min":1.284},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.9870000000000001,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.2629999999999999,"15min":1.4870000000000001},"min":{"1min":0.63300000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.8399999999999999,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.88700000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310480,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434672,"statfs":{"total":95995035648,"available":95677104128,"internally_reserved":0,"allocated":991232,"data_stored":503112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:27:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.093,"15min":1.363},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.4710000000000001,"5min":5.4710000000000001,"15min":90.597999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.143,"5min":1.157,"15min":1.5409999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.5389999999999997,"5min":12.236000000000001,"15min":90.676000000000002},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.022,"15min":0.95999999999999996},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.577,"5min":5.577,"15min":8.0530000000000008},"last":1.232},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.1619999999999999,"15min":1.1279999999999999},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":6.718,"5min":6.718,"15min":7.452},"last":0.91300000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":1.1259999999999999,"15min":1.1020000000000001},"min":{"1min":0.33300000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.2930000000000001,"5min":12.409000000000001,"15min":12.409000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.018,"15min":1.046},"min":{"1min":0.35499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.657,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":1.325}]},{"osd":5,"last update":"Mon Feb 12 16:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.96299999999999997,"15min":0.93100000000000005},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.0529999999999999,"5min":4.7009999999999996,"15min":5.5999999999999996},"last":1.1919999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.41299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.081,"5min":4.7160000000000002,"15min":5.7450000000000001},"last":1.4319999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.83299999999999996,"15min":0.97799999999999998},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.8660000000000001,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.218,"15min":1.071},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.5190000000000001,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":0.86399999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1379999999999999,"15min":1.101},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":5.4870000000000001,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.88100000000000001,"15min":1.143},"min":{"1min":0.36099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.6459999999999999,"5min":3.6459999999999999,"15min":7.6689999999999996},"last":1.6160000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.002,"15min":1.179},"min":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.6819999999999999,"5min":4.0439999999999996,"15min":7.8650000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.143,"15min":1.2230000000000001},"min":{"1min":0.42399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3200000000000001,"5min":5.4489999999999998,"15min":5.6699999999999999},"last":1.665}]},{"osd":9,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.998,"15min":1.399},"min":{"1min":0.53300000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.9750000000000001,"5min":3.2650000000000001,"15min":34.112000000000002},"last":1.679},{"interface":"front","average":{"1min":1.079,"5min":1.3440000000000001,"15min":1.637},"min":{"1min":0.45600000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.754,"5min":5.6420000000000003,"15min":34.590000000000003},"last":1.54}]},{"osd":10,"last update":"Mon Feb 12 16:27:50 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.373,"15min":1.381},"min":{"1min":0.503,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.48,"5min":5.6970000000000001,"15min":6.0899999999999999},"last":1.758},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.177,"15min":1.2350000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.4269999999999996,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":1.29}]},{"osd":11,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0800000000000001,"15min":1.4470000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6500000000000004},"last":1.736},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.357,"15min":1.482},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":3.52,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":1.6000000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306372,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438780,"statfs":{"total":95995035648,"available":95681310720,"internally_reserved":0,"allocated":978944,"data_stored":496834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":12.419,"5min":3.194,"15min":3.4750000000000001},"min":{"1min":0.27300000000000002,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":221.96600000000001,"5min":221.96600000000001,"15min":565.34500000000003},"last":1.1579999999999999},{"interface":"front","average":{"1min":12.515000000000001,"5min":3.3079999999999998,"15min":3.5859999999999999},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":222.06200000000001,"5min":222.06200000000001,"15min":565.98400000000004},"last":1.0169999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.516,"15min":1.145},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.8710000000000004,"5min":40.021999999999998,"15min":40.021999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.631,"15min":1.22},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.5549999999999997,"5min":40.225000000000001,"15min":40.225000000000001},"last":1.3700000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":1.4570000000000001,"15min":1.351},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":16.300999999999998,"5min":16.300999999999998,"15min":16.300999999999998},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.498,"5min":0.95999999999999996,"15min":0.80600000000000005},"min":{"1min":0.33700000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.5540000000000003,"5min":6.5540000000000003,"15min":6.5540000000000003},"last":0.75700000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.4119999999999999,"15min":1.425},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.9509999999999996,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.657,"5min":1.73,"15min":1.4750000000000001},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":5.4169999999999998,"5min":42.430999999999997,"15min":74.427000000000007},"last":1.044}]},{"osd":6,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1739999999999999,"15min":1.3320000000000001},"min":{"1min":0.34200000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.3490000000000002,"5min":6.4459999999999997,"15min":74.105000000000004},"last":0.877},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.2330000000000001,"15min":1.4390000000000001},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":9.5830000000000002,"5min":9.5830000000000002,"15min":75.013999999999996},"last":1.109}]},{"osd":7,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.1379999999999999,"15min":1.0449999999999999},"min":{"1min":0.42499999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.7429999999999999,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":1.226},{"interface":"front","average":{"1min":1.462,"5min":1.325,"15min":1.1599999999999999},"min":{"1min":0.36599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.423,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":1.3999999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.071,"15min":1.173},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.2850000000000001,"5min":4.2850000000000001,"15min":5.9509999999999996},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.2130000000000001,"15min":1.2210000000000001},"min":{"1min":0.48399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.7749999999999999,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":1.496}]},{"osd":9,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4550000000000001,"15min":3.327},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":4.54,"5min":6.2939999999999996,"15min":449.89499999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.829,"5min":1.321,"15min":3.3010000000000002},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":7.5789999999999997,"5min":9.6649999999999991,"15min":448.70800000000003},"last":1.637}]},{"osd":10,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.157,"15min":1.2509999999999999},"min":{"1min":0.55800000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.242,"5min":4.242,"15min":5.9210000000000003},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.5449999999999999,"15min":1.353},"min":{"1min":0.48899999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.4619999999999997,"5min":15.006,"15min":15.006},"last":1.6659999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.2969999999999999,"15min":1.2889999999999999},"min":{"1min":0.59999999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.6940000000000008,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":1.611},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.548,"15min":1.391},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":4.5019999999999998,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":1.5329999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:28:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:56.957 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:28:56.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:28:57.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:57.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:57.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:57.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:57.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:57.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:57.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.906755+0000 osd.6 (osd.6) 116 : cluster [DBG] 1.f scrub starts 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: cluster 2024-02-12T16:28:55.989467+0000 osd.6 (osd.6) 117 : cluster [DBG] 1.f scrub ok 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: cluster 2024-02-12T16:28:56.007349+0000 mgr.y (mgr.14146) 1322 : cluster [DBG] pgmap v1843: 21 pgs: 21 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: audit 2024-02-12T16:28:56.036673+0000 mgr.y (mgr.14146) 1323 : audit [DBG] from='client.16368 v1:172.21.15.121:0/338159708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: cluster 2024-02-12T16:28:56.311067+0000 osd.5 (osd.5) 101 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: cluster 2024-02-12T16:28:56.342748+0000 osd.10 (osd.10) 122 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: cluster 2024-02-12T16:28:56.394796+0000 osd.5 (osd.5) 102 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:28:57.366 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:57 smithi169 bash[19604]: cluster 2024-02-12T16:28:56.397250+0000 osd.10 (osd.10) 123 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:28:57.384 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:57.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.906755+0000 osd.6 (osd.6) 116 : cluster [DBG] 1.f scrub starts 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: cluster 2024-02-12T16:28:55.989467+0000 osd.6 (osd.6) 117 : cluster [DBG] 1.f scrub ok 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: cluster 2024-02-12T16:28:56.007349+0000 mgr.y (mgr.14146) 1322 : cluster [DBG] pgmap v1843: 21 pgs: 21 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: audit 2024-02-12T16:28:56.036673+0000 mgr.y (mgr.14146) 1323 : audit [DBG] from='client.16368 v1:172.21.15.121:0/338159708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: cluster 2024-02-12T16:28:56.311067+0000 osd.5 (osd.5) 101 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: cluster 2024-02-12T16:28:56.342748+0000 osd.10 (osd.10) 122 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: cluster 2024-02-12T16:28:56.394796+0000 osd.5 (osd.5) 102 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:28:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:57 smithi121 bash[18489]: cluster 2024-02-12T16:28:56.397250+0000 osd.10 (osd.10) 123 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:28:57.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:57.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:57.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:57.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.906755+0000 osd.6 (osd.6) 116 : cluster [DBG] 1.f scrub starts 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: cluster 2024-02-12T16:28:55.989467+0000 osd.6 (osd.6) 117 : cluster [DBG] 1.f scrub ok 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: cluster 2024-02-12T16:28:56.007349+0000 mgr.y (mgr.14146) 1322 : cluster [DBG] pgmap v1843: 21 pgs: 21 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: audit 2024-02-12T16:28:56.036673+0000 mgr.y (mgr.14146) 1323 : audit [DBG] from='client.16368 v1:172.21.15.121:0/338159708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: cluster 2024-02-12T16:28:56.311067+0000 osd.5 (osd.5) 101 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: cluster 2024-02-12T16:28:56.342748+0000 osd.10 (osd.10) 122 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: cluster 2024-02-12T16:28:56.394796+0000 osd.5 (osd.5) 102 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:28:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:57 smithi123 bash[19119]: cluster 2024-02-12T16:28:56.397250+0000 osd.10 (osd.10) 123 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:28:57.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:57.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:57.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:57.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:57.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:57.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:57.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:58.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:58.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:58.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:58.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:58.218 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:28:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:58.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:28:58.287 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:28:58.287 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:28:58.287 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:28:58.287 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:28:58.287 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:28:58.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:58.361 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:58.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:58.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:58.503 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:28:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:58.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:28:58.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:58.642 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:28:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:58.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:28:58.745 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:58.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:58.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:58.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:58.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:28:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:58.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:28:59.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-12T16:28:59.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:59.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:59.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:59.198 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:59.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:59 smithi169 bash[19604]: cluster 2024-02-12T16:28:57.805998+0000 osd.7 (osd.7) 153 : cluster [DBG] 1.d scrub ok 2024-02-12T16:28:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:59 smithi169 bash[19604]: cluster 2024-02-12T16:28:57.930109+0000 osd.1 (osd.1) 90 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:28:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:59 smithi169 bash[19604]: cluster 2024-02-12T16:28:58.008010+0000 mgr.y (mgr.14146) 1324 : cluster [DBG] pgmap v1844: 21 pgs: 21 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:59 smithi169 bash[19604]: cluster 2024-02-12T16:28:58.060246+0000 osd.1 (osd.1) 91 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:28:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:59 smithi169 bash[19604]: cluster 2024-02-12T16:28:58.334407+0000 osd.10 (osd.10) 124 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:28:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:28:59 smithi169 bash[19604]: cluster 2024-02-12T16:28:58.383733+0000 osd.10 (osd.10) 125 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:28:59.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:59.337 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:28:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:59.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:28:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:59 smithi121 bash[18489]: cluster 2024-02-12T16:28:57.805998+0000 osd.7 (osd.7) 153 : cluster [DBG] 1.d scrub ok 2024-02-12T16:28:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:59 smithi121 bash[18489]: cluster 2024-02-12T16:28:57.930109+0000 osd.1 (osd.1) 90 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:28:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:59 smithi121 bash[18489]: cluster 2024-02-12T16:28:58.008010+0000 mgr.y (mgr.14146) 1324 : cluster [DBG] pgmap v1844: 21 pgs: 21 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:59 smithi121 bash[18489]: cluster 2024-02-12T16:28:58.060246+0000 osd.1 (osd.1) 91 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:28:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:59 smithi121 bash[18489]: cluster 2024-02-12T16:28:58.334407+0000 osd.10 (osd.10) 124 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:28:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:28:59 smithi121 bash[18489]: cluster 2024-02-12T16:28:58.383733+0000 osd.10 (osd.10) 125 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:28:59.437 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:59.438 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:28:59.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:59.454 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:59 smithi123 bash[19119]: cluster 2024-02-12T16:28:57.805998+0000 osd.7 (osd.7) 153 : cluster [DBG] 1.d scrub ok 2024-02-12T16:28:59.454 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:59 smithi123 bash[19119]: cluster 2024-02-12T16:28:57.930109+0000 osd.1 (osd.1) 90 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:28:59.454 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:59 smithi123 bash[19119]: cluster 2024-02-12T16:28:58.008010+0000 mgr.y (mgr.14146) 1324 : cluster [DBG] pgmap v1844: 21 pgs: 21 active+clean; 0 B data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:28:59.454 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:59 smithi123 bash[19119]: cluster 2024-02-12T16:28:58.060246+0000 osd.1 (osd.1) 91 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:28:59.454 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:59 smithi123 bash[19119]: cluster 2024-02-12T16:28:58.334407+0000 osd.10 (osd.10) 124 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:28:59.454 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:28:59 smithi123 bash[19119]: cluster 2024-02-12T16:28:58.383733+0000 osd.10 (osd.10) 125 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:28:59.477 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:28:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:59.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:28:59.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:59.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:28:59.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:59.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:28:59.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:28:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:59.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:28:59.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:59.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:28:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:28:59.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:28:59.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:29:00.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:00.039 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:00.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:00.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:00.175 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:00.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:00.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:00.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:00 smithi169 bash[19604]: cluster 2024-02-12T16:28:58.711442+0000 osd.7 (osd.7) 154 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:29:00.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:00 smithi169 bash[19604]: cluster 2024-02-12T16:28:58.797567+0000 osd.7 (osd.7) 155 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:29:00.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:00 smithi169 bash[19604]: cluster 2024-02-12T16:28:59.331808+0000 osd.5 (osd.5) 103 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:29:00.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:00 smithi169 bash[19604]: cluster 2024-02-12T16:28:59.370398+0000 osd.5 (osd.5) 104 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:29:00.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:00.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:00 smithi121 bash[18489]: cluster 2024-02-12T16:28:58.711442+0000 osd.7 (osd.7) 154 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:29:00.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:00 smithi121 bash[18489]: cluster 2024-02-12T16:28:58.797567+0000 osd.7 (osd.7) 155 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:29:00.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:00 smithi121 bash[18489]: cluster 2024-02-12T16:28:59.331808+0000 osd.5 (osd.5) 103 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:29:00.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:00 smithi121 bash[18489]: cluster 2024-02-12T16:28:59.370398+0000 osd.5 (osd.5) 104 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:29:00.415 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:00.427 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:00 smithi123 bash[19119]: cluster 2024-02-12T16:28:58.711442+0000 osd.7 (osd.7) 154 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:29:00.428 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:00 smithi123 bash[19119]: cluster 2024-02-12T16:28:58.797567+0000 osd.7 (osd.7) 155 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:29:00.428 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:00 smithi123 bash[19119]: cluster 2024-02-12T16:28:59.331808+0000 osd.5 (osd.5) 103 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:29:00.428 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:00 smithi123 bash[19119]: cluster 2024-02-12T16:28:59.370398+0000 osd.5 (osd.5) 104 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:29:00.451 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:00.453 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:00.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:00.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:00.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:00.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:00.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:00.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:00.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:00.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:00.876 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:00.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:00.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:01.016 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:01.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:01.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:01.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:01.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:01.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:01.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:01.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:01 smithi123 bash[19119]: cluster 2024-02-12T16:28:59.645317+0000 osd.11 (osd.11) 142 : cluster [DBG] 1.b scrub starts 2024-02-12T16:29:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:01 smithi123 bash[19119]: cluster 2024-02-12T16:28:59.763856+0000 osd.11 (osd.11) 143 : cluster [DBG] 1.b scrub ok 2024-02-12T16:29:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:01 smithi123 bash[19119]: cluster 2024-02-12T16:29:00.008481+0000 mgr.y (mgr.14146) 1325 : cluster [DBG] pgmap v1845: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:01 smithi123 bash[19119]: cluster 2024-02-12T16:29:00.192866+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:01 smithi123 bash[19119]: cluster 2024-02-12T16:29:00.192936+0000 osd.2 (osd.2) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:01.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:01 smithi121 bash[18489]: cluster 2024-02-12T16:28:59.645317+0000 osd.11 (osd.11) 142 : cluster [DBG] 1.b scrub starts 2024-02-12T16:29:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:01 smithi121 bash[18489]: cluster 2024-02-12T16:28:59.763856+0000 osd.11 (osd.11) 143 : cluster [DBG] 1.b scrub ok 2024-02-12T16:29:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:01 smithi121 bash[18489]: cluster 2024-02-12T16:29:00.008481+0000 mgr.y (mgr.14146) 1325 : cluster [DBG] pgmap v1845: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:01 smithi121 bash[18489]: cluster 2024-02-12T16:29:00.192866+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:01 smithi121 bash[18489]: cluster 2024-02-12T16:29:00.192936+0000 osd.2 (osd.2) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:01.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:01.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:01.536 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:01 smithi169 bash[19604]: cluster 2024-02-12T16:28:59.645317+0000 osd.11 (osd.11) 142 : cluster [DBG] 1.b scrub starts 2024-02-12T16:29:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:01 smithi169 bash[19604]: cluster 2024-02-12T16:28:59.763856+0000 osd.11 (osd.11) 143 : cluster [DBG] 1.b scrub ok 2024-02-12T16:29:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:01 smithi169 bash[19604]: cluster 2024-02-12T16:29:00.008481+0000 mgr.y (mgr.14146) 1325 : cluster [DBG] pgmap v1845: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:01 smithi169 bash[19604]: cluster 2024-02-12T16:29:00.192866+0000 osd.2 (osd.2) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:01 smithi169 bash[19604]: cluster 2024-02-12T16:29:00.192936+0000 osd.2 (osd.2) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:01.574 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:01.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:01.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:01.717 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:01.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:01.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:01.854 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:01.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:01.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:01.997 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:02.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:02.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:02.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:02.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:02 smithi169 bash[19604]: cluster 2024-02-12T16:29:01.172498+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e635: 12 total, 12 up, 12 in 2024-02-12T16:29:02.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:02 smithi121 bash[18489]: cluster 2024-02-12T16:29:01.172498+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e635: 12 total, 12 up, 12 in 2024-02-12T16:29:02.410 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:02.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:02.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:02.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:02 smithi123 bash[19119]: cluster 2024-02-12T16:29:01.172498+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e635: 12 total, 12 up, 12 in 2024-02-12T16:29:02.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:02.684 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:02.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:02.823 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:02.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:02.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:02.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:03.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:03.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:03.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:03.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:03.238 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:03.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:03.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:03.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:03.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:03.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:03 smithi121 bash[18489]: cluster 2024-02-12T16:29:02.009740+0000 mgr.y (mgr.14146) 1326 : cluster [DBG] pgmap v1847: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:03.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:03.497 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:03 smithi169 bash[19604]: cluster 2024-02-12T16:29:02.009740+0000 mgr.y (mgr.14146) 1326 : cluster [DBG] pgmap v1847: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:03.522 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:03.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:03.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:03.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:03 smithi123 bash[19119]: cluster 2024-02-12T16:29:02.009740+0000 mgr.y (mgr.14146) 1326 : cluster [DBG] pgmap v1847: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:03.662 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:03.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:03.767 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:03.789 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:29:03.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:03.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:03.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:03.943 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:03.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:04.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:04.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:04.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:04.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:04.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:04.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:04.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:04.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:04.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:04.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:04.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:04.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:04.520 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:04.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:04.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:04.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:04.790 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:04.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:04.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:04.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:04.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:05.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:05.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:05.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:05.215 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:05.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:05.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:05.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:05.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:05.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:05 smithi121 bash[18489]: cluster 2024-02-12T16:29:04.010433+0000 mgr.y (mgr.14146) 1327 : cluster [DBG] pgmap v1848: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:05.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:05.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:05 smithi169 bash[19604]: cluster 2024-02-12T16:29:04.010433+0000 mgr.y (mgr.14146) 1327 : cluster [DBG] pgmap v1848: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:05.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:05.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:05.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:05.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:05 smithi123 bash[19119]: cluster 2024-02-12T16:29:04.010433+0000 mgr.y (mgr.14146) 1327 : cluster [DBG] pgmap v1848: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:05.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:05.772 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:05.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:05.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:05.914 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:05.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:06.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:06.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:06.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:06.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:06.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:06.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:06.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:06.395 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:06.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:06.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:06.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:06.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:06.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:06.642 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:29:06.662 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:29:06.664 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:06.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:06.687 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:29:06.707 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:29:06.731 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:29:06.763 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:29:06.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:06.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:06.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:06.810 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:29:06.839 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:06 smithi121 bash[18489]: cluster 2024-02-12T16:29:06.011253+0000 mgr.y (mgr.14146) 1328 : cluster [DBG] pgmap v1849: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:06.840 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:06 smithi121 bash[18489]: audit 2024-02-12T16:29:06.617735+0000 mon.a (mon.0) 2235 : audit [DBG] from='client.? v1:172.21.15.121:0/1086692720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:06.858 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:29:06.885 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:29:06.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:06.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:06 smithi169 bash[19604]: cluster 2024-02-12T16:29:06.011253+0000 mgr.y (mgr.14146) 1328 : cluster [DBG] pgmap v1849: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:06.909 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:06 smithi169 bash[19604]: audit 2024-02-12T16:29:06.617735+0000 mon.a (mon.0) 2235 : audit [DBG] from='client.? v1:172.21.15.121:0/1086692720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:06.917 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:29:06.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:06.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:06.940 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:29:06.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.970 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:29:06.971 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:29:07.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:07.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:06 smithi123 bash[19119]: cluster 2024-02-12T16:29:06.011253+0000 mgr.y (mgr.14146) 1328 : cluster [DBG] pgmap v1849: 21 pgs: 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:07.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:06 smithi123 bash[19119]: audit 2024-02-12T16:29:06.617735+0000 mon.a (mon.0) 2235 : audit [DBG] from='client.? v1:172.21.15.121:0/1086692720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:07.071 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:07.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:07.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:07.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:07.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:07.212 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:29:07.212 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:29:07.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:07.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:07.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:07.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:07.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:07.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:07.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:07.628 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:07.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:07.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:07.733 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_4' created 2024-02-12T16:29:07.734 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:29:07.758 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:07.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:07.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:07.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:07.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:07.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:07 smithi169 bash[19604]: audit 2024-02-12T16:29:06.734990+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.121:0/2630184726' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-12T16:29:07.919 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:07 smithi169 bash[19604]: audit 2024-02-12T16:29:07.204586+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? v1:172.21.15.121:0/15842271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:29:07.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:29:08.000 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:08.013 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:07 smithi123 bash[19119]: audit 2024-02-12T16:29:06.734990+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.121:0/2630184726' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-12T16:29:08.013 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:07 smithi123 bash[19119]: audit 2024-02-12T16:29:07.204586+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? v1:172.21.15.121:0/15842271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:29:08.038 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:08.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:08.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:08.149 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:07 smithi121 bash[18489]: audit 2024-02-12T16:29:06.734990+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.121:0/2630184726' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-12T16:29:08.149 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:07 smithi121 bash[18489]: audit 2024-02-12T16:29:07.204586+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? v1:172.21.15.121:0/15842271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:29:08.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:08.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:08.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:08.308 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:08.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:08.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:08.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:08.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:08.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:08.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:08.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:08.693 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:08.730 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:08.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:08.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:08.869 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:08.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:08.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:08.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:08 smithi169 bash[19604]: cluster 2024-02-12T16:29:07.705520+0000 mon.a (mon.0) 2238 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:29:08.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:08 smithi169 bash[19604]: cluster 2024-02-12T16:29:07.705593+0000 mon.a (mon.0) 2239 : cluster [INF] Cluster is now healthy 2024-02-12T16:29:08.986 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:08 smithi169 bash[19604]: audit 2024-02-12T16:29:07.720925+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? v1:172.21.15.121:0/2630184726' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-12T16:29:08.987 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:08 smithi169 bash[19604]: audit 2024-02-12T16:29:07.721033+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? v1:172.21.15.121:0/15842271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:29:08.987 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:08 smithi169 bash[19604]: cluster 2024-02-12T16:29:07.721112+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e636: 12 total, 12 up, 12 in 2024-02-12T16:29:08.987 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:08 smithi169 bash[19604]: cluster 2024-02-12T16:29:08.011894+0000 mgr.y (mgr.14146) 1329 : cluster [DBG] pgmap v1851: 37 pgs: 16 unknown, 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:09.010 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:09.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:09.064 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:08 smithi121 bash[18489]: cluster 2024-02-12T16:29:07.705520+0000 mon.a (mon.0) 2238 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:29:09.064 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:08 smithi121 bash[18489]: cluster 2024-02-12T16:29:07.705593+0000 mon.a (mon.0) 2239 : cluster [INF] Cluster is now healthy 2024-02-12T16:29:09.064 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:08 smithi121 bash[18489]: audit 2024-02-12T16:29:07.720925+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? v1:172.21.15.121:0/2630184726' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-12T16:29:09.064 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:08 smithi121 bash[18489]: audit 2024-02-12T16:29:07.721033+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? v1:172.21.15.121:0/15842271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:29:09.064 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:08 smithi121 bash[18489]: cluster 2024-02-12T16:29:07.721112+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e636: 12 total, 12 up, 12 in 2024-02-12T16:29:09.064 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:08 smithi121 bash[18489]: cluster 2024-02-12T16:29:08.011894+0000 mgr.y (mgr.14146) 1329 : cluster [DBG] pgmap v1851: 37 pgs: 16 unknown, 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:09.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:09.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:08 smithi123 bash[19119]: cluster 2024-02-12T16:29:07.705520+0000 mon.a (mon.0) 2238 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:29:09.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:08 smithi123 bash[19119]: cluster 2024-02-12T16:29:07.705593+0000 mon.a (mon.0) 2239 : cluster [INF] Cluster is now healthy 2024-02-12T16:29:09.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:08 smithi123 bash[19119]: audit 2024-02-12T16:29:07.720925+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? v1:172.21.15.121:0/2630184726' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-12T16:29:09.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:08 smithi123 bash[19119]: audit 2024-02-12T16:29:07.721033+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? v1:172.21.15.121:0/15842271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:29:09.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:08 smithi123 bash[19119]: cluster 2024-02-12T16:29:07.721112+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e636: 12 total, 12 up, 12 in 2024-02-12T16:29:09.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:08 smithi123 bash[19119]: cluster 2024-02-12T16:29:08.011894+0000 mgr.y (mgr.14146) 1329 : cluster [DBG] pgmap v1851: 37 pgs: 16 unknown, 21 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:09.149 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:09.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:09.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:09.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:09.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:09.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:09.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:09.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:09.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:09.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:09.687 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:09.725 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:09.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:09.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:09.842 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:09 smithi123 bash[19119]: cluster 2024-02-12T16:29:08.727662+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e637: 12 total, 12 up, 12 in 2024-02-12T16:29:09.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:09.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:09.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:09.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-02-12T16:29:09.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:09 smithi121 bash[18489]: cluster 2024-02-12T16:29:08.727662+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e637: 12 total, 12 up, 12 in 2024-02-12T16:29:10.006 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:10.008 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:09 smithi169 bash[19604]: cluster 2024-02-12T16:29:08.727662+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e637: 12 total, 12 up, 12 in 2024-02-12T16:29:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:10.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:10.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:10.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:10.284 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:10.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:10.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:10.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:10.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:10.619 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:29:10.621 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:29:10.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:10.702 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:10 smithi169 bash[19604]: cluster 2024-02-12T16:29:09.729080+0000 osd.11 (osd.11) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:10 smithi169 bash[19604]: cluster 2024-02-12T16:29:09.729184+0000 osd.11 (osd.11) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:10 smithi169 bash[19604]: cluster 2024-02-12T16:29:09.730349+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in 2024-02-12T16:29:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:10 smithi169 bash[19604]: cluster 2024-02-12T16:29:10.012601+0000 mgr.y (mgr.14146) 1330 : cluster [DBG] pgmap v1854: 37 pgs: 3 creating+peering, 10 unknown, 24 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:10 smithi169 bash[19604]: cluster 2024-02-12T16:29:10.694717+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e639: 12 total, 12 up, 12 in 2024-02-12T16:29:10.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:10.848 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:10.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:10.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:10 smithi121 bash[18489]: cluster 2024-02-12T16:29:09.729080+0000 osd.11 (osd.11) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:10.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:10 smithi121 bash[18489]: cluster 2024-02-12T16:29:09.729184+0000 osd.11 (osd.11) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:10.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:10 smithi121 bash[18489]: cluster 2024-02-12T16:29:09.730349+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in 2024-02-12T16:29:10.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:10 smithi121 bash[18489]: cluster 2024-02-12T16:29:10.012601+0000 mgr.y (mgr.14146) 1330 : cluster [DBG] pgmap v1854: 37 pgs: 3 creating+peering, 10 unknown, 24 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:10.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:10 smithi121 bash[18489]: cluster 2024-02-12T16:29:10.694717+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e639: 12 total, 12 up, 12 in 2024-02-12T16:29:10.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:10.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:11.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:29:11.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:10 smithi123 bash[19119]: cluster 2024-02-12T16:29:09.729080+0000 osd.11 (osd.11) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:10 smithi123 bash[19119]: cluster 2024-02-12T16:29:09.729184+0000 osd.11 (osd.11) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:10 smithi123 bash[19119]: cluster 2024-02-12T16:29:09.730349+0000 mon.a (mon.0) 2244 : cluster [DBG] osdmap e638: 12 total, 12 up, 12 in 2024-02-12T16:29:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:10 smithi123 bash[19119]: cluster 2024-02-12T16:29:10.012601+0000 mgr.y (mgr.14146) 1330 : cluster [DBG] pgmap v1854: 37 pgs: 3 creating+peering, 10 unknown, 24 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:10 smithi123 bash[19119]: cluster 2024-02-12T16:29:10.694717+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e639: 12 total, 12 up, 12 in 2024-02-12T16:29:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:11.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:11.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:11.286 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:11.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:11.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:11.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:11.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:11.472 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1854,"stamp":"2024-02-12T16:29:10.012206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5120008,"kb_used_data":9432,"kb_used_omap":91,"kb_used_meta":5108708,"kb_avail":1119821816,"statfs":{"total":1151940427776,"available":1146697539584,"internally_reserved":0,"allocated":9658368,"data_stored":4826666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":5231317008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-338776,"num_write_kb":-10840769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3147,"num_bytes_recovered":-206241792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.117313"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":127,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.986605+0000","last_change":"2024-02-12T16:28:06.991266+0000","last_active":"2024-02-12T16:28:55.986605+0000","last_peered":"2024-02-12T16:28:55.986605+0000","last_clean":"2024-02-12T16:28:55.986605+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:28:55.986605+0000","last_undegraded":"2024-02-12T16:28:55.986605+0000","last_fullsized":"2024-02-12T16:28:55.986605+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:06.991147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:28:06.991147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":244,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.988478+0000","last_change":"2024-02-12T16:28:53.018782+0000","last_active":"2024-02-12T16:28:55.988478+0000","last_peered":"2024-02-12T16:28:55.988478+0000","last_clean":"2024-02-12T16:28:55.988478+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:28:55.988478+0000","last_undegraded":"2024-02-12T16:28:55.988478+0000","last_fullsized":"2024-02-12T16:28:55.988478+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.018681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.018681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":8,"reported_epoch":637,"state":"active+clean","last_fresh":"2024-02-12T16:29:08.734672+0000","last_change":"2024-02-12T16:29:08.734672+0000","last_active":"2024-02-12T16:29:08.734672+0000","last_peered":"2024-02-12T16:29:08.734672+0000","last_clean":"2024-02-12T16:29:08.734672+0000","last_became_active":"2024-02-12T16:29:08.734223+0000","last_became_peered":"2024-02-12T16:29:08.734223+0000","last_unstale":"2024-02-12T16:29:08.734672+0000","last_undegraded":"2024-02-12T16:29:08.734672+0000","last_fullsized":"2024-02-12T16:29:08.734672+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":322,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-12T16:29:01.589700+0000","last_change":"2024-02-12T16:28:54.310544+0000","last_active":"2024-02-12T16:29:01.589700+0000","last_peered":"2024-02-12T16:29:01.589700+0000","last_clean":"2024-02-12T16:29:01.589700+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:29:01.589700+0000","last_undegraded":"2024-02-12T16:29:01.589700+0000","last_fullsized":"2024-02-12T16:29:01.589700+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.310448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.310448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.730163+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":727,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-12T16:29:07.789153+0000","last_change":"2024-02-12T16:28:50.494186+0000","last_active":"2024-02-12T16:29:07.789153+0000","last_peered":"2024-02-12T16:29:07.789153+0000","last_clean":"2024-02-12T16:29:07.789153+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:29:07.789153+0000","last_undegraded":"2024-02-12T16:29:07.789153+0000","last_fullsized":"2024-02-12T16:29:07.789153+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:50.494005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:50.494005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":365,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.974034+0000","last_change":"2024-02-12T16:28:53.942753+0000","last_active":"2024-02-12T16:28:55.974034+0000","last_peered":"2024-02-12T16:28:55.974034+0000","last_clean":"2024-02-12T16:28:55.974034+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:28:55.974034+0000","last_undegraded":"2024-02-12T16:28:55.974034+0000","last_fullsized":"2024-02-12T16:28:55.974034+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.942656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.942656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":8,"reported_epoch":637,"state":"active+clean","last_fresh":"2024-02-12T16:29:08.809293+0000","last_change":"2024-02-12T16:29:08.809293+0000","last_active":"2024-02-12T16:29:08.809293+0000","last_peered":"2024-02-12T16:29:08.809293+0000","last_clean":"2024-02-12T16:29:08.809293+0000","last_became_active":"2024-02-12T16:29:08.808944+0000","last_became_peered":"2024-02-12T16:29:08.808944+0000","last_unstale":"2024-02-12T16:29:08.809293+0000","last_undegraded":"2024-02-12T16:29:08.809293+0000","last_fullsized":"2024-02-12T16:29:08.809293+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":313,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-12T16:29:01.589789+0000","last_change":"2024-02-12T16:28:56.397511+0000","last_active":"2024-02-12T16:29:01.589789+0000","last_peered":"2024-02-12T16:29:01.589789+0000","last_clean":"2024-02-12T16:29:01.589789+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:29:01.589789+0000","last_undegraded":"2024-02-12T16:29:01.589789+0000","last_fullsized":"2024-02-12T16:29:01.589789+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:56.397266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:56.397266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":323,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-12T16:29:01.589700+0000","last_change":"2024-02-12T16:28:58.383870+0000","last_active":"2024-02-12T16:29:01.589700+0000","last_peered":"2024-02-12T16:29:01.589700+0000","last_clean":"2024-02-12T16:29:01.589700+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:29:01.589700+0000","last_undegraded":"2024-02-12T16:29:01.589700+0000","last_fullsized":"2024-02-12T16:29:01.589700+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.383742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.383742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":182,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.986276+0000","last_change":"2024-02-12T16:28:53.712516+0000","last_active":"2024-02-12T16:28:55.986276+0000","last_peered":"2024-02-12T16:28:55.986276+0000","last_clean":"2024-02-12T16:28:55.986276+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:28:55.986276+0000","last_undegraded":"2024-02-12T16:28:55.986276+0000","last_fullsized":"2024-02-12T16:28:55.986276+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.712421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.712421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.730242+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":754,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-12T16:29:01.793430+0000","last_change":"2024-02-12T16:28:53.646270+0000","last_active":"2024-02-12T16:29:01.793430+0000","last_peered":"2024-02-12T16:29:01.793430+0000","last_clean":"2024-02-12T16:29:01.793430+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:29:01.793430+0000","last_undegraded":"2024-02-12T16:29:01.793430+0000","last_fullsized":"2024-02-12T16:29:01.793430+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.646027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.646027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":129,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-12T16:29:01.793464+0000","last_change":"2024-02-12T16:28:57.806094+0000","last_active":"2024-02-12T16:29:01.793464+0000","last_peered":"2024-02-12T16:29:01.793464+0000","last_clean":"2024-02-12T16:29:01.793464+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:29:01.793464+0000","last_undegraded":"2024-02-12T16:29:01.793464+0000","last_fullsized":"2024-02-12T16:29:01.793464+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:57.806002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:57.806002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":154,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.986352+0000","last_change":"2024-02-12T16:28:51.703479+0000","last_active":"2024-02-12T16:28:55.986352+0000","last_peered":"2024-02-12T16:28:55.986352+0000","last_clean":"2024-02-12T16:28:55.986352+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:28:55.986352+0000","last_undegraded":"2024-02-12T16:28:55.986352+0000","last_fullsized":"2024-02-12T16:28:55.986352+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:51.703298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:51.703298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.730242+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[8],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":326,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.989658+0000","last_change":"2024-02-12T16:28:55.989602+0000","last_active":"2024-02-12T16:28:55.989658+0000","last_peered":"2024-02-12T16:28:55.989658+0000","last_clean":"2024-02-12T16:28:55.989658+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:28:55.989658+0000","last_undegraded":"2024-02-12T16:28:55.989658+0000","last_fullsized":"2024-02-12T16:28:55.989658+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:55.989476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:28:55.989476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":131,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-12T16:29:07.731208+0000","last_change":"2024-02-12T16:28:54.613666+0000","last_active":"2024-02-12T16:29:07.731208+0000","last_peered":"2024-02-12T16:29:07.731208+0000","last_clean":"2024-02-12T16:29:07.731208+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:29:07.731208+0000","last_undegraded":"2024-02-12T16:29:07.731208+0000","last_fullsized":"2024-02-12T16:29:07.731208+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.613501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.613501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":8,"reported_epoch":637,"state":"active+clean","last_fresh":"2024-02-12T16:29:08.736225+0000","last_change":"2024-02-12T16:29:08.736225+0000","last_active":"2024-02-12T16:29:08.736225+0000","last_peered":"2024-02-12T16:29:08.736225+0000","last_clean":"2024-02-12T16:29:08.736225+0000","last_became_active":"2024-02-12T16:29:08.735885+0000","last_became_peered":"2024-02-12T16:29:08.735885+0000","last_unstale":"2024-02-12T16:29:08.736225+0000","last_undegraded":"2024-02-12T16:29:08.736225+0000","last_fullsized":"2024-02-12T16:29:08.736225+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":729,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-12T16:29:01.793432+0000","last_change":"2024-02-12T16:28:58.797711+0000","last_active":"2024-02-12T16:29:01.793432+0000","last_peered":"2024-02-12T16:29:01.793432+0000","last_clean":"2024-02-12T16:29:01.793432+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:29:01.793432+0000","last_undegraded":"2024-02-12T16:29:01.793432+0000","last_fullsized":"2024-02-12T16:29:01.793432+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.797578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.797578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":315,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.906842+0000","last_change":"2024-02-12T16:28:53.361716+0000","last_active":"2024-02-12T16:28:55.906842+0000","last_peered":"2024-02-12T16:28:55.906842+0000","last_clean":"2024-02-12T16:28:55.906842+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:28:55.906842+0000","last_undegraded":"2024-02-12T16:28:55.906842+0000","last_fullsized":"2024-02-12T16:28:55.906842+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.361610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.361610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":69,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-12T16:29:07.789167+0000","last_change":"2024-02-12T16:28:59.764005+0000","last_active":"2024-02-12T16:29:07.789167+0000","last_peered":"2024-02-12T16:29:07.789167+0000","last_clean":"2024-02-12T16:29:07.789167+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:29:07.789167+0000","last_undegraded":"2024-02-12T16:29:07.789167+0000","last_fullsized":"2024-02-12T16:29:07.789167+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:59.763866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:59.763866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-12T16:29:07.704868+0000","last_change":"2024-02-12T16:29:07.704868+0000","last_active":"2024-02-12T16:29:07.704868+0000","last_peered":"2024-02-12T16:29:07.704868+0000","last_clean":"2024-02-12T16:29:07.704868+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-12T16:29:07.704868+0000","last_undegraded":"2024-02-12T16:29:07.704868+0000","last_fullsized":"2024-02-12T16:29:07.704868+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":303,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:55.988444+0000","last_change":"2024-02-12T16:28:54.979306+0000","last_active":"2024-02-12T16:28:55.988444+0000","last_peered":"2024-02-12T16:28:55.988444+0000","last_clean":"2024-02-12T16:28:55.988444+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:28:55.988444+0000","last_undegraded":"2024-02-12T16:28:55.988444+0000","last_fullsized":"2024-02-12T16:28:55.988444+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.979211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.979211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":124,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:59.370541+0000","last_change":"2024-02-12T16:28:59.370513+0000","last_active":"2024-02-12T16:28:59.370541+0000","last_peered":"2024-02-12T16:28:59.370541+0000","last_clean":"2024-02-12T16:28:59.370541+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:28:59.370541+0000","last_undegraded":"2024-02-12T16:28:59.370541+0000","last_fullsized":"2024-02-12T16:28:59.370541+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:59.370403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:28:59.370403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":123,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:56.127859+0000","last_change":"2024-02-12T16:28:55.212658+0000","last_active":"2024-02-12T16:28:56.127859+0000","last_peered":"2024-02-12T16:28:56.127859+0000","last_clean":"2024-02-12T16:28:56.127859+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:28:56.127859+0000","last_undegraded":"2024-02-12T16:28:56.127859+0000","last_fullsized":"2024-02-12T16:28:56.127859+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:55.212566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:55.212566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":138,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:56.394930+0000","last_change":"2024-02-12T16:28:56.394900+0000","last_active":"2024-02-12T16:28:56.394930+0000","last_peered":"2024-02-12T16:28:56.394930+0000","last_clean":"2024-02-12T16:28:56.394930+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:28:56.394930+0000","last_undegraded":"2024-02-12T16:28:56.394930+0000","last_fullsized":"2024-02-12T16:28:56.394930+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:56.394801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:28:56.394801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":117,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-02-12T16:28:58.060369+0000","last_change":"2024-02-12T16:28:58.060344+0000","last_active":"2024-02-12T16:28:58.060369+0000","last_peered":"2024-02-12T16:28:58.060369+0000","last_clean":"2024-02-12T16:28:58.060369+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:28:58.060369+0000","last_undegraded":"2024-02-12T16:28:58.060369+0000","last_fullsized":"2024-02-12T16:28:58.060369+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.060250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.060250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":3},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586412,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":93158740,"statfs":{"total":95995035648,"available":95394549760,"internally_reserved":0,"allocated":823296,"data_stored":411802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.77000000000000002,"15min":0.84199999999999997},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":1.3420000000000001,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.391,"5min":0.92700000000000005,"15min":0.90600000000000003},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.2279999999999998,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.94599999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":0.91200000000000003,"15min":1.056},"min":{"1min":0.39300000000000002,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":5.5430000000000001,"5min":5.5430000000000001,"15min":7.0110000000000001},"last":1.05},{"interface":"front","average":{"1min":1.089,"5min":0.89600000000000002,"15min":1.0609999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":3.6859999999999999,"5min":3.6859999999999999,"15min":5.6920000000000002},"last":0.70799999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":0.92600000000000005,"15min":1.274},"min":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.1639999999999997,"5min":5.343,"15min":5.343},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.94399999999999995,"15min":1.3420000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.617,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":1.1739999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":0.86599999999999999,"15min":1.0349999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.29499999999999998},"max":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":6.5970000000000004},"last":1.079},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.84799999999999998,"15min":0.93100000000000005},"min":{"1min":0.434,"5min":0.36799999999999999,"15min":0.34499999999999997},"max":{"1min":1.165,"5min":4.101,"15min":4.915},"last":1.1080000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.101,"15min":1.1419999999999999},"min":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.35099999999999998},"max":{"1min":3.7719999999999998,"5min":5.04,"15min":5.8789999999999996},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.98299999999999998,"15min":1.127},"min":{"1min":0.47799999999999998,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":4.0099999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.262}]},{"osd":6,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.875,"15min":0.99299999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.373},"max":{"1min":3.0819999999999999,"5min":4.7370000000000001,"15min":5.141},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.014,"15min":1.1279999999999999},"min":{"1min":0.46400000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":4.1630000000000003,"5min":4.7450000000000001,"15min":8.5709999999999997},"last":0.99399999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.93100000000000005,"15min":1.1910000000000001},"min":{"1min":0.49399999999999999,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":3.9089999999999998,"5min":3.9089999999999998,"15min":6.4370000000000003},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.82499999999999996,"15min":1.1200000000000001},"min":{"1min":0.498,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.5469999999999999,"5min":4.2350000000000003,"15min":5.0430000000000001},"last":1.218}]},{"osd":8,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":6.1779999999999999,"15min":2.8039999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.0329999999999999,"5min":537.54399999999998,"15min":537.54399999999998},"last":1.135},{"interface":"front","average":{"1min":1.0009999999999999,"5min":6.3239999999999998,"15min":2.98},"min":{"1min":0.51500000000000001,"5min":0.47399999999999998,"15min":0.34699999999999998},"max":{"1min":3.8719999999999999,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.62}]},{"osd":9,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1,"15min":1.2749999999999999},"min":{"1min":0.56899999999999995,"5min":0.46400000000000002,"15min":0.40400000000000003},"max":{"1min":3.8100000000000001,"5min":3.8100000000000001,"15min":10.743},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.05,"5min":0.998,"15min":1.363},"min":{"1min":0.55300000000000005,"5min":0.48799999999999999,"15min":0.39500000000000002},"max":{"1min":3.8399999999999999,"5min":3.8399999999999999,"15min":10.698},"last":1.153}]},{"osd":10,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":6.3019999999999996,"15min":2.96},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.433},"max":{"1min":3.1880000000000002,"5min":536.245,"15min":536.245},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":6.3170000000000002,"15min":2.9460000000000002},"min":{"1min":0.53200000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.6899999999999999,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.81599999999999995}]}]},{"osd":10,"up_from":83,"seq":356482285949,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":811008,"data_stored":405523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":2.778,"15min":1.55},"min":{"1min":0.33400000000000002,"5min":0.245,"15min":0.245},"max":{"1min":15.010999999999999,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.8109999999999999,"5min":2.8700000000000001,"15min":1.6339999999999999},"min":{"1min":0.34000000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":14.129,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.96099999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0640000000000001,"15min":1.0069999999999999},"min":{"1min":0.40699999999999997,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":2.5070000000000001,"5min":5.9000000000000004,"15min":19.866},"last":0.875},{"interface":"front","average":{"1min":1.262,"5min":1.05,"15min":1.1479999999999999},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":4.673,"5min":6.1970000000000001,"15min":20.277999999999999},"last":1.1279999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2230000000000001,"15min":1.0169999999999999},"min":{"1min":0.34999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8580000000000001,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":1.181},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.1359999999999999,"15min":1.0249999999999999},"min":{"1min":0.38900000000000001,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":2.7549999999999999,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.76200000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":0.92100000000000004,"15min":3.3599999999999999},"min":{"1min":0.41699999999999998,"5min":0.40400000000000003,"15min":0.32300000000000001},"max":{"1min":3.5489999999999999,"5min":5.8109999999999999,"15min":628.77099999999996},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.999,"15min":3.3300000000000001},"min":{"1min":0.47499999999999998,"5min":0.33500000000000002,"15min":0.30199999999999999},"max":{"1min":2.681,"5min":7.3070000000000004,"15min":624.19299999999998},"last":1.075}]},{"osd":5,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.1459999999999999,"15min":1.0800000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.8109999999999999,"5min":6.343,"15min":6.343},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.151,"15min":1.169},"min":{"1min":0.55300000000000005,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":4.7610000000000001,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":1.4079999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.036,"15min":3.391},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.8799999999999999,"5min":6.3010000000000002,"15min":626.26599999999996},"last":1.228},{"interface":"front","average":{"1min":1.24,"5min":1.056,"15min":3.492},"min":{"1min":0.49199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":2.7730000000000001,"5min":6.8200000000000003,"15min":625.90700000000004},"last":1.2050000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":0.99199999999999999,"15min":1.091},"min":{"1min":0.432,"5min":0.375,"15min":0.375},"max":{"1min":2.5640000000000001,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.202,"15min":1.2130000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.3519999999999999,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":1.153}]},{"osd":8,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.0649999999999999,"15min":1.157},"min":{"1min":0.51900000000000002,"5min":0.436,"15min":0.372},"max":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.038,"15min":1.151},"min":{"1min":0.46300000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":2.6200000000000001,"5min":7.282,"15min":7.282},"last":1.4299999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.218,"15min":1.2689999999999999},"min":{"1min":0.64900000000000002,"5min":0.48499999999999999,"15min":0.372},"max":{"1min":6.6239999999999997,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":1.3},{"interface":"front","average":{"1min":1.28,"5min":1.046,"15min":1.234},"min":{"1min":0.53600000000000003,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":2.593,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":1.7050000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.115,"15min":1.288},"min":{"1min":0.40699999999999997,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":3.2280000000000002,"5min":7.3689999999999998,"15min":10.218999999999999},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.087,"15min":1.2889999999999999},"min":{"1min":0.63400000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":6.5860000000000003,"5min":6.5860000000000003,"15min":7.3250000000000002},"last":1.464}]}]},{"osd":9,"up_from":74,"seq":317827580293,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306436,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438716,"statfs":{"total":95995035648,"available":95681245184,"internally_reserved":0,"allocated":798720,"data_stored":398964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.986,"5min":1.1930000000000001,"15min":1.1020000000000001},"min":{"1min":0.30599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.9560000000000004,"5min":7.9560000000000004,"15min":7.9560000000000004},"last":0.76300000000000001},{"interface":"front","average":{"1min":2.419,"5min":1.2370000000000001,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":1.296}]},{"osd":1,"last update":"Mon Feb 12 16:28:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.1499999999999999,"15min":0.995},"min":{"1min":0.33300000000000002,"5min":0.307,"15min":0.253},"max":{"1min":5.7750000000000004,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.0780000000000001,"15min":1.0669999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":5.7910000000000004,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":1.133}]},{"osd":2,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2829999999999999,"15min":1.4770000000000001},"min":{"1min":0.436,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":3.8959999999999999,"5min":12.441000000000001,"15min":97.275000000000006},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.198,"15min":1.363},"min":{"1min":0.42999999999999999,"5min":0.314,"15min":0.312},"max":{"1min":4.0039999999999996,"5min":6.1779999999999999,"15min":97.417000000000002},"last":0.82699999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.056,"15min":1.016},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":6.3090000000000002,"5min":6.3090000000000002,"15min":6.3090000000000002},"last":1.415},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.153,"15min":1.194},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":6.3490000000000002,"5min":6.3490000000000002,"15min":6.3490000000000002},"last":1.51}]},{"osd":5,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.325,"15min":1.175},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":5.3410000000000002,"5min":6.726,"15min":6.726},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.228,"15min":1.173},"min":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.3520000000000003,"5min":6.7350000000000003,"15min":7.3739999999999997},"last":1.484}]},{"osd":6,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.997,"15min":1.101},"min":{"1min":0.38400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":2.4020000000000001,"5min":4.6749999999999998,"15min":7.3360000000000003},"last":1.004},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.2270000000000001,"15min":1.125},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.36499999999999999},"max":{"1min":14.718,"5min":14.718,"15min":14.718},"last":1.6899999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.171,"15min":1.2669999999999999},"min":{"1min":0.45400000000000001,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":3.9830000000000001,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1910000000000001,"15min":1.23},"min":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.4390000000000001,"5min":4.7000000000000002,"15min":7.9160000000000004},"last":1.2250000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.99399999999999999,"15min":1.0860000000000001},"min":{"1min":0.46700000000000003,"5min":0.36399999999999999,"15min":0.35099999999999998},"max":{"1min":2.2290000000000001,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":1.375},{"interface":"front","average":{"1min":1.248,"5min":1.052,"15min":1.2070000000000001},"min":{"1min":0.49399999999999999,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":6.1980000000000004,"5min":6.1980000000000004,"15min":7.25},"last":1.7889999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.2,"15min":1.2549999999999999},"min":{"1min":0.57099999999999995,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":4.6900000000000004,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":1.534},{"interface":"front","average":{"1min":1.643,"5min":1.244,"15min":1.266},"min":{"1min":0.55800000000000005,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":1.8999999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.1579999999999999,"15min":1.351},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.45600000000000002},"max":{"1min":4.7990000000000004,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.073,"15min":1.2869999999999999},"min":{"1min":0.50700000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.7190000000000003,"5min":4.7190000000000003,"15min":6.3639999999999999},"last":1.5649999999999999}]}]},{"osd":8,"up_from":65,"seq":279172874639,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":811008,"data_stored":405531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.0289999999999999,"15min":0.88300000000000001},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":2.4279999999999999,"5min":19.196999999999999,"15min":19.196999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.208,"15min":0.92400000000000004},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":7.1840000000000002,"5min":18.584,"15min":18.584},"last":0.90300000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":1.968,"15min":1.5329999999999999},"min":{"1min":0.33300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":15.981999999999999,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.988,"5min":1.97,"15min":1.625},"min":{"1min":0.44,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":15.817,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.65700000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.85099999999999998,"15min":0.84399999999999997},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.903,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.012,"15min":1.018},"min":{"1min":0.40300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":1.0449999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.77500000000000002,"15min":1.71},"min":{"1min":0.378,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.89300000000000002,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.81299999999999994,"15min":1.6559999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":1.1519999999999999,"5min":5.0380000000000003,"15min":224.71000000000001},"last":1.2729999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.872,"15min":0.91100000000000003},"min":{"1min":0.39600000000000002,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":5.4059999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":1.234},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.93799999999999994,"15min":1.016},"min":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.8170000000000002,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.3340000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.071,"15min":1.742},"min":{"1min":0.373,"5min":0.373,"15min":0.37},"max":{"1min":3.8050000000000002,"5min":5.3639999999999999,"15min":221.36699999999999},"last":1.083},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.99299999999999999,"15min":1.843},"min":{"1min":0.497,"5min":0.38,"15min":0.38},"max":{"1min":1.1919999999999999,"5min":4.4930000000000003,"15min":221.203},"last":1.1240000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.052,"15min":1.131},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":2.5619999999999998,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.1279999999999999,"15min":1.2310000000000001},"min":{"1min":0.54000000000000004,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":2.5779999999999998,"5min":4.2240000000000002,"15min":7.899},"last":1.4530000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.028,"15min":1.167},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.7030000000000001,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":1.21},{"interface":"front","average":{"1min":1.351,"5min":1.1160000000000001,"15min":1.2490000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":1.9299999999999999,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":1.375}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.97199999999999998,"15min":1.4399999999999999},"min":{"1min":0.437,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.6120000000000001,"5min":5.2560000000000002,"15min":109.92},"last":1.397},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.999,"15min":1.484},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.38500000000000001},"max":{"1min":1.786,"5min":5.0250000000000004,"15min":110.249},"last":1.349}]},{"osd":11,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.028,"15min":1.1830000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":5.2530000000000001,"5min":5.2530000000000001,"15min":5.2530000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.135,"5min":0.93300000000000005,"15min":1.143},"min":{"1min":0.50700000000000001,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.4730000000000001}]}]},{"osd":7,"up_from":56,"seq":240518168982,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":811008,"data_stored":405531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":3.1120000000000001,"15min":1.673},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.8310000000000004,"5min":187.33699999999999,"15min":187.33699999999999},"last":1.083},{"interface":"front","average":{"1min":0.95799999999999996,"5min":3.0539999999999998,"15min":1.6619999999999999},"min":{"1min":0.32100000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.4580000000000002,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.81299999999999994}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.302,"15min":1.056},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4729999999999999,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.6060000000000001,"15min":1.2130000000000001},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.6840000000000002,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.98399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90000000000000002,"15min":1.198},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.7869999999999999,"5min":7.1689999999999996,"15min":7.1689999999999996},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.89600000000000002,"15min":0.98499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.7549999999999999,"5min":6.7549999999999999,"15min":6.7549999999999999},"last":1.1080000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.145,"15min":0.95699999999999996},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.0049999999999999,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.95999999999999996,"15min":1.0029999999999999},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":6.9470000000000001,"5min":6.9470000000000001,"15min":6.9470000000000001},"last":1.1759999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.97799999999999998,"15min":0.95099999999999996},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9379999999999997,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":1.004},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.0600000000000001,"15min":1.048},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.8840000000000003,"5min":5.992,"15min":9.3640000000000008},"last":1.151}]},{"osd":6,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0449999999999999,"15min":1.081},"min":{"1min":0.38800000000000001,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":4.7409999999999997,"5min":6.319,"15min":6.319},"last":1.139},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.173,"15min":1.1890000000000001},"min":{"1min":0.39300000000000002,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":1.228}]},{"osd":8,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":6.5270000000000001,"15min":2.9329999999999998},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.6719999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":6.641,"15min":3.0590000000000002},"min":{"1min":0.434,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.5359999999999996,"5min":498.69999999999999,"15min":498.69999999999999},"last":1.552}]},{"osd":9,"last update":"Mon Feb 12 16:29:04 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.006,"15min":1.181},"min":{"1min":0.51600000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":12.907999999999999},"last":1.571},{"interface":"front","average":{"1min":1.484,"5min":1.1950000000000001,"15min":1.2949999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":7.1219999999999999,"5min":7.1219999999999999,"15min":7.1219999999999999},"last":1.3640000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":7.7859999999999996,"15min":3.5179999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":7.0339999999999998,"5min":497.52699999999999,"15min":497.52699999999999},"last":1.605},{"interface":"front","average":{"1min":1.6990000000000001,"5min":7.6740000000000004,"15min":3.423},"min":{"1min":0.50600000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.5890000000000004,"5min":496.29599999999999,"15min":496.29599999999999},"last":1.429}]},{"osd":11,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0409999999999999,"15min":1.5940000000000001},"min":{"1min":0.626,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.847,"5min":4.9039999999999999,"15min":120.943},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.2410000000000001,"15min":1.708},"min":{"1min":0.56999999999999995,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":4.181,"5min":7.0679999999999996,"15min":120.661},"last":1.458}]}]},{"osd":6,"up_from":48,"seq":206158430622,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302340,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442812,"statfs":{"total":95995035648,"available":95685439488,"internally_reserved":0,"allocated":798720,"data_stored":398964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":7.04,"5min":2.367,"15min":1.464},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":107.723,"5min":107.723,"15min":107.723},"last":0.97399999999999998},{"interface":"front","average":{"1min":7.1260000000000003,"5min":2.5139999999999998,"15min":1.552},"min":{"1min":0.38,"5min":0.252,"15min":0.252},"max":{"1min":107.114,"5min":107.114,"15min":107.114},"last":1.085}]},{"osd":1,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0449999999999999,"15min":1.923},"min":{"1min":0.36499999999999999,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":3.5870000000000002,"5min":5.5179999999999998,"15min":252.22800000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.05,"15min":1.946},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.0549999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5009999999999999,"15min":1.0649999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.1369999999999996,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.7090000000000001,"15min":1.1719999999999999},"min":{"1min":0.33700000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3769999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.92300000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.2529999999999999,"15min":1.024},"min":{"1min":0.29099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.4509999999999996,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":1.125},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.079,"15min":1.0620000000000001},"min":{"1min":0.34999999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.242,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Feb 12 16:28:10 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.117,"15min":0.99099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":4.4779999999999998,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":1.3799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.99299999999999999,"15min":1.0109999999999999},"min":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1080000000000001,"5min":10.52,"15min":10.52},"last":1.264}]},{"osd":7,"last update":"Mon Feb 12 16:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.97299999999999998,"15min":1.089},"min":{"1min":0.34200000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.399,"5min":4.7149999999999999,"15min":7.641},"last":0.753},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.81000000000000005,"15min":1.034},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.679,"5min":4.5609999999999999,"15min":7.8380000000000001},"last":1.171}]},{"osd":8,"last update":"Mon Feb 12 16:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1200000000000001,"15min":1.1259999999999999},"min":{"1min":0.48899999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.5119999999999996,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":1.6950000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.98899999999999999,"15min":1.175},"min":{"1min":0.55100000000000005,"5min":0.371,"15min":0.371},"max":{"1min":2.7679999999999998,"5min":4.6349999999999998,"15min":5.4379999999999997},"last":1.677}]},{"osd":9,"last update":"Mon Feb 12 16:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.246},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.8089999999999999,"5min":4.1680000000000001,"15min":18.059999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.96099999999999997,"15min":1.2470000000000001},"min":{"1min":0.45300000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9970000000000001,"5min":2.673,"15min":17.425000000000001},"last":1.321}]},{"osd":10,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.24,"15min":1.2909999999999999},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.7400000000000002,"5min":5.726,"15min":6.5300000000000002},"last":1.776},{"interface":"front","average":{"1min":1.423,"5min":1.337,"15min":1.3839999999999999},"min":{"1min":0.57699999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.2080000000000002,"5min":5.7590000000000003,"15min":6.5730000000000004},"last":1.288}]},{"osd":11,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2290000000000001,"15min":2.48},"min":{"1min":0.54600000000000004,"5min":0.52100000000000002,"15min":0.499},"max":{"1min":2.7970000000000002,"5min":5.6859999999999999,"15min":139.727},"last":1.552},{"interface":"front","average":{"1min":0.995,"5min":0.98899999999999999,"15min":2.3559999999999999},"min":{"1min":0.55400000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.173,"5min":2.706,"15min":139.24199999999999},"last":1.4399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895236,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":894208,"kb_avail":92849916,"statfs":{"total":95995035648,"available":95078313984,"internally_reserved":0,"allocated":798720,"data_stored":398964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915668992},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.129,"15min":0.96099999999999997},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":4.4610000000000003,"5min":16.632999999999999,"15min":16.632999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.645,"15min":1.212},"min":{"1min":0.33200000000000002,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":6.4939999999999998,"5min":7.7249999999999996,"15min":11.433},"last":0.79600000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.345,"15min":1.1240000000000001},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":12.778,"5min":12.778,"15min":12.778},"last":0.87},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.5569999999999999,"15min":1.216},"min":{"1min":0.28299999999999997,"5min":0.246,"15min":0.246},"max":{"1min":8.9329999999999998,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.754}]},{"osd":3,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.0920000000000001,"15min":1.1200000000000001},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.625,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.1910000000000001,"15min":0.95499999999999996},"min":{"1min":0.38300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.1639999999999997,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.98899999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.274,"15min":1.1220000000000001},"min":{"1min":0.42499999999999999,"5min":0.31,"15min":0.31},"max":{"1min":3.536,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.417,"15min":1.389},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.6859999999999999,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.95599999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.482,"15min":1.27},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.7789999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.347,"15min":1.2190000000000001},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":4.1340000000000003,"5min":10.067,"15min":10.067},"last":1.099}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.6799999999999999,"15min":1.7849999999999999},"min":{"1min":0.503,"5min":0.439,"15min":0.36399999999999999},"max":{"1min":3.5,"5min":30.437000000000001,"15min":55.869999999999997},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.599,"15min":1.8540000000000001},"min":{"1min":0.47299999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":4.1680000000000001,"5min":29.382000000000001,"15min":55.892000000000003},"last":1.258}]},{"osd":8,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":7.4459999999999997,"15min":3.4529999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.2300000000000004,"5min":548.91600000000005,"15min":548.91600000000005},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":7.5579999999999998,"15min":3.4359999999999999},"min":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.3769999999999998,"5min":552.42100000000005,"15min":552.42100000000005},"last":1.2789999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.4009999999999998,"5min":5.7590000000000003,"15min":10.07},"last":1.391},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.3380000000000001,"15min":1.466},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":4.2919999999999998,"5min":10.106,"15min":10.106},"last":1.4850000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":8.4459999999999997,"15min":3.7690000000000001},"min":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.105,"5min":548.93499999999995,"15min":548.93499999999995},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":8.4209999999999994,"15min":3.8460000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1989999999999998,"5min":549.12099999999998,"15min":549.12099999999998},"last":1.4970000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.536,"15min":1.5880000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":4.2619999999999996,"5min":10.202,"15min":10.202},"last":1.345},{"interface":"front","average":{"1min":1.212,"5min":1.552,"15min":1.6359999999999999},"min":{"1min":0.52200000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.1710000000000003,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":1.444}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590724,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":93154428,"statfs":{"total":95995035648,"available":95390134272,"internally_reserved":0,"allocated":798720,"data_stored":398964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.1830000000000001,"15min":1.0800000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":3.6890000000000001,"5min":10.754,"15min":11.324},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.27,"5min":1.401,"15min":1.179},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1020000000000003,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.80800000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0600000000000001,"15min":1.056},"min":{"1min":0.26500000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.032,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.244,"5min":1.1579999999999999,"15min":1.002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":3.8050000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.73599999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":3.5310000000000001,"15min":2.1269999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":5.4729999999999999,"5min":193.77000000000001,"15min":193.77000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":3.4790000000000001,"15min":2.1909999999999998},"min":{"1min":0.29499999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":5.141,"5min":189.625,"15min":189.625},"last":0.96999999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9350000000000001,"5min":1.3320000000000001,"15min":1.0780000000000001},"min":{"1min":0.29799999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":10.49,"5min":10.49,"15min":10.49},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.772,"5min":1.3200000000000001,"15min":1.0940000000000001},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":7.3250000000000002,"5min":7.3250000000000002,"15min":7.3250000000000002},"last":1.125}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":10.422000000000001,"5min":3.161,"15min":1.9890000000000001},"min":{"1min":0.371,"5min":0.311,"15min":0.311},"max":{"1min":189.79400000000001,"5min":189.79400000000001,"15min":189.79400000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":10.676,"5min":3.238,"15min":2.0680000000000001},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":194.41200000000001,"5min":194.41200000000001,"15min":194.41200000000001},"last":1.508}]},{"osd":7,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":3.5750000000000002,"15min":1.9450000000000001},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":7.4509999999999996,"5min":191.47200000000001,"15min":191.47200000000001},"last":1.335},{"interface":"front","average":{"1min":1.6319999999999999,"5min":3.6760000000000002,"15min":2.052},"min":{"1min":0.47399999999999998,"5min":0.44,"15min":0.44},"max":{"1min":7.4779999999999998,"5min":191.50299999999999,"15min":191.50299999999999},"last":1.452}]},{"osd":8,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.3480000000000001,"15min":1.3160000000000001},"min":{"1min":0.45900000000000002,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":7.3920000000000003,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.4399999999999999,"15min":1.4059999999999999},"min":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":7.5060000000000002,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":1.6200000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.2010000000000001,"15min":1.361},"min":{"1min":0.46899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.6550000000000002,"5min":7.6539999999999999,"15min":46.246000000000002},"last":1.052},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.345,"15min":1.4219999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.5330000000000004,"5min":5.5330000000000004,"15min":46.167999999999999},"last":1.3939999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.52,"15min":1.377},"min":{"1min":0.45100000000000001,"5min":0.442,"15min":0.41399999999999998},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.6459999999999999,"15min":1.4079999999999999},"min":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":9.3490000000000002,"5min":9.3490000000000002,"15min":9.3490000000000002},"last":1.0369999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:28:20 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.556,"15min":1.617},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":46.064},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.895,"5min":1.458,"15min":1.5629999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":7.4210000000000003,"5min":7.4210000000000003,"15min":45.642000000000003},"last":1.276}]}]},{"osd":2,"up_from":21,"seq":90194313661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306436,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438716,"statfs":{"total":95995035648,"available":95681245184,"internally_reserved":0,"allocated":798720,"data_stored":398964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:28:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.3560000000000001,"15min":1.1080000000000001},"min":{"1min":0.26000000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.6440000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.753},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.4970000000000001,"15min":1.2170000000000001},"min":{"1min":0.33900000000000002,"5min":0.247,"15min":0.247},"max":{"1min":4.5300000000000002,"5min":9.6470000000000002,"15min":9.6470000000000002},"last":0.77800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.52,"15min":1.9239999999999999},"min":{"1min":0.29599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":6.2910000000000004,"5min":21.407,"15min":21.407},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1020000000000001,"15min":0.92000000000000004},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.8899999999999997,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.89600000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.393,"15min":1.079},"min":{"1min":0.33500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0019999999999998,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.419,"15min":1.1839999999999999},"min":{"1min":0.33600000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.6060000000000001,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":1.232}]},{"osd":5,"last update":"Mon Feb 12 16:28:33 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.365,"15min":1.1539999999999999},"min":{"1min":0.38200000000000001,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":4.8040000000000003,"5min":6.133,"15min":6.133},"last":1.155},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.6930000000000001,"15min":1.22},"min":{"1min":0.437,"5min":0.315,"15min":0.315},"max":{"1min":6.8170000000000002,"5min":11.081,"15min":11.081},"last":0.97699999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:28:59 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.429,"15min":1.294},"min":{"1min":0.378,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4160000000000004,"5min":7.6890000000000001,"15min":21.527000000000001},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.4179999999999999,"15min":1.2230000000000001},"min":{"1min":0.44500000000000001,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":2.6200000000000001,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":1.4039999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.6359999999999999,"15min":1.234},"min":{"1min":0.39500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.1260000000000003,"5min":11.044,"15min":11.044},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.2629999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.8739999999999997,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":1.302}]},{"osd":8,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3919999999999999,"15min":1.2909999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42099999999999999},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.504,"5min":1.462,"15min":1.327},"min":{"1min":0.46000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":6.593,"5min":9.7639999999999993,"15min":9.7639999999999993},"last":1.448}]},{"osd":9,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.607,"15min":1.3500000000000001},"min":{"1min":0.51500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":6.0910000000000002,"5min":10.972,"15min":10.972},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.71,"15min":1.429},"min":{"1min":0.45600000000000002,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":6.1600000000000001,"5min":11.006,"15min":11.006},"last":1.375}]},{"osd":10,"last update":"Mon Feb 12 16:28:33 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":1.702,"15min":1.454},"min":{"1min":0.52200000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":7.7539999999999996,"5min":11.454000000000001,"15min":11.454000000000001},"last":1.534},{"interface":"front","average":{"1min":1.9299999999999999,"5min":1.7210000000000001,"15min":1.476},"min":{"1min":0.55300000000000005,"5min":0.441,"15min":0.441},"max":{"1min":7.7140000000000004,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":1.6990000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:28:59 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4770000000000001,"15min":1.4970000000000001},"min":{"1min":0.64400000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":7.258,"5min":7.258,"15min":7.258},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.581,"15min":1.5800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":1.5009999999999999}]}]},{"osd":3,"up_from":522,"seq":2241972928586,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333252,"kb_used_data":780,"kb_used_omap":91,"kb_used_meta":332132,"kb_avail":93411900,"statfs":{"total":95995035648,"available":95653785600,"internally_reserved":0,"allocated":798720,"data_stored":398964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":340103184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.3280000000000001,"15min":1.0069999999999999},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.5529999999999999,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.4570000000000001,"15min":1.054},"min":{"1min":0.373,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.536,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.94399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.2270000000000001,"15min":0.98799999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5410000000000004,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.196,"15min":1.1020000000000001},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.1040000000000001,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":1.603}]},{"osd":4,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.89800000000000002,"15min":1.25},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":4.0899999999999999,"5min":10.004,"15min":10.004},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.032,"15min":1.0980000000000001},"min":{"1min":0.35999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.806,"5min":6.806,"15min":6.806},"last":1.204}]},{"osd":5,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.0820000000000001,"15min":1.325},"min":{"1min":0.36499999999999999,"5min":0.316,"15min":0.316},"max":{"1min":4.7530000000000001,"5min":6.7539999999999996,"15min":6.7539999999999996},"last":1.9219999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.318,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.1340000000000003,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":1.0069999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.0780000000000001,"15min":1.1319999999999999},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.7450000000000001,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":1.7709999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.002,"15min":1.012},"min":{"1min":0.38300000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":4.1689999999999996,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":1.173}]},{"osd":7,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.1140000000000001,"15min":1.4099999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.21,"5min":6.851,"15min":6.851},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.218,"15min":1.2769999999999999},"min":{"1min":0.41799999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":3.8479999999999999,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":1.3799999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.0449999999999999,"15min":1.173},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.8109999999999999,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":2.105},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.99399999999999999,"15min":1.0980000000000001},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.2430000000000003,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":1.2310000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.2909999999999999,"15min":1.331},"min":{"1min":0.45500000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.1539999999999999,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.335,"15min":1.3799999999999999},"min":{"1min":0.51800000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.863,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":2.194}]},{"osd":10,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.2130000000000001,"15min":1.4750000000000001},"min":{"1min":0.54600000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.8700000000000001,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":2.016},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.3089999999999999,"15min":1.3260000000000001},"min":{"1min":0.501,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":3.8180000000000001,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":1.5780000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.242,"15min":1.284},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.9870000000000001,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.2629999999999999,"15min":1.4870000000000001},"min":{"1min":0.63300000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.8399999999999999,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":2.2810000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953902,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598688,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":93146464,"statfs":{"total":95995035648,"available":95381979136,"internally_reserved":0,"allocated":811008,"data_stored":405531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.131,"15min":1.3680000000000001},"min":{"1min":0.29899999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.492,"5min":7.492,"15min":90.597999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.248,"15min":1.5329999999999999},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":7.569,"5min":12.236000000000001,"15min":90.676000000000002},"last":0.622}]},{"osd":2,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.121,"15min":1},"min":{"1min":0.32900000000000001,"5min":0.28399999999999997,"15min":0.24399999999999999},"max":{"1min":7.3630000000000004,"5min":7.3630000000000004,"15min":8.0530000000000008},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.276,"5min":1.1890000000000001,"15min":1.1299999999999999},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":7.633,"5min":7.633,"15min":7.633},"last":1.1899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":1.1259999999999999,"15min":1.1020000000000001},"min":{"1min":0.33300000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.2930000000000001,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.018,"15min":1.046},"min":{"1min":0.35499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.657,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":1.3480000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.96299999999999997,"15min":0.93100000000000005},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.0529999999999999,"5min":4.7009999999999996,"15min":5.5999999999999996},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.1240000000000001,"15min":1.0960000000000001},"min":{"1min":0.41299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.081,"5min":4.7160000000000002,"15min":5.7450000000000001},"last":0.98299999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.83299999999999996,"15min":0.97799999999999998},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.8660000000000001,"5min":3.2000000000000002,"15min":7.5410000000000004},"last":0.749},{"interface":"front","average":{"1min":1.018,"5min":1.218,"15min":1.071},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.5190000000000001,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":0.94599999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.0269999999999999,"15min":1.099},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.312},"max":{"1min":4.2140000000000004,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.143,"5min":0.90000000000000002,"15min":1.1419999999999999},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":7.6500000000000004,"5min":7.6500000000000004,"15min":7.6689999999999996},"last":1.0740000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.002,"15min":1.179},"min":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.6819999999999999,"5min":4.0439999999999996,"15min":7.8650000000000002},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.143,"15min":1.2230000000000001},"min":{"1min":0.42399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3200000000000001,"5min":5.4489999999999998,"15min":5.6699999999999999},"last":1.476}]},{"osd":9,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.0660000000000001,"15min":1.4419999999999999},"min":{"1min":0.59199999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":7.7750000000000004,"5min":7.7750000000000004,"15min":34.112000000000002},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3520000000000001,"15min":1.635},"min":{"1min":0.41499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.827,"5min":7.827,"15min":34.590000000000003},"last":1.601}]},{"osd":10,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4239999999999999,"15min":1.425},"min":{"1min":0.45500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":7.7619999999999996,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.093,"15min":1.2450000000000001},"min":{"1min":0.52700000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":4.3019999999999996,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":1.095}]},{"osd":11,"last update":"Mon Feb 12 16:28:05 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0800000000000001,"15min":1.4470000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6500000000000004},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.357,"15min":1.482},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":3.52,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":1.8560000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306436,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93438716,"statfs":{"total":95995035648,"available":95681245184,"internally_reserved":0,"allocated":798720,"data_stored":398964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":12.419,"5min":3.194,"15min":3.4750000000000001},"min":{"1min":0.27300000000000002,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":221.96600000000001,"5min":221.96600000000001,"15min":565.34500000000003},"last":1.367},{"interface":"front","average":{"1min":12.515000000000001,"5min":3.3079999999999998,"15min":3.5859999999999999},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":222.06200000000001,"5min":222.06200000000001,"15min":565.98400000000004},"last":1.3029999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.516,"15min":1.145},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.8710000000000004,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.631,"15min":1.22},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.5549999999999997,"5min":40.225000000000001,"15min":40.225000000000001},"last":1.095}]},{"osd":3,"last update":"Mon Feb 12 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":1.4570000000000001,"15min":1.351},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":16.300999999999998,"5min":16.300999999999998,"15min":16.300999999999998},"last":1.137},{"interface":"front","average":{"1min":1.498,"5min":0.95999999999999996,"15min":0.80600000000000005},"min":{"1min":0.33700000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.5540000000000003,"5min":6.5540000000000003,"15min":6.5540000000000003},"last":0.96699999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.4119999999999999,"15min":1.425},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.9509999999999996,"5min":41.908000000000001,"15min":75.257000000000005},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.657,"5min":1.73,"15min":1.4750000000000001},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":5.4169999999999998,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.88600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1739999999999999,"15min":1.3320000000000001},"min":{"1min":0.34200000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.3490000000000002,"5min":6.4459999999999997,"15min":74.105000000000004},"last":1.502},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.2330000000000001,"15min":1.4390000000000001},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":9.5830000000000002,"5min":9.5830000000000002,"15min":75.013999999999996},"last":1.357}]},{"osd":7,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.1379999999999999,"15min":1.0449999999999999},"min":{"1min":0.42499999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.7429999999999999,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.325,"15min":1.1599999999999999},"min":{"1min":0.36599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.423,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":1.2649999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.071,"15min":1.173},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.2850000000000001,"5min":4.2850000000000001,"15min":5.9509999999999996},"last":1.417},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.2130000000000001,"15min":1.2210000000000001},"min":{"1min":0.48399999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.7749999999999999,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":1.647}]},{"osd":9,"last update":"Mon Feb 12 16:28:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4550000000000001,"15min":3.327},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":4.54,"5min":6.2939999999999996,"15min":449.89499999999998},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.829,"5min":1.321,"15min":3.3010000000000002},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":7.5789999999999997,"5min":9.6649999999999991,"15min":448.70800000000003},"last":1.391}]},{"osd":10,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.157,"15min":1.2509999999999999},"min":{"1min":0.55800000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.242,"5min":4.242,"15min":5.9210000000000003},"last":1.478},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.5449999999999999,"15min":1.353},"min":{"1min":0.48899999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.4619999999999997,"5min":15.006,"15min":15.006},"last":1.546}]},{"osd":11,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.2969999999999999,"15min":1.2889999999999999},"min":{"1min":0.59999999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.6940000000000008,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":1.569},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.548,"15min":1.391},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":4.5019999999999998,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":1.6240000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:29:11.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph status --format=json 2024-02-12T16:29:11.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:11.565 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:11.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:11.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:11.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:11.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:11.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:11.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:11.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:11.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:29:11.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:11 smithi123 bash[19119]: audit 2024-02-12T16:29:10.597963+0000 mgr.y (mgr.14146) 1331 : audit [DBG] from='client.16392 v1:172.21.15.121:0/3147552177' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:29:11.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:11.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:11.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:11 smithi169 bash[19604]: audit 2024-02-12T16:29:10.597963+0000 mgr.y (mgr.14146) 1331 : audit [DBG] from='client.16392 v1:172.21.15.121:0/3147552177' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:29:12.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:12.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:12.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:12.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:11 smithi121 bash[18489]: audit 2024-02-12T16:29:10.597963+0000 mgr.y (mgr.14146) 1331 : audit [DBG] from='client.16392 v1:172.21.15.121:0/3147552177' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:29:12.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:12.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:12.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:12.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:12.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:12.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:12.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:12.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:12.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:12.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:12.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:12.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:12.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:12.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:12.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:12.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:12 smithi123 bash[19119]: cluster 2024-02-12T16:29:12.013106+0000 mgr.y (mgr.14146) 1332 : cluster [DBG] pgmap v1856: 37 pgs: 3 creating+peering, 3 unknown, 31 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:12.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:12.969 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:12.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:12.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:13.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:12 smithi169 bash[19604]: cluster 2024-02-12T16:29:12.013106+0000 mgr.y (mgr.14146) 1332 : cluster [DBG] pgmap v1856: 37 pgs: 3 creating+peering, 3 unknown, 31 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:13.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:13.116 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:13.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:13.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:12 smithi121 bash[18489]: cluster 2024-02-12T16:29:12.013106+0000 mgr.y (mgr.14146) 1332 : cluster [DBG] pgmap v1856: 37 pgs: 3 creating+peering, 3 unknown, 31 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:13.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:13.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:13.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:13.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:13.401 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:13.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:13.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:13.543 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:13.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:13.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:13.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:13.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:13.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:13.827 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:13.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:13.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:13.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:13.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:14.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:14.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:14.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:14.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:14.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:14.248 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:14.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:14.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:14.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:14.386 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:14.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:14.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:14.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:14.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:14.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:14.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:14.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:14.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:14.809 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:14.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:14.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:14.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:14.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:15.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:15.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:15.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:15.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:15.228 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:15.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:15.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:15.344 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:15 smithi123 bash[19119]: cluster 2024-02-12T16:29:14.013586+0000 mgr.y (mgr.14146) 1333 : cluster [DBG] pgmap v1857: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:15.369 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:15.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:15.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:15 smithi121 bash[18489]: cluster 2024-02-12T16:29:14.013586+0000 mgr.y (mgr.14146) 1333 : cluster [DBG] pgmap v1857: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:15.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:15.513 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:15.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:15 smithi169 bash[19604]: cluster 2024-02-12T16:29:14.013586+0000 mgr.y (mgr.14146) 1333 : cluster [DBG] pgmap v1857: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:15.581 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:15.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:15.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:15.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:15.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:15.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:15.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:15.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:15.932 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:15.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:15.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:16.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:16.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:16.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:16.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:16.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:16.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:16.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:16.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:16.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:16.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:16.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:16.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:16.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:16.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:16.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:16.734 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:16.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:16.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:16.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:16.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:16 smithi121 bash[18489]: cluster 2024-02-12T16:29:15.698129+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e640: 12 total, 12 up, 12 in 2024-02-12T16:29:16.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:16 smithi121 bash[18489]: cluster 2024-02-12T16:29:16.013999+0000 mgr.y (mgr.14146) 1334 : cluster [DBG] pgmap v1859: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:16.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:16.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:17.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:17.029 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:16 smithi123 bash[19119]: cluster 2024-02-12T16:29:15.698129+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e640: 12 total, 12 up, 12 in 2024-02-12T16:29:17.030 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:16 smithi123 bash[19119]: cluster 2024-02-12T16:29:16.013999+0000 mgr.y (mgr.14146) 1334 : cluster [DBG] pgmap v1859: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:16 smithi169 bash[19604]: cluster 2024-02-12T16:29:15.698129+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e640: 12 total, 12 up, 12 in 2024-02-12T16:29:17.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:16 smithi169 bash[19604]: cluster 2024-02-12T16:29:16.013999+0000 mgr.y (mgr.14146) 1334 : cluster [DBG] pgmap v1859: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:17.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:17.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:17.190 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:17.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:17.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:17.332 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:17.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:17.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:17.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:17.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:17.572 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:17.605 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:17.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:17.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:17.719 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:17 smithi121 bash[18489]: cluster 2024-02-12T16:29:16.532619+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:17.719 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:17 smithi121 bash[18489]: cluster 2024-02-12T16:29:16.532723+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:17.719 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:17 smithi121 bash[18489]: cluster 2024-02-12T16:29:16.712752+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e641: 12 total, 12 up, 12 in 2024-02-12T16:29:17.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:17.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:17.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:17.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:17 smithi169 bash[19604]: cluster 2024-02-12T16:29:16.532619+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:17.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:17 smithi169 bash[19604]: cluster 2024-02-12T16:29:16.532723+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:17.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:17 smithi169 bash[19604]: cluster 2024-02-12T16:29:16.712752+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e641: 12 total, 12 up, 12 in 2024-02-12T16:29:17.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:17.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:17.888 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:29:17.888 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:29:17.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:18.011 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:18.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:18.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:18.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:17 smithi123 bash[19119]: cluster 2024-02-12T16:29:16.532619+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:18.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:17 smithi123 bash[19119]: cluster 2024-02-12T16:29:16.532723+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:18.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:17 smithi123 bash[19119]: cluster 2024-02-12T16:29:16.712752+0000 mon.a (mon.0) 2247 : cluster [DBG] osdmap e641: 12 total, 12 up, 12 in 2024-02-12T16:29:18.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:18.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:18.251 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:18.289 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:18.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:18.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:18.431 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:18.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:18.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:18.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:18.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:18.718 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:18.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:18.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:18.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:18.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:18 smithi121 bash[18489]: cluster 2024-02-12T16:29:18.014436+0000 mgr.y (mgr.14146) 1335 : cluster [DBG] pgmap v1861: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:18.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:18.997 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:19.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:18 smithi169 bash[19604]: cluster 2024-02-12T16:29:18.014436+0000 mgr.y (mgr.14146) 1335 : cluster [DBG] pgmap v1861: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:19.102 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:19.113 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:18 smithi123 bash[19119]: cluster 2024-02-12T16:29:18.014436+0000 mgr.y (mgr.14146) 1335 : cluster [DBG] pgmap v1861: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:19.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:19.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:19.277 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:19.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:19.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:19.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:19.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:19.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:19.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:29:19.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:19.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:19.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:19.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:19.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:20.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:20.114 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:20.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:20.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:20.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:20.254 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:20.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:20.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:20.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:20.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:20.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:20.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:20.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:20.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:20.678 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:20.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:20.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:20.820 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:20.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:20.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:20.959 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:20.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:21.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:21.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:21.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:21.204 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:21.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:21.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:21.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:21.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:21 smithi123 bash[19119]: cluster 2024-02-12T16:29:20.015185+0000 mgr.y (mgr.14146) 1336 : cluster [DBG] pgmap v1862: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:21.431 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:21 smithi123 bash[19119]: cluster 2024-02-12T16:29:20.700684+0000 mon.a (mon.0) 2248 : cluster [DBG] osdmap e642: 12 total, 12 up, 12 in 2024-02-12T16:29:21.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:21 smithi121 bash[18489]: cluster 2024-02-12T16:29:20.015185+0000 mgr.y (mgr.14146) 1336 : cluster [DBG] pgmap v1862: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:21.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:21 smithi121 bash[18489]: cluster 2024-02-12T16:29:20.700684+0000 mon.a (mon.0) 2248 : cluster [DBG] osdmap e642: 12 total, 12 up, 12 in 2024-02-12T16:29:21.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:21.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:21 smithi169 bash[19604]: cluster 2024-02-12T16:29:20.015185+0000 mgr.y (mgr.14146) 1336 : cluster [DBG] pgmap v1862: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:21 smithi169 bash[19604]: cluster 2024-02-12T16:29:20.700684+0000 mon.a (mon.0) 2248 : cluster [DBG] osdmap e642: 12 total, 12 up, 12 in 2024-02-12T16:29:21.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:21.606 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:21.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:21.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:21.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:21.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:21.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:21.894 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:21.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:21.999 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:22.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:22.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:22.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:22.172 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:22.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:22.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:22.288 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:22 smithi123 bash[19119]: cluster 2024-02-12T16:29:20.240027+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:22.289 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:22 smithi123 bash[19119]: cluster 2024-02-12T16:29:20.240123+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:22.289 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:22 smithi123 bash[19119]: cluster 2024-02-12T16:29:20.770636+0000 osd.2 (osd.2) 90 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:29:22.289 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:22 smithi123 bash[19119]: cluster 2024-02-12T16:29:20.828144+0000 osd.2 (osd.2) 91 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:29:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:22 smithi169 bash[19604]: cluster 2024-02-12T16:29:20.240027+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:22 smithi169 bash[19604]: cluster 2024-02-12T16:29:20.240123+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:22 smithi169 bash[19604]: cluster 2024-02-12T16:29:20.770636+0000 osd.2 (osd.2) 90 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:29:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:22 smithi169 bash[19604]: cluster 2024-02-12T16:29:20.828144+0000 osd.2 (osd.2) 91 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:29:22.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:22.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:22 smithi121 bash[18489]: cluster 2024-02-12T16:29:20.240027+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:22.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:22 smithi121 bash[18489]: cluster 2024-02-12T16:29:20.240123+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:22.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:22 smithi121 bash[18489]: cluster 2024-02-12T16:29:20.770636+0000 osd.2 (osd.2) 90 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:29:22.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:22 smithi121 bash[18489]: cluster 2024-02-12T16:29:20.828144+0000 osd.2 (osd.2) 91 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:29:22.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:22.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:22.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:22.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:22.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:22.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:22.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:22.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:22.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:22.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:22.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:22.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:23.011 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:23.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:23.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:23.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:23 smithi123 bash[19119]: cluster 2024-02-12T16:29:22.015896+0000 mgr.y (mgr.14146) 1337 : cluster [DBG] pgmap v1864: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:23.153 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:23.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:23.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:23.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:23.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:23 smithi169 bash[19604]: cluster 2024-02-12T16:29:22.015896+0000 mgr.y (mgr.14146) 1337 : cluster [DBG] pgmap v1864: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:23.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:23 smithi121 bash[18489]: cluster 2024-02-12T16:29:22.015896+0000 mgr.y (mgr.14146) 1337 : cluster [DBG] pgmap v1864: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:23.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:23.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:23.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:23.578 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:23.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:23.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:23.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:23.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:23.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:23.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:23.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:23.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:24.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:24.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:24.140 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:24.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:24.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:24.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:24.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:24.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:24.566 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:24.568 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:24.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:24.709 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:24.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:24.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:24.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:24.991 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:24.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:25.071 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:29:25.072 INFO:teuthology.orchestra.run.smithi121.stdout:{"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2361,"monmap":{"epoch":3,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":642,"num_osds":12,"num_up_osds":12,"osd_up_since":1707754981,"num_in_osds":12,"osd_in_since":1707754767,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":37}],"num_pgs":37,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":5242040320,"bytes_avail":1146698387456,"bytes_total":1151940427776},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.121:8443/"}},"servicemap":{"epoch":55,"modified":"2024-02-12T16:28:40.001125+0000","services":{"osd":{"daemons":{"summary":"","11":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-12T16:29:25.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:25.107 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-02-12T16:29:25.121 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:25.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:25.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:25.257 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:25.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:25 smithi169 bash[19604]: cluster 2024-02-12T16:29:24.016685+0000 mgr.y (mgr.14146) 1338 : cluster [DBG] pgmap v1865: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:25 smithi169 bash[19604]: audit 2024-02-12T16:29:24.632553+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? v1:172.21.15.121:0/1371922650' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:29:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:25 smithi169 bash[19604]: cluster 2024-02-12T16:29:24.922845+0000 osd.0 (osd.0) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:25 smithi169 bash[19604]: cluster 2024-02-12T16:29:24.922953+0000 osd.0 (osd.0) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:25 smithi169 bash[19604]: audit 2024-02-12T16:29:25.066846+0000 mon.a (mon.0) 2250 : audit [DBG] from='client.? v1:172.21.15.121:0/4125562496' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:29:25.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:25 smithi123 bash[19119]: cluster 2024-02-12T16:29:24.016685+0000 mgr.y (mgr.14146) 1338 : cluster [DBG] pgmap v1865: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:25 smithi123 bash[19119]: audit 2024-02-12T16:29:24.632553+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? v1:172.21.15.121:0/1371922650' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:29:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:25 smithi123 bash[19119]: cluster 2024-02-12T16:29:24.922845+0000 osd.0 (osd.0) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:25 smithi123 bash[19119]: cluster 2024-02-12T16:29:24.922953+0000 osd.0 (osd.0) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:25.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:25 smithi123 bash[19119]: audit 2024-02-12T16:29:25.066846+0000 mon.a (mon.0) 2250 : audit [DBG] from='client.? v1:172.21.15.121:0/4125562496' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:29:25.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:25.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:25.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:25 smithi121 bash[18489]: cluster 2024-02-12T16:29:24.016685+0000 mgr.y (mgr.14146) 1338 : cluster [DBG] pgmap v1865: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:25 smithi121 bash[18489]: audit 2024-02-12T16:29:24.632553+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? v1:172.21.15.121:0/1371922650' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:29:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:25 smithi121 bash[18489]: cluster 2024-02-12T16:29:24.922845+0000 osd.0 (osd.0) 103 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:25 smithi121 bash[18489]: cluster 2024-02-12T16:29:24.922953+0000 osd.0 (osd.0) 104 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:25 smithi121 bash[18489]: audit 2024-02-12T16:29:25.066846+0000 mon.a (mon.0) 2250 : audit [DBG] from='client.? v1:172.21.15.121:0/4125562496' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-12T16:29:25.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:25.535 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:25.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:25.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:25.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:25.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:25.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:25.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:25.873 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:29:25.895 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:29:25.914 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:29:25.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:25.946 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:29:25.947 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:25.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:25.975 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:29:25.998 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:29:26.022 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:29:26.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:26.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:29:26.079 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:29:26.086 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:26.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:26.106 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:29:26.132 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:29:26.159 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:29:26.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:29:26.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:26.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:26 smithi121 bash[18489]: audit 2024-02-12T16:29:25.098988+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.121:0/1371922650' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:29:26.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:26 smithi121 bash[18489]: cluster 2024-02-12T16:29:25.099120+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in 2024-02-12T16:29:26.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:26 smithi121 bash[18489]: cluster 2024-02-12T16:29:25.699566+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e644: 12 total, 12 up, 12 in 2024-02-12T16:29:26.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:26 smithi121 bash[18489]: audit 2024-02-12T16:29:25.845395+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.121:0/568906130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:26.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:26 smithi121 bash[18489]: audit 2024-02-12T16:29:26.052752+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? v1:172.21.15.121:0/3280809081' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:29:26.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:26.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:26.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:26.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:26.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:26.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:26 smithi123 bash[19119]: audit 2024-02-12T16:29:25.098988+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.121:0/1371922650' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:29:26.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:26 smithi123 bash[19119]: cluster 2024-02-12T16:29:25.099120+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in 2024-02-12T16:29:26.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:26 smithi123 bash[19119]: cluster 2024-02-12T16:29:25.699566+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e644: 12 total, 12 up, 12 in 2024-02-12T16:29:26.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:26 smithi123 bash[19119]: audit 2024-02-12T16:29:25.845395+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.121:0/568906130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:26.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:26 smithi123 bash[19119]: audit 2024-02-12T16:29:26.052752+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? v1:172.21.15.121:0/3280809081' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:29:26.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:26.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:26.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:26 smithi169 bash[19604]: audit 2024-02-12T16:29:25.098988+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.121:0/1371922650' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:29:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:26 smithi169 bash[19604]: cluster 2024-02-12T16:29:25.099120+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e643: 12 total, 12 up, 12 in 2024-02-12T16:29:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:26 smithi169 bash[19604]: cluster 2024-02-12T16:29:25.699566+0000 mon.a (mon.0) 2253 : cluster [DBG] osdmap e644: 12 total, 12 up, 12 in 2024-02-12T16:29:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:26 smithi169 bash[19604]: audit 2024-02-12T16:29:25.845395+0000 mon.a (mon.0) 2254 : audit [DBG] from='client.? v1:172.21.15.121:0/568906130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:26 smithi169 bash[19604]: audit 2024-02-12T16:29:26.052752+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? v1:172.21.15.121:0/3280809081' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:29:26.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:26.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:26.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:26.724 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:29:26.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:26.800 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:26.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:26.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:26.904 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:26.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:26.954 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-02-12T16:29:26.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:29:27.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:27.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:27.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:27.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:27.199 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:27 smithi123 bash[19119]: cluster 2024-02-12T16:29:26.017358+0000 mgr.y (mgr.14146) 1339 : cluster [DBG] pgmap v1868: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:27.200 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:27 smithi123 bash[19119]: cluster 2024-02-12T16:29:26.704827+0000 mon.a (mon.0) 2256 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:29:27.200 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:27 smithi123 bash[19119]: audit 2024-02-12T16:29:26.715383+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? v1:172.21.15.121:0/3280809081' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:29:27.200 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:27 smithi123 bash[19119]: cluster 2024-02-12T16:29:26.715470+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e645: 12 total, 12 up, 12 in 2024-02-12T16:29:27.222 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:27.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:27.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:27.362 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:27.364 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:27 smithi121 bash[18489]: cluster 2024-02-12T16:29:26.017358+0000 mgr.y (mgr.14146) 1339 : cluster [DBG] pgmap v1868: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:27.365 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:27 smithi121 bash[18489]: cluster 2024-02-12T16:29:26.704827+0000 mon.a (mon.0) 2256 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:29:27.365 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:27 smithi121 bash[18489]: audit 2024-02-12T16:29:26.715383+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? v1:172.21.15.121:0/3280809081' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:29:27.365 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:27 smithi121 bash[18489]: cluster 2024-02-12T16:29:26.715470+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e645: 12 total, 12 up, 12 in 2024-02-12T16:29:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:27.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:27.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:27.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:27.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:27 smithi169 bash[19604]: cluster 2024-02-12T16:29:26.017358+0000 mgr.y (mgr.14146) 1339 : cluster [DBG] pgmap v1868: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:27 smithi169 bash[19604]: cluster 2024-02-12T16:29:26.704827+0000 mon.a (mon.0) 2256 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:29:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:27 smithi169 bash[19604]: audit 2024-02-12T16:29:26.715383+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? v1:172.21.15.121:0/3280809081' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:29:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:27 smithi169 bash[19604]: cluster 2024-02-12T16:29:26.715470+0000 mon.a (mon.0) 2258 : cluster [DBG] osdmap e645: 12 total, 12 up, 12 in 2024-02-12T16:29:27.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:27.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:27.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:27.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:27.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:27.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:27.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:27.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:28.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:28.072 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:28.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:28.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:28.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:28.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:28.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:28.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:28.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:28.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:28.481 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:28.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:28.522 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-02-12T16:29:28.524 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:29:28.525 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:29:28.563 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:29:28.563 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:29:28.563 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_96224 2024-02-12T16:29:28.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:29:28.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:29:28.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:28.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:28.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:28.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:28.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:28.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:28.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:28.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:28.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:28.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:29.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:29.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:29.130 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:29 smithi169 bash[19604]: cluster 2024-02-12T16:29:28.018133+0000 mgr.y (mgr.14146) 1340 : cluster [DBG] pgmap v1870: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:29.151 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:29.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:29.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:29.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:29.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:29 smithi123 bash[19119]: cluster 2024-02-12T16:29:28.018133+0000 mgr.y (mgr.14146) 1340 : cluster [DBG] pgmap v1870: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:29 smithi121 bash[18489]: cluster 2024-02-12T16:29:28.018133+0000 mgr.y (mgr.14146) 1340 : cluster [DBG] pgmap v1870: 37 pgs: 37 active+clean; 0 B data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:29.416 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:29.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:29.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:29.563 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 15 5765 5750 359.357 359.375 0.00175297 0.00277721 2024-02-12T16:29:29.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:29.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:29.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:29.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:29.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:29.952 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:29.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:30.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:30.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:30.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:30.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:30.198 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:30 smithi123 bash[19119]: audit 2024-02-12T16:29:30.054465+0000 mon.a (mon.0) 2259 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:29:30.228 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:30.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:30.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:29:30.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:29:30.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:30.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:30.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:30.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:30 smithi121 bash[18489]: audit 2024-02-12T16:29:30.054465+0000 mon.a (mon.0) 2259 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:29:30.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:30.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:30.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:30 smithi169 bash[19604]: audit 2024-02-12T16:29:30.054465+0000 mon.a (mon.0) 2259 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:29:30.563 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 16 10684 10668 333.349 307.375 0.00328357 0.00299588 2024-02-12T16:29:30.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:30.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:30.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:30.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:30.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:30.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:30.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:30.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:30.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:30.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:31.022 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:31.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:31.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:31.158 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:31.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:31.285 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:31.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:31 smithi123 bash[19119]: cluster 2024-02-12T16:29:29.737841+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:31 smithi123 bash[19119]: cluster 2024-02-12T16:29:29.737893+0000 osd.3 (osd.3) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:31 smithi123 bash[19119]: cluster 2024-02-12T16:29:30.018593+0000 mgr.y (mgr.14146) 1341 : cluster [DBG] pgmap v1871: 37 pgs: 37 active+clean; 42 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:31.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:31 smithi123 bash[19119]: cluster 2024-02-12T16:29:30.117085+0000 mon.a (mon.0) 2260 : cluster [DBG] osdmap e646: 12 total, 12 up, 12 in 2024-02-12T16:29:31.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:31 smithi121 bash[18489]: cluster 2024-02-12T16:29:29.737841+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:31.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:31 smithi121 bash[18489]: cluster 2024-02-12T16:29:29.737893+0000 osd.3 (osd.3) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:31.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:31 smithi121 bash[18489]: cluster 2024-02-12T16:29:30.018593+0000 mgr.y (mgr.14146) 1341 : cluster [DBG] pgmap v1871: 37 pgs: 37 active+clean; 42 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:31.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:31 smithi121 bash[18489]: cluster 2024-02-12T16:29:30.117085+0000 mon.a (mon.0) 2260 : cluster [DBG] osdmap e646: 12 total, 12 up, 12 in 2024-02-12T16:29:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:31.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:31.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:31 smithi169 bash[19604]: cluster 2024-02-12T16:29:29.737841+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:31 smithi169 bash[19604]: cluster 2024-02-12T16:29:29.737893+0000 osd.3 (osd.3) 27 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:31 smithi169 bash[19604]: cluster 2024-02-12T16:29:30.018593+0000 mgr.y (mgr.14146) 1341 : cluster [DBG] pgmap v1871: 37 pgs: 37 active+clean; 42 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:31 smithi169 bash[19604]: cluster 2024-02-12T16:29:30.117085+0000 mon.a (mon.0) 2260 : cluster [DBG] osdmap e646: 12 total, 12 up, 12 in 2024-02-12T16:29:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:31.563 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 15481 15465 322.161 299.812 0.00321362 0.00310127 2024-02-12T16:29:31.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:31.680 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:31.741 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:31.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:31.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:31.856 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:29:31.856 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:29:31.856 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:29:31.856 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:29:31.856 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:29:31.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:31.950 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:31.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:32.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:32.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:32.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:32.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:32.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:32.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:32.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:32.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 16 18571 18555 289.897 193.125 0.00383347 0.00341919 2024-02-12T16:29:32.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:32.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:32.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:32.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:32.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:32.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:32.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:32.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:32.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:33.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:33.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:33.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:33.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:33.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:33.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:33.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:33.380 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:33.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:33 smithi121 bash[18489]: cluster 2024-02-12T16:29:32.018993+0000 mgr.y (mgr.14146) 1342 : cluster [DBG] pgmap v1873: 37 pgs: 37 active+clean; 361 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:33.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:33.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:33 smithi123 bash[19119]: cluster 2024-02-12T16:29:32.018993+0000 mgr.y (mgr.14146) 1342 : cluster [DBG] pgmap v1873: 37 pgs: 37 active+clean; 361 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:33.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:33.513 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:33.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:33.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:33 smithi169 bash[19604]: cluster 2024-02-12T16:29:32.018993+0000 mgr.y (mgr.14146) 1342 : cluster [DBG] pgmap v1873: 37 pgs: 37 active+clean; 361 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:33.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 22678 22662 283.249 256.688 0.00345599 0.00352843 2024-02-12T16:29:33.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:33.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:33.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:33.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:33.767 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:33.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:33.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:33.899 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:33.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:34.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:34.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:29:34.164 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:34.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:34.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:34.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:34.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:34.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 26528 26512 276.142 240.625 0.00460267 0.00361917 2024-02-12T16:29:34.576 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:34.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:34.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:34.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:34.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.968 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:34.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:35.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:35.103 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:35.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:35.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:35.235 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:35.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:35.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:35.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:35.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:35.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:35 smithi123 bash[19119]: cluster 2024-02-12T16:29:34.019458+0000 mgr.y (mgr.14146) 1343 : cluster [DBG] pgmap v1874: 37 pgs: 37 active+clean; 895 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:35.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:35 smithi121 bash[18489]: cluster 2024-02-12T16:29:34.019458+0000 mgr.y (mgr.14146) 1343 : cluster [DBG] pgmap v1874: 37 pgs: 37 active+clean; 895 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:35.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:35.476 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:35 smithi169 bash[19604]: cluster 2024-02-12T16:29:34.019458+0000 mgr.y (mgr.14146) 1343 : cluster [DBG] pgmap v1874: 37 pgs: 37 active+clean; 895 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:35.492 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:35.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:35.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 16 29698 29682 264.994 198.125 0.00554119 0.00377159 2024-02-12T16:29:35.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:35.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:35.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:35.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:35.755 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:35.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:35.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:35.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:35.883 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:35.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:35.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:36.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:36.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.145 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:36.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:36.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:36.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.406 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:36.512 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:36.541 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:36.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 16 33257 33241 259.673 222.438 0.00215839 0.00384885 2024-02-12T16:29:36.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:36.763 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:36.764 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:36.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:36.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:36.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.914 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:36 smithi123 bash[19119]: cluster 2024-02-12T16:29:36.019930+0000 mgr.y (mgr.14146) 1344 : cluster [DBG] pgmap v1875: 37 pgs: 37 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:36.947 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:36.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:36 smithi169 bash[19604]: cluster 2024-02-12T16:29:36.019930+0000 mgr.y (mgr.14146) 1344 : cluster [DBG] pgmap v1875: 37 pgs: 37 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:37.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:37.073 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:37.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:36 smithi121 bash[18489]: cluster 2024-02-12T16:29:36.019930+0000 mgr.y (mgr.14146) 1344 : cluster [DBG] pgmap v1875: 37 pgs: 37 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:29:37.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:37.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:37.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:37.474 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:37.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 36526 36510 253.52 204.312 0.00293253 0.00394257 2024-02-12T16:29:37.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:37.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:37.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:37.717 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:37 smithi123 bash[19119]: cluster 2024-02-12T16:29:36.854640+0000 osd.8 (osd.8) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:37.717 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:37 smithi123 bash[19119]: cluster 2024-02-12T16:29:36.854698+0000 osd.8 (osd.8) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:37.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:37.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:37.858 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:37.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:37 smithi121 bash[18489]: cluster 2024-02-12T16:29:36.854640+0000 osd.8 (osd.8) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:37 smithi121 bash[18489]: cluster 2024-02-12T16:29:36.854698+0000 osd.8 (osd.8) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:37.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:37.968 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:37 smithi169 bash[19604]: cluster 2024-02-12T16:29:36.854640+0000 osd.8 (osd.8) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:37.968 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:37 smithi169 bash[19604]: cluster 2024-02-12T16:29:36.854698+0000 osd.8 (osd.8) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:37.986 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:37.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:38.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:38.110 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:38.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:38.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:38.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:38.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:38.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:38.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:38.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:38.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:38.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:38.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 16 39573 39557 247.21 190.438 0.00228014 0.00404324 2024-02-12T16:29:38.600 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:38.622 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:38.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:38.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:38.728 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:38 smithi121 bash[18489]: cluster 2024-02-12T16:29:37.718396+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e647: 12 total, 12 up, 12 in 2024-02-12T16:29:38.749 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:38.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:38.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:38.862 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:38 smithi169 bash[19604]: cluster 2024-02-12T16:29:37.718396+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e647: 12 total, 12 up, 12 in 2024-02-12T16:29:38.862 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:38 smithi169 bash[19604]: cluster 2024-02-12T16:29:38.020246+0000 mgr.y (mgr.14146) 1345 : cluster [DBG] pgmap v1877: 37 pgs: 37 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 MiB/s wr, 3.49k op/s 2024-02-12T16:29:38.881 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:38.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:38.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:39.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:39.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:39.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:38 smithi123 bash[19119]: cluster 2024-02-12T16:29:37.718396+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e647: 12 total, 12 up, 12 in 2024-02-12T16:29:39.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:38 smithi123 bash[19119]: cluster 2024-02-12T16:29:38.020246+0000 mgr.y (mgr.14146) 1345 : cluster [DBG] pgmap v1877: 37 pgs: 37 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 MiB/s wr, 3.49k op/s 2024-02-12T16:29:39.147 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:39.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:38 smithi121 bash[18489]: cluster 2024-02-12T16:29:38.020246+0000 mgr.y (mgr.14146) 1345 : cluster [DBG] pgmap v1877: 37 pgs: 37 active+clean; 1.7 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 218 MiB/s wr, 3.49k op/s 2024-02-12T16:29:39.251 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:39.284 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:39.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:39.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:39.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:39.545 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:39.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 42512 42496 241.433 183.688 0.0228656 0.00413621 2024-02-12T16:29:39.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:39.683 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:39.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:39.821 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:39.926 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:39.959 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:39.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:40.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:40.094 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:40.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:40.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:40.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:40.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:40.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:40.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:40.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:40.485 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:40.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:40.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 44805 44789 233.256 143.312 0.00408941 0.00428471 2024-02-12T16:29:40.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:40.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:40.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:40.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:40.751 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:40.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:29:40.814 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:29:40.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:40.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:40.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:40.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:41.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:41.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:41.139 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:41 smithi123 bash[19119]: cluster 2024-02-12T16:29:40.020691+0000 mgr.y (mgr.14146) 1346 : cluster [DBG] pgmap v1878: 37 pgs: 37 active+clean; 2.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-02-12T16:29:41.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:41 smithi123 bash[19119]: audit 2024-02-12T16:29:40.444514+0000 mon.a (mon.0) 2262 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:29:41.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:41 smithi123 bash[19119]: cluster 2024-02-12T16:29:40.703401+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e648: 12 total, 12 up, 12 in 2024-02-12T16:29:41.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:41 smithi123 bash[19119]: audit 2024-02-12T16:29:40.750016+0000 mon.a (mon.0) 2264 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:29:41.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:41 smithi123 bash[19119]: audit 2024-02-12T16:29:40.750975+0000 mon.a (mon.0) 2265 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:29:41.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:41 smithi123 bash[19119]: audit 2024-02-12T16:29:40.766944+0000 mon.a (mon.0) 2266 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:29:41.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:41 smithi121 bash[18489]: cluster 2024-02-12T16:29:40.020691+0000 mgr.y (mgr.14146) 1346 : cluster [DBG] pgmap v1878: 37 pgs: 37 active+clean; 2.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-02-12T16:29:41.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:41 smithi121 bash[18489]: audit 2024-02-12T16:29:40.444514+0000 mon.a (mon.0) 2262 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:29:41.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:41 smithi121 bash[18489]: cluster 2024-02-12T16:29:40.703401+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e648: 12 total, 12 up, 12 in 2024-02-12T16:29:41.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:41 smithi121 bash[18489]: audit 2024-02-12T16:29:40.750016+0000 mon.a (mon.0) 2264 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:29:41.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:41 smithi121 bash[18489]: audit 2024-02-12T16:29:40.750975+0000 mon.a (mon.0) 2265 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:29:41.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:41 smithi121 bash[18489]: audit 2024-02-12T16:29:40.766944+0000 mon.a (mon.0) 2266 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:29:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:41.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:41.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:41.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:41.374 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:41 smithi169 bash[19604]: cluster 2024-02-12T16:29:40.020691+0000 mgr.y (mgr.14146) 1346 : cluster [DBG] pgmap v1878: 37 pgs: 37 active+clean; 2.1 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-02-12T16:29:41.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:41 smithi169 bash[19604]: audit 2024-02-12T16:29:40.444514+0000 mon.a (mon.0) 2262 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:29:41.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:41 smithi169 bash[19604]: cluster 2024-02-12T16:29:40.703401+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e648: 12 total, 12 up, 12 in 2024-02-12T16:29:41.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:41 smithi169 bash[19604]: audit 2024-02-12T16:29:40.750016+0000 mon.a (mon.0) 2264 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:29:41.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:41 smithi169 bash[19604]: audit 2024-02-12T16:29:40.750975+0000 mon.a (mon.0) 2265 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:29:41.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:41 smithi169 bash[19604]: audit 2024-02-12T16:29:40.766944+0000 mon.a (mon.0) 2266 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:29:41.389 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:41.492 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:41.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:41.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 47381 47365 227.697 161 0.00907712 0.0043898 2024-02-12T16:29:41.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:41.644 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:41.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:41.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:41.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:41.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:41.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:41.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:42.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:42.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:42.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:42.156 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:42.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:42 smithi121 bash[18489]: audit 2024-02-12T16:29:40.792901+0000 mgr.y (mgr.14146) 1347 : audit [DBG] from='client.16422 v1:172.21.15.121:0/802325866' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:29:42.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:42.265 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:42 smithi169 bash[19604]: audit 2024-02-12T16:29:40.792901+0000 mgr.y (mgr.14146) 1347 : audit [DBG] from='client.16422 v1:172.21.15.121:0/802325866' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:29:42.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:42.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:42 smithi123 bash[19119]: audit 2024-02-12T16:29:40.792901+0000 mgr.y (mgr.14146) 1347 : audit [DBG] from='client.16422 v1:172.21.15.121:0/802325866' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:29:42.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:42.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:42.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 49708 49692 221.82 145.438 0.009803 0.00450644 2024-02-12T16:29:42.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:42.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:42.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:42.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:42.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:42.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:42.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:42.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:43.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:43.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:43.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:43.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:43.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:43.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:43.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:43.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:43.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:43.340 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:43 smithi169 bash[19604]: cluster 2024-02-12T16:29:42.021053+0000 mgr.y (mgr.14146) 1348 : cluster [DBG] pgmap v1880: 37 pgs: 37 active+clean; 2.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 212 MiB/s wr, 3.38k op/s 2024-02-12T16:29:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:43.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:43 smithi123 bash[19119]: cluster 2024-02-12T16:29:42.021053+0000 mgr.y (mgr.14146) 1348 : cluster [DBG] pgmap v1880: 37 pgs: 37 active+clean; 2.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 212 MiB/s wr, 3.38k op/s 2024-02-12T16:29:43.425 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:43 smithi121 bash[18489]: cluster 2024-02-12T16:29:42.021053+0000 mgr.y (mgr.14146) 1348 : cluster [DBG] pgmap v1880: 37 pgs: 37 active+clean; 2.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 212 MiB/s wr, 3.38k op/s 2024-02-12T16:29:43.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:43.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:43.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:43.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 52148 52132 217.198 152.5 0.00396822 0.00460223 2024-02-12T16:29:43.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:43.613 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:43.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:43.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:43.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:43.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:43.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:43.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:43.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:43.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:44.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:44.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:44.129 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:44.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:44.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:44.393 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:44.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:44.521 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:44.564 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 16 54756 54740 213.81 163 0.00647424 0.0046757 2024-02-12T16:29:44.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:44.646 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1879,"stamp":"2024-02-12T16:29:40.701167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2598305792,"num_objects":39647,"num_object_clones":0,"num_object_copies":79294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39647,"num_write_kb":2537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39647,"ondisk_log_size":39647,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10021684,"kb_used_data":4871828,"kb_used_omap":91,"kb_used_meta":5149668,"kb_avail":1114920140,"statfs":{"total":1151940427776,"available":1141678223360,"internally_reserved":0,"allocated":4988751872,"data_stored":4983127698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":5273260048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,1,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2219442176,"num_objects":33866,"num_object_clones":0,"num_object_copies":67732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33866,"num_write_kb":2167424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.682394"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":143,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-12T16:29:27.741919+0000","last_change":"2024-02-12T16:29:20.828257+0000","last_active":"2024-02-12T16:29:27.741919+0000","last_peered":"2024-02-12T16:29:27.741919+0000","last_clean":"2024-02-12T16:29:27.741919+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:29:27.741919+0000","last_undegraded":"2024-02-12T16:29:27.741919+0000","last_fullsized":"2024-02-12T16:29:27.741919+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:20.828149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:29:20.828149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"646'2211","reported_seq":2228,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:37.103130+0000","last_change":"2024-02-12T16:29:08.737178+0000","last_active":"2024-02-12T16:29:37.103130+0000","last_peered":"2024-02-12T16:29:37.103130+0000","last_clean":"2024-02-12T16:29:37.103130+0000","last_became_active":"2024-02-12T16:29:08.736890+0000","last_became_peered":"2024-02-12T16:29:08.736890+0000","last_unstale":"2024-02-12T16:29:37.103130+0000","last_undegraded":"2024-02-12T16:29:37.103130+0000","last_fullsized":"2024-02-12T16:29:37.103130+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2211,"ondisk_log_size":2211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":4422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":141504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":255,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-12T16:29:26.896245+0000","last_change":"2024-02-12T16:28:53.018782+0000","last_active":"2024-02-12T16:29:26.896245+0000","last_peered":"2024-02-12T16:29:26.896245+0000","last_clean":"2024-02-12T16:29:26.896245+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:29:26.896245+0000","last_undegraded":"2024-02-12T16:29:26.896245+0000","last_fullsized":"2024-02-12T16:29:26.896245+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.018681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.018681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"647'2642","reported_seq":2659,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:39.357237+0000","last_change":"2024-02-12T16:29:08.734672+0000","last_active":"2024-02-12T16:29:39.357237+0000","last_peered":"2024-02-12T16:29:39.357237+0000","last_clean":"2024-02-12T16:29:39.357237+0000","last_became_active":"2024-02-12T16:29:08.734223+0000","last_became_peered":"2024-02-12T16:29:08.734223+0000","last_unstale":"2024-02-12T16:29:39.357237+0000","last_undegraded":"2024-02-12T16:29:39.357237+0000","last_fullsized":"2024-02-12T16:29:39.357237+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2642,"ondisk_log_size":2642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2642,"num_write_kb":169088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":333,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.127827+0000","last_change":"2024-02-12T16:28:54.310544+0000","last_active":"2024-02-12T16:29:30.127827+0000","last_peered":"2024-02-12T16:29:30.127827+0000","last_clean":"2024-02-12T16:29:30.127827+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:29:30.127827+0000","last_undegraded":"2024-02-12T16:29:30.127827+0000","last_fullsized":"2024-02-12T16:29:30.127827+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.310448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.310448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"647'2390","reported_seq":2408,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:38.299525+0000","last_change":"2024-02-12T16:29:08.805574+0000","last_active":"2024-02-12T16:29:38.299525+0000","last_peered":"2024-02-12T16:29:38.299525+0000","last_clean":"2024-02-12T16:29:38.299525+0000","last_became_active":"2024-02-12T16:29:08.805299+0000","last_became_peered":"2024-02-12T16:29:08.805299+0000","last_unstale":"2024-02-12T16:29:38.299525+0000","last_undegraded":"2024-02-12T16:29:38.299525+0000","last_fullsized":"2024-02-12T16:29:38.299525+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2390,"ondisk_log_size":2390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156631040,"num_objects":2390,"num_object_clones":0,"num_object_copies":4780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":152960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":737,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.195089+0000","last_change":"2024-02-12T16:28:50.494186+0000","last_active":"2024-02-12T16:29:30.195089+0000","last_peered":"2024-02-12T16:29:30.195089+0000","last_clean":"2024-02-12T16:29:30.195089+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:29:30.195089+0000","last_undegraded":"2024-02-12T16:29:30.195089+0000","last_fullsized":"2024-02-12T16:29:30.195089+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:50.494005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:50.494005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.0","version":"647'2806","reported_seq":2824,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:40.470504+0000","last_change":"2024-02-12T16:29:08.733321+0000","last_active":"2024-02-12T16:29:40.470504+0000","last_peered":"2024-02-12T16:29:40.470504+0000","last_clean":"2024-02-12T16:29:40.470504+0000","last_became_active":"2024-02-12T16:29:08.732997+0000","last_became_peered":"2024-02-12T16:29:08.732997+0000","last_unstale":"2024-02-12T16:29:40.470504+0000","last_undegraded":"2024-02-12T16:29:40.470504+0000","last_fullsized":"2024-02-12T16:29:40.470504+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2806,"ondisk_log_size":2806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":183894016,"num_objects":2806,"num_object_clones":0,"num_object_copies":5612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2806,"num_write_kb":179584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":377,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.121930+0000","last_change":"2024-02-12T16:28:53.942753+0000","last_active":"2024-02-12T16:29:30.121930+0000","last_peered":"2024-02-12T16:29:30.121930+0000","last_clean":"2024-02-12T16:29:30.121930+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:29:30.121930+0000","last_undegraded":"2024-02-12T16:29:30.121930+0000","last_fullsized":"2024-02-12T16:29:30.121930+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.942656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.942656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"647'2539","reported_seq":2557,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:38.830772+0000","last_change":"2024-02-12T16:29:08.809293+0000","last_active":"2024-02-12T16:29:38.830772+0000","last_peered":"2024-02-12T16:29:38.830772+0000","last_clean":"2024-02-12T16:29:38.830772+0000","last_became_active":"2024-02-12T16:29:08.808944+0000","last_became_peered":"2024-02-12T16:29:08.808944+0000","last_unstale":"2024-02-12T16:29:38.830772+0000","last_undegraded":"2024-02-12T16:29:38.830772+0000","last_fullsized":"2024-02-12T16:29:38.830772+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2539,"ondisk_log_size":2539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":166395904,"num_objects":2539,"num_object_clones":0,"num_object_copies":5078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2539,"num_write_kb":162496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":324,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.127856+0000","last_change":"2024-02-12T16:28:56.397511+0000","last_active":"2024-02-12T16:29:30.127856+0000","last_peered":"2024-02-12T16:29:30.127856+0000","last_clean":"2024-02-12T16:29:30.127856+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:29:30.127856+0000","last_undegraded":"2024-02-12T16:29:30.127856+0000","last_fullsized":"2024-02-12T16:29:30.127856+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:56.397266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:56.397266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"647'2703","reported_seq":2720,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:40.220074+0000","last_change":"2024-02-12T16:29:08.733146+0000","last_active":"2024-02-12T16:29:40.220074+0000","last_peered":"2024-02-12T16:29:40.220074+0000","last_clean":"2024-02-12T16:29:40.220074+0000","last_became_active":"2024-02-12T16:29:08.732848+0000","last_became_peered":"2024-02-12T16:29:08.732848+0000","last_unstale":"2024-02-12T16:29:40.220074+0000","last_undegraded":"2024-02-12T16:29:40.220074+0000","last_fullsized":"2024-02-12T16:29:40.220074+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2703,"ondisk_log_size":2703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":177143808,"num_objects":2703,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2703,"num_write_kb":172992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":334,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.127816+0000","last_change":"2024-02-12T16:28:58.383870+0000","last_active":"2024-02-12T16:29:30.127816+0000","last_peered":"2024-02-12T16:29:30.127816+0000","last_clean":"2024-02-12T16:29:30.127816+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:29:30.127816+0000","last_undegraded":"2024-02-12T16:29:30.127816+0000","last_fullsized":"2024-02-12T16:29:30.127816+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.383742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.383742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"646'2132","reported_seq":2149,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:36.780815+0000","last_change":"2024-02-12T16:29:08.734515+0000","last_active":"2024-02-12T16:29:36.780815+0000","last_peered":"2024-02-12T16:29:36.780815+0000","last_clean":"2024-02-12T16:29:36.780815+0000","last_became_active":"2024-02-12T16:29:08.734296+0000","last_became_peered":"2024-02-12T16:29:08.734296+0000","last_unstale":"2024-02-12T16:29:36.780815+0000","last_undegraded":"2024-02-12T16:29:36.780815+0000","last_fullsized":"2024-02-12T16:29:36.780815+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2132,"ondisk_log_size":2132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":4264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":136448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":193,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.119171+0000","last_change":"2024-02-12T16:28:53.712516+0000","last_active":"2024-02-12T16:29:30.119171+0000","last_peered":"2024-02-12T16:29:30.119171+0000","last_clean":"2024-02-12T16:29:30.119171+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:29:30.119171+0000","last_undegraded":"2024-02-12T16:29:30.119171+0000","last_fullsized":"2024-02-12T16:29:30.119171+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.712421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.712421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"647'2417","reported_seq":2435,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:38.304194+0000","last_change":"2024-02-12T16:29:08.804530+0000","last_active":"2024-02-12T16:29:38.304194+0000","last_peered":"2024-02-12T16:29:38.304194+0000","last_clean":"2024-02-12T16:29:38.304194+0000","last_became_active":"2024-02-12T16:29:08.804218+0000","last_became_peered":"2024-02-12T16:29:08.804218+0000","last_unstale":"2024-02-12T16:29:38.304194+0000","last_undegraded":"2024-02-12T16:29:38.304194+0000","last_fullsized":"2024-02-12T16:29:38.304194+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2417,"ondisk_log_size":2417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158400512,"num_objects":2417,"num_object_clones":0,"num_object_copies":4834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":154688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":765,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.122546+0000","last_change":"2024-02-12T16:28:53.646270+0000","last_active":"2024-02-12T16:29:30.122546+0000","last_peered":"2024-02-12T16:29:30.122546+0000","last_clean":"2024-02-12T16:29:30.122546+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:29:30.122546+0000","last_undegraded":"2024-02-12T16:29:30.122546+0000","last_fullsized":"2024-02-12T16:29:30.122546+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.646027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.646027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"646'2180","reported_seq":2197,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:37.086918+0000","last_change":"2024-02-12T16:29:08.804335+0000","last_active":"2024-02-12T16:29:37.086918+0000","last_peered":"2024-02-12T16:29:37.086918+0000","last_clean":"2024-02-12T16:29:37.086918+0000","last_became_active":"2024-02-12T16:29:08.804047+0000","last_became_peered":"2024-02-12T16:29:08.804047+0000","last_unstale":"2024-02-12T16:29:37.086918+0000","last_undegraded":"2024-02-12T16:29:37.086918+0000","last_fullsized":"2024-02-12T16:29:37.086918+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2180,"ondisk_log_size":2180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":142868480,"num_objects":2180,"num_object_clones":0,"num_object_copies":4360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2180,"num_write_kb":139520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":140,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.122553+0000","last_change":"2024-02-12T16:28:57.806094+0000","last_active":"2024-02-12T16:29:30.122553+0000","last_peered":"2024-02-12T16:29:30.122553+0000","last_clean":"2024-02-12T16:29:30.122553+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:29:30.122553+0000","last_undegraded":"2024-02-12T16:29:30.122553+0000","last_fullsized":"2024-02-12T16:29:30.122553+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:57.806002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:57.806002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"647'2775","reported_seq":2792,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:40.244538+0000","last_change":"2024-02-12T16:29:08.734597+0000","last_active":"2024-02-12T16:29:40.244538+0000","last_peered":"2024-02-12T16:29:40.244538+0000","last_clean":"2024-02-12T16:29:40.244538+0000","last_became_active":"2024-02-12T16:29:08.734174+0000","last_became_peered":"2024-02-12T16:29:08.734174+0000","last_unstale":"2024-02-12T16:29:40.244538+0000","last_undegraded":"2024-02-12T16:29:40.244538+0000","last_fullsized":"2024-02-12T16:29:40.244538+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2775,"ondisk_log_size":2775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":181862400,"num_objects":2775,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2775,"num_write_kb":177600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":165,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.118367+0000","last_change":"2024-02-12T16:28:51.703479+0000","last_active":"2024-02-12T16:29:30.118367+0000","last_peered":"2024-02-12T16:29:30.118367+0000","last_clean":"2024-02-12T16:29:30.118367+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:29:30.118367+0000","last_undegraded":"2024-02-12T16:29:30.118367+0000","last_fullsized":"2024-02-12T16:29:30.118367+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:51.703298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:51.703298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"647'2319","reported_seq":2337,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:38.289004+0000","last_change":"2024-02-12T16:29:08.734976+0000","last_active":"2024-02-12T16:29:38.289004+0000","last_peered":"2024-02-12T16:29:38.289004+0000","last_clean":"2024-02-12T16:29:38.289004+0000","last_became_active":"2024-02-12T16:29:08.734560+0000","last_became_peered":"2024-02-12T16:29:08.734560+0000","last_unstale":"2024-02-12T16:29:38.289004+0000","last_undegraded":"2024-02-12T16:29:38.289004+0000","last_fullsized":"2024-02-12T16:29:38.289004+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2319,"ondisk_log_size":2319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":151977984,"num_objects":2319,"num_object_clones":0,"num_object_copies":4638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2319,"num_write_kb":148416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":338,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.121844+0000","last_change":"2024-02-12T16:28:55.989602+0000","last_active":"2024-02-12T16:29:30.121844+0000","last_peered":"2024-02-12T16:29:30.121844+0000","last_clean":"2024-02-12T16:29:30.121844+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:29:30.121844+0000","last_undegraded":"2024-02-12T16:29:30.121844+0000","last_fullsized":"2024-02-12T16:29:30.121844+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:55.989476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:28:55.989476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"647'2839","reported_seq":2857,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:40.468470+0000","last_change":"2024-02-12T16:29:08.807541+0000","last_active":"2024-02-12T16:29:40.468470+0000","last_peered":"2024-02-12T16:29:40.468470+0000","last_clean":"2024-02-12T16:29:40.468470+0000","last_became_active":"2024-02-12T16:29:08.807333+0000","last_became_peered":"2024-02-12T16:29:08.807333+0000","last_unstale":"2024-02-12T16:29:40.468470+0000","last_undegraded":"2024-02-12T16:29:40.468470+0000","last_fullsized":"2024-02-12T16:29:40.468470+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2839,"ondisk_log_size":2839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":186056704,"num_objects":2839,"num_object_clones":0,"num_object_copies":5678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2839,"num_write_kb":181696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":140,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.117789+0000","last_change":"2024-02-12T16:28:54.613666+0000","last_active":"2024-02-12T16:29:30.117789+0000","last_peered":"2024-02-12T16:29:30.117789+0000","last_clean":"2024-02-12T16:29:30.117789+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:29:30.117789+0000","last_undegraded":"2024-02-12T16:29:30.117789+0000","last_fullsized":"2024-02-12T16:29:30.117789+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.613501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.613501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"647'2638","reported_seq":2655,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:39.356181+0000","last_change":"2024-02-12T16:29:08.736225+0000","last_active":"2024-02-12T16:29:39.356181+0000","last_peered":"2024-02-12T16:29:39.356181+0000","last_clean":"2024-02-12T16:29:39.356181+0000","last_became_active":"2024-02-12T16:29:08.735885+0000","last_became_peered":"2024-02-12T16:29:08.735885+0000","last_unstale":"2024-02-12T16:29:39.356181+0000","last_undegraded":"2024-02-12T16:29:39.356181+0000","last_fullsized":"2024-02-12T16:29:39.356181+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2638,"ondisk_log_size":2638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":172883968,"num_objects":2638,"num_object_clones":0,"num_object_copies":5276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2638,"num_write_kb":168832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":740,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.122458+0000","last_change":"2024-02-12T16:28:58.797711+0000","last_active":"2024-02-12T16:29:30.122458+0000","last_peered":"2024-02-12T16:29:30.122458+0000","last_clean":"2024-02-12T16:29:30.122458+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:29:30.122458+0000","last_undegraded":"2024-02-12T16:29:30.122458+0000","last_fullsized":"2024-02-12T16:29:30.122458+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.797578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.797578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"646'2094","reported_seq":2111,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:36.416909+0000","last_change":"2024-02-12T16:29:08.834927+0000","last_active":"2024-02-12T16:29:36.416909+0000","last_peered":"2024-02-12T16:29:36.416909+0000","last_clean":"2024-02-12T16:29:36.416909+0000","last_became_active":"2024-02-12T16:29:08.834297+0000","last_became_peered":"2024-02-12T16:29:08.834297+0000","last_unstale":"2024-02-12T16:29:36.416909+0000","last_undegraded":"2024-02-12T16:29:36.416909+0000","last_fullsized":"2024-02-12T16:29:36.416909+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2094,"ondisk_log_size":2094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":137232384,"num_objects":2094,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2094,"num_write_kb":134016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":326,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-12T16:29:27.744015+0000","last_change":"2024-02-12T16:28:53.361716+0000","last_active":"2024-02-12T16:29:27.744015+0000","last_peered":"2024-02-12T16:29:27.744015+0000","last_clean":"2024-02-12T16:29:27.744015+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:29:27.744015+0000","last_undegraded":"2024-02-12T16:29:27.744015+0000","last_fullsized":"2024-02-12T16:29:27.744015+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.361610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.361610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"647'2824","reported_seq":2841,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-02-12T16:29:40.238667+0000","last_change":"2024-02-12T16:29:08.805152+0000","last_active":"2024-02-12T16:29:40.238667+0000","last_peered":"2024-02-12T16:29:40.238667+0000","last_clean":"2024-02-12T16:29:40.238667+0000","last_became_active":"2024-02-12T16:29:08.804528+0000","last_became_peered":"2024-02-12T16:29:08.804528+0000","last_unstale":"2024-02-12T16:29:40.238667+0000","last_undegraded":"2024-02-12T16:29:40.238667+0000","last_fullsized":"2024-02-12T16:29:40.238667+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2824,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":185073664,"num_objects":2824,"num_object_clones":0,"num_object_copies":5648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":180736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":79,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.195070+0000","last_change":"2024-02-12T16:28:59.764005+0000","last_active":"2024-02-12T16:29:30.195070+0000","last_peered":"2024-02-12T16:29:30.195070+0000","last_clean":"2024-02-12T16:29:30.195070+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:29:30.195070+0000","last_undegraded":"2024-02-12T16:29:30.195070+0000","last_fullsized":"2024-02-12T16:29:30.195070+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:59.763866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:59.763866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"646'2138","reported_seq":2155,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:37.100409+0000","last_change":"2024-02-12T16:29:08.735934+0000","last_active":"2024-02-12T16:29:37.100409+0000","last_peered":"2024-02-12T16:29:37.100409+0000","last_clean":"2024-02-12T16:29:37.100409+0000","last_became_active":"2024-02-12T16:29:08.735711+0000","last_became_peered":"2024-02-12T16:29:08.735711+0000","last_unstale":"2024-02-12T16:29:37.100409+0000","last_undegraded":"2024-02-12T16:29:37.100409+0000","last_fullsized":"2024-02-12T16:29:37.100409+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":2138,"ondisk_log_size":2138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":140115968,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2138,"num_write_kb":136832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":314,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-12T16:29:26.896375+0000","last_change":"2024-02-12T16:28:54.979306+0000","last_active":"2024-02-12T16:29:26.896375+0000","last_peered":"2024-02-12T16:29:26.896375+0000","last_clean":"2024-02-12T16:29:26.896375+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:29:26.896375+0000","last_undegraded":"2024-02-12T16:29:26.896375+0000","last_fullsized":"2024-02-12T16:29:26.896375+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.979211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.979211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":135,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-12T16:29:27.743951+0000","last_change":"2024-02-12T16:28:59.370513+0000","last_active":"2024-02-12T16:29:27.743951+0000","last_peered":"2024-02-12T16:29:27.743951+0000","last_clean":"2024-02-12T16:29:27.743951+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:29:27.743951+0000","last_undegraded":"2024-02-12T16:29:27.743951+0000","last_fullsized":"2024-02-12T16:29:27.743951+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:59.370403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:28:59.370403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":135,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-02-12T16:29:30.125511+0000","last_change":"2024-02-12T16:28:55.212658+0000","last_active":"2024-02-12T16:29:30.125511+0000","last_peered":"2024-02-12T16:29:30.125511+0000","last_clean":"2024-02-12T16:29:30.125511+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:29:30.125511+0000","last_undegraded":"2024-02-12T16:29:30.125511+0000","last_fullsized":"2024-02-12T16:29:30.125511+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:55.212566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:55.212566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":149,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-12T16:29:27.743924+0000","last_change":"2024-02-12T16:28:56.394900+0000","last_active":"2024-02-12T16:29:27.743924+0000","last_peered":"2024-02-12T16:29:27.743924+0000","last_clean":"2024-02-12T16:29:27.743924+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:29:27.743924+0000","last_undegraded":"2024-02-12T16:29:27.743924+0000","last_fullsized":"2024-02-12T16:29:27.743924+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:56.394801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:28:56.394801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":128,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-02-12T16:29:28.264284+0000","last_change":"2024-02-12T16:28:58.060344+0000","last_active":"2024-02-12T16:29:28.264284+0000","last_peered":"2024-02-12T16:29:28.264284+0000","last_clean":"2024-02-12T16:29:28.264284+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:29:28.264284+0000","last_undegraded":"2024-02-12T16:29:28.264284+0000","last_fullsized":"2024-02-12T16:29:28.264284+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.060250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.060250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2598305792,"num_objects":39647,"num_object_clones":0,"num_object_copies":79294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39647,"num_write_kb":2537408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5206638592,"data_stored":5206638592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39647,"ondisk_log_size":39647,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586532,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":93158620,"statfs":{"total":95995035648,"available":95394426880,"internally_reserved":0,"allocated":946176,"data_stored":474984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.80500000000000005,"15min":0.85499999999999998},"min":{"1min":0.315,"5min":0.246,"15min":0.246},"max":{"1min":1.5029999999999999,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.315},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.94799999999999995,"15min":0.90900000000000003},"min":{"1min":0.39000000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.395,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":1.395}]},{"osd":1,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.93400000000000005,"15min":1.0620000000000001},"min":{"1min":0.30099999999999999,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":1.385,"5min":5.5430000000000001,"15min":7.0110000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.89000000000000001,"15min":1.0640000000000001},"min":{"1min":0.36499999999999999,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":1.4830000000000001,"5min":3.6859999999999999,"15min":5.6920000000000002},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:29:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.83699999999999997,"15min":1.2450000000000001},"min":{"1min":0.46899999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.4379999999999999,"5min":4.1639999999999997,"15min":5.343},"last":1.6859999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.81699999999999995,"15min":1.3},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.268,"5min":5.4340000000000002,"15min":8.2759999999999998},"last":1.7789999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.874,"15min":1.034},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":1.8009999999999999,"5min":4.9900000000000002,"15min":6.5970000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.86799999999999999,"15min":0.93500000000000005},"min":{"1min":0.371,"5min":0.36799999999999999,"15min":0.34499999999999997},"max":{"1min":1.7350000000000001,"5min":4.101,"15min":4.915},"last":0.371}]},{"osd":5,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.125,"15min":1.1499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.35099999999999998},"max":{"1min":1.7769999999999999,"5min":5.04,"15min":5.8789999999999996},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.024,"15min":1.1319999999999999},"min":{"1min":0.498,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":1.794,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":0.78200000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.90400000000000003,"15min":0.98799999999999999},"min":{"1min":0.45200000000000001,"5min":0.378,"15min":0.373},"max":{"1min":1.522,"5min":4.7370000000000001,"15min":5.141},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.026,"15min":1.125},"min":{"1min":0.54100000000000004,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.905,"5min":4.7450000000000001,"15min":8.5709999999999997},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.91200000000000003,"15min":1.1859999999999999},"min":{"1min":0.46899999999999997,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.718,"5min":3.9089999999999998,"15min":6.4370000000000003},"last":0.495},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.84999999999999998,"15min":1.1160000000000001},"min":{"1min":0.52700000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.867,"5min":4.2350000000000003,"15min":5.0430000000000001},"last":0.52700000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":6.2039999999999997,"15min":2.7970000000000002},"min":{"1min":0.45600000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.625,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.27,"5min":6.319,"15min":2.9689999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.34699999999999998},"max":{"1min":1.722,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.443}]},{"osd":9,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98099999999999998,"15min":1.276},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.736,"5min":3.8100000000000001,"15min":10.743},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.397,"5min":0.998,"15min":1.357},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7809999999999999,"5min":3.8399999999999999,"15min":10.698},"last":0.46700000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":6.327,"15min":2.9590000000000001},"min":{"1min":0.50900000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.6850000000000001,"5min":536.245,"15min":536.245},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":6.3339999999999996,"15min":2.9399999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.901,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.432}]}]},{"osd":10,"up_from":83,"seq":356482285955,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467684,"kb_used_data":158172,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93277468,"statfs":{"total":95995035648,"available":95516127232,"internally_reserved":0,"allocated":161968128,"data_stored":161496936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":2.778,"15min":1.55},"min":{"1min":0.33400000000000002,"5min":0.245,"15min":0.245},"max":{"1min":15.010999999999999,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.8109999999999999,"5min":2.8700000000000001,"15min":1.6339999999999999},"min":{"1min":0.34000000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":14.129,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0640000000000001,"15min":1.0069999999999999},"min":{"1min":0.40699999999999997,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":2.5070000000000001,"5min":5.9000000000000004,"15min":19.866},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.05,"15min":1.1479999999999999},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":4.673,"5min":6.1970000000000001,"15min":20.277999999999999},"last":0.372}]},{"osd":2,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2230000000000001,"15min":1.0169999999999999},"min":{"1min":0.34999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8580000000000001,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":0.316},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.1359999999999999,"15min":1.0249999999999999},"min":{"1min":0.38900000000000001,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":2.7549999999999999,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":1.244}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":0.92100000000000004,"15min":3.3599999999999999},"min":{"1min":0.41699999999999998,"5min":0.40400000000000003,"15min":0.32300000000000001},"max":{"1min":3.5489999999999999,"5min":5.8109999999999999,"15min":628.77099999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.999,"15min":3.3300000000000001},"min":{"1min":0.47499999999999998,"5min":0.33500000000000002,"15min":0.30199999999999999},"max":{"1min":2.681,"5min":7.3070000000000004,"15min":624.19299999999998},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.1459999999999999,"15min":1.0800000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.8109999999999999,"5min":6.343,"15min":6.343},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.151,"15min":1.169},"min":{"1min":0.55300000000000005,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":4.7610000000000001,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":0.56999999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.036,"15min":3.391},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.8799999999999999,"5min":6.3010000000000002,"15min":626.26599999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.24,"5min":1.056,"15min":3.492},"min":{"1min":0.49199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":2.7730000000000001,"5min":6.8200000000000003,"15min":625.90700000000004},"last":0.443}]},{"osd":7,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":0.99199999999999999,"15min":1.091},"min":{"1min":0.432,"5min":0.375,"15min":0.375},"max":{"1min":2.5640000000000001,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.202,"15min":1.2130000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.3519999999999999,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.41199999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.0649999999999999,"15min":1.157},"min":{"1min":0.51900000000000002,"5min":0.436,"15min":0.372},"max":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.135,"5min":1.038,"15min":1.151},"min":{"1min":0.46300000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":2.6200000000000001,"5min":7.282,"15min":7.282},"last":0.42499999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:29:01 2024","interfaces":[{"interface":"back","average":{"1min":1.6220000000000001,"5min":1.218,"15min":1.2689999999999999},"min":{"1min":0.64900000000000002,"5min":0.48499999999999999,"15min":0.372},"max":{"1min":6.6239999999999997,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.046,"15min":1.234},"min":{"1min":0.53600000000000003,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":2.593,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.52100000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:29:30 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.1060000000000001,"15min":1.2809999999999999},"min":{"1min":0.67400000000000004,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":2.1219999999999999,"5min":7.3689999999999998,"15min":10.218999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.552,"5min":1.115,"15min":1.296},"min":{"1min":0.63900000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.2189999999999999,"5min":6.5860000000000003,"15min":7.3250000000000002},"last":0.497}]}]},{"osd":9,"up_from":74,"seq":317827580299,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560536,"kb_used_data":255120,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93184616,"statfs":{"total":95995035648,"available":95421046784,"internally_reserved":0,"allocated":261242880,"data_stored":260777706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2350000000000001,"15min":1.1160000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.77,"5min":7.9560000000000004,"15min":7.9560000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.234,"15min":1.0109999999999999},"min":{"1min":0.40500000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.3679999999999999,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1160000000000001,"15min":1.022},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.253},"max":{"1min":3.8180000000000001,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0600000000000001,"15min":1.079},"min":{"1min":0.32500000000000001,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":2.2789999999999999,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2829999999999999,"15min":1.4770000000000001},"min":{"1min":0.436,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":3.8959999999999999,"5min":12.441000000000001,"15min":97.275000000000006},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.198,"15min":1.363},"min":{"1min":0.42999999999999999,"5min":0.314,"15min":0.312},"max":{"1min":4.0039999999999996,"5min":6.1779999999999999,"15min":97.417000000000002},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.056,"15min":1.016},"min":{"1min":0.378,"5min":0.33300000000000002,"15min":0.32200000000000001},"max":{"1min":6.3090000000000002,"5min":6.3090000000000002,"15min":6.3090000000000002},"last":1.512},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.153,"15min":1.194},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":6.3490000000000002,"5min":6.3490000000000002,"15min":6.3490000000000002},"last":1.54}]},{"osd":5,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.325,"15min":1.175},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":5.3410000000000002,"5min":6.726,"15min":6.726},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.298,"5min":1.228,"15min":1.173},"min":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":5.3520000000000003,"5min":6.7350000000000003,"15min":7.3739999999999997},"last":0.67500000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.997,"15min":1.101},"min":{"1min":0.38400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":2.4020000000000001,"5min":4.6749999999999998,"15min":7.3360000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.2270000000000001,"15min":1.125},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.36499999999999999},"max":{"1min":14.718,"5min":14.718,"15min":14.718},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.171,"15min":1.2669999999999999},"min":{"1min":0.45400000000000001,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":3.9830000000000001,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1910000000000001,"15min":1.23},"min":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.4390000000000001,"5min":4.7000000000000002,"15min":7.9160000000000004},"last":0.57899999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:28:58 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.99399999999999999,"15min":1.0860000000000001},"min":{"1min":0.46700000000000003,"5min":0.36399999999999999,"15min":0.35099999999999998},"max":{"1min":2.2290000000000001,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.248,"5min":1.052,"15min":1.2070000000000001},"min":{"1min":0.49399999999999999,"5min":0.437,"15min":0.42599999999999999},"max":{"1min":6.1980000000000004,"5min":6.1980000000000004,"15min":7.25},"last":0.65900000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.2,"15min":1.2549999999999999},"min":{"1min":0.57099999999999995,"5min":0.498,"15min":0.36599999999999999},"max":{"1min":4.6900000000000004,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.643,"5min":1.244,"15min":1.266},"min":{"1min":0.55800000000000005,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":0.499}]},{"osd":11,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.1579999999999999,"15min":1.351},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.45600000000000002},"max":{"1min":4.7990000000000004,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.288,"5min":1.073,"15min":1.2869999999999999},"min":{"1min":0.50700000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.7190000000000003,"5min":4.7190000000000003,"15min":6.3639999999999999},"last":0.55500000000000005}]}]},{"osd":8,"up_from":65,"seq":279172874645,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":610084,"kb_used_data":308764,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93135068,"statfs":{"total":95995035648,"available":95370309632,"internally_reserved":0,"allocated":316174336,"data_stored":315703152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.077,"15min":0.90800000000000003},"min":{"1min":0.63500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":2.71,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.2430000000000001,"15min":0.93200000000000005},"min":{"1min":0.63800000000000001,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":2.5219999999999998,"5min":18.584,"15min":18.584},"last":0.497}]},{"osd":1,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.9810000000000001,"15min":1.5249999999999999},"min":{"1min":0.72199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.181,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.9670000000000001,"15min":1.6220000000000001},"min":{"1min":0.65700000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.3919999999999999,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.443}]},{"osd":2,"last update":"Mon Feb 12 16:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.85099999999999998,"15min":0.84399999999999997},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.903,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.012,"15min":1.018},"min":{"1min":0.40300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.754,"15min":1.708},"min":{"1min":0.46300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.458,"5min":5.0490000000000004,"15min":225.33099999999999},"last":1.556},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.84999999999999998,"15min":1.681},"min":{"1min":0.57899999999999996,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":2.0110000000000001,"5min":5.0380000000000003,"15min":224.71000000000001},"last":1.5720000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.88400000000000001,"15min":0.92000000000000004},"min":{"1min":0.51700000000000002,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":1.4259999999999999,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.345,"5min":0.95199999999999996,"15min":1.0229999999999999},"min":{"1min":0.53200000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.097,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.61499999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.0409999999999999,"15min":1.758},"min":{"1min":0.55100000000000005,"5min":0.373,"15min":0.37},"max":{"1min":1.5609999999999999,"5min":5.3639999999999999,"15min":221.36699999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.0189999999999999,"15min":1.8600000000000001},"min":{"1min":0.625,"5min":0.38,"15min":0.38},"max":{"1min":2.2269999999999999,"5min":4.4930000000000003,"15min":221.203},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.0629999999999999,"15min":1.1299999999999999},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":1.742,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.119,"15min":1.2509999999999999},"min":{"1min":0.64800000000000002,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":2.1709999999999998,"5min":4.2240000000000002,"15min":7.899},"last":0.69499999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.028,"15min":1.167},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.7030000000000001,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.351,"5min":1.1160000000000001,"15min":1.2490000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":1.9299999999999999,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":0.65600000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.97199999999999998,"15min":1.4399999999999999},"min":{"1min":0.437,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.6120000000000001,"5min":5.2560000000000002,"15min":109.92},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.999,"15min":1.484},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.38500000000000001},"max":{"1min":1.786,"5min":5.0250000000000004,"15min":110.249},"last":0.55200000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.028,"15min":1.1830000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":5.2530000000000001,"5min":5.2530000000000001,"15min":5.2530000000000001},"last":0.749},{"interface":"front","average":{"1min":1.135,"5min":0.93300000000000005,"15min":1.143},"min":{"1min":0.50700000000000001,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":0.57999999999999996}]}]},{"osd":7,"up_from":56,"seq":240518168988,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888664,"kb_used_data":575056,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":92856488,"statfs":{"total":95995035648,"available":95085043712,"internally_reserved":0,"allocated":588857344,"data_stored":588392170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":3.1120000000000001,"15min":1.673},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.8310000000000004,"5min":187.33699999999999,"15min":187.33699999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":3.0539999999999998,"15min":1.6619999999999999},"min":{"1min":0.32100000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.4580000000000002,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.872}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.302,"15min":1.056},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4729999999999999,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.6060000000000001,"15min":1.2130000000000001},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.6840000000000002,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78100000000000003,"15min":1.159},"min":{"1min":0.379,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.238,"5min":4.5140000000000002,"15min":7.1689999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86199999999999999,"15min":0.97399999999999998},"min":{"1min":0.39700000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.3759999999999999,"5min":6.7549999999999999,"15min":6.7549999999999999},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.145,"15min":0.95699999999999996},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.0049999999999999,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.95999999999999996,"15min":1.0029999999999999},"min":{"1min":0.29799999999999999,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":6.9470000000000001,"5min":6.9470000000000001,"15min":6.9470000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.97799999999999998,"15min":0.95099999999999996},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9379999999999997,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.0600000000000001,"15min":1.048},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.8840000000000003,"5min":5.992,"15min":9.3640000000000008},"last":0.59699999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.0449999999999999,"15min":1.081},"min":{"1min":0.38800000000000001,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":4.7409999999999997,"5min":6.319,"15min":6.319},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.173,"15min":1.1890000000000001},"min":{"1min":0.39300000000000002,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":0.97499999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":6.5270000000000001,"15min":2.9329999999999998},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.6719999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.5900000000000001,"5min":6.641,"15min":3.0590000000000002},"min":{"1min":0.434,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.5359999999999996,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.67200000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:29:04 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.006,"15min":1.181},"min":{"1min":0.51600000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":12.907999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.484,"5min":1.1950000000000001,"15min":1.2949999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":7.1219999999999999,"5min":7.1219999999999999,"15min":7.1219999999999999},"last":0.81399999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:28:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":7.7859999999999996,"15min":3.5179999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":7.0339999999999998,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.6990000000000001,"5min":7.6740000000000004,"15min":3.423},"min":{"1min":0.50600000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.5890000000000004,"5min":496.29599999999999,"15min":496.29599999999999},"last":0.81999999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0409999999999999,"15min":1.5940000000000001},"min":{"1min":0.626,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.847,"5min":4.9039999999999999,"15min":120.943},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.329,"5min":1.2410000000000001,"15min":1.708},"min":{"1min":0.56999999999999995,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":4.181,"5min":7.0679999999999996,"15min":120.661},"last":0.89600000000000002}]}]},{"osd":6,"up_from":48,"seq":206158430629,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994404,"kb_used_data":688988,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92750748,"statfs":{"total":95995035648,"available":94976765952,"internally_reserved":0,"allocated":705523712,"data_stored":705052528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":2.1219999999999999,"15min":1.4730000000000001},"min":{"1min":0.309,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.7889999999999997,"5min":107.723,"15min":107.723},"last":1.02},{"interface":"front","average":{"1min":0.99299999999999999,"5min":2.1680000000000001,"15min":1.544},"min":{"1min":0.27700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":2.9940000000000002,"5min":107.114,"15min":107.114},"last":0.98299999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.0109999999999999,"15min":1.9139999999999999},"min":{"1min":0.29199999999999998,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":1.7,"5min":5.5179999999999998,"15min":252.22800000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.90900000000000003,"15min":1.956},"min":{"1min":0.29199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.2410000000000001,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.464,"15min":1.0640000000000001},"min":{"1min":0.35499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.0089999999999999,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.317},{"interface":"front","average":{"1min":1.224,"5min":1.6739999999999999,"15min":1.1870000000000001},"min":{"1min":0.39500000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.3929999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.115,"15min":1.0309999999999999},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.6299999999999999,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.90500000000000003,"15min":1.0629999999999999},"min":{"1min":0.34399999999999997,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.5469999999999999,"5min":6.0259999999999998,"15min":7.5979999999999999},"last":2.0640000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.071,"15min":0.98899999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":2.9279999999999999,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.96699999999999997,"15min":1.0029999999999999},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.712,"5min":10.52,"15min":10.52},"last":0.88900000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.97299999999999998,"15min":1.089},"min":{"1min":0.34200000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.399,"5min":4.7149999999999999,"15min":7.641},"last":0.37},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.81000000000000005,"15min":1.034},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.679,"5min":4.5609999999999999,"15min":7.8380000000000001},"last":0.95099999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.139,"15min":1.139},"min":{"1min":0.436,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.21,"5min":6.21,"15min":6.21},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.98599999999999999,"15min":1.1859999999999999},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":6.2270000000000003,"5min":6.2270000000000003,"15min":6.2270000000000003},"last":0.77900000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.246},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.8089999999999999,"5min":4.1680000000000001,"15min":18.059999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.96099999999999997,"15min":1.2470000000000001},"min":{"1min":0.45300000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.9970000000000001,"5min":2.673,"15min":17.425000000000001},"last":1.0900000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.24,"15min":1.2909999999999999},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.7400000000000002,"5min":5.726,"15min":6.5300000000000002},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.423,"5min":1.337,"15min":1.3839999999999999},"min":{"1min":0.57699999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.2080000000000002,"5min":5.7590000000000003,"15min":6.5730000000000004},"last":0.91500000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.2290000000000001,"15min":2.48},"min":{"1min":0.54600000000000004,"5min":0.52100000000000002,"15min":0.499},"max":{"1min":2.7970000000000002,"5min":5.6859999999999999,"15min":139.727},"last":0.876},{"interface":"front","average":{"1min":0.995,"5min":0.98899999999999999,"15min":2.3559999999999999},"min":{"1min":0.55400000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.173,"5min":2.706,"15min":139.24199999999999},"last":1.127}]}]},{"osd":1,"up_from":13,"seq":55834575306,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1273944,"kb_used_data":371536,"kb_used_omap":0,"kb_used_meta":902400,"kb_avail":92471208,"statfs":{"total":95995035648,"available":94690516992,"internally_reserved":0,"allocated":380452864,"data_stored":379987690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":924057600},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.129,"15min":0.96099999999999997},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":4.4610000000000003,"5min":16.632999999999999,"15min":16.632999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.645,"15min":1.212},"min":{"1min":0.33200000000000002,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":6.4939999999999998,"5min":7.7249999999999996,"15min":11.433},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.3580000000000001,"15min":1.1339999999999999},"min":{"1min":0.52600000000000002,"5min":0.29599999999999999,"15min":0.248},"max":{"1min":1.552,"5min":12.778,"15min":12.778},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.5860000000000001,"15min":1.2190000000000001},"min":{"1min":0.55300000000000005,"5min":0.246,"15min":0.246},"max":{"1min":2.4169999999999998,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.42299999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:29:14 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.1419999999999999,"15min":1.137},"min":{"1min":0.47999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":4.1340000000000003,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":1.5089999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.208,"15min":0.95999999999999996},"min":{"1min":0.42599999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.8420000000000001,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":1.819}]},{"osd":4,"last update":"Mon Feb 12 16:29:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.3049999999999999,"15min":1.133},"min":{"1min":0.47199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":2.056,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.369,"15min":1.373},"min":{"1min":0.436,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.78,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.38300000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.514,"15min":1.282},"min":{"1min":0.67600000000000005,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.8169999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":1.244},{"interface":"front","average":{"1min":1.403,"5min":1.3680000000000001,"15min":1.2310000000000001},"min":{"1min":0.79100000000000004,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":2.3159999999999998,"5min":10.067,"15min":10.067},"last":2.8290000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.6799999999999999,"15min":1.7849999999999999},"min":{"1min":0.503,"5min":0.439,"15min":0.36399999999999999},"max":{"1min":3.5,"5min":30.437000000000001,"15min":55.869999999999997},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.599,"15min":1.8540000000000001},"min":{"1min":0.47299999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":4.1680000000000001,"5min":29.382000000000001,"15min":55.892000000000003},"last":1.5980000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":7.4459999999999997,"15min":3.4529999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.2300000000000004,"5min":548.91600000000005,"15min":548.91600000000005},"last":1.633},{"interface":"front","average":{"1min":1.0680000000000001,"5min":7.5579999999999998,"15min":3.4359999999999999},"min":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.3769999999999998,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.51500000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.425,"15min":1.4139999999999999},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.4009999999999998,"5min":5.7590000000000003,"15min":10.07},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.3380000000000001,"15min":1.466},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":4.2919999999999998,"5min":10.106,"15min":10.106},"last":0.55300000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":8.4459999999999997,"15min":3.7690000000000001},"min":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.105,"5min":548.93499999999995,"15min":548.93499999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.6919999999999999,"5min":8.4209999999999994,"15min":3.8460000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1989999999999998,"5min":549.12099999999998,"15min":549.12099999999998},"last":1.5580000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.538,"15min":1.5920000000000001},"min":{"1min":1.159,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4689999999999999,"5min":10.202,"15min":10.202},"last":1.667},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.597,"15min":1.6399999999999999},"min":{"1min":0.98399999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.387,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":0.63600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738838,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":954916,"kb_used_data":357020,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":92790236,"statfs":{"total":95995035648,"available":95017201664,"internally_reserved":0,"allocated":365588480,"data_stored":365117296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.1830000000000001,"15min":1.0800000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":3.6890000000000001,"5min":10.754,"15min":11.324},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.27,"5min":1.401,"15min":1.179},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1020000000000003,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:28:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0600000000000001,"15min":1.056},"min":{"1min":0.26500000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.032,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.244,"5min":1.1579999999999999,"15min":1.002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":3.8050000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.90100000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":3.5310000000000001,"15min":2.1269999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":5.4729999999999999,"5min":193.77000000000001,"15min":193.77000000000001},"last":0.436},{"interface":"front","average":{"1min":1.5469999999999999,"5min":3.4790000000000001,"15min":2.1909999999999998},"min":{"1min":0.29499999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":5.141,"5min":189.625,"15min":189.625},"last":0.48399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.3149999999999999,"15min":1.0589999999999999},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.236,"5min":10.49,"15min":10.49},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.304,"15min":1.083},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":1.369,"5min":7.3250000000000002,"15min":7.3250000000000002},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":3.145,"15min":2.0019999999999998},"min":{"1min":0.374,"5min":0.311,"15min":0.311},"max":{"1min":7.3600000000000003,"5min":189.79400000000001,"15min":189.79400000000001},"last":0.5},{"interface":"front","average":{"1min":1.367,"5min":3.2389999999999999,"15min":2.0790000000000002},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":7.2930000000000001,"5min":194.41200000000001,"15min":194.41200000000001},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":3.5750000000000002,"15min":1.9450000000000001},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":7.4509999999999996,"5min":191.47200000000001,"15min":191.47200000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.6319999999999999,"5min":3.6760000000000002,"15min":2.052},"min":{"1min":0.47399999999999998,"5min":0.44,"15min":0.44},"max":{"1min":7.4779999999999998,"5min":191.50299999999999,"15min":191.50299999999999},"last":0.56799999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:29:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3360000000000001,"15min":1.298},"min":{"1min":0.56699999999999995,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":1.6519999999999999,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4379999999999999,"15min":1.3919999999999999},"min":{"1min":0.68600000000000005,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.996,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.71099999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.274,"15min":1.371},"min":{"1min":0.44800000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.7130000000000001,"5min":7.6539999999999999,"15min":46.246000000000002},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.397,"15min":1.446},"min":{"1min":0.64500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":7.5369999999999999,"5min":7.5369999999999999,"15min":46.167999999999999},"last":0.64000000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:28:54 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.52,"15min":1.377},"min":{"1min":0.45100000000000001,"5min":0.442,"15min":0.41399999999999998},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.753},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.6459999999999999,"15min":1.4079999999999999},"min":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":9.3490000000000002,"5min":9.3490000000000002,"15min":9.3490000000000002},"last":0.79200000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:29:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.518,"15min":1.569},"min":{"1min":0.69799999999999995,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":2.0819999999999999,"5min":6.2830000000000004,"15min":46.064},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4410000000000001,"15min":1.5669999999999999},"min":{"1min":0.67500000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.9710000000000001,"5min":7.4210000000000003,"15min":45.642000000000003},"last":0.745}]}]},{"osd":2,"up_from":21,"seq":90194313667,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421400,"kb_used_data":124176,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93323752,"statfs":{"total":95995035648,"available":95563522048,"internally_reserved":0,"allocated":127156224,"data_stored":126691050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.3660000000000001,"15min":1.107},"min":{"1min":0.36199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1160000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.4059999999999999,"15min":1.222},"min":{"1min":0.40300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":1.4399999999999999,"5min":6.484,"15min":9.6470000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.272,"15min":1.841},"min":{"1min":0.25700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":2.7709999999999999,"5min":6.4509999999999996,"15min":21.407},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.109,"15min":0.92300000000000004},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.407,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.393,"15min":1.079},"min":{"1min":0.33500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0019999999999998,"5min":10.890000000000001,"15min":10.890000000000001},"last":1.4870000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":1.419,"15min":1.1839999999999999},"min":{"1min":0.33600000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.6060000000000001,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":1.5009999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.3080000000000001,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":1.4570000000000001,"5min":6.133,"15min":6.133},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.05,"5min":1.6140000000000001,"15min":1.214},"min":{"1min":0.55800000000000005,"5min":0.315,"15min":0.315},"max":{"1min":1.4370000000000001,"5min":11.081,"15min":11.081},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:28:59 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.429,"15min":1.294},"min":{"1min":0.378,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4160000000000004,"5min":7.6890000000000001,"15min":21.527000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.4179999999999999,"15min":1.2230000000000001},"min":{"1min":0.44500000000000001,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":2.6200000000000001,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":1.859}]},{"osd":7,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.6359999999999999,"15min":1.234},"min":{"1min":0.39500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.1260000000000003,"5min":11.044,"15min":11.044},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.2629999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.8739999999999997,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":0.79600000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4350000000000001,"15min":1.306},"min":{"1min":0.47799999999999998,"5min":0.42999999999999999,"15min":0.42099999999999999},"max":{"1min":4.8360000000000003,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4330000000000001,"15min":1.3420000000000001},"min":{"1min":0.47399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":4.7389999999999999,"5min":6.593,"15min":9.7639999999999993},"last":0.629}]},{"osd":9,"last update":"Mon Feb 12 16:28:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.607,"15min":1.3500000000000001},"min":{"1min":0.51500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":6.0910000000000002,"5min":10.972,"15min":10.972},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.71,"15min":1.429},"min":{"1min":0.45600000000000002,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":6.1600000000000001,"5min":11.006,"15min":11.006},"last":0.56000000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.609,"15min":1.462},"min":{"1min":0.72899999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.024,"5min":7.7539999999999996,"15min":11.454000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.605,"5min":1.6739999999999999,"15min":1.5009999999999999},"min":{"1min":0.76000000000000001,"5min":0.441,"15min":0.441},"max":{"1min":2.2280000000000002,"5min":7.7140000000000004,"15min":9.7170000000000005},"last":0.66200000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:28:59 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4770000000000001,"15min":1.4970000000000001},"min":{"1min":0.64400000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":7.258,"5min":7.258,"15min":7.258},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.581,"15min":1.5800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":0.71899999999999997}]}]},{"osd":3,"up_from":522,"seq":2241972928593,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1361252,"kb_used_data":1024924,"kb_used_omap":91,"kb_used_meta":336228,"kb_avail":92383900,"statfs":{"total":95995035648,"available":94601113600,"internally_reserved":0,"allocated":1049522176,"data_stored":1049050992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":344297488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.3280000000000001,"15min":1.0069999999999999},"min":{"1min":0.27000000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.5529999999999999,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.4570000000000001,"15min":1.054},"min":{"1min":0.373,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.536,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.2230000000000001,"15min":0.98699999999999999},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.597,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":0.316},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.175,"15min":1.095},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.9510000000000001,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":0.433}]},{"osd":4,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.78200000000000003,"15min":1.212},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.27,"5min":4.7080000000000002,"15min":10.004},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.052,"15min":1.105},"min":{"1min":0.45000000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.7350000000000003,"5min":6.806,"15min":6.806},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":1.294},"min":{"1min":0.41699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.9219999999999999,"5min":4.7530000000000001,"15min":6.7539999999999996},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.113,"5min":0.95099999999999996,"15min":0.94899999999999995},"min":{"1min":0.34599999999999997,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":3.5880000000000001,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.84999999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0600000000000001,"15min":1.1259999999999999},"min":{"1min":0.40100000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.778,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.996,"15min":1.01},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.1509999999999998,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.012,"15min":1.3759999999999999},"min":{"1min":0.41599999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.4630000000000001,"5min":4.8090000000000002,"15min":6.851},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1950000000000001,"15min":1.2689999999999999},"min":{"1min":0.52900000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.8330000000000002,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":0.58299999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.0529999999999999,"15min":1.1759999999999999},"min":{"1min":0.49299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.2280000000000002,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.999,"15min":1.099},"min":{"1min":0.44500000000000001,"5min":0.42699999999999999,"15min":0.39800000000000002},"max":{"1min":2.3650000000000002,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.91300000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.284,"15min":1.329},"min":{"1min":0.57199999999999995,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.7050000000000001,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3129999999999999,"15min":1.373},"min":{"1min":0.56200000000000006,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.194,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.86499999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.147,"15min":1.4530000000000001},"min":{"1min":0.60299999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.016,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.298,"15min":1.3220000000000001},"min":{"1min":0.61099999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.7330000000000001,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":15.140000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2450000000000001,"15min":1.2849999999999999},"min":{"1min":0.63400000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.766,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":2.0449999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.2589999999999999,"15min":1.486},"min":{"1min":0.57199999999999995,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":5.5410000000000004,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.93600000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953908,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070308,"kb_used_data":476508,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92674844,"statfs":{"total":95995035648,"available":94899040256,"internally_reserved":0,"allocated":487944192,"data_stored":487473008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.131,"15min":1.3680000000000001},"min":{"1min":0.29899999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.492,"5min":7.492,"15min":90.597999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.248,"15min":1.5329999999999999},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":7.569,"5min":12.236000000000001,"15min":90.676000000000002},"last":0.63100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.121,"15min":1},"min":{"1min":0.32900000000000001,"5min":0.28399999999999997,"15min":0.24399999999999999},"max":{"1min":7.3630000000000004,"5min":7.3630000000000004,"15min":8.0530000000000008},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.276,"5min":1.1890000000000001,"15min":1.1299999999999999},"min":{"1min":0.30599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":7.633,"5min":7.633,"15min":7.633},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.1759999999999999,"15min":1.1180000000000001},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":7.6029999999999998,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.294,"5min":1.0649999999999999,"15min":1.0609999999999999},"min":{"1min":0.45700000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.383,"5min":7.383,"15min":7.383},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.099,"15min":0.98099999999999998},"min":{"1min":0.372,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":7.431,"5min":7.431,"15min":7.431},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.7809999999999999,"5min":1.2470000000000001,"15min":1.149},"min":{"1min":0.42599999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":7.4569999999999999,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.78700000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.86799999999999999,"15min":1.0029999999999999},"min":{"1min":0.28000000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.5289999999999999,"5min":7.5289999999999999,"15min":7.5410000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.256,"5min":1.1160000000000001,"15min":1.101},"min":{"1min":0.28999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.9930000000000003,"5min":5.5190000000000001,"15min":9.6639999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.0269999999999999,"15min":1.099},"min":{"1min":0.34000000000000002,"5min":0.312,"15min":0.312},"max":{"1min":4.2140000000000004,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.143,"5min":0.90000000000000002,"15min":1.1419999999999999},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":7.6500000000000004,"5min":7.6500000000000004,"15min":7.6689999999999996},"last":3.016}]},{"osd":8,"last update":"Mon Feb 12 16:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.032,"15min":1.204},"min":{"1min":0.55300000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":7.6890000000000001,"5min":7.6890000000000001,"15min":7.8650000000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.7470000000000001,"5min":1.2410000000000001,"15min":1.26},"min":{"1min":0.52000000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":7.7110000000000003},"last":0.998}]},{"osd":9,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.0660000000000001,"15min":1.4419999999999999},"min":{"1min":0.59199999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":7.7750000000000004,"5min":7.7750000000000004,"15min":34.112000000000002},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.3520000000000001,"15min":1.635},"min":{"1min":0.41499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":7.827,"5min":7.827,"15min":34.590000000000003},"last":1.1819999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:28:55 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.4239999999999999,"15min":1.425},"min":{"1min":0.45500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":7.7619999999999996,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.093,"15min":1.2450000000000001},"min":{"1min":0.52700000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":4.3019999999999996,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":0.872}]},{"osd":11,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.1379999999999999,"15min":1.446},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":7.734,"5min":7.734,"15min":7.734},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.74,"5min":1.419,"15min":1.4950000000000001},"min":{"1min":0.56399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":7.7969999999999997,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":1.107}]}]},{"osd":5,"up_from":39,"seq":167503724971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831960,"kb_used_data":530640,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92913192,"statfs":{"total":95995035648,"available":95143108608,"internally_reserved":0,"allocated":543375360,"data_stored":542910186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":3.2469999999999999,"15min":3.484},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":1.387,"5min":221.96600000000001,"15min":565.34500000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.155,"5min":3.3519999999999999,"15min":3.5790000000000002},"min":{"1min":0.45900000000000002,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":1.4830000000000001,"5min":222.06200000000001,"15min":565.98400000000004},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.53,"15min":1.141},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.3779999999999999,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.599,"15min":1.2150000000000001},"min":{"1min":0.375,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.3700000000000001,"5min":40.225000000000001,"15min":40.225000000000001},"last":0.72799999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:29:22 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.4039999999999999,"15min":1.3340000000000001},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2650000000000001,"5min":16.300999999999998,"15min":16.300999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.999,"15min":0.81899999999999995},"min":{"1min":0.29699999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.284,"5min":6.5540000000000003,"15min":6.5540000000000003},"last":0.46100000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.3640000000000001,"15min":1.4239999999999999},"min":{"1min":0.40600000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.5149999999999999,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.109,"5min":1.736,"15min":1.476},"min":{"1min":0.499,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":3.165,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.89000000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:29:15 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.198,"15min":1.3480000000000001},"min":{"1min":0.39600000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.9089999999999998,"5min":6.4459999999999997,"15min":74.105000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.222,"5min":1.1639999999999999,"15min":1.4279999999999999},"min":{"1min":0.375,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.048,"5min":9.5830000000000002,"15min":75.013999999999996},"last":0.442}]},{"osd":7,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.1379999999999999,"15min":1.0449999999999999},"min":{"1min":0.42499999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.7429999999999999,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.462,"5min":1.325,"15min":1.1599999999999999},"min":{"1min":0.36599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.423,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":1.048}]},{"osd":8,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.1140000000000001,"15min":1.1799999999999999},"min":{"1min":0.51300000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.3159999999999998,"5min":4.2850000000000001,"15min":5.9509999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.58799999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.98,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":1.069}]},{"osd":9,"last update":"Mon Feb 12 16:29:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5669999999999999,"15min":3.3399999999999999},"min":{"1min":0.496,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":5.476,"5min":6.2939999999999996,"15min":449.89499999999998},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.361,"15min":3.2730000000000001},"min":{"1min":0.48199999999999998,"5min":0.375,"15min":0.375},"max":{"1min":2.4470000000000001,"5min":9.6649999999999991,"15min":448.70800000000003},"last":1.03}]},{"osd":10,"last update":"Mon Feb 12 16:28:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5089999999999999,"5min":1.157,"15min":1.2509999999999999},"min":{"1min":0.55800000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.242,"5min":4.242,"15min":5.9210000000000003},"last":55.518999999999998},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.5449999999999999,"15min":1.353},"min":{"1min":0.48899999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.4619999999999997,"5min":15.006,"15min":15.006},"last":55.497999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:28:43 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.2969999999999999,"15min":1.2889999999999999},"min":{"1min":0.59999999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.6940000000000008,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.548,"15min":1.391},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":4.5019999999999998,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":0.98099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":370147328,"data_stored":370147328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":414187520,"data_stored":414187520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137232384,"data_stored":137232384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1085538304,"data_stored":1085538304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":514588672,"data_stored":514588672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":571277312,"data_stored":571277312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":725090304,"data_stored":725090304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":627769344,"data_stored":627769344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":325189632,"data_stored":325189632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":269811712,"data_stored":269811712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":165806080,"data_stored":165806080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:29:44.651 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:44.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:44.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:44.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:44.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:45.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:45.033 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.036 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:45.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:45.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:45 smithi121 bash[18489]: cluster 2024-02-12T16:29:44.021473+0000 mgr.y (mgr.14146) 1349 : cluster [DBG] pgmap v1881: 37 pgs: 37 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:29:45.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:45.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:45.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:45 smithi169 bash[19604]: cluster 2024-02-12T16:29:44.021473+0000 mgr.y (mgr.14146) 1349 : cluster [DBG] pgmap v1881: 37 pgs: 37 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:29:45.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:45 smithi123 bash[19119]: cluster 2024-02-12T16:29:44.021473+0000 mgr.y (mgr.14146) 1349 : cluster [DBG] pgmap v1881: 37 pgs: 37 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:29:45.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:45.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:45.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:45.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 56719 56703 208.448 122.688 0.00227435 0.00473419 2024-02-12T16:29:45.583 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:45.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:45.720 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:45.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:45.830 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:45.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:45.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:45.986 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:45.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:46.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:46.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:46.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:46.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:46.258 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:46.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:46.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:46 smithi169 bash[19604]: cluster 2024-02-12T16:29:44.654492+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:46.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:46 smithi169 bash[19604]: cluster 2024-02-12T16:29:44.654536+0000 osd.7 (osd.7) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:46.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:46 smithi169 bash[19604]: cluster 2024-02-12T16:29:45.095837+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in 2024-02-12T16:29:46.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:46 smithi169 bash[19604]: cluster 2024-02-12T16:29:45.707879+0000 mon.a (mon.0) 2268 : cluster [DBG] osdmap e650: 12 total, 12 up, 12 in 2024-02-12T16:29:46.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:46.373 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:46 smithi123 bash[19119]: cluster 2024-02-12T16:29:44.654492+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:46.373 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:46 smithi123 bash[19119]: cluster 2024-02-12T16:29:44.654536+0000 osd.7 (osd.7) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:46.373 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:46 smithi123 bash[19119]: cluster 2024-02-12T16:29:45.095837+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in 2024-02-12T16:29:46.373 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:46 smithi123 bash[19119]: cluster 2024-02-12T16:29:45.707879+0000 mon.a (mon.0) 2268 : cluster [DBG] osdmap e650: 12 total, 12 up, 12 in 2024-02-12T16:29:46.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:46.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:46 smithi121 bash[18489]: cluster 2024-02-12T16:29:44.654492+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:46 smithi121 bash[18489]: cluster 2024-02-12T16:29:44.654536+0000 osd.7 (osd.7) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:46 smithi121 bash[18489]: cluster 2024-02-12T16:29:45.095837+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e649: 12 total, 12 up, 12 in 2024-02-12T16:29:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:46 smithi121 bash[18489]: cluster 2024-02-12T16:29:45.707879+0000 mon.a (mon.0) 2268 : cluster [DBG] osdmap e650: 12 total, 12 up, 12 in 2024-02-12T16:29:46.503 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:46.532 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:46.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:46.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 58870 58854 204.336 134.438 0.00684328 0.00489196 2024-02-12T16:29:46.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:46.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:46.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:46.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:46.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:46.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:46.891 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:46.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:46.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:47.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:47.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:47.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:47.139 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:47.164 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:47.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:47.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:47.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:47 smithi169 bash[19604]: cluster 2024-02-12T16:29:46.021849+0000 mgr.y (mgr.14146) 1350 : cluster [DBG] pgmap v1884: 37 pgs: 37 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.94k op/s 2024-02-12T16:29:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:47.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:47.389 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:47 smithi121 bash[18489]: cluster 2024-02-12T16:29:46.021849+0000 mgr.y (mgr.14146) 1350 : cluster [DBG] pgmap v1884: 37 pgs: 37 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.94k op/s 2024-02-12T16:29:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:47 smithi123 bash[19119]: cluster 2024-02-12T16:29:46.021849+0000 mgr.y (mgr.14146) 1350 : cluster [DBG] pgmap v1884: 37 pgs: 37 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.94k op/s 2024-02-12T16:29:47.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:47.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:47.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:47.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:47.553 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:47.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:47.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 61149 61133 201.077 142.438 0.0028248 0.00497046 2024-02-12T16:29:47.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:29:47.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:311591 4.f 4:f6eb50cf:::benchmark_data_smithi169_60938_object138254:head [delete] snapc 0=[] RETRY=10 ondisk+retry+write+known_if_redirected e523)", 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:03.145153+0000", 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 404.56278953499998, 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16071912299999999, 2024-02-12T16:29:47.720 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 311591 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145153+0000", 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145153+0000", 2024-02-12T16:29:47.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145150+0000", 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3649999999999993e-06 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145160+0000", 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6328999999999999e-05 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145176+0000", 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0690000000000002e-06 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145184+0000", 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3730000000000001e-05 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145258+0000", 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2114e-05 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145270+0000", 2024-02-12T16:29:47.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12569793100000001 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.270968+0000", 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031863600000000002 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.271286+0000", 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034415201999999999 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.305702+0000", 2024-02-12T16:29:47.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000171001 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.305873+0000", 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16071912299999999 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:311595 4.f 4:f7631ab2:::benchmark_data_smithi169_60938_object138258:head [delete] snapc 0=[] RETRY=9 ondisk+retry+write+known_if_redirected e523)", 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:03.145194+0000", 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 404.56274915400002, 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.156954327, 2024-02-12T16:29:47.725 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 311595 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145194+0000", 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145194+0000", 2024-02-12T16:29:47.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145193+0000", 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0969999999999999e-06 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145197+0000", 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0180000000000001e-06 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145202+0000", 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4800000000000001e-06 2024-02-12T16:29:47.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145206+0000", 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104571 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145310+0000", 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3180000000000007e-06 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145318+0000", 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14293967799999999 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.288258+0000", 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020941800000000001 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.288468+0000", 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028153899999999998 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.288749+0000", 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062641509999999999 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.295013+0000", 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1329999999999999e-06 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.295021+0000", 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068961159999999999 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.301918+0000", 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7917999999999996e-05 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.301985+0000", 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000162738 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.302148+0000", 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.156954327 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:311617 4.f 4:f050406c:::benchmark_data_smithi169_60938_object138280:head [delete] snapc 0=[] RETRY=9 ondisk+retry+write+known_if_redirected e523)", 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:03.145223+0000", 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 404.56271998900002, 2024-02-12T16:29:47.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16827498900000001, 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 311617 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145223+0000", 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145223+0000", 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145222+0000", 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.954e-06 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145225+0000", 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5122e-05 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145240+0000", 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0409999999999999e-06 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145244+0000", 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013050799999999999 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145375+0000", 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.546e-06 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145379+0000", 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14971584800000001 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.295094+0000", 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020284799999999999 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.295297+0000", 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018407800000000001 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.295481+0000", 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010162233 2024-02-12T16:29:47.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.305644+0000", 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031657899999999999 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.305960+0000", 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074089849999999999 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.313369+0000", 2024-02-12T16:29:47.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9801e-05 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.313419+0000", 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9105999999999998e-05 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.313498+0000", 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16827498900000001 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:311624 4.f 4:fe738e85:::benchmark_data_smithi169_60938_object138287:head [delete] snapc 0=[] RETRY=9 ondisk+retry+write+known_if_redirected e523)", 2024-02-12T16:29:47.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:03.145248+0000", 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 404.56269470000001, 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16854212599999999, 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 311624 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145248+0000", 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145248+0000", 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145248+0000", 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145251+0000", 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.702e-06 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145255+0000", 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7960000000000002e-06 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145259+0000", 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015143000000000001 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145411+0000", 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5850000000000002e-06 2024-02-12T16:29:47.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145418+0000", 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15315040699999999 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.298569+0000", 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019252700000000001 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.298761+0000", 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000184196 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.298945+0000", 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070326959999999997 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.305978+0000", 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117336 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.306095+0000", 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0075186389999999997 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.313614+0000", 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.952e-05 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.313664+0000", 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126816 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.313790+0000", 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16854212599999999 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:311687 4.f 4:f07899e2:::benchmark_data_smithi169_60938_object138350:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e523)", 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:03.145262+0000", 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 404.56268075999998, 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17291463500000001, 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 311687 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145262+0000", 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145262+0000", 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145262+0000", 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4109999999999999e-06 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145264+0000", 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0849999999999999e-06 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145268+0000", 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.393e-06 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145272+0000", 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016046100000000001 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145432+0000", 2024-02-12T16:29:47.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3830000000000001e-06 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.145436+0000", 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15676843700000001 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.302204+0000", 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017358200000000001 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.302378+0000", 2024-02-12T16:29:47.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019853699999999999 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9,11", 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.302576+0000", 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006829145 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.309405+0000", 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086086400000000007 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.318014+0000", 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9862999999999999e-05 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.318074+0000", 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7647000000000001e-05 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.318111+0000", 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5468000000000004e-05 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:03.318177+0000", 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17291463500000001 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:319473 4.f 4:ff5f2240:::benchmark_data_smithi169_60938_object146136:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e526)", 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:10.585059+0000", 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 397.12288360999997, 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12878116100000001, 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319473 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585059+0000", 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585059+0000", 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585059+0000", 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2200000000000001e-06 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585062+0000", 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.026e-05 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585072+0000", 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5770000000000004e-06 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585077+0000", 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048359500000000002 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585560+0000", 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7308e-05 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585588+0000", 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.113725916 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.699313+0000", 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010265899999999999 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.699416+0000", 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4221000000000005e-05 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.699510+0000", 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010296820999999999 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.709807+0000", 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039300589999999996 2024-02-12T16:29:47.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.713737+0000", 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5773000000000003e-05 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.713793+0000", 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7524000000000001e-05 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.713841+0000", 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12878116100000001 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:319485 4.f 4:f7ee96d2:::benchmark_data_smithi169_60938_object146148:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e526)", 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:10.585081+0000", 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 397.12286168999998, 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13115537299999999, 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319485 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.756 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585081+0000", 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585081+0000", 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585081+0000", 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1889999999999999e-06 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585084+0000", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0687999999999999e-05 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585104+0000", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5951999999999999e-05 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585120+0000", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052331899999999995 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585644+0000", 2024-02-12T16:29:47.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5689999999999994e-06 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585653+0000", 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12176441 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.707418+0000", 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104653 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.707522+0000", 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0780999999999998e-05 2024-02-12T16:29:47.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.707603+0000", 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085207560000000009 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716124+0000", 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.115e-05 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716135+0000", 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3764000000000002e-05 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716189+0000", 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7920000000000002e-05 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716237+0000", 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13115537299999999 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:319505 4.f 4:fbe9b191:::benchmark_data_smithi169_60938_object146168:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e526)", 2024-02-12T16:29:47.761 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:10.585125+0000", 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 397.12281775399998, 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.134785813, 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319505 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585125+0000", 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585125+0000", 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585124+0000", 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585128+0000", 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5502000000000001e-05 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585153+0000", 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7779999999999998e-06 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585158+0000", 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061002699999999997 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585768+0000", 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2668e-05 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585781+0000", 2024-02-12T16:29:47.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.124070607 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.709851+0000", 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012467500000000001 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.709976+0000", 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000168863 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.710145+0000", 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0061567719999999996 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716302+0000", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035061749999999998 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.719808+0000", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5912000000000003e-05 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.719864+0000", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7168e-05 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.719911+0000", 2024-02-12T16:29:47.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.134785813 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:319529 4.f 4:fc12b4da:::benchmark_data_smithi169_60938_object146192:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e526)", 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:10.585193+0000", 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 397.12274963499999, 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14127535199999999, 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319529 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585193+0000", 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585193+0000", 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585193+0000", 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.394e-06 2024-02-12T16:29:47.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585196+0000", 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1823e-05 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585208+0000", 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.831e-06 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585213+0000", 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000715012 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585928+0000", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7513999999999999e-05 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585945+0000", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12793460000000001 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.713880+0000", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4814000000000006e-05 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.713955+0000", 2024-02-12T16:29:47.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0340000000000007e-05 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.714035+0000", 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0058856070000000002 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.719920+0000", 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063908070000000001 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726311+0000", 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9382999999999996e-05 2024-02-12T16:29:47.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726381+0000", 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8048999999999997e-05 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726469+0000", 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14127535199999999 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:319533 4.f 4:fa3212b5:::benchmark_data_smithi169_60938_object146196:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e526)", 2024-02-12T16:29:47.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:10.585242+0000", 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 397.12270142699998, 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.141412764, 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319533 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585242+0000", 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585242+0000", 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585241+0000", 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3230000000000002e-06 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585244+0000", 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9737000000000001e-05 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585264+0000", 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5693999999999999e-05 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585280+0000", 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072260900000000003 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.586002+0000", 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.066e-05 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.586013+0000", 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13032468699999999 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716337+0000", 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120949 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716458+0000", 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015593599999999999 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.716614+0000", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099048599999999997 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726519+0000", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3302000000000001e-05 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726533+0000", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6241000000000004e-05 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726609+0000", 2024-02-12T16:29:47.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5565999999999997e-05 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726654+0000", 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.141412764 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:319554 4.f 4:f2571e11:::benchmark_data_smithi169_60938_object146217:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e526)", 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:10.585285+0000", 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 397.12265844400002, 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14159845200000001, 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319554 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585285+0000", 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585285+0000", 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585284+0000", 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4369999999999999e-06 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585287+0000", 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7470999999999999e-05 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585305+0000", 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8599999999999993e-06 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585313+0000", 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13465548099999999 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.719968+0000", 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012867 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.720097+0000", 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8539000000000001e-05 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.720175+0000", 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064905509999999998 2024-02-12T16:29:47.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726666+0000", 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030999999999997e-05 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726701+0000", 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014156800000000001 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726842+0000", 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0624000000000003e-05 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726883+0000", 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14159845200000001 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.25264.0:319555 4.f 4:f7bb9f7a:::benchmark_data_smithi169_60938_object146218:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e526)", 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:23:10.585354+0000", 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 397.12258917899999, 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.141669671, 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.25264", 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2667289283", 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 319555 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585354+0000", 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585354+0000", 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585353+0000", 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3510000000000002e-06 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585356+0000", 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5830000000000003e-06 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585363+0000", 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8969999999999999e-06 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.585368+0000", 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.137094774 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.722463+0000", 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012052 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.722583+0000", 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015616999999999999 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 9", 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.722739+0000", 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041877199999999998 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726927+0000", 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.043e-05 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726937+0000", 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2231000000000002e-05 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.726990+0000", 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3794999999999997e-05 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:23:10.727023+0000", 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.141669671 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26257.0:47948 6.7 e648/636 6:e98f99d5:::benchmark_data_smithi169_96224_object47947:head v 648'2968, mlcod=648'2963)", 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.773016+0000", 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9349270509999998, 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16073274400000001, 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773016+0000", 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773016+0000", 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773015+0000", 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9033000000000002e-05 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773074+0000", 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7839999999999998e-06 2024-02-12T16:29:47.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773080+0000", 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.72e-06 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773084+0000", 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2682999999999999e-05 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773117+0000", 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244000000000001e-05 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773130+0000", 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16016926000000001 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933299+0000", 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00044944199999999998 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933749+0000", 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16073274400000001 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:47961 6.c 6:3cb3305a:::benchmark_data_smithi169_96224_object47960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e648)", 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.773642+0000", 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9343012359999996, 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.168023271, 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 47961 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773642+0000", 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773642+0000", 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773640+0000", 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000112768 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773753+0000", 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1214e-05 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773764+0000", 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1530000000000002e-06 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773771+0000", 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5300999999999999e-05 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773816+0000", 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000109357 2024-02-12T16:29:47.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.773925+0000", 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2213000000000006e-05 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774017+0000", 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16458162800000001 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.938599+0000", 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002929914 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.941529+0000", 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2120999999999995e-05 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.941611+0000", 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3950999999999999e-05 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.941665+0000", 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.168023271 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26257.0:47949 6.7 e648/636 6:e4317d9f:::benchmark_data_smithi169_96224_object47948:head v 648'2969, mlcod=648'2966)", 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.774065+0000", 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9338783849999999, 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16504017700000001, 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774065+0000", 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774065+0000", 2024-02-12T16:29:47.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999804 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774045+0000", 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1539999999999997e-05 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774107+0000", 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8029999999999997e-06 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774115+0000", 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.378e-06 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774120+0000", 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774158+0000", 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7600000000000001e-05 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774176+0000", 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16489663399999999 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.939072+0000", 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2555999999999998e-05 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.939105+0000", 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16504017700000001 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:47965 6.b 6:d037726b:::benchmark_data_smithi169_96224_object47964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e648)", 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.774092+0000", 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9338514040000003, 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16668363, 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.800 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 47965 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774092+0000", 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774092+0000", 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:29:47.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774090+0000", 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7477000000000005e-05 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774158+0000", 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9289999999999998e-06 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774165+0000", 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2010000000000002e-06 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774173+0000", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8928000000000003e-05 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774222+0000", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6161000000000005e-05 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774298+0000", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3038e-05 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774381+0000", 2024-02-12T16:29:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16280750299999999 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.937188+0000", 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034966939999999998 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.940685+0000", 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1116999999999999e-05 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.940736+0000", 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9149000000000001e-05 2024-02-12T16:29:47.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.940775+0000", 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16668363 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26257.0:47962 6.0 e648/636 6:082306c4:::benchmark_data_smithi169_96224_object47961:head v 648'3014, mlcod=648'3012)", 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.774111+0000", 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9338317869999999, 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.159836704, 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774111+0000", 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774111+0000", 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774110+0000", 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6611e-05 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774137+0000", 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7669999999999998e-06 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774146+0000", 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1940000000000003e-06 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774153+0000", 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8783000000000003e-05 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774211+0000", 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8984999999999999e-05 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.774230+0000", 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15962319599999999 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933853+0000", 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4414999999999996e-05 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933948+0000", 2024-02-12T16:29:47.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.159836704 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:47966 6.c 6:394fcfb6:::benchmark_data_smithi169_96224_object47965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e648)", 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.775771+0000", 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9321718289999996, 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16607013200000001, 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 47966 2024-02-12T16:29:47.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.775771+0000", 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.775771+0000", 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.775769+0000", 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8056e-05 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.775837+0000", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.157509016 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933346+0000", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3028e-05 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933359+0000", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030923099999999998 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933668+0000", 2024-02-12T16:29:47.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034821899999999998 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.934016+0000", 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00096368599999999995 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.934980+0000", 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067472640000000002 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.941727+0000", 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6240000000000008e-06 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.941736+0000", 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9534999999999998e-05 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.941796+0000", 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5773999999999999e-05 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.941841+0000", 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16607013200000001 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26257.0:47964 6.7 e648/636 6:ed12fd77:::benchmark_data_smithi169_96224_object47963:head v 648'2970, mlcod=648'2966)", 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.776054+0000", 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9318891389999999, 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.163078108, 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.776054+0000", 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.776054+0000", 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.776052+0000", 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6601000000000002e-05 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.776109+0000", 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15711765999999999 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933227+0000", 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4363999999999998e-05 2024-02-12T16:29:47.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933261+0000", 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056897999999999996 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933830+0000", 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3559000000000004e-05 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933914+0000", 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051994859999999997 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.939113+0000", 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8935000000000001e-05 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.939132+0000", 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.163078108 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26257.0:47965 6.b e648/636 ondisk, result = 0)", 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:29:41.778065+0000", 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9298780889999998, 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15912501900000001, 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:29:47.817 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.778065+0000", 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.778065+0000", 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.778064+0000", 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9780000000000002e-06 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.778070+0000", 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.155223574 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933293+0000", 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3869000000000001e-05 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.933307+0000", 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038672189999999999 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:29:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.937174+0000", 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8279999999999997e-06 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.937181+0000", 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9220000000000003e-06 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:29:41.937190+0000", 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15912501900000001 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:29:47.820 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:29:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:47.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:47.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:47.822 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:29:47.822 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:29:47.822 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:29:47.842 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:29:47.875 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:29:47.916 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:29:47.923 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:47.948 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:47.949 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:29:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:47.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:47.981 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:29:48.006 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:29:48.032 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:29:48.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:48.058 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:29:48.084 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:29:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.086 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.087 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:29:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:48.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:48.117 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:29:48.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:48.219 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:48.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:48.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:48.344 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:48.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:48 smithi123 bash[19119]: audit 2024-02-12T16:29:47.672932+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.121:0/3387355783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:29:48.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:48 smithi123 bash[19119]: audit 2024-02-12T16:29:47.679447+0000 mon.a (mon.0) 2270 : audit [DBG] from='client.? v1:172.21.15.121:0/978959665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:48.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:48 smithi121 bash[18489]: audit 2024-02-12T16:29:47.672932+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.121:0/3387355783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:29:48.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:48 smithi121 bash[18489]: audit 2024-02-12T16:29:47.679447+0000 mon.a (mon.0) 2270 : audit [DBG] from='client.? v1:172.21.15.121:0/978959665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:48.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:48.452 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:48 smithi169 bash[19604]: audit 2024-02-12T16:29:47.672932+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? v1:172.21.15.121:0/3387355783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:29:48.453 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:48 smithi169 bash[19604]: audit 2024-02-12T16:29:47.679447+0000 mon.a (mon.0) 2270 : audit [DBG] from='client.? v1:172.21.15.121:0/978959665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:29:48.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:48.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:48.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:29:48.562638 min lat: 0.00101494 max lat: 0.31862 avg lat: 0.00506024 2024-02-12T16:29:48.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:29:48.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 16 63242 63226 197.564 130.812 0.00546822 0.00506024 2024-02-12T16:29:48.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:48.594 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:48.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:48.697 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:48.722 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:48.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:48.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:48.863 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:48.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:48.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:48.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:49.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:49.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:49.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:49.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:49.395 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:49 smithi121 bash[18489]: cluster 2024-02-12T16:29:47.727299+0000 osd.4 (osd.4) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:49 smithi121 bash[18489]: cluster 2024-02-12T16:29:47.727352+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:49 smithi121 bash[18489]: cluster 2024-02-12T16:29:48.022298+0000 mgr.y (mgr.14146) 1351 : cluster [DBG] pgmap v1885: 37 pgs: 37 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-02-12T16:29:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:49 smithi121 bash[18489]: cluster 2024-02-12T16:29:48.097022+0000 mon.a (mon.0) 2271 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:29:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:49 smithi121 bash[18489]: audit 2024-02-12T16:29:48.107791+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? v1:172.21.15.121:0/3387355783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:29:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:49 smithi121 bash[18489]: cluster 2024-02-12T16:29:48.107844+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e651: 12 total, 12 up, 12 in 2024-02-12T16:29:49.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:49 smithi123 bash[19119]: cluster 2024-02-12T16:29:47.727299+0000 osd.4 (osd.4) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:49.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:49 smithi123 bash[19119]: cluster 2024-02-12T16:29:47.727352+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:49.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:49 smithi123 bash[19119]: cluster 2024-02-12T16:29:48.022298+0000 mgr.y (mgr.14146) 1351 : cluster [DBG] pgmap v1885: 37 pgs: 37 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-02-12T16:29:49.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:49 smithi123 bash[19119]: cluster 2024-02-12T16:29:48.097022+0000 mon.a (mon.0) 2271 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:29:49.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:49 smithi123 bash[19119]: audit 2024-02-12T16:29:48.107791+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? v1:172.21.15.121:0/3387355783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:29:49.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:49.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:49 smithi169 bash[19604]: cluster 2024-02-12T16:29:47.727299+0000 osd.4 (osd.4) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:49.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:49 smithi169 bash[19604]: cluster 2024-02-12T16:29:47.727352+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:49.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:49 smithi169 bash[19604]: cluster 2024-02-12T16:29:48.022298+0000 mgr.y (mgr.14146) 1351 : cluster [DBG] pgmap v1885: 37 pgs: 37 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-02-12T16:29:49.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:49 smithi169 bash[19604]: cluster 2024-02-12T16:29:48.097022+0000 mon.a (mon.0) 2271 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:29:49.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:49 smithi169 bash[19604]: audit 2024-02-12T16:29:48.107791+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? v1:172.21.15.121:0/3387355783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:29:49.508 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:49 smithi169 bash[19604]: cluster 2024-02-12T16:29:48.107844+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e651: 12 total, 12 up, 12 in 2024-02-12T16:29:49.523 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:49.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 65492 65476 194.852 140.625 0.00334795 0.00513053 2024-02-12T16:29:49.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:49.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:49.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:49.765 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:49 smithi123 bash[19119]: cluster 2024-02-12T16:29:48.107844+0000 mon.a (mon.0) 2273 : cluster [DBG] osdmap e651: 12 total, 12 up, 12 in 2024-02-12T16:29:49.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:49.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:49.923 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:49.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:49.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:50.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:50.050 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:50.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:50.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:29:50.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:50.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:50.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:50.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:50.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:50.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:50.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:50.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:50.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:50.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 67665 67649 192.168 135.812 0.012463 0.0052026 2024-02-12T16:29:50.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:50.605 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:50.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:50.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:50.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:50.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:50.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:50.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:50.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:50.976 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:50.998 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:51.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:51.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:51 smithi121 bash[18489]: cluster 2024-02-12T16:29:50.022718+0000 mgr.y (mgr.14146) 1352 : cluster [DBG] pgmap v1887: 37 pgs: 37 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s wr, 2.19k op/s 2024-02-12T16:29:51.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:51 smithi121 bash[18489]: cluster 2024-02-12T16:29:50.047781+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:51.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:51 smithi121 bash[18489]: cluster 2024-02-12T16:29:50.047824+0000 osd.1 (osd.1) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:51.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:51 smithi121 bash[18489]: cluster 2024-02-12T16:29:50.114039+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in 2024-02-12T16:29:51.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:51.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:51.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:51.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:51 smithi169 bash[19604]: cluster 2024-02-12T16:29:50.022718+0000 mgr.y (mgr.14146) 1352 : cluster [DBG] pgmap v1887: 37 pgs: 37 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s wr, 2.19k op/s 2024-02-12T16:29:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:51 smithi169 bash[19604]: cluster 2024-02-12T16:29:50.047781+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:51 smithi169 bash[19604]: cluster 2024-02-12T16:29:50.047824+0000 osd.1 (osd.1) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:51 smithi169 bash[19604]: cluster 2024-02-12T16:29:50.114039+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in 2024-02-12T16:29:51.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:51.388 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:51 smithi123 bash[19119]: cluster 2024-02-12T16:29:50.022718+0000 mgr.y (mgr.14146) 1352 : cluster [DBG] pgmap v1887: 37 pgs: 37 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s wr, 2.19k op/s 2024-02-12T16:29:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:51 smithi123 bash[19119]: cluster 2024-02-12T16:29:50.047781+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:51 smithi123 bash[19119]: cluster 2024-02-12T16:29:50.047824+0000 osd.1 (osd.1) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:51 smithi123 bash[19119]: cluster 2024-02-12T16:29:50.114039+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e652: 12 total, 12 up, 12 in 2024-02-12T16:29:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:51.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:51.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 69884 69868 189.842 138.688 0.0127726 0.005266 2024-02-12T16:29:51.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:51.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:51.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:51.789 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:51.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:51.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:51.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:52.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:52.054 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:52.056 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:52.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:52.181 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:52.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:52.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:29:52.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:29:52.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:52.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:52.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:52.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:52.533 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:52.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:52.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 71876 71860 187.119 124.5 0.00880524 0.00534295 2024-02-12T16:29:52.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:52.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:52.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:52.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:52.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:52.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:52.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:52.956 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:52.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:53.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:53.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:53.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:53.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:53.224 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:53.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:53.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:53.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:53.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:53.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:53 smithi123 bash[19119]: cluster 2024-02-12T16:29:52.023207+0000 mgr.y (mgr.14146) 1353 : cluster [DBG] pgmap v1889: 37 pgs: 37 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-12T16:29:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:53 smithi121 bash[18489]: cluster 2024-02-12T16:29:52.023207+0000 mgr.y (mgr.14146) 1353 : cluster [DBG] pgmap v1889: 37 pgs: 37 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-12T16:29:53.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:53.467 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:53 smithi169 bash[19604]: cluster 2024-02-12T16:29:52.023207+0000 mgr.y (mgr.14146) 1353 : cluster [DBG] pgmap v1889: 37 pgs: 37 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.97k op/s 2024-02-12T16:29:53.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:53.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:53.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 73921 73905 184.746 127.812 0.00362078 0.00541113 2024-02-12T16:29:53.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:53.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:53.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:53.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:53.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:53.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:53.778 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:53.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:53.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:53.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:53.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:54.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:54.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:54.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:54.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:54.291 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:54.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:54.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:54.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:54.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:54.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 15 75819 75804 182.205 118.688 0.00475301 0.00548625 2024-02-12T16:29:54.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:54.698 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:54.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:54.932 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:54.965 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:54.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:55.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:55.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:55.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:55.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:55.222 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:55.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:55.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:55.332 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:55 smithi123 bash[19119]: cluster 2024-02-12T16:29:54.023643+0000 mgr.y (mgr.14146) 1354 : cluster [DBG] pgmap v1890: 37 pgs: 37 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-12T16:29:55.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:55.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:55.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:55.474 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:55 smithi121 bash[18489]: cluster 2024-02-12T16:29:54.023643+0000 mgr.y (mgr.14146) 1354 : cluster [DBG] pgmap v1890: 37 pgs: 37 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-12T16:29:55.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:55.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:55.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:55 smithi169 bash[19604]: cluster 2024-02-12T16:29:54.023643+0000 mgr.y (mgr.14146) 1354 : cluster [DBG] pgmap v1890: 37 pgs: 37 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.15k op/s 2024-02-12T16:29:55.565 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 77792 77776 180.022 123.25 0.0103126 0.00555371 2024-02-12T16:29:55.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:55.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:55.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:55.723 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:55.768 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:55.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:55.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:55.908 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:55.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:56.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:56.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:56.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:56.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:56.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:56.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:56.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:56.431 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:56.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:56.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 79757 79741 177.978 122.812 0.0121858 0.00561672 2024-02-12T16:29:56.574 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:56.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:56.719 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:56.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:56.824 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:29:56.847 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:56.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:56 smithi123 bash[19119]: cluster 2024-02-12T16:29:56.024108+0000 mgr.y (mgr.14146) 1355 : cluster [DBG] pgmap v1891: 37 pgs: 37 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-12T16:29:56.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:56.972 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:56.975 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:56 smithi169 bash[19604]: cluster 2024-02-12T16:29:56.024108+0000 mgr.y (mgr.14146) 1355 : cluster [DBG] pgmap v1891: 37 pgs: 37 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-12T16:29:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:56.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:57.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:57.110 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:57.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:56 smithi121 bash[18489]: cluster 2024-02-12T16:29:56.024108+0000 mgr.y (mgr.14146) 1355 : cluster [DBG] pgmap v1891: 37 pgs: 37 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 140 MiB/s wr, 2.24k op/s 2024-02-12T16:29:57.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:57.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:57.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:57.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:57.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:57.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:57.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:57.524 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:29:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:57.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:29:57.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 81615 81599 175.845 116.125 0.00443498 0.00568574 2024-02-12T16:29:57.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:57.656 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:29:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:57.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:29:57.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:57.796 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:29:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:57.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:29:57.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:57.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:57 smithi121 bash[18489]: cluster 2024-02-12T16:29:56.223640+0000 osd.10 (osd.10) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:57.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:57 smithi121 bash[18489]: cluster 2024-02-12T16:29:56.223724+0000 osd.10 (osd.10) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:57 smithi121 bash[18489]: cluster 2024-02-12T16:29:56.712123+0000 mon.a (mon.0) 2275 : cluster [DBG] osdmap e653: 12 total, 12 up, 12 in 2024-02-12T16:29:57.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:57 smithi169 bash[19604]: cluster 2024-02-12T16:29:56.223640+0000 osd.10 (osd.10) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:57.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:57 smithi169 bash[19604]: cluster 2024-02-12T16:29:56.223724+0000 osd.10 (osd.10) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:57.910 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:57 smithi169 bash[19604]: cluster 2024-02-12T16:29:56.712123+0000 mon.a (mon.0) 2275 : cluster [DBG] osdmap e653: 12 total, 12 up, 12 in 2024-02-12T16:29:57.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:29:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:57.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:29:58.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:58.068 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:29:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:58.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:29:58.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:57 smithi123 bash[19119]: cluster 2024-02-12T16:29:56.223640+0000 osd.10 (osd.10) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:29:58.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:57 smithi123 bash[19119]: cluster 2024-02-12T16:29:56.223724+0000 osd.10 (osd.10) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:29:58.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:57 smithi123 bash[19119]: cluster 2024-02-12T16:29:56.712123+0000 mon.a (mon.0) 2275 : cluster [DBG] osdmap e653: 12 total, 12 up, 12 in 2024-02-12T16:29:58.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:58.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:58.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:58.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:58.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:29:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:58.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:29:58.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:58.467 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:29:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:58.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:29:58.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 83523 83507 173.958 119.25 0.0101034 0.005747 2024-02-12T16:29:58.572 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:58.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:58.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:58.724 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:29:58 smithi169 bash[19604]: cluster 2024-02-12T16:29:58.024614+0000 mgr.y (mgr.14146) 1356 : cluster [DBG] pgmap v1893: 37 pgs: 37 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.12k op/s 2024-02-12T16:29:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:58.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:58.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:58.847 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:29:58 smithi121 bash[18489]: cluster 2024-02-12T16:29:58.024614+0000 mgr.y (mgr.14146) 1356 : cluster [DBG] pgmap v1893: 37 pgs: 37 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.12k op/s 2024-02-12T16:29:58.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:58.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:58.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:58.981 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:29:58 smithi123 bash[19119]: cluster 2024-02-12T16:29:58.024614+0000 mgr.y (mgr.14146) 1356 : cluster [DBG] pgmap v1893: 37 pgs: 37 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.12k op/s 2024-02-12T16:29:59.006 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:59.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:59.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:59.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:59.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:29:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:29:59.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:59.411 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:29:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:29:59.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:59.539 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:29:59.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 85406 85390 172.142 117.688 0.00354094 0.0058078 2024-02-12T16:29:59.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:59.680 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:29:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:29:59.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:59.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:29:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:29:59.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:59.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:29:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:29:59.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:00.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:00.064 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:00 smithi123 bash[19119]: cluster 2024-02-12T16:30:00.000102+0000 mon.a (mon.0) 2276 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:30:00.064 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:00 smithi123 bash[19119]: cluster 2024-02-12T16:30:00.000141+0000 mon.a (mon.0) 2277 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-12T16:30:00.066 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:00.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:00.193 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:00.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:00.326 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:00.382 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:00 smithi169 bash[19604]: cluster 2024-02-12T16:30:00.000102+0000 mon.a (mon.0) 2276 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:30:00.382 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:00 smithi169 bash[19604]: cluster 2024-02-12T16:30:00.000141+0000 mon.a (mon.0) 2277 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-12T16:30:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:00 smithi121 bash[18489]: cluster 2024-02-12T16:30:00.000102+0000 mon.a (mon.0) 2276 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:30:00.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:00 smithi121 bash[18489]: cluster 2024-02-12T16:30:00.000141+0000 mon.a (mon.0) 2277 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-12T16:30:00.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:00.456 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:00.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:00.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 86955 86939 169.788 96.8125 0.00668228 0.00588821 2024-02-12T16:30:00.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:00.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:00.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:00.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:00.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:00.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:00.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:01.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:01.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:01.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:01.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:01.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:01.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:01.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:01.375 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:01.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:01 smithi123 bash[19119]: cluster 2024-02-12T16:30:00.025083+0000 mgr.y (mgr.14146) 1357 : cluster [DBG] pgmap v1894: 37 pgs: 37 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-12T16:30:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:01 smithi123 bash[19119]: cluster 2024-02-12T16:30:00.710188+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e654: 12 total, 12 up, 12 in 2024-02-12T16:30:01.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:01 smithi121 bash[18489]: cluster 2024-02-12T16:30:00.025083+0000 mgr.y (mgr.14146) 1357 : cluster [DBG] pgmap v1894: 37 pgs: 37 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-12T16:30:01.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:01 smithi121 bash[18489]: cluster 2024-02-12T16:30:00.710188+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e654: 12 total, 12 up, 12 in 2024-02-12T16:30:01.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:01.527 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:01.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:01 smithi169 bash[19604]: cluster 2024-02-12T16:30:00.025083+0000 mgr.y (mgr.14146) 1357 : cluster [DBG] pgmap v1894: 37 pgs: 37 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-12T16:30:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:01 smithi169 bash[19604]: cluster 2024-02-12T16:30:00.710188+0000 mon.a (mon.0) 2278 : cluster [DBG] osdmap e654: 12 total, 12 up, 12 in 2024-02-12T16:30:01.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 88681 88665 167.912 107.875 0.00433953 0.00595417 2024-02-12T16:30:01.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:01.657 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:01.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:01.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:01.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:01.794 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:01.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:01.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:01.929 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:01.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:02.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:02.057 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:02.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:02.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:02.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:02.375 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:02.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:02.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:02.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 90435 90419 166.197 109.625 0.010903 0.00601538 2024-02-12T16:30:02.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:02.643 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:02.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:02.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:02.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:02.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:02.905 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:02.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:03.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:03.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:03.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:03 smithi121 bash[18489]: cluster 2024-02-12T16:30:02.025524+0000 mgr.y (mgr.14146) 1358 : cluster [DBG] pgmap v1896: 37 pgs: 37 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-02-12T16:30:03.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:03.145 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:03 smithi169 bash[19604]: cluster 2024-02-12T16:30:02.025524+0000 mgr.y (mgr.14146) 1358 : cluster [DBG] pgmap v1896: 37 pgs: 37 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-02-12T16:30:03.166 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:03.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:03.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:03 smithi123 bash[19119]: cluster 2024-02-12T16:30:02.025524+0000 mgr.y (mgr.14146) 1358 : cluster [DBG] pgmap v1896: 37 pgs: 37 active+clean; 5.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-02-12T16:30:03.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:03.434 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:03.537 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:03.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 92112 92096 164.443 104.812 0.00391744 0.00607936 2024-02-12T16:30:03.571 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:03.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:03.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:03.808 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:03.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:03.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:03.981 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:03.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:04.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:04.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:04.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:04.246 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:04.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:04.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.739785+0000 mon.a (mon.0) 2279 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1665130938,config=2903391811,health=1906662658,kv=1725960315,logm=2481501708}) 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.741272+0000 mon.a (mon.0) 2280 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051259567}) 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.743312+0000 mon.a (mon.0) 2281 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=58779077}) 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.744739+0000 mon.a (mon.0) 2282 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791897592}) 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.747129+0000 mon.a (mon.0) 2283 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150194309}) 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.749575+0000 mon.a (mon.0) 2284 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1427628979}) 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.754889+0000 mon.a (mon.0) 2285 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228876223}) 2024-02-12T16:30:04.357 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.758909+0000 mon.a (mon.0) 2286 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3932375424}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.762750+0000 mon.a (mon.0) 2287 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970132308}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.766692+0000 mon.a (mon.0) 2288 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500381692}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.773090+0000 mon.a (mon.0) 2289 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424849389}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.779984+0000 mon.a (mon.0) 2290 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=57} crc {logm=2128497849,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2617105149,mon_config_key=4249958317}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.780922+0000 mon.a (mon.0) 2291 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=693866673}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.781852+0000 mon.a (mon.0) 2292 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3134259950,monmap=2720272532,osd_metadata=3411596064}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.782836+0000 mon.a (mon.0) 2293 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=995221693,osd_pg_creating=3541239294,osdmap=726325750}) 2024-02-12T16:30:04.358 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:04 smithi169 bash[19604]: cluster 2024-02-12T16:30:03.791133+0000 mon.a (mon.0) 2294 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=60} crc {osdmap=1805939612}) 2024-02-12T16:30:04.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:04.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.739785+0000 mon.a (mon.0) 2279 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1665130938,config=2903391811,health=1906662658,kv=1725960315,logm=2481501708}) 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.741272+0000 mon.a (mon.0) 2280 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051259567}) 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.743312+0000 mon.a (mon.0) 2281 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=58779077}) 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.744739+0000 mon.a (mon.0) 2282 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791897592}) 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.747129+0000 mon.a (mon.0) 2283 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150194309}) 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.749575+0000 mon.a (mon.0) 2284 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1427628979}) 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.754889+0000 mon.a (mon.0) 2285 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228876223}) 2024-02-12T16:30:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.758909+0000 mon.a (mon.0) 2286 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3932375424}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.762750+0000 mon.a (mon.0) 2287 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970132308}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.766692+0000 mon.a (mon.0) 2288 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500381692}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.773090+0000 mon.a (mon.0) 2289 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424849389}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.779984+0000 mon.a (mon.0) 2290 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=57} crc {logm=2128497849,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2617105149,mon_config_key=4249958317}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.780922+0000 mon.a (mon.0) 2291 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=693866673}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.781852+0000 mon.a (mon.0) 2292 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3134259950,monmap=2720272532,osd_metadata=3411596064}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.782836+0000 mon.a (mon.0) 2293 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=995221693,osd_pg_creating=3541239294,osdmap=726325750}) 2024-02-12T16:30:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:04 smithi123 bash[19119]: cluster 2024-02-12T16:30:03.791133+0000 mon.a (mon.0) 2294 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=60} crc {osdmap=1805939612}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.739785+0000 mon.a (mon.0) 2279 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=1665130938,config=2903391811,health=1906662658,kv=1725960315,logm=2481501708}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.741272+0000 mon.a (mon.0) 2280 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051259567}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.743312+0000 mon.a (mon.0) 2281 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=58779077}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.744739+0000 mon.a (mon.0) 2282 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791897592}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.747129+0000 mon.a (mon.0) 2283 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150194309}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.749575+0000 mon.a (mon.0) 2284 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1427628979}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.754889+0000 mon.a (mon.0) 2285 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228876223}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.758909+0000 mon.a (mon.0) 2286 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3932375424}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.762750+0000 mon.a (mon.0) 2287 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970132308}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.766692+0000 mon.a (mon.0) 2288 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500381692}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.773090+0000 mon.a (mon.0) 2289 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424849389}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.779984+0000 mon.a (mon.0) 2290 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=3,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=57} crc {logm=2128497849,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2617105149,mon_config_key=4249958317}) 2024-02-12T16:30:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.780922+0000 mon.a (mon.0) 2291 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=693866673}) 2024-02-12T16:30:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.781852+0000 mon.a (mon.0) 2292 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3134259950,monmap=2720272532,osd_metadata=3411596064}) 2024-02-12T16:30:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.782836+0000 mon.a (mon.0) 2293 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=995221693,osd_pg_creating=3541239294,osdmap=726325750}) 2024-02-12T16:30:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:04 smithi121 bash[18489]: cluster 2024-02-12T16:30:03.791133+0000 mon.a (mon.0) 2294 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=60} crc {osdmap=1805939612}) 2024-02-12T16:30:04.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:04.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:04.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:04.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 16 93839 93823 162.873 107.938 0.00377561 0.00613801 2024-02-12T16:30:04.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:04.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:04.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:04.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:04.769 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:30:04.769 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:30:04.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:04.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:04.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:04.926 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:04.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:04.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:30:04.999 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:30:05.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:05.075 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:05.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:05.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:05.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:05.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:05 smithi169 bash[19604]: cluster 2024-02-12T16:30:04.025979+0000 mgr.y (mgr.14146) 1359 : cluster [DBG] pgmap v1897: 37 pgs: 37 active+clean; 5.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.83k op/s 2024-02-12T16:30:05.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:05.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:05.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:05.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:05 smithi123 bash[19119]: cluster 2024-02-12T16:30:04.025979+0000 mgr.y (mgr.14146) 1359 : cluster [DBG] pgmap v1897: 37 pgs: 37 active+clean; 5.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.83k op/s 2024-02-12T16:30:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:05 smithi121 bash[18489]: cluster 2024-02-12T16:30:04.025979+0000 mgr.y (mgr.14146) 1359 : cluster [DBG] pgmap v1897: 37 pgs: 37 active+clean; 5.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 1.83k op/s 2024-02-12T16:30:05.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:05.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:05.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 95462 95446 161.212 101.438 0.00394269 0.00620115 2024-02-12T16:30:05.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:05.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:05.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:05.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:05.748 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:05.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:05.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:05.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:05.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:05.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:06.026 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:06.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:06.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:06 smithi123 bash[19119]: audit 2024-02-12T16:30:04.979316+0000 mgr.y (mgr.14146) 1360 : audit [DBG] from='client.16440 v1:172.21.15.121:0/2822389029' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:30:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:06.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:06.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:06.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:06.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:06 smithi121 bash[18489]: audit 2024-02-12T16:30:04.979316+0000 mgr.y (mgr.14146) 1360 : audit [DBG] from='client.16440 v1:172.21.15.121:0/2822389029' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:30:06.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:06.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:06.542 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:06 smithi169 bash[19604]: audit 2024-02-12T16:30:04.979316+0000 mgr.y (mgr.14146) 1360 : audit [DBG] from='client.16440 v1:172.21.15.121:0/2822389029' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:30:06.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:06.566 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 16 96878 96862 159.299 88.5 0.00952916 0.00627584 2024-02-12T16:30:06.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:06.687 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:06.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:06.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:06.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:06.954 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:06.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:07.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:07.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:07 smithi121 bash[18489]: cluster 2024-02-12T16:30:06.026367+0000 mgr.y (mgr.14146) 1361 : cluster [DBG] pgmap v1898: 37 pgs: 37 active+clean; 5.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-02-12T16:30:07.099 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:07.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:07.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:07.229 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:07.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:07.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:07.341 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:07 smithi123 bash[19119]: cluster 2024-02-12T16:30:06.026367+0000 mgr.y (mgr.14146) 1361 : cluster [DBG] pgmap v1898: 37 pgs: 37 active+clean; 5.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-02-12T16:30:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:07.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:07.401 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1897,"stamp":"2024-02-12T16:30:04.025702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5813239808,"num_objects":88703,"num_object_clones":0,"num_object_copies":177406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88703,"num_write_kb":5676992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88703,"ondisk_log_size":88703,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16365656,"kb_used_data":11219832,"kb_used_omap":91,"kb_used_meta":5145572,"kb_avail":1108576168,"statfs":{"total":1151940427776,"available":1135181996032,"internally_reserved":0,"allocated":11489107968,"data_stored":11482983464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":5269065744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":959119360,"num_objects":14635,"num_object_clones":0,"num_object_copies":29270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14635,"num_write_kb":936640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001878"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":151,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.714943+0000","last_change":"2024-02-12T16:29:20.828257+0000","last_active":"2024-02-12T16:29:56.714943+0000","last_peered":"2024-02-12T16:29:56.714943+0000","last_clean":"2024-02-12T16:29:56.714943+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:29:56.714943+0000","last_undegraded":"2024-02-12T16:29:56.714943+0000","last_fullsized":"2024-02-12T16:29:56.714943+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:20.828149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:29:20.828149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"654'5566","reported_seq":5591,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:02.088247+0000","last_change":"2024-02-12T16:29:08.737178+0000","last_active":"2024-02-12T16:30:02.088247+0000","last_peered":"2024-02-12T16:30:02.088247+0000","last_clean":"2024-02-12T16:30:02.088247+0000","last_became_active":"2024-02-12T16:29:08.736890+0000","last_became_peered":"2024-02-12T16:29:08.736890+0000","last_unstale":"2024-02-12T16:30:02.088247+0000","last_undegraded":"2024-02-12T16:30:02.088247+0000","last_fullsized":"2024-02-12T16:30:02.088247+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5566,"ondisk_log_size":5566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":364773376,"num_objects":5566,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5566,"num_write_kb":356224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":263,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.720034+0000","last_change":"2024-02-12T16:28:53.018782+0000","last_active":"2024-02-12T16:29:56.720034+0000","last_peered":"2024-02-12T16:29:56.720034+0000","last_clean":"2024-02-12T16:29:56.720034+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:29:56.720034+0000","last_undegraded":"2024-02-12T16:29:56.720034+0000","last_fullsized":"2024-02-12T16:29:56.720034+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.018681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.018681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"653'5409","reported_seq":5432,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:59.365538+0000","last_change":"2024-02-12T16:29:08.734672+0000","last_active":"2024-02-12T16:29:59.365538+0000","last_peered":"2024-02-12T16:29:59.365538+0000","last_clean":"2024-02-12T16:29:59.365538+0000","last_became_active":"2024-02-12T16:29:08.734223+0000","last_became_peered":"2024-02-12T16:29:08.734223+0000","last_unstale":"2024-02-12T16:29:59.365538+0000","last_undegraded":"2024-02-12T16:29:59.365538+0000","last_fullsized":"2024-02-12T16:29:59.365538+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5409,"ondisk_log_size":5409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":354484224,"num_objects":5409,"num_object_clones":0,"num_object_copies":10818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5409,"num_write_kb":346176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":340,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.724126+0000","last_change":"2024-02-12T16:28:54.310544+0000","last_active":"2024-02-12T16:29:56.724126+0000","last_peered":"2024-02-12T16:29:56.724126+0000","last_clean":"2024-02-12T16:29:56.724126+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:29:56.724126+0000","last_undegraded":"2024-02-12T16:29:56.724126+0000","last_fullsized":"2024-02-12T16:29:56.724126+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.310448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.310448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"654'5642","reported_seq":5667,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:03.306291+0000","last_change":"2024-02-12T16:29:08.805574+0000","last_active":"2024-02-12T16:30:03.306291+0000","last_peered":"2024-02-12T16:30:03.306291+0000","last_clean":"2024-02-12T16:30:03.306291+0000","last_became_active":"2024-02-12T16:29:08.805299+0000","last_became_peered":"2024-02-12T16:29:08.805299+0000","last_unstale":"2024-02-12T16:30:03.306291+0000","last_undegraded":"2024-02-12T16:30:03.306291+0000","last_fullsized":"2024-02-12T16:30:03.306291+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5642,"ondisk_log_size":5642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369754112,"num_objects":5642,"num_object_clones":0,"num_object_copies":11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5642,"num_write_kb":361088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":743,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.498462+0000","last_change":"2024-02-12T16:28:50.494186+0000","last_active":"2024-02-12T16:29:50.498462+0000","last_peered":"2024-02-12T16:29:50.498462+0000","last_clean":"2024-02-12T16:29:50.498462+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:29:50.498462+0000","last_undegraded":"2024-02-12T16:29:50.498462+0000","last_fullsized":"2024-02-12T16:29:50.498462+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:50.494005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:50.494005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.0","version":"653'5394","reported_seq":5418,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:30:00.628255+0000","last_change":"2024-02-12T16:29:08.733321+0000","last_active":"2024-02-12T16:30:00.628255+0000","last_peered":"2024-02-12T16:30:00.628255+0000","last_clean":"2024-02-12T16:30:00.628255+0000","last_became_active":"2024-02-12T16:29:08.732997+0000","last_became_peered":"2024-02-12T16:29:08.732997+0000","last_unstale":"2024-02-12T16:30:00.628255+0000","last_undegraded":"2024-02-12T16:30:00.628255+0000","last_fullsized":"2024-02-12T16:30:00.628255+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5394,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":353501184,"num_objects":5394,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":345216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":383,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.114601+0000","last_change":"2024-02-12T16:28:53.942753+0000","last_active":"2024-02-12T16:29:50.114601+0000","last_peered":"2024-02-12T16:29:50.114601+0000","last_clean":"2024-02-12T16:29:50.114601+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:29:50.114601+0000","last_undegraded":"2024-02-12T16:29:50.114601+0000","last_fullsized":"2024-02-12T16:29:50.114601+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.942656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.942656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"654'5891","reported_seq":5916,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:03.816829+0000","last_change":"2024-02-12T16:29:08.809293+0000","last_active":"2024-02-12T16:30:03.816829+0000","last_peered":"2024-02-12T16:30:03.816829+0000","last_clean":"2024-02-12T16:30:03.816829+0000","last_became_active":"2024-02-12T16:29:08.808944+0000","last_became_peered":"2024-02-12T16:29:08.808944+0000","last_unstale":"2024-02-12T16:30:03.816829+0000","last_undegraded":"2024-02-12T16:30:03.816829+0000","last_fullsized":"2024-02-12T16:30:03.816829+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5891,"ondisk_log_size":5891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":386072576,"num_objects":5891,"num_object_clones":0,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5891,"num_write_kb":377024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":331,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.724175+0000","last_change":"2024-02-12T16:28:56.397511+0000","last_active":"2024-02-12T16:29:56.724175+0000","last_peered":"2024-02-12T16:29:56.724175+0000","last_clean":"2024-02-12T16:29:56.724175+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:29:56.724175+0000","last_undegraded":"2024-02-12T16:29:56.724175+0000","last_fullsized":"2024-02-12T16:29:56.724175+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:56.397266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:56.397266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"653'5332","reported_seq":5355,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:30:00.252890+0000","last_change":"2024-02-12T16:29:08.733146+0000","last_active":"2024-02-12T16:30:00.252890+0000","last_peered":"2024-02-12T16:30:00.252890+0000","last_clean":"2024-02-12T16:30:00.252890+0000","last_became_active":"2024-02-12T16:29:08.732848+0000","last_became_peered":"2024-02-12T16:29:08.732848+0000","last_unstale":"2024-02-12T16:30:00.252890+0000","last_undegraded":"2024-02-12T16:30:00.252890+0000","last_fullsized":"2024-02-12T16:30:00.252890+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5332,"ondisk_log_size":5332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":349437952,"num_objects":5332,"num_object_clones":0,"num_object_copies":10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5332,"num_write_kb":341248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":341,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.724088+0000","last_change":"2024-02-12T16:28:58.383870+0000","last_active":"2024-02-12T16:29:56.724088+0000","last_peered":"2024-02-12T16:29:56.724088+0000","last_clean":"2024-02-12T16:29:56.724088+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:29:56.724088+0000","last_undegraded":"2024-02-12T16:29:56.724088+0000","last_fullsized":"2024-02-12T16:29:56.724088+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.383742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.383742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"654'5652","reported_seq":5677,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:01.799288+0000","last_change":"2024-02-12T16:29:08.734515+0000","last_active":"2024-02-12T16:30:01.799288+0000","last_peered":"2024-02-12T16:30:01.799288+0000","last_clean":"2024-02-12T16:30:01.799288+0000","last_became_active":"2024-02-12T16:29:08.734296+0000","last_became_peered":"2024-02-12T16:29:08.734296+0000","last_unstale":"2024-02-12T16:30:01.799288+0000","last_undegraded":"2024-02-12T16:30:01.799288+0000","last_fullsized":"2024-02-12T16:30:01.799288+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5652,"ondisk_log_size":5652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":370409472,"num_objects":5652,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":361728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":199,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.113758+0000","last_change":"2024-02-12T16:28:53.712516+0000","last_active":"2024-02-12T16:29:50.113758+0000","last_peered":"2024-02-12T16:29:50.113758+0000","last_clean":"2024-02-12T16:29:50.113758+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:29:50.113758+0000","last_undegraded":"2024-02-12T16:29:50.113758+0000","last_fullsized":"2024-02-12T16:29:50.113758+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.712421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.712421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"654'5793","reported_seq":5818,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:03.304225+0000","last_change":"2024-02-12T16:29:08.804530+0000","last_active":"2024-02-12T16:30:03.304225+0000","last_peered":"2024-02-12T16:30:03.304225+0000","last_clean":"2024-02-12T16:30:03.304225+0000","last_became_active":"2024-02-12T16:29:08.804218+0000","last_became_peered":"2024-02-12T16:29:08.804218+0000","last_unstale":"2024-02-12T16:30:03.304225+0000","last_undegraded":"2024-02-12T16:30:03.304225+0000","last_fullsized":"2024-02-12T16:30:03.304225+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5793,"ondisk_log_size":5793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":379650048,"num_objects":5793,"num_object_clones":0,"num_object_copies":11586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":370752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":772,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.719795+0000","last_change":"2024-02-12T16:28:53.646270+0000","last_active":"2024-02-12T16:29:56.719795+0000","last_peered":"2024-02-12T16:29:56.719795+0000","last_clean":"2024-02-12T16:29:56.719795+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:29:56.719795+0000","last_undegraded":"2024-02-12T16:29:56.719795+0000","last_fullsized":"2024-02-12T16:29:56.719795+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.646027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.646027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"654'5525","reported_seq":5550,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:02.108526+0000","last_change":"2024-02-12T16:29:08.804335+0000","last_active":"2024-02-12T16:30:02.108526+0000","last_peered":"2024-02-12T16:30:02.108526+0000","last_clean":"2024-02-12T16:30:02.108526+0000","last_became_active":"2024-02-12T16:29:08.804047+0000","last_became_peered":"2024-02-12T16:29:08.804047+0000","last_unstale":"2024-02-12T16:30:02.108526+0000","last_undegraded":"2024-02-12T16:30:02.108526+0000","last_fullsized":"2024-02-12T16:30:02.108526+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5525,"ondisk_log_size":5525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":362086400,"num_objects":5525,"num_object_clones":0,"num_object_copies":11050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":353600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":147,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.719947+0000","last_change":"2024-02-12T16:28:57.806094+0000","last_active":"2024-02-12T16:29:56.719947+0000","last_peered":"2024-02-12T16:29:56.719947+0000","last_clean":"2024-02-12T16:29:56.719947+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:29:56.719947+0000","last_undegraded":"2024-02-12T16:29:56.719947+0000","last_fullsized":"2024-02-12T16:29:56.719947+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:57.806002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:57.806002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"653'5410","reported_seq":5433,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:30:00.244427+0000","last_change":"2024-02-12T16:29:08.734597+0000","last_active":"2024-02-12T16:30:00.244427+0000","last_peered":"2024-02-12T16:30:00.244427+0000","last_clean":"2024-02-12T16:30:00.244427+0000","last_became_active":"2024-02-12T16:29:08.734174+0000","last_became_peered":"2024-02-12T16:29:08.734174+0000","last_unstale":"2024-02-12T16:30:00.244427+0000","last_undegraded":"2024-02-12T16:30:00.244427+0000","last_fullsized":"2024-02-12T16:30:00.244427+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5410,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":354549760,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":346240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":171,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.115205+0000","last_change":"2024-02-12T16:28:51.703479+0000","last_active":"2024-02-12T16:29:50.115205+0000","last_peered":"2024-02-12T16:29:50.115205+0000","last_clean":"2024-02-12T16:29:50.115205+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:29:50.115205+0000","last_undegraded":"2024-02-12T16:29:50.115205+0000","last_fullsized":"2024-02-12T16:29:50.115205+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:51.703298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:51.703298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"654'5669","reported_seq":5693,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:03.300467+0000","last_change":"2024-02-12T16:29:08.734976+0000","last_active":"2024-02-12T16:30:03.300467+0000","last_peered":"2024-02-12T16:30:03.300467+0000","last_clean":"2024-02-12T16:30:03.300467+0000","last_became_active":"2024-02-12T16:29:08.734560+0000","last_became_peered":"2024-02-12T16:29:08.734560+0000","last_unstale":"2024-02-12T16:30:03.300467+0000","last_undegraded":"2024-02-12T16:30:03.300467+0000","last_fullsized":"2024-02-12T16:30:03.300467+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5669,"ondisk_log_size":5669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":371523584,"num_objects":5669,"num_object_clones":0,"num_object_copies":11338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":362816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":344,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.114530+0000","last_change":"2024-02-12T16:28:55.989602+0000","last_active":"2024-02-12T16:29:50.114530+0000","last_peered":"2024-02-12T16:29:50.114530+0000","last_clean":"2024-02-12T16:29:50.114530+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:29:50.114530+0000","last_undegraded":"2024-02-12T16:29:50.114530+0000","last_fullsized":"2024-02-12T16:29:50.114530+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:55.989476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:28:55.989476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"653'5501","reported_seq":5525,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:30:00.617458+0000","last_change":"2024-02-12T16:29:08.807541+0000","last_active":"2024-02-12T16:30:00.617458+0000","last_peered":"2024-02-12T16:30:00.617458+0000","last_clean":"2024-02-12T16:30:00.617458+0000","last_became_active":"2024-02-12T16:29:08.807333+0000","last_became_peered":"2024-02-12T16:29:08.807333+0000","last_unstale":"2024-02-12T16:30:00.617458+0000","last_undegraded":"2024-02-12T16:30:00.617458+0000","last_fullsized":"2024-02-12T16:30:00.617458+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5501,"ondisk_log_size":5501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":360513536,"num_objects":5501,"num_object_clones":0,"num_object_copies":11002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5501,"num_write_kb":352064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":146,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.116300+0000","last_change":"2024-02-12T16:28:54.613666+0000","last_active":"2024-02-12T16:29:50.116300+0000","last_peered":"2024-02-12T16:29:50.116300+0000","last_clean":"2024-02-12T16:29:50.116300+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:29:50.116300+0000","last_undegraded":"2024-02-12T16:29:50.116300+0000","last_fullsized":"2024-02-12T16:29:50.116300+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.613501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.613501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"653'5261","reported_seq":5284,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:59.355250+0000","last_change":"2024-02-12T16:29:08.736225+0000","last_active":"2024-02-12T16:29:59.355250+0000","last_peered":"2024-02-12T16:29:59.355250+0000","last_clean":"2024-02-12T16:29:59.355250+0000","last_became_active":"2024-02-12T16:29:08.735885+0000","last_became_peered":"2024-02-12T16:29:08.735885+0000","last_unstale":"2024-02-12T16:29:59.355250+0000","last_undegraded":"2024-02-12T16:29:59.355250+0000","last_fullsized":"2024-02-12T16:29:59.355250+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5261,"ondisk_log_size":5261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":344784896,"num_objects":5261,"num_object_clones":0,"num_object_copies":10522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5261,"num_write_kb":336704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":747,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.719785+0000","last_change":"2024-02-12T16:28:58.797711+0000","last_active":"2024-02-12T16:29:56.719785+0000","last_peered":"2024-02-12T16:29:56.719785+0000","last_clean":"2024-02-12T16:29:56.719785+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:29:56.719785+0000","last_undegraded":"2024-02-12T16:29:56.719785+0000","last_fullsized":"2024-02-12T16:29:56.719785+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.797578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.797578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"654'5599","reported_seq":5623,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:01.417167+0000","last_change":"2024-02-12T16:29:08.834927+0000","last_active":"2024-02-12T16:30:01.417167+0000","last_peered":"2024-02-12T16:30:01.417167+0000","last_clean":"2024-02-12T16:30:01.417167+0000","last_became_active":"2024-02-12T16:29:08.834297+0000","last_became_peered":"2024-02-12T16:29:08.834297+0000","last_unstale":"2024-02-12T16:30:01.417167+0000","last_undegraded":"2024-02-12T16:30:01.417167+0000","last_fullsized":"2024-02-12T16:30:01.417167+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5599,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":366936064,"num_objects":5599,"num_object_clones":0,"num_object_copies":11198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":358336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":334,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.721662+0000","last_change":"2024-02-12T16:28:53.361716+0000","last_active":"2024-02-12T16:29:56.721662+0000","last_peered":"2024-02-12T16:29:56.721662+0000","last_clean":"2024-02-12T16:29:56.721662+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:29:56.721662+0000","last_undegraded":"2024-02-12T16:29:56.721662+0000","last_fullsized":"2024-02-12T16:29:56.721662+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:53.361610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:53.361610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"653'5461","reported_seq":5484,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:30:00.222343+0000","last_change":"2024-02-12T16:29:08.805152+0000","last_active":"2024-02-12T16:30:00.222343+0000","last_peered":"2024-02-12T16:30:00.222343+0000","last_clean":"2024-02-12T16:30:00.222343+0000","last_became_active":"2024-02-12T16:29:08.804528+0000","last_became_peered":"2024-02-12T16:29:08.804528+0000","last_unstale":"2024-02-12T16:30:00.222343+0000","last_undegraded":"2024-02-12T16:30:00.222343+0000","last_fullsized":"2024-02-12T16:30:00.222343+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5461,"ondisk_log_size":5461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":357892096,"num_objects":5461,"num_object_clones":0,"num_object_copies":10922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5461,"num_write_kb":349504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":85,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.498396+0000","last_change":"2024-02-12T16:28:59.764005+0000","last_active":"2024-02-12T16:29:50.498396+0000","last_peered":"2024-02-12T16:29:50.498396+0000","last_clean":"2024-02-12T16:29:50.498396+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:29:50.498396+0000","last_undegraded":"2024-02-12T16:29:50.498396+0000","last_fullsized":"2024-02-12T16:29:50.498396+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:59.763866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:59.763866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"654'5598","reported_seq":5622,"reported_epoch":654,"state":"active+clean","last_fresh":"2024-02-12T16:30:02.092952+0000","last_change":"2024-02-12T16:29:08.735934+0000","last_active":"2024-02-12T16:30:02.092952+0000","last_peered":"2024-02-12T16:30:02.092952+0000","last_clean":"2024-02-12T16:30:02.092952+0000","last_became_active":"2024-02-12T16:29:08.735711+0000","last_became_peered":"2024-02-12T16:29:08.735711+0000","last_unstale":"2024-02-12T16:30:02.092952+0000","last_undegraded":"2024-02-12T16:30:02.092952+0000","last_fullsized":"2024-02-12T16:30:02.092952+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":5598,"ondisk_log_size":5598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":366870528,"num_objects":5598,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5598,"num_write_kb":358272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":322,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.720137+0000","last_change":"2024-02-12T16:28:54.979306+0000","last_active":"2024-02-12T16:29:56.720137+0000","last_peered":"2024-02-12T16:29:56.720137+0000","last_clean":"2024-02-12T16:29:56.720137+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:29:56.720137+0000","last_undegraded":"2024-02-12T16:29:56.720137+0000","last_fullsized":"2024-02-12T16:29:56.720137+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.979211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.979211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":143,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.721579+0000","last_change":"2024-02-12T16:28:59.370513+0000","last_active":"2024-02-12T16:29:56.721579+0000","last_peered":"2024-02-12T16:29:56.721579+0000","last_clean":"2024-02-12T16:29:56.721579+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:29:56.721579+0000","last_undegraded":"2024-02-12T16:29:56.721579+0000","last_fullsized":"2024-02-12T16:29:56.721579+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:59.370403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:28:59.370403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":141,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-02-12T16:29:50.117269+0000","last_change":"2024-02-12T16:28:55.212658+0000","last_active":"2024-02-12T16:29:50.117269+0000","last_peered":"2024-02-12T16:29:50.117269+0000","last_clean":"2024-02-12T16:29:50.117269+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:29:50.117269+0000","last_undegraded":"2024-02-12T16:29:50.117269+0000","last_fullsized":"2024-02-12T16:29:50.117269+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:55.212566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:55.212566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":157,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.721676+0000","last_change":"2024-02-12T16:28:56.394900+0000","last_active":"2024-02-12T16:29:56.721676+0000","last_peered":"2024-02-12T16:29:56.721676+0000","last_clean":"2024-02-12T16:29:56.721676+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:29:56.721676+0000","last_undegraded":"2024-02-12T16:29:56.721676+0000","last_fullsized":"2024-02-12T16:29:56.721676+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:56.394801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:28:56.394801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":136,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-02-12T16:29:56.718433+0000","last_change":"2024-02-12T16:28:58.060344+0000","last_active":"2024-02-12T16:29:56.718433+0000","last_peered":"2024-02-12T16:29:56.718433+0000","last_clean":"2024-02-12T16:29:56.718433+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:29:56.718433+0000","last_undegraded":"2024-02-12T16:29:56.718433+0000","last_fullsized":"2024-02-12T16:29:56.718433+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.060250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.060250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5813239808,"num_objects":88703,"num_object_clones":0,"num_object_copies":177406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88703,"num_write_kb":5676992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11600658432,"data_stored":11600658432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88703,"ondisk_log_size":88703,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586604,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":93158548,"statfs":{"total":95995035648,"available":95394353152,"internally_reserved":0,"allocated":1019904,"data_stored":512552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.80500000000000005,"15min":0.85499999999999998},"min":{"1min":0.315,"5min":0.246,"15min":0.246},"max":{"1min":1.5029999999999999,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.94799999999999995,"15min":0.90900000000000003},"min":{"1min":0.39000000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.395,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.29299999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.93400000000000005,"15min":1.0620000000000001},"min":{"1min":0.30099999999999999,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":1.385,"5min":5.5430000000000001,"15min":7.0110000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.89000000000000001,"15min":1.0640000000000001},"min":{"1min":0.36499999999999999,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":1.4830000000000001,"5min":3.6859999999999999,"15min":5.6920000000000002},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:29:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.83699999999999997,"15min":1.2450000000000001},"min":{"1min":0.46899999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.4379999999999999,"5min":4.1639999999999997,"15min":5.343},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.81699999999999995,"15min":1.3},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.268,"5min":5.4340000000000002,"15min":8.2759999999999998},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.874,"15min":1.034},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":1.8009999999999999,"5min":4.9900000000000002,"15min":6.5970000000000004},"last":2.5310000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.86799999999999999,"15min":0.93500000000000005},"min":{"1min":0.371,"5min":0.36799999999999999,"15min":0.34499999999999997},"max":{"1min":1.7350000000000001,"5min":4.101,"15min":4.915},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.125,"15min":1.1499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.35099999999999998},"max":{"1min":1.7769999999999999,"5min":5.04,"15min":5.8789999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.024,"15min":1.1319999999999999},"min":{"1min":0.498,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":1.794,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.90400000000000003,"15min":0.98799999999999999},"min":{"1min":0.45200000000000001,"5min":0.378,"15min":0.373},"max":{"1min":1.522,"5min":4.7370000000000001,"15min":5.141},"last":0.749},{"interface":"front","average":{"1min":1.274,"5min":1.026,"15min":1.125},"min":{"1min":0.54100000000000004,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.905,"5min":4.7450000000000001,"15min":8.5709999999999997},"last":1.677}]},{"osd":7,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.91200000000000003,"15min":1.1859999999999999},"min":{"1min":0.46899999999999997,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.718,"5min":3.9089999999999998,"15min":6.4370000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.84999999999999998,"15min":1.1160000000000001},"min":{"1min":0.52700000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.867,"5min":4.2350000000000003,"15min":5.0430000000000001},"last":2.3319999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":6.2039999999999997,"15min":2.7970000000000002},"min":{"1min":0.45600000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.625,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.27,"5min":6.319,"15min":2.9689999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.34699999999999998},"max":{"1min":1.722,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.67400000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98099999999999998,"15min":1.276},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.736,"5min":3.8100000000000001,"15min":10.743},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.397,"5min":0.998,"15min":1.357},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7809999999999999,"5min":3.8399999999999999,"15min":10.698},"last":0.76900000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":6.327,"15min":2.9590000000000001},"min":{"1min":0.50900000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.6850000000000001,"5min":536.245,"15min":536.245},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.2929999999999999,"5min":6.3339999999999996,"15min":2.9399999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.901,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.63500000000000001}]}]},{"osd":10,"up_from":83,"seq":356482285960,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676472,"kb_used_data":371056,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93068680,"statfs":{"total":95995035648,"available":95302328320,"internally_reserved":0,"allocated":379961344,"data_stored":379447932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":2.778,"15min":1.55},"min":{"1min":0.33400000000000002,"5min":0.245,"15min":0.245},"max":{"1min":15.010999999999999,"5min":162.38399999999999,"15min":162.38399999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.8109999999999999,"5min":2.8700000000000001,"15min":1.6339999999999999},"min":{"1min":0.34000000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":14.129,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.376}]},{"osd":1,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.008,"15min":1.006},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":1.6140000000000001,"5min":5.9000000000000004,"15min":19.866},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.999,"15min":1.1399999999999999},"min":{"1min":0.372,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":1.504,"5min":6.1970000000000001,"15min":20.277999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.1990000000000001,"15min":1.0009999999999999},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.28899999999999998},"max":{"1min":1.7230000000000001,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.095,"15min":1.03},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":1.77,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.57499999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.93200000000000005,"15min":1.034},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32300000000000001},"max":{"1min":2.9249999999999998,"5min":5.8109999999999999,"15min":5.8109999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.101,"15min":1.0469999999999999},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":5.9119999999999999,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":5.9119999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.1419999999999999,"15min":1.095},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.6349999999999998,"5min":6.343,"15min":6.343},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.196,"15min":1.196},"min":{"1min":0.54000000000000004,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":3.7759999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":3.7759999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0569999999999999,"15min":1.093},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":3.9100000000000001,"5min":6.3010000000000002,"15min":6.3010000000000002},"last":3.9100000000000001},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.0940000000000001,"15min":1.1739999999999999},"min":{"1min":0.443,"5min":0.376,"15min":0.376},"max":{"1min":2.1520000000000001,"5min":6.8200000000000003,"15min":6.8200000000000003},"last":0.621}]},{"osd":7,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.018,"15min":1.0900000000000001},"min":{"1min":0.433,"5min":0.375,"15min":0.375},"max":{"1min":2.484,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.5},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.177,"15min":1.2170000000000001},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":2.0859999999999999,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.56299999999999994}]},{"osd":8,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.054,"15min":1.1479999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.372},"max":{"1min":2.3300000000000001,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.99299999999999999,"15min":1.1359999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.35799999999999998},"max":{"1min":1.8129999999999999,"5min":7.282,"15min":7.282},"last":0.58599999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1859999999999999,"15min":1.2569999999999999},"min":{"1min":0.51300000000000001,"5min":0.48499999999999999,"15min":0.372},"max":{"1min":2.1499999999999999,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0449999999999999,"15min":1.218},"min":{"1min":0.51400000000000001,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":2.23,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.628}]},{"osd":11,"last update":"Mon Feb 12 16:29:30 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.1060000000000001,"15min":1.2809999999999999},"min":{"1min":0.67400000000000004,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":2.1219999999999999,"5min":7.3689999999999998,"15min":10.218999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.552,"5min":1.115,"15min":1.296},"min":{"1min":0.63900000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.2189999999999999,"5min":6.5860000000000003,"15min":7.3250000000000002},"last":0.69599999999999995}]}]},{"osd":9,"up_from":74,"seq":317827580304,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1011512,"kb_used_data":710192,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92733640,"statfs":{"total":95995035648,"available":94959247360,"internally_reserved":0,"allocated":727236608,"data_stored":726723204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.2350000000000001,"15min":1.1160000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.77,"5min":7.9560000000000004,"15min":7.9560000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.234,"15min":1.0109999999999999},"min":{"1min":0.40500000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.3679999999999999,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.33200000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1160000000000001,"15min":1.022},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.253},"max":{"1min":3.8180000000000001,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.0600000000000001,"15min":1.079},"min":{"1min":0.32500000000000001,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":2.2789999999999999,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.2649999999999999,"15min":1.4530000000000001},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":3.298,"5min":12.441000000000001,"15min":97.275000000000006},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.1659999999999999,"15min":1.3640000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.312},"max":{"1min":1.728,"5min":6.1779999999999999,"15min":97.417000000000002},"last":1.728}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0920000000000001,"15min":1.032},"min":{"1min":0.373,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.6600000000000001,"5min":6.3090000000000002,"15min":6.3090000000000002},"last":0.44},{"interface":"front","average":{"1min":1.113,"5min":1.155,"15min":1.2030000000000001},"min":{"1min":0.49099999999999999,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":3.6299999999999999,"5min":6.3490000000000002,"15min":6.3490000000000002},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.3049999999999999,"15min":1.1819999999999999},"min":{"1min":0.54500000000000004,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":1.468,"5min":6.726,"15min":6.726},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.204,"15min":1.159},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.6659999999999999,"5min":6.7350000000000003,"15min":6.7350000000000003},"last":0.66800000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.998,"15min":1.0940000000000001},"min":{"1min":0.38800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":5.6689999999999996,"5min":5.6689999999999996,"15min":6.1840000000000002},"last":0.621},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.264,"15min":1.145},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.36499999999999999},"max":{"1min":5.6920000000000002,"5min":14.718,"15min":14.718},"last":0.52900000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1599999999999999,"15min":1.2669999999999999},"min":{"1min":0.60999999999999999,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":1.8109999999999999,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.024,"5min":1.143,"15min":1.2110000000000001},"min":{"1min":0.53800000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.6479999999999999,"5min":4.7000000000000002,"15min":7.9160000000000004},"last":0.57099999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.95599999999999996,"15min":1.0940000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.105,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.272,"5min":1.0820000000000001,"15min":1.204},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42599999999999999},"max":{"1min":3.806,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.56000000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.1399999999999999,"15min":1.262},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.36599999999999999},"max":{"1min":1.913,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":0.504},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2010000000000001,"15min":1.27},"min":{"1min":0.44700000000000001,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":1.8999999999999999,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":0.59499999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:29:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.1319999999999999,"15min":1.3600000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.45600000000000002},"max":{"1min":2.238,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":0.751},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.0389999999999999,"15min":1.2849999999999999},"min":{"1min":0.53400000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.78,"5min":4.7190000000000003,"15min":6.3639999999999999},"last":0.76000000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874650,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1048312,"kb_used_data":738800,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92696840,"statfs":{"total":95995035648,"available":94921564160,"internally_reserved":0,"allocated":756531200,"data_stored":756017796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.077,"15min":0.90800000000000003},"min":{"1min":0.63500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":2.71,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.2430000000000001,"15min":0.93200000000000005},"min":{"1min":0.63800000000000001,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":2.5219999999999998,"5min":18.584,"15min":18.584},"last":0.38700000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.9810000000000001,"15min":1.5249999999999999},"min":{"1min":0.72199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.181,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.9670000000000001,"15min":1.6220000000000001},"min":{"1min":0.65700000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.3919999999999999,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.85099999999999998,"15min":0.84399999999999997},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.903,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.012,"15min":1.018},"min":{"1min":0.40300000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.754,"15min":1.708},"min":{"1min":0.46300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.458,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.84999999999999998,"15min":1.681},"min":{"1min":0.57899999999999996,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":2.0110000000000001,"5min":5.0380000000000003,"15min":224.71000000000001},"last":0.64700000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.88400000000000001,"15min":0.92000000000000004},"min":{"1min":0.51700000000000002,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":1.4259999999999999,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":3.198},{"interface":"front","average":{"1min":1.345,"5min":0.95199999999999996,"15min":1.0229999999999999},"min":{"1min":0.53200000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.097,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.6890000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.0409999999999999,"15min":1.758},"min":{"1min":0.55100000000000005,"5min":0.373,"15min":0.37},"max":{"1min":1.5609999999999999,"5min":5.3639999999999999,"15min":221.36699999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.0189999999999999,"15min":1.8600000000000001},"min":{"1min":0.625,"5min":0.38,"15min":0.38},"max":{"1min":2.2269999999999999,"5min":4.4930000000000003,"15min":221.203},"last":0.69599999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.0629999999999999,"15min":1.1299999999999999},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":1.742,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.119,"15min":1.2509999999999999},"min":{"1min":0.64800000000000002,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":2.1709999999999998,"5min":4.2240000000000002,"15min":7.899},"last":1.639}]},{"osd":9,"last update":"Mon Feb 12 16:29:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.028,"15min":1.167},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.7030000000000001,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":0.745},{"interface":"front","average":{"1min":1.351,"5min":1.1160000000000001,"15min":1.2490000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":1.9299999999999999,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":0.78000000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":0.95199999999999996,"15min":1.4430000000000001},"min":{"1min":0.50900000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.056,"5min":5.2560000000000002,"15min":109.92},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.25,"5min":0.99099999999999999,"15min":1.488},"min":{"1min":0.55200000000000005,"5min":0.44400000000000001,"15min":0.38500000000000001},"max":{"1min":2.6880000000000002,"5min":5.0250000000000004,"15min":110.249},"last":0.623}]},{"osd":11,"last update":"Mon Feb 12 16:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.0229999999999999,"15min":1.179},"min":{"1min":0.64200000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.724,"5min":5.2530000000000001,"15min":5.2530000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.2969999999999999,"5min":0.93500000000000005,"15min":1.1359999999999999},"min":{"1min":0.57999999999999996,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.6859999999999999,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":0.67400000000000004}]}]},{"osd":7,"up_from":56,"seq":240518168993,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753144,"kb_used_data":1443632,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91992008,"statfs":{"total":95995035648,"available":94199816192,"internally_reserved":0,"allocated":1478279168,"data_stored":1477765764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":3.1360000000000001,"15min":1.6819999999999999},"min":{"1min":0.39500000000000002,"5min":0.252,"15min":0.252},"max":{"1min":2.1120000000000001,"5min":187.33699999999999,"15min":187.33699999999999},"last":3.3180000000000001},{"interface":"front","average":{"1min":1.075,"5min":3.0739999999999998,"15min":1.679},"min":{"1min":0.72799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1850000000000001,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.2749999999999999,"15min":1.0669999999999999},"min":{"1min":0.42599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.6000000000000001,"5min":35.655000000000001,"15min":35.655000000000001},"last":2.6640000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.5720000000000001,"15min":1.228},"min":{"1min":0.47499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.145,"5min":35.164000000000001,"15min":35.164000000000001},"last":0.64600000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78100000000000003,"15min":1.159},"min":{"1min":0.379,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.238,"5min":4.5140000000000002,"15min":7.1689999999999996},"last":2.1110000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86199999999999999,"15min":0.97399999999999998},"min":{"1min":0.39700000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.3759999999999999,"5min":6.7549999999999999,"15min":6.7549999999999999},"last":2.7050000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1339999999999999,"15min":0.97099999999999997},"min":{"1min":0.41099999999999998,"5min":0.27400000000000002,"15min":0.26200000000000001},"max":{"1min":2.766,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.501},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.95699999999999996,"15min":1.014},"min":{"1min":0.63600000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":1.5209999999999999,"5min":6.9470000000000001,"15min":6.9470000000000001},"last":2.621}]},{"osd":5,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.018,"15min":0.98499999999999999},"min":{"1min":0.64700000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.4029999999999996,"5min":7.4029999999999996,"15min":7.4029999999999996},"last":2.411},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.095,"15min":1.0569999999999999},"min":{"1min":0.59699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.2549999999999999,"5min":5.992,"15min":9.3640000000000008},"last":0.755}]},{"osd":6,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.05,"15min":1.077},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":1.7090000000000001,"5min":6.319,"15min":6.319},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.363,"5min":1.1819999999999999,"15min":1.2},"min":{"1min":0.72499999999999998,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":3.169,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":2.7509999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":6.5270000000000001,"15min":2.9329999999999998},"min":{"1min":0.46999999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.6719999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.5900000000000001,"5min":6.641,"15min":3.0590000000000002},"min":{"1min":0.434,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":5.5359999999999996,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.68899999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:29:04 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.006,"15min":1.181},"min":{"1min":0.51600000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":12.907999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.484,"5min":1.1950000000000001,"15min":1.2949999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":7.1219999999999999,"5min":7.1219999999999999,"15min":7.1219999999999999},"last":2.8109999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":7.8109999999999999,"15min":3.5379999999999998},"min":{"1min":0.85899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":3.367,"5min":497.52699999999999,"15min":497.52699999999999},"last":2.7349999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":7.7069999999999999,"15min":3.4359999999999999},"min":{"1min":0.81999999999999995,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":2.2400000000000002,"5min":496.29599999999999,"15min":496.29599999999999},"last":0.73199999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0409999999999999,"15min":1.5940000000000001},"min":{"1min":0.626,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.847,"5min":4.9039999999999999,"15min":120.943},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.329,"5min":1.2410000000000001,"15min":1.708},"min":{"1min":0.56999999999999995,"5min":0.46899999999999997,"15min":0.45300000000000001},"max":{"1min":4.181,"5min":7.0679999999999996,"15min":120.661},"last":2.7799999999999998}]}]},{"osd":6,"up_from":48,"seq":206158430633,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1673644,"kb_used_data":1364132,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92071508,"statfs":{"total":95995035648,"available":94281224192,"internally_reserved":0,"allocated":1396871168,"data_stored":1396363824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":2.1219999999999999,"15min":1.4730000000000001},"min":{"1min":0.309,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.7889999999999997,"5min":107.723,"15min":107.723},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.99299999999999999,"5min":2.1680000000000001,"15min":1.544},"min":{"1min":0.27700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":2.9940000000000002,"5min":107.114,"15min":107.114},"last":0.626}]},{"osd":1,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.0109999999999999,"15min":1.9139999999999999},"min":{"1min":0.29199999999999998,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":1.7,"5min":5.5179999999999998,"15min":252.22800000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.90900000000000003,"15min":1.956},"min":{"1min":0.29199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.2410000000000001,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.464,"15min":1.0640000000000001},"min":{"1min":0.35499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.0089999999999999,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.6739999999999999,"15min":1.1870000000000001},"min":{"1min":0.39500000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.3929999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.46500000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.115,"15min":1.0309999999999999},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.6299999999999999,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.90500000000000003,"15min":1.0629999999999999},"min":{"1min":0.34399999999999997,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.5469999999999999,"5min":6.0259999999999998,"15min":7.5979999999999999},"last":0.71899999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.071,"15min":0.98899999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":2.9279999999999999,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.96699999999999997,"15min":1.0029999999999999},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.712,"5min":10.52,"15min":10.52},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.90300000000000002,"15min":1.073},"min":{"1min":0.37,"5min":0.312,"15min":0.30199999999999999},"max":{"1min":1.403,"5min":4.0970000000000004,"15min":7.641},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.81699999999999995,"15min":1.0389999999999999},"min":{"1min":0.51600000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.673,"5min":2.8210000000000002,"15min":7.8380000000000001},"last":0.69899999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.139,"15min":1.139},"min":{"1min":0.436,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.21,"5min":6.21,"15min":6.21},"last":0.753},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.98599999999999999,"15min":1.1859999999999999},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":6.2270000000000003,"5min":6.2270000000000003,"15min":6.2270000000000003},"last":0.505}]},{"osd":9,"last update":"Mon Feb 12 16:29:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0289999999999999,"15min":1.2350000000000001},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.5820000000000001,"5min":4.1680000000000001,"15min":18.059999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.3220000000000001,"5min":0.95099999999999996,"15min":1.256},"min":{"1min":0.46000000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.694,"5min":2.694,"15min":17.425000000000001},"last":0.65100000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.218,"15min":1.2869999999999999},"min":{"1min":0.54200000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.776,"5min":5.726,"15min":6.5300000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.306,"5min":1.3089999999999999,"15min":1.3819999999999999},"min":{"1min":0.69799999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.8320000000000001,"5min":5.7590000000000003,"15min":6.5730000000000004},"last":0.81999999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.222,"15min":2.0960000000000001},"min":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.499},"max":{"1min":1.7370000000000001,"5min":5.6859999999999999,"15min":139.727},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":0.98699999999999999,"15min":2.0049999999999999},"min":{"1min":0.64900000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.9430000000000001,"5min":2.706,"15min":139.24199999999999},"last":0.67700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575311,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1964856,"kb_used_data":1070576,"kb_used_omap":0,"kb_used_meta":894208,"kb_avail":91780296,"statfs":{"total":95995035648,"available":93983023104,"internally_reserved":0,"allocated":1096269824,"data_stored":1095756420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915668992},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":1.095,"15min":0.96599999999999997},"min":{"1min":0.27900000000000003,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":1.135,"5min":16.632999999999999,"15min":16.632999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.88,"5min":1.6339999999999999,"15min":1.2170000000000001},"min":{"1min":0.27600000000000002,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":1.7549999999999999,"5min":7.7249999999999996,"15min":11.433},"last":0.308}]},{"osd":2,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.3580000000000001,"15min":1.1339999999999999},"min":{"1min":0.52600000000000002,"5min":0.29599999999999999,"15min":0.248},"max":{"1min":1.552,"5min":12.778,"15min":12.778},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.048,"5min":1.5860000000000001,"15min":1.2190000000000001},"min":{"1min":0.55300000000000005,"5min":0.246,"15min":0.246},"max":{"1min":2.4169999999999998,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:29:14 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.1419999999999999,"15min":1.137},"min":{"1min":0.47999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":4.1340000000000003,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.208,"15min":0.95999999999999996},"min":{"1min":0.42599999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.8420000000000001,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:29:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.3049999999999999,"15min":1.133},"min":{"1min":0.47199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":2.056,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.369,"15min":1.373},"min":{"1min":0.436,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.78,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":3.5510000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.514,"15min":1.282},"min":{"1min":0.67600000000000005,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.8169999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.403,"5min":1.3680000000000001,"15min":1.2310000000000001},"min":{"1min":0.79100000000000004,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":2.3159999999999998,"5min":10.067,"15min":10.067},"last":0.505}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.712,"15min":1.4059999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.9670000000000001,"5min":30.437000000000001,"15min":30.437000000000001},"last":3.105},{"interface":"front","average":{"1min":1.294,"5min":1.5740000000000001,"15min":1.4610000000000001},"min":{"1min":0.47599999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.845,"5min":29.382000000000001,"15min":29.382000000000001},"last":0.63400000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":7.4509999999999996,"15min":3.464},"min":{"1min":0.42999999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.2549999999999999,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.333,"5min":7.5389999999999997,"15min":3.427},"min":{"1min":0.51500000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.0830000000000002,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.66700000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.419,"15min":1.393},"min":{"1min":0.48199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.9930000000000001,"5min":5.7590000000000003,"15min":10.07},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3420000000000001,"15min":1.4650000000000001},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.436},"max":{"1min":2.1920000000000002,"5min":10.106,"15min":10.106},"last":0.753}]},{"osd":10,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":8.4819999999999993,"15min":3.7959999999999998},"min":{"1min":0.59399999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.673,"5min":548.93499999999995,"15min":548.93499999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.5800000000000001,"5min":8.4350000000000005,"15min":3.8559999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.4929999999999999,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.59499999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.538,"15min":1.5920000000000001},"min":{"1min":1.159,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4689999999999999,"5min":10.202,"15min":10.202},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.597,"15min":1.6399999999999999},"min":{"1min":0.98399999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.387,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":0.64200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738842,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1282924,"kb_used_data":697316,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92462228,"statfs":{"total":95995035648,"available":94681321472,"internally_reserved":0,"allocated":714051584,"data_stored":713544240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.141,"15min":1.073},"min":{"1min":0.29099999999999998,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":3.278,"5min":10.754,"15min":11.324},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.3300000000000001,"15min":1.171},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.0609999999999999,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.78200000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.022,"15min":1.044},"min":{"1min":0.38900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.6419999999999999,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.157,"15min":1.0009999999999999},"min":{"1min":0.316,"5min":0.224,"15min":0.224},"max":{"1min":3.5259999999999998,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.40899999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":3.4630000000000001,"15min":1.8500000000000001},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":3.347,"5min":193.77000000000001,"15min":193.77000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.365,"5min":3.472,"15min":1.95},"min":{"1min":0.38,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":5.9180000000000001,"5min":189.625,"15min":189.625},"last":2.0640000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.3149999999999999,"15min":1.0589999999999999},"min":{"1min":0.439,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.236,"5min":10.49,"15min":10.49},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.304,"15min":1.083},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.31},"max":{"1min":1.369,"5min":7.3250000000000002,"15min":7.3250000000000002},"last":0.39000000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":3.145,"15min":2.0019999999999998},"min":{"1min":0.374,"5min":0.311,"15min":0.311},"max":{"1min":7.3600000000000003,"5min":189.79400000000001,"15min":189.79400000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.367,"5min":3.2389999999999999,"15min":2.0790000000000002},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":7.2930000000000001,"5min":194.41200000000001,"15min":194.41200000000001},"last":0.48699999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":3.629,"15min":1.9750000000000001},"min":{"1min":0.53700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":4.0380000000000003,"5min":191.47200000000001,"15min":191.47200000000001},"last":5.6379999999999999},{"interface":"front","average":{"1min":1.5589999999999999,"5min":3.7250000000000001,"15min":2.0840000000000001},"min":{"1min":0.56799999999999995,"5min":0.44,"15min":0.44},"max":{"1min":4.0800000000000001,"5min":191.50299999999999,"15min":191.50299999999999},"last":5.6740000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:29:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3360000000000001,"15min":1.298},"min":{"1min":0.56699999999999995,"5min":0.442,"15min":0.42899999999999999},"max":{"1min":1.6519999999999999,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.432},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4379999999999999,"15min":1.3919999999999999},"min":{"1min":0.68600000000000005,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":1.996,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.52900000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:29:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.274,"15min":1.371},"min":{"1min":0.44800000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.7130000000000001,"5min":7.6539999999999999,"15min":46.246000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.397,"15min":1.446},"min":{"1min":0.64500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":7.5369999999999999,"5min":7.5369999999999999,"15min":46.167999999999999},"last":0.46100000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.4630000000000001,"15min":1.397},"min":{"1min":0.48499999999999999,"5min":0.442,"15min":0.41999999999999998},"max":{"1min":3.9169999999999998,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.353,"5min":1.587,"15min":1.409},"min":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":4.0069999999999997,"5min":9.3490000000000002,"15min":9.3490000000000002},"last":0.999}]},{"osd":11,"last update":"Mon Feb 12 16:29:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.518,"15min":1.569},"min":{"1min":0.69799999999999995,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":2.0819999999999999,"5min":6.2830000000000004,"15min":46.064},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4410000000000001,"15min":1.5669999999999999},"min":{"1min":0.67500000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.9710000000000001,"5min":7.4210000000000003,"15min":45.642000000000003},"last":0.58999999999999997}]}]},{"osd":2,"up_from":21,"seq":90194313672,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663468,"kb_used_data":353956,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93081684,"statfs":{"total":95995035648,"available":95315644416,"internally_reserved":0,"allocated":362450944,"data_stored":361943600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.3660000000000001,"15min":1.107},"min":{"1min":0.36199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1160000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.4059999999999999,"15min":1.222},"min":{"1min":0.40300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":1.4399999999999999,"5min":6.484,"15min":9.6470000000000002},"last":0.65700000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.272,"15min":1.841},"min":{"1min":0.25700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":2.7709999999999999,"5min":6.4509999999999996,"15min":21.407},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.109,"15min":0.92300000000000004},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.407,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.81899999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.379,"15min":1.0840000000000001},"min":{"1min":0.33500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.5840000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3600000000000001,"15min":1.2110000000000001},"min":{"1min":0.36699999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.5710000000000002,"5min":6.601,"15min":9.8079999999999998},"last":1.5329999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.3080000000000001,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":1.4570000000000001,"5min":6.133,"15min":6.133},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.05,"5min":1.6140000000000001,"15min":1.214},"min":{"1min":0.55800000000000005,"5min":0.315,"15min":0.315},"max":{"1min":1.4370000000000001,"5min":11.081,"15min":11.081},"last":1.494}]},{"osd":6,"last update":"Mon Feb 12 16:28:59 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.429,"15min":1.294},"min":{"1min":0.378,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.4160000000000004,"5min":7.6890000000000001,"15min":21.527000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.4179999999999999,"15min":1.2230000000000001},"min":{"1min":0.44500000000000001,"5min":0.38700000000000001,"15min":0.33800000000000002},"max":{"1min":2.6200000000000001,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.74199999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.641,"15min":1.24},"min":{"1min":0.38300000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":2.6040000000000001,"5min":11.044,"15min":11.044},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.143,"5min":1.4299999999999999,"15min":1.272},"min":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.5840000000000001,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":0.78800000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4350000000000001,"15min":1.306},"min":{"1min":0.47799999999999998,"5min":0.42999999999999999,"15min":0.42099999999999999},"max":{"1min":4.8360000000000003,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4330000000000001,"15min":1.3420000000000001},"min":{"1min":0.47399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":4.7389999999999999,"5min":6.593,"15min":9.7639999999999993},"last":0.51000000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.5860000000000001,"15min":1.3520000000000001},"min":{"1min":0.42199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.1819999999999999,"5min":10.972,"15min":10.972},"last":1.78},{"interface":"front","average":{"1min":1.145,"5min":1.6819999999999999,"15min":1.4330000000000001},"min":{"1min":0.46600000000000003,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":2.2250000000000001,"5min":11.006,"15min":11.006},"last":1.825}]},{"osd":10,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.609,"15min":1.462},"min":{"1min":0.72899999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.024,"5min":7.7539999999999996,"15min":11.454000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.605,"5min":1.6739999999999999,"15min":1.5009999999999999},"min":{"1min":0.76000000000000001,"5min":0.441,"15min":0.441},"max":{"1min":2.2280000000000002,"5min":7.7140000000000004,"15min":9.7170000000000005},"last":0.625}]},{"osd":11,"last update":"Mon Feb 12 16:28:59 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4770000000000001,"15min":1.4970000000000001},"min":{"1min":0.64400000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":7.258,"5min":7.258,"15min":7.258},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.581,"15min":1.5800000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.3040000000000003,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":1.1779999999999999}]}]},{"osd":3,"up_from":522,"seq":2241972928597,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2398252,"kb_used_data":2053732,"kb_used_omap":91,"kb_used_meta":344420,"kb_avail":91346900,"statfs":{"total":95995035648,"available":93539225600,"internally_reserved":0,"allocated":2103021568,"data_stored":2102514224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":352686096},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.325,"15min":1.006},"min":{"1min":0.27400000000000002,"5min":0.27000000000000002,"15min":0.23799999999999999},"max":{"1min":3.9500000000000002,"5min":34.055999999999997,"15min":34.055999999999997},"last":2.5409999999999999},{"interface":"front","average":{"1min":0.751,"5min":1.4370000000000001,"15min":1.0469999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4220000000000002,"5min":27.600000000000001,"15min":27.600000000000001},"last":2.5659999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.2230000000000001,"15min":0.98699999999999999},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.597,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":2.7480000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.175,"15min":1.095},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.9510000000000001,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":2.7810000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.78200000000000003,"15min":1.212},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.27,"5min":4.7080000000000002,"15min":10.004},"last":2.7610000000000001},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.052,"15min":1.105},"min":{"1min":0.45000000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.7350000000000003,"5min":6.806,"15min":6.806},"last":2.8180000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":1.294},"min":{"1min":0.41699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.9219999999999999,"5min":4.7530000000000001,"15min":6.7539999999999996},"last":2.923},{"interface":"front","average":{"1min":1.113,"5min":0.95099999999999996,"15min":0.94899999999999995},"min":{"1min":0.34599999999999997,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":3.5880000000000001,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":2.7970000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0600000000000001,"15min":1.1259999999999999},"min":{"1min":0.40100000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.778,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":3.113},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.996,"15min":1.01},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.1509999999999998,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":2.7229999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.012,"15min":1.3759999999999999},"min":{"1min":0.41599999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.4630000000000001,"5min":4.8090000000000002,"15min":6.851},"last":2.8940000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1950000000000001,"15min":1.2689999999999999},"min":{"1min":0.52900000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.8330000000000002,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":3.1619999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.0529999999999999,"15min":1.1759999999999999},"min":{"1min":0.49299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.2280000000000002,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":2.968},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.999,"15min":1.099},"min":{"1min":0.44500000000000001,"5min":0.42699999999999999,"15min":0.39800000000000002},"max":{"1min":2.3650000000000002,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":2.855}]},{"osd":9,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.284,"15min":1.329},"min":{"1min":0.57199999999999995,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.7050000000000001,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":3.2440000000000002},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.3129999999999999,"15min":1.373},"min":{"1min":0.56200000000000006,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.194,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":3.0369999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.147,"15min":1.4530000000000001},"min":{"1min":0.60299999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.016,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":3.0030000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.298,"15min":1.3220000000000001},"min":{"1min":0.61099999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.7330000000000001,"5min":4.9320000000000004,"15min":4.9320000000000004},"last":3.2210000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:29:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.2450000000000001,"15min":1.2849999999999999},"min":{"1min":0.63400000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":2.766,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":3.194},{"interface":"front","average":{"1min":1.583,"5min":1.2589999999999999,"15min":1.486},"min":{"1min":0.57199999999999995,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":5.5410000000000004,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":3.0760000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953912,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1592428,"kb_used_data":1002724,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":92152724,"statfs":{"total":95995035648,"available":94364389376,"internally_reserved":0,"allocated":1026789376,"data_stored":1026282032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":1.101,"15min":1.361},"min":{"1min":0.27200000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.804,"5min":7.492,"15min":90.597999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.036,"5min":1.2470000000000001,"15min":1.542},"min":{"1min":0.44900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.9099999999999999,"5min":12.236000000000001,"15min":90.676000000000002},"last":0.44900000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.119,"15min":1.006},"min":{"1min":0.38300000000000001,"5min":0.28399999999999997,"15min":0.24399999999999999},"max":{"1min":2.0049999999999999,"5min":7.3630000000000004,"15min":8.0530000000000008},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2050000000000001,"15min":1.1359999999999999},"min":{"1min":0.312,"5min":0.254,"15min":0.254},"max":{"1min":1.8169999999999999,"5min":7.633,"15min":7.633},"last":0.312}]},{"osd":3,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.1759999999999999,"15min":1.1180000000000001},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":7.6029999999999998,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.294,"5min":1.0649999999999999,"15min":1.0609999999999999},"min":{"1min":0.45700000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.383,"5min":7.383,"15min":7.383},"last":0.34100000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.099,"15min":0.98099999999999998},"min":{"1min":0.372,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":7.431,"5min":7.431,"15min":7.431},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.7809999999999999,"5min":1.2470000000000001,"15min":1.149},"min":{"1min":0.42599999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":7.4569999999999999,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.86799999999999999,"15min":1.0029999999999999},"min":{"1min":0.28000000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.5289999999999999,"5min":7.5289999999999999,"15min":7.5410000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.1160000000000001,"15min":1.101},"min":{"1min":0.28999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.9930000000000003,"5min":5.5190000000000001,"15min":9.6639999999999997},"last":1.9299999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.081,"15min":1.0960000000000001},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.9610000000000001,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":1.9610000000000001},{"interface":"front","average":{"1min":1.264,"5min":0.94899999999999995,"15min":1.149},"min":{"1min":0.47199999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.016,"5min":7.6500000000000004,"15min":7.6689999999999996},"last":0.47199999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.032,"15min":1.204},"min":{"1min":0.55300000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":7.6890000000000001,"5min":7.6890000000000001,"15min":7.8650000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.7470000000000001,"5min":1.2410000000000001,"15min":1.26},"min":{"1min":0.52000000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":7.7110000000000003},"last":0.63500000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.0800000000000001,"15min":1.3660000000000001},"min":{"1min":0.52200000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.121,"5min":7.7750000000000004,"15min":34.112000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.383,"15min":1.5660000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.3170000000000002,"5min":7.827,"15min":34.590000000000003},"last":0.60799999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4530000000000001,"15min":1.4330000000000001},"min":{"1min":0.433,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4129999999999998,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":0.746},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.101,"15min":1.23},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.9950000000000001,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":1.9950000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:29:10 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.1379999999999999,"15min":1.446},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":7.734,"5min":7.734,"15min":7.734},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.74,"5min":1.419,"15min":1.4950000000000001},"min":{"1min":0.56399999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":7.7969999999999997,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":0.65700000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714040,"kb_used_data":1412720,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92031112,"statfs":{"total":95995035648,"available":94239858688,"internally_reserved":0,"allocated":1446625280,"data_stored":1446111876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":3.2469999999999999,"15min":3.484},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":1.387,"5min":221.96600000000001,"15min":565.34500000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.155,"5min":3.3519999999999999,"15min":3.5790000000000002},"min":{"1min":0.45900000000000002,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":1.4830000000000001,"5min":222.06200000000001,"15min":565.98400000000004},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.53,"15min":1.141},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.3779999999999999,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.599,"15min":1.2150000000000001},"min":{"1min":0.375,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.3700000000000001,"5min":40.225000000000001,"15min":40.225000000000001},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:29:22 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.4039999999999999,"15min":1.3340000000000001},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2650000000000001,"5min":16.300999999999998,"15min":16.300999999999998},"last":22.484000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.999,"15min":0.81899999999999995},"min":{"1min":0.29699999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.284,"5min":6.5540000000000003,"15min":6.5540000000000003},"last":23.34}]},{"osd":4,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.3640000000000001,"15min":1.4239999999999999},"min":{"1min":0.40600000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.5149999999999999,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.736,"15min":1.476},"min":{"1min":0.499,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":3.165,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:29:15 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.198,"15min":1.3480000000000001},"min":{"1min":0.39600000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.9089999999999998,"5min":6.4459999999999997,"15min":74.105000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.1639999999999999,"15min":1.4279999999999999},"min":{"1min":0.375,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.048,"5min":9.5830000000000002,"15min":75.013999999999996},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1100000000000001,"15min":1.0549999999999999},"min":{"1min":0.48199999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.5640000000000001,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.234,"5min":1.355,"15min":1.159},"min":{"1min":0.83699999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.802,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.45400000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.1140000000000001,"15min":1.1799999999999999},"min":{"1min":0.51300000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.3159999999999998,"5min":4.2850000000000001,"15min":5.9509999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.58799999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.98,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":0.58899999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:29:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5669999999999999,"15min":3.3399999999999999},"min":{"1min":0.496,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":5.476,"5min":6.2939999999999996,"15min":449.89499999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.361,"15min":3.2730000000000001},"min":{"1min":0.48199999999999998,"5min":0.375,"15min":0.375},"max":{"1min":2.4470000000000001,"5min":9.6649999999999991,"15min":448.70800000000003},"last":0.82499999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":4.2859999999999996,"5min":1.7430000000000001,"15min":1.4550000000000001},"min":{"1min":0.91100000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":55.518999999999998,"5min":55.518999999999998,"15min":55.518999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":4.218,"5min":2.105,"15min":1.5529999999999999},"min":{"1min":0.76300000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":55.497999999999998,"5min":55.497999999999998,"15min":55.497999999999998},"last":0.79200000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.321,"15min":1.3360000000000001},"min":{"1min":0.55800000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.5529999999999999,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.377,"5min":1.518,"15min":1.4119999999999999},"min":{"1min":0.63200000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.8149999999999999,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":714014720,"data_stored":714014720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1096548352,"data_stored":1096548352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366936064,"data_stored":366936064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2127364096,"data_stored":2127364096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1042808832,"data_stored":1042808832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1466302464,"data_stored":1466302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1414397952,"data_stored":1414397952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1495924736,"data_stored":1495924736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":761331712,"data_stored":761331712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":732037120,"data_stored":732037120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":382992384,"data_stored":382992384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:30:07.405 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T16:30:07.405 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:30:07.406 INFO:tasks.thrashosds.thrasher:Growing pool device_health_metrics 2024-02-12T16:30:07.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:30:07.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:30:07.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:07.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:07.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:07 smithi169 bash[19604]: cluster 2024-02-12T16:30:06.026367+0000 mgr.y (mgr.14146) 1361 : cluster [DBG] pgmap v1898: 37 pgs: 37 active+clean; 5.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-02-12T16:30:07.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 98518 98502 157.842 102.5 0.00924175 0.00633387 2024-02-12T16:30:07.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:07.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:07.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:07.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:07.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:07.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:07.981 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:07.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:08.034 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:30:08.074 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:30:08.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:08.105 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:30:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:08.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:08.135 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:30:08.170 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:30:08.204 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:30:08.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:30:08.242 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:08.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:08.269 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:30:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:30:08.323 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:30:08.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:08.352 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:30:08.352 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:08 smithi123 bash[19119]: cluster 2024-02-12T16:30:06.982247+0000 osd.2 (osd.2) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:08.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:08 smithi123 bash[19119]: cluster 2024-02-12T16:30:06.982300+0000 osd.2 (osd.2) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:08.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:08 smithi123 bash[19119]: cluster 2024-02-12T16:30:07.719457+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e655: 12 total, 12 up, 12 in 2024-02-12T16:30:08.353 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:08 smithi123 bash[19119]: audit 2024-02-12T16:30:07.999487+0000 mon.a (mon.0) 2296 : audit [DBG] from='client.? v1:172.21.15.121:0/19006251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:08.373 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:30:08.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.376 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:30:08.383 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:08.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:08 smithi121 bash[18489]: cluster 2024-02-12T16:30:06.982247+0000 osd.2 (osd.2) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:08.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:08 smithi121 bash[18489]: cluster 2024-02-12T16:30:06.982300+0000 osd.2 (osd.2) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:08 smithi121 bash[18489]: cluster 2024-02-12T16:30:07.719457+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e655: 12 total, 12 up, 12 in 2024-02-12T16:30:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:08 smithi121 bash[18489]: audit 2024-02-12T16:30:07.999487+0000 mon.a (mon.0) 2296 : audit [DBG] from='client.? v1:172.21.15.121:0/19006251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:08.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:08.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:08.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:08 smithi169 bash[19604]: cluster 2024-02-12T16:30:06.982247+0000 osd.2 (osd.2) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:08 smithi169 bash[19604]: cluster 2024-02-12T16:30:06.982300+0000 osd.2 (osd.2) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:08 smithi169 bash[19604]: cluster 2024-02-12T16:30:07.719457+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e655: 12 total, 12 up, 12 in 2024-02-12T16:30:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:08 smithi169 bash[19604]: audit 2024-02-12T16:30:07.999487+0000 mon.a (mon.0) 2296 : audit [DBG] from='client.? v1:172.21.15.121:0/19006251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:08.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:30:08.564321 min lat: 0.00101494 max lat: 0.31862 avg lat: 0.00639741 2024-02-12T16:30:08.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:30:08.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 100036 100020 156.268 94.875 0.0133664 0.00639741 2024-02-12T16:30:08.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:08.656 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:08.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:08.759 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:08.786 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:08.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:08.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:08.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:08.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:09.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:09.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:09.115 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:30:09.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:09.148 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:09 smithi169 bash[19604]: cluster 2024-02-12T16:30:08.026759+0000 mgr.y (mgr.14146) 1362 : cluster [DBG] pgmap v1900: 37 pgs: 37 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:30:09.148 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:09 smithi169 bash[19604]: audit 2024-02-12T16:30:08.234501+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? v1:172.21.15.121:0/172582020' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:30:09.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:09.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:09.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:09.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:09.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:09 smithi121 bash[18489]: cluster 2024-02-12T16:30:08.026759+0000 mgr.y (mgr.14146) 1362 : cluster [DBG] pgmap v1900: 37 pgs: 37 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:30:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:09 smithi121 bash[18489]: audit 2024-02-12T16:30:08.234501+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? v1:172.21.15.121:0/172582020' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:30:09.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:09.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:09.461 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:09 smithi123 bash[19119]: cluster 2024-02-12T16:30:08.026759+0000 mgr.y (mgr.14146) 1362 : cluster [DBG] pgmap v1900: 37 pgs: 37 active+clean; 5.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:30:09.461 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:09 smithi123 bash[19119]: audit 2024-02-12T16:30:08.234501+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? v1:172.21.15.121:0/172582020' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:30:09.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:09.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 16 101296 101280 154.377 78.75 0.00410427 0.00644935 2024-02-12T16:30:09.578 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:09.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:09.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:09.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:09.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:09.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:09.984 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:09.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:10.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:10.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:10 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.094904+0000 mon.a (mon.0) 2298 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:30:10.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:10 smithi123 bash[19119]: audit 2024-02-12T16:30:09.105493+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.121:0/172582020' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:30:10.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:10 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.105568+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e656: 12 total, 12 up, 12 in 2024-02-12T16:30:10.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:10 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.576271+0000 osd.7 (osd.7) 158 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:30:10.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:10 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.801496+0000 osd.9 (osd.9) 111 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:30:10.103 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:10 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.838464+0000 osd.9 (osd.9) 112 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:30:10.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:10.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:10.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:10.248 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:10.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:10.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:10.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:10.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:10.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:10 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.094904+0000 mon.a (mon.0) 2298 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:30:10.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:10 smithi121 bash[18489]: audit 2024-02-12T16:30:09.105493+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.121:0/172582020' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:30:10.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:10 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.105568+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e656: 12 total, 12 up, 12 in 2024-02-12T16:30:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:10 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.576271+0000 osd.7 (osd.7) 158 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:30:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:10 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.801496+0000 osd.9 (osd.9) 111 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:30:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:10 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.838464+0000 osd.9 (osd.9) 112 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:30:10.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:10.519 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:10.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:10 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.094904+0000 mon.a (mon.0) 2298 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:30:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:10 smithi169 bash[19604]: audit 2024-02-12T16:30:09.105493+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.121:0/172582020' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:30:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:10 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.105568+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e656: 12 total, 12 up, 12 in 2024-02-12T16:30:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:10 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.576271+0000 osd.7 (osd.7) 158 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:30:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:10 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.801496+0000 osd.9 (osd.9) 111 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:30:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:10 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.838464+0000 osd.9 (osd.9) 112 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:30:10.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 102628 102612 152.683 83.25 0.00412337 0.0065475 2024-02-12T16:30:10.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:10.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:10.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:10.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:10.790 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:10.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:10.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:10.924 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:10.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:11.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:11.073 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:11.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:11.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:11.185 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.612651+0000 osd.1 (osd.1) 94 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:30:11.185 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.726176+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:30:11.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.745389+0000 osd.1 (osd.1) 95 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:30:11.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.766204+0000 osd.4 (osd.4) 110 : cluster [DBG] 1.e scrub starts 2024-02-12T16:30:11.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:09.919667+0000 osd.4 (osd.4) 111 : cluster [DBG] 1.e scrub ok 2024-02-12T16:30:11.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.027193+0000 mgr.y (mgr.14146) 1363 : cluster [DBG] pgmap v1902: 37 pgs: 37 active+clean; 6.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:30:11.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.186678+0000 osd.11 (osd.11) 146 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:30:11.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:11 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.264887+0000 osd.11 (osd.11) 147 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:30:11.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:11.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:11.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:11.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:11.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:11.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.612651+0000 osd.1 (osd.1) 94 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:30:11.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.726176+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:30:11.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.745389+0000 osd.1 (osd.1) 95 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:30:11.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.766204+0000 osd.4 (osd.4) 110 : cluster [DBG] 1.e scrub starts 2024-02-12T16:30:11.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:09.919667+0000 osd.4 (osd.4) 111 : cluster [DBG] 1.e scrub ok 2024-02-12T16:30:11.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.027193+0000 mgr.y (mgr.14146) 1363 : cluster [DBG] pgmap v1902: 37 pgs: 37 active+clean; 6.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:30:11.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.186678+0000 osd.11 (osd.11) 146 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:30:11.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:11 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.264887+0000 osd.11 (osd.11) 147 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:30:11.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.612651+0000 osd.1 (osd.1) 94 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.726176+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.745389+0000 osd.1 (osd.1) 95 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.766204+0000 osd.4 (osd.4) 110 : cluster [DBG] 1.e scrub starts 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:09.919667+0000 osd.4 (osd.4) 111 : cluster [DBG] 1.e scrub ok 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.027193+0000 mgr.y (mgr.14146) 1363 : cluster [DBG] pgmap v1902: 37 pgs: 37 active+clean; 6.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.186678+0000 osd.11 (osd.11) 146 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:30:11.459 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:11 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.264887+0000 osd.11 (osd.11) 147 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:30:11.476 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:11.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:11.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 16 104148 104132 151.341 95 0.00896128 0.00660567 2024-02-12T16:30:11.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:11.603 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:11.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:11.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:11.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:11.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:11.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:11.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:11.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:11.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:11.998 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:12.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:12.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:12 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.552433+0000 osd.3 (osd.3) 28 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:30:12.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:12 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.565897+0000 osd.0 (osd.0) 105 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:30:12.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:12 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.604297+0000 osd.3 (osd.3) 29 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:30:12.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:12 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.699421+0000 osd.5 (osd.5) 107 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:30:12.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:12 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.717194+0000 osd.0 (osd.0) 106 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:30:12.115 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:12 smithi123 bash[19119]: cluster 2024-02-12T16:30:10.761830+0000 osd.5 (osd.5) 108 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:30:12.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:12.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:12.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:12.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:30:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:12 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.552433+0000 osd.3 (osd.3) 28 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:30:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:12 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.565897+0000 osd.0 (osd.0) 105 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:30:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:12 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.604297+0000 osd.3 (osd.3) 29 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:30:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:12 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.699421+0000 osd.5 (osd.5) 107 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:30:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:12 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.717194+0000 osd.0 (osd.0) 106 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:30:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:12 smithi169 bash[19604]: cluster 2024-02-12T16:30:10.761830+0000 osd.5 (osd.5) 108 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:30:12.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:12.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:12.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:12 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.552433+0000 osd.3 (osd.3) 28 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:30:12.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:12 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.565897+0000 osd.0 (osd.0) 105 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:30:12.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:12 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.604297+0000 osd.3 (osd.3) 29 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:30:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:12 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.699421+0000 osd.5 (osd.5) 107 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:30:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:12 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.717194+0000 osd.0 (osd.0) 106 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:30:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:12 smithi121 bash[18489]: cluster 2024-02-12T16:30:10.761830+0000 osd.5 (osd.5) 108 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:30:12.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:12.536 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:12.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 105702 105686 150.109 97.125 0.00858378 0.00666026 2024-02-12T16:30:12.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:30:12.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:12.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:12.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:12.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:12.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:12.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:13.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:13.052 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:13.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:13.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:13 smithi121 bash[18489]: cluster 2024-02-12T16:30:12.027693+0000 mgr.y (mgr.14146) 1364 : cluster [DBG] pgmap v1903: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-02-12T16:30:13.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:13.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:13 smithi169 bash[19604]: cluster 2024-02-12T16:30:12.027693+0000 mgr.y (mgr.14146) 1364 : cluster [DBG] pgmap v1903: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-02-12T16:30:13.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:13.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:13 smithi123 bash[19119]: cluster 2024-02-12T16:30:12.027693+0000 mgr.y (mgr.14146) 1364 : cluster [DBG] pgmap v1903: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-02-12T16:30:13.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:13.447 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:13.556 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:13.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 107221 107205 148.883 94.9375 0.00865812 0.00671516 2024-02-12T16:30:13.590 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:13.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:13.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:13.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:13.857 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:13.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:13.984 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:13.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:14.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:14.103 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:14.103 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:14.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:14.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:14.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:14.250 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:14 smithi121 bash[18489]: cluster 2024-02-12T16:30:12.764298+0000 osd.5 (osd.5) 109 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:30:14.250 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:14 smithi121 bash[18489]: cluster 2024-02-12T16:30:12.797109+0000 osd.5 (osd.5) 110 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:30:14.250 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:14 smithi121 bash[18489]: cluster 2024-02-12T16:30:13.268019+0000 osd.11 (osd.11) 148 : cluster [DBG] 1.b scrub starts 2024-02-12T16:30:14.250 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:14 smithi121 bash[18489]: cluster 2024-02-12T16:30:13.398409+0000 osd.8 (osd.8) 63 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:30:14.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:14.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:14.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:14.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:14 smithi123 bash[19119]: cluster 2024-02-12T16:30:12.764298+0000 osd.5 (osd.5) 109 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:30:14.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:14 smithi123 bash[19119]: cluster 2024-02-12T16:30:12.797109+0000 osd.5 (osd.5) 110 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:30:14.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:14 smithi123 bash[19119]: cluster 2024-02-12T16:30:13.268019+0000 osd.11 (osd.11) 148 : cluster [DBG] 1.b scrub starts 2024-02-12T16:30:14.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:14 smithi123 bash[19119]: cluster 2024-02-12T16:30:13.398409+0000 osd.8 (osd.8) 63 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:30:14.412 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:14.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:14.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:14 smithi169 bash[19604]: cluster 2024-02-12T16:30:12.764298+0000 osd.5 (osd.5) 109 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:30:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:14 smithi169 bash[19604]: cluster 2024-02-12T16:30:12.797109+0000 osd.5 (osd.5) 110 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:30:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:14 smithi169 bash[19604]: cluster 2024-02-12T16:30:13.268019+0000 osd.11 (osd.11) 148 : cluster [DBG] 1.b scrub starts 2024-02-12T16:30:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:14 smithi169 bash[19604]: cluster 2024-02-12T16:30:13.398409+0000 osd.8 (osd.8) 63 : cluster [DBG] 6.1 scrub starts 2024-02-12T16:30:14.553 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:14.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:14.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 16 108733 108717 147.7 94.5 0.00764045 0.00676883 2024-02-12T16:30:14.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:14.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:14.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:14.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:14.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:14.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:14.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:14.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:14.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:15.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:15.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:15.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:15 smithi121 bash[18489]: cluster 2024-02-12T16:30:13.391667+0000 osd.11 (osd.11) 149 : cluster [DBG] 1.b scrub ok 2024-02-12T16:30:15.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:15 smithi121 bash[18489]: cluster 2024-02-12T16:30:14.028120+0000 mgr.y (mgr.14146) 1365 : cluster [DBG] pgmap v1904: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-12T16:30:15.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:15 smithi121 bash[18489]: cluster 2024-02-12T16:30:14.571143+0000 osd.9 (osd.9) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:15.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:15 smithi121 bash[18489]: cluster 2024-02-12T16:30:14.571177+0000 osd.9 (osd.9) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:15.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:15.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:15.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:15.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:15.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:15.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:15 smithi123 bash[19119]: cluster 2024-02-12T16:30:13.391667+0000 osd.11 (osd.11) 149 : cluster [DBG] 1.b scrub ok 2024-02-12T16:30:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:15 smithi123 bash[19119]: cluster 2024-02-12T16:30:14.028120+0000 mgr.y (mgr.14146) 1365 : cluster [DBG] pgmap v1904: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-12T16:30:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:15 smithi123 bash[19119]: cluster 2024-02-12T16:30:14.571143+0000 osd.9 (osd.9) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:15 smithi123 bash[19119]: cluster 2024-02-12T16:30:14.571177+0000 osd.9 (osd.9) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:15 smithi123 bash[19119]: cluster 2024-02-12T16:30:14.669348+0000 osd.8 (osd.8) 64 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:30:15.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:15 smithi121 bash[18489]: cluster 2024-02-12T16:30:14.669348+0000 osd.8 (osd.8) 64 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:30:15.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:15.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:15.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:15.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:15.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:15 smithi169 bash[19604]: cluster 2024-02-12T16:30:13.391667+0000 osd.11 (osd.11) 149 : cluster [DBG] 1.b scrub ok 2024-02-12T16:30:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:15 smithi169 bash[19604]: cluster 2024-02-12T16:30:14.028120+0000 mgr.y (mgr.14146) 1365 : cluster [DBG] pgmap v1904: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-02-12T16:30:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:15 smithi169 bash[19604]: cluster 2024-02-12T16:30:14.571143+0000 osd.9 (osd.9) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:15 smithi169 bash[19604]: cluster 2024-02-12T16:30:14.571177+0000 osd.9 (osd.9) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:15 smithi169 bash[19604]: cluster 2024-02-12T16:30:14.669348+0000 osd.8 (osd.8) 64 : cluster [DBG] 6.1 scrub ok 2024-02-12T16:30:15.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 16 110243 110227 146.566 94.375 0.0130332 0.00682126 2024-02-12T16:30:15.600 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:15.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:15.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:15.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:15.761 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:15.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:15.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:15.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:15.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:16.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:16.035 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:16.134 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:16.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:16 smithi121 bash[18489]: cluster 2024-02-12T16:30:14.752808+0000 osd.6 (osd.6) 120 : cluster [DBG] 1.f scrub starts 2024-02-12T16:30:16.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:16 smithi121 bash[18489]: cluster 2024-02-12T16:30:14.849823+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.f scrub ok 2024-02-12T16:30:16.137 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:16 smithi121 bash[18489]: cluster 2024-02-12T16:30:15.124608+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e657: 12 total, 12 up, 12 in 2024-02-12T16:30:16.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:16.163 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:16.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:16.291 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:16 smithi123 bash[19119]: cluster 2024-02-12T16:30:14.752808+0000 osd.6 (osd.6) 120 : cluster [DBG] 1.f scrub starts 2024-02-12T16:30:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:16 smithi123 bash[19119]: cluster 2024-02-12T16:30:14.849823+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.f scrub ok 2024-02-12T16:30:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:16 smithi123 bash[19119]: cluster 2024-02-12T16:30:15.124608+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e657: 12 total, 12 up, 12 in 2024-02-12T16:30:16.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:16.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:16.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:16.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:16 smithi169 bash[19604]: cluster 2024-02-12T16:30:14.752808+0000 osd.6 (osd.6) 120 : cluster [DBG] 1.f scrub starts 2024-02-12T16:30:16.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:16 smithi169 bash[19604]: cluster 2024-02-12T16:30:14.849823+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.f scrub ok 2024-02-12T16:30:16.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:16 smithi169 bash[19604]: cluster 2024-02-12T16:30:15.124608+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e657: 12 total, 12 up, 12 in 2024-02-12T16:30:16.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:16.567 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 111694 111678 145.401 90.6875 0.00541824 0.0068757 2024-02-12T16:30:16.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:16.674 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:16.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:16.800 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:16.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:16.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:16.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:17.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:17.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:17.148 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:17.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:17.224 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:17.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:17.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:17.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:17 smithi121 bash[18489]: cluster 2024-02-12T16:30:15.581527+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.c scrub starts 2024-02-12T16:30:17.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:17 smithi121 bash[18489]: cluster 2024-02-12T16:30:16.028523+0000 mgr.y (mgr.14146) 1366 : cluster [DBG] pgmap v1906: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:30:17.357 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:17.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:17.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:17 smithi123 bash[19119]: cluster 2024-02-12T16:30:15.581527+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.c scrub starts 2024-02-12T16:30:17.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:17 smithi123 bash[19119]: cluster 2024-02-12T16:30:16.028523+0000 mgr.y (mgr.14146) 1366 : cluster [DBG] pgmap v1906: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:30:17.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:17.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:17 smithi169 bash[19604]: cluster 2024-02-12T16:30:15.581527+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.c scrub starts 2024-02-12T16:30:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:17 smithi169 bash[19604]: cluster 2024-02-12T16:30:16.028523+0000 mgr.y (mgr.14146) 1366 : cluster [DBG] pgmap v1906: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-02-12T16:30:17.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 113118 113102 144.25 89 0.0182944 0.00693096 2024-02-12T16:30:17.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:17.625 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:17.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:17.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:17.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:17.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:17.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:17.914 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:17.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:18.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:18.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:18.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:18.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:18.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:18.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:18.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:18 smithi121 bash[18489]: cluster 2024-02-12T16:30:17.486685+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.d scrub starts 2024-02-12T16:30:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:18 smithi121 bash[18489]: cluster 2024-02-12T16:30:17.602615+0000 osd.7 (osd.7) 161 : cluster [DBG] 1.d scrub ok 2024-02-12T16:30:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:18 smithi121 bash[18489]: cluster 2024-02-12T16:30:17.789126+0000 osd.9 (osd.9) 115 : cluster [DBG] 1.a scrub starts 2024-02-12T16:30:18.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:18 smithi121 bash[18489]: cluster 2024-02-12T16:30:17.846294+0000 osd.9 (osd.9) 116 : cluster [DBG] 1.a scrub ok 2024-02-12T16:30:18.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:18.434 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:18 smithi169 bash[19604]: cluster 2024-02-12T16:30:17.486685+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.d scrub starts 2024-02-12T16:30:18.435 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:18 smithi169 bash[19604]: cluster 2024-02-12T16:30:17.602615+0000 osd.7 (osd.7) 161 : cluster [DBG] 1.d scrub ok 2024-02-12T16:30:18.435 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:18 smithi169 bash[19604]: cluster 2024-02-12T16:30:17.789126+0000 osd.9 (osd.9) 115 : cluster [DBG] 1.a scrub starts 2024-02-12T16:30:18.435 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:18 smithi169 bash[19604]: cluster 2024-02-12T16:30:17.846294+0000 osd.9 (osd.9) 116 : cluster [DBG] 1.a scrub ok 2024-02-12T16:30:18.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:18.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:18.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 114588 114572 143.203 91.875 0.00603871 0.00698146 2024-02-12T16:30:18.581 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:18.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:18 smithi123 bash[19119]: cluster 2024-02-12T16:30:17.486685+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.d scrub starts 2024-02-12T16:30:18.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:18 smithi123 bash[19119]: cluster 2024-02-12T16:30:17.602615+0000 osd.7 (osd.7) 161 : cluster [DBG] 1.d scrub ok 2024-02-12T16:30:18.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:18 smithi123 bash[19119]: cluster 2024-02-12T16:30:17.789126+0000 osd.9 (osd.9) 115 : cluster [DBG] 1.a scrub starts 2024-02-12T16:30:18.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:18 smithi123 bash[19119]: cluster 2024-02-12T16:30:17.846294+0000 osd.9 (osd.9) 116 : cluster [DBG] 1.a scrub ok 2024-02-12T16:30:18.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:18.716 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:18.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:18.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:18.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:18.986 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:18.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:19.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:19.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:19.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:19.247 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:19.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:19.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:19.388 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:19.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:19.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:19 smithi121 bash[18489]: cluster 2024-02-12T16:30:17.583723+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.c scrub ok 2024-02-12T16:30:19.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:19 smithi121 bash[18489]: cluster 2024-02-12T16:30:18.028944+0000 mgr.y (mgr.14146) 1367 : cluster [DBG] pgmap v1907: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s 2024-02-12T16:30:19.494 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:19.501 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:19 smithi123 bash[19119]: cluster 2024-02-12T16:30:17.583723+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.c scrub ok 2024-02-12T16:30:19.501 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:19 smithi123 bash[19119]: cluster 2024-02-12T16:30:18.028944+0000 mgr.y (mgr.14146) 1367 : cluster [DBG] pgmap v1907: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s 2024-02-12T16:30:19.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:19.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:19.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:19 smithi169 bash[19604]: cluster 2024-02-12T16:30:17.583723+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.c scrub ok 2024-02-12T16:30:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:19 smithi169 bash[19604]: cluster 2024-02-12T16:30:18.028944+0000 mgr.y (mgr.14146) 1367 : cluster [DBG] pgmap v1907: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s 2024-02-12T16:30:19.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 116009 115993 142.136 88.8125 0.0121096 0.00703393 2024-02-12T16:30:19.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:19.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:19.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:19.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:19.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:19.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:19.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:19.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:19.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:20.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:20.060 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:20.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:20.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:20.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:20 smithi121 bash[18489]: cluster 2024-02-12T16:30:18.561687+0000 osd.11 (osd.11) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:20.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:20 smithi121 bash[18489]: cluster 2024-02-12T16:30:18.561771+0000 osd.11 (osd.11) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:20.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:20 smithi121 bash[18489]: cluster 2024-02-12T16:30:18.580117+0000 osd.3 (osd.3) 32 : cluster [DBG] 1.c scrub starts 2024-02-12T16:30:20.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:20 smithi121 bash[18489]: cluster 2024-02-12T16:30:18.631341+0000 osd.3 (osd.3) 33 : cluster [DBG] 1.c scrub ok 2024-02-12T16:30:20.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:20 smithi121 bash[18489]: cluster 2024-02-12T16:30:18.768595+0000 osd.5 (osd.5) 111 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:30:20.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:20 smithi121 bash[18489]: cluster 2024-02-12T16:30:18.901017+0000 osd.5 (osd.5) 112 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:30:20.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:20 smithi121 bash[18489]: cluster 2024-02-12T16:30:19.157792+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in 2024-02-12T16:30:20.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:20.338 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:20.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:20.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:20 smithi169 bash[19604]: cluster 2024-02-12T16:30:18.561687+0000 osd.11 (osd.11) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:20 smithi169 bash[19604]: cluster 2024-02-12T16:30:18.561771+0000 osd.11 (osd.11) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:20 smithi169 bash[19604]: cluster 2024-02-12T16:30:18.580117+0000 osd.3 (osd.3) 32 : cluster [DBG] 1.c scrub starts 2024-02-12T16:30:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:20 smithi169 bash[19604]: cluster 2024-02-12T16:30:18.631341+0000 osd.3 (osd.3) 33 : cluster [DBG] 1.c scrub ok 2024-02-12T16:30:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:20 smithi169 bash[19604]: cluster 2024-02-12T16:30:18.768595+0000 osd.5 (osd.5) 111 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:30:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:20 smithi169 bash[19604]: cluster 2024-02-12T16:30:18.901017+0000 osd.5 (osd.5) 112 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:30:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:20 smithi169 bash[19604]: cluster 2024-02-12T16:30:19.157792+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in 2024-02-12T16:30:20.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 16 117295 117279 140.948 80.375 0.00496875 0.00709327 2024-02-12T16:30:20.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:20.598 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:20.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:20 smithi123 bash[19119]: cluster 2024-02-12T16:30:18.561687+0000 osd.11 (osd.11) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:20.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:20 smithi123 bash[19119]: cluster 2024-02-12T16:30:18.561771+0000 osd.11 (osd.11) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:20.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:20 smithi123 bash[19119]: cluster 2024-02-12T16:30:18.580117+0000 osd.3 (osd.3) 32 : cluster [DBG] 1.c scrub starts 2024-02-12T16:30:20.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:20 smithi123 bash[19119]: cluster 2024-02-12T16:30:18.631341+0000 osd.3 (osd.3) 33 : cluster [DBG] 1.c scrub ok 2024-02-12T16:30:20.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:20 smithi123 bash[19119]: cluster 2024-02-12T16:30:18.768595+0000 osd.5 (osd.5) 111 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:30:20.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:20 smithi123 bash[19119]: cluster 2024-02-12T16:30:18.901017+0000 osd.5 (osd.5) 112 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:30:20.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:20 smithi123 bash[19119]: cluster 2024-02-12T16:30:19.157792+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e658: 12 total, 12 up, 12 in 2024-02-12T16:30:20.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:20.725 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:20.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:20.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:20.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:20.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:20.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:21.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:21.110 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:21.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:21.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:21.252 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:21.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:21.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:21.382 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:21.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:21 smithi123 bash[19119]: cluster 2024-02-12T16:30:19.687615+0000 osd.4 (osd.4) 112 : cluster [DBG] 6.e scrub starts 2024-02-12T16:30:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:21 smithi123 bash[19119]: cluster 2024-02-12T16:30:19.732697+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.a scrub starts 2024-02-12T16:30:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:21 smithi123 bash[19119]: cluster 2024-02-12T16:30:20.029437+0000 mgr.y (mgr.14146) 1368 : cluster [DBG] pgmap v1909: 37 pgs: 37 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:30:21.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:21.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:21.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:21.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:21 smithi169 bash[19604]: cluster 2024-02-12T16:30:19.687615+0000 osd.4 (osd.4) 112 : cluster [DBG] 6.e scrub starts 2024-02-12T16:30:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:21 smithi169 bash[19604]: cluster 2024-02-12T16:30:19.732697+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.a scrub starts 2024-02-12T16:30:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:21 smithi169 bash[19604]: cluster 2024-02-12T16:30:20.029437+0000 mgr.y (mgr.14146) 1368 : cluster [DBG] pgmap v1909: 37 pgs: 37 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:30:21.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 118539 118523 139.756 77.75 0.0113201 0.00715362 2024-02-12T16:30:21.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:21 smithi121 bash[18489]: cluster 2024-02-12T16:30:19.687615+0000 osd.4 (osd.4) 112 : cluster [DBG] 6.e scrub starts 2024-02-12T16:30:21.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:21 smithi121 bash[18489]: cluster 2024-02-12T16:30:19.732697+0000 osd.5 (osd.5) 113 : cluster [DBG] 6.a scrub starts 2024-02-12T16:30:21.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:21 smithi121 bash[18489]: cluster 2024-02-12T16:30:20.029437+0000 mgr.y (mgr.14146) 1368 : cluster [DBG] pgmap v1909: 37 pgs: 37 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-12T16:30:21.645 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:21.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:21.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:21.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:21.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:21.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:21.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:21.941 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:21.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:22.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:22.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:22.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:22.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:22.207 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:22.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:22.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:22.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:22.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:22.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:22.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:22.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:22.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 119790 119774 138.615 78.1875 0.0146203 0.00721279 2024-02-12T16:30:22.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:22.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:22.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:22.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:22.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:22.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:22.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:22.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:22.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:23.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:23.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:23.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:23.159 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:23.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:23.287 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:23.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:23.399 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:23 smithi123 bash[19119]: cluster 2024-02-12T16:30:21.757456+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.a scrub ok 2024-02-12T16:30:23.400 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:23 smithi123 bash[19119]: cluster 2024-02-12T16:30:22.029858+0000 mgr.y (mgr.14146) 1369 : cluster [DBG] pgmap v1910: 37 pgs: 37 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-02-12T16:30:23.421 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:23.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:23 smithi169 bash[19604]: cluster 2024-02-12T16:30:21.757456+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.a scrub ok 2024-02-12T16:30:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:23 smithi169 bash[19604]: cluster 2024-02-12T16:30:22.029858+0000 mgr.y (mgr.14146) 1369 : cluster [DBG] pgmap v1910: 37 pgs: 37 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-02-12T16:30:23.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:23.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 121080 121064 137.561 80.625 0.0051953 0.0072675 2024-02-12T16:30:23.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:23 smithi121 bash[18489]: cluster 2024-02-12T16:30:21.757456+0000 osd.5 (osd.5) 114 : cluster [DBG] 6.a scrub ok 2024-02-12T16:30:23.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:23 smithi121 bash[18489]: cluster 2024-02-12T16:30:22.029858+0000 mgr.y (mgr.14146) 1369 : cluster [DBG] pgmap v1910: 37 pgs: 37 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-02-12T16:30:23.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:23.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:23.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:23.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:23.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:23.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:23.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:24.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:24.086 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:24.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:24.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:24 smithi121 bash[18489]: cluster 2024-02-12T16:30:23.174448+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub ok 2024-02-12T16:30:24.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:24.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:24.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:24.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:24.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:24.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:24 smithi123 bash[19119]: cluster 2024-02-12T16:30:23.174448+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub ok 2024-02-12T16:30:24.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:24.487 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:24.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:24 smithi169 bash[19604]: cluster 2024-02-12T16:30:23.174448+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub ok 2024-02-12T16:30:24.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 122445 122429 136.628 85.3125 0.0149023 0.00731701 2024-02-12T16:30:24.592 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:24.617 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:24.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:24.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:24.759 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:24.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:24.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:24.893 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:24.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:24.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:25.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:25.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:25.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:25.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:25.291 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:25.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:25.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:25 smithi121 bash[18489]: cluster 2024-02-12T16:30:23.635977+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:30:25.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:25 smithi121 bash[18489]: cluster 2024-02-12T16:30:23.709524+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:30:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:25 smithi121 bash[18489]: cluster 2024-02-12T16:30:24.030328+0000 mgr.y (mgr.14146) 1370 : cluster [DBG] pgmap v1911: 37 pgs: 37 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-12T16:30:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:25 smithi121 bash[18489]: cluster 2024-02-12T16:30:24.466014+0000 osd.7 (osd.7) 162 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:30:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:25 smithi121 bash[18489]: cluster 2024-02-12T16:30:24.570140+0000 osd.7 (osd.7) 163 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:30:25.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:25.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:25 smithi169 bash[19604]: cluster 2024-02-12T16:30:23.635977+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:30:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:25 smithi169 bash[19604]: cluster 2024-02-12T16:30:23.709524+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:30:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:25 smithi169 bash[19604]: cluster 2024-02-12T16:30:24.030328+0000 mgr.y (mgr.14146) 1370 : cluster [DBG] pgmap v1911: 37 pgs: 37 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-12T16:30:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:25 smithi169 bash[19604]: cluster 2024-02-12T16:30:24.466014+0000 osd.7 (osd.7) 162 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:30:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:25 smithi169 bash[19604]: cluster 2024-02-12T16:30:24.570140+0000 osd.7 (osd.7) 163 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:30:25.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:25.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 123738 123722 135.648 80.8125 0.0125936 0.0073705 2024-02-12T16:30:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:25 smithi123 bash[19119]: cluster 2024-02-12T16:30:23.635977+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:30:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:25 smithi123 bash[19119]: cluster 2024-02-12T16:30:23.709524+0000 osd.6 (osd.6) 123 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:30:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:25 smithi123 bash[19119]: cluster 2024-02-12T16:30:24.030328+0000 mgr.y (mgr.14146) 1370 : cluster [DBG] pgmap v1911: 37 pgs: 37 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-02-12T16:30:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:25 smithi123 bash[19119]: cluster 2024-02-12T16:30:24.466014+0000 osd.7 (osd.7) 162 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:30:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:25 smithi123 bash[19119]: cluster 2024-02-12T16:30:24.570140+0000 osd.7 (osd.7) 163 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:30:25.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:25.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:25.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:25.839 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:25.865 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:30:25.865 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:30:25.865 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:30:25.865 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:30:25.865 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:30:25.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:25.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:25.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:26.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:26.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:26.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:26.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:26.214 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:26.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:26.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:26.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:26.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:26.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:26.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:26 smithi121 bash[18489]: cluster 2024-02-12T16:30:24.750619+0000 osd.1 (osd.1) 96 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:30:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:26 smithi121 bash[18489]: cluster 2024-02-12T16:30:24.971185+0000 osd.10 (osd.10) 128 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:30:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:26 smithi121 bash[18489]: cluster 2024-02-12T16:30:25.021147+0000 osd.10 (osd.10) 129 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:30:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:26 smithi121 bash[18489]: cluster 2024-02-12T16:30:25.508015+0000 osd.7 (osd.7) 164 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:30:26.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:26.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:26.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:26 smithi169 bash[19604]: cluster 2024-02-12T16:30:24.750619+0000 osd.1 (osd.1) 96 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:30:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:26 smithi169 bash[19604]: cluster 2024-02-12T16:30:24.971185+0000 osd.10 (osd.10) 128 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:30:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:26 smithi169 bash[19604]: cluster 2024-02-12T16:30:25.021147+0000 osd.10 (osd.10) 129 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:30:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:26 smithi169 bash[19604]: cluster 2024-02-12T16:30:25.508015+0000 osd.7 (osd.7) 164 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:30:26.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 16 124999 124983 134.668 78.8125 0.0192914 0.00742388 2024-02-12T16:30:26.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:26.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:26 smithi123 bash[19119]: cluster 2024-02-12T16:30:24.750619+0000 osd.1 (osd.1) 96 : cluster [DBG] 6.4 scrub starts 2024-02-12T16:30:26.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:26 smithi123 bash[19119]: cluster 2024-02-12T16:30:24.971185+0000 osd.10 (osd.10) 128 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:30:26.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:26 smithi123 bash[19119]: cluster 2024-02-12T16:30:25.021147+0000 osd.10 (osd.10) 129 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:30:26.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:26 smithi123 bash[19119]: cluster 2024-02-12T16:30:25.508015+0000 osd.7 (osd.7) 164 : cluster [DBG] 6.5 scrub starts 2024-02-12T16:30:26.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:26.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:26.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:26.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:26.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:26.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:26.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:26.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:30:27.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:27.069 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:30:27.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:27.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:27.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.236 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:27.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:27.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:27.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:27.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:27.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:27 smithi123 bash[19119]: cluster 2024-02-12T16:30:26.014682+0000 osd.10 (osd.10) 130 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:30:27.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:27 smithi123 bash[19119]: cluster 2024-02-12T16:30:26.030798+0000 mgr.y (mgr.14146) 1371 : cluster [DBG] pgmap v1912: 37 pgs: 37 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:30:27.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:27 smithi123 bash[19119]: cluster 2024-02-12T16:30:26.061159+0000 osd.10 (osd.10) 131 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:30:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:27 smithi121 bash[18489]: cluster 2024-02-12T16:30:26.014682+0000 osd.10 (osd.10) 130 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:30:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:27 smithi121 bash[18489]: cluster 2024-02-12T16:30:26.030798+0000 mgr.y (mgr.14146) 1371 : cluster [DBG] pgmap v1912: 37 pgs: 37 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:30:27.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:27 smithi121 bash[18489]: cluster 2024-02-12T16:30:26.061159+0000 osd.10 (osd.10) 131 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:30:27.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:27 smithi169 bash[19604]: cluster 2024-02-12T16:30:26.014682+0000 osd.10 (osd.10) 130 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:30:27.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:27 smithi169 bash[19604]: cluster 2024-02-12T16:30:26.030798+0000 mgr.y (mgr.14146) 1371 : cluster [DBG] pgmap v1912: 37 pgs: 37 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:30:27.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:27 smithi169 bash[19604]: cluster 2024-02-12T16:30:26.061159+0000 osd.10 (osd.10) 131 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:30:27.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:27.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:27.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 126021 126005 133.468 63.875 0.00980711 0.00748984 2024-02-12T16:30:27.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.630 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:27.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:27.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:27.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:27.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:27.892 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:27.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:27.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:28.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:28.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:28.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:28.264 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:28.294 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:28.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:28 smithi169 bash[19604]: audit 2024-02-12T16:30:27.047985+0000 mgr.y (mgr.14146) 1372 : audit [DBG] from='client.16458 v1:172.21.15.121:0/489442525' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:30:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:28 smithi169 bash[19604]: cluster 2024-02-12T16:30:27.112918+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:30:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:28 smithi169 bash[19604]: cluster 2024-02-12T16:30:27.497701+0000 osd.7 (osd.7) 165 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:30:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:28 smithi169 bash[19604]: cluster 2024-02-12T16:30:27.875628+0000 osd.0 (osd.0) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:28 smithi169 bash[19604]: cluster 2024-02-12T16:30:27.875679+0000 osd.0 (osd.0) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:28.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:28.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:28 smithi121 bash[18489]: audit 2024-02-12T16:30:27.047985+0000 mgr.y (mgr.14146) 1372 : audit [DBG] from='client.16458 v1:172.21.15.121:0/489442525' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:30:28.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:28 smithi121 bash[18489]: cluster 2024-02-12T16:30:27.112918+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:30:28.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:28 smithi121 bash[18489]: cluster 2024-02-12T16:30:27.497701+0000 osd.7 (osd.7) 165 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:30:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:28 smithi121 bash[18489]: cluster 2024-02-12T16:30:27.875628+0000 osd.0 (osd.0) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:28 smithi121 bash[18489]: cluster 2024-02-12T16:30:27.875679+0000 osd.0 (osd.0) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:28.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:28.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:28.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:30:28.566049 min lat: 0.00101494 max lat: 0.39282 avg lat: 0.00753496 2024-02-12T16:30:28.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:30:28.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 127403 127387 132.683 86.375 0.00540449 0.00753496 2024-02-12T16:30:28.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:28.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:28 smithi123 bash[19119]: audit 2024-02-12T16:30:27.047985+0000 mgr.y (mgr.14146) 1372 : audit [DBG] from='client.16458 v1:172.21.15.121:0/489442525' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:30:28.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:28 smithi123 bash[19119]: cluster 2024-02-12T16:30:27.112918+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.4 scrub ok 2024-02-12T16:30:28.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:28 smithi123 bash[19119]: cluster 2024-02-12T16:30:27.497701+0000 osd.7 (osd.7) 165 : cluster [DBG] 6.5 scrub ok 2024-02-12T16:30:28.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:28 smithi123 bash[19119]: cluster 2024-02-12T16:30:27.875628+0000 osd.0 (osd.0) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:28.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:28 smithi123 bash[19119]: cluster 2024-02-12T16:30:27.875679+0000 osd.0 (osd.0) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:28.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:28.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:28.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:28.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:28.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:30:28.943 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:28.966 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1912,"stamp":"2024-02-12T16:30:26.030523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7948664832,"num_objects":121287,"num_object_clones":0,"num_object_copies":242574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121287,"num_write_kb":7762368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121287,"ondisk_log_size":121287,"up":74,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":20610584,"kb_used_data":15452464,"kb_used_omap":91,"kb_used_meta":5157860,"kb_avail":1104331240,"statfs":{"total":1151940427776,"available":1130835189760,"internally_reserved":0,"allocated":15823323136,"data_stored":15819242628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":5281648656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,9,7,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":906428416,"num_objects":13831,"num_object_clones":0,"num_object_copies":27662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13831,"num_write_kb":885184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002227"},"pg_stats":[{"pgid":"1.14","version":"0'0","reported_seq":155,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.131293+0000","last_change":"2024-02-12T16:29:20.828257+0000","last_active":"2024-02-12T16:30:15.131293+0000","last_peered":"2024-02-12T16:30:15.131293+0000","last_clean":"2024-02-12T16:30:15.131293+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:30:15.131293+0000","last_undegraded":"2024-02-12T16:30:15.131293+0000","last_fullsized":"2024-02-12T16:30:15.131293+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:20.828149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:29:20.828149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"658'7410","reported_seq":7439,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:22.118747+0000","last_change":"2024-02-12T16:29:08.737178+0000","last_active":"2024-02-12T16:30:22.118747+0000","last_peered":"2024-02-12T16:30:22.118747+0000","last_clean":"2024-02-12T16:30:22.118747+0000","last_became_active":"2024-02-12T16:29:08.736890+0000","last_became_peered":"2024-02-12T16:29:08.736890+0000","last_unstale":"2024-02-12T16:30:22.118747+0000","last_undegraded":"2024-02-12T16:30:22.118747+0000","last_fullsized":"2024-02-12T16:30:22.118747+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7410,"ondisk_log_size":7410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":485621760,"num_objects":7410,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7410,"num_write_kb":474240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":272,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.132221+0000","last_change":"2024-02-12T16:30:09.838606+0000","last_active":"2024-02-12T16:30:15.132221+0000","last_peered":"2024-02-12T16:30:15.132221+0000","last_clean":"2024-02-12T16:30:15.132221+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:30:15.132221+0000","last_undegraded":"2024-02-12T16:30:15.132221+0000","last_fullsized":"2024-02-12T16:30:15.132221+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.838471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.838471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"658'7664","reported_seq":7692,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:24.362293+0000","last_change":"2024-02-12T16:29:08.734672+0000","last_active":"2024-02-12T16:30:24.362293+0000","last_peered":"2024-02-12T16:30:24.362293+0000","last_clean":"2024-02-12T16:30:24.362293+0000","last_became_active":"2024-02-12T16:29:08.734223+0000","last_became_peered":"2024-02-12T16:29:08.734223+0000","last_unstale":"2024-02-12T16:30:24.362293+0000","last_undegraded":"2024-02-12T16:30:24.362293+0000","last_fullsized":"2024-02-12T16:30:24.362293+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7664,"ondisk_log_size":7664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":502267904,"num_objects":7664,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":490496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":344,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.125983+0000","last_change":"2024-02-12T16:28:54.310544+0000","last_active":"2024-02-12T16:30:15.125983+0000","last_peered":"2024-02-12T16:30:15.125983+0000","last_clean":"2024-02-12T16:30:15.125983+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:17:37.453616+0000","last_unstale":"2024-02-12T16:30:15.125983+0000","last_undegraded":"2024-02-12T16:30:15.125983+0000","last_fullsized":"2024-02-12T16:30:15.125983+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:54.310448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:28:54.310448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"658'7490","reported_seq":7517,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:23.331032+0000","last_change":"2024-02-12T16:29:08.805574+0000","last_active":"2024-02-12T16:30:23.331032+0000","last_peered":"2024-02-12T16:30:23.331032+0000","last_clean":"2024-02-12T16:30:23.331032+0000","last_became_active":"2024-02-12T16:29:08.805299+0000","last_became_peered":"2024-02-12T16:29:08.805299+0000","last_unstale":"2024-02-12T16:30:23.331032+0000","last_undegraded":"2024-02-12T16:30:23.331032+0000","last_fullsized":"2024-02-12T16:30:23.331032+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7490,"ondisk_log_size":7490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":490864640,"num_objects":7490,"num_object_clones":0,"num_object_copies":14980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":753,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.262664+0000","last_change":"2024-02-12T16:30:10.264990+0000","last_active":"2024-02-12T16:30:15.262664+0000","last_peered":"2024-02-12T16:30:15.262664+0000","last_clean":"2024-02-12T16:30:15.262664+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:30:15.262664+0000","last_undegraded":"2024-02-12T16:30:15.262664+0000","last_fullsized":"2024-02-12T16:30:15.262664+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.264894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.264894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.0","version":"658'7726","reported_seq":7755,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:25.669333+0000","last_change":"2024-02-12T16:29:08.733321+0000","last_active":"2024-02-12T16:30:25.669333+0000","last_peered":"2024-02-12T16:30:25.669333+0000","last_clean":"2024-02-12T16:30:25.669333+0000","last_became_active":"2024-02-12T16:29:08.732997+0000","last_became_peered":"2024-02-12T16:29:08.732997+0000","last_unstale":"2024-02-12T16:30:25.669333+0000","last_undegraded":"2024-02-12T16:30:25.669333+0000","last_fullsized":"2024-02-12T16:30:25.669333+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7726,"ondisk_log_size":7726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":506331136,"num_objects":7726,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7726,"num_write_kb":494464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":393,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:23.709791+0000","last_change":"2024-02-12T16:30:23.709791+0000","last_active":"2024-02-12T16:30:23.709791+0000","last_peered":"2024-02-12T16:30:23.709791+0000","last_clean":"2024-02-12T16:30:23.709791+0000","last_became_active":"2024-02-12T16:17:37.451106+0000","last_became_peered":"2024-02-12T16:17:37.451106+0000","last_unstale":"2024-02-12T16:30:23.709791+0000","last_undegraded":"2024-02-12T16:30:23.709791+0000","last_fullsized":"2024-02-12T16:30:23.709791+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:23.709532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:23.709532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"658'7783","reported_seq":7817,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:23.837697+0000","last_change":"2024-02-12T16:30:14.669480+0000","last_active":"2024-02-12T16:30:23.837697+0000","last_peered":"2024-02-12T16:30:23.837697+0000","last_clean":"2024-02-12T16:30:23.837697+0000","last_became_active":"2024-02-12T16:29:08.808944+0000","last_became_peered":"2024-02-12T16:29:08.808944+0000","last_unstale":"2024-02-12T16:30:23.837697+0000","last_undegraded":"2024-02-12T16:30:23.837697+0000","last_fullsized":"2024-02-12T16:30:23.837697+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"656'6950","last_scrub_stamp":"2024-02-12T16:30:14.669359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:14.669359+0000","log_size":7783,"ondisk_log_size":7783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":510066688,"num_objects":7783,"num_object_clones":0,"num_object_copies":15566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7783,"num_write_kb":498112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":335,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.125994+0000","last_change":"2024-02-12T16:28:56.397511+0000","last_active":"2024-02-12T16:30:15.125994+0000","last_peered":"2024-02-12T16:30:15.125994+0000","last_clean":"2024-02-12T16:30:15.125994+0000","last_became_active":"2024-02-12T16:17:37.456792+0000","last_became_peered":"2024-02-12T16:17:37.456792+0000","last_unstale":"2024-02-12T16:30:15.125994+0000","last_undegraded":"2024-02-12T16:30:15.125994+0000","last_fullsized":"2024-02-12T16:30:15.125994+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:56.397266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:56.397266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"658'7608","reported_seq":7636,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:25.282973+0000","last_change":"2024-02-12T16:29:08.733146+0000","last_active":"2024-02-12T16:30:25.282973+0000","last_peered":"2024-02-12T16:30:25.282973+0000","last_clean":"2024-02-12T16:30:25.282973+0000","last_became_active":"2024-02-12T16:29:08.732848+0000","last_became_peered":"2024-02-12T16:29:08.732848+0000","last_unstale":"2024-02-12T16:30:25.282973+0000","last_undegraded":"2024-02-12T16:30:25.282973+0000","last_fullsized":"2024-02-12T16:30:25.282973+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7608,"ondisk_log_size":7608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":498597888,"num_objects":7608,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7608,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":347,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.125992+0000","last_change":"2024-02-12T16:30:14.074480+0000","last_active":"2024-02-12T16:30:15.125992+0000","last_peered":"2024-02-12T16:30:15.125992+0000","last_clean":"2024-02-12T16:30:15.125992+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:30:15.125992+0000","last_undegraded":"2024-02-12T16:30:15.125992+0000","last_fullsized":"2024-02-12T16:30:15.125992+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.383742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.383742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"658'7445","reported_seq":7474,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:21.689563+0000","last_change":"2024-02-12T16:29:08.734515+0000","last_active":"2024-02-12T16:30:21.689563+0000","last_peered":"2024-02-12T16:30:21.689563+0000","last_clean":"2024-02-12T16:30:21.689563+0000","last_became_active":"2024-02-12T16:29:08.734296+0000","last_became_peered":"2024-02-12T16:29:08.734296+0000","last_unstale":"2024-02-12T16:30:21.689563+0000","last_undegraded":"2024-02-12T16:30:21.689563+0000","last_fullsized":"2024-02-12T16:30:21.689563+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7445,"ondisk_log_size":7445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":487915520,"num_objects":7445,"num_object_clones":0,"num_object_copies":14890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7445,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":209,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.131661+0000","last_change":"2024-02-12T16:30:10.604389+0000","last_active":"2024-02-12T16:30:15.131661+0000","last_peered":"2024-02-12T16:30:15.131661+0000","last_clean":"2024-02-12T16:30:15.131661+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:30:15.131661+0000","last_undegraded":"2024-02-12T16:30:15.131661+0000","last_fullsized":"2024-02-12T16:30:15.131661+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.604303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.604303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"658'7628","reported_seq":7657,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:23.271260+0000","last_change":"2024-02-12T16:29:08.804530+0000","last_active":"2024-02-12T16:30:23.271260+0000","last_peered":"2024-02-12T16:30:23.271260+0000","last_clean":"2024-02-12T16:30:23.271260+0000","last_became_active":"2024-02-12T16:29:08.804218+0000","last_became_peered":"2024-02-12T16:29:08.804218+0000","last_unstale":"2024-02-12T16:30:23.271260+0000","last_undegraded":"2024-02-12T16:30:23.271260+0000","last_fullsized":"2024-02-12T16:30:23.271260+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7628,"ondisk_log_size":7628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":499908608,"num_objects":7628,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":488192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":781,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.131984+0000","last_change":"2024-02-12T16:30:09.726295+0000","last_active":"2024-02-12T16:30:15.131984+0000","last_peered":"2024-02-12T16:30:15.131984+0000","last_clean":"2024-02-12T16:30:15.131984+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:30:15.131984+0000","last_undegraded":"2024-02-12T16:30:15.131984+0000","last_fullsized":"2024-02-12T16:30:15.131984+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.726187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.726187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"658'7376","reported_seq":7410,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:22.100709+0000","last_change":"2024-02-12T16:30:21.757551+0000","last_active":"2024-02-12T16:30:22.100709+0000","last_peered":"2024-02-12T16:30:22.100709+0000","last_clean":"2024-02-12T16:30:22.100709+0000","last_became_active":"2024-02-12T16:29:08.804047+0000","last_became_peered":"2024-02-12T16:29:08.804047+0000","last_unstale":"2024-02-12T16:30:22.100709+0000","last_undegraded":"2024-02-12T16:30:22.100709+0000","last_fullsized":"2024-02-12T16:30:22.100709+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7354","last_scrub_stamp":"2024-02-12T16:30:21.757462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:21.757462+0000","log_size":7376,"ondisk_log_size":7376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":483393536,"num_objects":7376,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7376,"num_write_kb":472064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":156,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:17.602700+0000","last_change":"2024-02-12T16:30:17.602679+0000","last_active":"2024-02-12T16:30:17.602700+0000","last_peered":"2024-02-12T16:30:17.602700+0000","last_clean":"2024-02-12T16:30:17.602700+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:23:39.022679+0000","last_unstale":"2024-02-12T16:30:17.602700+0000","last_undegraded":"2024-02-12T16:30:17.602700+0000","last_fullsized":"2024-02-12T16:30:17.602700+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:17.602619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.602619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"658'7756","reported_seq":7784,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:25.279401+0000","last_change":"2024-02-12T16:29:08.734597+0000","last_active":"2024-02-12T16:30:25.279401+0000","last_peered":"2024-02-12T16:30:25.279401+0000","last_clean":"2024-02-12T16:30:25.279401+0000","last_became_active":"2024-02-12T16:29:08.734174+0000","last_became_peered":"2024-02-12T16:29:08.734174+0000","last_unstale":"2024-02-12T16:30:25.279401+0000","last_undegraded":"2024-02-12T16:30:25.279401+0000","last_fullsized":"2024-02-12T16:30:25.279401+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7756,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":508297216,"num_objects":7756,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":496384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":181,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:18.631494+0000","last_change":"2024-02-12T16:30:18.631464+0000","last_active":"2024-02-12T16:30:18.631494+0000","last_peered":"2024-02-12T16:30:18.631494+0000","last_clean":"2024-02-12T16:30:18.631494+0000","last_became_active":"2024-02-12T16:23:39.027758+0000","last_became_peered":"2024-02-12T16:23:39.027758+0000","last_unstale":"2024-02-12T16:30:18.631494+0000","last_undegraded":"2024-02-12T16:30:18.631494+0000","last_fullsized":"2024-02-12T16:30:18.631494+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:18.631349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:18.631349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"658'7453","reported_seq":7482,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:23.315368+0000","last_change":"2024-02-12T16:29:08.734976+0000","last_active":"2024-02-12T16:30:23.315368+0000","last_peered":"2024-02-12T16:30:23.315368+0000","last_clean":"2024-02-12T16:30:23.315368+0000","last_became_active":"2024-02-12T16:29:08.734560+0000","last_became_peered":"2024-02-12T16:29:08.734560+0000","last_unstale":"2024-02-12T16:30:23.315368+0000","last_undegraded":"2024-02-12T16:30:23.315368+0000","last_fullsized":"2024-02-12T16:30:23.315368+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7453,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":488439808,"num_objects":7453,"num_object_clones":0,"num_object_copies":14906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":476992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":354,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.129272+0000","last_change":"2024-02-12T16:30:14.850065+0000","last_active":"2024-02-12T16:30:15.129272+0000","last_peered":"2024-02-12T16:30:15.129272+0000","last_clean":"2024-02-12T16:30:15.129272+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:30:15.129272+0000","last_undegraded":"2024-02-12T16:30:15.129272+0000","last_fullsized":"2024-02-12T16:30:15.129272+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:14.849876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:30:14.849876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"658'7732","reported_seq":7761,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:25.664491+0000","last_change":"2024-02-12T16:29:08.807541+0000","last_active":"2024-02-12T16:30:25.664491+0000","last_peered":"2024-02-12T16:30:25.664491+0000","last_clean":"2024-02-12T16:30:25.664491+0000","last_became_active":"2024-02-12T16:29:08.807333+0000","last_became_peered":"2024-02-12T16:29:08.807333+0000","last_unstale":"2024-02-12T16:30:25.664491+0000","last_undegraded":"2024-02-12T16:30:25.664491+0000","last_fullsized":"2024-02-12T16:30:25.664491+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7732,"ondisk_log_size":7732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":506724352,"num_objects":7732,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":156,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.127307+0000","last_change":"2024-02-12T16:30:09.919779+0000","last_active":"2024-02-12T16:30:15.127307+0000","last_peered":"2024-02-12T16:30:15.127307+0000","last_clean":"2024-02-12T16:30:15.127307+0000","last_became_active":"2024-02-12T16:23:39.023052+0000","last_became_peered":"2024-02-12T16:23:39.023052+0000","last_unstale":"2024-02-12T16:30:15.127307+0000","last_undegraded":"2024-02-12T16:30:15.127307+0000","last_fullsized":"2024-02-12T16:30:15.127307+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.919678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.919678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"658'7582","reported_seq":7615,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:24.366905+0000","last_change":"2024-02-12T16:30:23.174523+0000","last_active":"2024-02-12T16:30:24.366905+0000","last_peered":"2024-02-12T16:30:24.366905+0000","last_clean":"2024-02-12T16:30:24.366905+0000","last_became_active":"2024-02-12T16:29:08.735885+0000","last_became_peered":"2024-02-12T16:29:08.735885+0000","last_unstale":"2024-02-12T16:30:24.366905+0000","last_undegraded":"2024-02-12T16:30:24.366905+0000","last_fullsized":"2024-02-12T16:30:24.366905+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7481","last_scrub_stamp":"2024-02-12T16:30:23.174453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:23.174453+0000","log_size":7582,"ondisk_log_size":7582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":496893952,"num_objects":7582,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7582,"num_write_kb":485248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":751,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.132060+0000","last_change":"2024-02-12T16:28:58.797711+0000","last_active":"2024-02-12T16:30:15.132060+0000","last_peered":"2024-02-12T16:30:15.132060+0000","last_clean":"2024-02-12T16:30:15.132060+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:15:24.790824+0000","last_unstale":"2024-02-12T16:30:15.132060+0000","last_undegraded":"2024-02-12T16:30:15.132060+0000","last_fullsized":"2024-02-12T16:30:15.132060+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:28:58.797578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:28:58.797578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"658'7422","reported_seq":7450,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:21.435950+0000","last_change":"2024-02-12T16:29:08.834927+0000","last_active":"2024-02-12T16:30:21.435950+0000","last_peered":"2024-02-12T16:30:21.435950+0000","last_clean":"2024-02-12T16:30:21.435950+0000","last_became_active":"2024-02-12T16:29:08.834297+0000","last_became_peered":"2024-02-12T16:29:08.834297+0000","last_unstale":"2024-02-12T16:30:21.435950+0000","last_undegraded":"2024-02-12T16:30:21.435950+0000","last_fullsized":"2024-02-12T16:30:21.435950+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7422,"ondisk_log_size":7422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":486408192,"num_objects":7422,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7422,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":343,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.130614+0000","last_change":"2024-02-12T16:30:10.761907+0000","last_active":"2024-02-12T16:30:15.130614+0000","last_peered":"2024-02-12T16:30:15.130614+0000","last_clean":"2024-02-12T16:30:15.130614+0000","last_became_active":"2024-02-12T16:15:24.790561+0000","last_became_peered":"2024-02-12T16:15:24.790561+0000","last_unstale":"2024-02-12T16:30:15.130614+0000","last_undegraded":"2024-02-12T16:30:15.130614+0000","last_fullsized":"2024-02-12T16:30:15.130614+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.761836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.761836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"658'7748","reported_seq":7781,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:25.292447+0000","last_change":"2024-02-12T16:30:17.583794+0000","last_active":"2024-02-12T16:30:25.292447+0000","last_peered":"2024-02-12T16:30:25.292447+0000","last_clean":"2024-02-12T16:30:25.292447+0000","last_became_active":"2024-02-12T16:29:08.804528+0000","last_became_peered":"2024-02-12T16:29:08.804528+0000","last_unstale":"2024-02-12T16:30:25.292447+0000","last_undegraded":"2024-02-12T16:30:25.292447+0000","last_fullsized":"2024-02-12T16:30:25.292447+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"657'7134","last_scrub_stamp":"2024-02-12T16:30:17.583727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.583727+0000","log_size":7748,"ondisk_log_size":7748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":507772928,"num_objects":7748,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":495872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":95,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.262631+0000","last_change":"2024-02-12T16:30:13.391795+0000","last_active":"2024-02-12T16:30:15.262631+0000","last_peered":"2024-02-12T16:30:15.262631+0000","last_clean":"2024-02-12T16:30:15.262631+0000","last_became_active":"2024-02-12T16:26:10.060510+0000","last_became_peered":"2024-02-12T16:26:10.060510+0000","last_unstale":"2024-02-12T16:30:15.262631+0000","last_undegraded":"2024-02-12T16:30:15.262631+0000","last_fullsized":"2024-02-12T16:30:15.262631+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:13.391676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:13.391676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"658'7464","reported_seq":7493,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-02-12T16:30:22.096463+0000","last_change":"2024-02-12T16:29:08.735934+0000","last_active":"2024-02-12T16:30:22.096463+0000","last_peered":"2024-02-12T16:30:22.096463+0000","last_clean":"2024-02-12T16:30:22.096463+0000","last_became_active":"2024-02-12T16:29:08.735711+0000","last_became_peered":"2024-02-12T16:29:08.735711+0000","last_unstale":"2024-02-12T16:30:22.096463+0000","last_undegraded":"2024-02-12T16:30:22.096463+0000","last_fullsized":"2024-02-12T16:30:22.096463+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":7464,"ondisk_log_size":7464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":489160704,"num_objects":7464,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7464,"num_write_kb":477696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":335,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:17.846402+0000","last_change":"2024-02-12T16:30:17.846377+0000","last_active":"2024-02-12T16:30:17.846402+0000","last_peered":"2024-02-12T16:30:17.846402+0000","last_clean":"2024-02-12T16:30:17.846402+0000","last_became_active":"2024-02-12T16:15:24.790723+0000","last_became_peered":"2024-02-12T16:15:24.790723+0000","last_unstale":"2024-02-12T16:30:17.846402+0000","last_undegraded":"2024-02-12T16:30:17.846402+0000","last_fullsized":"2024-02-12T16:30:17.846402+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:17.846300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.846300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":152,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.130602+0000","last_change":"2024-02-12T16:30:12.797215+0000","last_active":"2024-02-12T16:30:15.130602+0000","last_peered":"2024-02-12T16:30:15.130602+0000","last_clean":"2024-02-12T16:30:15.130602+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:30:15.130602+0000","last_undegraded":"2024-02-12T16:30:15.130602+0000","last_fullsized":"2024-02-12T16:30:15.130602+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:12.797116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:30:12.797116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":151,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.128509+0000","last_change":"2024-02-12T16:30:10.717293+0000","last_active":"2024-02-12T16:30:15.128509+0000","last_peered":"2024-02-12T16:30:15.128509+0000","last_clean":"2024-02-12T16:30:15.128509+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:30:15.128509+0000","last_undegraded":"2024-02-12T16:30:15.128509+0000","last_fullsized":"2024-02-12T16:30:15.128509+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.717201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.717201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":166,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:18.905788+0000","last_change":"2024-02-12T16:30:18.905746+0000","last_active":"2024-02-12T16:30:18.905788+0000","last_peered":"2024-02-12T16:30:18.905788+0000","last_clean":"2024-02-12T16:30:18.905788+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:30:18.905788+0000","last_undegraded":"2024-02-12T16:30:18.905788+0000","last_fullsized":"2024-02-12T16:30:18.905788+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:18.901023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:30:18.901023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":145,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-02-12T16:30:15.126933+0000","last_change":"2024-02-12T16:30:09.745507+0000","last_active":"2024-02-12T16:30:15.126933+0000","last_peered":"2024-02-12T16:30:15.126933+0000","last_clean":"2024-02-12T16:30:15.126933+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:30:15.126933+0000","last_undegraded":"2024-02-12T16:30:15.126933+0000","last_fullsized":"2024-02-12T16:30:15.126933+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.745399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.745399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":7948664832,"num_objects":121287,"num_object_clones":0,"num_object_copies":242574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121287,"num_write_kb":7762368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15904800768,"data_stored":15904800768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121287,"ondisk_log_size":121287,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958915,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586304,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":93158848,"statfs":{"total":95995035648,"available":95394660352,"internally_reserved":0,"allocated":675840,"data_stored":335791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.80500000000000005,"15min":0.85499999999999998},"min":{"1min":0.315,"5min":0.246,"15min":0.246},"max":{"1min":1.5029999999999999,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.94799999999999995,"15min":0.90900000000000003},"min":{"1min":0.39000000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.395,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.93400000000000005,"15min":1.0620000000000001},"min":{"1min":0.30099999999999999,"5min":0.27200000000000002,"15min":0.27100000000000002},"max":{"1min":1.385,"5min":5.5430000000000001,"15min":7.0110000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.89000000000000001,"15min":1.0640000000000001},"min":{"1min":0.36499999999999999,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":1.4830000000000001,"5min":3.6859999999999999,"15min":5.6920000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.89800000000000002,"15min":1.2090000000000001},"min":{"1min":0.441,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":1.6859999999999999,"5min":4.1639999999999997,"15min":5.343},"last":0.433},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.879,"15min":1.268},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":3.96,"5min":5.4340000000000002,"15min":8.2759999999999998},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.874,"15min":1.034},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":1.8009999999999999,"5min":4.9900000000000002,"15min":6.5970000000000004},"last":2.657},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.86799999999999999,"15min":0.93500000000000005},"min":{"1min":0.371,"5min":0.36799999999999999,"15min":0.34499999999999997},"max":{"1min":1.7350000000000001,"5min":4.101,"15min":4.915},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.125,"15min":1.1499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.35099999999999998},"max":{"1min":1.7769999999999999,"5min":5.04,"15min":5.8789999999999996},"last":2.3879999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.024,"15min":1.1319999999999999},"min":{"1min":0.498,"5min":0.35499999999999998,"15min":0.35399999999999998},"max":{"1min":1.794,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.90400000000000003,"15min":0.98799999999999999},"min":{"1min":0.45200000000000001,"5min":0.378,"15min":0.373},"max":{"1min":1.522,"5min":4.7370000000000001,"15min":5.141},"last":2.5329999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.026,"15min":1.125},"min":{"1min":0.54100000000000004,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.905,"5min":4.7450000000000001,"15min":8.5709999999999997},"last":2.5800000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.91200000000000003,"15min":1.1859999999999999},"min":{"1min":0.46899999999999997,"5min":0.39000000000000001,"15min":0.38600000000000001},"max":{"1min":1.718,"5min":3.9089999999999998,"15min":6.4370000000000003},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":0.84999999999999998,"15min":1.1160000000000001},"min":{"1min":0.52700000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.867,"5min":4.2350000000000003,"15min":5.0430000000000001},"last":0.54800000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":6.2039999999999997,"15min":2.7970000000000002},"min":{"1min":0.45600000000000002,"5min":0.371,"15min":0.371},"max":{"1min":1.625,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.27,"5min":6.319,"15min":2.9689999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.34699999999999998},"max":{"1min":1.722,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.502}]},{"osd":9,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98099999999999998,"15min":1.276},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.736,"5min":3.8100000000000001,"15min":10.743},"last":2.4329999999999998},{"interface":"front","average":{"1min":1.397,"5min":0.998,"15min":1.357},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7809999999999999,"5min":3.8399999999999999,"15min":10.698},"last":0.46500000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":6.327,"15min":2.9590000000000001},"min":{"1min":0.50900000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.6850000000000001,"5min":536.245,"15min":536.245},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":6.3339999999999996,"15min":2.9399999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.901,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.60199999999999998}]}]},{"osd":10,"up_from":83,"seq":356482285964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":800348,"kb_used_data":482644,"kb_used_omap":0,"kb_used_meta":317696,"kb_avail":92944804,"statfs":{"total":95995035648,"available":95175479296,"internally_reserved":0,"allocated":494227456,"data_stored":493887407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":325320704},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":2.7610000000000001,"15min":1.5249999999999999},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.245},"max":{"1min":1.591,"5min":162.38399999999999,"15min":162.38399999999999},"last":1.7629999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":2.8340000000000001,"15min":1.6040000000000001},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.4179999999999999,"5min":162.01300000000001,"15min":162.01300000000001},"last":1.4119999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.008,"15min":1.006},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":1.6140000000000001,"5min":5.9000000000000004,"15min":19.866},"last":1.698},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.999,"15min":1.1399999999999999},"min":{"1min":0.372,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":1.504,"5min":6.1970000000000001,"15min":20.277999999999999},"last":1.488}]},{"osd":2,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.1990000000000001,"15min":1.0009999999999999},"min":{"1min":0.316,"5min":0.29899999999999999,"15min":0.28899999999999998},"max":{"1min":1.7230000000000001,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":1.6220000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.095,"15min":1.03},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.29699999999999999},"max":{"1min":1.77,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.93200000000000005,"15min":1.034},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32300000000000001},"max":{"1min":2.9249999999999998,"5min":5.8109999999999999,"15min":5.8109999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.101,"15min":1.0469999999999999},"min":{"1min":0.34899999999999998,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":5.9119999999999999,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.1419999999999999,"15min":1.095},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.6349999999999998,"5min":6.343,"15min":6.343},"last":0.375},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.196,"15min":1.196},"min":{"1min":0.54000000000000004,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":3.7759999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":0.63100000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0569999999999999,"15min":1.093},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":3.9100000000000001,"5min":6.3010000000000002,"15min":6.3010000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.0940000000000001,"15min":1.1739999999999999},"min":{"1min":0.443,"5min":0.376,"15min":0.376},"max":{"1min":2.1520000000000001,"5min":6.8200000000000003,"15min":6.8200000000000003},"last":2.8069999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.018,"15min":1.0900000000000001},"min":{"1min":0.433,"5min":0.375,"15min":0.375},"max":{"1min":2.484,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":3.0249999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.177,"15min":1.2170000000000001},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":2.0859999999999999,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.91000000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.054,"15min":1.1479999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.372},"max":{"1min":2.3300000000000001,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.99299999999999999,"15min":1.1359999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.35799999999999998},"max":{"1min":1.8129999999999999,"5min":7.282,"15min":7.282},"last":0.42199999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1859999999999999,"15min":1.2569999999999999},"min":{"1min":0.51300000000000001,"5min":0.48499999999999999,"15min":0.372},"max":{"1min":2.1499999999999999,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0449999999999999,"15min":1.218},"min":{"1min":0.51400000000000001,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":2.23,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.80500000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:29:30 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.1060000000000001,"15min":1.2809999999999999},"min":{"1min":0.67400000000000004,"5min":0.39400000000000002,"15min":0.39300000000000002},"max":{"1min":2.1219999999999999,"5min":7.3689999999999998,"15min":10.218999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.552,"5min":1.115,"15min":1.296},"min":{"1min":0.63900000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.2189999999999999,"5min":6.5860000000000003,"15min":7.3250000000000002},"last":0.70199999999999996}]}]},{"osd":9,"up_from":74,"seq":317827580308,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1253504,"kb_used_data":952148,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92491648,"statfs":{"total":95995035648,"available":94711447552,"internally_reserved":0,"allocated":974999552,"data_stored":974659511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:09 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.397,"15min":1.145},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":19.120000000000001,"5min":19.120000000000001,"15min":19.120000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.411,"15min":1.0600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.28299999999999997},"max":{"1min":19.222000000000001,"5min":19.222000000000001,"15min":19.222000000000001},"last":0.443}]},{"osd":1,"last update":"Mon Feb 12 16:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.085,"15min":0.99199999999999999},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.253},"max":{"1min":1.6080000000000001,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.98499999999999999,"15min":1.024},"min":{"1min":0.35699999999999998,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":1.1659999999999999,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":0.88800000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.2649999999999999,"15min":1.4530000000000001},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":3.298,"5min":12.441000000000001,"15min":97.275000000000006},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.1659999999999999,"15min":1.3640000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.312},"max":{"1min":1.728,"5min":6.1779999999999999,"15min":97.417000000000002},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0920000000000001,"15min":1.032},"min":{"1min":0.373,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.6600000000000001,"5min":6.3090000000000002,"15min":6.3090000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.113,"5min":1.155,"15min":1.2030000000000001},"min":{"1min":0.49099999999999999,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":3.6299999999999999,"5min":6.3490000000000002,"15min":6.3490000000000002},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.3049999999999999,"15min":1.1819999999999999},"min":{"1min":0.54500000000000004,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":1.468,"5min":6.726,"15min":6.726},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.204,"15min":1.159},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.6659999999999999,"5min":6.7350000000000003,"15min":6.7350000000000003},"last":0.96999999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.998,"15min":1.0940000000000001},"min":{"1min":0.38800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":5.6689999999999996,"5min":5.6689999999999996,"15min":6.1840000000000002},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.264,"15min":1.145},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.36499999999999999},"max":{"1min":5.6920000000000002,"5min":14.718,"15min":14.718},"last":7.71}]},{"osd":7,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1599999999999999,"15min":1.2669999999999999},"min":{"1min":0.60999999999999999,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":1.8109999999999999,"5min":5.8010000000000002,"15min":5.8470000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.143,"15min":1.2110000000000001},"min":{"1min":0.53800000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.6479999999999999,"5min":4.7000000000000002,"15min":7.9160000000000004},"last":0.99399999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:30:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.95599999999999996,"15min":1.0940000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.105,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.272,"5min":1.0820000000000001,"15min":1.204},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42599999999999999},"max":{"1min":3.806,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.68200000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.1399999999999999,"15min":1.262},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.36599999999999999},"max":{"1min":1.913,"5min":5.6929999999999996,"15min":5.6929999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2010000000000001,"15min":1.27},"min":{"1min":0.44700000000000001,"5min":0.39200000000000002,"15min":0.38},"max":{"1min":1.8999999999999999,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":0.81699999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:29:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.1319999999999999,"15min":1.3600000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.45600000000000002},"max":{"1min":2.238,"5min":5.7690000000000001,"15min":5.7690000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.0389999999999999,"15min":1.2849999999999999},"min":{"1min":0.53400000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.78,"5min":4.7190000000000003,"15min":6.3639999999999999},"last":1.0149999999999999}]}]},{"osd":8,"up_from":65,"seq":279172874654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1283740,"kb_used_data":974228,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92461412,"statfs":{"total":95995035648,"available":94680485888,"internally_reserved":0,"allocated":997609472,"data_stored":997269431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.077,"15min":0.90800000000000003},"min":{"1min":0.63500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":2.71,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.2430000000000001,"15min":0.93200000000000005},"min":{"1min":0.63800000000000001,"5min":0.32100000000000001,"15min":0.29399999999999998},"max":{"1min":2.5219999999999998,"5min":18.584,"15min":18.584},"last":0.441}]},{"osd":1,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.9810000000000001,"15min":1.5249999999999999},"min":{"1min":0.72199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.181,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.9670000000000001,"15min":1.6220000000000001},"min":{"1min":0.65700000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.3919999999999999,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.33500000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.84399999999999997},"min":{"1min":0.40400000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.827,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.02,"15min":1.002},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.29899999999999999},"max":{"1min":1.726,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.375}]},{"osd":4,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.754,"15min":1.708},"min":{"1min":0.46300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.458,"5min":5.0490000000000004,"15min":225.33099999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.84999999999999998,"15min":1.681},"min":{"1min":0.57899999999999996,"5min":0.34699999999999998,"15min":0.34000000000000002},"max":{"1min":2.0110000000000001,"5min":5.0380000000000003,"15min":224.71000000000001},"last":0.57099999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.88400000000000001,"15min":0.92000000000000004},"min":{"1min":0.51700000000000002,"5min":0.36499999999999999,"15min":0.32200000000000001},"max":{"1min":1.4259999999999999,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.499},{"interface":"front","average":{"1min":1.345,"5min":0.95199999999999996,"15min":1.0229999999999999},"min":{"1min":0.53200000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.097,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.52500000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.0409999999999999,"15min":1.758},"min":{"1min":0.55100000000000005,"5min":0.373,"15min":0.37},"max":{"1min":1.5609999999999999,"5min":5.3639999999999999,"15min":221.36699999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.0189999999999999,"15min":1.8600000000000001},"min":{"1min":0.625,"5min":0.38,"15min":0.38},"max":{"1min":2.2269999999999999,"5min":4.4930000000000003,"15min":221.203},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.0629999999999999,"15min":1.1299999999999999},"min":{"1min":0.58599999999999997,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":1.742,"5min":4.1970000000000001,"15min":7.9160000000000004},"last":0.874},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.119,"15min":1.2509999999999999},"min":{"1min":0.64800000000000002,"5min":0.50900000000000001,"15min":0.42499999999999999},"max":{"1min":2.1709999999999998,"5min":4.2240000000000002,"15min":7.899},"last":0.68899999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.012,"15min":1.1599999999999999},"min":{"1min":0.45400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.9099999999999999,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.129,"5min":1.1100000000000001,"15min":1.254},"min":{"1min":0.52200000000000002,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":2.2719999999999998,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":0.72099999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":0.95199999999999996,"15min":1.4430000000000001},"min":{"1min":0.50900000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.056,"5min":5.2560000000000002,"15min":109.92},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.25,"5min":0.99099999999999999,"15min":1.488},"min":{"1min":0.55200000000000005,"5min":0.44400000000000001,"15min":0.38500000000000001},"max":{"1min":2.6880000000000002,"5min":5.0250000000000004,"15min":110.249},"last":0.58199999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:29:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.0229999999999999,"15min":1.179},"min":{"1min":0.64200000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.724,"5min":5.2530000000000001,"15min":5.2530000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.2969999999999999,"5min":0.93500000000000005,"15min":1.1359999999999999},"min":{"1min":0.57999999999999996,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.6859999999999999,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":0.60799999999999998}]}]},{"osd":7,"up_from":56,"seq":240518168997,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2214492,"kb_used_data":1913172,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91530660,"statfs":{"total":95995035648,"available":93727395840,"internally_reserved":0,"allocated":1959088128,"data_stored":1958748087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":3.1360000000000001,"15min":1.6819999999999999},"min":{"1min":0.39500000000000002,"5min":0.252,"15min":0.252},"max":{"1min":2.1120000000000001,"5min":187.33699999999999,"15min":187.33699999999999},"last":7.4500000000000002},{"interface":"front","average":{"1min":1.075,"5min":3.0739999999999998,"15min":1.679},"min":{"1min":0.72799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.1850000000000001,"5min":187.48400000000001,"15min":187.48400000000001},"last":7.4989999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.2749999999999999,"15min":1.0669999999999999},"min":{"1min":0.42599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.6000000000000001,"5min":35.655000000000001,"15min":35.655000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.5720000000000001,"15min":1.228},"min":{"1min":0.47499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.145,"5min":35.164000000000001,"15min":35.164000000000001},"last":7.6040000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.83999999999999997,"15min":1.1339999999999999},"min":{"1min":0.315,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.3339999999999996,"5min":4.5140000000000002,"15min":7.1689999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.702,"5min":1.0720000000000001,"15min":1.0189999999999999},"min":{"1min":0.52100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.9139999999999997,"5min":6.7549999999999999,"15min":6.7549999999999999},"last":3.468}]},{"osd":4,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1339999999999999,"15min":0.97099999999999997},"min":{"1min":0.41099999999999998,"5min":0.27400000000000002,"15min":0.26200000000000001},"max":{"1min":2.766,"5min":7.5910000000000002,"15min":7.5910000000000002},"last":0.622},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.95699999999999996,"15min":1.014},"min":{"1min":0.63600000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":1.5209999999999999,"5min":6.9470000000000001,"15min":6.9470000000000001},"last":3.7679999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.018,"15min":0.98499999999999999},"min":{"1min":0.64700000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":7.4029999999999996,"5min":7.4029999999999996,"15min":7.4029999999999996},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.095,"15min":1.0569999999999999},"min":{"1min":0.59699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.2549999999999999,"5min":5.992,"15min":9.3640000000000008},"last":7.556}]},{"osd":6,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.05,"15min":1.077},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.29499999999999998},"max":{"1min":1.7090000000000001,"5min":6.319,"15min":6.319},"last":3.7229999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.1819999999999999,"15min":1.2},"min":{"1min":0.72499999999999998,"5min":0.30599999999999999,"15min":0.29999999999999999},"max":{"1min":3.169,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":3.7360000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":6.5010000000000003,"15min":2.9279999999999999},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8839999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.4970000000000001,"5min":6.7460000000000004,"15min":3.0659999999999998},"min":{"1min":0.42199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":7.0110000000000001,"5min":498.69999999999999,"15min":498.69999999999999},"last":7.6859999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.018,"15min":1.1890000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.3719999999999999,"5min":4.8650000000000002,"15min":12.907999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.472,"5min":1.2709999999999999,"15min":1.323},"min":{"1min":0.58099999999999996,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.2810000000000001,"5min":7.1219999999999999,"15min":7.1219999999999999},"last":0.48099999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:29:39 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":7.8109999999999999,"15min":3.5379999999999998},"min":{"1min":0.85899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":3.367,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":7.7069999999999999,"15min":3.4359999999999999},"min":{"1min":0.81999999999999995,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":2.2400000000000002,"5min":496.29599999999999,"15min":496.29599999999999},"last":7.6470000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.0900000000000001,"15min":1.5940000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.4079999999999999,"5min":4.9039999999999999,"15min":120.943},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.3520000000000001,"15min":1.7490000000000001},"min":{"1min":0.66700000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":3.3279999999999998,"5min":7.0679999999999996,"15min":120.661},"last":0.56399999999999995}]}]},{"osd":6,"up_from":48,"seq":206158430638,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273948,"kb_used_data":1960340,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":91471204,"statfs":{"total":95995035648,"available":93666512896,"internally_reserved":0,"allocated":2007388160,"data_stored":2007048119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":2.1280000000000001,"15min":1.4750000000000001},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.0999999999999996,"5min":107.723,"15min":107.723},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.121,"5min":2.2149999999999999,"15min":1.544},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":6.1529999999999996,"5min":107.114,"15min":107.114},"last":0.51200000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.0489999999999999,"15min":1.9399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":6.1870000000000003,"5min":6.1870000000000003,"15min":252.22800000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.2549999999999999,"5min":0.998,"15min":1.982},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.9459999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.486,"15min":1.0800000000000001},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":6.2089999999999996,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.647,"15min":1.1930000000000001},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.28999999999999998},"max":{"1min":5.4169999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.89000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.2,"15min":1.0409999999999999},"min":{"1min":0.40799999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.117,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.97699999999999998,"15min":1.0580000000000001},"min":{"1min":0.33400000000000002,"5min":0.28799999999999998,"15min":0.23999999999999999},"max":{"1min":6.0170000000000003,"5min":6.0259999999999998,"15min":7.5979999999999999},"last":0.747}]},{"osd":5,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.2090000000000001,"15min":1.0289999999999999},"min":{"1min":0.54100000000000004,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":5.4690000000000003},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.329,"5min":1.038,"15min":1.0269999999999999},"min":{"1min":0.41499999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.0720000000000001,"5min":10.52,"15min":10.52},"last":0.62}]},{"osd":7,"last update":"Mon Feb 12 16:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.90300000000000002,"15min":1.073},"min":{"1min":0.37,"5min":0.312,"15min":0.30199999999999999},"max":{"1min":1.403,"5min":4.0970000000000004,"15min":7.641},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.81699999999999995,"15min":1.0389999999999999},"min":{"1min":0.51600000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.673,"5min":2.8210000000000002,"15min":7.8380000000000001},"last":0.64300000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.2230000000000001,"15min":1.171},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.2190000000000003,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.0289999999999999,"15min":1.204},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.371},"max":{"1min":5.5140000000000002,"5min":6.2270000000000003,"15min":6.2270000000000003},"last":3.3530000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:29:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0289999999999999,"15min":1.2350000000000001},"min":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.5820000000000001,"5min":4.1680000000000001,"15min":18.059999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.3220000000000001,"5min":0.95099999999999996,"15min":1.256},"min":{"1min":0.46000000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.694,"5min":2.694,"15min":17.425000000000001},"last":0.77900000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.218,"15min":1.2869999999999999},"min":{"1min":0.54200000000000004,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.776,"5min":5.726,"15min":6.5300000000000002},"last":3.4060000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.3089999999999999,"15min":1.3819999999999999},"min":{"1min":0.69799999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.8320000000000001,"5min":5.7590000000000003,"15min":6.5730000000000004},"last":3.4860000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:29:48 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.222,"15min":2.0960000000000001},"min":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.499},"max":{"1min":1.7370000000000001,"5min":5.6859999999999999,"15min":139.727},"last":3.4420000000000002},{"interface":"front","average":{"1min":1.4039999999999999,"5min":0.98699999999999999,"15min":2.0049999999999999},"min":{"1min":0.64900000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.9430000000000001,"5min":2.706,"15min":139.24199999999999},"last":0.81899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575315,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322972,"kb_used_data":1424660,"kb_used_omap":0,"kb_used_meta":898304,"kb_avail":91422180,"statfs":{"total":95995035648,"available":93616312320,"internally_reserved":0,"allocated":1458851840,"data_stored":1458511799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":919863296},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":1.095,"15min":0.96599999999999997},"min":{"1min":0.27900000000000003,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":1.135,"5min":16.632999999999999,"15min":16.632999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.88,"5min":1.6339999999999999,"15min":1.2170000000000001},"min":{"1min":0.27600000000000002,"5min":0.26400000000000001,"15min":0.23699999999999999},"max":{"1min":1.7549999999999999,"5min":7.7249999999999996,"15min":11.433},"last":2.3279999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.3580000000000001,"15min":1.1339999999999999},"min":{"1min":0.52600000000000002,"5min":0.29599999999999999,"15min":0.248},"max":{"1min":1.552,"5min":12.778,"15min":12.778},"last":2.3679999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.5860000000000001,"15min":1.2190000000000001},"min":{"1min":0.55300000000000005,"5min":0.246,"15min":0.246},"max":{"1min":2.4169999999999998,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:30:15 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.147,"15min":1.1240000000000001},"min":{"1min":0.39400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":2.8719999999999999,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.454,"5min":1.2669999999999999,"15min":1.0009999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.9100000000000001,"5min":6.5490000000000004,"15min":6.7110000000000003},"last":4.2469999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:29:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.3049999999999999,"15min":1.133},"min":{"1min":0.47199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":2.056,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":3.98},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.369,"15min":1.373},"min":{"1min":0.436,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.78,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.42199999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.514,"15min":1.282},"min":{"1min":0.67600000000000005,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.8169999999999999,"5min":10.109999999999999,"15min":10.109999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.403,"5min":1.3680000000000001,"15min":1.2310000000000001},"min":{"1min":0.79100000000000004,"5min":0.39300000000000002,"15min":0.35899999999999999},"max":{"1min":2.3159999999999998,"5min":10.067,"15min":10.067},"last":0.496}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.712,"15min":1.4059999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.9670000000000001,"5min":30.437000000000001,"15min":30.437000000000001},"last":4.0279999999999996},{"interface":"front","average":{"1min":1.294,"5min":1.5740000000000001,"15min":1.4610000000000001},"min":{"1min":0.47599999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.845,"5min":29.382000000000001,"15min":29.382000000000001},"last":0.56399999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":7.4509999999999996,"15min":3.464},"min":{"1min":0.42999999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.2549999999999999,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.333,"5min":7.5389999999999997,"15min":3.427},"min":{"1min":0.51500000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.0830000000000002,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.78800000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.419,"15min":1.393},"min":{"1min":0.48199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.9930000000000001,"5min":5.7590000000000003,"15min":10.07},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3420000000000001,"15min":1.4650000000000001},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.436},"max":{"1min":2.1920000000000002,"5min":10.106,"15min":10.106},"last":0.71099999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":8.4819999999999993,"15min":3.7959999999999998},"min":{"1min":0.59399999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.673,"5min":548.93499999999995,"15min":548.93499999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.5800000000000001,"5min":8.4350000000000005,"15min":3.8559999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.4929999999999999,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.60199999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:29:28 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.538,"15min":1.5920000000000001},"min":{"1min":1.159,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4689999999999999,"5min":10.202,"15min":10.202},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.597,"15min":1.6399999999999999},"min":{"1min":0.98399999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.387,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":0.82699999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738847,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574684,"kb_used_data":989076,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92170468,"statfs":{"total":95995035648,"available":94382559232,"internally_reserved":0,"allocated":1012813824,"data_stored":1012473783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.141,"15min":1.073},"min":{"1min":0.29099999999999998,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":3.278,"5min":10.754,"15min":11.324},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.3300000000000001,"15min":1.171},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.0609999999999999,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.29899999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.022,"15min":1.044},"min":{"1min":0.38900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.6419999999999999,"5min":7.6059999999999999,"15min":7.6059999999999999},"last":0.375},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.157,"15min":1.0009999999999999},"min":{"1min":0.316,"5min":0.224,"15min":0.224},"max":{"1min":3.5259999999999998,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":3.4630000000000001,"15min":1.8500000000000001},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":3.347,"5min":193.77000000000001,"15min":193.77000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.365,"5min":3.472,"15min":1.95},"min":{"1min":0.38,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":5.9180000000000001,"5min":189.625,"15min":189.625},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.3660000000000001,"15min":1.0700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":3.3700000000000001,"5min":10.49,"15min":10.49},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.3460000000000001,"15min":1.121},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.33600000000000002},"max":{"1min":3.6659999999999999,"5min":7.3250000000000002,"15min":7.3250000000000002},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":3.23,"15min":1.778},"min":{"1min":0.48899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":3.867,"5min":189.79400000000001,"15min":189.79400000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":3.27,"15min":1.8400000000000001},"min":{"1min":0.42099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.2559999999999998,"5min":194.41200000000001,"15min":194.41200000000001},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":3.629,"15min":1.9750000000000001},"min":{"1min":0.53700000000000003,"5min":0.374,"15min":0.374},"max":{"1min":4.0380000000000003,"5min":191.47200000000001,"15min":191.47200000000001},"last":1.75},{"interface":"front","average":{"1min":1.5589999999999999,"5min":3.7250000000000001,"15min":2.0840000000000001},"min":{"1min":0.56799999999999995,"5min":0.44,"15min":0.44},"max":{"1min":4.0800000000000001,"5min":191.50299999999999,"15min":191.50299999999999},"last":1.786}]},{"osd":8,"last update":"Mon Feb 12 16:30:23 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.3620000000000001,"15min":1.3109999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.4039999999999999,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.472,"15min":1.4079999999999999},"min":{"1min":0.52900000000000003,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":3.5299999999999998,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.84399999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.1619999999999999,"15min":1.367},"min":{"1min":0.497,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.73,"5min":5.6550000000000002,"15min":46.246000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.4119999999999999,"15min":1.4630000000000001},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.432},"max":{"1min":3.4900000000000002,"5min":7.5369999999999999,"15min":46.167999999999999},"last":0.63400000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.4630000000000001,"15min":1.397},"min":{"1min":0.48499999999999999,"5min":0.442,"15min":0.41999999999999998},"max":{"1min":3.9169999999999998,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.353,"5min":1.587,"15min":1.409},"min":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":4.0069999999999997,"5min":9.3490000000000002,"15min":9.3490000000000002},"last":0.69499999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:30:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.4570000000000001,"15min":1.5649999999999999},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":3.96,"5min":6.2830000000000004,"15min":46.064},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.482,"15min":1.583},"min":{"1min":0.50700000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.5649999999999999,"5min":7.4210000000000003,"15min":45.642000000000003},"last":0.76500000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313676,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":771804,"kb_used_data":470484,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92973348,"statfs":{"total":95995035648,"available":95204708352,"internally_reserved":0,"allocated":481775616,"data_stored":481435575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.3660000000000001,"15min":1.107},"min":{"1min":0.36199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1160000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.4059999999999999,"15min":1.222},"min":{"1min":0.40300000000000002,"5min":0.247,"15min":0.247},"max":{"1min":1.4399999999999999,"5min":6.484,"15min":9.6470000000000002},"last":0.317}]},{"osd":3,"last update":"Mon Feb 12 16:30:14 2024","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":1.494,"15min":1.837},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":26.745000000000001,"5min":26.745000000000001,"15min":26.745000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.982,"5min":1.371,"15min":1},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":25.260000000000002,"5min":25.260000000000002,"15min":25.260000000000002},"last":0.57899999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.379,"15min":1.0840000000000001},"min":{"1min":0.33500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.5840000000000001,"5min":10.890000000000001,"15min":10.890000000000001},"last":3.8730000000000002},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3600000000000001,"15min":1.2110000000000001},"min":{"1min":0.36699999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.5710000000000002,"5min":6.601,"15min":9.8079999999999998},"last":3.8809999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.3080000000000001,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.35199999999999998,"15min":0.315},"max":{"1min":1.4570000000000001,"5min":6.133,"15min":6.133},"last":3.6520000000000001},{"interface":"front","average":{"1min":1.05,"5min":1.6140000000000001,"15min":1.214},"min":{"1min":0.55800000000000005,"5min":0.315,"15min":0.315},"max":{"1min":1.4370000000000001,"5min":11.081,"15min":11.081},"last":3.5790000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.395,"15min":1.2989999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.444,"5min":7.6890000000000001,"15min":21.527000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.3740000000000001,"15min":1.228},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":1.859,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.63100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.641,"15min":1.24},"min":{"1min":0.38300000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":2.6040000000000001,"5min":11.044,"15min":11.044},"last":0.372},{"interface":"front","average":{"1min":1.143,"5min":1.4299999999999999,"15min":1.272},"min":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.5840000000000001,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":1.1279999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:30:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.367,"15min":1.3149999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.9399999999999999,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.381,"15min":1.355},"min":{"1min":0.497,"5min":0.38,"15min":0.38},"max":{"1min":2.121,"5min":6.593,"15min":9.7639999999999993},"last":0.441}]},{"osd":9,"last update":"Mon Feb 12 16:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.5860000000000001,"15min":1.3520000000000001},"min":{"1min":0.42199999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.1819999999999999,"5min":10.972,"15min":10.972},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.6819999999999999,"15min":1.4330000000000001},"min":{"1min":0.46600000000000003,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":2.2250000000000001,"5min":11.006,"15min":11.006},"last":0.48799999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.609,"15min":1.462},"min":{"1min":0.72899999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.024,"5min":7.7539999999999996,"15min":11.454000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.605,"5min":1.6739999999999999,"15min":1.5009999999999999},"min":{"1min":0.76000000000000001,"5min":0.441,"15min":0.441},"max":{"1min":2.2280000000000002,"5min":7.7140000000000004,"15min":9.7170000000000005},"last":0.70799999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.4450000000000001,"15min":1.4690000000000001},"min":{"1min":0.54400000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.2839999999999998,"5min":7.258,"15min":7.258},"last":0.755},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.5589999999999999,"15min":1.5509999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.0710000000000002,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":0.69799999999999995}]}]},{"osd":3,"up_from":522,"seq":2241972928602,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3273500,"kb_used_data":2941268,"kb_used_omap":91,"kb_used_meta":332132,"kb_avail":90471652,"statfs":{"total":95995035648,"available":92642971648,"internally_reserved":0,"allocated":3011858432,"data_stored":3011518391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":340103184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.325,"15min":1.006},"min":{"1min":0.27400000000000002,"5min":0.27000000000000002,"15min":0.23799999999999999},"max":{"1min":3.9500000000000002,"5min":34.055999999999997,"15min":34.055999999999997},"last":2.3540000000000001},{"interface":"front","average":{"1min":0.751,"5min":1.4370000000000001,"15min":1.0469999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4220000000000002,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.236,"15min":0.98899999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.7480000000000002,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":2.403},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.1850000000000001,"15min":1.091},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.2709999999999999,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":0.88300000000000001,"15min":1.1919999999999999},"min":{"1min":0.33400000000000002,"5min":0.307,"15min":0.307},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":10.004},"last":2.7370000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.1339999999999999,"15min":1.1120000000000001},"min":{"1min":0.42299999999999999,"5min":0.35999999999999999,"15min":0.32600000000000001},"max":{"1min":3.3620000000000001,"5min":6.806,"15min":6.806},"last":2.6269999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.038,"15min":1.272},"min":{"1min":0.47399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":2.923,"5min":4.7530000000000001,"15min":6.7539999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.05,"15min":0.96699999999999997},"min":{"1min":0.40799999999999997,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":2.7970000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":2.694}]},{"osd":6,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.113,"15min":1.1299999999999999},"min":{"1min":0.53700000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.2889999999999997,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":4.7110000000000003},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.1259999999999999,"15min":1.0369999999999999},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.359,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":6.8890000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.077,"15min":1.347},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.8940000000000001,"5min":4.8090000000000002,"15min":6.851},"last":6.7409999999999997},{"interface":"front","average":{"1min":1.27,"5min":1.25,"15min":1.2669999999999999},"min":{"1min":0.51700000000000002,"5min":0.40300000000000002,"15min":0.39200000000000002},"max":{"1min":3.1619999999999999,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":2.5950000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0800000000000001,"15min":1.169},"min":{"1min":0.503,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.968,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.26,"5min":1.0760000000000001,"15min":1.107},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.39800000000000002},"max":{"1min":2.855,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":2.7639999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3009999999999999,"15min":1.329},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.42799999999999999},"max":{"1min":3.2440000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":2.6469999999999998},{"interface":"front","average":{"1min":1.147,"5min":1.274,"15min":1.3560000000000001},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.0369999999999999,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.70999999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.171,"15min":1.425},"min":{"1min":0.58599999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":3.0030000000000001,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":2.1150000000000002,"5min":1.4670000000000001,"15min":1.3740000000000001},"min":{"1min":0.55300000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":15.140000000000001,"5min":15.140000000000001,"15min":15.140000000000001},"last":2.6779999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.3200000000000001,"15min":1.298},"min":{"1min":0.61799999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.194,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":2.71},{"interface":"front","average":{"1min":1.266,"5min":1.2829999999999999,"15min":1.464},"min":{"1min":0.57099999999999995,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":3.0760000000000001,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.80200000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953917,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2048860,"kb_used_data":1446868,"kb_used_omap":0,"kb_used_meta":601984,"kb_avail":91696292,"statfs":{"total":95995035648,"available":93897003008,"internally_reserved":0,"allocated":1481592832,"data_stored":1481252791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":616431616},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":1.101,"15min":1.361},"min":{"1min":0.27200000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.804,"5min":7.492,"15min":90.597999999999999},"last":3.9889999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.2470000000000001,"15min":1.542},"min":{"1min":0.44900000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.9099999999999999,"5min":12.236000000000001,"15min":90.676000000000002},"last":4.04}]},{"osd":2,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.119,"15min":1.006},"min":{"1min":0.38300000000000001,"5min":0.28399999999999997,"15min":0.24399999999999999},"max":{"1min":2.0049999999999999,"5min":7.3630000000000004,"15min":8.0530000000000008},"last":4.0780000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2050000000000001,"15min":1.1359999999999999},"min":{"1min":0.312,"5min":0.254,"15min":0.254},"max":{"1min":1.8169999999999999,"5min":7.633,"15min":7.633},"last":2.8100000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1539999999999999,"15min":1.1080000000000001},"min":{"1min":0.40600000000000003,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":2.802,"5min":12.409000000000001,"15min":12.409000000000001},"last":4.1120000000000001},{"interface":"front","average":{"1min":1.022,"5min":1.0549999999999999,"15min":1.0589999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.79,"5min":7.383,"15min":7.383},"last":2.7679999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.099,"15min":0.98099999999999998},"min":{"1min":0.372,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":7.431,"5min":7.431,"15min":7.431},"last":3.0289999999999999},{"interface":"front","average":{"1min":1.7809999999999999,"5min":1.2470000000000001,"15min":1.149},"min":{"1min":0.42599999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":7.4569999999999999,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":3.0859999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.94899999999999995,"15min":1.0249999999999999},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.3710000000000004,"5min":7.5289999999999999,"15min":7.5410000000000004},"last":4.2009999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.165,"15min":1.1060000000000001},"min":{"1min":0.34999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.1289999999999996,"5min":5.5190000000000001,"15min":9.6639999999999997},"last":3.8740000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.081,"15min":1.0960000000000001},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.9610000000000001,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":4.008},{"interface":"front","average":{"1min":1.264,"5min":0.94899999999999995,"15min":1.149},"min":{"1min":0.47199999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.016,"5min":7.6500000000000004,"15min":7.6689999999999996},"last":7.0979999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:29:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.032,"15min":1.204},"min":{"1min":0.55300000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":7.6890000000000001,"5min":7.6890000000000001,"15min":7.8650000000000002},"last":4.1459999999999999},{"interface":"front","average":{"1min":1.7470000000000001,"5min":1.2410000000000001,"15min":1.26},"min":{"1min":0.52000000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":7.7110000000000003},"last":3.1230000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.0800000000000001,"15min":1.3660000000000001},"min":{"1min":0.52200000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.121,"5min":7.7750000000000004,"15min":34.112000000000002},"last":4.2279999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.383,"15min":1.5660000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.3170000000000002,"5min":7.827,"15min":34.590000000000003},"last":3.0019999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:29:57 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.4530000000000001,"15min":1.4330000000000001},"min":{"1min":0.433,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.4129999999999998,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":2.8450000000000002},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.101,"15min":1.23},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.9950000000000001,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":3.964}]},{"osd":11,"last update":"Mon Feb 12 16:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.115,"15min":1.409},"min":{"1min":0.58499999999999996,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.1520000000000001,"5min":7.734,"15min":7.734},"last":4.1790000000000003},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.361,"15min":1.4710000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.1829999999999998,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":3.0619999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2206428,"kb_used_data":1896916,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91538724,"statfs":{"total":95995035648,"available":93735653376,"internally_reserved":0,"allocated":1942441984,"data_stored":1942101943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":3.2469999999999999,"15min":3.484},"min":{"1min":0.40699999999999997,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":1.387,"5min":221.96600000000001,"15min":565.34500000000003},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.155,"5min":3.3519999999999999,"15min":3.5790000000000002},"min":{"1min":0.45900000000000002,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":1.4830000000000001,"5min":222.06200000000001,"15min":565.98400000000004},"last":0.74399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.53,"15min":1.141},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.3779999999999999,"5min":40.021999999999998,"15min":40.021999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.599,"15min":1.2150000000000001},"min":{"1min":0.375,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.3700000000000001,"5min":40.225000000000001,"15min":40.225000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:29:22 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.4039999999999999,"15min":1.3340000000000001},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.2650000000000001,"5min":16.300999999999998,"15min":16.300999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.999,"15min":0.81899999999999995},"min":{"1min":0.29699999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.284,"5min":6.5540000000000003,"15min":6.5540000000000003},"last":0.628}]},{"osd":4,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.3640000000000001,"15min":1.4239999999999999},"min":{"1min":0.40600000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.5149999999999999,"5min":41.908000000000001,"15min":75.257000000000005},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.109,"5min":1.736,"15min":1.476},"min":{"1min":0.499,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":3.165,"5min":42.430999999999997,"15min":74.427000000000007},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:30:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.194,"15min":1.034},"min":{"1min":0.41299999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.9980000000000002,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.496},{"interface":"front","average":{"1min":1.02,"5min":1.1659999999999999,"15min":1.1000000000000001},"min":{"1min":0.38700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1629999999999998,"5min":9.5830000000000002,"15min":9.5830000000000002},"last":0.33100000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1100000000000001,"15min":1.0549999999999999},"min":{"1min":0.48199999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.5640000000000001,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":1.337},{"interface":"front","average":{"1min":1.234,"5min":1.355,"15min":1.159},"min":{"1min":0.83699999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.802,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.79400000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:29:35 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.1140000000000001,"15min":1.1799999999999999},"min":{"1min":0.51300000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.3159999999999998,"5min":4.2850000000000001,"15min":5.9509999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.58799999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.98,"5min":6.6159999999999997,"15min":6.6159999999999997},"last":0.58799999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:30:15 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.4810000000000001,"15min":1.3480000000000001},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":2.0209999999999999,"5min":6.2939999999999996,"15min":6.2939999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.3180000000000001,"15min":1.282},"min":{"1min":0.53600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":2.9420000000000002,"5min":9.6649999999999991,"15min":9.6649999999999991},"last":0.81799999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:29:38 2024","interfaces":[{"interface":"back","average":{"1min":4.2859999999999996,"5min":1.7430000000000001,"15min":1.4550000000000001},"min":{"1min":0.91100000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":55.518999999999998,"5min":55.518999999999998,"15min":55.518999999999998},"last":0.86199999999999999},{"interface":"front","average":{"1min":4.218,"5min":2.105,"15min":1.5529999999999999},"min":{"1min":0.76300000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":55.497999999999998,"5min":55.497999999999998,"15min":55.497999999999998},"last":0.73199999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:29:45 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.321,"15min":1.3360000000000001},"min":{"1min":0.55800000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.5529999999999999,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.377,"5min":1.518,"15min":1.4119999999999999},"min":{"1min":0.63200000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.8149999999999999,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1013121024,"data_stored":1013121024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1458962432,"data_stored":1458962432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":486408192,"data_stored":486408192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3027107840,"data_stored":3027107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1492975616,"data_stored":1492975616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1948385280,"data_stored":1948385280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2021982208,"data_stored":2021982208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1977352192,"data_stored":1977352192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1000931328,"data_stored":1000931328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":979632128,"data_stored":979632128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":497942528,"data_stored":497942528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:30:28.970 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T16:30:28.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set device_health_metrics pg_num 31 2024-02-12T16:30:28.971 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:28.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:29.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:29.115 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:29.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:29.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:29.245 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:29.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:29.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:29 smithi121 bash[18489]: cluster 2024-02-12T16:30:28.031277+0000 mgr.y (mgr.14146) 1373 : cluster [DBG] pgmap v1913: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-12T16:30:29.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:29 smithi121 bash[18489]: cluster 2024-02-12T16:30:28.193758+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e659: 12 total, 12 up, 12 in 2024-02-12T16:30:29.283 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:30:29.313 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:30:29.348 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:29.352 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:30:29.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:29 smithi123 bash[19119]: cluster 2024-02-12T16:30:28.031277+0000 mgr.y (mgr.14146) 1373 : cluster [DBG] pgmap v1913: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-12T16:30:29.355 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:29 smithi123 bash[19119]: cluster 2024-02-12T16:30:28.193758+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e659: 12 total, 12 up, 12 in 2024-02-12T16:30:29.373 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:29.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:29.383 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:30:29.423 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:30:29.455 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:30:29.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:29.486 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:30:29.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:29.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:29.525 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:30:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:29 smithi169 bash[19604]: cluster 2024-02-12T16:30:28.031277+0000 mgr.y (mgr.14146) 1373 : cluster [DBG] pgmap v1913: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-02-12T16:30:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:29 smithi169 bash[19604]: cluster 2024-02-12T16:30:28.193758+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e659: 12 total, 12 up, 12 in 2024-02-12T16:30:29.556 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:30:29.568 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 128625 128609 131.76 76.375 0.00539949 0.00758771 2024-02-12T16:30:29.583 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:30:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:30:29.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:29.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:29.636 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:30:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:29.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:30:29.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:29.767 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:29.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:29.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:29.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:29.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:30.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:30.038 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:30.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:30.166 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:30.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:30 smithi169 bash[19604]: cluster 2024-02-12T16:30:28.912951+0000 osd.10 (osd.10) 132 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:30:30.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:30 smithi169 bash[19604]: cluster 2024-02-12T16:30:28.981308+0000 osd.10 (osd.10) 133 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:30:30.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:30 smithi169 bash[19604]: audit 2024-02-12T16:30:29.239561+0000 mon.b (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.121:0/1349760565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:30.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:30 smithi169 bash[19604]: cluster 2024-02-12T16:30:29.693694+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:30.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:30 smithi169 bash[19604]: cluster 2024-02-12T16:30:29.693747+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:30.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:30 smithi169 bash[19604]: audit 2024-02-12T16:30:29.706808+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.121:0/2796971872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:30:30.210 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:30:30.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:30.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:30 smithi123 bash[19119]: cluster 2024-02-12T16:30:28.912951+0000 osd.10 (osd.10) 132 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:30:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:30 smithi123 bash[19119]: cluster 2024-02-12T16:30:28.981308+0000 osd.10 (osd.10) 133 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:30:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:30 smithi123 bash[19119]: audit 2024-02-12T16:30:29.239561+0000 mon.b (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.121:0/1349760565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:30 smithi123 bash[19119]: cluster 2024-02-12T16:30:29.693694+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:30 smithi123 bash[19119]: cluster 2024-02-12T16:30:29.693747+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:30.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:30 smithi123 bash[19119]: audit 2024-02-12T16:30:29.706808+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.121:0/2796971872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:30:30.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:30.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:30.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:30.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 129952 129936 130.972 82.9375 0.0174103 0.0076337 2024-02-12T16:30:30.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:30.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:30 smithi121 bash[18489]: cluster 2024-02-12T16:30:28.912951+0000 osd.10 (osd.10) 132 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:30:30.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:30 smithi121 bash[18489]: cluster 2024-02-12T16:30:28.981308+0000 osd.10 (osd.10) 133 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:30:30.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:30 smithi121 bash[18489]: audit 2024-02-12T16:30:29.239561+0000 mon.b (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.121:0/1349760565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:30.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:30 smithi121 bash[18489]: cluster 2024-02-12T16:30:29.693694+0000 osd.5 (osd.5) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:30 smithi121 bash[18489]: cluster 2024-02-12T16:30:29.693747+0000 osd.5 (osd.5) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:30 smithi121 bash[18489]: audit 2024-02-12T16:30:29.706808+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? v1:172.21.15.121:0/2796971872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:30:30.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:30.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:30.825 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:30.867 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:30.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:30.994 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:30.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:31.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:31.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:31.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:31.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:31.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:31.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:31.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:31.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:31.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:31 smithi121 bash[18489]: cluster 2024-02-12T16:30:30.031638+0000 mgr.y (mgr.14146) 1374 : cluster [DBG] pgmap v1915: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:30:31.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:31 smithi121 bash[18489]: cluster 2024-02-12T16:30:30.189953+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:30:31.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:31 smithi121 bash[18489]: cluster 2024-02-12T16:30:30.189992+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-02-12T16:30:31.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:31 smithi121 bash[18489]: audit 2024-02-12T16:30:30.201711+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.121:0/2796971872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:30:31.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:31 smithi121 bash[18489]: cluster 2024-02-12T16:30:30.201762+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in 2024-02-12T16:30:31.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:31 smithi121 bash[18489]: cluster 2024-02-12T16:30:30.744320+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:30:31.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:31.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:31.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:31.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:31.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:31 smithi169 bash[19604]: cluster 2024-02-12T16:30:30.031638+0000 mgr.y (mgr.14146) 1374 : cluster [DBG] pgmap v1915: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:30:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:31 smithi169 bash[19604]: cluster 2024-02-12T16:30:30.189953+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:30:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:31 smithi169 bash[19604]: cluster 2024-02-12T16:30:30.189992+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-02-12T16:30:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:31 smithi169 bash[19604]: audit 2024-02-12T16:30:30.201711+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.121:0/2796971872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:30:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:31 smithi169 bash[19604]: cluster 2024-02-12T16:30:30.201762+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in 2024-02-12T16:30:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:31 smithi169 bash[19604]: cluster 2024-02-12T16:30:30.744320+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:30:31.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 16 131184 131168 130.116 77 0.0231178 0.00768422 2024-02-12T16:30:31.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:31.632 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:31.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:31.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:31 smithi123 bash[19119]: cluster 2024-02-12T16:30:30.031638+0000 mgr.y (mgr.14146) 1374 : cluster [DBG] pgmap v1915: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 7.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:30:31.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:31 smithi123 bash[19119]: cluster 2024-02-12T16:30:30.189953+0000 mon.a (mon.0) 2305 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:30:31.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:31 smithi123 bash[19119]: cluster 2024-02-12T16:30:30.189992+0000 mon.a (mon.0) 2306 : cluster [INF] Cluster is now healthy 2024-02-12T16:30:31.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:31 smithi123 bash[19119]: audit 2024-02-12T16:30:30.201711+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.121:0/2796971872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:30:31.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:31 smithi123 bash[19119]: cluster 2024-02-12T16:30:30.201762+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e660: 12 total, 12 up, 12 in 2024-02-12T16:30:31.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:31 smithi123 bash[19119]: cluster 2024-02-12T16:30:30.744320+0000 osd.5 (osd.5) 117 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:30:31.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:31.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:31.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:31.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:31.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:31.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:32.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.040 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:32.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:32.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:32.170 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:32.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:32.284 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:32 smithi123 bash[19119]: cluster 2024-02-12T16:30:31.373326+0000 osd.7 (osd.7) 166 : cluster [DBG] 6.8 deep-scrub starts 2024-02-12T16:30:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:32.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:32 smithi121 bash[18489]: cluster 2024-02-12T16:30:31.373326+0000 osd.7 (osd.7) 166 : cluster [DBG] 6.8 deep-scrub starts 2024-02-12T16:30:32.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:32.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:32.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:32 smithi169 bash[19604]: cluster 2024-02-12T16:30:31.373326+0000 osd.7 (osd.7) 166 : cluster [DBG] 6.8 deep-scrub starts 2024-02-12T16:30:32.565 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:32.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 132279 132263 129.152 68.4375 0.0179409 0.00774059 2024-02-12T16:30:32.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:32.692 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:32.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:32.830 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:32.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:32.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:32.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:33.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:33.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:33.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:33.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:33.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:30:33.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:33.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:33.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:33.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:33.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:33.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:33.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:33 smithi121 bash[18489]: cluster 2024-02-12T16:30:32.032047+0000 mgr.y (mgr.14146) 1375 : cluster [DBG] pgmap v1917: 37 pgs: 37 active+clean; 7.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-12T16:30:33.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:33.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:33.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:30:33.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:33 smithi169 bash[19604]: cluster 2024-02-12T16:30:32.032047+0000 mgr.y (mgr.14146) 1375 : cluster [DBG] pgmap v1917: 37 pgs: 37 active+clean; 7.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-12T16:30:33.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 133347 133331 128.192 66.75 0.00999668 0.00779934 2024-02-12T16:30:33.575 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:33.582 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:33 smithi123 bash[19119]: cluster 2024-02-12T16:30:32.032047+0000 mgr.y (mgr.14146) 1375 : cluster [DBG] pgmap v1917: 37 pgs: 37 active+clean; 7.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-02-12T16:30:33.618 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:33.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:33.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:33.746 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:33.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:33.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:33.877 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:33.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:33.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:34.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:34.162 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:34.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:34.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:34 smithi123 bash[19119]: cluster 2024-02-12T16:30:32.444531+0000 osd.6 (osd.6) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:34.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:34 smithi123 bash[19119]: cluster 2024-02-12T16:30:32.444576+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:34.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:34 smithi123 bash[19119]: cluster 2024-02-12T16:30:33.214070+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e661: 12 total, 12 up, 12 in 2024-02-12T16:30:34.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:34 smithi123 bash[19119]: cluster 2024-02-12T16:30:33.673558+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:30:34.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:34 smithi123 bash[19119]: cluster 2024-02-12T16:30:33.785634+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d scrub starts 2024-02-12T16:30:34.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:34.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:34.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:34.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:34 smithi169 bash[19604]: cluster 2024-02-12T16:30:32.444531+0000 osd.6 (osd.6) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:34 smithi169 bash[19604]: cluster 2024-02-12T16:30:32.444576+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:34 smithi169 bash[19604]: cluster 2024-02-12T16:30:33.214070+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e661: 12 total, 12 up, 12 in 2024-02-12T16:30:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:34 smithi169 bash[19604]: cluster 2024-02-12T16:30:33.673558+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:30:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:34 smithi169 bash[19604]: cluster 2024-02-12T16:30:33.785634+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d scrub starts 2024-02-12T16:30:34.568 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:34.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 134472 134456 127.315 70.3125 0.00675981 0.00785289 2024-02-12T16:30:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:34.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:34 smithi121 bash[18489]: cluster 2024-02-12T16:30:32.444531+0000 osd.6 (osd.6) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:34.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:34 smithi121 bash[18489]: cluster 2024-02-12T16:30:32.444576+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:34 smithi121 bash[18489]: cluster 2024-02-12T16:30:33.214070+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e661: 12 total, 12 up, 12 in 2024-02-12T16:30:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:34 smithi121 bash[18489]: cluster 2024-02-12T16:30:33.673558+0000 osd.5 (osd.5) 118 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:30:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:34 smithi121 bash[18489]: cluster 2024-02-12T16:30:33.785634+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d scrub starts 2024-02-12T16:30:34.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:34.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:34.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:34.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:34.964 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:34.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:35.006 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:35.009 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:35.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:35.095 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:35.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:35.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:35.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:35.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:35.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:35.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:35.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:35.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:35 smithi121 bash[18489]: cluster 2024-02-12T16:30:34.032497+0000 mgr.y (mgr.14146) 1376 : cluster [DBG] pgmap v1919: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 35 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-12T16:30:35.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:35.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:35 smithi169 bash[19604]: cluster 2024-02-12T16:30:34.032497+0000 mgr.y (mgr.14146) 1376 : cluster [DBG] pgmap v1919: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 35 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-12T16:30:35.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:35.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:35.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 135472 135456 126.347 62.5 0.00827779 0.00791277 2024-02-12T16:30:35.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:35.634 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:35.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:35 smithi123 bash[19119]: cluster 2024-02-12T16:30:34.032497+0000 mgr.y (mgr.14146) 1376 : cluster [DBG] pgmap v1919: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 35 active+clean; 8.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-12T16:30:35.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:35.761 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:35.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:35.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:35.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:35.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:35.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:36.016 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:36.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:36.147 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:36.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:36.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:36.387 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:36 smithi169 bash[19604]: cluster 2024-02-12T16:30:34.545159+0000 osd.2 (osd.2) 94 : cluster [DBG] 6.f deep-scrub starts 2024-02-12T16:30:36.388 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:36 smithi169 bash[19604]: cluster 2024-02-12T16:30:36.029190+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d scrub ok 2024-02-12T16:30:36.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:36 smithi123 bash[19119]: cluster 2024-02-12T16:30:34.545159+0000 osd.2 (osd.2) 94 : cluster [DBG] 6.f deep-scrub starts 2024-02-12T16:30:36.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:36 smithi123 bash[19119]: cluster 2024-02-12T16:30:36.029190+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d scrub ok 2024-02-12T16:30:36.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:36.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:36.537 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:36.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 16 136483 136467 125.418 63.1875 0.00539899 0.00796888 2024-02-12T16:30:36.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:36.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:36 smithi121 bash[18489]: cluster 2024-02-12T16:30:34.545159+0000 osd.2 (osd.2) 94 : cluster [DBG] 6.f deep-scrub starts 2024-02-12T16:30:36.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:36 smithi121 bash[18489]: cluster 2024-02-12T16:30:36.029190+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d scrub ok 2024-02-12T16:30:36.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:36.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:36.815 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:36.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:36.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:36.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:37.033 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:37.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:37.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:37.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:37.211 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:37.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:37.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:37 smithi123 bash[19119]: cluster 2024-02-12T16:30:36.032938+0000 mgr.y (mgr.14146) 1377 : cluster [DBG] pgmap v1920: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 35 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-12T16:30:37.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:37 smithi123 bash[19119]: cluster 2024-02-12T16:30:36.474231+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.9 deep-scrub starts 2024-02-12T16:30:37.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:37.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:37 smithi169 bash[19604]: cluster 2024-02-12T16:30:36.032938+0000 mgr.y (mgr.14146) 1377 : cluster [DBG] pgmap v1920: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 35 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-12T16:30:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:37 smithi169 bash[19604]: cluster 2024-02-12T16:30:36.474231+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.9 deep-scrub starts 2024-02-12T16:30:37.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 137219 137203 124.267 46 0.0548217 0.00804504 2024-02-12T16:30:37.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:37.598 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:37.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:37 smithi121 bash[18489]: cluster 2024-02-12T16:30:36.032938+0000 mgr.y (mgr.14146) 1377 : cluster [DBG] pgmap v1920: 37 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 35 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-02-12T16:30:37.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:37 smithi121 bash[18489]: cluster 2024-02-12T16:30:36.474231+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.9 deep-scrub starts 2024-02-12T16:30:37.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:37.724 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:37.830 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:37.859 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:37.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:37.993 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:37.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:38.046 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:38.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:38.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:38.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:38.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:38.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:38.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:38.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:38 smithi123 bash[19119]: cluster 2024-02-12T16:30:36.614506+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:30:38.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:38.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:38.494 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:38.499 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:38 smithi121 bash[18489]: cluster 2024-02-12T16:30:36.614506+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:30:38.536 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:38.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:38 smithi169 bash[19604]: cluster 2024-02-12T16:30:36.614506+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.2 scrub starts 2024-02-12T16:30:38.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 16 137969 137953 123.162 46.875 0.10122 0.00811625 2024-02-12T16:30:38.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:38.678 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:38.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:38.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:38.810 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:38.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:38.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:38.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:38.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:39.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:39.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:39.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:39.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:39.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:39.207 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:39.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:39.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:39.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:39 smithi121 bash[18489]: cluster 2024-02-12T16:30:38.033310+0000 mgr.y (mgr.14146) 1378 : cluster [DBG] pgmap v1921: 37 pgs: 2 active+clean+scrubbing+deep, 35 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-12T16:30:39.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:39.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:39.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:39.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:39.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:39 smithi169 bash[19604]: cluster 2024-02-12T16:30:38.033310+0000 mgr.y (mgr.14146) 1378 : cluster [DBG] pgmap v1921: 37 pgs: 2 active+clean+scrubbing+deep, 35 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-12T16:30:39.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 138728 138712 122.095 47.4375 0.00442828 0.00818328 2024-02-12T16:30:39.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:39.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:39.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:39.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:39.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:39.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:39.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:39.872 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:39.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:39.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:40.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:40.151 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:40.218 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:39 smithi123 bash[19119]: cluster 2024-02-12T16:30:38.033310+0000 mgr.y (mgr.14146) 1378 : cluster [DBG] pgmap v1921: 37 pgs: 2 active+clean+scrubbing+deep, 35 active+clean; 8.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-02-12T16:30:40.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:40.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:40.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:40.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:40.535 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:40 smithi123 bash[19119]: cluster 2024-02-12T16:30:39.183503+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:30:40.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:40 smithi169 bash[19604]: cluster 2024-02-12T16:30:39.183503+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:30:40.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:40.569 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 139037 139021 120.667 19.3125 0.0114148 0.00828488 2024-02-12T16:30:40.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:40 smithi121 bash[18489]: cluster 2024-02-12T16:30:39.183503+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub ok 2024-02-12T16:30:40.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:40.686 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:40.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:40.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:40.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:40.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:40.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:41.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:41.115 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:41.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:41.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:41.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:41 smithi121 bash[18489]: cluster 2024-02-12T16:30:40.033734+0000 mgr.y (mgr.14146) 1379 : cluster [DBG] pgmap v1922: 37 pgs: 2 active+clean+scrubbing+deep, 35 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-12T16:30:41.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:41 smithi121 bash[18489]: audit 2024-02-12T16:30:40.770957+0000 mon.a (mon.0) 2310 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:30:41.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:41 smithi121 bash[18489]: audit 2024-02-12T16:30:41.197982+0000 mon.a (mon.0) 2311 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:41.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:41.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:41.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:41.389 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:41.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:41 smithi123 bash[19119]: cluster 2024-02-12T16:30:40.033734+0000 mgr.y (mgr.14146) 1379 : cluster [DBG] pgmap v1922: 37 pgs: 2 active+clean+scrubbing+deep, 35 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-12T16:30:41.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:41 smithi123 bash[19119]: audit 2024-02-12T16:30:40.770957+0000 mon.a (mon.0) 2310 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:30:41.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:41 smithi123 bash[19119]: audit 2024-02-12T16:30:41.197982+0000 mon.a (mon.0) 2311 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:41.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:41.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:41.524 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:41.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:41 smithi169 bash[19604]: cluster 2024-02-12T16:30:40.033734+0000 mgr.y (mgr.14146) 1379 : cluster [DBG] pgmap v1922: 37 pgs: 2 active+clean+scrubbing+deep, 35 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-02-12T16:30:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:41 smithi169 bash[19604]: audit 2024-02-12T16:30:40.770957+0000 mon.a (mon.0) 2310 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:30:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:41 smithi169 bash[19604]: audit 2024-02-12T16:30:41.197982+0000 mon.a (mon.0) 2311 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:41.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 139860 139844 119.719 51.4375 0.00540669 0.00834968 2024-02-12T16:30:41.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:41.660 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:41.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:41.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:41.800 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:41.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:41.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:41.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:41.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:42.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:42.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:42.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:42.195 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:42.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:42.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:42.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:42.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:42.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:42.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 140574 140558 118.704 44.625 0.00377863 0.00839958 2024-02-12T16:30:42.574 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:42.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:42.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:42.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:42.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:42.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:42.966 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:42.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:43.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:43.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:43.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:43.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:43.226 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:43.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:43.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:43.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:43 smithi121 bash[18489]: cluster 2024-02-12T16:30:42.034117+0000 mgr.y (mgr.14146) 1380 : cluster [DBG] pgmap v1923: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 997 op/s 2024-02-12T16:30:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:43.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:43.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:43.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:43.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:43 smithi169 bash[19604]: cluster 2024-02-12T16:30:42.034117+0000 mgr.y (mgr.14146) 1380 : cluster [DBG] pgmap v1923: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 997 op/s 2024-02-12T16:30:43.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 16 141315 141299 117.739 46.3125 0.00541197 0.0084902 2024-02-12T16:30:43.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:43.601 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:43 smithi123 bash[19119]: cluster 2024-02-12T16:30:42.034117+0000 mgr.y (mgr.14146) 1380 : cluster [DBG] pgmap v1923: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 997 op/s 2024-02-12T16:30:43.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:43.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:43.720 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:43.751 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:43.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:43.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:43.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:43.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:44.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:44.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:44.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:44.166 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:44.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:44.296 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:44.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:44.422 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:44.532 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:44.559 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:44.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 142106 142090 116.84 49.4375 0.00571672 0.00855557 2024-02-12T16:30:44.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:44.689 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:44.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:44.823 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:44.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:44.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:45.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:45.084 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:45.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:45.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:45.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:45.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:45.346 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:45.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:45 smithi123 bash[19119]: cluster 2024-02-12T16:30:44.034562+0000 mgr.y (mgr.14146) 1381 : cluster [DBG] pgmap v1924: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-02-12T16:30:45.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:45.456 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:45 smithi121 bash[18489]: cluster 2024-02-12T16:30:44.034562+0000 mgr.y (mgr.14146) 1381 : cluster [DBG] pgmap v1924: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-02-12T16:30:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:45.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:45 smithi169 bash[19604]: cluster 2024-02-12T16:30:44.034562+0000 mgr.y (mgr.14146) 1381 : cluster [DBG] pgmap v1924: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-02-12T16:30:45.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 16 142885 142869 115.955 48.6875 0.109892 0.00861889 2024-02-12T16:30:45.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:45.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:45.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:45.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:45.757 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:45.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:45.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:45.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:45.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:45.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:46.015 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:46.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:46.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:46.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:46.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:46.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:46.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:46.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 143630 143614 115.065 46.5625 0.0061303 0.0086849 2024-02-12T16:30:46.644 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:30:46.644 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:30:46.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:46.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:46.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:46.826 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:46.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:46.953 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:46.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:47.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:47.079 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:47.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:47.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:47.193 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:47 smithi169 bash[19604]: audit 2024-02-12T16:30:46.014049+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:47.193 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:47 smithi169 bash[19604]: cluster 2024-02-12T16:30:46.034964+0000 mgr.y (mgr.14146) 1382 : cluster [DBG] pgmap v1925: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s 2024-02-12T16:30:47.193 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:47 smithi169 bash[19604]: audit 2024-02-12T16:30:46.348324+0000 mon.a (mon.0) 2313 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:30:47.193 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:47 smithi169 bash[19604]: audit 2024-02-12T16:30:46.349211+0000 mon.a (mon.0) 2314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:30:47.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:47 smithi169 bash[19604]: audit 2024-02-12T16:30:46.354962+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:47.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:47 smithi169 bash[19604]: audit 2024-02-12T16:30:46.870462+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? v1:172.21.15.121:0/2959560121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "31"}]: dispatch 2024-02-12T16:30:47.211 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:47.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:47.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:47.344 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:47.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:47.370 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 1 pg_num to 31 2024-02-12T16:30:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:47 smithi123 bash[19119]: audit 2024-02-12T16:30:46.014049+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:47 smithi123 bash[19119]: cluster 2024-02-12T16:30:46.034964+0000 mgr.y (mgr.14146) 1382 : cluster [DBG] pgmap v1925: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s 2024-02-12T16:30:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:47 smithi123 bash[19119]: audit 2024-02-12T16:30:46.348324+0000 mon.a (mon.0) 2313 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:30:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:47 smithi123 bash[19119]: audit 2024-02-12T16:30:46.349211+0000 mon.a (mon.0) 2314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:30:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:47 smithi123 bash[19119]: audit 2024-02-12T16:30:46.354962+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:47 smithi123 bash[19119]: audit 2024-02-12T16:30:46.870462+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? v1:172.21.15.121:0/2959560121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "31"}]: dispatch 2024-02-12T16:30:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:47 smithi121 bash[18489]: audit 2024-02-12T16:30:46.014049+0000 mon.a (mon.0) 2312 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:47.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:47 smithi121 bash[18489]: cluster 2024-02-12T16:30:46.034964+0000 mgr.y (mgr.14146) 1382 : cluster [DBG] pgmap v1925: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s 2024-02-12T16:30:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:47 smithi121 bash[18489]: audit 2024-02-12T16:30:46.348324+0000 mon.a (mon.0) 2313 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:30:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:47 smithi121 bash[18489]: audit 2024-02-12T16:30:46.349211+0000 mon.a (mon.0) 2314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:30:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:47 smithi121 bash[18489]: audit 2024-02-12T16:30:46.354962+0000 mon.a (mon.0) 2315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:30:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:47 smithi121 bash[18489]: audit 2024-02-12T16:30:46.870462+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? v1:172.21.15.121:0/2959560121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "31"}]: dispatch 2024-02-12T16:30:47.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:47.469 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:47.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:47.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 144403 144387 114.22 48.3125 0.0106717 0.00875225 2024-02-12T16:30:47.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:47.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:47.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:47.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:47.735 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:47.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:47.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:47.865 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:47.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:47.952 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:30:47.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:47.980 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:30:48.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:48.018 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:30:48.050 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:30:48.074 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:30:48.103 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:30:48.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:48.131 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:30:48.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:48.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:48.161 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:30:48.191 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:30:48.220 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:30:48.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:48.255 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:30:48.275 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:48.290 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:30:48.295 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:30:48.372 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:30:48.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:48.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:48.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:48 smithi123 bash[19119]: audit 2024-02-12T16:30:47.356930+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? v1:172.21.15.121:0/2959560121' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "31"}]': finished 2024-02-12T16:30:48.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:48 smithi123 bash[19119]: cluster 2024-02-12T16:30:47.357001+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e662: 12 total, 12 up, 12 in 2024-02-12T16:30:48.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:48 smithi123 bash[19119]: audit 2024-02-12T16:30:47.542217+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? v1:172.21.15.121:0/3685885647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:30:48.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:48 smithi123 bash[19119]: audit 2024-02-12T16:30:47.917906+0000 mon.a (mon.0) 2320 : audit [DBG] from='client.? v1:172.21.15.121:0/1008538068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:48.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:48 smithi123 bash[19119]: audit 2024-02-12T16:30:48.036302+0000 mon.a (mon.0) 2321 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:30:48.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:30:48.567768 min lat: 0.00101494 max lat: 0.80513 avg lat: 0.00881537 2024-02-12T16:30:48.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:30:48.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 145170 145154 113.392 47.9375 0.0126151 0.00881537 2024-02-12T16:30:48.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:48.634 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:48 smithi121 bash[18489]: audit 2024-02-12T16:30:47.356930+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? v1:172.21.15.121:0/2959560121' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "31"}]': finished 2024-02-12T16:30:48.634 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:48 smithi121 bash[18489]: cluster 2024-02-12T16:30:47.357001+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e662: 12 total, 12 up, 12 in 2024-02-12T16:30:48.634 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:48 smithi121 bash[18489]: audit 2024-02-12T16:30:47.542217+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? v1:172.21.15.121:0/3685885647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:30:48.634 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:48 smithi121 bash[18489]: audit 2024-02-12T16:30:47.917906+0000 mon.a (mon.0) 2320 : audit [DBG] from='client.? v1:172.21.15.121:0/1008538068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:48.634 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:48 smithi121 bash[18489]: audit 2024-02-12T16:30:48.036302+0000 mon.a (mon.0) 2321 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:30:48.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:30:48.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:48.693 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:48 smithi169 bash[19604]: audit 2024-02-12T16:30:47.356930+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? v1:172.21.15.121:0/2959560121' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "31"}]': finished 2024-02-12T16:30:48.693 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:48 smithi169 bash[19604]: cluster 2024-02-12T16:30:47.357001+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e662: 12 total, 12 up, 12 in 2024-02-12T16:30:48.693 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:48 smithi169 bash[19604]: audit 2024-02-12T16:30:47.542217+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? v1:172.21.15.121:0/3685885647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:30:48.693 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:48 smithi169 bash[19604]: audit 2024-02-12T16:30:47.917906+0000 mon.a (mon.0) 2320 : audit [DBG] from='client.? v1:172.21.15.121:0/1008538068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:30:48.693 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:48 smithi169 bash[19604]: audit 2024-02-12T16:30:48.036302+0000 mon.a (mon.0) 2321 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-12T16:30:48.708 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:48.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:48.835 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:48.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:48.964 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:48.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:49.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:49.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:49.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:49.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:49.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:49.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:49.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:49.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:49.460 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:49.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:49.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:49.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 145961 145945 112.602 49.4375 0.091026 0.00887802 2024-02-12T16:30:49.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:49.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:49 smithi123 bash[19119]: cluster 2024-02-12T16:30:48.035368+0000 mgr.y (mgr.14146) 1383 : cluster [DBG] pgmap v1927: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-02-12T16:30:49.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:49 smithi123 bash[19119]: cluster 2024-02-12T16:30:48.354289+0000 mon.a (mon.0) 2322 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:30:49.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:49 smithi123 bash[19119]: audit 2024-02-12T16:30:48.363378+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.121:0/3685885647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:30:49.596 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:49 smithi123 bash[19119]: audit 2024-02-12T16:30:48.363416+0000 mon.a (mon.0) 2324 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T16:30:49.597 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:49 smithi123 bash[19119]: cluster 2024-02-12T16:30:48.363445+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e663: 12 total, 12 up, 12 in 2024-02-12T16:30:49.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:49.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:49.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:49 smithi121 bash[18489]: cluster 2024-02-12T16:30:48.035368+0000 mgr.y (mgr.14146) 1383 : cluster [DBG] pgmap v1927: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-02-12T16:30:49.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:49 smithi121 bash[18489]: cluster 2024-02-12T16:30:48.354289+0000 mon.a (mon.0) 2322 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:30:49.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:49 smithi121 bash[18489]: audit 2024-02-12T16:30:48.363378+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.121:0/3685885647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:30:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:49 smithi121 bash[18489]: audit 2024-02-12T16:30:48.363416+0000 mon.a (mon.0) 2324 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T16:30:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:49 smithi121 bash[18489]: cluster 2024-02-12T16:30:48.363445+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e663: 12 total, 12 up, 12 in 2024-02-12T16:30:49.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:49.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:49 smithi169 bash[19604]: cluster 2024-02-12T16:30:48.035368+0000 mgr.y (mgr.14146) 1383 : cluster [DBG] pgmap v1927: 37 pgs: 3 active+clean+scrubbing+deep, 34 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s 2024-02-12T16:30:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:49 smithi169 bash[19604]: cluster 2024-02-12T16:30:48.354289+0000 mon.a (mon.0) 2322 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:30:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:49 smithi169 bash[19604]: audit 2024-02-12T16:30:48.363378+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.121:0/3685885647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:30:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:49 smithi169 bash[19604]: audit 2024-02-12T16:30:48.363416+0000 mon.a (mon.0) 2324 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-12T16:30:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:49 smithi169 bash[19604]: cluster 2024-02-12T16:30:48.363445+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e663: 12 total, 12 up, 12 in 2024-02-12T16:30:49.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:49.873 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:49.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:49.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:50.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:50.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:50.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:50.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:50.271 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:50.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:50.376 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:50 smithi121 bash[18489]: cluster 2024-02-12T16:30:49.372509+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e664: 12 total, 12 up, 12 in 2024-02-12T16:30:50.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:50 smithi123 bash[19119]: cluster 2024-02-12T16:30:49.372509+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e664: 12 total, 12 up, 12 in 2024-02-12T16:30:50.403 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:50.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:50.531 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:50.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 146830 146814 111.891 54.3125 0.0408697 0.00893299 2024-02-12T16:30:50.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:50.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:50.786 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:50 smithi169 bash[19604]: cluster 2024-02-12T16:30:49.372509+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e664: 12 total, 12 up, 12 in 2024-02-12T16:30:50.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:50.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:50.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:51.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:51.052 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:51.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:51.181 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:51.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:51.314 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:51.418 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:51.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:51.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:51 smithi169 bash[19604]: cluster 2024-02-12T16:30:50.035840+0000 mgr.y (mgr.14146) 1384 : cluster [DBG] pgmap v1930: 47 pgs: 4 peering, 8 unknown, 3 active+clean+scrubbing+deep, 32 active+clean; 8.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 707 op/s 2024-02-12T16:30:51.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:51.571 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 147584 147568 111.111 47.125 0.00574863 0.00899589 2024-02-12T16:30:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:51 smithi123 bash[19119]: cluster 2024-02-12T16:30:50.035840+0000 mgr.y (mgr.14146) 1384 : cluster [DBG] pgmap v1930: 47 pgs: 4 peering, 8 unknown, 3 active+clean+scrubbing+deep, 32 active+clean; 8.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 707 op/s 2024-02-12T16:30:51.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:51 smithi121 bash[18489]: cluster 2024-02-12T16:30:50.035840+0000 mgr.y (mgr.14146) 1384 : cluster [DBG] pgmap v1930: 47 pgs: 4 peering, 8 unknown, 3 active+clean+scrubbing+deep, 32 active+clean; 8.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 707 op/s 2024-02-12T16:30:51.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:51.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:51.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:51.845 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:51.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:51.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:51.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:52.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:30:52.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:52.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:30:52.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:52.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:52.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:52.238 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:52.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:52.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:52.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:52.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:52.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:52.495 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:52.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:52.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 148330 148314 110.343 46.625 0.014514 0.00905897 2024-02-12T16:30:52.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:52.624 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:52.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:52.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:52.753 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:52.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:52.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:52.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:52.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:52.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:53.010 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:53.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:53.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:53.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:53.272 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:53.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:53.404 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:53.484 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:53.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:53.518 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:53 smithi121 bash[18489]: cluster 2024-02-12T16:30:52.036292+0000 mgr.y (mgr.14146) 1385 : cluster [DBG] pgmap v1931: 47 pgs: 4 peering, 5 unknown, 3 active+clean+scrubbing+deep, 35 active+clean; 8.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 715 op/s 2024-02-12T16:30:53.550 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:53.570 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 149116 149100 109.623 49.125 0.010666 0.0091193 2024-02-12T16:30:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:53 smithi123 bash[19119]: cluster 2024-02-12T16:30:52.036292+0000 mgr.y (mgr.14146) 1385 : cluster [DBG] pgmap v1931: 47 pgs: 4 peering, 5 unknown, 3 active+clean+scrubbing+deep, 35 active+clean; 8.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 715 op/s 2024-02-12T16:30:53.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:53.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:53.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:53 smithi169 bash[19604]: cluster 2024-02-12T16:30:52.036292+0000 mgr.y (mgr.14146) 1385 : cluster [DBG] pgmap v1931: 47 pgs: 4 peering, 5 unknown, 3 active+clean+scrubbing+deep, 35 active+clean; 8.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 715 op/s 2024-02-12T16:30:53.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:53.875 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-02-12T16:30:53.875 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:30:53.875 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-02-12T16:30:53.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:30:53.933 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:53.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:53.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:54.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:54.084 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:54.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:54.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:54.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:54.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:54.385 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:54 smithi169 bash[19604]: audit 2024-02-12T16:30:54.037721+0000 mon.a (mon.0) 2327 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-12T16:30:54.386 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:54 smithi123 bash[19119]: audit 2024-02-12T16:30:54.037721+0000 mon.a (mon.0) 2327 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-12T16:30:54.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:54.470 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:54.571 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 149866 149850 108.893 46.875 0.00565986 0.00917735 2024-02-12T16:30:54.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:54.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:54 smithi121 bash[18489]: audit 2024-02-12T16:30:54.037721+0000 mon.a (mon.0) 2327 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-12T16:30:54.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:54.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:54.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:54.836 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:54.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:54.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:54.995 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:54.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:55.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:55.128 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:55.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:55.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:55.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:55.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:55.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:55.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:55.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:55.553 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:55.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:55.571 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 150592 150576 108.163 45.375 0.00551167 0.00924007 2024-02-12T16:30:55.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:55 smithi123 bash[19119]: cluster 2024-02-12T16:30:54.036802+0000 mgr.y (mgr.14146) 1386 : cluster [DBG] pgmap v1932: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s 2024-02-12T16:30:55.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:55 smithi123 bash[19119]: audit 2024-02-12T16:30:54.386676+0000 mon.a (mon.0) 2328 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-12T16:30:55.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:55 smithi123 bash[19119]: cluster 2024-02-12T16:30:54.386738+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e665: 12 total, 12 up, 12 in 2024-02-12T16:30:55.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:55 smithi123 bash[19119]: cluster 2024-02-12T16:30:55.009555+0000 osd.8 (osd.8) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:55.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:55 smithi123 bash[19119]: cluster 2024-02-12T16:30:55.009632+0000 osd.8 (osd.8) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:55.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:55.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:55 smithi121 bash[18489]: cluster 2024-02-12T16:30:54.036802+0000 mgr.y (mgr.14146) 1386 : cluster [DBG] pgmap v1932: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s 2024-02-12T16:30:55.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:55 smithi121 bash[18489]: audit 2024-02-12T16:30:54.386676+0000 mon.a (mon.0) 2328 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-12T16:30:55.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:55 smithi121 bash[18489]: cluster 2024-02-12T16:30:54.386738+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e665: 12 total, 12 up, 12 in 2024-02-12T16:30:55.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:55 smithi121 bash[18489]: cluster 2024-02-12T16:30:55.009555+0000 osd.8 (osd.8) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:55.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:55 smithi121 bash[18489]: cluster 2024-02-12T16:30:55.009632+0000 osd.8 (osd.8) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:55.666 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:55 smithi169 bash[19604]: cluster 2024-02-12T16:30:54.036802+0000 mgr.y (mgr.14146) 1386 : cluster [DBG] pgmap v1932: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s 2024-02-12T16:30:55.666 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:55 smithi169 bash[19604]: audit 2024-02-12T16:30:54.386676+0000 mon.a (mon.0) 2328 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-12T16:30:55.666 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:55 smithi169 bash[19604]: cluster 2024-02-12T16:30:54.386738+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e665: 12 total, 12 up, 12 in 2024-02-12T16:30:55.666 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:55 smithi169 bash[19604]: cluster 2024-02-12T16:30:55.009555+0000 osd.8 (osd.8) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:55.666 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:55 smithi169 bash[19604]: cluster 2024-02-12T16:30:55.009632+0000 osd.8 (osd.8) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:55.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:55.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:55.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:55.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:55.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:55.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:55.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:56.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:56.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:56.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:56.202 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:56.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:56.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:56.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:56.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:56.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:56.571 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 151279 151263 107.422 42.9375 0.00583411 0.00930529 2024-02-12T16:30:56.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:56 smithi123 bash[19119]: cluster 2024-02-12T16:30:54.833741+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:56 smithi123 bash[19119]: cluster 2024-02-12T16:30:54.833816+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:56 smithi123 bash[19119]: cluster 2024-02-12T16:30:55.388564+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e666: 12 total, 12 up, 12 in 2024-02-12T16:30:56.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:56 smithi121 bash[18489]: cluster 2024-02-12T16:30:54.833741+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:56.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:56 smithi121 bash[18489]: cluster 2024-02-12T16:30:54.833816+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:56.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:56 smithi121 bash[18489]: cluster 2024-02-12T16:30:55.388564+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e666: 12 total, 12 up, 12 in 2024-02-12T16:30:56.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:56.724 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:56 smithi169 bash[19604]: cluster 2024-02-12T16:30:54.833741+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:30:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:56 smithi169 bash[19604]: cluster 2024-02-12T16:30:54.833816+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:30:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:56 smithi169 bash[19604]: cluster 2024-02-12T16:30:55.388564+0000 mon.a (mon.0) 2330 : cluster [DBG] osdmap e666: 12 total, 12 up, 12 in 2024-02-12T16:30:56.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:56.856 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:56.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:30:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:56.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:30:57.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:57.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:57.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:57.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:57.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:57.259 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:57.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:57.316 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:57.321 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:57.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:57.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:57.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:57.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:57.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:57.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:57 smithi169 bash[19604]: cluster 2024-02-12T16:30:56.037199+0000 mgr.y (mgr.14146) 1387 : cluster [DBG] pgmap v1935: 47 pgs: 2 peering, 3 active+clean+scrubbing+deep, 42 active+clean; 9.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 838 op/s 2024-02-12T16:30:57.571 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 152071 152055 106.771 49.5 0.00955847 0.00936086 2024-02-12T16:30:57.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:57.645 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:30:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:57.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:30:57.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:57 smithi121 bash[18489]: cluster 2024-02-12T16:30:56.037199+0000 mgr.y (mgr.14146) 1387 : cluster [DBG] pgmap v1935: 47 pgs: 2 peering, 3 active+clean+scrubbing+deep, 42 active+clean; 9.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 838 op/s 2024-02-12T16:30:57.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:57.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:57.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:57.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:57 smithi123 bash[19119]: cluster 2024-02-12T16:30:56.037199+0000 mgr.y (mgr.14146) 1387 : cluster [DBG] pgmap v1935: 47 pgs: 2 peering, 3 active+clean+scrubbing+deep, 42 active+clean; 9.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 838 op/s 2024-02-12T16:30:57.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:57.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:58.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:58.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:30:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:30:58.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:58.158 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:58.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:58.285 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:30:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:30:58.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:58.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:30:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:30:58.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:58.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:58.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:30:58.571 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 90 15 152833 152818 106.114 47.6875 0.0069758 0.00942086 2024-02-12T16:30:58.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:58.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:30:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:30:58.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:58.809 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:58.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:30:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:58.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:30:59.036 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:59.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:30:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:30:59.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:59.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:59.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:59.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:59.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:30:59 smithi121 bash[18489]: cluster 2024-02-12T16:30:58.037665+0000 mgr.y (mgr.14146) 1388 : cluster [DBG] pgmap v1936: 47 pgs: 2 peering, 3 active+clean+scrubbing+deep, 42 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 786 op/s 2024-02-12T16:30:59.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:30:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:30:59.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:30:59.564 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:30:59 smithi123 bash[19119]: cluster 2024-02-12T16:30:58.037665+0000 mgr.y (mgr.14146) 1388 : cluster [DBG] pgmap v1936: 47 pgs: 2 peering, 3 active+clean+scrubbing+deep, 42 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 786 op/s 2024-02-12T16:30:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:30:59.572 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.043542 2024-02-12T16:30:59.572 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 152833 2024-02-12T16:30:59.572 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:30:59.572 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:30:59.572 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 106.083 2024-02-12T16:30:59.572 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 63.1144 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 359.375 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 19.3125 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 1697 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 1009 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 5750 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 309 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.00942492 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.0152446 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 0.80513 2024-02-12T16:30:59.573 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.00101494 2024-02-12T16:30:59.580 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:30:59.616 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:30:59.616 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:30:59.617 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:30:59.617 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:30:59.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:59.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:30:59 smithi169 bash[19604]: cluster 2024-02-12T16:30:58.037665+0000 mgr.y (mgr.14146) 1388 : cluster [DBG] pgmap v1936: 47 pgs: 2 peering, 3 active+clean+scrubbing+deep, 42 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 786 op/s 2024-02-12T16:30:59.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:59.827 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:30:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:30:59.920 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:30:59.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:59.971 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:30:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:30:59.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:00.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:00.116 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:00.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:00.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:00.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:00.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:00.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:00.380 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:00.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:00.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:00.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:00.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:00.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:00.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:00.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:00.766 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:00.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:00.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:00.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:00.995 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:01.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:01.126 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:01.149 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:01.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:01.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:01.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:31:01.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:01.407 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:01.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:01.517 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:01 smithi169 bash[19604]: cluster 2024-02-12T16:31:00.038051+0000 mgr.y (mgr.14146) 1389 : cluster [DBG] pgmap v1937: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 735 op/s 2024-02-12T16:31:01.534 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:01.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:01.647 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:01 smithi121 bash[18489]: cluster 2024-02-12T16:31:00.038051+0000 mgr.y (mgr.14146) 1389 : cluster [DBG] pgmap v1937: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 735 op/s 2024-02-12T16:31:01.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:01.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:01.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:01 smithi123 bash[19119]: cluster 2024-02-12T16:31:00.038051+0000 mgr.y (mgr.14146) 1389 : cluster [DBG] pgmap v1937: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 735 op/s 2024-02-12T16:31:01.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:01.938 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:01.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:02.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:02.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:02.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:02.207 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:02.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:02.335 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:02.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:02.467 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:02.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:02.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:02.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:02.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:02.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:02.854 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:02.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:02.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:02.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:03.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:03.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:03.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:03.252 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:03.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:03.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:03.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:03.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:03.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:03.491 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:03 smithi121 bash[18489]: cluster 2024-02-12T16:31:02.038438+0000 mgr.y (mgr.14146) 1390 : cluster [DBG] pgmap v1938: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 691 op/s 2024-02-12T16:31:03.491 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:03 smithi121 bash[18489]: cluster 2024-02-12T16:31:02.429089+0000 osd.10 (osd.10) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:03.491 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:03 smithi121 bash[18489]: cluster 2024-02-12T16:31:02.429146+0000 osd.10 (osd.10) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:03.507 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:03.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:03.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:03.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:03 smithi169 bash[19604]: cluster 2024-02-12T16:31:02.038438+0000 mgr.y (mgr.14146) 1390 : cluster [DBG] pgmap v1938: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 691 op/s 2024-02-12T16:31:03.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:03 smithi169 bash[19604]: cluster 2024-02-12T16:31:02.429089+0000 osd.10 (osd.10) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:03.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:03 smithi169 bash[19604]: cluster 2024-02-12T16:31:02.429146+0000 osd.10 (osd.10) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:03.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:03.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:03.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:03.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:03.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:03.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:03 smithi123 bash[19119]: cluster 2024-02-12T16:31:02.038438+0000 mgr.y (mgr.14146) 1390 : cluster [DBG] pgmap v1938: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 691 op/s 2024-02-12T16:31:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:03 smithi123 bash[19119]: cluster 2024-02-12T16:31:02.429089+0000 osd.10 (osd.10) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:03.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:03 smithi123 bash[19119]: cluster 2024-02-12T16:31:02.429146+0000 osd.10 (osd.10) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:03.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:03.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:04.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:04.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:04.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:04.164 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:04.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:04.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:04.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:04.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:04.566 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:04.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:04 smithi123 bash[19119]: cluster 2024-02-12T16:31:03.420496+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e667: 12 total, 12 up, 12 in 2024-02-12T16:31:04.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:04 smithi121 bash[18489]: cluster 2024-02-12T16:31:03.420496+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e667: 12 total, 12 up, 12 in 2024-02-12T16:31:04.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:04.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:04.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:04.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:04 smithi169 bash[19604]: cluster 2024-02-12T16:31:03.420496+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e667: 12 total, 12 up, 12 in 2024-02-12T16:31:04.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:04.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:04.950 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:04.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:05.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:05.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:05.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:05.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:05.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:05.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:05.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:05.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:05.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:05.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:05.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:05.585 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:05 smithi123 bash[19119]: cluster 2024-02-12T16:31:04.038828+0000 mgr.y (mgr.14146) 1391 : cluster [DBG] pgmap v1940: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 118 B/s rd, 29 MiB/s wr, 645 op/s 2024-02-12T16:31:05.601 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:05.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:05.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:05.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:05.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:05.763 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:05 smithi121 bash[18489]: cluster 2024-02-12T16:31:04.038828+0000 mgr.y (mgr.14146) 1391 : cluster [DBG] pgmap v1940: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 118 B/s rd, 29 MiB/s wr, 645 op/s 2024-02-12T16:31:05.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:05 smithi169 bash[19604]: cluster 2024-02-12T16:31:04.038828+0000 mgr.y (mgr.14146) 1391 : cluster [DBG] pgmap v1940: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 118 B/s rd, 29 MiB/s wr, 645 op/s 2024-02-12T16:31:05.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:05.876 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:05.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:05.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:06.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:06.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:06.132 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:06.171 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:06.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:06.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:06.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:06.397 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:06.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:06.527 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:06.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:06.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:06.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:06.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:06.787 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:06 smithi121 bash[18489]: cluster 2024-02-12T16:31:06.039225+0000 mgr.y (mgr.14146) 1392 : cluster [DBG] pgmap v1941: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 22 MiB/s wr, 633 op/s 2024-02-12T16:31:06.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:06.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:06.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:07.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:07.025 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:06 smithi169 bash[19604]: cluster 2024-02-12T16:31:06.039225+0000 mgr.y (mgr.14146) 1392 : cluster [DBG] pgmap v1941: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 22 MiB/s wr, 633 op/s 2024-02-12T16:31:07.043 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:07.076 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:31:07.107 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:31:07.133 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:31:07.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:06 smithi123 bash[19119]: cluster 2024-02-12T16:31:06.039225+0000 mgr.y (mgr.14146) 1392 : cluster [DBG] pgmap v1941: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 22 MiB/s wr, 633 op/s 2024-02-12T16:31:07.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:07.161 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:31:07.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:07.199 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:31:07.232 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:31:07.258 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:31:07.287 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:31:07.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:07.319 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:07.323 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:31:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:07.367 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:31:07.408 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:31:07.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:07.431 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:31:07.435 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.436 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:31:07.454 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:07.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:07.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:07.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:07.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:07.746 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:31:07.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:07.839 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:07.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:07 smithi123 bash[19119]: audit 2024-02-12T16:31:07.045123+0000 mon.a (mon.0) 2332 : audit [DBG] from='client.? v1:172.21.15.121:0/2958713295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:07.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:07 smithi123 bash[19119]: audit 2024-02-12T16:31:07.408030+0000 mon.a (mon.0) 2333 : audit [INF] from='client.26326 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:07.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:07 smithi123 bash[19119]: audit 2024-02-12T16:31:07.408986+0000 mon.c (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.121:0/2608249284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:07 smithi121 bash[18489]: audit 2024-02-12T16:31:07.045123+0000 mon.a (mon.0) 2332 : audit [DBG] from='client.? v1:172.21.15.121:0/2958713295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:07 smithi121 bash[18489]: audit 2024-02-12T16:31:07.408030+0000 mon.a (mon.0) 2333 : audit [INF] from='client.26326 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:07.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:07 smithi121 bash[18489]: audit 2024-02-12T16:31:07.408986+0000 mon.c (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.121:0/2608249284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:07.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:07.949 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:07 smithi169 bash[19604]: audit 2024-02-12T16:31:07.045123+0000 mon.a (mon.0) 2332 : audit [DBG] from='client.? v1:172.21.15.121:0/2958713295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:07.949 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:07 smithi169 bash[19604]: audit 2024-02-12T16:31:07.408030+0000 mon.a (mon.0) 2333 : audit [INF] from='client.26326 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:07.949 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:07 smithi169 bash[19604]: audit 2024-02-12T16:31:07.408986+0000 mon.c (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.121:0/2608249284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:07.970 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:07.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:08.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:08.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:08.200 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:08.231 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:08.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:08.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:08.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:08.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:08.471 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:08.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:08.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:08.621 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:08.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:08.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:08.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:08.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:08 smithi123 bash[19119]: cluster 2024-02-12T16:31:07.733676+0000 mon.a (mon.0) 2334 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:31:08.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:08 smithi123 bash[19119]: audit 2024-02-12T16:31:07.737100+0000 mon.a (mon.0) 2335 : audit [INF] from='client.26326 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:31:08.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:08 smithi123 bash[19119]: cluster 2024-02-12T16:31:07.737168+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e668: 12 total, 12 up, 12 in 2024-02-12T16:31:08.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:08 smithi123 bash[19119]: cluster 2024-02-12T16:31:08.039590+0000 mgr.y (mgr.14146) 1393 : cluster [DBG] pgmap v1943: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 8.7 MiB/s wr, 688 op/s 2024-02-12T16:31:08.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:08.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:08.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:08.907 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:08.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:09.009 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:09.033 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:09.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:08 smithi169 bash[19604]: cluster 2024-02-12T16:31:07.733676+0000 mon.a (mon.0) 2334 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:31:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:08 smithi169 bash[19604]: audit 2024-02-12T16:31:07.737100+0000 mon.a (mon.0) 2335 : audit [INF] from='client.26326 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:31:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:08 smithi169 bash[19604]: cluster 2024-02-12T16:31:07.737168+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e668: 12 total, 12 up, 12 in 2024-02-12T16:31:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:08 smithi169 bash[19604]: cluster 2024-02-12T16:31:08.039590+0000 mgr.y (mgr.14146) 1393 : cluster [DBG] pgmap v1943: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 8.7 MiB/s wr, 688 op/s 2024-02-12T16:31:09.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:09.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:08 smithi121 bash[18489]: cluster 2024-02-12T16:31:07.733676+0000 mon.a (mon.0) 2334 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:31:09.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:08 smithi121 bash[18489]: audit 2024-02-12T16:31:07.737100+0000 mon.a (mon.0) 2335 : audit [INF] from='client.26326 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:31:09.146 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:08 smithi121 bash[18489]: cluster 2024-02-12T16:31:07.737168+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e668: 12 total, 12 up, 12 in 2024-02-12T16:31:09.146 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:08 smithi121 bash[18489]: cluster 2024-02-12T16:31:08.039590+0000 mgr.y (mgr.14146) 1393 : cluster [DBG] pgmap v1943: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 9.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 8.7 MiB/s wr, 688 op/s 2024-02-12T16:31:09.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:09.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:09.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:09.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:09.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:09.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:09.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:09.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:09.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:09.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:09.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:09.709 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:09.741 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:09.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:09.844 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:09.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:09.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:09.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:10.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:10.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:09 smithi121 bash[18489]: cluster 2024-02-12T16:31:08.379107+0000 osd.2 (osd.2) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:10.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:09 smithi121 bash[18489]: cluster 2024-02-12T16:31:08.379150+0000 osd.2 (osd.2) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:10.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:09 smithi121 bash[18489]: cluster 2024-02-12T16:31:08.783594+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e669: 12 total, 12 up, 12 in 2024-02-12T16:31:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:09 smithi169 bash[19604]: cluster 2024-02-12T16:31:08.379107+0000 osd.2 (osd.2) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:09 smithi169 bash[19604]: cluster 2024-02-12T16:31:08.379150+0000 osd.2 (osd.2) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:09 smithi169 bash[19604]: cluster 2024-02-12T16:31:08.783594+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e669: 12 total, 12 up, 12 in 2024-02-12T16:31:10.057 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:31:10.060 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:31:10.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:10.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:09 smithi123 bash[19119]: cluster 2024-02-12T16:31:08.379107+0000 osd.2 (osd.2) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:10.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:09 smithi123 bash[19119]: cluster 2024-02-12T16:31:08.379150+0000 osd.2 (osd.2) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:10.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:09 smithi123 bash[19119]: cluster 2024-02-12T16:31:08.783594+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e669: 12 total, 12 up, 12 in 2024-02-12T16:31:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:10.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:10.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:10.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:10.434 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:10.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:10.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:31:10.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:10.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:10.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:10.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:10 smithi123 bash[19119]: audit 2024-02-12T16:31:10.036779+0000 mgr.y (mgr.14146) 1394 : audit [DBG] from='client.16500 v1:172.21.15.121:0/1205205168' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:31:10.787 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:10 smithi123 bash[19119]: cluster 2024-02-12T16:31:10.050045+0000 mgr.y (mgr.14146) 1395 : cluster [DBG] pgmap v1945: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 9.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-02-12T16:31:10.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:10.843 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:10.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:10 smithi121 bash[18489]: audit 2024-02-12T16:31:10.036779+0000 mgr.y (mgr.14146) 1394 : audit [DBG] from='client.16500 v1:172.21.15.121:0/1205205168' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:31:10.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:10 smithi121 bash[18489]: cluster 2024-02-12T16:31:10.050045+0000 mgr.y (mgr.14146) 1395 : cluster [DBG] pgmap v1945: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 9.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-02-12T16:31:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:10.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:10.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:10.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:10 smithi169 bash[19604]: audit 2024-02-12T16:31:10.036779+0000 mgr.y (mgr.14146) 1394 : audit [DBG] from='client.16500 v1:172.21.15.121:0/1205205168' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:31:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:10 smithi169 bash[19604]: cluster 2024-02-12T16:31:10.050045+0000 mgr.y (mgr.14146) 1395 : cluster [DBG] pgmap v1945: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 9.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-02-12T16:31:11.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:11.112 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:11.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:11.215 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:11.249 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:11.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:11.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:11.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:11.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:11.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:11.517 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:11.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:11.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:31:11.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:11.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:11.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:11.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:11.723 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1944,"stamp":"2024-02-12T16:31:08.749429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9681240088,"num_objects":147725,"num_object_clones":0,"num_object_copies":295450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157943,"num_write_kb":9781313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158535,"ondisk_log_size":158535,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":24117940,"kb_used_data":18992660,"kb_used_omap":91,"kb_used_meta":5125092,"kb_avail":1100823884,"statfs":{"total":1151940427776,"available":1127243657216,"internally_reserved":0,"allocated":19448483840,"data_stored":19443753380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":5248094224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,10,10,12,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-287965160,"num_objects":-4393,"num_object_clones":0,"num_object_copies":-8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4393,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4867,"num_write_kb":15105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.711208"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":10,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.399414+0000","last_change":"2024-02-12T16:30:55.395938+0000","last_active":"2024-02-12T16:30:55.399414+0000","last_peered":"2024-02-12T16:30:55.399414+0000","last_clean":"2024-02-12T16:30:55.399414+0000","last_became_active":"2024-02-12T16:30:55.395625+0000","last_became_peered":"2024-02-12T16:30:55.395625+0000","last_unstale":"2024-02-12T16:30:55.399414+0000","last_undegraded":"2024-02-12T16:30:55.399414+0000","last_fullsized":"2024-02-12T16:30:55.399414+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.919678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.919678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":10,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.399159+0000","last_change":"2024-02-12T16:30:55.398809+0000","last_active":"2024-02-12T16:30:55.399159+0000","last_peered":"2024-02-12T16:30:55.399159+0000","last_clean":"2024-02-12T16:30:55.399159+0000","last_became_active":"2024-02-12T16:30:55.398431+0000","last_became_peered":"2024-02-12T16:30:55.398431+0000","last_unstale":"2024-02-12T16:30:55.399159+0000","last_undegraded":"2024-02-12T16:30:55.399159+0000","last_fullsized":"2024-02-12T16:30:55.399159+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:17.602619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.602619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":10,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.405826+0000","last_change":"2024-02-12T16:30:55.405502+0000","last_active":"2024-02-12T16:30:55.405826+0000","last_peered":"2024-02-12T16:30:55.405826+0000","last_clean":"2024-02-12T16:30:55.405826+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:30:55.405826+0000","last_undegraded":"2024-02-12T16:30:55.405826+0000","last_fullsized":"2024-02-12T16:30:55.405826+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:18.631349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:18.631349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":10,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.412462+0000","last_change":"2024-02-12T16:30:55.409915+0000","last_active":"2024-02-12T16:30:55.412462+0000","last_peered":"2024-02-12T16:30:55.412462+0000","last_clean":"2024-02-12T16:30:55.412462+0000","last_became_active":"2024-02-12T16:30:55.409579+0000","last_became_peered":"2024-02-12T16:30:55.409579+0000","last_unstale":"2024-02-12T16:30:55.412462+0000","last_undegraded":"2024-02-12T16:30:55.412462+0000","last_fullsized":"2024-02-12T16:30:55.412462+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:13.391676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:13.391676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":10,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.394673+0000","last_change":"2024-02-12T16:30:55.394405+0000","last_active":"2024-02-12T16:30:55.394673+0000","last_peered":"2024-02-12T16:30:55.394673+0000","last_clean":"2024-02-12T16:30:55.394673+0000","last_became_active":"2024-02-12T16:30:55.394105+0000","last_became_peered":"2024-02-12T16:30:55.394105+0000","last_unstale":"2024-02-12T16:30:55.394673+0000","last_undegraded":"2024-02-12T16:30:55.394673+0000","last_fullsized":"2024-02-12T16:30:55.394673+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:17.846300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.846300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":11,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.478015+0000","last_change":"2024-02-12T16:30:55.542797+0000","last_active":"2024-02-12T16:31:03.478015+0000","last_peered":"2024-02-12T16:31:03.478015+0000","last_clean":"2024-02-12T16:31:03.478015+0000","last_became_active":"2024-02-12T16:30:55.542451+0000","last_became_peered":"2024-02-12T16:30:55.542451+0000","last_unstale":"2024-02-12T16:31:03.478015+0000","last_undegraded":"2024-02-12T16:31:03.478015+0000","last_fullsized":"2024-02-12T16:31:03.478015+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:24.570146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:24.570146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":359,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.400468+0000","last_change":"2024-02-12T16:30:49.377194+0000","last_active":"2024-02-12T16:30:55.400468+0000","last_peered":"2024-02-12T16:30:55.400468+0000","last_clean":"2024-02-12T16:30:55.400468+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:30:55.400468+0000","last_undegraded":"2024-02-12T16:30:55.400468+0000","last_fullsized":"2024-02-12T16:30:55.400468+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.761836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.761836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":10,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.404441+0000","last_change":"2024-02-12T16:30:55.400943+0000","last_active":"2024-02-12T16:30:55.404441+0000","last_peered":"2024-02-12T16:30:55.404441+0000","last_clean":"2024-02-12T16:30:55.404441+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:30:55.404441+0000","last_undegraded":"2024-02-12T16:30:55.404441+0000","last_fullsized":"2024-02-12T16:30:55.404441+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:23.709532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:23.709532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":10,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.402993+0000","last_change":"2024-02-12T16:30:55.402535+0000","last_active":"2024-02-12T16:30:55.402993+0000","last_peered":"2024-02-12T16:30:55.402993+0000","last_clean":"2024-02-12T16:30:55.402993+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:30:55.402993+0000","last_undegraded":"2024-02-12T16:30:55.402993+0000","last_fullsized":"2024-02-12T16:30:55.402993+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:26.061171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:26.061171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":11,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.422968+0000","last_change":"2024-02-12T16:30:55.405054+0000","last_active":"2024-02-12T16:31:03.422968+0000","last_peered":"2024-02-12T16:31:03.422968+0000","last_clean":"2024-02-12T16:31:03.422968+0000","last_became_active":"2024-02-12T16:30:55.404763+0000","last_became_peered":"2024-02-12T16:30:55.404763+0000","last_unstale":"2024-02-12T16:31:03.422968+0000","last_undegraded":"2024-02-12T16:31:03.422968+0000","last_fullsized":"2024-02-12T16:31:03.422968+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:28.981314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:30:28.981314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":164,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.404214+0000","last_change":"2024-02-12T16:29:20.828257+0000","last_active":"2024-02-12T16:30:55.404214+0000","last_peered":"2024-02-12T16:30:55.404214+0000","last_clean":"2024-02-12T16:30:55.404214+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:30:55.404214+0000","last_undegraded":"2024-02-12T16:30:55.404214+0000","last_fullsized":"2024-02-12T16:30:55.404214+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:20.828149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:29:20.828149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"667'9847","reported_seq":9890,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:07.098870+0000","last_change":"2024-02-12T16:30:33.673651+0000","last_active":"2024-02-12T16:31:07.098870+0000","last_peered":"2024-02-12T16:31:07.098870+0000","last_clean":"2024-02-12T16:31:07.098870+0000","last_became_active":"2024-02-12T16:29:08.736890+0000","last_became_peered":"2024-02-12T16:29:08.736890+0000","last_unstale":"2024-02-12T16:31:07.098870+0000","last_undegraded":"2024-02-12T16:31:07.098870+0000","last_fullsized":"2024-02-12T16:31:07.098870+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8268","last_scrub_stamp":"2024-02-12T16:30:33.673564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:33.673564+0000","log_size":9847,"ondisk_log_size":9847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":598933504,"num_objects":9139,"num_object_clones":0,"num_object_copies":18278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9847,"num_write_kb":607552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":281,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.407740+0000","last_change":"2024-02-12T16:30:09.838606+0000","last_active":"2024-02-12T16:30:55.407740+0000","last_peered":"2024-02-12T16:30:55.407740+0000","last_clean":"2024-02-12T16:30:55.407740+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:30:55.407740+0000","last_undegraded":"2024-02-12T16:30:55.407740+0000","last_fullsized":"2024-02-12T16:30:55.407740+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.838471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.838471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"667'9822","reported_seq":9864,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:04.332522+0000","last_change":"2024-02-12T16:30:39.183658+0000","last_active":"2024-02-12T16:31:04.332522+0000","last_peered":"2024-02-12T16:31:04.332522+0000","last_clean":"2024-02-12T16:31:04.332522+0000","last_became_active":"2024-02-12T16:29:08.734223+0000","last_became_peered":"2024-02-12T16:29:08.734223+0000","last_unstale":"2024-02-12T16:31:04.332522+0000","last_undegraded":"2024-02-12T16:31:04.332522+0000","last_fullsized":"2024-02-12T16:31:04.332522+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8683","last_scrub_stamp":"2024-02-12T16:30:39.183515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:39.183515+0000","log_size":9822,"ondisk_log_size":9822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":613941248,"num_objects":9368,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9822,"num_write_kb":614080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":367,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.478033+0000","last_change":"2024-02-12T16:30:49.374675+0000","last_active":"2024-02-12T16:31:03.478033+0000","last_peered":"2024-02-12T16:31:03.478033+0000","last_clean":"2024-02-12T16:31:03.478033+0000","last_became_active":"2024-02-12T16:30:49.374358+0000","last_became_peered":"2024-02-12T16:30:49.374358+0000","last_unstale":"2024-02-12T16:31:03.478033+0000","last_undegraded":"2024-02-12T16:31:03.478033+0000","last_fullsized":"2024-02-12T16:31:03.478033+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:28.981314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:30:28.981314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"668'9914","reported_seq":9953,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-02-12T16:31:08.391573+0000","last_change":"2024-02-12T16:29:08.805574+0000","last_active":"2024-02-12T16:31:08.391573+0000","last_peered":"2024-02-12T16:31:08.391573+0000","last_clean":"2024-02-12T16:31:08.391573+0000","last_became_active":"2024-02-12T16:29:08.805299+0000","last_became_peered":"2024-02-12T16:29:08.805299+0000","last_unstale":"2024-02-12T16:31:08.391573+0000","last_undegraded":"2024-02-12T16:31:08.391573+0000","last_fullsized":"2024-02-12T16:31:08.391573+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":9914,"ondisk_log_size":9914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":593756160,"num_objects":9060,"num_object_clones":0,"num_object_copies":18120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9914,"num_write_kb":607168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":762,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.388821+0000","last_change":"2024-02-12T16:30:10.264990+0000","last_active":"2024-02-12T16:30:55.388821+0000","last_peered":"2024-02-12T16:30:55.388821+0000","last_clean":"2024-02-12T16:30:55.388821+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:30:55.388821+0000","last_undegraded":"2024-02-12T16:30:55.388821+0000","last_fullsized":"2024-02-12T16:30:55.388821+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.264894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.264894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.0","version":"667'9831","reported_seq":9869,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:06.005659+0000","last_change":"2024-02-12T16:29:08.733321+0000","last_active":"2024-02-12T16:31:06.005659+0000","last_peered":"2024-02-12T16:31:06.005659+0000","last_clean":"2024-02-12T16:31:06.005659+0000","last_became_active":"2024-02-12T16:29:08.732997+0000","last_became_peered":"2024-02-12T16:29:08.732997+0000","last_unstale":"2024-02-12T16:31:06.005659+0000","last_undegraded":"2024-02-12T16:31:06.005659+0000","last_fullsized":"2024-02-12T16:31:06.005659+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":9831,"ondisk_log_size":9831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":604045312,"num_objects":9217,"num_object_clones":0,"num_object_copies":18434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9831,"num_write_kb":609536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":410,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.404654+0000","last_change":"2024-02-12T16:30:49.379926+0000","last_active":"2024-02-12T16:30:55.404654+0000","last_peered":"2024-02-12T16:30:55.404654+0000","last_clean":"2024-02-12T16:30:55.404654+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:30:55.404654+0000","last_undegraded":"2024-02-12T16:30:55.404654+0000","last_fullsized":"2024-02-12T16:30:55.404654+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:23.709532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:23.709532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"667'9952","reported_seq":9995,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.880201+0000","last_change":"2024-02-12T16:30:14.669480+0000","last_active":"2024-02-12T16:31:03.880201+0000","last_peered":"2024-02-12T16:31:03.880201+0000","last_clean":"2024-02-12T16:31:03.880201+0000","last_became_active":"2024-02-12T16:29:08.808944+0000","last_became_peered":"2024-02-12T16:29:08.808944+0000","last_unstale":"2024-02-12T16:31:03.880201+0000","last_undegraded":"2024-02-12T16:31:03.880201+0000","last_fullsized":"2024-02-12T16:31:03.880201+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"656'6950","last_scrub_stamp":"2024-02-12T16:30:14.669359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:14.669359+0000","log_size":9952,"ondisk_log_size":9952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":629407744,"num_objects":9604,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":625792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":358,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.478045+0000","last_change":"2024-02-12T16:30:49.374289+0000","last_active":"2024-02-12T16:31:03.478045+0000","last_peered":"2024-02-12T16:31:03.478045+0000","last_clean":"2024-02-12T16:31:03.478045+0000","last_became_active":"2024-02-12T16:30:49.373979+0000","last_became_peered":"2024-02-12T16:30:49.373979+0000","last_unstale":"2024-02-12T16:31:03.478045+0000","last_undegraded":"2024-02-12T16:31:03.478045+0000","last_fullsized":"2024-02-12T16:31:03.478045+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:26.061171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:26.061171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"667'9711","reported_seq":9748,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:05.283172+0000","last_change":"2024-02-12T16:29:08.733146+0000","last_active":"2024-02-12T16:31:05.283172+0000","last_peered":"2024-02-12T16:31:05.283172+0000","last_clean":"2024-02-12T16:31:05.283172+0000","last_became_active":"2024-02-12T16:29:08.732848+0000","last_became_peered":"2024-02-12T16:29:08.732848+0000","last_unstale":"2024-02-12T16:31:05.283172+0000","last_undegraded":"2024-02-12T16:31:05.283172+0000","last_fullsized":"2024-02-12T16:31:05.283172+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":9711,"ondisk_log_size":9711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":598278144,"num_objects":9129,"num_object_clones":0,"num_object_copies":18258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9711,"num_write_kb":602880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":362,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.478023+0000","last_change":"2024-02-12T16:30:25.021242+0000","last_active":"2024-02-12T16:31:03.478023+0000","last_peered":"2024-02-12T16:31:03.478023+0000","last_clean":"2024-02-12T16:31:03.478023+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:31:03.478023+0000","last_undegraded":"2024-02-12T16:31:03.478023+0000","last_fullsized":"2024-02-12T16:31:03.478023+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:25.021151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:30:25.021151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"667'9925","reported_seq":9967,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:06.818409+0000","last_change":"2024-02-12T16:30:27.112997+0000","last_active":"2024-02-12T16:31:06.818409+0000","last_peered":"2024-02-12T16:31:06.818409+0000","last_clean":"2024-02-12T16:31:06.818409+0000","last_became_active":"2024-02-12T16:29:08.734296+0000","last_became_peered":"2024-02-12T16:29:08.734296+0000","last_unstale":"2024-02-12T16:31:06.818409+0000","last_undegraded":"2024-02-12T16:31:06.818409+0000","last_fullsized":"2024-02-12T16:31:06.818409+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7916","last_scrub_stamp":"2024-02-12T16:30:27.112924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:27.112924+0000","log_size":9925,"ondisk_log_size":9925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":604176384,"num_objects":9219,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":612608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":218,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.398032+0000","last_change":"2024-02-12T16:30:10.604389+0000","last_active":"2024-02-12T16:30:55.398032+0000","last_peered":"2024-02-12T16:30:55.398032+0000","last_clean":"2024-02-12T16:30:55.398032+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:30:55.398032+0000","last_undegraded":"2024-02-12T16:30:55.398032+0000","last_fullsized":"2024-02-12T16:30:55.398032+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.604303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.604303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"668'10062","reported_seq":10106,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-02-12T16:31:08.388196+0000","last_change":"2024-02-12T16:30:27.497822+0000","last_active":"2024-02-12T16:31:08.388196+0000","last_peered":"2024-02-12T16:31:08.388196+0000","last_clean":"2024-02-12T16:31:08.388196+0000","last_became_active":"2024-02-12T16:29:08.804218+0000","last_became_peered":"2024-02-12T16:29:08.804218+0000","last_unstale":"2024-02-12T16:31:08.388196+0000","last_undegraded":"2024-02-12T16:31:08.388196+0000","last_fullsized":"2024-02-12T16:31:08.388196+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7944","last_scrub_stamp":"2024-02-12T16:30:27.497708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:27.497708+0000","log_size":10062,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":601489408,"num_objects":9178,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10062,"num_write_kb":615680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":791,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.422936+0000","last_change":"2024-02-12T16:30:09.726295+0000","last_active":"2024-02-12T16:31:03.422936+0000","last_peered":"2024-02-12T16:31:03.422936+0000","last_clean":"2024-02-12T16:31:03.422936+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:31:03.422936+0000","last_undegraded":"2024-02-12T16:31:03.422936+0000","last_fullsized":"2024-02-12T16:31:03.422936+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.726187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.726187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"667'9853","reported_seq":9896,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:07.135406+0000","last_change":"2024-02-12T16:30:21.757551+0000","last_active":"2024-02-12T16:31:07.135406+0000","last_peered":"2024-02-12T16:31:07.135406+0000","last_clean":"2024-02-12T16:31:07.135406+0000","last_became_active":"2024-02-12T16:29:08.804047+0000","last_became_peered":"2024-02-12T16:29:08.804047+0000","last_unstale":"2024-02-12T16:31:07.135406+0000","last_undegraded":"2024-02-12T16:31:07.135406+0000","last_fullsized":"2024-02-12T16:31:07.135406+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7354","last_scrub_stamp":"2024-02-12T16:30:21.757462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:21.757462+0000","log_size":9853,"ondisk_log_size":9853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":601554944,"num_objects":9179,"num_object_clones":0,"num_object_copies":18358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9853,"num_write_kb":609024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":174,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.423035+0000","last_change":"2024-02-12T16:30:49.375200+0000","last_active":"2024-02-12T16:31:03.423035+0000","last_peered":"2024-02-12T16:31:03.423035+0000","last_clean":"2024-02-12T16:31:03.423035+0000","last_became_active":"2024-02-12T16:30:49.374934+0000","last_became_peered":"2024-02-12T16:30:49.374934+0000","last_unstale":"2024-02-12T16:31:03.423035+0000","last_undegraded":"2024-02-12T16:31:03.423035+0000","last_fullsized":"2024-02-12T16:31:03.423035+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:17.602619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.602619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"667'9895","reported_seq":9944,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:05.330678+0000","last_change":"2024-02-12T16:30:47.824137+0000","last_active":"2024-02-12T16:31:05.330678+0000","last_peered":"2024-02-12T16:31:05.330678+0000","last_clean":"2024-02-12T16:31:05.330678+0000","last_became_active":"2024-02-12T16:29:08.734174+0000","last_became_peered":"2024-02-12T16:29:08.734174+0000","last_unstale":"2024-02-12T16:31:05.330678+0000","last_undegraded":"2024-02-12T16:31:05.330678+0000","last_fullsized":"2024-02-12T16:31:05.330678+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":9895,"ondisk_log_size":9895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":610861056,"num_objects":9321,"num_object_clones":0,"num_object_copies":18642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":614912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":198,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.398970+0000","last_change":"2024-02-12T16:30:49.382635+0000","last_active":"2024-02-12T16:30:55.398970+0000","last_peered":"2024-02-12T16:30:55.398970+0000","last_clean":"2024-02-12T16:30:55.398970+0000","last_became_active":"2024-02-12T16:30:49.382268+0000","last_became_peered":"2024-02-12T16:30:49.382268+0000","last_unstale":"2024-02-12T16:30:55.398970+0000","last_undegraded":"2024-02-12T16:30:55.398970+0000","last_fullsized":"2024-02-12T16:30:55.398970+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:18.631349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:18.631349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"668'15144","reported_seq":15186,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-02-12T16:31:08.381395+0000","last_change":"2024-02-12T16:31:07.796335+0000","last_active":"2024-02-12T16:31:08.381395+0000","last_peered":"2024-02-12T16:31:08.381395+0000","last_clean":"2024-02-12T16:31:08.381395+0000","last_became_active":"2024-02-12T16:29:08.734560+0000","last_became_peered":"2024-02-12T16:29:08.734560+0000","last_unstale":"2024-02-12T16:31:08.381395+0000","last_undegraded":"2024-02-12T16:31:08.381395+0000","last_fullsized":"2024-02-12T16:31:08.381395+0000","mapping_epoch":636,"log_start":"653'5103","ondisk_log_start":"653'5103","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10041,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":594804736,"num_objects":9076,"num_object_clones":0,"num_object_copies":18152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9872,"num_write_kb":606336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":363,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.404720+0000","last_change":"2024-02-12T16:30:14.850065+0000","last_active":"2024-02-12T16:30:55.404720+0000","last_peered":"2024-02-12T16:30:55.404720+0000","last_clean":"2024-02-12T16:30:55.404720+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:30:55.404720+0000","last_undegraded":"2024-02-12T16:30:55.404720+0000","last_fullsized":"2024-02-12T16:30:55.404720+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:14.849876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:30:14.849876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"667'14102","reported_seq":14142,"reported_epoch":667,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-12T16:31:06.012101+0000","last_change":"2024-02-12T16:30:36.473592+0000","last_active":"2024-02-12T16:31:06.012101+0000","last_peered":"2024-02-12T16:31:06.012101+0000","last_clean":"2024-02-12T16:31:06.012101+0000","last_became_active":"2024-02-12T16:29:08.807333+0000","last_became_peered":"2024-02-12T16:29:08.807333+0000","last_unstale":"2024-02-12T16:31:06.012101+0000","last_undegraded":"2024-02-12T16:31:06.012101+0000","last_fullsized":"2024-02-12T16:31:06.012101+0000","mapping_epoch":636,"log_start":"651'4101","ondisk_log_start":"651'4101","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10001,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":597950464,"num_objects":9124,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":605504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":173,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.393328+0000","last_change":"2024-02-12T16:30:49.381550+0000","last_active":"2024-02-12T16:30:55.393328+0000","last_peered":"2024-02-12T16:30:55.393328+0000","last_clean":"2024-02-12T16:30:55.393328+0000","last_became_active":"2024-02-12T16:30:49.381004+0000","last_became_peered":"2024-02-12T16:30:49.381004+0000","last_unstale":"2024-02-12T16:30:55.393328+0000","last_undegraded":"2024-02-12T16:30:55.393328+0000","last_fullsized":"2024-02-12T16:30:55.393328+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.919678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.919678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"667'9773","reported_seq":9815,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:04.322813+0000","last_change":"2024-02-12T16:30:23.174523+0000","last_active":"2024-02-12T16:31:04.322813+0000","last_peered":"2024-02-12T16:31:04.322813+0000","last_clean":"2024-02-12T16:31:04.322813+0000","last_became_active":"2024-02-12T16:29:08.735885+0000","last_became_peered":"2024-02-12T16:29:08.735885+0000","last_unstale":"2024-02-12T16:31:04.322813+0000","last_undegraded":"2024-02-12T16:31:04.322813+0000","last_fullsized":"2024-02-12T16:31:04.322813+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7481","last_scrub_stamp":"2024-02-12T16:30:23.174453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:23.174453+0000","log_size":9773,"ondisk_log_size":9773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":609812480,"num_objects":9305,"num_object_clones":0,"num_object_copies":18610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9773,"num_write_kb":610496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":774,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:03.422912+0000","last_change":"2024-02-12T16:30:49.405085+0000","last_active":"2024-02-12T16:31:03.422912+0000","last_peered":"2024-02-12T16:31:03.422912+0000","last_clean":"2024-02-12T16:31:03.422912+0000","last_became_active":"2024-02-12T16:30:49.404783+0000","last_became_peered":"2024-02-12T16:30:49.404783+0000","last_unstale":"2024-02-12T16:31:03.422912+0000","last_undegraded":"2024-02-12T16:31:03.422912+0000","last_fullsized":"2024-02-12T16:31:03.422912+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:24.570146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:24.570146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"667'14793","reported_seq":14833,"reported_epoch":667,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-12T16:31:06.658440+0000","last_change":"2024-02-12T16:30:34.544505+0000","last_active":"2024-02-12T16:31:06.658440+0000","last_peered":"2024-02-12T16:31:06.658440+0000","last_clean":"2024-02-12T16:31:06.658440+0000","last_became_active":"2024-02-12T16:29:08.834297+0000","last_became_peered":"2024-02-12T16:29:08.834297+0000","last_unstale":"2024-02-12T16:31:06.658440+0000","last_undegraded":"2024-02-12T16:31:06.658440+0000","last_fullsized":"2024-02-12T16:31:06.658440+0000","mapping_epoch":636,"log_start":"652'4701","ondisk_log_start":"652'4701","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10092,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":604110848,"num_objects":9218,"num_object_clones":0,"num_object_copies":18436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9872,"num_write_kb":610880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":360,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.400470+0000","last_change":"2024-02-12T16:30:49.377059+0000","last_active":"2024-02-12T16:30:55.400470+0000","last_peered":"2024-02-12T16:30:55.400470+0000","last_clean":"2024-02-12T16:30:55.400470+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:30:55.400470+0000","last_undegraded":"2024-02-12T16:30:55.400470+0000","last_fullsized":"2024-02-12T16:30:55.400470+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.761836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.761836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"667'9878","reported_seq":9920,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:05.305473+0000","last_change":"2024-02-12T16:30:17.583794+0000","last_active":"2024-02-12T16:31:05.305473+0000","last_peered":"2024-02-12T16:31:05.305473+0000","last_clean":"2024-02-12T16:31:05.305473+0000","last_became_active":"2024-02-12T16:29:08.804528+0000","last_became_peered":"2024-02-12T16:29:08.804528+0000","last_unstale":"2024-02-12T16:31:05.305473+0000","last_undegraded":"2024-02-12T16:31:05.305473+0000","last_fullsized":"2024-02-12T16:31:05.305473+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"657'7134","last_scrub_stamp":"2024-02-12T16:30:17.583727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.583727+0000","log_size":9878,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":609878016,"num_objects":9306,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9878,"num_write_kb":613888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":112,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.388942+0000","last_change":"2024-02-12T16:30:49.376570+0000","last_active":"2024-02-12T16:30:55.388942+0000","last_peered":"2024-02-12T16:30:55.388942+0000","last_clean":"2024-02-12T16:30:55.388942+0000","last_became_active":"2024-02-12T16:30:49.376219+0000","last_became_peered":"2024-02-12T16:30:49.376219+0000","last_unstale":"2024-02-12T16:30:55.388942+0000","last_undegraded":"2024-02-12T16:30:55.388942+0000","last_fullsized":"2024-02-12T16:30:55.388942+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:13.391676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:13.391676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"667'9938","reported_seq":9982,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-02-12T16:31:07.094590+0000","last_change":"2024-02-12T16:30:36.029310+0000","last_active":"2024-02-12T16:31:07.094590+0000","last_peered":"2024-02-12T16:31:07.094590+0000","last_clean":"2024-02-12T16:31:07.094590+0000","last_became_active":"2024-02-12T16:29:08.735711+0000","last_became_peered":"2024-02-12T16:29:08.735711+0000","last_unstale":"2024-02-12T16:31:07.094590+0000","last_undegraded":"2024-02-12T16:31:07.094590+0000","last_fullsized":"2024-02-12T16:31:07.094590+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8520","last_scrub_stamp":"2024-02-12T16:30:36.029199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:36.029199+0000","log_size":9938,"ondisk_log_size":9938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":608239640,"num_objects":9282,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9938,"num_write_kb":614977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":352,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.407960+0000","last_change":"2024-02-12T16:30:49.384357+0000","last_active":"2024-02-12T16:30:55.407960+0000","last_peered":"2024-02-12T16:30:55.407960+0000","last_clean":"2024-02-12T16:30:55.407960+0000","last_became_active":"2024-02-12T16:30:49.384015+0000","last_became_peered":"2024-02-12T16:30:49.384015+0000","last_unstale":"2024-02-12T16:30:55.407960+0000","last_undegraded":"2024-02-12T16:30:55.407960+0000","last_fullsized":"2024-02-12T16:30:55.407960+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:17.846300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.846300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":161,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.400576+0000","last_change":"2024-02-12T16:30:12.797215+0000","last_active":"2024-02-12T16:30:55.400576+0000","last_peered":"2024-02-12T16:30:55.400576+0000","last_clean":"2024-02-12T16:30:55.400576+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:30:55.400576+0000","last_undegraded":"2024-02-12T16:30:55.400576+0000","last_fullsized":"2024-02-12T16:30:55.400576+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:12.797116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:30:12.797116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":160,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.539437+0000","last_change":"2024-02-12T16:30:10.717293+0000","last_active":"2024-02-12T16:30:55.539437+0000","last_peered":"2024-02-12T16:30:55.539437+0000","last_clean":"2024-02-12T16:30:55.539437+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:30:55.539437+0000","last_undegraded":"2024-02-12T16:30:55.539437+0000","last_fullsized":"2024-02-12T16:30:55.539437+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.717201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.717201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":175,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.400678+0000","last_change":"2024-02-12T16:30:18.905746+0000","last_active":"2024-02-12T16:30:55.400678+0000","last_peered":"2024-02-12T16:30:55.400678+0000","last_clean":"2024-02-12T16:30:55.400678+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:30:55.400678+0000","last_undegraded":"2024-02-12T16:30:55.400678+0000","last_fullsized":"2024-02-12T16:30:55.400678+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:18.901023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:30:18.901023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":154,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-02-12T16:30:55.390386+0000","last_change":"2024-02-12T16:30:09.745507+0000","last_active":"2024-02-12T16:30:55.390386+0000","last_peered":"2024-02-12T16:30:55.390386+0000","last_clean":"2024-02-12T16:30:55.390386+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:30:55.390386+0000","last_undegraded":"2024-02-12T16:30:55.390386+0000","last_fullsized":"2024-02-12T16:30:55.390386+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.745399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.745399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":9681240088,"num_objects":147725,"num_object_clones":0,"num_object_copies":295450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157943,"num_write_kb":9781313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19376906240,"data_stored":19376898096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158535,"ondisk_log_size":158535,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958923,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586376,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":93158776,"statfs":{"total":95995035648,"available":95394586624,"internally_reserved":0,"allocated":786432,"data_stored":393404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.747,"15min":0.86599999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.246},"max":{"1min":2.8879999999999999,"5min":5.6600000000000001,"15min":5.6870000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.93600000000000005,"15min":0.89700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28399999999999997},"max":{"1min":1.1950000000000001,"5min":5.7080000000000002,"15min":5.7080000000000002},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.0329999999999999,"15min":1.0620000000000001},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":8.9849999999999994,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.88900000000000001,"15min":1.0600000000000001},"min":{"1min":0.33900000000000002,"5min":0.311,"15min":0.311},"max":{"1min":2.2429999999999999,"5min":3.6859999999999999,"15min":5.6920000000000002},"last":0.62}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.89800000000000002,"15min":1.2090000000000001},"min":{"1min":0.441,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":1.6859999999999999,"5min":4.1639999999999997,"15min":5.343},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.879,"15min":1.268},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":3.96,"5min":5.4340000000000002,"15min":8.2759999999999998},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.91400000000000003,"15min":0.998},"min":{"1min":0.36099999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.657,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.88200000000000001,"15min":0.93000000000000005},"min":{"1min":0.378,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":1.5,"5min":4.101,"15min":4.915},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.145,"15min":1.1799999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.35099999999999998},"max":{"1min":5.968,"5min":5.968,"15min":5.968},"last":3.3010000000000002},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.1799999999999999,"15min":1.1679999999999999},"min":{"1min":0.47399999999999998,"5min":0.36699999999999999,"15min":0.35499999999999998},"max":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":5.8879999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":0.99399999999999999,"15min":1.006},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.373},"max":{"1min":2.9700000000000002,"5min":4.7370000000000001,"15min":5.141},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0669999999999999,"15min":1.1339999999999999},"min":{"1min":0.48699999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"max":{"1min":2.5800000000000001,"5min":4.7450000000000001,"15min":8.5709999999999997},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.97599999999999998,"15min":1.2110000000000001},"min":{"1min":0.61699999999999999,"5min":0.434,"15min":0.38600000000000001},"max":{"1min":2.8090000000000002,"5min":3.9089999999999998,"15min":6.4370000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.91500000000000004,"15min":1.129},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.4969999999999999,"5min":4.2350000000000003,"15min":5.0430000000000001},"last":3.137}]},{"osd":8,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":6.1959999999999997,"15min":2.7890000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.1020000000000001,"5min":537.54399999999998,"15min":537.54399999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":6.3129999999999997,"15min":2.9630000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.34699999999999998},"max":{"1min":0.94199999999999995,"5min":539.86099999999999,"15min":539.86099999999999},"last":0.751}]},{"osd":9,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.0349999999999999,"15min":1.252},"min":{"1min":0.55200000000000005,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.4329999999999998,"5min":3.8100000000000001,"15min":5.843},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.77100000000000002,"5min":1.0009999999999999,"15min":1.3149999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98699999999999999,"5min":3.8399999999999999,"15min":6.8869999999999996},"last":0.90000000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":6.327,"15min":2.9510000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.829,"5min":536.245,"15min":536.245},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":6.2850000000000001,"15min":2.9390000000000001},"min":{"1min":0.51300000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.4370000000000001,"5min":536.64300000000003,"15min":536.64300000000003},"last":0.73799999999999999}]}]},{"osd":10,"up_from":83,"seq":356482285973,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892884,"kb_used_data":583372,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92852268,"statfs":{"total":95995035648,"available":95080722432,"internally_reserved":0,"allocated":597372928,"data_stored":596973840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":2.7610000000000001,"15min":1.5249999999999999},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.245},"max":{"1min":1.591,"5min":162.38399999999999,"15min":162.38399999999999},"last":4.1769999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":2.8340000000000001,"15min":1.6040000000000001},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.4179999999999999,"5min":162.01300000000001,"15min":162.01300000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.90700000000000003,"15min":0.998},"min":{"1min":0.32600000000000001,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":3.1520000000000001,"5min":5.9000000000000004,"15min":19.866},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.131},"min":{"1min":0.32800000000000001,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":4.2309999999999999,"5min":6.1970000000000001,"15min":20.277999999999999},"last":0.751}]},{"osd":2,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.1359999999999999,"15min":1.0089999999999999},"min":{"1min":0.35999999999999999,"5min":0.29899999999999999,"15min":0.28899999999999998},"max":{"1min":5.016,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.083,"15min":1.0600000000000001},"min":{"1min":0.42299999999999999,"5min":0.34699999999999998,"15min":0.32000000000000001},"max":{"1min":5.1820000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.78800000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.0209999999999999,"15min":1.0740000000000001},"min":{"1min":0.46300000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.3689999999999998,"5min":6.3689999999999998,"15min":6.3689999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":1.03,"15min":1.04},"min":{"1min":0.34200000000000003,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":1.8440000000000001,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.0369999999999999,"15min":1.0700000000000001},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.8740000000000001,"5min":6.343,"15min":6.343},"last":0.505},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.2689999999999999,"15min":1.248},"min":{"1min":0.53200000000000003,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":7.2249999999999996,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.81999999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.03,"15min":1.0880000000000001},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":4.907,"5min":6.3010000000000002,"15min":6.3010000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.175,"15min":1.196},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.376},"max":{"1min":4.3689999999999998,"5min":6.8200000000000003,"15min":6.8200000000000003},"last":0.85499999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.099,"15min":1.0940000000000001},"min":{"1min":0.47999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":3.0249999999999999,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1519999999999999,"15min":1.2230000000000001},"min":{"1min":0.46500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":4.0659999999999998,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.57899999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1240000000000001,"15min":1.1719999999999999},"min":{"1min":0.51400000000000001,"5min":0.40400000000000003,"15min":0.372},"max":{"1min":4.3399999999999999,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.98999999999999999,"15min":1.129},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.35799999999999998},"max":{"1min":1.097,"5min":7.282,"15min":7.282},"last":0.60299999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.1200000000000001,"15min":1.2370000000000001},"min":{"1min":0.61499999999999999,"5min":0.505,"15min":0.372},"max":{"1min":1.204,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":1.04,"15min":1.1930000000000001},"min":{"1min":0.433,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":1.2490000000000001,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.88400000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.0900000000000001,"15min":1.264},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39300000000000002},"max":{"1min":1.149,"5min":7.3689999999999998,"15min":10.218999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.1060000000000001,"15min":1.282},"min":{"1min":0.497,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1830000000000001,"5min":6.5860000000000003,"15min":7.3250000000000002},"last":0.94199999999999995}]}]},{"osd":9,"up_from":74,"seq":317827580317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1500104,"kb_used_data":1194688,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92245048,"statfs":{"total":95995035648,"available":94458929152,"internally_reserved":0,"allocated":1223360512,"data_stored":1222967492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:09 2024","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.397,"15min":1.145},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":19.120000000000001,"5min":19.120000000000001,"15min":19.120000000000001},"last":0.434},{"interface":"front","average":{"1min":1.623,"5min":1.411,"15min":1.0600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.28299999999999997},"max":{"1min":19.222000000000001,"5min":19.222000000000001,"15min":19.222000000000001},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.085,"15min":0.99199999999999999},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.253},"max":{"1min":1.6080000000000001,"5min":5.8819999999999997,"15min":5.8819999999999997},"last":0.375},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.98499999999999999,"15min":1.024},"min":{"1min":0.35699999999999998,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":1.1659999999999999,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.196,"15min":1.4419999999999999},"min":{"1min":0.33700000000000002,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":3.3370000000000002,"5min":12.441000000000001,"15min":97.275000000000006},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.147,"15min":1.359},"min":{"1min":0.33500000000000002,"5min":0.33000000000000002,"15min":0.312},"max":{"1min":4.0019999999999998,"5min":4.6239999999999997,"15min":97.417000000000002},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.0800000000000001,"15min":1.0169999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.8900000000000001,"5min":6.3090000000000002,"15min":6.3090000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.1619999999999999,"15min":1.161},"min":{"1min":0.47799999999999998,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":3.9289999999999998,"5min":6.3490000000000002,"15min":6.3490000000000002},"last":0.73899999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2390000000000001,"15min":1.1839999999999999},"min":{"1min":0.378,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":6.8559999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.202,"5min":1.2090000000000001,"15min":1.159},"min":{"1min":0.46999999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.0880000000000001,"15min":1.087},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.378},"max":{"1min":4.3559999999999999,"5min":5.6689999999999996,"15min":5.6689999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.278,"15min":1.147},"min":{"1min":0.379,"5min":0.379,"15min":0.36499999999999999},"max":{"1min":7.71,"5min":14.718,"15min":14.718},"last":0.626}]},{"osd":7,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.194,"15min":1.242},"min":{"1min":0.502,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":5.3479999999999999,"5min":5.3479999999999999,"15min":5.8010000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.256,"5min":1.1919999999999999,"15min":1.214},"min":{"1min":0.45600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.1870000000000003,"5min":4.7000000000000002,"15min":7.9160000000000004},"last":4.4829999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0209999999999999,"15min":1.105},"min":{"1min":0.42599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.1100000000000003,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.1020000000000001,"15min":1.1859999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.42599999999999999},"max":{"1min":2.9260000000000002,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.70499999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.089,"15min":1.252},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36599999999999999},"max":{"1min":1.0349999999999999,"5min":4.6900000000000004,"15min":5.6929999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.875,"5min":1.2170000000000001,"15min":1.2629999999999999},"min":{"1min":0.51400000000000001,"5min":0.44700000000000001,"15min":0.38},"max":{"1min":3.8380000000000001,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":0.76500000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:30:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.077,"15min":1.3360000000000001},"min":{"1min":0.499,"5min":0.47599999999999998,"15min":0.45600000000000002},"max":{"1min":1.0669999999999999,"5min":4.7990000000000004,"15min":5.7690000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":1.022,"15min":1.2549999999999999},"min":{"1min":0.52800000000000002,"5min":0.50700000000000001,"15min":0.41599999999999998},"max":{"1min":1.0880000000000001,"5min":4.7190000000000003,"15min":6.3639999999999999},"last":0.79900000000000004}]}]},{"osd":8,"up_from":65,"seq":279172874662,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508488,"kb_used_data":1211264,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":92236664,"statfs":{"total":95995035648,"available":94450343936,"internally_reserved":0,"allocated":1240334336,"data_stored":1239941316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":1.0509999999999999,"15min":0.90500000000000003},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.86399999999999999,"5min":19.196999999999999,"15min":19.196999999999999},"last":0.307},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.2130000000000001,"15min":0.94099999999999995},"min":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.29399999999999998},"max":{"1min":2.625,"5min":18.584,"15min":18.584},"last":0.34200000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0190000000000001,"15min":1.3320000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.497},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.9870000000000001,"15min":1.4079999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4580000000000002,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.84399999999999997},"min":{"1min":0.40400000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.827,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":0.432},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.02,"15min":1.002},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.29899999999999999},"max":{"1min":1.726,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.45100000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.48999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.81599999999999995,"15min":0.96299999999999997},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.29599999999999999},"max":{"1min":1.829,"5min":5.0490000000000004,"15min":5.0490000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.89700000000000002,"15min":0.94599999999999995},"min":{"1min":0.40300000000000002,"5min":0.35499999999999998,"15min":0.34699999999999998},"max":{"1min":1.859,"5min":5.0380000000000003,"15min":5.0380000000000003},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.02,"15min":0.96199999999999997},"min":{"1min":0.41699999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.1239999999999997,"5min":5.4059999999999997,"15min":5.4059999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.95499999999999996,"15min":1.0289999999999999},"min":{"1min":0.47199999999999998,"5min":0.38600000000000001,"15min":0.38100000000000001},"max":{"1min":1.927,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":0.54700000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0880000000000001,"15min":1.048},"min":{"1min":0.51000000000000001,"5min":0.373,"15min":0.373},"max":{"1min":3.7229999999999999,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.1180000000000001,"15min":1.155},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":9.1110000000000007,"5min":9.1110000000000007,"15min":9.1110000000000007},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.167,"15min":1.1519999999999999},"min":{"1min":0.52200000000000002,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":4.4180000000000001,"5min":4.4180000000000001,"15min":7.9160000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.1930000000000001,"15min":1.2789999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.46400000000000002},"max":{"1min":4.4349999999999996,"5min":4.4349999999999996,"15min":7.899},"last":0.65300000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:30:10 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.012,"15min":1.1599999999999999},"min":{"1min":0.45400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.9099999999999999,"5min":3.8610000000000002,"15min":4.5099999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.129,"5min":1.1100000000000001,"15min":1.254},"min":{"1min":0.52200000000000002,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":2.2719999999999998,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":0.82399999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:30:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.995,"15min":1.4470000000000001},"min":{"1min":0.60999999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.587,"5min":5.2560000000000002,"15min":109.92},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.96499999999999997,"15min":1.492},"min":{"1min":0.53600000000000003,"5min":0.44400000000000001,"15min":0.38500000000000001},"max":{"1min":1.216,"5min":5.0250000000000004,"15min":110.249},"last":0.73799999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:30:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.0629999999999999,"15min":1.1739999999999999},"min":{"1min":0.64900000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.609,"5min":5.2530000000000001,"15min":5.2530000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.96499999999999997,"15min":1.1220000000000001},"min":{"1min":0.54300000000000004,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.1739999999999999,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":0.67500000000000004}]}]},{"osd":7,"up_from":56,"seq":240518169006,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2661960,"kb_used_data":2352448,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91083192,"statfs":{"total":95995035648,"available":93269188608,"internally_reserved":0,"allocated":2408906752,"data_stored":2408513732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":2.2909999999999999,"5min":3.4500000000000002,"15min":1.7769999999999999},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.252},"max":{"1min":7.4500000000000002,"5min":187.33699999999999,"15min":187.33699999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.9239999999999999,"5min":3.2709999999999999,"15min":1.7529999999999999},"min":{"1min":0.33700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.4989999999999997,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.0820000000000001,"15min":1.1040000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":35.655000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.9430000000000001,"5min":1.3129999999999999,"15min":1.2849999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.6040000000000001,"5min":7.6040000000000001,"15min":35.164000000000001},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:30:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.83999999999999997,"15min":1.1339999999999999},"min":{"1min":0.315,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.3339999999999996,"5min":4.5140000000000002,"15min":7.1689999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.0720000000000001,"15min":1.0189999999999999},"min":{"1min":0.52100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.9139999999999997,"5min":6.7549999999999999,"15min":6.7549999999999999},"last":0.68400000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.1659999999999999,"15min":1.0129999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":3.9580000000000002,"5min":6.899,"15min":7.5910000000000002},"last":1.587},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.1699999999999999,"15min":1.0649999999999999},"min":{"1min":0.40899999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.1539999999999999,"5min":6.9470000000000001,"15min":6.9470000000000001},"last":0.622}]},{"osd":5,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":14.285,"5min":3.7229999999999999,"15min":1.8959999999999999},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":216.84299999999999,"5min":216.84299999999999,"15min":216.84299999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":14.882,"5min":3.883,"15min":1.974},"min":{"1min":0.41399999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":217.02600000000001,"5min":217.02600000000001,"15min":217.02600000000001},"last":0.31}]},{"osd":6,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.1639999999999999,"15min":1.125},"min":{"1min":0.35799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.0270000000000001,"5min":6.319,"15min":6.319},"last":0.50700000000000001},{"interface":"front","average":{"1min":2.5379999999999998,"5min":1.484,"15min":1.2989999999999999},"min":{"1min":0.52900000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":6.3460000000000001,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":0.90600000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":6.5010000000000003,"15min":2.9279999999999999},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8839999999999999,"5min":498.68299999999999,"15min":498.68299999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":6.7460000000000004,"15min":3.0659999999999998},"min":{"1min":0.42199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":7.0110000000000001,"5min":498.69999999999999,"15min":498.69999999999999},"last":0.51800000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.018,"15min":1.1890000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.3719999999999999,"5min":4.8650000000000002,"15min":12.907999999999999},"last":1.623},{"interface":"front","average":{"1min":1.472,"5min":1.2709999999999999,"15min":1.323},"min":{"1min":0.58099999999999996,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.2810000000000001,"5min":7.1219999999999999,"15min":7.1219999999999999},"last":0.71399999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":7.9790000000000001,"15min":3.5739999999999998},"min":{"1min":0.51000000000000001,"5min":0.376,"15min":0.376},"max":{"1min":5.0839999999999996,"5min":497.52699999999999,"15min":497.52699999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":2.3540000000000001,"5min":8.0310000000000006,"15min":3.5169999999999999},"min":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":7.6470000000000002,"5min":496.29599999999999,"15min":496.29599999999999},"last":0.56000000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.0900000000000001,"15min":1.5940000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.4079999999999999,"5min":4.9039999999999999,"15min":120.943},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.3520000000000001,"15min":1.7490000000000001},"min":{"1min":0.66700000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":3.3279999999999998,"5min":7.0679999999999996,"15min":120.661},"last":0.76900000000000002}]}]},{"osd":6,"up_from":48,"seq":206158430646,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2695112,"kb_used_data":2385600,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91050040,"statfs":{"total":95995035648,"available":93235240960,"internally_reserved":0,"allocated":2442854400,"data_stored":2442461380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":2.1280000000000001,"15min":1.4750000000000001},"min":{"1min":0.32300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.0999999999999996,"5min":107.723,"15min":107.723},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.121,"5min":2.2149999999999999,"15min":1.544},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":6.1529999999999996,"5min":107.114,"15min":107.114},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.0489999999999999,"15min":1.9399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28199999999999997,"15min":0.27200000000000002},"max":{"1min":6.1870000000000003,"5min":6.1870000000000003,"15min":252.22800000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.2549999999999999,"5min":0.998,"15min":1.982},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.9459999999999997,"5min":6.6959999999999997,"15min":252.61199999999999},"last":0.53300000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.486,"15min":1.0800000000000001},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":6.2089999999999996,"5min":28.739999999999998,"15min":28.739999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.647,"15min":1.1930000000000001},"min":{"1min":0.34300000000000003,"5min":0.32300000000000001,"15min":0.28999999999999998},"max":{"1min":5.4169999999999998,"5min":32.328000000000003,"15min":32.328000000000003},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":1.3160000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.2,"15min":1.0409999999999999},"min":{"1min":0.40799999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.117,"5min":8.2260000000000009,"15min":8.2260000000000009},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.97699999999999998,"15min":1.0580000000000001},"min":{"1min":0.33400000000000002,"5min":0.28799999999999998,"15min":0.23999999999999999},"max":{"1min":6.0170000000000003,"5min":6.0259999999999998,"15min":7.5979999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.2090000000000001,"15min":1.0289999999999999},"min":{"1min":0.54100000000000004,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":5.4690000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.038,"15min":1.0269999999999999},"min":{"1min":0.41499999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.0720000000000001,"5min":10.52,"15min":10.52},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.9059999999999999,"5min":1.099,"15min":1.133},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":5.9939999999999998,"5min":5.9939999999999998,"15min":7.641},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.623,"5min":1.0129999999999999,"15min":1.0609999999999999},"min":{"1min":0.50900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":7.8380000000000001},"last":0.74299999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:30:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.2230000000000001,"15min":1.171},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.2190000000000003,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.0289999999999999,"15min":1.204},"min":{"1min":0.40999999999999998,"5min":0.39600000000000002,"15min":0.371},"max":{"1min":5.5140000000000002,"5min":6.2270000000000003,"15min":6.2270000000000003},"last":0.76000000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:30:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.2,"15min":1.25},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":6.2400000000000002,"5min":6.2400000000000002,"15min":18.059999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.23,"5min":1.0449999999999999,"15min":1.2509999999999999},"min":{"1min":0.56000000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.2729999999999997,"5min":6.2729999999999997,"15min":17.425000000000001},"last":0.89200000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:30:52 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.419,"15min":1.353},"min":{"1min":0.432,"5min":0.432,"15min":0.42799999999999999},"max":{"1min":8.6159999999999997,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":3.5859999999999999},{"interface":"front","average":{"1min":1.9990000000000001,"5min":1.5089999999999999,"15min":1.448},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":8.7100000000000009,"5min":8.7100000000000009,"15min":8.7100000000000009},"last":3.613}]},{"osd":11,"last update":"Mon Feb 12 16:30:52 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.391,"15min":1.7589999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":8.6649999999999991,"5min":8.6649999999999991,"15min":139.727},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.1120000000000001,"15min":1.651},"min":{"1min":0.61099999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":6.25,"5min":6.25,"15min":139.24199999999999},"last":0.84499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575324,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2674508,"kb_used_data":1780292,"kb_used_omap":0,"kb_used_meta":894208,"kb_avail":91070644,"statfs":{"total":95995035648,"available":93256339456,"internally_reserved":0,"allocated":1823019008,"data_stored":1822621916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915668992},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.1120000000000001,"15min":0.93700000000000006},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":4.3849999999999998,"5min":16.632999999999999,"15min":16.632999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.5269999999999999,"15min":1.2370000000000001},"min":{"1min":0.28199999999999997,"5min":0.27600000000000002,"15min":0.23899999999999999},"max":{"1min":6.4050000000000002,"5min":6.5759999999999996,"15min":11.433},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:30:29 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.3069999999999999,"15min":1.1459999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.248},"max":{"1min":5.7510000000000003,"5min":12.778,"15min":12.778},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":1.496,"15min":1.1950000000000001},"min":{"1min":0.316,"5min":0.246,"15min":0.246},"max":{"1min":2.3759999999999999,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:30:15 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.147,"15min":1.1240000000000001},"min":{"1min":0.39400000000000002,"5min":0.248,"15min":0.248},"max":{"1min":2.8719999999999999,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.454,"5min":1.2669999999999999,"15min":1.0009999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.9100000000000001,"5min":6.5490000000000004,"15min":6.7110000000000003},"last":0.68000000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.3979999999999999,"15min":1.1639999999999999},"min":{"1min":0.33700000000000002,"5min":0.31,"15min":0.31},"max":{"1min":7.7549999999999999,"5min":8.7479999999999993,"15min":8.7479999999999993},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.089,"5min":1.3109999999999999,"15min":1.353},"min":{"1min":0.38300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.5510000000000002,"5min":7.1710000000000003,"15min":7.1710000000000003},"last":0.42199999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:30:29 2024","interfaces":[{"interface":"back","average":{"1min":15.686,"5min":4.3440000000000003,"15min":2.2400000000000002},"min":{"1min":0.43099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":282.25999999999999,"5min":282.25999999999999,"15min":282.25999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":16.262,"5min":4.3710000000000004,"15min":2.2530000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":284.06200000000001,"5min":284.06200000000001,"15min":284.06200000000001},"last":0.38600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":29.050000000000001,"5min":7.1239999999999997,"15min":3.2509999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":522.79200000000003,"5min":522.79200000000003,"15min":522.79200000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":28.265999999999998,"5min":6.9619999999999997,"15min":3.234},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":520.64499999999998,"5min":520.64499999999998,"15min":520.64499999999998},"last":0.5}]},{"osd":8,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":7.3689999999999998,"15min":3.415},"min":{"1min":0.41699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.427,"5min":548.91600000000005,"15min":548.91600000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":7.4420000000000002,"15min":3.375},"min":{"1min":0.45200000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":3.512,"5min":552.42100000000005,"15min":552.42100000000005},"last":0.65100000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.204,"15min":1.3540000000000001},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.4950000000000001,"5min":5.7590000000000003,"15min":5.7590000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.2609999999999999,"15min":1.4199999999999999},"min":{"1min":0.505,"5min":0.436,"15min":0.436},"max":{"1min":2.6139999999999999,"5min":10.106,"15min":10.106},"last":0.63}]},{"osd":10,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":8.3170000000000002,"15min":3.738},"min":{"1min":0.53600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.5329999999999999,"5min":548.93499999999995,"15min":548.93499999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":8.2469999999999999,"15min":3.8109999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.46,"5min":549.12099999999998,"15min":549.12099999999998},"last":0.61599999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:30:29 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.4470000000000001,"15min":1.5609999999999999},"min":{"1min":0.52500000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.8959999999999999,"5min":10.202,"15min":10.202},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.431,"15min":1.6200000000000001},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.5720000000000001,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":0.66400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1761992,"kb_used_data":1176384,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":91983160,"statfs":{"total":95995035648,"available":94190755840,"internally_reserved":0,"allocated":1204617216,"data_stored":1204224196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.004,"15min":1.052},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":4.0949999999999998,"5min":5.5869999999999997,"15min":11.324},"last":2.9860000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.282,"15min":1.119},"min":{"1min":0.27300000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.4770000000000003,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.0329999999999999,"15min":1.0580000000000001},"min":{"1min":0.32900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":7.6059999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.173,"15min":1.014},"min":{"1min":0.32700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":5.9690000000000003,"5min":5.9690000000000003,"15min":5.9690000000000003},"last":2.8370000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":3.4580000000000002,"15min":1.873},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29599999999999999},"max":{"1min":4.5229999999999997,"5min":193.77000000000001,"15min":193.77000000000001},"last":0.31},{"interface":"front","average":{"1min":0.91700000000000004,"5min":3.3170000000000002,"15min":1.9339999999999999},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.2240000000000002,"5min":189.625,"15min":189.625},"last":0.29499999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.3660000000000001,"15min":1.0700000000000001},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":3.3700000000000001,"5min":10.49,"15min":10.49},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.161,"5min":1.3460000000000001,"15min":1.121},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.33600000000000002},"max":{"1min":3.6659999999999999,"5min":7.3250000000000002,"15min":7.3250000000000002},"last":0.497}]},{"osd":6,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":3.23,"15min":1.778},"min":{"1min":0.48899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":3.867,"5min":189.79400000000001,"15min":189.79400000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":3.27,"15min":1.8400000000000001},"min":{"1min":0.42099999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.2559999999999998,"5min":194.41200000000001,"15min":194.41200000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":5.3029999999999999,"15min":2.5659999999999998},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.374},"max":{"1min":184.39400000000001,"5min":191.47200000000001,"15min":191.47200000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":9.8190000000000008,"5min":5.4020000000000001,"15min":2.6749999999999998},"min":{"1min":0.443,"5min":0.44,"15min":0.44},"max":{"1min":184.43700000000001,"5min":191.50299999999999,"15min":191.50299999999999},"last":0.63500000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:30:23 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.3620000000000001,"15min":1.3109999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":3.4039999999999999,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.472,"15min":1.4079999999999999},"min":{"1min":0.52900000000000003,"5min":0.51400000000000001,"15min":0.51300000000000001},"max":{"1min":3.5299999999999998,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.76400000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.1619999999999999,"15min":1.367},"min":{"1min":0.497,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.73,"5min":5.6550000000000002,"15min":46.246000000000002},"last":0.371},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.4119999999999999,"15min":1.4630000000000001},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.432},"max":{"1min":3.4900000000000002,"5min":7.5369999999999999,"15min":46.167999999999999},"last":0.53700000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.4470000000000001,"15min":1.3600000000000001},"min":{"1min":0.46300000000000002,"5min":0.442,"15min":0.41999999999999998},"max":{"1min":4.6239999999999997,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.169,"5min":1.6060000000000001,"15min":1.4319999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.47999999999999998},"max":{"1min":8.4280000000000008,"5min":9.3490000000000002,"15min":9.3490000000000002},"last":0.58899999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:30:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.4570000000000001,"15min":1.5649999999999999},"min":{"1min":0.52300000000000002,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":3.96,"5min":6.2830000000000004,"15min":46.064},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.482,"15min":1.583},"min":{"1min":0.50700000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.5649999999999999,"5min":7.4210000000000003,"15min":45.642000000000003},"last":0.79400000000000004}]}]},{"osd":2,"up_from":21,"seq":90194313685,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892296,"kb_used_data":590976,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92852856,"statfs":{"total":95995035648,"available":95081324544,"internally_reserved":0,"allocated":605159424,"data_stored":604766404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.2849999999999999,"15min":1.1080000000000001},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.23599999999999999},"max":{"1min":4.7990000000000004,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":1.3140000000000001,"15min":1.222},"min":{"1min":0.255,"5min":0.247,"15min":0.247},"max":{"1min":4.774,"5min":6.484,"15min":9.6470000000000002},"last":0.29299999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:30:14 2024","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":1.494,"15min":1.837},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":26.745000000000001,"5min":26.745000000000001,"15min":26.745000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.982,"5min":1.371,"15min":1},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":25.260000000000002,"5min":25.260000000000002,"15min":25.260000000000002},"last":0.53600000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:30:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.2869999999999999,"15min":1.0840000000000001},"min":{"1min":0.37,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.8730000000000002,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.276,"15min":1.226},"min":{"1min":0.32300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.8809999999999998,"5min":6.601,"15min":9.8079999999999998},"last":0.438}]},{"osd":5,"last update":"Mon Feb 12 16:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.2909999999999999,"15min":1.1779999999999999},"min":{"1min":0.39600000000000002,"5min":0.38200000000000001,"15min":0.315},"max":{"1min":3.6520000000000001,"5min":6.133,"15min":6.133},"last":1.659},{"interface":"front","average":{"1min":1.014,"5min":1.6040000000000001,"15min":1.23},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.315},"max":{"1min":3.5790000000000002,"5min":11.081,"15min":11.081},"last":1.671}]},{"osd":6,"last update":"Mon Feb 12 16:31:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.2490000000000001,"15min":1.2829999999999999},"min":{"1min":0.39600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":4.2359999999999998,"5min":6.5279999999999996,"15min":21.527000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.2709999999999999,"15min":1.2250000000000001},"min":{"1min":0.40200000000000002,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":2.593,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.66200000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:30:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.548,"15min":1.2250000000000001},"min":{"1min":0.372,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.698,"5min":11.044,"15min":11.044},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.4039999999999999,"15min":1.2949999999999999},"min":{"1min":0.45000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.7080000000000002,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":0.61899999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:30:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.367,"15min":1.3149999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.9399999999999999,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":1.381,"15min":1.355},"min":{"1min":0.497,"5min":0.38,"15min":0.38},"max":{"1min":2.121,"5min":6.593,"15min":9.7639999999999993},"last":1.9970000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:30:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.4279999999999999,"15min":1.3220000000000001},"min":{"1min":0.45200000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.78,"5min":10.972,"15min":10.972},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.4990000000000001,"15min":1.4339999999999999},"min":{"1min":0.45600000000000002,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":1.825,"5min":11.006,"15min":11.006},"last":0.59399999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":1.53,"15min":1.4379999999999999},"min":{"1min":0.53500000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1439999999999999,"5min":7.7539999999999996,"15min":11.454000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.601,"15min":1.498},"min":{"1min":0.56599999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.113,"5min":7.7140000000000004,"15min":9.7170000000000005},"last":0.52300000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:31:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.349,"15min":1.4139999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.47799999999999998},"max":{"1min":1.1779999999999999,"5min":7.258,"15min":7.258},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.4330000000000001,"15min":1.4970000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.2719999999999998,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":0.72699999999999998}]}]},{"osd":3,"up_from":522,"seq":2241972928610,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3894088,"kb_used_data":3561856,"kb_used_omap":91,"kb_used_meta":332132,"kb_avail":89851064,"statfs":{"total":95995035648,"available":92007489536,"internally_reserved":0,"allocated":3647340544,"data_stored":3646947524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":340103184},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.405,"15min":1.0089999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.6659999999999999,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.4550000000000001,"15min":1.0469999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.8069999999999999,"5min":27.600000000000001,"15min":27.600000000000001},"last":1.48}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.236,"15min":0.98899999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.7480000000000002,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":0.38},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.1850000000000001,"15min":1.091},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.2709999999999999,"5min":6.5030000000000001,"15min":6.5030000000000001},"last":4.5819999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":0.88300000000000001,"15min":1.1919999999999999},"min":{"1min":0.33400000000000002,"5min":0.307,"15min":0.307},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":10.004},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.232,"5min":1.1339999999999999,"15min":1.1120000000000001},"min":{"1min":0.42299999999999999,"5min":0.35999999999999999,"15min":0.32600000000000001},"max":{"1min":3.3620000000000001,"5min":6.806,"15min":6.806},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.038,"15min":1.272},"min":{"1min":0.47399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":2.923,"5min":4.7530000000000001,"15min":6.7539999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.05,"15min":0.96699999999999997},"min":{"1min":0.40799999999999997,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":2.7970000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":2.3410000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.113,"15min":1.1299999999999999},"min":{"1min":0.53700000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.2889999999999997,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.1259999999999999,"15min":1.0369999999999999},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.359,"5min":6.1920000000000002,"15min":6.1920000000000002},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.077,"15min":1.347},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.8940000000000001,"5min":4.8090000000000002,"15min":6.851},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.27,"5min":1.25,"15min":1.2669999999999999},"min":{"1min":0.51700000000000002,"5min":0.40300000000000002,"15min":0.39200000000000002},"max":{"1min":3.1619999999999999,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":0.59199999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.0800000000000001,"15min":1.169},"min":{"1min":0.503,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.968,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.26,"5min":1.0760000000000001,"15min":1.107},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.39800000000000002},"max":{"1min":2.855,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":0.64900000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3009999999999999,"15min":1.329},"min":{"1min":0.52800000000000002,"5min":0.442,"15min":0.42799999999999999},"max":{"1min":3.2440000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.147,"5min":1.274,"15min":1.3560000000000001},"min":{"1min":0.56399999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":3.0369999999999999,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":1.419}]},{"osd":10,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.171,"15min":1.425},"min":{"1min":0.58599999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":3.0030000000000001,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":1.4530000000000001},{"interface":"front","average":{"1min":2.1150000000000002,"5min":1.4670000000000001,"15min":1.3740000000000001},"min":{"1min":0.55300000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":15.140000000000001,"5min":15.140000000000001,"15min":15.140000000000001},"last":0.71299999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:30:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.3200000000000001,"15min":1.298},"min":{"1min":0.61799999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.194,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.2829999999999999,"15min":1.464},"min":{"1min":0.57099999999999995,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":3.0760000000000001,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.84399999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953925,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2381448,"kb_used_data":1787648,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":91363704,"statfs":{"total":95995035648,"available":93556432896,"internally_reserved":0,"allocated":1830551552,"data_stored":1830158532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0149999999999999,"15min":1.054},"min":{"1min":0.248,"5min":0.248,"15min":0.23699999999999999},"max":{"1min":5.2889999999999997,"5min":7.492,"15min":7.6319999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.294,"15min":1.212},"min":{"1min":0.34899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.3220000000000001,"5min":12.236000000000001,"15min":12.236000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.246,"15min":1.056},"min":{"1min":0.29799999999999999,"5min":0.28399999999999997,"15min":0.24399999999999999},"max":{"1min":5.3920000000000003,"5min":7.3630000000000004,"15min":8.0530000000000008},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.212,"15min":1.153},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.254},"max":{"1min":3.8300000000000001,"5min":7.633,"15min":7.633},"last":2.9079999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1539999999999999,"15min":1.1080000000000001},"min":{"1min":0.40600000000000003,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":2.802,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.0549999999999999,"15min":1.0589999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.79,"5min":7.383,"15min":7.383},"last":2.944}]},{"osd":5,"last update":"Mon Feb 12 16:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.1659999999999999,"15min":1.0149999999999999},"min":{"1min":0.27200000000000002,"5min":0.25900000000000001,"15min":0.25700000000000001},"max":{"1min":3.7930000000000001,"5min":7.431,"15min":7.431},"last":2.855},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3080000000000001,"15min":1.179},"min":{"1min":0.38700000000000001,"5min":0.34599999999999997,"15min":0.33400000000000002},"max":{"1min":3.8879999999999999,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":2.976}]},{"osd":6,"last update":"Mon Feb 12 16:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":0.94899999999999995,"15min":1.0249999999999999},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.3710000000000004,"5min":7.5289999999999999,"15min":7.5410000000000004},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.165,"15min":1.1060000000000001},"min":{"1min":0.34999999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.1289999999999996,"5min":5.5190000000000001,"15min":9.6639999999999997},"last":2.9969999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.0720000000000001,"15min":1.1359999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.3440000000000003,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":2.9590000000000001},{"interface":"front","average":{"1min":1.897,"5min":1.1839999999999999,"15min":1.1990000000000001},"min":{"1min":0.55500000000000005,"5min":0.35499999999999998,"15min":0.34599999999999997},"max":{"1min":9.1440000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.57299999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.169,"15min":1.23},"min":{"1min":0.48299999999999998,"5min":0.39300000000000002,"15min":0.38300000000000001},"max":{"1min":5.4429999999999996,"5min":7.6890000000000001,"15min":7.8650000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.264,"5min":1.2769999999999999,"15min":1.2749999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":3.919,"5min":7.7110000000000003,"15min":7.7110000000000003},"last":3.0750000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.1759999999999999,"15min":1.3700000000000001},"min":{"1min":0.53400000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.4729999999999999,"5min":7.7750000000000004,"15min":34.112000000000002},"last":0.63},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3360000000000001,"15min":1.583},"min":{"1min":0.51100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":4.0060000000000002,"5min":7.827,"15min":34.590000000000003},"last":3.0419999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.429,"15min":1.4410000000000001},"min":{"1min":0.44,"5min":0.433,"15min":0.39200000000000002},"max":{"1min":3.976,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":3.105},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.093,"15min":1.2370000000000001},"min":{"1min":0.42299999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.964,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":3.0329999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:30:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.115,"15min":1.409},"min":{"1min":0.58499999999999996,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.1520000000000001,"5min":7.734,"15min":7.734},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.361,"15min":1.4710000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.1829999999999998,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":3.1320000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724989,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2668684,"kb_used_data":2367364,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91076468,"statfs":{"total":95995035648,"available":93262303232,"internally_reserved":0,"allocated":2424180736,"data_stored":2423783644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":3.27,"15min":3.0059999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.0640000000000001,"5min":221.96600000000001,"15min":565.34500000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":3.391,"15min":3.1030000000000002},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.29199999999999998},"max":{"1min":6.1029999999999998,"5min":222.06200000000001,"15min":565.98400000000004},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.2050000000000001,"15min":1.1779999999999999},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.0170000000000003,"5min":6.0709999999999997,"15min":40.021999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.23,"15min":1.236},"min":{"1min":0.372,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":11.448,"5min":11.448,"15min":40.225000000000001},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:30:26 2024","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":1.534,"15min":1.377},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":22.484000000000002,"5min":22.484000000000002,"15min":22.484000000000002},"last":1.089},{"interface":"front","average":{"1min":2.0739999999999998,"5min":1.268,"15min":0.90800000000000003},"min":{"1min":0.30299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":23.34,"5min":23.34,"15min":23.34},"last":0.5}]},{"osd":4,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.488,"15min":1.1819999999999999},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.9550000000000001,"5min":41.908000000000001,"15min":41.908000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.7629999999999999,"15min":1.2270000000000001},"min":{"1min":0.32400000000000001,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":4.9989999999999997,"5min":42.430999999999997,"15min":42.430999999999997},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:30:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":1.194,"15min":1.034},"min":{"1min":0.41299999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.9980000000000002,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.02,"5min":1.1659999999999999,"15min":1.1000000000000001},"min":{"1min":0.38700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.1629999999999998,"5min":9.5830000000000002,"15min":9.5830000000000002},"last":0.79500000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.1919999999999999,"15min":1.1060000000000001},"min":{"1min":0.41899999999999998,"5min":0.32000000000000001,"15min":0.28799999999999998},"max":{"1min":4.5570000000000004,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.4119999999999999,"15min":1.173},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.2560000000000002,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":0.92000000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":2.0950000000000002,"15min":1.528},"min":{"1min":0.42199999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":108.913,"5min":108.913,"15min":108.913},"last":0.61499999999999999},{"interface":"front","average":{"1min":5.633,"5min":2.1539999999999999,"15min":1.5369999999999999},"min":{"1min":0.51000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":108.93600000000001,"5min":108.93600000000001,"15min":108.93600000000001},"last":0.67100000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:30:15 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.4810000000000001,"15min":1.3480000000000001},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":2.0209999999999999,"5min":6.2939999999999996,"15min":6.2939999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.3180000000000001,"15min":1.282},"min":{"1min":0.53600000000000003,"5min":0.375,"15min":0.375},"max":{"1min":2.9420000000000002,"5min":9.6649999999999991,"15min":9.6649999999999991},"last":0.55200000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.835,"15min":1.494},"min":{"1min":0.48599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":6.1989999999999998,"5min":55.518999999999998,"15min":55.518999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":2.0819999999999999,"15min":1.5369999999999999},"min":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.0419999999999998,"5min":55.497999999999998,"15min":55.497999999999998},"last":1}]},{"osd":11,"last update":"Mon Feb 12 16:30:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8049999999999999,"5min":1.4610000000000001,"15min":1.401},"min":{"1min":0.48799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.2370000000000001,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.554,"15min":1.4179999999999999},"min":{"1min":0.58899999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":4.1630000000000003,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":1.0329999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1201930240,"data_stored":1201930240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1816399872,"data_stored":1816395800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":604110848,"data_stored":604110848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3639738368,"data_stored":3639738368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1825177600,"data_stored":1825177600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2411925504,"data_stored":2411921432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2432892928,"data_stored":2432892928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2391605248,"data_stored":2391605248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1237712896,"data_stored":1237712896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1219035136,"data_stored":1219035136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":596377600,"data_stored":596377600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:31:11.728 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T16:31:11.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set unique_pool_4 pg_num 26 2024-02-12T16:31:11.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:11.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:11.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:11.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:11.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:11.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:12.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:12.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:12.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:12.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:12.200 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:12.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:12.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:12.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:12.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:12.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:12.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:12.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:12.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:12.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:12.710 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:31:12.710 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:31:12.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:12.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:12.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:12.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:12.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:12.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:12.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:13.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:13.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:13.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:13.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:13.278 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:13.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:13.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:13.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:13 smithi169 bash[19604]: cluster 2024-02-12T16:31:12.055312+0000 mgr.y (mgr.14146) 1396 : cluster [DBG] pgmap v1946: 47 pgs: 47 active+clean; 8.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-02-12T16:31:13.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:13 smithi123 bash[19119]: cluster 2024-02-12T16:31:12.055312+0000 mgr.y (mgr.14146) 1396 : cluster [DBG] pgmap v1946: 47 pgs: 47 active+clean; 8.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-02-12T16:31:13.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:13 smithi121 bash[18489]: cluster 2024-02-12T16:31:12.055312+0000 mgr.y (mgr.14146) 1396 : cluster [DBG] pgmap v1946: 47 pgs: 47 active+clean; 8.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-02-12T16:31:13.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:13.534 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:13.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:31:13.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:13.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:13.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:13.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:13.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:13.987 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:13.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:14.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:14.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:14.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:14.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:14.236 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.302366+0000 mon.a (mon.0) 2338 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08417146191628573]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.302700+0000 mon.a (mon.0) 2339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08919413886208288]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.302950+0000 mon.a (mon.0) 2340 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08907834713277363]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.303237+0000 mon.a (mon.0) 2341 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09276905134279159]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.303519+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08865617662527901]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.303824+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08444061440373334]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.304149+0000 mon.a (mon.0) 2344 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08593040437391303]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.304404+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08532866113342863]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.304648+0000 mon.a (mon.0) 2346 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09182892692578389]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.304895+0000 mon.a (mon.0) 2347 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08657033654227161]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.305142+0000 mon.a (mon.0) 2348 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08521027300259604]}]: dispatch 2024-02-12T16:31:14.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:14 smithi169 bash[19604]: audit 2024-02-12T16:31:13.305386+0000 mon.a (mon.0) 2349 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08436799445781047]}]: dispatch 2024-02-12T16:31:14.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:14.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:14.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:14.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.302366+0000 mon.a (mon.0) 2338 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08417146191628573]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.302700+0000 mon.a (mon.0) 2339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08919413886208288]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.302950+0000 mon.a (mon.0) 2340 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08907834713277363]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.303237+0000 mon.a (mon.0) 2341 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09276905134279159]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.303519+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08865617662527901]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.303824+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08444061440373334]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.304149+0000 mon.a (mon.0) 2344 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08593040437391303]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.304404+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08532866113342863]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.304648+0000 mon.a (mon.0) 2346 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09182892692578389]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.304895+0000 mon.a (mon.0) 2347 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08657033654227161]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.305142+0000 mon.a (mon.0) 2348 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08521027300259604]}]: dispatch 2024-02-12T16:31:14.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:14 smithi121 bash[18489]: audit 2024-02-12T16:31:13.305386+0000 mon.a (mon.0) 2349 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08436799445781047]}]: dispatch 2024-02-12T16:31:14.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:14.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.302366+0000 mon.a (mon.0) 2338 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08417146191628573]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.302700+0000 mon.a (mon.0) 2339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08919413886208288]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.302950+0000 mon.a (mon.0) 2340 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08907834713277363]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.303237+0000 mon.a (mon.0) 2341 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09276905134279159]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.303519+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08865617662527901]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.303824+0000 mon.a (mon.0) 2343 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08444061440373334]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.304149+0000 mon.a (mon.0) 2344 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08593040437391303]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.304404+0000 mon.a (mon.0) 2345 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08532866113342863]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.304648+0000 mon.a (mon.0) 2346 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09182892692578389]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.304895+0000 mon.a (mon.0) 2347 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08657033654227161]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.305142+0000 mon.a (mon.0) 2348 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08521027300259604]}]: dispatch 2024-02-12T16:31:14.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:14 smithi123 bash[19119]: audit 2024-02-12T16:31:13.305386+0000 mon.a (mon.0) 2349 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08436799445781047]}]: dispatch 2024-02-12T16:31:14.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:14.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:14.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:14.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:14.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:14.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:14.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:14.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:14.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:14.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:14.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:14.931 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:14.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:15.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:15.070 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:15.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:15.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:15.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:15.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:15.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: cluster 2024-02-12T16:31:13.297253+0000 osd.7 (osd.7) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: cluster 2024-02-12T16:31:13.297306+0000 osd.7 (osd.7) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: cluster 2024-02-12T16:31:14.055770+0000 mgr.y (mgr.14146) 1397 : cluster [DBG] pgmap v1947: 47 pgs: 47 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118100+0000 mon.a (mon.0) 2350 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08417146191628573]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118231+0000 mon.a (mon.0) 2351 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08919413886208288]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118326+0000 mon.a (mon.0) 2352 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08907834713277363]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118385+0000 mon.a (mon.0) 2353 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09276905134279159]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118442+0000 mon.a (mon.0) 2354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08865617662527901]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118498+0000 mon.a (mon.0) 2355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08444061440373334]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118554+0000 mon.a (mon.0) 2356 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08593040437391303]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118609+0000 mon.a (mon.0) 2357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08532866113342863]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118750+0000 mon.a (mon.0) 2358 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09182892692578389]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118846+0000 mon.a (mon.0) 2359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08657033654227161]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.118957+0000 mon.a (mon.0) 2360 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08521027300259604]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: audit 2024-02-12T16:31:14.119065+0000 mon.a (mon.0) 2361 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08436799445781047]}]': finished 2024-02-12T16:31:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:15 smithi169 bash[19604]: cluster 2024-02-12T16:31:14.119119+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e670: 12 total, 12 up, 12 in 2024-02-12T16:31:15.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:15.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:15.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: cluster 2024-02-12T16:31:13.297253+0000 osd.7 (osd.7) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: cluster 2024-02-12T16:31:13.297306+0000 osd.7 (osd.7) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: cluster 2024-02-12T16:31:14.055770+0000 mgr.y (mgr.14146) 1397 : cluster [DBG] pgmap v1947: 47 pgs: 47 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-02-12T16:31:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118100+0000 mon.a (mon.0) 2350 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08417146191628573]}]': finished 2024-02-12T16:31:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118231+0000 mon.a (mon.0) 2351 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08919413886208288]}]': finished 2024-02-12T16:31:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118326+0000 mon.a (mon.0) 2352 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08907834713277363]}]': finished 2024-02-12T16:31:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118385+0000 mon.a (mon.0) 2353 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09276905134279159]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118442+0000 mon.a (mon.0) 2354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08865617662527901]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118498+0000 mon.a (mon.0) 2355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08444061440373334]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118554+0000 mon.a (mon.0) 2356 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08593040437391303]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118609+0000 mon.a (mon.0) 2357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08532866113342863]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118750+0000 mon.a (mon.0) 2358 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09182892692578389]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118846+0000 mon.a (mon.0) 2359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08657033654227161]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.118957+0000 mon.a (mon.0) 2360 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08521027300259604]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: audit 2024-02-12T16:31:14.119065+0000 mon.a (mon.0) 2361 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08436799445781047]}]': finished 2024-02-12T16:31:15.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:15 smithi123 bash[19119]: cluster 2024-02-12T16:31:14.119119+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e670: 12 total, 12 up, 12 in 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: cluster 2024-02-12T16:31:13.297253+0000 osd.7 (osd.7) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: cluster 2024-02-12T16:31:13.297306+0000 osd.7 (osd.7) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: cluster 2024-02-12T16:31:14.055770+0000 mgr.y (mgr.14146) 1397 : cluster [DBG] pgmap v1947: 47 pgs: 47 active+clean; 8.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118100+0000 mon.a (mon.0) 2350 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08417146191628573]}]': finished 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118231+0000 mon.a (mon.0) 2351 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08919413886208288]}]': finished 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118326+0000 mon.a (mon.0) 2352 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08907834713277363]}]': finished 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118385+0000 mon.a (mon.0) 2353 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09276905134279159]}]': finished 2024-02-12T16:31:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118442+0000 mon.a (mon.0) 2354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08865617662527901]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118498+0000 mon.a (mon.0) 2355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08444061440373334]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118554+0000 mon.a (mon.0) 2356 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08593040437391303]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118609+0000 mon.a (mon.0) 2357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08532866113342863]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118750+0000 mon.a (mon.0) 2358 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09182892692578389]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118846+0000 mon.a (mon.0) 2359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08657033654227161]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.118957+0000 mon.a (mon.0) 2360 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08521027300259604]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: audit 2024-02-12T16:31:14.119065+0000 mon.a (mon.0) 2361 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08436799445781047]}]': finished 2024-02-12T16:31:15.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:15 smithi121 bash[18489]: cluster 2024-02-12T16:31:14.119119+0000 mon.a (mon.0) 2362 : cluster [DBG] osdmap e670: 12 total, 12 up, 12 in 2024-02-12T16:31:15.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:15.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:15.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:15.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:15.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:15.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:15.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:15.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:15.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:15.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:15.901 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:15.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:16.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:16.044 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:16.118 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:16.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:16.185 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:16.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:16 smithi123 bash[19119]: cluster 2024-02-12T16:31:15.119794+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e671: 12 total, 12 up, 12 in 2024-02-12T16:31:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:16 smithi123 bash[19119]: cluster 2024-02-12T16:31:15.731684+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in 2024-02-12T16:31:16.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:16 smithi121 bash[18489]: cluster 2024-02-12T16:31:15.119794+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e671: 12 total, 12 up, 12 in 2024-02-12T16:31:16.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:16 smithi121 bash[18489]: cluster 2024-02-12T16:31:15.731684+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in 2024-02-12T16:31:16.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:16 smithi169 bash[19604]: cluster 2024-02-12T16:31:15.119794+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e671: 12 total, 12 up, 12 in 2024-02-12T16:31:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:16 smithi169 bash[19604]: cluster 2024-02-12T16:31:15.731684+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e672: 12 total, 12 up, 12 in 2024-02-12T16:31:16.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:16.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:16.715 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:16.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:16.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:16.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:17.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:17.107 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:17.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:17.135 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:17.135 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:17.138 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:17 smithi121 bash[18489]: cluster 2024-02-12T16:31:15.732500+0000 osd.7 (osd.7) 169 : cluster [DBG] 6.8 starting backfill to osd.11 from (0'0,0'0] MAX to 669'15525 2024-02-12T16:31:17.138 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:17 smithi121 bash[18489]: cluster 2024-02-12T16:31:16.032002+0000 osd.4 (osd.4) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:17.138 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:17 smithi121 bash[18489]: cluster 2024-02-12T16:31:16.032128+0000 osd.4 (osd.4) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:17.138 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:17 smithi121 bash[18489]: cluster 2024-02-12T16:31:16.056464+0000 mgr.y (mgr.14146) 1398 : cluster [DBG] pgmap v1951: 47 pgs: 47 active+clean; 8.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-12T16:31:17.138 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:17 smithi121 bash[18489]: cluster 2024-02-12T16:31:16.745217+0000 mon.a (mon.0) 2365 : cluster [DBG] osdmap e673: 12 total, 12 up, 12 in 2024-02-12T16:31:17.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:17.244 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:17.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:17.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:17.382 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:17.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:17.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:17.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:17.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:17 smithi169 bash[19604]: cluster 2024-02-12T16:31:15.732500+0000 osd.7 (osd.7) 169 : cluster [DBG] 6.8 starting backfill to osd.11 from (0'0,0'0] MAX to 669'15525 2024-02-12T16:31:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:17 smithi169 bash[19604]: cluster 2024-02-12T16:31:16.032002+0000 osd.4 (osd.4) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:17 smithi169 bash[19604]: cluster 2024-02-12T16:31:16.032128+0000 osd.4 (osd.4) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:17 smithi169 bash[19604]: cluster 2024-02-12T16:31:16.056464+0000 mgr.y (mgr.14146) 1398 : cluster [DBG] pgmap v1951: 47 pgs: 47 active+clean; 8.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-12T16:31:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:17 smithi169 bash[19604]: cluster 2024-02-12T16:31:16.745217+0000 mon.a (mon.0) 2365 : cluster [DBG] osdmap e673: 12 total, 12 up, 12 in 2024-02-12T16:31:17.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:17 smithi123 bash[19119]: cluster 2024-02-12T16:31:15.732500+0000 osd.7 (osd.7) 169 : cluster [DBG] 6.8 starting backfill to osd.11 from (0'0,0'0] MAX to 669'15525 2024-02-12T16:31:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:17 smithi123 bash[19119]: cluster 2024-02-12T16:31:16.032002+0000 osd.4 (osd.4) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:17 smithi123 bash[19119]: cluster 2024-02-12T16:31:16.032128+0000 osd.4 (osd.4) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:17 smithi123 bash[19119]: cluster 2024-02-12T16:31:16.056464+0000 mgr.y (mgr.14146) 1398 : cluster [DBG] pgmap v1951: 47 pgs: 47 active+clean; 8.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-12T16:31:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:17 smithi123 bash[19119]: cluster 2024-02-12T16:31:16.745217+0000 mon.a (mon.0) 2365 : cluster [DBG] osdmap e673: 12 total, 12 up, 12 in 2024-02-12T16:31:17.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:17.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:17.745 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:17.799 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:17.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:17.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:17.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:17.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:18.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:18.073 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:18.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:18.146 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:18.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:18.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:18.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:18.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:18.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:18.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:18.493 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:18.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:18.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:18.629 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:18.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:18.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:18.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:18.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:18.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:18.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:19.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:19.035 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:19.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:19.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:19 smithi169 bash[19604]: cluster 2024-02-12T16:31:18.057065+0000 mgr.y (mgr.14146) 1399 : cluster [DBG] pgmap v1953: 47 pgs: 47 active+clean; 8.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-02-12T16:31:19.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:19 smithi169 bash[19604]: cluster 2024-02-12T16:31:18.061573+0000 osd.1 (osd.1) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:19.143 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:19 smithi169 bash[19604]: cluster 2024-02-12T16:31:18.061624+0000 osd.1 (osd.1) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:19.144 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:19 smithi169 bash[19604]: cluster 2024-02-12T16:31:18.138812+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e674: 12 total, 12 up, 12 in 2024-02-12T16:31:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:19.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:19.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:19 smithi123 bash[19119]: cluster 2024-02-12T16:31:18.057065+0000 mgr.y (mgr.14146) 1399 : cluster [DBG] pgmap v1953: 47 pgs: 47 active+clean; 8.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-02-12T16:31:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:19 smithi123 bash[19119]: cluster 2024-02-12T16:31:18.061573+0000 osd.1 (osd.1) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:19 smithi123 bash[19119]: cluster 2024-02-12T16:31:18.061624+0000 osd.1 (osd.1) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:19 smithi123 bash[19119]: cluster 2024-02-12T16:31:18.138812+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e674: 12 total, 12 up, 12 in 2024-02-12T16:31:19.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:19.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:19 smithi121 bash[18489]: cluster 2024-02-12T16:31:18.057065+0000 mgr.y (mgr.14146) 1399 : cluster [DBG] pgmap v1953: 47 pgs: 47 active+clean; 8.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-02-12T16:31:19.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:19 smithi121 bash[18489]: cluster 2024-02-12T16:31:18.061573+0000 osd.1 (osd.1) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:19.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:19 smithi121 bash[18489]: cluster 2024-02-12T16:31:18.061624+0000 osd.1 (osd.1) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:19.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:19 smithi121 bash[18489]: cluster 2024-02-12T16:31:18.138812+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e674: 12 total, 12 up, 12 in 2024-02-12T16:31:19.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:19.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:19.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:19.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:19.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:19.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:19.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:19.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:19.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:19.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:20.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:20.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:20.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:20.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:20.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:20.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:20.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:20.366 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:20.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:20.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:20.494 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:20.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:20.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:20.650 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:20.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:20.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:20.778 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:20.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:20.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:20.908 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:20.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:21.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:21.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:21.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:21.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:21.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:21.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:21 smithi123 bash[19119]: cluster 2024-02-12T16:31:20.057557+0000 mgr.y (mgr.14146) 1400 : cluster [DBG] pgmap v1955: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 8533/276232 objects misplaced (3.089%); 324 KiB/s, 5 objects/s recovering 2024-02-12T16:31:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:21 smithi123 bash[19119]: cluster 2024-02-12T16:31:20.733243+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e675: 12 total, 12 up, 12 in 2024-02-12T16:31:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:21 smithi121 bash[18489]: cluster 2024-02-12T16:31:20.057557+0000 mgr.y (mgr.14146) 1400 : cluster [DBG] pgmap v1955: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 8533/276232 objects misplaced (3.089%); 324 KiB/s, 5 objects/s recovering 2024-02-12T16:31:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:21 smithi121 bash[18489]: cluster 2024-02-12T16:31:20.733243+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e675: 12 total, 12 up, 12 in 2024-02-12T16:31:21.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:21.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:21.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:21 smithi169 bash[19604]: cluster 2024-02-12T16:31:20.057557+0000 mgr.y (mgr.14146) 1400 : cluster [DBG] pgmap v1955: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 8533/276232 objects misplaced (3.089%); 324 KiB/s, 5 objects/s recovering 2024-02-12T16:31:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:21 smithi169 bash[19604]: cluster 2024-02-12T16:31:20.733243+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e675: 12 total, 12 up, 12 in 2024-02-12T16:31:21.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:21.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:21.695 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:21.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:21.828 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:21.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:21.978 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:21.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:22.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:22.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:22.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:22.248 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:22.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:22.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:22.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:22.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:22.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:22.518 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:22.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:22.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:22.647 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:22.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:22.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:22.800 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:22.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:22.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:22.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:22.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:23.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:23.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:23.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:23.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:23.224 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:23.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:23.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:23.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:23.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:23 smithi121 bash[18489]: cluster 2024-02-12T16:31:22.058035+0000 mgr.y (mgr.14146) 1401 : cluster [DBG] pgmap v1957: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 8533/271966 objects misplaced (3.138%); 267 KiB/s, 4 objects/s recovering 2024-02-12T16:31:23.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:23.492 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:23.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:23 smithi169 bash[19604]: cluster 2024-02-12T16:31:22.058035+0000 mgr.y (mgr.14146) 1401 : cluster [DBG] pgmap v1957: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 8533/271966 objects misplaced (3.138%); 267 KiB/s, 4 objects/s recovering 2024-02-12T16:31:23.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:23.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:23.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:23 smithi123 bash[19119]: cluster 2024-02-12T16:31:22.058035+0000 mgr.y (mgr.14146) 1401 : cluster [DBG] pgmap v1957: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 8533/271966 objects misplaced (3.138%); 267 KiB/s, 4 objects/s recovering 2024-02-12T16:31:23.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:23.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:23.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:23.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:23.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:23.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:24.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:24.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:24.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:24.154 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:24.192 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:24.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:24.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:24.333 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:24.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:24.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:24.462 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:24.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:24.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:24.598 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:24.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:24.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:24.725 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:24.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:24.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:24.859 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:24.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:24.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:25.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:25.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:25.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:25.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:25.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:25.363 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:25 smithi169 bash[19604]: cluster 2024-02-12T16:31:23.552135+0000 osd.11 (osd.11) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:25.363 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:25 smithi169 bash[19604]: cluster 2024-02-12T16:31:23.552196+0000 osd.11 (osd.11) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:25.363 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:25 smithi169 bash[19604]: cluster 2024-02-12T16:31:24.058485+0000 mgr.y (mgr.14146) 1402 : cluster [DBG] pgmap v1958: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 8174/267290 objects misplaced (3.058%); 726 KiB/s, 11 objects/s recovering 2024-02-12T16:31:25.363 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:25 smithi169 bash[19604]: cluster 2024-02-12T16:31:24.161558+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e676: 12 total, 12 up, 12 in 2024-02-12T16:31:25.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:25.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:25.494 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:25 smithi121 bash[18489]: cluster 2024-02-12T16:31:23.552135+0000 osd.11 (osd.11) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:25.494 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:25 smithi121 bash[18489]: cluster 2024-02-12T16:31:23.552196+0000 osd.11 (osd.11) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:25.494 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:25 smithi121 bash[18489]: cluster 2024-02-12T16:31:24.058485+0000 mgr.y (mgr.14146) 1402 : cluster [DBG] pgmap v1958: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 8174/267290 objects misplaced (3.058%); 726 KiB/s, 11 objects/s recovering 2024-02-12T16:31:25.494 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:25 smithi121 bash[18489]: cluster 2024-02-12T16:31:24.161558+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e676: 12 total, 12 up, 12 in 2024-02-12T16:31:25.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:25.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:25 smithi123 bash[19119]: cluster 2024-02-12T16:31:23.552135+0000 osd.11 (osd.11) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:25 smithi123 bash[19119]: cluster 2024-02-12T16:31:23.552196+0000 osd.11 (osd.11) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:25 smithi123 bash[19119]: cluster 2024-02-12T16:31:24.058485+0000 mgr.y (mgr.14146) 1402 : cluster [DBG] pgmap v1958: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 8174/267290 objects misplaced (3.058%); 726 KiB/s, 11 objects/s recovering 2024-02-12T16:31:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:25 smithi123 bash[19119]: cluster 2024-02-12T16:31:24.161558+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e676: 12 total, 12 up, 12 in 2024-02-12T16:31:25.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:25.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:25.790 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:31:25.791 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:25.817 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:31:25.847 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:31:25.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:31:25.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:25.903 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:31:25.926 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:25.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:25.938 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:31:25.968 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:31:26.007 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:31:26.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:26.033 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:31:26.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:31:26.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:26.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:26.097 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:31:26.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.139 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:31:26.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:26.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:26.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:26.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:26.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:26.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:26.482 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:26.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:26.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:26.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:26.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:26.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:26.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:26.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:26.755 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:31:26.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:26.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:26 smithi121 bash[18489]: cluster 2024-02-12T16:31:25.732970+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e677: 12 total, 12 up, 12 in 2024-02-12T16:31:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:26 smithi121 bash[18489]: audit 2024-02-12T16:31:25.761085+0000 mon.a (mon.0) 2370 : audit [DBG] from='client.? v1:172.21.15.121:0/3238561250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:26 smithi121 bash[18489]: cluster 2024-02-12T16:31:26.058881+0000 mgr.y (mgr.14146) 1403 : cluster [DBG] pgmap v1961: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 8174/263966 objects misplaced (3.097%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:31:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:26 smithi121 bash[18489]: audit 2024-02-12T16:31:26.506945+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? v1:172.21.15.121:0/4227921344' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:31:27.042 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:27.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:27.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:26 smithi169 bash[19604]: cluster 2024-02-12T16:31:25.732970+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e677: 12 total, 12 up, 12 in 2024-02-12T16:31:27.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:26 smithi169 bash[19604]: audit 2024-02-12T16:31:25.761085+0000 mon.a (mon.0) 2370 : audit [DBG] from='client.? v1:172.21.15.121:0/3238561250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:27.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:26 smithi169 bash[19604]: cluster 2024-02-12T16:31:26.058881+0000 mgr.y (mgr.14146) 1403 : cluster [DBG] pgmap v1961: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 8174/263966 objects misplaced (3.097%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:31:27.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:26 smithi169 bash[19604]: audit 2024-02-12T16:31:26.506945+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? v1:172.21.15.121:0/4227921344' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:31:27.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:26 smithi123 bash[19119]: cluster 2024-02-12T16:31:25.732970+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e677: 12 total, 12 up, 12 in 2024-02-12T16:31:27.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:26 smithi123 bash[19119]: audit 2024-02-12T16:31:25.761085+0000 mon.a (mon.0) 2370 : audit [DBG] from='client.? v1:172.21.15.121:0/3238561250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:27.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:26 smithi123 bash[19119]: cluster 2024-02-12T16:31:26.058881+0000 mgr.y (mgr.14146) 1403 : cluster [DBG] pgmap v1961: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 8.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 8174/263966 objects misplaced (3.097%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:31:27.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:26 smithi123 bash[19119]: audit 2024-02-12T16:31:26.506945+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? v1:172.21.15.121:0/4227921344' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:31:27.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:27.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:27.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:27.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:27.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:27.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:163266 5.c 5:3358d1c3:::benchmark_data_smithi169_84712_object163265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.316792+0000", 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.06649412000002, 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41050878600000001, 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163266 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.316792+0000", 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.316792+0000", 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.316790+0000", 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9188999999999999e-05 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.316849+0000", 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0342e-05 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.316859+0000", 2024-02-12T16:31:27.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3569999999999997e-06 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.316865+0000", 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017210249999999999 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.318586+0000", 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011827799999999999 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.318704+0000", 2024-02-12T16:31:27.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024218899999999999 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.318946+0000", 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011285223 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330231+0000", 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39698916000000001 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727221+0000", 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5401000000000001e-05 2024-02-12T16:31:27.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727266+0000", 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4801000000000001e-05 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727301+0000", 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41050878600000001 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:163283 5.0 5:03924079:::benchmark_data_smithi169_84712_object163282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:31:27.396 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.330305+0000", 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.05298098600002, 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.396550556, 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163283 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330305+0000", 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330305+0000", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330303+0000", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6774999999999997e-05 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330369+0000", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7380000000000006e-06 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330379+0000", 2024-02-12T16:31:27.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.964e-06 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330385+0000", 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0920000000000001e-05 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330426+0000", 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012696699999999999 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330553+0000", 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102122 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.330655+0000", 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39601703599999999 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.726672+0000", 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2321999999999998e-05 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.726724+0000", 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0829999999999999e-05 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.726785+0000", 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0532999999999997e-05 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.726856+0000", 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.396550556 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:163285 5.c 5:3e9669d5:::benchmark_data_smithi169_84712_object163284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.333508+0000", 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.04977782100002, 2024-02-12T16:31:27.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39394579299999999, 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163285 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333508+0000", 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333508+0000", 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:31:27.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333505+0000", 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7117000000000001e-05 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333572+0000", 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2863e-05 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333585+0000", 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9570000000000008e-06 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333594+0000", 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1894000000000002e-05 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333646+0000", 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013183499999999999 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333778+0000", 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018924499999999999 2024-02-12T16:31:27.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.333967+0000", 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39339477299999998 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727362+0000", 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0142e-05 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727372+0000", 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3899000000000001e-05 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727416+0000", 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8099000000000002e-05 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727454+0000", 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39394579299999999 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "rep_scrubmap(5.c e547 from shard 4)", 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.334271+0000", 2024-02-12T16:31:27.406 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.04901488199999, 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39290132799999999, 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.334271+0000", 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.334271+0000", 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.334270+0000", 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2140000000000001e-05 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.334282+0000", 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2090000000000005e-06 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.334290+0000", 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5539999999999999e-06 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.334295+0000", 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39282072800000001 2024-02-12T16:31:27.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727115+0000", 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.968e-06 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727122+0000", 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0238999999999998e-05 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727173+0000", 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39290132799999999 2024-02-12T16:31:27.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:163287 5.0 5:00b22bb4:::benchmark_data_smithi169_84712_object163286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.335075+0000", 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.04821096500001, 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39214079400000001, 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163287 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335075+0000", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335075+0000", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335073+0000", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7278999999999998e-05 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335131+0000", 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0708e-05 2024-02-12T16:31:27.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335141+0000", 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5739999999999996e-06 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335150+0000", 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090904899999999999 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.336059+0000", 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012397999999999999 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.336183+0000", 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015649300000000001 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.336339+0000", 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39033915800000002 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.726679+0000", 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045598399999999999 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727135+0000", 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9472999999999999e-05 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727184+0000", 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2005000000000003e-05 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727216+0000", 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39214079400000001 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.414 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16095.0:163266 5.c e578/547 ondisk, result = 0)", 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.335097+0000", 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.04818906200001, 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39221314299999999, 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335097+0000", 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335097+0000", 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335096+0000", 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4140000000000002e-06 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335101+0000", 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4619999999999999e-06 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335109+0000", 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.194e-06 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335113+0000", 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.392093519 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727206+0000", 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2790000000000001e-06 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727214+0000", 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6497999999999997e-05 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727310+0000", 2024-02-12T16:31:27.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39221314299999999 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:163289 5.0 5:09ebde68:::benchmark_data_smithi169_84712_object163288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.335979+0000", 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.04730726700001, 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.393605751, 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 163289 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335979+0000", 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335979+0000", 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.335977+0000", 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8367000000000002e-05 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.336076+0000", 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0781000000000001e-05 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.336087+0000", 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4339999999999992e-06 2024-02-12T16:31:27.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.336095+0000", 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38349497300000002 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.719590+0000", 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00034337899999999999 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.719933+0000", 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046979999999999998 2024-02-12T16:31:27.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.720403+0000", 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066175649999999997 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727021+0000", 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002486041 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.729507+0000", 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4399e-05 2024-02-12T16:31:27.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.729551+0000", 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3426000000000002e-05 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.729585+0000", 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.393605751 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16095.0:163285 5.c e578/547 ondisk, result = 0)", 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.339662+0000", 2024-02-12T16:31:27.423 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.04362437600003, 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38779838, 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.339662+0000", 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.339662+0000", 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.339661+0000", 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1889999999999998e-06 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.339666+0000", 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9129999999999992e-06 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.339675+0000", 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.276e-06 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.339680+0000", 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38766673800000001 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727347+0000", 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.401e-06 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727354+0000", 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105711 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.727460+0000", 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38779838 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16095.0:163294 5.5 e578/547 5:a433fdd0:::benchmark_data_smithi169_84712_object163293:head v 578'9935, mlcod=578'9932)", 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:25:48.343905+0000", 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 339.03938130900002, 2024-02-12T16:31:27.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40083268399999999, 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.343905+0000", 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.343905+0000", 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.343903+0000", 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8790999999999999e-05 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.343932+0000", 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.37538469400000002 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.719316+0000", 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1738000000000001e-05 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.719338+0000", 2024-02-12T16:31:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003820983 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.723159+0000", 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1469999999999999e-05 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.723181+0000", 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021345221000000001 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.744526+0000", 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021163700000000001 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:25:48.744738+0000", 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40083268399999999 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:265693 5.0 5:069ffb0a:::benchmark_data_smithi169_84712_object96303:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:26.871886+0000", 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 240.511400251, 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38030176999999998, 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 265693 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.871886+0000", 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.871886+0000", 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.871885+0000", 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3430000000000003e-06 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.871890+0000", 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4160000000000004e-06 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.871898+0000", 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7440000000000004e-06 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.871904+0000", 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050306719999999999 2024-02-12T16:31:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.876935+0000", 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117334 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.877052+0000", 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013961 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-02-12T16:31:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.877192+0000", 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0099807769999999997 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.887173+0000", 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3727000000000003e-05 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.887216+0000", 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.364796958 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:27.252013+0000", 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7102999999999999e-05 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:27.252070+0000", 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117376 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:27.252188+0000", 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38030176999999998 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:265705 5.0 5:02a78a1f:::benchmark_data_smithi169_84712_object96315:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:26.881255+0000", 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 240.50203082300001, 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38910847700000001, 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 265705 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.881255+0000", 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.881255+0000", 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.881253+0000", 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0069999999999997e-06 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.881261+0000", 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1382e-05 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.881272+0000", 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7279999999999998e-06 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.881279+0000", 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074963000000000004 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.882028+0000", 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011071899999999999 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.882139+0000", 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013621299999999999 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.882275+0000", 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049349509999999999 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.887210+0000", 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6041000000000001e-05 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:26.887226+0000", 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38303090899999997 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:27.270257+0000", 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8553000000000002e-05 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:27.270296+0000", 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8137000000000002e-05 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:27.270364+0000", 2024-02-12T16:31:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38910847700000001 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:296287 5.0 5:0db03e57:::benchmark_data_smithi169_84712_object126897:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.109630+0000", 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.273655859, 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44872978699999999, 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 296287 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109630+0000", 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109630+0000", 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109628+0000", 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.542e-06 2024-02-12T16:31:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109636+0000", 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0864e-05 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109646+0000", 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.037e-06 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109653+0000", 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7019000000000002e-05 2024-02-12T16:31:27.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109740+0000", 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116869 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109856+0000", 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013714599999999999 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109994+0000", 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014512963 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124506+0000", 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9371e-05 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124546+0000", 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43364284600000003 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558189+0000", 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0643000000000002e-05 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558249+0000", 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011073699999999999 2024-02-12T16:31:27.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558360+0000", 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44872978699999999 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:296288 5.0 5:0bf75f89:::benchmark_data_smithi169_84712_object126898:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.109682+0000", 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.27360379800001, 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45432622499999997, 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 296288 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109682+0000", 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109682+0000", 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109681+0000", 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0119999999999999e-06 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109685+0000", 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.558e-06 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109691+0000", 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8909999999999997e-06 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.109695+0000", 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052355370000000002 2024-02-12T16:31:27.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.114930+0000", 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000230652 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.115161+0000", 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000159307 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.115320+0000", 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44285537400000002 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558176+0000", 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028091500000000002 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558456+0000", 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053900670000000001 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.563847+0000", 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9667999999999999e-05 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.563906+0000", 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010238600000000001 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.564009+0000", 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45432622499999997 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.452 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:296302 5.c 5:35b32923:::benchmark_data_smithi169_84712_object126912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.118837+0000", 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.264449259, 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43898394899999998, 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 296302 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.118837+0000", 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.118837+0000", 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.118835+0000", 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9379999999999992e-06 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.118843+0000", 2024-02-12T16:31:27.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1948e-05 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.118854+0000", 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7240000000000003e-06 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.118861+0000", 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5108000000000002e-05 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.118906+0000", 2024-02-12T16:31:27.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111617 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119018+0000", 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013432799999999999 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119152+0000", 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004688617 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.123841+0000", 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43374882799999998 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557590+0000", 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012216199999999999 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557712+0000", 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010897599999999999 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557821+0000", 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43898394899999998 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16095.0:296288 5.0 e610/581 ondisk, result = 0)", 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.119508+0000", 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.263777842, 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.438856998, 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.458 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119508+0000", 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119508+0000", 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119507+0000", 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1349999999999992e-06 2024-02-12T16:31:27.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119515+0000", 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9789999999999993e-06 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119523+0000", 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7180000000000001e-06 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.119529+0000", 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43863014500000003 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558159+0000", 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9670000000000007e-06 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558168+0000", 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019759899999999999 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558365+0000", 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.438856998 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16095.0:296299 5.5 e610/547 5:abf2070f:::benchmark_data_smithi169_84712_object126909:head v 610'18019, mlcod=610'18016)", 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.120522+0000", 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.262764207, 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43701700999999998, 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120522+0000", 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120522+0000", 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120521+0000", 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4489999999999999e-06 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120529+0000", 2024-02-12T16:31:27.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4160000000000004e-06 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120537+0000", 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1540000000000001e-06 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120543+0000", 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9209999999999999e-05 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120582+0000", 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1083000000000001e-05 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.120603+0000", 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43686509800000001 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557468+0000", 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0939999999999995e-05 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557539+0000", 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43701700999999998 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16095.0:296288 5.0 e610/581 ondisk, result = 0)", 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.123985+0000", 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.25930098399999, 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43447960400000002, 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.123985+0000", 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.123985+0000", 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.123983+0000", 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0129999999999996e-06 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.123991+0000", 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8859999999999999e-06 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124000+0000", 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9499999999999998e-06 2024-02-12T16:31:27.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124006+0000", 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43442235400000001 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558429+0000", 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5443000000000001e-05 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558444+0000", 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0647000000000001e-05 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558465+0000", 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43447960400000002 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.16095.0:296302 5.c e610/547 ondisk, result = 0)", 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.124660+0000", 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.25862648, 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43317906299999998, 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124660+0000", 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124660+0000", 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124658+0000", 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3199999999999996e-06 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.124665+0000", 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43279536099999999 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557460+0000", 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557467+0000", 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1946000000000005e-05 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557549+0000", 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0849999999999999e-05 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557570+0000", 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026915100000000003 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557839+0000", 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43317906299999998 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.16095.0:296315 5.5 e610/547 5:ae0dd496:::benchmark_data_smithi169_84712_object126925:head v 610'18020, mlcod=610'18017)", 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.131149+0000", 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.252136772, 2024-02-12T16:31:27.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44967448199999999, 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.131149+0000", 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.131149+0000", 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.131148+0000", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6470000000000004e-06 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.131156+0000", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42628779 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557444+0000", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5959999999999993e-06 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557454+0000", 2024-02-12T16:31:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151094 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557605+0000", 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9536999999999997e-05 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557654+0000", 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023141947 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.580796+0000", 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7678000000000002e-05 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.580824+0000", 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44967448199999999 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.16095.0:296323 5.0 5:03ca451f:::benchmark_data_smithi169_84712_object126933:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:27:56.139654+0000", 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 211.243632244, 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43310763699999999, 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:31:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.16095", 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 296323 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.139654+0000", 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.139654+0000", 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.139651+0000", 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0669999999999999e-05 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.139661+0000", 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41773711200000002 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557398+0000", 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0899999999999998e-05 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.557439+0000", 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010480299999999999 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558487+0000", 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110221 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558598+0000", 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013843899999999999 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,10", 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.558736+0000", 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006686783 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.565423+0000", 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0064872610000000002 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.571910+0000", 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073875600000000003 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.572649+0000", 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4236000000000002e-05 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.572683+0000", 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8442999999999994e-05 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:27:56.572762+0000", 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43310763699999999 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:31:27.482 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:31:27.482 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:31:27.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:27.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:27.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:27.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:27.673 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:27.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:27.759 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 6 pg_num to 26 2024-02-12T16:31:27.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:27.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:27.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:27.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:27 smithi123 bash[19119]: cluster 2024-02-12T16:31:26.736940+0000 mon.a (mon.0) 2372 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:31:27.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:27 smithi123 bash[19119]: audit 2024-02-12T16:31:26.747156+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? v1:172.21.15.121:0/4227921344' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:31:27.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:27 smithi123 bash[19119]: cluster 2024-02-12T16:31:26.747330+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e678: 12 total, 12 up, 12 in 2024-02-12T16:31:27.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:27 smithi123 bash[19119]: audit 2024-02-12T16:31:26.886192+0000 mon.a (mon.0) 2375 : audit [INF] from='client.26350 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:31:27.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:27 smithi123 bash[19119]: audit 2024-02-12T16:31:26.886794+0000 mon.c (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.121:0/2686011688' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:31:27.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:27.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:28.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:28.042 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:27 smithi121 bash[18489]: cluster 2024-02-12T16:31:26.736940+0000 mon.a (mon.0) 2372 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:31:28.042 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:27 smithi121 bash[18489]: audit 2024-02-12T16:31:26.747156+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? v1:172.21.15.121:0/4227921344' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:31:28.042 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:27 smithi121 bash[18489]: cluster 2024-02-12T16:31:26.747330+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e678: 12 total, 12 up, 12 in 2024-02-12T16:31:28.042 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:27 smithi121 bash[18489]: audit 2024-02-12T16:31:26.886192+0000 mon.a (mon.0) 2375 : audit [INF] from='client.26350 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:31:28.042 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:27 smithi121 bash[18489]: audit 2024-02-12T16:31:26.886794+0000 mon.c (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.121:0/2686011688' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:31:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:27 smithi169 bash[19604]: cluster 2024-02-12T16:31:26.736940+0000 mon.a (mon.0) 2372 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:31:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:27 smithi169 bash[19604]: audit 2024-02-12T16:31:26.747156+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? v1:172.21.15.121:0/4227921344' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:31:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:27 smithi169 bash[19604]: cluster 2024-02-12T16:31:26.747330+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e678: 12 total, 12 up, 12 in 2024-02-12T16:31:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:27 smithi169 bash[19604]: audit 2024-02-12T16:31:26.886192+0000 mon.a (mon.0) 2375 : audit [INF] from='client.26350 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:31:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:27 smithi169 bash[19604]: audit 2024-02-12T16:31:26.886794+0000 mon.c (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.121:0/2686011688' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:31:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:28.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:28.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:28.190 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:28.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:28.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:28.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:28.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:28.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:28.456 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:28.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:28.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:28.594 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:28.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:28.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:28.739 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:28.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:28.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:28.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: cluster 2024-02-12T16:31:27.333549+0000 osd.6 (osd.6) 126 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:31:28.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: cluster 2024-02-12T16:31:27.413924+0000 osd.6 (osd.6) 127 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:31:28.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: audit 2024-02-12T16:31:27.748068+0000 mon.a (mon.0) 2376 : audit [INF] from='client.26350 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:31:28.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: cluster 2024-02-12T16:31:27.748132+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e679: 12 total, 12 up, 12 in 2024-02-12T16:31:28.850 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.059299+0000 mgr.y (mgr.14146) 1404 : cluster [DBG] pgmap v1964: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 7.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s; 8174/259326 objects misplaced (3.152%) 2024-02-12T16:31:28.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: audit 2024-02-12T16:31:28.060200+0000 mon.a (mon.0) 2378 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:31:28.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.077009+0000 osd.5 (osd.5) 121 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:31:28.851 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:28 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.087206+0000 osd.5 (osd.5) 122 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:31:28.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:28.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: cluster 2024-02-12T16:31:27.333549+0000 osd.6 (osd.6) 126 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: cluster 2024-02-12T16:31:27.413924+0000 osd.6 (osd.6) 127 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: audit 2024-02-12T16:31:27.748068+0000 mon.a (mon.0) 2376 : audit [INF] from='client.26350 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: cluster 2024-02-12T16:31:27.748132+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e679: 12 total, 12 up, 12 in 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.059299+0000 mgr.y (mgr.14146) 1404 : cluster [DBG] pgmap v1964: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 7.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s; 8174/259326 objects misplaced (3.152%) 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: audit 2024-02-12T16:31:28.060200+0000 mon.a (mon.0) 2378 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.077009+0000 osd.5 (osd.5) 121 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:31:28.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:28 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.087206+0000 osd.5 (osd.5) 122 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:31:28.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: cluster 2024-02-12T16:31:27.333549+0000 osd.6 (osd.6) 126 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: cluster 2024-02-12T16:31:27.413924+0000 osd.6 (osd.6) 127 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: audit 2024-02-12T16:31:27.748068+0000 mon.a (mon.0) 2376 : audit [INF] from='client.26350 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: cluster 2024-02-12T16:31:27.748132+0000 mon.a (mon.0) 2377 : cluster [DBG] osdmap e679: 12 total, 12 up, 12 in 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.059299+0000 mgr.y (mgr.14146) 1404 : cluster [DBG] pgmap v1964: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 7.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s; 8174/259326 objects misplaced (3.152%) 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: audit 2024-02-12T16:31:28.060200+0000 mon.a (mon.0) 2378 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.077009+0000 osd.5 (osd.5) 121 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:31:29.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:28 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.087206+0000 osd.5 (osd.5) 122 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:31:29.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:29.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:29.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:29.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:29.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:29.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:29.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:29.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:29.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:31:29.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:29.647 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:29.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:29.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:29.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:29.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:29.892 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:29.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.517027+0000 osd.8 (osd.8) 67 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.598512+0000 osd.8 (osd.8) 68 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.750536+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: audit 2024-02-12T16:31:28.752511+0000 mon.a (mon.0) 2379 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.752574+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e680: 12 total, 12 up, 12 in 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:28.976514+0000 osd.9 (osd.9) 117 : cluster [DBG] 1.a scrub starts 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.008668+0000 osd.9 (osd.9) 118 : cluster [DBG] 1.a scrub ok 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.057965+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub starts 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.096737+0000 osd.5 (osd.5) 123 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.124569+0000 osd.5 (osd.5) 124 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:31:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:29 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.276887+0000 osd.11 (osd.11) 154 : cluster [DBG] 1.b scrub starts 2024-02-12T16:31:29.916 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:29.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:30.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:30.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.517027+0000 osd.8 (osd.8) 67 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:31:30.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.598512+0000 osd.8 (osd.8) 68 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:31:30.033 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.750536+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:31:30.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: audit 2024-02-12T16:31:28.752511+0000 mon.a (mon.0) 2379 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:31:30.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.752574+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e680: 12 total, 12 up, 12 in 2024-02-12T16:31:30.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:28.976514+0000 osd.9 (osd.9) 117 : cluster [DBG] 1.a scrub starts 2024-02-12T16:31:30.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.008668+0000 osd.9 (osd.9) 118 : cluster [DBG] 1.a scrub ok 2024-02-12T16:31:30.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.057965+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub starts 2024-02-12T16:31:30.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.096737+0000 osd.5 (osd.5) 123 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:31:30.034 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.124569+0000 osd.5 (osd.5) 124 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:31:30.035 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:29 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.276887+0000 osd.11 (osd.11) 154 : cluster [DBG] 1.b scrub starts 2024-02-12T16:31:30.054 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:30.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:30.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.517027+0000 osd.8 (osd.8) 67 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:31:30.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.598512+0000 osd.8 (osd.8) 68 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:31:30.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.750536+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.9 scrub starts 2024-02-12T16:31:30.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: audit 2024-02-12T16:31:28.752511+0000 mon.a (mon.0) 2379 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:31:30.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.752574+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e680: 12 total, 12 up, 12 in 2024-02-12T16:31:30.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:28.976514+0000 osd.9 (osd.9) 117 : cluster [DBG] 1.a scrub starts 2024-02-12T16:31:30.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.008668+0000 osd.9 (osd.9) 118 : cluster [DBG] 1.a scrub ok 2024-02-12T16:31:30.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.057965+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub starts 2024-02-12T16:31:30.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.096737+0000 osd.5 (osd.5) 123 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:31:30.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.124569+0000 osd.5 (osd.5) 124 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:31:30.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:29 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.276887+0000 osd.11 (osd.11) 154 : cluster [DBG] 1.b scrub starts 2024-02-12T16:31:30.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:30.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:30.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:30.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:30.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:30.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:30.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:30.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:30.447 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:30.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:30.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:30.581 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:30.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:30.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:31:30.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:30.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:30.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:30.779 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.343607+0000 osd.6 (osd.6) 128 : cluster [DBG] 1.f scrub starts 2024-02-12T16:31:30.780 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.396868+0000 osd.11 (osd.11) 155 : cluster [DBG] 1.b scrub ok 2024-02-12T16:31:30.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:30.837 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.343607+0000 osd.6 (osd.6) 128 : cluster [DBG] 1.f scrub starts 2024-02-12T16:31:30.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.396868+0000 osd.11 (osd.11) 155 : cluster [DBG] 1.b scrub ok 2024-02-12T16:31:30.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.414169+0000 osd.6 (osd.6) 129 : cluster [DBG] 1.f scrub ok 2024-02-12T16:31:30.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.502818+0000 osd.8 (osd.8) 69 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:31:30.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.506835+0000 osd.8 (osd.8) 70 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:31:30.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.549243+0000 osd.10 (osd.10) 136 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:31:30.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.596274+0000 osd.10 (osd.10) 137 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:31:30.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.770773+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e681: 12 total, 12 up, 12 in 2024-02-12T16:31:30.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.777135+0000 osd.7 (osd.7) 170 : cluster [INF] 6.8 continuing backfill to osd.11 from (655'6203,679'16298] MIN to 679'16298 2024-02-12T16:31:30.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:29.931073+0000 osd.1 (osd.1) 100 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:31:30.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.059817+0000 mgr.y (mgr.14146) 1405 : cluster [DBG] pgmap v1967: 57 pgs: 2 peering, 10 unknown, 1 active+remapped+backfilling, 44 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 7811/239598 objects misplaced (3.260%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:31:30.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.111016+0000 osd.1 (osd.1) 101 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:31:30.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.420112+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.b scrub ok 2024-02-12T16:31:30.839 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:30 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.739000+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e682: 12 total, 12 up, 12 in 2024-02-12T16:31:30.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:30.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:30.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:30.977 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.343607+0000 osd.6 (osd.6) 128 : cluster [DBG] 1.f scrub starts 2024-02-12T16:31:30.977 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.396868+0000 osd.11 (osd.11) 155 : cluster [DBG] 1.b scrub ok 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.414169+0000 osd.6 (osd.6) 129 : cluster [DBG] 1.f scrub ok 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.502818+0000 osd.8 (osd.8) 69 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.506835+0000 osd.8 (osd.8) 70 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.549243+0000 osd.10 (osd.10) 136 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.596274+0000 osd.10 (osd.10) 137 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.770773+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e681: 12 total, 12 up, 12 in 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.777135+0000 osd.7 (osd.7) 170 : cluster [INF] 6.8 continuing backfill to osd.11 from (655'6203,679'16298] MIN to 679'16298 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:29.931073+0000 osd.1 (osd.1) 100 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.059817+0000 mgr.y (mgr.14146) 1405 : cluster [DBG] pgmap v1967: 57 pgs: 2 peering, 10 unknown, 1 active+remapped+backfilling, 44 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 7811/239598 objects misplaced (3.260%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.111016+0000 osd.1 (osd.1) 101 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.420112+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.b scrub ok 2024-02-12T16:31:30.978 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:30 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.739000+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e682: 12 total, 12 up, 12 in 2024-02-12T16:31:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:31.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:31.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.414169+0000 osd.6 (osd.6) 129 : cluster [DBG] 1.f scrub ok 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.502818+0000 osd.8 (osd.8) 69 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.506835+0000 osd.8 (osd.8) 70 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.549243+0000 osd.10 (osd.10) 136 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.596274+0000 osd.10 (osd.10) 137 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.770773+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e681: 12 total, 12 up, 12 in 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.777135+0000 osd.7 (osd.7) 170 : cluster [INF] 6.8 continuing backfill to osd.11 from (655'6203,679'16298] MIN to 679'16298 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:29.931073+0000 osd.1 (osd.1) 100 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.059817+0000 mgr.y (mgr.14146) 1405 : cluster [DBG] pgmap v1967: 57 pgs: 2 peering, 10 unknown, 1 active+remapped+backfilling, 44 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 7811/239598 objects misplaced (3.260%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.111016+0000 osd.1 (osd.1) 101 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.420112+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.b scrub ok 2024-02-12T16:31:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:30 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.739000+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e682: 12 total, 12 up, 12 in 2024-02-12T16:31:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:31.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:31:31.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:31.276 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:31.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:31.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:31.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:31.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:31.690 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:31.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:31.826 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:31.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:31.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:31.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.521998+0000 osd.10 (osd.10) 138 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:31:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.561067+0000 osd.10 (osd.10) 139 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:31:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.591519+0000 osd.7 (osd.7) 171 : cluster [DBG] 1.d scrub starts 2024-02-12T16:31:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.659468+0000 osd.7 (osd.7) 172 : cluster [DBG] 1.d scrub ok 2024-02-12T16:31:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.828318+0000 osd.0 (osd.0) 111 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:31:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.886617+0000 osd.0 (osd.0) 112 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:31:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.889225+0000 osd.4 (osd.4) 118 : cluster [DBG] 1.e scrub starts 2024-02-12T16:31:32.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:30.982704+0000 osd.4 (osd.4) 119 : cluster [DBG] 1.e scrub ok 2024-02-12T16:31:32.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.133137+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:31:32.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.191514+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:31:32.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.225616+0000 osd.11 (osd.11) 156 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:31:32.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.271437+0000 osd.11 (osd.11) 157 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:31:32.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:31 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.750368+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e683: 12 total, 12 up, 12 in 2024-02-12T16:31:32.072 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.521998+0000 osd.10 (osd.10) 138 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:31:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.561067+0000 osd.10 (osd.10) 139 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:31:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.591519+0000 osd.7 (osd.7) 171 : cluster [DBG] 1.d scrub starts 2024-02-12T16:31:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.659468+0000 osd.7 (osd.7) 172 : cluster [DBG] 1.d scrub ok 2024-02-12T16:31:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.828318+0000 osd.0 (osd.0) 111 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:31:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.886617+0000 osd.0 (osd.0) 112 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:31:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.889225+0000 osd.4 (osd.4) 118 : cluster [DBG] 1.e scrub starts 2024-02-12T16:31:32.086 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:30.982704+0000 osd.4 (osd.4) 119 : cluster [DBG] 1.e scrub ok 2024-02-12T16:31:32.086 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.133137+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:31:32.086 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.191514+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:31:32.086 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.225616+0000 osd.11 (osd.11) 156 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:31:32.086 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.271437+0000 osd.11 (osd.11) 157 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:31:32.086 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:31 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.750368+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e683: 12 total, 12 up, 12 in 2024-02-12T16:31:32.113 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:32.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:32.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.521998+0000 osd.10 (osd.10) 138 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.561067+0000 osd.10 (osd.10) 139 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.591519+0000 osd.7 (osd.7) 171 : cluster [DBG] 1.d scrub starts 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.659468+0000 osd.7 (osd.7) 172 : cluster [DBG] 1.d scrub ok 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.828318+0000 osd.0 (osd.0) 111 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.886617+0000 osd.0 (osd.0) 112 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.889225+0000 osd.4 (osd.4) 118 : cluster [DBG] 1.e scrub starts 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:30.982704+0000 osd.4 (osd.4) 119 : cluster [DBG] 1.e scrub ok 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.133137+0000 osd.5 (osd.5) 125 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.191514+0000 osd.5 (osd.5) 126 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.225616+0000 osd.11 (osd.11) 156 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.271437+0000 osd.11 (osd.11) 157 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:31:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:31 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.750368+0000 mon.a (mon.0) 2383 : cluster [DBG] osdmap e683: 12 total, 12 up, 12 in 2024-02-12T16:31:32.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:32.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:32.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:32.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:32.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:32.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:32.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:32.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:32.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:32.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:32.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:32.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:32.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:32.766 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:32.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:32.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:32.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:33.005 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:33.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.521461+0000 osd.10 (osd.10) 140 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:31:33.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.599851+0000 osd.10 (osd.10) 141 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:31:33.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.756656+0000 osd.7 (osd.7) 173 : cluster [INF] 6.18 continuing backfill to osd.11 from (655'6203,679'16298] MIN to 679'16298 2024-02-12T16:31:33.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:31.992361+0000 osd.1 (osd.1) 102 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:31:33.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:32.060447+0000 mgr.y (mgr.14146) 1406 : cluster [DBG] pgmap v1970: 57 pgs: 1 active+clean+scrubbing, 2 peering, 6 unknown, 1 active+remapped+backfilling, 47 active+clean; 7.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7811/236516 objects misplaced (3.303%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:31:33.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:32.088459+0000 osd.1 (osd.1) 103 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:31:33.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:32.097370+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:31:33.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:32.110214+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:31:33.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:32.151906+0000 osd.3 (osd.3) 38 : cluster [DBG] 1.c scrub starts 2024-02-12T16:31:33.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:32 smithi169 bash[19604]: cluster 2024-02-12T16:31:32.200426+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.c scrub ok 2024-02-12T16:31:33.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:33.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.521461+0000 osd.10 (osd.10) 140 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.599851+0000 osd.10 (osd.10) 141 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.756656+0000 osd.7 (osd.7) 173 : cluster [INF] 6.18 continuing backfill to osd.11 from (655'6203,679'16298] MIN to 679'16298 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:31.992361+0000 osd.1 (osd.1) 102 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:32.060447+0000 mgr.y (mgr.14146) 1406 : cluster [DBG] pgmap v1970: 57 pgs: 1 active+clean+scrubbing, 2 peering, 6 unknown, 1 active+remapped+backfilling, 47 active+clean; 7.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7811/236516 objects misplaced (3.303%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:32.088459+0000 osd.1 (osd.1) 103 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:32.097370+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:32.110214+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:32.151906+0000 osd.3 (osd.3) 38 : cluster [DBG] 1.c scrub starts 2024-02-12T16:31:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:32 smithi123 bash[19119]: cluster 2024-02-12T16:31:32.200426+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.c scrub ok 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.521461+0000 osd.10 (osd.10) 140 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.599851+0000 osd.10 (osd.10) 141 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.756656+0000 osd.7 (osd.7) 173 : cluster [INF] 6.18 continuing backfill to osd.11 from (655'6203,679'16298] MIN to 679'16298 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:31.992361+0000 osd.1 (osd.1) 102 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:32.060447+0000 mgr.y (mgr.14146) 1406 : cluster [DBG] pgmap v1970: 57 pgs: 1 active+clean+scrubbing, 2 peering, 6 unknown, 1 active+remapped+backfilling, 47 active+clean; 7.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7811/236516 objects misplaced (3.303%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:32.088459+0000 osd.1 (osd.1) 103 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:32.097370+0000 osd.5 (osd.5) 127 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:32.110214+0000 osd.5 (osd.5) 128 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:32.151906+0000 osd.3 (osd.3) 38 : cluster [DBG] 1.c scrub starts 2024-02-12T16:31:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:32 smithi121 bash[18489]: cluster 2024-02-12T16:31:32.200426+0000 osd.3 (osd.3) 39 : cluster [DBG] 1.c scrub ok 2024-02-12T16:31:33.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:33.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:33.290 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:33.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:33.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:33.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:33.545 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:33.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:33.672 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:33.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:33.813 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:33.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:33.930 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:33 smithi169 bash[19604]: cluster 2024-02-12T16:31:33.080475+0000 osd.5 (osd.5) 129 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:31:33.948 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:33.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:33.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:34.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:34.058 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:33 smithi123 bash[19119]: cluster 2024-02-12T16:31:33.080475+0000 osd.5 (osd.5) 129 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:31:34.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:34.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:33 smithi121 bash[18489]: cluster 2024-02-12T16:31:33.080475+0000 osd.5 (osd.5) 129 : cluster [DBG] 6.7 scrub starts 2024-02-12T16:31:34.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:34.231 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:34.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:34.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:34.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:34.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:34.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:34.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:34.511 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:34.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:34.542 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:34.543 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:34.565 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-02-12T16:31:34.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:31:34.566 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2024-02-12T16:31:34.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:31:34.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:34.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:34.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:34.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:34.804 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:34.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:34.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:34 smithi123 bash[19119]: cluster 2024-02-12T16:31:33.353943+0000 osd.2 (osd.2) 97 : cluster [DBG] 6.f scrub starts 2024-02-12T16:31:34.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:34 smithi123 bash[19119]: cluster 2024-02-12T16:31:33.684280+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:34.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:34 smithi123 bash[19119]: cluster 2024-02-12T16:31:33.684323+0000 osd.6 (osd.6) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:34.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:34 smithi123 bash[19119]: cluster 2024-02-12T16:31:33.794074+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e684: 12 total, 12 up, 12 in 2024-02-12T16:31:34.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:34 smithi123 bash[19119]: cluster 2024-02-12T16:31:34.060918+0000 mgr.y (mgr.14146) 1407 : cluster [DBG] pgmap v1972: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 1 unknown, 52 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.94k op/s; 7641/242488 objects misplaced (3.151%); 209 KiB/s, 3 objects/s recovering 2024-02-12T16:31:34.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:34 smithi123 bash[19119]: cluster 2024-02-12T16:31:34.152593+0000 osd.5 (osd.5) 130 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:31:34.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:34 smithi121 bash[18489]: cluster 2024-02-12T16:31:33.353943+0000 osd.2 (osd.2) 97 : cluster [DBG] 6.f scrub starts 2024-02-12T16:31:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:34 smithi121 bash[18489]: cluster 2024-02-12T16:31:33.684280+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:34 smithi121 bash[18489]: cluster 2024-02-12T16:31:33.684323+0000 osd.6 (osd.6) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:34 smithi121 bash[18489]: cluster 2024-02-12T16:31:33.794074+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e684: 12 total, 12 up, 12 in 2024-02-12T16:31:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:34 smithi121 bash[18489]: cluster 2024-02-12T16:31:34.060918+0000 mgr.y (mgr.14146) 1407 : cluster [DBG] pgmap v1972: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 1 unknown, 52 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.94k op/s; 7641/242488 objects misplaced (3.151%); 209 KiB/s, 3 objects/s recovering 2024-02-12T16:31:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:34 smithi121 bash[18489]: cluster 2024-02-12T16:31:34.152593+0000 osd.5 (osd.5) 130 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:31:34.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:34.932 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:34.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:35.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:35.062 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:35.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:34 smithi169 bash[19604]: cluster 2024-02-12T16:31:33.353943+0000 osd.2 (osd.2) 97 : cluster [DBG] 6.f scrub starts 2024-02-12T16:31:35.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:34 smithi169 bash[19604]: cluster 2024-02-12T16:31:33.684280+0000 osd.6 (osd.6) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:35.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:34 smithi169 bash[19604]: cluster 2024-02-12T16:31:33.684323+0000 osd.6 (osd.6) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:35.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:34 smithi169 bash[19604]: cluster 2024-02-12T16:31:33.794074+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e684: 12 total, 12 up, 12 in 2024-02-12T16:31:35.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:34 smithi169 bash[19604]: cluster 2024-02-12T16:31:34.060918+0000 mgr.y (mgr.14146) 1407 : cluster [DBG] pgmap v1972: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 1 unknown, 52 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.94k op/s; 7641/242488 objects misplaced (3.151%); 209 KiB/s, 3 objects/s recovering 2024-02-12T16:31:35.064 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:34 smithi169 bash[19604]: cluster 2024-02-12T16:31:34.152593+0000 osd.5 (osd.5) 130 : cluster [DBG] 6.7 scrub ok 2024-02-12T16:31:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:35.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:35.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:35.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:35.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:35.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:35.325 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:35.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:35.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:35.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:35.455 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:35.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:35.553 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:35.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:35.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:35.708 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:35.752 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:35.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:35.857 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:35.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:35.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:35.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:35 smithi123 bash[19119]: cluster 2024-02-12T16:31:34.790706+0000 mon.a (mon.0) 2385 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:31:35.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:35 smithi123 bash[19119]: cluster 2024-02-12T16:31:35.735415+0000 mon.a (mon.0) 2386 : cluster [DBG] osdmap e685: 12 total, 12 up, 12 in 2024-02-12T16:31:35.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:36.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:36.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:36.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:35 smithi121 bash[18489]: cluster 2024-02-12T16:31:34.790706+0000 mon.a (mon.0) 2385 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:31:36.117 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:35 smithi121 bash[18489]: cluster 2024-02-12T16:31:35.735415+0000 mon.a (mon.0) 2386 : cluster [DBG] osdmap e685: 12 total, 12 up, 12 in 2024-02-12T16:31:36.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:36.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:36.273 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:35 smithi169 bash[19604]: cluster 2024-02-12T16:31:34.790706+0000 mon.a (mon.0) 2385 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:31:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:35 smithi169 bash[19604]: cluster 2024-02-12T16:31:35.735415+0000 mon.a (mon.0) 2386 : cluster [DBG] osdmap e685: 12 total, 12 up, 12 in 2024-02-12T16:31:36.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:36.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:36.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:36.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:36.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:36.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:36.806 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:36.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:36 smithi121 bash[18489]: cluster 2024-02-12T16:31:35.808998+0000 osd.2 (osd.2) 98 : cluster [DBG] 6.f scrub ok 2024-02-12T16:31:36.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:36 smithi121 bash[18489]: cluster 2024-02-12T16:31:36.061363+0000 mgr.y (mgr.14146) 1408 : cluster [DBG] pgmap v1974: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 2.71k op/s; 7641/247908 objects misplaced (3.082%); 168 KiB/s, 2 objects/s recovering 2024-02-12T16:31:36.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:36 smithi121 bash[18489]: audit 2024-02-12T16:31:36.062113+0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:31:36.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:36 smithi121 bash[18489]: cluster 2024-02-12T16:31:36.148448+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:31:36.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:36.939 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:36.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:37.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:36 smithi169 bash[19604]: cluster 2024-02-12T16:31:35.808998+0000 osd.2 (osd.2) 98 : cluster [DBG] 6.f scrub ok 2024-02-12T16:31:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:36 smithi169 bash[19604]: cluster 2024-02-12T16:31:36.061363+0000 mgr.y (mgr.14146) 1408 : cluster [DBG] pgmap v1974: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 2.71k op/s; 7641/247908 objects misplaced (3.082%); 168 KiB/s, 2 objects/s recovering 2024-02-12T16:31:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:36 smithi169 bash[19604]: audit 2024-02-12T16:31:36.062113+0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:31:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:36 smithi169 bash[19604]: cluster 2024-02-12T16:31:36.148448+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:31:37.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:37.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:37.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:36 smithi123 bash[19119]: cluster 2024-02-12T16:31:35.808998+0000 osd.2 (osd.2) 98 : cluster [DBG] 6.f scrub ok 2024-02-12T16:31:37.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:36 smithi123 bash[19119]: cluster 2024-02-12T16:31:36.061363+0000 mgr.y (mgr.14146) 1408 : cluster [DBG] pgmap v1974: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 114 MiB/s wr, 2.71k op/s; 7641/247908 objects misplaced (3.082%); 168 KiB/s, 2 objects/s recovering 2024-02-12T16:31:37.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:36 smithi123 bash[19119]: audit 2024-02-12T16:31:36.062113+0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:31:37.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:36 smithi123 bash[19119]: cluster 2024-02-12T16:31:36.148448+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.6 scrub starts 2024-02-12T16:31:37.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:37.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:37.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:37.330 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:37.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:37.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:37.460 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:37.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:37.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:37.595 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:37.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:37.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:37.733 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:37.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:37 smithi123 bash[19119]: cluster 2024-02-12T16:31:36.673928+0000 osd.9 (osd.9) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:37 smithi123 bash[19119]: cluster 2024-02-12T16:31:36.673974+0000 osd.9 (osd.9) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:37 smithi123 bash[19119]: audit 2024-02-12T16:31:36.810673+0000 mon.a (mon.0) 2388 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:37 smithi123 bash[19119]: cluster 2024-02-12T16:31:36.810756+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e686: 12 total, 12 up, 12 in 2024-02-12T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:37 smithi123 bash[19119]: cluster 2024-02-12T16:31:37.183508+0000 osd.3 (osd.3) 41 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:31:37.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:37.870 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:37.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:37.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:37 smithi121 bash[18489]: cluster 2024-02-12T16:31:36.673928+0000 osd.9 (osd.9) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:37 smithi121 bash[18489]: cluster 2024-02-12T16:31:36.673974+0000 osd.9 (osd.9) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:37 smithi121 bash[18489]: audit 2024-02-12T16:31:36.810673+0000 mon.a (mon.0) 2388 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:31:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:37 smithi121 bash[18489]: cluster 2024-02-12T16:31:36.810756+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e686: 12 total, 12 up, 12 in 2024-02-12T16:31:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:37 smithi121 bash[18489]: cluster 2024-02-12T16:31:37.183508+0000 osd.3 (osd.3) 41 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:31:37.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:37.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:37 smithi169 bash[19604]: cluster 2024-02-12T16:31:36.673928+0000 osd.9 (osd.9) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:37.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:37 smithi169 bash[19604]: cluster 2024-02-12T16:31:36.673974+0000 osd.9 (osd.9) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:37.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:37 smithi169 bash[19604]: audit 2024-02-12T16:31:36.810673+0000 mon.a (mon.0) 2388 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:31:37.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:37 smithi169 bash[19604]: cluster 2024-02-12T16:31:36.810756+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e686: 12 total, 12 up, 12 in 2024-02-12T16:31:37.983 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:37 smithi169 bash[19604]: cluster 2024-02-12T16:31:37.183508+0000 osd.3 (osd.3) 41 : cluster [DBG] 6.6 scrub ok 2024-02-12T16:31:38.006 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:38.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:38.138 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:38.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:38.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:38.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:38.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:38.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:38.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:38.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:38.681 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:38.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:38.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:38.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:38.959 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:38.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:39.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:39.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:38 smithi121 bash[18489]: cluster 2024-02-12T16:31:37.812996+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e687: 12 total, 12 up, 12 in 2024-02-12T16:31:39.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:38 smithi121 bash[18489]: cluster 2024-02-12T16:31:38.061894+0000 mgr.y (mgr.14146) 1409 : cluster [DBG] pgmap v1977: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 2.31k op/s; 7641/242452 objects misplaced (3.152%) 2024-02-12T16:31:39.074 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:38 smithi121 bash[18489]: audit 2024-02-12T16:31:38.062745+0000 mon.a (mon.0) 2391 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:31:39.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:39.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:38 smithi123 bash[19119]: cluster 2024-02-12T16:31:37.812996+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e687: 12 total, 12 up, 12 in 2024-02-12T16:31:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:38 smithi123 bash[19119]: cluster 2024-02-12T16:31:38.061894+0000 mgr.y (mgr.14146) 1409 : cluster [DBG] pgmap v1977: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 2.31k op/s; 7641/242452 objects misplaced (3.152%) 2024-02-12T16:31:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:38 smithi123 bash[19119]: audit 2024-02-12T16:31:38.062745+0000 mon.a (mon.0) 2391 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:31:39.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:39.213 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:38 smithi169 bash[19604]: cluster 2024-02-12T16:31:37.812996+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e687: 12 total, 12 up, 12 in 2024-02-12T16:31:39.213 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:38 smithi169 bash[19604]: cluster 2024-02-12T16:31:38.061894+0000 mgr.y (mgr.14146) 1409 : cluster [DBG] pgmap v1977: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 2.31k op/s; 7641/242452 objects misplaced (3.152%) 2024-02-12T16:31:39.213 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:38 smithi169 bash[19604]: audit 2024-02-12T16:31:38.062745+0000 mon.a (mon.0) 2391 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:31:39.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:39.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:39.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:39.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:39.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:39.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:39.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:39.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:39.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:39.698 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:39.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:39.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:39.832 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:39.833 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:39 smithi121 bash[18489]: audit 2024-02-12T16:31:38.830302+0000 mon.a (mon.0) 2392 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:31:39.833 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:39 smithi121 bash[18489]: cluster 2024-02-12T16:31:38.830408+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e688: 12 total, 12 up, 12 in 2024-02-12T16:31:39.833 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:39 smithi121 bash[18489]: cluster 2024-02-12T16:31:38.830940+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.10 starting backfill to osd.8 from (0'0,0'0] MAX to 685'11400 2024-02-12T16:31:39.833 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:39 smithi121 bash[18489]: cluster 2024-02-12T16:31:39.173399+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:31:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:39.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:39.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:39.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:39 smithi123 bash[19119]: audit 2024-02-12T16:31:38.830302+0000 mon.a (mon.0) 2392 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:31:39.945 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:39 smithi123 bash[19119]: cluster 2024-02-12T16:31:38.830408+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e688: 12 total, 12 up, 12 in 2024-02-12T16:31:39.946 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:39 smithi123 bash[19119]: cluster 2024-02-12T16:31:38.830940+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.10 starting backfill to osd.8 from (0'0,0'0] MAX to 685'11400 2024-02-12T16:31:39.946 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:39 smithi123 bash[19119]: cluster 2024-02-12T16:31:39.173399+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:31:39.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:39.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:40.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:40.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:40.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:40.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:40.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:40.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:40.237 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:39 smithi169 bash[19604]: audit 2024-02-12T16:31:38.830302+0000 mon.a (mon.0) 2392 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:31:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:39 smithi169 bash[19604]: cluster 2024-02-12T16:31:38.830408+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e688: 12 total, 12 up, 12 in 2024-02-12T16:31:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:39 smithi169 bash[19604]: cluster 2024-02-12T16:31:38.830940+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.10 starting backfill to osd.8 from (0'0,0'0] MAX to 685'11400 2024-02-12T16:31:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:39 smithi169 bash[19604]: cluster 2024-02-12T16:31:39.173399+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.16 scrub starts 2024-02-12T16:31:40.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:40.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:40.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:40.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:40.512 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:40.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:40.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:40.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:40.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:40.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:40.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:40.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:40.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:40.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:40 smithi121 bash[18489]: cluster 2024-02-12T16:31:39.834701+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e689: 12 total, 12 up, 12 in 2024-02-12T16:31:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:40 smithi121 bash[18489]: cluster 2024-02-12T16:31:39.879614+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:31:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:40 smithi121 bash[18489]: cluster 2024-02-12T16:31:40.062523+0000 mgr.y (mgr.14146) 1410 : cluster [DBG] pgmap v1980: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7228/238670 objects misplaced (3.028%); 864 KiB/s, 13 objects/s recovering 2024-02-12T16:31:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:40 smithi121 bash[18489]: audit 2024-02-12T16:31:40.063645+0000 mon.a (mon.0) 2395 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:31:40.915 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:40.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:41.019 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:41.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:40 smithi123 bash[19119]: cluster 2024-02-12T16:31:39.834701+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e689: 12 total, 12 up, 12 in 2024-02-12T16:31:41.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:40 smithi123 bash[19119]: cluster 2024-02-12T16:31:39.879614+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:31:41.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:40 smithi123 bash[19119]: cluster 2024-02-12T16:31:40.062523+0000 mgr.y (mgr.14146) 1410 : cluster [DBG] pgmap v1980: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7228/238670 objects misplaced (3.028%); 864 KiB/s, 13 objects/s recovering 2024-02-12T16:31:41.027 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:40 smithi123 bash[19119]: audit 2024-02-12T16:31:40.063645+0000 mon.a (mon.0) 2395 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:31:41.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:41.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:41.168 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:40 smithi169 bash[19604]: cluster 2024-02-12T16:31:39.834701+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e689: 12 total, 12 up, 12 in 2024-02-12T16:31:41.169 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:40 smithi169 bash[19604]: cluster 2024-02-12T16:31:39.879614+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.16 scrub ok 2024-02-12T16:31:41.169 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:40 smithi169 bash[19604]: cluster 2024-02-12T16:31:40.062523+0000 mgr.y (mgr.14146) 1410 : cluster [DBG] pgmap v1980: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 55 active+clean; 7.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7228/238670 objects misplaced (3.028%); 864 KiB/s, 13 objects/s recovering 2024-02-12T16:31:41.169 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:40 smithi169 bash[19604]: audit 2024-02-12T16:31:40.063645+0000 mon.a (mon.0) 2395 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:31:41.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:41.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:41.315 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:41.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:41.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:41.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:41.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:41.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:41.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.700 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:41.801 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:41.826 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:41.854 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:41 smithi123 bash[19119]: audit 2024-02-12T16:31:40.844489+0000 mon.a (mon.0) 2396 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:31:41.854 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:41 smithi123 bash[19119]: cluster 2024-02-12T16:31:40.844610+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e690: 12 total, 12 up, 12 in 2024-02-12T16:31:41.854 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:41 smithi123 bash[19119]: cluster 2024-02-12T16:31:40.845666+0000 osd.8 (osd.8) 71 : cluster [DBG] 6.11 starting backfill to osd.3 from (0'0,0'0] MAX to 686'11693 2024-02-12T16:31:41.854 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:41 smithi123 bash[19119]: cluster 2024-02-12T16:31:40.855825+0000 osd.8 (osd.8) 72 : cluster [DBG] 6.11 starting backfill to osd.11 from (0'0,0'0] MAX to 686'11693 2024-02-12T16:31:41.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:41.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:41 smithi169 bash[19604]: audit 2024-02-12T16:31:40.844489+0000 mon.a (mon.0) 2396 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:31:41.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:41 smithi169 bash[19604]: cluster 2024-02-12T16:31:40.844610+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e690: 12 total, 12 up, 12 in 2024-02-12T16:31:41.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:41 smithi169 bash[19604]: cluster 2024-02-12T16:31:40.845666+0000 osd.8 (osd.8) 71 : cluster [DBG] 6.11 starting backfill to osd.3 from (0'0,0'0] MAX to 686'11693 2024-02-12T16:31:41.936 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:41 smithi169 bash[19604]: cluster 2024-02-12T16:31:40.855825+0000 osd.8 (osd.8) 72 : cluster [DBG] 6.11 starting backfill to osd.11 from (0'0,0'0] MAX to 686'11693 2024-02-12T16:31:41.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:41.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:42.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:42.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:42.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:42.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:41 smithi121 bash[18489]: audit 2024-02-12T16:31:40.844489+0000 mon.a (mon.0) 2396 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:31:42.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:41 smithi121 bash[18489]: cluster 2024-02-12T16:31:40.844610+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e690: 12 total, 12 up, 12 in 2024-02-12T16:31:42.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:41 smithi121 bash[18489]: cluster 2024-02-12T16:31:40.845666+0000 osd.8 (osd.8) 71 : cluster [DBG] 6.11 starting backfill to osd.3 from (0'0,0'0] MAX to 686'11693 2024-02-12T16:31:42.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:41 smithi121 bash[18489]: cluster 2024-02-12T16:31:40.855825+0000 osd.8 (osd.8) 72 : cluster [DBG] 6.11 starting backfill to osd.11 from (0'0,0'0] MAX to 686'11693 2024-02-12T16:31:42.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:42.219 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:42.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:42.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:42.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:42.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:42.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:42.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:42.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:31:42.605 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:31:42.605 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:31:42.606 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:31:42.606 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:31:42.612 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:42.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:42.624 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:31:42.644 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:31:42.667 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:31:42.691 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:31:42.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:42.722 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:31:42.740 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:42.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:42.745 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:31:42.773 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:31:42.793 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:31:42.812 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:31:42.832 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:31:42.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:31:42.866 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.866 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.866 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:31:42.867 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:42 smithi123 bash[19119]: cluster 2024-02-12T16:31:41.844016+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e691: 12 total, 12 up, 12 in 2024-02-12T16:31:42.867 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:42 smithi123 bash[19119]: cluster 2024-02-12T16:31:42.062996+0000 mgr.y (mgr.14146) 1411 : cluster [DBG] pgmap v1983: 57 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 54 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10890/236428 objects misplaced (4.606%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:31:42.868 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:42 smithi123 bash[19119]: audit 2024-02-12T16:31:42.063619+0000 mon.a (mon.0) 2399 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:31:42.868 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:42 smithi123 bash[19119]: audit 2024-02-12T16:31:42.578205+0000 mon.a (mon.0) 2400 : audit [DBG] from='client.? v1:172.21.15.121:0/1379886797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:42.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:42.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:42.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:42 smithi169 bash[19604]: cluster 2024-02-12T16:31:41.844016+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e691: 12 total, 12 up, 12 in 2024-02-12T16:31:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:42 smithi169 bash[19604]: cluster 2024-02-12T16:31:42.062996+0000 mgr.y (mgr.14146) 1411 : cluster [DBG] pgmap v1983: 57 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 54 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10890/236428 objects misplaced (4.606%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:31:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:42 smithi169 bash[19604]: audit 2024-02-12T16:31:42.063619+0000 mon.a (mon.0) 2399 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:31:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:42 smithi169 bash[19604]: audit 2024-02-12T16:31:42.578205+0000 mon.a (mon.0) 2400 : audit [DBG] from='client.? v1:172.21.15.121:0/1379886797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:43.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:43.125 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:43.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:42 smithi121 bash[18489]: cluster 2024-02-12T16:31:41.844016+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e691: 12 total, 12 up, 12 in 2024-02-12T16:31:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:42 smithi121 bash[18489]: cluster 2024-02-12T16:31:42.062996+0000 mgr.y (mgr.14146) 1411 : cluster [DBG] pgmap v1983: 57 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 54 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10890/236428 objects misplaced (4.606%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:31:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:42 smithi121 bash[18489]: audit 2024-02-12T16:31:42.063619+0000 mon.a (mon.0) 2399 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:31:43.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:42 smithi121 bash[18489]: audit 2024-02-12T16:31:42.578205+0000 mon.a (mon.0) 2400 : audit [DBG] from='client.? v1:172.21.15.121:0/1379886797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:31:43.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:43.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:43.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:43.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:43.508 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:43.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:43.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:43.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:43.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:43.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:43.872 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:43 smithi121 bash[18489]: cluster 2024-02-12T16:31:42.860320+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.12 starting backfill to osd.9 from (0'0,0'0] MAX to 688'11525 2024-02-12T16:31:43.872 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:43 smithi121 bash[18489]: audit 2024-02-12T16:31:42.860974+0000 mon.a (mon.0) 2401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:31:43.872 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:43 smithi121 bash[18489]: cluster 2024-02-12T16:31:42.861038+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e692: 12 total, 12 up, 12 in 2024-02-12T16:31:43.872 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:43 smithi121 bash[18489]: cluster 2024-02-12T16:31:42.873968+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.12 starting backfill to osd.11 from (0'0,0'0] MAX to 688'11525 2024-02-12T16:31:43.872 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:43 smithi121 bash[18489]: audit 2024-02-12T16:31:43.119303+0000 mon.a (mon.0) 2403 : audit [INF] from='client.26359 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:43.872 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:43 smithi121 bash[18489]: audit 2024-02-12T16:31:43.120211+0000 mon.c (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.121:0/226723965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:43.873 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:31:43.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:43.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:43.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:44.021 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:44.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:43 smithi123 bash[19119]: cluster 2024-02-12T16:31:42.860320+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.12 starting backfill to osd.9 from (0'0,0'0] MAX to 688'11525 2024-02-12T16:31:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:43 smithi123 bash[19119]: audit 2024-02-12T16:31:42.860974+0000 mon.a (mon.0) 2401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:31:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:43 smithi123 bash[19119]: cluster 2024-02-12T16:31:42.861038+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e692: 12 total, 12 up, 12 in 2024-02-12T16:31:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:43 smithi123 bash[19119]: cluster 2024-02-12T16:31:42.873968+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.12 starting backfill to osd.11 from (0'0,0'0] MAX to 688'11525 2024-02-12T16:31:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:43 smithi123 bash[19119]: audit 2024-02-12T16:31:43.119303+0000 mon.a (mon.0) 2403 : audit [INF] from='client.26359 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:44.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:43 smithi123 bash[19119]: audit 2024-02-12T16:31:43.120211+0000 mon.c (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.121:0/226723965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:44.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:44.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:44.256 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:43 smithi169 bash[19604]: cluster 2024-02-12T16:31:42.860320+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.12 starting backfill to osd.9 from (0'0,0'0] MAX to 688'11525 2024-02-12T16:31:44.256 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:43 smithi169 bash[19604]: audit 2024-02-12T16:31:42.860974+0000 mon.a (mon.0) 2401 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:31:44.256 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:43 smithi169 bash[19604]: cluster 2024-02-12T16:31:42.861038+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e692: 12 total, 12 up, 12 in 2024-02-12T16:31:44.256 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:43 smithi169 bash[19604]: cluster 2024-02-12T16:31:42.873968+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.12 starting backfill to osd.11 from (0'0,0'0] MAX to 688'11525 2024-02-12T16:31:44.256 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:43 smithi169 bash[19604]: audit 2024-02-12T16:31:43.119303+0000 mon.a (mon.0) 2403 : audit [INF] from='client.26359 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:44.256 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:43 smithi169 bash[19604]: audit 2024-02-12T16:31:43.120211+0000 mon.c (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.121:0/226723965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:31:44.274 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:44.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:44.405 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:44.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:44.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:44.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:44.663 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:44.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:44.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:44.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:44.900 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:44 smithi123 bash[19119]: cluster 2024-02-12T16:31:43.655732+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:44.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:44 smithi123 bash[19119]: cluster 2024-02-12T16:31:43.655774+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:44.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:44 smithi123 bash[19119]: cluster 2024-02-12T16:31:43.857383+0000 mon.a (mon.0) 2404 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:31:44.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:44 smithi123 bash[19119]: audit 2024-02-12T16:31:43.863955+0000 mon.a (mon.0) 2405 : audit [INF] from='client.26359 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:31:44.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:44 smithi123 bash[19119]: cluster 2024-02-12T16:31:43.864043+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in 2024-02-12T16:31:44.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:44 smithi123 bash[19119]: cluster 2024-02-12T16:31:44.063498+0000 mgr.y (mgr.14146) 1412 : cluster [DBG] pgmap v1986: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 15907/235342 objects misplaced (6.759%); 19 MiB/s, 310 objects/s recovering 2024-02-12T16:31:44.901 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:44 smithi123 bash[19119]: cluster 2024-02-12T16:31:44.864026+0000 mon.a (mon.0) 2407 : cluster [DBG] osdmap e694: 12 total, 12 up, 12 in 2024-02-12T16:31:44.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:44.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:44.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:31:45.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:45.028 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:44 smithi169 bash[19604]: cluster 2024-02-12T16:31:43.655732+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:45.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:44 smithi169 bash[19604]: cluster 2024-02-12T16:31:43.655774+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:45.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:44 smithi169 bash[19604]: cluster 2024-02-12T16:31:43.857383+0000 mon.a (mon.0) 2404 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:31:45.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:44 smithi169 bash[19604]: audit 2024-02-12T16:31:43.863955+0000 mon.a (mon.0) 2405 : audit [INF] from='client.26359 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:31:45.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:44 smithi169 bash[19604]: cluster 2024-02-12T16:31:43.864043+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in 2024-02-12T16:31:45.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:44 smithi169 bash[19604]: cluster 2024-02-12T16:31:44.063498+0000 mgr.y (mgr.14146) 1412 : cluster [DBG] pgmap v1986: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 15907/235342 objects misplaced (6.759%); 19 MiB/s, 310 objects/s recovering 2024-02-12T16:31:45.029 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:44 smithi169 bash[19604]: cluster 2024-02-12T16:31:44.864026+0000 mon.a (mon.0) 2407 : cluster [DBG] osdmap e694: 12 total, 12 up, 12 in 2024-02-12T16:31:45.046 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:45.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:45.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:45.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:44 smithi121 bash[18489]: cluster 2024-02-12T16:31:43.655732+0000 osd.0 (osd.0) 114 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:44 smithi121 bash[18489]: cluster 2024-02-12T16:31:43.655774+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:44 smithi121 bash[18489]: cluster 2024-02-12T16:31:43.857383+0000 mon.a (mon.0) 2404 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:31:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:44 smithi121 bash[18489]: audit 2024-02-12T16:31:43.863955+0000 mon.a (mon.0) 2405 : audit [INF] from='client.26359 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:31:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:44 smithi121 bash[18489]: cluster 2024-02-12T16:31:43.864043+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e693: 12 total, 12 up, 12 in 2024-02-12T16:31:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:44 smithi121 bash[18489]: cluster 2024-02-12T16:31:44.063498+0000 mgr.y (mgr.14146) 1412 : cluster [DBG] pgmap v1986: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 7.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 15907/235342 objects misplaced (6.759%); 19 MiB/s, 310 objects/s recovering 2024-02-12T16:31:45.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:44 smithi121 bash[18489]: cluster 2024-02-12T16:31:44.864026+0000 mon.a (mon.0) 2407 : cluster [DBG] osdmap e694: 12 total, 12 up, 12 in 2024-02-12T16:31:45.172 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:45.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:45.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:45.302 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:45.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:45.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:45.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:45.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:45.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:45.553 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:45.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:45.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:45.695 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:45.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:45.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:45.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:45.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:45.949 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:45 smithi123 bash[19119]: cluster 2024-02-12T16:31:44.863158+0000 osd.6 (osd.6) 132 : cluster [DBG] 6.13 starting backfill to osd.3 from (0'0,0'0] MAX to 690'11403 2024-02-12T16:31:45.949 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:45 smithi123 bash[19119]: cluster 2024-02-12T16:31:44.875339+0000 osd.6 (osd.6) 133 : cluster [DBG] 6.13 starting backfill to osd.11 from (0'0,0'0] MAX to 690'11403 2024-02-12T16:31:45.950 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:45 smithi123 bash[19119]: cluster 2024-02-12T16:31:45.740742+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e695: 12 total, 12 up, 12 in 2024-02-12T16:31:46.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:46.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:46.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:46.138 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:45 smithi169 bash[19604]: cluster 2024-02-12T16:31:44.863158+0000 osd.6 (osd.6) 132 : cluster [DBG] 6.13 starting backfill to osd.3 from (0'0,0'0] MAX to 690'11403 2024-02-12T16:31:46.138 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:45 smithi169 bash[19604]: cluster 2024-02-12T16:31:44.875339+0000 osd.6 (osd.6) 133 : cluster [DBG] 6.13 starting backfill to osd.11 from (0'0,0'0] MAX to 690'11403 2024-02-12T16:31:46.138 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:45 smithi169 bash[19604]: cluster 2024-02-12T16:31:45.740742+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e695: 12 total, 12 up, 12 in 2024-02-12T16:31:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:46.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:45 smithi121 bash[18489]: cluster 2024-02-12T16:31:44.863158+0000 osd.6 (osd.6) 132 : cluster [DBG] 6.13 starting backfill to osd.3 from (0'0,0'0] MAX to 690'11403 2024-02-12T16:31:46.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:45 smithi121 bash[18489]: cluster 2024-02-12T16:31:44.875339+0000 osd.6 (osd.6) 133 : cluster [DBG] 6.13 starting backfill to osd.11 from (0'0,0'0] MAX to 690'11403 2024-02-12T16:31:46.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:45 smithi121 bash[18489]: cluster 2024-02-12T16:31:45.740742+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e695: 12 total, 12 up, 12 in 2024-02-12T16:31:46.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:46.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:46.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:46.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:46.566 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:46.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:46.715 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:46.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:46.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:46.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:46.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:46.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:47.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:47.113 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:47.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:47.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:46 smithi123 bash[19119]: cluster 2024-02-12T16:31:46.063971+0000 mgr.y (mgr.14146) 1413 : cluster [DBG] pgmap v1989: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 30463/234226 objects misplaced (13.006%); 19 MiB/s, 305 objects/s recovering 2024-02-12T16:31:47.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:46 smithi123 bash[19119]: audit 2024-02-12T16:31:46.358461+0000 mon.a (mon.0) 2409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:31:47.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:46 smithi123 bash[19119]: audit 2024-02-12T16:31:46.723441+0000 mon.a (mon.0) 2410 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:31:47.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:46 smithi121 bash[18489]: cluster 2024-02-12T16:31:46.063971+0000 mgr.y (mgr.14146) 1413 : cluster [DBG] pgmap v1989: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 30463/234226 objects misplaced (13.006%); 19 MiB/s, 305 objects/s recovering 2024-02-12T16:31:47.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:46 smithi121 bash[18489]: audit 2024-02-12T16:31:46.358461+0000 mon.a (mon.0) 2409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:31:47.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:46 smithi121 bash[18489]: audit 2024-02-12T16:31:46.723441+0000 mon.a (mon.0) 2410 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:31:47.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:47.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:47.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:46 smithi169 bash[19604]: cluster 2024-02-12T16:31:46.063971+0000 mgr.y (mgr.14146) 1413 : cluster [DBG] pgmap v1989: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 30463/234226 objects misplaced (13.006%); 19 MiB/s, 305 objects/s recovering 2024-02-12T16:31:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:46 smithi169 bash[19604]: audit 2024-02-12T16:31:46.358461+0000 mon.a (mon.0) 2409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:31:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:46 smithi169 bash[19604]: audit 2024-02-12T16:31:46.723441+0000 mon.a (mon.0) 2410 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:31:47.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:31:47.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:47.386 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:47.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:47.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:47.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:47.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:47.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:31:47.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:47.646 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:47.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:47.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:47.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:47.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:47.876 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:31:47.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:47.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:47 smithi123 bash[19119]: cluster 2024-02-12T16:31:46.873168+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:31:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:47 smithi123 bash[19119]: audit 2024-02-12T16:31:47.082361+0000 mon.a (mon.0) 2412 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:31:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:47 smithi123 bash[19119]: audit 2024-02-12T16:31:47.083245+0000 mon.a (mon.0) 2413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:31:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:47 smithi123 bash[19119]: audit 2024-02-12T16:31:47.091282+0000 mon.a (mon.0) 2414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:31:47.891 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:31:47.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:47.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:48.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:48.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:48.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:48.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:48.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:47 smithi121 bash[18489]: cluster 2024-02-12T16:31:46.873168+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:31:48.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:47 smithi121 bash[18489]: audit 2024-02-12T16:31:47.082361+0000 mon.a (mon.0) 2412 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:31:48.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:47 smithi121 bash[18489]: audit 2024-02-12T16:31:47.083245+0000 mon.a (mon.0) 2413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:31:48.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:47 smithi121 bash[18489]: audit 2024-02-12T16:31:47.091282+0000 mon.a (mon.0) 2414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:31:48.191 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:48.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:48.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:47 smithi169 bash[19604]: cluster 2024-02-12T16:31:46.873168+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:31:48.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:47 smithi169 bash[19604]: audit 2024-02-12T16:31:47.082361+0000 mon.a (mon.0) 2412 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:31:48.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:47 smithi169 bash[19604]: audit 2024-02-12T16:31:47.083245+0000 mon.a (mon.0) 2413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:31:48.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:47 smithi169 bash[19604]: audit 2024-02-12T16:31:47.091282+0000 mon.a (mon.0) 2414 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:31:48.329 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:48.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:48.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:48.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:48.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:48.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:48.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:48.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:48.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:48.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:48.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:48.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:48.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:48.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:48.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:49.016 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":1989,"stamp":"2024-02-12T16:31:46.063695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7675019288,"num_objects":117113,"num_object_clones":0,"num_object_copies":234226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30463,"num_objects_unfound":0,"num_objects_dirty":117113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171367,"num_write_kb":8874657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":95092736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267104,"ondisk_log_size":267104,"up":114,"acting":114,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":20406244,"kb_used_data":15235908,"kb_used_omap":91,"kb_used_meta":5170148,"kb_avail":1104535580,"statfs":{"total":1151940427776,"available":1131044433920,"internally_reserved":0,"allocated":15601569792,"data_stored":15595132182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":5294231568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,6,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-72155136,"num_objects":-1101,"num_object_clones":0,"num_object_copies":-2202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19573,"num_objects_unfound":0,"num_objects_dirty":-1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1101,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1221,"num_bytes_recovered":80019456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000898"},"pg_stats":[{"pgid":"6.18","version":"692'16611","reported_seq":17224,"reported_epoch":692,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:31:43.367814+0000","last_change":"2024-02-12T16:31:32.441235+0000","last_active":"2024-02-12T16:31:43.367814+0000","last_peered":"2024-02-12T16:31:43.367814+0000","last_clean":"2024-02-12T16:31:14.115119+0000","last_became_active":"2024-02-12T16:31:32.326286+0000","last_became_peered":"2024-02-12T16:31:32.326286+0000","last_unstale":"2024-02-12T16:31:43.367814+0000","last_undegraded":"2024-02-12T16:31:43.367814+0000","last_fullsized":"2024-02-12T16:31:43.367814+0000","mapping_epoch":682,"log_start":"655'6203","ondisk_log_start":"655'6203","created":680,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10408,"ondisk_log_size":10408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":239075328,"num_objects":3648,"num_object_clones":0,"num_object_copies":7296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3467,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5826,"num_write_kb":303168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":12419072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,8],"avail_no_missing":["7","8"],"object_location_counts":[{"shards":"7,8","objects":3648}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"690'16000","reported_seq":16074,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.081316+0000","last_change":"2024-02-12T16:31:29.810668+0000","last_active":"2024-02-12T16:31:41.081316+0000","last_peered":"2024-02-12T16:31:41.081316+0000","last_clean":"2024-02-12T16:31:41.081316+0000","last_became_active":"2024-02-12T16:31:29.793937+0000","last_became_peered":"2024-02-12T16:31:29.793937+0000","last_unstale":"2024-02-12T16:31:41.081316+0000","last_undegraded":"2024-02-12T16:31:41.081316+0000","last_fullsized":"2024-02-12T16:31:41.081316+0000","mapping_epoch":680,"log_start":"654'5601","ondisk_log_start":"654'5601","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10399,"ondisk_log_size":10399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":234422272,"num_objects":3577,"num_object_clones":0,"num_object_copies":7154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":302752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"0'0","reported_seq":41,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.863577+0000","last_change":"2024-02-12T16:31:31.271540+0000","last_active":"2024-02-12T16:31:42.863577+0000","last_peered":"2024-02-12T16:31:42.863577+0000","last_clean":"2024-02-12T16:31:42.863577+0000","last_became_active":"2024-02-12T16:30:55.395625+0000","last_became_peered":"2024-02-12T16:30:55.395625+0000","last_unstale":"2024-02-12T16:31:42.863577+0000","last_undegraded":"2024-02-12T16:31:42.863577+0000","last_fullsized":"2024-02-12T16:31:42.863577+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:31.271443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:31.271443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":40,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T16:31:40.023970+0000","last_change":"2024-02-12T16:31:30.111133+0000","last_active":"2024-02-12T16:31:40.023970+0000","last_peered":"2024-02-12T16:31:40.023970+0000","last_clean":"2024-02-12T16:31:40.023970+0000","last_became_active":"2024-02-12T16:30:55.398431+0000","last_became_peered":"2024-02-12T16:30:55.398431+0000","last_unstale":"2024-02-12T16:31:40.023970+0000","last_undegraded":"2024-02-12T16:31:40.023970+0000","last_fullsized":"2024-02-12T16:31:40.023970+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:30.111034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:30.111034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":41,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.865891+0000","last_change":"2024-02-12T16:31:29.507118+0000","last_active":"2024-02-12T16:31:42.865891+0000","last_peered":"2024-02-12T16:31:42.865891+0000","last_clean":"2024-02-12T16:31:42.865891+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:31:42.865891+0000","last_undegraded":"2024-02-12T16:31:42.865891+0000","last_fullsized":"2024-02-12T16:31:42.865891+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:29.506861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:29.506861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":41,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.865895+0000","last_change":"2024-02-12T16:31:28.598604+0000","last_active":"2024-02-12T16:31:42.865895+0000","last_peered":"2024-02-12T16:31:42.865895+0000","last_clean":"2024-02-12T16:31:42.865895+0000","last_became_active":"2024-02-12T16:30:55.409579+0000","last_became_peered":"2024-02-12T16:30:55.409579+0000","last_unstale":"2024-02-12T16:31:42.865895+0000","last_undegraded":"2024-02-12T16:31:42.865895+0000","last_fullsized":"2024-02-12T16:31:42.865895+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:28.598517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:28.598517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":46,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.889267+0000","last_change":"2024-02-12T16:31:42.888513+0000","last_active":"2024-02-12T16:31:42.889267+0000","last_peered":"2024-02-12T16:31:42.889267+0000","last_clean":"2024-02-12T16:31:42.889267+0000","last_became_active":"2024-02-12T16:30:55.394105+0000","last_became_peered":"2024-02-12T16:30:55.394105+0000","last_unstale":"2024-02-12T16:31:42.889267+0000","last_undegraded":"2024-02-12T16:31:42.889267+0000","last_fullsized":"2024-02-12T16:31:42.889267+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:17.846300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.846300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":40,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.082249+0000","last_change":"2024-02-12T16:31:31.600023+0000","last_active":"2024-02-12T16:31:42.082249+0000","last_peered":"2024-02-12T16:31:42.082249+0000","last_clean":"2024-02-12T16:31:42.082249+0000","last_became_active":"2024-02-12T16:30:55.542451+0000","last_became_peered":"2024-02-12T16:30:55.542451+0000","last_unstale":"2024-02-12T16:31:42.082249+0000","last_undegraded":"2024-02-12T16:31:42.082249+0000","last_fullsized":"2024-02-12T16:31:42.082249+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:31.599861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:31.599861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":388,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:40.945692+0000","last_change":"2024-02-12T16:31:29.124659+0000","last_active":"2024-02-12T16:31:40.945692+0000","last_peered":"2024-02-12T16:31:40.945692+0000","last_clean":"2024-02-12T16:31:40.945692+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:31:40.945692+0000","last_undegraded":"2024-02-12T16:31:40.945692+0000","last_fullsized":"2024-02-12T16:31:40.945692+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:29.124575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:29.124575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"690'11437","reported_seq":11558,"reported_epoch":690,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:31:41.046748+0000","last_change":"2024-02-12T16:31:39.225729+0000","last_active":"2024-02-12T16:31:41.046748+0000","last_peered":"2024-02-12T16:31:41.046748+0000","last_clean":"2024-02-12T16:31:36.781460+0000","last_became_active":"2024-02-12T16:31:39.164121+0000","last_became_peered":"2024-02-12T16:31:39.164121+0000","last_unstale":"2024-02-12T16:31:41.046748+0000","last_undegraded":"2024-02-12T16:31:41.046748+0000","last_fullsized":"2024-02-12T16:31:41.046748+0000","mapping_epoch":687,"log_start":"646'1100","ondisk_log_start":"646'1100","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10337,"ondisk_log_size":10337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":241303552,"num_objects":3682,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3662,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":304768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":3682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":40,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.866071+0000","last_change":"2024-02-12T16:31:35.433532+0000","last_active":"2024-02-12T16:31:42.866071+0000","last_peered":"2024-02-12T16:31:42.866071+0000","last_clean":"2024-02-12T16:31:42.866071+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:31:42.866071+0000","last_undegraded":"2024-02-12T16:31:42.866071+0000","last_fullsized":"2024-02-12T16:31:42.866071+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:23.709532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:23.709532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.11","version":"692'11745","reported_seq":15335,"reported_epoch":693,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:31:43.905949+0000","last_change":"2024-02-12T16:31:40.952537+0000","last_active":"2024-02-12T16:31:43.905949+0000","last_peered":"2024-02-12T16:31:43.905949+0000","last_clean":"2024-02-12T16:31:37.816652+0000","last_became_active":"2024-02-12T16:31:40.927588+0000","last_became_peered":"2024-02-12T16:31:40.927588+0000","last_unstale":"2024-02-12T16:31:43.905949+0000","last_undegraded":"2024-02-12T16:31:43.905949+0000","last_fullsized":"2024-02-12T16:31:43.905949+0000","mapping_epoch":689,"log_start":"646'1400","ondisk_log_start":"646'1400","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"656'6950","last_scrub_stamp":"2024-02-12T16:30:14.669359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:14.669359+0000","log_size":10345,"ondisk_log_size":10345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":245465088,"num_objects":3746,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5164,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":312896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":76546048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[8,9],"avail_no_missing":["8","9"],"object_location_counts":[{"shards":"8,9","objects":3746}],"blocked_by":[],"up_primary":3,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":38,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T16:31:40.023863+0000","last_change":"2024-02-12T16:31:32.088557+0000","last_active":"2024-02-12T16:31:40.023863+0000","last_peered":"2024-02-12T16:31:40.023863+0000","last_clean":"2024-02-12T16:31:40.023863+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:31:40.023863+0000","last_undegraded":"2024-02-12T16:31:40.023863+0000","last_fullsized":"2024-02-12T16:31:40.023863+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:32.088463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:32.088463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"694'11573","reported_seq":11658,"reported_epoch":694,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:31:45.257387+0000","last_change":"2024-02-12T16:31:43.022665+0000","last_active":"2024-02-12T16:31:45.257387+0000","last_peered":"2024-02-12T16:31:45.257387+0000","last_clean":"2024-02-12T16:31:39.573115+0000","last_became_active":"2024-02-12T16:31:42.981292+0000","last_became_peered":"2024-02-12T16:31:42.981292+0000","last_unstale":"2024-02-12T16:31:45.257387+0000","last_undegraded":"2024-02-12T16:31:45.257387+0000","last_fullsized":"2024-02-12T16:31:45.257387+0000","mapping_epoch":691,"log_start":"646'1200","ondisk_log_start":"646'1200","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"661'8683","last_scrub_stamp":"2024-02-12T16:30:39.183515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:39.183515+0000","log_size":10373,"ondisk_log_size":10373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":239075328,"num_objects":3648,"num_object_clones":0,"num_object_copies":7296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7296,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":307040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":3648}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":35,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.845178+0000","last_change":"2024-02-12T16:30:55.405054+0000","last_active":"2024-02-12T16:31:41.845178+0000","last_peered":"2024-02-12T16:31:41.845178+0000","last_clean":"2024-02-12T16:31:41.845178+0000","last_became_active":"2024-02-12T16:30:55.404763+0000","last_became_peered":"2024-02-12T16:30:55.404763+0000","last_unstale":"2024-02-12T16:31:41.845178+0000","last_undegraded":"2024-02-12T16:31:41.845178+0000","last_fullsized":"2024-02-12T16:31:41.845178+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:28.981314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:30:28.981314+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"694'11432","reported_seq":11511,"reported_epoch":694,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:31:45.283291+0000","last_change":"2024-02-12T16:31:44.990213+0000","last_active":"2024-02-12T16:31:45.283291+0000","last_peered":"2024-02-12T16:31:45.283291+0000","last_clean":"2024-02-12T16:31:41.288399+0000","last_became_active":"2024-02-12T16:31:44.956987+0000","last_became_peered":"2024-02-12T16:31:44.956987+0000","last_unstale":"2024-02-12T16:31:45.283291+0000","last_undegraded":"2024-02-12T16:31:45.283291+0000","last_fullsized":"2024-02-12T16:31:45.283291+0000","mapping_epoch":693,"log_start":"646'1100","ondisk_log_start":"646'1100","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10332,"ondisk_log_size":10332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":237862912,"num_objects":3630,"num_object_clones":0,"num_object_copies":7260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7260,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":303584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":3630}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":191,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.304029+0000","last_change":"2024-02-12T16:31:41.304661+0000","last_active":"2024-02-12T16:31:41.304029+0000","last_peered":"2024-02-12T16:31:41.304029+0000","last_clean":"2024-02-12T16:31:41.304029+0000","last_became_active":"2024-02-12T16:23:39.022712+0000","last_became_peered":"2024-02-12T16:23:39.022712+0000","last_unstale":"2024-02-12T16:31:41.304029+0000","last_undegraded":"2024-02-12T16:31:41.304029+0000","last_fullsized":"2024-02-12T16:31:41.304029+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:20.828149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:29:20.828149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"690'11469","reported_seq":11550,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.472745+0000","last_change":"2024-02-12T16:31:34.152679+0000","last_active":"2024-02-12T16:31:41.472745+0000","last_peered":"2024-02-12T16:31:41.472745+0000","last_clean":"2024-02-12T16:31:41.472745+0000","last_became_active":"2024-02-12T16:31:29.781269+0000","last_became_peered":"2024-02-12T16:31:29.781269+0000","last_unstale":"2024-02-12T16:31:41.472745+0000","last_undegraded":"2024-02-12T16:31:41.472745+0000","last_fullsized":"2024-02-12T16:31:41.472745+0000","mapping_epoch":636,"log_start":"646'1200","ondisk_log_start":"646'1200","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'11302","last_scrub_stamp":"2024-02-12T16:31:34.152600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:31:34.152600+0000","log_size":10269,"ondisk_log_size":10269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":238288896,"num_objects":3636,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":304,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T16:31:39.837474+0000","last_change":"2024-02-12T16:30:09.838606+0000","last_active":"2024-02-12T16:31:39.837474+0000","last_peered":"2024-02-12T16:31:39.837474+0000","last_clean":"2024-02-12T16:31:39.837474+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:31:39.837474+0000","last_undegraded":"2024-02-12T16:31:39.837474+0000","last_fullsized":"2024-02-12T16:31:39.837474+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.838471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.838471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.2","version":"693'11559","reported_seq":11636,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.888276+0000","last_change":"2024-02-12T16:31:29.821558+0000","last_active":"2024-02-12T16:31:43.888276+0000","last_peered":"2024-02-12T16:31:43.888276+0000","last_clean":"2024-02-12T16:31:43.888276+0000","last_became_active":"2024-02-12T16:31:29.804115+0000","last_became_peered":"2024-02-12T16:31:29.804115+0000","last_unstale":"2024-02-12T16:31:43.888276+0000","last_undegraded":"2024-02-12T16:31:43.888276+0000","last_fullsized":"2024-02-12T16:31:43.888276+0000","mapping_epoch":636,"log_start":"646'1200","ondisk_log_start":"646'1200","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8683","last_scrub_stamp":"2024-02-12T16:30:39.183515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:39.183515+0000","log_size":10359,"ondisk_log_size":10359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":239992832,"num_objects":3662,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5933,"num_write_kb":307040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":396,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.082309+0000","last_change":"2024-02-12T16:31:29.596430+0000","last_active":"2024-02-12T16:31:42.082309+0000","last_peered":"2024-02-12T16:31:42.082309+0000","last_clean":"2024-02-12T16:31:42.082309+0000","last_became_active":"2024-02-12T16:30:49.374358+0000","last_became_peered":"2024-02-12T16:30:49.374358+0000","last_unstale":"2024-02-12T16:31:42.082309+0000","last_undegraded":"2024-02-12T16:31:42.082309+0000","last_fullsized":"2024-02-12T16:31:42.082309+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:29.596281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:31:29.596281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"692'11418","reported_seq":11488,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.335285+0000","last_change":"2024-02-12T16:31:29.788032+0000","last_active":"2024-02-12T16:31:43.335285+0000","last_peered":"2024-02-12T16:31:43.335285+0000","last_clean":"2024-02-12T16:31:43.335285+0000","last_became_active":"2024-02-12T16:31:29.774302+0000","last_became_peered":"2024-02-12T16:31:29.774302+0000","last_unstale":"2024-02-12T16:31:43.335285+0000","last_undegraded":"2024-02-12T16:31:43.335285+0000","last_fullsized":"2024-02-12T16:31:43.335285+0000","mapping_epoch":636,"log_start":"646'1100","ondisk_log_start":"646'1100","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10318,"ondisk_log_size":10318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":238780416,"num_objects":3643,"num_object_clones":0,"num_object_copies":7286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5843,"num_write_kb":303584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":788,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.863526+0000","last_change":"2024-02-12T16:30:10.264990+0000","last_active":"2024-02-12T16:31:42.863526+0000","last_peered":"2024-02-12T16:31:42.863526+0000","last_clean":"2024-02-12T16:31:42.863526+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:31:42.863526+0000","last_undegraded":"2024-02-12T16:31:42.863526+0000","last_fullsized":"2024-02-12T16:31:42.863526+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.264894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.264894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.0","version":"690'11445","reported_seq":11517,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.085800+0000","last_change":"2024-02-12T16:31:33.807583+0000","last_active":"2024-02-12T16:31:41.085800+0000","last_peered":"2024-02-12T16:31:41.085800+0000","last_clean":"2024-02-12T16:31:41.085800+0000","last_became_active":"2024-02-12T16:31:29.770479+0000","last_became_peered":"2024-02-12T16:31:29.770479+0000","last_unstale":"2024-02-12T16:31:41.085800+0000","last_undegraded":"2024-02-12T16:31:41.085800+0000","last_fullsized":"2024-02-12T16:31:41.085800+0000","mapping_epoch":636,"log_start":"646'1100","ondisk_log_start":"646'1100","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10345,"ondisk_log_size":10345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":240779264,"num_objects":3674,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5850,"num_write_kb":304768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":441,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.866944+0000","last_change":"2024-02-12T16:31:27.414044+0000","last_active":"2024-02-12T16:31:43.866944+0000","last_peered":"2024-02-12T16:31:43.866944+0000","last_clean":"2024-02-12T16:31:43.866944+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:31:43.866944+0000","last_undegraded":"2024-02-12T16:31:43.866944+0000","last_fullsized":"2024-02-12T16:31:43.866944+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:27.413933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:27.413933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"692'11735","reported_seq":11812,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.826643+0000","last_change":"2024-02-12T16:31:29.787171+0000","last_active":"2024-02-12T16:31:43.826643+0000","last_peered":"2024-02-12T16:31:43.826643+0000","last_clean":"2024-02-12T16:31:43.826643+0000","last_became_active":"2024-02-12T16:31:29.775552+0000","last_became_peered":"2024-02-12T16:31:29.775552+0000","last_unstale":"2024-02-12T16:31:43.826643+0000","last_undegraded":"2024-02-12T16:31:43.826643+0000","last_fullsized":"2024-02-12T16:31:43.826643+0000","mapping_epoch":636,"log_start":"646'1400","ondisk_log_start":"646'1400","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"656'6950","last_scrub_stamp":"2024-02-12T16:30:14.669359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:14.669359+0000","log_size":10335,"ondisk_log_size":10335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":246120448,"num_objects":3755,"num_object_clones":0,"num_object_copies":7510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":312896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":387,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.082305+0000","last_change":"2024-02-12T16:31:30.561239+0000","last_active":"2024-02-12T16:31:42.082305+0000","last_peered":"2024-02-12T16:31:42.082305+0000","last_clean":"2024-02-12T16:31:42.082305+0000","last_became_active":"2024-02-12T16:30:49.373979+0000","last_became_peered":"2024-02-12T16:30:49.373979+0000","last_unstale":"2024-02-12T16:31:42.082305+0000","last_undegraded":"2024-02-12T16:31:42.082305+0000","last_fullsized":"2024-02-12T16:31:42.082305+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:30.561077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:30.561077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"694'11375","reported_seq":11453,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-02-12T16:31:45.323569+0000","last_change":"2024-02-12T16:31:37.183574+0000","last_active":"2024-02-12T16:31:45.323569+0000","last_peered":"2024-02-12T16:31:45.323569+0000","last_clean":"2024-02-12T16:31:45.323569+0000","last_became_active":"2024-02-12T16:31:29.824700+0000","last_became_peered":"2024-02-12T16:31:29.824700+0000","last_unstale":"2024-02-12T16:31:45.323569+0000","last_undegraded":"2024-02-12T16:31:45.323569+0000","last_fullsized":"2024-02-12T16:31:45.323569+0000","mapping_epoch":636,"log_start":"646'1000","ondisk_log_start":"646'1000","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'11312","last_scrub_stamp":"2024-02-12T16:31:37.183512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:31:37.183512+0000","log_size":10375,"ondisk_log_size":10375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":235274240,"num_objects":3590,"num_object_clones":0,"num_object_copies":7180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":390,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.546633+0000","last_change":"2024-02-12T16:31:43.547269+0000","last_active":"2024-02-12T16:31:43.546633+0000","last_peered":"2024-02-12T16:31:43.546633+0000","last_clean":"2024-02-12T16:31:43.546633+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:31:43.546633+0000","last_undegraded":"2024-02-12T16:31:43.546633+0000","last_fullsized":"2024-02-12T16:31:43.546633+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:25.021151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:30:25.021151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"690'11517","reported_seq":11592,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.438660+0000","last_change":"2024-02-12T16:31:29.787484+0000","last_active":"2024-02-12T16:31:41.438660+0000","last_peered":"2024-02-12T16:31:41.438660+0000","last_clean":"2024-02-12T16:31:41.438660+0000","last_became_active":"2024-02-12T16:31:29.775215+0000","last_became_peered":"2024-02-12T16:31:29.775215+0000","last_unstale":"2024-02-12T16:31:41.438660+0000","last_undegraded":"2024-02-12T16:31:41.438660+0000","last_fullsized":"2024-02-12T16:31:41.438660+0000","mapping_epoch":636,"log_start":"646'1200","ondisk_log_start":"646'1200","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7916","last_scrub_stamp":"2024-02-12T16:30:27.112924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:27.112924+0000","log_size":10317,"ondisk_log_size":10317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":241565696,"num_objects":3686,"num_object_clones":0,"num_object_copies":7372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5886,"num_write_kb":306304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":245,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.863233+0000","last_change":"2024-02-12T16:30:10.604389+0000","last_active":"2024-02-12T16:31:43.863233+0000","last_peered":"2024-02-12T16:31:43.863233+0000","last_clean":"2024-02-12T16:31:43.863233+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:31:43.863233+0000","last_undegraded":"2024-02-12T16:31:43.863233+0000","last_fullsized":"2024-02-12T16:31:43.863233+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:10.604303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:10.604303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"692'11567","reported_seq":11644,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.411866+0000","last_change":"2024-02-12T16:31:29.811187+0000","last_active":"2024-02-12T16:31:43.411866+0000","last_peered":"2024-02-12T16:31:43.411866+0000","last_clean":"2024-02-12T16:31:43.411866+0000","last_became_active":"2024-02-12T16:31:29.799026+0000","last_became_peered":"2024-02-12T16:31:29.799026+0000","last_unstale":"2024-02-12T16:31:43.411866+0000","last_undegraded":"2024-02-12T16:31:43.411866+0000","last_fullsized":"2024-02-12T16:31:43.411866+0000","mapping_epoch":636,"log_start":"646'1300","ondisk_log_start":"646'1300","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7944","last_scrub_stamp":"2024-02-12T16:30:27.497708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:27.497708+0000","log_size":10267,"ondisk_log_size":10267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":242843648,"num_objects":3705,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":307840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":815,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.844847+0000","last_change":"2024-02-12T16:30:09.726295+0000","last_active":"2024-02-12T16:31:41.844847+0000","last_peered":"2024-02-12T16:31:41.844847+0000","last_clean":"2024-02-12T16:31:41.844847+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:31:41.844847+0000","last_undegraded":"2024-02-12T16:31:41.844847+0000","last_fullsized":"2024-02-12T16:31:41.844847+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.726187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.726187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"690'11722","reported_seq":11789,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.531956+0000","last_change":"2024-02-12T16:30:21.757551+0000","last_active":"2024-02-12T16:31:41.531956+0000","last_peered":"2024-02-12T16:31:41.531956+0000","last_clean":"2024-02-12T16:31:41.531956+0000","last_became_active":"2024-02-12T16:29:08.804047+0000","last_became_peered":"2024-02-12T16:29:08.804047+0000","last_unstale":"2024-02-12T16:31:41.531956+0000","last_undegraded":"2024-02-12T16:31:41.531956+0000","last_fullsized":"2024-02-12T16:31:41.531956+0000","mapping_epoch":636,"log_start":"646'1700","ondisk_log_start":"646'1700","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7354","last_scrub_stamp":"2024-02-12T16:30:21.757462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:21.757462+0000","log_size":10022,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":479068160,"num_objects":7310,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11722,"num_write_kb":609024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":203,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.845281+0000","last_change":"2024-02-12T16:31:30.659567+0000","last_active":"2024-02-12T16:31:41.845281+0000","last_peered":"2024-02-12T16:31:41.845281+0000","last_clean":"2024-02-12T16:31:41.845281+0000","last_became_active":"2024-02-12T16:30:49.374934+0000","last_became_peered":"2024-02-12T16:30:49.374934+0000","last_unstale":"2024-02-12T16:31:41.845281+0000","last_undegraded":"2024-02-12T16:31:41.845281+0000","last_fullsized":"2024-02-12T16:31:41.845281+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:30.659472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:30.659472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"694'11897","reported_seq":11982,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-02-12T16:31:45.297406+0000","last_change":"2024-02-12T16:31:30.420187+0000","last_active":"2024-02-12T16:31:45.297406+0000","last_peered":"2024-02-12T16:31:45.297406+0000","last_clean":"2024-02-12T16:31:45.297406+0000","last_became_active":"2024-02-12T16:29:08.734174+0000","last_became_peered":"2024-02-12T16:29:08.734174+0000","last_unstale":"2024-02-12T16:31:45.297406+0000","last_undegraded":"2024-02-12T16:31:45.297406+0000","last_fullsized":"2024-02-12T16:31:45.297406+0000","mapping_epoch":636,"log_start":"646'1800","ondisk_log_start":"646'1800","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'11328","last_scrub_stamp":"2024-02-12T16:31:30.420116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:31:30.420116+0000","log_size":10097,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":479657984,"num_objects":7319,"num_object_clones":0,"num_object_copies":14638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":614912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":230,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.863159+0000","last_change":"2024-02-12T16:31:32.200564+0000","last_active":"2024-02-12T16:31:43.863159+0000","last_peered":"2024-02-12T16:31:43.863159+0000","last_clean":"2024-02-12T16:31:43.863159+0000","last_became_active":"2024-02-12T16:30:49.382268+0000","last_became_peered":"2024-02-12T16:30:49.382268+0000","last_unstale":"2024-02-12T16:31:43.863159+0000","last_undegraded":"2024-02-12T16:31:43.863159+0000","last_fullsized":"2024-02-12T16:31:43.863159+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:32.200435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:32.200435+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"692'16576","reported_seq":16951,"reported_epoch":692,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:31:43.424963+0000","last_change":"2024-02-12T16:31:29.797813+0000","last_active":"2024-02-12T16:31:43.424963+0000","last_peered":"2024-02-12T16:31:43.424963+0000","last_clean":"2024-02-12T16:31:14.115119+0000","last_became_active":"2024-02-12T16:31:29.786496+0000","last_became_peered":"2024-02-12T16:31:29.786496+0000","last_unstale":"2024-02-12T16:31:43.424963+0000","last_undegraded":"2024-02-12T16:31:43.424963+0000","last_fullsized":"2024-02-12T16:31:43.424963+0000","mapping_epoch":671,"log_start":"655'6203","ondisk_log_start":"655'6203","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10373,"ondisk_log_size":10373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":241369088,"num_objects":3683,"num_object_clones":0,"num_object_copies":7366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3614,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5791,"num_write_kb":303168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4751360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,8],"avail_no_missing":["7","8"],"object_location_counts":[{"shards":"7,8","objects":3683}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":394,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.867021+0000","last_change":"2024-02-12T16:31:29.414390+0000","last_active":"2024-02-12T16:31:43.867021+0000","last_peered":"2024-02-12T16:31:43.867021+0000","last_clean":"2024-02-12T16:31:43.867021+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:31:43.867021+0000","last_undegraded":"2024-02-12T16:31:43.867021+0000","last_fullsized":"2024-02-12T16:31:43.867021+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:29.414184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:40.694324+0000","last_clean_scrub_stamp":"2024-02-12T16:31:29.414184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"690'15935","reported_seq":16010,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.013279+0000","last_change":"2024-02-12T16:31:29.792783+0000","last_active":"2024-02-12T16:31:41.013279+0000","last_peered":"2024-02-12T16:31:41.013279+0000","last_clean":"2024-02-12T16:31:41.013279+0000","last_became_active":"2024-02-12T16:31:29.779086+0000","last_became_peered":"2024-02-12T16:31:29.779086+0000","last_unstale":"2024-02-12T16:31:41.013279+0000","last_undegraded":"2024-02-12T16:31:41.013279+0000","last_fullsized":"2024-02-12T16:31:41.013279+0000","mapping_epoch":636,"log_start":"654'5601","ondisk_log_start":"654'5601","created":636,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:29:07.704868+0000","log_size":10334,"ondisk_log_size":10334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":238682112,"num_objects":3642,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":302752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":206,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.889313+0000","last_change":"2024-02-12T16:31:30.982791+0000","last_active":"2024-02-12T16:31:42.889313+0000","last_peered":"2024-02-12T16:31:42.889313+0000","last_clean":"2024-02-12T16:31:42.889313+0000","last_became_active":"2024-02-12T16:30:49.381004+0000","last_became_peered":"2024-02-12T16:30:49.381004+0000","last_unstale":"2024-02-12T16:31:42.889313+0000","last_undegraded":"2024-02-12T16:31:42.889313+0000","last_fullsized":"2024-02-12T16:31:42.889313+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:30.982709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:30.982709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"692'11770","reported_seq":11838,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.837459+0000","last_change":"2024-02-12T16:30:23.174523+0000","last_active":"2024-02-12T16:31:43.837459+0000","last_peered":"2024-02-12T16:31:43.837459+0000","last_clean":"2024-02-12T16:31:43.837459+0000","last_became_active":"2024-02-12T16:29:08.735885+0000","last_became_peered":"2024-02-12T16:29:08.735885+0000","last_unstale":"2024-02-12T16:31:43.837459+0000","last_undegraded":"2024-02-12T16:31:43.837459+0000","last_fullsized":"2024-02-12T16:31:43.837459+0000","mapping_epoch":636,"log_start":"646'1700","ondisk_log_start":"646'1700","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'7481","last_scrub_stamp":"2024-02-12T16:30:23.174453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:23.174453+0000","log_size":10070,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":478937088,"num_objects":7308,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11770,"num_write_kb":610496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":798,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.844746+0000","last_change":"2024-02-12T16:30:49.405085+0000","last_active":"2024-02-12T16:31:41.844746+0000","last_peered":"2024-02-12T16:31:41.844746+0000","last_clean":"2024-02-12T16:31:41.844746+0000","last_became_active":"2024-02-12T16:30:49.404783+0000","last_became_peered":"2024-02-12T16:30:49.404783+0000","last_unstale":"2024-02-12T16:31:41.844746+0000","last_undegraded":"2024-02-12T16:31:41.844746+0000","last_fullsized":"2024-02-12T16:31:41.844746+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:24.570146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:24.570146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"690'16871","reported_seq":16942,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.514430+0000","last_change":"2024-02-12T16:31:35.809107+0000","last_active":"2024-02-12T16:31:41.514430+0000","last_peered":"2024-02-12T16:31:41.514430+0000","last_clean":"2024-02-12T16:31:41.514430+0000","last_became_active":"2024-02-12T16:29:08.834297+0000","last_became_peered":"2024-02-12T16:29:08.834297+0000","last_unstale":"2024-02-12T16:31:41.514430+0000","last_undegraded":"2024-02-12T16:31:41.514430+0000","last_fullsized":"2024-02-12T16:31:41.514430+0000","mapping_epoch":636,"log_start":"656'6801","ondisk_log_start":"656'6801","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"685'16675","last_scrub_stamp":"2024-02-12T16:31:35.809006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:31:35.809006+0000","log_size":10070,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":478019584,"num_objects":7294,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":610880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":389,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:40.945761+0000","last_change":"2024-02-12T16:31:28.087294+0000","last_active":"2024-02-12T16:31:40.945761+0000","last_peered":"2024-02-12T16:31:40.945761+0000","last_clean":"2024-02-12T16:31:40.945761+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:31:40.945761+0000","last_undegraded":"2024-02-12T16:31:40.945761+0000","last_fullsized":"2024-02-12T16:31:40.945761+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:28.087212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:28.087212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"694'11939","reported_seq":12008,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-02-12T16:31:45.337706+0000","last_change":"2024-02-12T16:30:17.583794+0000","last_active":"2024-02-12T16:31:45.337706+0000","last_peered":"2024-02-12T16:31:45.337706+0000","last_clean":"2024-02-12T16:31:45.337706+0000","last_became_active":"2024-02-12T16:29:08.804528+0000","last_became_peered":"2024-02-12T16:29:08.804528+0000","last_unstale":"2024-02-12T16:31:45.337706+0000","last_undegraded":"2024-02-12T16:31:45.337706+0000","last_fullsized":"2024-02-12T16:31:45.337706+0000","mapping_epoch":636,"log_start":"646'1900","ondisk_log_start":"646'1900","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"657'7134","last_scrub_stamp":"2024-02-12T16:30:17.583727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:17.583727+0000","log_size":10039,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":474808320,"num_objects":7245,"num_object_clones":0,"num_object_copies":14490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11939,"num_write_kb":613888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":143,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.863532+0000","last_change":"2024-02-12T16:31:29.396957+0000","last_active":"2024-02-12T16:31:42.863532+0000","last_peered":"2024-02-12T16:31:42.863532+0000","last_clean":"2024-02-12T16:31:42.863532+0000","last_became_active":"2024-02-12T16:30:49.376219+0000","last_became_peered":"2024-02-12T16:30:49.376219+0000","last_unstale":"2024-02-12T16:31:42.863532+0000","last_undegraded":"2024-02-12T16:31:42.863532+0000","last_fullsized":"2024-02-12T16:31:42.863532+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:29.396872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:29.396872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"690'11764","reported_seq":11832,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.548577+0000","last_change":"2024-02-12T16:30:36.029310+0000","last_active":"2024-02-12T16:31:41.548577+0000","last_peered":"2024-02-12T16:31:41.548577+0000","last_clean":"2024-02-12T16:31:41.548577+0000","last_became_active":"2024-02-12T16:29:08.735711+0000","last_became_peered":"2024-02-12T16:29:08.735711+0000","last_unstale":"2024-02-12T16:31:41.548577+0000","last_undegraded":"2024-02-12T16:31:41.548577+0000","last_fullsized":"2024-02-12T16:31:41.548577+0000","mapping_epoch":636,"log_start":"646'1700","ondisk_log_start":"646'1700","created":636,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8520","last_scrub_stamp":"2024-02-12T16:30:36.029199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:36.029199+0000","log_size":10064,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":488570904,"num_objects":7456,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11764,"num_write_kb":614977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":380,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T16:31:39.837568+0000","last_change":"2024-02-12T16:31:29.008755+0000","last_active":"2024-02-12T16:31:39.837568+0000","last_peered":"2024-02-12T16:31:39.837568+0000","last_clean":"2024-02-12T16:31:39.837568+0000","last_became_active":"2024-02-12T16:30:49.384015+0000","last_became_peered":"2024-02-12T16:30:49.384015+0000","last_unstale":"2024-02-12T16:31:39.837568+0000","last_undegraded":"2024-02-12T16:31:39.837568+0000","last_fullsized":"2024-02-12T16:31:39.837568+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:29.008672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:31:29.008672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.17","version":"690'11471","reported_seq":11550,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:42.018498+0000","last_change":"2024-02-12T16:31:42.018498+0000","last_active":"2024-02-12T16:31:42.018498+0000","last_peered":"2024-02-12T16:31:42.018498+0000","last_clean":"2024-02-12T16:31:42.018498+0000","last_became_active":"2024-02-12T16:31:29.811685+0000","last_became_peered":"2024-02-12T16:31:29.811685+0000","last_unstale":"2024-02-12T16:31:42.018498+0000","last_undegraded":"2024-02-12T16:31:42.018498+0000","last_fullsized":"2024-02-12T16:31:42.018498+0000","mapping_epoch":680,"log_start":"646'1200","ondisk_log_start":"646'1200","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"661'8268","last_scrub_stamp":"2024-02-12T16:30:33.673564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:33.673564+0000","log_size":10271,"ondisk_log_size":10271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":237928448,"num_objects":3631,"num_object_clones":0,"num_object_copies":7262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5863,"num_write_kb":303776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":190,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:40.945780+0000","last_change":"2024-02-12T16:31:32.110358+0000","last_active":"2024-02-12T16:31:40.945780+0000","last_peered":"2024-02-12T16:31:40.945780+0000","last_clean":"2024-02-12T16:31:40.945780+0000","last_became_active":"2024-02-12T16:23:39.023021+0000","last_became_peered":"2024-02-12T16:23:39.023021+0000","last_unstale":"2024-02-12T16:31:40.945780+0000","last_undegraded":"2024-02-12T16:31:40.945780+0000","last_fullsized":"2024-02-12T16:31:40.945780+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:32.110222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:31:32.110222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"694'11376","reported_seq":11453,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-02-12T16:31:45.329482+0000","last_change":"2024-02-12T16:31:39.879746+0000","last_active":"2024-02-12T16:31:45.329482+0000","last_peered":"2024-02-12T16:31:45.329482+0000","last_clean":"2024-02-12T16:31:45.329482+0000","last_became_active":"2024-02-12T16:31:29.835922+0000","last_became_peered":"2024-02-12T16:31:29.835922+0000","last_unstale":"2024-02-12T16:31:45.329482+0000","last_undegraded":"2024-02-12T16:31:45.329482+0000","last_fullsized":"2024-02-12T16:31:45.329482+0000","mapping_epoch":680,"log_start":"646'1000","ondisk_log_start":"646'1000","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"688'11316","last_scrub_stamp":"2024-02-12T16:31:39.879624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:31:39.879624+0000","log_size":10376,"ondisk_log_size":10376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":234356736,"num_objects":3576,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":188,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T16:31:39.916270+0000","last_change":"2024-02-12T16:31:30.886757+0000","last_active":"2024-02-12T16:31:39.916270+0000","last_peered":"2024-02-12T16:31:39.916270+0000","last_clean":"2024-02-12T16:31:39.916270+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:31:39.916270+0000","last_undegraded":"2024-02-12T16:31:39.916270+0000","last_fullsized":"2024-02-12T16:31:39.916270+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:30.886627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:31:30.886627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"692'11585","reported_seq":11661,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-02-12T16:31:43.292588+0000","last_change":"2024-02-12T16:31:29.831778+0000","last_active":"2024-02-12T16:31:43.292588+0000","last_peered":"2024-02-12T16:31:43.292588+0000","last_clean":"2024-02-12T16:31:43.292588+0000","last_became_active":"2024-02-12T16:31:29.811228+0000","last_became_peered":"2024-02-12T16:31:29.811228+0000","last_unstale":"2024-02-12T16:31:43.292588+0000","last_undegraded":"2024-02-12T16:31:43.292588+0000","last_fullsized":"2024-02-12T16:31:43.292588+0000","mapping_epoch":680,"log_start":"646'1300","ondisk_log_start":"646'1300","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"658'7944","last_scrub_stamp":"2024-02-12T16:30:27.497708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:27.497708+0000","log_size":10285,"ondisk_log_size":10285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":241664000,"num_objects":3688,"num_object_clones":0,"num_object_copies":7376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5933,"num_write_kb":307840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":206,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:40.945856+0000","last_change":"2024-02-12T16:31:31.191622+0000","last_active":"2024-02-12T16:31:40.945856+0000","last_peered":"2024-02-12T16:31:40.945856+0000","last_clean":"2024-02-12T16:31:40.945856+0000","last_became_active":"2024-02-12T16:25:13.022182+0000","last_became_peered":"2024-02-12T16:25:13.022182+0000","last_unstale":"2024-02-12T16:31:40.945856+0000","last_undegraded":"2024-02-12T16:31:40.945856+0000","last_fullsized":"2024-02-12T16:31:40.945856+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:31:31.191518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:31:31.191518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"690'11524","reported_seq":11598,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-02-12T16:31:41.352271+0000","last_change":"2024-02-12T16:31:29.816486+0000","last_active":"2024-02-12T16:31:41.352271+0000","last_peered":"2024-02-12T16:31:41.352271+0000","last_clean":"2024-02-12T16:31:41.352271+0000","last_became_active":"2024-02-12T16:31:29.806445+0000","last_became_peered":"2024-02-12T16:31:29.806445+0000","last_unstale":"2024-02-12T16:31:41.352271+0000","last_undegraded":"2024-02-12T16:31:41.352271+0000","last_fullsized":"2024-02-12T16:31:41.352271+0000","mapping_epoch":680,"log_start":"646'1200","ondisk_log_start":"646'1200","created":680,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"658'7916","last_scrub_stamp":"2024-02-12T16:30:27.112924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:29:07.704868+0000","last_clean_scrub_stamp":"2024-02-12T16:30:27.112924+0000","log_size":10324,"ondisk_log_size":10324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":241106944,"num_objects":3679,"num_object_clones":0,"num_object_copies":7358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":306304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":181,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-02-12T16:31:40.023797+0000","last_change":"2024-02-12T16:31:38.996532+0000","last_active":"2024-02-12T16:31:40.023797+0000","last_peered":"2024-02-12T16:31:40.023797+0000","last_clean":"2024-02-12T16:31:40.023797+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:31:40.023797+0000","last_undegraded":"2024-02-12T16:31:40.023797+0000","last_fullsized":"2024-02-12T16:31:40.023797+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:30:09.745399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:30:09.745399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":7675019288,"num_objects":117113,"num_object_clones":0,"num_object_copies":234226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30463,"num_objects_unfound":0,"num_objects_dirty":117113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171367,"num_write_kb":8874657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":95092736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15580405760,"data_stored":15580397616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267104,"ondisk_log_size":267104,"up":52,"acting":52,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958931,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692608,"kb_used_data":102904,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":93052544,"statfs":{"total":95995035648,"available":95285805056,"internally_reserved":0,"allocated":105373696,"data_stored":104828832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":3.6789999999999998,"5min":1.3440000000000001,"15min":1.0580000000000001},"min":{"1min":0.32600000000000001,"5min":0.27400000000000002,"15min":0.246},"max":{"1min":33.941000000000003,"5min":33.941000000000003,"15min":33.941000000000003},"last":0.376},{"interface":"front","average":{"1min":3.8740000000000001,"5min":1.538,"15min":1.1160000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28399999999999997},"max":{"1min":33.890000000000001,"5min":33.890000000000001,"15min":33.890000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.97899999999999998,"15min":1.0069999999999999},"min":{"1min":0.35899999999999999,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":0.71299999999999997,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.81100000000000005,"15min":0.98799999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1950000000000001,"5min":3.6859999999999999,"15min":5.5590000000000002},"last":0.65800000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":2.4889999999999999,"5min":1.262,"15min":1.2789999999999999},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":31.326000000000001,"5min":31.326000000000001,"15min":31.326000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":2.5019999999999998,"5min":1.2310000000000001,"15min":1.3320000000000001},"min":{"1min":0.41999999999999998,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":31.234000000000002,"5min":31.234000000000002,"15min":31.234000000000002},"last":0.48999999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.92300000000000004,"15min":0.98599999999999999},"min":{"1min":0.33100000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.2489999999999997,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.85399999999999998,"15min":0.93300000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.34499999999999997},"max":{"1min":2.5409999999999999,"5min":4.101,"15min":4.915},"last":0.58599999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1279999999999999,"15min":1.1870000000000001},"min":{"1min":0.378,"5min":0.36199999999999999,"15min":0.35299999999999998},"max":{"1min":4.3760000000000003,"5min":5.968,"15min":5.968},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.147,"5min":1.115,"15min":1.1779999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.35499999999999998},"max":{"1min":4.266,"5min":5.8879999999999999,"15min":5.8879999999999999},"last":0.64200000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.0960000000000001,"15min":1.046},"min":{"1min":0.441,"5min":0.378,"15min":0.373},"max":{"1min":6.4160000000000004,"5min":6.4160000000000004,"15min":6.4160000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.448,"5min":1.171,"15min":1.1679999999999999},"min":{"1min":0.505,"5min":0.46400000000000002,"15min":0.40000000000000002},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":8.5709999999999997},"last":0.626}]},{"osd":7,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.0489999999999999,"15min":1.2230000000000001},"min":{"1min":0.47399999999999998,"5min":0.46899999999999997,"15min":0.38600000000000001},"max":{"1min":3.4529999999999998,"5min":3.9089999999999998,"15min":6.4370000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":0.98899999999999999,"15min":1.1160000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.137,"5min":4.2350000000000003,"15min":5.0430000000000001},"last":0.74099999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.80700000000000005,"15min":2.7749999999999999},"min":{"1min":0.35099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.1389999999999998,"5min":3.0329999999999999,"15min":537.54399999999998},"last":0.504},{"interface":"front","average":{"1min":0.749,"5min":0.89300000000000002,"15min":2.944},"min":{"1min":0.41499999999999998,"5min":0.41099999999999998,"15min":0.34699999999999998},"max":{"1min":1.1220000000000001,"5min":3.8719999999999999,"15min":539.86099999999999},"last":0.73399999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.0109999999999999,"15min":1.2270000000000001},"min":{"1min":0.55100000000000005,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2509999999999999,"5min":3.8100000000000001,"15min":5.843},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.96999999999999997,"15min":1.2909999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.173,"5min":3.8399999999999999,"15min":6.8869999999999996},"last":0.78400000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:31:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.96099999999999997,"15min":2.919},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.8929999999999998,"5min":4.0599999999999996,"15min":536.245},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.88300000000000001,"15min":2.9209999999999998},"min":{"1min":0.49199999999999999,"5min":0.432,"15min":0.432},"max":{"1min":0.88100000000000001,"5min":2.544,"15min":536.64300000000003},"last":0.70699999999999996}]}]},{"osd":10,"up_from":83,"seq":356482285980,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":767540,"kb_used_data":466220,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92977612,"statfs":{"total":95995035648,"available":95209074688,"internally_reserved":0,"allocated":477409280,"data_stored":476870215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.92100000000000004,"15min":1.5089999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.245},"max":{"1min":4.1769999999999996,"5min":15.010999999999999,"15min":162.38399999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":1.0089999999999999,"15min":1.5780000000000001},"min":{"1min":0.311,"5min":0.28699999999999998,"15min":0.26500000000000001},"max":{"1min":4.1909999999999998,"5min":14.129,"15min":162.01300000000001},"last":0.379}]},{"osd":1,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.90700000000000003,"15min":0.998},"min":{"1min":0.32600000000000001,"5min":0.30299999999999999,"15min":0.29599999999999999},"max":{"1min":3.1520000000000001,"5min":5.9000000000000004,"15min":19.866},"last":0.44},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.131},"min":{"1min":0.32800000000000001,"5min":0.318,"15min":0.27700000000000002},"max":{"1min":4.2309999999999999,"5min":6.1970000000000001,"15min":20.277999999999999},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.1359999999999999,"15min":1.0089999999999999},"min":{"1min":0.35999999999999999,"5min":0.29899999999999999,"15min":0.28899999999999998},"max":{"1min":5.016,"5min":8.3200000000000003,"15min":8.3200000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.083,"15min":1.0600000000000001},"min":{"1min":0.42299999999999999,"5min":0.34699999999999998,"15min":0.32000000000000001},"max":{"1min":5.1820000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:16:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":0.95899999999999996,"15min":1.006},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.7170000000000001,"5min":4.7229999999999999,"15min":4.8620000000000001},"last":0.378},{"interface":"front","average":{"1min":0.62,"5min":0.90500000000000003,"15min":0.93100000000000005},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29599999999999999},"max":{"1min":1.9330000000000001,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":1.4790000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.0209999999999999,"15min":1.0740000000000001},"min":{"1min":0.46300000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.3689999999999998,"5min":6.3689999999999998,"15min":6.3689999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":1.03,"15min":1.04},"min":{"1min":0.34200000000000003,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":1.8440000000000001,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.0369999999999999,"15min":1.0700000000000001},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.8740000000000001,"5min":6.343,"15min":6.343},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.2689999999999999,"15min":1.248},"min":{"1min":0.53200000000000003,"5min":0.40999999999999998,"15min":0.35799999999999998},"max":{"1min":7.2249999999999996,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.03,"15min":1.0880000000000001},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":4.907,"5min":6.3010000000000002,"15min":6.3010000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.175,"15min":1.196},"min":{"1min":0.53700000000000003,"5min":0.438,"15min":0.376},"max":{"1min":4.3689999999999998,"5min":6.8200000000000003,"15min":6.8200000000000003},"last":0.71999999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.099,"15min":1.0940000000000001},"min":{"1min":0.47999999999999998,"5min":0.375,"15min":0.375},"max":{"1min":3.0249999999999999,"5min":7.3280000000000003,"15min":7.3280000000000003},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1519999999999999,"15min":1.2230000000000001},"min":{"1min":0.46500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":4.0659999999999998,"5min":6.4539999999999997,"15min":6.4539999999999997},"last":0.84599999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.1240000000000001,"15min":1.1719999999999999},"min":{"1min":0.51400000000000001,"5min":0.40400000000000003,"15min":0.372},"max":{"1min":4.3399999999999999,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.98999999999999999,"15min":1.129},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.35799999999999998},"max":{"1min":1.097,"5min":7.282,"15min":7.282},"last":0.92600000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:31:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.1200000000000001,"15min":1.2370000000000001},"min":{"1min":0.61499999999999999,"5min":0.505,"15min":0.372},"max":{"1min":1.204,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.876},{"interface":"front","average":{"1min":0.76400000000000001,"5min":1.04,"15min":1.1930000000000001},"min":{"1min":0.433,"5min":0.42899999999999999,"15min":0.41399999999999998},"max":{"1min":1.2490000000000001,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.79200000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:31:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.1930000000000001,"15min":1.248},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.39300000000000002},"max":{"1min":7.3929999999999998,"5min":7.3929999999999998,"15min":7.3929999999999998},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.1759999999999999,"15min":1.292},"min":{"1min":0.58999999999999997,"5min":0.497,"15min":0.47899999999999998},"max":{"1min":4.3929999999999998,"5min":6.5860000000000003,"15min":7.3250000000000002},"last":0.90200000000000002}]}]},{"osd":9,"up_from":74,"seq":317827580324,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1254044,"kb_used_data":948628,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":92491108,"statfs":{"total":95995035648,"available":94710894592,"internally_reserved":0,"allocated":971395072,"data_stored":970867452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":1.2589999999999999,"15min":1.1020000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71499999999999997,"5min":19.120000000000001,"15min":19.120000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":1.3440000000000001,"15min":1.0329999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.28299999999999997},"max":{"1min":0.88400000000000001,"5min":19.222000000000001,"15min":19.222000000000001},"last":0.627}]},{"osd":1,"last update":"Mon Feb 12 16:31:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":1.0249999999999999,"15min":0.98899999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.253},"max":{"1min":3.5129999999999999,"5min":5.7750000000000004,"15min":5.8819999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.034,"15min":1},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":4.0499999999999998,"5min":5.7910000000000004,"15min":7.9550000000000001},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.196,"15min":1.4419999999999999},"min":{"1min":0.33700000000000002,"5min":0.32500000000000001,"15min":0.27600000000000002},"max":{"1min":3.3370000000000002,"5min":12.441000000000001,"15min":97.275000000000006},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.147,"15min":1.359},"min":{"1min":0.33500000000000002,"5min":0.33000000000000002,"15min":0.312},"max":{"1min":4.0019999999999998,"5min":4.6239999999999997,"15min":97.417000000000002},"last":0.90600000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.361,"15min":1.2150000000000001},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":10.206,"5min":34.344000000000001,"15min":34.591000000000001},"last":3.4950000000000001},{"interface":"front","average":{"1min":1.9650000000000001,"5min":1.6679999999999999,"15min":1.4239999999999999},"min":{"1min":0.504,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":6.2880000000000003,"5min":34.389000000000003,"15min":34.494},"last":3.5089999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.0800000000000001,"15min":1.0169999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.8900000000000001,"5min":6.3090000000000002,"15min":6.3090000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.1619999999999999,"15min":1.161},"min":{"1min":0.47799999999999998,"5min":0.38,"15min":0.36899999999999999},"max":{"1min":3.9289999999999998,"5min":6.3490000000000002,"15min":6.3490000000000002},"last":0.86499999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.2390000000000001,"15min":1.1839999999999999},"min":{"1min":0.378,"5min":0.371,"15min":0.35799999999999998},"max":{"1min":6.8559999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.2090000000000001,"15min":1.159},"min":{"1min":0.46999999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.97299999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.0880000000000001,"15min":1.087},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.378},"max":{"1min":4.3559999999999999,"5min":5.6689999999999996,"15min":5.6689999999999996},"last":1.022},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.278,"15min":1.147},"min":{"1min":0.379,"5min":0.379,"15min":0.36499999999999999},"max":{"1min":7.71,"5min":14.718,"15min":14.718},"last":0.94999999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.194,"15min":1.242},"min":{"1min":0.502,"5min":0.433,"15min":0.41899999999999998},"max":{"1min":5.3479999999999999,"5min":5.3479999999999999,"15min":5.8010000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.256,"5min":1.1919999999999999,"15min":1.214},"min":{"1min":0.45600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.1870000000000003,"5min":4.7000000000000002,"15min":7.9160000000000004},"last":2.54}]},{"osd":8,"last update":"Mon Feb 12 16:31:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0209999999999999,"15min":1.105},"min":{"1min":0.42599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.1100000000000003,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":1.01},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.1020000000000001,"15min":1.1859999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.42599999999999999},"max":{"1min":2.9260000000000002,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":1.054}]},{"osd":10,"last update":"Mon Feb 12 16:30:46 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.089,"15min":1.252},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36599999999999999},"max":{"1min":1.0349999999999999,"5min":4.6900000000000004,"15min":5.6929999999999996},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.875,"5min":1.2170000000000001,"15min":1.2629999999999999},"min":{"1min":0.51400000000000001,"5min":0.44700000000000001,"15min":0.38},"max":{"1min":3.8380000000000001,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":1.0429999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:30:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.077,"15min":1.3360000000000001},"min":{"1min":0.499,"5min":0.47599999999999998,"15min":0.45600000000000002},"max":{"1min":1.0669999999999999,"5min":4.7990000000000004,"15min":5.7690000000000001},"last":1.036},{"interface":"front","average":{"1min":0.78200000000000003,"5min":1.022,"15min":1.2549999999999999},"min":{"1min":0.52800000000000002,"5min":0.50700000000000001,"15min":0.41599999999999998},"max":{"1min":1.0880000000000001,"5min":4.7190000000000003,"15min":6.3639999999999999},"last":1.0960000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874670,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1255028,"kb_used_data":953708,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92490124,"statfs":{"total":95995035648,"available":94709886976,"internally_reserved":0,"allocated":976596992,"data_stored":976057927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.74099999999999999,"15min":0.88900000000000001},"min":{"1min":0.28599999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.84199999999999997,"5min":4.7270000000000003,"15min":19.196999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.86799999999999999,"15min":0.93400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29399999999999998},"max":{"1min":1.77,"5min":7.1840000000000002,"15min":18.584},"last":0.93600000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0190000000000001,"15min":1.3320000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.497},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.9870000000000001,"15min":1.4079999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4580000000000002,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.82499999999999996,"15min":0.82399999999999995},"min":{"1min":0.30599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.415,"5min":4.4160000000000004,"15min":4.4160000000000004},"last":1.2010000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.036,"15min":0.99099999999999999},"min":{"1min":0.45100000000000001,"5min":0.32100000000000001,"15min":0.29899999999999999},"max":{"1min":2.9830000000000001,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.97799999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":6.8079999999999998,"5min":10.4,"15min":10.4},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.95299999999999996,"15min":1.0129999999999999},"min":{"1min":0.33400000000000002,"5min":0.311,"15min":0.28899999999999998},"max":{"1min":6.8230000000000004,"5min":10.413,"15min":10.413},"last":0.77900000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:31:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.77400000000000002,"15min":0.95399999999999996},"min":{"1min":0.41299999999999998,"5min":0.33700000000000002,"15min":0.29599999999999999},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":5.0490000000000004},"last":1.3260000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.84499999999999997,"15min":0.94899999999999995},"min":{"1min":0.48699999999999999,"5min":0.35499999999999998,"15min":0.34699999999999998},"max":{"1min":2.762,"5min":2.762,"15min":5.0380000000000003},"last":1.0249999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1120000000000001,"15min":0.98899999999999999},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":6.4160000000000004,"5min":6.4160000000000004,"15min":6.4160000000000004},"last":1.002},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.0169999999999999,"15min":1.0589999999999999},"min":{"1min":0.502,"5min":0.40799999999999997,"15min":0.38100000000000001},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":1.097}]},{"osd":6,"last update":"Mon Feb 12 16:31:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0609999999999999,"15min":1.0660000000000001},"min":{"1min":0.499,"5min":0.373,"15min":0.373},"max":{"1min":4.1150000000000002,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.3,"5min":1.155,"15min":1.175},"min":{"1min":0.47699999999999998,"5min":0.38,"15min":0.38},"max":{"1min":5.8120000000000003,"5min":9.1110000000000007,"15min":9.1110000000000007},"last":1.1319999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:31:30 2024","interfaces":[{"interface":"back","average":{"1min":34.933999999999997,"5min":7.9669999999999996,"15min":3.4060000000000001},"min":{"1min":0.52700000000000002,"5min":0.47299999999999998,"15min":0.42199999999999999},"max":{"1min":706.67700000000002,"5min":706.67700000000002,"15min":706.67700000000002},"last":1.651},{"interface":"front","average":{"1min":35.029000000000003,"5min":8.0069999999999997,"15min":3.5449999999999999},"min":{"1min":0.57599999999999996,"5min":0.495,"15min":0.495},"max":{"1min":707.49099999999999,"5min":707.49099999999999,"15min":707.49099999999999},"last":1.1719999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.98499999999999999,"15min":1.141},"min":{"1min":0.45700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.298,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":1.2569999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":1.0860000000000001,"15min":1.2509999999999999},"min":{"1min":0.502,"5min":0.46100000000000002,"15min":0.44700000000000001},"max":{"1min":1.431,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":1.3700000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:30:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.995,"15min":1.4470000000000001},"min":{"1min":0.60999999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.587,"5min":5.2560000000000002,"15min":109.92},"last":1.5760000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.96499999999999997,"15min":1.492},"min":{"1min":0.53600000000000003,"5min":0.44400000000000001,"15min":0.38500000000000001},"max":{"1min":1.216,"5min":5.0250000000000004,"15min":110.249},"last":1.276}]},{"osd":11,"last update":"Mon Feb 12 16:30:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.0629999999999999,"15min":1.1739999999999999},"min":{"1min":0.64900000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.609,"5min":5.2530000000000001,"15min":5.2530000000000001},"last":1.7030000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.96499999999999997,"15min":1.1220000000000001},"min":{"1min":0.54300000000000004,"5min":0.48199999999999998,"15min":0.46300000000000002},"max":{"1min":1.1739999999999999,"5min":5.2320000000000002,"15min":5.2320000000000002},"last":1.1819999999999999}]}]},{"osd":7,"up_from":56,"seq":240518169013,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2192872,"kb_used_data":1887456,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":91552280,"statfs":{"total":95995035648,"available":93749534720,"internally_reserved":0,"allocated":1932754944,"data_stored":1932221446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":2.2909999999999999,"5min":3.4500000000000002,"15min":1.7769999999999999},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.252},"max":{"1min":7.4500000000000002,"5min":187.33699999999999,"15min":187.33699999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.9239999999999999,"5min":3.2709999999999999,"15min":1.7529999999999999},"min":{"1min":0.33700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.4989999999999997,"5min":187.48400000000001,"15min":187.48400000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.92700000000000005,"15min":0.96299999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.075,"15min":1.129},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.671,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.0820000000000001,"15min":1.1040000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":35.655000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.9430000000000001,"5min":1.3129999999999999,"15min":1.2849999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.6040000000000001,"5min":7.6040000000000001,"15min":35.164000000000001},"last":0.88900000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:31:23 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.873,"15min":1.1160000000000001},"min":{"1min":0.378,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.7149999999999999,"5min":4.3339999999999996,"15min":7.1689999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.897,"5min":1.2909999999999999,"15min":1.0760000000000001},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":12.193,"5min":12.193,"15min":12.193},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.1659999999999999,"15min":1.0129999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":3.9580000000000002,"5min":6.899,"15min":7.5910000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.1699999999999999,"15min":1.0649999999999999},"min":{"1min":0.40899999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.1539999999999999,"5min":6.9470000000000001,"15min":6.9470000000000001},"last":0.80000000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":14.285,"5min":3.7229999999999999,"15min":1.8959999999999999},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":216.84299999999999,"5min":216.84299999999999,"15min":216.84299999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":14.882,"5min":3.883,"15min":1.974},"min":{"1min":0.41399999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":217.02600000000001,"5min":217.02600000000001,"15min":217.02600000000001},"last":1.857}]},{"osd":6,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.1639999999999999,"15min":1.125},"min":{"1min":0.35799999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.0270000000000001,"5min":6.319,"15min":6.319},"last":0.93799999999999994},{"interface":"front","average":{"1min":2.5379999999999998,"5min":1.484,"15min":1.2989999999999999},"min":{"1min":0.52900000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":6.3460000000000001,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":1.0940000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96599999999999997,"15min":2.948},"min":{"1min":0.47899999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.6859999999999999,"5min":6.1619999999999999,"15min":498.68299999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.46,"15min":3.1389999999999998},"min":{"1min":0.49399999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":7.6859999999999999,"5min":9.9060000000000006,"15min":498.69999999999999},"last":2.2029999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.069,"15min":1.2030000000000001},"min":{"1min":0.45400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.7229999999999999,"5min":4.8650000000000002,"15min":12.907999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.607,"5min":1.4019999999999999,"15min":1.3560000000000001},"min":{"1min":0.48099999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.5469999999999997,"5min":7.1219999999999999,"15min":7.1219999999999999},"last":0.97599999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:30:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":7.9790000000000001,"15min":3.5739999999999998},"min":{"1min":0.51000000000000001,"5min":0.376,"15min":0.376},"max":{"1min":5.0839999999999996,"5min":497.52699999999999,"15min":497.52699999999999},"last":1.1859999999999999},{"interface":"front","average":{"1min":2.3540000000000001,"5min":8.0310000000000006,"15min":3.5169999999999999},"min":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":7.6470000000000002,"5min":496.29599999999999,"15min":496.29599999999999},"last":2.0880000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.099,"15min":1.601},"min":{"1min":0.52000000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.7589999999999999,"5min":4.9039999999999999,"15min":120.943},"last":1.056},{"interface":"front","average":{"1min":1.661,"5min":1.4870000000000001,"15min":1.796},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":5.5709999999999997,"5min":7.0679999999999996,"15min":120.661},"last":1.3049999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430654,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2180736,"kb_used_data":1871224,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":91564416,"statfs":{"total":95995035648,"available":93761961984,"internally_reserved":0,"allocated":1916133376,"data_stored":1915588512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":2.1539999999999999,"15min":1.4470000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7350000000000001,"5min":107.723,"15min":107.723},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":2.23,"15min":1.5349999999999999},"min":{"1min":0.38300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.1180000000000003,"5min":107.114,"15min":107.114},"last":0.92100000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.0429999999999999,"15min":1.9279999999999999},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.28199999999999997},"max":{"1min":3.7610000000000001,"5min":6.1870000000000003,"15min":252.22800000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.1990000000000001,"15min":2.0219999999999998},"min":{"1min":0.26100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.601,"5min":6.6959999999999997,"15min":252.61199999999999},"last":1.6200000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.7,"5min":1.3700000000000001,"15min":1.1619999999999999},"min":{"1min":0.33000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":7.8769999999999998,"5min":7.8769999999999998,"15min":28.739999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":2.1539999999999999,"5min":1.4750000000000001,"15min":1.278},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.28999999999999998},"max":{"1min":8.5129999999999999,"5min":8.5129999999999999,"15min":32.328000000000003},"last":1.6539999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.93000000000000005,"15min":1.1719999999999999},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":5.444,"5min":5.444,"15min":68.332999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.0069999999999999,"15min":1.206},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.32600000000000001},"max":{"1min":3.5169999999999999,"5min":5.1790000000000003,"15min":67.281999999999996},"last":2.3580000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2889999999999999,"15min":1.093},"min":{"1min":0.28299999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.4559999999999995,"5min":8.4559999999999995,"15min":8.4559999999999995},"last":3.0630000000000002},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.018,"15min":1.054},"min":{"1min":0.42399999999999999,"5min":0.28799999999999998,"15min":0.23999999999999999},"max":{"1min":3.246,"5min":6.0259999999999998,"15min":7.5979999999999999},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.3839999999999999,"15min":1.109},"min":{"1min":0.34000000000000002,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":8.5730000000000004,"5min":8.5730000000000004,"15min":8.5730000000000004},"last":3.4089999999999998},{"interface":"front","average":{"1min":1.131,"5min":1.1020000000000001,"15min":1.046},"min":{"1min":0.46000000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.6890000000000001,"5min":10.52,"15min":10.52},"last":0.52300000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.9059999999999999,"5min":1.099,"15min":1.133},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":5.9939999999999998,"5min":5.9939999999999998,"15min":7.641},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.0129999999999999,"15min":1.0609999999999999},"min":{"1min":0.50900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":7.8380000000000001},"last":0.61699999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.238,"15min":1.1870000000000001},"min":{"1min":0.50800000000000001,"5min":0.42099999999999999,"15min":0.39300000000000002},"max":{"1min":3.1640000000000001,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.198,"15min":1.2529999999999999},"min":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.371},"max":{"1min":8.5470000000000006,"5min":8.5470000000000006,"15min":8.5470000000000006},"last":3.3759999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:30:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6060000000000001,"5min":1.2,"15min":1.25},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":6.2400000000000002,"5min":6.2400000000000002,"15min":18.059999999999999},"last":1.71},{"interface":"front","average":{"1min":1.23,"5min":1.0449999999999999,"15min":1.2509999999999999},"min":{"1min":0.56000000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.2729999999999997,"5min":6.2729999999999997,"15min":17.425000000000001},"last":0.70299999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:30:52 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.419,"15min":1.353},"min":{"1min":0.432,"5min":0.432,"15min":0.42799999999999999},"max":{"1min":8.6159999999999997,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":3.4420000000000002},{"interface":"front","average":{"1min":1.9990000000000001,"5min":1.5089999999999999,"15min":1.448},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":8.7100000000000009,"5min":8.7100000000000009,"15min":8.7100000000000009},"last":3.4729999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:30:52 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.391,"15min":1.7589999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":8.6649999999999991,"5min":8.6649999999999991,"15min":139.727},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.1120000000000001,"15min":1.651},"min":{"1min":0.61099999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":6.25,"5min":6.25,"15min":139.24199999999999},"last":1.694}]}]},{"osd":1,"up_from":13,"seq":55834575331,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2327456,"kb_used_data":1425048,"kb_used_omap":0,"kb_used_meta":902400,"kb_avail":91417696,"statfs":{"total":95995035648,"available":93611720704,"internally_reserved":0,"allocated":1459249152,"data_stored":1458717460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":924057600},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.1120000000000001,"15min":0.93700000000000006},"min":{"1min":0.25600000000000001,"5min":0.23599999999999999,"15min":0.221},"max":{"1min":4.3849999999999998,"5min":16.632999999999999,"15min":16.632999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.5269999999999999,"15min":1.2370000000000001},"min":{"1min":0.28199999999999997,"5min":0.27600000000000002,"15min":0.23899999999999999},"max":{"1min":6.4050000000000002,"5min":6.5759999999999996,"15min":11.433},"last":0.86299999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:31:32 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.2749999999999999,"15min":1.133},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.4020000000000001,"5min":12.778,"15min":12.778},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.375,"15min":1.206},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.246},"max":{"1min":9.702,"5min":9.9689999999999994,"15min":9.9689999999999994},"last":0.75}]},{"osd":3,"last update":"Mon Feb 12 16:31:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.0629999999999999,"15min":1.0940000000000001},"min":{"1min":0.25900000000000001,"5min":0.248,"15min":0.248},"max":{"1min":3.2909999999999999,"5min":9.9529999999999994,"15min":9.9529999999999994},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.266,"15min":1.038},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.3689999999999998,"5min":6.5490000000000004,"15min":6.7110000000000003},"last":1.0049999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:31:25 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.2989999999999999,"15min":1.1870000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":6.4569999999999999,"5min":7.7549999999999999,"15min":8.7479999999999993},"last":1.159},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.165,"15min":1.3440000000000001},"min":{"1min":0.40100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.4669999999999996,"5min":5.633,"15min":7.1710000000000003},"last":1.0409999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:31:32 2024","interfaces":[{"interface":"back","average":{"1min":28.847000000000001,"5min":9.7059999999999995,"15min":4.0750000000000002},"min":{"1min":0.372,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":524.85299999999995,"5min":524.85299999999995,"15min":524.85299999999995},"last":0.95899999999999996},{"interface":"front","average":{"1min":28.268000000000001,"5min":9.7029999999999994,"15min":4.0289999999999999},"min":{"1min":0.38600000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":519.89700000000005,"5min":519.89700000000005,"15min":519.89700000000005},"last":1.0289999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":29.050000000000001,"5min":7.1239999999999997,"15min":3.2509999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":522.79200000000003,"5min":522.79200000000003,"15min":522.79200000000003},"last":1.0880000000000001},{"interface":"front","average":{"1min":28.265999999999998,"5min":6.9619999999999997,"15min":3.234},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":520.64499999999998,"5min":520.64499999999998,"15min":520.64499999999998},"last":0.98099999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":7.3689999999999998,"15min":3.415},"min":{"1min":0.41699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.427,"5min":548.91600000000005,"15min":548.91600000000005},"last":1.202},{"interface":"front","average":{"1min":0.89000000000000001,"5min":7.4420000000000002,"15min":3.375},"min":{"1min":0.45200000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":3.512,"5min":552.42100000000005,"15min":552.42100000000005},"last":1.2689999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.204,"15min":1.3540000000000001},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.4950000000000001,"5min":5.7590000000000003,"15min":5.7590000000000003},"last":1.1839999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.2609999999999999,"15min":1.4199999999999999},"min":{"1min":0.505,"5min":0.436,"15min":0.436},"max":{"1min":2.6139999999999999,"5min":10.106,"15min":10.106},"last":1.3680000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:30:48 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":8.3170000000000002,"15min":3.738},"min":{"1min":0.53600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.5329999999999999,"5min":548.93499999999995,"15min":548.93499999999995},"last":1.47},{"interface":"front","average":{"1min":0.89400000000000002,"5min":8.2469999999999999,"15min":3.8109999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.46,"5min":549.12099999999998,"15min":549.12099999999998},"last":1.5640000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.4099999999999999,"15min":1.49},"min":{"1min":0.53700000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":3.5470000000000002,"5min":10.202,"15min":10.202},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.325,"15min":1.5589999999999999},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":5.7939999999999996,"5min":5.8300000000000001,"15min":7.2489999999999997},"last":1.2529999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520784,"kb_used_data":935176,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92224368,"statfs":{"total":95995035648,"available":94437752832,"internally_reserved":0,"allocated":957620224,"data_stored":957098200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,6,10],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.004,"15min":1.052},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.215},"max":{"1min":4.0949999999999998,"5min":5.5869999999999997,"15min":11.324},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.282,"15min":1.119},"min":{"1min":0.27300000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.4770000000000003,"5min":5.5810000000000004,"15min":9.8249999999999993},"last":0.79100000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.0329999999999999,"15min":1.0580000000000001},"min":{"1min":0.32900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":7.6059999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.173,"15min":1.014},"min":{"1min":0.32700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":5.9690000000000003,"5min":5.9690000000000003,"15min":5.9690000000000003},"last":1.111}]},{"osd":4,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":3.4580000000000002,"15min":1.873},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29599999999999999},"max":{"1min":4.5229999999999997,"5min":193.77000000000001,"15min":193.77000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":3.3170000000000002,"15min":1.9339999999999999},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.2240000000000002,"5min":189.625,"15min":189.625},"last":1.0149999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:31:27 2024","interfaces":[{"interface":"back","average":{"1min":10.305999999999999,"5min":3.1309999999999998,"15min":1.7130000000000001},"min":{"1min":0.32800000000000001,"5min":0.29799999999999999,"15min":0.28699999999999998},"max":{"1min":181.46199999999999,"5min":181.46199999999999,"15min":181.46199999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":10.805999999999999,"5min":3.2149999999999999,"15min":1.8049999999999999},"min":{"1min":0.38300000000000001,"5min":0.34100000000000003,"15min":0.33600000000000002},"max":{"1min":186.41999999999999,"5min":186.41999999999999,"15min":186.41999999999999},"last":1.29}]},{"osd":6,"last update":"Mon Feb 12 16:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":3.1699999999999999,"15min":1.8129999999999999},"min":{"1min":0.371,"5min":0.311,"15min":0.311},"max":{"1min":4.3890000000000002,"5min":189.79400000000001,"15min":189.79400000000001},"last":1.369},{"interface":"front","average":{"1min":1.159,"5min":3.109,"15min":1.863},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.4889999999999999,"5min":194.41200000000001,"15min":194.41200000000001},"last":1.0600000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":9.7449999999999992,"5min":5.3029999999999999,"15min":2.5659999999999998},"min":{"1min":0.41299999999999998,"5min":0.38200000000000001,"15min":0.374},"max":{"1min":184.39400000000001,"5min":191.47200000000001,"15min":191.47200000000001},"last":1.196},{"interface":"front","average":{"1min":9.8190000000000008,"5min":5.4020000000000001,"15min":2.6749999999999998},"min":{"1min":0.443,"5min":0.44,"15min":0.44},"max":{"1min":184.43700000000001,"5min":191.50299999999999,"15min":191.50299999999999},"last":1.4299999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:31:27 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.3169999999999999,"15min":1.341},"min":{"1min":0.57499999999999996,"5min":0.432,"15min":0.432},"max":{"1min":4.5510000000000002,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.419,"15min":1.4379999999999999},"min":{"1min":0.63200000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":4.7889999999999997,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":1.681}]},{"osd":9,"last update":"Mon Feb 12 16:31:10 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.181,"15min":1.355},"min":{"1min":0.371,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.4359999999999999,"5min":5.6550000000000002,"15min":46.246000000000002},"last":1.331},{"interface":"front","average":{"1min":1.04,"5min":1.331,"15min":1.4730000000000001},"min":{"1min":0.45300000000000001,"5min":0.432,"15min":0.432},"max":{"1min":4.5910000000000002,"5min":7.5369999999999999,"15min":46.167999999999999},"last":1.51}]},{"osd":10,"last update":"Mon Feb 12 16:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.4470000000000001,"15min":1.3600000000000001},"min":{"1min":0.46300000000000002,"5min":0.442,"15min":0.41999999999999998},"max":{"1min":4.6239999999999997,"5min":6.3559999999999999,"15min":6.3559999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.6060000000000001,"15min":1.4319999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.47999999999999998},"max":{"1min":8.4280000000000008,"5min":9.3490000000000002,"15min":9.3490000000000002},"last":1.4750000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:31:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.4990000000000001,"15min":1.5640000000000001},"min":{"1min":0.51500000000000001,"5min":0.46600000000000003,"15min":0.435},"max":{"1min":8.2070000000000007,"5min":8.2070000000000007,"15min":46.064},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.4319999999999999,"15min":1.569},"min":{"1min":0.55900000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.6929999999999996,"5min":7.4210000000000003,"15min":45.642000000000003},"last":1.7649999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313692,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777948,"kb_used_data":468436,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92967204,"statfs":{"total":95995035648,"available":95198416896,"internally_reserved":0,"allocated":479678464,"data_stored":479150844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":1.0409999999999999,"15min":1.0820000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.0509999999999999,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.41599999999999998,"5min":1.0329999999999999,"15min":1.1830000000000001},"min":{"1min":0.26100000000000001,"5min":0.255,"15min":0.247},"max":{"1min":1.024,"5min":6.4180000000000001,"15min":9.6470000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.4079999999999999,"15min":1.768},"min":{"1min":0.28399999999999997,"5min":0.25700000000000001,"15min":0.253},"max":{"1min":7.0970000000000004,"5min":26.745000000000001,"15min":26.745000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.373,"15min":1.0229999999999999},"min":{"1min":0.32400000000000001,"5min":0.28799999999999998,"15min":0.25600000000000001},"max":{"1min":8.4410000000000007,"5min":25.260000000000002,"15min":25.260000000000002},"last":0.628}]},{"osd":4,"last update":"Mon Feb 12 16:30:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.2869999999999999,"15min":1.0840000000000001},"min":{"1min":0.37,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.8730000000000002,"5min":10.890000000000001,"15min":10.890000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.276,"15min":1.226},"min":{"1min":0.32300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.8809999999999998,"5min":6.601,"15min":9.8079999999999998},"last":0.45400000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.1319999999999999,"15min":1.177},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35199999999999998},"max":{"1min":3.9670000000000001,"5min":6.133,"15min":6.133},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.5249999999999999,"15min":1.252},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":5.6669999999999998,"5min":11.081,"15min":11.081},"last":0.49299999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:31:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.2490000000000001,"15min":1.2829999999999999},"min":{"1min":0.39600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":4.2359999999999998,"5min":6.5279999999999996,"15min":21.527000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.2709999999999999,"15min":1.2250000000000001},"min":{"1min":0.40200000000000002,"5min":0.35099999999999998,"15min":0.33800000000000002},"max":{"1min":2.593,"5min":6.2850000000000001,"15min":6.2850000000000001},"last":0.89200000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:30:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.548,"15min":1.2250000000000001},"min":{"1min":0.372,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.698,"5min":11.044,"15min":11.044},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.077,"5min":1.4039999999999999,"15min":1.2949999999999999},"min":{"1min":0.45000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.7080000000000002,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":0.94299999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.321,"15min":1.319},"min":{"1min":0.40100000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":4.1769999999999996,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.268,"15min":1.337},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.9970000000000001,"5min":6.593,"15min":9.7639999999999993},"last":0.73299999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:30:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.4279999999999999,"15min":1.3220000000000001},"min":{"1min":0.45200000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.78,"5min":10.972,"15min":10.972},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.4990000000000001,"15min":1.4339999999999999},"min":{"1min":0.45600000000000002,"5min":0.442,"15min":0.42399999999999999},"max":{"1min":1.825,"5min":11.006,"15min":11.006},"last":0.56699999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:31:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.3580000000000001,"15min":1.429},"min":{"1min":0.47699999999999998,"5min":0.42099999999999999,"15min":0.41399999999999998},"max":{"1min":1.2829999999999999,"5min":7.7539999999999996,"15min":11.454000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.3959999999999999,"15min":1.456},"min":{"1min":0.5,"5min":0.5,"15min":0.441},"max":{"1min":1.3440000000000001,"5min":7.7140000000000004,"15min":9.7170000000000005},"last":0.86899999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:31:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.349,"15min":1.4139999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.47799999999999998},"max":{"1min":1.1779999999999999,"5min":7.258,"15min":7.258},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.4330000000000001,"15min":1.4970000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.2719999999999998,"5min":7.3040000000000003,"15min":7.3040000000000003},"last":1.0309999999999999}]}]},{"osd":3,"up_from":522,"seq":2241972928618,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3241292,"kb_used_data":2896772,"kb_used_omap":91,"kb_used_meta":344420,"kb_avail":90503860,"statfs":{"total":95995035648,"available":92675952640,"internally_reserved":0,"allocated":2966294528,"data_stored":2965743786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":352686096},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:30:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.405,"15min":1.0089999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.6659999999999999,"5min":34.055999999999997,"15min":34.055999999999997},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.4550000000000001,"15min":1.0469999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.8069999999999999,"5min":27.600000000000001,"15min":27.600000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":1.127,"15min":0.97299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.6899999999999999,"5min":7.5410000000000004,"15min":7.5410000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.008,"15min":1.0669999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.5819999999999999,"5min":4.5819999999999999,"15min":6.5030000000000001},"last":1.5620000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.94999999999999996,"15min":1.1619999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.1319999999999997,"5min":4.9980000000000002,"15min":10.004},"last":1.633},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.177,"15min":1.115},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32600000000000001},"max":{"1min":4.1630000000000003,"5min":6.806,"15min":6.806},"last":0.84599999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.0009999999999999,"15min":1.24},"min":{"1min":0.38300000000000001,"5min":0.316,"15min":0.316},"max":{"1min":2.7530000000000001,"5min":4.7530000000000001,"15min":6.7539999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0800000000000001,"15min":0.97199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.165,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":1.6830000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.179,"15min":1.1479999999999999},"min":{"1min":0.38300000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.7110000000000003,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":2.968},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.149,"15min":1.0509999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":6.8890000000000002,"5min":6.8890000000000002,"15min":6.8890000000000002},"last":1.9139999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.1990000000000001,"15min":1.343},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.7409999999999997,"5min":6.7409999999999997,"15min":6.851},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.258,"15min":1.2609999999999999},"min":{"1min":0.45200000000000001,"5min":0.40300000000000002,"15min":0.39200000000000002},"max":{"1min":5.0890000000000004,"5min":5.0890000000000004,"15min":5.0890000000000004},"last":0.88800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.0429999999999999,"15min":1.139},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.762,"5min":5.1459999999999999,"15min":5.1459999999999999},"last":0.88},{"interface":"front","average":{"1min":0.996,"5min":1.1220000000000001,"15min":1.0960000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.234,"5min":4.8460000000000001,"15min":4.8460000000000001},"last":1.73}]},{"osd":9,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.218,"15min":1.304},"min":{"1min":0.44400000000000001,"5min":0.442,"15min":0.42799999999999999},"max":{"1min":4.2519999999999998,"5min":4.5810000000000004,"15min":5.0830000000000002},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.248,"15min":1.321},"min":{"1min":0.57699999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.7890000000000001,"5min":5.0780000000000003,"15min":5.0780000000000003},"last":0.91400000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.1259999999999999,"15min":1.3799999999999999},"min":{"1min":0.54600000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":3.6680000000000001,"5min":4.8499999999999996,"15min":4.8499999999999996},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.444,"15min":1.351},"min":{"1min":0.49199999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.4119999999999999,"5min":15.140000000000001,"15min":15.140000000000001},"last":1.3839999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.296,"15min":1.278},"min":{"1min":0.51000000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.444,"5min":4.9870000000000001,"15min":4.9870000000000001},"last":0.627},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.252,"15min":1.417},"min":{"1min":0.57599999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.8170000000000002,"5min":8.4540000000000006,"15min":8.4540000000000006},"last":0.66900000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953933,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2000128,"kb_used_data":1398136,"kb_used_omap":0,"kb_used_meta":601984,"kb_avail":91745024,"statfs":{"total":95995035648,"available":93946904576,"internally_reserved":0,"allocated":1431691264,"data_stored":1431146400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":616431616},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.0149999999999999,"15min":1.054},"min":{"1min":0.248,"5min":0.248,"15min":0.23699999999999999},"max":{"1min":5.2889999999999997,"5min":7.492,"15min":7.6319999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.294,"15min":1.212},"min":{"1min":0.34899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.3220000000000001,"5min":12.236000000000001,"15min":12.236000000000001},"last":0.67800000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.246,"15min":1.056},"min":{"1min":0.29799999999999999,"5min":0.28399999999999997,"15min":0.24399999999999999},"max":{"1min":5.3920000000000003,"5min":7.3630000000000004,"15min":8.0530000000000008},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.212,"15min":1.153},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.254},"max":{"1min":3.8300000000000001,"5min":7.633,"15min":7.633},"last":0.58699999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":1.3169999999999999,"15min":1.163},"min":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":8.5800000000000001,"5min":12.409000000000001,"15min":12.409000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.353,"5min":1.1419999999999999,"15min":1.0780000000000001},"min":{"1min":0.41199999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.8620000000000001,"5min":7.383,"15min":7.383},"last":0.69299999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:31:25 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.2350000000000001,"15min":1.077},"min":{"1min":0.34399999999999997,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":4.9489999999999998,"5min":7.431,"15min":7.431},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.841,"5min":1.3740000000000001,"15min":1.2390000000000001},"min":{"1min":0.49199999999999999,"5min":0.38700000000000001,"15min":0.33400000000000002},"max":{"1min":4.9829999999999997,"5min":7.4569999999999999,"15min":7.4569999999999999},"last":0.78700000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.1279999999999999,"15min":1.073},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.26400000000000001},"max":{"1min":6.5270000000000001,"5min":7.5289999999999999,"15min":7.5410000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.202,"15min":1.129},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.3949999999999996,"5min":5.5190000000000001,"15min":9.6639999999999997},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.0720000000000001,"15min":1.1359999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.3440000000000003,"5min":6.8659999999999997,"15min":8.2070000000000007},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.897,"5min":1.1839999999999999,"15min":1.1990000000000001},"min":{"1min":0.55500000000000005,"5min":0.35499999999999998,"15min":0.34599999999999997},"max":{"1min":9.1440000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":1.026}]},{"osd":8,"last update":"Mon Feb 12 16:31:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.1839999999999999,"15min":1.24},"min":{"1min":0.504,"5min":0.39300000000000002,"15min":0.38300000000000001},"max":{"1min":4.7009999999999996,"5min":7.6890000000000001,"15min":7.8650000000000002},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.2070000000000001,"15min":1.292},"min":{"1min":0.41199999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.4050000000000002,"5min":7.7110000000000003,"15min":7.7110000000000003},"last":0.89800000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.1759999999999999,"15min":1.3700000000000001},"min":{"1min":0.53400000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.4729999999999999,"5min":7.7750000000000004,"15min":34.112000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3360000000000001,"15min":1.583},"min":{"1min":0.51100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":4.0060000000000002,"5min":7.827,"15min":34.590000000000003},"last":1.0720000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:31:00 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.429,"15min":1.4410000000000001},"min":{"1min":0.44,"5min":0.433,"15min":0.39200000000000002},"max":{"1min":3.976,"5min":7.7619999999999996,"15min":7.7619999999999996},"last":1.004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.093,"15min":1.2370000000000001},"min":{"1min":0.42299999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.964,"5min":5.4269999999999996,"15min":8.5299999999999994},"last":0.96399999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:31:12 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.2010000000000001,"15min":1.3859999999999999},"min":{"1min":0.50900000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":5.5069999999999997,"5min":7.734,"15min":7.734},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.3680000000000001,"15min":1.464},"min":{"1min":0.48899999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.9489999999999998,"5min":7.7969999999999997,"15min":7.7969999999999997},"last":1.1279999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724996,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2195808,"kb_used_data":1882200,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":91549344,"statfs":{"total":95995035648,"available":93746528256,"internally_reserved":0,"allocated":1927372800,"data_stored":1926841108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.80500000000000005,"15min":0.92500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.44},{"interface":"front","average":{"1min":1.024,"5min":0.92000000000000004,"15min":0.95399999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":3.27,"15min":3.0059999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.0640000000000001,"5min":221.96600000000001,"15min":565.34500000000003},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":3.391,"15min":3.1030000000000002},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.29199999999999998},"max":{"1min":6.1029999999999998,"5min":222.06200000000001,"15min":565.98400000000004},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.2050000000000001,"15min":1.1779999999999999},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.0170000000000003,"5min":6.0709999999999997,"15min":40.021999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.23,"15min":1.236},"min":{"1min":0.372,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":11.448,"5min":11.448,"15min":40.225000000000001},"last":0.83499999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.5509999999999999,"15min":1.383},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":8.0060000000000002,"5min":22.484000000000002,"15min":22.484000000000002},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.3700000000000001,"15min":0.94199999999999995},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.5549999999999997,"5min":23.34,"15min":23.34},"last":0.91100000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.488,"15min":1.1819999999999999},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.9550000000000001,"5min":41.908000000000001,"15min":41.908000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.7629999999999999,"15min":1.2270000000000001},"min":{"1min":0.32400000000000001,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":4.9989999999999997,"5min":42.430999999999997,"15min":42.430999999999997},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.0860000000000001,"15min":1.036},"min":{"1min":0.29899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.0439999999999996,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":0.999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.135,"15min":1.1080000000000001},"min":{"1min":0.33100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4910000000000001,"5min":9.5830000000000002,"15min":9.5830000000000002},"last":0.94799999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.1919999999999999,"15min":1.1060000000000001},"min":{"1min":0.41899999999999998,"5min":0.32000000000000001,"15min":0.28799999999999998},"max":{"1min":4.5570000000000004,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":0.746},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.4119999999999999,"15min":1.173},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.2560000000000002,"5min":6.3979999999999997,"15min":6.3979999999999997},"last":1.181}]},{"osd":8,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":6.1109999999999998,"5min":2.0950000000000002,"15min":1.528},"min":{"1min":0.42199999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":108.913,"5min":108.913,"15min":108.913},"last":1.663},{"interface":"front","average":{"1min":5.633,"5min":2.1539999999999999,"15min":1.5369999999999999},"min":{"1min":0.51000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":108.93600000000001,"5min":108.93600000000001,"15min":108.93600000000001},"last":1.595}]},{"osd":9,"last update":"Mon Feb 12 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.365,"15min":1.3759999999999999},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.439},"max":{"1min":4.125,"5min":6.2939999999999996,"15min":6.2939999999999996},"last":1.694},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.3560000000000001,"15min":1.2749999999999999},"min":{"1min":0.50900000000000001,"5min":0.375,"15min":0.375},"max":{"1min":6.1669999999999998,"5min":9.6649999999999991,"15min":9.6649999999999991},"last":1.27}]},{"osd":10,"last update":"Mon Feb 12 16:30:40 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.835,"15min":1.494},"min":{"1min":0.48599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":6.1989999999999998,"5min":55.518999999999998,"15min":55.518999999999998},"last":1.333},{"interface":"front","average":{"1min":0.89200000000000002,"5min":2.0819999999999999,"15min":1.5369999999999999},"min":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.0419999999999998,"5min":55.497999999999998,"15min":55.497999999999998},"last":1.421}]},{"osd":11,"last update":"Mon Feb 12 16:30:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8049999999999999,"5min":1.4610000000000001,"15min":1.401},"min":{"1min":0.48799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.2370000000000001,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.554,"15min":1.4179999999999999},"min":{"1min":0.58899999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":4.1630000000000003,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":1.4910000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":955187200,"data_stored":955187200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1452412928,"data_stored":1452408856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":478019584,"data_stored":478019584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2965897216,"data_stored":2965897216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1430257664,"data_stored":1430257664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1926303744,"data_stored":1926299672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1907818496,"data_stored":1907818496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1924923392,"data_stored":1924923392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":974848000,"data_stored":974848000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":967311360,"data_stored":967311360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":476184576,"data_stored":476184576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":121241600,"data_stored":121241600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:31:49.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set device_health_metrics pgp_num 31 2024-02-12T16:31:49.025 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:49.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:49.140 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:48 smithi169 bash[19604]: audit 2024-02-12T16:31:47.847189+0000 mgr.y (mgr.14146) 1414 : audit [DBG] from='client.26408 v1:172.21.15.121:0/2000520722' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:31:49.140 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:48 smithi169 bash[19604]: cluster 2024-02-12T16:31:48.064520+0000 mgr.y (mgr.14146) 1415 : cluster [DBG] pgmap v1990: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 402 op/s; 30337/231818 objects misplaced (13.087%); 15 MiB/s, 245 objects/s recovering 2024-02-12T16:31:49.157 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:49.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:48 smithi121 bash[18489]: audit 2024-02-12T16:31:47.847189+0000 mgr.y (mgr.14146) 1414 : audit [DBG] from='client.26408 v1:172.21.15.121:0/2000520722' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:31:49.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:48 smithi121 bash[18489]: cluster 2024-02-12T16:31:48.064520+0000 mgr.y (mgr.14146) 1415 : cluster [DBG] pgmap v1990: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 402 op/s; 30337/231818 objects misplaced (13.087%); 15 MiB/s, 245 objects/s recovering 2024-02-12T16:31:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:49.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:49.289 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:49.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:48 smithi123 bash[19119]: audit 2024-02-12T16:31:47.847189+0000 mgr.y (mgr.14146) 1414 : audit [DBG] from='client.26408 v1:172.21.15.121:0/2000520722' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:31:49.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:48 smithi123 bash[19119]: cluster 2024-02-12T16:31:48.064520+0000 mgr.y (mgr.14146) 1415 : cluster [DBG] pgmap v1990: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 402 op/s; 30337/231818 objects misplaced (13.087%); 15 MiB/s, 245 objects/s recovering 2024-02-12T16:31:49.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:49.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:49.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:49.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:49.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:49.699 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:49.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:49.836 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:49.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:49.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:49.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:50.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:50.123 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:50.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:50.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:50.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:50.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:50.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:50.416 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:50.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:50.509 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:50.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:50.544 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:50.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:50.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:50.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:50.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:50.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:50.831 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:50.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:50.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:50.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:50.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:51.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:51.106 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:51.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:51.123 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:51 smithi123 bash[19119]: cluster 2024-02-12T16:31:50.065127+0000 mgr.y (mgr.14146) 1416 : cluster [DBG] pgmap v1991: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 27275/228768 objects misplaced (11.923%); 26 MiB/s, 419 objects/s recovering 2024-02-12T16:31:51.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:51.241 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:51.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:51.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:51.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:51 smithi169 bash[19604]: cluster 2024-02-12T16:31:50.065127+0000 mgr.y (mgr.14146) 1416 : cluster [DBG] pgmap v1991: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 27275/228768 objects misplaced (11.923%); 26 MiB/s, 419 objects/s recovering 2024-02-12T16:31:51.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:51.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:51.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:51 smithi121 bash[18489]: cluster 2024-02-12T16:31:50.065127+0000 mgr.y (mgr.14146) 1416 : cluster [DBG] pgmap v1991: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 7.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 27275/228768 objects misplaced (11.923%); 26 MiB/s, 419 objects/s recovering 2024-02-12T16:31:51.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:51.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:51.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:51.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:51.650 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:51.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:51.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:51.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:51.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:51.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:51.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:51.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:52.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:52.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:52.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:52.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:52.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:52.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:52.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:52.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:52.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:52.479 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:52.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:52.546 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:52.546 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:52.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:52.609 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:52.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:52.715 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:52.739 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:52.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:52.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:52.873 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:52.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:52.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:53.008 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:53.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:53.139 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:53.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:53.268 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:53.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:53.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:53 smithi123 bash[19119]: cluster 2024-02-12T16:31:52.065672+0000 mgr.y (mgr.14146) 1417 : cluster [DBG] pgmap v1992: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 6.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 26221/222140 objects misplaced (11.804%); 12 MiB/s, 186 objects/s recovering 2024-02-12T16:31:53.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:53 smithi123 bash[19119]: cluster 2024-02-12T16:31:52.176207+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:53.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:53 smithi123 bash[19119]: cluster 2024-02-12T16:31:52.176271+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:53 smithi121 bash[18489]: cluster 2024-02-12T16:31:52.065672+0000 mgr.y (mgr.14146) 1417 : cluster [DBG] pgmap v1992: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 6.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 26221/222140 objects misplaced (11.804%); 12 MiB/s, 186 objects/s recovering 2024-02-12T16:31:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:53 smithi121 bash[18489]: cluster 2024-02-12T16:31:52.176207+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:53 smithi121 bash[18489]: cluster 2024-02-12T16:31:52.176271+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:53.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:53.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:53 smithi169 bash[19604]: cluster 2024-02-12T16:31:52.065672+0000 mgr.y (mgr.14146) 1417 : cluster [DBG] pgmap v1992: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 6.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 26221/222140 objects misplaced (11.804%); 12 MiB/s, 186 objects/s recovering 2024-02-12T16:31:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:53 smithi169 bash[19604]: cluster 2024-02-12T16:31:52.176207+0000 osd.5 (osd.5) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:31:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:53 smithi169 bash[19604]: cluster 2024-02-12T16:31:52.176271+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:31:53.567 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:31:53.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:53.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:53.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.702 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:53.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:53.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:53.938 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:53.976 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:53.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:54.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:54.123 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:54.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:54.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:54.234 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:54 smithi123 bash[19119]: cluster 2024-02-12T16:31:53.131686+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e696: 12 total, 12 up, 12 in 2024-02-12T16:31:54.267 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:54.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:54.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:54.377 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:54 smithi121 bash[18489]: cluster 2024-02-12T16:31:53.131686+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e696: 12 total, 12 up, 12 in 2024-02-12T16:31:54.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:54.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:54.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:54.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:54.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:54.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:54 smithi169 bash[19604]: cluster 2024-02-12T16:31:53.131686+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e696: 12 total, 12 up, 12 in 2024-02-12T16:31:54.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:54.671 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:54.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:54.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:54.802 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:54.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:54.909 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:54.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:54.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:55.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:55.109 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:55.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:55.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:55.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:55.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:55.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:55.365 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:55.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:55.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:55 smithi123 bash[19119]: cluster 2024-02-12T16:31:54.066101+0000 mgr.y (mgr.14146) 1418 : cluster [DBG] pgmap v1994: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 23522/216666 objects misplaced (10.856%); 20 MiB/s, 313 objects/s recovering 2024-02-12T16:31:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:55 smithi121 bash[18489]: cluster 2024-02-12T16:31:54.066101+0000 mgr.y (mgr.14146) 1418 : cluster [DBG] pgmap v1994: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 23522/216666 objects misplaced (10.856%); 20 MiB/s, 313 objects/s recovering 2024-02-12T16:31:55.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:55.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:31:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:55.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:31:55.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:55.637 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:55.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:55.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:55.767 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:55.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:55.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:55.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:55.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:55.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:55.958 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:55 smithi169 bash[19604]: cluster 2024-02-12T16:31:54.066101+0000 mgr.y (mgr.14146) 1418 : cluster [DBG] pgmap v1994: 57 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 51 active+clean; 6.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 23522/216666 objects misplaced (10.856%); 20 MiB/s, 313 objects/s recovering 2024-02-12T16:31:56.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:56.034 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:56.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:56.164 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:56.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:56.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:56 smithi169 bash[19604]: cluster 2024-02-12T16:31:55.138048+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e697: 12 total, 12 up, 12 in 2024-02-12T16:31:56.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:56.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:56.402 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:56 smithi121 bash[18489]: cluster 2024-02-12T16:31:55.138048+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e697: 12 total, 12 up, 12 in 2024-02-12T16:31:56.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:56.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:56.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:56 smithi123 bash[19119]: cluster 2024-02-12T16:31:55.138048+0000 mon.a (mon.0) 2416 : cluster [DBG] osdmap e697: 12 total, 12 up, 12 in 2024-02-12T16:31:56.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:31:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:31:56.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:56.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:56.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:56.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:56.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:57.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:57.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:57.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:57.203 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:57.208 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:57 smithi169 bash[19604]: cluster 2024-02-12T16:31:56.015803+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.17 deep-scrub starts 2024-02-12T16:31:57.208 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:57 smithi169 bash[19604]: cluster 2024-02-12T16:31:56.066611+0000 mgr.y (mgr.14146) 1419 : cluster [DBG] pgmap v1996: 57 pgs: 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 6.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 22434/211988 objects misplaced (10.583%); 21 MiB/s, 333 objects/s recovering 2024-02-12T16:31:57.208 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:57 smithi169 bash[19604]: cluster 2024-02-12T16:31:56.150055+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e698: 12 total, 12 up, 12 in 2024-02-12T16:31:57.224 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:57.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:57.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:31:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:57.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:31:57.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:57 smithi121 bash[18489]: cluster 2024-02-12T16:31:56.015803+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.17 deep-scrub starts 2024-02-12T16:31:57.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:57 smithi121 bash[18489]: cluster 2024-02-12T16:31:56.066611+0000 mgr.y (mgr.14146) 1419 : cluster [DBG] pgmap v1996: 57 pgs: 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 6.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 22434/211988 objects misplaced (10.583%); 21 MiB/s, 333 objects/s recovering 2024-02-12T16:31:57.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:57 smithi121 bash[18489]: cluster 2024-02-12T16:31:56.150055+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e698: 12 total, 12 up, 12 in 2024-02-12T16:31:57.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:57.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:57.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:57.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:57.621 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:57.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:57 smithi123 bash[19119]: cluster 2024-02-12T16:31:56.015803+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.17 deep-scrub starts 2024-02-12T16:31:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:57 smithi123 bash[19119]: cluster 2024-02-12T16:31:56.066611+0000 mgr.y (mgr.14146) 1419 : cluster [DBG] pgmap v1996: 57 pgs: 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 6.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 22434/211988 objects misplaced (10.583%); 21 MiB/s, 333 objects/s recovering 2024-02-12T16:31:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:57 smithi123 bash[19119]: cluster 2024-02-12T16:31:56.150055+0000 mon.a (mon.0) 2417 : cluster [DBG] osdmap e698: 12 total, 12 up, 12 in 2024-02-12T16:31:57.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:57.748 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:31:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:57.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:31:57.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:57.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:57.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:57.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:58.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:58.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:58.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:58.179 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:58.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:58.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:58.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:58.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:58.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:58.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:58.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:58.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:58.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:58.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:58.771 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:31:58.771 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:31:58.775 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:31:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:58.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:31:58.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:58.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:31:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:58.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:31:59.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:59.039 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:59.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:59.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:59.161 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:31:59 smithi169 bash[19604]: cluster 2024-02-12T16:31:58.067164+0000 mgr.y (mgr.14146) 1420 : cluster [DBG] pgmap v1998: 57 pgs: 1 active+clean+scrubbing+deep, 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 50 active+clean; 6.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 22209/207034 objects misplaced (10.727%); 13 MiB/s, 214 objects/s recovering 2024-02-12T16:31:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:31:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:59.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:31:59.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:59.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:31:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:59.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:31:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:31:59 smithi123 bash[19119]: cluster 2024-02-12T16:31:58.067164+0000 mgr.y (mgr.14146) 1420 : cluster [DBG] pgmap v1998: 57 pgs: 1 active+clean+scrubbing+deep, 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 50 active+clean; 6.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 22209/207034 objects misplaced (10.727%); 13 MiB/s, 214 objects/s recovering 2024-02-12T16:31:59.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:31:59 smithi121 bash[18489]: cluster 2024-02-12T16:31:58.067164+0000 mgr.y (mgr.14146) 1420 : cluster [DBG] pgmap v1998: 57 pgs: 1 active+clean+scrubbing+deep, 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 50 active+clean; 6.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 22209/207034 objects misplaced (10.727%); 13 MiB/s, 214 objects/s recovering 2024-02-12T16:31:59.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:59.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:31:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:59.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:31:59.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:59.606 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:31:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:59.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:31:59.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:59.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:31:59.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:59.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:31:59.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:59.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:31:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:31:59.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:31:59.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:00.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:00.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:00.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:00.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:00.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:00.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:00.299 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:00.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:00.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:00.437 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:00.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:00.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:00.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:00.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:00.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:00.726 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:00.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:00.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:00.858 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:00.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:00.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:00.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:01.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:01.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:01.297 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:01 smithi121 bash[18489]: cluster 2024-02-12T16:32:00.067704+0000 mgr.y (mgr.14146) 1421 : cluster [DBG] pgmap v1999: 57 pgs: 1 active+clean+scrubbing+deep, 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 50 active+clean; 6.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 21836/204046 objects misplaced (10.702%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:32:01.297 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:01 smithi121 bash[18489]: cluster 2024-02-12T16:32:00.744712+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e699: 12 total, 12 up, 12 in 2024-02-12T16:32:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:01.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:01.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:01.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:01.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:01 smithi169 bash[19604]: cluster 2024-02-12T16:32:00.067704+0000 mgr.y (mgr.14146) 1421 : cluster [DBG] pgmap v1999: 57 pgs: 1 active+clean+scrubbing+deep, 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 50 active+clean; 6.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 21836/204046 objects misplaced (10.702%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:32:01.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:01 smithi169 bash[19604]: cluster 2024-02-12T16:32:00.744712+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e699: 12 total, 12 up, 12 in 2024-02-12T16:32:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:01.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:01.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:01.555 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:01.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:01.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:01 smithi123 bash[19119]: cluster 2024-02-12T16:32:00.067704+0000 mgr.y (mgr.14146) 1421 : cluster [DBG] pgmap v1999: 57 pgs: 1 active+clean+scrubbing+deep, 1 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 50 active+clean; 6.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 21836/204046 objects misplaced (10.702%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:32:01.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:01 smithi123 bash[19119]: cluster 2024-02-12T16:32:00.744712+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e699: 12 total, 12 up, 12 in 2024-02-12T16:32:01.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:01.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:01.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:01.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:01.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:01.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:32:01.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:02.000 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:02.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:02.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.271 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:02.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:02.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:02 smithi123 bash[19119]: cluster 2024-02-12T16:32:00.619203+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:02.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:02 smithi123 bash[19119]: cluster 2024-02-12T16:32:00.619251+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:02.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:02.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:02.534 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:02 smithi169 bash[19604]: cluster 2024-02-12T16:32:00.619203+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:02 smithi169 bash[19604]: cluster 2024-02-12T16:32:00.619251+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:02.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:02 smithi121 bash[18489]: cluster 2024-02-12T16:32:00.619203+0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:02.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:02 smithi121 bash[18489]: cluster 2024-02-12T16:32:00.619251+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:02.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:02.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.830 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:02.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:02.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:02.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:03.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:03.112 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:03.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:03.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:03.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:03.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:03.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:03.363 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:32:03.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:03.401 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:03.402 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:32:03.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:03.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:03.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:03 smithi121 bash[18489]: cluster 2024-02-12T16:32:02.068234+0000 mgr.y (mgr.14146) 1422 : cluster [DBG] pgmap v2001: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 6.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 21023/198460 objects misplaced (10.593%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:32:03.444 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:32:03.472 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:32:03.498 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:32:03.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:03.525 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:32:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:03 smithi169 bash[19604]: cluster 2024-02-12T16:32:02.068234+0000 mgr.y (mgr.14146) 1422 : cluster [DBG] pgmap v2001: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 6.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 21023/198460 objects misplaced (10.593%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:32:03.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:32:03.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:03.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:03.585 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:32:03.615 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:32:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:03 smithi123 bash[19119]: cluster 2024-02-12T16:32:02.068234+0000 mgr.y (mgr.14146) 1422 : cluster [DBG] pgmap v2001: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 6.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 21023/198460 objects misplaced (10.593%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:32:03.646 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:32:03.662 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:03.674 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:32:03.687 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:03.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:03.701 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:32:03.706 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.706 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.706 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.706 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.707 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:32:03.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:03.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:03.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:03.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:04.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:04.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:04.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:04.277 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:04.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:04.416 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:04.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:04.529 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:04 smithi169 bash[19604]: audit 2024-02-12T16:32:03.326250+0000 mon.b (mon.2) 142 : audit [DBG] from='client.? v1:172.21.15.121:0/1019314585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:04.529 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:04 smithi169 bash[19604]: cluster 2024-02-12T16:32:04.001845+0000 osd.8 (osd.8) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:04.529 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:04 smithi169 bash[19604]: cluster 2024-02-12T16:32:04.001896+0000 osd.8 (osd.8) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:04.545 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:04.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:04 smithi123 bash[19119]: audit 2024-02-12T16:32:03.326250+0000 mon.b (mon.2) 142 : audit [DBG] from='client.? v1:172.21.15.121:0/1019314585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:04.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:04 smithi123 bash[19119]: cluster 2024-02-12T16:32:04.001845+0000 osd.8 (osd.8) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:04.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:04 smithi123 bash[19119]: cluster 2024-02-12T16:32:04.001896+0000 osd.8 (osd.8) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:04.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:04.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:04 smithi121 bash[18489]: audit 2024-02-12T16:32:03.326250+0000 mon.b (mon.2) 142 : audit [DBG] from='client.? v1:172.21.15.121:0/1019314585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:04.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:04 smithi121 bash[18489]: cluster 2024-02-12T16:32:04.001845+0000 osd.8 (osd.8) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:04.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:04 smithi121 bash[18489]: cluster 2024-02-12T16:32:04.001896+0000 osd.8 (osd.8) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:04.684 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:04.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:04.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:04.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:04.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:04.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:05.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:05.088 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:05.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:05.184 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:32:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 1 pgp_num to 31 2024-02-12T16:32:05.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:05.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:05.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:05.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:05.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:05 smithi121 bash[18489]: cluster 2024-02-12T16:32:04.068678+0000 mgr.y (mgr.14146) 1423 : cluster [DBG] pgmap v2002: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 20668/194354 objects misplaced (10.634%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:05.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:05 smithi121 bash[18489]: cluster 2024-02-12T16:32:04.173565+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e700: 12 total, 12 up, 12 in 2024-02-12T16:32:05.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:05 smithi121 bash[18489]: audit 2024-02-12T16:32:04.232061+0000 mon.a (mon.0) 2420 : audit [INF] from='client.26420 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:32:05.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:05 smithi121 bash[18489]: audit 2024-02-12T16:32:04.233295+0000 mon.b (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.121:0/3361169907' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:32:05.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:05 smithi121 bash[18489]: audit 2024-02-12T16:32:04.395930+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? v1:172.21.15.121:0/2701469665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:32:05.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:05.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:05.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:05.478 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:05 smithi169 bash[19604]: cluster 2024-02-12T16:32:04.068678+0000 mgr.y (mgr.14146) 1423 : cluster [DBG] pgmap v2002: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 20668/194354 objects misplaced (10.634%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:05.478 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:05 smithi169 bash[19604]: cluster 2024-02-12T16:32:04.173565+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e700: 12 total, 12 up, 12 in 2024-02-12T16:32:05.478 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:05 smithi169 bash[19604]: audit 2024-02-12T16:32:04.232061+0000 mon.a (mon.0) 2420 : audit [INF] from='client.26420 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:32:05.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:05 smithi169 bash[19604]: audit 2024-02-12T16:32:04.233295+0000 mon.b (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.121:0/3361169907' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:32:05.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:05 smithi169 bash[19604]: audit 2024-02-12T16:32:04.395930+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? v1:172.21.15.121:0/2701469665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:32:05.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:05.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:05.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:05.623 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:05.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:05.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:05 smithi123 bash[19119]: cluster 2024-02-12T16:32:04.068678+0000 mgr.y (mgr.14146) 1423 : cluster [DBG] pgmap v2002: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 20668/194354 objects misplaced (10.634%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:05 smithi123 bash[19119]: cluster 2024-02-12T16:32:04.173565+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e700: 12 total, 12 up, 12 in 2024-02-12T16:32:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:05 smithi123 bash[19119]: audit 2024-02-12T16:32:04.232061+0000 mon.a (mon.0) 2420 : audit [INF] from='client.26420 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:32:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:05 smithi123 bash[19119]: audit 2024-02-12T16:32:04.233295+0000 mon.b (mon.2) 143 : audit [INF] from='client.? v1:172.21.15.121:0/3361169907' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:32:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:05 smithi123 bash[19119]: audit 2024-02-12T16:32:04.395930+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? v1:172.21.15.121:0/2701469665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:32:05.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:05.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:05.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:05.874 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:05.907 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:05.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:06.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:06.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:06.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:06.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:06.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:06.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:06.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:06 smithi169 bash[19604]: cluster 2024-02-12T16:32:05.170488+0000 mon.a (mon.0) 2422 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:06 smithi169 bash[19604]: audit 2024-02-12T16:32:05.174606+0000 mon.a (mon.0) 2423 : audit [INF] from='client.26420 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:32:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:06 smithi169 bash[19604]: audit 2024-02-12T16:32:05.174687+0000 mon.a (mon.0) 2424 : audit [INF] from='client.? v1:172.21.15.121:0/2701469665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:32:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:06 smithi169 bash[19604]: cluster 2024-02-12T16:32:05.174735+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e701: 12 total, 12 up, 12 in 2024-02-12T16:32:06.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:06.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:06.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:06.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:06 smithi123 bash[19119]: cluster 2024-02-12T16:32:05.170488+0000 mon.a (mon.0) 2422 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:06.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:06 smithi123 bash[19119]: audit 2024-02-12T16:32:05.174606+0000 mon.a (mon.0) 2423 : audit [INF] from='client.26420 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:32:06.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:06 smithi123 bash[19119]: audit 2024-02-12T16:32:05.174687+0000 mon.a (mon.0) 2424 : audit [INF] from='client.? v1:172.21.15.121:0/2701469665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:32:06.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:06 smithi123 bash[19119]: cluster 2024-02-12T16:32:05.174735+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e701: 12 total, 12 up, 12 in 2024-02-12T16:32:06.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:06 smithi121 bash[18489]: cluster 2024-02-12T16:32:05.170488+0000 mon.a (mon.0) 2422 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:06.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:06 smithi121 bash[18489]: audit 2024-02-12T16:32:05.174606+0000 mon.a (mon.0) 2423 : audit [INF] from='client.26420 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:32:06.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:06 smithi121 bash[18489]: audit 2024-02-12T16:32:05.174687+0000 mon.a (mon.0) 2424 : audit [INF] from='client.? v1:172.21.15.121:0/2701469665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:32:06.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:06 smithi121 bash[18489]: cluster 2024-02-12T16:32:05.174735+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e701: 12 total, 12 up, 12 in 2024-02-12T16:32:06.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:06.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:06.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:06.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:06.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:06.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:06.747 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:06.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:06.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:32:06.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:06.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:06.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:07.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:07.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:07.176 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:07.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:07.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:07 smithi121 bash[18489]: cluster 2024-02-12T16:32:06.069148+0000 mgr.y (mgr.14146) 1424 : cluster [DBG] pgmap v2005: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 20054/190504 objects misplaced (10.527%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:07.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:07.459 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:07 smithi169 bash[19604]: cluster 2024-02-12T16:32:06.069148+0000 mgr.y (mgr.14146) 1424 : cluster [DBG] pgmap v2005: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 20054/190504 objects misplaced (10.527%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:07.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:07.591 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:07.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:07 smithi123 bash[19119]: cluster 2024-02-12T16:32:06.069148+0000 mgr.y (mgr.14146) 1424 : cluster [DBG] pgmap v2005: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 20054/190504 objects misplaced (10.527%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:07.694 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:07.721 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:07.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:07.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.992 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:07.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:07.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:08.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:08.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:08.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:08.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:08.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:08.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:08.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:08.398 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:08.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:08.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:32:08.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:08.539 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:08.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:08.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:08.680 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:08.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:08.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:08.819 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:08.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:08.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:08.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:08.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:09.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:09.110 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:09.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:09.189 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:09 smithi123 bash[19119]: cluster 2024-02-12T16:32:08.069606+0000 mgr.y (mgr.14146) 1425 : cluster [DBG] pgmap v2006: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 19852/185828 objects misplaced (10.683%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:09.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:09.241 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:09.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:09.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:09.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:09.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:09.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:09.500 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:09 smithi121 bash[18489]: cluster 2024-02-12T16:32:08.069606+0000 mgr.y (mgr.14146) 1425 : cluster [DBG] pgmap v2006: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 19852/185828 objects misplaced (10.683%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:09.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:09.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:09 smithi169 bash[19604]: cluster 2024-02-12T16:32:08.069606+0000 mgr.y (mgr.14146) 1425 : cluster [DBG] pgmap v2006: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 19852/185828 objects misplaced (10.683%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:09.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:09.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:09.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:09.762 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:09.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:09.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:09.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:09.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:09.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:10.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:10.044 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:10.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:10.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:10.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:10.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:10.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:10 smithi123 bash[19119]: cluster 2024-02-12T16:32:09.270822+0000 osd.10 (osd.10) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:10.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:10 smithi123 bash[19119]: cluster 2024-02-12T16:32:09.270888+0000 osd.10 (osd.10) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:10.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:10.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:10.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:10.525 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:10.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:10 smithi169 bash[19604]: cluster 2024-02-12T16:32:09.270822+0000 osd.10 (osd.10) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:10 smithi169 bash[19604]: cluster 2024-02-12T16:32:09.270888+0000 osd.10 (osd.10) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:10.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:10.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:10.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:10.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:10 smithi121 bash[18489]: cluster 2024-02-12T16:32:09.270822+0000 osd.10 (osd.10) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:10.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:10 smithi121 bash[18489]: cluster 2024-02-12T16:32:09.270888+0000 osd.10 (osd.10) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:10.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:10.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:10.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:10.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:10.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:10.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:10.986 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:11.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:11.087 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:11.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:11.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:11.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:11.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:11.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:11.346 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:11.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:11.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:11 smithi123 bash[19119]: cluster 2024-02-12T16:32:10.070111+0000 mgr.y (mgr.14146) 1426 : cluster [DBG] pgmap v2007: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s; 19448/182710 objects misplaced (10.644%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:11 smithi123 bash[19119]: cluster 2024-02-12T16:32:10.194260+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e702: 12 total, 12 up, 12 in 2024-02-12T16:32:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:11 smithi123 bash[19119]: cluster 2024-02-12T16:32:10.748007+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in 2024-02-12T16:32:11.422 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-02-12T16:32:11.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T16:32:11.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:11.481 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:11.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:11 smithi169 bash[19604]: cluster 2024-02-12T16:32:10.070111+0000 mgr.y (mgr.14146) 1426 : cluster [DBG] pgmap v2007: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s; 19448/182710 objects misplaced (10.644%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:11 smithi169 bash[19604]: cluster 2024-02-12T16:32:10.194260+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e702: 12 total, 12 up, 12 in 2024-02-12T16:32:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:11 smithi169 bash[19604]: cluster 2024-02-12T16:32:10.748007+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in 2024-02-12T16:32:11.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:11.592 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:11 smithi121 bash[18489]: cluster 2024-02-12T16:32:10.070111+0000 mgr.y (mgr.14146) 1426 : cluster [DBG] pgmap v2007: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s; 19448/182710 objects misplaced (10.644%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:11.592 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:11 smithi121 bash[18489]: cluster 2024-02-12T16:32:10.194260+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e702: 12 total, 12 up, 12 in 2024-02-12T16:32:11.592 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:11 smithi121 bash[18489]: cluster 2024-02-12T16:32:10.748007+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e703: 12 total, 12 up, 12 in 2024-02-12T16:32:11.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:11.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:11.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:11.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:11.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:11.907 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:11.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:12.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:12.050 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:12.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:12.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:12.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:12.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:12.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:12.337 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:12.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:12.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:12.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:12.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:12.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:12.623 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:12.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:12.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:12.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:12.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:12.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:12.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:12.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:12.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:12.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:13.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:13.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:13.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:13.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:13.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:13.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:13.314 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:13.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:13.393 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:13 smithi121 bash[18489]: cluster 2024-02-12T16:32:11.477821+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.17 deep-scrub ok 2024-02-12T16:32:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:13 smithi121 bash[18489]: cluster 2024-02-12T16:32:12.070610+0000 mgr.y (mgr.14146) 1427 : cluster [DBG] pgmap v2010: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 18482/176788 objects misplaced (10.454%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:32:13.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:13.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:13.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:13.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:13.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:13 smithi169 bash[19604]: cluster 2024-02-12T16:32:11.477821+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.17 deep-scrub ok 2024-02-12T16:32:13.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:13 smithi169 bash[19604]: cluster 2024-02-12T16:32:12.070610+0000 mgr.y (mgr.14146) 1427 : cluster [DBG] pgmap v2010: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 18482/176788 objects misplaced (10.454%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:32:13.581 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:13.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:13 smithi123 bash[19119]: cluster 2024-02-12T16:32:11.477821+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.17 deep-scrub ok 2024-02-12T16:32:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:13 smithi123 bash[19119]: cluster 2024-02-12T16:32:12.070610+0000 mgr.y (mgr.14146) 1427 : cluster [DBG] pgmap v2010: 57 pgs: 1 active+clean+scrubbing+deep, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 51 active+clean; 5.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 18482/176788 objects misplaced (10.454%); 1.6 MiB/s, 25 objects/s recovering 2024-02-12T16:32:13.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:13.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:13.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:13.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:14.026 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:14.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:14.166 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:14.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:14.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:14.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:14.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:14.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:14.559 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:14.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:14.702 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:14.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:14.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:14.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:14.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:14.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:15.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:15.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:15.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:15.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:15.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:15.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:15.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:15.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:15.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:15.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:15.490 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:15 smithi123 bash[19119]: cluster 2024-02-12T16:32:14.071166+0000 mgr.y (mgr.14146) 1428 : cluster [DBG] pgmap v2011: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 314 op/s; 18092/172420 objects misplaced (10.493%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:15.530 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:15.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:15 smithi169 bash[19604]: cluster 2024-02-12T16:32:14.071166+0000 mgr.y (mgr.14146) 1428 : cluster [DBG] pgmap v2011: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 314 op/s; 18092/172420 objects misplaced (10.493%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:15.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:15.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:15 smithi121 bash[18489]: cluster 2024-02-12T16:32:14.071166+0000 mgr.y (mgr.14146) 1428 : cluster [DBG] pgmap v2011: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 314 op/s; 18092/172420 objects misplaced (10.493%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:15.671 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:15.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:15.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:15.802 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:15.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:15.905 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:15.935 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:15.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:16.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:16.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:16.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:16.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:16.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:16.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:16.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:16.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:16.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:16.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:16.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:16.483 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:16.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:16.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:16.623 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:16.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:16.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:16.767 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:16.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:16.811 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:16.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:16.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:16.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:16.914 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:16 smithi121 bash[18489]: cluster 2024-02-12T16:32:16.071680+0000 mgr.y (mgr.14146) 1429 : cluster [DBG] pgmap v2012: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 308 op/s; 17336/167840 objects misplaced (10.329%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:17.005 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:17.012 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:16 smithi169 bash[19604]: cluster 2024-02-12T16:32:16.071680+0000 mgr.y (mgr.14146) 1429 : cluster [DBG] pgmap v2012: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 308 op/s; 17336/167840 objects misplaced (10.329%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:17.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:17.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:17.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:17.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:16 smithi123 bash[19119]: cluster 2024-02-12T16:32:16.071680+0000 mgr.y (mgr.14146) 1429 : cluster [DBG] pgmap v2012: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 308 op/s; 17336/167840 objects misplaced (10.329%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:17.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:17.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:17.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:17.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:17.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:17.451 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:17.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:17.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:17.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:17.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:17.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:17.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:17.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:17.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:17.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:17.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:17.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:18.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:18.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:18.216 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:32:18.216 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:32:18.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:32:18.216 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:32:18.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.216 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:125506 6.f 6:f205cef0:::benchmark_data_smithi169_96224_object125505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-02-12T16:32:18.216 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:30:26.992110+0000", 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.21508231, 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32506945100000001, 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 125506 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992110+0000", 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992110+0000", 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992108+0000", 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5992000000000003e-05 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992154+0000", 2024-02-12T16:32:18.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0746e-05 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992165+0000", 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.046e-06 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992172+0000", 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030593579999999999 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.995231+0000", 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5210000000000007e-05 2024-02-12T16:32:18.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.995306+0000", 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7973999999999994e-05 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.995404+0000", 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32162321100000002 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317028+0000", 2024-02-12T16:32:18.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6496999999999997e-05 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317094+0000", 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0518e-05 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317145+0000", 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4313000000000002e-05 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317179+0000", 2024-02-12T16:32:18.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32506945100000001 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:125509 6.f 6:f0286b31:::benchmark_data_smithi169_96224_object125508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:30:26.992601+0000", 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.214591111, 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.324736572, 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.222 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 125509 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992601+0000", 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992601+0000", 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992599+0000", 2024-02-12T16:32:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013533299999999999 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992735+0000", 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1236999999999999e-05 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992746+0000", 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1940000000000003e-06 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.992752+0000", 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063471350000000003 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.999099+0000", 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118065 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.999217+0000", 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013904100000000001 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:26.999356+0000", 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31768661799999998 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317043+0000", 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021142900000000001 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317254+0000", 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0198999999999998e-05 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317304+0000", 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2891000000000001e-05 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317337+0000", 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.324736572 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:125522 6.f 6:f85803ba:::benchmark_data_smithi169_96224_object125521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:30:27.001891+0000", 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.20530079700001, 2024-02-12T16:32:18.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32238689100000001, 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 125522 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.001891+0000", 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.001891+0000", 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.001889+0000", 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100181 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.001989+0000", 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2609000000000001e-05 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.002001+0000", 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5060000000000001e-06 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.002008+0000", 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013852949999999999 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.003393+0000", 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000124213 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.003517+0000", 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30993001599999997 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.313447+0000", 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010722819999999999 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.324170+0000", 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.658e-05 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.324197+0000", 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0238000000000002e-05 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.324247+0000", 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0864999999999997e-05 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.324278+0000", 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.32238689100000001 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.232 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26257.0:125506 6.f e658/636 ondisk, result = 0)", 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:30:27.005232+0000", 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.201959965, 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31195691599999997, 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.005232+0000", 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.005232+0000", 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.005230+0000", 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2700000000000004e-06 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.005238+0000", 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30811075599999999 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.313349+0000", 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2713e-05 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.313362+0000", 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003718603 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317080+0000", 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3840000000000002e-06 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317087+0000", 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010211 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317189+0000", 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31195691599999997 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:125538 6.f 6:f4881649:::benchmark_data_smithi169_96224_object125537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-02-12T16:32:18.236 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:30:27.013463+0000", 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.19372839499999, 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31299448299999999, 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 125538 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.013463+0000", 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.013463+0000", 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.013461+0000", 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1211000000000006e-05 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.013532+0000", 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978150799999997 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.313313+0000", 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7033999999999997e-05 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.313361+0000", 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040229369999999999 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317383+0000", 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119805 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317503+0000", 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013902000000000001 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.317642+0000", 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0086932020000000006 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.326335+0000", 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5096999999999998e-05 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.326391+0000", 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0543000000000002e-05 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.326421+0000", 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6826999999999998e-05 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:27.326458+0000", 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31299448299999999 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:149425 6.f 6:fa9d18f7:::benchmark_data_smithi169_96224_object149424:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e664)", 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:30:53.941513+0000", 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.265678876999999, 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12909204899999999, 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 149425 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:53.941513+0000", 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:53.941513+0000", 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:53.941511+0000", 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3052999999999994e-05 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:53.941574+0000", 2024-02-12T16:32:18.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1201e-05 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:53.941585+0000", 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2589999999999997e-06 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:53.941591+0000", 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103950319 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:54.045541+0000", 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015748699999999999 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:54.045699+0000", 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00029625300000000001 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:54.045995+0000", 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01465079 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:54.060646+0000", 2024-02-12T16:32:18.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098530309999999999 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:54.070499+0000", 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8010999999999997e-05 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:54.070557+0000", 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8047999999999998e-05 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:54.070605+0000", 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12909204899999999 2024-02-12T16:32:18.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:151283 6.f 6:f6a6efd9:::benchmark_data_smithi169_96224_object151282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e666)", 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:30:56.571386+0000", 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.635806324000001, 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13200439999999999, 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 151283 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.571386+0000", 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.571386+0000", 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.571384+0000", 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010539099999999999 2024-02-12T16:32:18.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.571490+0000", 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.393e-05 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.571504+0000", 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.55e-06 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.571510+0000", 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.088174720999999998 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.659685+0000", 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141006 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.659826+0000", 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018278700000000001 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.660009+0000", 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013948004 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.673957+0000", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029354292000000001 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.703311+0000", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4799999999999999e-05 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.703346+0000", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4132999999999999e-05 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:30:56.703390+0000", 2024-02-12T16:32:18.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13200439999999999 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:154188 6.f 6:f077e1be:::benchmark_data_smithi169_96224_object1352:head [delete] snapc 0=[] ondisk+write+known_if_redirected e666)", 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:01.288669+0000", 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.918523202000003, 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13013376900000001, 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 154188 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.288669+0000", 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.288669+0000", 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.288667+0000", 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4760000000000004e-06 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.288676+0000", 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3356999999999999e-05 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.288690+0000", 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2840000000000003e-06 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.288696+0000", 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105145931 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.393842+0000", 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013979500000000001 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.393982+0000", 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017050400000000001 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.394152+0000", 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082430139999999999 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.402395+0000", 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016280907000000001 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.418676+0000", 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6984999999999996e-05 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.418743+0000", 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9262999999999998e-05 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.418802+0000", 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13013376900000001 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:154632 6.f 6:f6bdfb9a:::benchmark_data_smithi169_96224_object1796:head [delete] snapc 0=[] ondisk+write+known_if_redirected e666)", 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:01.863306+0000", 2024-02-12T16:32:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.343885990999993, 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13385513399999999, 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 154632 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.863306+0000", 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.863306+0000", 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.863304+0000", 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9510000000000001e-06 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.863314+0000", 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1902000000000001e-05 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.863326+0000", 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7780000000000004e-06 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.863331+0000", 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104592373 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.967924+0000", 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013621400000000001 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.968060+0000", 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000160054 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.968220+0000", 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042619770000000001 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.972482+0000", 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024610818 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.997093+0000", 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8704000000000001e-05 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.997131+0000", 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9518000000000001e-05 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.997161+0000", 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13385513399999999 2024-02-12T16:32:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:154634 6.f 6:fdc193d8:::benchmark_data_smithi169_96224_object1798:head [delete] snapc 0=[] ondisk+write+known_if_redirected e666)", 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:01.867713+0000", 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.339478886999999, 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.141997919, 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 154634 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.867713+0000", 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.867713+0000", 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.867711+0000", 2024-02-12T16:32:18.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5080000000000005e-06 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.867720+0000", 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1559999999999999e-05 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.867731+0000", 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2909999999999998e-06 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.867736+0000", 2024-02-12T16:32:18.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10545539700000001 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.973192+0000", 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1089999999999998e-05 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.973273+0000", 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9467000000000003e-05 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.973352+0000", 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011828861 2024-02-12T16:32:18.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:01.985181+0000", 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024452303000000002 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.009634+0000", 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3393999999999999e-05 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.009667+0000", 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3918000000000002e-05 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.009711+0000", 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.141997919 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:155331 6.f 6:fbc497e5:::benchmark_data_smithi169_96224_object2495:head [delete] snapc 0=[] ondisk+write+known_if_redirected e666)", 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:02.730428+0000", 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.476763998999999, 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14053428200000001, 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 155331 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.730428+0000", 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.730428+0000", 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:32:18.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.730425+0000", 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9890000000000004e-06 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.730435+0000", 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3135e-05 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.730448+0000", 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5209999999999997e-06 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.730455+0000", 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001578632 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.732034+0000", 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8294999999999995e-05 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.732112+0000", 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9135999999999995e-05 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.732201+0000", 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.138675101 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870876+0000", 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8161000000000001e-05 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870895+0000", 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9455000000000001e-05 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870934+0000", 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8161999999999999e-05 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870962+0000", 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.14053428200000001 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26257.0:155331 6.f e666/636 ondisk, result = 0)", 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:02.742777+0000", 2024-02-12T16:32:18.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.464415099000007, 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12810281800000001, 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742777+0000", 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742777+0000", 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742775+0000", 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7189999999999996e-06 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742784+0000", 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0668e-05 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742795+0000", 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0909999999999998e-06 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742801+0000", 2024-02-12T16:32:18.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.128064818 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870866+0000", 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2229999999999998e-06 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870871+0000", 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3739999999999998e-06 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870880+0000", 2024-02-12T16:32:18.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12810281800000001 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.2.0:104829 6.f e666/636 ondisk, result = 0)", 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:02.742923+0000", 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.464268419000007, 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12807811799999999, 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742923+0000", 2024-02-12T16:32:18.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742923+0000", 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742922+0000", 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4040000000000003e-06 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742928+0000", 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9389999999999997e-06 2024-02-12T16:32:18.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742936+0000", 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.622e-06 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.742940+0000", 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12805103300000001 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870991+0000", 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8689999999999997e-06 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.870995+0000", 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4620000000000001e-06 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.871002+0000", 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12807811799999999 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.280 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:155347 6.f 6:f4bf4fb2:::benchmark_data_smithi169_96224_object2511:head [delete] snapc 0=[] ondisk+write+known_if_redirected e666)", 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:02.749571+0000", 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.457620680999995, 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15040542600000001, 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 155347 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.749571+0000", 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.749571+0000", 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.749568+0000", 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3729999999999999e-06 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.749578+0000", 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022642078999999999 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.772220+0000", 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7397e-05 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.772237+0000", 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100040962 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.872278+0000", 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4448000000000002e-05 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.872363+0000", 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000132049 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.872495+0000", 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098715109999999995 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.882366+0000", 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.017509410999999999 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.899876+0000", 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3078000000000002e-05 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.899929+0000", 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7867000000000001e-05 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:02.899977+0000", 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15040542600000001 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:156025 6.f 6:f4238f49:::benchmark_data_smithi169_96224_object3189:head [delete] snapc 0=[] ondisk+write+known_if_redirected e667)", 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:03.672699+0000", 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.534492693999994, 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13100489600000001, 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 156025 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672699+0000", 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672699+0000", 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672697+0000", 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9020000000000004e-06 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672703+0000", 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5729999999999997e-06 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672713+0000", 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-06 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672717+0000", 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105357 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672823+0000", 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5254000000000007e-05 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.672918+0000", 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9639e-05 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.673008+0000", 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006080823 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.679089+0000", 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.124527656 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.803616+0000", 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7922e-05 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.803664+0000", 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9918e-05 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:03.803704+0000", 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13100489600000001 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:157307 6.f 6:f21ffccf:::benchmark_data_smithi169_96224_object4471:head [delete] snapc 0=[] ondisk+write+known_if_redirected e667)", 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:05.406370+0000", 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.800821530999997, 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13774598599999999, 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.291 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157307 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.406370+0000", 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.406370+0000", 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.406369+0000", 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9130000000000002e-06 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.406374+0000", 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4120000000000007e-06 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.406383+0000", 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2010000000000001e-06 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.406388+0000", 2024-02-12T16:32:18.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.096977731999999997 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.503365+0000", 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6562000000000002e-05 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.503452+0000", 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010336900000000001 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.503555+0000", 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010050966 2024-02-12T16:32:18.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.513606+0000", 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030444652999999999 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.544051+0000", 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9622000000000001e-05 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.544091+0000", 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5809e-05 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.544116+0000", 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13774598599999999 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:157310 6.f 6:f0579cac:::benchmark_data_smithi169_96224_object4474:head [delete] snapc 0=[] ondisk+write+known_if_redirected e667)", 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:05.407223+0000", 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.799968824000004, 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13698807599999999, 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157310 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.407223+0000", 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.407223+0000", 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:32:18.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.407222+0000", 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3450000000000002e-06 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.407228+0000", 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1425e-05 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.407240+0000", 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4509999999999997e-06 2024-02-12T16:32:18.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.407246+0000", 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099576612999999994 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.506823+0000", 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103162 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.506926+0000", 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9635000000000006e-05 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.507006+0000", 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066102060000000004 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.513616+0000", 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030546686999999999 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.544162+0000", 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6971e-05 2024-02-12T16:32:18.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.544189+0000", 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1708999999999998e-05 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.544211+0000", 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13698807599999999 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:157402 6.f 6:f9766d23:::benchmark_data_smithi169_96224_object4566:head [delete] snapc 0=[] ondisk+write+known_if_redirected e667)", 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:05.550680+0000", 2024-02-12T16:32:18.301 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.656512175000003, 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13463545900000001, 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157402 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.550680+0000", 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.550680+0000", 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.550677+0000", 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2204e-05 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.550689+0000", 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1022999999999999e-05 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.550700+0000", 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6450000000000002e-06 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.550706+0000", 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.116098204 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.666804+0000", 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8665999999999994e-05 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.666903+0000", 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011728899999999999 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.667020+0000", 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0085334120000000006 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.675554+0000", 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097063849999999997 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.685260+0000", 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1139000000000001e-05 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.685291+0000", 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4088e-05 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:05.685315+0000", 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13463545900000001 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:157820 6.f 6:f3c687b4:::benchmark_data_smithi169_96224_object4984:head [delete] snapc 0=[] ondisk+write+known_if_redirected e667)", 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:06.206882+0000", 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.000310232000004, 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13848407200000001, 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157820 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.307 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.206882+0000", 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.206882+0000", 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.206880+0000", 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3939999999999998e-06 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.206887+0000", 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5458999999999999e-05 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.206902+0000", 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2219999999999999e-06 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.206909+0000", 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.038714780999999997 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.245623+0000", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011247399999999999 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.245736+0000", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012881500000000001 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.245865+0000", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0096788059999999999 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.255543+0000", 2024-02-12T16:32:18.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089731250999999998 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.345275+0000", 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1156000000000003e-05 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.345326+0000", 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9814999999999997e-05 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.345366+0000", 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13848407200000001 2024-02-12T16:32:18.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:157823 6.f 6:f2ec3972:::benchmark_data_smithi169_96224_object4987:head [delete] snapc 0=[] ondisk+write+known_if_redirected e667)", 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:06.211005+0000", 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 71.996187282999998, 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13448210699999999, 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157823 2024-02-12T16:32:18.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.211005+0000", 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.211005+0000", 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.211003+0000", 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7780000000000002e-06 2024-02-12T16:32:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.211009+0000", 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2611e-05 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.211022+0000", 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0231e-05 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.211042+0000", 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.039385348000000001 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.250428+0000", 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010237400000000001 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.250530+0000", 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012758499999999999 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.250657+0000", 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.094717988000000003 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.345375+0000", 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1027e-05 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.345407+0000", 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7910999999999997e-05 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.345444+0000", 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2264000000000001e-05 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:32:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:06.345487+0000", 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.13448210699999999 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:32:18.317 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:32:18.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:18.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:18.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:18.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:18.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:18.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:18.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:18.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:18.886 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:18.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:18.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:19.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:19.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:19.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:19.156 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:19.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:19.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:19.270 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:19 smithi121 bash[18489]: cluster 2024-02-12T16:32:18.072216+0000 mgr.y (mgr.14146) 1430 : cluster [DBG] pgmap v2013: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 474 op/s; 17076/162058 objects misplaced (10.537%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:19.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:19.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:19 smithi123 bash[19119]: cluster 2024-02-12T16:32:18.072216+0000 mgr.y (mgr.14146) 1430 : cluster [DBG] pgmap v2013: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 474 op/s; 17076/162058 objects misplaced (10.537%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:19.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:19.429 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:19.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:19.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:19 smithi169 bash[19604]: cluster 2024-02-12T16:32:18.072216+0000 mgr.y (mgr.14146) 1430 : cluster [DBG] pgmap v2013: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 474 op/s; 17076/162058 objects misplaced (10.537%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:19.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:19.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:19.667 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:19.731 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:19.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:19.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:19.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:19.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:19.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:20.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:32:20.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:20.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:20.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:20.287 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:20.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:20 smithi123 bash[19119]: cluster 2024-02-12T16:32:19.111072+0000 osd.1 (osd.1) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:20.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:20 smithi123 bash[19119]: cluster 2024-02-12T16:32:19.111128+0000 osd.1 (osd.1) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:20.392 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:20 smithi123 bash[19119]: cluster 2024-02-12T16:32:19.130487+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e704: 12 total, 12 up, 12 in 2024-02-12T16:32:20.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:20.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:20 smithi121 bash[18489]: cluster 2024-02-12T16:32:19.111072+0000 osd.1 (osd.1) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:20.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:20 smithi121 bash[18489]: cluster 2024-02-12T16:32:19.111128+0000 osd.1 (osd.1) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:20 smithi121 bash[18489]: cluster 2024-02-12T16:32:19.130487+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e704: 12 total, 12 up, 12 in 2024-02-12T16:32:20.417 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:20.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:20 smithi169 bash[19604]: cluster 2024-02-12T16:32:19.111072+0000 osd.1 (osd.1) 104 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:20 smithi169 bash[19604]: cluster 2024-02-12T16:32:19.111128+0000 osd.1 (osd.1) 105 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:20 smithi169 bash[19604]: cluster 2024-02-12T16:32:19.130487+0000 mon.a (mon.0) 2428 : cluster [DBG] osdmap e704: 12 total, 12 up, 12 in 2024-02-12T16:32:20.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:20.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:20.699 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:20.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:20.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:20.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:20.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:20.962 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:20.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:21.051 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:32:21.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:21.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:32:21.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:21.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:21.124 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:32:21.165 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:32:21.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:21.195 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:32:21.201 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:21 smithi169 bash[19604]: cluster 2024-02-12T16:32:20.072717+0000 mgr.y (mgr.14146) 1431 : cluster [DBG] pgmap v2015: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 16636/158518 objects misplaced (10.495%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:32:21.201 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:21 smithi169 bash[19604]: cluster 2024-02-12T16:32:20.751119+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e705: 12 total, 12 up, 12 in 2024-02-12T16:32:21.201 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:21 smithi169 bash[19604]: audit 2024-02-12T16:32:21.013368+0000 mon.b (mon.2) 144 : audit [DBG] from='client.? v1:172.21.15.121:0/882085972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:21.216 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:21.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:21.226 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:32:21.257 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:32:21.285 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:32:21.318 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:32:21.319 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:21.326 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:21 smithi123 bash[19119]: cluster 2024-02-12T16:32:20.072717+0000 mgr.y (mgr.14146) 1431 : cluster [DBG] pgmap v2015: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 16636/158518 objects misplaced (10.495%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:32:21.326 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:21 smithi123 bash[19119]: cluster 2024-02-12T16:32:20.751119+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e705: 12 total, 12 up, 12 in 2024-02-12T16:32:21.326 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:21 smithi123 bash[19119]: audit 2024-02-12T16:32:21.013368+0000 mon.b (mon.2) 144 : audit [DBG] from='client.? v1:172.21.15.121:0/882085972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:21.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:32:21.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:21.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:21.375 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:32:21.402 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:32:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:21 smithi121 bash[18489]: cluster 2024-02-12T16:32:20.072717+0000 mgr.y (mgr.14146) 1431 : cluster [DBG] pgmap v2015: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 16636/158518 objects misplaced (10.495%); 1.5 MiB/s, 24 objects/s recovering 2024-02-12T16:32:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:21 smithi121 bash[18489]: cluster 2024-02-12T16:32:20.751119+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e705: 12 total, 12 up, 12 in 2024-02-12T16:32:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:21 smithi121 bash[18489]: audit 2024-02-12T16:32:21.013368+0000 mon.b (mon.2) 144 : audit [DBG] from='client.? v1:172.21.15.121:0/882085972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:21.416 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.417 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:32:21.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:21.479 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:21.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:21.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:21.613 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:21.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:21.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:21.743 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:21.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:21.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:21.848 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:21.901 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:21.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:22.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:22.035 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:22.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:22.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:22.164 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:22.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:22.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:22.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:22.291 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:22.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:22.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:22.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:22.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:22.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:22.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:22.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:22.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:22.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:22.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:22.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:22.936 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:22.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:23.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:23.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:23.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:23.157 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:23 smithi123 bash[19119]: cluster 2024-02-12T16:32:22.073228+0000 mgr.y (mgr.14146) 1432 : cluster [DBG] pgmap v2017: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 15593/151628 objects misplaced (10.284%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:23.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:23.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:23.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:23.366 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:23.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:23 smithi121 bash[18489]: cluster 2024-02-12T16:32:22.073228+0000 mgr.y (mgr.14146) 1432 : cluster [DBG] pgmap v2017: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 15593/151628 objects misplaced (10.284%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:23.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:23.480 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:23 smithi169 bash[19604]: cluster 2024-02-12T16:32:22.073228+0000 mgr.y (mgr.14146) 1432 : cluster [DBG] pgmap v2017: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 15593/151628 objects misplaced (10.284%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:23.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:23.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:23.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:23.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:23.628 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:23.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:23.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:23.760 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:23.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:23.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:23.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:23.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:23.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:24.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:24.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:24.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:24.210 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:24.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:24.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:24.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:24.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:24.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:24.504 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:24.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:24.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:24.635 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:24.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:24.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:24.772 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:24.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:24.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:24.910 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:32:24.918 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:24.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:25.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:25.053 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:25.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:25.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:32:25.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:25.179 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:32:25.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:25.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:25.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:25.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:25.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:25.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:25 smithi121 bash[18489]: cluster 2024-02-12T16:32:24.073791+0000 mgr.y (mgr.14146) 1433 : cluster [DBG] pgmap v2018: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 15171/146780 objects misplaced (10.336%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:25.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:25 smithi121 bash[18489]: audit 2024-02-12T16:32:24.187092+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.121:0/3858282069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:32:25.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:25.512 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:25 smithi123 bash[19119]: cluster 2024-02-12T16:32:24.073791+0000 mgr.y (mgr.14146) 1433 : cluster [DBG] pgmap v2018: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 15171/146780 objects misplaced (10.336%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:25.512 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:25 smithi123 bash[19119]: audit 2024-02-12T16:32:24.187092+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.121:0/3858282069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:32:25.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:25.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:25 smithi169 bash[19604]: cluster 2024-02-12T16:32:24.073791+0000 mgr.y (mgr.14146) 1433 : cluster [DBG] pgmap v2018: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 15171/146780 objects misplaced (10.336%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:25 smithi169 bash[19604]: audit 2024-02-12T16:32:24.187092+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.121:0/3858282069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:32:25.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:25.685 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:25.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:25.765 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T16:32:25.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:25.821 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:25.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:25.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:25.982 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:25.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:26.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:26.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:26.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:26.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:26.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:26.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:26.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:26.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:26 smithi123 bash[19119]: cluster 2024-02-12T16:32:25.152846+0000 mon.a (mon.0) 2431 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:26.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:26 smithi123 bash[19119]: audit 2024-02-12T16:32:25.159312+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? v1:172.21.15.121:0/3858282069' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:32:26.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:26 smithi123 bash[19119]: cluster 2024-02-12T16:32:25.159369+0000 mon.a (mon.0) 2433 : cluster [DBG] osdmap e706: 12 total, 12 up, 12 in 2024-02-12T16:32:26.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:26 smithi123 bash[19119]: audit 2024-02-12T16:32:25.306741+0000 mon.a (mon.0) 2434 : audit [INF] from='client.? v1:172.21.15.121:0/1510651082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:32:26.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:26 smithi123 bash[19119]: audit 2024-02-12T16:32:25.752675+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? v1:172.21.15.121:0/1510651082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:32:26.383 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:26 smithi123 bash[19119]: cluster 2024-02-12T16:32:25.752721+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e707: 12 total, 12 up, 12 in 2024-02-12T16:32:26.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:26 smithi121 bash[18489]: cluster 2024-02-12T16:32:25.152846+0000 mon.a (mon.0) 2431 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:26 smithi121 bash[18489]: audit 2024-02-12T16:32:25.159312+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? v1:172.21.15.121:0/3858282069' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:32:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:26 smithi121 bash[18489]: cluster 2024-02-12T16:32:25.159369+0000 mon.a (mon.0) 2433 : cluster [DBG] osdmap e706: 12 total, 12 up, 12 in 2024-02-12T16:32:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:26 smithi121 bash[18489]: audit 2024-02-12T16:32:25.306741+0000 mon.a (mon.0) 2434 : audit [INF] from='client.? v1:172.21.15.121:0/1510651082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:32:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:26 smithi121 bash[18489]: audit 2024-02-12T16:32:25.752675+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? v1:172.21.15.121:0/1510651082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:32:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:26 smithi121 bash[18489]: cluster 2024-02-12T16:32:25.752721+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e707: 12 total, 12 up, 12 in 2024-02-12T16:32:26.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:26.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:26.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:26 smithi169 bash[19604]: cluster 2024-02-12T16:32:25.152846+0000 mon.a (mon.0) 2431 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:26 smithi169 bash[19604]: audit 2024-02-12T16:32:25.159312+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? v1:172.21.15.121:0/3858282069' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:32:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:26 smithi169 bash[19604]: cluster 2024-02-12T16:32:25.159369+0000 mon.a (mon.0) 2433 : cluster [DBG] osdmap e706: 12 total, 12 up, 12 in 2024-02-12T16:32:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:26 smithi169 bash[19604]: audit 2024-02-12T16:32:25.306741+0000 mon.a (mon.0) 2434 : audit [INF] from='client.? v1:172.21.15.121:0/1510651082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:32:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:26 smithi169 bash[19604]: audit 2024-02-12T16:32:25.752675+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? v1:172.21.15.121:0/1510651082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:32:26.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:26 smithi169 bash[19604]: cluster 2024-02-12T16:32:25.752721+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e707: 12 total, 12 up, 12 in 2024-02-12T16:32:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:26.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:26.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:26.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:26.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:26.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:26.848 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:26.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:26.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:26.972 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:26.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:27.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T16:32:27.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:27.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:27.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:27.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:27 smithi169 bash[19604]: cluster 2024-02-12T16:32:26.074246+0000 mgr.y (mgr.14146) 1434 : cluster [DBG] pgmap v2021: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 14337/141954 objects misplaced (10.100%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:32:27.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:27.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:27.286 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:27 smithi121 bash[18489]: cluster 2024-02-12T16:32:26.074246+0000 mgr.y (mgr.14146) 1434 : cluster [DBG] pgmap v2021: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 14337/141954 objects misplaced (10.100%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:32:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:27.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:27.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:27.408 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:27.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:27.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:27.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:27.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:27.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:27 smithi123 bash[19119]: cluster 2024-02-12T16:32:26.074246+0000 mgr.y (mgr.14146) 1434 : cluster [DBG] pgmap v2021: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 14337/141954 objects misplaced (10.100%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:32:27.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:27.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:27.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:27.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:27.801 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:27.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:27.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:27.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:28.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:28.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:28.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:28.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:28.219 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:28.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:28.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:28.361 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:28.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:28.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:28.504 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:28.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:28.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:28.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:28.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:28.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:32:28.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:28.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:28.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:28.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:28.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:28.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:28.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:29.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:29.037 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:29.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:29.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:29.190 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:29.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:29.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:29.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:29.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:29.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:29.440 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:29 smithi121 bash[18489]: cluster 2024-02-12T16:32:28.074780+0000 mgr.y (mgr.14146) 1435 : cluster [DBG] pgmap v2022: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 14126/136582 objects misplaced (10.343%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:29.461 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:29.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:29 smithi169 bash[19604]: cluster 2024-02-12T16:32:28.074780+0000 mgr.y (mgr.14146) 1435 : cluster [DBG] pgmap v2022: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 14126/136582 objects misplaced (10.343%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:29.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:29.575 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:29 smithi123 bash[19119]: cluster 2024-02-12T16:32:28.074780+0000 mgr.y (mgr.14146) 1435 : cluster [DBG] pgmap v2022: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 14126/136582 objects misplaced (10.343%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:29.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:29.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:29.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:29.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:29.861 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:29.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:29.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:30.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:30.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:30.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:30.155 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:30.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:30.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:30.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:30.408 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:30.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:30.523 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:30.532 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:30.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:30.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:30.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:30.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:30.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:30.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:31.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:31.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:31.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:31.108 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:32:31.108 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:32:31.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:31.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:31.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:31.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:31.359 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:31.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:31.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:31.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:31 smithi121 bash[18489]: cluster 2024-02-12T16:32:30.075255+0000 mgr.y (mgr.14146) 1436 : cluster [DBG] pgmap v2023: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 13722/133100 objects misplaced (10.310%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:31.494 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:31.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:31.536 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:31 smithi169 bash[19604]: cluster 2024-02-12T16:32:30.075255+0000 mgr.y (mgr.14146) 1436 : cluster [DBG] pgmap v2023: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 13722/133100 objects misplaced (10.310%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:31.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:31.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:31.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:31 smithi123 bash[19119]: cluster 2024-02-12T16:32:30.075255+0000 mgr.y (mgr.14146) 1436 : cluster [DBG] pgmap v2023: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 13722/133100 objects misplaced (10.310%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:31.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:31.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:31.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:31.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:31.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:31.916 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:31.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:32.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:32.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:32.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:32:32.170 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:32.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:32.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:32.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:32.490 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:32.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:32.561 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:32.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:32.626 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:32.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:32.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:32.760 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:32.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:32.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:32.902 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:32.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:33.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:33.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:33.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:33.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:33.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:33.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:33.304 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:33.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:33.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:33.459 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:33.461 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:33 smithi123 bash[19119]: cluster 2024-02-12T16:32:31.422884+0000 osd.11 (osd.11) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:33.461 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:33 smithi123 bash[19119]: cluster 2024-02-12T16:32:31.422945+0000 osd.11 (osd.11) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:33.461 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:33 smithi123 bash[19119]: cluster 2024-02-12T16:32:32.075759+0000 mgr.y (mgr.14146) 1437 : cluster [DBG] pgmap v2024: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 12711/126522 objects misplaced (10.046%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:33.461 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:33 smithi123 bash[19119]: cluster 2024-02-12T16:32:32.182872+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e708: 12 total, 12 up, 12 in 2024-02-12T16:32:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:33.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:33 smithi169 bash[19604]: cluster 2024-02-12T16:32:31.422884+0000 osd.11 (osd.11) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:33 smithi169 bash[19604]: cluster 2024-02-12T16:32:31.422945+0000 osd.11 (osd.11) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:33 smithi169 bash[19604]: cluster 2024-02-12T16:32:32.075759+0000 mgr.y (mgr.14146) 1437 : cluster [DBG] pgmap v2024: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 12711/126522 objects misplaced (10.046%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:33 smithi169 bash[19604]: cluster 2024-02-12T16:32:32.182872+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e708: 12 total, 12 up, 12 in 2024-02-12T16:32:33.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:33.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:33 smithi121 bash[18489]: cluster 2024-02-12T16:32:31.422884+0000 osd.11 (osd.11) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:33.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:33 smithi121 bash[18489]: cluster 2024-02-12T16:32:31.422945+0000 osd.11 (osd.11) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:33.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:33 smithi121 bash[18489]: cluster 2024-02-12T16:32:32.075759+0000 mgr.y (mgr.14146) 1437 : cluster [DBG] pgmap v2024: 57 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 12711/126522 objects misplaced (10.046%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:33.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:33 smithi121 bash[18489]: cluster 2024-02-12T16:32:32.182872+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e708: 12 total, 12 up, 12 in 2024-02-12T16:32:33.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:33.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:33.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:33.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:33.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:33.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:33.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:33.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:33.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:34.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:34.182 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:34.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:34.295 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:34 smithi121 bash[18489]: cluster 2024-02-12T16:32:34.182057+0000 mon.a (mon.0) 2438 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:32:34.322 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:34.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:34.435 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:34 smithi123 bash[19119]: cluster 2024-02-12T16:32:34.182057+0000 mon.a (mon.0) 2438 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:32:34.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:34 smithi169 bash[19604]: cluster 2024-02-12T16:32:34.182057+0000 mon.a (mon.0) 2438 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:32:34.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:34.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:34.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:34.739 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:34.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:34.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:34.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:34.997 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:34.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:35.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:35.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:35.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:35.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:35.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:35.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:35.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:35.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:35.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:35.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:35.510 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:35 smithi123 bash[19119]: cluster 2024-02-12T16:32:34.076197+0000 mgr.y (mgr.14146) 1438 : cluster [DBG] pgmap v2026: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 12257/121972 objects misplaced (10.049%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:35.537 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:35.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:35 smithi169 bash[19604]: cluster 2024-02-12T16:32:34.076197+0000 mgr.y (mgr.14146) 1438 : cluster [DBG] pgmap v2026: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 12257/121972 objects misplaced (10.049%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:35.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:35.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:35 smithi121 bash[18489]: cluster 2024-02-12T16:32:34.076197+0000 mgr.y (mgr.14146) 1438 : cluster [DBG] pgmap v2026: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 12257/121972 objects misplaced (10.049%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:35.667 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:35.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:35.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:35.800 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:35.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:35.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:35.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:35.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:36.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:36.184 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:36.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:36.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:36.322 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:36.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:36.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:36.469 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:36.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:36.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:36.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:36.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:36.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:36.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:36.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:36.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:36.863 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:36.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:36.927 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:32:36.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:36.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:37.002 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:36 smithi121 bash[18489]: cluster 2024-02-12T16:32:35.753959+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e709: 12 total, 12 up, 12 in 2024-02-12T16:32:37.002 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:36 smithi121 bash[18489]: cluster 2024-02-12T16:32:36.076639+0000 mgr.y (mgr.14146) 1439 : cluster [DBG] pgmap v2028: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 11487/117384 objects misplaced (9.786%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:36 smithi169 bash[19604]: cluster 2024-02-12T16:32:35.753959+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e709: 12 total, 12 up, 12 in 2024-02-12T16:32:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:36 smithi169 bash[19604]: cluster 2024-02-12T16:32:36.076639+0000 mgr.y (mgr.14146) 1439 : cluster [DBG] pgmap v2028: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 11487/117384 objects misplaced (9.786%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:37.104 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:37.114 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:36 smithi123 bash[19119]: cluster 2024-02-12T16:32:35.753959+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e709: 12 total, 12 up, 12 in 2024-02-12T16:32:37.114 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:36 smithi123 bash[19119]: cluster 2024-02-12T16:32:36.076639+0000 mgr.y (mgr.14146) 1439 : cluster [DBG] pgmap v2028: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 11487/117384 objects misplaced (9.786%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:32:37.143 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:37.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:37.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:37.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:37.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:37.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:37.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:37.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:37.696 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:37.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:37.831 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:37.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:37.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:37.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:37 smithi169 bash[19604]: cluster 2024-02-12T16:32:36.601093+0000 osd.6 (osd.6) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:37 smithi169 bash[19604]: cluster 2024-02-12T16:32:36.601144+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:37 smithi169 bash[19604]: cluster 2024-02-12T16:32:36.769576+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e710: 12 total, 12 up, 12 in 2024-02-12T16:32:38.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:38.089 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:37 smithi123 bash[19119]: cluster 2024-02-12T16:32:36.601093+0000 osd.6 (osd.6) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:38.089 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:37 smithi123 bash[19119]: cluster 2024-02-12T16:32:36.601144+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:38.089 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:37 smithi123 bash[19119]: cluster 2024-02-12T16:32:36.769576+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e710: 12 total, 12 up, 12 in 2024-02-12T16:32:38.118 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:38.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:38.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:37 smithi121 bash[18489]: cluster 2024-02-12T16:32:36.601093+0000 osd.6 (osd.6) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:38.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:37 smithi121 bash[18489]: cluster 2024-02-12T16:32:36.601144+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:37 smithi121 bash[18489]: cluster 2024-02-12T16:32:36.769576+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e710: 12 total, 12 up, 12 in 2024-02-12T16:32:38.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:38.252 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:38.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:38.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:38.381 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:38.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:38.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:38.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:38.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:38.652 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:38.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:38.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:38.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:38.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:38.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:38.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:38 smithi121 bash[18489]: cluster 2024-02-12T16:32:37.577267+0000 osd.2 (osd.2) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:38.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:38 smithi121 bash[18489]: cluster 2024-02-12T16:32:37.577326+0000 osd.2 (osd.2) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:38.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:38 smithi121 bash[18489]: cluster 2024-02-12T16:32:37.773528+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e711: 12 total, 12 up, 12 in 2024-02-12T16:32:38.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:38 smithi121 bash[18489]: cluster 2024-02-12T16:32:38.077090+0000 mgr.y (mgr.14146) 1440 : cluster [DBG] pgmap v2031: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 11262/112408 objects misplaced (10.019%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:38.930 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:38.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:39.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:38 smithi169 bash[19604]: cluster 2024-02-12T16:32:37.577267+0000 osd.2 (osd.2) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:38 smithi169 bash[19604]: cluster 2024-02-12T16:32:37.577326+0000 osd.2 (osd.2) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:38 smithi169 bash[19604]: cluster 2024-02-12T16:32:37.773528+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e711: 12 total, 12 up, 12 in 2024-02-12T16:32:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:38 smithi169 bash[19604]: cluster 2024-02-12T16:32:38.077090+0000 mgr.y (mgr.14146) 1440 : cluster [DBG] pgmap v2031: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 11262/112408 objects misplaced (10.019%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:39.067 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:39.071 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:38 smithi123 bash[19119]: cluster 2024-02-12T16:32:37.577267+0000 osd.2 (osd.2) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:39.071 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:38 smithi123 bash[19119]: cluster 2024-02-12T16:32:37.577326+0000 osd.2 (osd.2) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:39.071 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:38 smithi123 bash[19119]: cluster 2024-02-12T16:32:37.773528+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e711: 12 total, 12 up, 12 in 2024-02-12T16:32:39.071 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:38 smithi123 bash[19119]: cluster 2024-02-12T16:32:38.077090+0000 mgr.y (mgr.14146) 1440 : cluster [DBG] pgmap v2031: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 11262/112408 objects misplaced (10.019%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:39.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:39.172 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:39.199 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:39.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:39.304 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:39.334 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:39.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:39.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:39.466 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:39.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:39.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:39.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:39.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:39.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:39.756 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:39.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:39.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:39.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:39.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:40.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:40.036 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:40.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:40.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:40.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:40.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:40.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:40.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:40.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:40.607 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:40.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:40.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:40.739 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:40.819 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:32:40.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:40.853 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:32:40.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:40.898 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:32:40.959 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:32:40.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:40.997 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:40.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:41.001 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:32:41.034 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:32:41.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:41.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:41 smithi169 bash[19604]: cluster 2024-02-12T16:32:40.077631+0000 mgr.y (mgr.14146) 1441 : cluster [DBG] pgmap v2032: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 10920/109372 objects misplaced (9.984%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:41.305 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:41 smithi169 bash[19604]: audit 2024-02-12T16:32:40.774472+0000 mon.c (mon.1) 60 : audit [DBG] from='client.? v1:172.21.15.121:0/412705480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:41.306 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:41 smithi169 bash[19604]: audit 2024-02-12T16:32:41.125665+0000 mon.a (mon.0) 2442 : audit [INF] from='client.26450 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:32:41.306 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:41 smithi169 bash[19604]: audit 2024-02-12T16:32:41.126928+0000 mon.b (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.121:0/1785385929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:32:41.306 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.308 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.308 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.308 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.308 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.308 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:32:41.338 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:41.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:41.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:41 smithi123 bash[19119]: cluster 2024-02-12T16:32:40.077631+0000 mgr.y (mgr.14146) 1441 : cluster [DBG] pgmap v2032: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 10920/109372 objects misplaced (9.984%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:41.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:41 smithi123 bash[19119]: audit 2024-02-12T16:32:40.774472+0000 mon.c (mon.1) 60 : audit [DBG] from='client.? v1:172.21.15.121:0/412705480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:41.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:41 smithi123 bash[19119]: audit 2024-02-12T16:32:41.125665+0000 mon.a (mon.0) 2442 : audit [INF] from='client.26450 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:32:41.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:41 smithi123 bash[19119]: audit 2024-02-12T16:32:41.126928+0000 mon.b (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.121:0/1785385929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:32:41.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:41 smithi121 bash[18489]: cluster 2024-02-12T16:32:40.077631+0000 mgr.y (mgr.14146) 1441 : cluster [DBG] pgmap v2032: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 10920/109372 objects misplaced (9.984%); 1.2 MiB/s, 19 objects/s recovering 2024-02-12T16:32:41.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:41 smithi121 bash[18489]: audit 2024-02-12T16:32:40.774472+0000 mon.c (mon.1) 60 : audit [DBG] from='client.? v1:172.21.15.121:0/412705480' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:41 smithi121 bash[18489]: audit 2024-02-12T16:32:41.125665+0000 mon.a (mon.0) 2442 : audit [INF] from='client.26450 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:32:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:41 smithi121 bash[18489]: audit 2024-02-12T16:32:41.126928+0000 mon.b (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.121:0/1785385929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:32:41.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:41.470 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:41.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:41.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:41.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:41.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:41.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:41.748 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:41.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:41.768 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:32:41.773 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T16:32:41.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:41.910 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:41.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:42.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:42.033 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:32:42.033 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:32:42.034 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2024-02-12T16:32:42.034 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2024-02-12T16:32:42.034 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:32:42.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:42.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:42.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:42 smithi121 bash[18489]: audit 2024-02-12T16:32:41.143346+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? v1:172.21.15.121:0/3507888081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:32:42.142 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:42 smithi121 bash[18489]: cluster 2024-02-12T16:32:41.753455+0000 mon.a (mon.0) 2444 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:42.142 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:42 smithi121 bash[18489]: audit 2024-02-12T16:32:41.758876+0000 mon.a (mon.0) 2445 : audit [INF] from='client.26450 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:32:42.142 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:42 smithi121 bash[18489]: audit 2024-02-12T16:32:41.758942+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? v1:172.21.15.121:0/3507888081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:32:42.142 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:42 smithi121 bash[18489]: cluster 2024-02-12T16:32:41.758992+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e712: 12 total, 12 up, 12 in 2024-02-12T16:32:42.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:42.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:42.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:42.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:42.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:42.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:42 smithi123 bash[19119]: audit 2024-02-12T16:32:41.143346+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? v1:172.21.15.121:0/3507888081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:32:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:42 smithi123 bash[19119]: cluster 2024-02-12T16:32:41.753455+0000 mon.a (mon.0) 2444 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:42 smithi123 bash[19119]: audit 2024-02-12T16:32:41.758876+0000 mon.a (mon.0) 2445 : audit [INF] from='client.26450 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:32:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:42 smithi123 bash[19119]: audit 2024-02-12T16:32:41.758942+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? v1:172.21.15.121:0/3507888081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:32:42.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:42 smithi123 bash[19119]: cluster 2024-02-12T16:32:41.758992+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e712: 12 total, 12 up, 12 in 2024-02-12T16:32:42.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:42.456 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:42.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:42 smithi169 bash[19604]: audit 2024-02-12T16:32:41.143346+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? v1:172.21.15.121:0/3507888081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:32:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:42 smithi169 bash[19604]: cluster 2024-02-12T16:32:41.753455+0000 mon.a (mon.0) 2444 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:32:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:42 smithi169 bash[19604]: audit 2024-02-12T16:32:41.758876+0000 mon.a (mon.0) 2445 : audit [INF] from='client.26450 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:32:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:42 smithi169 bash[19604]: audit 2024-02-12T16:32:41.758942+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? v1:172.21.15.121:0/3507888081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:32:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:42 smithi169 bash[19604]: cluster 2024-02-12T16:32:41.758992+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e712: 12 total, 12 up, 12 in 2024-02-12T16:32:42.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:42.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:42.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:42.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:42.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:42.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:42.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:42.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:42.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:42.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:32:43.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:43.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T16:32:43.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:43.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:43.249 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:43 smithi121 bash[18489]: cluster 2024-02-12T16:32:42.078100+0000 mgr.y (mgr.14146) 1442 : cluster [DBG] pgmap v2034: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 10110/103584 objects misplaced (9.760%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:43.279 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:43.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:43 smithi123 bash[19119]: cluster 2024-02-12T16:32:42.078100+0000 mgr.y (mgr.14146) 1442 : cluster [DBG] pgmap v2034: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 10110/103584 objects misplaced (9.760%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:43.392 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:43 smithi169 bash[19604]: cluster 2024-02-12T16:32:42.078100+0000 mgr.y (mgr.14146) 1442 : cluster [DBG] pgmap v2034: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 10110/103584 objects misplaced (9.760%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:32:43.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:43.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:43.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:43.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:43.662 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:43.707 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:43.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:43.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:43.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:43.981 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:43.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:44.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:44.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:44.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:44.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:32:44.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:44.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:44 smithi169 bash[19604]: cluster 2024-02-12T16:32:42.346375+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.d scrub starts 2024-02-12T16:32:44.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:44 smithi169 bash[19604]: cluster 2024-02-12T16:32:42.442647+0000 osd.7 (osd.7) 174 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:44.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:44 smithi169 bash[19604]: cluster 2024-02-12T16:32:42.442688+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:44.230 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:44 smithi169 bash[19604]: cluster 2024-02-12T16:32:43.146847+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in 2024-02-12T16:32:44.247 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:44.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:44.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:44.377 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:44.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:44 smithi121 bash[18489]: cluster 2024-02-12T16:32:42.346375+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.d scrub starts 2024-02-12T16:32:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:44 smithi121 bash[18489]: cluster 2024-02-12T16:32:42.442647+0000 osd.7 (osd.7) 174 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:44 smithi121 bash[18489]: cluster 2024-02-12T16:32:42.442688+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:44 smithi121 bash[18489]: cluster 2024-02-12T16:32:43.146847+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in 2024-02-12T16:32:44.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:44.490 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:44 smithi123 bash[19119]: cluster 2024-02-12T16:32:42.346375+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.d scrub starts 2024-02-12T16:32:44.490 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:44 smithi123 bash[19119]: cluster 2024-02-12T16:32:42.442647+0000 osd.7 (osd.7) 174 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:44.490 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:44 smithi123 bash[19119]: cluster 2024-02-12T16:32:42.442688+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:44.490 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:44 smithi123 bash[19119]: cluster 2024-02-12T16:32:43.146847+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e713: 12 total, 12 up, 12 in 2024-02-12T16:32:44.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:44.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:44.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:44.686 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:44.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:44.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:44.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:44.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:44.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:44.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:44.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:45.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:45.093 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:45.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:45.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:32:45.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:45.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:45.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:45.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:45.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:45 smithi123 bash[19119]: cluster 2024-02-12T16:32:43.465076+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.d scrub ok 2024-02-12T16:32:45.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:45 smithi123 bash[19119]: cluster 2024-02-12T16:32:43.763885+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.c scrub starts 2024-02-12T16:32:45.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:45 smithi123 bash[19119]: cluster 2024-02-12T16:32:44.078587+0000 mgr.y (mgr.14146) 1443 : cluster [DBG] pgmap v2036: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 9776/99202 objects misplaced (9.855%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:45.400 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:45.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:45.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:45 smithi121 bash[18489]: cluster 2024-02-12T16:32:43.465076+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.d scrub ok 2024-02-12T16:32:45.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:45 smithi121 bash[18489]: cluster 2024-02-12T16:32:43.763885+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.c scrub starts 2024-02-12T16:32:45.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:45 smithi121 bash[18489]: cluster 2024-02-12T16:32:44.078587+0000 mgr.y (mgr.14146) 1443 : cluster [DBG] pgmap v2036: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 9776/99202 objects misplaced (9.855%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:45.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:45.542 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:45.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:45 smithi169 bash[19604]: cluster 2024-02-12T16:32:43.465076+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.d scrub ok 2024-02-12T16:32:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:45 smithi169 bash[19604]: cluster 2024-02-12T16:32:43.763885+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.c scrub starts 2024-02-12T16:32:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:45 smithi169 bash[19604]: cluster 2024-02-12T16:32:44.078587+0000 mgr.y (mgr.14146) 1443 : cluster [DBG] pgmap v2036: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 9776/99202 objects misplaced (9.855%); 1.7 MiB/s, 27 objects/s recovering 2024-02-12T16:32:45.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:45.669 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:45.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:45.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:45.816 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:45.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:45.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:45.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:45.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:46.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:46.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:46.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:46.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:46.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:46.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:46.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:46.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:46.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:46 smithi121 bash[18489]: cluster 2024-02-12T16:32:45.014142+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub ok 2024-02-12T16:32:46.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:46.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:46.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:46.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:46 smithi169 bash[19604]: cluster 2024-02-12T16:32:45.014142+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub ok 2024-02-12T16:32:46.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:46 smithi123 bash[19119]: cluster 2024-02-12T16:32:45.014142+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub ok 2024-02-12T16:32:46.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:46.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:46.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:46.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:46.800 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:46.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:46.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:46.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:46.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:47.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:47.066 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:47.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:47 smithi123 bash[19119]: cluster 2024-02-12T16:32:45.491943+0000 osd.4 (osd.4) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:47 smithi123 bash[19119]: cluster 2024-02-12T16:32:45.491994+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:47 smithi123 bash[19119]: cluster 2024-02-12T16:32:45.745695+0000 osd.3 (osd.3) 50 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:47 smithi123 bash[19119]: cluster 2024-02-12T16:32:45.816882+0000 osd.3 (osd.3) 51 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:47 smithi123 bash[19119]: cluster 2024-02-12T16:32:46.079023+0000 mgr.y (mgr.14146) 1444 : cluster [DBG] pgmap v2037: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 9074/94852 objects misplaced (9.566%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:47 smithi123 bash[19119]: cluster 2024-02-12T16:32:46.162735+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e714: 12 total, 12 up, 12 in 2024-02-12T16:32:47.166 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:47 smithi123 bash[19119]: audit 2024-02-12T16:32:47.094724+0000 mon.a (mon.0) 2450 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:32:47.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:47.195 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:47.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:47 smithi169 bash[19604]: cluster 2024-02-12T16:32:45.491943+0000 osd.4 (osd.4) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:47 smithi169 bash[19604]: cluster 2024-02-12T16:32:45.491994+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:47 smithi169 bash[19604]: cluster 2024-02-12T16:32:45.745695+0000 osd.3 (osd.3) 50 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:32:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:47 smithi169 bash[19604]: cluster 2024-02-12T16:32:45.816882+0000 osd.3 (osd.3) 51 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:32:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:47 smithi169 bash[19604]: cluster 2024-02-12T16:32:46.079023+0000 mgr.y (mgr.14146) 1444 : cluster [DBG] pgmap v2037: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 9074/94852 objects misplaced (9.566%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:47.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:47 smithi169 bash[19604]: cluster 2024-02-12T16:32:46.162735+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e714: 12 total, 12 up, 12 in 2024-02-12T16:32:47.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:47 smithi169 bash[19604]: audit 2024-02-12T16:32:47.094724+0000 mon.a (mon.0) 2450 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:32:47.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:47.349 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:47.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:47.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:47.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:47.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:47.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:47.651 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:47.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:47.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:47 smithi121 bash[18489]: cluster 2024-02-12T16:32:45.491943+0000 osd.4 (osd.4) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:47 smithi121 bash[18489]: cluster 2024-02-12T16:32:45.491994+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:47 smithi121 bash[18489]: cluster 2024-02-12T16:32:45.745695+0000 osd.3 (osd.3) 50 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:32:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:47 smithi121 bash[18489]: cluster 2024-02-12T16:32:45.816882+0000 osd.3 (osd.3) 51 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:32:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:47 smithi121 bash[18489]: cluster 2024-02-12T16:32:46.079023+0000 mgr.y (mgr.14146) 1444 : cluster [DBG] pgmap v2037: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 9074/94852 objects misplaced (9.566%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:47 smithi121 bash[18489]: cluster 2024-02-12T16:32:46.162735+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e714: 12 total, 12 up, 12 in 2024-02-12T16:32:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:47 smithi121 bash[18489]: audit 2024-02-12T16:32:47.094724+0000 mon.a (mon.0) 2450 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:32:47.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:47.805 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:47.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:47.840 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:32:47.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:47.935 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:47.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:47.945 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:48.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:48.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:48.178 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:48.208 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:48.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:48.312 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:48.345 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:48.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:48.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:48.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:48.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:48 smithi169 bash[19604]: cluster 2024-02-12T16:32:47.369495+0000 osd.5 (osd.5) 137 : cluster [DBG] 6.a scrub starts 2024-02-12T16:32:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:48 smithi169 bash[19604]: audit 2024-02-12T16:32:47.393356+0000 mon.a (mon.0) 2451 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:32:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:48 smithi169 bash[19604]: audit 2024-02-12T16:32:47.719664+0000 mon.a (mon.0) 2452 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:32:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:48 smithi169 bash[19604]: audit 2024-02-12T16:32:47.720480+0000 mon.a (mon.0) 2453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:32:48.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:48 smithi169 bash[19604]: audit 2024-02-12T16:32:47.730541+0000 mon.a (mon.0) 2454 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:32:48.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:48 smithi169 bash[19604]: cluster 2024-02-12T16:32:48.198879+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.a scrub ok 2024-02-12T16:32:48.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:48.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:48.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:48.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:48 smithi123 bash[19119]: cluster 2024-02-12T16:32:47.369495+0000 osd.5 (osd.5) 137 : cluster [DBG] 6.a scrub starts 2024-02-12T16:32:48.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:48 smithi123 bash[19119]: audit 2024-02-12T16:32:47.393356+0000 mon.a (mon.0) 2451 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:32:48.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:48 smithi123 bash[19119]: audit 2024-02-12T16:32:47.719664+0000 mon.a (mon.0) 2452 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:32:48.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:48 smithi123 bash[19119]: audit 2024-02-12T16:32:47.720480+0000 mon.a (mon.0) 2453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:32:48.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:48 smithi123 bash[19119]: audit 2024-02-12T16:32:47.730541+0000 mon.a (mon.0) 2454 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:32:48.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:48 smithi123 bash[19119]: cluster 2024-02-12T16:32:48.198879+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.a scrub ok 2024-02-12T16:32:48.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:48.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:48.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:48.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:48.873 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:48 smithi121 bash[18489]: cluster 2024-02-12T16:32:47.369495+0000 osd.5 (osd.5) 137 : cluster [DBG] 6.a scrub starts 2024-02-12T16:32:48.874 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:48 smithi121 bash[18489]: audit 2024-02-12T16:32:47.393356+0000 mon.a (mon.0) 2451 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:32:48.874 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:48 smithi121 bash[18489]: audit 2024-02-12T16:32:47.719664+0000 mon.a (mon.0) 2452 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:32:48.874 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:48 smithi121 bash[18489]: audit 2024-02-12T16:32:47.720480+0000 mon.a (mon.0) 2453 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:32:48.874 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:48 smithi121 bash[18489]: audit 2024-02-12T16:32:47.730541+0000 mon.a (mon.0) 2454 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:32:48.874 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:48 smithi121 bash[18489]: cluster 2024-02-12T16:32:48.198879+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.a scrub ok 2024-02-12T16:32:48.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:48.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:48.968 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:48.968 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:32:49.038 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:49.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:49.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:49.210 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:49.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:49.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:49.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:49.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:49.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:49.499 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:49.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:49 smithi169 bash[19604]: cluster 2024-02-12T16:32:48.079474+0000 mgr.y (mgr.14146) 1445 : cluster [DBG] pgmap v2039: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 8883/89862 objects misplaced (9.885%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:32:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:49 smithi169 bash[19604]: cluster 2024-02-12T16:32:48.495330+0000 osd.6 (osd.6) 136 : cluster [DBG] 1.f scrub starts 2024-02-12T16:32:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:49 smithi169 bash[19604]: cluster 2024-02-12T16:32:48.560729+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.f scrub ok 2024-02-12T16:32:49.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:49.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:49.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:49.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:49 smithi121 bash[18489]: cluster 2024-02-12T16:32:48.079474+0000 mgr.y (mgr.14146) 1445 : cluster [DBG] pgmap v2039: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 8883/89862 objects misplaced (9.885%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:32:49.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:49 smithi121 bash[18489]: cluster 2024-02-12T16:32:48.495330+0000 osd.6 (osd.6) 136 : cluster [DBG] 1.f scrub starts 2024-02-12T16:32:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:49 smithi121 bash[18489]: cluster 2024-02-12T16:32:48.560729+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.f scrub ok 2024-02-12T16:32:49.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:49.772 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:49.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:49.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:49 smithi123 bash[19119]: cluster 2024-02-12T16:32:48.079474+0000 mgr.y (mgr.14146) 1445 : cluster [DBG] pgmap v2039: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 8883/89862 objects misplaced (9.885%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:32:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:49 smithi123 bash[19119]: cluster 2024-02-12T16:32:48.495330+0000 osd.6 (osd.6) 136 : cluster [DBG] 1.f scrub starts 2024-02-12T16:32:49.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:49 smithi123 bash[19119]: cluster 2024-02-12T16:32:48.560729+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.f scrub ok 2024-02-12T16:32:49.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:49.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:50.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:50.046 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:50.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:50.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:50.180 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:50.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:50.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:50.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:50.307 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:50.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:50.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:50.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:50 smithi121 bash[18489]: cluster 2024-02-12T16:32:49.506117+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:32:50.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:50 smithi121 bash[18489]: cluster 2024-02-12T16:32:49.602301+0000 osd.6 (osd.6) 139 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:32:50.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:50 smithi121 bash[18489]: cluster 2024-02-12T16:32:49.763445+0000 osd.9 (osd.9) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:50.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:50 smithi121 bash[18489]: cluster 2024-02-12T16:32:49.763495+0000 osd.9 (osd.9) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:50.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:50.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:50.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:50 smithi169 bash[19604]: cluster 2024-02-12T16:32:49.506117+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:32:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:50 smithi169 bash[19604]: cluster 2024-02-12T16:32:49.602301+0000 osd.6 (osd.6) 139 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:32:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:50 smithi169 bash[19604]: cluster 2024-02-12T16:32:49.763445+0000 osd.9 (osd.9) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:50 smithi169 bash[19604]: cluster 2024-02-12T16:32:49.763495+0000 osd.9 (osd.9) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:50.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:50.593 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:50.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:50.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:50.705 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:50 smithi123 bash[19119]: cluster 2024-02-12T16:32:49.506117+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:32:50.706 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:50 smithi123 bash[19119]: cluster 2024-02-12T16:32:49.602301+0000 osd.6 (osd.6) 139 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:32:50.706 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:50 smithi123 bash[19119]: cluster 2024-02-12T16:32:49.763445+0000 osd.9 (osd.9) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:32:50.706 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:50 smithi123 bash[19119]: cluster 2024-02-12T16:32:49.763495+0000 osd.9 (osd.9) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:32:50.731 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:50.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:50.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:50.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:50.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:50.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:51.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:51.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:51.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:51.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:51.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:51.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:51.257 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:51.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:51.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:51.394 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:51.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:51.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:51 smithi169 bash[19604]: cluster 2024-02-12T16:32:50.079935+0000 mgr.y (mgr.14146) 1446 : cluster [DBG] pgmap v2040: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 8537/86878 objects misplaced (9.826%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:51 smithi169 bash[19604]: cluster 2024-02-12T16:32:50.403201+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e715: 12 total, 12 up, 12 in 2024-02-12T16:32:51.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:51.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:51 smithi123 bash[19119]: cluster 2024-02-12T16:32:50.079935+0000 mgr.y (mgr.14146) 1446 : cluster [DBG] pgmap v2040: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 8537/86878 objects misplaced (9.826%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:51.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:51 smithi123 bash[19119]: cluster 2024-02-12T16:32:50.403201+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e715: 12 total, 12 up, 12 in 2024-02-12T16:32:51.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:51 smithi121 bash[18489]: cluster 2024-02-12T16:32:50.079935+0000 mgr.y (mgr.14146) 1446 : cluster [DBG] pgmap v2040: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 8537/86878 objects misplaced (9.826%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:51.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:51 smithi121 bash[18489]: cluster 2024-02-12T16:32:50.403201+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e715: 12 total, 12 up, 12 in 2024-02-12T16:32:51.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:51.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:51.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:51.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:51.884 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:51.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:51.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:52.027 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:52.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:52.159 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:52.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:52.285 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:52.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:52.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:52.539 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:52.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:52.667 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:52.771 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:52.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:52.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:52.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:52.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:53.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:53.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:53.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:265698 5.1 5:897ec097:::benchmark_data_smithi169_84712_object96308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.876792+0000", 2024-02-12T16:32:53.111 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 326.22380816100002, 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37771153099999999, 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265698 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876792+0000", 2024-02-12T16:32:53.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876792+0000", 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876789+0000", 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3130000000000004e-06 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876798+0000", 2024-02-12T16:32:53.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3217000000000001e-05 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876812+0000", 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0190000000000001e-06 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876819+0000", 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.897e-05 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876867+0000", 2024-02-12T16:32:53.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010273200000000001 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.876970+0000", 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116316 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.877087+0000", 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042691070000000003 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881356+0000", 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025114400000000002 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881607+0000", 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37278415100000001 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.254391+0000", 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011236 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.254503+0000", 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37771153099999999 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.16095.0:265693 5.0 e604/581 5:069ffb0a:::benchmark_data_smithi169_84712_object96303:head v 604'16565, mlcod=604'16562)", 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.880287+0000", 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 326.22031288599999, 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37310036200000002, 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880287+0000", 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880287+0000", 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880285+0000", 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4469999999999995e-06 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880295+0000", 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7669999999999998e-06 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880305+0000", 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6640000000000003e-06 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880310+0000", 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9721999999999997e-05 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880350+0000", 2024-02-12T16:32:53.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6410999999999998e-05 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.880376+0000", 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.372864537 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253241+0000", 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146487 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253387+0000", 2024-02-12T16:32:53.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37310036200000002 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.16095.0:265698 5.1 e604/547 ondisk, result = 0)", 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.881538+0000", 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 326.21906160999998, 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37298944299999998, 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881538+0000", 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881538+0000", 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881537+0000", 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8499999999999994e-06 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881545+0000", 2024-02-12T16:32:53.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0390000000000005e-06 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881553+0000", 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7039999999999999e-06 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881559+0000", 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4715000000000002e-05 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881593+0000", 2024-02-12T16:32:53.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.849e-06 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.881600+0000", 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37292752299999998 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.254528+0000", 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37298944299999998 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.16095.0:265705 5.0 e604/581 5:02a78a1f:::benchmark_data_smithi169_84712_object96315:head v 604'16566, mlcod=604'16562)", 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.884187+0000", 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 326.21641306999999, 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.38640768800000003, 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.884187+0000", 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.884187+0000", 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.884185+0000", 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0820000000000005e-06 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.884195+0000", 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36918208000000002 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253377+0000", 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5650000000000001e-05 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253392+0000", 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100201 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253492+0000", 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8338999999999997e-05 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253561+0000", 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016984035000000001 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.270545+0000", 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9727e-05 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.270595+0000", 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.38640768800000003 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:265736 5.1 5:8416ea08:::benchmark_data_smithi169_84712_object96346:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.909386+0000", 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 326.19121384800002, 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36358961000000001, 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265736 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.909386+0000", 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.909386+0000", 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.909384+0000", 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1470000000000007e-06 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.909393+0000", 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34383328800000001 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253226+0000", 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9181999999999998e-05 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253275+0000", 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013271330000000001 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.254602+0000", 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029244600000000003 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.254895+0000", 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000349112 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.255244+0000", 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01762203 2024-02-12T16:32:53.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.272866+0000", 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6800000000000005e-06 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.272876+0000", 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4055999999999999e-05 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.272930+0000", 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6072999999999997e-05 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.272976+0000", 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36358961000000001 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:16642 6.e 6:7f6c7094:::benchmark_data_smithi169_96224_object16641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:29:31.831135+0000", 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 201.26946446900001, 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15400594000000001, 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.134 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 16642 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831135+0000", 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831135+0000", 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831133+0000", 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5242999999999999e-05 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831188+0000", 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4993e-05 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831203+0000", 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9530000000000002e-06 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831210+0000", 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2713000000000002e-05 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831263+0000", 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012741299999999999 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831391+0000", 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015580100000000001 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.831546+0000", 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023695080000000002 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.833916+0000", 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.151003216 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.984919+0000", 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120963 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.985040+0000", 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101383 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.985141+0000", 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15400594000000001 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:16648 6.e 6:7b0e4e3f:::benchmark_data_smithi169_96224_object16647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:29:31.832048+0000", 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 201.26855219000001, 2024-02-12T16:32:53.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15328459999999999, 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 16648 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.832048+0000", 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.832048+0000", 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.832046+0000", 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3057000000000003e-05 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.832109+0000", 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3799e-05 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.832123+0000", 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.542e-06 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.832130+0000", 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089730699999999997 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.833028+0000", 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013263700000000001 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.833160+0000", 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016901199999999999 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.833329+0000", 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00080356899999999996 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.834133+0000", 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.151114204 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.985247+0000", 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7265999999999998e-05 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.985294+0000", 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8155000000000003e-05 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:31.985332+0000", 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15328459999999999 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.144 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:47973 6.2 6:49aa1340:::benchmark_data_smithi169_96224_object47972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e648)", 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:29:41.779461+0000", 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 191.321138762, 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.160203389, 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 47973 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779461+0000", 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779461+0000", 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779458+0000", 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7541999999999999e-05 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779515+0000", 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9086e-05 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779534+0000", 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5259999999999993e-06 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779543+0000", 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1450999999999999e-05 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779594+0000", 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105305 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779700+0000", 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5898999999999998e-05 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.779796+0000", 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015628459999999999 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.781358+0000", 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.158159933 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.939518+0000", 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9061000000000007e-05 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.939607+0000", 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7136000000000003e-05 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.939665+0000", 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.160203389 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:47980 6.2 6:4e6653a8:::benchmark_data_smithi169_96224_object47979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e648)", 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:29:41.783521+0000", 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 191.31707877100001, 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15661328699999999, 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.150 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 47980 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783521+0000", 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783521+0000", 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783520+0000", 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6595999999999996e-05 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783596+0000", 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3713999999999999e-05 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783610+0000", 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1850000000000001e-06 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783616+0000", 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6362000000000001e-05 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783663+0000", 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000130018 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783793+0000", 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000173933 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.783967+0000", 2024-02-12T16:32:53.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001772601 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.785739+0000", 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.154072657 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.939812+0000", 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000193313 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.940005+0000", 2024-02-12T16:32:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012923000000000001 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:29:41.940134+0000", 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15661328699999999 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101283 6.2 6:4b93b79d:::benchmark_data_smithi169_96224_object101282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.471525+0000", 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 163.629074586, 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20611299199999999, 2024-02-12T16:32:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101283 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471525+0000", 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471525+0000", 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999943 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471520+0000", 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4875999999999993e-05 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471585+0000", 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3581999999999998e-05 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471618+0000", 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8272999999999999e-05 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471636+0000", 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0072000000000003e-05 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471707+0000", 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2405999999999996e-05 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471799+0000", 2024-02-12T16:32:53.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010306699999999999 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471902+0000", 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032254499999999999 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475127+0000", 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20228344200000001 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677411+0000", 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017230699999999999 2024-02-12T16:32:53.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677583+0000", 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5136999999999997e-05 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677638+0000", 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20611299199999999 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26257.0:101283 6.2 e656/636 ondisk, result = 0)", 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.475050+0000", 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 163.62554943699999, 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19416351400000001, 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475050+0000", 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475050+0000", 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475049+0000", 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9430000000000006e-06 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475058+0000", 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5759999999999995e-06 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475067+0000", 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3329999999999996e-06 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475073+0000", 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1579999999999998e-05 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475114+0000", 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6719999999999998e-06 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475121+0000", 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19409317500000001 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669214+0000", 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19416351400000001 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101294 6.2 6:40d2734e:::benchmark_data_smithi169_96224_object101293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:32:53.164 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.487178+0000", 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 163.61342192699999, 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19086616300000001, 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101294 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.487178+0000", 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.487178+0000", 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.487175+0000", 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4419999999999998e-05 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.487219+0000", 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.18169859099999999 2024-02-12T16:32:53.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.668918+0000", 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4709000000000003e-05 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.668952+0000", 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034329199999999998 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669296+0000", 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000326103 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669622+0000", 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00044371999999999999 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.670065+0000", 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00783616 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677902+0000", 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4405e-05 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677936+0000", 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7668000000000003e-05 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.678004+0000", 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0544999999999999e-05 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.678044+0000", 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19086616300000001 2024-02-12T16:32:53.169 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:179498 6.2 6.b3092d82 (undecoded) ondisk+retry+write+known_if_redirected e680)", 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.755414+0000", 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.345186040000002, 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22293095900000001, 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 179498 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.170 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755414+0000", 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755414+0000", 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755412+0000", 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0990000000000001e-06 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755419+0000", 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0995999999999999e-05 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755430+0000", 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2589999999999997e-06 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755436+0000", 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2981000000000001e-05 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755479+0000", 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755485+0000", 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22271555600000001 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978201+0000", 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144134 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978345+0000", 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22293095900000001 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:179509 6.12 6:49413315:::benchmark_data_smithi169_96224_object26673:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.759446+0000", 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.341154133000003, 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0895270889999999, 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.174 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 179509 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759446+0000", 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759446+0000", 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759444+0000", 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5679999999999996e-06 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759450+0000", 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7540000000000001e-06 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759460+0000", 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.801e-06 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759466+0000", 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21890232700000001 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978368+0000", 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3490000000000003e-06 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978376+0000", 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.85934620900000003 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.837722+0000", 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5047000000000002e-05 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.837797+0000", 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4651e-05 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.837882+0000", 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042455849999999996 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.842127+0000", 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067655409999999999 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.848893+0000", 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5303999999999999e-05 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.848938+0000", 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4851000000000002e-05 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.848973+0000", 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0895270889999999 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:179516 6.2 6.caf0a7a2 (undecoded) ondisk+write+known_if_redirected e680)", 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.820275+0000", 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.280324634999999, 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15801873599999999, 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:32:53.180 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 179516 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.820275+0000", 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.820275+0000", 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999943 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.820270+0000", 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9483999999999998e-05 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.820289+0000", 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9209000000000001e-05 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.820318+0000", 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3621999999999999e-05 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.820332+0000", 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15795226000000001 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978284+0000", 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.713e-06 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978294+0000", 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15801873599999999 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.183 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:263648 6.2 6:448cfbca:::benchmark_data_smithi169_96224_object110812:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:48.072263+0000", 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0283364480000001, 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49358639599999998, 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 263648 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072263+0000", 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072263+0000", 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072260+0000", 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1686e-05 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072272+0000", 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1992e-05 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072284+0000", 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4900000000000006e-06 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072294+0000", 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021438499999999999 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072508+0000", 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105101 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072613+0000", 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058748800000000005 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.073201+0000", 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007237683 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.080438+0000", 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.485344098 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.565782+0000", 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270000000000002e-05 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.565822+0000", 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8271000000000001e-05 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.565850+0000", 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49358639599999998 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:263673 6.2 6:443ac971:::benchmark_data_smithi169_96224_object110837:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:48.103119+0000", 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9974808490000004, 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47670784100000002, 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.189 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 263673 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103119+0000", 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103119+0000", 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103117+0000", 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5930000000000002e-06 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103124+0000", 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2967e-05 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103137+0000", 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1629999999999999e-06 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103144+0000", 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3171000000000003e-05 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103197+0000", 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013269399999999999 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103330+0000", 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00067369999999999995 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.104003+0000", 2024-02-12T16:32:53.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002587134 2024-02-12T16:32:53.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:53.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.106590+0000", 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.473150768 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.579741+0000", 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8149000000000003e-05 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.579789+0000", 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7551e-05 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.579827+0000", 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47670784100000002 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264221 6.2 6:446377cf:::benchmark_data_smithi169_96224_object111385:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.076018+0000", 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.024582165, 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35090154800000001, 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.196 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264221 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076018+0000", 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076018+0000", 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076016+0000", 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2480000000000001e-06 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076022+0000", 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3334e-05 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076036+0000", 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2519999999999996e-06 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076042+0000", 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2206999999999997e-05 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076094+0000", 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125442 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076220+0000", 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068375100000000002 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076903+0000", 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068602649999999999 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.083764+0000", 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34298718500000003 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.426751+0000", 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1503999999999998e-05 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.426822+0000", 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6923999999999996e-05 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.426919+0000", 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35090154800000001 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264223 6.2 6:4772561a:::benchmark_data_smithi169_96224_object111387:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.076658+0000", 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.0239422429999996, 2024-02-12T16:32:53.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35736425399999999, 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264223 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076658+0000", 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076658+0000", 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076656+0000", 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3090000000000001e-06 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076662+0000", 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2891000000000001e-05 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076675+0000", 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4729999999999999e-06 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076681+0000", 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028843200000000001 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.079565+0000", 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000241368 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.079806+0000", 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056161700000000004 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080368+0000", 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076541070000000003 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.088022+0000", 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.345911091 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.433933+0000", 2024-02-12T16:32:53.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9709000000000001e-05 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.433983+0000", 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8949000000000003e-05 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434022+0000", 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35736425399999999 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264228 6.2 6:43e9cd2d:::benchmark_data_smithi169_96224_object111392:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.080709+0000", 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.0198908759999998, 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35371650599999999, 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264228 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:32:53.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080709+0000", 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080709+0000", 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080707+0000", 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7599999999999997e-06 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:32:53.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080714+0000", 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0532e-05 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080724+0000", 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1240000000000002e-06 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080731+0000", 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003134768 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.083865+0000", 2024-02-12T16:32:53.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107218 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.083973+0000", 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000601188 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.084574+0000", 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003583737 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.088157+0000", 2024-02-12T16:32:53.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34615111500000001 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434309+0000", 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0162999999999998e-05 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434389+0000", 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6810999999999997e-05 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434426+0000", 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35371650599999999 2024-02-12T16:32:53.211 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.212 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.212 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.212 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:32:53.212 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:32:53.212 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:32:53.219 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:53.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:53.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:53.347 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:53.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:53.452 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:53.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:53 smithi123 bash[19119]: cluster 2024-02-12T16:32:52.080477+0000 mgr.y (mgr.14146) 1447 : cluster [DBG] pgmap v2042: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 7816/81690 objects misplaced (9.568%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:53.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:53 smithi123 bash[19119]: cluster 2024-02-12T16:32:53.240177+0000 osd.5 (osd.5) 139 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:32:53.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:53.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:53.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:53 smithi169 bash[19604]: cluster 2024-02-12T16:32:52.080477+0000 mgr.y (mgr.14146) 1447 : cluster [DBG] pgmap v2042: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 7816/81690 objects misplaced (9.568%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:53 smithi169 bash[19604]: cluster 2024-02-12T16:32:53.240177+0000 osd.5 (osd.5) 139 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:32:53.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:53.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:53 smithi121 bash[18489]: cluster 2024-02-12T16:32:52.080477+0000 mgr.y (mgr.14146) 1447 : cluster [DBG] pgmap v2042: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 7816/81690 objects misplaced (9.568%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:53.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:53 smithi121 bash[18489]: cluster 2024-02-12T16:32:53.240177+0000 osd.5 (osd.5) 139 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:32:53.660 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:53.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:53.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:53.802 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:53.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:53.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:53.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:53.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:53.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:32:54.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:54.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:54.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:54.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:54.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:54.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:54.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:54.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:54.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:54.502 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:54.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:54.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:54.631 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:54.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:54.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:54 smithi123 bash[19119]: cluster 2024-02-12T16:32:53.336716+0000 osd.5 (osd.5) 140 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:32:54.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:54.770 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:54.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:54 smithi169 bash[19604]: cluster 2024-02-12T16:32:53.336716+0000 osd.5 (osd.5) 140 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:32:54.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:54.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:54 smithi121 bash[18489]: cluster 2024-02-12T16:32:53.336716+0000 osd.5 (osd.5) 140 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:32:54.901 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:54.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:55.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:55.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:55.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:55.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:55.169 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:55.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:55.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:55.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:55.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:55.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:55.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:55.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:55.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:55.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:32:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:55.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:32:55.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:55 smithi121 bash[18489]: cluster 2024-02-12T16:32:54.081026+0000 mgr.y (mgr.14146) 1448 : cluster [DBG] pgmap v2043: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 7517/77904 objects misplaced (9.649%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:32:55.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:55.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:55 smithi169 bash[19604]: cluster 2024-02-12T16:32:54.081026+0000 mgr.y (mgr.14146) 1448 : cluster [DBG] pgmap v2043: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 7517/77904 objects misplaced (9.649%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:32:55.713 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:55.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:55.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:55.824 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:55 smithi123 bash[19119]: cluster 2024-02-12T16:32:54.081026+0000 mgr.y (mgr.14146) 1448 : cluster [DBG] pgmap v2043: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 7517/77904 objects misplaced (9.649%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:32:55.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:55.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:55.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:56.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:56.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:56.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:56.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:32:56.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:32:56.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:56.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:56.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:56.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:32:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:32:56.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:56.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:56.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:56.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:56.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:56 smithi123 bash[19119]: cluster 2024-02-12T16:32:56.081521+0000 mgr.y (mgr.14146) 1449 : cluster [DBG] pgmap v2044: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 6805/73752 objects misplaced (9.227%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:56.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:56.943 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:56 smithi169 bash[19604]: cluster 2024-02-12T16:32:56.081521+0000 mgr.y (mgr.14146) 1449 : cluster [DBG] pgmap v2044: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 6805/73752 objects misplaced (9.227%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:56.960 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:56.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:57.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:57.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:57.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:56 smithi121 bash[18489]: cluster 2024-02-12T16:32:56.081521+0000 mgr.y (mgr.14146) 1449 : cluster [DBG] pgmap v2044: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 6805/73752 objects misplaced (9.227%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:32:57.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:57.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:57.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:57.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:57.464 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:57.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:57.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:57.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:32:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:57.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:32:57.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:57.787 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T16:32:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:57.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:57.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:57.944 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:57.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:58.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:58.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:57 smithi169 bash[19604]: audit 2024-02-12T16:32:57.551217+0000 mon.a (mon.0) 2456 : audit [INF] from='client.? v1:172.21.15.121:0/3804756559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:32:58.053 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:57 smithi121 bash[18489]: audit 2024-02-12T16:32:57.551217+0000 mon.a (mon.0) 2456 : audit [INF] from='client.? v1:172.21.15.121:0/3804756559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:32:58.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:58.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:58.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:57 smithi123 bash[19119]: audit 2024-02-12T16:32:57.551217+0000 mon.a (mon.0) 2456 : audit [INF] from='client.? v1:172.21.15.121:0/3804756559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:32:58.170 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:32:58.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:58.223 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:32:58.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:58.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:58.265 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:32:58.307 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:32:58.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:58.340 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:32:58.366 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:58.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:32:58.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:32:58.409 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:32:58.446 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:32:58.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:58.475 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:32:58.491 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:58.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:58.503 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:32:58.541 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:32:58.589 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:32:58.591 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.591 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.591 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.591 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.592 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:32:58.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:58.619 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:58.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:58.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:58.745 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:32:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:58.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:32:58.793 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:32:58.810 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:32:58.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:58.854 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:58 smithi169 bash[19604]: audit 2024-02-12T16:32:57.769014+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? v1:172.21.15.121:0/3804756559' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:32:58.855 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:58 smithi169 bash[19604]: cluster 2024-02-12T16:32:57.769085+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e716: 12 total, 12 up, 12 in 2024-02-12T16:32:58.855 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:58 smithi169 bash[19604]: cluster 2024-02-12T16:32:58.081994+0000 mgr.y (mgr.14146) 1450 : cluster [DBG] pgmap v2046: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 6643/68974 objects misplaced (9.631%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:58.855 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:58 smithi169 bash[19604]: audit 2024-02-12T16:32:58.110716+0000 mon.a (mon.0) 2459 : audit [DBG] from='client.? v1:172.21.15.121:0/1050832187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:58.855 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:58 smithi169 bash[19604]: audit 2024-02-12T16:32:58.434788+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? v1:172.21.15.121:0/805755919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:32:58.871 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:58.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:58 smithi121 bash[18489]: audit 2024-02-12T16:32:57.769014+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? v1:172.21.15.121:0/3804756559' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:32:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:58 smithi121 bash[18489]: cluster 2024-02-12T16:32:57.769085+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e716: 12 total, 12 up, 12 in 2024-02-12T16:32:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:58 smithi121 bash[18489]: cluster 2024-02-12T16:32:58.081994+0000 mgr.y (mgr.14146) 1450 : cluster [DBG] pgmap v2046: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 6643/68974 objects misplaced (9.631%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:58.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:58 smithi121 bash[18489]: audit 2024-02-12T16:32:58.110716+0000 mon.a (mon.0) 2459 : audit [DBG] from='client.? v1:172.21.15.121:0/1050832187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:58.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:58 smithi121 bash[18489]: audit 2024-02-12T16:32:58.434788+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? v1:172.21.15.121:0/805755919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:32:58.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:59.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:32:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:32:59.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-12T16:32:59.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:58 smithi123 bash[19119]: audit 2024-02-12T16:32:57.769014+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? v1:172.21.15.121:0/3804756559' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:32:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:58 smithi123 bash[19119]: cluster 2024-02-12T16:32:57.769085+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e716: 12 total, 12 up, 12 in 2024-02-12T16:32:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:58 smithi123 bash[19119]: cluster 2024-02-12T16:32:58.081994+0000 mgr.y (mgr.14146) 1450 : cluster [DBG] pgmap v2046: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 6643/68974 objects misplaced (9.631%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:32:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:58 smithi123 bash[19119]: audit 2024-02-12T16:32:58.110716+0000 mon.a (mon.0) 2459 : audit [DBG] from='client.? v1:172.21.15.121:0/1050832187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:32:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:58 smithi123 bash[19119]: audit 2024-02-12T16:32:58.434788+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? v1:172.21.15.121:0/805755919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:32:59.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:32:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:32:59.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:59.291 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:32:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:32:59.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:59.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:59.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:59.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:32:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:32:59.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:59.674 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:32:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:32:59.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:59.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:32:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:32:59.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:59.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:59 smithi169 bash[19604]: cluster 2024-02-12T16:32:58.772537+0000 mon.a (mon.0) 2461 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:32:59.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:59 smithi169 bash[19604]: audit 2024-02-12T16:32:58.783664+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? v1:172.21.15.121:0/805755919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:32:59.929 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:32:59 smithi169 bash[19604]: cluster 2024-02-12T16:32:58.783707+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e717: 12 total, 12 up, 12 in 2024-02-12T16:32:59.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:32:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:32:59.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:00.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:00.056 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:59 smithi123 bash[19119]: cluster 2024-02-12T16:32:58.772537+0000 mon.a (mon.0) 2461 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:33:00.056 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:59 smithi123 bash[19119]: audit 2024-02-12T16:32:58.783664+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? v1:172.21.15.121:0/805755919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:33:00.056 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:32:59 smithi123 bash[19119]: cluster 2024-02-12T16:32:58.783707+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e717: 12 total, 12 up, 12 in 2024-02-12T16:33:00.084 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:00.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:59 smithi121 bash[18489]: cluster 2024-02-12T16:32:58.772537+0000 mon.a (mon.0) 2461 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:33:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:59 smithi121 bash[18489]: audit 2024-02-12T16:32:58.783664+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? v1:172.21.15.121:0/805755919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:33:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:32:59 smithi121 bash[18489]: cluster 2024-02-12T16:32:58.783707+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e717: 12 total, 12 up, 12 in 2024-02-12T16:33:00.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:00.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:00.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:00.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:00.339 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:00.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:00.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:00.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:00.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:00.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:00.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:00.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:00.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:00.745 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:00.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:00.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:00.888 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:00.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:00.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:01.000 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:00 smithi169 bash[19604]: cluster 2024-02-12T16:33:00.082487+0000 mgr.y (mgr.14146) 1451 : cluster [DBG] pgmap v2048: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 998 op/s; 6304/65716 objects misplaced (9.593%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:01.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:01.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:01.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:00 smithi123 bash[19119]: cluster 2024-02-12T16:33:00.082487+0000 mgr.y (mgr.14146) 1451 : cluster [DBG] pgmap v2048: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 998 op/s; 6304/65716 objects misplaced (9.593%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:01.159 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:01.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:00 smithi121 bash[18489]: cluster 2024-02-12T16:33:00.082487+0000 mgr.y (mgr.14146) 1451 : cluster [DBG] pgmap v2048: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 998 op/s; 6304/65716 objects misplaced (9.593%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:01.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:01.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:01.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:01.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:01.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:01.674 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:01.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:01.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:01 smithi169 bash[19604]: cluster 2024-02-12T16:33:00.291581+0000 osd.5 (osd.5) 141 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:33:01.820 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:01 smithi169 bash[19604]: cluster 2024-02-12T16:33:00.304256+0000 osd.5 (osd.5) 142 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:33:01.854 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:01.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:01.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:01.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:02.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:33:02.086 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:02.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:01 smithi123 bash[19119]: cluster 2024-02-12T16:33:00.291581+0000 osd.5 (osd.5) 141 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:33:02.094 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:01 smithi123 bash[19119]: cluster 2024-02-12T16:33:00.304256+0000 osd.5 (osd.5) 142 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:33:02.115 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:02.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:02.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:01 smithi121 bash[18489]: cluster 2024-02-12T16:33:00.291581+0000 osd.5 (osd.5) 141 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:33:02.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:01 smithi121 bash[18489]: cluster 2024-02-12T16:33:00.304256+0000 osd.5 (osd.5) 142 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:33:02.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:02.267 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:02.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:02.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:33:02.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:02.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:02.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:02.517 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:02.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:02.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:02.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:02.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:02.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:02.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:02.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:02.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:02.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:02 smithi121 bash[18489]: cluster 2024-02-12T16:33:01.320613+0000 osd.5 (osd.5) 143 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:33:02.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:02 smithi121 bash[18489]: cluster 2024-02-12T16:33:01.358565+0000 osd.5 (osd.5) 144 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:33:02.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:02 smithi121 bash[18489]: cluster 2024-02-12T16:33:02.083052+0000 mgr.y (mgr.14146) 1452 : cluster [DBG] pgmap v2049: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.8 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 5479/59632 objects misplaced (9.188%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:02.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:03.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:03.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:02 smithi123 bash[19119]: cluster 2024-02-12T16:33:01.320613+0000 osd.5 (osd.5) 143 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:33:03.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:02 smithi123 bash[19119]: cluster 2024-02-12T16:33:01.358565+0000 osd.5 (osd.5) 144 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:33:03.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:02 smithi123 bash[19119]: cluster 2024-02-12T16:33:02.083052+0000 mgr.y (mgr.14146) 1452 : cluster [DBG] pgmap v2049: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.8 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 5479/59632 objects misplaced (9.188%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:03.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:03.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:03.264 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:02 smithi169 bash[19604]: cluster 2024-02-12T16:33:01.320613+0000 osd.5 (osd.5) 143 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:33:03.264 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:02 smithi169 bash[19604]: cluster 2024-02-12T16:33:01.358565+0000 osd.5 (osd.5) 144 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:33:03.264 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:02 smithi169 bash[19604]: cluster 2024-02-12T16:33:02.083052+0000 mgr.y (mgr.14146) 1452 : cluster [DBG] pgmap v2049: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.8 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 5479/59632 objects misplaced (9.188%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:03.282 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:03.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:03.410 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:03.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:03.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:03.544 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:03.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:03.669 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:03.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:03.797 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:03.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:03.908 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:03.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:03.967 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:33:03.967 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:33:04.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:04.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:04.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:04.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:04.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:04.336 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:04.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:04.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:04.361 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:04.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:04.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:04.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:04.586 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:04.613 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:04.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:04.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:04.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:04.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:04.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:04.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:33:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:04.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:04.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:05.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:05.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:05.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:05.179 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:05.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:05.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:05.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:05.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:05 smithi123 bash[19119]: cluster 2024-02-12T16:33:04.083600+0000 mgr.y (mgr.14146) 1453 : cluster [DBG] pgmap v2050: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 5156/55314 objects misplaced (9.321%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:33:05.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:05 smithi121 bash[18489]: cluster 2024-02-12T16:33:04.083600+0000 mgr.y (mgr.14146) 1453 : cluster [DBG] pgmap v2050: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 5156/55314 objects misplaced (9.321%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:33:05.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:05.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:05 smithi169 bash[19604]: cluster 2024-02-12T16:33:04.083600+0000 mgr.y (mgr.14146) 1453 : cluster [DBG] pgmap v2050: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 5156/55314 objects misplaced (9.321%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:33:05.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:05.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:05.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:05.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:05.816 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:05.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:05.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:05.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:06.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:06.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:06.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:06.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:06.245 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:06.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:06.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:06.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:06.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:06.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:06.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:06.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:06.675 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:06.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:06.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:06.820 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:06.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:06.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:06.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:06.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:07.016 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:07.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:07.121 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:07.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:07.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:07.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:07.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:07.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:07.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:07.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:07 smithi123 bash[19119]: cluster 2024-02-12T16:33:05.704495+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:07 smithi123 bash[19119]: cluster 2024-02-12T16:33:05.704548+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:07 smithi123 bash[19119]: cluster 2024-02-12T16:33:06.084132+0000 mgr.y (mgr.14146) 1454 : cluster [DBG] pgmap v2051: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 4454/51002 objects misplaced (8.733%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:33:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:07 smithi123 bash[19119]: cluster 2024-02-12T16:33:06.140404+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e718: 12 total, 12 up, 12 in 2024-02-12T16:33:07.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:07.517 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:07.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:07 smithi169 bash[19604]: cluster 2024-02-12T16:33:05.704495+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:07 smithi169 bash[19604]: cluster 2024-02-12T16:33:05.704548+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:07 smithi169 bash[19604]: cluster 2024-02-12T16:33:06.084132+0000 mgr.y (mgr.14146) 1454 : cluster [DBG] pgmap v2051: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 4454/51002 objects misplaced (8.733%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:33:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:07 smithi169 bash[19604]: cluster 2024-02-12T16:33:06.140404+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e718: 12 total, 12 up, 12 in 2024-02-12T16:33:07.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:07 smithi121 bash[18489]: cluster 2024-02-12T16:33:05.704495+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:07.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:07 smithi121 bash[18489]: cluster 2024-02-12T16:33:05.704548+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:07.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:07 smithi121 bash[18489]: cluster 2024-02-12T16:33:06.084132+0000 mgr.y (mgr.14146) 1454 : cluster [DBG] pgmap v2051: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 4454/51002 objects misplaced (8.733%); 1.3 MiB/s, 21 objects/s recovering 2024-02-12T16:33:07.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:07 smithi121 bash[18489]: cluster 2024-02-12T16:33:06.140404+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e718: 12 total, 12 up, 12 in 2024-02-12T16:33:07.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:07.643 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:07.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:07.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:07.776 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:07.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:07.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:07.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:07.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:08.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:08.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:08.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:08.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:08.195 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:08.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:08.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:08.325 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:08.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:08.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:08.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:08.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:08.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:08.602 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:08.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:08.710 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:08.731 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:08.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:08.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:08.866 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:08.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:08.971 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:09.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:09.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:09.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:09.153 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:09.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:09.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:09.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:09.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:09 smithi123 bash[19119]: cluster 2024-02-12T16:33:08.084626+0000 mgr.y (mgr.14146) 1455 : cluster [DBG] pgmap v2053: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 4306/46066 objects misplaced (9.347%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:33:09.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:09.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:09 smithi121 bash[18489]: cluster 2024-02-12T16:33:08.084626+0000 mgr.y (mgr.14146) 1455 : cluster [DBG] pgmap v2053: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 4306/46066 objects misplaced (9.347%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:33:09.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:09.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:09.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:09 smithi169 bash[19604]: cluster 2024-02-12T16:33:08.084626+0000 mgr.y (mgr.14146) 1455 : cluster [DBG] pgmap v2053: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 4306/46066 objects misplaced (9.347%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:33:09.575 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:09.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:09.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:09.728 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:09.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:09.758 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:33:09.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:09.871 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:09.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:09.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:10.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:10.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:10.126 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:10.161 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:10.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:10.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:10.310 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:10.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:10.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:10.451 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:10.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:10.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:10.598 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:10.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:10.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:10.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:10.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:10.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:10.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:10.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:11.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:11.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:11.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:11.129 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:11.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:11.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:11.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:11.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:11.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:11.392 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:11 smithi169 bash[19604]: cluster 2024-02-12T16:33:10.085155+0000 mgr.y (mgr.14146) 1456 : cluster [DBG] pgmap v2054: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 3970/42758 objects misplaced (9.285%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:33:11.392 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:11 smithi169 bash[19604]: cluster 2024-02-12T16:33:10.765372+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e719: 12 total, 12 up, 12 in 2024-02-12T16:33:11.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:11 smithi121 bash[18489]: cluster 2024-02-12T16:33:10.085155+0000 mgr.y (mgr.14146) 1456 : cluster [DBG] pgmap v2054: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 3970/42758 objects misplaced (9.285%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:33:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:11 smithi121 bash[18489]: cluster 2024-02-12T16:33:10.765372+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e719: 12 total, 12 up, 12 in 2024-02-12T16:33:11.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:11.419 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:11.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:11.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:11.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:11 smithi123 bash[19119]: cluster 2024-02-12T16:33:10.085155+0000 mgr.y (mgr.14146) 1456 : cluster [DBG] pgmap v2054: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 3970/42758 objects misplaced (9.285%); 1.5 MiB/s, 23 objects/s recovering 2024-02-12T16:33:11.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:11 smithi123 bash[19119]: cluster 2024-02-12T16:33:10.765372+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e719: 12 total, 12 up, 12 in 2024-02-12T16:33:11.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:11.695 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:11.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:11.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:11.836 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:11.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:11.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:12.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:12.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:12.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:12.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:12.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:12.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:12.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:12.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:12.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:12.466 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:12.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:12.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:12.605 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:12.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:12.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:12.731 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:12.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:12.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:12.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:12.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:13.002 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:13.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:13.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:13.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:13.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:13 smithi169 bash[19604]: cluster 2024-02-12T16:33:12.085651+0000 mgr.y (mgr.14146) 1457 : cluster [DBG] pgmap v2056: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 3085/36940 objects misplaced (8.351%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:13.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:13.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:13 smithi121 bash[18489]: cluster 2024-02-12T16:33:12.085651+0000 mgr.y (mgr.14146) 1457 : cluster [DBG] pgmap v2056: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 3085/36940 objects misplaced (8.351%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:13.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:13.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:13.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:13.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:13.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:13 smithi123 bash[19119]: cluster 2024-02-12T16:33:12.085651+0000 mgr.y (mgr.14146) 1457 : cluster [DBG] pgmap v2056: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 3085/36940 objects misplaced (8.351%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:33:13.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:13.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:13.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:13.825 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:13.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:13.959 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:13.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:14.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:14.084 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:14.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:14.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:14 smithi121 bash[18489]: audit 2024-02-12T16:33:13.155047+0000 mon.a (mon.0) 2466 : audit [INF] from='client.26425 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:33:14.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:14 smithi121 bash[18489]: audit 2024-02-12T16:33:13.155978+0000 mon.c (mon.1) 61 : audit [INF] from='client.? v1:172.21.15.121:0/2094756730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:33:14.165 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.3 to 1 (10000) 2024-02-12T16:33:14.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:14.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:14.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:14.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:14 smithi169 bash[19604]: audit 2024-02-12T16:33:13.155047+0000 mon.a (mon.0) 2466 : audit [INF] from='client.26425 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:33:14.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:14 smithi169 bash[19604]: audit 2024-02-12T16:33:13.155978+0000 mon.c (mon.1) 61 : audit [INF] from='client.? v1:172.21.15.121:0/2094756730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:33:14.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:14.355 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:14.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:14.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:14.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:14 smithi123 bash[19119]: audit 2024-02-12T16:33:13.155047+0000 mon.a (mon.0) 2466 : audit [INF] from='client.26425 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:33:14.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:14 smithi123 bash[19119]: audit 2024-02-12T16:33:13.155978+0000 mon.c (mon.1) 61 : audit [INF] from='client.? v1:172.21.15.121:0/2094756730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:33:14.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:14.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:14.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:14.624 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:14.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:14.683 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:33:14.683 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:33:14.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:14.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:14.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:14.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:14.898 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:14.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:15.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:15.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:15.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:15.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:15.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:15.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:15.291 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:15.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:15.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T16:33:15.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:15 smithi123 bash[19119]: cluster 2024-02-12T16:33:14.086184+0000 mgr.y (mgr.14146) 1458 : cluster [DBG] pgmap v2057: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1024 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 2801/32756 objects misplaced (8.551%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:33:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:15 smithi123 bash[19119]: audit 2024-02-12T16:33:14.154520+0000 mon.a (mon.0) 2467 : audit [INF] from='client.26425 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:33:15.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:15 smithi123 bash[19119]: cluster 2024-02-12T16:33:14.154612+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2024-02-12T16:33:15.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:15 smithi121 bash[18489]: cluster 2024-02-12T16:33:14.086184+0000 mgr.y (mgr.14146) 1458 : cluster [DBG] pgmap v2057: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1024 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 2801/32756 objects misplaced (8.551%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:33:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:15 smithi121 bash[18489]: audit 2024-02-12T16:33:14.154520+0000 mon.a (mon.0) 2467 : audit [INF] from='client.26425 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:33:15.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:15 smithi121 bash[18489]: cluster 2024-02-12T16:33:14.154612+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2024-02-12T16:33:15.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:15.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:15 smithi169 bash[19604]: cluster 2024-02-12T16:33:14.086184+0000 mgr.y (mgr.14146) 1458 : cluster [DBG] pgmap v2057: 57 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 52 active+clean; 1024 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 2801/32756 objects misplaced (8.551%); 1.8 MiB/s, 29 objects/s recovering 2024-02-12T16:33:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:15 smithi169 bash[19604]: audit 2024-02-12T16:33:14.154520+0000 mon.a (mon.0) 2467 : audit [INF] from='client.26425 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:33:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:15 smithi169 bash[19604]: cluster 2024-02-12T16:33:14.154612+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e720: 12 total, 12 up, 12 in 2024-02-12T16:33:15.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:15.604 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:15.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:15.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:33:15.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:15.712 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:33:15.747 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:33:15.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:15.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:15.805 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:33:15.833 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:33:15.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:15.871 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:33:15.896 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:15.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:15.906 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:33:15.953 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:33:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:33:16.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:16.017 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:33:16.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:16.046 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:33:16.079 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:33:16.108 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.122 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.122 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.122 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.122 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:33:16.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:16.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:16.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:16.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:16 smithi169 bash[19604]: audit 2024-02-12T16:33:15.666884+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? v1:172.21.15.121:0/1407158155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:16.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:16 smithi169 bash[19604]: cluster 2024-02-12T16:33:15.765664+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e721: 12 total, 12 up, 12 in 2024-02-12T16:33:16.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:16 smithi169 bash[19604]: audit 2024-02-12T16:33:16.003077+0000 mon.a (mon.0) 2471 : audit [INF] from='client.26486 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:33:16.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:16 smithi169 bash[19604]: audit 2024-02-12T16:33:16.004463+0000 mon.b (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.121:0/4029074390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:33:16.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:16.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:16.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:16.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:16.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:16 smithi123 bash[19119]: audit 2024-02-12T16:33:15.666884+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? v1:172.21.15.121:0/1407158155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:16 smithi123 bash[19119]: cluster 2024-02-12T16:33:15.765664+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e721: 12 total, 12 up, 12 in 2024-02-12T16:33:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:16 smithi123 bash[19119]: audit 2024-02-12T16:33:16.003077+0000 mon.a (mon.0) 2471 : audit [INF] from='client.26486 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:33:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:16 smithi123 bash[19119]: audit 2024-02-12T16:33:16.004463+0000 mon.b (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.121:0/4029074390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:33:16.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:16 smithi121 bash[18489]: audit 2024-02-12T16:33:15.666884+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? v1:172.21.15.121:0/1407158155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:16 smithi121 bash[18489]: cluster 2024-02-12T16:33:15.765664+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e721: 12 total, 12 up, 12 in 2024-02-12T16:33:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:16 smithi121 bash[18489]: audit 2024-02-12T16:33:16.003077+0000 mon.a (mon.0) 2471 : audit [INF] from='client.26486 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:33:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:16 smithi121 bash[18489]: audit 2024-02-12T16:33:16.004463+0000 mon.b (mon.2) 146 : audit [INF] from='client.? v1:172.21.15.121:0/4029074390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:33:16.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:16.794 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:33:16.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:16.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:16.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:16.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:16.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:17.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:17.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:17.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:17.212 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:17.263 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:17.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:17 smithi169 bash[19604]: cluster 2024-02-12T16:33:16.086588+0000 mgr.y (mgr.14146) 1459 : cluster [DBG] pgmap v2060: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 895 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 1961/28650 objects misplaced (6.845%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:17 smithi169 bash[19604]: cluster 2024-02-12T16:33:16.234170+0000 osd.5 (osd.5) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:17 smithi169 bash[19604]: cluster 2024-02-12T16:33:16.234212+0000 osd.5 (osd.5) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:17 smithi169 bash[19604]: cluster 2024-02-12T16:33:16.767189+0000 mon.a (mon.0) 2472 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:17 smithi169 bash[19604]: cluster 2024-02-12T16:33:16.767633+0000 mon.a (mon.0) 2473 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:17 smithi169 bash[19604]: audit 2024-02-12T16:33:16.774983+0000 mon.a (mon.0) 2474 : audit [INF] from='client.26486 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:33:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:17 smithi169 bash[19604]: cluster 2024-02-12T16:33:16.775085+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e722: 12 total, 12 up, 12 in 2024-02-12T16:33:17.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:17.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:17.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:17.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:17.519 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:17 smithi121 bash[18489]: cluster 2024-02-12T16:33:16.086588+0000 mgr.y (mgr.14146) 1459 : cluster [DBG] pgmap v2060: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 895 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 1961/28650 objects misplaced (6.845%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:17.519 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:17 smithi121 bash[18489]: cluster 2024-02-12T16:33:16.234170+0000 osd.5 (osd.5) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:17.520 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:17 smithi121 bash[18489]: cluster 2024-02-12T16:33:16.234212+0000 osd.5 (osd.5) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:17.520 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:17 smithi121 bash[18489]: cluster 2024-02-12T16:33:16.767189+0000 mon.a (mon.0) 2472 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:17.520 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:17 smithi121 bash[18489]: cluster 2024-02-12T16:33:16.767633+0000 mon.a (mon.0) 2473 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:17.520 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:17 smithi121 bash[18489]: audit 2024-02-12T16:33:16.774983+0000 mon.a (mon.0) 2474 : audit [INF] from='client.26486 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:33:17.520 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:17 smithi121 bash[18489]: cluster 2024-02-12T16:33:16.775085+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e722: 12 total, 12 up, 12 in 2024-02-12T16:33:17.535 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:17.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:17.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:17 smithi123 bash[19119]: cluster 2024-02-12T16:33:16.086588+0000 mgr.y (mgr.14146) 1459 : cluster [DBG] pgmap v2060: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 895 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 1961/28650 objects misplaced (6.845%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:17 smithi123 bash[19119]: cluster 2024-02-12T16:33:16.234170+0000 osd.5 (osd.5) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:17 smithi123 bash[19119]: cluster 2024-02-12T16:33:16.234212+0000 osd.5 (osd.5) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:17 smithi123 bash[19119]: cluster 2024-02-12T16:33:16.767189+0000 mon.a (mon.0) 2472 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:17 smithi123 bash[19119]: cluster 2024-02-12T16:33:16.767633+0000 mon.a (mon.0) 2473 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:17 smithi123 bash[19119]: audit 2024-02-12T16:33:16.774983+0000 mon.a (mon.0) 2474 : audit [INF] from='client.26486 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:33:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:17 smithi123 bash[19119]: cluster 2024-02-12T16:33:16.775085+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e722: 12 total, 12 up, 12 in 2024-02-12T16:33:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:17.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:17.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:17.842 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:17.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:17.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:17.972 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:17.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:18.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:18.108 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:18.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:18.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:18.235 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:18.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:18.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:18.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:18.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:18.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:18.506 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:18.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:18.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:18.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:18.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:18.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:18.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:18.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:18.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:18.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:19.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:19.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:19.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:19.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:19.183 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:19.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:33:19.290 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:19.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:19 smithi169 bash[19604]: cluster 2024-02-12T16:33:18.087046+0000 mgr.y (mgr.14146) 1460 : cluster [DBG] pgmap v2062: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 748 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1835/23950 objects misplaced (7.662%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:19.316 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:19.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:19.444 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:19.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:19.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:19 smithi123 bash[19119]: cluster 2024-02-12T16:33:18.087046+0000 mgr.y (mgr.14146) 1460 : cluster [DBG] pgmap v2062: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 748 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1835/23950 objects misplaced (7.662%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:19.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:19.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:19 smithi121 bash[18489]: cluster 2024-02-12T16:33:18.087046+0000 mgr.y (mgr.14146) 1460 : cluster [DBG] pgmap v2062: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 748 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1835/23950 objects misplaced (7.662%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:19.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:19.795 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:19.821 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:19.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:33:19.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:19.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:19.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:20.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:20.095 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:20.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:20.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:20.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:20.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:20.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:20.340 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:20 smithi169 bash[19604]: cluster 2024-02-12T16:33:19.174266+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e723: 12 total, 12 up, 12 in 2024-02-12T16:33:20.362 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:20.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:20.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:20.471 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:20 smithi123 bash[19119]: cluster 2024-02-12T16:33:19.174266+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e723: 12 total, 12 up, 12 in 2024-02-12T16:33:20.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:20.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:20.517 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:33:20.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:20.620 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:20.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:20.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:20 smithi121 bash[18489]: cluster 2024-02-12T16:33:19.174266+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e723: 12 total, 12 up, 12 in 2024-02-12T16:33:20.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:20.746 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:20.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:20.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:20.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:20.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:20.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:20.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:21.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:21.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:21.161 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:21.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:21.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:21.375 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:21 smithi123 bash[19119]: cluster 2024-02-12T16:33:20.087434+0000 mgr.y (mgr.14146) 1461 : cluster [DBG] pgmap v2064: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 655 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 993 op/s; 1569/20962 objects misplaced (7.485%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:21 smithi123 bash[19119]: cluster 2024-02-12T16:33:20.184897+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e724: 12 total, 12 up, 12 in 2024-02-12T16:33:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:21 smithi123 bash[19119]: cluster 2024-02-12T16:33:20.765528+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e725: 12 total, 12 up, 12 in 2024-02-12T16:33:21.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:21.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:21 smithi121 bash[18489]: cluster 2024-02-12T16:33:20.087434+0000 mgr.y (mgr.14146) 1461 : cluster [DBG] pgmap v2064: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 655 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 993 op/s; 1569/20962 objects misplaced (7.485%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:21.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:21 smithi121 bash[18489]: cluster 2024-02-12T16:33:20.184897+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e724: 12 total, 12 up, 12 in 2024-02-12T16:33:21.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:21 smithi121 bash[18489]: cluster 2024-02-12T16:33:20.765528+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e725: 12 total, 12 up, 12 in 2024-02-12T16:33:21.432 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:21.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:21 smithi169 bash[19604]: cluster 2024-02-12T16:33:20.087434+0000 mgr.y (mgr.14146) 1461 : cluster [DBG] pgmap v2064: 57 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 52 active+clean; 655 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 993 op/s; 1569/20962 objects misplaced (7.485%); 1.7 MiB/s, 26 objects/s recovering 2024-02-12T16:33:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:21 smithi169 bash[19604]: cluster 2024-02-12T16:33:20.184897+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e724: 12 total, 12 up, 12 in 2024-02-12T16:33:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:21 smithi169 bash[19604]: cluster 2024-02-12T16:33:20.765528+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e725: 12 total, 12 up, 12 in 2024-02-12T16:33:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:21.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:21.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:21.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:21.841 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:21.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:21.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:21.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:22.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:22.093 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:22.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:22.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:22.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:22.245 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:22.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:22 smithi169 bash[19604]: cluster 2024-02-12T16:33:21.664254+0000 osd.8 (osd.8) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:22 smithi169 bash[19604]: cluster 2024-02-12T16:33:21.664302+0000 osd.8 (osd.8) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:22 smithi169 bash[19604]: audit 2024-02-12T16:33:22.091877+0000 mon.a (mon.0) 2479 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:33:22.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:22.372 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:22.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:22.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:22.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:22 smithi123 bash[19119]: cluster 2024-02-12T16:33:21.664254+0000 osd.8 (osd.8) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:22.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:22 smithi123 bash[19119]: cluster 2024-02-12T16:33:21.664302+0000 osd.8 (osd.8) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:22.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:22 smithi123 bash[19119]: audit 2024-02-12T16:33:22.091877+0000 mon.a (mon.0) 2479 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:33:22.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:22.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:22.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:22.609 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:22 smithi121 bash[18489]: cluster 2024-02-12T16:33:21.664254+0000 osd.8 (osd.8) 75 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:22.609 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:22 smithi121 bash[18489]: cluster 2024-02-12T16:33:21.664302+0000 osd.8 (osd.8) 76 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:22.609 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:22 smithi121 bash[18489]: audit 2024-02-12T16:33:22.091877+0000 mon.a (mon.0) 2479 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:33:22.628 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:22.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:22.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:22.756 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:22.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:22.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:22.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:22.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:22.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:23.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:23.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:23.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:23.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:23.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:23 smithi123 bash[19119]: cluster 2024-02-12T16:33:22.087945+0000 mgr.y (mgr.14146) 1462 : cluster [DBG] pgmap v2067: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 497 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 565/15910 objects misplaced (3.551%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T16:33:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:23 smithi123 bash[19119]: audit 2024-02-12T16:33:22.191301+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:33:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:23 smithi123 bash[19119]: cluster 2024-02-12T16:33:22.191353+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e726: 12 total, 12 up, 12 in 2024-02-12T16:33:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:23 smithi123 bash[19119]: cluster 2024-02-12T16:33:22.403837+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:23 smithi123 bash[19119]: cluster 2024-02-12T16:33:22.403905+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:23 smithi123 bash[19119]: cluster 2024-02-12T16:33:22.821260+0000 osd.10 (osd.10) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:23 smithi123 bash[19119]: cluster 2024-02-12T16:33:22.821349+0000 osd.10 (osd.10) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:23.394 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:23.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:23.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:23.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:23.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:23 smithi121 bash[18489]: cluster 2024-02-12T16:33:22.087945+0000 mgr.y (mgr.14146) 1462 : cluster [DBG] pgmap v2067: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 497 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 565/15910 objects misplaced (3.551%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T16:33:23.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:23 smithi121 bash[18489]: audit 2024-02-12T16:33:22.191301+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:33:23.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:23 smithi121 bash[18489]: cluster 2024-02-12T16:33:22.191353+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e726: 12 total, 12 up, 12 in 2024-02-12T16:33:23.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:23 smithi121 bash[18489]: cluster 2024-02-12T16:33:22.403837+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:23.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:23 smithi121 bash[18489]: cluster 2024-02-12T16:33:22.403905+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:23.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:23 smithi121 bash[18489]: cluster 2024-02-12T16:33:22.821260+0000 osd.10 (osd.10) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:23.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:23 smithi121 bash[18489]: cluster 2024-02-12T16:33:22.821349+0000 osd.10 (osd.10) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:23 smithi169 bash[19604]: cluster 2024-02-12T16:33:22.087945+0000 mgr.y (mgr.14146) 1462 : cluster [DBG] pgmap v2067: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 497 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 565/15910 objects misplaced (3.551%); 4.4 MiB/s, 70 objects/s recovering 2024-02-12T16:33:23.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:23 smithi169 bash[19604]: audit 2024-02-12T16:33:22.191301+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:33:23.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:23 smithi169 bash[19604]: cluster 2024-02-12T16:33:22.191353+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e726: 12 total, 12 up, 12 in 2024-02-12T16:33:23.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:23 smithi169 bash[19604]: cluster 2024-02-12T16:33:22.403837+0000 osd.3 (osd.3) 52 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:23.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:23 smithi169 bash[19604]: cluster 2024-02-12T16:33:22.403905+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:23.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:23 smithi169 bash[19604]: cluster 2024-02-12T16:33:22.821260+0000 osd.10 (osd.10) 144 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:23.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:23 smithi169 bash[19604]: cluster 2024-02-12T16:33:22.821349+0000 osd.10 (osd.10) 145 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:23.573 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:23.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:23.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:23.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:23.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:23.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:23.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:23.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:24.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:24.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:24.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:24.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:24.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:24.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:24 smithi169 bash[19604]: cluster 2024-02-12T16:33:23.200809+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e727: 12 total, 12 up, 12 in 2024-02-12T16:33:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:24 smithi169 bash[19604]: audit 2024-02-12T16:33:24.089692+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:33:24.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:24.376 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:24.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:24.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:24 smithi121 bash[18489]: cluster 2024-02-12T16:33:23.200809+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e727: 12 total, 12 up, 12 in 2024-02-12T16:33:24.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:24 smithi121 bash[18489]: audit 2024-02-12T16:33:24.089692+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:33:24.411 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:24.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:24.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:24.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:24.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:24 smithi123 bash[19119]: cluster 2024-02-12T16:33:23.200809+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e727: 12 total, 12 up, 12 in 2024-02-12T16:33:24.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:24 smithi123 bash[19119]: audit 2024-02-12T16:33:24.089692+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:33:24.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:24.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:24.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:24.787 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:24.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:163290 5.7 5:ed721b17:::benchmark_data_smithi169_84712_object163289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:48.338310+0000", 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 456.53626921400001, 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.38410730700000001, 2024-02-12T16:33:24.880 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 163290 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.338310+0000", 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.338310+0000", 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.338307+0000", 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2176000000000007e-05 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.338400+0000", 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6549e-05 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.338416+0000", 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3699999999999997e-06 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.338423+0000", 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026998399999999998 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.341123+0000", 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012457200000000001 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.341248+0000", 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116218 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.341364+0000", 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046923290000000003 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.346056+0000", 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.376225963 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.722282+0000", 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5507000000000007e-05 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.722368+0000", 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9109e-05 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.722417+0000", 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.38410730700000001 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:163315 5.7 5:ed952438:::benchmark_data_smithi169_84712_object163314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:48.371704+0000", 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 456.50287537999998, 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37069886499999999, 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 163315 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371704+0000", 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371704+0000", 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371701+0000", 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0398000000000002e-05 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371742+0000", 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2809000000000001e-05 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371755+0000", 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0379999999999999e-06 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371760+0000", 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0652000000000002e-05 2024-02-12T16:33:24.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371810+0000", 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012910999999999999 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.371939+0000", 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016797700000000001 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.372107+0000", 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088899280000000001 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.380997+0000", 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.361298008 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.742295+0000", 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4323000000000001e-05 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.742350+0000", 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2729000000000003e-05 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.742402+0000", 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37069886499999999 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:163321 5.7 5:e61aae06:::benchmark_data_smithi169_84712_object163320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:48.381192+0000", 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 456.493387326, 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36133947900000002, 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.891 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 163321 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381192+0000", 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381192+0000", 2024-02-12T16:33:24.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381190+0000", 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2794000000000003e-05 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381242+0000", 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1446e-05 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381254+0000", 2024-02-12T16:33:24.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6899999999999997e-06 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381260+0000", 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5565999999999998e-05 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381295+0000", 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3467999999999993e-05 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381389+0000", 2024-02-12T16:33:24.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109394 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.381498+0000", 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066354500000000002 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.388133+0000", 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35431056300000002 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.742444+0000", 2024-02-12T16:33:24.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1482e-05 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.742485+0000", 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5561e-05 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:48.742531+0000", 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36133947900000002 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:265717 5.f 5:f1976e7b:::benchmark_data_smithi169_84712_object96327:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.895080+0000", 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 357.97949863999997, 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.38014435600000002, 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265717 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895080+0000", 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895080+0000", 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895078+0000", 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7390000000000005e-06 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895087+0000", 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2707e-05 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895100+0000", 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5360000000000004e-06 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895107+0000", 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9834999999999998e-05 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895177+0000", 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000156916 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895334+0000", 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9635999999999994e-05 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.895413+0000", 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079666990000000007 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.903380+0000", 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37173843699999998 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275119+0000", 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2632999999999995e-05 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275181+0000", 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3361e-05 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275225+0000", 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.38014435600000002 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:265720 5.f 5:f572d6b6:::benchmark_data_smithi169_84712_object96330:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.896607+0000", 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 357.977971615, 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37879277700000002, 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.902 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265720 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.896607+0000", 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.896607+0000", 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.896606+0000", 2024-02-12T16:33:24.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0769999999999996e-06 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.896612+0000", 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1773000000000001e-05 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.896624+0000", 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0700000000000003e-06 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.896630+0000", 2024-02-12T16:33:24.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023300030000000002 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.898960+0000", 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111611 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.899071+0000", 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012920600000000001 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.899200+0000", 2024-02-12T16:33:24.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043937280000000004 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.903594+0000", 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.371666257 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275260+0000", 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8582e-05 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275349+0000", 2024-02-12T16:33:24.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1004000000000002e-05 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275400+0000", 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37879277700000002 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:265731 5.f 5:fd524628:::benchmark_data_smithi169_84712_object96341:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.905072+0000", 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 357.96950693500003, 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.370765185, 2024-02-12T16:33:24.907 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265731 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905072+0000", 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905072+0000", 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905070+0000", 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7919999999999998e-06 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905076+0000", 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2088e-05 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905088+0000", 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4960000000000004e-06 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905094+0000", 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5590000000000003e-05 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905129+0000", 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8844999999999995e-05 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905208+0000", 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3840999999999999e-05 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 4", 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.905292+0000", 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033929189999999999 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.908685+0000", 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36704699099999999 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275732+0000", 2024-02-12T16:33:24.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4789000000000006e-05 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275797+0000", 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0429999999999997e-05 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.275837+0000", 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.370765185 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:296311 5.7 5:e69056b5:::benchmark_data_smithi169_84712_object126921:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:56.131343+0000", 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 328.743235664, 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43886127200000002, 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 296311 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131343+0000", 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131343+0000", 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131341+0000", 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0209999999999999e-06 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131349+0000", 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3103e-05 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131362+0000", 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4559999999999997e-06 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131368+0000", 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4143000000000001e-05 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131413+0000", 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7717000000000001e-05 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131500+0000", 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3992000000000005e-05 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.131584+0000", 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004125859 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.135710+0000", 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43420132500000003 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.569911+0000", 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000159483 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.570071+0000", 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013353400000000001 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.570204+0000", 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43886127200000002 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:296336 5.7 5:eff33d21:::benchmark_data_smithi169_84712_object126946:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:56.151559+0000", 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 328.72301944399999, 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42308567400000002, 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:24.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 296336 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151559+0000", 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151559+0000", 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151559+0000", 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.452e-06 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151561+0000", 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2609999999999996e-06 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151569+0000", 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.723e-06 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151571+0000", 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9267000000000001e-05 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151621+0000", 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9406999999999994e-05 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151700+0000", 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8201999999999994e-05 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.151778+0000", 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098183650000000008 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.161597+0000", 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.412939259 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.574536+0000", 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6700000000000003e-05 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.574593+0000", 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2608000000000002e-05 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:56.574645+0000", 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42308567400000002 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101250 6.a 6:50811963:::benchmark_data_smithi169_96224_object101249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:33:24.923 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.301704+0000", 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.572874391, 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36810108499999999, 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101250 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.301704+0000", 2024-02-12T16:33:24.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.301704+0000", 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.301703+0000", 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4010000000000005e-05 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.301767+0000", 2024-02-12T16:33:24.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3600000000000002e-06 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.301776+0000", 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5979999999999996e-06 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.301782+0000", 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029741540000000001 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.304756+0000", 2024-02-12T16:33:24.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121754 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.304878+0000", 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013557599999999999 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305013+0000", 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068538169999999999 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.311867+0000", 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00093467500000000005 2024-02-12T16:33:24.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312802+0000", 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35640179900000002 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669204+0000", 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060181600000000003 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669806+0000", 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36810108499999999 2024-02-12T16:33:24.928 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101253 6.a 6:5897011c:::benchmark_data_smithi169_96224_object101252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.305761+0000", 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.56881760600001, 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36418827199999998, 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101253 2024-02-12T16:33:24.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305761+0000", 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305761+0000", 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305759+0000", 2024-02-12T16:33:24.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2475000000000002e-05 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305811+0000", 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1642e-05 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305823+0000", 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9049999999999996e-06 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305828+0000", 2024-02-12T16:33:24.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002636733 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.308464+0000", 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5120999999999999e-05 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.308559+0000", 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016752499999999999 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.308727+0000", 2024-02-12T16:33:24.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035455899999999999 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312273+0000", 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35758468999999998 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669857+0000", 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4060000000000001e-05 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669911+0000", 2024-02-12T16:33:24.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8189000000000003e-05 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669950+0000", 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36418827199999998 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101256 6.7 6:eefcfb11:::benchmark_data_smithi169_96224_object101255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.305832+0000", 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.568746619, 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.39068206900000002, 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.934 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101256 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305832+0000", 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305832+0000", 2024-02-12T16:33:24.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305831+0000", 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2546999999999999e-05 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305864+0000", 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5590000000000002e-06 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305871+0000", 2024-02-12T16:33:24.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4610000000000001e-06 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305875+0000", 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9975000000000002e-05 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305925+0000", 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119788 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.306045+0000", 2024-02-12T16:33:24.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015047899999999999 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.306195+0000", 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.368617959 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.674813+0000", 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021614439999999999 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696428+0000", 2024-02-12T16:33:24.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8696000000000003e-05 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696476+0000", 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8021000000000001e-05 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696514+0000", 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.39068206900000002 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.939 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101258 6.7 6:ea31abde:::benchmark_data_smithi169_96224_object101257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.305889+0000", 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.56869017099999, 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.39071883099999999, 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101258 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305889+0000", 2024-02-12T16:33:24.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305889+0000", 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305888+0000", 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5568e-05 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305913+0000", 2024-02-12T16:33:24.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8300000000000001e-06 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305919+0000", 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7190000000000001e-06 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305924+0000", 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039963610000000004 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.309920+0000", 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011413699999999999 2024-02-12T16:33:24.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.310034+0000", 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013569499999999999 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.310170+0000", 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36468809200000002 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.674858+0000", 2024-02-12T16:33:24.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021666543999999999 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696525+0000", 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6052e-05 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696561+0000", 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6699000000000001e-05 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696608+0000", 2024-02-12T16:33:24.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.39071883099999999 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26257.0:101251 6.4 e656/636 6:2e05ab6b:::benchmark_data_smithi169_96224_object101250:head v 656'6397, mlcod=656'6394)", 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.305970+0000", 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.56860875500001, 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37175343799999999, 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305970+0000", 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305970+0000", 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.305969+0000", 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8994e-05 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.306008+0000", 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5849999999999999e-06 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.306014+0000", 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6789999999999996e-06 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.306019+0000", 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116724 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.306135+0000", 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9198000000000002e-05 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.306155+0000", 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37151505099999999 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677670+0000", 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3996000000000003e-05 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677724+0000", 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37175343799999999 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101262 6.7 6:ee173a7a:::benchmark_data_smithi169_96224_object101261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.307768+0000", 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.56681043099999, 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.388927105, 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:24.949 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101262 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307768+0000", 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307768+0000", 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307766+0000", 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1965999999999995e-05 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307838+0000", 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1909000000000001e-05 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307850+0000", 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3339999999999998e-06 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307856+0000", 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36130633299999998 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669163+0000", 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000372111 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669535+0000", 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00103323 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.670568+0000", 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017113921000000001 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.687682+0000", 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0089338549999999992 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696616+0000", 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5200000000000001e-05 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696661+0000", 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4578e-05 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.696696+0000", 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.388927105 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26257.0:101254 6.4 e656/636 6:2f99d632:::benchmark_data_smithi169_96224_object101253:head v 656'6398, mlcod=656'6394)", 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.307810+0000", 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.56676931699999, 2024-02-12T16:33:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36994137900000001, 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307810+0000", 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307810+0000", 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307809+0000", 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1729000000000003e-05 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307890+0000", 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0510000000000002e-06 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307896+0000", 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7790000000000003e-06 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.307901+0000", 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020135940000000001 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.309915+0000", 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7309e-05 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.309932+0000", 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.367796699 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677729+0000", 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2065e-05 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677751+0000", 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36994137900000001 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101266 6.d 6:b459c46e:::benchmark_data_smithi169_96224_object101265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:33:24.958 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.312635+0000", 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.561944356, 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37763475899999999, 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101266 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312635+0000", 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312635+0000", 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312633+0000", 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5839e-05 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312669+0000", 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0569999999999993e-06 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312677+0000", 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6450000000000004e-06 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312681+0000", 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013145 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312813+0000", 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36312753800000003 2024-02-12T16:33:24.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.675940+0000", 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000250826 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.676191+0000", 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013377095 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.689568+0000", 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059310199999999997 2024-02-12T16:33:24.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.690161+0000", 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2856999999999998e-05 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.690224+0000", 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4991999999999999e-05 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.690269+0000", 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.37763475899999999 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26257.0:101250 6.a e656/636 ondisk, result = 0)", 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.312712+0000", 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.561866385, 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35710272599999998, 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312712+0000", 2024-02-12T16:33:24.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312712+0000", 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312712+0000", 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.98e-06 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312716+0000", 2024-02-12T16:33:24.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.964e-06 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312722+0000", 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8569999999999997e-06 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312725+0000", 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3540999999999994e-05 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312789+0000", 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8129999999999998e-06 2024-02-12T16:33:24.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312796+0000", 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35701944299999999 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669815+0000", 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35710272599999998 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.967 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26257.0:101253 6.a e656/636 ondisk, result = 0)", 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.312868+0000", 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.56171054999999, 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35708597399999997, 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312868+0000", 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312868+0000", 2024-02-12T16:33:24.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312867+0000", 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9289999999999997e-06 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.312872+0000", 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.356055501 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.668928+0000", 2024-02-12T16:33:24.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00086391400000000002 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669792+0000", 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9731000000000002e-05 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669841+0000", 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.337e-06 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669850+0000", 2024-02-12T16:33:24.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104718 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669954+0000", 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35708597399999997 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26257.0:101256 6.7 e656/636 ondisk, result = 0)", 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.313775+0000", 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 195.56080408099999, 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36104019500000001, 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.971 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.313775+0000", 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.313775+0000", 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.313774+0000", 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0499999999999999e-06 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.313779+0000", 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35517570300000001 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.668954+0000", 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7220000000000001e-05 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.668972+0000", 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058242399999999996 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.674796+0000", 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.674804+0000", 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.128e-05 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.674815+0000", 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.36104019500000001 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:179502 6.17 6:ee3707bd:::benchmark_data_smithi169_96224_object26666:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e680)", 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.754833+0000", 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 116.119746021, 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.068464603, 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:24.975 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 179502 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754833+0000", 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754833+0000", 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754831+0000", 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9300000000000003e-06 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754839+0000", 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5509e-05 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754854+0000", 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4959999999999999e-06 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754862+0000", 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21626487999999999 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.971127+0000", 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2310000000000002e-06 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.971136+0000", 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.85209320600000005 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.823229+0000", 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8484000000000004e-05 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.823297+0000", 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.068464603 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:24.979 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:33:24.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:25.005 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:25.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:25.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:25.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:25.162 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:25.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:25.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:25.215 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:25 smithi123 bash[19119]: cluster 2024-02-12T16:33:24.088616+0000 mgr.y (mgr.14146) 1463 : cluster [DBG] pgmap v2070: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 395 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 435/12632 objects misplaced (3.444%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:33:25.215 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:25 smithi123 bash[19119]: audit 2024-02-12T16:33:24.212551+0000 mon.a (mon.0) 2484 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:33:25.215 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:25 smithi123 bash[19119]: cluster 2024-02-12T16:33:24.212629+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e728: 12 total, 12 up, 12 in 2024-02-12T16:33:25.215 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:25 smithi123 bash[19119]: cluster 2024-02-12T16:33:24.214659+0000 osd.1 (osd.1) 106 : cluster [DBG] 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 725'14996 2024-02-12T16:33:25.215 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:25 smithi123 bash[19119]: cluster 2024-02-12T16:33:24.232981+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 725'14996 2024-02-12T16:33:25.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:25.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:25 smithi169 bash[19604]: cluster 2024-02-12T16:33:24.088616+0000 mgr.y (mgr.14146) 1463 : cluster [DBG] pgmap v2070: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 395 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 435/12632 objects misplaced (3.444%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:33:25.272 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:25 smithi169 bash[19604]: audit 2024-02-12T16:33:24.212551+0000 mon.a (mon.0) 2484 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:33:25.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:25 smithi169 bash[19604]: cluster 2024-02-12T16:33:24.212629+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e728: 12 total, 12 up, 12 in 2024-02-12T16:33:25.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:25 smithi169 bash[19604]: cluster 2024-02-12T16:33:24.214659+0000 osd.1 (osd.1) 106 : cluster [DBG] 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 725'14996 2024-02-12T16:33:25.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:25 smithi169 bash[19604]: cluster 2024-02-12T16:33:24.232981+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 725'14996 2024-02-12T16:33:25.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:25.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:25.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:25.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:25.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:25.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:25.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:25 smithi121 bash[18489]: cluster 2024-02-12T16:33:24.088616+0000 mgr.y (mgr.14146) 1463 : cluster [DBG] pgmap v2070: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 395 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 435/12632 objects misplaced (3.444%); 5.9 MiB/s, 94 objects/s recovering 2024-02-12T16:33:25.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:25 smithi121 bash[18489]: audit 2024-02-12T16:33:24.212551+0000 mon.a (mon.0) 2484 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:33:25.541 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:25 smithi121 bash[18489]: cluster 2024-02-12T16:33:24.212629+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e728: 12 total, 12 up, 12 in 2024-02-12T16:33:25.541 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:25 smithi121 bash[18489]: cluster 2024-02-12T16:33:24.214659+0000 osd.1 (osd.1) 106 : cluster [DBG] 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 725'14996 2024-02-12T16:33:25.541 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:25 smithi121 bash[18489]: cluster 2024-02-12T16:33:24.232981+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 725'14996 2024-02-12T16:33:25.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:25.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:25.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:25.725 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:25.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:25.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:25.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:33:25.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:25.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:25.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:26.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:26.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:26.139 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:26.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:26.276 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:26.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:26.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:26.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:26.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:26.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:26.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:26.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:26.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:26.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:26 smithi121 bash[18489]: cluster 2024-02-12T16:33:25.767299+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e729: 12 total, 12 up, 12 in 2024-02-12T16:33:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:26 smithi121 bash[18489]: cluster 2024-02-12T16:33:26.089128+0000 mgr.y (mgr.14146) 1464 : cluster [DBG] pgmap v2073: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 328 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 239/10510 objects misplaced (2.274%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:33:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:26 smithi121 bash[18489]: audit 2024-02-12T16:33:26.090017+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:33:26.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:26.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:26.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:26 smithi169 bash[19604]: cluster 2024-02-12T16:33:25.767299+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e729: 12 total, 12 up, 12 in 2024-02-12T16:33:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:26 smithi169 bash[19604]: cluster 2024-02-12T16:33:26.089128+0000 mgr.y (mgr.14146) 1464 : cluster [DBG] pgmap v2073: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 328 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 239/10510 objects misplaced (2.274%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:33:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:26 smithi169 bash[19604]: audit 2024-02-12T16:33:26.090017+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:33:27.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:27.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:26 smithi123 bash[19119]: cluster 2024-02-12T16:33:25.767299+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e729: 12 total, 12 up, 12 in 2024-02-12T16:33:27.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:26 smithi123 bash[19119]: cluster 2024-02-12T16:33:26.089128+0000 mgr.y (mgr.14146) 1464 : cluster [DBG] pgmap v2073: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 328 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 239/10510 objects misplaced (2.274%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:33:27.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:26 smithi123 bash[19119]: audit 2024-02-12T16:33:26.090017+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:33:27.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:27.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:27.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:27.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:27.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:27.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:27.352 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:27.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:27.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:27.492 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:27.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:27.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:27.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:27.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:27.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:27.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:27.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:27.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:27.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:27.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:28.005 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:27 smithi169 bash[19604]: audit 2024-02-12T16:33:26.778626+0000 mon.a (mon.0) 2488 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:33:28.005 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:27 smithi169 bash[19604]: cluster 2024-02-12T16:33:26.778721+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e730: 12 total, 12 up, 12 in 2024-02-12T16:33:28.006 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:27 smithi169 bash[19604]: cluster 2024-02-12T16:33:27.154237+0000 osd.1 (osd.1) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:28.006 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:27 smithi169 bash[19604]: cluster 2024-02-12T16:33:27.154280+0000 osd.1 (osd.1) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:28.022 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:28.091 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:27 smithi121 bash[18489]: audit 2024-02-12T16:33:26.778626+0000 mon.a (mon.0) 2488 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:33:28.092 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:27 smithi121 bash[18489]: cluster 2024-02-12T16:33:26.778721+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e730: 12 total, 12 up, 12 in 2024-02-12T16:33:28.092 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:27 smithi121 bash[18489]: cluster 2024-02-12T16:33:27.154237+0000 osd.1 (osd.1) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:28.092 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:27 smithi121 bash[18489]: cluster 2024-02-12T16:33:27.154280+0000 osd.1 (osd.1) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:28.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:27 smithi123 bash[19119]: audit 2024-02-12T16:33:26.778626+0000 mon.a (mon.0) 2488 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:33:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:27 smithi123 bash[19119]: cluster 2024-02-12T16:33:26.778721+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e730: 12 total, 12 up, 12 in 2024-02-12T16:33:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:27 smithi123 bash[19119]: cluster 2024-02-12T16:33:27.154237+0000 osd.1 (osd.1) 108 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:28.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:27 smithi123 bash[19119]: cluster 2024-02-12T16:33:27.154280+0000 osd.1 (osd.1) 109 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:28.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:28.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:28.280 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:28.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:28.417 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:28.521 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:28.546 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:28.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:28.680 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:28.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:28.816 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:28.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:28.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:28 smithi169 bash[19604]: cluster 2024-02-12T16:33:27.791135+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e731: 12 total, 12 up, 12 in 2024-02-12T16:33:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:28 smithi169 bash[19604]: cluster 2024-02-12T16:33:28.089617+0000 mgr.y (mgr.14146) 1465 : cluster [DBG] pgmap v2076: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 260 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 239/8336 objects misplaced (2.867%) 2024-02-12T16:33:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:28 smithi169 bash[19604]: audit 2024-02-12T16:33:28.090388+0000 mon.a (mon.0) 2491 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:33:29.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:29.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:28 smithi123 bash[19119]: cluster 2024-02-12T16:33:27.791135+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e731: 12 total, 12 up, 12 in 2024-02-12T16:33:29.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:28 smithi123 bash[19119]: cluster 2024-02-12T16:33:28.089617+0000 mgr.y (mgr.14146) 1465 : cluster [DBG] pgmap v2076: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 260 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 239/8336 objects misplaced (2.867%) 2024-02-12T16:33:29.057 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:28 smithi123 bash[19119]: audit 2024-02-12T16:33:28.090388+0000 mon.a (mon.0) 2491 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:33:29.073 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:29.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:29.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:28 smithi121 bash[18489]: cluster 2024-02-12T16:33:27.791135+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e731: 12 total, 12 up, 12 in 2024-02-12T16:33:29.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:28 smithi121 bash[18489]: cluster 2024-02-12T16:33:28.089617+0000 mgr.y (mgr.14146) 1465 : cluster [DBG] pgmap v2076: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 260 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 239/8336 objects misplaced (2.867%) 2024-02-12T16:33:29.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:28 smithi121 bash[18489]: audit 2024-02-12T16:33:28.090388+0000 mon.a (mon.0) 2491 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:33:29.176 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:29.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:29.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:29.350 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:29.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:29.489 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:29.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:29.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:29.715 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:33:29.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:29.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:29.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.793228+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:33:29.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: cluster 2024-02-12T16:33:28.793386+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e732: 12 total, 12 up, 12 in 2024-02-12T16:33:29.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: cluster 2024-02-12T16:33:28.795962+0000 osd.3 (osd.3) 54 : cluster [DBG] 6.16 starting backfill to osd.0 from (0'0,0'0] MAX to 729'14853 2024-02-12T16:33:29.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.804440+0000 mon.b (mon.2) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:33:29.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.804657+0000 mon.b (mon.2) 148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:33:29.810 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.804843+0000 mon.b (mon.2) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:33:29.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: cluster 2024-02-12T16:33:28.815630+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.16 starting backfill to osd.7 from (0'0,0'0] MAX to 729'14853 2024-02-12T16:33:29.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.828784+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:33:29.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.831344+0000 mon.b (mon.2) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:33:29.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.832346+0000 mon.b (mon.2) 152 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:33:29.811 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:29 smithi123 bash[19119]: audit 2024-02-12T16:33:28.838270+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:33:29.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:29.873 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:29.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:29.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.793228+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: cluster 2024-02-12T16:33:28.793386+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e732: 12 total, 12 up, 12 in 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: cluster 2024-02-12T16:33:28.795962+0000 osd.3 (osd.3) 54 : cluster [DBG] 6.16 starting backfill to osd.0 from (0'0,0'0] MAX to 729'14853 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.804440+0000 mon.b (mon.2) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.804657+0000 mon.b (mon.2) 148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.804843+0000 mon.b (mon.2) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: cluster 2024-02-12T16:33:28.815630+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.16 starting backfill to osd.7 from (0'0,0'0] MAX to 729'14853 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.828784+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.831344+0000 mon.b (mon.2) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.832346+0000 mon.b (mon.2) 152 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:33:30.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:29 smithi169 bash[19604]: audit 2024-02-12T16:33:28.838270+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:33:30.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:30.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:30.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.793228+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:33:30.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: cluster 2024-02-12T16:33:28.793386+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e732: 12 total, 12 up, 12 in 2024-02-12T16:33:30.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: cluster 2024-02-12T16:33:28.795962+0000 osd.3 (osd.3) 54 : cluster [DBG] 6.16 starting backfill to osd.0 from (0'0,0'0] MAX to 729'14853 2024-02-12T16:33:30.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.804440+0000 mon.b (mon.2) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:33:30.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.804657+0000 mon.b (mon.2) 148 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:33:30.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.804843+0000 mon.b (mon.2) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:33:30.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: cluster 2024-02-12T16:33:28.815630+0000 osd.3 (osd.3) 55 : cluster [DBG] 6.16 starting backfill to osd.7 from (0'0,0'0] MAX to 729'14853 2024-02-12T16:33:30.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.828784+0000 mon.b (mon.2) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:33:30.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.831344+0000 mon.b (mon.2) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:33:30.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.832346+0000 mon.b (mon.2) 152 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:33:30.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:29 smithi121 bash[18489]: audit 2024-02-12T16:33:28.838270+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:33:30.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:30.274 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:30.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:30.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:30.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:30.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:30.557 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:30.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:30.700 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:30.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:30.832 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:30.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:30.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:30.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:31.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:31.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:30 smithi121 bash[18489]: cluster 2024-02-12T16:33:29.799704+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e733: 12 total, 12 up, 12 in 2024-02-12T16:33:31.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:30 smithi121 bash[18489]: cluster 2024-02-12T16:33:30.090281+0000 mgr.y (mgr.14146) 1466 : cluster [DBG] pgmap v2079: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 215 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 454 op/s; 212/6876 objects misplaced (3.083%); 944 KiB/s, 14 objects/s recovering 2024-02-12T16:33:31.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:30 smithi121 bash[18489]: audit 2024-02-12T16:33:30.092387+0000 mon.a (mon.0) 2496 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:33:31.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:30 smithi121 bash[18489]: audit 2024-02-12T16:33:30.092636+0000 mon.b (mon.2) 153 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:33:31.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:30 smithi121 bash[18489]: audit 2024-02-12T16:33:30.773264+0000 mon.a (mon.0) 2497 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:33:31.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:30 smithi121 bash[18489]: cluster 2024-02-12T16:33:30.773414+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e734: 12 total, 12 up, 12 in 2024-02-12T16:33:31.116 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:31.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:30 smithi123 bash[19119]: cluster 2024-02-12T16:33:29.799704+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e733: 12 total, 12 up, 12 in 2024-02-12T16:33:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:30 smithi123 bash[19119]: cluster 2024-02-12T16:33:30.090281+0000 mgr.y (mgr.14146) 1466 : cluster [DBG] pgmap v2079: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 215 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 454 op/s; 212/6876 objects misplaced (3.083%); 944 KiB/s, 14 objects/s recovering 2024-02-12T16:33:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:30 smithi123 bash[19119]: audit 2024-02-12T16:33:30.092387+0000 mon.a (mon.0) 2496 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:33:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:30 smithi123 bash[19119]: audit 2024-02-12T16:33:30.092636+0000 mon.b (mon.2) 153 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:33:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:30 smithi123 bash[19119]: audit 2024-02-12T16:33:30.773264+0000 mon.a (mon.0) 2497 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:33:31.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:30 smithi123 bash[19119]: cluster 2024-02-12T16:33:30.773414+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e734: 12 total, 12 up, 12 in 2024-02-12T16:33:31.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:31.258 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:31.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:30 smithi169 bash[19604]: cluster 2024-02-12T16:33:29.799704+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e733: 12 total, 12 up, 12 in 2024-02-12T16:33:31.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:30 smithi169 bash[19604]: cluster 2024-02-12T16:33:30.090281+0000 mgr.y (mgr.14146) 1466 : cluster [DBG] pgmap v2079: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 215 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 454 op/s; 212/6876 objects misplaced (3.083%); 944 KiB/s, 14 objects/s recovering 2024-02-12T16:33:31.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:30 smithi169 bash[19604]: audit 2024-02-12T16:33:30.092387+0000 mon.a (mon.0) 2496 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:33:31.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:30 smithi169 bash[19604]: audit 2024-02-12T16:33:30.092636+0000 mon.b (mon.2) 153 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:33:31.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:30 smithi169 bash[19604]: audit 2024-02-12T16:33:30.773264+0000 mon.a (mon.0) 2497 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:33:31.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:30 smithi169 bash[19604]: cluster 2024-02-12T16:33:30.773414+0000 mon.a (mon.0) 2498 : cluster [DBG] osdmap e734: 12 total, 12 up, 12 in 2024-02-12T16:33:31.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:31.390 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:31.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:31.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:31.519 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:31.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:31.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:31.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:31.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:31.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:31.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:31.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:31.790 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T16:33:31.854 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:33:31.881 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:33:31.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:31.907 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:33:31.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:31.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:31.935 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:33:31.960 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:33:31.988 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:33:32.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:32.014 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:33:32.026 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:31 smithi121 bash[18489]: cluster 2024-02-12T16:33:30.772075+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.17 starting backfill to osd.2 from (0'0,0'0] MAX to 731'17978 2024-02-12T16:33:32.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:31 smithi121 bash[18489]: cluster 2024-02-12T16:33:30.792658+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.17 starting backfill to osd.10 from (0'0,0'0] MAX to 731'17978 2024-02-12T16:33:32.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:31 smithi121 bash[18489]: audit 2024-02-12T16:33:30.823193+0000 mon.a (mon.0) 2499 : audit [INF] from='client.? v1:172.21.15.121:0/3618086636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:33:32.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:31 smithi121 bash[18489]: audit 2024-02-12T16:33:31.781371+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? v1:172.21.15.121:0/3618086636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:33:32.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:31 smithi121 bash[18489]: cluster 2024-02-12T16:33:31.781448+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e735: 12 total, 12 up, 12 in 2024-02-12T16:33:32.027 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:31 smithi121 bash[18489]: audit 2024-02-12T16:33:31.818438+0000 mon.a (mon.0) 2502 : audit [DBG] from='client.? v1:172.21.15.121:0/3698216038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:32.037 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:33:32.037 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:32.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:32.059 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:33:32.091 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:33:32.112 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:33:32.133 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:33:32.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.135 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.136 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.136 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.136 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.136 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.136 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.136 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:33:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:31 smithi123 bash[19119]: cluster 2024-02-12T16:33:30.772075+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.17 starting backfill to osd.2 from (0'0,0'0] MAX to 731'17978 2024-02-12T16:33:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:31 smithi123 bash[19119]: cluster 2024-02-12T16:33:30.792658+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.17 starting backfill to osd.10 from (0'0,0'0] MAX to 731'17978 2024-02-12T16:33:32.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:31 smithi123 bash[19119]: audit 2024-02-12T16:33:30.823193+0000 mon.a (mon.0) 2499 : audit [INF] from='client.? v1:172.21.15.121:0/3618086636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:33:32.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:31 smithi123 bash[19119]: audit 2024-02-12T16:33:31.781371+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? v1:172.21.15.121:0/3618086636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:33:32.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:31 smithi123 bash[19119]: cluster 2024-02-12T16:33:31.781448+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e735: 12 total, 12 up, 12 in 2024-02-12T16:33:32.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:31 smithi123 bash[19119]: audit 2024-02-12T16:33:31.818438+0000 mon.a (mon.0) 2502 : audit [DBG] from='client.? v1:172.21.15.121:0/3698216038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:32.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:32.165 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:32.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:32.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:32.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:32.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:31 smithi169 bash[19604]: cluster 2024-02-12T16:33:30.772075+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.17 starting backfill to osd.2 from (0'0,0'0] MAX to 731'17978 2024-02-12T16:33:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:31 smithi169 bash[19604]: cluster 2024-02-12T16:33:30.792658+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.17 starting backfill to osd.10 from (0'0,0'0] MAX to 731'17978 2024-02-12T16:33:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:31 smithi169 bash[19604]: audit 2024-02-12T16:33:30.823193+0000 mon.a (mon.0) 2499 : audit [INF] from='client.? v1:172.21.15.121:0/3618086636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:33:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:31 smithi169 bash[19604]: audit 2024-02-12T16:33:31.781371+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? v1:172.21.15.121:0/3618086636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:33:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:31 smithi169 bash[19604]: cluster 2024-02-12T16:33:31.781448+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e735: 12 total, 12 up, 12 in 2024-02-12T16:33:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:31 smithi169 bash[19604]: audit 2024-02-12T16:33:31.818438+0000 mon.a (mon.0) 2502 : audit [DBG] from='client.? v1:172.21.15.121:0/3698216038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:32.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:32.428 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:32.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:32.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:32.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:32.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:32.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:32.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:32.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:32.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:32.855 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:32.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:32.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:32.998 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:33.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:33.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:32 smithi169 bash[19604]: cluster 2024-02-12T16:33:32.090773+0000 mgr.y (mgr.14146) 1467 : cluster [DBG] pgmap v2082: 57 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 143 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 166/4574 objects misplaced (3.629%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:33:33.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:32 smithi169 bash[19604]: cluster 2024-02-12T16:33:32.783989+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e736: 12 total, 12 up, 12 in 2024-02-12T16:33:33.134 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:32 smithi123 bash[19119]: cluster 2024-02-12T16:33:32.090773+0000 mgr.y (mgr.14146) 1467 : cluster [DBG] pgmap v2082: 57 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 143 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 166/4574 objects misplaced (3.629%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:33:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:32 smithi123 bash[19119]: cluster 2024-02-12T16:33:32.783989+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e736: 12 total, 12 up, 12 in 2024-02-12T16:33:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:32 smithi121 bash[18489]: cluster 2024-02-12T16:33:32.090773+0000 mgr.y (mgr.14146) 1467 : cluster [DBG] pgmap v2082: 57 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 52 active+clean; 143 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 166/4574 objects misplaced (3.629%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:33:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:32 smithi121 bash[18489]: cluster 2024-02-12T16:33:32.783989+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e736: 12 total, 12 up, 12 in 2024-02-12T16:33:33.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T16:33:33.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:33.275 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:33.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:33.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:33.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:33.553 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:33.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:33.687 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:33.793 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:33.794 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:33:33.832 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:33.833 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:33:33.833 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:33:33.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:33.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:33.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:33.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:34.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:34.092 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:33 smithi123 bash[19119]: cluster 2024-02-12T16:33:32.786377+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.18 starting backfill to osd.0 from (0'0,0'0] MAX to 732'20139 2024-02-12T16:33:34.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:33 smithi123 bash[19119]: cluster 2024-02-12T16:33:32.805914+0000 osd.7 (osd.7) 177 : cluster [DBG] 6.18 starting backfill to osd.1 from (0'0,0'0] MAX to 732'20139 2024-02-12T16:33:34.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:33 smithi123 bash[19119]: cluster 2024-02-12T16:33:32.831335+0000 mon.a (mon.0) 2504 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:34.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:33 smithi123 bash[19119]: audit 2024-02-12T16:33:32.885596+0000 mon.a (mon.0) 2505 : audit [INF] from='client.? v1:172.21.15.121:0/2532529683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:33:34.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:33 smithi123 bash[19119]: cluster 2024-02-12T16:33:33.779853+0000 mon.a (mon.0) 2506 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:34.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:33 smithi123 bash[19119]: audit 2024-02-12T16:33:33.784035+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? v1:172.21.15.121:0/2532529683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:33:34.093 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:33 smithi123 bash[19119]: cluster 2024-02-12T16:33:33.784111+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e737: 12 total, 12 up, 12 in 2024-02-12T16:33:34.127 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:34.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:33 smithi121 bash[18489]: cluster 2024-02-12T16:33:32.786377+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.18 starting backfill to osd.0 from (0'0,0'0] MAX to 732'20139 2024-02-12T16:33:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:33 smithi121 bash[18489]: cluster 2024-02-12T16:33:32.805914+0000 osd.7 (osd.7) 177 : cluster [DBG] 6.18 starting backfill to osd.1 from (0'0,0'0] MAX to 732'20139 2024-02-12T16:33:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:33 smithi121 bash[18489]: cluster 2024-02-12T16:33:32.831335+0000 mon.a (mon.0) 2504 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:33 smithi121 bash[18489]: audit 2024-02-12T16:33:32.885596+0000 mon.a (mon.0) 2505 : audit [INF] from='client.? v1:172.21.15.121:0/2532529683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:33:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:33 smithi121 bash[18489]: cluster 2024-02-12T16:33:33.779853+0000 mon.a (mon.0) 2506 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:33 smithi121 bash[18489]: audit 2024-02-12T16:33:33.784035+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? v1:172.21.15.121:0/2532529683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:33:34.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:33 smithi121 bash[18489]: cluster 2024-02-12T16:33:33.784111+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e737: 12 total, 12 up, 12 in 2024-02-12T16:33:34.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:34.282 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:34.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:33 smithi169 bash[19604]: cluster 2024-02-12T16:33:32.786377+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.18 starting backfill to osd.0 from (0'0,0'0] MAX to 732'20139 2024-02-12T16:33:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:33 smithi169 bash[19604]: cluster 2024-02-12T16:33:32.805914+0000 osd.7 (osd.7) 177 : cluster [DBG] 6.18 starting backfill to osd.1 from (0'0,0'0] MAX to 732'20139 2024-02-12T16:33:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:33 smithi169 bash[19604]: cluster 2024-02-12T16:33:32.831335+0000 mon.a (mon.0) 2504 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:33 smithi169 bash[19604]: audit 2024-02-12T16:33:32.885596+0000 mon.a (mon.0) 2505 : audit [INF] from='client.? v1:172.21.15.121:0/2532529683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:33:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:33 smithi169 bash[19604]: cluster 2024-02-12T16:33:33.779853+0000 mon.a (mon.0) 2506 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:33 smithi169 bash[19604]: audit 2024-02-12T16:33:33.784035+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? v1:172.21.15.121:0/2532529683' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:33:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:33 smithi169 bash[19604]: cluster 2024-02-12T16:33:33.784111+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e737: 12 total, 12 up, 12 in 2024-02-12T16:33:34.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:34.410 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:34.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:34.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:34.542 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:34.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:34.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:33:34.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:34.675 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:34.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:34.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:34.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:34 smithi123 bash[19119]: cluster 2024-02-12T16:33:34.091275+0000 mgr.y (mgr.14146) 1468 : cluster [DBG] pgmap v2085: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+remapped, 53 active+clean; 118 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 391 op/s; 406/3764 objects misplaced (10.786%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:33:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:34 smithi121 bash[18489]: cluster 2024-02-12T16:33:34.091275+0000 mgr.y (mgr.14146) 1468 : cluster [DBG] pgmap v2085: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+remapped, 53 active+clean; 118 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 391 op/s; 406/3764 objects misplaced (10.786%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:33:34.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:34.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:34.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:35.025 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 152833 objects 2024-02-12T16:33:35.026 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :155.445741 2024-02-12T16:33:35.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:35.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:35.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:35.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:35.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:35.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:34 smithi169 bash[19604]: cluster 2024-02-12T16:33:34.091275+0000 mgr.y (mgr.14146) 1468 : cluster [DBG] pgmap v2085: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+remapped, 53 active+clean; 118 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 391 op/s; 406/3764 objects misplaced (10.786%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:33:35.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:35.370 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:35.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:35.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:35.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:33:35.510 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:35.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:35.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:35.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:35.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:35.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:35.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:35.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:35.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:35.925 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:35.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:36.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:36.065 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:36.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:36.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:36.205 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:36.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:36.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:36.350 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:36.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:36.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:36.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:36.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:36.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:36.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:36.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:36.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:33:36.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:36.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:36.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:36.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:36.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:36 smithi121 bash[18489]: cluster 2024-02-12T16:33:35.776578+0000 mon.a (mon.0) 2509 : cluster [DBG] osdmap e738: 12 total, 12 up, 12 in 2024-02-12T16:33:36.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:36 smithi121 bash[18489]: cluster 2024-02-12T16:33:36.091939+0000 mgr.y (mgr.14146) 1469 : cluster [DBG] pgmap v2087: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+remapped, 53 active+clean; 73 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 280 op/s; 240/2344 objects misplaced (10.239%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:33:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:36.912 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:36.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:37.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:37.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:36 smithi169 bash[19604]: cluster 2024-02-12T16:33:35.776578+0000 mon.a (mon.0) 2509 : cluster [DBG] osdmap e738: 12 total, 12 up, 12 in 2024-02-12T16:33:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:36 smithi169 bash[19604]: cluster 2024-02-12T16:33:36.091939+0000 mgr.y (mgr.14146) 1469 : cluster [DBG] pgmap v2087: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+remapped, 53 active+clean; 73 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 280 op/s; 240/2344 objects misplaced (10.239%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:33:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:37.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:37.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:36 smithi123 bash[19119]: cluster 2024-02-12T16:33:35.776578+0000 mon.a (mon.0) 2509 : cluster [DBG] osdmap e738: 12 total, 12 up, 12 in 2024-02-12T16:33:37.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:36 smithi123 bash[19119]: cluster 2024-02-12T16:33:36.091939+0000 mgr.y (mgr.14146) 1469 : cluster [DBG] pgmap v2087: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+remapped, 53 active+clean; 73 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 280 op/s; 240/2344 objects misplaced (10.239%); 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:33:37.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:37.192 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:37.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:37.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:37.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:37.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:37.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:37.476 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:37.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:37.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:37.618 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:37.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:37.723 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:37.758 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:37.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:37.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:37.873 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:37 smithi121 bash[18489]: cluster 2024-02-12T16:33:36.786270+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e739: 12 total, 12 up, 12 in 2024-02-12T16:33:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:37.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:37.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:38.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:38.040 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:37 smithi169 bash[19604]: cluster 2024-02-12T16:33:36.786270+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e739: 12 total, 12 up, 12 in 2024-02-12T16:33:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:38.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:38.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:37 smithi123 bash[19119]: cluster 2024-02-12T16:33:36.786270+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e739: 12 total, 12 up, 12 in 2024-02-12T16:33:38.142 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:38.180 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:38.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:38.189 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:38.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:38.319 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:38.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:38.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:38.464 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:38.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:38.569 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:38.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:38.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:38.676 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-02-12T16:33:38.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-02-12T16:33:38.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:38.757 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:38.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:38.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:38.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:38 smithi123 bash[19119]: cluster 2024-02-12T16:33:38.092734+0000 mgr.y (mgr.14146) 1470 : cluster [DBG] pgmap v2089: 57 pgs: 2 peering, 2 active+remapped+backfill_toofull, 53 active+clean; 29 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 333 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:33:38.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:38.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:38.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:39.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:38 smithi121 bash[18489]: cluster 2024-02-12T16:33:38.092734+0000 mgr.y (mgr.14146) 1470 : cluster [DBG] pgmap v2089: 57 pgs: 2 peering, 2 active+remapped+backfill_toofull, 53 active+clean; 29 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 333 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:33:39.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:39.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:39.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:39.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:39.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:39.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:39.286 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:38 smithi169 bash[19604]: cluster 2024-02-12T16:33:38.092734+0000 mgr.y (mgr.14146) 1470 : cluster [DBG] pgmap v2089: 57 pgs: 2 peering, 2 active+remapped+backfill_toofull, 53 active+clean; 29 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 333 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-02-12T16:33:39.314 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:39.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:39.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:39.457 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:39.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:39.482 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:33:39.559 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:39.596 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:39.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:39.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:39.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:39.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:39.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:39.880 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:39.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:39.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:40.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:40.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:40.151 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:40.223 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:40.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:40.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:40.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:40.428 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:40.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:40.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:40.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:40.710 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:40.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:40.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:40.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:40.989 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:40.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:41.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:41.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:41.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:41.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:41.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:41 smithi123 bash[19119]: cluster 2024-02-12T16:33:40.093707+0000 mgr.y (mgr.14146) 1471 : cluster [DBG] pgmap v2090: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 54 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 304 op/s; 152 KiB/s, 2 objects/s recovering 2024-02-12T16:33:41.271 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:41.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:41.350 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:41.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:41.405 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:41.407 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:41 smithi169 bash[19604]: cluster 2024-02-12T16:33:40.093707+0000 mgr.y (mgr.14146) 1471 : cluster [DBG] pgmap v2090: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 54 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 304 op/s; 152 KiB/s, 2 objects/s recovering 2024-02-12T16:33:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:41.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:41.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:41 smithi121 bash[18489]: cluster 2024-02-12T16:33:40.093707+0000 mgr.y (mgr.14146) 1471 : cluster [DBG] pgmap v2090: 57 pgs: 1 peering, 2 active+remapped+backfill_toofull, 54 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 304 op/s; 152 KiB/s, 2 objects/s recovering 2024-02-12T16:33:41.508 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:41.546 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:41.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:41.651 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:41.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:41.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:41.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:41.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:41.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:41.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:41.966 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:41.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:42.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:42.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:42.212 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:42.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:42.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:42.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:42.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:42.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:42.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:42 smithi121 bash[18489]: audit 2024-02-12T16:33:42.096974+0000 mon.a (mon.0) 2511 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:33:42.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:42 smithi121 bash[18489]: audit 2024-02-12T16:33:42.097203+0000 mon.b (mon.2) 154 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:33:42.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:42.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:42 smithi169 bash[19604]: audit 2024-02-12T16:33:42.096974+0000 mon.a (mon.0) 2511 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:33:42.501 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:42 smithi169 bash[19604]: audit 2024-02-12T16:33:42.097203+0000 mon.b (mon.2) 154 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:33:42.527 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:42.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:42.620 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:42 smithi123 bash[19119]: audit 2024-02-12T16:33:42.096974+0000 mon.a (mon.0) 2511 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:33:42.620 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:42 smithi123 bash[19119]: audit 2024-02-12T16:33:42.097203+0000 mon.b (mon.2) 154 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-12T16:33:42.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:42.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:42.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:42.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:42.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:42.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:42.873 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:33:42.873 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:33:42.911 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:42.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:42.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:43.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:43.135 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:43.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:43.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:43.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:43.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:43.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:43.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:43.426 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:43 smithi121 bash[18489]: cluster 2024-02-12T16:33:42.094547+0000 mgr.y (mgr.14146) 1472 : cluster [DBG] pgmap v2091: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 234 op/s; 120 KiB/s, 1 objects/s recovering 2024-02-12T16:33:43.426 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:43 smithi121 bash[18489]: audit 2024-02-12T16:33:42.165610+0000 mon.a (mon.0) 2512 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T16:33:43.426 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:43 smithi121 bash[18489]: cluster 2024-02-12T16:33:42.165759+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e740: 12 total, 12 up, 12 in 2024-02-12T16:33:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:43.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:43.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:43 smithi169 bash[19604]: cluster 2024-02-12T16:33:42.094547+0000 mgr.y (mgr.14146) 1472 : cluster [DBG] pgmap v2091: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 234 op/s; 120 KiB/s, 1 objects/s recovering 2024-02-12T16:33:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:43 smithi169 bash[19604]: audit 2024-02-12T16:33:42.165610+0000 mon.a (mon.0) 2512 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T16:33:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:43 smithi169 bash[19604]: cluster 2024-02-12T16:33:42.165759+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e740: 12 total, 12 up, 12 in 2024-02-12T16:33:43.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:43.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:43.587 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:43 smithi123 bash[19119]: cluster 2024-02-12T16:33:42.094547+0000 mgr.y (mgr.14146) 1472 : cluster [DBG] pgmap v2091: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 234 op/s; 120 KiB/s, 1 objects/s recovering 2024-02-12T16:33:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:43 smithi123 bash[19119]: audit 2024-02-12T16:33:42.165610+0000 mon.a (mon.0) 2512 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-12T16:33:43.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:43 smithi123 bash[19119]: cluster 2024-02-12T16:33:42.165759+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e740: 12 total, 12 up, 12 in 2024-02-12T16:33:43.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:33:43.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:43.703 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:43.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:43.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:43.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:43.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:43.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:43.981 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:43.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:44.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:44.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:44.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:44.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:44.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:44.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:44.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:44.374 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:44 smithi123 bash[19119]: cluster 2024-02-12T16:33:43.166853+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e741: 12 total, 12 up, 12 in 2024-02-12T16:33:44.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:44.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:44.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:44.512 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:44 smithi169 bash[19604]: cluster 2024-02-12T16:33:43.166853+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e741: 12 total, 12 up, 12 in 2024-02-12T16:33:44.539 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:44.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:44.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:44.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:44 smithi121 bash[18489]: cluster 2024-02-12T16:33:43.166853+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e741: 12 total, 12 up, 12 in 2024-02-12T16:33:44.680 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:44.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:44.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:44.821 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:44.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:44.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:44.962 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:44.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:45.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:45.104 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:45.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:45.191 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T16:33:45.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:45.231 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:45.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:45.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:45.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:45.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:45 smithi123 bash[19119]: cluster 2024-02-12T16:33:44.095543+0000 mgr.y (mgr.14146) 1473 : cluster [DBG] pgmap v2094: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s 2024-02-12T16:33:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:45 smithi123 bash[19119]: cluster 2024-02-12T16:33:44.168706+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e742: 12 total, 12 up, 12 in 2024-02-12T16:33:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:45 smithi123 bash[19119]: cluster 2024-02-12T16:33:44.171649+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.19 starting backfill to osd.1 from (0'0,0'0] MAX to 737'19641 2024-02-12T16:33:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:45 smithi123 bash[19119]: audit 2024-02-12T16:33:44.183824+0000 mon.a (mon.0) 2516 : audit [INF] from='client.? v1:172.21.15.121:0/1247893971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:33:45.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:45 smithi123 bash[19119]: cluster 2024-02-12T16:33:44.188347+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.19 starting backfill to osd.5 from (0'0,0'0] MAX to 737'19641 2024-02-12T16:33:45.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:45.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:45.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:45 smithi169 bash[19604]: cluster 2024-02-12T16:33:44.095543+0000 mgr.y (mgr.14146) 1473 : cluster [DBG] pgmap v2094: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s 2024-02-12T16:33:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:45 smithi169 bash[19604]: cluster 2024-02-12T16:33:44.168706+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e742: 12 total, 12 up, 12 in 2024-02-12T16:33:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:45 smithi169 bash[19604]: cluster 2024-02-12T16:33:44.171649+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.19 starting backfill to osd.1 from (0'0,0'0] MAX to 737'19641 2024-02-12T16:33:45.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:45 smithi169 bash[19604]: audit 2024-02-12T16:33:44.183824+0000 mon.a (mon.0) 2516 : audit [INF] from='client.? v1:172.21.15.121:0/1247893971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:33:45.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:45 smithi169 bash[19604]: cluster 2024-02-12T16:33:44.188347+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.19 starting backfill to osd.5 from (0'0,0'0] MAX to 737'19641 2024-02-12T16:33:45.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:45.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:45 smithi121 bash[18489]: cluster 2024-02-12T16:33:44.095543+0000 mgr.y (mgr.14146) 1473 : cluster [DBG] pgmap v2094: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s 2024-02-12T16:33:45.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:45 smithi121 bash[18489]: cluster 2024-02-12T16:33:44.168706+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e742: 12 total, 12 up, 12 in 2024-02-12T16:33:45.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:45 smithi121 bash[18489]: cluster 2024-02-12T16:33:44.171649+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.19 starting backfill to osd.1 from (0'0,0'0] MAX to 737'19641 2024-02-12T16:33:45.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:45 smithi121 bash[18489]: audit 2024-02-12T16:33:44.183824+0000 mon.a (mon.0) 2516 : audit [INF] from='client.? v1:172.21.15.121:0/1247893971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:33:45.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:45 smithi121 bash[18489]: cluster 2024-02-12T16:33:44.188347+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.19 starting backfill to osd.5 from (0'0,0'0] MAX to 737'19641 2024-02-12T16:33:45.645 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:45.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:45.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:45.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:45.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:45.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:45.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:46.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:46.066 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:46.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:46.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:46.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:46.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:46.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:46.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:46.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:46 smithi123 bash[19119]: audit 2024-02-12T16:33:45.179429+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? v1:172.21.15.121:0/1247893971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:33:46.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:46 smithi123 bash[19119]: cluster 2024-02-12T16:33:45.179582+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e743: 12 total, 12 up, 12 in 2024-02-12T16:33:46.439 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:46.451 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:46 smithi169 bash[19604]: audit 2024-02-12T16:33:45.179429+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? v1:172.21.15.121:0/1247893971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:33:46.452 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:46 smithi169 bash[19604]: cluster 2024-02-12T16:33:45.179582+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e743: 12 total, 12 up, 12 in 2024-02-12T16:33:46.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:46.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:46.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:46.620 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:46.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:46 smithi121 bash[18489]: audit 2024-02-12T16:33:45.179429+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? v1:172.21.15.121:0/1247893971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:33:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:46 smithi121 bash[18489]: cluster 2024-02-12T16:33:45.179582+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e743: 12 total, 12 up, 12 in 2024-02-12T16:33:46.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:46.761 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:46.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:46.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:46.901 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:46.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:47.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:47.046 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:47.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:47.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T16:33:47.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:47.191 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:47.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:47.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:47.334 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:47.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:47.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:47 smithi121 bash[18489]: cluster 2024-02-12T16:33:46.096352+0000 mgr.y (mgr.14146) 1474 : cluster [DBG] pgmap v2097: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:47 smithi121 bash[18489]: cluster 2024-02-12T16:33:46.195462+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e744: 12 total, 12 up, 12 in 2024-02-12T16:33:47.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:47.478 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:47 smithi123 bash[19119]: cluster 2024-02-12T16:33:46.096352+0000 mgr.y (mgr.14146) 1474 : cluster [DBG] pgmap v2097: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:47.478 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:47 smithi123 bash[19119]: cluster 2024-02-12T16:33:46.195462+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e744: 12 total, 12 up, 12 in 2024-02-12T16:33:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:47.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:47 smithi169 bash[19604]: cluster 2024-02-12T16:33:46.096352+0000 mgr.y (mgr.14146) 1474 : cluster [DBG] pgmap v2097: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:47 smithi169 bash[19604]: cluster 2024-02-12T16:33:46.195462+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e744: 12 total, 12 up, 12 in 2024-02-12T16:33:47.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:47.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:47.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:47.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:47.762 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:47.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:47.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:47.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:33:47.892 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:33:47.896 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:47.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:47.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:33:47.946 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:33:47.969 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:33:47.994 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:33:47.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:48.019 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:33:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:48.045 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:33:48.074 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:33:48.101 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:33:48.130 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:33:48.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:48.154 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:33:48.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.156 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.157 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:33:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:48.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:48.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:48 smithi121 bash[18489]: audit 2024-02-12T16:33:47.842153+0000 mon.a (mon.0) 2520 : audit [DBG] from='client.? v1:172.21.15.121:0/64266414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:48.288 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:48.389 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:33:48.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:48.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:48.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:48 smithi169 bash[19604]: audit 2024-02-12T16:33:47.842153+0000 mon.a (mon.0) 2520 : audit [DBG] from='client.? v1:172.21.15.121:0/64266414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:48.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:48.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:48 smithi123 bash[19119]: audit 2024-02-12T16:33:47.842153+0000 mon.a (mon.0) 2520 : audit [DBG] from='client.? v1:172.21.15.121:0/64266414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:33:48.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:48.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:48.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:48.848 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:48.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:48.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:48.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:48.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:49.094 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:49.129 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:49.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:49.229 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:33:49.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:49.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:49.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:49.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:49 smithi123 bash[19119]: cluster 2024-02-12T16:33:48.097136+0000 mgr.y (mgr.14146) 1475 : cluster [DBG] pgmap v2099: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:49.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:49 smithi123 bash[19119]: cluster 2024-02-12T16:33:48.195103+0000 mon.a (mon.0) 2521 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:49.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:49 smithi123 bash[19119]: audit 2024-02-12T16:33:48.555132+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? v1:172.21.15.121:0/2496767545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:33:49.404 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:49.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:49.507 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:49.544 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:49.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:49 smithi169 bash[19604]: cluster 2024-02-12T16:33:48.097136+0000 mgr.y (mgr.14146) 1475 : cluster [DBG] pgmap v2099: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:49 smithi169 bash[19604]: cluster 2024-02-12T16:33:48.195103+0000 mon.a (mon.0) 2521 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:49 smithi169 bash[19604]: audit 2024-02-12T16:33:48.555132+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? v1:172.21.15.121:0/2496767545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:33:49.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:49 smithi121 bash[18489]: cluster 2024-02-12T16:33:48.097136+0000 mgr.y (mgr.14146) 1475 : cluster [DBG] pgmap v2099: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:49 smithi121 bash[18489]: cluster 2024-02-12T16:33:48.195103+0000 mon.a (mon.0) 2521 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:33:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:49 smithi121 bash[18489]: audit 2024-02-12T16:33:48.555132+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? v1:172.21.15.121:0/2496767545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:33:49.683 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:49.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:49.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:49.819 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:49.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:49.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:49.960 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:49.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:49.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:50.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:50.101 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:50.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:50.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:50.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:50.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:50.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:50.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:50 smithi123 bash[19119]: cluster 2024-02-12T16:33:49.205014+0000 mon.a (mon.0) 2523 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:50.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:50 smithi123 bash[19119]: audit 2024-02-12T16:33:49.218106+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? v1:172.21.15.121:0/2496767545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:33:50.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:50 smithi123 bash[19119]: cluster 2024-02-12T16:33:49.218240+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e745: 12 total, 12 up, 12 in 2024-02-12T16:33:50.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:50.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:50.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:50.524 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:50.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:50 smithi169 bash[19604]: cluster 2024-02-12T16:33:49.205014+0000 mon.a (mon.0) 2523 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:50 smithi169 bash[19604]: audit 2024-02-12T16:33:49.218106+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? v1:172.21.15.121:0/2496767545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:33:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:50 smithi169 bash[19604]: cluster 2024-02-12T16:33:49.218240+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e745: 12 total, 12 up, 12 in 2024-02-12T16:33:50.627 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:50 smithi121 bash[18489]: cluster 2024-02-12T16:33:49.205014+0000 mon.a (mon.0) 2523 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:33:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:50 smithi121 bash[18489]: audit 2024-02-12T16:33:49.218106+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? v1:172.21.15.121:0/2496767545' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:33:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:50 smithi121 bash[18489]: cluster 2024-02-12T16:33:49.218240+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e745: 12 total, 12 up, 12 in 2024-02-12T16:33:50.663 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:50.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:50.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:50.805 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:50.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:50.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:50.947 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:50.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:51.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:51.085 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:51.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:51.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:51.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:51.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:51.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:51.361 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:49.430350+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:49.454603+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:49.944488+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.11 scrub starts 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:49.961509+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.11 scrub ok 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:50.097881+0000 mgr.y (mgr.14146) 1476 : cluster [DBG] pgmap v2101: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:50.232590+0000 osd.2 (osd.2) 101 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:50.291101+0000 osd.2 (osd.2) 102 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:50.672810+0000 osd.9 (osd.9) 123 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:33:51.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:51 smithi121 bash[18489]: cluster 2024-02-12T16:33:50.696483+0000 osd.9 (osd.9) 124 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:33:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:51.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:49.430350+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:49.454603+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:49.944488+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.11 scrub starts 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:49.961509+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.11 scrub ok 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:50.097881+0000 mgr.y (mgr.14146) 1476 : cluster [DBG] pgmap v2101: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:50.232590+0000 osd.2 (osd.2) 101 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:50.291101+0000 osd.2 (osd.2) 102 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:50.672810+0000 osd.9 (osd.9) 123 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:33:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:51 smithi123 bash[19119]: cluster 2024-02-12T16:33:50.696483+0000 osd.9 (osd.9) 124 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:33:51.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:51.502 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:51.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:49.430350+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.0 scrub starts 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:49.454603+0000 osd.0 (osd.0) 121 : cluster [DBG] 6.0 scrub ok 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:49.944488+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.11 scrub starts 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:49.961509+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.11 scrub ok 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:50.097881+0000 mgr.y (mgr.14146) 1476 : cluster [DBG] pgmap v2101: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:50.232590+0000 osd.2 (osd.2) 101 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:50.291101+0000 osd.2 (osd.2) 102 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:50.672810+0000 osd.9 (osd.9) 123 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:33:51.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:51 smithi169 bash[19604]: cluster 2024-02-12T16:33:50.696483+0000 osd.9 (osd.9) 124 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:33:51.605 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:51.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:51.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:51.717 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:33:51.717 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:33:51.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:33:51.717 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161965 5.2 5:4ccda5b4:::benchmark_data_smithi169_84712_object161964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:46.997872+0000", 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 484.71094371499998, 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.212414571, 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161965 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997872+0000", 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997872+0000", 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997869+0000", 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7025000000000001e-05 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997956+0000", 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4990999999999999e-05 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997971+0000", 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3990000000000001e-06 2024-02-12T16:33:51.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:46.997978+0000", 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023957980000000002 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.000374+0000", 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000151667 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.000525+0000", 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000183151 2024-02-12T16:33:51.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.000708+0000", 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010313325 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011022+0000", 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.198999169 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.210021+0000", 2024-02-12T16:33:51.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000177367 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.210198+0000", 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8038e-05 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.210286+0000", 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.212414571 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161971 5.2 5:41152504:::benchmark_data_smithi169_84712_object161970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.006154+0000", 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 484.702661283, 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23144814499999999, 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.724 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161971 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006154+0000", 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006154+0000", 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006152+0000", 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8842999999999999e-05 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006210+0000", 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3706999999999999e-05 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006224+0000", 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.241e-06 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006230+0000", 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6778000000000001e-05 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006327+0000", 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015125800000000001 2024-02-12T16:33:51.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006478+0000", 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018717199999999999 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.006666+0000", 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044819100000000004 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011147+0000", 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.226312452 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237460+0000", 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8177999999999995e-05 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237548+0000", 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4280000000000002e-05 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237602+0000", 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23144814499999999 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161977 5.2 5:4a9c9bc2:::benchmark_data_smithi169_84712_object161976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.011338+0000", 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 484.69747771300001, 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22639947999999999, 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.730 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161977 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011338+0000", 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011338+0000", 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011335+0000", 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5386e-05 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011401+0000", 2024-02-12T16:33:51.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7649999999999999e-05 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011418+0000", 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.605e-06 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011426+0000", 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2783e-05 2024-02-12T16:33:51.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011469+0000", 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013387800000000001 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011603+0000", 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017509700000000001 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.011778+0000", 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047544500000000003 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.016532+0000", 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.221132512 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237665+0000", 2024-02-12T16:33:51.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2134000000000001e-05 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237697+0000", 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0297999999999999e-05 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237737+0000", 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22639947999999999 2024-02-12T16:33:51.736 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161982 5.2 5:43ce8bef:::benchmark_data_smithi169_84712_object161981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.017144+0000", 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 484.69167151300002, 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22072623399999999, 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.737 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161982 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017144+0000", 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017144+0000", 2024-02-12T16:33:51.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017141+0000", 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012736299999999999 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017269+0000", 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5359999999999999e-05 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017284+0000", 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7969999999999997e-05 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017322+0000", 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7315000000000002e-05 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017389+0000", 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014794 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017537+0000", 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017065400000000001 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017708+0000", 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01007368 2024-02-12T16:33:51.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.027782+0000", 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.210005576 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237787+0000", 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6298000000000003e-05 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237833+0000", 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6692e-05 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237870+0000", 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22072623399999999 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161984 5.2 5:490f6ed5:::benchmark_data_smithi169_84712_object161983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.017334+0000", 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 484.691481426, 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22064978299999999, 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161984 2024-02-12T16:33:51.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017334+0000", 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017334+0000", 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017333+0000", 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7771000000000001e-05 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017371+0000", 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7780000000000007e-06 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017378+0000", 2024-02-12T16:33:51.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6789999999999999e-06 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.017385+0000", 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052991150000000001 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.022684+0000", 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000157882 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.022842+0000", 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000183788 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.023026+0000", 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096565990000000001 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.032682+0000", 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20524700800000001 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237930+0000", 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9995e-05 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237960+0000", 2024-02-12T16:33:51.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4318999999999999e-05 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.237984+0000", 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22064978299999999 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.750 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:161989 5.2 5:4ce3b2a3:::benchmark_data_smithi169_84712_object161988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:25:47.024035+0000", 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 484.68478038199999, 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21474912700000001, 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 161989 2024-02-12T16:33:51.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024035+0000", 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024035+0000", 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024032+0000", 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8520999999999998e-05 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024091+0000", 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7348000000000001e-05 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024108+0000", 2024-02-12T16:33:51.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8700000000000003e-06 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.024115+0000", 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037011940000000001 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.027816+0000", 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016248799999999999 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.027979+0000", 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019099899999999999 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.028170+0000", 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046490619999999998 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.032819+0000", 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20586957 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.238688+0000", 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6029000000000002e-05 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.238744+0000", 2024-02-12T16:33:51.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9734000000000003e-05 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:25:47.238784+0000", 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21474912700000001 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:213879 5.2 5:457142c1:::benchmark_data_smithi169_84712_object44489:head [delete] snapc 0=[] ondisk+write+known_if_redirected e594)", 2024-02-12T16:33:51.757 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:39.140072+0000", 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 432.56874326500002, 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15980639399999999, 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 213879 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140072+0000", 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140072+0000", 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140071+0000", 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9540000000000003e-06 2024-02-12T16:33:51.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140076+0000", 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3259999999999998e-06 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140083+0000", 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9320000000000004e-06 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.140088+0000", 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098086890000000006 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.149897+0000", 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000128794 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.150026+0000", 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000161223 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.150187+0000", 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011590903 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.161778+0000", 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.13804018900000001 2024-02-12T16:33:51.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299818+0000", 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5116000000000001e-05 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299853+0000", 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5465000000000001e-05 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299879+0000", 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15980639399999999 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:213882 5.2 5:4b0a2b37:::benchmark_data_smithi169_84712_object44492:head [delete] snapc 0=[] ondisk+write+known_if_redirected e594)", 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:39.143029+0000", 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 432.56578650900002, 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15696279499999999, 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.764 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 213882 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143029+0000", 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143029+0000", 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143027+0000", 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0739999999999993e-06 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143035+0000", 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0524e-05 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143045+0000", 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0979999999999999e-06 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.143051+0000", 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010504138999999999 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.153556+0000", 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8573000000000006e-05 2024-02-12T16:33:51.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.153654+0000", 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9264000000000005e-05 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.153743+0000", 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012003722 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.165747+0000", 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.134178724 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299926+0000", 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9660999999999999e-05 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299956+0000", 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6199999999999999e-05 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:39.299992+0000", 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.15696279499999999 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:229755 5.2 5:40ec6c5a:::benchmark_data_smithi169_84712_object60365:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:53.338179+0000", 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 418.37063651800003, 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32704555499999999, 2024-02-12T16:33:51.770 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 229755 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338179+0000", 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338179+0000", 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338177+0000", 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5910000000000003e-06 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338183+0000", 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7540000000000001e-06 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338192+0000", 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4179999999999999e-06 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338197+0000", 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2754999999999999e-05 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338250+0000", 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000131774 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338381+0000", 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100765 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.338482+0000", 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043336579999999998 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.342816+0000", 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32214183899999999 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.664958+0000", 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015849599999999999 2024-02-12T16:33:51.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.665116+0000", 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108354 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.665224+0000", 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32704555499999999 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:229770 5.2 5:4ed3e4ab:::benchmark_data_smithi169_84712_object60380:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:26:53.353029+0000", 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 418.35578643899998, 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31263463600000002, 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 229770 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353029+0000", 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353029+0000", 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353027+0000", 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5070000000000003e-06 2024-02-12T16:33:51.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353034+0000", 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.449e-05 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353049+0000", 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4459999999999998e-06 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353055+0000", 2024-02-12T16:33:51.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6449000000000002e-05 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353102+0000", 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121657 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353223+0000", 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119777 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.353343+0000", 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003830506 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.357174+0000", 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.30820228500000002 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.665376+0000", 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015195599999999999 2024-02-12T16:33:51.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.665528+0000", 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000135857 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:26:53.665664+0000", 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31263463600000002 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.16095.0:265724 5.2 5:42b60d38:::benchmark_data_smithi169_84712_object96334:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.902013+0000", 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 384.80680198599998, 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35881442800000002, 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.16095", 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265724 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902013+0000", 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902013+0000", 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902011+0000", 2024-02-12T16:33:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7710000000000006e-06 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902019+0000", 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1231e-05 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902030+0000", 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4250000000000003e-06 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902036+0000", 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8464999999999998e-05 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902075+0000", 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0357000000000004e-05 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902145+0000", 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8843000000000004e-05 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0", 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.902224+0000", 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35812087199999998 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.260345+0000", 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031904200000000001 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.260664+0000", 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109026 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.260773+0000", 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4845000000000001e-05 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.260828+0000", 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35881442800000002 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.16095.0:265724 5.2 e604/547 ondisk, result = 0)", 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:27:26.907639+0000", 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 384.80117633600003, 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35270918699999998, 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.907639+0000", 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.907639+0000", 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.907638+0000", 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3930000000000008e-06 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:26.907646+0000", 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34628388300000001 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253930+0000", 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5435999999999998e-05 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.253955+0000", 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063679110000000004 2024-02-12T16:33:51.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.260323+0000", 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1558e-05 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.260335+0000", 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3354e-05 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:27:27.260348+0000", 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35270918699999998 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26257.0:179495 6.3 e679/636 6:c520ae98:::benchmark_data_smithi169_96224_object26659:head v 679'11149, mlcod=679'11146)", 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.752699+0000", 2024-02-12T16:33:51.793 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 142.95611687900001, 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23754165399999999, 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752699+0000", 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752699+0000", 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752698+0000", 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4199999999999996e-06 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752707+0000", 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6110999999999999e-05 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752723+0000", 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5499999999999997e-06 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752731+0000", 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023781200000000001 2024-02-12T16:33:51.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752968+0000", 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6721999999999999e-05 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752995+0000", 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23724506000000001 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.990240+0000", 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23754165399999999 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:211062 6.13 6:c92921d6:::benchmark_data_smithi169_96224_object58226:head [delete] snapc 0=[] ondisk+write+known_if_redirected e699)", 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:04.025440+0000", 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 107.683375015, 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.066421078999999994, 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.798 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 211062 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025440+0000", 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025440+0000", 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:33:51.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025438+0000", 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0450000000000006e-06 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025447+0000", 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4453e-05 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025461+0000", 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8369999999999999e-06 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025468+0000", 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6221000000000003e-05 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025524+0000", 2024-02-12T16:33:51.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013982600000000001 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.025664+0000", 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043273900000000002 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.026097+0000", 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003145428 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.029242+0000", 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7278e-05 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.029260+0000", 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017547799999999999 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.029435+0000", 2024-02-12T16:33:51.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.062281784 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.091717+0000", 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9178000000000002e-05 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.091766+0000", 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5478999999999998e-05 2024-02-12T16:33:51.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:04.091862+0000", 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.066421078999999994 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:263668 6.13 6:cda3866b:::benchmark_data_smithi169_96224_object110832:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:48.094731+0000", 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 63.614084566999999, 2024-02-12T16:33:51.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.482651414, 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 263668 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094731+0000", 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094731+0000", 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094729+0000", 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8700000000000003e-06 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094736+0000", 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1467999999999999e-05 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094747+0000", 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5339999999999996e-06 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094754+0000", 2024-02-12T16:33:51.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8145000000000001e-05 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094802+0000", 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014538100000000001 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094947+0000", 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00055281199999999996 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.095500+0000", 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002824423 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.098324+0000", 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076364900000000003 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.099088+0000", 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2613000000000001e-05 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.099121+0000", 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47808740199999999 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.577208+0000", 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4317999999999997e-05 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.577262+0000", 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119839 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.577382+0000", 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.482651414 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264237 6.13 6:cc27572d:::benchmark_data_smithi169_96224_object111401:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.092307+0000", 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.616508519999996, 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35619952399999999, 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264237 2024-02-12T16:33:51.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092307+0000", 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092307+0000", 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:33:51.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092305+0000", 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.781e-06 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092312+0000", 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1891e-05 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092324+0000", 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.22e-06 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092329+0000", 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9597e-05 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092379+0000", 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109811 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092488+0000", 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050793500000000005 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092996+0000", 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00232073 2024-02-12T16:33:51.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.095317+0000", 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00052685299999999998 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.095844+0000", 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014834920000000001 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.097327+0000", 2024-02-12T16:33:51.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35100952099999999 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.448337+0000", 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3976e-05 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.448391+0000", 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011561300000000001 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.448506+0000", 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35619952399999999 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:267579 6.13 6:ce79d9f5:::benchmark_data_smithi169_96224_object114743:head [delete] snapc 0=[] ondisk+write+known_if_redirected e715)", 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:52.266294+0000", 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.442521798000001, 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.078541253000000005, 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.820 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 267579 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266294+0000", 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266294+0000", 2024-02-12T16:33:51.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266291+0000", 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0999999999999993e-06 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266300+0000", 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2106999999999999e-05 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266312+0000", 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9619999999999998e-06 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266318+0000", 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3980999999999999e-05 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266352+0000", 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018448499999999999 2024-02-12T16:33:51.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.266537+0000", 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059727900000000004 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.267134+0000", 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003566081 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270700+0000", 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047976019999999998 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.275498+0000", 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025116309999999998 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.278009+0000", 2024-02-12T16:33:51.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.066674251000000004 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.344683+0000", 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3029999999999997e-05 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.344756+0000", 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8481999999999998e-05 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.344835+0000", 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.078541253000000005 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:267582 6.13 6:ccbf01ed:::benchmark_data_smithi169_96224_object114746:head [delete] snapc 0=[] ondisk+write+known_if_redirected e715)", 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:52.270752+0000", 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.438063472000003, 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.085635978000000001, 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.827 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 267582 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270752+0000", 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270752+0000", 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270750+0000", 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1860000000000003e-06 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270757+0000", 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8130000000000007e-06 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270767+0000", 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9000000000000003e-06 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270773+0000", 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0806000000000002e-05 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270814+0000", 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120283 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.270934+0000", 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053645699999999995 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.271470+0000", 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033263279999999999 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.274797+0000", 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00082283499999999995 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.275619+0000", 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004690975 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.280310+0000", 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.075853218 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.356164+0000", 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1884000000000006e-05 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.356225+0000", 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016247399999999999 2024-02-12T16:33:51.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.356388+0000", 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.085635978000000001 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:267598 6.13 6:c8513d23:::benchmark_data_smithi169_96224_object114762:head [delete] snapc 0=[] ondisk+write+known_if_redirected e715)", 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:52.285094+0000", 2024-02-12T16:33:51.834 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.423721692000001, 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.076338254999999994, 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 267598 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285094+0000", 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285094+0000", 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285091+0000", 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2579999999999995e-06 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285100+0000", 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2799999999999999e-05 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285113+0000", 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1580000000000002e-06 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285119+0000", 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4011999999999998e-05 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285163+0000", 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001022 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285265+0000", 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050017400000000004 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.285765+0000", 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022994410000000002 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.288065+0000", 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000598413 2024-02-12T16:33:51.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.288663+0000", 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050377799999999997 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.289167+0000", 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.072050437999999994 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.361217+0000", 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0827000000000005e-05 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.361288+0000", 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014405599999999999 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:52.361432+0000", 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.076338254999999994 2024-02-12T16:33:51.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:273379 6.13 6:c8d70904:::benchmark_data_smithi169_96224_object120543:head [delete] snapc 0=[] ondisk+write+known_if_redirected e715)", 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:57.464649+0000", 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.244166127, 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.057521752000000002, 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:33:51.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 273379 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464649+0000", 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464649+0000", 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464647+0000", 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1710000000000002e-06 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:33:51.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464655+0000", 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.399e-05 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464669+0000", 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5919999999999997e-06 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464676+0000", 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3355000000000001e-05 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464719+0000", 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000134728 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.464854+0000", 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058469200000000005 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.465438+0000", 2024-02-12T16:33:51.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035289890000000002 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.468967+0000", 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029835159999999999 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.471951+0000", 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00067917499999999996 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.472630+0000", 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.049331238999999999 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.521961+0000", 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2218e-05 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.522014+0000", 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015754899999999999 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:57.522171+0000", 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.057521752000000002 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:33:51.849 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:33:51.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:51.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:33:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:51.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:51.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:52.022 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:52.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:52.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:52.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:52.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_4' removed 2024-02-12T16:33:52.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:52.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:52 smithi123 bash[19119]: cluster 2024-02-12T16:33:50.921455+0000 osd.7 (osd.7) 178 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:33:52.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:52 smithi123 bash[19119]: cluster 2024-02-12T16:33:50.997762+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:33:52.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:52 smithi123 bash[19119]: cluster 2024-02-12T16:33:51.277094+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-02-12T16:33:52.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:52 smithi123 bash[19119]: cluster 2024-02-12T16:33:51.291541+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-02-12T16:33:52.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:52 smithi123 bash[19119]: cluster 2024-02-12T16:33:51.322114+0000 osd.8 (osd.8) 77 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:33:52.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:52 smithi123 bash[19119]: cluster 2024-02-12T16:33:51.394965+0000 osd.8 (osd.8) 78 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:33:52.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:52 smithi123 bash[19119]: audit 2024-02-12T16:33:51.519442+0000 mon.a (mon.0) 2526 : audit [INF] from='client.? v1:172.21.15.121:0/1848786735' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:33:52.343 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:52.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:52.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:52 smithi121 bash[18489]: cluster 2024-02-12T16:33:50.921455+0000 osd.7 (osd.7) 178 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:33:52.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:52 smithi121 bash[18489]: cluster 2024-02-12T16:33:50.997762+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:33:52.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:52 smithi121 bash[18489]: cluster 2024-02-12T16:33:51.277094+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-02-12T16:33:52.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:52 smithi121 bash[18489]: cluster 2024-02-12T16:33:51.291541+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-02-12T16:33:52.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:52 smithi121 bash[18489]: cluster 2024-02-12T16:33:51.322114+0000 osd.8 (osd.8) 77 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:33:52.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:52 smithi121 bash[18489]: cluster 2024-02-12T16:33:51.394965+0000 osd.8 (osd.8) 78 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:33:52.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:52 smithi121 bash[18489]: audit 2024-02-12T16:33:51.519442+0000 mon.a (mon.0) 2526 : audit [INF] from='client.? v1:172.21.15.121:0/1848786735' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:33:52.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:52.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:33:52.482 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:52.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:52 smithi169 bash[19604]: cluster 2024-02-12T16:33:50.921455+0000 osd.7 (osd.7) 178 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:33:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:52 smithi169 bash[19604]: cluster 2024-02-12T16:33:50.997762+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:33:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:52 smithi169 bash[19604]: cluster 2024-02-12T16:33:51.277094+0000 osd.2 (osd.2) 103 : cluster [DBG] 6.f scrub starts 2024-02-12T16:33:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:52 smithi169 bash[19604]: cluster 2024-02-12T16:33:51.291541+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2024-02-12T16:33:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:52 smithi169 bash[19604]: cluster 2024-02-12T16:33:51.322114+0000 osd.8 (osd.8) 77 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:33:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:52 smithi169 bash[19604]: cluster 2024-02-12T16:33:51.394965+0000 osd.8 (osd.8) 78 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:33:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:52 smithi169 bash[19604]: audit 2024-02-12T16:33:51.519442+0000 mon.a (mon.0) 2526 : audit [INF] from='client.? v1:172.21.15.121:0/1848786735' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:33:52.585 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:52.622 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:52.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:52.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:52.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:52.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:52.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:33:52.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:52.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:53.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:53.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:53.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:53.108 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:33:53.109 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:33:53.109 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-02-12T16:33:53.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_5 16 2024-02-12T16:33:53.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:53.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:53.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:51.937244+0000 osd.7 (osd.7) 180 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.036060+0000 osd.7 (osd.7) 181 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.042209+0000 osd.11 (osd.11) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.042295+0000 osd.11 (osd.11) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.098480+0000 mgr.y (mgr.14146) 1477 : cluster [DBG] pgmap v2102: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: audit 2024-02-12T16:33:52.220300+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? v1:172.21.15.121:0/1848786735' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.220434+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e746: 12 total, 12 up, 12 in 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.295015+0000 osd.4 (osd.4) 122 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.408835+0000 osd.4 (osd.4) 123 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.752177+0000 osd.10 (osd.10) 146 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:33:53.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:53 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.779728+0000 osd.10 (osd.10) 147 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:33:53.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:53.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:51.937244+0000 osd.7 (osd.7) 180 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.036060+0000 osd.7 (osd.7) 181 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.042209+0000 osd.11 (osd.11) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.042295+0000 osd.11 (osd.11) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.098480+0000 mgr.y (mgr.14146) 1477 : cluster [DBG] pgmap v2102: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: audit 2024-02-12T16:33:52.220300+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? v1:172.21.15.121:0/1848786735' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.220434+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e746: 12 total, 12 up, 12 in 2024-02-12T16:33:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.295015+0000 osd.4 (osd.4) 122 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:33:53.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.408835+0000 osd.4 (osd.4) 123 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:33:53.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.752177+0000 osd.10 (osd.10) 146 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:33:53.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:53 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.779728+0000 osd.10 (osd.10) 147 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:33:53.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:53.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:53.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.590 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:53.604 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:53.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:51.937244+0000 osd.7 (osd.7) 180 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.036060+0000 osd.7 (osd.7) 181 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.042209+0000 osd.11 (osd.11) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.042295+0000 osd.11 (osd.11) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.098480+0000 mgr.y (mgr.14146) 1477 : cluster [DBG] pgmap v2102: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 16 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: audit 2024-02-12T16:33:52.220300+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? v1:172.21.15.121:0/1848786735' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.220434+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e746: 12 total, 12 up, 12 in 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.295015+0000 osd.4 (osd.4) 122 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.408835+0000 osd.4 (osd.4) 123 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.752177+0000 osd.10 (osd.10) 146 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:33:53.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:53 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.779728+0000 osd.10 (osd.10) 147 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:33:53.708 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:53.738 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:53.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:53.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:53.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:53.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:53.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:54.022 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:33:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:54.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:33:54.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:54.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:54.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:54.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:54.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.932792+0000 osd.3 (osd.3) 60 : cluster [DBG] 1.c scrub starts 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:52.998459+0000 osd.3 (osd.3) 61 : cluster [DBG] 1.c scrub ok 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.301900+0000 osd.4 (osd.4) 124 : cluster [DBG] 1.e scrub starts 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.343820+0000 osd.0 (osd.0) 122 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.367127+0000 osd.1 (osd.1) 110 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.411728+0000 osd.4 (osd.4) 125 : cluster [DBG] 1.e scrub ok 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.422047+0000 osd.0 (osd.0) 123 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.526799+0000 osd.1 (osd.1) 111 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.784622+0000 osd.10 (osd.10) 148 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:33:54.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:54 smithi169 bash[19604]: cluster 2024-02-12T16:33:53.839181+0000 osd.10 (osd.10) 149 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:33:54.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:54.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:54.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.932792+0000 osd.3 (osd.3) 60 : cluster [DBG] 1.c scrub starts 2024-02-12T16:33:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:52.998459+0000 osd.3 (osd.3) 61 : cluster [DBG] 1.c scrub ok 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.301900+0000 osd.4 (osd.4) 124 : cluster [DBG] 1.e scrub starts 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.343820+0000 osd.0 (osd.0) 122 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.367127+0000 osd.1 (osd.1) 110 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.411728+0000 osd.4 (osd.4) 125 : cluster [DBG] 1.e scrub ok 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.422047+0000 osd.0 (osd.0) 123 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.526799+0000 osd.1 (osd.1) 111 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.784622+0000 osd.10 (osd.10) 148 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:33:54.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:54 smithi121 bash[18489]: cluster 2024-02-12T16:33:53.839181+0000 osd.10 (osd.10) 149 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:33:54.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:54.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:54.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:54.565 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.932792+0000 osd.3 (osd.3) 60 : cluster [DBG] 1.c scrub starts 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:52.998459+0000 osd.3 (osd.3) 61 : cluster [DBG] 1.c scrub ok 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.301900+0000 osd.4 (osd.4) 124 : cluster [DBG] 1.e scrub starts 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.343820+0000 osd.0 (osd.0) 122 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.367127+0000 osd.1 (osd.1) 110 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.411728+0000 osd.4 (osd.4) 125 : cluster [DBG] 1.e scrub ok 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.422047+0000 osd.0 (osd.0) 123 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.526799+0000 osd.1 (osd.1) 111 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.784622+0000 osd.10 (osd.10) 148 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:33:54.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:54 smithi123 bash[19119]: cluster 2024-02-12T16:33:53.839181+0000 osd.10 (osd.10) 149 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:33:54.591 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:54.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:54.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:54.722 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:54.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:54.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:54.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:54.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:54.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:54.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:55.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:55.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:55.152 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:55.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:55.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:55.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:55 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.098944+0000 mgr.y (mgr.14146) 1478 : cluster [DBG] pgmap v2104: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:55.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:55 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.227226+0000 mon.a (mon.0) 2529 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:33:55.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:55 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.650641+0000 osd.9 (osd.9) 125 : cluster [DBG] 1.a scrub starts 2024-02-12T16:33:55.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:55 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.694425+0000 osd.9 (osd.9) 126 : cluster [DBG] 1.a scrub ok 2024-02-12T16:33:55.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:55 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.756111+0000 osd.10 (osd.10) 150 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:33:55.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:55 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.812852+0000 osd.10 (osd.10) 151 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:33:55.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:55 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.098944+0000 mgr.y (mgr.14146) 1478 : cluster [DBG] pgmap v2104: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:55 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.227226+0000 mon.a (mon.0) 2529 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:33:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:55 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.650641+0000 osd.9 (osd.9) 125 : cluster [DBG] 1.a scrub starts 2024-02-12T16:33:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:55 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.694425+0000 osd.9 (osd.9) 126 : cluster [DBG] 1.a scrub ok 2024-02-12T16:33:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:55 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.756111+0000 osd.10 (osd.10) 150 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:33:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:55 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.812852+0000 osd.10 (osd.10) 151 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:33:55.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:55.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:55 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.098944+0000 mgr.y (mgr.14146) 1478 : cluster [DBG] pgmap v2104: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:55 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.227226+0000 mon.a (mon.0) 2529 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:33:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:55 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.650641+0000 osd.9 (osd.9) 125 : cluster [DBG] 1.a scrub starts 2024-02-12T16:33:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:55 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.694425+0000 osd.9 (osd.9) 126 : cluster [DBG] 1.a scrub ok 2024-02-12T16:33:55.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:55 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.756111+0000 osd.10 (osd.10) 150 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:33:55.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:55 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.812852+0000 osd.10 (osd.10) 151 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:33:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:55.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:55.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:55.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:55.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:55.982 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:55.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:56.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:56.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:56.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:56.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:56.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:56.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:56.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:56.406 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:56.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:56.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.313499+0000 osd.11 (osd.11) 162 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.316665+0000 osd.11 (osd.11) 163 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.701343+0000 osd.5 (osd.5) 147 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.719826+0000 osd.5 (osd.5) 148 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.885323+0000 osd.7 (osd.7) 182 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:54.945888+0000 osd.7 (osd.7) 183 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.311116+0000 osd.1 (osd.1) 112 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.454570+0000 osd.1 (osd.1) 113 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.745945+0000 osd.10 (osd.10) 152 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.785149+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e747: 12 total, 12 up, 12 in 2024-02-12T16:33:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:56 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.831728+0000 osd.10 (osd.10) 153 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:33:56.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.313499+0000 osd.11 (osd.11) 162 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.316665+0000 osd.11 (osd.11) 163 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.701343+0000 osd.5 (osd.5) 147 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.719826+0000 osd.5 (osd.5) 148 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.885323+0000 osd.7 (osd.7) 182 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:54.945888+0000 osd.7 (osd.7) 183 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.311116+0000 osd.1 (osd.1) 112 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.454570+0000 osd.1 (osd.1) 113 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:33:56.523 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.745945+0000 osd.10 (osd.10) 152 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:33:56.524 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.785149+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e747: 12 total, 12 up, 12 in 2024-02-12T16:33:56.524 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:56 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.831728+0000 osd.10 (osd.10) 153 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:33:56.546 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.313499+0000 osd.11 (osd.11) 162 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:33:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.316665+0000 osd.11 (osd.11) 163 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:33:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.701343+0000 osd.5 (osd.5) 147 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.719826+0000 osd.5 (osd.5) 148 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.885323+0000 osd.7 (osd.7) 182 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:54.945888+0000 osd.7 (osd.7) 183 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.311116+0000 osd.1 (osd.1) 112 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.454570+0000 osd.1 (osd.1) 113 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.745945+0000 osd.10 (osd.10) 152 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.785149+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e747: 12 total, 12 up, 12 in 2024-02-12T16:33:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:56 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.831728+0000 osd.10 (osd.10) 153 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:33:56.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:56.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:56.641 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:56.641 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:33:56.651 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:56.690 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:56.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:56.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:56.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:56.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:56.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:56.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:56.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:57.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:57.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:33:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:57.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:33:57.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:57.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:57.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:57.264 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:33:57.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:57.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:57.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:57.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:57 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.849879+0000 osd.7 (osd.7) 184 : cluster [DBG] 1.d scrub starts 2024-02-12T16:33:57.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:57 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.901003+0000 osd.6 (osd.6) 140 : cluster [DBG] 1.f scrub starts 2024-02-12T16:33:57.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:57 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.962369+0000 osd.7 (osd.7) 185 : cluster [DBG] 1.d scrub ok 2024-02-12T16:33:57.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:57 smithi169 bash[19604]: cluster 2024-02-12T16:33:55.988980+0000 osd.6 (osd.6) 141 : cluster [DBG] 1.f scrub ok 2024-02-12T16:33:57.521 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:57 smithi169 bash[19604]: cluster 2024-02-12T16:33:56.099359+0000 mgr.y (mgr.14146) 1479 : cluster [DBG] pgmap v2106: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:57.522 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:57 smithi169 bash[19604]: cluster 2024-02-12T16:33:56.283244+0000 osd.11 (osd.11) 164 : cluster [DBG] 1.b scrub starts 2024-02-12T16:33:57.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:57.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:57 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.849879+0000 osd.7 (osd.7) 184 : cluster [DBG] 1.d scrub starts 2024-02-12T16:33:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:57 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.901003+0000 osd.6 (osd.6) 140 : cluster [DBG] 1.f scrub starts 2024-02-12T16:33:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:57 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.962369+0000 osd.7 (osd.7) 185 : cluster [DBG] 1.d scrub ok 2024-02-12T16:33:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:57 smithi123 bash[19119]: cluster 2024-02-12T16:33:55.988980+0000 osd.6 (osd.6) 141 : cluster [DBG] 1.f scrub ok 2024-02-12T16:33:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:57 smithi123 bash[19119]: cluster 2024-02-12T16:33:56.099359+0000 mgr.y (mgr.14146) 1479 : cluster [DBG] pgmap v2106: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:57 smithi123 bash[19119]: cluster 2024-02-12T16:33:56.283244+0000 osd.11 (osd.11) 164 : cluster [DBG] 1.b scrub starts 2024-02-12T16:33:57.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:57.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:57 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.849879+0000 osd.7 (osd.7) 184 : cluster [DBG] 1.d scrub starts 2024-02-12T16:33:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:57 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.901003+0000 osd.6 (osd.6) 140 : cluster [DBG] 1.f scrub starts 2024-02-12T16:33:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:57 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.962369+0000 osd.7 (osd.7) 185 : cluster [DBG] 1.d scrub ok 2024-02-12T16:33:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:57 smithi121 bash[18489]: cluster 2024-02-12T16:33:55.988980+0000 osd.6 (osd.6) 141 : cluster [DBG] 1.f scrub ok 2024-02-12T16:33:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:57 smithi121 bash[18489]: cluster 2024-02-12T16:33:56.099359+0000 mgr.y (mgr.14146) 1479 : cluster [DBG] pgmap v2106: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:57 smithi121 bash[18489]: cluster 2024-02-12T16:33:56.283244+0000 osd.11 (osd.11) 164 : cluster [DBG] 1.b scrub starts 2024-02-12T16:33:57.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:57.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:57.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:57.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:57.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:57.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:57.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:33:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:57.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:33:58.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:58.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:58.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:58.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:33:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:58.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:33:58.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:58 smithi121 bash[18489]: cluster 2024-02-12T16:33:56.410785+0000 osd.11 (osd.11) 165 : cluster [DBG] 1.b scrub ok 2024-02-12T16:33:58.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:58 smithi121 bash[18489]: cluster 2024-02-12T16:33:57.040391+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:58.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:58 smithi121 bash[18489]: cluster 2024-02-12T16:33:57.040497+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:58.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:58 smithi121 bash[18489]: cluster 2024-02-12T16:33:57.794459+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e748: 12 total, 12 up, 12 in 2024-02-12T16:33:58.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:58 smithi121 bash[18489]: audit 2024-02-12T16:33:58.181620+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? v1:172.21.15.121:0/3531332231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:33:58.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:58 smithi123 bash[19119]: cluster 2024-02-12T16:33:56.410785+0000 osd.11 (osd.11) 165 : cluster [DBG] 1.b scrub ok 2024-02-12T16:33:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:58 smithi123 bash[19119]: cluster 2024-02-12T16:33:57.040391+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:58 smithi123 bash[19119]: cluster 2024-02-12T16:33:57.040497+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:58 smithi123 bash[19119]: cluster 2024-02-12T16:33:57.794459+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e748: 12 total, 12 up, 12 in 2024-02-12T16:33:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:58 smithi123 bash[19119]: audit 2024-02-12T16:33:58.181620+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? v1:172.21.15.121:0/3531332231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:33:58.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:33:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:58.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:33:58.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:58.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:58 smithi169 bash[19604]: cluster 2024-02-12T16:33:56.410785+0000 osd.11 (osd.11) 165 : cluster [DBG] 1.b scrub ok 2024-02-12T16:33:58.510 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:58 smithi169 bash[19604]: cluster 2024-02-12T16:33:57.040391+0000 osd.6 (osd.6) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:33:58.510 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:58 smithi169 bash[19604]: cluster 2024-02-12T16:33:57.040497+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:33:58.510 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:58 smithi169 bash[19604]: cluster 2024-02-12T16:33:57.794459+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e748: 12 total, 12 up, 12 in 2024-02-12T16:33:58.510 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:58 smithi169 bash[19604]: audit 2024-02-12T16:33:58.181620+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? v1:172.21.15.121:0/3531332231' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:33:58.534 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:33:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:58.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:33:58.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:58.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:58.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:58.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:58.815 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T16:33:58.823 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:33:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:58.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:33:58.926 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:58.957 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:33:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:58.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:33:59.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:59.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:33:59.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:59.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:33:59.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:59.240 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:59.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:59.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:59.384 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:59.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:33:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:59 smithi123 bash[19119]: cluster 2024-02-12T16:33:58.099917+0000 mgr.y (mgr.14146) 1480 : cluster [DBG] pgmap v2108: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:59.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:59 smithi123 bash[19119]: audit 2024-02-12T16:33:58.804692+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? v1:172.21.15.121:0/3531332231' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:33:59.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:33:59 smithi123 bash[19119]: cluster 2024-02-12T16:33:58.804809+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e749: 12 total, 12 up, 12 in 2024-02-12T16:33:59.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:59.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:59.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:59 smithi169 bash[19604]: cluster 2024-02-12T16:33:58.099917+0000 mgr.y (mgr.14146) 1480 : cluster [DBG] pgmap v2108: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:59 smithi169 bash[19604]: audit 2024-02-12T16:33:58.804692+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? v1:172.21.15.121:0/3531332231' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:33:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:33:59 smithi169 bash[19604]: cluster 2024-02-12T16:33:58.804809+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e749: 12 total, 12 up, 12 in 2024-02-12T16:33:59.596 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:59 smithi121 bash[18489]: cluster 2024-02-12T16:33:58.099917+0000 mgr.y (mgr.14146) 1480 : cluster [DBG] pgmap v2108: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:33:59.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:59 smithi121 bash[18489]: audit 2024-02-12T16:33:58.804692+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? v1:172.21.15.121:0/3531332231' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:33:59.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:33:59 smithi121 bash[18489]: cluster 2024-02-12T16:33:58.804809+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e749: 12 total, 12 up, 12 in 2024-02-12T16:33:59.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:59.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:33:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:59.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:33:59.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:59.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:33:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:59.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:33:59.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:59.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:33:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:33:59.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:00.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:00.072 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:00.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:00.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:00.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:00.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:00.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:00.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:00.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:00.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:00 smithi123 bash[19119]: cluster 2024-02-12T16:33:59.287710+0000 osd.8 (osd.8) 79 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:34:00.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:00 smithi123 bash[19119]: cluster 2024-02-12T16:33:59.373700+0000 osd.8 (osd.8) 80 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:34:00.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:00.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:00.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:00 smithi169 bash[19604]: cluster 2024-02-12T16:33:59.287710+0000 osd.8 (osd.8) 79 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:34:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:00 smithi169 bash[19604]: cluster 2024-02-12T16:33:59.373700+0000 osd.8 (osd.8) 80 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:34:00.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:00.643 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:00.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:00.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:00.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:00 smithi121 bash[18489]: cluster 2024-02-12T16:33:59.287710+0000 osd.8 (osd.8) 79 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:34:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:00 smithi121 bash[18489]: cluster 2024-02-12T16:33:59.373700+0000 osd.8 (osd.8) 80 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:34:00.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:00.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T16:34:00.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:00.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:00.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:00.906 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:00.923 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:00.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:00.986 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:34:00.986 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:34:00.986 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:34:00.986 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:34:01.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:01.054 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:01.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:01.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:01.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:01.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:01.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:01.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:01 smithi123 bash[19119]: cluster 2024-02-12T16:34:00.100537+0000 mgr.y (mgr.14146) 1481 : cluster [DBG] pgmap v2110: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:01.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:01 smithi123 bash[19119]: cluster 2024-02-12T16:34:00.249089+0000 osd.8 (osd.8) 81 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:34:01.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:01 smithi123 bash[19119]: cluster 2024-02-12T16:34:00.252730+0000 osd.8 (osd.8) 82 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:34:01.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:01 smithi123 bash[19119]: cluster 2024-02-12T16:34:00.389807+0000 osd.1 (osd.1) 114 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:34:01.313 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:01 smithi123 bash[19119]: cluster 2024-02-12T16:34:00.490607+0000 osd.1 (osd.1) 115 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:34:01.333 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:01.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:01.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:01.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:01.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:01 smithi169 bash[19604]: cluster 2024-02-12T16:34:00.100537+0000 mgr.y (mgr.14146) 1481 : cluster [DBG] pgmap v2110: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:01 smithi169 bash[19604]: cluster 2024-02-12T16:34:00.249089+0000 osd.8 (osd.8) 81 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:34:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:01 smithi169 bash[19604]: cluster 2024-02-12T16:34:00.252730+0000 osd.8 (osd.8) 82 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:34:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:01 smithi169 bash[19604]: cluster 2024-02-12T16:34:00.389807+0000 osd.1 (osd.1) 114 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:34:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:01 smithi169 bash[19604]: cluster 2024-02-12T16:34:00.490607+0000 osd.1 (osd.1) 115 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:34:01.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:01.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:01.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:34:01.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:01.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:01.771 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:01 smithi121 bash[18489]: cluster 2024-02-12T16:34:00.100537+0000 mgr.y (mgr.14146) 1481 : cluster [DBG] pgmap v2110: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:01.771 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:01 smithi121 bash[18489]: cluster 2024-02-12T16:34:00.249089+0000 osd.8 (osd.8) 81 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:34:01.771 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:01 smithi121 bash[18489]: cluster 2024-02-12T16:34:00.252730+0000 osd.8 (osd.8) 82 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:34:01.771 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:01 smithi121 bash[18489]: cluster 2024-02-12T16:34:00.389807+0000 osd.1 (osd.1) 114 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:34:01.771 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:01 smithi121 bash[18489]: cluster 2024-02-12T16:34:00.490607+0000 osd.1 (osd.1) 115 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:34:01.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:01.902 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:01.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:02.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:02.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:02.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:02.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:02.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:02.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:02.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:02 smithi169 bash[19604]: cluster 2024-02-12T16:34:00.822992+0000 osd.5 (osd.5) 149 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:34:02.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:02 smithi169 bash[19604]: cluster 2024-02-12T16:34:00.913103+0000 osd.5 (osd.5) 150 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:34:02.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:02 smithi169 bash[19604]: cluster 2024-02-12T16:34:01.623631+0000 osd.9 (osd.9) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:02.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:02 smithi169 bash[19604]: cluster 2024-02-12T16:34:01.623725+0000 osd.9 (osd.9) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:02.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:02.333 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:02.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:02.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:02 smithi121 bash[18489]: cluster 2024-02-12T16:34:00.822992+0000 osd.5 (osd.5) 149 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:34:02.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:02 smithi121 bash[18489]: cluster 2024-02-12T16:34:00.913103+0000 osd.5 (osd.5) 150 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:34:02.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:02 smithi121 bash[18489]: cluster 2024-02-12T16:34:01.623631+0000 osd.9 (osd.9) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:02.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:02 smithi121 bash[18489]: cluster 2024-02-12T16:34:01.623725+0000 osd.9 (osd.9) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:02.437 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:02.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:02.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:02.599 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:34:02.610 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:02.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:02.620 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:34:02.639 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:34:02.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:02 smithi123 bash[19119]: cluster 2024-02-12T16:34:00.822992+0000 osd.5 (osd.5) 149 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:34:02.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:02 smithi123 bash[19119]: cluster 2024-02-12T16:34:00.913103+0000 osd.5 (osd.5) 150 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:34:02.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:02 smithi123 bash[19119]: cluster 2024-02-12T16:34:01.623631+0000 osd.9 (osd.9) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:02.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:02 smithi123 bash[19119]: cluster 2024-02-12T16:34:01.623725+0000 osd.9 (osd.9) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:02.661 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:34:02.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:34:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:34:02.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:02.737 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:34:02.749 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:02.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:02.761 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:34:02.786 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:34:02.812 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:34:02.844 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:34:02.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:02.873 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:34:02.875 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.875 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.875 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.875 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.875 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.875 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.876 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.876 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.876 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.876 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.876 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.876 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:34:02.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:02.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:02.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:02.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:03.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:03.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:03.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:03.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:03.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:34:03.298 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:03.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:03.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:03.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:03 smithi121 bash[18489]: cluster 2024-02-12T16:34:02.101286+0000 mgr.y (mgr.14146) 1482 : cluster [DBG] pgmap v2111: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:03.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:03 smithi121 bash[18489]: cluster 2024-02-12T16:34:02.274966+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e750: 12 total, 12 up, 12 in 2024-02-12T16:34:03.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:03 smithi121 bash[18489]: cluster 2024-02-12T16:34:02.279074+0000 osd.11 (osd.11) 166 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:34:03.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:03 smithi121 bash[18489]: audit 2024-02-12T16:34:02.570106+0000 mon.a (mon.0) 2536 : audit [DBG] from='client.? v1:172.21.15.121:0/2032857795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:03.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:03 smithi121 bash[18489]: audit 2024-02-12T16:34:03.203470+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? v1:172.21.15.121:0/1134891770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:34:03.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:03.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:03.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:03 smithi169 bash[19604]: cluster 2024-02-12T16:34:02.101286+0000 mgr.y (mgr.14146) 1482 : cluster [DBG] pgmap v2111: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:03 smithi169 bash[19604]: cluster 2024-02-12T16:34:02.274966+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e750: 12 total, 12 up, 12 in 2024-02-12T16:34:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:03 smithi169 bash[19604]: cluster 2024-02-12T16:34:02.279074+0000 osd.11 (osd.11) 166 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:34:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:03 smithi169 bash[19604]: audit 2024-02-12T16:34:02.570106+0000 mon.a (mon.0) 2536 : audit [DBG] from='client.? v1:172.21.15.121:0/2032857795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:03 smithi169 bash[19604]: audit 2024-02-12T16:34:03.203470+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? v1:172.21.15.121:0/1134891770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:34:03.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:03.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:03.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:03 smithi123 bash[19119]: cluster 2024-02-12T16:34:02.101286+0000 mgr.y (mgr.14146) 1482 : cluster [DBG] pgmap v2111: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:03.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:03 smithi123 bash[19119]: cluster 2024-02-12T16:34:02.274966+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e750: 12 total, 12 up, 12 in 2024-02-12T16:34:03.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:03 smithi123 bash[19119]: cluster 2024-02-12T16:34:02.279074+0000 osd.11 (osd.11) 166 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:34:03.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:03 smithi123 bash[19119]: audit 2024-02-12T16:34:02.570106+0000 mon.a (mon.0) 2536 : audit [DBG] from='client.? v1:172.21.15.121:0/2032857795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:03.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:03 smithi123 bash[19119]: audit 2024-02-12T16:34:03.203470+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? v1:172.21.15.121:0/1134891770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:34:03.684 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:03.721 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:03.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:03.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:03.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:03.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:03.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:04.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:04.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:04.141 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:04.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:04.283 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:04.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:04 smithi123 bash[19119]: cluster 2024-02-12T16:34:02.317567+0000 osd.11 (osd.11) 167 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:34:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:04 smithi123 bash[19119]: cluster 2024-02-12T16:34:02.875440+0000 osd.2 (osd.2) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:04 smithi123 bash[19119]: cluster 2024-02-12T16:34:02.875535+0000 osd.2 (osd.2) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:04 smithi123 bash[19119]: cluster 2024-02-12T16:34:03.269246+0000 mon.a (mon.0) 2538 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:34:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:04 smithi123 bash[19119]: cluster 2024-02-12T16:34:03.269277+0000 mon.a (mon.0) 2539 : cluster [INF] Cluster is now healthy 2024-02-12T16:34:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:04 smithi123 bash[19119]: audit 2024-02-12T16:34:03.282878+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? v1:172.21.15.121:0/1134891770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:34:04.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:04 smithi123 bash[19119]: cluster 2024-02-12T16:34:03.283049+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e751: 12 total, 12 up, 12 in 2024-02-12T16:34:04.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:04.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:04 smithi169 bash[19604]: cluster 2024-02-12T16:34:02.317567+0000 osd.11 (osd.11) 167 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:34:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:04 smithi169 bash[19604]: cluster 2024-02-12T16:34:02.875440+0000 osd.2 (osd.2) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:04 smithi169 bash[19604]: cluster 2024-02-12T16:34:02.875535+0000 osd.2 (osd.2) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:04 smithi169 bash[19604]: cluster 2024-02-12T16:34:03.269246+0000 mon.a (mon.0) 2538 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:34:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:04 smithi169 bash[19604]: cluster 2024-02-12T16:34:03.269277+0000 mon.a (mon.0) 2539 : cluster [INF] Cluster is now healthy 2024-02-12T16:34:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:04 smithi169 bash[19604]: audit 2024-02-12T16:34:03.282878+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? v1:172.21.15.121:0/1134891770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:34:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:04 smithi169 bash[19604]: cluster 2024-02-12T16:34:03.283049+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e751: 12 total, 12 up, 12 in 2024-02-12T16:34:04.566 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:04 smithi121 bash[18489]: cluster 2024-02-12T16:34:02.317567+0000 osd.11 (osd.11) 167 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:34:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:04 smithi121 bash[18489]: cluster 2024-02-12T16:34:02.875440+0000 osd.2 (osd.2) 105 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:04 smithi121 bash[18489]: cluster 2024-02-12T16:34:02.875535+0000 osd.2 (osd.2) 106 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:04 smithi121 bash[18489]: cluster 2024-02-12T16:34:03.269246+0000 mon.a (mon.0) 2538 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:34:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:04 smithi121 bash[18489]: cluster 2024-02-12T16:34:03.269277+0000 mon.a (mon.0) 2539 : cluster [INF] Cluster is now healthy 2024-02-12T16:34:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:04 smithi121 bash[18489]: audit 2024-02-12T16:34:03.282878+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? v1:172.21.15.121:0/1134891770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:34:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:04 smithi121 bash[18489]: cluster 2024-02-12T16:34:03.283049+0000 mon.a (mon.0) 2541 : cluster [DBG] osdmap e751: 12 total, 12 up, 12 in 2024-02-12T16:34:04.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:04.706 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:04.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:04.849 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:04.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:04.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:05.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:05.137 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:05.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:05.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:05.276 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:05.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:05.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:05.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:05 smithi121 bash[18489]: cluster 2024-02-12T16:34:04.101981+0000 mgr.y (mgr.14146) 1483 : cluster [DBG] pgmap v2114: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:05.422 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:05.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:05.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:05 smithi169 bash[19604]: cluster 2024-02-12T16:34:04.101981+0000 mgr.y (mgr.14146) 1483 : cluster [DBG] pgmap v2114: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:05.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:05.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:05 smithi123 bash[19119]: cluster 2024-02-12T16:34:04.101981+0000 mgr.y (mgr.14146) 1483 : cluster [DBG] pgmap v2114: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:05.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:05.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:05.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:05.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:05.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:34:05.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:05.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:05.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:05.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:05.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:06.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:06.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:06.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:06.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:06.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:06.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:06.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:06.396 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:06.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:06.478 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:34:06.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:06.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:06.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:06.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:06.675 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:06.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:06.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:06.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:34:06.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:06.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:06.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:06.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:06.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:07.040 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_5' created 2024-02-12T16:34:07.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:07.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:07.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:07.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:06 smithi121 bash[18489]: cluster 2024-02-12T16:34:06.102686+0000 mgr.y (mgr.14146) 1484 : cluster [DBG] pgmap v2115: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:07.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:07.217 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:07.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:06 smithi169 bash[19604]: cluster 2024-02-12T16:34:06.102686+0000 mgr.y (mgr.14146) 1484 : cluster [DBG] pgmap v2115: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:07.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:07.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:07.361 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:06 smithi123 bash[19119]: cluster 2024-02-12T16:34:06.102686+0000 mgr.y (mgr.14146) 1484 : cluster [DBG] pgmap v2115: 31 pgs: 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:07.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:07.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:07.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:07.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:07.638 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:07.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:07.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:07.772 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:07.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:07.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-02-12T16:34:07.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:07.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:07.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:07.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:08.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:08.053 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:08.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:08.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:08.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:07 smithi121 bash[18489]: audit 2024-02-12T16:34:06.504357+0000 mon.a (mon.0) 2542 : audit [INF] from='client.? v1:172.21.15.121:0/2669015050' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-02-12T16:34:08.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:07 smithi121 bash[18489]: audit 2024-02-12T16:34:07.030579+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? v1:172.21.15.121:0/2669015050' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-02-12T16:34:08.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:07 smithi121 bash[18489]: cluster 2024-02-12T16:34:07.030634+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e752: 12 total, 12 up, 12 in 2024-02-12T16:34:08.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:08.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:08.233 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:07 smithi123 bash[19119]: audit 2024-02-12T16:34:06.504357+0000 mon.a (mon.0) 2542 : audit [INF] from='client.? v1:172.21.15.121:0/2669015050' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-02-12T16:34:08.233 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:07 smithi123 bash[19119]: audit 2024-02-12T16:34:07.030579+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? v1:172.21.15.121:0/2669015050' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-02-12T16:34:08.233 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:07 smithi123 bash[19119]: cluster 2024-02-12T16:34:07.030634+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e752: 12 total, 12 up, 12 in 2024-02-12T16:34:08.277 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:08.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:07 smithi169 bash[19604]: audit 2024-02-12T16:34:06.504357+0000 mon.a (mon.0) 2542 : audit [INF] from='client.? v1:172.21.15.121:0/2669015050' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-02-12T16:34:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:07 smithi169 bash[19604]: audit 2024-02-12T16:34:07.030579+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? v1:172.21.15.121:0/2669015050' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-02-12T16:34:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:07 smithi169 bash[19604]: cluster 2024-02-12T16:34:07.030634+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e752: 12 total, 12 up, 12 in 2024-02-12T16:34:08.320 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:08.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:08.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:08.461 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:08.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:08.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:08.602 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:08.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:08.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:08.736 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:08.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:08.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:08.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:08.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:08.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:09.015 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:09.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:09.122 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:09 smithi121 bash[18489]: cluster 2024-02-12T16:34:07.927562+0000 osd.5 (osd.5) 151 : cluster [DBG] 1.18 deep-scrub starts 2024-02-12T16:34:09.122 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:09 smithi121 bash[18489]: cluster 2024-02-12T16:34:07.972830+0000 osd.5 (osd.5) 152 : cluster [DBG] 1.18 deep-scrub ok 2024-02-12T16:34:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:09.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:09.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:09 smithi123 bash[19119]: cluster 2024-02-12T16:34:07.927562+0000 osd.5 (osd.5) 151 : cluster [DBG] 1.18 deep-scrub starts 2024-02-12T16:34:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:09 smithi123 bash[19119]: cluster 2024-02-12T16:34:07.972830+0000 osd.5 (osd.5) 152 : cluster [DBG] 1.18 deep-scrub ok 2024-02-12T16:34:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:09 smithi123 bash[19119]: cluster 2024-02-12T16:34:08.031732+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e753: 12 total, 12 up, 12 in 2024-02-12T16:34:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:09 smithi123 bash[19119]: cluster 2024-02-12T16:34:08.103365+0000 mgr.y (mgr.14146) 1485 : cluster [DBG] pgmap v2118: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:09.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:09.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:09 smithi121 bash[18489]: cluster 2024-02-12T16:34:08.031732+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e753: 12 total, 12 up, 12 in 2024-02-12T16:34:09.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:09 smithi121 bash[18489]: cluster 2024-02-12T16:34:08.103365+0000 mgr.y (mgr.14146) 1485 : cluster [DBG] pgmap v2118: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:09.429 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:09.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:09 smithi169 bash[19604]: cluster 2024-02-12T16:34:07.927562+0000 osd.5 (osd.5) 151 : cluster [DBG] 1.18 deep-scrub starts 2024-02-12T16:34:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:09 smithi169 bash[19604]: cluster 2024-02-12T16:34:07.972830+0000 osd.5 (osd.5) 152 : cluster [DBG] 1.18 deep-scrub ok 2024-02-12T16:34:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:09 smithi169 bash[19604]: cluster 2024-02-12T16:34:08.031732+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e753: 12 total, 12 up, 12 in 2024-02-12T16:34:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:09 smithi169 bash[19604]: cluster 2024-02-12T16:34:08.103365+0000 mgr.y (mgr.14146) 1485 : cluster [DBG] pgmap v2118: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:09.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:09.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:09.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:09.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:09.854 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:09.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:09.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:09.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:10.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:10.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:10.156 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:34:10.156 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:34:10.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:10 smithi121 bash[18489]: cluster 2024-02-12T16:34:08.960224+0000 osd.6 (osd.6) 144 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:34:10.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:10 smithi121 bash[18489]: cluster 2024-02-12T16:34:09.042733+0000 osd.6 (osd.6) 145 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:34:10.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:10 smithi121 bash[18489]: cluster 2024-02-12T16:34:09.046802+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e754: 12 total, 12 up, 12 in 2024-02-12T16:34:10.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:10.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:10.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:10 smithi169 bash[19604]: cluster 2024-02-12T16:34:08.960224+0000 osd.6 (osd.6) 144 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:34:10.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:10 smithi169 bash[19604]: cluster 2024-02-12T16:34:09.042733+0000 osd.6 (osd.6) 145 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:34:10.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:10 smithi169 bash[19604]: cluster 2024-02-12T16:34:09.046802+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e754: 12 total, 12 up, 12 in 2024-02-12T16:34:10.352 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:10.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:10.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:10 smithi123 bash[19119]: cluster 2024-02-12T16:34:08.960224+0000 osd.6 (osd.6) 144 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:34:10.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:10 smithi123 bash[19119]: cluster 2024-02-12T16:34:09.042733+0000 osd.6 (osd.6) 145 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:34:10.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:10 smithi123 bash[19119]: cluster 2024-02-12T16:34:09.046802+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e754: 12 total, 12 up, 12 in 2024-02-12T16:34:10.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:10.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:10.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.539 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:10.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:10.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:10.677 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:10.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:10.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.816 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:10.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:10.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:34:10.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:10.954 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:10.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:11.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:11.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:11.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:11 smithi123 bash[19119]: cluster 2024-02-12T16:34:09.629971+0000 osd.7 (osd.7) 186 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:11 smithi123 bash[19119]: cluster 2024-02-12T16:34:09.630035+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:11 smithi123 bash[19119]: cluster 2024-02-12T16:34:10.054397+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e755: 12 total, 12 up, 12 in 2024-02-12T16:34:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:11 smithi123 bash[19119]: cluster 2024-02-12T16:34:10.103922+0000 mgr.y (mgr.14146) 1486 : cluster [DBG] pgmap v2121: 47 pgs: 1 creating+peering, 13 unknown, 33 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:11.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:11.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:11.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:11.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:11.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:11 smithi169 bash[19604]: cluster 2024-02-12T16:34:09.629971+0000 osd.7 (osd.7) 186 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:11.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:11 smithi169 bash[19604]: cluster 2024-02-12T16:34:09.630035+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:11.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:11 smithi169 bash[19604]: cluster 2024-02-12T16:34:10.054397+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e755: 12 total, 12 up, 12 in 2024-02-12T16:34:11.356 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:11 smithi169 bash[19604]: cluster 2024-02-12T16:34:10.103922+0000 mgr.y (mgr.14146) 1486 : cluster [DBG] pgmap v2121: 47 pgs: 1 creating+peering, 13 unknown, 33 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:11.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:11.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:11 smithi121 bash[18489]: cluster 2024-02-12T16:34:09.629971+0000 osd.7 (osd.7) 186 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:11 smithi121 bash[18489]: cluster 2024-02-12T16:34:09.630035+0000 osd.7 (osd.7) 187 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:11 smithi121 bash[18489]: cluster 2024-02-12T16:34:10.054397+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e755: 12 total, 12 up, 12 in 2024-02-12T16:34:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:11 smithi121 bash[18489]: cluster 2024-02-12T16:34:10.103922+0000 mgr.y (mgr.14146) 1486 : cluster [DBG] pgmap v2121: 47 pgs: 1 creating+peering, 13 unknown, 33 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:11.471 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:11.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:11.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:11.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:11.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:11.663 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:11.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:11.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:11.806 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:11.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:11.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:11.950 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:11.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:12.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:12.093 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:12.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:12.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:12.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:12.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:12.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:12.377 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:12.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:12.387 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:12.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:12.520 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:12.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:12.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:12.659 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:12.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:12.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:12.799 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:12.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:12.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:12.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:13.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:13.079 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:13.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:13.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:13.218 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:13.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:13.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:13.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:13.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:13.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:13.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:13.479 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:13 smithi121 bash[18489]: cluster 2024-02-12T16:34:11.888651+0000 osd.3 (osd.3) 62 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:34:13.480 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:13 smithi121 bash[18489]: cluster 2024-02-12T16:34:11.980893+0000 osd.3 (osd.3) 63 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:34:13.480 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:13 smithi121 bash[18489]: cluster 2024-02-12T16:34:12.104715+0000 mgr.y (mgr.14146) 1487 : cluster [DBG] pgmap v2122: 47 pgs: 1 creating+peering, 46 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:13.509 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:13.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:13 smithi169 bash[19604]: cluster 2024-02-12T16:34:11.888651+0000 osd.3 (osd.3) 62 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:34:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:13 smithi169 bash[19604]: cluster 2024-02-12T16:34:11.980893+0000 osd.3 (osd.3) 63 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:34:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:13 smithi169 bash[19604]: cluster 2024-02-12T16:34:12.104715+0000 mgr.y (mgr.14146) 1487 : cluster [DBG] pgmap v2122: 47 pgs: 1 creating+peering, 46 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:13.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:13.625 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:13 smithi123 bash[19119]: cluster 2024-02-12T16:34:11.888651+0000 osd.3 (osd.3) 62 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:34:13.625 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:13 smithi123 bash[19119]: cluster 2024-02-12T16:34:11.980893+0000 osd.3 (osd.3) 63 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:34:13.625 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:13 smithi123 bash[19119]: cluster 2024-02-12T16:34:12.104715+0000 mgr.y (mgr.14146) 1487 : cluster [DBG] pgmap v2122: 47 pgs: 1 creating+peering, 46 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:13.650 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:13.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:13.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:13.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:13.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:13.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:13.948 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:13.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:14.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:14.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:14.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:14.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:14.224 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:14.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:14.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:14.364 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:14.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:14 smithi121 bash[18489]: cluster 2024-02-12T16:34:12.632110+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:14 smithi121 bash[18489]: cluster 2024-02-12T16:34:12.632205+0000 osd.4 (osd.4) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:14 smithi121 bash[18489]: cluster 2024-02-12T16:34:13.173224+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e756: 12 total, 12 up, 12 in 2024-02-12T16:34:14.469 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:14.481 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:14 smithi123 bash[19119]: cluster 2024-02-12T16:34:12.632110+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:14.481 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:14 smithi123 bash[19119]: cluster 2024-02-12T16:34:12.632205+0000 osd.4 (osd.4) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:14.481 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:14 smithi123 bash[19119]: cluster 2024-02-12T16:34:13.173224+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e756: 12 total, 12 up, 12 in 2024-02-12T16:34:14.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:14.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:14 smithi169 bash[19604]: cluster 2024-02-12T16:34:12.632110+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:14 smithi169 bash[19604]: cluster 2024-02-12T16:34:12.632205+0000 osd.4 (osd.4) 127 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:14.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:14 smithi169 bash[19604]: cluster 2024-02-12T16:34:13.173224+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e756: 12 total, 12 up, 12 in 2024-02-12T16:34:14.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:14.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:14.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:14.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:14.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:14.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:14.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:14.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:15.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:15.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:15.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:15.148 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:15.179 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:15.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:15.203 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T16:34:15.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:15.318 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:15.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:15.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:15.444 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: cluster 2024-02-12T16:34:14.105517+0000 mgr.y (mgr.14146) 1488 : cluster [DBG] pgmap v2124: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:15.444 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.179609+0000 mon.b (mon.2) 155 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]: dispatch 2024-02-12T16:34:15.444 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.179929+0000 mon.b (mon.2) 156 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]: dispatch 2024-02-12T16:34:15.444 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.180211+0000 mon.b (mon.2) 157 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.180528+0000 mon.b (mon.2) 158 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.180838+0000 mon.b (mon.2) 159 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.181185+0000 mon.b (mon.2) 160 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.184829+0000 mon.a (mon.0) 2549 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.185762+0000 mon.a (mon.0) 2550 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.186520+0000 mon.a (mon.0) 2551 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.187126+0000 mon.a (mon.0) 2552 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.187800+0000 mon.a (mon.0) 2553 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.188147+0000 mon.b (mon.2) 161 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.188358+0000 mon.a (mon.0) 2554 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.188572+0000 mon.b (mon.2) 162 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.188991+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]: dispatch 2024-02-12T16:34:15.445 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.189043+0000 mon.a (mon.0) 2555 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.189374+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.189512+0000 mon.a (mon.0) 2556 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.189751+0000 mon.b (mon.2) 165 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.189965+0000 mon.a (mon.0) 2557 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.190212+0000 mon.b (mon.2) 166 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.190411+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.190859+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:14.191340+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]: dispatch 2024-02-12T16:34:15.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:15 smithi121 bash[18489]: audit 2024-02-12T16:34:15.109704+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? v1:172.21.15.121:0/1045328184' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:34:15.461 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: cluster 2024-02-12T16:34:14.105517+0000 mgr.y (mgr.14146) 1488 : cluster [DBG] pgmap v2124: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.179609+0000 mon.b (mon.2) 155 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.179929+0000 mon.b (mon.2) 156 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.180211+0000 mon.b (mon.2) 157 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.180528+0000 mon.b (mon.2) 158 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.180838+0000 mon.b (mon.2) 159 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.181185+0000 mon.b (mon.2) 160 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.184829+0000 mon.a (mon.0) 2549 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.185762+0000 mon.a (mon.0) 2550 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.186520+0000 mon.a (mon.0) 2551 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.187126+0000 mon.a (mon.0) 2552 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]: dispatch 2024-02-12T16:34:15.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.187800+0000 mon.a (mon.0) 2553 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]: dispatch 2024-02-12T16:34:15.465 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.188147+0000 mon.b (mon.2) 161 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]: dispatch 2024-02-12T16:34:15.465 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.188358+0000 mon.a (mon.0) 2554 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]: dispatch 2024-02-12T16:34:15.465 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.188572+0000 mon.b (mon.2) 162 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]: dispatch 2024-02-12T16:34:15.465 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.188991+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]: dispatch 2024-02-12T16:34:15.465 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.189043+0000 mon.a (mon.0) 2555 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]: dispatch 2024-02-12T16:34:15.465 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.189374+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]: dispatch 2024-02-12T16:34:15.465 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.189512+0000 mon.a (mon.0) 2556 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]: dispatch 2024-02-12T16:34:15.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.189751+0000 mon.b (mon.2) 165 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]: dispatch 2024-02-12T16:34:15.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.189965+0000 mon.a (mon.0) 2557 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]: dispatch 2024-02-12T16:34:15.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.190212+0000 mon.b (mon.2) 166 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]: dispatch 2024-02-12T16:34:15.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.190411+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]: dispatch 2024-02-12T16:34:15.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.190859+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]: dispatch 2024-02-12T16:34:15.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:14.191340+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]: dispatch 2024-02-12T16:34:15.466 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:15 smithi169 bash[19604]: audit 2024-02-12T16:34:15.109704+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? v1:172.21.15.121:0/1045328184' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:34:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:15.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:15.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:15.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:15.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:15.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: cluster 2024-02-12T16:34:14.105517+0000 mgr.y (mgr.14146) 1488 : cluster [DBG] pgmap v2124: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:15.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.179609+0000 mon.b (mon.2) 155 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.179929+0000 mon.b (mon.2) 156 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.180211+0000 mon.b (mon.2) 157 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.180528+0000 mon.b (mon.2) 158 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.180838+0000 mon.b (mon.2) 159 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.181185+0000 mon.b (mon.2) 160 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.184829+0000 mon.a (mon.0) 2549 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.185762+0000 mon.a (mon.0) 2550 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.186520+0000 mon.a (mon.0) 2551 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.187126+0000 mon.a (mon.0) 2552 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.187800+0000 mon.a (mon.0) 2553 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.188147+0000 mon.b (mon.2) 161 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.188358+0000 mon.a (mon.0) 2554 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.188572+0000 mon.b (mon.2) 162 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]: dispatch 2024-02-12T16:34:15.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.188991+0000 mon.b (mon.2) 163 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.189043+0000 mon.a (mon.0) 2555 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.189374+0000 mon.b (mon.2) 164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.189512+0000 mon.a (mon.0) 2556 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.189751+0000 mon.b (mon.2) 165 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.189965+0000 mon.a (mon.0) 2557 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.190212+0000 mon.b (mon.2) 166 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.190411+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.190859+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:14.191340+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]: dispatch 2024-02-12T16:34:15.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:15 smithi123 bash[19119]: audit 2024-02-12T16:34:15.109704+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? v1:172.21.15.121:0/1045328184' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:34:15.666 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:34:15.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:15.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:15.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:15.839 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:15.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:15.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:15.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:16.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:16.156 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:16.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:16.295 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.193974+0000 mon.a (mon.0) 2562 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]': finished 2024-02-12T16:34:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194031+0000 mon.a (mon.0) 2563 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]': finished 2024-02-12T16:34:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194077+0000 mon.a (mon.0) 2564 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]': finished 2024-02-12T16:34:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194130+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]': finished 2024-02-12T16:34:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194178+0000 mon.a (mon.0) 2566 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]': finished 2024-02-12T16:34:16.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194216+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194256+0000 mon.a (mon.0) 2568 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194293+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194333+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194369+0000 mon.a (mon.0) 2571 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194412+0000 mon.a (mon.0) 2572 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194452+0000 mon.a (mon.0) 2573 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: audit 2024-02-12T16:34:15.194494+0000 mon.a (mon.0) 2574 : audit [INF] from='client.? v1:172.21.15.121:0/1045328184' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:34:16.391 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:16 smithi123 bash[19119]: cluster 2024-02-12T16:34:15.194527+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e757: 12 total, 12 up, 12 in 2024-02-12T16:34:16.398 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:16.430 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:16.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:16.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.193974+0000 mon.a (mon.0) 2562 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]': finished 2024-02-12T16:34:16.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194031+0000 mon.a (mon.0) 2563 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]': finished 2024-02-12T16:34:16.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194077+0000 mon.a (mon.0) 2564 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]': finished 2024-02-12T16:34:16.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194130+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194178+0000 mon.a (mon.0) 2566 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194216+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194256+0000 mon.a (mon.0) 2568 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194293+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194333+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194369+0000 mon.a (mon.0) 2571 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194412+0000 mon.a (mon.0) 2572 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194452+0000 mon.a (mon.0) 2573 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: audit 2024-02-12T16:34:15.194494+0000 mon.a (mon.0) 2574 : audit [INF] from='client.? v1:172.21.15.121:0/1045328184' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:34:16.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:16 smithi121 bash[18489]: cluster 2024-02-12T16:34:15.194527+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e757: 12 total, 12 up, 12 in 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.193974+0000 mon.a (mon.0) 2562 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08279175840994103]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194031+0000 mon.a (mon.0) 2563 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09130229248437648]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194077+0000 mon.a (mon.0) 2564 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08871782917413848]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194130+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09381522914793146]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194178+0000 mon.a (mon.0) 2566 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09075557217009879]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194216+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08409726722982837]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194256+0000 mon.a (mon.0) 2568 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0855867904881915]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194293+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08421497207912525]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194333+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0911577012112254]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194369+0000 mon.a (mon.0) 2571 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08607853209122472]}]': finished 2024-02-12T16:34:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194412+0000 mon.a (mon.0) 2572 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08443984869158228]}]': finished 2024-02-12T16:34:16.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194452+0000 mon.a (mon.0) 2573 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08458859354108651]}]': finished 2024-02-12T16:34:16.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: audit 2024-02-12T16:34:15.194494+0000 mon.a (mon.0) 2574 : audit [INF] from='client.? v1:172.21.15.121:0/1045328184' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:34:16.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:16 smithi169 bash[19604]: cluster 2024-02-12T16:34:15.194527+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e757: 12 total, 12 up, 12 in 2024-02-12T16:34:16.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:16.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:16.709 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:16.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:16.850 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:16.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:16.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:16.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:17.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:17.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:17.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:17.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:17.267 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:17.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:17.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:17.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:17 smithi121 bash[18489]: cluster 2024-02-12T16:34:16.106486+0000 mgr.y (mgr.14146) 1489 : cluster [DBG] pgmap v2126: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:17.397 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:17 smithi121 bash[18489]: cluster 2024-02-12T16:34:16.211270+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e758: 12 total, 12 up, 12 in 2024-02-12T16:34:17.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 9 1 2024-02-12T16:34:17.410 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:17.412 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:17 smithi123 bash[19119]: cluster 2024-02-12T16:34:16.106486+0000 mgr.y (mgr.14146) 1489 : cluster [DBG] pgmap v2126: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:17.412 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:17 smithi123 bash[19119]: cluster 2024-02-12T16:34:16.211270+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e758: 12 total, 12 up, 12 in 2024-02-12T16:34:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:17.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:17.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:17.527 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:17 smithi169 bash[19604]: cluster 2024-02-12T16:34:16.106486+0000 mgr.y (mgr.14146) 1489 : cluster [DBG] pgmap v2126: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:17.527 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:17 smithi169 bash[19604]: cluster 2024-02-12T16:34:16.211270+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e758: 12 total, 12 up, 12 in 2024-02-12T16:34:17.551 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:17.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:17.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:17.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:17.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:17.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:17.824 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:17.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:17.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:17.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:17.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:18.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:18.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:18.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:18.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:18.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:18.348 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:18.382 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:18.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:18.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:18.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:18.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:18.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:18.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:18.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:18.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:18.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:18.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:18.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:18.953 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:18.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:19.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:34:19.053 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:34:19.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:19.077 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:34:19.092 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:34:19.092 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:34:19.094 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:19.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:19.098 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:34:19.122 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:34:19.153 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:34:19.175 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:34:19.197 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:34:19.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:19.221 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:34:19.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:19.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:19.246 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:34:19.271 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:34:19.294 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:34:19.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.299 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.300 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:34:19.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:19.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:19.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:19.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:19.492 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:19 smithi121 bash[18489]: cluster 2024-02-12T16:34:18.107253+0000 mgr.y (mgr.14146) 1490 : cluster [DBG] pgmap v2128: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:19.492 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:19 smithi121 bash[18489]: audit 2024-02-12T16:34:18.995785+0000 mon.a (mon.0) 2577 : audit [DBG] from='client.? v1:172.21.15.121:0/2703822507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:19.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:19.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:19 smithi169 bash[19604]: cluster 2024-02-12T16:34:18.107253+0000 mgr.y (mgr.14146) 1490 : cluster [DBG] pgmap v2128: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:19 smithi169 bash[19604]: audit 2024-02-12T16:34:18.995785+0000 mon.a (mon.0) 2577 : audit [DBG] from='client.? v1:172.21.15.121:0/2703822507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:19.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:19.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:19 smithi123 bash[19119]: cluster 2024-02-12T16:34:18.107253+0000 mgr.y (mgr.14146) 1490 : cluster [DBG] pgmap v2128: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:19.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:19 smithi123 bash[19119]: audit 2024-02-12T16:34:18.995785+0000 mon.a (mon.0) 2577 : audit [DBG] from='client.? v1:172.21.15.121:0/2703822507' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:19.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:19.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:19.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:19.798 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:19.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:19.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-12T16:34:19.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:19.936 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:19.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:20.039 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:20.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:20.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:20.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:20.218 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:20.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:20.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:20.355 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:20.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:20.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:20.495 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:20.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:20.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:20.637 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:20.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:20.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:20.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:20.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:20.804 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-02-12T16:34:20.809 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:34:20.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:20.916 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:20.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:21.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:21.054 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:21.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:21.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:21.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:21.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:21.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:21.337 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:21.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:21.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:21.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:21.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:21 smithi169 bash[19604]: cluster 2024-02-12T16:34:20.107852+0000 mgr.y (mgr.14146) 1491 : cluster [DBG] pgmap v2129: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:21.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:21 smithi169 bash[19604]: audit 2024-02-12T16:34:20.443290+0000 mon.a (mon.0) 2578 : audit [INF] from='client.? v1:172.21.15.121:0/1266932380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:34:21.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:21 smithi169 bash[19604]: audit 2024-02-12T16:34:20.454291+0000 mon.a (mon.0) 2579 : audit [INF] from='client.? v1:172.21.15.121:0/3868581307' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:34:21.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:21 smithi169 bash[19604]: cluster 2024-02-12T16:34:20.786613+0000 mon.a (mon.0) 2580 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:34:21.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:21 smithi169 bash[19604]: audit 2024-02-12T16:34:20.794281+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? v1:172.21.15.121:0/1266932380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:34:21.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:21 smithi169 bash[19604]: audit 2024-02-12T16:34:20.794390+0000 mon.a (mon.0) 2582 : audit [INF] from='client.? v1:172.21.15.121:0/3868581307' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:34:21.481 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:21 smithi169 bash[19604]: cluster 2024-02-12T16:34:20.794483+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e759: 12 total, 12 up, 12 in 2024-02-12T16:34:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:21.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:21.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:21.594 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:21 smithi121 bash[18489]: cluster 2024-02-12T16:34:20.107852+0000 mgr.y (mgr.14146) 1491 : cluster [DBG] pgmap v2129: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:21.594 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:21 smithi121 bash[18489]: audit 2024-02-12T16:34:20.443290+0000 mon.a (mon.0) 2578 : audit [INF] from='client.? v1:172.21.15.121:0/1266932380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:34:21.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:21 smithi121 bash[18489]: audit 2024-02-12T16:34:20.454291+0000 mon.a (mon.0) 2579 : audit [INF] from='client.? v1:172.21.15.121:0/3868581307' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:34:21.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:21 smithi121 bash[18489]: cluster 2024-02-12T16:34:20.786613+0000 mon.a (mon.0) 2580 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:34:21.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:21 smithi121 bash[18489]: audit 2024-02-12T16:34:20.794281+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? v1:172.21.15.121:0/1266932380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:34:21.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:21 smithi121 bash[18489]: audit 2024-02-12T16:34:20.794390+0000 mon.a (mon.0) 2582 : audit [INF] from='client.? v1:172.21.15.121:0/3868581307' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:34:21.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:21 smithi121 bash[18489]: cluster 2024-02-12T16:34:20.794483+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e759: 12 total, 12 up, 12 in 2024-02-12T16:34:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:21.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:21 smithi123 bash[19119]: cluster 2024-02-12T16:34:20.107852+0000 mgr.y (mgr.14146) 1491 : cluster [DBG] pgmap v2129: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:21 smithi123 bash[19119]: audit 2024-02-12T16:34:20.443290+0000 mon.a (mon.0) 2578 : audit [INF] from='client.? v1:172.21.15.121:0/1266932380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:34:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:21 smithi123 bash[19119]: audit 2024-02-12T16:34:20.454291+0000 mon.a (mon.0) 2579 : audit [INF] from='client.? v1:172.21.15.121:0/3868581307' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:34:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:21 smithi123 bash[19119]: cluster 2024-02-12T16:34:20.786613+0000 mon.a (mon.0) 2580 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:34:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:21 smithi123 bash[19119]: audit 2024-02-12T16:34:20.794281+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? v1:172.21.15.121:0/1266932380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:34:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:21 smithi123 bash[19119]: audit 2024-02-12T16:34:20.794390+0000 mon.a (mon.0) 2582 : audit [INF] from='client.? v1:172.21.15.121:0/3868581307' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:34:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:21 smithi123 bash[19119]: cluster 2024-02-12T16:34:20.794483+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e759: 12 total, 12 up, 12 in 2024-02-12T16:34:21.722 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:21.759 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:21.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:21.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:21.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:22.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:22.048 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:22.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:22.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:22.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:22.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:22.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:34:22.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:22.333 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:22.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:22.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:22.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:22.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:22.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:22.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:22.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:22.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:22.760 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:22.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:22.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:22.901 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:22.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:23.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:23.044 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:23.141 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-02-12T16:34:23.143 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:34:23.143 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:34:23.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:23.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:34:23.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:34:23.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_105252 2024-02-12T16:34:23.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:34:23.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:34:23.186 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:23.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:23.297 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:23 smithi123 bash[19119]: cluster 2024-02-12T16:34:22.108593+0000 mgr.y (mgr.14146) 1492 : cluster [DBG] pgmap v2131: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:23.316 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:23.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:23 smithi169 bash[19604]: cluster 2024-02-12T16:34:22.108593+0000 mgr.y (mgr.14146) 1492 : cluster [DBG] pgmap v2131: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:23.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:23.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:23.631 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:23.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:23 smithi121 bash[18489]: cluster 2024-02-12T16:34:22.108593+0000 mgr.y (mgr.14146) 1492 : cluster [DBG] pgmap v2131: 47 pgs: 47 active+clean; 0 B data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:23.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:23.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:23.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:23.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:23.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:23.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:24.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:34:24.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:24.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:24.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:24.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 16 5905 5889 368.043 368.062 0.00288275 0.00270944 2024-02-12T16:34:24.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:24.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:24.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:24.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:24.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:24.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:24.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:24.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:24.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:24.613 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:34:24.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:24.648 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:24.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:24.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:24.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:24.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:24.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:24.907 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:24.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:25.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:25.037 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:25.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:25.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:25.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:25.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:25.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 16 11174 11158 348.659 329.312 0.00210261 0.00286527 2024-02-12T16:34:25.230 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:25 smithi123 bash[19119]: cluster 2024-02-12T16:34:24.109117+0000 mgr.y (mgr.14146) 1493 : cluster [DBG] pgmap v2132: 47 pgs: 47 active+clean; 0 B data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:25.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:25 smithi169 bash[19604]: cluster 2024-02-12T16:34:24.109117+0000 mgr.y (mgr.14146) 1493 : cluster [DBG] pgmap v2132: 47 pgs: 47 active+clean; 0 B data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:25.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:25.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:25 smithi121 bash[18489]: cluster 2024-02-12T16:34:24.109117+0000 mgr.y (mgr.14146) 1493 : cluster [DBG] pgmap v2132: 47 pgs: 47 active+clean; 0 B data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:25.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:25.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:25.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:25.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:25.567 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:25.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:25.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:25.754 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:25.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:25.857 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:25.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:25.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:25.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:26.019 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:26.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:26.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:26.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:26.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:26.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 16085 16069 334.744 306.938 0.00398731 0.00298485 2024-02-12T16:34:26.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:26.289 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:26.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:26.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:26.420 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:26.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:26.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:26.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:26.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:26.681 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:26.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:26.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:26.852 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:26 smithi169 bash[19604]: cluster 2024-02-12T16:34:26.109588+0000 mgr.y (mgr.14146) 1494 : cluster [DBG] pgmap v2133: 47 pgs: 47 active+clean; 324 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:26.852 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:26 smithi169 bash[19604]: cluster 2024-02-12T16:34:26.243373+0000 osd.5 (osd.5) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:26.852 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:26 smithi169 bash[19604]: cluster 2024-02-12T16:34:26.243434+0000 osd.5 (osd.5) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:26.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:26.907 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:26.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:27.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:27.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:27.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:27.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:26 smithi121 bash[18489]: cluster 2024-02-12T16:34:26.109588+0000 mgr.y (mgr.14146) 1494 : cluster [DBG] pgmap v2133: 47 pgs: 47 active+clean; 324 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:27.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:26 smithi121 bash[18489]: cluster 2024-02-12T16:34:26.243373+0000 osd.5 (osd.5) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:27.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:26 smithi121 bash[18489]: cluster 2024-02-12T16:34:26.243434+0000 osd.5 (osd.5) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:27.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:26 smithi123 bash[19119]: cluster 2024-02-12T16:34:26.109588+0000 mgr.y (mgr.14146) 1494 : cluster [DBG] pgmap v2133: 47 pgs: 47 active+clean; 324 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:27.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:26 smithi123 bash[19119]: cluster 2024-02-12T16:34:26.243373+0000 osd.5 (osd.5) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:27.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:26 smithi123 bash[19119]: cluster 2024-02-12T16:34:26.243434+0000 osd.5 (osd.5) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:27.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:27.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:27.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:27.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 16 20575 20559 321.209 280.625 0.00506374 0.00310954 2024-02-12T16:34:27.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:27.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:27.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:27.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:27.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:27.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:27.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:27.634 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:27.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:27.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:27.763 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:27.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:27.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:27.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:27.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:27.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:27 smithi121 bash[18489]: cluster 2024-02-12T16:34:26.805348+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e760: 12 total, 12 up, 12 in 2024-02-12T16:34:27.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:27 smithi123 bash[19119]: cluster 2024-02-12T16:34:26.805348+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e760: 12 total, 12 up, 12 in 2024-02-12T16:34:28.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 24614 24598 307.45 252.438 0.00216889 0.00325051 2024-02-12T16:34:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:27 smithi169 bash[19604]: cluster 2024-02-12T16:34:26.805348+0000 mon.a (mon.0) 2584 : cluster [DBG] osdmap e760: 12 total, 12 up, 12 in 2024-02-12T16:34:28.330 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:28.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:28.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:28.459 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:28.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:28.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:28.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:28.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:28.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:28.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:28.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:28.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:28 smithi123 bash[19119]: cluster 2024-02-12T16:34:28.110102+0000 mgr.y (mgr.14146) 1495 : cluster [DBG] pgmap v2135: 47 pgs: 47 active+clean; 937 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:28.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:28.971 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:28.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:29.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:29.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:29.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:29.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:28 smithi121 bash[18489]: cluster 2024-02-12T16:34:28.110102+0000 mgr.y (mgr.14146) 1495 : cluster [DBG] pgmap v2135: 47 pgs: 47 active+clean; 937 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:29.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 27841 27825 289.82 201.688 0.00349537 0.00343707 2024-02-12T16:34:29.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:29.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:29.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:29.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:28 smithi169 bash[19604]: cluster 2024-02-12T16:34:28.110102+0000 mgr.y (mgr.14146) 1495 : cluster [DBG] pgmap v2135: 47 pgs: 47 active+clean; 937 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:29.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:29.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:29.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:29.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:29.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:29.521 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:34:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:34:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26530.0:29027 7.f e760/752 7:f0ea6794:::benchmark_data_smithi169_105252_object29026:head v 760'1820, mlcod=760'1816)", 2024-02-12T16:34:29.521 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.516201+0000", 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.00088781500000000002, 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00091784500000000003, 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516201+0000", 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516201+0000", 2024-02-12T16:34:29.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516200+0000", 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117114 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516317+0000", 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5349999999999993e-06 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516326+0000", 2024-02-12T16:34:29.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0227000000000001e-05 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516347+0000", 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5716000000000002e-05 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516382+0000", 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.208e-05 2024-02-12T16:34:29.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.516394+0000", 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019325000000000001 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 1 2024-02-12T16:34:29.525 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:34:29.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:29.622 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:29.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:29.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:29.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:29.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:29.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:29.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:29 smithi169 bash[19604]: audit 2024-02-12T16:34:28.845394+0000 mon.b (mon.2) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:34:29.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:29 smithi169 bash[19604]: audit 2024-02-12T16:34:29.323754+0000 mon.b (mon.2) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:34:29.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:29 smithi169 bash[19604]: audit 2024-02-12T16:34:29.327647+0000 mon.b (mon.2) 169 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:34:29.866 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:29 smithi169 bash[19604]: audit 2024-02-12T16:34:29.332829+0000 mon.a (mon.0) 2585 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:34:29.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:29.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:29.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:30.017 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:30.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:30.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:30.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:29 smithi123 bash[19119]: audit 2024-02-12T16:34:28.845394+0000 mon.b (mon.2) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:34:30.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:29 smithi123 bash[19119]: audit 2024-02-12T16:34:29.323754+0000 mon.b (mon.2) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:34:30.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:29 smithi123 bash[19119]: audit 2024-02-12T16:34:29.327647+0000 mon.b (mon.2) 169 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:34:30.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:29 smithi123 bash[19119]: audit 2024-02-12T16:34:29.332829+0000 mon.a (mon.0) 2585 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:34:30.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:29 smithi121 bash[18489]: audit 2024-02-12T16:34:28.845394+0000 mon.b (mon.2) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:34:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:29 smithi121 bash[18489]: audit 2024-02-12T16:34:29.323754+0000 mon.b (mon.2) 168 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:34:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:29 smithi121 bash[18489]: audit 2024-02-12T16:34:29.327647+0000 mon.b (mon.2) 169 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:34:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:29 smithi121 bash[18489]: audit 2024-02-12T16:34:29.332829+0000 mon.a (mon.0) 2585 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:34:30.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 16 31405 31389 280.236 222.75 0.00495989 0.00356609 2024-02-12T16:34:30.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:30.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:30.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:30.327 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:30.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:30.409 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:30.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:30.463 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:30.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:30.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-12T16:34:30.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:30.595 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:30.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:30.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:30.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:30.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:30.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:30.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:30.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:30.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:31.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:31.119 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:30 smithi123 bash[19119]: cluster 2024-02-12T16:34:30.110591+0000 mgr.y (mgr.14146) 1496 : cluster [DBG] pgmap v2136: 47 pgs: 47 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:31.119 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:30 smithi123 bash[19119]: cluster 2024-02-12T16:34:30.797795+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e761: 12 total, 12 up, 12 in 2024-02-12T16:34:31.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:31.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:30 smithi121 bash[18489]: cluster 2024-02-12T16:34:30.110591+0000 mgr.y (mgr.14146) 1496 : cluster [DBG] pgmap v2136: 47 pgs: 47 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:31.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:30 smithi121 bash[18489]: cluster 2024-02-12T16:34:30.797795+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e761: 12 total, 12 up, 12 in 2024-02-12T16:34:31.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 16 33372 33356 260.572 122.938 0.00239984 0.00383548 2024-02-12T16:34:31.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:31.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:30 smithi169 bash[19604]: cluster 2024-02-12T16:34:30.110591+0000 mgr.y (mgr.14146) 1496 : cluster [DBG] pgmap v2136: 47 pgs: 47 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:34:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:30 smithi169 bash[19604]: cluster 2024-02-12T16:34:30.797795+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e761: 12 total, 12 up, 12 in 2024-02-12T16:34:31.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:31.421 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:31.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:31.548 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:31.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:31.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:31.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:31.813 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:31.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:31.941 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:31.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:32.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:32.072 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:32.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:32.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:32.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 36806 36790 255.465 214.625 0.0107637 0.00391268 2024-02-12T16:34:32.198 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:32.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:32.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:32.330 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:32.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:32.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:32.460 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:32.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:32.564 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:32.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:32.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:32.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:32.719 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:32.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:32.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:32.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:32.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:32.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:33.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:32 smithi169 bash[19604]: cluster 2024-02-12T16:34:31.121877+0000 osd.10 (osd.10) 154 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:32 smithi169 bash[19604]: cluster 2024-02-12T16:34:31.121935+0000 osd.10 (osd.10) 155 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:32 smithi169 bash[19604]: cluster 2024-02-12T16:34:31.827666+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e762: 12 total, 12 up, 12 in 2024-02-12T16:34:33.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:32 smithi169 bash[19604]: cluster 2024-02-12T16:34:32.111039+0000 mgr.y (mgr.14146) 1497 : cluster [DBG] pgmap v2139: 47 pgs: 47 active+clean; 2.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 283 MiB/s wr, 4.53k op/s 2024-02-12T16:34:33.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.111 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:32 smithi121 bash[18489]: cluster 2024-02-12T16:34:31.121877+0000 osd.10 (osd.10) 154 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:33.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:32 smithi121 bash[18489]: cluster 2024-02-12T16:34:31.121935+0000 osd.10 (osd.10) 155 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:33.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:32 smithi121 bash[18489]: cluster 2024-02-12T16:34:31.827666+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e762: 12 total, 12 up, 12 in 2024-02-12T16:34:33.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:32 smithi121 bash[18489]: cluster 2024-02-12T16:34:32.111039+0000 mgr.y (mgr.14146) 1497 : cluster [DBG] pgmap v2139: 47 pgs: 47 active+clean; 2.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 283 MiB/s wr, 4.53k op/s 2024-02-12T16:34:33.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:32 smithi123 bash[19119]: cluster 2024-02-12T16:34:31.121877+0000 osd.10 (osd.10) 154 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:32 smithi123 bash[19119]: cluster 2024-02-12T16:34:31.121935+0000 osd.10 (osd.10) 155 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:32 smithi123 bash[19119]: cluster 2024-02-12T16:34:31.827666+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e762: 12 total, 12 up, 12 in 2024-02-12T16:34:33.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:32 smithi123 bash[19119]: cluster 2024-02-12T16:34:32.111039+0000 mgr.y (mgr.14146) 1497 : cluster [DBG] pgmap v2139: 47 pgs: 47 active+clean; 2.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 283 MiB/s wr, 4.53k op/s 2024-02-12T16:34:33.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 15 40111 40096 250.58 206.625 0.00447365 0.0039888 2024-02-12T16:34:33.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:33.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:33.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:33.390 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:33.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.523 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:33.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:33.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:33.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:33.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:33.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:34.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:33 smithi169 bash[19604]: cluster 2024-02-12T16:34:32.604542+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:33 smithi169 bash[19604]: cluster 2024-02-12T16:34:32.604600+0000 osd.0 (osd.0) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:33 smithi169 bash[19604]: cluster 2024-02-12T16:34:32.835830+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e763: 12 total, 12 up, 12 in 2024-02-12T16:34:34.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:34.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:33 smithi123 bash[19119]: cluster 2024-02-12T16:34:32.604542+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:34.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:33 smithi123 bash[19119]: cluster 2024-02-12T16:34:32.604600+0000 osd.0 (osd.0) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:34.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:33 smithi123 bash[19119]: cluster 2024-02-12T16:34:32.835830+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e763: 12 total, 12 up, 12 in 2024-02-12T16:34:34.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:33 smithi121 bash[18489]: cluster 2024-02-12T16:34:32.604542+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:33 smithi121 bash[18489]: cluster 2024-02-12T16:34:32.604600+0000 osd.0 (osd.0) 125 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:34.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:33 smithi121 bash[18489]: cluster 2024-02-12T16:34:32.835830+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e763: 12 total, 12 up, 12 in 2024-02-12T16:34:34.179 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:34.183 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 42666 42650 242.31 159.625 0.00453305 0.00412514 2024-02-12T16:34:34.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:34.324 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:34.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:34.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:34.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:34.596 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:34.700 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:34.725 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:34.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.9 to 1 (10000) 2024-02-12T16:34:34.852 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:34 smithi123 bash[19119]: cluster 2024-02-12T16:34:34.111413+0000 mgr.y (mgr.14146) 1498 : cluster [DBG] pgmap v2141: 47 pgs: 47 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s wr, 3.06k op/s 2024-02-12T16:34:34.853 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:34 smithi123 bash[19119]: audit 2024-02-12T16:34:34.462086+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? v1:172.21.15.121:0/3699343110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:34:34.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:34.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:34.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:34.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:35.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:35.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:35.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:35.118 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:34 smithi121 bash[18489]: cluster 2024-02-12T16:34:34.111413+0000 mgr.y (mgr.14146) 1498 : cluster [DBG] pgmap v2141: 47 pgs: 47 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s wr, 3.06k op/s 2024-02-12T16:34:35.118 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:34 smithi121 bash[18489]: audit 2024-02-12T16:34:34.462086+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? v1:172.21.15.121:0/3699343110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:34:35.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 45672 45656 237.772 187.875 0.00857415 0.00420404 2024-02-12T16:34:35.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:35.226 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:34:35.253 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:35.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:34 smithi169 bash[19604]: cluster 2024-02-12T16:34:34.111413+0000 mgr.y (mgr.14146) 1498 : cluster [DBG] pgmap v2141: 47 pgs: 47 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s wr, 3.06k op/s 2024-02-12T16:34:35.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:34 smithi169 bash[19604]: audit 2024-02-12T16:34:34.462086+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? v1:172.21.15.121:0/3699343110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:34:35.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:35.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:35.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:35.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:35.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:35.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:35.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:35.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:35.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:35.776 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:35.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:35.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:35.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:35.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:36.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:36.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:35 smithi169 bash[19604]: audit 2024-02-12T16:34:34.841567+0000 mon.a (mon.0) 2590 : audit [INF] from='client.? v1:172.21.15.121:0/3699343110' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:34:36.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:35 smithi169 bash[19604]: cluster 2024-02-12T16:34:34.841696+0000 mon.a (mon.0) 2591 : cluster [DBG] osdmap e764: 12 total, 12 up, 12 in 2024-02-12T16:34:36.016 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:35 smithi169 bash[19604]: cluster 2024-02-12T16:34:35.803689+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e765: 12 total, 12 up, 12 in 2024-02-12T16:34:36.036 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:36.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:36.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:35 smithi123 bash[19119]: audit 2024-02-12T16:34:34.841567+0000 mon.a (mon.0) 2590 : audit [INF] from='client.? v1:172.21.15.121:0/3699343110' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:34:36.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:35 smithi123 bash[19119]: cluster 2024-02-12T16:34:34.841696+0000 mon.a (mon.0) 2591 : cluster [DBG] osdmap e764: 12 total, 12 up, 12 in 2024-02-12T16:34:36.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:35 smithi123 bash[19119]: cluster 2024-02-12T16:34:35.803689+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e765: 12 total, 12 up, 12 in 2024-02-12T16:34:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:35 smithi121 bash[18489]: audit 2024-02-12T16:34:34.841567+0000 mon.a (mon.0) 2590 : audit [INF] from='client.? v1:172.21.15.121:0/3699343110' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:34:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:35 smithi121 bash[18489]: cluster 2024-02-12T16:34:34.841696+0000 mon.a (mon.0) 2591 : cluster [DBG] osdmap e764: 12 total, 12 up, 12 in 2024-02-12T16:34:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:35 smithi121 bash[18489]: cluster 2024-02-12T16:34:35.803689+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e765: 12 total, 12 up, 12 in 2024-02-12T16:34:36.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:36.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 48120 48104 231.25 153 0.00446866 0.00432252 2024-02-12T16:34:36.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:36.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:36.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:36.424 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:36.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:36.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:36.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:36.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:36.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:36.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:36.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:36.922 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:36 smithi123 bash[19119]: cluster 2024-02-12T16:34:35.431069+0000 osd.3 (osd.3) 64 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:36.922 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:36 smithi123 bash[19119]: cluster 2024-02-12T16:34:35.431148+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:36.922 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:36 smithi123 bash[19119]: cluster 2024-02-12T16:34:36.111827+0000 mgr.y (mgr.14146) 1499 : cluster [DBG] pgmap v2144: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.98k op/s 2024-02-12T16:34:36.944 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:36.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:37.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:36 smithi169 bash[19604]: cluster 2024-02-12T16:34:35.431069+0000 osd.3 (osd.3) 64 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:36 smithi169 bash[19604]: cluster 2024-02-12T16:34:35.431148+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:36 smithi169 bash[19604]: cluster 2024-02-12T16:34:36.111827+0000 mgr.y (mgr.14146) 1499 : cluster [DBG] pgmap v2144: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.98k op/s 2024-02-12T16:34:37.075 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:37.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:37.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:37.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:36 smithi121 bash[18489]: cluster 2024-02-12T16:34:35.431069+0000 osd.3 (osd.3) 64 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:36 smithi121 bash[18489]: cluster 2024-02-12T16:34:35.431148+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:37.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:36 smithi121 bash[18489]: cluster 2024-02-12T16:34:36.111827+0000 mgr.y (mgr.14146) 1499 : cluster [DBG] pgmap v2144: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.98k op/s 2024-02-12T16:34:37.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:37.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 50670 50654 226.115 159.375 0.0048555 0.00442042 2024-02-12T16:34:37.208 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:37.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:37.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:37.389 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:37.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:37.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:37.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:37.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:37.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:37.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:37.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 10 1 2024-02-12T16:34:37.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:37.782 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:37.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:37.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:37.910 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:37.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:38.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:38.041 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:38.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:38.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:38.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 53042 53026 220.924 148.25 0.00474061 0.00452489 2024-02-12T16:34:38.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:38.298 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:38.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:38.431 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:38.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:38.564 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:38.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:38.694 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:38.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:38.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:38.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:38.964 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:38.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:39.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:39.092 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:39.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:38 smithi123 bash[19119]: cluster 2024-02-12T16:34:37.785464+0000 osd.8 (osd.8) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:38 smithi123 bash[19119]: cluster 2024-02-12T16:34:37.785526+0000 osd.8 (osd.8) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:38 smithi123 bash[19119]: cluster 2024-02-12T16:34:37.866906+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e766: 12 total, 12 up, 12 in 2024-02-12T16:34:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:38 smithi123 bash[19119]: cluster 2024-02-12T16:34:38.112249+0000 mgr.y (mgr.14146) 1500 : cluster [DBG] pgmap v2146: 47 pgs: 47 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.72k op/s 2024-02-12T16:34:39.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:38 smithi121 bash[18489]: cluster 2024-02-12T16:34:37.785464+0000 osd.8 (osd.8) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:38 smithi121 bash[18489]: cluster 2024-02-12T16:34:37.785526+0000 osd.8 (osd.8) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:38 smithi121 bash[18489]: cluster 2024-02-12T16:34:37.866906+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e766: 12 total, 12 up, 12 in 2024-02-12T16:34:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:38 smithi121 bash[18489]: cluster 2024-02-12T16:34:38.112249+0000 mgr.y (mgr.14146) 1500 : cluster [DBG] pgmap v2146: 47 pgs: 47 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.72k op/s 2024-02-12T16:34:39.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 16 55766 55750 217.756 170.25 0.0029614 0.00459085 2024-02-12T16:34:39.195 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:38 smithi169 bash[19604]: cluster 2024-02-12T16:34:37.785464+0000 osd.8 (osd.8) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:38 smithi169 bash[19604]: cluster 2024-02-12T16:34:37.785526+0000 osd.8 (osd.8) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:38 smithi169 bash[19604]: cluster 2024-02-12T16:34:37.866906+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e766: 12 total, 12 up, 12 in 2024-02-12T16:34:39.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:38 smithi169 bash[19604]: cluster 2024-02-12T16:34:38.112249+0000 mgr.y (mgr.14146) 1500 : cluster [DBG] pgmap v2146: 47 pgs: 47 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.72k op/s 2024-02-12T16:34:39.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:39.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:39.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:39.347 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:39.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:39.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:39.478 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:39.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:39.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:39.609 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:39.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:39.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:39.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:39.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:39.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:39.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:39.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:39.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:40.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:40.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:40.141 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:40.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 58372 58356 214.527 162.875 0.00411912 0.0046594 2024-02-12T16:34:40.192 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:34:40.192 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:34:40.192 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:34:40.192 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:34:40.192 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:34:40.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:40.269 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:40.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:40.398 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:34:40.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:40.436 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:34:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:34:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:34:40.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:34:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:34:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:34:40.632 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:34:40.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:40.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:34:40.678 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:40.698 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:34:40.735 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:34:40.769 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:34:40.779 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.779 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.779 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.779 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:34:40.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:40.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:40.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:40.929 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:40.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:41.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:41.062 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:41.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:41.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-12T16:34:41.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:41.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:41 smithi169 bash[19604]: cluster 2024-02-12T16:34:40.112730+0000 mgr.y (mgr.14146) 1501 : cluster [DBG] pgmap v2147: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 166 MiB/s wr, 2.66k op/s 2024-02-12T16:34:41.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:41 smithi169 bash[19604]: audit 2024-02-12T16:34:40.360556+0000 mon.b (mon.2) 170 : audit [DBG] from='client.? v1:172.21.15.121:0/927635084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:41.176 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:41 smithi169 bash[19604]: cluster 2024-02-12T16:34:40.799879+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e767: 12 total, 12 up, 12 in 2024-02-12T16:34:41.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 60700 60684 210.691 145.5 0.00461736 0.00474473 2024-02-12T16:34:41.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:41.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:41.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:41.344 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:41.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:41.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:41.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:41.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:41.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:41.585 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:41 smithi123 bash[19119]: cluster 2024-02-12T16:34:40.112730+0000 mgr.y (mgr.14146) 1501 : cluster [DBG] pgmap v2147: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 166 MiB/s wr, 2.66k op/s 2024-02-12T16:34:41.585 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:41 smithi123 bash[19119]: audit 2024-02-12T16:34:40.360556+0000 mon.b (mon.2) 170 : audit [DBG] from='client.? v1:172.21.15.121:0/927635084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:41.586 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:41 smithi123 bash[19119]: cluster 2024-02-12T16:34:40.799879+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e767: 12 total, 12 up, 12 in 2024-02-12T16:34:41.601 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:41.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:41.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:41 smithi121 bash[18489]: cluster 2024-02-12T16:34:40.112730+0000 mgr.y (mgr.14146) 1501 : cluster [DBG] pgmap v2147: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 166 MiB/s wr, 2.66k op/s 2024-02-12T16:34:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:41 smithi121 bash[18489]: audit 2024-02-12T16:34:40.360556+0000 mon.b (mon.2) 170 : audit [DBG] from='client.? v1:172.21.15.121:0/927635084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:34:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:41 smithi121 bash[18489]: cluster 2024-02-12T16:34:40.799879+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e767: 12 total, 12 up, 12 in 2024-02-12T16:34:41.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:41.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:41.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:41.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:41.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:41.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:41.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:42.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:42.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:42.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:42.184 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 63223 63207 207.901 157.688 0.00793132 0.00480833 2024-02-12T16:34:42.192 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:34:42.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:42.297 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:42.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:42.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:42 smithi121 bash[18489]: audit 2024-02-12T16:34:41.303112+0000 mon.a (mon.0) 2595 : audit [INF] from='client.? v1:172.21.15.121:0/1475181190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:34:42.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:42.431 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:42 smithi169 bash[19604]: audit 2024-02-12T16:34:41.303112+0000 mon.a (mon.0) 2595 : audit [INF] from='client.? v1:172.21.15.121:0/1475181190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:34:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:42.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:42.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:42 smithi123 bash[19119]: audit 2024-02-12T16:34:41.303112+0000 mon.a (mon.0) 2595 : audit [INF] from='client.? v1:172.21.15.121:0/1475181190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:34:42.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:42.690 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:42.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:42.826 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:42.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:42.962 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:42.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:43.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:43.103 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:43.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:43.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:34:43.181568 min lat: 0.00104071 max lat: 0.477087 avg lat: 0.00486927 2024-02-12T16:34:43.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:34:43.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 16 65716 65700 205.296 155.812 0.012695 0.00486927 2024-02-12T16:34:43.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:43.235 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:43.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:43.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:43.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:43.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:43 smithi123 bash[19119]: cluster 2024-02-12T16:34:42.113197+0000 mgr.y (mgr.14146) 1502 : cluster [DBG] pgmap v2149: 47 pgs: 47 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-02-12T16:34:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:43 smithi123 bash[19119]: cluster 2024-02-12T16:34:42.169081+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:34:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:43 smithi123 bash[19119]: audit 2024-02-12T16:34:42.179965+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? v1:172.21.15.121:0/1475181190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:34:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:43 smithi123 bash[19119]: cluster 2024-02-12T16:34:42.180077+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e768: 12 total, 12 up, 12 in 2024-02-12T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:43 smithi121 bash[18489]: cluster 2024-02-12T16:34:42.113197+0000 mgr.y (mgr.14146) 1502 : cluster [DBG] pgmap v2149: 47 pgs: 47 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-02-12T16:34:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:43 smithi121 bash[18489]: cluster 2024-02-12T16:34:42.169081+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:34:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:43 smithi121 bash[18489]: audit 2024-02-12T16:34:42.179965+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? v1:172.21.15.121:0/1475181190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:34:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:43 smithi121 bash[18489]: cluster 2024-02-12T16:34:42.180077+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e768: 12 total, 12 up, 12 in 2024-02-12T16:34:43.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:43.490 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:43.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:43 smithi169 bash[19604]: cluster 2024-02-12T16:34:42.113197+0000 mgr.y (mgr.14146) 1502 : cluster [DBG] pgmap v2149: 47 pgs: 47 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.45k op/s 2024-02-12T16:34:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:43 smithi169 bash[19604]: cluster 2024-02-12T16:34:42.169081+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:34:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:43 smithi169 bash[19604]: audit 2024-02-12T16:34:42.179965+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? v1:172.21.15.121:0/1475181190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:34:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:43 smithi169 bash[19604]: cluster 2024-02-12T16:34:42.180077+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e768: 12 total, 12 up, 12 in 2024-02-12T16:34:43.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:43.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:43.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:43.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:43.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:43.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:43.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:43.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:43.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:43.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:44.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:44.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:34:44.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:44.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:44.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 68107 68091 202.635 149.438 0.00305114 0.00493285 2024-02-12T16:34:44.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:44.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:44.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:44.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:44.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:44.535 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:44.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:44.666 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:44.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:44.796 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:44.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:44.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:44.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:45.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:45.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:45.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:45.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 70436 70420 200.04 145.562 0.0126892 0.00499725 2024-02-12T16:34:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:45.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:45.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:45.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:45.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:34:45.452 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:45 smithi121 bash[18489]: cluster 2024-02-12T16:34:44.113775+0000 mgr.y (mgr.14146) 1503 : cluster [DBG] pgmap v2151: 47 pgs: 47 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-02-12T16:34:45.452 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:45.456 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:45 smithi123 bash[19119]: cluster 2024-02-12T16:34:44.113775+0000 mgr.y (mgr.14146) 1503 : cluster [DBG] pgmap v2151: 47 pgs: 47 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-02-12T16:34:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:45 smithi169 bash[19604]: cluster 2024-02-12T16:34:44.113775+0000 mgr.y (mgr.14146) 1503 : cluster [DBG] pgmap v2151: 47 pgs: 47 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-02-12T16:34:45.559 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:45.580 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:45.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:45.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:45.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:45.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:45.926 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:34:45.935 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:45.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:45.985 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:45.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:45.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:46.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:46.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:46.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:46.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 72585 72569 197.182 134.312 0.00406642 0.00507004 2024-02-12T16:34:46.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:46.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:46.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:46.404 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:46.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:46.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:46.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:46.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:46.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:46.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:46.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:46.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:46.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:46.784 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:46.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:46.887 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:46.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:46 smithi123 bash[19119]: cluster 2024-02-12T16:34:45.803839+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e769: 12 total, 12 up, 12 in 2024-02-12T16:34:46.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:46 smithi123 bash[19119]: cluster 2024-02-12T16:34:46.114264+0000 mgr.y (mgr.14146) 1504 : cluster [DBG] pgmap v2153: 47 pgs: 47 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.73k op/s 2024-02-12T16:34:46.911 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:46.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:47.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:47.041 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:47.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:47.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:46 smithi169 bash[19604]: cluster 2024-02-12T16:34:45.803839+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e769: 12 total, 12 up, 12 in 2024-02-12T16:34:47.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:46 smithi169 bash[19604]: cluster 2024-02-12T16:34:46.114264+0000 mgr.y (mgr.14146) 1504 : cluster [DBG] pgmap v2153: 47 pgs: 47 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.73k op/s 2024-02-12T16:34:47.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:47.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:46 smithi121 bash[18489]: cluster 2024-02-12T16:34:45.803839+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e769: 12 total, 12 up, 12 in 2024-02-12T16:34:47.153 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:46 smithi121 bash[18489]: cluster 2024-02-12T16:34:46.114264+0000 mgr.y (mgr.14146) 1504 : cluster [DBG] pgmap v2153: 47 pgs: 47 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.73k op/s 2024-02-12T16:34:47.181 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:47.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:47.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 74439 74423 193.794 115.875 0.00366508 0.0051587 2024-02-12T16:34:47.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:47.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:47.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:47.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:47.461 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:47.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:47.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:47.601 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:47.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:47.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:47.732 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:47.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:47.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:47.858 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:47.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:47.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:47.970 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:48.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:48.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:48.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:48.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 76647 76631 191.562 138 0.0102748 0.00521869 2024-02-12T16:34:48.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:48.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:48.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:48.396 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:48.499 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:48.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:48.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:48.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:48.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:48.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:48.936 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:48.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:49.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:49.170 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:49.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 16 78822 78806 189.422 135.938 0.0140126 0.0052775 2024-02-12T16:34:49.198 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:49.301 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:49.324 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:49.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:49 smithi121 bash[18489]: cluster 2024-02-12T16:34:48.114784+0000 mgr.y (mgr.14146) 1505 : cluster [DBG] pgmap v2154: 47 pgs: 47 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.81k op/s 2024-02-12T16:34:49.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:49.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:49 smithi123 bash[19119]: cluster 2024-02-12T16:34:48.114784+0000 mgr.y (mgr.14146) 1505 : cluster [DBG] pgmap v2154: 47 pgs: 47 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.81k op/s 2024-02-12T16:34:49.452 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:49 smithi169 bash[19604]: cluster 2024-02-12T16:34:48.114784+0000 mgr.y (mgr.14146) 1505 : cluster [DBG] pgmap v2154: 47 pgs: 47 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.81k op/s 2024-02-12T16:34:49.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:49.582 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:49.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:49.684 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:49.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:49.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:49.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:49.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:49.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:49.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:49.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:50.000 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:34:50.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:50.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:50.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:50.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 80889 80873 187.191 129.188 0.00343039 0.00533997 2024-02-12T16:34:50.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:50.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:50.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:50.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:50.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:50.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:50.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:50.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:50.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:50.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:50.609 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:34:50.610 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:34:50.633 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:50.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:50.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:50.774 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:50.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:50.880 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:50.902 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:50.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:51.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:51.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:51.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:51.176 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:51.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 82971 82955 185.152 130.125 0.00304545 0.00539757 2024-02-12T16:34:51.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:51.306 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:51.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:51 smithi121 bash[18489]: cluster 2024-02-12T16:34:50.115261+0000 mgr.y (mgr.14146) 1506 : cluster [DBG] pgmap v2155: 47 pgs: 47 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.74k op/s 2024-02-12T16:34:51.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:51.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:51.443 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:51 smithi169 bash[19604]: cluster 2024-02-12T16:34:50.115261+0000 mgr.y (mgr.14146) 1506 : cluster [DBG] pgmap v2155: 47 pgs: 47 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.74k op/s 2024-02-12T16:34:51.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-12T16:34:51.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:51.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:51 smithi123 bash[19119]: cluster 2024-02-12T16:34:50.115261+0000 mgr.y (mgr.14146) 1506 : cluster [DBG] pgmap v2155: 47 pgs: 47 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.74k op/s 2024-02-12T16:34:51.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:51.700 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:51.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:51.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:51.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:51.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:51.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:52.076 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:52.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:52.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:52.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 84462 84446 181.981 93.1875 0.00558017 0.00549355 2024-02-12T16:34:52.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:52.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:52.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:52.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:52.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:52.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:52.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:52.507 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:52.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:52.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:52.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:52.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:52.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:52.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:52.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:52.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:52.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:52.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:53.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:53.050 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:53.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:53.180 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:53.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 86448 86432 180.052 124.125 0.00474579 0.00555204 2024-02-12T16:34:53.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:53.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:53.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:53.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:53 smithi121 bash[18489]: cluster 2024-02-12T16:34:52.115754+0000 mgr.y (mgr.14146) 1507 : cluster [DBG] pgmap v2156: 47 pgs: 47 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.26k op/s 2024-02-12T16:34:53.418 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:53 smithi169 bash[19604]: cluster 2024-02-12T16:34:52.115754+0000 mgr.y (mgr.14146) 1507 : cluster [DBG] pgmap v2156: 47 pgs: 47 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.26k op/s 2024-02-12T16:34:53.435 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:53.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:53.567 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:53 smithi123 bash[19119]: cluster 2024-02-12T16:34:52.115754+0000 mgr.y (mgr.14146) 1507 : cluster [DBG] pgmap v2156: 47 pgs: 47 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.26k op/s 2024-02-12T16:34:53.670 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:53.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:53.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:53.844 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:53.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:53.976 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:53.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:54.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:54.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:54.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 88145 88129 177.665 106.062 0.00399633 0.00562705 2024-02-12T16:34:54.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:54.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:54.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:54.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:54.370 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:54.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:54.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:54.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:54.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:54.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:54.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:54.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:54.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:54.763 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:54.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:54.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:54.894 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:54.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:54.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:55.031 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:55.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:55.163 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:55.185 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 89910 89894 175.56 110.312 0.0085719 0.0056943 2024-02-12T16:34:55.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:55.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:55.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:55 smithi121 bash[18489]: cluster 2024-02-12T16:34:53.828008+0000 osd.1 (osd.1) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:55.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:55 smithi121 bash[18489]: cluster 2024-02-12T16:34:53.828078+0000 osd.1 (osd.1) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:55.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:55 smithi121 bash[18489]: cluster 2024-02-12T16:34:54.116194+0000 mgr.y (mgr.14146) 1508 : cluster [DBG] pgmap v2157: 47 pgs: 47 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-02-12T16:34:55.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:55 smithi121 bash[18489]: cluster 2024-02-12T16:34:54.185644+0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e770: 12 total, 12 up, 12 in 2024-02-12T16:34:55.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:55.422 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:55.527 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:55.534 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:55 smithi123 bash[19119]: cluster 2024-02-12T16:34:53.828008+0000 osd.1 (osd.1) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:55.534 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:55 smithi123 bash[19119]: cluster 2024-02-12T16:34:53.828078+0000 osd.1 (osd.1) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:55.534 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:55 smithi123 bash[19119]: cluster 2024-02-12T16:34:54.116194+0000 mgr.y (mgr.14146) 1508 : cluster [DBG] pgmap v2157: 47 pgs: 47 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-02-12T16:34:55.535 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:55 smithi123 bash[19119]: cluster 2024-02-12T16:34:54.185644+0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e770: 12 total, 12 up, 12 in 2024-02-12T16:34:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:55 smithi169 bash[19604]: cluster 2024-02-12T16:34:53.828008+0000 osd.1 (osd.1) 116 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:34:55.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:55 smithi169 bash[19604]: cluster 2024-02-12T16:34:53.828078+0000 osd.1 (osd.1) 117 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:34:55.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:55 smithi169 bash[19604]: cluster 2024-02-12T16:34:54.116194+0000 mgr.y (mgr.14146) 1508 : cluster [DBG] pgmap v2157: 47 pgs: 47 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-02-12T16:34:55.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:55 smithi169 bash[19604]: cluster 2024-02-12T16:34:54.185644+0000 mon.a (mon.0) 2600 : cluster [DBG] osdmap e770: 12 total, 12 up, 12 in 2024-02-12T16:34:55.551 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:34:55.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:34:55.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:55.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:55.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:55.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:55.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:55.950 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:55.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:56.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:56.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:34:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:34:56.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 91827 91811 173.87 119.812 0.00856969 0.00574976 2024-02-12T16:34:56.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:56.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:34:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:34:56.252 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:34:56.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:56.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:56.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:56.460 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:56.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:56.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:34:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:34:56.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:56.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:56.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:56.860 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:56.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:56.973 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:56 smithi169 bash[19604]: cluster 2024-02-12T16:34:55.804255+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e771: 12 total, 12 up, 12 in 2024-02-12T16:34:56.974 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:56 smithi169 bash[19604]: cluster 2024-02-12T16:34:56.116622+0000 mgr.y (mgr.14146) 1509 : cluster [DBG] pgmap v2160: 47 pgs: 47 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-12T16:34:56.988 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:56.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:57.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:57.128 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:57.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:57.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:56 smithi123 bash[19119]: cluster 2024-02-12T16:34:55.804255+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e771: 12 total, 12 up, 12 in 2024-02-12T16:34:57.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:56 smithi123 bash[19119]: cluster 2024-02-12T16:34:56.116622+0000 mgr.y (mgr.14146) 1509 : cluster [DBG] pgmap v2160: 47 pgs: 47 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-12T16:34:57.140 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:56 smithi121 bash[18489]: cluster 2024-02-12T16:34:55.804255+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e771: 12 total, 12 up, 12 in 2024-02-12T16:34:57.141 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:56 smithi121 bash[18489]: cluster 2024-02-12T16:34:56.116622+0000 mgr.y (mgr.14146) 1509 : cluster [DBG] pgmap v2160: 47 pgs: 47 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.79k op/s 2024-02-12T16:34:57.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 93099 93083 171.094 79.5 0.00423612 0.0058432 2024-02-12T16:34:57.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:57.256 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:57.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:57.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:57.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:57.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:57.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:57.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:57.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:34:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:57.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:34:57.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:57.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:57.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:57.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:57.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:57.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:58.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:58.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:58.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:58.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:58.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 94898 94882 169.418 112.438 0.00698183 0.00590115 2024-02-12T16:34:58.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:58.318 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:58.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:58.446 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:58.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:34:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:34:58.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:58.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:58.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:58.836 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:58.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:58.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:58.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:59.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:59.108 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:34:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:59.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:34:59.181 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.10 to 1 (10000) 2024-02-12T16:34:59.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 16 96413 96397 167.342 94.6875 0.00609025 0.00596921 2024-02-12T16:34:59.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:59.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:34:59.233 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:59 smithi121 bash[18489]: cluster 2024-02-12T16:34:58.117117+0000 mgr.y (mgr.14146) 1510 : cluster [DBG] pgmap v2161: 47 pgs: 47 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-02-12T16:34:59.233 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:34:59 smithi121 bash[18489]: audit 2024-02-12T16:34:58.975903+0000 mon.a (mon.0) 2602 : audit [INF] from='client.? v1:172.21.15.121:0/3648070297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:34:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:59.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:34:59.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:59.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:34:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:59.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:34:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:59 smithi123 bash[19119]: cluster 2024-02-12T16:34:58.117117+0000 mgr.y (mgr.14146) 1510 : cluster [DBG] pgmap v2161: 47 pgs: 47 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-02-12T16:34:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:34:59 smithi123 bash[19119]: audit 2024-02-12T16:34:58.975903+0000 mon.a (mon.0) 2602 : audit [INF] from='client.? v1:172.21.15.121:0/3648070297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:34:59.483 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:59.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:59 smithi169 bash[19604]: cluster 2024-02-12T16:34:58.117117+0000 mgr.y (mgr.14146) 1510 : cluster [DBG] pgmap v2161: 47 pgs: 47 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-02-12T16:34:59.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:34:59 smithi169 bash[19604]: audit 2024-02-12T16:34:58.975903+0000 mon.a (mon.0) 2602 : audit [INF] from='client.? v1:172.21.15.121:0/3648070297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:34:59.507 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:34:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:59.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:34:59.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:59.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:34:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:59.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:34:59.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:59.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:34:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:34:59.920 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:34:59.921 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:34:59.921 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:34:59.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:34:59.928 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:34:59.960 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:34:59.992 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:35:00.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:00.025 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:35:00.065 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:35:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:00.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:00.096 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:35:00.129 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:35:00.171 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:35:00.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:00.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 97877 97861 165.292 91.5 0.0123745 0.0060482 2024-02-12T16:35:00.207 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:35:00.211 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.212 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:35:00.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:00.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:00.288 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:00 smithi121 bash[18489]: audit 2024-02-12T16:34:59.173401+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? v1:172.21.15.121:0/3648070297' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:35:00.288 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:00 smithi121 bash[18489]: cluster 2024-02-12T16:34:59.173468+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e772: 12 total, 12 up, 12 in 2024-02-12T16:35:00.288 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:00 smithi121 bash[18489]: audit 2024-02-12T16:34:59.786740+0000 mon.a (mon.0) 2605 : audit [DBG] from='client.? v1:172.21.15.121:0/3262815410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:00.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:00.337 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:00.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:00.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:00.447 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:00 smithi169 bash[19604]: audit 2024-02-12T16:34:59.173401+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? v1:172.21.15.121:0/3648070297' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:35:00.447 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:00 smithi169 bash[19604]: cluster 2024-02-12T16:34:59.173468+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e772: 12 total, 12 up, 12 in 2024-02-12T16:35:00.448 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:00 smithi169 bash[19604]: audit 2024-02-12T16:34:59.786740+0000 mon.a (mon.0) 2605 : audit [DBG] from='client.? v1:172.21.15.121:0/3262815410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:00.481 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:00.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:00.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:98619 7.a 7:5131db29:::benchmark_data_smithi169_105252_object98618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e772)", 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:35:00.595864+0000", 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0093122320000000001, 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0093430649999999994, 2024-02-12T16:35:00.609 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 98619 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.595864+0000", 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.595864+0000", 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.595861+0000", 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7152999999999999e-05 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.595889+0000", 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6677999999999999e-05 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.595905+0000", 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0369999999999997e-06 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.595912+0000", 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8711999999999997e-05 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.595961+0000", 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015806300000000001 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.596119+0000", 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017437 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.596293+0000", 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052207800000000004 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.601514+0000", 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00565008 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:00.613 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:98621 7.a 7:555eedef:::benchmark_data_smithi169_105252_object98620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e772)", 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:35:00.598539+0000", 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0066375119999999999, 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067271839999999998, 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 98621 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:00.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.598539+0000", 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.598539+0000", 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.598536+0000", 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0131999999999999e-05 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.598566+0000", 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8726e-05 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.598585+0000", 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7660000000000004e-06 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.598594+0000", 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052143899999999995 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.599115+0000", 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101876 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.599217+0000", 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011127 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.599328+0000", 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023154059999999999 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:00.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:00.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:35:00.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:00.601644+0000", 2024-02-12T16:35:00.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0031049950000000002 2024-02-12T16:35:00.618 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:00.618 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:00.619 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:00.619 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:00.619 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-02-12T16:35:00.619 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 2 2024-02-12T16:35:00.619 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:35:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:00.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:00.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:00 smithi123 bash[19119]: audit 2024-02-12T16:34:59.173401+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? v1:172.21.15.121:0/3648070297' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:35:00.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:00 smithi123 bash[19119]: cluster 2024-02-12T16:34:59.173468+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e772: 12 total, 12 up, 12 in 2024-02-12T16:35:00.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:00 smithi123 bash[19119]: audit 2024-02-12T16:34:59.786740+0000 mon.a (mon.0) 2605 : audit [DBG] from='client.? v1:172.21.15.121:0/3262815410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:00.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:00.750 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:00.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:00.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:00.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:00.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:00.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 11 1 2024-02-12T16:35:00.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:01.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:01.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:01.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:01.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:01.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:01.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 16 99542 99526 163.681 104.062 0.00630398 0.00610737 2024-02-12T16:35:01.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:01.269 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:01.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:01.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:01.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:01 smithi123 bash[19119]: cluster 2024-02-12T16:35:00.117562+0000 mgr.y (mgr.14146) 1511 : cluster [DBG] pgmap v2163: 47 pgs: 47 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-02-12T16:35:01.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:01 smithi123 bash[19119]: cluster 2024-02-12T16:35:00.809834+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e773: 12 total, 12 up, 12 in 2024-02-12T16:35:01.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:01.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:01.449 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:01 smithi169 bash[19604]: cluster 2024-02-12T16:35:00.117562+0000 mgr.y (mgr.14146) 1511 : cluster [DBG] pgmap v2163: 47 pgs: 47 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-02-12T16:35:01.449 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:01 smithi169 bash[19604]: cluster 2024-02-12T16:35:00.809834+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e773: 12 total, 12 up, 12 in 2024-02-12T16:35:01.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-12T16:35:01.456 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:01 smithi121 bash[18489]: cluster 2024-02-12T16:35:00.117562+0000 mgr.y (mgr.14146) 1511 : cluster [DBG] pgmap v2163: 47 pgs: 47 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-02-12T16:35:01.456 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:01 smithi121 bash[18489]: cluster 2024-02-12T16:35:00.809834+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e773: 12 total, 12 up, 12 in 2024-02-12T16:35:01.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:01.535 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:01.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:01.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:01.673 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:01.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:01.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:01.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:01.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:01.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:01.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:02.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:02.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:02.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 101283 101267 162.273 108.812 0.012703 0.00616064 2024-02-12T16:35:02.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:02.284 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:02.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:02.415 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:02.519 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:02.541 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:02.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:02.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:02.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:02.820 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:02.925 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:02.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:02.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:03.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:03.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:03.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:03.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:35:03.183249 min lat: 0.00104071 max lat: 0.477087 avg lat: 0.00621239 2024-02-12T16:35:03.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:35:03.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 103017 103001 160.926 108.375 0.00426979 0.00621239 2024-02-12T16:35:03.191 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:35:03.219 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:03.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:03.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:03 smithi121 bash[18489]: cluster 2024-02-12T16:35:02.118027+0000 mgr.y (mgr.14146) 1512 : cluster [DBG] pgmap v2165: 47 pgs: 47 active+clean; 6.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-02-12T16:35:03.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:03 smithi121 bash[18489]: audit 2024-02-12T16:35:02.431347+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? v1:172.21.15.121:0/1355915110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:35:03.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:03.343 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:03.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:03.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:03.456 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:03 smithi169 bash[19604]: cluster 2024-02-12T16:35:02.118027+0000 mgr.y (mgr.14146) 1512 : cluster [DBG] pgmap v2165: 47 pgs: 47 active+clean; 6.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-02-12T16:35:03.456 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:03 smithi169 bash[19604]: audit 2024-02-12T16:35:02.431347+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? v1:172.21.15.121:0/1355915110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:35:03.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:03.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:03.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:03.586 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:03 smithi123 bash[19119]: cluster 2024-02-12T16:35:02.118027+0000 mgr.y (mgr.14146) 1512 : cluster [DBG] pgmap v2165: 47 pgs: 47 active+clean; 6.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-02-12T16:35:03.586 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:03 smithi123 bash[19119]: audit 2024-02-12T16:35:02.431347+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? v1:172.21.15.121:0/1355915110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:35:03.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:03.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:03.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:03.741 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:03.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:03.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:03.870 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:03.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:03.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:04.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:35:04.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 16 104718 104702 159.593 106.312 0.00696736 0.00626444 2024-02-12T16:35:04.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:04.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:04.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:04.458 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:02.759315+0000 osd.11 (osd.11) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:04.458 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:02.759370+0000 osd.11 (osd.11) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:04.458 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.186938+0000 mon.a (mon.0) 2608 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:35:04.458 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: audit 2024-02-12T16:35:03.196954+0000 mon.a (mon.0) 2609 : audit [INF] from='client.? v1:172.21.15.121:0/1355915110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.197060+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e774: 12 total, 12 up, 12 in 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.708861+0000 osd.9 (osd.9) 129 : cluster [DBG] 1.a scrub starts 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.750647+0000 osd.9 (osd.9) 130 : cluster [DBG] 1.a scrub ok 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.792944+0000 mon.a (mon.0) 2611 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=53,logm=6} crc {auth=4208671689,config=2903391811,health=4117320282,kv=4090957530,logm=503337891}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.794633+0000 mon.a (mon.0) 2612 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630279841}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.795484+0000 mon.a (mon.0) 2613 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314806009}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.796473+0000 mon.a (mon.0) 2614 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605969326}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.799160+0000 mon.a (mon.0) 2615 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636667940}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.800113+0000 mon.a (mon.0) 2616 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3240772620}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.804026+0000 mon.a (mon.0) 2617 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133098811}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.811837+0000 mon.a (mon.0) 2618 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610417284}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.816095+0000 mon.a (mon.0) 2619 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3090667846}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.819820+0000 mon.a (mon.0) 2620 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1837386276}) 2024-02-12T16:35:04.459 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.823847+0000 mon.a (mon.0) 2621 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469661242}) 2024-02-12T16:35:04.460 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.825203+0000 mon.a (mon.0) 2622 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=9,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=51} crc {logm=1222156288,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3613978866,mon_config_key=451609174}) 2024-02-12T16:35:04.460 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.826022+0000 mon.a (mon.0) 2623 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=829173151}) 2024-02-12T16:35:04.460 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.827400+0000 mon.a (mon.0) 2624 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=95,monmap=5} crc {mon_config_key=191894012,monmap=2720272532}) 2024-02-12T16:35:04.460 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.828208+0000 mon.a (mon.0) 2625 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=12,osd_pg_creating=1,osdmap=87} crc {osd_metadata=1720028586,osd_pg_creating=3434793684,osdmap=3829217889}) 2024-02-12T16:35:04.460 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.829761+0000 mon.a (mon.0) 2626 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1863709120}) 2024-02-12T16:35:04.460 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:04 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.837701+0000 mon.a (mon.0) 2627 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=6} crc {osdmap=1183972069}) 2024-02-12T16:35:04.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:04.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:02.759315+0000 osd.11 (osd.11) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:02.759370+0000 osd.11 (osd.11) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.186938+0000 mon.a (mon.0) 2608 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: audit 2024-02-12T16:35:03.196954+0000 mon.a (mon.0) 2609 : audit [INF] from='client.? v1:172.21.15.121:0/1355915110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.197060+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e774: 12 total, 12 up, 12 in 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.708861+0000 osd.9 (osd.9) 129 : cluster [DBG] 1.a scrub starts 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.750647+0000 osd.9 (osd.9) 130 : cluster [DBG] 1.a scrub ok 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.792944+0000 mon.a (mon.0) 2611 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=53,logm=6} crc {auth=4208671689,config=2903391811,health=4117320282,kv=4090957530,logm=503337891}) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.794633+0000 mon.a (mon.0) 2612 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630279841}) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.795484+0000 mon.a (mon.0) 2613 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314806009}) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.796473+0000 mon.a (mon.0) 2614 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605969326}) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.799160+0000 mon.a (mon.0) 2615 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636667940}) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.800113+0000 mon.a (mon.0) 2616 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3240772620}) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.804026+0000 mon.a (mon.0) 2617 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133098811}) 2024-02-12T16:35:04.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.811837+0000 mon.a (mon.0) 2618 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610417284}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.816095+0000 mon.a (mon.0) 2619 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3090667846}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.819820+0000 mon.a (mon.0) 2620 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1837386276}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.823847+0000 mon.a (mon.0) 2621 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469661242}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.825203+0000 mon.a (mon.0) 2622 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=9,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=51} crc {logm=1222156288,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3613978866,mon_config_key=451609174}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.826022+0000 mon.a (mon.0) 2623 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=829173151}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.827400+0000 mon.a (mon.0) 2624 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=95,monmap=5} crc {mon_config_key=191894012,monmap=2720272532}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.828208+0000 mon.a (mon.0) 2625 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=12,osd_pg_creating=1,osdmap=87} crc {osd_metadata=1720028586,osd_pg_creating=3434793684,osdmap=3829217889}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.829761+0000 mon.a (mon.0) 2626 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1863709120}) 2024-02-12T16:35:04.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:04 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.837701+0000 mon.a (mon.0) 2627 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=6} crc {osdmap=1183972069}) 2024-02-12T16:35:04.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:04.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:04.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:02.759315+0000 osd.11 (osd.11) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:02.759370+0000 osd.11 (osd.11) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.186938+0000 mon.a (mon.0) 2608 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:35:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: audit 2024-02-12T16:35:03.196954+0000 mon.a (mon.0) 2609 : audit [INF] from='client.? v1:172.21.15.121:0/1355915110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:35:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.197060+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e774: 12 total, 12 up, 12 in 2024-02-12T16:35:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.708861+0000 osd.9 (osd.9) 129 : cluster [DBG] 1.a scrub starts 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.750647+0000 osd.9 (osd.9) 130 : cluster [DBG] 1.a scrub ok 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.792944+0000 mon.a (mon.0) 2611 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=53,logm=6} crc {auth=4208671689,config=2903391811,health=4117320282,kv=4090957530,logm=503337891}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.794633+0000 mon.a (mon.0) 2612 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630279841}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.795484+0000 mon.a (mon.0) 2613 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314806009}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.796473+0000 mon.a (mon.0) 2614 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605969326}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.799160+0000 mon.a (mon.0) 2615 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636667940}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.800113+0000 mon.a (mon.0) 2616 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3240772620}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.804026+0000 mon.a (mon.0) 2617 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4133098811}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.811837+0000 mon.a (mon.0) 2618 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1610417284}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.816095+0000 mon.a (mon.0) 2619 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3090667846}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.819820+0000 mon.a (mon.0) 2620 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1837386276}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.823847+0000 mon.a (mon.0) 2621 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3469661242}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.825203+0000 mon.a (mon.0) 2622 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=9,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=51} crc {logm=1222156288,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3613978866,mon_config_key=451609174}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.826022+0000 mon.a (mon.0) 2623 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=829173151}) 2024-02-12T16:35:04.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.827400+0000 mon.a (mon.0) 2624 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=95,monmap=5} crc {mon_config_key=191894012,monmap=2720272532}) 2024-02-12T16:35:04.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.828208+0000 mon.a (mon.0) 2625 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=12,osd_pg_creating=1,osdmap=87} crc {osd_metadata=1720028586,osd_pg_creating=3434793684,osdmap=3829217889}) 2024-02-12T16:35:04.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.829761+0000 mon.a (mon.0) 2626 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1863709120}) 2024-02-12T16:35:04.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:04 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.837701+0000 mon.a (mon.0) 2627 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=6} crc {osdmap=1183972069}) 2024-02-12T16:35:04.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:04.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:04.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:04.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:04.970 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:04.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:05.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:05.099 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:05.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:05.186 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 106390 106374 158.281 104.5 0.0120929 0.00631607 2024-02-12T16:35:05.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:05.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:05 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.575831+0000 osd.7 (osd.7) 188 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:35:05.204 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:05 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.687665+0000 osd.7 (osd.7) 189 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:35:05.205 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:05 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.841295+0000 osd.5 (osd.5) 155 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:35:05.205 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:05 smithi169 bash[19604]: cluster 2024-02-12T16:35:03.857733+0000 osd.5 (osd.5) 156 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:35:05.205 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:05 smithi169 bash[19604]: cluster 2024-02-12T16:35:04.118491+0000 mgr.y (mgr.14146) 1513 : cluster [DBG] pgmap v2167: 47 pgs: 47 active+clean; 6.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:35:05.229 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:05.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:05.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:05.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:05.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:05.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:05.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:05 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.575831+0000 osd.7 (osd.7) 188 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:35:05.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:05 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.687665+0000 osd.7 (osd.7) 189 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:35:05.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:05 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.841295+0000 osd.5 (osd.5) 155 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:35:05.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:05 smithi121 bash[18489]: cluster 2024-02-12T16:35:03.857733+0000 osd.5 (osd.5) 156 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:35:05.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:05 smithi121 bash[18489]: cluster 2024-02-12T16:35:04.118491+0000 mgr.y (mgr.14146) 1513 : cluster [DBG] pgmap v2167: 47 pgs: 47 active+clean; 6.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:35:05.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:05 smithi121 bash[18489]: cluster 2024-02-12T16:35:04.303116+0000 osd.8 (osd.8) 85 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:35:05.483 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:05 smithi121 bash[18489]: cluster 2024-02-12T16:35:04.397370+0000 osd.8 (osd.8) 86 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:35:05.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:05.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:05 smithi169 bash[19604]: cluster 2024-02-12T16:35:04.303116+0000 osd.8 (osd.8) 85 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:35:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:05 smithi169 bash[19604]: cluster 2024-02-12T16:35:04.397370+0000 osd.8 (osd.8) 86 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:35:05.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:05 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.575831+0000 osd.7 (osd.7) 188 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:35:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:05 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.687665+0000 osd.7 (osd.7) 189 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:35:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:05 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.841295+0000 osd.5 (osd.5) 155 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:35:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:05 smithi123 bash[19119]: cluster 2024-02-12T16:35:03.857733+0000 osd.5 (osd.5) 156 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:35:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:05 smithi123 bash[19119]: cluster 2024-02-12T16:35:04.118491+0000 mgr.y (mgr.14146) 1513 : cluster [DBG] pgmap v2167: 47 pgs: 47 active+clean; 6.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-02-12T16:35:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:05 smithi123 bash[19119]: cluster 2024-02-12T16:35:04.303116+0000 osd.8 (osd.8) 85 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:35:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:05 smithi123 bash[19119]: cluster 2024-02-12T16:35:04.397370+0000 osd.8 (osd.8) 86 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:35:05.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:05.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:05.747 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:05.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:05.781 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:05.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:05.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:05.913 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:05.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:06.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:06.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:06.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:06.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:06.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 16 107999 107983 156.939 100.562 0.0149821 0.00637009 2024-02-12T16:35:06.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:06.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:06.233 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:35:06.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:06.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:06.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:06.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:06.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:06.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:06.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:06.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:06.723 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:06.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:06.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:06.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:06.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:06.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:06.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:06 smithi123 bash[19119]: cluster 2024-02-12T16:35:05.810057+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e775: 12 total, 12 up, 12 in 2024-02-12T16:35:06.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:06 smithi123 bash[19119]: cluster 2024-02-12T16:35:06.118889+0000 mgr.y (mgr.14146) 1514 : cluster [DBG] pgmap v2169: 47 pgs: 47 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.75k op/s 2024-02-12T16:35:07.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:07.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:35:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:06 smithi169 bash[19604]: cluster 2024-02-12T16:35:05.810057+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e775: 12 total, 12 up, 12 in 2024-02-12T16:35:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:06 smithi169 bash[19604]: cluster 2024-02-12T16:35:06.118889+0000 mgr.y (mgr.14146) 1514 : cluster [DBG] pgmap v2169: 47 pgs: 47 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.75k op/s 2024-02-12T16:35:07.109 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:07.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:06 smithi121 bash[18489]: cluster 2024-02-12T16:35:05.810057+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e775: 12 total, 12 up, 12 in 2024-02-12T16:35:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:06 smithi121 bash[18489]: cluster 2024-02-12T16:35:06.118889+0000 mgr.y (mgr.14146) 1514 : cluster [DBG] pgmap v2169: 47 pgs: 47 active+clean; 6.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.75k op/s 2024-02-12T16:35:07.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 109640 109624 155.703 102.562 0.0288817 0.00642076 2024-02-12T16:35:07.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:07.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:07.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:07.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:07.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:07.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:07.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:07.685 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:07.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:07.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:07.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:07.964 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:07.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:08.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:08.076 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:07 smithi121 bash[18489]: cluster 2024-02-12T16:35:06.327627+0000 osd.11 (osd.11) 170 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:35:08.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:07 smithi121 bash[18489]: cluster 2024-02-12T16:35:06.330502+0000 osd.11 (osd.11) 171 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:35:08.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:07 smithi121 bash[18489]: cluster 2024-02-12T16:35:06.708017+0000 osd.10 (osd.10) 156 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:35:08.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:07 smithi121 bash[18489]: cluster 2024-02-12T16:35:06.735405+0000 osd.10 (osd.10) 157 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:35:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:08.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:08.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:07 smithi169 bash[19604]: cluster 2024-02-12T16:35:06.327627+0000 osd.11 (osd.11) 170 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:35:08.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:07 smithi169 bash[19604]: cluster 2024-02-12T16:35:06.330502+0000 osd.11 (osd.11) 171 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:35:08.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:07 smithi169 bash[19604]: cluster 2024-02-12T16:35:06.708017+0000 osd.10 (osd.10) 156 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:35:08.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:07 smithi169 bash[19604]: cluster 2024-02-12T16:35:06.735405+0000 osd.10 (osd.10) 157 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:35:08.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:07 smithi123 bash[19119]: cluster 2024-02-12T16:35:06.327627+0000 osd.11 (osd.11) 170 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:35:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:07 smithi123 bash[19119]: cluster 2024-02-12T16:35:06.330502+0000 osd.11 (osd.11) 171 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:35:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:07 smithi123 bash[19119]: cluster 2024-02-12T16:35:06.708017+0000 osd.10 (osd.10) 156 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:35:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:07 smithi123 bash[19119]: cluster 2024-02-12T16:35:06.735405+0000 osd.10 (osd.10) 157 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:35:08.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 111256 111240 154.487 101 0.00501493 0.00647097 2024-02-12T16:35:08.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:08.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:08.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:08.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:08.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:08.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:08.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:08.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:08.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:08.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:08.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:08.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:08.750 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:08.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:08.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:08.884 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:08.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:08 smithi123 bash[19119]: cluster 2024-02-12T16:35:07.642039+0000 osd.2 (osd.2) 107 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:35:08.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:08 smithi123 bash[19119]: cluster 2024-02-12T16:35:07.718050+0000 osd.2 (osd.2) 108 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:35:08.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:08 smithi123 bash[19119]: cluster 2024-02-12T16:35:07.766444+0000 osd.5 (osd.5) 157 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:35:08.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:08 smithi123 bash[19119]: cluster 2024-02-12T16:35:07.846664+0000 osd.5 (osd.5) 158 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:35:08.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:08 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.119368+0000 mgr.y (mgr.14146) 1515 : cluster [DBG] pgmap v2170: 47 pgs: 47 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:35:08.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:08 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.604014+0000 osd.1 (osd.1) 118 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:35:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:08.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:08.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:09.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:09.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:09.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:09.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:08 smithi121 bash[18489]: cluster 2024-02-12T16:35:07.642039+0000 osd.2 (osd.2) 107 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:35:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:08 smithi121 bash[18489]: cluster 2024-02-12T16:35:07.718050+0000 osd.2 (osd.2) 108 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:35:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:08 smithi121 bash[18489]: cluster 2024-02-12T16:35:07.766444+0000 osd.5 (osd.5) 157 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:35:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:08 smithi121 bash[18489]: cluster 2024-02-12T16:35:07.846664+0000 osd.5 (osd.5) 158 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:35:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:08 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.119368+0000 mgr.y (mgr.14146) 1515 : cluster [DBG] pgmap v2170: 47 pgs: 47 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:35:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:08 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.604014+0000 osd.1 (osd.1) 118 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:35:09.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 15 112772 112757 153.19 94.8125 0.00751078 0.00652643 2024-02-12T16:35:09.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:09.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:08 smithi169 bash[19604]: cluster 2024-02-12T16:35:07.642039+0000 osd.2 (osd.2) 107 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:35:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:08 smithi169 bash[19604]: cluster 2024-02-12T16:35:07.718050+0000 osd.2 (osd.2) 108 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:35:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:08 smithi169 bash[19604]: cluster 2024-02-12T16:35:07.766444+0000 osd.5 (osd.5) 157 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:35:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:08 smithi169 bash[19604]: cluster 2024-02-12T16:35:07.846664+0000 osd.5 (osd.5) 158 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:35:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:08 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.119368+0000 mgr.y (mgr.14146) 1515 : cluster [DBG] pgmap v2170: 47 pgs: 47 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.90k op/s 2024-02-12T16:35:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:08 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.604014+0000 osd.1 (osd.1) 118 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:35:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:09.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:09.430 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:09.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:09.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:09.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:09.691 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:09.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:09.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.291046+0000 osd.8 (osd.8) 87 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:35:09.847 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:09.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:09.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:09.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:10.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.385262+0000 osd.8 (osd.8) 88 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.432667+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.432723+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.698900+0000 osd.2 (osd.2) 109 : cluster [DBG] 7.e scrub starts 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.802014+0000 osd.9 (osd.9) 131 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.806565+0000 osd.1 (osd.1) 119 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.825931+0000 osd.9 (osd.9) 132 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:35:10.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.827850+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e776: 12 total, 12 up, 12 in 2024-02-12T16:35:10.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:08.932870+0000 osd.0 (osd.0) 126 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:35:10.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:09 smithi121 bash[18489]: cluster 2024-02-12T16:35:09.018313+0000 osd.0 (osd.0) 127 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:35:10.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:10.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:10.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.291046+0000 osd.8 (osd.8) 87 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.385262+0000 osd.8 (osd.8) 88 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.432667+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.432723+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.698900+0000 osd.2 (osd.2) 109 : cluster [DBG] 7.e scrub starts 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.802014+0000 osd.9 (osd.9) 131 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.806565+0000 osd.1 (osd.1) 119 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.825931+0000 osd.9 (osd.9) 132 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.827850+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e776: 12 total, 12 up, 12 in 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:08.932870+0000 osd.0 (osd.0) 126 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:35:10.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:09 smithi123 bash[19119]: cluster 2024-02-12T16:35:09.018313+0000 osd.0 (osd.0) 127 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:35:10.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 16 114302 114286 151.963 95.5625 0.00431767 0.00657879 2024-02-12T16:35:10.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:10.254 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.291046+0000 osd.8 (osd.8) 87 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:35:10.254 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.385262+0000 osd.8 (osd.8) 88 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:35:10.254 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.432667+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:10.254 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.432723+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:10.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.698900+0000 osd.2 (osd.2) 109 : cluster [DBG] 7.e scrub starts 2024-02-12T16:35:10.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.802014+0000 osd.9 (osd.9) 131 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:35:10.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.806565+0000 osd.1 (osd.1) 119 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:35:10.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.825931+0000 osd.9 (osd.9) 132 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:35:10.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.827850+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e776: 12 total, 12 up, 12 in 2024-02-12T16:35:10.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:08.932870+0000 osd.0 (osd.0) 126 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:35:10.255 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:09 smithi169 bash[19604]: cluster 2024-02-12T16:35:09.018313+0000 osd.0 (osd.0) 127 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:35:10.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:10.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:10.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:10.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:10.521 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:35:10.521 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:35:10.521 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:35:10.521 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:35:10.521 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:35:10.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:10.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:10.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:10.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:10.697 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:10.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:10.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:10.839 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:10.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:10.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:10.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:10.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:11.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:11.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:10 smithi169 bash[19604]: cluster 2024-02-12T16:35:10.119853+0000 mgr.y (mgr.14146) 1516 : cluster [DBG] pgmap v2172: 47 pgs: 47 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-12T16:35:11.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:10 smithi169 bash[19604]: cluster 2024-02-12T16:35:10.266714+0000 osd.11 (osd.11) 172 : cluster [DBG] 1.b scrub starts 2024-02-12T16:35:11.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:10 smithi169 bash[19604]: cluster 2024-02-12T16:35:10.352916+0000 osd.4 (osd.4) 128 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:35:11.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:10 smithi169 bash[19604]: cluster 2024-02-12T16:35:10.432226+0000 osd.4 (osd.4) 129 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:35:11.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:10 smithi169 bash[19604]: cluster 2024-02-12T16:35:10.811080+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e777: 12 total, 12 up, 12 in 2024-02-12T16:35:11.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:11.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:10 smithi123 bash[19119]: cluster 2024-02-12T16:35:10.119853+0000 mgr.y (mgr.14146) 1516 : cluster [DBG] pgmap v2172: 47 pgs: 47 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-12T16:35:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:10 smithi123 bash[19119]: cluster 2024-02-12T16:35:10.266714+0000 osd.11 (osd.11) 172 : cluster [DBG] 1.b scrub starts 2024-02-12T16:35:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:10 smithi123 bash[19119]: cluster 2024-02-12T16:35:10.352916+0000 osd.4 (osd.4) 128 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:35:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:10 smithi123 bash[19119]: cluster 2024-02-12T16:35:10.432226+0000 osd.4 (osd.4) 129 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:35:11.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:10 smithi123 bash[19119]: cluster 2024-02-12T16:35:10.811080+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e777: 12 total, 12 up, 12 in 2024-02-12T16:35:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:10 smithi121 bash[18489]: cluster 2024-02-12T16:35:10.119853+0000 mgr.y (mgr.14146) 1516 : cluster [DBG] pgmap v2172: 47 pgs: 47 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-02-12T16:35:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:10 smithi121 bash[18489]: cluster 2024-02-12T16:35:10.266714+0000 osd.11 (osd.11) 172 : cluster [DBG] 1.b scrub starts 2024-02-12T16:35:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:10 smithi121 bash[18489]: cluster 2024-02-12T16:35:10.352916+0000 osd.4 (osd.4) 128 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:35:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:10 smithi121 bash[18489]: cluster 2024-02-12T16:35:10.432226+0000 osd.4 (osd.4) 129 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:35:11.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:10 smithi121 bash[18489]: cluster 2024-02-12T16:35:10.811080+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e777: 12 total, 12 up, 12 in 2024-02-12T16:35:11.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 115808 115792 150.758 94.125 0.00720524 0.00663153 2024-02-12T16:35:11.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:11.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:11.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:11.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-12T16:35:11.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:11.385 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:11.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:11.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:11.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:11.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:11.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:11.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:11.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:11.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:11.792 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:11.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:11.832 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:11.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:11.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:11.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:11.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:11 smithi121 bash[18489]: cluster 2024-02-12T16:35:10.417812+0000 osd.11 (osd.11) 173 : cluster [DBG] 1.b scrub ok 2024-02-12T16:35:11.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:11 smithi121 bash[18489]: cluster 2024-02-12T16:35:11.386921+0000 osd.2 (osd.2) 110 : cluster [DBG] 7.e scrub ok 2024-02-12T16:35:12.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:12.045 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:11 smithi169 bash[19604]: cluster 2024-02-12T16:35:10.417812+0000 osd.11 (osd.11) 173 : cluster [DBG] 1.b scrub ok 2024-02-12T16:35:12.045 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:11 smithi169 bash[19604]: cluster 2024-02-12T16:35:11.386921+0000 osd.2 (osd.2) 110 : cluster [DBG] 7.e scrub ok 2024-02-12T16:35:12.063 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:12.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:11 smithi123 bash[19119]: cluster 2024-02-12T16:35:10.417812+0000 osd.11 (osd.11) 173 : cluster [DBG] 1.b scrub ok 2024-02-12T16:35:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:11 smithi123 bash[19119]: cluster 2024-02-12T16:35:11.386921+0000 osd.2 (osd.2) 110 : cluster [DBG] 7.e scrub ok 2024-02-12T16:35:12.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:12.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 117317 117301 149.606 94.3125 0.0118208 0.0066826 2024-02-12T16:35:12.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:12.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:12.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:12.333 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:12.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:12.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:12.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:12.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:12.569 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:12.591 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:12.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:12.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:12.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:12.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:12.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:12.844 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:12 smithi123 bash[19119]: cluster 2024-02-12T16:35:11.509500+0000 osd.7 (osd.7) 190 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:35:12.844 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:12 smithi123 bash[19119]: cluster 2024-02-12T16:35:11.592465+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:35:12.844 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:12 smithi123 bash[19119]: cluster 2024-02-12T16:35:11.670768+0000 osd.10 (osd.10) 158 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:35:12.844 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:12 smithi123 bash[19119]: cluster 2024-02-12T16:35:11.714811+0000 osd.10 (osd.10) 159 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:35:12.844 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:12 smithi123 bash[19119]: cluster 2024-02-12T16:35:11.926776+0000 osd.0 (osd.0) 128 : cluster [DBG] 7.f scrub starts 2024-02-12T16:35:12.845 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:12 smithi123 bash[19119]: cluster 2024-02-12T16:35:12.120390+0000 mgr.y (mgr.14146) 1517 : cluster [DBG] pgmap v2174: 47 pgs: 47 active+clean; 7.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-12T16:35:12.866 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:12.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:12.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:13.002 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:13.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:13.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:12 smithi169 bash[19604]: cluster 2024-02-12T16:35:11.509500+0000 osd.7 (osd.7) 190 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:35:13.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:12 smithi169 bash[19604]: cluster 2024-02-12T16:35:11.592465+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:35:13.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:12 smithi169 bash[19604]: cluster 2024-02-12T16:35:11.670768+0000 osd.10 (osd.10) 158 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:35:13.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:12 smithi169 bash[19604]: cluster 2024-02-12T16:35:11.714811+0000 osd.10 (osd.10) 159 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:35:13.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:12 smithi169 bash[19604]: cluster 2024-02-12T16:35:11.926776+0000 osd.0 (osd.0) 128 : cluster [DBG] 7.f scrub starts 2024-02-12T16:35:13.113 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:12 smithi169 bash[19604]: cluster 2024-02-12T16:35:12.120390+0000 mgr.y (mgr.14146) 1517 : cluster [DBG] pgmap v2174: 47 pgs: 47 active+clean; 7.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-12T16:35:13.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:13.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 118812 118796 148.483 93.4375 0.00483782 0.00673362 2024-02-12T16:35:13.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:12 smithi121 bash[18489]: cluster 2024-02-12T16:35:11.509500+0000 osd.7 (osd.7) 190 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:35:13.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:12 smithi121 bash[18489]: cluster 2024-02-12T16:35:11.592465+0000 osd.7 (osd.7) 191 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:35:13.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:12 smithi121 bash[18489]: cluster 2024-02-12T16:35:11.670768+0000 osd.10 (osd.10) 158 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:35:13.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:12 smithi121 bash[18489]: cluster 2024-02-12T16:35:11.714811+0000 osd.10 (osd.10) 159 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:35:13.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:12 smithi121 bash[18489]: cluster 2024-02-12T16:35:11.926776+0000 osd.0 (osd.0) 128 : cluster [DBG] 7.f scrub starts 2024-02-12T16:35:13.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:12 smithi121 bash[18489]: cluster 2024-02-12T16:35:12.120390+0000 mgr.y (mgr.14146) 1517 : cluster [DBG] pgmap v2174: 47 pgs: 47 active+clean; 7.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-02-12T16:35:13.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:13.288 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:13.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:13.417 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:13.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:13.545 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:13.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:13.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:13.799 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:13.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:13.927 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:13.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:14.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:14.060 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:14.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:14.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:14.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 120294 120278 147.387 92.625 0.00872803 0.0067835 2024-02-12T16:35:14.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:14.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:14.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:14.336 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:14.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:14.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:14.465 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:14.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:14.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:14.609 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:14.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:14.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:14.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:14.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:14.863 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.11 to 1 (10000) 2024-02-12T16:35:14.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:14.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:14.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:14.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:15.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:15.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:15.132 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:14 smithi123 bash[19119]: cluster 2024-02-12T16:35:13.337239+0000 osd.7 (osd.7) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:15.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:14 smithi123 bash[19119]: cluster 2024-02-12T16:35:13.337281+0000 osd.7 (osd.7) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:15.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:14 smithi123 bash[19119]: cluster 2024-02-12T16:35:13.843092+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e778: 12 total, 12 up, 12 in 2024-02-12T16:35:15.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:14 smithi123 bash[19119]: cluster 2024-02-12T16:35:14.120838+0000 mgr.y (mgr.14146) 1518 : cluster [DBG] pgmap v2176: 47 pgs: 47 active+clean; 7.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-12T16:35:15.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:14 smithi123 bash[19119]: cluster 2024-02-12T16:35:14.175572+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.f scrub ok 2024-02-12T16:35:15.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:14 smithi123 bash[19119]: audit 2024-02-12T16:35:14.755721+0000 mon.a (mon.0) 2632 : audit [INF] from='client.26618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:35:15.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:14 smithi123 bash[19119]: audit 2024-02-12T16:35:14.757152+0000 mon.b (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.121:0/576432225' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:35:15.151 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:15.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:14 smithi121 bash[18489]: cluster 2024-02-12T16:35:13.337239+0000 osd.7 (osd.7) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:15.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:14 smithi121 bash[18489]: cluster 2024-02-12T16:35:13.337281+0000 osd.7 (osd.7) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:15.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:14 smithi121 bash[18489]: cluster 2024-02-12T16:35:13.843092+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e778: 12 total, 12 up, 12 in 2024-02-12T16:35:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:14 smithi121 bash[18489]: cluster 2024-02-12T16:35:14.120838+0000 mgr.y (mgr.14146) 1518 : cluster [DBG] pgmap v2176: 47 pgs: 47 active+clean; 7.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-12T16:35:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:14 smithi121 bash[18489]: cluster 2024-02-12T16:35:14.175572+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.f scrub ok 2024-02-12T16:35:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:14 smithi121 bash[18489]: audit 2024-02-12T16:35:14.755721+0000 mon.a (mon.0) 2632 : audit [INF] from='client.26618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:35:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:14 smithi121 bash[18489]: audit 2024-02-12T16:35:14.757152+0000 mon.b (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.121:0/576432225' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:35:15.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 16 121771 121755 146.328 92.3125 0.00642047 0.00683223 2024-02-12T16:35:15.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:15.284 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:15.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:14 smithi169 bash[19604]: cluster 2024-02-12T16:35:13.337239+0000 osd.7 (osd.7) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:15.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:14 smithi169 bash[19604]: cluster 2024-02-12T16:35:13.337281+0000 osd.7 (osd.7) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:15.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:14 smithi169 bash[19604]: cluster 2024-02-12T16:35:13.843092+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e778: 12 total, 12 up, 12 in 2024-02-12T16:35:15.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:14 smithi169 bash[19604]: cluster 2024-02-12T16:35:14.120838+0000 mgr.y (mgr.14146) 1518 : cluster [DBG] pgmap v2176: 47 pgs: 47 active+clean; 7.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-02-12T16:35:15.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:14 smithi169 bash[19604]: cluster 2024-02-12T16:35:14.175572+0000 osd.0 (osd.0) 129 : cluster [DBG] 7.f scrub ok 2024-02-12T16:35:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:14 smithi169 bash[19604]: audit 2024-02-12T16:35:14.755721+0000 mon.a (mon.0) 2632 : audit [INF] from='client.26618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:35:15.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:14 smithi169 bash[19604]: audit 2024-02-12T16:35:14.757152+0000 mon.b (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.121:0/576432225' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:35:15.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:15.420 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:15.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:15.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:15.662 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:15.689 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:15.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:15.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:15.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:15.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:14.848341+0000 mon.a (mon.0) 2633 : audit [INF] from='client.26618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:35:15.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: cluster 2024-02-12T16:35:14.848405+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e779: 12 total, 12 up, 12 in 2024-02-12T16:35:15.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.241031+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]: dispatch 2024-02-12T16:35:15.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.241751+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]: dispatch 2024-02-12T16:35:15.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.242307+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]: dispatch 2024-02-12T16:35:15.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.242405+0000 mon.b (mon.2) 172 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]: dispatch 2024-02-12T16:35:15.925 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.242714+0000 mon.b (mon.2) 173 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.242805+0000 mon.a (mon.0) 2638 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.242981+0000 mon.b (mon.2) 174 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.243244+0000 mon.b (mon.2) 175 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.243336+0000 mon.a (mon.0) 2639 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.243535+0000 mon.b (mon.2) 176 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.243798+0000 mon.a (mon.0) 2640 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.243812+0000 mon.b (mon.2) 177 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244082+0000 mon.a (mon.0) 2641 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244090+0000 mon.b (mon.2) 178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: cluster 2024-02-12T16:35:15.244211+0000 osd.11 (osd.11) 174 : cluster [DBG] 7.5 scrub starts 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244378+0000 mon.a (mon.0) 2642 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244389+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244569+0000 mon.b (mon.2) 180 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]: dispatch 2024-02-12T16:35:15.926 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244694+0000 mon.a (mon.0) 2643 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]: dispatch 2024-02-12T16:35:15.927 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244752+0000 mon.b (mon.2) 181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]: dispatch 2024-02-12T16:35:15.927 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244919+0000 mon.b (mon.2) 182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]: dispatch 2024-02-12T16:35:15.927 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.244996+0000 mon.a (mon.0) 2644 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]: dispatch 2024-02-12T16:35:15.927 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.245104+0000 mon.b (mon.2) 183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]: dispatch 2024-02-12T16:35:15.927 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.245280+0000 mon.a (mon.0) 2645 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]: dispatch 2024-02-12T16:35:15.927 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:15 smithi123 bash[19119]: audit 2024-02-12T16:35:15.245570+0000 mon.a (mon.0) 2646 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]: dispatch 2024-02-12T16:35:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:15.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:16.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:16.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:16.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:16.109 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:35:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:14.848341+0000 mon.a (mon.0) 2633 : audit [INF] from='client.26618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:35:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: cluster 2024-02-12T16:35:14.848405+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e779: 12 total, 12 up, 12 in 2024-02-12T16:35:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.241031+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]: dispatch 2024-02-12T16:35:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.241751+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]: dispatch 2024-02-12T16:35:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.242307+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]: dispatch 2024-02-12T16:35:16.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.242405+0000 mon.b (mon.2) 172 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.242714+0000 mon.b (mon.2) 173 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.242805+0000 mon.a (mon.0) 2638 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.242981+0000 mon.b (mon.2) 174 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.243244+0000 mon.b (mon.2) 175 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.243336+0000 mon.a (mon.0) 2639 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.243535+0000 mon.b (mon.2) 176 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.243798+0000 mon.a (mon.0) 2640 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.243812+0000 mon.b (mon.2) 177 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244082+0000 mon.a (mon.0) 2641 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244090+0000 mon.b (mon.2) 178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: cluster 2024-02-12T16:35:15.244211+0000 osd.11 (osd.11) 174 : cluster [DBG] 7.5 scrub starts 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244378+0000 mon.a (mon.0) 2642 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244389+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]: dispatch 2024-02-12T16:35:16.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244569+0000 mon.b (mon.2) 180 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]: dispatch 2024-02-12T16:35:16.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244694+0000 mon.a (mon.0) 2643 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]: dispatch 2024-02-12T16:35:16.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244752+0000 mon.b (mon.2) 181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]: dispatch 2024-02-12T16:35:16.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244919+0000 mon.b (mon.2) 182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]: dispatch 2024-02-12T16:35:16.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.244996+0000 mon.a (mon.0) 2644 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]: dispatch 2024-02-12T16:35:16.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.245104+0000 mon.b (mon.2) 183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]: dispatch 2024-02-12T16:35:16.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.245280+0000 mon.a (mon.0) 2645 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]: dispatch 2024-02-12T16:35:16.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:15 smithi121 bash[18489]: audit 2024-02-12T16:35:15.245570+0000 mon.a (mon.0) 2646 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]: dispatch 2024-02-12T16:35:16.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:16.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 123249 123233 145.31 92.375 0.00672784 0.00688048 2024-02-12T16:35:16.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:14.848341+0000 mon.a (mon.0) 2633 : audit [INF] from='client.26618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:35:16.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: cluster 2024-02-12T16:35:14.848405+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e779: 12 total, 12 up, 12 in 2024-02-12T16:35:16.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.241031+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]: dispatch 2024-02-12T16:35:16.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.241751+0000 mon.a (mon.0) 2636 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]: dispatch 2024-02-12T16:35:16.194 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.242307+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.242405+0000 mon.b (mon.2) 172 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.242714+0000 mon.b (mon.2) 173 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.242805+0000 mon.a (mon.0) 2638 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.242981+0000 mon.b (mon.2) 174 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.243244+0000 mon.b (mon.2) 175 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.243336+0000 mon.a (mon.0) 2639 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.243535+0000 mon.b (mon.2) 176 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.243798+0000 mon.a (mon.0) 2640 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.243812+0000 mon.b (mon.2) 177 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244082+0000 mon.a (mon.0) 2641 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244090+0000 mon.b (mon.2) 178 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: cluster 2024-02-12T16:35:15.244211+0000 osd.11 (osd.11) 174 : cluster [DBG] 7.5 scrub starts 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244378+0000 mon.a (mon.0) 2642 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244389+0000 mon.b (mon.2) 179 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]: dispatch 2024-02-12T16:35:16.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244569+0000 mon.b (mon.2) 180 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]: dispatch 2024-02-12T16:35:16.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244694+0000 mon.a (mon.0) 2643 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]: dispatch 2024-02-12T16:35:16.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244752+0000 mon.b (mon.2) 181 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]: dispatch 2024-02-12T16:35:16.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244919+0000 mon.b (mon.2) 182 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]: dispatch 2024-02-12T16:35:16.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.244996+0000 mon.a (mon.0) 2644 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]: dispatch 2024-02-12T16:35:16.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.245104+0000 mon.b (mon.2) 183 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]: dispatch 2024-02-12T16:35:16.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.245280+0000 mon.a (mon.0) 2645 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]: dispatch 2024-02-12T16:35:16.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:15 smithi169 bash[19604]: audit 2024-02-12T16:35:15.245570+0000 mon.a (mon.0) 2646 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]: dispatch 2024-02-12T16:35:16.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:16.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:16.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:16.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:16.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:16.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:16.610 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:16.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:16.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:16.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:16.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:16.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 8 1 2024-02-12T16:35:16.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:16.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:16.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:16.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:17.014 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:17.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.124 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: cluster 2024-02-12T16:35:15.284248+0000 osd.8 (osd.8) 89 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:35:17.124 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: cluster 2024-02-12T16:35:15.288956+0000 osd.8 (osd.8) 90 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: cluster 2024-02-12T16:35:15.819775+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub starts 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.859717+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.859812+0000 mon.a (mon.0) 2648 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.859890+0000 mon.a (mon.0) 2649 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.859962+0000 mon.a (mon.0) 2650 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860044+0000 mon.a (mon.0) 2651 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860113+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860211+0000 mon.a (mon.0) 2653 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860309+0000 mon.a (mon.0) 2654 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860373+0000 mon.a (mon.0) 2655 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860463+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860545+0000 mon.a (mon.0) 2657 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]': finished 2024-02-12T16:35:17.125 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: audit 2024-02-12T16:35:15.860624+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]': finished 2024-02-12T16:35:17.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: cluster 2024-02-12T16:35:15.860695+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e780: 12 total, 12 up, 12 in 2024-02-12T16:35:17.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: cluster 2024-02-12T16:35:16.121270+0000 mgr.y (mgr.14146) 1519 : cluster [DBG] pgmap v2179: 47 pgs: 47 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:35:17.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: cluster 2024-02-12T16:35:16.375605+0000 osd.4 (osd.4) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:17.126 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:16 smithi169 bash[19604]: cluster 2024-02-12T16:35:16.375643+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: cluster 2024-02-12T16:35:15.284248+0000 osd.8 (osd.8) 89 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: cluster 2024-02-12T16:35:15.288956+0000 osd.8 (osd.8) 90 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: cluster 2024-02-12T16:35:15.819775+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub starts 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.859717+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.859812+0000 mon.a (mon.0) 2648 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.859890+0000 mon.a (mon.0) 2649 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.859962+0000 mon.a (mon.0) 2650 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860044+0000 mon.a (mon.0) 2651 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860113+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860211+0000 mon.a (mon.0) 2653 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860309+0000 mon.a (mon.0) 2654 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860373+0000 mon.a (mon.0) 2655 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]': finished 2024-02-12T16:35:17.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860463+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]': finished 2024-02-12T16:35:17.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860545+0000 mon.a (mon.0) 2657 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]': finished 2024-02-12T16:35:17.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: audit 2024-02-12T16:35:15.860624+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]': finished 2024-02-12T16:35:17.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: cluster 2024-02-12T16:35:15.860695+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e780: 12 total, 12 up, 12 in 2024-02-12T16:35:17.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: cluster 2024-02-12T16:35:16.121270+0000 mgr.y (mgr.14146) 1519 : cluster [DBG] pgmap v2179: 47 pgs: 47 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:35:17.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: cluster 2024-02-12T16:35:16.375605+0000 osd.4 (osd.4) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:17.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:16 smithi123 bash[19119]: cluster 2024-02-12T16:35:16.375643+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:17.141 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: cluster 2024-02-12T16:35:15.284248+0000 osd.8 (osd.8) 89 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: cluster 2024-02-12T16:35:15.288956+0000 osd.8 (osd.8) 90 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: cluster 2024-02-12T16:35:15.819775+0000 osd.0 (osd.0) 130 : cluster [DBG] 7.6 scrub starts 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.859717+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08256131162534472]}]': finished 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.859812+0000 mon.a (mon.0) 2648 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.0913434535394806]}]': finished 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.859890+0000 mon.a (mon.0) 2649 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08866639033064766]}]': finished 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.859962+0000 mon.a (mon.0) 2650 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09419047043615004]}]': finished 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860044+0000 mon.a (mon.0) 2651 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09111104874492262]}]': finished 2024-02-12T16:35:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860113+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08404548969938068]}]': finished 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860211+0000 mon.a (mon.0) 2653 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08554003841840432]}]': finished 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860309+0000 mon.a (mon.0) 2654 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08409594209645589]}]': finished 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860373+0000 mon.a (mon.0) 2655 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09102901940283159]}]': finished 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860463+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08602805432665693]}]': finished 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860545+0000 mon.a (mon.0) 2657 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08430926755203667]}]': finished 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: audit 2024-02-12T16:35:15.860624+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08462590054643841]}]': finished 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: cluster 2024-02-12T16:35:15.860695+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e780: 12 total, 12 up, 12 in 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: cluster 2024-02-12T16:35:16.121270+0000 mgr.y (mgr.14146) 1519 : cluster [DBG] pgmap v2179: 47 pgs: 47 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: cluster 2024-02-12T16:35:16.375605+0000 osd.4 (osd.4) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:17.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:16 smithi121 bash[18489]: cluster 2024-02-12T16:35:16.375643+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:17.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 124529 124513 144.1 80 0.00671941 0.00693771 2024-02-12T16:35:17.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:17.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.407 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:17.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:17.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:17.689 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:17.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:17.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:17.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:17.933 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:17 smithi121 bash[18489]: cluster 2024-02-12T16:35:16.192894+0000 osd.3 (osd.3) 66 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:35:17.933 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:17 smithi121 bash[18489]: cluster 2024-02-12T16:35:16.743379+0000 osd.9 (osd.9) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:17.933 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:17 smithi121 bash[18489]: cluster 2024-02-12T16:35:16.743413+0000 osd.9 (osd.9) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:17.933 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:17 smithi121 bash[18489]: cluster 2024-02-12T16:35:16.884123+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e781: 12 total, 12 up, 12 in 2024-02-12T16:35:17.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:17.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:18.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:18.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:17 smithi169 bash[19604]: cluster 2024-02-12T16:35:16.192894+0000 osd.3 (osd.3) 66 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:35:18.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:17 smithi169 bash[19604]: cluster 2024-02-12T16:35:16.743379+0000 osd.9 (osd.9) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:18.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:17 smithi169 bash[19604]: cluster 2024-02-12T16:35:16.743413+0000 osd.9 (osd.9) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:18.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:17 smithi169 bash[19604]: cluster 2024-02-12T16:35:16.884123+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e781: 12 total, 12 up, 12 in 2024-02-12T16:35:18.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:18.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:17 smithi123 bash[19119]: cluster 2024-02-12T16:35:16.192894+0000 osd.3 (osd.3) 66 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:35:18.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:17 smithi123 bash[19119]: cluster 2024-02-12T16:35:16.743379+0000 osd.9 (osd.9) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:18.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:17 smithi123 bash[19119]: cluster 2024-02-12T16:35:16.743413+0000 osd.9 (osd.9) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:18.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:17 smithi123 bash[19119]: cluster 2024-02-12T16:35:16.884123+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e781: 12 total, 12 up, 12 in 2024-02-12T16:35:18.187 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 125838 125822 142.968 81.8125 0.00572296 0.00699249 2024-02-12T16:35:18.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:18.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:18.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:18.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:18.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:18.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:18.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:18.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:18.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:18.640 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:18.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:18.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:18.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:18.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:18.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:18.935 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:18.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:19.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:19.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:18 smithi121 bash[18489]: cluster 2024-02-12T16:35:17.683184+0000 osd.11 (osd.11) 175 : cluster [DBG] 7.5 scrub ok 2024-02-12T16:35:19.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:18 smithi121 bash[18489]: cluster 2024-02-12T16:35:17.793351+0000 osd.1 (osd.1) 120 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:35:19.046 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:18 smithi121 bash[18489]: cluster 2024-02-12T16:35:18.045028+0000 osd.1 (osd.1) 121 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:35:19.046 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:18 smithi121 bash[18489]: cluster 2024-02-12T16:35:18.121730+0000 mgr.y (mgr.14146) 1520 : cluster [DBG] pgmap v2181: 47 pgs: 1 active+clean+scrubbing, 1 peering, 45 active+clean; 7.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:35:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:19.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:19.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:19.177 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:18 smithi123 bash[19119]: cluster 2024-02-12T16:35:17.683184+0000 osd.11 (osd.11) 175 : cluster [DBG] 7.5 scrub ok 2024-02-12T16:35:19.177 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:18 smithi123 bash[19119]: cluster 2024-02-12T16:35:17.793351+0000 osd.1 (osd.1) 120 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:35:19.177 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:18 smithi123 bash[19119]: cluster 2024-02-12T16:35:18.045028+0000 osd.1 (osd.1) 121 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:35:19.177 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:18 smithi123 bash[19119]: cluster 2024-02-12T16:35:18.121730+0000 mgr.y (mgr.14146) 1520 : cluster [DBG] pgmap v2181: 47 pgs: 1 active+clean+scrubbing, 1 peering, 45 active+clean; 7.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:35:19.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 127077 127061 141.797 77.4375 0.00384341 0.00703488 2024-02-12T16:35:19.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:19.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:18 smithi169 bash[19604]: cluster 2024-02-12T16:35:17.683184+0000 osd.11 (osd.11) 175 : cluster [DBG] 7.5 scrub ok 2024-02-12T16:35:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:18 smithi169 bash[19604]: cluster 2024-02-12T16:35:17.793351+0000 osd.1 (osd.1) 120 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:35:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:18 smithi169 bash[19604]: cluster 2024-02-12T16:35:18.045028+0000 osd.1 (osd.1) 121 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:35:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:18 smithi169 bash[19604]: cluster 2024-02-12T16:35:18.121730+0000 mgr.y (mgr.14146) 1520 : cluster [DBG] pgmap v2181: 47 pgs: 1 active+clean+scrubbing, 1 peering, 45 active+clean; 7.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-02-12T16:35:19.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:19.334 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:19.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:19.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:19.469 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:19.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:19.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:19.600 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:19.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:19.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:19.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:19.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:19.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:19.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:19.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:19.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:19.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:19.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:20.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:20.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:20.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:20.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:20.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:20.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:19 smithi121 bash[18489]: cluster 2024-02-12T16:35:18.367310+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.6 scrub ok 2024-02-12T16:35:20.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:19 smithi121 bash[18489]: cluster 2024-02-12T16:35:18.718043+0000 osd.10 (osd.10) 160 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:35:20.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:19 smithi121 bash[18489]: cluster 2024-02-12T16:35:18.834350+0000 osd.10 (osd.10) 161 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:35:20.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:19 smithi121 bash[18489]: cluster 2024-02-12T16:35:19.447452+0000 osd.4 (osd.4) 132 : cluster [DBG] 1.e scrub starts 2024-02-12T16:35:20.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 128126 128110 140.46 65.5625 0.0140948 0.00711764 2024-02-12T16:35:20.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:20.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:19 smithi169 bash[19604]: cluster 2024-02-12T16:35:18.367310+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.6 scrub ok 2024-02-12T16:35:20.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:19 smithi169 bash[19604]: cluster 2024-02-12T16:35:18.718043+0000 osd.10 (osd.10) 160 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:35:20.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:19 smithi169 bash[19604]: cluster 2024-02-12T16:35:18.834350+0000 osd.10 (osd.10) 161 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:35:20.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:19 smithi169 bash[19604]: cluster 2024-02-12T16:35:19.447452+0000 osd.4 (osd.4) 132 : cluster [DBG] 1.e scrub starts 2024-02-12T16:35:20.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:20.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:20.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:20.371 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:19 smithi123 bash[19119]: cluster 2024-02-12T16:35:18.367310+0000 osd.0 (osd.0) 131 : cluster [DBG] 7.6 scrub ok 2024-02-12T16:35:20.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:19 smithi123 bash[19119]: cluster 2024-02-12T16:35:18.718043+0000 osd.10 (osd.10) 160 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:35:20.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:19 smithi123 bash[19119]: cluster 2024-02-12T16:35:18.834350+0000 osd.10 (osd.10) 161 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:35:20.372 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:19 smithi123 bash[19119]: cluster 2024-02-12T16:35:19.447452+0000 osd.4 (osd.4) 132 : cluster [DBG] 1.e scrub starts 2024-02-12T16:35:20.386 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:20.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:20.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:20.515 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:20.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:20.579 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.16095.0:229767 5.b 5:d8df3106:::benchmark_data_smithi169_84712_object60377:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:26:53.352265+0000", 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 507.22079255900002, 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33745658699999997, 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.16095", 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2579637782", 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 229767 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352265+0000", 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352265+0000", 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352263+0000", 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.117e-06 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352269+0000", 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2445e-05 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352281+0000", 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1699999999999996e-06 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352286+0000", 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3307000000000003e-05 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352340+0000", 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000114072 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352454+0000", 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9314999999999994e-05 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0", 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.352553+0000", 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004186576 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.356740+0000", 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33283253099999999 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.689572+0000", 2024-02-12T16:35:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4769000000000003e-05 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.689637+0000", 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4192999999999998e-05 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:26:53.689721+0000", 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33745658699999997 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187668 6.12 6.ef21b792 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:40.974102+0000", 2024-02-12T16:35:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.59895553300001, 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87369477799999995, 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187668 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974102+0000", 2024-02-12T16:35:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974102+0000", 2024-02-12T16:35:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974099+0000", 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0800000000000006e-06 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974107+0000", 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6269999999999995e-06 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974117+0000", 2024-02-12T16:35:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1190000000000002e-06 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974123+0000", 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0201999999999997e-05 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974183+0000", 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3610000000000003e-06 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:40.974191+0000", 2024-02-12T16:35:20.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87359188700000001 2024-02-12T16:35:20.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847783+0000", 2024-02-12T16:35:20.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3906e-05 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847796+0000", 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87369477799999995 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.592 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187720 6.12 6.c442a632 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.009563+0000", 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.56349382100001, 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.83829735800000005, 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187720 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009563+0000", 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009563+0000", 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009560+0000", 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0183000000000001e-05 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009570+0000", 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7260000000000001e-05 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009588+0000", 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1539999999999996e-06 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009595+0000", 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5950000000000004e-05 2024-02-12T16:35:20.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009661+0000", 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7557000000000001e-05 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.009688+0000", 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.83815804500000002 2024-02-12T16:35:20.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847846+0000", 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4403000000000001e-05 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847861+0000", 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.83829735800000005 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.597 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187746 6.12 6.7d35e672 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.028222+0000", 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.544834753, 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.8196696, 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187746 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.598 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028222+0000", 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028222+0000", 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028219+0000", 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2330999999999999e-05 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028232+0000", 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3801000000000001e-05 2024-02-12T16:35:20.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028255+0000", 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0550000000000005e-06 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028264+0000", 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7299999999999995e-05 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028342+0000", 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0910000000000001e-05 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.028353+0000", 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81952710200000001 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847880+0000", 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2329e-05 2024-02-12T16:35:20.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847892+0000", 2024-02-12T16:35:20.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.8196696 2024-02-12T16:35:20.603 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.603 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.603 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187763 6.12 6.cc7dced2 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.042778+0000", 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.530279304, 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80513970400000001, 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.604 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187763 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042778+0000", 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042778+0000", 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042775+0000", 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1157e-05 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042786+0000", 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0633e-05 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042807+0000", 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9979999999999992e-06 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042816+0000", 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6520999999999995e-05 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042882+0000", 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.19e-05 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.042894+0000", 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80501752299999996 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847911+0000", 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0939999999999996e-06 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847918+0000", 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80513970400000001 2024-02-12T16:35:20.609 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187859 6.12 6.c07b8ab2 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.114878+0000", 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.45817960599999, 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.73306662899999997, 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.610 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187859 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114878+0000", 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114878+0000", 2024-02-12T16:35:20.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:20.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114874+0000", 2024-02-12T16:35:20.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0406e-05 2024-02-12T16:35:20.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114885+0000", 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7850999999999999e-05 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114903+0000", 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5150000000000003e-06 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114909+0000", 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1171000000000002e-05 2024-02-12T16:35:20.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114960+0000", 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7050000000000005e-06 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.114969+0000", 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.73296538300000003 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847934+0000", 2024-02-12T16:35:20.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6979999999999993e-06 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847944+0000", 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.73306662899999997 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187869 6.12 6.a7e821d2 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.126473+0000", 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.446583944, 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72149101000000004, 2024-02-12T16:35:20.618 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187869 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126473+0000", 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126473+0000", 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999962 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126469+0000", 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3322000000000001e-05 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126483+0000", 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0296e-05 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126503+0000", 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9700000000000005e-06 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126512+0000", 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3619000000000002e-05 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126576+0000", 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9400000000000008e-06 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.126585+0000", 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72137515699999999 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847960+0000", 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4669999999999998e-06 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847964+0000", 2024-02-12T16:35:20.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72149101000000004 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187885 6.12 6.21e84f52 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.142572+0000", 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.43048468800001, 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.70541649699999998, 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.623 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187885 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142572+0000", 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142572+0000", 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142569+0000", 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1482e-05 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142581+0000", 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9411000000000001e-05 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142600+0000", 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4020000000000002e-06 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142609+0000", 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0028000000000001e-05 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142679+0000", 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2191999999999999e-05 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.142691+0000", 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.70529067300000003 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847982+0000", 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3989999999999998e-06 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847989+0000", 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.70541649699999998 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:187979 6.12 6.b1032852 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.627 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.225589+0000", 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.347468018, 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.622412678, 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 187979 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225589+0000", 2024-02-12T16:35:20.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225589+0000", 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225587+0000", 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6549999999999993e-06 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225595+0000", 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4166e-05 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225609+0000", 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4949999999999999e-06 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225615+0000", 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1854000000000002e-05 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225667+0000", 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7230000000000004e-06 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.225674+0000", 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.62232305700000001 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.847997+0000", 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3669999999999999e-06 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848002+0000", 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.622412678 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:188014 6.12 6.ba394ed2 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.258696+0000", 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.31436089900001, 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58932130999999999, 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.632 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 188014 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258696+0000", 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258696+0000", 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258693+0000", 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5610000000000004e-06 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258703+0000", 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4073999999999999e-05 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258717+0000", 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999992e-06 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258724+0000", 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5048000000000005e-05 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258789+0000", 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1992e-05 2024-02-12T16:35:20.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.258801+0000", 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58921177700000005 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848013+0000", 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3270000000000002e-06 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848018+0000", 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58932130999999999 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:188024 6.12 6.6ea34a92 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.266359+0000", 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.306697778, 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58167206999999999, 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.637 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 188024 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266359+0000", 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266359+0000", 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.638 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266357+0000", 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8580000000000006e-06 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266366+0000", 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1793e-05 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266378+0000", 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2689999999999999e-06 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266383+0000", 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1916999999999997e-05 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266445+0000", 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.266454+0000", 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58157367400000004 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848028+0000", 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.827e-06 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848031+0000", 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58167206999999999 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.641 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:188030 6.12 6.fea90172 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.274251+0000", 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.29880579300001, 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.57379316000000002, 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 188030 2024-02-12T16:35:20.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274251+0000", 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274251+0000", 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274249+0000", 2024-02-12T16:35:20.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1246999999999999e-05 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274260+0000", 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3124e-05 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274273+0000", 2024-02-12T16:35:20.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8860000000000001e-06 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274279+0000", 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9687e-05 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274329+0000", 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6699999999999999e-06 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.274335+0000", 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.57370665600000004 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848041+0000", 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3110000000000001e-06 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848045+0000", 2024-02-12T16:35:20.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.57379316000000002 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:188093 6.12 6.fd1ac052 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.343917+0000", 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.22914044800001, 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50414094399999998, 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.647 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 188093 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343917+0000", 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.648 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343917+0000", 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343914+0000", 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8650000000000007e-06 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343923+0000", 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4491000000000001e-05 2024-02-12T16:35:20.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343938+0000", 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.49e-06 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343943+0000", 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7675e-05 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343991+0000", 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4130000000000003e-06 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.343997+0000", 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50405723099999999 2024-02-12T16:35:20.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848054+0000", 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3280000000000002e-06 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848058+0000", 2024-02-12T16:35:20.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.50414094399999998 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:188119 6.12 6.2a4825d2 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.378559+0000", 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.19449797499999, 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46951162400000002, 2024-02-12T16:35:20.653 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 188119 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378559+0000", 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378559+0000", 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378556+0000", 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3570000000000004e-06 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378566+0000", 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5106000000000001e-05 2024-02-12T16:35:20.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378581+0000", 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5139999999999998e-06 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378587+0000", 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0247000000000001e-05 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378648+0000", 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5562000000000001e-05 2024-02-12T16:35:20.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.378663+0000", 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46940420500000002 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848067+0000", 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.444e-06 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848071+0000", 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.46951162400000002 2024-02-12T16:35:20.657 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:188150 6.12 6.ef8f8372 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.432071+0000", 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.140985933, 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41601409099999997, 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 188150 2024-02-12T16:35:20.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432071+0000", 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432071+0000", 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432068+0000", 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1083e-05 2024-02-12T16:35:20.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432079+0000", 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1291999999999999e-05 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432101+0000", 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0339999999999997e-06 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432109+0000", 2024-02-12T16:35:20.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4583000000000001e-05 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432183+0000", 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2478e-05 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.432196+0000", 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41588474800000003 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848080+0000", 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8930000000000004e-06 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848085+0000", 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.41601409099999997 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:188165 6.12 6.a07bcbb2 (undecoded) ondisk+write+known_if_redirected e690)", 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:31:41.466160+0000", 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 219.10689764099999, 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38194863000000001, 2024-02-12T16:35:20.662 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 188165 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466160+0000", 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466160+0000", 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999948 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466154+0000", 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4386000000000001e-05 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466169+0000", 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3754e-05 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466193+0000", 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3749999999999997e-06 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466200+0000", 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5611000000000006e-05 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466266+0000", 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.498e-06 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.466273+0000", 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.381828895 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848102+0000", 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2199999999999997e-06 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:31:41.848108+0000", 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.38194863000000001 2024-02-12T16:35:20.666 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:27801 7.a 7:54e7ce15:::benchmark_data_smithi169_105252_object27800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.073058+0000", 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.499998679000001, 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.097711489999999998, 2024-02-12T16:35:20.667 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 27801 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073058+0000", 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073058+0000", 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073057+0000", 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4576999999999999e-05 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073081+0000", 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4066999999999999e-05 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073095+0000", 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3090000000000001e-06 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073101+0000", 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00040983300000000002 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073511+0000", 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107048 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073618+0000", 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000174286 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.073793+0000", 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079826400000000003 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:35:20.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074591+0000", 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.096015586999999999 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.170607+0000", 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6148000000000002e-05 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.170703+0000", 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7315000000000002e-05 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.170770+0000", 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.097711489999999998 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:27811 7.a 7:5b697532:::benchmark_data_smithi169_105252_object27810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.074743+0000", 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 51.498314104000002, 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.101196569, 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:35:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 27811 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074743+0000", 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074743+0000", 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074741+0000", 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0767999999999999e-05 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074792+0000", 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0321e-05 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074802+0000", 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9550000000000003e-06 2024-02-12T16:35:20.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074809+0000", 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4878e-05 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.074854+0000", 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014610400000000001 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.075000+0000", 2024-02-12T16:35:20.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1569000000000004e-05 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.075092+0000", 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00074547399999999998 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.075837+0000", 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.099965251000000005 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.175802+0000", 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5413000000000003e-05 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.175898+0000", 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1774999999999999e-05 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.175940+0000", 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.101196569 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:42496 7.a 7:58f26340:::benchmark_data_smithi169_105252_object42495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e763)", 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:34.026666+0000", 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.546391642000003, 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.106897221, 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:20.679 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 42496 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026666+0000", 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026666+0000", 2024-02-12T16:35:20.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:20.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026662+0000", 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4539999999999998e-05 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026697+0000", 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5982e-05 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026713+0000", 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3209999999999999e-06 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026720+0000", 2024-02-12T16:35:20.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5208999999999999e-05 2024-02-12T16:35:20.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026775+0000", 2024-02-12T16:35:20.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011953800000000001 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.026895+0000", 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000132292 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.027027+0000", 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016227469999999999 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.028650+0000", 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.10477152300000001 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.133421+0000", 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3060999999999996e-05 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.133504+0000", 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8341000000000003e-05 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.133563+0000", 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.106897221 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:42515 7.a 7:5271f5af:::benchmark_data_smithi169_105252_object42514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e763)", 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:34.031275+0000", 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.541782630999997, 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.10562281499999999, 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 42515 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:20.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031275+0000", 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031275+0000", 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031272+0000", 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1522000000000002e-05 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031294+0000", 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0827e-05 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031305+0000", 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8980000000000004e-06 2024-02-12T16:35:20.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031310+0000", 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8121000000000003e-05 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031348+0000", 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5929999999999998e-05 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031444+0000", 2024-02-12T16:35:20.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106603 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.031550+0000", 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013006090000000001 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.032851+0000", 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.10380017900000001 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.136651+0000", 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000147183 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.136798+0000", 2024-02-12T16:35:20.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9100999999999999e-05 2024-02-12T16:35:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:34.136897+0000", 2024-02-12T16:35:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.10562281499999999 2024-02-12T16:35:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.694 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.695 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.695 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:20.695 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:20.695 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:35:20.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:20.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:20.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:20.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:20.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:20.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:20.952 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:35:20.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:20.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:20 smithi121 bash[18489]: cluster 2024-02-12T16:35:19.002945+0000 osd.3 (osd.3) 67 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:35:20.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:20 smithi121 bash[18489]: cluster 2024-02-12T16:35:19.431145+0000 osd.3 (osd.3) 68 : cluster [DBG] 1.c scrub starts 2024-02-12T16:35:20.964 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:20 smithi121 bash[18489]: cluster 2024-02-12T16:35:19.522637+0000 osd.3 (osd.3) 69 : cluster [DBG] 1.c scrub ok 2024-02-12T16:35:20.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:20 smithi121 bash[18489]: cluster 2024-02-12T16:35:19.574102+0000 osd.4 (osd.4) 133 : cluster [DBG] 1.e scrub ok 2024-02-12T16:35:20.965 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:20 smithi121 bash[18489]: cluster 2024-02-12T16:35:20.122133+0000 mgr.y (mgr.14146) 1521 : cluster [DBG] pgmap v2182: 47 pgs: 2 active+clean+scrubbing, 1 peering, 44 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:35:20.988 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:35:20.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:21.030 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:35:21.069 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:35:21.103 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:35:21.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:21.133 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:35:21.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:21.165 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:35:21.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 15 129509 129494 139.529 86.5 0.0186138 0.00716487 2024-02-12T16:35:21.193 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:35:21.228 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:35:21.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:35:21.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:21.272 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:20 smithi123 bash[19119]: cluster 2024-02-12T16:35:19.002945+0000 osd.3 (osd.3) 67 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:35:21.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:20 smithi123 bash[19119]: cluster 2024-02-12T16:35:19.431145+0000 osd.3 (osd.3) 68 : cluster [DBG] 1.c scrub starts 2024-02-12T16:35:21.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:20 smithi123 bash[19119]: cluster 2024-02-12T16:35:19.522637+0000 osd.3 (osd.3) 69 : cluster [DBG] 1.c scrub ok 2024-02-12T16:35:21.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:20 smithi123 bash[19119]: cluster 2024-02-12T16:35:19.574102+0000 osd.4 (osd.4) 133 : cluster [DBG] 1.e scrub ok 2024-02-12T16:35:21.273 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:20 smithi123 bash[19119]: cluster 2024-02-12T16:35:20.122133+0000 mgr.y (mgr.14146) 1521 : cluster [DBG] pgmap v2182: 47 pgs: 2 active+clean+scrubbing, 1 peering, 44 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:35:21.284 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:35:21.290 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:20 smithi169 bash[19604]: cluster 2024-02-12T16:35:19.002945+0000 osd.3 (osd.3) 67 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:35:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:20 smithi169 bash[19604]: cluster 2024-02-12T16:35:19.431145+0000 osd.3 (osd.3) 68 : cluster [DBG] 1.c scrub starts 2024-02-12T16:35:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:20 smithi169 bash[19604]: cluster 2024-02-12T16:35:19.522637+0000 osd.3 (osd.3) 69 : cluster [DBG] 1.c scrub ok 2024-02-12T16:35:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:20 smithi169 bash[19604]: cluster 2024-02-12T16:35:19.574102+0000 osd.4 (osd.4) 133 : cluster [DBG] 1.e scrub ok 2024-02-12T16:35:21.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:20 smithi169 bash[19604]: cluster 2024-02-12T16:35:20.122133+0000 mgr.y (mgr.14146) 1521 : cluster [DBG] pgmap v2182: 47 pgs: 2 active+clean+scrubbing, 1 peering, 44 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-02-12T16:35:21.323 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:35:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:35:21.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:21.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:21.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-12T16:35:21.524 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:21.555 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:21.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:21.693 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:21.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:21.820 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:35:21.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:21.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:21 smithi169 bash[19604]: cluster 2024-02-12T16:35:20.736000+0000 osd.6 (osd.6) 148 : cluster [DBG] 7.d scrub starts 2024-02-12T16:35:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:21 smithi169 bash[19604]: cluster 2024-02-12T16:35:20.747271+0000 osd.5 (osd.5) 159 : cluster [DBG] 7.0 scrub starts 2024-02-12T16:35:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:21 smithi169 bash[19604]: audit 2024-02-12T16:35:20.913273+0000 mon.a (mon.0) 2661 : audit [DBG] from='client.? v1:172.21.15.121:0/3991003898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:21 smithi169 bash[19604]: audit 2024-02-12T16:35:21.635932+0000 mon.a (mon.0) 2662 : audit [INF] from='client.? v1:172.21.15.121:0/3857089154' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:35:22.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:22.127 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:22.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:22.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 130774 130758 138.503 79 0.00716407 0.00721787 2024-02-12T16:35:22.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:22.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:22.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:22.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:22.365 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:21 smithi123 bash[19119]: cluster 2024-02-12T16:35:20.736000+0000 osd.6 (osd.6) 148 : cluster [DBG] 7.d scrub starts 2024-02-12T16:35:22.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:21 smithi123 bash[19119]: cluster 2024-02-12T16:35:20.747271+0000 osd.5 (osd.5) 159 : cluster [DBG] 7.0 scrub starts 2024-02-12T16:35:22.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:21 smithi123 bash[19119]: audit 2024-02-12T16:35:20.913273+0000 mon.a (mon.0) 2661 : audit [DBG] from='client.? v1:172.21.15.121:0/3991003898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:22.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:21 smithi123 bash[19119]: audit 2024-02-12T16:35:21.635932+0000 mon.a (mon.0) 2662 : audit [INF] from='client.? v1:172.21.15.121:0/3857089154' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:35:22.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:22.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:22.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:21 smithi121 bash[18489]: cluster 2024-02-12T16:35:20.736000+0000 osd.6 (osd.6) 148 : cluster [DBG] 7.d scrub starts 2024-02-12T16:35:22.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:21 smithi121 bash[18489]: cluster 2024-02-12T16:35:20.747271+0000 osd.5 (osd.5) 159 : cluster [DBG] 7.0 scrub starts 2024-02-12T16:35:22.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:21 smithi121 bash[18489]: audit 2024-02-12T16:35:20.913273+0000 mon.a (mon.0) 2661 : audit [DBG] from='client.? v1:172.21.15.121:0/3991003898' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:22.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:21 smithi121 bash[18489]: audit 2024-02-12T16:35:21.635932+0000 mon.a (mon.0) 2662 : audit [INF] from='client.? v1:172.21.15.121:0/3857089154' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:35:22.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:22.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:22.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:22.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:22.686 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:22.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:22.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:22.816 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:22.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:22.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:22.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:23.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:23.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:23.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:23.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:22 smithi121 bash[18489]: cluster 2024-02-12T16:35:21.417661+0000 osd.7 (osd.7) 194 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:35:23.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:22 smithi121 bash[18489]: cluster 2024-02-12T16:35:21.488271+0000 osd.7 (osd.7) 195 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:35:23.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:22 smithi121 bash[18489]: cluster 2024-02-12T16:35:21.897454+0000 mon.a (mon.0) 2663 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:35:23.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:22 smithi121 bash[18489]: cluster 2024-02-12T16:35:21.897492+0000 mon.a (mon.0) 2664 : cluster [INF] Cluster is now healthy 2024-02-12T16:35:23.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:22 smithi121 bash[18489]: audit 2024-02-12T16:35:21.901351+0000 mon.a (mon.0) 2665 : audit [INF] from='client.? v1:172.21.15.121:0/3857089154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:35:23.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:22 smithi121 bash[18489]: cluster 2024-02-12T16:35:21.901393+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e782: 12 total, 12 up, 12 in 2024-02-12T16:35:23.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:22 smithi121 bash[18489]: cluster 2024-02-12T16:35:22.122530+0000 mgr.y (mgr.14146) 1522 : cluster [DBG] pgmap v2184: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:35:23.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:23.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:35:23.184954 min lat: 0.00104071 max lat: 0.477087 avg lat: 0.00726344 2024-02-12T16:35:23.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:35:23.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 132167 132151 137.646 87.0625 0.0105629 0.00726344 2024-02-12T16:35:23.220 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:23.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:22 smithi169 bash[19604]: cluster 2024-02-12T16:35:21.417661+0000 osd.7 (osd.7) 194 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:35:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:22 smithi169 bash[19604]: cluster 2024-02-12T16:35:21.488271+0000 osd.7 (osd.7) 195 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:35:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:22 smithi169 bash[19604]: cluster 2024-02-12T16:35:21.897454+0000 mon.a (mon.0) 2663 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:35:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:22 smithi169 bash[19604]: cluster 2024-02-12T16:35:21.897492+0000 mon.a (mon.0) 2664 : cluster [INF] Cluster is now healthy 2024-02-12T16:35:23.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:22 smithi169 bash[19604]: audit 2024-02-12T16:35:21.901351+0000 mon.a (mon.0) 2665 : audit [INF] from='client.? v1:172.21.15.121:0/3857089154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:35:23.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:22 smithi169 bash[19604]: cluster 2024-02-12T16:35:21.901393+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e782: 12 total, 12 up, 12 in 2024-02-12T16:35:23.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:22 smithi169 bash[19604]: cluster 2024-02-12T16:35:22.122530+0000 mgr.y (mgr.14146) 1522 : cluster [DBG] pgmap v2184: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:35:23.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:23.355 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:23.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:22 smithi123 bash[19119]: cluster 2024-02-12T16:35:21.417661+0000 osd.7 (osd.7) 194 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:35:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:22 smithi123 bash[19119]: cluster 2024-02-12T16:35:21.488271+0000 osd.7 (osd.7) 195 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:35:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:22 smithi123 bash[19119]: cluster 2024-02-12T16:35:21.897454+0000 mon.a (mon.0) 2663 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:35:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:22 smithi123 bash[19119]: cluster 2024-02-12T16:35:21.897492+0000 mon.a (mon.0) 2664 : cluster [INF] Cluster is now healthy 2024-02-12T16:35:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:22 smithi123 bash[19119]: audit 2024-02-12T16:35:21.901351+0000 mon.a (mon.0) 2665 : audit [INF] from='client.? v1:172.21.15.121:0/3857089154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:35:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:22 smithi123 bash[19119]: cluster 2024-02-12T16:35:21.901393+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e782: 12 total, 12 up, 12 in 2024-02-12T16:35:23.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:22 smithi123 bash[19119]: cluster 2024-02-12T16:35:22.122530+0000 mgr.y (mgr.14146) 1522 : cluster [DBG] pgmap v2184: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-02-12T16:35:23.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:23.481 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:23.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:23.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:23.593 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:23.614 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:23.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:23.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:23.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:23.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:23.874 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:23.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:23.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:24.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:24.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:24.114 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:23 smithi123 bash[19119]: cluster 2024-02-12T16:35:22.520752+0000 osd.5 (osd.5) 160 : cluster [DBG] 7.0 scrub ok 2024-02-12T16:35:24.114 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:23 smithi123 bash[19119]: cluster 2024-02-12T16:35:22.869052+0000 osd.6 (osd.6) 149 : cluster [DBG] 7.d scrub ok 2024-02-12T16:35:24.132 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:24.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 133587 133571 136.844 88.75 0.00768556 0.00730612 2024-02-12T16:35:24.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:24.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:23 smithi121 bash[18489]: cluster 2024-02-12T16:35:22.520752+0000 osd.5 (osd.5) 160 : cluster [DBG] 7.0 scrub ok 2024-02-12T16:35:24.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:23 smithi121 bash[18489]: cluster 2024-02-12T16:35:22.869052+0000 osd.6 (osd.6) 149 : cluster [DBG] 7.d scrub ok 2024-02-12T16:35:24.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:23 smithi169 bash[19604]: cluster 2024-02-12T16:35:22.520752+0000 osd.5 (osd.5) 160 : cluster [DBG] 7.0 scrub ok 2024-02-12T16:35:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:23 smithi169 bash[19604]: cluster 2024-02-12T16:35:22.869052+0000 osd.6 (osd.6) 149 : cluster [DBG] 7.d scrub ok 2024-02-12T16:35:24.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:24.395 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:24.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:24.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:24.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:24.652 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:24.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:24.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:24.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:24.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:24.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:24.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:35:25.043 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:25.051 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:24 smithi123 bash[19119]: cluster 2024-02-12T16:35:23.701498+0000 osd.6 (osd.6) 150 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:35:25.051 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:24 smithi123 bash[19119]: cluster 2024-02-12T16:35:23.788551+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:35:25.051 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:24 smithi123 bash[19119]: cluster 2024-02-12T16:35:23.858960+0000 osd.5 (osd.5) 161 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:35:25.051 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:24 smithi123 bash[19119]: cluster 2024-02-12T16:35:23.904207+0000 osd.5 (osd.5) 162 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:35:25.051 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:24 smithi123 bash[19119]: cluster 2024-02-12T16:35:24.123004+0000 mgr.y (mgr.14146) 1523 : cluster [DBG] pgmap v2185: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-12T16:35:25.067 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:25.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:25.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:24 smithi121 bash[18489]: cluster 2024-02-12T16:35:23.701498+0000 osd.6 (osd.6) 150 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:35:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:24 smithi121 bash[18489]: cluster 2024-02-12T16:35:23.788551+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:35:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:24 smithi121 bash[18489]: cluster 2024-02-12T16:35:23.858960+0000 osd.5 (osd.5) 161 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:35:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:24 smithi121 bash[18489]: cluster 2024-02-12T16:35:23.904207+0000 osd.5 (osd.5) 162 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:35:25.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:24 smithi121 bash[18489]: cluster 2024-02-12T16:35:24.123004+0000 mgr.y (mgr.14146) 1523 : cluster [DBG] pgmap v2185: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-12T16:35:25.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:35:25.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:25.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 134965 134949 136.026 86.125 0.00450748 0.00734962 2024-02-12T16:35:25.224 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:25.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:24 smithi169 bash[19604]: cluster 2024-02-12T16:35:23.701498+0000 osd.6 (osd.6) 150 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:35:25.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:24 smithi169 bash[19604]: cluster 2024-02-12T16:35:23.788551+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:35:25.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:24 smithi169 bash[19604]: cluster 2024-02-12T16:35:23.858960+0000 osd.5 (osd.5) 161 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:35:25.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:24 smithi169 bash[19604]: cluster 2024-02-12T16:35:23.904207+0000 osd.5 (osd.5) 162 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:35:25.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:24 smithi169 bash[19604]: cluster 2024-02-12T16:35:24.123004+0000 mgr.y (mgr.14146) 1523 : cluster [DBG] pgmap v2185: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-02-12T16:35:25.331 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:25.354 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:25.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:25.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:25.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:25.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:25.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:25.632 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:25.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:25.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:25.763 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:25.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:25.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:25.895 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:25.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:25.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:25 smithi169 bash[19604]: cluster 2024-02-12T16:35:24.665495+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.c scrub starts 2024-02-12T16:35:25.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:25 smithi169 bash[19604]: cluster 2024-02-12T16:35:24.775904+0000 osd.9 (osd.9) 135 : cluster [DBG] 7.a scrub starts 2024-02-12T16:35:25.928 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:25 smithi169 bash[19604]: cluster 2024-02-12T16:35:25.812722+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e783: 12 total, 12 up, 12 in 2024-02-12T16:35:26.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:26.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:26.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:26.156 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:25 smithi123 bash[19119]: cluster 2024-02-12T16:35:24.665495+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.c scrub starts 2024-02-12T16:35:26.156 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:25 smithi123 bash[19119]: cluster 2024-02-12T16:35:24.775904+0000 osd.9 (osd.9) 135 : cluster [DBG] 7.a scrub starts 2024-02-12T16:35:26.157 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:25 smithi123 bash[19119]: cluster 2024-02-12T16:35:25.812722+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e783: 12 total, 12 up, 12 in 2024-02-12T16:35:26.186 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:26.187 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:25 smithi121 bash[18489]: cluster 2024-02-12T16:35:24.665495+0000 osd.6 (osd.6) 152 : cluster [DBG] 7.c scrub starts 2024-02-12T16:35:26.188 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:25 smithi121 bash[18489]: cluster 2024-02-12T16:35:24.775904+0000 osd.9 (osd.9) 135 : cluster [DBG] 7.a scrub starts 2024-02-12T16:35:26.188 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:25 smithi121 bash[18489]: cluster 2024-02-12T16:35:25.812722+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e783: 12 total, 12 up, 12 in 2024-02-12T16:35:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:26.189 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 16 136202 136186 135.094 77.3125 0.0136869 0.00740083 2024-02-12T16:35:26.267 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:35:26.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:26.322 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:26.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:26.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:26.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:26.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:26.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:26.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:26.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:26.856 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:26.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:26.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:26.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:27.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:27.095 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:26 smithi169 bash[19604]: cluster 2024-02-12T16:35:25.404207+0000 osd.11 (osd.11) 176 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:35:27.095 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:26 smithi169 bash[19604]: cluster 2024-02-12T16:35:25.445467+0000 osd.11 (osd.11) 177 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:35:27.095 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:26 smithi169 bash[19604]: cluster 2024-02-12T16:35:25.824372+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub starts 2024-02-12T16:35:27.095 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:26 smithi169 bash[19604]: cluster 2024-02-12T16:35:26.123395+0000 mgr.y (mgr.14146) 1524 : cluster [DBG] pgmap v2187: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 8.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-12T16:35:27.118 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:27.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:27.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:26 smithi121 bash[18489]: cluster 2024-02-12T16:35:25.404207+0000 osd.11 (osd.11) 176 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:35:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:26 smithi121 bash[18489]: cluster 2024-02-12T16:35:25.445467+0000 osd.11 (osd.11) 177 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:35:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:26 smithi121 bash[18489]: cluster 2024-02-12T16:35:25.824372+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub starts 2024-02-12T16:35:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:26 smithi121 bash[18489]: cluster 2024-02-12T16:35:26.123395+0000 mgr.y (mgr.14146) 1524 : cluster [DBG] pgmap v2187: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 8.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-12T16:35:27.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 137435 137419 134.187 77.0625 0.00639946 0.00745093 2024-02-12T16:35:27.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:27.247 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:27.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:27.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:27.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:27.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:26 smithi123 bash[19119]: cluster 2024-02-12T16:35:25.404207+0000 osd.11 (osd.11) 176 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:35:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:26 smithi123 bash[19119]: cluster 2024-02-12T16:35:25.445467+0000 osd.11 (osd.11) 177 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:35:27.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:26 smithi123 bash[19119]: cluster 2024-02-12T16:35:25.824372+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.9 scrub starts 2024-02-12T16:35:27.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:26 smithi123 bash[19119]: cluster 2024-02-12T16:35:26.123395+0000 mgr.y (mgr.14146) 1524 : cluster [DBG] pgmap v2187: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 8.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-02-12T16:35:27.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:27.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:27.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:27.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:27.634 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:27.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:27.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:27.764 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:27.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:27.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:27.891 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:27.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:27.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:28.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:28.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:27 smithi169 bash[19604]: cluster 2024-02-12T16:35:26.436205+0000 osd.7 (osd.7) 196 : cluster [DBG] 1.d scrub starts 2024-02-12T16:35:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:27 smithi169 bash[19604]: cluster 2024-02-12T16:35:26.539816+0000 osd.7 (osd.7) 197 : cluster [DBG] 1.d scrub ok 2024-02-12T16:35:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:27 smithi169 bash[19604]: cluster 2024-02-12T16:35:26.922560+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.c scrub ok 2024-02-12T16:35:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:27 smithi123 bash[19119]: cluster 2024-02-12T16:35:26.436205+0000 osd.7 (osd.7) 196 : cluster [DBG] 1.d scrub starts 2024-02-12T16:35:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:27 smithi123 bash[19119]: cluster 2024-02-12T16:35:26.539816+0000 osd.7 (osd.7) 197 : cluster [DBG] 1.d scrub ok 2024-02-12T16:35:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:27 smithi123 bash[19119]: cluster 2024-02-12T16:35:26.922560+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.c scrub ok 2024-02-12T16:35:28.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:28.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:28.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 138704 138688 133.343 79.3125 0.00652782 0.00749808 2024-02-12T16:35:28.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:28.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:28.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:28.315 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:27 smithi121 bash[18489]: cluster 2024-02-12T16:35:26.436205+0000 osd.7 (osd.7) 196 : cluster [DBG] 1.d scrub starts 2024-02-12T16:35:28.315 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:27 smithi121 bash[18489]: cluster 2024-02-12T16:35:26.539816+0000 osd.7 (osd.7) 197 : cluster [DBG] 1.d scrub ok 2024-02-12T16:35:28.315 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:27 smithi121 bash[18489]: cluster 2024-02-12T16:35:26.922560+0000 osd.6 (osd.6) 153 : cluster [DBG] 7.c scrub ok 2024-02-12T16:35:28.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:28.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:28.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:28.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:28.571 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:28.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:28.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:28.700 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:28.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:28.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:28.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:28.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:28.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:28.942 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:28 smithi123 bash[19119]: cluster 2024-02-12T16:35:27.977863+0000 osd.9 (osd.9) 136 : cluster [DBG] 7.a scrub ok 2024-02-12T16:35:28.942 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:28 smithi123 bash[19119]: cluster 2024-02-12T16:35:27.980648+0000 osd.2 (osd.2) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:28.942 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:28 smithi123 bash[19119]: cluster 2024-02-12T16:35:27.980682+0000 osd.2 (osd.2) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:28.942 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:28 smithi123 bash[19119]: cluster 2024-02-12T16:35:28.123856+0000 mgr.y (mgr.14146) 1525 : cluster [DBG] pgmap v2188: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-12T16:35:29.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:29.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:29.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:29.143 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:29.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:29.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 139914 139898 132.468 75.625 0.00918057 0.00754695 2024-02-12T16:35:29.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:29.275 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:29.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:28 smithi169 bash[19604]: cluster 2024-02-12T16:35:27.977863+0000 osd.9 (osd.9) 136 : cluster [DBG] 7.a scrub ok 2024-02-12T16:35:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:28 smithi169 bash[19604]: cluster 2024-02-12T16:35:27.980648+0000 osd.2 (osd.2) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:28 smithi169 bash[19604]: cluster 2024-02-12T16:35:27.980682+0000 osd.2 (osd.2) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:28 smithi169 bash[19604]: cluster 2024-02-12T16:35:28.123856+0000 mgr.y (mgr.14146) 1525 : cluster [DBG] pgmap v2188: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-12T16:35:29.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:29.401 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:29.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:29.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:28 smithi121 bash[18489]: cluster 2024-02-12T16:35:27.977863+0000 osd.9 (osd.9) 136 : cluster [DBG] 7.a scrub ok 2024-02-12T16:35:29.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:28 smithi121 bash[18489]: cluster 2024-02-12T16:35:27.980648+0000 osd.2 (osd.2) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:28 smithi121 bash[18489]: cluster 2024-02-12T16:35:27.980682+0000 osd.2 (osd.2) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:28 smithi121 bash[18489]: cluster 2024-02-12T16:35:28.123856+0000 mgr.y (mgr.14146) 1525 : cluster [DBG] pgmap v2188: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-02-12T16:35:29.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:29.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:29.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:29.680 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:29.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:29.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:29.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:29.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:29.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.8 to 1 (10000) 2024-02-12T16:35:29.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:30.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:30.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:30.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:30.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:30.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:30.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:29 smithi121 bash[18489]: cluster 2024-02-12T16:35:28.572334+0000 osd.2 (osd.2) 113 : cluster [DBG] 7.8 scrub starts 2024-02-12T16:35:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:29 smithi121 bash[18489]: cluster 2024-02-12T16:35:28.697717+0000 osd.6 (osd.6) 154 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:35:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:29 smithi121 bash[18489]: cluster 2024-02-12T16:35:28.794764+0000 osd.6 (osd.6) 155 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:35:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:29 smithi121 bash[18489]: cluster 2024-02-12T16:35:28.942311+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e784: 12 total, 12 up, 12 in 2024-02-12T16:35:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:29 smithi121 bash[18489]: audit 2024-02-12T16:35:29.187972+0000 mon.a (mon.0) 2669 : audit [INF] from='client.26584 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:35:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:29 smithi121 bash[18489]: audit 2024-02-12T16:35:29.188641+0000 mon.c (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.121:0/351744063' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:35:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:29 smithi121 bash[18489]: audit 2024-02-12T16:35:29.338948+0000 mon.b (mon.2) 184 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:35:30.188 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 141069 141053 131.568 72.1875 0.01821 0.00759887 2024-02-12T16:35:30.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:30.251 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:29 smithi123 bash[19119]: cluster 2024-02-12T16:35:28.572334+0000 osd.2 (osd.2) 113 : cluster [DBG] 7.8 scrub starts 2024-02-12T16:35:30.251 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:29 smithi123 bash[19119]: cluster 2024-02-12T16:35:28.697717+0000 osd.6 (osd.6) 154 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:35:30.251 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:29 smithi123 bash[19119]: cluster 2024-02-12T16:35:28.794764+0000 osd.6 (osd.6) 155 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:35:30.251 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:29 smithi123 bash[19119]: cluster 2024-02-12T16:35:28.942311+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e784: 12 total, 12 up, 12 in 2024-02-12T16:35:30.251 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:29 smithi123 bash[19119]: audit 2024-02-12T16:35:29.187972+0000 mon.a (mon.0) 2669 : audit [INF] from='client.26584 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:35:30.251 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:29 smithi123 bash[19119]: audit 2024-02-12T16:35:29.188641+0000 mon.c (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.121:0/351744063' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:35:30.251 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:29 smithi123 bash[19119]: audit 2024-02-12T16:35:29.338948+0000 mon.b (mon.2) 184 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:35:30.272 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:30.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:29 smithi169 bash[19604]: cluster 2024-02-12T16:35:28.572334+0000 osd.2 (osd.2) 113 : cluster [DBG] 7.8 scrub starts 2024-02-12T16:35:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:29 smithi169 bash[19604]: cluster 2024-02-12T16:35:28.697717+0000 osd.6 (osd.6) 154 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:35:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:29 smithi169 bash[19604]: cluster 2024-02-12T16:35:28.794764+0000 osd.6 (osd.6) 155 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:35:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:29 smithi169 bash[19604]: cluster 2024-02-12T16:35:28.942311+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e784: 12 total, 12 up, 12 in 2024-02-12T16:35:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:29 smithi169 bash[19604]: audit 2024-02-12T16:35:29.187972+0000 mon.a (mon.0) 2669 : audit [INF] from='client.26584 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:35:30.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:29 smithi169 bash[19604]: audit 2024-02-12T16:35:29.188641+0000 mon.c (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.121:0/351744063' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:35:30.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:29 smithi169 bash[19604]: audit 2024-02-12T16:35:29.338948+0000 mon.b (mon.2) 184 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:35:30.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:30.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:30.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:30.515 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:30.695 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:30.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:30.722 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:35:30.722 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:35:30.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:30.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:30.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:30.957 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:30 smithi123 bash[19119]: cluster 2024-02-12T16:35:29.184432+0000 osd.3 (osd.3) 70 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:35:30.957 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:30 smithi123 bash[19119]: cluster 2024-02-12T16:35:29.795888+0000 osd.0 (osd.0) 133 : cluster [DBG] 7.9 scrub ok 2024-02-12T16:35:30.957 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:30 smithi123 bash[19119]: cluster 2024-02-12T16:35:29.919621+0000 osd.0 (osd.0) 134 : cluster [DBG] 7.4 scrub starts 2024-02-12T16:35:30.957 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:30 smithi123 bash[19119]: audit 2024-02-12T16:35:29.943370+0000 mon.a (mon.0) 2670 : audit [INF] from='client.26584 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:35:30.957 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:30 smithi123 bash[19119]: cluster 2024-02-12T16:35:29.943431+0000 mon.a (mon.0) 2671 : cluster [DBG] osdmap e785: 12 total, 12 up, 12 in 2024-02-12T16:35:30.958 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:30 smithi123 bash[19119]: cluster 2024-02-12T16:35:30.124259+0000 mgr.y (mgr.14146) 1526 : cluster [DBG] pgmap v2191: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-02-12T16:35:30.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:30 smithi169 bash[19604]: cluster 2024-02-12T16:35:29.184432+0000 osd.3 (osd.3) 70 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:35:31.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:30 smithi169 bash[19604]: cluster 2024-02-12T16:35:29.795888+0000 osd.0 (osd.0) 133 : cluster [DBG] 7.9 scrub ok 2024-02-12T16:35:31.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:30 smithi169 bash[19604]: cluster 2024-02-12T16:35:29.919621+0000 osd.0 (osd.0) 134 : cluster [DBG] 7.4 scrub starts 2024-02-12T16:35:31.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:30 smithi169 bash[19604]: audit 2024-02-12T16:35:29.943370+0000 mon.a (mon.0) 2670 : audit [INF] from='client.26584 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:35:31.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:30 smithi169 bash[19604]: cluster 2024-02-12T16:35:29.943431+0000 mon.a (mon.0) 2671 : cluster [DBG] osdmap e785: 12 total, 12 up, 12 in 2024-02-12T16:35:31.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:30 smithi169 bash[19604]: cluster 2024-02-12T16:35:30.124259+0000 mgr.y (mgr.14146) 1526 : cluster [DBG] pgmap v2191: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-02-12T16:35:31.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:31.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:30 smithi121 bash[18489]: cluster 2024-02-12T16:35:29.184432+0000 osd.3 (osd.3) 70 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:35:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:30 smithi121 bash[18489]: cluster 2024-02-12T16:35:29.795888+0000 osd.0 (osd.0) 133 : cluster [DBG] 7.9 scrub ok 2024-02-12T16:35:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:30 smithi121 bash[18489]: cluster 2024-02-12T16:35:29.919621+0000 osd.0 (osd.0) 134 : cluster [DBG] 7.4 scrub starts 2024-02-12T16:35:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:30 smithi121 bash[18489]: audit 2024-02-12T16:35:29.943370+0000 mon.a (mon.0) 2670 : audit [INF] from='client.26584 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:35:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:30 smithi121 bash[18489]: cluster 2024-02-12T16:35:29.943431+0000 mon.a (mon.0) 2671 : cluster [DBG] osdmap e785: 12 total, 12 up, 12 in 2024-02-12T16:35:31.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:30 smithi121 bash[18489]: cluster 2024-02-12T16:35:30.124259+0000 mgr.y (mgr.14146) 1526 : cluster [DBG] pgmap v2191: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-02-12T16:35:31.189 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 16 142067 142051 130.551 62.375 0.0157801 0.00765824 2024-02-12T16:35:31.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:31.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:31.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:31.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:31.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:31.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:31.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:31.500 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T16:35:31.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:35:31.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-12T16:35:31.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:31.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:31.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:31.663 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:31.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:31.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:31.795 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:31.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:31.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:31.912 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:31.915 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:31.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:31.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:32.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:32.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:31 smithi169 bash[19604]: cluster 2024-02-12T16:35:30.719832+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.7 scrub starts 2024-02-12T16:35:32.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:31 smithi121 bash[18489]: cluster 2024-02-12T16:35:30.719832+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.7 scrub starts 2024-02-12T16:35:32.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:32.189 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 143210 143194 129.694 71.4375 0.00527198 0.00770823 2024-02-12T16:35:32.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:32.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:32.306 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:31 smithi123 bash[19119]: cluster 2024-02-12T16:35:30.719832+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.7 scrub starts 2024-02-12T16:35:32.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:32.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:32.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:32.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:32.583 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:32.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:32.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:32.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:32.853 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:32.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:32.961 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:32 smithi169 bash[19604]: cluster 2024-02-12T16:35:32.124790+0000 mgr.y (mgr.14146) 1527 : cluster [DBG] pgmap v2192: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:35:32.961 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:32 smithi169 bash[19604]: cluster 2024-02-12T16:35:32.405625+0000 osd.2 (osd.2) 114 : cluster [DBG] 7.8 scrub ok 2024-02-12T16:35:32.982 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:32.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:33.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:33.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:33.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:32 smithi123 bash[19119]: cluster 2024-02-12T16:35:32.124790+0000 mgr.y (mgr.14146) 1527 : cluster [DBG] pgmap v2192: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:35:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:32 smithi123 bash[19119]: cluster 2024-02-12T16:35:32.405625+0000 osd.2 (osd.2) 114 : cluster [DBG] 7.8 scrub ok 2024-02-12T16:35:33.189 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 16 144448 144432 128.946 77.375 0.0134803 0.00775341 2024-02-12T16:35:33.228 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:33.261 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:33.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:33.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:33.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:33.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:33.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:32 smithi121 bash[18489]: cluster 2024-02-12T16:35:32.124790+0000 mgr.y (mgr.14146) 1527 : cluster [DBG] pgmap v2192: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-02-12T16:35:33.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:32 smithi121 bash[18489]: cluster 2024-02-12T16:35:32.405625+0000 osd.2 (osd.2) 114 : cluster [DBG] 7.8 scrub ok 2024-02-12T16:35:33.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:33.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:33.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:33.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:33.662 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:33.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:33.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:33.795 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:33.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:33.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:33.931 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:33.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:34.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:34.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:34.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:34.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:33 smithi123 bash[19119]: cluster 2024-02-12T16:35:32.811272+0000 osd.0 (osd.0) 135 : cluster [DBG] 7.4 scrub ok 2024-02-12T16:35:34.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:33 smithi123 bash[19119]: cluster 2024-02-12T16:35:32.830406+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.7 scrub ok 2024-02-12T16:35:34.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:33 smithi123 bash[19119]: cluster 2024-02-12T16:35:32.912320+0000 osd.1 (osd.1) 122 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:35:34.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:33 smithi123 bash[19119]: cluster 2024-02-12T16:35:33.040353+0000 osd.1 (osd.1) 123 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:35:34.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:34.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 145764 145748 128.285 82.25 0.00754169 0.00779329 2024-02-12T16:35:34.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:34.215 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:33 smithi169 bash[19604]: cluster 2024-02-12T16:35:32.811272+0000 osd.0 (osd.0) 135 : cluster [DBG] 7.4 scrub ok 2024-02-12T16:35:34.215 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:33 smithi169 bash[19604]: cluster 2024-02-12T16:35:32.830406+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.7 scrub ok 2024-02-12T16:35:34.215 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:33 smithi169 bash[19604]: cluster 2024-02-12T16:35:32.912320+0000 osd.1 (osd.1) 122 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:35:34.215 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:33 smithi169 bash[19604]: cluster 2024-02-12T16:35:33.040353+0000 osd.1 (osd.1) 123 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:35:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:34.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:34.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:33 smithi121 bash[18489]: cluster 2024-02-12T16:35:32.811272+0000 osd.0 (osd.0) 135 : cluster [DBG] 7.4 scrub ok 2024-02-12T16:35:34.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:33 smithi121 bash[18489]: cluster 2024-02-12T16:35:32.830406+0000 osd.6 (osd.6) 157 : cluster [DBG] 7.7 scrub ok 2024-02-12T16:35:34.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:33 smithi121 bash[18489]: cluster 2024-02-12T16:35:32.912320+0000 osd.1 (osd.1) 122 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:35:34.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:33 smithi121 bash[18489]: cluster 2024-02-12T16:35:33.040353+0000 osd.1 (osd.1) 123 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:35:34.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:34.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:34.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:34.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:34.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:34.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:34.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:34.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:34.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:34.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:34.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:34.871 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:34.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:34.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:35.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:35.113 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:35.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:35.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 147045 147029 127.615 80.0625 0.00600169 0.00783448 2024-02-12T16:35:35.239 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:35 smithi121 bash[18489]: cluster 2024-02-12T16:35:34.125255+0000 mgr.y (mgr.14146) 1528 : cluster [DBG] pgmap v2193: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:35:35.239 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:35 smithi121 bash[18489]: cluster 2024-02-12T16:35:34.717304+0000 osd.10 (osd.10) 162 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:35:35.239 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:35 smithi121 bash[18489]: cluster 2024-02-12T16:35:34.775047+0000 osd.10 (osd.10) 163 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:35:35.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:35.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:34 smithi169 bash[19604]: cluster 2024-02-12T16:35:34.125255+0000 mgr.y (mgr.14146) 1528 : cluster [DBG] pgmap v2193: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:35:35.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:34 smithi169 bash[19604]: cluster 2024-02-12T16:35:34.717304+0000 osd.10 (osd.10) 162 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:35:35.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:34 smithi169 bash[19604]: cluster 2024-02-12T16:35:34.775047+0000 osd.10 (osd.10) 163 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:35:35.268 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:35.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:35.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:35 smithi123 bash[19119]: cluster 2024-02-12T16:35:34.125255+0000 mgr.y (mgr.14146) 1528 : cluster [DBG] pgmap v2193: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:35:35.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:35 smithi123 bash[19119]: cluster 2024-02-12T16:35:34.717304+0000 osd.10 (osd.10) 162 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:35:35.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:35 smithi123 bash[19119]: cluster 2024-02-12T16:35:34.775047+0000 osd.10 (osd.10) 163 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:35:35.400 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:35.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:35.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:35.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:35.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:35.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.796 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:35.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:35.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:35.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:35.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:35.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:36.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:36.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:35 smithi169 bash[19604]: cluster 2024-02-12T16:35:34.921672+0000 osd.5 (osd.5) 163 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:35:36.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:35 smithi169 bash[19604]: cluster 2024-02-12T16:35:34.988663+0000 osd.5 (osd.5) 164 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:35:36.053 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:36.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:36.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 148306 148290 126.947 78.8125 0.0105888 0.0078755 2024-02-12T16:35:36.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:36.307 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:35:36.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:36.316 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:35 smithi123 bash[19119]: cluster 2024-02-12T16:35:34.921672+0000 osd.5 (osd.5) 163 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:35:36.316 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:35 smithi123 bash[19119]: cluster 2024-02-12T16:35:34.988663+0000 osd.5 (osd.5) 164 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:35:36.334 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:35 smithi121 bash[18489]: cluster 2024-02-12T16:35:34.921672+0000 osd.5 (osd.5) 163 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:35:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:35 smithi121 bash[18489]: cluster 2024-02-12T16:35:34.988663+0000 osd.5 (osd.5) 164 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:35:36.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:36.463 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:36.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:36.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:36.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:36.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:36.827 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:36.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:36.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:36.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:36.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:37.090 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:37.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:37.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:37.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 149476 149460 126.219 73.125 0.0213129 0.00792101 2024-02-12T16:35:37.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:37.248 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:37.250 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:36 smithi169 bash[19604]: cluster 2024-02-12T16:35:36.125720+0000 mgr.y (mgr.14146) 1529 : cluster [DBG] pgmap v2194: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 8.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-02-12T16:35:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:37.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:37.316 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:37.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:37.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:36 smithi121 bash[18489]: cluster 2024-02-12T16:35:36.125720+0000 mgr.y (mgr.14146) 1529 : cluster [DBG] pgmap v2194: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 8.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-02-12T16:35:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:36 smithi123 bash[19119]: cluster 2024-02-12T16:35:36.125720+0000 mgr.y (mgr.14146) 1529 : cluster [DBG] pgmap v2194: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 8.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-02-12T16:35:37.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:37.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:37.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:37.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:37.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:37.710 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:37.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:37.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:37.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:37.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:37.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:37.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:37.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:38.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:38.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:38.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:38.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 16 150712 150696 125.566 77.25 0.00580533 0.00796233 2024-02-12T16:35:38.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:38.264 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:38.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:38.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:38.394 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:38.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:38.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:38.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:38.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:38.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:38.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:38 smithi123 bash[19119]: audit 2024-02-12T16:35:37.369581+0000 mon.a (mon.0) 2672 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:38.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:38 smithi123 bash[19119]: audit 2024-02-12T16:35:37.734649+0000 mon.b (mon.2) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:35:38.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:38 smithi123 bash[19119]: audit 2024-02-12T16:35:37.735528+0000 mon.b (mon.2) 186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:35:38.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:38 smithi123 bash[19119]: audit 2024-02-12T16:35:37.741030+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:38.640 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:38 smithi121 bash[18489]: audit 2024-02-12T16:35:37.369581+0000 mon.a (mon.0) 2672 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:38.640 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:38 smithi121 bash[18489]: audit 2024-02-12T16:35:37.734649+0000 mon.b (mon.2) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:35:38.640 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:38 smithi121 bash[18489]: audit 2024-02-12T16:35:37.735528+0000 mon.b (mon.2) 186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:35:38.640 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:38 smithi121 bash[18489]: audit 2024-02-12T16:35:37.741030+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:38.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:38.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:38.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:38.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:38 smithi169 bash[19604]: audit 2024-02-12T16:35:37.369581+0000 mon.a (mon.0) 2672 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:38 smithi169 bash[19604]: audit 2024-02-12T16:35:37.734649+0000 mon.b (mon.2) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:35:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:38 smithi169 bash[19604]: audit 2024-02-12T16:35:37.735528+0000 mon.b (mon.2) 186 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:35:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:38 smithi169 bash[19604]: audit 2024-02-12T16:35:37.741030+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:38.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:38.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:38.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:39.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:39.056 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:39.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:39.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:39.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 151869 151853 124.865 72.3125 0.0179361 0.00800718 2024-02-12T16:35:39.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:39.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:39.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:39.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:39.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:39.376 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:39 smithi123 bash[19119]: cluster 2024-02-12T16:35:37.535995+0000 osd.4 (osd.4) 134 : cluster [DBG] 7.1 scrub starts 2024-02-12T16:35:39.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:39 smithi123 bash[19119]: cluster 2024-02-12T16:35:38.126149+0000 mgr.y (mgr.14146) 1530 : cluster [DBG] pgmap v2195: 47 pgs: 47 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-12T16:35:39.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:39 smithi123 bash[19119]: cluster 2024-02-12T16:35:38.487276+0000 osd.5 (osd.5) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:39.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:39 smithi123 bash[19119]: cluster 2024-02-12T16:35:38.487333+0000 osd.5 (osd.5) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:39.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:39.474 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:39.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:39.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:39.609 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:39.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:39.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:39 smithi121 bash[18489]: cluster 2024-02-12T16:35:37.535995+0000 osd.4 (osd.4) 134 : cluster [DBG] 7.1 scrub starts 2024-02-12T16:35:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:39 smithi121 bash[18489]: cluster 2024-02-12T16:35:38.126149+0000 mgr.y (mgr.14146) 1530 : cluster [DBG] pgmap v2195: 47 pgs: 47 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-12T16:35:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:39 smithi121 bash[18489]: cluster 2024-02-12T16:35:38.487276+0000 osd.5 (osd.5) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:39 smithi121 bash[18489]: cluster 2024-02-12T16:35:38.487333+0000 osd.5 (osd.5) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:39.712 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:39.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:39 smithi169 bash[19604]: cluster 2024-02-12T16:35:37.535995+0000 osd.4 (osd.4) 134 : cluster [DBG] 7.1 scrub starts 2024-02-12T16:35:39.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:39 smithi169 bash[19604]: cluster 2024-02-12T16:35:38.126149+0000 mgr.y (mgr.14146) 1530 : cluster [DBG] pgmap v2195: 47 pgs: 47 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s 2024-02-12T16:35:39.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:39 smithi169 bash[19604]: cluster 2024-02-12T16:35:38.487276+0000 osd.5 (osd.5) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:39.718 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:39 smithi169 bash[19604]: cluster 2024-02-12T16:35:38.487333+0000 osd.5 (osd.5) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:39.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:39.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:39.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:39.865 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:39.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:39.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:39.999 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:40.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:40.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:40.152 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:40.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:40.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 15 153015 153000 124.175 71.6875 0.00771045 0.00805165 2024-02-12T16:35:40.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:40.305 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:40.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:40.362 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:35:40.362 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:35:40.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:40.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:40.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:40.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:40.602 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:40.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:40 smithi123 bash[19119]: cluster 2024-02-12T16:35:38.882280+0000 osd.1 (osd.1) 124 : cluster [DBG] 7.b scrub starts 2024-02-12T16:35:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:40 smithi123 bash[19119]: cluster 2024-02-12T16:35:39.378705+0000 mon.a (mon.0) 2674 : cluster [DBG] osdmap e786: 12 total, 12 up, 12 in 2024-02-12T16:35:40.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:40.712 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:40 smithi169 bash[19604]: cluster 2024-02-12T16:35:38.882280+0000 osd.1 (osd.1) 124 : cluster [DBG] 7.b scrub starts 2024-02-12T16:35:40.712 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:40 smithi169 bash[19604]: cluster 2024-02-12T16:35:39.378705+0000 mon.a (mon.0) 2674 : cluster [DBG] osdmap e786: 12 total, 12 up, 12 in 2024-02-12T16:35:40.724 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:40 smithi121 bash[18489]: cluster 2024-02-12T16:35:38.882280+0000 osd.1 (osd.1) 124 : cluster [DBG] 7.b scrub starts 2024-02-12T16:35:40.724 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:40 smithi121 bash[18489]: cluster 2024-02-12T16:35:39.378705+0000 mon.a (mon.0) 2674 : cluster [DBG] osdmap e786: 12 total, 12 up, 12 in 2024-02-12T16:35:40.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:40.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:40.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:40.880 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:40.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:40.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:41.025 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:41.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:41.133 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:35:41.169 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:35:41.176 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:41.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-12T16:35:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:41.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 154115 154099 123.463 68.6875 0.00661644 0.00809813 2024-02-12T16:35:41.197 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:35:41.237 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:35:41.267 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:35:41.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:41.303 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:35:41.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:41.334 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:35:41.366 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:35:41.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:41 smithi121 bash[18489]: cluster 2024-02-12T16:35:40.126570+0000 mgr.y (mgr.14146) 1531 : cluster [DBG] pgmap v2197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:35:41.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:41 smithi121 bash[18489]: cluster 2024-02-12T16:35:40.233261+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:35:41.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:41 smithi121 bash[18489]: cluster 2024-02-12T16:35:40.400924+0000 osd.3 (osd.3) 73 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:35:41.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:41 smithi121 bash[18489]: audit 2024-02-12T16:35:40.501649+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? v1:172.21.15.121:0/2233759319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:35:41.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:41 smithi121 bash[18489]: audit 2024-02-12T16:35:41.089330+0000 mon.a (mon.0) 2676 : audit [DBG] from='client.? v1:172.21.15.121:0/3434267646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:41.400 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:35:41.401 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:35:41.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:41.443 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:35:41.459 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:41.479 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:35:41.511 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:35:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:35:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:41 smithi169 bash[19604]: cluster 2024-02-12T16:35:40.126570+0000 mgr.y (mgr.14146) 1531 : cluster [DBG] pgmap v2197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:35:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:41 smithi169 bash[19604]: cluster 2024-02-12T16:35:40.233261+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:35:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:41 smithi169 bash[19604]: cluster 2024-02-12T16:35:40.400924+0000 osd.3 (osd.3) 73 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:35:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:41 smithi169 bash[19604]: audit 2024-02-12T16:35:40.501649+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? v1:172.21.15.121:0/2233759319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:35:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:41 smithi169 bash[19604]: audit 2024-02-12T16:35:41.089330+0000 mon.a (mon.0) 2676 : audit [DBG] from='client.? v1:172.21.15.121:0/3434267646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:41.566 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:41.593 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:41.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:41.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:41.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:41.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:41.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:41 smithi123 bash[19119]: cluster 2024-02-12T16:35:40.126570+0000 mgr.y (mgr.14146) 1531 : cluster [DBG] pgmap v2197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-02-12T16:35:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:41 smithi123 bash[19119]: cluster 2024-02-12T16:35:40.233261+0000 osd.3 (osd.3) 72 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:35:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:41 smithi123 bash[19119]: cluster 2024-02-12T16:35:40.400924+0000 osd.3 (osd.3) 73 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:35:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:41 smithi123 bash[19119]: audit 2024-02-12T16:35:40.501649+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? v1:172.21.15.121:0/2233759319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:35:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:41 smithi123 bash[19119]: audit 2024-02-12T16:35:41.089330+0000 mon.a (mon.0) 2676 : audit [DBG] from='client.? v1:172.21.15.121:0/3434267646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:35:41.952 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:41.975 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:41.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:42.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:42.102 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:42.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:42.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 155330 155314 122.862 75.9375 0.00738035 0.00813685 2024-02-12T16:35:42.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:42.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:42.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:42.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:42.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:42.401 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:42 smithi169 bash[19604]: cluster 2024-02-12T16:35:40.519582+0000 osd.4 (osd.4) 135 : cluster [DBG] 7.1 scrub ok 2024-02-12T16:35:42.401 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:42 smithi169 bash[19604]: cluster 2024-02-12T16:35:41.381617+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:35:42.401 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:42 smithi169 bash[19604]: audit 2024-02-12T16:35:41.388395+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? v1:172.21.15.121:0/2233759319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:35:42.402 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:42 smithi169 bash[19604]: cluster 2024-02-12T16:35:41.388483+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e787: 12 total, 12 up, 12 in 2024-02-12T16:35:42.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:42.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:42.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:42.602 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:42.610 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:42 smithi123 bash[19119]: cluster 2024-02-12T16:35:40.519582+0000 osd.4 (osd.4) 135 : cluster [DBG] 7.1 scrub ok 2024-02-12T16:35:42.610 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:42 smithi123 bash[19119]: cluster 2024-02-12T16:35:41.381617+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:35:42.610 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:42 smithi123 bash[19119]: audit 2024-02-12T16:35:41.388395+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? v1:172.21.15.121:0/2233759319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:35:42.610 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:42 smithi123 bash[19119]: cluster 2024-02-12T16:35:41.388483+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e787: 12 total, 12 up, 12 in 2024-02-12T16:35:42.629 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:42.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:42.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:42 smithi121 bash[18489]: cluster 2024-02-12T16:35:40.519582+0000 osd.4 (osd.4) 135 : cluster [DBG] 7.1 scrub ok 2024-02-12T16:35:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:42 smithi121 bash[18489]: cluster 2024-02-12T16:35:41.381617+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:35:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:42 smithi121 bash[18489]: audit 2024-02-12T16:35:41.388395+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? v1:172.21.15.121:0/2233759319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:35:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:42 smithi121 bash[18489]: cluster 2024-02-12T16:35:41.388483+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e787: 12 total, 12 up, 12 in 2024-02-12T16:35:42.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:42.768 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:42.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:42.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:42.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:42.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:43.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:43.042 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:43.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.173 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:43.191 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:35:43.188682 min lat: 0.00104071 max lat: 0.477087 avg lat: 0.00817489 2024-02-12T16:35:43.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:35:43.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 156582 156566 122.304 78.25 0.0166707 0.00817489 2024-02-12T16:35:43.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:43.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:43.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:43.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:43.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:43 smithi121 bash[18489]: cluster 2024-02-12T16:35:42.127024+0000 mgr.y (mgr.14146) 1532 : cluster [DBG] pgmap v2199: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-02-12T16:35:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:43 smithi121 bash[18489]: cluster 2024-02-12T16:35:42.742636+0000 osd.3 (osd.3) 74 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:43 smithi121 bash[18489]: cluster 2024-02-12T16:35:42.742709+0000 osd.3 (osd.3) 75 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:43.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:43.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:43 smithi169 bash[19604]: cluster 2024-02-12T16:35:42.127024+0000 mgr.y (mgr.14146) 1532 : cluster [DBG] pgmap v2199: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-02-12T16:35:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:43 smithi169 bash[19604]: cluster 2024-02-12T16:35:42.742636+0000 osd.3 (osd.3) 74 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:43 smithi169 bash[19604]: cluster 2024-02-12T16:35:42.742709+0000 osd.3 (osd.3) 75 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:43.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.821 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:43 smithi123 bash[19119]: cluster 2024-02-12T16:35:42.127024+0000 mgr.y (mgr.14146) 1532 : cluster [DBG] pgmap v2199: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-02-12T16:35:43.822 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:43 smithi123 bash[19119]: cluster 2024-02-12T16:35:42.742636+0000 osd.3 (osd.3) 74 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:43.822 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:43 smithi123 bash[19119]: cluster 2024-02-12T16:35:42.742709+0000 osd.3 (osd.3) 75 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:43.841 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:43.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:43.982 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:43.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:44.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:44.132 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:44.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:44.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 157846 157830 121.769 79 0.0104666 0.00821087 2024-02-12T16:35:44.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:44.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:44.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:44.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:44.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:44.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:44.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:44.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:44.545 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:44 smithi121 bash[18489]: cluster 2024-02-12T16:35:43.400669+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e788: 12 total, 12 up, 12 in 2024-02-12T16:35:44.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:44.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:44 smithi123 bash[19119]: cluster 2024-02-12T16:35:43.400669+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e788: 12 total, 12 up, 12 in 2024-02-12T16:35:44.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:44.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:44.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:44.788 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:44 smithi169 bash[19604]: cluster 2024-02-12T16:35:43.400669+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e788: 12 total, 12 up, 12 in 2024-02-12T16:35:44.804 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:44.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:44.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:44.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:45.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:45.080 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:35:45.082 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:35:45.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:45.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:45.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:45.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 158997 158981 121.161 71.9375 0.00620538 0.00825187 2024-02-12T16:35:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:45.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:45.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:45.362 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:45.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:45.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:45 smithi169 bash[19604]: cluster 2024-02-12T16:35:44.127521+0000 mgr.y (mgr.14146) 1533 : cluster [DBG] pgmap v2201: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 971 op/s 2024-02-12T16:35:45.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:45.476 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:45 smithi121 bash[18489]: cluster 2024-02-12T16:35:44.127521+0000 mgr.y (mgr.14146) 1533 : cluster [DBG] pgmap v2201: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 971 op/s 2024-02-12T16:35:45.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:45.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:45.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:45.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:45.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:45 smithi123 bash[19119]: cluster 2024-02-12T16:35:44.127521+0000 mgr.y (mgr.14146) 1533 : cluster [DBG] pgmap v2201: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 9.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 971 op/s 2024-02-12T16:35:45.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:45.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:45.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:45.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:45.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:45.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:45.968 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:35:46.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:46.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:46.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:46.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:46.194 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 160188 160172 120.598 74.4375 0.0265543 0.00829043 2024-02-12T16:35:46.205 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2201,"stamp":"2024-02-12T16:35:44.127216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10094116864,"num_objects":154024,"num_object_clones":0,"num_object_copies":308048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154024,"num_write_kb":9857536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154024,"ondisk_log_size":154024,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":26630940,"kb_used_data":19745252,"kb_used_omap":91,"kb_used_meta":6882980,"kb_avail":1098310884,"statfs":{"total":1151940427776,"available":1124670345216,"internally_reserved":0,"allocated":20219138048,"data_stored":20213278966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":7048171536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":381943808,"num_objects":5828,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":372992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001319"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":145,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.945999+0000","last_change":"2024-02-12T16:35:25.445580+0000","last_active":"2024-02-12T16:35:29.945999+0000","last_peered":"2024-02-12T16:35:29.945999+0000","last_clean":"2024-02-12T16:35:29.945999+0000","last_became_active":"2024-02-12T16:30:55.395625+0000","last_became_peered":"2024-02-12T16:30:55.395625+0000","last_unstale":"2024-02-12T16:35:29.945999+0000","last_undegraded":"2024-02-12T16:35:29.945999+0000","last_fullsized":"2024-02-12T16:35:29.945999+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:25.445474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:25.445474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":148,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.391641+0000","last_change":"2024-02-12T16:35:18.045317+0000","last_active":"2024-02-12T16:35:39.391641+0000","last_peered":"2024-02-12T16:35:39.391641+0000","last_clean":"2024-02-12T16:35:39.391641+0000","last_became_active":"2024-02-12T16:30:55.398431+0000","last_became_peered":"2024-02-12T16:30:55.398431+0000","last_unstale":"2024-02-12T16:35:39.391641+0000","last_undegraded":"2024-02-12T16:35:39.391641+0000","last_fullsized":"2024-02-12T16:35:39.391641+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:18.045236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:18.045236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":146,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.408038+0000","last_change":"2024-02-12T16:35:15.289078+0000","last_active":"2024-02-12T16:35:41.408038+0000","last_peered":"2024-02-12T16:35:41.408038+0000","last_clean":"2024-02-12T16:35:41.408038+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:35:41.408038+0000","last_undegraded":"2024-02-12T16:35:41.408038+0000","last_fullsized":"2024-02-12T16:35:41.408038+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:15.288966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:15.288966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":146,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.408084+0000","last_change":"2024-02-12T16:35:04.397528+0000","last_active":"2024-02-12T16:35:41.408084+0000","last_peered":"2024-02-12T16:35:41.408084+0000","last_clean":"2024-02-12T16:35:41.408084+0000","last_became_active":"2024-02-12T16:30:55.409579+0000","last_became_peered":"2024-02-12T16:30:55.409579+0000","last_unstale":"2024-02-12T16:35:41.408084+0000","last_undegraded":"2024-02-12T16:35:41.408084+0000","last_fullsized":"2024-02-12T16:35:41.408084+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:04.397374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:04.397374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":193,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.951530+0000","last_change":"2024-02-12T16:35:10.432374+0000","last_active":"2024-02-12T16:35:29.951530+0000","last_peered":"2024-02-12T16:35:29.951530+0000","last_clean":"2024-02-12T16:35:29.951530+0000","last_became_active":"2024-02-12T16:30:55.394105+0000","last_became_peered":"2024-02-12T16:30:55.394105+0000","last_unstale":"2024-02-12T16:35:29.951530+0000","last_undegraded":"2024-02-12T16:35:29.951530+0000","last_fullsized":"2024-02-12T16:35:29.951530+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:10.432253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:35:10.432253+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":146,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.396973+0000","last_change":"2024-02-12T16:35:18.834610+0000","last_active":"2024-02-12T16:35:41.396973+0000","last_peered":"2024-02-12T16:35:41.396973+0000","last_clean":"2024-02-12T16:35:41.396973+0000","last_became_active":"2024-02-12T16:30:55.542451+0000","last_became_peered":"2024-02-12T16:30:55.542451+0000","last_unstale":"2024-02-12T16:35:41.396973+0000","last_undegraded":"2024-02-12T16:35:41.396973+0000","last_fullsized":"2024-02-12T16:35:41.396973+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:18.834357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:18.834357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":501,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.379146+0000","last_change":"2024-02-12T16:35:34.988787+0000","last_active":"2024-02-12T16:35:39.379146+0000","last_peered":"2024-02-12T16:35:39.379146+0000","last_clean":"2024-02-12T16:35:39.379146+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:35:39.379146+0000","last_undegraded":"2024-02-12T16:35:39.379146+0000","last_fullsized":"2024-02-12T16:35:39.379146+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:34.988672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:35:34.988672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":145,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.411579+0000","last_change":"2024-02-12T16:35:08.385373+0000","last_active":"2024-02-12T16:35:41.411579+0000","last_peered":"2024-02-12T16:35:41.411579+0000","last_clean":"2024-02-12T16:35:41.411579+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:35:41.411579+0000","last_undegraded":"2024-02-12T16:35:41.411579+0000","last_fullsized":"2024-02-12T16:35:41.411579+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:08.385272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:08.385272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":150,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.392060+0000","last_change":"2024-02-12T16:35:33.040481+0000","last_active":"2024-02-12T16:35:39.392060+0000","last_peered":"2024-02-12T16:35:39.392060+0000","last_clean":"2024-02-12T16:35:39.392060+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:35:39.392060+0000","last_undegraded":"2024-02-12T16:35:39.392060+0000","last_fullsized":"2024-02-12T16:35:39.392060+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:33.040362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:33.040362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":140,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.399178+0000","last_change":"2024-02-12T16:35:03.687751+0000","last_active":"2024-02-12T16:35:41.399178+0000","last_peered":"2024-02-12T16:35:41.399178+0000","last_clean":"2024-02-12T16:35:41.399178+0000","last_became_active":"2024-02-12T16:30:55.404763+0000","last_became_peered":"2024-02-12T16:30:55.404763+0000","last_unstale":"2024-02-12T16:35:41.399178+0000","last_undegraded":"2024-02-12T16:35:41.399178+0000","last_fullsized":"2024-02-12T16:35:41.399178+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:03.687669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:35:03.687669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":351,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.382661+0000","last_change":"2024-02-12T16:35:16.894972+0000","last_active":"2024-02-12T16:35:39.382661+0000","last_peered":"2024-02-12T16:35:39.382661+0000","last_clean":"2024-02-12T16:35:39.382661+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:35:39.382661+0000","last_undegraded":"2024-02-12T16:35:39.382661+0000","last_fullsized":"2024-02-12T16:35:39.382661+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:07.718055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:35:07.718055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"786'9617","reported_seq":9663,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.262989+0000","last_change":"2024-02-12T16:35:18.367383+0000","last_active":"2024-02-12T16:35:41.262989+0000","last_peered":"2024-02-12T16:35:41.262989+0000","last_clean":"2024-02-12T16:35:41.262989+0000","last_became_active":"2024-02-12T16:34:08.148188+0000","last_became_peered":"2024-02-12T16:34:08.148188+0000","last_unstale":"2024-02-12T16:35:41.262989+0000","last_undegraded":"2024-02-12T16:35:41.262989+0000","last_fullsized":"2024-02-12T16:35:41.262989+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'7864","last_scrub_stamp":"2024-02-12T16:35:18.367314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:18.367314+0000","log_size":9617,"ondisk_log_size":9617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630259712,"num_objects":9617,"num_object_clones":0,"num_object_copies":19234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9617,"num_write_kb":615488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.392706+0000","last_change":"2024-02-12T16:35:08.826049+0000","last_active":"2024-02-12T16:35:39.392706+0000","last_peered":"2024-02-12T16:35:39.392706+0000","last_clean":"2024-02-12T16:35:39.392706+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:35:39.392706+0000","last_undegraded":"2024-02-12T16:35:39.392706+0000","last_fullsized":"2024-02-12T16:35:39.392706+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:08.825938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:08.825938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.3","version":"786'9670","reported_seq":9718,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.078149+0000","last_change":"2024-02-12T16:35:19.003072+0000","last_active":"2024-02-12T16:35:41.078149+0000","last_peered":"2024-02-12T16:35:41.078149+0000","last_clean":"2024-02-12T16:35:41.078149+0000","last_became_active":"2024-02-12T16:34:08.036294+0000","last_became_peered":"2024-02-12T16:34:08.036294+0000","last_unstale":"2024-02-12T16:35:41.078149+0000","last_undegraded":"2024-02-12T16:35:41.078149+0000","last_fullsized":"2024-02-12T16:35:41.078149+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'7952","last_scrub_stamp":"2024-02-12T16:35:19.002953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:19.002953+0000","log_size":9670,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":633733120,"num_objects":9670,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9670,"num_write_kb":618880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":506,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.393843+0000","last_change":"2024-02-12T16:35:34.775120+0000","last_active":"2024-02-12T16:35:41.393843+0000","last_peered":"2024-02-12T16:35:41.393843+0000","last_clean":"2024-02-12T16:35:41.393843+0000","last_became_active":"2024-02-12T16:30:49.374358+0000","last_became_peered":"2024-02-12T16:30:49.374358+0000","last_unstale":"2024-02-12T16:35:41.393843+0000","last_undegraded":"2024-02-12T16:35:41.393843+0000","last_fullsized":"2024-02-12T16:35:41.393843+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:34.775051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:35:34.775051+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.2","version":"786'9659","reported_seq":9705,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:40.998751+0000","last_change":"2024-02-12T16:35:31.873084+0000","last_active":"2024-02-12T16:35:40.998751+0000","last_peered":"2024-02-12T16:35:40.998751+0000","last_clean":"2024-02-12T16:35:40.998751+0000","last_became_active":"2024-02-12T16:34:08.148290+0000","last_became_peered":"2024-02-12T16:34:08.148290+0000","last_unstale":"2024-02-12T16:35:40.998751+0000","last_undegraded":"2024-02-12T16:35:40.998751+0000","last_fullsized":"2024-02-12T16:35:40.998751+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'8980","last_scrub_stamp":"2024-02-12T16:35:31.872998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:31.872998+0000","log_size":9659,"ondisk_log_size":9659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":633012224,"num_objects":9659,"num_object_clones":0,"num_object_copies":19318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":618176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":892,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.945972+0000","last_change":"2024-02-12T16:35:06.330589+0000","last_active":"2024-02-12T16:35:29.945972+0000","last_peered":"2024-02-12T16:35:29.945972+0000","last_clean":"2024-02-12T16:35:29.945972+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:35:29.945972+0000","last_undegraded":"2024-02-12T16:35:29.945972+0000","last_fullsized":"2024-02-12T16:35:29.945972+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:06.330507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:06.330507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.1","version":"786'9510","reported_seq":9553,"reported_epoch":786,"state":"active+clean+scrubbing","last_fresh":"2024-02-12T16:35:39.551126+0000","last_change":"2024-02-12T16:35:37.527368+0000","last_active":"2024-02-12T16:35:39.551126+0000","last_peered":"2024-02-12T16:35:39.551126+0000","last_clean":"2024-02-12T16:35:39.551126+0000","last_became_active":"2024-02-12T16:34:08.149664+0000","last_became_peered":"2024-02-12T16:34:08.149664+0000","last_unstale":"2024-02-12T16:35:39.551126+0000","last_undegraded":"2024-02-12T16:35:39.551126+0000","last_fullsized":"2024-02-12T16:35:39.551126+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:34:07.024012+0000","log_size":9510,"ondisk_log_size":9510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":623247360,"num_objects":9510,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":608640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":548,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.947264+0000","last_change":"2024-02-12T16:35:23.788654+0000","last_active":"2024-02-12T16:35:29.947264+0000","last_peered":"2024-02-12T16:35:29.947264+0000","last_clean":"2024-02-12T16:35:29.947264+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:35:29.947264+0000","last_undegraded":"2024-02-12T16:35:29.947264+0000","last_fullsized":"2024-02-12T16:35:29.947264+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:23.788558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:23.788558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"787'9768","reported_seq":9815,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:42.405441+0000","last_change":"2024-02-12T16:35:22.520848+0000","last_active":"2024-02-12T16:35:42.405441+0000","last_peered":"2024-02-12T16:35:42.405441+0000","last_clean":"2024-02-12T16:35:42.405441+0000","last_became_active":"2024-02-12T16:34:08.041885+0000","last_became_peered":"2024-02-12T16:34:08.041885+0000","last_unstale":"2024-02-12T16:35:42.405441+0000","last_undegraded":"2024-02-12T16:35:42.405441+0000","last_fullsized":"2024-02-12T16:35:42.405441+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'8277","last_scrub_stamp":"2024-02-12T16:35:22.520759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:22.520759+0000","log_size":9768,"ondisk_log_size":9768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":640155648,"num_objects":9768,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9768,"num_write_kb":625152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":493,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.393893+0000","last_change":"2024-02-12T16:35:11.714927+0000","last_active":"2024-02-12T16:35:41.393893+0000","last_peered":"2024-02-12T16:35:41.393893+0000","last_clean":"2024-02-12T16:35:41.393893+0000","last_became_active":"2024-02-12T16:30:49.373979+0000","last_became_peered":"2024-02-12T16:30:49.373979+0000","last_unstale":"2024-02-12T16:35:41.393893+0000","last_undegraded":"2024-02-12T16:35:41.393893+0000","last_fullsized":"2024-02-12T16:35:41.393893+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:11.714823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:11.714823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.7","version":"786'9562","reported_seq":9608,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:40.450275+0000","last_change":"2024-02-12T16:35:32.830560+0000","last_active":"2024-02-12T16:35:40.450275+0000","last_peered":"2024-02-12T16:35:40.450275+0000","last_clean":"2024-02-12T16:35:40.450275+0000","last_became_active":"2024-02-12T16:34:08.150853+0000","last_became_peered":"2024-02-12T16:34:08.150853+0000","last_unstale":"2024-02-12T16:35:40.450275+0000","last_undegraded":"2024-02-12T16:35:40.450275+0000","last_fullsized":"2024-02-12T16:35:40.450275+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'8955","last_scrub_stamp":"2024-02-12T16:35:32.830417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:32.830417+0000","log_size":9562,"ondisk_log_size":9562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":626655232,"num_objects":9562,"num_object_clones":0,"num_object_copies":19124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":611968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":530,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.396906+0000","last_change":"2024-02-12T16:35:06.735527+0000","last_active":"2024-02-12T16:35:41.396906+0000","last_peered":"2024-02-12T16:35:41.396906+0000","last_clean":"2024-02-12T16:35:41.396906+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:35:41.396906+0000","last_undegraded":"2024-02-12T16:35:41.396906+0000","last_fullsized":"2024-02-12T16:35:41.396906+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:06.735411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:35:06.735411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.5","version":"786'9485","reported_seq":9531,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.854402+0000","last_change":"2024-02-12T16:35:17.683301+0000","last_active":"2024-02-12T16:35:39.854402+0000","last_peered":"2024-02-12T16:35:39.854402+0000","last_clean":"2024-02-12T16:35:39.854402+0000","last_became_active":"2024-02-12T16:34:08.041057+0000","last_became_peered":"2024-02-12T16:34:08.041057+0000","last_unstale":"2024-02-12T16:35:39.854402+0000","last_undegraded":"2024-02-12T16:35:39.854402+0000","last_fullsized":"2024-02-12T16:35:39.854402+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'7774","last_scrub_stamp":"2024-02-12T16:35:17.683194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:17.683194+0000","log_size":9485,"ondisk_log_size":9485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":621608960,"num_objects":9485,"num_object_clones":0,"num_object_copies":18970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":607040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":351,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:40.401022+0000","last_change":"2024-02-12T16:35:40.401022+0000","last_active":"2024-02-12T16:35:40.401022+0000","last_peered":"2024-02-12T16:35:40.401022+0000","last_clean":"2024-02-12T16:35:40.401022+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:35:40.401022+0000","last_undegraded":"2024-02-12T16:35:40.401022+0000","last_fullsized":"2024-02-12T16:35:40.401022+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:40.400930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:40.400930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"786'9693","reported_seq":9739,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.250735+0000","last_change":"2024-02-12T16:35:32.811366+0000","last_active":"2024-02-12T16:35:41.250735+0000","last_peered":"2024-02-12T16:35:41.250735+0000","last_clean":"2024-02-12T16:35:41.250735+0000","last_became_active":"2024-02-12T16:34:08.039442+0000","last_became_peered":"2024-02-12T16:34:08.039442+0000","last_unstale":"2024-02-12T16:35:41.250735+0000","last_undegraded":"2024-02-12T16:35:41.250735+0000","last_fullsized":"2024-02-12T16:35:41.250735+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'9039","last_scrub_stamp":"2024-02-12T16:35:32.811276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:32.811276+0000","log_size":9693,"ondisk_log_size":9693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":635240448,"num_objects":9693,"num_object_clones":0,"num_object_copies":19386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9693,"num_write_kb":620352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":920,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.399303+0000","last_change":"2024-02-12T16:35:21.488378+0000","last_active":"2024-02-12T16:35:41.399303+0000","last_peered":"2024-02-12T16:35:41.399303+0000","last_clean":"2024-02-12T16:35:41.399303+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:35:41.399303+0000","last_undegraded":"2024-02-12T16:35:41.399303+0000","last_fullsized":"2024-02-12T16:35:41.399303+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:21.488276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:21.488276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"787'9770","reported_seq":9815,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.975597+0000","last_change":"2024-02-12T16:35:41.411122+0000","last_active":"2024-02-12T16:35:41.975597+0000","last_peered":"2024-02-12T16:35:41.975597+0000","last_clean":"2024-02-12T16:35:41.975597+0000","last_became_active":"2024-02-12T16:34:08.036767+0000","last_became_peered":"2024-02-12T16:34:08.036767+0000","last_unstale":"2024-02-12T16:35:41.975597+0000","last_undegraded":"2024-02-12T16:35:41.975597+0000","last_fullsized":"2024-02-12T16:35:41.975597+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:34:07.024012+0000","log_size":9770,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":640286720,"num_objects":9770,"num_object_clones":0,"num_object_copies":19540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":625280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":308,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.399297+0000","last_change":"2024-02-12T16:35:26.539904+0000","last_active":"2024-02-12T16:35:41.399297+0000","last_peered":"2024-02-12T16:35:41.399297+0000","last_clean":"2024-02-12T16:35:41.399297+0000","last_became_active":"2024-02-12T16:30:49.374934+0000","last_became_peered":"2024-02-12T16:30:49.374934+0000","last_unstale":"2024-02-12T16:35:41.399297+0000","last_undegraded":"2024-02-12T16:35:41.399297+0000","last_fullsized":"2024-02-12T16:35:41.399297+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:26.539828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:26.539828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"787'9614","reported_seq":9661,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.396729+0000","last_change":"2024-02-12T16:35:27.977979+0000","last_active":"2024-02-12T16:35:41.396729+0000","last_peered":"2024-02-12T16:35:41.396729+0000","last_clean":"2024-02-12T16:35:41.396729+0000","last_became_active":"2024-02-12T16:34:08.041413+0000","last_became_peered":"2024-02-12T16:34:08.041413+0000","last_unstale":"2024-02-12T16:35:41.396729+0000","last_undegraded":"2024-02-12T16:35:41.396729+0000","last_fullsized":"2024-02-12T16:35:41.396729+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'8586","last_scrub_stamp":"2024-02-12T16:35:27.977872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:27.977872+0000","log_size":9614,"ondisk_log_size":9614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630063104,"num_objects":9614,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9614,"num_write_kb":615296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":331,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.953145+0000","last_change":"2024-02-12T16:35:19.522713+0000","last_active":"2024-02-12T16:35:29.953145+0000","last_peered":"2024-02-12T16:35:29.953145+0000","last_clean":"2024-02-12T16:35:29.953145+0000","last_became_active":"2024-02-12T16:30:49.382268+0000","last_became_peered":"2024-02-12T16:30:49.382268+0000","last_unstale":"2024-02-12T16:35:29.953145+0000","last_undegraded":"2024-02-12T16:35:29.953145+0000","last_fullsized":"2024-02-12T16:35:29.953145+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:19.522641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:19.522641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"786'9513","reported_seq":9558,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.297276+0000","last_change":"2024-02-12T16:35:29.795980+0000","last_active":"2024-02-12T16:35:41.297276+0000","last_peered":"2024-02-12T16:35:41.297276+0000","last_clean":"2024-02-12T16:35:41.297276+0000","last_became_active":"2024-02-12T16:34:08.040236+0000","last_became_peered":"2024-02-12T16:34:08.040236+0000","last_unstale":"2024-02-12T16:35:41.297276+0000","last_undegraded":"2024-02-12T16:35:41.297276+0000","last_fullsized":"2024-02-12T16:35:41.297276+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"784'8676","last_scrub_stamp":"2024-02-12T16:35:29.795895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:29.795895+0000","log_size":9513,"ondisk_log_size":9513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":623443968,"num_objects":9513,"num_object_clones":0,"num_object_copies":19026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9513,"num_write_kb":608832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":501,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.947437+0000","last_change":"2024-02-12T16:35:28.794885+0000","last_active":"2024-02-12T16:35:29.947437+0000","last_peered":"2024-02-12T16:35:29.947437+0000","last_clean":"2024-02-12T16:35:29.947437+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:35:29.947437+0000","last_undegraded":"2024-02-12T16:35:29.947437+0000","last_fullsized":"2024-02-12T16:35:29.947437+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:28.794774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:35:28.794774+0000","last_clean_scrub_stamp":"2024-02-12T16:35:28.794774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"787'9695","reported_seq":9742,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.829318+0000","last_change":"2024-02-12T16:35:32.405722+0000","last_active":"2024-02-12T16:35:41.829318+0000","last_peered":"2024-02-12T16:35:41.829318+0000","last_clean":"2024-02-12T16:35:41.829318+0000","last_became_active":"2024-02-12T16:34:08.147817+0000","last_became_peered":"2024-02-12T16:34:08.147817+0000","last_unstale":"2024-02-12T16:35:41.829318+0000","last_undegraded":"2024-02-12T16:35:41.829318+0000","last_fullsized":"2024-02-12T16:35:41.829318+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'9023","last_scrub_stamp":"2024-02-12T16:35:32.405631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:32.405631+0000","log_size":9695,"ondisk_log_size":9695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":635371520,"num_objects":9695,"num_object_clones":0,"num_object_copies":19390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":620480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":309,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.951695+0000","last_change":"2024-02-12T16:35:19.574228+0000","last_active":"2024-02-12T16:35:29.951695+0000","last_peered":"2024-02-12T16:35:29.951695+0000","last_clean":"2024-02-12T16:35:29.951695+0000","last_became_active":"2024-02-12T16:30:49.381004+0000","last_became_peered":"2024-02-12T16:30:49.381004+0000","last_unstale":"2024-02-12T16:35:29.951695+0000","last_undegraded":"2024-02-12T16:35:29.951695+0000","last_fullsized":"2024-02-12T16:35:29.951695+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:19.574110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:19.574110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"786'9662","reported_seq":9708,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.268716+0000","last_change":"2024-02-12T16:35:14.175766+0000","last_active":"2024-02-12T16:35:41.268716+0000","last_peered":"2024-02-12T16:35:41.268716+0000","last_clean":"2024-02-12T16:35:41.268716+0000","last_became_active":"2024-02-12T16:34:08.149769+0000","last_became_peered":"2024-02-12T16:34:08.149769+0000","last_unstale":"2024-02-12T16:35:41.268716+0000","last_undegraded":"2024-02-12T16:35:41.268716+0000","last_fullsized":"2024-02-12T16:35:41.268716+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'7499","last_scrub_stamp":"2024-02-12T16:35:14.175580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:14.175580+0000","log_size":9662,"ondisk_log_size":9662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":633208832,"num_objects":9662,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9662,"num_write_kb":618368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":903,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.399191+0000","last_change":"2024-02-12T16:35:11.592580+0000","last_active":"2024-02-12T16:35:41.399191+0000","last_peered":"2024-02-12T16:35:41.399191+0000","last_clean":"2024-02-12T16:35:41.399191+0000","last_became_active":"2024-02-12T16:30:49.404783+0000","last_became_peered":"2024-02-12T16:30:49.404783+0000","last_unstale":"2024-02-12T16:35:41.399191+0000","last_undegraded":"2024-02-12T16:35:41.399191+0000","last_fullsized":"2024-02-12T16:35:41.399191+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:11.592474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:11.592474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"787'9687","reported_seq":9734,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.833381+0000","last_change":"2024-02-12T16:35:11.387198+0000","last_active":"2024-02-12T16:35:41.833381+0000","last_peered":"2024-02-12T16:35:41.833381+0000","last_clean":"2024-02-12T16:35:41.833381+0000","last_became_active":"2024-02-12T16:34:08.149135+0000","last_became_peered":"2024-02-12T16:34:08.149135+0000","last_unstale":"2024-02-12T16:35:41.833381+0000","last_undegraded":"2024-02-12T16:35:41.833381+0000","last_fullsized":"2024-02-12T16:35:41.833381+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"777'7222","last_scrub_stamp":"2024-02-12T16:35:11.386929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:11.386929+0000","log_size":9687,"ondisk_log_size":9687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":634847232,"num_objects":9687,"num_object_clones":0,"num_object_copies":19374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":619968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":495,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.379253+0000","last_change":"2024-02-12T16:35:03.857816+0000","last_active":"2024-02-12T16:35:39.379253+0000","last_peered":"2024-02-12T16:35:39.379253+0000","last_clean":"2024-02-12T16:35:39.379253+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:35:39.379253+0000","last_undegraded":"2024-02-12T16:35:39.379253+0000","last_fullsized":"2024-02-12T16:35:39.379253+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:03.857737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:03.857737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"786'9554","reported_seq":9602,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:40.483564+0000","last_change":"2024-02-12T16:35:22.869146+0000","last_active":"2024-02-12T16:35:40.483564+0000","last_peered":"2024-02-12T16:35:40.483564+0000","last_clean":"2024-02-12T16:35:40.483564+0000","last_became_active":"2024-02-12T16:34:08.037484+0000","last_became_peered":"2024-02-12T16:34:08.037484+0000","last_unstale":"2024-02-12T16:35:40.483564+0000","last_undegraded":"2024-02-12T16:35:40.483564+0000","last_fullsized":"2024-02-12T16:35:40.483564+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'8268","last_scrub_stamp":"2024-02-12T16:35:22.869057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:22.869057+0000","log_size":9554,"ondisk_log_size":9554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":626130944,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9554,"num_write_kb":611456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":245,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.945984+0000","last_change":"2024-02-12T16:35:10.417915+0000","last_active":"2024-02-12T16:35:29.945984+0000","last_peered":"2024-02-12T16:35:29.945984+0000","last_clean":"2024-02-12T16:35:29.945984+0000","last_became_active":"2024-02-12T16:30:49.376219+0000","last_became_peered":"2024-02-12T16:30:49.376219+0000","last_unstale":"2024-02-12T16:35:29.945984+0000","last_undegraded":"2024-02-12T16:35:29.945984+0000","last_fullsized":"2024-02-12T16:35:29.945984+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:10.417816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:10.417816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"786'9565","reported_seq":9610,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:40.467418+0000","last_change":"2024-02-12T16:35:26.922675+0000","last_active":"2024-02-12T16:35:40.467418+0000","last_peered":"2024-02-12T16:35:40.467418+0000","last_clean":"2024-02-12T16:35:40.467418+0000","last_became_active":"2024-02-12T16:34:08.037558+0000","last_became_peered":"2024-02-12T16:34:08.037558+0000","last_unstale":"2024-02-12T16:35:40.467418+0000","last_undegraded":"2024-02-12T16:35:40.467418+0000","last_fullsized":"2024-02-12T16:35:40.467418+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'8571","last_scrub_stamp":"2024-02-12T16:35:26.922569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:26.922569+0000","log_size":9565,"ondisk_log_size":9565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":626851840,"num_objects":9565,"num_object_clones":0,"num_object_copies":19130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":612160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":487,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.392593+0000","last_change":"2024-02-12T16:35:03.750833+0000","last_active":"2024-02-12T16:35:39.392593+0000","last_peered":"2024-02-12T16:35:39.392593+0000","last_clean":"2024-02-12T16:35:39.392593+0000","last_became_active":"2024-02-12T16:30:49.384015+0000","last_became_peered":"2024-02-12T16:30:49.384015+0000","last_unstale":"2024-02-12T16:35:39.392593+0000","last_undegraded":"2024-02-12T16:35:39.392593+0000","last_fullsized":"2024-02-12T16:35:39.392593+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:03.750662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:35:03.750662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":312,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.379258+0000","last_change":"2024-02-12T16:35:23.904322+0000","last_active":"2024-02-12T16:35:39.379258+0000","last_peered":"2024-02-12T16:35:39.379258+0000","last_clean":"2024-02-12T16:35:39.379258+0000","last_became_active":"2024-02-12T16:34:16.403771+0000","last_became_peered":"2024-02-12T16:34:16.403771+0000","last_unstale":"2024-02-12T16:35:39.379258+0000","last_undegraded":"2024-02-12T16:35:39.379258+0000","last_fullsized":"2024-02-12T16:35:39.379258+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:23.904216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:35:23.904216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":293,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-12T16:35:29.943374+0000","last_change":"2024-02-12T16:35:09.018448+0000","last_active":"2024-02-12T16:35:29.943374+0000","last_peered":"2024-02-12T16:35:29.943374+0000","last_clean":"2024-02-12T16:35:29.943374+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:35:29.943374+0000","last_undegraded":"2024-02-12T16:35:29.943374+0000","last_fullsized":"2024-02-12T16:35:29.943374+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:09.018317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:09.018317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":333,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.379190+0000","last_change":"2024-02-12T16:35:16.899312+0000","last_active":"2024-02-12T16:35:39.379190+0000","last_peered":"2024-02-12T16:35:39.379190+0000","last_clean":"2024-02-12T16:35:39.379190+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:35:39.379190+0000","last_undegraded":"2024-02-12T16:35:39.379190+0000","last_fullsized":"2024-02-12T16:35:39.379190+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:07.846670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:35:07.846670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":317,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-12T16:35:39.391720+0000","last_change":"2024-02-12T16:35:08.806700+0000","last_active":"2024-02-12T16:35:39.391720+0000","last_peered":"2024-02-12T16:35:39.391720+0000","last_clean":"2024-02-12T16:35:39.391720+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:35:39.391720+0000","last_undegraded":"2024-02-12T16:35:39.391720+0000","last_fullsized":"2024-02-12T16:35:39.391720+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:08.806574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:08.806574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":10094116864,"num_objects":154024,"num_object_clones":0,"num_object_copies":308048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":154024,"num_write_kb":9857536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20291518464,"data_stored":20291518464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154024,"ondisk_log_size":154024,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958978,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2100024,"kb_used_data":1207280,"kb_used_omap":0,"kb_used_meta":892736,"kb_avail":91645128,"statfs":{"total":95995035648,"available":93844611072,"internally_reserved":0,"allocated":1236254720,"data_stored":1235526101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914161664},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.3200000000000001,"15min":1.02},"min":{"1min":0.249,"5min":0.249,"15min":0.246},"max":{"1min":2.4980000000000002,"5min":33.941000000000003,"15min":33.941000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.006,"5min":1.4550000000000001,"15min":1.137},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.28399999999999997},"max":{"1min":2.137,"5min":33.890000000000001,"15min":33.890000000000001},"last":4.3529999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.5459999999999998,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":4.1749999999999998},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.85899999999999999,"15min":0.91900000000000004},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.5389999999999999,"5min":11.167,"15min":11.167},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3759999999999999,"15min":1.1850000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.33200000000000002},"max":{"1min":4.9240000000000004,"5min":31.326000000000001,"15min":31.326000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.125,"5min":1.363,"15min":1.208},"min":{"1min":0.40300000000000002,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":4.9009999999999998,"5min":31.234000000000002,"15min":31.234000000000002},"last":1.3240000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.88600000000000001,"15min":0.98199999999999998},"min":{"1min":0.42399999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":1.4910000000000001,"5min":4.2770000000000001,"15min":4.9900000000000002},"last":3.8380000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":0.95199999999999996,"15min":0.94699999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.34499999999999997},"max":{"1min":2.3889999999999998,"5min":5.0730000000000004,"15min":5.0730000000000004},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.2290000000000001,"15min":1.2050000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.036,"5min":5.968,"15min":5.968},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.321,"5min":1.405,"15min":1.2390000000000001},"min":{"1min":0.40600000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.3239999999999998,"5min":6.7720000000000002,"15min":6.7720000000000002},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.248,"15min":1.0860000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.8109999999999999,"5min":6.4160000000000004,"15min":6.4160000000000004},"last":3.8940000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.4199999999999999,"15min":1.258},"min":{"1min":0.51100000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.9970000000000001,"5min":9.5389999999999997,"15min":9.5389999999999997},"last":3.9620000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.367,"15min":1.242},"min":{"1min":0.442,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.1190000000000002,"5min":7.851,"15min":7.851},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.411,"5min":1.3400000000000001,"15min":1.165},"min":{"1min":0.45200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.4500000000000002,"5min":8.3000000000000007,"15min":8.3000000000000007},"last":0.45800000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":0.77800000000000002,"15min":2.7269999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.1059999999999999,"5min":2.1389999999999998,"15min":537.54399999999998},"last":3.9929999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.96199999999999997,"15min":2.8980000000000001},"min":{"1min":0.48499999999999999,"5min":0.41099999999999998,"15min":0.40500000000000003},"max":{"1min":2.0910000000000002,"5min":4.9119999999999999,"15min":539.86099999999999},"last":0.70599999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.044,"15min":1.165},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.38600000000000001},"max":{"1min":2.194,"5min":4.9770000000000003,"15min":5.7430000000000003},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.0209999999999999,"15min":1.1819999999999999},"min":{"1min":0.54800000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.262,"5min":5.0129999999999999,"15min":5.359},"last":0.89900000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:34:41 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":0.96699999999999997,"15min":2.8999999999999999},"min":{"1min":0.47999999999999998,"5min":0.439,"15min":0.439},"max":{"1min":2.4990000000000001,"5min":2.8929999999999998,"15min":536.245},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.339,"5min":0.93500000000000005,"15min":2.8799999999999999},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.432},"max":{"1min":2.1869999999999998,"5min":3.1600000000000001,"15min":536.64300000000003},"last":3.9239999999999999}]}]},{"osd":10,"up_from":83,"seq":356482286028,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2176448,"kb_used_data":1875128,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":91568704,"statfs":{"total":95995035648,"available":93766352896,"internally_reserved":0,"allocated":1920131072,"data_stored":1919741974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64800000000000002,"15min":1.4199999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":2.8769999999999998,"5min":4.1769999999999996,"15min":162.38399999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.87,"15min":1.534},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.26500000000000001},"max":{"1min":3.1469999999999998,"5min":4.6180000000000003,"15min":162.01300000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.70999999999999996,"15min":0.875},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.0940000000000001,"5min":3.2850000000000001,"15min":5.9000000000000004},"last":0.438},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.93700000000000006,"15min":0.98299999999999998},"min":{"1min":0.34100000000000003,"5min":0.32800000000000001,"15min":0.309},"max":{"1min":3.3079999999999998,"5min":4.673,"15min":6.1970000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.85599999999999998,"15min":0.98699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.9700000000000002,"5min":5.016,"15min":8.3200000000000003},"last":1.073},{"interface":"front","average":{"1min":1.123,"5min":0.93600000000000005,"15min":1.034},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.5489999999999999,"5min":5.1820000000000004,"15min":5.7220000000000004},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:35:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.181,"15min":1.1859999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.6989999999999998,"5min":3.6989999999999998,"15min":3.6989999999999998},"last":1.028},{"interface":"front","average":{"1min":1.137,"5min":1.0049999999999999,"15min":0.98299999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.927,"5min":2.927,"15min":2.927},"last":0.374}]},{"osd":4,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.056,"15min":1.0289999999999999},"min":{"1min":0.38800000000000001,"5min":0.371,"15min":0.33500000000000002},"max":{"1min":1.4810000000000001,"5min":6.3689999999999998,"15min":6.3689999999999998},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.96899999999999997,"15min":1.018},"min":{"1min":0.318,"5min":0.318,"15min":0.30199999999999999},"max":{"1min":1.5700000000000001,"5min":4.3959999999999999,"15min":7.3070000000000004},"last":1.101}]},{"osd":5,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.998,"15min":1.0640000000000001},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":3.0190000000000001,"5min":5.0010000000000003,"15min":6.343},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.3440000000000001,"15min":1.252},"min":{"1min":0.47599999999999998,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":1.8700000000000001,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.496}]},{"osd":6,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.2430000000000001,"15min":1.129},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.35499999999999998},"max":{"1min":2.964,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.87,"5min":1.2150000000000001,"15min":1.1719999999999999},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.376},"max":{"1min":1.6359999999999999,"5min":4.4889999999999999,"15min":6.8200000000000003},"last":0.60099999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.1659999999999999,"15min":1.097},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.375},"max":{"1min":3.0739999999999998,"5min":7.2610000000000001,"15min":7.3280000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.139,"5min":1.2649999999999999,"15min":1.2170000000000001},"min":{"1min":0.59099999999999997,"5min":0.434,"15min":0.374},"max":{"1min":3.9460000000000002,"5min":5.3730000000000002,"15min":6.4539999999999997},"last":0.64000000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.071,"15min":1.125},"min":{"1min":0.47299999999999998,"5min":0.44900000000000001,"15min":0.40400000000000003},"max":{"1min":1.679,"5min":4.3399999999999999,"15min":6.5300000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":1.0580000000000001,"15min":1.095},"min":{"1min":0.378,"5min":0.376,"15min":0.35799999999999998},"max":{"1min":1.595,"5min":7.306,"15min":7.306},"last":1.284}]},{"osd":9,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.97999999999999998,"15min":1.1479999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.0489999999999999,"5min":3.0489999999999999,"15min":6.6239999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.054,"15min":1.127},"min":{"1min":0.44900000000000001,"5min":0.433,"15min":0.42899999999999999},"max":{"1min":3.3660000000000001,"5min":3.5049999999999999,"15min":7.3479999999999999},"last":0.86299999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:34:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.1200000000000001,"15min":1.2190000000000001},"min":{"1min":0.51500000000000001,"5min":0.40699999999999997,"15min":0.39400000000000002},"max":{"1min":2.2389999999999999,"5min":7.3929999999999998,"15min":7.3929999999999998},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.125,"15min":1.2250000000000001},"min":{"1min":0.58099999999999996,"5min":0.497,"15min":0.47899999999999998},"max":{"1min":1.962,"5min":4.3929999999999998,"15min":6.5860000000000003},"last":0.69999999999999996}]}]},{"osd":9,"up_from":74,"seq":317827580372,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2428764,"kb_used_data":1839060,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":91316388,"statfs":{"total":95995035648,"available":93507981312,"internally_reserved":0,"allocated":1883197440,"data_stored":1882638265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":3.1669999999999998,"5min":1.206,"15min":1.1890000000000001},"min":{"1min":0.313,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":51.289000000000001,"5min":51.289000000000001,"15min":51.289000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":3.1230000000000002,"5min":1.24,"15min":1.175},"min":{"1min":0.312,"5min":0.308,"15min":0.28299999999999997},"max":{"1min":51.448,"5min":51.448,"15min":51.448},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":0.80100000000000005,"15min":0.97299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.8819999999999997},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.4730000000000001,"5min":0.93100000000000005,"15min":0.96299999999999997},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.6349999999999998,"5min":4.6349999999999998,"15min":5.7910000000000004},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:35:07 2024","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":3.125,"15min":3.125},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":52.097000000000001,"5min":52.097000000000001,"15min":52.097000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":51.209000000000003,"5min":51.209000000000003,"15min":51.209000000000003},"last":0.64600000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:33:43 2024","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":8.8390000000000004,"15min":8.6829999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":197.661,"5min":197.661,"15min":197.661},"last":0.84499999999999997},{"interface":"front","average":{"1min":9.8330000000000002,"5min":10.432,"15min":10.532},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":197.31200000000001,"5min":197.31200000000001,"15min":197.31200000000001},"last":0.89300000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.84299999999999997,"15min":1.008},"min":{"1min":0.436,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.585,"5min":3.8900000000000001,"15min":6.3090000000000002},"last":0.434},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.94699999999999995,"15min":1.1399999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.36899999999999999},"max":{"1min":1.6619999999999999,"5min":3.9289999999999998,"15min":6.3490000000000002},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0660000000000001,"15min":1.198},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.7389999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.349,"5min":1.1299999999999999,"15min":1.1719999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":3.1989999999999998,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.1890000000000001,"15min":1.1160000000000001},"min":{"1min":0.44400000000000001,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":1.885,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":0.498},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.212,"15min":1.214},"min":{"1min":0.40000000000000002,"5min":0.379,"15min":0.36499999999999999},"max":{"1min":3.234,"5min":7.71,"15min":14.718},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.3109999999999999,"15min":1.286},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.7859999999999996,"5min":7.9489999999999998,"15min":7.9489999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4870000000000001,"15min":1.3089999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.43099999999999999},"max":{"1min":5.8979999999999997,"5min":5.8979999999999997,"15min":7.9160000000000004},"last":0.752}]},{"osd":8,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.143,"15min":1.1220000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34300000000000003},"max":{"1min":7.593,"5min":7.593,"15min":7.593},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.036,"15min":1.139},"min":{"1min":0.47599999999999998,"5min":0.45300000000000001,"15min":0.42599999999999999},"max":{"1min":1.736,"5min":3.6280000000000001,"15min":6.1980000000000004},"last":0.627}]},{"osd":10,"last update":"Mon Feb 12 16:34:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":0.98099999999999998,"15min":1.218},"min":{"1min":0.47999999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":5.8609999999999998,"5min":5.8609999999999998,"15min":5.8609999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0109999999999999,"15min":1.194},"min":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.39200000000000002},"max":{"1min":1.865,"5min":3.8380000000000001,"15min":6.2560000000000002},"last":0.67000000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:34:58 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.05,"15min":1.23},"min":{"1min":0.55800000000000005,"5min":0.499,"15min":0.47599999999999998},"max":{"1min":5.944,"5min":5.944,"15min":5.944},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.5,"5min":1.1619999999999999,"15min":1.2070000000000001},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.41599999999999998},"max":{"1min":5.5069999999999997,"5min":5.5069999999999997,"15min":5.5069999999999997},"last":0.96699999999999997}]}]},{"osd":8,"up_from":65,"seq":279172874718,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1555880,"kb_used_data":1254200,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":92189272,"statfs":{"total":95995035648,"available":94401814528,"internally_reserved":0,"allocated":1284300800,"data_stored":1283911702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":1.7490000000000001,"15min":1.1870000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.246,"5min":134.01499999999999,"15min":134.01499999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.891,"15min":1.3200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.3119999999999998,"5min":133.667,"15min":133.667},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0190000000000001,"15min":1.3320000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.497},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.9870000000000001,"15min":1.4079999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4580000000000002,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.70499999999999996,"15min":0.79800000000000004},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":4.0659999999999998,"5min":4.0659999999999998,"15min":4.4160000000000004},"last":3.4590000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.79100000000000004,"15min":0.92300000000000004},"min":{"1min":0.28999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.5510000000000002,"5min":3.3620000000000001,"15min":5.4260000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.70299999999999996,"15min":0.64300000000000002},"min":{"1min":0.45600000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.0289999999999999,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":3.2559999999999998},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.92600000000000005,"15min":0.82499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.2360000000000002,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":3.3540000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.753,"15min":0.876},"min":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.29599999999999999},"max":{"1min":1.3879999999999999,"5min":2.7040000000000002,"15min":5.0490000000000004},"last":0.38},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.86699999999999999,"15min":0.92600000000000005},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":2.6930000000000001,"5min":2.762,"15min":5.0380000000000003},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.93600000000000005,"15min":0.95999999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.3560000000000001,"5min":6.4160000000000004,"15min":6.4160000000000004},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.1419999999999999,"15min":1.0720000000000001},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.7229999999999999,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.39500000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.0840000000000001,"15min":1.0940000000000001},"min":{"1min":0.40600000000000003,"5min":0.38200000000000001,"15min":0.373},"max":{"1min":1.7649999999999999,"5min":4.6920000000000002,"15min":5.3639999999999999},"last":0.441},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.3009999999999999,"15min":1.208},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.38},"max":{"1min":3.2719999999999998,"5min":6.915,"15min":9.1110000000000007},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":7.9119999999999999,"15min":3.3999999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45400000000000001},"max":{"1min":1.581,"5min":706.67700000000002,"15min":706.67700000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":8.0139999999999993,"15min":3.5110000000000001},"min":{"1min":0.47899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.8820000000000001,"5min":707.49099999999999,"15min":707.49099999999999},"last":0.59499999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93300000000000005,"15min":1.075},"min":{"1min":0.371,"5min":0.371,"15min":0.34200000000000003},"max":{"1min":1.714,"5min":2.5699999999999998,"15min":3.8610000000000002},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0369999999999999,"15min":1.1890000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.44700000000000001},"max":{"1min":1.8180000000000001,"5min":2.6059999999999999,"15min":3.8940000000000001},"last":0.85199999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:34:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":0.91400000000000003,"15min":1.0409999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.36699999999999999},"max":{"1min":4.202,"5min":4.202,"15min":5.2560000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":0.85799999999999998,"15min":1.0369999999999999},"min":{"1min":0.48599999999999999,"5min":0.43099999999999999,"15min":0.38500000000000001},"max":{"1min":2.4430000000000001,"5min":2.46,"15min":5.0250000000000004},"last":0.626}]},{"osd":11,"last update":"Mon Feb 12 16:34:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":0.99399999999999999,"15min":1.1160000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":2.0630000000000002,"5min":2.306,"15min":5.2530000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.103,"5min":0.89300000000000002,"15min":1.034},"min":{"1min":0.53000000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6910000000000001,"5min":2.323,"15min":5.2320000000000002},"last":0.73999999999999999}]}]},{"osd":7,"up_from":56,"seq":240518169061,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1833984,"kb_used_data":1248376,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":91911168,"statfs":{"total":95995035648,"available":94117036032,"internally_reserved":0,"allocated":1278337024,"data_stored":1277947926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:34:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.419,"15min":1.853},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":2.4220000000000002,"5min":13.763999999999999,"15min":187.33699999999999},"last":3.2850000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.5029999999999999,"15min":1.867},"min":{"1min":0.34999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.4300000000000002,"5min":14.066000000000001,"15min":187.48400000000001},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.3360000000000001,"15min":1.4179999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.5789999999999997,"5min":14.106,"15min":14.106},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0389999999999999,"15min":1.111},"min":{"1min":0.32700000000000001,"5min":0.316,"15min":0.316},"max":{"1min":2.1429999999999998,"5min":5.3220000000000001,"15min":5.3220000000000001},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:32:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.046,"15min":1.143},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.9299999999999997,"5min":5.4340000000000002,"15min":35.655000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.3089999999999999,"15min":1.3360000000000001},"min":{"1min":0.378,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.0599999999999996,"5min":7.6040000000000001,"15min":35.164000000000001},"last":0.998}]},{"osd":3,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.3460000000000001,"15min":1.133},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26100000000000001},"max":{"1min":2.1989999999999998,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.3999999999999999,"15min":1.1419999999999999},"min":{"1min":0.30499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.7000000000000002,"5min":12.193,"15min":12.193},"last":1.73}]},{"osd":4,"last update":"Mon Feb 12 16:34:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.425,"15min":1.163},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.26200000000000001},"max":{"1min":1.6319999999999999,"5min":10.958,"15min":10.958},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.516,"15min":1.1739999999999999},"min":{"1min":0.312,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":1.6799999999999999,"5min":13.925000000000001,"15min":13.925000000000001},"last":0.626}]},{"osd":5,"last update":"Mon Feb 12 16:34:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":4.21,"15min":2.0790000000000002},"min":{"1min":0.36699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.5940000000000001,"5min":216.84299999999999,"15min":216.84299999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.876,"5min":4.3879999999999999,"15min":2.1850000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":1.7749999999999999,"5min":217.02600000000001,"15min":217.02600000000001},"last":0.501}]},{"osd":6,"last update":"Mon Feb 12 16:34:55 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.625,"15min":1.264},"min":{"1min":0.36299999999999999,"5min":0.31,"15min":0.31},"max":{"1min":2.2360000000000002,"5min":11.380000000000001,"15min":11.380000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.8140000000000001,"15min":1.395},"min":{"1min":0.51900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.0760000000000001,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.61899999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":1.248,"15min":2.9710000000000001},"min":{"1min":0.42599999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.98,"5min":8.3659999999999997,"15min":498.68299999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.5429999999999999,"15min":3.1920000000000002},"min":{"1min":0.52900000000000003,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":1.327,"5min":14.021000000000001,"15min":498.69999999999999},"last":0.52600000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.3129999999999999,"15min":1.1759999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.4289999999999998,"5min":8.4459999999999997,"15min":8.4459999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.997,"5min":1.353,"15min":1.3120000000000001},"min":{"1min":0.51400000000000001,"5min":0.41399999999999998,"15min":0.40100000000000002},"max":{"1min":3.3919999999999999,"5min":6.1379999999999999,"15min":7.1219999999999999},"last":0.68100000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:34:55 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.423,"15min":3.5449999999999999},"min":{"1min":0.42399999999999999,"5min":0.40899999999999997,"15min":0.376},"max":{"1min":2.1909999999999998,"5min":5.5179999999999998,"15min":497.52699999999999},"last":2.0219999999999998},{"interface":"front","average":{"1min":1.296,"5min":1.72,"15min":3.6259999999999999},"min":{"1min":0.60599999999999998,"5min":0.44900000000000001,"15min":0.39200000000000002},"max":{"1min":2.319,"5min":13.98,"15min":496.29599999999999},"last":0.59499999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.361,"15min":1.25},"min":{"1min":0.52200000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.0390000000000001,"5min":8.4199999999999999,"15min":8.4199999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.024,"5min":1.425,"15min":1.371},"min":{"1min":0.48299999999999998,"5min":0.47499999999999998,"15min":0.46899999999999997},"max":{"1min":1.915,"5min":5.5830000000000002,"15min":7.0679999999999996},"last":0.72299999999999998}]}]},{"osd":6,"up_from":48,"seq":206158430701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2420868,"kb_used_data":1830804,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":91324284,"statfs":{"total":95995035648,"available":93516066816,"internally_reserved":0,"allocated":1874743296,"data_stored":1874184121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":2.1539999999999999,"15min":1.4470000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7350000000000001,"5min":107.723,"15min":107.723},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":2.23,"15min":1.5349999999999999},"min":{"1min":0.38300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.1180000000000003,"5min":107.114,"15min":107.114},"last":0.92100000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.98399999999999999,"15min":1.0149999999999999},"min":{"1min":0.29199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.1099999999999999,"5min":5.6479999999999997,"15min":6.1870000000000003},"last":3.9900000000000002},{"interface":"front","average":{"1min":1.115,"5min":1.272,"15min":1.087},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":5.468,"5min":6.0910000000000002,"15min":6.6959999999999997},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.2969999999999999,"15min":1.252},"min":{"1min":0.36599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.859,"5min":7.8769999999999998,"15min":28.739999999999998},"last":4.0380000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.272,"15min":1.319},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.0249999999999999,"5min":8.5129999999999999,"15min":32.328000000000003},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:34:49 2024","interfaces":[{"interface":"back","average":{"1min":21.321000000000002,"5min":6.157,"15min":3.9540000000000002},"min":{"1min":0.40999999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":365.58999999999997,"5min":365.58999999999997,"15min":365.58999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":21.594000000000001,"5min":6.9329999999999998,"15min":4.9489999999999998},"min":{"1min":0.47899999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":366.93299999999999,"5min":366.93299999999999,"15min":366.93299999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.2509999999999999,"15min":1.1779999999999999},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":2.7930000000000001,"5min":8.4559999999999995,"15min":8.4559999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.1890000000000001,"15min":1.109},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.23999999999999999},"max":{"1min":2.6000000000000001,"5min":7.7610000000000001,"15min":7.7610000000000001},"last":5.6100000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.4179999999999999,"15min":1.2010000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.9180000000000001,"5min":8.5730000000000004,"15min":8.5730000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.877,"5min":1.347,"15min":1.1499999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":2.6379999999999999,"5min":7.835,"15min":10.52},"last":4.0970000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.5720000000000001,"15min":1.246},"min":{"1min":0.38100000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.9260000000000002,"5min":9.1199999999999992,"15min":9.1199999999999992},"last":4.0670000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.6160000000000001,"15min":1.2230000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.32200000000000001},"max":{"1min":1.9630000000000001,"5min":8.8879999999999999,"15min":8.8879999999999999},"last":0.501}]},{"osd":8,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":6.1219999999999999,"15min":2.827},"min":{"1min":0.47499999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.863,"5min":501.892,"15min":501.892},"last":4.1369999999999996},{"interface":"front","average":{"1min":0.97199999999999998,"5min":6.1479999999999997,"15min":2.8220000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.371},"max":{"1min":3.0030000000000001,"5min":501.76600000000002,"15min":501.76600000000002},"last":2.5579999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:35:00 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.3169999999999999,"15min":1.214},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8160000000000001,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":4.2409999999999997},{"interface":"front","average":{"1min":1.698,"5min":1.5229999999999999,"15min":1.2889999999999999},"min":{"1min":0.49399999999999999,"5min":0.46100000000000002,"15min":0.42799999999999999},"max":{"1min":4.327,"5min":7.6870000000000003,"15min":7.6870000000000003},"last":0.54300000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:35:00 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.385},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":4.0839999999999996,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.577,"5min":1.653,"15min":1.4690000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":4.1280000000000001,"5min":8.7100000000000009,"15min":8.7100000000000009},"last":0.85399999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:35:00 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.752,"15min":1.4419999999999999},"min":{"1min":0.51200000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.375,"5min":8.6649999999999991,"15min":8.6649999999999991},"last":4.2000000000000002},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.252,"15min":1.226},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.0579999999999998,"5min":6.25,"15min":6.25},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575379,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2146560,"kb_used_data":1244152,"kb_used_omap":0,"kb_used_meta":902400,"kb_avail":91598592,"statfs":{"total":95995035648,"available":93796958208,"internally_reserved":0,"allocated":1274011648,"data_stored":1273622550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":924057600},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":0.93200000000000005,"15min":0.97399999999999998},"min":{"1min":0.33500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":10.420999999999999,"5min":10.420999999999999,"15min":16.632999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.256,"5min":1.1830000000000001,"15min":1.2450000000000001},"min":{"1min":0.30199999999999999,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":10.26,"5min":10.26,"15min":10.26},"last":4.8380000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.1950000000000001,"15min":1.147},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.755000000000001,"5min":10.755000000000001,"15min":12.778},"last":8.5199999999999996},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.1919999999999999,"15min":1.248},"min":{"1min":0.308,"5min":0.249,"15min":0.246},"max":{"1min":7.3600000000000003,"5min":9.702,"15min":9.9689999999999994},"last":0.80500000000000005}]},{"osd":3,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.3280000000000001,"15min":1.1890000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.248},"max":{"1min":3.2610000000000001,"5min":26.664000000000001,"15min":26.664000000000001},"last":0.75},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.5820000000000001,"15min":1.25},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.3260000000000001,"5min":27.148,"15min":27.148},"last":0.84599999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":1.034,"15min":1.1599999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":2.516,"5min":7.0720000000000001,"15min":8.7479999999999993},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.72699999999999998,"5min":1.1319999999999999,"15min":1.2729999999999999},"min":{"1min":0.35499999999999998,"5min":0.312,"15min":0.30199999999999999},"max":{"1min":2.5350000000000001,"5min":6.9119999999999999,"15min":7.1710000000000003},"last":0.89000000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":6.8010000000000002,"15min":4.1120000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.165,"5min":524.85299999999995,"15min":524.85299999999995},"last":8.4299999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":6.6509999999999998,"15min":4.0499999999999998},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":2.468,"5min":519.89700000000005,"15min":519.89700000000005},"last":0.83299999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":6.9219999999999997,"15min":3.254},"min":{"1min":0.39200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.371,"5min":522.79200000000003,"15min":522.79200000000003},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.173,"5min":6.7190000000000003,"15min":3.2010000000000001},"min":{"1min":0.44500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.3940000000000001,"5min":520.64499999999998,"15min":520.64499999999998},"last":0.79500000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.1439999999999999,"15min":3.355},"min":{"1min":0.52500000000000002,"5min":0.41699999999999998,"15min":0.39500000000000002},"max":{"1min":3.593,"5min":7.0309999999999997,"15min":548.91600000000005},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.446,"5min":1.1990000000000001,"15min":3.3700000000000001},"min":{"1min":0.57499999999999996,"5min":0.42499999999999999,"15min":0.36599999999999999},"max":{"1min":3.5550000000000002,"5min":7.0730000000000004,"15min":552.42100000000005},"last":0.96699999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:35:07 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.131,"15min":1.3140000000000001},"min":{"1min":0.44700000000000001,"5min":0.42599999999999999,"15min":0.42099999999999999},"max":{"1min":4.1210000000000004,"5min":7.444,"15min":7.444},"last":8.3949999999999996},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2210000000000001,"15min":1.339},"min":{"1min":0.53600000000000003,"5min":0.49099999999999999,"15min":0.436},"max":{"1min":4.2960000000000003,"5min":7.4870000000000001,"15min":10.106},"last":8.4969999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1930000000000001,"15min":3.7080000000000002},"min":{"1min":0.49399999999999999,"5min":0.46200000000000002,"15min":0.39500000000000002},"max":{"1min":4.258,"5min":7.3620000000000001,"15min":548.93499999999995},"last":8.3469999999999995},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.1679999999999999,"15min":3.7029999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.46999999999999997},"max":{"1min":3.504,"5min":7.1260000000000003,"15min":549.12099999999998},"last":0.86299999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.2010000000000001,"15min":1.3899999999999999},"min":{"1min":0.54400000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.5800000000000001,"5min":7.1669999999999998,"15min":10.202},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2070000000000001,"15min":1.411},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.44900000000000001},"max":{"1min":4.2140000000000004,"5min":7.4029999999999996,"15min":7.4029999999999996},"last":8.4649999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738911,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3351388,"kb_used_data":2456276,"kb_used_omap":0,"kb_used_meta":895104,"kb_avail":90393764,"statfs":{"total":95995035648,"available":92563214336,"internally_reserved":0,"allocated":2515226624,"data_stored":2514667449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":916586496},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.83899999999999997,"15min":0.91900000000000004},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23699999999999999},"max":{"1min":1.7170000000000001,"5min":6.0030000000000001,"15min":10.754},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.0029999999999999,"15min":1.0820000000000001},"min":{"1min":0.29899999999999999,"5min":0.23400000000000001,"15min":0.22600000000000001},"max":{"1min":4.1849999999999996,"5min":4.7939999999999996,"15min":5.5810000000000004},"last":4.1769999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:34:56 2024","interfaces":[{"interface":"back","average":{"1min":18.202000000000002,"5min":4.8760000000000003,"15min":2.3170000000000002},"min":{"1min":0.28399999999999997,"5min":0.254,"15min":0.254},"max":{"1min":377.06299999999999,"5min":377.06299999999999,"15min":377.06299999999999},"last":0.435},{"interface":"front","average":{"1min":18.343,"5min":4.8310000000000004,"15min":2.3039999999999998},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.224},"max":{"1min":379.18000000000001,"5min":379.18000000000001,"15min":379.18000000000001},"last":4.024}]},{"osd":4,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":1.1339999999999999,"15min":1.9239999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.6890000000000001,"5min":5.6859999999999999,"15min":193.77000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.97699999999999998,"15min":1.8879999999999999},"min":{"1min":0.34399999999999997,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":4.3230000000000004,"5min":5.0659999999999998,"15min":189.625},"last":4.0780000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:35:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":3.1499999999999999,"15min":1.8120000000000001},"min":{"1min":0.311,"5min":0.30399999999999999,"15min":0.28699999999999998},"max":{"1min":4.5670000000000002,"5min":181.46199999999999,"15min":181.46199999999999},"last":3.2370000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":3.218,"15min":1.8779999999999999},"min":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.718,"5min":186.41999999999999,"15min":186.41999999999999},"last":3.5099999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3819999999999999,"15min":1.9330000000000001},"min":{"1min":0.379,"5min":0.371,"15min":0.311},"max":{"1min":4.5919999999999996,"5min":7.8220000000000001,"15min":189.79400000000001},"last":3.2909999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.427,"15min":1.9770000000000001},"min":{"1min":0.45100000000000001,"5min":0.38,"15min":0.35499999999999998},"max":{"1min":4.6280000000000001,"5min":5.907,"15min":194.41200000000001},"last":3.3380000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":3.2839999999999998,"15min":2.73},"min":{"1min":0.44500000000000001,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":4.7850000000000001,"5min":184.39400000000001,"15min":191.47200000000001},"last":3.403},{"interface":"front","average":{"1min":1.458,"5min":3.3730000000000002,"15min":2.8300000000000001},"min":{"1min":0.49199999999999999,"5min":0.432,"15min":0.432},"max":{"1min":4.7359999999999998,"5min":184.43700000000001,"15min":191.50299999999999},"last":3.4340000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:35:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.29,"15min":1.3220000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.7619999999999996,"5min":5.9379999999999997,"15min":7.3920000000000003},"last":3.3759999999999999},{"interface":"front","average":{"1min":1.621,"5min":1.373,"15min":1.4179999999999999},"min":{"1min":0.51200000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":4.8079999999999998,"5min":6.0209999999999999,"15min":7.5060000000000002},"last":3.4710000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.1279999999999999,"15min":1.1819999999999999},"min":{"1min":0.47199999999999998,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":8.4629999999999992,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":4.1139999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.1950000000000001,"15min":1.3520000000000001},"min":{"1min":0.54000000000000004,"5min":0.45300000000000001,"15min":0.432},"max":{"1min":4.6699999999999999,"5min":5.9640000000000004,"15min":7.5369999999999999},"last":8.7590000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.113,"15min":1.3109999999999999},"min":{"1min":0.53000000000000003,"5min":0.46100000000000002,"15min":0.442},"max":{"1min":2.0430000000000001,"5min":4.774,"15min":6.3559999999999999},"last":0.748},{"interface":"front","average":{"1min":1.147,"5min":1.222,"15min":1.421},"min":{"1min":0.59999999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":2.069,"5min":8.4280000000000008,"15min":9.3490000000000002},"last":0.84999999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.45,"15min":1.4179999999999999},"min":{"1min":0.51600000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":8.4280000000000008,"5min":8.4280000000000008,"15min":8.4280000000000008},"last":4.1449999999999996},{"interface":"front","average":{"1min":1.093,"5min":1.2030000000000001,"15min":1.3600000000000001},"min":{"1min":0.52000000000000002,"5min":0.47299999999999998,"15min":0.45200000000000001},"max":{"1min":4.0129999999999999,"5min":5.9930000000000003,"15min":7.4210000000000003},"last":0.78900000000000003}]}]},{"osd":2,"up_from":21,"seq":90194313741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2772944,"kb_used_data":2475000,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":90972208,"statfs":{"total":95995035648,"available":93155540992,"internally_reserved":0,"allocated":2534400000,"data_stored":2534010902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.69699999999999995,"15min":0.98099999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":1.704,"5min":4.7990000000000004,"15min":6.9870000000000001},"last":6.7880000000000003},{"interface":"front","average":{"1min":1.01,"5min":0.80900000000000005,"15min":1.0940000000000001},"min":{"1min":0.42699999999999999,"5min":0.255,"15min":0.247},"max":{"1min":2.1970000000000001,"5min":6.7640000000000002,"15min":9.6470000000000002},"last":3.8500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:35:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.6499999999999999,"15min":1.6779999999999999},"min":{"1min":0.31,"5min":0.28399999999999997,"15min":0.253},"max":{"1min":7.1040000000000001,"5min":54.091000000000001,"15min":54.091000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.651,"15min":1.2729999999999999},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":5.3399999999999999,"5min":54.203000000000003,"15min":54.203000000000003},"last":6.7670000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":0.89600000000000002,"15min":1.089},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.28799999999999998},"max":{"1min":7.194,"5min":7.194,"15min":10.890000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.119,"5min":0.98999999999999999,"15min":1.2010000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":3.7189999999999999,"5min":7.0019999999999998,"15min":9.8079999999999998},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:34:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.85199999999999998,"15min":1.1240000000000001},"min":{"1min":0.442,"5min":0.311,"15min":0.311},"max":{"1min":1.3879999999999999,"5min":3.9670000000000001,"15min":6.133},"last":6.7229999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.026,"15min":1.25},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":2.754,"5min":7.0590000000000002,"15min":11.081},"last":0.50800000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.198,"15min":1.3340000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":7.157,"5min":7.157,"15min":21.527000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.1890000000000001,"15min":1.2509999999999999},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.35099999999999998},"max":{"1min":3.9100000000000001,"5min":7.931,"15min":7.931},"last":7.0609999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.115,"15min":1.28},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":3.774,"5min":7.1189999999999998,"15min":11.044},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.196,"5min":1.0629999999999999,"15min":1.2529999999999999},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":3.8759999999999999,"5min":4.867,"15min":6.2050000000000001},"last":7.0069999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:35:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":0.96799999999999997,"15min":1.234},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.4950000000000001,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.1499999999999999,"5min":0.89900000000000002,"15min":1.2350000000000001},"min":{"1min":0.498,"5min":0.377,"15min":0.377},"max":{"1min":3.9390000000000001,"5min":3.9390000000000001,"15min":9.7639999999999993},"last":7.1029999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.97799999999999998,"15min":1.2769999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.8170000000000002,"5min":7.1689999999999996,"15min":10.972},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.044,"15min":1.3720000000000001},"min":{"1min":0.53900000000000003,"5min":0.45100000000000001,"15min":0.442},"max":{"1min":3.847,"5min":7.2110000000000003,"15min":11.006},"last":0.70199999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:34:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":0.90600000000000003,"15min":1.351},"min":{"1min":0.54900000000000004,"5min":0.46300000000000002,"15min":0.41399999999999998},"max":{"1min":3.2370000000000001,"5min":6.2990000000000004,"15min":11.454000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.5089999999999999,"5min":0.96199999999999997,"15min":1.397},"min":{"1min":0.65000000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":3.202,"5min":4.4299999999999997,"15min":9.7170000000000005},"last":0.85699999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.93899999999999995,"15min":1.256},"min":{"1min":0.52200000000000002,"5min":0.52100000000000002,"15min":0.47799999999999998},"max":{"1min":3.673,"5min":3.8119999999999998,"15min":7.258},"last":7.1369999999999996},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.0209999999999999,"15min":1.345},"min":{"1min":0.54000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.6459999999999999,"5min":5.1769999999999996,"15min":7.3040000000000003},"last":7.1710000000000003}]}]},{"osd":3,"up_from":522,"seq":2241972928665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2186820,"kb_used_data":1841940,"kb_used_omap":91,"kb_used_meta":344420,"kb_avail":91558332,"statfs":{"total":95995035648,"available":93755731968,"internally_reserved":0,"allocated":1886146560,"data_stored":1885587385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":352686096},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.052,"15min":1.0740000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":34.055999999999997},"last":3.3860000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.0209999999999999,"15min":1.103},"min":{"1min":0.29499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.1559999999999997,"5min":10.436999999999999,"15min":27.600000000000001},"last":3.8199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":0.85299999999999998,"15min":0.98399999999999999},"min":{"1min":0.28399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":8.8109999999999999},"last":3.1190000000000002},{"interface":"front","average":{"1min":1.151,"5min":0.85999999999999999,"15min":1.026},"min":{"1min":0.27800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":3.2909999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.0600000000000001,"15min":1.0700000000000001},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.2380000000000004,"5min":5.2809999999999997,"15min":10.004},"last":3.3570000000000002},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.1200000000000001,"15min":1.1080000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.2650000000000001,"5min":6.1829999999999998,"15min":6.806},"last":3.5659999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.2170000000000001,"15min":1.196},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":11.884,"5min":11.884,"15min":11.884},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.367,"15min":1.107},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":12.164999999999999,"5min":12.164999999999999,"15min":12.164999999999999},"last":3.1640000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2669999999999999,"15min":1.167},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.2359999999999998,"5min":5.7309999999999999,"15min":8.3320000000000007},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.2609999999999999,"15min":1.1180000000000001},"min":{"1min":0.40200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.4100000000000001,"5min":6.8890000000000002,"15min":6.8890000000000002},"last":3.2629999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.2749999999999999,"15min":1.252},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.3419999999999996,"5min":6.7409999999999997,"15min":6.851},"last":3.2290000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.4199999999999999,"15min":1.3049999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.3039999999999998,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":3.415}]},{"osd":8,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.131,"15min":1.133},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.3730000000000002,"5min":5.3220000000000001,"15min":5.3220000000000001},"last":3.3279999999999998},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.204,"15min":1.125},"min":{"1min":0.44400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.3620000000000001,"5min":6.2610000000000001,"15min":6.2610000000000001},"last":3.4470000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2,"15min":1.2789999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":3.419,"5min":6.218,"15min":6.218},"last":3.5350000000000001},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.169,"15min":1.278},"min":{"1min":0.48799999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":4.2649999999999997,"5min":4.7320000000000002,"15min":5.0780000000000003},"last":0.51400000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2130000000000001,"15min":1.294},"min":{"1min":0.53000000000000003,"5min":0.47099999999999997,"15min":0.46000000000000002},"max":{"1min":4.2990000000000004,"5min":4.7720000000000002,"15min":4.8499999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2410000000000001,"15min":1.343},"min":{"1min":0.495,"5min":0.49199999999999999,"15min":0.47099999999999997},"max":{"1min":3.4319999999999999,"5min":6.2939999999999996,"15min":15.140000000000001},"last":3.4780000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":4.7030000000000003,"5min":1.9470000000000001,"15min":1.512},"min":{"1min":0.45700000000000002,"5min":0.437,"15min":0.437},"max":{"1min":73.037999999999997,"5min":73.037999999999997,"15min":73.037999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":4.6120000000000001,"5min":1.895,"15min":1.5620000000000001},"min":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.49299999999999999},"max":{"1min":73.057000000000002,"5min":73.057000000000002,"15min":73.057000000000002},"last":3.5070000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953980,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1821700,"kb_used_data":1223444,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":91923452,"statfs":{"total":95995035648,"available":94129614848,"internally_reserved":0,"allocated":1252806656,"data_stored":1252247481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0350000000000001,"5min":1.6599999999999999,"15min":1.246},"min":{"1min":0.27300000000000002,"5min":0.248,"15min":0.23699999999999999},"max":{"1min":56.188000000000002,"5min":56.188000000000002,"15min":56.188000000000002},"last":3.492},{"interface":"front","average":{"1min":4.1529999999999996,"5min":1.704,"15min":1.3380000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":55.414000000000001,"5min":55.414000000000001,"15min":55.414000000000001},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.101,"15min":1.0580000000000001},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.24399999999999999},"max":{"1min":4.0090000000000003,"5min":5.3920000000000003,"15min":7.3630000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":1.129,"15min":1.1240000000000001},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":1.8859999999999999,"5min":5.2069999999999999,"15min":7.633},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.448,"15min":1.228},"min":{"1min":0.40200000000000002,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":3.6110000000000002,"5min":8.5800000000000001,"15min":12.409000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.3620000000000001,"15min":1.1599999999999999},"min":{"1min":0.35099999999999998,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":2.9550000000000001,"5min":6.5860000000000003,"15min":7.383},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":1.3999999999999999,"15min":1.1499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":2.004,"5min":10.289999999999999,"15min":10.289999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.5069999999999999,"15min":1.3049999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.04,"5min":10.321999999999999,"15min":10.321999999999999},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.2649999999999999,"15min":1.083},"min":{"1min":0.32400000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":1.8600000000000001,"5min":6.5270000000000001,"15min":7.5289999999999999},"last":0.5},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.2629999999999999,"15min":1.173},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":2.871,"5min":7.0910000000000002,"15min":9.6639999999999997},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.1499999999999999,"15min":1.0860000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.5139999999999998,"5min":5.8099999999999996,"15min":6.8659999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.7849999999999999,"15min":1.302},"min":{"1min":0.38700000000000001,"5min":0.35599999999999998,"15min":0.34599999999999997},"max":{"1min":1.7270000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.435}]},{"osd":8,"last update":"Mon Feb 12 16:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.2,"15min":1.1950000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.38300000000000001},"max":{"1min":1.4350000000000001,"5min":7.5179999999999998,"15min":7.6890000000000001},"last":3.5619999999999998},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.224,"15min":1.256},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.2749999999999999,"5min":5.5259999999999998,"15min":7.7110000000000003},"last":0.78400000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.3460000000000001,"15min":1.26},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.0190000000000001,"5min":7.4470000000000001,"15min":7.7750000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.01,"5min":1.294,"15min":1.3580000000000001},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.36699999999999999},"max":{"1min":1.9259999999999999,"5min":5.6470000000000002,"15min":7.827},"last":0.753}]},{"osd":10,"last update":"Mon Feb 12 16:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.3,"15min":1.383},"min":{"1min":0.46899999999999997,"5min":0.44,"15min":0.39200000000000002},"max":{"1min":1.6459999999999999,"5min":5.6109999999999998,"15min":7.7619999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1499999999999999,"15min":1.1819999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":2.9100000000000001,"5min":5.6459999999999999,"15min":5.6459999999999999},"last":0.66500000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.375,"15min":1.3069999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.45400000000000001},"max":{"1min":1.4930000000000001,"5min":7.4880000000000004,"15min":7.734},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":1.339,"15min":1.391},"min":{"1min":0.51400000000000001,"5min":0.48899999999999999,"15min":0.46400000000000002},"max":{"1min":1.589,"5min":5.681,"15min":7.7969999999999997},"last":0.59299999999999997}]}]},{"osd":5,"up_from":39,"seq":167503725044,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1835560,"kb_used_data":1249592,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":91909592,"statfs":{"total":95995035648,"available":94115422208,"internally_reserved":0,"allocated":1279582208,"data_stored":1279193110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:34:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":11.859999999999999,"5min":11.859999999999999,"15min":11.859999999999999},"last":0.443},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":11.166,"5min":11.166,"15min":11.166},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.121,"15min":1.7430000000000001},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.245},"max":{"1min":1.4910000000000001,"5min":6.6870000000000003,"15min":221.96600000000001},"last":0.747},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.278,"15min":1.867},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.311},"max":{"1min":1.635,"5min":6.774,"15min":222.06200000000001},"last":0.80500000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.2350000000000001,"15min":1.246},"min":{"1min":0.33800000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.9670000000000001,"5min":6.6459999999999999,"15min":40.021999999999998},"last":0.626},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.2270000000000001,"15min":1.2629999999999999},"min":{"1min":0.38200000000000001,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":4.375,"5min":11.448,"15min":40.225000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:35:41 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.5389999999999999,"15min":1.4570000000000001},"min":{"1min":0.35999999999999999,"5min":0.33900000000000002,"15min":0.27200000000000002},"max":{"1min":3.9809999999999999,"5min":19.067,"15min":22.484000000000002},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.427,"15min":1.141},"min":{"1min":0.378,"5min":0.308,"15min":0.29299999999999998},"max":{"1min":2.4660000000000002,"5min":18.216000000000001,"15min":23.34},"last":0.77700000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.2889999999999999,"15min":1.2350000000000001},"min":{"1min":0.34100000000000003,"5min":0.28299999999999997,"15min":0.23200000000000001},"max":{"1min":1.744,"5min":6.5999999999999996,"15min":41.908000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":1.2190000000000001,"15min":1.3089999999999999},"min":{"1min":0.34799999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.5129999999999999,"5min":5.2599999999999998,"15min":42.430999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":1.1459999999999999,"15min":1.103},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.7569999999999999,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":0.495},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.3939999999999999,"15min":1.23},"min":{"1min":0.40500000000000003,"5min":0.33100000000000002,"15min":0.27100000000000002},"max":{"1min":3.29,"5min":5.3810000000000002,"15min":9.5830000000000002},"last":0.66300000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.52,"15min":1.238},"min":{"1min":0.40999999999999998,"5min":0.39100000000000001,"15min":0.28799999999999998},"max":{"1min":1.6639999999999999,"5min":9.4179999999999993,"15min":9.7710000000000008},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.002,"5min":1.464,"15min":1.2909999999999999},"min":{"1min":0.39800000000000002,"5min":0.378,"15min":0.34300000000000003},"max":{"1min":1.9550000000000001,"5min":6.7599999999999998,"15min":6.7599999999999998},"last":0.72599999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:32:45 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":2.3969999999999998,"15min":1.6140000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":6.7300000000000004,"5min":108.913,"15min":108.913},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.169,"5min":2.1309999999999998,"15min":1.577},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1550000000000002,"5min":108.93600000000001,"15min":108.93600000000001},"last":1.8009999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":1.266,"15min":1.347},"min":{"1min":0.44,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.0819999999999999,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.448,"15min":1.345},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.375},"max":{"1min":2.5099999999999998,"5min":7.9710000000000001,"15min":9.6649999999999991},"last":0.86799999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:34:49 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.49399999999999999,"5min":0.48599999999999999,"15min":0.41099999999999998},"max":{"1min":2.1880000000000002,"5min":8.0299999999999994,"15min":55.518999999999998},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.147,"5min":1.3700000000000001,"15min":1.609},"min":{"1min":0.54700000000000004,"5min":0.48699999999999999,"15min":0.47999999999999998},"max":{"1min":1.835,"5min":6.867,"15min":55.497999999999998},"last":0.95399999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.5409999999999999,"15min":1.421},"min":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.38300000000000001},"max":{"1min":2.2690000000000001,"5min":8.0640000000000001,"15min":8.6940000000000008},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.492,"15min":1.4670000000000001},"min":{"1min":0.59699999999999998,"5min":0.53100000000000003,"15min":0.51500000000000001},"max":{"1min":2.0569999999999999,"5min":6.9029999999999996,"15min":6.9029999999999996},"last":0.91000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2522284032,"data_stored":2522284032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1274806272,"data_stored":1274806272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2539716608,"data_stored":2539716608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1896677376,"data_stored":1896677376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1251999744,"data_stored":1251999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1282408448,"data_stored":1282408448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1879638016,"data_stored":1879638016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1287585792,"data_stored":1287585792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1291780096,"data_stored":1291780096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1894776832,"data_stored":1894776832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1930100736,"data_stored":1930100736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1239744512,"data_stored":1239744512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:35:46.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:35:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:46.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:46.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:46.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:35:46.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:35:46.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:46.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:46.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:46.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:46.455 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:46 smithi123 bash[19119]: audit 2024-02-12T16:35:45.058483+0000 mgr.y (mgr.14146) 1534 : audit [DBG] from='client.26645 v1:172.21.15.121:0/2160119439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:35:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:46.477 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:46.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:46.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:46.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:46 smithi169 bash[19604]: audit 2024-02-12T16:35:45.058483+0000 mgr.y (mgr.14146) 1534 : audit [DBG] from='client.26645 v1:172.21.15.121:0/2160119439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:35:46.604 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:46.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:46.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:46 smithi121 bash[18489]: audit 2024-02-12T16:35:45.058483+0000 mgr.y (mgr.14146) 1534 : audit [DBG] from='client.26645 v1:172.21.15.121:0/2160119439' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:35:46.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:46.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:46.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:46.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:46.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:46.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:46.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:47.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:47.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:47.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:47.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 161327 161311 120.01 71.1875 0.0148542 0.0083312 2024-02-12T16:35:47.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:47.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:47.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:47.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:47.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:47.543 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:47 smithi169 bash[19604]: cluster 2024-02-12T16:35:46.127981+0000 mgr.y (mgr.14146) 1535 : cluster [DBG] pgmap v2202: 47 pgs: 47 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-12T16:35:47.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:47.670 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:47.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:47.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:47 smithi123 bash[19119]: cluster 2024-02-12T16:35:46.127981+0000 mgr.y (mgr.14146) 1535 : cluster [DBG] pgmap v2202: 47 pgs: 47 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-12T16:35:47.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:47 smithi121 bash[18489]: cluster 2024-02-12T16:35:46.127981+0000 mgr.y (mgr.14146) 1535 : cluster [DBG] pgmap v2202: 47 pgs: 47 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-02-12T16:35:47.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:47.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:47.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:48.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:48.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:48.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:48.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 162537 162521 119.488 75.625 0.00658673 0.00836772 2024-02-12T16:35:48.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:48.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:48.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:48.359 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:48.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:48.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:48.489 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:48.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:48.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:48.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:48.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:48.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:48.766 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:48.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:48.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:48.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:49.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:49.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:49.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:49.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:49.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:49.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 163746 163730 118.977 75.5625 0.00845474 0.00840355 2024-02-12T16:35:49.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:49.318 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:49.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:49.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:49.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:49.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:49.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:49.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:49.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:49 smithi169 bash[19604]: cluster 2024-02-12T16:35:48.128410+0000 mgr.y (mgr.14146) 1536 : cluster [DBG] pgmap v2203: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-02-12T16:35:49.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:49.710 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:49 smithi123 bash[19119]: cluster 2024-02-12T16:35:48.128410+0000 mgr.y (mgr.14146) 1536 : cluster [DBG] pgmap v2203: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-02-12T16:35:49.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:49.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:49.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:49.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:49 smithi121 bash[18489]: cluster 2024-02-12T16:35:48.128410+0000 mgr.y (mgr.14146) 1536 : cluster [DBG] pgmap v2203: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-02-12T16:35:49.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:49.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:49.892 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.16095.0:255640 5.0 e603/581 5:05bcfd21:::benchmark_data_smithi169_84712_object86250:head v 603'15941, mlcod=603'15938)", 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:27:17.626215+0000", 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 512.26049293899996, 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.153774877, 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.626215+0000", 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.626215+0000", 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.626213+0000", 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0012e-05 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.626223+0000", 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14030864700000001 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.766532+0000", 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1600000000000001e-05 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.766543+0000", 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030770999999999999 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.766851+0000", 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2731999999999999e-05 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.766874+0000", 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013071665 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.779946+0000", 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4192e-05 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.779990+0000", 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.153774877 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.16095.0:255645 5.8 e603/585 5:1a0809ee:::benchmark_data_smithi169_84712_object86255:head v 603'15955, mlcod=603'15953)", 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:27:17.628479+0000", 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 512.25822852199997, 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15618848499999999, 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.628479+0000", 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.628479+0000", 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.628478+0000", 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9610000000000005e-06 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.628486+0000", 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14111628000000001 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.769602+0000", 2024-02-12T16:35:49.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4950000000000002e-05 2024-02-12T16:35:49.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.769647+0000", 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060146239999999997 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.775662+0000", 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7419000000000002e-05 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.775699+0000", 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089246970000000005 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.784624+0000", 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4121e-05 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:27:17.784668+0000", 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15618848499999999 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303055 6.17 6.6feb8017 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.902 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.124172+0000", 2024-02-12T16:35:49.903 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.762535611, 2024-02-12T16:35:49.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.68241777299999995, 2024-02-12T16:35:49.903 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.903 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.903 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.903 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303055 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124172+0000", 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124172+0000", 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999948 2024-02-12T16:35:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124167+0000", 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5682999999999999e-05 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124183+0000", 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6696e-05 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124210+0000", 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.433e-05 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124224+0000", 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1990000000000006e-05 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124306+0000", 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9810000000000003e-06 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124314+0000", 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.68226310099999998 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806577+0000", 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3002e-05 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806590+0000", 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.68241777299999995 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.907 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303097 6.17 6.608bff37 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.175939+0000", 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.71076875899999, 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63069973599999996, 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.908 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303097 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175939+0000", 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175939+0000", 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999957 2024-02-12T16:35:49.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175935+0000", 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4701e-05 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175949+0000", 2024-02-12T16:35:49.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2152000000000001e-05 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175972+0000", 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3624e-05 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175985+0000", 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9125000000000001e-05 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.176034+0000", 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7129999999999999e-06 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.176041+0000", 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.630590922 2024-02-12T16:35:49.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806632+0000", 2024-02-12T16:35:49.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8449999999999997e-06 2024-02-12T16:35:49.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806639+0000", 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63069973599999996 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303129 6.17 6.d8224497 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.205408+0000", 2024-02-12T16:35:49.914 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.681299413, 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.60125489899999995, 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303129 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205408+0000", 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205408+0000", 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205405+0000", 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0176999999999999e-05 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205416+0000", 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5583999999999999e-05 2024-02-12T16:35:49.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205431+0000", 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2930000000000003e-06 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205439+0000", 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0902999999999997e-05 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205489+0000", 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9179999999999997e-06 2024-02-12T16:35:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205496+0000", 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.60115846799999995 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806655+0000", 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4719999999999995e-06 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806663+0000", 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.60125489899999995 2024-02-12T16:35:49.918 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303143 6.17 6.c304e917 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.218206+0000", 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.66850164600001, 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58847930999999998, 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303143 2024-02-12T16:35:49.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218206+0000", 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218206+0000", 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218204+0000", 2024-02-12T16:35:49.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0628000000000001e-05 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218214+0000", 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9261000000000001e-05 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218233+0000", 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2899999999999997e-06 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218241+0000", 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0324000000000001e-05 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218291+0000", 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9949999999999999e-06 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.218298+0000", 2024-02-12T16:35:49.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58838061100000005 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806679+0000", 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8070000000000002e-06 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806685+0000", 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.58847930999999998 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303154 6.17 6.4cdaa2d7 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.227421+0000", 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.659286748, 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.57928588199999997, 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.924 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303154 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227421+0000", 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227421+0000", 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227418+0000", 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2077e-05 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227430+0000", 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0914e-05 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227451+0000", 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7449999999999995e-06 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227458+0000", 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0139999999999998e-05 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227509+0000", 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3559999999999998e-06 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.227515+0000", 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.57918453000000003 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806699+0000", 2024-02-12T16:35:49.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4710000000000001e-06 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806707+0000", 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.57928588199999997 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303299 6.17 6.55477837 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.381349+0000", 2024-02-12T16:35:49.928 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.505359201, 2024-02-12T16:35:49.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42538305500000001, 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303299 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381349+0000", 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381349+0000", 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381345+0000", 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0394000000000001e-05 2024-02-12T16:35:49.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381356+0000", 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8417e-05 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381374+0000", 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9529999999999999e-06 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381382+0000", 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4865999999999999e-05 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381447+0000", 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1591e-05 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.381459+0000", 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42526808199999999 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806727+0000", 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8520000000000003e-06 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806732+0000", 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.42538305500000001 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303331 6.17 6.75691037 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.413904+0000", 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.472803586, 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39284830900000001, 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.935 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303331 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413904+0000", 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413904+0000", 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:49.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413902+0000", 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.242e-06 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413910+0000", 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3261999999999999e-05 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413923+0000", 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3419999999999998e-06 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413929+0000", 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4671000000000001e-05 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413963+0000", 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7219999999999996e-06 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.413969+0000", 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39277653600000001 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806746+0000", 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8569999999999997e-06 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806753+0000", 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.39284830900000001 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303414 6.17 6.dd56bf37 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.939 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.503674+0000", 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.38303375999999, 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.303100129, 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303414 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503674+0000", 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503674+0000", 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999957 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503670+0000", 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3356000000000001e-05 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503683+0000", 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4329999999999999e-05 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503708+0000", 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8789999999999998e-06 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503718+0000", 2024-02-12T16:35:49.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1952000000000001e-05 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503759+0000", 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0499999999999997e-06 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.503766+0000", 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.30300151600000003 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806767+0000", 2024-02-12T16:35:49.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1280000000000002e-06 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806774+0000", 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.303100129 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303428 6.17 6.923469d7 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.524557+0000", 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.36215051900001, 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28224349599999998, 2024-02-12T16:35:49.944 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303428 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524557+0000", 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524557+0000", 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524555+0000", 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2292e-05 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524567+0000", 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4280999999999999e-05 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524581+0000", 2024-02-12T16:35:49.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8610000000000003e-06 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524587+0000", 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4425000000000002e-05 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524632+0000", 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3810000000000001e-06 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.524639+0000", 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28215449999999997 2024-02-12T16:35:49.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806794+0000", 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2049999999999996e-06 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806801+0000", 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28224349599999998 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.948 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303431 6.17 6.c930af7 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.529242+0000", 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.35746622900001, 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.277583994, 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303431 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529242+0000", 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529242+0000", 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529240+0000", 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3010000000000001e-06 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529247+0000", 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6926999999999999e-05 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529264+0000", 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7289999999999997e-06 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529270+0000", 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9338999999999997e-05 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529309+0000", 2024-02-12T16:35:49.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9030000000000004e-06 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.529317+0000", 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27750394299999998 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806821+0000", 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7369999999999997e-06 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806826+0000", 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.277583994 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303482 6.17 6.f38098f7 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.587272+0000", 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.29943607800001, 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.219576833, 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.953 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303482 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587272+0000", 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587272+0000", 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587269+0000", 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8099999999999992e-06 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587279+0000", 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8558000000000001e-05 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587297+0000", 2024-02-12T16:35:49.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9810000000000003e-06 2024-02-12T16:35:49.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587305+0000", 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6961e-05 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587352+0000", 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0597999999999999e-05 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.587363+0000", 2024-02-12T16:35:49.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21947847600000001 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806841+0000", 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2930000000000003e-06 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806849+0000", 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.219576833 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303487 6.17 6.e7fa0377 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.593566+0000", 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 140.29314159800001, 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21331007599999999, 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303487 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593566+0000", 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593566+0000", 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999962 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593563+0000", 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1134e-05 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593574+0000", 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0262999999999999e-05 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593594+0000", 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5120000000000008e-06 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593602+0000", 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6650999999999998e-05 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593649+0000", 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9449999999999993e-06 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.593657+0000", 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.213212717 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806870+0000", 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5180000000000001e-06 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806876+0000", 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21331007599999999 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26530.0:30119 7.2 e760/752 7:4fe122fc:::benchmark_data_smithi169_105252_object30118:head v 760'1919, mlcod=760'1916)", 2024-02-12T16:35:49.963 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.797077+0000", 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 80.089630602, 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.041190095000000003, 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797077+0000", 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797077+0000", 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797076+0000", 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9869000000000005e-05 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797146+0000", 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8230000000000001e-06 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797155+0000", 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6949999999999999e-06 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797159+0000", 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000479896 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797639+0000", 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1883e-05 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797651+0000", 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.040586689000000002 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.838238+0000", 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9436000000000001e-05 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.838267+0000", 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.041190095000000003 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.967 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26530.0:30122 7.1 e760/752 7:84444c6f:::benchmark_data_smithi169_105252_object30121:head v 760'1917, mlcod=760'1915)", 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.797726+0000", 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 80.088981481999994, 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.039098156000000002, 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797726+0000", 2024-02-12T16:35:49.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797726+0000", 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797725+0000", 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5376e-05 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797781+0000", 2024-02-12T16:35:49.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.613e-06 2024-02-12T16:35:49.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797787+0000", 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7869999999999999e-06 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797791+0000", 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4703000000000003e-05 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797826+0000", 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5866999999999999e-05 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.797842+0000", 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010640510000000001 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798906+0000", 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.037918871 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.836824+0000", 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.039098156000000002 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.972 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26530.0:30120 7.2 e760/752 7:4a2180a2:::benchmark_data_smithi169_105252_object30119:head v 760'1920, mlcod=760'1916)", 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.798422+0000", 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 80.088285636999998, 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.041838912999999998, 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798422+0000", 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798422+0000", 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798421+0000", 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022539200000000001 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798647+0000", 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3209999999999994e-06 2024-02-12T16:35:49.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798656+0000", 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6829999999999996e-06 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798662+0000", 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5413999999999997e-05 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798697+0000", 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5743000000000002e-05 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.798713+0000", 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.041502041000000003 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.840215+0000", 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6038000000000002e-05 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.840261+0000", 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.041838912999999998 2024-02-12T16:35:49.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26530.0:30116 7.e e760/752 7:772f734d:::benchmark_data_smithi169_105252_object30115:head v 760'1881, mlcod=760'1879)", 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.799124+0000", 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 80.087583773999995, 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.039974569000000001, 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.799124+0000", 2024-02-12T16:35:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.799124+0000", 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.799123+0000", 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9855999999999997e-05 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.799183+0000", 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.037554056000000002 2024-02-12T16:35:49.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.836737+0000", 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.122e-05 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.836758+0000", 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014687899999999999 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.836905+0000", 2024-02-12T16:35:49.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9935999999999998e-05 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.836945+0000", 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021230369999999999 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.839068+0000", 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0744000000000002e-05 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.839099+0000", 2024-02-12T16:35:49.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.039974569000000001 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26530.0:30126 7.2 e760/752 7:4517884e:::benchmark_data_smithi169_105252_object30125:head v 760'1921, mlcod=760'1917)", 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:34:29.799913+0000", 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 80.086794746999999, 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.040615766999999997, 2024-02-12T16:35:49.981 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.799913+0000", 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.799913+0000", 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.799912+0000", 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7583000000000004e-05 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.800009+0000", 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036734779000000002 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.836744+0000", 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1921e-05 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.836756+0000", 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001572688 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.838329+0000", 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0762e-05 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.838370+0000", 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021329550000000002 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.840503+0000", 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308999999999998e-05 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:34:29.840529+0000", 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.040615766999999997 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26530.0:163822 7.e e788/752 7:7ddb78ab:::benchmark_data_smithi169_105252_object163821:head v 788'10254, mlcod=788'10250)", 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:35:49.261610+0000", 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.62509804300000005, 2024-02-12T16:35:49.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.034795685, 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.261610+0000", 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.261610+0000", 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.261608+0000", 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0649999999999998e-05 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.261659+0000", 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4021e-05 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.261673+0000", 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.131e-06 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.261680+0000", 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040673130000000004 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.265747+0000", 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5687999999999999e-05 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.265773+0000", 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030593089 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.296366+0000", 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9422999999999998e-05 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:35:49.296405+0000", 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.034795685 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:35:49.989 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:35:49.990 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:35:49.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:50.014 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:50.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:50.143 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:50.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 164971 164955 118.489 76.5625 0.00660398 0.00843821 2024-02-12T16:35:50.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:50.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:50.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:50.408 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:50.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:50.537 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:50.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:50.667 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:50.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-12T16:35:50.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:50.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:50.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:50.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:50.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:51.072 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:51.103 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:51.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:51.137 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:51.138 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:51.138 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:35:51.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 166173 166157 117.997 75.125 0.0327233 0.00847283 2024-02-12T16:35:51.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:51.240 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:51.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:51.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:51.373 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:51.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:51.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:51.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:51 smithi169 bash[19604]: cluster 2024-02-12T16:35:50.128887+0000 mgr.y (mgr.14146) 1537 : cluster [DBG] pgmap v2204: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 986 op/s 2024-02-12T16:35:51.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:51.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:51.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:51.640 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:51.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:51.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:51.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:51 smithi121 bash[18489]: cluster 2024-02-12T16:35:50.128887+0000 mgr.y (mgr.14146) 1537 : cluster [DBG] pgmap v2204: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 986 op/s 2024-02-12T16:35:51.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:51.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:51 smithi123 bash[19119]: cluster 2024-02-12T16:35:50.128887+0000 mgr.y (mgr.14146) 1537 : cluster [DBG] pgmap v2204: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 986 op/s 2024-02-12T16:35:51.890 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:51.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:52.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:52.051 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:52.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:52.188 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:52.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 167380 167364 117.518 75.4375 0.0060617 0.00850784 2024-02-12T16:35:52.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:52.316 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:52.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:52.450 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:52.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:52.593 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:52.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:52.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:52.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:52.864 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:52.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:52.991 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:52.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:53.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:53.144 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:53.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:53.192 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 90 15 168591 168576 117.054 75.75 0.0105978 0.00854074 2024-02-12T16:35:53.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:53.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:53.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:53.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:53.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:53.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:53.566 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:53.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:53.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:53.683 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:53 smithi121 bash[18489]: cluster 2024-02-12T16:35:52.129319+0000 mgr.y (mgr.14146) 1538 : cluster [DBG] pgmap v2205: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-02-12T16:35:53.710 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:53.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:53 smithi169 bash[19604]: cluster 2024-02-12T16:35:52.129319+0000 mgr.y (mgr.14146) 1538 : cluster [DBG] pgmap v2205: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-02-12T16:35:53.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:53.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:53.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:53 smithi123 bash[19119]: cluster 2024-02-12T16:35:52.129319+0000 mgr.y (mgr.14146) 1538 : cluster [DBG] pgmap v2205: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-02-12T16:35:53.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:53.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:53.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:54.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:54.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:54.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.011367 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 168591 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 117.062 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 59.8613 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 368.062 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 62.375 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 1872 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 957 2024-02-12T16:35:54.193 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 5889 2024-02-12T16:35:54.194 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 998 2024-02-12T16:35:54.194 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.00854158 2024-02-12T16:35:54.194 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.0102617 2024-02-12T16:35:54.194 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 0.477087 2024-02-12T16:35:54.194 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.00104071 2024-02-12T16:35:54.205 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:35:54.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:54.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:54.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:54.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:54.398 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:54.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:54.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:54.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:54.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:54.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:54.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:54.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:54.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:54.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:54.911 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:54.936 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:54.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:55.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:55.075 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:55.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:55.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:55.224 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:55.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:55.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:55.363 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:55.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:55.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:55.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:55.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:55.524 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:55 smithi121 bash[18489]: cluster 2024-02-12T16:35:54.129936+0000 mgr.y (mgr.14146) 1539 : cluster [DBG] pgmap v2206: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-12T16:35:55.545 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:35:55.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:55.631 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:55.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:55 smithi123 bash[19119]: cluster 2024-02-12T16:35:54.129936+0000 mgr.y (mgr.14146) 1539 : cluster [DBG] pgmap v2206: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-12T16:35:55.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:55.763 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:35:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:55.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:35:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:55 smithi169 bash[19604]: cluster 2024-02-12T16:35:54.129936+0000 mgr.y (mgr.14146) 1539 : cluster [DBG] pgmap v2206: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-02-12T16:35:55.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:55.888 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:55.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:55.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:56.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:56.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:56.150 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:56.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:56.288 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:56.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:56.392 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:56.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:56.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:56.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:56.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:56.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:56.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:56.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:56.828 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:56.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:56 smithi121 bash[18489]: cluster 2024-02-12T16:35:56.130365+0000 mgr.y (mgr.14146) 1540 : cluster [DBG] pgmap v2207: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 70 MiB/s wr, 1.18k op/s 2024-02-12T16:35:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:56.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:56.976 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:56.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:57.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:57.089 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:56 smithi123 bash[19119]: cluster 2024-02-12T16:35:56.130365+0000 mgr.y (mgr.14146) 1540 : cluster [DBG] pgmap v2207: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 70 MiB/s wr, 1.18k op/s 2024-02-12T16:35:57.108 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:35:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:57.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:35:57.146 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:56 smithi169 bash[19604]: cluster 2024-02-12T16:35:56.130365+0000 mgr.y (mgr.14146) 1540 : cluster [DBG] pgmap v2207: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 70 MiB/s wr, 1.18k op/s 2024-02-12T16:35:57.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:57.236 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:35:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:57.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:35:57.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:57.371 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:57.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:57.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:57.500 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:57.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:57.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:57.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:57.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:57.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:57.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:57.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:57.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:35:57.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:35:58.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:58.041 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:58.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:58.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:35:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:35:58.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:58.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:58.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:58.446 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:35:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:35:58.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:35:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:35:58.693 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:58.725 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:35:58.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:58.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:58.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:58.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:58.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: cluster 2024-02-12T16:35:57.752095+0000 osd.10 (osd.10) 164 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: cluster 2024-02-12T16:35:57.752151+0000 osd.10 (osd.10) 165 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: cluster 2024-02-12T16:35:57.830626+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e789: 12 total, 12 up, 12 in 2024-02-12T16:35:59.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: audit 2024-02-12T16:35:58.506967+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:35:59.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: audit 2024-02-12T16:35:58.507240+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:35:59.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: audit 2024-02-12T16:35:58.507503+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:35:59.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: audit 2024-02-12T16:35:58.563696+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:35:59.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: audit 2024-02-12T16:35:58.570374+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:35:59.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: audit 2024-02-12T16:35:58.571634+0000 mon.c (mon.1) 68 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:35:59.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:35:58 smithi169 bash[19604]: audit 2024-02-12T16:35:58.577404+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:59.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:59.124 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:35:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:59.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:35:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: cluster 2024-02-12T16:35:57.752095+0000 osd.10 (osd.10) 164 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: cluster 2024-02-12T16:35:57.752151+0000 osd.10 (osd.10) 165 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: cluster 2024-02-12T16:35:57.830626+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e789: 12 total, 12 up, 12 in 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: audit 2024-02-12T16:35:58.506967+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: audit 2024-02-12T16:35:58.507240+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: audit 2024-02-12T16:35:58.507503+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: audit 2024-02-12T16:35:58.563696+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: audit 2024-02-12T16:35:58.570374+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: audit 2024-02-12T16:35:58.571634+0000 mon.c (mon.1) 68 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:35:59.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:35:58 smithi123 bash[19119]: audit 2024-02-12T16:35:58.577404+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:59.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: cluster 2024-02-12T16:35:57.752095+0000 osd.10 (osd.10) 164 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: cluster 2024-02-12T16:35:57.752151+0000 osd.10 (osd.10) 165 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: cluster 2024-02-12T16:35:57.830626+0000 mon.a (mon.0) 2681 : cluster [DBG] osdmap e789: 12 total, 12 up, 12 in 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: audit 2024-02-12T16:35:58.506967+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: audit 2024-02-12T16:35:58.507240+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: audit 2024-02-12T16:35:58.507503+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: audit 2024-02-12T16:35:58.563696+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: audit 2024-02-12T16:35:58.570374+0000 mon.c (mon.1) 67 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: audit 2024-02-12T16:35:58.571634+0000 mon.c (mon.1) 68 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:35:59.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:35:58 smithi121 bash[18489]: audit 2024-02-12T16:35:58.577404+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:35:59.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:59.257 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:35:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:59.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:35:59.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:59.385 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:59.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:59.412 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:35:59.412 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:35:59.412 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-02-12T16:35:59.412 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:35:59.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:59.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:35:59.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:59.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:35:59.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:59.662 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:59.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:59.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:59.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:35:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:59.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:35:59.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:59.939 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:35:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:35:59.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:00.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:00.143 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:00.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:36:00.225 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:36:00.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-12T16:36:00.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:00.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:00.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:00.375 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:36:00.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:00.403 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:36:00.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:00.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:00.445 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:36:00.478 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:36:00.516 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:36:00.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:00.539 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:36:00.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:00.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:00.570 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:36:00.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:36:00.647 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:36:00.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:00.678 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:36:00.685 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:00.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:00.709 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:36:00.740 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.752 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:36:00.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:00.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:00.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:00.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:00.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:00.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:01.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:01.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:01.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:01.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:01.206 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:01.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:01.310 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:01.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:01.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:01 smithi121 bash[18489]: cluster 2024-02-12T16:36:00.131285+0000 mgr.y (mgr.14146) 1542 : cluster [DBG] pgmap v2210: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 48 MiB/s wr, 1.07k op/s 2024-02-12T16:36:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:01 smithi121 bash[18489]: audit 2024-02-12T16:36:00.190184+0000 mgr.y (mgr.14146) 1543 : audit [DBG] from='client.26602 v1:172.21.15.121:0/4055238644' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:36:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:01 smithi121 bash[18489]: audit 2024-02-12T16:36:00.323339+0000 mon.a (mon.0) 2683 : audit [DBG] from='client.? v1:172.21.15.121:0/3187430561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:01 smithi121 bash[18489]: audit 2024-02-12T16:36:01.027206+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? v1:172.21.15.121:0/2224641037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:01.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:01.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:01 smithi169 bash[19604]: cluster 2024-02-12T16:36:00.131285+0000 mgr.y (mgr.14146) 1542 : cluster [DBG] pgmap v2210: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 48 MiB/s wr, 1.07k op/s 2024-02-12T16:36:01.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:01 smithi169 bash[19604]: audit 2024-02-12T16:36:00.190184+0000 mgr.y (mgr.14146) 1543 : audit [DBG] from='client.26602 v1:172.21.15.121:0/4055238644' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:36:01.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:01 smithi169 bash[19604]: audit 2024-02-12T16:36:00.323339+0000 mon.a (mon.0) 2683 : audit [DBG] from='client.? v1:172.21.15.121:0/3187430561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:01.454 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:01 smithi169 bash[19604]: audit 2024-02-12T16:36:01.027206+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? v1:172.21.15.121:0/2224641037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:01.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:01.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:01.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:01.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:01 smithi123 bash[19119]: cluster 2024-02-12T16:36:00.131285+0000 mgr.y (mgr.14146) 1542 : cluster [DBG] pgmap v2210: 47 pgs: 47 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 48 MiB/s wr, 1.07k op/s 2024-02-12T16:36:01.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:01 smithi123 bash[19119]: audit 2024-02-12T16:36:00.190184+0000 mgr.y (mgr.14146) 1543 : audit [DBG] from='client.26602 v1:172.21.15.121:0/4055238644' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:36:01.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:01 smithi123 bash[19119]: audit 2024-02-12T16:36:00.323339+0000 mon.a (mon.0) 2683 : audit [DBG] from='client.? v1:172.21.15.121:0/3187430561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:01.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:01 smithi123 bash[19119]: audit 2024-02-12T16:36:01.027206+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? v1:172.21.15.121:0/2224641037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:01.613 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:01.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:01.691 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2210,"stamp":"2024-02-12T16:36:00.130952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10855514136,"num_objects":165643,"num_object_clones":0,"num_object_copies":331286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171541,"num_write_kb":10789825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":28119740,"kb_used_data":21199708,"kb_used_omap":91,"kb_used_meta":6919844,"kb_avail":1096822084,"statfs":{"total":1151940427776,"available":1123145814016,"internally_reserved":0,"allocated":21708500992,"data_stored":21704283962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":7085920272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,8,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":313655320,"num_objects":4787,"num_object_clones":0,"num_object_copies":9574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10685,"num_write_kb":495041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002358"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":148,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.415668+0000","last_change":"2024-02-12T16:35:25.445580+0000","last_active":"2024-02-12T16:35:43.415668+0000","last_peered":"2024-02-12T16:35:43.415668+0000","last_clean":"2024-02-12T16:35:43.415668+0000","last_became_active":"2024-02-12T16:30:55.395625+0000","last_became_peered":"2024-02-12T16:30:55.395625+0000","last_unstale":"2024-02-12T16:35:43.415668+0000","last_undegraded":"2024-02-12T16:35:43.415668+0000","last_fullsized":"2024-02-12T16:35:43.415668+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:25.445474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:25.445474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":149,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.392546+0000","last_change":"2024-02-12T16:35:18.045317+0000","last_active":"2024-02-12T16:35:41.392546+0000","last_peered":"2024-02-12T16:35:41.392546+0000","last_clean":"2024-02-12T16:35:41.392546+0000","last_became_active":"2024-02-12T16:30:55.398431+0000","last_became_peered":"2024-02-12T16:30:55.398431+0000","last_unstale":"2024-02-12T16:35:41.392546+0000","last_undegraded":"2024-02-12T16:35:41.392546+0000","last_fullsized":"2024-02-12T16:35:41.392546+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:18.045236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:18.045236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":147,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.400499+0000","last_change":"2024-02-12T16:35:15.289078+0000","last_active":"2024-02-12T16:35:43.400499+0000","last_peered":"2024-02-12T16:35:43.400499+0000","last_clean":"2024-02-12T16:35:43.400499+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:35:43.400499+0000","last_undegraded":"2024-02-12T16:35:43.400499+0000","last_fullsized":"2024-02-12T16:35:43.400499+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:15.288966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:15.288966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":147,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.400458+0000","last_change":"2024-02-12T16:35:04.397528+0000","last_active":"2024-02-12T16:35:43.400458+0000","last_peered":"2024-02-12T16:35:43.400458+0000","last_clean":"2024-02-12T16:35:43.400458+0000","last_became_active":"2024-02-12T16:30:55.409579+0000","last_became_peered":"2024-02-12T16:30:55.409579+0000","last_unstale":"2024-02-12T16:35:43.400458+0000","last_undegraded":"2024-02-12T16:35:43.400458+0000","last_fullsized":"2024-02-12T16:35:43.400458+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:04.397374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:04.397374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":196,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.407978+0000","last_change":"2024-02-12T16:35:10.432374+0000","last_active":"2024-02-12T16:35:43.407978+0000","last_peered":"2024-02-12T16:35:43.407978+0000","last_clean":"2024-02-12T16:35:43.407978+0000","last_became_active":"2024-02-12T16:30:55.394105+0000","last_became_peered":"2024-02-12T16:30:55.394105+0000","last_unstale":"2024-02-12T16:35:43.407978+0000","last_undegraded":"2024-02-12T16:35:43.407978+0000","last_fullsized":"2024-02-12T16:35:43.407978+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:10.432253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:35:10.432253+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":147,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.412049+0000","last_change":"2024-02-12T16:35:18.834610+0000","last_active":"2024-02-12T16:35:43.412049+0000","last_peered":"2024-02-12T16:35:43.412049+0000","last_clean":"2024-02-12T16:35:43.412049+0000","last_became_active":"2024-02-12T16:30:55.542451+0000","last_became_peered":"2024-02-12T16:30:55.542451+0000","last_unstale":"2024-02-12T16:35:43.412049+0000","last_undegraded":"2024-02-12T16:35:43.412049+0000","last_fullsized":"2024-02-12T16:35:43.412049+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:18.834357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:18.834357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":502,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.404184+0000","last_change":"2024-02-12T16:35:34.988787+0000","last_active":"2024-02-12T16:35:41.404184+0000","last_peered":"2024-02-12T16:35:41.404184+0000","last_clean":"2024-02-12T16:35:41.404184+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:35:41.404184+0000","last_undegraded":"2024-02-12T16:35:41.404184+0000","last_fullsized":"2024-02-12T16:35:41.404184+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:34.988672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:35:34.988672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":146,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.401720+0000","last_change":"2024-02-12T16:35:08.385373+0000","last_active":"2024-02-12T16:35:43.401720+0000","last_peered":"2024-02-12T16:35:43.401720+0000","last_clean":"2024-02-12T16:35:43.401720+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:35:43.401720+0000","last_undegraded":"2024-02-12T16:35:43.401720+0000","last_fullsized":"2024-02-12T16:35:43.401720+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:08.385272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:08.385272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":151,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.392454+0000","last_change":"2024-02-12T16:35:33.040481+0000","last_active":"2024-02-12T16:35:41.392454+0000","last_peered":"2024-02-12T16:35:41.392454+0000","last_clean":"2024-02-12T16:35:41.392454+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:35:41.392454+0000","last_undegraded":"2024-02-12T16:35:41.392454+0000","last_fullsized":"2024-02-12T16:35:41.392454+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:33.040362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:33.040362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":141,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.406385+0000","last_change":"2024-02-12T16:35:03.687751+0000","last_active":"2024-02-12T16:35:43.406385+0000","last_peered":"2024-02-12T16:35:43.406385+0000","last_clean":"2024-02-12T16:35:43.406385+0000","last_became_active":"2024-02-12T16:30:55.404763+0000","last_became_peered":"2024-02-12T16:30:55.404763+0000","last_unstale":"2024-02-12T16:35:43.406385+0000","last_undegraded":"2024-02-12T16:35:43.406385+0000","last_fullsized":"2024-02-12T16:35:43.406385+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:03.687669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:35:03.687669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":352,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.387270+0000","last_change":"2024-02-12T16:35:16.894972+0000","last_active":"2024-02-12T16:35:41.387270+0000","last_peered":"2024-02-12T16:35:41.387270+0000","last_clean":"2024-02-12T16:35:41.387270+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:35:41.387270+0000","last_undegraded":"2024-02-12T16:35:41.387270+0000","last_fullsized":"2024-02-12T16:35:41.387270+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:07.718055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:35:07.718055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"788'10685","reported_seq":10732,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.317988+0000","last_change":"2024-02-12T16:35:18.367383+0000","last_active":"2024-02-12T16:35:56.317988+0000","last_peered":"2024-02-12T16:35:56.317988+0000","last_clean":"2024-02-12T16:35:56.317988+0000","last_became_active":"2024-02-12T16:34:08.148188+0000","last_became_peered":"2024-02-12T16:34:08.148188+0000","last_unstale":"2024-02-12T16:35:56.317988+0000","last_undegraded":"2024-02-12T16:35:56.317988+0000","last_fullsized":"2024-02-12T16:35:56.317988+0000","mapping_epoch":752,"log_start":"759'600","ondisk_log_start":"759'600","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'7864","last_scrub_stamp":"2024-02-12T16:35:18.367314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:18.367314+0000","log_size":10085,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":679804928,"num_objects":10373,"num_object_clones":0,"num_object_copies":20746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10685,"num_write_kb":673856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":458,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.396603+0000","last_change":"2024-02-12T16:35:08.826049+0000","last_active":"2024-02-12T16:35:41.396603+0000","last_peered":"2024-02-12T16:35:41.396603+0000","last_clean":"2024-02-12T16:35:41.396603+0000","last_became_active":"2024-02-12T16:21:09.577897+0000","last_became_peered":"2024-02-12T16:21:09.577897+0000","last_unstale":"2024-02-12T16:35:41.396603+0000","last_undegraded":"2024-02-12T16:35:41.396603+0000","last_fullsized":"2024-02-12T16:35:41.396603+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:08.825938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:08.825938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.3","version":"788'10714","reported_seq":10764,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.047894+0000","last_change":"2024-02-12T16:35:19.003072+0000","last_active":"2024-02-12T16:35:56.047894+0000","last_peered":"2024-02-12T16:35:56.047894+0000","last_clean":"2024-02-12T16:35:56.047894+0000","last_became_active":"2024-02-12T16:34:08.036294+0000","last_became_peered":"2024-02-12T16:34:08.036294+0000","last_unstale":"2024-02-12T16:35:56.047894+0000","last_undegraded":"2024-02-12T16:35:56.047894+0000","last_fullsized":"2024-02-12T16:35:56.047894+0000","mapping_epoch":752,"log_start":"759'700","ondisk_log_start":"759'700","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'7952","last_scrub_stamp":"2024-02-12T16:35:19.002953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:19.002953+0000","log_size":10014,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":684720128,"num_objects":10448,"num_object_clones":0,"num_object_copies":20896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10714,"num_write_kb":677184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":507,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.408738+0000","last_change":"2024-02-12T16:35:34.775120+0000","last_active":"2024-02-12T16:35:43.408738+0000","last_peered":"2024-02-12T16:35:43.408738+0000","last_clean":"2024-02-12T16:35:43.408738+0000","last_became_active":"2024-02-12T16:30:49.374358+0000","last_became_peered":"2024-02-12T16:30:49.374358+0000","last_unstale":"2024-02-12T16:35:43.408738+0000","last_undegraded":"2024-02-12T16:35:43.408738+0000","last_fullsized":"2024-02-12T16:35:43.408738+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:34.775051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:35:34.775051+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.2","version":"788'10663","reported_seq":10711,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.052269+0000","last_change":"2024-02-12T16:35:31.873084+0000","last_active":"2024-02-12T16:35:56.052269+0000","last_peered":"2024-02-12T16:35:56.052269+0000","last_clean":"2024-02-12T16:35:56.052269+0000","last_became_active":"2024-02-12T16:34:08.148290+0000","last_became_peered":"2024-02-12T16:34:08.148290+0000","last_unstale":"2024-02-12T16:35:56.052269+0000","last_undegraded":"2024-02-12T16:35:56.052269+0000","last_fullsized":"2024-02-12T16:35:56.052269+0000","mapping_epoch":752,"log_start":"759'600","ondisk_log_start":"759'600","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'8980","last_scrub_stamp":"2024-02-12T16:35:31.872998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:31.872998+0000","log_size":10063,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":680460288,"num_objects":10383,"num_object_clones":0,"num_object_copies":20766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":673472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":895,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.415794+0000","last_change":"2024-02-12T16:35:06.330589+0000","last_active":"2024-02-12T16:35:43.415794+0000","last_peered":"2024-02-12T16:35:43.415794+0000","last_clean":"2024-02-12T16:35:43.415794+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:35:43.415794+0000","last_undegraded":"2024-02-12T16:35:43.415794+0000","last_fullsized":"2024-02-12T16:35:43.415794+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:06.330507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:06.330507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.1","version":"789'10926","reported_seq":10975,"reported_epoch":789,"state":"active+clean","last_fresh":"2024-02-12T16:35:59.315970+0000","last_change":"2024-02-12T16:35:40.519702+0000","last_active":"2024-02-12T16:35:59.315970+0000","last_peered":"2024-02-12T16:35:59.315970+0000","last_clean":"2024-02-12T16:35:59.315970+0000","last_became_active":"2024-02-12T16:34:08.149664+0000","last_became_peered":"2024-02-12T16:34:08.149664+0000","last_unstale":"2024-02-12T16:35:59.315970+0000","last_undegraded":"2024-02-12T16:35:59.315970+0000","last_fullsized":"2024-02-12T16:35:59.315970+0000","mapping_epoch":752,"log_start":"759'900","ondisk_log_start":"759'900","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"786'9571","last_scrub_stamp":"2024-02-12T16:35:40.519592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:40.519592+0000","log_size":10026,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":664928256,"num_objects":10146,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10926,"num_write_kb":674304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":550,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.389105+0000","last_change":"2024-02-12T16:35:23.788654+0000","last_active":"2024-02-12T16:35:41.389105+0000","last_peered":"2024-02-12T16:35:41.389105+0000","last_clean":"2024-02-12T16:35:41.389105+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:35:41.389105+0000","last_undegraded":"2024-02-12T16:35:41.389105+0000","last_fullsized":"2024-02-12T16:35:41.389105+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:23.788558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:23.788558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"788'10867","reported_seq":10915,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:57.425130+0000","last_change":"2024-02-12T16:35:22.520848+0000","last_active":"2024-02-12T16:35:57.425130+0000","last_peered":"2024-02-12T16:35:57.425130+0000","last_clean":"2024-02-12T16:35:57.425130+0000","last_became_active":"2024-02-12T16:34:08.041885+0000","last_became_peered":"2024-02-12T16:34:08.041885+0000","last_unstale":"2024-02-12T16:35:57.425130+0000","last_undegraded":"2024-02-12T16:35:57.425130+0000","last_fullsized":"2024-02-12T16:35:57.425130+0000","mapping_epoch":752,"log_start":"759'800","ondisk_log_start":"759'800","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'8277","last_scrub_stamp":"2024-02-12T16:35:22.520759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:22.520759+0000","log_size":10067,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":681246720,"num_objects":10395,"num_object_clones":0,"num_object_copies":20790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10867,"num_write_kb":680384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":494,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.408829+0000","last_change":"2024-02-12T16:35:11.714927+0000","last_active":"2024-02-12T16:35:43.408829+0000","last_peered":"2024-02-12T16:35:43.408829+0000","last_clean":"2024-02-12T16:35:43.408829+0000","last_became_active":"2024-02-12T16:30:49.373979+0000","last_became_peered":"2024-02-12T16:30:49.373979+0000","last_unstale":"2024-02-12T16:35:43.408829+0000","last_undegraded":"2024-02-12T16:35:43.408829+0000","last_fullsized":"2024-02-12T16:35:43.408829+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:11.714823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:11.714823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.7","version":"788'10553","reported_seq":10601,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:55.381080+0000","last_change":"2024-02-12T16:35:32.830560+0000","last_active":"2024-02-12T16:35:55.381080+0000","last_peered":"2024-02-12T16:35:55.381080+0000","last_clean":"2024-02-12T16:35:55.381080+0000","last_became_active":"2024-02-12T16:34:08.150853+0000","last_became_peered":"2024-02-12T16:34:08.150853+0000","last_unstale":"2024-02-12T16:35:55.381080+0000","last_undegraded":"2024-02-12T16:35:55.381080+0000","last_fullsized":"2024-02-12T16:35:55.381080+0000","mapping_epoch":752,"log_start":"759'500","ondisk_log_start":"759'500","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'8955","last_scrub_stamp":"2024-02-12T16:35:32.830417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:32.830417+0000","log_size":10053,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":680067072,"num_objects":10377,"num_object_clones":0,"num_object_copies":20754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":669760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":531,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.412000+0000","last_change":"2024-02-12T16:35:06.735527+0000","last_active":"2024-02-12T16:35:43.412000+0000","last_peered":"2024-02-12T16:35:43.412000+0000","last_clean":"2024-02-12T16:35:43.412000+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:35:43.412000+0000","last_undegraded":"2024-02-12T16:35:43.412000+0000","last_fullsized":"2024-02-12T16:35:43.412000+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:06.735411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:35:06.735411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.5","version":"789'10854","reported_seq":10903,"reported_epoch":789,"state":"active+clean","last_fresh":"2024-02-12T16:35:59.893654+0000","last_change":"2024-02-12T16:35:17.683301+0000","last_active":"2024-02-12T16:35:59.893654+0000","last_peered":"2024-02-12T16:35:59.893654+0000","last_clean":"2024-02-12T16:35:59.893654+0000","last_became_active":"2024-02-12T16:34:08.041057+0000","last_became_peered":"2024-02-12T16:34:08.041057+0000","last_unstale":"2024-02-12T16:35:59.893654+0000","last_undegraded":"2024-02-12T16:35:59.893654+0000","last_fullsized":"2024-02-12T16:35:59.893654+0000","mapping_epoch":752,"log_start":"759'800","ondisk_log_start":"759'800","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'7774","last_scrub_stamp":"2024-02-12T16:35:17.683194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:17.683194+0000","log_size":10054,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":662831104,"num_objects":10114,"num_object_clones":0,"num_object_copies":20228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":670976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":353,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.401376+0000","last_change":"2024-02-12T16:35:40.401022+0000","last_active":"2024-02-12T16:35:41.401376+0000","last_peered":"2024-02-12T16:35:41.401376+0000","last_clean":"2024-02-12T16:35:41.401376+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:35:41.401376+0000","last_undegraded":"2024-02-12T16:35:41.401376+0000","last_fullsized":"2024-02-12T16:35:41.401376+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:40.400930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:40.400930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"788'10715","reported_seq":10763,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.309177+0000","last_change":"2024-02-12T16:35:32.811366+0000","last_active":"2024-02-12T16:35:56.309177+0000","last_peered":"2024-02-12T16:35:56.309177+0000","last_clean":"2024-02-12T16:35:56.309177+0000","last_became_active":"2024-02-12T16:34:08.039442+0000","last_became_peered":"2024-02-12T16:34:08.039442+0000","last_unstale":"2024-02-12T16:35:56.309177+0000","last_undegraded":"2024-02-12T16:35:56.309177+0000","last_fullsized":"2024-02-12T16:35:56.309177+0000","mapping_epoch":752,"log_start":"759'700","ondisk_log_start":"759'700","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'9039","last_scrub_stamp":"2024-02-12T16:35:32.811276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:32.811276+0000","log_size":10015,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":681246720,"num_objects":10395,"num_object_clones":0,"num_object_copies":20790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10715,"num_write_kb":675520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":921,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.406486+0000","last_change":"2024-02-12T16:35:21.488378+0000","last_active":"2024-02-12T16:35:43.406486+0000","last_peered":"2024-02-12T16:35:43.406486+0000","last_clean":"2024-02-12T16:35:43.406486+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:35:43.406486+0000","last_undegraded":"2024-02-12T16:35:43.406486+0000","last_fullsized":"2024-02-12T16:35:43.406486+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:21.488276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:21.488276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"788'10871","reported_seq":10917,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.988640+0000","last_change":"2024-02-12T16:35:41.411122+0000","last_active":"2024-02-12T16:35:56.988640+0000","last_peered":"2024-02-12T16:35:56.988640+0000","last_clean":"2024-02-12T16:35:56.988640+0000","last_became_active":"2024-02-12T16:34:08.036767+0000","last_became_peered":"2024-02-12T16:34:08.036767+0000","last_unstale":"2024-02-12T16:35:56.988640+0000","last_undegraded":"2024-02-12T16:35:56.988640+0000","last_fullsized":"2024-02-12T16:35:56.988640+0000","mapping_epoch":752,"log_start":"759'800","ondisk_log_start":"759'800","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:34:07.024012+0000","log_size":10071,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":682950656,"num_objects":10421,"num_object_clones":0,"num_object_copies":20842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10871,"num_write_kb":681344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":309,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.406477+0000","last_change":"2024-02-12T16:35:26.539904+0000","last_active":"2024-02-12T16:35:43.406477+0000","last_peered":"2024-02-12T16:35:43.406477+0000","last_clean":"2024-02-12T16:35:43.406477+0000","last_became_active":"2024-02-12T16:30:49.374934+0000","last_became_peered":"2024-02-12T16:30:49.374934+0000","last_unstale":"2024-02-12T16:35:43.406477+0000","last_undegraded":"2024-02-12T16:35:43.406477+0000","last_fullsized":"2024-02-12T16:35:43.406477+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:26.539828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:26.539828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"788'10666","reported_seq":10714,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.437621+0000","last_change":"2024-02-12T16:35:27.977979+0000","last_active":"2024-02-12T16:35:56.437621+0000","last_peered":"2024-02-12T16:35:56.437621+0000","last_clean":"2024-02-12T16:35:56.437621+0000","last_became_active":"2024-02-12T16:34:08.041413+0000","last_became_peered":"2024-02-12T16:34:08.041413+0000","last_unstale":"2024-02-12T16:35:56.437621+0000","last_undegraded":"2024-02-12T16:35:56.437621+0000","last_fullsized":"2024-02-12T16:35:56.437621+0000","mapping_epoch":752,"log_start":"759'600","ondisk_log_start":"759'600","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'8586","last_scrub_stamp":"2024-02-12T16:35:27.977872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:27.977872+0000","log_size":10066,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":677904384,"num_objects":10344,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10666,"num_write_kb":672320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":333,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.403983+0000","last_change":"2024-02-12T16:35:19.522713+0000","last_active":"2024-02-12T16:35:41.403983+0000","last_peered":"2024-02-12T16:35:41.403983+0000","last_clean":"2024-02-12T16:35:41.403983+0000","last_became_active":"2024-02-12T16:30:49.382268+0000","last_became_peered":"2024-02-12T16:30:49.382268+0000","last_unstale":"2024-02-12T16:35:41.403983+0000","last_undegraded":"2024-02-12T16:35:41.403983+0000","last_fullsized":"2024-02-12T16:35:41.403983+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:19.522641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:19.522641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"788'10546","reported_seq":10594,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.256455+0000","last_change":"2024-02-12T16:35:29.795980+0000","last_active":"2024-02-12T16:35:56.256455+0000","last_peered":"2024-02-12T16:35:56.256455+0000","last_clean":"2024-02-12T16:35:56.256455+0000","last_became_active":"2024-02-12T16:34:08.040236+0000","last_became_peered":"2024-02-12T16:34:08.040236+0000","last_unstale":"2024-02-12T16:35:56.256455+0000","last_undegraded":"2024-02-12T16:35:56.256455+0000","last_fullsized":"2024-02-12T16:35:56.256455+0000","mapping_epoch":752,"log_start":"759'500","ondisk_log_start":"759'500","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"784'8676","last_scrub_stamp":"2024-02-12T16:35:29.795895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:29.795895+0000","log_size":10046,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":672792576,"num_objects":10266,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":665984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":503,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.389283+0000","last_change":"2024-02-12T16:35:28.794885+0000","last_active":"2024-02-12T16:35:41.389283+0000","last_peered":"2024-02-12T16:35:41.389283+0000","last_clean":"2024-02-12T16:35:41.389283+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:35:41.389283+0000","last_undegraded":"2024-02-12T16:35:41.389283+0000","last_fullsized":"2024-02-12T16:35:41.389283+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:28.794774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:35:28.794774+0000","last_clean_scrub_stamp":"2024-02-12T16:35:28.794774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"788'10778","reported_seq":10825,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.879413+0000","last_change":"2024-02-12T16:35:32.405722+0000","last_active":"2024-02-12T16:35:56.879413+0000","last_peered":"2024-02-12T16:35:56.879413+0000","last_clean":"2024-02-12T16:35:56.879413+0000","last_became_active":"2024-02-12T16:34:08.147817+0000","last_became_peered":"2024-02-12T16:34:08.147817+0000","last_unstale":"2024-02-12T16:35:56.879413+0000","last_undegraded":"2024-02-12T16:35:56.879413+0000","last_fullsized":"2024-02-12T16:35:56.879413+0000","mapping_epoch":752,"log_start":"759'700","ondisk_log_start":"759'700","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"785'9023","last_scrub_stamp":"2024-02-12T16:35:32.405631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:32.405631+0000","log_size":10078,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":681574400,"num_objects":10400,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":677696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":312,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.408000+0000","last_change":"2024-02-12T16:35:19.574228+0000","last_active":"2024-02-12T16:35:43.408000+0000","last_peered":"2024-02-12T16:35:43.408000+0000","last_clean":"2024-02-12T16:35:43.408000+0000","last_became_active":"2024-02-12T16:30:49.381004+0000","last_became_peered":"2024-02-12T16:30:49.381004+0000","last_unstale":"2024-02-12T16:35:43.408000+0000","last_undegraded":"2024-02-12T16:35:43.408000+0000","last_fullsized":"2024-02-12T16:35:43.408000+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:19.574110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:19.574110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"788'10717","reported_seq":10765,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.322011+0000","last_change":"2024-02-12T16:35:14.175766+0000","last_active":"2024-02-12T16:35:56.322011+0000","last_peered":"2024-02-12T16:35:56.322011+0000","last_clean":"2024-02-12T16:35:56.322011+0000","last_became_active":"2024-02-12T16:34:08.149769+0000","last_became_peered":"2024-02-12T16:34:08.149769+0000","last_unstale":"2024-02-12T16:35:56.322011+0000","last_undegraded":"2024-02-12T16:35:56.322011+0000","last_fullsized":"2024-02-12T16:35:56.322011+0000","mapping_epoch":752,"log_start":"759'700","ondisk_log_start":"759'700","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'7499","last_scrub_stamp":"2024-02-12T16:35:14.175580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:14.175580+0000","log_size":10017,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":679542784,"num_objects":10369,"num_object_clones":0,"num_object_copies":20738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":674752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":904,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.406378+0000","last_change":"2024-02-12T16:35:11.592580+0000","last_active":"2024-02-12T16:35:43.406378+0000","last_peered":"2024-02-12T16:35:43.406378+0000","last_clean":"2024-02-12T16:35:43.406378+0000","last_became_active":"2024-02-12T16:30:49.404783+0000","last_became_peered":"2024-02-12T16:30:49.404783+0000","last_unstale":"2024-02-12T16:35:43.406378+0000","last_undegraded":"2024-02-12T16:35:43.406378+0000","last_fullsized":"2024-02-12T16:35:43.406378+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:11.592474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:11.592474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.e","version":"788'10754","reported_seq":10802,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:56.872947+0000","last_change":"2024-02-12T16:35:11.387198+0000","last_active":"2024-02-12T16:35:56.872947+0000","last_peered":"2024-02-12T16:35:56.872947+0000","last_clean":"2024-02-12T16:35:56.872947+0000","last_became_active":"2024-02-12T16:34:08.149135+0000","last_became_peered":"2024-02-12T16:34:08.149135+0000","last_unstale":"2024-02-12T16:35:56.872947+0000","last_undegraded":"2024-02-12T16:35:56.872947+0000","last_fullsized":"2024-02-12T16:35:56.872947+0000","mapping_epoch":752,"log_start":"759'700","ondisk_log_start":"759'700","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"777'7222","last_scrub_stamp":"2024-02-12T16:35:11.386929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:11.386929+0000","log_size":10054,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":680001536,"num_objects":10376,"num_object_clones":0,"num_object_copies":20752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":676160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":496,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.404244+0000","last_change":"2024-02-12T16:35:03.857816+0000","last_active":"2024-02-12T16:35:41.404244+0000","last_peered":"2024-02-12T16:35:41.404244+0000","last_clean":"2024-02-12T16:35:41.404244+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:35:41.404244+0000","last_undegraded":"2024-02-12T16:35:41.404244+0000","last_fullsized":"2024-02-12T16:35:41.404244+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:03.857737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:03.857737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"788'10649","reported_seq":10700,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:55.397374+0000","last_change":"2024-02-12T16:35:22.869146+0000","last_active":"2024-02-12T16:35:55.397374+0000","last_peered":"2024-02-12T16:35:55.397374+0000","last_clean":"2024-02-12T16:35:55.397374+0000","last_became_active":"2024-02-12T16:34:08.037484+0000","last_became_peered":"2024-02-12T16:34:08.037484+0000","last_unstale":"2024-02-12T16:35:55.397374+0000","last_undegraded":"2024-02-12T16:35:55.397374+0000","last_fullsized":"2024-02-12T16:35:55.397374+0000","mapping_epoch":752,"log_start":"759'600","ondisk_log_start":"759'600","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"782'8268","last_scrub_stamp":"2024-02-12T16:35:22.869057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:22.869057+0000","log_size":10049,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":684326936,"num_objects":10443,"num_object_clones":0,"num_object_copies":20886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10649,"num_write_kb":674881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":248,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:43.415751+0000","last_change":"2024-02-12T16:35:10.417915+0000","last_active":"2024-02-12T16:35:43.415751+0000","last_peered":"2024-02-12T16:35:43.415751+0000","last_clean":"2024-02-12T16:35:43.415751+0000","last_became_active":"2024-02-12T16:30:49.376219+0000","last_became_peered":"2024-02-12T16:30:49.376219+0000","last_unstale":"2024-02-12T16:35:43.415751+0000","last_undegraded":"2024-02-12T16:35:43.415751+0000","last_fullsized":"2024-02-12T16:35:43.415751+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:10.417816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:10.417816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"788'10583","reported_seq":10631,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-12T16:35:55.377456+0000","last_change":"2024-02-12T16:35:26.922675+0000","last_active":"2024-02-12T16:35:55.377456+0000","last_peered":"2024-02-12T16:35:55.377456+0000","last_clean":"2024-02-12T16:35:55.377456+0000","last_became_active":"2024-02-12T16:34:08.037558+0000","last_became_peered":"2024-02-12T16:34:08.037558+0000","last_unstale":"2024-02-12T16:35:55.377456+0000","last_undegraded":"2024-02-12T16:35:55.377456+0000","last_fullsized":"2024-02-12T16:35:55.377456+0000","mapping_epoch":752,"log_start":"759'500","ondisk_log_start":"759'500","created":752,"last_epoch_clean":753,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'8571","last_scrub_stamp":"2024-02-12T16:35:26.922569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.024012+0000","last_clean_scrub_stamp":"2024-02-12T16:35:26.922569+0000","log_size":10083,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":681115648,"num_objects":10393,"num_object_clones":0,"num_object_copies":20786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10583,"num_write_kb":671232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":488,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.396737+0000","last_change":"2024-02-12T16:35:03.750833+0000","last_active":"2024-02-12T16:35:41.396737+0000","last_peered":"2024-02-12T16:35:41.396737+0000","last_clean":"2024-02-12T16:35:41.396737+0000","last_became_active":"2024-02-12T16:30:49.384015+0000","last_became_peered":"2024-02-12T16:30:49.384015+0000","last_unstale":"2024-02-12T16:35:41.396737+0000","last_undegraded":"2024-02-12T16:35:41.396737+0000","last_fullsized":"2024-02-12T16:35:41.396737+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:03.750662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:35:03.750662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":313,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.404246+0000","last_change":"2024-02-12T16:35:23.904322+0000","last_active":"2024-02-12T16:35:41.404246+0000","last_peered":"2024-02-12T16:35:41.404246+0000","last_clean":"2024-02-12T16:35:41.404246+0000","last_became_active":"2024-02-12T16:34:16.403771+0000","last_became_peered":"2024-02-12T16:34:16.403771+0000","last_unstale":"2024-02-12T16:35:41.404246+0000","last_undegraded":"2024-02-12T16:35:41.404246+0000","last_fullsized":"2024-02-12T16:35:41.404246+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:23.904216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:35:23.904216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":295,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.392462+0000","last_change":"2024-02-12T16:35:09.018448+0000","last_active":"2024-02-12T16:35:41.392462+0000","last_peered":"2024-02-12T16:35:41.392462+0000","last_clean":"2024-02-12T16:35:41.392462+0000","last_became_active":"2024-02-12T16:23:39.185503+0000","last_became_peered":"2024-02-12T16:23:39.185503+0000","last_unstale":"2024-02-12T16:35:41.392462+0000","last_undegraded":"2024-02-12T16:35:41.392462+0000","last_fullsized":"2024-02-12T16:35:41.392462+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:09.018317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:09.018317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":334,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.404140+0000","last_change":"2024-02-12T16:35:16.899312+0000","last_active":"2024-02-12T16:35:41.404140+0000","last_peered":"2024-02-12T16:35:41.404140+0000","last_clean":"2024-02-12T16:35:41.404140+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:35:41.404140+0000","last_undegraded":"2024-02-12T16:35:41.404140+0000","last_fullsized":"2024-02-12T16:35:41.404140+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:07.846670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:35:07.846670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":318,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-12T16:35:41.393316+0000","last_change":"2024-02-12T16:35:08.806700+0000","last_active":"2024-02-12T16:35:41.393316+0000","last_peered":"2024-02-12T16:35:41.393316+0000","last_clean":"2024-02-12T16:35:41.393316+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:35:41.393316+0000","last_undegraded":"2024-02-12T16:35:41.393316+0000","last_fullsized":"2024-02-12T16:35:41.393316+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:08.806574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:08.806574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":10855514136,"num_objects":165643,"num_object_clones":0,"num_object_copies":331286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165643,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171541,"num_write_kb":10789825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21612077056,"data_stored":21612068912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431958982,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2191928,"kb_used_data":1290992,"kb_used_omap":0,"kb_used_meta":900928,"kb_avail":91553224,"statfs":{"total":95995035648,"available":93750501376,"internally_reserved":0,"allocated":1321975808,"data_stored":1321620560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922550272},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.4490000000000001,"15min":1.0269999999999999},"min":{"1min":0.316,"5min":0.249,"15min":0.246},"max":{"1min":4.0129999999999999,"5min":33.941000000000003,"15min":33.941000000000003},"last":0.434},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.6060000000000001,"15min":1.1559999999999999},"min":{"1min":0.30499999999999999,"5min":0.29199999999999998,"15min":0.28399999999999997},"max":{"1min":4.3529999999999998,"5min":33.890000000000001,"15min":33.890000000000001},"last":2.0030000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.83899999999999997,"15min":0.95799999999999996},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":8.9849999999999994},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.85699999999999998,"15min":0.88800000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.8450000000000002,"5min":11.167,"15min":11.167},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.3759999999999999,"15min":1.1850000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.33200000000000002},"max":{"1min":4.9240000000000004,"5min":31.326000000000001,"15min":31.326000000000001},"last":1.284},{"interface":"front","average":{"1min":1.125,"5min":1.363,"15min":1.208},"min":{"1min":0.40300000000000002,"5min":0.34100000000000003,"15min":0.31900000000000001},"max":{"1min":4.9009999999999998,"5min":31.234000000000002,"15min":31.234000000000002},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.90800000000000003,"15min":0.95599999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.8740000000000001,"5min":4.2770000000000001,"15min":4.9900000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.93999999999999995,"15min":0.92000000000000004},"min":{"1min":0.39200000000000002,"5min":0.35499999999999998,"15min":0.34499999999999997},"max":{"1min":1.569,"5min":5.0730000000000004,"15min":5.0730000000000004},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.131,"15min":1.167},"min":{"1min":0.48199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.1159999999999997,"5min":5.1360000000000001,"15min":5.968},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72599999999999998,"5min":1.2589999999999999,"15min":1.1930000000000001},"min":{"1min":0.46300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.6459999999999999,"5min":6.7720000000000002,"15min":6.7720000000000002},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.2090000000000001,"15min":1.075},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.0359999999999996,"5min":6.4160000000000004,"15min":6.4160000000000004},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.302,"5min":1.4730000000000001,"15min":1.2549999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":4.1449999999999996,"5min":9.5389999999999997,"15min":9.5389999999999997},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.347,"15min":1.208},"min":{"1min":0.51300000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":4.1660000000000004,"5min":7.851,"15min":7.851},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.2889999999999999,"15min":1.121},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.681,"5min":8.3000000000000007,"15min":8.3000000000000007},"last":0.72199999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.85199999999999998,"15min":2.7000000000000002},"min":{"1min":0.435,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":537.54399999999998},"last":1.012},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.006,"15min":2.8559999999999999},"min":{"1min":0.502,"5min":0.41499999999999998,"15min":0.40500000000000003},"max":{"1min":4.2089999999999996,"5min":4.9119999999999999,"15min":539.86099999999999},"last":0.64900000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.129,"15min":1.153},"min":{"1min":0.58499999999999996,"5min":0.45800000000000002,"15min":0.38600000000000001},"max":{"1min":4.7439999999999998,"5min":4.9770000000000003,"15min":5.7430000000000003},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.109,"15min":1.149},"min":{"1min":0.53900000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":4.8019999999999996,"5min":5.0129999999999999,"15min":5.359},"last":0.79500000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:35:42 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.03,"15min":2.8690000000000002},"min":{"1min":0.47999999999999998,"5min":0.439,"15min":0.439},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":536.245},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.071,"15min":2.871},"min":{"1min":0.52800000000000002,"5min":0.44,"15min":0.432},"max":{"1min":5.0730000000000004,"5min":5.0730000000000004,"15min":536.64300000000003},"last":0.68200000000000005}]}]},{"osd":10,"up_from":83,"seq":356482286031,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2286252,"kb_used_data":1972644,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":91458900,"statfs":{"total":95995035648,"available":93653913600,"internally_reserved":0,"allocated":2019987456,"data_stored":2019638218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64800000000000002,"15min":1.4199999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":2.8769999999999998,"5min":4.1769999999999996,"15min":162.38399999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.87,"15min":1.534},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.26500000000000001},"max":{"1min":3.1469999999999998,"5min":4.6180000000000003,"15min":162.01300000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.70999999999999996,"15min":0.875},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.0940000000000001,"5min":3.2850000000000001,"15min":5.9000000000000004},"last":0.438},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.93700000000000006,"15min":0.98299999999999998},"min":{"1min":0.34100000000000003,"5min":0.32800000000000001,"15min":0.309},"max":{"1min":3.3079999999999998,"5min":4.673,"15min":6.1970000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.85599999999999998,"15min":0.98699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.9700000000000002,"5min":5.016,"15min":8.3200000000000003},"last":2.4359999999999999},{"interface":"front","average":{"1min":1.123,"5min":0.93600000000000005,"15min":1.034},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.5489999999999999,"5min":5.1820000000000004,"15min":5.7220000000000004},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:35:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.181,"15min":1.1859999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.6989999999999998,"5min":3.6989999999999998,"15min":3.6989999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.0049999999999999,"15min":0.98299999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.927,"5min":2.927,"15min":2.927},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.056,"15min":1.0289999999999999},"min":{"1min":0.38800000000000001,"5min":0.371,"15min":0.33500000000000002},"max":{"1min":1.4810000000000001,"5min":6.3689999999999998,"15min":6.3689999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.96899999999999997,"15min":1.018},"min":{"1min":0.318,"5min":0.318,"15min":0.30199999999999999},"max":{"1min":1.5700000000000001,"5min":4.3959999999999999,"15min":7.3070000000000004},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.998,"15min":1.0640000000000001},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.32700000000000001},"max":{"1min":3.0190000000000001,"5min":5.0010000000000003,"15min":6.343},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.3440000000000001,"15min":1.252},"min":{"1min":0.47599999999999998,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":1.8700000000000001,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.2430000000000001,"15min":1.129},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.35499999999999998},"max":{"1min":2.964,"5min":7.7300000000000004,"15min":7.7300000000000004},"last":0.505},{"interface":"front","average":{"1min":0.87,"5min":1.2150000000000001,"15min":1.1719999999999999},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.376},"max":{"1min":1.6359999999999999,"5min":4.4889999999999999,"15min":6.8200000000000003},"last":0.59599999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.1659999999999999,"15min":1.097},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.375},"max":{"1min":3.0739999999999998,"5min":7.2610000000000001,"15min":7.3280000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.139,"5min":1.2649999999999999,"15min":1.2170000000000001},"min":{"1min":0.59099999999999997,"5min":0.434,"15min":0.374},"max":{"1min":3.9460000000000002,"5min":5.3730000000000002,"15min":6.4539999999999997},"last":0.629}]},{"osd":8,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.071,"15min":1.125},"min":{"1min":0.47299999999999998,"5min":0.44900000000000001,"15min":0.40400000000000003},"max":{"1min":1.679,"5min":4.3399999999999999,"15min":6.5300000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.83899999999999997,"5min":1.0580000000000001,"15min":1.095},"min":{"1min":0.378,"5min":0.376,"15min":0.35799999999999998},"max":{"1min":1.595,"5min":7.306,"15min":7.306},"last":0.51700000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.97999999999999998,"15min":1.1479999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":3.0489999999999999,"5min":3.0489999999999999,"15min":6.6239999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.069,"5min":1.054,"15min":1.127},"min":{"1min":0.44900000000000001,"5min":0.433,"15min":0.42899999999999999},"max":{"1min":3.3660000000000001,"5min":3.5049999999999999,"15min":7.3479999999999999},"last":0.71399999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:35:44 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2350000000000001,"15min":1.194},"min":{"1min":0.46300000000000002,"5min":0.42299999999999999,"15min":0.39400000000000002},"max":{"1min":4.2050000000000001,"5min":7.3929999999999998,"15min":7.3929999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.2190000000000001,"15min":1.2030000000000001},"min":{"1min":0.59399999999999997,"5min":0.53000000000000003,"15min":0.47899999999999998},"max":{"1min":3.198,"5min":4.3929999999999998,"15min":6.5860000000000003},"last":0.68500000000000005}]}]},{"osd":9,"up_from":74,"seq":317827580375,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2593388,"kb_used_data":2007780,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":91151764,"statfs":{"total":95995035648,"available":93339406336,"internally_reserved":0,"allocated":2055966720,"data_stored":2055617482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":3.1669999999999998,"5min":1.206,"15min":1.1890000000000001},"min":{"1min":0.313,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":51.289000000000001,"5min":51.289000000000001,"15min":51.289000000000001},"last":51.82},{"interface":"front","average":{"1min":3.1230000000000002,"5min":1.24,"15min":1.175},"min":{"1min":0.312,"5min":0.308,"15min":0.28299999999999997},"max":{"1min":51.448,"5min":51.448,"15min":51.448},"last":51.798999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":0.80100000000000005,"15min":0.97299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":5.8819999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.4730000000000001,"5min":0.93100000000000005,"15min":0.96299999999999997},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.6349999999999998,"5min":4.6349999999999998,"15min":5.7910000000000004},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:35:07 2024","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":3.125,"15min":3.125},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":52.097000000000001,"5min":52.097000000000001,"15min":52.097000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":3.6899999999999999,"5min":3.6899999999999999,"15min":3.6899999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":51.209000000000003,"5min":51.209000000000003,"15min":51.209000000000003},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:33:43 2024","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":8.8390000000000004,"15min":8.6829999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":197.661,"5min":197.661,"15min":197.661},"last":0.84499999999999997},{"interface":"front","average":{"1min":9.8330000000000002,"5min":10.432,"15min":10.532},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":197.31200000000001,"5min":197.31200000000001,"15min":197.31200000000001},"last":0.89300000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.84299999999999997,"15min":1.008},"min":{"1min":0.436,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.585,"5min":3.8900000000000001,"15min":6.3090000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.94699999999999995,"15min":1.1399999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.36899999999999999},"max":{"1min":1.6619999999999999,"5min":3.9289999999999998,"15min":6.3490000000000002},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0660000000000001,"15min":1.198},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.7389999999999999,"5min":6.8559999999999999,"15min":6.8559999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.349,"5min":1.1299999999999999,"15min":1.1719999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40300000000000002},"max":{"1min":3.1989999999999998,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.1890000000000001,"15min":1.1160000000000001},"min":{"1min":0.44400000000000001,"5min":0.39400000000000002,"15min":0.378},"max":{"1min":1.885,"5min":7.0339999999999998,"15min":7.0339999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.212,"15min":1.214},"min":{"1min":0.40000000000000002,"5min":0.379,"15min":0.36499999999999999},"max":{"1min":3.234,"5min":7.71,"15min":14.718},"last":0.81699999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.3109999999999999,"15min":1.286},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.7859999999999996,"5min":7.9489999999999998,"15min":7.9489999999999998},"last":0.871},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.4870000000000001,"15min":1.3089999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.43099999999999999},"max":{"1min":5.8979999999999997,"5min":5.8979999999999997,"15min":7.9160000000000004},"last":0.97599999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:35:12 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.143,"15min":1.1220000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34300000000000003},"max":{"1min":7.593,"5min":7.593,"15min":7.593},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.036,"15min":1.139},"min":{"1min":0.47599999999999998,"5min":0.45300000000000001,"15min":0.42599999999999999},"max":{"1min":1.736,"5min":3.6280000000000001,"15min":6.1980000000000004},"last":0.85499999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.98899999999999999,"15min":1.157},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.9950000000000001,"5min":5.8609999999999998,"15min":5.8609999999999998},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.054,"15min":1.1679999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":3.9729999999999999,"5min":3.9729999999999999,"15min":6.2560000000000002},"last":0.68300000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:34:58 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.05,"15min":1.23},"min":{"1min":0.55800000000000005,"5min":0.499,"15min":0.47599999999999998},"max":{"1min":5.944,"5min":5.944,"15min":5.944},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.1619999999999999,"15min":1.2070000000000001},"min":{"1min":0.495,"5min":0.46700000000000003,"15min":0.41599999999999998},"max":{"1min":5.5069999999999997,"5min":5.5069999999999997,"15min":5.5069999999999997},"last":0.77000000000000002}]}]},{"osd":8,"up_from":65,"seq":279172874721,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1617016,"kb_used_data":1307504,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":92128136,"statfs":{"total":95995035648,"available":94339211264,"internally_reserved":0,"allocated":1338884096,"data_stored":1338528848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":1.7490000000000001,"15min":1.1870000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.246,"5min":134.01499999999999,"15min":134.01499999999999},"last":0.376},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.891,"15min":1.3200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.3119999999999998,"5min":133.667,"15min":133.667},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0190000000000001,"15min":1.3320000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.497},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.9870000000000001,"15min":1.4079999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4580000000000002,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.70499999999999996,"15min":0.79800000000000004},"min":{"1min":0.307,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":4.0659999999999998,"5min":4.0659999999999998,"15min":4.4160000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.79100000000000004,"15min":0.92300000000000004},"min":{"1min":0.28999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.5510000000000002,"5min":3.3620000000000001,"15min":5.4260000000000002},"last":0.96399999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.70299999999999996,"15min":0.64300000000000002},"min":{"1min":0.45600000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.0289999999999999,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.92600000000000005,"15min":0.82499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.2360000000000002,"5min":4.1559999999999997,"15min":4.1559999999999997},"last":0.82199999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.753,"15min":0.876},"min":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.29599999999999999},"max":{"1min":1.3879999999999999,"5min":2.7040000000000002,"15min":5.0490000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.86699999999999999,"15min":0.92600000000000005},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":2.6930000000000001,"5min":2.762,"15min":5.0380000000000003},"last":0.90900000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.93600000000000005,"15min":0.95999999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.3560000000000001,"5min":6.4160000000000004,"15min":6.4160000000000004},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.1419999999999999,"15min":1.0720000000000001},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.7229999999999999,"5min":7.0880000000000001,"15min":7.0880000000000001},"last":0.88300000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.0840000000000001,"15min":1.0940000000000001},"min":{"1min":0.40600000000000003,"5min":0.38200000000000001,"15min":0.373},"max":{"1min":1.7649999999999999,"5min":4.6920000000000002,"15min":5.3639999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.3009999999999999,"15min":1.208},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.38},"max":{"1min":3.2719999999999998,"5min":6.915,"15min":9.1110000000000007},"last":0.93500000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":7.9119999999999999,"15min":3.3999999999999999},"min":{"1min":0.49399999999999999,"5min":0.45800000000000002,"15min":0.45400000000000001},"max":{"1min":1.581,"5min":706.67700000000002,"15min":706.67700000000002},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":8.0139999999999993,"15min":3.5110000000000001},"min":{"1min":0.47899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.8820000000000001,"5min":707.49099999999999,"15min":707.49099999999999},"last":0.85499999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.93300000000000005,"15min":1.075},"min":{"1min":0.371,"5min":0.371,"15min":0.34200000000000003},"max":{"1min":1.714,"5min":2.5699999999999998,"15min":3.8610000000000002},"last":2.5670000000000002},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0369999999999999,"15min":1.1890000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.44700000000000001},"max":{"1min":1.8180000000000001,"5min":2.6059999999999999,"15min":3.8940000000000001},"last":2.6000000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:34:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":0.91400000000000003,"15min":1.0409999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.36699999999999999},"max":{"1min":4.202,"5min":4.202,"15min":5.2560000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":0.85799999999999998,"15min":1.0369999999999999},"min":{"1min":0.48599999999999999,"5min":0.43099999999999999,"15min":0.38500000000000001},"max":{"1min":2.4430000000000001,"5min":2.46,"15min":5.0250000000000004},"last":0.65400000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:34:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":0.99399999999999999,"15min":1.1160000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":2.0630000000000002,"5min":2.306,"15min":5.2530000000000001},"last":1.244},{"interface":"front","average":{"1min":1.103,"5min":0.89300000000000002,"15min":1.034},"min":{"1min":0.53000000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.6910000000000001,"5min":2.323,"15min":5.2320000000000002},"last":1.2649999999999999}]}]},{"osd":7,"up_from":56,"seq":240518169064,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914540,"kb_used_data":1320740,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":91830612,"statfs":{"total":95995035648,"available":94034546688,"internally_reserved":0,"allocated":1352437760,"data_stored":1352088522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:57 2024","interfaces":[{"interface":"back","average":{"1min":11.169,"5min":3.1949999999999998,"15min":2.528},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":184.77199999999999,"5min":184.77199999999999,"15min":187.33699999999999},"last":0.496},{"interface":"front","average":{"1min":10.694000000000001,"5min":3.2570000000000001,"15min":2.5070000000000001},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":184.64599999999999,"5min":184.64599999999999,"15min":187.48400000000001},"last":0.373}]},{"osd":1,"last update":"Mon Feb 12 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.3360000000000001,"15min":1.4179999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":4.5789999999999997,"5min":14.106,"15min":14.106},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.0389999999999999,"15min":1.111},"min":{"1min":0.32700000000000001,"5min":0.316,"15min":0.316},"max":{"1min":2.1429999999999998,"5min":5.3220000000000001,"15min":5.3220000000000001},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:32:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.046,"15min":1.143},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.9299999999999997,"5min":5.4340000000000002,"15min":35.655000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.3089999999999999,"15min":1.3360000000000001},"min":{"1min":0.378,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.0599999999999996,"5min":7.6040000000000001,"15min":35.164000000000001},"last":0.998}]},{"osd":3,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.3460000000000001,"15min":1.133},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26100000000000001},"max":{"1min":2.1989999999999998,"5min":8.3190000000000008,"15min":8.3190000000000008},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.3999999999999999,"15min":1.1419999999999999},"min":{"1min":0.30499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.7000000000000002,"5min":12.193,"15min":12.193},"last":3.4820000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:35:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":1.254,"15min":1.135},"min":{"1min":0.307,"5min":0.28799999999999998,"15min":0.26200000000000001},"max":{"1min":0.996,"5min":10.958,"15min":10.958},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.292,"15min":1.1359999999999999},"min":{"1min":0.35999999999999999,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":0.96599999999999997,"5min":13.925000000000001,"15min":13.925000000000001},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:35:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":1.466,"15min":2.0470000000000002},"min":{"1min":0.42299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.0189999999999999,"5min":12.101000000000001,"15min":216.84299999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.556,"15min":2.1520000000000001},"min":{"1min":0.308,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":3.0179999999999998,"5min":13.805999999999999,"15min":217.02600000000001},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:35:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.5629999999999999,"15min":1.264},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":4.649,"5min":11.380000000000001,"15min":11.380000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.5209999999999999,"15min":1.371},"min":{"1min":0.38200000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.6079999999999997,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.77300000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":1.248,"15min":2.9710000000000001},"min":{"1min":0.42599999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.98,"5min":8.3659999999999997,"15min":498.68299999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.5429999999999999,"15min":3.1920000000000002},"min":{"1min":0.52900000000000003,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":1.327,"5min":14.021000000000001,"15min":498.69999999999999},"last":0.66700000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.3129999999999999,"15min":1.1759999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.4289999999999998,"5min":8.4459999999999997,"15min":8.4459999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.353,"15min":1.3120000000000001},"min":{"1min":0.51400000000000001,"5min":0.41399999999999998,"15min":0.40100000000000002},"max":{"1min":3.3919999999999999,"5min":6.1379999999999999,"15min":7.1219999999999999},"last":0.84099999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:35:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.3149999999999999,"15min":3.504},"min":{"1min":0.497,"5min":0.40899999999999997,"15min":0.376},"max":{"1min":2.968,"5min":5.5179999999999998,"15min":497.52699999999999},"last":2.121},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.4410000000000001,"15min":3.585},"min":{"1min":0.51800000000000002,"5min":0.44900000000000001,"15min":0.39200000000000002},"max":{"1min":3.9289999999999998,"5min":13.98,"15min":496.29599999999999},"last":0.60899999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.361,"15min":1.25},"min":{"1min":0.52200000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.0390000000000001,"5min":8.4199999999999999,"15min":8.4199999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.024,"5min":1.425,"15min":1.371},"min":{"1min":0.48299999999999998,"5min":0.47499999999999998,"15min":0.46899999999999997},"max":{"1min":1.915,"5min":5.5830000000000002,"15min":7.0679999999999996},"last":0.91500000000000004}]}]},{"osd":6,"up_from":48,"seq":206158430704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2589488,"kb_used_data":2003880,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":91155664,"statfs":{"total":95995035648,"available":93343399936,"internally_reserved":0,"allocated":2051973120,"data_stored":2051619810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":2.1539999999999999,"15min":1.4470000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7350000000000001,"5min":107.723,"15min":107.723},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":2.23,"15min":1.5349999999999999},"min":{"1min":0.38300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.1180000000000003,"5min":107.114,"15min":107.114},"last":0.92100000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.98399999999999999,"15min":1.0149999999999999},"min":{"1min":0.29199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.1099999999999999,"5min":5.6479999999999997,"15min":6.1870000000000003},"last":0.443},{"interface":"front","average":{"1min":1.115,"5min":1.272,"15min":1.087},"min":{"1min":0.312,"5min":0.26100000000000001,"15min":0.25700000000000001},"max":{"1min":5.468,"5min":6.0910000000000002,"15min":6.6959999999999997},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.2969999999999999,"15min":1.252},"min":{"1min":0.36599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.859,"5min":7.8769999999999998,"15min":28.739999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.272,"15min":1.319},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.0249999999999999,"5min":8.5129999999999999,"15min":32.328000000000003},"last":0.625}]},{"osd":3,"last update":"Mon Feb 12 16:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":5.7720000000000002,"15min":3.8260000000000001},"min":{"1min":0.28699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.3040000000000003,"5min":365.58999999999997,"15min":365.58999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.111,"5min":6.3630000000000004,"15min":4.7590000000000003},"min":{"1min":0.438,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.9980000000000002,"5min":366.93299999999999,"15min":366.93299999999999},"last":0.81000000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.2509999999999999,"15min":1.1779999999999999},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":2.7930000000000001,"5min":8.4559999999999995,"15min":8.4559999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.1890000000000001,"15min":1.109},"min":{"1min":0.35599999999999998,"5min":0.29999999999999999,"15min":0.23999999999999999},"max":{"1min":2.6000000000000001,"5min":7.7610000000000001,"15min":7.7610000000000001},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.4179999999999999,"15min":1.2010000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.9180000000000001,"5min":8.5730000000000004,"15min":8.5730000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.877,"5min":1.347,"15min":1.1499999999999999},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.33100000000000002},"max":{"1min":2.6379999999999999,"5min":7.835,"15min":10.52},"last":0.47199999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:35:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.411,"15min":1.2310000000000001},"min":{"1min":0.41999999999999998,"5min":0.29599999999999999,"15min":0.29499999999999998},"max":{"1min":4.0670000000000002,"5min":9.1199999999999992,"15min":9.1199999999999992},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":1.4279999999999999,"15min":1.198},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32200000000000001},"max":{"1min":3.012,"5min":8.8879999999999999,"15min":8.8879999999999999},"last":0.54900000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:35:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":6.1219999999999999,"15min":2.827},"min":{"1min":0.47499999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.863,"5min":501.892,"15min":501.892},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":6.1479999999999997,"15min":2.8220000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.371},"max":{"1min":3.0030000000000001,"5min":501.76600000000002,"15min":501.76600000000002},"last":0.68100000000000005}]},{"osd":9,"last update":"Mon Feb 12 16:35:00 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.3169999999999999,"15min":1.214},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8160000000000001,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.698,"5min":1.5229999999999999,"15min":1.2889999999999999},"min":{"1min":0.49399999999999999,"5min":0.46100000000000002,"15min":0.42799999999999999},"max":{"1min":4.327,"5min":7.6870000000000003,"15min":7.6870000000000003},"last":0.748}]},{"osd":10,"last update":"Mon Feb 12 16:35:00 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.599,"15min":1.385},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":4.0839999999999996,"5min":8.6159999999999997,"15min":8.6159999999999997},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.577,"5min":1.653,"15min":1.4690000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":4.1280000000000001,"5min":8.7100000000000009,"15min":8.7100000000000009},"last":0.75600000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:35:00 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.752,"15min":1.4419999999999999},"min":{"1min":0.51200000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":4.375,"5min":8.6649999999999991,"15min":8.6649999999999991},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.252,"15min":1.226},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":2.0579999999999998,"5min":6.25,"15min":6.25},"last":0.80100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2225584,"kb_used_data":1331368,"kb_used_omap":0,"kb_used_meta":894208,"kb_avail":91519568,"statfs":{"total":95995035648,"available":93716037632,"internally_reserved":0,"allocated":1363320832,"data_stored":1362967522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915668992},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":0.93200000000000005,"15min":0.97399999999999998},"min":{"1min":0.33500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":10.420999999999999,"5min":10.420999999999999,"15min":16.632999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.256,"5min":1.1830000000000001,"15min":1.2450000000000001},"min":{"1min":0.30199999999999999,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":10.26,"5min":10.26,"15min":10.26},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.1950000000000001,"15min":1.147},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.755000000000001,"5min":10.755000000000001,"15min":12.778},"last":7.5369999999999999},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.1919999999999999,"15min":1.248},"min":{"1min":0.308,"5min":0.249,"15min":0.246},"max":{"1min":7.3600000000000003,"5min":9.702,"15min":9.9689999999999994},"last":6.9850000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:35:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.3280000000000001,"15min":1.1890000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.248},"max":{"1min":3.2610000000000001,"5min":26.664000000000001,"15min":26.664000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.5820000000000001,"15min":1.25},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.3260000000000001,"5min":27.148,"15min":27.148},"last":2.3530000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":1.034,"15min":1.1599999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.31},"max":{"1min":2.516,"5min":7.0720000000000001,"15min":8.7479999999999993},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":1.1319999999999999,"15min":1.2729999999999999},"min":{"1min":0.35499999999999998,"5min":0.312,"15min":0.30199999999999999},"max":{"1min":2.5350000000000001,"5min":6.9119999999999999,"15min":7.1710000000000003},"last":0.435}]},{"osd":5,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":6.8010000000000002,"15min":4.1120000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.165,"5min":524.85299999999995,"15min":524.85299999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":6.6509999999999998,"15min":4.0499999999999998},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.32800000000000001},"max":{"1min":2.468,"5min":519.89700000000005,"15min":519.89700000000005},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":6.9219999999999997,"15min":3.254},"min":{"1min":0.39200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.371,"5min":522.79200000000003,"15min":522.79200000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.173,"5min":6.7190000000000003,"15min":3.2010000000000001},"min":{"1min":0.44500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.3940000000000001,"5min":520.64499999999998,"15min":520.64499999999998},"last":0.70099999999999996}]},{"osd":8,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.1439999999999999,"15min":3.355},"min":{"1min":0.52500000000000002,"5min":0.41699999999999998,"15min":0.39500000000000002},"max":{"1min":3.593,"5min":7.0309999999999997,"15min":548.91600000000005},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.446,"5min":1.1990000000000001,"15min":3.3700000000000001},"min":{"1min":0.57499999999999996,"5min":0.42499999999999999,"15min":0.36599999999999999},"max":{"1min":3.5550000000000002,"5min":7.0730000000000004,"15min":552.42100000000005},"last":0.752}]},{"osd":9,"last update":"Mon Feb 12 16:35:07 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.131,"15min":1.3140000000000001},"min":{"1min":0.44700000000000001,"5min":0.42599999999999999,"15min":0.42099999999999999},"max":{"1min":4.1210000000000004,"5min":7.444,"15min":7.444},"last":0.498},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2210000000000001,"15min":1.339},"min":{"1min":0.53600000000000003,"5min":0.49099999999999999,"15min":0.436},"max":{"1min":4.2960000000000003,"5min":7.4870000000000001,"15min":10.106},"last":0.63300000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:34:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1930000000000001,"15min":3.7080000000000002},"min":{"1min":0.49399999999999999,"5min":0.46200000000000002,"15min":0.39500000000000002},"max":{"1min":4.258,"5min":7.3620000000000001,"15min":548.93499999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.1679999999999999,"15min":3.7029999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.46999999999999997},"max":{"1min":3.504,"5min":7.1260000000000003,"15min":549.12099999999998},"last":0.64400000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.2010000000000001,"15min":1.3899999999999999},"min":{"1min":0.54400000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.5800000000000001,"5min":7.1669999999999998,"15min":10.202},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.2070000000000001,"15min":1.411},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.44900000000000001},"max":{"1min":4.2140000000000004,"5min":7.4029999999999996,"15min":7.4029999999999996},"last":0.54600000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738914,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3554284,"kb_used_data":2659172,"kb_used_omap":0,"kb_used_meta":895104,"kb_avail":90190868,"statfs":{"total":95995035648,"available":92355448832,"internally_reserved":0,"allocated":2722992128,"data_stored":2722642890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":916586496},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.83899999999999997,"15min":0.91900000000000004},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23699999999999999},"max":{"1min":1.7170000000000001,"5min":6.0030000000000001,"15min":10.754},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.022,"5min":1.0029999999999999,"15min":1.0820000000000001},"min":{"1min":0.29899999999999999,"5min":0.23400000000000001,"15min":0.22600000000000001},"max":{"1min":4.1849999999999996,"5min":4.7939999999999996,"15min":5.5810000000000004},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:34:56 2024","interfaces":[{"interface":"back","average":{"1min":18.202000000000002,"5min":4.8760000000000003,"15min":2.3170000000000002},"min":{"1min":0.28399999999999997,"5min":0.254,"15min":0.254},"max":{"1min":377.06299999999999,"5min":377.06299999999999,"15min":377.06299999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":18.343,"5min":4.8310000000000004,"15min":2.3039999999999998},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.224},"max":{"1min":379.18000000000001,"5min":379.18000000000001,"15min":379.18000000000001},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":1.1339999999999999,"15min":1.9239999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.6890000000000001,"5min":5.6859999999999999,"15min":193.77000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.97699999999999998,"15min":1.8879999999999999},"min":{"1min":0.34399999999999997,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":4.3230000000000004,"5min":5.0659999999999998,"15min":189.625},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:35:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":3.1499999999999999,"15min":1.8120000000000001},"min":{"1min":0.311,"5min":0.30399999999999999,"15min":0.28699999999999998},"max":{"1min":4.5670000000000002,"5min":181.46199999999999,"15min":181.46199999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.4610000000000001,"5min":3.218,"15min":1.8779999999999999},"min":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.718,"5min":186.41999999999999,"15min":186.41999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3819999999999999,"15min":1.9330000000000001},"min":{"1min":0.379,"5min":0.371,"15min":0.311},"max":{"1min":4.5919999999999996,"5min":7.8220000000000001,"15min":189.79400000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.276,"5min":1.427,"15min":1.9770000000000001},"min":{"1min":0.45100000000000001,"5min":0.38,"15min":0.35499999999999998},"max":{"1min":4.6280000000000001,"5min":5.907,"15min":194.41200000000001},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":3.2839999999999998,"15min":2.73},"min":{"1min":0.44500000000000001,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":4.7850000000000001,"5min":184.39400000000001,"15min":191.47200000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.458,"5min":3.3730000000000002,"15min":2.8300000000000001},"min":{"1min":0.49199999999999999,"5min":0.432,"15min":0.432},"max":{"1min":4.7359999999999998,"5min":184.43700000000001,"15min":191.50299999999999},"last":0.63500000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:35:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.29,"15min":1.3220000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.7619999999999996,"5min":5.9379999999999997,"15min":7.3920000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.621,"5min":1.373,"15min":1.4179999999999999},"min":{"1min":0.51200000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":4.8079999999999998,"5min":6.0209999999999999,"15min":7.5060000000000002},"last":0.66800000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:35:17 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.1279999999999999,"15min":1.1819999999999999},"min":{"1min":0.47199999999999998,"5min":0.371,"15min":0.35699999999999998},"max":{"1min":8.4629999999999992,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.363,"5min":1.1950000000000001,"15min":1.3520000000000001},"min":{"1min":0.54000000000000004,"5min":0.45300000000000001,"15min":0.432},"max":{"1min":4.6699999999999999,"5min":5.9640000000000004,"15min":7.5369999999999999},"last":0.57599999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.113,"15min":1.3109999999999999},"min":{"1min":0.53000000000000003,"5min":0.46100000000000002,"15min":0.442},"max":{"1min":2.0430000000000001,"5min":4.774,"15min":6.3559999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.147,"5min":1.222,"15min":1.421},"min":{"1min":0.59999999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":2.069,"5min":8.4280000000000008,"15min":9.3490000000000002},"last":0.72599999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:35:39 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.45,"15min":1.4179999999999999},"min":{"1min":0.51600000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":8.4280000000000008,"5min":8.4280000000000008,"15min":8.4280000000000008},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.093,"5min":1.2030000000000001,"15min":1.3600000000000001},"min":{"1min":0.52000000000000002,"5min":0.47299999999999998,"15min":0.45200000000000001},"max":{"1min":4.0129999999999999,"5min":5.9930000000000003,"15min":7.4210000000000003},"last":0.68600000000000005}]}]},{"osd":2,"up_from":21,"seq":90194313744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2973420,"kb_used_data":2659812,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":90771732,"statfs":{"total":95995035648,"available":92950253568,"internally_reserved":0,"allocated":2723647488,"data_stored":2723298250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":0.83699999999999997,"15min":1.008},"min":{"1min":0.252,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":7.6050000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.4550000000000001,"5min":0.95599999999999996,"15min":1.1220000000000001},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.247},"max":{"1min":7.641,"5min":7.641,"15min":9.6470000000000002},"last":1.2290000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:35:27 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.6499999999999999,"15min":1.6779999999999999},"min":{"1min":0.31,"5min":0.28399999999999997,"15min":0.253},"max":{"1min":7.1040000000000001,"5min":54.091000000000001,"15min":54.091000000000001},"last":1.111},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.651,"15min":1.2729999999999999},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":5.3399999999999999,"5min":54.203000000000003,"15min":54.203000000000003},"last":1.137}]},{"osd":4,"last update":"Mon Feb 12 16:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":0.89600000000000002,"15min":1.089},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.28799999999999998},"max":{"1min":7.194,"5min":7.194,"15min":10.890000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.119,"5min":0.98999999999999999,"15min":1.2010000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":3.7189999999999999,"5min":7.0019999999999998,"15min":9.8079999999999998},"last":0.95799999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":0.95899999999999996,"15min":1.133},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":6.7229999999999999,"5min":6.7229999999999999,"15min":6.7229999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.119,"15min":1.2729999999999999},"min":{"1min":0.38500000000000001,"5min":0.31,"15min":0.31},"max":{"1min":5.8209999999999997,"5min":7.0590000000000002,"15min":11.081},"last":1.1950000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.198,"15min":1.3340000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":7.157,"5min":7.157,"15min":21.527000000000001},"last":1.302},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.1890000000000001,"15min":1.2509999999999999},"min":{"1min":0.45100000000000001,"5min":0.40200000000000002,"15min":0.35099999999999998},"max":{"1min":3.9100000000000001,"5min":7.931,"15min":7.931},"last":1.256}]},{"osd":7,"last update":"Mon Feb 12 16:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.115,"15min":1.28},"min":{"1min":0.34799999999999998,"5min":0.314,"15min":0.314},"max":{"1min":3.774,"5min":7.1189999999999998,"15min":11.044},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.0629999999999999,"15min":1.2529999999999999},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":3.8759999999999999,"5min":4.867,"15min":6.2050000000000001},"last":1.417}]},{"osd":8,"last update":"Mon Feb 12 16:35:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":0.96799999999999997,"15min":1.234},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.4950000000000001,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":0.89900000000000002,"15min":1.2350000000000001},"min":{"1min":0.498,"5min":0.377,"15min":0.377},"max":{"1min":3.9390000000000001,"5min":3.9390000000000001,"15min":9.7639999999999993},"last":1.7569999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:35:04 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.97799999999999998,"15min":1.2769999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.8170000000000002,"5min":7.1689999999999996,"15min":10.972},"last":1.282},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.044,"15min":1.3720000000000001},"min":{"1min":0.53900000000000003,"5min":0.45100000000000001,"15min":0.442},"max":{"1min":3.847,"5min":7.2110000000000003,"15min":11.006},"last":1.4970000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.0469999999999999,"15min":1.341},"min":{"1min":0.433,"5min":0.433,"15min":0.41399999999999998},"max":{"1min":7.5709999999999997,"5min":7.5709999999999997,"15min":11.454000000000001},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.1100000000000001,"15min":1.389},"min":{"1min":0.45800000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.5380000000000003,"5min":7.5380000000000003,"15min":9.7170000000000005},"last":1.5549999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:35:08 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.93899999999999995,"15min":1.256},"min":{"1min":0.52200000000000002,"5min":0.52100000000000002,"15min":0.47799999999999998},"max":{"1min":3.673,"5min":3.8119999999999998,"15min":7.258},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.0209999999999999,"15min":1.345},"min":{"1min":0.54000000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.6459999999999999,"5min":5.1769999999999996,"15min":7.3040000000000003},"last":1.643}]}]},{"osd":3,"up_from":522,"seq":2241972928668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2344876,"kb_used_data":2004452,"kb_used_omap":91,"kb_used_meta":340324,"kb_avail":91400276,"statfs":{"total":95995035648,"available":93593882624,"internally_reserved":0,"allocated":2052558848,"data_stored":2052209610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":348491792},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.052,"15min":1.0740000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.1419999999999995,"5min":8.1419999999999995,"15min":34.055999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.224,"5min":1.0209999999999999,"15min":1.103},"min":{"1min":0.29499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.1559999999999997,"5min":10.436999999999999,"15min":27.600000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":0.85299999999999998,"15min":0.98399999999999999},"min":{"1min":0.28399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":8.8109999999999999},"last":5.7549999999999999},{"interface":"front","average":{"1min":1.151,"5min":0.85999999999999999,"15min":1.026},"min":{"1min":0.27800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.9500000000000002,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":10.369999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.0600000000000001,"15min":1.0700000000000001},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.2380000000000004,"5min":5.2809999999999997,"15min":10.004},"last":5.8719999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.1200000000000001,"15min":1.1080000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.2650000000000001,"5min":6.1829999999999998,"15min":6.806},"last":0.35299999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.2170000000000001,"15min":1.196},"min":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":11.884,"5min":11.884,"15min":11.884},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.367,"15min":1.107},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":12.164999999999999,"5min":12.164999999999999,"15min":12.164999999999999},"last":6.0010000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2669999999999999,"15min":1.167},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.2359999999999998,"5min":5.7309999999999999,"15min":8.3320000000000007},"last":1.756},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.2609999999999999,"15min":1.1180000000000001},"min":{"1min":0.40200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.4100000000000001,"5min":6.8890000000000002,"15min":6.8890000000000002},"last":9.4640000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.2749999999999999,"15min":1.252},"min":{"1min":0.36199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.3419999999999996,"5min":6.7409999999999997,"15min":6.851},"last":5.9260000000000002},{"interface":"front","average":{"1min":1.171,"5min":1.4199999999999999,"15min":1.3049999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":3.3039999999999998,"5min":7.2249999999999996,"15min":7.2249999999999996},"last":0.40999999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.131,"15min":1.133},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.3730000000000002,"5min":5.3220000000000001,"15min":5.3220000000000001},"last":5.9669999999999996},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.204,"15min":1.125},"min":{"1min":0.44400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.3620000000000001,"5min":6.2610000000000001,"15min":6.2610000000000001},"last":0.44500000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2,"15min":1.2789999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":3.419,"5min":6.218,"15min":6.218},"last":2.7149999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.169,"15min":1.278},"min":{"1min":0.48799999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":4.2649999999999997,"5min":4.7320000000000002,"15min":5.0780000000000003},"last":0.55300000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2130000000000001,"15min":1.294},"min":{"1min":0.53000000000000003,"5min":0.47099999999999997,"15min":0.46000000000000002},"max":{"1min":4.2990000000000004,"5min":4.7720000000000002,"15min":4.8499999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2410000000000001,"15min":1.343},"min":{"1min":0.495,"5min":0.49199999999999999,"15min":0.47099999999999997},"max":{"1min":3.4319999999999999,"5min":6.2939999999999996,"15min":15.140000000000001},"last":2.6749999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":4.7030000000000003,"5min":1.9470000000000001,"15min":1.512},"min":{"1min":0.45700000000000002,"5min":0.437,"15min":0.437},"max":{"1min":73.037999999999997,"5min":73.037999999999997,"15min":73.037999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":4.6120000000000001,"5min":1.895,"15min":1.5620000000000001},"min":{"1min":0.53500000000000003,"5min":0.53400000000000003,"15min":0.49299999999999999},"max":{"1min":73.057000000000002,"5min":73.057000000000002,"15min":73.057000000000002},"last":0.56699999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953984,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1900024,"kb_used_data":1306224,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":91845128,"statfs":{"total":95995035648,"available":94049411072,"internally_reserved":0,"allocated":1337573376,"data_stored":1337218128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.98199999999999998,"15min":1.417},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.25700000000000001},"max":{"1min":1.2829999999999999,"5min":3.992,"15min":96.052000000000007},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.98799999999999999,"15min":1.5289999999999999},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.24099999999999999},"max":{"1min":1.163,"5min":3.7400000000000002,"15min":96.307000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":4.0350000000000001,"5min":1.6599999999999999,"15min":1.246},"min":{"1min":0.27300000000000002,"5min":0.248,"15min":0.23699999999999999},"max":{"1min":56.188000000000002,"5min":56.188000000000002,"15min":56.188000000000002},"last":4.9320000000000004},{"interface":"front","average":{"1min":4.1529999999999996,"5min":1.704,"15min":1.3380000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":55.414000000000001,"5min":55.414000000000001,"15min":55.414000000000001},"last":2.577}]},{"osd":2,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.101,"15min":1.0580000000000001},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.24399999999999999},"max":{"1min":4.0090000000000003,"5min":5.3920000000000003,"15min":7.3630000000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":1.129,"15min":1.1240000000000001},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":1.8859999999999999,"5min":5.2069999999999999,"15min":7.633},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.448,"15min":1.228},"min":{"1min":0.40200000000000002,"5min":0.33400000000000002,"15min":0.315},"max":{"1min":3.6110000000000002,"5min":8.5800000000000001,"15min":12.409000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.3620000000000001,"15min":1.1599999999999999},"min":{"1min":0.35099999999999998,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":2.9550000000000001,"5min":6.5860000000000003,"15min":7.383},"last":0.89800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":1.3999999999999999,"15min":1.1499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":2.004,"5min":10.289999999999999,"15min":10.289999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.5069999999999999,"15min":1.3049999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.04,"5min":10.321999999999999,"15min":10.321999999999999},"last":0.435}]},{"osd":6,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.2649999999999999,"15min":1.083},"min":{"1min":0.32400000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":1.8600000000000001,"5min":6.5270000000000001,"15min":7.5289999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.2629999999999999,"15min":1.173},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":2.871,"5min":7.0910000000000002,"15min":9.6639999999999997},"last":0.42199999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.1499999999999999,"15min":1.0860000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.5139999999999998,"5min":5.8099999999999996,"15min":6.8659999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.7849999999999999,"15min":1.302},"min":{"1min":0.38700000000000001,"5min":0.35599999999999998,"15min":0.34599999999999997},"max":{"1min":1.7270000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"last":0.63}]},{"osd":8,"last update":"Mon Feb 12 16:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.2,"15min":1.1950000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.38300000000000001},"max":{"1min":1.4350000000000001,"5min":7.5179999999999998,"15min":7.6890000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.224,"15min":1.256},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.2749999999999999,"5min":5.5259999999999998,"15min":7.7110000000000003},"last":0.47499999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:35:05 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.3460000000000001,"15min":1.26},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.0190000000000001,"5min":7.4470000000000001,"15min":7.7750000000000004},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.01,"5min":1.294,"15min":1.3580000000000001},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.36699999999999999},"max":{"1min":1.9259999999999999,"5min":5.6470000000000002,"15min":7.827},"last":0.52800000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:35:08 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.3,"15min":1.383},"min":{"1min":0.46899999999999997,"5min":0.44,"15min":0.39200000000000002},"max":{"1min":1.6459999999999999,"5min":5.6109999999999998,"15min":7.7619999999999996},"last":0.503},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1499999999999999,"15min":1.1819999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41199999999999998},"max":{"1min":2.9100000000000001,"5min":5.6459999999999999,"15min":5.6459999999999999},"last":0.57599999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.375,"15min":1.3069999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.45400000000000001},"max":{"1min":1.4930000000000001,"5min":7.4880000000000004,"15min":7.734},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":1.339,"15min":1.391},"min":{"1min":0.51400000000000001,"5min":0.48899999999999999,"15min":0.46400000000000002},"max":{"1min":1.589,"5min":5.681,"15min":7.7969999999999997},"last":0.85799999999999998}]}]},{"osd":5,"up_from":39,"seq":167503725047,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1928940,"kb_used_data":1335140,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":91816212,"statfs":{"total":95995035648,"available":94019801088,"internally_reserved":0,"allocated":1367183360,"data_stored":1366834122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:35:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0219999999999998,"5min":2.0099999999999998,"15min":1.841},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":38.366,"5min":38.366,"15min":38.366},"last":0.39400000000000002},{"interface":"front","average":{"1min":3.2090000000000001,"5min":1.9870000000000001,"15min":1.7829999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":37.310000000000002,"5min":37.310000000000002,"15min":37.310000000000002},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.03,"15min":1.7190000000000001},"min":{"1min":0.33400000000000002,"5min":0.245,"15min":0.245},"max":{"1min":2.7229999999999999,"5min":6.6870000000000003,"15min":221.96600000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.1639999999999999,"15min":1.837},"min":{"1min":0.40500000000000003,"5min":0.36599999999999999,"15min":0.311},"max":{"1min":2.7599999999999998,"5min":6.774,"15min":222.06200000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.21,"15min":1.2509999999999999},"min":{"1min":0.51100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.8730000000000002,"5min":6.6459999999999999,"15min":40.021999999999998},"last":0.623},{"interface":"front","average":{"1min":1.806,"5min":1.3580000000000001,"15min":1.3160000000000001},"min":{"1min":0.34300000000000003,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":40.225000000000001},"last":0.56999999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:35:41 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.5389999999999999,"15min":1.4570000000000001},"min":{"1min":0.35999999999999999,"5min":0.33900000000000002,"15min":0.27200000000000002},"max":{"1min":3.9809999999999999,"5min":19.067,"15min":22.484000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.427,"15min":1.141},"min":{"1min":0.378,"5min":0.308,"15min":0.29299999999999998},"max":{"1min":2.4660000000000002,"5min":18.216000000000001,"15min":23.34},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.095,"15min":1.1910000000000001},"min":{"1min":0.32400000000000001,"5min":0.307,"15min":0.23200000000000001},"max":{"1min":0.73999999999999999,"5min":6.5999999999999996,"15min":41.908000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.1659999999999999,"15min":1.294},"min":{"1min":0.377,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.5579999999999998,"5min":5.2599999999999998,"15min":42.430999999999997},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":1.1459999999999999,"15min":1.103},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.7569999999999999,"5min":6.9829999999999997,"15min":6.9829999999999997},"last":3.3540000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.3939999999999999,"15min":1.23},"min":{"1min":0.40500000000000003,"5min":0.33100000000000002,"15min":0.27100000000000002},"max":{"1min":3.29,"5min":5.3810000000000002,"15min":9.5830000000000002},"last":3.3929999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.3520000000000001,"15min":1.1930000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.28799999999999998},"max":{"1min":0.81299999999999994,"5min":9.4179999999999993,"15min":9.7710000000000008},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.4159999999999999,"15min":1.2709999999999999},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.34300000000000003},"max":{"1min":3.2240000000000002,"5min":6.7599999999999998,"15min":6.7599999999999998},"last":0.72599999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:32:45 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":2.3969999999999998,"15min":1.6140000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":6.7300000000000004,"5min":108.913,"15min":108.913},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.169,"5min":2.1309999999999998,"15min":1.577},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1550000000000002,"5min":108.93600000000001,"15min":108.93600000000001},"last":1.8009999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":1.266,"15min":1.347},"min":{"1min":0.44,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":2.0819999999999999,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.448,"15min":1.345},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.375},"max":{"1min":2.5099999999999998,"5min":7.9710000000000001,"15min":9.6649999999999991},"last":0.73999999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:35:52 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.393,"15min":1.486},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.41099999999999998},"max":{"1min":3.4569999999999999,"5min":8.0299999999999994,"15min":55.518999999999998},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.3999999999999999,"15min":1.5720000000000001},"min":{"1min":0.55200000000000005,"5min":0.49299999999999999,"15min":0.47999999999999998},"max":{"1min":3.7189999999999999,"5min":6.867,"15min":55.497999999999998},"last":0.94899999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.5409999999999999,"15min":1.421},"min":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.38300000000000001},"max":{"1min":2.2690000000000001,"5min":8.0640000000000001,"15min":8.6940000000000008},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.492,"15min":1.4670000000000001},"min":{"1min":0.59699999999999998,"5min":0.53100000000000003,"15min":0.51500000000000001},"max":{"1min":2.0569999999999999,"5min":6.9029999999999996,"15min":6.9029999999999996},"last":0.68500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2713321472,"data_stored":2713321472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1359482880,"data_stored":1359478808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2716467200,"data_stored":2716467200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2043346944,"data_stored":2043346944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1329463296,"data_stored":1329463296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1361772544,"data_stored":1361772544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2045513728,"data_stored":2045509656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1343750144,"data_stored":1343750144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1330184192,"data_stored":1330184192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2039742464,"data_stored":2039742464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2008743936,"data_stored":2008743936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1320288256,"data_stored":1320288256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:36:01.696 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T16:36:01.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:36:01.697 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-12T16:36:01.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight-by-utilization 110 1.0 2024-02-12T16:36:01.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:01.750 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:01.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:01.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:01.841 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:36:01.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:01.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:01.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:02.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:02.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:02.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:02.240 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:02.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:02.245 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:02 smithi121 bash[18489]: cluster 2024-02-12T16:36:01.820291+0000 mon.a (mon.0) 2685 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:36:02.245 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:02 smithi121 bash[18489]: audit 2024-02-12T16:36:01.829561+0000 mon.a (mon.0) 2686 : audit [INF] from='client.? v1:172.21.15.121:0/2224641037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:36:02.245 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:02 smithi121 bash[18489]: cluster 2024-02-12T16:36:01.829617+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e790: 12 total, 12 up, 12 in 2024-02-12T16:36:02.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:02.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:02.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:02.487 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:02 smithi123 bash[19119]: cluster 2024-02-12T16:36:01.820291+0000 mon.a (mon.0) 2685 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:36:02.487 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:02 smithi123 bash[19119]: audit 2024-02-12T16:36:01.829561+0000 mon.a (mon.0) 2686 : audit [INF] from='client.? v1:172.21.15.121:0/2224641037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:36:02.487 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:02 smithi123 bash[19119]: cluster 2024-02-12T16:36:01.829617+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e790: 12 total, 12 up, 12 in 2024-02-12T16:36:02.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:02.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:02 smithi169 bash[19604]: cluster 2024-02-12T16:36:01.820291+0000 mon.a (mon.0) 2685 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:36:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:02 smithi169 bash[19604]: audit 2024-02-12T16:36:01.829561+0000 mon.a (mon.0) 2686 : audit [INF] from='client.? v1:172.21.15.121:0/2224641037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:36:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:02 smithi169 bash[19604]: cluster 2024-02-12T16:36:01.829617+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e790: 12 total, 12 up, 12 in 2024-02-12T16:36:02.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:02.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:02.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:02.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:02.784 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:02.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:02.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:02.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:02.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:03.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:03.045 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:03.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:03.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:03.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:03.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:03.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:03 smithi121 bash[18489]: cluster 2024-02-12T16:36:02.131734+0000 mgr.y (mgr.14146) 1544 : cluster [DBG] pgmap v2212: 47 pgs: 47 active+clean; 9.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 19 MiB/s wr, 1.21k op/s 2024-02-12T16:36:03.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:03.452 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:03 smithi169 bash[19604]: cluster 2024-02-12T16:36:02.131734+0000 mgr.y (mgr.14146) 1544 : cluster [DBG] pgmap v2212: 47 pgs: 47 active+clean; 9.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 19 MiB/s wr, 1.21k op/s 2024-02-12T16:36:03.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:03.563 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:03 smithi123 bash[19119]: cluster 2024-02-12T16:36:02.131734+0000 mgr.y (mgr.14146) 1544 : cluster [DBG] pgmap v2212: 47 pgs: 47 active+clean; 9.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 19 MiB/s wr, 1.21k op/s 2024-02-12T16:36:03.580 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:03.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:03.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:03.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:03.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:03.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:03.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:04.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:36:04.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:04.127 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:04.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:04.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:04.261 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:04.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:04.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:04 smithi123 bash[19119]: cluster 2024-02-12T16:36:02.697966+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:04 smithi123 bash[19119]: cluster 2024-02-12T16:36:02.698005+0000 osd.0 (osd.0) 137 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:04 smithi123 bash[19119]: cluster 2024-02-12T16:36:03.199916+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e791: 12 total, 12 up, 12 in 2024-02-12T16:36:04.393 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:04.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:04.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:04 smithi121 bash[18489]: cluster 2024-02-12T16:36:02.697966+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:04.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:04 smithi121 bash[18489]: cluster 2024-02-12T16:36:02.698005+0000 osd.0 (osd.0) 137 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:04 smithi121 bash[18489]: cluster 2024-02-12T16:36:03.199916+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e791: 12 total, 12 up, 12 in 2024-02-12T16:36:04.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:04.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:04.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:04 smithi169 bash[19604]: cluster 2024-02-12T16:36:02.697966+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:04 smithi169 bash[19604]: cluster 2024-02-12T16:36:02.698005+0000 osd.0 (osd.0) 137 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:04 smithi169 bash[19604]: cluster 2024-02-12T16:36:03.199916+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e791: 12 total, 12 up, 12 in 2024-02-12T16:36:04.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:04.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:04.765 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:04.791 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:04.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:04.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:36:04.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:04.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:04.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:05.019 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:05.035 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:36:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:05.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:05.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:05.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:05.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:05.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:05.434 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:05.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:05.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:05 smithi121 bash[18489]: cluster 2024-02-12T16:36:04.132195+0000 mgr.y (mgr.14146) 1545 : cluster [DBG] pgmap v2214: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-02-12T16:36:05.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:05 smithi121 bash[18489]: cluster 2024-02-12T16:36:04.604889+0000 osd.8 (osd.8) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:05.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:05 smithi121 bash[18489]: cluster 2024-02-12T16:36:04.604941+0000 osd.8 (osd.8) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:05 smithi169 bash[19604]: cluster 2024-02-12T16:36:04.132195+0000 mgr.y (mgr.14146) 1545 : cluster [DBG] pgmap v2214: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-02-12T16:36:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:05 smithi169 bash[19604]: cluster 2024-02-12T16:36:04.604889+0000 osd.8 (osd.8) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:05 smithi169 bash[19604]: cluster 2024-02-12T16:36:04.604941+0000 osd.8 (osd.8) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:05.583 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:05 smithi123 bash[19119]: cluster 2024-02-12T16:36:04.132195+0000 mgr.y (mgr.14146) 1545 : cluster [DBG] pgmap v2214: 47 pgs: 47 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s 2024-02-12T16:36:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:05 smithi123 bash[19119]: cluster 2024-02-12T16:36:04.604889+0000 osd.8 (osd.8) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:05.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:05 smithi123 bash[19119]: cluster 2024-02-12T16:36:04.604941+0000 osd.8 (osd.8) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:05.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:05.715 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:05.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:05.857 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:05.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:05.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:05.989 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:05.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:06.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:06.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:06.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:06.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:06.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:06.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:06.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:06.402 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:06.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:06.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:06.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:06.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:06.602 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:06.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:06.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:06 smithi123 bash[19119]: cluster 2024-02-12T16:36:05.350424+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e792: 12 total, 12 up, 12 in 2024-02-12T16:36:06.641 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:06 smithi121 bash[18489]: cluster 2024-02-12T16:36:05.350424+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e792: 12 total, 12 up, 12 in 2024-02-12T16:36:06.670 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:06 smithi169 bash[19604]: cluster 2024-02-12T16:36:05.350424+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e792: 12 total, 12 up, 12 in 2024-02-12T16:36:06.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:06.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:06.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:06.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:06.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:06.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:06.954 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:06.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:07.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:07.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:07.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:07.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:07.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:07.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:07.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:07.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:07.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:07.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:07.497 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:07.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:07.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:07.610 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:07 smithi169 bash[19604]: cluster 2024-02-12T16:36:06.132631+0000 mgr.y (mgr.14146) 1546 : cluster [DBG] pgmap v2216: 47 pgs: 47 active+clean; 9.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:36:07.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:07.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:07 smithi123 bash[19119]: cluster 2024-02-12T16:36:06.132631+0000 mgr.y (mgr.14146) 1546 : cluster [DBG] pgmap v2216: 47 pgs: 47 active+clean; 9.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:36:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:07.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:07.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:07 smithi121 bash[18489]: cluster 2024-02-12T16:36:06.132631+0000 mgr.y (mgr.14146) 1546 : cluster [DBG] pgmap v2216: 47 pgs: 47 active+clean; 9.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-02-12T16:36:07.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:07.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:07.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:07.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:07.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:08.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:08.044 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:08.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:08.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:08.182 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:08.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:08.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:08.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:08.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:08.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:08.539 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:08.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:08.639 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:08.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:08.672 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:08.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:08.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:08.801 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:08.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:08.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:08.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:08.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:09.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:09.093 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:09.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:09.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:09.214 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:36:09.214 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:36:09.226 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:09.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:09.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:09.358 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:09.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:09.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:09.475 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:09 smithi123 bash[19119]: cluster 2024-02-12T16:36:08.133106+0000 mgr.y (mgr.14146) 1547 : cluster [DBG] pgmap v2217: 47 pgs: 47 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-02-12T16:36:09.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:09.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:09 smithi169 bash[19604]: cluster 2024-02-12T16:36:08.133106+0000 mgr.y (mgr.14146) 1547 : cluster [DBG] pgmap v2217: 47 pgs: 47 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-02-12T16:36:09.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:09.607 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:09 smithi121 bash[18489]: cluster 2024-02-12T16:36:08.133106+0000 mgr.y (mgr.14146) 1547 : cluster [DBG] pgmap v2217: 47 pgs: 47 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-02-12T16:36:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:09.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:09.650 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:09.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:09.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:09.874 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:09.901 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:09.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:10.005 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:10.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:10.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-12T16:36:10.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:10.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:10.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:10.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:10.469 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:10.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:10.601 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:10.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:10.731 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:10.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:10.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:10.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:10.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:10.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:11.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:11.124 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:11.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:11.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:11.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:11.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:11.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:11.359 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:11 smithi121 bash[18489]: cluster 2024-02-12T16:36:10.133582+0000 mgr.y (mgr.14146) 1548 : cluster [DBG] pgmap v2218: 47 pgs: 47 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-12T16:36:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:11.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:11.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:11.527 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:11.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:11.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:11 smithi123 bash[19119]: cluster 2024-02-12T16:36:10.133582+0000 mgr.y (mgr.14146) 1548 : cluster [DBG] pgmap v2218: 47 pgs: 47 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-12T16:36:11.640 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:11 smithi169 bash[19604]: cluster 2024-02-12T16:36:10.133582+0000 mgr.y (mgr.14146) 1548 : cluster [DBG] pgmap v2218: 47 pgs: 47 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-02-12T16:36:11.660 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:11.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:11.763 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:11.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:11.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:11.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:11.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:12.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:12.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:12.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:12.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:12.194 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:12.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:12.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:12.324 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:12.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:12.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:12.464 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:12.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:12.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:12.598 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:12.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:12.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:12.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:12.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:12.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:12.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:13.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:13.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:13.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:13.147 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:13.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:13.274 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:13.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:13.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:13.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:13.566 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:13 smithi121 bash[18489]: cluster 2024-02-12T16:36:12.133998+0000 mgr.y (mgr.14146) 1549 : cluster [DBG] pgmap v2219: 47 pgs: 47 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-02-12T16:36:13.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:13 smithi123 bash[19119]: cluster 2024-02-12T16:36:12.133998+0000 mgr.y (mgr.14146) 1549 : cluster [DBG] pgmap v2219: 47 pgs: 47 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-02-12T16:36:13.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:13.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:13.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:13.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:13 smithi169 bash[19604]: cluster 2024-02-12T16:36:12.133998+0000 mgr.y (mgr.14146) 1549 : cluster [DBG] pgmap v2219: 47 pgs: 47 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-02-12T16:36:13.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:13.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:13.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:13.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:14.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:14.076 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:14.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:14.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:14.239 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:14.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:14.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:14.374 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:14 smithi123 bash[19119]: cluster 2024-02-12T16:36:12.818163+0000 osd.1 (osd.1) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:14.374 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:14 smithi123 bash[19119]: cluster 2024-02-12T16:36:12.818217+0000 osd.1 (osd.1) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:14.374 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:14 smithi123 bash[19119]: cluster 2024-02-12T16:36:13.359931+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e793: 12 total, 12 up, 12 in 2024-02-12T16:36:14.374 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:14 smithi123 bash[19119]: cluster 2024-02-12T16:36:13.436342+0000 osd.11 (osd.11) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:14.374 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:14 smithi123 bash[19119]: cluster 2024-02-12T16:36:13.436404+0000 osd.11 (osd.11) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:14.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:14.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:14.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:14 smithi121 bash[18489]: cluster 2024-02-12T16:36:12.818163+0000 osd.1 (osd.1) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:14.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:14 smithi121 bash[18489]: cluster 2024-02-12T16:36:12.818217+0000 osd.1 (osd.1) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:14.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:14 smithi121 bash[18489]: cluster 2024-02-12T16:36:13.359931+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e793: 12 total, 12 up, 12 in 2024-02-12T16:36:14.483 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:14 smithi121 bash[18489]: cluster 2024-02-12T16:36:13.436342+0000 osd.11 (osd.11) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:14.483 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:14 smithi121 bash[18489]: cluster 2024-02-12T16:36:13.436404+0000 osd.11 (osd.11) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:14.488 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:14.518 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:14.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:14.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:14.665 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:14.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:14.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:14.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:14 smithi169 bash[19604]: cluster 2024-02-12T16:36:12.818163+0000 osd.1 (osd.1) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:14.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:14 smithi169 bash[19604]: cluster 2024-02-12T16:36:12.818217+0000 osd.1 (osd.1) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:14.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:14 smithi169 bash[19604]: cluster 2024-02-12T16:36:13.359931+0000 mon.a (mon.0) 2690 : cluster [DBG] osdmap e793: 12 total, 12 up, 12 in 2024-02-12T16:36:14.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:14 smithi169 bash[19604]: cluster 2024-02-12T16:36:13.436342+0000 osd.11 (osd.11) 178 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:14.777 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:14 smithi169 bash[19604]: cluster 2024-02-12T16:36:13.436404+0000 osd.11 (osd.11) 179 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:14.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:14.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:14.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:14.911 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:36:14.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:14.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:15.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:15.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:15.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:15.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:15.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:15.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:15.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:15.339 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:15.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:15.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:15.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:15.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:15.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:15.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:15.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:15 smithi123 bash[19119]: cluster 2024-02-12T16:36:14.134421+0000 mgr.y (mgr.14146) 1550 : cluster [DBG] pgmap v2221: 47 pgs: 47 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-02-12T16:36:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:15 smithi123 bash[19119]: cluster 2024-02-12T16:36:14.378882+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e794: 12 total, 12 up, 12 in 2024-02-12T16:36:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:15 smithi123 bash[19119]: cluster 2024-02-12T16:36:14.589622+0000 osd.7 (osd.7) 198 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:15.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:15 smithi123 bash[19119]: cluster 2024-02-12T16:36:14.589672+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:15.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:15 smithi121 bash[18489]: cluster 2024-02-12T16:36:14.134421+0000 mgr.y (mgr.14146) 1550 : cluster [DBG] pgmap v2221: 47 pgs: 47 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-02-12T16:36:15.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:15 smithi121 bash[18489]: cluster 2024-02-12T16:36:14.378882+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e794: 12 total, 12 up, 12 in 2024-02-12T16:36:15.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:15 smithi121 bash[18489]: cluster 2024-02-12T16:36:14.589622+0000 osd.7 (osd.7) 198 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:15.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:15 smithi121 bash[18489]: cluster 2024-02-12T16:36:14.589672+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:15.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:15.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:15.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:15 smithi169 bash[19604]: cluster 2024-02-12T16:36:14.134421+0000 mgr.y (mgr.14146) 1550 : cluster [DBG] pgmap v2221: 47 pgs: 47 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-02-12T16:36:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:15 smithi169 bash[19604]: cluster 2024-02-12T16:36:14.378882+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e794: 12 total, 12 up, 12 in 2024-02-12T16:36:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:15 smithi169 bash[19604]: cluster 2024-02-12T16:36:14.589622+0000 osd.7 (osd.7) 198 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:15 smithi169 bash[19604]: cluster 2024-02-12T16:36:14.589672+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:15.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:15.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:15.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:15.977 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:16.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:16.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:16.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:16.283 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:16.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: cluster 2024-02-12T16:36:15.384420+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e795: 12 total, 12 up, 12 in 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.964466+0000 mon.a (mon.0) 2693 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.964990+0000 mon.a (mon.0) 2694 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965149+0000 mon.c (mon.1) 69 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965327+0000 mon.a (mon.0) 2695 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965368+0000 mon.c (mon.1) 70 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965565+0000 mon.c (mon.1) 71 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965609+0000 mon.a (mon.0) 2696 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965749+0000 mon.c (mon.1) 72 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]: dispatch 2024-02-12T16:36:16.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965889+0000 mon.a (mon.0) 2697 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.965949+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966136+0000 mon.c (mon.1) 74 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966230+0000 mon.a (mon.0) 2698 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966306+0000 mon.c (mon.1) 75 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966516+0000 mon.c (mon.1) 76 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966576+0000 mon.a (mon.0) 2699 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966750+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966829+0000 mon.a (mon.0) 2700 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.966948+0000 mon.c (mon.1) 78 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.967040+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.967141+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.967246+0000 mon.a (mon.0) 2702 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.967316+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]: dispatch 2024-02-12T16:36:16.394 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.967443+0000 mon.a (mon.0) 2703 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]: dispatch 2024-02-12T16:36:16.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:15.967640+0000 mon.a (mon.0) 2704 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]: dispatch 2024-02-12T16:36:16.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:16.287647+0000 mon.a (mon.0) 2705 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]: dispatch 2024-02-12T16:36:16.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:16 smithi121 bash[18489]: audit 2024-02-12T16:36:16.288271+0000 mon.c (mon.1) 81 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]: dispatch 2024-02-12T16:36:16.410 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:16.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:16.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: cluster 2024-02-12T16:36:15.384420+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e795: 12 total, 12 up, 12 in 2024-02-12T16:36:16.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.964466+0000 mon.a (mon.0) 2693 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.964990+0000 mon.a (mon.0) 2694 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965149+0000 mon.c (mon.1) 69 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965327+0000 mon.a (mon.0) 2695 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965368+0000 mon.c (mon.1) 70 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965565+0000 mon.c (mon.1) 71 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965609+0000 mon.a (mon.0) 2696 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965749+0000 mon.c (mon.1) 72 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]: dispatch 2024-02-12T16:36:16.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965889+0000 mon.a (mon.0) 2697 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]: dispatch 2024-02-12T16:36:16.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.965949+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]: dispatch 2024-02-12T16:36:16.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966136+0000 mon.c (mon.1) 74 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]: dispatch 2024-02-12T16:36:16.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966230+0000 mon.a (mon.0) 2698 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]: dispatch 2024-02-12T16:36:16.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966306+0000 mon.c (mon.1) 75 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]: dispatch 2024-02-12T16:36:16.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966516+0000 mon.c (mon.1) 76 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]: dispatch 2024-02-12T16:36:16.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966576+0000 mon.a (mon.0) 2699 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]: dispatch 2024-02-12T16:36:16.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966750+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]: dispatch 2024-02-12T16:36:16.643 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966829+0000 mon.a (mon.0) 2700 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]: dispatch 2024-02-12T16:36:16.643 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.966948+0000 mon.c (mon.1) 78 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]: dispatch 2024-02-12T16:36:16.643 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.967040+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]: dispatch 2024-02-12T16:36:16.643 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.967141+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]: dispatch 2024-02-12T16:36:16.643 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.967246+0000 mon.a (mon.0) 2702 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]: dispatch 2024-02-12T16:36:16.643 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.967316+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]: dispatch 2024-02-12T16:36:16.643 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.967443+0000 mon.a (mon.0) 2703 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]: dispatch 2024-02-12T16:36:16.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:15.967640+0000 mon.a (mon.0) 2704 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]: dispatch 2024-02-12T16:36:16.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:16.287647+0000 mon.a (mon.0) 2705 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]: dispatch 2024-02-12T16:36:16.644 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:16 smithi123 bash[19119]: audit 2024-02-12T16:36:16.288271+0000 mon.c (mon.1) 81 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]: dispatch 2024-02-12T16:36:16.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:16.673 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:16.682 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: cluster 2024-02-12T16:36:15.384420+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e795: 12 total, 12 up, 12 in 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.964466+0000 mon.a (mon.0) 2693 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.964990+0000 mon.a (mon.0) 2694 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965149+0000 mon.c (mon.1) 69 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965327+0000 mon.a (mon.0) 2695 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965368+0000 mon.c (mon.1) 70 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965565+0000 mon.c (mon.1) 71 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965609+0000 mon.a (mon.0) 2696 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965749+0000 mon.c (mon.1) 72 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965889+0000 mon.a (mon.0) 2697 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.965949+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966136+0000 mon.c (mon.1) 74 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]: dispatch 2024-02-12T16:36:16.683 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966230+0000 mon.a (mon.0) 2698 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966306+0000 mon.c (mon.1) 75 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966516+0000 mon.c (mon.1) 76 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966576+0000 mon.a (mon.0) 2699 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966750+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966829+0000 mon.a (mon.0) 2700 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.966948+0000 mon.c (mon.1) 78 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.967040+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.967141+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.967246+0000 mon.a (mon.0) 2702 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.967316+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.967443+0000 mon.a (mon.0) 2703 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:15.967640+0000 mon.a (mon.0) 2704 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]: dispatch 2024-02-12T16:36:16.684 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:16.287647+0000 mon.a (mon.0) 2705 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]: dispatch 2024-02-12T16:36:16.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:16 smithi169 bash[19604]: audit 2024-02-12T16:36:16.288271+0000 mon.c (mon.1) 81 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]: dispatch 2024-02-12T16:36:16.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:16.805 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:moved 6 / 94 (6.38298%) 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:avg 7.83333 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 1.40436 -> 1.51841 (expected baseline 2.67966) 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.1 with 6 -> 7 pgs (0.765957 -> 0.893617 * mean) 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.0 with 11 -> 6 pgs (1.40426 -> 0.765957 * mean) 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:oload 110 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 1 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0232 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.0255 2024-02-12T16:36:16.847 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 weight 1.0000 -> 0.6635 2024-02-12T16:36:16.848 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 weight 1.0000 -> 0.8111 2024-02-12T16:36:16.848 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9 weight 1.0000 -> 0.9061 2024-02-12T16:36:16.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:16.929 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:16.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:17.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:17.073 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:17.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:17.176 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:17.213 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:17.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:17.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:17.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:17.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:17.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:17.500 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:17.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:17.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:17.619 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: cluster 2024-02-12T16:36:16.134813+0000 mgr.y (mgr.14146) 1551 : cluster [DBG] pgmap v2224: 47 pgs: 47 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-12T16:36:17.619 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.286515+0000 mgr.y (mgr.14146) 1552 : audit [DBG] from='client.26669 v1:172.21.15.121:0/904458153' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:17.619 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837143+0000 mon.a (mon.0) 2706 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]': finished 2024-02-12T16:36:17.620 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837207+0000 mon.a (mon.0) 2707 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]': finished 2024-02-12T16:36:17.620 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837258+0000 mon.a (mon.0) 2708 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]': finished 2024-02-12T16:36:17.620 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837303+0000 mon.a (mon.0) 2709 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]': finished 2024-02-12T16:36:17.620 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837348+0000 mon.a (mon.0) 2710 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]': finished 2024-02-12T16:36:17.620 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837393+0000 mon.a (mon.0) 2711 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]': finished 2024-02-12T16:36:17.620 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837437+0000 mon.a (mon.0) 2712 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]': finished 2024-02-12T16:36:17.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837503+0000 mon.a (mon.0) 2713 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]': finished 2024-02-12T16:36:17.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837582+0000 mon.a (mon.0) 2714 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]': finished 2024-02-12T16:36:17.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837634+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]': finished 2024-02-12T16:36:17.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837695+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]': finished 2024-02-12T16:36:17.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837759+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]': finished 2024-02-12T16:36:17.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: audit 2024-02-12T16:36:16.837805+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]': finished 2024-02-12T16:36:17.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:17 smithi121 bash[18489]: cluster 2024-02-12T16:36:16.837840+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e796: 12 total, 12 up, 12 in 2024-02-12T16:36:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: cluster 2024-02-12T16:36:16.134813+0000 mgr.y (mgr.14146) 1551 : cluster [DBG] pgmap v2224: 47 pgs: 47 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-12T16:36:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.286515+0000 mgr.y (mgr.14146) 1552 : audit [DBG] from='client.26669 v1:172.21.15.121:0/904458153' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837143+0000 mon.a (mon.0) 2706 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837207+0000 mon.a (mon.0) 2707 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837258+0000 mon.a (mon.0) 2708 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837303+0000 mon.a (mon.0) 2709 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837348+0000 mon.a (mon.0) 2710 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837393+0000 mon.a (mon.0) 2711 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837437+0000 mon.a (mon.0) 2712 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837503+0000 mon.a (mon.0) 2713 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837582+0000 mon.a (mon.0) 2714 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837634+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837695+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837759+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]': finished 2024-02-12T16:36:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: audit 2024-02-12T16:36:16.837805+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]': finished 2024-02-12T16:36:17.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:17 smithi123 bash[19119]: cluster 2024-02-12T16:36:16.837840+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e796: 12 total, 12 up, 12 in 2024-02-12T16:36:17.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:17.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:17.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:17.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: cluster 2024-02-12T16:36:16.134813+0000 mgr.y (mgr.14146) 1551 : cluster [DBG] pgmap v2224: 47 pgs: 47 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.286515+0000 mgr.y (mgr.14146) 1552 : audit [DBG] from='client.26669 v1:172.21.15.121:0/904458153' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837143+0000 mon.a (mon.0) 2706 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08089842078410396]}]': finished 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837207+0000 mon.a (mon.0) 2707 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09289307375685198]}]': finished 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837258+0000 mon.a (mon.0) 2708 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08834919777762579]}]': finished 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837303+0000 mon.a (mon.0) 2709 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09577949402393758]}]': finished 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837348+0000 mon.a (mon.0) 2710 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09212534718538293]}]': finished 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837393+0000 mon.a (mon.0) 2711 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.0841232274648511]}]': finished 2024-02-12T16:36:17.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837437+0000 mon.a (mon.0) 2712 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08523188529149611]}]': finished 2024-02-12T16:36:17.801 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837503+0000 mon.a (mon.0) 2713 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08380248346858787]}]': finished 2024-02-12T16:36:17.801 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837582+0000 mon.a (mon.0) 2714 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0903975986912007]}]': finished 2024-02-12T16:36:17.801 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837634+0000 mon.a (mon.0) 2715 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08542686736333584]}]': finished 2024-02-12T16:36:17.801 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837695+0000 mon.a (mon.0) 2716 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08344468825734862]}]': finished 2024-02-12T16:36:17.801 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837759+0000 mon.a (mon.0) 2717 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08507410265402755]}]': finished 2024-02-12T16:36:17.801 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: audit 2024-02-12T16:36:16.837805+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'43481','2':'53157','9':'59382'}"}]': finished 2024-02-12T16:36:17.801 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:17 smithi169 bash[19604]: cluster 2024-02-12T16:36:16.837840+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e796: 12 total, 12 up, 12 in 2024-02-12T16:36:17.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:17.919 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:17.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:18.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:18.061 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:18.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:18.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:18.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:18.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:18.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:18.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:18.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:18.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:18.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:18.566 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-12T16:36:18.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight-by-utilization 1000 3.0 2024-02-12T16:36:18.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:18.621 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:18.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:18.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:18.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:18.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:18.791 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:36:18.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:36:18.840 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:36:18.874 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:36:18.876 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:18.896 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:36:18.902 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:18.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:18.918 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:36:18.948 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:36:18.973 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:36:19.005 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:19.016 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:36:19.016 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:36:19.026 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:36:19.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:19.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:18 smithi169 bash[19604]: cluster 2024-02-12T16:36:17.843376+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e797: 12 total, 12 up, 12 in 2024-02-12T16:36:19.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:18 smithi169 bash[19604]: cluster 2024-02-12T16:36:17.896774+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:19.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:18 smithi169 bash[19604]: cluster 2024-02-12T16:36:17.896866+0000 osd.4 (osd.4) 137 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:19.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:18 smithi169 bash[19604]: cluster 2024-02-12T16:36:18.135437+0000 mgr.y (mgr.14146) 1553 : cluster [DBG] pgmap v2227: 47 pgs: 47 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.53k op/s 2024-02-12T16:36:19.032 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:18 smithi169 bash[19604]: audit 2024-02-12T16:36:18.757385+0000 mon.a (mon.0) 2721 : audit [DBG] from='client.? v1:172.21.15.121:0/82415753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:19.049 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:36:19.082 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:36:19.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:36:19.120 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.121 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:36:19.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:19.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:18 smithi123 bash[19119]: cluster 2024-02-12T16:36:17.843376+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e797: 12 total, 12 up, 12 in 2024-02-12T16:36:19.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:18 smithi123 bash[19119]: cluster 2024-02-12T16:36:17.896774+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:19.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:18 smithi123 bash[19119]: cluster 2024-02-12T16:36:17.896866+0000 osd.4 (osd.4) 137 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:19.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:18 smithi123 bash[19119]: cluster 2024-02-12T16:36:18.135437+0000 mgr.y (mgr.14146) 1553 : cluster [DBG] pgmap v2227: 47 pgs: 47 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.53k op/s 2024-02-12T16:36:19.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:18 smithi123 bash[19119]: audit 2024-02-12T16:36:18.757385+0000 mon.a (mon.0) 2721 : audit [DBG] from='client.? v1:172.21.15.121:0/82415753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:19.143 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:18 smithi121 bash[18489]: cluster 2024-02-12T16:36:17.843376+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e797: 12 total, 12 up, 12 in 2024-02-12T16:36:19.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:18 smithi121 bash[18489]: cluster 2024-02-12T16:36:17.896774+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:19.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:18 smithi121 bash[18489]: cluster 2024-02-12T16:36:17.896866+0000 osd.4 (osd.4) 137 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:19.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:18 smithi121 bash[18489]: cluster 2024-02-12T16:36:18.135437+0000 mgr.y (mgr.14146) 1553 : cluster [DBG] pgmap v2227: 47 pgs: 47 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.53k op/s 2024-02-12T16:36:19.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:18 smithi121 bash[18489]: audit 2024-02-12T16:36:18.757385+0000 mon.a (mon.0) 2721 : audit [DBG] from='client.? v1:172.21.15.121:0/82415753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:19.177 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:19.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:19.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:19.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:19.452 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:19.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:19.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:19.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:19.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:19.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:19.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:19.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:36:19.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:36:19.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:19 smithi121 bash[18489]: cluster 2024-02-12T16:36:18.849966+0000 osd.0 (osd.0) 138 : cluster [DBG] 7.f starting backfill to osd.3 from (0'0,0'0] MAX to 795'12222 2024-02-12T16:36:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:19 smithi121 bash[18489]: cluster 2024-02-12T16:36:18.852303+0000 osd.11 (osd.11) 180 : cluster [DBG] 7.9 starting backfill to osd.8 from (0'0,0'0] MAX to 795'12013 2024-02-12T16:36:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:19 smithi121 bash[18489]: cluster 2024-02-12T16:36:18.852688+0000 osd.4 (osd.4) 138 : cluster [DBG] 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 795'12238 2024-02-12T16:36:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:19 smithi121 bash[18489]: cluster 2024-02-12T16:36:18.853642+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e798: 12 total, 12 up, 12 in 2024-02-12T16:36:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:19 smithi121 bash[18489]: audit 2024-02-12T16:36:19.239222+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? v1:172.21.15.121:0/3178273688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:36:19.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:19.986 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:19.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:20.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:20.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:19 smithi123 bash[19119]: cluster 2024-02-12T16:36:18.849966+0000 osd.0 (osd.0) 138 : cluster [DBG] 7.f starting backfill to osd.3 from (0'0,0'0] MAX to 795'12222 2024-02-12T16:36:20.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:19 smithi123 bash[19119]: cluster 2024-02-12T16:36:18.852303+0000 osd.11 (osd.11) 180 : cluster [DBG] 7.9 starting backfill to osd.8 from (0'0,0'0] MAX to 795'12013 2024-02-12T16:36:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:19 smithi123 bash[19119]: cluster 2024-02-12T16:36:18.852688+0000 osd.4 (osd.4) 138 : cluster [DBG] 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 795'12238 2024-02-12T16:36:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:19 smithi123 bash[19119]: cluster 2024-02-12T16:36:18.853642+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e798: 12 total, 12 up, 12 in 2024-02-12T16:36:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:19 smithi123 bash[19119]: audit 2024-02-12T16:36:19.239222+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? v1:172.21.15.121:0/3178273688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:36:20.141 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:20.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:20.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:20.253 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:19 smithi169 bash[19604]: cluster 2024-02-12T16:36:18.849966+0000 osd.0 (osd.0) 138 : cluster [DBG] 7.f starting backfill to osd.3 from (0'0,0'0] MAX to 795'12222 2024-02-12T16:36:20.253 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:19 smithi169 bash[19604]: cluster 2024-02-12T16:36:18.852303+0000 osd.11 (osd.11) 180 : cluster [DBG] 7.9 starting backfill to osd.8 from (0'0,0'0] MAX to 795'12013 2024-02-12T16:36:20.253 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:19 smithi169 bash[19604]: cluster 2024-02-12T16:36:18.852688+0000 osd.4 (osd.4) 138 : cluster [DBG] 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 795'12238 2024-02-12T16:36:20.253 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:19 smithi169 bash[19604]: cluster 2024-02-12T16:36:18.853642+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e798: 12 total, 12 up, 12 in 2024-02-12T16:36:20.253 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:19 smithi169 bash[19604]: audit 2024-02-12T16:36:19.239222+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? v1:172.21.15.121:0/3178273688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:36:20.276 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:20.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:20.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:20.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:20.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:20.518 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:20.545 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:20.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:20.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:20.688 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:20.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:20.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:20.838 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:20.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:20.858 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:20 smithi123 bash[19119]: cluster 2024-02-12T16:36:19.846868+0000 mon.a (mon.0) 2724 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:36:20.858 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:20 smithi123 bash[19119]: audit 2024-02-12T16:36:19.859470+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? v1:172.21.15.121:0/3178273688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:36:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:20 smithi121 bash[18489]: cluster 2024-02-12T16:36:19.846868+0000 mon.a (mon.0) 2724 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:36:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:20 smithi121 bash[18489]: audit 2024-02-12T16:36:19.859470+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? v1:172.21.15.121:0/3178273688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:36:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:20 smithi121 bash[18489]: cluster 2024-02-12T16:36:19.859545+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e799: 12 total, 12 up, 12 in 2024-02-12T16:36:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:20 smithi121 bash[18489]: cluster 2024-02-12T16:36:20.135922+0000 mgr.y (mgr.14146) 1554 : cluster [DBG] pgmap v2230: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s; 17366/283528 objects misplaced (6.125%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:36:20.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:20.976 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:20.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:21.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:21.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:21.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:20 smithi169 bash[19604]: cluster 2024-02-12T16:36:19.846868+0000 mon.a (mon.0) 2724 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:36:21.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:20 smithi169 bash[19604]: audit 2024-02-12T16:36:19.859470+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? v1:172.21.15.121:0/3178273688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:36:21.117 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:20 smithi169 bash[19604]: cluster 2024-02-12T16:36:19.859545+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e799: 12 total, 12 up, 12 in 2024-02-12T16:36:21.117 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:20 smithi169 bash[19604]: cluster 2024-02-12T16:36:20.135922+0000 mgr.y (mgr.14146) 1554 : cluster [DBG] pgmap v2230: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s; 17366/283528 objects misplaced (6.125%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:36:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:21.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:21.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:20 smithi123 bash[19119]: cluster 2024-02-12T16:36:19.859545+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e799: 12 total, 12 up, 12 in 2024-02-12T16:36:21.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:20 smithi123 bash[19119]: cluster 2024-02-12T16:36:20.135922+0000 mgr.y (mgr.14146) 1554 : cluster [DBG] pgmap v2230: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.33k op/s; 17366/283528 objects misplaced (6.125%); 3.4 MiB/s, 54 objects/s recovering 2024-02-12T16:36:21.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:21.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:21.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:21.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:21.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:21.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:21.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:21.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:21.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:21.664 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:21.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:21.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:21.795 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:21.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:21.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:21.941 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:21.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:22.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:22.077 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:22.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:22.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:22.224 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:22.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:22.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:36:22.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:22.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:22.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:22.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:22.493 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:22.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:22.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:22.636 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:22.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:22.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:22.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:22.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:22.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:22.905 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:22.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:23.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:36:23.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:23.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:23.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:23.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:23.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:23.360 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:23.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:23 smithi121 bash[18489]: cluster 2024-02-12T16:36:21.834327+0000 osd.7 (osd.7) 200 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:36:23.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:23 smithi121 bash[18489]: cluster 2024-02-12T16:36:21.958622+0000 osd.7 (osd.7) 201 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:36:23.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:23 smithi121 bash[18489]: cluster 2024-02-12T16:36:22.136403+0000 mgr.y (mgr.14146) 1555 : cluster [DBG] pgmap v2231: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 25823/280310 objects misplaced (9.212%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T16:36:23.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.515 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:23.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:23.551 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:23 smithi169 bash[19604]: cluster 2024-02-12T16:36:21.834327+0000 osd.7 (osd.7) 200 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:36:23.551 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:23 smithi169 bash[19604]: cluster 2024-02-12T16:36:21.958622+0000 osd.7 (osd.7) 201 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:36:23.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:23 smithi169 bash[19604]: cluster 2024-02-12T16:36:22.136403+0000 mgr.y (mgr.14146) 1555 : cluster [DBG] pgmap v2231: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 25823/280310 objects misplaced (9.212%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T16:36:23.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:23 smithi123 bash[19119]: cluster 2024-02-12T16:36:21.834327+0000 osd.7 (osd.7) 200 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:36:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:23 smithi123 bash[19119]: cluster 2024-02-12T16:36:21.958622+0000 osd.7 (osd.7) 201 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:36:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:23 smithi123 bash[19119]: cluster 2024-02-12T16:36:22.136403+0000 mgr.y (mgr.14146) 1555 : cluster [DBG] pgmap v2231: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 25823/280310 objects misplaced (9.212%); 5.7 MiB/s, 90 objects/s recovering 2024-02-12T16:36:23.642 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:23.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:23.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:23.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:23.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:23.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:23.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:24.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:24.060 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:24.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:24.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:24.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:24.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:24.301 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:24.328 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:24.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:24.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:24.459 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:24.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:24.507 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:24.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:24.587 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:24.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:24.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:24.728 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:24.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:24.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:24.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:24.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:24.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:25.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:25.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:25.132 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:25.198 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:25 smithi169 bash[19604]: cluster 2024-02-12T16:36:24.136829+0000 mgr.y (mgr.14146) 1556 : cluster [DBG] pgmap v2232: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 25823/279842 objects misplaced (9.228%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:36:25.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:25.247 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:25 smithi121 bash[18489]: cluster 2024-02-12T16:36:24.136829+0000 mgr.y (mgr.14146) 1556 : cluster [DBG] pgmap v2232: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 25823/279842 objects misplaced (9.228%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:36:25.271 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:25.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:25 smithi123 bash[19119]: cluster 2024-02-12T16:36:24.136829+0000 mgr.y (mgr.14146) 1556 : cluster [DBG] pgmap v2232: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 25823/279842 objects misplaced (9.228%); 4.8 MiB/s, 76 objects/s recovering 2024-02-12T16:36:25.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:25.521 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:25.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.551 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:25.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:25.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:25.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:25.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:25.808 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:25.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:25.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:25.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:26.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:26.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:26.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:26.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:26.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:26.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:26.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:26.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:26.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:26.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:26.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:26.534 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:26.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:26.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:26.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:26.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:26.760 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:26.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:26.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:26.871 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:26 smithi121 bash[18489]: cluster 2024-02-12T16:36:26.141202+0000 mgr.y (mgr.14146) 1557 : cluster [DBG] pgmap v2233: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 23804/275022 objects misplaced (8.655%); 14 MiB/s, 228 objects/s recovering 2024-02-12T16:36:26.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:26.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:27.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:27.008 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:26 smithi123 bash[19119]: cluster 2024-02-12T16:36:26.141202+0000 mgr.y (mgr.14146) 1557 : cluster [DBG] pgmap v2233: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 23804/275022 objects misplaced (8.655%); 14 MiB/s, 228 objects/s recovering 2024-02-12T16:36:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.037 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:27.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:27.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:27.172 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:27.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:26 smithi169 bash[19604]: cluster 2024-02-12T16:36:26.141202+0000 mgr.y (mgr.14146) 1557 : cluster [DBG] pgmap v2233: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 23804/275022 objects misplaced (8.655%); 14 MiB/s, 228 objects/s recovering 2024-02-12T16:36:27.303 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:27.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:27.438 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:27.545 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:27.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.578 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:27.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:27.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:27.705 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:27.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:27.846 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:27.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:27.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:27.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:28.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:28.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:28.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:28.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:28.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:28.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:28.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:28.393 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:28.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:28.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:28.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:28.527 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:28.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:28.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:28.663 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:28.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:28.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:28.797 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:28.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:28.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:28.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:29.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:29.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:29.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:29.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:29.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:29.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:29.310 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:29 smithi121 bash[18489]: cluster 2024-02-12T16:36:28.141664+0000 mgr.y (mgr.14146) 1558 : cluster [DBG] pgmap v2234: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 22905/268592 objects misplaced (8.528%); 16 MiB/s, 257 objects/s recovering 2024-02-12T16:36:29.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:29.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:29.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:29.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:29.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:29.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:29.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:29 smithi169 bash[19604]: cluster 2024-02-12T16:36:28.141664+0000 mgr.y (mgr.14146) 1558 : cluster [DBG] pgmap v2234: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 22905/268592 objects misplaced (8.528%); 16 MiB/s, 257 objects/s recovering 2024-02-12T16:36:29.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:29 smithi123 bash[19119]: cluster 2024-02-12T16:36:28.141664+0000 mgr.y (mgr.14146) 1558 : cluster [DBG] pgmap v2234: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 22905/268592 objects misplaced (8.528%); 16 MiB/s, 257 objects/s recovering 2024-02-12T16:36:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:29.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:29.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:29.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:29.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:29.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:29.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:29.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:30.020 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:30.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:30.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:30.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:30.197 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:30.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:30.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:30.343 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:30.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:30.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:30.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:30.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:30.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:30.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:30.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:30.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:30.745 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:30.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:30.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:30.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:30.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:30.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:31.016 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:31.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:31.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:31.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:31.260 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:31 smithi123 bash[19119]: cluster 2024-02-12T16:36:30.142141+0000 mgr.y (mgr.14146) 1559 : cluster [DBG] pgmap v2235: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 21037/263766 objects misplaced (7.976%); 22 MiB/s, 357 objects/s recovering 2024-02-12T16:36:31.260 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:31 smithi123 bash[19119]: cluster 2024-02-12T16:36:30.301759+0000 osd.6 (osd.6) 158 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:36:31.261 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:31 smithi123 bash[19119]: cluster 2024-02-12T16:36:30.387875+0000 osd.6 (osd.6) 159 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:36:31.278 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:31.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:31.434 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:31 smithi121 bash[18489]: cluster 2024-02-12T16:36:30.142141+0000 mgr.y (mgr.14146) 1559 : cluster [DBG] pgmap v2235: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 21037/263766 objects misplaced (7.976%); 22 MiB/s, 357 objects/s recovering 2024-02-12T16:36:31.434 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:31 smithi121 bash[18489]: cluster 2024-02-12T16:36:30.301759+0000 osd.6 (osd.6) 158 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:36:31.434 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:31 smithi121 bash[18489]: cluster 2024-02-12T16:36:30.387875+0000 osd.6 (osd.6) 159 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:36:31.520 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:31.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:31 smithi169 bash[19604]: cluster 2024-02-12T16:36:30.142141+0000 mgr.y (mgr.14146) 1559 : cluster [DBG] pgmap v2235: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 21037/263766 objects misplaced (7.976%); 22 MiB/s, 357 objects/s recovering 2024-02-12T16:36:31.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:31 smithi169 bash[19604]: cluster 2024-02-12T16:36:30.301759+0000 osd.6 (osd.6) 158 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:36:31.531 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:31 smithi169 bash[19604]: cluster 2024-02-12T16:36:30.387875+0000 osd.6 (osd.6) 159 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:36:31.555 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:31.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:31.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:31.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:31.817 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:31.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:31.954 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:31.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:32.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:32.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:32.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:32.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:32.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:32.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:32.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:32.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:32.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:32.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:32.532 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:32.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:32.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:32.675 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:32.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:32.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:32.813 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:32.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:32.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:32.964 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:33.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:33.117 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:32 smithi169 bash[19604]: cluster 2024-02-12T16:36:31.088246+0000 osd.6 (osd.6) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:33.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:32 smithi169 bash[19604]: cluster 2024-02-12T16:36:31.088307+0000 osd.6 (osd.6) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:33.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:32 smithi169 bash[19604]: cluster 2024-02-12T16:36:31.833156+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e800: 12 total, 12 up, 12 in 2024-02-12T16:36:33.118 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:32 smithi169 bash[19604]: cluster 2024-02-12T16:36:32.142595+0000 mgr.y (mgr.14146) 1560 : cluster [DBG] pgmap v2237: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 20149/258748 objects misplaced (7.787%); 24 MiB/s, 382 objects/s recovering 2024-02-12T16:36:33.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:32 smithi123 bash[19119]: cluster 2024-02-12T16:36:31.088246+0000 osd.6 (osd.6) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:32 smithi123 bash[19119]: cluster 2024-02-12T16:36:31.088307+0000 osd.6 (osd.6) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:32 smithi123 bash[19119]: cluster 2024-02-12T16:36:31.833156+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e800: 12 total, 12 up, 12 in 2024-02-12T16:36:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:32 smithi123 bash[19119]: cluster 2024-02-12T16:36:32.142595+0000 mgr.y (mgr.14146) 1560 : cluster [DBG] pgmap v2237: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 20149/258748 objects misplaced (7.787%); 24 MiB/s, 382 objects/s recovering 2024-02-12T16:36:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:32 smithi121 bash[18489]: cluster 2024-02-12T16:36:31.088246+0000 osd.6 (osd.6) 160 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:32 smithi121 bash[18489]: cluster 2024-02-12T16:36:31.088307+0000 osd.6 (osd.6) 161 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:32 smithi121 bash[18489]: cluster 2024-02-12T16:36:31.833156+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e800: 12 total, 12 up, 12 in 2024-02-12T16:36:33.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:32 smithi121 bash[18489]: cluster 2024-02-12T16:36:32.142595+0000 mgr.y (mgr.14146) 1560 : cluster [DBG] pgmap v2237: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 20149/258748 objects misplaced (7.787%); 24 MiB/s, 382 objects/s recovering 2024-02-12T16:36:33.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:33.271 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:33.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:33.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:33.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:33.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:33.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:33.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:33.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:33.855 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:33.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:33.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:33.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:33.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:34.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:34.128 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:34.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:34.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:34.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:34.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:34.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:34.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:34.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:34.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:34.544 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:34.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:34.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:34.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:34.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:34.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:34.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:34.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:34.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:35.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:35.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:35.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:35.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:35.285 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:35 smithi121 bash[18489]: cluster 2024-02-12T16:36:34.143008+0000 mgr.y (mgr.14146) 1561 : cluster [DBG] pgmap v2238: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 20149/258124 objects misplaced (7.806%); 24 MiB/s, 382 objects/s recovering 2024-02-12T16:36:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:35.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:35.426 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:35.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:35.539 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:35 smithi169 bash[19604]: cluster 2024-02-12T16:36:34.143008+0000 mgr.y (mgr.14146) 1561 : cluster [DBG] pgmap v2238: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 20149/258124 objects misplaced (7.806%); 24 MiB/s, 382 objects/s recovering 2024-02-12T16:36:35.562 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:35 smithi123 bash[19119]: cluster 2024-02-12T16:36:34.143008+0000 mgr.y (mgr.14146) 1561 : cluster [DBG] pgmap v2238: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 20149/258124 objects misplaced (7.806%); 24 MiB/s, 382 objects/s recovering 2024-02-12T16:36:35.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:35.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:35.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:35.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:35.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:35.950 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:35.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:36.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:36.089 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:36.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:36.191 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:36.219 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:36.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:36.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:36.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:36.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:36.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:36.499 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:36.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:36.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:36.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:36.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:36.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:36.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:36.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:36.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:36.930 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:36.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:37.038 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:37.058 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:36 smithi121 bash[18489]: cluster 2024-02-12T16:36:36.143473+0000 mgr.y (mgr.14146) 1562 : cluster [DBG] pgmap v2239: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 18160/253624 objects misplaced (7.160%); 25 MiB/s, 392 objects/s recovering 2024-02-12T16:36:37.063 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:37.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:36 smithi123 bash[19119]: cluster 2024-02-12T16:36:36.143473+0000 mgr.y (mgr.14146) 1562 : cluster [DBG] pgmap v2239: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 18160/253624 objects misplaced (7.160%); 25 MiB/s, 392 objects/s recovering 2024-02-12T16:36:37.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:37.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:37.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:36 smithi169 bash[19604]: cluster 2024-02-12T16:36:36.143473+0000 mgr.y (mgr.14146) 1562 : cluster [DBG] pgmap v2239: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 18160/253624 objects misplaced (7.160%); 25 MiB/s, 392 objects/s recovering 2024-02-12T16:36:37.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:37.322 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:37.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:37.413 INFO:teuthology.orchestra.run.smithi121.stdout:moved 0 / 94 (0%) 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:avg 7.83333 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 1.28019 -> 1.28019 (expected baseline 2.67966) 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.0 with 6 -> 6 pgs (0.765957 -> 0.765957 * mean) 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.6 with 10 -> 10 pgs (1.2766 -> 1.2766 * mean) 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:oload 1000 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 3 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0212 2024-02-12T16:36:37.414 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.2121 2024-02-12T16:36:37.418 INFO:teuthology.orchestra.run.smithi121.stderr:no change 2024-02-12T16:36:37.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:37.433 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26530.0:214902 7.2 7:4942c911:::benchmark_data_smithi169_105252_object46308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e800)", 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:36:37.403066+0000", 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.020288368000000001, 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020326442, 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26530", 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 214902 2024-02-12T16:36:37.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.403066+0000", 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.403066+0000", 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.403064+0000", 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4429999999999997e-06 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.403071+0000", 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6049000000000002e-05 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.403087+0000", 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4390000000000003e-06 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.403095+0000", 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012045857 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:36:37.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.415140+0000", 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012548799999999999 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.415266+0000", 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014587399999999999 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.415412+0000", 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012346207999999999 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26530.0:214909 7.2 7:454634ef:::benchmark_data_smithi169_105252_object46315:head [delete] snapc 0=[] ondisk+write+known_if_redirected e800)", 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:36:37.409348+0000", 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.014006261000000001, 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014132603000000001, 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:36:37.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26530", 2024-02-12T16:36:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:36:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 214909 2024-02-12T16:36:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:36:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.409348+0000", 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.409348+0000", 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.409345+0000", 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.426e-06 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.409352+0000", 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2532e-05 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.409364+0000", 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3090000000000003e-06 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.409370+0000", 2024-02-12T16:36:37.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010325799 2024-02-12T16:36:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:36:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419695+0000", 2024-02-12T16:36:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012279500000000001 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419818+0000", 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014600799999999999 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 10", 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419964+0000", 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010616591 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:36:37.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26530.0:214908 7.f e800/797 7:f2a0bc2d:::benchmark_data_smithi169_105252_object46314:head v 800'13441, mlcod=800'13439)", 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:36:37.410612+0000", 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.012741924999999999, 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012953111999999999, 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410612+0000", 2024-02-12T16:36:37.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410612+0000", 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410610+0000", 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0078000000000001e-05 2024-02-12T16:36:37.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410621+0000", 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4959e-05 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410636+0000", 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4330000000000001e-06 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:36:37.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410642+0000", 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0426999999999993e-05 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410732+0000", 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3584e-05 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.410756+0000", 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014393300000000001 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:36:37.448 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26530.0:214911 7.2 7.a42290a2 (undecoded) ondisk+write+known_if_redirected e800)", 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:36:37.411995+0000", 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.011358686999999999, 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011635309, 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26530", 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:36:37.449 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 214911 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.411995+0000", 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.411995+0000", 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.411993+0000", 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1613e-05 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.412004+0000", 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1335000000000001e-05 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.412016+0000", 2024-02-12T16:36:37.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8660000000000002e-06 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.412022+0000", 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6359999999999998e-05 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26530.0:214902 7.2 e800/752)", 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:36:37.419516+0000", 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0038378449999999999, 2024-02-12T16:36:37.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004175972, 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419516+0000", 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419516+0000", 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:36:37.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419514+0000", 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.679e-06 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419523+0000", 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8281999999999999e-05 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:36:37.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419542+0000", 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1459999999999999e-06 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.419549+0000", 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2588e-05 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.455 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26530.0:214912 7.f e800/797 7:fbd9fac4:::benchmark_data_smithi169_105252_object46318:head v 800'13442, mlcod=800'13439)", 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:36:37.423327+0000", 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 2.6548999999999999e-05, 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000417947, 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423327+0000", 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423327+0000", 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423325+0000", 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5136e-05 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423340+0000", 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9324999999999999e-05 2024-02-12T16:36:37.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423360+0000", 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9279999999999993e-06 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423367+0000", 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023378599999999999 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423601+0000", 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7617e-05 2024-02-12T16:36:37.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:36:37.423629+0000", 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030145600000000002 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 6 2024-02-12T16:36:37.459 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:36:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:37.461 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:37.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:37.529 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:36:37.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:37.577 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:36:37.589 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:37.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:37.621 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:36:37.647 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:36:37.679 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:36:37.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:37.713 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:36:37.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:37.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:37.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:36:37.771 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:36:37.803 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:36:37.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:37.834 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:36:37.863 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:37.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:37.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:36:37.905 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.912 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:36:37.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:38.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:38.016 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:37 smithi121 bash[18489]: audit 2024-02-12T16:36:37.407276+0000 mgr.y (mgr.14146) 1563 : audit [DBG] from='client.26687 v1:172.21.15.121:0/43720742' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:38.016 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:37 smithi121 bash[18489]: audit 2024-02-12T16:36:37.472051+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.121:0/2632038650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:37 smithi169 bash[19604]: audit 2024-02-12T16:36:37.407276+0000 mgr.y (mgr.14146) 1563 : audit [DBG] from='client.26687 v1:172.21.15.121:0/43720742' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:37 smithi169 bash[19604]: audit 2024-02-12T16:36:37.472051+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.121:0/2632038650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:38.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:38.139 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:38.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:37 smithi123 bash[19119]: audit 2024-02-12T16:36:37.407276+0000 mgr.y (mgr.14146) 1563 : audit [DBG] from='client.26687 v1:172.21.15.121:0/43720742' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:38.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:37 smithi123 bash[19119]: audit 2024-02-12T16:36:37.472051+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? v1:172.21.15.121:0/2632038650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:38.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:38.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:38.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:38.407 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:38.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:38.537 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:38.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:38.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:38.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:38.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:38.862 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:36:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:38 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.143951+0000 mgr.y (mgr.14146) 1564 : cluster [DBG] pgmap v2240: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 17384/247618 objects misplaced (7.020%); 24 MiB/s, 386 objects/s recovering 2024-02-12T16:36:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:38 smithi121 bash[18489]: audit 2024-02-12T16:36:38.355047+0000 mon.a (mon.0) 2728 : audit [INF] from='client.26641 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:38 smithi121 bash[18489]: audit 2024-02-12T16:36:38.355774+0000 mon.c (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.121:0/85255432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:38.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:38.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:38.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:39.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:39.116 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:39.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:39.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:38 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.143951+0000 mgr.y (mgr.14146) 1564 : cluster [DBG] pgmap v2240: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 17384/247618 objects misplaced (7.020%); 24 MiB/s, 386 objects/s recovering 2024-02-12T16:36:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:38 smithi123 bash[19119]: audit 2024-02-12T16:36:38.355047+0000 mon.a (mon.0) 2728 : audit [INF] from='client.26641 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:38 smithi123 bash[19119]: audit 2024-02-12T16:36:38.355774+0000 mon.c (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.121:0/85255432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:39.225 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:39.231 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:38 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.143951+0000 mgr.y (mgr.14146) 1564 : cluster [DBG] pgmap v2240: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 17384/247618 objects misplaced (7.020%); 24 MiB/s, 386 objects/s recovering 2024-02-12T16:36:39.232 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:38 smithi169 bash[19604]: audit 2024-02-12T16:36:38.355047+0000 mon.a (mon.0) 2728 : audit [INF] from='client.26641 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:39.232 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:38 smithi169 bash[19604]: audit 2024-02-12T16:36:38.355774+0000 mon.c (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.121:0/85255432' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:36:39.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:39.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:39.354 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:39.361 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-12T16:36:39.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight-by-utilization 1000 3.0 2024-02-12T16:36:39.378 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:39.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:39.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:39.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:39.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:39.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:39.641 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:39.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:39.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:39.779 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:39.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:39.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:39.928 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:39.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:40.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:40.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:40.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.152942+0000 osd.9 (osd.9) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.152981+0000 osd.9 (osd.9) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.813738+0000 osd.2 (osd.2) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.813785+0000 osd.2 (osd.2) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.840082+0000 mon.a (mon.0) 2729 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:36:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.840120+0000 mon.a (mon.0) 2730 : cluster [INF] Cluster is now healthy 2024-02-12T16:36:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: audit 2024-02-12T16:36:38.851375+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26641 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:36:40.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:39 smithi123 bash[19119]: cluster 2024-02-12T16:36:38.851482+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e801: 12 total, 12 up, 12 in 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.152942+0000 osd.9 (osd.9) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.152981+0000 osd.9 (osd.9) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.813738+0000 osd.2 (osd.2) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.813785+0000 osd.2 (osd.2) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.840082+0000 mon.a (mon.0) 2729 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.840120+0000 mon.a (mon.0) 2730 : cluster [INF] Cluster is now healthy 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: audit 2024-02-12T16:36:38.851375+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26641 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:36:40.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:39 smithi121 bash[18489]: cluster 2024-02-12T16:36:38.851482+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e801: 12 total, 12 up, 12 in 2024-02-12T16:36:40.169 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:40.198 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:40.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.152942+0000 osd.9 (osd.9) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.152981+0000 osd.9 (osd.9) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.813738+0000 osd.2 (osd.2) 115 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.813785+0000 osd.2 (osd.2) 116 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.840082+0000 mon.a (mon.0) 2729 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:36:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.840120+0000 mon.a (mon.0) 2730 : cluster [INF] Cluster is now healthy 2024-02-12T16:36:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: audit 2024-02-12T16:36:38.851375+0000 mon.a (mon.0) 2731 : audit [INF] from='client.26641 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:36:40.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:39 smithi169 bash[19604]: cluster 2024-02-12T16:36:38.851482+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e801: 12 total, 12 up, 12 in 2024-02-12T16:36:40.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:40.330 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:40.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:40.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:40.436 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:40.463 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:40.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:40.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:36:40.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:40.600 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:40.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:40.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:40.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:40.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:40.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:40.871 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:40.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:40.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:41.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:41.009 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:40 smithi121 bash[18489]: cluster 2024-02-12T16:36:40.144442+0000 mgr.y (mgr.14146) 1565 : cluster [DBG] pgmap v2242: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 15468/243102 objects misplaced (6.363%); 26 MiB/s, 412 objects/s recovering 2024-02-12T16:36:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:41.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:41.135 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:41.136 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:40 smithi123 bash[19119]: cluster 2024-02-12T16:36:40.144442+0000 mgr.y (mgr.14146) 1565 : cluster [DBG] pgmap v2242: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 15468/243102 objects misplaced (6.363%); 26 MiB/s, 412 objects/s recovering 2024-02-12T16:36:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:41.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:41.245 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:40 smithi169 bash[19604]: cluster 2024-02-12T16:36:40.144442+0000 mgr.y (mgr.14146) 1565 : cluster [DBG] pgmap v2242: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 15468/243102 objects misplaced (6.363%); 26 MiB/s, 412 objects/s recovering 2024-02-12T16:36:41.264 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:41.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:41.395 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:41.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:41.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:41.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:41.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:41.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:41.787 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:41.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:41.917 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:41.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:42.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:42.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:42.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:42.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:42.188 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:42.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:42.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:42.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:42.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:42.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:42.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:42.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:42.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:42.606 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:42.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:42.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:42.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:42.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:36:42.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:42.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:36:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:42.907 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:42.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:43.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:43.040 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:43.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:43.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:43.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:43.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:43.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:43.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:43.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:43 smithi123 bash[19119]: cluster 2024-02-12T16:36:42.144919+0000 mgr.y (mgr.14146) 1566 : cluster [DBG] pgmap v2243: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 14714/237730 objects misplaced (6.189%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:36:43.306 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:43.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:43.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:43.437 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:43.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:43.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:43 smithi169 bash[19604]: cluster 2024-02-12T16:36:42.144919+0000 mgr.y (mgr.14146) 1566 : cluster [DBG] pgmap v2243: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 14714/237730 objects misplaced (6.189%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:36:43.565 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:43.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:43.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:43 smithi121 bash[18489]: cluster 2024-02-12T16:36:42.144919+0000 mgr.y (mgr.14146) 1566 : cluster [DBG] pgmap v2243: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 14714/237730 objects misplaced (6.189%); 25 MiB/s, 397 objects/s recovering 2024-02-12T16:36:43.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:43.736 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:43.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:43.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:43.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:43.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:43.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:44.002 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:44.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:44.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:44.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:44.279 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:44.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:44.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:44.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:44.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:44.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:44.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:44.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:44.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:44.945 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:44.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:45.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:45.087 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:45.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:45.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:45.218 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:45.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:45.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:45.329 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:45 smithi123 bash[19119]: cluster 2024-02-12T16:36:44.145353+0000 mgr.y (mgr.14146) 1567 : cluster [DBG] pgmap v2244: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 14353/235830 objects misplaced (6.086%); 25 MiB/s, 402 objects/s recovering 2024-02-12T16:36:45.347 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:45.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:45.451 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:45.457 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:45 smithi169 bash[19604]: cluster 2024-02-12T16:36:44.145353+0000 mgr.y (mgr.14146) 1567 : cluster [DBG] pgmap v2244: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 14353/235830 objects misplaced (6.086%); 25 MiB/s, 402 objects/s recovering 2024-02-12T16:36:45.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:45.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:45.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:45.604 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:45.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:45.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:45 smithi121 bash[18489]: cluster 2024-02-12T16:36:44.145353+0000 mgr.y (mgr.14146) 1567 : cluster [DBG] pgmap v2244: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 14353/235830 objects misplaced (6.086%); 25 MiB/s, 402 objects/s recovering 2024-02-12T16:36:45.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:45.736 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:45.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:45.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:45.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:45.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:45.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:45.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:46.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:46.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:46.258 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:46.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:46.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:46.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:46.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:46.578 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:46.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:46.705 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:46.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:46.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:46.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:46.969 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:46.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:47.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:47.082 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:46 smithi169 bash[19604]: cluster 2024-02-12T16:36:46.145746+0000 mgr.y (mgr.14146) 1568 : cluster [DBG] pgmap v2245: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 13068/232544 objects misplaced (5.620%); 24 MiB/s, 376 objects/s recovering 2024-02-12T16:36:47.104 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:47.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:47.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:46 smithi123 bash[19119]: cluster 2024-02-12T16:36:46.145746+0000 mgr.y (mgr.14146) 1568 : cluster [DBG] pgmap v2245: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 13068/232544 objects misplaced (5.620%); 24 MiB/s, 376 objects/s recovering 2024-02-12T16:36:47.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:46 smithi121 bash[18489]: cluster 2024-02-12T16:36:46.145746+0000 mgr.y (mgr.14146) 1568 : cluster [DBG] pgmap v2245: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 13068/232544 objects misplaced (5.620%); 24 MiB/s, 376 objects/s recovering 2024-02-12T16:36:47.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:47.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:47.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:47.271 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:47.276 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:36:47.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:47.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:47.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:47.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:47.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:47.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:47.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:47.650 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:47.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:47.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:47.789 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:47.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:47.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:47.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:47.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:48.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:48.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:48.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:48.174 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:48.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:48.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:48.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:48.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:48.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:48.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:48.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:48.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:48.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:48.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:48.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:48.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:48.764 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:48.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:48.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:48.899 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:48.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:49.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:49.035 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:49.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:49.178 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:49.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:49.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:49.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:49.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:49.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:49.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:49 smithi169 bash[19604]: cluster 2024-02-12T16:36:48.146202+0000 mgr.y (mgr.14146) 1569 : cluster [DBG] pgmap v2246: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 12266/226244 objects misplaced (5.422%); 24 MiB/s, 383 objects/s recovering 2024-02-12T16:36:49.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:49 smithi123 bash[19119]: cluster 2024-02-12T16:36:48.146202+0000 mgr.y (mgr.14146) 1569 : cluster [DBG] pgmap v2246: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 12266/226244 objects misplaced (5.422%); 24 MiB/s, 383 objects/s recovering 2024-02-12T16:36:49.577 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:49.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:49 smithi121 bash[18489]: cluster 2024-02-12T16:36:48.146202+0000 mgr.y (mgr.14146) 1569 : cluster [DBG] pgmap v2246: 47 pgs: 3 active+remapped+backfilling, 44 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 12266/226244 objects misplaced (5.422%); 24 MiB/s, 383 objects/s recovering 2024-02-12T16:36:49.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:49.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:49.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:49.855 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:49.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:49.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:49.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:50.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:50.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:50.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:50.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:50.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:50.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:50.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:50.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:50.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:50.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:50.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:50.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:50.491 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:50 smithi121 bash[18489]: cluster 2024-02-12T16:36:49.207062+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e802: 12 total, 12 up, 12 in 2024-02-12T16:36:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:50.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:50 smithi169 bash[19604]: cluster 2024-02-12T16:36:49.207062+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e802: 12 total, 12 up, 12 in 2024-02-12T16:36:50.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:50.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:50 smithi123 bash[19119]: cluster 2024-02-12T16:36:49.207062+0000 mon.a (mon.0) 2733 : cluster [DBG] osdmap e802: 12 total, 12 up, 12 in 2024-02-12T16:36:50.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:50.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:50.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:50.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:50.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:50.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:50.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:50.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:50.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:51.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:51.063 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:51.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:51.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:51.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:51 smithi169 bash[19604]: cluster 2024-02-12T16:36:50.019845+0000 osd.11 (osd.11) 181 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:36:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:51 smithi169 bash[19604]: cluster 2024-02-12T16:36:50.023251+0000 osd.11 (osd.11) 182 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:36:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:51 smithi169 bash[19604]: cluster 2024-02-12T16:36:50.146592+0000 mgr.y (mgr.14146) 1570 : cluster [DBG] pgmap v2248: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 10849/221844 objects misplaced (4.890%); 21 MiB/s, 343 objects/s recovering 2024-02-12T16:36:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:51 smithi169 bash[19604]: cluster 2024-02-12T16:36:50.213836+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e803: 12 total, 12 up, 12 in 2024-02-12T16:36:51.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:51.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:51.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:51.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:51.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:51.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:51.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:51.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:51 smithi121 bash[18489]: cluster 2024-02-12T16:36:50.019845+0000 osd.11 (osd.11) 181 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:36:51.596 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:51 smithi121 bash[18489]: cluster 2024-02-12T16:36:50.023251+0000 osd.11 (osd.11) 182 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:36:51.596 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:51 smithi121 bash[18489]: cluster 2024-02-12T16:36:50.146592+0000 mgr.y (mgr.14146) 1570 : cluster [DBG] pgmap v2248: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 10849/221844 objects misplaced (4.890%); 21 MiB/s, 343 objects/s recovering 2024-02-12T16:36:51.596 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:51 smithi121 bash[18489]: cluster 2024-02-12T16:36:50.213836+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e803: 12 total, 12 up, 12 in 2024-02-12T16:36:51.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:51.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:51 smithi123 bash[19119]: cluster 2024-02-12T16:36:50.019845+0000 osd.11 (osd.11) 181 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:36:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:51 smithi123 bash[19119]: cluster 2024-02-12T16:36:50.023251+0000 osd.11 (osd.11) 182 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:36:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:51 smithi123 bash[19119]: cluster 2024-02-12T16:36:50.146592+0000 mgr.y (mgr.14146) 1570 : cluster [DBG] pgmap v2248: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 10849/221844 objects misplaced (4.890%); 21 MiB/s, 343 objects/s recovering 2024-02-12T16:36:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:51 smithi123 bash[19119]: cluster 2024-02-12T16:36:50.213836+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e803: 12 total, 12 up, 12 in 2024-02-12T16:36:51.726 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:51.753 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:51.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:51.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:51.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:51.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:51.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:52.023 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:52.127 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:52.158 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:52.225 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:52 smithi169 bash[19604]: cluster 2024-02-12T16:36:51.089971+0000 osd.10 (osd.10) 166 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:36:52.225 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:52 smithi169 bash[19604]: cluster 2024-02-12T16:36:51.114415+0000 osd.10 (osd.10) 167 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:36:52.261 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:52.268 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:52 smithi123 bash[19119]: cluster 2024-02-12T16:36:51.089971+0000 osd.10 (osd.10) 166 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:36:52.268 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:52 smithi123 bash[19119]: cluster 2024-02-12T16:36:51.114415+0000 osd.10 (osd.10) 167 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:36:52.287 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:52.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:52.400 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:52 smithi121 bash[18489]: cluster 2024-02-12T16:36:51.089971+0000 osd.10 (osd.10) 166 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:36:52.400 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:52 smithi121 bash[18489]: cluster 2024-02-12T16:36:51.114415+0000 osd.10 (osd.10) 167 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:36:52.446 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:52.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:52.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:52.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:52.704 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:52.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:52.842 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:52.945 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:52.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:52.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:53.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:53.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:53.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:53.215 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:53.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:53 smithi121 bash[18489]: cluster 2024-02-12T16:36:52.049886+0000 osd.10 (osd.10) 168 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:36:53.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:53 smithi121 bash[18489]: cluster 2024-02-12T16:36:52.082043+0000 osd.10 (osd.10) 169 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:36:53.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:53 smithi121 bash[18489]: cluster 2024-02-12T16:36:52.147078+0000 mgr.y (mgr.14146) 1571 : cluster [DBG] pgmap v2250: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 9710/217214 objects misplaced (4.470%); 30 MiB/s, 476 objects/s recovering 2024-02-12T16:36:53.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:53 smithi121 bash[18489]: cluster 2024-02-12T16:36:52.865752+0000 osd.8 (osd.8) 93 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:36:53.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:53 smithi121 bash[18489]: cluster 2024-02-12T16:36:52.969820+0000 osd.8 (osd.8) 94 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:36:53.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:53.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:53.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:53.380 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:53.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:53.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:53.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:53.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:53.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:53 smithi169 bash[19604]: cluster 2024-02-12T16:36:52.049886+0000 osd.10 (osd.10) 168 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:36:53.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:53 smithi169 bash[19604]: cluster 2024-02-12T16:36:52.082043+0000 osd.10 (osd.10) 169 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:36:53.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:53 smithi169 bash[19604]: cluster 2024-02-12T16:36:52.147078+0000 mgr.y (mgr.14146) 1571 : cluster [DBG] pgmap v2250: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 9710/217214 objects misplaced (4.470%); 30 MiB/s, 476 objects/s recovering 2024-02-12T16:36:53.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:53 smithi169 bash[19604]: cluster 2024-02-12T16:36:52.865752+0000 osd.8 (osd.8) 93 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:36:53.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:53 smithi169 bash[19604]: cluster 2024-02-12T16:36:52.969820+0000 osd.8 (osd.8) 94 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:36:53.618 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:53.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:53 smithi123 bash[19119]: cluster 2024-02-12T16:36:52.049886+0000 osd.10 (osd.10) 168 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:36:53.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:53 smithi123 bash[19119]: cluster 2024-02-12T16:36:52.082043+0000 osd.10 (osd.10) 169 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:36:53.626 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:53 smithi123 bash[19119]: cluster 2024-02-12T16:36:52.147078+0000 mgr.y (mgr.14146) 1571 : cluster [DBG] pgmap v2250: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 9710/217214 objects misplaced (4.470%); 30 MiB/s, 476 objects/s recovering 2024-02-12T16:36:53.627 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:53 smithi123 bash[19119]: cluster 2024-02-12T16:36:52.865752+0000 osd.8 (osd.8) 93 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:36:53.627 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:53 smithi123 bash[19119]: cluster 2024-02-12T16:36:52.969820+0000 osd.8 (osd.8) 94 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:36:53.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:53.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:53.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:53.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:53.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:53.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:53.936 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:53.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:54.040 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:54.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:54.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:54.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:54.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:54.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:54.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:54.347 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:54.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:54.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:54.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:54.480 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:54.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:54.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:54.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:54.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:54.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:54.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:54.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:54.874 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:54.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:54.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:55.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:55.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:55.149 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:55.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:55.284 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:55.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:55.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:55 smithi121 bash[18489]: cluster 2024-02-12T16:36:54.147501+0000 mgr.y (mgr.14146) 1572 : cluster [DBG] pgmap v2251: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 9380/215376 objects misplaced (4.355%); 21 MiB/s, 335 objects/s recovering 2024-02-12T16:36:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:55 smithi121 bash[18489]: cluster 2024-02-12T16:36:54.888569+0000 osd.8 (osd.8) 95 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:36:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:55 smithi121 bash[18489]: cluster 2024-02-12T16:36:54.947874+0000 osd.8 (osd.8) 96 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:36:55.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:55.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:55.524 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:55 smithi169 bash[19604]: cluster 2024-02-12T16:36:54.147501+0000 mgr.y (mgr.14146) 1572 : cluster [DBG] pgmap v2251: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 9380/215376 objects misplaced (4.355%); 21 MiB/s, 335 objects/s recovering 2024-02-12T16:36:55.525 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:55 smithi169 bash[19604]: cluster 2024-02-12T16:36:54.888569+0000 osd.8 (osd.8) 95 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:36:55.525 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:55 smithi169 bash[19604]: cluster 2024-02-12T16:36:54.947874+0000 osd.8 (osd.8) 96 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:36:55.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:36:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:36:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:55 smithi123 bash[19119]: cluster 2024-02-12T16:36:54.147501+0000 mgr.y (mgr.14146) 1572 : cluster [DBG] pgmap v2251: 47 pgs: 1 peering, 2 active+remapped+backfilling, 44 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 9380/215376 objects misplaced (4.355%); 21 MiB/s, 335 objects/s recovering 2024-02-12T16:36:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:55 smithi123 bash[19119]: cluster 2024-02-12T16:36:54.888569+0000 osd.8 (osd.8) 95 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:36:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:55 smithi123 bash[19119]: cluster 2024-02-12T16:36:54.947874+0000 osd.8 (osd.8) 96 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:36:55.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:55.681 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:55.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:55.915 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:36:55.915 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:36:55.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:55.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:55.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:56.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:56.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:56.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:56.178 INFO:teuthology.orchestra.run.smithi121.stdout:moved 0 / 94 (0%) 2024-02-12T16:36:56.178 INFO:teuthology.orchestra.run.smithi121.stdout:avg 7.83333 2024-02-12T16:36:56.178 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 1.28019 -> 1.28019 (expected baseline 2.67966) 2024-02-12T16:36:56.178 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.0 with 6 -> 6 pgs (0.765957 -> 0.765957 * mean) 2024-02-12T16:36:56.179 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.6 with 10 -> 10 pgs (1.2766 -> 1.2766 * mean) 2024-02-12T16:36:56.179 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:36:56.179 INFO:teuthology.orchestra.run.smithi121.stdout:oload 1000 2024-02-12T16:36:56.179 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 3 2024-02-12T16:36:56.179 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:36:56.179 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0194 2024-02-12T16:36:56.179 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.1939 2024-02-12T16:36:56.182 INFO:teuthology.orchestra.run.smithi121.stderr:no change 2024-02-12T16:36:56.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:56.218 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:56.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:56.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:56.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:56.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:56.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:56.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:56.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:56.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:56.646 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:56.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:56.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:56.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:56.779 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:56.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:56.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:56.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:56 smithi169 bash[19604]: cluster 2024-02-12T16:36:55.826217+0000 osd.3 (osd.3) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:56.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:56 smithi169 bash[19604]: cluster 2024-02-12T16:36:55.826287+0000 osd.3 (osd.3) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:56.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:56 smithi169 bash[19604]: cluster 2024-02-12T16:36:55.896710+0000 osd.8 (osd.8) 97 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:36:56.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:56 smithi169 bash[19604]: cluster 2024-02-12T16:36:55.992470+0000 osd.8 (osd.8) 98 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:36:56.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:56 smithi169 bash[19604]: cluster 2024-02-12T16:36:56.147962+0000 mgr.y (mgr.14146) 1573 : cluster [DBG] pgmap v2252: 47 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 44 active+clean; 6.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 9380/211956 objects misplaced (4.425%); 16 MiB/s, 263 objects/s recovering 2024-02-12T16:36:56.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:56 smithi169 bash[19604]: audit 2024-02-12T16:36:56.173262+0000 mgr.y (mgr.14146) 1574 : audit [DBG] from='client.16809 v1:172.21.15.121:0/1603614990' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:56.892 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:56 smithi169 bash[19604]: cluster 2024-02-12T16:36:56.229415+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e804: 12 total, 12 up, 12 in 2024-02-12T16:36:56.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:56.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:57.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:57.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:56 smithi123 bash[19119]: cluster 2024-02-12T16:36:55.826217+0000 osd.3 (osd.3) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:56 smithi123 bash[19119]: cluster 2024-02-12T16:36:55.826287+0000 osd.3 (osd.3) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:56 smithi123 bash[19119]: cluster 2024-02-12T16:36:55.896710+0000 osd.8 (osd.8) 97 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:36:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:56 smithi123 bash[19119]: cluster 2024-02-12T16:36:55.992470+0000 osd.8 (osd.8) 98 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:36:57.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:56 smithi123 bash[19119]: cluster 2024-02-12T16:36:56.147962+0000 mgr.y (mgr.14146) 1573 : cluster [DBG] pgmap v2252: 47 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 44 active+clean; 6.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 9380/211956 objects misplaced (4.425%); 16 MiB/s, 263 objects/s recovering 2024-02-12T16:36:57.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:56 smithi123 bash[19119]: audit 2024-02-12T16:36:56.173262+0000 mgr.y (mgr.14146) 1574 : audit [DBG] from='client.16809 v1:172.21.15.121:0/1603614990' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:57.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:56 smithi123 bash[19119]: cluster 2024-02-12T16:36:56.229415+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e804: 12 total, 12 up, 12 in 2024-02-12T16:36:57.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:56 smithi121 bash[18489]: cluster 2024-02-12T16:36:55.826217+0000 osd.3 (osd.3) 76 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:36:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:56 smithi121 bash[18489]: cluster 2024-02-12T16:36:55.826287+0000 osd.3 (osd.3) 77 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:36:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:56 smithi121 bash[18489]: cluster 2024-02-12T16:36:55.896710+0000 osd.8 (osd.8) 97 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:36:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:56 smithi121 bash[18489]: cluster 2024-02-12T16:36:55.992470+0000 osd.8 (osd.8) 98 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:36:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:56 smithi121 bash[18489]: cluster 2024-02-12T16:36:56.147962+0000 mgr.y (mgr.14146) 1573 : cluster [DBG] pgmap v2252: 47 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 44 active+clean; 6.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 9380/211956 objects misplaced (4.425%); 16 MiB/s, 263 objects/s recovering 2024-02-12T16:36:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:56 smithi121 bash[18489]: audit 2024-02-12T16:36:56.173262+0000 mgr.y (mgr.14146) 1574 : audit [DBG] from='client.16809 v1:172.21.15.121:0/1603614990' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:36:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:56 smithi121 bash[18489]: cluster 2024-02-12T16:36:56.229415+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e804: 12 total, 12 up, 12 in 2024-02-12T16:36:57.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:57.198 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:57.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:57.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:57.339 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:36:57.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:57.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:36:57.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:57.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:57.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:57.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:57.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:57.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:57.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:36:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:57.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:36:57.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:57.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:36:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:57.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:36:57.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:58.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:58.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:58.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:36:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:36:58.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:58.260 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:57 smithi121 bash[18489]: cluster 2024-02-12T16:36:57.013190+0000 osd.10 (osd.10) 170 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:36:58.260 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:57 smithi121 bash[18489]: cluster 2024-02-12T16:36:57.057475+0000 osd.10 (osd.10) 171 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:36:58.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:57 smithi169 bash[19604]: cluster 2024-02-12T16:36:57.013190+0000 osd.10 (osd.10) 170 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:36:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:57 smithi169 bash[19604]: cluster 2024-02-12T16:36:57.057475+0000 osd.10 (osd.10) 171 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:36:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:57 smithi123 bash[19119]: cluster 2024-02-12T16:36:57.013190+0000 osd.10 (osd.10) 170 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:36:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:57 smithi123 bash[19119]: cluster 2024-02-12T16:36:57.057475+0000 osd.10 (osd.10) 171 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:36:58.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:58.427 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:58.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:36:58.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:58.555 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:58.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:58.569 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-12T16:36:58.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight-by-utilization 1000 3.0 2024-02-12T16:36:58.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:58.721 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:58.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:58.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:36:58.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:36:58.930 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:36:58.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:58.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:36:58.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:36:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:58.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:36:59.018 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:36:59.062 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:36:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:36:59.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:59.122 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:36:59.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:59.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:59.153 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:36:59.182 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:36:59.215 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:36:59.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:59.245 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:36:59.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:59 smithi123 bash[19119]: cluster 2024-02-12T16:36:57.946980+0000 osd.8 (osd.8) 99 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:36:59.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:59 smithi123 bash[19119]: cluster 2024-02-12T16:36:57.953139+0000 osd.8 (osd.8) 100 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:36:59.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:59 smithi123 bash[19119]: cluster 2024-02-12T16:36:58.148438+0000 mgr.y (mgr.14146) 1575 : cluster [DBG] pgmap v2254: 47 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 44 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 8660/205548 objects misplaced (4.213%); 12 MiB/s, 199 objects/s recovering 2024-02-12T16:36:59.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:59 smithi123 bash[19119]: audit 2024-02-12T16:36:58.586605+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:36:59.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:36:59 smithi123 bash[19119]: audit 2024-02-12T16:36:58.824449+0000 mon.b (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.121:0/2287063788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:59.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:36:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:59.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:36:59.277 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:36:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:59 smithi169 bash[19604]: cluster 2024-02-12T16:36:57.946980+0000 osd.8 (osd.8) 99 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:36:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:59 smithi169 bash[19604]: cluster 2024-02-12T16:36:57.953139+0000 osd.8 (osd.8) 100 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:36:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:59 smithi169 bash[19604]: cluster 2024-02-12T16:36:58.148438+0000 mgr.y (mgr.14146) 1575 : cluster [DBG] pgmap v2254: 47 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 44 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 8660/205548 objects misplaced (4.213%); 12 MiB/s, 199 objects/s recovering 2024-02-12T16:36:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:59 smithi169 bash[19604]: audit 2024-02-12T16:36:58.586605+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:36:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:36:59 smithi169 bash[19604]: audit 2024-02-12T16:36:58.824449+0000 mon.b (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.121:0/2287063788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:59.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:59 smithi121 bash[18489]: cluster 2024-02-12T16:36:57.946980+0000 osd.8 (osd.8) 99 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:36:59.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:59 smithi121 bash[18489]: cluster 2024-02-12T16:36:57.953139+0000 osd.8 (osd.8) 100 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:36:59.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:59 smithi121 bash[18489]: cluster 2024-02-12T16:36:58.148438+0000 mgr.y (mgr.14146) 1575 : cluster [DBG] pgmap v2254: 47 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 44 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 8660/205548 objects misplaced (4.213%); 12 MiB/s, 199 objects/s recovering 2024-02-12T16:36:59.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:59 smithi121 bash[18489]: audit 2024-02-12T16:36:58.586605+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:36:59.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:36:59 smithi121 bash[18489]: audit 2024-02-12T16:36:58.824449+0000 mon.b (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.121:0/2287063788' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:36:59.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:59.400 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:36:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:59.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:36:59.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:59.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:36:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:59.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:36:59.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:59.702 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:36:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:59.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:36:59.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:59.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:36:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:59.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:36:59.951 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:59.979 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:36:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:36:59.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:37:00.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:00.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:00.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:00.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:00.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: cluster 2024-02-12T16:36:58.662941+0000 osd.5 (osd.5) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:00.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: cluster 2024-02-12T16:36:58.662992+0000 osd.5 (osd.5) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:00.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: cluster 2024-02-12T16:36:59.017414+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e805: 12 total, 12 up, 12 in 2024-02-12T16:37:00.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: audit 2024-02-12T16:36:59.042852+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:37:00.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: audit 2024-02-12T16:36:59.043782+0000 mon.c (mon.1) 86 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:37:00.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: audit 2024-02-12T16:36:59.048786+0000 mon.a (mon.0) 2737 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:37:00.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: audit 2024-02-12T16:36:59.740824+0000 mon.a (mon.0) 2738 : audit [INF] from='client.26711 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:37:00.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:00 smithi169 bash[19604]: audit 2024-02-12T16:36:59.742431+0000 mon.b (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.121:0/3383990572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:37:00.268 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:00.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:00.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:00.379 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: cluster 2024-02-12T16:36:58.662941+0000 osd.5 (osd.5) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:00.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: cluster 2024-02-12T16:36:58.662992+0000 osd.5 (osd.5) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:00.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: cluster 2024-02-12T16:36:59.017414+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e805: 12 total, 12 up, 12 in 2024-02-12T16:37:00.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: audit 2024-02-12T16:36:59.042852+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:37:00.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: audit 2024-02-12T16:36:59.043782+0000 mon.c (mon.1) 86 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:37:00.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: audit 2024-02-12T16:36:59.048786+0000 mon.a (mon.0) 2737 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:37:00.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: audit 2024-02-12T16:36:59.740824+0000 mon.a (mon.0) 2738 : audit [INF] from='client.26711 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:37:00.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:00 smithi121 bash[18489]: audit 2024-02-12T16:36:59.742431+0000 mon.b (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.121:0/3383990572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:37:00.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: cluster 2024-02-12T16:36:58.662941+0000 osd.5 (osd.5) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:00.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: cluster 2024-02-12T16:36:58.662992+0000 osd.5 (osd.5) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:00.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: cluster 2024-02-12T16:36:59.017414+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e805: 12 total, 12 up, 12 in 2024-02-12T16:37:00.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: audit 2024-02-12T16:36:59.042852+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:37:00.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: audit 2024-02-12T16:36:59.043782+0000 mon.c (mon.1) 86 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:37:00.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: audit 2024-02-12T16:36:59.048786+0000 mon.a (mon.0) 2737 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:37:00.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: audit 2024-02-12T16:36:59.740824+0000 mon.a (mon.0) 2738 : audit [INF] from='client.26711 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:37:00.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:00 smithi123 bash[19119]: audit 2024-02-12T16:36:59.742431+0000 mon.b (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.121:0/3383990572' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:37:00.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:00.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:00.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:00.561 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:00.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:00.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:00.698 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:00.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:00.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:00.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:00.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:01.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:01.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:01.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:01 smithi121 bash[18489]: cluster 2024-02-12T16:37:00.046174+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:01.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:01 smithi121 bash[18489]: audit 2024-02-12T16:37:00.054691+0000 mon.a (mon.0) 2740 : audit [INF] from='client.26711 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:37:01.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:01 smithi121 bash[18489]: cluster 2024-02-12T16:37:00.054803+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e806: 12 total, 12 up, 12 in 2024-02-12T16:37:01.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:01 smithi121 bash[18489]: cluster 2024-02-12T16:37:00.148898+0000 mgr.y (mgr.14146) 1576 : cluster [DBG] pgmap v2257: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 8292/200724 objects misplaced (4.131%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:37:01.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:01.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:01.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:01 smithi169 bash[19604]: cluster 2024-02-12T16:37:00.046174+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:01 smithi169 bash[19604]: audit 2024-02-12T16:37:00.054691+0000 mon.a (mon.0) 2740 : audit [INF] from='client.26711 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:37:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:01 smithi169 bash[19604]: cluster 2024-02-12T16:37:00.054803+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e806: 12 total, 12 up, 12 in 2024-02-12T16:37:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:01 smithi169 bash[19604]: cluster 2024-02-12T16:37:00.148898+0000 mgr.y (mgr.14146) 1576 : cluster [DBG] pgmap v2257: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 8292/200724 objects misplaced (4.131%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:37:01.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:01.374 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:01.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:01 smithi123 bash[19119]: cluster 2024-02-12T16:37:00.046174+0000 mon.a (mon.0) 2739 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:01 smithi123 bash[19119]: audit 2024-02-12T16:37:00.054691+0000 mon.a (mon.0) 2740 : audit [INF] from='client.26711 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:37:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:01 smithi123 bash[19119]: cluster 2024-02-12T16:37:00.054803+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e806: 12 total, 12 up, 12 in 2024-02-12T16:37:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:01 smithi123 bash[19119]: cluster 2024-02-12T16:37:00.148898+0000 mgr.y (mgr.14146) 1576 : cluster [DBG] pgmap v2257: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 8292/200724 objects misplaced (4.131%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:37:01.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:01.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:01.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:01.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:01.652 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:01.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:01.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:01.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:01.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:01.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:01.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:01.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:02.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:02.064 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:02.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:02.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:02.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:02.328 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:02.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:02.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:37:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:02.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:02.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:02.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:02.722 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:02.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:02.854 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:02.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:02.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:03.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:03.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:03.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:03.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:03.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:37:03.252 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:03 smithi121 bash[18489]: cluster 2024-02-12T16:37:02.149358+0000 mgr.y (mgr.14146) 1577 : cluster [DBG] pgmap v2258: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 7601/195258 objects misplaced (3.893%); 13 MiB/s, 201 objects/s recovering 2024-02-12T16:37:03.269 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:03.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:03.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:03.421 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:03.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:03.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:03.539 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:03 smithi169 bash[19604]: cluster 2024-02-12T16:37:02.149358+0000 mgr.y (mgr.14146) 1577 : cluster [DBG] pgmap v2258: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 7601/195258 objects misplaced (3.893%); 13 MiB/s, 201 objects/s recovering 2024-02-12T16:37:03.556 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:03.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:03 smithi123 bash[19119]: cluster 2024-02-12T16:37:02.149358+0000 mgr.y (mgr.14146) 1577 : cluster [DBG] pgmap v2258: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.39k op/s; 7601/195258 objects misplaced (3.893%); 13 MiB/s, 201 objects/s recovering 2024-02-12T16:37:03.661 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:03.696 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:03.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:03.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:03.881 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:03.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:03.977 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:03.983 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:03.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:04.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:04.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:04.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:04.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:04.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:04.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:04.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:04.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:04.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:04.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:04.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:04.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:04.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:04.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:05.135 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:05.167 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:05.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:05.298 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:05.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:05.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:05 smithi121 bash[18489]: cluster 2024-02-12T16:37:04.149852+0000 mgr.y (mgr.14146) 1578 : cluster [DBG] pgmap v2259: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 7166/193088 objects misplaced (3.711%); 10 MiB/s, 160 objects/s recovering 2024-02-12T16:37:05.431 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:05.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:05 smithi169 bash[19604]: cluster 2024-02-12T16:37:04.149852+0000 mgr.y (mgr.14146) 1578 : cluster [DBG] pgmap v2259: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 7166/193088 objects misplaced (3.711%); 10 MiB/s, 160 objects/s recovering 2024-02-12T16:37:05.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:05 smithi123 bash[19119]: cluster 2024-02-12T16:37:04.149852+0000 mgr.y (mgr.14146) 1578 : cluster [DBG] pgmap v2259: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 7166/193088 objects misplaced (3.711%); 10 MiB/s, 160 objects/s recovering 2024-02-12T16:37:05.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:05.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:05.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:05.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:05.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:05.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:06.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:06.106 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:06.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:06.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:06.243 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:06.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:06.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:06.379 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:06.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:06.485 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:06.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:06.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:06.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:06.692 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:06.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:06.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:06.824 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:06.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:06.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:06.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:06.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:07.039 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:07.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:07.068 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:06 smithi121 bash[18489]: cluster 2024-02-12T16:37:06.150333+0000 mgr.y (mgr.14146) 1579 : cluster [DBG] pgmap v2260: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 7166/189410 objects misplaced (3.783%); 5.5 MiB/s, 87 objects/s recovering 2024-02-12T16:37:07.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:07.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:07.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:06 smithi123 bash[19119]: cluster 2024-02-12T16:37:06.150333+0000 mgr.y (mgr.14146) 1579 : cluster [DBG] pgmap v2260: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 7166/189410 objects misplaced (3.783%); 5.5 MiB/s, 87 objects/s recovering 2024-02-12T16:37:07.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:07.217 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:06 smithi169 bash[19604]: cluster 2024-02-12T16:37:06.150333+0000 mgr.y (mgr.14146) 1579 : cluster [DBG] pgmap v2260: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 7166/189410 objects misplaced (3.783%); 5.5 MiB/s, 87 objects/s recovering 2024-02-12T16:37:07.234 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:07.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:07.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:07.380 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:07.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:07.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:07.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:07.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:07.626 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:07.659 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:07.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:07.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:07.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:07.795 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:07.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:07.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:07.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:07 smithi169 bash[19604]: cluster 2024-02-12T16:37:06.490568+0000 osd.10 (osd.10) 172 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:07.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:07 smithi169 bash[19604]: cluster 2024-02-12T16:37:06.490611+0000 osd.10 (osd.10) 173 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:07.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:07 smithi169 bash[19604]: cluster 2024-02-12T16:37:06.868169+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e807: 12 total, 12 up, 12 in 2024-02-12T16:37:07.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:07 smithi169 bash[19604]: cluster 2024-02-12T16:37:06.982592+0000 osd.0 (osd.0) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:07.905 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:07 smithi169 bash[19604]: cluster 2024-02-12T16:37:06.982634+0000 osd.0 (osd.0) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:07.925 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:07.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:07.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:08.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:08.052 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:08.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:07 smithi123 bash[19119]: cluster 2024-02-12T16:37:06.490568+0000 osd.10 (osd.10) 172 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:08.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:07 smithi123 bash[19119]: cluster 2024-02-12T16:37:06.490611+0000 osd.10 (osd.10) 173 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:07 smithi123 bash[19119]: cluster 2024-02-12T16:37:06.868169+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e807: 12 total, 12 up, 12 in 2024-02-12T16:37:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:07 smithi123 bash[19119]: cluster 2024-02-12T16:37:06.982592+0000 osd.0 (osd.0) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:07 smithi123 bash[19119]: cluster 2024-02-12T16:37:06.982634+0000 osd.0 (osd.0) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:07 smithi121 bash[18489]: cluster 2024-02-12T16:37:06.490568+0000 osd.10 (osd.10) 172 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:07 smithi121 bash[18489]: cluster 2024-02-12T16:37:06.490611+0000 osd.10 (osd.10) 173 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:07 smithi121 bash[18489]: cluster 2024-02-12T16:37:06.868169+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e807: 12 total, 12 up, 12 in 2024-02-12T16:37:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:07 smithi121 bash[18489]: cluster 2024-02-12T16:37:06.982592+0000 osd.0 (osd.0) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:07 smithi121 bash[18489]: cluster 2024-02-12T16:37:06.982634+0000 osd.0 (osd.0) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:08.163 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:08.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:08.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:08.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:08.315 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:08.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:08.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:08.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:08.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:08.580 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:08.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:08.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:08.710 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:08.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:08.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:08.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:08.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:08.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:09.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:09.117 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:09.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:08 smithi123 bash[19119]: cluster 2024-02-12T16:37:07.877741+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e808: 12 total, 12 up, 12 in 2024-02-12T16:37:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:08 smithi123 bash[19119]: cluster 2024-02-12T16:37:08.150756+0000 mgr.y (mgr.14146) 1580 : cluster [DBG] pgmap v2263: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 6435/183024 objects misplaced (3.516%); 10 MiB/s, 161 objects/s recovering 2024-02-12T16:37:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:08 smithi121 bash[18489]: cluster 2024-02-12T16:37:07.877741+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e808: 12 total, 12 up, 12 in 2024-02-12T16:37:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:08 smithi121 bash[18489]: cluster 2024-02-12T16:37:08.150756+0000 mgr.y (mgr.14146) 1580 : cluster [DBG] pgmap v2263: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 6435/183024 objects misplaced (3.516%); 10 MiB/s, 161 objects/s recovering 2024-02-12T16:37:09.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:09.259 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:09.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:08 smithi169 bash[19604]: cluster 2024-02-12T16:37:07.877741+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e808: 12 total, 12 up, 12 in 2024-02-12T16:37:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:08 smithi169 bash[19604]: cluster 2024-02-12T16:37:08.150756+0000 mgr.y (mgr.14146) 1580 : cluster [DBG] pgmap v2263: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 6435/183024 objects misplaced (3.516%); 10 MiB/s, 161 objects/s recovering 2024-02-12T16:37:09.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:09.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:09.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:09.421 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:09.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:09.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:09.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:09.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:09.657 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:09.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:09.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:09.793 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:09.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:09.896 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:09.924 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:09.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:10.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:10.058 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:10.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:10.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:10.190 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:10.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:10.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:10.339 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:10.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:10.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:10.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:10.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:10.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:10.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:10.625 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:10.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:10.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:10.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:10.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:10.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:10.910 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:10.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:11.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:11.132 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:11.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:11.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:11.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:11.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:11.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:11.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:11 smithi123 bash[19119]: cluster 2024-02-12T16:37:10.151289+0000 mgr.y (mgr.14146) 1581 : cluster [DBG] pgmap v2264: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 6063/180314 objects misplaced (3.362%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:37:11.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:11.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:11.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:11.531 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:11.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:11 smithi169 bash[19604]: cluster 2024-02-12T16:37:10.151289+0000 mgr.y (mgr.14146) 1581 : cluster [DBG] pgmap v2264: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 6063/180314 objects misplaced (3.362%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:37:11.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:11.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:11 smithi121 bash[18489]: cluster 2024-02-12T16:37:10.151289+0000 mgr.y (mgr.14146) 1581 : cluster [DBG] pgmap v2264: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 6063/180314 objects misplaced (3.362%); 6.0 MiB/s, 95 objects/s recovering 2024-02-12T16:37:11.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:11.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:11.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:11.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:11.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:11.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:11.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:11.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:12.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:12.079 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:12.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:12.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:12.215 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:12.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:12.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:12.345 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:12.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:12.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:12.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:12.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:12.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:12.594 INFO:teuthology.orchestra.run.smithi121.stdout:moved 0 / 94 (0%) 2024-02-12T16:37:12.594 INFO:teuthology.orchestra.run.smithi121.stdout:avg 7.83333 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 1.28019 -> 1.28019 (expected baseline 2.67966) 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.0 with 6 -> 6 pgs (0.765957 -> 0.765957 * mean) 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.6 with 10 -> 10 pgs (1.2766 -> 1.2766 * mean) 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:oload 1000 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 3 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0173 2024-02-12T16:37:12.595 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.1727 2024-02-12T16:37:12.600 INFO:teuthology.orchestra.run.smithi121.stderr:no change 2024-02-12T16:37:12.620 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:12.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:12.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:12.748 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:12.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:12.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:12.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:12.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:12.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:13.022 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:13.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:13.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:13.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:13.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:179492 6.16 6:6c69f484:::benchmark_data_smithi169_96224_object26656:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e680)", 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:28.752991+0000", 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 344.45978722699999, 2024-02-12T16:37:13.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1067958879999999, 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 179492 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.752991+0000", 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.752991+0000", 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.752989+0000", 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4590000000000003e-06 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.752995+0000", 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9819999999999997e-06 2024-02-12T16:37:13.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.753004+0000", 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7049999999999998e-06 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.753009+0000", 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21091458099999999 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.963923+0000", 2024-02-12T16:37:13.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0474e-05 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:28.963934+0000", 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.89573373999999994 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:29.859668+0000", 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011889500000000001 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:29.859787+0000", 2024-02-12T16:37:13.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1067958879999999 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:187130 6.11 6.63dfcb91 (undecoded) ondisk+write+known_if_redirected e688)", 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:39.228234+0000", 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 333.98454423099997, 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60914430100000005, 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187130 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228234+0000", 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228234+0000", 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228232+0000", 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4909999999999999e-06 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228239+0000", 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9070000000000002e-06 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228249+0000", 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8220000000000001e-06 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228256+0000", 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8046e-05 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228304+0000", 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0199999999999997e-06 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.228311+0000", 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60905642699999996 2024-02-12T16:37:13.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.837367+0000", 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0614e-05 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.837378+0000", 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60914430100000005 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:187223 6.11 6.e6ef1df1 (undecoded) ondisk+write+known_if_redirected e688)", 2024-02-12T16:37:13.234 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:31:39.315801+0000", 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 333.896977338, 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.52162434999999996, 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187223 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315801+0000", 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315801+0000", 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315800+0000", 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9129999999999999e-06 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315803+0000", 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2660000000000003e-06 2024-02-12T16:37:13.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315807+0000", 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4029999999999999e-06 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315811+0000", 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9951999999999999e-05 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315861+0000", 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4710000000000001e-06 2024-02-12T16:37:13.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.315868+0000", 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.52154792699999997 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.837416+0000", 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8769999999999999e-06 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:31:39.837425+0000", 2024-02-12T16:37:13.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.52162434999999996 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:263626 6.16 6:6b42ac10:::benchmark_data_smithi169_96224_object110790:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:32:48.050661+0000", 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 265.16211721000002, 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51815513800000002, 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.239 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 263626 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050661+0000", 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050661+0000", 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999657 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050626+0000", 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9971000000000001e-05 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050666+0000", 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0784999999999999e-05 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050677+0000", 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8109999999999999e-06 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050683+0000", 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6153999999999999e-05 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050729+0000", 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9776000000000002e-05 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.050829+0000", 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060172899999999996 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:37:13.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.051430+0000", 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010242523 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.061673+0000", 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50693039100000004 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.568603+0000", 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011762800000000001 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.568721+0000", 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4812e-05 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.568816+0000", 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51815513800000002 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:263656 6.12 6:4d1a6202:::benchmark_data_smithi169_96224_object110820:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:32:48.078234+0000", 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 265.13454408299998, 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51251717100000005, 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 263656 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.245 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.078234+0000", 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.078234+0000", 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.078232+0000", 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1630000000000002e-06 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.078241+0000", 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48565713700000002 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.563898+0000", 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7309999999999993e-06 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.563906+0000", 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098621009999999999 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.573768+0000", 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5372999999999997e-05 2024-02-12T16:37:13.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.573853+0000", 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00046566100000000003 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4,9,11", 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.574319+0000", 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036898539999999998 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.578009+0000", 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001747983 2024-02-12T16:37:13.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.579757+0000", 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000561167 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.580318+0000", 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.010334065 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.590652+0000", 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8657999999999999e-05 2024-02-12T16:37:13.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.590681+0000", 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0388000000000004e-05 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:32:48.590751+0000", 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51251717100000005 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303055 6.17 6:e801d7f6:::benchmark_data_smithi169_96224_object150219:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.251 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811161+0000", 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40161674300001, 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.605079751, 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303055 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811161+0000", 2024-02-12T16:37:13.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811161+0000", 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811158+0000", 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1732999999999999e-05 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811169+0000", 2024-02-12T16:37:13.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8087000000000001e-05 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811187+0000", 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3346999999999999e-05 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811211+0000", 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1950000000000001e-05 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811273+0000", 2024-02-12T16:37:13.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2109999999999998e-06 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811281+0000", 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58389507500000004 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.395176+0000", 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000199543 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.395376+0000", 2024-02-12T16:37:13.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081416300000000002 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.396190+0000", 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003661938 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.399852+0000", 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.008196867 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.408049+0000", 2024-02-12T16:37:13.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080025740000000001 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416051+0000", 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5882000000000004e-05 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416117+0000", 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2190000000000001e-05 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416159+0000", 2024-02-12T16:37:13.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1772000000000002e-05 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416241+0000", 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.605079751 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303097 6.17 6:ecffd106:::benchmark_data_smithi169_96224_object150261:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811221+0000", 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40155644199999, 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.61234877300000001, 2024-02-12T16:37:13.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303097 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811221+0000", 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811221+0000", 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811221+0000", 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7780000000000001e-06 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811224+0000", 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7400000000000004e-06 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811232+0000", 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.23e-06 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811235+0000", 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8858999999999998e-05 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811314+0000", 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4279999999999998e-06 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811319+0000", 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58857671300000003 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.399895+0000", 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7357999999999999e-05 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.399993+0000", 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045102299999999999 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.400444+0000", 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0078261890000000008 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.408270+0000", 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7950000000000002e-06 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.408277+0000", 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080098089999999997 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416287+0000", 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071591170000000004 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423446+0000", 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6863999999999997e-05 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423493+0000", 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7731999999999994e-05 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423570+0000", 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.61234877300000001 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303129 6.17 6:e922441b:::benchmark_data_smithi169_96224_object150293:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811259+0000", 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40151861699999, 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.61255049699999997, 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.265 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303129 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811259+0000", 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811259+0000", 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:37:13.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811259+0000", 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7979999999999999e-06 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811262+0000", 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1983e-05 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811274+0000", 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1240000000000004e-06 2024-02-12T16:37:13.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811279+0000", 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0622000000000002e-05 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811350+0000", 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811354+0000", 2024-02-12T16:37:13.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59217924899999996 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.403533+0000", 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104199 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.403637+0000", 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00067849999999999996 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.404315+0000", 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040079599999999996 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.408323+0000", 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0043495890000000001 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.412673+0000", 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036899279999999999 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416363+0000", 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073351980000000002 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423698+0000", 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1128000000000003e-05 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423739+0000", 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0542000000000003e-05 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423810+0000", 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.61255049699999997 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303143 6.17 6:e89720c3:::benchmark_data_smithi169_96224_object150307:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811285+0000", 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.401493399, 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62004562399999996, 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303143 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811285+0000", 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811285+0000", 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811284+0000", 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6169999999999998e-06 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811287+0000", 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6620000000000002e-06 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811293+0000", 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1829999999999996e-06 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811297+0000", 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1637999999999993e-05 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811379+0000", 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3790000000000002e-06 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811382+0000", 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59698127999999995 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.408364+0000", 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014094300000000001 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.408505+0000", 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000448076 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.408953+0000", 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071054109999999998 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416058+0000", 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5950000000000001e-05 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416084+0000", 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031380999999999998 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416398+0000", 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014795234000000001 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431193+0000", 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1074e-05 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431234+0000", 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6080000000000002e-05 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431330+0000", 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62004562399999996 2024-02-12T16:37:13.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303154 6.17 6:eb455b32:::benchmark_data_smithi169_96224_object150318:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811311+0000", 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40146736200001, 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62033069900000004, 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303154 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811311+0000", 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811311+0000", 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811310+0000", 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5288e-05 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811325+0000", 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4941e-05 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811340+0000", 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6030000000000004e-06 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811346+0000", 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7812000000000001e-05 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811403+0000", 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3560000000000001e-06 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811407+0000", 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60129366500000003 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.412700+0000", 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013454299999999999 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.412835+0000", 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045888899999999998 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.413294+0000", 2024-02-12T16:37:13.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0030283599999999999 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416322+0000", 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7729999999999999e-06 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416330+0000", 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070510349999999998 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423381+0000", 2024-02-12T16:37:13.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080288640000000001 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431410+0000", 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5463999999999994e-05 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431485+0000", 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155897 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431641+0000", 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62033069900000004 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303299 6.17 6:ec1ee2aa:::benchmark_data_smithi169_96224_object150463:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811353+0000", 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40142455899999, 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62055890599999997, 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303299 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811353+0000", 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811353+0000", 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811352+0000", 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811357+0000", 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8219999999999996e-06 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811366+0000", 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7939999999999997e-06 2024-02-12T16:37:13.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811373+0000", 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1901000000000002e-05 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811435+0000", 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4510000000000001e-06 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811438+0000", 2024-02-12T16:37:13.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60498615499999997 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416424+0000", 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5548000000000001e-05 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.416520+0000", 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000502616 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.417022+0000", 2024-02-12T16:37:13.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0063682529999999999 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423391+0000", 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0233e-05 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423421+0000", 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000210919 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423632+0000", 2024-02-12T16:37:13.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080656600000000005 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431698+0000", 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1973999999999999e-05 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431770+0000", 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014276299999999999 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431912+0000", 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62055890599999997 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303331 6.17 6:ec0896ae:::benchmark_data_smithi169_96224_object150495:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811377+0000", 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40140063300001, 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62524346200000003, 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303331 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811377+0000", 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811377+0000", 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811377+0000", 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811380+0000", 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4620000000000004e-06 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811385+0000", 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.827e-06 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811389+0000", 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2892000000000006e-05 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811462+0000", 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4829999999999999e-06 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811465+0000", 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60848573800000005 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.419951+0000", 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106384 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.420058+0000", 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041537699999999998 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.420473+0000", 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031097059999999998 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423583+0000", 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5915999999999995e-05 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423669+0000", 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018951800000000001 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423858+0000", 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012531354 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436389+0000", 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9710999999999997e-05 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436459+0000", 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016163199999999999 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436621+0000", 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62524346200000003 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303414 6.17 6:ecfd6abb:::benchmark_data_smithi169_96224_object150578:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811393+0000", 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.401384992, 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.63036996899999997, 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303414 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811393+0000", 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811393+0000", 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811392+0000", 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811395+0000", 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5849999999999999e-06 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811401+0000", 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0860000000000004e-06 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811405+0000", 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1199000000000006e-05 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811486+0000", 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.343e-06 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811489+0000", 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.61238783299999999 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423877+0000", 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5079000000000002e-05 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.423972+0000", 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00042831 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.424401+0000", 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011833096 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436234+0000", 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7028999999999997e-05 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436311+0000", 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8208000000000002e-05 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436339+0000", 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052570689999999996 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.441596+0000", 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2812999999999999e-05 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.441629+0000", 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013410700000000001 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.441763+0000", 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.63036996899999997 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303428 6.17 6:eb962c49:::benchmark_data_smithi169_96224_object150592:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811410+0000", 2024-02-12T16:37:13.306 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.401368254, 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.63516610699999998, 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303428 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811410+0000", 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811410+0000", 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811409+0000", 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9720000000000001e-06 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811412+0000", 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811417+0000", 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0539999999999996e-06 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811421+0000", 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5737999999999997e-05 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811517+0000", 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3100000000000001e-06 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811520+0000", 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.61582559599999998 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.427346+0000", 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013689 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.427482+0000", 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055208600000000003 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.428035+0000", 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0082054629999999996 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436240+0000", 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00043541899999999999 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436675+0000", 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9679000000000002e-05 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436735+0000", 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051061320000000002 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.441841+0000", 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8525e-05 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.441900+0000", 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046760480000000004 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.446576+0000", 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.63516610699999998 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303431 6.17 6:ef50c930:::benchmark_data_smithi169_96224_object150595:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811434+0000", 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.401344322, 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.63555167999999995, 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.313 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303431 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811434+0000", 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811434+0000", 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811433+0000", 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7519999999999998e-06 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811437+0000", 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3219999999999998e-06 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811443+0000", 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1239999999999998e-06 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811447+0000", 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4969999999999994e-05 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811542+0000", 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2949999999999998e-06 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811545+0000", 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.62041930899999997 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.431965+0000", 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013606299999999999 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.432101+0000", 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00068168700000000005 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.432782+0000", 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040132409999999999 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436796+0000", 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2159999999999995e-06 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.436805+0000", 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0098565000000000007 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.446661+0000", 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014684499999999999 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.446808+0000", 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8085999999999999e-05 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.446866+0000", 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011904399999999999 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.446985+0000", 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.63555167999999995 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303482 6.17 6:ef1901cf:::benchmark_data_smithi169_96224_object150646:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811460+0000", 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40131767, 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.65949601899999999, 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303482 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811460+0000", 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811460+0000", 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811460+0000", 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4800000000000001e-06 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811463+0000", 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3849999999999999e-06 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811469+0000", 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811472+0000", 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5345000000000003e-05 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811567+0000", 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.728e-06 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811573+0000", 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.63563906800000003 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.447212+0000", 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000138357 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.447351+0000", 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00086026500000000003 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.448211+0000", 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012214498000000001 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.460425+0000", 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1124000000000001e-05 2024-02-12T16:37:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.460466+0000", 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01027696 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.470743+0000", 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.199e-05 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.470755+0000", 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7454000000000003e-05 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.470813+0000", 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014339499999999999 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.470956+0000", 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.65949601899999999 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303487 6.17 6:eec05fe7:::benchmark_data_smithi169_96224_object150651:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811475+0000", 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40130339999999, 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66384843999999998, 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303487 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811475+0000", 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811475+0000", 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811474+0000", 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.951e-06 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811476+0000", 2024-02-12T16:37:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9239999999999999e-06 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811479+0000", 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2500000000000001e-06 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811481+0000", 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114604 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811596+0000", 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4419999999999998e-06 2024-02-12T16:37:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811599+0000", 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64432854799999995 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.455928+0000", 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012283699999999999 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.456051+0000", 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072751700000000001 2024-02-12T16:37:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.456778+0000", 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014190825000000001 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.470969+0000", 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3893000000000001e-05 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.471013+0000", 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041520300000000001 2024-02-12T16:37:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.475165+0000", 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9148999999999999e-05 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.475184+0000", 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6415999999999998e-05 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.475221+0000", 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102433 2024-02-12T16:37:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.475323+0000", 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66384843999999998 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303538 6.17 6:ea2927d2:::benchmark_data_smithi169_96224_object150702:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811483+0000", 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40129467400001, 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66789793799999997, 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303538 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811483+0000", 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811483+0000", 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811483+0000", 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5740000000000001e-06 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811484+0000", 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8619999999999999e-06 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811487+0000", 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.044e-06 2024-02-12T16:37:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811490+0000", 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013317099999999999 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811624+0000", 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.305e-06 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811629+0000", 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64886946999999995 2024-02-12T16:37:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.460498+0000", 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012857999999999999 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.460627+0000", 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080572 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.461433+0000", 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095874610000000002 2024-02-12T16:37:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.471020+0000", 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015076900000000001 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.471171+0000", 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041771339999999999 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.475348+0000", 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003821362 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479169+0000", 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9553000000000003e-05 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479239+0000", 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142268 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479381+0000", 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66789793799999997 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303553 6.17 6:ee39f542:::benchmark_data_smithi169_96224_object150717:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811506+0000", 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40127236000001, 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66825667099999997, 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303553 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811506+0000", 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811506+0000", 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811505+0000", 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4149999999999999e-06 2024-02-12T16:37:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811508+0000", 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3339999999999998e-06 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811515+0000", 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4560000000000002e-06 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811519+0000", 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133428 2024-02-12T16:37:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811653+0000", 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1049999999999997e-06 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811657+0000", 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.653671526 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.465328+0000", 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154573 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.465483+0000", 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000886526 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.466369+0000", 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0046717019999999998 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.471041+0000", 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080877770000000009 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479129+0000", 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030241300000000002 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479431+0000", 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7017000000000002e-05 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479468+0000", 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107352 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479576+0000", 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018670199999999999 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.479762+0000", 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66825667099999997 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:303600 6.17 6:ed570c8d:::benchmark_data_smithi169_96224_object150764:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e734)", 2024-02-12T16:37:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:30.811523+0000", 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 222.40125528799999, 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.685226789, 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 303600 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811523+0000", 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:37:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811523+0000", 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811522+0000", 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3860000000000001e-06 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811525+0000", 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0960000000000001e-06 2024-02-12T16:37:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811528+0000", 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811530+0000", 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014012200000000001 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811670+0000", 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0080000000000004e-06 2024-02-12T16:37:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:30.811674+0000", 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.66923038599999995 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.480905+0000", 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013901000000000001 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.481044+0000", 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00080849600000000002 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2,5,10", 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.481852+0000", 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014691272 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.496543+0000", 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1573000000000003e-05 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.496585+0000", 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3881999999999998e-05 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.496629+0000", 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.635e-06 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.496635+0000", 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0697000000000001e-05 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.496666+0000", 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3339999999999998e-05 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:31.496749+0000", 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.685226789 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:13.353 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:37:13.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:13.381 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:13.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:13.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:13 smithi121 bash[18489]: cluster 2024-02-12T16:37:12.151770+0000 mgr.y (mgr.14146) 1582 : cluster [DBG] pgmap v2265: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 5421/172824 objects misplaced (3.137%); 10 MiB/s, 163 objects/s recovering 2024-02-12T16:37:13.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:13.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:13.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:13 smithi169 bash[19604]: cluster 2024-02-12T16:37:12.151770+0000 mgr.y (mgr.14146) 1582 : cluster [DBG] pgmap v2265: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 5421/172824 objects misplaced (3.137%); 10 MiB/s, 163 objects/s recovering 2024-02-12T16:37:13.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:13.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:13 smithi123 bash[19119]: cluster 2024-02-12T16:37:12.151770+0000 mgr.y (mgr.14146) 1582 : cluster [DBG] pgmap v2265: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 5421/172824 objects misplaced (3.137%); 10 MiB/s, 163 objects/s recovering 2024-02-12T16:37:13.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:13.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:13.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:13.794 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:13.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:13.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:13.938 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:13.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:14.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:14.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:14.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:14.180 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:14.212 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:14.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:14.224 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:14 smithi169 bash[19604]: audit 2024-02-12T16:37:12.583892+0000 mgr.y (mgr.14146) 1583 : audit [DBG] from='client.16821 v1:172.21.15.121:0/2491804309' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:37:14.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:14.345 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:14.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:14.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:14 smithi121 bash[18489]: audit 2024-02-12T16:37:12.583892+0000 mgr.y (mgr.14146) 1583 : audit [DBG] from='client.16821 v1:172.21.15.121:0/2491804309' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:37:14.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:14.492 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:14.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:14.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:14.606 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:14 smithi123 bash[19119]: audit 2024-02-12T16:37:12.583892+0000 mgr.y (mgr.14146) 1583 : audit [DBG] from='client.16821 v1:172.21.15.121:0/2491804309' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:37:14.634 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:14.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:14.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:14.770 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:14.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:14.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:14.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:14.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:15.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:15.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:15.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:15.181 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:15.185 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-02-12T16:37:15.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-02-12T16:37:15.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:15.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:15.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:15 smithi121 bash[18489]: cluster 2024-02-12T16:37:14.152215+0000 mgr.y (mgr.14146) 1584 : cluster [DBG] pgmap v2266: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 5049/170834 objects misplaced (2.956%); 11 MiB/s, 171 objects/s recovering 2024-02-12T16:37:15.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:15.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:15 smithi169 bash[19604]: cluster 2024-02-12T16:37:14.152215+0000 mgr.y (mgr.14146) 1584 : cluster [DBG] pgmap v2266: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 5049/170834 objects misplaced (2.956%); 11 MiB/s, 171 objects/s recovering 2024-02-12T16:37:15.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:15.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:15.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:15 smithi123 bash[19119]: cluster 2024-02-12T16:37:14.152215+0000 mgr.y (mgr.14146) 1584 : cluster [DBG] pgmap v2266: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 5049/170834 objects misplaced (2.956%); 11 MiB/s, 171 objects/s recovering 2024-02-12T16:37:15.574 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:15.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:15.812 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:15.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:15.981 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:15.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:16.084 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:16.117 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:16.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:16.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:16.255 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:16.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:16.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:37:16.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:16.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:16.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:16.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:16 smithi169 bash[19604]: cluster 2024-02-12T16:37:15.224272+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e809: 12 total, 12 up, 12 in 2024-02-12T16:37:16.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:16 smithi123 bash[19119]: cluster 2024-02-12T16:37:15.224272+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e809: 12 total, 12 up, 12 in 2024-02-12T16:37:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:16.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:16.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:16 smithi121 bash[18489]: cluster 2024-02-12T16:37:15.224272+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e809: 12 total, 12 up, 12 in 2024-02-12T16:37:16.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:16.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:16.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:16.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:16.832 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:16.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:16.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:16.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:16.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:17.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:17.119 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:17.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:17.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:17.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:17.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:17.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:17.388 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:17 smithi123 bash[19119]: cluster 2024-02-12T16:37:16.152848+0000 mgr.y (mgr.14146) 1585 : cluster [DBG] pgmap v2268: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s; 5049/167220 objects misplaced (3.019%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T16:37:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:17 smithi123 bash[19119]: cluster 2024-02-12T16:37:16.234141+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e810: 12 total, 12 up, 12 in 2024-02-12T16:37:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:17.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:17.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:17.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:17 smithi169 bash[19604]: cluster 2024-02-12T16:37:16.152848+0000 mgr.y (mgr.14146) 1585 : cluster [DBG] pgmap v2268: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s; 5049/167220 objects misplaced (3.019%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T16:37:17.498 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:17 smithi169 bash[19604]: cluster 2024-02-12T16:37:16.234141+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e810: 12 total, 12 up, 12 in 2024-02-12T16:37:17.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:17.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:17.562 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:37:17.594 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:37:17.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:17.621 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:37:17.644 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:17.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:17.656 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:37:17.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:17 smithi121 bash[18489]: cluster 2024-02-12T16:37:16.152848+0000 mgr.y (mgr.14146) 1585 : cluster [DBG] pgmap v2268: 47 pgs: 2 active+remapped+backfilling, 45 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s; 5049/167220 objects misplaced (3.019%); 5.4 MiB/s, 86 objects/s recovering 2024-02-12T16:37:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:17 smithi121 bash[18489]: cluster 2024-02-12T16:37:16.234141+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e810: 12 total, 12 up, 12 in 2024-02-12T16:37:17.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:37:17.719 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:37:17.744 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:37:17.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:17.775 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:37:17.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:17.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:17.801 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:37:17.827 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:37:17.864 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:37:17.894 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:37:17.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.900 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:37:17.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:17.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:18.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:18.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:18.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:18.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:18.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:18.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:18.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:18 smithi121 bash[18489]: audit 2024-02-12T16:37:17.514846+0000 mon.a (mon.0) 2746 : audit [DBG] from='client.? v1:172.21.15.121:0/381816157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:18.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:18.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:18.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:18.473 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:18 smithi169 bash[19604]: audit 2024-02-12T16:37:17.514846+0000 mon.a (mon.0) 2746 : audit [DBG] from='client.? v1:172.21.15.121:0/381816157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:18.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:18.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:18.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:18.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:18.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:18.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:18 smithi123 bash[19119]: audit 2024-02-12T16:37:17.514846+0000 mon.a (mon.0) 2746 : audit [DBG] from='client.? v1:172.21.15.121:0/381816157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:18.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:18.772 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:18.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:18.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:18.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:18.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:19.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:19.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:19.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:19.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:19.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:19.253 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:19 smithi121 bash[18489]: cluster 2024-02-12T16:37:18.153373+0000 mgr.y (mgr.14146) 1586 : cluster [DBG] pgmap v2270: 47 pgs: 1 peering, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 4634/162130 objects misplaced (2.858%); 8.3 MiB/s, 133 objects/s recovering 2024-02-12T16:37:19.253 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:19 smithi121 bash[18489]: cluster 2024-02-12T16:37:18.713014+0000 osd.1 (osd.1) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:19.253 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:19 smithi121 bash[18489]: cluster 2024-02-12T16:37:18.713070+0000 osd.1 (osd.1) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:19.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:19 smithi169 bash[19604]: cluster 2024-02-12T16:37:18.153373+0000 mgr.y (mgr.14146) 1586 : cluster [DBG] pgmap v2270: 47 pgs: 1 peering, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 4634/162130 objects misplaced (2.858%); 8.3 MiB/s, 133 objects/s recovering 2024-02-12T16:37:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:19 smithi169 bash[19604]: cluster 2024-02-12T16:37:18.713014+0000 osd.1 (osd.1) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:19 smithi169 bash[19604]: cluster 2024-02-12T16:37:18.713070+0000 osd.1 (osd.1) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:19.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:19.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:19.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:19.439 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:19.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:19.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:19.574 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:19.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:19 smithi123 bash[19119]: cluster 2024-02-12T16:37:18.153373+0000 mgr.y (mgr.14146) 1586 : cluster [DBG] pgmap v2270: 47 pgs: 1 peering, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 4634/162130 objects misplaced (2.858%); 8.3 MiB/s, 133 objects/s recovering 2024-02-12T16:37:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:19 smithi123 bash[19119]: cluster 2024-02-12T16:37:18.713014+0000 osd.1 (osd.1) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:19.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:19 smithi123 bash[19119]: cluster 2024-02-12T16:37:18.713070+0000 osd.1 (osd.1) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:19.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:19.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:19.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:19.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:19.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:19.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.019 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:20.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:20.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:20.284 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:37:20.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:20.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:20.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:20 smithi121 bash[18489]: cluster 2024-02-12T16:37:19.255448+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e811: 12 total, 12 up, 12 in 2024-02-12T16:37:20.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:20 smithi121 bash[18489]: audit 2024-02-12T16:37:19.496316+0000 mon.a (mon.0) 2748 : audit [INF] from='client.26729 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:20.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:20 smithi121 bash[18489]: audit 2024-02-12T16:37:19.497964+0000 mon.b (mon.2) 189 : audit [INF] from='client.26729 v1:172.21.15.121:0/4272273608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:20.430 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:20.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:20 smithi169 bash[19604]: cluster 2024-02-12T16:37:19.255448+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e811: 12 total, 12 up, 12 in 2024-02-12T16:37:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:20 smithi169 bash[19604]: audit 2024-02-12T16:37:19.496316+0000 mon.a (mon.0) 2748 : audit [INF] from='client.26729 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:20 smithi169 bash[19604]: audit 2024-02-12T16:37:19.497964+0000 mon.b (mon.2) 189 : audit [INF] from='client.26729 v1:172.21.15.121:0/4272273608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:20.568 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:20 smithi123 bash[19119]: cluster 2024-02-12T16:37:19.255448+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e811: 12 total, 12 up, 12 in 2024-02-12T16:37:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:20 smithi123 bash[19119]: audit 2024-02-12T16:37:19.496316+0000 mon.a (mon.0) 2748 : audit [INF] from='client.26729 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:20 smithi123 bash[19119]: audit 2024-02-12T16:37:19.497964+0000 mon.b (mon.2) 189 : audit [INF] from='client.26729 v1:172.21.15.121:0/4272273608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:20.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:20.814 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:20.850 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:20.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:37:20.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:20.985 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:20.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:21.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:21.124 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:21.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:21.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:21.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:21.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:21.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:21.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:21.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:21.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:21.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:21.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:21.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:21 smithi169 bash[19604]: cluster 2024-02-12T16:37:20.154005+0000 mgr.y (mgr.14146) 1587 : cluster [DBG] pgmap v2272: 47 pgs: 1 active+remapped+backfilling, 1 peering, 45 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 921 op/s; 4293/159776 objects misplaced (2.687%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:37:21.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:21 smithi169 bash[19604]: cluster 2024-02-12T16:37:20.252401+0000 mon.a (mon.0) 2749 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:21.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:21 smithi169 bash[19604]: audit 2024-02-12T16:37:20.266786+0000 mon.a (mon.0) 2750 : audit [INF] from='client.26729 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:37:21.540 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:21 smithi169 bash[19604]: cluster 2024-02-12T16:37:20.266836+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e812: 12 total, 12 up, 12 in 2024-02-12T16:37:21.568 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:21.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:21 smithi123 bash[19119]: cluster 2024-02-12T16:37:20.154005+0000 mgr.y (mgr.14146) 1587 : cluster [DBG] pgmap v2272: 47 pgs: 1 active+remapped+backfilling, 1 peering, 45 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 921 op/s; 4293/159776 objects misplaced (2.687%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:37:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:21 smithi123 bash[19119]: cluster 2024-02-12T16:37:20.252401+0000 mon.a (mon.0) 2749 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:21 smithi123 bash[19119]: audit 2024-02-12T16:37:20.266786+0000 mon.a (mon.0) 2750 : audit [INF] from='client.26729 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:37:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:21 smithi123 bash[19119]: cluster 2024-02-12T16:37:20.266836+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e812: 12 total, 12 up, 12 in 2024-02-12T16:37:21.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:21 smithi121 bash[18489]: cluster 2024-02-12T16:37:20.154005+0000 mgr.y (mgr.14146) 1587 : cluster [DBG] pgmap v2272: 47 pgs: 1 active+remapped+backfilling, 1 peering, 45 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 921 op/s; 4293/159776 objects misplaced (2.687%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:37:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:21 smithi121 bash[18489]: cluster 2024-02-12T16:37:20.252401+0000 mon.a (mon.0) 2749 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:21 smithi121 bash[18489]: audit 2024-02-12T16:37:20.266786+0000 mon.a (mon.0) 2750 : audit [INF] from='client.26729 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:37:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:21 smithi121 bash[18489]: cluster 2024-02-12T16:37:20.266836+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e812: 12 total, 12 up, 12 in 2024-02-12T16:37:21.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:21.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:21.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:21.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:21.846 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:21.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:21.929 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:21.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:21.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:21.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:22.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:22.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:22.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:22.240 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:22.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:22.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:22.375 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:22.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:22.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:22.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:22.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:22.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:22.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:22.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:22.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:22.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:22.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:22.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:22.906 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:22.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:23.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:23.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:23.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:23.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:23.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:23.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:23.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:23.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:23 smithi123 bash[19119]: cluster 2024-02-12T16:37:22.154487+0000 mgr.y (mgr.14146) 1588 : cluster [DBG] pgmap v2274: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 4293/152050 objects misplaced (2.823%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:37:23.301 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:23.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:23.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:23.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:23.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:23.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:37:23.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:23 smithi169 bash[19604]: cluster 2024-02-12T16:37:22.154487+0000 mgr.y (mgr.14146) 1588 : cluster [DBG] pgmap v2274: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 4293/152050 objects misplaced (2.823%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:37:23.573 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:23.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:23.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:23 smithi121 bash[18489]: cluster 2024-02-12T16:37:22.154487+0000 mgr.y (mgr.14146) 1588 : cluster [DBG] pgmap v2274: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 4293/152050 objects misplaced (2.823%); 4.8 MiB/s, 77 objects/s recovering 2024-02-12T16:37:23.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:23.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:23.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:23.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:23.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:23.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:23.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:24.005 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:24.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:24.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:24.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:24.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:24.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:24.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:24.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:24.554 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:24.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:24.683 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:24.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:24.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:24.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:24.942 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:24.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:25.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:25.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:25.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:25.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:25.209 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:25.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:25.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:25.357 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:25.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:25.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:25 smithi121 bash[18489]: cluster 2024-02-12T16:37:24.154929+0000 mgr.y (mgr.14146) 1589 : cluster [DBG] pgmap v2275: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 3939/149780 objects misplaced (2.630%); 937 KiB/s, 14 objects/s recovering 2024-02-12T16:37:25.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:25.492 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:25.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:25 smithi169 bash[19604]: cluster 2024-02-12T16:37:24.154929+0000 mgr.y (mgr.14146) 1589 : cluster [DBG] pgmap v2275: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 3939/149780 objects misplaced (2.630%); 937 KiB/s, 14 objects/s recovering 2024-02-12T16:37:25.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:25.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:25.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:25 smithi123 bash[19119]: cluster 2024-02-12T16:37:24.154929+0000 mgr.y (mgr.14146) 1589 : cluster [DBG] pgmap v2275: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 3939/149780 objects misplaced (2.630%); 937 KiB/s, 14 objects/s recovering 2024-02-12T16:37:25.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:25.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:25.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:25.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:25.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:25.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:25.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:26.021 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:26.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:26.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:26.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:26.157 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:26.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:26.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:26.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:26.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:26.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:26 smithi169 bash[19604]: cluster 2024-02-12T16:37:24.591715+0000 osd.11 (osd.11) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:26 smithi169 bash[19604]: cluster 2024-02-12T16:37:24.591776+0000 osd.11 (osd.11) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:26 smithi169 bash[19604]: cluster 2024-02-12T16:37:25.272746+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e813: 12 total, 12 up, 12 in 2024-02-12T16:37:26.556 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:26.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:26 smithi123 bash[19119]: cluster 2024-02-12T16:37:24.591715+0000 osd.11 (osd.11) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:26.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:26 smithi123 bash[19119]: cluster 2024-02-12T16:37:24.591776+0000 osd.11 (osd.11) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:26.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:26 smithi123 bash[19119]: cluster 2024-02-12T16:37:25.272746+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e813: 12 total, 12 up, 12 in 2024-02-12T16:37:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:26.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:26 smithi121 bash[18489]: cluster 2024-02-12T16:37:24.591715+0000 osd.11 (osd.11) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:26 smithi121 bash[18489]: cluster 2024-02-12T16:37:24.591776+0000 osd.11 (osd.11) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:26 smithi121 bash[18489]: cluster 2024-02-12T16:37:25.272746+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e813: 12 total, 12 up, 12 in 2024-02-12T16:37:26.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:26.693 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:26.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:26.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:26.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:26.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:26.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:26.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:26.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:27.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:27.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:27.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:27.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:27.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:27.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:27.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:27.349 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:27 smithi123 bash[19119]: cluster 2024-02-12T16:37:26.155369+0000 mgr.y (mgr.14146) 1590 : cluster [DBG] pgmap v2277: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 3939/146082 objects misplaced (2.696%); 538 KiB/s, 8 objects/s recovering 2024-02-12T16:37:27.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:27.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:27.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:27.488 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:27 smithi121 bash[18489]: cluster 2024-02-12T16:37:26.155369+0000 mgr.y (mgr.14146) 1590 : cluster [DBG] pgmap v2277: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 3939/146082 objects misplaced (2.696%); 538 KiB/s, 8 objects/s recovering 2024-02-12T16:37:27.516 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:27.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:27 smithi169 bash[19604]: cluster 2024-02-12T16:37:26.155369+0000 mgr.y (mgr.14146) 1590 : cluster [DBG] pgmap v2277: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 3939/146082 objects misplaced (2.696%); 538 KiB/s, 8 objects/s recovering 2024-02-12T16:37:27.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:27.647 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:27.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:27.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:27.776 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:27.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:27.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:27.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:27.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:27.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:27.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:28.001 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:28.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:28.052 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:28.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:28.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:28.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:28.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:28.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:28.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:28.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:28.586 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:28.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:28.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:28.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:28.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:28.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:28.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:28.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:29.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:29.128 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:29.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:29.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:29.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:29.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:29.294 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:29 smithi169 bash[19604]: cluster 2024-02-12T16:37:28.155816+0000 mgr.y (mgr.14146) 1591 : cluster [DBG] pgmap v2278: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 3939/139130 objects misplaced (2.831%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:29.373 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:29.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:29.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:29.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:29.526 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:29 smithi121 bash[18489]: cluster 2024-02-12T16:37:28.155816+0000 mgr.y (mgr.14146) 1591 : cluster [DBG] pgmap v2278: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 3939/139130 objects misplaced (2.831%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:29.557 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:29.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:29 smithi123 bash[19119]: cluster 2024-02-12T16:37:28.155816+0000 mgr.y (mgr.14146) 1591 : cluster [DBG] pgmap v2278: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 3939/139130 objects misplaced (2.831%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:29.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:29.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:29.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:29.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:29.826 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:29.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:37:29.912 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:37:29.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:29.966 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:29.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:30.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:30.099 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:30.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:30.170 INFO:teuthology.orchestra.run.smithi121.stdout:moved 0 / 94 (0%) 2024-02-12T16:37:30.170 INFO:teuthology.orchestra.run.smithi121.stdout:avg 7.83333 2024-02-12T16:37:30.170 INFO:teuthology.orchestra.run.smithi121.stdout:stddev 1.28019 -> 1.28019 (expected baseline 2.67966) 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:min osd.0 with 6 -> 6 pgs (0.765957 -> 0.765957 * mean) 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:max osd.6 with 10 -> 10 pgs (1.2766 -> 1.2766 * mean) 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:oload 1000 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:max_change 1 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:max_change_osds 4 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:average_utilization 0.0151 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:overload_utilization 0.1507 2024-02-12T16:37:30.171 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 weight 0.8111 -> 0.8134 2024-02-12T16:37:30.175 INFO:teuthology.orchestra.run.smithi121.stderr:no change 2024-02-12T16:37:30.206 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:30.232 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 16:37:30 smithi121 bash[18805]: debug 2024-02-12T16:37:30.161+0000 7fb564e0b700 -1 mgr.server reply reply (1) Operation not permitted no change 2024-02-12T16:37:30.235 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:30.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:30.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:30.373 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:30.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:30 smithi123 bash[19119]: cluster 2024-02-12T16:37:28.599903+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:30 smithi123 bash[19119]: cluster 2024-02-12T16:37:28.599955+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:30 smithi123 bash[19119]: cluster 2024-02-12T16:37:29.281693+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e814: 12 total, 12 up, 12 in 2024-02-12T16:37:30.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:30.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:30.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:30.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:30.616 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:30 smithi121 bash[18489]: cluster 2024-02-12T16:37:28.599903+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:30.616 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:30 smithi121 bash[18489]: cluster 2024-02-12T16:37:28.599955+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:30.616 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:30 smithi121 bash[18489]: cluster 2024-02-12T16:37:29.281693+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e814: 12 total, 12 up, 12 in 2024-02-12T16:37:30.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:30.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:30.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:30.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:30.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:30 smithi169 bash[19604]: cluster 2024-02-12T16:37:28.599903+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:30 smithi169 bash[19604]: cluster 2024-02-12T16:37:28.599955+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:30 smithi169 bash[19604]: cluster 2024-02-12T16:37:29.281693+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e814: 12 total, 12 up, 12 in 2024-02-12T16:37:30.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:30.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:30.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:31.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:31.040 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:31.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:31.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:31.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:31.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:31.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:31.303 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:31.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:31 smithi121 bash[18489]: cluster 2024-02-12T16:37:30.156259+0000 mgr.y (mgr.14146) 1592 : cluster [DBG] pgmap v2280: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 987 op/s; 3543/136246 objects misplaced (2.600%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:31 smithi121 bash[18489]: audit 2024-02-12T16:37:30.165054+0000 mgr.y (mgr.14146) 1593 : audit [DBG] from='client.16836 v1:172.21.15.121:0/282624590' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:37:31.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:31.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:31.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:31.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:31.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:31 smithi123 bash[19119]: cluster 2024-02-12T16:37:30.156259+0000 mgr.y (mgr.14146) 1592 : cluster [DBG] pgmap v2280: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 987 op/s; 3543/136246 objects misplaced (2.600%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:31.554 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:31 smithi123 bash[19119]: audit 2024-02-12T16:37:30.165054+0000 mgr.y (mgr.14146) 1593 : audit [DBG] from='client.16836 v1:172.21.15.121:0/282624590' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:37:31.580 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:31.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:31.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:31.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:31.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:31 smithi169 bash[19604]: cluster 2024-02-12T16:37:30.156259+0000 mgr.y (mgr.14146) 1592 : cluster [DBG] pgmap v2280: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 987 op/s; 3543/136246 objects misplaced (2.600%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:31 smithi169 bash[19604]: audit 2024-02-12T16:37:30.165054+0000 mgr.y (mgr.14146) 1593 : audit [DBG] from='client.16836 v1:172.21.15.121:0/282624590' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-12T16:37:31.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:31.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:31.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:31.963 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:32.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:32.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:32.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:32.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:32.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:32.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:32.280 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:32.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:32.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:32.423 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:32.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:32.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:32.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:32.560 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:32.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:32.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:32.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:32.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:32.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:32.836 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:32.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:32.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:33.016 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:33.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:33.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:37:33.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:33.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:33.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:33.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:33.287 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:33.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:33.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:33.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:33.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:33.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:33 smithi169 bash[19604]: cluster 2024-02-12T16:37:31.743101+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:33 smithi169 bash[19604]: cluster 2024-02-12T16:37:31.743152+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:33 smithi169 bash[19604]: cluster 2024-02-12T16:37:32.156757+0000 mgr.y (mgr.14146) 1594 : cluster [DBG] pgmap v2281: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s; 3543/127914 objects misplaced (2.770%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:33 smithi169 bash[19604]: cluster 2024-02-12T16:37:32.306598+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e815: 12 total, 12 up, 12 in 2024-02-12T16:37:33.591 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:33.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:33 smithi121 bash[18489]: cluster 2024-02-12T16:37:31.743101+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:33.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:33 smithi121 bash[18489]: cluster 2024-02-12T16:37:31.743152+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:33.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:33 smithi121 bash[18489]: cluster 2024-02-12T16:37:32.156757+0000 mgr.y (mgr.14146) 1594 : cluster [DBG] pgmap v2281: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s; 3543/127914 objects misplaced (2.770%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:33.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:33 smithi121 bash[18489]: cluster 2024-02-12T16:37:32.306598+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e815: 12 total, 12 up, 12 in 2024-02-12T16:37:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:33.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:33 smithi123 bash[19119]: cluster 2024-02-12T16:37:31.743101+0000 osd.4 (osd.4) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:33.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:33 smithi123 bash[19119]: cluster 2024-02-12T16:37:31.743152+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:33.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:33 smithi123 bash[19119]: cluster 2024-02-12T16:37:32.156757+0000 mgr.y (mgr.14146) 1594 : cluster [DBG] pgmap v2281: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s; 3543/127914 objects misplaced (2.770%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:33.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:33 smithi123 bash[19119]: cluster 2024-02-12T16:37:32.306598+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e815: 12 total, 12 up, 12 in 2024-02-12T16:37:33.698 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:37:33.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:37:33.730 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:33.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:33.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:33.762 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:37:33.792 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:37:33.823 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:37:33.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:33.857 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:37:33.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:33.887 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:37:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:33.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:33.915 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:37:33.942 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:37:33.973 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:37:33.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:34.009 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:37:34.022 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:34.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:34.037 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:37:34.044 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.044 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.044 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.044 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:37:34.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:34.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:34.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:34.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:34.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:34.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:34.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:34.420 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:34 smithi121 bash[18489]: audit 2024-02-12T16:37:33.655256+0000 mon.a (mon.0) 2755 : audit [DBG] from='client.? v1:172.21.15.121:0/2191282051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:34.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:34 smithi121 bash[18489]: cluster 2024-02-12T16:37:33.786164+0000 osd.8 (osd.8) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:34.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:34 smithi121 bash[18489]: cluster 2024-02-12T16:37:33.786213+0000 osd.8 (osd.8) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:34.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:34.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:34 smithi169 bash[19604]: audit 2024-02-12T16:37:33.655256+0000 mon.a (mon.0) 2755 : audit [DBG] from='client.? v1:172.21.15.121:0/2191282051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:34 smithi169 bash[19604]: cluster 2024-02-12T16:37:33.786164+0000 osd.8 (osd.8) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:34 smithi169 bash[19604]: cluster 2024-02-12T16:37:33.786213+0000 osd.8 (osd.8) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:34.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:34.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:34.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:34 smithi123 bash[19119]: audit 2024-02-12T16:37:33.655256+0000 mon.a (mon.0) 2755 : audit [DBG] from='client.? v1:172.21.15.121:0/2191282051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:34 smithi123 bash[19119]: cluster 2024-02-12T16:37:33.786164+0000 osd.8 (osd.8) 101 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:34 smithi123 bash[19119]: cluster 2024-02-12T16:37:33.786213+0000 osd.8 (osd.8) 102 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:34.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:34.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:34.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:34.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:34.845 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:34.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:34.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:34.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:35.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.151 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:35.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:35.279 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:35.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:35 smithi121 bash[18489]: cluster 2024-02-12T16:37:34.157178+0000 mgr.y (mgr.14146) 1595 : cluster [DBG] pgmap v2283: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 3168/125758 objects misplaced (2.519%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:37:35.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:35 smithi121 bash[18489]: cluster 2024-02-12T16:37:34.316324+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e816: 12 total, 12 up, 12 in 2024-02-12T16:37:35.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:35.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:35.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:35 smithi169 bash[19604]: cluster 2024-02-12T16:37:34.157178+0000 mgr.y (mgr.14146) 1595 : cluster [DBG] pgmap v2283: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 3168/125758 objects misplaced (2.519%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:37:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:35 smithi169 bash[19604]: cluster 2024-02-12T16:37:34.316324+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e816: 12 total, 12 up, 12 in 2024-02-12T16:37:35.556 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:35 smithi123 bash[19119]: cluster 2024-02-12T16:37:34.157178+0000 mgr.y (mgr.14146) 1595 : cluster [DBG] pgmap v2283: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 3168/125758 objects misplaced (2.519%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:37:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:35 smithi123 bash[19119]: cluster 2024-02-12T16:37:34.316324+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e816: 12 total, 12 up, 12 in 2024-02-12T16:37:35.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:35.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:35.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:35.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:35.988 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:35.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:36.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:36.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:36.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:36.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:36.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:36.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:36.400 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:36.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:36.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:36.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:36.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:36.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:36.673 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:36.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:36.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:36.803 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:36.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:36.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:36.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:36.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:37.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:36 smithi169 bash[19604]: cluster 2024-02-12T16:37:36.157523+0000 mgr.y (mgr.14146) 1596 : cluster [DBG] pgmap v2285: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3168/122068 objects misplaced (2.595%); 549 KiB/s, 8 objects/s recovering 2024-02-12T16:37:37.079 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:37.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:37.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:36 smithi123 bash[19119]: cluster 2024-02-12T16:37:36.157523+0000 mgr.y (mgr.14146) 1596 : cluster [DBG] pgmap v2285: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3168/122068 objects misplaced (2.595%); 549 KiB/s, 8 objects/s recovering 2024-02-12T16:37:37.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:36 smithi121 bash[18489]: cluster 2024-02-12T16:37:36.157523+0000 mgr.y (mgr.14146) 1596 : cluster [DBG] pgmap v2285: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3168/122068 objects misplaced (2.595%); 549 KiB/s, 8 objects/s recovering 2024-02-12T16:37:37.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:37.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:37.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:37.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:37.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:37.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:37.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:37.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:37.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:37:37.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:37.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:37.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:37.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:37.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:37.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:37.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:37.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:37.880 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:37:37.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:37.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:37.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:38.007 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-02-12T16:37:38.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:37:38.008 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T16:37:38.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:37:38.016 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:37 smithi121 bash[18489]: audit 2024-02-12T16:37:36.924772+0000 mon.a (mon.0) 2757 : audit [INF] from='client.26747 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:37:38.016 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:37 smithi121 bash[18489]: audit 2024-02-12T16:37:36.926330+0000 mon.b (mon.2) 190 : audit [INF] from='client.? v1:172.21.15.121:0/2425408455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:37:38.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:38.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:38.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:37 smithi169 bash[19604]: audit 2024-02-12T16:37:36.924772+0000 mon.a (mon.0) 2757 : audit [INF] from='client.26747 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:37:38.132 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:37 smithi169 bash[19604]: audit 2024-02-12T16:37:36.926330+0000 mon.b (mon.2) 190 : audit [INF] from='client.? v1:172.21.15.121:0/2425408455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:37:38.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:37 smithi123 bash[19119]: audit 2024-02-12T16:37:36.924772+0000 mon.a (mon.0) 2757 : audit [INF] from='client.26747 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:37:38.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:37 smithi123 bash[19119]: audit 2024-02-12T16:37:36.926330+0000 mon.b (mon.2) 190 : audit [INF] from='client.? v1:172.21.15.121:0/2425408455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:37:38.153 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:38.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:38.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:38.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:38.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:38.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:38.572 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:38.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:38.706 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:38.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:38.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:38.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:38.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:38.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:38 smithi169 bash[19604]: cluster 2024-02-12T16:37:37.856423+0000 mon.a (mon.0) 2758 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:38 smithi169 bash[19604]: audit 2024-02-12T16:37:37.859761+0000 mon.a (mon.0) 2759 : audit [INF] from='client.26747 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:37:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:38 smithi169 bash[19604]: cluster 2024-02-12T16:37:37.859825+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e817: 12 total, 12 up, 12 in 2024-02-12T16:37:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:38 smithi169 bash[19604]: cluster 2024-02-12T16:37:38.157964+0000 mgr.y (mgr.14146) 1597 : cluster [DBG] pgmap v2287: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 3168/115352 objects misplaced (2.746%); 629 KiB/s, 9 objects/s recovering 2024-02-12T16:37:39.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:39.114 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:39.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:38 smithi123 bash[19119]: cluster 2024-02-12T16:37:37.856423+0000 mon.a (mon.0) 2758 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:39.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:38 smithi123 bash[19119]: audit 2024-02-12T16:37:37.859761+0000 mon.a (mon.0) 2759 : audit [INF] from='client.26747 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:37:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:38 smithi123 bash[19119]: cluster 2024-02-12T16:37:37.859825+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e817: 12 total, 12 up, 12 in 2024-02-12T16:37:39.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:38 smithi123 bash[19119]: cluster 2024-02-12T16:37:38.157964+0000 mgr.y (mgr.14146) 1597 : cluster [DBG] pgmap v2287: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 3168/115352 objects misplaced (2.746%); 629 KiB/s, 9 objects/s recovering 2024-02-12T16:37:39.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:38 smithi121 bash[18489]: cluster 2024-02-12T16:37:37.856423+0000 mon.a (mon.0) 2758 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:37:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:38 smithi121 bash[18489]: audit 2024-02-12T16:37:37.859761+0000 mon.a (mon.0) 2759 : audit [INF] from='client.26747 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:37:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:38 smithi121 bash[18489]: cluster 2024-02-12T16:37:37.859825+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e817: 12 total, 12 up, 12 in 2024-02-12T16:37:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:38 smithi121 bash[18489]: cluster 2024-02-12T16:37:38.157964+0000 mgr.y (mgr.14146) 1597 : cluster [DBG] pgmap v2287: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 3168/115352 objects misplaced (2.746%); 629 KiB/s, 9 objects/s recovering 2024-02-12T16:37:39.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:39.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:39.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:39.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:39.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:39.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:39.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:39.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:39.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:39.564 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:39.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:39.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:39.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:39.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:39.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:39.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:39.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:39.928 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:39.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:40.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:40.042 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:39 smithi121 bash[18489]: cluster 2024-02-12T16:37:38.203670+0000 osd.3 (osd.3) 78 : cluster [DBG] 7.f scrub starts 2024-02-12T16:37:40.042 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:39 smithi121 bash[18489]: cluster 2024-02-12T16:37:39.110529+0000 osd.0 (osd.0) 141 : cluster [DBG] 7.6 scrub starts 2024-02-12T16:37:40.075 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:40.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:39 smithi123 bash[19119]: cluster 2024-02-12T16:37:38.203670+0000 osd.3 (osd.3) 78 : cluster [DBG] 7.f scrub starts 2024-02-12T16:37:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:39 smithi123 bash[19119]: cluster 2024-02-12T16:37:39.110529+0000 osd.0 (osd.0) 141 : cluster [DBG] 7.6 scrub starts 2024-02-12T16:37:40.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:40.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:40.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:39 smithi169 bash[19604]: cluster 2024-02-12T16:37:38.203670+0000 osd.3 (osd.3) 78 : cluster [DBG] 7.f scrub starts 2024-02-12T16:37:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:39 smithi169 bash[19604]: cluster 2024-02-12T16:37:39.110529+0000 osd.0 (osd.0) 141 : cluster [DBG] 7.6 scrub starts 2024-02-12T16:37:40.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:40.364 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:40.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:40.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:40.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:40.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:40.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:40.639 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:40.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:40.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:40.774 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:40.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:40.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:40.914 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:40.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:41.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:41.046 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:41.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:41.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:37:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:40 smithi123 bash[19119]: cluster 2024-02-12T16:37:39.388503+0000 osd.3 (osd.3) 79 : cluster [DBG] 7.f scrub ok 2024-02-12T16:37:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:40 smithi123 bash[19119]: cluster 2024-02-12T16:37:39.723281+0000 osd.10 (osd.10) 174 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:37:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:40 smithi123 bash[19119]: cluster 2024-02-12T16:37:39.765741+0000 osd.10 (osd.10) 175 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:37:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:40 smithi123 bash[19119]: cluster 2024-02-12T16:37:40.120847+0000 osd.0 (osd.0) 142 : cluster [DBG] 7.6 scrub ok 2024-02-12T16:37:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:40 smithi123 bash[19119]: cluster 2024-02-12T16:37:40.158462+0000 mgr.y (mgr.14146) 1598 : cluster [DBG] pgmap v2288: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 987 op/s; 2802/112408 objects misplaced (2.493%); 954 KiB/s, 14 objects/s recovering 2024-02-12T16:37:41.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:41.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:40 smithi121 bash[18489]: cluster 2024-02-12T16:37:39.388503+0000 osd.3 (osd.3) 79 : cluster [DBG] 7.f scrub ok 2024-02-12T16:37:41.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:40 smithi121 bash[18489]: cluster 2024-02-12T16:37:39.723281+0000 osd.10 (osd.10) 174 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:37:41.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:40 smithi121 bash[18489]: cluster 2024-02-12T16:37:39.765741+0000 osd.10 (osd.10) 175 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:37:41.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:40 smithi121 bash[18489]: cluster 2024-02-12T16:37:40.120847+0000 osd.0 (osd.0) 142 : cluster [DBG] 7.6 scrub ok 2024-02-12T16:37:41.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:40 smithi121 bash[18489]: cluster 2024-02-12T16:37:40.158462+0000 mgr.y (mgr.14146) 1598 : cluster [DBG] pgmap v2288: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 987 op/s; 2802/112408 objects misplaced (2.493%); 954 KiB/s, 14 objects/s recovering 2024-02-12T16:37:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:41.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:41.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:40 smithi169 bash[19604]: cluster 2024-02-12T16:37:39.388503+0000 osd.3 (osd.3) 79 : cluster [DBG] 7.f scrub ok 2024-02-12T16:37:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:40 smithi169 bash[19604]: cluster 2024-02-12T16:37:39.723281+0000 osd.10 (osd.10) 174 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:37:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:40 smithi169 bash[19604]: cluster 2024-02-12T16:37:39.765741+0000 osd.10 (osd.10) 175 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:37:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:40 smithi169 bash[19604]: cluster 2024-02-12T16:37:40.120847+0000 osd.0 (osd.0) 142 : cluster [DBG] 7.6 scrub ok 2024-02-12T16:37:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:40 smithi169 bash[19604]: cluster 2024-02-12T16:37:40.158462+0000 mgr.y (mgr.14146) 1598 : cluster [DBG] pgmap v2288: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 987 op/s; 2802/112408 objects misplaced (2.493%); 954 KiB/s, 14 objects/s recovering 2024-02-12T16:37:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:41.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:41.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:41.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:41.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:41.582 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:41.614 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:41.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:41.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:41.747 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:41.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:41.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:41.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:41.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:41.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.022 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:42.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:42.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:42.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:41 smithi121 bash[18489]: cluster 2024-02-12T16:37:41.121698+0000 osd.3 (osd.3) 80 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:37:42.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:41 smithi121 bash[18489]: cluster 2024-02-12T16:37:41.178388+0000 osd.11 (osd.11) 185 : cluster [DBG] 1.b scrub starts 2024-02-12T16:37:42.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:41 smithi121 bash[18489]: cluster 2024-02-12T16:37:41.320660+0000 osd.11 (osd.11) 186 : cluster [DBG] 1.b scrub ok 2024-02-12T16:37:42.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:41 smithi123 bash[19119]: cluster 2024-02-12T16:37:41.121698+0000 osd.3 (osd.3) 80 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:37:42.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:41 smithi123 bash[19119]: cluster 2024-02-12T16:37:41.178388+0000 osd.11 (osd.11) 185 : cluster [DBG] 1.b scrub starts 2024-02-12T16:37:42.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:41 smithi123 bash[19119]: cluster 2024-02-12T16:37:41.320660+0000 osd.11 (osd.11) 186 : cluster [DBG] 1.b scrub ok 2024-02-12T16:37:42.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:42.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:42.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:41 smithi169 bash[19604]: cluster 2024-02-12T16:37:41.121698+0000 osd.3 (osd.3) 80 : cluster [DBG] 7.3 scrub starts 2024-02-12T16:37:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:41 smithi169 bash[19604]: cluster 2024-02-12T16:37:41.178388+0000 osd.11 (osd.11) 185 : cluster [DBG] 1.b scrub starts 2024-02-12T16:37:42.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:41 smithi169 bash[19604]: cluster 2024-02-12T16:37:41.320660+0000 osd.11 (osd.11) 186 : cluster [DBG] 1.b scrub ok 2024-02-12T16:37:42.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:42.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:42.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:42.650 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:42.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:42.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:42.816 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:42.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:42.949 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:42.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:43.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:43.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:43.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:43.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:43.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:43.193 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:42 smithi169 bash[19604]: cluster 2024-02-12T16:37:41.778207+0000 osd.9 (osd.9) 139 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:37:43.193 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:42 smithi169 bash[19604]: cluster 2024-02-12T16:37:41.801854+0000 osd.9 (osd.9) 140 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:37:43.193 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:42 smithi169 bash[19604]: cluster 2024-02-12T16:37:42.158928+0000 mgr.y (mgr.14146) 1599 : cluster [DBG] pgmap v2289: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 45 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 2802/104360 objects misplaced (2.685%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:43.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:43.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:43.265 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:43.265 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:43.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:43.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:43.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:42 smithi123 bash[19119]: cluster 2024-02-12T16:37:41.778207+0000 osd.9 (osd.9) 139 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:37:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:42 smithi123 bash[19119]: cluster 2024-02-12T16:37:41.801854+0000 osd.9 (osd.9) 140 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:37:43.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:42 smithi123 bash[19119]: cluster 2024-02-12T16:37:42.158928+0000 mgr.y (mgr.14146) 1599 : cluster [DBG] pgmap v2289: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 45 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 2802/104360 objects misplaced (2.685%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:42 smithi121 bash[18489]: cluster 2024-02-12T16:37:41.778207+0000 osd.9 (osd.9) 139 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:37:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:42 smithi121 bash[18489]: cluster 2024-02-12T16:37:41.801854+0000 osd.9 (osd.9) 140 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:37:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:42 smithi121 bash[18489]: cluster 2024-02-12T16:37:42.158928+0000 mgr.y (mgr.14146) 1599 : cluster [DBG] pgmap v2289: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 45 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 2802/104360 objects misplaced (2.685%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:37:43.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:43.510 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:43.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:43.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:43.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:43.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:43.758 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:43.786 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:43.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:43.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:43.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:43.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:44.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:43 smithi169 bash[19604]: cluster 2024-02-12T16:37:42.280679+0000 osd.3 (osd.3) 81 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:37:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:43 smithi169 bash[19604]: cluster 2024-02-12T16:37:43.023604+0000 osd.6 (osd.6) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:43 smithi169 bash[19604]: cluster 2024-02-12T16:37:43.023659+0000 osd.6 (osd.6) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:44.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:44.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:44.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:43 smithi121 bash[18489]: cluster 2024-02-12T16:37:42.280679+0000 osd.3 (osd.3) 81 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:37:44.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:43 smithi121 bash[18489]: cluster 2024-02-12T16:37:43.023604+0000 osd.6 (osd.6) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:44.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:43 smithi121 bash[18489]: cluster 2024-02-12T16:37:43.023659+0000 osd.6 (osd.6) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:44.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:44.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:44.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:44.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:44.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:44.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:43 smithi123 bash[19119]: cluster 2024-02-12T16:37:42.280679+0000 osd.3 (osd.3) 81 : cluster [DBG] 7.3 scrub ok 2024-02-12T16:37:44.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:43 smithi123 bash[19119]: cluster 2024-02-12T16:37:43.023604+0000 osd.6 (osd.6) 162 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:44.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:43 smithi123 bash[19119]: cluster 2024-02-12T16:37:43.023659+0000 osd.6 (osd.6) 163 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:44.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:44.488 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:44.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:44.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:44.623 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:44.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:44.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:44.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:44.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:44.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:44.896 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:44.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:44.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:45.033 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:45.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:45.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:45.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:44 smithi121 bash[18489]: cluster 2024-02-12T16:37:43.180171+0000 osd.3 (osd.3) 82 : cluster [DBG] 1.c scrub starts 2024-02-12T16:37:45.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:44 smithi121 bash[18489]: cluster 2024-02-12T16:37:43.255286+0000 osd.3 (osd.3) 83 : cluster [DBG] 1.c scrub ok 2024-02-12T16:37:45.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:44 smithi121 bash[18489]: cluster 2024-02-12T16:37:43.919958+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e818: 12 total, 12 up, 12 in 2024-02-12T16:37:45.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:44 smithi121 bash[18489]: cluster 2024-02-12T16:37:44.150208+0000 osd.3 (osd.3) 84 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:37:45.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:44 smithi121 bash[18489]: cluster 2024-02-12T16:37:44.159380+0000 mgr.y (mgr.14146) 1600 : cluster [DBG] pgmap v2291: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 45 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 2444/102110 objects misplaced (2.393%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:45.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:45.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:45.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:45.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:44 smithi123 bash[19119]: cluster 2024-02-12T16:37:43.180171+0000 osd.3 (osd.3) 82 : cluster [DBG] 1.c scrub starts 2024-02-12T16:37:45.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:44 smithi123 bash[19119]: cluster 2024-02-12T16:37:43.255286+0000 osd.3 (osd.3) 83 : cluster [DBG] 1.c scrub ok 2024-02-12T16:37:45.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:44 smithi123 bash[19119]: cluster 2024-02-12T16:37:43.919958+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e818: 12 total, 12 up, 12 in 2024-02-12T16:37:45.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:44 smithi123 bash[19119]: cluster 2024-02-12T16:37:44.150208+0000 osd.3 (osd.3) 84 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:37:45.294 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:44 smithi123 bash[19119]: cluster 2024-02-12T16:37:44.159380+0000 mgr.y (mgr.14146) 1600 : cluster [DBG] pgmap v2291: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 45 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 2444/102110 objects misplaced (2.393%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:45.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:44 smithi169 bash[19604]: cluster 2024-02-12T16:37:43.180171+0000 osd.3 (osd.3) 82 : cluster [DBG] 1.c scrub starts 2024-02-12T16:37:45.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:44 smithi169 bash[19604]: cluster 2024-02-12T16:37:43.255286+0000 osd.3 (osd.3) 83 : cluster [DBG] 1.c scrub ok 2024-02-12T16:37:45.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:44 smithi169 bash[19604]: cluster 2024-02-12T16:37:43.919958+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e818: 12 total, 12 up, 12 in 2024-02-12T16:37:45.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:44 smithi169 bash[19604]: cluster 2024-02-12T16:37:44.150208+0000 osd.3 (osd.3) 84 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:37:45.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:44 smithi169 bash[19604]: cluster 2024-02-12T16:37:44.159380+0000 mgr.y (mgr.14146) 1600 : cluster [DBG] pgmap v2291: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 45 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 2444/102110 objects misplaced (2.393%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:45.322 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:45.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:45.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:45.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:45.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:45.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:45.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:45.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:45.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:45.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:45.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:45.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:45.849 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:37:45.849 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:37:45.849 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:37:45.849 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:37:45.849 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:37:45.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:45.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:45.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:46.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:45 smithi169 bash[19604]: cluster 2024-02-12T16:37:44.252844+0000 osd.3 (osd.3) 85 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:37:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:45 smithi169 bash[19604]: cluster 2024-02-12T16:37:44.743354+0000 osd.9 (osd.9) 141 : cluster [DBG] 1.a scrub starts 2024-02-12T16:37:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:45 smithi169 bash[19604]: cluster 2024-02-12T16:37:44.746345+0000 osd.9 (osd.9) 142 : cluster [DBG] 1.a scrub ok 2024-02-12T16:37:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:45 smithi169 bash[19604]: cluster 2024-02-12T16:37:45.117735+0000 osd.11 (osd.11) 187 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:37:46.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:45 smithi169 bash[19604]: cluster 2024-02-12T16:37:45.169458+0000 osd.11 (osd.11) 188 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:37:46.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:46.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:45 smithi121 bash[18489]: cluster 2024-02-12T16:37:44.252844+0000 osd.3 (osd.3) 85 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:37:46.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:45 smithi121 bash[18489]: cluster 2024-02-12T16:37:44.743354+0000 osd.9 (osd.9) 141 : cluster [DBG] 1.a scrub starts 2024-02-12T16:37:46.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:45 smithi121 bash[18489]: cluster 2024-02-12T16:37:44.746345+0000 osd.9 (osd.9) 142 : cluster [DBG] 1.a scrub ok 2024-02-12T16:37:46.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:45 smithi121 bash[18489]: cluster 2024-02-12T16:37:45.117735+0000 osd.11 (osd.11) 187 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:37:46.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:45 smithi121 bash[18489]: cluster 2024-02-12T16:37:45.169458+0000 osd.11 (osd.11) 188 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:37:46.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:46.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:46.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:46.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:45 smithi123 bash[19119]: cluster 2024-02-12T16:37:44.252844+0000 osd.3 (osd.3) 85 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:37:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:45 smithi123 bash[19119]: cluster 2024-02-12T16:37:44.743354+0000 osd.9 (osd.9) 141 : cluster [DBG] 1.a scrub starts 2024-02-12T16:37:46.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:45 smithi123 bash[19119]: cluster 2024-02-12T16:37:44.746345+0000 osd.9 (osd.9) 142 : cluster [DBG] 1.a scrub ok 2024-02-12T16:37:46.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:45 smithi123 bash[19119]: cluster 2024-02-12T16:37:45.117735+0000 osd.11 (osd.11) 187 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:37:46.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:45 smithi123 bash[19119]: cluster 2024-02-12T16:37:45.169458+0000 osd.11 (osd.11) 188 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:37:46.428 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:46.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:46.563 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:46.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:46.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:46.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:46.829 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:46.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:46.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:46.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:47.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:47.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:47.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:47.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:47.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:46 smithi169 bash[19604]: cluster 2024-02-12T16:37:45.187060+0000 osd.3 (osd.3) 86 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:37:47.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:46 smithi169 bash[19604]: cluster 2024-02-12T16:37:46.032919+0000 osd.3 (osd.3) 87 : cluster [DBG] 7.2 scrub ok 2024-02-12T16:37:47.221 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:46 smithi169 bash[19604]: cluster 2024-02-12T16:37:46.159776+0000 mgr.y (mgr.14146) 1601 : cluster [DBG] pgmap v2292: 47 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 44 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2444/98454 objects misplaced (2.482%); 895 KiB/s, 13 objects/s recovering 2024-02-12T16:37:47.251 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:47.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:47.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:46 smithi121 bash[18489]: cluster 2024-02-12T16:37:45.187060+0000 osd.3 (osd.3) 86 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:37:47.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:46 smithi121 bash[18489]: cluster 2024-02-12T16:37:46.032919+0000 osd.3 (osd.3) 87 : cluster [DBG] 7.2 scrub ok 2024-02-12T16:37:47.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:46 smithi121 bash[18489]: cluster 2024-02-12T16:37:46.159776+0000 mgr.y (mgr.14146) 1601 : cluster [DBG] pgmap v2292: 47 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 44 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2444/98454 objects misplaced (2.482%); 895 KiB/s, 13 objects/s recovering 2024-02-12T16:37:47.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:47.362 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:46 smithi123 bash[19119]: cluster 2024-02-12T16:37:45.187060+0000 osd.3 (osd.3) 86 : cluster [DBG] 7.2 scrub starts 2024-02-12T16:37:47.362 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:46 smithi123 bash[19119]: cluster 2024-02-12T16:37:46.032919+0000 osd.3 (osd.3) 87 : cluster [DBG] 7.2 scrub ok 2024-02-12T16:37:47.362 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:46 smithi123 bash[19119]: cluster 2024-02-12T16:37:46.159776+0000 mgr.y (mgr.14146) 1601 : cluster [DBG] pgmap v2292: 47 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 44 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2444/98454 objects misplaced (2.482%); 895 KiB/s, 13 objects/s recovering 2024-02-12T16:37:47.386 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:47.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:47.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:47.514 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:47.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:47.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:47.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:47.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:47.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:47.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:47.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:47.922 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:47.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:48.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:48.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:48.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:48.184 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:47 smithi169 bash[19604]: cluster 2024-02-12T16:37:46.720407+0000 osd.5 (osd.5) 169 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:37:48.185 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:47 smithi169 bash[19604]: cluster 2024-02-12T16:37:46.809946+0000 osd.5 (osd.5) 170 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:37:48.185 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:47 smithi169 bash[19604]: cluster 2024-02-12T16:37:47.140106+0000 osd.11 (osd.11) 189 : cluster [DBG] 7.9 scrub starts 2024-02-12T16:37:48.185 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:47 smithi121 bash[18489]: cluster 2024-02-12T16:37:46.720407+0000 osd.5 (osd.5) 169 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:37:48.185 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:47 smithi121 bash[18489]: cluster 2024-02-12T16:37:46.809946+0000 osd.5 (osd.5) 170 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:37:48.185 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:47 smithi121 bash[18489]: cluster 2024-02-12T16:37:47.140106+0000 osd.11 (osd.11) 189 : cluster [DBG] 7.9 scrub starts 2024-02-12T16:37:48.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:37:48.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:48.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:48.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:48.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:47 smithi123 bash[19119]: cluster 2024-02-12T16:37:46.720407+0000 osd.5 (osd.5) 169 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:37:48.332 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:47 smithi123 bash[19119]: cluster 2024-02-12T16:37:46.809946+0000 osd.5 (osd.5) 170 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:37:48.332 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:47 smithi123 bash[19119]: cluster 2024-02-12T16:37:47.140106+0000 osd.11 (osd.11) 189 : cluster [DBG] 7.9 scrub starts 2024-02-12T16:37:48.347 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:48.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:48.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:48.498 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:48.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:48.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:48.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:48.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:48.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:48.755 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:48.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:48.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:48.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:48.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:49.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:49.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:48 smithi123 bash[19119]: cluster 2024-02-12T16:37:47.691433+0000 osd.9 (osd.9) 143 : cluster [DBG] 7.a scrub starts 2024-02-12T16:37:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:48 smithi123 bash[19119]: cluster 2024-02-12T16:37:47.945758+0000 osd.11 (osd.11) 190 : cluster [DBG] 7.9 scrub ok 2024-02-12T16:37:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:48 smithi123 bash[19119]: cluster 2024-02-12T16:37:48.002559+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:48 smithi123 bash[19119]: cluster 2024-02-12T16:37:48.002610+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:49.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:48 smithi123 bash[19119]: cluster 2024-02-12T16:37:48.160316+0000 mgr.y (mgr.14146) 1602 : cluster [DBG] pgmap v2293: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 45 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 2444/91686 objects misplaced (2.666%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:37:49.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:49.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:49.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:49.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:48 smithi169 bash[19604]: cluster 2024-02-12T16:37:47.691433+0000 osd.9 (osd.9) 143 : cluster [DBG] 7.a scrub starts 2024-02-12T16:37:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:48 smithi169 bash[19604]: cluster 2024-02-12T16:37:47.945758+0000 osd.11 (osd.11) 190 : cluster [DBG] 7.9 scrub ok 2024-02-12T16:37:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:48 smithi169 bash[19604]: cluster 2024-02-12T16:37:48.002559+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:48 smithi169 bash[19604]: cluster 2024-02-12T16:37:48.002610+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:48 smithi169 bash[19604]: cluster 2024-02-12T16:37:48.160316+0000 mgr.y (mgr.14146) 1602 : cluster [DBG] pgmap v2293: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 45 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 2444/91686 objects misplaced (2.666%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:37:49.331 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:49.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:48 smithi121 bash[18489]: cluster 2024-02-12T16:37:47.691433+0000 osd.9 (osd.9) 143 : cluster [DBG] 7.a scrub starts 2024-02-12T16:37:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:48 smithi121 bash[18489]: cluster 2024-02-12T16:37:47.945758+0000 osd.11 (osd.11) 190 : cluster [DBG] 7.9 scrub ok 2024-02-12T16:37:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:48 smithi121 bash[18489]: cluster 2024-02-12T16:37:48.002559+0000 osd.2 (osd.2) 117 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:48 smithi121 bash[18489]: cluster 2024-02-12T16:37:48.002610+0000 osd.2 (osd.2) 118 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:48 smithi121 bash[18489]: cluster 2024-02-12T16:37:48.160316+0000 mgr.y (mgr.14146) 1602 : cluster [DBG] pgmap v2293: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 45 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 2444/91686 objects misplaced (2.666%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:37:49.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:49.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:49.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:49.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:49.601 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:49.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:49.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:49.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:49.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:49.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:49.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:49.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:49.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:50.015 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:50.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:50.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:50.144 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:50.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:50.248 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:50.254 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:49 smithi123 bash[19119]: cluster 2024-02-12T16:37:48.923256+0000 osd.9 (osd.9) 144 : cluster [DBG] 7.a scrub ok 2024-02-12T16:37:50.254 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:49 smithi123 bash[19119]: cluster 2024-02-12T16:37:48.940719+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e819: 12 total, 12 up, 12 in 2024-02-12T16:37:50.274 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:50.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:50.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:49 smithi169 bash[19604]: cluster 2024-02-12T16:37:48.923256+0000 osd.9 (osd.9) 144 : cluster [DBG] 7.a scrub ok 2024-02-12T16:37:50.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:49 smithi169 bash[19604]: cluster 2024-02-12T16:37:48.940719+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e819: 12 total, 12 up, 12 in 2024-02-12T16:37:50.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:50.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:49 smithi121 bash[18489]: cluster 2024-02-12T16:37:48.923256+0000 osd.9 (osd.9) 144 : cluster [DBG] 7.a scrub ok 2024-02-12T16:37:50.388 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:49 smithi121 bash[18489]: cluster 2024-02-12T16:37:48.940719+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e819: 12 total, 12 up, 12 in 2024-02-12T16:37:50.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:50.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:50.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:50.548 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:50.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:50.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:50.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:50.685 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:50.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:50.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:50.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:50.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:50.953 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:50 smithi123 bash[19119]: cluster 2024-02-12T16:37:50.161086+0000 mgr.y (mgr.14146) 1603 : cluster [DBG] pgmap v2295: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 2076/88832 objects misplaced (2.337%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:50.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:50.996 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:51.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:51.116 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:50 smithi121 bash[18489]: cluster 2024-02-12T16:37:50.161086+0000 mgr.y (mgr.14146) 1603 : cluster [DBG] pgmap v2295: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 2076/88832 objects misplaced (2.337%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:51.143 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:51.248 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.283 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:51.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:50 smithi169 bash[19604]: cluster 2024-02-12T16:37:50.161086+0000 mgr.y (mgr.14146) 1603 : cluster [DBG] pgmap v2295: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 2076/88832 objects misplaced (2.337%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:37:51.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:51.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:51.568 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:51.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:51.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:51.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:51.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:51.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:51.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:52.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:52.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:52.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:52.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:51 smithi123 bash[19119]: cluster 2024-02-12T16:37:50.672742+0000 osd.5 (osd.5) 171 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:37:52.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:51 smithi123 bash[19119]: cluster 2024-02-12T16:37:50.700997+0000 osd.5 (osd.5) 172 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:37:52.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:52.217 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:51 smithi169 bash[19604]: cluster 2024-02-12T16:37:50.672742+0000 osd.5 (osd.5) 171 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:37:52.217 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:51 smithi169 bash[19604]: cluster 2024-02-12T16:37:50.700997+0000 osd.5 (osd.5) 172 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:37:52.234 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:52.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:52.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:52.361 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:52.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:52.415 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:51 smithi121 bash[18489]: cluster 2024-02-12T16:37:50.672742+0000 osd.5 (osd.5) 171 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:37:52.415 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:51 smithi121 bash[18489]: cluster 2024-02-12T16:37:50.700997+0000 osd.5 (osd.5) 172 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:37:52.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:52.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:52.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:52.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:52.624 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:52.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:52.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:52.766 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:52.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:52.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:52.903 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:52.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:53.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:53.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:53.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:53.045 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:37:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:52 smithi169 bash[19604]: cluster 2024-02-12T16:37:52.161561+0000 mgr.y (mgr.14146) 1604 : cluster [DBG] pgmap v2296: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s; 2076/81058 objects misplaced (2.561%); 901 KiB/s, 14 objects/s recovering 2024-02-12T16:37:53.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:53.166 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:53.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:53.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:53.279 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:52 smithi123 bash[19119]: cluster 2024-02-12T16:37:52.161561+0000 mgr.y (mgr.14146) 1604 : cluster [DBG] pgmap v2296: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s; 2076/81058 objects misplaced (2.561%); 901 KiB/s, 14 objects/s recovering 2024-02-12T16:37:53.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:53.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:53.392 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:37:53.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:53.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:52 smithi121 bash[18489]: cluster 2024-02-12T16:37:52.161561+0000 mgr.y (mgr.14146) 1604 : cluster [DBG] pgmap v2296: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s; 2076/81058 objects misplaced (2.561%); 901 KiB/s, 14 objects/s recovering 2024-02-12T16:37:53.443 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:37:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:53.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:53.482 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:37:53.507 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:37:53.531 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 819, 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-02-12T15:47:18.273813+0000", 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-02-12T16:37:48.930491+0000", 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: "last_up_change": "2024-02-12T16:23:01.123593+0000", 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: "last_in_change": "2024-02-12T16:19:27.892497+0000", 2024-02-12T16:37:53.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_num": 5804032, 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_set": [ 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "nodeep-scrub", 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "pglog_hardlimit", 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "purged_snapdirs", 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_deletes", 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "sortbitwise" 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_version": 36, 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_snapshot": "", 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_max": 7, 2024-02-12T16:37:53.555 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd": 12, 2024-02-12T16:37:53.556 INFO:teuthology.orchestra.run.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:37:53.556 INFO:teuthology.orchestra.run.smithi121.stdout: "min_compat_client": "luminous", 2024-02-12T16:37:53.556 INFO:teuthology.orchestra.run.smithi121.stdout: "require_osd_release": "pacific", 2024-02-12T16:37:53.556 INFO:teuthology.orchestra.run.smithi121.stdout: "pools": [ 2024-02-12T16:37:53.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.556 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 1, 2024-02-12T16:37:53.556 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T15:51:35.886878+0000", 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 31, 2024-02-12T16:37:53.557 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 31, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 31, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 31, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 31, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "701", 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:37:53.558 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "663", 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:37:53.559 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:37:53.560 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:37:53.560 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:37:53.560 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:37:53.560 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:37:53.560 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:37:53.560 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:37:53.560 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:37:53.561 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "options": { 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_max": 32, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_min": 1 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:37:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_devicehealth": {} 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 7, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "unique_pool_5", 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T16:34:06.973798+0000", 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:37:53.563 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 16, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 16, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 16, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 16, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 16, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:37:53.564 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "759", 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:37:53.565 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:37:53.566 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:37:53.567 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "options": {}, 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: "rados": {} 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0", 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.6634674072265625, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 8, 2024-02-12T16:37:53.569 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 808, 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.570 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6802", 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6804", 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.571 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6803", 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6801/1815232143", 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6802/1815232143", 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6804/1815232143", 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6803/1815232143", 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.572 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "c6140309-5703-4538-9739-276fbbdaa880", 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 0.9075927734375, 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.573 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 13, 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 752, 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6805", 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6806", 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6808", 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.575 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6807", 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6805/2968433670", 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6806/2968433670", 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6808/2968433670", 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6807/2968433670", 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170", 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.8111114501953125, 2024-02-12T16:37:53.577 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 21, 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 780, 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6809", 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6810", 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6812", 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6811", 2024-02-12T16:37:53.580 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6809/2158187852", 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6810/2158187852", 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6812/2158187852", 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6811/2158187852", 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:37:53.581 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd", 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 26, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 414, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 522, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 809, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 415, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.582 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6813", 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6814", 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.583 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6816", 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.584 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6815", 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6813/2551349828", 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6814/2551349828", 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6816/2551349828", 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6815/2551349828", 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.585 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31", 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 32, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 797, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.586 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6800", 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6801", 2024-02-12T16:37:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6803", 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6802", 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.589 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6800/3447700123", 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6801/3447700123", 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6803/3447700123", 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6802/3447700123", 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602", 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.590 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 39, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 780, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6804", 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.591 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6805", 2024-02-12T16:37:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:37:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6807", 2024-02-12T16:37:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:37:53.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.594 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.594 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.594 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.594 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6806", 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6804/961317306", 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6805/961317306", 2024-02-12T16:37:53.595 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6807/961317306", 2024-02-12T16:37:53.596 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6806/961317306", 2024-02-12T16:37:53.596 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.596 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.596 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.596 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.597 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:37:53.597 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0", 2024-02-12T16:37:53.597 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.597 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.597 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.597 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.597 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.598 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.598 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 48, 2024-02-12T16:37:53.598 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 752, 2024-02-12T16:37:53.598 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.598 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.598 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.598 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6808", 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.599 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.600 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.600 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.600 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6809", 2024-02-12T16:37:53.600 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:37:53.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.601 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.601 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.601 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.601 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6811", 2024-02-12T16:37:53.602 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:37:53.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.602 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.602 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6810", 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.603 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6808/1377261784", 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6809/1377261784", 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6811/1377261784", 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6810/1377261784", 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.605 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:37:53.605 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515", 2024-02-12T16:37:53.605 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.605 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.605 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.606 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.606 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.606 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.606 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 56, 2024-02-12T16:37:53.606 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 737, 2024-02-12T16:37:53.606 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6812", 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:37:53.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6813", 2024-02-12T16:37:53.608 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6815", 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:37:53.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6814", 2024-02-12T16:37:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:37:53.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.611 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6812/1702613957", 2024-02-12T16:37:53.611 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6813/1702613957", 2024-02-12T16:37:53.611 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6815/1702613957", 2024-02-12T16:37:53.611 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6814/1702613957", 2024-02-12T16:37:53.612 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.612 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.612 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.613 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:37:53.613 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75", 2024-02-12T16:37:53.613 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.613 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.613 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.613 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.613 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 65, 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 796, 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.615 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.615 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6800", 2024-02-12T16:37:53.615 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:37:53.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.615 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.616 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.616 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.616 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.616 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6801", 2024-02-12T16:37:53.616 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:37:53.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.617 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.617 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.617 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.617 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.618 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6803", 2024-02-12T16:37:53.618 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:37:53.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.618 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.618 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.618 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.619 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6802", 2024-02-12T16:37:53.619 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:37:53.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6800/2551075225", 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6801/2551075225", 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6803/2551075225", 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6802/2551075225", 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.620 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15", 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.906097412109375, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 74, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 752, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6804", 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6805", 2024-02-12T16:37:53.622 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.623 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6807", 2024-02-12T16:37:53.624 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:37:53.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.624 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.624 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.625 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6806", 2024-02-12T16:37:53.625 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:37:53.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.626 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6804/2339555071", 2024-02-12T16:37:53.626 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6805/2339555071", 2024-02-12T16:37:53.626 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6807/2339555071", 2024-02-12T16:37:53.626 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6806/2339555071", 2024-02-12T16:37:53.626 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.626 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.626 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.627 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.627 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:37:53.627 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "91f74dca-503a-45d1-9647-df4804d58515", 2024-02-12T16:37:53.627 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.628 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.628 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.628 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.628 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.628 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.628 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 83, 2024-02-12T16:37:53.628 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 796, 2024-02-12T16:37:53.629 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.629 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.629 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.629 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.629 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.629 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6808", 2024-02-12T16:37:53.630 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:37:53.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.630 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.630 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.630 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6809", 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.631 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.632 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.632 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6811", 2024-02-12T16:37:53.632 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:37:53.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.632 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.633 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.633 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.633 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.633 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6810", 2024-02-12T16:37:53.633 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:37:53.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.634 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.634 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6808/391306610", 2024-02-12T16:37:53.634 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6809/391306610", 2024-02-12T16:37:53.634 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6811/391306610", 2024-02-12T16:37:53.634 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6810/391306610", 2024-02-12T16:37:53.634 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.635 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.635 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.635 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.635 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:37:53.636 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf", 2024-02-12T16:37:53.636 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:37:53.636 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:37:53.636 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:37:53.636 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:37:53.636 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:37:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:37:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 93, 2024-02-12T16:37:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 802, 2024-02-12T16:37:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:37:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:37:53.637 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6812", 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.638 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:37:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6813", 2024-02-12T16:37:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:37:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.640 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:37:53.640 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.640 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6815", 2024-02-12T16:37:53.640 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:37:53.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.641 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.641 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:37:53.641 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:37:53.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.641 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:37:53.642 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6814", 2024-02-12T16:37:53.642 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:37:53.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.642 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.642 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6812/3696812305", 2024-02-12T16:37:53.642 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6813/3696812305", 2024-02-12T16:37:53.643 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3696812305", 2024-02-12T16:37:53.643 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6814/3696812305", 2024-02-12T16:37:53.643 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:37:53.643 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:37:53.643 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:37:53.643 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_xinfo": [ 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.644 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:06.982636+0000", 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:18.713072+0000", 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.645 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:48.002612+0000", 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:37:53.646 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:36:55.826290+0000", 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 414 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.647 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.648 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:31.743154+0000", 2024-02-12T16:37:53.648 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.648 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:37:53.648 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.648 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.649 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.649 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.649 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.649 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:36:58.662994+0000", 2024-02-12T16:37:53.649 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.650 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:37:53.650 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.650 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.650 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.650 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.650 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.651 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:43.023664+0000", 2024-02-12T16:37:53.651 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.651 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:37:53.651 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.651 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:28.599958+0000", 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:37:53.652 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:33.786215+0000", 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.653 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:36:38.152983+0000", 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:06.490613+0000", 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:24.591779+0000", 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap": [ 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "1.0", 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:37:53.655 INFO:teuthology.orchestra.run.smithi121.stdout: 9, 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: 0 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap_items": [], 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_temp": [ 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "7.4", 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: 0 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_temp": [], 2024-02-12T16:37:53.656 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklist": { 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/2474849789": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/253689008": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/3531151725": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/1778611055": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/28718860": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/445290372": "2024-02-13T15:47:47.232761+0000" 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "range_blocklist": {}, 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profiles": { 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "default": { 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-failure-domain": "osd", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "k": "2", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "m": "1", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin": "jerasure", 2024-02-12T16:37:53.657 INFO:teuthology.orchestra.run.smithi121.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "technique": "reed_sol_van" 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps_queue": [], 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "new_removed_snaps": [], 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "new_purged_snaps": [], 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_node_flags": {}, 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "device_class_flags": {}, 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": { 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:37:53.658 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:37:53.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:53.660 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:37:53.660 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:37:53.660 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:37:53.660 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:37:53.682 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:37:53.682 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:53.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:53.716 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:37:53.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.745 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:37:53.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:53.821 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:53.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:53.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:53.948 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:53.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:53.962 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:53 smithi123 bash[19119]: audit 2024-02-12T16:37:53.347985+0000 mon.b (mon.2) 191 : audit [DBG] from='client.? v1:172.21.15.121:0/596418035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:53.962 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:53 smithi123 bash[19119]: audit 2024-02-12T16:37:53.547301+0000 mon.a (mon.0) 2763 : audit [DBG] from='client.? v1:172.21.15.121:0/3904437949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:37:54.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:54.079 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:54.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:54.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:53 smithi121 bash[18489]: audit 2024-02-12T16:37:53.347985+0000 mon.b (mon.2) 191 : audit [DBG] from='client.? v1:172.21.15.121:0/596418035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:54 smithi121 bash[18489]: audit 2024-02-12T16:37:53.547301+0000 mon.a (mon.0) 2763 : audit [DBG] from='client.? v1:172.21.15.121:0/3904437949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:37:54.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:54.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:53 smithi169 bash[19604]: audit 2024-02-12T16:37:53.347985+0000 mon.b (mon.2) 191 : audit [DBG] from='client.? v1:172.21.15.121:0/596418035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:37:54.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:53 smithi169 bash[19604]: audit 2024-02-12T16:37:53.547301+0000 mon.a (mon.0) 2763 : audit [DBG] from='client.? v1:172.21.15.121:0/3904437949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:37:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:54.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:54.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:54.342 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:54.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:54.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:54.471 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:54.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:54.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:54.597 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:54.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:54.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:54.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:54.725 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:54.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:54.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:54.874 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:54.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:54.970 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:54 smithi169 bash[19604]: cluster 2024-02-12T16:37:54.161952+0000 mgr.y (mgr.14146) 1605 : cluster [DBG] pgmap v2297: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 1749/79010 objects misplaced (2.214%); 749 KiB/s, 11 objects/s recovering 2024-02-12T16:37:54.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:55.005 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:55.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:55.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:55.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:54 smithi121 bash[18489]: cluster 2024-02-12T16:37:54.161952+0000 mgr.y (mgr.14146) 1605 : cluster [DBG] pgmap v2297: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 1749/79010 objects misplaced (2.214%); 749 KiB/s, 11 objects/s recovering 2024-02-12T16:37:55.242 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:55.248 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:54 smithi123 bash[19119]: cluster 2024-02-12T16:37:54.161952+0000 mgr.y (mgr.14146) 1605 : cluster [DBG] pgmap v2297: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 1749/79010 objects misplaced (2.214%); 749 KiB/s, 11 objects/s recovering 2024-02-12T16:37:55.267 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:55.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:55.401 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:55.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:55.537 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:55.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:55.672 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:55.746 INFO:tasks.thrashosds.thrasher:j is {'epoch': 819, 'fsid': 'eb2349b2-c9bd-11ee-95b9-87774f69a715', 'created': '2024-02-12T15:47:18.273813+0000', 'modified': '2024-02-12T16:37:48.930491+0000', 'last_up_change': '2024-02-12T16:23:01.123593+0000', 'last_in_change': '2024-02-12T16:19:27.892497+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 36, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '701', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '663', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-02-12T16:34:06.973798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '759', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'up': 1, 'in': 1, 'weight': 0.6634674072265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 808, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 1815232143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 1815232143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 1815232143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 1815232143}]}, 'public_addr': '172.21.15.121:6801/1815232143', 'cluster_addr': '172.21.15.121:6802/1815232143', 'heartbeat_back_addr': '172.21.15.121:6804/1815232143', 'heartbeat_front_addr': '172.21.15.121:6803/1815232143', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c6140309-5703-4538-9739-276fbbdaa880', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9075927734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 752, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2968433670}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2968433670}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 2968433670}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2968433670}]}, 'public_addr': '172.21.15.121:6805/2968433670', 'cluster_addr': '172.21.15.121:6806/2968433670', 'heartbeat_back_addr': '172.21.15.121:6808/2968433670', 'heartbeat_front_addr': '172.21.15.121:6807/2968433670', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'up': 1, 'in': 1, 'weight': 0.8111114501953125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 780, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 2158187852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 2158187852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 2158187852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 2158187852}]}, 'public_addr': '172.21.15.121:6809/2158187852', 'cluster_addr': '172.21.15.121:6810/2158187852', 'heartbeat_back_addr': '172.21.15.121:6812/2158187852', 'heartbeat_front_addr': '172.21.15.121:6811/2158187852', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5f824526-596d-42e1-8b7e-cd8676ded6dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 26, 'last_clean_end': 414, 'up_from': 522, 'up_thru': 809, 'down_at': 415, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 2551349828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 2551349828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6816', 'nonce': 2551349828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 2551349828}]}, 'public_addr': '172.21.15.121:6813/2551349828', 'cluster_addr': '172.21.15.121:6814/2551349828', 'heartbeat_back_addr': '172.21.15.121:6816/2551349828', 'heartbeat_front_addr': '172.21.15.121:6815/2551349828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '14c2f878-99ce-4b51-8d9c-131c1d883b31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 797, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6800', 'nonce': 3447700123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6801', 'nonce': 3447700123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6803', 'nonce': 3447700123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6802', 'nonce': 3447700123}]}, 'public_addr': '172.21.15.123:6800/3447700123', 'cluster_addr': '172.21.15.123:6801/3447700123', 'heartbeat_back_addr': '172.21.15.123:6803/3447700123', 'heartbeat_front_addr': '172.21.15.123:6802/3447700123', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 780, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6804', 'nonce': 961317306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6805', 'nonce': 961317306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6807', 'nonce': 961317306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6806', 'nonce': 961317306}]}, 'public_addr': '172.21.15.123:6804/961317306', 'cluster_addr': '172.21.15.123:6805/961317306', 'heartbeat_back_addr': '172.21.15.123:6807/961317306', 'heartbeat_front_addr': '172.21.15.123:6806/961317306', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '71e715e6-eac1-4386-927e-51df0cc419c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 752, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6808', 'nonce': 1377261784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6809', 'nonce': 1377261784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6811', 'nonce': 1377261784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6810', 'nonce': 1377261784}]}, 'public_addr': '172.21.15.123:6808/1377261784', 'cluster_addr': '172.21.15.123:6809/1377261784', 'heartbeat_back_addr': '172.21.15.123:6811/1377261784', 'heartbeat_front_addr': '172.21.15.123:6810/1377261784', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b419b04f-0c0c-495a-aa9b-25c334159515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 737, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6812', 'nonce': 1702613957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6813', 'nonce': 1702613957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6815', 'nonce': 1702613957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6814', 'nonce': 1702613957}]}, 'public_addr': '172.21.15.123:6812/1702613957', 'cluster_addr': '172.21.15.123:6813/1702613957', 'heartbeat_back_addr': '172.21.15.123:6815/1702613957', 'heartbeat_front_addr': '172.21.15.123:6814/1702613957', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1591e7bc-5b86-4401-803d-5325062ffb75', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 796, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6800', 'nonce': 2551075225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 2551075225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 2551075225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6802', 'nonce': 2551075225}]}, 'public_addr': '172.21.15.169:6800/2551075225', 'cluster_addr': '172.21.15.169:6801/2551075225', 'heartbeat_back_addr': '172.21.15.169:6803/2551075225', 'heartbeat_front_addr': '172.21.15.169:6802/2551075225', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '522d61df-8dc1-4915-83fd-f94d0d572c15', 'up': 1, 'in': 1, 'weight': 0.906097412109375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 752, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6804', 'nonce': 2339555071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 2339555071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 2339555071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6806', 'nonce': 2339555071}]}, 'public_addr': '172.21.15.169:6804/2339555071', 'cluster_addr': '172.21.15.169:6805/2339555071', 'heartbeat_back_addr': '172.21.15.169:6807/2339555071', 'heartbeat_front_addr': '172.21.15.169:6806/2339555071', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '91f74dca-503a-45d1-9647-df4804d58515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 796, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6808', 'nonce': 391306610}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 391306610}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 391306610}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6810', 'nonce': 391306610}]}, 'public_addr': '172.21.15.169:6808/391306610', 'cluster_addr': '172.21.15.169:6809/391306610', 'heartbeat_back_addr': '172.21.15.169:6811/391306610', 'heartbeat_front_addr': '172.21.15.169:6810/391306610', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '788b4f47-eadf-4601-a5b4-51fb6f04fbdf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 93, 'up_thru': 802, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6812', 'nonce': 3696812305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3696812305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3696812305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6814', 'nonce': 3696812305}]}, 'public_addr': '172.21.15.169:6812/3696812305', 'cluster_addr': '172.21.15.169:6813/3696812305', 'heartbeat_back_addr': '172.21.15.169:6815/3696812305', 'heartbeat_front_addr': '172.21.15.169:6814/3696812305', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:06.982636+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:18.713072+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:48.002612+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:36:55.826290+0000', 'dead_epoch': 414}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:31.743154+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:36:58.662994+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:43.023664+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:28.599958+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:33.786215+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:36:38.152983+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:06.490613+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:24.591779+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [9, 0]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '7.4', 'osds': [4, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.121:0/2474849789': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/253689008': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:6800/3531151725': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:0/1778611055': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:6800/28718860': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/445290372': '2024-02-13T15:47:47.232761+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:37:55.747 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 1.0 2024-02-12T16:37:55.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd rm-pg-upmap 1.0 2024-02-12T16:37:55.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:55.802 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:55.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:55.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:37:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:55.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:37:56.008 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:55 smithi121 bash[18489]: cluster 2024-02-12T16:37:54.198016+0000 osd.9 (osd.9) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:56.009 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:55 smithi121 bash[18489]: cluster 2024-02-12T16:37:54.198059+0000 osd.9 (osd.9) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:56.009 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:55 smithi121 bash[18489]: cluster 2024-02-12T16:37:54.966419+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e820: 12 total, 12 up, 12 in 2024-02-12T16:37:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:55 smithi169 bash[19604]: cluster 2024-02-12T16:37:54.198016+0000 osd.9 (osd.9) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:55 smithi169 bash[19604]: cluster 2024-02-12T16:37:54.198059+0000 osd.9 (osd.9) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:55 smithi169 bash[19604]: cluster 2024-02-12T16:37:54.966419+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e820: 12 total, 12 up, 12 in 2024-02-12T16:37:56.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:56.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:56.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:56.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:56.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:56.231 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:55 smithi123 bash[19119]: cluster 2024-02-12T16:37:54.198016+0000 osd.9 (osd.9) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:37:56.231 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:55 smithi123 bash[19119]: cluster 2024-02-12T16:37:54.198059+0000 osd.9 (osd.9) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:37:56.231 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:55 smithi123 bash[19119]: cluster 2024-02-12T16:37:54.966419+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e820: 12 total, 12 up, 12 in 2024-02-12T16:37:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:56.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:56.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:56.363 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:56.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:56.471 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:56.499 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:56.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:56.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:56.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:56.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:56.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:56.766 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:56.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:56.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:56.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:56.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:56.983 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:37:56.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:57.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:57.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:57.149 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:57.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:57.259 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:56 smithi123 bash[19119]: cluster 2024-02-12T16:37:56.162412+0000 mgr.y (mgr.14146) 1606 : cluster [DBG] pgmap v2299: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 1749/75402 objects misplaced (2.320%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:37:57.259 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:56 smithi123 bash[19119]: audit 2024-02-12T16:37:56.570439+0000 mon.a (mon.0) 2765 : audit [INF] from='client.26698 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:57.259 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:56 smithi123 bash[19119]: audit 2024-02-12T16:37:56.571326+0000 mon.c (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.121:0/3517908422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:57.277 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:56 smithi169 bash[19604]: cluster 2024-02-12T16:37:56.162412+0000 mgr.y (mgr.14146) 1606 : cluster [DBG] pgmap v2299: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 1749/75402 objects misplaced (2.320%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:37:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:56 smithi169 bash[19604]: audit 2024-02-12T16:37:56.570439+0000 mon.a (mon.0) 2765 : audit [INF] from='client.26698 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:56 smithi169 bash[19604]: audit 2024-02-12T16:37:56.571326+0000 mon.c (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.121:0/3517908422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:57.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:57.405 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:57.427 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:56 smithi121 bash[18489]: cluster 2024-02-12T16:37:56.162412+0000 mgr.y (mgr.14146) 1606 : cluster [DBG] pgmap v2299: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 1749/75402 objects misplaced (2.320%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:37:57.427 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:56 smithi121 bash[18489]: audit 2024-02-12T16:37:56.570439+0000 mon.a (mon.0) 2765 : audit [INF] from='client.26698 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:57.427 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:56 smithi121 bash[18489]: audit 2024-02-12T16:37:56.571326+0000 mon.c (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.121:0/3517908422' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:37:57.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:57.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:37:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:37:57.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:57.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:57.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:57.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:37:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:37:57.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:37:57.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:57.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:37:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:57.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:37:58.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:58.084 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:57 smithi123 bash[19119]: cluster 2024-02-12T16:37:56.760651+0000 osd.5 (osd.5) 173 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:37:58.084 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:57 smithi123 bash[19119]: cluster 2024-02-12T16:37:56.814092+0000 osd.5 (osd.5) 174 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:37:58.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:57 smithi123 bash[19119]: cluster 2024-02-12T16:37:56.962659+0000 mon.a (mon.0) 2766 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:37:58.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:57 smithi123 bash[19119]: cluster 2024-02-12T16:37:56.962691+0000 mon.a (mon.0) 2767 : cluster [INF] Cluster is now healthy 2024-02-12T16:37:58.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:57 smithi123 bash[19119]: audit 2024-02-12T16:37:56.973336+0000 mon.a (mon.0) 2768 : audit [INF] from='client.26698 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:37:58.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:57 smithi123 bash[19119]: cluster 2024-02-12T16:37:56.973395+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e821: 12 total, 12 up, 12 in 2024-02-12T16:37:58.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:57 smithi123 bash[19119]: cluster 2024-02-12T16:37:57.158038+0000 osd.11 (osd.11) 191 : cluster [DBG] 7.5 scrub starts 2024-02-12T16:37:58.145 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:37:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:58.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:37:58.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:58.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:57 smithi121 bash[18489]: cluster 2024-02-12T16:37:56.760651+0000 osd.5 (osd.5) 173 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:37:58.258 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:57 smithi121 bash[18489]: cluster 2024-02-12T16:37:56.814092+0000 osd.5 (osd.5) 174 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:37:58.258 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:57 smithi121 bash[18489]: cluster 2024-02-12T16:37:56.962659+0000 mon.a (mon.0) 2766 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:37:58.258 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:57 smithi121 bash[18489]: cluster 2024-02-12T16:37:56.962691+0000 mon.a (mon.0) 2767 : cluster [INF] Cluster is now healthy 2024-02-12T16:37:58.258 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:57 smithi121 bash[18489]: audit 2024-02-12T16:37:56.973336+0000 mon.a (mon.0) 2768 : audit [INF] from='client.26698 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:37:58.258 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:57 smithi121 bash[18489]: cluster 2024-02-12T16:37:56.973395+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e821: 12 total, 12 up, 12 in 2024-02-12T16:37:58.258 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:57 smithi121 bash[18489]: cluster 2024-02-12T16:37:57.158038+0000 osd.11 (osd.11) 191 : cluster [DBG] 7.5 scrub starts 2024-02-12T16:37:58.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:58.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:57 smithi169 bash[19604]: cluster 2024-02-12T16:37:56.760651+0000 osd.5 (osd.5) 173 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:37:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:57 smithi169 bash[19604]: cluster 2024-02-12T16:37:56.814092+0000 osd.5 (osd.5) 174 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:37:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:57 smithi169 bash[19604]: cluster 2024-02-12T16:37:56.962659+0000 mon.a (mon.0) 2766 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:37:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:57 smithi169 bash[19604]: cluster 2024-02-12T16:37:56.962691+0000 mon.a (mon.0) 2767 : cluster [INF] Cluster is now healthy 2024-02-12T16:37:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:57 smithi169 bash[19604]: audit 2024-02-12T16:37:56.973336+0000 mon.a (mon.0) 2768 : audit [INF] from='client.26698 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:37:58.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:57 smithi169 bash[19604]: cluster 2024-02-12T16:37:56.973395+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e821: 12 total, 12 up, 12 in 2024-02-12T16:37:58.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:57 smithi169 bash[19604]: cluster 2024-02-12T16:37:57.158038+0000 osd.11 (osd.11) 191 : cluster [DBG] 7.5 scrub starts 2024-02-12T16:37:58.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:58.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:58.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:58.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:58.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:58.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:58.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:58.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:58.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:58.807 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:37:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:58.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:37:58.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:58.950 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:37:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:58.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:37:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:58 smithi169 bash[19604]: cluster 2024-02-12T16:37:58.003119+0000 osd.11 (osd.11) 192 : cluster [DBG] 7.5 scrub ok 2024-02-12T16:37:59.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:58 smithi169 bash[19604]: cluster 2024-02-12T16:37:58.162898+0000 mgr.y (mgr.14146) 1607 : cluster [DBG] pgmap v2301: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 1749/68866 objects misplaced (2.540%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:37:59.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:59.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:37:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:59.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:37:59.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:37:59.236 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:58 smithi121 bash[18489]: cluster 2024-02-12T16:37:58.003119+0000 osd.11 (osd.11) 192 : cluster [DBG] 7.5 scrub ok 2024-02-12T16:37:59.236 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:58 smithi121 bash[18489]: cluster 2024-02-12T16:37:58.162898+0000 mgr.y (mgr.14146) 1607 : cluster [DBG] pgmap v2301: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 1749/68866 objects misplaced (2.540%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:37:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:59.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:37:59.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:59.367 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:37:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:59.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:37:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:58 smithi123 bash[19119]: cluster 2024-02-12T16:37:58.003119+0000 osd.11 (osd.11) 192 : cluster [DBG] 7.5 scrub ok 2024-02-12T16:37:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:58 smithi123 bash[19119]: cluster 2024-02-12T16:37:58.162898+0000 mgr.y (mgr.14146) 1607 : cluster [DBG] pgmap v2301: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 1749/68866 objects misplaced (2.540%); 472 KiB/s, 7 objects/s recovering 2024-02-12T16:37:59.427 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:37:59.427 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:37:59.469 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:59.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:37:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:59.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:37:59.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:59.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:37:59.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:37:59.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:37:59.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:00.036 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:00.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:37:59 smithi123 bash[19119]: audit 2024-02-12T16:37:59.054843+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:00.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:00.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:37:59 smithi121 bash[18489]: audit 2024-02-12T16:37:59.054843+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:00.172 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:00.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:00.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:37:59 smithi169 bash[19604]: audit 2024-02-12T16:37:59.054843+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:00.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:00.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:00.440 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:00.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:38:00.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:00.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:38:00.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:00.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:00.730 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:00.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:00.855 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:00.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:00.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:01.088 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:01.114 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:01.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:01.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:01.227 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:00 smithi169 bash[19604]: cluster 2024-02-12T16:38:00.163394+0000 mgr.y (mgr.14146) 1608 : cluster [DBG] pgmap v2302: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 1411/65886 objects misplaced (2.142%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:38:01.245 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:01.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:01.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:01.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:01.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:01.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:00 smithi123 bash[19119]: cluster 2024-02-12T16:38:00.163394+0000 mgr.y (mgr.14146) 1608 : cluster [DBG] pgmap v2302: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 1411/65886 objects misplaced (2.142%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:38:01.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:00 smithi121 bash[18489]: cluster 2024-02-12T16:38:00.163394+0000 mgr.y (mgr.14146) 1608 : cluster [DBG] pgmap v2302: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s; 1411/65886 objects misplaced (2.142%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:38:01.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:01.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:01.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:01.631 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:01.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:01.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:01.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:01.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:01.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:01.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:01.926 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:01.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:02.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:02.053 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:02.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:02.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:02.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:01 smithi121 bash[18489]: cluster 2024-02-12T16:38:01.114063+0000 osd.11 (osd.11) 193 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:38:02.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:01 smithi121 bash[18489]: cluster 2024-02-12T16:38:01.117063+0000 osd.11 (osd.11) 194 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:38:02.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:01 smithi169 bash[19604]: cluster 2024-02-12T16:38:01.114063+0000 osd.11 (osd.11) 193 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:38:02.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:01 smithi169 bash[19604]: cluster 2024-02-12T16:38:01.117063+0000 osd.11 (osd.11) 194 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:38:02.195 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:02.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:02.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:02.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:02.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:01 smithi123 bash[19119]: cluster 2024-02-12T16:38:01.114063+0000 osd.11 (osd.11) 193 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:38:02.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:01 smithi123 bash[19119]: cluster 2024-02-12T16:38:01.117063+0000 osd.11 (osd.11) 194 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:38:02.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:02.472 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:02.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:02.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:02.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:02.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:02.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:02.771 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:02.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:02.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:02.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:02.908 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:02.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:03.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:03.012 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:03.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:02 smithi123 bash[19119]: cluster 2024-02-12T16:38:02.163899+0000 mgr.y (mgr.14146) 1609 : cluster [DBG] pgmap v2303: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 1411/57550 objects misplaced (2.452%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:03.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:03.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:03.297 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:02 smithi169 bash[19604]: cluster 2024-02-12T16:38:02.163899+0000 mgr.y (mgr.14146) 1609 : cluster [DBG] pgmap v2303: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 1411/57550 objects misplaced (2.452%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:03.319 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:03.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:02 smithi121 bash[18489]: cluster 2024-02-12T16:38:02.163899+0000 mgr.y (mgr.14146) 1609 : cluster [DBG] pgmap v2303: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.34k op/s; 1411/57550 objects misplaced (2.452%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:03.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:03.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:03.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:03.580 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:03.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:03.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:03.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:03.842 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:03.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:03.978 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:03.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:04.002 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:03 smithi123 bash[19119]: cluster 2024-02-12T16:38:02.621775+0000 osd.5 (osd.5) 175 : cluster [DBG] 7.0 scrub starts 2024-02-12T16:38:04.002 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:03 smithi123 bash[19119]: cluster 2024-02-12T16:38:02.992977+0000 osd.5 (osd.5) 176 : cluster [DBG] 7.0 scrub ok 2024-02-12T16:38:04.020 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:04.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:04.091 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:03 smithi169 bash[19604]: cluster 2024-02-12T16:38:02.621775+0000 osd.5 (osd.5) 175 : cluster [DBG] 7.0 scrub starts 2024-02-12T16:38:04.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:03 smithi169 bash[19604]: cluster 2024-02-12T16:38:02.992977+0000 osd.5 (osd.5) 176 : cluster [DBG] 7.0 scrub ok 2024-02-12T16:38:04.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:04.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:04.214 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:04.241 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:04.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:04.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:04.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:03 smithi121 bash[18489]: cluster 2024-02-12T16:38:02.621775+0000 osd.5 (osd.5) 175 : cluster [DBG] 7.0 scrub starts 2024-02-12T16:38:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:03 smithi121 bash[18489]: cluster 2024-02-12T16:38:02.992977+0000 osd.5 (osd.5) 176 : cluster [DBG] 7.0 scrub ok 2024-02-12T16:38:04.422 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:04.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:04.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:04.552 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:04.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:04.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:04.680 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:04.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:04.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:04.818 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:04.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:04.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:04.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:04.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:05.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:05 smithi169 bash[19604]: cluster 2024-02-12T16:38:04.164386+0000 mgr.y (mgr.14146) 1610 : cluster [DBG] pgmap v2304: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 1075/55294 objects misplaced (1.944%); 807 KiB/s, 12 objects/s recovering 2024-02-12T16:38:05.011 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:05 smithi169 bash[19604]: audit 2024-02-12T16:38:04.781807+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:38:05.036 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:05.036 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:05.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:05.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:05.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:05.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:05 smithi121 bash[18489]: cluster 2024-02-12T16:38:04.164386+0000 mgr.y (mgr.14146) 1610 : cluster [DBG] pgmap v2304: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 1075/55294 objects misplaced (1.944%); 807 KiB/s, 12 objects/s recovering 2024-02-12T16:38:05.168 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:05 smithi121 bash[18489]: audit 2024-02-12T16:38:04.781807+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:38:05.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:05.208 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:05 smithi123 bash[19119]: cluster 2024-02-12T16:38:04.164386+0000 mgr.y (mgr.14146) 1610 : cluster [DBG] pgmap v2304: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 1075/55294 objects misplaced (1.944%); 807 KiB/s, 12 objects/s recovering 2024-02-12T16:38:05.208 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:05 smithi123 bash[19119]: audit 2024-02-12T16:38:04.781807+0000 mon.a (mon.0) 2770 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:38:05.239 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:05.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:05.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:05.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:05.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:05.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:05.503 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:05.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:05.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:05.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:05.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:05.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:05.774 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:05.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:05.879 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:05.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:05.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:06.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:06.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:06 smithi121 bash[18489]: audit 2024-02-12T16:38:05.114997+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:06.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:06 smithi121 bash[18489]: audit 2024-02-12T16:38:05.116241+0000 mon.c (mon.1) 90 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:06.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:06 smithi121 bash[18489]: audit 2024-02-12T16:38:05.120172+0000 mon.a (mon.0) 2771 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:38:06.029 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:06.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:06.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:06 smithi169 bash[19604]: audit 2024-02-12T16:38:05.114997+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:06 smithi169 bash[19604]: audit 2024-02-12T16:38:05.116241+0000 mon.c (mon.1) 90 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:06 smithi169 bash[19604]: audit 2024-02-12T16:38:05.120172+0000 mon.a (mon.0) 2771 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:38:06.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:06 smithi123 bash[19119]: audit 2024-02-12T16:38:05.114997+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:06 smithi123 bash[19119]: audit 2024-02-12T16:38:05.116241+0000 mon.c (mon.1) 90 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:06 smithi123 bash[19119]: audit 2024-02-12T16:38:05.120172+0000 mon.a (mon.0) 2771 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:38:06.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:06.444 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:06.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:06.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:06.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:06.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:06.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:06.824 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:06.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:06.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:06.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:07.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:07.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:07.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:07.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:07 smithi121 bash[18489]: cluster 2024-02-12T16:38:05.626589+0000 osd.3 (osd.3) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:07.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:07 smithi121 bash[18489]: cluster 2024-02-12T16:38:05.626645+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:07.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:07 smithi121 bash[18489]: cluster 2024-02-12T16:38:06.121445+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e822: 12 total, 12 up, 12 in 2024-02-12T16:38:07.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:07 smithi121 bash[18489]: cluster 2024-02-12T16:38:06.164729+0000 mgr.y (mgr.14146) 1611 : cluster [DBG] pgmap v2306: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 1075/51614 objects misplaced (2.083%); 808 KiB/s, 12 objects/s recovering 2024-02-12T16:38:07.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:07.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:07.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:07.339 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:07.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:07.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:07 smithi123 bash[19119]: cluster 2024-02-12T16:38:05.626589+0000 osd.3 (osd.3) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:07 smithi123 bash[19119]: cluster 2024-02-12T16:38:05.626645+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:07 smithi123 bash[19119]: cluster 2024-02-12T16:38:06.121445+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e822: 12 total, 12 up, 12 in 2024-02-12T16:38:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:07 smithi123 bash[19119]: cluster 2024-02-12T16:38:06.164729+0000 mgr.y (mgr.14146) 1611 : cluster [DBG] pgmap v2306: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 1075/51614 objects misplaced (2.083%); 808 KiB/s, 12 objects/s recovering 2024-02-12T16:38:07.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:07.514 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:07.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:07 smithi169 bash[19604]: cluster 2024-02-12T16:38:05.626589+0000 osd.3 (osd.3) 88 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:07 smithi169 bash[19604]: cluster 2024-02-12T16:38:05.626645+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:07 smithi169 bash[19604]: cluster 2024-02-12T16:38:06.121445+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e822: 12 total, 12 up, 12 in 2024-02-12T16:38:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:07 smithi169 bash[19604]: cluster 2024-02-12T16:38:06.164729+0000 mgr.y (mgr.14146) 1611 : cluster [DBG] pgmap v2306: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 1075/51614 objects misplaced (2.083%); 808 KiB/s, 12 objects/s recovering 2024-02-12T16:38:07.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:07.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:07.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:07.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:07.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:07.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:07.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:07.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:07.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:08.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:08.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:08.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:08.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:08.187 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:08.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:08.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:08.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:08.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:08.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:08.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:08.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:08.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:08.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:08.603 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:08.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:08.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:08.744 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:08.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:08.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:08.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:08.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:08.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:09.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:09.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:09.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:09.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:09.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:09.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:09.270 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:09.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:09.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:09.410 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:09.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:09.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:09.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:09 smithi123 bash[19119]: cluster 2024-02-12T16:38:08.165153+0000 mgr.y (mgr.14146) 1612 : cluster [DBG] pgmap v2307: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 1075/44788 objects misplaced (2.400%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:38:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:09 smithi169 bash[19604]: cluster 2024-02-12T16:38:08.165153+0000 mgr.y (mgr.14146) 1612 : cluster [DBG] pgmap v2307: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 1075/44788 objects misplaced (2.400%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:38:09.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:09.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:09.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:09 smithi121 bash[18489]: cluster 2024-02-12T16:38:08.165153+0000 mgr.y (mgr.14146) 1612 : cluster [DBG] pgmap v2307: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 1075/44788 objects misplaced (2.400%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:38:09.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:09.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:09.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:09.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:09.884 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:09.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:09.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:10.018 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:10.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:10.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:10.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:10.293 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:10.396 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:10.421 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:10.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:10.547 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:10.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:10.680 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:10.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:10.827 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:10.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:10.957 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:10.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:11.060 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:11.084 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:11.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:11.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:11.224 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:11.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:11.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:11.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:11.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:11.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:11.479 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:11 smithi121 bash[18489]: cluster 2024-02-12T16:38:10.165646+0000 mgr.y (mgr.14146) 1613 : cluster [DBG] pgmap v2308: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 752/41728 objects misplaced (1.802%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:38:11.499 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:11.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:11 smithi169 bash[19604]: cluster 2024-02-12T16:38:10.165646+0000 mgr.y (mgr.14146) 1613 : cluster [DBG] pgmap v2308: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 752/41728 objects misplaced (1.802%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:38:11.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:11.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:11.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:11.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:11.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:11.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:11.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:11.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:12.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:12.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:12.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:12.181 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:12.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:12.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:12.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:12.445 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:12 smithi169 bash[19604]: cluster 2024-02-12T16:38:11.128125+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:12 smithi169 bash[19604]: cluster 2024-02-12T16:38:11.128179+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:12 smithi169 bash[19604]: cluster 2024-02-12T16:38:11.747495+0000 osd.8 (osd.8) 103 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:38:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:12 smithi169 bash[19604]: cluster 2024-02-12T16:38:11.810476+0000 osd.8 (osd.8) 104 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:38:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:12 smithi169 bash[19604]: cluster 2024-02-12T16:38:11.854618+0000 osd.10 (osd.10) 176 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:38:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:12 smithi169 bash[19604]: cluster 2024-02-12T16:38:11.858283+0000 mon.a (mon.0) 2773 : cluster [DBG] osdmap e823: 12 total, 12 up, 12 in 2024-02-12T16:38:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:12 smithi169 bash[19604]: cluster 2024-02-12T16:38:11.892661+0000 osd.10 (osd.10) 177 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:38:12.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:12.557 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:12 smithi121 bash[18489]: cluster 2024-02-12T16:38:11.128125+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:12.557 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:12 smithi121 bash[18489]: cluster 2024-02-12T16:38:11.128179+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:12.558 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:12 smithi121 bash[18489]: cluster 2024-02-12T16:38:11.747495+0000 osd.8 (osd.8) 103 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:38:12.558 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:12 smithi121 bash[18489]: cluster 2024-02-12T16:38:11.810476+0000 osd.8 (osd.8) 104 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:38:12.558 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:12 smithi121 bash[18489]: cluster 2024-02-12T16:38:11.854618+0000 osd.10 (osd.10) 176 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:38:12.558 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:12 smithi121 bash[18489]: cluster 2024-02-12T16:38:11.858283+0000 mon.a (mon.0) 2773 : cluster [DBG] osdmap e823: 12 total, 12 up, 12 in 2024-02-12T16:38:12.558 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:12 smithi121 bash[18489]: cluster 2024-02-12T16:38:11.892661+0000 osd.10 (osd.10) 177 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:38:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:12.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:11 smithi123 bash[19119]: cluster 2024-02-12T16:38:10.165646+0000 mgr.y (mgr.14146) 1613 : cluster [DBG] pgmap v2308: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 752/41728 objects misplaced (1.802%); 742 KiB/s, 11 objects/s recovering 2024-02-12T16:38:12.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:12 smithi123 bash[19119]: cluster 2024-02-12T16:38:11.128125+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:12.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:12 smithi123 bash[19119]: cluster 2024-02-12T16:38:11.128179+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:12.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:12 smithi123 bash[19119]: cluster 2024-02-12T16:38:11.747495+0000 osd.8 (osd.8) 103 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:38:12.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:12 smithi123 bash[19119]: cluster 2024-02-12T16:38:11.810476+0000 osd.8 (osd.8) 104 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:38:12.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:12 smithi123 bash[19119]: cluster 2024-02-12T16:38:11.854618+0000 osd.10 (osd.10) 176 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:38:12.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:12 smithi123 bash[19119]: cluster 2024-02-12T16:38:11.858283+0000 mon.a (mon.0) 2773 : cluster [DBG] osdmap e823: 12 total, 12 up, 12 in 2024-02-12T16:38:12.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:12 smithi123 bash[19119]: cluster 2024-02-12T16:38:11.892661+0000 osd.10 (osd.10) 177 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:38:12.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:12.720 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:12.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:12.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:12.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:12.979 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:12.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:13.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:13.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:13.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:13.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:13.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:13.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:13.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:13.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:13.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:13.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:13.501 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:13 smithi121 bash[18489]: cluster 2024-02-12T16:38:12.166048+0000 mgr.y (mgr.14146) 1614 : cluster [DBG] pgmap v2310: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s; 752/33510 objects misplaced (2.244%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:13.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:13.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:13 smithi169 bash[19604]: cluster 2024-02-12T16:38:12.166048+0000 mgr.y (mgr.14146) 1614 : cluster [DBG] pgmap v2310: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s; 752/33510 objects misplaced (2.244%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:13.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:13 smithi123 bash[19119]: cluster 2024-02-12T16:38:12.166048+0000 mgr.y (mgr.14146) 1614 : cluster [DBG] pgmap v2310: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s; 752/33510 objects misplaced (2.244%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:13.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:13.692 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:13.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:13.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:13.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:13.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:13.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:13.979 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:13.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:14.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:14.111 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:14.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:14.215 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:14.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:14.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:14.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:14.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:14.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:14 smithi121 bash[18489]: cluster 2024-02-12T16:38:13.727691+0000 osd.8 (osd.8) 105 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:38:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:14 smithi121 bash[18489]: cluster 2024-02-12T16:38:13.833790+0000 osd.8 (osd.8) 106 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:38:14.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:14.497 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:38:14.500 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:14.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:14.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:14 smithi169 bash[19604]: cluster 2024-02-12T16:38:13.727691+0000 osd.8 (osd.8) 105 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:38:14.507 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:14 smithi169 bash[19604]: cluster 2024-02-12T16:38:13.833790+0000 osd.8 (osd.8) 106 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:38:14.528 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:38:14.572 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:38:14.603 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:38:14.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:14.613 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:14 smithi123 bash[19119]: cluster 2024-02-12T16:38:13.727691+0000 osd.8 (osd.8) 105 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:38:14.613 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:14 smithi123 bash[19119]: cluster 2024-02-12T16:38:13.833790+0000 osd.8 (osd.8) 106 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:38:14.635 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:38:14.640 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:14.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:14.661 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:38:14.706 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:38:14.741 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:38:14.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:14.785 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:38:14.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:14.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:14.823 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:38:14.856 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:38:14.900 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:38:14.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.917 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:38:14.968 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:14.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26530.0:325599 7.b 7:df900611:::benchmark_data_smithi169_105252_object157005:head [delete] snapc 0=[] ondisk+write+known_if_redirected e823)", 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:38:14.975212+0000", 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.027735666999999999, 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.027770587999999999, 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:38:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26530", 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 325599 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.975212+0000", 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.975212+0000", 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.975210+0000", 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2429999999999999e-06 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.975217+0000", 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7900000000000005e-06 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.975226+0000", 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8989999999999998e-06 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.975232+0000", 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015281796 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990513+0000", 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000135947 2024-02-12T16:38:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990649+0000", 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013217099999999999 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990782+0000", 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015569681 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26530.0:325613 7.b 7:d90dfd62:::benchmark_data_smithi169_105252_object157019:head [delete] snapc 0=[] ondisk+write+known_if_redirected e823)", 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:38:14.988006+0000", 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.014941942999999999, 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015088186, 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26530", 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:38:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 325613 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.988006+0000", 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.988006+0000", 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.988002+0000", 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7680000000000003e-06 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.988012+0000", 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7881999999999999e-05 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.988030+0000", 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1640000000000004e-06 2024-02-12T16:38:15.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.988037+0000", 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071885309999999997 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995226+0000", 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104073 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:38:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995330+0000", 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118448 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995448+0000", 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0074426370000000002 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26530.0:325616 7.b 7:d47346f2:::benchmark_data_smithi169_105252_object157022:head [delete] snapc 0=[] ondisk+write+known_if_redirected e823)", 2024-02-12T16:38:15.022 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:38:14.990582+0000", 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.012365833999999999, 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012607126, 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26530", 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 325616 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:38:15.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990582+0000", 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990582+0000", 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990580+0000", 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.17e-06 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990587+0000", 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9139999999999997e-06 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990596+0000", 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2389999999999999e-06 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.990602+0000", 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095593770000000008 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000161+0000", 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8058000000000003e-05 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000249+0000", 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8628000000000006e-05 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000338+0000", 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0097563470000000003 2024-02-12T16:38:15.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26530.0:325617 7.b 7.49b4b3eb (undecoded) ondisk+write+known_if_redirected e823)", 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:38:14.991279+0000", 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.011668183, 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01199794, 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26530", 2024-02-12T16:38:15.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:38:15.028 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 325617 2024-02-12T16:38:15.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.028 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:38:15.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:38:15.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.991279+0000", 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.991279+0000", 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.991278+0000", 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7270000000000007e-06 2024-02-12T16:38:15.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.991285+0000", 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4508000000000001e-05 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.991300+0000", 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2790000000000004e-06 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:38:15.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.991306+0000", 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6784e-05 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26530.0:325599 7.b e823/752)", 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:38:14.995271+0000", 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0076767650000000003, 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0080713710000000008, 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:38:15.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995271+0000", 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995271+0000", 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995269+0000", 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8089999999999999e-06 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:38:15.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995278+0000", 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4950000000000008e-06 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995287+0000", 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0549999999999999e-06 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:14.995292+0000", 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0999e-05 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:15.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26530.0:325613 7.b e823/752)", 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:38:15.000761+0000", 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0021868669999999999, 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026370650000000001, 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000761+0000", 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000761+0000", 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000759+0000", 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1229999999999997e-06 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000765+0000", 2024-02-12T16:38:15.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9459999999999993e-06 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000775+0000", 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7819999999999999e-06 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:38:15.000781+0000", 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0256000000000001e-05 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:15.037 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:38:15.037 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 6 2024-02-12T16:38:15.037 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:38:15.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:15.114 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:15.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:15.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:15.252 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:15.255 INFO:teuthology.orchestra.run.smithi121.stderr:clear 1.0 pg_upmap mapping 2024-02-12T16:38:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:15.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:15.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:15 smithi121 bash[18489]: cluster 2024-02-12T16:38:14.166518+0000 mgr.y (mgr.14146) 1615 : cluster [DBG] pgmap v2311: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 976 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 455/31246 objects misplaced (1.456%); 923 KiB/s, 14 objects/s recovering 2024-02-12T16:38:15.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:15 smithi121 bash[18489]: audit 2024-02-12T16:38:14.450748+0000 mon.c (mon.1) 91 : audit [DBG] from='client.? v1:172.21.15.121:0/1541330851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:15.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:15 smithi121 bash[18489]: audit 2024-02-12T16:38:15.127848+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? v1:172.21.15.121:0/2493663862' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]: dispatch 2024-02-12T16:38:15.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:15.399 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:15.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:15.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:15.513 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:15 smithi123 bash[19119]: cluster 2024-02-12T16:38:14.166518+0000 mgr.y (mgr.14146) 1615 : cluster [DBG] pgmap v2311: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 976 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 455/31246 objects misplaced (1.456%); 923 KiB/s, 14 objects/s recovering 2024-02-12T16:38:15.513 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:15 smithi123 bash[19119]: audit 2024-02-12T16:38:14.450748+0000 mon.c (mon.1) 91 : audit [DBG] from='client.? v1:172.21.15.121:0/1541330851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:15.513 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:15 smithi123 bash[19119]: audit 2024-02-12T16:38:15.127848+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? v1:172.21.15.121:0/2493663862' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]: dispatch 2024-02-12T16:38:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:15 smithi169 bash[19604]: cluster 2024-02-12T16:38:14.166518+0000 mgr.y (mgr.14146) 1615 : cluster [DBG] pgmap v2311: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 976 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 455/31246 objects misplaced (1.456%); 923 KiB/s, 14 objects/s recovering 2024-02-12T16:38:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:15 smithi169 bash[19604]: audit 2024-02-12T16:38:14.450748+0000 mon.c (mon.1) 91 : audit [DBG] from='client.? v1:172.21.15.121:0/1541330851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:15.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:15 smithi169 bash[19604]: audit 2024-02-12T16:38:15.127848+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? v1:172.21.15.121:0/2493663862' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]: dispatch 2024-02-12T16:38:15.549 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:15.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:15.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:15.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:15.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:15.817 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:15.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:15.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:15.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:15.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:16.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:16.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:16.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:16.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:16.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:16.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:16.254 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:38:16.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:16.343 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:16 smithi169 bash[19604]: audit 2024-02-12T16:38:15.242871+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? v1:172.21.15.121:0/2493663862' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]': finished 2024-02-12T16:38:16.343 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:16 smithi169 bash[19604]: cluster 2024-02-12T16:38:15.243008+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e824: 12 total, 12 up, 12 in 2024-02-12T16:38:16.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:16 smithi169 bash[19604]: audit 2024-02-12T16:38:15.329856+0000 mon.a (mon.0) 2777 : audit [INF] from='client.26777 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:38:16.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:16 smithi169 bash[19604]: audit 2024-02-12T16:38:15.331608+0000 mon.b (mon.2) 192 : audit [INF] from='client.? v1:172.21.15.121:0/1959671670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:38:16.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:16 smithi169 bash[19604]: cluster 2024-02-12T16:38:15.784371+0000 osd.10 (osd.10) 178 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:38:16.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:16 smithi169 bash[19604]: cluster 2024-02-12T16:38:15.825685+0000 osd.10 (osd.10) 179 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:38:16.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:16.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:16.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:16.506 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:16.507 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:16 smithi123 bash[19119]: audit 2024-02-12T16:38:15.242871+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? v1:172.21.15.121:0/2493663862' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]': finished 2024-02-12T16:38:16.507 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:16 smithi123 bash[19119]: cluster 2024-02-12T16:38:15.243008+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e824: 12 total, 12 up, 12 in 2024-02-12T16:38:16.507 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:16 smithi123 bash[19119]: audit 2024-02-12T16:38:15.329856+0000 mon.a (mon.0) 2777 : audit [INF] from='client.26777 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:38:16.507 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:16 smithi123 bash[19119]: audit 2024-02-12T16:38:15.331608+0000 mon.b (mon.2) 192 : audit [INF] from='client.? v1:172.21.15.121:0/1959671670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:38:16.507 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:16 smithi123 bash[19119]: cluster 2024-02-12T16:38:15.784371+0000 osd.10 (osd.10) 178 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:38:16.507 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:16 smithi123 bash[19119]: cluster 2024-02-12T16:38:15.825685+0000 osd.10 (osd.10) 179 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:38:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:16.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:16.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:16.635 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:16.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:16.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:16.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:16 smithi121 bash[18489]: audit 2024-02-12T16:38:15.242871+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? v1:172.21.15.121:0/2493663862' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.0"}]': finished 2024-02-12T16:38:16.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:16 smithi121 bash[18489]: cluster 2024-02-12T16:38:15.243008+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e824: 12 total, 12 up, 12 in 2024-02-12T16:38:16.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:16 smithi121 bash[18489]: audit 2024-02-12T16:38:15.329856+0000 mon.a (mon.0) 2777 : audit [INF] from='client.26777 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:38:16.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:16 smithi121 bash[18489]: audit 2024-02-12T16:38:15.331608+0000 mon.b (mon.2) 192 : audit [INF] from='client.? v1:172.21.15.121:0/1959671670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:38:16.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:16 smithi121 bash[18489]: cluster 2024-02-12T16:38:15.784371+0000 osd.10 (osd.10) 178 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:38:16.754 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:16 smithi121 bash[18489]: cluster 2024-02-12T16:38:15.825685+0000 osd.10 (osd.10) 179 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:38:16.775 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:16.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:16.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:16.954 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:16.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:17.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:17.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:17.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:17.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:17.226 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:17.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:17.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:17.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:17.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:17.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:38:17.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:17.463 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: cluster 2024-02-12T16:38:16.032846+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:17.463 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: cluster 2024-02-12T16:38:16.032895+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:17.463 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: cluster 2024-02-12T16:38:16.166920+0000 mgr.y (mgr.14146) 1616 : cluster [DBG] pgmap v2313: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 861 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 455/27568 objects misplaced (1.650%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: cluster 2024-02-12T16:38:16.239699+0000 mon.a (mon.0) 2778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.243622+0000 mon.a (mon.0) 2779 : audit [INF] from='client.26777 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: cluster 2024-02-12T16:38:16.243665+0000 mon.a (mon.0) 2780 : cluster [DBG] osdmap e825: 12 total, 12 up, 12 in 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.970531+0000 mon.a (mon.0) 2781 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]: dispatch 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.971287+0000 mon.c (mon.1) 92 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]: dispatch 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.971476+0000 mon.a (mon.0) 2782 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]: dispatch 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.971612+0000 mon.c (mon.1) 93 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]: dispatch 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.971926+0000 mon.c (mon.1) 94 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]: dispatch 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.972236+0000 mon.c (mon.1) 95 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]: dispatch 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.972343+0000 mon.a (mon.0) 2783 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]: dispatch 2024-02-12T16:38:17.464 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.972539+0000 mon.c (mon.1) 96 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.972735+0000 mon.c (mon.1) 97 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.972804+0000 mon.a (mon.0) 2784 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.972898+0000 mon.c (mon.1) 98 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973058+0000 mon.c (mon.1) 99 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973154+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973226+0000 mon.c (mon.1) 100 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973405+0000 mon.c (mon.1) 101 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973440+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973576+0000 mon.c (mon.1) 102 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973704+0000 mon.a (mon.0) 2787 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973743+0000 mon.c (mon.1) 103 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973906+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:17.465 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.973958+0000 mon.a (mon.0) 2788 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:17.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.974215+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:17.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.974459+0000 mon.a (mon.0) 2790 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:17.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.974727+0000 mon.a (mon.0) 2791 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:17.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.974977+0000 mon.a (mon.0) 2792 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:17.466 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:17 smithi121 bash[18489]: audit 2024-02-12T16:38:16.975240+0000 mon.a (mon.0) 2793 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:17.489 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:17.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: cluster 2024-02-12T16:38:16.032846+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: cluster 2024-02-12T16:38:16.032895+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: cluster 2024-02-12T16:38:16.166920+0000 mgr.y (mgr.14146) 1616 : cluster [DBG] pgmap v2313: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 861 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 455/27568 objects misplaced (1.650%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: cluster 2024-02-12T16:38:16.239699+0000 mon.a (mon.0) 2778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.243622+0000 mon.a (mon.0) 2779 : audit [INF] from='client.26777 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: cluster 2024-02-12T16:38:16.243665+0000 mon.a (mon.0) 2780 : cluster [DBG] osdmap e825: 12 total, 12 up, 12 in 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.970531+0000 mon.a (mon.0) 2781 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.971287+0000 mon.c (mon.1) 92 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.971476+0000 mon.a (mon.0) 2782 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.971612+0000 mon.c (mon.1) 93 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.971926+0000 mon.c (mon.1) 94 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.972236+0000 mon.c (mon.1) 95 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.972343+0000 mon.a (mon.0) 2783 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.972539+0000 mon.c (mon.1) 96 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.972735+0000 mon.c (mon.1) 97 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]: dispatch 2024-02-12T16:38:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.972804+0000 mon.a (mon.0) 2784 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.972898+0000 mon.c (mon.1) 98 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973058+0000 mon.c (mon.1) 99 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973154+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973226+0000 mon.c (mon.1) 100 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973405+0000 mon.c (mon.1) 101 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973440+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973576+0000 mon.c (mon.1) 102 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973704+0000 mon.a (mon.0) 2787 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973743+0000 mon.c (mon.1) 103 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973906+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.973958+0000 mon.a (mon.0) 2788 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.974215+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.974459+0000 mon.a (mon.0) 2790 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:17.550 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.974727+0000 mon.a (mon.0) 2791 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:17.551 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.974977+0000 mon.a (mon.0) 2792 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:17.551 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:17 smithi169 bash[19604]: audit 2024-02-12T16:38:16.975240+0000 mon.a (mon.0) 2793 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:17.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:17.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: cluster 2024-02-12T16:38:16.032846+0000 osd.5 (osd.5) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: cluster 2024-02-12T16:38:16.032895+0000 osd.5 (osd.5) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: cluster 2024-02-12T16:38:16.166920+0000 mgr.y (mgr.14146) 1616 : cluster [DBG] pgmap v2313: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 861 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 455/27568 objects misplaced (1.650%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: cluster 2024-02-12T16:38:16.239699+0000 mon.a (mon.0) 2778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.243622+0000 mon.a (mon.0) 2779 : audit [INF] from='client.26777 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: cluster 2024-02-12T16:38:16.243665+0000 mon.a (mon.0) 2780 : cluster [DBG] osdmap e825: 12 total, 12 up, 12 in 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.970531+0000 mon.a (mon.0) 2781 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]: dispatch 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.971287+0000 mon.c (mon.1) 92 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]: dispatch 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.971476+0000 mon.a (mon.0) 2782 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]: dispatch 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.971612+0000 mon.c (mon.1) 93 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]: dispatch 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.971926+0000 mon.c (mon.1) 94 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]: dispatch 2024-02-12T16:38:17.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.972236+0000 mon.c (mon.1) 95 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.972343+0000 mon.a (mon.0) 2783 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.972539+0000 mon.c (mon.1) 96 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.972735+0000 mon.c (mon.1) 97 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.972804+0000 mon.a (mon.0) 2784 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.972898+0000 mon.c (mon.1) 98 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973058+0000 mon.c (mon.1) 99 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973154+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973226+0000 mon.c (mon.1) 100 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973405+0000 mon.c (mon.1) 101 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973440+0000 mon.a (mon.0) 2786 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973576+0000 mon.c (mon.1) 102 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973704+0000 mon.a (mon.0) 2787 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973743+0000 mon.c (mon.1) 103 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973906+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:17.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.973958+0000 mon.a (mon.0) 2788 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:17.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.974215+0000 mon.a (mon.0) 2789 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:17.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.974459+0000 mon.a (mon.0) 2790 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:17.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.974727+0000 mon.a (mon.0) 2791 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:17.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.974977+0000 mon.a (mon.0) 2792 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:17.642 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:17 smithi123 bash[19119]: audit 2024-02-12T16:38:16.975240+0000 mon.a (mon.0) 2793 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:17.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:17.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:17.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:17.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:17.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:17.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:17.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:17.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:18.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:18.061 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:18.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:18.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:18.195 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:18.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:18.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:18.329 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:18.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:18.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:18.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:18.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:18.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:18.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:18.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:18.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:18.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:18.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:18.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:38:18.880 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:18.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:18.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:19.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:19.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.862268+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.862345+0000 mon.a (mon.0) 2795 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.862614+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.862791+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.862945+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.863114+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.863273+0000 mon.a (mon.0) 2800 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.863428+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.863587+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.863747+0000 mon.a (mon.0) 2803 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.863900+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:19.125 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.864064+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:19.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.864228+0000 mon.a (mon.0) 2806 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:19.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: cluster 2024-02-12T16:38:17.864368+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e826: 12 total, 12 up, 12 in 2024-02-12T16:38:19.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.870594+0000 mon.b (mon.2) 193 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.870785+0000 mon.b (mon.2) 194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.870936+0000 mon.b (mon.2) 195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.871095+0000 mon.b (mon.2) 196 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.126 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.871265+0000 mon.b (mon.2) 197 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.871448+0000 mon.b (mon.2) 198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.871627+0000 mon.b (mon.2) 199 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.871756+0000 mon.b (mon.2) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.871930+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.872529+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.873355+0000 mon.a (mon.0) 2809 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.873713+0000 mon.a (mon.0) 2810 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.874063+0000 mon.a (mon.0) 2811 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.874414+0000 mon.a (mon.0) 2812 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.875188+0000 mon.a (mon.0) 2813 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.900737+0000 mon.b (mon.2) 202 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.901555+0000 mon.a (mon.0) 2814 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.901883+0000 mon.a (mon.0) 2815 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.127 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.902117+0000 mon.a (mon.0) 2816 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.902544+0000 mon.a (mon.0) 2817 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.902828+0000 mon.a (mon.0) 2818 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.903070+0000 mon.a (mon.0) 2819 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.903303+0000 mon.a (mon.0) 2820 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.903473+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.903632+0000 mon.a (mon.0) 2822 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.910050+0000 mon.a (mon.0) 2823 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.911967+0000 mon.a (mon.0) 2824 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:19.128 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.921228+0000 mon.a (mon.0) 2825 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:19.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.924715+0000 mon.a (mon.0) 2826 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:19.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.926755+0000 mon.a (mon.0) 2827 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:19.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.927629+0000 mon.a (mon.0) 2828 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:19.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: audit 2024-02-12T16:38:17.932817+0000 mon.a (mon.0) 2829 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:19.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:18 smithi123 bash[19119]: cluster 2024-02-12T16:38:18.167280+0000 mgr.y (mgr.14146) 1617 : cluster [DBG] pgmap v2316: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 658 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 455/21070 objects misplaced (2.159%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:38:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.862268+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]': finished 2024-02-12T16:38:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.862345+0000 mon.a (mon.0) 2795 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]': finished 2024-02-12T16:38:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.862614+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]': finished 2024-02-12T16:38:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.862791+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]': finished 2024-02-12T16:38:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.862945+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]': finished 2024-02-12T16:38:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.863114+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]': finished 2024-02-12T16:38:19.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.863273+0000 mon.a (mon.0) 2800 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]': finished 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.863428+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.863587+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.863747+0000 mon.a (mon.0) 2803 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.863900+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.864064+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.864228+0000 mon.a (mon.0) 2806 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: cluster 2024-02-12T16:38:17.864368+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e826: 12 total, 12 up, 12 in 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.870594+0000 mon.b (mon.2) 193 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.870785+0000 mon.b (mon.2) 194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.870936+0000 mon.b (mon.2) 195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.871095+0000 mon.b (mon.2) 196 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.871265+0000 mon.b (mon.2) 197 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.871448+0000 mon.b (mon.2) 198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.871627+0000 mon.b (mon.2) 199 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:19.162 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.871756+0000 mon.b (mon.2) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.871930+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.872529+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.873355+0000 mon.a (mon.0) 2809 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.873713+0000 mon.a (mon.0) 2810 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.874063+0000 mon.a (mon.0) 2811 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.874414+0000 mon.a (mon.0) 2812 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.875188+0000 mon.a (mon.0) 2813 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.900737+0000 mon.b (mon.2) 202 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.901555+0000 mon.a (mon.0) 2814 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.901883+0000 mon.a (mon.0) 2815 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.902117+0000 mon.a (mon.0) 2816 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.902544+0000 mon.a (mon.0) 2817 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.902828+0000 mon.a (mon.0) 2818 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.903070+0000 mon.a (mon.0) 2819 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.903303+0000 mon.a (mon.0) 2820 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.903473+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.903632+0000 mon.a (mon.0) 2822 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.910050+0000 mon.a (mon.0) 2823 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.911967+0000 mon.a (mon.0) 2824 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.921228+0000 mon.a (mon.0) 2825 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.924715+0000 mon.a (mon.0) 2826 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.926755+0000 mon.a (mon.0) 2827 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.927629+0000 mon.a (mon.0) 2828 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: audit 2024-02-12T16:38:17.932817+0000 mon.a (mon.0) 2829 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:19.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:18 smithi121 bash[18489]: cluster 2024-02-12T16:38:18.167280+0000 mgr.y (mgr.14146) 1617 : cluster [DBG] pgmap v2316: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 658 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 455/21070 objects misplaced (2.159%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:38:19.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:19.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.862268+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07989320358507691]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.862345+0000 mon.a (mon.0) 2795 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09387555493544754]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.862614+0000 mon.a (mon.0) 2796 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08748240086468646]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.862791+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09680579239852216]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.862945+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09310443981358581]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.863114+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08502315333647482]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.863273+0000 mon.a (mon.0) 2800 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08459917760643616]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.863428+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.863587+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.863747+0000 mon.a (mon.0) 2803 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.863900+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.864064+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:19.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.864228+0000 mon.a (mon.0) 2806 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: cluster 2024-02-12T16:38:17.864368+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e826: 12 total, 12 up, 12 in 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.870594+0000 mon.b (mon.2) 193 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.870785+0000 mon.b (mon.2) 194 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.870936+0000 mon.b (mon.2) 195 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.871095+0000 mon.b (mon.2) 196 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.871265+0000 mon.b (mon.2) 197 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.871448+0000 mon.b (mon.2) 198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.871627+0000 mon.b (mon.2) 199 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.871756+0000 mon.b (mon.2) 200 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.871930+0000 mon.b (mon.2) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.872529+0000 mon.a (mon.0) 2808 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.873355+0000 mon.a (mon.0) 2809 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.873713+0000 mon.a (mon.0) 2810 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.874063+0000 mon.a (mon.0) 2811 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.874414+0000 mon.a (mon.0) 2812 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.875188+0000 mon.a (mon.0) 2813 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.900737+0000 mon.b (mon.2) 202 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.901555+0000 mon.a (mon.0) 2814 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.901883+0000 mon.a (mon.0) 2815 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.902117+0000 mon.a (mon.0) 2816 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.902544+0000 mon.a (mon.0) 2817 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.902828+0000 mon.a (mon.0) 2818 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.903070+0000 mon.a (mon.0) 2819 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.903303+0000 mon.a (mon.0) 2820 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.903473+0000 mon.a (mon.0) 2821 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.903632+0000 mon.a (mon.0) 2822 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.910050+0000 mon.a (mon.0) 2823 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.911967+0000 mon.a (mon.0) 2824 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:19.301 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.921228+0000 mon.a (mon.0) 2825 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:19.302 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.924715+0000 mon.a (mon.0) 2826 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:19.302 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.926755+0000 mon.a (mon.0) 2827 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:19.302 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.927629+0000 mon.a (mon.0) 2828 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:19.302 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: audit 2024-02-12T16:38:17.932817+0000 mon.a (mon.0) 2829 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:19.302 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:18 smithi169 bash[19604]: cluster 2024-02-12T16:38:18.167280+0000 mgr.y (mgr.14146) 1617 : cluster [DBG] pgmap v2316: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 658 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 455/21070 objects misplaced (2.159%); 619 KiB/s, 9 objects/s recovering 2024-02-12T16:38:19.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:19.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:19.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:19.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:38:19.630 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:19.658 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:19.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:19.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:19.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:19.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:19.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864250+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864325+0000 mon.a (mon.0) 2831 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864379+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864430+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864481+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864532+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864583+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864626+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864668+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864710+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864751+0000 mon.a (mon.0) 2840 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: audit 2024-02-12T16:38:18.864793+0000 mon.a (mon.0) 2841 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:19 smithi169 bash[19604]: cluster 2024-02-12T16:38:18.864827+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e827: 12 total, 12 up, 12 in 2024-02-12T16:38:20.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864250+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864325+0000 mon.a (mon.0) 2831 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864379+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864430+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864481+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864532+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864583+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864626+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864668+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864710+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:20.061 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864751+0000 mon.a (mon.0) 2840 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:20.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: audit 2024-02-12T16:38:18.864793+0000 mon.a (mon.0) 2841 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:20.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:19 smithi121 bash[18489]: cluster 2024-02-12T16:38:18.864827+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e827: 12 total, 12 up, 12 in 2024-02-12T16:38:20.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:20.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:20.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864250+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:20.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864325+0000 mon.a (mon.0) 2831 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864379+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864430+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864481+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864532+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14146 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864583+0000 mon.a (mon.0) 2836 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08358027641091073]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864626+0000 mon.a (mon.0) 2837 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08986322998984983]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864668+0000 mon.a (mon.0) 2838 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08476471369834393]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864710+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08321503121173703]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864751+0000 mon.a (mon.0) 2840 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.0853394128676787]}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: audit 2024-02-12T16:38:18.864793+0000 mon.a (mon.0) 2841 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"0\": \"43808\", \"2\": \"53484\", \"9\": \"59709\"}"}]': finished 2024-02-12T16:38:20.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:19 smithi123 bash[19119]: cluster 2024-02-12T16:38:18.864827+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e827: 12 total, 12 up, 12 in 2024-02-12T16:38:20.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:20.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:20.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:20.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:20.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:20.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:20.481 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:20.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:20.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:20.671 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:20.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:20.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:20.814 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:20.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:20.909 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:20 smithi121 bash[18489]: cluster 2024-02-12T16:38:20.167758+0000 mgr.y (mgr.14146) 1618 : cluster [DBG] pgmap v2318: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 614 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 204/19660 objects misplaced (1.038%); 766 KiB/s, 11 objects/s recovering 2024-02-12T16:38:20.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:20.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:20.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:21.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:20 smithi169 bash[19604]: cluster 2024-02-12T16:38:20.167758+0000 mgr.y (mgr.14146) 1618 : cluster [DBG] pgmap v2318: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 614 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 204/19660 objects misplaced (1.038%); 766 KiB/s, 11 objects/s recovering 2024-02-12T16:38:21.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:21.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:21.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:21.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:20 smithi123 bash[19119]: cluster 2024-02-12T16:38:20.167758+0000 mgr.y (mgr.14146) 1618 : cluster [DBG] pgmap v2318: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 614 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 802 op/s; 204/19660 objects misplaced (1.038%); 766 KiB/s, 11 objects/s recovering 2024-02-12T16:38:21.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:21.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:21.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:21.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:21.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:21.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:21.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:21.478 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:21.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:21.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:21.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:21.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:21.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:21.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:21.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:21.897 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:21.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:22.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:22.028 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:22.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:22.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:22.168 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:22.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:22.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:22.306 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:22.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:22.319 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:22.320 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-02-12T16:38:22.320 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:38:22.320 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-12T16:38:22.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:38:22.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:22.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:22.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:22.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:22.608 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:22.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:22.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:22.733 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:22.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:22.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:22.868 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:22.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:22.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:22.998 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:23.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:23.129 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:23.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:23.243 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:23 smithi123 bash[19119]: cluster 2024-02-12T16:38:22.168252+0000 mgr.y (mgr.14146) 1619 : cluster [DBG] pgmap v2319: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 365 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s; 204/11666 objects misplaced (1.749%); 629 KiB/s, 9 objects/s recovering 2024-02-12T16:38:23.263 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:23.331 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:23.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:23.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:23.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:23.509 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:23 smithi169 bash[19604]: cluster 2024-02-12T16:38:22.168252+0000 mgr.y (mgr.14146) 1619 : cluster [DBG] pgmap v2319: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 365 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s; 204/11666 objects misplaced (1.749%); 629 KiB/s, 9 objects/s recovering 2024-02-12T16:38:23.530 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:23.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:23.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:23 smithi121 bash[18489]: cluster 2024-02-12T16:38:22.168252+0000 mgr.y (mgr.14146) 1619 : cluster [DBG] pgmap v2319: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 365 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s; 204/11666 objects misplaced (1.749%); 629 KiB/s, 9 objects/s recovering 2024-02-12T16:38:23.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:23.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:23.801 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:23.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:23.930 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:23.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:24.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:24.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:24.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:24.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:24.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:24.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:24.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:24.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:24.352 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:24.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:24.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:24.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:24.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:24.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:24.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:24.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:24.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:24.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:24.762 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:24.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:24.854 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 168591 objects 2024-02-12T16:38:24.854 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :150.648856 2024-02-12T16:38:24.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:24.895 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:24.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:24.999 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:25.036 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:25.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:25.177 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:25.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:25.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:25 smithi123 bash[19119]: cluster 2024-02-12T16:38:24.168759+0000 mgr.y (mgr.14146) 1620 : cluster [DBG] pgmap v2320: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 244 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 5/7806 objects misplaced (0.064%); 945 KiB/s, 14 objects/s recovering 2024-02-12T16:38:25.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:25.421 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:25 smithi169 bash[19604]: cluster 2024-02-12T16:38:24.168759+0000 mgr.y (mgr.14146) 1620 : cluster [DBG] pgmap v2320: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 244 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 5/7806 objects misplaced (0.064%); 945 KiB/s, 14 objects/s recovering 2024-02-12T16:38:25.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:25.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:25.551 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:25 smithi121 bash[18489]: cluster 2024-02-12T16:38:24.168759+0000 mgr.y (mgr.14146) 1620 : cluster [DBG] pgmap v2320: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 244 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 5/7806 objects misplaced (0.064%); 945 KiB/s, 14 objects/s recovering 2024-02-12T16:38:25.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:25.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:25.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:25.820 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:25.858 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:25.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:25.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:25.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:26.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:26.126 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:26.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:26.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:26.262 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:26.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:26.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:26.376 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:26 smithi169 bash[19604]: cluster 2024-02-12T16:38:25.241810+0000 mon.a (mon.0) 2843 : cluster [DBG] osdmap e828: 12 total, 12 up, 12 in 2024-02-12T16:38:26.402 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:26.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:26.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:26.545 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:26.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:26 smithi123 bash[19119]: cluster 2024-02-12T16:38:25.241810+0000 mon.a (mon.0) 2843 : cluster [DBG] osdmap e828: 12 total, 12 up, 12 in 2024-02-12T16:38:26.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:26.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:26 smithi121 bash[18489]: cluster 2024-02-12T16:38:25.241810+0000 mon.a (mon.0) 2843 : cluster [DBG] osdmap e828: 12 total, 12 up, 12 in 2024-02-12T16:38:26.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:26.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:26.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:26.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:26.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:26.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:26.962 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:26.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:27.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:27.104 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:27.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:27.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:27.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:27.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:27.282 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-02-12T16:38:27.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-02-12T16:38:27.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:27.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:27 smithi169 bash[19604]: cluster 2024-02-12T16:38:26.169439+0000 mgr.y (mgr.14146) 1621 : cluster [DBG] pgmap v2322: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 137 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 5/4378 objects misplaced (0.114%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:38:27.359 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:27 smithi169 bash[19604]: cluster 2024-02-12T16:38:26.254620+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e829: 12 total, 12 up, 12 in 2024-02-12T16:38:27.386 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:27.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:27.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:27.530 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:27.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:27.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:27 smithi123 bash[19119]: cluster 2024-02-12T16:38:26.169439+0000 mgr.y (mgr.14146) 1621 : cluster [DBG] pgmap v2322: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 137 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 5/4378 objects misplaced (0.114%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:38:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:27 smithi123 bash[19119]: cluster 2024-02-12T16:38:26.254620+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e829: 12 total, 12 up, 12 in 2024-02-12T16:38:27.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:27 smithi121 bash[18489]: cluster 2024-02-12T16:38:26.169439+0000 mgr.y (mgr.14146) 1621 : cluster [DBG] pgmap v2322: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 137 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 5/4378 objects misplaced (0.114%); 936 KiB/s, 14 objects/s recovering 2024-02-12T16:38:27.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:27 smithi121 bash[18489]: cluster 2024-02-12T16:38:26.254620+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e829: 12 total, 12 up, 12 in 2024-02-12T16:38:27.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:27.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:27.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:27.813 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:27.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:27.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:27.950 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:27.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:28.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:28.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:28.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:28.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:28.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:28.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:28.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:28.369 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:28.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:28.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:28.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:28.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:28.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:28.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:28.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:28.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:28.794 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:28.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:28.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:28.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:28.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:29.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:29.077 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:29.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:29.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:29.217 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:29.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:29.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:29.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:29 smithi121 bash[18489]: cluster 2024-02-12T16:38:28.170267+0000 mgr.y (mgr.14146) 1622 : cluster [DBG] pgmap v2324: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 6.1 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 5/194 objects misplaced (2.577%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:29.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:29.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:29.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:29.474 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:29 smithi169 bash[19604]: cluster 2024-02-12T16:38:28.170267+0000 mgr.y (mgr.14146) 1622 : cluster [DBG] pgmap v2324: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 6.1 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 5/194 objects misplaced (2.577%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:29.500 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:29.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:29.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:29 smithi123 bash[19119]: cluster 2024-02-12T16:38:28.170267+0000 mgr.y (mgr.14146) 1622 : cluster [DBG] pgmap v2324: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 6.1 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s; 5/194 objects misplaced (2.577%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:38:29.642 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:29.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:29.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:29.783 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:29.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:29.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:29.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:29.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:30.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:30.067 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:30.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:30.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:30.209 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:30.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:30.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:30.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:30.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:30.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:30.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:30.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:30.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:30.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:30.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:30.777 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:30.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:30.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:30.937 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:30.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:31.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:31.077 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:31.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:31.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:31.224 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:31.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:31.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:31.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:31.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:31.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:31.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:31.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:31 smithi169 bash[19604]: cluster 2024-02-12T16:38:30.171187+0000 mgr.y (mgr.14146) 1623 : cluster [DBG] pgmap v2325: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 496 KiB/s, 7 objects/s recovering 2024-02-12T16:38:31.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:31 smithi123 bash[19119]: cluster 2024-02-12T16:38:30.171187+0000 mgr.y (mgr.14146) 1623 : cluster [DBG] pgmap v2325: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 496 KiB/s, 7 objects/s recovering 2024-02-12T16:38:31.648 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:31.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:31 smithi121 bash[18489]: cluster 2024-02-12T16:38:30.171187+0000 mgr.y (mgr.14146) 1623 : cluster [DBG] pgmap v2325: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 496 KiB/s, 7 objects/s recovering 2024-02-12T16:38:31.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:31.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:31.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:31.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:31.926 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:31.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:32.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:32.067 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:32.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:32.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:32.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:32.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:32.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:32.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:32.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:32.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:32.486 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:32.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:32.497 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:38:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:38:32.534 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:38:32.534 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:38:32.553 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:38:32.580 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:38:32.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:38:32.628 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:32.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:32.641 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:38:32.665 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:38:32.691 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:38:32.718 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:38:32.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:32.756 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:38:32.768 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:32.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:32.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:38:32.803 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:38:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:38:32.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:32.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:32.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:33.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:33.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:33.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:33.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:33.192 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:33.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:33.294 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:38:33.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:33.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:33.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:33.427 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:33.438 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:33 smithi169 bash[19604]: cluster 2024-02-12T16:38:32.171994+0000 mgr.y (mgr.14146) 1624 : cluster [DBG] pgmap v2326: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T16:38:33.439 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:33 smithi169 bash[19604]: audit 2024-02-12T16:38:32.467230+0000 mon.a (mon.0) 2845 : audit [DBG] from='client.? v1:172.21.15.121:0/3127153681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:33.439 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:33 smithi169 bash[19604]: audit 2024-02-12T16:38:33.121903+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.121:0/3230500737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:38:33.465 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:33.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:33.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:33.609 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:33.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:33 smithi123 bash[19119]: cluster 2024-02-12T16:38:32.171994+0000 mgr.y (mgr.14146) 1624 : cluster [DBG] pgmap v2326: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T16:38:33.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:33 smithi123 bash[19119]: audit 2024-02-12T16:38:32.467230+0000 mon.a (mon.0) 2845 : audit [DBG] from='client.? v1:172.21.15.121:0/3127153681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:33.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:33 smithi123 bash[19119]: audit 2024-02-12T16:38:33.121903+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.121:0/3230500737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:38:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:33 smithi121 bash[18489]: cluster 2024-02-12T16:38:32.171994+0000 mgr.y (mgr.14146) 1624 : cluster [DBG] pgmap v2326: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T16:38:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:33 smithi121 bash[18489]: audit 2024-02-12T16:38:32.467230+0000 mon.a (mon.0) 2845 : audit [DBG] from='client.? v1:172.21.15.121:0/3127153681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:33 smithi121 bash[18489]: audit 2024-02-12T16:38:33.121903+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.121:0/3230500737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:38:33.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:33.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:33.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:33.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:33.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:33.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:33.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:34.034 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:34.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:34.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:34.178 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:34.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:34.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:34.316 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:34.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:34.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:38:34.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:34.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:34.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:34 smithi169 bash[19604]: cluster 2024-02-12T16:38:33.266651+0000 mon.a (mon.0) 2847 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:34 smithi169 bash[19604]: audit 2024-02-12T16:38:33.283852+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? v1:172.21.15.121:0/3230500737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:38:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:34 smithi169 bash[19604]: cluster 2024-02-12T16:38:33.283954+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e830: 12 total, 12 up, 12 in 2024-02-12T16:38:34.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:34.596 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:34.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:34 smithi123 bash[19119]: cluster 2024-02-12T16:38:33.266651+0000 mon.a (mon.0) 2847 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:34 smithi123 bash[19119]: audit 2024-02-12T16:38:33.283852+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? v1:172.21.15.121:0/3230500737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:38:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:34 smithi123 bash[19119]: cluster 2024-02-12T16:38:33.283954+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e830: 12 total, 12 up, 12 in 2024-02-12T16:38:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:34 smithi121 bash[18489]: cluster 2024-02-12T16:38:33.266651+0000 mon.a (mon.0) 2847 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:34 smithi121 bash[18489]: audit 2024-02-12T16:38:33.283852+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? v1:172.21.15.121:0/3230500737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:38:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:34 smithi121 bash[18489]: cluster 2024-02-12T16:38:33.283954+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e830: 12 total, 12 up, 12 in 2024-02-12T16:38:34.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:34.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:34.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:34.878 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:34.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:34.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:35.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:35.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:35.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:35.161 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:35.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:35.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:35.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:35.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:35.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:35.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:35 smithi121 bash[18489]: cluster 2024-02-12T16:38:34.172752+0000 mgr.y (mgr.14146) 1625 : cluster [DBG] pgmap v2328: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 273 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T16:38:35.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:35.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:35 smithi169 bash[19604]: cluster 2024-02-12T16:38:34.172752+0000 mgr.y (mgr.14146) 1625 : cluster [DBG] pgmap v2328: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 273 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T16:38:35.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:35.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:35.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:35 smithi123 bash[19119]: cluster 2024-02-12T16:38:34.172752+0000 mgr.y (mgr.14146) 1625 : cluster [DBG] pgmap v2328: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 273 op/s; 32 KiB/s, 0 objects/s recovering 2024-02-12T16:38:35.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:35.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:35.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 830, 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-02-12T15:47:18.273813+0000", 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-02-12T16:38:33.266391+0000", 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "last_up_change": "2024-02-12T16:23:01.123593+0000", 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "last_in_change": "2024-02-12T16:19:27.892497+0000", 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_num": 5806080, 2024-02-12T16:38:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_set": [ 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "nodeep-scrub", 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "noscrub", 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "pglog_hardlimit", 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "purged_snapdirs", 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_deletes", 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "sortbitwise" 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_version": 38, 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:38:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:38:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:38:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_snapshot": "", 2024-02-12T16:38:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_max": 7, 2024-02-12T16:38:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd": 12, 2024-02-12T16:38:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "min_compat_client": "jewel", 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "require_osd_release": "pacific", 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "pools": [ 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 1, 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T15:51:35.886878+0000", 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:38:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 31, 2024-02-12T16:38:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 31, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 31, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 31, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 31, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "701", 2024-02-12T16:38:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "663", 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:38:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:38:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:38:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "options": { 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_max": 32, 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_min": 1 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:38:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_devicehealth": {} 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 7, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "unique_pool_5", 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T16:34:06.973798+0000", 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:38:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 16, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 16, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 16, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 16, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 16, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:38:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "759", 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "0", 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:38:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:38:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:38:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "options": {}, 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "rados": {} 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:38:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0", 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.66845703125, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 8, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 808, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6802", 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6804", 2024-02-12T16:38:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6803", 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6801/1815232143", 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6802/1815232143", 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6804/1815232143", 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6803/1815232143", 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "c6140309-5703-4538-9739-276fbbdaa880", 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 0.9075927734375, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 13, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 752, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6805", 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6806", 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6808", 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6807", 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6805/2968433670", 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6806/2968433670", 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6808/2968433670", 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6807/2968433670", 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170", 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.81610107421875, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 21, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 780, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6809", 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6810", 2024-02-12T16:38:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6812", 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6811", 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6809/2158187852", 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6810/2158187852", 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6812/2158187852", 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6811/2158187852", 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd", 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 26, 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 414, 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 522, 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 809, 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 415, 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6813", 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6814", 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6816", 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6815", 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:38:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6813/2551349828", 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6814/2551349828", 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6816/2551349828", 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6815/2551349828", 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:38:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31", 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 32, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 828, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6800", 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6801", 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6803", 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6802", 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6800/3447700123", 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6801/3447700123", 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6803/3447700123", 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6802/3447700123", 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602", 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 39, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 780, 2024-02-12T16:38:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6804", 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6805", 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6807", 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6806", 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6804/961317306", 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6805/961317306", 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6807/961317306", 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6806/961317306", 2024-02-12T16:38:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0", 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 48, 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 752, 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6808", 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6809", 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6811", 2024-02-12T16:38:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6810", 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6808/1377261784", 2024-02-12T16:38:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6809/1377261784", 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6811/1377261784", 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6810/1377261784", 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515", 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 56, 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 824, 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6812", 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6813", 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6815", 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6814", 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1702613957 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6812/1702613957", 2024-02-12T16:38:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6813/1702613957", 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6815/1702613957", 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6814/1702613957", 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75", 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 65, 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 796, 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6800", 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6801", 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6803", 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6802", 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6800/2551075225", 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6801/2551075225", 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6803/2551075225", 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6802/2551075225", 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15", 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.9110870361328125, 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 74, 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 752, 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6804", 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6805", 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6807", 2024-02-12T16:38:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6806", 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6804/2339555071", 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6805/2339555071", 2024-02-12T16:38:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6807/2339555071", 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6806/2339555071", 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "91f74dca-503a-45d1-9647-df4804d58515", 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 83, 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 796, 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6808", 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6809", 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6811", 2024-02-12T16:38:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6810", 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6808/391306610", 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6809/391306610", 2024-02-12T16:38:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6811/391306610", 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6810/391306610", 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf", 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:38:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 93, 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 802, 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6812", 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6813", 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6815", 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6814", 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6812/3696812305", 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6813/3696812305", 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3696812305", 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6814/3696812305", 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_xinfo": [ 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:38:11.128181+0000", 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:18.713072+0000", 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:48.002612+0000", 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:38:05.626646+0000", 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 414 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:31.743154+0000", 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:38:16.032900+0000", 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:43.023664+0000", 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:28.599958+0000", 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:33.786215+0000", 2024-02-12T16:38:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:54.198061+0000", 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:06.490613+0000", 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:38:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:37:24.591779+0000", 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap": [], 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap_items": [], 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_temp": [], 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_temp": [], 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklist": { 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/2474849789": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/253689008": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/3531151725": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/1778611055": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/28718860": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:38:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/445290372": "2024-02-13T15:47:47.232761+0000" 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "range_blocklist": {}, 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profiles": { 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "default": { 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-failure-domain": "osd", 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "k": "2", 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "m": "1", 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin": "jerasure", 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "technique": "reed_sol_van" 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps_queue": [], 2024-02-12T16:38:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "new_removed_snaps": [], 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "new_purged_snaps": [], 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_node_flags": {}, 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "device_class_flags": {}, 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": { 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:38:35.818 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:38:35.830 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:35.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:35.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:35.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:35.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:36.021 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:36.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:36.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:36.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:36.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:36.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:36.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:36.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:36.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:36 smithi121 bash[18489]: cluster 2024-02-12T16:38:35.646002+0000 osd.10 (osd.10) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:36 smithi121 bash[18489]: cluster 2024-02-12T16:38:35.646099+0000 osd.10 (osd.10) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:36 smithi121 bash[18489]: audit 2024-02-12T16:38:35.732300+0000 mon.a (mon.0) 2850 : audit [DBG] from='client.? v1:172.21.15.121:0/4149608104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:38:36.448 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:36.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:36 smithi169 bash[19604]: cluster 2024-02-12T16:38:35.646002+0000 osd.10 (osd.10) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:36 smithi169 bash[19604]: cluster 2024-02-12T16:38:35.646099+0000 osd.10 (osd.10) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:36 smithi169 bash[19604]: audit 2024-02-12T16:38:35.732300+0000 mon.a (mon.0) 2850 : audit [DBG] from='client.? v1:172.21.15.121:0/4149608104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:38:36.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:36.582 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:36.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:36 smithi123 bash[19119]: cluster 2024-02-12T16:38:35.646002+0000 osd.10 (osd.10) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:36 smithi123 bash[19119]: cluster 2024-02-12T16:38:35.646099+0000 osd.10 (osd.10) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:36.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:36 smithi123 bash[19119]: audit 2024-02-12T16:38:35.732300+0000 mon.a (mon.0) 2850 : audit [DBG] from='client.? v1:172.21.15.121:0/4149608104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:38:36.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:36.722 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:36.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:36.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:36.865 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:36.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:36.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:37.005 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:37.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:37.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:38:37.145 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:37.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:37.270 INFO:tasks.thrashosds.thrasher:j is {'epoch': 830, 'fsid': 'eb2349b2-c9bd-11ee-95b9-87774f69a715', 'created': '2024-02-12T15:47:18.273813+0000', 'modified': '2024-02-12T16:38:33.266391+0000', 'last_up_change': '2024-02-12T16:23:01.123593+0000', 'last_in_change': '2024-02-12T16:19:27.892497+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 38, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '701', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '663', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-02-12T16:34:06.973798+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '759', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'up': 1, 'in': 1, 'weight': 0.66845703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 808, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 1815232143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 1815232143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 1815232143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 1815232143}]}, 'public_addr': '172.21.15.121:6801/1815232143', 'cluster_addr': '172.21.15.121:6802/1815232143', 'heartbeat_back_addr': '172.21.15.121:6804/1815232143', 'heartbeat_front_addr': '172.21.15.121:6803/1815232143', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c6140309-5703-4538-9739-276fbbdaa880', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9075927734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 752, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2968433670}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2968433670}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 2968433670}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2968433670}]}, 'public_addr': '172.21.15.121:6805/2968433670', 'cluster_addr': '172.21.15.121:6806/2968433670', 'heartbeat_back_addr': '172.21.15.121:6808/2968433670', 'heartbeat_front_addr': '172.21.15.121:6807/2968433670', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'up': 1, 'in': 1, 'weight': 0.81610107421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 780, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 2158187852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 2158187852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 2158187852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 2158187852}]}, 'public_addr': '172.21.15.121:6809/2158187852', 'cluster_addr': '172.21.15.121:6810/2158187852', 'heartbeat_back_addr': '172.21.15.121:6812/2158187852', 'heartbeat_front_addr': '172.21.15.121:6811/2158187852', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5f824526-596d-42e1-8b7e-cd8676ded6dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 26, 'last_clean_end': 414, 'up_from': 522, 'up_thru': 809, 'down_at': 415, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 2551349828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 2551349828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6816', 'nonce': 2551349828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 2551349828}]}, 'public_addr': '172.21.15.121:6813/2551349828', 'cluster_addr': '172.21.15.121:6814/2551349828', 'heartbeat_back_addr': '172.21.15.121:6816/2551349828', 'heartbeat_front_addr': '172.21.15.121:6815/2551349828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '14c2f878-99ce-4b51-8d9c-131c1d883b31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 828, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6800', 'nonce': 3447700123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6801', 'nonce': 3447700123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6803', 'nonce': 3447700123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6802', 'nonce': 3447700123}]}, 'public_addr': '172.21.15.123:6800/3447700123', 'cluster_addr': '172.21.15.123:6801/3447700123', 'heartbeat_back_addr': '172.21.15.123:6803/3447700123', 'heartbeat_front_addr': '172.21.15.123:6802/3447700123', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 780, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6804', 'nonce': 961317306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6805', 'nonce': 961317306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6807', 'nonce': 961317306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6806', 'nonce': 961317306}]}, 'public_addr': '172.21.15.123:6804/961317306', 'cluster_addr': '172.21.15.123:6805/961317306', 'heartbeat_back_addr': '172.21.15.123:6807/961317306', 'heartbeat_front_addr': '172.21.15.123:6806/961317306', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '71e715e6-eac1-4386-927e-51df0cc419c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 752, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6808', 'nonce': 1377261784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6809', 'nonce': 1377261784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6811', 'nonce': 1377261784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6810', 'nonce': 1377261784}]}, 'public_addr': '172.21.15.123:6808/1377261784', 'cluster_addr': '172.21.15.123:6809/1377261784', 'heartbeat_back_addr': '172.21.15.123:6811/1377261784', 'heartbeat_front_addr': '172.21.15.123:6810/1377261784', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b419b04f-0c0c-495a-aa9b-25c334159515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 824, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6812', 'nonce': 1702613957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6813', 'nonce': 1702613957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6815', 'nonce': 1702613957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6814', 'nonce': 1702613957}]}, 'public_addr': '172.21.15.123:6812/1702613957', 'cluster_addr': '172.21.15.123:6813/1702613957', 'heartbeat_back_addr': '172.21.15.123:6815/1702613957', 'heartbeat_front_addr': '172.21.15.123:6814/1702613957', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1591e7bc-5b86-4401-803d-5325062ffb75', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 796, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6800', 'nonce': 2551075225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 2551075225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 2551075225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6802', 'nonce': 2551075225}]}, 'public_addr': '172.21.15.169:6800/2551075225', 'cluster_addr': '172.21.15.169:6801/2551075225', 'heartbeat_back_addr': '172.21.15.169:6803/2551075225', 'heartbeat_front_addr': '172.21.15.169:6802/2551075225', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '522d61df-8dc1-4915-83fd-f94d0d572c15', 'up': 1, 'in': 1, 'weight': 0.9110870361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 752, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6804', 'nonce': 2339555071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 2339555071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 2339555071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6806', 'nonce': 2339555071}]}, 'public_addr': '172.21.15.169:6804/2339555071', 'cluster_addr': '172.21.15.169:6805/2339555071', 'heartbeat_back_addr': '172.21.15.169:6807/2339555071', 'heartbeat_front_addr': '172.21.15.169:6806/2339555071', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '91f74dca-503a-45d1-9647-df4804d58515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 796, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6808', 'nonce': 391306610}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 391306610}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 391306610}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6810', 'nonce': 391306610}]}, 'public_addr': '172.21.15.169:6808/391306610', 'cluster_addr': '172.21.15.169:6809/391306610', 'heartbeat_back_addr': '172.21.15.169:6811/391306610', 'heartbeat_front_addr': '172.21.15.169:6810/391306610', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '788b4f47-eadf-4601-a5b4-51fb6f04fbdf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 93, 'up_thru': 802, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6812', 'nonce': 3696812305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3696812305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3696812305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6814', 'nonce': 3696812305}]}, 'public_addr': '172.21.15.169:6812/3696812305', 'cluster_addr': '172.21.15.169:6813/3696812305', 'heartbeat_back_addr': '172.21.15.169:6815/3696812305', 'heartbeat_front_addr': '172.21.15.169:6814/3696812305', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:38:11.128181+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:18.713072+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:48.002612+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:38:05.626646+0000', 'dead_epoch': 414}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:31.743154+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:38:16.032900+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:43.023664+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:28.599958+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:33.786215+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:54.198061+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:06.490613+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:37:24.591779+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.121:0/2474849789': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/253689008': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:6800/3531151725': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:0/1778611055': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:6800/28718860': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/445290372': '2024-02-13T15:47:47.232761+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:38:37.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:38:37.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:37.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:38:37.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:37 smithi123 bash[19119]: cluster 2024-02-12T16:38:36.173539+0000 mgr.y (mgr.14146) 1626 : cluster [DBG] pgmap v2329: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-12T16:38:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:37 smithi123 bash[19119]: cluster 2024-02-12T16:38:36.295724+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e831: 12 total, 12 up, 12 in 2024-02-12T16:38:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:37 smithi121 bash[18489]: cluster 2024-02-12T16:38:36.173539+0000 mgr.y (mgr.14146) 1626 : cluster [DBG] pgmap v2329: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-12T16:38:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:37 smithi121 bash[18489]: cluster 2024-02-12T16:38:36.295724+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e831: 12 total, 12 up, 12 in 2024-02-12T16:38:37.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:37.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:37.555 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:37 smithi169 bash[19604]: cluster 2024-02-12T16:38:36.173539+0000 mgr.y (mgr.14146) 1626 : cluster [DBG] pgmap v2329: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-12T16:38:37.555 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:37 smithi169 bash[19604]: cluster 2024-02-12T16:38:36.295724+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e831: 12 total, 12 up, 12 in 2024-02-12T16:38:37.580 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:37.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:37.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:37.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:37.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:37.959 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:37.994 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:37.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:38.099 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:38.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:38.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:38.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:38.279 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:38.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:38.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:38 smithi123 bash[19119]: cluster 2024-02-12T16:38:37.455635+0000 osd.1 (osd.1) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:38 smithi123 bash[19119]: cluster 2024-02-12T16:38:37.455715+0000 osd.1 (osd.1) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:38.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:38.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:38.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:38.562 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:38.564 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:38 smithi121 bash[18489]: cluster 2024-02-12T16:38:37.455635+0000 osd.1 (osd.1) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:38.564 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:38 smithi121 bash[18489]: cluster 2024-02-12T16:38:37.455715+0000 osd.1 (osd.1) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:38.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:38.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:38.701 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:38.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:38 smithi169 bash[19604]: cluster 2024-02-12T16:38:37.455635+0000 osd.1 (osd.1) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:38 smithi169 bash[19604]: cluster 2024-02-12T16:38:37.455715+0000 osd.1 (osd.1) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:38.807 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:38.845 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:38.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:38.949 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:38.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:38.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:39.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:39.122 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:39.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:39.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:39.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:39.261 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:39.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:39.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:39.404 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:39.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:39.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:39 smithi121 bash[18489]: cluster 2024-02-12T16:38:38.174381+0000 mgr.y (mgr.14146) 1627 : cluster [DBG] pgmap v2331: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:39.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:39 smithi121 bash[18489]: cluster 2024-02-12T16:38:38.307513+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e832: 12 total, 12 up, 12 in 2024-02-12T16:38:39.508 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:39.550 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:39.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:39 smithi123 bash[19119]: cluster 2024-02-12T16:38:38.174381+0000 mgr.y (mgr.14146) 1627 : cluster [DBG] pgmap v2331: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:39 smithi123 bash[19119]: cluster 2024-02-12T16:38:38.307513+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e832: 12 total, 12 up, 12 in 2024-02-12T16:38:39.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:39.690 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:39.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:39.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:39 smithi169 bash[19604]: cluster 2024-02-12T16:38:38.174381+0000 mgr.y (mgr.14146) 1627 : cluster [DBG] pgmap v2331: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:39 smithi169 bash[19604]: cluster 2024-02-12T16:38:38.307513+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e832: 12 total, 12 up, 12 in 2024-02-12T16:38:39.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:39.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:39.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:39.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:39.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:40.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:40.111 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:40.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:40.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:40.249 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:40.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:40.353 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_5' removed 2024-02-12T16:38:40.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:40.366 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:40 smithi123 bash[19119]: audit 2024-02-12T16:38:40.275414+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? v1:172.21.15.121:0/2174106115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:38:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:40.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:40.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:40.535 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:40.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:40.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:40.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:40 smithi169 bash[19604]: audit 2024-02-12T16:38:40.275414+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? v1:172.21.15.121:0/2174106115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:38:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:40 smithi121 bash[18489]: audit 2024-02-12T16:38:40.275414+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? v1:172.21.15.121:0/2174106115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:38:40.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:40.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:40.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:40.817 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:40.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:40.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:40.949 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:40.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:41.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:41.090 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:41.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:41.149 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:38:41.150 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:38:41.150 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-02-12T16:38:41.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_6 16 2024-02-12T16:38:41.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:41.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:41.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:41.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:41.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:41.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:41.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:41 smithi121 bash[18489]: cluster 2024-02-12T16:38:40.175098+0000 mgr.y (mgr.14146) 1628 : cluster [DBG] pgmap v2333: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:41 smithi121 bash[18489]: audit 2024-02-12T16:38:40.320011+0000 mon.a (mon.0) 2854 : audit [INF] from='client.? v1:172.21.15.121:0/2174106115' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:38:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:41 smithi121 bash[18489]: cluster 2024-02-12T16:38:40.320110+0000 mon.a (mon.0) 2855 : cluster [DBG] osdmap e833: 12 total, 12 up, 12 in 2024-02-12T16:38:41.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:41.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:41 smithi169 bash[19604]: cluster 2024-02-12T16:38:40.175098+0000 mgr.y (mgr.14146) 1628 : cluster [DBG] pgmap v2333: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:41.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:41 smithi169 bash[19604]: audit 2024-02-12T16:38:40.320011+0000 mon.a (mon.0) 2854 : audit [INF] from='client.? v1:172.21.15.121:0/2174106115' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:38:41.483 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:41 smithi169 bash[19604]: cluster 2024-02-12T16:38:40.320110+0000 mon.a (mon.0) 2855 : cluster [DBG] osdmap e833: 12 total, 12 up, 12 in 2024-02-12T16:38:41.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:41.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:41.566 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:41.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:41.621 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:41 smithi123 bash[19119]: cluster 2024-02-12T16:38:40.175098+0000 mgr.y (mgr.14146) 1628 : cluster [DBG] pgmap v2333: 47 pgs: 47 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:41.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:41 smithi123 bash[19119]: audit 2024-02-12T16:38:40.320011+0000 mon.a (mon.0) 2854 : audit [INF] from='client.? v1:172.21.15.121:0/2174106115' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:38:41.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:41 smithi123 bash[19119]: cluster 2024-02-12T16:38:40.320110+0000 mon.a (mon.0) 2855 : cluster [DBG] osdmap e833: 12 total, 12 up, 12 in 2024-02-12T16:38:41.642 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:41.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:41.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:41.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:41.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:41.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:41.820 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:41.820 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:41.821 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:41.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:41.924 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:41.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:42.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:42.058 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:42.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:42.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:42.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:42.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:42.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:42.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:42.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:42.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:42.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:42.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:42.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:42.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:42.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:42.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:42.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:42.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:42.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:42.902 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:42.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:43.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:43.043 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:43.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:43.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:43.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:43.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:43.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:43.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:43.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:43.333 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:43 smithi123 bash[19119]: cluster 2024-02-12T16:38:42.175770+0000 mgr.y (mgr.14146) 1629 : cluster [DBG] pgmap v2335: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:43.333 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:43 smithi123 bash[19119]: cluster 2024-02-12T16:38:42.283547+0000 osd.11 (osd.11) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:43.333 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:43 smithi123 bash[19119]: cluster 2024-02-12T16:38:42.283649+0000 osd.11 (osd.11) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:43.333 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:43 smithi123 bash[19119]: cluster 2024-02-12T16:38:42.327786+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e834: 12 total, 12 up, 12 in 2024-02-12T16:38:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:43 smithi121 bash[18489]: cluster 2024-02-12T16:38:42.175770+0000 mgr.y (mgr.14146) 1629 : cluster [DBG] pgmap v2335: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:43 smithi121 bash[18489]: cluster 2024-02-12T16:38:42.283547+0000 osd.11 (osd.11) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:43 smithi121 bash[18489]: cluster 2024-02-12T16:38:42.283649+0000 osd.11 (osd.11) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:43 smithi121 bash[18489]: cluster 2024-02-12T16:38:42.327786+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e834: 12 total, 12 up, 12 in 2024-02-12T16:38:43.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:43.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:43.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:43.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:43.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:43.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:43.722 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:43 smithi169 bash[19604]: cluster 2024-02-12T16:38:42.175770+0000 mgr.y (mgr.14146) 1629 : cluster [DBG] pgmap v2335: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:43.723 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:43 smithi169 bash[19604]: cluster 2024-02-12T16:38:42.283547+0000 osd.11 (osd.11) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:43.723 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:43 smithi169 bash[19604]: cluster 2024-02-12T16:38:42.283649+0000 osd.11 (osd.11) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:43.723 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:43 smithi169 bash[19604]: cluster 2024-02-12T16:38:42.327786+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e834: 12 total, 12 up, 12 in 2024-02-12T16:38:43.749 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:43.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:43.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:43.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:43.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:44.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:44.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:44.131 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:44.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:44.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:44.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:44.313 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:44.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:44.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:44.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:44.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:44.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:44.588 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:44.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:44.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:44.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:44.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:44.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:44.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:44.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:45.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:45.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:45.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:45.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:45.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:45.304 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:45.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:45.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:45.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:45 smithi121 bash[18489]: cluster 2024-02-12T16:38:44.176529+0000 mgr.y (mgr.14146) 1630 : cluster [DBG] pgmap v2337: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:45.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:45.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:45 smithi169 bash[19604]: cluster 2024-02-12T16:38:44.176529+0000 mgr.y (mgr.14146) 1630 : cluster [DBG] pgmap v2337: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:45.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:45.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:45.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:45 smithi123 bash[19119]: cluster 2024-02-12T16:38:44.176529+0000 mgr.y (mgr.14146) 1630 : cluster [DBG] pgmap v2337: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:45.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:45.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:45.727 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:45.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:45.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:45.871 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:45.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:45.975 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:46.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:46.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:46.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:46.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:46.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:46.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:46.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:46.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:46.442 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:46.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:46.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:46.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:46.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:46.689 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:46.726 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:46.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:46.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:46.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:46.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:46.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:47.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:47.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:47.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:46 smithi121 bash[18489]: cluster 2024-02-12T16:38:46.177275+0000 mgr.y (mgr.14146) 1631 : cluster [DBG] pgmap v2338: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:47.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:46 smithi121 bash[18489]: cluster 2024-02-12T16:38:46.189185+0000 osd.7 (osd.7) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:47.126 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:46 smithi121 bash[18489]: cluster 2024-02-12T16:38:46.189279+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:47.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:47.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:47.289 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:46 smithi169 bash[19604]: cluster 2024-02-12T16:38:46.177275+0000 mgr.y (mgr.14146) 1631 : cluster [DBG] pgmap v2338: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:46 smithi169 bash[19604]: cluster 2024-02-12T16:38:46.189185+0000 osd.7 (osd.7) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:46 smithi169 bash[19604]: cluster 2024-02-12T16:38:46.189279+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:46 smithi123 bash[19119]: cluster 2024-02-12T16:38:46.177275+0000 mgr.y (mgr.14146) 1631 : cluster [DBG] pgmap v2338: 31 pgs: 31 active+clean; 0 B data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:46 smithi123 bash[19119]: cluster 2024-02-12T16:38:46.189185+0000 osd.7 (osd.7) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:46 smithi123 bash[19119]: cluster 2024-02-12T16:38:46.189279+0000 osd.7 (osd.7) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:47.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:47.423 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:47.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:47.526 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:47.565 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:47.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:47.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:47.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:47.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:47.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:47.989 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:47.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:48.028 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:47 smithi121 bash[18489]: cluster 2024-02-12T16:38:46.891739+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e835: 12 total, 12 up, 12 in 2024-02-12T16:38:48.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:48.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:48.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:48.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:47 smithi123 bash[19119]: cluster 2024-02-12T16:38:46.891739+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e835: 12 total, 12 up, 12 in 2024-02-12T16:38:48.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:48.246 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:47 smithi169 bash[19604]: cluster 2024-02-12T16:38:46.891739+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e835: 12 total, 12 up, 12 in 2024-02-12T16:38:48.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:48.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:48.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:48.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:48.420 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:38:48.421 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:38:48.519 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:48.554 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:48.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:48.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:48.698 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:48.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:48.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:48.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:48.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:48.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:48.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:49.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:49.122 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:49.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:48 smithi121 bash[18489]: cluster 2024-02-12T16:38:48.177935+0000 mgr.y (mgr.14146) 1632 : cluster [DBG] pgmap v2340: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:49.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:49.258 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:49.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:48 smithi169 bash[19604]: cluster 2024-02-12T16:38:48.177935+0000 mgr.y (mgr.14146) 1632 : cluster [DBG] pgmap v2340: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:49.364 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:48 smithi123 bash[19119]: cluster 2024-02-12T16:38:48.177935+0000 mgr.y (mgr.14146) 1632 : cluster [DBG] pgmap v2340: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:49.402 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:49.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:49.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:49.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:49.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:49.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:38:49.662 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:38:49.680 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:49.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:49.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:49.819 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:49.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:49.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:49.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:49.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:50.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:38:50.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: cluster 2024-02-12T16:38:49.005426+0000 osd.4 (osd.4) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:50.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: cluster 2024-02-12T16:38:49.005529+0000 osd.4 (osd.4) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:50.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: audit 2024-02-12T16:38:49.066654+0000 mon.a (mon.0) 2858 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:50.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: audit 2024-02-12T16:38:49.067341+0000 mon.a (mon.0) 2859 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:50.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: audit 2024-02-12T16:38:49.067757+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:50.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: audit 2024-02-12T16:38:49.081142+0000 mon.a (mon.0) 2861 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:50.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: audit 2024-02-12T16:38:49.083782+0000 mon.a (mon.0) 2862 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:50.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: audit 2024-02-12T16:38:49.084834+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:50.052 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:49 smithi121 bash[18489]: audit 2024-02-12T16:38:49.092689+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:50.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:50.101 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:50.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:50.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: cluster 2024-02-12T16:38:49.005426+0000 osd.4 (osd.4) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:50.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: cluster 2024-02-12T16:38:49.005529+0000 osd.4 (osd.4) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:50.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: audit 2024-02-12T16:38:49.066654+0000 mon.a (mon.0) 2858 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:50.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: audit 2024-02-12T16:38:49.067341+0000 mon.a (mon.0) 2859 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:50.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: audit 2024-02-12T16:38:49.067757+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:50.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: audit 2024-02-12T16:38:49.081142+0000 mon.a (mon.0) 2861 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:50.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: audit 2024-02-12T16:38:49.083782+0000 mon.a (mon.0) 2862 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:50.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: audit 2024-02-12T16:38:49.084834+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:50.141 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:49 smithi123 bash[19119]: audit 2024-02-12T16:38:49.092689+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:50.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:50.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:50.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:50.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: cluster 2024-02-12T16:38:49.005426+0000 osd.4 (osd.4) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:50.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: cluster 2024-02-12T16:38:49.005529+0000 osd.4 (osd.4) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:50.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: audit 2024-02-12T16:38:49.066654+0000 mon.a (mon.0) 2858 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:38:50.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: audit 2024-02-12T16:38:49.067341+0000 mon.a (mon.0) 2859 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:38:50.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: audit 2024-02-12T16:38:49.067757+0000 mon.a (mon.0) 2860 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:38:50.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: audit 2024-02-12T16:38:49.081142+0000 mon.a (mon.0) 2861 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:38:50.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: audit 2024-02-12T16:38:49.083782+0000 mon.a (mon.0) 2862 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:38:50.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: audit 2024-02-12T16:38:49.084834+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:38:50.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:49 smithi169 bash[19604]: audit 2024-02-12T16:38:49.092689+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:38:50.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:50.374 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:50.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:50.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:50.516 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:50.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:50.621 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:50.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:50.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:50.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:50.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:50.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:50.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:50.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:50.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:51.032 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:51.069 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:51.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:51.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:51.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:50 smithi121 bash[18489]: audit 2024-02-12T16:38:49.638953+0000 mgr.y (mgr.14146) 1633 : audit [DBG] from='client.16902 v1:172.21.15.121:0/3382242086' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:38:51.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:50 smithi121 bash[18489]: cluster 2024-02-12T16:38:49.911131+0000 mon.a (mon.0) 2865 : cluster [DBG] osdmap e836: 12 total, 12 up, 12 in 2024-02-12T16:38:51.173 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:50 smithi121 bash[18489]: cluster 2024-02-12T16:38:50.178392+0000 mgr.y (mgr.14146) 1634 : cluster [DBG] pgmap v2342: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:51.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:50 smithi123 bash[19119]: audit 2024-02-12T16:38:49.638953+0000 mgr.y (mgr.14146) 1633 : audit [DBG] from='client.16902 v1:172.21.15.121:0/3382242086' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:38:51.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:50 smithi123 bash[19119]: cluster 2024-02-12T16:38:49.911131+0000 mon.a (mon.0) 2865 : cluster [DBG] osdmap e836: 12 total, 12 up, 12 in 2024-02-12T16:38:51.186 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:50 smithi123 bash[19119]: cluster 2024-02-12T16:38:50.178392+0000 mgr.y (mgr.14146) 1634 : cluster [DBG] pgmap v2342: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:51.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:51.306 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:50 smithi169 bash[19604]: audit 2024-02-12T16:38:49.638953+0000 mgr.y (mgr.14146) 1633 : audit [DBG] from='client.16902 v1:172.21.15.121:0/3382242086' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:38:51.306 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:50 smithi169 bash[19604]: cluster 2024-02-12T16:38:49.911131+0000 mon.a (mon.0) 2865 : cluster [DBG] osdmap e836: 12 total, 12 up, 12 in 2024-02-12T16:38:51.307 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:50 smithi169 bash[19604]: cluster 2024-02-12T16:38:50.178392+0000 mgr.y (mgr.14146) 1634 : cluster [DBG] pgmap v2342: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:51.308 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2340,"stamp":"2024-02-12T16:38:48.177595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7904512,"kb_used_data":379360,"kb_used_omap":91,"kb_used_meta":7524964,"kb_avail":1117037312,"statfs":{"total":1151940427776,"available":1143846207488,"internally_reserved":0,"allocated":388464640,"data_stored":383989036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":7705563152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-337184,"num_write_kb":-10789825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-16071,"num_bytes_recovered":-1053229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.142841"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":198,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.585687+0000","last_change":"2024-02-12T16:37:45.169571+0000","last_active":"2024-02-12T16:38:41.585687+0000","last_peered":"2024-02-12T16:38:41.585687+0000","last_clean":"2024-02-12T16:38:41.585687+0000","last_became_active":"2024-02-12T16:30:55.395625+0000","last_became_peered":"2024-02-12T16:30:55.395625+0000","last_unstale":"2024-02-12T16:38:41.585687+0000","last_undegraded":"2024-02-12T16:38:41.585687+0000","last_fullsized":"2024-02-12T16:38:41.585687+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:45.169464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:37:45.169464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":195,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.321885+0000","last_change":"2024-02-12T16:35:18.045317+0000","last_active":"2024-02-12T16:38:40.321885+0000","last_peered":"2024-02-12T16:38:40.321885+0000","last_clean":"2024-02-12T16:38:40.321885+0000","last_became_active":"2024-02-12T16:30:55.398431+0000","last_became_peered":"2024-02-12T16:30:55.398431+0000","last_unstale":"2024-02-12T16:38:40.321885+0000","last_undegraded":"2024-02-12T16:38:40.321885+0000","last_fullsized":"2024-02-12T16:38:40.321885+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:18.045236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:18.045236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":197,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.904067+0000","last_change":"2024-02-12T16:36:57.953264+0000","last_active":"2024-02-12T16:38:41.904067+0000","last_peered":"2024-02-12T16:38:41.904067+0000","last_clean":"2024-02-12T16:38:41.904067+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:38:41.904067+0000","last_undegraded":"2024-02-12T16:38:41.904067+0000","last_fullsized":"2024-02-12T16:38:41.904067+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:36:57.953149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:36:57.953149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":202,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.904145+0000","last_change":"2024-02-12T16:38:13.834040+0000","last_active":"2024-02-12T16:38:41.904145+0000","last_peered":"2024-02-12T16:38:41.904145+0000","last_clean":"2024-02-12T16:38:41.904145+0000","last_became_active":"2024-02-12T16:30:55.409579+0000","last_became_peered":"2024-02-12T16:30:55.409579+0000","last_unstale":"2024-02-12T16:38:41.904145+0000","last_undegraded":"2024-02-12T16:38:41.904145+0000","last_fullsized":"2024-02-12T16:38:41.904145+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:13.833796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:38:13.833796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":241,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.614097+0000","last_change":"2024-02-12T16:35:10.432374+0000","last_active":"2024-02-12T16:38:40.614097+0000","last_peered":"2024-02-12T16:38:40.614097+0000","last_clean":"2024-02-12T16:38:40.614097+0000","last_became_active":"2024-02-12T16:30:55.394105+0000","last_became_peered":"2024-02-12T16:30:55.394105+0000","last_unstale":"2024-02-12T16:38:40.614097+0000","last_undegraded":"2024-02-12T16:38:40.614097+0000","last_fullsized":"2024-02-12T16:38:40.614097+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:10.432253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:35:10.432253+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":210,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.903175+0000","last_change":"2024-02-12T16:38:15.825816+0000","last_active":"2024-02-12T16:38:40.903175+0000","last_peered":"2024-02-12T16:38:40.903175+0000","last_clean":"2024-02-12T16:38:40.903175+0000","last_became_active":"2024-02-12T16:36:17.847960+0000","last_became_peered":"2024-02-12T16:36:17.847960+0000","last_unstale":"2024-02-12T16:38:40.903175+0000","last_undegraded":"2024-02-12T16:38:40.903175+0000","last_fullsized":"2024-02-12T16:38:40.903175+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:15.825691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:38:15.825691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":554,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-02-12T16:38:42.786962+0000","last_change":"2024-02-12T16:37:56.814210+0000","last_active":"2024-02-12T16:38:42.786962+0000","last_peered":"2024-02-12T16:38:42.786962+0000","last_clean":"2024-02-12T16:38:42.786962+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:38:42.786962+0000","last_undegraded":"2024-02-12T16:38:42.786962+0000","last_fullsized":"2024-02-12T16:38:42.786962+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:56.814102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:37:56.814102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":196,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.904169+0000","last_change":"2024-02-12T16:36:55.992593+0000","last_active":"2024-02-12T16:38:41.904169+0000","last_peered":"2024-02-12T16:38:41.904169+0000","last_clean":"2024-02-12T16:38:41.904169+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:38:41.904169+0000","last_undegraded":"2024-02-12T16:38:41.904169+0000","last_fullsized":"2024-02-12T16:38:41.904169+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:36:55.992473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:36:55.992473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":197,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.319799+0000","last_change":"2024-02-12T16:35:33.040481+0000","last_active":"2024-02-12T16:38:40.319799+0000","last_peered":"2024-02-12T16:38:40.319799+0000","last_clean":"2024-02-12T16:38:40.319799+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:38:40.319799+0000","last_undegraded":"2024-02-12T16:38:40.319799+0000","last_fullsized":"2024-02-12T16:38:40.319799+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:33.040362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:33.040362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":191,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.906218+0000","last_change":"2024-02-12T16:36:21.958710+0000","last_active":"2024-02-12T16:38:41.906218+0000","last_peered":"2024-02-12T16:38:41.906218+0000","last_clean":"2024-02-12T16:38:41.906218+0000","last_became_active":"2024-02-12T16:30:55.404763+0000","last_became_peered":"2024-02-12T16:30:55.404763+0000","last_unstale":"2024-02-12T16:38:41.906218+0000","last_undegraded":"2024-02-12T16:38:41.906218+0000","last_fullsized":"2024-02-12T16:38:41.906218+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:36:21.958626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:36:21.958626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":446,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.315618+0000","last_change":"2024-02-12T16:38:12.822793+0000","last_active":"2024-02-12T16:38:40.315618+0000","last_peered":"2024-02-12T16:38:40.315618+0000","last_clean":"2024-02-12T16:38:40.315618+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:38:40.315618+0000","last_undegraded":"2024-02-12T16:38:40.315618+0000","last_fullsized":"2024-02-12T16:38:40.315618+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:07.718055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:35:07.718055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.905710+0000","last_change":"2024-02-12T16:38:16.254285+0000","last_active":"2024-02-12T16:38:41.905710+0000","last_peered":"2024-02-12T16:38:41.905710+0000","last_clean":"2024-02-12T16:38:41.905710+0000","last_became_active":"2024-02-12T16:38:16.253962+0000","last_became_peered":"2024-02-12T16:38:16.253962+0000","last_unstale":"2024-02-12T16:38:41.905710+0000","last_undegraded":"2024-02-12T16:38:41.905710+0000","last_fullsized":"2024-02-12T16:38:41.905710+0000","mapping_epoch":824,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":825,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:41.801861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:37:41.801861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":565,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.903075+0000","last_change":"2024-02-12T16:37:39.765865+0000","last_active":"2024-02-12T16:38:40.903075+0000","last_peered":"2024-02-12T16:38:40.903075+0000","last_clean":"2024-02-12T16:38:40.903075+0000","last_became_active":"2024-02-12T16:36:17.846631+0000","last_became_peered":"2024-02-12T16:36:17.846631+0000","last_unstale":"2024-02-12T16:38:40.903075+0000","last_undegraded":"2024-02-12T16:38:40.903075+0000","last_fullsized":"2024-02-12T16:38:40.903075+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:39.765751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:37:39.765751+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":950,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.585591+0000","last_change":"2024-02-12T16:38:01.117143+0000","last_active":"2024-02-12T16:38:41.585591+0000","last_peered":"2024-02-12T16:38:41.585591+0000","last_clean":"2024-02-12T16:38:41.585591+0000","last_became_active":"2024-02-12T16:19:29.356001+0000","last_became_peered":"2024-02-12T16:19:29.356001+0000","last_unstale":"2024-02-12T16:38:41.585591+0000","last_undegraded":"2024-02-12T16:38:41.585591+0000","last_fullsized":"2024-02-12T16:38:41.585591+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:01.117069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:38:01.117069+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":601,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.585719+0000","last_change":"2024-02-12T16:36:30.388002+0000","last_active":"2024-02-12T16:38:41.585719+0000","last_peered":"2024-02-12T16:38:41.585719+0000","last_clean":"2024-02-12T16:38:41.585719+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:38:41.585719+0000","last_undegraded":"2024-02-12T16:38:41.585719+0000","last_fullsized":"2024-02-12T16:38:41.585719+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:36:30.387884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:36:30.387884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":544,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.903375+0000","last_change":"2024-02-12T16:36:57.057605+0000","last_active":"2024-02-12T16:38:40.903375+0000","last_peered":"2024-02-12T16:38:40.903375+0000","last_clean":"2024-02-12T16:38:40.903375+0000","last_became_active":"2024-02-12T16:30:49.373979+0000","last_became_peered":"2024-02-12T16:30:49.373979+0000","last_unstale":"2024-02-12T16:38:40.903375+0000","last_undegraded":"2024-02-12T16:38:40.903375+0000","last_fullsized":"2024-02-12T16:38:40.903375+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:36:57.057484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:57.057484+0000","last_clean_scrub_stamp":"2024-02-12T16:36:57.057484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":600,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.902951+0000","last_change":"2024-02-12T16:38:11.892762+0000","last_active":"2024-02-12T16:38:40.902951+0000","last_peered":"2024-02-12T16:38:40.902951+0000","last_clean":"2024-02-12T16:38:40.902951+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:38:40.902951+0000","last_undegraded":"2024-02-12T16:38:40.902951+0000","last_fullsized":"2024-02-12T16:38:40.902951+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:11.892668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:38:11.892668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":404,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.315905+0000","last_change":"2024-02-12T16:37:44.252931+0000","last_active":"2024-02-12T16:38:40.315905+0000","last_peered":"2024-02-12T16:38:40.315905+0000","last_clean":"2024-02-12T16:38:40.315905+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:38:40.315905+0000","last_undegraded":"2024-02-12T16:38:40.315905+0000","last_fullsized":"2024-02-12T16:38:40.315905+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:44.252850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:37:44.252850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":966,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.906068+0000","last_change":"2024-02-12T16:35:21.488378+0000","last_active":"2024-02-12T16:38:41.906068+0000","last_peered":"2024-02-12T16:38:41.906068+0000","last_clean":"2024-02-12T16:38:41.906068+0000","last_became_active":"2024-02-12T16:17:37.456323+0000","last_became_peered":"2024-02-12T16:17:37.456323+0000","last_unstale":"2024-02-12T16:38:41.906068+0000","last_undegraded":"2024-02-12T16:38:41.906068+0000","last_fullsized":"2024-02-12T16:38:41.906068+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:21.488276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:21.488276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":390,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.905513+0000","last_change":"2024-02-12T16:38:10.836649+0000","last_active":"2024-02-12T16:38:41.905513+0000","last_peered":"2024-02-12T16:38:41.905513+0000","last_clean":"2024-02-12T16:38:41.905513+0000","last_became_active":"2024-02-12T16:30:49.374934+0000","last_became_peered":"2024-02-12T16:30:49.374934+0000","last_unstale":"2024-02-12T16:38:41.905513+0000","last_undegraded":"2024-02-12T16:38:41.905513+0000","last_fullsized":"2024-02-12T16:38:41.905513+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:26.539828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:26.539828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":384,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.321561+0000","last_change":"2024-02-12T16:37:43.255399+0000","last_active":"2024-02-12T16:38:40.321561+0000","last_peered":"2024-02-12T16:38:40.321561+0000","last_clean":"2024-02-12T16:38:40.321561+0000","last_became_active":"2024-02-12T16:30:49.382268+0000","last_became_peered":"2024-02-12T16:30:49.382268+0000","last_unstale":"2024-02-12T16:38:40.321561+0000","last_undegraded":"2024-02-12T16:38:40.321561+0000","last_fullsized":"2024-02-12T16:38:40.321561+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:43.255293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:37:43.255293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":549,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.585879+0000","last_change":"2024-02-12T16:35:28.794885+0000","last_active":"2024-02-12T16:38:41.585879+0000","last_peered":"2024-02-12T16:38:41.585879+0000","last_clean":"2024-02-12T16:38:41.585879+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:38:41.585879+0000","last_undegraded":"2024-02-12T16:38:41.585879+0000","last_fullsized":"2024-02-12T16:38:41.585879+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:28.794774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:35:28.794774+0000","last_clean_scrub_stamp":"2024-02-12T16:35:28.794774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":357,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.614185+0000","last_change":"2024-02-12T16:35:19.574228+0000","last_active":"2024-02-12T16:38:40.614185+0000","last_peered":"2024-02-12T16:38:40.614185+0000","last_clean":"2024-02-12T16:38:40.614185+0000","last_became_active":"2024-02-12T16:30:49.381004+0000","last_became_peered":"2024-02-12T16:30:49.381004+0000","last_unstale":"2024-02-12T16:38:40.614185+0000","last_undegraded":"2024-02-12T16:38:40.614185+0000","last_fullsized":"2024-02-12T16:38:40.614185+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:19.574110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:19.574110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":949,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.905678+0000","last_change":"2024-02-12T16:35:11.592580+0000","last_active":"2024-02-12T16:38:41.905678+0000","last_peered":"2024-02-12T16:38:41.905678+0000","last_clean":"2024-02-12T16:38:41.905678+0000","last_became_active":"2024-02-12T16:30:49.404783+0000","last_became_peered":"2024-02-12T16:30:49.404783+0000","last_unstale":"2024-02-12T16:38:41.905678+0000","last_undegraded":"2024-02-12T16:38:41.905678+0000","last_fullsized":"2024-02-12T16:38:41.905678+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:11.592474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:11.592474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":548,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-02-12T16:38:42.787032+0000","last_change":"2024-02-12T16:37:50.701109+0000","last_active":"2024-02-12T16:38:42.787032+0000","last_peered":"2024-02-12T16:38:42.787032+0000","last_clean":"2024-02-12T16:38:42.787032+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:38:42.787032+0000","last_undegraded":"2024-02-12T16:38:42.787032+0000","last_fullsized":"2024-02-12T16:38:42.787032+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:50.701008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:37:50.701008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":306,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.585690+0000","last_change":"2024-02-12T16:37:41.320773+0000","last_active":"2024-02-12T16:38:41.585690+0000","last_peered":"2024-02-12T16:38:41.585690+0000","last_clean":"2024-02-12T16:38:41.585690+0000","last_became_active":"2024-02-12T16:30:49.376219+0000","last_became_peered":"2024-02-12T16:30:49.376219+0000","last_unstale":"2024-02-12T16:38:41.585690+0000","last_undegraded":"2024-02-12T16:38:41.585690+0000","last_fullsized":"2024-02-12T16:38:41.585690+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:41.320666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:37:41.320666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":539,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.825487+0000","last_change":"2024-02-12T16:37:44.746418+0000","last_active":"2024-02-12T16:38:41.825487+0000","last_peered":"2024-02-12T16:38:41.825487+0000","last_clean":"2024-02-12T16:38:41.825487+0000","last_became_active":"2024-02-12T16:30:49.384015+0000","last_became_peered":"2024-02-12T16:30:49.384015+0000","last_unstale":"2024-02-12T16:38:41.825487+0000","last_undegraded":"2024-02-12T16:38:41.825487+0000","last_fullsized":"2024-02-12T16:38:41.825487+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:44.746349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:37:44.746349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":46,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.614033+0000","last_change":"2024-02-12T16:36:17.850421+0000","last_active":"2024-02-12T16:38:40.614033+0000","last_peered":"2024-02-12T16:38:40.614033+0000","last_clean":"2024-02-12T16:38:40.614033+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:38:40.614033+0000","last_undegraded":"2024-02-12T16:38:40.614033+0000","last_fullsized":"2024-02-12T16:38:40.614033+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:23.904216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.186362+0000","last_clean_scrub_stamp":"2024-02-12T16:35:23.904216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":61,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:41.903968+0000","last_change":"2024-02-12T16:38:11.810599+0000","last_active":"2024-02-12T16:38:41.903968+0000","last_peered":"2024-02-12T16:38:41.903968+0000","last_clean":"2024-02-12T16:38:41.903968+0000","last_became_active":"2024-02-12T16:36:18.479362+0000","last_became_peered":"2024-02-12T16:36:18.479362+0000","last_unstale":"2024-02-12T16:38:41.903968+0000","last_undegraded":"2024-02-12T16:38:41.903968+0000","last_fullsized":"2024-02-12T16:38:41.903968+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:11.810484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:54.947881+0000","last_clean_scrub_stamp":"2024-02-12T16:38:11.810484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":416,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-02-12T16:38:42.787160+0000","last_change":"2024-02-12T16:37:46.810041+0000","last_active":"2024-02-12T16:38:42.787160+0000","last_peered":"2024-02-12T16:38:42.787160+0000","last_clean":"2024-02-12T16:38:42.787160+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:38:42.787160+0000","last_undegraded":"2024-02-12T16:38:42.787160+0000","last_fullsized":"2024-02-12T16:38:42.787160+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:37:46.809950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:37:46.809950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":364,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-02-12T16:38:40.322258+0000","last_change":"2024-02-12T16:35:08.806700+0000","last_active":"2024-02-12T16:38:40.322258+0000","last_peered":"2024-02-12T16:38:40.322258+0000","last_clean":"2024-02-12T16:38:40.322258+0000","last_became_active":"2024-02-12T16:23:53.120611+0000","last_became_peered":"2024-02-12T16:23:53.120611+0000","last_unstale":"2024-02-12T16:38:40.322258+0000","last_undegraded":"2024-02-12T16:38:40.322258+0000","last_fullsized":"2024-02-12T16:38:40.322258+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:35:08.806574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:35:08.806574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431959016,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901664,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":900928,"kb_avail":92843488,"statfs":{"total":95995035648,"available":95071731712,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922550272},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":3.149,"15min":1.736},"min":{"1min":0.28000000000000003,"5min":0.249,"15min":0.246},"max":{"1min":12.287000000000001,"5min":124.437,"15min":124.437},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.7749999999999999,"5min":3.262,"15min":1.897},"min":{"1min":0.33900000000000002,"5min":0.30499999999999999,"15min":0.28399999999999997},"max":{"1min":8.3610000000000007,"5min":126.926,"15min":126.926},"last":1.446}]},{"osd":1,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":1.0980000000000001,"15min":0.98699999999999999},"min":{"1min":0.38300000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":7.3129999999999997,"5min":7.3129999999999997,"15min":8.9849999999999994},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":0.85599999999999998,"15min":0.89100000000000001},"min":{"1min":0.38900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.3949999999999996,"5min":5.3949999999999996,"15min":11.167},"last":0.72499999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:22:51 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0649999999999999,"15min":1.0469999999999999},"min":{"1min":0.64700000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.512,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0820000000000001,"15min":1.0109999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3420000000000001,"5min":1.982,"15min":1.982},"last":0.999}]},{"osd":3,"last update":"Mon Feb 12 16:38:25 2024","interfaces":[{"interface":"back","average":{"1min":2.4580000000000002,"5min":2.887,"15min":1.6910000000000001},"min":{"1min":0.38200000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":27.327999999999999,"5min":121.169,"15min":121.169},"last":1.0820000000000001},{"interface":"front","average":{"1min":2.7389999999999999,"5min":2.8719999999999999,"15min":1.698},"min":{"1min":0.436,"5min":0.40300000000000002,"15min":0.31900000000000001},"max":{"1min":27.346,"5min":121.181,"15min":121.181},"last":1.1479999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.0209999999999999,"15min":0.92400000000000004},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":6.2039999999999997,"5min":6.2039999999999997,"15min":6.2039999999999997},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.667,"5min":1.0409999999999999,"15min":0.94499999999999995},"min":{"1min":0.40999999999999998,"5min":0.35499999999999998,"15min":0.34499999999999997},"max":{"1min":7.3920000000000003,"5min":7.3920000000000003,"15min":7.3920000000000003},"last":1.3420000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.085,"15min":1.1339999999999999},"min":{"1min":0.46400000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.5289999999999999,"5min":5.5289999999999999,"15min":5.968},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.556,"5min":1.0940000000000001,"15min":1.163},"min":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.33100000000000002},"max":{"1min":7.4249999999999998,"5min":7.4249999999999998,"15min":7.4249999999999998},"last":1.748}]},{"osd":6,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.2150000000000001,"15min":1.0860000000000001},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":6.2649999999999997,"5min":6.2649999999999997,"15min":6.4160000000000004},"last":0.877},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.4219999999999999,"15min":1.2689999999999999},"min":{"1min":0.52500000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":6.242,"5min":9.5389999999999997,"15min":9.5389999999999997},"last":1.238}]},{"osd":7,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.194,"15min":1.1699999999999999},"min":{"1min":0.496,"5min":0.43099999999999999,"15min":0.38700000000000001},"max":{"1min":5.5670000000000002,"5min":7.851,"15min":7.851},"last":1.212},{"interface":"front","average":{"1min":1.738,"5min":1.3029999999999999,"15min":1.125},"min":{"1min":0.503,"5min":0.45200000000000001,"15min":0.36699999999999999},"max":{"1min":7.3529999999999998,"5min":7.3529999999999998,"15min":8.3000000000000007},"last":1.5740000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.105,"15min":2.7080000000000002},"min":{"1min":0.437,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":537.54399999999998},"last":1.403},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.153,"15min":2.8330000000000002},"min":{"1min":0.55900000000000005,"5min":0.47099999999999997,"15min":0.40500000000000003},"max":{"1min":5.4880000000000004,"5min":5.4880000000000004,"15min":539.86099999999999},"last":1.28}]},{"osd":9,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":2.153,"5min":1.4039999999999999,"15min":1.175},"min":{"1min":0.59299999999999997,"5min":0.45800000000000002,"15min":0.38600000000000001},"max":{"1min":7.5709999999999997,"5min":7.5709999999999997,"15min":7.5709999999999997},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.2769999999999999,"15min":1.1220000000000001},"min":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.44700000000000001},"max":{"1min":5.7089999999999996,"5min":5.7089999999999996,"15min":5.7089999999999996},"last":1.6100000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.306,"15min":2.8660000000000001},"min":{"1min":0.56999999999999995,"5min":0.46700000000000003,"15min":0.439},"max":{"1min":7.4539999999999997,"5min":7.4539999999999997,"15min":536.245},"last":1.819},{"interface":"front","average":{"1min":1.484,"5min":1.2929999999999999,"15min":2.863},"min":{"1min":0.52500000000000002,"5min":0.44,"15min":0.432},"max":{"1min":6.4930000000000003,"5min":6.4930000000000003,"15min":536.64300000000003},"last":1.5449999999999999}]}]},{"osd":10,"up_from":83,"seq":356482286064,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306144,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93439008,"statfs":{"total":95995035648,"available":95681544192,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":1.724,"15min":1.7150000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.245},"max":{"1min":4.7140000000000004,"5min":111.935,"15min":162.38399999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.903,"15min":1.8560000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26500000000000001},"max":{"1min":4.3920000000000003,"5min":111.316,"15min":162.01300000000001},"last":1.046}]},{"osd":1,"last update":"Mon Feb 12 16:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.70999999999999996,"15min":0.875},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.0940000000000001,"5min":3.2850000000000001,"15min":5.9000000000000004},"last":0.438},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.93700000000000006,"15min":0.98299999999999998},"min":{"1min":0.34100000000000003,"5min":0.32800000000000001,"15min":0.309},"max":{"1min":3.3079999999999998,"5min":4.673,"15min":6.1970000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":2.125,"15min":1.379},"min":{"1min":0.30399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.1509999999999998,"5min":106.822,"15min":106.822},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.252,"5min":2.2210000000000001,"15min":1.411},"min":{"1min":0.41099999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4320000000000004,"5min":107.88800000000001,"15min":107.88800000000001},"last":1.321}]},{"osd":3,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":2.1739999999999999,"15min":1.5169999999999999},"min":{"1min":0.34899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.8970000000000002,"5min":107.014,"15min":107.014},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.151,"5min":2.133,"15min":1.359},"min":{"1min":0.375,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.9299999999999997,"5min":110.80800000000001,"15min":110.80800000000001},"last":1.3740000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.80600000000000005,"15min":0.93500000000000005},"min":{"1min":0.33000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.9749999999999996,"5min":4.9749999999999996,"15min":6.3689999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.083,"5min":0.88900000000000001,"15min":0.97999999999999998},"min":{"1min":0.495,"5min":0.318,"15min":0.30199999999999999},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":7.3070000000000004},"last":1.6140000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.83999999999999997,"15min":0.98199999999999998},"min":{"1min":0.378,"5min":0.35399999999999998,"15min":0.32700000000000001},"max":{"1min":5.0110000000000001,"5min":5.0110000000000001,"15min":6.343},"last":1.129},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.0349999999999999,"15min":1.1850000000000001},"min":{"1min":0.50700000000000001,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":4.5570000000000004,"5min":5.4400000000000004,"15min":7.2249999999999996},"last":1.446}]},{"osd":6,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":5.1719999999999997,"15min":2.4809999999999999},"min":{"1min":0.373,"5min":0.36099999999999999,"15min":0.35499999999999998},"max":{"1min":5.0780000000000003,"5min":460.90199999999999,"15min":460.90199999999999},"last":1.587},{"interface":"front","average":{"1min":1.2250000000000001,"5min":5.2610000000000001,"15min":2.512},"min":{"1min":0.55500000000000005,"5min":0.44,"15min":0.376},"max":{"1min":4.5979999999999999,"5min":460.93299999999999,"15min":460.93299999999999},"last":1.7809999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.93000000000000005,"15min":1.014},"min":{"1min":0.41599999999999998,"5min":0.40799999999999997,"15min":0.375},"max":{"1min":5.0419999999999998,"5min":5.0419999999999998,"15min":7.3280000000000003},"last":1.488},{"interface":"front","average":{"1min":1.212,"5min":1.1819999999999999,"15min":1.175},"min":{"1min":0.57599999999999996,"5min":0.434,"15min":0.374},"max":{"1min":4.532,"5min":5.306,"15min":6.4539999999999997},"last":1.6930000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.127,"15min":1.0880000000000001},"min":{"1min":0.60399999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.4640000000000004,"5min":8.4420000000000002,"15min":8.4420000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.948,"5min":1.175,"15min":1.0660000000000001},"min":{"1min":0.44800000000000001,"5min":0.36299999999999999,"15min":0.35799999999999998},"max":{"1min":7.569,"5min":7.569,"15min":7.569},"last":1.6619999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.026,"15min":1.069},"min":{"1min":0.48299999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.2560000000000002,"5min":6.2560000000000002,"15min":6.6239999999999997},"last":1.645},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.125,"15min":1.0549999999999999},"min":{"1min":0.56100000000000005,"5min":0.44900000000000001,"15min":0.42899999999999999},"max":{"1min":4.4989999999999997,"5min":6.0030000000000001,"15min":7.3479999999999999},"last":1.859}]},{"osd":11,"last update":"Mon Feb 12 16:37:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.258,"15min":1.165},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.39400000000000002},"max":{"1min":6.5570000000000004,"5min":6.5570000000000004,"15min":7.3929999999999998},"last":1.8089999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.1679999999999999,"15min":1.137},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.9300000000000002,"5min":3.9300000000000002,"15min":6.5860000000000003},"last":1.903}]}]},{"osd":9,"up_from":74,"seq":317827580409,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594528,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93150624,"statfs":{"total":95995035648,"available":95386238976,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":2.8260000000000001,"15min":1.6559999999999999},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.4880000000000004,"5min":112.508,"15min":112.508},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":2.839,"15min":1.6759999999999999},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.492,"5min":110.15300000000001,"15min":110.15300000000001},"last":1.04}]},{"osd":1,"last update":"Mon Feb 12 16:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.01,"15min":0.96499999999999997},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.9630000000000001,"5min":6.6799999999999997,"15min":6.6799999999999997},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.097,"15min":0.97499999999999998},"min":{"1min":0.34100000000000003,"5min":0.29899999999999999,"15min":0.26100000000000001},"max":{"1min":4.2469999999999999,"5min":5.367,"15min":5.7910000000000004},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:38:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":2.2069999999999999,"15min":2.819},"min":{"1min":0.35299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.0019999999999998,"5min":52.097000000000001,"15min":52.097000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":2.3719999999999999,"15min":3.25},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.3479999999999999,"5min":51.209000000000003,"15min":51.209000000000003},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:33:43 2024","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":8.8390000000000004,"15min":8.6829999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":197.661,"5min":197.661,"15min":197.661},"last":0.84499999999999997},{"interface":"front","average":{"1min":9.8330000000000002,"5min":10.432,"15min":10.532},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":197.31200000000001,"5min":197.31200000000001,"15min":197.31200000000001},"last":0.89300000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.91600000000000004,"15min":0.96199999999999997},"min":{"1min":0.39000000000000001,"5min":0.33200000000000002,"15min":0.32200000000000001},"max":{"1min":4.569,"5min":4.569,"15min":6.3090000000000002},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.125,"5min":1.01,"15min":1.0620000000000001},"min":{"1min":0.42599999999999999,"5min":0.39500000000000002,"15min":0.36899999999999999},"max":{"1min":4.633,"5min":4.633,"15min":6.3490000000000002},"last":0.97799999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.88,"15min":1.093},"min":{"1min":0.42299999999999999,"5min":0.311,"15min":0.311},"max":{"1min":1.526,"5min":5.7389999999999999,"15min":6.8559999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.94399999999999995,"15min":1.081},"min":{"1min":0.432,"5min":0.376,"15min":0.376},"max":{"1min":4.319,"5min":4.319,"15min":6.8659999999999997},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.0820000000000001,"15min":1.1120000000000001},"min":{"1min":0.41999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.3540000000000001,"5min":13.208,"15min":13.208},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2390000000000001,"15min":1.2370000000000001},"min":{"1min":0.36799999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.8289999999999997,"5min":12.968999999999999,"15min":14.718},"last":1.02}]},{"osd":7,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.95799999999999996,"15min":1.169},"min":{"1min":0.45600000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":6.7050000000000001,"15min":7.9489999999999998},"last":0.877},{"interface":"front","average":{"1min":0.73299999999999998,"5min":1.1950000000000001,"15min":1.2529999999999999},"min":{"1min":0.53700000000000003,"5min":0.44800000000000001,"15min":0.43099999999999999},"max":{"1min":1.3220000000000001,"5min":6.7720000000000002,"15min":7.9160000000000004},"last":1.444}]},{"osd":8,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.1699999999999999,"15min":1.0660000000000001},"min":{"1min":0.40500000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.4269999999999996,"5min":7.593,"15min":7.593},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.103,"15min":1.0740000000000001},"min":{"1min":0.48899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.2869999999999999,"5min":4.2869999999999999,"15min":6.1980000000000004},"last":0.85299999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:38:01 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0389999999999999,"15min":1.0900000000000001},"min":{"1min":0.434,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.133,"5min":6.7960000000000003,"15min":6.7960000000000003},"last":1.4950000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.0820000000000001,"15min":1.1259999999999999},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.8799999999999999,"5min":3.9729999999999999,"15min":6.2560000000000002},"last":0.93300000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:38:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.0649999999999999,"15min":1.087},"min":{"1min":0.50900000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.208,"5min":6.7329999999999997,"15min":6.7329999999999997},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.1180000000000001,"15min":1.0800000000000001},"min":{"1min":0.45900000000000002,"5min":0.44500000000000001,"15min":0.41599999999999998},"max":{"1min":1.264,"5min":5.5069999999999997,"15min":5.5069999999999997},"last":0.95799999999999996}]}]},{"osd":8,"up_from":65,"seq":279172874754,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590432,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":93154720,"statfs":{"total":95995035648,"available":95390433280,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":0.997,"15min":1.2669999999999999},"min":{"1min":0.28100000000000003,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.835,"5min":7.4379999999999997,"15min":134.01499999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.325,"15min":1.468},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.7349999999999999,"5min":9.0180000000000007,"15min":133.667},"last":1.3240000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0190000000000001,"15min":1.3320000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":69.956000000000003,"15min":69.956000000000003},"last":0.497},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.9870000000000001,"15min":1.4079999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4580000000000002,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":1.014,"15min":0.85199999999999998},"min":{"1min":0.32500000000000001,"5min":0.30299999999999999,"15min":0.29299999999999998},"max":{"1min":0.96299999999999997,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.871,"5min":1.337,"15min":1.0389999999999999},"min":{"1min":0.39000000000000001,"5min":0.28999999999999998,"15min":0.26800000000000002},"max":{"1min":8.9399999999999995,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.90300000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:38:12 2024","interfaces":[{"interface":"back","average":{"1min":3.0670000000000002,"5min":1.554,"15min":0.92900000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.30399999999999999},"max":{"1min":25.919,"5min":25.919,"15min":25.919},"last":1.1910000000000001},{"interface":"front","average":{"1min":3.0289999999999999,"5min":1.681,"15min":1.087},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":25.984999999999999,"5min":25.984999999999999,"15min":25.984999999999999},"last":1.1240000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.91300000000000003,"15min":0.82699999999999996},"min":{"1min":0.35699999999999998,"5min":0.34000000000000002,"15min":0.29599999999999999},"max":{"1min":1.7370000000000001,"5min":8.609,"15min":8.609},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3180000000000001,"15min":1.0109999999999999},"min":{"1min":0.41899999999999998,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":3.8180000000000001,"5min":9.0890000000000004,"15min":9.0890000000000004},"last":1.5249999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.0640000000000001,"15min":1.0129999999999999},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":4.8710000000000004,"5min":7.2960000000000003,"15min":7.2960000000000003},"last":1.165},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.397,"15min":1.1659999999999999},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.35199999999999998},"max":{"1min":3.7919999999999998,"5min":9.0540000000000003,"15min":9.0540000000000003},"last":1.645}]},{"osd":6,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.0209999999999999,"15min":1.077},"min":{"1min":0.46100000000000002,"5min":0.38200000000000001,"15min":0.373},"max":{"1min":4.8330000000000002,"5min":8.5220000000000002,"15min":8.5220000000000002},"last":1.486},{"interface":"front","average":{"1min":1.224,"5min":1.3620000000000001,"15min":1.2450000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.0099999999999998,"5min":9.1500000000000004,"15min":9.1500000000000004},"last":1.7569999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.085,"15min":3.3860000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.9710000000000001,"5min":8.5709999999999997,"15min":706.67700000000002},"last":1.792},{"interface":"front","average":{"1min":1.373,"5min":1.4870000000000001,"15min":3.5720000000000001},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":5.1079999999999997,"5min":9.1799999999999997,"15min":707.49099999999999},"last":1.923}]},{"osd":9,"last update":"Mon Feb 12 16:38:26 2024","interfaces":[{"interface":"back","average":{"1min":1.9650000000000001,"5min":1.466,"15min":1.1439999999999999},"min":{"1min":0.59199999999999997,"5min":0.371,"15min":0.34200000000000003},"max":{"1min":9.1170000000000009,"5min":9.1170000000000009,"15min":9.1170000000000009},"last":1.8320000000000001},{"interface":"front","average":{"1min":2.056,"5min":1.569,"15min":1.24},"min":{"1min":0.748,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":9.1980000000000004,"5min":9.1980000000000004,"15min":9.1980000000000004},"last":2.02}]},{"osd":10,"last update":"Mon Feb 12 16:38:06 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.099,"15min":0.99099999999999999},"min":{"1min":0.46600000000000003,"5min":0.39000000000000001,"15min":0.36699999999999999},"max":{"1min":3.6669999999999998,"5min":8.452,"15min":8.452},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.2490000000000001,"15min":1.046},"min":{"1min":0.48699999999999999,"5min":0.43099999999999999,"15min":0.38500000000000001},"max":{"1min":5.0140000000000002,"5min":5.0720000000000001,"15min":5.0720000000000001},"last":1.6870000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:38:06 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":1.157,"15min":1.052},"min":{"1min":0.52100000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0369999999999999,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":1.857},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2470000000000001,"15min":1.0229999999999999},"min":{"1min":0.54300000000000004,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":5.0540000000000003,"5min":5.141,"15min":5.2320000000000002},"last":1.5649999999999999}]}]},{"osd":7,"up_from":56,"seq":240518169097,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594516,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93150636,"statfs":{"total":95995035648,"available":95386251264,"internally_reserved":0,"allocated":733184,"data_stored":366621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:38:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":4.4649999999999999,"15min":2.968},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.252},"max":{"1min":4.5620000000000003,"5min":184.77199999999999,"15min":187.33699999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":4.4260000000000002,"15min":2.9409999999999998},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.27400000000000002},"max":{"1min":6.758,"5min":184.64599999999999,"15min":187.48400000000001},"last":1.119}]},{"osd":1,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.149,"15min":1.3560000000000001},"min":{"1min":0.309,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.4630000000000001,"5min":14.106,"15min":14.106},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.1859999999999999,"5min":0.94599999999999995,"15min":1.0800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.2690000000000001,"5min":7.2690000000000001,"15min":7.2690000000000001},"last":1.1739999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:32:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.046,"15min":1.143},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.9299999999999997,"5min":5.4340000000000002,"15min":35.655000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.3089999999999999,"15min":1.3360000000000001},"min":{"1min":0.378,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.0599999999999996,"5min":7.6040000000000001,"15min":35.164000000000001},"last":0.998}]},{"osd":3,"last update":"Mon Feb 12 16:38:35 2024","interfaces":[{"interface":"back","average":{"1min":4.7119999999999997,"5min":1.9590000000000001,"15min":1.353},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.26100000000000001},"max":{"1min":77.603999999999999,"5min":77.603999999999999,"15min":77.603999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":4.54,"5min":2.1080000000000001,"15min":1.492},"min":{"1min":0.35499999999999998,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":78.031999999999996,"5min":78.031999999999996,"15min":78.031999999999996},"last":1.0800000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:38:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.95399999999999996,"15min":1.135},"min":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.26200000000000001},"max":{"1min":4.5529999999999999,"5min":10.958,"15min":10.958},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.077,"15min":1.127},"min":{"1min":0.41699999999999998,"5min":0.312,"15min":0.25800000000000001},"max":{"1min":4.5940000000000003,"5min":13.925000000000001,"15min":13.925000000000001},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:38:00 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.84099999999999997,"15min":2.0230000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.26500000000000001},"max":{"1min":4.3520000000000003,"5min":5.3799999999999999,"15min":216.84299999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.125,"15min":2.1379999999999999},"min":{"1min":0.313,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":4.0679999999999996,"5min":13.805999999999999,"15min":217.02600000000001},"last":1.371}]},{"osd":6,"last update":"Mon Feb 12 16:38:00 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.294,"15min":1.2669999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.31},"max":{"1min":4.1479999999999997,"5min":5.9509999999999996,"15min":11.380000000000001},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.264,"15min":1.357},"min":{"1min":0.432,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":3.6299999999999999,"5min":7.3259999999999996,"15min":8.0719999999999992},"last":1.343}]},{"osd":8,"last update":"Mon Feb 12 16:38:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.0229999999999999,"15min":2.9249999999999998},"min":{"1min":0.441,"5min":0.377,"15min":0.377},"max":{"1min":5.2229999999999999,"5min":5.2229999999999999,"15min":498.68299999999999},"last":1.474},{"interface":"front","average":{"1min":1.296,"5min":1.159,"15min":3.1459999999999999},"min":{"1min":0.5,"5min":0.40799999999999997,"15min":0.35999999999999999},"max":{"1min":5.2350000000000003,"5min":14.021000000000001,"15min":498.69999999999999},"last":1.4510000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:38:28 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":2.0129999999999999,"15min":1.444},"min":{"1min":0.47699999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":8.5299999999999994,"5min":90.025999999999996,"15min":90.025999999999996},"last":1.631},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.9410000000000001,"15min":1.5029999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.40100000000000002},"max":{"1min":1.671,"5min":90.049999999999997,"15min":90.049999999999997},"last":1.5409999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:38:00 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":1.177,"15min":3.4449999999999998},"min":{"1min":0.56999999999999995,"5min":0.42399999999999999,"15min":0.376},"max":{"1min":0.83499999999999996,"5min":5.6260000000000003,"15min":497.52699999999999},"last":1.607},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.3220000000000001,"15min":3.548},"min":{"1min":0.57699999999999996,"5min":0.44900000000000001,"15min":0.39200000000000002},"max":{"1min":4.7430000000000003,"5min":13.98,"15min":496.29599999999999},"last":1.6799999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:38:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":0.51600000000000001,"5min":0.501,"15min":0.43099999999999999},"max":{"1min":3.5550000000000002,"5min":4.0389999999999997,"15min":8.4199999999999999},"last":1.722},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.0860000000000001,"15min":1.2649999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.46899999999999997},"max":{"1min":1.724,"5min":5.4139999999999997,"15min":7.0679999999999996},"last":1.665}]}]},{"osd":6,"up_from":48,"seq":206158430738,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598624,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":93146528,"statfs":{"total":95995035648,"available":95382044672,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":2.1539999999999999,"15min":1.4470000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7350000000000001,"5min":107.723,"15min":107.723},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":2.23,"15min":1.5349999999999999},"min":{"1min":0.38300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.1180000000000003,"5min":107.114,"15min":107.114},"last":0.92100000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.1020000000000001,"15min":1.0449999999999999},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":5.3550000000000004,"5min":6.0780000000000003,"15min":6.1870000000000003},"last":0.878},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.25,"15min":1.1799999999999999},"min":{"1min":0.34000000000000002,"5min":0.28699999999999998,"15min":0.25700000000000001},"max":{"1min":4.8239999999999998,"5min":6.6559999999999997,"15min":6.6959999999999997},"last":0.93500000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.5249999999999999,"15min":1.4319999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":5.3220000000000001,"5min":14.68,"15min":28.739999999999998},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.256,"15min":1.429},"min":{"1min":0.31,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.8410000000000002,"5min":15.138,"15min":32.328000000000003},"last":0.79500000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:38:03 2024","interfaces":[{"interface":"back","average":{"1min":1.8939999999999999,"5min":5.4219999999999997,"15min":3.7090000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.1549999999999994,"5min":365.58999999999997,"15min":365.58999999999997},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.456,"5min":5.6150000000000002,"15min":4.5099999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":5.1849999999999996,"5min":366.93299999999999,"15min":366.93299999999999},"last":0.753}]},{"osd":4,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.98299999999999998,"15min":1.1919999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":4.9219999999999997,"5min":5.1849999999999996,"15min":8.4559999999999995},"last":0.996},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.131,"15min":1.137},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.23999999999999999},"max":{"1min":5.2069999999999999,"5min":6.1059999999999999,"15min":7.7610000000000001},"last":1.238}]},{"osd":5,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.008,"15min":1.224},"min":{"1min":0.41899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.4740000000000002,"5min":5.4740000000000002,"15min":8.5730000000000004},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.1259999999999999,"15min":1.165},"min":{"1min":0.40500000000000003,"5min":0.379,"15min":0.33100000000000002},"max":{"1min":5.3739999999999997,"5min":6.0359999999999996,"15min":10.52},"last":1.0720000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.214,"15min":1.24},"min":{"1min":0.42499999999999999,"5min":0.29599999999999999,"15min":0.29499999999999998},"max":{"1min":5.274,"5min":6.359,"15min":9.1199999999999992},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.129,"15min":1.2150000000000001},"min":{"1min":0.33500000000000002,"5min":0.32800000000000001,"15min":0.32200000000000001},"max":{"1min":6.9279999999999999,"5min":8.0099999999999998,"15min":8.8879999999999999},"last":1.2709999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.3540000000000001,"15min":2.867},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.35699999999999998},"max":{"1min":5.4489999999999998,"5min":6.2590000000000003,"15min":501.892},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.177,"15min":2.7879999999999998},"min":{"1min":0.47899999999999998,"5min":0.38700000000000001,"15min":0.371},"max":{"1min":5.3150000000000004,"5min":5.3150000000000004,"15min":501.76600000000002},"last":1.125}]},{"osd":9,"last update":"Mon Feb 12 16:38:03 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.169,"15min":1.2250000000000001},"min":{"1min":0.48999999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.4029999999999996,"5min":6.319,"15min":6.319},"last":1.488},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.2689999999999999,"15min":1.2190000000000001},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":7.0119999999999996,"5min":7.0119999999999996,"15min":7.6870000000000003},"last":1.1419999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:38:03 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.3400000000000001,"15min":1.363},"min":{"1min":0.55300000000000005,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":5.3479999999999999,"5min":11.536,"15min":11.536},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.4019999999999999,"15min":1.4370000000000001},"min":{"1min":0.65600000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":5.4569999999999999,"5min":11.585000000000001,"15min":11.585000000000001},"last":1.3839999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:38:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.52,"15min":1.4450000000000001},"min":{"1min":0.63300000000000001,"5min":0.51200000000000001,"15min":0.49299999999999999},"max":{"1min":5.4980000000000002,"5min":6.2910000000000004,"15min":8.6649999999999991},"last":1.548},{"interface":"front","average":{"1min":1.446,"5min":1.117,"15min":1.1439999999999999},"min":{"1min":0.56599999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":7.04,"5min":7.04,"15min":7.04},"last":1.444}]}]},{"osd":1,"up_from":13,"seq":55834575416,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903136,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":902400,"kb_avail":92842016,"statfs":{"total":95995035648,"available":95070224384,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":924057600},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:37:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.23,"15min":1.0680000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22900000000000001},"max":{"1min":9.1999999999999993,"5min":10.420999999999999,"15min":16.632999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.748,"5min":1.1220000000000001,"15min":1.292},"min":{"1min":0.28799999999999998,"5min":0.254,"15min":0.254},"max":{"1min":2.492,"5min":10.26,"15min":10.26},"last":0.77100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.363,"15min":1.254},"min":{"1min":0.34599999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.9740000000000002,"5min":10.755000000000001,"15min":12.778},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.363,"15min":1.3},"min":{"1min":0.30099999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.9860000000000002,"5min":7.3600000000000003,"15min":9.9689999999999994},"last":0.90600000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:38:26 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.179,"15min":1.224},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.248},"max":{"1min":6.2779999999999996,"5min":7.3019999999999996,"15min":26.664000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.181,"5min":1.486,"15min":1.4379999999999999},"min":{"1min":0.34899999999999998,"5min":0.28899999999999998,"15min":0.26200000000000001},"max":{"1min":6.952,"5min":11.699999999999999,"15min":27.148},"last":1.103}]},{"osd":4,"last update":"Mon Feb 12 16:38:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.90000000000000002,"15min":1.1319999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.31},"max":{"1min":2.0099999999999998,"5min":4.6680000000000001,"15min":8.7479999999999993},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.006,"15min":1.214},"min":{"1min":0.38900000000000001,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":4.9379999999999997,"5min":4.9379999999999997,"15min":7.1710000000000003},"last":0.96699999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":1.107,"15min":4.1219999999999999},"min":{"1min":0.34300000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.819,"5min":8.4299999999999997,"15min":524.85299999999995},"last":1.331},{"interface":"front","average":{"1min":1.095,"5min":1.351,"15min":4.1550000000000002},"min":{"1min":0.39900000000000002,"5min":0.35499999999999998,"15min":0.32800000000000001},"max":{"1min":4.3090000000000002,"5min":8.9540000000000006,"15min":519.89700000000005},"last":1.1839999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:23:47 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.2689999999999999,"15min":1.6739999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.312},"max":{"1min":5.6760000000000002,"5min":5.6760000000000002,"15min":56.137},"last":1.294},{"interface":"front","average":{"1min":1.585,"5min":1.407,"15min":1.778},"min":{"1min":0.499,"5min":0.499,"15min":0.39000000000000001},"max":{"1min":5.7130000000000001,"5min":5.7130000000000001,"15min":56.195999999999998},"last":1.4419999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:37:58 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.1100000000000001,"15min":3.234},"min":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":3.569,"5min":6.7130000000000001,"15min":522.79200000000003},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.248,"15min":3.2000000000000002},"min":{"1min":0.45800000000000002,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":9.0069999999999997,"5min":9.0069999999999997,"15min":520.64499999999998},"last":1.2470000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:37:58 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.3340000000000001,"15min":3.3130000000000002},"min":{"1min":0.40300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":9.1430000000000007,"5min":9.1430000000000007,"15min":548.91600000000005},"last":1.516},{"interface":"front","average":{"1min":1.639,"5min":1.3999999999999999,"15min":3.3460000000000001},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.36599999999999999},"max":{"1min":9.1739999999999995,"5min":9.1739999999999995,"15min":552.42100000000005},"last":1.5820000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:38:09 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.284,"15min":1.2669999999999999},"min":{"1min":0.51800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":3.7069999999999999,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.3759999999999999,"15min":1.298},"min":{"1min":0.60999999999999999,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":3.742,"5min":8.4969999999999999,"15min":10.106},"last":1.629}]},{"osd":10,"last update":"Mon Feb 12 16:37:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.3260000000000001,"15min":3.6680000000000001},"min":{"1min":0.45400000000000001,"5min":0.42999999999999999,"15min":0.39500000000000002},"max":{"1min":4.3819999999999997,"5min":8.3469999999999995,"15min":548.93499999999995},"last":1.698},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.341,"15min":3.657},"min":{"1min":0.56699999999999995,"5min":0.47999999999999998,"15min":0.46999999999999997},"max":{"1min":9.0440000000000005,"5min":9.0440000000000005,"15min":549.12099999999998},"last":1.3600000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":2.3439999999999999,"15min":1.698},"min":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.47099999999999997},"max":{"1min":4.3959999999999999,"5min":88.679000000000002,"15min":88.679000000000002},"last":1.454},{"interface":"front","average":{"1min":1.2869999999999999,"5min":2.2989999999999999,"15min":1.694},"min":{"1min":0.46700000000000003,"5min":0.46100000000000002,"15min":0.44900000000000001},"max":{"1min":3.6699999999999999,"5min":88.519000000000005,"15min":88.519000000000005},"last":1.54}]}]},{"osd":0,"up_from":8,"seq":34359738948,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266464,"kb_used_data":371352,"kb_used_omap":0,"kb_used_meta":895104,"kb_avail":92478688,"statfs":{"total":95995035648,"available":94698176512,"internally_reserved":0,"allocated":380264448,"data_stored":379891481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":916586496},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:38:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.077,"15min":1.022},"min":{"1min":0.252,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.75,"5min":13.359,"15min":13.359},"last":1.181},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3089999999999999,"15min":1.2190000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.22600000000000001},"max":{"1min":4.3650000000000002,"5min":8.3780000000000001,"15min":8.3780000000000001},"last":1.018}]},{"osd":2,"last update":"Mon Feb 12 16:23:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.81399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1319999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.878,"15min":0.872},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.222,"5min":1.647,"15min":1.647},"last":0.98899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:38:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":4.71,"15min":2.391},"min":{"1min":0.26000000000000001,"5min":0.252,"15min":0.252},"max":{"1min":4.6509999999999998,"5min":377.06299999999999,"15min":377.06299999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.77,"5min":5.1020000000000003,"15min":2.5499999999999998},"min":{"1min":0.39300000000000002,"5min":0.23799999999999999,"15min":0.224},"max":{"1min":4.5460000000000003,"5min":379.18000000000001,"15min":379.18000000000001},"last":1.3100000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:38:12 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.96999999999999997,"15min":1.9219999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.794,"5min":7.2869999999999999,"15min":193.77000000000001},"last":1.296},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.466,"15min":2.016},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":4.4550000000000001,"5min":8.4949999999999992,"15min":189.625},"last":1.429}]},{"osd":5,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.101,"15min":1.8220000000000001},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.28699999999999998},"max":{"1min":1.6970000000000001,"5min":5.9240000000000004,"15min":181.46199999999999},"last":1.149},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.2490000000000001,"15min":1.901},"min":{"1min":0.45800000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":1.768,"5min":6.0880000000000001,"15min":186.41999999999999},"last":1.796}]},{"osd":6,"last update":"Mon Feb 12 16:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.2090000000000001,"15min":1.931},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.311},"max":{"1min":2.9889999999999999,"5min":7.2389999999999999,"15min":189.79400000000001},"last":1.617},{"interface":"front","average":{"1min":0.998,"5min":1.355,"15min":2.008},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.35499999999999998},"max":{"1min":3.9209999999999998,"5min":5.968,"15min":194.41200000000001},"last":1.7190000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:12 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.3620000000000001,"15min":2.7090000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.374},"max":{"1min":3.1139999999999999,"5min":6.1130000000000004,"15min":191.47200000000001},"last":1.458},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.51,"15min":2.819},"min":{"1min":0.54800000000000004,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":3.141,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.3460000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.4059999999999999,"15min":1.302},"min":{"1min":0.49299999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.806,"5min":6.056,"15min":7.3920000000000003},"last":1.9119999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.4990000000000001,"15min":1.3959999999999999},"min":{"1min":0.621,"5min":0.51200000000000001,"15min":0.48299999999999998},"max":{"1min":1.96,"5min":6.1589999999999998,"15min":7.5060000000000002},"last":1.9650000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:38:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.5840000000000001,"15min":1.2569999999999999},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.0540000000000003,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":1.593},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.577,"15min":1.3859999999999999},"min":{"1min":0.51700000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":12.943,"5min":12.943,"15min":12.943},"last":1.855}]},{"osd":10,"last update":"Mon Feb 12 16:38:12 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.21,"15min":1.284},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":3.8820000000000001,"5min":7.3369999999999997,"15min":7.3369999999999997},"last":1.504},{"interface":"front","average":{"1min":1.077,"5min":1.288,"15min":1.387},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48299999999999998},"max":{"1min":3.9100000000000001,"5min":7.415,"15min":9.3490000000000002},"last":1.7430000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.6759999999999999,"15min":1.482},"min":{"1min":0.50600000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.3410000000000002,"5min":8.4280000000000008,"15min":8.4280000000000008},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"min":{"1min":0.51100000000000001,"5min":0.47599999999999998,"15min":0.45200000000000001},"max":{"1min":2.7589999999999999,"5min":7.3799999999999999,"15min":7.4210000000000003},"last":1.641}]}]},{"osd":2,"up_from":21,"seq":90194313778,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310240,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93434912,"statfs":{"total":95995035648,"available":95677349888,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.873,"15min":1.7509999999999999},"min":{"1min":0.38900000000000001,"5min":0.29099999999999998,"15min":0.247},"max":{"1min":1.337,"5min":3.3410000000000002,"15min":177.68899999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.998,"15min":1.8700000000000001},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":1.212,"5min":6.2800000000000002,"15min":178.26900000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:37:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1379999999999999,"15min":1.0600000000000001},"min":{"1min":0.27200000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.4880000000000004,"5min":7.6050000000000004,"15min":7.6050000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":2.7909999999999999,"5min":1.6890000000000001,"15min":1.3080000000000001},"min":{"1min":0.28899999999999998,"5min":0.254,"15min":0.247},"max":{"1min":27.562999999999999,"5min":27.562999999999999,"15min":27.562999999999999},"last":1.0189999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.5640000000000001,"15min":1.5800000000000001},"min":{"1min":0.28999999999999998,"5min":0.28100000000000003,"15min":0.253},"max":{"1min":2.0649999999999999,"5min":8.1859999999999999,"15min":54.091000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.877,"5min":1.387,"15min":1.3979999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.25600000000000001},"max":{"1min":4.4550000000000001,"5min":9.1110000000000007,"15min":54.203000000000003},"last":0.82099999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:38:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.232,"15min":1.163},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28799999999999998},"max":{"1min":8.3439999999999994,"5min":8.3439999999999994,"15min":10.890000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.3460000000000001,"15min":1.268},"min":{"1min":0.34699999999999998,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":6.1699999999999999,"5min":7.0019999999999998,"15min":9.8079999999999998},"last":0.996}]},{"osd":5,"last update":"Mon Feb 12 16:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.0549999999999999,"15min":1.1200000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.1890000000000001,"5min":8.7509999999999994,"15min":8.7509999999999994},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.2509999999999999,"15min":1.3009999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.31},"max":{"1min":6.2220000000000004,"5min":7.0590000000000002,"15min":11.081},"last":0.90100000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.3240000000000001,"15min":1.3600000000000001},"min":{"1min":0.36599999999999999,"5min":0.32000000000000001,"15min":0.316},"max":{"1min":2.0390000000000001,"5min":8.4510000000000005,"15min":21.527000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.169,"15min":1.236},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":0.72599999999999998,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":0.97899999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:38:07 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.405,"15min":1.3380000000000001},"min":{"1min":0.41599999999999998,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":6.2560000000000002,"5min":7.1189999999999998,"15min":11.044},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.2709999999999999,"15min":1.2350000000000001},"min":{"1min":0.35499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.2709999999999999,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":0.91200000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:38:32 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.3759999999999999,"15min":1.2170000000000001},"min":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1920000000000002,"5min":8.3870000000000005,"15min":8.3870000000000005},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.3140000000000001,"15min":1.2250000000000001},"min":{"1min":0.48699999999999999,"5min":0.41199999999999998,"15min":0.377},"max":{"1min":1.7330000000000001,"5min":8.9000000000000004,"15min":9.7639999999999993},"last":1.1399999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:38:07 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":1.641,"15min":1.3819999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":7.7759999999999998,"5min":7.7759999999999998,"15min":10.972},"last":1.079},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.498,"15min":1.3779999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":6.29,"5min":7.2110000000000003,"15min":11.006},"last":1.236}]},{"osd":10,"last update":"Mon Feb 12 16:37:52 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.417,"15min":1.3520000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":8.4209999999999994,"5min":8.4209999999999994,"15min":11.454000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.4179999999999999,"15min":1.393},"min":{"1min":0.50600000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":8.4879999999999995,"5min":8.4879999999999995,"15min":9.7170000000000005},"last":1.321}]},{"osd":11,"last update":"Mon Feb 12 16:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.379,"15min":1.246},"min":{"1min":0.56000000000000005,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.1880000000000002,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":1.208},{"interface":"front","average":{"1min":1.423,"5min":1.5249999999999999,"15min":1.3600000000000001},"min":{"1min":0.56699999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":5.2640000000000002,"5min":8.968,"15min":8.968},"last":1.296}]}]},{"osd":3,"up_from":522,"seq":2241972928702,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345248,"kb_used_data":728,"kb_used_omap":91,"kb_used_meta":344420,"kb_avail":93399904,"statfs":{"total":95995035648,"available":95641501696,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":352686096},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:38:17 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.4370000000000001,"15min":1.2350000000000001},"min":{"1min":0.27500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":7.0940000000000003,"5min":10.101000000000001,"15min":34.055999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.667,"5min":1.3839999999999999,"15min":1.268},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.5819999999999999,"5min":9.3490000000000002,"15min":27.600000000000001},"last":1.369}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.4079999999999999,"15min":1.151},"min":{"1min":0.253,"5min":0.253,"15min":0.252},"max":{"1min":6.2329999999999997,"5min":8.8109999999999999,"15min":8.8109999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":2.254,"5min":1.786,"15min":1.268},"min":{"1min":0.32800000000000001,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":8.7810000000000006,"5min":10.369999999999999,"15min":10.369999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":1.401,"15min":1.1240000000000001},"min":{"1min":0.33000000000000002,"5min":0.316,"15min":0.27400000000000002},"max":{"1min":6.8220000000000001,"5min":6.8220000000000001,"15min":10.004},"last":1.181},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.1639999999999999,"15min":1.137},"min":{"1min":0.38300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.6840000000000002,"5min":6.6840000000000002,"15min":6.806},"last":1.0549999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.329,"15min":1.165},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.9420000000000002,"5min":11.884,"15min":11.884},"last":1.01},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.484,"15min":1.2310000000000001},"min":{"1min":0.376,"5min":0.312,"15min":0.29899999999999999},"max":{"1min":5.9279999999999999,"5min":12.164999999999999,"15min":12.164999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.26,"15min":1.208},"min":{"1min":0.32300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.9849999999999999,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.8440000000000001,"5min":1.581,"15min":1.272},"min":{"1min":0.41599999999999998,"5min":0.377,"15min":0.30499999999999999},"max":{"1min":6.9130000000000003,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":1.3280000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.5149999999999999,"15min":1.298},"min":{"1min":0.55000000000000004,"5min":0.36199999999999999,"15min":0.33300000000000002},"max":{"1min":7.9909999999999997,"5min":7.9909999999999997,"15min":7.9909999999999997},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.609,"5min":1.3859999999999999,"15min":1.335},"min":{"1min":0.621,"5min":0.371,"15min":0.371},"max":{"1min":6.0899999999999999,"5min":6.7590000000000003,"15min":7.2249999999999996},"last":1.413}]},{"osd":8,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.4950000000000001,"15min":1.2190000000000001},"min":{"1min":0.47899999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.3789999999999996,"5min":5.9669999999999996,"15min":5.9669999999999996},"last":1.288},{"interface":"front","average":{"1min":1.675,"5min":1.504,"15min":1.224},"min":{"1min":0.55800000000000005,"5min":0.42299999999999999,"15min":0.35599999999999998},"max":{"1min":6.056,"5min":7.9569999999999999,"15min":7.9569999999999999},"last":1.4710000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:38:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1280000000000001,"5min":1.6259999999999999,"15min":1.3779999999999999},"min":{"1min":0.51600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":8.9350000000000005,"5min":8.9350000000000005,"15min":8.9350000000000005},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.7390000000000001,"5min":1.502,"15min":1.3360000000000001},"min":{"1min":0.40899999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.8490000000000002,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":1.5069999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4199999999999999,"15min":1.2729999999999999},"min":{"1min":0.46200000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":4.2439999999999998,"5min":9.5649999999999995,"15min":9.5649999999999995},"last":1.137},{"interface":"front","average":{"1min":1.9530000000000001,"5min":1.675,"15min":1.4490000000000001},"min":{"1min":0.55800000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":10.032999999999999,"5min":10.032999999999999,"15min":15.140000000000001},"last":1.653}]},{"osd":11,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":2.077,"15min":1.542},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":4.0439999999999996,"5min":73.037999999999997,"15min":73.037999999999997},"last":1.0820000000000001},{"interface":"front","average":{"1min":2.0739999999999998,"5min":2.3679999999999999,"15min":1.631},"min":{"1min":0.63600000000000001,"5min":0.53200000000000003,"15min":0.49299999999999999},"max":{"1min":5.5330000000000004,"5min":73.057000000000002,"15min":73.057000000000002},"last":1.55}]}]},{"osd":4,"up_from":32,"seq":137438954017,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894880,"kb_used_data":728,"kb_used_omap":0,"kb_used_meta":894144,"kb_avail":92850272,"statfs":{"total":95995035648,"available":95078678528,"internally_reserved":0,"allocated":745472,"data_stored":372505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915603456},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:38:21 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.1160000000000001,"15min":1.119},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9670000000000001,"5min":4.1050000000000004,"15min":4.1050000000000004},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.212,"15min":1.2190000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":5.0330000000000004,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":1.401}]},{"osd":1,"last update":"Mon Feb 12 16:38:16 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.6870000000000001,"15min":1.272},"min":{"1min":0.27300000000000002,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":8.1699999999999999,"5min":56.188000000000002,"15min":56.188000000000002},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.8049999999999999,"15min":1.381},"min":{"1min":0.29399999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.9260000000000002,"5min":55.414000000000001,"15min":55.414000000000001},"last":1.29}]},{"osd":2,"last update":"Mon Feb 12 16:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.0009999999999999,"15min":1.044},"min":{"1min":0.307,"5min":0.307,"15min":0.24399999999999999},"max":{"1min":4.2850000000000001,"5min":4.9660000000000002,"15min":7.3630000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.0349999999999999,"15min":1.0860000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.254},"max":{"1min":3.1110000000000002,"5min":5.2069999999999999,"15min":7.633},"last":1.889}]},{"osd":3,"last update":"Mon Feb 12 16:38:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.1579999999999999,"15min":1.2529999999999999},"min":{"1min":0.32300000000000001,"5min":0.311,"15min":0.311},"max":{"1min":5.1340000000000003,"5min":5.1340000000000003,"15min":12.409000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.734,"5min":1.204,"15min":1.2090000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.25800000000000001},"max":{"1min":6.8099999999999996,"5min":6.8099999999999996,"15min":7.383},"last":0.998}]},{"osd":5,"last update":"Mon Feb 12 16:38:35 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":0.872,"15min":1.157},"min":{"1min":0.27200000000000002,"5min":0.252,"15min":0.252},"max":{"1min":7.8310000000000004,"5min":7.8310000000000004,"15min":10.289999999999999},"last":1.161},{"interface":"front","average":{"1min":1.645,"5min":0.98899999999999999,"15min":1.2869999999999999},"min":{"1min":0.34599999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":7.8620000000000001,"5min":7.8620000000000001,"15min":10.321999999999999},"last":1.323}]},{"osd":6,"last update":"Mon Feb 12 16:38:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":0.98199999999999998,"15min":1.073},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.26400000000000001},"max":{"1min":5.1059999999999999,"5min":5.7560000000000002,"15min":7.5289999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.722,"5min":1.087,"15min":1.22},"min":{"1min":0.35099999999999998,"5min":0.30599999999999999,"15min":0.28899999999999998},"max":{"1min":11.968999999999999,"5min":11.968999999999999,"15min":11.968999999999999},"last":1.4910000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.98099999999999998,"15min":1.075},"min":{"1min":0.35699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.6580000000000004,"5min":5.4160000000000004,"15min":6.8659999999999997},"last":1.272},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.0249999999999999,"15min":1.248},"min":{"1min":0.377,"5min":0.377,"15min":0.34599999999999997},"max":{"1min":8.8000000000000007,"5min":8.8000000000000007,"15min":9.1440000000000001},"last":1.7070000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:38:35 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.0589999999999999,"15min":1.147},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38300000000000001},"max":{"1min":5.1950000000000003,"5min":5.1950000000000003,"15min":7.6890000000000001},"last":1.611},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.1200000000000001,"15min":1.216},"min":{"1min":0.48199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.4859999999999998,"5min":7.9349999999999996,"15min":7.9349999999999996},"last":1.5549999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:38:18 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.732,"15min":1.3879999999999999},"min":{"1min":0.42699999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":5.2249999999999996,"5min":63.545000000000002,"15min":63.545000000000002},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.7490000000000001,"15min":1.4730000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.36699999999999999},"max":{"1min":5.5179999999999998,"5min":63.421999999999997,"15min":63.421999999999997},"last":1.954}]},{"osd":10,"last update":"Mon Feb 12 16:38:16 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.1599999999999999,"15min":1.3129999999999999},"min":{"1min":0.51000000000000001,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":5.548,"5min":5.548,"15min":7.7619999999999996},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.145,"15min":1.135},"min":{"1min":0.52300000000000002,"5min":0.41799999999999998,"15min":0.41199999999999998},"max":{"1min":3.7789999999999999,"5min":5.6459999999999999,"15min":5.6459999999999999},"last":1.583}]},{"osd":11,"last update":"Mon Feb 12 16:38:31 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.204,"15min":1.2490000000000001},"min":{"1min":0.54700000000000004,"5min":0.48999999999999999,"15min":0.45400000000000001},"max":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":7.734},"last":1.7450000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.119,"15min":1.3089999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.46400000000000002},"max":{"1min":5.2539999999999996,"5min":5.2539999999999996,"15min":7.7969999999999997},"last":1.879}]}]},{"osd":5,"up_from":39,"seq":167503725081,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598636,"kb_used_data":740,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":93146516,"statfs":{"total":95995035648,"available":95382032384,"internally_reserved":0,"allocated":757760,"data_stored":378389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":1.762,"15min":1.758},"min":{"1min":0.28999999999999998,"5min":0.245,"15min":0.245},"max":{"1min":3.1829999999999998,"5min":38.366,"15min":38.366},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.494,"5min":2.0670000000000002,"15min":1.8100000000000001},"min":{"1min":0.32100000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.2430000000000003,"5min":37.310000000000002,"15min":37.310000000000002},"last":0.76900000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.018,"15min":1.7569999999999999},"min":{"1min":0.34599999999999997,"5min":0.25,"15min":0.245},"max":{"1min":4.0999999999999996,"5min":4.4349999999999996,"15min":221.96600000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.198,"5min":1.135,"15min":1.8700000000000001},"min":{"1min":0.45700000000000002,"5min":0.36599999999999999,"15min":0.311},"max":{"1min":4.1500000000000004,"5min":4.5369999999999999,"15min":222.06200000000001},"last":1.2869999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.159,"15min":1.2649999999999999},"min":{"1min":0.40600000000000003,"5min":0.32600000000000001,"15min":0.28799999999999998},"max":{"1min":4.923,"5min":4.923,"15min":40.021999999999998},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.427,"5min":1.5129999999999999,"15min":1.3740000000000001},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.27600000000000002},"max":{"1min":5.5030000000000001,"5min":8.2439999999999998,"15min":40.225000000000001},"last":1.238}]},{"osd":3,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.3600000000000001,"15min":1.49},"min":{"1min":0.39600000000000002,"5min":0.34799999999999998,"15min":0.27200000000000002},"max":{"1min":4.3949999999999996,"5min":7.2300000000000004,"15min":22.484000000000002},"last":1.042},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.357,"15min":1.3149999999999999},"min":{"1min":0.41799999999999998,"5min":0.36299999999999999,"15min":0.29299999999999998},"max":{"1min":5.1719999999999997,"5min":5.5510000000000002,"15min":23.34},"last":0.82299999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:37:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.96899999999999997,"15min":1.1859999999999999},"min":{"1min":0.39300000000000002,"5min":0.312,"15min":0.23200000000000001},"max":{"1min":3.2949999999999999,"5min":4.3860000000000001,"15min":41.908000000000001},"last":1.476},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.1719999999999999,"15min":1.3169999999999999},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.27600000000000002},"max":{"1min":5.5389999999999997,"5min":5.5389999999999997,"15min":42.430999999999997},"last":1.143}]},{"osd":6,"last update":"Mon Feb 12 16:38:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.9770000000000001,"15min":1.4610000000000001},"min":{"1min":0.30499999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.8490000000000002,"5min":68.263999999999996,"15min":68.263999999999996},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.49,"5min":2.0409999999999999,"15min":1.5600000000000001},"min":{"1min":0.57999999999999996,"5min":0.35199999999999998,"15min":0.27100000000000002},"max":{"1min":4.8760000000000003,"5min":67.873000000000005,"15min":67.873000000000005},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.1659999999999999,"15min":1.167},"min":{"1min":0.372,"5min":0.372,"15min":0.28799999999999998},"max":{"1min":5.6609999999999996,"5min":9.4179999999999993,"15min":9.7710000000000008},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2,"15min":1.2609999999999999},"min":{"1min":0.52700000000000002,"5min":0.39800000000000002,"15min":0.34300000000000003},"max":{"1min":5.6120000000000001,"5min":5.6120000000000001,"15min":6.7599999999999998},"last":1.262}]},{"osd":8,"last update":"Mon Feb 12 16:32:45 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":2.3969999999999998,"15min":1.6140000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":6.7300000000000004,"5min":108.913,"15min":108.913},"last":1.8300000000000001},{"interface":"front","average":{"1min":1.169,"5min":2.1309999999999998,"15min":1.577},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1550000000000002,"5min":108.93600000000001,"15min":108.93600000000001},"last":1.8009999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:38:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1,"15min":1.274},"min":{"1min":0.50600000000000001,"5min":0.44,"15min":0.41899999999999998},"max":{"1min":3.5049999999999999,"5min":3.8730000000000002,"15min":6.9509999999999996},"last":1.569},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.1040000000000001,"15min":1.2869999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.375},"max":{"1min":2.9470000000000001,"5min":6.3390000000000004,"15min":9.6649999999999991},"last":1.4419999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:37:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3220000000000001,"15min":1.4450000000000001},"min":{"1min":0.60499999999999998,"5min":0.48599999999999999,"15min":0.41099999999999998},"max":{"1min":5.681,"5min":5.681,"15min":55.518999999999998},"last":1.5029999999999999},{"interface":"front","average":{"1min":0.997,"5min":1.1719999999999999,"15min":1.5169999999999999},"min":{"1min":0.59599999999999997,"5min":0.49199999999999999,"15min":0.47999999999999998},"max":{"1min":4.5069999999999997,"5min":4.6239999999999997,"15min":55.497999999999998},"last":1.7070000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:38:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.349,"15min":1.3560000000000001},"min":{"1min":0.63200000000000001,"5min":0.52000000000000002,"15min":0.38300000000000001},"max":{"1min":5.7329999999999997,"5min":5.7329999999999997,"15min":8.6940000000000008},"last":1.403},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.173,"15min":1.371},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.51500000000000001},"max":{"1min":3.532,"5min":4.5970000000000004,"15min":6.9029999999999996},"last":1.639}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:38:51.312 INFO:tasks.thrashosds.thrasher:Setting 1.1b to [10, 5] 2024-02-12T16:38:51.312 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.1b', '10', '5'] 2024-02-12T16:38:51.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pg-upmap 1.1b 10 5 2024-02-12T16:38:51.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:51.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:51.456 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:51.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:51.475 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:38:51.497 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:38:51.527 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:38:51.546 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:38:51.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:51.569 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:38:51.585 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:51.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:51.594 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:38:51.617 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:38:51.641 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:38:51.666 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:38:51.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:51.691 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:38:51.717 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:38:51.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:51.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:51.741 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:38:51.743 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.743 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.743 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:38:51.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:51.877 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:51.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:51.940 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:38:51.981 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:51.995 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:51 smithi123 bash[19119]: audit 2024-02-12T16:38:51.106854+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.121:0/266956682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:38:51.995 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:51 smithi123 bash[19119]: audit 2024-02-12T16:38:51.448686+0000 mon.a (mon.0) 2867 : audit [DBG] from='client.? v1:172.21.15.121:0/1099420479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:52.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:52.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:52.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:52.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:51 smithi121 bash[18489]: audit 2024-02-12T16:38:51.106854+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.121:0/266956682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:38:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:51 smithi121 bash[18489]: audit 2024-02-12T16:38:51.448686+0000 mon.a (mon.0) 2867 : audit [DBG] from='client.? v1:172.21.15.121:0/1099420479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:52.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:52.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:52.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:51 smithi169 bash[19604]: audit 2024-02-12T16:38:51.106854+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.121:0/266956682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:38:52.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:51 smithi169 bash[19604]: audit 2024-02-12T16:38:51.448686+0000 mon.a (mon.0) 2867 : audit [DBG] from='client.? v1:172.21.15.121:0/1099420479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:38:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:52.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:52.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:52.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:52.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:52.539 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:52.577 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:52.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:52.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:52.718 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:52.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:52.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:52.858 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:52.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:52.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:52.999 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:53.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:53.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:52 smithi169 bash[19604]: cluster 2024-02-12T16:38:51.918352+0000 mon.a (mon.0) 2868 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:53.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:52 smithi169 bash[19604]: audit 2024-02-12T16:38:51.930696+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? v1:172.21.15.121:0/266956682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:38:53.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:52 smithi169 bash[19604]: cluster 2024-02-12T16:38:51.930766+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e837: 12 total, 12 up, 12 in 2024-02-12T16:38:53.116 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:52 smithi169 bash[19604]: cluster 2024-02-12T16:38:52.179055+0000 mgr.y (mgr.14146) 1635 : cluster [DBG] pgmap v2344: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:53.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:52 smithi121 bash[18489]: cluster 2024-02-12T16:38:51.918352+0000 mon.a (mon.0) 2868 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:52 smithi121 bash[18489]: audit 2024-02-12T16:38:51.930696+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? v1:172.21.15.121:0/266956682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:38:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:52 smithi121 bash[18489]: cluster 2024-02-12T16:38:51.930766+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e837: 12 total, 12 up, 12 in 2024-02-12T16:38:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:52 smithi121 bash[18489]: cluster 2024-02-12T16:38:52.179055+0000 mgr.y (mgr.14146) 1635 : cluster [DBG] pgmap v2344: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:53.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:53.279 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:53.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:52 smithi123 bash[19119]: cluster 2024-02-12T16:38:51.918352+0000 mon.a (mon.0) 2868 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:38:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:52 smithi123 bash[19119]: audit 2024-02-12T16:38:51.930696+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? v1:172.21.15.121:0/266956682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:38:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:52 smithi123 bash[19119]: cluster 2024-02-12T16:38:51.930766+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e837: 12 total, 12 up, 12 in 2024-02-12T16:38:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:52 smithi123 bash[19119]: cluster 2024-02-12T16:38:52.179055+0000 mgr.y (mgr.14146) 1635 : cluster [DBG] pgmap v2344: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:53.421 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:53.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:53.564 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:53.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:53.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:53.846 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:53.947 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:52.490603+0000 osd.10 (osd.10) 182 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:38:53.947 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:52.534705+0000 osd.10 (osd.10) 183 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:38:53.947 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:52.861835+0000 osd.8 (osd.8) 107 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:38:53.947 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:52.952711+0000 osd.8 (osd.8) 108 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:38:53.947 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.073532+0000 osd.11 (osd.11) 197 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:38:53.947 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.119000+0000 osd.11 (osd.11) 198 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:38:53.947 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.651668+0000 osd.6 (osd.6) 164 : cluster [DBG] 1.f scrub starts 2024-02-12T16:38:53.948 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:53 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.766019+0000 osd.6 (osd.6) 165 : cluster [DBG] 1.f scrub ok 2024-02-12T16:38:53.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:53.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:53.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:54.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:54.108 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:54.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:52.490603+0000 osd.10 (osd.10) 182 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:52.534705+0000 osd.10 (osd.10) 183 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:52.861835+0000 osd.8 (osd.8) 107 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:52.952711+0000 osd.8 (osd.8) 108 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.073532+0000 osd.11 (osd.11) 197 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.119000+0000 osd.11 (osd.11) 198 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.651668+0000 osd.6 (osd.6) 164 : cluster [DBG] 1.f scrub starts 2024-02-12T16:38:54.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:53 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.766019+0000 osd.6 (osd.6) 165 : cluster [DBG] 1.f scrub ok 2024-02-12T16:38:54.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:54.226 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:52.490603+0000 osd.10 (osd.10) 182 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:38:54.226 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:52.534705+0000 osd.10 (osd.10) 183 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:38:54.226 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:52.861835+0000 osd.8 (osd.8) 107 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:38:54.226 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:52.952711+0000 osd.8 (osd.8) 108 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:38:54.226 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.073532+0000 osd.11 (osd.11) 197 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:38:54.227 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.119000+0000 osd.11 (osd.11) 198 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:38:54.227 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.651668+0000 osd.6 (osd.6) 164 : cluster [DBG] 1.f scrub starts 2024-02-12T16:38:54.227 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:53 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.766019+0000 osd.6 (osd.6) 165 : cluster [DBG] 1.f scrub ok 2024-02-12T16:38:54.249 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:54.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:54.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:54.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:54.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:54.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:54.534 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:54.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:54.638 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:54.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:54.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:54.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:54.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:54.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:54.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:54.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:54.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:55.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:55.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:55.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:55.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:55.200 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_6' created 2024-02-12T16:38:55.207 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:55 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.293563+0000 osd.8 (osd.8) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:55.207 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:55 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.293657+0000 osd.8 (osd.8) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:55.207 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:55 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.934737+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e838: 12 total, 12 up, 12 in 2024-02-12T16:38:55.207 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:55 smithi169 bash[19604]: cluster 2024-02-12T16:38:54.179805+0000 mgr.y (mgr.14146) 1636 : cluster [DBG] pgmap v2346: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:55.207 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:55 smithi169 bash[19604]: audit 2024-02-12T16:38:54.665126+0000 mon.a (mon.0) 2872 : audit [INF] from='client.? v1:172.21.15.121:0/929679918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-02-12T16:38:55.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:55.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:55.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:55.360 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:38:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:55.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:38:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:55 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.293563+0000 osd.8 (osd.8) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:55 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.293657+0000 osd.8 (osd.8) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:55 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.934737+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e838: 12 total, 12 up, 12 in 2024-02-12T16:38:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:55 smithi121 bash[18489]: cluster 2024-02-12T16:38:54.179805+0000 mgr.y (mgr.14146) 1636 : cluster [DBG] pgmap v2346: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:55.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:55 smithi121 bash[18489]: audit 2024-02-12T16:38:54.665126+0000 mon.a (mon.0) 2872 : audit [INF] from='client.? v1:172.21.15.121:0/929679918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-02-12T16:38:55.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:55.477 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:55 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.293563+0000 osd.8 (osd.8) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:38:55.478 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:55 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.293657+0000 osd.8 (osd.8) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:38:55.478 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:55 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.934737+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e838: 12 total, 12 up, 12 in 2024-02-12T16:38:55.478 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:55 smithi123 bash[19119]: cluster 2024-02-12T16:38:54.179805+0000 mgr.y (mgr.14146) 1636 : cluster [DBG] pgmap v2346: 31 pgs: 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:55.478 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:55 smithi123 bash[19119]: audit 2024-02-12T16:38:54.665126+0000 mon.a (mon.0) 2872 : audit [INF] from='client.? v1:172.21.15.121:0/929679918' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-02-12T16:38:55.501 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:55.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:55.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:55.643 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:55.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:55.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:38:55.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:55.782 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:38:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:55.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:38:55.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:55.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:55.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:55.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:38:56.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:56.055 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:56.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:56.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-02-12T16:38:56.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:56.196 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:56.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:56.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:56.337 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:56.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.825589+0000 osd.1 (osd.1) 131 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:38:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:53.994590+0000 osd.1 (osd.1) 132 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:38:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:54.495200+0000 osd.10 (osd.10) 184 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:38:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:54.548615+0000 osd.10 (osd.10) 185 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:38:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:54.648952+0000 osd.6 (osd.6) 166 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:38:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:54.755024+0000 osd.6 (osd.6) 167 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:38:56.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:54.832529+0000 osd.8 (osd.8) 111 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:38:56.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:54.835970+0000 osd.8 (osd.8) 112 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:38:56.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: audit 2024-02-12T16:38:55.192137+0000 mon.a (mon.0) 2873 : audit [INF] from='client.? v1:172.21.15.121:0/929679918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-02-12T16:38:56.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:56 smithi121 bash[18489]: cluster 2024-02-12T16:38:55.192279+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e839: 12 total, 12 up, 12 in 2024-02-12T16:38:56.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:56.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:56.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.825589+0000 osd.1 (osd.1) 131 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:38:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:53.994590+0000 osd.1 (osd.1) 132 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:38:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:54.495200+0000 osd.10 (osd.10) 184 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:38:56.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:54.548615+0000 osd.10 (osd.10) 185 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:38:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:54.648952+0000 osd.6 (osd.6) 166 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:38:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:54.755024+0000 osd.6 (osd.6) 167 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:38:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:54.832529+0000 osd.8 (osd.8) 111 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:38:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:54.835970+0000 osd.8 (osd.8) 112 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:38:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: audit 2024-02-12T16:38:55.192137+0000 mon.a (mon.0) 2873 : audit [INF] from='client.? v1:172.21.15.121:0/929679918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-02-12T16:38:56.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:56 smithi169 bash[19604]: cluster 2024-02-12T16:38:55.192279+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e839: 12 total, 12 up, 12 in 2024-02-12T16:38:56.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:56.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:56.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:56.627 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.825589+0000 osd.1 (osd.1) 131 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:38:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:53.994590+0000 osd.1 (osd.1) 132 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:38:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:54.495200+0000 osd.10 (osd.10) 184 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:38:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:54.548615+0000 osd.10 (osd.10) 185 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:38:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:54.648952+0000 osd.6 (osd.6) 166 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:38:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:54.755024+0000 osd.6 (osd.6) 167 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:38:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:54.832529+0000 osd.8 (osd.8) 111 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:38:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:54.835970+0000 osd.8 (osd.8) 112 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:38:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: audit 2024-02-12T16:38:55.192137+0000 mon.a (mon.0) 2873 : audit [INF] from='client.? v1:172.21.15.121:0/929679918' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-02-12T16:38:56.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:56 smithi123 bash[19119]: cluster 2024-02-12T16:38:55.192279+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e839: 12 total, 12 up, 12 in 2024-02-12T16:38:56.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:56.744 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:56.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:56.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:56.885 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:56.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:56.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:57.026 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:57.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:57.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:57.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:57.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:57.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:57.307 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:57.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:57 smithi121 bash[18489]: cluster 2024-02-12T16:38:55.862867+0000 osd.1 (osd.1) 133 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:38:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:57 smithi121 bash[18489]: cluster 2024-02-12T16:38:56.008233+0000 osd.1 (osd.1) 134 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:38:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:57 smithi121 bash[18489]: cluster 2024-02-12T16:38:56.180473+0000 mgr.y (mgr.14146) 1637 : cluster [DBG] pgmap v2348: 47 pgs: 16 unknown, 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:57 smithi121 bash[18489]: cluster 2024-02-12T16:38:56.195243+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e840: 12 total, 12 up, 12 in 2024-02-12T16:38:57.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:57.449 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:57.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:57 smithi169 bash[19604]: cluster 2024-02-12T16:38:55.862867+0000 osd.1 (osd.1) 133 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:38:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:57 smithi169 bash[19604]: cluster 2024-02-12T16:38:56.008233+0000 osd.1 (osd.1) 134 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:38:57.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:57 smithi169 bash[19604]: cluster 2024-02-12T16:38:56.180473+0000 mgr.y (mgr.14146) 1637 : cluster [DBG] pgmap v2348: 47 pgs: 16 unknown, 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:57.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:57 smithi169 bash[19604]: cluster 2024-02-12T16:38:56.195243+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e840: 12 total, 12 up, 12 in 2024-02-12T16:38:57.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:57.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:57 smithi123 bash[19119]: cluster 2024-02-12T16:38:55.862867+0000 osd.1 (osd.1) 133 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:38:57.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:57 smithi123 bash[19119]: cluster 2024-02-12T16:38:56.008233+0000 osd.1 (osd.1) 134 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:38:57.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:57 smithi123 bash[19119]: cluster 2024-02-12T16:38:56.180473+0000 mgr.y (mgr.14146) 1637 : cluster [DBG] pgmap v2348: 47 pgs: 16 unknown, 31 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:57.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:57 smithi123 bash[19119]: cluster 2024-02-12T16:38:56.195243+0000 mon.a (mon.0) 2875 : cluster [DBG] osdmap e840: 12 total, 12 up, 12 in 2024-02-12T16:38:57.588 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:57.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:57.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:57.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:57.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:57.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:57.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:57.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:57.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:57.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:38:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:58.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:38:58.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:58.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:38:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:58.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:38:58.242 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:58.283 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:38:58.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:58.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:38:58.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:58.402 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:58 smithi169 bash[19604]: cluster 2024-02-12T16:38:56.548881+0000 osd.4 (osd.4) 143 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:38:58.402 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:58 smithi169 bash[19604]: cluster 2024-02-12T16:38:56.627534+0000 osd.4 (osd.4) 144 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:38:58.403 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:58 smithi169 bash[19604]: cluster 2024-02-12T16:38:57.197836+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e841: 12 total, 12 up, 12 in 2024-02-12T16:38:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:58 smithi121 bash[18489]: cluster 2024-02-12T16:38:56.548881+0000 osd.4 (osd.4) 143 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:38:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:58 smithi121 bash[18489]: cluster 2024-02-12T16:38:56.627534+0000 osd.4 (osd.4) 144 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:38:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:58 smithi121 bash[18489]: cluster 2024-02-12T16:38:57.197836+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e841: 12 total, 12 up, 12 in 2024-02-12T16:38:58.429 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:38:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:58.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:38:58.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:58.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:58 smithi123 bash[19119]: cluster 2024-02-12T16:38:56.548881+0000 osd.4 (osd.4) 143 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:38:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:58 smithi123 bash[19119]: cluster 2024-02-12T16:38:56.627534+0000 osd.4 (osd.4) 144 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:38:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:58 smithi123 bash[19119]: cluster 2024-02-12T16:38:57.197836+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e841: 12 total, 12 up, 12 in 2024-02-12T16:38:58.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:58.714 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:58.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:58.819 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:58.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:38:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:58.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:38:58.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:58.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:38:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:38:59.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:38:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:38:59.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:59.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:59 smithi121 bash[18489]: cluster 2024-02-12T16:38:57.647212+0000 osd.3 (osd.3) 90 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:38:59.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:59 smithi121 bash[18489]: cluster 2024-02-12T16:38:57.758733+0000 osd.3 (osd.3) 91 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:38:59.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:59 smithi121 bash[18489]: cluster 2024-02-12T16:38:57.798457+0000 osd.1 (osd.1) 135 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:38:59.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:59 smithi121 bash[18489]: cluster 2024-02-12T16:38:57.918117+0000 osd.1 (osd.1) 136 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:38:59.259 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:38:59 smithi121 bash[18489]: cluster 2024-02-12T16:38:58.181044+0000 mgr.y (mgr.14146) 1638 : cluster [DBG] pgmap v2351: 47 pgs: 9 unknown, 38 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:38:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:38:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:59 smithi123 bash[19119]: cluster 2024-02-12T16:38:57.647212+0000 osd.3 (osd.3) 90 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:38:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:59 smithi123 bash[19119]: cluster 2024-02-12T16:38:57.758733+0000 osd.3 (osd.3) 91 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:38:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:59 smithi123 bash[19119]: cluster 2024-02-12T16:38:57.798457+0000 osd.1 (osd.1) 135 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:38:59.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:59 smithi123 bash[19119]: cluster 2024-02-12T16:38:57.918117+0000 osd.1 (osd.1) 136 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:38:59.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:38:59 smithi123 bash[19119]: cluster 2024-02-12T16:38:58.181044+0000 mgr.y (mgr.14146) 1638 : cluster [DBG] pgmap v2351: 47 pgs: 9 unknown, 38 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:59.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:59.401 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:38:59.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:59.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:59.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:59 smithi169 bash[19604]: cluster 2024-02-12T16:38:57.647212+0000 osd.3 (osd.3) 90 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:38:59.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:59 smithi169 bash[19604]: cluster 2024-02-12T16:38:57.758733+0000 osd.3 (osd.3) 91 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:38:59.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:59 smithi169 bash[19604]: cluster 2024-02-12T16:38:57.798457+0000 osd.1 (osd.1) 135 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:38:59.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:59 smithi169 bash[19604]: cluster 2024-02-12T16:38:57.918117+0000 osd.1 (osd.1) 136 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:38:59.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:38:59 smithi169 bash[19604]: cluster 2024-02-12T16:38:58.181044+0000 mgr.y (mgr.14146) 1638 : cluster [DBG] pgmap v2351: 47 pgs: 9 unknown, 38 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:38:59.565 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:38:59.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:59.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:59.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:59.847 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:38:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:38:59.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:59.990 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:38:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:38:59.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:00.095 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:00.124 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:00.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:00.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:00.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:00.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:00.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:00.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:00.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:00.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:00.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:00.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:00.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:00.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:00.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:00.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:00.840 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:00.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:00.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:00.979 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:00.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:01.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:01.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:01.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:01.223 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:01.260 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:01.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:01.275 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:39:01.275 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:39:01.275 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:39:01.275 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:39:01.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:01.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:01.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:01.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:01.509 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:01 smithi123 bash[19119]: cluster 2024-02-12T16:38:59.550601+0000 osd.4 (osd.4) 145 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:39:01.509 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:01 smithi123 bash[19119]: cluster 2024-02-12T16:38:59.614139+0000 osd.4 (osd.4) 146 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:39:01.509 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:01 smithi123 bash[19119]: cluster 2024-02-12T16:38:59.803210+0000 osd.2 (osd.2) 119 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:39:01.509 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:01 smithi123 bash[19119]: cluster 2024-02-12T16:38:59.895522+0000 osd.2 (osd.2) 120 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:39:01.509 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:01 smithi123 bash[19119]: cluster 2024-02-12T16:38:59.996443+0000 osd.7 (osd.7) 206 : cluster [DBG] 1.d scrub starts 2024-02-12T16:39:01.509 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:01 smithi123 bash[19119]: cluster 2024-02-12T16:39:00.113514+0000 osd.7 (osd.7) 207 : cluster [DBG] 1.d scrub ok 2024-02-12T16:39:01.509 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:01 smithi123 bash[19119]: cluster 2024-02-12T16:39:00.181999+0000 mgr.y (mgr.14146) 1639 : cluster [DBG] pgmap v2352: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:01.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:01 smithi169 bash[19604]: cluster 2024-02-12T16:38:59.550601+0000 osd.4 (osd.4) 145 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:39:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:01 smithi169 bash[19604]: cluster 2024-02-12T16:38:59.614139+0000 osd.4 (osd.4) 146 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:39:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:01 smithi169 bash[19604]: cluster 2024-02-12T16:38:59.803210+0000 osd.2 (osd.2) 119 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:39:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:01 smithi169 bash[19604]: cluster 2024-02-12T16:38:59.895522+0000 osd.2 (osd.2) 120 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:39:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:01 smithi169 bash[19604]: cluster 2024-02-12T16:38:59.996443+0000 osd.7 (osd.7) 206 : cluster [DBG] 1.d scrub starts 2024-02-12T16:39:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:01 smithi169 bash[19604]: cluster 2024-02-12T16:39:00.113514+0000 osd.7 (osd.7) 207 : cluster [DBG] 1.d scrub ok 2024-02-12T16:39:01.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:01 smithi169 bash[19604]: cluster 2024-02-12T16:39:00.181999+0000 mgr.y (mgr.14146) 1639 : cluster [DBG] pgmap v2352: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:01.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:01 smithi121 bash[18489]: cluster 2024-02-12T16:38:59.550601+0000 osd.4 (osd.4) 145 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:39:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:01 smithi121 bash[18489]: cluster 2024-02-12T16:38:59.614139+0000 osd.4 (osd.4) 146 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:39:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:01 smithi121 bash[18489]: cluster 2024-02-12T16:38:59.803210+0000 osd.2 (osd.2) 119 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:39:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:01 smithi121 bash[18489]: cluster 2024-02-12T16:38:59.895522+0000 osd.2 (osd.2) 120 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:39:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:01 smithi121 bash[18489]: cluster 2024-02-12T16:38:59.996443+0000 osd.7 (osd.7) 206 : cluster [DBG] 1.d scrub starts 2024-02-12T16:39:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:01 smithi121 bash[18489]: cluster 2024-02-12T16:39:00.113514+0000 osd.7 (osd.7) 207 : cluster [DBG] 1.d scrub ok 2024-02-12T16:39:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:01 smithi121 bash[18489]: cluster 2024-02-12T16:39:00.181999+0000 mgr.y (mgr.14146) 1639 : cluster [DBG] pgmap v2352: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:01.673 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:01.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:01.717 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:01.717 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:01.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:01.814 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:01.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:01.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:01.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:02.060 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:02.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:02.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:02.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:02.238 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:02.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:02.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:02.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:02 smithi123 bash[19119]: cluster 2024-02-12T16:39:00.573283+0000 osd.4 (osd.4) 147 : cluster [DBG] 1.e scrub starts 2024-02-12T16:39:02.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:02 smithi123 bash[19119]: cluster 2024-02-12T16:39:00.695508+0000 osd.4 (osd.4) 148 : cluster [DBG] 1.e scrub ok 2024-02-12T16:39:02.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:02 smithi123 bash[19119]: cluster 2024-02-12T16:39:01.200365+0000 osd.6 (osd.6) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:02.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:02 smithi123 bash[19119]: cluster 2024-02-12T16:39:01.200457+0000 osd.6 (osd.6) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:02.354 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:02 smithi123 bash[19119]: cluster 2024-02-12T16:39:01.883895+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e842: 12 total, 12 up, 12 in 2024-02-12T16:39:02.380 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:02.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:02.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:02.525 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:02.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:02 smithi169 bash[19604]: cluster 2024-02-12T16:39:00.573283+0000 osd.4 (osd.4) 147 : cluster [DBG] 1.e scrub starts 2024-02-12T16:39:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:02 smithi169 bash[19604]: cluster 2024-02-12T16:39:00.695508+0000 osd.4 (osd.4) 148 : cluster [DBG] 1.e scrub ok 2024-02-12T16:39:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:02 smithi169 bash[19604]: cluster 2024-02-12T16:39:01.200365+0000 osd.6 (osd.6) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:02 smithi169 bash[19604]: cluster 2024-02-12T16:39:01.200457+0000 osd.6 (osd.6) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:02 smithi169 bash[19604]: cluster 2024-02-12T16:39:01.883895+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e842: 12 total, 12 up, 12 in 2024-02-12T16:39:02.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:02.637 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:02 smithi121 bash[18489]: cluster 2024-02-12T16:39:00.573283+0000 osd.4 (osd.4) 147 : cluster [DBG] 1.e scrub starts 2024-02-12T16:39:02.637 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:02 smithi121 bash[18489]: cluster 2024-02-12T16:39:00.695508+0000 osd.4 (osd.4) 148 : cluster [DBG] 1.e scrub ok 2024-02-12T16:39:02.637 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:02 smithi121 bash[18489]: cluster 2024-02-12T16:39:01.200365+0000 osd.6 (osd.6) 168 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:02.638 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:02 smithi121 bash[18489]: cluster 2024-02-12T16:39:01.200457+0000 osd.6 (osd.6) 169 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:02.638 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:02 smithi121 bash[18489]: cluster 2024-02-12T16:39:01.883895+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e842: 12 total, 12 up, 12 in 2024-02-12T16:39:02.660 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:02.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:02.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:02.801 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:02.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:02.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:02.941 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:02.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:03.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:03.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:03.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:03.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:03.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:03.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:03.370 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:03.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:03 smithi123 bash[19119]: cluster 2024-02-12T16:39:02.182627+0000 mgr.y (mgr.14146) 1640 : cluster [DBG] pgmap v2354: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:03.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:03.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:03 smithi169 bash[19604]: cluster 2024-02-12T16:39:02.182627+0000 mgr.y (mgr.14146) 1640 : cluster [DBG] pgmap v2354: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:03.514 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:03.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:03.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:03 smithi121 bash[18489]: cluster 2024-02-12T16:39:02.182627+0000 mgr.y (mgr.14146) 1640 : cluster [DBG] pgmap v2354: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:03.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:39:03.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:03.695 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:03.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:03.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:03.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:03.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:03.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:03.972 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:03.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:04.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:04.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:04.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:04.246 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:04.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:04.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:04.387 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:04.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:04.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:04.504 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:04 smithi123 bash[19119]: cluster 2024-02-12T16:39:03.058138+0000 osd.7 (osd.7) 208 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:39:04.504 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:04 smithi123 bash[19119]: cluster 2024-02-12T16:39:03.155037+0000 osd.7 (osd.7) 209 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:39:04.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:04.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:04 smithi169 bash[19604]: cluster 2024-02-12T16:39:03.058138+0000 osd.7 (osd.7) 208 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:39:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:04 smithi169 bash[19604]: cluster 2024-02-12T16:39:03.155037+0000 osd.7 (osd.7) 209 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:39:04.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:04 smithi121 bash[18489]: cluster 2024-02-12T16:39:03.058138+0000 osd.7 (osd.7) 208 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:39:04.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:04 smithi121 bash[18489]: cluster 2024-02-12T16:39:03.155037+0000 osd.7 (osd.7) 209 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:39:04.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:04.671 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:04.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:04.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:04.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:04.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:04.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:04.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:04.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:05.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:05.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:05.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:05.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:05.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:05.236 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:05.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:05.339 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:05.377 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:05.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:05.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:05.493 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:05 smithi123 bash[19119]: cluster 2024-02-12T16:39:04.183462+0000 mgr.y (mgr.14146) 1641 : cluster [DBG] pgmap v2355: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:05.518 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:05.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:05 smithi169 bash[19604]: cluster 2024-02-12T16:39:04.183462+0000 mgr.y (mgr.14146) 1641 : cluster [DBG] pgmap v2355: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:05.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:05.634 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:05 smithi121 bash[18489]: cluster 2024-02-12T16:39:04.183462+0000 mgr.y (mgr.14146) 1641 : cluster [DBG] pgmap v2355: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:05.658 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:05.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:05.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:05.803 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:05.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:05.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:05.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:06.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:06.087 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:06.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:06.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:06.229 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:06.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:06.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:06.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:06.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:06.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:06.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:06.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:06.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:06.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:06.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:06.745 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:06.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:06.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:06.886 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:06.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:06.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:07.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:07.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:07.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:06 smithi123 bash[19119]: cluster 2024-02-12T16:39:05.874742+0000 mon.a (mon.0) 2878 : cluster [DBG] osdmap e843: 12 total, 12 up, 12 in 2024-02-12T16:39:07.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:06 smithi123 bash[19119]: cluster 2024-02-12T16:39:06.184120+0000 mgr.y (mgr.14146) 1642 : cluster [DBG] pgmap v2357: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:06 smithi121 bash[18489]: cluster 2024-02-12T16:39:05.874742+0000 mon.a (mon.0) 2878 : cluster [DBG] osdmap e843: 12 total, 12 up, 12 in 2024-02-12T16:39:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:06 smithi121 bash[18489]: cluster 2024-02-12T16:39:06.184120+0000 mgr.y (mgr.14146) 1642 : cluster [DBG] pgmap v2357: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:07.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:07.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:07.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:06 smithi169 bash[19604]: cluster 2024-02-12T16:39:05.874742+0000 mon.a (mon.0) 2878 : cluster [DBG] osdmap e843: 12 total, 12 up, 12 in 2024-02-12T16:39:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:06 smithi169 bash[19604]: cluster 2024-02-12T16:39:06.184120+0000 mgr.y (mgr.14146) 1642 : cluster [DBG] pgmap v2357: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:07.303 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:07.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:07.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:07.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:07.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:07.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:07.587 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:07.621 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:07.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:07.725 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:07.761 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:07.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:07.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:07.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:07.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:07.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:07.910 INFO:teuthology.orchestra.run.smithi121.stderr:set 1.1b pg_upmap mapping to [10,5] 2024-02-12T16:39:08.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:08.046 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:08.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:08.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:07 smithi123 bash[19119]: cluster 2024-02-12T16:39:06.288462+0000 osd.9 (osd.9) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:07 smithi123 bash[19119]: cluster 2024-02-12T16:39:06.288548+0000 osd.9 (osd.9) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:07 smithi123 bash[19119]: cluster 2024-02-12T16:39:06.889618+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e844: 12 total, 12 up, 12 in 2024-02-12T16:39:08.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:07 smithi123 bash[19119]: audit 2024-02-12T16:39:07.164009+0000 mon.a (mon.0) 2880 : audit [INF] from='client.? v1:172.21.15.121:0/549713495' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1b", "id": [10, 5]}]: dispatch 2024-02-12T16:39:08.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:08.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:07 smithi121 bash[18489]: cluster 2024-02-12T16:39:06.288462+0000 osd.9 (osd.9) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:07 smithi121 bash[18489]: cluster 2024-02-12T16:39:06.288548+0000 osd.9 (osd.9) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:07 smithi121 bash[18489]: cluster 2024-02-12T16:39:06.889618+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e844: 12 total, 12 up, 12 in 2024-02-12T16:39:08.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:07 smithi121 bash[18489]: audit 2024-02-12T16:39:07.164009+0000 mon.a (mon.0) 2880 : audit [INF] from='client.? v1:172.21.15.121:0/549713495' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1b", "id": [10, 5]}]: dispatch 2024-02-12T16:39:08.181 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:08.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:08.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:07 smithi169 bash[19604]: cluster 2024-02-12T16:39:06.288462+0000 osd.9 (osd.9) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:07 smithi169 bash[19604]: cluster 2024-02-12T16:39:06.288548+0000 osd.9 (osd.9) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:07 smithi169 bash[19604]: cluster 2024-02-12T16:39:06.889618+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e844: 12 total, 12 up, 12 in 2024-02-12T16:39:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:07 smithi169 bash[19604]: audit 2024-02-12T16:39:07.164009+0000 mon.a (mon.0) 2880 : audit [INF] from='client.? v1:172.21.15.121:0/549713495' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1b", "id": [10, 5]}]: dispatch 2024-02-12T16:39:08.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:08.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:08.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:08.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:08.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:08.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:08.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:08.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:08.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:08.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:08.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:08.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:08.878 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:08.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:08.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:09.014 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:09.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:09.156 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:07.427998+0000 osd.5 (osd.5) 179 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:07.505024+0000 osd.5 (osd.5) 180 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:07.692245+0000 osd.9 (osd.9) 149 : cluster [DBG] 1.a scrub starts 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:07.695955+0000 osd.9 (osd.9) 150 : cluster [DBG] 1.a scrub ok 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: audit 2024-02-12T16:39:07.900763+0000 mon.a (mon.0) 2881 : audit [INF] from='client.? v1:172.21.15.121:0/549713495' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1b", "id": [10, 5]}]': finished 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:07.900870+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e845: 12 total, 12 up, 12 in 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.014295+0000 osd.2 (osd.2) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.014392+0000 osd.2 (osd.2) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:08 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.184770+0000 mgr.y (mgr.14146) 1643 : cluster [DBG] pgmap v2360: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:09.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:09.297 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:07.427998+0000 osd.5 (osd.5) 179 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:39:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:07.505024+0000 osd.5 (osd.5) 180 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:39:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:07.692245+0000 osd.9 (osd.9) 149 : cluster [DBG] 1.a scrub starts 2024-02-12T16:39:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:07.695955+0000 osd.9 (osd.9) 150 : cluster [DBG] 1.a scrub ok 2024-02-12T16:39:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: audit 2024-02-12T16:39:07.900763+0000 mon.a (mon.0) 2881 : audit [INF] from='client.? v1:172.21.15.121:0/549713495' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1b", "id": [10, 5]}]': finished 2024-02-12T16:39:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:07.900870+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e845: 12 total, 12 up, 12 in 2024-02-12T16:39:09.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.014295+0000 osd.2 (osd.2) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:09.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.014392+0000 osd.2 (osd.2) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:09.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:08 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.184770+0000 mgr.y (mgr.14146) 1643 : cluster [DBG] pgmap v2360: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:07.427998+0000 osd.5 (osd.5) 179 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:07.505024+0000 osd.5 (osd.5) 180 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:07.692245+0000 osd.9 (osd.9) 149 : cluster [DBG] 1.a scrub starts 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:07.695955+0000 osd.9 (osd.9) 150 : cluster [DBG] 1.a scrub ok 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: audit 2024-02-12T16:39:07.900763+0000 mon.a (mon.0) 2881 : audit [INF] from='client.? v1:172.21.15.121:0/549713495' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1b", "id": [10, 5]}]': finished 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:07.900870+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e845: 12 total, 12 up, 12 in 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.014295+0000 osd.2 (osd.2) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.014392+0000 osd.2 (osd.2) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:09.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:08 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.184770+0000 mgr.y (mgr.14146) 1643 : cluster [DBG] pgmap v2360: 47 pgs: 47 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:09.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:09.435 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:09.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:09.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:09.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:09.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:09.851 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:09.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:09.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:10.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:10.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:09 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.331454+0000 osd.3 (osd.3) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:10.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:09 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.331550+0000 osd.3 (osd.3) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:10.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:09 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.655550+0000 osd.3 (osd.3) 94 : cluster [DBG] 1.c scrub starts 2024-02-12T16:39:10.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:09 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.746902+0000 osd.3 (osd.3) 95 : cluster [DBG] 1.c scrub ok 2024-02-12T16:39:10.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:09 smithi123 bash[19119]: cluster 2024-02-12T16:39:08.912353+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e846: 12 total, 12 up, 12 in 2024-02-12T16:39:10.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:09 smithi123 bash[19119]: cluster 2024-02-12T16:39:09.095598+0000 osd.7 (osd.7) 210 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:39:10.112 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:09 smithi123 bash[19119]: cluster 2024-02-12T16:39:09.159795+0000 osd.7 (osd.7) 211 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:39:10.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:09 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.331454+0000 osd.3 (osd.3) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:10.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:09 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.331550+0000 osd.3 (osd.3) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:10.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:09 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.655550+0000 osd.3 (osd.3) 94 : cluster [DBG] 1.c scrub starts 2024-02-12T16:39:10.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:09 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.746902+0000 osd.3 (osd.3) 95 : cluster [DBG] 1.c scrub ok 2024-02-12T16:39:10.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:09 smithi121 bash[18489]: cluster 2024-02-12T16:39:08.912353+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e846: 12 total, 12 up, 12 in 2024-02-12T16:39:10.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:09 smithi121 bash[18489]: cluster 2024-02-12T16:39:09.095598+0000 osd.7 (osd.7) 210 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:39:10.134 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:09 smithi121 bash[18489]: cluster 2024-02-12T16:39:09.159795+0000 osd.7 (osd.7) 211 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:39:10.138 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:10.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:10.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:10.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:10.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:10.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:09 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.331454+0000 osd.3 (osd.3) 92 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:10.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:09 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.331550+0000 osd.3 (osd.3) 93 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:10.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:09 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.655550+0000 osd.3 (osd.3) 94 : cluster [DBG] 1.c scrub starts 2024-02-12T16:39:10.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:09 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.746902+0000 osd.3 (osd.3) 95 : cluster [DBG] 1.c scrub ok 2024-02-12T16:39:10.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:09 smithi169 bash[19604]: cluster 2024-02-12T16:39:08.912353+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e846: 12 total, 12 up, 12 in 2024-02-12T16:39:10.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:09 smithi169 bash[19604]: cluster 2024-02-12T16:39:09.095598+0000 osd.7 (osd.7) 210 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:39:10.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:09 smithi169 bash[19604]: cluster 2024-02-12T16:39:09.159795+0000 osd.7 (osd.7) 211 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:39:10.300 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:39:10.324 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:39:10.348 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:39:10.372 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:39:10.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:10.402 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:39:10.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:10.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:10.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:10.426 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:39:10.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:39:10.476 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:39:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:39:10.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:39:10.549 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:39:10.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:10.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:10.573 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:39:10.575 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.575 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.575 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.575 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.576 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:39:10.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:10.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:10.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:10.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:10.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:10.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:10.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:10.924 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:10 smithi169 bash[19604]: cluster 2024-02-12T16:39:09.434428+0000 osd.5 (osd.5) 181 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:39:10.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:10 smithi169 bash[19604]: cluster 2024-02-12T16:39:09.548901+0000 osd.5 (osd.5) 182 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:39:10.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:10 smithi169 bash[19604]: cluster 2024-02-12T16:39:09.866155+0000 osd.11 (osd.11) 199 : cluster [DBG] 1.b scrub starts 2024-02-12T16:39:10.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:10 smithi169 bash[19604]: cluster 2024-02-12T16:39:10.006232+0000 osd.11 (osd.11) 200 : cluster [DBG] 1.b scrub ok 2024-02-12T16:39:10.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:10 smithi169 bash[19604]: cluster 2024-02-12T16:39:10.185441+0000 mgr.y (mgr.14146) 1644 : cluster [DBG] pgmap v2362: 47 pgs: 1 peering, 46 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:10.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:10 smithi169 bash[19604]: audit 2024-02-12T16:39:10.274694+0000 mon.a (mon.0) 2884 : audit [DBG] from='client.? v1:172.21.15.121:0/40580458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:10.925 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:10 smithi169 bash[19604]: cluster 2024-02-12T16:39:10.877515+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e847: 12 total, 12 up, 12 in 2024-02-12T16:39:10.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:10.981 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:10.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:11.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:11.113 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:11.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:11.219 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:11.257 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:11.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:11.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:10 smithi121 bash[18489]: cluster 2024-02-12T16:39:09.434428+0000 osd.5 (osd.5) 181 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:39:11.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:10 smithi121 bash[18489]: cluster 2024-02-12T16:39:09.548901+0000 osd.5 (osd.5) 182 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:39:11.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:10 smithi121 bash[18489]: cluster 2024-02-12T16:39:09.866155+0000 osd.11 (osd.11) 199 : cluster [DBG] 1.b scrub starts 2024-02-12T16:39:11.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:10 smithi121 bash[18489]: cluster 2024-02-12T16:39:10.006232+0000 osd.11 (osd.11) 200 : cluster [DBG] 1.b scrub ok 2024-02-12T16:39:11.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:10 smithi121 bash[18489]: cluster 2024-02-12T16:39:10.185441+0000 mgr.y (mgr.14146) 1644 : cluster [DBG] pgmap v2362: 47 pgs: 1 peering, 46 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:11.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:10 smithi121 bash[18489]: audit 2024-02-12T16:39:10.274694+0000 mon.a (mon.0) 2884 : audit [DBG] from='client.? v1:172.21.15.121:0/40580458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:11.265 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:10 smithi121 bash[18489]: cluster 2024-02-12T16:39:10.877515+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e847: 12 total, 12 up, 12 in 2024-02-12T16:39:11.362 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:11.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:10 smithi123 bash[19119]: cluster 2024-02-12T16:39:09.434428+0000 osd.5 (osd.5) 181 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:39:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:10 smithi123 bash[19119]: cluster 2024-02-12T16:39:09.548901+0000 osd.5 (osd.5) 182 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:39:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:10 smithi123 bash[19119]: cluster 2024-02-12T16:39:09.866155+0000 osd.11 (osd.11) 199 : cluster [DBG] 1.b scrub starts 2024-02-12T16:39:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:10 smithi123 bash[19119]: cluster 2024-02-12T16:39:10.006232+0000 osd.11 (osd.11) 200 : cluster [DBG] 1.b scrub ok 2024-02-12T16:39:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:10 smithi123 bash[19119]: cluster 2024-02-12T16:39:10.185441+0000 mgr.y (mgr.14146) 1644 : cluster [DBG] pgmap v2362: 47 pgs: 1 peering, 46 active+clean; 0 B data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:10 smithi123 bash[19119]: audit 2024-02-12T16:39:10.274694+0000 mon.a (mon.0) 2884 : audit [DBG] from='client.? v1:172.21.15.121:0/40580458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:11.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:10 smithi123 bash[19119]: cluster 2024-02-12T16:39:10.877515+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e847: 12 total, 12 up, 12 in 2024-02-12T16:39:11.399 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:11.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:11.503 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:11.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:11.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:11.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:11.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:11.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:11.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:11.810 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:11.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:11.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:11.945 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-02-12T16:39:11.946 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:39:11.956 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:11.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:12.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:12.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:12.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:11 smithi123 bash[19119]: cluster 2024-02-12T16:39:11.092825+0000 osd.7 (osd.7) 212 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:39:12.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:11 smithi123 bash[19119]: cluster 2024-02-12T16:39:11.207551+0000 osd.7 (osd.7) 213 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:39:12.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:11 smithi123 bash[19119]: audit 2024-02-12T16:39:11.691897+0000 mon.a (mon.0) 2886 : audit [INF] from='client.26846 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:39:12.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:11 smithi123 bash[19119]: audit 2024-02-12T16:39:11.693442+0000 mon.b (mon.2) 203 : audit [INF] from='client.? v1:172.21.15.121:0/4277854890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:39:12.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:11 smithi123 bash[19119]: audit 2024-02-12T16:39:11.695853+0000 mon.a (mon.0) 2887 : audit [INF] from='client.? v1:172.21.15.121:0/798994946' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:12.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:12.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:11 smithi121 bash[18489]: cluster 2024-02-12T16:39:11.092825+0000 osd.7 (osd.7) 212 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:39:12.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:11 smithi121 bash[18489]: cluster 2024-02-12T16:39:11.207551+0000 osd.7 (osd.7) 213 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:39:12.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:11 smithi121 bash[18489]: audit 2024-02-12T16:39:11.691897+0000 mon.a (mon.0) 2886 : audit [INF] from='client.26846 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:39:12.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:11 smithi121 bash[18489]: audit 2024-02-12T16:39:11.693442+0000 mon.b (mon.2) 203 : audit [INF] from='client.? v1:172.21.15.121:0/4277854890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:39:12.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:11 smithi121 bash[18489]: audit 2024-02-12T16:39:11.695853+0000 mon.a (mon.0) 2887 : audit [INF] from='client.? v1:172.21.15.121:0/798994946' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:12.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:12.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:11 smithi169 bash[19604]: cluster 2024-02-12T16:39:11.092825+0000 osd.7 (osd.7) 212 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:39:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:11 smithi169 bash[19604]: cluster 2024-02-12T16:39:11.207551+0000 osd.7 (osd.7) 213 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:39:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:11 smithi169 bash[19604]: audit 2024-02-12T16:39:11.691897+0000 mon.a (mon.0) 2886 : audit [INF] from='client.26846 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:39:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:11 smithi169 bash[19604]: audit 2024-02-12T16:39:11.693442+0000 mon.b (mon.2) 203 : audit [INF] from='client.? v1:172.21.15.121:0/4277854890' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:39:12.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:11 smithi169 bash[19604]: audit 2024-02-12T16:39:11.695853+0000 mon.a (mon.0) 2887 : audit [INF] from='client.? v1:172.21.15.121:0/798994946' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:12.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:12.370 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:12.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:12.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:12.509 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:12.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:12.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:12.650 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:12.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:12.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:12.789 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:12.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:12.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:12.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:12.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:13.040 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:13.070 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:13.076 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: cluster 2024-02-12T16:39:11.919087+0000 mon.a (mon.0) 2888 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:39:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: cluster 2024-02-12T16:39:11.919145+0000 mon.a (mon.0) 2889 : cluster [INF] Cluster is now healthy 2024-02-12T16:39:13.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: audit 2024-02-12T16:39:11.934908+0000 mon.a (mon.0) 2890 : audit [INF] from='client.26846 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:39:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: audit 2024-02-12T16:39:11.935117+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? v1:172.21.15.121:0/798994946' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:39:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: cluster 2024-02-12T16:39:11.935209+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e848: 12 total, 12 up, 12 in 2024-02-12T16:39:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: cluster 2024-02-12T16:39:12.083029+0000 osd.7 (osd.7) 214 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:39:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: cluster 2024-02-12T16:39:12.128719+0000 osd.7 (osd.7) 215 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:39:13.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:12 smithi123 bash[19119]: cluster 2024-02-12T16:39:12.186008+0000 mgr.y (mgr.14146) 1645 : cluster [DBG] pgmap v2365: 47 pgs: 1 peering, 46 active+clean; 0 B data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:13.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:13.215 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:13.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: cluster 2024-02-12T16:39:11.919087+0000 mon.a (mon.0) 2888 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: cluster 2024-02-12T16:39:11.919145+0000 mon.a (mon.0) 2889 : cluster [INF] Cluster is now healthy 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: audit 2024-02-12T16:39:11.934908+0000 mon.a (mon.0) 2890 : audit [INF] from='client.26846 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: audit 2024-02-12T16:39:11.935117+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? v1:172.21.15.121:0/798994946' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: cluster 2024-02-12T16:39:11.935209+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e848: 12 total, 12 up, 12 in 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: cluster 2024-02-12T16:39:12.083029+0000 osd.7 (osd.7) 214 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: cluster 2024-02-12T16:39:12.128719+0000 osd.7 (osd.7) 215 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:39:13.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:12 smithi121 bash[18489]: cluster 2024-02-12T16:39:12.186008+0000 mgr.y (mgr.14146) 1645 : cluster [DBG] pgmap v2365: 47 pgs: 1 peering, 46 active+clean; 0 B data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: cluster 2024-02-12T16:39:11.919087+0000 mon.a (mon.0) 2888 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:39:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: cluster 2024-02-12T16:39:11.919145+0000 mon.a (mon.0) 2889 : cluster [INF] Cluster is now healthy 2024-02-12T16:39:13.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: audit 2024-02-12T16:39:11.934908+0000 mon.a (mon.0) 2890 : audit [INF] from='client.26846 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:39:13.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: audit 2024-02-12T16:39:11.935117+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? v1:172.21.15.121:0/798994946' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:39:13.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: cluster 2024-02-12T16:39:11.935209+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e848: 12 total, 12 up, 12 in 2024-02-12T16:39:13.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: cluster 2024-02-12T16:39:12.083029+0000 osd.7 (osd.7) 214 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:39:13.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: cluster 2024-02-12T16:39:12.128719+0000 osd.7 (osd.7) 215 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:39:13.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:12 smithi169 bash[19604]: cluster 2024-02-12T16:39:12.186008+0000 mgr.y (mgr.14146) 1645 : cluster [DBG] pgmap v2365: 47 pgs: 1 peering, 46 active+clean; 0 B data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:13.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:13.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:13.459 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:13.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:13.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:13.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:39:13.639 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:13.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:13.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:39:13.728 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:13.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:13.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:13.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:13.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:13.909 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:13.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:14.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:14.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:14.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:14.177 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-02-12T16:39:14.180 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:39:14.180 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:39:14.190 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:14.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:14.302 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:39:14.302 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:39:14.302 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_115677 2024-02-12T16:39:14.303 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:39:14.304 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:39:14.330 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:14.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:14.464 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:14.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:14.567 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:14.704 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:14.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:14.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:14.857 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:14.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:14.984 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:14.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:15.087 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:15.125 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:15.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:15.136 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-02-12T16:39:15.136 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:39:15.136 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-02-12T16:39:15.136 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-12T16:39:15.136 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.7 2024-02-12T16:39:15.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:15.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:39:15.249 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:39:15 smithi123 systemd[1]: Stopping Ceph osd.7 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:39:15.249 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:39:15 smithi123 bash[113661]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:39:15.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:15.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:15.304 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 16 5514 5498 343.609 343.625 0.00214899 0.00290173 2024-02-12T16:39:15.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:15.375 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:15 smithi169 bash[19604]: cluster 2024-02-12T16:39:14.186799+0000 mgr.y (mgr.14146) 1646 : cluster [DBG] pgmap v2366: 47 pgs: 47 active+clean; 0 B data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:15.396 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:15.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:15.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:15 smithi121 bash[18489]: cluster 2024-02-12T16:39:14.186799+0000 mgr.y (mgr.14146) 1646 : cluster [DBG] pgmap v2366: 47 pgs: 47 active+clean; 0 B data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:15.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:15.507 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:15 smithi123 bash[19119]: cluster 2024-02-12T16:39:14.186799+0000 mgr.y (mgr.14146) 1646 : cluster [DBG] pgmap v2366: 47 pgs: 47 active+clean; 0 B data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:15.507 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:39:15 smithi123 bash[37755]: debug 2024-02-12T16:39:15.336+0000 7f860fc46700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:39:15.507 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:39:15 smithi123 bash[37755]: debug 2024-02-12T16:39:15.336+0000 7f860fc46700 -1 osd.7 848 *** Got signal Terminated *** 2024-02-12T16:39:15.507 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:39:15 smithi123 bash[37755]: debug 2024-02-12T16:39:15.336+0000 7f860fc46700 -1 osd.7 848 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:39:15.522 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:15.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:15.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:15.653 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:15.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:15.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:15.792 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:15.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:15.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:15.920 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:15.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:16.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:16.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:16.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:16.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:16.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:16.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:16.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:16 smithi169 bash[19604]: cluster 2024-02-12T16:39:15.340289+0000 mon.a (mon.0) 2893 : cluster [INF] osd.7 marked itself down and dead 2024-02-12T16:39:16.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:16.304 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 16 9947 9931 310.314 277.062 0.00162062 0.00320448 2024-02-12T16:39:16.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:16.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:16.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:16 smithi121 bash[18489]: cluster 2024-02-12T16:39:15.340289+0000 mon.a (mon.0) 2893 : cluster [INF] osd.7 marked itself down and dead 2024-02-12T16:39:16.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:16.451 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:16.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:16.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:16.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:16.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:16.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:16 smithi123 bash[19119]: cluster 2024-02-12T16:39:15.340289+0000 mon.a (mon.0) 2893 : cluster [INF] osd.7 marked itself down and dead 2024-02-12T16:39:16.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:16.732 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:16.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:16.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:16.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:16.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:16.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:17.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:17.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:17.167 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:17.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:17.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:17.304 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 10115 10099 210.373 10.5 0.00153764 0.00317964 2024-02-12T16:39:17.398 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:17.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[19119]: cluster 2024-02-12T16:39:15.430135+0000 osd.5 (osd.5) 183 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:39:17.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[19119]: cluster 2024-02-12T16:39:15.465208+0000 osd.5 (osd.5) 184 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:39:17.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[19119]: cluster 2024-02-12T16:39:15.943740+0000 osd.11 (osd.11) 201 : cluster [DBG] 1.4 deep-scrub starts 2024-02-12T16:39:17.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[19119]: cluster 2024-02-12T16:39:15.947500+0000 osd.11 (osd.11) 202 : cluster [DBG] 1.4 deep-scrub ok 2024-02-12T16:39:17.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[19119]: cluster 2024-02-12T16:39:16.187277+0000 mgr.y (mgr.14146) 1647 : cluster [DBG] pgmap v2367: 47 pgs: 47 active+clean; 76 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:17.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[19119]: cluster 2024-02-12T16:39:16.242302+0000 mon.a (mon.0) 2894 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T16:39:17.407 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[19119]: cluster 2024-02-12T16:39:16.250353+0000 mon.a (mon.0) 2895 : cluster [DBG] osdmap e849: 12 total, 11 up, 12 in 2024-02-12T16:39:17.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:17.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:17.541 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:17 smithi121 bash[18489]: cluster 2024-02-12T16:39:15.430135+0000 osd.5 (osd.5) 183 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:39:17.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:17 smithi121 bash[18489]: cluster 2024-02-12T16:39:15.465208+0000 osd.5 (osd.5) 184 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:39:17.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:17 smithi121 bash[18489]: cluster 2024-02-12T16:39:15.943740+0000 osd.11 (osd.11) 201 : cluster [DBG] 1.4 deep-scrub starts 2024-02-12T16:39:17.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:17 smithi121 bash[18489]: cluster 2024-02-12T16:39:15.947500+0000 osd.11 (osd.11) 202 : cluster [DBG] 1.4 deep-scrub ok 2024-02-12T16:39:17.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:17 smithi121 bash[18489]: cluster 2024-02-12T16:39:16.187277+0000 mgr.y (mgr.14146) 1647 : cluster [DBG] pgmap v2367: 47 pgs: 47 active+clean; 76 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:17.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:17 smithi121 bash[18489]: cluster 2024-02-12T16:39:16.242302+0000 mon.a (mon.0) 2894 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T16:39:17.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:17 smithi121 bash[18489]: cluster 2024-02-12T16:39:16.250353+0000 mon.a (mon.0) 2895 : cluster [DBG] osdmap e849: 12 total, 11 up, 12 in 2024-02-12T16:39:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:17 smithi169 bash[19604]: cluster 2024-02-12T16:39:15.430135+0000 osd.5 (osd.5) 183 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:39:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:17 smithi169 bash[19604]: cluster 2024-02-12T16:39:15.465208+0000 osd.5 (osd.5) 184 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:39:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:17 smithi169 bash[19604]: cluster 2024-02-12T16:39:15.943740+0000 osd.11 (osd.11) 201 : cluster [DBG] 1.4 deep-scrub starts 2024-02-12T16:39:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:17 smithi169 bash[19604]: cluster 2024-02-12T16:39:15.947500+0000 osd.11 (osd.11) 202 : cluster [DBG] 1.4 deep-scrub ok 2024-02-12T16:39:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:17 smithi169 bash[19604]: cluster 2024-02-12T16:39:16.187277+0000 mgr.y (mgr.14146) 1647 : cluster [DBG] pgmap v2367: 47 pgs: 47 active+clean; 76 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:17 smithi169 bash[19604]: cluster 2024-02-12T16:39:16.242302+0000 mon.a (mon.0) 2894 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-12T16:39:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:17 smithi169 bash[19604]: cluster 2024-02-12T16:39:16.250353+0000 mon.a (mon.0) 2895 : cluster [DBG] osdmap e849: 12 total, 11 up, 12 in 2024-02-12T16:39:17.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:17.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:17.710 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:17.784 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:17.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:17.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:17.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:17.985 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:17.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:18.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:18.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:18.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:18.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:18.266 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:18.267 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:39:17 smithi123 bash[113661]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-7 2024-02-12T16:39:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:18.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:18.304 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 16 10115 10099 157.776 0 - 0.00317964 2024-02-12T16:39:18.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:18.409 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:18 smithi121 bash[18489]: cluster 2024-02-12T16:39:17.257899+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e850: 12 total, 11 up, 12 in 2024-02-12T16:39:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:18.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:18.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:18 smithi169 bash[19604]: cluster 2024-02-12T16:39:17.257899+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e850: 12 total, 11 up, 12 in 2024-02-12T16:39:18.550 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:18.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:18.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:18 smithi123 bash[19119]: cluster 2024-02-12T16:39:17.257899+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e850: 12 total, 11 up, 12 in 2024-02-12T16:39:18.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:18.693 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:18.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:18.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:18.826 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:18.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:18.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:18.941 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:39:18.941 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:39:18.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:18.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:19.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:19.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:19.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:19.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:19.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:19.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:19.304 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 10115 10099 126.221 0 - 0.00317964 2024-02-12T16:39:19.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:19.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:19 smithi121 bash[18489]: cluster 2024-02-12T16:39:18.187967+0000 mgr.y (mgr.14146) 1648 : cluster [DBG] pgmap v2370: 47 pgs: 3 peering, 2 stale+active+clean, 42 active+clean; 350 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:19.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:19 smithi121 bash[18489]: cluster 2024-02-12T16:39:18.253011+0000 mon.a (mon.0) 2897 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-12T16:39:19.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:19 smithi121 bash[18489]: cluster 2024-02-12T16:39:18.696315+0000 osd.10 (osd.10) 186 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:39:19.366 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:19 smithi121 bash[18489]: cluster 2024-02-12T16:39:18.726995+0000 osd.10 (osd.10) 187 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:39:19.387 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:19.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:19.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:19.529 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:19.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:19 smithi169 bash[19604]: cluster 2024-02-12T16:39:18.187967+0000 mgr.y (mgr.14146) 1648 : cluster [DBG] pgmap v2370: 47 pgs: 3 peering, 2 stale+active+clean, 42 active+clean; 350 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:19.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:19 smithi169 bash[19604]: cluster 2024-02-12T16:39:18.253011+0000 mon.a (mon.0) 2897 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-12T16:39:19.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:19 smithi169 bash[19604]: cluster 2024-02-12T16:39:18.696315+0000 osd.10 (osd.10) 186 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:39:19.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:19 smithi169 bash[19604]: cluster 2024-02-12T16:39:18.726995+0000 osd.10 (osd.10) 187 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:39:19.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:19 smithi123 bash[19119]: cluster 2024-02-12T16:39:18.187967+0000 mgr.y (mgr.14146) 1648 : cluster [DBG] pgmap v2370: 47 pgs: 3 peering, 2 stale+active+clean, 42 active+clean; 350 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:39:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:19 smithi123 bash[19119]: cluster 2024-02-12T16:39:18.253011+0000 mon.a (mon.0) 2897 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-12T16:39:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:19 smithi123 bash[19119]: cluster 2024-02-12T16:39:18.696315+0000 osd.10 (osd.10) 186 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:39:19.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:19 smithi123 bash[19119]: cluster 2024-02-12T16:39:18.726995+0000 osd.10 (osd.10) 187 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:39:19.669 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:19.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:19.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:19.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:19.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:19.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:39:19.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:19.956 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:19.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:20.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:20.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:20.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:20.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:20.224 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:20.230 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:20.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:20.305 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 10115 10099 105.184 0 - 0.00317964 2024-02-12T16:39:20.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:20.374 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:20.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:20.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:20.518 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:20.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:20.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:20.634 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:20 smithi169 bash[19604]: cluster 2024-02-12T16:39:20.265334+0000 mon.a (mon.0) 2898 : cluster [WRN] Health check failed: Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded (PG_DEGRADED) 2024-02-12T16:39:20.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:20 smithi123 bash[19119]: cluster 2024-02-12T16:39:20.265334+0000 mon.a (mon.0) 2898 : cluster [WRN] Health check failed: Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded (PG_DEGRADED) 2024-02-12T16:39:20.659 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:20.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:20 smithi121 bash[18489]: cluster 2024-02-12T16:39:20.265334+0000 mon.a (mon.0) 2898 : cluster [WRN] Health check failed: Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded (PG_DEGRADED) 2024-02-12T16:39:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:20.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:20.765 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:20.803 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:20.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:20.857 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.7.service' 2024-02-12T16:39:20.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:39:20.886 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-12T16:39:20.887 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-02-12T16:39:20.887 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi123.front.sepia.ceph.com 2024-02-12T16:39:20.887 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.7 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-02-12T16:39:20.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:20.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:20.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:21.047 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:21.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:21.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:21.117 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.7/config 2024-02-12T16:39:21.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:21.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:21.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:21.305 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 16 10115 10099 90.1572 0 - 0.00317964 2024-02-12T16:39:21.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:21.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:21.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:21.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:21.502 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:21.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:21.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:21.618 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:21 smithi169 bash[19604]: cluster 2024-02-12T16:39:20.188974+0000 mgr.y (mgr.14146) 1649 : cluster [DBG] pgmap v2371: 47 pgs: 1 undersized+degraded+peered, 4 undersized+peered, 3 peering, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:21 smithi123 bash[19119]: cluster 2024-02-12T16:39:20.188974+0000 mgr.y (mgr.14146) 1649 : cluster [DBG] pgmap v2371: 47 pgs: 1 undersized+degraded+peered, 4 undersized+peered, 3 peering, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:21.642 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:21.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:21 smithi121 bash[18489]: cluster 2024-02-12T16:39:20.188974+0000 mgr.y (mgr.14146) 1649 : cluster [DBG] pgmap v2371: 47 pgs: 1 undersized+degraded+peered, 4 undersized+peered, 3 peering, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:21.747 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:21.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:21.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:21.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:21.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:22.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:22.068 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:22.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:22.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:22.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:22.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:22.305 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 16 10115 10099 78.8873 0 - 0.00317964 2024-02-12T16:39:22.305 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:22.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:22.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:22.478 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:22.479 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:22 smithi123 bash[19119]: cluster 2024-02-12T16:39:21.995475+0000 osd.8 (osd.8) 113 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:39:22.479 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:22 smithi123 bash[19119]: cluster 2024-02-12T16:39:22.102237+0000 osd.8 (osd.8) 114 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:39:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:22.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:22.504 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:39:22.524 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:39:22.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:39:22.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:22.584 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:39:22.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:22 smithi121 bash[18489]: cluster 2024-02-12T16:39:21.995475+0000 osd.8 (osd.8) 113 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:39:22.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:22 smithi121 bash[18489]: cluster 2024-02-12T16:39:22.102237+0000 osd.8 (osd.8) 114 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:39:22.611 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:22.613 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:39:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:22.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:22.638 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:39:22.659 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:39:22.690 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:39:22.714 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:39:22.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:22.725 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:22 smithi169 bash[19604]: cluster 2024-02-12T16:39:21.995475+0000 osd.8 (osd.8) 113 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:39:22.725 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:22 smithi169 bash[19604]: cluster 2024-02-12T16:39:22.102237+0000 osd.8 (osd.8) 114 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:39:22.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:39:22.753 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:22.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:22.766 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:39:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:39:22.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:22.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:22.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:22.987 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:23.026 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:23.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:23.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:23.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:23.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:23.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:23.305 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 10115 10099 70.1217 0 - 0.00317964 2024-02-12T16:39:23.306 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:23.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:23.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:23.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:23 smithi121 bash[18489]: cluster 2024-02-12T16:39:22.189956+0000 mgr.y (mgr.14146) 1650 : cluster [DBG] pgmap v2372: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:23.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:23 smithi121 bash[18489]: audit 2024-02-12T16:39:22.477063+0000 mon.a (mon.0) 2899 : audit [DBG] from='client.? v1:172.21.15.121:0/550099081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:23.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:23 smithi121 bash[18489]: cluster 2024-02-12T16:39:22.761796+0000 osd.0 (osd.0) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:23.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:23 smithi121 bash[18489]: cluster 2024-02-12T16:39:22.761867+0000 osd.0 (osd.0) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:23.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:23.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:23.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:23.590 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:23.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:23 smithi123 bash[19119]: cluster 2024-02-12T16:39:22.189956+0000 mgr.y (mgr.14146) 1650 : cluster [DBG] pgmap v2372: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:23.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:23 smithi123 bash[19119]: audit 2024-02-12T16:39:22.477063+0000 mon.a (mon.0) 2899 : audit [DBG] from='client.? v1:172.21.15.121:0/550099081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:23.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:23 smithi123 bash[19119]: cluster 2024-02-12T16:39:22.761796+0000 osd.0 (osd.0) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:23.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:23 smithi123 bash[19119]: cluster 2024-02-12T16:39:22.761867+0000 osd.0 (osd.0) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:23.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:23.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:23 smithi169 bash[19604]: cluster 2024-02-12T16:39:22.189956+0000 mgr.y (mgr.14146) 1650 : cluster [DBG] pgmap v2372: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:23.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:23 smithi169 bash[19604]: audit 2024-02-12T16:39:22.477063+0000 mon.a (mon.0) 2899 : audit [DBG] from='client.? v1:172.21.15.121:0/550099081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:23.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:23 smithi169 bash[19604]: cluster 2024-02-12T16:39:22.761796+0000 osd.0 (osd.0) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:23.707 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:23 smithi169 bash[19604]: cluster 2024-02-12T16:39:22.761867+0000 osd.0 (osd.0) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:23.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:23.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:23.871 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:23.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:23.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:24.011 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:24.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:24.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:24.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:24.290 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:24.305 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 16 10115 10099 63.1094 0 - 0.00317964 2024-02-12T16:39:24.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:24 smithi123 bash[19119]: cluster 2024-02-12T16:39:23.340225+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e851: 12 total, 11 up, 12 in 2024-02-12T16:39:24.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:24.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:39:24.513 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:39:24.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:24 smithi169 bash[19604]: cluster 2024-02-12T16:39:23.340225+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e851: 12 total, 11 up, 12 in 2024-02-12T16:39:24.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:24.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:24.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:24 smithi121 bash[18489]: cluster 2024-02-12T16:39:23.340225+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e851: 12 total, 11 up, 12 in 2024-02-12T16:39:24.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:24.710 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:24.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:24.853 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:24.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:24.983 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:24.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:25.088 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:25.123 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:25.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:25.231 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:25.266 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:25.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:25.305 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 10115 10099 57.3721 0 - 0.00317964 2024-02-12T16:39:25.368 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:39:25.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:25.397 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:25.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:25.501 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:25.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:25.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:25 smithi169 bash[19604]: cluster 2024-02-12T16:39:24.190900+0000 mgr.y (mgr.14146) 1651 : cluster [DBG] pgmap v2374: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:25 smithi169 bash[19604]: audit 2024-02-12T16:39:25.322988+0000 mon.a (mon.0) 2901 : audit [INF] from='client.26861 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:39:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:25 smithi169 bash[19604]: audit 2024-02-12T16:39:25.324631+0000 mon.b (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.121:0/3096511419' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:39:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:25 smithi123 bash[19119]: cluster 2024-02-12T16:39:24.190900+0000 mgr.y (mgr.14146) 1651 : cluster [DBG] pgmap v2374: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:25 smithi123 bash[19119]: audit 2024-02-12T16:39:25.322988+0000 mon.a (mon.0) 2901 : audit [INF] from='client.26861 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:39:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:25 smithi123 bash[19119]: audit 2024-02-12T16:39:25.324631+0000 mon.b (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.121:0/3096511419' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:39:25.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:25 smithi121 bash[18489]: cluster 2024-02-12T16:39:24.190900+0000 mgr.y (mgr.14146) 1651 : cluster [DBG] pgmap v2374: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:25 smithi121 bash[18489]: audit 2024-02-12T16:39:25.322988+0000 mon.a (mon.0) 2901 : audit [INF] from='client.26861 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:39:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:25 smithi121 bash[18489]: audit 2024-02-12T16:39:25.324631+0000 mon.b (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.121:0/3096511419' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:39:25.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:25.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:25.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:25.825 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:25.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:25.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:25.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:25.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:26.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:26.098 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:26.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:26.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:26.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:26.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:26.306 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 10115 10099 52.591 0 - 0.00317964 2024-02-12T16:39:26.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:26.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:26.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:26.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:26.507 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:26.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:26.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:39:26.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:26.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:26 smithi123 bash[19119]: cluster 2024-02-12T16:39:25.352722+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:26.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:26 smithi123 bash[19119]: audit 2024-02-12T16:39:25.360003+0000 mon.a (mon.0) 2903 : audit [INF] from='client.26861 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:39:26.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:26 smithi123 bash[19119]: cluster 2024-02-12T16:39:25.360127+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e852: 12 total, 11 up, 12 in 2024-02-12T16:39:26.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:26 smithi123 bash[19119]: cluster 2024-02-12T16:39:25.876501+0000 mon.a (mon.0) 2905 : cluster [WRN] Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:39:26.646 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:26.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:26 smithi121 bash[18489]: cluster 2024-02-12T16:39:25.352722+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:26 smithi121 bash[18489]: audit 2024-02-12T16:39:25.360003+0000 mon.a (mon.0) 2903 : audit [INF] from='client.26861 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:39:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:26 smithi121 bash[18489]: cluster 2024-02-12T16:39:25.360127+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e852: 12 total, 11 up, 12 in 2024-02-12T16:39:26.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:26 smithi121 bash[18489]: cluster 2024-02-12T16:39:25.876501+0000 mon.a (mon.0) 2905 : cluster [WRN] Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:39:26.714 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:26.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:26.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:26 smithi169 bash[19604]: cluster 2024-02-12T16:39:25.352722+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:26.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:26 smithi169 bash[19604]: audit 2024-02-12T16:39:25.360003+0000 mon.a (mon.0) 2903 : audit [INF] from='client.26861 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:39:26.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:26 smithi169 bash[19604]: cluster 2024-02-12T16:39:25.360127+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e852: 12 total, 11 up, 12 in 2024-02-12T16:39:26.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:26 smithi169 bash[19604]: cluster 2024-02-12T16:39:25.876501+0000 mon.a (mon.0) 2905 : cluster [WRN] Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:39:26.786 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:26.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:26.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:26.926 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:26.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:27.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:27.067 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:27.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:27.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:27.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:27.306 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 10115 10099 48.5454 0 - 0.00317964 2024-02-12T16:39:27.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:27.370 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:27.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:27.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:27.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:27.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:27.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:27.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:27 smithi169 bash[19604]: cluster 2024-02-12T16:39:26.191487+0000 mgr.y (mgr.14146) 1652 : cluster [DBG] pgmap v2376: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 561 op/s; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:27 smithi123 bash[19119]: cluster 2024-02-12T16:39:26.191487+0000 mgr.y (mgr.14146) 1652 : cluster [DBG] pgmap v2376: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 561 op/s; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:27.654 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:27.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:27.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:27 smithi121 bash[18489]: cluster 2024-02-12T16:39:26.191487+0000 mgr.y (mgr.14146) 1652 : cluster [DBG] pgmap v2376: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 561 op/s; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:27.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:27.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:27.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:27.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:27.929 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:27.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:28.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:28.070 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:28.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:28.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:28.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:28.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:28.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:39:28.306 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 10115 10099 45.0779 0 - 0.00317964 2024-02-12T16:39:28.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:28.349 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:28.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:28.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:28.482 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:28.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:28.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:28.618 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:28.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:28.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:28.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:28.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:28.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:28.895 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:28.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:28.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:29.037 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:29.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:29.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:29.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:29.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:29.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:29.306 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 10115 10099 42.0727 0 - 0.00317964 2024-02-12T16:39:29.324 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:29.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:29 smithi121 bash[18489]: cluster 2024-02-12T16:39:28.192135+0000 mgr.y (mgr.14146) 1653 : cluster [DBG] pgmap v2377: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:29.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:29.459 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:29.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:29 smithi169 bash[19604]: cluster 2024-02-12T16:39:28.192135+0000 mgr.y (mgr.14146) 1653 : cluster [DBG] pgmap v2377: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:29.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:29.594 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:29.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:29.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:29 smithi123 bash[19119]: cluster 2024-02-12T16:39:28.192135+0000 mgr.y (mgr.14146) 1653 : cluster [DBG] pgmap v2377: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:29.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:29.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:29.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:29.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:29.876 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:29.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:29.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:30.018 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:30.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:30.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:30.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:30.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:30.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:30.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:30.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:30.306 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 16 10115 10099 39.4432 0 - 0.00317964 2024-02-12T16:39:30.388 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:30 smithi123 bash[19119]: cluster 2024-02-12T16:39:30.288489+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:30 smithi123 bash[19119]: cluster 2024-02-12T16:39:30.288572+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:30.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:30.420 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:39:30.420 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:39:30.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:39:30.420 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:39:30.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.420 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300188 6.14 6.4ceffcb4 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.204645+0000", 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.20694479899998, 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.99970946400000005, 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300188 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204645+0000", 2024-02-12T16:39:30.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204645+0000", 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204642+0000", 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4810000000000006e-06 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204651+0000", 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5613000000000001e-05 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204666+0000", 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.567e-06 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204673+0000", 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5782000000000002e-05 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204729+0000", 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3799999999999999e-06 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.204735+0000", 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.99960395199999996 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204339+0000", 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5299e-05 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204354+0000", 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.99970946400000005 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300253 6.14 6.bbdd42d4 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.262545+0000", 2024-02-12T16:39:30.425 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.149044366, 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94204153599999996, 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300253 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262545+0000", 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262545+0000", 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262542+0000", 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0456e-05 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262553+0000", 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6531000000000001e-05 2024-02-12T16:39:30.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262569+0000", 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6279999999999994e-06 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262578+0000", 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9821999999999998e-05 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262628+0000", 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5719999999999997e-06 2024-02-12T16:39:30.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.262635+0000", 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94191894099999995 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204554+0000", 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2468000000000002e-05 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204587+0000", 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.94204153599999996 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.429 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300272 6.14 6.949ce094 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.281827+0000", 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12976281700003, 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92279888200000004, 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300272 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.430 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281827+0000", 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281827+0000", 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281824+0000", 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6055000000000001e-05 2024-02-12T16:39:30.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281840+0000", 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0126e-05 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281860+0000", 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6910000000000003e-06 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281869+0000", 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1373999999999999e-05 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281930+0000", 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6770000000000001e-06 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281939+0000", 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92267843100000002 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204617+0000", 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5990000000000002e-06 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204626+0000", 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92279888200000004 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300274 6.14 6.c8ffe254 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.281887+0000", 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12970225800001, 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92277841199999999, 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300274 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281887+0000", 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281887+0000", 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281886+0000", 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6e-06 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281891+0000", 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8089999999999998e-06 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281897+0000", 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6290000000000001e-06 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281901+0000", 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2589999999999995e-05 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281964+0000", 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7250000000000002e-06 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.281967+0000", 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92268319200000004 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204650+0000", 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5871000000000001e-05 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204666+0000", 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92277841199999999 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300278 6.14 6.f59c9594 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.283259+0000", 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12833091099998, 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.921427039, 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300278 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283259+0000", 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283259+0000", 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283257+0000", 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.964e-06 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283263+0000", 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9029999999999999e-06 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283273+0000", 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5999999999999997e-06 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283279+0000", 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8618000000000002e-05 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283317+0000", 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3319999999999997e-06 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.283324+0000", 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.92135630700000004 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204680+0000", 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8150000000000002e-06 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204686+0000", 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.921427039 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300188 6.14 6.4ceffcb4 (undecoded) ondisk+retry+write+known_if_redirected e726)", 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.288228+0000", 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12336180099999, 2024-02-12T16:39:30.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91654532799999999, 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300188 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288228+0000", 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288228+0000", 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288226+0000", 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0389999999999993e-06 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288235+0000", 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2454e-05 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288248+0000", 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3550000000000005e-06 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288256+0000", 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5189999999999998e-05 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288311+0000", 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0670000000000003e-06 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288318+0000", 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91643662000000004 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204755+0000", 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8417e-05 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204773+0000", 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91654532799999999 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300253 6.14 6.bbdd42d4 (undecoded) ondisk+retry+write+known_if_redirected e726)", 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.288269+0000", 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12332112199999, 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91652246500000001, 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.448 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300253 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288269+0000", 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288269+0000", 2024-02-12T16:39:30.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288268+0000", 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.501e-06 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288271+0000", 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3798000000000001e-05 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288285+0000", 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1690000000000002e-06 2024-02-12T16:39:30.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288289+0000", 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8326999999999998e-05 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288358+0000", 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2770000000000002e-06 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288364+0000", 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91642254700000003 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204786+0000", 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6310000000000002e-06 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204791+0000", 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91652246500000001 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300272 6.14 6.949ce094 (undecoded) ondisk+retry+write+known_if_redirected e726)", 2024-02-12T16:39:30.452 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.288306+0000", 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12328345700001, 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91653186099999995, 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300272 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288306+0000", 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288306+0000", 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288306+0000", 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0070000000000002e-06 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288309+0000", 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.011e-06 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288314+0000", 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0039999999999999e-06 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288319+0000", 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1581999999999993e-05 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288390+0000", 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2079999999999999e-06 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288395+0000", 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91642617999999998 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204822+0000", 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6480000000000001e-05 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204838+0000", 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91653186099999995 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.456 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300274 6.14 6.c8ffe254 (undecoded) ondisk+retry+write+known_if_redirected e726)", 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.288323+0000", 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12326691599998, 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91654689199999995, 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300274 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288323+0000", 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288323+0000", 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288322+0000", 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5009999999999999e-06 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288325+0000", 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2699999999999998e-06 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288329+0000", 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4709999999999999e-06 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288333+0000", 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3035999999999996e-05 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288416+0000", 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9929999999999995e-06 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288424+0000", 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.916440484 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204864+0000", 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204870+0000", 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91654689199999995 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300278 6.14 6.f59c9594 (undecoded) ondisk+retry+write+known_if_redirected e726)", 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.288435+0000", 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12315443900002, 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91651608699999998, 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300278 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288435+0000", 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288435+0000", 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288435+0000", 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.394e-06 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288438+0000", 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6410000000000001e-06 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288443+0000", 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.303e-06 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288446+0000", 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6951999999999999e-05 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288473+0000", 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0139999999999998e-06 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288478+0000", 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91644451999999998 2024-02-12T16:39:30.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204922+0000", 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8917000000000001e-05 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204951+0000", 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91651608699999998 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.465 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300281 6.14 6.2e127d74 (undecoded) ondisk+retry+write+known_if_redirected e726)", 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.288468+0000", 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.12312184000001, 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91651088300000005, 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300281 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288468+0000", 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288468+0000", 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288467+0000", 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1670000000000002e-06 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288470+0000", 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3259999999999997e-06 2024-02-12T16:39:30.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288474+0000", 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2789999999999999e-06 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288477+0000", 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0482000000000002e-05 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288528+0000", 2024-02-12T16:39:30.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7580000000000001e-06 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.288530+0000", 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91644381399999997 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204974+0000", 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4819999999999996e-06 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204979+0000", 2024-02-12T16:39:30.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.91651088300000005 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300334 6.14 6.6217d134 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.334967+0000", 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.07662241600002, 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87002833700000004, 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300334 2024-02-12T16:39:30.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.334967+0000", 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.334967+0000", 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.334964+0000", 2024-02-12T16:39:30.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0158e-05 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.334975+0000", 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8949000000000001e-05 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.334994+0000", 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.335002+0000", 2024-02-12T16:39:30.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0866000000000003e-05 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.335063+0000", 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1912999999999999e-05 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.335075+0000", 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.86991445499999998 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204990+0000", 2024-02-12T16:39:30.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1020000000000002e-06 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.204996+0000", 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87002833700000004 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300354 6.14 6.95bcd9b4 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.354714+0000", 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 368.056875934, 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85029916699999997, 2024-02-12T16:39:30.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300354 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354714+0000", 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354714+0000", 2024-02-12T16:39:30.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354712+0000", 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9009999999999994e-06 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354720+0000", 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.129e-05 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354731+0000", 2024-02-12T16:39:30.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8749999999999999e-06 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354736+0000", 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3320999999999999e-05 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354800+0000", 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0879999999999999e-05 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.354811+0000", 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85019869599999998 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205009+0000", 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7170000000000002e-06 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205013+0000", 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.85029916699999997 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300407 6.14 6.3a5b6d34 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.419771+0000", 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.99181905900002, 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78525490200000003, 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300407 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419771+0000", 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419771+0000", 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419768+0000", 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4789999999999999e-06 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419776+0000", 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3329000000000001e-05 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419789+0000", 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0319999999999998e-06 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419796+0000", 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8297000000000003e-05 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419834+0000", 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.746e-06 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.419839+0000", 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78518287600000003 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205022+0000", 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4829999999999999e-06 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205026+0000", 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78525490200000003 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300416 6.14 6.c8f05914 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.426434+0000", 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.98515607299998, 2024-02-12T16:39:30.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77869392100000001, 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300416 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426434+0000", 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426434+0000", 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426432+0000", 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4999999999999996e-06 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426438+0000", 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2207e-05 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426450+0000", 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2279999999999998e-06 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426456+0000", 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6622000000000003e-05 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426492+0000", 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7389999999999999e-06 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.426497+0000", 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.778602825 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205100+0000", 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7773999999999999e-05 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205128+0000", 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77869392100000001 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300424 6.14 6.6df01a34 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.432994+0000", 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.97859529700003, 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77218374300000003, 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300424 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.432994+0000", 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.432994+0000", 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.432992+0000", 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3469999999999999e-06 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.433000+0000", 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0574e-05 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.433021+0000", 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3400000000000003e-06 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.433027+0000", 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7658999999999999e-05 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.433075+0000", 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.259e-06 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.433080+0000", 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77208355699999998 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205163+0000", 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4715999999999999e-05 2024-02-12T16:39:30.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205178+0000", 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77218374300000003 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300484 6.14 6.5dc85134 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.515769+0000", 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.89582029600001, 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68948472599999999, 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300484 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515769+0000", 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515769+0000", 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:39:30.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515767+0000", 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3470000000000005e-06 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515776+0000", 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6568000000000001e-05 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515793+0000", 2024-02-12T16:39:30.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5309999999999998e-06 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515799+0000", 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9432999999999999e-05 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515849+0000", 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0369999999999995e-06 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.515857+0000", 2024-02-12T16:39:30.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68937050300000002 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205227+0000", 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7093999999999998e-05 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205254+0000", 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68948472599999999 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.497 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300487 6.14 6.4378beb4 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.521320+0000", 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.89026956499998, 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68398171200000002, 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300487 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521320+0000", 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521320+0000", 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521318+0000", 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5990000000000002e-06 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521326+0000", 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6113999999999999e-05 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521342+0000", 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521348+0000", 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.257e-05 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521401+0000", 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5689999999999994e-06 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.521410+0000", 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68387275000000003 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205283+0000", 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8799000000000001e-05 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205302+0000", 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.68398171200000002 2024-02-12T16:39:30.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300548 6.14 6.fc6f4774 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.649673+0000", 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.76191628700002, 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55568853500000004, 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300548 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.502 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649673+0000", 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649673+0000", 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649671+0000", 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9829999999999996e-06 2024-02-12T16:39:30.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649679+0000", 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.482e-05 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649694+0000", 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6480000000000003e-06 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649701+0000", 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8726999999999994e-05 2024-02-12T16:39:30.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649770+0000", 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1491999999999999e-05 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.649782+0000", 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55556510800000003 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205347+0000", 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5140000000000001e-05 2024-02-12T16:39:30.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205362+0000", 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55568853500000004 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26257.0:300555 6.14 6.8ce912d4 (undecoded) ondisk+write+known_if_redirected e726)", 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:33:22.670229+0000", 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 367.74136108699997, 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53516879399999995, 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:39:30.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26257", 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 300555 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670229+0000", 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670229+0000", 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670225+0000", 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1415e-05 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670237+0000", 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0469999999999999e-05 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670257+0000", 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1579999999999999e-05 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670269+0000", 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7800000000000003e-05 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670316+0000", 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5890000000000002e-06 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:22.670322+0000", 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.535066559 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205389+0000", 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8710000000000003e-06 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:33:23.205397+0000", 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53516879399999995 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:39:30.510 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:39:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:30.511 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:30 smithi121 bash[18489]: cluster 2024-02-12T16:39:30.288489+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:30.511 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:30 smithi121 bash[18489]: cluster 2024-02-12T16:39:30.288572+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:30.512 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:30.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:30.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:30.650 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:30.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:30 smithi169 bash[19604]: cluster 2024-02-12T16:39:30.288489+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:30.652 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:30 smithi169 bash[19604]: cluster 2024-02-12T16:39:30.288572+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:30.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:30.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:30.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:30.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:30.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:30.927 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:30.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:31.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:31.067 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:31.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:31.163 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:39:31.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:31.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:31.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:31.306 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 10115 10099 37.123 0 - 0.00317964 2024-02-12T16:39:31.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:31.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:31.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:31.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:31.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:31.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:31.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:31.597 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:31 smithi123 bash[19119]: cluster 2024-02-12T16:39:30.193086+0000 mgr.y (mgr.14146) 1654 : cluster [DBG] pgmap v2378: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:31.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:31 smithi123 bash[19119]: cluster 2024-02-12T16:39:30.384382+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e853: 12 total, 11 up, 12 in 2024-02-12T16:39:31.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:31 smithi123 bash[19119]: cluster 2024-02-12T16:39:30.888265+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e854: 12 total, 11 up, 12 in 2024-02-12T16:39:31.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:31.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:31 smithi121 bash[18489]: cluster 2024-02-12T16:39:30.193086+0000 mgr.y (mgr.14146) 1654 : cluster [DBG] pgmap v2378: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:31 smithi121 bash[18489]: cluster 2024-02-12T16:39:30.384382+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e853: 12 total, 11 up, 12 in 2024-02-12T16:39:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:31 smithi121 bash[18489]: cluster 2024-02-12T16:39:30.888265+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e854: 12 total, 11 up, 12 in 2024-02-12T16:39:31.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:31.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:31.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:31 smithi169 bash[19604]: cluster 2024-02-12T16:39:30.193086+0000 mgr.y (mgr.14146) 1654 : cluster [DBG] pgmap v2378: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:31 smithi169 bash[19604]: cluster 2024-02-12T16:39:30.384382+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e853: 12 total, 11 up, 12 in 2024-02-12T16:39:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:31 smithi169 bash[19604]: cluster 2024-02-12T16:39:30.888265+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e854: 12 total, 11 up, 12 in 2024-02-12T16:39:31.807 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:31.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:31.888 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:31.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:31.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:32.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:32.081 INFO:teuthology.orchestra.run.smithi123.stdout:1.15 2024-02-12T16:39:32.081 INFO:teuthology.orchestra.run.smithi123.stdout:1.0 2024-02-12T16:39:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:1.2 2024-02-12T16:39:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:8.d 2024-02-12T16:39:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:1.e 2024-02-12T16:39:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:1.1a 2024-02-12T16:39:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:1.9 2024-02-12T16:39:32.082 INFO:teuthology.orchestra.run.smithi123.stdout:1.d 2024-02-12T16:39:32.132 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:32.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:32.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:32.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:32.306 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 10115 10099 35.0606 0 - 0.00317964 2024-02-12T16:39:32.313 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:32.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:32.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:32.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:32.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:32.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:32.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:32.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:32.698 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:32.737 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:32.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:32.826 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:32.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:32.877 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:32.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:32.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:33.021 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:33.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:33.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:33.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:33.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 10115 10099 33.2154 0 - 0.00317964 2024-02-12T16:39:33.309 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:33.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:33.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.7 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 1.0 --file /var/log/ceph/exp.1.0.7 2024-02-12T16:39:33.447 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:33 smithi169 bash[19604]: cluster 2024-02-12T16:39:32.194022+0000 mgr.y (mgr.14146) 1655 : cluster [DBG] pgmap v2381: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:33.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:33 smithi121 bash[18489]: cluster 2024-02-12T16:39:32.194022+0000 mgr.y (mgr.14146) 1655 : cluster [DBG] pgmap v2381: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:33.675 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.7/config 2024-02-12T16:39:33.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:33.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:33.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:33.835 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:33 smithi123 bash[19119]: cluster 2024-02-12T16:39:32.194022+0000 mgr.y (mgr.14146) 1655 : cluster [DBG] pgmap v2381: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:33.853 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:33.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:33.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:33.991 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:33.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:34.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:34.134 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:34.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:34.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:34.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:34.274 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:34.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:34.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:39:34.304059 min lat: 0.00106883 max lat: 0.0956592 avg lat: 0.00317964 2024-02-12T16:39:34.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:39:34.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 16 10115 10099 31.5546 0 - 0.00317964 2024-02-12T16:39:34.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:34.415 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:34.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:34.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:34.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:34.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:34.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:34.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:34.688 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:34.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:34.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:34.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:34.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:34.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:34.962 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:34.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:35.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:35.096 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:35.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:35.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:35.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:35.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:35.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 10115 10099 30.0519 0 - 0.00317964 2024-02-12T16:39:35.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:35.377 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:35.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:35.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:35.494 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:35 smithi121 bash[18489]: cluster 2024-02-12T16:39:34.194932+0000 mgr.y (mgr.14146) 1656 : cluster [DBG] pgmap v2382: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:35.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:35.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:35.636 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:35 smithi169 bash[19604]: cluster 2024-02-12T16:39:34.194932+0000 mgr.y (mgr.14146) 1656 : cluster [DBG] pgmap v2382: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:35.662 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:35.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:35.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:35.807 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:35.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:35 smithi123 bash[19119]: cluster 2024-02-12T16:39:34.194932+0000 mgr.y (mgr.14146) 1656 : cluster [DBG] pgmap v2382: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:35.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:35.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:35.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:36.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:36.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:36.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:36.113 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:39:36.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:36.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:36.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:36.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 10115 10099 28.6859 0 - 0.00317964 2024-02-12T16:39:36.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:36.369 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:36.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:36.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:36.512 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:36.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:36.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:36.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:36.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:36.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:36.797 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:36.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:36.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:36.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:36.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:37.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:37.080 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:37.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:37.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:36 smithi121 bash[18489]: cluster 2024-02-12T16:39:36.195852+0000 mgr.y (mgr.14146) 1657 : cluster [DBG] pgmap v2383: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:37.172 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:36 smithi123 bash[19119]: cluster 2024-02-12T16:39:36.195852+0000 mgr.y (mgr.14146) 1657 : cluster [DBG] pgmap v2383: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:37.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:37.220 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:37.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:37.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:36 smithi169 bash[19604]: cluster 2024-02-12T16:39:36.195852+0000 mgr.y (mgr.14146) 1657 : cluster [DBG] pgmap v2383: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:37.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 10115 10099 27.4387 0 - 0.00317964 2024-02-12T16:39:37.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:37.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:37.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:37.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:37.481 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:39:37.493 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:37.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:37.508 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:39:37.526 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:39:37.545 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:39:37.565 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:39:37.588 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:39:37.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:37.618 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:39:37.625 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:37.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:37.641 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:39:37.668 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:39:37.694 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:39:37.716 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:39:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.718 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:39:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.719 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:39:37.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:37.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:37.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:37.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:37.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:37.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:37.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:38.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:38.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:38.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:38.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:37 smithi123 bash[19119]: audit 2024-02-12T16:39:37.448881+0000 mon.a (mon.0) 2908 : audit [DBG] from='client.? v1:172.21.15.121:0/3514098579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:38.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:37 smithi121 bash[18489]: audit 2024-02-12T16:39:37.448881+0000 mon.a (mon.0) 2908 : audit [DBG] from='client.? v1:172.21.15.121:0/3514098579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:38.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:38.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:38.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:38.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:37 smithi169 bash[19604]: audit 2024-02-12T16:39:37.448881+0000 mon.a (mon.0) 2908 : audit [DBG] from='client.? v1:172.21.15.121:0/3514098579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:38.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 10115 10099 26.2955 0 - 0.00317964 2024-02-12T16:39:38.311 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:38.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:38.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:38.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:38.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:39:38.517 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:39:38.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:38.591 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:38.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:38.695 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:38.733 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:38.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:38.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:38.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:38.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:38.916 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:39:38.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:38.986 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:38 smithi123 bash[19119]: audit 2024-02-12T16:39:38.014403+0000 mon.a (mon.0) 2909 : audit [INF] from='client.26870 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:38.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:38 smithi123 bash[19119]: audit 2024-02-12T16:39:38.015958+0000 mon.b (mon.2) 205 : audit [INF] from='client.? v1:172.21.15.121:0/164540018' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:38.987 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:38 smithi123 bash[19119]: cluster 2024-02-12T16:39:38.196856+0000 mgr.y (mgr.14146) 1658 : cluster [DBG] pgmap v2384: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:39.011 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:39.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:39.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:39.157 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:38 smithi121 bash[18489]: audit 2024-02-12T16:39:38.014403+0000 mon.a (mon.0) 2909 : audit [INF] from='client.26870 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:38 smithi121 bash[18489]: audit 2024-02-12T16:39:38.015958+0000 mon.b (mon.2) 205 : audit [INF] from='client.? v1:172.21.15.121:0/164540018' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:39.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:38 smithi121 bash[18489]: cluster 2024-02-12T16:39:38.196856+0000 mgr.y (mgr.14146) 1658 : cluster [DBG] pgmap v2384: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:39.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:39.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:38 smithi169 bash[19604]: audit 2024-02-12T16:39:38.014403+0000 mon.a (mon.0) 2909 : audit [INF] from='client.26870 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:39.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:38 smithi169 bash[19604]: audit 2024-02-12T16:39:38.015958+0000 mon.b (mon.2) 205 : audit [INF] from='client.? v1:172.21.15.121:0/164540018' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:39:39.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:38 smithi169 bash[19604]: cluster 2024-02-12T16:39:38.196856+0000 mgr.y (mgr.14146) 1658 : cluster [DBG] pgmap v2384: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:39.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:39.307 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 10115 10099 25.2436 0 - 0.00317964 2024-02-12T16:39:39.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:39.434 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:39.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:39.564 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:39.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:39.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:39.724 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:39:39.724 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:39:39.807 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:39.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:39.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:39 smithi121 bash[18489]: cluster 2024-02-12T16:39:38.895634+0000 mon.a (mon.0) 2910 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:39.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:39 smithi121 bash[18489]: audit 2024-02-12T16:39:38.905188+0000 mon.a (mon.0) 2911 : audit [INF] from='client.26870 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:39:39.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:39 smithi121 bash[18489]: cluster 2024-02-12T16:39:38.905260+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e855: 12 total, 11 up, 12 in 2024-02-12T16:39:39.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:39.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:40.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:40.121 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:40.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:39 smithi123 bash[19119]: cluster 2024-02-12T16:39:38.895634+0000 mon.a (mon.0) 2910 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:39 smithi123 bash[19119]: audit 2024-02-12T16:39:38.905188+0000 mon.a (mon.0) 2911 : audit [INF] from='client.26870 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:39:40.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:39 smithi123 bash[19119]: cluster 2024-02-12T16:39:38.905260+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e855: 12 total, 11 up, 12 in 2024-02-12T16:39:40.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:40.261 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:40.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:39 smithi169 bash[19604]: cluster 2024-02-12T16:39:38.895634+0000 mon.a (mon.0) 2910 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:39 smithi169 bash[19604]: audit 2024-02-12T16:39:38.905188+0000 mon.a (mon.0) 2911 : audit [INF] from='client.26870 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:39:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:39 smithi169 bash[19604]: cluster 2024-02-12T16:39:38.905260+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e855: 12 total, 11 up, 12 in 2024-02-12T16:39:40.308 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 16 10115 10099 24.2727 0 - 0.00317964 2024-02-12T16:39:40.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:40.403 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:40.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:40.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:39:40.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:40.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:39:40.545 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:40.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:40.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:40.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:40.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:40.708 INFO:teuthology.orchestra.run.smithi123.stderr:Exporting 1.0 info 1.0( empty local-lis/les=824/825 n=0 ec=14/14 lis/c=824/824 les/c/f=825/825/0 sis=824) 2024-02-12T16:39:40.722 INFO:teuthology.orchestra.run.smithi123.stderr:Export successful 2024-02-12T16:39:40.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:40.816 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:40.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:40.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:40.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:40.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:41.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:41.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:41.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:40 smithi123 bash[19119]: cluster 2024-02-12T16:39:40.197802+0000 mgr.y (mgr.14146) 1659 : cluster [DBG] pgmap v2386: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:41.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:40 smithi123 bash[19119]: cluster 2024-02-12T16:39:40.891518+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e856: 12 total, 11 up, 12 in 2024-02-12T16:39:41.200 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:41.238 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:41.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:40 smithi169 bash[19604]: cluster 2024-02-12T16:39:40.197802+0000 mgr.y (mgr.14146) 1659 : cluster [DBG] pgmap v2386: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:41.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:40 smithi169 bash[19604]: cluster 2024-02-12T16:39:40.891518+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e856: 12 total, 11 up, 12 in 2024-02-12T16:39:41.308 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 10115 10099 23.3737 0 - 0.00317964 2024-02-12T16:39:41.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:41.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:40 smithi121 bash[18489]: cluster 2024-02-12T16:39:40.197802+0000 mgr.y (mgr.14146) 1659 : cluster [DBG] pgmap v2386: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:41.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:40 smithi121 bash[18489]: cluster 2024-02-12T16:39:40.891518+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e856: 12 total, 11 up, 12 in 2024-02-12T16:39:41.381 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:41.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:41.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:41.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:41.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:41.656 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:41.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:41.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:41.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:39:41.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:41.797 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:41.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:41.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:41.939 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:41.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:42.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:42.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.7 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 1.0 2024-02-12T16:39:42.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:42.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:42.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:42.219 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:42.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:42.232 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.7/config 2024-02-12T16:39:42.308 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 10115 10099 22.5389 0 - 0.00317964 2024-02-12T16:39:42.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:42.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:42.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:42.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:42.527 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:42.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:42.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:42.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:42.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:42.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:42.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:42.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:42.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:42.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:42.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:43.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:43.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:43.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:43.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:43.232 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:43.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:43.308 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 10115 10099 21.7617 0 - 0.00317964 2024-02-12T16:39:43.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:43.374 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:43.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:43 smithi121 bash[18489]: cluster 2024-02-12T16:39:42.198727+0000 mgr.y (mgr.14146) 1660 : cluster [DBG] pgmap v2388: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:43.446 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:43 smithi123 bash[19119]: cluster 2024-02-12T16:39:42.198727+0000 mgr.y (mgr.14146) 1660 : cluster [DBG] pgmap v2388: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:43.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:43.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:43.516 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:43 smithi169 bash[19604]: cluster 2024-02-12T16:39:42.198727+0000 mgr.y (mgr.14146) 1660 : cluster [DBG] pgmap v2388: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:43.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:43.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:43.657 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:43.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:43.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:43.802 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:43.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:43.906 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:43.943 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:43.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:44.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:44.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:44.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:44.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:44.229 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:44.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:44.308 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 10115 10099 21.0364 0 - 0.00317964 2024-02-12T16:39:44.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:44.349 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:44 smithi121 bash[18489]: cluster 2024-02-12T16:39:43.657074+0000 osd.10 (osd.10) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:44.349 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:44 smithi121 bash[18489]: cluster 2024-02-12T16:39:43.657165+0000 osd.10 (osd.10) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:44.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:44.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:44.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:44.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:44.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:44.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:44 smithi169 bash[19604]: cluster 2024-02-12T16:39:43.657074+0000 osd.10 (osd.10) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:44.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:44 smithi169 bash[19604]: cluster 2024-02-12T16:39:43.657165+0000 osd.10 (osd.10) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:44.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:44.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:44 smithi123 bash[19119]: cluster 2024-02-12T16:39:43.657074+0000 osd.10 (osd.10) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:44.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:44 smithi123 bash[19119]: cluster 2024-02-12T16:39:43.657165+0000 osd.10 (osd.10) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:44.653 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:44.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:44.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:44.794 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:44.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:44.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:44.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:45.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:45.075 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:45.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:45.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:45.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:45.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:45.308 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 10115 10099 20.3578 0 - 0.00317964 2024-02-12T16:39:45.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:45.329 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:45 smithi123 bash[19119]: cluster 2024-02-12T16:39:44.199340+0000 mgr.y (mgr.14146) 1661 : cluster [DBG] pgmap v2389: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:45.329 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:45 smithi123 bash[19119]: cluster 2024-02-12T16:39:44.271657+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e857: 12 total, 11 up, 12 in 2024-02-12T16:39:45.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:45.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:45.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:45.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:45.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:45 smithi169 bash[19604]: cluster 2024-02-12T16:39:44.199340+0000 mgr.y (mgr.14146) 1661 : cluster [DBG] pgmap v2389: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:45 smithi169 bash[19604]: cluster 2024-02-12T16:39:44.271657+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e857: 12 total, 11 up, 12 in 2024-02-12T16:39:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:39:45.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:45.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:45 smithi121 bash[18489]: cluster 2024-02-12T16:39:44.199340+0000 mgr.y (mgr.14146) 1661 : cluster [DBG] pgmap v2389: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:45.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:45 smithi121 bash[18489]: cluster 2024-02-12T16:39:44.271657+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e857: 12 total, 11 up, 12 in 2024-02-12T16:39:45.637 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:45.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:45.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:45.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:45.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:45.883 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:45.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:45.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:46.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:46.062 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:46.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:46.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:46.204 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:46.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:46.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:46.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 10115 10099 19.7216 0 - 0.00317964 2024-02-12T16:39:46.343 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:46.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:46.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:46.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:46.487 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:46.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:46.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:46.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:46.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:46.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:46.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:46.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:46.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:46.913 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:46.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:47.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:47.054 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:47.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:47.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:47.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:46 smithi121 bash[18489]: cluster 2024-02-12T16:39:45.894496+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e858: 12 total, 11 up, 12 in 2024-02-12T16:39:47.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:46 smithi121 bash[18489]: cluster 2024-02-12T16:39:46.200255+0000 mgr.y (mgr.14146) 1662 : cluster [DBG] pgmap v2392: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:47.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:47.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:47.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:46 smithi169 bash[19604]: cluster 2024-02-12T16:39:45.894496+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e858: 12 total, 11 up, 12 in 2024-02-12T16:39:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:46 smithi169 bash[19604]: cluster 2024-02-12T16:39:46.200255+0000 mgr.y (mgr.14146) 1662 : cluster [DBG] pgmap v2392: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:47.301 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:47.309 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 10115 10099 19.1239 0 - 0.00317964 2024-02-12T16:39:47.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:46 smithi123 bash[19119]: cluster 2024-02-12T16:39:45.894496+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e858: 12 total, 11 up, 12 in 2024-02-12T16:39:47.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:46 smithi123 bash[19119]: cluster 2024-02-12T16:39:46.200255+0000 mgr.y (mgr.14146) 1662 : cluster [DBG] pgmap v2392: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:47.337 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:47.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:47.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:47.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:47.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:47.581 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:47.622 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:47.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:47.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:47.765 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:47.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:47.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:47.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:48.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:48.045 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:48.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:48.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:48.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:47 smithi121 bash[18489]: cluster 2024-02-12T16:39:47.398529+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:48.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:47 smithi121 bash[18489]: cluster 2024-02-12T16:39:47.398631+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:48.186 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:48.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:47 smithi169 bash[19604]: cluster 2024-02-12T16:39:47.398529+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:48.188 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:47 smithi169 bash[19604]: cluster 2024-02-12T16:39:47.398631+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:48.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:48.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:48.309 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 10115 10099 18.5615 0 - 0.00317964 2024-02-12T16:39:48.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:48.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:47 smithi123 bash[19119]: cluster 2024-02-12T16:39:47.398529+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:39:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:47 smithi123 bash[19119]: cluster 2024-02-12T16:39:47.398631+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:39:48.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:48.456 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:48.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:48.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:48.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:48.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:48.619 INFO:teuthology.orchestra.run.smithi123.stdout: marking collection for removal 2024-02-12T16:39:48.619 INFO:teuthology.orchestra.run.smithi123.stdout:setting '_remove' omap key 2024-02-12T16:39:48.619 INFO:teuthology.orchestra.run.smithi123.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-02-12T16:39:48.646 INFO:teuthology.orchestra.run.smithi123.stdout:Remove successful 2024-02-12T16:39:48.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:48.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:48.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:48.745 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:39:48.745 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:39:48.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:48.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:48.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:49.000 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:48 smithi123 bash[19119]: cluster 2024-02-12T16:39:47.904916+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e859: 12 total, 11 up, 12 in 2024-02-12T16:39:49.000 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:48 smithi123 bash[19119]: cluster 2024-02-12T16:39:48.201160+0000 mgr.y (mgr.14146) 1663 : cluster [DBG] pgmap v2394: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:49.024 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:49.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:49.128 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:48 smithi121 bash[18489]: cluster 2024-02-12T16:39:47.904916+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e859: 12 total, 11 up, 12 in 2024-02-12T16:39:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:48 smithi121 bash[18489]: cluster 2024-02-12T16:39:48.201160+0000 mgr.y (mgr.14146) 1663 : cluster [DBG] pgmap v2394: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:49.165 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:49.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:49.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:49.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:48 smithi169 bash[19604]: cluster 2024-02-12T16:39:47.904916+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e859: 12 total, 11 up, 12 in 2024-02-12T16:39:49.281 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:48 smithi169 bash[19604]: cluster 2024-02-12T16:39:48.201160+0000 mgr.y (mgr.14146) 1663 : cluster [DBG] pgmap v2394: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:49.301 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:49.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:49.309 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 10115 10099 18.0312 0 - 0.00317964 2024-02-12T16:39:49.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:49.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:49.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:49.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:49.581 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:49.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:49.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:49.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:49.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:49.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:49.857 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:49.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:49.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:39:49.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:50.027 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:50.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:50.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:49 smithi169 bash[19604]: audit 2024-02-12T16:39:49.097686+0000 mon.a (mon.0) 2917 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:39:50.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:49 smithi169 bash[19604]: audit 2024-02-12T16:39:49.721604+0000 mon.a (mon.0) 2918 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:39:50.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:49 smithi169 bash[19604]: audit 2024-02-12T16:39:49.723087+0000 mon.a (mon.0) 2919 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:39:50.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:49 smithi169 bash[19604]: audit 2024-02-12T16:39:49.733087+0000 mon.a (mon.0) 2920 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:39:50.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:50.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell -n osd.7 --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.0.7 2024-02-12T16:39:50.172 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:50.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:50.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:50.287 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:49 smithi121 bash[18489]: audit 2024-02-12T16:39:49.097686+0000 mon.a (mon.0) 2917 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:39:50.287 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:49 smithi121 bash[18489]: audit 2024-02-12T16:39:49.721604+0000 mon.a (mon.0) 2918 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:39:50.287 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:49 smithi121 bash[18489]: audit 2024-02-12T16:39:49.723087+0000 mon.a (mon.0) 2919 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:39:50.287 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:49 smithi121 bash[18489]: audit 2024-02-12T16:39:49.733087+0000 mon.a (mon.0) 2920 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:39:50.309 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 16 10115 10099 17.5303 0 - 0.00317964 2024-02-12T16:39:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:50.315 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/osd.7/config 2024-02-12T16:39:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:50.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:49 smithi123 bash[19119]: audit 2024-02-12T16:39:49.097686+0000 mon.a (mon.0) 2917 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:39:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:49 smithi123 bash[19119]: audit 2024-02-12T16:39:49.721604+0000 mon.a (mon.0) 2918 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:39:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:49 smithi123 bash[19119]: audit 2024-02-12T16:39:49.723087+0000 mon.a (mon.0) 2919 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:39:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:49 smithi123 bash[19119]: audit 2024-02-12T16:39:49.733087+0000 mon.a (mon.0) 2920 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:39:50.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:50.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:50.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:50.594 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:50.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:50.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:50.734 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:50.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:50.824 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:39:50.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:39:50.874 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:39:50.876 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:50.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:50.895 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:39:50.929 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:39:50.952 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:39:50.977 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:39:50.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:51.005 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:51.006 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:39:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:51.029 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:39:51.059 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:39:51.088 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:39:51.088 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.089 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:39:51.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:51.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:51.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:50 smithi169 bash[19604]: cluster 2024-02-12T16:39:50.202065+0000 mgr.y (mgr.14146) 1664 : cluster [DBG] pgmap v2395: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:51.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:50 smithi169 bash[19604]: audit 2024-02-12T16:39:50.793984+0000 mon.a (mon.0) 2921 : audit [DBG] from='client.? v1:172.21.15.121:0/3828497101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:51.260 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:50 smithi169 bash[19604]: cluster 2024-02-12T16:39:50.895623+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e860: 12 total, 11 up, 12 in 2024-02-12T16:39:51.286 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:51.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 10115 10099 17.0565 0 - 0.00317964 2024-02-12T16:39:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:50 smithi123 bash[19119]: cluster 2024-02-12T16:39:50.202065+0000 mgr.y (mgr.14146) 1664 : cluster [DBG] pgmap v2395: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:50 smithi123 bash[19119]: audit 2024-02-12T16:39:50.793984+0000 mon.a (mon.0) 2921 : audit [DBG] from='client.? v1:172.21.15.121:0/3828497101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:51.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:50 smithi123 bash[19119]: cluster 2024-02-12T16:39:50.895623+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e860: 12 total, 11 up, 12 in 2024-02-12T16:39:51.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:50 smithi121 bash[18489]: cluster 2024-02-12T16:39:50.202065+0000 mgr.y (mgr.14146) 1664 : cluster [DBG] pgmap v2395: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:50 smithi121 bash[18489]: audit 2024-02-12T16:39:50.793984+0000 mon.a (mon.0) 2921 : audit [DBG] from='client.? v1:172.21.15.121:0/3828497101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:39:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:50 smithi121 bash[18489]: cluster 2024-02-12T16:39:50.895623+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e860: 12 total, 11 up, 12 in 2024-02-12T16:39:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:51.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:51.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:51.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:51.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:51.704 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:51.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:51.845 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:39:51.856 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:39:51.949 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:51.989 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:39:51.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:51.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:52.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:52.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:52.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:52.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:51 smithi123 bash[19119]: audit 2024-02-12T16:39:51.367995+0000 mon.a (mon.0) 2923 : audit [INF] from='client.? v1:172.21.15.121:0/3346583081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:39:52.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:51 smithi121 bash[18489]: audit 2024-02-12T16:39:51.367995+0000 mon.a (mon.0) 2923 : audit [INF] from='client.? v1:172.21.15.121:0/3346583081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:39:52.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:52.248 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:51 smithi169 bash[19604]: audit 2024-02-12T16:39:51.367995+0000 mon.a (mon.0) 2923 : audit [INF] from='client.? v1:172.21.15.121:0/3346583081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:39:52.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:52.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:52.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 16 10115 10099 16.6076 0 - 0.00317964 2024-02-12T16:39:52.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:52.411 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:52.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:52.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:52.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:52.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:52.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:52.708 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:52.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:52.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:52.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:52.945 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:52.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:52.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:53.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:53.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:53.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:52 smithi121 bash[18489]: cluster 2024-02-12T16:39:51.966985+0000 mon.a (mon.0) 2924 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:52 smithi121 bash[18489]: audit 2024-02-12T16:39:51.980844+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? v1:172.21.15.121:0/3346583081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:39:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:52 smithi121 bash[18489]: cluster 2024-02-12T16:39:51.980959+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e861: 12 total, 11 up, 12 in 2024-02-12T16:39:53.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:52 smithi121 bash[18489]: cluster 2024-02-12T16:39:52.202936+0000 mgr.y (mgr.14146) 1665 : cluster [DBG] pgmap v2398: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:53.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:53.238 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:52 smithi169 bash[19604]: cluster 2024-02-12T16:39:51.966985+0000 mon.a (mon.0) 2924 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:53.238 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:52 smithi169 bash[19604]: audit 2024-02-12T16:39:51.980844+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? v1:172.21.15.121:0/3346583081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:39:53.238 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:52 smithi169 bash[19604]: cluster 2024-02-12T16:39:51.980959+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e861: 12 total, 11 up, 12 in 2024-02-12T16:39:53.239 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:52 smithi169 bash[19604]: cluster 2024-02-12T16:39:52.202936+0000 mgr.y (mgr.14146) 1665 : cluster [DBG] pgmap v2398: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:53.264 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:53.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:53.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 10115 10099 16.1818 0 - 0.00317964 2024-02-12T16:39:53.368 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:53.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:52 smithi123 bash[19119]: cluster 2024-02-12T16:39:51.966985+0000 mon.a (mon.0) 2924 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:39:53.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:52 smithi123 bash[19119]: audit 2024-02-12T16:39:51.980844+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? v1:172.21.15.121:0/3346583081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:39:53.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:52 smithi123 bash[19119]: cluster 2024-02-12T16:39:51.980959+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e861: 12 total, 11 up, 12 in 2024-02-12T16:39:53.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:52 smithi123 bash[19119]: cluster 2024-02-12T16:39:52.202936+0000 mgr.y (mgr.14146) 1665 : cluster [DBG] pgmap v2398: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:53.402 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:53.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:53.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:53.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:53.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:53.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:53.680 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:53.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:53.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:53.821 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:53.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:53.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:39:53.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:53.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:53.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:54.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:54.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:54.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:54.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:54.242 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:54.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:54.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:39:54.307250 min lat: 0.00106883 max lat: 0.0956592 avg lat: 0.00317964 2024-02-12T16:39:54.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:39:54.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 10115 10099 15.7772 0 - 0.00317964 2024-02-12T16:39:54.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:54.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:54.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:54.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:54.528 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:54.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:54.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:54.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:54.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:54.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:54.813 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:54.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:54.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:39:54.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:54.951 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:54.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:55.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:55.092 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:55.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:55.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:55.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:55.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:55.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 16 10115 10099 15.3924 0 - 0.00317964 2024-02-12T16:39:55.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:55.347 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:55 smithi169 bash[19604]: cluster 2024-02-12T16:39:54.203842+0000 mgr.y (mgr.14146) 1666 : cluster [DBG] pgmap v2399: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:55.373 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:55.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:55.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:55.490 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:55 smithi121 bash[18489]: cluster 2024-02-12T16:39:54.203842+0000 mgr.y (mgr.14146) 1666 : cluster [DBG] pgmap v2399: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:55.516 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:55.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:55.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:55.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:55 smithi123 bash[19119]: cluster 2024-02-12T16:39:54.203842+0000 mgr.y (mgr.14146) 1666 : cluster [DBG] pgmap v2399: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:55.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:55.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:55.741 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:39:55.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:55.787 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:39:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:55.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:39:55.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:55.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:55.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:56.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:56.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:39:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:56.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:39:56.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:56.197 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:56.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:56.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:56.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 10115 10099 15.0259 0 - 0.00317964 2024-02-12T16:39:56.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:56.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:56.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:56.479 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:56.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:56.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:56.619 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:56.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:56.722 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:56.756 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:39:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:56.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:39:56.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:56.891 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:39:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:56.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:39:56.952 INFO:teuthology.orchestra.run.smithi123.stdout:get_pg_num_history pg_num_history pg_num_history(e848 pg_nums {1={663=31},7={752=16},8={839=16}} deleted_pools 833,7) 2024-02-12T16:39:56.952 INFO:teuthology.orchestra.run.smithi123.stdout:Importing pgid 1.0 2024-02-12T16:39:56.952 INFO:teuthology.orchestra.run.smithi123.stdout:write_pg epoch 847 info 1.0( empty local-lis/les=824/825 n=0 ec=14/14 lis/c=824/824 les/c/f=825/825/0 sis=824) 2024-02-12T16:39:56.973 INFO:teuthology.orchestra.run.smithi123.stdout:Import successful 2024-02-12T16:39:56.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:57.026 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:57.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:57.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:57.131 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:56 smithi123 bash[19119]: cluster 2024-02-12T16:39:55.532058+0000 osd.10 (osd.10) 190 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:39:57.131 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:56 smithi123 bash[19119]: cluster 2024-02-12T16:39:55.622726+0000 osd.10 (osd.10) 191 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:39:57.131 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:56 smithi123 bash[19119]: cluster 2024-02-12T16:39:55.900123+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e862: 12 total, 11 up, 12 in 2024-02-12T16:39:57.131 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:56 smithi123 bash[19119]: cluster 2024-02-12T16:39:56.204612+0000 mgr.y (mgr.14146) 1667 : cluster [DBG] pgmap v2401: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:56 smithi121 bash[18489]: cluster 2024-02-12T16:39:55.532058+0000 osd.10 (osd.10) 190 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:39:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:56 smithi121 bash[18489]: cluster 2024-02-12T16:39:55.622726+0000 osd.10 (osd.10) 191 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:39:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:56 smithi121 bash[18489]: cluster 2024-02-12T16:39:55.900123+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e862: 12 total, 11 up, 12 in 2024-02-12T16:39:57.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:56 smithi121 bash[18489]: cluster 2024-02-12T16:39:56.204612+0000 mgr.y (mgr.14146) 1667 : cluster [DBG] pgmap v2401: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:57.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:57.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:57.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:57.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:56 smithi169 bash[19604]: cluster 2024-02-12T16:39:55.532058+0000 osd.10 (osd.10) 190 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:39:57.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:56 smithi169 bash[19604]: cluster 2024-02-12T16:39:55.622726+0000 osd.10 (osd.10) 191 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:39:57.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:56 smithi169 bash[19604]: cluster 2024-02-12T16:39:55.900123+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e862: 12 total, 11 up, 12 in 2024-02-12T16:39:57.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:56 smithi169 bash[19604]: cluster 2024-02-12T16:39:56.204612+0000 mgr.y (mgr.14146) 1667 : cluster [DBG] pgmap v2401: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:57.303 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:57.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:57.310 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 16 10115 10099 14.6765 0 - 0.00317964 2024-02-12T16:39:57.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:57.442 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:39:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:57.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:39:57.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:57.579 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:57.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:57.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:57.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:57.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:57.825 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:57.863 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:57.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:57.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:58.005 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:58.108 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:58.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:57 smithi169 bash[19604]: cluster 2024-02-12T16:39:56.486766+0000 osd.10 (osd.10) 192 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:39:58.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:57 smithi169 bash[19604]: cluster 2024-02-12T16:39:56.521415+0000 osd.10 (osd.10) 193 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:39:58.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:57 smithi169 bash[19604]: cluster 2024-02-12T16:39:56.940537+0000 osd.11 (osd.11) 203 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:39:58.121 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:57 smithi169 bash[19604]: cluster 2024-02-12T16:39:56.998117+0000 osd.11 (osd.11) 204 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:39:58.146 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:58.159 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:57 smithi121 bash[18489]: cluster 2024-02-12T16:39:56.486766+0000 osd.10 (osd.10) 192 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:39:58.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:57 smithi121 bash[18489]: cluster 2024-02-12T16:39:56.521415+0000 osd.10 (osd.10) 193 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:39:58.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:57 smithi121 bash[18489]: cluster 2024-02-12T16:39:56.940537+0000 osd.11 (osd.11) 203 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:39:58.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:57 smithi121 bash[18489]: cluster 2024-02-12T16:39:56.998117+0000 osd.11 (osd.11) 204 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:39:58.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:58.287 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:58.311 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 10115 10099 14.3429 0 - 0.00317964 2024-02-12T16:39:58.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:58.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:58.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:57 smithi123 bash[19119]: cluster 2024-02-12T16:39:56.486766+0000 osd.10 (osd.10) 192 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:39:58.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:57 smithi123 bash[19119]: cluster 2024-02-12T16:39:56.521415+0000 osd.10 (osd.10) 193 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:39:58.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:57 smithi123 bash[19119]: cluster 2024-02-12T16:39:56.940537+0000 osd.11 (osd.11) 203 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:39:58.434 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:57 smithi123 bash[19119]: cluster 2024-02-12T16:39:56.998117+0000 osd.11 (osd.11) 204 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:39:58.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:58.567 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:58.661 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:58.662 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:39:58.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:58.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:58.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/exp.1.0.7 2024-02-12T16:39:58.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:58.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:39:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:39:58.937 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:58 smithi123 bash[19119]: cluster 2024-02-12T16:39:58.205644+0000 mgr.y (mgr.14146) 1668 : cluster [DBG] pgmap v2402: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:58.952 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:58.988 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:39:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:58.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:39:59.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:59.106 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:58 smithi121 bash[18489]: cluster 2024-02-12T16:39:58.205644+0000 mgr.y (mgr.14146) 1668 : cluster [DBG] pgmap v2402: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:59.133 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:39:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:59.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:39:59.236 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:59.249 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:58 smithi169 bash[19604]: cluster 2024-02-12T16:39:58.205644+0000 mgr.y (mgr.14146) 1668 : cluster [DBG] pgmap v2402: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:39:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:59.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:59.311 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 10115 10099 14.0242 0 - 0.00317964 2024-02-12T16:39:59.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:59.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:39:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:59.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:39:59.515 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:59.520 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:39:59.520 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:39:59.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:39:59.520 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:39:59.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.520 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:101283 6.2 6:4b93b79d:::benchmark_data_smithi169_96224_object101282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e656)", 2024-02-12T16:39:59.520 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.471525+0000", 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 590.04211767000004, 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20611299199999999, 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 101283 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471525+0000", 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471525+0000", 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999943 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471520+0000", 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4875999999999993e-05 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471585+0000", 2024-02-12T16:39:59.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3581999999999998e-05 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471618+0000", 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8272999999999999e-05 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471636+0000", 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0072000000000003e-05 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471707+0000", 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2405999999999996e-05 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471799+0000", 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010306699999999999 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.471902+0000", 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032254499999999999 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475127+0000", 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20228344200000001 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677411+0000", 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017230699999999999 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677583+0000", 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5136999999999997e-05 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.677638+0000", 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20611299199999999 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26257.0:101283 6.2 e656/636 ondisk, result = 0)", 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:30:09.475050+0000", 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 590.03859252100006, 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19416351400000001, 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.526 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475050+0000", 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475050+0000", 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475049+0000", 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9430000000000006e-06 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475058+0000", 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5759999999999995e-06 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475067+0000", 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3329999999999996e-06 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475073+0000", 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1579999999999998e-05 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475114+0000", 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6719999999999998e-06 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.475121+0000", 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19409317500000001 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:30:09.669214+0000", 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19416351400000001 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:179498 6.2 6.b3092d82 (undecoded) ondisk+retry+write+known_if_redirected e680)", 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.755414+0000", 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 510.75822912400002, 2024-02-12T16:39:59.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22293095900000001, 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 179498 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755414+0000", 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755414+0000", 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755412+0000", 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0990000000000001e-06 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755419+0000", 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0995999999999999e-05 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755430+0000", 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2589999999999997e-06 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755436+0000", 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2981000000000001e-05 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755479+0000", 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.755485+0000", 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22271555600000001 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978201+0000", 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144134 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978345+0000", 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.22293095900000001 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:179509 6.12 6:49413315:::benchmark_data_smithi169_96224_object26673:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.759446+0000", 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 510.75419721700001, 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0895270889999999, 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 179509 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759446+0000", 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759446+0000", 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759444+0000", 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5679999999999996e-06 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759450+0000", 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7540000000000001e-06 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759460+0000", 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.801e-06 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.759466+0000", 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21890232700000001 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978368+0000", 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3490000000000003e-06 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.978376+0000", 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.85934620900000003 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.837722+0000", 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5047000000000002e-05 2024-02-12T16:39:59.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.837797+0000", 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4651e-05 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.837882+0000", 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042455849999999996 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.842127+0000", 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067655409999999999 2024-02-12T16:39:59.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.848893+0000", 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5303999999999999e-05 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.848938+0000", 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4851000000000002e-05 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.848973+0000", 2024-02-12T16:39:59.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0895270889999999 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:263648 6.2 6:448cfbca:::benchmark_data_smithi169_96224_object110812:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:48.072263+0000", 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 431.44137953199998, 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49358639599999998, 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.541 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 263648 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072263+0000", 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072263+0000", 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072260+0000", 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1686e-05 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072272+0000", 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1992e-05 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072284+0000", 2024-02-12T16:39:59.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4900000000000006e-06 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072294+0000", 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021438499999999999 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072508+0000", 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000105101 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.072613+0000", 2024-02-12T16:39:59.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058748800000000005 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.073201+0000", 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007237683 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.080438+0000", 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.485344098 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.565782+0000", 2024-02-12T16:39:59.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270000000000002e-05 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.565822+0000", 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8271000000000001e-05 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.565850+0000", 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.49358639599999998 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:263673 6.2 6:443ac971:::benchmark_data_smithi169_96224_object110837:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:48.103119+0000", 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 431.41052393299998, 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47670784100000002, 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 263673 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103119+0000", 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103119+0000", 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103117+0000", 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5930000000000002e-06 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103124+0000", 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2967e-05 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103137+0000", 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1629999999999999e-06 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103144+0000", 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3171000000000003e-05 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103197+0000", 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013269399999999999 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.103330+0000", 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00067369999999999995 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.104003+0000", 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002587134 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.106590+0000", 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.473150768 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.579741+0000", 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8149000000000003e-05 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.579789+0000", 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7551e-05 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.579827+0000", 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47670784100000002 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264221 6.2 6:446377cf:::benchmark_data_smithi169_96224_object111385:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.076018+0000", 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 430.43762524900001, 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35090154800000001, 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.552 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264221 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076018+0000", 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076018+0000", 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:59.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076016+0000", 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2480000000000001e-06 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076022+0000", 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3334e-05 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076036+0000", 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2519999999999996e-06 2024-02-12T16:39:59.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076042+0000", 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2206999999999997e-05 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076094+0000", 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125442 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076220+0000", 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068375100000000002 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076903+0000", 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068602649999999999 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.083764+0000", 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34298718500000003 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.426751+0000", 2024-02-12T16:39:59.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1503999999999998e-05 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.426822+0000", 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6923999999999996e-05 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.426919+0000", 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35090154800000001 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.557 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264223 6.2 6:4772561a:::benchmark_data_smithi169_96224_object111387:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.076658+0000", 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 430.436985327, 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35736425399999999, 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264223 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076658+0000", 2024-02-12T16:39:59.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076658+0000", 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076656+0000", 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3090000000000001e-06 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076662+0000", 2024-02-12T16:39:59.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2891000000000001e-05 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076675+0000", 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4729999999999999e-06 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.076681+0000", 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028843200000000001 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.079565+0000", 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000241368 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.079806+0000", 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056161700000000004 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080368+0000", 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076541070000000003 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.088022+0000", 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.345911091 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.433933+0000", 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9709000000000001e-05 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.433983+0000", 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8949000000000003e-05 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434022+0000", 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35736425399999999 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264228 6.2 6:43e9cd2d:::benchmark_data_smithi169_96224_object111392:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.080709+0000", 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 430.43293396000001, 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35371650599999999, 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.563 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264228 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080709+0000", 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080709+0000", 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080707+0000", 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7599999999999997e-06 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080714+0000", 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0532e-05 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080724+0000", 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1240000000000002e-06 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.080731+0000", 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003134768 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.083865+0000", 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107218 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.083973+0000", 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000601188 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.084574+0000", 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003583737 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.088157+0000", 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34615111500000001 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434309+0000", 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0162999999999998e-05 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434389+0000", 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6810999999999997e-05 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.434426+0000", 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35371650599999999 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:207857 7.4 7:27e57e7f:::benchmark_data_smithi169_105252_object39263:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:39:59.568 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:36:30.491943+0000", 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 209.02169961000001, 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19406817300000001, 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207857 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.491943+0000", 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.491943+0000", 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.491941+0000", 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7540000000000006e-06 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.491949+0000", 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1795e-05 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.491961+0000", 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3690000000000003e-06 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.491967+0000", 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9441000000000002e-05 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.492027+0000", 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014318499999999999 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.492170+0000", 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018097799999999999 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.492351+0000", 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055046009999999996 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497855+0000", 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4700000000000008e-06 2024-02-12T16:39:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497865+0000", 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.18779837499999999 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.685663+0000", 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100581 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.685764+0000", 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024777499999999999 2024-02-12T16:39:59.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.686012+0000", 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19406817300000001 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:207867 7.4 7:2ab4f6ad:::benchmark_data_smithi169_105252_object39273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:36:30.497565+0000", 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 209.01607806000001, 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21385816399999999, 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.574 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207867 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497565+0000", 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497565+0000", 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497563+0000", 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1489999999999997e-06 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497570+0000", 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2952999999999999e-05 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497583+0000", 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4647000000000001e-05 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.497598+0000", 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026926239999999998 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.500290+0000", 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000147999 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.500438+0000", 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019308899999999999 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.500631+0000", 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042146670000000001 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.504846+0000", 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124048 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.504970+0000", 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.20624688599999999 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711217+0000", 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2693000000000004e-05 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711280+0000", 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014343099999999999 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711423+0000", 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21385816399999999 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:207875 7.4 7:26b85289:::benchmark_data_smithi169_105252_object39281:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:36:30.507637+0000", 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 209.00600624099999, 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.204049799, 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207875 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.507637+0000", 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.507637+0000", 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.507635+0000", 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8000000000000001e-06 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.507641+0000", 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5616000000000001e-05 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.507657+0000", 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3830000000000003e-06 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.507664+0000", 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087851800000000001 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508543+0000", 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120056 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508663+0000", 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142485 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508805+0000", 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083393849999999995 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.517145+0000", 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4053000000000001e-05 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.517159+0000", 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19434365200000001 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711503+0000", 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5924000000000001e-05 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711568+0000", 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118088 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711687+0000", 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.204049799 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:207878 7.4 7:298b7c63:::benchmark_data_smithi169_105252_object39284:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:36:30.508509+0000", 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 209.00513430300001, 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.203407384, 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207878 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508509+0000", 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508509+0000", 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508507+0000", 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2534999999999999e-05 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508519+0000", 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0671e-05 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508530+0000", 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2280000000000001e-06 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.508537+0000", 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041083990000000004 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.512646+0000", 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103146 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.512749+0000", 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012762900000000001 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.512876+0000", 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043475079999999999 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.517224+0000", 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040583520000000003 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.521282+0000", 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.190468897 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711751+0000", 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2586000000000001e-05 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711804+0000", 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011231 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711916+0000", 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.203407384 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:207881 7.4 7:2985b5b3:::benchmark_data_smithi169_105252_object39287:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:36:30.513406+0000", 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 209.00023741999999, 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19879212199999999, 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.592 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207881 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.513406+0000", 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.513406+0000", 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.513404+0000", 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.438e-06 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.513410+0000", 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2497000000000001e-05 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.513423+0000", 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9429999999999999e-06 2024-02-12T16:39:59.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.513427+0000", 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038343460000000002 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.517262+0000", 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000147255 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.517409+0000", 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019437400000000001 2024-02-12T16:39:59.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.517603+0000", 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007747304 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.525351+0000", 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9047e-05 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.525400+0000", 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.18659250599999999 2024-02-12T16:39:59.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.711992+0000", 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3515000000000004e-05 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.712056+0000", 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014188499999999999 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.712198+0000", 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19879212199999999 2024-02-12T16:39:59.597 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:207891 7.4 7:25f37ef7:::benchmark_data_smithi169_105252_object39297:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:36:30.520085+0000", 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 208.99355767899999, 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19240966400000001, 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.598 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 207891 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.520085+0000", 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.520085+0000", 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.520084+0000", 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5579999999999997e-06 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.520090+0000", 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1482e-05 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.520102+0000", 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0339999999999999e-06 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.520109+0000", 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012181200000000001 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.521327+0000", 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015259599999999999 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.521479+0000", 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018135499999999999 2024-02-12T16:39:59.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.521661+0000", 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037258230000000001 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.525387+0000", 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4406999999999993e-05 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.525471+0000", 2024-02-12T16:39:59.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.186808167 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.712279+0000", 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1198000000000006e-05 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.712340+0000", 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015462700000000001 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:30.712495+0000", 2024-02-12T16:39:59.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19240966400000001 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:283976 7.4 7:2245208a:::benchmark_data_smithi169_105252_object115382:head [delete] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:39.653372+0000", 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 139.860271447, 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046079847, 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 283976 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653372+0000", 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653372+0000", 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653369+0000", 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8270000000000004e-06 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653378+0000", 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6291999999999999e-05 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653394+0000", 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6669999999999999e-06 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653401+0000", 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005732694 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.659134+0000", 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010278 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.659236+0000", 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106172 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.659343+0000", 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038289280000000001 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.663171+0000", 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013068400000000001 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.664478+0000", 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034742268999999999 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.699221+0000", 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5315999999999993e-05 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.699316+0000", 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013549600000000001 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.699451+0000", 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046079847 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:283989 7.4 7:265dc6f4:::benchmark_data_smithi169_105252_object115395:head [delete] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:39.666261+0000", 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 139.84738178999999, 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045926209000000003, 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 283989 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666261+0000", 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666261+0000", 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666260+0000", 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0759999999999997e-06 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666267+0000", 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1304000000000001e-05 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666278+0000", 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.583e-06 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666283+0000", 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025589900000000001 2024-02-12T16:39:59.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.668842+0000", 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014255 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.668984+0000", 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020421400000000001 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669188+0000", 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075747119999999999 2024-02-12T16:39:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.676763+0000", 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035943390000000002 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.680358+0000", 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031702014000000001 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.712060+0000", 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7827999999999997e-05 2024-02-12T16:39:59.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.712107+0000", 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0061000000000005e-05 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.712187+0000", 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045926209000000003 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.615 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:283993 7.4 7:23c03c92:::benchmark_data_smithi169_105252_object115399:head [delete] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:39.669389+0000", 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 139.84425406899999, 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.053297408999999997, 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 283993 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669389+0000", 2024-02-12T16:39:59.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669389+0000", 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669388+0000", 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6469999999999997e-06 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669392+0000", 2024-02-12T16:39:59.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8920000000000006e-06 2024-02-12T16:39:59.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669401+0000", 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.048e-06 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669405+0000", 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033621039999999999 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.672767+0000", 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145298 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.672913+0000", 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191416 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.673104+0000", 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038252529999999998 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.676929+0000", 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003472181 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.680402+0000", 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.042179254999999999 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.722581+0000", 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6111e-05 2024-02-12T16:39:59.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.722617+0000", 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9481999999999996e-05 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.722686+0000", 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.053297408999999997 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.622 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:328656 7.4 7:220a6bfe:::benchmark_data_smithi169_105252_object160062:head [delete] snapc 0=[] ondisk+write+known_if_redirected e825)", 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:38:17.736120+0000", 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 101.77752338099999, 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045947122999999999, 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 328656 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736120+0000", 2024-02-12T16:39:59.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736120+0000", 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736118+0000", 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7559999999999996e-06 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736124+0000", 2024-02-12T16:39:59.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1121000000000001e-05 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736135+0000", 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0599999999999996e-06 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736141+0000", 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2299999999999999e-05 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736174+0000", 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6514999999999993e-05 2024-02-12T16:39:59.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736270+0000", 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6551000000000005e-05 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736357+0000", 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036948409999999999 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.740051+0000", 2024-02-12T16:39:59.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252682 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.753304+0000", 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028601433999999998 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.781906+0000", 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5561e-05 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.781951+0000", 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011567 2024-02-12T16:39:59.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.782067+0000", 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045947122999999999 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:330813 7.4 7:2a76caf2:::benchmark_data_smithi169_105252_object162219:head [delete] snapc 0=[] ondisk+write+known_if_redirected e827)", 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:38:19.515960+0000", 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 99.997682620000006, 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044954721000000003, 2024-02-12T16:39:59.628 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 330813 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515960+0000", 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515960+0000", 2024-02-12T16:39:59.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515959+0000", 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7969999999999997e-06 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515965+0000", 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2486e-05 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515977+0000", 2024-02-12T16:39:59.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0549999999999996e-06 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515983+0000", 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014179647 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.530163+0000", 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015306799999999999 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.530316+0000", 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019076100000000001 2024-02-12T16:39:59.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.530507+0000", 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088279550000000002 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.539335+0000", 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5001999999999997e-05 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.539420+0000", 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.02137379 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.560794+0000", 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4208000000000002e-05 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.560828+0000", 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7287999999999995e-05 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.560915+0000", 2024-02-12T16:39:59.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044954721000000003 2024-02-12T16:39:59.634 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.634 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.634 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.634 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:39:59.634 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:39:59.634 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:39:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:59.635 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:59.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:39:59.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:59.775 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:39:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:59.778 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:39:59.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:59.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:39:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:39:59.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:00.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:59 smithi169 bash[19604]: cluster 2024-02-12T16:39:58.542120+0000 osd.10 (osd.10) 194 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:40:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:59 smithi169 bash[19604]: cluster 2024-02-12T16:39:58.582546+0000 osd.10 (osd.10) 195 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:40:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:59 smithi169 bash[19604]: cluster 2024-02-12T16:39:59.353728+0000 osd.11 (osd.11) 205 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:39:59 smithi169 bash[19604]: cluster 2024-02-12T16:39:59.353823+0000 osd.11 (osd.11) 206 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:00.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:00.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:00.162 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:00.175 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:59 smithi123 bash[19119]: cluster 2024-02-12T16:39:58.542120+0000 osd.10 (osd.10) 194 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:40:00.175 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:59 smithi123 bash[19119]: cluster 2024-02-12T16:39:58.582546+0000 osd.10 (osd.10) 195 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:40:00.175 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:59 smithi123 bash[19119]: cluster 2024-02-12T16:39:59.353728+0000 osd.11 (osd.11) 205 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:00.175 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:39:59 smithi123 bash[19119]: cluster 2024-02-12T16:39:59.353823+0000 osd.11 (osd.11) 206 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:00.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:00.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:00.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:40:00.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:00.311 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 16 10115 10099 13.7193 0 - 0.00317964 2024-02-12T16:40:00.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:59 smithi121 bash[18489]: cluster 2024-02-12T16:39:58.542120+0000 osd.10 (osd.10) 194 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:40:00.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:59 smithi121 bash[18489]: cluster 2024-02-12T16:39:58.582546+0000 osd.10 (osd.10) 195 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:40:00.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:59 smithi121 bash[18489]: cluster 2024-02-12T16:39:59.353728+0000 osd.11 (osd.11) 205 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:00.312 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:39:59 smithi121 bash[18489]: cluster 2024-02-12T16:39:59.353823+0000 osd.11 (osd.11) 206 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:00.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:00.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:00.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:00.479 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:00.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:00.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:00.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:00.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:00.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:00.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:00.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:00.889 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:00.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:00.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:01.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:01.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:01.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:01.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:39:59.539839+0000 osd.5 (osd.5) 187 : cluster [DBG] 8.8 scrub starts 2024-02-12T16:40:01.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:39:59.646881+0000 osd.5 (osd.5) 188 : cluster [DBG] 8.8 scrub ok 2024-02-12T16:40:01.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:39:59.931293+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e863: 12 total, 11 up, 12 in 2024-02-12T16:40:01.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:39:59.967657+0000 osd.8 (osd.8) 115 : cluster [DBG] 8.b scrub starts 2024-02-12T16:40:01.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000194+0000 mon.a (mon.0) 2929 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Reduced data availability: 5 pgs inactive; Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000253+0000 mon.a (mon.0) 2930 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000271+0000 mon.a (mon.0) 2931 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000287+0000 mon.a (mon.0) 2932 : cluster [WRN] osd.7 (root=default,host=smithi123) is down 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000314+0000 mon.a (mon.0) 2933 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000360+0000 mon.a (mon.0) 2934 : cluster [WRN] pg 1.0 is stuck inactive for 18m, current state undersized+peered, last acting [11] 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000398+0000 mon.a (mon.0) 2935 : cluster [WRN] pg 1.2 is stuck inactive for 22m, current state undersized+peered, last acting [0] 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000431+0000 mon.a (mon.0) 2936 : cluster [WRN] pg 1.9 is stuck inactive for 9m, current state undersized+peered, last acting [9] 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000455+0000 mon.a (mon.0) 2937 : cluster [WRN] pg 1.d is stuck inactive for 9m, current state undersized+peered, last acting [4] 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000476+0000 mon.a (mon.0) 2938 : cluster [WRN] pg 1.15 is stuck inactive for 9m, current state undersized+peered, last acting [5] 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000496+0000 mon.a (mon.0) 2939 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.000521+0000 mon.a (mon.0) 2940 : cluster [WRN] pg 8.d is undersized+degraded+peered, acting [11] 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.105615+0000 osd.8 (osd.8) 116 : cluster [DBG] 8.b scrub ok 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.206478+0000 mgr.y (mgr.14146) 1669 : cluster [DBG] pgmap v2404: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:01.148 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.900682+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e864: 12 total, 11 up, 12 in 2024-02-12T16:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:01.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:01.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:01.285 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:39:59.539839+0000 osd.5 (osd.5) 187 : cluster [DBG] 8.8 scrub starts 2024-02-12T16:40:01.285 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:39:59.646881+0000 osd.5 (osd.5) 188 : cluster [DBG] 8.8 scrub ok 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:39:59.931293+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e863: 12 total, 11 up, 12 in 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:39:59.967657+0000 osd.8 (osd.8) 115 : cluster [DBG] 8.b scrub starts 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000194+0000 mon.a (mon.0) 2929 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Reduced data availability: 5 pgs inactive; Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000253+0000 mon.a (mon.0) 2930 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000271+0000 mon.a (mon.0) 2931 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000287+0000 mon.a (mon.0) 2932 : cluster [WRN] osd.7 (root=default,host=smithi123) is down 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000314+0000 mon.a (mon.0) 2933 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000360+0000 mon.a (mon.0) 2934 : cluster [WRN] pg 1.0 is stuck inactive for 18m, current state undersized+peered, last acting [11] 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000398+0000 mon.a (mon.0) 2935 : cluster [WRN] pg 1.2 is stuck inactive for 22m, current state undersized+peered, last acting [0] 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000431+0000 mon.a (mon.0) 2936 : cluster [WRN] pg 1.9 is stuck inactive for 9m, current state undersized+peered, last acting [9] 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000455+0000 mon.a (mon.0) 2937 : cluster [WRN] pg 1.d is stuck inactive for 9m, current state undersized+peered, last acting [4] 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000476+0000 mon.a (mon.0) 2938 : cluster [WRN] pg 1.15 is stuck inactive for 9m, current state undersized+peered, last acting [5] 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000496+0000 mon.a (mon.0) 2939 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.000521+0000 mon.a (mon.0) 2940 : cluster [WRN] pg 8.d is undersized+degraded+peered, acting [11] 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.105615+0000 osd.8 (osd.8) 116 : cluster [DBG] 8.b scrub ok 2024-02-12T16:40:01.286 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.206478+0000 mgr.y (mgr.14146) 1669 : cluster [DBG] pgmap v2404: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:01.287 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.900682+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e864: 12 total, 11 up, 12 in 2024-02-12T16:40:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:39:59.539839+0000 osd.5 (osd.5) 187 : cluster [DBG] 8.8 scrub starts 2024-02-12T16:40:01.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:39:59.646881+0000 osd.5 (osd.5) 188 : cluster [DBG] 8.8 scrub ok 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:39:59.931293+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e863: 12 total, 11 up, 12 in 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:39:59.967657+0000 osd.8 (osd.8) 115 : cluster [DBG] 8.b scrub starts 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000194+0000 mon.a (mon.0) 2929 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Reduced data availability: 5 pgs inactive; Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000253+0000 mon.a (mon.0) 2930 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000271+0000 mon.a (mon.0) 2931 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000287+0000 mon.a (mon.0) 2932 : cluster [WRN] osd.7 (root=default,host=smithi123) is down 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000314+0000 mon.a (mon.0) 2933 : cluster [WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000360+0000 mon.a (mon.0) 2934 : cluster [WRN] pg 1.0 is stuck inactive for 18m, current state undersized+peered, last acting [11] 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000398+0000 mon.a (mon.0) 2935 : cluster [WRN] pg 1.2 is stuck inactive for 22m, current state undersized+peered, last acting [0] 2024-02-12T16:40:01.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000431+0000 mon.a (mon.0) 2936 : cluster [WRN] pg 1.9 is stuck inactive for 9m, current state undersized+peered, last acting [9] 2024-02-12T16:40:01.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000455+0000 mon.a (mon.0) 2937 : cluster [WRN] pg 1.d is stuck inactive for 9m, current state undersized+peered, last acting [4] 2024-02-12T16:40:01.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000476+0000 mon.a (mon.0) 2938 : cluster [WRN] pg 1.15 is stuck inactive for 9m, current state undersized+peered, last acting [5] 2024-02-12T16:40:01.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000496+0000 mon.a (mon.0) 2939 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 570/20198 objects degraded (2.822%), 1 pg degraded 2024-02-12T16:40:01.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.000521+0000 mon.a (mon.0) 2940 : cluster [WRN] pg 8.d is undersized+degraded+peered, acting [11] 2024-02-12T16:40:01.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.105615+0000 osd.8 (osd.8) 116 : cluster [DBG] 8.b scrub ok 2024-02-12T16:40:01.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.206478+0000 mgr.y (mgr.14146) 1669 : cluster [DBG] pgmap v2404: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:01.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.900682+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e864: 12 total, 11 up, 12 in 2024-02-12T16:40:01.311 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 16 10115 10099 13.4274 0 - 0.00317964 2024-02-12T16:40:01.312 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:01.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:01.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:01.457 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:01.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:01.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:01.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:01.704 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:01.740 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:01.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:01.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:01.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:01.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:01.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:01.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:01.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:01 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.570759+0000 osd.10 (osd.10) 196 : cluster [DBG] 8.9 scrub starts 2024-02-12T16:40:01.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:01 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.709849+0000 osd.2 (osd.2) 123 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:40:01.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:01 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.717865+0000 osd.10 (osd.10) 197 : cluster [DBG] 8.9 scrub ok 2024-02-12T16:40:01.993 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:01 smithi123 bash[19119]: cluster 2024-02-12T16:40:00.803483+0000 osd.2 (osd.2) 124 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:40:02.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:02.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:02.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:02.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:01 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.570759+0000 osd.10 (osd.10) 196 : cluster [DBG] 8.9 scrub starts 2024-02-12T16:40:02.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:01 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.709849+0000 osd.2 (osd.2) 123 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:40:02.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:01 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.717865+0000 osd.10 (osd.10) 197 : cluster [DBG] 8.9 scrub ok 2024-02-12T16:40:02.133 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:01 smithi121 bash[18489]: cluster 2024-02-12T16:40:00.803483+0000 osd.2 (osd.2) 124 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:40:02.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:02.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:02.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:02.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:01 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.570759+0000 osd.10 (osd.10) 196 : cluster [DBG] 8.9 scrub starts 2024-02-12T16:40:02.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:01 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.709849+0000 osd.2 (osd.2) 123 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:40:02.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:01 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.717865+0000 osd.10 (osd.10) 197 : cluster [DBG] 8.9 scrub ok 2024-02-12T16:40:02.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:01 smithi169 bash[19604]: cluster 2024-02-12T16:40:00.803483+0000 osd.2 (osd.2) 124 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:40:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:02.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:02.311 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 10115 10099 13.1477 0 - 0.00317964 2024-02-12T16:40:02.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:02.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:02.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:02.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:02.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:02.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:02.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:02.717 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:02.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:02.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:02.860 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:02.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:02.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:03.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:03.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:03.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:03.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:03.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:02 smithi121 bash[18489]: cluster 2024-02-12T16:40:02.207448+0000 mgr.y (mgr.14146) 1670 : cluster [DBG] pgmap v2406: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:03.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:02 smithi121 bash[18489]: cluster 2024-02-12T16:40:02.544802+0000 osd.1 (osd.1) 139 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:40:03.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:02 smithi121 bash[18489]: cluster 2024-02-12T16:40:02.707506+0000 osd.1 (osd.1) 140 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:40:03.292 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:03.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:02 smithi169 bash[19604]: cluster 2024-02-12T16:40:02.207448+0000 mgr.y (mgr.14146) 1670 : cluster [DBG] pgmap v2406: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:02 smithi169 bash[19604]: cluster 2024-02-12T16:40:02.544802+0000 osd.1 (osd.1) 139 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:40:03.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:02 smithi169 bash[19604]: cluster 2024-02-12T16:40:02.707506+0000 osd.1 (osd.1) 140 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:40:03.311 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 10115 10099 12.8794 0 - 0.00317964 2024-02-12T16:40:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:02 smithi123 bash[19119]: cluster 2024-02-12T16:40:02.207448+0000 mgr.y (mgr.14146) 1670 : cluster [DBG] pgmap v2406: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:02 smithi123 bash[19119]: cluster 2024-02-12T16:40:02.544802+0000 osd.1 (osd.1) 139 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:40:03.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:02 smithi123 bash[19119]: cluster 2024-02-12T16:40:02.707506+0000 osd.1 (osd.1) 140 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:40:03.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:03.433 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:03.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:03.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:03.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:03.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:03.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:03.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:03.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:03.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:03.855 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:03.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:03.939 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3] 2024-02-12T16:40:03.940 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:40:03.940 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 8] 2024-02-12T16:40:03.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd out 11 2024-02-12T16:40:03.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:04.014 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:04.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:04.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.839901+0000 mon.a (mon.0) 2942 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=3031727831,config=2903391811,health=2051210885,kv=3472000986,logm=1409215847}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.841411+0000 mon.a (mon.0) 2943 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2613917437}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.842971+0000 mon.a (mon.0) 2944 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2333044274}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.844577+0000 mon.a (mon.0) 2945 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800749582}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.846889+0000 mon.a (mon.0) 2946 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242103240}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.848462+0000 mon.a (mon.0) 2947 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3837993979}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.855346+0000 mon.a (mon.0) 2948 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2773679205}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.860775+0000 mon.a (mon.0) 2949 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370116797}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.865139+0000 mon.a (mon.0) 2950 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814072851}) 2024-02-12T16:40:04.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.869602+0000 mon.a (mon.0) 2951 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879143891}) 2024-02-12T16:40:04.130 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.875434+0000 mon.a (mon.0) 2952 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3226927901}) 2024-02-12T16:40:04.130 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.877130+0000 mon.a (mon.0) 2953 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=15,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=45} crc {logm=1753745927,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2357140987,mon_config_key=918508178}) 2024-02-12T16:40:04.130 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.882073+0000 mon.a (mon.0) 2954 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=557140444}) 2024-02-12T16:40:04.130 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.883757+0000 mon.a (mon.0) 2955 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3384376550}) 2024-02-12T16:40:04.130 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.885386+0000 mon.a (mon.0) 2956 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=318043308,monmap=2720272532,osd_metadata=1720028586,osd_pg_creating=3423679030,osdmap=928983819}) 2024-02-12T16:40:04.130 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:03 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.900213+0000 mon.a (mon.0) 2957 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=861463493}) 2024-02-12T16:40:04.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:04.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:04.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.839901+0000 mon.a (mon.0) 2942 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=3031727831,config=2903391811,health=2051210885,kv=3472000986,logm=1409215847}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.841411+0000 mon.a (mon.0) 2943 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2613917437}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.842971+0000 mon.a (mon.0) 2944 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2333044274}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.844577+0000 mon.a (mon.0) 2945 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800749582}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.846889+0000 mon.a (mon.0) 2946 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242103240}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.848462+0000 mon.a (mon.0) 2947 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3837993979}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.855346+0000 mon.a (mon.0) 2948 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2773679205}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.860775+0000 mon.a (mon.0) 2949 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370116797}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.865139+0000 mon.a (mon.0) 2950 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814072851}) 2024-02-12T16:40:04.268 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.869602+0000 mon.a (mon.0) 2951 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879143891}) 2024-02-12T16:40:04.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.875434+0000 mon.a (mon.0) 2952 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3226927901}) 2024-02-12T16:40:04.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.877130+0000 mon.a (mon.0) 2953 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=15,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=45} crc {logm=1753745927,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2357140987,mon_config_key=918508178}) 2024-02-12T16:40:04.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.882073+0000 mon.a (mon.0) 2954 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=557140444}) 2024-02-12T16:40:04.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.883757+0000 mon.a (mon.0) 2955 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3384376550}) 2024-02-12T16:40:04.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.885386+0000 mon.a (mon.0) 2956 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=318043308,monmap=2720272532,osd_metadata=1720028586,osd_pg_creating=3423679030,osdmap=928983819}) 2024-02-12T16:40:04.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:03 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.900213+0000 mon.a (mon.0) 2957 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=861463493}) 2024-02-12T16:40:04.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:04.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.839901+0000 mon.a (mon.0) 2942 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=3031727831,config=2903391811,health=2051210885,kv=3472000986,logm=1409215847}) 2024-02-12T16:40:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.841411+0000 mon.a (mon.0) 2943 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2613917437}) 2024-02-12T16:40:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.842971+0000 mon.a (mon.0) 2944 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2333044274}) 2024-02-12T16:40:04.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.844577+0000 mon.a (mon.0) 2945 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800749582}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.846889+0000 mon.a (mon.0) 2946 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242103240}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.848462+0000 mon.a (mon.0) 2947 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3837993979}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.855346+0000 mon.a (mon.0) 2948 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2773679205}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.860775+0000 mon.a (mon.0) 2949 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370116797}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.865139+0000 mon.a (mon.0) 2950 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814072851}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.869602+0000 mon.a (mon.0) 2951 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2879143891}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.875434+0000 mon.a (mon.0) 2952 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3226927901}) 2024-02-12T16:40:04.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.877130+0000 mon.a (mon.0) 2953 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=15,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=45} crc {logm=1753745927,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=2357140987,mon_config_key=918508178}) 2024-02-12T16:40:04.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.882073+0000 mon.a (mon.0) 2954 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=557140444}) 2024-02-12T16:40:04.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.883757+0000 mon.a (mon.0) 2955 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3384376550}) 2024-02-12T16:40:04.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.885386+0000 mon.a (mon.0) 2956 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=318043308,monmap=2720272532,osd_metadata=1720028586,osd_pg_creating=3423679030,osdmap=928983819}) 2024-02-12T16:40:04.300 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:03 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.900213+0000 mon.a (mon.0) 2957 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=861463493}) 2024-02-12T16:40:04.312 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 10115 10099 12.6218 0 - 0.00317964 2024-02-12T16:40:04.398 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:04.422 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:40:04.437 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:04.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:04.444 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:40:04.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:40:04.494 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:40:04.523 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:40:04.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:04.547 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:40:04.569 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:40:04.575 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:04.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:04.600 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:40:04.622 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:40:04.653 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:40:04.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:40:04.684 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.684 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.684 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.684 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:40:04.721 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:04.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:04.825 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:04.864 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:04.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:04.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:04.971 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:40:04.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:05.056 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:40:05.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:05.140 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:05.244 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.520492+0000 osd.5 (osd.5) 189 : cluster [DBG] 8.4 scrub starts 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.606564+0000 osd.10 (osd.10) 198 : cluster [DBG] 8.5 scrub starts 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.638117+0000 osd.5 (osd.5) 190 : cluster [DBG] 8.4 scrub ok 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: cluster 2024-02-12T16:40:03.706257+0000 osd.10 (osd.10) 199 : cluster [DBG] 8.5 scrub ok 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: audit 2024-02-12T16:40:04.101573+0000 mon.a (mon.0) 2958 : audit [INF] from='client.26891 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: audit 2024-02-12T16:40:04.103511+0000 mon.b (mon.2) 206 : audit [INF] from='client.? v1:172.21.15.121:0/3796142340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: cluster 2024-02-12T16:40:04.208339+0000 mgr.y (mgr.14146) 1671 : cluster [DBG] pgmap v2407: 47 pgs: 1 active+clean+scrubbing, 1 undersized+degraded+peered, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:05.257 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:04 smithi123 bash[19119]: audit 2024-02-12T16:40:04.392991+0000 mon.a (mon.0) 2959 : audit [DBG] from='client.? v1:172.21.15.121:0/1879882812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:05.284 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.520492+0000 osd.5 (osd.5) 189 : cluster [DBG] 8.4 scrub starts 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.606564+0000 osd.10 (osd.10) 198 : cluster [DBG] 8.5 scrub starts 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.638117+0000 osd.5 (osd.5) 190 : cluster [DBG] 8.4 scrub ok 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: cluster 2024-02-12T16:40:03.706257+0000 osd.10 (osd.10) 199 : cluster [DBG] 8.5 scrub ok 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: audit 2024-02-12T16:40:04.101573+0000 mon.a (mon.0) 2958 : audit [INF] from='client.26891 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: audit 2024-02-12T16:40:04.103511+0000 mon.b (mon.2) 206 : audit [INF] from='client.? v1:172.21.15.121:0/3796142340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: cluster 2024-02-12T16:40:04.208339+0000 mgr.y (mgr.14146) 1671 : cluster [DBG] pgmap v2407: 47 pgs: 1 active+clean+scrubbing, 1 undersized+degraded+peered, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:04 smithi169 bash[19604]: audit 2024-02-12T16:40:04.392991+0000 mon.a (mon.0) 2959 : audit [DBG] from='client.? v1:172.21.15.121:0/1879882812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:05.312 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 10115 10099 12.3743 0 - 0.00317964 2024-02-12T16:40:05.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.520492+0000 osd.5 (osd.5) 189 : cluster [DBG] 8.4 scrub starts 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.606564+0000 osd.10 (osd.10) 198 : cluster [DBG] 8.5 scrub starts 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.638117+0000 osd.5 (osd.5) 190 : cluster [DBG] 8.4 scrub ok 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: cluster 2024-02-12T16:40:03.706257+0000 osd.10 (osd.10) 199 : cluster [DBG] 8.5 scrub ok 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: audit 2024-02-12T16:40:04.101573+0000 mon.a (mon.0) 2958 : audit [INF] from='client.26891 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: audit 2024-02-12T16:40:04.103511+0000 mon.b (mon.2) 206 : audit [INF] from='client.? v1:172.21.15.121:0/3796142340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: cluster 2024-02-12T16:40:04.208339+0000 mgr.y (mgr.14146) 1671 : cluster [DBG] pgmap v2407: 47 pgs: 1 active+clean+scrubbing, 1 undersized+degraded+peered, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:04 smithi121 bash[18489]: audit 2024-02-12T16:40:04.392991+0000 mon.a (mon.0) 2959 : audit [DBG] from='client.? v1:172.21.15.121:0/1879882812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:05.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:05.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:05.564 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:05.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:05.700 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:05.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:05.842 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:05.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:05.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:05.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:40:05.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:40:06.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:06.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:06.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:05 smithi123 bash[19119]: cluster 2024-02-12T16:40:04.951441+0000 mon.a (mon.0) 2960 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:40:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:05 smithi123 bash[19119]: audit 2024-02-12T16:40:04.959150+0000 mon.a (mon.0) 2961 : audit [INF] from='client.26891 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:40:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:05 smithi123 bash[19119]: cluster 2024-02-12T16:40:04.959294+0000 mon.a (mon.0) 2962 : cluster [DBG] osdmap e865: 12 total, 11 up, 12 in 2024-02-12T16:40:06.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:05 smithi123 bash[19119]: cluster 2024-02-12T16:40:05.534503+0000 osd.1 (osd.1) 141 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:40:06.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:05 smithi123 bash[19119]: cluster 2024-02-12T16:40:05.661965+0000 osd.1 (osd.1) 142 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:40:06.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:05 smithi123 bash[19119]: cluster 2024-02-12T16:40:05.906125+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e866: 12 total, 11 up, 12 in 2024-02-12T16:40:06.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:06.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:05 smithi121 bash[18489]: cluster 2024-02-12T16:40:04.951441+0000 mon.a (mon.0) 2960 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:40:06.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:05 smithi121 bash[18489]: audit 2024-02-12T16:40:04.959150+0000 mon.a (mon.0) 2961 : audit [INF] from='client.26891 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:40:06.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:05 smithi121 bash[18489]: cluster 2024-02-12T16:40:04.959294+0000 mon.a (mon.0) 2962 : cluster [DBG] osdmap e865: 12 total, 11 up, 12 in 2024-02-12T16:40:06.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:05 smithi121 bash[18489]: cluster 2024-02-12T16:40:05.534503+0000 osd.1 (osd.1) 141 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:40:06.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:05 smithi121 bash[18489]: cluster 2024-02-12T16:40:05.661965+0000 osd.1 (osd.1) 142 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:40:06.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:05 smithi121 bash[18489]: cluster 2024-02-12T16:40:05.906125+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e866: 12 total, 11 up, 12 in 2024-02-12T16:40:06.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:06.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:06.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:05 smithi169 bash[19604]: cluster 2024-02-12T16:40:04.951441+0000 mon.a (mon.0) 2960 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:40:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:05 smithi169 bash[19604]: audit 2024-02-12T16:40:04.959150+0000 mon.a (mon.0) 2961 : audit [INF] from='client.26891 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:40:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:05 smithi169 bash[19604]: cluster 2024-02-12T16:40:04.959294+0000 mon.a (mon.0) 2962 : cluster [DBG] osdmap e865: 12 total, 11 up, 12 in 2024-02-12T16:40:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:05 smithi169 bash[19604]: cluster 2024-02-12T16:40:05.534503+0000 osd.1 (osd.1) 141 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:40:06.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:05 smithi169 bash[19604]: cluster 2024-02-12T16:40:05.661965+0000 osd.1 (osd.1) 142 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:40:06.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:05 smithi169 bash[19604]: cluster 2024-02-12T16:40:05.906125+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e866: 12 total, 11 up, 12 in 2024-02-12T16:40:06.312 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 16 10115 10099 12.1363 0 - 0.00317964 2024-02-12T16:40:06.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:06.383 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:06.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:06.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:06.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:06.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:06.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:06.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:06.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:06.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:06.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:06.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:06.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:06.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:06 smithi169 bash[19604]: cluster 2024-02-12T16:40:05.752429+0000 osd.4 (osd.4) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:06 smithi169 bash[19604]: cluster 2024-02-12T16:40:05.752542+0000 osd.4 (osd.4) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:06 smithi169 bash[19604]: cluster 2024-02-12T16:40:05.927517+0000 osd.3 (osd.3) 96 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:40:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:06 smithi169 bash[19604]: cluster 2024-02-12T16:40:06.039586+0000 osd.3 (osd.3) 97 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:40:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:06 smithi169 bash[19604]: cluster 2024-02-12T16:40:06.209160+0000 mgr.y (mgr.14146) 1672 : cluster [DBG] pgmap v2410: 47 pgs: 1 undersized+degraded+peered, 1 active+clean+scrubbing, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:07.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:06 smithi169 bash[19604]: cluster 2024-02-12T16:40:06.555694+0000 osd.6 (osd.6) 170 : cluster [DBG] 1.f scrub starts 2024-02-12T16:40:07.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:06 smithi169 bash[19604]: cluster 2024-02-12T16:40:06.662375+0000 osd.6 (osd.6) 171 : cluster [DBG] 1.f scrub ok 2024-02-12T16:40:07.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:07.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:07.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:07.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:07.231 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:07.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:07.312 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 10115 10099 11.9073 0 - 0.00317964 2024-02-12T16:40:07.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:06 smithi121 bash[18489]: cluster 2024-02-12T16:40:05.752429+0000 osd.4 (osd.4) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:06 smithi121 bash[18489]: cluster 2024-02-12T16:40:05.752542+0000 osd.4 (osd.4) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:06 smithi121 bash[18489]: cluster 2024-02-12T16:40:05.927517+0000 osd.3 (osd.3) 96 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:40:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:06 smithi121 bash[18489]: cluster 2024-02-12T16:40:06.039586+0000 osd.3 (osd.3) 97 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:40:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:06 smithi121 bash[18489]: cluster 2024-02-12T16:40:06.209160+0000 mgr.y (mgr.14146) 1672 : cluster [DBG] pgmap v2410: 47 pgs: 1 undersized+degraded+peered, 1 active+clean+scrubbing, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:06 smithi121 bash[18489]: cluster 2024-02-12T16:40:06.555694+0000 osd.6 (osd.6) 170 : cluster [DBG] 1.f scrub starts 2024-02-12T16:40:07.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:06 smithi121 bash[18489]: cluster 2024-02-12T16:40:06.662375+0000 osd.6 (osd.6) 171 : cluster [DBG] 1.f scrub ok 2024-02-12T16:40:07.371 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:07.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:07.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:06 smithi123 bash[19119]: cluster 2024-02-12T16:40:05.752429+0000 osd.4 (osd.4) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:06 smithi123 bash[19119]: cluster 2024-02-12T16:40:05.752542+0000 osd.4 (osd.4) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:06 smithi123 bash[19119]: cluster 2024-02-12T16:40:05.927517+0000 osd.3 (osd.3) 96 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:40:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:06 smithi123 bash[19119]: cluster 2024-02-12T16:40:06.039586+0000 osd.3 (osd.3) 97 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:40:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:06 smithi123 bash[19119]: cluster 2024-02-12T16:40:06.209160+0000 mgr.y (mgr.14146) 1672 : cluster [DBG] pgmap v2410: 47 pgs: 1 undersized+degraded+peered, 1 active+clean+scrubbing, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:06 smithi123 bash[19119]: cluster 2024-02-12T16:40:06.555694+0000 osd.6 (osd.6) 170 : cluster [DBG] 1.f scrub starts 2024-02-12T16:40:07.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:06 smithi123 bash[19119]: cluster 2024-02-12T16:40:06.662375+0000 osd.6 (osd.6) 171 : cluster [DBG] 1.f scrub ok 2024-02-12T16:40:07.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:07.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:07.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:07.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:07.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:07.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:07.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:07.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:07.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:07.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:07.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:07.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:07.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:40:08.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:08.077 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:08.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:08.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:40:08.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:07 smithi121 bash[18489]: cluster 2024-02-12T16:40:06.859904+0000 osd.11 (osd.11) 207 : cluster [DBG] 8.6 scrub starts 2024-02-12T16:40:08.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:07 smithi121 bash[18489]: cluster 2024-02-12T16:40:06.972386+0000 osd.11 (osd.11) 208 : cluster [DBG] 8.6 scrub ok 2024-02-12T16:40:08.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:08.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:08.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:08.225 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:40:08.225 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:40:08.279 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:07 smithi123 bash[19119]: cluster 2024-02-12T16:40:06.859904+0000 osd.11 (osd.11) 207 : cluster [DBG] 8.6 scrub starts 2024-02-12T16:40:08.280 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:07 smithi123 bash[19119]: cluster 2024-02-12T16:40:06.972386+0000 osd.11 (osd.11) 208 : cluster [DBG] 8.6 scrub ok 2024-02-12T16:40:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:07 smithi169 bash[19604]: cluster 2024-02-12T16:40:06.859904+0000 osd.11 (osd.11) 207 : cluster [DBG] 8.6 scrub starts 2024-02-12T16:40:08.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:07 smithi169 bash[19604]: cluster 2024-02-12T16:40:06.972386+0000 osd.11 (osd.11) 208 : cluster [DBG] 8.6 scrub ok 2024-02-12T16:40:08.312 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 10115 10099 11.6868 0 - 0.00317964 2024-02-12T16:40:08.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:08.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:08.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:08.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:08.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.644 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:08.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:08.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:08.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:08.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:08.864 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:40:08.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.929 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:08.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:09.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:09.070 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:09.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:09.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:08 smithi123 bash[19119]: cluster 2024-02-12T16:40:08.209860+0000 mgr.y (mgr.14146) 1673 : cluster [DBG] pgmap v2411: 47 pgs: 1 undersized+degraded+peered, 1 active+clean+scrubbing, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:09.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:09.186 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:08 smithi169 bash[19604]: cluster 2024-02-12T16:40:08.209860+0000 mgr.y (mgr.14146) 1673 : cluster [DBG] pgmap v2411: 47 pgs: 1 undersized+degraded+peered, 1 active+clean+scrubbing, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:09.210 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:09.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:09.312 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 10115 10099 11.4743 0 - 0.00317964 2024-02-12T16:40:09.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:09.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:09.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:08 smithi121 bash[18489]: cluster 2024-02-12T16:40:08.209860+0000 mgr.y (mgr.14146) 1673 : cluster [DBG] pgmap v2411: 47 pgs: 1 undersized+degraded+peered, 1 active+clean+scrubbing, 7 undersized+peered, 38 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:09.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:09.497 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:09.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:09.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:09.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:09.622 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:09.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:09.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:09.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:09.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:09.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:09.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:09.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:09.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:10.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:10.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:10.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:10.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:10.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:10.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:10.312 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 10115 10099 11.2695 0 - 0.00317964 2024-02-12T16:40:10.344 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:10.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:10.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:10.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:10.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:10.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:10.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:10.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:10.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:10.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:10.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:10.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:10.903 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:10.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:11.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:11.044 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:11.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:11.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:11.182 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:11.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:11.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:11.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 10115 10099 11.0717 0 - 0.00317964 2024-02-12T16:40:11.324 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:11.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:11.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:11.464 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:11.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:11 smithi169 bash[19604]: cluster 2024-02-12T16:40:09.797354+0000 osd.11 (osd.11) 209 : cluster [DBG] 8.f scrub starts 2024-02-12T16:40:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:11 smithi169 bash[19604]: cluster 2024-02-12T16:40:09.935998+0000 osd.11 (osd.11) 210 : cluster [DBG] 8.f scrub ok 2024-02-12T16:40:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:11 smithi169 bash[19604]: cluster 2024-02-12T16:40:10.210873+0000 mgr.y (mgr.14146) 1674 : cluster [DBG] pgmap v2412: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:11.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:11.604 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:11.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:11 smithi123 bash[19119]: cluster 2024-02-12T16:40:09.797354+0000 osd.11 (osd.11) 209 : cluster [DBG] 8.f scrub starts 2024-02-12T16:40:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:11 smithi123 bash[19119]: cluster 2024-02-12T16:40:09.935998+0000 osd.11 (osd.11) 210 : cluster [DBG] 8.f scrub ok 2024-02-12T16:40:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:11 smithi123 bash[19119]: cluster 2024-02-12T16:40:10.210873+0000 mgr.y (mgr.14146) 1674 : cluster [DBG] pgmap v2412: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:11 smithi121 bash[18489]: cluster 2024-02-12T16:40:09.797354+0000 osd.11 (osd.11) 209 : cluster [DBG] 8.f scrub starts 2024-02-12T16:40:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:11 smithi121 bash[18489]: cluster 2024-02-12T16:40:09.935998+0000 osd.11 (osd.11) 210 : cluster [DBG] 8.f scrub ok 2024-02-12T16:40:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:11 smithi121 bash[18489]: cluster 2024-02-12T16:40:10.210873+0000 mgr.y (mgr.14146) 1674 : cluster [DBG] pgmap v2412: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:11.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:11.744 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:11.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:11.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:11.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:11.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:12.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:12.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:12.165 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:12.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:12.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:12.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 16 10115 10099 10.8809 0 - 0.00317964 2024-02-12T16:40:12.404 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:12 smithi121 bash[18489]: cluster 2024-02-12T16:40:11.855324+0000 osd.8 (osd.8) 117 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:40:12.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:12 smithi121 bash[18489]: cluster 2024-02-12T16:40:11.950998+0000 osd.8 (osd.8) 118 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:40:12.442 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:12.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:12 smithi169 bash[19604]: cluster 2024-02-12T16:40:11.855324+0000 osd.8 (osd.8) 117 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:40:12.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:12 smithi169 bash[19604]: cluster 2024-02-12T16:40:11.950998+0000 osd.8 (osd.8) 118 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:40:12.583 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:12.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:12 smithi123 bash[19119]: cluster 2024-02-12T16:40:11.855324+0000 osd.8 (osd.8) 117 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:40:12.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:12 smithi123 bash[19119]: cluster 2024-02-12T16:40:11.950998+0000 osd.8 (osd.8) 118 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:40:12.675 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:12.676 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:12.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:12.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:12.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:12.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:12.994 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:12.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:13.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:13.136 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:13.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:13.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:13.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:13.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:13.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 10115 10099 10.6964 0 - 0.00317964 2024-02-12T16:40:13.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:13.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:13 smithi123 bash[19119]: cluster 2024-02-12T16:40:12.211784+0000 mgr.y (mgr.14146) 1675 : cluster [DBG] pgmap v2413: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:13.419 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:13.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:13.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:13.536 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:13 smithi169 bash[19604]: cluster 2024-02-12T16:40:12.211784+0000 mgr.y (mgr.14146) 1675 : cluster [DBG] pgmap v2413: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:13.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:13.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:13.656 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:40:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:13 smithi121 bash[18489]: cluster 2024-02-12T16:40:12.211784+0000 mgr.y (mgr.14146) 1675 : cluster [DBG] pgmap v2413: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:13.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:13.700 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:13.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:13.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:13.841 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:13.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:13.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:13.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:13.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:14.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:14.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:14.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:14.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:14.256 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:14.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:marked out osd.11. 2024-02-12T16:40:14.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:40:14.310346 min lat: 0.00106883 max lat: 0.0956592 avg lat: 0.00317964 2024-02-12T16:40:14.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:40:14.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 10115 10099 10.5182 0 - 0.00317964 2024-02-12T16:40:14.360 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:14.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:14.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:14 smithi121 bash[18489]: debug 2024-02-12T16:40:14.243+0000 7fbba85de700 -1 mon.a@0(leader).osd e866 definitely_dead 0 2024-02-12T16:40:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:14 smithi121 bash[18489]: cluster 2024-02-12T16:40:13.445710+0000 osd.1 (osd.1) 143 : cluster [DBG] 8.1 scrub starts 2024-02-12T16:40:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:14 smithi121 bash[18489]: cluster 2024-02-12T16:40:13.573525+0000 osd.1 (osd.1) 144 : cluster [DBG] 8.1 scrub ok 2024-02-12T16:40:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:14 smithi121 bash[18489]: audit 2024-02-12T16:40:14.250903+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? v1:172.21.15.121:0/2637373996' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-02-12T16:40:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:14 smithi121 bash[18489]: cluster 2024-02-12T16:40:14.251002+0000 mon.a (mon.0) 2965 : cluster [INF] Client client.admin marked osd.11 out, while it was still marked up 2024-02-12T16:40:14.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:14.542 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:14.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:14 smithi169 bash[19604]: cluster 2024-02-12T16:40:13.445710+0000 osd.1 (osd.1) 143 : cluster [DBG] 8.1 scrub starts 2024-02-12T16:40:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:14 smithi169 bash[19604]: cluster 2024-02-12T16:40:13.573525+0000 osd.1 (osd.1) 144 : cluster [DBG] 8.1 scrub ok 2024-02-12T16:40:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:14 smithi169 bash[19604]: audit 2024-02-12T16:40:14.250903+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? v1:172.21.15.121:0/2637373996' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-02-12T16:40:14.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:14 smithi169 bash[19604]: cluster 2024-02-12T16:40:14.251002+0000 mon.a (mon.0) 2965 : cluster [INF] Client client.admin marked osd.11 out, while it was still marked up 2024-02-12T16:40:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:14 smithi123 bash[19119]: cluster 2024-02-12T16:40:13.445710+0000 osd.1 (osd.1) 143 : cluster [DBG] 8.1 scrub starts 2024-02-12T16:40:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:14 smithi123 bash[19119]: cluster 2024-02-12T16:40:13.573525+0000 osd.1 (osd.1) 144 : cluster [DBG] 8.1 scrub ok 2024-02-12T16:40:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:14 smithi123 bash[19119]: audit 2024-02-12T16:40:14.250903+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? v1:172.21.15.121:0/2637373996' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-02-12T16:40:14.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:14 smithi123 bash[19119]: cluster 2024-02-12T16:40:14.251002+0000 mon.a (mon.0) 2965 : cluster [INF] Client client.admin marked osd.11 out, while it was still marked up 2024-02-12T16:40:14.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:14.684 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:14.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:14.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:14.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:14.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:14.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:14.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:14.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:15.072 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:15.110 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:15.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:15.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:15.251 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:15.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:15.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 10115 10099 10.3457 0 - 0.00317964 2024-02-12T16:40:15.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:15.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:15.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:15.487 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:15.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:15.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:15.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:15.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:15 smithi121 bash[18489]: cluster 2024-02-12T16:40:14.212655+0000 mgr.y (mgr.14146) 1676 : cluster [DBG] pgmap v2414: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:15.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:15 smithi121 bash[18489]: audit 2024-02-12T16:40:14.293838+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? v1:172.21.15.121:0/2637373996' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-02-12T16:40:15.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:15 smithi121 bash[18489]: cluster 2024-02-12T16:40:14.293928+0000 mon.a (mon.0) 2967 : cluster [DBG] osdmap e867: 12 total, 11 up, 11 in 2024-02-12T16:40:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:15 smithi123 bash[19119]: cluster 2024-02-12T16:40:14.212655+0000 mgr.y (mgr.14146) 1676 : cluster [DBG] pgmap v2414: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:15 smithi123 bash[19119]: audit 2024-02-12T16:40:14.293838+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? v1:172.21.15.121:0/2637373996' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-02-12T16:40:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:15 smithi123 bash[19119]: cluster 2024-02-12T16:40:14.293928+0000 mon.a (mon.0) 2967 : cluster [DBG] osdmap e867: 12 total, 11 up, 11 in 2024-02-12T16:40:15.648 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:15.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:15.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:15.759 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:15 smithi169 bash[19604]: cluster 2024-02-12T16:40:14.212655+0000 mgr.y (mgr.14146) 1676 : cluster [DBG] pgmap v2414: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:15.759 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:15 smithi169 bash[19604]: audit 2024-02-12T16:40:14.293838+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? v1:172.21.15.121:0/2637373996' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-02-12T16:40:15.759 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:15 smithi169 bash[19604]: cluster 2024-02-12T16:40:14.293928+0000 mon.a (mon.0) 2967 : cluster [DBG] osdmap e867: 12 total, 11 up, 11 in 2024-02-12T16:40:15.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:15.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:15.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:15.901 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:15.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:16.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:16.037 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:16.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:16.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:16.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 10115 10099 10.1789 0 - 0.00317964 2024-02-12T16:40:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:16.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:16.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:16.560 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:16 smithi121 bash[18489]: cluster 2024-02-12T16:40:15.294411+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e868: 12 total, 11 up, 11 in 2024-02-12T16:40:16.560 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:16 smithi121 bash[18489]: cluster 2024-02-12T16:40:16.295599+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e869: 12 total, 11 up, 11 in 2024-02-12T16:40:16.566 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:16 smithi123 bash[19119]: cluster 2024-02-12T16:40:15.294411+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e868: 12 total, 11 up, 11 in 2024-02-12T16:40:16.567 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:16 smithi123 bash[19119]: cluster 2024-02-12T16:40:16.295599+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e869: 12 total, 11 up, 11 in 2024-02-12T16:40:16.584 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:16.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:16.714 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:40:16.714 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:40:16.723 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:16.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:16 smithi169 bash[19604]: cluster 2024-02-12T16:40:15.294411+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e868: 12 total, 11 up, 11 in 2024-02-12T16:40:16.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:16 smithi169 bash[19604]: cluster 2024-02-12T16:40:16.295599+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e869: 12 total, 11 up, 11 in 2024-02-12T16:40:16.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:16.854 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:16.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:16.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:16.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:16.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:17.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:17.124 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:17.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:17.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:17.254 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:17.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:17.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 15 13076 13061 12.9554 3.08542 0.00216739 0.0771826 2024-02-12T16:40:17.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:17.380 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:17.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:17.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:17.490 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:17 smithi123 bash[19119]: cluster 2024-02-12T16:40:16.213246+0000 mgr.y (mgr.14146) 1677 : cluster [DBG] pgmap v2417: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:17.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:17.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:17.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:40:17.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:17.618 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:17 smithi121 bash[18489]: cluster 2024-02-12T16:40:16.213246+0000 mgr.y (mgr.14146) 1677 : cluster [DBG] pgmap v2417: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:17.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:17.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:17.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:17.753 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:17 smithi169 bash[19604]: cluster 2024-02-12T16:40:16.213246+0000 mgr.y (mgr.14146) 1677 : cluster [DBG] pgmap v2417: 47 pgs: 1 undersized+degraded+peered, 7 undersized+peered, 39 active+clean; 631 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 570/20198 objects degraded (2.822%) 2024-02-12T16:40:17.777 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:17.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:17.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:17.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:17.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:18.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:18.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:18.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:18.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:18.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:18.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:18.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:18.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 17413 17397 16.9867 271 0.00610722 0.0588649 2024-02-12T16:40:18.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:18.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:18.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:18.477 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:18.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:18.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:18.587 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:18 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.303916+0000 mon.a (mon.0) 2970 : cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:40:18.588 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:18 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.303955+0000 mon.a (mon.0) 2971 : cluster [WRN] Health check update: Degraded data redundancy: 570/20916 objects degraded (2.725%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:18.588 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:18 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.313177+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e870: 12 total, 11 up, 11 in 2024-02-12T16:40:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:18 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.303916+0000 mon.a (mon.0) 2970 : cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:40:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:18 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.303955+0000 mon.a (mon.0) 2971 : cluster [WRN] Health check update: Degraded data redundancy: 570/20916 objects degraded (2.725%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:18 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.313177+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e870: 12 total, 11 up, 11 in 2024-02-12T16:40:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:18 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.303916+0000 mon.a (mon.0) 2970 : cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:40:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:18 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.303955+0000 mon.a (mon.0) 2971 : cluster [WRN] Health check update: Degraded data redundancy: 570/20916 objects degraded (2.725%), 1 pg degraded, 8 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:18 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.313177+0000 mon.a (mon.0) 2972 : cluster [DBG] osdmap e870: 12 total, 11 up, 11 in 2024-02-12T16:40:18.829 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:18.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:18.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:18.959 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:18.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:19.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:19.091 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:19.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:19.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:19.230 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:19.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:19.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 21521 21505 20.6748 256.75 0.00516581 0.0483653 2024-02-12T16:40:19.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:19.362 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:19.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:19.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:19.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:19.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:19.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:19.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:19 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.213925+0000 mgr.y (mgr.14146) 1678 : cluster [DBG] pgmap v2419: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 1 undersized+degraded+peered, 7 undersized+peered, 35 active+clean; 654 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 3.7 MiB/s wr, 59 op/s; 570/20916 objects degraded (2.725%); 821/20916 objects misplaced (3.925%); 11 MiB/s, 169 objects/s recovering 2024-02-12T16:40:19.597 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:19 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.274153+0000 osd.8 (osd.8) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:19.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:19 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.274210+0000 osd.8 (osd.8) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:19.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:19 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.289432+0000 osd.4 (osd.4) 151 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:40:19.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:19 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.349201+0000 osd.4 (osd.4) 152 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:40:19.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:19 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.634034+0000 osd.0 (osd.0) 147 : cluster [DBG] 1.b scrub starts 2024-02-12T16:40:19.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:19 smithi121 bash[18489]: cluster 2024-02-12T16:40:18.778795+0000 osd.0 (osd.0) 148 : cluster [DBG] 1.b scrub ok 2024-02-12T16:40:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:19.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:19 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.213925+0000 mgr.y (mgr.14146) 1678 : cluster [DBG] pgmap v2419: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 1 undersized+degraded+peered, 7 undersized+peered, 35 active+clean; 654 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 3.7 MiB/s wr, 59 op/s; 570/20916 objects degraded (2.725%); 821/20916 objects misplaced (3.925%); 11 MiB/s, 169 objects/s recovering 2024-02-12T16:40:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:19 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.274153+0000 osd.8 (osd.8) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:19 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.274210+0000 osd.8 (osd.8) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:19 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.289432+0000 osd.4 (osd.4) 151 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:40:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:19 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.349201+0000 osd.4 (osd.4) 152 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:40:19.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:19 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.634034+0000 osd.0 (osd.0) 147 : cluster [DBG] 1.b scrub starts 2024-02-12T16:40:19.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:19 smithi123 bash[19119]: cluster 2024-02-12T16:40:18.778795+0000 osd.0 (osd.0) 148 : cluster [DBG] 1.b scrub ok 2024-02-12T16:40:19.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:19.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:19.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:19 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.213925+0000 mgr.y (mgr.14146) 1678 : cluster [DBG] pgmap v2419: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 1 undersized+degraded+peered, 7 undersized+peered, 35 active+clean; 654 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 3.7 MiB/s wr, 59 op/s; 570/20916 objects degraded (2.725%); 821/20916 objects misplaced (3.925%); 11 MiB/s, 169 objects/s recovering 2024-02-12T16:40:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:19 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.274153+0000 osd.8 (osd.8) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:19 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.274210+0000 osd.8 (osd.8) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:19 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.289432+0000 osd.4 (osd.4) 151 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:40:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:19 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.349201+0000 osd.4 (osd.4) 152 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:40:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:19 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.634034+0000 osd.0 (osd.0) 147 : cluster [DBG] 1.b scrub starts 2024-02-12T16:40:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:19 smithi169 bash[19604]: cluster 2024-02-12T16:40:18.778795+0000 osd.0 (osd.0) 148 : cluster [DBG] 1.b scrub ok 2024-02-12T16:40:19.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:19.877 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:19.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:19.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:20.015 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:20.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:20.144 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:20.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:20.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:20.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 25074 25058 23.7256 222.062 0.00191189 0.0421185 2024-02-12T16:40:20.372 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:20 smithi121 bash[18489]: cluster 2024-02-12T16:40:19.461194+0000 osd.5 (osd.5) 191 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:40:20.372 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:20 smithi121 bash[18489]: cluster 2024-02-12T16:40:19.529302+0000 osd.5 (osd.5) 192 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:40:20.372 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:20 smithi121 bash[18489]: cluster 2024-02-12T16:40:19.623400+0000 osd.6 (osd.6) 172 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:40:20.372 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:20 smithi121 bash[18489]: cluster 2024-02-12T16:40:19.721109+0000 osd.6 (osd.6) 173 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:40:20.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:20.408 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:20.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:20.525 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:40:20.543 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:20 smithi169 bash[19604]: cluster 2024-02-12T16:40:19.461194+0000 osd.5 (osd.5) 191 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:40:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:20 smithi169 bash[19604]: cluster 2024-02-12T16:40:19.529302+0000 osd.5 (osd.5) 192 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:40:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:20 smithi169 bash[19604]: cluster 2024-02-12T16:40:19.623400+0000 osd.6 (osd.6) 172 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:40:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:20 smithi169 bash[19604]: cluster 2024-02-12T16:40:19.721109+0000 osd.6 (osd.6) 173 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:40:20.566 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:40:20.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:40:20.632 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:40:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:20 smithi123 bash[19119]: cluster 2024-02-12T16:40:19.461194+0000 osd.5 (osd.5) 191 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:40:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:20 smithi123 bash[19119]: cluster 2024-02-12T16:40:19.529302+0000 osd.5 (osd.5) 192 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:40:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:20 smithi123 bash[19119]: cluster 2024-02-12T16:40:19.623400+0000 osd.6 (osd.6) 172 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:40:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:20 smithi123 bash[19119]: cluster 2024-02-12T16:40:19.721109+0000 osd.6 (osd.6) 173 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:40:20.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:20.666 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:40:20.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:20.696 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:40:20.722 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:40:20.753 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:40:20.771 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:20.784 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:40:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:20.810 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:40:20.840 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.841 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:40:20.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:20.942 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:20.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:21.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:21.070 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:21.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:21.177 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8] out_osds: [11] dead_osds: [7] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3] 2024-02-12T16:40:21.178 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:40:21.178 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-02-12T16:40:21.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:40:21.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:21.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:21.313 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 28323 28307 26.4019 203.062 0.00373593 0.0378736 2024-02-12T16:40:21.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:21.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:21.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:21.594 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:21 smithi123 bash[19119]: cluster 2024-02-12T16:40:20.214571+0000 mgr.y (mgr.14146) 1679 : cluster [DBG] pgmap v2421: 47 pgs: 3 active+clean+remapped, 1 active+clean+scrubbing, 3 active+recovering, 7 undersized+peered, 1 active+recovering+degraded+remapped, 32 active+clean; 935 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 51 MiB/s wr, 820 op/s; 1563/29914 objects degraded (5.225%); 2498/29914 objects misplaced (8.351%); 11 MiB/s, 179 objects/s recovering 2024-02-12T16:40:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:21 smithi123 bash[19119]: audit 2024-02-12T16:40:20.488828+0000 mon.c (mon.1) 105 : audit [DBG] from='client.? v1:172.21.15.121:0/3916516753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:21 smithi123 bash[19119]: cluster 2024-02-12T16:40:20.679230+0000 osd.10 (osd.10) 200 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:40:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:21 smithi123 bash[19119]: cluster 2024-02-12T16:40:20.722814+0000 osd.10 (osd.10) 201 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:40:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:21 smithi123 bash[19119]: audit 2024-02-12T16:40:21.309693+0000 mon.a (mon.0) 2973 : audit [INF] from='client.26915 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:40:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:21 smithi123 bash[19119]: audit 2024-02-12T16:40:21.311618+0000 mon.b (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.121:0/3614204608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:40:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:21 smithi121 bash[18489]: cluster 2024-02-12T16:40:20.214571+0000 mgr.y (mgr.14146) 1679 : cluster [DBG] pgmap v2421: 47 pgs: 3 active+clean+remapped, 1 active+clean+scrubbing, 3 active+recovering, 7 undersized+peered, 1 active+recovering+degraded+remapped, 32 active+clean; 935 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 51 MiB/s wr, 820 op/s; 1563/29914 objects degraded (5.225%); 2498/29914 objects misplaced (8.351%); 11 MiB/s, 179 objects/s recovering 2024-02-12T16:40:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:21 smithi121 bash[18489]: audit 2024-02-12T16:40:20.488828+0000 mon.c (mon.1) 105 : audit [DBG] from='client.? v1:172.21.15.121:0/3916516753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:21 smithi121 bash[18489]: cluster 2024-02-12T16:40:20.679230+0000 osd.10 (osd.10) 200 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:40:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:21 smithi121 bash[18489]: cluster 2024-02-12T16:40:20.722814+0000 osd.10 (osd.10) 201 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:40:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:21 smithi121 bash[18489]: audit 2024-02-12T16:40:21.309693+0000 mon.a (mon.0) 2973 : audit [INF] from='client.26915 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:40:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:21 smithi121 bash[18489]: audit 2024-02-12T16:40:21.311618+0000 mon.b (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.121:0/3614204608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:40:21.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:21.720 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:21 smithi169 bash[19604]: cluster 2024-02-12T16:40:20.214571+0000 mgr.y (mgr.14146) 1679 : cluster [DBG] pgmap v2421: 47 pgs: 3 active+clean+remapped, 1 active+clean+scrubbing, 3 active+recovering, 7 undersized+peered, 1 active+recovering+degraded+remapped, 32 active+clean; 935 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 51 MiB/s wr, 820 op/s; 1563/29914 objects degraded (5.225%); 2498/29914 objects misplaced (8.351%); 11 MiB/s, 179 objects/s recovering 2024-02-12T16:40:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:21 smithi169 bash[19604]: audit 2024-02-12T16:40:20.488828+0000 mon.c (mon.1) 105 : audit [DBG] from='client.? v1:172.21.15.121:0/3916516753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:21 smithi169 bash[19604]: cluster 2024-02-12T16:40:20.679230+0000 osd.10 (osd.10) 200 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:40:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:21 smithi169 bash[19604]: cluster 2024-02-12T16:40:20.722814+0000 osd.10 (osd.10) 201 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:40:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:21 smithi169 bash[19604]: audit 2024-02-12T16:40:21.309693+0000 mon.a (mon.0) 2973 : audit [INF] from='client.26915 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:40:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:21 smithi169 bash[19604]: audit 2024-02-12T16:40:21.311618+0000 mon.b (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.121:0/3614204608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:40:21.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:21.844 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:21.927 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:40:21.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:21.970 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:21.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:22.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:22.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:22.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:22.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:22.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:22.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:40:22.234 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:40:22.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 15 31852 31837 29.2576 220.625 0.0022827 0.0341768 2024-02-12T16:40:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:40:22.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:22.351 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:22.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:22.375 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:22 smithi121 bash[18489]: cluster 2024-02-12T16:40:21.911585+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:22.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:22.492 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:22.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:22.595 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:22.625 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:22.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:22.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:22 smithi123 bash[19119]: cluster 2024-02-12T16:40:21.911585+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:22.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:22 smithi123 bash[19119]: audit 2024-02-12T16:40:21.917748+0000 mon.a (mon.0) 2975 : audit [INF] from='client.26915 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:40:22.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:22 smithi123 bash[19119]: cluster 2024-02-12T16:40:21.917840+0000 mon.a (mon.0) 2976 : cluster [DBG] osdmap e871: 12 total, 11 up, 11 in 2024-02-12T16:40:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:22 smithi121 bash[18489]: audit 2024-02-12T16:40:21.917748+0000 mon.a (mon.0) 2975 : audit [INF] from='client.26915 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:40:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:22 smithi121 bash[18489]: cluster 2024-02-12T16:40:21.917840+0000 mon.a (mon.0) 2976 : cluster [DBG] osdmap e871: 12 total, 11 up, 11 in 2024-02-12T16:40:22.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:22.736 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:22 smithi169 bash[19604]: cluster 2024-02-12T16:40:21.911585+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:22.737 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:22 smithi169 bash[19604]: audit 2024-02-12T16:40:21.917748+0000 mon.a (mon.0) 2975 : audit [INF] from='client.26915 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:40:22.737 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:22 smithi169 bash[19604]: cluster 2024-02-12T16:40:21.917840+0000 mon.a (mon.0) 2976 : cluster [DBG] osdmap e871: 12 total, 11 up, 11 in 2024-02-12T16:40:22.758 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:22.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:22.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:22.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:22.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:22.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:23.029 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:23.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:23.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:23.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:23.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:23.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 34750 34734 31.4574 181.062 0.00657204 0.031787 2024-02-12T16:40:23.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:23.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:23.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:23.539 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:23 smithi169 bash[19604]: cluster 2024-02-12T16:40:22.215161+0000 mgr.y (mgr.14146) 1680 : cluster [DBG] pgmap v2423: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 1.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 160 MiB/s wr, 2.56k op/s; 1563/50956 objects degraded (3.067%); 1677/50956 objects misplaced (3.291%); 20 MiB/s, 313 objects/s recovering 2024-02-12T16:40:23.561 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:23 smithi123 bash[19119]: cluster 2024-02-12T16:40:22.215161+0000 mgr.y (mgr.14146) 1680 : cluster [DBG] pgmap v2423: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 1.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 160 MiB/s wr, 2.56k op/s; 1563/50956 objects degraded (3.067%); 1677/50956 objects misplaced (3.291%); 20 MiB/s, 313 objects/s recovering 2024-02-12T16:40:23.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:23 smithi121 bash[18489]: cluster 2024-02-12T16:40:22.215161+0000 mgr.y (mgr.14146) 1680 : cluster [DBG] pgmap v2423: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 1.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 160 MiB/s wr, 2.56k op/s; 1563/50956 objects degraded (3.067%); 1677/50956 objects misplaced (3.291%); 20 MiB/s, 313 objects/s recovering 2024-02-12T16:40:23.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:23.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:23.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:23.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:23.922 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:23.944 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:23.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:24.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:24.072 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:24.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:24.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:24.205 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:24.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:24.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:40:24.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:24.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 16 37980 37964 33.8915 201.875 0.00491259 0.0295034 2024-02-12T16:40:24.354 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:24.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:24.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:24.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:24.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:24.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:24.618 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:24.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:24.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:24 smithi121 bash[18489]: cluster 2024-02-12T16:40:24.373038+0000 mon.a (mon.0) 2977 : cluster [WRN] Health check update: Degraded data redundancy: 1563/63432 objects degraded (2.464%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:24.694 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:24 smithi123 bash[19119]: cluster 2024-02-12T16:40:24.373038+0000 mon.a (mon.0) 2977 : cluster [WRN] Health check update: Degraded data redundancy: 1563/63432 objects degraded (2.464%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:24.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:24.728 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:24 smithi169 bash[19604]: cluster 2024-02-12T16:40:24.373038+0000 mon.a (mon.0) 2977 : cluster [WRN] Health check update: Degraded data redundancy: 1563/63432 objects degraded (2.464%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:24.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:24.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:24.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:24.878 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:24.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:24.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:25.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:25.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:40:25.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:25.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:25.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:25.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:25.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 40834 40818 35.9261 178.375 0.00437273 0.0278326 2024-02-12T16:40:25.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:25.402 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:25.433 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: cluster 2024-02-12T16:40:24.215698+0000 mgr.y (mgr.14146) 1681 : cluster [DBG] pgmap v2424: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s; 1563/63432 objects degraded (2.464%); 2736/63432 objects misplaced (4.313%); 15 MiB/s, 237 objects/s recovering 2024-02-12T16:40:25.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:25.538 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: cluster 2024-02-12T16:40:24.215698+0000 mgr.y (mgr.14146) 1681 : cluster [DBG] pgmap v2424: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s; 1563/63432 objects degraded (2.464%); 2736/63432 objects misplaced (4.313%); 15 MiB/s, 237 objects/s recovering 2024-02-12T16:40:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.871499+0000 mon.b (mon.2) 208 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:40:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.871633+0000 mon.b (mon.2) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:40:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.872111+0000 mon.b (mon.2) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:40:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.873129+0000 mon.a (mon.0) 2978 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:40:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.891288+0000 mon.b (mon.2) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:40:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.893272+0000 mon.b (mon.2) 212 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:40:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.894263+0000 mon.b (mon.2) 213 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:40:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:25 smithi123 bash[19119]: audit 2024-02-12T16:40:24.899356+0000 mon.a (mon.0) 2979 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:40:25.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:25.669 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:25.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:25.780 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.871499+0000 mon.b (mon.2) 208 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:40:25.780 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.871633+0000 mon.b (mon.2) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:40:25.780 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.872111+0000 mon.b (mon.2) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:40:25.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.873129+0000 mon.a (mon.0) 2978 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:40:25.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.891288+0000 mon.b (mon.2) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:40:25.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.893272+0000 mon.b (mon.2) 212 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:40:25.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.894263+0000 mon.b (mon.2) 213 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:40:25.781 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:25 smithi169 bash[19604]: audit 2024-02-12T16:40:24.899356+0000 mon.a (mon.0) 2979 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:40:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:25.905 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: cluster 2024-02-12T16:40:24.215698+0000 mgr.y (mgr.14146) 1681 : cluster [DBG] pgmap v2424: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s; 1563/63432 objects degraded (2.464%); 2736/63432 objects misplaced (4.313%); 15 MiB/s, 237 objects/s recovering 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.871499+0000 mon.b (mon.2) 208 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.871633+0000 mon.b (mon.2) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.872111+0000 mon.b (mon.2) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.873129+0000 mon.a (mon.0) 2978 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.891288+0000 mon.b (mon.2) 211 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.893272+0000 mon.b (mon.2) 212 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.894263+0000 mon.b (mon.2) 213 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:40:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:25 smithi121 bash[18489]: audit 2024-02-12T16:40:24.899356+0000 mon.a (mon.0) 2979 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:40:25.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:25.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:26.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:26.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:26.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:26.183 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:26.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:26.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 43677 43661 37.8947 177.688 0.00604799 0.0263869 2024-02-12T16:40:26.318 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:26.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:26.448 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:26.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:26.588 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:26.589 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:26.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:26.718 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:26.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:26.850 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:26.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:26.983 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:26.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:27.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:27.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:26 smithi169 bash[19604]: cluster 2024-02-12T16:40:25.382091+0000 osd.9 (osd.9) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:27.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:26 smithi169 bash[19604]: cluster 2024-02-12T16:40:25.382141+0000 osd.9 (osd.9) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:27.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:26 smithi169 bash[19604]: cluster 2024-02-12T16:40:25.902449+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e872: 12 total, 11 up, 11 in 2024-02-12T16:40:27.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:26 smithi169 bash[19604]: cluster 2024-02-12T16:40:25.903843+0000 mon.a (mon.0) 2981 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:40:27.092 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:26 smithi169 bash[19604]: cluster 2024-02-12T16:40:26.216389+0000 mgr.y (mgr.14146) 1682 : cluster [DBG] pgmap v2426: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 176 MiB/s wr, 2.82k op/s; 2594/65494 objects degraded (3.961%); 2678/65494 objects misplaced (4.089%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T16:40:27.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:27.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:26 smithi121 bash[18489]: cluster 2024-02-12T16:40:25.382091+0000 osd.9 (osd.9) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:26 smithi121 bash[18489]: cluster 2024-02-12T16:40:25.382141+0000 osd.9 (osd.9) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:26 smithi121 bash[18489]: cluster 2024-02-12T16:40:25.902449+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e872: 12 total, 11 up, 11 in 2024-02-12T16:40:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:26 smithi121 bash[18489]: cluster 2024-02-12T16:40:25.903843+0000 mon.a (mon.0) 2981 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:40:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:26 smithi121 bash[18489]: cluster 2024-02-12T16:40:26.216389+0000 mgr.y (mgr.14146) 1682 : cluster [DBG] pgmap v2426: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 176 MiB/s wr, 2.82k op/s; 2594/65494 objects degraded (3.961%); 2678/65494 objects misplaced (4.089%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T16:40:27.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:26 smithi123 bash[19119]: cluster 2024-02-12T16:40:25.382091+0000 osd.9 (osd.9) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:27.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:26 smithi123 bash[19119]: cluster 2024-02-12T16:40:25.382141+0000 osd.9 (osd.9) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:27.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:26 smithi123 bash[19119]: cluster 2024-02-12T16:40:25.902449+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e872: 12 total, 11 up, 11 in 2024-02-12T16:40:27.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:26 smithi123 bash[19119]: cluster 2024-02-12T16:40:25.903843+0000 mon.a (mon.0) 2981 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-02-12T16:40:27.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:26 smithi123 bash[19119]: cluster 2024-02-12T16:40:26.216389+0000 mgr.y (mgr.14146) 1682 : cluster [DBG] pgmap v2426: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 176 MiB/s wr, 2.82k op/s; 2594/65494 objects degraded (3.961%); 2678/65494 objects misplaced (4.089%); 7.3 MiB/s, 116 objects/s recovering 2024-02-12T16:40:27.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:27.247 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:27.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:27.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 46233 46217 39.5637 159.75 0.01165 0.0252736 2024-02-12T16:40:27.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:27.377 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:27.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:27.427 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:179502 6.17 6:ee3707bd:::benchmark_data_smithi169_96224_object26666:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e680)", 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.754833+0000", 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 538.66789744499999, 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.068464603, 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.428 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 179502 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754833+0000", 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754833+0000", 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754831+0000", 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9300000000000003e-06 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754839+0000", 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5509e-05 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754854+0000", 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4959999999999999e-06 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.754862+0000", 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.21626487999999999 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.971127+0000", 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2310000000000002e-06 2024-02-12T16:40:27.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.971136+0000", 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.85209320600000005 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.823229+0000", 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8484000000000004e-05 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:29.823297+0000", 2024-02-12T16:40:27.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.068464603 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:204247 7.0 7:05bd7ef5:::benchmark_data_smithi169_105252_object35653:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:36:27.109343+0000", 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 240.31338709799999, 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039870944999999998, 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:27.433 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 204247 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.109343+0000", 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.109343+0000", 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.109342+0000", 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8660000000000002e-06 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.109348+0000", 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4710000000000007e-06 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.109356+0000", 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0669999999999999e-06 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.109361+0000", 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013065558 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.122427+0000", 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0694999999999999e-05 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.122507+0000", 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101374 2024-02-12T16:40:27.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.122609+0000", 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080119869999999999 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.130621+0000", 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018488343000000001 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.149109+0000", 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6116999999999998e-05 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.149165+0000", 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8983999999999997e-05 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:36:27.149214+0000", 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039870944999999998 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.438 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:239288 7.0 7:0ce56820:::benchmark_data_smithi169_105252_object70694:head [delete] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:00.319875+0000", 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 207.102855772, 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.038802300999999997, 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 239288 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.319875+0000", 2024-02-12T16:40:27.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.319875+0000", 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.319873+0000", 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3230000000000003e-06 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.319879+0000", 2024-02-12T16:40:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3159000000000001e-05 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.319892+0000", 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0639999999999999e-06 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.319899+0000", 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010708188 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.330607+0000", 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014544300000000001 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.330752+0000", 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000163053 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.330915+0000", 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071947529999999999 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.338110+0000", 2024-02-12T16:40:27.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020468806999999999 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.358579+0000", 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5772000000000001e-05 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.358635+0000", 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259000000000003e-05 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:00.358677+0000", 2024-02-12T16:40:27.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.038802300999999997 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:281860 7.0 7:06f3fbe7:::benchmark_data_smithi169_105252_object113266:head [delete] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:37.827027+0000", 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 169.59570315600001, 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044420260000000003, 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:27.444 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 281860 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.827027+0000", 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.827027+0000", 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:40:27.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.827024+0000", 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7070000000000004e-06 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.827034+0000", 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3592e-05 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.827047+0000", 2024-02-12T16:40:27.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4829999999999999e-06 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.827056+0000", 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0051380749999999998 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.832194+0000", 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016692000000000001 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.832361+0000", 2024-02-12T16:40:27.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013816699999999999 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.832499+0000", 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013775757 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.846275+0000", 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025071369999999999 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.871346+0000", 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4821999999999998e-05 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.871401+0000", 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6517999999999997e-05 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.871447+0000", 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044420260000000003 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:281863 7.0 7:07ef509f:::benchmark_data_smithi169_105252_object113269:head [delete] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:37.829437+0000", 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 169.59329300799999, 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.042118787999999997, 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 281863 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.829437+0000", 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.829437+0000", 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.829435+0000", 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5120000000000002e-06 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.829443+0000", 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4252e-05 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.829457+0000", 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5629999999999997e-06 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.829463+0000", 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0073811629999999996 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.836845+0000", 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014165400000000001 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.836986+0000", 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015206 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.837138+0000", 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012653765000000001 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.849792+0000", 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021702382999999999 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.871494+0000", 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5657000000000001e-05 2024-02-12T16:40:27.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.871530+0000", 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6030999999999999e-05 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:37.871556+0000", 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.042118787999999997 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:283343 7.0 7:0726b2fa:::benchmark_data_smithi169_105252_object114749:head [delete] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:39.114683+0000", 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 168.30804725499999, 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039650756000000002, 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 283343 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.114683+0000", 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.114683+0000", 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.114681+0000", 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9740000000000002e-06 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.114689+0000", 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5877999999999999e-05 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.114705+0000", 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7329999999999997e-06 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.114711+0000", 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061720680000000002 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.120883+0000", 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001225 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.121006+0000", 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1235000000000004e-05 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.121087+0000", 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037139320000000001 2024-02-12T16:40:27.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.124801+0000", 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029455436000000002 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.154257+0000", 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6489999999999998e-05 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.154293+0000", 2024-02-12T16:40:27.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0732999999999998e-05 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.154334+0000", 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039650756000000002 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:289451 7.0 7:0bf7f0fe:::benchmark_data_smithi169_105252_object120857:head [delete] snapc 0=[] ondisk+write+known_if_redirected e818)", 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:44.302845+0000", 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 163.11988504199999, 2024-02-12T16:40:27.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.040572550999999998, 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 289451 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.302845+0000", 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.302845+0000", 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.302843+0000", 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2439999999999996e-06 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.302850+0000", 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5790999999999998e-05 2024-02-12T16:40:27.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.302866+0000", 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7460000000000003e-06 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.302873+0000", 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029073089999999998 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.305780+0000", 2024-02-12T16:40:27.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1647e-05 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.305852+0000", 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7770999999999998e-05 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.305930+0000", 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014248472999999999 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.320178+0000", 2024-02-12T16:40:27.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023170168000000001 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.343348+0000", 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0976000000000003e-05 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.343379+0000", 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8615999999999998e-05 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:44.343418+0000", 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.040572550999999998 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:32476 8.8 8:14dd2810:::benchmark_data_smithi169_115677_object32475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e871)", 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.513675+0000", 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9090551829999995, 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037572442999999997, 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32476 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513675+0000", 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513675+0000", 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:27.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513673+0000", 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103188 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513777+0000", 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.208e-06 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513785+0000", 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0409999999999999e-06 2024-02-12T16:40:27.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513789+0000", 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9473999999999999e-05 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513818+0000", 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012506 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513943+0000", 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7151999999999995e-05 2024-02-12T16:40:27.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.514031+0000", 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078888199999999997 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.514819+0000", 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.03631272 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551132+0000", 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8483000000000001e-05 2024-02-12T16:40:27.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551211+0000", 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6956000000000003e-05 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551248+0000", 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037572442999999997 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26785.0:32469 8.9 e871/839 8:91dbd50e:::benchmark_data_smithi169_115677_object32468:head v 871'2017, mlcod=871'2012)", 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.513744+0000", 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9089863449999998, 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037393401999999999, 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513744+0000", 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513744+0000", 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513743+0000", 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012752999999999999 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513870+0000", 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2559999999999999e-06 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513876+0000", 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.755e-06 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513880+0000", 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085669299999999995 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.514737+0000", 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3903000000000001e-05 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.514751+0000", 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036340263999999997 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551091+0000", 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6249999999999999e-05 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551137+0000", 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037393401999999999 2024-02-12T16:40:27.476 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:32477 8.4 8:28d70d5b:::benchmark_data_smithi169_115677_object32476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e871)", 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.513858+0000", 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9088723400000003, 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037520805999999997, 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.477 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32477 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513858+0000", 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513858+0000", 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513857+0000", 2024-02-12T16:40:27.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0717000000000002e-05 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513907+0000", 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5580000000000003e-06 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513915+0000", 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6019999999999999e-06 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513921+0000", 2024-02-12T16:40:27.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0098000000000001e-05 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.513961+0000", 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3007000000000001e-05 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.514034+0000", 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012309899999999999 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.514157+0000", 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099962800000000002 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.515156+0000", 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036133475999999998 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551290+0000", 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.939e-05 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551339+0000", 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9529e-05 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551379+0000", 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037520805999999997 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26785.0:32482 8.0 e871/839 8:0a53aab3:::benchmark_data_smithi169_115677_object32481:head v 871'1993, mlcod=871'1991)", 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.516580+0000", 2024-02-12T16:40:27.482 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9061498170000002, 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.040751272999999998, 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.516580+0000", 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.516580+0000", 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:40:27.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.516580+0000", 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6304000000000002e-05 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.516636+0000", 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034457044999999999 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551093+0000", 2024-02-12T16:40:27.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2296000000000001e-05 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551115+0000", 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8483999999999998e-05 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551174+0000", 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3797000000000002e-05 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.551207+0000", 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060994869999999998 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.557307+0000", 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4794e-05 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.557332+0000", 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.040751272999999998 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.486 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:32528 8.4 8:2621676e:::benchmark_data_smithi169_115677_object32527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e871)", 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.559777+0000", 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.8629536599999996, 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.081942963999999993, 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32528 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559777+0000", 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559777+0000", 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559775+0000", 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0521999999999998e-05 2024-02-12T16:40:27.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559865+0000", 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6139999999999992e-06 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559874+0000", 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.403e-06 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559878+0000", 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.286e-05 2024-02-12T16:40:27.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559911+0000", 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6049000000000003e-05 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.559987+0000", 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5797000000000005e-05 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.560073+0000", 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010174680000000001 2024-02-12T16:40:27.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.561090+0000", 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.080530598999999994 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641621+0000", 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4889000000000003e-05 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641676+0000", 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3705999999999998e-05 2024-02-12T16:40:27.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641720+0000", 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.081942963999999993 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:32540 8.4 8:2083a6a9:::benchmark_data_smithi169_115677_object32539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e871)", 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.562426+0000", 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.8603040239999995, 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.082988233999999994, 2024-02-12T16:40:27.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32540 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562426+0000", 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562426+0000", 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562425+0000", 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012515900000000001 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562550+0000", 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6450000000000002e-06 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562556+0000", 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3349999999999998e-06 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562559+0000", 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9625000000000001e-05 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562599+0000", 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9208000000000004e-05 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562688+0000", 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1800999999999997e-05 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562780+0000", 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.078776709 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641557+0000", 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036926419999999999 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.645249+0000", 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111821 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.645361+0000", 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3473000000000001e-05 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.645415+0000", 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.082988233999999994 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.497 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26785.0:32528 8.4 e871/839 ondisk, result = 0)", 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.562584+0000", 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.8601463530000002, 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.079145793000000006, 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562584+0000", 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562584+0000", 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562583+0000", 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7799999999999998e-06 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562587+0000", 2024-02-12T16:40:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5759999999999996e-06 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562593+0000", 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9530000000000001e-06 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.562597+0000", 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.079008445999999996 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641605+0000", 2024-02-12T16:40:27.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9419999999999998e-06 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641612+0000", 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011781 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641730+0000", 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.079145793000000006 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26785.0:32536 8.9 e871/839 8:9981ac54:::benchmark_data_smithi169_115677_object32535:head v 871'2027, mlcod=871'2023)", 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.563580+0000", 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.8591506119999996, 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.082731418000000001, 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.563580+0000", 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.563580+0000", 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.563579+0000", 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8279000000000002e-05 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.563637+0000", 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.077853965999999997 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641491+0000", 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6322000000000001e-05 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641507+0000", 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0735000000000001e-05 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641558+0000", 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1103000000000001e-05 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641579+0000", 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047054330000000002 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.646284+0000", 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6611999999999999e-05 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.646311+0000", 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.082731418000000001 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.505 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:32547 8.4 8:21ed5775:::benchmark_data_smithi169_115677_object32546:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e871)", 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:22.564572+0000", 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.8581578309999998, 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.082771869999999997, 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32547 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.564572+0000", 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.564572+0000", 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.564571+0000", 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1736999999999995e-05 2024-02-12T16:40:27.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.564633+0000", 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.076887068000000003 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641520+0000", 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8919999999999998e-05 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641539+0000", 2024-02-12T16:40:27.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000218011 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.641757+0000", 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032342699999999999 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.642080+0000", 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030937599999999999 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.642390+0000", 2024-02-12T16:40:27.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030661569999999999 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.645456+0000", 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017513749999999999 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.647207+0000", 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4425999999999994e-05 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.647292+0000", 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2651000000000001e-05 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:22.647344+0000", 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.082771869999999997 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:40194 8.4 8:2f4e0f88:::benchmark_data_smithi169_115677_object40193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e871)", 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:25.055710+0000", 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 2.3670207560000001, 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044384676999999997, 2024-02-12T16:40:27.511 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 40194 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055710+0000", 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055710+0000", 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055708+0000", 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3924000000000002e-05 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055752+0000", 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0746e-05 2024-02-12T16:40:27.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055763+0000", 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4040000000000003e-06 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055768+0000", 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0167000000000001e-05 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055798+0000", 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4439999999999999e-05 2024-02-12T16:40:27.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055873+0000", 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3417000000000004e-05 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.055966+0000", 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095131800000000004 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.056917+0000", 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.043035047 2024-02-12T16:40:27.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.099953+0000", 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6903000000000006e-05 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.100029+0000", 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4776000000000005e-05 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.100094+0000", 2024-02-12T16:40:27.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044384676999999997 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:40212 8.4 8:226fff83:::benchmark_data_smithi169_115677_object40211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e871)", 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:25.063627+0000", 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 2.3591033530000001, 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.040101309000000002, 2024-02-12T16:40:27.517 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 40212 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.063627+0000", 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.063627+0000", 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.063626+0000", 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0764999999999998e-05 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.063687+0000", 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.716e-06 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.063692+0000", 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.553e-06 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.063698+0000", 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037003699999999998 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.064068+0000", 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7689999999999996e-05 2024-02-12T16:40:27.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.064156+0000", 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.033813430999999998 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.097969+0000", 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005482705 2024-02-12T16:40:27.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.103452+0000", 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8985000000000004e-05 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.103521+0000", 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9351000000000006e-05 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.103610+0000", 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118069 2024-02-12T16:40:27.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.103728+0000", 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.040101309000000002 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26785.0:40215 8.0 e871/839 8:020e9d42:::benchmark_data_smithi169_115677_object40214:head v 871'2451, mlcod=871'2449)", 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:25.065857+0000", 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 2.3568728889999999, 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037412359999999999, 2024-02-12T16:40:27.524 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.065857+0000", 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.065857+0000", 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.065856+0000", 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8357999999999999e-05 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.065915+0000", 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031983729000000002 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.097898+0000", 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1500000000000005e-06 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.097908+0000", 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8343000000000001e-05 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.097956+0000", 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6089999999999999e-05 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.097982+0000", 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052640559999999996 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.103246+0000", 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3697e-05 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.103270+0000", 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.037412359999999999 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:42616 8.4 8:2017556e:::benchmark_data_smithi169_115677_object42615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e872)", 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:25.927902+0000", 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.49482867, 2024-02-12T16:40:27.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039777109999999997, 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 42616 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.927902+0000", 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:27.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.927902+0000", 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.927900+0000", 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4857999999999994e-05 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.927975+0000", 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9750000000000002e-06 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.927985+0000", 2024-02-12T16:40:27.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9909999999999999e-06 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.927991+0000", 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7493999999999997e-05 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.928028+0000", 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9483999999999999e-05 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:27.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.928118+0000", 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010084700000000001 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 9", 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.928219+0000", 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034652097 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.962871+0000", 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047088319999999996 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.967580+0000", 2024-02-12T16:40:27.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4172999999999998e-05 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.967634+0000", 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5037999999999998e-05 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:25.967679+0000", 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039777109999999997 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:27.533 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:40:27.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:27.567 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:27.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:27.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:27.693 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:27.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:27.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:27.821 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:27.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:27.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:27.947 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:27.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:28.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:28.071 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:28.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:28.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:28.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:28.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 48709 48693 41.12 154.75 0.00590579 0.0243171 2024-02-12T16:40:28.331 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:28.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:40:28.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:28.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:28.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:28.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:28.611 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:28.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:28.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:28.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:28.846 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:28.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:28.983 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:28.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:29.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:29.114 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:29.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:29.217 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:29.241 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:29.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:29.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 16 51328 51312 42.754 163.688 0.00572619 0.0233881 2024-02-12T16:40:29.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:29.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:29.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:29.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:29 smithi169 bash[19604]: cluster 2024-02-12T16:40:28.216983+0000 mgr.y (mgr.14146) 1683 : cluster [DBG] pgmap v2427: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 206 MiB/s wr, 3.29k op/s; 2594/82554 objects degraded (3.142%); 2678/82554 objects misplaced (3.244%); 6.9 MiB/s, 109 objects/s recovering 2024-02-12T16:40:29.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:29.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:29.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:29.622 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:29.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:29 smithi123 bash[19119]: cluster 2024-02-12T16:40:28.216983+0000 mgr.y (mgr.14146) 1683 : cluster [DBG] pgmap v2427: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 206 MiB/s wr, 3.29k op/s; 2594/82554 objects degraded (3.142%); 2678/82554 objects misplaced (3.244%); 6.9 MiB/s, 109 objects/s recovering 2024-02-12T16:40:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:29.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:29 smithi121 bash[18489]: cluster 2024-02-12T16:40:28.216983+0000 mgr.y (mgr.14146) 1683 : cluster [DBG] pgmap v2427: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 206 MiB/s wr, 3.29k op/s; 2594/82554 objects degraded (3.142%); 2678/82554 objects misplaced (3.244%); 6.9 MiB/s, 109 objects/s recovering 2024-02-12T16:40:29.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:29.773 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:29.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:29.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:29.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:30.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:30.038 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:30.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:30.167 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:30.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:30.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:30.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 53811 53795 44.2331 155.188 0.00333966 0.0226059 2024-02-12T16:40:30.415 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:30.436 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:30.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:30.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:30 smithi169 bash[19604]: cluster 2024-02-12T16:40:29.088578+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:30.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:30 smithi169 bash[19604]: cluster 2024-02-12T16:40:29.088637+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:30.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:30 smithi169 bash[19604]: cluster 2024-02-12T16:40:29.273936+0000 mon.a (mon.0) 2982 : cluster [DBG] osdmap e873: 12 total, 11 up, 11 in 2024-02-12T16:40:30.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:30 smithi169 bash[19604]: cluster 2024-02-12T16:40:29.487058+0000 osd.6 (osd.6) 174 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:30.552 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:30 smithi169 bash[19604]: cluster 2024-02-12T16:40:29.487103+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:30.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:30.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:30 smithi121 bash[18489]: cluster 2024-02-12T16:40:29.088578+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:30.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:30 smithi121 bash[18489]: cluster 2024-02-12T16:40:29.088637+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:30.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:30 smithi121 bash[18489]: cluster 2024-02-12T16:40:29.273936+0000 mon.a (mon.0) 2982 : cluster [DBG] osdmap e873: 12 total, 11 up, 11 in 2024-02-12T16:40:30.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:30 smithi121 bash[18489]: cluster 2024-02-12T16:40:29.487058+0000 osd.6 (osd.6) 174 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:30.591 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:30 smithi121 bash[18489]: cluster 2024-02-12T16:40:29.487103+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:30 smithi123 bash[19119]: cluster 2024-02-12T16:40:29.088578+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:30 smithi123 bash[19119]: cluster 2024-02-12T16:40:29.088637+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:30 smithi123 bash[19119]: cluster 2024-02-12T16:40:29.273936+0000 mon.a (mon.0) 2982 : cluster [DBG] osdmap e873: 12 total, 11 up, 11 in 2024-02-12T16:40:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:30 smithi123 bash[19119]: cluster 2024-02-12T16:40:29.487058+0000 osd.6 (osd.6) 174 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:30 smithi123 bash[19119]: cluster 2024-02-12T16:40:29.487103+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:30.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:30.706 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:30.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:30.834 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:30.937 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:30.958 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:30.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:31.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:31.087 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:31.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:31.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:31.215 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:31.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:31.291 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:31 smithi123 bash[19119]: cluster 2024-02-12T16:40:30.217549+0000 mgr.y (mgr.14146) 1684 : cluster [DBG] pgmap v2429: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 169 MiB/s wr, 2.71k op/s; 3416/94332 objects degraded (3.621%); 3445/94332 objects misplaced (3.652%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:40:31.291 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:31 smithi123 bash[19119]: cluster 2024-02-12T16:40:30.276833+0000 mon.a (mon.0) 2983 : cluster [WRN] Health check update: Degraded data redundancy: 3416/94332 objects degraded (3.621%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:31.291 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:31 smithi123 bash[19119]: cluster 2024-02-12T16:40:30.286080+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e874: 12 total, 11 up, 11 in 2024-02-12T16:40:31.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 16 56161 56145 45.5659 146.875 0.00394874 0.0219437 2024-02-12T16:40:31.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:31.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:31.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:31.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:31.470 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:31 smithi121 bash[18489]: cluster 2024-02-12T16:40:30.217549+0000 mgr.y (mgr.14146) 1684 : cluster [DBG] pgmap v2429: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 169 MiB/s wr, 2.71k op/s; 3416/94332 objects degraded (3.621%); 3445/94332 objects misplaced (3.652%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:40:31.470 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:31 smithi121 bash[18489]: cluster 2024-02-12T16:40:30.276833+0000 mon.a (mon.0) 2983 : cluster [WRN] Health check update: Degraded data redundancy: 3416/94332 objects degraded (3.621%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:31.470 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:31 smithi121 bash[18489]: cluster 2024-02-12T16:40:30.286080+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e874: 12 total, 11 up, 11 in 2024-02-12T16:40:31.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:31.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:31 smithi169 bash[19604]: cluster 2024-02-12T16:40:30.217549+0000 mgr.y (mgr.14146) 1684 : cluster [DBG] pgmap v2429: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 169 MiB/s wr, 2.71k op/s; 3416/94332 objects degraded (3.621%); 3445/94332 objects misplaced (3.652%); 928 KiB/s, 14 objects/s recovering 2024-02-12T16:40:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:31 smithi169 bash[19604]: cluster 2024-02-12T16:40:30.276833+0000 mon.a (mon.0) 2983 : cluster [WRN] Health check update: Degraded data redundancy: 3416/94332 objects degraded (3.621%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:31 smithi169 bash[19604]: cluster 2024-02-12T16:40:30.286080+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e874: 12 total, 11 up, 11 in 2024-02-12T16:40:31.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:31.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:31.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:31.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:31.743 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:31.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:31.848 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:31.870 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:31.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:31.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:32.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:32.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:32.133 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:32.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:32.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:32.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:32.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 58592 58576 46.9294 151.938 0.00522724 0.0213064 2024-02-12T16:40:32.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:32.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:32.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:32.531 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:32.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:32.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:32.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:32.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:32.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:32.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:32.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:33.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:33.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:33.095 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:40:33.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:33.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:33.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:33.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:33.314 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 60209 60193 47.6145 101.062 0.00816021 0.0209997 2024-02-12T16:40:33.333 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:33.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:33 smithi123 bash[19119]: audit 2024-02-12T16:40:32.552872+0000 mon.a (mon.0) 2985 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:40:33.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:33 smithi123 bash[19119]: audit 2024-02-12T16:40:32.553089+0000 mon.a (mon.0) 2986 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:40:33.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:33 smithi123 bash[19119]: audit 2024-02-12T16:40:32.553280+0000 mon.a (mon.0) 2987 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:40:33.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:33 smithi123 bash[19119]: audit 2024-02-12T16:40:32.588142+0000 mon.a (mon.0) 2988 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:40:33.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:33 smithi123 bash[19119]: audit 2024-02-12T16:40:32.591157+0000 mon.a (mon.0) 2989 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:40:33.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:33 smithi123 bash[19119]: audit 2024-02-12T16:40:32.592176+0000 mon.a (mon.0) 2990 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:40:33.393 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:33 smithi123 bash[19119]: audit 2024-02-12T16:40:32.597889+0000 mon.a (mon.0) 2991 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:40:33.435 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:33.457 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:33 smithi169 bash[19604]: audit 2024-02-12T16:40:32.552872+0000 mon.a (mon.0) 2985 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:40:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:33 smithi169 bash[19604]: audit 2024-02-12T16:40:32.553089+0000 mon.a (mon.0) 2986 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:40:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:33 smithi169 bash[19604]: audit 2024-02-12T16:40:32.553280+0000 mon.a (mon.0) 2987 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:40:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:33 smithi169 bash[19604]: audit 2024-02-12T16:40:32.588142+0000 mon.a (mon.0) 2988 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:40:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:33 smithi169 bash[19604]: audit 2024-02-12T16:40:32.591157+0000 mon.a (mon.0) 2989 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:40:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:33 smithi169 bash[19604]: audit 2024-02-12T16:40:32.592176+0000 mon.a (mon.0) 2990 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:40:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:33 smithi169 bash[19604]: audit 2024-02-12T16:40:32.597889+0000 mon.a (mon.0) 2991 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:40:33.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:33.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:33 smithi121 bash[18489]: audit 2024-02-12T16:40:32.552872+0000 mon.a (mon.0) 2985 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:40:33.690 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:33 smithi121 bash[18489]: audit 2024-02-12T16:40:32.553089+0000 mon.a (mon.0) 2986 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:40:33.690 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:33 smithi121 bash[18489]: audit 2024-02-12T16:40:32.553280+0000 mon.a (mon.0) 2987 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:40:33.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:33 smithi121 bash[18489]: audit 2024-02-12T16:40:32.588142+0000 mon.a (mon.0) 2988 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:40:33.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:33 smithi121 bash[18489]: audit 2024-02-12T16:40:32.591157+0000 mon.a (mon.0) 2989 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:40:33.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:33 smithi121 bash[18489]: audit 2024-02-12T16:40:32.592176+0000 mon.a (mon.0) 2990 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:40:33.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:33 smithi121 bash[18489]: audit 2024-02-12T16:40:32.597889+0000 mon.a (mon.0) 2991 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:40:33.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:33.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:33.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:33.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:33.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:33.987 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:33.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:34.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:34.113 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:34.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:34.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:34.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:34.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:34.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:40:34.312051 min lat: 0.00106883 max lat: 60.0841 avg lat: 0.0204988 2024-02-12T16:40:34.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:40:34.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 62462 62446 48.7792 140.812 0.00661627 0.0204988 2024-02-12T16:40:34.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:34.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:34.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:34.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:34.511 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:34.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:34.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:34.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:34.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:34.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:34.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:34.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:34.901 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:34.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:35.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:35.037 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:35.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:35.167 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:35.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:35.296 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:35 smithi169 bash[19604]: cluster 2024-02-12T16:40:34.218961+0000 mgr.y (mgr.14146) 1686 : cluster [DBG] pgmap v2432: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 200 MiB/s wr, 3.20k op/s; 3416/116648 objects degraded (2.928%); 4135/116648 objects misplaced (3.545%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:40:35.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:35.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 64814 64798 49.9916 147 0.0030239 0.020001 2024-02-12T16:40:35.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:35.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:35.539 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:35.570 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:35 smithi123 bash[19119]: cluster 2024-02-12T16:40:34.218961+0000 mgr.y (mgr.14146) 1686 : cluster [DBG] pgmap v2432: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 200 MiB/s wr, 3.20k op/s; 3416/116648 objects degraded (2.928%); 4135/116648 objects misplaced (3.545%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:40:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:35 smithi121 bash[18489]: cluster 2024-02-12T16:40:34.218961+0000 mgr.y (mgr.14146) 1686 : cluster [DBG] pgmap v2432: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 200 MiB/s wr, 3.20k op/s; 3416/116648 objects degraded (2.928%); 4135/116648 objects misplaced (3.545%); 464 KiB/s, 7 objects/s recovering 2024-02-12T16:40:35.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:35.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:35.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:35.843 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:35.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:35.972 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:35.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:36.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:36.112 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:36.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:36.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:36.242 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:36.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:36.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 66988 66972 51.0388 135.875 0.00509656 0.0195908 2024-02-12T16:40:36.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:36.375 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:36.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:36 smithi121 bash[18489]: cluster 2024-02-12T16:40:35.905887+0000 mon.a (mon.0) 2992 : cluster [WRN] Health check update: Degraded data redundancy: 3416/116648 objects degraded (2.928%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:36.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:36.500 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:36.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:36 smithi169 bash[19604]: cluster 2024-02-12T16:40:35.905887+0000 mon.a (mon.0) 2992 : cluster [WRN] Health check update: Degraded data redundancy: 3416/116648 objects degraded (2.928%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:36.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:36.636 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:36 smithi123 bash[19119]: cluster 2024-02-12T16:40:35.905887+0000 mon.a (mon.0) 2992 : cluster [WRN] Health check update: Degraded data redundancy: 3416/116648 objects degraded (2.928%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:36.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:36.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:36.762 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:36.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:36.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:36.903 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:36.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:37.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:37.030 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:37.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:37.157 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:37.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:37.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:37.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:37.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 68688 68672 51.7038 106.25 0.00899928 0.0193388 2024-02-12T16:40:37.333 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:40:37.333 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:40:37.333 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:40:37.333 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:40:37.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:37.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:37 smithi121 bash[18489]: cluster 2024-02-12T16:40:36.219476+0000 mgr.y (mgr.14146) 1687 : cluster [DBG] pgmap v2433: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 138 MiB/s wr, 2.21k op/s; 4056/117928 objects degraded (3.439%); 4079/117928 objects misplaced (3.459%); 912 KiB/s, 14 objects/s recovering 2024-02-12T16:40:37.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:37 smithi121 bash[18489]: cluster 2024-02-12T16:40:36.228693+0000 osd.2 (osd.2) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:37.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:37 smithi121 bash[18489]: cluster 2024-02-12T16:40:36.228734+0000 osd.2 (osd.2) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:37.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:37 smithi121 bash[18489]: cluster 2024-02-12T16:40:36.932625+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e875: 12 total, 11 up, 11 in 2024-02-12T16:40:37.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:37.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:37.542 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:37 smithi169 bash[19604]: cluster 2024-02-12T16:40:36.219476+0000 mgr.y (mgr.14146) 1687 : cluster [DBG] pgmap v2433: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 138 MiB/s wr, 2.21k op/s; 4056/117928 objects degraded (3.439%); 4079/117928 objects misplaced (3.459%); 912 KiB/s, 14 objects/s recovering 2024-02-12T16:40:37.542 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:37 smithi169 bash[19604]: cluster 2024-02-12T16:40:36.228693+0000 osd.2 (osd.2) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:37.542 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:37 smithi169 bash[19604]: cluster 2024-02-12T16:40:36.228734+0000 osd.2 (osd.2) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:37.542 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:37 smithi169 bash[19604]: cluster 2024-02-12T16:40:36.932625+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e875: 12 total, 11 up, 11 in 2024-02-12T16:40:37.562 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:37 smithi123 bash[19119]: cluster 2024-02-12T16:40:36.219476+0000 mgr.y (mgr.14146) 1687 : cluster [DBG] pgmap v2433: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 138 MiB/s wr, 2.21k op/s; 4056/117928 objects degraded (3.439%); 4079/117928 objects misplaced (3.459%); 912 KiB/s, 14 objects/s recovering 2024-02-12T16:40:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:37 smithi123 bash[19119]: cluster 2024-02-12T16:40:36.228693+0000 osd.2 (osd.2) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:37 smithi123 bash[19119]: cluster 2024-02-12T16:40:36.228734+0000 osd.2 (osd.2) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:37 smithi123 bash[19119]: cluster 2024-02-12T16:40:36.932625+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e875: 12 total, 11 up, 11 in 2024-02-12T16:40:37.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:37.699 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:37.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:37.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:37.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:37.963 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:37.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:37.991 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:40:37.999 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:40:38.032 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:40:38.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:38.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:40:38.103 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:40:38.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:38.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:38.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:40:38.134 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:40:38.161 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:40:38.186 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:40:38.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:38.213 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:40:38.241 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:38.242 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:40:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:38.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:38.275 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:40:38.305 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:40:38.305 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:38 smithi123 bash[19119]: cluster 2024-02-12T16:40:37.606328+0000 osd.3 (osd.3) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:38.305 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:38 smithi123 bash[19119]: cluster 2024-02-12T16:40:37.606400+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:38.306 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:38 smithi123 bash[19119]: audit 2024-02-12T16:40:37.994417+0000 mon.a (mon.0) 2994 : audit [DBG] from='client.? v1:172.21.15.121:0/1280882394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:38.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 70655 70639 52.5517 122.938 0.00473118 0.0190262 2024-02-12T16:40:38.333 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:40:38.333 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.333 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.334 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:40:38.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:38.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:38.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:38.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:38.508 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:38.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:38 smithi169 bash[19604]: cluster 2024-02-12T16:40:37.606328+0000 osd.3 (osd.3) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:38 smithi169 bash[19604]: cluster 2024-02-12T16:40:37.606400+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:38 smithi169 bash[19604]: audit 2024-02-12T16:40:37.994417+0000 mon.a (mon.0) 2994 : audit [DBG] from='client.? v1:172.21.15.121:0/1280882394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:38.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:38.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:38.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:38 smithi121 bash[18489]: cluster 2024-02-12T16:40:37.606328+0000 osd.3 (osd.3) 98 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:38 smithi121 bash[18489]: cluster 2024-02-12T16:40:37.606400+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:38.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:38 smithi121 bash[18489]: audit 2024-02-12T16:40:37.994417+0000 mon.a (mon.0) 2994 : audit [DBG] from='client.? v1:172.21.15.121:0/1280882394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:38.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:38.765 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:38.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:38.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:38.893 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:38.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:38.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:39.028 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:39.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:39.163 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:39.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:39.311 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:39.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 72739 72723 53.4656 130.25 0.00799089 0.0187017 2024-02-12T16:40:39.327 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:40:39.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:39.450 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:39 smithi169 bash[19604]: audit 2024-02-12T16:40:37.968434+0000 mgr.y (mgr.14146) 1688 : audit [DBG] from='client.26921 v1:172.21.15.121:0/45142241' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:40:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:39 smithi169 bash[19604]: cluster 2024-02-12T16:40:38.220046+0000 mgr.y (mgr.14146) 1689 : cluster [DBG] pgmap v2435: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 140 MiB/s wr, 2.24k op/s; 4056/130138 objects degraded (3.117%); 4079/130138 objects misplaced (3.134%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:40:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:39 smithi169 bash[19604]: cluster 2024-02-12T16:40:38.302217+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e876: 12 total, 11 up, 11 in 2024-02-12T16:40:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:39 smithi169 bash[19604]: audit 2024-02-12T16:40:38.551142+0000 mon.a (mon.0) 2996 : audit [INF] from='client.? v1:172.21.15.121:0/2139733705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:39.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:39.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:39 smithi123 bash[19119]: audit 2024-02-12T16:40:37.968434+0000 mgr.y (mgr.14146) 1688 : audit [DBG] from='client.26921 v1:172.21.15.121:0/45142241' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:40:39.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:39 smithi123 bash[19119]: cluster 2024-02-12T16:40:38.220046+0000 mgr.y (mgr.14146) 1689 : cluster [DBG] pgmap v2435: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 140 MiB/s wr, 2.24k op/s; 4056/130138 objects degraded (3.117%); 4079/130138 objects misplaced (3.134%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:40:39.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:39 smithi123 bash[19119]: cluster 2024-02-12T16:40:38.302217+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e876: 12 total, 11 up, 11 in 2024-02-12T16:40:39.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:39 smithi123 bash[19119]: audit 2024-02-12T16:40:38.551142+0000 mon.a (mon.0) 2996 : audit [INF] from='client.? v1:172.21.15.121:0/2139733705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:39.581 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:39 smithi121 bash[18489]: audit 2024-02-12T16:40:37.968434+0000 mgr.y (mgr.14146) 1688 : audit [DBG] from='client.26921 v1:172.21.15.121:0/45142241' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:40:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:39 smithi121 bash[18489]: cluster 2024-02-12T16:40:38.220046+0000 mgr.y (mgr.14146) 1689 : cluster [DBG] pgmap v2435: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 140 MiB/s wr, 2.24k op/s; 4056/130138 objects degraded (3.117%); 4079/130138 objects misplaced (3.134%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:40:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:39 smithi121 bash[18489]: cluster 2024-02-12T16:40:38.302217+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e876: 12 total, 11 up, 11 in 2024-02-12T16:40:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:39 smithi121 bash[18489]: audit 2024-02-12T16:40:38.551142+0000 mon.a (mon.0) 2996 : audit [INF] from='client.? v1:172.21.15.121:0/2139733705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:40:39.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:39.717 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:39.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:39.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:39.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:39.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:39.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:40.086 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:40.114 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:40.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:40.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:40.241 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:40.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:40.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:40 smithi121 bash[18489]: cluster 2024-02-12T16:40:38.769203+0000 osd.5 (osd.5) 193 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:40.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:40 smithi121 bash[18489]: cluster 2024-02-12T16:40:38.769252+0000 osd.5 (osd.5) 194 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:40.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:40 smithi121 bash[18489]: cluster 2024-02-12T16:40:39.299845+0000 mon.a (mon.0) 2997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:40.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:40 smithi121 bash[18489]: audit 2024-02-12T16:40:39.312053+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.121:0/2139733705' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:40:40.309 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:40 smithi121 bash[18489]: cluster 2024-02-12T16:40:39.312095+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e877: 12 total, 11 up, 11 in 2024-02-12T16:40:40.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 74805 74789 54.3452 129.125 0.0102533 0.018399 2024-02-12T16:40:40.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:40.349 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:40 smithi123 bash[19119]: cluster 2024-02-12T16:40:38.769203+0000 osd.5 (osd.5) 193 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:40.350 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:40 smithi123 bash[19119]: cluster 2024-02-12T16:40:38.769252+0000 osd.5 (osd.5) 194 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:40.350 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:40 smithi123 bash[19119]: cluster 2024-02-12T16:40:39.299845+0000 mon.a (mon.0) 2997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:40.350 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:40 smithi123 bash[19119]: audit 2024-02-12T16:40:39.312053+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.121:0/2139733705' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:40:40.350 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:40 smithi123 bash[19119]: cluster 2024-02-12T16:40:39.312095+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e877: 12 total, 11 up, 11 in 2024-02-12T16:40:40.370 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:40.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:40.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:40:40.503 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:40:40.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:40.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:40.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:40.625 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2434,"stamp":"2024-02-12T16:40:36.929084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4264361984,"num_objects":65069,"num_object_clones":0,"num_object_copies":130138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":4079,"num_objects_unfound":0,"num_objects_dirty":65069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65069,"num_write_kb":4164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2055,"num_bytes_recovered":134676480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65069,"ondisk_log_size":65069,"up":83,"acting":88,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":14756604,"kb_used_data":7580196,"kb_used_omap":91,"kb_used_meta":7176228,"kb_avail":1016440068,"statfs":{"total":1055945392128,"available":1040834629632,"internally_reserved":0,"allocated":7762120704,"data_stored":7755533110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":7348457488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,5,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1227161600,"num_objects":18725,"num_object_clones":0,"num_object_copies":37450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1462,"num_objects_misplaced":576,"num_objects_unfound":0,"num_objects_dirty":18725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18725,"num_write_kb":1198400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":7471104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.656835"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":27,"reported_epoch":873,"state":"active+clean+remapped","last_fresh":"2024-02-12T16:40:29.276709+0000","last_change":"2024-02-12T16:40:21.804444+0000","last_active":"2024-02-12T16:40:29.276709+0000","last_peered":"2024-02-12T16:40:29.276709+0000","last_clean":"2024-02-12T16:40:29.276709+0000","last_became_active":"2024-02-12T16:40:16.300715+0000","last_became_peered":"2024-02-12T16:40:16.300715+0000","last_unstale":"2024-02-12T16:40:29.276709+0000","last_undegraded":"2024-02-12T16:40:29.276709+0000","last_fullsized":"2024-02-12T16:40:29.276709+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:56.998123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:56.998123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8,11],"avail_no_missing":["8","11"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":248,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.281144+0000","last_change":"2024-02-12T16:40:18.632972+0000","last_active":"2024-02-12T16:40:29.281144+0000","last_peered":"2024-02-12T16:40:29.281144+0000","last_clean":"2024-02-12T16:40:29.281144+0000","last_became_active":"2024-02-12T16:40:15.300768+0000","last_became_peered":"2024-02-12T16:40:15.300768+0000","last_unstale":"2024-02-12T16:40:29.281144+0000","last_undegraded":"2024-02-12T16:40:29.281144+0000","last_fullsized":"2024-02-12T16:40:29.281144+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:53.994595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:38:53.994595+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":242,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.277212+0000","last_change":"2024-02-12T16:38:54.836099+0000","last_active":"2024-02-12T16:40:29.277212+0000","last_peered":"2024-02-12T16:40:29.277212+0000","last_clean":"2024-02-12T16:40:29.277212+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:40:29.277212+0000","last_undegraded":"2024-02-12T16:40:29.277212+0000","last_fullsized":"2024-02-12T16:40:29.277212+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:54.835978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:38:54.835978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":42,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.282539+0000","last_change":"2024-02-12T16:39:55.622860+0000","last_active":"2024-02-12T16:40:29.282539+0000","last_peered":"2024-02-12T16:40:29.282539+0000","last_clean":"2024-02-12T16:40:29.282539+0000","last_became_active":"2024-02-12T16:39:09.436840+0000","last_became_peered":"2024-02-12T16:39:09.436840+0000","last_unstale":"2024-02-12T16:40:29.282539+0000","last_undegraded":"2024-02-12T16:40:29.282539+0000","last_fullsized":"2024-02-12T16:40:29.282539+0000","mapping_epoch":845,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":846,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:55.622733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:55.622733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":290,"reported_epoch":873,"state":"undersized+peered","last_fresh":"2024-02-12T16:40:29.278937+0000","last_change":"2024-02-12T16:39:17.263790+0000","last_active":"2024-02-12T16:39:16.252427+0000","last_peered":"2024-02-12T16:40:29.278937+0000","last_clean":"2024-02-12T16:39:11.935822+0000","last_became_active":"2024-02-12T16:30:55.394105+0000","last_became_peered":"2024-02-12T16:39:17.263790+0000","last_unstale":"2024-02-12T16:40:29.278937+0000","last_undegraded":"2024-02-12T16:40:29.278937+0000","last_fullsized":"2024-02-12T16:39:17.262718+0000","mapping_epoch":849,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:56.627539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:38:56.627539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":255,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.282573+0000","last_change":"2024-02-12T16:39:18.727200+0000","last_active":"2024-02-12T16:40:29.282573+0000","last_peered":"2024-02-12T16:40:29.282573+0000","last_clean":"2024-02-12T16:40:29.282573+0000","last_became_active":"2024-02-12T16:36:17.847960+0000","last_became_peered":"2024-02-12T16:36:17.847960+0000","last_unstale":"2024-02-12T16:40:29.282573+0000","last_undegraded":"2024-02-12T16:40:29.282573+0000","last_fullsized":"2024-02-12T16:40:29.282573+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:18.727010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:18.727010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":598,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.280850+0000","last_change":"2024-02-12T16:39:07.505234+0000","last_active":"2024-02-12T16:40:29.280850+0000","last_peered":"2024-02-12T16:40:29.280850+0000","last_clean":"2024-02-12T16:40:29.280850+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:40:29.280850+0000","last_undegraded":"2024-02-12T16:40:29.280850+0000","last_fullsized":"2024-02-12T16:40:29.280850+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:07.505037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:39:07.505037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":246,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.276854+0000","last_change":"2024-02-12T16:40:11.951091+0000","last_active":"2024-02-12T16:40:29.276854+0000","last_peered":"2024-02-12T16:40:29.276854+0000","last_clean":"2024-02-12T16:40:29.276854+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:40:29.276854+0000","last_undegraded":"2024-02-12T16:40:29.276854+0000","last_fullsized":"2024-02-12T16:40:29.276854+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:11.951002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:40:11.951002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":245,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.278886+0000","last_change":"2024-02-12T16:40:05.662055+0000","last_active":"2024-02-12T16:40:29.278886+0000","last_peered":"2024-02-12T16:40:29.278886+0000","last_clean":"2024-02-12T16:40:29.278886+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:40:29.278886+0000","last_undegraded":"2024-02-12T16:40:29.278886+0000","last_fullsized":"2024-02-12T16:40:29.278886+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:05.661969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:40:05.661969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":33,"reported_epoch":873,"state":"undersized+peered","last_fresh":"2024-02-12T16:40:29.280309+0000","last_change":"2024-02-12T16:39:17.263710+0000","last_active":"2024-02-12T16:30:48.365000+0000","last_peered":"2024-02-12T16:40:29.280309+0000","last_clean":"2024-02-12T16:30:47.477423+0000","last_became_active":"2024-02-12T16:17:37.453616+0000","last_became_peered":"2024-02-12T16:39:17.263710+0000","last_unstale":"2024-02-12T16:40:29.280309+0000","last_undegraded":"2024-02-12T16:40:29.280309+0000","last_fullsized":"2024-02-12T16:39:17.263322+0000","mapping_epoch":849,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:11.207557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:39:11.207557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":496,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.275086+0000","last_change":"2024-02-12T16:40:00.803581+0000","last_active":"2024-02-12T16:40:29.275086+0000","last_peered":"2024-02-12T16:40:29.275086+0000","last_clean":"2024-02-12T16:40:29.275086+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:40:29.275086+0000","last_undegraded":"2024-02-12T16:40:29.275086+0000","last_fullsized":"2024-02-12T16:40:29.275086+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:00.803488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:40:00.803488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"874'3811","reported_seq":3858,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:33.694588+0000","last_change":"2024-02-12T16:40:00.717962+0000","last_active":"2024-02-12T16:40:33.694588+0000","last_peered":"2024-02-12T16:40:33.694588+0000","last_clean":"2024-02-12T16:40:33.694588+0000","last_became_active":"2024-02-12T16:38:56.203785+0000","last_became_peered":"2024-02-12T16:38:56.203785+0000","last_unstale":"2024-02-12T16:40:33.694588+0000","last_undegraded":"2024-02-12T16:40:33.694588+0000","last_fullsized":"2024-02-12T16:40:33.694588+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'650","last_scrub_stamp":"2024-02-12T16:40:00.717870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:40:00.717870+0000","log_size":3811,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":249757696,"num_objects":3811,"num_object_clones":0,"num_object_copies":7622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":243904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":873,"state":"active+clean+remapped","last_fresh":"2024-02-12T16:40:29.274899+0000","last_change":"2024-02-12T16:40:16.300000+0000","last_active":"2024-02-12T16:40:29.274899+0000","last_peered":"2024-02-12T16:40:29.274899+0000","last_clean":"2024-02-12T16:40:29.274899+0000","last_became_active":"2024-02-12T16:40:16.299588+0000","last_became_peered":"2024-02-12T16:40:16.299588+0000","last_unstale":"2024-02-12T16:40:29.274899+0000","last_undegraded":"2024-02-12T16:40:29.274899+0000","last_fullsized":"2024-02-12T16:40:29.274899+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:12.128724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:12.128724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6,11],"avail_no_missing":["6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"874'4146","reported_seq":4188,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.564148+0000","last_change":"2024-02-12T16:38:56.202943+0000","last_active":"2024-02-12T16:40:36.564148+0000","last_peered":"2024-02-12T16:40:36.564148+0000","last_clean":"2024-02-12T16:40:36.564148+0000","last_became_active":"2024-02-12T16:38:56.202631+0000","last_became_peered":"2024-02-12T16:38:56.202631+0000","last_unstale":"2024-02-12T16:40:36.564148+0000","last_undegraded":"2024-02-12T16:40:36.564148+0000","last_fullsized":"2024-02-12T16:40:36.564148+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":4146,"ondisk_log_size":4146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":271712256,"num_objects":4146,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":265344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":615,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.282737+0000","last_change":"2024-02-12T16:40:20.722915+0000","last_active":"2024-02-12T16:40:29.282737+0000","last_peered":"2024-02-12T16:40:29.282737+0000","last_clean":"2024-02-12T16:40:29.282737+0000","last_became_active":"2024-02-12T16:36:17.846631+0000","last_became_peered":"2024-02-12T16:36:17.846631+0000","last_unstale":"2024-02-12T16:40:29.282737+0000","last_undegraded":"2024-02-12T16:40:29.282737+0000","last_fullsized":"2024-02-12T16:40:29.282737+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:20.722821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:40:20.722821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.d","version":"874'4056","reported_seq":4543,"reported_epoch":874,"state":"active+recovering+degraded+remapped","last_fresh":"2024-02-12T16:40:35.184821+0000","last_change":"2024-02-12T16:40:16.345360+0000","last_active":"2024-02-12T16:40:35.184821+0000","last_peered":"2024-02-12T16:40:35.184821+0000","last_clean":"2024-02-12T16:39:16.241600+0000","last_became_active":"2024-02-12T16:40:16.321705+0000","last_became_peered":"2024-02-12T16:40:16.321705+0000","last_unstale":"2024-02-12T16:40:35.184821+0000","last_undegraded":"2024-02-12T16:40:16.301574+0000","last_fullsized":"2024-02-12T16:40:35.184821+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":4056,"ondisk_log_size":4056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":265814016,"num_objects":4056,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":353,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4056,"num_write_kb":259584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":14221312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[11,2],"avail_no_missing":["11"],"object_location_counts":[{"shards":"2,11","objects":3703},{"shards":"11","objects":353}],"blocked_by":[],"up_primary":2,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":532,"reported_epoch":873,"state":"active+clean+remapped","last_fresh":"2024-02-12T16:40:29.277299+0000","last_change":"2024-02-12T16:40:16.300188+0000","last_active":"2024-02-12T16:40:29.277299+0000","last_peered":"2024-02-12T16:40:29.277299+0000","last_clean":"2024-02-12T16:40:29.277299+0000","last_became_active":"2024-02-12T16:40:16.299879+0000","last_became_peered":"2024-02-12T16:40:16.299879+0000","last_unstale":"2024-02-12T16:40:29.277299+0000","last_undegraded":"2024-02-12T16:40:29.277299+0000","last_fullsized":"2024-02-12T16:40:29.277299+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:15.947507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:39:15.947507+0000","last_clean_scrub_stamp":"2024-02-12T16:39:15.947507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8,11],"avail_no_missing":["8","11"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.e","version":"874'4228","reported_seq":4267,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.649389+0000","last_change":"2024-02-12T16:38:56.230891+0000","last_active":"2024-02-12T16:40:36.649389+0000","last_peered":"2024-02-12T16:40:36.649389+0000","last_clean":"2024-02-12T16:40:36.649389+0000","last_became_active":"2024-02-12T16:38:56.230466+0000","last_became_peered":"2024-02-12T16:38:56.230466+0000","last_unstale":"2024-02-12T16:40:36.649389+0000","last_undegraded":"2024-02-12T16:40:36.649389+0000","last_fullsized":"2024-02-12T16:40:36.649389+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":4228,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":277086208,"num_objects":4228,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":270592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":653,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.275054+0000","last_change":"2024-02-12T16:40:19.721247+0000","last_active":"2024-02-12T16:40:29.275054+0000","last_peered":"2024-02-12T16:40:29.275054+0000","last_clean":"2024-02-12T16:40:29.275054+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:40:29.275054+0000","last_undegraded":"2024-02-12T16:40:29.275054+0000","last_fullsized":"2024-02-12T16:40:29.275054+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:19.721113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:40:19.721113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"874'4308","reported_seq":5622,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.541887+0000","last_change":"2024-02-12T16:40:17.997355+0000","last_active":"2024-02-12T16:40:36.541887+0000","last_peered":"2024-02-12T16:40:36.541887+0000","last_clean":"2024-02-12T16:40:36.541887+0000","last_became_active":"2024-02-12T16:40:15.357892+0000","last_became_peered":"2024-02-12T16:40:15.357892+0000","last_unstale":"2024-02-12T16:40:36.541887+0000","last_undegraded":"2024-02-12T16:40:36.541887+0000","last_fullsized":"2024-02-12T16:40:36.541887+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'641","last_scrub_stamp":"2024-02-12T16:40:09.936005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:40:09.936005+0000","log_size":4308,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":8616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4308,"num_write_kb":275712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":42008576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":594,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.282651+0000","last_change":"2024-02-12T16:39:58.582777+0000","last_active":"2024-02-12T16:40:29.282651+0000","last_peered":"2024-02-12T16:40:29.282651+0000","last_clean":"2024-02-12T16:40:29.282651+0000","last_became_active":"2024-02-12T16:30:49.373979+0000","last_became_peered":"2024-02-12T16:30:49.373979+0000","last_unstale":"2024-02-12T16:40:29.282651+0000","last_undegraded":"2024-02-12T16:40:29.282651+0000","last_fullsized":"2024-02-12T16:40:29.282651+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:58.582561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:57.057484+0000","last_clean_scrub_stamp":"2024-02-12T16:39:58.582561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.8","version":"874'4275","reported_seq":4322,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.727394+0000","last_change":"2024-02-12T16:39:59.647052+0000","last_active":"2024-02-12T16:40:36.727394+0000","last_peered":"2024-02-12T16:40:36.727394+0000","last_clean":"2024-02-12T16:40:36.727394+0000","last_became_active":"2024-02-12T16:38:56.202948+0000","last_became_peered":"2024-02-12T16:38:56.202948+0000","last_unstale":"2024-02-12T16:40:36.727394+0000","last_undegraded":"2024-02-12T16:40:36.727394+0000","last_fullsized":"2024-02-12T16:40:36.727394+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'663","last_scrub_stamp":"2024-02-12T16:39:59.646892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:39:59.646892+0000","log_size":4275,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":280166400,"num_objects":4275,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":273600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":645,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.282557+0000","last_change":"2024-02-12T16:39:56.521580+0000","last_active":"2024-02-12T16:40:29.282557+0000","last_peered":"2024-02-12T16:40:29.282557+0000","last_clean":"2024-02-12T16:40:29.282557+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:40:29.282557+0000","last_undegraded":"2024-02-12T16:40:29.282557+0000","last_fullsized":"2024-02-12T16:40:29.282557+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:56.521425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:39:56.521425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.a","version":"874'3887","reported_seq":3928,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:34.097578+0000","last_change":"2024-02-12T16:38:56.232262+0000","last_active":"2024-02-12T16:40:34.097578+0000","last_peered":"2024-02-12T16:40:34.097578+0000","last_clean":"2024-02-12T16:40:34.097578+0000","last_became_active":"2024-02-12T16:38:56.231966+0000","last_became_peered":"2024-02-12T16:38:56.231966+0000","last_unstale":"2024-02-12T16:40:34.097578+0000","last_undegraded":"2024-02-12T16:40:34.097578+0000","last_fullsized":"2024-02-12T16:40:34.097578+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":3887,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":254738432,"num_objects":3887,"num_object_clones":0,"num_object_copies":7774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3887,"num_write_kb":248768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":454,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.276272+0000","last_change":"2024-02-12T16:40:06.039681+0000","last_active":"2024-02-12T16:40:29.276272+0000","last_peered":"2024-02-12T16:40:29.276272+0000","last_clean":"2024-02-12T16:40:29.276272+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:40:29.276272+0000","last_undegraded":"2024-02-12T16:40:29.276272+0000","last_fullsized":"2024-02-12T16:40:29.276272+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:06.039591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:40:06.039591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"874'3833","reported_seq":3879,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:34.086961+0000","last_change":"2024-02-12T16:40:00.105722+0000","last_active":"2024-02-12T16:40:34.086961+0000","last_peered":"2024-02-12T16:40:34.086961+0000","last_clean":"2024-02-12T16:40:34.086961+0000","last_became_active":"2024-02-12T16:38:56.202404+0000","last_became_peered":"2024-02-12T16:38:56.202404+0000","last_unstale":"2024-02-12T16:40:34.086961+0000","last_undegraded":"2024-02-12T16:40:34.086961+0000","last_fullsized":"2024-02-12T16:40:34.086961+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'666","last_scrub_stamp":"2024-02-12T16:40:00.105621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:40:00.105621+0000","log_size":3833,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":251199488,"num_objects":3833,"num_object_clones":0,"num_object_copies":7666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":245312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":492,"reported_epoch":873,"state":"undersized+peered","last_fresh":"2024-02-12T16:40:29.278718+0000","last_change":"2024-02-12T16:39:17.262778+0000","last_active":"2024-02-12T16:17:36.439563+0000","last_peered":"2024-02-12T16:40:29.278718+0000","last_clean":"2024-02-12T16:17:34.860067+0000","last_became_active":"2024-02-12T16:15:24.799547+0000","last_became_peered":"2024-02-12T16:39:17.262778+0000","last_unstale":"2024-02-12T16:40:29.278718+0000","last_undegraded":"2024-02-12T16:40:29.278718+0000","last_fullsized":"2024-02-12T16:39:17.261901+0000","mapping_epoch":849,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:09.159808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:09.159808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"874'4272","reported_seq":4319,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.737069+0000","last_change":"2024-02-12T16:40:03.638276+0000","last_active":"2024-02-12T16:40:36.737069+0000","last_peered":"2024-02-12T16:40:36.737069+0000","last_clean":"2024-02-12T16:40:36.737069+0000","last_became_active":"2024-02-12T16:38:56.201855+0000","last_became_peered":"2024-02-12T16:38:56.201855+0000","last_unstale":"2024-02-12T16:40:36.737069+0000","last_undegraded":"2024-02-12T16:40:36.737069+0000","last_fullsized":"2024-02-12T16:40:36.737069+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'606","last_scrub_stamp":"2024-02-12T16:40:03.638128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:40:03.638128+0000","log_size":4272,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":279969792,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":273408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":193,"reported_epoch":873,"state":"undersized+peered","last_fresh":"2024-02-12T16:40:29.279049+0000","last_change":"2024-02-12T16:39:17.263777+0000","last_active":"2024-02-12T16:30:48.367228+0000","last_peered":"2024-02-12T16:40:29.279049+0000","last_clean":"2024-02-12T16:30:47.361313+0000","last_became_active":"2024-02-12T16:23:39.022679+0000","last_became_peered":"2024-02-12T16:39:17.263777+0000","last_unstale":"2024-02-12T16:40:29.279049+0000","last_undegraded":"2024-02-12T16:40:29.279049+0000","last_fullsized":"2024-02-12T16:39:17.262764+0000","mapping_epoch":849,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:00.113518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:00.113518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.5","version":"874'3768","reported_seq":3815,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:33.696864+0000","last_change":"2024-02-12T16:40:03.706477+0000","last_active":"2024-02-12T16:40:33.696864+0000","last_peered":"2024-02-12T16:40:33.696864+0000","last_clean":"2024-02-12T16:40:33.696864+0000","last_became_active":"2024-02-12T16:38:56.203819+0000","last_became_peered":"2024-02-12T16:38:56.203819+0000","last_unstale":"2024-02-12T16:40:33.696864+0000","last_undegraded":"2024-02-12T16:40:33.696864+0000","last_fullsized":"2024-02-12T16:40:33.696864+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'646","last_scrub_stamp":"2024-02-12T16:40:03.706273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:40:03.706273+0000","log_size":3768,"ondisk_log_size":3768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":246939648,"num_objects":3768,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3768,"num_write_kb":241152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":429,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.276268+0000","last_change":"2024-02-12T16:39:08.747017+0000","last_active":"2024-02-12T16:40:29.276268+0000","last_peered":"2024-02-12T16:40:29.276268+0000","last_clean":"2024-02-12T16:40:29.276268+0000","last_became_active":"2024-02-12T16:30:49.382268+0000","last_became_peered":"2024-02-12T16:30:49.382268+0000","last_unstale":"2024-02-12T16:40:29.276268+0000","last_undegraded":"2024-02-12T16:40:29.276268+0000","last_fullsized":"2024-02-12T16:40:29.276268+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:08.746907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:08.746907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"874'3726","reported_seq":3759,"reported_epoch":874,"state":"active+clean+remapped","last_fresh":"2024-02-12T16:40:33.696587+0000","last_change":"2024-02-12T16:40:16.303930+0000","last_active":"2024-02-12T16:40:33.696587+0000","last_peered":"2024-02-12T16:40:33.696587+0000","last_clean":"2024-02-12T16:40:33.696587+0000","last_became_active":"2024-02-12T16:40:16.302744+0000","last_became_peered":"2024-02-12T16:40:16.302744+0000","last_unstale":"2024-02-12T16:40:33.696587+0000","last_undegraded":"2024-02-12T16:40:33.696587+0000","last_fullsized":"2024-02-12T16:40:33.696587+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'632","last_scrub_stamp":"2024-02-12T16:40:06.972391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:40:06.972391+0000","log_size":3726,"ondisk_log_size":3726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":244187136,"num_objects":3726,"num_object_clones":0,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3726,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3726,"num_write_kb":238464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10],"acting":[10,11],"avail_no_missing":["10","11"],"object_location_counts":[{"shards":"10,11","objects":3726}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":599,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.275109+0000","last_change":"2024-02-12T16:40:06.662477+0000","last_active":"2024-02-12T16:40:29.275109+0000","last_peered":"2024-02-12T16:40:29.275109+0000","last_clean":"2024-02-12T16:40:29.275109+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:40:29.275109+0000","last_undegraded":"2024-02-12T16:40:29.275109+0000","last_fullsized":"2024-02-12T16:40:29.275109+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:06.662379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:35:28.794774+0000","last_clean_scrub_stamp":"2024-02-12T16:40:06.662379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"874'4150","reported_seq":5416,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.529114+0000","last_change":"2024-02-12T16:40:17.998598+0000","last_active":"2024-02-12T16:40:36.529114+0000","last_peered":"2024-02-12T16:40:36.529114+0000","last_clean":"2024-02-12T16:40:36.529114+0000","last_became_active":"2024-02-12T16:40:15.324661+0000","last_became_peered":"2024-02-12T16:40:15.324661+0000","last_unstale":"2024-02-12T16:40:36.529114+0000","last_undegraded":"2024-02-12T16:40:36.529114+0000","last_fullsized":"2024-02-12T16:40:36.529114+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":4150,"ondisk_log_size":4150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":271974400,"num_objects":4150,"num_object_clones":0,"num_object_copies":8300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4150,"num_write_kb":265600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":39845888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":406,"reported_epoch":873,"state":"undersized+peered","last_fresh":"2024-02-12T16:40:29.279009+0000","last_change":"2024-02-12T16:39:17.263385+0000","last_active":"2024-02-12T16:39:16.252436+0000","last_peered":"2024-02-12T16:40:29.279009+0000","last_clean":"2024-02-12T16:39:11.935953+0000","last_became_active":"2024-02-12T16:30:49.381004+0000","last_became_peered":"2024-02-12T16:39:17.263385+0000","last_unstale":"2024-02-12T16:40:29.279009+0000","last_undegraded":"2024-02-12T16:40:29.279009+0000","last_fullsized":"2024-02-12T16:39:17.262722+0000","mapping_epoch":849,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:00.695513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:00.695513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"874'4165","reported_seq":4207,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.376440+0000","last_change":"2024-02-12T16:38:56.201830+0000","last_active":"2024-02-12T16:40:36.376440+0000","last_peered":"2024-02-12T16:40:36.376440+0000","last_clean":"2024-02-12T16:40:36.376440+0000","last_became_active":"2024-02-12T16:38:56.201608+0000","last_became_peered":"2024-02-12T16:38:56.201608+0000","last_unstale":"2024-02-12T16:40:36.376440+0000","last_undegraded":"2024-02-12T16:40:36.376440+0000","last_fullsized":"2024-02-12T16:40:36.376440+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":4165,"ondisk_log_size":4165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":272957440,"num_objects":4165,"num_object_clones":0,"num_object_copies":8330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4165,"num_write_kb":266560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":794,"reported_epoch":873,"state":"undersized+peered","last_fresh":"2024-02-12T16:40:29.278075+0000","last_change":"2024-02-12T16:39:17.260491+0000","last_active":"2024-02-12T16:30:48.367304+0000","last_peered":"2024-02-12T16:40:29.278075+0000","last_clean":"2024-02-12T16:30:47.360932+0000","last_became_active":"2024-02-12T16:15:24.790824+0000","last_became_peered":"2024-02-12T16:39:17.260491+0000","last_unstale":"2024-02-12T16:40:29.278075+0000","last_undegraded":"2024-02-12T16:40:29.278075+0000","last_fullsized":"2024-02-12T16:39:17.259222+0000","mapping_epoch":849,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:03.155045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:03.155045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.1","version":"874'4253","reported_seq":4298,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.544488+0000","last_change":"2024-02-12T16:40:13.573741+0000","last_active":"2024-02-12T16:40:36.544488+0000","last_peered":"2024-02-12T16:40:36.544488+0000","last_clean":"2024-02-12T16:40:36.544488+0000","last_became_active":"2024-02-12T16:38:56.230408+0000","last_became_peered":"2024-02-12T16:38:56.230408+0000","last_unstale":"2024-02-12T16:40:36.544488+0000","last_undegraded":"2024-02-12T16:40:36.544488+0000","last_fullsized":"2024-02-12T16:40:36.544488+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'630","last_scrub_stamp":"2024-02-12T16:40:13.573540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:40:13.573540+0000","log_size":4253,"ondisk_log_size":4253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":278724608,"num_objects":4253,"num_object_clones":0,"num_object_copies":8506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":272192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":592,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.280874+0000","last_change":"2024-02-12T16:39:15.465278+0000","last_active":"2024-02-12T16:40:29.280874+0000","last_peered":"2024-02-12T16:40:29.280874+0000","last_clean":"2024-02-12T16:40:29.280874+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:40:29.280874+0000","last_undegraded":"2024-02-12T16:40:29.280874+0000","last_fullsized":"2024-02-12T16:40:29.280874+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:15.465213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:39:15.465213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"874'3848","reported_seq":3890,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:33.692187+0000","last_change":"2024-02-12T16:38:56.204161+0000","last_active":"2024-02-12T16:40:33.692187+0000","last_peered":"2024-02-12T16:40:33.692187+0000","last_clean":"2024-02-12T16:40:33.692187+0000","last_became_active":"2024-02-12T16:38:56.203800+0000","last_became_peered":"2024-02-12T16:38:56.203800+0000","last_unstale":"2024-02-12T16:40:33.692187+0000","last_undegraded":"2024-02-12T16:40:33.692187+0000","last_fullsized":"2024-02-12T16:40:33.692187+0000","mapping_epoch":839,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":840,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":3848,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":252182528,"num_objects":3848,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":246272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":125,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.278716+0000","last_change":"2024-02-12T16:40:18.778910+0000","last_active":"2024-02-12T16:40:29.278716+0000","last_peered":"2024-02-12T16:40:29.278716+0000","last_clean":"2024-02-12T16:40:29.278716+0000","last_became_active":"2024-02-12T16:40:15.527187+0000","last_became_peered":"2024-02-12T16:40:15.527187+0000","last_unstale":"2024-02-12T16:40:29.278716+0000","last_undegraded":"2024-02-12T16:40:29.278716+0000","last_fullsized":"2024-02-12T16:40:29.278716+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:18.778805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:40:18.778805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"874'4343","reported_seq":5555,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-02-12T16:40:36.551858+0000","last_change":"2024-02-12T16:40:17.630126+0000","last_active":"2024-02-12T16:40:36.551858+0000","last_peered":"2024-02-12T16:40:36.551858+0000","last_clean":"2024-02-12T16:40:36.551858+0000","last_became_active":"2024-02-12T16:40:15.528229+0000","last_became_peered":"2024-02-12T16:40:15.528229+0000","last_unstale":"2024-02-12T16:40:36.551858+0000","last_undegraded":"2024-02-12T16:40:36.551858+0000","last_fullsized":"2024-02-12T16:40:36.551858+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":839,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:38:55.185285+0000","last_clean_scrub_stamp":"2024-02-12T16:38:55.185285+0000","log_size":4343,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":284622848,"num_objects":4343,"num_object_clones":0,"num_object_copies":8686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":277952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":589,"num_bytes_recovered":38600704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":584,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.278034+0000","last_change":"2024-02-12T16:39:07.696190+0000","last_active":"2024-02-12T16:40:29.278034+0000","last_peered":"2024-02-12T16:40:29.278034+0000","last_clean":"2024-02-12T16:40:29.278034+0000","last_became_active":"2024-02-12T16:30:49.384015+0000","last_became_peered":"2024-02-12T16:30:49.384015+0000","last_unstale":"2024-02-12T16:40:29.278034+0000","last_undegraded":"2024-02-12T16:40:29.278034+0000","last_fullsized":"2024-02-12T16:40:29.278034+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:07.695969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:39:07.695969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":95,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.278926+0000","last_change":"2024-02-12T16:40:18.349314+0000","last_active":"2024-02-12T16:40:29.278926+0000","last_peered":"2024-02-12T16:40:29.278926+0000","last_clean":"2024-02-12T16:40:29.278926+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:40:29.278926+0000","last_undegraded":"2024-02-12T16:40:29.278926+0000","last_fullsized":"2024-02-12T16:40:29.278926+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:18.349211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:40:18.349211+0000","last_clean_scrub_stamp":"2024-02-12T16:40:18.349211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":106,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.276954+0000","last_change":"2024-02-12T16:39:22.102350+0000","last_active":"2024-02-12T16:40:29.276954+0000","last_peered":"2024-02-12T16:40:29.276954+0000","last_clean":"2024-02-12T16:40:29.276954+0000","last_became_active":"2024-02-12T16:36:18.479362+0000","last_became_peered":"2024-02-12T16:36:18.479362+0000","last_unstale":"2024-02-12T16:40:29.276954+0000","last_undegraded":"2024-02-12T16:40:29.276954+0000","last_fullsized":"2024-02-12T16:40:29.276954+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:22.102241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:54.947881+0000","last_clean_scrub_stamp":"2024-02-12T16:39:22.102241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":465,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.280190+0000","last_change":"2024-02-12T16:40:19.529370+0000","last_active":"2024-02-12T16:40:29.280190+0000","last_peered":"2024-02-12T16:40:29.280190+0000","last_clean":"2024-02-12T16:40:29.280190+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:40:29.280190+0000","last_undegraded":"2024-02-12T16:40:29.280190+0000","last_fullsized":"2024-02-12T16:40:29.280190+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:19.529306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:40:19.529306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":420,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-02-12T16:40:29.278904+0000","last_change":"2024-02-12T16:40:15.302314+0000","last_active":"2024-02-12T16:40:29.278904+0000","last_peered":"2024-02-12T16:40:29.278904+0000","last_clean":"2024-02-12T16:40:29.278904+0000","last_became_active":"2024-02-12T16:40:15.302015+0000","last_became_peered":"2024-02-12T16:40:15.302015+0000","last_unstale":"2024-02-12T16:40:29.278904+0000","last_undegraded":"2024-02-12T16:40:29.278904+0000","last_fullsized":"2024-02-12T16:40:29.278904+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:40:02.707515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:40:02.707515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10],"acting":[1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4264361984,"num_objects":65069,"num_object_clones":0,"num_object_copies":130138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4056,"num_objects_misplaced":4079,"num_objects_unfound":0,"num_objects_dirty":65069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65069,"num_write_kb":4164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2055,"num_bytes_recovered":134676480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8411348992,"data_stored":8411348992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65069,"ondisk_log_size":65069,"up":30,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":53,"acting":56,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":399431959038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":10,"up_from":83,"seq":356482286086,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1774720,"kb_used_data":1185016,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":91970432,"statfs":{"total":95995035648,"available":94177722368,"internally_reserved":0,"allocated":1213456384,"data_stored":1212842873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.8380000000000001,"15min":1.7490000000000001},"min":{"1min":0.40699999999999997,"5min":0.247,"15min":0.247},"max":{"1min":1.5469999999999999,"5min":111.935,"15min":162.38399999999999},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.9650000000000001,"15min":1.8899999999999999},"min":{"1min":0.65500000000000003,"5min":0.28499999999999998,"15min":0.26500000000000001},"max":{"1min":1.6519999999999999,"5min":111.316,"15min":162.01300000000001},"last":0.32200000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:40:17 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.157,"5min":1.157,"15min":1.157},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":2.1549999999999998,"15min":1.403},"min":{"1min":0.32600000000000001,"5min":0.30399999999999999,"15min":0.28100000000000003},"max":{"1min":1.4550000000000001,"5min":106.822,"15min":106.822},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.167,"5min":2.2429999999999999,"15min":1.425},"min":{"1min":0.50700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.605,"5min":107.88800000000001,"15min":107.88800000000001},"last":0.38900000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":2.1309999999999998,"15min":1.5029999999999999},"min":{"1min":0.39300000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.456,"5min":107.014,"15min":107.014},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.1339999999999999,"5min":2.165,"15min":1.3700000000000001},"min":{"1min":0.69199999999999995,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.98,"5min":110.80800000000001,"15min":110.80800000000001},"last":1.5680000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":0.93600000000000005,"15min":0.97399999999999998},"min":{"1min":0.58799999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.2090000000000001,"5min":4.9749999999999996,"15min":6.3689999999999998},"last":1.6899999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.032,"15min":1.034},"min":{"1min":0.57699999999999996,"5min":0.41699999999999998,"15min":0.318},"max":{"1min":1.6930000000000001,"5min":4.6189999999999998,"15min":7.3070000000000004},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":0.95099999999999996,"15min":1.0309999999999999},"min":{"1min":0.53800000000000003,"5min":0.35399999999999998,"15min":0.32700000000000001},"max":{"1min":2.4279999999999999,"5min":5.0110000000000001,"15min":6.343},"last":1.657},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.115,"15min":1.218},"min":{"1min":0.69599999999999995,"5min":0.41299999999999998,"15min":0.40999999999999998},"max":{"1min":1.8200000000000001,"5min":4.5570000000000004,"15min":7.2249999999999996},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":5.3419999999999996,"15min":2.5470000000000002},"min":{"1min":0.68100000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.327,"5min":460.90199999999999,"15min":460.90199999999999},"last":1.851},{"interface":"front","average":{"1min":1.5509999999999999,"5min":5.4400000000000004,"15min":2.5830000000000002},"min":{"1min":0.747,"5min":0.44,"15min":0.376},"max":{"1min":1.9830000000000001,"5min":460.93299999999999,"15min":460.93299999999999},"last":0.54900000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.93000000000000005,"15min":1.014},"min":{"1min":0.41599999999999998,"5min":0.40799999999999997,"15min":0.375},"max":{"1min":5.0419999999999998,"5min":5.0419999999999998,"15min":7.3280000000000003},"last":1.181},{"interface":"front","average":{"1min":1.212,"5min":1.1819999999999999,"15min":1.175},"min":{"1min":0.57599999999999996,"5min":0.434,"15min":0.374},"max":{"1min":4.532,"5min":5.306,"15min":6.4539999999999997},"last":0.65400000000000003}]},{"osd":8,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.3109999999999999,"15min":1.1459999999999999},"min":{"1min":0.61599999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.9870000000000001,"5min":8.4420000000000002,"15min":8.4420000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.2789999999999999,"15min":1.1100000000000001},"min":{"1min":0.52400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.6059999999999999,"5min":7.569,"15min":7.569},"last":0.44500000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.198,"15min":1.121},"min":{"1min":0.625,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.536,"5min":6.2560000000000002,"15min":6.6239999999999997},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.2909999999999999,"15min":1.1299999999999999},"min":{"1min":0.67600000000000005,"5min":0.48099999999999998,"15min":0.42899999999999999},"max":{"1min":2.1840000000000002,"5min":6.0030000000000001,"15min":7.3479999999999999},"last":1.752}]},{"osd":11,"last update":"Mon Feb 12 16:39:53 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.349,"15min":1.1910000000000001},"min":{"1min":1.004,"5min":0.44700000000000001,"15min":0.39400000000000002},"max":{"1min":1.879,"5min":6.5570000000000004,"15min":7.3929999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.7490000000000001,"5min":1.343,"15min":1.194},"min":{"1min":0.77400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.2480000000000002,"5min":5.46,"15min":6.5860000000000003},"last":0.58499999999999996}]}]},{"osd":9,"up_from":74,"seq":317827580431,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121472,"kb_used_data":527672,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92623680,"statfs":{"total":95995035648,"available":94846648320,"internally_reserved":0,"allocated":540336128,"data_stored":539722617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:40:32 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.3500000000000001,"15min":1.651},"min":{"1min":0.34200000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.5409999999999999,"5min":112.508,"15min":112.508},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":2.4300000000000002,"15min":1.694},"min":{"1min":0.33200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.7150000000000001,"5min":110.15300000000001,"15min":110.15300000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:40:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.016,"15min":0.96699999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28699999999999998},"max":{"1min":1.581,"5min":6.6799999999999997,"15min":6.6799999999999997},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.98099999999999998,"15min":0.96599999999999997},"min":{"1min":0.33200000000000002,"5min":0.29899999999999999,"15min":0.26100000000000001},"max":{"1min":1.611,"5min":5.367,"15min":5.7910000000000004},"last":2.1240000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.353,"15min":2.5339999999999998},"min":{"1min":0.54000000000000004,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.4370000000000001,"5min":27.879000000000001,"15min":52.097000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.204,"5min":1.3759999999999999,"15min":2.9180000000000001},"min":{"1min":0.71499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.863,"5min":28.446000000000002,"15min":51.209000000000003},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Feb 12 16:40:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.63},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.0069999999999999,"15min":0.97999999999999998},"min":{"1min":0.39500000000000002,"5min":0.33200000000000002,"15min":0.32200000000000001},"max":{"1min":1.7709999999999999,"5min":4.569,"15min":6.3090000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.258,"5min":1.089,"15min":1.0640000000000001},"min":{"1min":0.48599999999999999,"5min":0.39600000000000002,"15min":0.38},"max":{"1min":1.8620000000000001,"5min":4.633,"15min":6.3490000000000002},"last":0.72499999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":0.94599999999999995,"15min":1.1060000000000001},"min":{"1min":0.44900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.978,"5min":4.9930000000000003,"15min":6.8559999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.88400000000000001,"15min":1.073},"min":{"1min":0.434,"5min":0.376,"15min":0.376},"max":{"1min":1.681,"5min":4.319,"15min":6.8659999999999997},"last":2.2519999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1679999999999999,"15min":1.1180000000000001},"min":{"1min":0.56200000000000006,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.679,"5min":13.208,"15min":13.208},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.22,"15min":1.232},"min":{"1min":0.495,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.5229999999999999,"5min":12.968999999999999,"15min":14.718},"last":2.294}]},{"osd":7,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.95799999999999996,"15min":1.169},"min":{"1min":0.45600000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":6.7050000000000001,"15min":7.9489999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":1.1950000000000001,"15min":1.2529999999999999},"min":{"1min":0.53700000000000003,"5min":0.44800000000000001,"15min":0.43099999999999999},"max":{"1min":1.3220000000000001,"5min":6.7720000000000002,"15min":7.9160000000000004},"last":0.56799999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.1259999999999999,"15min":1.075},"min":{"1min":0.46999999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.9370000000000001,"5min":4.4269999999999996,"15min":7.593},"last":0.749},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.091,"15min":1.0700000000000001},"min":{"1min":0.46200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8400000000000001,"5min":4.2869999999999999,"15min":6.1980000000000004},"last":2.1789999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:40:04 2024","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.1339999999999999,"15min":1.085},"min":{"1min":0.54100000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.5209999999999999,"5min":6.7960000000000003,"15min":6.7960000000000003},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.548,"5min":1.3169999999999999,"15min":1.1759999999999999},"min":{"1min":0.83899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.2559999999999998,"5min":8.077,"15min":8.077},"last":0.85899999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:40:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5720000000000001,"5min":1.129,"15min":1.1040000000000001},"min":{"1min":0.61599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.0910000000000002,"5min":6.7329999999999997,"15min":6.7329999999999997},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.0660000000000001,"15min":1.089},"min":{"1min":0.65700000000000003,"5min":0.44500000000000001,"15min":0.41599999999999998},"max":{"1min":1.7470000000000001,"5min":4.6550000000000002,"15min":5.5069999999999997},"last":2.2130000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874776,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1316480,"kb_used_data":730872,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92428672,"statfs":{"total":95995035648,"available":94646960128,"internally_reserved":0,"allocated":748412928,"data_stored":747799417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.995,"15min":1.2689999999999999},"min":{"1min":0.67000000000000004,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.2749999999999999,"5min":7.4379999999999997,"15min":134.01499999999999},"last":2.0369999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.367,"15min":1.48},"min":{"1min":0.748,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.3120000000000001,"5min":9.0180000000000007,"15min":133.667},"last":0.371}]},{"osd":1,"last update":"Mon Feb 12 16:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0190000000000001,"15min":1.3320000000000001},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":69.956000000000003,"15min":69.956000000000003},"last":1.9970000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.9870000000000001,"15min":1.4079999999999999},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4580000000000002,"5min":71.010999999999996,"15min":71.010999999999996},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:40:29 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.046,"15min":0.86899999999999999},"min":{"1min":0.33900000000000002,"5min":0.30299999999999999,"15min":0.29299999999999998},"max":{"1min":2.0659999999999998,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.381,"15min":1.0640000000000001},"min":{"1min":0.41099999999999998,"5min":0.34000000000000002,"15min":0.26800000000000002},"max":{"1min":1.74,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:40:20 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.8109999999999999,"15min":1.042},"min":{"1min":0.45500000000000002,"5min":0.307,"15min":0.30399999999999999},"max":{"1min":1.823,"5min":25.919,"15min":25.919},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.8140000000000001,"15min":1.171},"min":{"1min":0.46899999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.788,"5min":25.984999999999999,"15min":25.984999999999999},"last":0.495}]},{"osd":4,"last update":"Mon Feb 12 16:39:43 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":0.93500000000000005,"15min":0.82899999999999996},"min":{"1min":0.80700000000000005,"5min":0.35399999999999998,"15min":0.29599999999999999},"max":{"1min":1.7010000000000001,"5min":8.609,"15min":8.609},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.369,"5min":1.339,"15min":1.024},"min":{"1min":0.77300000000000002,"5min":0.38500000000000001,"15min":0.34699999999999998},"max":{"1min":1.8069999999999999,"5min":9.0890000000000004,"15min":9.0890000000000004},"last":0.72699999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:39:43 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0880000000000001,"15min":1.008},"min":{"1min":0.64200000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.4339999999999999,"5min":7.2960000000000003,"15min":7.2960000000000003},"last":2.0950000000000002},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4159999999999999,"15min":1.1679999999999999},"min":{"1min":0.69199999999999995,"5min":0.36499999999999999,"15min":0.35199999999999998},"max":{"1min":1.613,"5min":9.0540000000000003,"15min":9.0540000000000003},"last":0.69299999999999995}]},{"osd":6,"last update":"Mon Feb 12 16:39:43 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.054,"15min":1.073},"min":{"1min":0.91800000000000004,"5min":0.39400000000000002,"15min":0.373},"max":{"1min":1.6259999999999999,"5min":8.5220000000000002,"15min":8.5220000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.391,"15min":1.25},"min":{"1min":0.89900000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.708,"5min":9.1500000000000004,"15min":9.1500000000000004},"last":0.75700000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.085,"15min":3.3860000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.9710000000000001,"5min":8.5709999999999997,"15min":706.67700000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.4870000000000001,"15min":3.5720000000000001},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":5.1079999999999997,"5min":9.1799999999999997,"15min":707.49099999999999},"last":1.5700000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:40:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.5980000000000001,"15min":1.181},"min":{"1min":0.47399999999999998,"5min":0.42499999999999999,"15min":0.34200000000000003},"max":{"1min":1.99,"5min":9.1170000000000009,"15min":9.1170000000000009},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.407,"5min":1.6990000000000001,"15min":1.282},"min":{"1min":0.61099999999999999,"5min":0.51700000000000002,"15min":0.46100000000000002},"max":{"1min":2.052,"5min":9.1980000000000004,"15min":9.1980000000000004},"last":0.78300000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:40:09 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1859999999999999,"15min":1.0169999999999999},"min":{"1min":0.95099999999999996,"5min":0.39000000000000001,"15min":0.36699999999999999},"max":{"1min":1.8540000000000001,"5min":8.452,"15min":8.452},"last":2.1240000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.3779999999999999,"15min":1.0760000000000001},"min":{"1min":1.145,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.0219999999999998,"5min":5.0720000000000001,"15min":5.0720000000000001},"last":0.67300000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:40:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.206,"15min":1.0740000000000001},"min":{"1min":0.86899999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6240000000000001,"5min":8.6790000000000003,"15min":8.6790000000000003},"last":2.0640000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.3600000000000001,"15min":1.0629999999999999},"min":{"1min":0.623,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.982,"5min":5.141,"15min":5.2320000000000002},"last":3.153}]}]},{"osd":7,"up_from":56,"seq":240518169103,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594684,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93150468,"statfs":{"total":95995035648,"available":95386079232,"internally_reserved":0,"allocated":905216,"data_stored":452732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:39:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":4.2969999999999997,"15min":2.9489999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.252},"max":{"1min":2.113,"5min":184.77199999999999,"15min":187.33699999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.018,"5min":4.2329999999999997,"15min":2.9359999999999999},"min":{"1min":0.315,"5min":0.28699999999999998,"15min":0.27400000000000002},"max":{"1min":3.492,"5min":184.64599999999999,"15min":187.48400000000001},"last":1.242}]},{"osd":1,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.149,"15min":1.3560000000000001},"min":{"1min":0.309,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.4630000000000001,"5min":14.106,"15min":14.106},"last":1.335},{"interface":"front","average":{"1min":1.1859999999999999,"5min":0.94599999999999995,"15min":1.0800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.2690000000000001,"5min":7.2690000000000001,"15min":7.2690000000000001},"last":1.0740000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:32:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.046,"15min":1.143},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.9299999999999997,"5min":5.4340000000000002,"15min":35.655000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.3089999999999999,"15min":1.3360000000000001},"min":{"1min":0.378,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":7.0599999999999996,"5min":7.6040000000000001,"15min":35.164000000000001},"last":0.998}]},{"osd":3,"last update":"Mon Feb 12 16:38:35 2024","interfaces":[{"interface":"back","average":{"1min":4.7119999999999997,"5min":1.9590000000000001,"15min":1.353},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.26100000000000001},"max":{"1min":77.603999999999999,"5min":77.603999999999999,"15min":77.603999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":4.54,"5min":2.1080000000000001,"15min":1.492},"min":{"1min":0.35499999999999998,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":78.031999999999996,"5min":78.031999999999996,"15min":78.031999999999996},"last":1.113}]},{"osd":4,"last update":"Mon Feb 12 16:39:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.81299999999999994,"15min":1.1359999999999999},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.26200000000000001},"max":{"1min":3.6120000000000001,"5min":4.5529999999999999,"15min":10.958},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":0.91000000000000003,"15min":1.1399999999999999},"min":{"1min":0.433,"5min":0.312,"15min":0.25800000000000001},"max":{"1min":3.5750000000000002,"5min":4.5940000000000003,"15min":13.925000000000001},"last":1.034}]},{"osd":5,"last update":"Mon Feb 12 16:39:04 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.753,"15min":2.02},"min":{"1min":0.32700000000000001,"5min":0.32200000000000001,"15min":0.26500000000000001},"max":{"1min":1.292,"5min":4.3520000000000003,"15min":216.84299999999999},"last":1.2649999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.89000000000000001,"15min":2.1339999999999999},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28299999999999997},"max":{"1min":1.4770000000000001,"5min":5.1550000000000002,"15min":217.02600000000001},"last":1.6439999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:39:04 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.1359999999999999,"15min":1.278},"min":{"1min":0.50900000000000001,"5min":0.33800000000000002,"15min":0.31},"max":{"1min":3.9750000000000001,"5min":4.649,"15min":11.380000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.1299999999999999,"15min":1.3859999999999999},"min":{"1min":0.38400000000000001,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":3.996,"5min":4.6079999999999997,"15min":8.0719999999999992},"last":1.5049999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:38:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.0229999999999999,"15min":2.9249999999999998},"min":{"1min":0.441,"5min":0.377,"15min":0.377},"max":{"1min":5.2229999999999999,"5min":5.2229999999999999,"15min":498.68299999999999},"last":1.849},{"interface":"front","average":{"1min":1.296,"5min":1.159,"15min":3.1459999999999999},"min":{"1min":0.5,"5min":0.40799999999999997,"15min":0.35999999999999999},"max":{"1min":5.2350000000000003,"5min":14.021000000000001,"15min":498.69999999999999},"last":1.587}]},{"osd":9,"last update":"Mon Feb 12 16:38:28 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":2.0129999999999999,"15min":1.444},"min":{"1min":0.47699999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":8.5299999999999994,"5min":90.025999999999996,"15min":90.025999999999996},"last":2.0609999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.9410000000000001,"15min":1.5029999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.40100000000000002},"max":{"1min":1.671,"5min":90.049999999999997,"15min":90.049999999999997},"last":1.825}]},{"osd":10,"last update":"Mon Feb 12 16:39:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.1579999999999999,"15min":3.4780000000000002},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.376},"max":{"1min":3.4460000000000002,"5min":5.6260000000000003,"15min":497.52699999999999},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.169,"15min":3.5369999999999999},"min":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.39200000000000002},"max":{"1min":2.0129999999999999,"5min":4.7430000000000003,"15min":496.29599999999999},"last":1.677}]},{"osd":11,"last update":"Mon Feb 12 16:38:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":0.51600000000000001,"5min":0.501,"15min":0.43099999999999999},"max":{"1min":3.5550000000000002,"5min":4.0389999999999997,"15min":8.4199999999999999},"last":1.966},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.0860000000000001,"15min":1.2649999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.46899999999999997},"max":{"1min":1.724,"5min":5.4139999999999997,"15min":7.0679999999999996},"last":1.8879999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599104,"kb_used_data":1208,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":93146048,"statfs":{"total":95995035648,"available":95381553152,"internally_reserved":0,"allocated":1236992,"data_stored":623481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.376}]},{"osd":1,"last update":"Mon Feb 12 16:39:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.0820000000000001,"15min":1.056},"min":{"1min":0.78500000000000003,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":2.7200000000000002,"5min":6.0780000000000003,"15min":6.1870000000000003},"last":2.806},{"interface":"front","average":{"1min":1.079,"5min":1.2509999999999999,"15min":1.153},"min":{"1min":0.85899999999999999,"5min":0.28699999999999998,"15min":0.25700000000000001},"max":{"1min":1.6499999999999999,"5min":6.6559999999999997,"15min":6.6959999999999997},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:39:35 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.5349999999999999,"15min":1.4330000000000001},"min":{"1min":0.72999999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":1.6950000000000001,"5min":14.68,"15min":28.739999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.034,"5min":1.274,"15min":1.405},"min":{"1min":0.71499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.762,"5min":15.138,"15min":32.328000000000003},"last":0.58799999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.3839999999999999,"15min":3.4649999999999999},"min":{"1min":0.68999999999999995,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.75,"5min":14.981999999999999,"15min":365.58999999999997},"last":2.2770000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.5329999999999999,"15min":4.141},"min":{"1min":0.628,"5min":0.317,"15min":0.317},"max":{"1min":1.504,"5min":14.788,"15min":366.93299999999999},"last":0.51900000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:39:35 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":0.95799999999999996,"15min":1.1539999999999999},"min":{"1min":0.68300000000000005,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":1.847,"5min":5.1849999999999996,"15min":8.4559999999999995},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.131,"15min":1.1060000000000001},"min":{"1min":0.85899999999999999,"5min":0.33400000000000002,"15min":0.23999999999999999},"max":{"1min":2.5430000000000001,"5min":6.1059999999999999,"15min":7.7610000000000001},"last":0.436}]},{"osd":5,"last update":"Mon Feb 12 16:39:35 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.0069999999999999,"15min":1.222},"min":{"1min":0.68300000000000005,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.4039999999999999,"5min":5.4740000000000002,"15min":8.5730000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.3,"5min":1.129,"15min":1.1779999999999999},"min":{"1min":0.89000000000000001,"5min":0.379,"15min":0.33100000000000002},"max":{"1min":2.1150000000000002,"5min":6.0359999999999996,"15min":10.52},"last":0.60499999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:39:06 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.29,"15min":1.256},"min":{"1min":0.54800000000000004,"5min":0.38100000000000001,"15min":0.29499999999999998},"max":{"1min":4.4429999999999996,"5min":6.359,"15min":9.1199999999999992},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.0600000000000001,"15min":1.173},"min":{"1min":0.505,"5min":0.32800000000000001,"15min":0.32200000000000001},"max":{"1min":4.476,"5min":6.9279999999999999,"15min":8.8879999999999999},"last":2.0640000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:39:35 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.3460000000000001,"15min":2.8839999999999999},"min":{"1min":0.93999999999999995,"5min":0.45600000000000002,"15min":0.35699999999999998},"max":{"1min":3.113,"5min":6.2590000000000003,"15min":501.892},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.1879999999999999,"15min":2.794},"min":{"1min":1.105,"5min":0.38700000000000001,"15min":0.371},"max":{"1min":2.2679999999999998,"5min":5.3150000000000004,"15min":501.76600000000002},"last":0.58099999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:40:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.3819999999999999,"15min":1.2430000000000001},"min":{"1min":0.78500000000000003,"5min":0.39100000000000001,"15min":0.35299999999999998},"max":{"1min":2.1150000000000002,"5min":6.319,"15min":6.319},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.22,"5min":1.1619999999999999,"15min":1.212},"min":{"1min":0.91600000000000004,"5min":0.44800000000000001,"15min":0.42799999999999999},"max":{"1min":1.97,"5min":7.0119999999999996,"15min":7.6870000000000003},"last":0.48499999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:40:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.3680000000000001,"15min":1.395},"min":{"1min":0.90200000000000002,"5min":0.49299999999999999,"15min":0.42499999999999999},"max":{"1min":2.0880000000000001,"5min":11.536,"15min":11.536},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.4430000000000001,"15min":1.468},"min":{"1min":0.96799999999999997,"5min":0.55100000000000005,"15min":0.48099999999999998},"max":{"1min":2.2799999999999998,"5min":11.585000000000001,"15min":11.585000000000001},"last":0.72299999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:40:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4850000000000001,"15min":1.486},"min":{"1min":0.83699999999999997,"5min":0.63300000000000001,"15min":0.49299999999999999},"max":{"1min":2.306,"5min":6.2910000000000004,"15min":8.6649999999999991},"last":0.622},{"interface":"front","average":{"1min":1.339,"5min":1.2090000000000001,"15min":1.149},"min":{"1min":0.86699999999999999,"5min":0.52600000000000002,"15min":0.40699999999999997},"max":{"1min":2.161,"5min":7.04,"15min":7.04},"last":0.54200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1993536,"kb_used_data":1091128,"kb_used_omap":0,"kb_used_meta":902400,"kb_avail":91751616,"statfs":{"total":95995035648,"available":93953654784,"internally_reserved":0,"allocated":1117315072,"data_stored":1116701561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":924057600},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:40:00 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":1.1579999999999999,"15min":1.0620000000000001},"min":{"1min":0.70999999999999996,"5min":0.24399999999999999,"15min":0.22900000000000001},"max":{"1min":1.133,"5min":9.1999999999999993,"15min":16.632999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.995,"15min":1.2709999999999999},"min":{"1min":0.45500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.3700000000000001,"5min":4.8730000000000002,"15min":10.26},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:39:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.4179999999999999,"15min":1.2669999999999999},"min":{"1min":0.67500000000000004,"5min":0.28199999999999997,"15min":0.22700000000000001},"max":{"1min":1.5860000000000001,"5min":10.755000000000001,"15min":12.778},"last":0.501},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.321,"15min":1.3009999999999999},"min":{"1min":0.65600000000000003,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.23,"5min":7.3600000000000003,"15min":9.9689999999999994},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1859999999999999,"15min":1.22},"min":{"1min":0.32500000000000001,"5min":0.26800000000000002,"15min":0.248},"max":{"1min":3.8380000000000001,"5min":7.3019999999999996,"15min":26.664000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.4710000000000001,"15min":1.4399999999999999},"min":{"1min":0.432,"5min":0.29699999999999999,"15min":0.26200000000000001},"max":{"1min":2.4039999999999999,"5min":11.699999999999999,"15min":27.148},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:39:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.90400000000000003,"15min":1.1359999999999999},"min":{"1min":0.72099999999999997,"5min":0.312,"15min":0.31},"max":{"1min":1.5600000000000001,"5min":4.6680000000000001,"15min":8.7479999999999993},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.0309999999999999,"15min":1.2010000000000001},"min":{"1min":0.75900000000000001,"5min":0.35499999999999998,"15min":0.30199999999999999},"max":{"1min":1.452,"5min":4.9379999999999997,"15min":7.1710000000000003},"last":0.438}]},{"osd":5,"last update":"Mon Feb 12 16:39:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1299999999999999,"15min":4.125},"min":{"1min":0.77500000000000002,"5min":0.309,"15min":0.309},"max":{"1min":1.4079999999999999,"5min":8.4299999999999997,"15min":524.85299999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.353,"15min":4.1589999999999998},"min":{"1min":1.0780000000000001,"5min":0.35499999999999998,"15min":0.32800000000000001},"max":{"1min":1.6140000000000001,"5min":8.9540000000000006,"15min":519.89700000000005},"last":0.34999999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:40:18 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.353,"15min":1.353},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.522,"5min":1.522,"15min":1.522},"min":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.46200000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:38:59 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0629999999999999,"15min":3.2370000000000001},"min":{"1min":0.44700000000000001,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":522.79200000000003},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.25,"15min":3.198},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":4.3360000000000003,"5min":9.0069999999999997,"15min":520.64499999999998},"last":1.5800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:40:00 2024","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.3779999999999999,"15min":3.3239999999999998},"min":{"1min":1.3460000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.917,"5min":9.1430000000000007,"15min":548.91600000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.4550000000000001,"15min":3.3980000000000001},"min":{"1min":1.518,"5min":0.47799999999999998,"15min":0.36599999999999999},"max":{"1min":1.9690000000000001,"5min":9.1739999999999995,"15min":552.42100000000005},"last":0.58599999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:40:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.2889999999999999,"15min":1.2789999999999999},"min":{"1min":0.73999999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.9610000000000001,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.3919999999999999,"15min":1.319},"min":{"1min":0.86699999999999999,"5min":0.45300000000000001,"15min":0.436},"max":{"1min":1.9850000000000001,"5min":8.4969999999999999,"15min":10.106},"last":0.71599999999999997}]},{"osd":10,"last update":"Mon Feb 12 16:40:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.339,"15min":3.6709999999999998},"min":{"1min":0.98699999999999999,"5min":0.42999999999999999,"15min":0.39500000000000002},"max":{"1min":1.9450000000000001,"5min":8.3469999999999995,"15min":548.93499999999995},"last":2.222},{"interface":"front","average":{"1min":1.714,"5min":1.413,"15min":3.6720000000000002},"min":{"1min":1.3939999999999999,"5min":0.51000000000000001,"15min":0.46999999999999997},"max":{"1min":2.0249999999999999,"5min":9.0440000000000005,"15min":549.12099999999998},"last":2.2109999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:39:44 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":2.347,"15min":1.6930000000000001},"min":{"1min":1.2969999999999999,"5min":0.52400000000000002,"15min":0.47099999999999997},"max":{"1min":1.9830000000000001,"5min":88.679000000000002,"15min":88.679000000000002},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":2.3519999999999999,"15min":1.702},"min":{"1min":0.93600000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"max":{"1min":1.8540000000000001,"5min":88.519000000000005,"15min":88.519000000000005},"last":0.61799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738970,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1428992,"kb_used_data":546168,"kb_used_omap":0,"kb_used_meta":882816,"kb_avail":92316160,"statfs":{"total":95995035648,"available":94531747840,"internally_reserved":0,"allocated":559276032,"data_stored":558662521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":904003584},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.1299999999999999,"15min":1.036},"min":{"1min":0.69599999999999995,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.607,"5min":13.359,"15min":13.359},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.321,"15min":1.218},"min":{"1min":0.52600000000000002,"5min":0.29899999999999999,"15min":0.22600000000000001},"max":{"1min":1.365,"5min":8.3780000000000001,"15min":8.3780000000000001},"last":4.093}]},{"osd":2,"last update":"Mon Feb 12 16:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":2.7360000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:40:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.2529999999999999,"15min":2.383},"min":{"1min":0.54000000000000004,"5min":0.252,"15min":0.252},"max":{"1min":1.411,"5min":14.315,"15min":377.06299999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.756,"15min":2.581},"min":{"1min":0.58299999999999996,"5min":0.24399999999999999,"15min":0.224},"max":{"1min":1.419,"5min":13.986000000000001,"15min":379.18000000000001},"last":2.8180000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:40:14 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.028,"15min":1.875},"min":{"1min":0.93000000000000005,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.6950000000000001,"5min":7.2869999999999999,"15min":193.77000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.4830000000000001,"15min":1.9770000000000001},"min":{"1min":0.65700000000000003,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":1.536,"5min":8.4949999999999992,"15min":189.625},"last":2.7909999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:39:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.1140000000000001,"15min":1.8300000000000001},"min":{"1min":0.63,"5min":0.311,"15min":0.28699999999999998},"max":{"1min":1.343,"5min":5.9240000000000004,"15min":181.46199999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2789999999999999,"15min":1.9139999999999999},"min":{"1min":0.77400000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":1.796,"5min":6.0880000000000001,"15min":186.41999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:40:19 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":0.61299999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:39:13 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.24,"15min":2.706},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.374},"max":{"1min":1.835,"5min":6.1130000000000004,"15min":191.47200000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4019999999999999,"15min":2.823},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":1.897,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.081}]},{"osd":8,"last update":"Mon Feb 12 16:39:45 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.377,"15min":1.3009999999999999},"min":{"1min":0.80400000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.9119999999999999,"5min":6.056,"15min":7.3920000000000003},"last":0.498},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.486,"15min":1.401},"min":{"1min":0.95499999999999996,"5min":0.51200000000000001,"15min":0.48299999999999998},"max":{"1min":1.9690000000000001,"5min":6.1589999999999998,"15min":7.5060000000000002},"last":0.64400000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:40:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.5049999999999999,"15min":1.2649999999999999},"min":{"1min":0.60399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.8660000000000001,"5min":5.9279999999999999,"15min":8.4629999999999992},"last":2.8839999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.599,"15min":1.4019999999999999},"min":{"1min":0.53400000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.0710000000000002,"5min":12.943,"15min":12.943},"last":0.60299999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:40:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.4179999999999999,"15min":1.331},"min":{"1min":1.1850000000000001,"5min":0.44,"15min":0.44},"max":{"1min":2.0510000000000002,"5min":13.157,"15min":13.157},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.7330000000000001,"5min":1.5049999999999999,"15min":1.4379999999999999},"min":{"1min":1.2170000000000001,"5min":0.48899999999999999,"15min":0.48299999999999998},"max":{"1min":2.1629999999999998,"5min":13.238,"15min":13.238},"last":0.71199999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:39:45 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.704,"15min":1.4870000000000001},"min":{"1min":0.96199999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.8320000000000001,"5min":8.4280000000000008,"15min":8.4280000000000008},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.3740000000000001,"15min":1.347},"min":{"1min":0.94899999999999995,"5min":0.47599999999999998,"15min":0.45200000000000001},"max":{"1min":1.7769999999999999,"5min":7.3799999999999999,"15min":7.4210000000000003},"last":2.851}]}]},{"osd":2,"up_from":21,"seq":90194313799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1493504,"kb_used_data":895608,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":92251648,"statfs":{"total":95995035648,"available":94465687552,"internally_reserved":0,"allocated":917102592,"data_stored":916489081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:40:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":3.496},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.69599999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1830000000000001,"15min":1.0820000000000001},"min":{"1min":0.373,"5min":0.252,"15min":0.19500000000000001},"max":{"1min":1.48,"5min":7.6050000000000004,"15min":7.6050000000000004},"last":3.552},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.6359999999999999,"15min":1.284},"min":{"1min":0.42499999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.486,"5min":27.562999999999999,"15min":27.562999999999999},"last":1.3220000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:39:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.5629999999999999,"15min":1.504},"min":{"1min":0.47699999999999998,"5min":0.28100000000000003,"15min":0.253},"max":{"1min":1.403,"5min":8.1859999999999999,"15min":54.091000000000001},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.385,"15min":1.411},"min":{"1min":0.46000000000000002,"5min":0.29099999999999998,"15min":0.25600000000000001},"max":{"1min":1.411,"5min":9.1110000000000007,"15min":54.203000000000003},"last":2.1539999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:40:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2050000000000001,"15min":1.1599999999999999},"min":{"1min":0.72499999999999998,"5min":0.29499999999999998,"15min":0.28799999999999998},"max":{"1min":1.5129999999999999,"5min":8.3439999999999994,"15min":10.890000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.349,"15min":1.2330000000000001},"min":{"1min":0.879,"5min":0.30099999999999999,"15min":0.28999999999999998},"max":{"1min":1.556,"5min":6.1699999999999999,"15min":7.0019999999999998},"last":0.505}]},{"osd":5,"last update":"Mon Feb 12 16:39:51 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.069,"15min":1.077},"min":{"1min":0.57099999999999995,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.627,"5min":8.7509999999999994,"15min":8.7509999999999994},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.2969999999999999,"15min":1.3120000000000001},"min":{"1min":0.624,"5min":0.311,"15min":0.31},"max":{"1min":1.6100000000000001,"5min":6.2220000000000004,"15min":11.081},"last":3.6080000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:40:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.27,"15min":1.2869999999999999},"min":{"1min":0.71999999999999997,"5min":0.36599999999999999,"15min":0.316},"max":{"1min":1.742,"5min":8.4510000000000005,"15min":8.4510000000000005},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.206,"15min":1.256},"min":{"1min":0.78400000000000003,"5min":0.379,"15min":0.35099999999999998},"max":{"1min":1.885,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":3.6520000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.58399999999999996,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":1.9019999999999999,"5min":6.2560000000000002,"15min":11.044},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2529999999999999,"15min":1.248},"min":{"1min":0.44600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.4219999999999997,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":1.6499999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:39:38 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.397,"15min":1.228},"min":{"1min":0.748,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.952,"5min":8.3870000000000005,"15min":8.3870000000000005},"last":1.302},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.345,"15min":1.2150000000000001},"min":{"1min":0.93799999999999994,"5min":0.41199999999999998,"15min":0.377},"max":{"1min":2.1709999999999998,"5min":8.9000000000000004,"15min":8.9000000000000004},"last":3.6869999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:40:12 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.6579999999999999,"15min":1.407},"min":{"1min":1.095,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.847,"5min":7.7759999999999998,"15min":10.972},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4950000000000001,"15min":1.407},"min":{"1min":1.02,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.7989999999999999,"5min":6.29,"15min":11.006},"last":0.753}]},{"osd":10,"last update":"Mon Feb 12 16:39:55 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.5529999999999999,"15min":1.3560000000000001},"min":{"1min":0.83999999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.173,"5min":8.4209999999999994,"15min":8.4209999999999994},"last":1.345},{"interface":"front","average":{"1min":1.502,"5min":1.5329999999999999,"15min":1.3899999999999999},"min":{"1min":0.80200000000000005,"5min":0.45800000000000002,"15min":0.41799999999999998},"max":{"1min":2.2400000000000002,"5min":8.4879999999999995,"15min":8.4879999999999995},"last":1.369}]},{"osd":11,"last update":"Mon Feb 12 16:40:14 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.4850000000000001,"15min":1.29},"min":{"1min":1.1890000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.1160000000000001,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":3.7400000000000002},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.3919999999999999},"min":{"1min":1.181,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.2930000000000001,"5min":8.968,"15min":8.968},"last":3.774}]}]},{"osd":3,"up_from":522,"seq":2241972928724,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870528,"kb_used_data":526008,"kb_used_omap":91,"kb_used_meta":344420,"kb_avail":92874624,"statfs":{"total":95995035648,"available":95103614976,"internally_reserved":0,"allocated":538632192,"data_stored":538018681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":352686096},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.302,"15min":1.226},"min":{"1min":0.47399999999999998,"5min":0.23499999999999999,"15min":0.20699999999999999},"max":{"1min":1.218,"5min":10.101000000000001,"15min":34.055999999999997},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.018,"5min":1.3500000000000001,"15min":1.2689999999999999},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.1600000000000001,"5min":9.3490000000000002,"15min":27.600000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":2.798,"5min":2.798,"15min":2.798},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.409,"15min":1.1659999999999999},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.252},"max":{"1min":2.2770000000000001,"5min":6.4950000000000001,"15min":8.8109999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.796,"15min":1.2809999999999999},"min":{"1min":0.32300000000000001,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":2.0430000000000001,"5min":10.369999999999999,"15min":10.369999999999999},"last":2.0430000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.468,"15min":1.137},"min":{"1min":0.42199999999999999,"5min":0.318,"15min":0.27400000000000002},"max":{"1min":1.605,"5min":6.8220000000000001,"15min":6.8220000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.1559999999999999,"15min":1.137},"min":{"1min":0.36699999999999999,"5min":0.32700000000000001,"15min":0.29599999999999999},"max":{"1min":1.651,"5min":6.6840000000000002,"15min":6.806},"last":0.76800000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.206,"15min":1.1539999999999999},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.4890000000000001,"5min":9.4100000000000001,"15min":11.884},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.163,"5min":1.4259999999999999,"15min":1.2809999999999999},"min":{"1min":0.45600000000000002,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":1.8300000000000001,"5min":6.6680000000000001,"15min":12.164999999999999},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.2829999999999999,"15min":1.2210000000000001},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":1.619,"5min":9.4689999999999994,"15min":9.4689999999999994},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.579,"15min":1.3220000000000001},"min":{"1min":0.496,"5min":0.377,"15min":0.30499999999999999},"max":{"1min":2.1890000000000001,"5min":9.4640000000000004,"15min":9.4640000000000004},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.5149999999999999,"15min":1.298},"min":{"1min":0.55000000000000004,"5min":0.36199999999999999,"15min":0.33300000000000002},"max":{"1min":7.9909999999999997,"5min":7.9909999999999997,"15min":7.9909999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.609,"5min":1.3859999999999999,"15min":1.335},"min":{"1min":0.621,"5min":0.371,"15min":0.371},"max":{"1min":6.0899999999999999,"5min":6.7590000000000003,"15min":7.2249999999999996},"last":0.59299999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.5009999999999999,"15min":1.2370000000000001},"min":{"1min":0.53900000000000003,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":2.3490000000000002,"5min":5.9669999999999996,"15min":5.9669999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.5900000000000001,"15min":1.29},"min":{"1min":0.57799999999999996,"5min":0.42299999999999999,"15min":0.35599999999999998},"max":{"1min":2.383,"5min":7.9569999999999999,"15min":7.9569999999999999},"last":2.109}]},{"osd":9,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.6319999999999999,"15min":1.3779999999999999},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.39700000000000002},"max":{"1min":2.4390000000000001,"5min":8.9350000000000005,"15min":8.9350000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.5860000000000001,"15min":1.343},"min":{"1min":0.54700000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.476,"5min":9.5239999999999991,"15min":9.5239999999999991},"last":0.68999999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4330000000000001,"15min":1.272},"min":{"1min":0.45500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.8919999999999999,"5min":9.5649999999999995,"15min":9.5649999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.373,"5min":1.661,"15min":1.456},"min":{"1min":0.58599999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.5299999999999998,"5min":10.032999999999999,"15min":15.140000000000001},"last":2.097}]},{"osd":11,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.4239999999999999,"15min":1.5640000000000001},"min":{"1min":0.59899999999999998,"5min":0.437,"15min":0.437},"max":{"1min":2.0840000000000001,"5min":9.6059999999999999,"15min":73.037999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.71,"15min":1.629},"min":{"1min":0.64900000000000002,"5min":0.53200000000000003,"15min":0.49299999999999999},"max":{"1min":2.7069999999999999,"5min":11.445,"15min":73.057000000000002},"last":0.80700000000000005}]}]},{"osd":4,"up_from":32,"seq":137438954039,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1613760,"kb_used_data":723704,"kb_used_omap":0,"kb_used_meta":890048,"kb_avail":92131392,"statfs":{"total":95995035648,"available":94342545408,"internally_reserved":0,"allocated":741072896,"data_stored":740459385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911409152},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:40:28 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.996,"15min":1.079},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.028,"5min":4.1050000000000004,"15min":4.1050000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.123,"15min":1.1890000000000001},"min":{"1min":0.40200000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.5569999999999999,"5min":5.0599999999999996,"15min":5.0599999999999996},"last":0.59999999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:40:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.4610000000000001,"15min":1.407},"min":{"1min":0.46899999999999997,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":1.5980000000000001,"5min":30.137,"15min":56.188000000000002},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.617,"15min":1.5229999999999999},"min":{"1min":0.63600000000000001,"5min":0.28100000000000003,"15min":0.25700000000000001},"max":{"1min":5.2039999999999997,"5min":32.481999999999999,"15min":55.414000000000001},"last":0.89500000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:40:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":212.685},{"interface":"front","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.831,"5min":3.831,"15min":3.831},"last":210.95699999999999}]},{"osd":3,"last update":"Mon Feb 12 16:40:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.556,"15min":1.3859999999999999},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.311},"max":{"1min":1.482,"5min":30.282,"15min":30.282},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.875,"5min":1.54,"15min":1.319},"min":{"1min":0.41399999999999998,"5min":0.308,"15min":0.25800000000000001},"max":{"1min":1.1499999999999999,"5min":30.478000000000002,"15min":30.478000000000002},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.877,"15min":1.1699999999999999},"min":{"1min":0.66900000000000004,"5min":0.252,"15min":0.252},"max":{"1min":2.1880000000000002,"5min":7.8310000000000004,"15min":10.289999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.26,"5min":0.99199999999999999,"15min":1.2929999999999999},"min":{"1min":0.74099999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.4780000000000002,"5min":7.8620000000000001,"15min":10.321999999999999},"last":0.86699999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:40:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1,"15min":1.071},"min":{"1min":0.43099999999999999,"5min":0.34599999999999997,"15min":0.26400000000000001},"max":{"1min":1.258,"5min":5.7560000000000002,"15min":7.5289999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.2110000000000001,"15min":1.2130000000000001},"min":{"1min":0.36499999999999999,"5min":0.33100000000000002,"15min":0.28899999999999998},"max":{"1min":1.6080000000000001,"5min":11.968999999999999,"15min":11.968999999999999},"last":0.755}]},{"osd":7,"last update":"Mon Feb 12 16:38:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.98099999999999998,"15min":1.075},"min":{"1min":0.35699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.6580000000000004,"5min":5.4160000000000004,"15min":6.8659999999999997},"last":0.753},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.0249999999999999,"15min":1.248},"min":{"1min":0.377,"5min":0.377,"15min":0.34599999999999997},"max":{"1min":8.8000000000000007,"5min":8.8000000000000007,"15min":9.1440000000000001},"last":0.77100000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.0529999999999999,"15min":1.149},"min":{"1min":0.88,"5min":0.40300000000000002,"15min":0.38300000000000001},"max":{"1min":2.306,"5min":5.1950000000000003,"15min":7.6890000000000001},"last":1.139},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.107,"15min":1.2230000000000001},"min":{"1min":0.79100000000000004,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.657,"5min":7.9349999999999996,"15min":7.9349999999999996},"last":0.95799999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:40:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.8480000000000001,"15min":1.425},"min":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.39300000000000002},"max":{"1min":1.8380000000000001,"5min":63.545000000000002,"15min":63.545000000000002},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.8819999999999999,"15min":1.52},"min":{"1min":0.57899999999999996,"5min":0.46600000000000003,"15min":0.36699999999999999},"max":{"1min":1.968,"5min":63.421999999999997,"15min":63.421999999999997},"last":1.23}]},{"osd":10,"last update":"Mon Feb 12 16:40:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.2070000000000001,"15min":1.3200000000000001},"min":{"1min":0.51800000000000002,"5min":0.45300000000000001,"15min":0.39200000000000002},"max":{"1min":1.9790000000000001,"5min":5.548,"15min":7.7619999999999996},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.1719999999999999,"15min":1.141},"min":{"1min":0.54000000000000004,"5min":0.41799999999999998,"15min":0.41199999999999998},"max":{"1min":1.6759999999999999,"5min":4.0490000000000004,"15min":5.6459999999999999},"last":0.84999999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:40:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.294,"15min":1.2609999999999999},"min":{"1min":0.56100000000000005,"5min":0.54700000000000004,"15min":0.45400000000000001},"max":{"1min":1.7609999999999999,"5min":5.5739999999999998,"15min":7.734},"last":1.034},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.264,"15min":1.321},"min":{"1min":0.55200000000000005,"5min":0.51000000000000001,"15min":0.46400000000000002},"max":{"1min":1.849,"5min":5.2539999999999996,"15min":7.7969999999999997},"last":1.2789999999999999}]}]},{"osd":5,"up_from":39,"seq":167503725103,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949824,"kb_used_data":1351928,"kb_used_omap":0,"kb_used_meta":597888,"kb_avail":91795328,"statfs":{"total":95995035648,"available":93998415872,"internally_reserved":0,"allocated":1384374272,"data_stored":1383760761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":612237312},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.456,"15min":1.6559999999999999},"min":{"1min":0.378,"5min":0.245,"15min":0.245},"max":{"1min":1.3759999999999999,"5min":38.366,"15min":38.366},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.7170000000000001,"15min":1.6930000000000001},"min":{"1min":0.55100000000000005,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.222,"5min":37.310000000000002,"15min":37.310000000000002},"last":0.432}]},{"osd":1,"last update":"Mon Feb 12 16:39:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.995,"15min":1.7869999999999999},"min":{"1min":0.74099999999999999,"5min":0.318,"15min":0.245},"max":{"1min":1.387,"5min":4.0999999999999996,"15min":221.96600000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.232,"5min":1.105,"15min":1.9119999999999999},"min":{"1min":0.88600000000000001,"5min":0.40500000000000003,"15min":0.311},"max":{"1min":1.6919999999999999,"5min":4.1500000000000004,"15min":222.06200000000001},"last":0.59999999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:39:55 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.139,"15min":1.3009999999999999},"min":{"1min":0.81599999999999995,"5min":0.373,"15min":0.28799999999999998},"max":{"1min":1.534,"5min":4.923,"15min":40.021999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.3069999999999999,"15min":1.3779999999999999},"min":{"1min":0.35499999999999998,"5min":0.312,"15min":0.27600000000000002},"max":{"1min":1.3560000000000001,"5min":8.2439999999999998,"15min":40.225000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:39:44 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.371,"15min":1.476},"min":{"1min":0.59799999999999998,"5min":0.35999999999999999,"15min":0.27200000000000002},"max":{"1min":1.4810000000000001,"5min":7.2300000000000004,"15min":22.484000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.054,"5min":1.3560000000000001,"15min":1.3360000000000001},"min":{"1min":0.40000000000000002,"5min":0.36299999999999999,"15min":0.29299999999999998},"max":{"1min":1.573,"5min":5.5510000000000002,"15min":23.34},"last":0.46700000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:39:55 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.93799999999999994,"15min":1.1970000000000001},"min":{"1min":0.64800000000000002,"5min":0.312,"15min":0.23200000000000001},"max":{"1min":1.706,"5min":3.4710000000000001,"15min":41.908000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.103,"5min":1.097,"15min":1.351},"min":{"1min":0.503,"5min":0.32000000000000001,"15min":0.27600000000000002},"max":{"1min":1.593,"5min":5.5389999999999997,"15min":42.430999999999997},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":2.028,"15min":1.456},"min":{"1min":0.38700000000000001,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":1.2270000000000001,"5min":68.263999999999996,"15min":68.263999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.92700000000000005,"5min":2.0579999999999998,"15min":1.5389999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27100000000000002},"max":{"1min":1.5720000000000001,"5min":67.873000000000005,"15min":67.873000000000005},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:38:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.86799999999999999,"15min":1.1830000000000001},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.28799999999999998},"max":{"1min":1.9570000000000001,"5min":5.6609999999999996,"15min":9.7710000000000008},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.024,"15min":1.286},"min":{"1min":0.376,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.9670000000000001,"5min":5.6120000000000001,"15min":6.7599999999999998},"last":0.627}]},{"osd":8,"last update":"Mon Feb 12 16:40:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.71199999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:40:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.153,"15min":1.3},"min":{"1min":0.45300000000000001,"5min":0.442,"15min":0.41899999999999998},"max":{"1min":1.986,"5min":3.8730000000000002,"15min":6.9509999999999996},"last":0.752},{"interface":"front","average":{"1min":1.135,"5min":1.1539999999999999,"15min":1.306},"min":{"1min":0.442,"5min":0.42599999999999999,"15min":0.375},"max":{"1min":2.0510000000000002,"5min":6.3390000000000004,"15min":9.6649999999999991},"last":0.60899999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:39:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.2430000000000001,"15min":1.484},"min":{"1min":0.66700000000000004,"5min":0.48599999999999999,"15min":0.41099999999999998},"max":{"1min":1.9259999999999999,"5min":5.681,"15min":55.518999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.1870000000000001,"15min":1.554},"min":{"1min":1.117,"5min":0.49199999999999999,"15min":0.47999999999999998},"max":{"1min":2.1850000000000001,"5min":4.5069999999999997,"15min":55.497999999999998},"last":0.81599999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:40:07 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.2689999999999999,"15min":1.377},"min":{"1min":1.0860000000000001,"5min":0.52000000000000002,"15min":0.38300000000000001},"max":{"1min":1.9410000000000001,"5min":5.7329999999999997,"15min":8.6940000000000008},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.611,"5min":1.181,"15min":1.397},"min":{"1min":1.2689999999999999,"5min":0.52500000000000002,"15min":0.51500000000000001},"max":{"1min":2.085,"5min":3.54,"15min":6.9029999999999996},"last":0.78800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":557973504,"data_stored":557973504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1117847552,"data_stored":1117847552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":937623552,"data_stored":937623552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":549650432,"data_stored":549650432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":761921536,"data_stored":761921536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1400111104,"data_stored":1400111104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":755040256,"data_stored":755040256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":550371328,"data_stored":550371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1241186304,"data_stored":1241186304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":539623424,"data_stored":539623424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:40:40.630 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T16:40:40.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set unique_pool_6 pg_num 26 2024-02-12T16:40:40.632 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:40.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:40.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:40.747 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:40 smithi169 bash[19604]: cluster 2024-02-12T16:40:38.769203+0000 osd.5 (osd.5) 193 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:40.747 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:40 smithi169 bash[19604]: cluster 2024-02-12T16:40:38.769252+0000 osd.5 (osd.5) 194 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:40.747 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:40 smithi169 bash[19604]: cluster 2024-02-12T16:40:39.299845+0000 mon.a (mon.0) 2997 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:40.747 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:40 smithi169 bash[19604]: audit 2024-02-12T16:40:39.312053+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.121:0/2139733705' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:40:40.747 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:40 smithi169 bash[19604]: cluster 2024-02-12T16:40:39.312095+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e877: 12 total, 11 up, 11 in 2024-02-12T16:40:40.765 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:40.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:40.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:40.896 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:40.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:41.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:41.033 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:41.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:41.170 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:41.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:41.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 76829 76813 55.1743 126.5 0.00384862 0.0181221 2024-02-12T16:40:41.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:41.428 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:41.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:41.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:41 smithi123 bash[19119]: cluster 2024-02-12T16:40:40.220607+0000 mgr.y (mgr.14146) 1690 : cluster [DBG] pgmap v2438: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 118 MiB/s wr, 1.88k op/s; 4655/139248 objects degraded (3.343%); 4676/139248 objects misplaced (3.358%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:40:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:41 smithi123 bash[19119]: cluster 2024-02-12T16:40:40.907202+0000 mon.a (mon.0) 3000 : cluster [WRN] Health check update: Degraded data redundancy: 4655/139248 objects degraded (3.343%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:41 smithi123 bash[19119]: cluster 2024-02-12T16:40:40.914327+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e878: 12 total, 11 up, 11 in 2024-02-12T16:40:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:41 smithi121 bash[18489]: cluster 2024-02-12T16:40:40.220607+0000 mgr.y (mgr.14146) 1690 : cluster [DBG] pgmap v2438: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 118 MiB/s wr, 1.88k op/s; 4655/139248 objects degraded (3.343%); 4676/139248 objects misplaced (3.358%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:40:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:41 smithi121 bash[18489]: cluster 2024-02-12T16:40:40.907202+0000 mon.a (mon.0) 3000 : cluster [WRN] Health check update: Degraded data redundancy: 4655/139248 objects degraded (3.343%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:41 smithi121 bash[18489]: cluster 2024-02-12T16:40:40.914327+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e878: 12 total, 11 up, 11 in 2024-02-12T16:40:41.663 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:41.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:41 smithi169 bash[19604]: cluster 2024-02-12T16:40:40.220607+0000 mgr.y (mgr.14146) 1690 : cluster [DBG] pgmap v2438: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 118 MiB/s wr, 1.88k op/s; 4655/139248 objects degraded (3.343%); 4676/139248 objects misplaced (3.358%); 1.2 MiB/s, 18 objects/s recovering 2024-02-12T16:40:41.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:41 smithi169 bash[19604]: cluster 2024-02-12T16:40:40.907202+0000 mon.a (mon.0) 3000 : cluster [WRN] Health check update: Degraded data redundancy: 4655/139248 objects degraded (3.343%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:41.671 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:41 smithi169 bash[19604]: cluster 2024-02-12T16:40:40.914327+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e878: 12 total, 11 up, 11 in 2024-02-12T16:40:41.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:41.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:41.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:41.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:41.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:41.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:42.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:42.082 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:42.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:42.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:42.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:42.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:42.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 78817 78801 55.9591 124.25 0.00888228 0.0178683 2024-02-12T16:40:42.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:42.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:42.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:42.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:42.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:40:42.517 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:42.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:42.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:42.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:42.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:42.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:40:42.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:42.812 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:42.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:42.908 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:40:42.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:42.947 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:42.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:43.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:43.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:43.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:43.224 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:43.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:43.315 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 80723 80707 56.6687 119.125 0.00246422 0.0176445 2024-02-12T16:40:43.321 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:43 smithi121 bash[18489]: cluster 2024-02-12T16:40:42.221157+0000 mgr.y (mgr.14146) 1691 : cluster [DBG] pgmap v2440: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s; 4655/150318 objects degraded (3.097%); 4676/150318 objects misplaced (3.111%); 677 KiB/s, 10 objects/s recovering 2024-02-12T16:40:43.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:43.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:43.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:43.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:43.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:43.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:43.613 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:43 smithi169 bash[19604]: cluster 2024-02-12T16:40:42.221157+0000 mgr.y (mgr.14146) 1691 : cluster [DBG] pgmap v2440: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s; 4655/150318 objects degraded (3.097%); 4676/150318 objects misplaced (3.111%); 677 KiB/s, 10 objects/s recovering 2024-02-12T16:40:43.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:43.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:43 smithi123 bash[19119]: cluster 2024-02-12T16:40:42.221157+0000 mgr.y (mgr.14146) 1691 : cluster [DBG] pgmap v2440: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s; 4655/150318 objects degraded (3.097%); 4676/150318 objects misplaced (3.111%); 677 KiB/s, 10 objects/s recovering 2024-02-12T16:40:43.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.770 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:43.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:43.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.901 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:43.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:44.007 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:44.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:44.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:44.166 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:44.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:44.298 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.004294 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 82625 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 57.3757 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 86.9715 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 343.625 2024-02-12T16:40:44.316 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 0 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 918 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 1391 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 5498 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 0 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.017428 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.835227 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 60.0841 2024-02-12T16:40:44.317 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.00106883 2024-02-12T16:40:44.320 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:40:44.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:44.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:44.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:44.562 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:44.665 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:44.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:44.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:44.827 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:44.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:44.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:44.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:45.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:45.093 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:45.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:45.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:45.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:45.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:45.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:45.332 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:45 smithi123 bash[19119]: cluster 2024-02-12T16:40:44.221649+0000 mgr.y (mgr.14146) 1692 : cluster [DBG] pgmap v2441: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s; 4655/157700 objects degraded (2.952%); 5289/157700 objects misplaced (3.354%); 597 KiB/s, 9 objects/s recovering 2024-02-12T16:40:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:45.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:45.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:45.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:45 smithi169 bash[19604]: cluster 2024-02-12T16:40:44.221649+0000 mgr.y (mgr.14146) 1692 : cluster [DBG] pgmap v2441: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s; 4655/157700 objects degraded (2.952%); 5289/157700 objects misplaced (3.354%); 597 KiB/s, 9 objects/s recovering 2024-02-12T16:40:45.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:45.615 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:45.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:45.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:45 smithi121 bash[18489]: cluster 2024-02-12T16:40:44.221649+0000 mgr.y (mgr.14146) 1692 : cluster [DBG] pgmap v2441: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s; 4655/157700 objects degraded (2.952%); 5289/157700 objects misplaced (3.354%); 597 KiB/s, 9 objects/s recovering 2024-02-12T16:40:45.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:45.751 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:45.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:45.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:45.884 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:45.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:45.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:46.016 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:46.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:46.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:46.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:46.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:46.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:46 smithi121 bash[18489]: cluster 2024-02-12T16:40:45.908567+0000 mon.a (mon.0) 3002 : cluster [WRN] Health check update: Degraded data redundancy: 4655/157700 objects degraded (2.952%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:46.430 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:46.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:46.482 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:46.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:46.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:46.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:46 smithi123 bash[19119]: cluster 2024-02-12T16:40:45.908567+0000 mon.a (mon.0) 3002 : cluster [WRN] Health check update: Degraded data redundancy: 4655/157700 objects degraded (2.952%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:46.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:46.676 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:46 smithi169 bash[19604]: cluster 2024-02-12T16:40:45.908567+0000 mon.a (mon.0) 3002 : cluster [WRN] Health check update: Degraded data redundancy: 4655/157700 objects degraded (2.952%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:46.705 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:46.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:46.847 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:46.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:46.973 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:40:46.973 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:40:46.990 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:46.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:47.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:47.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:47.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:47.248 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:47.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:47.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:47.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:47 smithi121 bash[18489]: cluster 2024-02-12T16:40:46.222226+0000 mgr.y (mgr.14146) 1693 : cluster [DBG] pgmap v2442: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 129 B/s rd, 113 MiB/s wr, 1.82k op/s; 5064/158518 objects degraded (3.195%); 5231/158518 objects misplaced (3.300%); 921 KiB/s, 14 objects/s recovering 2024-02-12T16:40:47.391 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:47.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:47.490 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:40:47.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:47.538 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:47.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:47 smithi169 bash[19604]: cluster 2024-02-12T16:40:46.222226+0000 mgr.y (mgr.14146) 1693 : cluster [DBG] pgmap v2442: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 129 B/s rd, 113 MiB/s wr, 1.82k op/s; 5064/158518 objects degraded (3.195%); 5231/158518 objects misplaced (3.300%); 921 KiB/s, 14 objects/s recovering 2024-02-12T16:40:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:47 smithi123 bash[19119]: cluster 2024-02-12T16:40:46.222226+0000 mgr.y (mgr.14146) 1693 : cluster [DBG] pgmap v2442: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 129 B/s rd, 113 MiB/s wr, 1.82k op/s; 5064/158518 objects degraded (3.195%); 5231/158518 objects misplaced (3.300%); 921 KiB/s, 14 objects/s recovering 2024-02-12T16:40:47.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:47.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:47.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:47.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:40:47.773 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:47.799 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:47.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:47.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:47.934 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:47.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:48.038 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:48.062 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:48.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:48.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:48.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:48.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:48.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:48.322 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:48.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:48.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:48.449 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:48.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:48.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:48.585 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:48.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:48.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:48.714 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:48.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:48.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:48.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:48.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:48.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:48.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:49.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:49.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:49.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:49.265 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:49.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:49.395 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:49.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:49.506 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:49 smithi169 bash[19604]: cluster 2024-02-12T16:40:48.222859+0000 mgr.y (mgr.14146) 1694 : cluster [DBG] pgmap v2443: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 114 B/s rd, 93 MiB/s wr, 1.78k op/s; 5064/159294 objects degraded (3.179%); 5231/159294 objects misplaced (3.284%); 819 KiB/s, 12 objects/s recovering 2024-02-12T16:40:49.537 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:49.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:49 smithi123 bash[19119]: cluster 2024-02-12T16:40:48.222859+0000 mgr.y (mgr.14146) 1694 : cluster [DBG] pgmap v2443: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 114 B/s rd, 93 MiB/s wr, 1.78k op/s; 5064/159294 objects degraded (3.179%); 5231/159294 objects misplaced (3.284%); 819 KiB/s, 12 objects/s recovering 2024-02-12T16:40:49.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:49 smithi121 bash[18489]: cluster 2024-02-12T16:40:48.222859+0000 mgr.y (mgr.14146) 1694 : cluster [DBG] pgmap v2443: 47 pgs: 4 active+clean+remapped, 6 undersized+peered, 1 active+recovering+degraded+remapped, 36 active+clean; 4.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 114 B/s rd, 93 MiB/s wr, 1.78k op/s; 5064/159294 objects degraded (3.179%); 5231/159294 objects misplaced (3.284%); 819 KiB/s, 12 objects/s recovering 2024-02-12T16:40:49.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:49.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:49.800 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:49.903 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:49.926 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:49.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:50.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:50.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:50.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:50.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:50.183 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:50.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:50.286 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:50.312 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:50.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:50.336 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:50 smithi123 bash[19119]: cluster 2024-02-12T16:40:49.412081+0000 osd.1 (osd.1) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:50.336 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:50 smithi123 bash[19119]: cluster 2024-02-12T16:40:49.412135+0000 osd.1 (osd.1) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:50.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:50.422 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:50 smithi169 bash[19604]: cluster 2024-02-12T16:40:49.412081+0000 osd.1 (osd.1) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:50.423 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:50 smithi169 bash[19604]: cluster 2024-02-12T16:40:49.412135+0000 osd.1 (osd.1) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:50.441 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:50.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:50.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:50 smithi121 bash[18489]: cluster 2024-02-12T16:40:49.412081+0000 osd.1 (osd.1) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:40:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:50 smithi121 bash[18489]: cluster 2024-02-12T16:40:49.412135+0000 osd.1 (osd.1) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:40:51.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:51.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:51.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:51.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:51.518 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:51.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:51.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:51.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:51 smithi123 bash[19119]: cluster 2024-02-12T16:40:50.223344+0000 mgr.y (mgr.14146) 1695 : cluster [DBG] pgmap v2444: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 102 B/s rd, 81 MiB/s wr, 1.96k op/s; 9045/152118 objects misplaced (5.946%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:51.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:51 smithi123 bash[19119]: cluster 2024-02-12T16:40:50.392251+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e879: 12 total, 11 up, 11 in 2024-02-12T16:40:51.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:51 smithi123 bash[19119]: cluster 2024-02-12T16:40:50.909717+0000 mon.a (mon.0) 3004 : cluster [WRN] Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:51.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:51 smithi123 bash[19119]: cluster 2024-02-12T16:40:50.917877+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e880: 12 total, 11 up, 11 in 2024-02-12T16:40:51.651 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:51.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:51 smithi121 bash[18489]: cluster 2024-02-12T16:40:50.223344+0000 mgr.y (mgr.14146) 1695 : cluster [DBG] pgmap v2444: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 102 B/s rd, 81 MiB/s wr, 1.96k op/s; 9045/152118 objects misplaced (5.946%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:51 smithi121 bash[18489]: cluster 2024-02-12T16:40:50.392251+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e879: 12 total, 11 up, 11 in 2024-02-12T16:40:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:51 smithi121 bash[18489]: cluster 2024-02-12T16:40:50.909717+0000 mon.a (mon.0) 3004 : cluster [WRN] Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:51 smithi121 bash[18489]: cluster 2024-02-12T16:40:50.917877+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e880: 12 total, 11 up, 11 in 2024-02-12T16:40:51.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:51.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:51.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:51 smithi169 bash[19604]: cluster 2024-02-12T16:40:50.223344+0000 mgr.y (mgr.14146) 1695 : cluster [DBG] pgmap v2444: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 102 B/s rd, 81 MiB/s wr, 1.96k op/s; 9045/152118 objects misplaced (5.946%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:51 smithi169 bash[19604]: cluster 2024-02-12T16:40:50.392251+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e879: 12 total, 11 up, 11 in 2024-02-12T16:40:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:51 smithi169 bash[19604]: cluster 2024-02-12T16:40:50.909717+0000 mon.a (mon.0) 3004 : cluster [WRN] Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:40:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:51 smithi169 bash[19604]: cluster 2024-02-12T16:40:50.917877+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e880: 12 total, 11 up, 11 in 2024-02-12T16:40:51.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:51.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:51.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:52.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:52.059 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:52.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:52.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:52.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:52.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:52.302 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:52.338 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:52.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:52.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:52.467 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:52.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:52.513 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:40:52.572 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:52.606 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:52.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:52.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:52.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:52.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:52.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:52.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:52.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:53.010 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:53.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:53.151 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:53.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:53.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:53.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:53.412 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:53.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:53.527 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:53 smithi121 bash[18489]: cluster 2024-02-12T16:40:52.223927+0000 mgr.y (mgr.14146) 1696 : cluster [DBG] pgmap v2447: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 1.82k op/s; 8941/143712 objects misplaced (6.221%); 2.3 MiB/s, 37 objects/s recovering 2024-02-12T16:40:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:53 smithi169 bash[19604]: cluster 2024-02-12T16:40:52.223927+0000 mgr.y (mgr.14146) 1696 : cluster [DBG] pgmap v2447: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 1.82k op/s; 8941/143712 objects misplaced (6.221%); 2.3 MiB/s, 37 objects/s recovering 2024-02-12T16:40:53.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:53 smithi123 bash[19119]: cluster 2024-02-12T16:40:52.223927+0000 mgr.y (mgr.14146) 1696 : cluster [DBG] pgmap v2447: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 127 B/s rd, 29 MiB/s wr, 1.82k op/s; 8941/143712 objects misplaced (6.221%); 2.3 MiB/s, 37 objects/s recovering 2024-02-12T16:40:53.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:53.676 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:53.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:53.805 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:53.911 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:53.936 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:53.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:54.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:54.065 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:54.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:54.197 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:54.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:54.325 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:54.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:54.456 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:54.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:54.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:54.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:54.716 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:54.819 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:54.845 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:54.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:54.974 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:54.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:55.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:55.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:55.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:55.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:55.234 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:55.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:55.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:55.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:55.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:55.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:55.480 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:55 smithi169 bash[19604]: cluster 2024-02-12T16:40:54.224703+0000 mgr.y (mgr.14146) 1697 : cluster [DBG] pgmap v2448: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 26 MiB/s wr, 2.06k op/s; 8538/138668 objects misplaced (6.157%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:55.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:55.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:55.508 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:40:55.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:40:55.598 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:40:55.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:55 smithi123 bash[19119]: cluster 2024-02-12T16:40:54.224703+0000 mgr.y (mgr.14146) 1697 : cluster [DBG] pgmap v2448: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 26 MiB/s wr, 2.06k op/s; 8538/138668 objects misplaced (6.157%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:55.639 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:55.640 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:40:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:55.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:55.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:55 smithi121 bash[18489]: cluster 2024-02-12T16:40:54.224703+0000 mgr.y (mgr.14146) 1697 : cluster [DBG] pgmap v2448: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 26 MiB/s wr, 2.06k op/s; 8538/138668 objects misplaced (6.157%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:55.673 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:40:55.707 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:40:55.738 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:40:55.745 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:55.770 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:55.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:55.775 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:40:55.800 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:40:55.827 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:40:55.857 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.858 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:40:55.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:55.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:55.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:56.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:56.048 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:56.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:56.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:56.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:56.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:56.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:56.315 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:56.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:56.351 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 8 pg_num to 26 2024-02-12T16:40:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:40:56.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:56.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:56 smithi121 bash[18489]: audit 2024-02-12T16:40:55.462963+0000 mon.c (mon.1) 106 : audit [DBG] from='client.? v1:172.21.15.121:0/2082385872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:56.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:56 smithi121 bash[18489]: audit 2024-02-12T16:40:55.766481+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? v1:172.21.15.121:0/3258639381' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:40:56.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:56 smithi121 bash[18489]: audit 2024-02-12T16:40:55.769782+0000 mon.a (mon.0) 3007 : audit [INF] from='client.26939 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:40:56.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:56 smithi121 bash[18489]: audit 2024-02-12T16:40:55.771956+0000 mon.b (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.121:0/1807442995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:40:56.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:40:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:56.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:40:56.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26257.0:179495 6.3 e679/636 6:c520ae98:::benchmark_data_smithi169_96224_object26659:head v 679'11149, mlcod=679'11146)", 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:31:28.752699+0000", 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 567.84694840600002, 2024-02-12T16:40:56.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23754165399999999, 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752699+0000", 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752699+0000", 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752698+0000", 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4199999999999996e-06 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752707+0000", 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6110999999999999e-05 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752723+0000", 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5499999999999997e-06 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752731+0000", 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023781200000000001 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752968+0000", 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6721999999999999e-05 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.752995+0000", 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23724506000000001 2024-02-12T16:40:56.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:31:28.990240+0000", 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.23754165399999999 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:263668 6.13 6:cda3866b:::benchmark_data_smithi169_96224_object110832:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:48.094731+0000", 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 488.50491609400001, 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.482651414, 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.609 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 263668 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094731+0000", 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094731+0000", 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094729+0000", 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8700000000000003e-06 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094736+0000", 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1467999999999999e-05 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094747+0000", 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5339999999999996e-06 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094754+0000", 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8145000000000001e-05 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094802+0000", 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014538100000000001 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.094947+0000", 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00055281199999999996 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.095500+0000", 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002824423 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.098324+0000", 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076364900000000003 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.099088+0000", 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2613000000000001e-05 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.099121+0000", 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47808740199999999 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.577208+0000", 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4317999999999997e-05 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.577262+0000", 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119839 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:48.577382+0000", 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.482651414 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26257.0:264237 6.13 6:cc27572d:::benchmark_data_smithi169_96224_object111401:head [delete] snapc 0=[] ondisk+write+known_if_redirected e714)", 2024-02-12T16:40:56.615 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:32:49.092307+0000", 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 487.50734004700001, 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35619952399999999, 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26257", 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 264237 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092307+0000", 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092307+0000", 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092305+0000", 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.781e-06 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092312+0000", 2024-02-12T16:40:56.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1891e-05 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092324+0000", 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.22e-06 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092329+0000", 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9597e-05 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092379+0000", 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109811 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092488+0000", 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050793500000000005 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,7,11", 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.092996+0000", 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00232073 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.095317+0000", 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00052685299999999998 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.095844+0000", 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014834920000000001 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.097327+0000", 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35100952099999999 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.448337+0000", 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3976e-05 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.448391+0000", 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011561300000000001 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:32:49.448506+0000", 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.35619952399999999 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:32530 7.c 7:381b837d:::benchmark_data_smithi169_105252_object32529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:34:30.495547+0000", 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 386.10409963299998, 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47412637499999999, 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32530 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495547+0000", 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495547+0000", 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495546+0000", 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7082999999999995e-05 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495613+0000", 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6919999999999996e-06 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495620+0000", 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0580000000000002e-06 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495623+0000", 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4523000000000001e-05 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495658+0000", 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011336900000000001 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495771+0000", 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015399500000000001 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.495925+0000", 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001924638 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.497850+0000", 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47156733200000001 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.969417+0000", 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013156999999999999 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.969549+0000", 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125091 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.969674+0000", 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47412637499999999 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:32539 7.c 7:35dfeca2:::benchmark_data_smithi169_105252_object32538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:34:30.498163+0000", 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 386.10148366099997, 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47178509800000001, 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32539 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498163+0000", 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498163+0000", 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498162+0000", 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5704999999999993e-05 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498247+0000", 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3978999999999999e-05 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498261+0000", 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9567000000000001e-05 2024-02-12T16:40:56.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498281+0000", 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6601999999999999e-05 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498328+0000", 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133244 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498461+0000", 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013966200000000001 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.498600+0000", 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012310419999999999 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.499832+0000", 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46996233599999998 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.969794+0000", 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6411999999999999e-05 2024-02-12T16:40:56.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.969880+0000", 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8149000000000001e-05 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.969948+0000", 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47178509800000001 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.632 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:32571 7.c 7:32fb11a2:::benchmark_data_smithi169_105252_object32570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:34:30.508151+0000", 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 386.09149598300002, 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46486913899999999, 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32571 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508151+0000", 2024-02-12T16:40:56.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508151+0000", 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508149+0000", 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9498000000000004e-05 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508219+0000", 2024-02-12T16:40:56.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0686000000000001e-05 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508230+0000", 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5891000000000001e-05 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508245+0000", 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0731e-05 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508286+0000", 2024-02-12T16:40:56.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6896000000000002e-05 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508383+0000", 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103402 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.508486+0000", 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001087972 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.509574+0000", 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.463355446 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.972930+0000", 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1322000000000001e-05 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.972981+0000", 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8939000000000001e-05 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.973020+0000", 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46486913899999999 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:32608 7.c 7:3948ffc3:::benchmark_data_smithi169_105252_object32607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:34:30.523691+0000", 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 386.07595611900001, 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.44969811799999998, 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.638 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32608 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523691+0000", 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523691+0000", 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999928 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523684+0000", 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9212000000000001e-05 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523713+0000", 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2243e-05 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523725+0000", 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8119999999999998e-06 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523730+0000", 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7470999999999999e-05 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523788+0000", 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102056 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523890+0000", 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010483499999999999 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.523994+0000", 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287163 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.525282+0000", 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.44784286499999998 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.973125+0000", 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014454 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.973269+0000", 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011992700000000001 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.973389+0000", 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.44969811799999998 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:32613 7.c 7:3e29b717:::benchmark_data_smithi169_105252_object32612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:40:56.643 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:34:30.526220+0000", 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 386.07342739699999, 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.447603101, 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32613 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526220+0000", 2024-02-12T16:40:56.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526220+0000", 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526218+0000", 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3367e-05 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526261+0000", 2024-02-12T16:40:56.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3430000000000002e-06 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526271+0000", 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9990000000000002e-06 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526275+0000", 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1256e-05 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526306+0000", 2024-02-12T16:40:56.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5488000000000005e-05 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526391+0000", 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100431 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.526492+0000", 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070680500000000002 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.527199+0000", 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.446363338 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.973562+0000", 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000130257 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.973692+0000", 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013042800000000001 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.973823+0000", 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.447603101 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:32630 7.c 7:3ca5494a:::benchmark_data_smithi169_105252_object32629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e760)", 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:34:30.541081+0000", 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 386.05856609099999, 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43319382200000001, 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 32630 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541081+0000", 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541081+0000", 2024-02-12T16:40:56.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541080+0000", 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2222000000000003e-05 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541122+0000", 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5640000000000002e-06 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541128+0000", 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0649999999999998e-06 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541133+0000", 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5265999999999997e-05 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541169+0000", 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4135000000000007e-05 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541253+0000", 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010051399999999999 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.541353+0000", 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00097257300000000001 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.542326+0000", 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43169965300000002 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.974025+0000", 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000131851 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.974157+0000", 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117371 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:34:30.974275+0000", 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43319382200000001 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.654 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:174594 7.c 7:3d140a15:::benchmark_data_smithi169_105252_object6000:head [delete] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:35:59.278759+0000", 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 297.320888094, 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.54895232699999996, 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 174594 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278759+0000", 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278759+0000", 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278757+0000", 2024-02-12T16:40:56.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7139999999999999e-06 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278765+0000", 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.224e-05 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278777+0000", 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7839999999999998e-06 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278784+0000", 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7769999999999999e-05 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278822+0000", 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5606999999999996e-05 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278898+0000", 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3904000000000002e-05 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.278982+0000", 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076676219999999998 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.286649+0000", 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085816920000000001 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295231+0000", 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53242219700000004 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827653+0000", 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8183999999999997e-05 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827711+0000", 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.54895232699999996 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.660 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:174597 7.c 7:30ef62d7:::benchmark_data_smithi169_105252_object6003:head [delete] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:35:59.279672+0000", 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 297.319975097, 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.54848531899999997, 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 174597 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.279672+0000", 2024-02-12T16:40:56.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.279672+0000", 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.279671+0000", 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6210000000000001e-06 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.279676+0000", 2024-02-12T16:40:56.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2219999999999996e-06 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.279683+0000", 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.527e-06 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.279688+0000", 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030919599999999999 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.282780+0000", 2024-02-12T16:40:56.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2550000000000002e-05 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.282852+0000", 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1213999999999999e-05 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.282934+0000", 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0054220079999999999 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.288356+0000", 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53956807299999998 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827924+0000", 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124647 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.828048+0000", 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010884500000000001 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.828157+0000", 2024-02-12T16:40:56.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.54848531899999997 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26530.0:174594 7.c e789/752 ondisk, result = 0)", 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:35:59.295162+0000", 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 297.30448528300002, 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.532567075, 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295162+0000", 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295162+0000", 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295160+0000", 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1940000000000003e-06 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295166+0000", 2024-02-12T16:40:56.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2309999999999999e-06 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295173+0000", 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3000000000000003e-06 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295178+0000", 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9823000000000001e-05 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295218+0000", 2024-02-12T16:40:56.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8890000000000002e-06 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295225+0000", 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53250333699999997 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827729+0000", 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.532567075 2024-02-12T16:40:56.669 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.26530.0:174597 7.c e789/752 ondisk, result = 0)", 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:35:59.295188+0000", 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 297.30445897999999, 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53298291499999995, 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295188+0000", 2024-02-12T16:40:56.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295188+0000", 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295187+0000", 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9809999999999997e-06 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295191+0000", 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3979999999999998e-06 2024-02-12T16:40:56.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295197+0000", 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2430000000000001e-06 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.295200+0000", 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53264677199999999 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827847+0000", 2024-02-12T16:40:56.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3801999999999998e-05 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827901+0000", 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000270265 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.828171+0000", 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53298291499999995 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.673 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:174635 7.7 7:e11e63c7:::benchmark_data_smithi169_105252_object6041:head [delete] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:35:59.313355+0000", 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 297.28629246600002, 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53779860899999998, 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 174635 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.313355+0000", 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.313355+0000", 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.313351+0000", 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2087e-05 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.313363+0000", 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.514467377 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827831+0000", 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827843+0000", 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7793999999999997e-05 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.827891+0000", 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016141199999999999 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.828052+0000", 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016303699999999999 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 8", 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.828215+0000", 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022819731999999999 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.851035+0000", 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3171e-05 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.851048+0000", 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1864000000000003e-05 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.851110+0000", 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3028000000000002e-05 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:35:59.851153+0000", 2024-02-12T16:40:56.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.53779860899999998 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:249803 7.c 7:39b98a62:::benchmark_data_smithi169_105252_object81209:head [delete] snapc 0=[] ondisk+write+known_if_redirected e808)", 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:09.622410+0000", 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 226.977237167, 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.200661173, 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.679 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 249803 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622410+0000", 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622410+0000", 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622407+0000", 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0899999999999999e-06 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622415+0000", 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3550000000000001e-05 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622428+0000", 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7450000000000001e-06 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622434+0000", 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9530999999999997e-05 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622483+0000", 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0742000000000006e-05 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622574+0000", 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8228999999999998e-05 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.622662+0000", 2024-02-12T16:40:56.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049362080000000001 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.627599+0000", 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.19533467500000001 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.822933+0000", 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5303000000000002e-05 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.822989+0000", 2024-02-12T16:40:56.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2486999999999998e-05 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:09.823071+0000", 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.200661173 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:296069 7.c 7:3a195bdf:::benchmark_data_smithi169_105252_object127475:head [delete] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:49.867479+0000", 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 186.73216749400001, 2024-02-12T16:40:56.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.331843006, 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 296069 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.867479+0000", 2024-02-12T16:40:56.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.867479+0000", 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.867478+0000", 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0584000000000001e-05 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.867488+0000", 2024-02-12T16:40:56.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0395e-05 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.867499+0000", 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.28e-06 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.867505+0000", 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004929096 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872434+0000", 2024-02-12T16:40:56.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120344 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872554+0000", 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2029999999999998e-05 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872646+0000", 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011970692999999999 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.884617+0000", 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31443031799999999 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.199047+0000", 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013529699999999999 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.199183+0000", 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013990500000000001 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.199323+0000", 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.331843006 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:296072 7.c 7:3d34fc30:::benchmark_data_smithi169_105252_object127478:head [delete] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:49.872941+0000", 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 186.72670639699999, 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34663622100000002, 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.690 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 296072 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872941+0000", 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872941+0000", 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872938+0000", 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7540000000000006e-06 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872946+0000", 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3376e-05 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872960+0000", 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6699999999999997e-06 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.872966+0000", 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038978760000000002 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.876864+0000", 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101879 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.876966+0000", 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011272 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.877079+0000", 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075413859999999997 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.884620+0000", 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.33490329200000002 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219523+0000", 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9366999999999999e-05 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219553+0000", 2024-02-12T16:40:56.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4045000000000001e-05 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219577+0000", 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34663622100000002 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:296073 7.c 7:3ca24106:::benchmark_data_smithi169_105252_object127479:head [delete] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:49.873795+0000", 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 186.72585163400001, 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.345861749, 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.695 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 296073 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.873795+0000", 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.873795+0000", 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.873794+0000", 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1729999999999995e-06 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.873802+0000", 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1829e-05 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.873814+0000", 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2320000000000001e-06 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.873819+0000", 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0073785420000000001 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.881198+0000", 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7843999999999995e-05 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.881276+0000", 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0552999999999999e-05 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.881356+0000", 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033976359999999999 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.884754+0000", 2024-02-12T16:40:56.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.33485849099999998 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219612+0000", 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3e-05 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219635+0000", 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1849999999999999e-05 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219657+0000", 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.345861749 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:296097 7.c 7:365f4911:::benchmark_data_smithi169_105252_object127503:head [delete] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:49.893671+0000", 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 186.705976214, 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32605700900000001, 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 296097 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893671+0000", 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893671+0000", 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:40:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893669+0000", 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5460000000000003e-06 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893677+0000", 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0304e-05 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893687+0000", 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5129999999999998e-06 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893692+0000", 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7778000000000003e-05 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893730+0000", 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0123000000000005e-05 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893810+0000", 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2933e-05 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.893893+0000", 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039710359999999998 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.897864+0000", 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32181891299999998 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219683+0000", 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4079000000000001e-05 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219707+0000", 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1138e-05 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.219728+0000", 2024-02-12T16:40:56.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32605700900000001 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:296102 7.c 7:3afc18e3:::benchmark_data_smithi169_105252_object127508:head [delete] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:49.900574+0000", 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 186.69907317799999, 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32498954299999999, 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:40:56.707 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 296102 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900574+0000", 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900574+0000", 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900572+0000", 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4760000000000003e-06 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900578+0000", 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0723000000000001e-05 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900589+0000", 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8829999999999997e-06 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900594+0000", 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0951999999999997e-05 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900635+0000", 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125728 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900761+0000", 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013913700000000001 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.900900+0000", 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038646499999999999 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:49.904765+0000", 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32069467899999998 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.225459+0000", 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6759000000000003e-05 2024-02-12T16:40:56.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.225516+0000", 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7367000000000002e-05 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:50.225563+0000", 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32498954299999999 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:40:56.712 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:40:56.713 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:40:56.713 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:56 smithi123 bash[19119]: audit 2024-02-12T16:40:55.462963+0000 mon.c (mon.1) 106 : audit [DBG] from='client.? v1:172.21.15.121:0/2082385872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:56.713 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:56 smithi123 bash[19119]: audit 2024-02-12T16:40:55.766481+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? v1:172.21.15.121:0/3258639381' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:40:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:56 smithi123 bash[19119]: audit 2024-02-12T16:40:55.769782+0000 mon.a (mon.0) 3007 : audit [INF] from='client.26939 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:40:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:56 smithi123 bash[19119]: audit 2024-02-12T16:40:55.771956+0000 mon.b (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.121:0/1807442995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:40:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:56.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:56.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:56 smithi169 bash[19604]: audit 2024-02-12T16:40:55.462963+0000 mon.c (mon.1) 106 : audit [DBG] from='client.? v1:172.21.15.121:0/2082385872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:40:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:56 smithi169 bash[19604]: audit 2024-02-12T16:40:55.766481+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? v1:172.21.15.121:0/3258639381' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:40:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:56 smithi169 bash[19604]: audit 2024-02-12T16:40:55.769782+0000 mon.a (mon.0) 3007 : audit [INF] from='client.26939 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:40:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:56 smithi169 bash[19604]: audit 2024-02-12T16:40:55.771956+0000 mon.b (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.121:0/1807442995' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:40:56.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:56.846 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:56.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:56.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:56.987 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:56.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:57.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:57.127 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:40:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:57.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:40:57.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:40:57.276 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:40:57.281 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:57.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:57.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: cluster 2024-02-12T16:40:56.227497+0000 mgr.y (mgr.14146) 1698 : cluster [DBG] pgmap v2449: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 MiB/s wr, 1.37k op/s; 8538/138668 objects misplaced (6.157%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:57.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: cluster 2024-02-12T16:40:56.337141+0000 mon.a (mon.0) 3008 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:57.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: audit 2024-02-12T16:40:56.342951+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? v1:172.21.15.121:0/3258639381' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:40:57.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: audit 2024-02-12T16:40:56.343005+0000 mon.a (mon.0) 3010 : audit [INF] from='client.26939 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:40:57.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: cluster 2024-02-12T16:40:56.343073+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e881: 12 total, 11 up, 11 in 2024-02-12T16:40:57.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: cluster 2024-02-12T16:40:56.439206+0000 osd.0 (osd.0) 151 : cluster [DBG] 8.e scrub starts 2024-02-12T16:40:57.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: cluster 2024-02-12T16:40:56.470577+0000 osd.6 (osd.6) 176 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:40:57.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:57 smithi121 bash[18489]: cluster 2024-02-12T16:40:56.518195+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:40:57.388 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:57.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-12T16:40:57.418 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:57.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:57.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:57.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:57.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: cluster 2024-02-12T16:40:56.227497+0000 mgr.y (mgr.14146) 1698 : cluster [DBG] pgmap v2449: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 MiB/s wr, 1.37k op/s; 8538/138668 objects misplaced (6.157%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: cluster 2024-02-12T16:40:56.337141+0000 mon.a (mon.0) 3008 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: audit 2024-02-12T16:40:56.342951+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? v1:172.21.15.121:0/3258639381' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:40:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: audit 2024-02-12T16:40:56.343005+0000 mon.a (mon.0) 3010 : audit [INF] from='client.26939 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:40:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: cluster 2024-02-12T16:40:56.343073+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e881: 12 total, 11 up, 11 in 2024-02-12T16:40:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: cluster 2024-02-12T16:40:56.439206+0000 osd.0 (osd.0) 151 : cluster [DBG] 8.e scrub starts 2024-02-12T16:40:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: cluster 2024-02-12T16:40:56.470577+0000 osd.6 (osd.6) 176 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:40:57.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:57 smithi123 bash[19119]: cluster 2024-02-12T16:40:56.518195+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:40:57.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:57.662 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: cluster 2024-02-12T16:40:56.227497+0000 mgr.y (mgr.14146) 1698 : cluster [DBG] pgmap v2449: 47 pgs: 5 active+clean+remapped, 6 undersized+peered, 36 active+clean; 4.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 2.7 MiB/s wr, 1.37k op/s; 8538/138668 objects misplaced (6.157%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:40:57.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: cluster 2024-02-12T16:40:56.337141+0000 mon.a (mon.0) 3008 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:40:57.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: audit 2024-02-12T16:40:56.342951+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? v1:172.21.15.121:0/3258639381' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:40:57.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: audit 2024-02-12T16:40:56.343005+0000 mon.a (mon.0) 3010 : audit [INF] from='client.26939 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:40:57.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: cluster 2024-02-12T16:40:56.343073+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e881: 12 total, 11 up, 11 in 2024-02-12T16:40:57.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: cluster 2024-02-12T16:40:56.439206+0000 osd.0 (osd.0) 151 : cluster [DBG] 8.e scrub starts 2024-02-12T16:40:57.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: cluster 2024-02-12T16:40:56.470577+0000 osd.6 (osd.6) 176 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:40:57.663 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:57 smithi169 bash[19604]: cluster 2024-02-12T16:40:56.518195+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:40:57.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:57.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:57.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:57.826 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:40:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:57.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:40:57.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:57.959 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:57.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:58.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:58.089 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:58.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:58.192 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:58.217 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:58.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:58.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:58.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:58.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:58.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:58.526 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:40:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:58.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:40:58.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:58 smithi123 bash[19119]: cluster 2024-02-12T16:40:57.597256+0000 osd.0 (osd.0) 152 : cluster [DBG] 8.e scrub ok 2024-02-12T16:40:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:58 smithi123 bash[19119]: cluster 2024-02-12T16:40:57.754131+0000 osd.3 (osd.3) 100 : cluster [DBG] 8.0 scrub starts 2024-02-12T16:40:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:58 smithi123 bash[19119]: audit 2024-02-12T16:40:58.229026+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:40:58.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:58.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:58.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:58.807 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:58.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:58.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:58 smithi121 bash[18489]: cluster 2024-02-12T16:40:57.597256+0000 osd.0 (osd.0) 152 : cluster [DBG] 8.e scrub ok 2024-02-12T16:40:58.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:58 smithi121 bash[18489]: cluster 2024-02-12T16:40:57.754131+0000 osd.3 (osd.3) 100 : cluster [DBG] 8.0 scrub starts 2024-02-12T16:40:58.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:58 smithi121 bash[18489]: audit 2024-02-12T16:40:58.229026+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:40:58.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:58.947 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:40:58.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:58.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:40:58.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:58 smithi169 bash[19604]: cluster 2024-02-12T16:40:57.597256+0000 osd.0 (osd.0) 152 : cluster [DBG] 8.e scrub ok 2024-02-12T16:40:58.992 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:58 smithi169 bash[19604]: cluster 2024-02-12T16:40:57.754131+0000 osd.3 (osd.3) 100 : cluster [DBG] 8.0 scrub starts 2024-02-12T16:40:58.993 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:58 smithi169 bash[19604]: audit 2024-02-12T16:40:58.229026+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:40:59.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:59.079 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:59.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:59.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:59.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:40:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:59.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:40:59.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:40:59.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:59.361 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:40:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:59.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:40:59.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:59.492 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:40:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:59.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:40:59.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:59.625 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:40:59.626 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:59 smithi169 bash[19604]: cluster 2024-02-12T16:40:58.170321+0000 osd.9 (osd.9) 153 : cluster [DBG] 8.c scrub starts 2024-02-12T16:40:59.626 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:59 smithi169 bash[19604]: cluster 2024-02-12T16:40:58.228107+0000 mgr.y (mgr.14146) 1699 : cluster [DBG] pgmap v2451: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 6 undersized+peered, 35 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.53k op/s; 7972/127916 objects misplaced (6.232%) 2024-02-12T16:40:59.626 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:59 smithi169 bash[19604]: audit 2024-02-12T16:40:58.358362+0000 mon.a (mon.0) 3013 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:40:59.626 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:59 smithi169 bash[19604]: cluster 2024-02-12T16:40:58.358448+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e882: 12 total, 11 up, 11 in 2024-02-12T16:40:59.627 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:40:59 smithi169 bash[19604]: cluster 2024-02-12T16:40:59.015791+0000 osd.9 (osd.9) 154 : cluster [DBG] 8.c scrub ok 2024-02-12T16:40:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:59.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:40:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:59 smithi123 bash[19119]: cluster 2024-02-12T16:40:58.170321+0000 osd.9 (osd.9) 153 : cluster [DBG] 8.c scrub starts 2024-02-12T16:40:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:59 smithi123 bash[19119]: cluster 2024-02-12T16:40:58.228107+0000 mgr.y (mgr.14146) 1699 : cluster [DBG] pgmap v2451: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 6 undersized+peered, 35 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.53k op/s; 7972/127916 objects misplaced (6.232%) 2024-02-12T16:40:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:59 smithi123 bash[19119]: audit 2024-02-12T16:40:58.358362+0000 mon.a (mon.0) 3013 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:40:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:59 smithi123 bash[19119]: cluster 2024-02-12T16:40:58.358448+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e882: 12 total, 11 up, 11 in 2024-02-12T16:40:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:40:59 smithi123 bash[19119]: cluster 2024-02-12T16:40:59.015791+0000 osd.9 (osd.9) 154 : cluster [DBG] 8.c scrub ok 2024-02-12T16:40:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:59 smithi121 bash[18489]: cluster 2024-02-12T16:40:58.170321+0000 osd.9 (osd.9) 153 : cluster [DBG] 8.c scrub starts 2024-02-12T16:40:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:59 smithi121 bash[18489]: cluster 2024-02-12T16:40:58.228107+0000 mgr.y (mgr.14146) 1699 : cluster [DBG] pgmap v2451: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 6 undersized+peered, 35 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.53k op/s; 7972/127916 objects misplaced (6.232%) 2024-02-12T16:40:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:59 smithi121 bash[18489]: audit 2024-02-12T16:40:58.358362+0000 mon.a (mon.0) 3013 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:40:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:59 smithi121 bash[18489]: cluster 2024-02-12T16:40:58.358448+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e882: 12 total, 11 up, 11 in 2024-02-12T16:40:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:40:59 smithi121 bash[18489]: cluster 2024-02-12T16:40:59.015791+0000 osd.9 (osd.9) 154 : cluster [DBG] 8.c scrub ok 2024-02-12T16:40:59.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:59.754 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:40:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:59.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:40:59.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:59.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:40:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:40:59.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:40:59.994 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:00.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:00.135 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:00.173 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:00.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:00.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:00.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:00.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:00.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:00.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:58.529284+0000 osd.10 (osd.10) 202 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:41:00.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:58.581378+0000 osd.10 (osd.10) 203 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:41:00.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:58.990181+0000 osd.2 (osd.2) 127 : cluster [DBG] 8.d scrub starts 2024-02-12T16:41:00.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:00 smithi169 bash[19604]: cluster 2024-02-12T16:40:59.365351+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e883: 12 total, 11 up, 11 in 2024-02-12T16:41:00.420 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:00 smithi169 bash[19604]: audit 2024-02-12T16:40:59.896596+0000 mon.a (mon.0) 3016 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:00.444 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:00.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:00.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:00.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:00.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:58.529284+0000 osd.10 (osd.10) 202 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:41:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:58.581378+0000 osd.10 (osd.10) 203 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:41:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:58.990181+0000 osd.2 (osd.2) 127 : cluster [DBG] 8.d scrub starts 2024-02-12T16:41:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:00 smithi123 bash[19119]: cluster 2024-02-12T16:40:59.365351+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e883: 12 total, 11 up, 11 in 2024-02-12T16:41:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:00 smithi123 bash[19119]: audit 2024-02-12T16:40:59.896596+0000 mon.a (mon.0) 3016 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:58.529284+0000 osd.10 (osd.10) 202 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:41:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:58.581378+0000 osd.10 (osd.10) 203 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:41:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:58.990181+0000 osd.2 (osd.2) 127 : cluster [DBG] 8.d scrub starts 2024-02-12T16:41:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:00 smithi121 bash[18489]: cluster 2024-02-12T16:40:59.365351+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e883: 12 total, 11 up, 11 in 2024-02-12T16:41:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:00 smithi121 bash[18489]: audit 2024-02-12T16:40:59.896596+0000 mon.a (mon.0) 3016 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:00.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:00.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:00.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:00.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:00.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:00.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:00.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:01.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:41:01.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.158 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:01.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:01.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:01.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.433 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:01.539 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:01 smithi169 bash[19604]: cluster 2024-02-12T16:40:59.806805+0000 osd.2 (osd.2) 128 : cluster [DBG] 8.d scrub ok 2024-02-12T16:41:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:01 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.229000+0000 mgr.y (mgr.14146) 1700 : cluster [DBG] pgmap v2454: 57 pgs: 1 remapped+peering, 3 peering, 10 unknown, 1 active+clean+scrubbing, 4 active+clean+remapped, 6 undersized+peered, 32 active+clean; 3.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3838/107368 objects misplaced (3.575%) 2024-02-12T16:41:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:01 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.235225+0000 osd.1 (osd.1) 147 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:41:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:01 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.376330+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e884: 12 total, 11 up, 11 in 2024-02-12T16:41:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:01 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.413182+0000 osd.1 (osd.1) 148 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:41:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:01 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.427343+0000 osd.5 (osd.5) 195 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:41:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:01 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.457802+0000 osd.5 (osd.5) 196 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:41:01.564 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:01 smithi123 bash[19119]: cluster 2024-02-12T16:40:59.806805+0000 osd.2 (osd.2) 128 : cluster [DBG] 8.d scrub ok 2024-02-12T16:41:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:01 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.229000+0000 mgr.y (mgr.14146) 1700 : cluster [DBG] pgmap v2454: 57 pgs: 1 remapped+peering, 3 peering, 10 unknown, 1 active+clean+scrubbing, 4 active+clean+remapped, 6 undersized+peered, 32 active+clean; 3.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3838/107368 objects misplaced (3.575%) 2024-02-12T16:41:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:01 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.235225+0000 osd.1 (osd.1) 147 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:41:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:01 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.376330+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e884: 12 total, 11 up, 11 in 2024-02-12T16:41:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:01 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.413182+0000 osd.1 (osd.1) 148 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:41:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:01 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.427343+0000 osd.5 (osd.5) 195 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:41:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:01 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.457802+0000 osd.5 (osd.5) 196 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:41:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:01 smithi121 bash[18489]: cluster 2024-02-12T16:40:59.806805+0000 osd.2 (osd.2) 128 : cluster [DBG] 8.d scrub ok 2024-02-12T16:41:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:01 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.229000+0000 mgr.y (mgr.14146) 1700 : cluster [DBG] pgmap v2454: 57 pgs: 1 remapped+peering, 3 peering, 10 unknown, 1 active+clean+scrubbing, 4 active+clean+remapped, 6 undersized+peered, 32 active+clean; 3.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3838/107368 objects misplaced (3.575%) 2024-02-12T16:41:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:01 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.235225+0000 osd.1 (osd.1) 147 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:41:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:01 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.376330+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e884: 12 total, 11 up, 11 in 2024-02-12T16:41:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:01 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.413182+0000 osd.1 (osd.1) 148 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:41:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:01 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.427343+0000 osd.5 (osd.5) 195 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:41:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:01 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.457802+0000 osd.5 (osd.5) 196 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:41:01.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:01.694 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:01.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:01.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:01.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.970 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:01.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:02.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:02.100 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:02.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:02.204 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:02.236 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:02.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:02.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:02.372 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:02.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:02.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.573356+0000 osd.10 (osd.10) 204 : cluster [DBG] 8.6 scrub starts 2024-02-12T16:41:02.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:00.862575+0000 osd.10 (osd.10) 205 : cluster [DBG] 8.6 scrub ok 2024-02-12T16:41:02.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:01.185104+0000 osd.9 (osd.9) 155 : cluster [DBG] 1.a scrub starts 2024-02-12T16:41:02.425 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:41:02.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:02.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:02.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:02.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:02.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:02.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.573356+0000 osd.10 (osd.10) 204 : cluster [DBG] 8.6 scrub starts 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:00.862575+0000 osd.10 (osd.10) 205 : cluster [DBG] 8.6 scrub ok 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:01.185104+0000 osd.9 (osd.9) 155 : cluster [DBG] 1.a scrub starts 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:01.225324+0000 osd.9 (osd.9) 156 : cluster [DBG] 1.a scrub ok 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:01.379445+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e885: 12 total, 11 up, 11 in 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:01.469764+0000 osd.5 (osd.5) 197 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:01.510064+0000 osd.5 (osd.5) 198 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:01.711051+0000 osd.3 (osd.3) 101 : cluster [DBG] 1.c scrub starts 2024-02-12T16:41:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:02 smithi123 bash[19119]: cluster 2024-02-12T16:41:01.822268+0000 osd.3 (osd.3) 102 : cluster [DBG] 1.c scrub ok 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.573356+0000 osd.10 (osd.10) 204 : cluster [DBG] 8.6 scrub starts 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:00.862575+0000 osd.10 (osd.10) 205 : cluster [DBG] 8.6 scrub ok 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:01.185104+0000 osd.9 (osd.9) 155 : cluster [DBG] 1.a scrub starts 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:01.225324+0000 osd.9 (osd.9) 156 : cluster [DBG] 1.a scrub ok 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:01.379445+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e885: 12 total, 11 up, 11 in 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:01.469764+0000 osd.5 (osd.5) 197 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:01.510064+0000 osd.5 (osd.5) 198 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:01.711051+0000 osd.3 (osd.3) 101 : cluster [DBG] 1.c scrub starts 2024-02-12T16:41:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:02 smithi121 bash[18489]: cluster 2024-02-12T16:41:01.822268+0000 osd.3 (osd.3) 102 : cluster [DBG] 1.c scrub ok 2024-02-12T16:41:02.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:02.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:01.225324+0000 osd.9 (osd.9) 156 : cluster [DBG] 1.a scrub ok 2024-02-12T16:41:02.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:01.379445+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e885: 12 total, 11 up, 11 in 2024-02-12T16:41:02.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:01.469764+0000 osd.5 (osd.5) 197 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:41:02.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:01.510064+0000 osd.5 (osd.5) 198 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:41:02.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:01.711051+0000 osd.3 (osd.3) 101 : cluster [DBG] 1.c scrub starts 2024-02-12T16:41:02.744 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:02 smithi169 bash[19604]: cluster 2024-02-12T16:41:01.822268+0000 osd.3 (osd.3) 102 : cluster [DBG] 1.c scrub ok 2024-02-12T16:41:02.760 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:02.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:02.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:02.886 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:02.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:02.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:03.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:03.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:03.160 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:03.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:03.293 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:03.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:03.424 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:03.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:03.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:03 smithi121 bash[18489]: cluster 2024-02-12T16:41:02.229682+0000 mgr.y (mgr.14146) 1701 : cluster [DBG] pgmap v2457: 57 pgs: 1 remapped+peering, 3 peering, 4 unknown, 4 active+clean+remapped, 6 undersized+peered, 39 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3552/102460 objects misplaced (3.467%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:03.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:03 smithi121 bash[18489]: cluster 2024-02-12T16:41:02.246552+0000 osd.1 (osd.1) 149 : cluster [DBG] 8.7 scrub starts 2024-02-12T16:41:03.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:03 smithi121 bash[18489]: cluster 2024-02-12T16:41:02.490204+0000 osd.5 (osd.5) 199 : cluster [DBG] 8.4 scrub starts 2024-02-12T16:41:03.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:03 smithi121 bash[18489]: cluster 2024-02-12T16:41:02.724374+0000 osd.1 (osd.1) 150 : cluster [DBG] 8.7 scrub ok 2024-02-12T16:41:03.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:03 smithi121 bash[18489]: cluster 2024-02-12T16:41:02.785193+0000 osd.8 (osd.8) 121 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:41:03.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:03 smithi121 bash[18489]: cluster 2024-02-12T16:41:02.859010+0000 osd.5 (osd.5) 200 : cluster [DBG] 8.4 scrub ok 2024-02-12T16:41:03.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:03 smithi121 bash[18489]: cluster 2024-02-12T16:41:02.908274+0000 osd.8 (osd.8) 122 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:41:03.566 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:03.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:03.697 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:03 smithi169 bash[19604]: cluster 2024-02-12T16:41:02.229682+0000 mgr.y (mgr.14146) 1701 : cluster [DBG] pgmap v2457: 57 pgs: 1 remapped+peering, 3 peering, 4 unknown, 4 active+clean+remapped, 6 undersized+peered, 39 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3552/102460 objects misplaced (3.467%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:03 smithi169 bash[19604]: cluster 2024-02-12T16:41:02.246552+0000 osd.1 (osd.1) 149 : cluster [DBG] 8.7 scrub starts 2024-02-12T16:41:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:03 smithi169 bash[19604]: cluster 2024-02-12T16:41:02.490204+0000 osd.5 (osd.5) 199 : cluster [DBG] 8.4 scrub starts 2024-02-12T16:41:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:03 smithi169 bash[19604]: cluster 2024-02-12T16:41:02.724374+0000 osd.1 (osd.1) 150 : cluster [DBG] 8.7 scrub ok 2024-02-12T16:41:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:03 smithi169 bash[19604]: cluster 2024-02-12T16:41:02.785193+0000 osd.8 (osd.8) 121 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:41:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:03 smithi169 bash[19604]: cluster 2024-02-12T16:41:02.859010+0000 osd.5 (osd.5) 200 : cluster [DBG] 8.4 scrub ok 2024-02-12T16:41:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:03 smithi169 bash[19604]: cluster 2024-02-12T16:41:02.908274+0000 osd.8 (osd.8) 122 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:41:03.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:03.825 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:03.874 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8] out_osds: [11] dead_osds: [7] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3] 2024-02-12T16:41:03.874 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:41:03.874 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-02-12T16:41:03.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd in 11 2024-02-12T16:41:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:03 smithi123 bash[19119]: cluster 2024-02-12T16:41:02.229682+0000 mgr.y (mgr.14146) 1701 : cluster [DBG] pgmap v2457: 57 pgs: 1 remapped+peering, 3 peering, 4 unknown, 4 active+clean+remapped, 6 undersized+peered, 39 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3552/102460 objects misplaced (3.467%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:03 smithi123 bash[19119]: cluster 2024-02-12T16:41:02.246552+0000 osd.1 (osd.1) 149 : cluster [DBG] 8.7 scrub starts 2024-02-12T16:41:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:03 smithi123 bash[19119]: cluster 2024-02-12T16:41:02.490204+0000 osd.5 (osd.5) 199 : cluster [DBG] 8.4 scrub starts 2024-02-12T16:41:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:03 smithi123 bash[19119]: cluster 2024-02-12T16:41:02.724374+0000 osd.1 (osd.1) 150 : cluster [DBG] 8.7 scrub ok 2024-02-12T16:41:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:03 smithi123 bash[19119]: cluster 2024-02-12T16:41:02.785193+0000 osd.8 (osd.8) 121 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:41:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:03 smithi123 bash[19119]: cluster 2024-02-12T16:41:02.859010+0000 osd.5 (osd.5) 200 : cluster [DBG] 8.4 scrub ok 2024-02-12T16:41:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:03 smithi123 bash[19119]: cluster 2024-02-12T16:41:02.908274+0000 osd.8 (osd.8) 122 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:41:03.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:03.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:03.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:04.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:04.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:04.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:04.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:04.228 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:04.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:04.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:04.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:04.451 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:04.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:04.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:04.582 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:04.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:04.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:04.709 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:04.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:04.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:04.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:04.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:04.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:04.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:04.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:04.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:05.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:05.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:05.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:05.111 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:05.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:05.233 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:05.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:05.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:05.368 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:05.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:05.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:05.478 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:05 smithi169 bash[19604]: cluster 2024-02-12T16:41:03.566434+0000 osd.10 (osd.10) 206 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:05.478 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:05 smithi169 bash[19604]: cluster 2024-02-12T16:41:03.566488+0000 osd.10 (osd.10) 207 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:05.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:05 smithi169 bash[19604]: cluster 2024-02-12T16:41:04.230205+0000 mgr.y (mgr.14146) 1702 : cluster [DBG] pgmap v2458: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 6857/113110 objects misplaced (6.062%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:05.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:05 smithi169 bash[19604]: cluster 2024-02-12T16:41:04.400334+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e886: 12 total, 11 up, 11 in 2024-02-12T16:41:05.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:05 smithi169 bash[19604]: cluster 2024-02-12T16:41:04.448238+0000 osd.10 (osd.10) 208 : cluster [DBG] 8.9 scrub starts 2024-02-12T16:41:05.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:05 smithi169 bash[19604]: cluster 2024-02-12T16:41:04.760601+0000 osd.8 (osd.8) 123 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:41:05.479 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:05 smithi169 bash[19604]: cluster 2024-02-12T16:41:04.844360+0000 osd.8 (osd.8) 124 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:41:05.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:05.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:05.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:05.633 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:05.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:05.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:05.762 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:05.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:05.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:05 smithi121 bash[18489]: cluster 2024-02-12T16:41:03.566434+0000 osd.10 (osd.10) 206 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:05.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:05 smithi121 bash[18489]: cluster 2024-02-12T16:41:03.566488+0000 osd.10 (osd.10) 207 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:05.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:05 smithi121 bash[18489]: cluster 2024-02-12T16:41:04.230205+0000 mgr.y (mgr.14146) 1702 : cluster [DBG] pgmap v2458: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 6857/113110 objects misplaced (6.062%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:05.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:05 smithi121 bash[18489]: cluster 2024-02-12T16:41:04.400334+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e886: 12 total, 11 up, 11 in 2024-02-12T16:41:05.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:05 smithi121 bash[18489]: cluster 2024-02-12T16:41:04.448238+0000 osd.10 (osd.10) 208 : cluster [DBG] 8.9 scrub starts 2024-02-12T16:41:05.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:05 smithi121 bash[18489]: cluster 2024-02-12T16:41:04.760601+0000 osd.8 (osd.8) 123 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:41:05.792 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:05 smithi121 bash[18489]: cluster 2024-02-12T16:41:04.844360+0000 osd.8 (osd.8) 124 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:41:05.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:05.872 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:05 smithi123 bash[19119]: cluster 2024-02-12T16:41:03.566434+0000 osd.10 (osd.10) 206 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:05.873 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:05 smithi123 bash[19119]: cluster 2024-02-12T16:41:03.566488+0000 osd.10 (osd.10) 207 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:05.873 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:05 smithi123 bash[19119]: cluster 2024-02-12T16:41:04.230205+0000 mgr.y (mgr.14146) 1702 : cluster [DBG] pgmap v2458: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 6857/113110 objects misplaced (6.062%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:05.873 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:05 smithi123 bash[19119]: cluster 2024-02-12T16:41:04.400334+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e886: 12 total, 11 up, 11 in 2024-02-12T16:41:05.873 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:05 smithi123 bash[19119]: cluster 2024-02-12T16:41:04.448238+0000 osd.10 (osd.10) 208 : cluster [DBG] 8.9 scrub starts 2024-02-12T16:41:05.873 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:05 smithi123 bash[19119]: cluster 2024-02-12T16:41:04.760601+0000 osd.8 (osd.8) 123 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:41:05.873 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:05 smithi123 bash[19119]: cluster 2024-02-12T16:41:04.844360+0000 osd.8 (osd.8) 124 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:41:05.891 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:05.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:05.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:06.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:06.118 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:06.144 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:06.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:06.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:06.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:06.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:06.409 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:06 smithi169 bash[19604]: cluster 2024-02-12T16:41:04.895759+0000 osd.10 (osd.10) 209 : cluster [DBG] 8.9 scrub ok 2024-02-12T16:41:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:06.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:06.542 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:06.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:06 smithi121 bash[18489]: cluster 2024-02-12T16:41:04.895759+0000 osd.10 (osd.10) 209 : cluster [DBG] 8.9 scrub ok 2024-02-12T16:41:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:06 smithi121 bash[18489]: cluster 2024-02-12T16:41:05.253211+0000 osd.1 (osd.1) 151 : cluster [DBG] 8.1 scrub starts 2024-02-12T16:41:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:06 smithi121 bash[18489]: cluster 2024-02-12T16:41:05.713804+0000 osd.1 (osd.1) 152 : cluster [DBG] 8.1 scrub ok 2024-02-12T16:41:06.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:06.679 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:06 smithi169 bash[19604]: cluster 2024-02-12T16:41:05.253211+0000 osd.1 (osd.1) 151 : cluster [DBG] 8.1 scrub starts 2024-02-12T16:41:06.680 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:06 smithi169 bash[19604]: cluster 2024-02-12T16:41:05.713804+0000 osd.1 (osd.1) 152 : cluster [DBG] 8.1 scrub ok 2024-02-12T16:41:06.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:06.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:06.815 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:06 smithi123 bash[19119]: cluster 2024-02-12T16:41:04.895759+0000 osd.10 (osd.10) 209 : cluster [DBG] 8.9 scrub ok 2024-02-12T16:41:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:06 smithi123 bash[19119]: cluster 2024-02-12T16:41:05.253211+0000 osd.1 (osd.1) 151 : cluster [DBG] 8.1 scrub starts 2024-02-12T16:41:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:06 smithi123 bash[19119]: cluster 2024-02-12T16:41:05.713804+0000 osd.1 (osd.1) 152 : cluster [DBG] 8.1 scrub ok 2024-02-12T16:41:06.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:06.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:07.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:07.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:07.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:07.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:07.219 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:07.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:07.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:07.351 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:07.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:07.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:07.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:07.464 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:07 smithi169 bash[19604]: cluster 2024-02-12T16:41:06.230691+0000 mgr.y (mgr.14146) 1703 : cluster [DBG] pgmap v2460: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 81 MiB/s wr, 2.40k op/s; 6857/113110 objects misplaced (6.062%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:07.484 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:07.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:07.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:07.614 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:07.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:07.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:07 smithi121 bash[18489]: cluster 2024-02-12T16:41:06.230691+0000 mgr.y (mgr.14146) 1703 : cluster [DBG] pgmap v2460: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 81 MiB/s wr, 2.40k op/s; 6857/113110 objects misplaced (6.062%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:07.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:07.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:07.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:07.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:07.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:07.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:07 smithi123 bash[19119]: cluster 2024-02-12T16:41:06.230691+0000 mgr.y (mgr.14146) 1703 : cluster [DBG] pgmap v2460: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 81 MiB/s wr, 2.40k op/s; 6857/113110 objects misplaced (6.062%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26785.0:120310 8.c e886/839 8:347819b9:::benchmark_data_smithi169_115677_object37682:head v 886'7401, mlcod=886'7399)", 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:07.951799+0000", 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.010455018999999999, 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010486641999999999, 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:07.966 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951799+0000", 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951799+0000", 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951797+0000", 2024-02-12T16:41:07.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5030000000000003e-06 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951807+0000", 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5389000000000001e-05 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951822+0000", 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5830000000000003e-06 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:07.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951829+0000", 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0282999999999999e-05 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951889+0000", 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5522000000000001e-05 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.951915+0000", 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115535 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:07.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26785.0:120314 8.c e886/839 8:37c19ba5:::benchmark_data_smithi169_115677_object37686:head v 886'7402, mlcod=886'7399)", 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:07.955856+0000", 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.006398094, 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0064943819999999999, 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955856+0000", 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955856+0000", 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955855+0000", 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2500000000000006e-06 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955863+0000", 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3331e-05 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955876+0000", 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8479999999999997e-06 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955882+0000", 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1384000000000005e-05 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955964+0000", 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7944e-05 2024-02-12T16:41:07.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.955982+0000", 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000125375 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26785.0:120316 8.c e886/839 8:3427cc52:::benchmark_data_smithi169_115677_object37688:head v 886'7403, mlcod=886'7399)", 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:07.959810+0000", 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0024442650000000002, 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025785529999999999, 2024-02-12T16:41:07.973 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.959810+0000", 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.959810+0000", 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:07.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.959809+0000", 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0739999999999999e-06 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.959818+0000", 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2490999999999999e-05 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.959830+0000", 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7400000000000001e-06 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:07.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.959836+0000", 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013371600000000001 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.959970+0000", 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6999999999999997e-05 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:07.960017+0000", 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020659000000000001 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:07.976 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-02-12T16:41:07.977 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 3 2024-02-12T16:41:07.977 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:41:07.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:08.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:08.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:08.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:08.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:08.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:08.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:08.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:08.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:08.520 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:08.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:08 smithi123 bash[19119]: cluster 2024-02-12T16:41:07.402274+0000 osd.5 (osd.5) 201 : cluster [DBG] 8.8 scrub starts 2024-02-12T16:41:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:08 smithi123 bash[19119]: cluster 2024-02-12T16:41:07.635008+0000 osd.5 (osd.5) 202 : cluster [DBG] 8.8 scrub ok 2024-02-12T16:41:08.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:08.659 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:08 smithi121 bash[18489]: cluster 2024-02-12T16:41:07.402274+0000 osd.5 (osd.5) 201 : cluster [DBG] 8.8 scrub starts 2024-02-12T16:41:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:08 smithi121 bash[18489]: cluster 2024-02-12T16:41:07.635008+0000 osd.5 (osd.5) 202 : cluster [DBG] 8.8 scrub ok 2024-02-12T16:41:08.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:08.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:08 smithi169 bash[19604]: cluster 2024-02-12T16:41:07.402274+0000 osd.5 (osd.5) 201 : cluster [DBG] 8.8 scrub starts 2024-02-12T16:41:08.761 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:08 smithi169 bash[19604]: cluster 2024-02-12T16:41:07.635008+0000 osd.5 (osd.5) 202 : cluster [DBG] 8.8 scrub ok 2024-02-12T16:41:08.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:08.883 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:08.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:08.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:08.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:09.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:09.043 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:09.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-12T16:41:09.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:09.173 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:09.203 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:09.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:09.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:09.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:09.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:09.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:09 smithi169 bash[19604]: cluster 2024-02-12T16:41:08.231337+0000 mgr.y (mgr.14146) 1704 : cluster [DBG] pgmap v2461: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.36k op/s; 6159/99916 objects misplaced (6.164%) 2024-02-12T16:41:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:09 smithi169 bash[19604]: cluster 2024-02-12T16:41:08.252067+0000 osd.1 (osd.1) 153 : cluster [DBG] 8.3 scrub starts 2024-02-12T16:41:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:09 smithi169 bash[19604]: cluster 2024-02-12T16:41:08.546820+0000 osd.1 (osd.1) 154 : cluster [DBG] 8.3 scrub ok 2024-02-12T16:41:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:09 smithi169 bash[19604]: cluster 2024-02-12T16:41:08.748952+0000 osd.8 (osd.8) 125 : cluster [DBG] 8.a scrub starts 2024-02-12T16:41:09.571 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:09.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:09.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:09 smithi121 bash[18489]: cluster 2024-02-12T16:41:08.231337+0000 mgr.y (mgr.14146) 1704 : cluster [DBG] pgmap v2461: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.36k op/s; 6159/99916 objects misplaced (6.164%) 2024-02-12T16:41:09.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:09 smithi121 bash[18489]: cluster 2024-02-12T16:41:08.252067+0000 osd.1 (osd.1) 153 : cluster [DBG] 8.3 scrub starts 2024-02-12T16:41:09.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:09 smithi121 bash[18489]: cluster 2024-02-12T16:41:08.546820+0000 osd.1 (osd.1) 154 : cluster [DBG] 8.3 scrub ok 2024-02-12T16:41:09.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:09 smithi121 bash[18489]: cluster 2024-02-12T16:41:08.748952+0000 osd.8 (osd.8) 125 : cluster [DBG] 8.a scrub starts 2024-02-12T16:41:09.701 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:09.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:09.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:09.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:09 smithi123 bash[19119]: cluster 2024-02-12T16:41:08.231337+0000 mgr.y (mgr.14146) 1704 : cluster [DBG] pgmap v2461: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 3.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.36k op/s; 6159/99916 objects misplaced (6.164%) 2024-02-12T16:41:09.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:09 smithi123 bash[19119]: cluster 2024-02-12T16:41:08.252067+0000 osd.1 (osd.1) 153 : cluster [DBG] 8.3 scrub starts 2024-02-12T16:41:09.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:09 smithi123 bash[19119]: cluster 2024-02-12T16:41:08.546820+0000 osd.1 (osd.1) 154 : cluster [DBG] 8.3 scrub ok 2024-02-12T16:41:09.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:09 smithi123 bash[19119]: cluster 2024-02-12T16:41:08.748952+0000 osd.8 (osd.8) 125 : cluster [DBG] 8.a scrub starts 2024-02-12T16:41:09.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:09.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:09.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:10.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:10.090 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:10.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:10.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:10.220 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:10.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:10.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:10.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:10.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:10.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:10.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:10.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:10.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:10.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:10.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:10 smithi123 bash[19119]: cluster 2024-02-12T16:41:09.455233+0000 osd.10 (osd.10) 210 : cluster [DBG] 8.5 scrub starts 2024-02-12T16:41:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:10 smithi123 bash[19119]: cluster 2024-02-12T16:41:09.539950+0000 osd.8 (osd.8) 126 : cluster [DBG] 8.a scrub ok 2024-02-12T16:41:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:10 smithi123 bash[19119]: cluster 2024-02-12T16:41:09.881312+0000 osd.3 (osd.3) 103 : cluster [DBG] 8.0 scrub starts 2024-02-12T16:41:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:10 smithi123 bash[19119]: cluster 2024-02-12T16:41:10.091177+0000 osd.3 (osd.3) 104 : cluster [DBG] 8.0 scrub ok 2024-02-12T16:41:10.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:10.746 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:10.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:10.782 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:10 smithi121 bash[18489]: cluster 2024-02-12T16:41:09.455233+0000 osd.10 (osd.10) 210 : cluster [DBG] 8.5 scrub starts 2024-02-12T16:41:10.783 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:10 smithi121 bash[18489]: cluster 2024-02-12T16:41:09.539950+0000 osd.8 (osd.8) 126 : cluster [DBG] 8.a scrub ok 2024-02-12T16:41:10.783 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:10 smithi121 bash[18489]: cluster 2024-02-12T16:41:09.881312+0000 osd.3 (osd.3) 103 : cluster [DBG] 8.0 scrub starts 2024-02-12T16:41:10.783 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:10 smithi121 bash[18489]: cluster 2024-02-12T16:41:10.091177+0000 osd.3 (osd.3) 104 : cluster [DBG] 8.0 scrub ok 2024-02-12T16:41:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:10 smithi169 bash[19604]: cluster 2024-02-12T16:41:09.455233+0000 osd.10 (osd.10) 210 : cluster [DBG] 8.5 scrub starts 2024-02-12T16:41:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:10 smithi169 bash[19604]: cluster 2024-02-12T16:41:09.539950+0000 osd.8 (osd.8) 126 : cluster [DBG] 8.a scrub ok 2024-02-12T16:41:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:10 smithi169 bash[19604]: cluster 2024-02-12T16:41:09.881312+0000 osd.3 (osd.3) 103 : cluster [DBG] 8.0 scrub starts 2024-02-12T16:41:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:10 smithi169 bash[19604]: cluster 2024-02-12T16:41:10.091177+0000 osd.3 (osd.3) 104 : cluster [DBG] 8.0 scrub ok 2024-02-12T16:41:10.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:10.874 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:10.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:10.977 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:11.003 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:11.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:11.134 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:11.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:11.262 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:11.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:11.394 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:11.475 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:11 smithi121 bash[18489]: cluster 2024-02-12T16:41:09.704104+0000 osd.10 (osd.10) 211 : cluster [DBG] 8.5 scrub ok 2024-02-12T16:41:11.475 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:11 smithi121 bash[18489]: cluster 2024-02-12T16:41:10.231838+0000 mgr.y (mgr.14146) 1705 : cluster [DBG] pgmap v2462: 57 pgs: 1 active+clean+scrubbing, 6 active+clean+remapped, 6 undersized+peered, 44 active+clean; 2.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s; 5539/92258 objects misplaced (6.004%) 2024-02-12T16:41:11.476 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:11 smithi121 bash[18489]: cluster 2024-02-12T16:41:10.439584+0000 osd.10 (osd.10) 212 : cluster [DBG] 8.2 scrub starts 2024-02-12T16:41:11.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:11.528 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:11.546 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:41:11.596 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:41:11.631 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:41:11.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:11.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:11.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:11.677 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:41:11.704 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:41:11.735 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:41:11.763 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:41:11.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:11.778 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:11 smithi169 bash[19604]: cluster 2024-02-12T16:41:09.704104+0000 osd.10 (osd.10) 211 : cluster [DBG] 8.5 scrub ok 2024-02-12T16:41:11.778 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:11 smithi169 bash[19604]: cluster 2024-02-12T16:41:10.231838+0000 mgr.y (mgr.14146) 1705 : cluster [DBG] pgmap v2462: 57 pgs: 1 active+clean+scrubbing, 6 active+clean+remapped, 6 undersized+peered, 44 active+clean; 2.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s; 5539/92258 objects misplaced (6.004%) 2024-02-12T16:41:11.778 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:11 smithi169 bash[19604]: cluster 2024-02-12T16:41:10.439584+0000 osd.10 (osd.10) 212 : cluster [DBG] 8.2 scrub starts 2024-02-12T16:41:11.795 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:11.797 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:41:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:11.825 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:41:11.860 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:41:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:11 smithi123 bash[19119]: cluster 2024-02-12T16:41:09.704104+0000 osd.10 (osd.10) 211 : cluster [DBG] 8.5 scrub ok 2024-02-12T16:41:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:11 smithi123 bash[19119]: cluster 2024-02-12T16:41:10.231838+0000 mgr.y (mgr.14146) 1705 : cluster [DBG] pgmap v2462: 57 pgs: 1 active+clean+scrubbing, 6 active+clean+remapped, 6 undersized+peered, 44 active+clean; 2.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s; 5539/92258 objects misplaced (6.004%) 2024-02-12T16:41:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:11 smithi123 bash[19119]: cluster 2024-02-12T16:41:10.439584+0000 osd.10 (osd.10) 212 : cluster [DBG] 8.2 scrub starts 2024-02-12T16:41:11.889 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:41:11.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:11.952 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:11.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:12.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:12.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:12.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:12.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:12.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:12.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:12.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:12.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:12.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:12.449 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:41:12.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:12.485 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:12.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:12.510 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:12 smithi121 bash[18489]: cluster 2024-02-12T16:41:10.870687+0000 osd.10 (osd.10) 213 : cluster [DBG] 8.2 scrub ok 2024-02-12T16:41:12.510 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:12 smithi121 bash[18489]: cluster 2024-02-12T16:41:11.425075+0000 osd.10 (osd.10) 214 : cluster [DBG] 8.15 scrub starts 2024-02-12T16:41:12.510 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:12 smithi121 bash[18489]: audit 2024-02-12T16:41:11.505102+0000 mon.a (mon.0) 3020 : audit [DBG] from='client.? v1:172.21.15.121:0/3913230981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:41:12.510 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:12 smithi121 bash[18489]: audit 2024-02-12T16:41:12.065409+0000 mon.a (mon.0) 3021 : audit [INF] from='client.26954 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:41:12.510 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:12 smithi121 bash[18489]: audit 2024-02-12T16:41:12.067590+0000 mon.b (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.121:0/1071808828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:41:12.511 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:12 smithi121 bash[18489]: cluster 2024-02-12T16:41:12.271107+0000 osd.5 (osd.5) 203 : cluster [DBG] 8.14 scrub starts 2024-02-12T16:41:12.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:12.622 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:12.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:12.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:12.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:12.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:12 smithi169 bash[19604]: cluster 2024-02-12T16:41:10.870687+0000 osd.10 (osd.10) 213 : cluster [DBG] 8.2 scrub ok 2024-02-12T16:41:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:12 smithi169 bash[19604]: cluster 2024-02-12T16:41:11.425075+0000 osd.10 (osd.10) 214 : cluster [DBG] 8.15 scrub starts 2024-02-12T16:41:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:12 smithi169 bash[19604]: audit 2024-02-12T16:41:11.505102+0000 mon.a (mon.0) 3020 : audit [DBG] from='client.? v1:172.21.15.121:0/3913230981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:41:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:12 smithi169 bash[19604]: audit 2024-02-12T16:41:12.065409+0000 mon.a (mon.0) 3021 : audit [INF] from='client.26954 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:41:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:12 smithi169 bash[19604]: audit 2024-02-12T16:41:12.067590+0000 mon.b (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.121:0/1071808828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:41:12.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:12 smithi169 bash[19604]: cluster 2024-02-12T16:41:12.271107+0000 osd.5 (osd.5) 203 : cluster [DBG] 8.14 scrub starts 2024-02-12T16:41:12.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:12 smithi123 bash[19119]: cluster 2024-02-12T16:41:10.870687+0000 osd.10 (osd.10) 213 : cluster [DBG] 8.2 scrub ok 2024-02-12T16:41:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:12 smithi123 bash[19119]: cluster 2024-02-12T16:41:11.425075+0000 osd.10 (osd.10) 214 : cluster [DBG] 8.15 scrub starts 2024-02-12T16:41:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:12 smithi123 bash[19119]: audit 2024-02-12T16:41:11.505102+0000 mon.a (mon.0) 3020 : audit [DBG] from='client.? v1:172.21.15.121:0/3913230981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:41:12.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:12 smithi123 bash[19119]: audit 2024-02-12T16:41:12.065409+0000 mon.a (mon.0) 3021 : audit [INF] from='client.26954 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:41:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:12 smithi123 bash[19119]: audit 2024-02-12T16:41:12.067590+0000 mon.b (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.121:0/1071808828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:41:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:12 smithi123 bash[19119]: cluster 2024-02-12T16:41:12.271107+0000 osd.5 (osd.5) 203 : cluster [DBG] 8.14 scrub starts 2024-02-12T16:41:12.893 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:12.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:12.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:13.034 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:13.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:13.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:13.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:13.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:13.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:13.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:13.582 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:13 smithi121 bash[18489]: cluster 2024-02-12T16:41:11.603477+0000 osd.10 (osd.10) 215 : cluster [DBG] 8.15 scrub ok 2024-02-12T16:41:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:13 smithi121 bash[18489]: cluster 2024-02-12T16:41:12.232438+0000 mgr.y (mgr.14146) 1706 : cluster [DBG] pgmap v2463: 57 pgs: 1 active+clean+scrubbing, 6 active+clean+remapped, 6 undersized+peered, 44 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 312 op/s; 4911/80372 objects misplaced (6.110%) 2024-02-12T16:41:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:13 smithi121 bash[18489]: cluster 2024-02-12T16:41:12.422264+0000 mon.a (mon.0) 3022 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:41:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:13 smithi121 bash[18489]: audit 2024-02-12T16:41:12.434200+0000 mon.a (mon.0) 3023 : audit [INF] from='client.26954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:41:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:13 smithi121 bash[18489]: cluster 2024-02-12T16:41:12.434276+0000 mon.a (mon.0) 3024 : cluster [DBG] osdmap e887: 12 total, 11 up, 11 in 2024-02-12T16:41:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:13 smithi121 bash[18489]: cluster 2024-02-12T16:41:12.527996+0000 osd.5 (osd.5) 204 : cluster [DBG] 8.14 scrub ok 2024-02-12T16:41:13.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:13 smithi169 bash[19604]: cluster 2024-02-12T16:41:11.603477+0000 osd.10 (osd.10) 215 : cluster [DBG] 8.15 scrub ok 2024-02-12T16:41:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:13 smithi169 bash[19604]: cluster 2024-02-12T16:41:12.232438+0000 mgr.y (mgr.14146) 1706 : cluster [DBG] pgmap v2463: 57 pgs: 1 active+clean+scrubbing, 6 active+clean+remapped, 6 undersized+peered, 44 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 312 op/s; 4911/80372 objects misplaced (6.110%) 2024-02-12T16:41:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:13 smithi169 bash[19604]: cluster 2024-02-12T16:41:12.422264+0000 mon.a (mon.0) 3022 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:41:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:13 smithi169 bash[19604]: audit 2024-02-12T16:41:12.434200+0000 mon.a (mon.0) 3023 : audit [INF] from='client.26954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:41:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:13 smithi169 bash[19604]: cluster 2024-02-12T16:41:12.434276+0000 mon.a (mon.0) 3024 : cluster [DBG] osdmap e887: 12 total, 11 up, 11 in 2024-02-12T16:41:13.697 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:13 smithi169 bash[19604]: cluster 2024-02-12T16:41:12.527996+0000 osd.5 (osd.5) 204 : cluster [DBG] 8.14 scrub ok 2024-02-12T16:41:13.730 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:13.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:13.863 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:13.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:13 smithi123 bash[19119]: cluster 2024-02-12T16:41:11.603477+0000 osd.10 (osd.10) 215 : cluster [DBG] 8.15 scrub ok 2024-02-12T16:41:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:13 smithi123 bash[19119]: cluster 2024-02-12T16:41:12.232438+0000 mgr.y (mgr.14146) 1706 : cluster [DBG] pgmap v2463: 57 pgs: 1 active+clean+scrubbing, 6 active+clean+remapped, 6 undersized+peered, 44 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 312 op/s; 4911/80372 objects misplaced (6.110%) 2024-02-12T16:41:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:13 smithi123 bash[19119]: cluster 2024-02-12T16:41:12.422264+0000 mon.a (mon.0) 3022 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:41:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:13 smithi123 bash[19119]: audit 2024-02-12T16:41:12.434200+0000 mon.a (mon.0) 3023 : audit [INF] from='client.26954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:41:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:13 smithi123 bash[19119]: cluster 2024-02-12T16:41:12.434276+0000 mon.a (mon.0) 3024 : cluster [DBG] osdmap e887: 12 total, 11 up, 11 in 2024-02-12T16:41:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:13 smithi123 bash[19119]: cluster 2024-02-12T16:41:12.527996+0000 osd.5 (osd.5) 204 : cluster [DBG] 8.14 scrub ok 2024-02-12T16:41:13.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:13.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:13.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:14.043 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:41:14.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:14.122 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:14.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:14.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:14.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:14.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:14.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:14.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:14.391 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:14.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:14.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:14.538 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:14.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:41:14.541 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:41:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:14 smithi123 bash[19119]: cluster 2024-02-12T16:41:13.197942+0000 osd.1 (osd.1) 155 : cluster [DBG] 8.11 deep-scrub starts 2024-02-12T16:41:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:14 smithi123 bash[19119]: cluster 2024-02-12T16:41:13.357656+0000 osd.10 (osd.10) 216 : cluster [DBG] 8.19 scrub starts 2024-02-12T16:41:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:14 smithi123 bash[19119]: cluster 2024-02-12T16:41:13.635184+0000 osd.8 (osd.8) 127 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:41:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:14 smithi123 bash[19119]: cluster 2024-02-12T16:41:13.754011+0000 osd.8 (osd.8) 128 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:41:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:14 smithi123 bash[19119]: cluster 2024-02-12T16:41:13.827303+0000 osd.3 (osd.3) 105 : cluster [DBG] 8.10 scrub starts 2024-02-12T16:41:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:14 smithi123 bash[19119]: cluster 2024-02-12T16:41:14.027448+0000 osd.3 (osd.3) 106 : cluster [DBG] 8.10 scrub ok 2024-02-12T16:41:14.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:14 smithi121 bash[18489]: cluster 2024-02-12T16:41:13.197942+0000 osd.1 (osd.1) 155 : cluster [DBG] 8.11 deep-scrub starts 2024-02-12T16:41:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:14 smithi121 bash[18489]: cluster 2024-02-12T16:41:13.357656+0000 osd.10 (osd.10) 216 : cluster [DBG] 8.19 scrub starts 2024-02-12T16:41:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:14 smithi121 bash[18489]: cluster 2024-02-12T16:41:13.635184+0000 osd.8 (osd.8) 127 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:41:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:14 smithi121 bash[18489]: cluster 2024-02-12T16:41:13.754011+0000 osd.8 (osd.8) 128 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:41:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:14 smithi121 bash[18489]: cluster 2024-02-12T16:41:13.827303+0000 osd.3 (osd.3) 105 : cluster [DBG] 8.10 scrub starts 2024-02-12T16:41:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:14 smithi121 bash[18489]: cluster 2024-02-12T16:41:14.027448+0000 osd.3 (osd.3) 106 : cluster [DBG] 8.10 scrub ok 2024-02-12T16:41:14.663 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:14.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:14.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:14.766 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:14.789 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:14.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:14.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:14 smithi169 bash[19604]: cluster 2024-02-12T16:41:13.197942+0000 osd.1 (osd.1) 155 : cluster [DBG] 8.11 deep-scrub starts 2024-02-12T16:41:14.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:14 smithi169 bash[19604]: cluster 2024-02-12T16:41:13.357656+0000 osd.10 (osd.10) 216 : cluster [DBG] 8.19 scrub starts 2024-02-12T16:41:14.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:14 smithi169 bash[19604]: cluster 2024-02-12T16:41:13.635184+0000 osd.8 (osd.8) 127 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:41:14.807 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:14 smithi169 bash[19604]: cluster 2024-02-12T16:41:13.754011+0000 osd.8 (osd.8) 128 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:41:14.808 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:14 smithi169 bash[19604]: cluster 2024-02-12T16:41:13.827303+0000 osd.3 (osd.3) 105 : cluster [DBG] 8.10 scrub starts 2024-02-12T16:41:14.808 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:14 smithi169 bash[19604]: cluster 2024-02-12T16:41:14.027448+0000 osd.3 (osd.3) 106 : cluster [DBG] 8.10 scrub ok 2024-02-12T16:41:14.891 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:14.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:14.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:15.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:15.064 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:15.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:15.169 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:15.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:15.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:15.296 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:15.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:15.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:15.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:15.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:15.717 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:15 smithi121 bash[18489]: cluster 2024-02-12T16:41:13.635705+0000 osd.10 (osd.10) 217 : cluster [DBG] 8.19 scrub ok 2024-02-12T16:41:15.717 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:15 smithi121 bash[18489]: cluster 2024-02-12T16:41:14.233012+0000 mgr.y (mgr.14146) 1707 : cluster [DBG] pgmap v2465: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 4352/73462 objects misplaced (5.924%) 2024-02-12T16:41:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:15.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:15 smithi169 bash[19604]: cluster 2024-02-12T16:41:13.635705+0000 osd.10 (osd.10) 217 : cluster [DBG] 8.19 scrub ok 2024-02-12T16:41:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:15 smithi169 bash[19604]: cluster 2024-02-12T16:41:14.233012+0000 mgr.y (mgr.14146) 1707 : cluster [DBG] pgmap v2465: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 4352/73462 objects misplaced (5.924%) 2024-02-12T16:41:15.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:15.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:15.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:15 smithi123 bash[19119]: cluster 2024-02-12T16:41:13.635705+0000 osd.10 (osd.10) 217 : cluster [DBG] 8.19 scrub ok 2024-02-12T16:41:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:15 smithi123 bash[19119]: cluster 2024-02-12T16:41:14.233012+0000 mgr.y (mgr.14146) 1707 : cluster [DBG] pgmap v2465: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 4352/73462 objects misplaced (5.924%) 2024-02-12T16:41:15.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:15.972 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:15.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:16.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:16.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:16.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:16.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:16.225 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:16.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:16.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:16.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:16.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:16.456 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:16.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:16.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:16.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:41:16.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:16.609 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:16.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:16.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:16.740 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:16.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:16.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:16.875 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:16.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:16.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:16 smithi121 bash[18489]: debug 2024-02-12T16:41:16.666+0000 7fbba85de700 -1 mon.a@0(leader).osd e888 definitely_dead 0 2024-02-12T16:41:16.951 INFO:teuthology.orchestra.run.smithi121.stderr:marked in osd.11. 2024-02-12T16:41:16.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:16.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:16 smithi169 bash[19604]: cluster 2024-02-12T16:41:15.922454+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e888: 12 total, 11 up, 11 in 2024-02-12T16:41:16.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:16 smithi169 bash[19604]: cluster 2024-02-12T16:41:16.233532+0000 mgr.y (mgr.14146) 1708 : cluster [DBG] pgmap v2467: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 4352/73462 objects misplaced (5.924%) 2024-02-12T16:41:16.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:16 smithi169 bash[19604]: cluster 2024-02-12T16:41:16.280956+0000 osd.10 (osd.10) 218 : cluster [DBG] 8.12 deep-scrub starts 2024-02-12T16:41:16.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:16 smithi169 bash[19604]: cluster 2024-02-12T16:41:16.357702+0000 osd.5 (osd.5) 205 : cluster [DBG] 8.18 deep-scrub starts 2024-02-12T16:41:16.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:16 smithi169 bash[19604]: audit 2024-02-12T16:41:16.670293+0000 mon.a (mon.0) 3026 : audit [DBG] from='client.26960 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-02-12T16:41:16.984 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:16 smithi169 bash[19604]: audit 2024-02-12T16:41:16.672491+0000 mon.b (mon.2) 216 : audit [DBG] from='client.? v1:172.21.15.121:0/840038041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-02-12T16:41:16.998 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:17.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:41:17.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:17.127 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:17.230 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:17.253 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:17.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:17.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:16 smithi121 bash[18489]: cluster 2024-02-12T16:41:15.922454+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e888: 12 total, 11 up, 11 in 2024-02-12T16:41:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:16 smithi121 bash[18489]: cluster 2024-02-12T16:41:16.233532+0000 mgr.y (mgr.14146) 1708 : cluster [DBG] pgmap v2467: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 4352/73462 objects misplaced (5.924%) 2024-02-12T16:41:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:16 smithi121 bash[18489]: cluster 2024-02-12T16:41:16.280956+0000 osd.10 (osd.10) 218 : cluster [DBG] 8.12 deep-scrub starts 2024-02-12T16:41:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:16 smithi121 bash[18489]: cluster 2024-02-12T16:41:16.357702+0000 osd.5 (osd.5) 205 : cluster [DBG] 8.18 deep-scrub starts 2024-02-12T16:41:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:16 smithi121 bash[18489]: audit 2024-02-12T16:41:16.670293+0000 mon.a (mon.0) 3026 : audit [DBG] from='client.26960 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-02-12T16:41:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:16 smithi121 bash[18489]: audit 2024-02-12T16:41:16.672491+0000 mon.b (mon.2) 216 : audit [DBG] from='client.? v1:172.21.15.121:0/840038041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-02-12T16:41:17.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:17.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:17.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:17.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:17.619 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:16 smithi123 bash[19119]: cluster 2024-02-12T16:41:15.922454+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e888: 12 total, 11 up, 11 in 2024-02-12T16:41:17.619 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:16 smithi123 bash[19119]: cluster 2024-02-12T16:41:16.233532+0000 mgr.y (mgr.14146) 1708 : cluster [DBG] pgmap v2467: 57 pgs: 6 active+clean+remapped, 6 undersized+peered, 45 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 4352/73462 objects misplaced (5.924%) 2024-02-12T16:41:17.620 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:16 smithi123 bash[19119]: cluster 2024-02-12T16:41:16.280956+0000 osd.10 (osd.10) 218 : cluster [DBG] 8.12 deep-scrub starts 2024-02-12T16:41:17.620 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:16 smithi123 bash[19119]: cluster 2024-02-12T16:41:16.357702+0000 osd.5 (osd.5) 205 : cluster [DBG] 8.18 deep-scrub starts 2024-02-12T16:41:17.620 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:16 smithi123 bash[19119]: audit 2024-02-12T16:41:16.670293+0000 mon.a (mon.0) 3026 : audit [DBG] from='client.26960 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-02-12T16:41:17.620 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:16 smithi123 bash[19119]: audit 2024-02-12T16:41:16.672491+0000 mon.b (mon.2) 216 : audit [DBG] from='client.? v1:172.21.15.121:0/840038041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-02-12T16:41:17.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:17.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:17.760 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:17.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:17.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:17.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:17.953 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:17 smithi169 bash[19604]: cluster 2024-02-12T16:41:16.576053+0000 osd.8 (osd.8) 129 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:41:17.954 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:17 smithi169 bash[19604]: cluster 2024-02-12T16:41:16.606084+0000 osd.8 (osd.8) 130 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:41:17.954 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:17 smithi169 bash[19604]: audit 2024-02-12T16:41:16.940038+0000 mon.a (mon.0) 3027 : audit [INF] from='client.26960 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-02-12T16:41:17.954 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:17 smithi169 bash[19604]: cluster 2024-02-12T16:41:16.940127+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e889: 12 total, 11 up, 12 in 2024-02-12T16:41:17.954 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:17 smithi169 bash[19604]: cluster 2024-02-12T16:41:16.952965+0000 osd.1 (osd.1) 156 : cluster [DBG] 8.11 deep-scrub ok 2024-02-12T16:41:17.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.013 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:18.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:18.117 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-02-12T16:41:18.139 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:17 smithi123 bash[19119]: cluster 2024-02-12T16:41:16.576053+0000 osd.8 (osd.8) 129 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:41:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:17 smithi123 bash[19119]: cluster 2024-02-12T16:41:16.606084+0000 osd.8 (osd.8) 130 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:41:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:17 smithi123 bash[19119]: audit 2024-02-12T16:41:16.940038+0000 mon.a (mon.0) 3027 : audit [INF] from='client.26960 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-02-12T16:41:18.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:17 smithi123 bash[19119]: cluster 2024-02-12T16:41:16.940127+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e889: 12 total, 11 up, 12 in 2024-02-12T16:41:18.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:17 smithi123 bash[19119]: cluster 2024-02-12T16:41:16.952965+0000 osd.1 (osd.1) 156 : cluster [DBG] 8.11 deep-scrub ok 2024-02-12T16:41:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:17 smithi121 bash[18489]: cluster 2024-02-12T16:41:16.576053+0000 osd.8 (osd.8) 129 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:41:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:17 smithi121 bash[18489]: cluster 2024-02-12T16:41:16.606084+0000 osd.8 (osd.8) 130 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:41:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:17 smithi121 bash[18489]: audit 2024-02-12T16:41:16.940038+0000 mon.a (mon.0) 3027 : audit [INF] from='client.26960 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-02-12T16:41:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:17 smithi121 bash[18489]: cluster 2024-02-12T16:41:16.940127+0000 mon.a (mon.0) 3028 : cluster [DBG] osdmap e889: 12 total, 11 up, 12 in 2024-02-12T16:41:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:17 smithi121 bash[18489]: cluster 2024-02-12T16:41:16.952965+0000 osd.1 (osd.1) 156 : cluster [DBG] 8.11 deep-scrub ok 2024-02-12T16:41:18.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:18.266 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:18.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:18.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:18.494 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.515 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:18.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:18.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.765 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:18.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:18.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:18.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:18.996 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:19.026 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:18 smithi169 bash[19604]: cluster 2024-02-12T16:41:17.612388+0000 osd.8 (osd.8) 131 : cluster [DBG] 8.b scrub starts 2024-02-12T16:41:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:18 smithi169 bash[19604]: cluster 2024-02-12T16:41:17.926335+0000 osd.8 (osd.8) 132 : cluster [DBG] 8.b scrub ok 2024-02-12T16:41:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:18 smithi169 bash[19604]: cluster 2024-02-12T16:41:17.940801+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e890: 12 total, 11 up, 12 in 2024-02-12T16:41:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:18 smithi169 bash[19604]: cluster 2024-02-12T16:41:18.234077+0000 mgr.y (mgr.14146) 1709 : cluster [DBG] pgmap v2470: 57 pgs: 1 remapped+peering, 2 active+clean+scrubbing+deep, 5 active+clean+remapped, 6 undersized+peered, 43 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 2126/62996 objects misplaced (3.375%) 2024-02-12T16:41:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:18 smithi169 bash[19604]: cluster 2024-02-12T16:41:18.944317+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e891: 12 total, 11 up, 12 in 2024-02-12T16:41:19.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:19.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:19.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:19.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:18 smithi121 bash[18489]: cluster 2024-02-12T16:41:17.612388+0000 osd.8 (osd.8) 131 : cluster [DBG] 8.b scrub starts 2024-02-12T16:41:19.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:18 smithi121 bash[18489]: cluster 2024-02-12T16:41:17.926335+0000 osd.8 (osd.8) 132 : cluster [DBG] 8.b scrub ok 2024-02-12T16:41:19.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:18 smithi121 bash[18489]: cluster 2024-02-12T16:41:17.940801+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e890: 12 total, 11 up, 12 in 2024-02-12T16:41:19.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:18 smithi121 bash[18489]: cluster 2024-02-12T16:41:18.234077+0000 mgr.y (mgr.14146) 1709 : cluster [DBG] pgmap v2470: 57 pgs: 1 remapped+peering, 2 active+clean+scrubbing+deep, 5 active+clean+remapped, 6 undersized+peered, 43 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 2126/62996 objects misplaced (3.375%) 2024-02-12T16:41:19.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:18 smithi121 bash[18489]: cluster 2024-02-12T16:41:18.944317+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e891: 12 total, 11 up, 12 in 2024-02-12T16:41:19.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:19.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:18 smithi123 bash[19119]: cluster 2024-02-12T16:41:17.612388+0000 osd.8 (osd.8) 131 : cluster [DBG] 8.b scrub starts 2024-02-12T16:41:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:18 smithi123 bash[19119]: cluster 2024-02-12T16:41:17.926335+0000 osd.8 (osd.8) 132 : cluster [DBG] 8.b scrub ok 2024-02-12T16:41:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:18 smithi123 bash[19119]: cluster 2024-02-12T16:41:17.940801+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e890: 12 total, 11 up, 12 in 2024-02-12T16:41:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:18 smithi123 bash[19119]: cluster 2024-02-12T16:41:18.234077+0000 mgr.y (mgr.14146) 1709 : cluster [DBG] pgmap v2470: 57 pgs: 1 remapped+peering, 2 active+clean+scrubbing+deep, 5 active+clean+remapped, 6 undersized+peered, 43 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 2126/62996 objects misplaced (3.375%) 2024-02-12T16:41:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:18 smithi123 bash[19119]: cluster 2024-02-12T16:41:18.944317+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e891: 12 total, 11 up, 12 in 2024-02-12T16:41:19.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:19.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:19.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:19.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:19.788 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:19.806 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:41:19.806 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:41:19.815 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:19.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:19.948 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:19.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:19 smithi169 bash[19604]: cluster 2024-02-12T16:41:18.535443+0000 osd.10 (osd.10) 219 : cluster [DBG] 8.12 deep-scrub ok 2024-02-12T16:41:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:19 smithi169 bash[19604]: cluster 2024-02-12T16:41:18.687906+0000 osd.5 (osd.5) 206 : cluster [DBG] 8.18 deep-scrub ok 2024-02-12T16:41:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:19 smithi169 bash[19604]: cluster 2024-02-12T16:41:19.275345+0000 osd.10 (osd.10) 220 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:41:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:19 smithi169 bash[19604]: cluster 2024-02-12T16:41:19.313383+0000 osd.10 (osd.10) 221 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:41:20.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:20.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:20.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:20.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:20.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:20.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:19 smithi121 bash[18489]: cluster 2024-02-12T16:41:18.535443+0000 osd.10 (osd.10) 219 : cluster [DBG] 8.12 deep-scrub ok 2024-02-12T16:41:20.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:19 smithi121 bash[18489]: cluster 2024-02-12T16:41:18.687906+0000 osd.5 (osd.5) 206 : cluster [DBG] 8.18 deep-scrub ok 2024-02-12T16:41:20.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:19 smithi121 bash[18489]: cluster 2024-02-12T16:41:19.275345+0000 osd.10 (osd.10) 220 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:41:20.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:19 smithi121 bash[18489]: cluster 2024-02-12T16:41:19.313383+0000 osd.10 (osd.10) 221 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:41:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:20.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:20.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:20.321 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:19 smithi123 bash[19119]: cluster 2024-02-12T16:41:18.535443+0000 osd.10 (osd.10) 219 : cluster [DBG] 8.12 deep-scrub ok 2024-02-12T16:41:20.322 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:19 smithi123 bash[19119]: cluster 2024-02-12T16:41:18.687906+0000 osd.5 (osd.5) 206 : cluster [DBG] 8.18 deep-scrub ok 2024-02-12T16:41:20.322 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:19 smithi123 bash[19119]: cluster 2024-02-12T16:41:19.275345+0000 osd.10 (osd.10) 220 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:41:20.322 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:19 smithi123 bash[19119]: cluster 2024-02-12T16:41:19.313383+0000 osd.10 (osd.10) 221 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:41:20.338 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:20.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:20.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:20.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:20.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:20.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:20.630 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:20.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:20.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:20.763 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:20.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:20.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:20.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-12T16:41:20.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:20.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:20.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:21.005 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:20 smithi169 bash[19604]: cluster 2024-02-12T16:41:20.234628+0000 mgr.y (mgr.14146) 1710 : cluster [DBG] pgmap v2472: 57 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+remapped, 6 undersized+peered, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1718/59348 objects misplaced (2.895%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:21.005 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:20 smithi169 bash[19604]: cluster 2024-02-12T16:41:20.923430+0000 mon.a (mon.0) 3031 : cluster [DBG] osdmap e892: 12 total, 11 up, 12 in 2024-02-12T16:41:21.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:21.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:21.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:21.176 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:21.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:21.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:21.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:20 smithi121 bash[18489]: cluster 2024-02-12T16:41:20.234628+0000 mgr.y (mgr.14146) 1710 : cluster [DBG] pgmap v2472: 57 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+remapped, 6 undersized+peered, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1718/59348 objects misplaced (2.895%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:21.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:20 smithi121 bash[18489]: cluster 2024-02-12T16:41:20.923430+0000 mon.a (mon.0) 3031 : cluster [DBG] osdmap e892: 12 total, 11 up, 12 in 2024-02-12T16:41:21.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:21.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:20 smithi123 bash[19119]: cluster 2024-02-12T16:41:20.234628+0000 mgr.y (mgr.14146) 1710 : cluster [DBG] pgmap v2472: 57 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+remapped, 6 undersized+peered, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1718/59348 objects misplaced (2.895%); 0 B/s, 0 objects/s recovering 2024-02-12T16:41:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:20 smithi123 bash[19119]: cluster 2024-02-12T16:41:20.923430+0000 mon.a (mon.0) 3031 : cluster [DBG] osdmap e892: 12 total, 11 up, 12 in 2024-02-12T16:41:21.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:21.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:21.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:21.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:21.578 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:21.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:21.637 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:21.637 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:21.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:21.704 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:21.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:21.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:21.838 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:21.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:21.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:22.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:22.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:22.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:22.236 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:22.237 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:21 smithi123 bash[19119]: cluster 2024-02-12T16:41:21.237602+0000 osd.10 (osd.10) 222 : cluster [DBG] 1.1b deep-scrub starts 2024-02-12T16:41:22.237 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:21 smithi123 bash[19119]: cluster 2024-02-12T16:41:21.337422+0000 osd.10 (osd.10) 223 : cluster [DBG] 1.1b deep-scrub ok 2024-02-12T16:41:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:22.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:21 smithi169 bash[19604]: cluster 2024-02-12T16:41:21.237602+0000 osd.10 (osd.10) 222 : cluster [DBG] 1.1b deep-scrub starts 2024-02-12T16:41:22.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:21 smithi169 bash[19604]: cluster 2024-02-12T16:41:21.337422+0000 osd.10 (osd.10) 223 : cluster [DBG] 1.1b deep-scrub ok 2024-02-12T16:41:22.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:22.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:22.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:21 smithi121 bash[18489]: cluster 2024-02-12T16:41:21.237602+0000 osd.10 (osd.10) 222 : cluster [DBG] 1.1b deep-scrub starts 2024-02-12T16:41:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:21 smithi121 bash[18489]: cluster 2024-02-12T16:41:21.337422+0000 osd.10 (osd.10) 223 : cluster [DBG] 1.1b deep-scrub ok 2024-02-12T16:41:22.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:22.503 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:22.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:22.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:22.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:22.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:22.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:22.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:22.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:22.895 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:22.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:22.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:23.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:23.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8, 11] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3] 2024-02-12T16:41:23.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:41:23.118 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.9749978270940473 2024-02-12T16:41:23.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 3 0.9749978270940473 2024-02-12T16:41:23.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:22 smithi121 bash[18489]: cluster 2024-02-12T16:41:22.235325+0000 mgr.y (mgr.14146) 1711 : cluster [DBG] pgmap v2474: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/55836 objects degraded (13.742%); 4055/55836 objects misplaced (7.262%); 8.9 MiB/s, 143 objects/s recovering 2024-02-12T16:41:23.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:23.163 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:23.269 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:23.276 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:22 smithi169 bash[19604]: cluster 2024-02-12T16:41:22.235325+0000 mgr.y (mgr.14146) 1711 : cluster [DBG] pgmap v2474: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/55836 objects degraded (13.742%); 4055/55836 objects misplaced (7.262%); 8.9 MiB/s, 143 objects/s recovering 2024-02-12T16:41:23.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:23.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:22 smithi123 bash[19119]: cluster 2024-02-12T16:41:22.235325+0000 mgr.y (mgr.14146) 1711 : cluster [DBG] pgmap v2474: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/55836 objects degraded (13.742%); 4055/55836 objects misplaced (7.262%); 8.9 MiB/s, 143 objects/s recovering 2024-02-12T16:41:23.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:23.431 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:23.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:23.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:23.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:23.696 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:23.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:23.825 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:23.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:23.951 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:23.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:24.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:24.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:23 smithi121 bash[18489]: cluster 2024-02-12T16:41:22.803813+0000 osd.2 (osd.2) 129 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:41:24.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:23 smithi121 bash[18489]: cluster 2024-02-12T16:41:22.900074+0000 osd.2 (osd.2) 130 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:41:24.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:23 smithi121 bash[18489]: cluster 2024-02-12T16:41:22.968317+0000 mon.a (mon.0) 3032 : cluster [WRN] Health check update: Degraded data redundancy: 7673/55836 objects degraded (13.742%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:24.087 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:24.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:24.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:24.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:24.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:23 smithi169 bash[19604]: cluster 2024-02-12T16:41:22.803813+0000 osd.2 (osd.2) 129 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:41:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:23 smithi169 bash[19604]: cluster 2024-02-12T16:41:22.900074+0000 osd.2 (osd.2) 130 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:41:24.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:23 smithi169 bash[19604]: cluster 2024-02-12T16:41:22.968317+0000 mon.a (mon.0) 3032 : cluster [WRN] Health check update: Degraded data redundancy: 7673/55836 objects degraded (13.742%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:24.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:24.336 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:23 smithi123 bash[19119]: cluster 2024-02-12T16:41:22.803813+0000 osd.2 (osd.2) 129 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:41:24.336 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:23 smithi123 bash[19119]: cluster 2024-02-12T16:41:22.900074+0000 osd.2 (osd.2) 130 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:41:24.337 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:23 smithi123 bash[19119]: cluster 2024-02-12T16:41:22.968317+0000 mon.a (mon.0) 3032 : cluster [WRN] Health check update: Degraded data redundancy: 7673/55836 objects degraded (13.742%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:24.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:24.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:24.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:24.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:24.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:24.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:24.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:24.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:24.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:24.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:24.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:24.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:24.883 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:24.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:24.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:25.006 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:25.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:25.131 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:25.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:24 smithi123 bash[19119]: cluster 2024-02-12T16:41:24.170455+0000 osd.10 (osd.10) 224 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:41:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:24 smithi123 bash[19119]: cluster 2024-02-12T16:41:24.217101+0000 osd.10 (osd.10) 225 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:41:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:24 smithi123 bash[19119]: cluster 2024-02-12T16:41:24.235952+0000 mgr.y (mgr.14146) 1712 : cluster [DBG] pgmap v2475: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/52698 objects degraded (14.560%); 4055/52698 objects misplaced (7.695%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T16:41:25.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:25.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:24 smithi121 bash[18489]: cluster 2024-02-12T16:41:24.170455+0000 osd.10 (osd.10) 224 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:41:25.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:24 smithi121 bash[18489]: cluster 2024-02-12T16:41:24.217101+0000 osd.10 (osd.10) 225 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:41:25.242 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:24 smithi121 bash[18489]: cluster 2024-02-12T16:41:24.235952+0000 mgr.y (mgr.14146) 1712 : cluster [DBG] pgmap v2475: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/52698 objects degraded (14.560%); 4055/52698 objects misplaced (7.695%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T16:41:25.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:24 smithi169 bash[19604]: cluster 2024-02-12T16:41:24.170455+0000 osd.10 (osd.10) 224 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:41:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:24 smithi169 bash[19604]: cluster 2024-02-12T16:41:24.217101+0000 osd.10 (osd.10) 225 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:41:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:24 smithi169 bash[19604]: cluster 2024-02-12T16:41:24.235952+0000 mgr.y (mgr.14146) 1712 : cluster [DBG] pgmap v2475: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/52698 objects degraded (14.560%); 4055/52698 objects misplaced (7.695%); 7.5 MiB/s, 120 objects/s recovering 2024-02-12T16:41:25.364 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:25.387 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:25.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:25.519 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:25.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:25.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:25.702 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:41:25.759 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:25.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:25.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:25.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:26.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:26.044 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:25 smithi123 bash[19119]: cluster 2024-02-12T16:41:25.190936+0000 osd.10 (osd.10) 226 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:41:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:25 smithi123 bash[19119]: cluster 2024-02-12T16:41:25.248415+0000 osd.10 (osd.10) 227 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:41:26.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:26.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:25 smithi121 bash[18489]: cluster 2024-02-12T16:41:25.190936+0000 osd.10 (osd.10) 226 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:41:26.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:25 smithi121 bash[18489]: cluster 2024-02-12T16:41:25.248415+0000 osd.10 (osd.10) 227 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:41:26.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:26.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:25 smithi169 bash[19604]: cluster 2024-02-12T16:41:25.190936+0000 osd.10 (osd.10) 226 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:41:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:25 smithi169 bash[19604]: cluster 2024-02-12T16:41:25.248415+0000 osd.10 (osd.10) 227 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:41:26.322 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:26.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:26.459 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:26.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:26.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:26.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:26.723 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:26.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:26.856 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:26.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:26.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:26.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:27.003 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:26 smithi169 bash[19604]: cluster 2024-02-12T16:41:25.733157+0000 osd.4 (osd.4) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:27.003 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:27 smithi169 bash[19604]: cluster 2024-02-12T16:41:25.733203+0000 osd.4 (osd.4) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:27.003 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:27 smithi169 bash[19604]: cluster 2024-02-12T16:41:25.991208+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e893: 12 total, 11 up, 12 in 2024-02-12T16:41:27.003 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:27 smithi169 bash[19604]: cluster 2024-02-12T16:41:26.236508+0000 mgr.y (mgr.14146) 1713 : cluster [DBG] pgmap v2477: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/51400 objects degraded (14.928%); 3732/51400 objects misplaced (7.261%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:41:27.088 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:27.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:27.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:27 smithi123 bash[19119]: cluster 2024-02-12T16:41:25.733157+0000 osd.4 (osd.4) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:27 smithi123 bash[19119]: cluster 2024-02-12T16:41:25.733203+0000 osd.4 (osd.4) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:27 smithi123 bash[19119]: cluster 2024-02-12T16:41:25.991208+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e893: 12 total, 11 up, 12 in 2024-02-12T16:41:27.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:27 smithi123 bash[19119]: cluster 2024-02-12T16:41:26.236508+0000 mgr.y (mgr.14146) 1713 : cluster [DBG] pgmap v2477: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/51400 objects degraded (14.928%); 3732/51400 objects misplaced (7.261%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:41:27.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:27.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:27.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:27.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:27.359 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:26 smithi121 bash[18489]: cluster 2024-02-12T16:41:25.733157+0000 osd.4 (osd.4) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:27.359 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:27 smithi121 bash[18489]: cluster 2024-02-12T16:41:25.733203+0000 osd.4 (osd.4) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:27.359 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:27 smithi121 bash[18489]: cluster 2024-02-12T16:41:25.991208+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e893: 12 total, 11 up, 12 in 2024-02-12T16:41:27.359 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:27 smithi121 bash[18489]: cluster 2024-02-12T16:41:26.236508+0000 mgr.y (mgr.14146) 1713 : cluster [DBG] pgmap v2477: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7673/51400 objects degraded (14.928%); 3732/51400 objects misplaced (7.261%); 6.5 MiB/s, 104 objects/s recovering 2024-02-12T16:41:27.375 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:27.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:27.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:27.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:27.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:27.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:27.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:27.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:27.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:27.776 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:27.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:27.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:27.909 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:27.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:28.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:28.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:28.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:28.165 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:28.268 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:28.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:28 smithi169 bash[19604]: cluster 2024-02-12T16:41:27.523702+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:41:28.275 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:28 smithi169 bash[19604]: cluster 2024-02-12T16:41:27.641869+0000 osd.6 (osd.6) 179 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:41:28.299 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:28.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:28 smithi123 bash[19119]: cluster 2024-02-12T16:41:27.523702+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:41:28.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:28 smithi123 bash[19119]: cluster 2024-02-12T16:41:27.641869+0000 osd.6 (osd.6) 179 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:41:28.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:28 smithi121 bash[18489]: cluster 2024-02-12T16:41:27.523702+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:41:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:28 smithi121 bash[18489]: cluster 2024-02-12T16:41:27.641869+0000 osd.6 (osd.6) 179 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:41:28.428 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:28.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:28.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:28.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:28.697 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:28.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:28.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:28.955 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:28.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:29.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:29.090 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:29.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:29.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:29 smithi123 bash[19119]: cluster 2024-02-12T16:41:28.237200+0000 mgr.y (mgr.14146) 1714 : cluster [DBG] pgmap v2478: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7129/44970 objects degraded (15.853%); 3055/44970 objects misplaced (6.793%); 15 MiB/s, 247 objects/s recovering 2024-02-12T16:41:29.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:29.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:29.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:29.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:29 smithi169 bash[19604]: cluster 2024-02-12T16:41:28.237200+0000 mgr.y (mgr.14146) 1714 : cluster [DBG] pgmap v2478: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7129/44970 objects degraded (15.853%); 3055/44970 objects misplaced (6.793%); 15 MiB/s, 247 objects/s recovering 2024-02-12T16:41:29.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:29.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:29 smithi121 bash[18489]: cluster 2024-02-12T16:41:28.237200+0000 mgr.y (mgr.14146) 1714 : cluster [DBG] pgmap v2478: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7129/44970 objects degraded (15.853%); 3055/44970 objects misplaced (6.793%); 15 MiB/s, 247 objects/s recovering 2024-02-12T16:41:29.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:29.500 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:29.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:29.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:29.642 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:29.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:29.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:29.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:29.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:29.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:29.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:30.006 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:30.018 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:30 smithi121 bash[18489]: cluster 2024-02-12T16:41:28.528197+0000 osd.11 (osd.11) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:30.018 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:30 smithi121 bash[18489]: cluster 2024-02-12T16:41:28.528246+0000 osd.11 (osd.11) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:30.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.107 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:30.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:30.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:30.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:30 smithi169 bash[19604]: cluster 2024-02-12T16:41:28.528197+0000 osd.11 (osd.11) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:30 smithi169 bash[19604]: cluster 2024-02-12T16:41:28.528246+0000 osd.11 (osd.11) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:30 smithi169 bash[19604]: cluster 2024-02-12T16:41:29.003102+0000 mon.a (mon.0) 3034 : cluster [WRN] Health check update: Degraded data redundancy: 7129/44970 objects degraded (15.853%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:30 smithi169 bash[19604]: cluster 2024-02-12T16:41:29.014729+0000 mon.a (mon.0) 3035 : cluster [DBG] osdmap e894: 12 total, 11 up, 12 in 2024-02-12T16:41:30.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:30.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:30 smithi121 bash[18489]: cluster 2024-02-12T16:41:29.003102+0000 mon.a (mon.0) 3034 : cluster [WRN] Health check update: Degraded data redundancy: 7129/44970 objects degraded (15.853%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:30.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:30 smithi121 bash[18489]: cluster 2024-02-12T16:41:29.014729+0000 mon.a (mon.0) 3035 : cluster [DBG] osdmap e894: 12 total, 11 up, 12 in 2024-02-12T16:41:30.371 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:30.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:30 smithi123 bash[19119]: cluster 2024-02-12T16:41:28.528197+0000 osd.11 (osd.11) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:30 smithi123 bash[19119]: cluster 2024-02-12T16:41:28.528246+0000 osd.11 (osd.11) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:30 smithi123 bash[19119]: cluster 2024-02-12T16:41:29.003102+0000 mon.a (mon.0) 3034 : cluster [WRN] Health check update: Degraded data redundancy: 7129/44970 objects degraded (15.853%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:30 smithi123 bash[19119]: cluster 2024-02-12T16:41:29.014729+0000 mon.a (mon.0) 3035 : cluster [DBG] osdmap e894: 12 total, 11 up, 12 in 2024-02-12T16:41:30.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:30.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:30.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.644 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:30.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:30.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:30.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:30.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:30.893 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:41:30.893 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:41:30.921 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:30.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:31.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:31.051 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:31.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:31.190 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:31.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:31 smithi169 bash[19604]: cluster 2024-02-12T16:41:29.297027+0000 osd.8 (osd.8) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:31 smithi169 bash[19604]: cluster 2024-02-12T16:41:29.297100+0000 osd.8 (osd.8) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:31 smithi169 bash[19604]: cluster 2024-02-12T16:41:30.019494+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e895: 12 total, 11 up, 12 in 2024-02-12T16:41:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:31 smithi169 bash[19604]: cluster 2024-02-12T16:41:30.237749+0000 mgr.y (mgr.14146) 1715 : cluster [DBG] pgmap v2481: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 7129/40316 objects degraded (17.683%); 2717/40316 objects misplaced (6.739%); 13 MiB/s, 203 objects/s recovering 2024-02-12T16:41:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:31 smithi169 bash[19604]: cluster 2024-02-12T16:41:30.927153+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e896: 12 total, 11 up, 12 in 2024-02-12T16:41:31.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:31 smithi121 bash[18489]: cluster 2024-02-12T16:41:29.297027+0000 osd.8 (osd.8) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:31.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:31 smithi121 bash[18489]: cluster 2024-02-12T16:41:29.297100+0000 osd.8 (osd.8) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:31.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:31 smithi121 bash[18489]: cluster 2024-02-12T16:41:30.019494+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e895: 12 total, 11 up, 12 in 2024-02-12T16:41:31.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:31 smithi121 bash[18489]: cluster 2024-02-12T16:41:30.237749+0000 mgr.y (mgr.14146) 1715 : cluster [DBG] pgmap v2481: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 7129/40316 objects degraded (17.683%); 2717/40316 objects misplaced (6.739%); 13 MiB/s, 203 objects/s recovering 2024-02-12T16:41:31.303 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:31 smithi121 bash[18489]: cluster 2024-02-12T16:41:30.927153+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e896: 12 total, 11 up, 12 in 2024-02-12T16:41:31.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:31 smithi123 bash[19119]: cluster 2024-02-12T16:41:29.297027+0000 osd.8 (osd.8) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:31 smithi123 bash[19119]: cluster 2024-02-12T16:41:29.297100+0000 osd.8 (osd.8) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:31 smithi123 bash[19119]: cluster 2024-02-12T16:41:30.019494+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e895: 12 total, 11 up, 12 in 2024-02-12T16:41:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:31 smithi123 bash[19119]: cluster 2024-02-12T16:41:30.237749+0000 mgr.y (mgr.14146) 1715 : cluster [DBG] pgmap v2481: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 7129/40316 objects degraded (17.683%); 2717/40316 objects misplaced (6.739%); 13 MiB/s, 203 objects/s recovering 2024-02-12T16:41:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:31 smithi123 bash[19119]: cluster 2024-02-12T16:41:30.927153+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e896: 12 total, 11 up, 12 in 2024-02-12T16:41:31.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:31.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:31.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:31.593 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:31.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:31.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:31.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:31.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:31.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-12T16:41:31.971 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:31.994 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:31.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:32.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:32.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:32.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:32.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:32 smithi121 bash[18489]: cluster 2024-02-12T16:41:31.289084+0000 osd.5 (osd.5) 207 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:41:32.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:32.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:32.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:32.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:32 smithi169 bash[19604]: cluster 2024-02-12T16:41:31.289084+0000 osd.5 (osd.5) 207 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:41:32.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:32.381 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:32 smithi123 bash[19119]: cluster 2024-02-12T16:41:31.289084+0000 osd.5 (osd.5) 207 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:41:32.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:32.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:32.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:32.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:32.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:32.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:32.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:32.671 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:32.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:32.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:32.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:32.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:32.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:32.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:32.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:32.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:33.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:33.088 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:33.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:33.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:33.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:33.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:33 smithi169 bash[19604]: cluster 2024-02-12T16:41:31.381410+0000 osd.5 (osd.5) 208 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:41:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:33 smithi169 bash[19604]: cluster 2024-02-12T16:41:32.238500+0000 mgr.y (mgr.14146) 1716 : cluster [DBG] pgmap v2483: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s; 6592/34344 objects degraded (19.194%); 2086/34344 objects misplaced (6.074%); 25 MiB/s, 419 objects/s recovering 2024-02-12T16:41:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:33 smithi169 bash[19604]: audit 2024-02-12T16:41:32.605990+0000 mon.a (mon.0) 3038 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:41:33.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:33.378 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:33 smithi121 bash[18489]: cluster 2024-02-12T16:41:31.381410+0000 osd.5 (osd.5) 208 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:41:33.378 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:33 smithi121 bash[18489]: cluster 2024-02-12T16:41:32.238500+0000 mgr.y (mgr.14146) 1716 : cluster [DBG] pgmap v2483: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s; 6592/34344 objects degraded (19.194%); 2086/34344 objects misplaced (6.074%); 25 MiB/s, 419 objects/s recovering 2024-02-12T16:41:33.378 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:33 smithi121 bash[18489]: audit 2024-02-12T16:41:32.605990+0000 mon.a (mon.0) 3038 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:41:33.379 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:33.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:33 smithi123 bash[19119]: cluster 2024-02-12T16:41:31.381410+0000 osd.5 (osd.5) 208 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:41:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:33 smithi123 bash[19119]: cluster 2024-02-12T16:41:32.238500+0000 mgr.y (mgr.14146) 1716 : cluster [DBG] pgmap v2483: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.42k op/s; 6592/34344 objects degraded (19.194%); 2086/34344 objects misplaced (6.074%); 25 MiB/s, 419 objects/s recovering 2024-02-12T16:41:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:33 smithi123 bash[19119]: audit 2024-02-12T16:41:32.605990+0000 mon.a (mon.0) 3038 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:41:33.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:33.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:33.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:33.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:33.639 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:33.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:33.648 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:41:33.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:41:33.724 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:41:33.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:33.769 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:41:33.779 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:33.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:33.799 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:41:33.833 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:41:33.859 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:41:33.885 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:33.907 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:41:33.908 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:33.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:33.948 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:41:33.977 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:41:34.008 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:41:34.008 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.008 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.008 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.008 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.009 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:41:34.011 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:34.038 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:34.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:34.168 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:34.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:34.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:34.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:34 smithi121 bash[18489]: audit 2024-02-12T16:41:33.143898+0000 mon.a (mon.0) 3039 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:34 smithi121 bash[18489]: cluster 2024-02-12T16:41:33.320015+0000 osd.6 (osd.6) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:34 smithi121 bash[18489]: cluster 2024-02-12T16:41:33.320072+0000 osd.6 (osd.6) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:34.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:34 smithi121 bash[18489]: audit 2024-02-12T16:41:33.618616+0000 mon.a (mon.0) 3040 : audit [DBG] from='client.? v1:172.21.15.121:0/3698752993' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:41:34.425 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:34.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:34.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:34 smithi169 bash[19604]: audit 2024-02-12T16:41:33.143898+0000 mon.a (mon.0) 3039 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:34.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:34 smithi169 bash[19604]: cluster 2024-02-12T16:41:33.320015+0000 osd.6 (osd.6) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:34.537 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:34 smithi169 bash[19604]: cluster 2024-02-12T16:41:33.320072+0000 osd.6 (osd.6) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:34.538 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:34 smithi169 bash[19604]: audit 2024-02-12T16:41:33.618616+0000 mon.a (mon.0) 3040 : audit [DBG] from='client.? v1:172.21.15.121:0/3698752993' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:41:34.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:34 smithi123 bash[19119]: audit 2024-02-12T16:41:33.143898+0000 mon.a (mon.0) 3039 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:34 smithi123 bash[19119]: cluster 2024-02-12T16:41:33.320015+0000 osd.6 (osd.6) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:34 smithi123 bash[19119]: cluster 2024-02-12T16:41:33.320072+0000 osd.6 (osd.6) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:34 smithi123 bash[19119]: audit 2024-02-12T16:41:33.618616+0000 mon.a (mon.0) 3040 : audit [DBG] from='client.? v1:172.21.15.121:0/3698752993' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:41:34.663 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:34.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:34.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:34.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:34.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:34.946 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:34.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:35.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:35.078 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:35.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:35.171 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:41:35.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:35.190 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:35 smithi121 bash[18489]: cluster 2024-02-12T16:41:33.770089+0000 osd.3 (osd.3) 107 : cluster [DBG] 1.3 deep-scrub starts 2024-02-12T16:41:35.190 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:35 smithi121 bash[18489]: cluster 2024-02-12T16:41:33.930330+0000 osd.3 (osd.3) 108 : cluster [DBG] 1.3 deep-scrub ok 2024-02-12T16:41:35.190 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:35 smithi121 bash[18489]: cluster 2024-02-12T16:41:34.152413+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e897: 12 total, 11 up, 12 in 2024-02-12T16:41:35.190 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:35 smithi121 bash[18489]: audit 2024-02-12T16:41:34.211655+0000 mon.a (mon.0) 3042 : audit [INF] from='client.26972 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:41:35.190 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:35 smithi121 bash[18489]: audit 2024-02-12T16:41:34.213885+0000 mon.b (mon.2) 217 : audit [INF] from='client.? v1:172.21.15.121:0/3868402179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:41:35.190 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:35 smithi121 bash[18489]: cluster 2024-02-12T16:41:34.238978+0000 mgr.y (mgr.14146) 1717 : cluster [DBG] pgmap v2485: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 968 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 6592/30986 objects degraded (21.274%); 2086/30986 objects misplaced (6.732%); 14 MiB/s, 247 objects/s recovering 2024-02-12T16:41:35.213 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:35.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:35.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:35.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:35.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:35.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:35.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:35 smithi123 bash[19119]: cluster 2024-02-12T16:41:33.770089+0000 osd.3 (osd.3) 107 : cluster [DBG] 1.3 deep-scrub starts 2024-02-12T16:41:35.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:35 smithi123 bash[19119]: cluster 2024-02-12T16:41:33.930330+0000 osd.3 (osd.3) 108 : cluster [DBG] 1.3 deep-scrub ok 2024-02-12T16:41:35.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:35 smithi123 bash[19119]: cluster 2024-02-12T16:41:34.152413+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e897: 12 total, 11 up, 12 in 2024-02-12T16:41:35.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:35 smithi123 bash[19119]: audit 2024-02-12T16:41:34.211655+0000 mon.a (mon.0) 3042 : audit [INF] from='client.26972 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:41:35.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:35 smithi123 bash[19119]: audit 2024-02-12T16:41:34.213885+0000 mon.b (mon.2) 217 : audit [INF] from='client.? v1:172.21.15.121:0/3868402179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:41:35.468 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:35 smithi123 bash[19119]: cluster 2024-02-12T16:41:34.238978+0000 mgr.y (mgr.14146) 1717 : cluster [DBG] pgmap v2485: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 968 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 6592/30986 objects degraded (21.274%); 2086/30986 objects misplaced (6.732%); 14 MiB/s, 247 objects/s recovering 2024-02-12T16:41:35.489 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:35.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:35 smithi169 bash[19604]: cluster 2024-02-12T16:41:33.770089+0000 osd.3 (osd.3) 107 : cluster [DBG] 1.3 deep-scrub starts 2024-02-12T16:41:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:35 smithi169 bash[19604]: cluster 2024-02-12T16:41:33.930330+0000 osd.3 (osd.3) 108 : cluster [DBG] 1.3 deep-scrub ok 2024-02-12T16:41:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:35 smithi169 bash[19604]: cluster 2024-02-12T16:41:34.152413+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e897: 12 total, 11 up, 12 in 2024-02-12T16:41:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:35 smithi169 bash[19604]: audit 2024-02-12T16:41:34.211655+0000 mon.a (mon.0) 3042 : audit [INF] from='client.26972 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:41:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:35 smithi169 bash[19604]: audit 2024-02-12T16:41:34.213885+0000 mon.b (mon.2) 217 : audit [INF] from='client.? v1:172.21.15.121:0/3868402179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:41:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:35 smithi169 bash[19604]: cluster 2024-02-12T16:41:34.238978+0000 mgr.y (mgr.14146) 1717 : cluster [DBG] pgmap v2485: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 968 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 6592/30986 objects degraded (21.274%); 2086/30986 objects misplaced (6.732%); 14 MiB/s, 247 objects/s recovering 2024-02-12T16:41:35.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:35.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:35.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:35.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:35.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:35.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:35.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:35.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:35.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:35.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:36.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:36.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:36.137 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:36.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:36.272 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:36.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:36.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:36 smithi123 bash[19119]: cluster 2024-02-12T16:41:35.150288+0000 mon.a (mon.0) 3043 : cluster [WRN] Health check update: Degraded data redundancy: 6592/30986 objects degraded (21.274%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:36.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:36 smithi123 bash[19119]: cluster 2024-02-12T16:41:35.151013+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:41:36.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:36 smithi123 bash[19119]: audit 2024-02-12T16:41:35.162201+0000 mon.a (mon.0) 3045 : audit [INF] from='client.26972 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:41:36.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:36 smithi123 bash[19119]: cluster 2024-02-12T16:41:35.162258+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e898: 12 total, 11 up, 12 in 2024-02-12T16:41:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:36 smithi121 bash[18489]: cluster 2024-02-12T16:41:35.150288+0000 mon.a (mon.0) 3043 : cluster [WRN] Health check update: Degraded data redundancy: 6592/30986 objects degraded (21.274%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:36 smithi121 bash[18489]: cluster 2024-02-12T16:41:35.151013+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:41:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:36 smithi121 bash[18489]: audit 2024-02-12T16:41:35.162201+0000 mon.a (mon.0) 3045 : audit [INF] from='client.26972 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:41:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:36 smithi121 bash[18489]: cluster 2024-02-12T16:41:35.162258+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e898: 12 total, 11 up, 12 in 2024-02-12T16:41:36.413 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:36.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:36.538 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:36 smithi169 bash[19604]: cluster 2024-02-12T16:41:35.150288+0000 mon.a (mon.0) 3043 : cluster [WRN] Health check update: Degraded data redundancy: 6592/30986 objects degraded (21.274%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:36 smithi169 bash[19604]: cluster 2024-02-12T16:41:35.151013+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:41:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:36 smithi169 bash[19604]: audit 2024-02-12T16:41:35.162201+0000 mon.a (mon.0) 3045 : audit [INF] from='client.26972 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:41:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:36 smithi169 bash[19604]: cluster 2024-02-12T16:41:35.162258+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e898: 12 total, 11 up, 12 in 2024-02-12T16:41:36.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:36.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:36.754 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:41:36.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:36.801 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:36.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:36.934 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:36.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:37.037 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:37.063 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:37.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:37.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:37.296 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:37.303 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:37 smithi123 bash[19119]: cluster 2024-02-12T16:41:36.239573+0000 mgr.y (mgr.14146) 1718 : cluster [DBG] pgmap v2487: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 927 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 6592/29680 objects degraded (22.210%); 1763/29680 objects misplaced (5.940%); 12 MiB/s, 215 objects/s recovering 2024-02-12T16:41:37.320 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:41:37.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:41:37.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:37.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:37 smithi121 bash[18489]: cluster 2024-02-12T16:41:36.239573+0000 mgr.y (mgr.14146) 1718 : cluster [DBG] pgmap v2487: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 927 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 6592/29680 objects degraded (22.210%); 1763/29680 objects misplaced (5.940%); 12 MiB/s, 215 objects/s recovering 2024-02-12T16:41:37.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:37 smithi169 bash[19604]: cluster 2024-02-12T16:41:36.239573+0000 mgr.y (mgr.14146) 1718 : cluster [DBG] pgmap v2487: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 927 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 6592/29680 objects degraded (22.210%); 1763/29680 objects misplaced (5.940%); 12 MiB/s, 215 objects/s recovering 2024-02-12T16:41:37.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:37.597 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:37.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:37.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:37.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:37.863 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:37.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:37.988 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:37.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:38.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:38.126 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:38.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:38.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:38.257 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:38.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:38.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:38.388 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:38.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:38.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:38.513 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:38.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:38.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:38.647 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:38.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:38.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:38.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:38.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:38.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:38.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:38.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:39.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:39.071 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:39.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:39.200 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:39.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:39.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:41:39.328 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:39.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:39.436 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:39 smithi169 bash[19604]: cluster 2024-02-12T16:41:38.240230+0000 mgr.y (mgr.14146) 1719 : cluster [DBG] pgmap v2488: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 742 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 6073/23744 objects degraded (25.577%); 1438/23744 objects misplaced (6.056%); 17 MiB/s, 367 objects/s recovering 2024-02-12T16:41:39.436 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:39 smithi169 bash[19604]: cluster 2024-02-12T16:41:38.494436+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:39.436 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:39 smithi169 bash[19604]: cluster 2024-02-12T16:41:38.494535+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:39.436 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:39 smithi169 bash[19604]: audit 2024-02-12T16:41:39.104234+0000 mon.a (mon.0) 3047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:39.457 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:39.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:39.569 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:39 smithi123 bash[19119]: cluster 2024-02-12T16:41:38.240230+0000 mgr.y (mgr.14146) 1719 : cluster [DBG] pgmap v2488: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 742 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 6073/23744 objects degraded (25.577%); 1438/23744 objects misplaced (6.056%); 17 MiB/s, 367 objects/s recovering 2024-02-12T16:41:39.569 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:39 smithi123 bash[19119]: cluster 2024-02-12T16:41:38.494436+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:39.569 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:39 smithi123 bash[19119]: cluster 2024-02-12T16:41:38.494535+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:39.569 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:39 smithi123 bash[19119]: audit 2024-02-12T16:41:39.104234+0000 mon.a (mon.0) 3047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:39.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:41:39.585 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:39.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:39 smithi121 bash[18489]: cluster 2024-02-12T16:41:38.240230+0000 mgr.y (mgr.14146) 1719 : cluster [DBG] pgmap v2488: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 742 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 6073/23744 objects degraded (25.577%); 1438/23744 objects misplaced (6.056%); 17 MiB/s, 367 objects/s recovering 2024-02-12T16:41:39.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:39 smithi121 bash[18489]: cluster 2024-02-12T16:41:38.494436+0000 osd.3 (osd.3) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:39.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:39 smithi121 bash[18489]: cluster 2024-02-12T16:41:38.494535+0000 osd.3 (osd.3) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:39.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:39 smithi121 bash[18489]: audit 2024-02-12T16:41:39.104234+0000 mon.a (mon.0) 3047 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:39.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:39.733 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:39.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:39.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:39.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:39.993 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:39.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:40.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:40.120 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:40.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:40.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:40.255 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:40.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:40.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:40.385 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:40.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:40 smithi121 bash[18489]: cluster 2024-02-12T16:41:39.296716+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e899: 12 total, 11 up, 12 in 2024-02-12T16:41:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:40 smithi121 bash[18489]: cluster 2024-02-12T16:41:39.359653+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:40 smithi121 bash[18489]: cluster 2024-02-12T16:41:39.359713+0000 osd.0 (osd.0) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:40 smithi121 bash[18489]: audit 2024-02-12T16:41:39.451851+0000 mon.a (mon.0) 3049 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:41:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:40 smithi121 bash[18489]: audit 2024-02-12T16:41:39.452653+0000 mon.a (mon.0) 3050 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:41:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:40 smithi121 bash[18489]: audit 2024-02-12T16:41:39.459205+0000 mon.a (mon.0) 3051 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:40.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:40.516 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:40.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:40.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:40.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:40 smithi169 bash[19604]: cluster 2024-02-12T16:41:39.296716+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e899: 12 total, 11 up, 12 in 2024-02-12T16:41:40.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:40 smithi169 bash[19604]: cluster 2024-02-12T16:41:39.359653+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:40.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:40 smithi169 bash[19604]: cluster 2024-02-12T16:41:39.359713+0000 osd.0 (osd.0) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:40.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:40 smithi169 bash[19604]: audit 2024-02-12T16:41:39.451851+0000 mon.a (mon.0) 3049 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:41:40.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:40 smithi169 bash[19604]: audit 2024-02-12T16:41:39.452653+0000 mon.a (mon.0) 3050 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:41:40.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:40 smithi169 bash[19604]: audit 2024-02-12T16:41:39.459205+0000 mon.a (mon.0) 3051 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:40 smithi123 bash[19119]: cluster 2024-02-12T16:41:39.296716+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e899: 12 total, 11 up, 12 in 2024-02-12T16:41:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:40 smithi123 bash[19119]: cluster 2024-02-12T16:41:39.359653+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:40 smithi123 bash[19119]: cluster 2024-02-12T16:41:39.359713+0000 osd.0 (osd.0) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:40 smithi123 bash[19119]: audit 2024-02-12T16:41:39.451851+0000 mon.a (mon.0) 3049 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:41:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:40 smithi123 bash[19119]: audit 2024-02-12T16:41:39.452653+0000 mon.a (mon.0) 3050 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:41:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:40 smithi123 bash[19119]: audit 2024-02-12T16:41:39.459205+0000 mon.a (mon.0) 3051 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:40.651 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:40.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:40.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:40.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:40.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:40.910 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:40.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:41.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:41.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:41.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:41.178 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:41.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:41.304 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:41.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:41.473 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.3 to 0.974998 (f999) 2024-02-12T16:41:41.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:41 smithi169 bash[19604]: cluster 2024-02-12T16:41:40.240728+0000 mgr.y (mgr.14146) 1720 : cluster [DBG] pgmap v2490: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 636 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s; 6073/20366 objects degraded (29.819%); 1438/20366 objects misplaced (7.061%); 8.7 MiB/s, 229 objects/s recovering 2024-02-12T16:41:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:41 smithi169 bash[19604]: cluster 2024-02-12T16:41:40.307762+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e900: 12 total, 11 up, 12 in 2024-02-12T16:41:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:41 smithi169 bash[19604]: cluster 2024-02-12T16:41:40.456097+0000 mon.a (mon.0) 3053 : cluster [WRN] Health check update: Degraded data redundancy: 6073/20366 objects degraded (29.819%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:41 smithi169 bash[19604]: audit 2024-02-12T16:41:40.478348+0000 mon.a (mon.0) 3054 : audit [INF] from='client.26978 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]: dispatch 2024-02-12T16:41:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:41 smithi169 bash[19604]: audit 2024-02-12T16:41:40.480603+0000 mon.b (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.121:0/3688294025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]: dispatch 2024-02-12T16:41:41.575 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:41 smithi123 bash[19119]: cluster 2024-02-12T16:41:40.240728+0000 mgr.y (mgr.14146) 1720 : cluster [DBG] pgmap v2490: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 636 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s; 6073/20366 objects degraded (29.819%); 1438/20366 objects misplaced (7.061%); 8.7 MiB/s, 229 objects/s recovering 2024-02-12T16:41:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:41 smithi123 bash[19119]: cluster 2024-02-12T16:41:40.307762+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e900: 12 total, 11 up, 12 in 2024-02-12T16:41:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:41 smithi123 bash[19119]: cluster 2024-02-12T16:41:40.456097+0000 mon.a (mon.0) 3053 : cluster [WRN] Health check update: Degraded data redundancy: 6073/20366 objects degraded (29.819%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:41 smithi123 bash[19119]: audit 2024-02-12T16:41:40.478348+0000 mon.a (mon.0) 3054 : audit [INF] from='client.26978 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]: dispatch 2024-02-12T16:41:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:41 smithi123 bash[19119]: audit 2024-02-12T16:41:40.480603+0000 mon.b (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.121:0/3688294025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]: dispatch 2024-02-12T16:41:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:41 smithi121 bash[18489]: cluster 2024-02-12T16:41:40.240728+0000 mgr.y (mgr.14146) 1720 : cluster [DBG] pgmap v2490: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 636 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s; 6073/20366 objects degraded (29.819%); 1438/20366 objects misplaced (7.061%); 8.7 MiB/s, 229 objects/s recovering 2024-02-12T16:41:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:41 smithi121 bash[18489]: cluster 2024-02-12T16:41:40.307762+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e900: 12 total, 11 up, 12 in 2024-02-12T16:41:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:41 smithi121 bash[18489]: cluster 2024-02-12T16:41:40.456097+0000 mon.a (mon.0) 3053 : cluster [WRN] Health check update: Degraded data redundancy: 6073/20366 objects degraded (29.819%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:41 smithi121 bash[18489]: audit 2024-02-12T16:41:40.478348+0000 mon.a (mon.0) 3054 : audit [INF] from='client.26978 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]: dispatch 2024-02-12T16:41:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:41 smithi121 bash[18489]: audit 2024-02-12T16:41:40.480603+0000 mon.b (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.121:0/3688294025' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]: dispatch 2024-02-12T16:41:41.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:41.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:41.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:41.848 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:41.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:41.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:41.977 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:41.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:160084 8.c 8:3f465a46:::benchmark_data_smithi169_115677_object77456:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.002961+0000", 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.013938485, 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013969947, 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:42.021 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 160084 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.002961+0000", 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.002961+0000", 2024-02-12T16:41:42.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.002958+0000", 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1008e-05 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.002969+0000", 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6208000000000001e-05 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:42.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.002985+0000", 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7510000000000008e-06 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.002993+0000", 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8162000000000003e-05 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.003051+0000", 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012712499999999999 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:41:42.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.003178+0000", 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1709000000000001e-05 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.003250+0000", 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096252160000000007 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.012875+0000", 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0099139009999999993 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:160090 8.c 8:3069e953:::benchmark_data_smithi169_115677_object77462:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.007629+0000", 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0092704940000000007, 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0093973040000000004, 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 160090 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:42.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.007629+0000", 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.007629+0000", 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.007626+0000", 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1790000000000008e-06 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:42.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.007635+0000", 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4742999999999999e-05 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.007649+0000", 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6399999999999997e-06 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.007657+0000", 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048550200000000001 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:41:42.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.008142+0000", 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012361800000000001 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.008266+0000", 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000149006 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.008415+0000", 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078618600000000002 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:160091 8.c 8:33773bf4:::benchmark_data_smithi169_115677_object77463:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.009355+0000", 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.0075439829999999998, 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077474199999999997, 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "waiting for sub ops", 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 160091 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:42.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.009355+0000", 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.009355+0000", 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.009353+0000", 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0979999999999996e-06 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:42.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.009360+0000", 2024-02-12T16:41:42.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1368e-05 2024-02-12T16:41:42.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.009371+0000", 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1620000000000002e-06 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.009378+0000", 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0035389810000000001 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.012917+0000", 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015935100000000001 2024-02-12T16:41:42.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013077+0000", 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014896000000000001 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013226+0000", 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038702989999999998 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.034 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop_reply(client.26785.0:160084 8.c e901/839)", 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.013662+0000", 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.003237406, 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003513609, 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "queued for pg", 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013662+0000", 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013662+0000", 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:41:42.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013660+0000", 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8379999999999996e-06 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013667+0000", 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4105e-05 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013681+0000", 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4379999999999998e-06 2024-02-12T16:41:42.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.013689+0000", 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6628e-05 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:160093 8.c 8.87698d5c (undecoded) ondisk+write+known_if_redirected e901)", 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.016440+0000", 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.00045919400000000001, 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078367900000000002, 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "queued for pg", 2024-02-12T16:41:42.037 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 160093 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.016440+0000", 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.016440+0000", 2024-02-12T16:41:42.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.016437+0000", 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.219e-05 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.016449+0000", 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0066999999999999e-05 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:41:42.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.016469+0000", 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3250000000000008e-06 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.016477+0000", 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7205e-05 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: ], 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 5 2024-02-12T16:41:42.040 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:41:42.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:42.105 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:42.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:42.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:42.235 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:42.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:42.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:42.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:42.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:42.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:42.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:42.472 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:42 smithi169 bash[19604]: cluster 2024-02-12T16:41:41.387551+0000 osd.9 (osd.9) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:42.473 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:42 smithi169 bash[19604]: cluster 2024-02-12T16:41:41.387629+0000 osd.9 (osd.9) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:42.473 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:42 smithi169 bash[19604]: audit 2024-02-12T16:41:41.461794+0000 mon.a (mon.0) 3055 : audit [INF] from='client.26978 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]': finished 2024-02-12T16:41:42.473 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:42 smithi169 bash[19604]: cluster 2024-02-12T16:41:41.461942+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e901: 12 total, 11 up, 12 in 2024-02-12T16:41:42.499 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:42.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:42.602 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:42.632 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:42.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:42 smithi123 bash[19119]: cluster 2024-02-12T16:41:41.387551+0000 osd.9 (osd.9) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:42 smithi123 bash[19119]: cluster 2024-02-12T16:41:41.387629+0000 osd.9 (osd.9) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:42 smithi123 bash[19119]: audit 2024-02-12T16:41:41.461794+0000 mon.a (mon.0) 3055 : audit [INF] from='client.26978 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]': finished 2024-02-12T16:41:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:42 smithi123 bash[19119]: cluster 2024-02-12T16:41:41.461942+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e901: 12 total, 11 up, 12 in 2024-02-12T16:41:42.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:42.759 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:42.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:42.779 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:42 smithi121 bash[18489]: cluster 2024-02-12T16:41:41.387551+0000 osd.9 (osd.9) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:42.780 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:42 smithi121 bash[18489]: cluster 2024-02-12T16:41:41.387629+0000 osd.9 (osd.9) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:42.780 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:42 smithi121 bash[18489]: audit 2024-02-12T16:41:41.461794+0000 mon.a (mon.0) 3055 : audit [INF] from='client.26978 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.9749978270940473}]': finished 2024-02-12T16:41:42.780 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:42 smithi121 bash[18489]: cluster 2024-02-12T16:41:41.461942+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e901: 12 total, 11 up, 12 in 2024-02-12T16:41:42.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:42.887 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:42.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:42.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:43.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-12T16:41:43.027 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:43.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:43.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:43.300 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:43.404 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:43.427 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:43.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:43.569 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:43.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:43.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:43 smithi169 bash[19604]: cluster 2024-02-12T16:41:42.241364+0000 mgr.y (mgr.14146) 1721 : cluster [DBG] pgmap v2493: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 420 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s; 5585/13448 objects degraded (41.530%); 812/13448 objects misplaced (6.038%); 17 MiB/s, 482 objects/s recovering 2024-02-12T16:41:43.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:43.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:43 smithi121 bash[18489]: cluster 2024-02-12T16:41:42.241364+0000 mgr.y (mgr.14146) 1721 : cluster [DBG] pgmap v2493: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 420 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s; 5585/13448 objects degraded (41.530%); 812/13448 objects misplaced (6.038%); 17 MiB/s, 482 objects/s recovering 2024-02-12T16:41:43.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:43 smithi123 bash[19119]: cluster 2024-02-12T16:41:42.241364+0000 mgr.y (mgr.14146) 1721 : cluster [DBG] pgmap v2493: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 420 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s; 5585/13448 objects degraded (41.530%); 812/13448 objects misplaced (6.038%); 17 MiB/s, 482 objects/s recovering 2024-02-12T16:41:43.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:43.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:43.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:44.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:44.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:44.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:44.221 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:44.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:44.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:44.352 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:44.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:44.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:44.482 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:44.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:44.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:44.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:44.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:44.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:44.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:44.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:44.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:44.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:44.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:44.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:45.000 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:45.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:45.136 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:45.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:45.271 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:45.284 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:45.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:45.413 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:45.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:45.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:45 smithi121 bash[18489]: cluster 2024-02-12T16:41:44.241945+0000 mgr.y (mgr.14146) 1722 : cluster [DBG] pgmap v2494: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 317 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 5585/10134 objects degraded (55.112%); 812/10134 objects misplaced (8.013%); 8.6 MiB/s, 249 objects/s recovering 2024-02-12T16:41:45.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:45 smithi121 bash[18489]: cluster 2024-02-12T16:41:44.476719+0000 mon.a (mon.0) 3057 : cluster [DBG] osdmap e902: 12 total, 11 up, 12 in 2024-02-12T16:41:45.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:45 smithi121 bash[18489]: audit 2024-02-12T16:41:44.924877+0000 mon.a (mon.0) 3058 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:45.546 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:45.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:45.675 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:45.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:45.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:45 smithi169 bash[19604]: cluster 2024-02-12T16:41:44.241945+0000 mgr.y (mgr.14146) 1722 : cluster [DBG] pgmap v2494: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 317 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 5585/10134 objects degraded (55.112%); 812/10134 objects misplaced (8.013%); 8.6 MiB/s, 249 objects/s recovering 2024-02-12T16:41:45.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:45 smithi169 bash[19604]: cluster 2024-02-12T16:41:44.476719+0000 mon.a (mon.0) 3057 : cluster [DBG] osdmap e902: 12 total, 11 up, 12 in 2024-02-12T16:41:45.785 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:45 smithi169 bash[19604]: audit 2024-02-12T16:41:44.924877+0000 mon.a (mon.0) 3058 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:45.807 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:45 smithi123 bash[19119]: cluster 2024-02-12T16:41:44.241945+0000 mgr.y (mgr.14146) 1722 : cluster [DBG] pgmap v2494: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 317 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 5585/10134 objects degraded (55.112%); 812/10134 objects misplaced (8.013%); 8.6 MiB/s, 249 objects/s recovering 2024-02-12T16:41:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:45 smithi123 bash[19119]: cluster 2024-02-12T16:41:44.476719+0000 mon.a (mon.0) 3057 : cluster [DBG] osdmap e902: 12 total, 11 up, 12 in 2024-02-12T16:41:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:45 smithi123 bash[19119]: audit 2024-02-12T16:41:44.924877+0000 mon.a (mon.0) 3058 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:41:45.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:45.940 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:45.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:46.047 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:46.071 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:46.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:46.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:46.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:46.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:46.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:46.310 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:46.334 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:46.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:46.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:46.480 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:46.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:46.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:46.592 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:46 smithi123 bash[19119]: cluster 2024-02-12T16:41:45.481687+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e903: 12 total, 11 up, 12 in 2024-02-12T16:41:46.592 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:46 smithi123 bash[19119]: cluster 2024-02-12T16:41:45.925423+0000 mon.a (mon.0) 3060 : cluster [WRN] Health check update: Degraded data redundancy: 5585/10134 objects degraded (55.112%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:46.592 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:46 smithi123 bash[19119]: cluster 2024-02-12T16:41:45.931267+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e904: 12 total, 11 up, 12 in 2024-02-12T16:41:46.608 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:46.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:46.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:46.737 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:46.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:46 smithi169 bash[19604]: cluster 2024-02-12T16:41:45.481687+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e903: 12 total, 11 up, 12 in 2024-02-12T16:41:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:46 smithi169 bash[19604]: cluster 2024-02-12T16:41:45.925423+0000 mon.a (mon.0) 3060 : cluster [WRN] Health check update: Degraded data redundancy: 5585/10134 objects degraded (55.112%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:46 smithi169 bash[19604]: cluster 2024-02-12T16:41:45.931267+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e904: 12 total, 11 up, 12 in 2024-02-12T16:41:46.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:46.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:46.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:46 smithi121 bash[18489]: cluster 2024-02-12T16:41:45.481687+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e903: 12 total, 11 up, 12 in 2024-02-12T16:41:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:46 smithi121 bash[18489]: cluster 2024-02-12T16:41:45.925423+0000 mon.a (mon.0) 3060 : cluster [WRN] Health check update: Degraded data redundancy: 5585/10134 objects degraded (55.112%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:46 smithi121 bash[18489]: cluster 2024-02-12T16:41:45.931267+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e904: 12 total, 11 up, 12 in 2024-02-12T16:41:46.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:46.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:47.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:47.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:47.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:47.263 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:47.337 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 82625 objects 2024-02-12T16:41:47.338 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :63.017420 2024-02-12T16:41:47.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:47.401 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:47.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:47.532 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:47.635 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:47 smithi123 bash[19119]: cluster 2024-02-12T16:41:46.242528+0000 mgr.y (mgr.14146) 1723 : cluster [DBG] pgmap v2498: 57 pgs: 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 271 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 5585/8670 objects degraded (64.418%); 497/8670 objects misplaced (5.732%); 11 MiB/s, 382 objects/s recovering 2024-02-12T16:41:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:47 smithi121 bash[18489]: cluster 2024-02-12T16:41:46.242528+0000 mgr.y (mgr.14146) 1723 : cluster [DBG] pgmap v2498: 57 pgs: 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 271 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 5585/8670 objects degraded (64.418%); 497/8670 objects misplaced (5.732%); 11 MiB/s, 382 objects/s recovering 2024-02-12T16:41:47.671 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:47.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:47.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8, 11] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3] 2024-02-12T16:41:47.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:41:47.781 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.5543352723013449 2024-02-12T16:41:47.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 3 0.5543352723013449 2024-02-12T16:41:47.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:47 smithi169 bash[19604]: cluster 2024-02-12T16:41:46.242528+0000 mgr.y (mgr.14146) 1723 : cluster [DBG] pgmap v2498: 57 pgs: 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 2 active+recovery_wait+degraded+remapped, 44 active+clean; 271 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 5585/8670 objects degraded (64.418%); 497/8670 objects misplaced (5.732%); 11 MiB/s, 382 objects/s recovering 2024-02-12T16:41:47.810 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:47.842 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:41:47.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:47.951 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:47.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:48.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:48.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:48.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:48.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:48.223 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:48.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:48.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:48.348 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:48.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:48.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:48.487 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:48.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:48.591 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:48.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:48.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:48.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:48.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:48.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:48.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:48.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:48.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:49.029 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:49.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:49.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:49.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:49.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:49.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:49.310 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:49.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:49.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:49.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:49.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:49.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:49.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:49.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:49.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:49.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:49 smithi121 bash[18489]: cluster 2024-02-12T16:41:48.243410+0000 mgr.y (mgr.14146) 1724 : cluster [DBG] pgmap v2499: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 44 active+clean; 109 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 3416/3502 objects degraded (97.544%); 127/3502 objects misplaced (3.626%); 8.7 MiB/s, 265 objects/s recovering 2024-02-12T16:41:49.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:49.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:49 smithi169 bash[19604]: cluster 2024-02-12T16:41:48.243410+0000 mgr.y (mgr.14146) 1724 : cluster [DBG] pgmap v2499: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 44 active+clean; 109 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 3416/3502 objects degraded (97.544%); 127/3502 objects misplaced (3.626%); 8.7 MiB/s, 265 objects/s recovering 2024-02-12T16:41:49.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:49.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:49 smithi123 bash[19119]: cluster 2024-02-12T16:41:48.243410+0000 mgr.y (mgr.14146) 1724 : cluster [DBG] pgmap v2499: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 44 active+clean; 109 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 3416/3502 objects degraded (97.544%); 127/3502 objects misplaced (3.626%); 8.7 MiB/s, 265 objects/s recovering 2024-02-12T16:41:49.970 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:50.012 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:50.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:50.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:50.256 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:50.280 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:50.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:50.417 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:50.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:50 smithi121 bash[18489]: cluster 2024-02-12T16:41:49.764550+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:50 smithi121 bash[18489]: cluster 2024-02-12T16:41:49.764627+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:50.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:50.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:50 smithi169 bash[19604]: cluster 2024-02-12T16:41:49.764550+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:50 smithi169 bash[19604]: cluster 2024-02-12T16:41:49.764627+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:50.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:50 smithi123 bash[19119]: cluster 2024-02-12T16:41:49.764550+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:41:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:50 smithi123 bash[19119]: cluster 2024-02-12T16:41:49.764627+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:41:50.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:50.983 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:50.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:51.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:51.119 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:51.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:51.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:51.253 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:51.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:51.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:51.378 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:51.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:51.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:51.520 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:51.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:51.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:51.666 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:51.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:51.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:51.778 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:51 smithi169 bash[19604]: cluster 2024-02-12T16:41:50.244256+0000 mgr.y (mgr.14146) 1725 : cluster [DBG] pgmap v2500: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 44 active+clean; 49 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 3416/1586 objects degraded (215.385%); 127/1586 objects misplaced (8.008%); 8.7 MiB/s, 265 objects/s recovering 2024-02-12T16:41:51.778 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:51 smithi169 bash[19604]: cluster 2024-02-12T16:41:50.504230+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e905: 12 total, 11 up, 12 in 2024-02-12T16:41:51.778 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:51 smithi169 bash[19604]: cluster 2024-02-12T16:41:50.927428+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: Degraded data redundancy: 3416/1586 objects degraded (215.385%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:51.778 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:51 smithi169 bash[19604]: cluster 2024-02-12T16:41:50.935141+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e906: 12 total, 11 up, 12 in 2024-02-12T16:41:51.797 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:51.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:51 smithi123 bash[19119]: cluster 2024-02-12T16:41:50.244256+0000 mgr.y (mgr.14146) 1725 : cluster [DBG] pgmap v2500: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 44 active+clean; 49 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 3416/1586 objects degraded (215.385%); 127/1586 objects misplaced (8.008%); 8.7 MiB/s, 265 objects/s recovering 2024-02-12T16:41:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:51 smithi123 bash[19119]: cluster 2024-02-12T16:41:50.504230+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e905: 12 total, 11 up, 12 in 2024-02-12T16:41:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:51 smithi123 bash[19119]: cluster 2024-02-12T16:41:50.927428+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: Degraded data redundancy: 3416/1586 objects degraded (215.385%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:51 smithi123 bash[19119]: cluster 2024-02-12T16:41:50.935141+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e906: 12 total, 11 up, 12 in 2024-02-12T16:41:51.901 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:51 smithi121 bash[18489]: cluster 2024-02-12T16:41:50.244256+0000 mgr.y (mgr.14146) 1725 : cluster [DBG] pgmap v2500: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 44 active+clean; 49 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 3416/1586 objects degraded (215.385%); 127/1586 objects misplaced (8.008%); 8.7 MiB/s, 265 objects/s recovering 2024-02-12T16:41:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:51 smithi121 bash[18489]: cluster 2024-02-12T16:41:50.504230+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e905: 12 total, 11 up, 12 in 2024-02-12T16:41:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:51 smithi121 bash[18489]: cluster 2024-02-12T16:41:50.927428+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: Degraded data redundancy: 3416/1586 objects degraded (215.385%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:51 smithi121 bash[18489]: cluster 2024-02-12T16:41:50.935141+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e906: 12 total, 11 up, 12 in 2024-02-12T16:41:51.937 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:51.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:52.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:52.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:52.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:52.182 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:52.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:52.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:52.354 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:52.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:52.360 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:41:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:41:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:41:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:41:52.361 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:41:52.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:52.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:52.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:52.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:52.625 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:52.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:52.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:52.756 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:52.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:52.853 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-02-12T16:41:52.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-02-12T16:41:52.860 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:52.897 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:52.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:53.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:53.037 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:53.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-12T16:41:53.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:53.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:53.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:53.321 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:53.424 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:53.447 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:53.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:53.581 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:53 smithi123 bash[19119]: cluster 2024-02-12T16:41:52.245062+0000 mgr.y (mgr.14146) 1726 : cluster [DBG] pgmap v2503: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 45 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 3363/156 objects degraded (2155.769%); 8.8 MiB/s, 640 objects/s recovering 2024-02-12T16:41:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:53 smithi123 bash[19119]: cluster 2024-02-12T16:41:52.515502+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e907: 12 total, 11 up, 12 in 2024-02-12T16:41:53.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:53.692 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:53 smithi169 bash[19604]: cluster 2024-02-12T16:41:52.245062+0000 mgr.y (mgr.14146) 1726 : cluster [DBG] pgmap v2503: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 45 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 3363/156 objects degraded (2155.769%); 8.8 MiB/s, 640 objects/s recovering 2024-02-12T16:41:53.692 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:53 smithi169 bash[19604]: cluster 2024-02-12T16:41:52.515502+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e907: 12 total, 11 up, 12 in 2024-02-12T16:41:53.707 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:53.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:53.837 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:53 smithi121 bash[18489]: cluster 2024-02-12T16:41:52.245062+0000 mgr.y (mgr.14146) 1726 : cluster [DBG] pgmap v2503: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 45 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 3363/156 objects degraded (2155.769%); 8.8 MiB/s, 640 objects/s recovering 2024-02-12T16:41:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:53 smithi121 bash[18489]: cluster 2024-02-12T16:41:52.515502+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e907: 12 total, 11 up, 12 in 2024-02-12T16:41:53.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:53.964 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:53.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:54.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:54.104 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:54.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:54.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:54.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:54.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:54.386 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:54.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:54.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:54.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:54 smithi123 bash[19119]: cluster 2024-02-12T16:41:53.518753+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e908: 12 total, 11 up, 12 in 2024-02-12T16:41:54.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:54.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:54.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:54.779 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:54.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:54 smithi169 bash[19604]: cluster 2024-02-12T16:41:53.518753+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e908: 12 total, 11 up, 12 in 2024-02-12T16:41:54.831 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:54.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:54.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:54 smithi121 bash[18489]: cluster 2024-02-12T16:41:53.518753+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e908: 12 total, 11 up, 12 in 2024-02-12T16:41:54.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:54.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:55.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:55.062 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:55.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:55.196 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:55.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:55.323 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:55.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:55.446 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:55.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:55.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:55.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:55 smithi169 bash[19604]: cluster 2024-02-12T16:41:54.245956+0000 mgr.y (mgr.14146) 1727 : cluster [DBG] pgmap v2506: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 45 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 3363/156 objects degraded (2155.769%); 848 KiB/s, 695 objects/s recovering 2024-02-12T16:41:55.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:55 smithi121 bash[18489]: cluster 2024-02-12T16:41:54.245956+0000 mgr.y (mgr.14146) 1727 : cluster [DBG] pgmap v2506: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 45 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 3363/156 objects degraded (2155.769%); 848 KiB/s, 695 objects/s recovering 2024-02-12T16:41:55.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:55 smithi123 bash[19119]: cluster 2024-02-12T16:41:54.245956+0000 mgr.y (mgr.14146) 1727 : cluster [DBG] pgmap v2506: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 45 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 3363/156 objects degraded (2155.769%); 848 KiB/s, 695 objects/s recovering 2024-02-12T16:41:55.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:55.994 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:55.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:56.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:56.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:56.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:56.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:56.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:56.373 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:56.409 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:41:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:56.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:41:56.513 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:56.550 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:56.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:56.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:56.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:56.676 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:41:56.688 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:56.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:56.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:56 smithi169 bash[19604]: cluster 2024-02-12T16:41:55.929291+0000 mon.a (mon.0) 3067 : cluster [WRN] Health check update: Degraded data redundancy: 3363/156 objects degraded (2155.769%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:56.804 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:56 smithi123 bash[19119]: cluster 2024-02-12T16:41:55.929291+0000 mon.a (mon.0) 3067 : cluster [WRN] Health check update: Degraded data redundancy: 3363/156 objects degraded (2155.769%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:56.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:56.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:56.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:56 smithi121 bash[18489]: cluster 2024-02-12T16:41:55.929291+0000 mon.a (mon.0) 3067 : cluster [WRN] Health check update: Degraded data redundancy: 3363/156 objects degraded (2155.769%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:41:56.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:56.966 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:56.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:56.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:57.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:57.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:57.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:57.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:57.235 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:41:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:57.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:41:57.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:57.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:57.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:57.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:57.481 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:57.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:57.583 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:57.602 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:57.603 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:57 smithi169 bash[19604]: cluster 2024-02-12T16:41:56.246700+0000 mgr.y (mgr.14146) 1728 : cluster [DBG] pgmap v2507: 57 pgs: 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 65 op/s; 3363/156 objects degraded (2155.769%); 590 KiB/s, 273 objects/s recovering 2024-02-12T16:41:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:57.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:57.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:57.744 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:41:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:57.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:41:57.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:57.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:57.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:57 smithi123 bash[19119]: cluster 2024-02-12T16:41:56.246700+0000 mgr.y (mgr.14146) 1728 : cluster [DBG] pgmap v2507: 57 pgs: 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 65 op/s; 3363/156 objects degraded (2155.769%); 590 KiB/s, 273 objects/s recovering 2024-02-12T16:41:57.891 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:41:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:57 smithi121 bash[18489]: cluster 2024-02-12T16:41:56.246700+0000 mgr.y (mgr.14146) 1728 : cluster [DBG] pgmap v2507: 57 pgs: 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 1 active+recovery_wait+degraded+remapped, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 65 op/s; 3363/156 objects degraded (2155.769%); 590 KiB/s, 273 objects/s recovering 2024-02-12T16:41:57.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:58.003 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:58.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:58.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:58.126 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:41:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:41:58.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:58.250 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:41:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:41:58.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:58.388 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:58.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:58.524 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:41:58.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:41:58.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:58.656 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:41:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:41:58.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:58.782 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:58.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:58.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:59.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:41:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:41:59.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:41:59.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:59.327 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:59.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:59.452 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:59.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:59.581 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:59.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:59.707 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:41:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:41:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:41:59 smithi169 bash[19604]: cluster 2024-02-12T16:41:58.247381+0000 mgr.y (mgr.14146) 1729 : cluster [DBG] pgmap v2508: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 51 op/s; 464 KiB/s, 554 objects/s recovering 2024-02-12T16:41:59.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:59.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:41:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:41:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:41:59 smithi123 bash[19119]: cluster 2024-02-12T16:41:58.247381+0000 mgr.y (mgr.14146) 1729 : cluster [DBG] pgmap v2508: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 51 op/s; 464 KiB/s, 554 objects/s recovering 2024-02-12T16:41:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:41:59 smithi121 bash[18489]: cluster 2024-02-12T16:41:58.247381+0000 mgr.y (mgr.14146) 1729 : cluster [DBG] pgmap v2508: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 51 op/s; 464 KiB/s, 554 objects/s recovering 2024-02-12T16:41:59.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:59.973 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:41:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:41:59.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:00.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:00.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:00.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:00.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:00.238 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:00.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:00.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:00.364 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:00.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:00.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:00.502 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:00.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:00.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:00.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:00.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:00.777 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:00.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:00.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:00.917 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:00.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:01.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:01.053 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:01.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:01.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:01.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:01.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:01.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:01.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:01.407 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:42:01.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:01.442 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:42:01.463 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:42:01.467 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:01.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:01.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:42:01.515 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:42:01.543 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:42:01.553 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:01 smithi121 bash[18489]: cluster 2024-02-12T16:42:00.248288+0000 mgr.y (mgr.14146) 1730 : cluster [DBG] pgmap v2509: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 159 objects/s recovering 2024-02-12T16:42:01.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:01.571 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:42:01.600 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:42:01.605 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:01.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:01.622 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:42:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:42:01.652 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:42:01.652 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:42:01.677 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:42:01.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.677 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.678 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:42:01.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:01 smithi169 bash[19604]: cluster 2024-02-12T16:42:00.248288+0000 mgr.y (mgr.14146) 1730 : cluster [DBG] pgmap v2509: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 159 objects/s recovering 2024-02-12T16:42:01.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:01 smithi169 bash[19604]: cluster 2024-02-12T16:42:00.930833+0000 mon.a (mon.0) 3068 : cluster [WRN] Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:42:01.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:01 smithi169 bash[19604]: audit 2024-02-12T16:42:01.378993+0000 mon.a (mon.0) 3069 : audit [DBG] from='client.? v1:172.21.15.121:0/2745558858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:01.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:01 smithi169 bash[19604]: audit 2024-02-12T16:42:01.444720+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? v1:172.21.15.121:0/2294168503' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:42:01.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:01.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:01.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:01.811 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:01 smithi121 bash[18489]: cluster 2024-02-12T16:42:00.930833+0000 mon.a (mon.0) 3068 : cluster [WRN] Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:42:01.811 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:01 smithi121 bash[18489]: audit 2024-02-12T16:42:01.378993+0000 mon.a (mon.0) 3069 : audit [DBG] from='client.? v1:172.21.15.121:0/2745558858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:01.811 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:01 smithi121 bash[18489]: audit 2024-02-12T16:42:01.444720+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? v1:172.21.15.121:0/2294168503' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:42:01.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:01.876 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:01.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:01 smithi123 bash[19119]: cluster 2024-02-12T16:42:00.248288+0000 mgr.y (mgr.14146) 1730 : cluster [DBG] pgmap v2509: 57 pgs: 1 active+recovering+remapped, 2 active+recovering, 2 active+clean+remapped, 6 undersized+peered, 46 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 159 objects/s recovering 2024-02-12T16:42:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:01 smithi123 bash[19119]: cluster 2024-02-12T16:42:00.930833+0000 mon.a (mon.0) 3068 : cluster [WRN] Health check update: Degraded data redundancy: 6 pgs undersized (PG_DEGRADED) 2024-02-12T16:42:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:01 smithi123 bash[19119]: audit 2024-02-12T16:42:01.378993+0000 mon.a (mon.0) 3069 : audit [DBG] from='client.? v1:172.21.15.121:0/2745558858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:01 smithi123 bash[19119]: audit 2024-02-12T16:42:01.444720+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? v1:172.21.15.121:0/2294168503' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:42:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:42:01.979 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:02.016 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:02.120 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:02.159 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:02.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:02.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:02.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:02.414 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:02.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-12T16:42:02.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:02.553 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:02.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:02.688 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:02.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:02.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:02.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:02 smithi169 bash[19604]: cluster 2024-02-12T16:42:01.940120+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:02.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:02 smithi169 bash[19604]: audit 2024-02-12T16:42:01.947640+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? v1:172.21.15.121:0/2294168503' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:42:02.812 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:02 smithi169 bash[19604]: cluster 2024-02-12T16:42:01.947720+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e909: 12 total, 11 up, 12 in 2024-02-12T16:42:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:02 smithi123 bash[19119]: cluster 2024-02-12T16:42:01.940120+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:02 smithi123 bash[19119]: audit 2024-02-12T16:42:01.947640+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? v1:172.21.15.121:0/2294168503' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:42:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:02 smithi123 bash[19119]: cluster 2024-02-12T16:42:01.947720+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e909: 12 total, 11 up, 12 in 2024-02-12T16:42:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:02 smithi121 bash[18489]: cluster 2024-02-12T16:42:01.940120+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:02 smithi121 bash[18489]: audit 2024-02-12T16:42:01.947640+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? v1:172.21.15.121:0/2294168503' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:42:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:02 smithi121 bash[18489]: cluster 2024-02-12T16:42:01.947720+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e909: 12 total, 11 up, 12 in 2024-02-12T16:42:02.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:02.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:02.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:03.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:03.092 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:03.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:03.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:03.235 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:03.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:03.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:03.378 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:03.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:03.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:03.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:03.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:03.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:03.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:03.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:03.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:03.752 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:03 smithi169 bash[19604]: cluster 2024-02-12T16:42:02.248842+0000 mgr.y (mgr.14146) 1731 : cluster [DBG] pgmap v2511: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 283 objects/s recovering 2024-02-12T16:42:03.776 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:03.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:03.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:03 smithi123 bash[19119]: cluster 2024-02-12T16:42:02.248842+0000 mgr.y (mgr.14146) 1731 : cluster [DBG] pgmap v2511: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 283 objects/s recovering 2024-02-12T16:42:03.893 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:03 smithi121 bash[18489]: cluster 2024-02-12T16:42:02.248842+0000 mgr.y (mgr.14146) 1731 : cluster [DBG] pgmap v2511: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 283 objects/s recovering 2024-02-12T16:42:03.921 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:03.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:04.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:04.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:04.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:04.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:04.184 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:04.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:04.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:04.465 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:04.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:04.588 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:04.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:04.729 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:04.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:04.868 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:04.962 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:04 smithi169 bash[19604]: cluster 2024-02-12T16:42:03.530146+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:04.962 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:04 smithi169 bash[19604]: cluster 2024-02-12T16:42:03.530253+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:04.962 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:04 smithi169 bash[19604]: cluster 2024-02-12T16:42:03.954181+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e910: 12 total, 11 up, 12 in 2024-02-12T16:42:04.962 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:04 smithi169 bash[19604]: cluster 2024-02-12T16:42:04.249808+0000 mgr.y (mgr.14146) 1732 : cluster [DBG] pgmap v2513: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 461 objects/s recovering 2024-02-12T16:42:04.963 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:04 smithi121 bash[18489]: cluster 2024-02-12T16:42:03.530146+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:04.963 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:04 smithi121 bash[18489]: cluster 2024-02-12T16:42:03.530253+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:04.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:04.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:04.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:42:05.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:42:05.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.116 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:05.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:05.219 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:05.245 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:05.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:05.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:05.384 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:05.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:04 smithi123 bash[19119]: cluster 2024-02-12T16:42:03.530146+0000 osd.1 (osd.1) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:04 smithi123 bash[19119]: cluster 2024-02-12T16:42:03.530253+0000 osd.1 (osd.1) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:04 smithi123 bash[19119]: cluster 2024-02-12T16:42:03.954181+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e910: 12 total, 11 up, 12 in 2024-02-12T16:42:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:04 smithi123 bash[19119]: cluster 2024-02-12T16:42:04.249808+0000 mgr.y (mgr.14146) 1732 : cluster [DBG] pgmap v2513: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 461 objects/s recovering 2024-02-12T16:42:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:04 smithi121 bash[18489]: cluster 2024-02-12T16:42:03.954181+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e910: 12 total, 11 up, 12 in 2024-02-12T16:42:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:04 smithi121 bash[18489]: cluster 2024-02-12T16:42:04.249808+0000 mgr.y (mgr.14146) 1732 : cluster [DBG] pgmap v2513: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 461 objects/s recovering 2024-02-12T16:42:05.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:05.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:05.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.674 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:05.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:05.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:05.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:05.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:06.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:06.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:06.142 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:06.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:06.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:06.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:06.423 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:06.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:06.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:06.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:06.698 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:06.801 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:06.831 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:06.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:06.972 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:06.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:07.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:42:07.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:07.106 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:07.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:07.206 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:42:07.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:07.221 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:06 smithi123 bash[19119]: cluster 2024-02-12T16:42:05.943551+0000 mon.a (mon.0) 3075 : cluster [DBG] osdmap e911: 12 total, 11 up, 12 in 2024-02-12T16:42:07.221 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:06 smithi123 bash[19119]: cluster 2024-02-12T16:42:06.236863+0000 osd.2 (osd.2) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:07.221 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:06 smithi123 bash[19119]: cluster 2024-02-12T16:42:06.236988+0000 osd.2 (osd.2) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:07.221 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:06 smithi123 bash[19119]: cluster 2024-02-12T16:42:06.250748+0000 mgr.y (mgr.14146) 1733 : cluster [DBG] pgmap v2515: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 200 objects/s recovering 2024-02-12T16:42:07.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:07.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:06 smithi169 bash[19604]: cluster 2024-02-12T16:42:05.943551+0000 mon.a (mon.0) 3075 : cluster [DBG] osdmap e911: 12 total, 11 up, 12 in 2024-02-12T16:42:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:06 smithi169 bash[19604]: cluster 2024-02-12T16:42:06.236863+0000 osd.2 (osd.2) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:06 smithi169 bash[19604]: cluster 2024-02-12T16:42:06.236988+0000 osd.2 (osd.2) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:06 smithi169 bash[19604]: cluster 2024-02-12T16:42:06.250748+0000 mgr.y (mgr.14146) 1733 : cluster [DBG] pgmap v2515: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 200 objects/s recovering 2024-02-12T16:42:07.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:07.389 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:07.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:06 smithi121 bash[18489]: cluster 2024-02-12T16:42:05.943551+0000 mon.a (mon.0) 3075 : cluster [DBG] osdmap e911: 12 total, 11 up, 12 in 2024-02-12T16:42:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:06 smithi121 bash[18489]: cluster 2024-02-12T16:42:06.236863+0000 osd.2 (osd.2) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:06 smithi121 bash[18489]: cluster 2024-02-12T16:42:06.236988+0000 osd.2 (osd.2) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:06 smithi121 bash[18489]: cluster 2024-02-12T16:42:06.250748+0000 mgr.y (mgr.14146) 1733 : cluster [DBG] pgmap v2515: 57 pgs: 1 active+recovering+remapped, 2 active+clean+remapped, 6 undersized+peered, 48 active+clean; 4.8 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 200 objects/s recovering 2024-02-12T16:42:07.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:07.524 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:07.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:07.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:42:07.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:07.668 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:07.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:07.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:07.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:07.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:07.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:07.952 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:07.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:07.992 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_6' removed 2024-02-12T16:42:08.003 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.3 to 0.554335 (8de8) 2024-02-12T16:42:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:07 smithi169 bash[19604]: cluster 2024-02-12T16:42:06.967825+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e912: 12 total, 11 up, 12 in 2024-02-12T16:42:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:07 smithi169 bash[19604]: audit 2024-02-12T16:42:07.125477+0000 mon.a (mon.0) 3077 : audit [INF] from='client.? v1:172.21.15.121:0/1491243651' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:42:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:07 smithi169 bash[19604]: audit 2024-02-12T16:42:07.150191+0000 mon.a (mon.0) 3078 : audit [INF] from='client.? v1:172.21.15.121:0/814683860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.5543352723013449}]: dispatch 2024-02-12T16:42:08.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:08.089 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:08.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:08.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:08.224 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:08.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:08.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:08.339 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:07 smithi123 bash[19119]: cluster 2024-02-12T16:42:06.967825+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e912: 12 total, 11 up, 12 in 2024-02-12T16:42:08.339 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:07 smithi123 bash[19119]: audit 2024-02-12T16:42:07.125477+0000 mon.a (mon.0) 3077 : audit [INF] from='client.? v1:172.21.15.121:0/1491243651' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:42:08.339 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:07 smithi123 bash[19119]: audit 2024-02-12T16:42:07.150191+0000 mon.a (mon.0) 3078 : audit [INF] from='client.? v1:172.21.15.121:0/814683860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.5543352723013449}]: dispatch 2024-02-12T16:42:08.364 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:08.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:07 smithi121 bash[18489]: cluster 2024-02-12T16:42:06.967825+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e912: 12 total, 11 up, 12 in 2024-02-12T16:42:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:07 smithi121 bash[18489]: audit 2024-02-12T16:42:07.125477+0000 mon.a (mon.0) 3077 : audit [INF] from='client.? v1:172.21.15.121:0/1491243651' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:42:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:07 smithi121 bash[18489]: audit 2024-02-12T16:42:07.150191+0000 mon.a (mon.0) 3078 : audit [INF] from='client.? v1:172.21.15.121:0/814683860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.5543352723013449}]: dispatch 2024-02-12T16:42:08.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:08.501 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:08.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:08.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:08.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:08.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:08.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:08.783 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:08.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:08.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:08.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:08.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:09.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:09.062 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:09.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:08 smithi121 bash[18489]: audit 2024-02-12T16:42:07.979964+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? v1:172.21.15.121:0/1491243651' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:42:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:08 smithi121 bash[18489]: audit 2024-02-12T16:42:07.980076+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? v1:172.21.15.121:0/814683860' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5543352723013449}]': finished 2024-02-12T16:42:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:08 smithi121 bash[18489]: cluster 2024-02-12T16:42:07.980155+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e913: 12 total, 11 up, 12 in 2024-02-12T16:42:09.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:08 smithi121 bash[18489]: cluster 2024-02-12T16:42:08.251737+0000 mgr.y (mgr.14146) 1734 : cluster [DBG] pgmap v2518: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:09.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:09.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:09.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:08 smithi169 bash[19604]: audit 2024-02-12T16:42:07.979964+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? v1:172.21.15.121:0/1491243651' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:42:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:08 smithi169 bash[19604]: audit 2024-02-12T16:42:07.980076+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? v1:172.21.15.121:0/814683860' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5543352723013449}]': finished 2024-02-12T16:42:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:08 smithi169 bash[19604]: cluster 2024-02-12T16:42:07.980155+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e913: 12 total, 11 up, 12 in 2024-02-12T16:42:09.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:08 smithi169 bash[19604]: cluster 2024-02-12T16:42:08.251737+0000 mgr.y (mgr.14146) 1734 : cluster [DBG] pgmap v2518: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:09.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:09.346 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:09.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:09.366 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-12T16:42:09.367 INFO:tasks.radosbench:Beginning radosbench... 2024-02-12T16:42:09.367 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-02-12T16:42:09.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool create unique_pool_7 16 2024-02-12T16:42:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:08 smithi123 bash[19119]: audit 2024-02-12T16:42:07.979964+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? v1:172.21.15.121:0/1491243651' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:42:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:08 smithi123 bash[19119]: audit 2024-02-12T16:42:07.980076+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? v1:172.21.15.121:0/814683860' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5543352723013449}]': finished 2024-02-12T16:42:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:08 smithi123 bash[19119]: cluster 2024-02-12T16:42:07.980155+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e913: 12 total, 11 up, 12 in 2024-02-12T16:42:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:08 smithi123 bash[19119]: cluster 2024-02-12T16:42:08.251737+0000 mgr.y (mgr.14146) 1734 : cluster [DBG] pgmap v2518: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:09.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:09.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:09.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:09.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:09.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:09.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:09.767 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:09.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:09.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:09.908 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:09.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:10.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:10.051 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:10.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:10.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:10.192 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:10.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:10.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:10.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:10.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:10.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:10.481 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:10.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:10.585 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:10.619 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:10.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:10.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:10.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:10.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:10.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:10.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:10.904 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:42:10.904 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:42:10.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:11.019 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:11.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:11.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:11.159 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:11.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:11.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:11.300 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:11.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:11.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:11 smithi121 bash[18489]: cluster 2024-02-12T16:42:10.252730+0000 mgr.y (mgr.14146) 1735 : cluster [DBG] pgmap v2519: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:11 smithi121 bash[18489]: cluster 2024-02-12T16:42:10.942766+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e914: 12 total, 11 up, 12 in 2024-02-12T16:42:11.438 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:11.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:11.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:11.580 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:11.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:11 smithi169 bash[19604]: cluster 2024-02-12T16:42:10.252730+0000 mgr.y (mgr.14146) 1735 : cluster [DBG] pgmap v2519: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:11.582 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:11 smithi169 bash[19604]: cluster 2024-02-12T16:42:10.942766+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e914: 12 total, 11 up, 12 in 2024-02-12T16:42:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:11.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:11.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-12T16:42:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:11 smithi123 bash[19119]: cluster 2024-02-12T16:42:10.252730+0000 mgr.y (mgr.14146) 1735 : cluster [DBG] pgmap v2519: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:11 smithi123 bash[19119]: cluster 2024-02-12T16:42:10.942766+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e914: 12 total, 11 up, 12 in 2024-02-12T16:42:11.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:11.719 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:11.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:11.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:11.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:11.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:11.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:11.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:11.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:12.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:12.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:12.240 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:12.277 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:12.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:12.414 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:12.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:12.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:12.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:12.691 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:12.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:12.831 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:12.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:12.953 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:12.954 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:12.971 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:12.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:13.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:13.110 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:13.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:13.214 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:13.251 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:13.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:13.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:13.393 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:13.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:13.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:13.511 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:13 smithi123 bash[19119]: cluster 2024-02-12T16:42:12.253770+0000 mgr.y (mgr.14146) 1736 : cluster [DBG] pgmap v2521: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:13.533 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:13.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:13.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:13 smithi169 bash[19604]: cluster 2024-02-12T16:42:12.253770+0000 mgr.y (mgr.14146) 1736 : cluster [DBG] pgmap v2521: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:13.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:13.649 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:13 smithi121 bash[18489]: cluster 2024-02-12T16:42:12.253770+0000 mgr.y (mgr.14146) 1736 : cluster [DBG] pgmap v2521: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:13.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:13.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:13.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:13.823 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:13.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:13.928 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:13.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:13.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:13.972 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:14.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:14.103 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:14.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:14.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:14.243 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:14.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:14.348 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:14.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:14.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:14.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8, 11] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3] 2024-02-12T16:42:14.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:42:14.426 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-12T16:42:14.426 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:42:14.426 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:42:14.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-12T16:42:14.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:14.524 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:14.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:14.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:14.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:14.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:14.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:14.805 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:14.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:14.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:14.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:15.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:15.083 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:15.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:15.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:15.225 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:15.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:15.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:15.365 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:15.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:15.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:15.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:15.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:15.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:15 smithi121 bash[18489]: cluster 2024-02-12T16:42:14.254786+0000 mgr.y (mgr.14146) 1737 : cluster [DBG] pgmap v2522: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:15.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:15 smithi123 bash[19119]: cluster 2024-02-12T16:42:14.254786+0000 mgr.y (mgr.14146) 1737 : cluster [DBG] pgmap v2522: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:15.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:15.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:15.744 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:15.759 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:15 smithi169 bash[19604]: cluster 2024-02-12T16:42:14.254786+0000 mgr.y (mgr.14146) 1737 : cluster [DBG] pgmap v2522: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:15.785 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:15.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:15.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:15.929 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:15.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:16.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:16.067 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:16.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:16.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:16.209 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:16.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:16.312 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:16.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:16.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:16.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:16.464 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:42:16.490 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:16.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:16.594 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:16.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:16.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:16.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:16.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:16.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:16.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:16.911 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:16.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:17.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:17.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:17.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:16 smithi121 bash[18489]: cluster 2024-02-12T16:42:16.255728+0000 mgr.y (mgr.14146) 1738 : cluster [DBG] pgmap v2523: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:17.193 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:17.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:17.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:17.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:16 smithi169 bash[19604]: cluster 2024-02-12T16:42:16.255728+0000 mgr.y (mgr.14146) 1738 : cluster [DBG] pgmap v2523: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:17.332 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:17.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:16 smithi123 bash[19119]: cluster 2024-02-12T16:42:16.255728+0000 mgr.y (mgr.14146) 1738 : cluster [DBG] pgmap v2523: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:17.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:17.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:17.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:17.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:17.613 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:17.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:17.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:17.754 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:17.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:17.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:17.887 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:17.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:17.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.027 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:18.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:18.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.169 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:18.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:18.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:18.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:18.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:18.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:18.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:18.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:18.559 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:18.594 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:18.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:18.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.734 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:18.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:18.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:18.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:18.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:19.015 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:19.120 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:19.157 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:19.223 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:42:19.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:19.297 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:19 smithi123 bash[19119]: cluster 2024-02-12T16:42:18.256678+0000 mgr.y (mgr.14146) 1739 : cluster [DBG] pgmap v2524: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:19.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:19.438 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:19.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:19 smithi169 bash[19604]: cluster 2024-02-12T16:42:18.256678+0000 mgr.y (mgr.14146) 1739 : cluster [DBG] pgmap v2524: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:19.572 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:19 smithi121 bash[18489]: cluster 2024-02-12T16:42:18.256678+0000 mgr.y (mgr.14146) 1739 : cluster [DBG] pgmap v2524: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:19.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:19.696 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:42:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:19.715 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:42:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:19.720 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:42:19.720 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:42:19.738 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:42:19.757 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:42:19.780 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:42:19.805 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:42:19.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:19.828 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:42:19.851 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:19.858 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:42:19.882 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:42:19.905 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:42:19.929 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:42:19.929 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.929 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.930 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:42:19.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:19.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:19.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:20.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:20.129 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:20.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:20.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:20.268 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:20.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:20.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:20.401 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:20.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:20.466 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-12T16:42:20.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:20.511 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:20 smithi123 bash[19119]: audit 2024-02-12T16:42:19.663122+0000 mon.a (mon.0) 3083 : audit [DBG] from='client.? v1:172.21.15.121:0/3637612767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:20.537 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:20.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:20.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:20.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:20 smithi121 bash[18489]: audit 2024-02-12T16:42:19.663122+0000 mon.a (mon.0) 3083 : audit [DBG] from='client.? v1:172.21.15.121:0/3637612767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:20.684 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:20.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:20.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:20 smithi169 bash[19604]: audit 2024-02-12T16:42:19.663122+0000 mon.a (mon.0) 3083 : audit [DBG] from='client.? v1:172.21.15.121:0/3637612767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:20.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:20.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:42:20.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:42:20.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:20.961 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_7' created 2024-02-12T16:42:20.962 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:42:20.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:20.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:21.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:21.107 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:21.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:21.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:21.246 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:21.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:21.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:21.389 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: cluster 2024-02-12T16:42:19.520368+0000 osd.10 (osd.10) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: cluster 2024-02-12T16:42:19.520479+0000 osd.10 (osd.10) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: cluster 2024-02-12T16:42:20.257328+0000 mgr.y (mgr.14146) 1740 : cluster [DBG] pgmap v2525: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: cluster 2024-02-12T16:42:20.323975+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e915: 12 total, 11 up, 12 in 2024-02-12T16:42:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: audit 2024-02-12T16:42:20.373538+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? v1:172.21.15.121:0/1271620762' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:42:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: audit 2024-02-12T16:42:20.439000+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? v1:172.21.15.121:0/661094684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-02-12T16:42:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: cluster 2024-02-12T16:42:20.942398+0000 mon.a (mon.0) 3087 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: audit 2024-02-12T16:42:20.951578+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? v1:172.21.15.121:0/1271620762' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:42:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: audit 2024-02-12T16:42:20.951637+0000 mon.a (mon.0) 3089 : audit [INF] from='client.? v1:172.21.15.121:0/661094684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-02-12T16:42:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:21 smithi123 bash[19119]: cluster 2024-02-12T16:42:20.951682+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e916: 12 total, 11 up, 12 in 2024-02-12T16:42:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:21.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:21.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:21.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:21.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:21.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: cluster 2024-02-12T16:42:19.520368+0000 osd.10 (osd.10) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: cluster 2024-02-12T16:42:19.520479+0000 osd.10 (osd.10) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: cluster 2024-02-12T16:42:20.257328+0000 mgr.y (mgr.14146) 1740 : cluster [DBG] pgmap v2525: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: cluster 2024-02-12T16:42:20.323975+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e915: 12 total, 11 up, 12 in 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: audit 2024-02-12T16:42:20.373538+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? v1:172.21.15.121:0/1271620762' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: audit 2024-02-12T16:42:20.439000+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? v1:172.21.15.121:0/661094684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: cluster 2024-02-12T16:42:20.942398+0000 mon.a (mon.0) 3087 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: audit 2024-02-12T16:42:20.951578+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? v1:172.21.15.121:0/1271620762' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: audit 2024-02-12T16:42:20.951637+0000 mon.a (mon.0) 3089 : audit [INF] from='client.? v1:172.21.15.121:0/661094684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-02-12T16:42:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:21 smithi121 bash[18489]: cluster 2024-02-12T16:42:20.951682+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e916: 12 total, 11 up, 12 in 2024-02-12T16:42:21.662 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:21.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:21.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: cluster 2024-02-12T16:42:19.520368+0000 osd.10 (osd.10) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: cluster 2024-02-12T16:42:19.520479+0000 osd.10 (osd.10) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: cluster 2024-02-12T16:42:20.257328+0000 mgr.y (mgr.14146) 1740 : cluster [DBG] pgmap v2525: 31 pgs: 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: cluster 2024-02-12T16:42:20.323975+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e915: 12 total, 11 up, 12 in 2024-02-12T16:42:21.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: audit 2024-02-12T16:42:20.373538+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? v1:172.21.15.121:0/1271620762' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:42:21.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: audit 2024-02-12T16:42:20.439000+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? v1:172.21.15.121:0/661094684' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-02-12T16:42:21.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: cluster 2024-02-12T16:42:20.942398+0000 mon.a (mon.0) 3087 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:21.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: audit 2024-02-12T16:42:20.951578+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? v1:172.21.15.121:0/1271620762' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:42:21.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: audit 2024-02-12T16:42:20.951637+0000 mon.a (mon.0) 3089 : audit [INF] from='client.? v1:172.21.15.121:0/661094684' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-02-12T16:42:21.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:21 smithi169 bash[19604]: cluster 2024-02-12T16:42:20.951682+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e916: 12 total, 11 up, 12 in 2024-02-12T16:42:21.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:21.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:21.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:21.940 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:21.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:22.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:22.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:22.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-02-12T16:42:22.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:22.223 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:22.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:22.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:22.359 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:22.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:22.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:22.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:22.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:22.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:22.646 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:22.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:22.727 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:42:22.727 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "" 2024-02-12T16:42:22.727 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:42:22.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:22.789 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:22.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:22.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:42:22.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:22.932 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:22.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:23.036 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:21.569169+0000 osd.11 (osd.11) 213 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:21.631853+0000 osd.11 (osd.11) 214 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:21.955245+0000 mon.a (mon.0) 3091 : cluster [DBG] osdmap e917: 12 total, 11 up, 12 in 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:22.043142+0000 osd.4 (osd.4) 155 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:22.106264+0000 osd.4 (osd.4) 156 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:22.258133+0000 mgr.y (mgr.14146) 1741 : cluster [DBG] pgmap v2529: 47 pgs: 10 creating+peering, 6 unknown, 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:22.394638+0000 osd.0 (osd.0) 155 : cluster [DBG] 1.c scrub starts 2024-02-12T16:42:23.048 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:22 smithi123 bash[19119]: cluster 2024-02-12T16:42:22.478959+0000 osd.0 (osd.0) 156 : cluster [DBG] 1.c scrub ok 2024-02-12T16:42:23.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:23.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:21.569169+0000 osd.11 (osd.11) 213 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:42:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:21.631853+0000 osd.11 (osd.11) 214 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:42:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:21.955245+0000 mon.a (mon.0) 3091 : cluster [DBG] osdmap e917: 12 total, 11 up, 12 in 2024-02-12T16:42:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:22.043142+0000 osd.4 (osd.4) 155 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:42:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:22.106264+0000 osd.4 (osd.4) 156 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:42:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:22.258133+0000 mgr.y (mgr.14146) 1741 : cluster [DBG] pgmap v2529: 47 pgs: 10 creating+peering, 6 unknown, 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:22.394638+0000 osd.0 (osd.0) 155 : cluster [DBG] 1.c scrub starts 2024-02-12T16:42:23.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:22 smithi121 bash[18489]: cluster 2024-02-12T16:42:22.478959+0000 osd.0 (osd.0) 156 : cluster [DBG] 1.c scrub ok 2024-02-12T16:42:23.174 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:23.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:23.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:21.569169+0000 osd.11 (osd.11) 213 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:42:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:21.631853+0000 osd.11 (osd.11) 214 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:42:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:21.955245+0000 mon.a (mon.0) 3091 : cluster [DBG] osdmap e917: 12 total, 11 up, 12 in 2024-02-12T16:42:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:22.043142+0000 osd.4 (osd.4) 155 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:42:23.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:22.106264+0000 osd.4 (osd.4) 156 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:42:23.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:22.258133+0000 mgr.y (mgr.14146) 1741 : cluster [DBG] pgmap v2529: 47 pgs: 10 creating+peering, 6 unknown, 1 active+clean+remapped, 6 undersized+peered, 24 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:23.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:22.394638+0000 osd.0 (osd.0) 155 : cluster [DBG] 1.c scrub starts 2024-02-12T16:42:23.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:22 smithi169 bash[19604]: cluster 2024-02-12T16:42:22.478959+0000 osd.0 (osd.0) 156 : cluster [DBG] 1.c scrub ok 2024-02-12T16:42:23.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:23.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:23.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:23.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:23.493 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:23.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:23.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:23.634 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:23.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:23.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:23.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:23.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:23.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:23.915 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:23.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:24.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:24.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:24.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:24.163 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:23 smithi169 bash[19604]: cluster 2024-02-12T16:42:22.962087+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e918: 12 total, 11 up, 12 in 2024-02-12T16:42:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:23 smithi169 bash[19604]: cluster 2024-02-12T16:42:23.430134+0000 osd.6 (osd.6) 182 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:42:24.175 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:23 smithi169 bash[19604]: cluster 2024-02-12T16:42:23.543381+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:42:24.199 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:24.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:24.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:42:24.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:23 smithi121 bash[18489]: cluster 2024-02-12T16:42:22.962087+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e918: 12 total, 11 up, 12 in 2024-02-12T16:42:24.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:23 smithi121 bash[18489]: cluster 2024-02-12T16:42:23.430134+0000 osd.6 (osd.6) 182 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:42:24.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:23 smithi121 bash[18489]: cluster 2024-02-12T16:42:23.543381+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:42:24.303 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:24.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:24.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:24.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:23 smithi123 bash[19119]: cluster 2024-02-12T16:42:22.962087+0000 mon.a (mon.0) 3092 : cluster [DBG] osdmap e918: 12 total, 11 up, 12 in 2024-02-12T16:42:24.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:23 smithi123 bash[19119]: cluster 2024-02-12T16:42:23.430134+0000 osd.6 (osd.6) 182 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:42:24.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:23 smithi123 bash[19119]: cluster 2024-02-12T16:42:23.543381+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:42:24.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:24.483 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:24.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:24.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:24.623 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:24.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:24.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:24.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:24.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:24.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:24.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:24.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:25.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:25.052 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:25.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:24 smithi123 bash[19119]: cluster 2024-02-12T16:42:23.465564+0000 osd.5 (osd.5) 211 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:42:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:24 smithi123 bash[19119]: cluster 2024-02-12T16:42:23.507533+0000 osd.5 (osd.5) 212 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:42:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:24 smithi123 bash[19119]: cluster 2024-02-12T16:42:23.576891+0000 osd.11 (osd.11) 215 : cluster [DBG] 1.b scrub starts 2024-02-12T16:42:25.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:24 smithi123 bash[19119]: cluster 2024-02-12T16:42:23.724516+0000 osd.11 (osd.11) 216 : cluster [DBG] 1.b scrub ok 2024-02-12T16:42:25.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:24 smithi123 bash[19119]: cluster 2024-02-12T16:42:24.259124+0000 mgr.y (mgr.14146) 1742 : cluster [DBG] pgmap v2531: 47 pgs: 10 creating+peering, 1 unknown, 1 active+clean+remapped, 8 undersized+peered, 27 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:25.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:25.193 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:25.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:25.242 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:42:25.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:24 smithi169 bash[19604]: cluster 2024-02-12T16:42:23.465564+0000 osd.5 (osd.5) 211 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:42:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:24 smithi169 bash[19604]: cluster 2024-02-12T16:42:23.507533+0000 osd.5 (osd.5) 212 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:42:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:24 smithi169 bash[19604]: cluster 2024-02-12T16:42:23.576891+0000 osd.11 (osd.11) 215 : cluster [DBG] 1.b scrub starts 2024-02-12T16:42:25.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:24 smithi169 bash[19604]: cluster 2024-02-12T16:42:23.724516+0000 osd.11 (osd.11) 216 : cluster [DBG] 1.b scrub ok 2024-02-12T16:42:25.299 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:24 smithi169 bash[19604]: cluster 2024-02-12T16:42:24.259124+0000 mgr.y (mgr.14146) 1742 : cluster [DBG] pgmap v2531: 47 pgs: 10 creating+peering, 1 unknown, 1 active+clean+remapped, 8 undersized+peered, 27 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:25.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:24 smithi121 bash[18489]: cluster 2024-02-12T16:42:23.465564+0000 osd.5 (osd.5) 211 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:42:25.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:24 smithi121 bash[18489]: cluster 2024-02-12T16:42:23.507533+0000 osd.5 (osd.5) 212 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:42:25.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:24 smithi121 bash[18489]: cluster 2024-02-12T16:42:23.576891+0000 osd.11 (osd.11) 215 : cluster [DBG] 1.b scrub starts 2024-02-12T16:42:25.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:24 smithi121 bash[18489]: cluster 2024-02-12T16:42:23.724516+0000 osd.11 (osd.11) 216 : cluster [DBG] 1.b scrub ok 2024-02-12T16:42:25.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:24 smithi121 bash[18489]: cluster 2024-02-12T16:42:24.259124+0000 mgr.y (mgr.14146) 1742 : cluster [DBG] pgmap v2531: 47 pgs: 10 creating+peering, 1 unknown, 1 active+clean+remapped, 8 undersized+peered, 27 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:25.333 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:25.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:25.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:25.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:25.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:25.613 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:25.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:25.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:25.753 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:25.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:25.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:25.898 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:25.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:26.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:26.014 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:25 smithi121 bash[18489]: cluster 2024-02-12T16:42:24.884788+0000 osd.10 (osd.10) 230 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:42:26.014 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:25 smithi121 bash[18489]: cluster 2024-02-12T16:42:24.921221+0000 osd.10 (osd.10) 231 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:42:26.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:26.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:26.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:26.181 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:26.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:26.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:26.296 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:25 smithi123 bash[19119]: cluster 2024-02-12T16:42:24.884788+0000 osd.10 (osd.10) 230 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:42:26.296 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:25 smithi123 bash[19119]: cluster 2024-02-12T16:42:24.921221+0000 osd.10 (osd.10) 231 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:42:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:25 smithi169 bash[19604]: cluster 2024-02-12T16:42:24.884788+0000 osd.10 (osd.10) 230 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:42:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:25 smithi169 bash[19604]: cluster 2024-02-12T16:42:24.921221+0000 osd.10 (osd.10) 231 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:42:26.320 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:26.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:26.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:26.461 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:26.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:26.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:26.604 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:26.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:26.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:26.737 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:26.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:26.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:26.876 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:26.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:26.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:26.959 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:26.959 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:26.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:27.115 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:27.147 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:26 smithi121 bash[18489]: cluster 2024-02-12T16:42:25.205645+0000 osd.1 (osd.1) 159 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:42:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:26 smithi121 bash[18489]: cluster 2024-02-12T16:42:25.380184+0000 osd.1 (osd.1) 160 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:42:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:26 smithi121 bash[18489]: cluster 2024-02-12T16:42:25.743225+0000 osd.8 (osd.8) 135 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:42:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:26 smithi121 bash[18489]: cluster 2024-02-12T16:42:25.848571+0000 osd.8 (osd.8) 136 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:42:27.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:26 smithi121 bash[18489]: cluster 2024-02-12T16:42:26.259759+0000 mgr.y (mgr.14146) 1743 : cluster [DBG] pgmap v2532: 47 pgs: 1 active+clean+remapped, 9 creating+peering, 8 undersized+peered, 29 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:27.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:27.263 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:26 smithi123 bash[19119]: cluster 2024-02-12T16:42:25.205645+0000 osd.1 (osd.1) 159 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:42:27.263 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:26 smithi123 bash[19119]: cluster 2024-02-12T16:42:25.380184+0000 osd.1 (osd.1) 160 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:42:27.263 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:26 smithi123 bash[19119]: cluster 2024-02-12T16:42:25.743225+0000 osd.8 (osd.8) 135 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:42:27.263 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:26 smithi123 bash[19119]: cluster 2024-02-12T16:42:25.848571+0000 osd.8 (osd.8) 136 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:42:27.263 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:26 smithi123 bash[19119]: cluster 2024-02-12T16:42:26.259759+0000 mgr.y (mgr.14146) 1743 : cluster [DBG] pgmap v2532: 47 pgs: 1 active+clean+remapped, 9 creating+peering, 8 undersized+peered, 29 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:27.288 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:26 smithi169 bash[19604]: cluster 2024-02-12T16:42:25.205645+0000 osd.1 (osd.1) 159 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:42:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:26 smithi169 bash[19604]: cluster 2024-02-12T16:42:25.380184+0000 osd.1 (osd.1) 160 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:42:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:26 smithi169 bash[19604]: cluster 2024-02-12T16:42:25.743225+0000 osd.8 (osd.8) 135 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:42:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:26 smithi169 bash[19604]: cluster 2024-02-12T16:42:25.848571+0000 osd.8 (osd.8) 136 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:42:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:26 smithi169 bash[19604]: cluster 2024-02-12T16:42:26.259759+0000 mgr.y (mgr.14146) 1743 : cluster [DBG] pgmap v2532: 47 pgs: 1 active+clean+remapped, 9 creating+peering, 8 undersized+peered, 29 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:27.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:27.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:27.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:27.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:27.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:27.696 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:27.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:27.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:27.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:27.972 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:27.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:28.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:28.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:28.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:27 smithi123 bash[19119]: cluster 2024-02-12T16:42:26.236364+0000 osd.1 (osd.1) 161 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:42:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:27 smithi123 bash[19119]: cluster 2024-02-12T16:42:26.378302+0000 osd.1 (osd.1) 162 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:42:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:27 smithi123 bash[19119]: cluster 2024-02-12T16:42:26.608266+0000 osd.11 (osd.11) 217 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:42:28.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:27 smithi123 bash[19119]: cluster 2024-02-12T16:42:26.611850+0000 osd.11 (osd.11) 218 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:42:28.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:28.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:28.249 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:27 smithi121 bash[18489]: cluster 2024-02-12T16:42:26.236364+0000 osd.1 (osd.1) 161 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:42:28.249 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:27 smithi121 bash[18489]: cluster 2024-02-12T16:42:26.378302+0000 osd.1 (osd.1) 162 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:42:28.249 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:27 smithi121 bash[18489]: cluster 2024-02-12T16:42:26.608266+0000 osd.11 (osd.11) 217 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:42:28.250 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:27 smithi121 bash[18489]: cluster 2024-02-12T16:42:26.611850+0000 osd.11 (osd.11) 218 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:42:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:28.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:27 smithi169 bash[19604]: cluster 2024-02-12T16:42:26.236364+0000 osd.1 (osd.1) 161 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:42:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:27 smithi169 bash[19604]: cluster 2024-02-12T16:42:26.378302+0000 osd.1 (osd.1) 162 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:42:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:27 smithi169 bash[19604]: cluster 2024-02-12T16:42:26.608266+0000 osd.11 (osd.11) 217 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:42:28.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:27 smithi169 bash[19604]: cluster 2024-02-12T16:42:26.611850+0000 osd.11 (osd.11) 218 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:42:28.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:28.360 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8, 11] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3] 2024-02-12T16:42:28.360 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:42:28.360 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-02-12T16:42:28.360 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-02-12T16:42:28.360 DEBUG:teuthology.orchestra.run.smithi123:osd.7> sudo journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.7.service 2024-02-12T16:42:28.388 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:28.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl start ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.7 2024-02-12T16:42:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:28.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:28.400 INFO:journalctl@ceph.osd.7.smithi123.stdout:-- Logs begin at Sun 2022-10-16 12:17:23 UTC. -- 2024-02-12T16:42:28.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:42:28.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:28.525 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:28.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:28.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:28.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:28 smithi123 systemd[1]: Started Ceph osd.7 for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T16:42:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:28.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:28.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:28.799 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:28.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:28.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:28.935 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:28.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:28.991 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:29.027 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:42:29.027 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:42:29.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:42:29.027 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303055 6.17 6.6feb8017 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.124172+0000", 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 539.89585631900002, 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.68241777299999995, 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303055 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124172+0000", 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124172+0000", 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999948 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124167+0000", 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5682999999999999e-05 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124183+0000", 2024-02-12T16:42:29.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6696e-05 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124210+0000", 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.433e-05 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124224+0000", 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1990000000000006e-05 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124306+0000", 2024-02-12T16:42:29.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9810000000000003e-06 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.124314+0000", 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.68226310099999998 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806577+0000", 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3002e-05 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806590+0000", 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.68241777299999995 2024-02-12T16:42:29.031 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303097 6.17 6.608bff37 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.175939+0000", 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 539.844089467, 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63069973599999996, 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:42:29.032 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303097 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175939+0000", 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175939+0000", 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999957 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175935+0000", 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4701e-05 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175949+0000", 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2152000000000001e-05 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175972+0000", 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3624e-05 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.175985+0000", 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9125000000000001e-05 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.176034+0000", 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7129999999999999e-06 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.176041+0000", 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.630590922 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806632+0000", 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8449999999999997e-06 2024-02-12T16:42:29.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806639+0000", 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63069973599999996 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26257.0:303129 6.17 6.d8224497 (undecoded) ondisk+write+known_if_redirected e732)", 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:33:29.205408+0000", 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 539.81462012099996, 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.60125489899999995, 2024-02-12T16:42:29.036 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26257", 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3310897821", 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 303129 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205408+0000", 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205408+0000", 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:42:29.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205405+0000", 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0176999999999999e-05 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205416+0000", 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5583999999999999e-05 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205431+0000", 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2930000000000003e-06 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205439+0000", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0902999999999997e-05 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205489+0000", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9179999999999997e-06 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.205496+0000", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.60115846799999995 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806655+0000", 2024-02-12T16:42:29.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4719999999999995e-06 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:33:29.806663+0000", 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.60125489899999995 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106015 8.12 8:4d0dbb88:::benchmark_data_smithi169_115677_object23387:head [delete] snapc 0=[] ondisk+write+known_if_redirected e882)", 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:58.360267+0000", 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 90.659761841000005, 2024-02-12T16:42:29.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0432394679999999, 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106015 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360267+0000", 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360267+0000", 2024-02-12T16:42:29.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360264+0000", 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9529999999999999e-06 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360272+0000", 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0678e-05 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360283+0000", 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7360000000000004e-06 2024-02-12T16:42:29.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360290+0000", 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.166646664 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.526936+0000", 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2949999999999996e-06 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.526944+0000", 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87383641700000003 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.400780+0000", 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109707 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.400890+0000", 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9071000000000002e-05 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0", 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.400979+0000", 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024375270000000001 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403416+0000", 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.241e-05 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403429+0000", 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1751000000000001e-05 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403470+0000", 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5691000000000001e-05 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403506+0000", 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0432394679999999 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106019 8.16 8.e008b6b6 (undecoded) ondisk+write+known_if_redirected e882)", 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:58.368844+0000", 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 90.651184717000007, 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.012615437, 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.046 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106019 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368844+0000", 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368844+0000", 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368840+0000", 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0348e-05 2024-02-12T16:42:29.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368851+0000", 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3125000000000001e-05 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368864+0000", 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3629999999999996e-06 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368871+0000", 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15464666199999999 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.523518+0000", 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.977e-06 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.523526+0000", 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.857852783 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381379+0000", 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0574999999999993e-05 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381459+0000", 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.012615437 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106036 8.16 8.8b727756 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.408773+0000", 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.611255335999999, 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97285082300000003, 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106036 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.050 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408773+0000", 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408773+0000", 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408771+0000", 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0509999999999997e-06 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408779+0000", 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2703e-05 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408832+0000", 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115853 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408948+0000", 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0973999999999997e-05 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408999+0000", 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.524e-06 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409007+0000", 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97260508099999998 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381612+0000", 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1589e-05 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381624+0000", 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97285082300000003 2024-02-12T16:42:29.053 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106038 8.16 8.8e8b3fd6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.408988+0000", 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.611040936999999, 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97266427899999996, 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106038 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408988+0000", 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408988+0000", 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408986+0000", 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0949999999999998e-06 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408993+0000", 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4509999999999992e-06 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409001+0000", 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7910000000000001e-06 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409007+0000", 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5868000000000002e-05 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409053+0000", 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2830000000000004e-06 2024-02-12T16:42:29.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409060+0000", 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97258542299999995 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381646+0000", 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.993e-06 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381652+0000", 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97266427899999996 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106043 8.16 8.f5bba616 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.415033+0000", 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.604995535, 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96665297100000003, 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106043 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415033+0000", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415033+0000", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415031+0000", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6410000000000002e-06 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415039+0000", 2024-02-12T16:42:29.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3165999999999999e-05 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415092+0000", 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6299999999999993e-06 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415102+0000", 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1401000000000004e-05 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415163+0000", 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0858e-05 2024-02-12T16:42:29.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415174+0000", 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96650533500000002 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381679+0000", 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.708e-06 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381686+0000", 2024-02-12T16:42:29.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96665297100000003 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106103 8.16 8.720be176 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.457924+0000", 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.562104250999994, 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.92378728799999998, 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106103 2024-02-12T16:42:29.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457924+0000", 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457924+0000", 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457921+0000", 2024-02-12T16:42:29.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457931+0000", 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5812999999999999e-05 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457947+0000", 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3550000000000005e-06 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457956+0000", 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8015e-05 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.458004+0000", 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2286000000000001e-05 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.458016+0000", 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.92368782999999999 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381704+0000", 2024-02-12T16:42:29.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.79e-06 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381712+0000", 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.92378728799999998 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106122 8.16 8.5e63e396 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.468415+0000", 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.551613302000007, 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91331797599999998, 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.066 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106122 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468415+0000", 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468415+0000", 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468414+0000", 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8350000000000003e-06 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468419+0000", 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5565e-05 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468444+0000", 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0990000000000004e-06 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468450+0000", 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8056000000000002e-05 2024-02-12T16:42:29.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468498+0000", 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4099999999999997e-06 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468508+0000", 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91321966300000001 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381727+0000", 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8629999999999996e-06 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381733+0000", 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91331797599999998 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106151 8.16 8.48750896 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.481942+0000", 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.538086226000004, 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89981281300000004, 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106151 2024-02-12T16:42:29.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481942+0000", 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481942+0000", 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481940+0000", 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1019999999999999e-06 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.071 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481947+0000", 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.381e-05 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481961+0000", 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0169999999999999e-06 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481968+0000", 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4472000000000003e-05 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.482023+0000", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4709999999999996e-06 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.482032+0000", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89971682799999997 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381749+0000", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0480000000000004e-06 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381755+0000", 2024-02-12T16:42:29.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89981281300000004 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106158 8.16 8.eba0cfd6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.485343+0000", 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.534685323000005, 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89643377400000002, 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.074 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106158 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485343+0000", 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485343+0000", 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485341+0000", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3170000000000002e-06 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485349+0000", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4054e-05 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485363+0000", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.674e-06 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485370+0000", 2024-02-12T16:42:29.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8694000000000003e-05 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485439+0000", 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1878999999999999e-05 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485451+0000", 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89631989000000001 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381771+0000", 2024-02-12T16:42:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4330000000000001e-06 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381777+0000", 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89643377400000002 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106172 8.16 8.6585dbd6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.493796+0000", 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.526232508999996, 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88803040600000005, 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106172 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493796+0000", 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493796+0000", 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493794+0000", 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2820000000000004e-06 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493800+0000", 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4489999999999999e-06 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493808+0000", 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9259999999999999e-06 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493813+0000", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5454000000000002e-05 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493859+0000", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3679999999999996e-06 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493866+0000", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88795410500000005 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381820+0000", 2024-02-12T16:42:29.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3890000000000002e-06 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381826+0000", 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88803040600000005 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106277 8.16 8.7fbde36 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.560827+0000", 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.459201476000004, 2024-02-12T16:42:29.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82102122700000002, 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106277 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560827+0000", 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560827+0000", 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560825+0000", 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2920000000000001e-06 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560831+0000", 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0268000000000001e-05 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560841+0000", 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8590000000000004e-06 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560846+0000", 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8593000000000001e-05 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560885+0000", 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3249999999999998e-06 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560890+0000", 2024-02-12T16:42:29.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82095215499999996 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381842+0000", 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410000000000002e-06 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381848+0000", 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82102122700000002 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106283 8.16 8.641c5ef6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.563678+0000", 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.456350197000006, 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81818920500000003, 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106283 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563678+0000", 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563678+0000", 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563677+0000", 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6919999999999996e-06 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563684+0000", 2024-02-12T16:42:29.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2099000000000001e-05 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563696+0000", 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7170000000000002e-06 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563702+0000", 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9125999999999997e-05 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563751+0000", 2024-02-12T16:42:29.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7589999999999997e-06 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563759+0000", 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81810208600000001 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381861+0000", 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1639999999999998e-06 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381868+0000", 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81818920500000003 2024-02-12T16:42:29.090 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106306 8.16 8.f15ebc16 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.579963+0000", 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.440065532000006, 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80192957499999995, 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106306 2024-02-12T16:42:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579963+0000", 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579963+0000", 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579962+0000", 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7659999999999998e-06 2024-02-12T16:42:29.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579966+0000", 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4961e-05 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579981+0000", 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2300000000000002e-06 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579986+0000", 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4341000000000001e-05 2024-02-12T16:42:29.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.580030+0000", 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1549999999999998e-06 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.580035+0000", 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80185269400000003 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381888+0000", 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6859999999999997e-06 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381893+0000", 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80192957499999995 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106316 8.16 8.8ca12656 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.588489+0000", 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.431539510999997, 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.79342752000000005, 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.095 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106316 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588489+0000", 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588489+0000", 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588487+0000", 2024-02-12T16:42:29.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2480000000000001e-06 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588493+0000", 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2643000000000001e-05 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588506+0000", 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8010000000000004e-06 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588514+0000", 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4776e-05 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588559+0000", 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6049999999999997e-06 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588564+0000", 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.79334808199999995 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381912+0000", 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1019999999999999e-06 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381917+0000", 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.79342752000000005 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106402 8.16 8.59a95bf6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.683695+0000", 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.336333456000006, 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.69824561600000001, 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.099 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106402 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683695+0000", 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683695+0000", 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683694+0000", 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1589999999999998e-06 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683700+0000", 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7410000000000004e-06 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683709+0000", 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3750000000000002e-06 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683715+0000", 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0726999999999998e-05 2024-02-12T16:42:29.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683756+0000", 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2310000000000004e-06 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683761+0000", 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.69817597200000003 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381937+0000", 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9210000000000002e-06 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381941+0000", 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.69824561600000001 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106432 8.16 8.75870976 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.724576+0000", 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.295452939, 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65739000999999997, 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106432 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724576+0000", 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724576+0000", 2024-02-12T16:42:29.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724574+0000", 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2650000000000004e-06 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724579+0000", 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7809999999999994e-06 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724588+0000", 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6969999999999996e-06 2024-02-12T16:42:29.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724595+0000", 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4279000000000001e-05 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724639+0000", 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1219999999999995e-06 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724647+0000", 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65731423300000003 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381961+0000", 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0840000000000002e-06 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381966+0000", 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65739000999999997 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106443 8.16 8.4816aff6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.750934+0000", 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 89.269094013, 2024-02-12T16:42:29.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63104969, 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106443 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750934+0000", 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:42:29.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750934+0000", 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750932+0000", 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.525e-06 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750940+0000", 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6228999999999999e-05 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750966+0000", 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6800000000000005e-06 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750976+0000", 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6941999999999997e-05 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.751043+0000", 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0635000000000001e-05 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.751053+0000", 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63092733499999998 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:42:29.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381981+0000", 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6239999999999999e-06 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381984+0000", 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63104969 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:42:29.111 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:42:29.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:29.112 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:28 smithi169 bash[19604]: cluster 2024-02-12T16:42:28.260798+0000 mgr.y (mgr.14146) 1744 : cluster [DBG] pgmap v2533: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:29.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:28 smithi121 bash[18489]: cluster 2024-02-12T16:42:28.260798+0000 mgr.y (mgr.14146) 1744 : cluster [DBG] pgmap v2533: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:29.150 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:29.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:29.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:29.291 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:29.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:28 smithi123 bash[19119]: cluster 2024-02-12T16:42:28.260798+0000 mgr.y (mgr.14146) 1744 : cluster [DBG] pgmap v2533: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:29.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:29.430 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:29.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:29.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:29.568 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:29.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:29.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:29.705 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:29.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:29.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-12T16:42:29.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:29.838 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:29.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:29.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:29.978 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:29.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:30.081 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:30.120 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:30.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:30.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:30.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:30.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:30 smithi169 bash[19604]: cluster 2024-02-12T16:42:29.316611+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:42:30.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:30 smithi169 bash[19604]: cluster 2024-02-12T16:42:29.489289+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:42:30.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:30.375 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:30 smithi121 bash[18489]: cluster 2024-02-12T16:42:29.316611+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:42:30.375 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:30 smithi121 bash[18489]: cluster 2024-02-12T16:42:29.489289+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:42:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[19119]: cluster 2024-02-12T16:42:29.316611+0000 osd.1 (osd.1) 163 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:42:30.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[19119]: cluster 2024-02-12T16:42:29.489289+0000 osd.1 (osd.1) 164 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:42:30.406 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:30.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:30.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:30.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:30.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:30.690 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:30.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:30.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:30.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:30.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:30.889 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-12T16:42:30.889 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-12T16:42:30.889 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2024-02-12T16:42:30.889 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-12T16:42:30.889 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2024-02-12T16:42:30.890 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-12T16:42:30.890 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-12T16:42:30.890 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:30 smithi123 bash[123421]: --> ceph-volume lvm activate successful for osd ID: 7 2024-02-12T16:42:30.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:30.962 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:30.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:31.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:31.109 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:31.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:31.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:31.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:31.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:31.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:31.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:31 smithi121 bash[18489]: cluster 2024-02-12T16:42:30.261797+0000 mgr.y (mgr.14146) 1745 : cluster [DBG] pgmap v2534: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:31.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:31 smithi123 bash[19119]: cluster 2024-02-12T16:42:30.261797+0000 mgr.y (mgr.14146) 1745 : cluster [DBG] pgmap v2534: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:31.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:31.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:31.503 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:31 smithi169 bash[19604]: cluster 2024-02-12T16:42:30.261797+0000 mgr.y (mgr.14146) 1745 : cluster [DBG] pgmap v2534: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:31.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:31.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:31.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:31.674 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:31.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:31.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:31.816 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:31.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:31.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:31.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:32.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:32.092 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:32.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:32.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:32.237 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:32.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:32.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:32.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:32 smithi121 bash[18489]: cluster 2024-02-12T16:42:30.715477+0000 osd.9 (osd.9) 159 : cluster [DBG] 1.a scrub starts 2024-02-12T16:42:32.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:32 smithi121 bash[18489]: cluster 2024-02-12T16:42:30.755990+0000 osd.9 (osd.9) 160 : cluster [DBG] 1.a scrub ok 2024-02-12T16:42:32.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:32 smithi121 bash[18489]: cluster 2024-02-12T16:42:30.844923+0000 osd.4 (osd.4) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:32.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:32 smithi121 bash[18489]: cluster 2024-02-12T16:42:30.845003+0000 osd.4 (osd.4) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:32.354 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:32 smithi121 bash[18489]: cluster 2024-02-12T16:42:31.095144+0000 mon.a (mon.0) 3093 : cluster [DBG] osdmap e919: 12 total, 11 up, 12 in 2024-02-12T16:42:32.378 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:32.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:32.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:32 smithi123 bash[19119]: cluster 2024-02-12T16:42:30.715477+0000 osd.9 (osd.9) 159 : cluster [DBG] 1.a scrub starts 2024-02-12T16:42:32.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:32 smithi123 bash[19119]: cluster 2024-02-12T16:42:30.755990+0000 osd.9 (osd.9) 160 : cluster [DBG] 1.a scrub ok 2024-02-12T16:42:32.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:32 smithi123 bash[19119]: cluster 2024-02-12T16:42:30.844923+0000 osd.4 (osd.4) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:32.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:32 smithi123 bash[19119]: cluster 2024-02-12T16:42:30.845003+0000 osd.4 (osd.4) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:32.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:32 smithi123 bash[19119]: cluster 2024-02-12T16:42:31.095144+0000 mon.a (mon.0) 3093 : cluster [DBG] osdmap e919: 12 total, 11 up, 12 in 2024-02-12T16:42:32.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:32.523 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:32.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:32 smithi169 bash[19604]: cluster 2024-02-12T16:42:30.715477+0000 osd.9 (osd.9) 159 : cluster [DBG] 1.a scrub starts 2024-02-12T16:42:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:32 smithi169 bash[19604]: cluster 2024-02-12T16:42:30.755990+0000 osd.9 (osd.9) 160 : cluster [DBG] 1.a scrub ok 2024-02-12T16:42:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:32 smithi169 bash[19604]: cluster 2024-02-12T16:42:30.844923+0000 osd.4 (osd.4) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:32 smithi169 bash[19604]: cluster 2024-02-12T16:42:30.845003+0000 osd.4 (osd.4) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:32 smithi169 bash[19604]: cluster 2024-02-12T16:42:31.095144+0000 mon.a (mon.0) 3093 : cluster [DBG] osdmap e919: 12 total, 11 up, 12 in 2024-02-12T16:42:32.628 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:32.665 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:32.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:32.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:32.806 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:32.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:32.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:32.923 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:42:32.939 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:32.943 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:42:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:32.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:32.979 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:42:33.011 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:42:33.034 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:42:33.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:33.062 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:42:33.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:33.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:33.092 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:42:33.117 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:42:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:42:33.139 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:32 smithi123 bash[123421]: debug 2024-02-12T16:42:32.872+0000 7f77ce85b380 -1 Falling back to public interface 2024-02-12T16:42:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:33 smithi123 bash[19119]: cluster 2024-02-12T16:42:31.494478+0000 osd.5 (osd.5) 213 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:42:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:33 smithi123 bash[19119]: cluster 2024-02-12T16:42:31.552612+0000 osd.5 (osd.5) 214 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:42:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:33 smithi123 bash[19119]: cluster 2024-02-12T16:42:32.262838+0000 mgr.y (mgr.14146) 1746 : cluster [DBG] pgmap v2536: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:33.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:33 smithi123 bash[19119]: audit 2024-02-12T16:42:32.897311+0000 mon.a (mon.0) 3094 : audit [DBG] from='client.? v1:172.21.15.121:0/2610611307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:33.167 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:42:33.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:33.193 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:42:33.193 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.194 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:42:33.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:33.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:33.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:33.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:33 smithi121 bash[18489]: cluster 2024-02-12T16:42:31.494478+0000 osd.5 (osd.5) 213 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:42:33.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:33 smithi121 bash[18489]: cluster 2024-02-12T16:42:31.552612+0000 osd.5 (osd.5) 214 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:42:33.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:33 smithi121 bash[18489]: cluster 2024-02-12T16:42:32.262838+0000 mgr.y (mgr.14146) 1746 : cluster [DBG] pgmap v2536: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:33.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:33 smithi121 bash[18489]: audit 2024-02-12T16:42:32.897311+0000 mon.a (mon.0) 3094 : audit [DBG] from='client.? v1:172.21.15.121:0/2610611307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:33.353 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:33.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:33.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:33.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:33.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:33 smithi169 bash[19604]: cluster 2024-02-12T16:42:31.494478+0000 osd.5 (osd.5) 213 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:42:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:33 smithi169 bash[19604]: cluster 2024-02-12T16:42:31.552612+0000 osd.5 (osd.5) 214 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:42:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:33 smithi169 bash[19604]: cluster 2024-02-12T16:42:32.262838+0000 mgr.y (mgr.14146) 1746 : cluster [DBG] pgmap v2536: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:33 smithi169 bash[19604]: audit 2024-02-12T16:42:32.897311+0000 mon.a (mon.0) 3094 : audit [DBG] from='client.? v1:172.21.15.121:0/2610611307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:33.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:33.631 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:33.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:33.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:33.771 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:33.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:33.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:33.913 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:33.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:34.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:34.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:34.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:34.127 INFO:teuthology.orchestra.run.smithi121.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-02-12T16:42:34.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:34.190 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:34.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:34.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:34 smithi169 bash[19604]: cluster 2024-02-12T16:42:32.758176+0000 osd.2 (osd.2) 133 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:42:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:34 smithi169 bash[19604]: cluster 2024-02-12T16:42:32.870852+0000 osd.2 (osd.2) 134 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:42:34.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:34 smithi169 bash[19604]: audit 2024-02-12T16:42:33.239992+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? v1:172.21.15.121:0/2579333865' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:42:34.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:34 smithi121 bash[18489]: cluster 2024-02-12T16:42:32.758176+0000 osd.2 (osd.2) 133 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:42:34.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:34 smithi121 bash[18489]: cluster 2024-02-12T16:42:32.870852+0000 osd.2 (osd.2) 134 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:42:34.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:34 smithi121 bash[18489]: audit 2024-02-12T16:42:33.239992+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? v1:172.21.15.121:0/2579333865' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:42:34.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:34.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:34 smithi123 bash[19119]: cluster 2024-02-12T16:42:32.758176+0000 osd.2 (osd.2) 133 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:42:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:34 smithi123 bash[19119]: cluster 2024-02-12T16:42:32.870852+0000 osd.2 (osd.2) 134 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:42:34.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:34 smithi123 bash[19119]: audit 2024-02-12T16:42:33.239992+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? v1:172.21.15.121:0/2579333865' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-12T16:42:34.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:34.471 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:34.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:34.533 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:42:34.575 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:34.614 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:34.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:34.696 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:42:34.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:34.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:34.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:34.860 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:34.897 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:34.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:35.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:35.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:35.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:35.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:35.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-12T16:42:35.182 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-12T16:42:35.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:35.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:35.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:35 smithi123 bash[19119]: cluster 2024-02-12T16:42:33.643061+0000 osd.11 (osd.11) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:35 smithi123 bash[19119]: cluster 2024-02-12T16:42:33.643154+0000 osd.11 (osd.11) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:35 smithi123 bash[19119]: audit 2024-02-12T16:42:34.115840+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? v1:172.21.15.121:0/2579333865' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:42:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:35 smithi123 bash[19119]: cluster 2024-02-12T16:42:34.115926+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e920: 12 total, 11 up, 12 in 2024-02-12T16:42:35.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:35 smithi123 bash[19119]: cluster 2024-02-12T16:42:34.263754+0000 mgr.y (mgr.14146) 1747 : cluster [DBG] pgmap v2538: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:35 smithi121 bash[18489]: cluster 2024-02-12T16:42:33.643061+0000 osd.11 (osd.11) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:35 smithi121 bash[18489]: cluster 2024-02-12T16:42:33.643154+0000 osd.11 (osd.11) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:35 smithi121 bash[18489]: audit 2024-02-12T16:42:34.115840+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? v1:172.21.15.121:0/2579333865' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:42:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:35 smithi121 bash[18489]: cluster 2024-02-12T16:42:34.115926+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e920: 12 total, 11 up, 12 in 2024-02-12T16:42:35.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:35 smithi121 bash[18489]: cluster 2024-02-12T16:42:34.263754+0000 mgr.y (mgr.14146) 1747 : cluster [DBG] pgmap v2538: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:35.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:35.453 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:35.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:35 smithi169 bash[19604]: cluster 2024-02-12T16:42:33.643061+0000 osd.11 (osd.11) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:35 smithi169 bash[19604]: cluster 2024-02-12T16:42:33.643154+0000 osd.11 (osd.11) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:35 smithi169 bash[19604]: audit 2024-02-12T16:42:34.115840+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? v1:172.21.15.121:0/2579333865' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-12T16:42:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:35 smithi169 bash[19604]: cluster 2024-02-12T16:42:34.115926+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e920: 12 total, 11 up, 12 in 2024-02-12T16:42:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:35 smithi169 bash[19604]: cluster 2024-02-12T16:42:34.263754+0000 mgr.y (mgr.14146) 1747 : cluster [DBG] pgmap v2538: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:35.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:35.592 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:35.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:35.696 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:35.727 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:35.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:35.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:35.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:35.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:35.968 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:36.003 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:36.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:36.130 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:36.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:36.270 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:36 smithi169 bash[19604]: cluster 2024-02-12T16:42:34.775057+0000 osd.10 (osd.10) 232 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:42:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:36 smithi169 bash[19604]: cluster 2024-02-12T16:42:34.813148+0000 osd.10 (osd.10) 233 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:42:36.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:36.385 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:36 smithi123 bash[19119]: cluster 2024-02-12T16:42:34.775057+0000 osd.10 (osd.10) 232 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:42:36.385 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:36 smithi123 bash[19119]: cluster 2024-02-12T16:42:34.813148+0000 osd.10 (osd.10) 233 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:42:36.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:36.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:36 smithi121 bash[18489]: cluster 2024-02-12T16:42:34.775057+0000 osd.10 (osd.10) 232 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:42:36.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:36 smithi121 bash[18489]: cluster 2024-02-12T16:42:34.813148+0000 osd.10 (osd.10) 233 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:42:36.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:36.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:36.566 DEBUG:teuthology.orchestra.run.smithi169:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-02-12T16:42:36.596 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-12T16:42:36.596 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-12T16:42:36.642 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:hints = 1 2024-02-12T16:42:36.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-12T16:42:36.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object prefix: benchmark_data_smithi169_124168 2024-02-12T16:42:36.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:42:36.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 0 0 0 0 0 0 - 0 2024-02-12T16:42:36.652 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:36.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:36.781 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:36.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:36.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:36.962 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:36.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:37.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:37.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:37.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:37.152 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:42:37.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:42:37.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:37.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:37.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:37.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:37.367 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:37.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:37 smithi123 bash[19119]: cluster 2024-02-12T16:42:35.805028+0000 osd.8 (osd.8) 137 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:42:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:37 smithi123 bash[19119]: cluster 2024-02-12T16:42:35.918511+0000 osd.8 (osd.8) 138 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:42:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:37 smithi123 bash[19119]: audit 2024-02-12T16:42:36.160361+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.121:0/3731167202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:42:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:37 smithi123 bash[19119]: cluster 2024-02-12T16:42:36.264724+0000 mgr.y (mgr.14146) 1748 : cluster [DBG] pgmap v2539: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:37 smithi121 bash[18489]: cluster 2024-02-12T16:42:35.805028+0000 osd.8 (osd.8) 137 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:42:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:37 smithi121 bash[18489]: cluster 2024-02-12T16:42:35.918511+0000 osd.8 (osd.8) 138 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:42:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:37 smithi121 bash[18489]: audit 2024-02-12T16:42:36.160361+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.121:0/3731167202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:42:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:37 smithi121 bash[18489]: cluster 2024-02-12T16:42:36.264724+0000 mgr.y (mgr.14146) 1748 : cluster [DBG] pgmap v2539: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:37.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:37.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:37 smithi169 bash[19604]: cluster 2024-02-12T16:42:35.805028+0000 osd.8 (osd.8) 137 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:42:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:37 smithi169 bash[19604]: cluster 2024-02-12T16:42:35.918511+0000 osd.8 (osd.8) 138 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:42:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:37 smithi169 bash[19604]: audit 2024-02-12T16:42:36.160361+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.121:0/3731167202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:42:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:37 smithi169 bash[19604]: cluster 2024-02-12T16:42:36.264724+0000 mgr.y (mgr.14146) 1748 : cluster [DBG] pgmap v2539: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 0 B data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:37.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:37.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:37.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 1 16 77 61 3.81207 3.8125 0.00140402 0.00292698 2024-02-12T16:42:37.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:37.778 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:37.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:37.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:37.923 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:37.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:38.026 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:42:38.027 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:42:38.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:no valid command found; 10 closest matches: 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:0 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:1 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:2 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:abort 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:assert 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:bluefs debug_inject_read_zeros 2024-02-12T16:42:38.043 INFO:teuthology.orchestra.run.smithi123.stderr:bluefs files list 2024-02-12T16:42:38.044 INFO:teuthology.orchestra.run.smithi123.stderr:bluefs stats 2024-02-12T16:42:38.044 INFO:teuthology.orchestra.run.smithi123.stderr:bluestore allocator dump block 2024-02-12T16:42:38.044 INFO:teuthology.orchestra.run.smithi123.stderr:bluestore allocator fragmentation block 2024-02-12T16:42:38.044 INFO:teuthology.orchestra.run.smithi123.stderr:admin_socket: invalid command 2024-02-12T16:42:38.105 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:38.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:38 smithi121 bash[18489]: cluster 2024-02-12T16:42:37.124128+0000 mon.a (mon.0) 3099 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:42:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:38 smithi121 bash[18489]: audit 2024-02-12T16:42:37.139561+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? v1:172.21.15.121:0/3731167202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:42:38.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:38 smithi121 bash[18489]: cluster 2024-02-12T16:42:37.139732+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e921: 12 total, 11 up, 12 in 2024-02-12T16:42:38.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:38.245 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:38.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:38.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:38.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:38.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:38 smithi123 bash[19119]: cluster 2024-02-12T16:42:37.124128+0000 mon.a (mon.0) 3099 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:42:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:38 smithi123 bash[19119]: audit 2024-02-12T16:42:37.139561+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? v1:172.21.15.121:0/3731167202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:42:38.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:38 smithi123 bash[19119]: cluster 2024-02-12T16:42:37.139732+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e921: 12 total, 11 up, 12 in 2024-02-12T16:42:38.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:38.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:38 smithi169 bash[19604]: cluster 2024-02-12T16:42:37.124128+0000 mon.a (mon.0) 3099 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:42:38.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:38 smithi169 bash[19604]: audit 2024-02-12T16:42:37.139561+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? v1:172.21.15.121:0/3731167202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:42:38.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:38 smithi169 bash[19604]: cluster 2024-02-12T16:42:37.139732+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e921: 12 total, 11 up, 12 in 2024-02-12T16:42:38.530 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:38.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:38.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:38.643 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 2 16 77 61 1.90598 0 - 0.00292698 2024-02-12T16:42:38.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:38.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-12T16:42:38.774 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-02-12T16:42:38.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:38.780 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-12T16:42:38.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:38.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:38.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:38.944 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:38.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:39.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:39.083 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:39.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:39.139 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:38 smithi123 bash[123421]: debug 2024-02-12T16:42:38.936+0000 7f77ce85b380 -1 osd.7 848 log_to_monitors {default=true} 2024-02-12T16:42:39.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:39.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:39.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:39.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:39.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:39 smithi121 bash[18489]: cluster 2024-02-12T16:42:38.210565+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:39.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:39 smithi121 bash[18489]: cluster 2024-02-12T16:42:38.210658+0000 osd.6 (osd.6) 185 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:39.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:39 smithi121 bash[18489]: cluster 2024-02-12T16:42:38.265640+0000 mgr.y (mgr.14146) 1749 : cluster [DBG] pgmap v2541: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 2 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:39.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:39 smithi121 bash[18489]: cluster 2024-02-12T16:42:38.436474+0000 osd.11 (osd.11) 221 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:42:39.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:39 smithi121 bash[18489]: cluster 2024-02-12T16:42:38.439658+0000 osd.11 (osd.11) 222 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:42:39.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:39 smithi121 bash[18489]: audit 2024-02-12T16:42:38.941748+0000 mon.a (mon.0) 3102 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:42:39.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:39 smithi121 bash[18489]: audit 2024-02-12T16:42:38.943767+0000 mon.b (mon.2) 219 : audit [INF] from='osd.7 v1:172.21.15.123:6812/4284140534' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:42:39.349 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:39.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:39.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:39.491 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:39.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:39 smithi169 bash[19604]: cluster 2024-02-12T16:42:38.210565+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:39 smithi169 bash[19604]: cluster 2024-02-12T16:42:38.210658+0000 osd.6 (osd.6) 185 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:39 smithi169 bash[19604]: cluster 2024-02-12T16:42:38.265640+0000 mgr.y (mgr.14146) 1749 : cluster [DBG] pgmap v2541: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 2 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:39 smithi169 bash[19604]: cluster 2024-02-12T16:42:38.436474+0000 osd.11 (osd.11) 221 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:42:39.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:39 smithi169 bash[19604]: cluster 2024-02-12T16:42:38.439658+0000 osd.11 (osd.11) 222 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:42:39.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:39 smithi169 bash[19604]: audit 2024-02-12T16:42:38.941748+0000 mon.a (mon.0) 3102 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:42:39.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:39 smithi169 bash[19604]: audit 2024-02-12T16:42:38.943767+0000 mon.b (mon.2) 219 : audit [INF] from='osd.7 v1:172.21.15.123:6812/4284140534' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:42:39.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:39.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:39.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:39.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[123421]: debug 2024-02-12T16:42:39.180+0000 7f77be53f700 -1 osd.7 922 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-12T16:42:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[19119]: cluster 2024-02-12T16:42:38.210565+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[19119]: cluster 2024-02-12T16:42:38.210658+0000 osd.6 (osd.6) 185 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:39.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[19119]: cluster 2024-02-12T16:42:38.265640+0000 mgr.y (mgr.14146) 1749 : cluster [DBG] pgmap v2541: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 2 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:39.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[19119]: cluster 2024-02-12T16:42:38.436474+0000 osd.11 (osd.11) 221 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:42:39.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[19119]: cluster 2024-02-12T16:42:38.439658+0000 osd.11 (osd.11) 222 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:42:39.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[19119]: audit 2024-02-12T16:42:38.941748+0000 mon.a (mon.0) 3102 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:42:39.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:39 smithi123 bash[19119]: audit 2024-02-12T16:42:38.943767+0000 mon.b (mon.2) 219 : audit [INF] from='osd.7 v1:172.21.15.123:6812/4284140534' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-12T16:42:39.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 3 16 77 61 1.27066 0 - 0.00292698 2024-02-12T16:42:39.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:39.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:39.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:39.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:39.905 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:39.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:40.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:40.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:42:40.035 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:40.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:40.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:40.173 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:40.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:40.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:40.313 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:40.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:39.144451+0000 mon.a (mon.0) 3103 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T16:42:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: cluster 2024-02-12T16:42:39.144614+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e922: 12 total, 11 up, 12 in 2024-02-12T16:42:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:39.146181+0000 mon.a (mon.0) 3105 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T16:42:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:39.148422+0000 mon.b (mon.2) 220 : audit [INF] from='osd.7 v1:172.21.15.123:6812/4284140534' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T16:42:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:39.463153+0000 mon.a (mon.0) 3106 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:42:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:39.755371+0000 mon.a (mon.0) 3107 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:40.025662+0000 mon.a (mon.0) 3108 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:42:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:40.027308+0000 mon.a (mon.0) 3109 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:42:40.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:40 smithi121 bash[18489]: audit 2024-02-12T16:42:40.036271+0000 mon.a (mon.0) 3110 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:40.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:40.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:40.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:39.144451+0000 mon.a (mon.0) 3103 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T16:42:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: cluster 2024-02-12T16:42:39.144614+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e922: 12 total, 11 up, 12 in 2024-02-12T16:42:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:39.146181+0000 mon.a (mon.0) 3105 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T16:42:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:39.148422+0000 mon.b (mon.2) 220 : audit [INF] from='osd.7 v1:172.21.15.123:6812/4284140534' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T16:42:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:39.463153+0000 mon.a (mon.0) 3106 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:42:40.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:39.755371+0000 mon.a (mon.0) 3107 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:40.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:40.025662+0000 mon.a (mon.0) 3108 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:42:40.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:40.027308+0000 mon.a (mon.0) 3109 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:42:40.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:40 smithi169 bash[19604]: audit 2024-02-12T16:42:40.036271+0000 mon.a (mon.0) 3110 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:40.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:40.594 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:40.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:39.144451+0000 mon.a (mon.0) 3103 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-12T16:42:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: cluster 2024-02-12T16:42:39.144614+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e922: 12 total, 11 up, 12 in 2024-02-12T16:42:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:39.146181+0000 mon.a (mon.0) 3105 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T16:42:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:39.148422+0000 mon.b (mon.2) 220 : audit [INF] from='osd.7 v1:172.21.15.123:6812/4284140534' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-02-12T16:42:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:39.463153+0000 mon.a (mon.0) 3106 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:42:40.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:39.755371+0000 mon.a (mon.0) 3107 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:40.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:40.025662+0000 mon.a (mon.0) 3108 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:42:40.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:40.027308+0000 mon.a (mon.0) 3109 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:42:40.641 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:40 smithi123 bash[19119]: audit 2024-02-12T16:42:40.036271+0000 mon.a (mon.0) 3110 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:40.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 4 16 77 61 0.95299 0 - 0.00292698 2024-02-12T16:42:40.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:40.726 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:40.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:40.829 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:40.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:40.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:40.972 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:41.009 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:41.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:41.148 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:41.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:41.283 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:41.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:41.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: cluster 2024-02-12T16:42:40.138694+0000 mon.a (mon.0) 3111 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T16:42:41.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: cluster 2024-02-12T16:42:40.152326+0000 mon.a (mon.0) 3112 : cluster [INF] osd.7 v1:172.21.15.123:6812/4284140534 boot 2024-02-12T16:42:41.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: cluster 2024-02-12T16:42:40.152383+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e923: 12 total, 12 up, 12 in 2024-02-12T16:42:41.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.154723+0000 mon.a (mon.0) 3114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:42:41.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.155060+0000 mon.a (mon.0) 3115 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:42:41.395 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.155273+0000 mon.a (mon.0) 3116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:42:41.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.155496+0000 mon.a (mon.0) 3117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:42:41.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.206231+0000 mon.a (mon.0) 3118 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:42:41.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.209182+0000 mon.a (mon.0) 3119 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:42:41.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.210470+0000 mon.a (mon.0) 3120 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:42:41.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: audit 2024-02-12T16:42:40.219514+0000 mon.a (mon.0) 3121 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:41.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: cluster 2024-02-12T16:42:40.266401+0000 mgr.y (mgr.14146) 1750 : cluster [DBG] pgmap v2544: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 2.9 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:41.396 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:41 smithi169 bash[19604]: cluster 2024-02-12T16:42:40.962403+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e924: 12 total, 12 up, 12 in 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: cluster 2024-02-12T16:42:40.138694+0000 mon.a (mon.0) 3111 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: cluster 2024-02-12T16:42:40.152326+0000 mon.a (mon.0) 3112 : cluster [INF] osd.7 v1:172.21.15.123:6812/4284140534 boot 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: cluster 2024-02-12T16:42:40.152383+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e923: 12 total, 12 up, 12 in 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.154723+0000 mon.a (mon.0) 3114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.155060+0000 mon.a (mon.0) 3115 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.155273+0000 mon.a (mon.0) 3116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.155496+0000 mon.a (mon.0) 3117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.206231+0000 mon.a (mon.0) 3118 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.209182+0000 mon.a (mon.0) 3119 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:42:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.210470+0000 mon.a (mon.0) 3120 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:42:41.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: audit 2024-02-12T16:42:40.219514+0000 mon.a (mon.0) 3121 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:41.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: cluster 2024-02-12T16:42:40.266401+0000 mgr.y (mgr.14146) 1750 : cluster [DBG] pgmap v2544: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 2.9 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:41.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:41 smithi121 bash[18489]: cluster 2024-02-12T16:42:40.962403+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e924: 12 total, 12 up, 12 in 2024-02-12T16:42:41.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:41.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: cluster 2024-02-12T16:42:40.138694+0000 mon.a (mon.0) 3111 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: cluster 2024-02-12T16:42:40.152326+0000 mon.a (mon.0) 3112 : cluster [INF] osd.7 v1:172.21.15.123:6812/4284140534 boot 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: cluster 2024-02-12T16:42:40.152383+0000 mon.a (mon.0) 3113 : cluster [DBG] osdmap e923: 12 total, 12 up, 12 in 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.154723+0000 mon.a (mon.0) 3114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.155060+0000 mon.a (mon.0) 3115 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.155273+0000 mon.a (mon.0) 3116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.155496+0000 mon.a (mon.0) 3117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.206231+0000 mon.a (mon.0) 3118 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.209182+0000 mon.a (mon.0) 3119 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.210470+0000 mon.a (mon.0) 3120 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: audit 2024-02-12T16:42:40.219514+0000 mon.a (mon.0) 3121 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: cluster 2024-02-12T16:42:40.266401+0000 mgr.y (mgr.14146) 1750 : cluster [DBG] pgmap v2544: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 2.9 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:42:41.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:41 smithi123 bash[19119]: cluster 2024-02-12T16:42:40.962403+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e924: 12 total, 12 up, 12 in 2024-02-12T16:42:41.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 5 16 3976 3960 49.4929 60.9219 0.00143903 0.020198 2024-02-12T16:42:41.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:41.679 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:41.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:41.809 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:41.913 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:41.942 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:41.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:41.971 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:42.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:42.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:42.206 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:42.225 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:42 smithi169 bash[19604]: cluster 2024-02-12T16:42:39.988396+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:42.225 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:42 smithi169 bash[19604]: cluster 2024-02-12T16:42:39.988552+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:42.225 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:42 smithi169 bash[19604]: cluster 2024-02-12T16:42:41.214062+0000 osd.4 (osd.4) 159 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:42:42.225 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:42 smithi169 bash[19604]: cluster 2024-02-12T16:42:41.310836+0000 osd.4 (osd.4) 160 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:42:42.225 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:42 smithi169 bash[19604]: cluster 2024-02-12T16:42:41.971731+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e925: 12 total, 12 up, 12 in 2024-02-12T16:42:42.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:42.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:42 smithi121 bash[18489]: cluster 2024-02-12T16:42:39.988396+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:42.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:42 smithi121 bash[18489]: cluster 2024-02-12T16:42:39.988552+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:42.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:42 smithi121 bash[18489]: cluster 2024-02-12T16:42:41.214062+0000 osd.4 (osd.4) 159 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:42:42.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:42 smithi121 bash[18489]: cluster 2024-02-12T16:42:41.310836+0000 osd.4 (osd.4) 160 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:42:42.317 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:42 smithi121 bash[18489]: cluster 2024-02-12T16:42:41.971731+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e925: 12 total, 12 up, 12 in 2024-02-12T16:42:42.335 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:42.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:42.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:42.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:42 smithi123 bash[19119]: cluster 2024-02-12T16:42:39.988396+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:42 smithi123 bash[19119]: cluster 2024-02-12T16:42:39.988552+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:42 smithi123 bash[19119]: cluster 2024-02-12T16:42:41.214062+0000 osd.4 (osd.4) 159 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:42:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:42 smithi123 bash[19119]: cluster 2024-02-12T16:42:41.310836+0000 osd.4 (osd.4) 160 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:42:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:42 smithi123 bash[19119]: cluster 2024-02-12T16:42:41.971731+0000 mon.a (mon.0) 3123 : cluster [DBG] osdmap e925: 12 total, 12 up, 12 in 2024-02-12T16:42:42.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 6 16 9806 9790 101.966 364.375 0.00363173 0.00980413 2024-02-12T16:42:42.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:42.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:42.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:42.854 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:42.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:42.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:42.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:43.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:43.113 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:43.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:43.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:43.244 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:43.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:43.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:43.391 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:43.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:43 smithi121 bash[18489]: cluster 2024-02-12T16:42:42.225329+0000 osd.4 (osd.4) 161 : cluster [DBG] 1.e deep-scrub starts 2024-02-12T16:42:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:43 smithi121 bash[18489]: cluster 2024-02-12T16:42:42.267013+0000 mgr.y (mgr.14146) 1751 : cluster [DBG] pgmap v2547: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 147 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s 2024-02-12T16:42:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:43 smithi121 bash[18489]: cluster 2024-02-12T16:42:42.373017+0000 osd.4 (osd.4) 162 : cluster [DBG] 1.e deep-scrub ok 2024-02-12T16:42:43.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:43 smithi121 bash[18489]: cluster 2024-02-12T16:42:42.926426+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.d scrub starts 2024-02-12T16:42:43.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:43.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:43 smithi169 bash[19604]: cluster 2024-02-12T16:42:42.225329+0000 osd.4 (osd.4) 161 : cluster [DBG] 1.e deep-scrub starts 2024-02-12T16:42:43.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:43 smithi169 bash[19604]: cluster 2024-02-12T16:42:42.267013+0000 mgr.y (mgr.14146) 1751 : cluster [DBG] pgmap v2547: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 147 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s 2024-02-12T16:42:43.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:43 smithi169 bash[19604]: cluster 2024-02-12T16:42:42.373017+0000 osd.4 (osd.4) 162 : cluster [DBG] 1.e deep-scrub ok 2024-02-12T16:42:43.504 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:43 smithi169 bash[19604]: cluster 2024-02-12T16:42:42.926426+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.d scrub starts 2024-02-12T16:42:43.524 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:43.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:43.548 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:42:43.628 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:43 smithi123 bash[19119]: cluster 2024-02-12T16:42:42.225329+0000 osd.4 (osd.4) 161 : cluster [DBG] 1.e deep-scrub starts 2024-02-12T16:42:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:43 smithi123 bash[19119]: cluster 2024-02-12T16:42:42.267013+0000 mgr.y (mgr.14146) 1751 : cluster [DBG] pgmap v2547: 47 pgs: 1 active+clean+remapped, 9 undersized+peered, 37 active+clean; 147 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s 2024-02-12T16:42:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:43 smithi123 bash[19119]: cluster 2024-02-12T16:42:42.373017+0000 osd.4 (osd.4) 162 : cluster [DBG] 1.e deep-scrub ok 2024-02-12T16:42:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:43 smithi123 bash[19119]: cluster 2024-02-12T16:42:42.926426+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.d scrub starts 2024-02-12T16:42:43.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 7 16 15081 15065 134.492 329.688 0.00189119 0.00743204 2024-02-12T16:42:43.655 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:43.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:43.759 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:43.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:42:43.787 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:43.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:43.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:43.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:43.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:43.994 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:42:44.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:44.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:44.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:44.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:44.386 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:44.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:44.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:44 smithi123 bash[19119]: cluster 2024-02-12T16:42:43.033572+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.d scrub ok 2024-02-12T16:42:44.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:44.518 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:44.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:44.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:44 smithi169 bash[19604]: cluster 2024-02-12T16:42:43.033572+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.d scrub ok 2024-02-12T16:42:44.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:44.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 8 15 18177 18162 141.874 193.562 0.0071741 0.0070456 2024-02-12T16:42:44.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:44 smithi121 bash[18489]: cluster 2024-02-12T16:42:43.033572+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.d scrub ok 2024-02-12T16:42:44.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:44.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:44.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:44.821 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:44.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:44.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:44.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:45.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:45.084 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:45.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:45.186 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:45.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:45.239 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:45 smithi121 bash[18489]: cluster 2024-02-12T16:42:43.873875+0000 osd.10 (osd.10) 234 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:42:45.239 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:45 smithi121 bash[18489]: cluster 2024-02-12T16:42:43.923198+0000 osd.10 (osd.10) 235 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:42:45.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:45 smithi121 bash[18489]: cluster 2024-02-12T16:42:44.267764+0000 mgr.y (mgr.14146) 1752 : cluster [DBG] pgmap v2548: 47 pgs: 47 active+clean; 516 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-12T16:42:45.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:45 smithi121 bash[18489]: cluster 2024-02-12T16:42:44.288873+0000 osd.0 (osd.0) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:45.240 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:45 smithi121 bash[18489]: cluster 2024-02-12T16:42:44.288956+0000 osd.0 (osd.0) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:45.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:45.343 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:45.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:45.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:45.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:45.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:45.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:45 smithi169 bash[19604]: cluster 2024-02-12T16:42:43.873875+0000 osd.10 (osd.10) 234 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:42:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:45 smithi169 bash[19604]: cluster 2024-02-12T16:42:43.923198+0000 osd.10 (osd.10) 235 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:42:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:45 smithi169 bash[19604]: cluster 2024-02-12T16:42:44.267764+0000 mgr.y (mgr.14146) 1752 : cluster [DBG] pgmap v2548: 47 pgs: 47 active+clean; 516 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-12T16:42:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:45 smithi169 bash[19604]: cluster 2024-02-12T16:42:44.288873+0000 osd.0 (osd.0) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:45.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:45 smithi169 bash[19604]: cluster 2024-02-12T16:42:44.288956+0000 osd.0 (osd.0) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:45.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:45 smithi123 bash[19119]: cluster 2024-02-12T16:42:43.873875+0000 osd.10 (osd.10) 234 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:42:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:45 smithi123 bash[19119]: cluster 2024-02-12T16:42:43.923198+0000 osd.10 (osd.10) 235 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:42:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:45 smithi123 bash[19119]: cluster 2024-02-12T16:42:44.267764+0000 mgr.y (mgr.14146) 1752 : cluster [DBG] pgmap v2548: 47 pgs: 47 active+clean; 516 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s 2024-02-12T16:42:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:45 smithi123 bash[19119]: cluster 2024-02-12T16:42:44.288873+0000 osd.0 (osd.0) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:45 smithi123 bash[19119]: cluster 2024-02-12T16:42:44.288956+0000 osd.0 (osd.0) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:45.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 9 16 22238 22222 154.302 253.75 0.00327587 0.00647855 2024-02-12T16:42:45.705 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:45.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:45.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:45.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:45.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:45.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:45.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:46.099 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:46.124 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:46.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:46.227 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:46.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:46 smithi169 bash[19604]: cluster 2024-02-12T16:42:45.226644+0000 mon.a (mon.0) 3124 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive) 2024-02-12T16:42:46.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:46 smithi169 bash[19604]: cluster 2024-02-12T16:42:45.226670+0000 mon.a (mon.0) 3125 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6 pgs undersized) 2024-02-12T16:42:46.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:46 smithi169 bash[19604]: cluster 2024-02-12T16:42:45.226687+0000 mon.a (mon.0) 3126 : cluster [INF] Cluster is now healthy 2024-02-12T16:42:46.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:46 smithi169 bash[19604]: cluster 2024-02-12T16:42:45.229922+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e926: 12 total, 12 up, 12 in 2024-02-12T16:42:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:46 smithi123 bash[19119]: cluster 2024-02-12T16:42:45.226644+0000 mon.a (mon.0) 3124 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive) 2024-02-12T16:42:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:46 smithi123 bash[19119]: cluster 2024-02-12T16:42:45.226670+0000 mon.a (mon.0) 3125 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6 pgs undersized) 2024-02-12T16:42:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:46 smithi123 bash[19119]: cluster 2024-02-12T16:42:45.226687+0000 mon.a (mon.0) 3126 : cluster [INF] Cluster is now healthy 2024-02-12T16:42:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:46 smithi123 bash[19119]: cluster 2024-02-12T16:42:45.229922+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e926: 12 total, 12 up, 12 in 2024-02-12T16:42:46.480 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:46.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:46.496 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:46 smithi121 bash[18489]: cluster 2024-02-12T16:42:45.226644+0000 mon.a (mon.0) 3124 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive) 2024-02-12T16:42:46.496 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:46 smithi121 bash[18489]: cluster 2024-02-12T16:42:45.226670+0000 mon.a (mon.0) 3125 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6 pgs undersized) 2024-02-12T16:42:46.496 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:46 smithi121 bash[18489]: cluster 2024-02-12T16:42:45.226687+0000 mon.a (mon.0) 3126 : cluster [INF] Cluster is now healthy 2024-02-12T16:42:46.496 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:46 smithi121 bash[18489]: cluster 2024-02-12T16:42:45.229922+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e926: 12 total, 12 up, 12 in 2024-02-12T16:42:46.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:46.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:46.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:46.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 10 16 26334 26318 164.469 256 0.00293633 0.00607789 2024-02-12T16:42:46.719 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:46.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:46.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:46.849 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:46.875 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:46.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:46.978 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:47.028 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:47.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:47.168 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:47.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:47.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:47 smithi169 bash[19604]: cluster 2024-02-12T16:42:46.268227+0000 mgr.y (mgr.14146) 1753 : cluster [DBG] pgmap v2550: 47 pgs: 47 active+clean; 691 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s wr, 1.83k op/s 2024-02-12T16:42:47.298 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:47.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:47.530 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:47.559 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:47 smithi123 bash[19119]: cluster 2024-02-12T16:42:46.268227+0000 mgr.y (mgr.14146) 1753 : cluster [DBG] pgmap v2550: 47 pgs: 47 active+clean; 691 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s wr, 1.83k op/s 2024-02-12T16:42:47.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 11 16 30262 30246 171.833 245.5 0.00226854 0.0058167 2024-02-12T16:42:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:47 smithi121 bash[18489]: cluster 2024-02-12T16:42:46.268227+0000 mgr.y (mgr.14146) 1753 : cluster [DBG] pgmap v2550: 47 pgs: 47 active+clean; 691 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 115 MiB/s wr, 1.83k op/s 2024-02-12T16:42:47.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:47.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:47.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:47.829 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:47.933 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:47.963 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:47.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:48.068 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:48.106 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:48.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:48.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:48.242 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:48.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:48.346 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:48.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:48.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:48.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:48.510 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:48.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:48.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:48.607 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:42:48.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:48.639 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:48.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:48.644 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 12 16 33979 33963 176.872 232.312 0.00252624 0.00565193 2024-02-12T16:42:48.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:48.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:48.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:48.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:48.924 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:48.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:49.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:49.063 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:49.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:49.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:49.203 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:49.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:49.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:49.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:49.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:49.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:49.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:49.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:49.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:49.602 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:49 smithi121 bash[18489]: cluster 2024-02-12T16:42:47.993797+0000 osd.7 (osd.7) 5 : cluster [DBG] 1.9 deep-scrub starts 2024-02-12T16:42:49.602 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:49 smithi121 bash[18489]: cluster 2024-02-12T16:42:48.099440+0000 osd.7 (osd.7) 6 : cluster [DBG] 1.9 deep-scrub ok 2024-02-12T16:42:49.602 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:49 smithi121 bash[18489]: cluster 2024-02-12T16:42:48.268640+0000 mgr.y (mgr.14146) 1754 : cluster [DBG] pgmap v2551: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:42:49.627 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:49.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:49.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 13 16 35129 35113 168.795 71.875 0.0028697 0.00592235 2024-02-12T16:42:49.730 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:49.737 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:49 smithi123 bash[19119]: cluster 2024-02-12T16:42:47.993797+0000 osd.7 (osd.7) 5 : cluster [DBG] 1.9 deep-scrub starts 2024-02-12T16:42:49.737 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:49 smithi123 bash[19119]: cluster 2024-02-12T16:42:48.099440+0000 osd.7 (osd.7) 6 : cluster [DBG] 1.9 deep-scrub ok 2024-02-12T16:42:49.738 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:49 smithi123 bash[19119]: cluster 2024-02-12T16:42:48.268640+0000 mgr.y (mgr.14146) 1754 : cluster [DBG] pgmap v2551: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:42:49.752 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:49.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:49 smithi169 bash[19604]: cluster 2024-02-12T16:42:47.993797+0000 osd.7 (osd.7) 5 : cluster [DBG] 1.9 deep-scrub starts 2024-02-12T16:42:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:49 smithi169 bash[19604]: cluster 2024-02-12T16:42:48.099440+0000 osd.7 (osd.7) 6 : cluster [DBG] 1.9 deep-scrub ok 2024-02-12T16:42:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:49 smithi169 bash[19604]: cluster 2024-02-12T16:42:48.268640+0000 mgr.y (mgr.14146) 1754 : cluster [DBG] pgmap v2551: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 189 MiB/s wr, 3.02k op/s 2024-02-12T16:42:49.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:49.882 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:49.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:49.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:50.010 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:50.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:50.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:50.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:50.274 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:50.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:50.442 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:50.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:50.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:50.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 14 16 38131 38115 170.139 187.625 0.00704925 0.00587518 2024-02-12T16:42:50.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:50.704 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:50.801 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-02-12T16:42:50.801 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:42:50.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:50.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:50.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:50.969 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:50.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:51.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:51.077 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:42:51.097 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:51.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:51.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:51.108 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:42:51.136 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:42:51.174 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:42:51.203 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:42:51.203 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:51.210 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:50 smithi169 bash[19604]: audit 2024-02-12T16:42:49.972440+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:51.210 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:50 smithi169 bash[19604]: cluster 2024-02-12T16:42:50.269170+0000 mgr.y (mgr.14146) 1755 : cluster [DBG] pgmap v2552: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 209 MiB/s wr, 3.34k op/s 2024-02-12T16:42:51.228 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:42:51.257 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:42:51.292 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:42:51.321 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:42:51.346 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:51.348 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:42:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:51.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:51.378 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:42:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:50 smithi123 bash[19119]: audit 2024-02-12T16:42:49.972440+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:50 smithi123 bash[19119]: cluster 2024-02-12T16:42:50.269170+0000 mgr.y (mgr.14146) 1755 : cluster [DBG] pgmap v2552: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 209 MiB/s wr, 3.34k op/s 2024-02-12T16:42:51.409 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:42:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:50 smithi121 bash[18489]: audit 2024-02-12T16:42:49.972440+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:42:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:50 smithi121 bash[18489]: cluster 2024-02-12T16:42:50.269170+0000 mgr.y (mgr.14146) 1755 : cluster [DBG] pgmap v2552: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 209 MiB/s wr, 3.34k op/s 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.415 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.416 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.416 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.416 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:42:51.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:51.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:51.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:51.580 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:51.613 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:51.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:51.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 15 16 41120 41104 171.249 186.812 0.00245177 0.00583745 2024-02-12T16:42:51.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:51.744 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:51.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:51.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:51.877 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:51.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:51.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:51.997 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:42:52.007 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:52.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:51 smithi121 bash[18489]: audit 2024-02-12T16:42:51.047969+0000 mon.b (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.121:0/1518939336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:52.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:51 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.440042+0000 osd.6 (osd.6) 186 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:42:52.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:51 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.559257+0000 osd.6 (osd.6) 187 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:42:52.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:51 smithi121 bash[18489]: audit 2024-02-12T16:42:51.620387+0000 mon.a (mon.0) 3129 : audit [INF] from='client.27050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:42:52.048 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:51 smithi121 bash[18489]: audit 2024-02-12T16:42:51.622790+0000 mon.b (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.121:0/104176764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:42:52.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-12T16:42:52.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:52.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:52.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:52.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:51 smithi123 bash[19119]: audit 2024-02-12T16:42:51.047969+0000 mon.b (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.121:0/1518939336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:52.245 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:51 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.440042+0000 osd.6 (osd.6) 186 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:42:52.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:51 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.559257+0000 osd.6 (osd.6) 187 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:42:52.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:51 smithi123 bash[19119]: audit 2024-02-12T16:42:51.620387+0000 mon.a (mon.0) 3129 : audit [INF] from='client.27050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:42:52.246 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:51 smithi123 bash[19119]: audit 2024-02-12T16:42:51.622790+0000 mon.b (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.121:0/104176764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:42:52.264 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:52.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:51 smithi169 bash[19604]: audit 2024-02-12T16:42:51.047969+0000 mon.b (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.121:0/1518939336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:42:52.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:51 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.440042+0000 osd.6 (osd.6) 186 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:42:52.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:51 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.559257+0000 osd.6 (osd.6) 187 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:42:52.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:51 smithi169 bash[19604]: audit 2024-02-12T16:42:51.620387+0000 mon.a (mon.0) 3129 : audit [INF] from='client.27050 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:42:52.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:51 smithi169 bash[19604]: audit 2024-02-12T16:42:51.622790+0000 mon.b (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.121:0/104176764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:42:52.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:52.411 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:52.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:52.542 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:52.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 16 16 44110 44094 172.225 186.875 0.00397636 0.00580459 2024-02-12T16:42:52.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:52.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:52.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:52.809 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:52.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:52.941 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:52.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:53.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.382118+0000 osd.9 (osd.9) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.382198+0000 osd.9 (osd.9) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.633442+0000 osd.8 (osd.8) 139 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.683923+0000 osd.8 (osd.8) 140 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.979304+0000 mon.a (mon.0) 3130 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: audit 2024-02-12T16:42:51.985322+0000 mon.a (mon.0) 3131 : audit [INF] from='client.27050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: cluster 2024-02-12T16:42:51.985384+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e927: 12 total, 12 up, 12 in 2024-02-12T16:42:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:53 smithi169 bash[19604]: cluster 2024-02-12T16:42:52.269584+0000 mgr.y (mgr.14146) 1756 : cluster [DBG] pgmap v2554: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 236 MiB/s wr, 3.77k op/s 2024-02-12T16:42:53.072 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:53.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:53.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:53.200 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:53.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:53.262 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.382118+0000 osd.9 (osd.9) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:53.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.382198+0000 osd.9 (osd.9) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:53.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.633442+0000 osd.8 (osd.8) 139 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:42:53.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.683923+0000 osd.8 (osd.8) 140 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:42:53.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.979304+0000 mon.a (mon.0) 3130 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:53.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: audit 2024-02-12T16:42:51.985322+0000 mon.a (mon.0) 3131 : audit [INF] from='client.27050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:42:53.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: cluster 2024-02-12T16:42:51.985384+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e927: 12 total, 12 up, 12 in 2024-02-12T16:42:53.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:52 smithi121 bash[18489]: cluster 2024-02-12T16:42:52.269584+0000 mgr.y (mgr.14146) 1756 : cluster [DBG] pgmap v2554: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 236 MiB/s wr, 3.77k op/s 2024-02-12T16:42:53.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:53.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.382118+0000 osd.9 (osd.9) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:53.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.382198+0000 osd.9 (osd.9) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:53.312 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.633442+0000 osd.8 (osd.8) 139 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:42:53.313 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.683923+0000 osd.8 (osd.8) 140 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:42:53.313 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.979304+0000 mon.a (mon.0) 3130 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:42:53.313 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: audit 2024-02-12T16:42:51.985322+0000 mon.a (mon.0) 3131 : audit [INF] from='client.27050 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:42:53.313 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: cluster 2024-02-12T16:42:51.985384+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e927: 12 total, 12 up, 12 in 2024-02-12T16:42:53.313 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:53 smithi123 bash[19119]: cluster 2024-02-12T16:42:52.269584+0000 mgr.y (mgr.14146) 1756 : cluster [DBG] pgmap v2554: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 236 MiB/s wr, 3.77k op/s 2024-02-12T16:42:53.333 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:53.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:53.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:53.466 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:53.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:53.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:53.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:53.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 17 16 47052 47036 172.909 183.875 0.0058684 0.00578174 2024-02-12T16:42:53.708 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:53.735 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:53.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:53.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:53.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:53.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:53.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:54.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:54.126 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:54.148 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:54.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:54.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:54.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:54.418 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:54.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:54.547 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:54.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 18 16 49897 49881 173.181 177.812 0.00308114 0.0057719 2024-02-12T16:42:54.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:54.674 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:54.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:54.812 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:54.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:54.941 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:54.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:55.046 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:55.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:55.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:55.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:55.191 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:42:55.191 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:42:55.191 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:42:55.191 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:42:55.211 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:55.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:55.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:42:55.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:55.332 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:55 smithi169 bash[19604]: cluster 2024-02-12T16:42:54.269999+0000 mgr.y (mgr.14146) 1757 : cluster [DBG] pgmap v2555: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.89k op/s 2024-02-12T16:42:55.347 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:55.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:55.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:55.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:55.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:55.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:42:55.511 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:55 smithi121 bash[18489]: cluster 2024-02-12T16:42:54.269999+0000 mgr.y (mgr.14146) 1757 : cluster [DBG] pgmap v2555: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.89k op/s 2024-02-12T16:42:55.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:55.624 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:55.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:55 smithi123 bash[19119]: cluster 2024-02-12T16:42:54.269999+0000 mgr.y (mgr.14146) 1757 : cluster [DBG] pgmap v2555: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 243 MiB/s wr, 3.89k op/s 2024-02-12T16:42:55.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 19 16 52650 52634 173.121 172.062 0.00702863 0.00577438 2024-02-12T16:42:55.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:55.756 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:55.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:55.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:42:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:55.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:42:55.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:56.018 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:56.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:56.149 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:56.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:56.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:56.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:56.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:56.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:56.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:56.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:42:56.642564 min lat: 0.000995729 max lat: 4.35085 avg lat: 0.00577772 2024-02-12T16:42:56.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:42:56.645 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 20 15 55388 55373 173.024 171.188 0.00658598 0.00577772 2024-02-12T16:42:56.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:56.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:56.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:56.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:56.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:56.924 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:42:56.933 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:42:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:56.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:42:57.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:56 smithi169 bash[19604]: cluster 2024-02-12T16:42:55.965141+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e928: 12 total, 12 up, 12 in 2024-02-12T16:42:57.242 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:56 smithi169 bash[19604]: cluster 2024-02-12T16:42:56.270423+0000 mgr.y (mgr.14146) 1758 : cluster [DBG] pgmap v2557: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 180 MiB/s wr, 2.89k op/s 2024-02-12T16:42:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:57.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:57.344 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:57.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:57.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:56 smithi123 bash[19119]: cluster 2024-02-12T16:42:55.965141+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e928: 12 total, 12 up, 12 in 2024-02-12T16:42:57.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:56 smithi123 bash[19119]: cluster 2024-02-12T16:42:56.270423+0000 mgr.y (mgr.14146) 1758 : cluster [DBG] pgmap v2557: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 180 MiB/s wr, 2.89k op/s 2024-02-12T16:42:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:56 smithi121 bash[18489]: cluster 2024-02-12T16:42:55.965141+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e928: 12 total, 12 up, 12 in 2024-02-12T16:42:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:56 smithi121 bash[18489]: cluster 2024-02-12T16:42:56.270423+0000 mgr.y (mgr.14146) 1758 : cluster [DBG] pgmap v2557: 47 pgs: 47 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 180 MiB/s wr, 2.89k op/s 2024-02-12T16:42:57.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:57.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:42:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:57.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:42:57.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 21 16 57948 57932 172.395 159.938 0.00361289 0.00579799 2024-02-12T16:42:57.647 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:57.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:57.751 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:57.784 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:42:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:57.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:42:57.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.914 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:42:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:57.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:42:58.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:58.051 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:58.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:58.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:58.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:42:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:58.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:42:58.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:58.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:42:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:58.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:42:58.416 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:58.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 22 16 60591 60575 172.067 165.188 0.00719463 0.00581002 2024-02-12T16:42:58.756 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:58.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:58.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:58.884 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:58.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:58.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:59.019 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:59.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:59.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:59.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:59.150 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:42:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:59.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:42:59.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:59.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:59.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:59.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:59 smithi123 bash[19119]: cluster 2024-02-12T16:42:58.270903+0000 mgr.y (mgr.14146) 1759 : cluster [DBG] pgmap v2558: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.79k op/s 2024-02-12T16:42:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:59 smithi123 bash[19119]: cluster 2024-02-12T16:42:58.982712+0000 osd.8 (osd.8) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:42:59 smithi123 bash[19119]: cluster 2024-02-12T16:42:58.982771+0000 osd.8 (osd.8) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:59.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:42:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:59.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:42:59.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:59.618 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:59.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:59 smithi169 bash[19604]: cluster 2024-02-12T16:42:58.270903+0000 mgr.y (mgr.14146) 1759 : cluster [DBG] pgmap v2558: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.79k op/s 2024-02-12T16:42:59.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:59 smithi169 bash[19604]: cluster 2024-02-12T16:42:58.982712+0000 osd.8 (osd.8) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:59.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:42:59 smithi169 bash[19604]: cluster 2024-02-12T16:42:58.982771+0000 osd.8 (osd.8) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:59.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:59.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 23 16 63169 63153 171.591 161.125 0.00487772 0.00582581 2024-02-12T16:42:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:59 smithi121 bash[18489]: cluster 2024-02-12T16:42:58.270903+0000 mgr.y (mgr.14146) 1759 : cluster [DBG] pgmap v2558: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 174 MiB/s wr, 2.79k op/s 2024-02-12T16:42:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:59 smithi121 bash[18489]: cluster 2024-02-12T16:42:58.982712+0000 osd.8 (osd.8) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:42:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:42:59 smithi121 bash[18489]: cluster 2024-02-12T16:42:58.982771+0000 osd.8 (osd.8) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:42:59.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:59.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:42:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:59.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:42:59.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:59.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:42:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:42:59.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:42:59.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:00.019 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:00.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:00.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:00.131 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:00.131 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:00.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:00.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:00.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:00.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:00.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:00.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:00.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:00 smithi121 bash[18489]: cluster 2024-02-12T16:42:59.326638+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e929: 12 total, 12 up, 12 in 2024-02-12T16:43:00.430 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:00.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:00.537 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:00 smithi169 bash[19604]: cluster 2024-02-12T16:42:59.326638+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e929: 12 total, 12 up, 12 in 2024-02-12T16:43:00.563 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:00.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:00 smithi123 bash[19119]: cluster 2024-02-12T16:42:59.326638+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e929: 12 total, 12 up, 12 in 2024-02-12T16:43:00.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 24 16 65527 65511 170.581 147.375 0.0330527 0.00585692 2024-02-12T16:43:00.667 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:00.691 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:00.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:00.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:00.819 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:00.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:00.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:01.059 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:43:01.060 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:43:01.060 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 8, 11] 2024-02-12T16:43:01.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd out 9 2024-02-12T16:43:01.220 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:01.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:01.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:01.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:01.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:01.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:01.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:01.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:01 smithi169 bash[19604]: cluster 2024-02-12T16:43:00.271345+0000 mgr.y (mgr.14146) 1760 : cluster [DBG] pgmap v2560: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-12T16:43:01.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:01 smithi169 bash[19604]: cluster 2024-02-12T16:43:00.963388+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e930: 12 total, 12 up, 12 in 2024-02-12T16:43:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:01.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:01.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:01.632 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:01 smithi121 bash[18489]: cluster 2024-02-12T16:43:00.271345+0000 mgr.y (mgr.14146) 1760 : cluster [DBG] pgmap v2560: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-12T16:43:01.632 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:01 smithi121 bash[18489]: cluster 2024-02-12T16:43:00.963388+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e930: 12 total, 12 up, 12 in 2024-02-12T16:43:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:01 smithi123 bash[19119]: cluster 2024-02-12T16:43:00.271345+0000 mgr.y (mgr.14146) 1760 : cluster [DBG] pgmap v2560: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-12T16:43:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:01 smithi123 bash[19119]: cluster 2024-02-12T16:43:00.963388+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e930: 12 total, 12 up, 12 in 2024-02-12T16:43:01.643 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:01.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:01.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 25 16 66934 66918 167.275 87.9375 0.00619503 0.00597602 2024-02-12T16:43:01.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:01.772 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:01.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:01.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:01.907 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:01.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:02.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:02.038 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:02.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:02.178 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:02.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:02.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:02.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:02.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:02.434 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:02.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:02 smithi169 bash[19604]: cluster 2024-02-12T16:43:01.355426+0000 osd.3 (osd.3) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:02 smithi169 bash[19604]: cluster 2024-02-12T16:43:01.355527+0000 osd.3 (osd.3) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:02.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:02 smithi121 bash[18489]: cluster 2024-02-12T16:43:01.355426+0000 osd.3 (osd.3) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:02.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:02 smithi121 bash[18489]: cluster 2024-02-12T16:43:01.355527+0000 osd.3 (osd.3) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:02.566 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:02 smithi123 bash[19119]: cluster 2024-02-12T16:43:01.355426+0000 osd.3 (osd.3) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:02 smithi123 bash[19119]: cluster 2024-02-12T16:43:01.355527+0000 osd.3 (osd.3) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:02.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 26 16 69331 69315 166.603 149.812 0.00480206 0.00600068 2024-02-12T16:43:02.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:02.727 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:02.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:02.861 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:02.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:02.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:03.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:03.136 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:03.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:03.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:03.270 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:03.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:03.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPushReply(7.9 799/797 [PushReplyOp(7:94fe17e1:::benchmark_data_smithi169_105252_object99065:head)])", 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:36:29.868205+0000", 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 393.42779587000001, 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19116248999999999, 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.306 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.868205+0000", 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.868205+0000", 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.868205+0000", 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.211e-06 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.868209+0000", 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2299999999999999e-06 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.868214+0000", 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9309999999999999e-06 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.868219+0000", 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0779999999999998e-05 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.868270+0000", 2024-02-12T16:43:03.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19109783699999999 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.059368+0000", 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19116248999999999 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:207298 7.9 7:92e1ec80:::benchmark_data_smithi169_105252_object38704:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:36:29.869375+0000", 2024-02-12T16:43:03.310 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 393.42662581000002, 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.215142958, 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 207298 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.869375+0000", 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.869375+0000", 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.869373+0000", 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2849999999999995e-06 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:29.869381+0000", 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18829702000000001 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.057678+0000", 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7137000000000001e-05 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.057705+0000", 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001691875 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.059397+0000", 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115666 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.059513+0000", 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000137797 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,8", 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.059651+0000", 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010076909 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.069728+0000", 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054305919999999997 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.075158+0000", 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091284450000000007 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.084287+0000", 2024-02-12T16:43:03.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1258e-05 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.084368+0000", 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015054000000000001 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.084518+0000", 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.215142958 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.317 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:207870 7.9 7:95056476:::benchmark_data_smithi169_105252_object39276:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:36:30.500755+0000", 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 392.79524584799998, 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19929270199999999, 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:43:03.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 207870 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500755+0000", 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500755+0000", 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:43:03.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500754+0000", 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0959999999999997e-06 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500760+0000", 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6160000000000008e-06 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500769+0000", 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8310000000000003e-06 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500776+0000", 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9857000000000001e-05 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500826+0000", 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4469999999999995e-05 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.500920+0000", 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012521200000000001 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,8", 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.501046+0000", 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0095770460000000005 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.510623+0000", 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025133700000000002 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.510874+0000", 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.189014458 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.699888+0000", 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3028000000000001e-05 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.699942+0000", 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106649 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.700048+0000", 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19929270199999999 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:207872 7.9 7:98bd0476:::benchmark_data_smithi169_105252_object39278:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:36:30.502534+0000", 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 392.79346724099997, 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19773361, 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 207872 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.325 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.502534+0000", 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.502534+0000", 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.502532+0000", 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7020000000000004e-06 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.502539+0000", 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6627000000000001e-05 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.502566+0000", 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.477e-06 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.502572+0000", 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00271354 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.505286+0000", 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000149452 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.505435+0000", 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018990399999999999 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,8", 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.505625+0000", 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050577770000000003 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.510683+0000", 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001261161 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.511944+0000", 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18819139500000001 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.700135+0000", 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9400000000000002e-05 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.700175+0000", 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2965000000000002e-05 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.700268+0000", 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19773361 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.331 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26530.0:207893 7.9 7:9d60d02a:::benchmark_data_smithi169_105252_object39299:head [delete] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:36:30.522998+0000", 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 392.77300357299998, 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17795888400000001, 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26530", 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 207893 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.522998+0000", 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.522998+0000", 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.522995+0000", 2024-02-12T16:43:03.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1189999999999999e-06 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.523002+0000", 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2099999999999999e-06 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.523012+0000", 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2120000000000002e-06 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.523017+0000", 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055822999999999995 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.523575+0000", 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9185999999999995e-05 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.523674+0000", 2024-02-12T16:43:03.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7732000000000006e-05 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0,8", 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.523772+0000", 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041564100000000001 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.527928+0000", 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000835384 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.528764+0000", 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17165388000000001 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.700418+0000", 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017039400000000001 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.700588+0000", 2024-02-12T16:43:03.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00036866299999999998 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:36:30.700957+0000", 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17795888400000001 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59811 8.d 8:ba970246:::benchmark_data_smithi169_115677_object59810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.338 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.823440+0000", 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 150.47256080400001, 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34065904400000002, 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59811 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823440+0000", 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823440+0000", 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823439+0000", 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3519999999999998e-05 2024-02-12T16:43:03.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823462+0000", 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0699e-05 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823473+0000", 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1129999999999997e-06 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823479+0000", 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024674580000000001 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.825947+0000", 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00023116099999999999 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.826178+0000", 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000178221 2024-02-12T16:43:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.826356+0000", 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006138274 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.832494+0000", 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33121484800000001 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.163709+0000", 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0861000000000001e-05 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.163750+0000", 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034960299999999998 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.164100+0000", 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34065904400000002 2024-02-12T16:43:03.344 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59822 8.d 8:bd106337:::benchmark_data_smithi169_115677_object59821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.828921+0000", 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 150.46707982000001, 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33677383700000002, 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.345 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59822 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828921+0000", 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828921+0000", 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:03.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828919+0000", 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3802999999999999e-05 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828943+0000", 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.521e-05 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828958+0000", 2024-02-12T16:43:03.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5080000000000002e-06 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828965+0000", 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00116552 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830130+0000", 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013362500000000001 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830264+0000", 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000191225 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830455+0000", 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034418320000000001 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.833897+0000", 2024-02-12T16:43:03.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33036304500000002 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.164260+0000", 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051311100000000001 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.164773+0000", 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00092227200000000002 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.165695+0000", 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33677383700000002 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59837 8.d 8:bce42dd4:::benchmark_data_smithi169_115677_object59836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.836326+0000", 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 150.45967485400001, 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32974853500000001, 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.351 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59837 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836326+0000", 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836326+0000", 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836324+0000", 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1529000000000003e-05 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836356+0000", 2024-02-12T16:43:03.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3139000000000001e-05 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836369+0000", 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2120000000000002e-06 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836374+0000", 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9558999999999998e-05 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836423+0000", 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015596699999999999 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836579+0000", 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000199817 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836779+0000", 2024-02-12T16:43:03.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0019116210000000001 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.838691+0000", 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32708284300000001 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.165774+0000", 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9215000000000001e-05 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.165833+0000", 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024211 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166075+0000", 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32974853500000001 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59858 8.d 8:b9b6efa3:::benchmark_data_smithi169_115677_object59857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.357 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.851078+0000", 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 150.44492317800001, 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.315281547, 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59858 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851078+0000", 2024-02-12T16:43:03.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851078+0000", 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851076+0000", 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1844e-05 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851098+0000", 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3205e-05 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851111+0000", 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7649999999999993e-06 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851119+0000", 2024-02-12T16:43:03.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5439000000000003e-05 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851164+0000", 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000108528 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851273+0000", 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129102 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851402+0000", 2024-02-12T16:43:03.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034160670000000001 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.854818+0000", 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31130226599999999 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166120+0000", 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3087999999999998e-05 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166163+0000", 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019629000000000001 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166360+0000", 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.315281547 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.363 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59860 8.d 8:b39a1c77:::benchmark_data_smithi169_115677_object59859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.852616+0000", 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 150.44338498600001, 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31403088899999998, 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59860 2024-02-12T16:43:03.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852616+0000", 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852616+0000", 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852615+0000", 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9853000000000001e-05 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852635+0000", 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0856000000000001e-05 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852646+0000", 2024-02-12T16:43:03.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7729999999999999e-06 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852650+0000", 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00041727000000000002 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.853068+0000", 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016523699999999999 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.853233+0000", 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000208827 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.853442+0000", 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001487012 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.854929+0000", 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.311478274 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166407+0000", 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7182999999999999e-05 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166454+0000", 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019307600000000001 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166647+0000", 2024-02-12T16:43:03.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31403088899999998 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67828 8.d 8:bbb16db0:::benchmark_data_smithi169_115677_object67827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.702612+0000", 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 146.59338964299999, 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24922882399999999, 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67828 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702612+0000", 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702612+0000", 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702609+0000", 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2708e-05 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702632+0000", 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2493999999999999e-05 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702644+0000", 2024-02-12T16:43:03.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3149999999999999e-06 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702650+0000", 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2055000000000003e-05 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702692+0000", 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101516 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702793+0000", 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113244 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702906+0000", 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004734266 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.707641+0000", 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24380940600000001 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951450+0000", 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6556000000000003e-05 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951537+0000", 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030377900000000001 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951840+0000", 2024-02-12T16:43:03.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24922882399999999 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67829 8.d 8:b4a23540:::benchmark_data_smithi169_115677_object67828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.704059+0000", 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 146.59194250600001, 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24810375800000001, 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.376 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67829 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704059+0000", 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704059+0000", 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704057+0000", 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0411000000000002e-05 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704078+0000", 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1507e-05 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704089+0000", 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7840000000000003e-06 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704094+0000", 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00087896600000000001 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704973+0000", 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000179872 2024-02-12T16:43:03.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.705153+0000", 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018307100000000001 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.705336+0000", 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002813701 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.708149+0000", 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24374633600000001 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951896+0000", 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9889999999999997e-05 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951966+0000", 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019684999999999999 2024-02-12T16:43:03.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952163+0000", 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24810375800000001 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67857 8.d 8:b50e86ce:::benchmark_data_smithi169_115677_object67856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.382 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.716963+0000", 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 146.579038529, 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23546748300000001, 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67857 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716963+0000", 2024-02-12T16:43:03.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716963+0000", 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716960+0000", 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9241999999999999e-05 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716989+0000", 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0531999999999999e-05 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717010+0000", 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717018+0000", 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0773999999999998e-05 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717079+0000", 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017184899999999999 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717251+0000", 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019772499999999999 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717449+0000", 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026654080000000002 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.720114+0000", 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23208046700000001 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952194+0000", 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8588999999999999e-05 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952233+0000", 2024-02-12T16:43:03.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019727699999999999 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952430+0000", 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23546748300000001 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67883 8.d 8:b86b9654:::benchmark_data_smithi169_115677_object67882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.730676+0000", 2024-02-12T16:43:03.388 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 146.56532546899999, 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.223248011, 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67883 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730676+0000", 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730676+0000", 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730672+0000", 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9053000000000001e-05 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730702+0000", 2024-02-12T16:43:03.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6235e-05 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730718+0000", 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7000000000000008e-06 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730725+0000", 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9512999999999998e-05 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730785+0000", 2024-02-12T16:43:03.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015649500000000001 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730941+0000", 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000138787 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.731080+0000", 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021907390000000001 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.733271+0000", 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22032115799999999 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953592+0000", 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2156e-05 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953644+0000", 2024-02-12T16:43:03.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027951300000000001 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953924+0000", 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.223248011 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67899 8.d 8:b3bc5adf:::benchmark_data_smithi169_115677_object67898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.756677+0000", 2024-02-12T16:43:03.394 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 146.53932390400001, 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197548369, 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67899 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756677+0000", 2024-02-12T16:43:03.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756677+0000", 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756674+0000", 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5304999999999999e-05 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756700+0000", 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5061e-05 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756715+0000", 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9519999999999999e-06 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756721+0000", 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5686000000000002e-05 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756776+0000", 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015034 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756927+0000", 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017099899999999999 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.757098+0000", 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021726800000000002 2024-02-12T16:43:03.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.759270+0000", 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19469667800000001 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953967+0000", 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1038000000000002e-05 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.954018+0000", 2024-02-12T16:43:03.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000207672 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.954226+0000", 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197548369 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 898/889 [PushOp(8:e9d21599:::benchmark_data_smithi169_115677_object70736:head, version: 876'4346, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e9d21599:::benchmark_data_smithi169_115677_object70736:head@876'4346, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:35.267146+0000", 2024-02-12T16:43:03.400 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 88.028855450999998, 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19494173300000001, 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267146+0000", 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267146+0000", 2024-02-12T16:43:03.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267145+0000", 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0874000000000001e-05 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267196+0000", 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7368999999999999e-05 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267213+0000", 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1689999999999998e-06 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267222+0000", 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18740217200000001 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.454625+0000", 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3699999999999995e-06 2024-02-12T16:43:03.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.454633+0000", 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074546960000000002 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.462088+0000", 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19494173300000001 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.7 898/889 [PushOp(8:e69a8233:::benchmark_data_smithi169_115677_object70702:head, version: 876'4345, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e69a8233:::benchmark_data_smithi169_115677_object70702:head@876'4345, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:35.279424+0000", 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 88.016577495000007, 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18170157200000001, 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279424+0000", 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279424+0000", 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279423+0000", 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4529000000000001e-05 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279457+0000", 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7051999999999999e-05 2024-02-12T16:43:03.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279474+0000", 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7479999999999993e-06 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279482+0000", 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.179421944 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.458904+0000", 2024-02-12T16:43:03.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7399999999999998e-06 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.458911+0000", 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022147529999999999 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.461125+0000", 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18170157200000001 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.408 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 898/889 [PushOp(8:e94d15ac:::benchmark_data_smithi169_115677_object70790:head, version: 876'4350, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e94d15ac:::benchmark_data_smithi169_115677_object70790:head@876'4350, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:35.291573+0000", 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 88.004428781000001, 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17258230399999999, 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291573+0000", 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291573+0000", 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291571+0000", 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000003e-05 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291607+0000", 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3944999999999999e-05 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291621+0000", 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5499999999999995e-06 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291630+0000", 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17034442399999999 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.461974+0000", 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.216e-06 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.461981+0000", 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002173473 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.464155+0000", 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17258230399999999 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.412 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 899/889 [PushOp(8:e9f88ef0:::benchmark_data_smithi169_115677_object75347:head, version: 877'4655, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e9f88ef0:::benchmark_data_smithi169_115677_object75347:head@877'4655, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:39.921862+0000", 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 83.374139244000006, 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18364805100000001, 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921862+0000", 2024-02-12T16:43:03.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921862+0000", 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921861+0000", 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4548000000000006e-05 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921925+0000", 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6736999999999999e-05 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921942+0000", 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.166e-05 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921954+0000", 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.181234334 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:40.103188+0000", 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9610000000000005e-06 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:40.103196+0000", 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002313973 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:40.105510+0000", 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18364805100000001 2024-02-12T16:43:03.416 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 900/889 [PushOp(8:eeb3a8f1:::benchmark_data_smithi169_115677_object76694:head, version: 878'4743, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:eeb3a8f1:::benchmark_data_smithi169_115677_object76694:head@878'4743, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:41.135011+0000", 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 82.160990239, 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21948980300000001, 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:43:03.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135011+0000", 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135011+0000", 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135010+0000", 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8582000000000001e-05 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135088+0000", 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4621e-05 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135113+0000", 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3378e-05 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:43:03.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135126+0000", 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20962963100000001 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.344756+0000", 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1490000000000002e-06 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.344761+0000", 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0097400060000000007 2024-02-12T16:43:03.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.354501+0000", 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21948980300000001 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:43:03.421 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:43:03.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:03.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:03.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:03.578 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:03.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:03.602 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:03 smithi121 bash[18489]: cluster 2024-02-12T16:43:02.271747+0000 mgr.y (mgr.14146) 1761 : cluster [DBG] pgmap v2562: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 203 MiB/s wr, 3.25k op/s 2024-02-12T16:43:03.602 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:03 smithi121 bash[18489]: cluster 2024-02-12T16:43:02.346275+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e931: 12 total, 12 up, 12 in 2024-02-12T16:43:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:03 smithi123 bash[19119]: cluster 2024-02-12T16:43:02.271747+0000 mgr.y (mgr.14146) 1761 : cluster [DBG] pgmap v2562: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 203 MiB/s wr, 3.25k op/s 2024-02-12T16:43:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:03 smithi123 bash[19119]: cluster 2024-02-12T16:43:02.346275+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e931: 12 total, 12 up, 12 in 2024-02-12T16:43:03.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 27 16 71598 71582 165.68 141.688 0.00365567 0.00603405 2024-02-12T16:43:03.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:03.690 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:03 smithi169 bash[19604]: cluster 2024-02-12T16:43:02.271747+0000 mgr.y (mgr.14146) 1761 : cluster [DBG] pgmap v2562: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 203 MiB/s wr, 3.25k op/s 2024-02-12T16:43:03.690 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:03 smithi169 bash[19604]: cluster 2024-02-12T16:43:02.346275+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e931: 12 total, 12 up, 12 in 2024-02-12T16:43:03.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:03.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:03.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:04.170 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:04.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:04.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:04.305 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:04.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:04.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:43:04.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:04.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:04.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:04.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:04.565 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:04.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:04.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 28 16 73803 73787 164.684 137.812 0.00663641 0.00607059 2024-02-12T16:43:04.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:04.693 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:04.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:04.796 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:04.827 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:04.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:04.934 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:04.960 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:04.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:05.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:05.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:05.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:05.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:05.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:05.223 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:05.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:05.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:05.357 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:05.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:05 smithi121 bash[18489]: cluster 2024-02-12T16:43:03.714900+0000 osd.5 (osd.5) 215 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:05.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:05 smithi121 bash[18489]: cluster 2024-02-12T16:43:03.714950+0000 osd.5 (osd.5) 216 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:05.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:05 smithi121 bash[18489]: cluster 2024-02-12T16:43:04.272208+0000 mgr.y (mgr.14146) 1762 : cluster [DBG] pgmap v2564: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.75k op/s 2024-02-12T16:43:05.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:05 smithi121 bash[18489]: cluster 2024-02-12T16:43:04.356575+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e932: 12 total, 12 up, 12 in 2024-02-12T16:43:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:05.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:05.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:05.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:05.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:05.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:05 smithi123 bash[19119]: cluster 2024-02-12T16:43:03.714900+0000 osd.5 (osd.5) 215 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:05 smithi123 bash[19119]: cluster 2024-02-12T16:43:03.714950+0000 osd.5 (osd.5) 216 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:05 smithi123 bash[19119]: cluster 2024-02-12T16:43:04.272208+0000 mgr.y (mgr.14146) 1762 : cluster [DBG] pgmap v2564: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.75k op/s 2024-02-12T16:43:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:05 smithi123 bash[19119]: cluster 2024-02-12T16:43:04.356575+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e932: 12 total, 12 up, 12 in 2024-02-12T16:43:05.646 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 29 16 75718 75702 163.133 119.688 0.00484466 0.00612826 2024-02-12T16:43:05.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:05.744 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:05.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:05 smithi169 bash[19604]: cluster 2024-02-12T16:43:03.714900+0000 osd.5 (osd.5) 215 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:05 smithi169 bash[19604]: cluster 2024-02-12T16:43:03.714950+0000 osd.5 (osd.5) 216 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:05 smithi169 bash[19604]: cluster 2024-02-12T16:43:04.272208+0000 mgr.y (mgr.14146) 1762 : cluster [DBG] pgmap v2564: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.75k op/s 2024-02-12T16:43:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:05 smithi169 bash[19604]: cluster 2024-02-12T16:43:04.356575+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e932: 12 total, 12 up, 12 in 2024-02-12T16:43:05.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:05.874 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:05.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:05.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:06.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:06.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:06.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:06.227 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:06.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:06.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:06.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:06.422 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:06.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:06.552 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:06.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 30 16 77889 77873 162.217 135.688 0.00545167 0.00616278 2024-02-12T16:43:06.657 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:06.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:06.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:06.824 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:06.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:06.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:06.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:07.063 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:07.090 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:07.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:07.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:06 smithi121 bash[18489]: cluster 2024-02-12T16:43:06.272710+0000 mgr.y (mgr.14146) 1763 : cluster [DBG] pgmap v2566: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-02-12T16:43:07.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:07.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:07.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:07.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:06 smithi169 bash[19604]: cluster 2024-02-12T16:43:06.272710+0000 mgr.y (mgr.14146) 1763 : cluster [DBG] pgmap v2566: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-02-12T16:43:07.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:07.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:06 smithi123 bash[19119]: cluster 2024-02-12T16:43:06.272710+0000 mgr.y (mgr.14146) 1763 : cluster [DBG] pgmap v2566: 47 pgs: 47 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 133 MiB/s wr, 2.13k op/s 2024-02-12T16:43:07.349 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:07.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:07.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:07.479 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:07.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:07.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:07.605 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:07.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:07.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 31 16 80062 80046 161.365 135.812 0.00747947 0.00619542 2024-02-12T16:43:07.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:07.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:07.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:07.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:07.869 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:07.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:07.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:08.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:08.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:08.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:08.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:08.251 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:08.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:08.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:08.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:08.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:08.522 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:08.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:08.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 32 16 82130 82114 160.361 129.25 0.0066114 0.0062343 2024-02-12T16:43:08.649 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:08.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:08.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:08.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:08.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:08.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:08.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:09.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:09.041 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:09.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:09.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:09.088 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:43:09.114 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:43:09.142 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:43:09.145 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:09.169 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:43:09.172 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:09.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:09.193 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:43:09.222 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:43:09.241 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:43:09.258 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:43:09.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:09.291 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:43:09.303 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:09.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:09.318 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:43:09.345 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:43:09.350 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:43:09.369 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:43:09.374 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.374 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.374 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.374 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.374 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:43:09.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:09 smithi121 bash[18489]: cluster 2024-02-12T16:43:08.274759+0000 mgr.y (mgr.14146) 1764 : cluster [DBG] pgmap v2567: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-12T16:43:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:09 smithi121 bash[18489]: audit 2024-02-12T16:43:08.535455+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? v1:172.21.15.121:0/2496546553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:09 smithi121 bash[18489]: audit 2024-02-12T16:43:09.052101+0000 mon.a (mon.0) 3139 : audit [DBG] from='client.? v1:172.21.15.121:0/4148206723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:09.414 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:09 smithi123 bash[19119]: cluster 2024-02-12T16:43:08.274759+0000 mgr.y (mgr.14146) 1764 : cluster [DBG] pgmap v2567: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-12T16:43:09.414 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:09 smithi123 bash[19119]: audit 2024-02-12T16:43:08.535455+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? v1:172.21.15.121:0/2496546553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:09.414 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:09 smithi123 bash[19119]: audit 2024-02-12T16:43:09.052101+0000 mon.a (mon.0) 3139 : audit [DBG] from='client.? v1:172.21.15.121:0/4148206723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:09.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:09.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:09.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:09.572 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:09.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:09.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 33 16 84066 84050 159.168 121 0.00393408 0.00628049 2024-02-12T16:43:09.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:09.699 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:09.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:09 smithi169 bash[19604]: cluster 2024-02-12T16:43:08.274759+0000 mgr.y (mgr.14146) 1764 : cluster [DBG] pgmap v2567: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 148 MiB/s wr, 2.37k op/s 2024-02-12T16:43:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:09 smithi169 bash[19604]: audit 2024-02-12T16:43:08.535455+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? v1:172.21.15.121:0/2496546553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:09 smithi169 bash[19604]: audit 2024-02-12T16:43:09.052101+0000 mon.a (mon.0) 3139 : audit [DBG] from='client.? v1:172.21.15.121:0/4148206723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:09.805 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:09.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:09.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:09.970 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:10.000 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:10.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:10.134 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:10.238 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:10.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:10.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:10.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:10 smithi169 bash[19604]: cluster 2024-02-12T16:43:09.327348+0000 mon.a (mon.0) 3140 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:43:10.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:10 smithi169 bash[19604]: audit 2024-02-12T16:43:09.340754+0000 mon.a (mon.0) 3141 : audit [INF] from='client.? v1:172.21.15.121:0/2496546553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:43:10.379 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:10 smithi169 bash[19604]: cluster 2024-02-12T16:43:09.340811+0000 mon.a (mon.0) 3142 : cluster [DBG] osdmap e933: 12 total, 12 up, 12 in 2024-02-12T16:43:10.403 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:10.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:10.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:10 smithi123 bash[19119]: cluster 2024-02-12T16:43:09.327348+0000 mon.a (mon.0) 3140 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:43:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:10 smithi123 bash[19119]: audit 2024-02-12T16:43:09.340754+0000 mon.a (mon.0) 3141 : audit [INF] from='client.? v1:172.21.15.121:0/2496546553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:43:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:10 smithi123 bash[19119]: cluster 2024-02-12T16:43:09.340811+0000 mon.a (mon.0) 3142 : cluster [DBG] osdmap e933: 12 total, 12 up, 12 in 2024-02-12T16:43:10.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:10.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 34 16 86124 86108 158.269 128.625 0.00377627 0.00631663 2024-02-12T16:43:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:10 smithi121 bash[18489]: cluster 2024-02-12T16:43:09.327348+0000 mon.a (mon.0) 3140 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:43:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:10 smithi121 bash[18489]: audit 2024-02-12T16:43:09.340754+0000 mon.a (mon.0) 3141 : audit [INF] from='client.? v1:172.21.15.121:0/2496546553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:43:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:10 smithi121 bash[18489]: cluster 2024-02-12T16:43:09.340811+0000 mon.a (mon.0) 3142 : cluster [DBG] osdmap e933: 12 total, 12 up, 12 in 2024-02-12T16:43:10.670 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:10.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:10.799 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:10.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:10.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:10.931 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:10.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:11.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:11.096 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:11.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:11.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:11.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:11.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:11.331 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:11.345 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:11 smithi169 bash[19604]: cluster 2024-02-12T16:43:10.275142+0000 mgr.y (mgr.14146) 1765 : cluster [DBG] pgmap v2569: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-12T16:43:11.345 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:11 smithi169 bash[19604]: cluster 2024-02-12T16:43:10.965494+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e934: 12 total, 12 up, 12 in 2024-02-12T16:43:11.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:11.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:11.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:11.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:11.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:11.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:11.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:11 smithi121 bash[18489]: cluster 2024-02-12T16:43:10.275142+0000 mgr.y (mgr.14146) 1765 : cluster [DBG] pgmap v2569: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-12T16:43:11.596 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:11 smithi121 bash[18489]: cluster 2024-02-12T16:43:10.965494+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e934: 12 total, 12 up, 12 in 2024-02-12T16:43:11.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:11.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:11 smithi123 bash[19119]: cluster 2024-02-12T16:43:10.275142+0000 mgr.y (mgr.14146) 1765 : cluster [DBG] pgmap v2569: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.94k op/s 2024-02-12T16:43:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:11 smithi123 bash[19119]: cluster 2024-02-12T16:43:10.965494+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e934: 12 total, 12 up, 12 in 2024-02-12T16:43:11.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 35 16 88085 88069 157.249 122.562 0.00624794 0.00635733 2024-02-12T16:43:11.718 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:11.747 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:11.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:11.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:11.878 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:11.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:11.981 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:12.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:12.137 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:12.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:12.280 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:12.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:12.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:12.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:12.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:12.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 36 16 89803 89787 155.863 107.375 0.00327797 0.00639788 2024-02-12T16:43:12.683 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:12.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:12.832 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:12.938 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:12.961 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:12.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:13.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:13.090 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:13.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:13.192 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:13.219 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:13.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:13.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:13.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:13.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:13.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:13.498 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:13.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:13.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:13.608 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:13 smithi123 bash[19119]: cluster 2024-02-12T16:43:12.275625+0000 mgr.y (mgr.14146) 1766 : cluster [DBG] pgmap v2571: 47 pgs: 47 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.14k op/s 2024-02-12T16:43:13.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:13.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:13.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 37 16 91697 91681 154.85 118.375 0.00510662 0.00645635 2024-02-12T16:43:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:13 smithi121 bash[18489]: cluster 2024-02-12T16:43:12.275625+0000 mgr.y (mgr.14146) 1766 : cluster [DBG] pgmap v2571: 47 pgs: 47 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.14k op/s 2024-02-12T16:43:13.734 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:13.763 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:13.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:13.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:43:13.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:13 smithi169 bash[19604]: cluster 2024-02-12T16:43:12.275625+0000 mgr.y (mgr.14146) 1766 : cluster [DBG] pgmap v2571: 47 pgs: 47 active+clean; 5.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 134 MiB/s wr, 2.14k op/s 2024-02-12T16:43:13.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:13.895 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:13.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:13.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:14.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.036 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:14.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:14.166 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:14.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:14.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:14.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:14.424 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:14.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:43:14.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:14.560 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:14.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 38 16 93514 93498 153.763 113.562 0.00949956 0.00650175 2024-02-12T16:43:14.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:14.698 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:14.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:14.830 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:14.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:14.963 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:14.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:15.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:15.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:15.200 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:15.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:15.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:15.359 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:15.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:15 smithi123 bash[19119]: cluster 2024-02-12T16:43:14.276108+0000 mgr.y (mgr.14146) 1767 : cluster [DBG] pgmap v2572: 47 pgs: 47 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.42k op/s 2024-02-12T16:43:15.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:15 smithi123 bash[19119]: audit 2024-02-12T16:43:15.241627+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? v1:172.21.15.121:0/1960375034' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-02-12T16:43:15.360 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:15 smithi123 bash[19119]: cluster 2024-02-12T16:43:15.241708+0000 mon.a (mon.0) 3145 : cluster [INF] Client client.admin marked osd.9 out, while it was still marked up 2024-02-12T16:43:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:15.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:15.366 INFO:teuthology.orchestra.run.smithi121.stderr:marked out osd.9. 2024-02-12T16:43:15.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:15.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:15.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:15.613 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:15 smithi169 bash[19604]: cluster 2024-02-12T16:43:14.276108+0000 mgr.y (mgr.14146) 1767 : cluster [DBG] pgmap v2572: 47 pgs: 47 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.42k op/s 2024-02-12T16:43:15.613 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:15 smithi169 bash[19604]: audit 2024-02-12T16:43:15.241627+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? v1:172.21.15.121:0/1960375034' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-02-12T16:43:15.613 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:15 smithi169 bash[19604]: cluster 2024-02-12T16:43:15.241708+0000 mon.a (mon.0) 3145 : cluster [INF] Client client.admin marked osd.9 out, while it was still marked up 2024-02-12T16:43:15.637 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:15.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:15.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 39 16 94956 94940 152.131 90.125 0.00459753 0.0065307 2024-02-12T16:43:15.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:15 smithi121 bash[18489]: debug 2024-02-12T16:43:15.236+0000 7fbba85de700 -1 mon.a@0(leader).osd e934 definitely_dead 0 2024-02-12T16:43:15.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:15 smithi121 bash[18489]: cluster 2024-02-12T16:43:14.276108+0000 mgr.y (mgr.14146) 1767 : cluster [DBG] pgmap v2572: 47 pgs: 47 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.42k op/s 2024-02-12T16:43:15.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:15 smithi121 bash[18489]: audit 2024-02-12T16:43:15.241627+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? v1:172.21.15.121:0/1960375034' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-02-12T16:43:15.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:15 smithi121 bash[18489]: cluster 2024-02-12T16:43:15.241708+0000 mon.a (mon.0) 3145 : cluster [INF] Client client.admin marked osd.9 out, while it was still marked up 2024-02-12T16:43:15.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:15.778 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:15.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:15.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:15.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:15.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:16.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:16.053 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:16.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:16.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:16.187 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:16.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:16.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:16.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:16.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:16 smithi123 bash[19119]: audit 2024-02-12T16:43:15.352130+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? v1:172.21.15.121:0/1960375034' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-02-12T16:43:16.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:16 smithi123 bash[19119]: cluster 2024-02-12T16:43:15.352185+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e935: 12 total, 12 up, 11 in 2024-02-12T16:43:16.361 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:16 smithi123 bash[19119]: cluster 2024-02-12T16:43:15.971113+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e936: 12 total, 12 up, 11 in 2024-02-12T16:43:16.423 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:16.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:16 smithi121 bash[18489]: audit 2024-02-12T16:43:15.352130+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? v1:172.21.15.121:0/1960375034' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-02-12T16:43:16.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:16 smithi121 bash[18489]: cluster 2024-02-12T16:43:15.352185+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e935: 12 total, 12 up, 11 in 2024-02-12T16:43:16.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:16 smithi121 bash[18489]: cluster 2024-02-12T16:43:15.971113+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e936: 12 total, 12 up, 11 in 2024-02-12T16:43:16.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:16.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:16.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:16.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:16.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:16.647 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:43:16.644908 min lat: 0.000995729 max lat: 4.35085 avg lat: 0.0066977 2024-02-12T16:43:16.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:43:16.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 40 16 95545 95529 149.248 36.8125 0.0192146 0.0066977 2024-02-12T16:43:16.725 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:16.733 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:16 smithi169 bash[19604]: audit 2024-02-12T16:43:15.352130+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? v1:172.21.15.121:0/1960375034' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-02-12T16:43:16.734 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:16 smithi169 bash[19604]: cluster 2024-02-12T16:43:15.352185+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e935: 12 total, 12 up, 11 in 2024-02-12T16:43:16.734 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:16 smithi169 bash[19604]: cluster 2024-02-12T16:43:15.971113+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e936: 12 total, 12 up, 11 in 2024-02-12T16:43:16.752 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:16.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:16.777 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:43:16.777 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:43:16.861 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:16.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:16.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:16.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:17.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:17.128 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:17.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:17.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:17 smithi123 bash[19119]: cluster 2024-02-12T16:43:16.276543+0000 mgr.y (mgr.14146) 1768 : cluster [DBG] pgmap v2575: 47 pgs: 47 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-02-12T16:43:17.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:17.418 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:17.523 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:17.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:17.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 41 16 96844 96828 147.588 81.1875 0.0104284 0.00677312 2024-02-12T16:43:17.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:17 smithi121 bash[18489]: cluster 2024-02-12T16:43:16.276543+0000 mgr.y (mgr.14146) 1768 : cluster [DBG] pgmap v2575: 47 pgs: 47 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-02-12T16:43:17.660 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:17 smithi169 bash[19604]: cluster 2024-02-12T16:43:16.276543+0000 mgr.y (mgr.14146) 1768 : cluster [DBG] pgmap v2575: 47 pgs: 47 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-02-12T16:43:17.680 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:17.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:17.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:17.921 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:17.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:18.047 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:18.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:18.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:43:18.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:18.222 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:18.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:18.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:18.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:18.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:18 smithi121 bash[18489]: cluster 2024-02-12T16:43:17.667402+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:18 smithi121 bash[18489]: cluster 2024-02-12T16:43:17.667454+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:18.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:18.478 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:18.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:18 smithi169 bash[19604]: cluster 2024-02-12T16:43:17.667402+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:18 smithi169 bash[19604]: cluster 2024-02-12T16:43:17.667454+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:18.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:18.611 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:18.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:18 smithi123 bash[19119]: cluster 2024-02-12T16:43:17.667402+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:18 smithi123 bash[19119]: cluster 2024-02-12T16:43:17.667454+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:18.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 42 16 98208 98192 146.103 85.25 0.00389144 0.00684147 2024-02-12T16:43:18.716 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:18.745 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:18.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:18.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:18.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:18.884 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:18.935 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:18.935 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:18.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:19.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:19.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:19.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:19.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:19.371 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:19 smithi123 bash[19119]: cluster 2024-02-12T16:43:18.277054+0000 mgr.y (mgr.14146) 1769 : cluster [DBG] pgmap v2576: 47 pgs: 2 active+recovering, 45 active+clean; 5.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 11716/187618 objects misplaced (6.245%); 79 KiB/s, 1 objects/s recovering 2024-02-12T16:43:19.371 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:19 smithi123 bash[19119]: cluster 2024-02-12T16:43:18.366014+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e937: 12 total, 12 up, 11 in 2024-02-12T16:43:19.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:19.407 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:19 smithi121 bash[18489]: cluster 2024-02-12T16:43:18.277054+0000 mgr.y (mgr.14146) 1769 : cluster [DBG] pgmap v2576: 47 pgs: 2 active+recovering, 45 active+clean; 5.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 11716/187618 objects misplaced (6.245%); 79 KiB/s, 1 objects/s recovering 2024-02-12T16:43:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:19 smithi121 bash[18489]: cluster 2024-02-12T16:43:18.366014+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e937: 12 total, 12 up, 11 in 2024-02-12T16:43:19.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.411 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:19.512 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:19.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:19.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:19.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 43 16 99467 99451 144.536 78.6875 0.0355921 0.00691611 2024-02-12T16:43:19.670 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:19.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:19 smithi169 bash[19604]: cluster 2024-02-12T16:43:18.277054+0000 mgr.y (mgr.14146) 1769 : cluster [DBG] pgmap v2576: 47 pgs: 2 active+recovering, 45 active+clean; 5.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 11716/187618 objects misplaced (6.245%); 79 KiB/s, 1 objects/s recovering 2024-02-12T16:43:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:19 smithi169 bash[19604]: cluster 2024-02-12T16:43:18.366014+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e937: 12 total, 12 up, 11 in 2024-02-12T16:43:19.810 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:19.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:19.938 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:19.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:20.043 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:20.076 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:20.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:20.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:20.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:20.337 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:20.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:20.465 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:20.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:20.596 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:20.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 44 16 100751 100735 143.074 80.25 0.00383798 0.00698716 2024-02-12T16:43:20.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:20.732 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:20.837 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:20.862 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:20.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:20.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:20.995 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:20.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:21.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:21.128 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:21.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:21.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:21.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:21.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:21.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:21.392 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:21.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:21.441 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 10, 8, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:43:21.441 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:43:21.441 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2024-02-12T16:43:21.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:43:21.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:21.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:21 smithi169 bash[19604]: cluster 2024-02-12T16:43:20.277827+0000 mgr.y (mgr.14146) 1770 : cluster [DBG] pgmap v2578: 47 pgs: 2 active+recovering, 45 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 11716/192386 objects misplaced (6.090%); 96 KiB/s, 1 objects/s recovering 2024-02-12T16:43:21.502 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:21 smithi169 bash[19604]: cluster 2024-02-12T16:43:20.970832+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e938: 12 total, 12 up, 11 in 2024-02-12T16:43:21.522 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:21.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:21.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:21.637 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:21 smithi121 bash[18489]: cluster 2024-02-12T16:43:20.277827+0000 mgr.y (mgr.14146) 1770 : cluster [DBG] pgmap v2578: 47 pgs: 2 active+recovering, 45 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 11716/192386 objects misplaced (6.090%); 96 KiB/s, 1 objects/s recovering 2024-02-12T16:43:21.637 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:21 smithi121 bash[18489]: cluster 2024-02-12T16:43:20.970832+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e938: 12 total, 12 up, 11 in 2024-02-12T16:43:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:21 smithi123 bash[19119]: cluster 2024-02-12T16:43:20.277827+0000 mgr.y (mgr.14146) 1770 : cluster [DBG] pgmap v2578: 47 pgs: 2 active+recovering, 45 active+clean; 5.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 11716/192386 objects misplaced (6.090%); 96 KiB/s, 1 objects/s recovering 2024-02-12T16:43:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:21 smithi123 bash[19119]: cluster 2024-02-12T16:43:20.970832+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e938: 12 total, 12 up, 11 in 2024-02-12T16:43:21.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 45 16 102026 102010 141.666 79.6875 0.0111677 0.00705574 2024-02-12T16:43:21.654 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:21.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:21.759 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:21.789 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:21.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:21.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:21.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:21.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:22.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:22.047 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:22.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:22.178 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:22.284 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:22.309 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:22.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:22.440 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:22.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:22.572 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:22.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 46 16 103240 103224 140.235 75.875 0.00595819 0.00712737 2024-02-12T16:43:22.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:22.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:22.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:22.835 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:22.942 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:22.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:22.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:23.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:23.107 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:23.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:23.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:23.322 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:23.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:23.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:23.458 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:23.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:23.485 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:23.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:23.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:23 smithi121 bash[18489]: cluster 2024-02-12T16:43:22.278321+0000 mgr.y (mgr.14146) 1771 : cluster [DBG] pgmap v2580: 47 pgs: 3 active+recovering, 44 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 103 MiB/s wr, 1.66k op/s; 17337/201090 objects misplaced (8.622%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:43:23.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:23.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:23 smithi123 bash[19119]: cluster 2024-02-12T16:43:22.278321+0000 mgr.y (mgr.14146) 1771 : cluster [DBG] pgmap v2580: 47 pgs: 3 active+recovering, 44 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 103 MiB/s wr, 1.66k op/s; 17337/201090 objects misplaced (8.622%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:43:23.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 47 16 104252 104236 138.597 63.25 0.00373343 0.00720324 2024-02-12T16:43:23.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:23.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:23 smithi169 bash[19604]: cluster 2024-02-12T16:43:22.278321+0000 mgr.y (mgr.14146) 1771 : cluster [DBG] pgmap v2580: 47 pgs: 3 active+recovering, 44 active+clean; 6.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 103 MiB/s wr, 1.66k op/s; 17337/201090 objects misplaced (8.622%); 2.3 MiB/s, 36 objects/s recovering 2024-02-12T16:43:23.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:23.849 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:23.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:23.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:23.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:24.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:24.130 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:24.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:24.260 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:24.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:24.395 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:24.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:24.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:24.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:24.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 48 16 105355 105339 137.146 68.9375 0.0170127 0.00728895 2024-02-12T16:43:24.655 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:24.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:24.785 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:24.888 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:24.920 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:24.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:25.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:25.052 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:25.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:25.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:25.284 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:25.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:25.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:25.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:25.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:25.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:25 smithi123 bash[19119]: cluster 2024-02-12T16:43:24.278744+0000 mgr.y (mgr.14146) 1772 : cluster [DBG] pgmap v2581: 47 pgs: 3 active+recovering, 44 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 17337/205422 objects misplaced (8.440%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:43:25.581 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:25.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 49 16 106558 106542 135.881 75.1875 0.00730294 0.00735637 2024-02-12T16:43:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:25 smithi121 bash[18489]: cluster 2024-02-12T16:43:24.278744+0000 mgr.y (mgr.14146) 1772 : cluster [DBG] pgmap v2581: 47 pgs: 3 active+recovering, 44 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 17337/205422 objects misplaced (8.440%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:43:25.687 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:25.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:25 smithi169 bash[19604]: cluster 2024-02-12T16:43:24.278744+0000 mgr.y (mgr.14146) 1772 : cluster [DBG] pgmap v2581: 47 pgs: 3 active+recovering, 44 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 17337/205422 objects misplaced (8.440%); 1.8 MiB/s, 28 objects/s recovering 2024-02-12T16:43:25.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:25.852 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:25.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:25.990 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:25.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:26.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:26.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:26.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:26.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:26.258 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:26.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:26.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:26.392 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:26.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:26.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:26.521 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:26.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:26.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:26.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:26.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 50 16 107719 107703 134.615 72.5625 0.00436461 0.00742545 2024-02-12T16:43:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:26.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:26.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:26.795 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:26.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:26.901 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:26.904 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:43:26.923 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:26.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:26.935 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:43:26.972 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:43:26.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:43:27.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:27.036 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:43:27.048 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:43:27.093 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:43:27.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:43:27.144 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:43:27.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:26 smithi121 bash[18489]: cluster 2024-02-12T16:43:26.279181+0000 mgr.y (mgr.14146) 1773 : cluster [DBG] pgmap v2582: 47 pgs: 3 active+recovering, 44 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.21k op/s; 17278/206938 objects misplaced (8.349%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:43:27.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:26 smithi121 bash[18489]: audit 2024-02-12T16:43:26.867598+0000 mon.c (mon.1) 107 : audit [DBG] from='client.? v1:172.21.15.121:0/660586582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:27.170 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:43:27.179 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:27.198 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:43:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.225 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.226 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:43:27.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:26 smithi169 bash[19604]: cluster 2024-02-12T16:43:26.279181+0000 mgr.y (mgr.14146) 1773 : cluster [DBG] pgmap v2582: 47 pgs: 3 active+recovering, 44 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.21k op/s; 17278/206938 objects misplaced (8.349%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:43:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:26 smithi169 bash[19604]: audit 2024-02-12T16:43:26.867598+0000 mon.c (mon.1) 107 : audit [DBG] from='client.? v1:172.21.15.121:0/660586582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:27.316 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:26 smithi123 bash[19119]: cluster 2024-02-12T16:43:26.279181+0000 mgr.y (mgr.14146) 1773 : cluster [DBG] pgmap v2582: 47 pgs: 3 active+recovering, 44 active+clean; 6.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.21k op/s; 17278/206938 objects misplaced (8.349%); 2.2 MiB/s, 35 objects/s recovering 2024-02-12T16:43:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:26 smithi123 bash[19119]: audit 2024-02-12T16:43:26.867598+0000 mon.c (mon.1) 107 : audit [DBG] from='client.? v1:172.21.15.121:0/660586582' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:27.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:27.455 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:27.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:27.583 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:27.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 51 16 108920 108904 133.447 75.0625 0.00748371 0.00749087 2024-02-12T16:43:27.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:27.713 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:27.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:27.846 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:27.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:27.974 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:27.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:28.002 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:43:28.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:28.086 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:27 smithi169 bash[19604]: audit 2024-02-12T16:43:27.155365+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? v1:172.21.15.121:0/2461217556' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:43:28.103 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:28.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:28.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:28.232 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:28.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:28.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:28.345 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:27 smithi121 bash[18489]: audit 2024-02-12T16:43:27.155365+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? v1:172.21.15.121:0/2461217556' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:43:28.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:28.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:28.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:27 smithi123 bash[19119]: audit 2024-02-12T16:43:27.155365+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? v1:172.21.15.121:0/2461217556' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:43:28.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:28.494 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:28.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:28.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:28.628 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:28.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:28.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 52 16 110084 110068 132.28 72.75 0.0396485 0.00755471 2024-02-12T16:43:28.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:28.780 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:28.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:28.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:28.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:28.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:28.991 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:28 smithi169 bash[19604]: cluster 2024-02-12T16:43:27.982298+0000 mon.a (mon.0) 3152 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:43:28.991 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:28 smithi169 bash[19604]: audit 2024-02-12T16:43:27.994135+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.121:0/2461217556' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:43:28.991 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:28 smithi169 bash[19604]: cluster 2024-02-12T16:43:27.994185+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e939: 12 total, 12 up, 11 in 2024-02-12T16:43:28.991 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:28 smithi169 bash[19604]: cluster 2024-02-12T16:43:28.254547+0000 osd.2 (osd.2) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:28.991 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:28 smithi169 bash[19604]: cluster 2024-02-12T16:43:28.254606+0000 osd.2 (osd.2) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:28.991 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:28 smithi169 bash[19604]: cluster 2024-02-12T16:43:28.282203+0000 mgr.y (mgr.14146) 1774 : cluster [DBG] pgmap v2584: 47 pgs: 3 active+recovering, 44 active+clean; 6.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s; 17093/212612 objects misplaced (8.040%); 3.6 MiB/s, 58 objects/s recovering 2024-02-12T16:43:29.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:29.045 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:29.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:29.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:28 smithi121 bash[18489]: cluster 2024-02-12T16:43:27.982298+0000 mon.a (mon.0) 3152 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:43:29.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:28 smithi121 bash[18489]: audit 2024-02-12T16:43:27.994135+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.121:0/2461217556' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:43:29.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:28 smithi121 bash[18489]: cluster 2024-02-12T16:43:27.994185+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e939: 12 total, 12 up, 11 in 2024-02-12T16:43:29.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:28 smithi121 bash[18489]: cluster 2024-02-12T16:43:28.254547+0000 osd.2 (osd.2) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:29.178 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:28 smithi121 bash[18489]: cluster 2024-02-12T16:43:28.254606+0000 osd.2 (osd.2) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:29.179 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:28 smithi121 bash[18489]: cluster 2024-02-12T16:43:28.282203+0000 mgr.y (mgr.14146) 1774 : cluster [DBG] pgmap v2584: 47 pgs: 3 active+recovering, 44 active+clean; 6.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s; 17093/212612 objects misplaced (8.040%); 3.6 MiB/s, 58 objects/s recovering 2024-02-12T16:43:29.189 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:29.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:29.326 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:28 smithi123 bash[19119]: cluster 2024-02-12T16:43:27.982298+0000 mon.a (mon.0) 3152 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:43:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:28 smithi123 bash[19119]: audit 2024-02-12T16:43:27.994135+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.121:0/2461217556' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:43:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:28 smithi123 bash[19119]: cluster 2024-02-12T16:43:27.994185+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e939: 12 total, 12 up, 11 in 2024-02-12T16:43:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:28 smithi123 bash[19119]: cluster 2024-02-12T16:43:28.254547+0000 osd.2 (osd.2) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:28 smithi123 bash[19119]: cluster 2024-02-12T16:43:28.254606+0000 osd.2 (osd.2) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:29.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:28 smithi123 bash[19119]: cluster 2024-02-12T16:43:28.282203+0000 mgr.y (mgr.14146) 1774 : cluster [DBG] pgmap v2584: 47 pgs: 3 active+recovering, 44 active+clean; 6.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s; 17093/212612 objects misplaced (8.040%); 3.6 MiB/s, 58 objects/s recovering 2024-02-12T16:43:29.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:29.462 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:29.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:29.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 53 16 111174 111158 131.069 68.125 0.0405208 0.00762554 2024-02-12T16:43:29.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:29.728 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:29.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:29.859 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:29.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:29.990 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:29.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:30.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:30.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:30.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:30 smithi123 bash[19119]: cluster 2024-02-12T16:43:28.531774+0000 osd.8 (osd.8) 143 : cluster [DBG] 9.b scrub starts 2024-02-12T16:43:30.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:30 smithi123 bash[19119]: cluster 2024-02-12T16:43:29.002026+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e940: 12 total, 12 up, 11 in 2024-02-12T16:43:30.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:30 smithi123 bash[19119]: audit 2024-02-12T16:43:29.996208+0000 mon.a (mon.0) 3156 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:30.238 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:30.271 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:30.273 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:30 smithi169 bash[19604]: cluster 2024-02-12T16:43:28.531774+0000 osd.8 (osd.8) 143 : cluster [DBG] 9.b scrub starts 2024-02-12T16:43:30.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:30 smithi169 bash[19604]: cluster 2024-02-12T16:43:29.002026+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e940: 12 total, 12 up, 11 in 2024-02-12T16:43:30.274 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:30 smithi169 bash[19604]: audit 2024-02-12T16:43:29.996208+0000 mon.a (mon.0) 3156 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:30.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:30.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:30.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:30.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:30 smithi121 bash[18489]: cluster 2024-02-12T16:43:28.531774+0000 osd.8 (osd.8) 143 : cluster [DBG] 9.b scrub starts 2024-02-12T16:43:30.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:30 smithi121 bash[18489]: cluster 2024-02-12T16:43:29.002026+0000 mon.a (mon.0) 3155 : cluster [DBG] osdmap e940: 12 total, 12 up, 11 in 2024-02-12T16:43:30.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:30 smithi121 bash[18489]: audit 2024-02-12T16:43:29.996208+0000 mon.a (mon.0) 3156 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:30.511 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:30.538 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:30.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:30.555 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:30.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:30.648 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 54 16 112335 112319 129.986 72.5625 0.0174104 0.00769005 2024-02-12T16:43:30.774 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:30.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:30.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:30.901 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:30.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:31.004 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:31.029 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:31.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:31.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:31.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:31.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:31 smithi121 bash[18489]: cluster 2024-02-12T16:43:30.282638+0000 mgr.y (mgr.14146) 1775 : cluster [DBG] pgmap v2586: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 43 active+clean; 6.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s; 17093/217164 objects misplaced (7.871%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:43:31.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:31 smithi169 bash[19604]: cluster 2024-02-12T16:43:30.282638+0000 mgr.y (mgr.14146) 1775 : cluster [DBG] pgmap v2586: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 43 active+clean; 6.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s; 17093/217164 objects misplaced (7.871%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:43:31.302 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:31.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:31.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:31 smithi123 bash[19119]: cluster 2024-02-12T16:43:30.282638+0000 mgr.y (mgr.14146) 1775 : cluster [DBG] pgmap v2586: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 43 active+clean; 6.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s; 17093/217164 objects misplaced (7.871%); 1.9 MiB/s, 30 objects/s recovering 2024-02-12T16:43:31.406 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:31.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:31.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:31.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:31.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 55 16 113435 113419 128.872 68.75 0.00650169 0.0077576 2024-02-12T16:43:31.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:31.709 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:31.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:31.839 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:31.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:31.972 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:31.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:32.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:32 smithi123 bash[19119]: cluster 2024-02-12T16:43:30.452517+0000 osd.8 (osd.8) 144 : cluster [DBG] 9.b scrub ok 2024-02-12T16:43:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:32 smithi123 bash[19119]: cluster 2024-02-12T16:43:30.950463+0000 osd.7 (osd.7) 7 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:43:32.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:32 smithi123 bash[19119]: cluster 2024-02-12T16:43:31.016002+0000 osd.7 (osd.7) 8 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:43:32.105 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:32.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:32.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:43:32.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:32.216 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:32 smithi169 bash[19604]: cluster 2024-02-12T16:43:30.452517+0000 osd.8 (osd.8) 144 : cluster [DBG] 9.b scrub ok 2024-02-12T16:43:32.216 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:32 smithi169 bash[19604]: cluster 2024-02-12T16:43:30.950463+0000 osd.7 (osd.7) 7 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:43:32.216 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:32 smithi169 bash[19604]: cluster 2024-02-12T16:43:31.016002+0000 osd.7 (osd.7) 8 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:43:32.234 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:32.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:32.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:32.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:32 smithi121 bash[18489]: cluster 2024-02-12T16:43:30.452517+0000 osd.8 (osd.8) 144 : cluster [DBG] 9.b scrub ok 2024-02-12T16:43:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:32 smithi121 bash[18489]: cluster 2024-02-12T16:43:30.950463+0000 osd.7 (osd.7) 7 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:43:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:32 smithi121 bash[18489]: cluster 2024-02-12T16:43:31.016002+0000 osd.7 (osd.7) 8 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:43:32.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:32.496 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:32.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:32.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:32.631 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:32.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:32.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 56 16 114579 114563 127.848 71.5 0.00464831 0.00781994 2024-02-12T16:43:32.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:32.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:32.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:32.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:32.893 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:32.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:32.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:43:32.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:33.020 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:33.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:33.167 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:33.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:33.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:33 smithi121 bash[18489]: cluster 2024-02-12T16:43:31.499450+0000 osd.8 (osd.8) 145 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:43:33.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:33 smithi121 bash[18489]: cluster 2024-02-12T16:43:31.617628+0000 osd.8 (osd.8) 146 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:43:33.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:33 smithi121 bash[18489]: cluster 2024-02-12T16:43:32.283158+0000 mgr.y (mgr.14146) 1776 : cluster [DBG] pgmap v2587: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 43 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 16858/224302 objects misplaced (7.516%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T16:43:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:33 smithi169 bash[19604]: cluster 2024-02-12T16:43:31.499450+0000 osd.8 (osd.8) 145 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:43:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:33 smithi169 bash[19604]: cluster 2024-02-12T16:43:31.617628+0000 osd.8 (osd.8) 146 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:43:33.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:33 smithi169 bash[19604]: cluster 2024-02-12T16:43:32.283158+0000 mgr.y (mgr.14146) 1776 : cluster [DBG] pgmap v2587: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 43 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 16858/224302 objects misplaced (7.516%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T16:43:33.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:33.372 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:43:33.372 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:43:33.372 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:43:33.372 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:43:33.372 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:43:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:33 smithi123 bash[19119]: cluster 2024-02-12T16:43:31.499450+0000 osd.8 (osd.8) 145 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:43:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:33 smithi123 bash[19119]: cluster 2024-02-12T16:43:31.617628+0000 osd.8 (osd.8) 146 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:43:33.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:33 smithi123 bash[19119]: cluster 2024-02-12T16:43:32.283158+0000 mgr.y (mgr.14146) 1776 : cluster [DBG] pgmap v2587: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering, 43 active+clean; 6.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 16858/224302 objects misplaced (7.516%); 3.7 MiB/s, 59 objects/s recovering 2024-02-12T16:43:33.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:33.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:33.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:33.577 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:33.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 57 16 115728 115712 126.864 71.8125 0.0124162 0.00787952 2024-02-12T16:43:33.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:33.706 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:33.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:33.839 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:33.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:33.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:33.966 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:33.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:34.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:34.095 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:34.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:34.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:34.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:34.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:34.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:34.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:34.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:34.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:34.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:34.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:34.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:34.626 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:34.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:34.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 58 16 116924 116908 125.966 74.75 0.00504956 0.00793657 2024-02-12T16:43:34.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:34.899 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:34.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:35.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:35.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:43:35.028 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:35.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:35.164 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:35.272 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:35.300 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:35.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:35.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:35.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:35 smithi169 bash[19604]: cluster 2024-02-12T16:43:34.283657+0000 mgr.y (mgr.14146) 1777 : cluster [DBG] pgmap v2588: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s; 16858/228566 objects misplaced (7.376%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:43:35.563 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:35 smithi123 bash[19119]: cluster 2024-02-12T16:43:34.283657+0000 mgr.y (mgr.14146) 1777 : cluster [DBG] pgmap v2588: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s; 16858/228566 objects misplaced (7.376%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:43:35.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 59 16 118149 118133 125.128 76.5625 0.00491758 0.00798964 2024-02-12T16:43:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:35 smithi121 bash[18489]: cluster 2024-02-12T16:43:34.283657+0000 mgr.y (mgr.14146) 1777 : cluster [DBG] pgmap v2588: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s; 16858/228566 objects misplaced (7.376%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:43:35.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:35.689 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:35.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:35.820 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:35.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:35.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:35.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:36.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:36.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:36.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:36.212 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:36.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:36.315 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:36.353 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:36.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:36.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:36.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:36.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:36 smithi169 bash[19604]: cluster 2024-02-12T16:43:34.871494+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:36 smithi169 bash[19604]: cluster 2024-02-12T16:43:34.871553+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:36 smithi169 bash[19604]: cluster 2024-02-12T16:43:35.339300+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e941: 12 total, 12 up, 11 in 2024-02-12T16:43:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:36 smithi169 bash[19604]: cluster 2024-02-12T16:43:35.976554+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e942: 12 total, 12 up, 11 in 2024-02-12T16:43:36.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:36.612 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:36.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:36 smithi123 bash[19119]: cluster 2024-02-12T16:43:34.871494+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:36 smithi123 bash[19119]: cluster 2024-02-12T16:43:34.871553+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:36 smithi123 bash[19119]: cluster 2024-02-12T16:43:35.339300+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e941: 12 total, 12 up, 11 in 2024-02-12T16:43:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:36 smithi123 bash[19119]: cluster 2024-02-12T16:43:35.976554+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e942: 12 total, 12 up, 11 in 2024-02-12T16:43:36.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:43:36.646646 min lat: 0.000995729 max lat: 4.35085 avg lat: 0.00806269 2024-02-12T16:43:36.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:43:36.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 60 16 119065 119049 123.997 57.25 0.010066 0.00806269 2024-02-12T16:43:36.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:36 smithi121 bash[18489]: cluster 2024-02-12T16:43:34.871494+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:36.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:36 smithi121 bash[18489]: cluster 2024-02-12T16:43:34.871553+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:36.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:36 smithi121 bash[18489]: cluster 2024-02-12T16:43:35.339300+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e941: 12 total, 12 up, 11 in 2024-02-12T16:43:36.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:36 smithi121 bash[18489]: cluster 2024-02-12T16:43:35.976554+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e942: 12 total, 12 up, 11 in 2024-02-12T16:43:36.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:36.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:36.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:36.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:36.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:36.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:36.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:37.015 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:37.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:37.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:37.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:37.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:37.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:37 smithi121 bash[18489]: cluster 2024-02-12T16:43:36.284090+0000 mgr.y (mgr.14146) 1778 : cluster [DBG] pgmap v2591: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 61 MiB/s wr, 982 op/s; 16800/229920 objects misplaced (7.307%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T16:43:37.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:37.517 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:37.542 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:37 smithi169 bash[19604]: cluster 2024-02-12T16:43:36.284090+0000 mgr.y (mgr.14146) 1778 : cluster [DBG] pgmap v2591: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 61 MiB/s wr, 982 op/s; 16800/229920 objects misplaced (7.307%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T16:43:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:37 smithi123 bash[19119]: cluster 2024-02-12T16:43:36.284090+0000 mgr.y (mgr.14146) 1778 : cluster [DBG] pgmap v2591: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 61 MiB/s wr, 982 op/s; 16800/229920 objects misplaced (7.307%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T16:43:37.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:37.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 61 16 120202 120186 123.129 71.0625 0.0203239 0.00811967 2024-02-12T16:43:37.679 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:37.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:37.908 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:37.940 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:37.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:37.952 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:43:37.969 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:43:38.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:38.074 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:38.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:38.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:38.216 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:38.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:38.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:38.357 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:38.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:38.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:38 smithi121 bash[18489]: cluster 2024-02-12T16:43:36.741765+0000 osd.11 (osd.11) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:38.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:38 smithi121 bash[18489]: cluster 2024-02-12T16:43:36.741819+0000 osd.11 (osd.11) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:38.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:38 smithi121 bash[18489]: cluster 2024-02-12T16:43:37.351256+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e943: 12 total, 12 up, 11 in 2024-02-12T16:43:38.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:38.496 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:38.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:38.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:38.607 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:38 smithi123 bash[19119]: cluster 2024-02-12T16:43:36.741765+0000 osd.11 (osd.11) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:38.608 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:38 smithi123 bash[19119]: cluster 2024-02-12T16:43:36.741819+0000 osd.11 (osd.11) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:38.608 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:38 smithi123 bash[19119]: cluster 2024-02-12T16:43:37.351256+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e943: 12 total, 12 up, 11 in 2024-02-12T16:43:38.641 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:38.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:38.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 62 16 121344 121328 122.294 71.375 0.00941551 0.00817456 2024-02-12T16:43:38.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:38.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:38 smithi169 bash[19604]: cluster 2024-02-12T16:43:36.741765+0000 osd.11 (osd.11) 223 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:38 smithi169 bash[19604]: cluster 2024-02-12T16:43:36.741819+0000 osd.11 (osd.11) 224 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:38.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:38 smithi169 bash[19604]: cluster 2024-02-12T16:43:37.351256+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e943: 12 total, 12 up, 11 in 2024-02-12T16:43:38.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:38.913 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:38.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:39.019 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:39.036 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2592,"stamp":"2024-02-12T16:43:37.348498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7721844736,"num_objects":117826,"num_object_clones":0,"num_object_copies":235652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16601,"num_objects_unfound":0,"num_objects_dirty":117826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117826,"num_write_kb":7540864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":63438848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117826,"ondisk_log_size":117826,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":22022676,"kb_used_data":13958908,"kb_used_omap":92,"kb_used_meta":8063587,"kb_avail":1009173996,"statfs":{"total":1055945392128,"available":1033394171904,"internally_reserved":0,"allocated":14293921792,"data_stored":14285737856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95164,"internal_metadata":8257113156},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,8,5,5,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":605814784,"num_objects":9244,"num_object_clones":0,"num_object_copies":18488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-492,"num_objects_unfound":0,"num_objects_dirty":9244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9244,"num_write_kb":591616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":32243712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.066111"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":304,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.004113+0000","last_change":"2024-02-12T16:42:21.631979+0000","last_active":"2024-02-12T16:43:28.004113+0000","last_peered":"2024-02-12T16:43:28.004113+0000","last_clean":"2024-02-12T16:43:28.004113+0000","last_became_active":"2024-02-12T16:41:18.947003+0000","last_became_peered":"2024-02-12T16:41:18.947003+0000","last_unstale":"2024-02-12T16:43:28.004113+0000","last_undegraded":"2024-02-12T16:43:28.004113+0000","last_fullsized":"2024-02-12T16:43:28.004113+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:21.631859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:21.631859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":334,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.345937+0000","last_change":"2024-02-12T16:42:29.489386+0000","last_active":"2024-02-12T16:43:35.345937+0000","last_peered":"2024-02-12T16:43:35.345937+0000","last_clean":"2024-02-12T16:43:35.345937+0000","last_became_active":"2024-02-12T16:41:17.943883+0000","last_became_peered":"2024-02-12T16:41:17.943883+0000","last_unstale":"2024-02-12T16:43:35.345937+0000","last_undegraded":"2024-02-12T16:43:35.345937+0000","last_fullsized":"2024-02-12T16:43:35.345937+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:29.489294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:29.489294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":317,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:27.997407+0000","last_change":"2024-02-12T16:42:51.684068+0000","last_active":"2024-02-12T16:43:27.997407+0000","last_peered":"2024-02-12T16:43:27.997407+0000","last_clean":"2024-02-12T16:43:27.997407+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:43:27.997407+0000","last_undegraded":"2024-02-12T16:43:27.997407+0000","last_fullsized":"2024-02-12T16:43:27.997407+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.683932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.683932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":114,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:32.114160+0000","last_change":"2024-02-12T16:43:32.114759+0000","last_active":"2024-02-12T16:43:32.114160+0000","last_peered":"2024-02-12T16:43:32.114160+0000","last_clean":"2024-02-12T16:43:32.114160+0000","last_became_active":"2024-02-12T16:39:09.436840+0000","last_became_peered":"2024-02-12T16:39:09.436840+0000","last_unstale":"2024-02-12T16:43:32.114160+0000","last_undegraded":"2024-02-12T16:43:32.114160+0000","last_fullsized":"2024-02-12T16:43:32.114160+0000","mapping_epoch":845,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":846,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_clean_scrub_stamp":"2024-02-12T16:41:21.337430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":369,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.006807+0000","last_change":"2024-02-12T16:42:41.310928+0000","last_active":"2024-02-12T16:43:28.006807+0000","last_peered":"2024-02-12T16:43:28.006807+0000","last_clean":"2024-02-12T16:43:28.006807+0000","last_became_active":"2024-02-12T16:42:40.975420+0000","last_became_peered":"2024-02-12T16:42:40.975420+0000","last_unstale":"2024-02-12T16:43:28.006807+0000","last_undegraded":"2024-02-12T16:43:28.006807+0000","last_fullsized":"2024-02-12T16:43:28.006807+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:41.310840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:41.310840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":330,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.000438+0000","last_change":"2024-02-12T16:42:24.921381+0000","last_active":"2024-02-12T16:43:28.000438+0000","last_peered":"2024-02-12T16:43:28.000438+0000","last_clean":"2024-02-12T16:43:28.000438+0000","last_became_active":"2024-02-12T16:36:17.847960+0000","last_became_peered":"2024-02-12T16:36:17.847960+0000","last_unstale":"2024-02-12T16:43:28.000438+0000","last_undegraded":"2024-02-12T16:43:28.000438+0000","last_fullsized":"2024-02-12T16:43:28.000438+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:24.921234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:24.921234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":676,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.346751+0000","last_change":"2024-02-12T16:42:31.552860+0000","last_active":"2024-02-12T16:43:35.346751+0000","last_peered":"2024-02-12T16:43:35.346751+0000","last_clean":"2024-02-12T16:43:35.346751+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:43:35.346751+0000","last_undegraded":"2024-02-12T16:43:35.346751+0000","last_fullsized":"2024-02-12T16:43:35.346751+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:31.552623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:42:31.552623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":321,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:31.617757+0000","last_change":"2024-02-12T16:43:31.617757+0000","last_active":"2024-02-12T16:43:31.617757+0000","last_peered":"2024-02-12T16:43:31.617757+0000","last_clean":"2024-02-12T16:43:31.617757+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:43:31.617757+0000","last_undegraded":"2024-02-12T16:43:31.617757+0000","last_fullsized":"2024-02-12T16:43:31.617757+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.617636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.617636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":318,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.342753+0000","last_change":"2024-02-12T16:42:26.378442+0000","last_active":"2024-02-12T16:43:35.342753+0000","last_peered":"2024-02-12T16:43:35.342753+0000","last_clean":"2024-02-12T16:43:35.342753+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:43:35.342753+0000","last_undegraded":"2024-02-12T16:43:35.342753+0000","last_fullsized":"2024-02-12T16:43:35.342753+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.378312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":237,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:32.933207+0000","last_change":"2024-02-12T16:43:32.933826+0000","last_active":"2024-02-12T16:43:32.933207+0000","last_peered":"2024-02-12T16:43:32.933207+0000","last_clean":"2024-02-12T16:43:32.933207+0000","last_became_active":"2024-02-12T16:42:40.982346+0000","last_became_peered":"2024-02-12T16:42:40.982346+0000","last_unstale":"2024-02-12T16:43:32.933207+0000","last_undegraded":"2024-02-12T16:43:32.933207+0000","last_fullsized":"2024-02-12T16:43:32.933207+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:11.207557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:39:11.207557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":574,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.713041+0000","last_change":"2024-02-12T16:42:32.870964+0000","last_active":"2024-02-12T16:43:35.713041+0000","last_peered":"2024-02-12T16:43:35.713041+0000","last_clean":"2024-02-12T16:43:35.713041+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:43:35.713041+0000","last_undegraded":"2024-02-12T16:43:35.713041+0000","last_fullsized":"2024-02-12T16:43:35.713041+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:32.870856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:42:32.870856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"940'7278","reported_seq":7757,"reported_epoch":940,"state":"active+recovering","last_fresh":"2024-02-12T16:43:35.272351+0000","last_change":"2024-02-12T16:43:16.238278+0000","last_active":"2024-02-12T16:43:35.272351+0000","last_peered":"2024-02-12T16:43:35.272351+0000","last_clean":"2024-02-12T16:43:15.331170+0000","last_became_active":"2024-02-12T16:43:16.169213+0000","last_became_peered":"2024-02-12T16:43:16.169213+0000","last_unstale":"2024-02-12T16:43:35.272351+0000","last_undegraded":"2024-02-12T16:43:35.272351+0000","last_fullsized":"2024-02-12T16:43:35.272351+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7278,"ondisk_log_size":7278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":476971008,"num_objects":7278,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5622,"num_objects_unfound":0,"num_objects_dirty":7278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7278,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14548992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","9"],"object_location_counts":[{"shards":"5,11","objects":1656},{"shards":"9,11","objects":5622}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:27.999164+0000","last_change":"2024-02-12T16:42:41.980681+0000","last_active":"2024-02-12T16:43:27.999164+0000","last_peered":"2024-02-12T16:43:27.999164+0000","last_clean":"2024-02-12T16:43:27.999164+0000","last_became_active":"2024-02-12T16:42:41.980413+0000","last_became_peered":"2024-02-12T16:42:41.980413+0000","last_unstale":"2024-02-12T16:43:27.999164+0000","last_undegraded":"2024-02-12T16:43:27.999164+0000","last_fullsized":"2024-02-12T16:43:27.999164+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.611860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.611860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"940'7142","reported_seq":7166,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:34.152763+0000","last_change":"2024-02-12T16:42:40.991625+0000","last_active":"2024-02-12T16:43:34.152763+0000","last_peered":"2024-02-12T16:43:34.152763+0000","last_clean":"2024-02-12T16:43:34.152763+0000","last_became_active":"2024-02-12T16:42:40.984344+0000","last_became_peered":"2024-02-12T16:42:40.984344+0000","last_unstale":"2024-02-12T16:43:34.152763+0000","last_undegraded":"2024-02-12T16:43:34.152763+0000","last_fullsized":"2024-02-12T16:43:34.152763+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7142,"ondisk_log_size":7142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468058112,"num_objects":7142,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":457088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":685,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.000214+0000","last_change":"2024-02-12T16:41:25.248527+0000","last_active":"2024-02-12T16:43:28.000214+0000","last_peered":"2024-02-12T16:43:28.000214+0000","last_clean":"2024-02-12T16:43:28.000214+0000","last_became_active":"2024-02-12T16:36:17.846631+0000","last_became_peered":"2024-02-12T16:36:17.846631+0000","last_unstale":"2024-02-12T16:43:28.000214+0000","last_undegraded":"2024-02-12T16:43:28.000214+0000","last_fullsized":"2024-02-12T16:43:28.000214+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:25.248423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:41:25.248423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"942'7443","reported_seq":7476,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-02-12T16:43:36.414695+0000","last_change":"2024-02-12T16:42:21.999716+0000","last_active":"2024-02-12T16:43:36.414695+0000","last_peered":"2024-02-12T16:43:36.414695+0000","last_clean":"2024-02-12T16:43:36.414695+0000","last_became_active":"2024-02-12T16:42:21.999181+0000","last_became_peered":"2024-02-12T16:42:21.999181+0000","last_unstale":"2024-02-12T16:43:36.414695+0000","last_undegraded":"2024-02-12T16:43:36.414695+0000","last_fullsized":"2024-02-12T16:43:36.414695+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7443,"ondisk_log_size":7443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":487784448,"num_objects":7443,"num_object_clones":0,"num_object_copies":14886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":476352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1051,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.004036+0000","last_change":"2024-02-12T16:42:38.439762+0000","last_active":"2024-02-12T16:43:28.004036+0000","last_peered":"2024-02-12T16:43:28.004036+0000","last_clean":"2024-02-12T16:43:28.004036+0000","last_became_active":"2024-02-12T16:41:18.947424+0000","last_became_peered":"2024-02-12T16:41:18.947424+0000","last_unstale":"2024-02-12T16:43:28.004036+0000","last_undegraded":"2024-02-12T16:43:28.004036+0000","last_fullsized":"2024-02-12T16:43:28.004036+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:38.439663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:39:15.947507+0000","last_clean_scrub_stamp":"2024-02-12T16:42:38.439663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.f","version":"942'7584","reported_seq":7617,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-02-12T16:43:36.976680+0000","last_change":"2024-02-12T16:42:21.964154+0000","last_active":"2024-02-12T16:43:36.976680+0000","last_peered":"2024-02-12T16:43:36.976680+0000","last_clean":"2024-02-12T16:43:36.976680+0000","last_became_active":"2024-02-12T16:42:21.963855+0000","last_became_peered":"2024-02-12T16:42:21.963855+0000","last_unstale":"2024-02-12T16:43:36.976680+0000","last_undegraded":"2024-02-12T16:43:36.976680+0000","last_fullsized":"2024-02-12T16:43:36.976680+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7584,"ondisk_log_size":7584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":497025024,"num_objects":7584,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":723,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.014725+0000","last_change":"2024-02-12T16:42:23.543471+0000","last_active":"2024-02-12T16:43:28.014725+0000","last_peered":"2024-02-12T16:43:28.014725+0000","last_clean":"2024-02-12T16:43:28.014725+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:43:28.014725+0000","last_undegraded":"2024-02-12T16:43:28.014725+0000","last_fullsized":"2024-02-12T16:43:28.014725+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.543386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.543386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"940'7399","reported_seq":7423,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:34.177887+0000","last_change":"2024-02-12T16:42:40.974589+0000","last_active":"2024-02-12T16:43:34.177887+0000","last_peered":"2024-02-12T16:43:34.177887+0000","last_clean":"2024-02-12T16:43:34.177887+0000","last_became_active":"2024-02-12T16:42:40.971979+0000","last_became_peered":"2024-02-12T16:42:40.971979+0000","last_unstale":"2024-02-12T16:43:34.177887+0000","last_undegraded":"2024-02-12T16:43:34.177887+0000","last_fullsized":"2024-02-12T16:43:34.177887+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7399,"ondisk_log_size":7399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":484900864,"num_objects":7399,"num_object_clones":0,"num_object_copies":14798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":677,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.000424+0000","last_change":"2024-02-12T16:43:15.977479+0000","last_active":"2024-02-12T16:43:28.000424+0000","last_peered":"2024-02-12T16:43:28.000424+0000","last_clean":"2024-02-12T16:43:28.000424+0000","last_became_active":"2024-02-12T16:43:15.976997+0000","last_became_peered":"2024-02-12T16:43:15.976997+0000","last_unstale":"2024-02-12T16:43:28.000424+0000","last_undegraded":"2024-02-12T16:43:28.000424+0000","last_fullsized":"2024-02-12T16:43:28.000424+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.923206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:57.057484+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.923206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"940'7310","reported_seq":7341,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:33.798413+0000","last_change":"2024-02-12T16:42:21.962569+0000","last_active":"2024-02-12T16:43:33.798413+0000","last_peered":"2024-02-12T16:43:33.798413+0000","last_clean":"2024-02-12T16:43:33.798413+0000","last_became_active":"2024-02-12T16:42:21.962130+0000","last_became_peered":"2024-02-12T16:42:21.962130+0000","last_unstale":"2024-02-12T16:43:33.798413+0000","last_undegraded":"2024-02-12T16:43:33.798413+0000","last_fullsized":"2024-02-12T16:43:33.798413+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7310,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":479068160,"num_objects":7310,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":467840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":720,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.000353+0000","last_change":"2024-02-12T16:42:34.813265+0000","last_active":"2024-02-12T16:43:28.000353+0000","last_peered":"2024-02-12T16:43:28.000353+0000","last_clean":"2024-02-12T16:43:28.000353+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:43:28.000353+0000","last_undegraded":"2024-02-12T16:43:28.000353+0000","last_fullsized":"2024-02-12T16:43:28.000353+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:34.813152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:42:34.813152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"940'7122","reported_seq":7158,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:33.789017+0000","last_change":"2024-02-12T16:43:30.452610+0000","last_active":"2024-02-12T16:43:33.789017+0000","last_peered":"2024-02-12T16:43:33.789017+0000","last_clean":"2024-02-12T16:43:33.789017+0000","last_became_active":"2024-02-12T16:42:21.963637+0000","last_became_peered":"2024-02-12T16:42:21.963637+0000","last_unstale":"2024-02-12T16:43:33.789017+0000","last_undegraded":"2024-02-12T16:43:33.789017+0000","last_fullsized":"2024-02-12T16:43:33.789017+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"940'6865","last_scrub_stamp":"2024-02-12T16:43:30.452522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:43:30.452522+0000","log_size":7122,"ondisk_log_size":7122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466747392,"num_objects":7122,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":527,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.338689+0000","last_change":"2024-02-12T16:41:33.930537+0000","last_active":"2024-02-12T16:43:35.338689+0000","last_peered":"2024-02-12T16:43:35.338689+0000","last_clean":"2024-02-12T16:43:35.338689+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:43:35.338689+0000","last_undegraded":"2024-02-12T16:43:35.338689+0000","last_fullsized":"2024-02-12T16:43:35.338689+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:33.930337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:33.930337+0000","last_clean_scrub_stamp":"2024-02-12T16:41:33.930337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"942'7595","reported_seq":7628,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-02-12T16:43:36.957468+0000","last_change":"2024-02-12T16:42:21.964459+0000","last_active":"2024-02-12T16:43:36.957468+0000","last_peered":"2024-02-12T16:43:36.957468+0000","last_clean":"2024-02-12T16:43:36.957468+0000","last_became_active":"2024-02-12T16:42:21.964092+0000","last_became_peered":"2024-02-12T16:42:21.964092+0000","last_unstale":"2024-02-12T16:43:36.957468+0000","last_undegraded":"2024-02-12T16:43:36.957468+0000","last_fullsized":"2024-02-12T16:43:36.957468+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7595,"ondisk_log_size":7595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":497745920,"num_objects":7595,"num_object_clones":0,"num_object_copies":15190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":486080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1013,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:31.016133+0000","last_change":"2024-02-12T16:43:31.016133+0000","last_active":"2024-02-12T16:43:31.016133+0000","last_peered":"2024-02-12T16:43:31.016133+0000","last_clean":"2024-02-12T16:43:31.016133+0000","last_became_active":"2024-02-12T16:42:41.321180+0000","last_became_peered":"2024-02-12T16:42:41.321180+0000","last_unstale":"2024-02-12T16:43:31.016133+0000","last_undegraded":"2024-02-12T16:43:31.016133+0000","last_fullsized":"2024-02-12T16:43:31.016133+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.016013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.016013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"940'7387","reported_seq":7418,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.242650+0000","last_change":"2024-02-12T16:42:21.961833+0000","last_active":"2024-02-12T16:43:35.242650+0000","last_peered":"2024-02-12T16:43:35.242650+0000","last_clean":"2024-02-12T16:43:35.242650+0000","last_became_active":"2024-02-12T16:42:21.961543+0000","last_became_peered":"2024-02-12T16:42:21.961543+0000","last_unstale":"2024-02-12T16:43:35.242650+0000","last_undegraded":"2024-02-12T16:43:35.242650+0000","last_fullsized":"2024-02-12T16:43:35.242650+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7387,"ondisk_log_size":7387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":484114432,"num_objects":7387,"num_object_clones":0,"num_object_copies":14774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":432,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:27.999304+0000","last_change":"2024-02-12T16:42:43.033723+0000","last_active":"2024-02-12T16:43:27.999304+0000","last_peered":"2024-02-12T16:43:27.999304+0000","last_clean":"2024-02-12T16:43:27.999304+0000","last_became_active":"2024-02-12T16:42:40.975171+0000","last_became_peered":"2024-02-12T16:42:40.975171+0000","last_unstale":"2024-02-12T16:43:27.999304+0000","last_undegraded":"2024-02-12T16:43:27.999304+0000","last_fullsized":"2024-02-12T16:43:27.999304+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.033576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.033576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"942'7492","reported_seq":7525,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-02-12T16:43:36.641271+0000","last_change":"2024-02-12T16:42:21.999246+0000","last_active":"2024-02-12T16:43:36.641271+0000","last_peered":"2024-02-12T16:43:36.641271+0000","last_clean":"2024-02-12T16:43:36.641271+0000","last_became_active":"2024-02-12T16:42:21.998975+0000","last_became_peered":"2024-02-12T16:42:21.998975+0000","last_unstale":"2024-02-12T16:43:36.641271+0000","last_undegraded":"2024-02-12T16:43:36.641271+0000","last_fullsized":"2024-02-12T16:43:36.641271+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7492,"ondisk_log_size":7492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":490995712,"num_objects":7492,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7492,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":232,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.709086+0000","last_change":"2024-02-12T16:43:34.683532+0000","last_active":"2024-02-12T16:43:35.709086+0000","last_peered":"2024-02-12T16:43:35.709086+0000","last_clean":"2024-02-12T16:43:35.709086+0000","last_became_active":"2024-02-12T16:42:10.950266+0000","last_became_peered":"2024-02-12T16:42:10.950266+0000","last_unstale":"2024-02-12T16:43:35.709086+0000","last_undegraded":"2024-02-12T16:43:35.709086+0000","last_fullsized":"2024-02-12T16:43:35.709086+0000","mapping_epoch":913,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":914,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:22.478963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:22.478963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"940'7289","reported_seq":7325,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:33.737225+0000","last_change":"2024-02-12T16:42:40.974611+0000","last_active":"2024-02-12T16:43:33.737225+0000","last_peered":"2024-02-12T16:43:33.737225+0000","last_clean":"2024-02-12T16:43:33.737225+0000","last_became_active":"2024-02-12T16:42:40.973291+0000","last_became_peered":"2024-02-12T16:42:40.973291+0000","last_unstale":"2024-02-12T16:43:33.737225+0000","last_undegraded":"2024-02-12T16:43:33.737225+0000","last_fullsized":"2024-02-12T16:43:33.737225+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7289,"ondisk_log_size":7289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":477691904,"num_objects":7289,"num_object_clones":0,"num_object_copies":14578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7289,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":674,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.014828+0000","last_change":"2024-02-12T16:42:51.559357+0000","last_active":"2024-02-12T16:43:28.014828+0000","last_peered":"2024-02-12T16:43:28.014828+0000","last_clean":"2024-02-12T16:43:28.014828+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:43:28.014828+0000","last_undegraded":"2024-02-12T16:43:28.014828+0000","last_fullsized":"2024-02-12T16:43:28.014828+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.559264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"942'7441","reported_seq":7474,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-02-12T16:43:36.945158+0000","last_change":"2024-02-12T16:42:21.964135+0000","last_active":"2024-02-12T16:43:36.945158+0000","last_peered":"2024-02-12T16:43:36.945158+0000","last_clean":"2024-02-12T16:43:36.945158+0000","last_became_active":"2024-02-12T16:42:21.963899+0000","last_became_peered":"2024-02-12T16:42:21.963899+0000","last_unstale":"2024-02-12T16:43:36.945158+0000","last_undegraded":"2024-02-12T16:43:36.945158+0000","last_fullsized":"2024-02-12T16:43:36.945158+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7441,"ondisk_log_size":7441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":487653376,"num_objects":7441,"num_object_clones":0,"num_object_copies":14882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7441,"num_write_kb":476224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":485,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.006940+0000","last_change":"2024-02-12T16:42:42.373090+0000","last_active":"2024-02-12T16:43:28.006940+0000","last_peered":"2024-02-12T16:43:28.006940+0000","last_clean":"2024-02-12T16:43:28.006940+0000","last_became_active":"2024-02-12T16:42:40.975493+0000","last_became_peered":"2024-02-12T16:42:40.975493+0000","last_unstale":"2024-02-12T16:43:28.006940+0000","last_undegraded":"2024-02-12T16:43:28.006940+0000","last_fullsized":"2024-02-12T16:43:28.006940+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_clean_scrub_stamp":"2024-02-12T16:42:42.373026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"942'7297","reported_seq":8360,"reported_epoch":942,"state":"active+recovering","last_fresh":"2024-02-12T16:43:36.424428+0000","last_change":"2024-02-12T16:43:16.215367+0000","last_active":"2024-02-12T16:43:36.424428+0000","last_peered":"2024-02-12T16:43:36.424428+0000","last_clean":"2024-02-12T16:43:15.351398+0000","last_became_active":"2024-02-12T16:43:16.158964+0000","last_became_peered":"2024-02-12T16:43:16.158964+0000","last_unstale":"2024-02-12T16:43:36.424428+0000","last_undegraded":"2024-02-12T16:43:36.424428+0000","last_fullsized":"2024-02-12T16:43:36.424428+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7297,"ondisk_log_size":7297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":478216192,"num_objects":7297,"num_object_clones":0,"num_object_copies":14594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5292,"num_objects_unfound":0,"num_objects_dirty":7297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7297,"num_write_kb":467008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":33619968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","9"],"object_location_counts":[{"shards":"3,6","objects":2005},{"shards":"3,9","objects":5292}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":1000,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:27.999168+0000","last_change":"2024-02-12T16:43:16.350572+0000","last_active":"2024-02-12T16:43:27.999168+0000","last_peered":"2024-02-12T16:43:27.999168+0000","last_clean":"2024-02-12T16:43:27.999168+0000","last_became_active":"2024-02-12T16:43:16.350353+0000","last_became_peered":"2024-02-12T16:43:16.350353+0000","last_unstale":"2024-02-12T16:43:27.999168+0000","last_undegraded":"2024-02-12T16:43:27.999168+0000","last_fullsized":"2024-02-12T16:43:27.999168+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_clean_scrub_stamp":"2024-02-12T16:42:48.099448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"942'7404","reported_seq":7437,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-02-12T16:43:36.690701+0000","last_change":"2024-02-12T16:42:21.959286+0000","last_active":"2024-02-12T16:43:36.690701+0000","last_peered":"2024-02-12T16:43:36.690701+0000","last_clean":"2024-02-12T16:43:36.690701+0000","last_became_active":"2024-02-12T16:42:21.959009+0000","last_became_peered":"2024-02-12T16:42:21.959009+0000","last_unstale":"2024-02-12T16:43:36.690701+0000","last_undegraded":"2024-02-12T16:43:36.690701+0000","last_fullsized":"2024-02-12T16:43:36.690701+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7404,"ondisk_log_size":7404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":485228544,"num_objects":7404,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7404,"num_write_kb":473856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":670,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.346907+0000","last_change":"2024-02-12T16:42:23.507674+0000","last_active":"2024-02-12T16:43:35.346907+0000","last_peered":"2024-02-12T16:43:35.346907+0000","last_clean":"2024-02-12T16:43:35.346907+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:43:35.346907+0000","last_undegraded":"2024-02-12T16:43:35.346907+0000","last_fullsized":"2024-02-12T16:43:35.346907+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.507539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.507539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"940'7240","reported_seq":7271,"reported_epoch":940,"state":"active+clean","last_fresh":"2024-02-12T16:43:33.789164+0000","last_change":"2024-02-12T16:42:21.963116+0000","last_active":"2024-02-12T16:43:33.789164+0000","last_peered":"2024-02-12T16:43:33.789164+0000","last_clean":"2024-02-12T16:43:33.789164+0000","last_became_active":"2024-02-12T16:42:21.962544+0000","last_became_peered":"2024-02-12T16:42:21.962544+0000","last_unstale":"2024-02-12T16:43:33.789164+0000","last_undegraded":"2024-02-12T16:43:33.789164+0000","last_fullsized":"2024-02-12T16:43:33.789164+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7240,"ondisk_log_size":7240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":64,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.004083+0000","last_change":"2024-02-12T16:42:23.724712+0000","last_active":"2024-02-12T16:43:28.004083+0000","last_peered":"2024-02-12T16:43:28.004083+0000","last_clean":"2024-02-12T16:43:28.004083+0000","last_became_active":"2024-02-12T16:41:17.958639+0000","last_became_peered":"2024-02-12T16:41:17.958639+0000","last_unstale":"2024-02-12T16:43:28.004083+0000","last_undegraded":"2024-02-12T16:43:28.004083+0000","last_fullsized":"2024-02-12T16:43:28.004083+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.724526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.724526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"942'7403","reported_seq":7909,"reported_epoch":942,"state":"active+recovering","last_fresh":"2024-02-12T16:43:36.693435+0000","last_change":"2024-02-12T16:43:16.243860+0000","last_active":"2024-02-12T16:43:36.693435+0000","last_peered":"2024-02-12T16:43:36.693435+0000","last_clean":"2024-02-12T16:43:15.356599+0000","last_became_active":"2024-02-12T16:43:16.169927+0000","last_became_peered":"2024-02-12T16:43:16.169927+0000","last_unstale":"2024-02-12T16:43:36.693435+0000","last_undegraded":"2024-02-12T16:43:36.693435+0000","last_fullsized":"2024-02-12T16:43:36.693435+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":7403,"ondisk_log_size":7403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":485163008,"num_objects":7403,"num_object_clones":0,"num_object_copies":14806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5687,"num_objects_unfound":0,"num_objects_dirty":7403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7403,"num_write_kb":473792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":15269888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","9"],"object_location_counts":[{"shards":"1,9","objects":5687},{"shards":"1,11","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":356,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.014746+0000","last_change":"2024-02-12T16:43:16.012096+0000","last_active":"2024-02-12T16:43:28.014746+0000","last_peered":"2024-02-12T16:43:28.014746+0000","last_clean":"2024-02-12T16:43:28.014746+0000","last_became_active":"2024-02-12T16:43:16.011685+0000","last_became_peered":"2024-02-12T16:43:16.011685+0000","last_unstale":"2024-02-12T16:43:28.014746+0000","last_undegraded":"2024-02-12T16:43:28.014746+0000","last_fullsized":"2024-02-12T16:43:28.014746+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:30.755995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:30.755995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":170,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:28.006674+0000","last_change":"2024-02-12T16:42:22.106390+0000","last_active":"2024-02-12T16:43:28.006674+0000","last_peered":"2024-02-12T16:43:28.006674+0000","last_clean":"2024-02-12T16:43:28.006674+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:43:28.006674+0000","last_undegraded":"2024-02-12T16:43:28.006674+0000","last_fullsized":"2024-02-12T16:43:28.006674+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:22.106269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:40:18.349211+0000","last_clean_scrub_stamp":"2024-02-12T16:42:22.106269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":189,"reported_epoch":939,"state":"active+clean","last_fresh":"2024-02-12T16:43:27.997243+0000","last_change":"2024-02-12T16:43:15.981259+0000","last_active":"2024-02-12T16:43:27.997243+0000","last_peered":"2024-02-12T16:43:27.997243+0000","last_clean":"2024-02-12T16:43:27.997243+0000","last_became_active":"2024-02-12T16:43:15.981046+0000","last_became_peered":"2024-02-12T16:43:15.981046+0000","last_unstale":"2024-02-12T16:43:27.997243+0000","last_undegraded":"2024-02-12T16:43:27.997243+0000","last_fullsized":"2024-02-12T16:43:27.997243+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:35.918516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:13.754017+0000","last_clean_scrub_stamp":"2024-02-12T16:42:35.918516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":538,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.347161+0000","last_change":"2024-02-12T16:41:31.381514+0000","last_active":"2024-02-12T16:43:35.347161+0000","last_peered":"2024-02-12T16:43:35.347161+0000","last_clean":"2024-02-12T16:43:35.347161+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:43:35.347161+0000","last_undegraded":"2024-02-12T16:43:35.347161+0000","last_fullsized":"2024-02-12T16:43:35.347161+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:31.381415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:41:31.381415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":501,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-02-12T16:43:35.342671+0000","last_change":"2024-02-12T16:42:25.380280+0000","last_active":"2024-02-12T16:43:35.342671+0000","last_peered":"2024-02-12T16:43:35.342671+0000","last_clean":"2024-02-12T16:43:35.342671+0000","last_became_active":"2024-02-12T16:41:17.950595+0000","last_became_peered":"2024-02-12T16:41:17.950595+0000","last_unstale":"2024-02-12T16:43:35.342671+0000","last_undegraded":"2024-02-12T16:43:35.342671+0000","last_fullsized":"2024-02-12T16:43:35.342671+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:25.380188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:25.380188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7721844736,"num_objects":117826,"num_object_clones":0,"num_object_copies":235652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16601,"num_objects_unfound":0,"num_objects_dirty":117826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117826,"num_write_kb":7540864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":968,"num_bytes_recovered":63438848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15507062784,"data_stored":15507062784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117826,"ondisk_log_size":117826,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431959074,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2413888,"kb_used_data":1512952,"kb_used_omap":0,"kb_used_meta":900928,"kb_avail":91331264,"statfs":{"total":95995035648,"available":93523214336,"internally_reserved":0,"allocated":1549262848,"data_stored":1548522692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922550272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88900000000000001,"15min":1.756},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.8320000000000001,"5min":3.9159999999999999,"15min":124.437},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.109,"15min":1.972},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.29199999999999998},"max":{"1min":3.492,"5min":4.4450000000000003,"15min":126.926},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 12 16:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.282,"15min":1.054},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":8.8450000000000006,"5min":8.8450000000000006,"15min":8.9849999999999994},"last":1.131},{"interface":"front","average":{"1min":1.117,"5min":1.1080000000000001,"15min":0.92700000000000005},"min":{"1min":0.33100000000000002,"5min":0.29899999999999999,"15min":0.28499999999999998},"max":{"1min":4.391,"5min":5.3949999999999996,"15min":11.167},"last":1.3180000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:43:20 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":3.2959999999999998,"15min":1.873},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.4980000000000002,"5min":219.56100000000001,"15min":219.56100000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":3.1469999999999998,"15min":1.855},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.7010000000000001,"5min":219.79300000000001,"15min":219.79300000000001},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.391,"15min":1.766},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.32800000000000001},"max":{"1min":1.6140000000000001,"5min":27.327999999999999,"15min":121.169},"last":3.7709999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.5149999999999999,"15min":1.7470000000000001},"min":{"1min":0.41799999999999998,"5min":0.35299999999999998,"15min":0.32300000000000001},"max":{"1min":4.6420000000000003,"5min":27.346,"15min":121.181},"last":2.1379999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.87,"15min":0.92500000000000004},"min":{"1min":0.39000000000000001,"5min":0.30399999999999999,"15min":0.30199999999999999},"max":{"1min":3.379,"5min":6.1669999999999998,"15min":6.2039999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.141,"5min":1.167,"15min":1.036},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.6509999999999998,"5min":4.6529999999999996,"15min":7.3920000000000003},"last":0.751}]},{"osd":5,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.0820000000000001,"15min":1.1279999999999999},"min":{"1min":0.42099999999999999,"5min":0.37,"15min":0.34100000000000003},"max":{"1min":3.6219999999999999,"5min":3.6219999999999999,"15min":5.968},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.1859999999999999,"15min":1.212},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.33100000000000002},"max":{"1min":1.7,"5min":4.7300000000000004,"15min":7.4249999999999998},"last":0.83399999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.97599999999999998,"15min":1.099},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.35299999999999998},"max":{"1min":3.4420000000000002,"5min":6.202,"15min":6.4160000000000004},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.151,"5min":1.056,"15min":1.258},"min":{"1min":0.49299999999999999,"5min":0.41699999999999998,"15min":0.38700000000000001},"max":{"1min":3.4790000000000001,"5min":6.1289999999999996,"15min":9.5389999999999997},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:38:45 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.159,"15min":1.145},"min":{"1min":0.497,"5min":0.43099999999999999,"15min":0.38700000000000001},"max":{"1min":4.968,"5min":5.5670000000000002,"15min":7.851},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.244,"5min":1.222,"15min":1.1220000000000001},"min":{"1min":0.50900000000000001,"5min":0.45200000000000001,"15min":0.36699999999999999},"max":{"1min":2.8140000000000001,"5min":7.3529999999999998,"15min":8.3000000000000007},"last":0.67800000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.079,"15min":0.98999999999999999},"min":{"1min":0.42099999999999999,"5min":0.35099999999999998,"15min":0.32300000000000001},"max":{"1min":3.5390000000000001,"5min":6.306,"15min":6.4459999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1850000000000001,"15min":1.099},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.885,"5min":2.9860000000000002,"15min":5.4880000000000004},"last":0.622}]},{"osd":9,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2330000000000001,"15min":1.2290000000000001},"min":{"1min":0.44800000000000001,"5min":0.437,"15min":0.38600000000000001},"max":{"1min":2.48,"5min":2.907,"15min":7.5709999999999997},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.2949999999999999,"15min":1.1899999999999999},"min":{"1min":0.52800000000000002,"5min":0.47399999999999998,"15min":0.44700000000000001},"max":{"1min":2.5129999999999999,"5min":2.9870000000000001,"15min":5.7089999999999996},"last":0.876}]},{"osd":10,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.399,"15min":1.2090000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":3.722,"5min":4.5990000000000002,"15min":7.4539999999999997},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3260000000000001,"15min":1.171},"min":{"1min":0.52900000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":4.4530000000000003,"5min":6.3330000000000002,"15min":6.4930000000000003},"last":0.86399999999999999}]}]},{"osd":10,"up_from":83,"seq":356482286122,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2735872,"kb_used_data":1849592,"kb_used_omap":0,"kb_used_meta":886272,"kb_avail":91009280,"statfs":{"total":95995035648,"available":93193502720,"internally_reserved":0,"allocated":1893982208,"data_stored":1893242052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":907542528},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.99399999999999999,"15min":1.196},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":2.9199999999999999,"5min":4.7140000000000004,"15min":111.935},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88400000000000001,"15min":1.274},"min":{"1min":0.29899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.641,"5min":4.3920000000000003,"15min":111.316},"last":1.353}]},{"osd":1,"last update":"Mon Feb 12 16:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.0069999999999999,"15min":1.119},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0590000000000002,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.032,"15min":1.115},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.534,"5min":7.2000000000000002,"15min":7.2000000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0269999999999999,"15min":1.3400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.28100000000000003},"max":{"1min":4.2380000000000004,"5min":9.4179999999999993,"15min":106.822},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.92700000000000005,"15min":1.3420000000000001},"min":{"1min":0.40400000000000003,"5min":0.34699999999999998,"15min":0.30199999999999999},"max":{"1min":1.5009999999999999,"5min":1.923,"15min":107.88800000000001},"last":0.438}]},{"osd":3,"last update":"Mon Feb 12 16:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.157,"15min":1.4870000000000001},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.27400000000000002},"max":{"1min":1.6830000000000001,"5min":9.3699999999999992,"15min":107.014},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.214,"15min":1.4279999999999999},"min":{"1min":0.39800000000000002,"5min":0.375,"15min":0.27900000000000003},"max":{"1min":1.506,"5min":4.9299999999999997,"15min":110.80800000000001},"last":0.95699999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.167,"15min":1.0389999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":6.3689999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.03,"15min":0.995},"min":{"1min":0.40100000000000002,"5min":0.38600000000000001,"15min":0.318},"max":{"1min":1.4219999999999999,"5min":2.0840000000000001,"15min":5.9119999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":4.8849999999999998,"5min":1.992,"15min":1.2929999999999999},"min":{"1min":0.42099999999999999,"5min":0.39500000000000002,"15min":0.32700000000000001},"max":{"1min":74.137,"5min":74.137,"15min":74.137},"last":1.0249999999999999},{"interface":"front","average":{"1min":4.8120000000000003,"5min":1.9039999999999999,"15min":1.4530000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.41299999999999998},"max":{"1min":74.063999999999993,"5min":74.063999999999993,"15min":74.063999999999993},"last":2.8500000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.327,"15min":2.581},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.36099999999999999},"max":{"1min":3.843,"5min":3.843,"15min":460.90199999999999},"last":1.056},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.234,"15min":2.589},"min":{"1min":0.47899999999999998,"5min":0.44800000000000001,"15min":0.44},"max":{"1min":3.1600000000000001,"5min":3.1600000000000001,"15min":460.93299999999999},"last":0.97799999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.93000000000000005,"15min":1.014},"min":{"1min":0.41599999999999998,"5min":0.40799999999999997,"15min":0.375},"max":{"1min":5.0419999999999998,"5min":5.0419999999999998,"15min":7.3280000000000003},"last":0.998},{"interface":"front","average":{"1min":1.212,"5min":1.1819999999999999,"15min":1.175},"min":{"1min":0.57599999999999996,"5min":0.434,"15min":0.374},"max":{"1min":4.532,"5min":5.306,"15min":6.4539999999999997},"last":0.92800000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.2,"15min":1.1619999999999999},"min":{"1min":0.436,"5min":0.42199999999999999,"15min":0.35499999999999998},"max":{"1min":3.0209999999999999,"5min":3.0209999999999999,"15min":8.4420000000000002},"last":1.01},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2869999999999999,"15min":1.165},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36299999999999999},"max":{"1min":4.444,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.98899999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.375,"15min":1.157},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.34699999999999998},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":6.6239999999999997},"last":1.077},{"interface":"front","average":{"1min":1.194,"5min":1.29,"15min":1.1479999999999999},"min":{"1min":0.58399999999999996,"5min":0.378,"15min":0.378},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":6.0030000000000001},"last":1.0469999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:43:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.488,"15min":1.2749999999999999},"min":{"1min":0.495,"5min":0.46300000000000002,"15min":0.40699999999999997},"max":{"1min":2.1040000000000001,"5min":8.7569999999999997,"15min":8.7569999999999997},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.476,"5min":1.466,"15min":1.2869999999999999},"min":{"1min":0.60299999999999998,"5min":0.504,"15min":0.38100000000000001},"max":{"1min":2.1579999999999999,"5min":5.46,"15min":6.5860000000000003},"last":1.034}]}]},{"osd":9,"up_from":0,"seq":317827580467,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":8,"up_from":65,"seq":279172874812,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503168,"kb_used_data":917560,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92241984,"statfs":{"total":95995035648,"available":94455791616,"internally_reserved":0,"allocated":939581440,"data_stored":938841284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.91300000000000003,"15min":1.2070000000000001},"min":{"1min":0.34100000000000003,"5min":0.27500000000000002,"15min":0.23899999999999999},"max":{"1min":1.222,"5min":5.96,"15min":134.01499999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":1.0069999999999999,"15min":1.423},"min":{"1min":0.33000000000000002,"5min":0.308,"15min":0.28799999999999998},"max":{"1min":1.087,"5min":6.7729999999999997,"15min":133.667},"last":0.315}]},{"osd":1,"last update":"Mon Feb 12 16:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.8319999999999999,"5min":1.671,"15min":1.571},"min":{"1min":0.44,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":33.415999999999997,"5min":33.415999999999997,"15min":33.415999999999997},"last":4.173},{"interface":"front","average":{"1min":2.7719999999999998,"5min":1.343,"15min":1.091},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":33.337000000000003,"5min":33.337000000000003,"15min":33.337000000000003},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":1.9399999999999999,"5min":3.2320000000000002,"15min":8.6430000000000007},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2609999999999999,"15min":1.091},"min":{"1min":0.48999999999999999,"5min":0.378,"15min":0.26800000000000002},"max":{"1min":2.0070000000000001,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.84099999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.1699999999999999,"15min":1.115},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.5549999999999999,"5min":13.784000000000001,"15min":25.919},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2110000000000001,"15min":1.2330000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.8580000000000001,"5min":13.773999999999999,"15min":25.984999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:42:48 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.028,"15min":0.89000000000000001},"min":{"1min":0.41299999999999998,"5min":0.35699999999999998,"15min":0.34000000000000002},"max":{"1min":2.931,"5min":3.117,"15min":8.609},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.2230000000000001,"15min":1.107},"min":{"1min":0.52300000000000002,"5min":0.38200000000000001,"15min":0.37},"max":{"1min":2.4780000000000002,"5min":4.4500000000000002,"15min":9.0890000000000004},"last":0.44400000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.044,"15min":1.0449999999999999},"min":{"1min":0.46800000000000003,"5min":0.36899999999999999,"15min":0.318},"max":{"1min":1.8169999999999999,"5min":4.8710000000000004,"15min":7.2960000000000003},"last":2.359},{"interface":"front","average":{"1min":1.111,"5min":1.212,"15min":1.226},"min":{"1min":0.55700000000000005,"5min":0.379,"15min":0.35199999999999998},"max":{"1min":2.524,"5min":4.3499999999999996,"15min":9.0540000000000003},"last":0.86099999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:42:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.119,"15min":1.0700000000000001},"min":{"1min":0.56899999999999995,"5min":0.434,"15min":0.373},"max":{"1min":2.9809999999999999,"5min":4.8330000000000002,"15min":8.5220000000000002},"last":2.2069999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.2529999999999999,"15min":1.2989999999999999},"min":{"1min":0.58799999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.6139999999999999,"5min":4.5140000000000002,"15min":9.1500000000000004},"last":2.302}]},{"osd":7,"last update":"Mon Feb 12 16:38:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.085,"15min":3.3860000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.9710000000000001,"5min":8.5709999999999997,"15min":706.67700000000002},"last":4.218},{"interface":"front","average":{"1min":1.373,"5min":1.4870000000000001,"15min":3.5720000000000001},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":5.1079999999999997,"5min":9.1799999999999997,"15min":707.49099999999999},"last":0.41599999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.214,"15min":1.206},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.34200000000000003},"max":{"1min":2.6440000000000001,"5min":4.407,"15min":9.1170000000000009},"last":0.751},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.357,"15min":1.3200000000000001},"min":{"1min":0.57399999999999995,"5min":0.49199999999999999,"15min":0.46100000000000002},"max":{"1min":2.6960000000000002,"5min":4.5380000000000003,"15min":9.1980000000000004},"last":0.78900000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.167,"15min":1.048},"min":{"1min":0.61099999999999999,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":1.554,"5min":5.0709999999999997,"15min":8.452},"last":4.1269999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.3049999999999999,"15min":1.137},"min":{"1min":0.48399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.0190000000000001,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":0.80100000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:43:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.268,"15min":1.1399999999999999},"min":{"1min":0.53500000000000003,"5min":0.437,"15min":0.42699999999999999},"max":{"1min":3.024,"5min":4.7750000000000004,"15min":8.6790000000000003},"last":4.2750000000000004},{"interface":"front","average":{"1min":1.496,"5min":1.371,"15min":1.1850000000000001},"min":{"1min":0.46500000000000002,"5min":0.45800000000000002,"15min":0.44600000000000001},"max":{"1min":3.0920000000000001,"5min":4.7549999999999999,"15min":5.2320000000000002},"last":0.83199999999999996}]}]},{"osd":7,"up_from":923,"seq":3964254814220,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1715200,"kb_used_data":1396728,"kb_used_omap":0,"kb_used_meta":318463,"kb_avail":92029952,"statfs":{"total":95995035648,"available":94238670848,"internally_reserved":0,"allocated":1430249472,"data_stored":1429509316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":326106164},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]},{"osd":6,"up_from":48,"seq":206158430796,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1166592,"kb_used_data":580984,"kb_used_omap":0,"kb_used_meta":585600,"kb_avail":92578560,"statfs":{"total":95995035648,"available":94800445440,"internally_reserved":0,"allocated":594927616,"data_stored":594187460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":599654400},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.82399999999999995,"15min":0.81000000000000005},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.419,"5min":5.7430000000000003,"15min":5.7430000000000003},"last":0.87},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.751,"15min":0.77500000000000002},"min":{"1min":0.39900000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.421,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.98399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:42:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1000000000000001,"15min":1.054},"min":{"1min":0.39600000000000002,"5min":0.31900000000000001,"15min":0.28000000000000003},"max":{"1min":2.8119999999999998,"5min":5.8140000000000001,"15min":6.1870000000000003},"last":2.7440000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0700000000000001,"15min":1.159},"min":{"1min":0.41699999999999998,"5min":0.30399999999999999,"15min":0.25700000000000001},"max":{"1min":1.429,"5min":5.375,"15min":6.6559999999999997},"last":0.35799999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:42:40 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":0.96199999999999997,"15min":1.286},"min":{"1min":0.57299999999999995,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":1.464,"5min":5.3220000000000001,"15min":14.68},"last":6.3380000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89800000000000002,"15min":1.1930000000000001},"min":{"1min":0.435,"5min":0.31,"15min":0.27700000000000002},"max":{"1min":1.369,"5min":4.8410000000000002,"15min":15.138},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.82399999999999995,"15min":3.0329999999999999},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.27800000000000002},"max":{"1min":1.4830000000000001,"5min":2.4590000000000001,"15min":365.58999999999997},"last":2.8490000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.0660000000000001,"15min":3.5459999999999998},"min":{"1min":0.39600000000000002,"5min":0.374,"15min":0.317},"max":{"1min":1.6080000000000001,"5min":4.1699999999999999,"15min":366.93299999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:42:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.93000000000000005,"15min":1.075},"min":{"1min":0.53200000000000003,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":1.6719999999999999,"5min":4.9219999999999997,"15min":8.4559999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.97799999999999998,"15min":1.0820000000000001},"min":{"1min":0.46300000000000002,"5min":0.33400000000000002,"15min":0.29999999999999999},"max":{"1min":1.631,"5min":5.2069999999999999,"15min":7.7610000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Feb 12 16:42:40 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.264,"15min":1.2749999999999999},"min":{"1min":0.51900000000000002,"5min":0.41899999999999998,"15min":0.26200000000000001},"max":{"1min":1.702,"5min":24.870000000000001,"15min":24.870000000000001},"last":5.3730000000000002},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.325,"15min":1.242},"min":{"1min":0.48699999999999999,"5min":0.40500000000000003,"15min":0.33800000000000002},"max":{"1min":1.7669999999999999,"5min":25.98,"15min":25.98},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:39:06 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.29,"15min":1.256},"min":{"1min":0.54800000000000004,"5min":0.38100000000000001,"15min":0.29499999999999998},"max":{"1min":4.4429999999999996,"5min":6.359,"15min":9.1199999999999992},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.0600000000000001,"15min":1.173},"min":{"1min":0.505,"5min":0.32800000000000001,"15min":0.32200000000000001},"max":{"1min":4.476,"5min":6.9279999999999999,"15min":8.8879999999999999},"last":0.84999999999999998}]},{"osd":8,"last update":"Mon Feb 12 16:42:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.222,"15min":2.895},"min":{"1min":0.73599999999999999,"5min":0.45600000000000002,"15min":0.35699999999999998},"max":{"1min":3,"5min":5.4489999999999998,"15min":501.892},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.2110000000000001,"15min":2.8359999999999999},"min":{"1min":0.76700000000000002,"5min":0.40500000000000003,"15min":0.38700000000000001},"max":{"1min":2.956,"5min":5.3150000000000004,"15min":501.76600000000002},"last":0.53200000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.224,"15min":1.238},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.35299999999999998},"max":{"1min":2.2109999999999999,"5min":3.3279999999999998,"15min":6.319},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.179,"5min":1.1060000000000001,"15min":1.2470000000000001},"min":{"1min":0.60499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.1819999999999999,"5min":2.6280000000000001,"15min":7.6870000000000003},"last":0.624}]},{"osd":10,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.2130000000000001,"15min":1.3600000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.42499999999999999},"max":{"1min":2.0920000000000001,"5min":2.6850000000000001,"15min":11.536},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.2889999999999999,"15min":1.4350000000000001},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48099999999999998},"max":{"1min":2.1659999999999999,"5min":2.8050000000000002,"15min":11.585000000000001},"last":0.73799999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.2889999999999999,"15min":1.466},"min":{"1min":0.61899999999999999,"5min":0.54800000000000004,"15min":0.49299999999999999},"max":{"1min":1.8819999999999999,"5min":5.3799999999999999,"15min":8.6649999999999991},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.238,"15min":1.214},"min":{"1min":0.55800000000000005,"5min":0.45600000000000002,"15min":0.40699999999999997},"max":{"1min":2.5649999999999999,"5min":5.6130000000000004,"15min":7.04},"last":0.81299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834575474,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4213260,"kb_used_data":3319044,"kb_used_omap":0,"kb_used_meta":894208,"kb_avail":89531892,"statfs":{"total":95995035648,"available":91680657408,"internally_reserved":0,"allocated":3398701056,"data_stored":3397954890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915668992},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.98799999999999999},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":1.3560000000000001,"5min":7.532,"15min":10.420999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.89100000000000001,"15min":1.0589999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.3939999999999999,"5min":7.3540000000000001,"15min":10.26},"last":0.42099999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:42:52 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.96899999999999997,"15min":1.175},"min":{"1min":0.34799999999999998,"5min":0.30299999999999999,"15min":0.22700000000000001},"max":{"1min":1.5049999999999999,"5min":4.0119999999999996,"15min":12.778},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.96099999999999997,"15min":1.1759999999999999},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":1.2769999999999999,"5min":7.7190000000000003,"15min":9.702},"last":0.434}]},{"osd":3,"last update":"Mon Feb 12 16:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":1.161},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":4.1180000000000003,"5min":4.1180000000000003,"15min":26.664000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.97699999999999998,"15min":1.367},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.26200000000000001},"max":{"1min":1.55,"5min":3.629,"15min":27.148},"last":0.73399999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:42:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.94999999999999996,"15min":1.0469999999999999},"min":{"1min":0.70999999999999996,"5min":0.312,"15min":0.312},"max":{"1min":1.978,"5min":3.6549999999999998,"15min":7.7549999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.0629999999999999,"15min":1.1080000000000001},"min":{"1min":0.66700000000000004,"5min":0.377,"15min":0.309},"max":{"1min":2.1019999999999999,"5min":4.9379999999999997,"15min":6.9119999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:42:52 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.038,"15min":4.0110000000000001},"min":{"1min":0.443,"5min":0.34300000000000003,"15min":0.309},"max":{"1min":2.1909999999999998,"5min":3.2519999999999998,"15min":524.85299999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2050000000000001,"15min":4.1200000000000001},"min":{"1min":0.495,"5min":0.34799999999999998,"15min":0.32800000000000001},"max":{"1min":2.274,"5min":4.3090000000000002,"15min":519.89700000000005},"last":0.88300000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:43:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.1160000000000001,"15min":1.274},"min":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.331,"5min":3.7829999999999999,"15min":3.7829999999999999},"last":0.746},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3620000000000001,"15min":1.468},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.5099999999999998,"5min":4.5099999999999998,"15min":4.5099999999999998},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Feb 12 16:38:59 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0629999999999999,"15min":3.2370000000000001},"min":{"1min":0.44700000000000001,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":522.79200000000003},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.25,"15min":3.198},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":4.3360000000000003,"5min":9.0069999999999997,"15min":520.64499999999998},"last":1.5800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:43:04 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.444,"15min":1.319},"min":{"1min":0.503,"5min":0.48499999999999999,"15min":0.379},"max":{"1min":4.8879999999999999,"5min":4.8879999999999999,"15min":9.1430000000000007},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.7569999999999999,"5min":1.5489999999999999,"15min":1.345},"min":{"1min":0.59699999999999998,"5min":0.54300000000000004,"15min":0.38400000000000001},"max":{"1min":4.9409999999999998,"5min":4.9409999999999998,"15min":9.1739999999999995},"last":0.85099999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.224,"15min":1.194},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":2.0329999999999999,"5min":3.1869999999999998,"15min":8.3949999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.234,"5min":1.3360000000000001,"15min":1.302},"min":{"1min":0.48099999999999998,"5min":0.47299999999999998,"15min":0.436},"max":{"1min":2.266,"5min":3.2229999999999999,"15min":8.4969999999999999},"last":0.61799999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:43:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4279999999999999,"15min":1.3129999999999999},"min":{"1min":0.64400000000000002,"5min":0.434,"15min":0.42999999999999999},"max":{"1min":2.4329999999999998,"5min":3.746,"15min":8.3469999999999995},"last":0.87},{"interface":"front","average":{"1min":1.534,"5min":1.5669999999999999,"15min":1.391},"min":{"1min":0.65000000000000002,"5min":0.51200000000000001,"15min":0.46999999999999997},"max":{"1min":2.6600000000000001,"5min":4.3659999999999997,"15min":9.0440000000000005},"last":0.98099999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:42:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.5409999999999999,"15min":1.7070000000000001},"min":{"1min":0.59599999999999997,"5min":0.52400000000000002,"15min":0.47099999999999997},"max":{"1min":2.4860000000000002,"5min":6.8639999999999999,"15min":88.679000000000002},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.331,"5min":1.3360000000000001,"15min":1.617},"min":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.46100000000000002},"max":{"1min":2.5099999999999998,"5min":3.6699999999999999,"15min":88.519000000000005},"last":0.75800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739006,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888384,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":886912,"kb_avail":92856768,"statfs":{"total":95995035648,"available":95085330432,"internally_reserved":0,"allocated":1499136,"data_stored":758980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":908197888},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":2.1030000000000002,"15min":1.3720000000000001},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":1.7589999999999999,"5min":118.657,"15min":118.657},"last":0.623},{"interface":"front","average":{"1min":1.3149999999999999,"5min":2.4300000000000002,"15min":1.591},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.22600000000000001},"max":{"1min":7.6429999999999998,"5min":118.58,"15min":118.58},"last":0.99399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:43:02 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.89500000000000002,"15min":0.93899999999999995},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2070000000000001,"5min":3.2909999999999999,"15min":3.2909999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.878,"15min":0.91000000000000003},"min":{"1min":0.40999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0720000000000001,"5min":2.8660000000000001,"15min":2.8660000000000001},"last":1.377}]},{"osd":3,"last update":"Mon Feb 12 16:43:14 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.98899999999999999,"15min":2.3660000000000001},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.252},"max":{"1min":2.0569999999999999,"5min":3.681,"15min":377.06299999999999},"last":1.083},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.97999999999999998,"15min":2.512},"min":{"1min":0.39300000000000002,"5min":0.26600000000000001,"15min":0.224},"max":{"1min":1.262,"5min":4.6779999999999999,"15min":379.18000000000001},"last":1.5449999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.94699999999999995,"15min":1.0189999999999999},"min":{"1min":0.35699999999999998,"5min":0.32700000000000001,"15min":0.29099999999999998},"max":{"1min":1.5109999999999999,"5min":1.748,"15min":7.2869999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.014,"15min":1.1859999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.29499999999999998},"max":{"1min":1.431,"5min":4.0220000000000002,"15min":8.4949999999999992},"last":1.784}]},{"osd":5,"last update":"Mon Feb 12 16:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96699999999999997,"15min":1.7829999999999999},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.29799999999999999},"max":{"1min":1.3180000000000001,"5min":6.6420000000000003,"15min":181.46199999999999},"last":8.532},{"interface":"front","average":{"1min":1.117,"5min":1.1180000000000001,"15min":1.891},"min":{"1min":0.46800000000000003,"5min":0.41299999999999998,"15min":0.32800000000000001},"max":{"1min":1.4650000000000001,"5min":6.8170000000000002,"15min":186.41999999999999},"last":8.6519999999999992}]},{"osd":6,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.0169999999999999,"15min":1.0669999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3280000000000001,"5min":6.6950000000000003,"15min":6.6950000000000003},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.109,"15min":1.2789999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":1.252}]},{"osd":7,"last update":"Mon Feb 12 16:39:13 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.24,"15min":2.706},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.374},"max":{"1min":1.835,"5min":6.1130000000000004,"15min":191.47200000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4019999999999999,"15min":2.823},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":1.897,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.081}]},{"osd":8,"last update":"Mon Feb 12 16:42:51 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.1599999999999999,"15min":1.2889999999999999},"min":{"1min":0.66900000000000004,"5min":0.45000000000000001,"15min":0.41299999999999998},"max":{"1min":1.5780000000000001,"5min":6.7400000000000002,"15min":7.3920000000000003},"last":1.155},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2829999999999999,"15min":1.3919999999999999},"min":{"1min":0.75600000000000001,"5min":0.55000000000000004,"15min":0.48299999999999998},"max":{"1min":1.589,"5min":6.8570000000000002,"15min":7.5060000000000002},"last":1.4350000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:43:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.155,"15min":1.242},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.34999999999999998},"max":{"1min":1.772,"5min":2.9750000000000001,"15min":8.4629999999999992},"last":1.879},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.2,"15min":1.3380000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41499999999999998},"max":{"1min":1.504,"5min":6.7770000000000001,"15min":12.943},"last":1.3180000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:43:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.3260000000000001,"15min":1.254},"min":{"1min":0.496,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":1.9059999999999999,"5min":13.157,"15min":13.157},"last":1.667},{"interface":"front","average":{"1min":1.115,"5min":1.4079999999999999,"15min":1.3600000000000001},"min":{"1min":0.56699999999999995,"5min":0.495,"15min":0.48299999999999998},"max":{"1min":1.9850000000000001,"5min":13.238,"15min":13.238},"last":1.7549999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:42:51 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.345,"15min":1.429},"min":{"1min":0.66600000000000004,"5min":0.50600000000000001,"15min":0.40300000000000002},"max":{"1min":2.0800000000000001,"5min":4.3410000000000002,"15min":8.4280000000000008},"last":1.948},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.284,"15min":1.341},"min":{"1min":0.71799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"max":{"1min":1.9419999999999999,"5min":4.9960000000000004,"15min":7.4210000000000003},"last":1.476}]}]},{"osd":2,"up_from":21,"seq":90194313836,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595288,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93149864,"statfs":{"total":95995035648,"available":95385460736,"internally_reserved":0,"allocated":1523712,"data_stored":771486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.73999999999999999,"15min":0.79900000000000004},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.876,"5min":3.496,"15min":3.496},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.73399999999999999,"15min":0.81799999999999995},"min":{"1min":0.26700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.762,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":0.90300000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.88100000000000001,"15min":0.91400000000000003},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.19500000000000001},"max":{"1min":1.1579999999999999,"5min":5.1379999999999999,"15min":7.6050000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.89700000000000002,"15min":1.125},"min":{"1min":0.32300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.2809999999999999,"5min":2.9220000000000002,"15min":27.562999999999999},"last":0.89500000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.95199999999999996,"15min":1.419},"min":{"1min":0.38300000000000001,"5min":0.28999999999999998,"15min":0.25700000000000001},"max":{"1min":1.0960000000000001,"5min":6.5090000000000003,"15min":54.091000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.042,"15min":1.3959999999999999},"min":{"1min":0.40000000000000002,"5min":0.29099999999999998,"15min":0.25800000000000001},"max":{"1min":1.462,"5min":6.5419999999999998,"15min":54.203000000000003},"last":0.79000000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.94199999999999995,"15min":0.998},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.29499999999999998},"max":{"1min":1.3360000000000001,"5min":3.0470000000000002,"15min":8.3439999999999994},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.94399999999999995,"15min":1.0740000000000001},"min":{"1min":0.34999999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":1.6950000000000001,"5min":5.1710000000000003,"15min":7.0019999999999998},"last":0.77600000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.90200000000000002,"15min":0.97299999999999998},"min":{"1min":0.38300000000000001,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":1.403,"5min":5.1319999999999997,"15min":8.7509999999999994},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.1060000000000001,"15min":1.1599999999999999},"min":{"1min":0.44800000000000001,"5min":0.35299999999999998,"15min":0.31},"max":{"1min":1.607,"5min":4.3819999999999997,"15min":7.0590000000000002},"last":0.64700000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0269999999999999,"15min":1.1519999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.316},"max":{"1min":1.758,"5min":3.0209999999999999,"15min":8.4510000000000005},"last":0.755},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.081,"15min":1.137},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":1.242,"5min":3.7879999999999998,"15min":8.8179999999999996},"last":0.60899999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.58399999999999996,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":1.9019999999999999,"5min":6.2560000000000002,"15min":11.044},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2529999999999999,"15min":1.248},"min":{"1min":0.44600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.4219999999999997,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":1.6499999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.764,"15min":1.385},"min":{"1min":0.58699999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.073,"5min":59.164999999999999,"15min":59.164999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.744,"15min":1.365},"min":{"1min":0.63500000000000001,"5min":0.48699999999999999,"15min":0.377},"max":{"1min":1.9630000000000001,"5min":58.993000000000002,"15min":58.993000000000002},"last":0.68799999999999994}]},{"osd":9,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1599999999999999,"15min":1.2529999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.38600000000000001},"max":{"1min":1.6319999999999999,"5min":5.2210000000000001,"15min":7.7759999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2609999999999999,"15min":1.2569999999999999},"min":{"1min":0.434,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.3609999999999998,"5min":5.2549999999999999,"15min":7.2110000000000003},"last":0.997}]},{"osd":10,"last update":"Mon Feb 12 16:43:02 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":5.2770000000000001,"15min":2.5899999999999999},"min":{"1min":0.66900000000000004,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":2.7869999999999999,"5min":407.577,"15min":407.577},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.1160000000000001,"5min":4.9930000000000003,"15min":2.5310000000000001},"min":{"1min":0.60199999999999998,"5min":0.47799999999999998,"15min":0.41799999999999998},"max":{"1min":2.105,"5min":407.45100000000002,"15min":407.45100000000002},"last":0.82099999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.2869999999999999,"15min":1.212},"min":{"1min":0.63100000000000001,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":1.825,"5min":3.8540000000000001,"15min":8.9359999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.4339999999999999,"15min":1.3360000000000001},"min":{"1min":0.61699999999999999,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":1.6080000000000001,"5min":4.8849999999999998,"15min":8.968},"last":0.67100000000000004}]}]},{"osd":3,"up_from":522,"seq":2241972928760,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2043032,"kb_used_data":1418320,"kb_used_omap":91,"kb_used_meta":624612,"kb_avail":91702120,"statfs":{"total":95995035648,"available":93902970880,"internally_reserved":0,"allocated":1452359680,"data_stored":1451607454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":639602704},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.748,"15min":1.032},"min":{"1min":0.26600000000000001,"5min":0.26200000000000001,"15min":0.20699999999999999},"max":{"1min":1.7350000000000001,"5min":4.0469999999999997,"15min":10.101000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.82999999999999996,"15min":1.0580000000000001},"min":{"1min":0.27800000000000002,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":1.794,"5min":4.9969999999999999,"15min":10.436999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":14.590999999999999,"5min":3.8420000000000001,"15min":1.9219999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":245.10900000000001,"5min":245.10900000000001,"15min":245.10900000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":14.506,"5min":3.8919999999999999,"15min":2.0640000000000001},"min":{"1min":0.38,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":245.21199999999999,"5min":245.21199999999999,"15min":245.21199999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.893,"15min":1.357},"min":{"1min":0.40300000000000002,"5min":0.253,"15min":0.252},"max":{"1min":1.6080000000000001,"5min":81.769999999999996,"15min":81.769999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.109,"5min":2.1120000000000001,"15min":1.4930000000000001},"min":{"1min":0.53200000000000003,"5min":0.32300000000000001,"15min":0.23699999999999999},"max":{"1min":1.6890000000000001,"5min":81.876999999999995,"15min":81.876999999999995},"last":0.65400000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.2290000000000001,"15min":1.1519999999999999},"min":{"1min":0.44,"5min":0.33000000000000002,"15min":0.27400000000000002},"max":{"1min":1.585,"5min":6.8220000000000001,"15min":6.8220000000000001},"last":0.621},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.0820000000000001,"15min":1.115},"min":{"1min":0.378,"5min":0.33500000000000002,"15min":0.29599999999999999},"max":{"1min":4.6159999999999997,"5min":6.6840000000000002,"15min":6.806},"last":0.54700000000000004}]},{"osd":5,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1830000000000001,"15min":1.1930000000000001},"min":{"1min":0.38900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.2549999999999999,"5min":8.9489999999999998,"15min":11.884},"last":0.441},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2130000000000001,"15min":1.2909999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.29899999999999999},"max":{"1min":1.6890000000000001,"5min":5.9279999999999999,"15min":12.164999999999999},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.2470000000000001,"15min":1.2390000000000001},"min":{"1min":0.45400000000000001,"5min":0.32300000000000001,"15min":0.31900000000000001},"max":{"1min":3.6080000000000001,"5min":9,"15min":9.4689999999999994},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3140000000000001},"min":{"1min":0.68999999999999995,"5min":0.41599999999999998,"15min":0.30499999999999999},"max":{"1min":1.8,"5min":6.9130000000000003,"15min":9.4640000000000004},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.5149999999999999,"15min":1.298},"min":{"1min":0.55000000000000004,"5min":0.36199999999999999,"15min":0.33300000000000002},"max":{"1min":7.9909999999999997,"5min":7.9909999999999997,"15min":7.9909999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.609,"5min":1.3859999999999999,"15min":1.335},"min":{"1min":0.621,"5min":0.371,"15min":0.371},"max":{"1min":6.0899999999999999,"5min":6.7590000000000003,"15min":7.2249999999999996},"last":0.59299999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.379,"15min":1.278},"min":{"1min":0.5,"5min":0.439,"15min":0.38200000000000001},"max":{"1min":4.1390000000000002,"5min":5.8079999999999998,"15min":5.9669999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.385,"5min":1.484,"15min":1.3360000000000001},"min":{"1min":0.48999999999999999,"5min":0.47899999999999998,"15min":0.35599999999999998},"max":{"1min":1.9079999999999999,"5min":6.056,"15min":7.9569999999999999},"last":0.82699999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.454,"15min":1.3560000000000001},"min":{"1min":0.54800000000000004,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":4.25,"5min":8.9350000000000005,"15min":8.9350000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.4359999999999999,"15min":1.3260000000000001},"min":{"1min":0.54600000000000004,"5min":0.40899999999999997,"15min":0.38300000000000001},"max":{"1min":1.819,"5min":8.8490000000000002,"15min":9.5239999999999991},"last":0.76000000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":9.1989999999999998,"15min":3.899},"min":{"1min":0.69599999999999995,"5min":0.45500000000000002,"15min":0.40799999999999997},"max":{"1min":4.3890000000000002,"5min":738.99900000000002,"15min":738.99900000000002},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.5800000000000001,"5min":9.2609999999999992,"15min":4.0410000000000004},"min":{"1min":0.626,"5min":0.55800000000000005,"15min":0.45600000000000002},"max":{"1min":3.7480000000000002,"5min":746.45299999999997,"15min":746.45299999999997},"last":0.69999999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:42:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.4790000000000001,"15min":1.607},"min":{"1min":0.57899999999999996,"5min":0.437,"15min":0.437},"max":{"1min":3.1640000000000001,"5min":8.9030000000000005,"15min":73.037999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.53,"5min":1.51,"15min":1.639},"min":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.53200000000000003},"max":{"1min":4.4989999999999997,"5min":5.5330000000000004,"15min":73.057000000000002},"last":0.74399999999999999}]}]},{"osd":4,"up_from":32,"seq":137438954075,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1825280,"kb_used_data":935224,"kb_used_omap":0,"kb_used_meta":890048,"kb_avail":91919872,"statfs":{"total":95995035648,"available":94125948928,"internally_reserved":0,"allocated":957669376,"data_stored":956929220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911409152},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81999999999999995,"15min":1.02},"min":{"1min":0.372,"5min":0.27100000000000002,"15min":0.27000000000000002},"max":{"1min":1.1080000000000001,"5min":3.9670000000000001,"15min":4.1050000000000004},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.92100000000000004,"15min":1.1220000000000001},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.4470000000000001,"5min":5.0330000000000004,"15min":5.0599999999999996},"last":1.1850000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.232,"15min":1.3180000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":3.1520000000000001,"5min":30.137,"15min":56.188000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.353,"15min":1.446},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.25700000000000001},"max":{"1min":2.1429999999999998,"5min":32.481999999999999,"15min":55.414000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":3.2050000000000001,"15min":1.8740000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3100000000000001,"5min":212.685,"15min":212.685},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":3.1019999999999999,"15min":1.786},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.9390000000000001,"5min":210.95699999999999,"15min":210.95699999999999},"last":1.3540000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:42:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.488,"15min":1.323},"min":{"1min":0.56499999999999995,"5min":0.32300000000000001,"15min":0.311},"max":{"1min":1.8580000000000001,"5min":30.282,"15min":30.282},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.4730000000000001,"15min":1.292},"min":{"1min":0.42299999999999999,"5min":0.308,"15min":0.25800000000000001},"max":{"1min":2.0419999999999998,"5min":30.478000000000002,"15min":30.478000000000002},"last":1.2929999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.98399999999999999,"15min":1.1080000000000001},"min":{"1min":0.32900000000000001,"5min":0.27200000000000002,"15min":0.252},"max":{"1min":1.2430000000000001,"5min":7.8310000000000004,"15min":10.289999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.008,"5min":1.1479999999999999,"15min":1.242},"min":{"1min":0.48499999999999999,"5min":0.34599999999999997,"15min":0.33000000000000002},"max":{"1min":1.468,"5min":7.8620000000000001,"15min":10.321999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:42:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.97199999999999998,"15min":1.0640000000000001},"min":{"1min":0.51100000000000001,"5min":0.34599999999999997,"15min":0.28000000000000003},"max":{"1min":1.196,"5min":5.1059999999999999,"15min":7.5289999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.24,"15min":1.1799999999999999},"min":{"1min":0.55900000000000005,"5min":0.35099999999999998,"15min":0.28899999999999998},"max":{"1min":1.778,"5min":11.968999999999999,"15min":11.968999999999999},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:38:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.98099999999999998,"15min":1.075},"min":{"1min":0.35699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.6580000000000004,"5min":5.4160000000000004,"15min":6.8659999999999997},"last":3.4489999999999998},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.0249999999999999,"15min":1.248},"min":{"1min":0.377,"5min":0.377,"15min":0.34599999999999997},"max":{"1min":8.8000000000000007,"5min":8.8000000000000007,"15min":9.1440000000000001},"last":0.58799999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:42:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.22,"15min":1.204},"min":{"1min":0.58899999999999997,"5min":0.40300000000000002,"15min":0.39300000000000002},"max":{"1min":1.6759999999999999,"5min":5.1950000000000003,"15min":7.6890000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1890000000000001,"15min":1.1819999999999999},"min":{"1min":0.57399999999999995,"5min":0.41399999999999998,"15min":0.39100000000000001},"max":{"1min":1.417,"5min":5.4859999999999998,"15min":7.9349999999999996},"last":0.52900000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.2490000000000001,"15min":1.4710000000000001},"min":{"1min":0.60699999999999998,"5min":0.46899999999999997,"15min":0.39300000000000002},"max":{"1min":2.0169999999999999,"5min":2.2879999999999998,"15min":63.545000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.327,"15min":1.468},"min":{"1min":0.65100000000000002,"5min":0.56699999999999995,"15min":0.36699999999999999},"max":{"1min":2.0979999999999999,"5min":2.395,"15min":63.421999999999997},"last":0.68899999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.23,"15min":1.266},"min":{"1min":0.50700000000000001,"5min":0.49099999999999999,"15min":0.433},"max":{"1min":3.9569999999999999,"5min":3.9569999999999999,"15min":7.7619999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1499999999999999,"15min":1.125},"min":{"1min":0.59299999999999997,"5min":0.47599999999999998,"15min":0.41799999999999998},"max":{"1min":3.048,"5min":3.048,"15min":5.6459999999999999},"last":0.50800000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:42:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.294,"15min":1.298},"min":{"1min":0.58499999999999996,"5min":0.50900000000000001,"15min":0.45400000000000001},"max":{"1min":1.76,"5min":5.5739999999999998,"15min":7.734},"last":0.63},{"interface":"front","average":{"1min":1.462,"5min":1.425,"15min":1.3100000000000001},"min":{"1min":0.80500000000000005,"5min":0.51000000000000001,"15min":0.46400000000000002},"max":{"1min":1.794,"5min":5.2539999999999996,"15min":7.7969999999999997},"last":0.63800000000000001}]}]},{"osd":5,"up_from":39,"seq":167503725139,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2922712,"kb_used_data":2025552,"kb_used_omap":0,"kb_used_meta":897152,"kb_avail":90822440,"statfs":{"total":95995035648,"available":93002178560,"internally_reserved":0,"allocated":2074165248,"data_stored":2073413022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":918683648},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.86899999999999999,"15min":1.462},"min":{"1min":0.35499999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1839999999999999,"5min":5.2240000000000002,"15min":38.366},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.76700000000000002,"15min":1.5049999999999999},"min":{"1min":0.28899999999999998,"5min":0.255,"15min":0.255},"max":{"1min":1.663,"5min":2.1920000000000002,"15min":37.310000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.94999999999999996,"15min":1.7949999999999999},"min":{"1min":0.504,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.3959999999999999,"5min":4.2599999999999998,"15min":221.96600000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.038,"5min":1.0840000000000001,"15min":1.919},"min":{"1min":0.55300000000000005,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.488,"5min":4.2910000000000004,"15min":222.06200000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.0720000000000001,"15min":1.161},"min":{"1min":0.441,"5min":0.35999999999999999,"15min":0.28799999999999998},"max":{"1min":1.3580000000000001,"5min":4.5990000000000002,"15min":6.6459999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.92700000000000005,"15min":1.1910000000000001},"min":{"1min":0.34699999999999998,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":1.3919999999999999,"5min":7.3470000000000004,"15min":11.448},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0620000000000001,"15min":1.3879999999999999},"min":{"1min":0.51600000000000001,"5min":0.28299999999999997,"15min":0.27200000000000002},"max":{"1min":1.3700000000000001,"5min":4.3949999999999996,"15min":22.484000000000002},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.0249999999999999,"15min":1.3360000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.29699999999999999},"max":{"1min":1.427,"5min":5.1719999999999997,"15min":23.34},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.89800000000000002,"15min":1.0409999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.23200000000000001},"max":{"1min":1.2190000000000001,"5min":1.706,"15min":6.5999999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96999999999999997,"15min":1.1299999999999999},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":1.831,"5min":3.117,"15min":5.5389999999999997},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:42:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.006,"15min":1.415},"min":{"1min":0.626,"5min":0.30499999999999999,"15min":0.28699999999999998},"max":{"1min":1.641,"5min":4.8490000000000002,"15min":68.263999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.085,"5min":1.081,"15min":1.514},"min":{"1min":0.45600000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6140000000000001,"5min":4.8760000000000003,"15min":67.873000000000005},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:38:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.86799999999999999,"15min":1.1830000000000001},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.28799999999999998},"max":{"1min":1.9570000000000001,"5min":5.6609999999999996,"15min":9.7710000000000008},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.024,"15min":1.286},"min":{"1min":0.376,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.9670000000000001,"5min":5.6120000000000001,"15min":6.7599999999999998},"last":0.627}]},{"osd":8,"last update":"Mon Feb 12 16:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1419999999999999,"15min":1.202},"min":{"1min":0.5,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.0019999999999998,"5min":4.0019999999999998,"15min":4.0019999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":1,"5min":1.107,"15min":1.3009999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"last":0.59099999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:42:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.224,"15min":1.2250000000000001},"min":{"1min":0.64800000000000002,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":1.863,"5min":3.5049999999999999,"15min":6.9509999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.1160000000000001,"15min":1.2769999999999999},"min":{"1min":0.67700000000000005,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":1.7749999999999999,"5min":2.9470000000000001,"15min":7.9710000000000001},"last":0.61699999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:42:57 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2130000000000001,"15min":1.5109999999999999},"min":{"1min":0.64700000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":2.0670000000000002,"5min":2.351,"15min":55.518999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.327,"15min":1.482},"min":{"1min":0.60999999999999999,"5min":0.53100000000000003,"15min":0.48699999999999999},"max":{"1min":2.2570000000000001,"5min":2.7429999999999999,"15min":55.497999999999998},"last":0.68000000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:43:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.3380000000000001,"15min":1.425},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":3.6880000000000002,"5min":5.6749999999999998,"15min":8.6940000000000008},"last":0.626},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.3839999999999999,"15min":1.3520000000000001},"min":{"1min":0.65200000000000002,"5min":0.54400000000000004,"15min":0.51500000000000001},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":6.9029999999999996},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3418095616,"data_stored":3418095616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1454702592,"data_stored":1454702592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":957612032,"data_stored":957612032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2088042496,"data_stored":2088042496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":603783168,"data_stored":603783168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1432682496,"data_stored":1432682496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":941359104,"data_stored":941359104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1151401984,"data_stored":1151401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1906114560,"data_stored":1906114560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1553268736,"data_stored":1553268736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:43:39.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set device_health_metrics pgp_num 31 2024-02-12T16:43:39.049 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:39.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:39.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:39.183 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:39.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:39.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:39.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:39.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:39.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:39.451 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:39.452 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:39.475 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:39.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:39.541 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:39.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:39 smithi123 bash[19119]: audit 2024-02-12T16:43:37.935112+0000 mgr.y (mgr.14146) 1779 : audit [DBG] from='client.17169 v1:172.21.15.121:0/802515251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:43:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:39 smithi123 bash[19119]: cluster 2024-02-12T16:43:38.284627+0000 mgr.y (mgr.14146) 1780 : cluster [DBG] pgmap v2593: 47 pgs: 3 active+recovering, 44 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 59 MiB/s wr, 945 op/s; 16601/235652 objects misplaced (7.045%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:43:39.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 63 16 122489 122473 121.489 71.5625 0.00418861 0.00822907 2024-02-12T16:43:39.763 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:39.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:39 smithi169 bash[19604]: audit 2024-02-12T16:43:37.935112+0000 mgr.y (mgr.14146) 1779 : audit [DBG] from='client.17169 v1:172.21.15.121:0/802515251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:43:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:39 smithi169 bash[19604]: cluster 2024-02-12T16:43:38.284627+0000 mgr.y (mgr.14146) 1780 : cluster [DBG] pgmap v2593: 47 pgs: 3 active+recovering, 44 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 59 MiB/s wr, 945 op/s; 16601/235652 objects misplaced (7.045%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:43:39.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:39.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:39 smithi121 bash[18489]: audit 2024-02-12T16:43:37.935112+0000 mgr.y (mgr.14146) 1779 : audit [DBG] from='client.17169 v1:172.21.15.121:0/802515251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:43:39.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:39 smithi121 bash[18489]: cluster 2024-02-12T16:43:38.284627+0000 mgr.y (mgr.14146) 1780 : cluster [DBG] pgmap v2593: 47 pgs: 3 active+recovering, 44 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 59 MiB/s wr, 945 op/s; 16601/235652 objects misplaced (7.045%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:43:39.894 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:39.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:39.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:40.024 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:40.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:40.154 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:40.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:40.303 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:40.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:40.456 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:40 smithi169 bash[19604]: audit 2024-02-12T16:43:40.223088+0000 mon.a (mon.0) 3160 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:43:40.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:40.595 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:40.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:40 smithi123 bash[19119]: audit 2024-02-12T16:43:40.223088+0000 mon.a (mon.0) 3160 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:43:40.649 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 64 16 123441 123425 120.52 59.5 0.0132409 0.00829563 2024-02-12T16:43:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:40 smithi121 bash[18489]: audit 2024-02-12T16:43:40.223088+0000 mon.a (mon.0) 3160 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:43:40.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:40.833 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:40.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:40.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:40.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:40.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:41.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:41.124 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:41.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:41.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:41.264 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:41.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:41.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:41.400 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:41.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:41.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:41.534 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:41.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:41.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:41.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 65 16 124556 124540 119.738 69.6875 0.0087665 0.00834898 2024-02-12T16:43:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:41 smithi121 bash[18489]: cluster 2024-02-12T16:43:40.285242+0000 mgr.y (mgr.14146) 1781 : cluster [DBG] pgmap v2594: 47 pgs: 3 active+recovering, 44 active+clean; 7.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 59 MiB/s wr, 940 op/s; 16601/239860 objects misplaced (6.921%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:43:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:41 smithi121 bash[18489]: audit 2024-02-12T16:43:40.560802+0000 mon.a (mon.0) 3161 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:41 smithi121 bash[18489]: audit 2024-02-12T16:43:40.927472+0000 mon.a (mon.0) 3162 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:43:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:41 smithi121 bash[18489]: audit 2024-02-12T16:43:40.928388+0000 mon.a (mon.0) 3163 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:43:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:41 smithi121 bash[18489]: audit 2024-02-12T16:43:40.933455+0000 mon.a (mon.0) 3164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:41 smithi121 bash[18489]: cluster 2024-02-12T16:43:40.978663+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e944: 12 total, 12 up, 11 in 2024-02-12T16:43:41.662 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:41.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:41.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:41.808 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:41.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:41 smithi123 bash[19119]: cluster 2024-02-12T16:43:40.285242+0000 mgr.y (mgr.14146) 1781 : cluster [DBG] pgmap v2594: 47 pgs: 3 active+recovering, 44 active+clean; 7.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 59 MiB/s wr, 940 op/s; 16601/239860 objects misplaced (6.921%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:43:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:41 smithi123 bash[19119]: audit 2024-02-12T16:43:40.560802+0000 mon.a (mon.0) 3161 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:41 smithi123 bash[19119]: audit 2024-02-12T16:43:40.927472+0000 mon.a (mon.0) 3162 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:43:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:41 smithi123 bash[19119]: audit 2024-02-12T16:43:40.928388+0000 mon.a (mon.0) 3163 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:43:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:41 smithi123 bash[19119]: audit 2024-02-12T16:43:40.933455+0000 mon.a (mon.0) 3164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:41 smithi123 bash[19119]: cluster 2024-02-12T16:43:40.978663+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e944: 12 total, 12 up, 11 in 2024-02-12T16:43:41.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:41.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:41 smithi169 bash[19604]: cluster 2024-02-12T16:43:40.285242+0000 mgr.y (mgr.14146) 1781 : cluster [DBG] pgmap v2594: 47 pgs: 3 active+recovering, 44 active+clean; 7.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 59 MiB/s wr, 940 op/s; 16601/239860 objects misplaced (6.921%); 2.7 MiB/s, 42 objects/s recovering 2024-02-12T16:43:41.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:41 smithi169 bash[19604]: audit 2024-02-12T16:43:40.560802+0000 mon.a (mon.0) 3161 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:41.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:41 smithi169 bash[19604]: audit 2024-02-12T16:43:40.927472+0000 mon.a (mon.0) 3162 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:43:41.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:41 smithi169 bash[19604]: audit 2024-02-12T16:43:40.928388+0000 mon.a (mon.0) 3163 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:43:41.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:41 smithi169 bash[19604]: audit 2024-02-12T16:43:40.933455+0000 mon.a (mon.0) 3164 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:43:41.917 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:41 smithi169 bash[19604]: cluster 2024-02-12T16:43:40.978663+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e944: 12 total, 12 up, 11 in 2024-02-12T16:43:41.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:41.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:42.038 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:42.067 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:42.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:42.210 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:42.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:42.340 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:42.444 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:42.470 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:42.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:42.599 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:42.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 66 16 125688 125672 118.996 70.75 0.00920886 0.00840133 2024-02-12T16:43:42.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:42.708 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:42 smithi123 bash[19119]: cluster 2024-02-12T16:43:41.012297+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:42.708 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:42 smithi123 bash[19119]: cluster 2024-02-12T16:43:41.012347+0000 osd.6 (osd.6) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:42.708 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:42 smithi123 bash[19119]: cluster 2024-02-12T16:43:41.987929+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e945: 12 total, 12 up, 11 in 2024-02-12T16:43:42.726 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:42.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:42.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:42 smithi121 bash[18489]: cluster 2024-02-12T16:43:41.012297+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:42.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:42 smithi121 bash[18489]: cluster 2024-02-12T16:43:41.012347+0000 osd.6 (osd.6) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:42.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:42 smithi121 bash[18489]: cluster 2024-02-12T16:43:41.987929+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e945: 12 total, 12 up, 11 in 2024-02-12T16:43:42.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:42.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:42.995 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:42.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:42.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:42 smithi169 bash[19604]: cluster 2024-02-12T16:43:41.012297+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:42 smithi169 bash[19604]: cluster 2024-02-12T16:43:41.012347+0000 osd.6 (osd.6) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:42 smithi169 bash[19604]: cluster 2024-02-12T16:43:41.987929+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e945: 12 total, 12 up, 11 in 2024-02-12T16:43:43.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:43.135 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:43.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:43.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:43.274 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:43.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:43.378 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:43.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:43.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:43.509 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:43.538 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:43.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:43.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:43.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:43 smithi169 bash[19604]: cluster 2024-02-12T16:43:42.285621+0000 mgr.y (mgr.14146) 1782 : cluster [DBG] pgmap v2597: 47 pgs: 3 active+recovering, 44 active+clean; 7.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s; 16361/246660 objects misplaced (6.633%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T16:43:43.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 67 16 126879 126863 118.331 74.4375 0.0185535 0.00844834 2024-02-12T16:43:43.672 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:43.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:43.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:43.801 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:43.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:43 smithi123 bash[19119]: cluster 2024-02-12T16:43:42.285621+0000 mgr.y (mgr.14146) 1782 : cluster [DBG] pgmap v2597: 47 pgs: 3 active+recovering, 44 active+clean; 7.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s; 16361/246660 objects misplaced (6.633%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T16:43:43.903 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:43 smithi121 bash[18489]: cluster 2024-02-12T16:43:42.285621+0000 mgr.y (mgr.14146) 1782 : cluster [DBG] pgmap v2597: 47 pgs: 3 active+recovering, 44 active+clean; 7.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s; 16361/246660 objects misplaced (6.633%); 4.6 MiB/s, 73 objects/s recovering 2024-02-12T16:43:43.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:43.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:44.032 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:44.061 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:44.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:44.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:44.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:44.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:44.423 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:44.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:44.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:44.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:44.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 68 16 128005 127989 117.625 70.375 0.00784849 0.00849833 2024-02-12T16:43:44.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:44.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:44.829 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:44.859 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:44.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:44.990 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:44.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:45.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:45.127 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:45.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:45.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:45.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:45.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:45.397 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:45.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:45.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:45.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:45.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:45.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:45 smithi123 bash[19119]: cluster 2024-02-12T16:43:44.286102+0000 mgr.y (mgr.14146) 1783 : cluster [DBG] pgmap v2598: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 16361/250856 objects misplaced (6.522%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:43:45.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:45.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 69 16 129148 129132 116.956 71.4375 0.0162242 0.00854827 2024-02-12T16:43:45.666 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:45.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:45.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:45.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:45 smithi121 bash[18489]: cluster 2024-02-12T16:43:44.286102+0000 mgr.y (mgr.14146) 1783 : cluster [DBG] pgmap v2598: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 16361/250856 objects misplaced (6.522%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:43:45.799 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:45.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:45.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:45.911 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:45 smithi169 bash[19604]: cluster 2024-02-12T16:43:44.286102+0000 mgr.y (mgr.14146) 1783 : cluster [DBG] pgmap v2598: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 16361/250856 objects misplaced (6.522%); 2.2 MiB/s, 34 objects/s recovering 2024-02-12T16:43:45.933 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:45.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:46.038 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:46.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:46.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:46.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:46.202 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:46.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:46.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:46.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:46.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:46.459 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:46.485 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:46.487 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:43:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:46.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:46.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:46.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:46.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:46.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 70 15 130239 130224 116.26 68.25 0.00506736 0.00859858 2024-02-12T16:43:46.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:46.757 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:46.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:46.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:46.893 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:46.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:46.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:47.028 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:47.131 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:47.158 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:47.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:46 smithi121 bash[18489]: cluster 2024-02-12T16:43:45.979773+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e946: 12 total, 12 up, 11 in 2024-02-12T16:43:47.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:46 smithi121 bash[18489]: cluster 2024-02-12T16:43:46.286524+0000 mgr.y (mgr.14146) 1784 : cluster [DBG] pgmap v2600: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 16303/252344 objects misplaced (6.461%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:43:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:47.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:47.269 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:46 smithi169 bash[19604]: cluster 2024-02-12T16:43:45.979773+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e946: 12 total, 12 up, 11 in 2024-02-12T16:43:47.269 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:46 smithi169 bash[19604]: cluster 2024-02-12T16:43:46.286524+0000 mgr.y (mgr.14146) 1784 : cluster [DBG] pgmap v2600: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 16303/252344 objects misplaced (6.461%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:43:47.292 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:46 smithi123 bash[19119]: cluster 2024-02-12T16:43:45.979773+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e946: 12 total, 12 up, 11 in 2024-02-12T16:43:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:46 smithi123 bash[19119]: cluster 2024-02-12T16:43:46.286524+0000 mgr.y (mgr.14146) 1784 : cluster [DBG] pgmap v2600: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 16303/252344 objects misplaced (6.461%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:43:47.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:47.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:47.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:47.558 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:47.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 71 16 131288 131272 115.545 65.5 0.0414593 0.00865131 2024-02-12T16:43:47.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:47.688 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:47.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:47.823 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:47.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:47.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:47.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:48.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:48.091 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:48.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:48.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:48.269 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:48.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:48 smithi123 bash[19119]: cluster 2024-02-12T16:43:47.429427+0000 osd.0 (osd.0) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:48.270 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:48 smithi123 bash[19119]: cluster 2024-02-12T16:43:47.429481+0000 osd.0 (osd.0) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:48.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:47 smithi169 bash[19604]: cluster 2024-02-12T16:43:47.429427+0000 osd.0 (osd.0) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:48.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:47 smithi169 bash[19604]: cluster 2024-02-12T16:43:47.429481+0000 osd.0 (osd.0) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:48.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:48.379 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:47 smithi121 bash[18489]: cluster 2024-02-12T16:43:47.429427+0000 osd.0 (osd.0) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:48.379 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:47 smithi121 bash[18489]: cluster 2024-02-12T16:43:47.429481+0000 osd.0 (osd.0) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:48.401 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:48.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:48.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:48.531 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:48.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:48.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:48.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 72 16 132408 132392 114.912 70 0.00570127 0.00870018 2024-02-12T16:43:48.659 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:48.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:48.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:48.793 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:48.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:48.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:48.928 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:48.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:49.007 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:49 smithi121 bash[18489]: cluster 2024-02-12T16:43:47.991120+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e947: 12 total, 12 up, 11 in 2024-02-12T16:43:49.007 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:49 smithi121 bash[18489]: cluster 2024-02-12T16:43:48.286977+0000 mgr.y (mgr.14146) 1785 : cluster [DBG] pgmap v2602: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s; 16117/257930 objects misplaced (6.249%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:43:49.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:49.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:49.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:49.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:49.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:49.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:49.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:49.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:49 smithi169 bash[19604]: cluster 2024-02-12T16:43:47.991120+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e947: 12 total, 12 up, 11 in 2024-02-12T16:43:49.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:49 smithi169 bash[19604]: cluster 2024-02-12T16:43:48.286977+0000 mgr.y (mgr.14146) 1785 : cluster [DBG] pgmap v2602: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s; 16117/257930 objects misplaced (6.249%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:43:49.303 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:49 smithi123 bash[19119]: cluster 2024-02-12T16:43:47.991120+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e947: 12 total, 12 up, 11 in 2024-02-12T16:43:49.303 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:49 smithi123 bash[19119]: cluster 2024-02-12T16:43:48.286977+0000 mgr.y (mgr.14146) 1785 : cluster [DBG] pgmap v2602: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s; 16117/257930 objects misplaced (6.249%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:43:49.321 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:49.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:49.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:49.455 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:49.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:49.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:49.589 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:49.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:49.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 73 16 133476 133460 114.253 66.75 0.00525746 0.00874888 2024-02-12T16:43:49.695 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:49.722 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:49.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:49.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:49.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:49.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:49.999 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:50.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:50.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:50 smithi169 bash[19604]: cluster 2024-02-12T16:43:49.393958+0000 osd.10 (osd.10) 236 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:50.096 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:50 smithi169 bash[19604]: cluster 2024-02-12T16:43:49.394008+0000 osd.10 (osd.10) 237 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:50.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:50.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:50.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:50.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:50 smithi121 bash[18489]: cluster 2024-02-12T16:43:49.393958+0000 osd.10 (osd.10) 236 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:50.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:50 smithi121 bash[18489]: cluster 2024-02-12T16:43:49.394008+0000 osd.10 (osd.10) 237 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:50.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:50.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:50.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:50.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:50 smithi123 bash[19119]: cluster 2024-02-12T16:43:49.393958+0000 osd.10 (osd.10) 236 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:50.384 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:50 smithi123 bash[19119]: cluster 2024-02-12T16:43:49.394008+0000 osd.10 (osd.10) 237 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:50.401 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:50.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:50.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:50.530 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:50.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:50.601 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:43:50.601 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:43:50.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:43:50.601 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:43:50.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26785.0:106004 8.10 8:0c16ca9a:::benchmark_data_smithi169_115677_object23376:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e882)", 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 172.23223202200001, 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.07780718, 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26785", 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106004 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2119999999999999e-06 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355593+0000", 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1560000000000003e-06 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355598+0000", 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8820000000000003e-06 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355602+0000", 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.149617209 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.505219+0000", 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5519999999999998e-06 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.505224+0000", 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.928077547 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:59.433301+0000", 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6343999999999998e-05 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:59.433397+0000", 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.07780718 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:110525 9.1 9:8ce17ec1:::benchmark_data_smithi169_124168_object110524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e939)", 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:28.998347+0000", 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.589475197999999, 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19730310300000001, 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110525 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:28.998347+0000", 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:28.998347+0000", 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:28.998344+0000", 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8818000000000004e-05 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:28.998413+0000", 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9313e-05 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:28.998432+0000", 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2310000000000002e-06 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:28.998442+0000", 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.043655544999999997 2024-02-12T16:43:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.042097+0000", 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010524899999999999 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.042202+0000", 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011296199999999999 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.042315+0000", 2024-02-12T16:43:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020214543000000001 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.062530+0000", 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12949914600000001 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.192029+0000", 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2972000000000001e-05 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.192072+0000", 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003578164 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.195650+0000", 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19730310300000001 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:110529 9.1 9:8b22da8f:::benchmark_data_smithi169_124168_object110528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:29.003192+0000", 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.584630253, 2024-02-12T16:43:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19497829, 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110529 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.003192+0000", 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.003192+0000", 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.003189+0000", 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6011000000000003e-05 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.003245+0000", 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3783e-05 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.003259+0000", 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7339999999999996e-06 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.003265+0000", 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.042710247 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.045975+0000", 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000136577 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.046111+0000", 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000168892 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.046280+0000", 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024220055000000001 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.070500+0000", 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.125217781 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.195718+0000", 2024-02-12T16:43:50.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0905000000000002e-05 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.195759+0000", 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024112119999999998 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.198170+0000", 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19497829 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:110541 9.1 9:889ba44f:::benchmark_data_smithi169_124168_object110540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:29.019665+0000", 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.568157199000002, 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18121289900000001, 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110541 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.019665+0000", 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.019665+0000", 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.019662+0000", 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8446000000000003e-05 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.019721+0000", 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2824e-05 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.019733+0000", 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9179999999999999e-06 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.019739+0000", 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029947899 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.049687+0000", 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012386 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.049811+0000", 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155768 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.049967+0000", 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020538154999999999 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.070505+0000", 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.127725428 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.198231+0000", 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1959000000000003e-05 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.198272+0000", 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0026055029999999999 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.200878+0000", 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18121289900000001 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:110554 9.1 9:84be5dea:::benchmark_data_smithi169_124168_object110553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:29.030606+0000", 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.557216378, 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17263824899999999, 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110554 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.030606+0000", 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.030606+0000", 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999847 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.030591+0000", 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.893e-05 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.030650+0000", 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5919000000000001e-05 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.030666+0000", 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9360000000000002e-06 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.030673+0000", 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.022523206 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.053196+0000", 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015042399999999999 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.053346+0000", 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000184697 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.053531+0000", 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016982456999999999 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.070513+0000", 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.130413689 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.200927+0000", 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2199e-05 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.200969+0000", 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022749189999999998 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:29.203244+0000", 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17263824899999999 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:115368 9.1 9:8cf30c7b:::benchmark_data_smithi169_124168_object115367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:33.355543+0000", 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.23227894, 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085396062999999994, 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115368 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.355543+0000", 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.355543+0000", 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.355542+0000", 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399000000000001e-05 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.355571+0000", 2024-02-12T16:43:50.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1001e-05 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.355582+0000", 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1970000000000004e-06 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.355588+0000", 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.037584944000000002 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.393172+0000", 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114721 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.393287+0000", 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000119405 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.393407+0000", 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012863247 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.406270+0000", 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030187446 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.436457+0000", 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7402000000000003e-05 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.436525+0000", 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044147040000000002 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.440939+0000", 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.085396062999999994 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:115378 9.1 9:8befa8cf:::benchmark_data_smithi169_124168_object115377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:33.362286+0000", 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.225536701999999, 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082837957000000004, 2024-02-12T16:43:50.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115378 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.362286+0000", 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.362286+0000", 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.362283+0000", 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3499999999999999e-05 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.362346+0000", 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1453e-05 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.362358+0000", 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7680000000000003e-06 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.362364+0000", 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034138955999999998 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.396503+0000", 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105517 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.396609+0000", 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000138708 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.396748+0000", 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095334089999999996 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.406281+0000", 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034763201000000001 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.441044+0000", 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4834999999999998e-05 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.441119+0000", 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00400454 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.445124+0000", 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082837957000000004 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:115379 9.1 9:8b5398aa:::benchmark_data_smithi169_124168_object115378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:33.363366+0000", 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.224455787, 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.087369726999999994, 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115379 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.363366+0000", 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.363366+0000", 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.363365+0000", 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110733 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.363475+0000", 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049430100000000005 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.363970+0000", 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3259999999999996e-06 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.363978+0000", 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035206978999999999 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.399185+0000", 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012904100000000001 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.399314+0000", 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113328 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.399427+0000", 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015508495000000001 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.414936+0000", 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030246215 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.445182+0000", 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2435999999999997e-05 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.445234+0000", 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055017219999999997 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:33.450736+0000", 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.087369726999999994 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:118841 9.1 9:81a3c9cf:::benchmark_data_smithi169_124168_object118840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e942)", 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:36.435063+0000", 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.152759138, 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079629757999999995, 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 118841 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.435063+0000", 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.435063+0000", 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.435061+0000", 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010378399999999999 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.435165+0000", 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4278e-05 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.435179+0000", 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2060000000000001e-06 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.435186+0000", 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019711045999999999 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.454897+0000", 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020251899999999999 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.455100+0000", 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000225004 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.455325+0000", 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.020515839000000001 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.475841+0000", 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035160968000000001 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.511002+0000", 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8916e-05 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.511091+0000", 2024-02-12T16:43:50.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036021619999999999 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.514693+0000", 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079629757999999995 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:118851 9.1 9:80b3887b:::benchmark_data_smithi169_124168_object118850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e942)", 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:36.441156+0000", 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.146666765999999, 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078532571999999995, 2024-02-12T16:43:50.652 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 118851 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.441156+0000", 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.441156+0000", 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.441153+0000", 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5181999999999999e-05 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.441238+0000", 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6559000000000001e-05 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.441265+0000", 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4240000000000004e-06 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.441272+0000", 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.018096475000000001 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.459368+0000", 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016741600000000001 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.459536+0000", 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016955900000000001 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.459705+0000", 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.016142608999999999 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.475848+0000", 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.038903106999999999 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.514751+0000", 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6923999999999998e-05 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.514818+0000", 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048700449999999999 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:36.519688+0000", 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.078532571999999995 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:120821 9.1 9:8d3fec68:::benchmark_data_smithi169_124168_object120820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:38.199801+0000", 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.388021604, 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079691656, 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 120821 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.199801+0000", 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.199801+0000", 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999924 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.199793+0000", 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5822999999999998e-05 2024-02-12T16:43:50.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.199839+0000", 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6271000000000001e-05 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.199855+0000", 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6909999999999997e-06 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.199863+0000", 2024-02-12T16:43:50.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.028333549999999999 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.228196+0000", 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000166384 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.228363+0000", 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000199729 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.228562+0000", 2024-02-12T16:43:50.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011224252000000001 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.239787+0000", 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.036069125 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.275856+0000", 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6558000000000003e-05 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.275912+0000", 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0035801140000000001 2024-02-12T16:43:50.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:38.279492+0000", 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.079691656 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:122532 9.1 9:8f6a5060:::benchmark_data_smithi169_124168_object122531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.678976+0000", 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.908846575, 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20243556900000001, 2024-02-12T16:43:50.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 122532 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.678976+0000", 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.678976+0000", 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999847 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.678960+0000", 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9978000000000001e-05 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679020+0000", 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9273999999999999e-05 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679040+0000", 2024-02-12T16:43:50.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9089999999999999e-06 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679045+0000", 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0336999999999997e-05 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679085+0000", 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014626099999999999 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679231+0000", 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174907 2024-02-12T16:43:50.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679406+0000", 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091274470000000003 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.688534+0000", 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015220551000000001 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.703754+0000", 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17174315200000001 2024-02-12T16:43:50.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875497+0000", 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059140199999999999 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.881411+0000", 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20243556900000001 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(9.1 943/935 [PushReplyOp(9:85e0efaa:::benchmark_data_smithi169_124168_object9727:head)])", 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.693117+0000", 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.894704968999999, 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20235745399999999, 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693117+0000", 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693117+0000", 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:43:50.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693116+0000", 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7070000000000003e-06 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693123+0000", 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1909999999999997e-06 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693129+0000", 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8430000000000003e-06 2024-02-12T16:43:50.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693133+0000", 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19611678199999999 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.889250+0000", 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006224907 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.895475+0000", 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20235745399999999 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26977.0:122532 9.1 e943/935 ondisk, result = 0)", 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.693138+0000", 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.894683832, 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.188294935, 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693138+0000", 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693138+0000", 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693138+0000", 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7489999999999999e-06 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693141+0000", 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6609e-05 2024-02-12T16:43:50.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693157+0000", 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.882e-06 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693162+0000", 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.01057366 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.703736+0000", 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5529999999999998e-06 2024-02-12T16:43:50.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.703746+0000", 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.177687867 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.881433+0000", 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.188294935 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:122555 9.1 9:8efe72fe:::benchmark_data_smithi169_124168_object122554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:43:50.675 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.697530+0000", 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.890292664, 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20841303999999999, 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 122555 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697530+0000", 2024-02-12T16:43:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697530+0000", 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999938 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697523+0000", 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5047000000000002e-05 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697589+0000", 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.117e-05 2024-02-12T16:43:50.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697600+0000", 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5139999999999998e-06 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697606+0000", 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18391962000000001 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.881526+0000", 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026639600000000002 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.881792+0000", 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032232199999999999 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.882115+0000", 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071066419999999998 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.889221+0000", 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013541442000000001 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.902763+0000", 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1150999999999997e-05 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.902834+0000", 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003108857 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.905943+0000", 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20841303999999999 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.680 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:122557 9.c 9:3dfe9060:::benchmark_data_smithi169_124168_object122556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.700212+0000", 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.887610449, 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.175907234, 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 122557 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700212+0000", 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700212+0000", 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700209+0000", 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6629000000000003e-05 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700266+0000", 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4141999999999999e-05 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700280+0000", 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8810000000000004e-06 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700285+0000", 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3908e-05 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700329+0000", 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127377 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700456+0000", 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140617 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.700597+0000", 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17530027300000001 2024-02-12T16:43:50.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875897+0000", 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7297000000000001e-05 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875914+0000", 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011037899999999999 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.876025+0000", 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4238000000000001e-05 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.876119+0000", 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.175907234 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26977.0:122557 9.c e943/916 ondisk, result = 0)", 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.704576+0000", 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.883245857, 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17154913599999999, 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.704576+0000", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.704576+0000", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.704574+0000", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4519999999999997e-06 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.704582+0000", 2024-02-12T16:43:50.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17086191000000001 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875444+0000", 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5712999999999999e-05 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875470+0000", 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038910600000000001 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875859+0000", 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9357000000000001e-05 2024-02-12T16:43:50.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875878+0000", 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000247125 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.876126+0000", 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17154913599999999 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:122572 9.c 9:3e05ac4a:::benchmark_data_smithi169_124168_object122571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.711516+0000", 2024-02-12T16:43:50.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.876306502, 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20045165000000001, 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 122572 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711516+0000", 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711516+0000", 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711513+0000", 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5548000000000001e-05 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711609+0000", 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.163957779 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875567+0000", 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6982999999999999e-05 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875594+0000", 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058365700000000001 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.876177+0000", 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035321000000000001 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.876531+0000", 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012794799999999999 2024-02-12T16:43:50.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.877810+0000", 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093099129999999995 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.887120+0000", 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024732095999999999 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.911852+0000", 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7362999999999999e-05 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.911919+0000", 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8062000000000002e-05 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.911967+0000", 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20045165000000001 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:125467 9.1 9:87c054f8:::benchmark_data_smithi169_124168_object125466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e945)", 2024-02-12T16:43:50.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:42.436871+0000", 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.1509516810000004, 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080970225000000007, 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 125467 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.436871+0000", 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.436871+0000", 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.436869+0000", 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8125999999999999e-05 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.436928+0000", 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7640000000000005e-06 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.436935+0000", 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1309999999999997e-06 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.436940+0000", 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.026300881000000002 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.463241+0000", 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1934000000000004e-05 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.463333+0000", 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000103188 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.463436+0000", 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.012571956 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.476008+0000", 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.038325113000000001 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.514333+0000", 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2237999999999996e-05 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.514396+0000", 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034450890000000001 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:42.517841+0000", 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.080970225000000007 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:129899 9.1 9:84df9405:::benchmark_data_smithi169_124168_object129898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e946)", 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:46.331590+0000", 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.2562323040000001, 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082750475000000004, 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:43:50.700 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 129899 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.331590+0000", 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.331590+0000", 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:43:50.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.331588+0000", 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020795200000000001 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.333667+0000", 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.063e-05 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.333678+0000", 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6760000000000004e-06 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.333684+0000", 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014133849 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.347817+0000", 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014141800000000001 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.347959+0000", 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000156732 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.348116+0000", 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0118265 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.359942+0000", 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.050894911000000001 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.410837+0000", 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4589999999999999e-05 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.410872+0000", 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034688269999999998 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:46.414340+0000", 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082750475000000004 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:43:50.705 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:43:50.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:50.707 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 74 16 134559 134543 113.623 67.6875 0.0373057 0.00879875 2024-02-12T16:43:50.740 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:50.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:50.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:50.880 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:50.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:50.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:51.017 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:51.122 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:51.156 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:51.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:51 smithi121 bash[18489]: cluster 2024-02-12T16:43:50.013439+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e948: 12 total, 12 up, 11 in 2024-02-12T16:43:51.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:51 smithi121 bash[18489]: cluster 2024-02-12T16:43:50.287430+0000 mgr.y (mgr.14146) 1786 : cluster [DBG] pgmap v2604: 47 pgs: 3 active+recovering, 44 active+clean; 8.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s; 16117/262096 objects misplaced (6.149%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T16:43:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:51.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:51.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:51 smithi169 bash[19604]: cluster 2024-02-12T16:43:50.013439+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e948: 12 total, 12 up, 11 in 2024-02-12T16:43:51.270 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:51 smithi169 bash[19604]: cluster 2024-02-12T16:43:50.287430+0000 mgr.y (mgr.14146) 1786 : cluster [DBG] pgmap v2604: 47 pgs: 3 active+recovering, 44 active+clean; 8.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s; 16117/262096 objects misplaced (6.149%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T16:43:51.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:51 smithi123 bash[19119]: cluster 2024-02-12T16:43:50.013439+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e948: 12 total, 12 up, 11 in 2024-02-12T16:43:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:51 smithi123 bash[19119]: cluster 2024-02-12T16:43:50.287430+0000 mgr.y (mgr.14146) 1786 : cluster [DBG] pgmap v2604: 47 pgs: 3 active+recovering, 44 active+clean; 8.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s; 16117/262096 objects misplaced (6.149%); 2.5 MiB/s, 40 objects/s recovering 2024-02-12T16:43:51.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:51.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:51.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:51.550 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:51.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 75 16 135649 135633 113.017 68.125 0.00461143 0.00884586 2024-02-12T16:43:51.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:51.678 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:51.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:51.808 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:51.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:51.944 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:51.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:52.051 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:52.082 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:52.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:52.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:52.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:52.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:52.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:52.363 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:52.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:52.470 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:52.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:52.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:52.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:52.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:52.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:52.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 76 16 136626 136610 112.333 61.0625 0.151819 0.00889835 2024-02-12T16:43:52.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:52.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:52.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:52.871 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:52.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:52.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:53.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:53.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:53.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:53.162 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:53.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:53.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:53.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:53.405 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:53 smithi169 bash[19604]: cluster 2024-02-12T16:43:52.287855+0000 mgr.y (mgr.14146) 1787 : cluster [DBG] pgmap v2605: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 15879/268856 objects misplaced (5.906%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T16:43:53.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:53.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:53.556 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:43:53.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:53.579 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:43:53.609 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:43:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:53 smithi123 bash[19119]: cluster 2024-02-12T16:43:52.287855+0000 mgr.y (mgr.14146) 1787 : cluster [DBG] pgmap v2605: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 15879/268856 objects misplaced (5.906%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T16:43:53.639 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:43:53.650 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 77 16 137649 137633 111.704 63.9375 0.0392247 0.00894997 2024-02-12T16:43:53.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:53 smithi121 bash[18489]: cluster 2024-02-12T16:43:52.287855+0000 mgr.y (mgr.14146) 1787 : cluster [DBG] pgmap v2605: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 15879/268856 objects misplaced (5.906%); 4.2 MiB/s, 67 objects/s recovering 2024-02-12T16:43:53.670 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:43:53.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:53.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:53.694 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:43:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:53.720 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:43:53.754 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:43:53.783 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:43:53.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:53.810 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:43:53.836 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:43:53.837 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:53.861 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.865 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.866 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:43:53.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:53.969 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:53.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:54.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:54.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:54.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:54.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:54.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:54.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:54.370 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:43:54.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:43:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:54.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:43:54.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:54.520 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:54.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:54.623 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:54.630 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:54 smithi123 bash[19119]: audit 2024-02-12T16:43:53.524269+0000 mon.b (mon.2) 223 : audit [DBG] from='client.? v1:172.21.15.121:0/2404989543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:54.630 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:54 smithi123 bash[19119]: audit 2024-02-12T16:43:53.817070+0000 mon.a (mon.0) 3170 : audit [INF] from='client.27034 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:54.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:54 smithi123 bash[19119]: audit 2024-02-12T16:43:53.818714+0000 mon.c (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.121:0/4052532330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:54.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:54.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 78 16 138678 138662 111.097 64.3125 0.00972509 0.00899875 2024-02-12T16:43:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:54.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:54.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:54 smithi121 bash[18489]: audit 2024-02-12T16:43:53.524269+0000 mon.b (mon.2) 223 : audit [DBG] from='client.? v1:172.21.15.121:0/2404989543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:54.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:54 smithi121 bash[18489]: audit 2024-02-12T16:43:53.817070+0000 mon.a (mon.0) 3170 : audit [INF] from='client.27034 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:54.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:54 smithi121 bash[18489]: audit 2024-02-12T16:43:53.818714+0000 mon.c (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.121:0/4052532330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:54.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:54.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:54.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:54 smithi169 bash[19604]: audit 2024-02-12T16:43:53.524269+0000 mon.b (mon.2) 223 : audit [DBG] from='client.? v1:172.21.15.121:0/2404989543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:43:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:54 smithi169 bash[19604]: audit 2024-02-12T16:43:53.817070+0000 mon.a (mon.0) 3170 : audit [INF] from='client.27034 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:54 smithi169 bash[19604]: audit 2024-02-12T16:43:53.818714+0000 mon.c (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.121:0/4052532330' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:43:54.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:54.912 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:54.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:54.970 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:43:55.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:55.043 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:55.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:55.188 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:55.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:55.314 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:55.357 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:55 smithi121 bash[18489]: cluster 2024-02-12T16:43:54.288377+0000 mgr.y (mgr.14146) 1788 : cluster [DBG] pgmap v2606: 47 pgs: 3 active+recovering, 44 active+clean; 8.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s; 15879/272814 objects misplaced (5.820%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:43:55.357 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:55 smithi121 bash[18489]: cluster 2024-02-12T16:43:54.341728+0000 mon.a (mon.0) 3171 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:43:55.357 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:55 smithi121 bash[18489]: cluster 2024-02-12T16:43:54.341762+0000 mon.a (mon.0) 3172 : cluster [INF] Cluster is now healthy 2024-02-12T16:43:55.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:55 smithi121 bash[18489]: audit 2024-02-12T16:43:54.353740+0000 mon.a (mon.0) 3173 : audit [INF] from='client.27034 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:43:55.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:55 smithi121 bash[18489]: cluster 2024-02-12T16:43:54.353801+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e949: 12 total, 12 up, 11 in 2024-02-12T16:43:55.417 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:55.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:55 smithi169 bash[19604]: cluster 2024-02-12T16:43:54.288377+0000 mgr.y (mgr.14146) 1788 : cluster [DBG] pgmap v2606: 47 pgs: 3 active+recovering, 44 active+clean; 8.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s; 15879/272814 objects misplaced (5.820%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:43:55.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:55 smithi169 bash[19604]: cluster 2024-02-12T16:43:54.341728+0000 mon.a (mon.0) 3171 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:43:55.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:55 smithi169 bash[19604]: cluster 2024-02-12T16:43:54.341762+0000 mon.a (mon.0) 3172 : cluster [INF] Cluster is now healthy 2024-02-12T16:43:55.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:55 smithi169 bash[19604]: audit 2024-02-12T16:43:54.353740+0000 mon.a (mon.0) 3173 : audit [INF] from='client.27034 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:43:55.425 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:55 smithi169 bash[19604]: cluster 2024-02-12T16:43:54.353801+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e949: 12 total, 12 up, 11 in 2024-02-12T16:43:55.443 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:55.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:55.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:55 smithi123 bash[19119]: cluster 2024-02-12T16:43:54.288377+0000 mgr.y (mgr.14146) 1788 : cluster [DBG] pgmap v2606: 47 pgs: 3 active+recovering, 44 active+clean; 8.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s; 15879/272814 objects misplaced (5.820%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:43:55.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:55 smithi123 bash[19119]: cluster 2024-02-12T16:43:54.341728+0000 mon.a (mon.0) 3171 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:43:55.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:55 smithi123 bash[19119]: cluster 2024-02-12T16:43:54.341762+0000 mon.a (mon.0) 3172 : cluster [INF] Cluster is now healthy 2024-02-12T16:43:55.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:55 smithi123 bash[19119]: audit 2024-02-12T16:43:54.353740+0000 mon.a (mon.0) 3173 : audit [INF] from='client.27034 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:43:55.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:55 smithi123 bash[19119]: cluster 2024-02-12T16:43:54.353801+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e949: 12 total, 12 up, 11 in 2024-02-12T16:43:55.574 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:55.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 79 16 139498 139482 110.339 51.25 0.0554797 0.00906042 2024-02-12T16:43:55.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:55.720 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:55.826 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:55.854 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:55.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:55.984 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:55.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:56.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:56.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:56.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:56.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:56.251 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:43:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:56.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:43:56.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:56.384 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:56.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:56.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:56.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:56.515 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:56.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:56.617 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:56.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:56 smithi169 bash[19604]: cluster 2024-02-12T16:43:55.816336+0000 osd.7 (osd.7) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:56.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:56 smithi169 bash[19604]: cluster 2024-02-12T16:43:55.816386+0000 osd.7 (osd.7) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:56.624 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:56 smithi169 bash[19604]: cluster 2024-02-12T16:43:55.986561+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e950: 12 total, 12 up, 11 in 2024-02-12T16:43:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:56 smithi123 bash[19119]: cluster 2024-02-12T16:43:55.816336+0000 osd.7 (osd.7) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:56 smithi123 bash[19119]: cluster 2024-02-12T16:43:55.816386+0000 osd.7 (osd.7) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:56 smithi123 bash[19119]: cluster 2024-02-12T16:43:55.986561+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e950: 12 total, 12 up, 11 in 2024-02-12T16:43:56.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:2024-02-12 16:43:56.648492 min lat: 0.000995729 max lat: 4.35085 avg lat: 0.00910758 2024-02-12T16:43:56.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-12T16:43:56.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 80 16 140531 140515 109.767 64.5625 0.0101395 0.00910758 2024-02-12T16:43:56.652 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:56.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:56.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:56 smithi121 bash[18489]: cluster 2024-02-12T16:43:55.816336+0000 osd.7 (osd.7) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:43:56.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:56 smithi121 bash[18489]: cluster 2024-02-12T16:43:55.816386+0000 osd.7 (osd.7) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:43:56.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:56 smithi121 bash[18489]: cluster 2024-02-12T16:43:55.986561+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e950: 12 total, 12 up, 11 in 2024-02-12T16:43:56.756 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:56.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:56.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:56.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:56.913 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:56.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:57.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:57.044 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:57.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:57.175 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:57.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:57.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:57.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:57.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:57.439 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:57.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:57 smithi169 bash[19604]: cluster 2024-02-12T16:43:56.288794+0000 mgr.y (mgr.14146) 1789 : cluster [DBG] pgmap v2609: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s; 15821/274074 objects misplaced (5.773%); 2.9 MiB/s, 47 objects/s recovering 2024-02-12T16:43:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:57 smithi169 bash[19604]: audit 2024-02-12T16:43:57.144366+0000 mon.a (mon.0) 3176 : audit [INF] from='client.27098 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:43:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:57 smithi169 bash[19604]: audit 2024-02-12T16:43:57.147061+0000 mon.b (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.121:0/3524237734' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:43:57.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:43:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:43:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:57 smithi123 bash[19119]: cluster 2024-02-12T16:43:56.288794+0000 mgr.y (mgr.14146) 1789 : cluster [DBG] pgmap v2609: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s; 15821/274074 objects misplaced (5.773%); 2.9 MiB/s, 47 objects/s recovering 2024-02-12T16:43:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:57 smithi123 bash[19119]: audit 2024-02-12T16:43:57.144366+0000 mon.a (mon.0) 3176 : audit [INF] from='client.27098 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:43:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:57 smithi123 bash[19119]: audit 2024-02-12T16:43:57.147061+0000 mon.b (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.121:0/3524237734' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:43:57.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 81 16 141592 141576 109.23 66.3125 0.0182547 0.00915317 2024-02-12T16:43:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:57 smithi121 bash[18489]: cluster 2024-02-12T16:43:56.288794+0000 mgr.y (mgr.14146) 1789 : cluster [DBG] pgmap v2609: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s; 15821/274074 objects misplaced (5.773%); 2.9 MiB/s, 47 objects/s recovering 2024-02-12T16:43:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:57 smithi121 bash[18489]: audit 2024-02-12T16:43:57.144366+0000 mon.a (mon.0) 3176 : audit [INF] from='client.27098 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:43:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:57 smithi121 bash[18489]: audit 2024-02-12T16:43:57.147061+0000 mon.b (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.121:0/3524237734' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:43:57.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:57.699 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:43:57.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:43:57.806 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:57.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:57.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:57.980 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:57.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:58.003 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 1 pgp_num to 31 2024-02-12T16:43:58.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:58.117 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:58.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:58.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:58.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:58.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:58.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:58.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:58.484 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:58.509 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:43:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:58.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:43:58.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:43:58.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:43:58.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:58.638 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:58.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:58.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 82 16 142674 142658 108.723 67.625 0.0153452 0.00919577 2024-02-12T16:43:58.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:58.796 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:58.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:58.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:58.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:43:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:58.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:43:59.005 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:58 smithi121 bash[18489]: cluster 2024-02-12T16:43:57.729438+0000 osd.2 (osd.2) 137 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:43:59.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:59.081 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:43:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:59.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:43:59.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:43:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:59.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:43:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:59 smithi169 bash[19604]: cluster 2024-02-12T16:43:57.729438+0000 osd.2 (osd.2) 137 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:43:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:59 smithi169 bash[19604]: cluster 2024-02-12T16:43:57.836980+0000 osd.2 (osd.2) 138 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:43:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:59 smithi169 bash[19604]: audit 2024-02-12T16:43:57.992903+0000 mon.a (mon.0) 3177 : audit [INF] from='client.27098 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:43:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:59 smithi169 bash[19604]: cluster 2024-02-12T16:43:57.993062+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e951: 12 total, 12 up, 11 in 2024-02-12T16:43:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:43:59 smithi169 bash[19604]: cluster 2024-02-12T16:43:58.289213+0000 mgr.y (mgr.14146) 1790 : cluster [DBG] pgmap v2611: 47 pgs: 3 active+recovering, 44 active+clean; 8.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 52 MiB/s wr, 838 op/s; 15630/278916 objects misplaced (5.604%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:43:59.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:59.345 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:59.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:59 smithi123 bash[19119]: cluster 2024-02-12T16:43:57.729438+0000 osd.2 (osd.2) 137 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:43:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:59 smithi123 bash[19119]: cluster 2024-02-12T16:43:57.836980+0000 osd.2 (osd.2) 138 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:43:59.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:59 smithi123 bash[19119]: audit 2024-02-12T16:43:57.992903+0000 mon.a (mon.0) 3177 : audit [INF] from='client.27098 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:43:59.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:59 smithi123 bash[19119]: cluster 2024-02-12T16:43:57.993062+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e951: 12 total, 12 up, 11 in 2024-02-12T16:43:59.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:43:59 smithi123 bash[19119]: cluster 2024-02-12T16:43:58.289213+0000 mgr.y (mgr.14146) 1790 : cluster [DBG] pgmap v2611: 47 pgs: 3 active+recovering, 44 active+clean; 8.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 52 MiB/s wr, 838 op/s; 15630/278916 objects misplaced (5.604%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:43:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:58 smithi121 bash[18489]: cluster 2024-02-12T16:43:57.836980+0000 osd.2 (osd.2) 138 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:43:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:58 smithi121 bash[18489]: audit 2024-02-12T16:43:57.992903+0000 mon.a (mon.0) 3177 : audit [INF] from='client.27098 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:43:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:58 smithi121 bash[18489]: cluster 2024-02-12T16:43:57.993062+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e951: 12 total, 12 up, 11 in 2024-02-12T16:43:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:43:58 smithi121 bash[18489]: cluster 2024-02-12T16:43:58.289213+0000 mgr.y (mgr.14146) 1790 : cluster [DBG] pgmap v2611: 47 pgs: 3 active+recovering, 44 active+clean; 8.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 52 MiB/s wr, 838 op/s; 15630/278916 objects misplaced (5.604%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:43:59.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:59.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:43:59.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:59.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:43:59.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:59.625 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:43:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:59.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:43:59.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 83 16 143708 143692 108.191 64.625 0.0144511 0.00924153 2024-02-12T16:43:59.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:59.758 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:43:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:59.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:43:59.803 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:43:59.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:43:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:43:59.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:43:59.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:00.043 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:00.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:00.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:00.277 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:00.297 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:00.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:00.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:00.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:00.563 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:00.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 84 16 144764 144748 107.689 66 0.00498551 0.00928408 2024-02-12T16:44:00.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:00.691 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:00.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:00.822 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:00.926 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:00.952 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:00.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:01.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:01.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:01 smithi169 bash[19604]: cluster 2024-02-12T16:43:59.982234+0000 osd.8 (osd.8) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:01.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:01 smithi169 bash[19604]: cluster 2024-02-12T16:43:59.982298+0000 osd.8 (osd.8) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:01.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:01 smithi169 bash[19604]: cluster 2024-02-12T16:44:00.009287+0000 mon.a (mon.0) 3179 : cluster [DBG] osdmap e952: 12 total, 12 up, 11 in 2024-02-12T16:44:01.063 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:01 smithi169 bash[19604]: cluster 2024-02-12T16:44:00.289582+0000 mgr.y (mgr.14146) 1791 : cluster [DBG] pgmap v2613: 47 pgs: 3 active+recovering, 44 active+clean; 8.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 52 MiB/s wr, 835 op/s; 15630/282736 objects misplaced (5.528%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:44:01.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:01.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:01.184 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:01.190 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:01 smithi123 bash[19119]: cluster 2024-02-12T16:43:59.982234+0000 osd.8 (osd.8) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:01.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:01 smithi123 bash[19119]: cluster 2024-02-12T16:43:59.982298+0000 osd.8 (osd.8) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:01.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:01 smithi123 bash[19119]: cluster 2024-02-12T16:44:00.009287+0000 mon.a (mon.0) 3179 : cluster [DBG] osdmap e952: 12 total, 12 up, 11 in 2024-02-12T16:44:01.191 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:01 smithi123 bash[19119]: cluster 2024-02-12T16:44:00.289582+0000 mgr.y (mgr.14146) 1791 : cluster [DBG] pgmap v2613: 47 pgs: 3 active+recovering, 44 active+clean; 8.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 52 MiB/s wr, 835 op/s; 15630/282736 objects misplaced (5.528%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:44:01.210 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:01.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:01.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:01.346 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:01.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:01 smithi121 bash[18489]: cluster 2024-02-12T16:43:59.982234+0000 osd.8 (osd.8) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:01 smithi121 bash[18489]: cluster 2024-02-12T16:43:59.982298+0000 osd.8 (osd.8) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:01 smithi121 bash[18489]: cluster 2024-02-12T16:44:00.009287+0000 mon.a (mon.0) 3179 : cluster [DBG] osdmap e952: 12 total, 12 up, 11 in 2024-02-12T16:44:01.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:01 smithi121 bash[18489]: cluster 2024-02-12T16:44:00.289582+0000 mgr.y (mgr.14146) 1791 : cluster [DBG] pgmap v2613: 47 pgs: 3 active+recovering, 44 active+clean; 8.6 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 52 MiB/s wr, 835 op/s; 15630/282736 objects misplaced (5.528%); 2.6 MiB/s, 41 objects/s recovering 2024-02-12T16:44:01.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:01.475 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:01.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:01.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:01.605 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:01.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:01.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 85 16 145760 145744 107.154 62.25 0.00508641 0.00932889 2024-02-12T16:44:01.708 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:01.738 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:01.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:01.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:01.875 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:01.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:01.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:02.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:02.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:02.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:02.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:02.281 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:02.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:02.415 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:02.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:02.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:02.557 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:02.651 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 86 16 146816 146800 106.676 66 0.0259868 0.00937169 2024-02-12T16:44:02.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:02.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:02.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:02.836 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:02.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:02.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:02.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:03.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:03.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:03.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:03.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:03.231 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:03.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:03.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:03.349 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:03 smithi123 bash[19119]: cluster 2024-02-12T16:44:02.290217+0000 mgr.y (mgr.14146) 1792 : cluster [DBG] pgmap v2614: 47 pgs: 3 active+recovering, 44 active+clean; 8.8 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 15372/289166 objects misplaced (5.316%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:03.378 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:03.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:03.419 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:03.422 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:03.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:03.490 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:03 smithi169 bash[19604]: cluster 2024-02-12T16:44:02.290217+0000 mgr.y (mgr.14146) 1792 : cluster [DBG] pgmap v2614: 47 pgs: 3 active+recovering, 44 active+clean; 8.8 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 15372/289166 objects misplaced (5.316%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:03.507 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:03.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:03.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:03.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:03.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:03.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 87 16 147847 147831 106.19 64.4375 0.0105744 0.00941543 2024-02-12T16:44:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:03 smithi121 bash[18489]: cluster 2024-02-12T16:44:02.290217+0000 mgr.y (mgr.14146) 1792 : cluster [DBG] pgmap v2614: 47 pgs: 3 active+recovering, 44 active+clean; 8.8 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 15372/289166 objects misplaced (5.316%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:03.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:03.777 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:03.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:03.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:03.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:03.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:04.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:04.036 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:04.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:04.116 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 10, 8, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:44:04.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:44:04.116 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2024-02-12T16:44:04.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:44:04.143 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:04.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:04.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:04.309 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:04.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:04.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:04.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:04.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:04.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:04.559 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:04.591 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:04.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:04.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 88 16 148699 148683 105.588 53.25 0.0578694 0.00946832 2024-02-12T16:44:04.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:04.729 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:04.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:04.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:04.872 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:04.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:04.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:05.020 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:05.123 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:05.154 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:05.221 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:44:05.221 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:44:05.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:05.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:05.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:05.406 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:05 smithi121 bash[18489]: cluster 2024-02-12T16:44:04.290808+0000 mgr.y (mgr.14146) 1793 : cluster [DBG] pgmap v2615: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s; 15372/293094 objects misplaced (5.245%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T16:44:05.431 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:05.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:05 smithi169 bash[19604]: cluster 2024-02-12T16:44:04.290808+0000 mgr.y (mgr.14146) 1793 : cluster [DBG] pgmap v2615: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s; 15372/293094 objects misplaced (5.245%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T16:44:05.567 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:05 smithi123 bash[19119]: cluster 2024-02-12T16:44:04.290808+0000 mgr.y (mgr.14146) 1793 : cluster [DBG] pgmap v2615: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s; 15372/293094 objects misplaced (5.245%); 3.5 MiB/s, 56 objects/s recovering 2024-02-12T16:44:05.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 89 16 149719 149703 105.118 63.75 0.00840406 0.00951066 2024-02-12T16:44:05.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:05.703 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:05.807 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:05.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:05.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:05.962 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:05.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:05.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:44:06.064 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:06.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:06.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:06.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:06.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:06.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:06.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:06.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:06.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:06.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:06.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:06.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:06.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:06.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:06.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:06.652 INFO:tasks.radosbench.radosbench.2.smithi169.stdout: 90 15 150681 150666 104.619 60.1875 0.00515496 0.00955582 2024-02-12T16:44:06.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:06.768 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:06.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:06.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:06.903 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:06.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:07.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:07.038 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:07.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:07.152 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:06 smithi169 bash[19604]: cluster 2024-02-12T16:44:06.291330+0000 mgr.y (mgr.14146) 1794 : cluster [DBG] pgmap v2616: 47 pgs: 3 active+recovering, 44 active+clean; 9.0 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 15314/294296 objects misplaced (5.204%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:44:07.172 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:07.260 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:06 smithi121 bash[18489]: cluster 2024-02-12T16:44:06.291330+0000 mgr.y (mgr.14146) 1794 : cluster [DBG] pgmap v2616: 47 pgs: 3 active+recovering, 44 active+clean; 9.0 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 15314/294296 objects misplaced (5.204%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:44:07.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:07.310 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:06 smithi123 bash[19119]: cluster 2024-02-12T16:44:06.291330+0000 mgr.y (mgr.14146) 1794 : cluster [DBG] pgmap v2616: 47 pgs: 3 active+recovering, 44 active+clean; 9.0 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 15314/294296 objects misplaced (5.204%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:44:07.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:07.448 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:07.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:07.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total time run: 90.021886 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Total writes made: 150681 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Write size: 65536 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Object size: 65536 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Bandwidth (MB/sec): 104.614 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Bandwidth: 65.8175 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max bandwidth (MB/sec): 364.375 2024-02-12T16:44:07.653 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min bandwidth (MB/sec): 0 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average IOPS: 1673 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev IOPS: 1053 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max IOPS: 5830 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min IOPS: 0 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Average Latency(s): 0.00955783 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Stddev Latency(s): 0.047298 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Max latency(s): 4.35085 2024-02-12T16:44:07.654 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Min latency(s): 0.000995729 2024-02-12T16:44:07.663 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Cleaning up (deleting benchmark objects) 2024-02-12T16:44:07.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:07.710 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:07.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:07.843 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:07.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:07.971 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:07.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:08.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:08.102 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:08.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:08.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:08.234 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:08.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:08.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:08.371 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:08.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:08.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:08.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:08.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:08.640 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:08.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:08.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:08.835 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:08.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:08.938 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:08.964 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:08.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:09.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:09.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:09.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:09.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:09.232 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:09.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:09.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:09.382 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:09.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:09.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:09.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:09.495 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:09 smithi169 bash[19604]: cluster 2024-02-12T16:44:08.291768+0000 mgr.y (mgr.14146) 1795 : cluster [DBG] pgmap v2617: 47 pgs: 3 active+recovering, 44 active+clean; 9.1 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 15118/299060 objects misplaced (5.055%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:09.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:09.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:09.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:09.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:09 smithi123 bash[19119]: cluster 2024-02-12T16:44:08.291768+0000 mgr.y (mgr.14146) 1795 : cluster [DBG] pgmap v2617: 47 pgs: 3 active+recovering, 44 active+clean; 9.1 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 15118/299060 objects misplaced (5.055%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:09.652 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:09.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:09.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:09 smithi121 bash[18489]: cluster 2024-02-12T16:44:08.291768+0000 mgr.y (mgr.14146) 1795 : cluster [DBG] pgmap v2617: 47 pgs: 3 active+recovering, 44 active+clean; 9.1 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 15118/299060 objects misplaced (5.055%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:09.757 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:09.783 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:09.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:09.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:09.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:09.923 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:09.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:10.030 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:10.062 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:10.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:10.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:10.201 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:10.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:10.305 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:10.330 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:10.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:10.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:10.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:10.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:10.574 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:10.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:10.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:10.717 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:10.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:10.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:10.788 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:44:10.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:10.893 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:10.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:10.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:11.022 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:11.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:11.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:11.166 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:11.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:11.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:11.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:11.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:11.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:11 smithi123 bash[19119]: cluster 2024-02-12T16:44:10.292305+0000 mgr.y (mgr.14146) 1796 : cluster [DBG] pgmap v2618: 47 pgs: 3 active+recovering, 44 active+clean; 9.2 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 99 B/s rd, 56 MiB/s wr, 932 op/s; 15118/300090 objects misplaced (5.038%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:44:11.398 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:11.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:11 smithi121 bash[18489]: cluster 2024-02-12T16:44:10.292305+0000 mgr.y (mgr.14146) 1796 : cluster [DBG] pgmap v2618: 47 pgs: 3 active+recovering, 44 active+clean; 9.2 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 99 B/s rd, 56 MiB/s wr, 932 op/s; 15118/300090 objects misplaced (5.038%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:44:11.428 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:11.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:11.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:11.561 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:11.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:11.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:11.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:11.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:11 smithi169 bash[19604]: cluster 2024-02-12T16:44:10.292305+0000 mgr.y (mgr.14146) 1796 : cluster [DBG] pgmap v2618: 47 pgs: 3 active+recovering, 44 active+clean; 9.2 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 99 B/s rd, 56 MiB/s wr, 932 op/s; 15118/300090 objects misplaced (5.038%); 3.1 MiB/s, 49 objects/s recovering 2024-02-12T16:44:11.827 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:11.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:11.930 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:12.160 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:12.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:12.265 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:12.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:12.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:12.432 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:12.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:12.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:12.573 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:12.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:12.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:12.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:12.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:12.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:12.808 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:12.842 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:12.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:12.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:12.975 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:12.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:13.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:13.104 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:13.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:13.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:13.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:13.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:13.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:13.382 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:44:13.390 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:13.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:13.476 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:13 smithi121 bash[18489]: cluster 2024-02-12T16:44:12.292857+0000 mgr.y (mgr.14146) 1797 : cluster [DBG] pgmap v2619: 47 pgs: 3 active+recovering, 44 active+clean; 9.0 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 85 B/s rd, 48 MiB/s wr, 1.01k op/s; 14842/295706 objects misplaced (5.019%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:44:13.476 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:13 smithi121 bash[18489]: audit 2024-02-12T16:44:12.866099+0000 mon.a (mon.0) 3180 : audit [INF] from='client.27101 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:44:13.476 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:13 smithi121 bash[18489]: audit 2024-02-12T16:44:12.868819+0000 mon.b (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.121:0/244428179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:44:13.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:13.518 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:13.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:13.548 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:44:13.576 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:44:13.602 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:44:13.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:13.628 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:44:13.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:13 smithi123 bash[19119]: cluster 2024-02-12T16:44:12.292857+0000 mgr.y (mgr.14146) 1797 : cluster [DBG] pgmap v2619: 47 pgs: 3 active+recovering, 44 active+clean; 9.0 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 85 B/s rd, 48 MiB/s wr, 1.01k op/s; 14842/295706 objects misplaced (5.019%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:44:13.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:13 smithi123 bash[19119]: audit 2024-02-12T16:44:12.866099+0000 mon.a (mon.0) 3180 : audit [INF] from='client.27101 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:44:13.632 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:13 smithi123 bash[19119]: audit 2024-02-12T16:44:12.868819+0000 mon.b (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.121:0/244428179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:44:13.657 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:13.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:44:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:13.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:13.693 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:44:13.721 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:44:13.748 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:44:13.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:13.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:13 smithi169 bash[19604]: cluster 2024-02-12T16:44:12.292857+0000 mgr.y (mgr.14146) 1797 : cluster [DBG] pgmap v2619: 47 pgs: 3 active+recovering, 44 active+clean; 9.0 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 85 B/s rd, 48 MiB/s wr, 1.01k op/s; 14842/295706 objects misplaced (5.019%); 4.1 MiB/s, 65 objects/s recovering 2024-02-12T16:44:13.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:13 smithi169 bash[19604]: audit 2024-02-12T16:44:12.866099+0000 mon.a (mon.0) 3180 : audit [INF] from='client.27101 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:44:13.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:13 smithi169 bash[19604]: audit 2024-02-12T16:44:12.868819+0000 mon.b (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.121:0/244428179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:44:13.782 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:44:13.791 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:13.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:13.806 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:44:13.829 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:44:13.847 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:44:13.849 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.849 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.849 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.850 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:44:13.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:13.942 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:13.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:14.047 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:14.076 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:14.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:14.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:14.210 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:14.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:14.313 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:14.347 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:14.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:14.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:14.478 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:14.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:14.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:14.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:14 smithi123 bash[19119]: cluster 2024-02-12T16:44:13.349332+0000 mon.a (mon.0) 3181 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:14.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:14 smithi123 bash[19119]: audit 2024-02-12T16:44:13.365973+0000 mon.a (mon.0) 3182 : audit [INF] from='client.27101 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:44:14.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:14 smithi123 bash[19119]: cluster 2024-02-12T16:44:13.366041+0000 mon.a (mon.0) 3183 : cluster [DBG] osdmap e953: 12 total, 12 up, 11 in 2024-02-12T16:44:14.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:14 smithi123 bash[19119]: audit 2024-02-12T16:44:13.508183+0000 mon.a (mon.0) 3184 : audit [DBG] from='client.? v1:172.21.15.121:0/1302350463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:14.607 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:14.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:14 smithi121 bash[18489]: cluster 2024-02-12T16:44:13.349332+0000 mon.a (mon.0) 3181 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:14 smithi121 bash[18489]: audit 2024-02-12T16:44:13.365973+0000 mon.a (mon.0) 3182 : audit [INF] from='client.27101 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:44:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:14 smithi121 bash[18489]: cluster 2024-02-12T16:44:13.366041+0000 mon.a (mon.0) 3183 : cluster [DBG] osdmap e953: 12 total, 12 up, 11 in 2024-02-12T16:44:14.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:14 smithi121 bash[18489]: audit 2024-02-12T16:44:13.508183+0000 mon.a (mon.0) 3184 : audit [DBG] from='client.? v1:172.21.15.121:0/1302350463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:14.713 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:14.721 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:14 smithi169 bash[19604]: cluster 2024-02-12T16:44:13.349332+0000 mon.a (mon.0) 3181 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:14.721 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:14 smithi169 bash[19604]: audit 2024-02-12T16:44:13.365973+0000 mon.a (mon.0) 3182 : audit [INF] from='client.27101 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:44:14.721 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:14 smithi169 bash[19604]: cluster 2024-02-12T16:44:13.366041+0000 mon.a (mon.0) 3183 : cluster [DBG] osdmap e953: 12 total, 12 up, 11 in 2024-02-12T16:44:14.721 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:14 smithi169 bash[19604]: audit 2024-02-12T16:44:13.508183+0000 mon.a (mon.0) 3184 : audit [DBG] from='client.? v1:172.21.15.121:0/1302350463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:14.742 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:14.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:14.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:14.878 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:14.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:14.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:15.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:15.059 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:44:15.059 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:44:15.059 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2024-02-12T16:44:15.059 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2024-02-12T16:44:15.059 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:44:15.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:15.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:15.242 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:15.270 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:15.379 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:15.406 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:15.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:15.541 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:15 smithi123 bash[19119]: cluster 2024-02-12T16:44:14.293298+0000 mgr.y (mgr.14146) 1798 : cluster [DBG] pgmap v2621: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 26 MiB/s wr, 871 op/s; 14842/292194 objects misplaced (5.080%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:44:15.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:15.653 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:15 smithi169 bash[19604]: cluster 2024-02-12T16:44:14.293298+0000 mgr.y (mgr.14146) 1798 : cluster [DBG] pgmap v2621: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 26 MiB/s wr, 871 op/s; 14842/292194 objects misplaced (5.080%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:44:15.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:15 smithi121 bash[18489]: cluster 2024-02-12T16:44:14.293298+0000 mgr.y (mgr.14146) 1798 : cluster [DBG] pgmap v2621: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 26 MiB/s wr, 871 op/s; 14842/292194 objects misplaced (5.080%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:44:15.681 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:15.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:15.810 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:15.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:44:15.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:15.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:15.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:16.043 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:16.070 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:16.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:16.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:16.205 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:16.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:16.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:16.335 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:16.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:16.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:16.467 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:16.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:16.570 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:16.595 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:16.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:16.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:16.727 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:16.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:16.830 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:16.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:16.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:16.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:17.000 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:17.003 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:17 smithi169 bash[19604]: cluster 2024-02-12T16:44:16.293717+0000 mgr.y (mgr.14146) 1799 : cluster [DBG] pgmap v2622: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 22 MiB/s wr, 872 op/s; 14785/290970 objects misplaced (5.081%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:44:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:17.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:17.117 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:17 smithi123 bash[19119]: cluster 2024-02-12T16:44:16.293717+0000 mgr.y (mgr.14146) 1799 : cluster [DBG] pgmap v2622: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 22 MiB/s wr, 872 op/s; 14785/290970 objects misplaced (5.081%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:44:17.134 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:17.237 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:17.264 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:17.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:16 smithi121 bash[18489]: cluster 2024-02-12T16:44:16.293717+0000 mgr.y (mgr.14146) 1799 : cluster [DBG] pgmap v2622: 47 pgs: 3 active+recovering, 44 active+clean; 8.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 22 MiB/s wr, 872 op/s; 14785/290970 objects misplaced (5.081%); 3.3 MiB/s, 52 objects/s recovering 2024-02-12T16:44:17.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:17.396 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:17.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:17.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:17.536 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:17.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:17.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:17.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:17.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:44:17.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:17.814 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:17.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:44:17.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:17.941 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:17.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:18.044 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:18.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:18.189 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:44:18.190 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:44:18.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:18.217 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:18.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:18.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:18.355 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:18.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:18.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:18.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:18.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:18.599 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:18.624 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:18.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:18.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:18.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:18.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:18.898 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:18.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:19.004 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:19.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:19.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:19.180 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:19.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:19.185 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2622,"stamp":"2024-02-12T16:44:16.293474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9534439448,"num_objects":145485,"num_object_clones":0,"num_object_copies":290970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14785,"num_objects_unfound":0,"num_objects_dirty":145485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155879,"num_write_kb":9643585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2784,"num_bytes_recovered":182452224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155879,"ondisk_log_size":155879,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":25837992,"kb_used_data":17731760,"kb_used_omap":92,"kb_used_meta":8104547,"kb_avail":1005358680,"statfs":{"total":1055945392128,"available":1029487288320,"internally_reserved":0,"allocated":18157322240,"data_stored":18149073971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95164,"internal_metadata":8299056196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-109051880,"num_objects":-1663,"num_object_clones":0,"num_object_copies":-3326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-529,"num_objects_unfound":0,"num_objects_dirty":-1663,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8731,"num_write_kb":226113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":529,"num_bytes_recovered":34668544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002462"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":317,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.017658+0000","last_change":"2024-02-12T16:42:21.631979+0000","last_active":"2024-02-12T16:44:00.017658+0000","last_peered":"2024-02-12T16:44:00.017658+0000","last_clean":"2024-02-12T16:44:00.017658+0000","last_became_active":"2024-02-12T16:41:18.947003+0000","last_became_peered":"2024-02-12T16:41:18.947003+0000","last_unstale":"2024-02-12T16:44:00.017658+0000","last_undegraded":"2024-02-12T16:44:00.017658+0000","last_fullsized":"2024-02-12T16:44:00.017658+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:21.631859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:21.631859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":344,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:57.998977+0000","last_change":"2024-02-12T16:42:29.489386+0000","last_active":"2024-02-12T16:43:57.998977+0000","last_peered":"2024-02-12T16:43:57.998977+0000","last_clean":"2024-02-12T16:43:57.998977+0000","last_became_active":"2024-02-12T16:41:17.943883+0000","last_became_peered":"2024-02-12T16:41:17.943883+0000","last_unstale":"2024-02-12T16:43:57.998977+0000","last_undegraded":"2024-02-12T16:43:57.998977+0000","last_fullsized":"2024-02-12T16:43:57.998977+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:29.489294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:29.489294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":330,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.020323+0000","last_change":"2024-02-12T16:42:51.684068+0000","last_active":"2024-02-12T16:44:00.020323+0000","last_peered":"2024-02-12T16:44:00.020323+0000","last_clean":"2024-02-12T16:44:00.020323+0000","last_became_active":"2024-02-12T16:30:55.405213+0000","last_became_peered":"2024-02-12T16:30:55.405213+0000","last_unstale":"2024-02-12T16:44:00.020323+0000","last_undegraded":"2024-02-12T16:44:00.020323+0000","last_fullsized":"2024-02-12T16:44:00.020323+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.683932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.683932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":147,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:10.958086+0000","last_change":"2024-02-12T16:44:10.958086+0000","last_active":"2024-02-12T16:44:10.958086+0000","last_peered":"2024-02-12T16:44:10.958086+0000","last_clean":"2024-02-12T16:44:10.958086+0000","last_became_active":"2024-02-12T16:39:09.436840+0000","last_became_peered":"2024-02-12T16:39:09.436840+0000","last_unstale":"2024-02-12T16:44:10.958086+0000","last_undegraded":"2024-02-12T16:44:10.958086+0000","last_fullsized":"2024-02-12T16:44:10.958086+0000","mapping_epoch":845,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":846,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_clean_scrub_stamp":"2024-02-12T16:41:21.337430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":382,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.011786+0000","last_change":"2024-02-12T16:42:41.310928+0000","last_active":"2024-02-12T16:44:00.011786+0000","last_peered":"2024-02-12T16:44:00.011786+0000","last_clean":"2024-02-12T16:44:00.011786+0000","last_became_active":"2024-02-12T16:42:40.975420+0000","last_became_peered":"2024-02-12T16:42:40.975420+0000","last_unstale":"2024-02-12T16:44:00.011786+0000","last_undegraded":"2024-02-12T16:44:00.011786+0000","last_fullsized":"2024-02-12T16:44:00.011786+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:41.310840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:41.310840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":343,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.019582+0000","last_change":"2024-02-12T16:42:24.921381+0000","last_active":"2024-02-12T16:44:00.019582+0000","last_peered":"2024-02-12T16:44:00.019582+0000","last_clean":"2024-02-12T16:44:00.019582+0000","last_became_active":"2024-02-12T16:36:17.847960+0000","last_became_peered":"2024-02-12T16:36:17.847960+0000","last_unstale":"2024-02-12T16:44:00.019582+0000","last_undegraded":"2024-02-12T16:44:00.019582+0000","last_fullsized":"2024-02-12T16:44:00.019582+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:24.921234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:24.921234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":686,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:58.012965+0000","last_change":"2024-02-12T16:42:31.552860+0000","last_active":"2024-02-12T16:43:58.012965+0000","last_peered":"2024-02-12T16:43:58.012965+0000","last_clean":"2024-02-12T16:43:58.012965+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:43:58.012965+0000","last_undegraded":"2024-02-12T16:43:58.012965+0000","last_fullsized":"2024-02-12T16:43:58.012965+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:31.552623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:42:31.552623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":334,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.020616+0000","last_change":"2024-02-12T16:43:31.617757+0000","last_active":"2024-02-12T16:44:00.020616+0000","last_peered":"2024-02-12T16:44:00.020616+0000","last_clean":"2024-02-12T16:44:00.020616+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:44:00.020616+0000","last_undegraded":"2024-02-12T16:44:00.020616+0000","last_fullsized":"2024-02-12T16:44:00.020616+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.617636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.617636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":328,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:58.001580+0000","last_change":"2024-02-12T16:42:26.378442+0000","last_active":"2024-02-12T16:43:58.001580+0000","last_peered":"2024-02-12T16:43:58.001580+0000","last_clean":"2024-02-12T16:43:58.001580+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:43:58.001580+0000","last_undegraded":"2024-02-12T16:43:58.001580+0000","last_fullsized":"2024-02-12T16:43:58.001580+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.378312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":284,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:13.220903+0000","last_change":"2024-02-12T16:44:13.220903+0000","last_active":"2024-02-12T16:44:13.220903+0000","last_peered":"2024-02-12T16:44:13.220903+0000","last_clean":"2024-02-12T16:44:13.220903+0000","last_became_active":"2024-02-12T16:42:40.982346+0000","last_became_peered":"2024-02-12T16:42:40.982346+0000","last_unstale":"2024-02-12T16:44:13.220903+0000","last_undegraded":"2024-02-12T16:44:13.220903+0000","last_fullsized":"2024-02-12T16:44:13.220903+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:11.207557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:39:11.207557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":589,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:58.078781+0000","last_change":"2024-02-12T16:43:57.837069+0000","last_active":"2024-02-12T16:43:58.078781+0000","last_peered":"2024-02-12T16:43:58.078781+0000","last_clean":"2024-02-12T16:43:58.078781+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:43:58.078781+0000","last_undegraded":"2024-02-12T16:43:58.078781+0000","last_fullsized":"2024-02-12T16:43:58.078781+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:57.836984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:43:57.836984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"953'9786","reported_seq":11204,"reported_epoch":953,"state":"active+recovering","last_fresh":"2024-02-12T16:44:15.339707+0000","last_change":"2024-02-12T16:43:16.238278+0000","last_active":"2024-02-12T16:44:15.339707+0000","last_peered":"2024-02-12T16:44:15.339707+0000","last_clean":"2024-02-12T16:43:15.331170+0000","last_became_active":"2024-02-12T16:43:16.169213+0000","last_became_peered":"2024-02-12T16:43:16.169213+0000","last_unstale":"2024-02-12T16:44:15.339707+0000","last_undegraded":"2024-02-12T16:44:15.339707+0000","last_fullsized":"2024-02-12T16:44:15.339707+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9786,"ondisk_log_size":9786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578682880,"num_objects":8830,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5159,"num_objects_unfound":0,"num_objects_dirty":8830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":595712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":44892160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","9"],"object_location_counts":[{"shards":"5,11","objects":3671},{"shards":"9,11","objects":5159}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.019748+0000","last_change":"2024-02-12T16:42:41.980681+0000","last_active":"2024-02-12T16:44:00.019748+0000","last_peered":"2024-02-12T16:44:00.019748+0000","last_clean":"2024-02-12T16:44:00.019748+0000","last_became_active":"2024-02-12T16:42:41.980413+0000","last_became_peered":"2024-02-12T16:42:41.980413+0000","last_unstale":"2024-02-12T16:44:00.019748+0000","last_undegraded":"2024-02-12T16:44:00.019748+0000","last_fullsized":"2024-02-12T16:44:00.019748+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.611860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.611860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"953'9723","reported_seq":9761,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-02-12T16:44:14.172366+0000","last_change":"2024-02-12T16:42:40.991625+0000","last_active":"2024-02-12T16:44:14.172366+0000","last_peered":"2024-02-12T16:44:14.172366+0000","last_clean":"2024-02-12T16:44:14.172366+0000","last_became_active":"2024-02-12T16:42:40.984344+0000","last_became_peered":"2024-02-12T16:42:40.984344+0000","last_unstale":"2024-02-12T16:44:14.172366+0000","last_undegraded":"2024-02-12T16:44:14.172366+0000","last_fullsized":"2024-02-12T16:44:14.172366+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9723,"ondisk_log_size":9723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":587726872,"num_objects":8969,"num_object_clones":0,"num_object_copies":17938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8969,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9723,"num_write_kb":598081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":698,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.019410+0000","last_change":"2024-02-12T16:41:25.248527+0000","last_active":"2024-02-12T16:44:00.019410+0000","last_peered":"2024-02-12T16:44:00.019410+0000","last_clean":"2024-02-12T16:44:00.019410+0000","last_became_active":"2024-02-12T16:36:17.846631+0000","last_became_peered":"2024-02-12T16:36:17.846631+0000","last_unstale":"2024-02-12T16:44:00.019410+0000","last_undegraded":"2024-02-12T16:44:00.019410+0000","last_fullsized":"2024-02-12T16:44:00.019410+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:25.248423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:41:25.248423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"952'9663","reported_seq":9706,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:11.516600+0000","last_change":"2024-02-12T16:42:21.999716+0000","last_active":"2024-02-12T16:44:11.516600+0000","last_peered":"2024-02-12T16:44:11.516600+0000","last_clean":"2024-02-12T16:44:11.516600+0000","last_became_active":"2024-02-12T16:42:21.999181+0000","last_became_peered":"2024-02-12T16:42:21.999181+0000","last_unstale":"2024-02-12T16:44:11.516600+0000","last_undegraded":"2024-02-12T16:44:11.516600+0000","last_fullsized":"2024-02-12T16:44:11.516600+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9663,"ondisk_log_size":9663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":604045312,"num_objects":9217,"num_object_clones":0,"num_object_copies":18434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":604160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1064,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.017582+0000","last_change":"2024-02-12T16:42:38.439762+0000","last_active":"2024-02-12T16:44:00.017582+0000","last_peered":"2024-02-12T16:44:00.017582+0000","last_clean":"2024-02-12T16:44:00.017582+0000","last_became_active":"2024-02-12T16:41:18.947424+0000","last_became_peered":"2024-02-12T16:41:18.947424+0000","last_unstale":"2024-02-12T16:44:00.017582+0000","last_undegraded":"2024-02-12T16:44:00.017582+0000","last_fullsized":"2024-02-12T16:44:00.017582+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:38.439663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:39:15.947507+0000","last_clean_scrub_stamp":"2024-02-12T16:42:38.439663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.f","version":"952'9788","reported_seq":9831,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:11.993770+0000","last_change":"2024-02-12T16:42:21.964154+0000","last_active":"2024-02-12T16:44:11.993770+0000","last_peered":"2024-02-12T16:44:11.993770+0000","last_clean":"2024-02-12T16:44:11.993770+0000","last_became_active":"2024-02-12T16:42:21.963855+0000","last_became_peered":"2024-02-12T16:42:21.963855+0000","last_unstale":"2024-02-12T16:44:11.993770+0000","last_undegraded":"2024-02-12T16:44:11.993770+0000","last_fullsized":"2024-02-12T16:44:11.993770+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9788,"ondisk_log_size":9788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":603848704,"num_objects":9214,"num_object_clones":0,"num_object_copies":18428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9788,"num_write_kb":608064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":736,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.016194+0000","last_change":"2024-02-12T16:42:23.543471+0000","last_active":"2024-02-12T16:44:00.016194+0000","last_peered":"2024-02-12T16:44:00.016194+0000","last_clean":"2024-02-12T16:44:00.016194+0000","last_became_active":"2024-02-12T16:30:49.379666+0000","last_became_peered":"2024-02-12T16:30:49.379666+0000","last_unstale":"2024-02-12T16:44:00.016194+0000","last_undegraded":"2024-02-12T16:44:00.016194+0000","last_fullsized":"2024-02-12T16:44:00.016194+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.543386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.543386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"953'9863","reported_seq":9900,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-02-12T16:44:14.158427+0000","last_change":"2024-02-12T16:42:40.974589+0000","last_active":"2024-02-12T16:44:14.158427+0000","last_peered":"2024-02-12T16:44:14.158427+0000","last_clean":"2024-02-12T16:44:14.158427+0000","last_became_active":"2024-02-12T16:42:40.971979+0000","last_became_peered":"2024-02-12T16:42:40.971979+0000","last_unstale":"2024-02-12T16:44:14.158427+0000","last_undegraded":"2024-02-12T16:44:14.158427+0000","last_fullsized":"2024-02-12T16:44:14.158427+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9863,"ondisk_log_size":9863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":598409216,"num_objects":9131,"num_object_clones":0,"num_object_copies":18262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9863,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":690,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.020522+0000","last_change":"2024-02-12T16:43:15.977479+0000","last_active":"2024-02-12T16:44:00.020522+0000","last_peered":"2024-02-12T16:44:00.020522+0000","last_clean":"2024-02-12T16:44:00.020522+0000","last_became_active":"2024-02-12T16:43:15.976997+0000","last_became_peered":"2024-02-12T16:43:15.976997+0000","last_unstale":"2024-02-12T16:44:00.020522+0000","last_undegraded":"2024-02-12T16:44:00.020522+0000","last_fullsized":"2024-02-12T16:44:00.020522+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.923206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:57.057484+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.923206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"953'9844","reported_seq":9888,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-02-12T16:44:13.782546+0000","last_change":"2024-02-12T16:42:21.962569+0000","last_active":"2024-02-12T16:44:13.782546+0000","last_peered":"2024-02-12T16:44:13.782546+0000","last_clean":"2024-02-12T16:44:13.782546+0000","last_became_active":"2024-02-12T16:42:21.962130+0000","last_became_peered":"2024-02-12T16:42:21.962130+0000","last_unstale":"2024-02-12T16:44:13.782546+0000","last_undegraded":"2024-02-12T16:44:13.782546+0000","last_fullsized":"2024-02-12T16:44:13.782546+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9844,"ondisk_log_size":9844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":595722240,"num_objects":9090,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":605888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":733,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.019509+0000","last_change":"2024-02-12T16:42:34.813265+0000","last_active":"2024-02-12T16:44:00.019509+0000","last_peered":"2024-02-12T16:44:00.019509+0000","last_clean":"2024-02-12T16:44:00.019509+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:44:00.019509+0000","last_undegraded":"2024-02-12T16:44:00.019509+0000","last_fullsized":"2024-02-12T16:44:00.019509+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:34.813152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:42:34.813152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"953'9605","reported_seq":9654,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-02-12T16:44:13.815371+0000","last_change":"2024-02-12T16:43:30.452610+0000","last_active":"2024-02-12T16:44:13.815371+0000","last_peered":"2024-02-12T16:44:13.815371+0000","last_clean":"2024-02-12T16:44:13.815371+0000","last_became_active":"2024-02-12T16:42:21.963637+0000","last_became_peered":"2024-02-12T16:42:21.963637+0000","last_unstale":"2024-02-12T16:44:13.815371+0000","last_undegraded":"2024-02-12T16:44:13.815371+0000","last_fullsized":"2024-02-12T16:44:13.815371+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"940'6865","last_scrub_stamp":"2024-02-12T16:43:30.452522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:43:30.452522+0000","log_size":9605,"ondisk_log_size":9605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":584646656,"num_objects":8921,"num_object_clones":0,"num_object_copies":17842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9605,"num_write_kb":592832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":537,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:58.002163+0000","last_change":"2024-02-12T16:41:33.930537+0000","last_active":"2024-02-12T16:43:58.002163+0000","last_peered":"2024-02-12T16:43:58.002163+0000","last_clean":"2024-02-12T16:43:58.002163+0000","last_became_active":"2024-02-12T16:23:02.144007+0000","last_became_peered":"2024-02-12T16:23:02.144007+0000","last_unstale":"2024-02-12T16:43:58.002163+0000","last_undegraded":"2024-02-12T16:43:58.002163+0000","last_fullsized":"2024-02-12T16:43:58.002163+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:33.930337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:33.930337+0000","last_clean_scrub_stamp":"2024-02-12T16:41:33.930337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"952'9792","reported_seq":9835,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:12.015117+0000","last_change":"2024-02-12T16:42:21.964459+0000","last_active":"2024-02-12T16:44:12.015117+0000","last_peered":"2024-02-12T16:44:12.015117+0000","last_clean":"2024-02-12T16:44:12.015117+0000","last_became_active":"2024-02-12T16:42:21.964092+0000","last_became_peered":"2024-02-12T16:42:21.964092+0000","last_unstale":"2024-02-12T16:44:12.015117+0000","last_undegraded":"2024-02-12T16:44:12.015117+0000","last_fullsized":"2024-02-12T16:44:12.015117+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9792,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":606208000,"num_objects":9250,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":609344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1026,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.019889+0000","last_change":"2024-02-12T16:43:31.016133+0000","last_active":"2024-02-12T16:44:00.019889+0000","last_peered":"2024-02-12T16:44:00.019889+0000","last_clean":"2024-02-12T16:44:00.019889+0000","last_became_active":"2024-02-12T16:42:41.321180+0000","last_became_peered":"2024-02-12T16:42:41.321180+0000","last_unstale":"2024-02-12T16:44:00.019889+0000","last_undegraded":"2024-02-12T16:44:00.019889+0000","last_fullsized":"2024-02-12T16:44:00.019889+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.016013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.016013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"953'9925","reported_seq":9969,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-02-12T16:44:15.290947+0000","last_change":"2024-02-12T16:42:21.961833+0000","last_active":"2024-02-12T16:44:15.290947+0000","last_peered":"2024-02-12T16:44:15.290947+0000","last_clean":"2024-02-12T16:44:15.290947+0000","last_became_active":"2024-02-12T16:42:21.961543+0000","last_became_peered":"2024-02-12T16:42:21.961543+0000","last_unstale":"2024-02-12T16:44:15.290947+0000","last_undegraded":"2024-02-12T16:44:15.290947+0000","last_fullsized":"2024-02-12T16:44:15.290947+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9925,"ondisk_log_size":9925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590413824,"num_objects":9009,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9925,"num_write_kb":605888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":445,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.020201+0000","last_change":"2024-02-12T16:42:43.033723+0000","last_active":"2024-02-12T16:44:00.020201+0000","last_peered":"2024-02-12T16:44:00.020201+0000","last_clean":"2024-02-12T16:44:00.020201+0000","last_became_active":"2024-02-12T16:42:40.975171+0000","last_became_peered":"2024-02-12T16:42:40.975171+0000","last_unstale":"2024-02-12T16:44:00.020201+0000","last_undegraded":"2024-02-12T16:44:00.020201+0000","last_fullsized":"2024-02-12T16:44:00.020201+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.033576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.033576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"952'9751","reported_seq":9794,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:11.708289+0000","last_change":"2024-02-12T16:42:21.999246+0000","last_active":"2024-02-12T16:44:11.708289+0000","last_peered":"2024-02-12T16:44:11.708289+0000","last_clean":"2024-02-12T16:44:11.708289+0000","last_became_active":"2024-02-12T16:42:21.998975+0000","last_became_peered":"2024-02-12T16:42:21.998975+0000","last_unstale":"2024-02-12T16:44:11.708289+0000","last_undegraded":"2024-02-12T16:44:11.708289+0000","last_fullsized":"2024-02-12T16:44:11.708289+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9751,"ondisk_log_size":9751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":604176384,"num_objects":9219,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9751,"num_write_kb":607040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":266,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:07.680002+0000","last_change":"2024-02-12T16:44:07.680516+0000","last_active":"2024-02-12T16:44:07.680002+0000","last_peered":"2024-02-12T16:44:07.680002+0000","last_clean":"2024-02-12T16:44:07.680002+0000","last_became_active":"2024-02-12T16:42:10.950266+0000","last_became_peered":"2024-02-12T16:42:10.950266+0000","last_unstale":"2024-02-12T16:44:07.680002+0000","last_undegraded":"2024-02-12T16:44:07.680002+0000","last_fullsized":"2024-02-12T16:44:07.680002+0000","mapping_epoch":913,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":914,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:22.478963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:22.478963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"953'9978","reported_seq":10027,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-02-12T16:44:13.809190+0000","last_change":"2024-02-12T16:42:40.974611+0000","last_active":"2024-02-12T16:44:13.809190+0000","last_peered":"2024-02-12T16:44:13.809190+0000","last_clean":"2024-02-12T16:44:13.809190+0000","last_became_active":"2024-02-12T16:42:40.973291+0000","last_became_peered":"2024-02-12T16:42:40.973291+0000","last_unstale":"2024-02-12T16:44:13.809190+0000","last_undegraded":"2024-02-12T16:44:13.809190+0000","last_fullsized":"2024-02-12T16:44:13.809190+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9978,"ondisk_log_size":9978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":603717632,"num_objects":9212,"num_object_clones":0,"num_object_copies":18424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9978,"num_write_kb":614080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":687,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.016302+0000","last_change":"2024-02-12T16:42:51.559357+0000","last_active":"2024-02-12T16:44:00.016302+0000","last_peered":"2024-02-12T16:44:00.016302+0000","last_clean":"2024-02-12T16:44:00.016302+0000","last_became_active":"2024-02-12T16:23:39.022915+0000","last_became_peered":"2024-02-12T16:23:39.022915+0000","last_unstale":"2024-02-12T16:44:00.016302+0000","last_undegraded":"2024-02-12T16:44:00.016302+0000","last_fullsized":"2024-02-12T16:44:00.016302+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.559264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"952'9624","reported_seq":9667,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:11.998573+0000","last_change":"2024-02-12T16:42:21.964135+0000","last_active":"2024-02-12T16:44:11.998573+0000","last_peered":"2024-02-12T16:44:11.998573+0000","last_clean":"2024-02-12T16:44:11.998573+0000","last_became_active":"2024-02-12T16:42:21.963899+0000","last_became_peered":"2024-02-12T16:42:21.963899+0000","last_unstale":"2024-02-12T16:44:11.998573+0000","last_undegraded":"2024-02-12T16:44:11.998573+0000","last_fullsized":"2024-02-12T16:44:11.998573+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9624,"ondisk_log_size":9624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":596639744,"num_objects":9104,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":599296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":498,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.011897+0000","last_change":"2024-02-12T16:42:42.373090+0000","last_active":"2024-02-12T16:44:00.011897+0000","last_peered":"2024-02-12T16:44:00.011897+0000","last_clean":"2024-02-12T16:44:00.011897+0000","last_became_active":"2024-02-12T16:42:40.975493+0000","last_became_peered":"2024-02-12T16:42:40.975493+0000","last_unstale":"2024-02-12T16:44:00.011897+0000","last_undegraded":"2024-02-12T16:44:00.011897+0000","last_fullsized":"2024-02-12T16:44:00.011897+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_clean_scrub_stamp":"2024-02-12T16:42:42.373026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"952'9443","reported_seq":12426,"reported_epoch":952,"state":"active+recovering","last_fresh":"2024-02-12T16:44:11.532904+0000","last_change":"2024-02-12T16:43:16.215367+0000","last_active":"2024-02-12T16:44:11.532904+0000","last_peered":"2024-02-12T16:44:11.532904+0000","last_clean":"2024-02-12T16:43:15.351398+0000","last_became_active":"2024-02-12T16:43:16.158964+0000","last_became_peered":"2024-02-12T16:43:16.158964+0000","last_unstale":"2024-02-12T16:44:11.532904+0000","last_undegraded":"2024-02-12T16:44:11.532904+0000","last_fullsized":"2024-02-12T16:44:11.532904+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9443,"ondisk_log_size":9443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":587661312,"num_objects":8967,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4338,"num_objects_unfound":0,"num_objects_dirty":8967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9443,"num_write_kb":589120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":96141312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","9"],"object_location_counts":[{"shards":"3,6","objects":4629},{"shards":"3,9","objects":4338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":1013,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.019742+0000","last_change":"2024-02-12T16:43:16.350572+0000","last_active":"2024-02-12T16:44:00.019742+0000","last_peered":"2024-02-12T16:44:00.019742+0000","last_clean":"2024-02-12T16:44:00.019742+0000","last_became_active":"2024-02-12T16:43:16.350353+0000","last_became_peered":"2024-02-12T16:43:16.350353+0000","last_unstale":"2024-02-12T16:44:00.019742+0000","last_undegraded":"2024-02-12T16:44:00.019742+0000","last_fullsized":"2024-02-12T16:44:00.019742+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_clean_scrub_stamp":"2024-02-12T16:42:48.099448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"952'9633","reported_seq":9676,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:11.731234+0000","last_change":"2024-02-12T16:42:21.959286+0000","last_active":"2024-02-12T16:44:11.731234+0000","last_peered":"2024-02-12T16:44:11.731234+0000","last_clean":"2024-02-12T16:44:11.731234+0000","last_became_active":"2024-02-12T16:42:21.959009+0000","last_became_peered":"2024-02-12T16:42:21.959009+0000","last_unstale":"2024-02-12T16:44:11.731234+0000","last_undegraded":"2024-02-12T16:44:11.731234+0000","last_fullsized":"2024-02-12T16:44:11.731234+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9633,"ondisk_log_size":9633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":598802432,"num_objects":9137,"num_object_clones":0,"num_object_copies":18274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9633,"num_write_kb":600640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":680,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:58.013029+0000","last_change":"2024-02-12T16:42:23.507674+0000","last_active":"2024-02-12T16:43:58.013029+0000","last_peered":"2024-02-12T16:43:58.013029+0000","last_clean":"2024-02-12T16:43:58.013029+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:43:58.013029+0000","last_undegraded":"2024-02-12T16:43:58.013029+0000","last_fullsized":"2024-02-12T16:43:58.013029+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.507539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.507539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"953'9814","reported_seq":9858,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-02-12T16:44:13.800435+0000","last_change":"2024-02-12T16:42:21.963116+0000","last_active":"2024-02-12T16:44:13.800435+0000","last_peered":"2024-02-12T16:44:13.800435+0000","last_clean":"2024-02-12T16:44:13.800435+0000","last_became_active":"2024-02-12T16:42:21.962544+0000","last_became_peered":"2024-02-12T16:42:21.962544+0000","last_unstale":"2024-02-12T16:44:13.800435+0000","last_undegraded":"2024-02-12T16:44:13.800435+0000","last_fullsized":"2024-02-12T16:44:13.800435+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9814,"ondisk_log_size":9814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":594804736,"num_objects":9076,"num_object_clones":0,"num_object_copies":18152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9814,"num_write_kb":604480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":77,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.017604+0000","last_change":"2024-02-12T16:42:23.724712+0000","last_active":"2024-02-12T16:44:00.017604+0000","last_peered":"2024-02-12T16:44:00.017604+0000","last_clean":"2024-02-12T16:44:00.017604+0000","last_became_active":"2024-02-12T16:41:17.958639+0000","last_became_peered":"2024-02-12T16:41:17.958639+0000","last_unstale":"2024-02-12T16:44:00.017604+0000","last_undegraded":"2024-02-12T16:44:00.017604+0000","last_fullsized":"2024-02-12T16:44:00.017604+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.724526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.724526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"952'9647","reported_seq":10961,"reported_epoch":952,"state":"active+recovering","last_fresh":"2024-02-12T16:44:11.731264+0000","last_change":"2024-02-12T16:43:16.243860+0000","last_active":"2024-02-12T16:44:11.731264+0000","last_peered":"2024-02-12T16:44:11.731264+0000","last_clean":"2024-02-12T16:43:15.356599+0000","last_became_active":"2024-02-12T16:43:16.169927+0000","last_became_peered":"2024-02-12T16:43:16.169927+0000","last_unstale":"2024-02-12T16:44:11.731264+0000","last_undegraded":"2024-02-12T16:44:11.731264+0000","last_fullsized":"2024-02-12T16:44:11.731264+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":9647,"ondisk_log_size":9647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":598933504,"num_objects":9139,"num_object_clones":0,"num_object_copies":18278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5288,"num_objects_unfound":0,"num_objects_dirty":9139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":601152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":41418752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","9"],"object_location_counts":[{"shards":"1,9","objects":5288},{"shards":"1,11","objects":3851}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":369,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:00.016035+0000","last_change":"2024-02-12T16:43:16.012096+0000","last_active":"2024-02-12T16:44:00.016035+0000","last_peered":"2024-02-12T16:44:00.016035+0000","last_clean":"2024-02-12T16:44:00.016035+0000","last_became_active":"2024-02-12T16:43:16.011685+0000","last_became_peered":"2024-02-12T16:43:16.011685+0000","last_unstale":"2024-02-12T16:44:00.016035+0000","last_undegraded":"2024-02-12T16:44:00.016035+0000","last_fullsized":"2024-02-12T16:44:00.016035+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:30.755995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:30.755995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":205,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:12.896865+0000","last_change":"2024-02-12T16:44:12.896865+0000","last_active":"2024-02-12T16:44:12.896865+0000","last_peered":"2024-02-12T16:44:12.896865+0000","last_clean":"2024-02-12T16:44:12.896865+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:44:12.896865+0000","last_undegraded":"2024-02-12T16:44:12.896865+0000","last_fullsized":"2024-02-12T16:44:12.896865+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:22.106269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:40:18.349211+0000","last_clean_scrub_stamp":"2024-02-12T16:42:22.106269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":228,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-02-12T16:44:10.580797+0000","last_change":"2024-02-12T16:44:10.580797+0000","last_active":"2024-02-12T16:44:10.580797+0000","last_peered":"2024-02-12T16:44:10.580797+0000","last_clean":"2024-02-12T16:44:10.580797+0000","last_became_active":"2024-02-12T16:43:15.981046+0000","last_became_peered":"2024-02-12T16:43:15.981046+0000","last_unstale":"2024-02-12T16:44:10.580797+0000","last_undegraded":"2024-02-12T16:44:10.580797+0000","last_fullsized":"2024-02-12T16:44:10.580797+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:35.918516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:13.754017+0000","last_clean_scrub_stamp":"2024-02-12T16:42:35.918516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":548,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:58.000970+0000","last_change":"2024-02-12T16:41:31.381514+0000","last_active":"2024-02-12T16:43:58.000970+0000","last_peered":"2024-02-12T16:43:58.000970+0000","last_clean":"2024-02-12T16:43:58.000970+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:43:58.000970+0000","last_undegraded":"2024-02-12T16:43:58.000970+0000","last_fullsized":"2024-02-12T16:43:58.000970+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:31.381415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:41:31.381415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":511,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-02-12T16:43:57.998898+0000","last_change":"2024-02-12T16:42:25.380280+0000","last_active":"2024-02-12T16:43:57.998898+0000","last_peered":"2024-02-12T16:43:57.998898+0000","last_clean":"2024-02-12T16:43:57.998898+0000","last_became_active":"2024-02-12T16:41:17.950595+0000","last_became_peered":"2024-02-12T16:41:17.950595+0000","last_unstale":"2024-02-12T16:43:57.998898+0000","last_undegraded":"2024-02-12T16:43:57.998898+0000","last_fullsized":"2024-02-12T16:43:57.998898+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:25.380188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:25.380188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":9534439448,"num_objects":145485,"num_object_clones":0,"num_object_copies":290970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14785,"num_objects_unfound":0,"num_objects_dirty":145485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155879,"num_write_kb":9643585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2784,"num_bytes_recovered":182452224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19239018496,"data_stored":19239010352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155879,"ondisk_log_size":155879,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431959082,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2853340,"kb_used_data":1960236,"kb_used_omap":0,"kb_used_meta":892736,"kb_avail":90891812,"statfs":{"total":95995035648,"available":93073215488,"internally_reserved":0,"allocated":2007281664,"data_stored":2006642883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914161664},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.84499999999999997,"15min":1.7669999999999999},"min":{"1min":0.28100000000000003,"5min":0.248,"15min":0.248},"max":{"1min":2.9399999999999999,"5min":3.9159999999999999,"15min":124.437},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.454,"5min":1.216,"15min":1.976},"min":{"1min":0.38,"5min":0.33500000000000002,"15min":0.29199999999999998},"max":{"1min":3.3279999999999998,"5min":4.4450000000000003,"15min":126.926},"last":0.73299999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.282,"15min":1.054},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":8.8450000000000006,"5min":8.8450000000000006,"15min":8.9849999999999994},"last":1.131},{"interface":"front","average":{"1min":1.117,"5min":1.1080000000000001,"15min":0.92700000000000005},"min":{"1min":0.33100000000000002,"5min":0.29899999999999999,"15min":0.28499999999999998},"max":{"1min":4.391,"5min":5.3949999999999996,"15min":11.167},"last":1.3180000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:43:20 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":3.2959999999999998,"15min":1.873},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":3.4980000000000002,"5min":219.56100000000001,"15min":219.56100000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.97299999999999998,"5min":3.1469999999999998,"15min":1.855},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.7010000000000001,"5min":219.79300000000001,"15min":219.79300000000001},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:43:36 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.2270000000000001,"15min":1.7909999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.7709999999999999,"5min":5.4379999999999997,"15min":121.169},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1799999999999999,"15min":1.7769999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.9700000000000002,"5min":4.6420000000000003,"15min":121.181},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.90100000000000002,"15min":0.90300000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.75,"5min":6.1669999999999998,"15min":6.2039999999999997},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.272,"15min":1.0920000000000001},"min":{"1min":0.45100000000000001,"5min":0.315,"15min":0.315},"max":{"1min":3.6299999999999999,"5min":4.6529999999999996,"15min":7.3920000000000003},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.1919999999999999,"15min":1.1579999999999999},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.34100000000000003},"max":{"1min":3.6000000000000001,"5min":3.6219999999999999,"15min":5.968},"last":1.159},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.181,"15min":1.2170000000000001},"min":{"1min":0.48699999999999999,"5min":0.38300000000000001,"15min":0.33100000000000002},"max":{"1min":3.0920000000000001,"5min":4.7300000000000004,"15min":7.4249999999999998},"last":1.2769999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.0449999999999999,"15min":1.1100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":4.2869999999999999,"5min":6.202,"15min":6.4160000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1359999999999999,"15min":1.258},"min":{"1min":0.432,"5min":0.41699999999999998,"15min":0.38700000000000001},"max":{"1min":4.2370000000000001,"5min":6.1289999999999996,"15min":9.5389999999999997},"last":1.0169999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:43:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":1.7330000000000001},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":3.1789999999999998,"5min":3.1789999999999998,"15min":3.1789999999999998},"last":1.8859999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.143,"15min":1.0249999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":5.319,"5min":6.306,"15min":6.4459999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.133,"15min":1.093},"min":{"1min":0.44600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.1190000000000002,"5min":3.1190000000000002,"15min":5.4880000000000004},"last":0.754}]},{"osd":9,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.1870000000000001,"15min":1.222},"min":{"1min":0.56399999999999995,"5min":0.437,"15min":0.38600000000000001},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":7.5709999999999997},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.091,"5min":1.2629999999999999,"15min":1.1930000000000001},"min":{"1min":0.60199999999999998,"5min":0.47399999999999998,"15min":0.44700000000000001},"max":{"1min":3.21,"5min":3.21,"15min":5.7089999999999996},"last":0.80000000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":2.1259999999999999,"5min":1.5660000000000001,"15min":1.2849999999999999},"min":{"1min":0.53600000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.5960000000000001,"5min":4.5990000000000002,"15min":7.4539999999999997},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.925,"5min":1.49,"15min":1.246},"min":{"1min":0.39700000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":7.6479999999999997,"5min":7.6479999999999997,"15min":7.6479999999999997},"last":1.8340000000000001}]}]},{"osd":10,"up_from":83,"seq":356482286130,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3237712,"kb_used_data":2343240,"kb_used_omap":0,"kb_used_meta":894464,"kb_avail":90507440,"statfs":{"total":95995035648,"available":92679618560,"internally_reserved":0,"allocated":2399477760,"data_stored":2398665617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915931136},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.99399999999999999,"15min":1.196},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":2.9199999999999999,"5min":4.7140000000000004,"15min":111.935},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88400000000000001,"15min":1.274},"min":{"1min":0.29899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.641,"5min":4.3920000000000003,"15min":111.316},"last":1.353}]},{"osd":1,"last update":"Mon Feb 12 16:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.0069999999999999,"15min":1.119},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0590000000000002,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.032,"15min":1.115},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.534,"5min":7.2000000000000002,"15min":7.2000000000000002},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0269999999999999,"15min":1.3400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.28100000000000003},"max":{"1min":4.2380000000000004,"5min":9.4179999999999993,"15min":106.822},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.92700000000000005,"15min":1.3420000000000001},"min":{"1min":0.40400000000000003,"5min":0.34699999999999998,"15min":0.30199999999999999},"max":{"1min":1.5009999999999999,"5min":1.923,"15min":107.88800000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:43:44 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0549999999999999,"15min":1.472},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":4.1890000000000001,"5min":9.3699999999999992,"15min":107.014},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":1.151,"15min":1.419},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.27900000000000003},"max":{"1min":3.7210000000000001,"5min":3.7210000000000001,"15min":110.80800000000001},"last":0.58399999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.167,"15min":1.0389999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":6.3689999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.03,"15min":0.995},"min":{"1min":0.40100000000000002,"5min":0.38600000000000001,"15min":0.318},"max":{"1min":1.4219999999999999,"5min":2.0840000000000001,"15min":5.9119999999999999},"last":0.59599999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":4.8849999999999998,"5min":1.992,"15min":1.2929999999999999},"min":{"1min":0.42099999999999999,"5min":0.39500000000000002,"15min":0.32700000000000001},"max":{"1min":74.137,"5min":74.137,"15min":74.137},"last":0.624},{"interface":"front","average":{"1min":4.8120000000000003,"5min":1.9039999999999999,"15min":1.4530000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.41299999999999998},"max":{"1min":74.063999999999993,"5min":74.063999999999993,"15min":74.063999999999993},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.327,"15min":2.581},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.36099999999999999},"max":{"1min":3.843,"5min":3.843,"15min":460.90199999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.234,"15min":2.589},"min":{"1min":0.47899999999999998,"5min":0.44800000000000001,"15min":0.44},"max":{"1min":3.1600000000000001,"5min":3.1600000000000001,"15min":460.93299999999999},"last":0.754}]},{"osd":7,"last update":"Mon Feb 12 16:43:41 2024","interfaces":[{"interface":"back","average":{"1min":5.2519999999999998,"5min":5.2519999999999998,"15min":5.2519999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":74.007000000000005,"5min":74.007000000000005,"15min":74.007000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":4.8860000000000001,"5min":4.8860000000000001,"15min":4.8860000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":73.819000000000003,"5min":73.819000000000003,"15min":73.819000000000003},"last":0.64400000000000002}]},{"osd":8,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.2,"15min":1.1619999999999999},"min":{"1min":0.436,"5min":0.42199999999999999,"15min":0.35499999999999998},"max":{"1min":3.0209999999999999,"5min":3.0209999999999999,"15min":8.4420000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2869999999999999,"15min":1.165},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36299999999999999},"max":{"1min":4.444,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.84199999999999997}]},{"osd":9,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.375,"15min":1.157},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.34699999999999998},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":6.6239999999999997},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.194,"5min":1.29,"15min":1.1479999999999999},"min":{"1min":0.58399999999999996,"5min":0.378,"15min":0.378},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":6.0030000000000001},"last":0.72199999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.4350000000000001,"15min":1.2929999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.4119999999999999,"5min":8.7569999999999997,"15min":8.7569999999999997},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.054,"5min":1.351,"15min":1.26},"min":{"1min":0.57599999999999996,"5min":0.504,"15min":0.38100000000000001},"max":{"1min":2.468,"5min":3.8010000000000002,"15min":5.46},"last":0.78900000000000003}]}]},{"osd":9,"up_from":0,"seq":317827580475,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":8,"up_from":65,"seq":279172874820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1744860,"kb_used_data":1154796,"kb_used_omap":0,"kb_used_meta":589696,"kb_avail":92000292,"statfs":{"total":95995035648,"available":94208299008,"internally_reserved":0,"allocated":1182511104,"data_stored":1181872323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":603848704},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.78600000000000003,"15min":1.1850000000000001},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.23899999999999999},"max":{"1min":0.69399999999999995,"5min":5.96,"15min":134.01499999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.91700000000000004,"15min":1.3720000000000001},"min":{"1min":0.315,"5min":0.308,"15min":0.28799999999999998},"max":{"1min":0.871,"5min":6.7729999999999997,"15min":133.667},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.8319999999999999,"5min":1.671,"15min":1.571},"min":{"1min":0.44,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":33.415999999999997,"5min":33.415999999999997,"15min":33.415999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":2.7719999999999998,"5min":1.343,"15min":1.091},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":33.337000000000003,"5min":33.337000000000003,"15min":33.337000000000003},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":1.9399999999999999,"5min":3.2320000000000002,"15min":8.6430000000000007},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2609999999999999,"15min":1.091},"min":{"1min":0.48999999999999999,"5min":0.378,"15min":0.26800000000000002},"max":{"1min":2.0070000000000001,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.84099999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.1699999999999999,"15min":1.115},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.5549999999999999,"5min":13.784000000000001,"15min":25.919},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2110000000000001,"15min":1.2330000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.8580000000000001,"5min":13.773999999999999,"15min":25.984999999999999},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Feb 12 16:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":0.92200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.3620000000000001,"5min":3.3620000000000001,"15min":8.609},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.1000000000000001,"15min":1.107},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.37},"max":{"1min":0.95299999999999996,"5min":4.4500000000000002,"15min":9.0890000000000004},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:43:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.93300000000000005,"15min":1.034},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.318},"max":{"1min":2.359,"5min":4.5069999999999997,"15min":7.2960000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.2090000000000001,"15min":1.256},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.35199999999999998},"max":{"1min":6.681,"5min":6.681,"15min":9.0540000000000003},"last":0.70199999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.1439999999999999,"15min":1.101},"min":{"1min":0.432,"5min":0.432,"15min":0.38200000000000001},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":8.5220000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.1619999999999999,"15min":1.302},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.34599999999999997},"max":{"1min":2.302,"5min":4.5140000000000002,"15min":9.1500000000000004},"last":0.64100000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:43:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":4.218,"5min":4.218,"15min":4.218},"last":0.495},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.7850000000000001,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":0.56999999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.214,"15min":1.206},"min":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.34200000000000003},"max":{"1min":2.6440000000000001,"5min":4.407,"15min":9.1170000000000009},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.357,"15min":1.3200000000000001},"min":{"1min":0.57399999999999995,"5min":0.49199999999999999,"15min":0.46100000000000002},"max":{"1min":2.6960000000000002,"5min":4.5380000000000003,"15min":9.1980000000000004},"last":0.72399999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.167,"15min":1.048},"min":{"1min":0.61099999999999999,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":1.554,"5min":5.0709999999999997,"15min":8.452},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.3049999999999999,"15min":1.137},"min":{"1min":0.48399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":3.0190000000000001,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":0.83199999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:43:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.268,"15min":1.1399999999999999},"min":{"1min":0.53500000000000003,"5min":0.437,"15min":0.42699999999999999},"max":{"1min":3.024,"5min":4.7750000000000004,"15min":8.6790000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.496,"5min":1.371,"15min":1.1850000000000001},"min":{"1min":0.46500000000000002,"5min":0.45800000000000002,"15min":0.44600000000000001},"max":{"1min":3.0920000000000001,"5min":4.7549999999999999,"15min":5.2320000000000002},"last":0.73899999999999999}]}]},{"osd":7,"up_from":923,"seq":3964254814228,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088596,"kb_used_data":1757836,"kb_used_omap":0,"kb_used_meta":330751,"kb_avail":91656556,"statfs":{"total":95995035648,"available":93856313344,"internally_reserved":0,"allocated":1800024064,"data_stored":1799207849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":338689076},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.0170000000000003,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":2.9860000000000002},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.3019999999999996,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":3.0329999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":4.4459999999999997,"5min":4.4459999999999997,"15min":4.4459999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":4.5110000000000001,"5min":4.5110000000000001,"15min":4.5110000000000001},"last":0.48399999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.4540000000000002,"5min":3.4540000000000002,"15min":3.4540000000000002},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":3.1749999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.4590000000000001},"last":3.2010000000000001},{"interface":"front","average":{"1min":1.516,"5min":1.516,"15min":1.516},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.1269999999999998,"5min":5.1269999999999998,"15min":5.1269999999999998},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":2.1000000000000001},"last":3.226},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.7360000000000002,"5min":3.7360000000000002,"15min":3.7360000000000002},"last":2.3439999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.4790000000000001,"5min":4.4790000000000001,"15min":4.4790000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":2.6099999999999999},"last":3.1110000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":3.073}]},{"osd":11,"last update":"Mon Feb 12 16:43:42 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.103,"15min":1.103},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.198,"5min":3.198,"15min":3.198},"last":3.1480000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":5.1790000000000003,"5min":5.1790000000000003,"15min":5.1790000000000003},"last":1.02}]}]},{"osd":6,"up_from":48,"seq":206158430805,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1456924,"kb_used_data":862764,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92288228,"statfs":{"total":95995035648,"available":94503145472,"internally_reserved":0,"allocated":883470336,"data_stored":882831555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.82399999999999995,"15min":0.81000000000000005},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.419,"5min":5.7430000000000003,"15min":5.7430000000000003},"last":0.87},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.751,"15min":0.77500000000000002},"min":{"1min":0.39900000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.421,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.98399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:43:45 2024","interfaces":[{"interface":"back","average":{"1min":6.3129999999999997,"5min":2.1520000000000001,"15min":1.4139999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":118.962,"5min":118.962,"15min":118.962},"last":0.39800000000000002},{"interface":"front","average":{"1min":6.3159999999999998,"5min":2.0289999999999999,"15min":1.506},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":122.48,"5min":122.48,"15min":122.48},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.93799999999999994,"15min":1.244},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.27000000000000002},"max":{"1min":6.3380000000000001,"5min":6.3380000000000001,"15min":14.68},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.82599999999999996,"15min":1.135},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.27700000000000002},"max":{"1min":1.032,"5min":3.298,"15min":15.138},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.82399999999999995,"15min":3.0329999999999999},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.27800000000000002},"max":{"1min":1.4830000000000001,"5min":2.4590000000000001,"15min":365.58999999999997},"last":124.44499999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.0660000000000001,"15min":3.5459999999999998},"min":{"1min":0.39600000000000002,"5min":0.374,"15min":0.317},"max":{"1min":1.6080000000000001,"5min":4.1699999999999999,"15min":366.93299999999999},"last":124.407}]},{"osd":4,"last update":"Mon Feb 12 16:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.876,"15min":1.0669999999999999},"min":{"1min":0.316,"5min":0.29699999999999999,"15min":0.249},"max":{"1min":3.5,"5min":3.5,"15min":8.4559999999999995},"last":0.374},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.92700000000000005,"15min":1.0860000000000001},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.29999999999999999},"max":{"1min":1.919,"5min":2.5430000000000001,"15min":7.7610000000000001},"last":0.59499999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:43:45 2024","interfaces":[{"interface":"back","average":{"1min":14.409000000000001,"5min":3.9529999999999998,"15min":2.169},"min":{"1min":0.52700000000000002,"5min":0.47099999999999997,"15min":0.26200000000000001},"max":{"1min":281.48500000000001,"5min":281.48500000000001,"15min":281.48500000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":14.4,"5min":3.992,"15min":2.165},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":282.96199999999999,"5min":282.96199999999999,"15min":282.96199999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 12 16:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":0.68400000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":1.1419999999999999,"15min":2.875},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.35699999999999998},"max":{"1min":1.48,"5min":3.113,"15min":501.892},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.876,"5min":1.143,"15min":2.839},"min":{"1min":0.51800000000000002,"5min":0.40500000000000003,"15min":0.38700000000000001},"max":{"1min":2.0870000000000002,"5min":2.956,"15min":501.76600000000002},"last":0.72199999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.224,"15min":1.238},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.35299999999999998},"max":{"1min":2.2109999999999999,"5min":3.3279999999999998,"15min":6.319},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.1060000000000001,"15min":1.2470000000000001},"min":{"1min":0.60499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.1819999999999999,"5min":2.6280000000000001,"15min":7.6870000000000003},"last":0.77900000000000003}]},{"osd":10,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.2130000000000001,"15min":1.3600000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.42499999999999999},"max":{"1min":2.0920000000000001,"5min":2.6850000000000001,"15min":11.536},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.2889999999999999,"15min":1.4350000000000001},"min":{"1min":0.58999999999999997,"5min":0.58399999999999996,"15min":0.48099999999999998},"max":{"1min":2.1659999999999999,"5min":2.8050000000000002,"15min":11.585000000000001},"last":0.76600000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.2889999999999999,"15min":1.466},"min":{"1min":0.61899999999999999,"5min":0.54800000000000004,"15min":0.49299999999999999},"max":{"1min":1.8819999999999999,"5min":5.3799999999999999,"15min":8.6649999999999991},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.238,"15min":1.214},"min":{"1min":0.55800000000000005,"5min":0.45600000000000002,"15min":0.40699999999999997},"max":{"1min":2.5649999999999999,"5min":5.6130000000000004,"15min":7.04},"last":0.874}]}]},{"osd":1,"up_from":13,"seq":55834575481,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5032724,"kb_used_data":4130316,"kb_used_omap":0,"kb_used_meta":902400,"kb_avail":88712428,"statfs":{"total":95995035648,"available":90841526272,"internally_reserved":0,"allocated":4229443584,"data_stored":4228627369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":924057600},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.82199999999999995,"15min":0.96399999999999997},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":2.6230000000000002,"5min":7.532,"15min":10.420999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.83599999999999997,"15min":1},"min":{"1min":0.33700000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.6080000000000001,"5min":7.3540000000000001,"15min":10.26},"last":4.0499999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:43:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.91900000000000004,"15min":1.1120000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.22700000000000001},"max":{"1min":4.2679999999999998,"5min":4.2679999999999998,"15min":10.755000000000001},"last":0.318},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.97599999999999998,"15min":1.1060000000000001},"min":{"1min":0.29899999999999999,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":7.9539999999999997,"5min":7.9539999999999997,"15min":9.702},"last":1.028}]},{"osd":3,"last update":"Mon Feb 12 16:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.90200000000000002,"15min":1.161},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":4.1180000000000003,"5min":4.1180000000000003,"15min":26.664000000000001},"last":4.1109999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.97699999999999998,"15min":1.367},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.26200000000000001},"max":{"1min":1.55,"5min":3.629,"15min":27.148},"last":4.1539999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:43:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.94999999999999996,"15min":1.024},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.312},"max":{"1min":1.1830000000000001,"5min":3.6549999999999998,"15min":7.7549999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.044,"15min":1.0900000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.312},"max":{"1min":2.746,"5min":3.7309999999999999,"15min":6.9119999999999999},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:43:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.1359999999999999,"15min":4.0069999999999997},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.309},"max":{"1min":8.0190000000000001,"5min":8.0190000000000001,"15min":524.85299999999995},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.276,"15min":4.1319999999999997},"min":{"1min":0.53100000000000003,"5min":0.34799999999999998,"15min":0.32800000000000001},"max":{"1min":8.7050000000000001,"5min":8.7050000000000001,"15min":519.89700000000005},"last":4.194}]},{"osd":6,"last update":"Mon Feb 12 16:43:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.1160000000000001,"15min":1.274},"min":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.331,"5min":3.7829999999999999,"15min":3.7829999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3620000000000001,"15min":1.468},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":4.5099999999999998,"5min":4.5099999999999998,"15min":4.5099999999999998},"last":4.2450000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:59 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0629999999999999,"15min":3.2370000000000001},"min":{"1min":0.44700000000000001,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":522.79200000000003},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.25,"15min":3.198},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":4.3360000000000003,"5min":9.0069999999999997,"15min":520.64499999999998},"last":1.5800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:44:06 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.4650000000000001,"15min":1.3200000000000001},"min":{"1min":0.59399999999999997,"5min":0.48499999999999999,"15min":0.379},"max":{"1min":8.8100000000000005,"5min":8.8100000000000005,"15min":9.1430000000000007},"last":4.3220000000000001},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5640000000000001,"15min":1.3779999999999999},"min":{"1min":0.67400000000000004,"5min":0.54300000000000004,"15min":0.42499999999999999},"max":{"1min":8.9269999999999996,"5min":8.9269999999999996,"15min":9.1739999999999995},"last":4.3719999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:43:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.224,"15min":1.194},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":2.0329999999999999,"5min":3.1869999999999998,"15min":8.3949999999999996},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.3360000000000001,"15min":1.302},"min":{"1min":0.48099999999999998,"5min":0.47299999999999998,"15min":0.436},"max":{"1min":2.266,"5min":3.2229999999999999,"15min":8.4969999999999999},"last":1.3540000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:44:06 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.4790000000000001,"15min":1.3400000000000001},"min":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.42999999999999999},"max":{"1min":8.1140000000000008,"5min":8.1140000000000008,"15min":8.3469999999999995},"last":1.175},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.6220000000000001,"15min":1.3919999999999999},"min":{"1min":0.63,"5min":0.51200000000000001,"15min":0.46999999999999997},"max":{"1min":8.8469999999999995,"5min":8.8469999999999995,"15min":9.0440000000000005},"last":4.2809999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:43:56 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.6339999999999999,"15min":1.7310000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.47099999999999997},"max":{"1min":8.8849999999999998,"5min":8.8849999999999998,"15min":88.679000000000002},"last":4.4109999999999996},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3400000000000001,"15min":1.6240000000000001},"min":{"1min":0.56100000000000005,"5min":0.51800000000000002,"15min":0.46100000000000002},"max":{"1min":8.0730000000000004,"5min":8.0730000000000004,"15min":88.519000000000005},"last":1.212}]}]},{"osd":0,"up_from":8,"seq":34359739013,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888528,"kb_used_data":1608,"kb_used_omap":0,"kb_used_meta":886912,"kb_avail":92856624,"statfs":{"total":95995035648,"available":95085182976,"internally_reserved":0,"allocated":1646592,"data_stored":834449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":908197888},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":2.1030000000000002,"15min":1.3720000000000001},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":1.7589999999999999,"5min":118.657,"15min":118.657},"last":5.4470000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":2.4300000000000002,"15min":1.591},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.22600000000000001},"max":{"1min":7.6429999999999998,"5min":118.58,"15min":118.58},"last":4.0060000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:44:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.83699999999999997,"15min":0.92000000000000004},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.0950000000000002,"5min":3.2909999999999999,"15min":3.2909999999999999},"last":4.0999999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.84499999999999997,"15min":0.89900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.641,"5min":2.8660000000000001,"15min":2.8660000000000001},"last":4.0469999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:43:14 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.98899999999999999,"15min":2.3660000000000001},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.252},"max":{"1min":2.0569999999999999,"5min":3.681,"15min":377.06299999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.97999999999999998,"15min":2.512},"min":{"1min":0.39300000000000002,"5min":0.26600000000000001,"15min":0.224},"max":{"1min":1.262,"5min":4.6779999999999999,"15min":379.18000000000001},"last":3.8340000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.94699999999999995,"15min":1.0189999999999999},"min":{"1min":0.35699999999999998,"5min":0.32700000000000001,"15min":0.29099999999999998},"max":{"1min":1.5109999999999999,"5min":1.748,"15min":7.2869999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.014,"15min":1.1859999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.29499999999999998},"max":{"1min":1.431,"5min":4.0220000000000002,"15min":8.4949999999999992},"last":3.9580000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:43:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0509999999999999,"15min":1.728},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.30399999999999999},"max":{"1min":8.532,"5min":8.532,"15min":181.46199999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.234,"5min":1.181,"15min":1.855},"min":{"1min":0.48099999999999998,"5min":0.41299999999999998,"15min":0.32800000000000001},"max":{"1min":8.6519999999999992,"5min":8.6519999999999992,"15min":186.41999999999999},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.0169999999999999,"15min":1.0669999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3280000000000001,"5min":6.6950000000000003,"15min":6.6950000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.109,"15min":1.2789999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.747}]},{"osd":7,"last update":"Mon Feb 12 16:39:13 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.24,"15min":2.706},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.374},"max":{"1min":1.835,"5min":6.1130000000000004,"15min":191.47200000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4019999999999999,"15min":2.823},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":1.897,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.081}]},{"osd":8,"last update":"Mon Feb 12 16:43:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.1220000000000001,"15min":1.2270000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.41299999999999998},"max":{"1min":1.2,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.2410000000000001,"15min":1.3300000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.48299999999999998},"max":{"1min":1.4350000000000001,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":0.69899999999999995}]},{"osd":9,"last update":"Mon Feb 12 16:43:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.155,"15min":1.242},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.34999999999999998},"max":{"1min":1.772,"5min":2.9750000000000001,"15min":8.4629999999999992},"last":4.1130000000000004},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.2,"15min":1.3380000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41499999999999998},"max":{"1min":1.504,"5min":6.7770000000000001,"15min":12.943},"last":0.56399999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:43:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.3260000000000001,"15min":1.254},"min":{"1min":0.496,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":1.9059999999999999,"5min":13.157,"15min":13.157},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.115,"5min":1.4079999999999999,"15min":1.3600000000000001},"min":{"1min":0.56699999999999995,"5min":0.495,"15min":0.48299999999999998},"max":{"1min":1.9850000000000001,"5min":13.238,"15min":13.238},"last":0.91600000000000004}]},{"osd":11,"last update":"Mon Feb 12 16:43:51 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3069999999999999,"15min":1.399},"min":{"1min":0.56100000000000005,"5min":0.50900000000000001,"15min":0.40300000000000002},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":8.4280000000000008},"last":4.157},{"interface":"front","average":{"1min":1.012,"5min":1.2490000000000001,"15min":1.282},"min":{"1min":0.56699999999999995,"5min":0.50600000000000001,"15min":0.47299999999999998},"max":{"1min":2.9380000000000002,"5min":4.9960000000000004,"15min":7.3799999999999999},"last":0.84099999999999997}]}]},{"osd":2,"up_from":21,"seq":90194313843,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595408,"kb_used_data":1608,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93149744,"statfs":{"total":95995035648,"available":95385337856,"internally_reserved":0,"allocated":1646592,"data_stored":834449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.73999999999999999,"15min":0.79900000000000004},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.876,"5min":3.496,"15min":3.496},"last":0.373},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.73399999999999999,"15min":0.81799999999999995},"min":{"1min":0.26700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.762,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:43:58 2024","interfaces":[{"interface":"back","average":{"1min":1.9730000000000001,"5min":1.0940000000000001,"15min":0.98499999999999999},"min":{"1min":0.34599999999999997,"5min":0.27200000000000002,"15min":0.19500000000000001},"max":{"1min":24.184999999999999,"5min":24.184999999999999,"15min":24.184999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.929,"5min":1.0960000000000001,"15min":1.1830000000000001},"min":{"1min":0.26600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":24.292000000000002,"5min":24.292000000000002,"15min":27.562999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:43:46 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.125,"15min":1.448},"min":{"1min":0.34899999999999998,"5min":0.308,"15min":0.25700000000000001},"max":{"1min":24.114000000000001,"5min":24.114000000000001,"15min":54.091000000000001},"last":0.499},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.2270000000000001,"15min":1.4399999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.25800000000000001},"max":{"1min":24.155999999999999,"5min":24.155999999999999,"15min":54.203000000000003},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.94199999999999995,"15min":0.998},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.29499999999999998},"max":{"1min":1.3360000000000001,"5min":3.0470000000000002,"15min":8.3439999999999994},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.94399999999999995,"15min":1.0740000000000001},"min":{"1min":0.34999999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":1.6950000000000001,"5min":5.1710000000000003,"15min":7.0019999999999998},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:43:58 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.91000000000000003,"15min":0.94799999999999995},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":2.7669999999999999,"5min":5.1319999999999997,"15min":8.7509999999999994},"last":4.056},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.97499999999999998,"15min":1.0900000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.91800000000000004,"5min":3.694,"15min":7.0590000000000002},"last":4.2089999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0269999999999999,"15min":1.1519999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.316},"max":{"1min":1.758,"5min":3.0209999999999999,"15min":8.4510000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.081,"15min":1.137},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35099999999999998},"max":{"1min":1.242,"5min":3.7879999999999998,"15min":8.8179999999999996},"last":2.1429999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.58399999999999996,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":1.9019999999999999,"5min":6.2560000000000002,"15min":11.044},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2529999999999999,"15min":1.248},"min":{"1min":0.44600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.4219999999999997,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":1.6499999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:43:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.7070000000000001,"15min":1.3220000000000001},"min":{"1min":0.41299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.361,"5min":59.164999999999999,"15min":59.164999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.7350000000000001,"15min":1.3160000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.377},"max":{"1min":1.3660000000000001,"5min":58.993000000000002,"15min":58.993000000000002},"last":0.63}]},{"osd":9,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1599999999999999,"15min":1.2529999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.38600000000000001},"max":{"1min":1.6319999999999999,"5min":5.2210000000000001,"15min":7.7759999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.2609999999999999,"15min":1.2569999999999999},"min":{"1min":0.434,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.3609999999999998,"5min":5.2549999999999999,"15min":7.2110000000000003},"last":0.61599999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:44:03 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":5.1689999999999996,"15min":2.54},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41399999999999998},"max":{"1min":7.1820000000000004,"5min":407.577,"15min":407.577},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":4.9029999999999996,"15min":2.4729999999999999},"min":{"1min":0.47899999999999998,"5min":0.47799999999999998,"15min":0.41799999999999998},"max":{"1min":7.1470000000000002,"5min":407.45100000000002,"15min":407.45100000000002},"last":0.84499999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.2869999999999999,"15min":1.212},"min":{"1min":0.63100000000000001,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":1.825,"5min":3.8540000000000001,"15min":8.9359999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.4339999999999999,"15min":1.3360000000000001},"min":{"1min":0.61699999999999999,"5min":0.53600000000000003,"15min":0.47699999999999998},"max":{"1min":1.6080000000000001,"5min":4.8849999999999998,"15min":8.968},"last":0.76500000000000001}]}]},{"osd":3,"up_from":522,"seq":2241972928767,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2387792,"kb_used_data":1754888,"kb_used_omap":91,"kb_used_meta":632804,"kb_avail":91357360,"statfs":{"total":95995035648,"available":93549936640,"internally_reserved":0,"allocated":1797005312,"data_stored":1796193169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":647991312},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.748,"15min":1.032},"min":{"1min":0.26600000000000001,"5min":0.26200000000000001,"15min":0.20699999999999999},"max":{"1min":1.7350000000000001,"5min":4.0469999999999997,"15min":10.101000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.82999999999999996,"15min":1.0580000000000001},"min":{"1min":0.27800000000000002,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":1.794,"5min":4.9969999999999999,"15min":10.436999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:43:30 2024","interfaces":[{"interface":"back","average":{"1min":14.590999999999999,"5min":3.8420000000000001,"15min":1.9219999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":245.10900000000001,"5min":245.10900000000001,"15min":245.10900000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":14.506,"5min":3.8919999999999999,"15min":2.0640000000000001},"min":{"1min":0.38,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":245.21199999999999,"5min":245.21199999999999,"15min":245.21199999999999},"last":1.22}]},{"osd":2,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.6899999999999999,"15min":1.3049999999999999},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.252},"max":{"1min":2.0899999999999999,"5min":81.769999999999996,"15min":81.769999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.806,"15min":1.4710000000000001},"min":{"1min":0.45500000000000002,"5min":0.32300000000000001,"15min":0.23699999999999999},"max":{"1min":2.1230000000000002,"5min":81.876999999999995,"15min":81.876999999999995},"last":1.0529999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.0049999999999999,"15min":1.1599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.27400000000000002},"max":{"1min":2.2629999999999999,"5min":2.2679999999999998,"15min":6.8220000000000001},"last":0.754},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.92600000000000005,"15min":1.1020000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":1.825,"5min":4.6159999999999997,"15min":6.6840000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":1.1299999999999999,"15min":1.181},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.30499999999999999},"max":{"1min":2.4740000000000002,"5min":8.9489999999999998,"15min":11.884},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.143,"15min":1.306},"min":{"1min":0.48199999999999998,"5min":0.36199999999999999,"15min":0.29899999999999999},"max":{"1min":6.633,"5min":6.633,"15min":12.164999999999999},"last":0.79900000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.21,"15min":1.252},"min":{"1min":0.40100000000000002,"5min":0.379,"15min":0.31900000000000001},"max":{"1min":4.3449999999999998,"5min":9,"15min":9.4689999999999994},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.139,"15min":1.3169999999999999},"min":{"1min":0.52000000000000002,"5min":0.496,"15min":0.30499999999999999},"max":{"1min":2.1869999999999998,"5min":2.4470000000000001,"15min":9.4640000000000004},"last":0.83799999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.5149999999999999,"15min":1.298},"min":{"1min":0.55000000000000004,"5min":0.36199999999999999,"15min":0.33300000000000002},"max":{"1min":7.9909999999999997,"5min":7.9909999999999997,"15min":7.9909999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.609,"5min":1.3859999999999999,"15min":1.335},"min":{"1min":0.621,"5min":0.371,"15min":0.371},"max":{"1min":6.0899999999999999,"5min":6.7590000000000003,"15min":7.2249999999999996},"last":0.59299999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.2290000000000001,"15min":1.28},"min":{"1min":0.49399999999999999,"5min":0.439,"15min":0.38200000000000001},"max":{"1min":1.982,"5min":5.8079999999999998,"15min":5.9669999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.3360000000000001,"15min":1.3380000000000001},"min":{"1min":0.64500000000000002,"5min":0.47899999999999998,"15min":0.35599999999999998},"max":{"1min":2.2309999999999999,"5min":4.9569999999999999,"15min":7.9569999999999999},"last":0.90200000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.2170000000000001,"15min":1.353},"min":{"1min":0.53800000000000003,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":2.1930000000000001,"5min":5.9260000000000002,"15min":8.9350000000000005},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.097,"5min":1.3080000000000001,"15min":1.327},"min":{"1min":0.58499999999999996,"5min":0.54600000000000004,"15min":0.38300000000000001},"max":{"1min":3.4649999999999999,"5min":4.7489999999999997,"15min":9.5239999999999991},"last":0.872}]},{"osd":10,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":9.0869999999999997,"15min":3.8980000000000001},"min":{"1min":0.57999999999999996,"5min":0.45500000000000002,"15min":0.40799999999999997},"max":{"1min":1.877,"5min":738.99900000000002,"15min":738.99900000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":9.0869999999999997,"15min":4.0419999999999998},"min":{"1min":0.60299999999999998,"5min":0.58599999999999997,"15min":0.45600000000000002},"max":{"1min":4.3769999999999998,"5min":746.45299999999997,"15min":746.45299999999997},"last":0.76800000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.466,"15min":1.6160000000000001},"min":{"1min":0.56299999999999994,"5min":0.502,"15min":0.437},"max":{"1min":4.4029999999999996,"5min":8.9030000000000005,"15min":73.037999999999997},"last":4.9130000000000003},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.367,"15min":1.665},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.53200000000000003},"max":{"1min":4.1100000000000003,"5min":4.4989999999999997,"15min":73.057000000000002},"last":4.8979999999999997}]}]},{"osd":4,"up_from":32,"seq":137438954083,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2058588,"kb_used_data":1164076,"kb_used_omap":0,"kb_used_meta":894144,"kb_avail":91686564,"statfs":{"total":95995035648,"available":93887041536,"internally_reserved":0,"allocated":1192013824,"data_stored":1191375043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915603456},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81999999999999995,"15min":1.02},"min":{"1min":0.372,"5min":0.27100000000000002,"15min":0.27000000000000002},"max":{"1min":1.1080000000000001,"5min":3.9670000000000001,"15min":4.1050000000000004},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.92100000000000004,"15min":1.1220000000000001},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.4470000000000001,"5min":5.0330000000000004,"15min":5.0599999999999996},"last":1.1850000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.232,"15min":1.3180000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":3.1520000000000001,"5min":30.137,"15min":56.188000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.353,"15min":1.446},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.25700000000000001},"max":{"1min":2.1429999999999998,"5min":32.481999999999999,"15min":55.414000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":3.2050000000000001,"15min":1.8740000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3100000000000001,"5min":212.685,"15min":212.685},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":3.1019999999999999,"15min":1.786},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.9390000000000001,"5min":210.95699999999999,"15min":210.95699999999999},"last":0.505}]},{"osd":3,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.3440000000000001,"15min":1.3260000000000001},"min":{"1min":0.38100000000000001,"5min":0.33600000000000002,"15min":0.311},"max":{"1min":3.0859999999999999,"5min":30.282,"15min":30.282},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":1.27,"15min":1.2929999999999999},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.25800000000000001},"max":{"1min":3.3719999999999999,"5min":30.478000000000002,"15min":30.478000000000002},"last":4.1609999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:43:52 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.77600000000000002,"15min":1.095},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.252},"max":{"1min":1.0900000000000001,"5min":2.5960000000000001,"15min":10.289999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.93400000000000005,"15min":1.2230000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":1.1240000000000001,"5min":2.6469999999999998,"15min":10.321999999999999},"last":0.90300000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.82299999999999995,"15min":1.069},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28000000000000003},"max":{"1min":2.8809999999999998,"5min":2.8809999999999998,"15min":7.5289999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.024,"15min":1.155},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.28999999999999998},"max":{"1min":1.4259999999999999,"5min":2.46,"15min":11.968999999999999},"last":0.67400000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:43:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.569,"5min":3.569,"15min":3.569},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.0019999999999998,"5min":3.0019999999999998,"15min":3.0019999999999998},"last":0.624}]},{"osd":8,"last update":"Mon Feb 12 16:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.1859999999999999,"15min":1.2090000000000001},"min":{"1min":0.47399999999999998,"5min":0.44400000000000001,"15min":0.40300000000000002},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":7.6890000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":1.0189999999999999,"15min":1.1739999999999999},"min":{"1min":0.439,"5min":0.41399999999999998,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":3.6659999999999999,"15min":7.9349999999999996},"last":0.49399999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.2490000000000001,"15min":1.4710000000000001},"min":{"1min":0.60699999999999998,"5min":0.46899999999999997,"15min":0.39300000000000002},"max":{"1min":2.0169999999999999,"5min":2.2879999999999998,"15min":63.545000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.327,"15min":1.468},"min":{"1min":0.65100000000000002,"5min":0.56699999999999995,"15min":0.36699999999999999},"max":{"1min":2.0979999999999999,"5min":2.395,"15min":63.421999999999997},"last":0.79500000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.23,"15min":1.266},"min":{"1min":0.50700000000000001,"5min":0.49099999999999999,"15min":0.433},"max":{"1min":3.9569999999999999,"5min":3.9569999999999999,"15min":7.7619999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1499999999999999,"15min":1.125},"min":{"1min":0.59299999999999997,"5min":0.47599999999999998,"15min":0.41799999999999998},"max":{"1min":3.048,"5min":3.048,"15min":5.6459999999999999},"last":0.879}]},{"osd":11,"last update":"Mon Feb 12 16:43:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.258,"15min":1.3080000000000001},"min":{"1min":0.55700000000000005,"5min":0.50900000000000001,"15min":0.45400000000000001},"max":{"1min":3.4710000000000001,"5min":3.4710000000000001,"15min":7.734},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.3520000000000001,"15min":1.3069999999999999},"min":{"1min":0.56399999999999995,"5min":0.55200000000000005,"15min":0.46400000000000002},"max":{"1min":3.4580000000000002,"5min":3.4580000000000002,"15min":7.7969999999999997},"last":1.855}]}]},{"osd":5,"up_from":39,"seq":167503725146,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3493520,"kb_used_data":2600392,"kb_used_omap":0,"kb_used_meta":893056,"kb_avail":90251632,"statfs":{"total":95995035648,"available":92417671168,"internally_reserved":0,"allocated":2662801408,"data_stored":2661989265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914489344},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.85499999999999998,"15min":1.415},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.7610000000000001,"5min":3.7610000000000001,"15min":38.366},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.45,"5min":0.92200000000000004,"15min":1.49},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":4.7720000000000002,"5min":4.7720000000000002,"15min":37.310000000000002},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":23.995000000000001,"5min":5.5830000000000002,"15min":2.5659999999999998},"min":{"1min":0.29099999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":387.76299999999998,"5min":387.76299999999998,"15min":387.76299999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":24.143999999999998,"5min":5.7149999999999999,"15min":2.694},"min":{"1min":0.39600000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":387.88099999999997,"5min":387.88099999999997,"15min":387.88099999999997},"last":0.82699999999999996}]},{"osd":2,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.1140000000000001,"15min":1.155},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":3.867,"5min":4.5990000000000002,"15min":6.6459999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91000000000000003,"15min":1.1479999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.8380000000000001,"5min":7.3470000000000004,"15min":11.448},"last":1.8460000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.1040000000000001,"15min":1.373},"min":{"1min":0.36799999999999999,"5min":0.28299999999999997,"15min":0.27200000000000002},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":22.484000000000002},"last":5.8879999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":0.94899999999999995,"15min":1.3109999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29699999999999999},"max":{"1min":4.7839999999999998,"5min":4.7839999999999998,"15min":23.34},"last":1.889}]},{"osd":4,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":0.93100000000000005,"15min":1.044},"min":{"1min":0.40899999999999997,"5min":0.255,"15min":0.255},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":6.5999999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.0409999999999999,"15min":1.119},"min":{"1min":0.39600000000000002,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":4.7430000000000003,"5min":4.7430000000000003,"15min":5.5389999999999997},"last":0.44600000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:43:43 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.0269999999999999,"15min":1.4119999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.28699999999999998},"max":{"1min":4.6699999999999999,"5min":4.6699999999999999,"15min":68.263999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.97999999999999998,"15min":1.476},"min":{"1min":0.34599999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.5730000000000004,"5min":4.5730000000000004,"15min":67.873000000000005},"last":1.9450000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.86799999999999999,"15min":1.1830000000000001},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.28799999999999998},"max":{"1min":1.9570000000000001,"5min":5.6609999999999996,"15min":9.7710000000000008},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.024,"15min":1.286},"min":{"1min":0.376,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.9670000000000001,"5min":5.6120000000000001,"15min":6.7599999999999998},"last":0.627}]},{"osd":8,"last update":"Mon Feb 12 16:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1419999999999999,"15min":1.202},"min":{"1min":0.5,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.0019999999999998,"5min":4.0019999999999998,"15min":4.0019999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":1,"5min":1.107,"15min":1.3009999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"last":2.0179999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:43:43 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":1.3620000000000001,"15min":1.2430000000000001},"min":{"1min":0.46300000000000002,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":9.0830000000000002,"5min":9.0830000000000002,"15min":9.0830000000000002},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.1879999999999999,"15min":1.242},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.21,"5min":6.21,"15min":7.9710000000000001},"last":2.0579999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:43:57 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":1.391,"15min":1.5529999999999999},"min":{"1min":0.46400000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":8.3879999999999999,"5min":8.3879999999999999,"15min":55.518999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.3939999999999999,"15min":1.494},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":4.9029999999999996,"5min":4.9029999999999996,"15min":55.497999999999998},"last":0.89600000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:43:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.3380000000000001,"15min":1.425},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":3.6880000000000002,"5min":5.6749999999999998,"15min":8.6940000000000008},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.3839999999999999,"15min":1.3520000000000001},"min":{"1min":0.65200000000000002,"5min":0.54400000000000004,"15min":0.51500000000000001},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":6.9029999999999996},"last":0.86299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4207546368,"data_stored":4207542296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1794834432,"data_stored":1794834432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1188429824,"data_stored":1188429824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2661023744,"data_stored":2661023744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":881655808,"data_stored":881655808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1788809216,"data_stored":1788805144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1179254784,"data_stored":1179254784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1151401984,"data_stored":1151401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2383675392,"data_stored":2383675392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2002386944,"data_stored":2002386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:44:19.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set device_health_metrics pgp_num 31 2024-02-12T16:44:19.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:19.319 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:19.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:19 smithi121 bash[18489]: audit 2024-02-12T16:44:18.167883+0000 mgr.y (mgr.14146) 1800 : audit [DBG] from='client.17205 v1:172.21.15.121:0/3762891711' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:44:19.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:19 smithi121 bash[18489]: cluster 2024-02-12T16:44:18.294184+0000 mgr.y (mgr.14146) 1801 : cluster [DBG] pgmap v2623: 47 pgs: 3 active+recovering, 44 active+clean; 8.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 7.2 MiB/s wr, 855 op/s; 14600/286552 objects misplaced (5.095%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:19.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:19.461 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:19.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:19.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:19.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:19.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:19 smithi123 bash[19119]: audit 2024-02-12T16:44:18.167883+0000 mgr.y (mgr.14146) 1800 : audit [DBG] from='client.17205 v1:172.21.15.121:0/3762891711' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:44:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:19 smithi123 bash[19119]: cluster 2024-02-12T16:44:18.294184+0000 mgr.y (mgr.14146) 1801 : cluster [DBG] pgmap v2623: 47 pgs: 3 active+recovering, 44 active+clean; 8.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 7.2 MiB/s wr, 855 op/s; 14600/286552 objects misplaced (5.095%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:19.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:19.734 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:19.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:19 smithi169 bash[19604]: audit 2024-02-12T16:44:18.167883+0000 mgr.y (mgr.14146) 1800 : audit [DBG] from='client.17205 v1:172.21.15.121:0/3762891711' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:44:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:19 smithi169 bash[19604]: cluster 2024-02-12T16:44:18.294184+0000 mgr.y (mgr.14146) 1801 : cluster [DBG] pgmap v2623: 47 pgs: 3 active+recovering, 44 active+clean; 8.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 102 B/s rd, 7.2 MiB/s wr, 855 op/s; 14600/286552 objects misplaced (5.095%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:19.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:19.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:19.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:19.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:20.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:20.155 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:20.259 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:20.288 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:20.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:20.421 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:20.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:20.560 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:20.656 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:44:20.664 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:20.693 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:20.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:20.822 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:20.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:20.951 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:20.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:21.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:21.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:21.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:21.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:21.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:21.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:21.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:21.351 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:21.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:21.359 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:21 smithi123 bash[19119]: cluster 2024-02-12T16:44:20.256810+0000 osd.9 (osd.9) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:21.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:21.480 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:21.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:21 smithi169 bash[19604]: cluster 2024-02-12T16:44:20.256810+0000 osd.9 (osd.9) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:21 smithi169 bash[19604]: cluster 2024-02-12T16:44:20.256864+0000 osd.9 (osd.9) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:21 smithi169 bash[19604]: cluster 2024-02-12T16:44:20.294654+0000 mgr.y (mgr.14146) 1802 : cluster [DBG] pgmap v2624: 47 pgs: 3 active+recovering, 44 active+clean; 8.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 1.1 MiB/s wr, 866 op/s; 14600/283468 objects misplaced (5.150%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:21 smithi169 bash[19604]: cluster 2024-02-12T16:44:20.354323+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e954: 12 total, 12 up, 11 in 2024-02-12T16:44:21.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:21.638 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:21 smithi123 bash[19119]: cluster 2024-02-12T16:44:20.256864+0000 osd.9 (osd.9) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:21 smithi123 bash[19119]: cluster 2024-02-12T16:44:20.294654+0000 mgr.y (mgr.14146) 1802 : cluster [DBG] pgmap v2624: 47 pgs: 3 active+recovering, 44 active+clean; 8.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 1.1 MiB/s wr, 866 op/s; 14600/283468 objects misplaced (5.150%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:21 smithi123 bash[19119]: cluster 2024-02-12T16:44:20.354323+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e954: 12 total, 12 up, 11 in 2024-02-12T16:44:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:21.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:21 smithi121 bash[18489]: cluster 2024-02-12T16:44:20.256810+0000 osd.9 (osd.9) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:21 smithi121 bash[18489]: cluster 2024-02-12T16:44:20.256864+0000 osd.9 (osd.9) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:21 smithi121 bash[18489]: cluster 2024-02-12T16:44:20.294654+0000 mgr.y (mgr.14146) 1802 : cluster [DBG] pgmap v2624: 47 pgs: 3 active+recovering, 44 active+clean; 8.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 1.1 MiB/s wr, 866 op/s; 14600/283468 objects misplaced (5.150%); 3.2 MiB/s, 51 objects/s recovering 2024-02-12T16:44:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:21 smithi121 bash[18489]: cluster 2024-02-12T16:44:20.354323+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e954: 12 total, 12 up, 11 in 2024-02-12T16:44:21.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:21.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:21.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:21.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:21.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:22.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:22.034 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:22.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:22.172 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:22.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:22.301 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:22.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:22 smithi121 bash[18489]: cluster 2024-02-12T16:44:20.644347+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:22 smithi121 bash[18489]: cluster 2024-02-12T16:44:20.644416+0000 osd.3 (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:22 smithi121 bash[18489]: cluster 2024-02-12T16:44:21.361759+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e955: 12 total, 12 up, 11 in 2024-02-12T16:44:22.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:22.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:22.545 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:22 smithi123 bash[19119]: cluster 2024-02-12T16:44:20.644347+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:22.546 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:22 smithi123 bash[19119]: cluster 2024-02-12T16:44:20.644416+0000 osd.3 (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:22.546 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:22 smithi123 bash[19119]: cluster 2024-02-12T16:44:21.361759+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e955: 12 total, 12 up, 11 in 2024-02-12T16:44:22.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:22.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:22.680 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:22 smithi169 bash[19604]: cluster 2024-02-12T16:44:20.644347+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:22.680 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:22 smithi169 bash[19604]: cluster 2024-02-12T16:44:20.644416+0000 osd.3 (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:22.681 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:22 smithi169 bash[19604]: cluster 2024-02-12T16:44:21.361759+0000 mon.a (mon.0) 3186 : cluster [DBG] osdmap e955: 12 total, 12 up, 11 in 2024-02-12T16:44:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:22.692 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:22.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:22.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:22.830 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:22.937 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:22.961 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:22.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:23.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:23.094 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:23.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:23.198 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:23.228 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:23.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:23.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:23.362 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:23.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:23.468 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:23.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:23.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:23 smithi169 bash[19604]: cluster 2024-02-12T16:44:22.295142+0000 mgr.y (mgr.14146) 1803 : cluster [DBG] pgmap v2627: 47 pgs: 3 active+recovering, 44 active+clean; 8.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 14269/277866 objects misplaced (5.135%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:23.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:23.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:23 smithi121 bash[18489]: cluster 2024-02-12T16:44:22.295142+0000 mgr.y (mgr.14146) 1803 : cluster [DBG] pgmap v2627: 47 pgs: 3 active+recovering, 44 active+clean; 8.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 14269/277866 objects misplaced (5.135%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:23.636 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:23 smithi123 bash[19119]: cluster 2024-02-12T16:44:22.295142+0000 mgr.y (mgr.14146) 1803 : cluster [DBG] pgmap v2627: 47 pgs: 3 active+recovering, 44 active+clean; 8.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 14269/277866 objects misplaced (5.135%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:23.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:23.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:23.767 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:23.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:23.871 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:23.896 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:23.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:24.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:24.054 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:24.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:24.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:24.180 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:24.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:24.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:24.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:24.448 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:24.554 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:24.584 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:24.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:24.714 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:24.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:24.842 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:24.947 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:24.991 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:24.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:25.065 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:44:25.065 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:44:25.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:11 9.d 9.a8d11bfd (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.175715+0000", 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.88518038799999, 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64770053699999997, 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 11 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175715+0000", 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175715+0000", 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175712+0000", 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7687000000000002e-05 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175810+0000", 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9610999999999997e-05 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175850+0000", 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9979999999999998e-06 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175860+0000", 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64751687700000005 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823377+0000", 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8554999999999999e-05 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823415+0000", 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64770053699999997 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:24 9.d 9.1c59118d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.175929+0000", 2024-02-12T16:44:25.069 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.884966474, 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64759051999999995, 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 24 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175929+0000", 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175929+0000", 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175927+0000", 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021789300000000001 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176145+0000", 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7478999999999998e-05 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176182+0000", 2024-02-12T16:44:25.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8719999999999997e-06 2024-02-12T16:44:25.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176192+0000", 2024-02-12T16:44:25.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64730112900000003 2024-02-12T16:44:25.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823493+0000", 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5669000000000001e-05 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823519+0000", 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64759051999999995 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:26 9.d 9.83ee18ed (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.176227+0000", 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.88466853600001, 2024-02-12T16:44:25.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64731014499999995, 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 26 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176227+0000", 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176227+0000", 2024-02-12T16:44:25.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176225+0000", 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000230604 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176456+0000", 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4382000000000001e-05 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176470+0000", 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9782000000000001e-05 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176490+0000", 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64704144900000005 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823531+0000", 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.152e-06 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823537+0000", 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64731014499999995 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:29 9.d 9.3cf8248d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.176576+0000", 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.88431944200001, 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.646997551, 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 29 2024-02-12T16:44:25.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176576+0000", 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176576+0000", 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176574+0000", 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014659699999999999 2024-02-12T16:44:25.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176721+0000", 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2809999999999999e-05 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176734+0000", 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9348999999999999e-05 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176773+0000", 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64679205900000003 2024-02-12T16:44:25.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823565+0000", 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1899999999999995e-06 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823573+0000", 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.646997551 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:47 9.d 9.59200e7d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.081 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177055+0000", 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.883839702, 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64655418099999995, 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 47 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177055+0000", 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177055+0000", 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177054+0000", 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6201e-05 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177100+0000", 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2798e-05 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177113+0000", 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5350000000000004e-06 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177121+0000", 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64648264099999997 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823604+0000", 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7559999999999996e-06 2024-02-12T16:44:25.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823610+0000", 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64655418099999995 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66 9.d 9.5f28cf2d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177228+0000", 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.88366686800001, 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64642686800000004, 2024-02-12T16:44:25.085 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177228+0000", 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177228+0000", 2024-02-12T16:44:25.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177227+0000", 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7198e-05 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177324+0000", 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0577999999999999e-05 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177334+0000", 2024-02-12T16:44:25.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0620000000000007e-06 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177343+0000", 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64628943900000002 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823633+0000", 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2263e-05 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823655+0000", 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64642686800000004 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:69 9.d 9.6c15de4d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177361+0000", 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.883533993, 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64630940299999995, 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.089 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 69 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177361+0000", 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177361+0000", 2024-02-12T16:44:25.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177359+0000", 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3695000000000001e-05 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177403+0000", 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1970000000000001e-05 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177415+0000", 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7847999999999999e-05 2024-02-12T16:44:25.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177433+0000", 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64623225900000003 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823665+0000", 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3229999999999997e-06 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823670+0000", 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64630940299999995 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:77 9.d 9.dc88835d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177450+0000", 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.883444579, 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64624936799999999, 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 77 2024-02-12T16:44:25.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177450+0000", 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177450+0000", 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177449+0000", 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1155999999999997e-05 2024-02-12T16:44:25.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177490+0000", 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3303e-05 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177504+0000", 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0199999999999994e-06 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177512+0000", 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64618072800000004 2024-02-12T16:44:25.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823692+0000", 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823700+0000", 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64624936799999999 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:16995 9.d 9:b1101a0d:::benchmark_data_smithi169_124168_object16994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e925)", 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:44.072076+0000", 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 100.988819522, 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32354703800000001, 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 16995 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072076+0000", 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072076+0000", 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072074+0000", 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8467000000000002e-05 2024-02-12T16:44:25.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072123+0000", 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2581e-05 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072135+0000", 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0599999999999996e-06 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072141+0000", 2024-02-12T16:44:25.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6850000000000001e-05 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072178+0000", 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4089e-05 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072272+0000", 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7096999999999995e-05 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072369+0000", 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059913000000000002 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072968+0000", 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32242240300000002 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395391+0000", 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133757 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395524+0000", 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8088e-05 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395623+0000", 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32354703800000001 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:17001 9.d 9:bc5e5ff6:::benchmark_data_smithi169_124168_object17000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e925)", 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:44.076282+0000", 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 100.984613283, 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31989659399999998, 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 17001 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076282+0000", 2024-02-12T16:44:25.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076282+0000", 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076280+0000", 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000147696 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076428+0000", 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2692e-05 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076441+0000", 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3680000000000004e-06 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076445+0000", 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8293000000000001e-05 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076483+0000", 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012841400000000001 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076612+0000", 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017282800000000001 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076785+0000", 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050753599999999995 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.077292+0000", 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31846323399999998 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395755+0000", 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000151831 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395907+0000", 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027103400000000002 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.396178+0000", 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31989659399999998 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:34530 9.e 9:7f31c9ea:::benchmark_data_smithi169_124168_object34529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e926)", 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:48.794362+0000", 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 96.266533339000006, 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10611812900000001, 2024-02-12T16:44:25.108 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 34530 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.794362+0000", 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.794362+0000", 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.794361+0000", 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2212000000000004e-05 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.794453+0000", 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4560000000000002e-06 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.794457+0000", 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5000000000000001e-06 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.794462+0000", 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7354000000000002e-05 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.794499+0000", 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.099690521000000004 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.894190+0000", 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039427499999999997 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.894584+0000", 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055049290000000004 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900089+0000", 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102292 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900191+0000", 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016479400000000001 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900356+0000", 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012376600000000001 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900480+0000", 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10611812900000001 2024-02-12T16:44:25.113 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:34532 9.e 9:74b9ce45:::benchmark_data_smithi169_124168_object34531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e926)", 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:48.795464+0000", 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 96.265431441999993, 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10520813599999999, 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.114 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 34532 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.795464+0000", 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.795464+0000", 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.795463+0000", 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5155999999999997e-05 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.795538+0000", 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.098424422999999997 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.893962+0000", 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2744000000000001e-05 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.893985+0000", 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030432419999999998 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.897028+0000", 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000348396 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.897377+0000", 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042342200000000001 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 10", 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.897800+0000", 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027597210000000001 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900560+0000", 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3492000000000001e-05 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900573+0000", 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5646999999999998e-05 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900629+0000", 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2796000000000002e-05 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:48.900672+0000", 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.10520813599999999 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66895 9.d 9:bbd5c3e8:::benchmark_data_smithi169_124168_object66894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:44:25.119 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.209519+0000", 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 83.851375852999993, 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42805788099999997, 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66895 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209519+0000", 2024-02-12T16:44:25.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209519+0000", 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209517+0000", 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5612000000000005e-05 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209612+0000", 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1770000000000001e-05 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209624+0000", 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1610000000000002e-06 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209629+0000", 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5796e-05 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209675+0000", 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102526 2024-02-12T16:44:25.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209778+0000", 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107908 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209885+0000", 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001571228 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.211457+0000", 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42588541000000002 2024-02-12T16:44:25.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637342+0000", 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000154523 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637497+0000", 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0442e-05 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637577+0000", 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42805788099999997 2024-02-12T16:44:25.124 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66902 9.d 9:b99a6cc2:::benchmark_data_smithi169_124168_object66901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.215250+0000", 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 83.845645302999998, 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42328817499999999, 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66902 2024-02-12T16:44:25.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215250+0000", 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215250+0000", 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215249+0000", 2024-02-12T16:44:25.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6382e-05 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215345+0000", 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.374e-06 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215353+0000", 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.101e-06 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215356+0000", 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8970000000000001e-05 2024-02-12T16:44:25.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215395+0000", 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8777999999999996e-05 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215483+0000", 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7287999999999994e-05 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215581+0000", 2024-02-12T16:44:25.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001935855 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.217517+0000", 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42092468100000002 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638441+0000", 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3287e-05 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638495+0000", 2024-02-12T16:44:25.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3380000000000001e-05 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638538+0000", 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42328817499999999 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66906 9.d 9:bfe77eb9:::benchmark_data_smithi169_124168_object66905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.215599+0000", 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 83.845295766000007, 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42307459600000002, 2024-02-12T16:44:25.130 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66906 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215599+0000", 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215599+0000", 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215598+0000", 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0842999999999998e-05 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215679+0000", 2024-02-12T16:44:25.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.216e-06 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215686+0000", 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5140000000000003e-06 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215691+0000", 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078608499999999995 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.216477+0000", 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2298000000000004e-05 2024-02-12T16:44:25.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.216559+0000", 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012651700000000001 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.216686+0000", 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00096934199999999999 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.217655+0000", 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42093917199999997 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638594+0000", 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3069000000000003e-05 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638647+0000", 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6625999999999999e-05 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638674+0000", 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42307459600000002 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:74421 9.d 9:bfca4afb:::benchmark_data_smithi169_124168_object74420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e932)", 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:04.922847+0000", 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 80.138048294000001, 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12929467, 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 74421 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.922847+0000", 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.922847+0000", 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999928 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.922839+0000", 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5308000000000006e-05 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.922905+0000", 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.294e-06 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.922912+0000", 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.58e-06 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.922916+0000", 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4480000000000002e-05 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.922950+0000", 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1749000000000005e-05 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.923042+0000", 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010302999999999999 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.923145+0000", 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021000329999999998 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.925245+0000", 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12668695399999999 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:05.051932+0000", 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014306400000000001 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:05.052075+0000", 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6458000000000006e-05 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:05.052141+0000", 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12929467 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:74429 9.d 9:b11397c3:::benchmark_data_smithi169_124168_object74428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e932)", 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:04.929310+0000", 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 80.131585129000001, 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.122959237, 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.141 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 74429 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929310+0000", 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929310+0000", 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929308+0000", 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0563999999999999e-05 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929359+0000", 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2982e-05 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929372+0000", 2024-02-12T16:44:25.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5779999999999998e-06 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929377+0000", 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3488000000000001e-05 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929421+0000", 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6714999999999998e-05 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929518+0000", 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7621999999999995e-05 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.929615+0000", 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001542812 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:04.931158+0000", 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.121054731 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:05.052213+0000", 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6232999999999997e-05 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:05.052249+0000", 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.022e-05 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:05.052269+0000", 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.122959237 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.146 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:110594 9.d 9:bb778e75:::benchmark_data_smithi169_124168_object110593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:29.072725+0000", 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 55.988169749999997, 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.109529423, 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 110594 2024-02-12T16:44:25.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072725+0000", 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072725+0000", 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072722+0000", 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0377999999999999e-05 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072762+0000", 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9975000000000001e-05 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072782+0000", 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8949999999999998e-06 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072790+0000", 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7290999999999997e-05 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072847+0000", 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121835 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.072969+0000", 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011950700000000001 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.073089+0000", 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003513024 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.076602+0000", 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.105457289 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.182059+0000", 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117058 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.182176+0000", 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8721999999999996e-05 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:29.182255+0000", 2024-02-12T16:44:25.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.109529423 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:156676 9.d 9:b89b52aa:::benchmark_data_smithi169_124168_object5992:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:13.797395+0000", 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 11.263500349999999, 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12854365700000001, 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.153 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 156676 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797395+0000", 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797395+0000", 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797392+0000", 2024-02-12T16:44:25.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6040000000000004e-06 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797400+0000", 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3053e-05 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797413+0000", 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2769999999999996e-06 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797418+0000", 2024-02-12T16:44:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9094999999999997e-05 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797467+0000", 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127222 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797595+0000", 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000163926 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.797759+0000", 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040514150000000001 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.801810+0000", 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.124005457 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.925816+0000", 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4244000000000004e-05 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.925880+0000", 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8603000000000001e-05 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.925938+0000", 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.12854365700000001 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:156685 9.d 9:b927f605:::benchmark_data_smithi169_124168_object6001:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:13.810304+0000", 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 11.250590942000001, 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11576286, 2024-02-12T16:44:25.158 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 156685 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810304+0000", 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810304+0000", 2024-02-12T16:44:25.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810302+0000", 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2540000000000003e-06 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810309+0000", 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2150000000000007e-06 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810318+0000", 2024-02-12T16:44:25.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7759999999999997e-06 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810323+0000", 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9100999999999997e-05 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810362+0000", 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6314e-05 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810458+0000", 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3902999999999999e-05 2024-02-12T16:44:25.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.810552+0000", 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037385360000000002 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.814291+0000", 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.111684145 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.925975+0000", 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4243999999999999e-05 2024-02-12T16:44:25.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.926019+0000", 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7676000000000002e-05 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:13.926067+0000", 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11576286 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:44:25.164 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:44:25.164 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:44:25.164 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:25.206 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:25.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:25.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:25.339 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:25.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:25.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:25 smithi123 bash[19119]: cluster 2024-02-12T16:44:24.295603+0000 mgr.y (mgr.14146) 1804 : cluster [DBG] pgmap v2628: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 14269/274652 objects misplaced (5.195%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:44:25.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:25.473 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:25.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:25.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:25.587 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:25 smithi169 bash[19604]: cluster 2024-02-12T16:44:24.295603+0000 mgr.y (mgr.14146) 1804 : cluster [DBG] pgmap v2628: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 14269/274652 objects misplaced (5.195%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:44:25.609 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:25.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:25 smithi121 bash[18489]: cluster 2024-02-12T16:44:24.295603+0000 mgr.y (mgr.14146) 1804 : cluster [DBG] pgmap v2628: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 14269/274652 objects misplaced (5.195%); 4.0 MiB/s, 64 objects/s recovering 2024-02-12T16:44:25.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:25.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:25.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:25.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:25.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-12T16:44:25.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:25.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:25.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:26.016 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:26.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:26.155 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:26.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:26.286 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:26.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:26.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:26.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:26.545 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:26.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:26.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:26.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:26.823 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:26.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:26.962 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:26.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:27.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:27.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:27.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:27.198 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:27.231 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:27.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:27.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:27 smithi169 bash[19604]: cluster 2024-02-12T16:44:26.296077+0000 mgr.y (mgr.14146) 1805 : cluster [DBG] pgmap v2629: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 803 op/s; 14029/273688 objects misplaced (5.126%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:27.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:27.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:27 smithi123 bash[19119]: cluster 2024-02-12T16:44:26.296077+0000 mgr.y (mgr.14146) 1805 : cluster [DBG] pgmap v2629: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 803 op/s; 14029/273688 objects misplaced (5.126%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:27.362 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:27.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:27 smithi121 bash[18489]: cluster 2024-02-12T16:44:26.296077+0000 mgr.y (mgr.14146) 1805 : cluster [DBG] pgmap v2629: 47 pgs: 3 active+recovering, 44 active+clean; 8.4 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 803 op/s; 14029/273688 objects misplaced (5.126%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:44:27.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:27.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:27.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:27.619 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:27.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:27.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:27.754 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:27.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:27.859 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:27.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:27.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:27.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:28.011 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:28.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:28.142 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:28.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:28.271 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:28.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:28.408 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:28.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:28.542 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:28.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:28.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:28.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:28.815 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:28.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:28.952 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:28.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:29.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:29.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:29.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:29.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:29.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:29.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:29.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:29.351 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:29.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:29.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:29.482 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:29.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:29.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:29.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:29.614 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:29.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:29 smithi169 bash[19604]: cluster 2024-02-12T16:44:28.296524+0000 mgr.y (mgr.14146) 1806 : cluster [DBG] pgmap v2630: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 851 op/s; 13614/269834 objects misplaced (5.045%); 7.7 MiB/s, 123 objects/s recovering 2024-02-12T16:44:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:29.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:29 smithi123 bash[19119]: cluster 2024-02-12T16:44:28.296524+0000 mgr.y (mgr.14146) 1806 : cluster [DBG] pgmap v2630: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 851 op/s; 13614/269834 objects misplaced (5.045%); 7.7 MiB/s, 123 objects/s recovering 2024-02-12T16:44:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:29 smithi121 bash[18489]: cluster 2024-02-12T16:44:28.296524+0000 mgr.y (mgr.14146) 1806 : cluster [DBG] pgmap v2630: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 851 op/s; 13614/269834 objects misplaced (5.045%); 7.7 MiB/s, 123 objects/s recovering 2024-02-12T16:44:29.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:29.755 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:29.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:29.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:29.891 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:29.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:29.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:30.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:30.135 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:30.169 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:30.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:30.298 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:30.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:30.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:30.539 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:30.547 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:30 smithi123 bash[19119]: cluster 2024-02-12T16:44:29.538997+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:30.547 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:30 smithi123 bash[19119]: cluster 2024-02-12T16:44:29.539062+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:30.575 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:30 smithi121 bash[18489]: cluster 2024-02-12T16:44:29.538997+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:30 smithi121 bash[18489]: cluster 2024-02-12T16:44:29.539062+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:30.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:30.708 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:30 smithi169 bash[19604]: cluster 2024-02-12T16:44:29.538997+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:30 smithi169 bash[19604]: cluster 2024-02-12T16:44:29.539062+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:30.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:30.843 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:30.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:30.980 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:30.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:31.084 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:31.120 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:31.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:31.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:31.250 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:31.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:31.356 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:31.362 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:31 smithi169 bash[19604]: cluster 2024-02-12T16:44:30.297036+0000 mgr.y (mgr.14146) 1807 : cluster [DBG] pgmap v2631: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 821 op/s; 13614/267136 objects misplaced (5.096%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:44:31.362 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:31 smithi169 bash[19604]: cluster 2024-02-12T16:44:30.358080+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e956: 12 total, 12 up, 11 in 2024-02-12T16:44:31.380 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:31.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:31.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:31.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:31.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:31.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:31.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:31 smithi123 bash[19119]: cluster 2024-02-12T16:44:30.297036+0000 mgr.y (mgr.14146) 1807 : cluster [DBG] pgmap v2631: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 821 op/s; 13614/267136 objects misplaced (5.096%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:44:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:31 smithi123 bash[19119]: cluster 2024-02-12T16:44:30.358080+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e956: 12 total, 12 up, 11 in 2024-02-12T16:44:31.644 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:31.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:31 smithi121 bash[18489]: cluster 2024-02-12T16:44:30.297036+0000 mgr.y (mgr.14146) 1807 : cluster [DBG] pgmap v2631: 47 pgs: 3 active+recovering, 44 active+clean; 8.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 821 op/s; 13614/267136 objects misplaced (5.096%); 6.2 MiB/s, 99 objects/s recovering 2024-02-12T16:44:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:31 smithi121 bash[18489]: cluster 2024-02-12T16:44:30.358080+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e956: 12 total, 12 up, 11 in 2024-02-12T16:44:31.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:31.776 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:31.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:31.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:31.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:31.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:32.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:32.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:32.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:32.179 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:32.259 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:44:32.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:32.286 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:44:32.315 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:44:32.316 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:32.336 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:44:32.371 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:44:32.401 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:44:32.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:32 smithi121 bash[18489]: audit 2024-02-12T16:44:32.231111+0000 mon.b (mon.2) 226 : audit [DBG] from='client.? v1:172.21.15.121:0/530742405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:32.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:32.428 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:44:32.453 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:32.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:32.456 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:44:32.489 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:44:32.517 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:44:32.542 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:44:32.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:32.570 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:44:32.572 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.572 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.572 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.572 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.572 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.572 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.573 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.573 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.573 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.573 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.573 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.573 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:44:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:32 smithi123 bash[19119]: audit 2024-02-12T16:44:32.231111+0000 mon.b (mon.2) 226 : audit [DBG] from='client.? v1:172.21.15.121:0/530742405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:32.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:32.730 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:32.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:32 smithi169 bash[19604]: audit 2024-02-12T16:44:32.231111+0000 mon.b (mon.2) 226 : audit [DBG] from='client.? v1:172.21.15.121:0/530742405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:32.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:32.861 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:32.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:32.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:32.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:33.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:33.127 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:33.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:33.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:33.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:33.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:33.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:33.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:33.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:33.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:33 smithi123 bash[19119]: cluster 2024-02-12T16:44:32.297578+0000 mgr.y (mgr.14146) 1808 : cluster [DBG] pgmap v2633: 47 pgs: 3 active+recovering, 44 active+clean; 8.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 12923/262256 objects misplaced (4.928%); 8.4 MiB/s, 134 objects/s recovering 2024-02-12T16:44:33.377 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:33 smithi123 bash[19119]: audit 2024-02-12T16:44:33.155638+0000 mon.a (mon.0) 3188 : audit [INF] from='client.27125 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:44:33.379 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:44:33.379 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 1 pgp_num to 31 2024-02-12T16:44:33.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:33.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:33.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:33.543 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:33.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:33 smithi123 bash[19119]: audit 2024-02-12T16:44:33.158472+0000 mon.b (mon.2) 227 : audit [INF] from='client.? v1:172.21.15.121:0/659491745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:44:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:33 smithi123 bash[19119]: audit 2024-02-12T16:44:33.208903+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? v1:172.21.15.121:0/1089086353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:44:33.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:33 smithi121 bash[18489]: cluster 2024-02-12T16:44:32.297578+0000 mgr.y (mgr.14146) 1808 : cluster [DBG] pgmap v2633: 47 pgs: 3 active+recovering, 44 active+clean; 8.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 12923/262256 objects misplaced (4.928%); 8.4 MiB/s, 134 objects/s recovering 2024-02-12T16:44:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:33 smithi121 bash[18489]: audit 2024-02-12T16:44:33.155638+0000 mon.a (mon.0) 3188 : audit [INF] from='client.27125 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:44:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:33 smithi121 bash[18489]: audit 2024-02-12T16:44:33.158472+0000 mon.b (mon.2) 227 : audit [INF] from='client.? v1:172.21.15.121:0/659491745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:44:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:33 smithi121 bash[18489]: audit 2024-02-12T16:44:33.208903+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? v1:172.21.15.121:0/1089086353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:44:33.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:33.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:33.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:33 smithi169 bash[19604]: cluster 2024-02-12T16:44:32.297578+0000 mgr.y (mgr.14146) 1808 : cluster [DBG] pgmap v2633: 47 pgs: 3 active+recovering, 44 active+clean; 8.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 12923/262256 objects misplaced (4.928%); 8.4 MiB/s, 134 objects/s recovering 2024-02-12T16:44:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:33 smithi169 bash[19604]: audit 2024-02-12T16:44:33.155638+0000 mon.a (mon.0) 3188 : audit [INF] from='client.27125 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:44:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:33 smithi169 bash[19604]: audit 2024-02-12T16:44:33.158472+0000 mon.b (mon.2) 227 : audit [INF] from='client.? v1:172.21.15.121:0/659491745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:44:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:33 smithi169 bash[19604]: audit 2024-02-12T16:44:33.208903+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? v1:172.21.15.121:0/1089086353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]: dispatch 2024-02-12T16:44:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:33.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:33.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:33.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:33.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:34.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:34.070 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:34.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:34.177 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:34.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:34.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:34.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:34.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:34.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:34.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:34.476 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:34.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:34 smithi169 bash[19604]: cluster 2024-02-12T16:44:33.361503+0000 mon.a (mon.0) 3190 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:34 smithi169 bash[19604]: audit 2024-02-12T16:44:33.366005+0000 mon.a (mon.0) 3191 : audit [INF] from='client.27125 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:44:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:34 smithi169 bash[19604]: audit 2024-02-12T16:44:33.366123+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? v1:172.21.15.121:0/1089086353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:44:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:34 smithi169 bash[19604]: cluster 2024-02-12T16:44:33.366213+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e957: 12 total, 12 up, 11 in 2024-02-12T16:44:34.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:34.588 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:34 smithi123 bash[19119]: cluster 2024-02-12T16:44:33.361503+0000 mon.a (mon.0) 3190 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:34.589 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:34 smithi123 bash[19119]: audit 2024-02-12T16:44:33.366005+0000 mon.a (mon.0) 3191 : audit [INF] from='client.27125 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:44:34.589 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:34 smithi123 bash[19119]: audit 2024-02-12T16:44:33.366123+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? v1:172.21.15.121:0/1089086353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:44:34.589 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:34 smithi123 bash[19119]: cluster 2024-02-12T16:44:33.366213+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e957: 12 total, 12 up, 11 in 2024-02-12T16:44:34.612 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:34.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:34 smithi121 bash[18489]: cluster 2024-02-12T16:44:33.361503+0000 mon.a (mon.0) 3190 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:34 smithi121 bash[18489]: audit 2024-02-12T16:44:33.366005+0000 mon.a (mon.0) 3191 : audit [INF] from='client.27125 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:44:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:34 smithi121 bash[18489]: audit 2024-02-12T16:44:33.366123+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? v1:172.21.15.121:0/1089086353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "31"}]': finished 2024-02-12T16:44:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:34 smithi121 bash[18489]: cluster 2024-02-12T16:44:33.366213+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e957: 12 total, 12 up, 11 in 2024-02-12T16:44:34.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:34.740 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:34.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:34.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:34.872 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:34.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:34.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:35.001 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:35.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:35.133 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:35.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:35.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:35.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:35.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:35 smithi121 bash[18489]: cluster 2024-02-12T16:44:34.009658+0000 osd.5 (osd.5) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:35.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:35 smithi121 bash[18489]: cluster 2024-02-12T16:44:34.009731+0000 osd.5 (osd.5) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:35.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:35 smithi121 bash[18489]: cluster 2024-02-12T16:44:34.298103+0000 mgr.y (mgr.14146) 1809 : cluster [DBG] pgmap v2635: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 12923/259352 objects misplaced (4.983%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:35.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:35 smithi121 bash[18489]: cluster 2024-02-12T16:44:34.465502+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e958: 12 total, 12 up, 11 in 2024-02-12T16:44:35.403 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:35.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:35.536 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:35 smithi123 bash[19119]: cluster 2024-02-12T16:44:34.009658+0000 osd.5 (osd.5) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:35 smithi123 bash[19119]: cluster 2024-02-12T16:44:34.009731+0000 osd.5 (osd.5) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:35 smithi123 bash[19119]: cluster 2024-02-12T16:44:34.298103+0000 mgr.y (mgr.14146) 1809 : cluster [DBG] pgmap v2635: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 12923/259352 objects misplaced (4.983%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:35 smithi123 bash[19119]: cluster 2024-02-12T16:44:34.465502+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e958: 12 total, 12 up, 11 in 2024-02-12T16:44:35.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:35.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:35 smithi169 bash[19604]: cluster 2024-02-12T16:44:34.009658+0000 osd.5 (osd.5) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:35.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:35 smithi169 bash[19604]: cluster 2024-02-12T16:44:34.009731+0000 osd.5 (osd.5) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:35.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:35 smithi169 bash[19604]: cluster 2024-02-12T16:44:34.298103+0000 mgr.y (mgr.14146) 1809 : cluster [DBG] pgmap v2635: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 12923/259352 objects misplaced (4.983%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:35.648 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:35 smithi169 bash[19604]: cluster 2024-02-12T16:44:34.465502+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e958: 12 total, 12 up, 11 in 2024-02-12T16:44:35.665 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:35.767 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:35.796 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:35.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:35.930 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:35.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:36.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:36.067 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:36.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:36.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:36.199 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:36.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:36.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:36.332 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:36.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:36.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:36.469 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:36.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:36.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:36.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:36.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:36.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:36.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:36.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:36.843 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:36.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:36.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:36.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:36.998 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:37.104 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:37.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:37.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:37.263 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:37.264 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:37 smithi169 bash[19604]: cluster 2024-02-12T16:44:36.298600+0000 mgr.y (mgr.14146) 1810 : cluster [DBG] pgmap v2637: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 12702/258418 objects misplaced (4.915%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T16:44:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:37.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:37 smithi123 bash[19119]: cluster 2024-02-12T16:44:36.298600+0000 mgr.y (mgr.14146) 1810 : cluster [DBG] pgmap v2637: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 12702/258418 objects misplaced (4.915%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T16:44:37.395 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:37.415 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:37 smithi121 bash[18489]: cluster 2024-02-12T16:44:36.298600+0000 mgr.y (mgr.14146) 1810 : cluster [DBG] pgmap v2637: 47 pgs: 3 active+recovering, 44 active+clean; 7.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 12702/258418 objects misplaced (4.915%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T16:44:37.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:37.531 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:37.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:37.661 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:37.768 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:37.795 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:37.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:37.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:44:37.925 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:37.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:38.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:38.071 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:38.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:38.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:38.211 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:38.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:38.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:38.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:38.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:44:38.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:38.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:38.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:38.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:38.606 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:38.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:38.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:38.738 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:38.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:38.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:38.866 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:38.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:38.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:39.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:39.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:39.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:39.270 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:39.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:39.400 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:39.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:39.544 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:39 smithi123 bash[19119]: cluster 2024-02-12T16:44:38.299149+0000 mgr.y (mgr.14146) 1811 : cluster [DBG] pgmap v2638: 47 pgs: 3 active+recovering, 44 active+clean; 7.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 12239/254760 objects misplaced (4.804%); 11 MiB/s, 173 objects/s recovering 2024-02-12T16:44:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:39 smithi123 bash[19119]: cluster 2024-02-12T16:44:38.706704+0000 osd.1 (osd.1) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:39 smithi123 bash[19119]: cluster 2024-02-12T16:44:38.706773+0000 osd.1 (osd.1) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:39.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:39 smithi121 bash[18489]: cluster 2024-02-12T16:44:38.299149+0000 mgr.y (mgr.14146) 1811 : cluster [DBG] pgmap v2638: 47 pgs: 3 active+recovering, 44 active+clean; 7.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 12239/254760 objects misplaced (4.804%); 11 MiB/s, 173 objects/s recovering 2024-02-12T16:44:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:39 smithi121 bash[18489]: cluster 2024-02-12T16:44:38.706704+0000 osd.1 (osd.1) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:39 smithi121 bash[18489]: cluster 2024-02-12T16:44:38.706773+0000 osd.1 (osd.1) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:39.675 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:39.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:39.805 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:39.910 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:39.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:39.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:40.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:40.085 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:40.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:40.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:40.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:40.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:39 smithi169 bash[19604]: cluster 2024-02-12T16:44:38.299149+0000 mgr.y (mgr.14146) 1811 : cluster [DBG] pgmap v2638: 47 pgs: 3 active+recovering, 44 active+clean; 7.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 779 op/s; 12239/254760 objects misplaced (4.804%); 11 MiB/s, 173 objects/s recovering 2024-02-12T16:44:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:39 smithi169 bash[19604]: cluster 2024-02-12T16:44:38.706704+0000 osd.1 (osd.1) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:40.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:39 smithi169 bash[19604]: cluster 2024-02-12T16:44:38.706773+0000 osd.1 (osd.1) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:40.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:40.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:40.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:40.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:40.488 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:40.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:40.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:40.598 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:40 smithi123 bash[19119]: cluster 2024-02-12T16:44:39.427051+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e959: 12 total, 12 up, 11 in 2024-02-12T16:44:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:40.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:40 smithi121 bash[18489]: cluster 2024-02-12T16:44:39.427051+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e959: 12 total, 12 up, 11 in 2024-02-12T16:44:40.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:40.729 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:40 smithi169 bash[19604]: cluster 2024-02-12T16:44:39.427051+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e959: 12 total, 12 up, 11 in 2024-02-12T16:44:40.748 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:40.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:40.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:40.881 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:40.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:40.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:41.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:41.137 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.178 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:41.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:41.315 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:41.377 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 10, 8, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:44:41.377 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:44:41.377 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 10, 8, 11] 2024-02-12T16:44:41.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd out 6 2024-02-12T16:44:41.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.448 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:41.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:41.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:41 smithi123 bash[19119]: cluster 2024-02-12T16:44:40.299604+0000 mgr.y (mgr.14146) 1812 : cluster [DBG] pgmap v2640: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 12239/251912 objects misplaced (4.858%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:44:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:41 smithi123 bash[19119]: audit 2024-02-12T16:44:40.936699+0000 mon.a (mon.0) 3196 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:41 smithi123 bash[19119]: audit 2024-02-12T16:44:41.322979+0000 mon.a (mon.0) 3197 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:44:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:41 smithi123 bash[19119]: audit 2024-02-12T16:44:41.328150+0000 mon.a (mon.0) 3198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:44:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:41 smithi123 bash[19119]: audit 2024-02-12T16:44:41.334680+0000 mon.a (mon.0) 3199 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:44:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:41 smithi121 bash[18489]: cluster 2024-02-12T16:44:40.299604+0000 mgr.y (mgr.14146) 1812 : cluster [DBG] pgmap v2640: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 12239/251912 objects misplaced (4.858%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:44:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:41 smithi121 bash[18489]: audit 2024-02-12T16:44:40.936699+0000 mon.a (mon.0) 3196 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:41 smithi121 bash[18489]: audit 2024-02-12T16:44:41.322979+0000 mon.a (mon.0) 3197 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:44:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:41 smithi121 bash[18489]: audit 2024-02-12T16:44:41.328150+0000 mon.a (mon.0) 3198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:44:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:41 smithi121 bash[18489]: audit 2024-02-12T16:44:41.334680+0000 mon.a (mon.0) 3199 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:44:41.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:41 smithi169 bash[19604]: cluster 2024-02-12T16:44:40.299604+0000 mgr.y (mgr.14146) 1812 : cluster [DBG] pgmap v2640: 47 pgs: 3 active+recovering, 44 active+clean; 7.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 12239/251912 objects misplaced (4.858%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:44:41.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:41 smithi169 bash[19604]: audit 2024-02-12T16:44:40.936699+0000 mon.a (mon.0) 3196 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:44:41.688 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:41 smithi169 bash[19604]: audit 2024-02-12T16:44:41.322979+0000 mon.a (mon.0) 3197 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:44:41.689 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:41 smithi169 bash[19604]: audit 2024-02-12T16:44:41.328150+0000 mon.a (mon.0) 3198 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:44:41.689 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:41 smithi169 bash[19604]: audit 2024-02-12T16:44:41.334680+0000 mon.a (mon.0) 3199 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:44:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:41.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:41.841 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:41.949 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.978 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:41.981 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:42.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:42.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:42.212 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:42.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:42.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:42.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:42.384 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:42.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:42.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:42.525 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:42.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:42.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:42.661 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:42.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:42.713 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:42.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:42.788 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:42.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:42.892 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:42.924 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:42.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:43.029 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:43.056 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:43.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:43.162 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:43.186 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:43.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:43.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:43.328 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:43.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:43.435 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:43.470 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:43.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:43.516 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:44:43.516 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:44:43.559 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:43 smithi121 bash[18489]: cluster 2024-02-12T16:44:42.300251+0000 mgr.y (mgr.14146) 1813 : cluster [DBG] pgmap v2641: 47 pgs: 3 active+recovering, 44 active+clean; 7.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 767 op/s; 11556/247062 objects misplaced (4.677%); 11 MiB/s, 170 objects/s recovering 2024-02-12T16:44:43.571 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:43.600 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:43.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:43.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:43.743 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:43.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:43 smithi169 bash[19604]: cluster 2024-02-12T16:44:42.300251+0000 mgr.y (mgr.14146) 1813 : cluster [DBG] pgmap v2641: 47 pgs: 3 active+recovering, 44 active+clean; 7.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 767 op/s; 11556/247062 objects misplaced (4.677%); 11 MiB/s, 170 objects/s recovering 2024-02-12T16:44:43.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:43.874 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:43.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:43 smithi123 bash[19119]: cluster 2024-02-12T16:44:42.300251+0000 mgr.y (mgr.14146) 1813 : cluster [DBG] pgmap v2641: 47 pgs: 3 active+recovering, 44 active+clean; 7.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 767 op/s; 11556/247062 objects misplaced (4.677%); 11 MiB/s, 170 objects/s recovering 2024-02-12T16:44:43.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:44.000 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:44.103 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:44.130 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:44.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:44.259 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:44.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:44.398 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:44.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:44.539 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:44.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:44.670 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:44.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:44.800 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:44.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-12T16:44:44.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:44.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:45.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:45.097 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:45.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:45.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:45.239 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:45.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:45.345 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:45.376 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:45.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:45.450 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:45 smithi169 bash[19604]: cluster 2024-02-12T16:44:44.300749+0000 mgr.y (mgr.14146) 1814 : cluster [DBG] pgmap v2642: 47 pgs: 3 active+recovering, 44 active+clean; 7.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 11556/244036 objects misplaced (4.735%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T16:44:45.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:45.511 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:45.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:45.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:45.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:45 smithi123 bash[19119]: cluster 2024-02-12T16:44:44.300749+0000 mgr.y (mgr.14146) 1814 : cluster [DBG] pgmap v2642: 47 pgs: 3 active+recovering, 44 active+clean; 7.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 11556/244036 objects misplaced (4.735%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T16:44:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:45.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:45.750 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:45.757 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:45 smithi121 bash[18489]: cluster 2024-02-12T16:44:44.300749+0000 mgr.y (mgr.14146) 1814 : cluster [DBG] pgmap v2642: 47 pgs: 3 active+recovering, 44 active+clean; 7.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 11556/244036 objects misplaced (4.735%); 8.6 MiB/s, 137 objects/s recovering 2024-02-12T16:44:45.781 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:45.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:45.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:45.908 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:45.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:46.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:46.040 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:46.145 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:46.174 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:46.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:46.307 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:46.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:46.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:46.570 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:46.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:46.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:46.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:46.834 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:46.938 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:46.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:46.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:47.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:47.102 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:47.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:47.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:47.246 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:47.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:47.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:47.374 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:47.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:47.478 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:47.509 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:47.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:47.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:47.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:47 smithi169 bash[19604]: cluster 2024-02-12T16:44:45.863904+0000 osd.4 (osd.4) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:47.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:47 smithi169 bash[19604]: cluster 2024-02-12T16:44:45.863954+0000 osd.4 (osd.4) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:47.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:47 smithi169 bash[19604]: cluster 2024-02-12T16:44:46.301156+0000 mgr.y (mgr.14146) 1815 : cluster [DBG] pgmap v2643: 47 pgs: 3 active+recovering, 44 active+clean; 7.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 11321/243084 objects misplaced (4.657%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:47.619 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:47 smithi169 bash[19604]: cluster 2024-02-12T16:44:46.445646+0000 mon.a (mon.0) 3200 : cluster [DBG] osdmap e960: 12 total, 12 up, 11 in 2024-02-12T16:44:47.637 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:47.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:47.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:47.748 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:47 smithi123 bash[19119]: cluster 2024-02-12T16:44:45.863904+0000 osd.4 (osd.4) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:47.748 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:47 smithi123 bash[19119]: cluster 2024-02-12T16:44:45.863954+0000 osd.4 (osd.4) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:47.748 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:47 smithi123 bash[19119]: cluster 2024-02-12T16:44:46.301156+0000 mgr.y (mgr.14146) 1815 : cluster [DBG] pgmap v2643: 47 pgs: 3 active+recovering, 44 active+clean; 7.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 11321/243084 objects misplaced (4.657%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:47.748 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:47 smithi123 bash[19119]: cluster 2024-02-12T16:44:46.445646+0000 mon.a (mon.0) 3200 : cluster [DBG] osdmap e960: 12 total, 12 up, 11 in 2024-02-12T16:44:47.770 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:47.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:47.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:47.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:47 smithi121 bash[18489]: cluster 2024-02-12T16:44:45.863904+0000 osd.4 (osd.4) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:47.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:47 smithi121 bash[18489]: cluster 2024-02-12T16:44:45.863954+0000 osd.4 (osd.4) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:47.880 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:47 smithi121 bash[18489]: cluster 2024-02-12T16:44:46.301156+0000 mgr.y (mgr.14146) 1815 : cluster [DBG] pgmap v2643: 47 pgs: 3 active+recovering, 44 active+clean; 7.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 11321/243084 objects misplaced (4.657%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:47.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:47 smithi121 bash[18489]: cluster 2024-02-12T16:44:46.445646+0000 mon.a (mon.0) 3200 : cluster [DBG] osdmap e960: 12 total, 12 up, 11 in 2024-02-12T16:44:47.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:47.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:48.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:48.090 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:48.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:48.176 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:48.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:48.304 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:48.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:48.436 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:48.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:48.571 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:48.676 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:48.706 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:48.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:48.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:48.941 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:48.967 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:48.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:49.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:49.102 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:49.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:49.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:49.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:49.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:49.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:49.369 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:49.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:49.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:49.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:49.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:49 smithi169 bash[19604]: cluster 2024-02-12T16:44:48.301610+0000 mgr.y (mgr.14146) 1816 : cluster [DBG] pgmap v2645: 47 pgs: 3 active+recovering, 44 active+clean; 7.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 10856/238772 objects misplaced (4.547%); 9.7 MiB/s, 155 objects/s recovering 2024-02-12T16:44:49.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:49.633 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:49.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:49.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:49.744 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:49 smithi121 bash[18489]: cluster 2024-02-12T16:44:48.301610+0000 mgr.y (mgr.14146) 1816 : cluster [DBG] pgmap v2645: 47 pgs: 3 active+recovering, 44 active+clean; 7.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 10856/238772 objects misplaced (4.547%); 9.7 MiB/s, 155 objects/s recovering 2024-02-12T16:44:49.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:49.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:49.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:49 smithi123 bash[19119]: cluster 2024-02-12T16:44:48.301610+0000 mgr.y (mgr.14146) 1816 : cluster [DBG] pgmap v2645: 47 pgs: 3 active+recovering, 44 active+clean; 7.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 10856/238772 objects misplaced (4.547%); 9.7 MiB/s, 155 objects/s recovering 2024-02-12T16:44:49.893 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:49.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:49.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:50.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:50.153 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:50.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:50.286 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:50.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:50.418 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:50.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:50.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:50.668 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:50.678 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:44:50.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:50.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:50.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:50.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:50.971 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:50.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:51.054 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:44:51.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:51.088 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:44:51.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:51.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:51.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:44:51.144 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:44:51.180 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:44:51.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:51.215 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:44:51.233 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:51.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:51.247 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:44:51.272 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:44:51.300 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:44:51.329 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:44:51.336 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:51.358 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:44:51.374 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:51.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:51.386 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.389 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.389 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:44:51.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:51.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:51 smithi169 bash[19604]: cluster 2024-02-12T16:44:50.302098+0000 mgr.y (mgr.14146) 1817 : cluster [DBG] pgmap v2646: 47 pgs: 3 active+recovering, 44 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 10856/235602 objects misplaced (4.608%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:51.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:51 smithi169 bash[19604]: audit 2024-02-12T16:44:50.996569+0000 mon.b (mon.2) 228 : audit [DBG] from='client.? v1:172.21.15.121:0/2975667285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:51.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:51 smithi169 bash[19604]: audit 2024-02-12T16:44:51.328882+0000 mon.a (mon.0) 3201 : audit [INF] from='client.? v1:172.21.15.121:0/1764923353' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:44:51.510 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:51.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:51.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:51 smithi123 bash[19119]: cluster 2024-02-12T16:44:50.302098+0000 mgr.y (mgr.14146) 1817 : cluster [DBG] pgmap v2646: 47 pgs: 3 active+recovering, 44 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 10856/235602 objects misplaced (4.608%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:51 smithi123 bash[19119]: audit 2024-02-12T16:44:50.996569+0000 mon.b (mon.2) 228 : audit [DBG] from='client.? v1:172.21.15.121:0/2975667285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:51 smithi123 bash[19119]: audit 2024-02-12T16:44:51.328882+0000 mon.a (mon.0) 3201 : audit [INF] from='client.? v1:172.21.15.121:0/1764923353' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:44:51.640 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:51.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:51.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:51.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:51 smithi121 bash[18489]: cluster 2024-02-12T16:44:50.302098+0000 mgr.y (mgr.14146) 1817 : cluster [DBG] pgmap v2646: 47 pgs: 3 active+recovering, 44 active+clean; 7.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 10856/235602 objects misplaced (4.608%); 8.6 MiB/s, 138 objects/s recovering 2024-02-12T16:44:51.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:51 smithi121 bash[18489]: audit 2024-02-12T16:44:50.996569+0000 mon.b (mon.2) 228 : audit [DBG] from='client.? v1:172.21.15.121:0/2975667285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:44:51.753 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:51 smithi121 bash[18489]: audit 2024-02-12T16:44:51.328882+0000 mon.a (mon.0) 3201 : audit [INF] from='client.? v1:172.21.15.121:0/1764923353' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:44:51.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:51.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:51.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:51.909 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:51.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:51.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:52.012 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:44:52.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:52.046 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:52.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:52.176 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:52.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:52.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:52.419 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:52.450 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:52.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:52.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:52 smithi121 bash[18489]: cluster 2024-02-12T16:44:51.994072+0000 mon.a (mon.0) 3202 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:52 smithi121 bash[18489]: audit 2024-02-12T16:44:52.000913+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? v1:172.21.15.121:0/1764923353' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:44:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:52 smithi121 bash[18489]: cluster 2024-02-12T16:44:52.001063+0000 mon.a (mon.0) 3204 : cluster [DBG] osdmap e961: 12 total, 12 up, 11 in 2024-02-12T16:44:52.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:52.694 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:52 smithi123 bash[19119]: cluster 2024-02-12T16:44:51.994072+0000 mon.a (mon.0) 3202 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:52.694 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:52 smithi123 bash[19119]: audit 2024-02-12T16:44:52.000913+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? v1:172.21.15.121:0/1764923353' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:44:52.694 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:52 smithi123 bash[19119]: cluster 2024-02-12T16:44:52.001063+0000 mon.a (mon.0) 3204 : cluster [DBG] osdmap e961: 12 total, 12 up, 11 in 2024-02-12T16:44:52.713 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:52.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:52 smithi169 bash[19604]: cluster 2024-02-12T16:44:51.994072+0000 mon.a (mon.0) 3202 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:44:52.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:52 smithi169 bash[19604]: audit 2024-02-12T16:44:52.000913+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? v1:172.21.15.121:0/1764923353' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:44:52.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:52 smithi169 bash[19604]: cluster 2024-02-12T16:44:52.001063+0000 mon.a (mon.0) 3204 : cluster [DBG] osdmap e961: 12 total, 12 up, 11 in 2024-02-12T16:44:52.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:52.849 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:52.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:52.976 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:52.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:53.081 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:53.111 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:53.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:53.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:53.243 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:53.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:53.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:53.377 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:53.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:53.457 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:53 smithi169 bash[19604]: cluster 2024-02-12T16:44:52.302577+0000 mgr.y (mgr.14146) 1818 : cluster [DBG] pgmap v2648: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 847 op/s; 10153/230468 objects misplaced (4.405%); 11 MiB/s, 175 objects/s recovering 2024-02-12T16:44:53.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:53.504 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:53.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:53.522 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:53 smithi121 bash[18489]: cluster 2024-02-12T16:44:52.302577+0000 mgr.y (mgr.14146) 1818 : cluster [DBG] pgmap v2648: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 847 op/s; 10153/230468 objects misplaced (4.405%); 11 MiB/s, 175 objects/s recovering 2024-02-12T16:44:53.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:53.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:53.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:53.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:53.776 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:53.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:53.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:53 smithi123 bash[19119]: cluster 2024-02-12T16:44:52.302577+0000 mgr.y (mgr.14146) 1818 : cluster [DBG] pgmap v2648: 47 pgs: 3 active+recovering, 44 active+clean; 7.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 847 op/s; 10153/230468 objects misplaced (4.405%); 11 MiB/s, 175 objects/s recovering 2024-02-12T16:44:53.915 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:53.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:54.018 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.056 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:54.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.190 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:54.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:54.326 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:54.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:54.465 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:54.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:54.595 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:54.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.725 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:54.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:54.862 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:54.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:54.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.997 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:54.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:55.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:55.123 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:44:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:55.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:44:55.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:55.254 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:55.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:55.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:55.392 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:55.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:55.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:55.521 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:55.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:55 smithi169 bash[19604]: cluster 2024-02-12T16:44:54.145237+0000 osd.11 (osd.11) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:55 smithi169 bash[19604]: cluster 2024-02-12T16:44:54.303019+0000 mgr.y (mgr.14146) 1819 : cluster [DBG] pgmap v2649: 47 pgs: 3 active+recovering, 44 active+clean; 6.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 972 op/s; 10153/227520 objects misplaced (4.462%); 9.1 MiB/s, 145 objects/s recovering 2024-02-12T16:44:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:55 smithi169 bash[19604]: cluster 2024-02-12T16:44:54.462361+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e962: 12 total, 12 up, 11 in 2024-02-12T16:44:55.625 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:55.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:55 smithi123 bash[19119]: cluster 2024-02-12T16:44:54.145237+0000 osd.11 (osd.11) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:55.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:55 smithi123 bash[19119]: cluster 2024-02-12T16:44:54.303019+0000 mgr.y (mgr.14146) 1819 : cluster [DBG] pgmap v2649: 47 pgs: 3 active+recovering, 44 active+clean; 6.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 972 op/s; 10153/227520 objects misplaced (4.462%); 9.1 MiB/s, 145 objects/s recovering 2024-02-12T16:44:55.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:55 smithi123 bash[19119]: cluster 2024-02-12T16:44:54.462361+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e962: 12 total, 12 up, 11 in 2024-02-12T16:44:55.647 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:44:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:55.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:44:55.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:55.777 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:55.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:55.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:55.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:55 smithi121 bash[18489]: cluster 2024-02-12T16:44:54.145237+0000 osd.11 (osd.11) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:55.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:55 smithi121 bash[18489]: cluster 2024-02-12T16:44:54.303019+0000 mgr.y (mgr.14146) 1819 : cluster [DBG] pgmap v2649: 47 pgs: 3 active+recovering, 44 active+clean; 6.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 972 op/s; 10153/227520 objects misplaced (4.462%); 9.1 MiB/s, 145 objects/s recovering 2024-02-12T16:44:55.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:55 smithi121 bash[18489]: cluster 2024-02-12T16:44:54.462361+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e962: 12 total, 12 up, 11 in 2024-02-12T16:44:55.910 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:55.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:56.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:56.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:56.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:56.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:56.295 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:56.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:56.425 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:56.529 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:56.555 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:56.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:56.687 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:56.790 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:56.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:56.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:56.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:56.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:56.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:56.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:44:57.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:44:57.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:57.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:57.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:57.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:57 smithi123 bash[19119]: cluster 2024-02-12T16:44:56.303719+0000 mgr.y (mgr.14146) 1820 : cluster [DBG] pgmap v2651: 47 pgs: 3 active+recovering, 44 active+clean; 6.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 9883/226418 objects misplaced (4.365%); 7.6 MiB/s, 121 objects/s recovering 2024-02-12T16:44:57.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:57.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:57 smithi121 bash[18489]: cluster 2024-02-12T16:44:56.303719+0000 mgr.y (mgr.14146) 1820 : cluster [DBG] pgmap v2651: 47 pgs: 3 active+recovering, 44 active+clean; 6.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 9883/226418 objects misplaced (4.365%); 7.6 MiB/s, 121 objects/s recovering 2024-02-12T16:44:57.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:57.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:57 smithi169 bash[19604]: cluster 2024-02-12T16:44:56.303719+0000 mgr.y (mgr.14146) 1820 : cluster [DBG] pgmap v2651: 47 pgs: 3 active+recovering, 44 active+clean; 6.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 9883/226418 objects misplaced (4.365%); 7.6 MiB/s, 121 objects/s recovering 2024-02-12T16:44:57.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:44:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:57.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:44:57.471 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:57.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:57.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:57.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:57.629 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:44:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:57.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:44:57.732 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:57.758 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:57.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:57.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:57.888 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:57.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:57.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:58.031 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:58.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:58.168 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:58.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:58.288 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:58 smithi121 bash[18489]: cluster 2024-02-12T16:44:57.703904+0000 osd.6 (osd.6) 190 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:58.288 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:58 smithi121 bash[18489]: cluster 2024-02-12T16:44:57.703970+0000 osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:58 smithi169 bash[19604]: cluster 2024-02-12T16:44:57.703904+0000 osd.6 (osd.6) 190 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:58.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:58 smithi169 bash[19604]: cluster 2024-02-12T16:44:57.703970+0000 osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:58.306 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:58 smithi123 bash[19119]: cluster 2024-02-12T16:44:57.703904+0000 osd.6 (osd.6) 190 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:44:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:58 smithi123 bash[19119]: cluster 2024-02-12T16:44:57.703970+0000 osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:44:58.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:58.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:58.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:58.569 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:44:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:44:58.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:58.700 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:58.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:58.829 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:58.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:58.960 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:44:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:58.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:44:59.037 INFO:teuthology.orchestra.run.smithi121.stderr:marked out osd.6. 2024-02-12T16:44:59.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:59.101 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:44:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:59.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:44:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:59 smithi123 bash[19119]: cluster 2024-02-12T16:44:58.022289+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e963: 12 total, 12 up, 11 in 2024-02-12T16:44:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:59 smithi123 bash[19119]: cluster 2024-02-12T16:44:58.304243+0000 mgr.y (mgr.14146) 1821 : cluster [DBG] pgmap v2653: 47 pgs: 3 active+recovering, 44 active+clean; 6.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 9397/222528 objects misplaced (4.223%); 7.5 MiB/s, 119 objects/s recovering 2024-02-12T16:44:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:59 smithi123 bash[19119]: audit 2024-02-12T16:44:58.960899+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? v1:172.21.15.121:0/2072425354' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:44:59.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:44:59 smithi123 bash[19119]: cluster 2024-02-12T16:44:58.961032+0000 mon.a (mon.0) 3208 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-12T16:44:59.204 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:59.233 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:44:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:59.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:44:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:59 smithi169 bash[19604]: cluster 2024-02-12T16:44:58.022289+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e963: 12 total, 12 up, 11 in 2024-02-12T16:44:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:59 smithi169 bash[19604]: cluster 2024-02-12T16:44:58.304243+0000 mgr.y (mgr.14146) 1821 : cluster [DBG] pgmap v2653: 47 pgs: 3 active+recovering, 44 active+clean; 6.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 9397/222528 objects misplaced (4.223%); 7.5 MiB/s, 119 objects/s recovering 2024-02-12T16:44:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:59 smithi169 bash[19604]: audit 2024-02-12T16:44:58.960899+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? v1:172.21.15.121:0/2072425354' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:44:59.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:44:59 smithi169 bash[19604]: cluster 2024-02-12T16:44:58.961032+0000 mon.a (mon.0) 3208 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-12T16:44:59.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:59.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:58 smithi121 bash[18489]: debug 2024-02-12T16:44:58.955+0000 7fbba85de700 -1 mon.a@0(leader).osd e963 definitely_dead 0 2024-02-12T16:44:59.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:59 smithi121 bash[18489]: cluster 2024-02-12T16:44:58.022289+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e963: 12 total, 12 up, 11 in 2024-02-12T16:44:59.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:59 smithi121 bash[18489]: cluster 2024-02-12T16:44:58.304243+0000 mgr.y (mgr.14146) 1821 : cluster [DBG] pgmap v2653: 47 pgs: 3 active+recovering, 44 active+clean; 6.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 9397/222528 objects misplaced (4.223%); 7.5 MiB/s, 119 objects/s recovering 2024-02-12T16:44:59.343 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:59 smithi121 bash[18489]: audit 2024-02-12T16:44:58.960899+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? v1:172.21.15.121:0/2072425354' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-12T16:44:59.343 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:44:59 smithi121 bash[18489]: cluster 2024-02-12T16:44:58.961032+0000 mon.a (mon.0) 3208 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-12T16:44:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:59.360 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:44:59.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:44:59.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:59.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:59.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:44:59.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:59.642 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:44:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:59.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:44:59.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:59.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:44:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:59.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:44:59.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:59.924 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:44:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:44:59.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:00.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:00.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:00.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:00.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:00 smithi123 bash[19119]: audit 2024-02-12T16:44:59.027077+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? v1:172.21.15.121:0/2072425354' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-12T16:45:00.081 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:00 smithi123 bash[19119]: cluster 2024-02-12T16:44:59.027263+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e964: 12 total, 12 up, 10 in 2024-02-12T16:45:00.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:00.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:00 smithi121 bash[18489]: audit 2024-02-12T16:44:59.027077+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? v1:172.21.15.121:0/2072425354' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-12T16:45:00.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:00 smithi121 bash[18489]: cluster 2024-02-12T16:44:59.027263+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e964: 12 total, 12 up, 10 in 2024-02-12T16:45:00.189 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:00.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:00.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:00.323 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:00.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:00.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:00.434 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:00 smithi169 bash[19604]: audit 2024-02-12T16:44:59.027077+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? v1:172.21.15.121:0/2072425354' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-12T16:45:00.434 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:00 smithi169 bash[19604]: cluster 2024-02-12T16:44:59.027263+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e964: 12 total, 12 up, 10 in 2024-02-12T16:45:00.457 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:00.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:00.561 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:00.589 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:00.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:00.695 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:00.732 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:00.732 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:45:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:45:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:45:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:45:00.732 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:45:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:00.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:00.836 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:00.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:00.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:00.973 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:01.008 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:01.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:01.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:01 smithi169 bash[19604]: cluster 2024-02-12T16:44:59.967114+0000 osd.2 (osd.2) 141 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:45:01.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:01 smithi169 bash[19604]: cluster 2024-02-12T16:45:00.040755+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e965: 12 total, 12 up, 10 in 2024-02-12T16:45:01.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:01 smithi169 bash[19604]: cluster 2024-02-12T16:45:00.089748+0000 osd.2 (osd.2) 142 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:45:01.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:01 smithi169 bash[19604]: cluster 2024-02-12T16:45:00.304663+0000 mgr.y (mgr.14146) 1822 : cluster [DBG] pgmap v2656: 47 pgs: 1 peering, 3 active+recovering, 43 active+clean; 6.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 680 op/s; 9397/219566 objects misplaced (4.280%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T16:45:01.120 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:01 smithi169 bash[19604]: cluster 2024-02-12T16:45:01.029396+0000 mon.a (mon.0) 3212 : cluster [DBG] osdmap e966: 12 total, 12 up, 10 in 2024-02-12T16:45:01.142 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:01.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:01.269 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:01.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:01.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:01 smithi121 bash[18489]: cluster 2024-02-12T16:44:59.967114+0000 osd.2 (osd.2) 141 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:45:01.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:01 smithi121 bash[18489]: cluster 2024-02-12T16:45:00.040755+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e965: 12 total, 12 up, 10 in 2024-02-12T16:45:01.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:01 smithi121 bash[18489]: cluster 2024-02-12T16:45:00.089748+0000 osd.2 (osd.2) 142 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:45:01.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:01 smithi121 bash[18489]: cluster 2024-02-12T16:45:00.304663+0000 mgr.y (mgr.14146) 1822 : cluster [DBG] pgmap v2656: 47 pgs: 1 peering, 3 active+recovering, 43 active+clean; 6.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 680 op/s; 9397/219566 objects misplaced (4.280%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T16:45:01.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:01 smithi121 bash[18489]: cluster 2024-02-12T16:45:01.029396+0000 mon.a (mon.0) 3212 : cluster [DBG] osdmap e966: 12 total, 12 up, 10 in 2024-02-12T16:45:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:01 smithi123 bash[19119]: cluster 2024-02-12T16:44:59.967114+0000 osd.2 (osd.2) 141 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:45:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:01 smithi123 bash[19119]: cluster 2024-02-12T16:45:00.040755+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e965: 12 total, 12 up, 10 in 2024-02-12T16:45:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:01 smithi123 bash[19119]: cluster 2024-02-12T16:45:00.089748+0000 osd.2 (osd.2) 142 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:45:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:01 smithi123 bash[19119]: cluster 2024-02-12T16:45:00.304663+0000 mgr.y (mgr.14146) 1822 : cluster [DBG] pgmap v2656: 47 pgs: 1 peering, 3 active+recovering, 43 active+clean; 6.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 680 op/s; 9397/219566 objects misplaced (4.280%); 8.1 MiB/s, 129 objects/s recovering 2024-02-12T16:45:01.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:01 smithi123 bash[19119]: cluster 2024-02-12T16:45:01.029396+0000 mon.a (mon.0) 3212 : cluster [DBG] osdmap e966: 12 total, 12 up, 10 in 2024-02-12T16:45:01.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:01.510 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:01.540 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:01.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:01.676 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:01.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-12T16:45:01.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:01.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:01.917 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:01.944 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:01.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:02.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:02.053 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:02 smithi121 bash[18489]: cluster 2024-02-12T16:45:00.796685+0000 osd.3 (osd.3) 115 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:45:02.054 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:02 smithi123 bash[19119]: cluster 2024-02-12T16:45:00.796685+0000 osd.3 (osd.3) 115 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:45:02.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:02 smithi123 bash[19119]: cluster 2024-02-12T16:45:00.933024+0000 osd.3 (osd.3) 116 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:45:02.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:02 smithi123 bash[19119]: cluster 2024-02-12T16:45:01.030936+0000 osd.1 (osd.1) 169 : cluster [DBG] 9.0 starting backfill to osd.11 from (0'0,0'0] MAX to 963'12057 2024-02-12T16:45:02.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:02 smithi123 bash[19119]: cluster 2024-02-12T16:45:01.032590+0000 osd.3 (osd.3) 117 : cluster [DBG] 9.1 starting backfill to osd.10 from (0'0,0'0] MAX to 963'11779 2024-02-12T16:45:02.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:02 smithi123 bash[19119]: cluster 2024-02-12T16:45:01.043063+0000 mon.a (mon.0) 3213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:45:02.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:02 smithi123 bash[19119]: cluster 2024-02-12T16:45:01.848213+0000 osd.7 (osd.7) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:02.055 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:02 smithi123 bash[19119]: cluster 2024-02-12T16:45:01.848293+0000 osd.7 (osd.7) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:02.081 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:02.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:02.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:02.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:02.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:02.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:02.318 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:02 smithi121 bash[18489]: cluster 2024-02-12T16:45:00.933024+0000 osd.3 (osd.3) 116 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:45:02.318 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:02 smithi121 bash[18489]: cluster 2024-02-12T16:45:01.030936+0000 osd.1 (osd.1) 169 : cluster [DBG] 9.0 starting backfill to osd.11 from (0'0,0'0] MAX to 963'12057 2024-02-12T16:45:02.318 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:02 smithi121 bash[18489]: cluster 2024-02-12T16:45:01.032590+0000 osd.3 (osd.3) 117 : cluster [DBG] 9.1 starting backfill to osd.10 from (0'0,0'0] MAX to 963'11779 2024-02-12T16:45:02.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:02 smithi121 bash[18489]: cluster 2024-02-12T16:45:01.043063+0000 mon.a (mon.0) 3213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:45:02.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:02 smithi121 bash[18489]: cluster 2024-02-12T16:45:01.848213+0000 osd.7 (osd.7) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:02.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:02 smithi121 bash[18489]: cluster 2024-02-12T16:45:01.848293+0000 osd.7 (osd.7) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:02.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:02.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:02.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:02.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:02 smithi169 bash[19604]: cluster 2024-02-12T16:45:00.796685+0000 osd.3 (osd.3) 115 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:45:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:02 smithi169 bash[19604]: cluster 2024-02-12T16:45:00.933024+0000 osd.3 (osd.3) 116 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:45:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:02 smithi169 bash[19604]: cluster 2024-02-12T16:45:01.030936+0000 osd.1 (osd.1) 169 : cluster [DBG] 9.0 starting backfill to osd.11 from (0'0,0'0] MAX to 963'12057 2024-02-12T16:45:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:02 smithi169 bash[19604]: cluster 2024-02-12T16:45:01.032590+0000 osd.3 (osd.3) 117 : cluster [DBG] 9.1 starting backfill to osd.10 from (0'0,0'0] MAX to 963'11779 2024-02-12T16:45:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:02 smithi169 bash[19604]: cluster 2024-02-12T16:45:01.043063+0000 mon.a (mon.0) 3213 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-12T16:45:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:02 smithi169 bash[19604]: cluster 2024-02-12T16:45:01.848213+0000 osd.7 (osd.7) 11 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:02.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:02 smithi169 bash[19604]: cluster 2024-02-12T16:45:01.848293+0000 osd.7 (osd.7) 12 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:02.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:02.602 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:02.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:02.706 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:02.734 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:02.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:02.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:02.869 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:02.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:02.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:03.002 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:03.014 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:03.017 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:03.107 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:03.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:03 smithi121 bash[18489]: cluster 2024-02-12T16:45:02.057067+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e967: 12 total, 12 up, 10 in 2024-02-12T16:45:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:03 smithi121 bash[18489]: cluster 2024-02-12T16:45:02.305090+0000 mgr.y (mgr.14146) 1823 : cluster [DBG] pgmap v2659: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 1 peering, 2 active+recovering, 42 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 765 op/s; 15587/216402 objects misplaced (7.203%); 7.4 MiB/s, 118 objects/s recovering 2024-02-12T16:45:03.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:03.268 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:03.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:03.378 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:03 smithi123 bash[19119]: cluster 2024-02-12T16:45:02.057067+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e967: 12 total, 12 up, 10 in 2024-02-12T16:45:03.379 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:03 smithi123 bash[19119]: cluster 2024-02-12T16:45:02.305090+0000 mgr.y (mgr.14146) 1823 : cluster [DBG] pgmap v2659: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 1 peering, 2 active+recovering, 42 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 765 op/s; 15587/216402 objects misplaced (7.203%); 7.4 MiB/s, 118 objects/s recovering 2024-02-12T16:45:03.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:03.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:03.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:03.531 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:03 smithi169 bash[19604]: cluster 2024-02-12T16:45:02.057067+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e967: 12 total, 12 up, 10 in 2024-02-12T16:45:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:03 smithi169 bash[19604]: cluster 2024-02-12T16:45:02.305090+0000 mgr.y (mgr.14146) 1823 : cluster [DBG] pgmap v2659: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 1 peering, 2 active+recovering, 42 active+clean; 6.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 765 op/s; 15587/216402 objects misplaced (7.203%); 7.4 MiB/s, 118 objects/s recovering 2024-02-12T16:45:03.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:03.668 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:03.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:03.807 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:03.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:03.944 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:03.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:04.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:04.084 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:04.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:04.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.054620+0000 mon.a (mon.0) 3215 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.901526+0000 mon.a (mon.0) 3216 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=4116140735,config=2903391811,health=27270746,kv=907397368}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.902587+0000 mon.a (mon.0) 3217 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3221839024,logm=320555359}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.903524+0000 mon.a (mon.0) 3218 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1215561094}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.904371+0000 mon.a (mon.0) 3219 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1810830127}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.905258+0000 mon.a (mon.0) 3220 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2904458400}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.906978+0000 mon.a (mon.0) 3221 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302155314}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.911198+0000 mon.a (mon.0) 3222 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=741982891}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.915596+0000 mon.a (mon.0) 3223 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=358927418}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.919677+0000 mon.a (mon.0) 3224 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2744939720}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.923731+0000 mon.a (mon.0) 3225 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2593003180}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.927790+0000 mon.a (mon.0) 3226 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734474486}) 2024-02-12T16:45:04.195 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.929972+0000 mon.a (mon.0) 3227 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=22,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=38} crc {logm=1505259141,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3906500417,mon_config_key=310623174}) 2024-02-12T16:45:04.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.930885+0000 mon.a (mon.0) 3228 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3130161295}) 2024-02-12T16:45:04.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.931854+0000 mon.a (mon.0) 3229 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1815389167}) 2024-02-12T16:45:04.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.932978+0000 mon.a (mon.0) 3230 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=74} crc {mon_config_key=2796904410,monmap=2720272532,osd_metadata=4093509710,osd_pg_creating=1122032377,osdmap=517479979}) 2024-02-12T16:45:04.196 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:04 smithi169 bash[19604]: cluster 2024-02-12T16:45:03.943456+0000 mon.a (mon.0) 3231 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=41} crc {osdmap=658076882}) 2024-02-12T16:45:04.219 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:04.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:04.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:04.356 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:04.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.054620+0000 mon.a (mon.0) 3215 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.901526+0000 mon.a (mon.0) 3216 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=4116140735,config=2903391811,health=27270746,kv=907397368}) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.902587+0000 mon.a (mon.0) 3217 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3221839024,logm=320555359}) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.903524+0000 mon.a (mon.0) 3218 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1215561094}) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.904371+0000 mon.a (mon.0) 3219 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1810830127}) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.905258+0000 mon.a (mon.0) 3220 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2904458400}) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.906978+0000 mon.a (mon.0) 3221 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302155314}) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.911198+0000 mon.a (mon.0) 3222 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=741982891}) 2024-02-12T16:45:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.915596+0000 mon.a (mon.0) 3223 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=358927418}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.919677+0000 mon.a (mon.0) 3224 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2744939720}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.923731+0000 mon.a (mon.0) 3225 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2593003180}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.927790+0000 mon.a (mon.0) 3226 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734474486}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.929972+0000 mon.a (mon.0) 3227 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=22,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=38} crc {logm=1505259141,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3906500417,mon_config_key=310623174}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.930885+0000 mon.a (mon.0) 3228 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3130161295}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.931854+0000 mon.a (mon.0) 3229 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1815389167}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.932978+0000 mon.a (mon.0) 3230 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=74} crc {mon_config_key=2796904410,monmap=2720272532,osd_metadata=4093509710,osd_pg_creating=1122032377,osdmap=517479979}) 2024-02-12T16:45:04.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:04 smithi123 bash[19119]: cluster 2024-02-12T16:45:03.943456+0000 mon.a (mon.0) 3231 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=41} crc {osdmap=658076882}) 2024-02-12T16:45:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.054620+0000 mon.a (mon.0) 3215 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:45:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.901526+0000 mon.a (mon.0) 3216 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=4116140735,config=2903391811,health=27270746,kv=907397368}) 2024-02-12T16:45:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.902587+0000 mon.a (mon.0) 3217 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3221839024,logm=320555359}) 2024-02-12T16:45:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.903524+0000 mon.a (mon.0) 3218 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1215561094}) 2024-02-12T16:45:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.904371+0000 mon.a (mon.0) 3219 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1810830127}) 2024-02-12T16:45:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.905258+0000 mon.a (mon.0) 3220 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2904458400}) 2024-02-12T16:45:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.906978+0000 mon.a (mon.0) 3221 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302155314}) 2024-02-12T16:45:04.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.911198+0000 mon.a (mon.0) 3222 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=741982891}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.915596+0000 mon.a (mon.0) 3223 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=358927418}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.919677+0000 mon.a (mon.0) 3224 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2744939720}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.923731+0000 mon.a (mon.0) 3225 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2593003180}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.927790+0000 mon.a (mon.0) 3226 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=734474486}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.929972+0000 mon.a (mon.0) 3227 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=22,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=38} crc {logm=1505259141,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3906500417,mon_config_key=310623174}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.930885+0000 mon.a (mon.0) 3228 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3130161295}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.931854+0000 mon.a (mon.0) 3229 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1815389167}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.932978+0000 mon.a (mon.0) 3230 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=8,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=74} crc {mon_config_key=2796904410,monmap=2720272532,osd_metadata=4093509710,osd_pg_creating=1122032377,osdmap=517479979}) 2024-02-12T16:45:04.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:04 smithi121 bash[18489]: cluster 2024-02-12T16:45:03.943456+0000 mon.a (mon.0) 3231 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=41} crc {osdmap=658076882}) 2024-02-12T16:45:04.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:04.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:04.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:04.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:04.621 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:04.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:04.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:04.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:04.857 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:04.881 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:04.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:04.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:05.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:05.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:05.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:05 smithi123 bash[19119]: cluster 2024-02-12T16:45:04.305555+0000 mgr.y (mgr.14146) 1824 : cluster [DBG] pgmap v2660: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 584 op/s; 15587/214352 objects misplaced (7.272%); 5.6 MiB/s, 90 objects/s recovering 2024-02-12T16:45:05.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:05 smithi123 bash[19119]: cluster 2024-02-12T16:45:04.576244+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:05.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:05 smithi123 bash[19119]: cluster 2024-02-12T16:45:04.576313+0000 osd.0 (osd.0) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:05.156 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:05.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:05.302 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:05.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:05 smithi121 bash[18489]: cluster 2024-02-12T16:45:04.305555+0000 mgr.y (mgr.14146) 1824 : cluster [DBG] pgmap v2660: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 584 op/s; 15587/214352 objects misplaced (7.272%); 5.6 MiB/s, 90 objects/s recovering 2024-02-12T16:45:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:05 smithi121 bash[18489]: cluster 2024-02-12T16:45:04.576244+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:05 smithi121 bash[18489]: cluster 2024-02-12T16:45:04.576313+0000 osd.0 (osd.0) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:05.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:05 smithi169 bash[19604]: cluster 2024-02-12T16:45:04.305555+0000 mgr.y (mgr.14146) 1824 : cluster [DBG] pgmap v2660: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 584 op/s; 15587/214352 objects misplaced (7.272%); 5.6 MiB/s, 90 objects/s recovering 2024-02-12T16:45:05.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:05 smithi169 bash[19604]: cluster 2024-02-12T16:45:04.576244+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:05.419 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:05 smithi169 bash[19604]: cluster 2024-02-12T16:45:04.576313+0000 osd.0 (osd.0) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:05.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:05.508 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 10, 8, 11] out_osds: [9, 6] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:45:05.509 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:45:05.509 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-12T16:45:05.509 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-02-12T16:45:05.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-12T16:45:05.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:05.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:05.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:05.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:05.826 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:05.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:05.955 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:05.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:06.056 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:06.060 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:06.086 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:06.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:06.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:06.218 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:06.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:06.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:06.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:06 smithi121 bash[18489]: cluster 2024-02-12T16:45:05.061098+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T16:45:06.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:06 smithi121 bash[18489]: cluster 2024-02-12T16:45:05.072738+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e968: 12 total, 12 up, 10 in 2024-02-12T16:45:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:06.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:06 smithi123 bash[19119]: cluster 2024-02-12T16:45:05.061098+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T16:45:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:06 smithi123 bash[19119]: cluster 2024-02-12T16:45:05.072738+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e968: 12 total, 12 up, 10 in 2024-02-12T16:45:06.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:06.490 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:06.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:06 smithi169 bash[19604]: cluster 2024-02-12T16:45:05.061098+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-12T16:45:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:06 smithi169 bash[19604]: cluster 2024-02-12T16:45:05.072738+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e968: 12 total, 12 up, 10 in 2024-02-12T16:45:06.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:06.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:06.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:06.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:06.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:06.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:06.886 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:06.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:06.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:07.016 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:07.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:07.127 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:07 smithi121 bash[18489]: cluster 2024-02-12T16:45:06.306024+0000 mgr.y (mgr.14146) 1825 : cluster [DBG] pgmap v2662: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 502 op/s; 15396/213536 objects misplaced (7.210%); 6.9 MiB/s, 111 objects/s recovering 2024-02-12T16:45:07.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:07.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:07.262 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:07 smithi169 bash[19604]: cluster 2024-02-12T16:45:06.306024+0000 mgr.y (mgr.14146) 1825 : cluster [DBG] pgmap v2662: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 502 op/s; 15396/213536 objects misplaced (7.210%); 6.9 MiB/s, 111 objects/s recovering 2024-02-12T16:45:07.283 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:07.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:07 smithi123 bash[19119]: cluster 2024-02-12T16:45:06.306024+0000 mgr.y (mgr.14146) 1825 : cluster [DBG] pgmap v2662: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 502 op/s; 15396/213536 objects misplaced (7.210%); 6.9 MiB/s, 111 objects/s recovering 2024-02-12T16:45:07.412 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:07.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:07.545 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:07.651 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:07.790 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:07.819 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:07.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:07.966 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:07.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:08.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:08.108 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:08.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:08.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:08.238 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:08.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:08.342 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:08.376 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:08.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:08.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:08.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:08.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:08.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:08.657 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:08.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:08.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:08.798 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:08.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:08.905 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:08.933 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:08.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:09.037 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:09.066 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:09.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:09.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:09.208 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:09.315 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:09.339 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:09.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:09.473 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:09.479 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:09 smithi121 bash[18489]: cluster 2024-02-12T16:45:08.306556+0000 mgr.y (mgr.14146) 1826 : cluster [DBG] pgmap v2663: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 654 op/s; 14622/209366 objects misplaced (6.984%); 8.7 MiB/s, 138 objects/s recovering 2024-02-12T16:45:09.576 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:09.611 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:09.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:09 smithi123 bash[19119]: cluster 2024-02-12T16:45:08.306556+0000 mgr.y (mgr.14146) 1826 : cluster [DBG] pgmap v2663: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 654 op/s; 14622/209366 objects misplaced (6.984%); 8.7 MiB/s, 138 objects/s recovering 2024-02-12T16:45:09.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:09.738 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:09 smithi169 bash[19604]: cluster 2024-02-12T16:45:08.306556+0000 mgr.y (mgr.14146) 1826 : cluster [DBG] pgmap v2663: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 654 op/s; 14622/209366 objects misplaced (6.984%); 8.7 MiB/s, 138 objects/s recovering 2024-02-12T16:45:09.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:09.880 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:09.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:09.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:10.014 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:10.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:10.146 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:10.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:10.281 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:10.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:10 smithi123 bash[19119]: cluster 2024-02-12T16:45:08.676043+0000 osd.10 (osd.10) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:10 smithi123 bash[19119]: cluster 2024-02-12T16:45:08.676094+0000 osd.10 (osd.10) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:10 smithi123 bash[19119]: cluster 2024-02-12T16:45:09.364156+0000 mon.a (mon.0) 3234 : cluster [DBG] osdmap e969: 12 total, 12 up, 10 in 2024-02-12T16:45:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:10 smithi123 bash[19119]: audit 2024-02-12T16:45:10.051613+0000 mon.a (mon.0) 3235 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:10.410 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:10.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:10.542 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:10.647 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:10.677 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:10.678 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:10 smithi121 bash[18489]: cluster 2024-02-12T16:45:08.676043+0000 osd.10 (osd.10) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:10.678 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:10 smithi121 bash[18489]: cluster 2024-02-12T16:45:08.676094+0000 osd.10 (osd.10) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:10.678 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:10 smithi121 bash[18489]: cluster 2024-02-12T16:45:09.364156+0000 mon.a (mon.0) 3234 : cluster [DBG] osdmap e969: 12 total, 12 up, 10 in 2024-02-12T16:45:10.678 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:10 smithi121 bash[18489]: audit 2024-02-12T16:45:10.051613+0000 mon.a (mon.0) 3235 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:10.784 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:10 smithi169 bash[19604]: cluster 2024-02-12T16:45:08.676043+0000 osd.10 (osd.10) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:10 smithi169 bash[19604]: cluster 2024-02-12T16:45:08.676094+0000 osd.10 (osd.10) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:10 smithi169 bash[19604]: cluster 2024-02-12T16:45:09.364156+0000 mon.a (mon.0) 3234 : cluster [DBG] osdmap e969: 12 total, 12 up, 10 in 2024-02-12T16:45:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:10 smithi169 bash[19604]: audit 2024-02-12T16:45:10.051613+0000 mon.a (mon.0) 3235 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:10.810 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:10.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:10.942 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:10.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:11.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:11.085 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:11.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:11.183 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:11.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:11.221 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:11.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:11.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:11.352 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:11.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:11.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:11.485 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:11.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:11.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:11.597 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:11 smithi123 bash[19119]: cluster 2024-02-12T16:45:10.306999+0000 mgr.y (mgr.14146) 1827 : cluster [DBG] pgmap v2665: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 630 op/s; 14622/206314 objects misplaced (7.087%); 5.6 MiB/s, 89 objects/s recovering 2024-02-12T16:45:11.616 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:11.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:11.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:11 smithi121 bash[18489]: cluster 2024-02-12T16:45:10.306999+0000 mgr.y (mgr.14146) 1827 : cluster [DBG] pgmap v2665: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 630 op/s; 14622/206314 objects misplaced (7.087%); 5.6 MiB/s, 89 objects/s recovering 2024-02-12T16:45:11.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:11.753 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:11.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:11 smithi169 bash[19604]: cluster 2024-02-12T16:45:10.306999+0000 mgr.y (mgr.14146) 1827 : cluster [DBG] pgmap v2665: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 630 op/s; 14622/206314 objects misplaced (7.087%); 5.6 MiB/s, 89 objects/s recovering 2024-02-12T16:45:11.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:11.882 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:11.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:11.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:12.011 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:12.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:12.148 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:12.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:12.279 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:12.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:12.412 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:12.521 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:12.545 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:12.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:12.683 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:12.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:12.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:12.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:12.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:12.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:12.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:13.054 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:13.081 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:13.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:13.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:13.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:13.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:13.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:13.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:13.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:13.469 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:13 smithi169 bash[19604]: cluster 2024-02-12T16:45:12.307518+0000 mgr.y (mgr.14146) 1828 : cluster [DBG] pgmap v2666: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 812 op/s; 13560/201352 objects misplaced (6.734%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:13.490 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:13.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:13.589 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:13 smithi121 bash[18489]: cluster 2024-02-12T16:45:12.307518+0000 mgr.y (mgr.14146) 1828 : cluster [DBG] pgmap v2666: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 812 op/s; 13560/201352 objects misplaced (6.734%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:13.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:13.620 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:13.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:13.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:13 smithi123 bash[19119]: cluster 2024-02-12T16:45:12.307518+0000 mgr.y (mgr.14146) 1828 : cluster [DBG] pgmap v2666: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 812 op/s; 13560/201352 objects misplaced (6.734%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:13.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:13.768 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:13.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:13.863 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:45:13.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:13.892 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:45:13.911 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:13.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:13.925 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:45:13.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:45:13.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:45:13.984 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:45:14.006 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:45:14.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:14.029 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:45:14.044 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:14.058 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:45:14.082 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:45:14.123 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:45:14.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:14.153 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:45:14.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.161 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.162 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:45:14.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:14.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:14.315 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:14.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:14 smithi121 bash[18489]: audit 2024-02-12T16:45:13.833505+0000 mon.a (mon.0) 3236 : audit [DBG] from='client.? v1:172.21.15.121:0/915675618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:14.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:14.448 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:14.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:14.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:14 smithi123 bash[19119]: audit 2024-02-12T16:45:13.833505+0000 mon.a (mon.0) 3236 : audit [DBG] from='client.? v1:172.21.15.121:0/915675618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:14.684 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:14.711 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:14 smithi169 bash[19604]: audit 2024-02-12T16:45:13.833505+0000 mon.a (mon.0) 3236 : audit [DBG] from='client.? v1:172.21.15.121:0/915675618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:14.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:14.841 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:14.946 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:14.977 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:14.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:15.079 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:15.112 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:15.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:15.216 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:15.241 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:15.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:15.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:15.373 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:15.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:15.387 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:45:15.480 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:15.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:15 smithi169 bash[19604]: cluster 2024-02-12T16:45:14.308086+0000 mgr.y (mgr.14146) 1829 : cluster [DBG] pgmap v2667: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 860 op/s; 13560/198450 objects misplaced (6.833%); 10 MiB/s, 167 objects/s recovering 2024-02-12T16:45:15.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:15 smithi169 bash[19604]: audit 2024-02-12T16:45:14.377198+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? v1:172.21.15.121:0/4248682151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:15.488 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:15 smithi169 bash[19604]: cluster 2024-02-12T16:45:15.374009+0000 mon.a (mon.0) 3238 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:45:15.507 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:15.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:15.612 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:15 smithi123 bash[19119]: cluster 2024-02-12T16:45:14.308086+0000 mgr.y (mgr.14146) 1829 : cluster [DBG] pgmap v2667: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 860 op/s; 13560/198450 objects misplaced (6.833%); 10 MiB/s, 167 objects/s recovering 2024-02-12T16:45:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:15 smithi123 bash[19119]: audit 2024-02-12T16:45:14.377198+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? v1:172.21.15.121:0/4248682151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:15.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:15 smithi123 bash[19119]: cluster 2024-02-12T16:45:15.374009+0000 mon.a (mon.0) 3238 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:45:15.649 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:15.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:15.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:15 smithi121 bash[18489]: cluster 2024-02-12T16:45:14.308086+0000 mgr.y (mgr.14146) 1829 : cluster [DBG] pgmap v2667: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 860 op/s; 13560/198450 objects misplaced (6.833%); 10 MiB/s, 167 objects/s recovering 2024-02-12T16:45:15.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:15 smithi121 bash[18489]: audit 2024-02-12T16:45:14.377198+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? v1:172.21.15.121:0/4248682151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:15.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:15 smithi121 bash[18489]: cluster 2024-02-12T16:45:15.374009+0000 mon.a (mon.0) 3238 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:45:15.696 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:45:15.696 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:45:15.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:15.779 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:15.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:15.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:15.924 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:15.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:16.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:16.070 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:16.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:16.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:16.208 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:16.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:16.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:16.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:16.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:16 smithi123 bash[19119]: audit 2024-02-12T16:45:15.377617+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? v1:172.21.15.121:0/4248682151' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:45:16.387 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:16 smithi123 bash[19119]: cluster 2024-02-12T16:45:15.377705+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e970: 12 total, 12 up, 10 in 2024-02-12T16:45:16.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:16.468 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:16.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:16.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:16 smithi169 bash[19604]: audit 2024-02-12T16:45:15.377617+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? v1:172.21.15.121:0/4248682151' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:45:16.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:16 smithi169 bash[19604]: cluster 2024-02-12T16:45:15.377705+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e970: 12 total, 12 up, 10 in 2024-02-12T16:45:16.576 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:16.605 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:16.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:16 smithi121 bash[18489]: audit 2024-02-12T16:45:15.377617+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? v1:172.21.15.121:0/4248682151' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:45:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:16 smithi121 bash[18489]: cluster 2024-02-12T16:45:15.377705+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e970: 12 total, 12 up, 10 in 2024-02-12T16:45:16.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:16.735 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:16.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:16.841 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:16.865 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:16.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:16.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:17.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:17.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:17.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:17.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:17.273 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:17.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:17.412 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:17.488 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:17 smithi121 bash[18489]: cluster 2024-02-12T16:45:16.308529+0000 mgr.y (mgr.14146) 1830 : cluster [DBG] pgmap v2669: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 744 op/s; 13308/197448 objects misplaced (6.740%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T16:45:17.488 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:17 smithi121 bash[18489]: cluster 2024-02-12T16:45:17.117016+0000 osd.8 (osd.8) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:17.488 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:17 smithi121 bash[18489]: cluster 2024-02-12T16:45:17.117068+0000 osd.8 (osd.8) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:17.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:17.524 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:17 smithi169 bash[19604]: cluster 2024-02-12T16:45:16.308529+0000 mgr.y (mgr.14146) 1830 : cluster [DBG] pgmap v2669: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 744 op/s; 13308/197448 objects misplaced (6.740%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T16:45:17.524 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:17 smithi169 bash[19604]: cluster 2024-02-12T16:45:17.117016+0000 osd.8 (osd.8) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:17.524 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:17 smithi169 bash[19604]: cluster 2024-02-12T16:45:17.117068+0000 osd.8 (osd.8) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:17.548 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:17.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:17.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:17 smithi123 bash[19119]: cluster 2024-02-12T16:45:16.308529+0000 mgr.y (mgr.14146) 1830 : cluster [DBG] pgmap v2669: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 6.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 744 op/s; 13308/197448 objects misplaced (6.740%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T16:45:17.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:17 smithi123 bash[19119]: cluster 2024-02-12T16:45:17.117016+0000 osd.8 (osd.8) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:17.658 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:17 smithi123 bash[19119]: cluster 2024-02-12T16:45:17.117068+0000 osd.8 (osd.8) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:17.799 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:17.835 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:17.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:45:17.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:17.971 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:17.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:18.078 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:18.108 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:18.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:18.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:18.239 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:18.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:18.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:18.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:18.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:18.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:18.499 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:18.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:18.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-12T16:45:18.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:18.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:18.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:18.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:18.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:18.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:18.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:18.918 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:18.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:19.017 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:19 smithi123 bash[19119]: cluster 2024-02-12T16:45:18.007531+0000 mon.a (mon.0) 3241 : cluster [DBG] osdmap e971: 12 total, 12 up, 10 in 2024-02-12T16:45:19.017 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:19 smithi123 bash[19119]: cluster 2024-02-12T16:45:18.308965+0000 mgr.y (mgr.14146) 1831 : cluster [DBG] pgmap v2671: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 813 op/s; 12528/193288 objects misplaced (6.482%); 13 MiB/s, 202 objects/s recovering 2024-02-12T16:45:19.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:19.058 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:19.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:19.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:19.187 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:19.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:19.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:19.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:19 smithi169 bash[19604]: cluster 2024-02-12T16:45:18.007531+0000 mon.a (mon.0) 3241 : cluster [DBG] osdmap e971: 12 total, 12 up, 10 in 2024-02-12T16:45:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:19 smithi169 bash[19604]: cluster 2024-02-12T16:45:18.308965+0000 mgr.y (mgr.14146) 1831 : cluster [DBG] pgmap v2671: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 813 op/s; 12528/193288 objects misplaced (6.482%); 13 MiB/s, 202 objects/s recovering 2024-02-12T16:45:19.299 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:19 smithi121 bash[18489]: cluster 2024-02-12T16:45:18.007531+0000 mon.a (mon.0) 3241 : cluster [DBG] osdmap e971: 12 total, 12 up, 10 in 2024-02-12T16:45:19.299 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:19 smithi121 bash[18489]: cluster 2024-02-12T16:45:18.308965+0000 mgr.y (mgr.14146) 1831 : cluster [DBG] pgmap v2671: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 813 op/s; 12528/193288 objects misplaced (6.482%); 13 MiB/s, 202 objects/s recovering 2024-02-12T16:45:19.336 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:19.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:19.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:19.473 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:19.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:19.576 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:19.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:19.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:19.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:19.748 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:19.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:19.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:19.902 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:19.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:20.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:20.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:20.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:20.167 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:20.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:20.294 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:20.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:20.428 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:20.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:45:20.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:20.558 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:20.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:20.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:20.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:20.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:20.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:20.948 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:20.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:21.051 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:21.078 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:21.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:21.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:21.214 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:21.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:21.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:21.358 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:21.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:21.379 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-12T16:45:21.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:21.498 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:21.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:21 smithi169 bash[19604]: cluster 2024-02-12T16:45:20.309484+0000 mgr.y (mgr.14146) 1832 : cluster [DBG] pgmap v2672: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 12528/190306 objects misplaced (6.583%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:45:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:21 smithi169 bash[19604]: audit 2024-02-12T16:45:20.669303+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? v1:172.21.15.121:0/2693191779' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:45:21.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:21.625 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:21.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:21 smithi123 bash[19119]: cluster 2024-02-12T16:45:20.309484+0000 mgr.y (mgr.14146) 1832 : cluster [DBG] pgmap v2672: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 12528/190306 objects misplaced (6.583%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:45:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:21 smithi123 bash[19119]: audit 2024-02-12T16:45:20.669303+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? v1:172.21.15.121:0/2693191779' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:45:21.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:21 smithi121 bash[18489]: cluster 2024-02-12T16:45:20.309484+0000 mgr.y (mgr.14146) 1832 : cluster [DBG] pgmap v2672: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 12528/190306 objects misplaced (6.583%); 6.1 MiB/s, 97 objects/s recovering 2024-02-12T16:45:21.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:21 smithi121 bash[18489]: audit 2024-02-12T16:45:20.669303+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? v1:172.21.15.121:0/2693191779' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:45:21.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:21.761 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:21.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:21.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:21.898 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:21.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:22.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:22.037 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:22.141 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:22.173 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:22.276 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:22.303 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:22.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:22.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:22.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:22.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:22 smithi123 bash[19119]: audit 2024-02-12T16:45:21.366137+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? v1:172.21.15.121:0/2693191779' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:45:22.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:22 smithi123 bash[19119]: cluster 2024-02-12T16:45:21.366197+0000 mon.a (mon.0) 3244 : cluster [DBG] osdmap e972: 12 total, 12 up, 10 in 2024-02-12T16:45:22.584 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:22 smithi121 bash[18489]: audit 2024-02-12T16:45:21.366137+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? v1:172.21.15.121:0/2693191779' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:45:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:22 smithi121 bash[18489]: cluster 2024-02-12T16:45:21.366197+0000 mon.a (mon.0) 3244 : cluster [DBG] osdmap e972: 12 total, 12 up, 10 in 2024-02-12T16:45:22.692 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:22.723 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:22 smithi169 bash[19604]: audit 2024-02-12T16:45:21.366137+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? v1:172.21.15.121:0/2693191779' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:45:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:22 smithi169 bash[19604]: cluster 2024-02-12T16:45:21.366197+0000 mon.a (mon.0) 3244 : cluster [DBG] osdmap e972: 12 total, 12 up, 10 in 2024-02-12T16:45:22.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:22.853 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:22.910 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:22.911 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:22.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:22.983 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:22.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:23.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:23.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:23.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:23.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:23.252 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:23.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:23.357 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:23.381 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:23 smithi121 bash[18489]: cluster 2024-02-12T16:45:22.310006+0000 mgr.y (mgr.14146) 1833 : cluster [DBG] pgmap v2674: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.6 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 988 op/s; 11448/184740 objects misplaced (6.197%); 14 MiB/s, 229 objects/s recovering 2024-02-12T16:45:23.387 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:23.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:23.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:23.520 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:23.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:23.623 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:23.631 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:23 smithi123 bash[19119]: cluster 2024-02-12T16:45:22.310006+0000 mgr.y (mgr.14146) 1833 : cluster [DBG] pgmap v2674: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.6 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 988 op/s; 11448/184740 objects misplaced (6.197%); 14 MiB/s, 229 objects/s recovering 2024-02-12T16:45:23.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:23.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:23.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:23.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:23.792 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:23.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:23 smithi169 bash[19604]: cluster 2024-02-12T16:45:22.310006+0000 mgr.y (mgr.14146) 1833 : cluster [DBG] pgmap v2674: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.6 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 988 op/s; 11448/184740 objects misplaced (6.197%); 14 MiB/s, 229 objects/s recovering 2024-02-12T16:45:23.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:23.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:23.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:24.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:24.059 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:24.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:24.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:24.195 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:24.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:24.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:24.340 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:24.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:24.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:24.478 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:24.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:24.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:24.610 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:24.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:24.714 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:24.742 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:24.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:24.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:24.879 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:24.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:24.931 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:24.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:25.007 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:25.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:25.138 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:25.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:25.283 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:25.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:25.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:25 smithi169 bash[19604]: cluster 2024-02-12T16:45:24.310560+0000 mgr.y (mgr.14146) 1834 : cluster [DBG] pgmap v2675: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 11448/181376 objects misplaced (6.312%); 10 MiB/s, 166 objects/s recovering 2024-02-12T16:45:25.426 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:25.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:25.559 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:25 smithi123 bash[19119]: cluster 2024-02-12T16:45:24.310560+0000 mgr.y (mgr.14146) 1834 : cluster [DBG] pgmap v2675: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 11448/181376 objects misplaced (6.312%); 10 MiB/s, 166 objects/s recovering 2024-02-12T16:45:25.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:25 smithi121 bash[18489]: cluster 2024-02-12T16:45:24.310560+0000 mgr.y (mgr.14146) 1834 : cluster [DBG] pgmap v2675: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.00k op/s; 11448/181376 objects misplaced (6.312%); 10 MiB/s, 166 objects/s recovering 2024-02-12T16:45:25.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:25.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:25.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:25.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:25.965 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:25.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:26.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:26.098 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:26.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:26.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:26.241 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:26.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:26.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:26.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:26.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:26.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:26.502 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:26.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:26.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:26.618 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:26 smithi123 bash[19119]: cluster 2024-02-12T16:45:24.534585+0000 osd.3 (osd.3) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:26.618 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:26 smithi123 bash[19119]: cluster 2024-02-12T16:45:24.534651+0000 osd.3 (osd.3) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:26.618 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:26 smithi123 bash[19119]: cluster 2024-02-12T16:45:25.381170+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e973: 12 total, 12 up, 10 in 2024-02-12T16:45:26.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:26.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:26 smithi121 bash[18489]: cluster 2024-02-12T16:45:24.534585+0000 osd.3 (osd.3) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:26.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:26 smithi121 bash[18489]: cluster 2024-02-12T16:45:24.534651+0000 osd.3 (osd.3) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:26.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:26 smithi121 bash[18489]: cluster 2024-02-12T16:45:25.381170+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e973: 12 total, 12 up, 10 in 2024-02-12T16:45:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:26.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:26.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:26.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:26 smithi169 bash[19604]: cluster 2024-02-12T16:45:24.534585+0000 osd.3 (osd.3) 118 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:26.756 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:26 smithi169 bash[19604]: cluster 2024-02-12T16:45:24.534651+0000 osd.3 (osd.3) 119 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:26.756 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:26 smithi169 bash[19604]: cluster 2024-02-12T16:45:25.381170+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e973: 12 total, 12 up, 10 in 2024-02-12T16:45:26.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:26.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:26.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:26.912 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:26.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:27.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:27.046 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:27.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:27.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:27.174 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:27.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:27.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:27.307 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:27.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:27 smithi121 bash[18489]: cluster 2024-02-12T16:45:26.311069+0000 mgr.y (mgr.14146) 1835 : cluster [DBG] pgmap v2677: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 812 op/s; 11181/180280 objects misplaced (6.202%); 8.4 MiB/s, 134 objects/s recovering 2024-02-12T16:45:27.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:27.440 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 10, 8, 11] out_osds: [9, 6] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:45:27.440 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:45:27.440 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_7 2024-02-12T16:45:27.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:45:27.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:27.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:27 smithi169 bash[19604]: cluster 2024-02-12T16:45:26.311069+0000 mgr.y (mgr.14146) 1835 : cluster [DBG] pgmap v2677: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 812 op/s; 11181/180280 objects misplaced (6.202%); 8.4 MiB/s, 134 objects/s recovering 2024-02-12T16:45:27.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:27.581 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:27.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:27.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:27.697 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:27 smithi123 bash[19119]: cluster 2024-02-12T16:45:26.311069+0000 mgr.y (mgr.14146) 1835 : cluster [DBG] pgmap v2677: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 812 op/s; 11181/180280 objects misplaced (6.202%); 8.4 MiB/s, 134 objects/s recovering 2024-02-12T16:45:27.731 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:27.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:27.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:27.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:27.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:27.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:28.011 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:28.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:28.154 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:28.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:28.296 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:28.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:28.432 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:28.537 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:28.576 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:28.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:28.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:28.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:28.843 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:28.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:28.982 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:28.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:29.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:29.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:29.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:29.253 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:29.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:29.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:29.383 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:29.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:29.489 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:29.497 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:29 smithi123 bash[19119]: cluster 2024-02-12T16:45:28.311577+0000 mgr.y (mgr.14146) 1836 : cluster [DBG] pgmap v2678: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 900 op/s; 10299/175892 objects misplaced (5.855%); 13 MiB/s, 207 objects/s recovering 2024-02-12T16:45:29.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:29.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:29.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:29 smithi121 bash[18489]: cluster 2024-02-12T16:45:28.311577+0000 mgr.y (mgr.14146) 1836 : cluster [DBG] pgmap v2678: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 900 op/s; 10299/175892 objects misplaced (5.855%); 13 MiB/s, 207 objects/s recovering 2024-02-12T16:45:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:29.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:29.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:29.773 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:29 smithi169 bash[19604]: cluster 2024-02-12T16:45:28.311577+0000 mgr.y (mgr.14146) 1836 : cluster [DBG] pgmap v2678: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 900 op/s; 10299/175892 objects misplaced (5.855%); 13 MiB/s, 207 objects/s recovering 2024-02-12T16:45:29.804 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:29.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:29.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:29.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:29.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:30.043 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:30.070 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:30.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:30.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:30.201 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:30.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:30.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:30.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:30.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:30.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:30.487 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:30.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:30.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:30.643 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:30.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:30.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:30.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:30.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:30.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:30.909 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:30.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:31.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:31.047 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:31.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:31.177 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:31.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:31.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:31.420 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:31 smithi123 bash[19119]: cluster 2024-02-12T16:45:30.312143+0000 mgr.y (mgr.14146) 1837 : cluster [DBG] pgmap v2679: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 915 op/s; 10299/172808 objects misplaced (5.960%); 9.7 MiB/s, 154 objects/s recovering 2024-02-12T16:45:31.437 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:31.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:31.547 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:31 smithi121 bash[18489]: cluster 2024-02-12T16:45:30.312143+0000 mgr.y (mgr.14146) 1837 : cluster [DBG] pgmap v2679: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 915 op/s; 10299/172808 objects misplaced (5.960%); 9.7 MiB/s, 154 objects/s recovering 2024-02-12T16:45:31.572 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:31.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:31.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:31 smithi169 bash[19604]: cluster 2024-02-12T16:45:30.312143+0000 mgr.y (mgr.14146) 1837 : cluster [DBG] pgmap v2679: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering, 43 active+clean; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 915 op/s; 10299/172808 objects misplaced (5.960%); 9.7 MiB/s, 154 objects/s recovering 2024-02-12T16:45:31.711 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:31.816 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:31.840 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:31.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:31.978 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:31.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:32.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:32.107 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:32.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:32.211 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:32.249 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:32.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:32.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:32.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:32.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:32.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:32.527 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:32.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:32 smithi169 bash[19604]: cluster 2024-02-12T16:45:32.391759+0000 mon.a (mon.0) 3246 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:45:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:32 smithi169 bash[19604]: cluster 2024-02-12T16:45:32.391798+0000 mon.a (mon.0) 3247 : cluster [INF] Cluster is now healthy 2024-02-12T16:45:32.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:32 smithi121 bash[18489]: cluster 2024-02-12T16:45:32.391759+0000 mon.a (mon.0) 3246 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:45:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:32 smithi121 bash[18489]: cluster 2024-02-12T16:45:32.391798+0000 mon.a (mon.0) 3247 : cluster [INF] Cluster is now healthy 2024-02-12T16:45:32.668 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:32.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:32.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:32.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:32 smithi123 bash[19119]: cluster 2024-02-12T16:45:32.391759+0000 mon.a (mon.0) 3246 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:45:32.785 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:32 smithi123 bash[19119]: cluster 2024-02-12T16:45:32.391798+0000 mon.a (mon.0) 3247 : cluster [INF] Cluster is now healthy 2024-02-12T16:45:32.802 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:32.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:32.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:32.929 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:32.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:33.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:33.064 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:33.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:33.088 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:45:33.127 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:45:33.158 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:219893 9.f e973/916 9:f51f5b8a:::benchmark_data_smithi169_124168_object69209:head v 973'13902, mlcod=973'13898)", 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.134507+0000", 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.01485363, 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.014886516000000001, 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.134507+0000", 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.134507+0000", 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.134506+0000", 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4679999999999992e-06 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.134514+0000", 2024-02-12T16:45:33.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2646000000000001e-05 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.134527+0000", 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.134533+0000", 2024-02-12T16:45:33.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015220909999999999 2024-02-12T16:45:33.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.136055+0000", 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1452000000000002e-05 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.136096+0000", 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015894419999999999 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:219896 9.a e973/916 9:531fc894:::benchmark_data_smithi169_124168_object69212:head v 973'13911, mlcod=973'13909)", 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.138213+0000", 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.011147057, 2024-02-12T16:45:33.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.011289162, 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138213+0000", 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138213+0000", 2024-02-12T16:45:33.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:45:33.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138212+0000", 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7759999999999992e-06 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138222+0000", 2024-02-12T16:45:33.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8990000000000007e-06 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138231+0000", 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2899999999999999e-06 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138237+0000", 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000406998 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138644+0000", 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1973000000000001e-05 2024-02-12T16:45:33.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.138666+0000", 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045256499999999999 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:219902 9.2 9:4756a9e7:::benchmark_data_smithi169_124168_object69218:head [delete] snapc 0=[] ondisk+write+known_if_redirected e973)", 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.140136+0000", 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0092245190000000005, 2024-02-12T16:45:33.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0094503120000000006, 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for degraded object", 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 219902 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140136+0000", 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140136+0000", 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140134+0000", 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5040000000000008e-06 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140144+0000", 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3999999999999998e-06 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140153+0000", 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7450000000000001e-06 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140159+0000", 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0066725910000000003 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146831+0000", 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025504049999999999 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-02-12T16:45:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.149382+0000", 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0092458369999999998 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:219903 9.2 9.c117dde2 (undecoded) ondisk+write+known_if_redirected e973)", 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.140167+0000", 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0091938060000000005, 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0095074849999999995, 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:45:33.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 219903 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140167+0000", 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140167+0000", 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140166+0000", 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4319999999999999e-06 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140169+0000", 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8819999999999998e-06 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140175+0000", 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1320000000000004e-06 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140179+0000", 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2812e-05 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:219895 9.f e973/916 9:fd67c23f:::benchmark_data_smithi169_124168_object69211:head v 973'13903, mlcod=973'13900)", 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.140583+0000", 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0087770929999999997, 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091628899999999999, 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140583+0000", 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140583+0000", 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140583+0000", 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.38e-06 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140592+0000", 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7409999999999992e-06 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140600+0000", 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.977e-06 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140605+0000", 2024-02-12T16:45:33.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073233090000000001 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.147928+0000", 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8127999999999999e-05 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.147956+0000", 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0073726379999999999 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:219904 9.d e973/923 9:bef0ff23:::benchmark_data_smithi169_124168_object69220:head v 973'13540, mlcod=973'13538)", 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.146463+0000", 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0028972989999999999, 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033598510000000001, 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146463+0000", 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146463+0000", 2024-02-12T16:45:33.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146461+0000", 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1445e-05 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146472+0000", 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0825999999999999e-05 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146483+0000", 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8149999999999996e-06 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146488+0000", 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5379000000000001e-05 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146523+0000", 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1338999999999999e-05 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146555+0000", 2024-02-12T16:45:33.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1580000000000001e-05 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:219898 9.f e973/916)", 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.146835+0000", 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0025254090000000002, 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003059764, 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146835+0000", 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146835+0000", 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146834+0000", 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7049999999999994e-06 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146844+0000", 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4909999999999999e-06 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146851+0000", 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2129999999999999e-06 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146857+0000", 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1528000000000001e-05 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.184 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:219911 9.a e973/916 9:57331b0e:::benchmark_data_smithi169_124168_object69227:head v 973'13912, mlcod=973'13909)", 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.148841+0000", 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00051998500000000004, 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001116941, 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148841+0000", 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148841+0000", 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148839+0000", 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.204e-05 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148851+0000", 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3852000000000001e-05 2024-02-12T16:45:33.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148865+0000", 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3219999999999998e-06 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148871+0000", 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000537238 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.149409+0000", 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6925000000000001e-05 2024-02-12T16:45:33.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.149425+0000", 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000584951 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:219907 9.6 e973/916 9:6e7e3a9f:::benchmark_data_smithi169_124168_object69223:head v 973'13654, mlcod=973'13652)", 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.148877+0000", 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.00048372500000000001, 2024-02-12T16:45:33.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011587170000000001, 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148877+0000", 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148877+0000", 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148876+0000", 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1320000000000004e-06 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148880+0000", 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.831e-06 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148885+0000", 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.608e-06 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148889+0000", 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0890999999999999e-05 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148930+0000", 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.579e-05 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.148946+0000", 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8798000000000001e-05 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:219881 9.2 9:47b5b798:::benchmark_data_smithi169_124168_object69197:head [delete] snapc 0=[] ondisk+write+known_if_redirected e973)", 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:33.116959+0000", 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.032401422999999999, 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.033151938999999998, 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for degraded object", 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 219881 2024-02-12T16:45:33.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.116959+0000", 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.116959+0000", 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.116956+0000", 2024-02-12T16:45:33.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3750000000000003e-06 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.116965+0000", 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1505000000000001e-05 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.116976+0000", 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2869999999999999e-06 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.116984+0000", 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.023023277000000002 2024-02-12T16:45:33.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.140007+0000", 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0067871479999999998 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:33.146794+0000", 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029835008 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:45:33.195 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:45:33.196 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 10 2024-02-12T16:45:33.196 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:45:33.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:33.197 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:45:33.197 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:45:33.224 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:45:33.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:33.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:33.260 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:45:33.297 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:45:33.325 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:45:33.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:33.362 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:45:33.368 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:33.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:33.396 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:45:33.425 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:45:33.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.460 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:45:33.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:33.481 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:33 smithi123 bash[19119]: cluster 2024-02-12T16:45:32.312697+0000 mgr.y (mgr.14146) 1838 : cluster [DBG] pgmap v2680: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 860 op/s; 9084/167520 objects misplaced (5.423%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:33.481 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:33 smithi123 bash[19119]: audit 2024-02-12T16:45:33.045813+0000 mon.a (mon.0) 3248 : audit [DBG] from='client.? v1:172.21.15.121:0/2695318107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:33.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:33.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:33 smithi121 bash[18489]: cluster 2024-02-12T16:45:32.312697+0000 mgr.y (mgr.14146) 1838 : cluster [DBG] pgmap v2680: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 860 op/s; 9084/167520 objects misplaced (5.423%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:33.514 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:33 smithi121 bash[18489]: audit 2024-02-12T16:45:33.045813+0000 mon.a (mon.0) 3248 : audit [DBG] from='client.? v1:172.21.15.121:0/2695318107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:33.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:33.643 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:33.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:33.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:33.769 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:33.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:33 smithi169 bash[19604]: cluster 2024-02-12T16:45:32.312697+0000 mgr.y (mgr.14146) 1838 : cluster [DBG] pgmap v2680: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 860 op/s; 9084/167520 objects misplaced (5.423%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:33 smithi169 bash[19604]: audit 2024-02-12T16:45:33.045813+0000 mon.a (mon.0) 3248 : audit [DBG] from='client.? v1:172.21.15.121:0/2695318107' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:33.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:33.906 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:33.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:34.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:34.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:34.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:34.148 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:34.184 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:34.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:34.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:34.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:34.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:34.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:34.457 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:34.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:34.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:34.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:34.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:34.732 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:34.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:34.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:34.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:34.868 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:34.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:34.974 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:35.001 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:35.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:35.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:35.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-12T16:45:35.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.299 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:35.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:35.439 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:35.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:35.467 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:45:35.529 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:35 smithi121 bash[18489]: cluster 2024-02-12T16:45:34.313210+0000 mgr.y (mgr.14146) 1839 : cluster [DBG] pgmap v2681: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 9084/164006 objects misplaced (5.539%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:35.529 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:35 smithi121 bash[18489]: audit 2024-02-12T16:45:34.972914+0000 mon.a (mon.0) 3249 : audit [INF] from='client.? v1:172.21.15.121:0/2727589390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:45:35.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:35.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:35.705 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:35.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:35 smithi169 bash[19604]: cluster 2024-02-12T16:45:34.313210+0000 mgr.y (mgr.14146) 1839 : cluster [DBG] pgmap v2681: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 9084/164006 objects misplaced (5.539%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:35.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:35 smithi169 bash[19604]: audit 2024-02-12T16:45:34.972914+0000 mon.a (mon.0) 3249 : audit [INF] from='client.? v1:172.21.15.121:0/2727589390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:45:35.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:35 smithi123 bash[19119]: cluster 2024-02-12T16:45:34.313210+0000 mgr.y (mgr.14146) 1839 : cluster [DBG] pgmap v2681: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 868 op/s; 9084/164006 objects misplaced (5.539%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:35 smithi123 bash[19119]: audit 2024-02-12T16:45:34.972914+0000 mon.a (mon.0) 3249 : audit [INF] from='client.? v1:172.21.15.121:0/2727589390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:45:35.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:35.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:36.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:36.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:36.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:36.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:36.276 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:36.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:36.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:36.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:36.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:36.455 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:36 smithi169 bash[19604]: cluster 2024-02-12T16:45:35.446613+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:45:36.455 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:36 smithi169 bash[19604]: audit 2024-02-12T16:45:35.455389+0000 mon.a (mon.0) 3251 : audit [INF] from='client.? v1:172.21.15.121:0/2727589390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:45:36.455 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:36 smithi169 bash[19604]: cluster 2024-02-12T16:45:35.455467+0000 mon.a (mon.0) 3252 : cluster [DBG] osdmap e974: 12 total, 12 up, 10 in 2024-02-12T16:45:36.515 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:36.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:36 smithi123 bash[19119]: cluster 2024-02-12T16:45:35.446613+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:45:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:36 smithi123 bash[19119]: audit 2024-02-12T16:45:35.455389+0000 mon.a (mon.0) 3251 : audit [INF] from='client.? v1:172.21.15.121:0/2727589390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:45:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:36 smithi123 bash[19119]: cluster 2024-02-12T16:45:35.455467+0000 mon.a (mon.0) 3252 : cluster [DBG] osdmap e974: 12 total, 12 up, 10 in 2024-02-12T16:45:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:36 smithi121 bash[18489]: cluster 2024-02-12T16:45:35.446613+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:45:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:36 smithi121 bash[18489]: audit 2024-02-12T16:45:35.455389+0000 mon.a (mon.0) 3251 : audit [INF] from='client.? v1:172.21.15.121:0/2727589390' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:45:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:36 smithi121 bash[18489]: cluster 2024-02-12T16:45:35.455467+0000 mon.a (mon.0) 3252 : cluster [DBG] osdmap e974: 12 total, 12 up, 10 in 2024-02-12T16:45:36.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:36.687 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:36.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:36.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:36.832 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:36.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:36.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:36.977 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:36.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:37.034 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:37.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:37.118 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:37.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:37.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:37.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:37.252 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:37.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:37.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:45:37.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:37.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:37.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:37.532 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:37.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:37.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:37 smithi121 bash[18489]: cluster 2024-02-12T16:45:36.313594+0000 mgr.y (mgr.14146) 1840 : cluster [DBG] pgmap v2683: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 8812/162884 objects misplaced (5.410%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:37.675 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:37.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:37.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:37.790 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:37 smithi123 bash[19119]: cluster 2024-02-12T16:45:36.313594+0000 mgr.y (mgr.14146) 1840 : cluster [DBG] pgmap v2683: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 8812/162884 objects misplaced (5.410%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:37.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:37 smithi169 bash[19604]: cluster 2024-02-12T16:45:36.313594+0000 mgr.y (mgr.14146) 1840 : cluster [DBG] pgmap v2683: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 5.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 8812/162884 objects misplaced (5.410%); 11 MiB/s, 180 objects/s recovering 2024-02-12T16:45:37.811 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:37.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:37.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:37.947 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:37.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:38.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:38.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:38.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:38.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:38.213 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:38.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:38.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:38.345 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:38.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:38.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:38.479 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:38.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:38.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:38.616 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:38.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:38.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:45:38.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:38.749 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:38.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:38.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:38.881 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:38.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:38.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:39.017 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:39.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:39.151 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:39.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:39.283 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:39.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:39.417 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:39.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:39.561 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:39 smithi123 bash[19119]: cluster 2024-02-12T16:45:38.314035+0000 mgr.y (mgr.14146) 1841 : cluster [DBG] pgmap v2684: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 7789/158280 objects misplaced (4.921%); 12 MiB/s, 194 objects/s recovering 2024-02-12T16:45:39.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:39.674 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:39 smithi169 bash[19604]: cluster 2024-02-12T16:45:38.314035+0000 mgr.y (mgr.14146) 1841 : cluster [DBG] pgmap v2684: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 7789/158280 objects misplaced (4.921%); 12 MiB/s, 194 objects/s recovering 2024-02-12T16:45:39.701 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:39.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:39.833 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:39 smithi121 bash[18489]: cluster 2024-02-12T16:45:38.314035+0000 mgr.y (mgr.14146) 1841 : cluster [DBG] pgmap v2684: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 7789/158280 objects misplaced (4.921%); 12 MiB/s, 194 objects/s recovering 2024-02-12T16:45:39.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:39.973 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:39.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:40.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:40.106 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:40.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:40.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:40.233 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:40.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:40.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:40.358 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:40.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:40.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:40.489 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:40.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:40.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:40.632 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:40.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:40.692 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:40.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:40.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:40.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:40.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:40.917 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:40.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:41.021 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:41.057 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:41.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:41.154 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:41.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:41.211 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:41.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:41.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:41.341 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:41.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:41.447 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:41.477 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:41.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:41.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:41.615 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:41.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:41 smithi121 bash[18489]: cluster 2024-02-12T16:45:40.314446+0000 mgr.y (mgr.14146) 1842 : cluster [DBG] pgmap v2685: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 906 op/s; 7789/154678 objects misplaced (5.036%); 12 MiB/s, 194 objects/s recovering 2024-02-12T16:45:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:41 smithi121 bash[18489]: audit 2024-02-12T16:45:41.338983+0000 mon.a (mon.0) 3253 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:45:41.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:41.729 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:41 smithi169 bash[19604]: cluster 2024-02-12T16:45:40.314446+0000 mgr.y (mgr.14146) 1842 : cluster [DBG] pgmap v2685: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 906 op/s; 7789/154678 objects misplaced (5.036%); 12 MiB/s, 194 objects/s recovering 2024-02-12T16:45:41.729 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:41 smithi169 bash[19604]: audit 2024-02-12T16:45:41.338983+0000 mon.a (mon.0) 3253 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:45:41.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:41.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:41.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:41 smithi123 bash[19119]: cluster 2024-02-12T16:45:40.314446+0000 mgr.y (mgr.14146) 1842 : cluster [DBG] pgmap v2685: 47 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 906 op/s; 7789/154678 objects misplaced (5.036%); 12 MiB/s, 194 objects/s recovering 2024-02-12T16:45:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:41 smithi123 bash[19119]: audit 2024-02-12T16:45:41.338983+0000 mon.a (mon.0) 3253 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:45:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:41.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:41.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:42.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:42.125 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:42.163 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:42.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:42.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:42.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:42.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:42.399 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:42.444 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:42.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:42.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:42.574 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:42.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:42 smithi121 bash[18489]: cluster 2024-02-12T16:45:42.455197+0000 mon.a (mon.0) 3254 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:45:42.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:42.685 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:42 smithi169 bash[19604]: cluster 2024-02-12T16:45:42.455197+0000 mon.a (mon.0) 3254 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:45:42.718 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:42.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:42.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:42.830 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:42 smithi123 bash[19119]: cluster 2024-02-12T16:45:42.455197+0000 mon.a (mon.0) 3254 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:45:42.853 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:42.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:42.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:43.005 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:43.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:43.132 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:43.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:43.180 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:43.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:43.259 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:43.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:43.399 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:43.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:43.514 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:43 smithi169 bash[19604]: cluster 2024-02-12T16:45:42.314974+0000 mgr.y (mgr.14146) 1843 : cluster [DBG] pgmap v2686: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 941 op/s; 10986/148694 objects misplaced (7.388%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:43.529 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:43.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:43 smithi121 bash[18489]: cluster 2024-02-12T16:45:42.314974+0000 mgr.y (mgr.14146) 1843 : cluster [DBG] pgmap v2686: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 941 op/s; 10986/148694 objects misplaced (7.388%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:43.661 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:43.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:43.793 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:43 smithi123 bash[19119]: cluster 2024-02-12T16:45:42.314974+0000 mgr.y (mgr.14146) 1843 : cluster [DBG] pgmap v2686: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 941 op/s; 10986/148694 objects misplaced (7.388%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:43.896 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:43.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:44.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:44.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:44.196 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:44.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:44.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:44.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:44.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:44.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:44.469 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:44.497 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:44.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:44.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:44.644 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:44.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:44.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:44.777 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:44.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:44.804 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:45:44.805 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:45:44.881 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:44.907 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:44.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:45.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:45.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:45.151 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:45.205 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:45.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:45.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:45.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:45.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:45.486 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:45.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:45.517 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:45 smithi169 bash[19604]: cluster 2024-02-12T16:45:44.315510+0000 mgr.y (mgr.14146) 1844 : cluster [DBG] pgmap v2687: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 943 op/s; 10986/145130 objects misplaced (7.570%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:45.589 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:45.620 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:45.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:45.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:45 smithi121 bash[18489]: cluster 2024-02-12T16:45:44.315510+0000 mgr.y (mgr.14146) 1844 : cluster [DBG] pgmap v2687: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 943 op/s; 10986/145130 objects misplaced (7.570%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:45.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:45.745 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:45.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:45.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:45.860 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:45 smithi123 bash[19119]: cluster 2024-02-12T16:45:44.315510+0000 mgr.y (mgr.14146) 1844 : cluster [DBG] pgmap v2687: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 943 op/s; 10986/145130 objects misplaced (7.570%); 12 MiB/s, 193 objects/s recovering 2024-02-12T16:45:45.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:45.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:45.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:46.012 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:46.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:46.147 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:46.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:46.285 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2687,"stamp":"2024-02-12T16:45:44.315174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4755554328,"num_objects":72565,"num_object_clones":0,"num_object_copies":145130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10986,"num_objects_unfound":0,"num_objects_dirty":72565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":228799,"num_write_kb":9643585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15525,"num_bytes_recovered":1017446400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16434120,"kb_used_data":8643000,"kb_used_omap":92,"kb_used_meta":7790947,"kb_avail":921017400,"statfs":{"total":959950356480,"available":943121817600,"internally_reserved":0,"allocated":8850432000,"data_stored":8846328196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95164,"internal_metadata":7977929796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,6,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-618528768,"num_objects":-9438,"num_object_clones":0,"num_object_copies":-18876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1902,"num_objects_unfound":0,"num_objects_dirty":-9438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":127008768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002283"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":338,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.385331+0000","last_change":"2024-02-12T16:42:21.631979+0000","last_active":"2024-02-12T16:45:25.385331+0000","last_peered":"2024-02-12T16:45:25.385331+0000","last_clean":"2024-02-12T16:45:25.385331+0000","last_became_active":"2024-02-12T16:41:18.947003+0000","last_became_peered":"2024-02-12T16:41:18.947003+0000","last_unstale":"2024-02-12T16:45:25.385331+0000","last_undegraded":"2024-02-12T16:45:25.385331+0000","last_fullsized":"2024-02-12T16:45:25.385331+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:21.631859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:21.631859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":366,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.384070+0000","last_change":"2024-02-12T16:42:29.489386+0000","last_active":"2024-02-12T16:45:25.384070+0000","last_peered":"2024-02-12T16:45:25.384070+0000","last_clean":"2024-02-12T16:45:25.384070+0000","last_became_active":"2024-02-12T16:41:17.943883+0000","last_became_peered":"2024-02-12T16:41:17.943883+0000","last_unstale":"2024-02-12T16:45:25.384070+0000","last_undegraded":"2024-02-12T16:45:25.384070+0000","last_fullsized":"2024-02-12T16:45:25.384070+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:29.489294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:29.489294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":359,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.382422+0000","last_change":"2024-02-12T16:45:00.042491+0000","last_active":"2024-02-12T16:45:25.382422+0000","last_peered":"2024-02-12T16:45:25.382422+0000","last_clean":"2024-02-12T16:45:25.382422+0000","last_became_active":"2024-02-12T16:45:00.042127+0000","last_became_peered":"2024-02-12T16:45:00.042127+0000","last_unstale":"2024-02-12T16:45:25.382422+0000","last_undegraded":"2024-02-12T16:45:25.382422+0000","last_fullsized":"2024-02-12T16:45:25.382422+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.683932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.683932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":172,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.388469+0000","last_change":"2024-02-12T16:44:57.998874+0000","last_active":"2024-02-12T16:45:25.388469+0000","last_peered":"2024-02-12T16:45:25.388469+0000","last_clean":"2024-02-12T16:45:25.388469+0000","last_became_active":"2024-02-12T16:39:09.436840+0000","last_became_peered":"2024-02-12T16:39:09.436840+0000","last_unstale":"2024-02-12T16:45:25.388469+0000","last_undegraded":"2024-02-12T16:45:25.388469+0000","last_fullsized":"2024-02-12T16:45:25.388469+0000","mapping_epoch":845,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":846,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_clean_scrub_stamp":"2024-02-12T16:41:21.337430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":403,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.404187+0000","last_change":"2024-02-12T16:42:41.310928+0000","last_active":"2024-02-12T16:45:25.404187+0000","last_peered":"2024-02-12T16:45:25.404187+0000","last_clean":"2024-02-12T16:45:25.404187+0000","last_became_active":"2024-02-12T16:42:40.975420+0000","last_became_peered":"2024-02-12T16:42:40.975420+0000","last_unstale":"2024-02-12T16:45:25.404187+0000","last_undegraded":"2024-02-12T16:45:25.404187+0000","last_fullsized":"2024-02-12T16:45:25.404187+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:41.310840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:41.310840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":372,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.394077+0000","last_change":"2024-02-12T16:45:00.376435+0000","last_active":"2024-02-12T16:45:25.394077+0000","last_peered":"2024-02-12T16:45:25.394077+0000","last_clean":"2024-02-12T16:45:25.394077+0000","last_became_active":"2024-02-12T16:45:00.375895+0000","last_became_peered":"2024-02-12T16:45:00.375895+0000","last_unstale":"2024-02-12T16:45:25.394077+0000","last_undegraded":"2024-02-12T16:45:25.394077+0000","last_fullsized":"2024-02-12T16:45:25.394077+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:24.921234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:24.921234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":708,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.390073+0000","last_change":"2024-02-12T16:42:31.552860+0000","last_active":"2024-02-12T16:45:25.390073+0000","last_peered":"2024-02-12T16:45:25.390073+0000","last_clean":"2024-02-12T16:45:25.390073+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:45:25.390073+0000","last_undegraded":"2024-02-12T16:45:25.390073+0000","last_fullsized":"2024-02-12T16:45:25.390073+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:31.552623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:42:31.552623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":355,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.382440+0000","last_change":"2024-02-12T16:43:31.617757+0000","last_active":"2024-02-12T16:45:25.382440+0000","last_peered":"2024-02-12T16:45:25.382440+0000","last_clean":"2024-02-12T16:45:25.382440+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:45:25.382440+0000","last_undegraded":"2024-02-12T16:45:25.382440+0000","last_fullsized":"2024-02-12T16:45:25.382440+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.617636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.617636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":350,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.384082+0000","last_change":"2024-02-12T16:42:26.378442+0000","last_active":"2024-02-12T16:45:25.384082+0000","last_peered":"2024-02-12T16:45:25.384082+0000","last_clean":"2024-02-12T16:45:25.384082+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:45:25.384082+0000","last_undegraded":"2024-02-12T16:45:25.384082+0000","last_fullsized":"2024-02-12T16:45:25.384082+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.378312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":331,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:34.491905+0000","last_change":"2024-02-12T16:45:34.491905+0000","last_active":"2024-02-12T16:45:34.491905+0000","last_peered":"2024-02-12T16:45:34.491905+0000","last_clean":"2024-02-12T16:45:34.491905+0000","last_became_active":"2024-02-12T16:42:40.982346+0000","last_became_peered":"2024-02-12T16:42:40.982346+0000","last_unstale":"2024-02-12T16:45:34.491905+0000","last_undegraded":"2024-02-12T16:45:34.491905+0000","last_fullsized":"2024-02-12T16:45:34.491905+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:11.207557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:39:11.207557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":618,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:26.287210+0000","last_change":"2024-02-12T16:45:00.089844+0000","last_active":"2024-02-12T16:45:26.287210+0000","last_peered":"2024-02-12T16:45:26.287210+0000","last_clean":"2024-02-12T16:45:26.287210+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:45:26.287210+0000","last_undegraded":"2024-02-12T16:45:26.287210+0000","last_fullsized":"2024-02-12T16:45:26.287210+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:45:00.089752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:45:00.089752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"974'14053","reported_seq":23614,"reported_epoch":974,"state":"active+recovering","last_fresh":"2024-02-12T16:45:40.421979+0000","last_change":"2024-02-12T16:43:16.238278+0000","last_active":"2024-02-12T16:45:40.421979+0000","last_peered":"2024-02-12T16:45:40.421979+0000","last_clean":"2024-02-12T16:43:15.331170+0000","last_became_active":"2024-02-12T16:43:16.169213+0000","last_became_peered":"2024-02-12T16:43:16.169213+0000","last_unstale":"2024-02-12T16:45:40.421979+0000","last_undegraded":"2024-02-12T16:45:40.421979+0000","last_fullsized":"2024-02-12T16:45:40.421979+0000","mapping_epoch":935,"log_start":"929'4000","ondisk_log_start":"929'4000","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10053,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":299040768,"num_objects":4563,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1099,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14053,"num_write_kb":595712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4745,"num_bytes_recovered":310968320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","9"],"object_location_counts":[{"shards":"5,11","objects":3464},{"shards":"9,11","objects":1099}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":95,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.396880+0000","last_change":"2024-02-12T16:42:41.980681+0000","last_active":"2024-02-12T16:45:25.396880+0000","last_peered":"2024-02-12T16:45:25.396880+0000","last_clean":"2024-02-12T16:45:25.396880+0000","last_became_active":"2024-02-12T16:42:41.980413+0000","last_became_peered":"2024-02-12T16:42:41.980413+0000","last_unstale":"2024-02-12T16:45:25.396880+0000","last_undegraded":"2024-02-12T16:45:25.396880+0000","last_fullsized":"2024-02-12T16:45:25.396880+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.611860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.611860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"974'14199","reported_seq":14258,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:44.244270+0000","last_change":"2024-02-12T16:42:40.991625+0000","last_active":"2024-02-12T16:45:44.244270+0000","last_peered":"2024-02-12T16:45:44.244270+0000","last_clean":"2024-02-12T16:45:44.244270+0000","last_became_active":"2024-02-12T16:42:40.984344+0000","last_became_peered":"2024-02-12T16:42:40.984344+0000","last_unstale":"2024-02-12T16:45:44.244270+0000","last_undegraded":"2024-02-12T16:45:44.244270+0000","last_fullsized":"2024-02-12T16:45:44.244270+0000","mapping_epoch":923,"log_start":"930'4100","ondisk_log_start":"930'4100","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10099,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":294387736,"num_objects":4493,"num_object_clones":0,"num_object_copies":8986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14199,"num_write_kb":598081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":727,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.388226+0000","last_change":"2024-02-12T16:45:00.039447+0000","last_active":"2024-02-12T16:45:25.388226+0000","last_peered":"2024-02-12T16:45:25.388226+0000","last_clean":"2024-02-12T16:45:25.388226+0000","last_became_active":"2024-02-12T16:45:00.039179+0000","last_became_peered":"2024-02-12T16:45:00.039179+0000","last_unstale":"2024-02-12T16:45:25.388226+0000","last_undegraded":"2024-02-12T16:45:25.388226+0000","last_fullsized":"2024-02-12T16:45:25.388226+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:25.248423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:41:25.248423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"974'14331","reported_seq":14396,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:41.698789+0000","last_change":"2024-02-12T16:42:21.999716+0000","last_active":"2024-02-12T16:45:41.698789+0000","last_peered":"2024-02-12T16:45:41.698789+0000","last_clean":"2024-02-12T16:45:41.698789+0000","last_became_active":"2024-02-12T16:42:21.999181+0000","last_became_peered":"2024-02-12T16:42:21.999181+0000","last_unstale":"2024-02-12T16:45:41.698789+0000","last_undegraded":"2024-02-12T16:45:41.698789+0000","last_fullsized":"2024-02-12T16:45:41.698789+0000","mapping_epoch":916,"log_start":"930'4300","ondisk_log_start":"930'4300","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10031,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":298123264,"num_objects":4549,"num_object_clones":0,"num_object_copies":9098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14331,"num_write_kb":604160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1085,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.385209+0000","last_change":"2024-02-12T16:42:38.439762+0000","last_active":"2024-02-12T16:45:25.385209+0000","last_peered":"2024-02-12T16:45:25.385209+0000","last_clean":"2024-02-12T16:45:25.385209+0000","last_became_active":"2024-02-12T16:41:18.947424+0000","last_became_peered":"2024-02-12T16:41:18.947424+0000","last_unstale":"2024-02-12T16:45:25.385209+0000","last_undegraded":"2024-02-12T16:45:25.385209+0000","last_fullsized":"2024-02-12T16:45:25.385209+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:38.439663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:39:15.947507+0000","last_clean_scrub_stamp":"2024-02-12T16:42:38.439663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.f","version":"974'14403","reported_seq":14468,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:42.086985+0000","last_change":"2024-02-12T16:42:21.964154+0000","last_active":"2024-02-12T16:45:42.086985+0000","last_peered":"2024-02-12T16:45:42.086985+0000","last_clean":"2024-02-12T16:45:42.086985+0000","last_became_active":"2024-02-12T16:42:21.963855+0000","last_became_peered":"2024-02-12T16:42:21.963855+0000","last_unstale":"2024-02-12T16:45:42.086985+0000","last_undegraded":"2024-02-12T16:45:42.086985+0000","last_fullsized":"2024-02-12T16:45:42.086985+0000","mapping_epoch":916,"log_start":"931'4400","ondisk_log_start":"931'4400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10003,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":9198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14403,"num_write_kb":608064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":419,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.388515+0000","last_change":"2024-02-12T16:45:00.034498+0000","last_active":"2024-02-12T16:45:25.388515+0000","last_peered":"2024-02-12T16:45:25.388515+0000","last_clean":"2024-02-12T16:45:25.388515+0000","last_became_active":"2024-02-12T16:45:00.034042+0000","last_became_peered":"2024-02-12T16:45:00.034042+0000","last_unstale":"2024-02-12T16:45:25.388515+0000","last_undegraded":"2024-02-12T16:45:25.388515+0000","last_fullsized":"2024-02-12T16:45:25.388515+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.543386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.543386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"974'14521","reported_seq":14579,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:44.234019+0000","last_change":"2024-02-12T16:42:40.974589+0000","last_active":"2024-02-12T16:45:44.234019+0000","last_peered":"2024-02-12T16:45:44.234019+0000","last_clean":"2024-02-12T16:45:44.234019+0000","last_became_active":"2024-02-12T16:42:40.971979+0000","last_became_peered":"2024-02-12T16:42:40.971979+0000","last_unstale":"2024-02-12T16:45:44.234019+0000","last_undegraded":"2024-02-12T16:45:44.234019+0000","last_fullsized":"2024-02-12T16:45:44.234019+0000","mapping_epoch":923,"log_start":"931'4500","ondisk_log_start":"931'4500","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10021,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":293142528,"num_objects":4473,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14521,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":711,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.388352+0000","last_change":"2024-02-12T16:43:15.977479+0000","last_active":"2024-02-12T16:45:25.388352+0000","last_peered":"2024-02-12T16:45:25.388352+0000","last_clean":"2024-02-12T16:45:25.388352+0000","last_became_active":"2024-02-12T16:43:15.976997+0000","last_became_peered":"2024-02-12T16:43:15.976997+0000","last_unstale":"2024-02-12T16:45:25.388352+0000","last_undegraded":"2024-02-12T16:45:25.388352+0000","last_fullsized":"2024-02-12T16:45:25.388352+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.923206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:57.057484+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.923206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"974'14510","reported_seq":14575,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:43.866333+0000","last_change":"2024-02-12T16:42:21.962569+0000","last_active":"2024-02-12T16:45:43.866333+0000","last_peered":"2024-02-12T16:45:43.866333+0000","last_clean":"2024-02-12T16:45:43.866333+0000","last_became_active":"2024-02-12T16:42:21.962130+0000","last_became_peered":"2024-02-12T16:42:21.962130+0000","last_unstale":"2024-02-12T16:45:43.866333+0000","last_undegraded":"2024-02-12T16:45:43.866333+0000","last_fullsized":"2024-02-12T16:45:43.866333+0000","mapping_epoch":916,"log_start":"931'4500","ondisk_log_start":"931'4500","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10010,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289931264,"num_objects":4424,"num_object_clones":0,"num_object_copies":8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14510,"num_write_kb":605888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":754,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.388194+0000","last_change":"2024-02-12T16:42:34.813265+0000","last_active":"2024-02-12T16:45:25.388194+0000","last_peered":"2024-02-12T16:45:25.388194+0000","last_clean":"2024-02-12T16:45:25.388194+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:45:25.388194+0000","last_undegraded":"2024-02-12T16:45:25.388194+0000","last_fullsized":"2024-02-12T16:45:25.388194+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:34.813152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:42:34.813152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"974'14072","reported_seq":14140,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:43.824698+0000","last_change":"2024-02-12T16:43:30.452610+0000","last_active":"2024-02-12T16:45:43.824698+0000","last_peered":"2024-02-12T16:45:43.824698+0000","last_clean":"2024-02-12T16:45:43.824698+0000","last_became_active":"2024-02-12T16:42:21.963637+0000","last_became_peered":"2024-02-12T16:42:21.963637+0000","last_unstale":"2024-02-12T16:45:43.824698+0000","last_undegraded":"2024-02-12T16:45:43.824698+0000","last_fullsized":"2024-02-12T16:45:43.824698+0000","mapping_epoch":916,"log_start":"930'4000","ondisk_log_start":"930'4000","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"940'6865","last_scrub_stamp":"2024-02-12T16:43:30.452522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:43:30.452522+0000","log_size":10072,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":291897344,"num_objects":4454,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14072,"num_write_kb":592832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":572,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.383279+0000","last_change":"2024-02-12T16:45:00.933115+0000","last_active":"2024-02-12T16:45:25.383279+0000","last_peered":"2024-02-12T16:45:25.383279+0000","last_clean":"2024-02-12T16:45:25.383279+0000","last_became_active":"2024-02-12T16:45:00.030971+0000","last_became_peered":"2024-02-12T16:45:00.030971+0000","last_unstale":"2024-02-12T16:45:25.383279+0000","last_undegraded":"2024-02-12T16:45:25.383279+0000","last_fullsized":"2024-02-12T16:45:25.383279+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:45:00.933028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:33.930337+0000","last_clean_scrub_stamp":"2024-02-12T16:45:00.933028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"974'14449","reported_seq":14514,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:42.106340+0000","last_change":"2024-02-12T16:42:21.964459+0000","last_active":"2024-02-12T16:45:42.106340+0000","last_peered":"2024-02-12T16:45:42.106340+0000","last_clean":"2024-02-12T16:45:42.106340+0000","last_became_active":"2024-02-12T16:42:21.964092+0000","last_became_peered":"2024-02-12T16:42:21.964092+0000","last_unstale":"2024-02-12T16:45:42.106340+0000","last_undegraded":"2024-02-12T16:45:42.106340+0000","last_fullsized":"2024-02-12T16:45:42.106340+0000","mapping_epoch":916,"log_start":"931'4400","ondisk_log_start":"931'4400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10049,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":301006848,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14449,"num_write_kb":609344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1047,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.397077+0000","last_change":"2024-02-12T16:43:31.016133+0000","last_active":"2024-02-12T16:45:25.397077+0000","last_peered":"2024-02-12T16:45:25.397077+0000","last_clean":"2024-02-12T16:45:25.397077+0000","last_became_active":"2024-02-12T16:42:41.321180+0000","last_became_peered":"2024-02-12T16:42:41.321180+0000","last_unstale":"2024-02-12T16:45:25.397077+0000","last_undegraded":"2024-02-12T16:45:25.397077+0000","last_fullsized":"2024-02-12T16:45:25.397077+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.016013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.016013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"974'14268","reported_seq":14333,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:40.448260+0000","last_change":"2024-02-12T16:42:21.961833+0000","last_active":"2024-02-12T16:45:40.448260+0000","last_peered":"2024-02-12T16:45:40.448260+0000","last_clean":"2024-02-12T16:45:40.448260+0000","last_became_active":"2024-02-12T16:42:21.961543+0000","last_became_peered":"2024-02-12T16:42:21.961543+0000","last_unstale":"2024-02-12T16:45:40.448260+0000","last_undegraded":"2024-02-12T16:45:40.448260+0000","last_fullsized":"2024-02-12T16:45:40.448260+0000","mapping_epoch":916,"log_start":"930'4200","ondisk_log_start":"930'4200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10068,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":305790976,"num_objects":4666,"num_object_clones":0,"num_object_copies":9332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14268,"num_write_kb":605888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":466,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.401054+0000","last_change":"2024-02-12T16:42:43.033723+0000","last_active":"2024-02-12T16:45:25.401054+0000","last_peered":"2024-02-12T16:45:25.401054+0000","last_clean":"2024-02-12T16:45:25.401054+0000","last_became_active":"2024-02-12T16:42:40.975171+0000","last_became_peered":"2024-02-12T16:42:40.975171+0000","last_unstale":"2024-02-12T16:45:25.401054+0000","last_undegraded":"2024-02-12T16:45:25.401054+0000","last_fullsized":"2024-02-12T16:45:25.401054+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.033576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.033576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"974'14343","reported_seq":14408,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:41.904238+0000","last_change":"2024-02-12T16:42:21.999246+0000","last_active":"2024-02-12T16:45:41.904238+0000","last_peered":"2024-02-12T16:45:41.904238+0000","last_clean":"2024-02-12T16:45:41.904238+0000","last_became_active":"2024-02-12T16:42:21.998975+0000","last_became_peered":"2024-02-12T16:42:21.998975+0000","last_unstale":"2024-02-12T16:45:41.904238+0000","last_undegraded":"2024-02-12T16:45:41.904238+0000","last_fullsized":"2024-02-12T16:45:41.904238+0000","mapping_epoch":916,"log_start":"931'4300","ondisk_log_start":"931'4300","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10043,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":303235072,"num_objects":4627,"num_object_clones":0,"num_object_copies":9254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14343,"num_write_kb":607040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":322,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:28.521013+0000","last_change":"2024-02-12T16:45:28.521013+0000","last_active":"2024-02-12T16:45:28.521013+0000","last_peered":"2024-02-12T16:45:28.521013+0000","last_clean":"2024-02-12T16:45:28.521013+0000","last_became_active":"2024-02-12T16:45:00.031187+0000","last_became_peered":"2024-02-12T16:45:00.031187+0000","last_unstale":"2024-02-12T16:45:28.521013+0000","last_undegraded":"2024-02-12T16:45:28.521013+0000","last_fullsized":"2024-02-12T16:45:28.521013+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:22.478963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:22.478963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"974'14589","reported_seq":14659,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:43.856631+0000","last_change":"2024-02-12T16:42:40.974611+0000","last_active":"2024-02-12T16:45:43.856631+0000","last_peered":"2024-02-12T16:45:43.856631+0000","last_clean":"2024-02-12T16:45:43.856631+0000","last_became_active":"2024-02-12T16:42:40.973291+0000","last_became_peered":"2024-02-12T16:42:40.973291+0000","last_unstale":"2024-02-12T16:45:43.856631+0000","last_undegraded":"2024-02-12T16:45:43.856631+0000","last_fullsized":"2024-02-12T16:45:43.856631+0000","mapping_epoch":923,"log_start":"931'4500","ondisk_log_start":"931'4500","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10089,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":301531136,"num_objects":4601,"num_object_clones":0,"num_object_copies":9202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14589,"num_write_kb":614080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":224,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.383237+0000","last_change":"2024-02-12T16:45:00.036284+0000","last_active":"2024-02-12T16:45:25.383237+0000","last_peered":"2024-02-12T16:45:25.383237+0000","last_clean":"2024-02-12T16:45:25.383237+0000","last_became_active":"2024-02-12T16:45:00.035400+0000","last_became_peered":"2024-02-12T16:45:00.035400+0000","last_unstale":"2024-02-12T16:45:25.383237+0000","last_undegraded":"2024-02-12T16:45:25.383237+0000","last_fullsized":"2024-02-12T16:45:25.383237+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.559264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"974'14147","reported_seq":14212,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:42.112137+0000","last_change":"2024-02-12T16:42:21.964135+0000","last_active":"2024-02-12T16:45:42.112137+0000","last_peered":"2024-02-12T16:45:42.112137+0000","last_clean":"2024-02-12T16:45:42.112137+0000","last_became_active":"2024-02-12T16:42:21.963899+0000","last_became_peered":"2024-02-12T16:42:21.963899+0000","last_unstale":"2024-02-12T16:45:42.112137+0000","last_undegraded":"2024-02-12T16:45:42.112137+0000","last_fullsized":"2024-02-12T16:45:42.112137+0000","mapping_epoch":916,"log_start":"929'4100","ondisk_log_start":"929'4100","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10047,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":300220416,"num_objects":4581,"num_object_clones":0,"num_object_copies":9162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14147,"num_write_kb":599296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":519,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.404162+0000","last_change":"2024-02-12T16:42:42.373090+0000","last_active":"2024-02-12T16:45:25.404162+0000","last_peered":"2024-02-12T16:45:25.404162+0000","last_clean":"2024-02-12T16:45:25.404162+0000","last_became_active":"2024-02-12T16:42:40.975493+0000","last_became_peered":"2024-02-12T16:42:40.975493+0000","last_unstale":"2024-02-12T16:45:25.404162+0000","last_undegraded":"2024-02-12T16:45:25.404162+0000","last_fullsized":"2024-02-12T16:45:25.404162+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_clean_scrub_stamp":"2024-02-12T16:42:42.373026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"974'13985","reported_seq":25681,"reported_epoch":974,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:45:41.687840+0000","last_change":"2024-02-12T16:45:39.718212+0000","last_active":"2024-02-12T16:45:41.687840+0000","last_peered":"2024-02-12T16:45:41.687840+0000","last_clean":"2024-02-12T16:43:15.351398+0000","last_became_active":"2024-02-12T16:45:01.185991+0000","last_became_peered":"2024-02-12T16:45:01.185991+0000","last_unstale":"2024-02-12T16:45:41.687840+0000","last_undegraded":"2024-02-12T16:45:41.687840+0000","last_fullsized":"2024-02-12T16:45:41.687840+0000","mapping_epoch":965,"log_start":"929'3900","ondisk_log_start":"929'3900","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10085,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289996800,"num_objects":4425,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4425,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13985,"num_write_kb":589120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5805,"num_bytes_recovered":380436480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,6],"avail_no_missing":["3","6","9"],"object_location_counts":[{"shards":"3,6","objects":4425}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":1034,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.396975+0000","last_change":"2024-02-12T16:43:16.350572+0000","last_active":"2024-02-12T16:45:25.396975+0000","last_peered":"2024-02-12T16:45:25.396975+0000","last_clean":"2024-02-12T16:45:25.396975+0000","last_became_active":"2024-02-12T16:43:16.350353+0000","last_became_peered":"2024-02-12T16:43:16.350353+0000","last_unstale":"2024-02-12T16:45:25.396975+0000","last_undegraded":"2024-02-12T16:45:25.396975+0000","last_fullsized":"2024-02-12T16:45:25.396975+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_clean_scrub_stamp":"2024-02-12T16:42:48.099448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"974'14264","reported_seq":14588,"reported_epoch":974,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:45:41.893847+0000","last_change":"2024-02-12T16:45:31.287568+0000","last_active":"2024-02-12T16:45:41.893847+0000","last_peered":"2024-02-12T16:45:41.893847+0000","last_clean":"2024-02-12T16:44:59.014278+0000","last_became_active":"2024-02-12T16:45:01.200731+0000","last_became_peered":"2024-02-12T16:45:01.200731+0000","last_unstale":"2024-02-12T16:45:41.893847+0000","last_undegraded":"2024-02-12T16:45:41.893847+0000","last_fullsized":"2024-02-12T16:45:41.893847+0000","mapping_epoch":965,"log_start":"930'4200","ondisk_log_start":"930'4200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10064,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":295305216,"num_objects":4506,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4394,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14264,"num_write_kb":600640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":8060928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4506}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":702,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.390166+0000","last_change":"2024-02-12T16:42:23.507674+0000","last_active":"2024-02-12T16:45:25.390166+0000","last_peered":"2024-02-12T16:45:25.390166+0000","last_clean":"2024-02-12T16:45:25.390166+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:45:25.390166+0000","last_undegraded":"2024-02-12T16:45:25.390166+0000","last_fullsized":"2024-02-12T16:45:25.390166+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.507539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.507539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"974'14420","reported_seq":14485,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-02-12T16:45:43.830617+0000","last_change":"2024-02-12T16:42:21.963116+0000","last_active":"2024-02-12T16:45:43.830617+0000","last_peered":"2024-02-12T16:45:43.830617+0000","last_clean":"2024-02-12T16:45:43.830617+0000","last_became_active":"2024-02-12T16:42:21.962544+0000","last_became_peered":"2024-02-12T16:42:21.962544+0000","last_unstale":"2024-02-12T16:45:43.830617+0000","last_undegraded":"2024-02-12T16:45:43.830617+0000","last_fullsized":"2024-02-12T16:45:43.830617+0000","mapping_epoch":916,"log_start":"931'4400","ondisk_log_start":"931'4400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10020,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":292945920,"num_objects":4470,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14420,"num_write_kb":604480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":98,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.385115+0000","last_change":"2024-02-12T16:42:23.724712+0000","last_active":"2024-02-12T16:45:25.385115+0000","last_peered":"2024-02-12T16:45:25.385115+0000","last_clean":"2024-02-12T16:45:25.385115+0000","last_became_active":"2024-02-12T16:41:17.958639+0000","last_became_peered":"2024-02-12T16:41:17.958639+0000","last_unstale":"2024-02-12T16:45:25.385115+0000","last_undegraded":"2024-02-12T16:45:25.385115+0000","last_fullsized":"2024-02-12T16:45:25.385115+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.724526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.724526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"974'14245","reported_seq":24020,"reported_epoch":974,"state":"active+recovering","last_fresh":"2024-02-12T16:45:41.915181+0000","last_change":"2024-02-12T16:43:16.243860+0000","last_active":"2024-02-12T16:45:41.915181+0000","last_peered":"2024-02-12T16:45:41.915181+0000","last_clean":"2024-02-12T16:43:15.356599+0000","last_became_active":"2024-02-12T16:43:16.169927+0000","last_became_peered":"2024-02-12T16:43:16.169927+0000","last_unstale":"2024-02-12T16:45:41.915181+0000","last_undegraded":"2024-02-12T16:45:41.915181+0000","last_fullsized":"2024-02-12T16:45:41.915181+0000","mapping_epoch":935,"log_start":"930'4200","ondisk_log_start":"930'4200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10045,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":297598976,"num_objects":4541,"num_object_clones":0,"num_object_copies":9082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1068,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14245,"num_write_kb":601152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4852,"num_bytes_recovered":317980672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","9"],"object_location_counts":[{"shards":"1,9","objects":1068},{"shards":"1,11","objects":3473}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":364,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.404099+0000","last_change":"2024-02-12T16:45:00.034722+0000","last_active":"2024-02-12T16:45:25.404099+0000","last_peered":"2024-02-12T16:45:25.404099+0000","last_clean":"2024-02-12T16:45:25.404099+0000","last_became_active":"2024-02-12T16:45:00.034337+0000","last_became_peered":"2024-02-12T16:45:00.034337+0000","last_unstale":"2024-02-12T16:45:25.404099+0000","last_undegraded":"2024-02-12T16:45:25.404099+0000","last_fullsized":"2024-02-12T16:45:25.404099+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:30.755995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:30.755995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":256,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:34.633597+0000","last_change":"2024-02-12T16:45:34.633597+0000","last_active":"2024-02-12T16:45:34.633597+0000","last_peered":"2024-02-12T16:45:34.633597+0000","last_clean":"2024-02-12T16:45:34.633597+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:45:34.633597+0000","last_undegraded":"2024-02-12T16:45:34.633597+0000","last_fullsized":"2024-02-12T16:45:34.633597+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:22.106269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:40:18.349211+0000","last_clean_scrub_stamp":"2024-02-12T16:42:22.106269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":271,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:31.847474+0000","last_change":"2024-02-12T16:45:31.847474+0000","last_active":"2024-02-12T16:45:31.847474+0000","last_peered":"2024-02-12T16:45:31.847474+0000","last_clean":"2024-02-12T16:45:31.847474+0000","last_became_active":"2024-02-12T16:43:15.981046+0000","last_became_peered":"2024-02-12T16:43:15.981046+0000","last_unstale":"2024-02-12T16:45:31.847474+0000","last_undegraded":"2024-02-12T16:45:31.847474+0000","last_fullsized":"2024-02-12T16:45:31.847474+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:35.918516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:13.754017+0000","last_clean_scrub_stamp":"2024-02-12T16:42:35.918516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":570,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.385970+0000","last_change":"2024-02-12T16:41:31.381514+0000","last_active":"2024-02-12T16:45:25.385970+0000","last_peered":"2024-02-12T16:45:25.385970+0000","last_clean":"2024-02-12T16:45:25.385970+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:45:25.385970+0000","last_undegraded":"2024-02-12T16:45:25.385970+0000","last_fullsized":"2024-02-12T16:45:25.385970+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:31.381415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:41:31.381415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":533,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-02-12T16:45:25.383969+0000","last_change":"2024-02-12T16:42:25.380280+0000","last_active":"2024-02-12T16:45:25.383969+0000","last_peered":"2024-02-12T16:45:25.383969+0000","last_clean":"2024-02-12T16:45:25.383969+0000","last_became_active":"2024-02-12T16:41:17.950595+0000","last_became_peered":"2024-02-12T16:41:17.950595+0000","last_unstale":"2024-02-12T16:45:25.383969+0000","last_undegraded":"2024-02-12T16:45:25.383969+0000","last_fullsized":"2024-02-12T16:45:25.383969+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:25.380188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:25.380188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4755554328,"num_objects":72565,"num_object_clones":0,"num_object_copies":145130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10986,"num_objects_unfound":0,"num_objects_dirty":72565,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":228799,"num_write_kb":9643585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15525,"num_bytes_recovered":1017446400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10545864704,"data_stored":10545856560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431959099,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2011424,"kb_used_data":1122776,"kb_used_omap":0,"kb_used_meta":888640,"kb_avail":91733728,"statfs":{"total":95995035648,"available":93935337472,"internally_reserved":0,"allocated":1149722624,"data_stored":1149313058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":909967360},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:44:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.72399999999999998,"15min":1.7410000000000001},"min":{"1min":0.31,"5min":0.248,"15min":0.248},"max":{"1min":2.903,"5min":3.9159999999999999,"15min":124.437},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.149,"15min":1.978},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.29199999999999998},"max":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":126.926},"last":3.3500000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.282,"15min":1.054},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":8.8450000000000006,"5min":8.8450000000000006,"15min":8.9849999999999994},"last":1.131},{"interface":"front","average":{"1min":1.117,"5min":1.1080000000000001,"15min":0.92700000000000005},"min":{"1min":0.33100000000000002,"5min":0.29899999999999999,"15min":0.28499999999999998},"max":{"1min":4.391,"5min":5.3949999999999996,"15min":11.167},"last":1.3180000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":3.2770000000000001,"15min":1.867},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":4.2750000000000004,"5min":219.56100000000001,"15min":219.56100000000001},"last":3.4409999999999998},{"interface":"front","average":{"1min":0.626,"5min":2.9159999999999999,"15min":1.778},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.1120000000000001,"5min":219.79300000000001,"15min":219.79300000000001},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Feb 12 16:45:39 2024","interfaces":[{"interface":"back","average":{"1min":4.9969999999999999,"5min":1.9379999999999999,"15min":2.0640000000000001},"min":{"1min":0.376,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":46.445,"5min":46.445,"15min":121.169},"last":3.4769999999999999},{"interface":"front","average":{"1min":4.6719999999999997,"5min":1.8149999999999999,"15min":2.0169999999999999},"min":{"1min":0.36899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":46.774999999999999,"5min":46.774999999999999,"15min":121.181},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:44:58 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":0.93700000000000006,"15min":0.91400000000000003},"min":{"1min":0.40400000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.4710000000000001,"5min":6.1669999999999998,"15min":6.2039999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1990000000000001,"15min":1.0840000000000001},"min":{"1min":0.35799999999999998,"5min":0.315,"15min":0.315},"max":{"1min":5.7469999999999999,"5min":5.7469999999999999,"15min":7.3920000000000003},"last":3.4020000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:44:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6319999999999999,"5min":1.232,"15min":1.1819999999999999},"min":{"1min":0.44900000000000001,"5min":0.37,"15min":0.34100000000000003},"max":{"1min":5.1280000000000001,"5min":5.1280000000000001,"15min":5.968},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.581,"5min":1.202,"15min":1.236},"min":{"1min":0.497,"5min":0.38300000000000001,"15min":0.33100000000000002},"max":{"1min":5.8200000000000003,"5min":5.8200000000000003,"15min":7.4249999999999998},"last":3.5139999999999998}]},{"osd":6,"last update":"Mon Feb 12 16:44:58 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.127,"15min":1.139},"min":{"1min":0.46700000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.9379999999999997,"5min":6.202,"15min":6.4160000000000004},"last":0.622},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.161,"15min":1.268},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":5.9980000000000002,"5min":6.1289999999999996,"15min":9.5389999999999997},"last":0.65400000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:44:49 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.1870000000000001,"15min":1.137},"min":{"1min":0.53400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":5.9660000000000002,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.1799999999999999,"5min":0.94399999999999995,"15min":0.90400000000000003},"min":{"1min":0.52900000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":5.0739999999999998,"5min":5.0739999999999998,"15min":5.0739999999999998},"last":0.754}]},{"osd":8,"last update":"Mon Feb 12 16:44:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.1599999999999999,"15min":1.038},"min":{"1min":0.43099999999999999,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":5.9039999999999999,"5min":6.306,"15min":6.4459999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.998,"15min":1.0680000000000001},"min":{"1min":0.47199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.0640000000000001,"5min":3.1190000000000002,"15min":5.4880000000000004},"last":3.7530000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:44:58 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.073,"15min":1.2050000000000001},"min":{"1min":0.501,"5min":0.437,"15min":0.437},"max":{"1min":3.1269999999999998,"5min":3.1480000000000001,"15min":7.5709999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.091,"5min":1.137,"15min":1.1719999999999999},"min":{"1min":0.55200000000000005,"5min":0.47399999999999998,"15min":0.44700000000000001},"max":{"1min":3.1629999999999998,"5min":3.21,"15min":5.7089999999999996},"last":0.80300000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:44:58 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.5089999999999999,"15min":1.29},"min":{"1min":0.624,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.6159999999999997,"5min":9.6159999999999997,"15min":9.6159999999999997},"last":3.8610000000000002},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.4990000000000001,"15min":1.262},"min":{"1min":0.59999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.8710000000000004,"5min":7.6479999999999997,"15min":7.6479999999999997},"last":0.81200000000000006}]}]},{"osd":10,"up_from":83,"seq":356482286148,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2055456,"kb_used_data":1160984,"kb_used_omap":0,"kb_used_meta":894464,"kb_avail":91689696,"statfs":{"total":95995035648,"available":93890248704,"internally_reserved":0,"allocated":1188847616,"data_stored":1188438050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915931136},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.99399999999999999,"15min":1.196},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":2.9199999999999999,"5min":4.7140000000000004,"15min":111.935},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88400000000000001,"15min":1.274},"min":{"1min":0.29899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.641,"5min":4.3920000000000003,"15min":111.316},"last":1.353}]},{"osd":1,"last update":"Mon Feb 12 16:45:26 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.872,"15min":1.0740000000000001},"min":{"1min":0.313,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.5170000000000003,"5min":5.5170000000000003,"15min":5.5170000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1,"15min":1.1040000000000001},"min":{"1min":0.32200000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.4690000000000003,"5min":7.2000000000000002,"15min":7.2000000000000002},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.012,"15min":1.341},"min":{"1min":0.36699999999999999,"5min":0.314,"15min":0.28100000000000003},"max":{"1min":3.9020000000000001,"5min":9.4179999999999993,"15min":106.822},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.85199999999999998,"15min":1.3440000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":5.5599999999999996,"5min":5.5599999999999996,"15min":107.88800000000001},"last":2.2000000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.1519999999999999,"15min":1.49},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":4.4189999999999996,"5min":9.3699999999999992,"15min":107.014},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":1.0820000000000001,"15min":1.407},"min":{"1min":0.35999999999999999,"5min":0.28699999999999998,"15min":0.27900000000000003},"max":{"1min":3.0699999999999998,"5min":3.7210000000000001,"15min":110.80800000000001},"last":0.74299999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.0740000000000001,"15min":1.022},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":5.8380000000000001,"5min":5.8380000000000001,"15min":6.3689999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.1579999999999999,"5min":0.89700000000000002,"15min":0.96599999999999997},"min":{"1min":0.51900000000000002,"5min":0.38600000000000001,"15min":0.318},"max":{"1min":5.649,"5min":5.649,"15min":5.649},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.903,"15min":1.284},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":5.734,"5min":74.137,"15min":74.137},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.47,"5min":1.8029999999999999,"15min":1.421},"min":{"1min":0.55500000000000005,"5min":0.44,"15min":0.41299999999999998},"max":{"1min":5.9249999999999998,"5min":74.063999999999993,"15min":74.063999999999993},"last":0.61599999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1759999999999999,"15min":2.5870000000000002},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.36099999999999999},"max":{"1min":5.7729999999999997,"5min":5.7729999999999997,"15min":460.90199999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.0640000000000001,"15min":2.573},"min":{"1min":0.61799999999999999,"5min":0.44800000000000001,"15min":0.44},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":460.93299999999999},"last":0.64400000000000002}]},{"osd":7,"last update":"Mon Feb 12 16:44:42 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":4.3579999999999997,"15min":4.9539999999999997},"min":{"1min":0.47699999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.4929999999999999,"5min":74.007000000000005,"15min":74.007000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.323,"5min":4.173,"15min":4.6479999999999997},"min":{"1min":0.57199999999999995,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":5.8129999999999997,"5min":73.819000000000003,"15min":73.819000000000003},"last":1.409}]},{"osd":8,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.1819999999999999,"15min":1.1879999999999999},"min":{"1min":0.60899999999999999,"5min":0.42199999999999999,"15min":0.35499999999999998},"max":{"1min":7.5810000000000004,"5min":7.5810000000000004,"15min":8.4420000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.3560000000000001,"15min":1.2310000000000001},"min":{"1min":0.45600000000000002,"5min":0.41299999999999998,"15min":0.36299999999999999},"max":{"1min":3.859,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":0.73299999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.212,"15min":1.1299999999999999},"min":{"1min":0.51800000000000002,"5min":0.35499999999999998,"15min":0.34699999999999998},"max":{"1min":5.8029999999999999,"5min":5.8029999999999999,"15min":6.2560000000000002},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.427,"5min":1.101,"15min":1.149},"min":{"1min":0.67300000000000004,"5min":0.378,"15min":0.378},"max":{"1min":5.7519999999999998,"5min":5.7519999999999998,"15min":6.0030000000000001},"last":0.68500000000000005}]},{"osd":11,"last update":"Mon Feb 12 16:45:06 2024","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.5309999999999999,"15min":1.333},"min":{"1min":0.51300000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":7.8289999999999997,"5min":8.7569999999999997,"15min":8.7569999999999997},"last":3.2890000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.3180000000000001,"15min":1.262},"min":{"1min":0.67800000000000005,"5min":0.504,"15min":0.38100000000000001},"max":{"1min":7.891,"5min":7.891,"15min":7.891},"last":3.2789999999999999}]}]},{"osd":9,"up_from":0,"seq":317827580492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":8,"up_from":65,"seq":279172874838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1166112,"kb_used_data":572312,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":92579040,"statfs":{"total":95995035648,"available":94800936960,"internally_reserved":0,"allocated":586047488,"data_stored":585637922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.70299999999999996,"15min":1.1439999999999999},"min":{"1min":0.33900000000000002,"5min":0.27500000000000002,"15min":0.23899999999999999},"max":{"1min":0.66200000000000003,"5min":5.96,"15min":134.01499999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.81200000000000006,"15min":1.3360000000000001},"min":{"1min":0.35199999999999998,"5min":0.308,"15min":0.28799999999999998},"max":{"1min":0.76200000000000001,"5min":6.7729999999999997,"15min":133.667},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:45:18 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.022,"15min":0.98899999999999999},"min":{"1min":0.35199999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":8.1210000000000004,"5min":8.1210000000000004,"15min":8.1210000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.2989999999999999,"5min":0.83699999999999997,"15min":0.76000000000000001},"min":{"1min":0.44400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.4819999999999993,"5min":8.4819999999999993,"15min":8.4819999999999993},"last":0.68600000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":1.9399999999999999,"5min":3.2320000000000002,"15min":8.6430000000000007},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2609999999999999,"15min":1.091},"min":{"1min":0.48999999999999999,"5min":0.378,"15min":0.26800000000000002},"max":{"1min":2.0070000000000001,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.84099999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.008,"15min":1.1739999999999999},"min":{"1min":0.38100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":10.231,"5min":10.231,"15min":25.919},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.0840000000000001,"15min":1.2749999999999999},"min":{"1min":0.39600000000000002,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":10.265000000000001,"5min":10.265000000000001,"15min":25.984999999999999},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.91200000000000003,"15min":0.88100000000000001},"min":{"1min":0.35499999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":3.3620000000000001,"15min":8.609},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.95999999999999996,"15min":1.0609999999999999},"min":{"1min":0.45200000000000001,"5min":0.38200000000000001,"15min":0.37},"max":{"1min":0.97399999999999998,"5min":4.4500000000000002,"15min":9.0890000000000004},"last":0.77300000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:44:56 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.86299999999999999,"15min":1.0009999999999999},"min":{"1min":0.51000000000000001,"5min":0.38,"15min":0.318},"max":{"1min":0.84499999999999997,"5min":4.5069999999999997,"15min":7.2960000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":1.123,"15min":1.2250000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.35199999999999998},"max":{"1min":5.452,"5min":6.681,"15min":9.0540000000000003},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Feb 12 16:44:56 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1100000000000001,"15min":1.0960000000000001},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.38200000000000001},"max":{"1min":5.5369999999999999,"5min":5.5369999999999999,"15min":8.5220000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.026,"15min":1.2529999999999999},"min":{"1min":0.52200000000000002,"5min":0.41399999999999998,"15min":0.34599999999999997},"max":{"1min":1.1359999999999999,"5min":4.5140000000000002,"15min":9.1500000000000004},"last":0.72099999999999997}]},{"osd":7,"last update":"Mon Feb 12 16:44:43 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.244,"15min":1.2110000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":6.5970000000000004,"5min":6.5970000000000004,"15min":6.5970000000000004},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.012,"15min":0.99099999999999999},"min":{"1min":0.56999999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":3.8530000000000002,"5min":3.8530000000000002,"15min":3.8530000000000002},"last":0.878}]},{"osd":9,"last update":"Mon Feb 12 16:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.96699999999999997,"15min":1.1659999999999999},"min":{"1min":0.48499999999999999,"5min":0.40600000000000003,"15min":0.371},"max":{"1min":1.425,"5min":4.407,"15min":9.1170000000000009},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.0760000000000001,"15min":1.2709999999999999},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.47099999999999997},"max":{"1min":1.4610000000000001,"5min":4.5380000000000003,"15min":9.1980000000000004},"last":1.097}]},{"osd":10,"last update":"Mon Feb 12 16:45:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.0409999999999999,"15min":1.0469999999999999},"min":{"1min":0.45900000000000002,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":2.5209999999999999,"5min":4.5289999999999999,"15min":8.452},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.099,"15min":1.1120000000000001},"min":{"1min":0.53800000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.359,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":0.625}]},{"osd":11,"last update":"Mon Feb 12 16:45:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.1359999999999999,"15min":1.1120000000000001},"min":{"1min":0.50800000000000001,"5min":0.437,"15min":0.42699999999999999},"max":{"1min":2.6349999999999998,"5min":4.7750000000000004,"15min":8.6790000000000003},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":1.169,"15min":1.141},"min":{"1min":0.57399999999999995,"5min":0.45800000000000002,"15min":0.44600000000000001},"max":{"1min":1.0860000000000001,"5min":4.7549999999999999,"15min":5.141},"last":0.98199999999999998}]}]},{"osd":7,"up_from":923,"seq":3964254814246,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1199012,"kb_used_data":876444,"kb_used_omap":0,"kb_used_meta":322559,"kb_avail":92546140,"statfs":{"total":95995035648,"available":94767247360,"internally_reserved":0,"allocated":897478656,"data_stored":897065018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":330300468},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.82099999999999995,"15min":0.80000000000000004},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.871,"15min":0.86899999999999999},"min":{"1min":0.317,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.2110000000000003,"5min":5.2110000000000003,"15min":5.2110000000000003},"last":0.33300000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.026,"15min":1.0409999999999999},"min":{"1min":0.30599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.4909999999999997,"5min":5.4909999999999997,"15min":5.4909999999999997},"last":0.377},{"interface":"front","average":{"1min":1.528,"5min":0.76000000000000001,"15min":0.63200000000000001},"min":{"1min":0.33100000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.444,"5min":5.444,"15min":5.444},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:45:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.75,"15min":0.72199999999999998},"min":{"1min":0.36499999999999999,"5min":0.318,"15min":0.318},"max":{"1min":5.2359999999999998,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.0269999999999999,"15min":1.0549999999999999},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.21,"15min":1.1859999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":5.7389999999999999,"5min":5.7389999999999999,"15min":5.7389999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.2050000000000001,"15min":1.169},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":5.7720000000000002,"5min":5.7720000000000002,"15min":5.7720000000000002},"last":0.65000000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.72299999999999998,"15min":0.68300000000000005},"min":{"1min":0.35199999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.4199999999999999,"5min":5.4199999999999999,"15min":5.4199999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.397,"5min":0.84599999999999997,"15min":0.754},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":8.4719999999999995,"5min":8.4719999999999995,"15min":8.4719999999999995},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.0589999999999999,"15min":1.0009999999999999},"min":{"1min":0.42399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.474,"15min":1.502},"min":{"1min":0.46400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.7069999999999999,"5min":5.7069999999999999,"15min":5.7069999999999999},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":0.871,"15min":0.80700000000000005},"min":{"1min":0.41799999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.54,"5min":5.54,"15min":5.54},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.101,"5min":0.92900000000000005,"15min":0.90100000000000002},"min":{"1min":0.46300000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.508,"5min":5.508,"15min":5.508},"last":0.622}]},{"osd":8,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0529999999999999,"15min":1.0429999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.874},{"interface":"front","average":{"1min":1.323,"5min":0.98999999999999999,"15min":0.93400000000000005},"min":{"1min":0.58699999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.73699999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1659999999999999,"15min":1.1759999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":5.6559999999999997,"5min":5.6559999999999997,"15min":5.6559999999999997},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.3620000000000001,"5min":0.92300000000000004,"15min":0.84999999999999998},"min":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":5.7679999999999998,"5min":5.7679999999999998,"15min":5.7679999999999998},"last":0.70899999999999996}]},{"osd":11,"last update":"Mon Feb 12 16:44:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":1.244,"15min":1.1499999999999999},"min":{"1min":0.60799999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":9.7409999999999997,"5min":9.7409999999999997,"15min":9.7409999999999997},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3899999999999999,"15min":1.429},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":5.5670000000000002,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":0.76400000000000001}]}]},{"osd":6,"up_from":0,"seq":206158430822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575499,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3243428,"kb_used_data":2056732,"kb_used_omap":0,"kb_used_meta":1186688,"kb_avail":90501724,"statfs":{"total":95995035648,"available":92673765376,"internally_reserved":0,"allocated":2106093568,"data_stored":2105679930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1215168512},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.79700000000000004,"15min":0.96199999999999997},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":5.8700000000000001,"5min":7.532,"15min":10.420999999999999},"last":0.499},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.78700000000000003,"15min":0.98799999999999999},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":4.0499999999999998,"5min":7.3540000000000001,"15min":10.26},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:44:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.83299999999999996,"15min":1.091},"min":{"1min":0.318,"5min":0.29399999999999998,"15min":0.22700000000000001},"max":{"1min":2.6579999999999999,"5min":4.2679999999999998,"15min":10.755000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.93700000000000006,"15min":1.0840000000000001},"min":{"1min":0.27700000000000002,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":2.927,"5min":7.9539999999999997,"15min":9.702},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":0.98899999999999999,"15min":1.1679999999999999},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":4.1820000000000004,"5min":8.9920000000000009,"15min":26.664000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.113,"15min":1.389},"min":{"1min":0.50700000000000001,"5min":0.34599999999999997,"15min":0.26200000000000001},"max":{"1min":5.2080000000000002,"5min":8.9610000000000003,"15min":27.148},"last":0.68799999999999994}]},{"osd":4,"last update":"Mon Feb 12 16:44:46 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91300000000000003,"15min":1.0049999999999999},"min":{"1min":0.38200000000000001,"5min":0.34699999999999998,"15min":0.312},"max":{"1min":3.226,"5min":3.6549999999999998,"15min":7.7549999999999999},"last":0.755},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.97299999999999998,"15min":1.0700000000000001},"min":{"1min":0.41599999999999998,"5min":0.372,"15min":0.312},"max":{"1min":2.73,"5min":3.7309999999999999,"15min":6.9119999999999999},"last":0.83499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.137,"15min":3.9990000000000001},"min":{"1min":0.38700000000000001,"5min":0.32100000000000001,"15min":0.309},"max":{"1min":5.2530000000000001,"5min":8.0190000000000001,"15min":524.85299999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.198,"15min":4.1020000000000003},"min":{"1min":0.432,"5min":0.34799999999999998,"15min":0.32800000000000001},"max":{"1min":4.194,"5min":8.7050000000000001,"15min":519.89700000000005},"last":0.76500000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:45:30 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.0529999999999999,"15min":1.2529999999999999},"min":{"1min":0.43099999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":7.1539999999999999,"5min":7.1539999999999999,"15min":7.1539999999999999},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.248,"15min":1.4299999999999999},"min":{"1min":0.54100000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.1600000000000001,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":0.873}]},{"osd":7,"last update":"Mon Feb 12 16:38:59 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0629999999999999,"15min":3.2370000000000001},"min":{"1min":0.44700000000000001,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":522.79200000000003},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.25,"15min":3.198},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":4.3360000000000003,"5min":9.0069999999999997,"15min":520.64499999999998},"last":1.5800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:45:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4039999999999999,"15min":1.3089999999999999},"min":{"1min":0.63600000000000001,"5min":0.48499999999999999,"15min":0.379},"max":{"1min":5.3390000000000004,"5min":8.8100000000000005,"15min":9.1430000000000007},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.5,"15min":1.385},"min":{"1min":0.70599999999999996,"5min":0.54300000000000004,"15min":0.42499999999999999},"max":{"1min":5.3789999999999996,"5min":8.9269999999999996,"15min":9.1739999999999995},"last":1.008}]},{"osd":9,"last update":"Mon Feb 12 16:45:22 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.081,"15min":1.167},"min":{"1min":0.497,"5min":0.40200000000000002,"15min":0.38400000000000001},"max":{"1min":1.4099999999999999,"5min":8.1539999999999999,"15min":8.3949999999999996},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.1619999999999999,"15min":1.258},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.45300000000000001},"max":{"1min":1.474,"5min":8.2409999999999997,"15min":8.4969999999999999},"last":0.88800000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:45:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.321,"15min":1.284},"min":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.42999999999999999},"max":{"1min":1.3700000000000001,"5min":8.1140000000000008,"15min":8.3469999999999995},"last":0.747},{"interface":"front","average":{"1min":1.345,"5min":1.548,"15min":1.3759999999999999},"min":{"1min":0.66900000000000004,"5min":0.51200000000000001,"15min":0.47999999999999998},"max":{"1min":5.2990000000000004,"5min":8.8469999999999995,"15min":9.0440000000000005},"last":0.96399999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:44:57 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.5920000000000001,"15min":1.7110000000000001},"min":{"1min":0.58899999999999997,"5min":0.53800000000000003,"15min":0.47099999999999997},"max":{"1min":4.4829999999999997,"5min":8.8849999999999998,"15min":88.679000000000002},"last":4.6529999999999996},{"interface":"front","average":{"1min":1.125,"5min":1.252,"15min":1.587},"min":{"1min":0.56200000000000006,"5min":0.51800000000000002,"15min":0.46100000000000002},"max":{"1min":3.657,"5min":8.0730000000000004,"15min":88.519000000000005},"last":0.82999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359739031,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887712,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":886912,"kb_avail":92857440,"statfs":{"total":95995035648,"available":95086018560,"internally_reserved":0,"allocated":811008,"data_stored":401442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":908197888},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":2.1349999999999998,"15min":1.3680000000000001},"min":{"1min":0.26300000000000001,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":2.6989999999999998,"5min":118.657,"15min":118.657},"last":1.008},{"interface":"front","average":{"1min":0.68500000000000005,"5min":2.3650000000000002,"15min":1.5629999999999999},"min":{"1min":0.313,"5min":0.29699999999999999,"15min":0.23400000000000001},"max":{"1min":2.1360000000000001,"5min":118.58,"15min":118.58},"last":0.75700000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.90100000000000002},"min":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0999999999999996,"5min":4.0999999999999996,"15min":4.0999999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.83799999999999997,"15min":0.89700000000000002},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.0469999999999997,"5min":4.0469999999999997,"15min":4.0469999999999997},"last":0.65700000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:45:17 2024","interfaces":[{"interface":"back","average":{"1min":11.071999999999999,"5min":2.9889999999999999,"15min":3.0390000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.252},"max":{"1min":193.114,"5min":193.114,"15min":377.06299999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":10.912000000000001,"5min":2.9260000000000002,"15min":3.1709999999999998},"min":{"1min":0.314,"5min":0.26600000000000001,"15min":0.23799999999999999},"max":{"1min":193.22,"5min":193.22,"15min":379.18000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.75700000000000001,"15min":0.97299999999999998},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.29099999999999998},"max":{"1min":1.0069999999999999,"5min":2.21,"15min":7.2869999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.82599999999999996,"15min":1.095},"min":{"1min":0.39500000000000002,"5min":0.35199999999999998,"15min":0.29499999999999998},"max":{"1min":2.2869999999999999,"5min":3.9580000000000002,"15min":8.4949999999999992},"last":0.70499999999999996}]},{"osd":5,"last update":"Mon Feb 12 16:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.97099999999999997,"15min":1.716},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.30399999999999999},"max":{"1min":3.129,"5min":8.532,"15min":181.46199999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":1.099,"15min":1.845},"min":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.32800000000000001},"max":{"1min":4.4640000000000004,"5min":8.6519999999999992,"15min":186.41999999999999},"last":0.63400000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:45:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.85899999999999999,"15min":1.0149999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.403,"5min":6.6950000000000003,"15min":6.6950000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.83999999999999997,"15min":1.1899999999999999},"min":{"1min":0.45200000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91000000000000003,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":0.749}]},{"osd":7,"last update":"Mon Feb 12 16:39:13 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.24,"15min":2.706},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.374},"max":{"1min":1.835,"5min":6.1130000000000004,"15min":191.47200000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4019999999999999,"15min":2.823},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":1.897,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.081}]},{"osd":8,"last update":"Mon Feb 12 16:44:55 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.0189999999999999,"15min":1.1950000000000001},"min":{"1min":0.49099999999999999,"5min":0.442,"15min":0.41299999999999998},"max":{"1min":3.2160000000000002,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.1379999999999999,"15min":1.3009999999999999},"min":{"1min":0.56100000000000005,"5min":0.51600000000000001,"15min":0.48299999999999998},"max":{"1min":4.5229999999999997,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":0.76900000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:45:34 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.99299999999999999,"15min":1.2090000000000001},"min":{"1min":0.46400000000000002,"5min":0.42299999999999999,"15min":0.34999999999999998},"max":{"1min":2.3460000000000001,"5min":4.1130000000000004,"15min":8.4629999999999992},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.96999999999999997,"15min":1.254},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6240000000000001,"5min":6.7770000000000001,"15min":12.943},"last":0.73399999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.98799999999999999,"15min":1.173},"min":{"1min":0.49099999999999999,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":1.119,"5min":3.4569999999999999,"15min":13.157},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.046,"15min":1.258},"min":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.48299999999999998},"max":{"1min":1.155,"5min":3.4860000000000002,"15min":13.238},"last":0.84199999999999997}]},{"osd":11,"last update":"Mon Feb 12 16:44:55 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.3009999999999999,"15min":1.415},"min":{"1min":0.58499999999999996,"5min":0.50900000000000001,"15min":0.40300000000000002},"max":{"1min":5.8079999999999998,"5min":5.8079999999999998,"15min":8.4280000000000008},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.127,"15min":1.2430000000000001},"min":{"1min":0.59299999999999997,"5min":0.50600000000000001,"15min":0.47299999999999998},"max":{"1min":0.96999999999999997,"5min":4.9960000000000004,"15min":7.3799999999999999},"last":0.81000000000000005}]}]},{"osd":2,"up_from":21,"seq":90194313861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594592,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93150560,"statfs":{"total":95995035648,"available":95386173440,"internally_reserved":0,"allocated":811008,"data_stored":401442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.66200000000000003,"15min":0.77300000000000002},"min":{"1min":0.29399999999999998,"5min":0.223,"15min":0.223},"max":{"1min":2.0529999999999999,"5min":3.996,"15min":3.996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.878,"5min":0.68799999999999994,"15min":0.80300000000000005},"min":{"1min":0.33600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.9690000000000003,"5min":4.9690000000000003,"15min":4.9690000000000003},"last":0.29299999999999998}]},{"osd":1,"last update":"Mon Feb 12 16:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.1080000000000001,"15min":0.996},"min":{"1min":0.32000000000000001,"5min":0.27200000000000002,"15min":0.19500000000000001},"max":{"1min":4.2779999999999996,"5min":24.184999999999999,"15min":24.184999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.1299999999999999,"15min":1.1919999999999999},"min":{"1min":0.34699999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.0839999999999996,"5min":24.292000000000002,"15min":27.562999999999999},"last":0.441}]},{"osd":3,"last update":"Mon Feb 12 16:44:46 2024","interfaces":[{"interface":"back","average":{"1min":6.1879999999999997,"5min":2.1659999999999999,"15min":1.802},"min":{"1min":0.35799999999999998,"5min":0.308,"15min":0.28100000000000003},"max":{"1min":113.68000000000001,"5min":113.68000000000001,"15min":113.68000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":6.2759999999999998,"5min":2.278,"15min":1.8},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.25800000000000001},"max":{"1min":113.815,"5min":113.815,"15min":113.815},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.755,"15min":0.95199999999999996},"min":{"1min":0.38900000000000001,"5min":0.309,"15min":0.29499999999999998},"max":{"1min":1.268,"5min":3.0470000000000002,"15min":8.3439999999999994},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.81299999999999994,"15min":1.05},"min":{"1min":0.41399999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":1.431,"5min":6.9409999999999998,"15min":7.0019999999999998},"last":0.52000000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:45:03 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.878,"15min":0.93300000000000005},"min":{"1min":0.42899999999999999,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":4.056,"5min":5.1319999999999997,"15min":8.7509999999999994},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.91000000000000003,"15min":1.0780000000000001},"min":{"1min":0.44800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":7.0590000000000002},"last":0.88900000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.88700000000000001,"15min":1.1180000000000001},"min":{"1min":0.47099999999999997,"5min":0.40999999999999998,"15min":0.32000000000000001},"max":{"1min":1.5109999999999999,"5min":7.0049999999999999,"15min":8.4510000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.92300000000000004,"15min":1.1060000000000001},"min":{"1min":0.438,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.3049999999999999,"5min":4.0940000000000003,"15min":8.8179999999999996},"last":0.92300000000000004}]},{"osd":7,"last update":"Mon Feb 12 16:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.58399999999999996,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":1.9019999999999999,"5min":6.2560000000000002,"15min":11.044},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2529999999999999,"15min":1.248},"min":{"1min":0.44600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.4219999999999997,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":1.6499999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.679,"15min":1.3100000000000001},"min":{"1min":0.51300000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":7.0780000000000003,"5min":59.164999999999999,"15min":59.164999999999999},"last":0.62},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.625,"15min":1.2789999999999999},"min":{"1min":0.60299999999999998,"5min":0.44500000000000001,"15min":0.377},"max":{"1min":4.1749999999999998,"5min":58.993000000000002,"15min":58.993000000000002},"last":0.95399999999999996}]},{"osd":9,"last update":"Mon Feb 12 16:45:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.95299999999999996,"15min":1.1970000000000001},"min":{"1min":0.48999999999999999,"5min":0.435,"15min":0.38600000000000001},"max":{"1min":1.552,"5min":5.2210000000000001,"15min":7.7759999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.0760000000000001,"15min":1.2050000000000001},"min":{"1min":0.56799999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.2989999999999999,"5min":5.2549999999999999,"15min":7.2110000000000003},"last":0.48599999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:45:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":5.0780000000000003,"15min":2.512},"min":{"1min":0.63100000000000001,"5min":0.45600000000000002,"15min":0.41399999999999998},"max":{"1min":2.839,"5min":407.577,"15min":407.577},"last":5.7240000000000002},{"interface":"front","average":{"1min":1.3779999999999999,"5min":4.8780000000000001,"15min":2.4580000000000002},"min":{"1min":0.71099999999999997,"5min":0.47799999999999998,"15min":0.41799999999999998},"max":{"1min":4.8019999999999996,"5min":407.45100000000002,"15min":407.45100000000002},"last":0.64900000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:45:27 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4350000000000001,"15min":1.286},"min":{"1min":0.69399999999999995,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":4.3659999999999997,"5min":31.835999999999999,"15min":31.835999999999999},"last":5.5659999999999998},{"interface":"front","average":{"1min":1.171,"5min":1.55,"15min":1.389},"min":{"1min":0.68600000000000005,"5min":0.53500000000000003,"15min":0.47699999999999998},"max":{"1min":5.7510000000000003,"5min":31.978999999999999,"15min":31.978999999999999},"last":0.98799999999999999}]}]},{"osd":3,"up_from":522,"seq":2241972928785,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503840,"kb_used_data":879128,"kb_used_omap":91,"kb_used_meta":624612,"kb_avail":92241312,"statfs":{"total":95995035648,"available":94455103488,"internally_reserved":0,"allocated":900227072,"data_stored":899817506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":639602704},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.72299999999999998,"15min":1.0249999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.20699999999999999},"max":{"1min":4.2530000000000001,"5min":4.2530000000000001,"15min":10.101000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.81799999999999995,"15min":1.0629999999999999},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":9.907,"5min":9.907,"15min":10.436999999999999},"last":2.1179999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":3.8679999999999999,"15min":1.9299999999999999},"min":{"1min":0.32800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.3869999999999996,"5min":245.10900000000001,"15min":245.10900000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.3380000000000001,"5min":3.9540000000000002,"15min":2.085},"min":{"1min":0.503,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.5780000000000003,"5min":245.21199999999999,"15min":245.21199999999999},"last":1.9970000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.7170000000000001,"15min":1.3140000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.252},"max":{"1min":4.4669999999999996,"5min":81.769999999999996,"15min":81.769999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.8149999999999999,"15min":1.482},"min":{"1min":0.439,"5min":0.32300000000000001,"15min":0.23699999999999999},"max":{"1min":4.4989999999999997,"5min":81.876999999999995,"15min":81.876999999999995},"last":0.59599999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":0.99399999999999999,"15min":1.1779999999999999},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.27400000000000002},"max":{"1min":4.5739999999999998,"5min":4.5739999999999998,"15min":6.8220000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.90400000000000003,"15min":1.0780000000000001},"min":{"1min":0.46300000000000002,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":2.3410000000000002,"5min":4.6159999999999997,"15min":6.6840000000000002},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.0580000000000001,"15min":1.169},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.30499999999999999},"max":{"1min":2.2309999999999999,"5min":8.9489999999999998,"15min":11.884},"last":2.1749999999999998},{"interface":"front","average":{"1min":1.573,"5min":1.206,"15min":1.3360000000000001},"min":{"1min":0.38900000000000001,"5min":0.36199999999999999,"15min":0.29899999999999999},"max":{"1min":8.032,"5min":8.032,"15min":12.164999999999999},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.1499999999999999,"15min":1.242},"min":{"1min":0.45300000000000001,"5min":0.379,"15min":0.31900000000000001},"max":{"1min":2.5430000000000001,"5min":9,"15min":9.4689999999999994},"last":2.2160000000000002},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.099,"15min":1.3260000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.30499999999999999},"max":{"1min":4.5350000000000001,"5min":4.5350000000000001,"15min":9.4640000000000004},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.5149999999999999,"15min":1.298},"min":{"1min":0.55000000000000004,"5min":0.36199999999999999,"15min":0.33300000000000002},"max":{"1min":7.9909999999999997,"5min":7.9909999999999997,"15min":7.9909999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.609,"5min":1.3859999999999999,"15min":1.335},"min":{"1min":0.621,"5min":0.371,"15min":0.371},"max":{"1min":6.0899999999999999,"5min":6.7590000000000003,"15min":7.2249999999999996},"last":0.59299999999999997}]},{"osd":8,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":1.153,"15min":1.2529999999999999},"min":{"1min":0.58099999999999996,"5min":0.439,"15min":0.38200000000000001},"max":{"1min":2.5099999999999998,"5min":5.8079999999999998,"15min":5.9669999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.278,"15min":1.3420000000000001},"min":{"1min":0.496,"5min":0.47899999999999998,"15min":0.35599999999999998},"max":{"1min":4.609,"5min":4.9569999999999999,"15min":7.9569999999999999},"last":0.73899999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.1419999999999999,"15min":1.329},"min":{"1min":0.54900000000000004,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":2.54,"5min":5.9260000000000002,"15min":8.9350000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.256,"15min":1.3220000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.38300000000000001},"max":{"1min":4.6440000000000001,"5min":4.7489999999999997,"15min":9.5239999999999991},"last":0.77200000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":9.0899999999999999,"15min":3.907},"min":{"1min":0.60099999999999998,"5min":0.45500000000000002,"15min":0.40799999999999997},"max":{"1min":4.5019999999999998,"5min":738.99900000000002,"15min":738.99900000000002},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":9.0960000000000001,"15min":4.0490000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.45600000000000002},"max":{"1min":4.5190000000000001,"5min":746.45299999999997,"15min":746.45299999999997},"last":2.2559999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:44:38 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.498,"15min":1.633},"min":{"1min":0.55300000000000005,"5min":0.55000000000000004,"15min":0.437},"max":{"1min":4.9130000000000003,"5min":8.9030000000000005,"15min":73.037999999999997},"last":2.3399999999999999},{"interface":"front","average":{"1min":1.6970000000000001,"5min":1.4399999999999999,"15min":1.673},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.53200000000000003},"max":{"1min":4.9370000000000003,"5min":4.9370000000000003,"15min":73.057000000000002},"last":0.51400000000000001}]}]},{"osd":4,"up_from":32,"seq":137438954101,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472864,"kb_used_data":574616,"kb_used_omap":0,"kb_used_meta":898240,"kb_avail":92272288,"statfs":{"total":95995035648,"available":94486822912,"internally_reserved":0,"allocated":588406784,"data_stored":587997218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":919797760},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81999999999999995,"15min":1.02},"min":{"1min":0.372,"5min":0.27100000000000002,"15min":0.27000000000000002},"max":{"1min":1.1080000000000001,"5min":3.9670000000000001,"15min":4.1050000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.92100000000000004,"15min":1.1220000000000001},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.4470000000000001,"5min":5.0330000000000004,"15min":5.0599999999999996},"last":0.65600000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:45:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68200000000000005,"15min":1.268},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":1.3260000000000001,"5min":3.1520000000000001,"15min":56.188000000000002},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.74299999999999999,"15min":1.355},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.25700000000000001},"max":{"1min":0.97299999999999998,"5min":3.0880000000000001,"15min":55.414000000000001},"last":0.80300000000000005}]},{"osd":2,"last update":"Mon Feb 12 16:44:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":3.0840000000000001,"15min":1.8340000000000001},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3100000000000001,"5min":212.685,"15min":212.685},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":2.9910000000000001,"15min":1.7490000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.3540000000000001,"5min":210.95699999999999,"15min":210.95699999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:44:42 2024","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":1.1160000000000001,"15min":1.365},"min":{"1min":0.34000000000000002,"5min":0.33600000000000002,"15min":0.311},"max":{"1min":17.876999999999999,"5min":17.876999999999999,"15min":30.282},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.1080000000000001,"15min":1.347},"min":{"1min":0.34699999999999998,"5min":0.312,"15min":0.25800000000000001},"max":{"1min":17.623999999999999,"5min":17.623999999999999,"15min":30.478000000000002},"last":0.85499999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:44:53 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.75700000000000001,"15min":1.056},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.252},"max":{"1min":4.4690000000000003,"5min":4.4690000000000003,"15min":10.289999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":0.89900000000000002,"15min":1.177},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":4.5720000000000001,"5min":4.5720000000000001,"15min":10.321999999999999},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Feb 12 16:44:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.72499999999999998,"15min":1.0269999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29499999999999998},"max":{"1min":1.075,"5min":2.8809999999999998,"15min":6.5270000000000001},"last":0.625},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.89000000000000001,"15min":1.121},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.29999999999999999},"max":{"1min":1.391,"5min":2.46,"15min":11.968999999999999},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.98199999999999998,"15min":1.046},"min":{"1min":0.35499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.93000000000000005,"5min":3.569,"15min":3.569},"last":1.3169999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.73599999999999999,"15min":0.69799999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.879,"5min":3.879,"15min":3.879},"last":0.67900000000000005}]},{"osd":8,"last update":"Mon Feb 12 16:44:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.0489999999999999,"15min":1.1539999999999999},"min":{"1min":0.52300000000000002,"5min":0.44400000000000001,"15min":0.40300000000000002},"max":{"1min":1.7430000000000001,"5min":3.8279999999999998,"15min":7.5179999999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.94399999999999995,"15min":1.1240000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":4.6470000000000002,"5min":4.6470000000000002,"15min":7.9349999999999996},"last":0.39300000000000002}]},{"osd":9,"last update":"Mon Feb 12 16:45:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.96899999999999997,"15min":1.387},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.39300000000000002},"max":{"1min":1.0840000000000001,"5min":2.2879999999999998,"15min":63.545000000000002},"last":0.88},{"interface":"front","average":{"1min":0.875,"5min":1.0449999999999999,"15min":1.407},"min":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.44600000000000001},"max":{"1min":1.1240000000000001,"5min":2.3370000000000002,"15min":63.421999999999997},"last":0.90200000000000002}]},{"osd":10,"last update":"Mon Feb 12 16:45:35 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.139,"15min":1.2150000000000001},"min":{"1min":0.46300000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":4.4210000000000003,"5min":4.6079999999999997,"15min":5.6109999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.079,"15min":1.1339999999999999},"min":{"1min":0.53500000000000003,"5min":0.45500000000000002,"15min":0.41799999999999998},"max":{"1min":6.8920000000000003,"5min":6.8920000000000003,"15min":6.8920000000000003},"last":0.76000000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:44:42 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.101,"15min":1.2509999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.48999999999999999},"max":{"1min":1.351,"5min":3.4710000000000001,"15min":7.4880000000000004},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.2110000000000001,"15min":1.2589999999999999},"min":{"1min":0.59299999999999997,"5min":0.55200000000000005,"15min":0.46400000000000002},"max":{"1min":2.117,"5min":3.4580000000000002,"15min":5.681},"last":0.92500000000000004}]}]},{"osd":5,"up_from":39,"seq":167503725164,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2299680,"kb_used_data":1398424,"kb_used_omap":0,"kb_used_meta":901248,"kb_avail":91445472,"statfs":{"total":95995035648,"available":93640163328,"internally_reserved":0,"allocated":1431986176,"data_stored":1431576610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922877952},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.82199999999999995,"15min":1.345},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.7469999999999999,"5min":3.7610000000000001,"15min":38.366},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.89400000000000002,"15min":1.431},"min":{"1min":0.33000000000000002,"5min":0.255,"15min":0.255},"max":{"1min":5.1459999999999999,"5min":5.1459999999999999,"15min":37.310000000000002},"last":0.38}]},{"osd":1,"last update":"Mon Feb 12 16:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":5.5069999999999997,"15min":2.5409999999999999},"min":{"1min":0.317,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.7879999999999998,"5min":387.76299999999998,"15min":387.76299999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":5.633,"15min":2.6720000000000002},"min":{"1min":0.45600000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.8300000000000001,"5min":387.88099999999997,"15min":387.88099999999997},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.056,"15min":1.143},"min":{"1min":0.41299999999999998,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":3.706,"5min":4.5990000000000002,"15min":6.6459999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.85999999999999999,"15min":1.131},"min":{"1min":0.35399999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.1989999999999998,"5min":7.3470000000000004,"15min":11.448},"last":0.437}]},{"osd":3,"last update":"Mon Feb 12 16:44:59 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.1160000000000001,"15min":1.3799999999999999},"min":{"1min":0.40699999999999997,"5min":0.28299999999999997,"15min":0.27200000000000002},"max":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":22.484000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92000000000000004,"15min":1.3100000000000001},"min":{"1min":0.42099999999999999,"5min":0.32000000000000001,"15min":0.30299999999999999},"max":{"1min":3.794,"5min":4.7839999999999998,"15min":23.34},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Feb 12 16:44:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.96499999999999997,"15min":1.0640000000000001},"min":{"1min":0.36599999999999999,"5min":0.255,"15min":0.255},"max":{"1min":5.7460000000000004,"5min":5.7460000000000004,"15min":6.5999999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0169999999999999,"15min":1.111},"min":{"1min":0.32700000000000001,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":5.069,"5min":5.069,"15min":5.5389999999999997},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:44:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.022,"15min":1.4039999999999999},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.28699999999999998},"max":{"1min":5.1760000000000002,"5min":5.1760000000000002,"15min":68.263999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.877,"5min":0.88900000000000001,"15min":1.4530000000000001},"min":{"1min":0.38,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.6520000000000001,"5min":4.5730000000000004,"15min":67.873000000000005},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Feb 12 16:38:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.86799999999999999,"15min":1.1830000000000001},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.28799999999999998},"max":{"1min":1.9570000000000001,"5min":5.6609999999999996,"15min":9.7710000000000008},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.024,"15min":1.286},"min":{"1min":0.376,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.9670000000000001,"5min":5.6120000000000001,"15min":6.7599999999999998},"last":0.627}]},{"osd":8,"last update":"Mon Feb 12 16:45:28 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.1379999999999999,"15min":1.2010000000000001},"min":{"1min":0.496,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.5449999999999999,"5min":5.8090000000000002,"15min":5.8090000000000002},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.95799999999999996,"15min":1.252},"min":{"1min":0.47499999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.661,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.746}]},{"osd":9,"last update":"Mon Feb 12 16:44:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3260000000000001,"15min":1.2310000000000001},"min":{"1min":0.629,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":5.8540000000000001,"5min":9.0830000000000002,"15min":9.0830000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1060000000000001,"15min":1.2250000000000001},"min":{"1min":0.53500000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.7559999999999998,"5min":6.21,"15min":7.9710000000000001},"last":0.79100000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:44:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3540000000000001,"15min":1.355},"min":{"1min":0.48899999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":5.2119999999999997,"5min":8.3879999999999999,"15min":8.3879999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.476,"5min":1.375,"15min":1.3109999999999999},"min":{"1min":0.63800000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":5.9359999999999999,"5min":5.9359999999999999,"15min":6.867},"last":0.78100000000000003}]},{"osd":11,"last update":"Mon Feb 12 16:45:17 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.4590000000000001,"15min":1.423},"min":{"1min":0.65200000000000002,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":5.2469999999999999,"5min":5.6749999999999998,"15min":8.0640000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.367,"15min":1.347},"min":{"1min":0.76400000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":5.8929999999999998,"5min":5.8929999999999998,"15min":6.9029999999999996},"last":0.67800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2103840768,"data_stored":2103836696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":888733696,"data_stored":888733696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":582615040,"data_stored":582615040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1429733376,"data_stored":1429733376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":596901888,"data_stored":596901888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":887099392,"data_stored":887095320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":584777728,"data_stored":584777728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1151401984,"data_stored":1151401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1178533888,"data_stored":1178533888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1142226944,"data_stored":1142226944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:45:46.289 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-12T16:45:46.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool set unique_pool_7 pg_num 26 2024-02-12T16:45:46.290 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:46.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:46.417 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:46.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:46.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:46.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:46.686 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:46.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:46 smithi169 bash[19604]: audit 2024-02-12T16:45:44.783241+0000 mgr.y (mgr.14146) 1845 : audit [DBG] from='client.17274 v1:172.21.15.121:0/2086524644' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:45:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:46 smithi169 bash[19604]: cluster 2024-02-12T16:45:46.061065+0000 osd.9 (osd.9) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:46 smithi169 bash[19604]: cluster 2024-02-12T16:45:46.061140+0000 osd.9 (osd.9) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:46.844 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:46 smithi123 bash[19119]: audit 2024-02-12T16:45:44.783241+0000 mgr.y (mgr.14146) 1845 : audit [DBG] from='client.17274 v1:172.21.15.121:0/2086524644' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:45:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:46 smithi123 bash[19119]: cluster 2024-02-12T16:45:46.061065+0000 osd.9 (osd.9) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:46 smithi123 bash[19119]: cluster 2024-02-12T16:45:46.061140+0000 osd.9 (osd.9) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:46 smithi121 bash[18489]: audit 2024-02-12T16:45:44.783241+0000 mgr.y (mgr.14146) 1845 : audit [DBG] from='client.17274 v1:172.21.15.121:0/2086524644' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:45:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:46 smithi121 bash[18489]: cluster 2024-02-12T16:45:46.061065+0000 osd.9 (osd.9) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:46 smithi121 bash[18489]: cluster 2024-02-12T16:45:46.061140+0000 osd.9 (osd.9) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:46.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:46.980 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:46.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:47.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:47.106 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:47.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:47.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:47.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:47.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:47.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:47.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:47.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:47.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:47.516 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:47.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:47.622 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:47.650 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:47.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:47 smithi121 bash[18489]: cluster 2024-02-12T16:45:46.315905+0000 mgr.y (mgr.14146) 1846 : cluster [DBG] pgmap v2688: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 10681/143934 objects misplaced (7.421%); 13 MiB/s, 206 objects/s recovering 2024-02-12T16:45:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:47 smithi121 bash[18489]: cluster 2024-02-12T16:45:47.023993+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e975: 12 total, 12 up, 10 in 2024-02-12T16:45:47.760 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:47.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:47 smithi169 bash[19604]: cluster 2024-02-12T16:45:46.315905+0000 mgr.y (mgr.14146) 1846 : cluster [DBG] pgmap v2688: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 10681/143934 objects misplaced (7.421%); 13 MiB/s, 206 objects/s recovering 2024-02-12T16:45:47.767 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:47 smithi169 bash[19604]: cluster 2024-02-12T16:45:47.023993+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e975: 12 total, 12 up, 10 in 2024-02-12T16:45:47.871 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:47.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:47 smithi123 bash[19119]: cluster 2024-02-12T16:45:46.315905+0000 mgr.y (mgr.14146) 1846 : cluster [DBG] pgmap v2688: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 10681/143934 objects misplaced (7.421%); 13 MiB/s, 206 objects/s recovering 2024-02-12T16:45:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:47 smithi123 bash[19119]: cluster 2024-02-12T16:45:47.023993+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e975: 12 total, 12 up, 10 in 2024-02-12T16:45:47.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:48.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:48.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:48.134 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:48.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:48.269 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:48.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:48.403 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:48.508 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:48.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:48.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:48.682 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:48.790 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:48.818 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:48.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:48.960 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:48.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:49.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:49.091 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:49.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:49.195 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:49.225 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:49.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:49.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:49.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:49.366 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:49.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:49.473 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:49.497 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:49.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:49.602 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:49.610 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:49 smithi123 bash[19119]: cluster 2024-02-12T16:45:48.316321+0000 mgr.y (mgr.14146) 1847 : cluster [DBG] pgmap v2690: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 962 op/s; 9718/139020 objects misplaced (6.990%); 10 MiB/s, 162 objects/s recovering 2024-02-12T16:45:49.632 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:49.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:49.736 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:49.769 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:49.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:49 smithi169 bash[19604]: cluster 2024-02-12T16:45:48.316321+0000 mgr.y (mgr.14146) 1847 : cluster [DBG] pgmap v2690: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 962 op/s; 9718/139020 objects misplaced (6.990%); 10 MiB/s, 162 objects/s recovering 2024-02-12T16:45:49.873 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:49.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:49.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:49 smithi121 bash[18489]: cluster 2024-02-12T16:45:48.316321+0000 mgr.y (mgr.14146) 1847 : cluster [DBG] pgmap v2690: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.2 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 962 op/s; 9718/139020 objects misplaced (6.990%); 10 MiB/s, 162 objects/s recovering 2024-02-12T16:45:50.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:50.039 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:50.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:50.171 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:50.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:50.303 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:50.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:50.435 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:50.538 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:50.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:50.668 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:50.696 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:50.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:50.830 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:50.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:50.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:50.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:51.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:51.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:51 smithi123 bash[19119]: audit 2024-02-12T16:45:50.074595+0000 mon.a (mon.0) 3256 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:51 smithi123 bash[19119]: cluster 2024-02-12T16:45:50.316742+0000 mgr.y (mgr.14146) 1848 : cluster [DBG] pgmap v2691: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 969 op/s; 9718/135274 objects misplaced (7.184%); 10 MiB/s, 162 objects/s recovering 2024-02-12T16:45:51.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:51 smithi123 bash[19119]: audit 2024-02-12T16:45:50.519412+0000 mon.a (mon.0) 3257 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:51 smithi123 bash[19119]: audit 2024-02-12T16:45:50.797212+0000 mon.a (mon.0) 3258 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:51.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:51 smithi123 bash[19119]: audit 2024-02-12T16:45:50.798439+0000 mon.a (mon.0) 3259 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:45:51.085 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:51 smithi123 bash[19119]: audit 2024-02-12T16:45:50.805250+0000 mon.a (mon.0) 3260 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:51.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:51.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:51.238 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:51.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:51.272 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:51 smithi121 bash[18489]: audit 2024-02-12T16:45:50.074595+0000 mon.a (mon.0) 3256 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.272 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:51 smithi121 bash[18489]: cluster 2024-02-12T16:45:50.316742+0000 mgr.y (mgr.14146) 1848 : cluster [DBG] pgmap v2691: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 969 op/s; 9718/135274 objects misplaced (7.184%); 10 MiB/s, 162 objects/s recovering 2024-02-12T16:45:51.273 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:51 smithi121 bash[18489]: audit 2024-02-12T16:45:50.519412+0000 mon.a (mon.0) 3257 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.273 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:51 smithi121 bash[18489]: audit 2024-02-12T16:45:50.797212+0000 mon.a (mon.0) 3258 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:51.273 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:51 smithi121 bash[18489]: audit 2024-02-12T16:45:50.798439+0000 mon.a (mon.0) 3259 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:45:51.273 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:51 smithi121 bash[18489]: audit 2024-02-12T16:45:50.805250+0000 mon.a (mon.0) 3260 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:51.367 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:51.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:51.470 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:45:51.470 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:45:51.470 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:45:51.470 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:45:51.470 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:45:51.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:51.505 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:51.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:51 smithi169 bash[19604]: audit 2024-02-12T16:45:50.074595+0000 mon.a (mon.0) 3256 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:51 smithi169 bash[19604]: cluster 2024-02-12T16:45:50.316742+0000 mgr.y (mgr.14146) 1848 : cluster [DBG] pgmap v2691: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 969 op/s; 9718/135274 objects misplaced (7.184%); 10 MiB/s, 162 objects/s recovering 2024-02-12T16:45:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:51 smithi169 bash[19604]: audit 2024-02-12T16:45:50.519412+0000 mon.a (mon.0) 3257 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:51 smithi169 bash[19604]: audit 2024-02-12T16:45:50.797212+0000 mon.a (mon.0) 3258 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:45:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:51 smithi169 bash[19604]: audit 2024-02-12T16:45:50.798439+0000 mon.a (mon.0) 3259 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:45:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:51 smithi169 bash[19604]: audit 2024-02-12T16:45:50.805250+0000 mon.a (mon.0) 3260 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:45:51.610 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:51.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:51.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:51.731 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:45:51.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:51.759 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:45:51.775 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:51.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:51.787 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:45:51.819 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:45:51.859 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:45:51.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:45:51.915 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:51.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:51.918 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:45:51.945 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:45:51.972 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:45:52.006 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:45:52.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:52.034 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:45:52.048 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:52.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:52.064 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:45:52.131 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:52 smithi121 bash[18489]: cluster 2024-02-12T16:45:50.737863+0000 osd.5 (osd.5) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:52.132 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:52 smithi121 bash[18489]: cluster 2024-02-12T16:45:50.737917+0000 osd.5 (osd.5) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:52.132 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:52 smithi121 bash[18489]: cluster 2024-02-12T16:45:51.520930+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e976: 12 total, 12 up, 10 in 2024-02-12T16:45:52.132 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:52 smithi121 bash[18489]: audit 2024-02-12T16:45:51.691463+0000 mon.a (mon.0) 3262 : audit [DBG] from='client.? v1:172.21.15.121:0/1624308630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:52.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:52.178 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:52.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:52.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:52.317 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:52.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:52.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:52 smithi123 bash[19119]: cluster 2024-02-12T16:45:50.737863+0000 osd.5 (osd.5) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:52.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:52 smithi123 bash[19119]: cluster 2024-02-12T16:45:50.737917+0000 osd.5 (osd.5) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:52.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:52 smithi123 bash[19119]: cluster 2024-02-12T16:45:51.520930+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e976: 12 total, 12 up, 10 in 2024-02-12T16:45:52.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:52 smithi123 bash[19119]: audit 2024-02-12T16:45:51.691463+0000 mon.a (mon.0) 3262 : audit [DBG] from='client.? v1:172.21.15.121:0/1624308630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:52.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:52.427 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:52 smithi169 bash[19604]: cluster 2024-02-12T16:45:50.737863+0000 osd.5 (osd.5) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:52.427 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:52 smithi169 bash[19604]: cluster 2024-02-12T16:45:50.737917+0000 osd.5 (osd.5) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:52.427 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:52 smithi169 bash[19604]: cluster 2024-02-12T16:45:51.520930+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e976: 12 total, 12 up, 10 in 2024-02-12T16:45:52.427 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:52 smithi169 bash[19604]: audit 2024-02-12T16:45:51.691463+0000 mon.a (mon.0) 3262 : audit [DBG] from='client.? v1:172.21.15.121:0/1624308630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:45:52.449 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:52.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:52.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:52.592 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:52.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:52.696 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:52.725 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:52.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:52.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:52.864 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:52.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:52.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:52.996 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:53.102 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:53.131 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:53.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:53.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:53.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:53.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-12T16:45:53.408 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:53.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:53.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:53 smithi121 bash[18489]: cluster 2024-02-12T16:45:51.791585+0000 osd.4 (osd.4) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:53.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:53 smithi121 bash[18489]: cluster 2024-02-12T16:45:51.791645+0000 osd.4 (osd.4) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:53.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:53 smithi121 bash[18489]: cluster 2024-02-12T16:45:52.317238+0000 mgr.y (mgr.14146) 1849 : cluster [DBG] pgmap v2693: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 8652/129896 objects misplaced (6.661%); 9.8 MiB/s, 156 objects/s recovering 2024-02-12T16:45:53.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:53 smithi121 bash[18489]: cluster 2024-02-12T16:45:52.522890+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e977: 12 total, 12 up, 10 in 2024-02-12T16:45:53.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:53 smithi121 bash[18489]: audit 2024-02-12T16:45:52.778273+0000 mon.a (mon.0) 3264 : audit [INF] from='client.27133 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:53.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:53 smithi121 bash[18489]: audit 2024-02-12T16:45:52.779814+0000 mon.c (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.121:0/1318946391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:53.540 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:53.541 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:45:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:53.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:53.672 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:53.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:53 smithi169 bash[19604]: cluster 2024-02-12T16:45:51.791585+0000 osd.4 (osd.4) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:53 smithi169 bash[19604]: cluster 2024-02-12T16:45:51.791645+0000 osd.4 (osd.4) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:53 smithi169 bash[19604]: cluster 2024-02-12T16:45:52.317238+0000 mgr.y (mgr.14146) 1849 : cluster [DBG] pgmap v2693: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 8652/129896 objects misplaced (6.661%); 9.8 MiB/s, 156 objects/s recovering 2024-02-12T16:45:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:53 smithi169 bash[19604]: cluster 2024-02-12T16:45:52.522890+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e977: 12 total, 12 up, 10 in 2024-02-12T16:45:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:53 smithi169 bash[19604]: audit 2024-02-12T16:45:52.778273+0000 mon.a (mon.0) 3264 : audit [INF] from='client.27133 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:53 smithi169 bash[19604]: audit 2024-02-12T16:45:52.779814+0000 mon.c (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.121:0/1318946391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:53.814 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:53.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:53 smithi123 bash[19119]: cluster 2024-02-12T16:45:51.791585+0000 osd.4 (osd.4) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:53 smithi123 bash[19119]: cluster 2024-02-12T16:45:51.791645+0000 osd.4 (osd.4) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:53 smithi123 bash[19119]: cluster 2024-02-12T16:45:52.317238+0000 mgr.y (mgr.14146) 1849 : cluster [DBG] pgmap v2693: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 8652/129896 objects misplaced (6.661%); 9.8 MiB/s, 156 objects/s recovering 2024-02-12T16:45:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:53 smithi123 bash[19119]: cluster 2024-02-12T16:45:52.522890+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e977: 12 total, 12 up, 10 in 2024-02-12T16:45:53.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:53 smithi123 bash[19119]: audit 2024-02-12T16:45:52.778273+0000 mon.a (mon.0) 3264 : audit [INF] from='client.27133 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:53.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:53 smithi123 bash[19119]: audit 2024-02-12T16:45:52.779814+0000 mon.c (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.121:0/1318946391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:45:53.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:53.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:53.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:54.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:54.079 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:54.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:54.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:54.215 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:54.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:54.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:54.350 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:54.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:54.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:54.488 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:54.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:54.542 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:54 smithi123 bash[19119]: cluster 2024-02-12T16:45:53.519927+0000 mon.a (mon.0) 3265 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:45:54.542 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:54 smithi123 bash[19119]: audit 2024-02-12T16:45:53.526783+0000 mon.a (mon.0) 3266 : audit [INF] from='client.27133 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:45:54.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:54.617 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:54.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:54.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:54.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:54 smithi121 bash[18489]: cluster 2024-02-12T16:45:53.519927+0000 mon.a (mon.0) 3265 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:45:54.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:54 smithi121 bash[18489]: audit 2024-02-12T16:45:53.526783+0000 mon.a (mon.0) 3266 : audit [INF] from='client.27133 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:45:54.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:54 smithi121 bash[18489]: cluster 2024-02-12T16:45:53.526841+0000 mon.a (mon.0) 3267 : cluster [DBG] osdmap e978: 12 total, 12 up, 10 in 2024-02-12T16:45:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:54.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:54 smithi169 bash[19604]: cluster 2024-02-12T16:45:53.519927+0000 mon.a (mon.0) 3265 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:45:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:54 smithi169 bash[19604]: audit 2024-02-12T16:45:53.526783+0000 mon.a (mon.0) 3266 : audit [INF] from='client.27133 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:45:54.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:54 smithi169 bash[19604]: cluster 2024-02-12T16:45:53.526841+0000 mon.a (mon.0) 3267 : cluster [DBG] osdmap e978: 12 total, 12 up, 10 in 2024-02-12T16:45:54.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:54.887 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:54.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:54 smithi123 bash[19119]: cluster 2024-02-12T16:45:53.526841+0000 mon.a (mon.0) 3267 : cluster [DBG] osdmap e978: 12 total, 12 up, 10 in 2024-02-12T16:45:54.943 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:54.989 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:55.018 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:45:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:45:55.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:55.147 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:55.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:55.283 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:55.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:45:55.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:45:55.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:55.561 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:55.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:55.694 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:55.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:55 smithi169 bash[19604]: cluster 2024-02-12T16:45:54.317686+0000 mgr.y (mgr.14146) 1850 : cluster [DBG] pgmap v2696: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 8652/126486 objects misplaced (6.840%); 5.8 MiB/s, 93 objects/s recovering 2024-02-12T16:45:55.839 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:55 smithi123 bash[19119]: cluster 2024-02-12T16:45:54.317686+0000 mgr.y (mgr.14146) 1850 : cluster [DBG] pgmap v2696: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 8652/126486 objects misplaced (6.840%); 5.8 MiB/s, 93 objects/s recovering 2024-02-12T16:45:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:55 smithi121 bash[18489]: cluster 2024-02-12T16:45:54.317686+0000 mgr.y (mgr.14146) 1850 : cluster [DBG] pgmap v2696: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s; 8652/126486 objects misplaced (6.840%); 5.8 MiB/s, 93 objects/s recovering 2024-02-12T16:45:55.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:55.971 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:55.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:56.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:56.098 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:56.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:56.202 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:56.234 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:56.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:56.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:56.373 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:56.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:56.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:56.505 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:56.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:56.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:45:56.607 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:56 smithi123 bash[19119]: cluster 2024-02-12T16:45:55.380556+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:56 smithi123 bash[19119]: cluster 2024-02-12T16:45:55.380608+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:56 smithi123 bash[19119]: cluster 2024-02-12T16:45:55.540473+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e979: 12 total, 12 up, 10 in 2024-02-12T16:45:56.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:56 smithi123 bash[19119]: cluster 2024-02-12T16:45:56.014105+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e980: 12 total, 12 up, 10 in 2024-02-12T16:45:56.640 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:56.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:56.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:56.773 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:56.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:56 smithi169 bash[19604]: cluster 2024-02-12T16:45:55.380556+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:56 smithi169 bash[19604]: cluster 2024-02-12T16:45:55.380608+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:56 smithi169 bash[19604]: cluster 2024-02-12T16:45:55.540473+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e979: 12 total, 12 up, 10 in 2024-02-12T16:45:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:56 smithi169 bash[19604]: cluster 2024-02-12T16:45:56.014105+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e980: 12 total, 12 up, 10 in 2024-02-12T16:45:56.877 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:56.897 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:56 smithi121 bash[18489]: cluster 2024-02-12T16:45:55.380556+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:56.897 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:56 smithi121 bash[18489]: cluster 2024-02-12T16:45:55.380608+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:56.897 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:56 smithi121 bash[18489]: cluster 2024-02-12T16:45:55.540473+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e979: 12 total, 12 up, 10 in 2024-02-12T16:45:56.897 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:56 smithi121 bash[18489]: cluster 2024-02-12T16:45:56.014105+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e980: 12 total, 12 up, 10 in 2024-02-12T16:45:56.904 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:56.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:56.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:57.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:57.048 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:45:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:57.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:45:57.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:57.189 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:57.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:57.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:57.325 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:57.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:57.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:57.468 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:57.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:57.552 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:57 smithi121 bash[18489]: cluster 2024-02-12T16:45:55.921448+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:57.552 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:57 smithi121 bash[18489]: cluster 2024-02-12T16:45:55.921492+0000 osd.1 (osd.1) 171 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:57.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:57.598 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:57.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:57.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:45:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:57.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:45:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:57 smithi169 bash[19604]: cluster 2024-02-12T16:45:55.921448+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:57 smithi169 bash[19604]: cluster 2024-02-12T16:45:55.921492+0000 osd.1 (osd.1) 171 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:57 smithi169 bash[19604]: cluster 2024-02-12T16:45:56.318129+0000 mgr.y (mgr.14146) 1851 : cluster [DBG] pgmap v2699: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 571 op/s; 8364/125322 objects misplaced (6.674%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:45:57.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:57.859 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:57.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:57 smithi123 bash[19119]: cluster 2024-02-12T16:45:55.921448+0000 osd.1 (osd.1) 170 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:57 smithi123 bash[19119]: cluster 2024-02-12T16:45:55.921492+0000 osd.1 (osd.1) 171 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:57 smithi123 bash[19119]: cluster 2024-02-12T16:45:56.318129+0000 mgr.y (mgr.14146) 1851 : cluster [DBG] pgmap v2699: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 571 op/s; 8364/125322 objects misplaced (6.674%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:45:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:57 smithi121 bash[18489]: cluster 2024-02-12T16:45:56.318129+0000 mgr.y (mgr.14146) 1851 : cluster [DBG] pgmap v2699: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 571 op/s; 8364/125322 objects misplaced (6.674%); 4.5 MiB/s, 71 objects/s recovering 2024-02-12T16:45:57.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:58.004 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:45:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:45:58.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:58.140 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:58.246 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:45:58.275 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:58.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:58.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:58.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:58.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:58.689 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:45:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:45:58.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:58.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:58.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:58.956 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:45:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:58.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:45:59.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:59.088 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:45:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:59.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:45:59.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:45:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:59.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:45:59.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:59.356 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:45:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:59.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:45:59.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:59.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:59 smithi169 bash[19604]: cluster 2024-02-12T16:45:58.318645+0000 mgr.y (mgr.14146) 1852 : cluster [DBG] pgmap v2700: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 839 op/s; 7377/120164 objects misplaced (6.139%); 7.2 MiB/s, 115 objects/s recovering 2024-02-12T16:45:59.629 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:59 smithi169 bash[19604]: cluster 2024-02-12T16:45:58.901639+0000 osd.11 (osd.11) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:59.630 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:45:59 smithi169 bash[19604]: cluster 2024-02-12T16:45:58.901683+0000 osd.11 (osd.11) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:59.647 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:45:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:59.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:45:59.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:59.788 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:45:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:59.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:45:59.857 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:59 smithi121 bash[18489]: cluster 2024-02-12T16:45:58.318645+0000 mgr.y (mgr.14146) 1852 : cluster [DBG] pgmap v2700: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 839 op/s; 7377/120164 objects misplaced (6.139%); 7.2 MiB/s, 115 objects/s recovering 2024-02-12T16:45:59.857 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:59 smithi121 bash[18489]: cluster 2024-02-12T16:45:58.901639+0000 osd.11 (osd.11) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:59.857 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:45:59 smithi121 bash[18489]: cluster 2024-02-12T16:45:58.901683+0000 osd.11 (osd.11) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:59 smithi123 bash[19119]: cluster 2024-02-12T16:45:58.318645+0000 mgr.y (mgr.14146) 1852 : cluster [DBG] pgmap v2700: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 839 op/s; 7377/120164 objects misplaced (6.139%); 7.2 MiB/s, 115 objects/s recovering 2024-02-12T16:45:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:59 smithi123 bash[19119]: cluster 2024-02-12T16:45:58.901639+0000 osd.11 (osd.11) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:45:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:45:59 smithi123 bash[19119]: cluster 2024-02-12T16:45:58.901683+0000 osd.11 (osd.11) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:45:59.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:59.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:45:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:45:59.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:00.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:00.070 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:00.081 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:00.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:00.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:00.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:00.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:00.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:00.348 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:00.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:00.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:00.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:00.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:00.609 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:00.635 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:00.640 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:00 smithi121 bash[18489]: cluster 2024-02-12T16:45:59.553323+0000 mon.a (mon.0) 3270 : cluster [DBG] osdmap e981: 12 total, 12 up, 10 in 2024-02-12T16:46:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:00.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:00.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:00.765 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:00.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:00.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:00.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:00 smithi169 bash[19604]: cluster 2024-02-12T16:45:59.553323+0000 mon.a (mon.0) 3270 : cluster [DBG] osdmap e981: 12 total, 12 up, 10 in 2024-02-12T16:46:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:00 smithi123 bash[19119]: cluster 2024-02-12T16:45:59.553323+0000 mon.a (mon.0) 3270 : cluster [DBG] osdmap e981: 12 total, 12 up, 10 in 2024-02-12T16:46:00.898 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:00.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:01.001 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:01.029 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 9 pg_num to 26 2024-02-12T16:46:01.039 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:01.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:01.171 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:01.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:01.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:01.304 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:01.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:01.458 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:01.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:01.589 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:01.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:01.717 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:01.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:01.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:01 smithi169 bash[19604]: cluster 2024-02-12T16:46:00.319085+0000 mgr.y (mgr.14146) 1853 : cluster [DBG] pgmap v2702: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 864 op/s; 7377/116114 objects misplaced (6.353%); 7.0 MiB/s, 111 objects/s recovering 2024-02-12T16:46:01.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:01 smithi169 bash[19604]: audit 2024-02-12T16:46:00.695841+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? v1:172.21.15.121:0/1678583156' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:46:01.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:01 smithi169 bash[19604]: audit 2024-02-12T16:46:01.017252+0000 mon.a (mon.0) 3272 : audit [INF] from='client.? v1:172.21.15.121:0/1678583156' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:46:01.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:01 smithi169 bash[19604]: cluster 2024-02-12T16:46:01.017295+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e982: 12 total, 12 up, 10 in 2024-02-12T16:46:01.851 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:01 smithi123 bash[19119]: cluster 2024-02-12T16:46:00.319085+0000 mgr.y (mgr.14146) 1853 : cluster [DBG] pgmap v2702: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 864 op/s; 7377/116114 objects misplaced (6.353%); 7.0 MiB/s, 111 objects/s recovering 2024-02-12T16:46:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:01 smithi123 bash[19119]: audit 2024-02-12T16:46:00.695841+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? v1:172.21.15.121:0/1678583156' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:46:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:01 smithi123 bash[19119]: audit 2024-02-12T16:46:01.017252+0000 mon.a (mon.0) 3272 : audit [INF] from='client.? v1:172.21.15.121:0/1678583156' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:46:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:01 smithi123 bash[19119]: cluster 2024-02-12T16:46:01.017295+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e982: 12 total, 12 up, 10 in 2024-02-12T16:46:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:01 smithi121 bash[18489]: cluster 2024-02-12T16:46:00.319085+0000 mgr.y (mgr.14146) 1853 : cluster [DBG] pgmap v2702: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+recovering, 43 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 864 op/s; 7377/116114 objects misplaced (6.353%); 7.0 MiB/s, 111 objects/s recovering 2024-02-12T16:46:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:01 smithi121 bash[18489]: audit 2024-02-12T16:46:00.695841+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? v1:172.21.15.121:0/1678583156' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]: dispatch 2024-02-12T16:46:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:01 smithi121 bash[18489]: audit 2024-02-12T16:46:01.017252+0000 mon.a (mon.0) 3272 : audit [INF] from='client.? v1:172.21.15.121:0/1678583156' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-02-12T16:46:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:01 smithi121 bash[18489]: cluster 2024-02-12T16:46:01.017295+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e982: 12 total, 12 up, 10 in 2024-02-12T16:46:01.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:01.985 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:01.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:02.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:02.121 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:02.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:02.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:02.257 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:02.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:02.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:02.391 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:02.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:02.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:02.516 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:02.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:02.574 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:02 smithi123 bash[19119]: audit 2024-02-12T16:46:02.320494+0000 mon.a (mon.0) 3274 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:46:02.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:02.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:02.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:02.786 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:02 smithi121 bash[18489]: audit 2024-02-12T16:46:02.320494+0000 mon.a (mon.0) 3274 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:46:02.807 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:02.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:02.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:02.952 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:02.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:03.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:02 smithi169 bash[19604]: audit 2024-02-12T16:46:02.320494+0000 mon.a (mon.0) 3274 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-12T16:46:03.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:03.087 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:03.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:03.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:03.226 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:03.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:03.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:03.368 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:03.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:03.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:03.510 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:03.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:03.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:03.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:03 smithi123 bash[19119]: cluster 2024-02-12T16:46:02.319582+0000 mgr.y (mgr.14146) 1854 : cluster [DBG] pgmap v2704: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 45 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s; 6261/109208 objects misplaced (5.733%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T16:46:03.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:03 smithi123 bash[19119]: audit 2024-02-12T16:46:02.565235+0000 mon.a (mon.0) 3275 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:46:03.622 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:03 smithi123 bash[19119]: cluster 2024-02-12T16:46:02.565291+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e983: 12 total, 12 up, 10 in 2024-02-12T16:46:03.637 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:03.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:03.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:03.764 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:03.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:03.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:03.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:03 smithi169 bash[19604]: cluster 2024-02-12T16:46:02.319582+0000 mgr.y (mgr.14146) 1854 : cluster [DBG] pgmap v2704: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 45 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s; 6261/109208 objects misplaced (5.733%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T16:46:03.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:03 smithi169 bash[19604]: audit 2024-02-12T16:46:02.565235+0000 mon.a (mon.0) 3275 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:46:03.876 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:03 smithi169 bash[19604]: cluster 2024-02-12T16:46:02.565291+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e983: 12 total, 12 up, 10 in 2024-02-12T16:46:03.890 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:03.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:03.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:04.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:03 smithi121 bash[18489]: cluster 2024-02-12T16:46:02.319582+0000 mgr.y (mgr.14146) 1854 : cluster [DBG] pgmap v2704: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 45 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s; 6261/109208 objects misplaced (5.733%); 8.5 MiB/s, 135 objects/s recovering 2024-02-12T16:46:04.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:03 smithi121 bash[18489]: audit 2024-02-12T16:46:02.565235+0000 mon.a (mon.0) 3275 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-12T16:46:04.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:03 smithi121 bash[18489]: cluster 2024-02-12T16:46:02.565291+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e983: 12 total, 12 up, 10 in 2024-02-12T16:46:04.020 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:04.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:04.160 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:04.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:04.294 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:04.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:04.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:04.565 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:04.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:04.680 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:04 smithi123 bash[19119]: cluster 2024-02-12T16:46:03.583571+0000 osd.3 (osd.3) 120 : cluster [INF] 9.1 continuing backfill to osd.10 from (933'5200,982'15217] MIN to 982'15217 2024-02-12T16:46:04.680 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:04 smithi123 bash[19119]: cluster 2024-02-12T16:46:03.583773+0000 mon.a (mon.0) 3277 : cluster [DBG] osdmap e984: 12 total, 12 up, 10 in 2024-02-12T16:46:04.681 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:04 smithi123 bash[19119]: cluster 2024-02-12T16:46:03.586207+0000 osd.1 (osd.1) 172 : cluster [INF] 9.0 continuing backfill to osd.11 from (934'5500,982'15558] MIN to 982'15558 2024-02-12T16:46:04.697 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:04.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:04.837 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:04 smithi121 bash[18489]: cluster 2024-02-12T16:46:03.583571+0000 osd.3 (osd.3) 120 : cluster [INF] 9.1 continuing backfill to osd.10 from (933'5200,982'15217] MIN to 982'15217 2024-02-12T16:46:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:04 smithi121 bash[18489]: cluster 2024-02-12T16:46:03.583773+0000 mon.a (mon.0) 3277 : cluster [DBG] osdmap e984: 12 total, 12 up, 10 in 2024-02-12T16:46:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:04 smithi121 bash[18489]: cluster 2024-02-12T16:46:03.586207+0000 osd.1 (osd.1) 172 : cluster [INF] 9.0 continuing backfill to osd.11 from (934'5500,982'15558] MIN to 982'15558 2024-02-12T16:46:04.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:04.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:04.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:04 smithi169 bash[19604]: cluster 2024-02-12T16:46:03.583571+0000 osd.3 (osd.3) 120 : cluster [INF] 9.1 continuing backfill to osd.10 from (933'5200,982'15217] MIN to 982'15217 2024-02-12T16:46:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:04 smithi169 bash[19604]: cluster 2024-02-12T16:46:03.583773+0000 mon.a (mon.0) 3277 : cluster [DBG] osdmap e984: 12 total, 12 up, 10 in 2024-02-12T16:46:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:04 smithi169 bash[19604]: cluster 2024-02-12T16:46:03.586207+0000 osd.1 (osd.1) 172 : cluster [INF] 9.0 continuing backfill to osd.11 from (934'5500,982'15558] MIN to 982'15558 2024-02-12T16:46:05.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:05.102 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:05.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:05.206 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:05.242 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:05.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:05.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:05.365 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:05.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:05.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:05.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:05.603 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:05 smithi123 bash[19119]: cluster 2024-02-12T16:46:04.320402+0000 mgr.y (mgr.14146) 1855 : cluster [DBG] pgmap v2707: 57 pgs: 7 unknown, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 48 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 6261/105996 objects misplaced (5.907%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:46:05.603 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:05 smithi123 bash[19119]: cluster 2024-02-12T16:46:04.590668+0000 mon.a (mon.0) 3278 : cluster [DBG] osdmap e985: 12 total, 12 up, 10 in 2024-02-12T16:46:05.604 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:05.629 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:05.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:05.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:05.743 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:05 smithi121 bash[18489]: cluster 2024-02-12T16:46:04.320402+0000 mgr.y (mgr.14146) 1855 : cluster [DBG] pgmap v2707: 57 pgs: 7 unknown, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 48 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 6261/105996 objects misplaced (5.907%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:46:05.743 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:05 smithi121 bash[18489]: cluster 2024-02-12T16:46:04.590668+0000 mon.a (mon.0) 3278 : cluster [DBG] osdmap e985: 12 total, 12 up, 10 in 2024-02-12T16:46:05.765 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:05.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:05.868 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:05.874 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:05 smithi169 bash[19604]: cluster 2024-02-12T16:46:04.320402+0000 mgr.y (mgr.14146) 1855 : cluster [DBG] pgmap v2707: 57 pgs: 7 unknown, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 48 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 6261/105996 objects misplaced (5.907%); 6.2 MiB/s, 98 objects/s recovering 2024-02-12T16:46:05.875 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:05 smithi169 bash[19604]: cluster 2024-02-12T16:46:04.590668+0000 mon.a (mon.0) 3278 : cluster [DBG] osdmap e985: 12 total, 12 up, 10 in 2024-02-12T16:46:05.896 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:05.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:05.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:06.026 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:06.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:06.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:06.293 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:06.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:06.445 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:06.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:06.577 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:06 smithi121 bash[18489]: cluster 2024-02-12T16:46:05.592582+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e986: 12 total, 12 up, 10 in 2024-02-12T16:46:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:06 smithi121 bash[18489]: cluster 2024-02-12T16:46:05.595612+0000 osd.1 (osd.1) 173 : cluster [INF] 9.10 continuing backfill to osd.11 from (934'5500,982'15558] MIN to 982'15558 2024-02-12T16:46:06.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:06 smithi121 bash[18489]: cluster 2024-02-12T16:46:06.017378+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e987: 12 total, 12 up, 10 in 2024-02-12T16:46:06.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:06.706 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:06.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:06.852 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:06 smithi123 bash[19119]: cluster 2024-02-12T16:46:05.592582+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e986: 12 total, 12 up, 10 in 2024-02-12T16:46:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:06 smithi123 bash[19119]: cluster 2024-02-12T16:46:05.595612+0000 osd.1 (osd.1) 173 : cluster [INF] 9.10 continuing backfill to osd.11 from (934'5500,982'15558] MIN to 982'15558 2024-02-12T16:46:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:06 smithi123 bash[19119]: cluster 2024-02-12T16:46:06.017378+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e987: 12 total, 12 up, 10 in 2024-02-12T16:46:06.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:06.965 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:06 smithi169 bash[19604]: cluster 2024-02-12T16:46:05.592582+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e986: 12 total, 12 up, 10 in 2024-02-12T16:46:06.966 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:06 smithi169 bash[19604]: cluster 2024-02-12T16:46:05.595612+0000 osd.1 (osd.1) 173 : cluster [INF] 9.10 continuing backfill to osd.11 from (934'5500,982'15558] MIN to 982'15558 2024-02-12T16:46:06.966 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:06 smithi169 bash[19604]: cluster 2024-02-12T16:46:06.017378+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e987: 12 total, 12 up, 10 in 2024-02-12T16:46:06.986 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:06.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:07.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:07.115 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:07.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:07.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:07.259 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:07.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:07.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:07.394 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:46:07.395 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:46:07.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:07.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:07.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:07.511 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:07.534 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:07.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:07.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:07.646 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:07 smithi121 bash[18489]: cluster 2024-02-12T16:46:05.740320+0000 osd.3 (osd.3) 121 : cluster [INF] 9.11 continuing backfill to osd.10 from (933'5200,982'15217] MIN to 982'15217 2024-02-12T16:46:07.646 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:07 smithi121 bash[18489]: cluster 2024-02-12T16:46:06.320871+0000 mgr.y (mgr.14146) 1856 : cluster [DBG] pgmap v2711: 57 pgs: 5 unknown, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 50 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 6261/105324 objects misplaced (5.945%) 2024-02-12T16:46:07.646 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:07 smithi121 bash[18489]: cluster 2024-02-12T16:46:06.571037+0000 osd.7 (osd.7) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:07.646 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:07 smithi121 bash[18489]: cluster 2024-02-12T16:46:06.571100+0000 osd.7 (osd.7) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:07.646 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:07 smithi121 bash[18489]: cluster 2024-02-12T16:46:07.029095+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e988: 12 total, 12 up, 10 in 2024-02-12T16:46:07.667 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:07.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:07.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:07.800 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:07.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:07 smithi123 bash[19119]: cluster 2024-02-12T16:46:05.740320+0000 osd.3 (osd.3) 121 : cluster [INF] 9.11 continuing backfill to osd.10 from (933'5200,982'15217] MIN to 982'15217 2024-02-12T16:46:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:07 smithi123 bash[19119]: cluster 2024-02-12T16:46:06.320871+0000 mgr.y (mgr.14146) 1856 : cluster [DBG] pgmap v2711: 57 pgs: 5 unknown, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 50 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 6261/105324 objects misplaced (5.945%) 2024-02-12T16:46:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:07 smithi123 bash[19119]: cluster 2024-02-12T16:46:06.571037+0000 osd.7 (osd.7) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:07 smithi123 bash[19119]: cluster 2024-02-12T16:46:06.571100+0000 osd.7 (osd.7) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:07 smithi123 bash[19119]: cluster 2024-02-12T16:46:07.029095+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e988: 12 total, 12 up, 10 in 2024-02-12T16:46:07.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:07.938 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:07.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:08.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:07 smithi169 bash[19604]: cluster 2024-02-12T16:46:05.740320+0000 osd.3 (osd.3) 121 : cluster [INF] 9.11 continuing backfill to osd.10 from (933'5200,982'15217] MIN to 982'15217 2024-02-12T16:46:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:07 smithi169 bash[19604]: cluster 2024-02-12T16:46:06.320871+0000 mgr.y (mgr.14146) 1856 : cluster [DBG] pgmap v2711: 57 pgs: 5 unknown, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 50 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 6261/105324 objects misplaced (5.945%) 2024-02-12T16:46:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:07 smithi169 bash[19604]: cluster 2024-02-12T16:46:06.571037+0000 osd.7 (osd.7) 13 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:07 smithi169 bash[19604]: cluster 2024-02-12T16:46:06.571100+0000 osd.7 (osd.7) 14 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:07 smithi169 bash[19604]: cluster 2024-02-12T16:46:07.029095+0000 mon.a (mon.0) 3281 : cluster [DBG] osdmap e988: 12 total, 12 up, 10 in 2024-02-12T16:46:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:08.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:08.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:08.207 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:08.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:08.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:08.341 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:08.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:08.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:08.483 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:08.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:08.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:08.619 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:08.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:08.642 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 10, 8, 11] out_osds: [9, 6] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:46:08.642 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:46:08.642 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-12T16:46:08.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-12T16:46:08.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:08.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:08 smithi121 bash[18489]: cluster 2024-02-12T16:46:08.321415+0000 mgr.y (mgr.14146) 1857 : cluster [DBG] pgmap v2713: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 4641/102492 objects misplaced (4.528%); 16 MiB/s, 259 objects/s recovering 2024-02-12T16:46:08.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:08 smithi121 bash[18489]: audit 2024-02-12T16:46:08.322495+0000 mon.a (mon.0) 3282 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:46:08.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:08.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:08.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:08.874 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:08 smithi123 bash[19119]: cluster 2024-02-12T16:46:08.321415+0000 mgr.y (mgr.14146) 1857 : cluster [DBG] pgmap v2713: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 4641/102492 objects misplaced (4.528%); 16 MiB/s, 259 objects/s recovering 2024-02-12T16:46:08.874 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:08 smithi123 bash[19119]: audit 2024-02-12T16:46:08.322495+0000 mon.a (mon.0) 3282 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:46:08.892 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:08.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:08.996 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:09.018 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:08 smithi169 bash[19604]: cluster 2024-02-12T16:46:08.321415+0000 mgr.y (mgr.14146) 1857 : cluster [DBG] pgmap v2713: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 437 op/s; 4641/102492 objects misplaced (4.528%); 16 MiB/s, 259 objects/s recovering 2024-02-12T16:46:09.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:08 smithi169 bash[19604]: audit 2024-02-12T16:46:08.322495+0000 mon.a (mon.0) 3282 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]: dispatch 2024-02-12T16:46:09.121 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:09.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:09.277 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:09.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:09.419 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:09.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:09.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:09.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:09.685 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:09.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:09 smithi169 bash[19604]: audit 2024-02-12T16:46:08.681892+0000 mon.a (mon.0) 3283 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:46:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:09 smithi169 bash[19604]: cluster 2024-02-12T16:46:08.682007+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e989: 12 total, 12 up, 10 in 2024-02-12T16:46:09.821 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:09.925 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:09.954 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:09.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:10.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:10.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:09 smithi123 bash[19119]: audit 2024-02-12T16:46:08.681892+0000 mon.a (mon.0) 3283 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:46:10.070 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:09 smithi123 bash[19119]: cluster 2024-02-12T16:46:08.682007+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e989: 12 total, 12 up, 10 in 2024-02-12T16:46:10.095 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:10.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:10.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:09 smithi121 bash[18489]: audit 2024-02-12T16:46:08.681892+0000 mon.a (mon.0) 3283 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]': finished 2024-02-12T16:46:10.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:09 smithi121 bash[18489]: cluster 2024-02-12T16:46:08.682007+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e989: 12 total, 12 up, 10 in 2024-02-12T16:46:10.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:10.225 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:10.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:10.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-12T16:46:10.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:10.361 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:10.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:10.464 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:10.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:10.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:10.596 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:10.623 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:10.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:10.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:10.752 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:10.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:10 smithi169 bash[19604]: cluster 2024-02-12T16:46:09.691334+0000 mon.a (mon.0) 3285 : cluster [DBG] osdmap e990: 12 total, 12 up, 10 in 2024-02-12T16:46:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:10 smithi169 bash[19604]: cluster 2024-02-12T16:46:09.692369+0000 osd.1 (osd.1) 174 : cluster [DBG] 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 988'15689 2024-02-12T16:46:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:10 smithi169 bash[19604]: cluster 2024-02-12T16:46:09.702921+0000 osd.1 (osd.1) 175 : cluster [DBG] 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 988'15689 2024-02-12T16:46:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:10 smithi169 bash[19604]: cluster 2024-02-12T16:46:10.322088+0000 mgr.y (mgr.14146) 1858 : cluster [DBG] pgmap v2716: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 723 op/s; 4641/99088 objects misplaced (4.684%); 15 MiB/s, 241 objects/s recovering 2024-02-12T16:46:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:10 smithi169 bash[19604]: audit 2024-02-12T16:46:10.323083+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:46:10.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:10 smithi121 bash[18489]: cluster 2024-02-12T16:46:09.691334+0000 mon.a (mon.0) 3285 : cluster [DBG] osdmap e990: 12 total, 12 up, 10 in 2024-02-12T16:46:10.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:10 smithi121 bash[18489]: cluster 2024-02-12T16:46:09.692369+0000 osd.1 (osd.1) 174 : cluster [DBG] 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 988'15689 2024-02-12T16:46:10.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:10 smithi121 bash[18489]: cluster 2024-02-12T16:46:09.702921+0000 osd.1 (osd.1) 175 : cluster [DBG] 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 988'15689 2024-02-12T16:46:10.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:10 smithi121 bash[18489]: cluster 2024-02-12T16:46:10.322088+0000 mgr.y (mgr.14146) 1858 : cluster [DBG] pgmap v2716: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 723 op/s; 4641/99088 objects misplaced (4.684%); 15 MiB/s, 241 objects/s recovering 2024-02-12T16:46:10.822 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:10 smithi121 bash[18489]: audit 2024-02-12T16:46:10.323083+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:46:10.847 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:46:10.856 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:10.877 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:46:10.881 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:10.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:10.905 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:46:10.939 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:46:10.979 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:46:10.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:11.012 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:46:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:46:11.064 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:46:11.088 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:46:11.119 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:46:11.126 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:11.132 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:10 smithi123 bash[19119]: cluster 2024-02-12T16:46:09.691334+0000 mon.a (mon.0) 3285 : cluster [DBG] osdmap e990: 12 total, 12 up, 10 in 2024-02-12T16:46:11.132 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:10 smithi123 bash[19119]: cluster 2024-02-12T16:46:09.692369+0000 osd.1 (osd.1) 174 : cluster [DBG] 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 988'15689 2024-02-12T16:46:11.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:10 smithi123 bash[19119]: cluster 2024-02-12T16:46:09.702921+0000 osd.1 (osd.1) 175 : cluster [DBG] 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 988'15689 2024-02-12T16:46:11.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:10 smithi123 bash[19119]: cluster 2024-02-12T16:46:10.322088+0000 mgr.y (mgr.14146) 1858 : cluster [DBG] pgmap v2716: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 723 op/s; 4641/99088 objects misplaced (4.684%); 15 MiB/s, 241 objects/s recovering 2024-02-12T16:46:11.133 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:10 smithi123 bash[19119]: audit 2024-02-12T16:46:10.323083+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]: dispatch 2024-02-12T16:46:11.142 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:46:11.148 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:11.168 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:46:11.169 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.170 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:46:11.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:11.275 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:11.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:11.415 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:11.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:11.549 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:11.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:11.692 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:11.717 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:46:11.778 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:11 smithi121 bash[18489]: audit 2024-02-12T16:46:10.701966+0000 mon.a (mon.0) 3287 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:46:11.778 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:11 smithi121 bash[18489]: cluster 2024-02-12T16:46:10.702026+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e991: 12 total, 12 up, 10 in 2024-02-12T16:46:11.779 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:11 smithi121 bash[18489]: audit 2024-02-12T16:46:10.808858+0000 mon.b (mon.2) 229 : audit [DBG] from='client.? v1:172.21.15.121:0/579957375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:11.779 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:11 smithi121 bash[18489]: audit 2024-02-12T16:46:11.317282+0000 mon.a (mon.0) 3289 : audit [INF] from='client.27212 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:46:11.779 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:11 smithi121 bash[18489]: audit 2024-02-12T16:46:11.319647+0000 mon.b (mon.2) 230 : audit [INF] from='client.? v1:172.21.15.121:0/1536780300' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:46:11.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:11.827 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:11.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:11.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:11 smithi169 bash[19604]: audit 2024-02-12T16:46:10.701966+0000 mon.a (mon.0) 3287 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:46:11.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:11 smithi169 bash[19604]: cluster 2024-02-12T16:46:10.702026+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e991: 12 total, 12 up, 10 in 2024-02-12T16:46:11.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:11 smithi169 bash[19604]: audit 2024-02-12T16:46:10.808858+0000 mon.b (mon.2) 229 : audit [DBG] from='client.? v1:172.21.15.121:0/579957375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:11.939 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:11 smithi169 bash[19604]: audit 2024-02-12T16:46:11.317282+0000 mon.a (mon.0) 3289 : audit [INF] from='client.27212 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:46:11.940 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:11 smithi169 bash[19604]: audit 2024-02-12T16:46:11.319647+0000 mon.b (mon.2) 230 : audit [INF] from='client.? v1:172.21.15.121:0/1536780300' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:46:11.958 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:11.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:12.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:12.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:11 smithi123 bash[19119]: audit 2024-02-12T16:46:10.701966+0000 mon.a (mon.0) 3287 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]': finished 2024-02-12T16:46:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:11 smithi123 bash[19119]: cluster 2024-02-12T16:46:10.702026+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e991: 12 total, 12 up, 10 in 2024-02-12T16:46:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:11 smithi123 bash[19119]: audit 2024-02-12T16:46:10.808858+0000 mon.b (mon.2) 229 : audit [DBG] from='client.? v1:172.21.15.121:0/579957375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:11 smithi123 bash[19119]: audit 2024-02-12T16:46:11.317282+0000 mon.a (mon.0) 3289 : audit [INF] from='client.27212 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:46:12.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:11 smithi123 bash[19119]: audit 2024-02-12T16:46:11.319647+0000 mon.b (mon.2) 230 : audit [INF] from='client.? v1:172.21.15.121:0/1536780300' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:46:12.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:12.237 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:12.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:12.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:12.376 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:12.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:12.482 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:12.508 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:12.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:12.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:12.653 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:12.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:12.756 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:12.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:12 smithi169 bash[19604]: cluster 2024-02-12T16:46:11.697271+0000 mon.a (mon.0) 3290 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:46:12.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:12 smithi169 bash[19604]: cluster 2024-02-12T16:46:11.704364+0000 osd.3 (osd.3) 122 : cluster [DBG] 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 989'15383 2024-02-12T16:46:12.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:12 smithi169 bash[19604]: audit 2024-02-12T16:46:11.704481+0000 mon.a (mon.0) 3291 : audit [INF] from='client.27212 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:46:12.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:12 smithi169 bash[19604]: cluster 2024-02-12T16:46:11.704566+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e992: 12 total, 12 up, 10 in 2024-02-12T16:46:12.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:12 smithi169 bash[19604]: cluster 2024-02-12T16:46:11.720051+0000 osd.3 (osd.3) 123 : cluster [DBG] 9.11 starting backfill to osd.7 from (0'0,0'0] MAX to 989'15383 2024-02-12T16:46:12.764 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:12 smithi169 bash[19604]: cluster 2024-02-12T16:46:12.322769+0000 mgr.y (mgr.14146) 1859 : cluster [DBG] pgmap v2719: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 997 op/s; 6722/94510 objects misplaced (7.112%); 29 MiB/s, 465 objects/s recovering 2024-02-12T16:46:12.783 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:12.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:12.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:12 smithi123 bash[19119]: cluster 2024-02-12T16:46:11.697271+0000 mon.a (mon.0) 3290 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:46:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:12 smithi123 bash[19119]: cluster 2024-02-12T16:46:11.704364+0000 osd.3 (osd.3) 122 : cluster [DBG] 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 989'15383 2024-02-12T16:46:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:12 smithi123 bash[19119]: audit 2024-02-12T16:46:11.704481+0000 mon.a (mon.0) 3291 : audit [INF] from='client.27212 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:46:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:12 smithi123 bash[19119]: cluster 2024-02-12T16:46:11.704566+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e992: 12 total, 12 up, 10 in 2024-02-12T16:46:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:12 smithi123 bash[19119]: cluster 2024-02-12T16:46:11.720051+0000 osd.3 (osd.3) 123 : cluster [DBG] 9.11 starting backfill to osd.7 from (0'0,0'0] MAX to 989'15383 2024-02-12T16:46:12.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:12 smithi123 bash[19119]: cluster 2024-02-12T16:46:12.322769+0000 mgr.y (mgr.14146) 1859 : cluster [DBG] pgmap v2719: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 997 op/s; 6722/94510 objects misplaced (7.112%); 29 MiB/s, 465 objects/s recovering 2024-02-12T16:46:12.925 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:12.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:13.031 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:13.049 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:12 smithi121 bash[18489]: cluster 2024-02-12T16:46:11.697271+0000 mon.a (mon.0) 3290 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:46:13.049 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:12 smithi121 bash[18489]: cluster 2024-02-12T16:46:11.704364+0000 osd.3 (osd.3) 122 : cluster [DBG] 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 989'15383 2024-02-12T16:46:13.049 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:12 smithi121 bash[18489]: audit 2024-02-12T16:46:11.704481+0000 mon.a (mon.0) 3291 : audit [INF] from='client.27212 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:46:13.049 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:12 smithi121 bash[18489]: cluster 2024-02-12T16:46:11.704566+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e992: 12 total, 12 up, 10 in 2024-02-12T16:46:13.049 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:12 smithi121 bash[18489]: cluster 2024-02-12T16:46:11.720051+0000 osd.3 (osd.3) 123 : cluster [DBG] 9.11 starting backfill to osd.7 from (0'0,0'0] MAX to 989'15383 2024-02-12T16:46:13.049 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:12 smithi121 bash[18489]: cluster 2024-02-12T16:46:12.322769+0000 mgr.y (mgr.14146) 1859 : cluster [DBG] pgmap v2719: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 997 op/s; 6722/94510 objects misplaced (7.112%); 29 MiB/s, 465 objects/s recovering 2024-02-12T16:46:13.055 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:13.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:13.161 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:13.186 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:13.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:13.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:13.318 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:13.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:13.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:13.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:13.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:13.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:13.603 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:13.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:13.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:13.724 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:13 smithi121 bash[18489]: cluster 2024-02-12T16:46:12.980643+0000 osd.2 (osd.2) 145 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:46:13.724 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:13 smithi169 bash[19604]: cluster 2024-02-12T16:46:12.980643+0000 osd.2 (osd.2) 145 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:46:13.724 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:13 smithi169 bash[19604]: cluster 2024-02-12T16:46:13.097327+0000 osd.2 (osd.2) 146 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:46:13.742 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:13.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:13.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:13.871 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:13.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:13.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:13.986 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:13 smithi121 bash[18489]: cluster 2024-02-12T16:46:13.097327+0000 osd.2 (osd.2) 146 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:46:14.006 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:14.113 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:13 smithi123 bash[19119]: cluster 2024-02-12T16:46:12.980643+0000 osd.2 (osd.2) 145 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:46:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:13 smithi123 bash[19119]: cluster 2024-02-12T16:46:13.097327+0000 osd.2 (osd.2) 146 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:46:14.143 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:14.250 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:14.277 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:14.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:14.413 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:14.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:14.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:14.653 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:14.682 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:14.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:14.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:46:14.818 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:14.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:14.961 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:14.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:14 smithi169 bash[19604]: cluster 2024-02-12T16:46:13.727736+0000 osd.10 (osd.10) 240 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:14 smithi169 bash[19604]: cluster 2024-02-12T16:46:13.727780+0000 osd.10 (osd.10) 241 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:14 smithi169 bash[19604]: cluster 2024-02-12T16:46:14.323317+0000 mgr.y (mgr.14146) 1860 : cluster [DBG] pgmap v2720: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 6722/91662 objects misplaced (7.333%); 21 MiB/s, 330 objects/s recovering 2024-02-12T16:46:15.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:46:15.057 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:14 smithi121 bash[18489]: cluster 2024-02-12T16:46:13.727736+0000 osd.10 (osd.10) 240 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:15.057 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:14 smithi121 bash[18489]: cluster 2024-02-12T16:46:13.727780+0000 osd.10 (osd.10) 241 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:15.057 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:14 smithi121 bash[18489]: cluster 2024-02-12T16:46:14.323317+0000 mgr.y (mgr.14146) 1860 : cluster [DBG] pgmap v2720: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 6722/91662 objects misplaced (7.333%); 21 MiB/s, 330 objects/s recovering 2024-02-12T16:46:15.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:15.074 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:14 smithi123 bash[19119]: cluster 2024-02-12T16:46:13.727736+0000 osd.10 (osd.10) 240 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:15.074 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:14 smithi123 bash[19119]: cluster 2024-02-12T16:46:13.727780+0000 osd.10 (osd.10) 241 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:15.074 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:14 smithi123 bash[19119]: cluster 2024-02-12T16:46:14.323317+0000 mgr.y (mgr.14146) 1860 : cluster [DBG] pgmap v2720: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 959 op/s; 6722/91662 objects misplaced (7.333%); 21 MiB/s, 330 objects/s recovering 2024-02-12T16:46:15.090 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:15.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:15.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:15.247 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:15.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:15.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:15.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:15.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:15.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:15.517 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:15.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:15.542 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:15.621 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:15.649 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:15.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:15.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:15.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:15.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:15 smithi123 bash[19119]: cluster 2024-02-12T16:46:14.731091+0000 mon.a (mon.0) 3293 : cluster [DBG] osdmap e993: 12 total, 12 up, 10 in 2024-02-12T16:46:15.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:15 smithi123 bash[19119]: cluster 2024-02-12T16:46:14.900491+0000 osd.6 (osd.6) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:15.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:15 smithi123 bash[19119]: cluster 2024-02-12T16:46:14.900536+0000 osd.6 (osd.6) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:15.918 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:15.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:16.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:16.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:15 smithi121 bash[18489]: cluster 2024-02-12T16:46:14.731091+0000 mon.a (mon.0) 3293 : cluster [DBG] osdmap e993: 12 total, 12 up, 10 in 2024-02-12T16:46:16.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:15 smithi121 bash[18489]: cluster 2024-02-12T16:46:14.900491+0000 osd.6 (osd.6) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:16.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:15 smithi121 bash[18489]: cluster 2024-02-12T16:46:14.900536+0000 osd.6 (osd.6) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:15 smithi169 bash[19604]: cluster 2024-02-12T16:46:14.731091+0000 mon.a (mon.0) 3293 : cluster [DBG] osdmap e993: 12 total, 12 up, 10 in 2024-02-12T16:46:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:15 smithi169 bash[19604]: cluster 2024-02-12T16:46:14.900491+0000 osd.6 (osd.6) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:15 smithi169 bash[19604]: cluster 2024-02-12T16:46:14.900536+0000 osd.6 (osd.6) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:16.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:16.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:16.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:16.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:16.312 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:16.343 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:16.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:16.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:16.474 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:16.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:16.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:16.610 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:16.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:16.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:16.744 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:16.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:16.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:16.861 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:16 smithi169 bash[19604]: cluster 2024-02-12T16:46:15.733642+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e994: 12 total, 12 up, 10 in 2024-02-12T16:46:16.862 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:16 smithi169 bash[19604]: cluster 2024-02-12T16:46:16.019073+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e995: 12 total, 12 up, 10 in 2024-02-12T16:46:16.862 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:16 smithi169 bash[19604]: cluster 2024-02-12T16:46:16.323884+0000 mgr.y (mgr.14146) 1861 : cluster [DBG] pgmap v2724: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 844 op/s; 6722/90122 objects misplaced (7.459%); 25 MiB/s, 402 objects/s recovering 2024-02-12T16:46:16.885 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:16.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:16.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:16.995 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:16 smithi121 bash[18489]: cluster 2024-02-12T16:46:15.733642+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e994: 12 total, 12 up, 10 in 2024-02-12T16:46:16.995 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:16 smithi121 bash[18489]: cluster 2024-02-12T16:46:16.019073+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e995: 12 total, 12 up, 10 in 2024-02-12T16:46:16.995 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:16 smithi121 bash[18489]: cluster 2024-02-12T16:46:16.323884+0000 mgr.y (mgr.14146) 1861 : cluster [DBG] pgmap v2724: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 844 op/s; 6722/90122 objects misplaced (7.459%); 25 MiB/s, 402 objects/s recovering 2024-02-12T16:46:17.012 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:17.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:17.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:16 smithi123 bash[19119]: cluster 2024-02-12T16:46:15.733642+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e994: 12 total, 12 up, 10 in 2024-02-12T16:46:17.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:16 smithi123 bash[19119]: cluster 2024-02-12T16:46:16.019073+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e995: 12 total, 12 up, 10 in 2024-02-12T16:46:17.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:16 smithi123 bash[19119]: cluster 2024-02-12T16:46:16.323884+0000 mgr.y (mgr.14146) 1861 : cluster [DBG] pgmap v2724: 57 pgs: 1 active+remapped+backfill_toofull, 1 activating+remapped, 1 active+remapped+backfilling, 54 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 844 op/s; 6722/90122 objects misplaced (7.459%); 25 MiB/s, 402 objects/s recovering 2024-02-12T16:46:17.145 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:17.248 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:17.276 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:17.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:17.408 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:17.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:17.553 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:17.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:17.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:17.840 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:17.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:17.975 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:17.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:18.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:18.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:18 smithi121 bash[18489]: cluster 2024-02-12T16:46:17.503203+0000 osd.0 (osd.0) 163 : cluster [DBG] 1.c scrub starts 2024-02-12T16:46:18.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:18 smithi121 bash[18489]: cluster 2024-02-12T16:46:17.567855+0000 osd.0 (osd.0) 164 : cluster [DBG] 1.c scrub ok 2024-02-12T16:46:18.113 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:18.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:18.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:18.244 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:18.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:18 smithi169 bash[19604]: cluster 2024-02-12T16:46:17.503203+0000 osd.0 (osd.0) 163 : cluster [DBG] 1.c scrub starts 2024-02-12T16:46:18.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:18 smithi169 bash[19604]: cluster 2024-02-12T16:46:17.567855+0000 osd.0 (osd.0) 164 : cluster [DBG] 1.c scrub ok 2024-02-12T16:46:18.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:18.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:18 smithi123 bash[19119]: cluster 2024-02-12T16:46:17.503203+0000 osd.0 (osd.0) 163 : cluster [DBG] 1.c scrub starts 2024-02-12T16:46:18.357 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:18 smithi123 bash[19119]: cluster 2024-02-12T16:46:17.567855+0000 osd.0 (osd.0) 164 : cluster [DBG] 1.c scrub ok 2024-02-12T16:46:18.378 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:18.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:18.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:18.512 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:18.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:18.615 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:18.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:18.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:18.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:18.775 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:18.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:18.882 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:18.906 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:18.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:19.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:19.051 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:19.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:19.158 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:19.181 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:19.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:19.285 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:19.292 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:19 smithi123 bash[19119]: cluster 2024-02-12T16:46:18.324698+0000 mgr.y (mgr.14146) 1862 : cluster [DBG] pgmap v2725: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 988 op/s; 3935/82648 objects misplaced (4.761%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T16:46:19.292 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:19 smithi123 bash[19119]: audit 2024-02-12T16:46:18.325665+0000 mon.a (mon.0) 3296 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:46:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:19 smithi169 bash[19604]: cluster 2024-02-12T16:46:18.324698+0000 mgr.y (mgr.14146) 1862 : cluster [DBG] pgmap v2725: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 988 op/s; 3935/82648 objects misplaced (4.761%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T16:46:19.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:19 smithi169 bash[19604]: audit 2024-02-12T16:46:18.325665+0000 mon.a (mon.0) 3296 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:46:19.305 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:19.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:19.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:19.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:19 smithi121 bash[18489]: cluster 2024-02-12T16:46:18.324698+0000 mgr.y (mgr.14146) 1862 : cluster [DBG] pgmap v2725: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 988 op/s; 3935/82648 objects misplaced (4.761%); 9.5 MiB/s, 151 objects/s recovering 2024-02-12T16:46:19.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:19 smithi121 bash[18489]: audit 2024-02-12T16:46:18.325665+0000 mon.a (mon.0) 3296 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]: dispatch 2024-02-12T16:46:19.443 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:19.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:19.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:19.579 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:19.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:19.683 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:19.720 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:19.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:19.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:19.856 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:19.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:19.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:19.999 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:20.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:20.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:20 smithi121 bash[18489]: audit 2024-02-12T16:46:19.038119+0000 mon.a (mon.0) 3297 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:46:20.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:20 smithi121 bash[18489]: cluster 2024-02-12T16:46:19.038212+0000 mon.a (mon.0) 3298 : cluster [DBG] osdmap e996: 12 total, 12 up, 10 in 2024-02-12T16:46:20.112 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:20 smithi121 bash[18489]: cluster 2024-02-12T16:46:19.134053+0000 osd.5 (osd.5) 221 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:46:20.113 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:20 smithi121 bash[18489]: cluster 2024-02-12T16:46:19.207600+0000 osd.5 (osd.5) 222 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:46:20.136 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:20.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:20.266 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:20.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:20.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:20 smithi169 bash[19604]: audit 2024-02-12T16:46:19.038119+0000 mon.a (mon.0) 3297 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:46:20.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:20 smithi169 bash[19604]: cluster 2024-02-12T16:46:19.038212+0000 mon.a (mon.0) 3298 : cluster [DBG] osdmap e996: 12 total, 12 up, 10 in 2024-02-12T16:46:20.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:20 smithi169 bash[19604]: cluster 2024-02-12T16:46:19.134053+0000 osd.5 (osd.5) 221 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:46:20.377 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:20 smithi169 bash[19604]: cluster 2024-02-12T16:46:19.207600+0000 osd.5 (osd.5) 222 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:46:20.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:20 smithi123 bash[19119]: audit 2024-02-12T16:46:19.038119+0000 mon.a (mon.0) 3297 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]': finished 2024-02-12T16:46:20.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:20 smithi123 bash[19119]: cluster 2024-02-12T16:46:19.038212+0000 mon.a (mon.0) 3298 : cluster [DBG] osdmap e996: 12 total, 12 up, 10 in 2024-02-12T16:46:20.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:20 smithi123 bash[19119]: cluster 2024-02-12T16:46:19.134053+0000 osd.5 (osd.5) 221 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:46:20.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:20 smithi123 bash[19119]: cluster 2024-02-12T16:46:19.207600+0000 osd.5 (osd.5) 222 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:46:20.395 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:20.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:20.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:20.632 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:20.668 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:20.772 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:20.812 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:20.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:20.948 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:20.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:21.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:21.082 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:21.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:21.179 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:21.184 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:21.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:21.197 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:21 smithi169 bash[19604]: cluster 2024-02-12T16:46:20.043093+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e997: 12 total, 12 up, 10 in 2024-02-12T16:46:21.197 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:21 smithi169 bash[19604]: cluster 2024-02-12T16:46:20.325228+0000 mgr.y (mgr.14146) 1863 : cluster [DBG] pgmap v2728: 57 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 54 active+clean; 2.3 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.46k op/s; 3163/76724 objects misplaced (4.123%); 18 MiB/s, 282 objects/s recovering 2024-02-12T16:46:21.215 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:21.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:21.319 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:21.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:21 smithi123 bash[19119]: cluster 2024-02-12T16:46:20.043093+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e997: 12 total, 12 up, 10 in 2024-02-12T16:46:21.331 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:21 smithi123 bash[19119]: cluster 2024-02-12T16:46:20.325228+0000 mgr.y (mgr.14146) 1863 : cluster [DBG] pgmap v2728: 57 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 54 active+clean; 2.3 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.46k op/s; 3163/76724 objects misplaced (4.123%); 18 MiB/s, 282 objects/s recovering 2024-02-12T16:46:21.354 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:21.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:21 smithi121 bash[18489]: cluster 2024-02-12T16:46:20.043093+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e997: 12 total, 12 up, 10 in 2024-02-12T16:46:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:21 smithi121 bash[18489]: cluster 2024-02-12T16:46:20.325228+0000 mgr.y (mgr.14146) 1863 : cluster [DBG] pgmap v2728: 57 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 54 active+clean; 2.3 GiB data, 13 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.46k op/s; 3163/76724 objects misplaced (4.123%); 18 MiB/s, 282 objects/s recovering 2024-02-12T16:46:21.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:21.501 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:21.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:21.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:21.627 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:21.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:21.730 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:21.755 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:21.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:21.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:21.898 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:21.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:22.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:22.031 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:22.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:22 smithi123 bash[19119]: cluster 2024-02-12T16:46:21.047755+0000 mon.a (mon.0) 3300 : cluster [DBG] osdmap e998: 12 total, 12 up, 10 in 2024-02-12T16:46:22.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:22 smithi123 bash[19119]: cluster 2024-02-12T16:46:21.049305+0000 osd.1 (osd.1) 176 : cluster [DBG] 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 995'16049 2024-02-12T16:46:22.159 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:22.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:22.296 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:22.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:22.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:22 smithi121 bash[18489]: cluster 2024-02-12T16:46:21.047755+0000 mon.a (mon.0) 3300 : cluster [DBG] osdmap e998: 12 total, 12 up, 10 in 2024-02-12T16:46:22.408 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:22 smithi121 bash[18489]: cluster 2024-02-12T16:46:21.049305+0000 osd.1 (osd.1) 176 : cluster [DBG] 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 995'16049 2024-02-12T16:46:22.427 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:22.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:22 smithi169 bash[19604]: cluster 2024-02-12T16:46:21.047755+0000 mon.a (mon.0) 3300 : cluster [DBG] osdmap e998: 12 total, 12 up, 10 in 2024-02-12T16:46:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:22 smithi169 bash[19604]: cluster 2024-02-12T16:46:21.049305+0000 osd.1 (osd.1) 176 : cluster [DBG] 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 995'16049 2024-02-12T16:46:22.577 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:22.684 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:22.711 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:22.817 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:22.843 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:22.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:22.973 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:22.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:23.072 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:23 smithi123 bash[19119]: cluster 2024-02-12T16:46:22.325717+0000 mgr.y (mgr.14146) 1864 : cluster [DBG] pgmap v2730: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.59k op/s; 3852/71004 objects misplaced (5.425%); 14 MiB/s, 226 objects/s recovering 2024-02-12T16:46:23.072 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:23 smithi123 bash[19119]: cluster 2024-02-12T16:46:22.774209+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:23.072 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:23 smithi123 bash[19119]: cluster 2024-02-12T16:46:22.774261+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:23.077 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:23.084 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:23 smithi169 bash[19604]: cluster 2024-02-12T16:46:22.325717+0000 mgr.y (mgr.14146) 1864 : cluster [DBG] pgmap v2730: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.59k op/s; 3852/71004 objects misplaced (5.425%); 14 MiB/s, 226 objects/s recovering 2024-02-12T16:46:23.084 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:23 smithi169 bash[19604]: cluster 2024-02-12T16:46:22.774209+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:23.084 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:23 smithi169 bash[19604]: cluster 2024-02-12T16:46:22.774261+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:23.106 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:23.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:23.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:23.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:23.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:23.373 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:23.373 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:23 smithi121 bash[18489]: cluster 2024-02-12T16:46:22.325717+0000 mgr.y (mgr.14146) 1864 : cluster [DBG] pgmap v2730: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 1 peering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.59k op/s; 3852/71004 objects misplaced (5.425%); 14 MiB/s, 226 objects/s recovering 2024-02-12T16:46:23.373 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:23 smithi121 bash[18489]: cluster 2024-02-12T16:46:22.774209+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:23.373 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:23 smithi121 bash[18489]: cluster 2024-02-12T16:46:22.774261+0000 osd.0 (osd.0) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:23.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:23.482 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:23.507 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:23.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:23.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:23.651 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:23.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:23.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:23.788 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:23.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:23.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:23.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:23.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:24.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:24.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:24.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:24.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:24.207 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:24.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:24.226 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:46:24.226 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:46:24.226 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 999, 2024-02-12T16:46:24.226 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "eb2349b2-c9bd-11ee-95b9-87774f69a715", 2024-02-12T16:46:24.226 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-02-12T15:47:18.273813+0000", 2024-02-12T16:46:24.226 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-02-12T16:46:23.058122+0000", 2024-02-12T16:46:24.226 INFO:teuthology.orchestra.run.smithi121.stdout: "last_up_change": "2024-02-12T16:42:40.138095+0000", 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "last_in_change": "2024-02-12T16:44:59.017069+0000", 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_num": 5804032, 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_set": [ 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "nodeep-scrub", 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "pglog_hardlimit", 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "purged_snapdirs", 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_deletes", 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "sortbitwise" 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_version": 39, 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "full_ratio": 0.94999998807907104, 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-12T16:46:24.227 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_snapshot": "", 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_max": 9, 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd": 12, 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "require_min_compat_client": "luminous", 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "min_compat_client": "luminous", 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "require_osd_release": "pacific", 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "pools": [ 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 1, 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "device_health_metrics", 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T15:51:35.886878+0000", 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:46:24.228 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 31, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 31, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 31, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 31, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 31, 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:46:24.229 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "957", 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "663", 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:46:24.230 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:46:24.231 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:46:24.232 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "options": { 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_max": 32, 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_min": 1 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_devicehealth": {} 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "pool": 9, 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": "unique_pool_7", 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "create_time": "2024-02-12T16:42:20.926635+0000", 2024-02-12T16:46:24.233 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flags_names": "hashpspool", 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type": 1, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 2, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": 2, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_rule": 0, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_count": 0, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_target": 0, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_barrier": 0, 2024-02-12T16:46:24.234 INFO:teuthology.orchestra.run.smithi121.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-12T16:46:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "object_hash": 2, 2024-02-12T16:46:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscale_mode": "off", 2024-02-12T16:46:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": 26, 2024-02-12T16:46:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num": 19, 2024-02-12T16:46:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_placement_num_target": 26, 2024-02-12T16:46:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_target": 26, 2024-02-12T16:46:24.235 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num_pending": 26, 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "last_pg_merge_meta": { 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "source_pgid": "0.0", 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "ready_epoch": 0, 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_started": 0, 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "last_epoch_clean": 0, 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "source_version": "0'0", 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "target_version": "0'0" 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "last_change": "996", 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend": "0", 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "last_force_op_resend_preluminous": "983", 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "auid": 0, 2024-02-12T16:46:24.236 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_mode": "selfmanaged", 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_seq": 0, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_epoch": 0, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_snaps": [], 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps": "[]", 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_bytes": 0, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "quota_max_objects": 0, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tiers": [], 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tier_of": -1, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "read_tier": -1, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "write_tier": -1, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_mode": "none", 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_bytes": 0, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "target_max_objects": 0, 2024-02-12T16:46:24.237 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_flush_age": 0, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "cache_min_evict_age": 0, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profile": "", 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_params": { 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "none" 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_period": 0, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_count": 0, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "use_gmt_hitset": true, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "min_read_recency_for_promote": 0, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "min_write_recency_for_promote": 0, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_grade_decay_rate": 0, 2024-02-12T16:46:24.238 INFO:teuthology.orchestra.run.smithi121.stdout: "hit_set_search_last_n": 0, 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "grade_table": [], 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "stripe_width": 0, 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "expected_num_objects": 0, 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "fast_read": false, 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "options": {}, 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "application_metadata": { 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "rados": {} 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "3a3badac-97ea-4ee1-ad82-1b67a4fe94f0", 2024-02-12T16:46:24.239 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.66845703125, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 8, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 964, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6802", 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.241 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6804", 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6803", 2024-02-12T16:46:24.242 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1815232143 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6801/1815232143", 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6802/1815232143", 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6804/1815232143", 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6803/1815232143", 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.243 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "c6140309-5703-4538-9739-276fbbdaa880", 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 0.9075927734375, 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.244 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 13, 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 997, 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6805", 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.247 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6806", 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6808", 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:46:24.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6807", 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2968433670 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6805/2968433670", 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6806/2968433670", 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6808/2968433670", 2024-02-12T16:46:24.249 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6807/2968433670", 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "7c81e1a4-cae0-48d4-98ab-0becb6b31170", 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.81610107421875, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.250 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 21, 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 889, 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6809", 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.251 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6810", 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6812", 2024-02-12T16:46:24.252 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6811", 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2158187852 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6809/2158187852", 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6810/2158187852", 2024-02-12T16:46:24.253 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6812/2158187852", 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6811/2158187852", 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "5f824526-596d-42e1-8b7e-cd8676ded6dd", 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0.5543212890625, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 26, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 414, 2024-02-12T16:46:24.254 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 522, 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 995, 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 415, 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6813", 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.255 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6814", 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6816", 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:46:24.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6815", 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551349828 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6813/2551349828", 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.121:6814/2551349828", 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.121:6816/2551349828", 2024-02-12T16:46:24.257 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.121:6815/2551349828", 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "14c2f878-99ce-4b51-8d9c-131c1d883b31", 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.258 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 32, 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 964, 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6800", 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.259 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6801", 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6803", 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6802", 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447700123 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6800/3447700123", 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6801/3447700123", 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6803/3447700123", 2024-02-12T16:46:24.261 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6802/3447700123", 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "75ecf26a-d7ed-46e7-a40e-6f719e2c3602", 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.262 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 39, 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 996, 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6804", 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.263 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6805", 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6807", 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.264 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6806", 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 961317306 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6804/961317306", 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6805/961317306", 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6807/961317306", 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6806/961317306", 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.265 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "71e715e6-eac1-4386-927e-51df0cc419c0", 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 0, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 48, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 935, 2024-02-12T16:46:24.266 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6808", 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.267 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6809", 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6811", 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.268 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6810", 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1377261784 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6808/1377261784", 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6809/1377261784", 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6811/1377261784", 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6810/1377261784", 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.269 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "b419b04f-0c0c-495a-aa9b-25c334159515", 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 56, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 848, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 923, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 996, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 849, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.270 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6812", 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 4284140534 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6813", 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 4284140534 2024-02-12T16:46:24.271 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6815", 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 4284140534 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.123:6814", 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 4284140534 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.123:6812/4284140534", 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.123:6813/4284140534", 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.123:6815/4284140534", 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.123:6814/4284140534", 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "1591e7bc-5b86-4401-803d-5325062ffb75", 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 65, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 983, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.274 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6800", 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6801", 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:46:24.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6803", 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6802", 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2551075225 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6800/2551075225", 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6801/2551075225", 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6803/2551075225", 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6802/2551075225", 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "522d61df-8dc1-4915-83fd-f94d0d572c15", 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 0, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 74, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 916, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.278 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6804", 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6805", 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6807", 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6806", 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2339555071 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6804/2339555071", 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6805/2339555071", 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6807/2339555071", 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6806/2339555071", 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "91f74dca-503a-45d1-9647-df4804d58515", 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 83, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 983, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.282 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6808", 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6809", 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:46:24.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6811", 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.284 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6810", 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 391306610 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6808/391306610", 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6809/391306610", 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6811/391306610", 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6810/391306610", 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.285 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "uuid": "788b4f47-eadf-4601-a5b4-51fb6f04fbdf", 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "up": 1, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "in": 1, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 1, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_affinity": 1, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_begin": 0, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "last_clean_end": 0, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "up_from": 93, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "up_thru": 983, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "down_at": 0, 2024-02-12T16:46:24.286 INFO:teuthology.orchestra.run.smithi121.stdout: "lost_at": 0, 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6812", 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addrs": { 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.287 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6813", 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addrs": { 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6815", 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.288 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addrs": { 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.169:6814", 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3696812305 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.169:6812/3696812305", 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_addr": "172.21.15.169:6813/3696812305", 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3696812305", 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "heartbeat_front_addr": "172.21.15.169:6814/3696812305", 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "state": [ 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "exists", 2024-02-12T16:46:24.289 INFO:teuthology.orchestra.run.smithi121.stdout: "up" 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_xinfo": [ 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 0, 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:46:22.774266+0000", 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 1, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:55.921494+0000", 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 2, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.291 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:55.380611+0000", 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 3, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:24.534653+0000", 2024-02-12T16:46:24.292 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 414 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 4, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:51.791648+0000", 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 5, 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.293 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:50.737919+0000", 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 6, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 65536, 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:46:14.900537+0000", 2024-02-12T16:46:24.294 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 7, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:46:06.571103+0000", 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 848 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 8, 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.295 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:17.117070+0000", 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 9, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 59709, 2024-02-12T16:46:24.296 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:46.061144+0000", 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 10, 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:46:13.727782+0000", 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.297 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": 11, 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "down_stamp": "0.000000", 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_probability": 0, 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "laggy_interval": 0, 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "features": 4540138314316775423, 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "old_weight": 0, 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "last_purged_snaps_scrub": "2024-02-12T16:45:58.901685+0000", 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "dead_epoch": 0 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap": [ 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "1.1b", 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: 10, 2024-02-12T16:46:24.298 INFO:teuthology.orchestra.run.smithi121.stdout: 5 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_upmap_items": [], 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_temp": [ 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "9.0", 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: 6 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "9.10", 2024-02-12T16:46:24.299 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: 6 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: "pgid": "9.12", 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: "osds": [ 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: 11 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: "primary_temp": [], 2024-02-12T16:46:24.300 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklist": { 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/2474849789": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/253689008": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/3531151725": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/1778611055": "2024-02-13T15:47:47.232761+0000", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:6800/28718860": "2024-02-13T15:48:17.261083+0000", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "172.21.15.121:0/445290372": "2024-02-13T15:47:47.232761+0000" 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "range_blocklist": {}, 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "erasure_code_profiles": { 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "default": { 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-failure-domain": "osd", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "k": "2", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "m": "1", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "plugin": "jerasure", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "ruleset-failure-domain": "osd", 2024-02-12T16:46:24.301 INFO:teuthology.orchestra.run.smithi121.stdout: "technique": "reed_sol_van" 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_snaps_queue": [], 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "new_removed_snaps": [], 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "new_purged_snaps": [], 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_node_flags": {}, 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "device_class_flags": {}, 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": { 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_enabled": false, 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_bucket_count": 0, 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "degraded_stretch_mode": 0, 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "recovering_stretch_mode": 0, 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode_bucket": 0 2024-02-12T16:46:24.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:24.303 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:46:24.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:24.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:24 smithi121 bash[18489]: cluster 2024-02-12T16:46:23.070392+0000 mon.a (mon.0) 3301 : cluster [DBG] osdmap e999: 12 total, 12 up, 10 in 2024-02-12T16:46:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:24.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:24.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:24 smithi123 bash[19119]: cluster 2024-02-12T16:46:23.070392+0000 mon.a (mon.0) 3301 : cluster [DBG] osdmap e999: 12 total, 12 up, 10 in 2024-02-12T16:46:24.461 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:46:24.461 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:46:24.461 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:46:24.461 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:46:24.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:24.502 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:24.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:24 smithi169 bash[19604]: cluster 2024-02-12T16:46:23.070392+0000 mon.a (mon.0) 3301 : cluster [DBG] osdmap e999: 12 total, 12 up, 10 in 2024-02-12T16:46:24.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:24.630 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:24.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:24.733 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:24.769 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:24.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:24.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:24.901 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:24.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:25.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:25.037 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:25.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:25 smithi121 bash[18489]: audit 2024-02-12T16:46:24.221249+0000 mon.b (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.121:0/3204700742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:46:25.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:25 smithi121 bash[18489]: cluster 2024-02-12T16:46:24.326316+0000 mgr.y (mgr.14146) 1865 : cluster [DBG] pgmap v2732: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 3852/67470 objects misplaced (5.709%); 5.3 MiB/s, 84 objects/s recovering 2024-02-12T16:46:25.140 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:25.174 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:25.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:25.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:25.309 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:25 smithi123 bash[19119]: audit 2024-02-12T16:46:24.221249+0000 mon.b (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.121:0/3204700742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:46:25.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:25 smithi123 bash[19119]: cluster 2024-02-12T16:46:24.326316+0000 mgr.y (mgr.14146) 1865 : cluster [DBG] pgmap v2732: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 3852/67470 objects misplaced (5.709%); 5.3 MiB/s, 84 objects/s recovering 2024-02-12T16:46:25.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:25.437 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:25.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:25 smithi169 bash[19604]: audit 2024-02-12T16:46:24.221249+0000 mon.b (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.121:0/3204700742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-12T16:46:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:25 smithi169 bash[19604]: cluster 2024-02-12T16:46:24.326316+0000 mgr.y (mgr.14146) 1865 : cluster [DBG] pgmap v2732: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 3852/67470 objects misplaced (5.709%); 5.3 MiB/s, 84 objects/s recovering 2024-02-12T16:46:25.570 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:25.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:25.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:25.706 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:25.813 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:25.841 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:25.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:25.976 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:25.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:26.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:26.114 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:26.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:26.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:26.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:26 smithi169 bash[19604]: cluster 2024-02-12T16:46:24.918958+0000 osd.3 (osd.3) 124 : cluster [DBG] 9.c scrub starts 2024-02-12T16:46:26.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:26 smithi169 bash[19604]: cluster 2024-02-12T16:46:25.665923+0000 osd.3 (osd.3) 125 : cluster [DBG] 9.c scrub ok 2024-02-12T16:46:26.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:26.362 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:26 smithi123 bash[19119]: cluster 2024-02-12T16:46:24.918958+0000 osd.3 (osd.3) 124 : cluster [DBG] 9.c scrub starts 2024-02-12T16:46:26.362 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:26 smithi123 bash[19119]: cluster 2024-02-12T16:46:25.665923+0000 osd.3 (osd.3) 125 : cluster [DBG] 9.c scrub ok 2024-02-12T16:46:26.382 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:26.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:26 smithi121 bash[18489]: cluster 2024-02-12T16:46:24.918958+0000 osd.3 (osd.3) 124 : cluster [DBG] 9.c scrub starts 2024-02-12T16:46:26.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:26 smithi121 bash[18489]: cluster 2024-02-12T16:46:25.665923+0000 osd.3 (osd.3) 125 : cluster [DBG] 9.c scrub ok 2024-02-12T16:46:26.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:26.518 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:26.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:26.623 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:26.646 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:26.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:26.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:26.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:26.890 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:26.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:26.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:27.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:27.050 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:27.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:27.163 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:27 smithi169 bash[19604]: cluster 2024-02-12T16:46:26.326828+0000 mgr.y (mgr.14146) 1866 : cluster [DBG] pgmap v2733: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 845 op/s; 3852/66092 objects misplaced (5.828%); 621 KiB/s, 9 objects/s recovering 2024-02-12T16:46:27.185 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:27.264 INFO:tasks.thrashosds.thrasher:j is {'epoch': 999, 'fsid': 'eb2349b2-c9bd-11ee-95b9-87774f69a715', 'created': '2024-02-12T15:47:18.273813+0000', 'modified': '2024-02-12T16:46:23.058122+0000', 'last_up_change': '2024-02-12T16:42:40.138095+0000', 'last_in_change': '2024-02-12T16:44:59.017069+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 39, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-12T15:51:35.886878+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '957', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '663', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-02-12T16:42:20.926635+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 19, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '996', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '983', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '3a3badac-97ea-4ee1-ad82-1b67a4fe94f0', 'up': 1, 'in': 1, 'weight': 0.66845703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 964, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 1815232143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 1815232143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 1815232143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 1815232143}]}, 'public_addr': '172.21.15.121:6801/1815232143', 'cluster_addr': '172.21.15.121:6802/1815232143', 'heartbeat_back_addr': '172.21.15.121:6804/1815232143', 'heartbeat_front_addr': '172.21.15.121:6803/1815232143', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c6140309-5703-4538-9739-276fbbdaa880', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9075927734375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 997, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 2968433670}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 2968433670}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 2968433670}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 2968433670}]}, 'public_addr': '172.21.15.121:6805/2968433670', 'cluster_addr': '172.21.15.121:6806/2968433670', 'heartbeat_back_addr': '172.21.15.121:6808/2968433670', 'heartbeat_front_addr': '172.21.15.121:6807/2968433670', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c81e1a4-cae0-48d4-98ab-0becb6b31170', 'up': 1, 'in': 1, 'weight': 0.81610107421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 889, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 2158187852}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 2158187852}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 2158187852}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 2158187852}]}, 'public_addr': '172.21.15.121:6809/2158187852', 'cluster_addr': '172.21.15.121:6810/2158187852', 'heartbeat_back_addr': '172.21.15.121:6812/2158187852', 'heartbeat_front_addr': '172.21.15.121:6811/2158187852', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '5f824526-596d-42e1-8b7e-cd8676ded6dd', 'up': 1, 'in': 1, 'weight': 0.5543212890625, 'primary_affinity': 1, 'last_clean_begin': 26, 'last_clean_end': 414, 'up_from': 522, 'up_thru': 995, 'down_at': 415, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 2551349828}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 2551349828}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6816', 'nonce': 2551349828}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 2551349828}]}, 'public_addr': '172.21.15.121:6813/2551349828', 'cluster_addr': '172.21.15.121:6814/2551349828', 'heartbeat_back_addr': '172.21.15.121:6816/2551349828', 'heartbeat_front_addr': '172.21.15.121:6815/2551349828', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '14c2f878-99ce-4b51-8d9c-131c1d883b31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 964, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6800', 'nonce': 3447700123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6801', 'nonce': 3447700123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6803', 'nonce': 3447700123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6802', 'nonce': 3447700123}]}, 'public_addr': '172.21.15.123:6800/3447700123', 'cluster_addr': '172.21.15.123:6801/3447700123', 'heartbeat_back_addr': '172.21.15.123:6803/3447700123', 'heartbeat_front_addr': '172.21.15.123:6802/3447700123', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '75ecf26a-d7ed-46e7-a40e-6f719e2c3602', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 996, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6804', 'nonce': 961317306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6805', 'nonce': 961317306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6807', 'nonce': 961317306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6806', 'nonce': 961317306}]}, 'public_addr': '172.21.15.123:6804/961317306', 'cluster_addr': '172.21.15.123:6805/961317306', 'heartbeat_back_addr': '172.21.15.123:6807/961317306', 'heartbeat_front_addr': '172.21.15.123:6806/961317306', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '71e715e6-eac1-4386-927e-51df0cc419c0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 935, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6808', 'nonce': 1377261784}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6809', 'nonce': 1377261784}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6811', 'nonce': 1377261784}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6810', 'nonce': 1377261784}]}, 'public_addr': '172.21.15.123:6808/1377261784', 'cluster_addr': '172.21.15.123:6809/1377261784', 'heartbeat_back_addr': '172.21.15.123:6811/1377261784', 'heartbeat_front_addr': '172.21.15.123:6810/1377261784', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b419b04f-0c0c-495a-aa9b-25c334159515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 56, 'last_clean_end': 848, 'up_from': 923, 'up_thru': 996, 'down_at': 849, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6812', 'nonce': 4284140534}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6813', 'nonce': 4284140534}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6815', 'nonce': 4284140534}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.123:6814', 'nonce': 4284140534}]}, 'public_addr': '172.21.15.123:6812/4284140534', 'cluster_addr': '172.21.15.123:6813/4284140534', 'heartbeat_back_addr': '172.21.15.123:6815/4284140534', 'heartbeat_front_addr': '172.21.15.123:6814/4284140534', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1591e7bc-5b86-4401-803d-5325062ffb75', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 983, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6800', 'nonce': 2551075225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 2551075225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 2551075225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6802', 'nonce': 2551075225}]}, 'public_addr': '172.21.15.169:6800/2551075225', 'cluster_addr': '172.21.15.169:6801/2551075225', 'heartbeat_back_addr': '172.21.15.169:6803/2551075225', 'heartbeat_front_addr': '172.21.15.169:6802/2551075225', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '522d61df-8dc1-4915-83fd-f94d0d572c15', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 74, 'up_thru': 916, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6804', 'nonce': 2339555071}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 2339555071}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 2339555071}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6806', 'nonce': 2339555071}]}, 'public_addr': '172.21.15.169:6804/2339555071', 'cluster_addr': '172.21.15.169:6805/2339555071', 'heartbeat_back_addr': '172.21.15.169:6807/2339555071', 'heartbeat_front_addr': '172.21.15.169:6806/2339555071', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '91f74dca-503a-45d1-9647-df4804d58515', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 983, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6808', 'nonce': 391306610}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 391306610}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 391306610}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6810', 'nonce': 391306610}]}, 'public_addr': '172.21.15.169:6808/391306610', 'cluster_addr': '172.21.15.169:6809/391306610', 'heartbeat_back_addr': '172.21.15.169:6811/391306610', 'heartbeat_front_addr': '172.21.15.169:6810/391306610', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '788b4f47-eadf-4601-a5b4-51fb6f04fbdf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 93, 'up_thru': 983, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6812', 'nonce': 3696812305}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3696812305}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3696812305}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.169:6814', 'nonce': 3696812305}]}, 'public_addr': '172.21.15.169:6812/3696812305', 'cluster_addr': '172.21.15.169:6813/3696812305', 'heartbeat_back_addr': '172.21.15.169:6815/3696812305', 'heartbeat_front_addr': '172.21.15.169:6814/3696812305', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:46:22.774266+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:45:55.921494+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:45:55.380611+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:45:24.534653+0000', 'dead_epoch': 414}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:45:51.791648+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:45:50.737919+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-12T16:46:14.900537+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:46:06.571103+0000', 'dead_epoch': 848}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:45:17.117070+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 59709, 'last_purged_snaps_scrub': '2024-02-12T16:45:46.061144+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:46:13.727782+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-12T16:45:58.901685+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1b', 'osds': [10, 5]}], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '9.0', 'osds': [1, 6]}, {'pgid': '9.10', 'osds': [1, 6]}, {'pgid': '9.12', 'osds': [1, 11]}], 'primary_temp': [], 'blocklist': {'172.21.15.121:0/2474849789': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/253689008': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:6800/3531151725': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:0/1778611055': '2024-02-13T15:47:47.232761+0000', '172.21.15.121:6800/28718860': '2024-02-13T15:48:17.261083+0000', '172.21.15.121:0/445290372': '2024-02-13T15:47:47.232761+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-12T16:46:27.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:46:27.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:27.312 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:27 smithi123 bash[19119]: cluster 2024-02-12T16:46:26.326828+0000 mgr.y (mgr.14146) 1866 : cluster [DBG] pgmap v2733: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 845 op/s; 3852/66092 objects misplaced (5.828%); 621 KiB/s, 9 objects/s recovering 2024-02-12T16:46:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:27 smithi121 bash[18489]: cluster 2024-02-12T16:46:26.326828+0000 mgr.y (mgr.14146) 1866 : cluster [DBG] pgmap v2733: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 2.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 845 op/s; 3852/66092 objects misplaced (5.828%); 621 KiB/s, 9 objects/s recovering 2024-02-12T16:46:27.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:27.445 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:27.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:27.577 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:27.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-12T16:46:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:27.684 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:27.712 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:27.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:27.841 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:27.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:27.975 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:27.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:28.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:28.105 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:28.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:28.212 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:28.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:28.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:28.389 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:28.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:28.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:28.519 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:28.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:28.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:28.629 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:28 smithi121 bash[18489]: audit 2024-02-12T16:46:28.328200+0000 mon.a (mon.0) 3302 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:46:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:28 smithi123 bash[19119]: audit 2024-02-12T16:46:28.328200+0000 mon.a (mon.0) 3302 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:46:28.650 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:28.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:28.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:28.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:28.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:28.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:28 smithi169 bash[19604]: audit 2024-02-12T16:46:28.328200+0000 mon.a (mon.0) 3302 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]: dispatch 2024-02-12T16:46:28.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:28.905 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:28.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:29.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:29.032 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:29.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:29.161 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:29.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:29.264 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:29.291 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:29.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:29 smithi121 bash[18489]: cluster 2024-02-12T16:46:28.327257+0000 mgr.y (mgr.14146) 1867 : cluster [DBG] pgmap v2734: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 2936/59600 objects misplaced (4.926%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:46:29.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:29 smithi121 bash[18489]: audit 2024-02-12T16:46:28.391361+0000 mon.a (mon.0) 3303 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:46:29.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:29 smithi121 bash[18489]: cluster 2024-02-12T16:46:28.391458+0000 mon.a (mon.0) 3304 : cluster [DBG] osdmap e1000: 12 total, 12 up, 10 in 2024-02-12T16:46:29.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:29.426 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:29.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:29.540 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:29 smithi123 bash[19119]: cluster 2024-02-12T16:46:28.327257+0000 mgr.y (mgr.14146) 1867 : cluster [DBG] pgmap v2734: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 2936/59600 objects misplaced (4.926%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:46:29.540 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:29 smithi123 bash[19119]: audit 2024-02-12T16:46:28.391361+0000 mon.a (mon.0) 3303 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:46:29.540 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:29 smithi123 bash[19119]: cluster 2024-02-12T16:46:28.391458+0000 mon.a (mon.0) 3304 : cluster [DBG] osdmap e1000: 12 total, 12 up, 10 in 2024-02-12T16:46:29.556 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:29.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:29.694 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:29.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:29 smithi169 bash[19604]: cluster 2024-02-12T16:46:28.327257+0000 mgr.y (mgr.14146) 1867 : cluster [DBG] pgmap v2734: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 2936/59600 objects misplaced (4.926%); 1.4 MiB/s, 21 objects/s recovering 2024-02-12T16:46:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:29 smithi169 bash[19604]: audit 2024-02-12T16:46:28.391361+0000 mon.a (mon.0) 3303 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]': finished 2024-02-12T16:46:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:29 smithi169 bash[19604]: cluster 2024-02-12T16:46:28.391458+0000 mon.a (mon.0) 3304 : cluster [DBG] osdmap e1000: 12 total, 12 up, 10 in 2024-02-12T16:46:29.828 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:29.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:29.967 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:29.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:30.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:30.107 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:30.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:30.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:30.236 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:30.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:30.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:30.366 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:30.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:30.445 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:46:30.467 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:46:30.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:30.476 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:30 smithi121 bash[18489]: cluster 2024-02-12T16:46:29.397665+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e1001: 12 total, 12 up, 10 in 2024-02-12T16:46:30.490 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:46:30.495 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:30.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:30.525 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:46:30.561 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:46:30.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:30.599 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:46:30.628 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:30.629 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:46:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:30.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:46:30.690 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:46:30.720 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:46:30.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:30.758 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:46:30.772 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:30.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:30.793 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:46:30.797 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:46:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:30 smithi169 bash[19604]: cluster 2024-02-12T16:46:29.397665+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e1001: 12 total, 12 up, 10 in 2024-02-12T16:46:30.875 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:30 smithi123 bash[19119]: cluster 2024-02-12T16:46:29.397665+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e1001: 12 total, 12 up, 10 in 2024-02-12T16:46:30.907 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:30.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:31.014 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:31.041 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:46:31.047 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:31.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:31.178 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:31.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:31.313 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:30.028458+0000 osd.3 (osd.3) 126 : cluster [DBG] 9.1 scrub starts 2024-02-12T16:46:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:30.159627+0000 osd.3 (osd.3) 127 : cluster [DBG] 9.1 scrub ok 2024-02-12T16:46:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:30.327791+0000 mgr.y (mgr.14146) 1868 : cluster [DBG] pgmap v2737: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 2936/54528 objects misplaced (5.384%); 1014 KiB/s, 15 objects/s recovering 2024-02-12T16:46:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:30.408750+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e1002: 12 total, 12 up, 10 in 2024-02-12T16:46:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:30.411421+0000 osd.3 (osd.3) 128 : cluster [DBG] 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 999'16421 2024-02-12T16:46:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: audit 2024-02-12T16:46:30.416620+0000 mon.a (mon.0) 3307 : audit [DBG] from='client.? v1:172.21.15.121:0/1245205385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:30.426439+0000 osd.3 (osd.3) 129 : cluster [DBG] 9.13 starting backfill to osd.11 from (0'0,0'0] MAX to 999'16421 2024-02-12T16:46:31.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: audit 2024-02-12T16:46:30.737600+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? v1:172.21.15.121:0/1663901787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:46:31.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:31.019529+0000 mon.a (mon.0) 3309 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:46:31.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: audit 2024-02-12T16:46:31.024014+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? v1:172.21.15.121:0/1663901787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:46:31.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:31 smithi121 bash[18489]: cluster 2024-02-12T16:46:31.024084+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e1003: 12 total, 12 up, 10 in 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:30.028458+0000 osd.3 (osd.3) 126 : cluster [DBG] 9.1 scrub starts 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:30.159627+0000 osd.3 (osd.3) 127 : cluster [DBG] 9.1 scrub ok 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:30.327791+0000 mgr.y (mgr.14146) 1868 : cluster [DBG] pgmap v2737: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 2936/54528 objects misplaced (5.384%); 1014 KiB/s, 15 objects/s recovering 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:30.408750+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e1002: 12 total, 12 up, 10 in 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:30.411421+0000 osd.3 (osd.3) 128 : cluster [DBG] 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 999'16421 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: audit 2024-02-12T16:46:30.416620+0000 mon.a (mon.0) 3307 : audit [DBG] from='client.? v1:172.21.15.121:0/1245205385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:30.426439+0000 osd.3 (osd.3) 129 : cluster [DBG] 9.13 starting backfill to osd.11 from (0'0,0'0] MAX to 999'16421 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: audit 2024-02-12T16:46:30.737600+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? v1:172.21.15.121:0/1663901787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:31.019529+0000 mon.a (mon.0) 3309 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: audit 2024-02-12T16:46:31.024014+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? v1:172.21.15.121:0/1663901787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:46:31.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:31 smithi169 bash[19604]: cluster 2024-02-12T16:46:31.024084+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e1003: 12 total, 12 up, 10 in 2024-02-12T16:46:31.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:31.452 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:31.558 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:31.587 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:31.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:31.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:30.028458+0000 osd.3 (osd.3) 126 : cluster [DBG] 9.1 scrub starts 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:30.159627+0000 osd.3 (osd.3) 127 : cluster [DBG] 9.1 scrub ok 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:30.327791+0000 mgr.y (mgr.14146) 1868 : cluster [DBG] pgmap v2737: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s; 2936/54528 objects misplaced (5.384%); 1014 KiB/s, 15 objects/s recovering 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:30.408750+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e1002: 12 total, 12 up, 10 in 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:30.411421+0000 osd.3 (osd.3) 128 : cluster [DBG] 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 999'16421 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: audit 2024-02-12T16:46:30.416620+0000 mon.a (mon.0) 3307 : audit [DBG] from='client.? v1:172.21.15.121:0/1245205385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:30.426439+0000 osd.3 (osd.3) 129 : cluster [DBG] 9.13 starting backfill to osd.11 from (0'0,0'0] MAX to 999'16421 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: audit 2024-02-12T16:46:30.737600+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? v1:172.21.15.121:0/1663901787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:46:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:31.019529+0000 mon.a (mon.0) 3309 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:46:31.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: audit 2024-02-12T16:46:31.024014+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? v1:172.21.15.121:0/1663901787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:46:31.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:31 smithi123 bash[19119]: cluster 2024-02-12T16:46:31.024084+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e1003: 12 total, 12 up, 10 in 2024-02-12T16:46:31.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:31.994 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:31.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:32.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:32.125 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:32.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:32.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:32.261 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:32.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:32.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:32.398 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:32.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:32.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:32.533 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:32.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:32.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:32.666 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:32.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:32.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:32.796 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:32.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:32.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:32.924 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:32.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:33.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:33.062 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:33.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:33.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:33.199 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:33.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:33.256 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:33.257 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:33.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:33.326 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:33.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:33.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:33.468 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:33.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:33.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:33.607 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:33.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:33 smithi123 bash[19119]: cluster 2024-02-12T16:46:32.328232+0000 mgr.y (mgr.14146) 1869 : cluster [DBG] pgmap v2740: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 1.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 3450/48502 objects misplaced (7.113%) 2024-02-12T16:46:33.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:33.720 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:33 smithi121 bash[18489]: cluster 2024-02-12T16:46:32.328232+0000 mgr.y (mgr.14146) 1869 : cluster [DBG] pgmap v2740: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 1.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 3450/48502 objects misplaced (7.113%) 2024-02-12T16:46:33.747 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:33.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:33 smithi169 bash[19604]: cluster 2024-02-12T16:46:32.328232+0000 mgr.y (mgr.14146) 1869 : cluster [DBG] pgmap v2740: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 1.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 3450/48502 objects misplaced (7.113%) 2024-02-12T16:46:33.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:33.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:33.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:34.014 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:34.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:34.153 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:34.260 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:34.290 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:34.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:34.421 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:34.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:46:34.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:34.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:34.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:34.671 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:34.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:34.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set noscrub 2024-02-12T16:46:34.802 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:34.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:34.931 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:34.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:35.034 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:35.057 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:35.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:35.196 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:35.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:35.332 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:35.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:35.564 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:35.590 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:35 smithi121 bash[18489]: cluster 2024-02-12T16:46:34.328740+0000 mgr.y (mgr.14146) 1870 : cluster [DBG] pgmap v2741: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 1.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 3450/45078 objects misplaced (7.653%) 2024-02-12T16:46:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:35 smithi121 bash[18489]: cluster 2024-02-12T16:46:34.396745+0000 osd.3 (osd.3) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:35 smithi121 bash[18489]: cluster 2024-02-12T16:46:34.396821+0000 osd.3 (osd.3) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:35 smithi121 bash[18489]: cluster 2024-02-12T16:46:34.415773+0000 mon.a (mon.0) 3312 : cluster [DBG] osdmap e1004: 12 total, 12 up, 10 in 2024-02-12T16:46:35.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:35.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:35 smithi169 bash[19604]: cluster 2024-02-12T16:46:34.328740+0000 mgr.y (mgr.14146) 1870 : cluster [DBG] pgmap v2741: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 1.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 3450/45078 objects misplaced (7.653%) 2024-02-12T16:46:35.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:35 smithi169 bash[19604]: cluster 2024-02-12T16:46:34.396745+0000 osd.3 (osd.3) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:35.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:35 smithi169 bash[19604]: cluster 2024-02-12T16:46:34.396821+0000 osd.3 (osd.3) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:35.703 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:35 smithi169 bash[19604]: cluster 2024-02-12T16:46:34.415773+0000 mon.a (mon.0) 3312 : cluster [DBG] osdmap e1004: 12 total, 12 up, 10 in 2024-02-12T16:46:35.721 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:35.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:35.833 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:35 smithi123 bash[19119]: cluster 2024-02-12T16:46:34.328740+0000 mgr.y (mgr.14146) 1870 : cluster [DBG] pgmap v2741: 57 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 53 active+clean; 1.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 3450/45078 objects misplaced (7.653%) 2024-02-12T16:46:35.833 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:35 smithi123 bash[19119]: cluster 2024-02-12T16:46:34.396745+0000 osd.3 (osd.3) 130 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:35.834 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:35 smithi123 bash[19119]: cluster 2024-02-12T16:46:34.396821+0000 osd.3 (osd.3) 131 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:35.834 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:35 smithi123 bash[19119]: cluster 2024-02-12T16:46:34.415773+0000 mon.a (mon.0) 3312 : cluster [DBG] osdmap e1004: 12 total, 12 up, 10 in 2024-02-12T16:46:35.854 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:35.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:35.988 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:35.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:36.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:36.119 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:36.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:36.224 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:36.252 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:36.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:36.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:36.386 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:36.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:36.431 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:36 smithi169 bash[19604]: cluster 2024-02-12T16:46:35.425250+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e1005: 12 total, 12 up, 10 in 2024-02-12T16:46:36.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:36.497 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:36 smithi123 bash[19119]: cluster 2024-02-12T16:46:35.425250+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e1005: 12 total, 12 up, 10 in 2024-02-12T16:46:36.525 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:36.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:36.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:36.656 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:36.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:36 smithi121 bash[18489]: cluster 2024-02-12T16:46:35.425250+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e1005: 12 total, 12 up, 10 in 2024-02-12T16:46:36.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:36.811 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:36.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:36.915 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:36.946 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:36.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:37.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:37.094 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:37.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:37.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:37.226 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:37.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:37.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:37.360 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:37.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:37.463 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:37.492 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:37.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:37 smithi169 bash[19604]: cluster 2024-02-12T16:46:36.329200+0000 mgr.y (mgr.14146) 1871 : cluster [DBG] pgmap v2744: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.3 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 2320/43422 objects misplaced (5.343%) 2024-02-12T16:46:37.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:37.603 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:37 smithi123 bash[19119]: cluster 2024-02-12T16:46:36.329200+0000 mgr.y (mgr.14146) 1871 : cluster [DBG] pgmap v2744: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.3 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 2320/43422 objects misplaced (5.343%) 2024-02-12T16:46:37.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:37.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:37.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:37.745 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:37 smithi121 bash[18489]: cluster 2024-02-12T16:46:36.329200+0000 mgr.y (mgr.14146) 1871 : cluster [DBG] pgmap v2744: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.3 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 2320/43422 objects misplaced (5.343%) 2024-02-12T16:46:37.780 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:37.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:37.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:37.919 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:37.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:38.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:38.058 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:38.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:38.187 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:38.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:38.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:38.317 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:38.420 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:38.454 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:38.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:38.567 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:38 smithi169 bash[19604]: audit 2024-02-12T16:46:38.330380+0000 mon.a (mon.0) 3314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:46:38.587 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:38.690 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:38.716 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:38.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:38.845 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:38.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:38 smithi123 bash[19119]: audit 2024-02-12T16:46:38.330380+0000 mon.a (mon.0) 3314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:46:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:38 smithi121 bash[18489]: audit 2024-02-12T16:46:38.330380+0000 mon.a (mon.0) 3314 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-12T16:46:38.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:38.972 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:38.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:39.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:39.097 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:39.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:39.200 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:39.230 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:39.230 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-02-12T16:46:39.230 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:46:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:39.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:39.309 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:39.310 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:39.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:39.364 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:39.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:39.452 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:46:39.453 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:46:39.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:39.490 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:39.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:39 smithi169 bash[19604]: cluster 2024-02-12T16:46:38.329697+0000 mgr.y (mgr.14146) 1872 : cluster [DBG] pgmap v2745: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 1555/37720 objects misplaced (4.122%) 2024-02-12T16:46:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:39 smithi169 bash[19604]: audit 2024-02-12T16:46:38.433761+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:46:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:39 smithi169 bash[19604]: cluster 2024-02-12T16:46:38.433824+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e1006: 12 total, 12 up, 10 in 2024-02-12T16:46:39.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:39.601 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:39 smithi123 bash[19119]: cluster 2024-02-12T16:46:38.329697+0000 mgr.y (mgr.14146) 1872 : cluster [DBG] pgmap v2745: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 1555/37720 objects misplaced (4.122%) 2024-02-12T16:46:39.602 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:39 smithi123 bash[19119]: audit 2024-02-12T16:46:38.433761+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:46:39.602 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:39 smithi123 bash[19119]: cluster 2024-02-12T16:46:38.433824+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e1006: 12 total, 12 up, 10 in 2024-02-12T16:46:39.617 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:39.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:39 smithi121 bash[18489]: cluster 2024-02-12T16:46:38.329697+0000 mgr.y (mgr.14146) 1872 : cluster [DBG] pgmap v2745: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 1555/37720 objects misplaced (4.122%) 2024-02-12T16:46:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:39 smithi121 bash[18489]: audit 2024-02-12T16:46:38.433761+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-12T16:46:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:39 smithi121 bash[18489]: cluster 2024-02-12T16:46:38.433824+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e1006: 12 total, 12 up, 10 in 2024-02-12T16:46:39.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:39.745 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:39.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:39.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:39.870 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:39.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:39.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:39.998 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:40.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:40.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:40.273 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:40.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:40.409 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:40.512 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:40.542 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:40.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:40.652 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:40 smithi121 bash[18489]: cluster 2024-02-12T16:46:39.060678+0000 osd.3 (osd.3) 132 : cluster [DBG] 1.3 deep-scrub starts 2024-02-12T16:46:40.652 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:40 smithi121 bash[18489]: cluster 2024-02-12T16:46:39.182346+0000 osd.3 (osd.3) 133 : cluster [DBG] 1.3 deep-scrub ok 2024-02-12T16:46:40.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:40 smithi121 bash[18489]: cluster 2024-02-12T16:46:39.383517+0000 osd.8 (osd.8) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:40.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:40 smithi121 bash[18489]: cluster 2024-02-12T16:46:39.383585+0000 osd.8 (osd.8) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:40.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:40 smithi121 bash[18489]: audit 2024-02-12T16:46:39.431990+0000 mgr.y (mgr.14146) 1873 : audit [DBG] from='client.17325 v1:172.21.15.121:0/1458020971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:46:40.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:40 smithi121 bash[18489]: cluster 2024-02-12T16:46:39.441783+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e1007: 12 total, 12 up, 10 in 2024-02-12T16:46:40.653 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:40 smithi121 bash[18489]: audit 2024-02-12T16:46:40.331161+0000 mon.a (mon.0) 3318 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:46:40.676 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:40.783 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:40.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:40 smithi169 bash[19604]: cluster 2024-02-12T16:46:39.060678+0000 osd.3 (osd.3) 132 : cluster [DBG] 1.3 deep-scrub starts 2024-02-12T16:46:40.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:40 smithi169 bash[19604]: cluster 2024-02-12T16:46:39.182346+0000 osd.3 (osd.3) 133 : cluster [DBG] 1.3 deep-scrub ok 2024-02-12T16:46:40.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:40 smithi169 bash[19604]: cluster 2024-02-12T16:46:39.383517+0000 osd.8 (osd.8) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:40.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:40 smithi169 bash[19604]: cluster 2024-02-12T16:46:39.383585+0000 osd.8 (osd.8) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:40.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:40 smithi169 bash[19604]: audit 2024-02-12T16:46:39.431990+0000 mgr.y (mgr.14146) 1873 : audit [DBG] from='client.17325 v1:172.21.15.121:0/1458020971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:46:40.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:40 smithi169 bash[19604]: cluster 2024-02-12T16:46:39.441783+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e1007: 12 total, 12 up, 10 in 2024-02-12T16:46:40.791 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:40 smithi169 bash[19604]: audit 2024-02-12T16:46:40.331161+0000 mon.a (mon.0) 3318 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:46:40.811 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:40.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-12T16:46:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:40 smithi123 bash[19119]: cluster 2024-02-12T16:46:39.060678+0000 osd.3 (osd.3) 132 : cluster [DBG] 1.3 deep-scrub starts 2024-02-12T16:46:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:40 smithi123 bash[19119]: cluster 2024-02-12T16:46:39.182346+0000 osd.3 (osd.3) 133 : cluster [DBG] 1.3 deep-scrub ok 2024-02-12T16:46:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:40 smithi123 bash[19119]: cluster 2024-02-12T16:46:39.383517+0000 osd.8 (osd.8) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:46:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:40 smithi123 bash[19119]: cluster 2024-02-12T16:46:39.383585+0000 osd.8 (osd.8) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:46:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:40 smithi123 bash[19119]: audit 2024-02-12T16:46:39.431990+0000 mgr.y (mgr.14146) 1873 : audit [DBG] from='client.17325 v1:172.21.15.121:0/1458020971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:46:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:40 smithi123 bash[19119]: cluster 2024-02-12T16:46:39.441783+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e1007: 12 total, 12 up, 10 in 2024-02-12T16:46:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:40 smithi123 bash[19119]: audit 2024-02-12T16:46:40.331161+0000 mon.a (mon.0) 3318 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-12T16:46:40.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:40.949 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:40.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:41.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:41.076 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:41.178 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:41.206 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:41.294 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":2746,"stamp":"2024-02-12T16:46:38.431653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1235910680,"num_objects":18860,"num_object_clones":0,"num_object_copies":37720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1555,"num_objects_unfound":0,"num_objects_dirty":18860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":274054,"num_write_kb":9349025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18794,"num_bytes_recovered":1231585280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282212,"ondisk_log_size":282212,"up":114,"acting":114,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":10699044,"kb_used_data":2600020,"kb_used_omap":92,"kb_used_meta":8098851,"kb_avail":926752476,"statfs":{"total":959950356480,"available":948994535424,"internally_reserved":0,"allocated":2662420480,"data_stored":2656547027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95164,"internal_metadata":8293223492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-353304576,"num_objects":-5391,"num_object_clones":0,"num_object_copies":-10782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1895,"num_objects_unfound":0,"num_objects_dirty":-5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":40632320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.103682"},"pg_stats":[{"pgid":"9.17","version":"1003'16765","reported_seq":16872,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.800685+0000","last_change":"2024-02-12T16:46:03.664548+0000","last_active":"2024-02-12T16:46:33.800685+0000","last_peered":"2024-02-12T16:46:33.800685+0000","last_clean":"2024-02-12T16:46:33.800685+0000","last_became_active":"2024-02-12T16:46:03.628055+0000","last_became_peered":"2024-02-12T16:46:03.628055+0000","last_unstale":"2024-02-12T16:46:33.800685+0000","last_undegraded":"2024-02-12T16:46:33.800685+0000","last_fullsized":"2024-02-12T16:46:33.800685+0000","mapping_epoch":983,"log_start":"934'5700","ondisk_log_start":"934'5700","created":983,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11065,"ondisk_log_size":11065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47546368,"num_objects":726,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8870,"num_write_kb":307040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.16","version":"1005'16526","reported_seq":16629,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:37.139794+0000","last_change":"2024-02-12T16:46:03.652111+0000","last_active":"2024-02-12T16:46:37.139794+0000","last_peered":"2024-02-12T16:46:37.139794+0000","last_clean":"2024-02-12T16:46:37.139794+0000","last_became_active":"2024-02-12T16:46:03.625834+0000","last_became_peered":"2024-02-12T16:46:03.625834+0000","last_unstale":"2024-02-12T16:46:37.139794+0000","last_undegraded":"2024-02-12T16:46:37.139794+0000","last_fullsized":"2024-02-12T16:46:37.139794+0000","mapping_epoch":983,"log_start":"934'5400","ondisk_log_start":"934'5400","created":983,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11126,"ondisk_log_size":11126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36503552,"num_objects":557,"num_object_clones":0,"num_object_copies":1114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8807,"num_write_kb":299648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1e","version":"0'0","reported_seq":369,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.422631+0000","last_change":"2024-02-12T16:42:21.631979+0000","last_active":"2024-02-12T16:46:34.422631+0000","last_peered":"2024-02-12T16:46:34.422631+0000","last_clean":"2024-02-12T16:46:34.422631+0000","last_became_active":"2024-02-12T16:41:18.947003+0000","last_became_peered":"2024-02-12T16:41:18.947003+0000","last_unstale":"2024-02-12T16:46:34.422631+0000","last_undegraded":"2024-02-12T16:46:34.422631+0000","last_fullsized":"2024-02-12T16:46:34.422631+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:21.631859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:21.631859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.15","version":"1005'16677","reported_seq":16781,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:35.479539+0000","last_change":"2024-02-12T16:46:03.699900+0000","last_active":"2024-02-12T16:46:35.479539+0000","last_peered":"2024-02-12T16:46:35.479539+0000","last_clean":"2024-02-12T16:46:35.479539+0000","last_became_active":"2024-02-12T16:46:03.653135+0000","last_became_peered":"2024-02-12T16:46:03.653135+0000","last_unstale":"2024-02-12T16:46:35.479539+0000","last_undegraded":"2024-02-12T16:46:35.479539+0000","last_fullsized":"2024-02-12T16:46:35.479539+0000","mapping_epoch":983,"log_start":"934'5600","ondisk_log_start":"934'5600","created":983,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11077,"ondisk_log_size":11077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40665088,"num_objects":621,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8847,"num_write_kb":302944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":397,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.419064+0000","last_change":"2024-02-12T16:42:29.489386+0000","last_active":"2024-02-12T16:46:34.419064+0000","last_peered":"2024-02-12T16:46:34.419064+0000","last_clean":"2024-02-12T16:46:34.419064+0000","last_became_active":"2024-02-12T16:41:17.943883+0000","last_became_peered":"2024-02-12T16:41:17.943883+0000","last_unstale":"2024-02-12T16:46:34.419064+0000","last_undegraded":"2024-02-12T16:46:34.419064+0000","last_fullsized":"2024-02-12T16:46:34.419064+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:29.489294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:29.489294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.14","version":"1005'16838","reported_seq":16942,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:36.899223+0000","last_change":"2024-02-12T16:46:03.647823+0000","last_active":"2024-02-12T16:46:36.899223+0000","last_peered":"2024-02-12T16:46:36.899223+0000","last_clean":"2024-02-12T16:46:36.899223+0000","last_became_active":"2024-02-12T16:46:03.621779+0000","last_became_peered":"2024-02-12T16:46:03.621779+0000","last_unstale":"2024-02-12T16:46:36.899223+0000","last_undegraded":"2024-02-12T16:46:36.899223+0000","last_fullsized":"2024-02-12T16:46:36.899223+0000","mapping_epoch":983,"log_start":"934'5700","ondisk_log_start":"934'5700","created":983,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11138,"ondisk_log_size":11138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":303520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":388,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.443411+0000","last_change":"2024-02-12T16:45:00.042491+0000","last_active":"2024-02-12T16:46:30.443411+0000","last_peered":"2024-02-12T16:46:30.443411+0000","last_clean":"2024-02-12T16:46:30.443411+0000","last_became_active":"2024-02-12T16:45:00.042127+0000","last_became_peered":"2024-02-12T16:45:00.042127+0000","last_unstale":"2024-02-12T16:46:30.443411+0000","last_undegraded":"2024-02-12T16:46:30.443411+0000","last_fullsized":"2024-02-12T16:46:30.443411+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.683932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.683932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"1005'16608","reported_seq":18779,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:35.507865+0000","last_change":"2024-02-12T16:46:35.469596+0000","last_active":"2024-02-12T16:46:35.507865+0000","last_peered":"2024-02-12T16:46:35.507865+0000","last_clean":"2024-02-12T16:46:35.507865+0000","last_became_active":"2024-02-12T16:46:35.432057+0000","last_became_peered":"2024-02-12T16:46:35.432057+0000","last_unstale":"2024-02-12T16:46:35.507865+0000","last_undegraded":"2024-02-12T16:46:35.507865+0000","last_fullsized":"2024-02-12T16:46:35.507865+0000","mapping_epoch":1004,"log_start":"934'5500","ondisk_log_start":"934'5500","created":983,"last_epoch_clean":1005,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11108,"ondisk_log_size":11108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40927232,"num_objects":625,"num_object_clones":0,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":302240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":683,"num_bytes_recovered":44761088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":201,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.581733+0000","last_change":"2024-02-12T16:44:57.998874+0000","last_active":"2024-02-12T16:46:30.581733+0000","last_peered":"2024-02-12T16:46:30.581733+0000","last_clean":"2024-02-12T16:46:30.581733+0000","last_became_active":"2024-02-12T16:39:09.436840+0000","last_became_peered":"2024-02-12T16:39:09.436840+0000","last_unstale":"2024-02-12T16:46:30.581733+0000","last_undegraded":"2024-02-12T16:46:30.581733+0000","last_fullsized":"2024-02-12T16:46:30.581733+0000","mapping_epoch":845,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":846,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_clean_scrub_stamp":"2024-02-12T16:41:21.337430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.12","version":"1005'16633","reported_seq":28955,"reported_epoch":1005,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:46:36.989644+0000","last_change":"2024-02-12T16:46:21.371739+0000","last_active":"2024-02-12T16:46:36.989644+0000","last_peered":"2024-02-12T16:46:36.989644+0000","last_clean":"2024-02-12T16:46:18.994225+0000","last_became_active":"2024-02-12T16:46:21.282521+0000","last_became_peered":"2024-02-12T16:46:21.282521+0000","last_unstale":"2024-02-12T16:46:36.989644+0000","last_undegraded":"2024-02-12T16:46:36.989644+0000","last_fullsized":"2024-02-12T16:46:36.989644+0000","mapping_epoch":997,"log_start":"934'5500","ondisk_log_start":"934'5500","created":983,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11133,"ondisk_log_size":11133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34308096,"num_objects":524,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":393,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8870,"num_write_kb":300576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3141,"num_bytes_recovered":205848576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[1,11],"avail_no_missing":["1","9","11"],"object_location_counts":[{"shards":"1,11","objects":524}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":432,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.463399+0000","last_change":"2024-02-12T16:42:41.310928+0000","last_active":"2024-02-12T16:46:30.463399+0000","last_peered":"2024-02-12T16:46:30.463399+0000","last_clean":"2024-02-12T16:46:30.463399+0000","last_became_active":"2024-02-12T16:42:40.975420+0000","last_became_peered":"2024-02-12T16:42:40.975420+0000","last_unstale":"2024-02-12T16:46:30.463399+0000","last_undegraded":"2024-02-12T16:46:30.463399+0000","last_fullsized":"2024-02-12T16:46:30.463399+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:41.310840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:41.310840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"1003'16237","reported_seq":34306,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.185586+0000","last_change":"2024-02-12T16:46:20.164676+0000","last_active":"2024-02-12T16:46:34.185586+0000","last_peered":"2024-02-12T16:46:34.185586+0000","last_clean":"2024-02-12T16:46:34.185586+0000","last_became_active":"2024-02-12T16:46:20.048117+0000","last_became_peered":"2024-02-12T16:46:20.048117+0000","last_unstale":"2024-02-12T16:46:34.185586+0000","last_undegraded":"2024-02-12T16:46:34.185586+0000","last_fullsized":"2024-02-12T16:46:34.185586+0000","mapping_epoch":996,"log_start":"933'5200","ondisk_log_start":"933'5200","created":983,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11037,"ondisk_log_size":11037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37781504,"num_objects":577,"num_object_clones":0,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8629,"num_write_kb":294560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5441,"num_bytes_recovered":356548608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":401,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.577128+0000","last_change":"2024-02-12T16:45:00.376435+0000","last_active":"2024-02-12T16:46:30.577128+0000","last_peered":"2024-02-12T16:46:30.577128+0000","last_clean":"2024-02-12T16:46:30.577128+0000","last_became_active":"2024-02-12T16:45:00.375895+0000","last_became_peered":"2024-02-12T16:45:00.375895+0000","last_unstale":"2024-02-12T16:46:30.577128+0000","last_undegraded":"2024-02-12T16:46:30.577128+0000","last_fullsized":"2024-02-12T16:46:30.577128+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:24.921234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:24.921234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"1005'16662","reported_seq":17504,"reported_epoch":1005,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-12T16:46:36.887805+0000","last_change":"2024-02-12T16:46:10.765171+0000","last_active":"2024-02-12T16:46:36.887805+0000","last_peered":"2024-02-12T16:46:36.887805+0000","last_clean":"2024-02-12T16:44:59.014278+0000","last_became_active":"2024-02-12T16:46:10.683158+0000","last_became_peered":"2024-02-12T16:46:10.683158+0000","last_unstale":"2024-02-12T16:46:36.887805+0000","last_undegraded":"2024-02-12T16:46:36.887805+0000","last_fullsized":"2024-02-12T16:46:36.887805+0000","mapping_epoch":989,"log_start":"934'5500","ondisk_log_start":"934'5500","created":983,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11162,"ondisk_log_size":11162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32899072,"num_objects":502,"num_object_clones":0,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1004,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8883,"num_write_kb":300320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":11698176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":502}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":739,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.421258+0000","last_change":"2024-02-12T16:42:31.552860+0000","last_active":"2024-02-12T16:46:34.421258+0000","last_peered":"2024-02-12T16:46:34.421258+0000","last_clean":"2024-02-12T16:46:34.421258+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:46:34.421258+0000","last_undegraded":"2024-02-12T16:46:34.421258+0000","last_fullsized":"2024-02-12T16:46:34.421258+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:31.552623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:34:07.972836+0000","last_clean_scrub_stamp":"2024-02-12T16:42:31.552623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":384,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.443745+0000","last_change":"2024-02-12T16:43:31.617757+0000","last_active":"2024-02-12T16:46:30.443745+0000","last_peered":"2024-02-12T16:46:30.443745+0000","last_clean":"2024-02-12T16:46:30.443745+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:46:30.443745+0000","last_undegraded":"2024-02-12T16:46:30.443745+0000","last_fullsized":"2024-02-12T16:46:30.443745+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.617636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.617636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":381,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.415649+0000","last_change":"2024-02-12T16:42:26.378442+0000","last_active":"2024-02-12T16:46:34.415649+0000","last_peered":"2024-02-12T16:46:34.415649+0000","last_clean":"2024-02-12T16:46:34.415649+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:46:34.415649+0000","last_undegraded":"2024-02-12T16:46:34.415649+0000","last_fullsized":"2024-02-12T16:46:34.415649+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.378312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.378312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":366,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.578326+0000","last_change":"2024-02-12T16:46:27.567194+0000","last_active":"2024-02-12T16:46:30.578326+0000","last_peered":"2024-02-12T16:46:30.578326+0000","last_clean":"2024-02-12T16:46:30.578326+0000","last_became_active":"2024-02-12T16:42:40.982346+0000","last_became_peered":"2024-02-12T16:42:40.982346+0000","last_unstale":"2024-02-12T16:46:30.578326+0000","last_undegraded":"2024-02-12T16:46:30.578326+0000","last_fullsized":"2024-02-12T16:46:30.578326+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:39:11.207557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:39:11.207557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":650,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.636310+0000","last_change":"2024-02-12T16:46:13.097407+0000","last_active":"2024-02-12T16:46:34.636310+0000","last_peered":"2024-02-12T16:46:34.636310+0000","last_clean":"2024-02-12T16:46:34.636310+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:46:34.636310+0000","last_undegraded":"2024-02-12T16:46:34.636310+0000","last_fullsized":"2024-02-12T16:46:34.636310+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:46:13.097331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:05.516165+0000","last_clean_scrub_stamp":"2024-02-12T16:46:13.097331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"1005'16411","reported_seq":28213,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:35.504747+0000","last_change":"2024-02-12T16:46:03.648726+0000","last_active":"2024-02-12T16:46:35.504747+0000","last_peered":"2024-02-12T16:46:35.504747+0000","last_clean":"2024-02-12T16:46:35.504747+0000","last_became_active":"2024-02-12T16:46:03.608969+0000","last_became_peered":"2024-02-12T16:46:03.608969+0000","last_unstale":"2024-02-12T16:46:35.504747+0000","last_undegraded":"2024-02-12T16:46:35.504747+0000","last_fullsized":"2024-02-12T16:46:35.504747+0000","mapping_epoch":935,"log_start":"933'5300","ondisk_log_start":"933'5300","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11111,"ondisk_log_size":11111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37748736,"num_objects":576,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":297856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2922,"num_bytes_recovered":191496192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":124,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.578614+0000","last_change":"2024-02-12T16:42:41.980681+0000","last_active":"2024-02-12T16:46:30.578614+0000","last_peered":"2024-02-12T16:46:30.578614+0000","last_clean":"2024-02-12T16:46:30.578614+0000","last_became_active":"2024-02-12T16:42:41.980413+0000","last_became_peered":"2024-02-12T16:42:41.980413+0000","last_unstale":"2024-02-12T16:46:30.578614+0000","last_undegraded":"2024-02-12T16:46:30.578614+0000","last_fullsized":"2024-02-12T16:46:30.578614+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:26.611860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:26.611860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"1003'17375","reported_seq":17463,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.265095+0000","last_change":"2024-02-12T16:42:40.991625+0000","last_active":"2024-02-12T16:46:34.265095+0000","last_peered":"2024-02-12T16:46:34.265095+0000","last_clean":"2024-02-12T16:46:34.265095+0000","last_became_active":"2024-02-12T16:42:40.984344+0000","last_became_peered":"2024-02-12T16:42:40.984344+0000","last_unstale":"2024-02-12T16:46:34.265095+0000","last_undegraded":"2024-02-12T16:46:34.265095+0000","last_fullsized":"2024-02-12T16:46:34.265095+0000","mapping_epoch":923,"log_start":"942'7300","ondisk_log_start":"942'7300","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10075,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":86245400,"num_objects":1317,"num_object_clones":0,"num_object_copies":2634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17375,"num_write_kb":598081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":774,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.304398+0000","last_change":"2024-02-12T16:46:33.304776+0000","last_active":"2024-02-12T16:46:33.304398+0000","last_peered":"2024-02-12T16:46:33.304398+0000","last_clean":"2024-02-12T16:46:33.304398+0000","last_became_active":"2024-02-12T16:45:00.039179+0000","last_became_peered":"2024-02-12T16:45:00.039179+0000","last_unstale":"2024-02-12T16:46:33.304398+0000","last_undegraded":"2024-02-12T16:46:33.304398+0000","last_fullsized":"2024-02-12T16:46:33.304398+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:41:25.248423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:41:25.248423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"1005'17785","reported_seq":17886,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:36.720074+0000","last_change":"2024-02-12T16:46:25.666039+0000","last_active":"2024-02-12T16:46:36.720074+0000","last_peered":"2024-02-12T16:46:36.720074+0000","last_clean":"2024-02-12T16:46:36.720074+0000","last_became_active":"2024-02-12T16:42:21.999181+0000","last_became_peered":"2024-02-12T16:42:21.999181+0000","last_unstale":"2024-02-12T16:46:36.720074+0000","last_undegraded":"2024-02-12T16:46:36.720074+0000","last_fullsized":"2024-02-12T16:46:36.720074+0000","mapping_epoch":916,"log_start":"943'7700","ondisk_log_start":"943'7700","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"999'17027","last_scrub_stamp":"2024-02-12T16:46:25.665930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:46:25.665930+0000","log_size":10085,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":2190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17785,"num_write_kb":604160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1116,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.422588+0000","last_change":"2024-02-12T16:42:38.439762+0000","last_active":"2024-02-12T16:46:34.422588+0000","last_peered":"2024-02-12T16:46:34.422588+0000","last_clean":"2024-02-12T16:46:34.422588+0000","last_became_active":"2024-02-12T16:41:18.947424+0000","last_became_peered":"2024-02-12T16:41:18.947424+0000","last_unstale":"2024-02-12T16:46:34.422588+0000","last_undegraded":"2024-02-12T16:46:34.422588+0000","last_fullsized":"2024-02-12T16:46:34.422588+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:38.439663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:39:15.947507+0000","last_clean_scrub_stamp":"2024-02-12T16:42:38.439663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.f","version":"1005'17944","reported_seq":18042,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:37.164310+0000","last_change":"2024-02-12T16:46:21.209862+0000","last_active":"2024-02-12T16:46:37.164310+0000","last_peered":"2024-02-12T16:46:37.164310+0000","last_clean":"2024-02-12T16:46:37.164310+0000","last_became_active":"2024-02-12T16:42:21.963855+0000","last_became_peered":"2024-02-12T16:42:21.963855+0000","last_unstale":"2024-02-12T16:46:37.164310+0000","last_undegraded":"2024-02-12T16:46:37.164310+0000","last_fullsized":"2024-02-12T16:46:37.164310+0000","mapping_epoch":916,"log_start":"944'7900","ondisk_log_start":"944'7900","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10044,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":69337088,"num_objects":1058,"num_object_clones":0,"num_object_copies":2116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17944,"num_write_kb":608064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":448,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.577128+0000","last_change":"2024-02-12T16:45:00.034498+0000","last_active":"2024-02-12T16:46:30.577128+0000","last_peered":"2024-02-12T16:46:30.577128+0000","last_clean":"2024-02-12T16:46:30.577128+0000","last_became_active":"2024-02-12T16:45:00.034042+0000","last_became_peered":"2024-02-12T16:45:00.034042+0000","last_unstale":"2024-02-12T16:46:30.577128+0000","last_undegraded":"2024-02-12T16:46:30.577128+0000","last_fullsized":"2024-02-12T16:46:30.577128+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.543386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.543386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"1003'17755","reported_seq":17842,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.265881+0000","last_change":"2024-02-12T16:42:40.974589+0000","last_active":"2024-02-12T16:46:34.265881+0000","last_peered":"2024-02-12T16:46:34.265881+0000","last_clean":"2024-02-12T16:46:34.265881+0000","last_became_active":"2024-02-12T16:42:40.971979+0000","last_became_peered":"2024-02-12T16:42:40.971979+0000","last_unstale":"2024-02-12T16:46:34.265881+0000","last_undegraded":"2024-02-12T16:46:34.265881+0000","last_fullsized":"2024-02-12T16:46:34.265881+0000","mapping_epoch":923,"log_start":"943'7700","ondisk_log_start":"943'7700","created":916,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10055,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":81199104,"num_objects":1239,"num_object_clones":0,"num_object_copies":2478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17755,"num_write_kb":607808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":740,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.577008+0000","last_change":"2024-02-12T16:43:15.977479+0000","last_active":"2024-02-12T16:46:30.577008+0000","last_peered":"2024-02-12T16:46:30.577008+0000","last_clean":"2024-02-12T16:46:30.577008+0000","last_became_active":"2024-02-12T16:43:15.976997+0000","last_became_peered":"2024-02-12T16:43:15.976997+0000","last_unstale":"2024-02-12T16:46:30.577008+0000","last_undegraded":"2024-02-12T16:46:30.577008+0000","last_fullsized":"2024-02-12T16:46:30.577008+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.923206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:36:57.057484+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.923206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"1003'16675","reported_seq":16777,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.808282+0000","last_change":"2024-02-12T16:46:03.642100+0000","last_active":"2024-02-12T16:46:33.808282+0000","last_peered":"2024-02-12T16:46:33.808282+0000","last_clean":"2024-02-12T16:46:33.808282+0000","last_became_active":"2024-02-12T16:46:03.610670+0000","last_became_peered":"2024-02-12T16:46:03.610670+0000","last_unstale":"2024-02-12T16:46:33.808282+0000","last_undegraded":"2024-02-12T16:46:33.808282+0000","last_fullsized":"2024-02-12T16:46:33.808282+0000","mapping_epoch":916,"log_start":"934'5600","ondisk_log_start":"934'5600","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11075,"ondisk_log_size":11075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42041344,"num_objects":641,"num_object_clones":0,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8825,"num_write_kb":302944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":783,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.577145+0000","last_change":"2024-02-12T16:42:34.813265+0000","last_active":"2024-02-12T16:46:30.577145+0000","last_peered":"2024-02-12T16:46:30.577145+0000","last_clean":"2024-02-12T16:46:30.577145+0000","last_became_active":"2024-02-12T16:28:14.153716+0000","last_became_peered":"2024-02-12T16:28:14.153716+0000","last_unstale":"2024-02-12T16:46:30.577145+0000","last_undegraded":"2024-02-12T16:46:30.577145+0000","last_fullsized":"2024-02-12T16:46:30.577145+0000","mapping_epoch":616,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":617,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:34.813152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:43.068344+0000","last_clean_scrub_stamp":"2024-02-12T16:42:34.813152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"1003'17248","reported_seq":17347,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.865550+0000","last_change":"2024-02-12T16:43:30.452610+0000","last_active":"2024-02-12T16:46:33.865550+0000","last_peered":"2024-02-12T16:46:33.865550+0000","last_clean":"2024-02-12T16:46:33.865550+0000","last_became_active":"2024-02-12T16:42:21.963637+0000","last_became_peered":"2024-02-12T16:42:21.963637+0000","last_unstale":"2024-02-12T16:46:33.865550+0000","last_undegraded":"2024-02-12T16:46:33.865550+0000","last_fullsized":"2024-02-12T16:46:33.865550+0000","mapping_epoch":916,"log_start":"940'7200","ondisk_log_start":"940'7200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"940'6865","last_scrub_stamp":"2024-02-12T16:43:30.452522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:43:30.452522+0000","log_size":10048,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83755008,"num_objects":1278,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17248,"num_write_kb":592832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":603,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.417369+0000","last_change":"2024-02-12T16:45:00.933115+0000","last_active":"2024-02-12T16:46:34.417369+0000","last_peered":"2024-02-12T16:46:34.417369+0000","last_clean":"2024-02-12T16:46:34.417369+0000","last_became_active":"2024-02-12T16:45:00.030971+0000","last_became_peered":"2024-02-12T16:45:00.030971+0000","last_unstale":"2024-02-12T16:46:34.417369+0000","last_undegraded":"2024-02-12T16:46:34.417369+0000","last_fullsized":"2024-02-12T16:46:34.417369+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:45:00.933028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:33.930337+0000","last_clean_scrub_stamp":"2024-02-12T16:45:00.933028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"1005'18009","reported_seq":18105,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:37.171768+0000","last_change":"2024-02-12T16:42:21.964459+0000","last_active":"2024-02-12T16:46:37.171768+0000","last_peered":"2024-02-12T16:46:37.171768+0000","last_clean":"2024-02-12T16:46:37.171768+0000","last_became_active":"2024-02-12T16:42:21.964092+0000","last_became_peered":"2024-02-12T16:42:21.964092+0000","last_unstale":"2024-02-12T16:46:37.171768+0000","last_undegraded":"2024-02-12T16:46:37.171768+0000","last_fullsized":"2024-02-12T16:46:37.171768+0000","mapping_epoch":916,"log_start":"945'8000","ondisk_log_start":"945'8000","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":10009,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67698688,"num_objects":1033,"num_object_clones":0,"num_object_copies":2066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18009,"num_write_kb":609344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1076,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.578165+0000","last_change":"2024-02-12T16:43:31.016133+0000","last_active":"2024-02-12T16:46:30.578165+0000","last_peered":"2024-02-12T16:46:30.578165+0000","last_clean":"2024-02-12T16:46:30.578165+0000","last_became_active":"2024-02-12T16:42:41.321180+0000","last_became_peered":"2024-02-12T16:42:41.321180+0000","last_unstale":"2024-02-12T16:46:30.578165+0000","last_undegraded":"2024-02-12T16:46:30.578165+0000","last_fullsized":"2024-02-12T16:46:30.578165+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:43:31.016013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:43:31.016013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"1005'16681","reported_seq":16786,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:35.429950+0000","last_change":"2024-02-12T16:46:03.649055+0000","last_active":"2024-02-12T16:46:35.429950+0000","last_peered":"2024-02-12T16:46:35.429950+0000","last_clean":"2024-02-12T16:46:35.429950+0000","last_became_active":"2024-02-12T16:46:03.609293+0000","last_became_peered":"2024-02-12T16:46:03.609293+0000","last_unstale":"2024-02-12T16:46:35.429950+0000","last_undegraded":"2024-02-12T16:46:35.429950+0000","last_fullsized":"2024-02-12T16:46:35.429950+0000","mapping_epoch":916,"log_start":"934'5600","ondisk_log_start":"934'5600","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11081,"ondisk_log_size":11081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40402944,"num_objects":616,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8850,"num_write_kb":302944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":495,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.578409+0000","last_change":"2024-02-12T16:42:43.033723+0000","last_active":"2024-02-12T16:46:30.578409+0000","last_peered":"2024-02-12T16:46:30.578409+0000","last_clean":"2024-02-12T16:46:30.578409+0000","last_became_active":"2024-02-12T16:42:40.975171+0000","last_became_peered":"2024-02-12T16:42:40.975171+0000","last_unstale":"2024-02-12T16:46:30.578409+0000","last_undegraded":"2024-02-12T16:46:30.578409+0000","last_fullsized":"2024-02-12T16:46:30.578409+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:43.033576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:43.033576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"1005'16728","reported_seq":16832,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:36.974408+0000","last_change":"2024-02-12T16:46:03.619264+0000","last_active":"2024-02-12T16:46:36.974408+0000","last_peered":"2024-02-12T16:46:36.974408+0000","last_clean":"2024-02-12T16:46:36.974408+0000","last_became_active":"2024-02-12T16:46:03.588876+0000","last_became_peered":"2024-02-12T16:46:03.588876+0000","last_unstale":"2024-02-12T16:46:36.974408+0000","last_undegraded":"2024-02-12T16:46:36.974408+0000","last_fullsized":"2024-02-12T16:46:36.974408+0000","mapping_epoch":916,"log_start":"934'5700","ondisk_log_start":"934'5700","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11028,"ondisk_log_size":11028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39976960,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8875,"num_write_kb":303520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":356,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.414861+0000","last_change":"2024-02-12T16:46:17.567969+0000","last_active":"2024-02-12T16:46:34.414861+0000","last_peered":"2024-02-12T16:46:34.414861+0000","last_clean":"2024-02-12T16:46:34.414861+0000","last_became_active":"2024-02-12T16:45:00.031187+0000","last_became_peered":"2024-02-12T16:45:00.031187+0000","last_unstale":"2024-02-12T16:46:34.414861+0000","last_undegraded":"2024-02-12T16:46:34.414861+0000","last_fullsized":"2024-02-12T16:46:34.414861+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:46:17.567861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:46:17.567861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"1003'16808","reported_seq":16916,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.837744+0000","last_change":"2024-02-12T16:46:03.625625+0000","last_active":"2024-02-12T16:46:33.837744+0000","last_peered":"2024-02-12T16:46:33.837744+0000","last_clean":"2024-02-12T16:46:33.837744+0000","last_became_active":"2024-02-12T16:46:03.592907+0000","last_became_peered":"2024-02-12T16:46:03.592907+0000","last_unstale":"2024-02-12T16:46:33.837744+0000","last_undegraded":"2024-02-12T16:46:33.837744+0000","last_fullsized":"2024-02-12T16:46:33.837744+0000","mapping_epoch":923,"log_start":"934'5700","ondisk_log_start":"934'5700","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11108,"ondisk_log_size":11108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44728320,"num_objects":682,"num_object_clones":0,"num_object_copies":1364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":307040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":255,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.425343+0000","last_change":"2024-02-12T16:45:00.036284+0000","last_active":"2024-02-12T16:46:34.425343+0000","last_peered":"2024-02-12T16:46:34.425343+0000","last_clean":"2024-02-12T16:46:34.425343+0000","last_became_active":"2024-02-12T16:45:00.035400+0000","last_became_peered":"2024-02-12T16:45:00.035400+0000","last_unstale":"2024-02-12T16:46:34.425343+0000","last_undegraded":"2024-02-12T16:46:34.425343+0000","last_fullsized":"2024-02-12T16:46:34.425343+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:51.559264+0000","last_clean_scrub_stamp":"2024-02-12T16:42:51.559264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"1005'16562","reported_seq":16666,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:37.156304+0000","last_change":"2024-02-12T16:46:03.625749+0000","last_active":"2024-02-12T16:46:37.156304+0000","last_peered":"2024-02-12T16:46:37.156304+0000","last_clean":"2024-02-12T16:46:37.156304+0000","last_became_active":"2024-02-12T16:46:03.605891+0000","last_became_peered":"2024-02-12T16:46:03.605891+0000","last_unstale":"2024-02-12T16:46:37.156304+0000","last_undegraded":"2024-02-12T16:46:37.156304+0000","last_fullsized":"2024-02-12T16:46:37.156304+0000","mapping_epoch":916,"log_start":"934'5400","ondisk_log_start":"934'5400","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11162,"ondisk_log_size":11162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34144256,"num_objects":521,"num_object_clones":0,"num_object_copies":1042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":299648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":548,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.463630+0000","last_change":"2024-02-12T16:42:42.373090+0000","last_active":"2024-02-12T16:46:30.463630+0000","last_peered":"2024-02-12T16:46:30.463630+0000","last_clean":"2024-02-12T16:46:30.463630+0000","last_became_active":"2024-02-12T16:42:40.975493+0000","last_became_peered":"2024-02-12T16:42:40.975493+0000","last_unstale":"2024-02-12T16:46:30.463630+0000","last_undegraded":"2024-02-12T16:46:30.463630+0000","last_fullsized":"2024-02-12T16:46:30.463630+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_clean_scrub_stamp":"2024-02-12T16:42:42.373026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"1005'16278","reported_seq":31048,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:36.758789+0000","last_change":"2024-02-12T16:46:30.159762+0000","last_active":"2024-02-12T16:46:36.758789+0000","last_peered":"2024-02-12T16:46:36.758789+0000","last_clean":"2024-02-12T16:46:36.758789+0000","last_became_active":"2024-02-12T16:46:10.702563+0000","last_became_peered":"2024-02-12T16:46:10.702563+0000","last_unstale":"2024-02-12T16:46:36.758789+0000","last_undegraded":"2024-02-12T16:46:36.758789+0000","last_fullsized":"2024-02-12T16:46:36.758789+0000","mapping_epoch":990,"log_start":"933'5200","ondisk_log_start":"933'5200","created":916,"last_epoch_clean":991,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'16037","last_scrub_stamp":"2024-02-12T16:46:30.159637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:46:30.159637+0000","log_size":11078,"ondisk_log_size":11078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33619968,"num_objects":513,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":1063,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.578218+0000","last_change":"2024-02-12T16:43:16.350572+0000","last_active":"2024-02-12T16:46:30.578218+0000","last_peered":"2024-02-12T16:46:30.578218+0000","last_clean":"2024-02-12T16:46:30.578218+0000","last_became_active":"2024-02-12T16:43:16.350353+0000","last_became_peered":"2024-02-12T16:43:16.350353+0000","last_unstale":"2024-02-12T16:46:30.578218+0000","last_undegraded":"2024-02-12T16:46:30.578218+0000","last_fullsized":"2024-02-12T16:46:30.578218+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:48.099448+0000","last_clean_scrub_stamp":"2024-02-12T16:42:48.099448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"1005'16598","reported_seq":18168,"reported_epoch":1005,"state":"active+remapped+backfilling","last_fresh":"2024-02-12T16:46:36.965308+0000","last_change":"2024-02-12T16:46:03.726632+0000","last_active":"2024-02-12T16:46:36.965308+0000","last_peered":"2024-02-12T16:46:36.965308+0000","last_clean":"2024-02-12T16:44:59.014278+0000","last_became_active":"2024-02-12T16:46:03.604945+0000","last_became_peered":"2024-02-12T16:46:03.604945+0000","last_unstale":"2024-02-12T16:46:36.965308+0000","last_undegraded":"2024-02-12T16:46:36.965308+0000","last_fullsized":"2024-02-12T16:46:36.965308+0000","mapping_epoch":965,"log_start":"934'5500","ondisk_log_start":"934'5500","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11098,"ondisk_log_size":11098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37093376,"num_objects":566,"num_object_clones":0,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":158,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8819,"num_write_kb":300320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":35749888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":566}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":733,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.421295+0000","last_change":"2024-02-12T16:42:23.507674+0000","last_active":"2024-02-12T16:46:34.421295+0000","last_peered":"2024-02-12T16:46:34.421295+0000","last_clean":"2024-02-12T16:46:34.421295+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:46:34.421295+0000","last_undegraded":"2024-02-12T16:46:34.421295+0000","last_fullsized":"2024-02-12T16:46:34.421295+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.507539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.507539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"1003'16578","reported_seq":16681,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.866282+0000","last_change":"2024-02-12T16:46:03.624229+0000","last_active":"2024-02-12T16:46:33.866282+0000","last_peered":"2024-02-12T16:46:33.866282+0000","last_clean":"2024-02-12T16:46:33.866282+0000","last_became_active":"2024-02-12T16:46:03.591299+0000","last_became_peered":"2024-02-12T16:46:03.591299+0000","last_unstale":"2024-02-12T16:46:33.866282+0000","last_undegraded":"2024-02-12T16:46:33.866282+0000","last_fullsized":"2024-02-12T16:46:33.866282+0000","mapping_epoch":916,"log_start":"934'5500","ondisk_log_start":"934'5500","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11078,"ondisk_log_size":11078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42893312,"num_objects":654,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":302240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":129,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.423948+0000","last_change":"2024-02-12T16:42:23.724712+0000","last_active":"2024-02-12T16:46:34.423948+0000","last_peered":"2024-02-12T16:46:34.423948+0000","last_clean":"2024-02-12T16:46:34.423948+0000","last_became_active":"2024-02-12T16:41:17.958639+0000","last_became_peered":"2024-02-12T16:41:17.958639+0000","last_unstale":"2024-02-12T16:46:34.423948+0000","last_undegraded":"2024-02-12T16:46:34.423948+0000","last_fullsized":"2024-02-12T16:46:34.423948+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:23.724526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:23.724526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"1005'16586","reported_seq":28539,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:36.842586+0000","last_change":"2024-02-12T16:46:03.655177+0000","last_active":"2024-02-12T16:46:36.842586+0000","last_peered":"2024-02-12T16:46:36.842586+0000","last_clean":"2024-02-12T16:46:36.842586+0000","last_became_active":"2024-02-12T16:46:03.637814+0000","last_became_peered":"2024-02-12T16:46:03.637814+0000","last_unstale":"2024-02-12T16:46:36.842586+0000","last_undegraded":"2024-02-12T16:46:36.842586+0000","last_fullsized":"2024-02-12T16:46:36.842586+0000","mapping_epoch":935,"log_start":"934'5500","ondisk_log_start":"934'5500","created":916,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11086,"ondisk_log_size":11086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37388288,"num_objects":570,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8822,"num_write_kb":300576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2960,"num_bytes_recovered":193986560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":393,"reported_epoch":1002,"state":"active+clean","last_fresh":"2024-02-12T16:46:30.463397+0000","last_change":"2024-02-12T16:45:00.034722+0000","last_active":"2024-02-12T16:46:30.463397+0000","last_peered":"2024-02-12T16:46:30.463397+0000","last_clean":"2024-02-12T16:46:30.463397+0000","last_became_active":"2024-02-12T16:45:00.034337+0000","last_became_peered":"2024-02-12T16:45:00.034337+0000","last_unstale":"2024-02-12T16:46:30.463397+0000","last_undegraded":"2024-02-12T16:46:30.463397+0000","last_fullsized":"2024-02-12T16:46:30.463397+0000","mapping_epoch":964,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":965,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:30.755995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:25:03.361757+0000","last_clean_scrub_stamp":"2024-02-12T16:42:30.755995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"1005'16371","reported_seq":28172,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-02-12T16:46:35.476361+0000","last_change":"2024-02-12T16:46:03.703922+0000","last_active":"2024-02-12T16:46:35.476361+0000","last_peered":"2024-02-12T16:46:35.476361+0000","last_clean":"2024-02-12T16:46:35.476361+0000","last_became_active":"2024-02-12T16:46:03.650004+0000","last_became_peered":"2024-02-12T16:46:03.650004+0000","last_unstale":"2024-02-12T16:46:35.476361+0000","last_undegraded":"2024-02-12T16:46:35.476361+0000","last_fullsized":"2024-02-12T16:46:35.476361+0000","mapping_epoch":983,"log_start":"933'5300","ondisk_log_start":"933'5300","created":983,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11071,"ondisk_log_size":11071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40370176,"num_objects":616,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":297856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2922,"num_bytes_recovered":191496192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":297,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.466182+0000","last_change":"2024-02-12T16:46:33.468310+0000","last_active":"2024-02-12T16:46:33.466182+0000","last_peered":"2024-02-12T16:46:33.466182+0000","last_clean":"2024-02-12T16:46:33.466182+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:46:33.466182+0000","last_undegraded":"2024-02-12T16:46:33.466182+0000","last_fullsized":"2024-02-12T16:46:33.466182+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:22.106269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:40:18.349211+0000","last_clean_scrub_stamp":"2024-02-12T16:42:22.106269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.19","version":"1003'16674","reported_seq":16775,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:33.750819+0000","last_change":"2024-02-12T16:46:03.686626+0000","last_active":"2024-02-12T16:46:33.750819+0000","last_peered":"2024-02-12T16:46:33.750819+0000","last_clean":"2024-02-12T16:46:33.750819+0000","last_became_active":"2024-02-12T16:46:03.649059+0000","last_became_peered":"2024-02-12T16:46:03.649059+0000","last_unstale":"2024-02-12T16:46:33.750819+0000","last_undegraded":"2024-02-12T16:46:33.750819+0000","last_fullsized":"2024-02-12T16:46:33.750819+0000","mapping_epoch":983,"log_start":"934'5600","ondisk_log_start":"934'5600","created":983,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:20.941655+0000","last_clean_scrub_stamp":"2024-02-12T16:42:20.941655+0000","log_size":11074,"ondisk_log_size":11074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42106880,"num_objects":643,"num_object_clones":0,"num_object_copies":1286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8825,"num_write_kb":302944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":314,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-02-12T16:46:32.973806+0000","last_change":"2024-02-12T16:46:32.974203+0000","last_active":"2024-02-12T16:46:32.973806+0000","last_peered":"2024-02-12T16:46:32.973806+0000","last_clean":"2024-02-12T16:46:32.973806+0000","last_became_active":"2024-02-12T16:43:15.981046+0000","last_became_peered":"2024-02-12T16:43:15.981046+0000","last_unstale":"2024-02-12T16:46:32.973806+0000","last_undegraded":"2024-02-12T16:46:32.973806+0000","last_fullsized":"2024-02-12T16:46:32.973806+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:35.918516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:13.754017+0000","last_clean_scrub_stamp":"2024-02-12T16:42:35.918516+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":612,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.420605+0000","last_change":"2024-02-12T16:46:19.207678+0000","last_active":"2024-02-12T16:46:34.420605+0000","last_peered":"2024-02-12T16:46:34.420605+0000","last_clean":"2024-02-12T16:46:34.420605+0000","last_became_active":"2024-02-12T16:35:16.899034+0000","last_became_peered":"2024-02-12T16:35:16.899034+0000","last_unstale":"2024-02-12T16:46:34.420605+0000","last_undegraded":"2024-02-12T16:46:34.420605+0000","last_fullsized":"2024-02-12T16:46:34.420605+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:46:19.207604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:46:19.207604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":564,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-02-12T16:46:34.415437+0000","last_change":"2024-02-12T16:42:25.380280+0000","last_active":"2024-02-12T16:46:34.415437+0000","last_peered":"2024-02-12T16:46:34.415437+0000","last_clean":"2024-02-12T16:46:34.415437+0000","last_became_active":"2024-02-12T16:41:17.950595+0000","last_became_peered":"2024-02-12T16:41:17.950595+0000","last_unstale":"2024-02-12T16:46:34.415437+0000","last_undegraded":"2024-02-12T16:46:34.415437+0000","last_fullsized":"2024-02-12T16:46:34.415437+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:42:25.380188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:42:25.380188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":1235910680,"num_objects":18860,"num_object_clones":0,"num_object_copies":37720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1555,"num_objects_unfound":0,"num_objects_dirty":18860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":274054,"num_write_kb":9349025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18794,"num_bytes_recovered":1231585280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3572899840,"data_stored":3572891696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282212,"ondisk_log_size":282212,"up":52,"acting":52,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431959110,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1272264,"kb_used_data":371328,"kb_used_omap":0,"kb_used_meta":900928,"kb_avail":92472888,"statfs":{"total":95995035648,"available":94692237312,"internally_reserved":0,"allocated":380239872,"data_stored":379653966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922550272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.66400000000000003,"15min":1.7370000000000001},"min":{"1min":0.312,"5min":0.248,"15min":0.248},"max":{"1min":0.93899999999999995,"5min":3.9159999999999999,"15min":124.437},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.1759999999999999,"15min":2.0190000000000001},"min":{"1min":0.39700000000000002,"5min":0.33500000000000002,"15min":0.29199999999999998},"max":{"1min":4.4219999999999997,"5min":5.7009999999999996,"15min":126.926},"last":0.91600000000000004}]},{"osd":1,"last update":"Mon Feb 12 16:41:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.282,"15min":1.054},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":8.8450000000000006,"5min":8.8450000000000006,"15min":8.9849999999999994},"last":1.131},{"interface":"front","average":{"1min":1.117,"5min":1.1080000000000001,"15min":0.92700000000000005},"min":{"1min":0.33100000000000002,"5min":0.29899999999999999,"15min":0.28499999999999998},"max":{"1min":4.391,"5min":5.3949999999999996,"15min":11.167},"last":1.3180000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.1240000000000001,"15min":1.8560000000000001},"min":{"1min":0.36399999999999999,"5min":0.308,"15min":0.308},"max":{"1min":4.452,"5min":5.6669999999999998,"15min":219.56100000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.74099999999999999,"15min":1.7490000000000001},"min":{"1min":0.40600000000000003,"5min":0.28699999999999998,"15min":0.26900000000000002},"max":{"1min":2.605,"5min":2.968,"15min":219.79300000000001},"last":0.88}]},{"osd":3,"last update":"Mon Feb 12 16:45:39 2024","interfaces":[{"interface":"back","average":{"1min":4.9969999999999999,"5min":1.9379999999999999,"15min":2.0640000000000001},"min":{"1min":0.376,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":46.445,"5min":46.445,"15min":121.169},"last":0.85999999999999999},{"interface":"front","average":{"1min":4.6719999999999997,"5min":1.8149999999999999,"15min":2.0169999999999999},"min":{"1min":0.36899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":46.774999999999999,"5min":46.774999999999999,"15min":121.181},"last":1.347}]},{"osd":4,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.93300000000000005,"15min":0.91800000000000004},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.4239999999999999,"5min":6.1669999999999998,"15min":6.2039999999999997},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.304,"5min":1.234,"15min":1.121},"min":{"1min":0.48099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.4989999999999997,"5min":5.7469999999999999,"15min":7.3920000000000003},"last":0.96899999999999997}]},{"osd":5,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.208,"15min":1.153},"min":{"1min":0.52200000000000002,"5min":0.38600000000000001,"15min":0.34100000000000003},"max":{"1min":3.8519999999999999,"5min":5.1280000000000001,"15min":5.5289999999999999},"last":1.244},{"interface":"front","average":{"1min":1.417,"5min":1.2470000000000001,"15min":1.2330000000000001},"min":{"1min":0.503,"5min":0.38300000000000001,"15min":0.33100000000000002},"max":{"1min":4.5199999999999996,"5min":5.8200000000000003,"15min":7.4249999999999998},"last":1.113}]},{"osd":6,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1599999999999999,"15min":1.139},"min":{"1min":0.40300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.4780000000000002,"5min":6.202,"15min":6.4160000000000004},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.198,"15min":1.2809999999999999},"min":{"1min":0.49399999999999999,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":3.512,"5min":6.1289999999999996,"15min":9.5389999999999997},"last":1.1990000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:45:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.29,"15min":1.171},"min":{"1min":0.49399999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.79,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":1.272},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.92800000000000005,"15min":0.89900000000000002},"min":{"1min":0.57299999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.3759999999999999,"5min":5.0739999999999998,"15min":5.0739999999999998},"last":1.4179999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.159,"15min":1.0669999999999999},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":3.5409999999999999,"5min":6.306,"15min":6.4459999999999997},"last":1.3720000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.93500000000000005,"15min":1.0820000000000001},"min":{"1min":0.53700000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.7530000000000001,"5min":3.7530000000000001,"15min":5.4880000000000004},"last":1.464}]},{"osd":9,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.0169999999999999,"15min":1.194},"min":{"1min":0.64000000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.206,"5min":3.1480000000000001,"15min":7.5709999999999997},"last":1.506},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.075,"15min":1.1799999999999999},"min":{"1min":0.67800000000000005,"5min":0.496,"15min":0.44700000000000001},"max":{"1min":1.3109999999999999,"5min":3.21,"15min":5.7089999999999996},"last":1.55}]},{"osd":10,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.506,"15min":1.333},"min":{"1min":0.63300000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":4.3780000000000001,"5min":9.6159999999999997,"15min":9.6159999999999997},"last":1.222},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.508,"15min":1.306},"min":{"1min":0.52000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":8.2129999999999992,"5min":8.2129999999999992,"15min":8.2129999999999992},"last":1.3240000000000001}]}]},{"osd":10,"up_from":83,"seq":356482286158,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1333564,"kb_used_data":443188,"kb_used_omap":0,"kb_used_meta":890368,"kb_avail":92411588,"statfs":{"total":95995035648,"available":94629466112,"internally_reserved":0,"allocated":453824512,"data_stored":453244500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911736832},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.99399999999999999,"15min":1.196},"min":{"1min":0.28100000000000003,"5min":0.247,"15min":0.247},"max":{"1min":2.9199999999999999,"5min":4.7140000000000004,"15min":111.935},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88400000000000001,"15min":1.274},"min":{"1min":0.29899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.641,"5min":4.3920000000000003,"15min":111.316},"last":1.353}]},{"osd":1,"last update":"Mon Feb 12 16:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":0.871,"15min":1.0669999999999999},"min":{"1min":0.38600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.524,"5min":5.524,"15min":5.524},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.29,"5min":0.999,"15min":1.113},"min":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.5609999999999999,"5min":5.5609999999999999,"15min":7.2000000000000002},"last":0.97399999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.012,"15min":1.341},"min":{"1min":0.36699999999999999,"5min":0.314,"15min":0.28100000000000003},"max":{"1min":3.9020000000000001,"5min":9.4179999999999993,"15min":106.822},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.85199999999999998,"15min":1.3440000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":5.5599999999999996,"5min":5.5599999999999996,"15min":107.88800000000001},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:45:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1459999999999999,"15min":1.482},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27400000000000002},"max":{"1min":3.9550000000000001,"5min":9.3699999999999992,"15min":107.014},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.0549999999999999,"15min":1.429},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.27900000000000003},"max":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":110.80800000000001},"last":0.76500000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.0740000000000001,"15min":1.022},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":5.8380000000000001,"5min":5.8380000000000001,"15min":6.3689999999999998},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":0.89700000000000002,"15min":0.96599999999999997},"min":{"1min":0.51900000000000002,"5min":0.38600000000000001,"15min":0.318},"max":{"1min":5.649,"5min":5.649,"15min":5.649},"last":1.474}]},{"osd":5,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.903,"15min":1.284},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.35399999999999998},"max":{"1min":5.734,"5min":74.137,"15min":74.137},"last":1.083},{"interface":"front","average":{"1min":1.47,"5min":1.8029999999999999,"15min":1.421},"min":{"1min":0.55500000000000005,"5min":0.44,"15min":0.41299999999999998},"max":{"1min":5.9249999999999998,"5min":74.063999999999993,"15min":74.063999999999993},"last":1.5569999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1759999999999999,"15min":2.5870000000000002},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.36099999999999999},"max":{"1min":5.7729999999999997,"5min":5.7729999999999997,"15min":460.90199999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.0640000000000001,"15min":2.573},"min":{"1min":0.61799999999999999,"5min":0.44800000000000001,"15min":0.44},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":460.93299999999999},"last":1.3320000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:45:45 2024","interfaces":[{"interface":"back","average":{"1min":1.7969999999999999,"5min":3.6669999999999998,"15min":4.7229999999999999},"min":{"1min":0.57099999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.8739999999999997,"5min":74.007000000000005,"15min":74.007000000000005},"last":1.641},{"interface":"front","average":{"1min":1.0680000000000001,"5min":3.4089999999999998,"15min":4.3929999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":3.7170000000000001,"5min":73.819000000000003,"15min":73.819000000000003},"last":1.107}]},{"osd":8,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.1819999999999999,"15min":1.1879999999999999},"min":{"1min":0.60899999999999999,"5min":0.42199999999999999,"15min":0.35499999999999998},"max":{"1min":7.5810000000000004,"5min":7.5810000000000004,"15min":8.4420000000000002},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.3560000000000001,"15min":1.2310000000000001},"min":{"1min":0.45600000000000002,"5min":0.41299999999999998,"15min":0.36299999999999999},"max":{"1min":3.859,"5min":9.5090000000000003,"15min":9.5090000000000003},"last":1.2090000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.212,"15min":1.1299999999999999},"min":{"1min":0.51800000000000002,"5min":0.35499999999999998,"15min":0.34699999999999998},"max":{"1min":5.8029999999999999,"5min":5.8029999999999999,"15min":6.2560000000000002},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.101,"15min":1.149},"min":{"1min":0.67300000000000004,"5min":0.378,"15min":0.378},"max":{"1min":5.7519999999999998,"5min":5.7519999999999998,"15min":6.0030000000000001},"last":1.7869999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:46:06 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.4119999999999999,"15min":1.3839999999999999},"min":{"1min":0.40500000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":5.2880000000000003,"5min":7.8289999999999997,"15min":8.7569999999999997},"last":1.2949999999999999},{"interface":"front","average":{"1min":2.0790000000000002,"5min":1.4339999999999999,"15min":1.3500000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.38100000000000001},"max":{"1min":5.7770000000000001,"5min":7.891,"15min":7.891},"last":1.8799999999999999}]}]},{"osd":9,"up_from":0,"seq":317827580503,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":8,"up_from":65,"seq":279172874848,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061756,"kb_used_data":176436,"kb_used_omap":0,"kb_used_meta":885312,"kb_avail":92683396,"statfs":{"total":95995035648,"available":94907797504,"internally_reserved":0,"allocated":180670464,"data_stored":180090452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":906559488},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.624,"15min":1.161},"min":{"1min":0.35899999999999999,"5min":0.27500000000000002,"15min":0.23899999999999999},"max":{"1min":2.4460000000000002,"5min":4.4189999999999996,"15min":134.01499999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.80700000000000005,"15min":1.335},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.28799999999999998},"max":{"1min":2.5699999999999998,"5min":6.7729999999999997,"15min":133.667},"last":0.504}]},{"osd":1,"last update":"Mon Feb 12 16:46:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.93300000000000005,"15min":0.95899999999999996},"min":{"1min":0.308,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.96899999999999997,"5min":8.1210000000000004,"15min":8.1210000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.88500000000000001,"15min":0.77600000000000002},"min":{"1min":0.318,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.2270000000000003,"5min":8.4819999999999993,"15min":8.4819999999999993},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":1.9399999999999999,"5min":3.2320000000000002,"15min":8.6430000000000007},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2609999999999999,"15min":1.091},"min":{"1min":0.48999999999999999,"5min":0.378,"15min":0.26800000000000002},"max":{"1min":2.0070000000000001,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.84099999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.008,"15min":1.1739999999999999},"min":{"1min":0.38100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":10.231,"5min":10.231,"15min":25.919},"last":0.624},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.0840000000000001,"15min":1.2749999999999999},"min":{"1min":0.39600000000000002,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":10.265000000000001,"5min":10.265000000000001,"15min":25.984999999999999},"last":0.57999999999999996}]},{"osd":4,"last update":"Mon Feb 12 16:45:57 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.93300000000000005,"15min":0.89100000000000001},"min":{"1min":0.438,"5min":0.313,"15min":0.313},"max":{"1min":3.0960000000000001,"5min":3.3620000000000001,"15min":8.609},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.93799999999999994,"15min":1.0620000000000001},"min":{"1min":0.48699999999999999,"5min":0.38200000000000001,"15min":0.37},"max":{"1min":2.4449999999999998,"5min":4.4500000000000002,"15min":9.0890000000000004},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Feb 12 16:45:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.85699999999999998,"15min":0.98799999999999999},"min":{"1min":0.54300000000000004,"5min":0.38,"15min":0.318},"max":{"1min":3.3999999999999999,"5min":4.5069999999999997,"15min":7.2960000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.2470000000000001,"15min":1.276},"min":{"1min":0.499,"5min":0.42399999999999999,"15min":0.35199999999999998},"max":{"1min":6.2869999999999999,"5min":6.681,"15min":9.0540000000000003},"last":0.63700000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.1020000000000001,"15min":1.0880000000000001},"min":{"1min":0.48599999999999999,"5min":0.432,"15min":0.38200000000000001},"max":{"1min":3.177,"5min":5.5369999999999999,"15min":8.5220000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.98599999999999999,"15min":1.2310000000000001},"min":{"1min":0.505,"5min":0.41399999999999998,"15min":0.34599999999999997},"max":{"1min":2.5179999999999998,"5min":4.5140000000000002,"15min":9.1500000000000004},"last":0.60899999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.167,"15min":1.1850000000000001},"min":{"1min":0.42199999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.5419999999999998,"5min":6.5970000000000004,"15min":6.5970000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.02,"15min":0.99399999999999999},"min":{"1min":0.61899999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.1200000000000001,"5min":3.8530000000000002,"15min":3.8530000000000002},"last":0.751}]},{"osd":9,"last update":"Mon Feb 12 16:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.96699999999999997,"15min":1.1659999999999999},"min":{"1min":0.48499999999999999,"5min":0.40600000000000003,"15min":0.371},"max":{"1min":1.425,"5min":4.407,"15min":9.1170000000000009},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.0760000000000001,"15min":1.2709999999999999},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.47099999999999997},"max":{"1min":1.4610000000000001,"5min":4.5380000000000003,"15min":9.1980000000000004},"last":0.79700000000000004}]},{"osd":10,"last update":"Mon Feb 12 16:46:21 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.97999999999999998,"15min":1.0409999999999999},"min":{"1min":0.42799999999999999,"5min":0.39500000000000002,"15min":0.39000000000000001},"max":{"1min":1.484,"5min":4.5289999999999999,"15min":8.452},"last":1.254},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.0620000000000001,"15min":1.127},"min":{"1min":0.55600000000000005,"5min":0.439,"15min":0.40100000000000002},"max":{"1min":3.2040000000000002,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":1.2649999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:46:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.204,"15min":1.1419999999999999},"min":{"1min":0.503,"5min":0.437,"15min":0.42699999999999999},"max":{"1min":6.1699999999999999,"5min":6.1699999999999999,"15min":8.6790000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.6890000000000001,"5min":1.2450000000000001,"15min":1.1990000000000001},"min":{"1min":0.629,"5min":0.46500000000000002,"15min":0.44600000000000001},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.66200000000000003}]}]},{"osd":7,"up_from":923,"seq":3964254814256,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626496,"kb_used_data":303928,"kb_used_omap":0,"kb_used_meta":322559,"kb_avail":93118656,"statfs":{"total":95995035648,"available":95353503744,"internally_reserved":0,"allocated":311222272,"data_stored":310638188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":330300468},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.4630000000000001,"5min":6.4630000000000001,"15min":6.4630000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.83899999999999997,"15min":0.85799999999999998},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.8580000000000001,"5min":5.2110000000000003,"15min":5.2110000000000003},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":10.833,"5min":2.9830000000000001,"15min":1.6930000000000001},"min":{"1min":0.377,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":177.82400000000001,"5min":177.82400000000001,"15min":177.82400000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":10.349,"5min":2.7160000000000002,"15min":1.284},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":177.30500000000001,"5min":177.30500000000001,"15min":177.30500000000001},"last":0.313}]},{"osd":2,"last update":"Mon Feb 12 16:46:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.0229999999999999,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.98899999999999999,"15min":1.042},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.9039999999999999,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.145,"15min":1.1639999999999999},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.8820000000000001,"5min":5.7389999999999999,"15min":5.7389999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.157,"15min":1.153},"min":{"1min":0.53800000000000003,"5min":0.443,"15min":0.443},"max":{"1min":3.9249999999999998,"5min":5.7720000000000002,"15min":5.7720000000000002},"last":0.71099999999999997}]},{"osd":4,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.69999999999999996,"15min":0.67500000000000004},"min":{"1min":0.39000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.98999999999999999,"5min":5.4199999999999999,"15min":5.4199999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.93600000000000005,"15min":0.78400000000000003},"min":{"1min":0.374,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.4119999999999999,"5min":8.4719999999999995,"15min":8.4719999999999995},"last":0.77100000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.1319999999999999,"15min":1.0249999999999999},"min":{"1min":0.46999999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.3480000000000001,"15min":1.46},"min":{"1min":0.505,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.766,"5min":5.7069999999999999,"15min":5.7069999999999999},"last":0.997}]},{"osd":6,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.92500000000000004,"15min":0.82499999999999996},"min":{"1min":0.51800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.5069999999999997,"5min":6.5069999999999997,"15min":6.5069999999999997},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.877,"15min":0.88300000000000001},"min":{"1min":0.46800000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86899999999999999,"5min":5.508,"15min":5.508},"last":0.48999999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.008,"15min":1.028},"min":{"1min":0.64200000000000002,"5min":0.437,"15min":0.437},"max":{"1min":1.0680000000000001,"5min":5.6719999999999997,"15min":5.6719999999999997},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0289999999999999,"15min":0.94699999999999995},"min":{"1min":0.57199999999999995,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":6.5880000000000001,"5min":6.5880000000000001,"15min":6.5880000000000001},"last":0.64500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":10,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.0940000000000001,"15min":1.1519999999999999},"min":{"1min":0.52400000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.871,"5min":5.6559999999999997,"15min":5.6559999999999997},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.1220000000000001,"5min":0.98499999999999999,"15min":0.871},"min":{"1min":0.55900000000000005,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.5650000000000004},"last":0.57099999999999995}]},{"osd":11,"last update":"Mon Feb 12 16:45:44 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.3140000000000001,"15min":1.173},"min":{"1min":0.73099999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":6.5410000000000004,"5min":9.7409999999999997,"15min":9.7409999999999997},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.02,"5min":1.304,"15min":1.3999999999999999},"min":{"1min":0.57299999999999995,"5min":0.437,"15min":0.437},"max":{"1min":4.9169999999999998,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":0.61199999999999999}]}]},{"osd":6,"up_from":0,"seq":206158430833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575510,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1716248,"kb_used_data":521360,"kb_used_omap":0,"kb_used_meta":1194880,"kb_avail":92028904,"statfs":{"total":95995035648,"available":94237597696,"internally_reserved":0,"allocated":533872640,"data_stored":533276697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1223557120},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:46:08 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":0.96899999999999997,"15min":1.014},"min":{"1min":0.33900000000000002,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":7.5010000000000003,"5min":7.532,"15min":10.420999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.753,"15min":0.94399999999999995},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.254},"max":{"1min":2.835,"5min":7.3540000000000001,"15min":10.26},"last":0.56599999999999995}]},{"osd":2,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":0.91800000000000004,"15min":1.105},"min":{"1min":0.30199999999999999,"5min":0.29399999999999998,"15min":0.22700000000000001},"max":{"1min":7.54,"5min":7.54,"15min":10.755000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.85099999999999998,"15min":1.0700000000000001},"min":{"1min":0.26000000000000001,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":1.113,"5min":7.9539999999999997,"15min":9.702},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":0.98899999999999999,"15min":1.1679999999999999},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":4.1820000000000004,"5min":8.9920000000000009,"15min":26.664000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.466,"5min":1.113,"15min":1.389},"min":{"1min":0.50700000000000001,"5min":0.34599999999999997,"15min":0.26200000000000001},"max":{"1min":5.2080000000000002,"5min":8.9610000000000003,"15min":27.148},"last":1.133}]},{"osd":4,"last update":"Mon Feb 12 16:45:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":0.94899999999999995,"15min":0.98999999999999999},"min":{"1min":0.375,"5min":0.34699999999999998,"15min":0.312},"max":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.6029999999999998},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.341,"5min":1.0129999999999999,"15min":1.0860000000000001},"min":{"1min":0.47699999999999998,"5min":0.372,"15min":0.312},"max":{"1min":7.7149999999999999,"5min":7.7149999999999999,"15min":7.7149999999999999},"last":1.107}]},{"osd":5,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.0409999999999999,"15min":2.9990000000000001},"min":{"1min":0.33900000000000002,"5min":0.32100000000000001,"15min":0.309},"max":{"1min":1.4079999999999999,"5min":8.0190000000000001,"15min":524.85299999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1890000000000001,"15min":3.0939999999999999},"min":{"1min":0.52700000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.2599999999999998,"5min":8.7050000000000001,"15min":519.89700000000005},"last":1.2569999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.151,"15min":1.254},"min":{"1min":0.437,"5min":0.40300000000000002,"15min":0.39000000000000001},"max":{"1min":7.9450000000000003,"5min":7.9450000000000003,"15min":7.9450000000000003},"last":1.298},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.2190000000000001,"15min":1.3939999999999999},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.927,"5min":8.7669999999999995,"15min":8.7669999999999995},"last":1.0680000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:38:59 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.0629999999999999,"15min":3.2370000000000001},"min":{"1min":0.44700000000000001,"5min":0.39200000000000002,"15min":0.34999999999999998},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":522.79200000000003},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.25,"15min":3.198},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.35099999999999998},"max":{"1min":4.3360000000000003,"5min":9.0069999999999997,"15min":520.64499999999998},"last":1.016}]},{"osd":8,"last update":"Mon Feb 12 16:46:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.389,"15min":1.331},"min":{"1min":0.61199999999999999,"5min":0.48499999999999999,"15min":0.379},"max":{"1min":2.968,"5min":8.8100000000000005,"15min":9.1430000000000007},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.4770000000000001,"15min":1.411},"min":{"1min":0.64400000000000002,"5min":0.54300000000000004,"15min":0.42499999999999999},"max":{"1min":3.0430000000000001,"5min":8.9269999999999996,"15min":9.1739999999999995},"last":1.3200000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.101,"15min":1.1899999999999999},"min":{"1min":0.441,"5min":0.41199999999999998,"15min":0.38400000000000001},"max":{"1min":4.6239999999999997,"5min":8.1539999999999999,"15min":8.3949999999999996},"last":1.238},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1839999999999999,"15min":1.28},"min":{"1min":0.60499999999999998,"5min":0.48099999999999998,"15min":0.45300000000000001},"max":{"1min":4.766,"5min":8.2409999999999997,"15min":8.4969999999999999},"last":1.1659999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:46:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.3009999999999999,"15min":1.3200000000000001},"min":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.42999999999999999},"max":{"1min":4.7220000000000004,"5min":8.1140000000000008,"15min":8.3469999999999995},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.462,"15min":1.4039999999999999},"min":{"1min":0.55100000000000005,"5min":0.51200000000000001,"15min":0.47999999999999998},"max":{"1min":6.6059999999999999,"5min":8.8469999999999995,"15min":9.0440000000000005},"last":1.3620000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:46:00 2024","interfaces":[{"interface":"back","average":{"1min":2.4060000000000001,"5min":1.77,"15min":1.8140000000000001},"min":{"1min":0.66000000000000003,"5min":0.53800000000000003,"15min":0.47099999999999997},"max":{"1min":5.423,"5min":8.8849999999999998,"15min":88.679000000000002},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.242,"15min":1.607},"min":{"1min":0.60399999999999998,"5min":0.51800000000000002,"15min":0.46100000000000002},"max":{"1min":4.2169999999999996,"5min":8.0730000000000004,"15min":88.519000000000005},"last":0.91600000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739042,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888084,"kb_used_data":1164,"kb_used_omap":0,"kb_used_meta":886912,"kb_avail":92857068,"statfs":{"total":95995035648,"available":95085637632,"internally_reserved":0,"allocated":1191936,"data_stored":600065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":908197888},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":2.1259999999999999,"15min":1.371},"min":{"1min":0.27700000000000002,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":1.766,"5min":118.657,"15min":118.657},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":2.2080000000000002,"15min":1.5660000000000001},"min":{"1min":0.34399999999999997,"5min":0.29699999999999999,"15min":0.23400000000000001},"max":{"1min":3.448,"5min":118.58,"15min":118.58},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:46:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.68500000000000005,"15min":0.86799999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.82599999999999996,"5min":4.0999999999999996,"15min":4.0999999999999996},"last":1.79},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.73699999999999999,"15min":0.86699999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0189999999999999,"5min":4.0469999999999997,"15min":4.0469999999999997},"last":0.60899999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:46:18 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":2.9769999999999999,"15min":3.0139999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.252},"max":{"1min":3.3159999999999998,"5min":193.114,"15min":377.06299999999999},"last":1.2889999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.9020000000000001,"15min":3.1440000000000001},"min":{"1min":0.308,"5min":0.26600000000000001,"15min":0.23799999999999999},"max":{"1min":1.8109999999999999,"5min":193.22,"15min":379.18000000000001},"last":0.80900000000000005}]},{"osd":4,"last update":"Mon Feb 12 16:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72899999999999998,"15min":0.95899999999999996},"min":{"1min":0.44500000000000001,"5min":0.35699999999999998,"15min":0.29099999999999998},"max":{"1min":2.0699999999999998,"5min":2.21,"15min":7.2869999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.79300000000000004,"15min":1.077},"min":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.29499999999999998},"max":{"1min":1.2789999999999999,"5min":3.9580000000000002,"15min":8.4949999999999992},"last":1.456}]},{"osd":5,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.90200000000000002,"15min":1.698},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.30399999999999999},"max":{"1min":1.752,"5min":8.532,"15min":181.46199999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.0269999999999999,"15min":1.8200000000000001},"min":{"1min":0.47099999999999997,"5min":0.41299999999999998,"15min":0.32800000000000001},"max":{"1min":1.8109999999999999,"5min":8.6519999999999992,"15min":186.41999999999999},"last":1.0089999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.76100000000000001,"15min":0.98499999999999999},"min":{"1min":0.38400000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.5620000000000001,"5min":3.1749999999999998,"15min":6.6950000000000003},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.80200000000000005,"15min":1.1459999999999999},"min":{"1min":0.47299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4219999999999999,"5min":2.1779999999999999,"15min":2.1779999999999999},"last":1.3700000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:39:13 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.24,"15min":2.706},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.374},"max":{"1min":1.835,"5min":6.1130000000000004,"15min":191.47200000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4019999999999999,"15min":2.823},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":1.897,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.081}]},{"osd":8,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93899999999999995,"15min":1.1850000000000001},"min":{"1min":0.48899999999999999,"5min":0.442,"15min":0.41299999999999998},"max":{"1min":3.5049999999999999,"5min":6.7400000000000002,"15min":6.7400000000000002},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.048,"15min":1.288},"min":{"1min":0.61399999999999999,"5min":0.51600000000000001,"15min":0.48299999999999998},"max":{"1min":3.5339999999999998,"5min":6.8570000000000002,"15min":6.8570000000000002},"last":1.1359999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.91200000000000003,"15min":1.1850000000000001},"min":{"1min":0.45800000000000002,"5min":0.42299999999999999,"15min":0.34999999999999998},"max":{"1min":1.4430000000000001,"5min":4.1130000000000004,"15min":8.4629999999999992},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.86099999999999999,"15min":1.2370000000000001},"min":{"1min":0.52800000000000002,"5min":0.47499999999999998,"15min":0.41499999999999998},"max":{"1min":1.393,"5min":3.25,"15min":12.943},"last":1.0429999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:46:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.018,"15min":1.1890000000000001},"min":{"1min":0.58999999999999997,"5min":0.46100000000000002,"15min":0.44},"max":{"1min":6.0880000000000001,"5min":6.0880000000000001,"15min":13.157},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.0740000000000001,"15min":1.262},"min":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.48299999999999998},"max":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":13.238},"last":1.532}]},{"osd":11,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.218,"15min":1.403},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.40300000000000002},"max":{"1min":3.726,"5min":5.8079999999999998,"15min":8.4280000000000008},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0820000000000001,"15min":1.2310000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.47299999999999998},"max":{"1min":3.6819999999999999,"5min":4.9960000000000004,"15min":7.3799999999999999},"last":1.496}]}]},{"osd":2,"up_from":21,"seq":90194313872,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594964,"kb_used_data":1164,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93150188,"statfs":{"total":95995035648,"available":95385792512,"internally_reserved":0,"allocated":1191936,"data_stored":600065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.56899999999999995,"15min":0.751},"min":{"1min":0.29199999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.94299999999999995,"5min":3.996,"15min":3.996},"last":0.37},{"interface":"front","average":{"1min":0.439,"5min":0.66800000000000004,"15min":0.77500000000000002},"min":{"1min":0.29099999999999998,"5min":0.25900000000000001,"15min":0.23400000000000001},"max":{"1min":0.747,"5min":4.9690000000000003,"15min":4.9690000000000003},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:46:07 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":1.425,"15min":1.1140000000000001},"min":{"1min":0.34000000000000002,"5min":0.28899999999999998,"15min":0.19500000000000001},"max":{"1min":34.843000000000004,"5min":34.843000000000004,"15min":34.843000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":2.4209999999999998,"5min":1.4319999999999999,"15min":1.3049999999999999},"min":{"1min":0.33700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":34.935000000000002,"5min":34.935000000000002,"15min":34.935000000000002},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":2.1120000000000001,"15min":1.712},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":2.7639999999999998,"5min":113.68000000000001,"15min":113.68000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":2.2170000000000001,"15min":1.722},"min":{"1min":0.40400000000000003,"5min":0.38100000000000001,"15min":0.25800000000000001},"max":{"1min":2.7970000000000002,"5min":113.815,"15min":113.815},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Feb 12 16:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.93799999999999994},"min":{"1min":0.312,"5min":0.309,"15min":0.29499999999999998},"max":{"1min":1.218,"5min":3.0470000000000002,"15min":8.3439999999999994},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.80900000000000005,"15min":1.0409999999999999},"min":{"1min":0.41399999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":1.24,"5min":6.9409999999999998,"15min":7.0019999999999998},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:46:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.86299999999999999,"15min":0.92500000000000004},"min":{"1min":0.372,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":4.1820000000000004,"5min":5.1319999999999997,"15min":8.7509999999999994},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.82999999999999996,"15min":1.0589999999999999},"min":{"1min":0.45000000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.206,"5min":4.2089999999999996,"15min":7.0590000000000002},"last":0.437}]},{"osd":6,"last update":"Mon Feb 12 16:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.872,"15min":1.119},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.32000000000000001},"max":{"1min":3.4199999999999999,"5min":7.0049999999999999,"15min":8.4510000000000005},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.83399999999999996,"15min":1.1020000000000001},"min":{"1min":0.42699999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.3500000000000001,"5min":4.0940000000000003,"15min":8.8179999999999996},"last":0.498}]},{"osd":7,"last update":"Mon Feb 12 16:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.58399999999999996,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":1.9019999999999999,"5min":6.2560000000000002,"15min":11.044},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2529999999999999,"15min":1.248},"min":{"1min":0.44600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.4219999999999997,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":1.6499999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":1.5469999999999999,"15min":1.2969999999999999},"min":{"1min":0.51300000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.92700000000000005,"5min":59.164999999999999,"15min":59.164999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.4990000000000001,"15min":1.2649999999999999},"min":{"1min":0.47299999999999998,"5min":0.44500000000000001,"15min":0.377},"max":{"1min":1.266,"5min":58.993000000000002,"15min":58.993000000000002},"last":0.54100000000000004}]},{"osd":9,"last update":"Mon Feb 12 16:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.94199999999999995,"15min":1.2130000000000001},"min":{"1min":0.45200000000000001,"5min":0.435,"15min":0.38600000000000001},"max":{"1min":2.9140000000000001,"5min":5.2210000000000001,"15min":7.7759999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.127,"15min":1.238},"min":{"1min":0.48499999999999999,"5min":0.434,"15min":0.41299999999999998},"max":{"1min":4.6050000000000004,"5min":5.2549999999999999,"15min":7.2110000000000003},"last":0.60499999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:46:05 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.2809999999999999,"15min":2.54},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.41399999999999998},"max":{"1min":5.7240000000000002,"5min":7.1820000000000004,"15min":407.577},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.1220000000000001,"15min":2.4740000000000002},"min":{"1min":0.61299999999999999,"5min":0.47799999999999998,"15min":0.41799999999999998},"max":{"1min":1.9930000000000001,"5min":7.1470000000000002,"15min":407.45100000000002},"last":4.5830000000000002}]},{"osd":11,"last update":"Mon Feb 12 16:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.4910000000000001,"15min":1.345},"min":{"1min":0.65800000000000003,"5min":0.50700000000000001,"15min":0.46100000000000002},"max":{"1min":6.1429999999999998,"5min":31.835999999999999,"15min":31.835999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.474,"15min":1.411},"min":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.47699999999999998},"max":{"1min":4.8289999999999997,"5min":31.978999999999999,"15min":31.978999999999999},"last":0.70899999999999996}]}]},{"osd":3,"up_from":522,"seq":2241972928796,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854612,"kb_used_data":225804,"kb_used_omap":91,"kb_used_meta":628708,"kb_avail":92890540,"statfs":{"total":95995035648,"available":95119912960,"internally_reserved":0,"allocated":231223296,"data_stored":230631425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":643797008},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.72299999999999998,"15min":1.0249999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.20699999999999999},"max":{"1min":4.2530000000000001,"5min":4.2530000000000001,"15min":10.101000000000001},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.81799999999999995,"15min":1.0629999999999999},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.23300000000000001},"max":{"1min":9.907,"5min":9.907,"15min":10.436999999999999},"last":0.70399999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":3.718,"15min":1.9370000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26200000000000001},"max":{"1min":3.819,"5min":245.10900000000001,"15min":245.10900000000001},"last":1.022},{"interface":"front","average":{"1min":1.5900000000000001,"5min":3.9359999999999999,"15min":2.1139999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":5.0279999999999996,"5min":245.21199999999999,"15min":245.21199999999999},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.732,"15min":1.331},"min":{"1min":0.35199999999999998,"5min":0.30399999999999999,"15min":0.252},"max":{"1min":4.6699999999999999,"5min":81.769999999999996,"15min":81.769999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.833,"15min":1.4970000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.23699999999999999},"max":{"1min":5.165,"5min":81.876999999999995,"15min":81.876999999999995},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98199999999999998,"15min":1.1699999999999999},"min":{"1min":0.442,"5min":0.39800000000000002,"15min":0.27400000000000002},"max":{"1min":4.806,"5min":4.806,"15min":6.8220000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.224,"5min":0.95799999999999996,"15min":1.0780000000000001},"min":{"1min":0.36699999999999999,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":8.0069999999999997,"5min":8.0069999999999997,"15min":8.0069999999999997},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.1339999999999999,"15min":1.1859999999999999},"min":{"1min":0.38600000000000001,"5min":0.35299999999999998,"15min":0.30499999999999999},"max":{"1min":10.058,"5min":10.058,"15min":11.884},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.258,"5min":1.2250000000000001,"15min":1.339},"min":{"1min":0.434,"5min":0.36199999999999999,"15min":0.29899999999999999},"max":{"1min":4.7480000000000002,"5min":8.032,"15min":12.164999999999999},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.1859999999999999,"15min":1.2450000000000001},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.31900000000000001},"max":{"1min":9.7710000000000008,"5min":9.7710000000000008,"15min":9.7710000000000008},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.321,"5min":1.121,"15min":1.3200000000000001},"min":{"1min":0.47999999999999998,"5min":0.46500000000000002,"15min":0.30499999999999999},"max":{"1min":4.9790000000000001,"5min":4.9790000000000001,"15min":9.4640000000000004},"last":0.47599999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:38:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.5149999999999999,"15min":1.298},"min":{"1min":0.55000000000000004,"5min":0.36199999999999999,"15min":0.33300000000000002},"max":{"1min":7.9909999999999997,"5min":7.9909999999999997,"15min":7.9909999999999997},"last":0.87},{"interface":"front","average":{"1min":1.609,"5min":1.3859999999999999,"15min":1.335},"min":{"1min":0.621,"5min":0.371,"15min":0.371},"max":{"1min":6.0899999999999999,"5min":6.7590000000000003,"15min":7.2249999999999996},"last":0.66800000000000004}]},{"osd":8,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.1160000000000001,"15min":1.2490000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.38200000000000001},"max":{"1min":8.2710000000000008,"5min":8.2710000000000008,"15min":8.2710000000000008},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.252,"15min":1.3480000000000001},"min":{"1min":0.51600000000000001,"5min":0.47899999999999998,"15min":0.35599999999999998},"max":{"1min":5.0279999999999996,"5min":5.0279999999999996,"15min":7.9569999999999999},"last":0.65600000000000003}]},{"osd":9,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.113,"15min":1.3149999999999999},"min":{"1min":0.52400000000000002,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.9350000000000005},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.262,"15min":1.339},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.38300000000000001},"max":{"1min":4.9080000000000004,"5min":4.9080000000000004,"15min":9.5239999999999991},"last":0.56699999999999995}]},{"osd":10,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":2.2189999999999999,"5min":9.282,"15min":3.976},"min":{"1min":0.55900000000000005,"5min":0.52100000000000002,"15min":0.40799999999999997},"max":{"1min":21.669,"5min":738.99900000000002,"15min":738.99900000000002},"last":3.1280000000000001},{"interface":"front","average":{"1min":2.3180000000000001,"5min":9.2850000000000001,"15min":4.0620000000000003},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.45600000000000002},"max":{"1min":21.475999999999999,"5min":746.45299999999997,"15min":746.45299999999997},"last":3.1629999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:45:41 2024","interfaces":[{"interface":"back","average":{"1min":2.476,"5min":1.722,"15min":1.698},"min":{"1min":0.56799999999999995,"5min":0.55000000000000004,"15min":0.437},"max":{"1min":21.888000000000002,"5min":21.888000000000002,"15min":73.037999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":2.2349999999999999,"5min":1.6080000000000001,"15min":1.738},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":22.079999999999998,"5min":22.079999999999998,"15min":73.057000000000002},"last":0.93000000000000005}]}]},{"osd":4,"up_from":32,"seq":137438954111,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1069308,"kb_used_data":175156,"kb_used_omap":0,"kb_used_meta":894144,"kb_avail":92675844,"statfs":{"total":95995035648,"available":94900064256,"internally_reserved":0,"allocated":179359744,"data_stored":178779732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915603456},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:46:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3410000000000002,"5min":3.3410000000000002,"15min":3.3410000000000002},"last":0.34300000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:45:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68200000000000005,"15min":1.268},"min":{"1min":0.29399999999999998,"5min":0.27600000000000002,"15min":0.248},"max":{"1min":1.3260000000000001,"5min":3.1520000000000001,"15min":56.188000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.74299999999999999,"15min":1.355},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.25700000000000001},"max":{"1min":0.97299999999999998,"5min":3.0880000000000001,"15min":55.414000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:44:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":3.0840000000000001,"15min":1.8340000000000001},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3100000000000001,"5min":212.685,"15min":212.685},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":2.9910000000000001,"15min":1.7490000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.3540000000000001,"5min":210.95699999999999,"15min":210.95699999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.016,"15min":1.3400000000000001},"min":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.311},"max":{"1min":0.88300000000000001,"5min":17.876999999999999,"15min":30.282},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.103,"5min":1.153,"15min":1.3520000000000001},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":10.209,"5min":17.623999999999999,"15min":30.478000000000002},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.79700000000000004,"15min":1.0349999999999999},"min":{"1min":0.307,"5min":0.28699999999999998,"15min":0.252},"max":{"1min":4.7370000000000001,"5min":4.7370000000000001,"15min":10.289999999999999},"last":0.627},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.91200000000000003,"15min":1.1539999999999999},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.33000000000000002},"max":{"1min":4.8150000000000004,"5min":4.8150000000000004,"15min":10.321999999999999},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Feb 12 16:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.73499999999999999,"15min":1},"min":{"1min":0.41399999999999998,"5min":0.29899999999999999,"15min":0.29499999999999998},"max":{"1min":3.4830000000000001,"5min":3.4830000000000001,"15min":6.5270000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.874,"5min":0.84499999999999997,"15min":1.1060000000000001},"min":{"1min":0.5,"5min":0.41999999999999998,"15min":0.29999999999999999},"max":{"1min":3.508,"5min":3.508,"15min":11.968999999999999},"last":0.62}]},{"osd":7,"last update":"Mon Feb 12 16:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.93799999999999994,"15min":1.0309999999999999},"min":{"1min":0.41699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.3839999999999999,"5min":3.569,"15min":3.569},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.80700000000000005,"15min":0.72199999999999998},"min":{"1min":0.496,"5min":0.376,"15min":0.376},"max":{"1min":4.0709999999999997,"5min":4.0709999999999997,"15min":4.0709999999999997},"last":0.42099999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:45:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.98799999999999999,"15min":1.115},"min":{"1min":0.56399999999999995,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":2.1480000000000001,"5min":3.8279999999999998,"15min":7.5179999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.89100000000000001,"15min":1.089},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8720000000000001,"5min":4.6470000000000002,"15min":7.9349999999999996},"last":0.5}]},{"osd":9,"last update":"Mon Feb 12 16:45:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.96899999999999997,"15min":1.387},"min":{"1min":0.56100000000000005,"5min":0.5,"15min":0.39300000000000002},"max":{"1min":1.0840000000000001,"5min":2.2879999999999998,"15min":63.545000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.875,"5min":1.0449999999999999,"15min":1.407},"min":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.44600000000000001},"max":{"1min":1.1240000000000001,"5min":2.3370000000000002,"15min":63.421999999999997},"last":0.83099999999999996}]},{"osd":10,"last update":"Mon Feb 12 16:45:35 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.139,"15min":1.2150000000000001},"min":{"1min":0.46300000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":4.4210000000000003,"5min":4.6079999999999997,"15min":5.6109999999999998},"last":2.1139999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.079,"15min":1.1339999999999999},"min":{"1min":0.53500000000000003,"5min":0.45500000000000002,"15min":0.41799999999999998},"max":{"1min":6.8920000000000003,"5min":6.8920000000000003,"15min":6.8920000000000003},"last":0.61499999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.038,"15min":1.2350000000000001},"min":{"1min":0.51600000000000001,"5min":0.496,"15min":0.48999999999999999},"max":{"1min":3.4140000000000001,"5min":3.4710000000000001,"15min":7.4880000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.101,"15min":1.2350000000000001},"min":{"1min":0.59899999999999998,"5min":0.56399999999999995,"15min":0.48899999999999999},"max":{"1min":1.1659999999999999,"5min":3.4580000000000002,"15min":5.681},"last":0.65500000000000003}]}]},{"osd":5,"up_from":39,"seq":167503725176,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1281748,"kb_used_data":380492,"kb_used_omap":0,"kb_used_meta":901248,"kb_avail":92463404,"statfs":{"total":95995035648,"available":94682525696,"internally_reserved":0,"allocated":389623808,"data_stored":389031937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":922877952},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.74199999999999999,"15min":1.2609999999999999},"min":{"1min":0.28899999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2649999999999999,"5min":3.7610000000000001,"15min":38.366},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.94799999999999995,"15min":1.385},"min":{"1min":0.317,"5min":0.28899999999999998,"15min":0.255},"max":{"1min":4.915,"5min":5.1459999999999999,"15min":37.310000000000002},"last":0.56399999999999995}]},{"osd":1,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":3.3199999999999998,"5min":6.0279999999999996,"15min":2.6890000000000001},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.23499999999999999},"max":{"1min":43.521999999999998,"5min":387.76299999999998,"15min":387.76299999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":3.4399999999999999,"5min":6.1500000000000004,"15min":2.8159999999999998},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.34100000000000003},"max":{"1min":43.585999999999999,"5min":387.88099999999997,"15min":387.88099999999997},"last":5.1580000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.93600000000000005,"15min":1.091},"min":{"1min":0.39700000000000002,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":1.157,"5min":4.5389999999999997,"15min":6.6459999999999999},"last":5.1109999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.872,"15min":1.1040000000000001},"min":{"1min":0.38300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.976,"5min":7.3470000000000004,"15min":8.2439999999999998},"last":1.6399999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.135,"15min":1.325},"min":{"1min":0.372,"5min":0.36799999999999999,"15min":0.28299999999999997},"max":{"1min":4.9809999999999999,"5min":5.8879999999999999,"15min":19.067},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.871,"15min":1.2210000000000001},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.308},"max":{"1min":3.0529999999999999,"5min":4.7839999999999998,"15min":18.216000000000001},"last":1.6659999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.93000000000000005,"15min":1.004},"min":{"1min":0.35199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":1.4650000000000001,"5min":5.7460000000000004,"15min":6.5999999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.077,"5min":1.0629999999999999,"15min":1.1040000000000001},"min":{"1min":0.43099999999999999,"5min":0.28199999999999997,"15min":0.27600000000000002},"max":{"1min":4.8600000000000003,"5min":5.069,"15min":5.5389999999999997},"last":0.76700000000000002}]},{"osd":6,"last update":"Mon Feb 12 16:45:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.079,"15min":1.4179999999999999},"min":{"1min":0.44400000000000001,"5min":0.35099999999999998,"15min":0.28699999999999998},"max":{"1min":4.9459999999999997,"5min":5.1760000000000002,"15min":68.263999999999996},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.069,"5min":0.91700000000000004,"15min":1.456},"min":{"1min":0.42999999999999999,"5min":0.34599999999999997,"15min":0.33000000000000002},"max":{"1min":7.3029999999999999,"5min":7.3029999999999999,"15min":67.873000000000005},"last":1.597}]},{"osd":7,"last update":"Mon Feb 12 16:38:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.86799999999999999,"15min":1.1830000000000001},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.28799999999999998},"max":{"1min":1.9570000000000001,"5min":5.6609999999999996,"15min":9.7710000000000008},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.024,"15min":1.286},"min":{"1min":0.376,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.9670000000000001,"5min":5.6120000000000001,"15min":6.7599999999999998},"last":0.627}]},{"osd":8,"last update":"Mon Feb 12 16:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.1419999999999999,"15min":1.1719999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.46100000000000002},"max":{"1min":1.409,"5min":5.8090000000000002,"15min":5.8090000000000002},"last":5.202},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.002,"15min":1.2190000000000001},"min":{"1min":0.41899999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.1110000000000002,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":1.708}]},{"osd":9,"last update":"Mon Feb 12 16:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":1.238,"15min":1.2190000000000001},"min":{"1min":0.55900000000000005,"5min":0.46300000000000002,"15min":0.41899999999999998},"max":{"1min":1.409,"5min":9.0830000000000002,"15min":9.0830000000000002},"last":5.282},{"interface":"front","average":{"1min":1.113,"5min":1.101,"15min":1.234},"min":{"1min":0.48199999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.5419999999999998,"5min":6.5419999999999998,"15min":7.9710000000000001},"last":1.6890000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.391,"15min":1.3340000000000001},"min":{"1min":0.55000000000000004,"5min":0.434,"15min":0.43099999999999999},"max":{"1min":5.0190000000000001,"5min":8.3879999999999999,"15min":8.3879999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.375,"15min":1.339},"min":{"1min":0.61799999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":6.173,"5min":6.173,"15min":6.867},"last":5.2530000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:46:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5369999999999999,"15min":1.397},"min":{"1min":0.57499999999999996,"5min":0.51300000000000001,"15min":0.47199999999999998},"max":{"1min":7.4210000000000003,"5min":7.4210000000000003,"15min":8.0640000000000001},"last":0.874},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.5149999999999999,"15min":1.385},"min":{"1min":0.65000000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":5.226}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532156416,"data_stored":532152344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217055232,"data_stored":217055232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":167837696,"data_stored":167837696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":386072576,"data_stored":386072576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":162332672,"data_stored":162332672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":296161280,"data_stored":296157208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":169672704,"data_stored":169672704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":827654144,"data_stored":827654144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":436994048,"data_stored":436994048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":376963072,"data_stored":376963072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:46:41.299 INFO:tasks.thrashosds.thrasher:Setting 1.9 to [0, 1, 5, 10] 2024-02-12T16:46:41.299 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.9', '0', '1', '5', '10'] 2024-02-12T16:46:41.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pg-upmap-items 1.9 0 1 5 10 2024-02-12T16:46:41.309 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:41.331 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:41.434 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:41.458 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:41.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:41.600 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:41.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:41 smithi121 bash[18489]: cluster 2024-02-12T16:46:40.330219+0000 mgr.y (mgr.14146) 1874 : cluster [DBG] pgmap v2748: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s; 1555/33120 objects misplaced (4.695%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:46:41.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:41 smithi121 bash[18489]: cluster 2024-02-12T16:46:40.456978+0000 osd.1 (osd.1) 177 : cluster [DBG] 9.14 starting backfill to osd.11 from (0'0,0'0] MAX to 1005'16900 2024-02-12T16:46:41.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:41 smithi121 bash[18489]: audit 2024-02-12T16:46:40.457282+0000 mon.a (mon.0) 3319 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:46:41.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:41 smithi121 bash[18489]: cluster 2024-02-12T16:46:40.457351+0000 mon.a (mon.0) 3320 : cluster [DBG] osdmap e1008: 12 total, 12 up, 10 in 2024-02-12T16:46:41.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:41 smithi121 bash[18489]: cluster 2024-02-12T16:46:41.027186+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e1009: 12 total, 12 up, 10 in 2024-02-12T16:46:41.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:41.725 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:41 smithi169 bash[19604]: cluster 2024-02-12T16:46:40.330219+0000 mgr.y (mgr.14146) 1874 : cluster [DBG] pgmap v2748: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s; 1555/33120 objects misplaced (4.695%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:46:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:41 smithi169 bash[19604]: cluster 2024-02-12T16:46:40.456978+0000 osd.1 (osd.1) 177 : cluster [DBG] 9.14 starting backfill to osd.11 from (0'0,0'0] MAX to 1005'16900 2024-02-12T16:46:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:41 smithi169 bash[19604]: audit 2024-02-12T16:46:40.457282+0000 mon.a (mon.0) 3319 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:46:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:41 smithi169 bash[19604]: cluster 2024-02-12T16:46:40.457351+0000 mon.a (mon.0) 3320 : cluster [DBG] osdmap e1008: 12 total, 12 up, 10 in 2024-02-12T16:46:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:41 smithi169 bash[19604]: cluster 2024-02-12T16:46:41.027186+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e1009: 12 total, 12 up, 10 in 2024-02-12T16:46:41.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:41.857 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:41.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:41 smithi123 bash[19119]: cluster 2024-02-12T16:46:40.330219+0000 mgr.y (mgr.14146) 1874 : cluster [DBG] pgmap v2748: 57 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 54 active+clean; 1.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s; 1555/33120 objects misplaced (4.695%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:46:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:41 smithi123 bash[19119]: cluster 2024-02-12T16:46:40.456978+0000 osd.1 (osd.1) 177 : cluster [DBG] 9.14 starting backfill to osd.11 from (0'0,0'0] MAX to 1005'16900 2024-02-12T16:46:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:41 smithi123 bash[19119]: audit 2024-02-12T16:46:40.457282+0000 mon.a (mon.0) 3319 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-12T16:46:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:41 smithi123 bash[19119]: cluster 2024-02-12T16:46:40.457351+0000 mon.a (mon.0) 3320 : cluster [DBG] osdmap e1008: 12 total, 12 up, 10 in 2024-02-12T16:46:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:41 smithi123 bash[19119]: cluster 2024-02-12T16:46:41.027186+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e1009: 12 total, 12 up, 10 in 2024-02-12T16:46:41.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:41.993 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:41.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:42.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:42.131 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:42.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:42.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:42.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:42.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:42.365 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:42.399 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:42.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:42.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:42.531 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:42.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:42.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:42.658 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:42.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:42.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:42.781 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:42.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:42.884 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:42.904 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:42.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:43.008 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:43.032 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:43.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:43.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:43.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:43 smithi169 bash[19604]: cluster 2024-02-12T16:46:42.032526+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e1010: 12 total, 12 up, 10 in 2024-02-12T16:46:43.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:43 smithi169 bash[19604]: cluster 2024-02-12T16:46:42.038865+0000 osd.5 (osd.5) 223 : cluster [DBG] 9.15 starting backfill to osd.7 from (0'0,0'0] MAX to 1007'16850 2024-02-12T16:46:43.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:43 smithi169 bash[19604]: cluster 2024-02-12T16:46:42.053000+0000 osd.5 (osd.5) 224 : cluster [DBG] 9.15 starting backfill to osd.8 from (0'0,0'0] MAX to 1007'16850 2024-02-12T16:46:43.279 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:43 smithi169 bash[19604]: cluster 2024-02-12T16:46:42.330731+0000 mgr.y (mgr.14146) 1875 : cluster [DBG] pgmap v2752: 57 pgs: 1 active+clean+remapped, 1 activating+remapped, 3 active+remapped+backfilling, 52 active+clean; 888 MiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 1984/28422 objects misplaced (6.981%); 9.2 MiB/s, 146 objects/s recovering 2024-02-12T16:46:43.298 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:43 smithi123 bash[19119]: cluster 2024-02-12T16:46:42.032526+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e1010: 12 total, 12 up, 10 in 2024-02-12T16:46:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:43 smithi123 bash[19119]: cluster 2024-02-12T16:46:42.038865+0000 osd.5 (osd.5) 223 : cluster [DBG] 9.15 starting backfill to osd.7 from (0'0,0'0] MAX to 1007'16850 2024-02-12T16:46:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:43 smithi123 bash[19119]: cluster 2024-02-12T16:46:42.053000+0000 osd.5 (osd.5) 224 : cluster [DBG] 9.15 starting backfill to osd.8 from (0'0,0'0] MAX to 1007'16850 2024-02-12T16:46:43.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:43 smithi123 bash[19119]: cluster 2024-02-12T16:46:42.330731+0000 mgr.y (mgr.14146) 1875 : cluster [DBG] pgmap v2752: 57 pgs: 1 active+clean+remapped, 1 activating+remapped, 3 active+remapped+backfilling, 52 active+clean; 888 MiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 1984/28422 objects misplaced (6.981%); 9.2 MiB/s, 146 objects/s recovering 2024-02-12T16:46:43.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:43 smithi121 bash[18489]: cluster 2024-02-12T16:46:42.032526+0000 mon.a (mon.0) 3322 : cluster [DBG] osdmap e1010: 12 total, 12 up, 10 in 2024-02-12T16:46:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:43 smithi121 bash[18489]: cluster 2024-02-12T16:46:42.038865+0000 osd.5 (osd.5) 223 : cluster [DBG] 9.15 starting backfill to osd.7 from (0'0,0'0] MAX to 1007'16850 2024-02-12T16:46:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:43 smithi121 bash[18489]: cluster 2024-02-12T16:46:42.053000+0000 osd.5 (osd.5) 224 : cluster [DBG] 9.15 starting backfill to osd.8 from (0'0,0'0] MAX to 1007'16850 2024-02-12T16:46:43.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:43 smithi121 bash[18489]: cluster 2024-02-12T16:46:42.330731+0000 mgr.y (mgr.14146) 1875 : cluster [DBG] pgmap v2752: 57 pgs: 1 active+clean+remapped, 1 activating+remapped, 3 active+remapped+backfilling, 52 active+clean; 888 MiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 1984/28422 objects misplaced (6.981%); 9.2 MiB/s, 146 objects/s recovering 2024-02-12T16:46:43.436 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:43.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:43.576 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:43.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.707 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:43.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.848 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:43.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:43.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:44.082 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:44.110 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:44.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:44.235 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:44.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:44 smithi169 bash[19604]: cluster 2024-02-12T16:46:43.031540+0000 mon.a (mon.0) 3323 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:46:44.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:44 smithi169 bash[19604]: cluster 2024-02-12T16:46:43.031619+0000 mon.a (mon.0) 3324 : cluster [INF] Cluster is now healthy 2024-02-12T16:46:44.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:44 smithi169 bash[19604]: cluster 2024-02-12T16:46:43.044721+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e1011: 12 total, 12 up, 10 in 2024-02-12T16:46:44.341 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:44.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:44.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:44.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:44 smithi123 bash[19119]: cluster 2024-02-12T16:46:43.031540+0000 mon.a (mon.0) 3323 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:46:44.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:44 smithi123 bash[19119]: cluster 2024-02-12T16:46:43.031619+0000 mon.a (mon.0) 3324 : cluster [INF] Cluster is now healthy 2024-02-12T16:46:44.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:44 smithi123 bash[19119]: cluster 2024-02-12T16:46:43.044721+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e1011: 12 total, 12 up, 10 in 2024-02-12T16:46:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:44 smithi121 bash[18489]: cluster 2024-02-12T16:46:43.031540+0000 mon.a (mon.0) 3323 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-12T16:46:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:44 smithi121 bash[18489]: cluster 2024-02-12T16:46:43.031619+0000 mon.a (mon.0) 3324 : cluster [INF] Cluster is now healthy 2024-02-12T16:46:44.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:44 smithi121 bash[18489]: cluster 2024-02-12T16:46:43.044721+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e1011: 12 total, 12 up, 10 in 2024-02-12T16:46:44.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:44.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:44.613 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:44.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:44.741 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:44.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:44.867 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:44.973 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:44.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:44.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:45.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:45.128 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:45.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:45.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:45.237 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:46:45.269 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:46:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:45.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:45.290 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:45 smithi121 bash[18489]: cluster 2024-02-12T16:46:44.044722+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e1012: 12 total, 12 up, 10 in 2024-02-12T16:46:45.290 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:45 smithi121 bash[18489]: cluster 2024-02-12T16:46:44.331122+0000 mgr.y (mgr.14146) 1876 : cluster [DBG] pgmap v2755: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 52 active+clean; 827 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 857 op/s; 1088/26472 objects misplaced (4.110%); 16 MiB/s, 255 objects/s recovering 2024-02-12T16:46:45.296 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:46:45.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:45 smithi169 bash[19604]: cluster 2024-02-12T16:46:44.044722+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e1012: 12 total, 12 up, 10 in 2024-02-12T16:46:45.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:45 smithi169 bash[19604]: cluster 2024-02-12T16:46:44.331122+0000 mgr.y (mgr.14146) 1876 : cluster [DBG] pgmap v2755: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 52 active+clean; 827 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 857 op/s; 1088/26472 objects misplaced (4.110%); 16 MiB/s, 255 objects/s recovering 2024-02-12T16:46:45.329 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:46:45.352 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:46:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:45 smithi123 bash[19119]: cluster 2024-02-12T16:46:44.044722+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e1012: 12 total, 12 up, 10 in 2024-02-12T16:46:45.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:45 smithi123 bash[19119]: cluster 2024-02-12T16:46:44.331122+0000 mgr.y (mgr.14146) 1876 : cluster [DBG] pgmap v2755: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 52 active+clean; 827 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 857 op/s; 1088/26472 objects misplaced (4.110%); 16 MiB/s, 255 objects/s recovering 2024-02-12T16:46:45.391 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:45.394 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:46:45.415 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:45.416 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:46:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:45.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:45.453 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:46:45.485 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:46:45.509 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:46:45.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:45.536 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:46:45.543 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:45.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:45.567 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:46:45.569 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.569 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.569 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.569 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.569 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:46:45.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:45.672 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:45.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:45.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:45.804 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:45.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:45.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:45.946 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:45.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:46.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:46.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:46.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:46.102 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is set 2024-02-12T16:46:46.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:46.209 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:46.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:46.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:46.344 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:46.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:46.353 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:46 smithi169 bash[19604]: cluster 2024-02-12T16:46:45.056648+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e1013: 12 total, 12 up, 10 in 2024-02-12T16:46:46.353 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:46 smithi169 bash[19604]: audit 2024-02-12T16:46:45.199280+0000 mon.a (mon.0) 3328 : audit [DBG] from='client.? v1:172.21.15.121:0/2942139452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:46.353 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:46 smithi169 bash[19604]: audit 2024-02-12T16:46:45.548171+0000 mon.a (mon.0) 3329 : audit [INF] from='client.? v1:172.21.15.121:0/2328056596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:46:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:46 smithi123 bash[19119]: cluster 2024-02-12T16:46:45.056648+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e1013: 12 total, 12 up, 10 in 2024-02-12T16:46:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:46 smithi123 bash[19119]: audit 2024-02-12T16:46:45.199280+0000 mon.a (mon.0) 3328 : audit [DBG] from='client.? v1:172.21.15.121:0/2942139452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:46.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:46 smithi123 bash[19119]: audit 2024-02-12T16:46:45.548171+0000 mon.a (mon.0) 3329 : audit [INF] from='client.? v1:172.21.15.121:0/2328056596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:46:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:46 smithi121 bash[18489]: cluster 2024-02-12T16:46:45.056648+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e1013: 12 total, 12 up, 10 in 2024-02-12T16:46:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:46 smithi121 bash[18489]: audit 2024-02-12T16:46:45.199280+0000 mon.a (mon.0) 3328 : audit [DBG] from='client.? v1:172.21.15.121:0/2942139452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:46:46.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:46 smithi121 bash[18489]: audit 2024-02-12T16:46:45.548171+0000 mon.a (mon.0) 3329 : audit [INF] from='client.? v1:172.21.15.121:0/2328056596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-12T16:46:46.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:46.476 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:46.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:46.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:46.612 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:46.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:46.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:46.752 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:46.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:46.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:46.905 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:46.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:47.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:47.040 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:47.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:47.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:47.169 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:47.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:47.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:47.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:47 smithi121 bash[18489]: cluster 2024-02-12T16:46:46.068428+0000 mon.a (mon.0) 3330 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:46:47.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:47 smithi121 bash[18489]: audit 2024-02-12T16:46:46.093622+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? v1:172.21.15.121:0/2328056596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:46:47.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:47 smithi121 bash[18489]: cluster 2024-02-12T16:46:46.093722+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e1014: 12 total, 12 up, 10 in 2024-02-12T16:46:47.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:47 smithi121 bash[18489]: cluster 2024-02-12T16:46:46.331493+0000 mgr.y (mgr.14146) 1877 : cluster [DBG] pgmap v2758: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 52 active+clean; 802 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 345 op/s; 1088/25660 objects misplaced (4.240%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:46:47.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:47.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:47 smithi123 bash[19119]: cluster 2024-02-12T16:46:46.068428+0000 mon.a (mon.0) 3330 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:46:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:47 smithi123 bash[19119]: audit 2024-02-12T16:46:46.093622+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? v1:172.21.15.121:0/2328056596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:46:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:47 smithi123 bash[19119]: cluster 2024-02-12T16:46:46.093722+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e1014: 12 total, 12 up, 10 in 2024-02-12T16:46:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:47 smithi123 bash[19119]: cluster 2024-02-12T16:46:46.331493+0000 mgr.y (mgr.14146) 1877 : cluster [DBG] pgmap v2758: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 52 active+clean; 802 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 345 op/s; 1088/25660 objects misplaced (4.240%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:46:47.406 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:47.435 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:47.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:47.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:47 smithi169 bash[19604]: cluster 2024-02-12T16:46:46.068428+0000 mon.a (mon.0) 3330 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:46:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:47 smithi169 bash[19604]: audit 2024-02-12T16:46:46.093622+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? v1:172.21.15.121:0/2328056596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-12T16:46:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:47 smithi169 bash[19604]: cluster 2024-02-12T16:46:46.093722+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e1014: 12 total, 12 up, 10 in 2024-02-12T16:46:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:47 smithi169 bash[19604]: cluster 2024-02-12T16:46:46.331493+0000 mgr.y (mgr.14146) 1877 : cluster [DBG] pgmap v2758: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 52 active+clean; 802 MiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 345 op/s; 1088/25660 objects misplaced (4.240%); 6.6 MiB/s, 104 objects/s recovering 2024-02-12T16:46:47.573 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:47.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:47.613 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:47.615 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:47.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:47.719 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:47.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:47.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:47.867 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:47.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:47.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:48.004 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:48.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:48.146 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:48.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:48.382 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:48.415 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:48.519 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:48.546 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:48.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:48.674 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:48.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:48.804 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:48.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-12T16:46:48.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:48.943 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:48.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:49.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:49.088 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:49.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:49.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-12T16:46:49.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:49.225 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:49.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:49.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:49.363 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:49.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:49.469 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:49.476 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:49 smithi123 bash[19119]: cluster 2024-02-12T16:46:48.331971+0000 mgr.y (mgr.14146) 1878 : cluster [DBG] pgmap v2759: 57 pgs: 1 peering, 3 active+remapped+backfilling, 53 active+clean; 652 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 715 op/s; 535/20856 objects misplaced (2.565%); 6.3 MiB/s, 101 objects/s recovering 2024-02-12T16:46:49.505 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:49.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:49.608 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:49.620 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:49 smithi121 bash[18489]: cluster 2024-02-12T16:46:48.331971+0000 mgr.y (mgr.14146) 1878 : cluster [DBG] pgmap v2759: 57 pgs: 1 peering, 3 active+remapped+backfilling, 53 active+clean; 652 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 715 op/s; 535/20856 objects misplaced (2.565%); 6.3 MiB/s, 101 objects/s recovering 2024-02-12T16:46:49.642 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:49.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:49.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:49.755 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:49 smithi169 bash[19604]: cluster 2024-02-12T16:46:48.331971+0000 mgr.y (mgr.14146) 1878 : cluster [DBG] pgmap v2759: 57 pgs: 1 peering, 3 active+remapped+backfilling, 53 active+clean; 652 MiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 715 op/s; 535/20856 objects misplaced (2.565%); 6.3 MiB/s, 101 objects/s recovering 2024-02-12T16:46:49.772 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:49.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:49.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:49.913 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:49.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:50.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:50.057 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:50.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:50.185 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:50.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:50.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:50.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:50.449 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:50 smithi169 bash[19604]: audit 2024-02-12T16:46:50.333133+0000 mon.a (mon.0) 3333 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:46:50.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:50.576 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:50 smithi121 bash[18489]: audit 2024-02-12T16:46:50.333133+0000 mon.a (mon.0) 3333 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:46:50.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:50.703 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:50.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:50.833 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:50 smithi123 bash[19119]: audit 2024-02-12T16:46:50.333133+0000 mon.a (mon.0) 3333 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-12T16:46:50.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:50.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:51.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:51.103 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:51.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:51.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:51.237 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:51.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:51.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:51.372 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:51.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:51.476 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:51.504 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:51.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:51.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:51.645 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:51.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: cluster 2024-02-12T16:46:50.332468+0000 mgr.y (mgr.14146) 1879 : cluster [DBG] pgmap v2760: 57 pgs: 3 active+remapped+backfilling, 54 active+clean; 484 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 535/15502 objects misplaced (3.451%); 5.3 MiB/s, 85 objects/s recovering 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: audit 2024-02-12T16:46:50.399279+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: cluster 2024-02-12T16:46:50.399601+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e1015: 12 total, 12 up, 10 in 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: audit 2024-02-12T16:46:50.808471+0000 mon.a (mon.0) 3336 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: cluster 2024-02-12T16:46:51.031990+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e1016: 12 total, 12 up, 10 in 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: audit 2024-02-12T16:46:51.189758+0000 mon.a (mon.0) 3338 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: audit 2024-02-12T16:46:51.190565+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:51 smithi121 bash[18489]: audit 2024-02-12T16:46:51.196415+0000 mon.a (mon.0) 3340 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:46:51.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: cluster 2024-02-12T16:46:50.332468+0000 mgr.y (mgr.14146) 1879 : cluster [DBG] pgmap v2760: 57 pgs: 3 active+remapped+backfilling, 54 active+clean; 484 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 535/15502 objects misplaced (3.451%); 5.3 MiB/s, 85 objects/s recovering 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: audit 2024-02-12T16:46:50.399279+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: cluster 2024-02-12T16:46:50.399601+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e1015: 12 total, 12 up, 10 in 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: audit 2024-02-12T16:46:50.808471+0000 mon.a (mon.0) 3336 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: cluster 2024-02-12T16:46:51.031990+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e1016: 12 total, 12 up, 10 in 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: audit 2024-02-12T16:46:51.189758+0000 mon.a (mon.0) 3338 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: audit 2024-02-12T16:46:51.190565+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:51.760 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:51 smithi123 bash[19119]: audit 2024-02-12T16:46:51.196415+0000 mon.a (mon.0) 3340 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:46:51.784 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:51.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: cluster 2024-02-12T16:46:50.332468+0000 mgr.y (mgr.14146) 1879 : cluster [DBG] pgmap v2760: 57 pgs: 3 active+remapped+backfilling, 54 active+clean; 484 MiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 535/15502 objects misplaced (3.451%); 5.3 MiB/s, 85 objects/s recovering 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: audit 2024-02-12T16:46:50.399279+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: cluster 2024-02-12T16:46:50.399601+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e1015: 12 total, 12 up, 10 in 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: audit 2024-02-12T16:46:50.808471+0000 mon.a (mon.0) 3336 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: cluster 2024-02-12T16:46:51.031990+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e1016: 12 total, 12 up, 10 in 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: audit 2024-02-12T16:46:51.189758+0000 mon.a (mon.0) 3338 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: audit 2024-02-12T16:46:51.190565+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:51 smithi169 bash[19604]: audit 2024-02-12T16:46:51.196415+0000 mon.a (mon.0) 3340 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:46:51.887 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:51.921 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:51.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:52.024 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:52.059 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:52.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:52.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:52.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:52.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:52.321 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:52.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:52.425 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:52.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:52.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:52.579 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:52.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:52.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:52.712 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:52.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:52.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:52.861 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:52.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:52.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:53.002 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:53.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:53.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:53 smithi169 bash[19604]: cluster 2024-02-12T16:46:52.032155+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e1017: 12 total, 12 up, 10 in 2024-02-12T16:46:53.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:53 smithi169 bash[19604]: cluster 2024-02-12T16:46:52.033596+0000 osd.1 (osd.1) 178 : cluster [DBG] 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 1014'16929 2024-02-12T16:46:53.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:53 smithi169 bash[19604]: cluster 2024-02-12T16:46:52.046087+0000 osd.1 (osd.1) 179 : cluster [DBG] 9.16 starting backfill to osd.11 from (0'0,0'0] MAX to 1014'16929 2024-02-12T16:46:53.046 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:53 smithi169 bash[19604]: cluster 2024-02-12T16:46:52.332961+0000 mgr.y (mgr.14146) 1880 : cluster [DBG] pgmap v2764: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 273 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s; 118/8742 objects misplaced (1.350%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "rep_scrubmap(1.14 e780 from shard 4)", 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:45:00.045950+0000", 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 113.03995136, 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.042618379999999997, 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.045950+0000", 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.045950+0000", 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.045944+0000", 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9578000000000001e-05 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.045964+0000", 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0568e-05 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.045985+0000", 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5040000000000002e-06 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.045993+0000", 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.042540624999999999 2024-02-12T16:46:53.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.088534+0000", 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2229999999999996e-06 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.088543+0000", 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5772999999999999e-05 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.088569+0000", 2024-02-12T16:46:53.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.042618379999999997 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "rep_scrubmap(1.14 e780 from shard 4)", 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:13.038409+0000", 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.047492941999998, 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.057733297000000003, 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.038409+0000", 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.038409+0000", 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.038405+0000", 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.007e-05 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.038415+0000", 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2379e-05 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.038427+0000", 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9580000000000007e-06 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.038435+0000", 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.057675379999999998 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.096110+0000", 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6389999999999992e-06 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.096118+0000", 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4000000000000001e-05 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:13.096142+0000", 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.057733297000000003 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297564 9.16 e1017/1016 9:6ec26774:::benchmark_data_smithi169_124168_object146880:head v 1017'16930, mlcod=0'0)", 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.204328+0000", 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.88157322400000004, 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004144366, 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204328+0000", 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204328+0000", 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204324+0000", 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9514e-05 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204344+0000", 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4096000000000001e-05 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204368+0000", 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6929999999999996e-06 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204376+0000", 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9148999999999997e-05 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204456+0000", 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6168000000000002e-05 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.204512+0000", 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039239339999999996 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.208436+0000", 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7218999999999997e-05 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.208473+0000", 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004144366 2024-02-12T16:46:53.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.110 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.110 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297789 9.16 e1017/1016 9:6f730b78:::benchmark_data_smithi169_124168_object147105:head v 1017'16938, mlcod=1017'16936)", 2024-02-12T16:46:53.110 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.231152+0000", 2024-02-12T16:46:53.110 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.85474996199999997, 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059089240000000003, 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231152+0000", 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231152+0000", 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:46:53.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231150+0000", 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3700000000000001e-06 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231159+0000", 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3339999999999999e-05 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231172+0000", 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1459999999999999e-06 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231180+0000", 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3900999999999999e-05 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231223+0000", 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0543e-05 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.231244+0000", 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0057848470000000001 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237029+0000", 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1782000000000002e-05 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237061+0000", 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059089240000000003 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.114 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297798 9.16 e1017/1016 9:6a024d0c:::benchmark_data_smithi169_124168_object147114:head v 1017'16939, mlcod=1017'16936)", 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.233172+0000", 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.85272972899999999, 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059901659999999999, 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.233172+0000", 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.233172+0000", 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.233171+0000", 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9870000000000005e-06 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.233180+0000", 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4803e-05 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.233194+0000", 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6619999999999999e-06 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.233201+0000", 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00095543899999999998 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.234156+0000", 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1365999999999998e-05 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.234178+0000", 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049538200000000003 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.239132+0000", 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0546999999999997e-05 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.239162+0000", 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059901659999999999 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297864 9.16 e1017/1016 9:6a79dd51:::benchmark_data_smithi169_124168_object147180:head v 1017'16940, mlcod=1017'16936)", 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.235240+0000", 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.85066182199999996, 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006435133, 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.235240+0000", 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.235240+0000", 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.235238+0000", 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8480000000000007e-06 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.235247+0000", 2024-02-12T16:46:53.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4081e-05 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.235261+0000", 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5969999999999997e-06 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.235268+0000", 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001820374 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237088+0000", 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9525e-05 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237108+0000", 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0045246330000000001 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241633+0000", 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2484999999999997e-05 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241675+0000", 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006435133 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297871 9.16 e1017/1016 9:6866ce93:::benchmark_data_smithi169_124168_object147187:head v 1017'16941, mlcod=1017'16936)", 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.237353+0000", 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.84854828800000004, 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068753039999999996, 2024-02-12T16:46:53.123 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237353+0000", 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237353+0000", 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237352+0000", 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3410000000000003e-06 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237361+0000", 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2196e-05 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237373+0000", 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0090000000000004e-06 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.237379+0000", 2024-02-12T16:46:53.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001811895 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.239191+0000", 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9380000000000001e-05 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.239210+0000", 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0049889089999999997 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.244199+0000", 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9815000000000002e-05 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.244229+0000", 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068753039999999996 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297924 9.16 e1017/1016 9:69b5b442:::benchmark_data_smithi169_124168_object147240:head v 1017'16942, mlcod=1017'16936)", 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.241287+0000", 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.84461436400000001, 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056339500000000004, 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241287+0000", 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241287+0000", 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241286+0000", 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3399999999999998e-06 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241294+0000", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2634e-05 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241307+0000", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.246e-06 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241313+0000", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039324299999999999 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241706+0000", 2024-02-12T16:46:53.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1704000000000001e-05 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.241728+0000", 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051553720000000001 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.246883+0000", 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7868999999999999e-05 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.246921+0000", 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056339500000000004 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.130 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297935 9.16 e1017/1016 9:68757d5c:::benchmark_data_smithi169_124168_object147251:head v 1017'16943, mlcod=1017'16936)", 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.243605+0000", 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.84229687399999997, 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059774210000000001, 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.243605+0000", 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.243605+0000", 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.243603+0000", 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2970000000000004e-06 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.243612+0000", 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3508e-05 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.243625+0000", 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0979999999999999e-06 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.243631+0000", 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062765800000000001 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.244259+0000", 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5698e-05 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.244285+0000", 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052635260000000001 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.249548+0000", 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3992999999999998e-05 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.249582+0000", 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059774210000000001 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297936 9.16 e1017/1016 9:6d113ee6:::benchmark_data_smithi169_124168_object147252:head v 1017'16944, mlcod=1017'16936)", 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.245837+0000", 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.84006445799999996, 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039412099999999997, 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.245837+0000", 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.245837+0000", 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.245836+0000", 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8040000000000007e-06 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.245844+0000", 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3276e-05 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.245858+0000", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7459999999999997e-06 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.245863+0000", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001088107 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.246952+0000", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2626e-05 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.246974+0000", 2024-02-12T16:46:53.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027708950000000002 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.249745+0000", 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3417999999999998e-05 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.249778+0000", 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039412099999999997 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297952 9.16 e1017/1016 9:6983460e:::benchmark_data_smithi169_124168_object147268:head v 1017'16945, mlcod=1017'16938)", 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.252938+0000", 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.83296350200000002, 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055668219999999999, 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252938+0000", 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252938+0000", 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252936+0000", 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0530000000000001e-05 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252947+0000", 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5062000000000001e-05 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252962+0000", 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4200000000000004e-06 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252968+0000", 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9604000000000002e-05 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.253008+0000", 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8464000000000003e-05 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.253046+0000", 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054274960000000004 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.258474+0000", 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1293e-05 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.258505+0000", 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0055668219999999999 2024-02-12T16:46:53.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297979 9.16 e1017/1016 9:689ed4f1:::benchmark_data_smithi169_124168_object147295:head v 1017'16946, mlcod=1017'16938)", 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.252979+0000", 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.83292233999999998, 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084652709999999999, 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252979+0000", 2024-02-12T16:46:53.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252979+0000", 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252979+0000", 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2859999999999999e-06 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252984+0000", 2024-02-12T16:46:53.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3670000000000002e-06 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252990+0000", 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.252994+0000", 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029683740000000002 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.255962+0000", 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.012e-05 2024-02-12T16:46:53.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.255982+0000", 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0054235120000000001 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.261406+0000", 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8702999999999999e-05 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.261445+0000", 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0084652709999999999 2024-02-12T16:46:53.146 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:297988 9.16 e1017/1016 9:6d9484f8:::benchmark_data_smithi169_124168_object147304:head v 1017'16947, mlcod=1017'16939)", 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.257518+0000", 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.82838345899999999, 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040009499999999996, 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.257518+0000", 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.257518+0000", 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.257516+0000", 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6180000000000001e-06 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.257526+0000", 2024-02-12T16:46:53.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3431999999999999e-05 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.257540+0000", 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9580000000000004e-06 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.257546+0000", 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099531500000000009 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.258541+0000", 2024-02-12T16:46:53.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.755e-05 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.258558+0000", 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029371760000000001 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.261496+0000", 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3655e-05 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.261519+0000", 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040009499999999996 2024-02-12T16:46:53.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:298238 9.16 e1017/1016 9:6995aa47:::benchmark_data_smithi169_124168_object147554:head v 1017'16953, mlcod=1017'16950)", 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.436592+0000", 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.64931017499999999, 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006216938, 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436592+0000", 2024-02-12T16:46:53.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436592+0000", 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436590+0000", 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2019999999999993e-06 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436599+0000", 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5254e-05 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436614+0000", 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6340000000000003e-06 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436621+0000", 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3155000000000001e-05 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436684+0000", 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4685000000000002e-05 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.436709+0000", 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060575589999999997 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.442766+0000", 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2116999999999997e-05 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.442808+0000", 2024-02-12T16:46:53.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006216938 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:298239 9.16 e1017/1016 9:6d14031c:::benchmark_data_smithi169_124168_object147555:head v 1017'16954, mlcod=1017'16950)", 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.438615+0000", 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.64728662199999998, 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004294656, 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.438615+0000", 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.438615+0000", 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.438614+0000", 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1080000000000004e-06 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.438621+0000", 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8789999999999998e-06 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.438631+0000", 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.08e-06 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.438638+0000", 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010995 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.439737+0000", 2024-02-12T16:46:53.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8736000000000002e-05 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.439756+0000", 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031277200000000001 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.442884+0000", 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5882999999999999e-05 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.442910+0000", 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004294656 2024-02-12T16:46:53.158 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:298379 9.16 e1017/1016 9:6c63ac4a:::benchmark_data_smithi169_124168_object147695:head v 1017'16956, mlcod=1017'16954)", 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.543606+0000", 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.54229551300000001, 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051470070000000003, 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543606+0000", 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543606+0000", 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543604+0000", 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.132e-05 2024-02-12T16:46:53.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543616+0000", 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.482e-05 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543630+0000", 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.041e-06 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543637+0000", 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8993999999999999e-05 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543686+0000", 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3839999999999999e-05 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.543710+0000", 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050098230000000001 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.548720+0000", 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3147e-05 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.548753+0000", 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051470070000000003 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:298381 9.16 e1017/1016 9:6f17596c:::benchmark_data_smithi169_124168_object147697:head v 1017'16957, mlcod=1017'16954)", 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.544699+0000", 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.54120310400000005, 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041774910000000002, 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.544699+0000", 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.544699+0000", 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.544697+0000", 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9079999999999995e-06 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.544705+0000", 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1708e-05 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.544717+0000", 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9610000000000002e-06 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.544723+0000", 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016998359999999999 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.546423+0000", 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6254999999999999e-05 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.546439+0000", 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0024069970000000001 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.548846+0000", 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0111999999999999e-05 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.548876+0000", 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041774910000000002 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:298760 9.16 e1017/1016 9:6d024baa:::benchmark_data_smithi169_124168_object148076:head v 1017'16975, mlcod=1017'16972)", 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.820863+0000", 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.26503837600000002, 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004045814, 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.820863+0000", 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.820863+0000", 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.820860+0000", 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.738e-05 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.820877+0000", 2024-02-12T16:46:53.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7504e-05 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.820905+0000", 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2548000000000001e-05 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.820917+0000", 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000173912 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.821091+0000", 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1227999999999999e-05 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.821133+0000", 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037357319999999999 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.824868+0000", 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0883999999999997e-05 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.824909+0000", 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004045814 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.170 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:298874 9.16 e1017/1016 9:6dc2c573:::benchmark_data_smithi169_124168_object148190:head v 1017'16980, mlcod=1017'16977)", 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.902436+0000", 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.18346525899999999, 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062557339999999998, 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902436+0000", 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902436+0000", 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902434+0000", 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6646e-05 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902451+0000", 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0129999999999999e-05 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902471+0000", 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2770000000000005e-06 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902479+0000", 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3707000000000004e-05 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902553+0000", 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8469000000000001e-05 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.902581+0000", 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.005955241 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.908536+0000", 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015572399999999999 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.908692+0000", 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0062557339999999998 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:298875 9.16 e1017/1016 9:6cdeb9cc:::benchmark_data_smithi169_124168_object148191:head v 1017'16981, mlcod=1017'16977)", 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:52.905411+0000", 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.18049095500000001, 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056879139999999996, 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905411+0000", 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905411+0000", 2024-02-12T16:46:53.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905408+0000", 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2125e-05 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905421+0000", 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6416999999999999e-05 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905437+0000", 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5329999999999997e-06 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905444+0000", 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000362825 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905807+0000", 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1197999999999999e-05 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:46:53.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.905828+0000", 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052359980000000004 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.911064+0000", 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4180999999999997e-05 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:52.911099+0000", 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0056879139999999996 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:46:53.178 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:46:53.179 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:46:53.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:53.204 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:53.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:53.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:53.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:53 smithi121 bash[18489]: cluster 2024-02-12T16:46:52.032155+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e1017: 12 total, 12 up, 10 in 2024-02-12T16:46:53.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:53 smithi121 bash[18489]: cluster 2024-02-12T16:46:52.033596+0000 osd.1 (osd.1) 178 : cluster [DBG] 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 1014'16929 2024-02-12T16:46:53.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:53 smithi121 bash[18489]: cluster 2024-02-12T16:46:52.046087+0000 osd.1 (osd.1) 179 : cluster [DBG] 9.16 starting backfill to osd.11 from (0'0,0'0] MAX to 1014'16929 2024-02-12T16:46:53.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:53 smithi121 bash[18489]: cluster 2024-02-12T16:46:52.332961+0000 mgr.y (mgr.14146) 1880 : cluster [DBG] pgmap v2764: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 273 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s; 118/8742 objects misplaced (1.350%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:46:53.338 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:53.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:53.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:53 smithi123 bash[19119]: cluster 2024-02-12T16:46:52.032155+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e1017: 12 total, 12 up, 10 in 2024-02-12T16:46:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:53 smithi123 bash[19119]: cluster 2024-02-12T16:46:52.033596+0000 osd.1 (osd.1) 178 : cluster [DBG] 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 1014'16929 2024-02-12T16:46:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:53 smithi123 bash[19119]: cluster 2024-02-12T16:46:52.046087+0000 osd.1 (osd.1) 179 : cluster [DBG] 9.16 starting backfill to osd.11 from (0'0,0'0] MAX to 1014'16929 2024-02-12T16:46:53.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:53 smithi123 bash[19119]: cluster 2024-02-12T16:46:52.332961+0000 mgr.y (mgr.14146) 1880 : cluster [DBG] pgmap v2764: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 273 MiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.41k op/s; 118/8742 objects misplaced (1.350%); 2.4 MiB/s, 38 objects/s recovering 2024-02-12T16:46:53.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:53.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:53.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:53.606 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:53.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:53.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:53.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:53.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:53.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:53.882 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:53.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:53.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:54.051 INFO:teuthology.orchestra.run.smithi121.stderr:set 1.9 pg_upmap_items mapping to [0->1,5->10] 2024-02-12T16:46:54.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-12T16:46:54.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:54.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:54 smithi169 bash[19604]: cluster 2024-02-12T16:46:53.039497+0000 mon.a (mon.0) 3342 : cluster [DBG] osdmap e1018: 12 total, 12 up, 10 in 2024-02-12T16:46:54.134 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:54 smithi169 bash[19604]: audit 2024-02-12T16:46:53.369128+0000 mon.a (mon.0) 3343 : audit [INF] from='client.? v1:172.21.15.121:0/938952488' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [0, 1, 5, 10]}]: dispatch 2024-02-12T16:46:54.161 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:54.266 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:54.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:54.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:54 smithi123 bash[19119]: cluster 2024-02-12T16:46:53.039497+0000 mon.a (mon.0) 3342 : cluster [DBG] osdmap e1018: 12 total, 12 up, 10 in 2024-02-12T16:46:54.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:54 smithi123 bash[19119]: audit 2024-02-12T16:46:53.369128+0000 mon.a (mon.0) 3343 : audit [INF] from='client.? v1:172.21.15.121:0/938952488' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [0, 1, 5, 10]}]: dispatch 2024-02-12T16:46:54.406 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:54 smithi121 bash[18489]: cluster 2024-02-12T16:46:53.039497+0000 mon.a (mon.0) 3342 : cluster [DBG] osdmap e1018: 12 total, 12 up, 10 in 2024-02-12T16:46:54.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:54 smithi121 bash[18489]: audit 2024-02-12T16:46:53.369128+0000 mon.a (mon.0) 3343 : audit [INF] from='client.? v1:172.21.15.121:0/938952488' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [0, 1, 5, 10]}]: dispatch 2024-02-12T16:46:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:54.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:54.561 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:54.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:54.687 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:54.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:54.824 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:54.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:54.966 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:54.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:55.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:55.093 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:55.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:55 smithi121 bash[18489]: audit 2024-02-12T16:46:54.042774+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? v1:172.21.15.121:0/938952488' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [0, 1, 5, 10]}]': finished 2024-02-12T16:46:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:55 smithi121 bash[18489]: cluster 2024-02-12T16:46:54.043047+0000 mon.a (mon.0) 3345 : cluster [DBG] osdmap e1019: 12 total, 12 up, 10 in 2024-02-12T16:46:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:55 smithi121 bash[18489]: cluster 2024-02-12T16:46:54.333375+0000 mgr.y (mgr.14146) 1881 : cluster [DBG] pgmap v2767: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 174 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s; 118/5582 objects misplaced (2.114%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:46:55.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:55 smithi121 bash[18489]: cluster 2024-02-12T16:46:55.046333+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e1020: 12 total, 12 up, 10 in 2024-02-12T16:46:55.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:55.219 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:55.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:55.285 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:55.285 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:46:55.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:55.331 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:55 smithi169 bash[19604]: audit 2024-02-12T16:46:54.042774+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? v1:172.21.15.121:0/938952488' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [0, 1, 5, 10]}]': finished 2024-02-12T16:46:55.331 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:55 smithi169 bash[19604]: cluster 2024-02-12T16:46:54.043047+0000 mon.a (mon.0) 3345 : cluster [DBG] osdmap e1019: 12 total, 12 up, 10 in 2024-02-12T16:46:55.331 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:55 smithi169 bash[19604]: cluster 2024-02-12T16:46:54.333375+0000 mgr.y (mgr.14146) 1881 : cluster [DBG] pgmap v2767: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 174 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s; 118/5582 objects misplaced (2.114%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:46:55.331 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:55 smithi169 bash[19604]: cluster 2024-02-12T16:46:55.046333+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e1020: 12 total, 12 up, 10 in 2024-02-12T16:46:55.349 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:55.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:55.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:55 smithi123 bash[19119]: audit 2024-02-12T16:46:54.042774+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? v1:172.21.15.121:0/938952488' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [0, 1, 5, 10]}]': finished 2024-02-12T16:46:55.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:55 smithi123 bash[19119]: cluster 2024-02-12T16:46:54.043047+0000 mon.a (mon.0) 3345 : cluster [DBG] osdmap e1019: 12 total, 12 up, 10 in 2024-02-12T16:46:55.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:55 smithi123 bash[19119]: cluster 2024-02-12T16:46:54.333375+0000 mgr.y (mgr.14146) 1881 : cluster [DBG] pgmap v2767: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 174 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s; 118/5582 objects misplaced (2.114%); 1.9 MiB/s, 29 objects/s recovering 2024-02-12T16:46:55.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:55 smithi123 bash[19119]: cluster 2024-02-12T16:46:55.046333+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e1020: 12 total, 12 up, 10 in 2024-02-12T16:46:55.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:55.491 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:46:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:55.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:46:55.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:55.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:55.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:55.759 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:55.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:55.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:55.891 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:55.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:55.939 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Removed 150681 objects 2024-02-12T16:46:55.939 INFO:tasks.radosbench.radosbench.2.smithi169.stdout:Clean up completed and total clean up time :168.275896 2024-02-12T16:46:55.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:56.028 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:56.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:56.161 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:56.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:56.300 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:56.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:56.437 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:56.541 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:56.575 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:56.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:56.706 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:46:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:46:56.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:56.841 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:46:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:46:56.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:56.979 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:56.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:57.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:57.107 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:57.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:57.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:57.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:57 smithi121 bash[18489]: cluster 2024-02-12T16:46:56.033959+0000 mon.a (mon.0) 3347 : cluster [DBG] osdmap e1021: 12 total, 12 up, 10 in 2024-02-12T16:46:57.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:57 smithi121 bash[18489]: cluster 2024-02-12T16:46:56.334074+0000 mgr.y (mgr.14146) 1882 : cluster [DBG] pgmap v2770: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 150 MiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 491 op/s; 118/4808 objects misplaced (2.454%) 2024-02-12T16:46:57.245 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:57.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:57 smithi169 bash[19604]: cluster 2024-02-12T16:46:56.033959+0000 mon.a (mon.0) 3347 : cluster [DBG] osdmap e1021: 12 total, 12 up, 10 in 2024-02-12T16:46:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:57 smithi169 bash[19604]: cluster 2024-02-12T16:46:56.334074+0000 mgr.y (mgr.14146) 1882 : cluster [DBG] pgmap v2770: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 150 MiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 491 op/s; 118/4808 objects misplaced (2.454%) 2024-02-12T16:46:57.350 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:57.363 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:57 smithi123 bash[19119]: cluster 2024-02-12T16:46:56.033959+0000 mon.a (mon.0) 3347 : cluster [DBG] osdmap e1021: 12 total, 12 up, 10 in 2024-02-12T16:46:57.363 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:57 smithi123 bash[19119]: cluster 2024-02-12T16:46:56.334074+0000 mgr.y (mgr.14146) 1882 : cluster [DBG] pgmap v2770: 57 pgs: 1 remapped+peering, 3 active+remapped+backfilling, 53 active+clean; 150 MiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 491 op/s; 118/4808 objects misplaced (2.454%) 2024-02-12T16:46:57.387 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:57.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:57.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:57.529 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:57.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:57.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:57.672 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:57.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:57.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:57.813 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:46:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:57.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:46:57.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:57.954 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:57.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:58.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:58.101 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:46:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:58.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:46:58.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:58.234 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:58.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:58.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: cluster 2024-02-12T16:46:57.046644+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e1022: 12 total, 12 up, 10 in 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: audit 2024-02-12T16:46:57.511851+0000 mon.a (mon.0) 3349 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: audit 2024-02-12T16:46:57.512466+0000 mon.a (mon.0) 3350 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: audit 2024-02-12T16:46:57.512990+0000 mon.a (mon.0) 3351 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: audit 2024-02-12T16:46:57.575412+0000 mon.a (mon.0) 3352 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: audit 2024-02-12T16:46:57.578904+0000 mon.a (mon.0) 3353 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: audit 2024-02-12T16:46:57.580525+0000 mon.a (mon.0) 3354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:58.351 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:58 smithi169 bash[19604]: audit 2024-02-12T16:46:57.590290+0000 mon.a (mon.0) 3355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:46:58.376 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:58.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: cluster 2024-02-12T16:46:57.046644+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e1022: 12 total, 12 up, 10 in 2024-02-12T16:46:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: audit 2024-02-12T16:46:57.511851+0000 mon.a (mon.0) 3349 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:46:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: audit 2024-02-12T16:46:57.512466+0000 mon.a (mon.0) 3350 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:46:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: audit 2024-02-12T16:46:57.512990+0000 mon.a (mon.0) 3351 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: audit 2024-02-12T16:46:57.575412+0000 mon.a (mon.0) 3352 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: audit 2024-02-12T16:46:57.578904+0000 mon.a (mon.0) 3353 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: audit 2024-02-12T16:46:57.580525+0000 mon.a (mon.0) 3354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:58.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:58 smithi123 bash[19119]: audit 2024-02-12T16:46:57.590290+0000 mon.a (mon.0) 3355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:46:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: cluster 2024-02-12T16:46:57.046644+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e1022: 12 total, 12 up, 10 in 2024-02-12T16:46:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: audit 2024-02-12T16:46:57.511851+0000 mon.a (mon.0) 3349 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:46:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: audit 2024-02-12T16:46:57.512466+0000 mon.a (mon.0) 3350 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:46:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: audit 2024-02-12T16:46:57.512990+0000 mon.a (mon.0) 3351 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:46:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: audit 2024-02-12T16:46:57.575412+0000 mon.a (mon.0) 3352 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:46:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: audit 2024-02-12T16:46:57.578904+0000 mon.a (mon.0) 3353 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:46:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: audit 2024-02-12T16:46:57.580525+0000 mon.a (mon.0) 3354 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:46:58.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:58 smithi121 bash[18489]: audit 2024-02-12T16:46:57.590290+0000 mon.a (mon.0) 3355 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:46:58.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:58.517 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:58.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:58.620 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:58.659 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:58.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:58.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:58.802 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:46:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:58.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:46:58.898 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:46:58.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:58.937 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:46:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:58.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:46:58.949 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-02-12T16:46:58.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-02-12T16:46:59.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:59.074 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:46:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:59.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:46:59.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:59.207 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:59.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:59.310 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:59.323 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:46:59 smithi123 bash[19119]: cluster 2024-02-12T16:46:58.334597+0000 mgr.y (mgr.14146) 1883 : cluster [DBG] pgmap v2772: 57 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 55 active+clean; 63 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 602 op/s; 136/2004 objects misplaced (6.786%); 179 KiB/s, 2 objects/s recovering 2024-02-12T16:46:59.347 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:46:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:59.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:46:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:46:59 smithi121 bash[18489]: cluster 2024-02-12T16:46:58.334597+0000 mgr.y (mgr.14146) 1883 : cluster [DBG] pgmap v2772: 57 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 55 active+clean; 63 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 602 op/s; 136/2004 objects misplaced (6.786%); 179 KiB/s, 2 objects/s recovering 2024-02-12T16:46:59.450 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:59.486 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:46:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:59.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:46:59.581 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:46:59 smithi169 bash[19604]: cluster 2024-02-12T16:46:58.334597+0000 mgr.y (mgr.14146) 1883 : cluster [DBG] pgmap v2772: 57 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 55 active+clean; 63 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 602 op/s; 136/2004 objects misplaced (6.786%); 179 KiB/s, 2 objects/s recovering 2024-02-12T16:46:59.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:59.626 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:59.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:59.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:59.763 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:46:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:59.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:46:59.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 10, 8, 11] out_osds: [9, 6] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 3, 7] 2024-02-12T16:46:59.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-12T16:46:59.864 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-12T16:46:59.865 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-12T16:46:59.865 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-12T16:46:59.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-12T16:46:59.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:59.895 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:46:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:46:59.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:46:59.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:00.035 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:00.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:00.139 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:00.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:00 smithi121 bash[18489]: cluster 2024-02-12T16:46:59.043326+0000 mon.a (mon.0) 3356 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:00.176 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:00.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:00.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:00.320 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:00.322 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:00 smithi169 bash[19604]: cluster 2024-02-12T16:46:59.043326+0000 mon.a (mon.0) 3356 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:00.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:00.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:00 smithi123 bash[19119]: cluster 2024-02-12T16:46:59.043326+0000 mon.a (mon.0) 3356 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:00.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:00.464 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:00.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:00.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:00.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:00.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:00.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:00.735 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:00.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:00.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:00.864 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:00.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:00.921 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:47:00.941 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:47:00.959 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:47:00.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:00.981 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:47:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:01.013 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:47:01.040 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:47:01.070 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:47:01.091 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:47:01.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:01.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:47:01.132 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:47:01.137 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:47:01.161 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:47:01.183 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:47:01.185 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.185 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.185 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.186 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'false' 2024-02-12T16:47:01.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:01 smithi121 bash[18489]: cluster 2024-02-12T16:47:00.335483+0000 mgr.y (mgr.14146) 1884 : cluster [DBG] pgmap v2773: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 17 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 457 op/s; 18/546 objects misplaced (3.297%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:47:01.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:01 smithi121 bash[18489]: audit 2024-02-12T16:47:00.337001+0000 mon.a (mon.0) 3357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:47:01.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:01 smithi121 bash[18489]: audit 2024-02-12T16:47:00.884388+0000 mon.a (mon.0) 3358 : audit [DBG] from='client.? v1:172.21.15.121:0/3336082522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:47:01.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:01 smithi121 bash[18489]: audit 2024-02-12T16:47:01.038943+0000 mon.a (mon.0) 3359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:47:01.232 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:01 smithi121 bash[18489]: cluster 2024-02-12T16:47:01.039019+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e1023: 12 total, 12 up, 10 in 2024-02-12T16:47:01.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:01.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:01.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:01 smithi123 bash[19119]: cluster 2024-02-12T16:47:00.335483+0000 mgr.y (mgr.14146) 1884 : cluster [DBG] pgmap v2773: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 17 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 457 op/s; 18/546 objects misplaced (3.297%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:47:01.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:01 smithi123 bash[19119]: audit 2024-02-12T16:47:00.337001+0000 mon.a (mon.0) 3357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:47:01.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:01 smithi123 bash[19119]: audit 2024-02-12T16:47:00.884388+0000 mon.a (mon.0) 3358 : audit [DBG] from='client.? v1:172.21.15.121:0/3336082522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:47:01.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:01 smithi123 bash[19119]: audit 2024-02-12T16:47:01.038943+0000 mon.a (mon.0) 3359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:47:01.382 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:01 smithi123 bash[19119]: cluster 2024-02-12T16:47:01.039019+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e1023: 12 total, 12 up, 10 in 2024-02-12T16:47:01.407 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:01.512 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:01.525 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:01 smithi169 bash[19604]: cluster 2024-02-12T16:47:00.335483+0000 mgr.y (mgr.14146) 1884 : cluster [DBG] pgmap v2773: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 17 MiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 457 op/s; 18/546 objects misplaced (3.297%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:47:01.525 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:01 smithi169 bash[19604]: audit 2024-02-12T16:47:00.337001+0000 mon.a (mon.0) 3357 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]: dispatch 2024-02-12T16:47:01.525 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:01 smithi169 bash[19604]: audit 2024-02-12T16:47:00.884388+0000 mon.a (mon.0) 3358 : audit [DBG] from='client.? v1:172.21.15.121:0/3336082522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:47:01.525 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:01 smithi169 bash[19604]: audit 2024-02-12T16:47:01.038943+0000 mon.a (mon.0) 3359 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]': finished 2024-02-12T16:47:01.525 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:01 smithi169 bash[19604]: cluster 2024-02-12T16:47:01.039019+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e1023: 12 total, 12 up, 10 in 2024-02-12T16:47:01.547 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:01.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:01.679 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:01.782 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:01.819 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:01.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:01.960 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:01.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:02.062 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is set 2024-02-12T16:47:02.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:02.088 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:02.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:02.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:02.225 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:02.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:02.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:02.341 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:02 smithi123 bash[19119]: audit 2024-02-12T16:47:01.236762+0000 mon.a (mon.0) 3361 : audit [INF] from='client.? v1:172.21.15.121:0/1297713892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:47:02.342 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:02 smithi123 bash[19119]: cluster 2024-02-12T16:47:02.043806+0000 mon.a (mon.0) 3362 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:47:02.342 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:02 smithi123 bash[19119]: audit 2024-02-12T16:47:02.051815+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? v1:172.21.15.121:0/1297713892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:47:02.342 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:02 smithi123 bash[19119]: cluster 2024-02-12T16:47:02.051972+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e1024: 12 total, 12 up, 10 in 2024-02-12T16:47:02.370 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:02.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:02 smithi121 bash[18489]: audit 2024-02-12T16:47:01.236762+0000 mon.a (mon.0) 3361 : audit [INF] from='client.? v1:172.21.15.121:0/1297713892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:47:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:02 smithi121 bash[18489]: cluster 2024-02-12T16:47:02.043806+0000 mon.a (mon.0) 3362 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:47:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:02 smithi121 bash[18489]: audit 2024-02-12T16:47:02.051815+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? v1:172.21.15.121:0/1297713892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:47:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:02 smithi121 bash[18489]: cluster 2024-02-12T16:47:02.051972+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e1024: 12 total, 12 up, 10 in 2024-02-12T16:47:02.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:02.486 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:02 smithi169 bash[19604]: audit 2024-02-12T16:47:01.236762+0000 mon.a (mon.0) 3361 : audit [INF] from='client.? v1:172.21.15.121:0/1297713892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:47:02.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:02 smithi169 bash[19604]: cluster 2024-02-12T16:47:02.043806+0000 mon.a (mon.0) 3362 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:47:02.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:02 smithi169 bash[19604]: audit 2024-02-12T16:47:02.051815+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? v1:172.21.15.121:0/1297713892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-12T16:47:02.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:02 smithi169 bash[19604]: cluster 2024-02-12T16:47:02.051972+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e1024: 12 total, 12 up, 10 in 2024-02-12T16:47:02.513 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:02.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:02.617 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:02.655 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:02.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:02.759 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:02.791 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:02.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:02.894 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:02.927 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:02.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:02.939 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:47:02.939 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:47:03.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:03.069 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:03.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:03.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:03 smithi121 bash[18489]: cluster 2024-02-12T16:47:02.191179+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:03 smithi121 bash[18489]: cluster 2024-02-12T16:47:02.191288+0000 osd.1 (osd.1) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:03 smithi121 bash[18489]: cluster 2024-02-12T16:47:02.336426+0000 mgr.y (mgr.14146) 1885 : cluster [DBG] pgmap v2776: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 404 op/s; 18/412 objects misplaced (4.369%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:47:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:03 smithi121 bash[18489]: audit 2024-02-12T16:47:02.337677+0000 mon.a (mon.0) 3365 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:47:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:03 smithi121 bash[18489]: audit 2024-02-12T16:47:03.053979+0000 mon.a (mon.0) 3366 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:47:03.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:03 smithi121 bash[18489]: cluster 2024-02-12T16:47:03.054126+0000 mon.a (mon.0) 3367 : cluster [DBG] osdmap e1025: 12 total, 12 up, 10 in 2024-02-12T16:47:03.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:03.202 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:03.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:47:03.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:03.342 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:03.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:03 smithi169 bash[19604]: cluster 2024-02-12T16:47:02.191179+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:03.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:03 smithi169 bash[19604]: cluster 2024-02-12T16:47:02.191288+0000 osd.1 (osd.1) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:03.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:03 smithi169 bash[19604]: cluster 2024-02-12T16:47:02.336426+0000 mgr.y (mgr.14146) 1885 : cluster [DBG] pgmap v2776: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 404 op/s; 18/412 objects misplaced (4.369%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:47:03.344 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:03 smithi169 bash[19604]: audit 2024-02-12T16:47:02.337677+0000 mon.a (mon.0) 3365 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:47:03.345 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:03 smithi169 bash[19604]: audit 2024-02-12T16:47:03.053979+0000 mon.a (mon.0) 3366 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:47:03.345 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:03 smithi169 bash[19604]: cluster 2024-02-12T16:47:03.054126+0000 mon.a (mon.0) 3367 : cluster [DBG] osdmap e1025: 12 total, 12 up, 10 in 2024-02-12T16:47:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:03.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:03 smithi123 bash[19119]: cluster 2024-02-12T16:47:02.191179+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:03 smithi123 bash[19119]: cluster 2024-02-12T16:47:02.191288+0000 osd.1 (osd.1) 181 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:03 smithi123 bash[19119]: cluster 2024-02-12T16:47:02.336426+0000 mgr.y (mgr.14146) 1885 : cluster [DBG] pgmap v2776: 57 pgs: 1 active+remapped+backfill_toofull, 56 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 404 op/s; 18/412 objects misplaced (4.369%); 448 KiB/s, 6 objects/s recovering 2024-02-12T16:47:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:03 smithi123 bash[19119]: audit 2024-02-12T16:47:02.337677+0000 mon.a (mon.0) 3365 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]: dispatch 2024-02-12T16:47:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:03 smithi123 bash[19119]: audit 2024-02-12T16:47:03.053979+0000 mon.a (mon.0) 3366 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]': finished 2024-02-12T16:47:03.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:03 smithi123 bash[19119]: cluster 2024-02-12T16:47:03.054126+0000 mon.a (mon.0) 3367 : cluster [DBG] osdmap e1025: 12 total, 12 up, 10 in 2024-02-12T16:47:03.446 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:03.484 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:03.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:03.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:03.622 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:03.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:03.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-12T16:47:03.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:03.760 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:03.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:03.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:03.904 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:03.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:04.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:04.045 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:04.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:47:04.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-12T16:47:04.149 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:04.183 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:04.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:47:04.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:04.323 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:04.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:04 smithi123 bash[19119]: cluster 2024-02-12T16:47:03.055869+0000 osd.7 (osd.7) 15 : cluster [DBG] 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 1020'17477 2024-02-12T16:47:04.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:04 smithi123 bash[19119]: cluster 2024-02-12T16:47:03.068082+0000 osd.7 (osd.7) 16 : cluster [DBG] 9.17 starting backfill to osd.8 from (0'0,0'0] MAX to 1020'17477 2024-02-12T16:47:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:04 smithi121 bash[18489]: cluster 2024-02-12T16:47:03.055869+0000 osd.7 (osd.7) 15 : cluster [DBG] 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 1020'17477 2024-02-12T16:47:04.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:04 smithi121 bash[18489]: cluster 2024-02-12T16:47:03.068082+0000 osd.7 (osd.7) 16 : cluster [DBG] 9.17 starting backfill to osd.8 from (0'0,0'0] MAX to 1020'17477 2024-02-12T16:47:04.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:04.454 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:04.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:47:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:04 smithi169 bash[19604]: cluster 2024-02-12T16:47:03.055869+0000 osd.7 (osd.7) 15 : cluster [DBG] 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 1020'17477 2024-02-12T16:47:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:04 smithi169 bash[19604]: cluster 2024-02-12T16:47:03.068082+0000 osd.7 (osd.7) 16 : cluster [DBG] 9.17 starting backfill to osd.8 from (0'0,0'0] MAX to 1020'17477 2024-02-12T16:47:04.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:04.601 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:04.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:47:04.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:04.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:04.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:04.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:47:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:04.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:04.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:04.987 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:05.025 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:05.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:05.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:05.168 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:05.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:47:05.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:05 smithi169 bash[19604]: cluster 2024-02-12T16:47:04.082792+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e1026: 12 total, 12 up, 10 in 2024-02-12T16:47:05.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:05 smithi169 bash[19604]: cluster 2024-02-12T16:47:04.337481+0000 mgr.y (mgr.14146) 1886 : cluster [DBG] pgmap v2779: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 16 op/s; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:05.312 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:05.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:05 smithi123 bash[19119]: cluster 2024-02-12T16:47:04.082792+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e1026: 12 total, 12 up, 10 in 2024-02-12T16:47:05.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:05 smithi123 bash[19119]: cluster 2024-02-12T16:47:04.337481+0000 mgr.y (mgr.14146) 1886 : cluster [DBG] pgmap v2779: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 16 op/s; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:05 smithi121 bash[18489]: cluster 2024-02-12T16:47:04.082792+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e1026: 12 total, 12 up, 10 in 2024-02-12T16:47:05.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:05 smithi121 bash[18489]: cluster 2024-02-12T16:47:04.337481+0000 mgr.y (mgr.14146) 1886 : cluster [DBG] pgmap v2779: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 16 op/s; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:05.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:05.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:47:05.556 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:05.594 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:05.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:47:05.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:05.734 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:05.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:05.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:05.873 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:05.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:05.881 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:05.881 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:05.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:06.013 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:06.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:06.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:06.150 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:06.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:06.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:06.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:06 smithi121 bash[18489]: cluster 2024-02-12T16:47:05.085401+0000 mon.a (mon.0) 3369 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:06.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:06 smithi121 bash[18489]: cluster 2024-02-12T16:47:05.096793+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e1027: 12 total, 12 up, 10 in 2024-02-12T16:47:06.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:06 smithi121 bash[18489]: cluster 2024-02-12T16:47:05.102387+0000 osd.5 (osd.5) 225 : cluster [DBG] 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 1020'16987 2024-02-12T16:47:06.267 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:06 smithi121 bash[18489]: cluster 2024-02-12T16:47:05.118229+0000 osd.5 (osd.5) 226 : cluster [DBG] 9.18 starting backfill to osd.10 from (0'0,0'0] MAX to 1020'16987 2024-02-12T16:47:06.295 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:06.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:06 smithi123 bash[19119]: cluster 2024-02-12T16:47:05.085401+0000 mon.a (mon.0) 3369 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:06 smithi123 bash[19119]: cluster 2024-02-12T16:47:05.096793+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e1027: 12 total, 12 up, 10 in 2024-02-12T16:47:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:06 smithi123 bash[19119]: cluster 2024-02-12T16:47:05.102387+0000 osd.5 (osd.5) 225 : cluster [DBG] 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 1020'16987 2024-02-12T16:47:06.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:06 smithi123 bash[19119]: cluster 2024-02-12T16:47:05.118229+0000 osd.5 (osd.5) 226 : cluster [DBG] 9.18 starting backfill to osd.10 from (0'0,0'0] MAX to 1020'16987 2024-02-12T16:47:06.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:06.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:06 smithi169 bash[19604]: cluster 2024-02-12T16:47:05.085401+0000 mon.a (mon.0) 3369 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:06.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:06 smithi169 bash[19604]: cluster 2024-02-12T16:47:05.096793+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e1027: 12 total, 12 up, 10 in 2024-02-12T16:47:06.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:06 smithi169 bash[19604]: cluster 2024-02-12T16:47:05.102387+0000 osd.5 (osd.5) 225 : cluster [DBG] 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 1020'16987 2024-02-12T16:47:06.411 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:06 smithi169 bash[19604]: cluster 2024-02-12T16:47:05.118229+0000 osd.5 (osd.5) 226 : cluster [DBG] 9.18 starting backfill to osd.10 from (0'0,0'0] MAX to 1020'16987 2024-02-12T16:47:06.437 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:06.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:47:06.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:06.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:06.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:06.726 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:06.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:06.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:06.863 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:06.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:06.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:07.004 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:07.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:07.138 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:07.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:07.278 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:07.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:07 smithi123 bash[19119]: cluster 2024-02-12T16:47:06.338289+0000 mgr.y (mgr.14146) 1887 : cluster [DBG] pgmap v2781: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:07.392 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:07 smithi169 bash[19604]: cluster 2024-02-12T16:47:06.338289+0000 mgr.y (mgr.14146) 1887 : cluster [DBG] pgmap v2781: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:07 smithi121 bash[18489]: cluster 2024-02-12T16:47:06.338289+0000 mgr.y (mgr.14146) 1887 : cluster [DBG] pgmap v2781: 57 pgs: 2 active+remapped+backfill_toofull, 55 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:07.419 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:07.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:07.560 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:07.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:07.704 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:07.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:07.840 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:47:07.912 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:07.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:07.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:08.086 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:08.125 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:08.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:08.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:08.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:08.266 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:08.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:08.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:08.410 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:08.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:08.502 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:47:08.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:08.549 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:08.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:08.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:08.688 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:08.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:08.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:08.832 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:08.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:08.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:08.975 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:08.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:09.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:09.118 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:09.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:09.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:09.260 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:09.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:09.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:09.401 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:09.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:09.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:09.538 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:09.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:09.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:09 smithi123 bash[19119]: cluster 2024-02-12T16:47:08.339158+0000 mgr.y (mgr.14146) 1888 : cluster [DBG] pgmap v2782: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:09.642 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:09.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:09 smithi121 bash[18489]: cluster 2024-02-12T16:47:08.339158+0000 mgr.y (mgr.14146) 1888 : cluster [DBG] pgmap v2782: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:09.676 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:09.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:47:09.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:09 smithi169 bash[19604]: cluster 2024-02-12T16:47:08.339158+0000 mgr.y (mgr.14146) 1888 : cluster [DBG] pgmap v2782: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:09.822 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:47:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:09.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1' 2024-02-12T16:47:09.926 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:09.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:10.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:10.104 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:10.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:10.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:10.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:10.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:10.385 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:47:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:10.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7' 2024-02-12T16:47:10.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:10.527 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:10.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:10.565 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:10.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:10.669 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:10.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:10.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:10.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:47:10.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:10.949 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:10.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:11.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:11.085 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:11.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:11.189 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:11.223 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:11.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:11.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:11.365 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:11.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:47:11.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:11.504 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:11.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:11.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:11 smithi123 bash[19119]: cluster 2024-02-12T16:47:10.340212+0000 mgr.y (mgr.14146) 1889 : cluster [DBG] pgmap v2783: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:11 smithi123 bash[19119]: cluster 2024-02-12T16:47:11.036921+0000 mon.a (mon.0) 3371 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:11.646 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:11.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:11 smithi121 bash[18489]: cluster 2024-02-12T16:47:10.340212+0000 mgr.y (mgr.14146) 1889 : cluster [DBG] pgmap v2783: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:11 smithi121 bash[18489]: cluster 2024-02-12T16:47:11.036921+0000 mon.a (mon.0) 3371 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:11.750 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:11.762 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:11 smithi169 bash[19604]: cluster 2024-02-12T16:47:10.340212+0000 mgr.y (mgr.14146) 1889 : cluster [DBG] pgmap v2783: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:11.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:11 smithi169 bash[19604]: cluster 2024-02-12T16:47:11.036921+0000 mon.a (mon.0) 3371 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-12T16:47:11.787 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:11.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:47:11.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:11.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:11.988 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2024-02-12T16:47:11.988 INFO:teuthology.orchestra.run.smithi123.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-12T16:47:12.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:12.055 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:12.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:12.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:47:12.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:12.198 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:12.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:12.301 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:12.336 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:12.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:47:12.441 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:12.474 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:12.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:12.578 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:12.614 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:12.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:12.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-12T16:47:12.718 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:12.748 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:12.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:12.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:12.886 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:12.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:12.990 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:13.030 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:13.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:47:13.124 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "" 2024-02-12T16:47:13.124 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:47:13.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:13.166 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:13.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:13.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:13.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:13.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:13.443 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:13.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:13.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:13.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:13.582 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:13.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:13.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:13.698 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:13 smithi121 bash[18489]: cluster 2024-02-12T16:47:12.341191+0000 mgr.y (mgr.14146) 1890 : cluster [DBG] pgmap v2784: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:13.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:13.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:13 smithi169 bash[19604]: cluster 2024-02-12T16:47:12.341191+0000 mgr.y (mgr.14146) 1890 : cluster [DBG] pgmap v2784: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:13.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:13.862 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:13.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:13 smithi123 bash[19119]: cluster 2024-02-12T16:47:12.341191+0000 mgr.y (mgr.14146) 1890 : cluster [DBG] pgmap v2784: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:13.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:13.997 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:14.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:14.129 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:14.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:14.268 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:14.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:14.407 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'unique_pool_7' removed 2024-02-12T16:47:14.511 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:14.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:14.568 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:14 smithi121 bash[18489]: cluster 2024-02-12T16:47:13.501960+0000 osd.9 (osd.9) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:14.568 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:14 smithi121 bash[18489]: cluster 2024-02-12T16:47:13.502065+0000 osd.9 (osd.9) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:14.568 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:14 smithi121 bash[18489]: audit 2024-02-12T16:47:14.222658+0000 mon.a (mon.0) 3372 : audit [INF] from='client.? v1:172.21.15.121:0/400661602' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:47:14.643 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:14.677 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:14.679 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:14 smithi169 bash[19604]: cluster 2024-02-12T16:47:13.501960+0000 osd.9 (osd.9) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:14.679 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:14 smithi169 bash[19604]: cluster 2024-02-12T16:47:13.502065+0000 osd.9 (osd.9) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:14.679 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:14 smithi169 bash[19604]: audit 2024-02-12T16:47:14.222658+0000 mon.a (mon.0) 3372 : audit [INF] from='client.? v1:172.21.15.121:0/400661602' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:47:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:14.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:14.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:14 smithi123 bash[19119]: cluster 2024-02-12T16:47:13.501960+0000 osd.9 (osd.9) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:14.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:14 smithi123 bash[19119]: cluster 2024-02-12T16:47:13.502065+0000 osd.9 (osd.9) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:14.792 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:14 smithi123 bash[19119]: audit 2024-02-12T16:47:14.222658+0000 mon.a (mon.0) 3372 : audit [INF] from='client.? v1:172.21.15.121:0/400661602' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-12T16:47:14.814 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:47:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5' 2024-02-12T16:47:14.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:14.945 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:14.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:15.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:15.072 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:47:15.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:15.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0' 2024-02-12T16:47:15.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:15.212 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:15.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:15.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:15.354 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:15.356 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:15.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:15.493 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:47:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:15.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10' 2024-02-12T16:47:15.597 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:15.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:15 smithi169 bash[19604]: cluster 2024-02-12T16:47:14.342089+0000 mgr.y (mgr.14146) 1891 : cluster [DBG] pgmap v2785: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:15.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:15 smithi169 bash[19604]: audit 2024-02-12T16:47:14.416826+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? v1:172.21.15.121:0/400661602' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:47:15.609 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:15 smithi169 bash[19604]: cluster 2024-02-12T16:47:14.416924+0000 mon.a (mon.0) 3374 : cluster [DBG] osdmap e1028: 12 total, 12 up, 10 in 2024-02-12T16:47:15.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:15 smithi121 bash[18489]: cluster 2024-02-12T16:47:14.342089+0000 mgr.y (mgr.14146) 1891 : cluster [DBG] pgmap v2785: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:15.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:15 smithi121 bash[18489]: audit 2024-02-12T16:47:14.416826+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? v1:172.21.15.121:0/400661602' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:47:15.630 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:15 smithi121 bash[18489]: cluster 2024-02-12T16:47:14.416924+0000 mon.a (mon.0) 3374 : cluster [DBG] osdmap e1028: 12 total, 12 up, 10 in 2024-02-12T16:47:15.631 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:47:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:15.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9' 2024-02-12T16:47:15.735 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:15.773 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:15.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:15.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:15 smithi123 bash[19119]: cluster 2024-02-12T16:47:14.342089+0000 mgr.y (mgr.14146) 1891 : cluster [DBG] pgmap v2785: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 13 MiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 46/412 objects misplaced (11.165%) 2024-02-12T16:47:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:15 smithi123 bash[19119]: audit 2024-02-12T16:47:14.416826+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? v1:172.21.15.121:0/400661602' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-02-12T16:47:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:15 smithi123 bash[19119]: cluster 2024-02-12T16:47:14.416924+0000 mon.a (mon.0) 3374 : cluster [DBG] osdmap e1028: 12 total, 12 up, 10 in 2024-02-12T16:47:15.916 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:47:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:15.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8' 2024-02-12T16:47:16.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:16.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:47:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:16.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6' 2024-02-12T16:47:16.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:16.203 INFO:teuthology.orchestra.run.smithi169.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:47:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:16.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi169 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11' 2024-02-12T16:47:16.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:16.342 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:16.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:16.446 DEBUG:teuthology.orchestra.run.smithi123:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:16.484 INFO:teuthology.orchestra.run.smithi123.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:47:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:16.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi123 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4' 2024-02-12T16:47:16.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:16.625 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:47:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:16.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3' 2024-02-12T16:47:16.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:16.735 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-12T16:47:16.745 INFO:tasks.thrashosds:joining thrashosds 2024-02-12T16:47:16.762 INFO:teuthology.orchestra.run.smithi121.stderr:Error response from daemon: Cannot kill container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:47:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T16:47:16.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi121 with status 1: 'sudo docker kill -s 1 ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2' 2024-02-12T16:47:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:17 smithi121 bash[18489]: cluster 2024-02-12T16:47:16.047342+0000 mon.a (mon.0) 3375 : cluster [DBG] osdmap e1029: 12 total, 12 up, 10 in 2024-02-12T16:47:17.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:17 smithi121 bash[18489]: cluster 2024-02-12T16:47:16.342631+0000 mgr.y (mgr.14146) 1892 : cluster [DBG] pgmap v2788: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:17.372 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0: {} 2024-02-12T16:47:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:17 smithi123 bash[19119]: cluster 2024-02-12T16:47:16.047342+0000 mon.a (mon.0) 3375 : cluster [DBG] osdmap e1029: 12 total, 12 up, 10 in 2024-02-12T16:47:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:17 smithi123 bash[19119]: cluster 2024-02-12T16:47:16.342631+0000 mgr.y (mgr.14146) 1892 : cluster [DBG] pgmap v2788: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:17.393 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1: {} 2024-02-12T16:47:17.412 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2: {} 2024-02-12T16:47:17.434 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3: {} 2024-02-12T16:47:17.462 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:47:17.463 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4: {} 2024-02-12T16:47:17.487 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5: {} 2024-02-12T16:47:17.512 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6: {} 2024-02-12T16:47:17.535 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7: {} 2024-02-12T16:47:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:17 smithi169 bash[19604]: cluster 2024-02-12T16:47:16.047342+0000 mon.a (mon.0) 3375 : cluster [DBG] osdmap e1029: 12 total, 12 up, 10 in 2024-02-12T16:47:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:17 smithi169 bash[19604]: cluster 2024-02-12T16:47:16.342631+0000 mgr.y (mgr.14146) 1892 : cluster [DBG] pgmap v2788: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:17.559 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8: {} 2024-02-12T16:47:17.591 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9: {} 2024-02-12T16:47:17.615 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10: {} 2024-02-12T16:47:17.640 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11: {} 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.8: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.9: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.10: osd_enable_op_tracker = 'true' 2024-02-12T16:47:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:osd.11: osd_enable_op_tracker = 'true' 2024-02-12T16:47:18.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:18 smithi123 bash[19119]: cluster 2024-02-12T16:47:17.052062+0000 mon.a (mon.0) 3376 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-12T16:47:18.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:18 smithi123 bash[19119]: audit 2024-02-12T16:47:17.345194+0000 mon.a (mon.0) 3377 : audit [DBG] from='client.? v1:172.21.15.121:0/2588091226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:47:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:18 smithi121 bash[18489]: cluster 2024-02-12T16:47:17.052062+0000 mon.a (mon.0) 3376 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-12T16:47:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:18 smithi121 bash[18489]: audit 2024-02-12T16:47:17.345194+0000 mon.a (mon.0) 3377 : audit [DBG] from='client.? v1:172.21.15.121:0/2588091226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:47:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:18 smithi169 bash[19604]: cluster 2024-02-12T16:47:17.052062+0000 mon.a (mon.0) 3376 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-12T16:47:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:18 smithi169 bash[19604]: audit 2024-02-12T16:47:17.345194+0000 mon.a (mon.0) 3377 : audit [DBG] from='client.? v1:172.21.15.121:0/2588091226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-12T16:47:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:19 smithi123 bash[19119]: cluster 2024-02-12T16:47:17.745083+0000 osd.5 (osd.5) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:19 smithi123 bash[19119]: cluster 2024-02-12T16:47:17.745180+0000 osd.5 (osd.5) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:19 smithi123 bash[19119]: cluster 2024-02-12T16:47:18.060728+0000 mon.a (mon.0) 3378 : cluster [DBG] osdmap e1030: 12 total, 12 up, 10 in 2024-02-12T16:47:19.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:19 smithi123 bash[19119]: cluster 2024-02-12T16:47:18.343266+0000 mgr.y (mgr.14146) 1893 : cluster [DBG] pgmap v2790: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:19 smithi123 bash[19119]: cluster 2024-02-12T16:47:18.514828+0000 osd.2 (osd.2) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:19.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:19 smithi123 bash[19119]: cluster 2024-02-12T16:47:18.514940+0000 osd.2 (osd.2) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:19 smithi121 bash[18489]: cluster 2024-02-12T16:47:17.745083+0000 osd.5 (osd.5) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:19 smithi121 bash[18489]: cluster 2024-02-12T16:47:17.745180+0000 osd.5 (osd.5) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:19 smithi121 bash[18489]: cluster 2024-02-12T16:47:18.060728+0000 mon.a (mon.0) 3378 : cluster [DBG] osdmap e1030: 12 total, 12 up, 10 in 2024-02-12T16:47:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:19 smithi121 bash[18489]: cluster 2024-02-12T16:47:18.343266+0000 mgr.y (mgr.14146) 1893 : cluster [DBG] pgmap v2790: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:19 smithi121 bash[18489]: cluster 2024-02-12T16:47:18.514828+0000 osd.2 (osd.2) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:19.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:19 smithi121 bash[18489]: cluster 2024-02-12T16:47:18.514940+0000 osd.2 (osd.2) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:19 smithi169 bash[19604]: cluster 2024-02-12T16:47:17.745083+0000 osd.5 (osd.5) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:19 smithi169 bash[19604]: cluster 2024-02-12T16:47:17.745180+0000 osd.5 (osd.5) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:19.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:19 smithi169 bash[19604]: cluster 2024-02-12T16:47:18.060728+0000 mon.a (mon.0) 3378 : cluster [DBG] osdmap e1030: 12 total, 12 up, 10 in 2024-02-12T16:47:19.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:19 smithi169 bash[19604]: cluster 2024-02-12T16:47:18.343266+0000 mgr.y (mgr.14146) 1893 : cluster [DBG] pgmap v2790: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:19.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:19 smithi169 bash[19604]: cluster 2024-02-12T16:47:18.514828+0000 osd.2 (osd.2) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:19.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:19 smithi169 bash[19604]: cluster 2024-02-12T16:47:18.514940+0000 osd.2 (osd.2) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:20.084 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:47:20.342 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:20 smithi123 bash[19119]: cluster 2024-02-12T16:47:18.816200+0000 osd.4 (osd.4) 169 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:20.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:20 smithi123 bash[19119]: cluster 2024-02-12T16:47:18.816322+0000 osd.4 (osd.4) 170 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:20.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:20 smithi123 bash[19119]: cluster 2024-02-12T16:47:19.067107+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e1031: 12 total, 12 up, 10 in 2024-02-12T16:47:20.343 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:20 smithi123 bash[19119]: audit 2024-02-12T16:47:19.150713+0000 mon.a (mon.0) 3380 : audit [INF] from='client.? v1:172.21.15.121:0/1016116987' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:47:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:20 smithi121 bash[18489]: cluster 2024-02-12T16:47:18.816200+0000 osd.4 (osd.4) 169 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:20 smithi121 bash[18489]: cluster 2024-02-12T16:47:18.816322+0000 osd.4 (osd.4) 170 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:20 smithi121 bash[18489]: cluster 2024-02-12T16:47:19.067107+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e1031: 12 total, 12 up, 10 in 2024-02-12T16:47:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:20 smithi121 bash[18489]: audit 2024-02-12T16:47:19.150713+0000 mon.a (mon.0) 3380 : audit [INF] from='client.? v1:172.21.15.121:0/1016116987' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:47:20.508 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:47:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:47:20.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:283976 7.4 7:2245208a:::benchmark_data_smithi169_105252_object115382:head [delete] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:39.653372+0000", 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 580.84860930000002, 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046079847, 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 283976 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653372+0000", 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653372+0000", 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653369+0000", 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8270000000000004e-06 2024-02-12T16:47:20.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653378+0000", 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6291999999999999e-05 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653394+0000", 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6669999999999999e-06 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.653401+0000", 2024-02-12T16:47:20.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005732694 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.659134+0000", 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010278 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.659236+0000", 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106172 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:47:20.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.659343+0000", 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038289280000000001 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.663171+0000", 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013068400000000001 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.664478+0000", 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.034742268999999999 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.699221+0000", 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5315999999999993e-05 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.699316+0000", 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013549600000000001 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.699451+0000", 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.046079847 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:283989 7.4 7:265dc6f4:::benchmark_data_smithi169_105252_object115395:head [delete] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:39.666261+0000", 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 580.83571964299995, 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045926209000000003, 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 283989 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666261+0000", 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666261+0000", 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666260+0000", 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0759999999999997e-06 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666267+0000", 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1304000000000001e-05 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666278+0000", 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.583e-06 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.666283+0000", 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025589900000000001 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.668842+0000", 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014255 2024-02-12T16:47:20.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.668984+0000", 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020421400000000001 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669188+0000", 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075747119999999999 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.676763+0000", 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035943390000000002 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.680358+0000", 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031702014000000001 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.712060+0000", 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7827999999999997e-05 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.712107+0000", 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0061000000000005e-05 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.712187+0000", 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045926209000000003 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:283993 7.4 7:23c03c92:::benchmark_data_smithi169_105252_object115399:head [delete] snapc 0=[] ondisk+write+known_if_redirected e817)", 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:37:39.669389+0000", 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 580.83259192200001, 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.053297408999999997, 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:20.520 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 283993 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669389+0000", 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669389+0000", 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669388+0000", 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6469999999999997e-06 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669392+0000", 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8920000000000006e-06 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669401+0000", 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.048e-06 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.669405+0000", 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033621039999999999 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.672767+0000", 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145298 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.672913+0000", 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191416 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:47:20.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.673104+0000", 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038252529999999998 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.676929+0000", 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003472181 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.680402+0000", 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.042179254999999999 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.722581+0000", 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6111e-05 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.722617+0000", 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9481999999999996e-05 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:37:39.722686+0000", 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.053297408999999997 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:328656 7.4 7:220a6bfe:::benchmark_data_smithi169_105252_object160062:head [delete] snapc 0=[] ondisk+write+known_if_redirected e825)", 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:38:17.736120+0000", 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 542.765861234, 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045947122999999999, 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 328656 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736120+0000", 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736120+0000", 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736118+0000", 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7559999999999996e-06 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736124+0000", 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1121000000000001e-05 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736135+0000", 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0599999999999996e-06 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736141+0000", 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2299999999999999e-05 2024-02-12T16:47:20.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736174+0000", 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6514999999999993e-05 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736270+0000", 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6551000000000005e-05 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.736357+0000", 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036948409999999999 2024-02-12T16:47:20.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.740051+0000", 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252682 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.753304+0000", 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028601433999999998 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.781906+0000", 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5561e-05 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.781951+0000", 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011567 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:17.782067+0000", 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.045947122999999999 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26530.0:330813 7.4 7:2a76caf2:::benchmark_data_smithi169_105252_object162219:head [delete] snapc 0=[] ondisk+write+known_if_redirected e827)", 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:38:19.515960+0000", 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 540.98602047300005, 2024-02-12T16:47:20.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044954721000000003, 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26530", 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/2225654437", 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 330813 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515960+0000", 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515960+0000", 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515959+0000", 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7969999999999997e-06 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515965+0000", 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2486e-05 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515977+0000", 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0549999999999996e-06 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.515983+0000", 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014179647 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.530163+0000", 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015306799999999999 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.530316+0000", 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019076100000000001 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,1", 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.530507+0000", 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088279550000000002 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.539335+0000", 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5001999999999997e-05 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.539420+0000", 2024-02-12T16:47:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.02137379 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.560794+0000", 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4208000000000002e-05 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.560828+0000", 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7287999999999995e-05 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:38:19.560915+0000", 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044954721000000003 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:129537 9.4 e946/916 9:2d548fbd:::benchmark_data_smithi169_124168_object129536:head v 946'8123, mlcod=945'8120)", 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:46.010285+0000", 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 214.491695795, 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028124712, 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010285+0000", 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010285+0000", 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010282+0000", 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1179000000000004e-05 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010353+0000", 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0302000000000001e-05 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010364+0000", 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2553e-05 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010376+0000", 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4969000000000002e-05 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010431+0000", 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1345999999999999e-05 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.010453+0000", 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.027914406999999999 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.038367+0000", 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2877000000000003e-05 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:46.038410+0000", 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028124712 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:157744 9.c e953/916 9:3a2e2945:::benchmark_data_smithi169_124168_object7060:head v 953'9867, mlcod=953'9862)", 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:15.190421+0000", 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 185.31155960500001, 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031279361999999998, 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.541 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.190421+0000", 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.190421+0000", 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.190420+0000", 2024-02-12T16:47:20.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0835999999999999e-05 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.190431+0000", 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2550000000000003e-06 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.190440+0000", 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0769999999999998e-06 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.190445+0000", 2024-02-12T16:47:20.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058176310000000002 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.196263+0000", 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1883e-05 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.196284+0000", 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025381486000000002 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.221666+0000", 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4656000000000002e-05 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:15.221701+0000", 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031279361999999998 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:220260 9.c e973/916 9:34fd6b3f:::benchmark_data_smithi169_124168_object69576:head v 973'13819, mlcod=973'13814)", 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:33.534263+0000", 2024-02-12T16:47:20.545 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 106.967718007, 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028071433, 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.534263+0000", 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.534263+0000", 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.534261+0000", 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2979000000000001e-05 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.534274+0000", 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2853e-05 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.534287+0000", 2024-02-12T16:47:20.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9559999999999996e-06 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.534292+0000", 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020700129999999999 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.536362+0000", 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7728999999999999e-05 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.536380+0000", 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.025925396 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.562305+0000", 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9017999999999999e-05 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:33.562334+0000", 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028071433 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:223040 9.4 e974/916 9:224e1b8c:::benchmark_data_smithi169_124168_object72356:head v 974'13988, mlcod=974'13985)", 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:36.603647+0000", 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 103.898333482, 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.02901024, 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.603647+0000", 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.603647+0000", 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.603647+0000", 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0429999999999998e-06 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.603652+0000", 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8039999999999998e-06 2024-02-12T16:47:20.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.603658+0000", 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.603662+0000", 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045994690000000001 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.608261+0000", 2024-02-12T16:47:20.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0939000000000001e-05 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.608282+0000", 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024345412 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.632628+0000", 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0090000000000002e-05 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:36.632658+0000", 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.02901024 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:225262 9.c e974/916 9:3258fc8d:::benchmark_data_smithi169_124168_object74578:head v 974'14149, mlcod=974'14145)", 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:38.830141+0000", 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 101.671840182, 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.027986364, 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.830141+0000", 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.830141+0000", 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.830139+0000", 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3140000000000004e-06 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.830147+0000", 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3020000000000001e-05 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.830160+0000", 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6299999999999997e-06 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.830165+0000", 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001753452 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.831919+0000", 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2636000000000001e-05 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.831941+0000", 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026156195 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.858097+0000", 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9671000000000001e-05 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:38.858127+0000", 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.027986364 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:272221 9.c e999/916 9:344bcf60:::benchmark_data_smithi169_124168_object121537:head v 999'17061, mlcod=999'17056)", 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:26.018208+0000", 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.483772598999998, 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028874416999999999, 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.558 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.018208+0000", 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.018208+0000", 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.018208+0000", 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6419999999999998e-06 2024-02-12T16:47:20.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.018212+0000", 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.198e-06 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.018218+0000", 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7529999999999999e-06 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.018221+0000", 2024-02-12T16:47:20.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004044448 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.022266+0000", 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0178000000000001e-05 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.022296+0000", 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024745248000000001 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.047041+0000", 2024-02-12T16:47:20.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1535999999999997e-05 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.047083+0000", 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.028874416999999999 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:272234 9.c e999/916 9:30e16a02:::benchmark_data_smithi169_124168_object121550:head v 999'17062, mlcod=999'17056)", 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:26.020449+0000", 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.481531725000004, 2024-02-12T16:47:20.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031368382, 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.020449+0000", 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.020449+0000", 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.020448+0000", 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0274e-05 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.020458+0000", 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.119e-05 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.020469+0000", 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4299999999999997e-06 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.020475+0000", 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0078176530000000008 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.028292+0000", 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5418999999999998e-05 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.028318+0000", 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023457407 2024-02-12T16:47:20.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.051775+0000", 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2457999999999999e-05 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.051818+0000", 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031368382 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:272245 9.c e999/916 9:3d07e73a:::benchmark_data_smithi169_124168_object121561:head v 999'17063, mlcod=999'17057)", 2024-02-12T16:47:20.566 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:26.025980+0000", 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.476000562000003, 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029198663999999999, 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.025980+0000", 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.025980+0000", 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.025979+0000", 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1744e-05 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.025990+0000", 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3937000000000001e-05 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.026004+0000", 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0190000000000003e-06 2024-02-12T16:47:20.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.026010+0000", 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074798989999999999 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.033490+0000", 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3408e-05 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.033514+0000", 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021643565 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.055157+0000", 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1716999999999999e-05 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.055179+0000", 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029198663999999999 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:272253 9.c e999/916 9:3a61b6bf:::benchmark_data_smithi169_124168_object121569:head v 999'17066, mlcod=999'17058)", 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:26.042562+0000", 2024-02-12T16:47:20.570 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.459418927999998, 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029271111999999998, 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.042562+0000", 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.042562+0000", 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.042560+0000", 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5838999999999999e-05 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.042576+0000", 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5051999999999999e-05 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.042591+0000", 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9149999999999999e-06 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.042598+0000", 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00451674 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.047115+0000", 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4572000000000001e-05 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.047139+0000", 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024659073 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.071798+0000", 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4749999999999998e-05 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.071833+0000", 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029271111999999998 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:291057 9.c e1013/916 9:324f86f7:::benchmark_data_smithi169_124168_object140373:head v 1013'18202, mlcod=1013'18198)", 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:46.023273+0000", 2024-02-12T16:47:20.574 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.478707980000003, 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024212480000000002, 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023273+0000", 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023273+0000", 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023271+0000", 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3701e-05 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023284+0000", 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6033e-05 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023300+0000", 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9450000000000004e-06 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023307+0000", 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058711600000000005 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023894+0000", 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3374e-05 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.023918+0000", 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023527987 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.047446+0000", 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9544e-05 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.047485+0000", 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024212480000000002 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:291065 9.c e1013/916 9:3331255b:::benchmark_data_smithi169_124168_object140381:head v 1013'18203, mlcod=1013'18198)", 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:46.028999+0000", 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.472982189, 2024-02-12T16:47:20.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024732186, 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.028999+0000", 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.028999+0000", 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.028997+0000", 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1809e-05 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.029009+0000", 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3492999999999999e-05 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.029022+0000", 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.674e-06 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.029029+0000", 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007169173 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.036198+0000", 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9805999999999999e-05 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.036228+0000", 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017453584000000001 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.053682+0000", 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9190000000000002e-05 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.053731+0000", 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024732186 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:291104 9.c e1013/916 9:3828c164:::benchmark_data_smithi169_124168_object140420:head v 1013'18206, mlcod=1013'18203)", 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:46.057743+0000", 2024-02-12T16:47:20.582 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.444237526999999, 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024374212999999999, 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057743+0000", 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057743+0000", 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057741+0000", 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5158e-05 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057756+0000", 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1426000000000001e-05 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057777+0000", 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0129999999999999e-06 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057786+0000", 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1944999999999998e-05 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057838+0000", 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9255000000000001e-05 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.057857+0000", 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024200874000000001 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.082058+0000", 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9320999999999997e-05 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:46.082118+0000", 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024374212999999999 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:296312 9.c e1014/916 9:3d0fb0e3:::benchmark_data_smithi169_124168_object145628:head v 1014'18535, mlcod=1014'18533)", 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:49.534355+0000", 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.967625774999998, 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022626007, 2024-02-12T16:47:20.586 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534355+0000", 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534355+0000", 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534354+0000", 2024-02-12T16:47:20.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1595e-05 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534365+0000", 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0729999999999994e-06 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534374+0000", 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7049999999999996e-06 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534380+0000", 2024-02-12T16:47:20.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9257e-05 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534429+0000", 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.842e-05 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.534448+0000", 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022491251 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.556939+0000", 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2132000000000003e-05 2024-02-12T16:47:20.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:49.556981+0000", 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022626007 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:301050 9.c e1020/916 9:396ee8d9:::benchmark_data_smithi169_124168_object150366:head v 1020'18856, mlcod=1020'18852)", 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:55.715708+0000", 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.786273368, 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023069342999999999, 2024-02-12T16:47:20.590 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.715708+0000", 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.715708+0000", 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.715705+0000", 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.668e-05 2024-02-12T16:47:20.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.715722+0000", 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2665e-05 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.715734+0000", 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3220000000000004e-06 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.715742+0000", 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035619549999999999 2024-02-12T16:47:20.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.719304+0000", 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4217e-05 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.719318+0000", 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019423777 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.738742+0000", 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5250000000000003e-05 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.738777+0000", 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023069342999999999 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:301261 9.c e1020/916 9:38402c46:::benchmark_data_smithi169_124168_object150577:head v 1020'18875, mlcod=1020'18871)", 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:55.863543+0000", 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.638437762999999, 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024172658, 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.863543+0000", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.863543+0000", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.863541+0000", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0135999999999999e-05 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.863552+0000", 2024-02-12T16:47:20.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3878e-05 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.863565+0000", 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.93e-06 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.863571+0000", 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004908567 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.868480+0000", 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3414e-05 2024-02-12T16:47:20.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.868503+0000", 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019165016999999999 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.887668+0000", 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7398000000000003e-05 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:55.887716+0000", 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024172658 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.597 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.598 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:20.598 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:20.598 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:47:20.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:20 smithi169 bash[19604]: cluster 2024-02-12T16:47:18.816200+0000 osd.4 (osd.4) 169 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:20.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:20 smithi169 bash[19604]: cluster 2024-02-12T16:47:18.816322+0000 osd.4 (osd.4) 170 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:20.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:20 smithi169 bash[19604]: cluster 2024-02-12T16:47:19.067107+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e1031: 12 total, 12 up, 10 in 2024-02-12T16:47:20.598 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:20 smithi169 bash[19604]: audit 2024-02-12T16:47:19.150713+0000 mon.a (mon.0) 3380 : audit [INF] from='client.? v1:172.21.15.121:0/1016116987' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:47:20.633 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-12T16:47:20.633 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-02-12T16:47:20.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd in 6 2024-02-12T16:47:21.129 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-12T16:47:21.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:19.751838+0000 osd.10 (osd.10) 242 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:19.751924+0000 osd.10 (osd.10) 243 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:20.065092+0000 mon.a (mon.0) 3381 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: audit 2024-02-12T16:47:20.072264+0000 mon.a (mon.0) 3382 : audit [INF] from='client.? v1:172.21.15.121:0/1016116987' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:20.072370+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e1032: 12 total, 12 up, 10 in 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: audit 2024-02-12T16:47:20.130279+0000 mon.a (mon.0) 3384 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:20.343768+0000 mgr.y (mgr.14146) 1894 : cluster [DBG] pgmap v2793: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:20.479466+0000 osd.10 (osd.10) 244 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:20.562045+0000 osd.10 (osd.10) 245 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:47:21.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:21 smithi123 bash[19119]: cluster 2024-02-12T16:47:21.051626+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e1033: 12 total, 12 up, 10 in 2024-02-12T16:47:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:19.751838+0000 osd.10 (osd.10) 242 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:19.751924+0000 osd.10 (osd.10) 243 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:20.065092+0000 mon.a (mon.0) 3381 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:47:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: audit 2024-02-12T16:47:20.072264+0000 mon.a (mon.0) 3382 : audit [INF] from='client.? v1:172.21.15.121:0/1016116987' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:47:21.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:20.072370+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e1032: 12 total, 12 up, 10 in 2024-02-12T16:47:21.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: audit 2024-02-12T16:47:20.130279+0000 mon.a (mon.0) 3384 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:21.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:20.343768+0000 mgr.y (mgr.14146) 1894 : cluster [DBG] pgmap v2793: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:21.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:20.479466+0000 osd.10 (osd.10) 244 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:47:21.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:20.562045+0000 osd.10 (osd.10) 245 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:47:21.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:21 smithi121 bash[18489]: cluster 2024-02-12T16:47:21.051626+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e1033: 12 total, 12 up, 10 in 2024-02-12T16:47:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:19.751838+0000 osd.10 (osd.10) 242 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:19.751924+0000 osd.10 (osd.10) 243 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:20.065092+0000 mon.a (mon.0) 3381 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-12T16:47:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: audit 2024-02-12T16:47:20.072264+0000 mon.a (mon.0) 3382 : audit [INF] from='client.? v1:172.21.15.121:0/1016116987' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:47:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:20.072370+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e1032: 12 total, 12 up, 10 in 2024-02-12T16:47:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: audit 2024-02-12T16:47:20.130279+0000 mon.a (mon.0) 3384 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:20.343768+0000 mgr.y (mgr.14146) 1894 : cluster [DBG] pgmap v2793: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:20.479466+0000 osd.10 (osd.10) 244 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:47:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:20.562045+0000 osd.10 (osd.10) 245 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:47:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:21 smithi169 bash[19604]: cluster 2024-02-12T16:47:21.051626+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e1033: 12 total, 12 up, 10 in 2024-02-12T16:47:22.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:22 smithi123 bash[19119]: cluster 2024-02-12T16:47:20.633344+0000 osd.11 (osd.11) 229 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:47:22.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:22 smithi123 bash[19119]: cluster 2024-02-12T16:47:20.672223+0000 osd.11 (osd.11) 230 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:47:22.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:22 smithi123 bash[19119]: cluster 2024-02-12T16:47:21.190002+0000 osd.8 (osd.8) 153 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:47:22.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:22 smithi123 bash[19119]: cluster 2024-02-12T16:47:21.623183+0000 osd.7 (osd.7) 17 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:47:22.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:22 smithi123 bash[19119]: cluster 2024-02-12T16:47:21.686884+0000 osd.7 (osd.7) 18 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:47:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:22 smithi121 bash[18489]: cluster 2024-02-12T16:47:20.633344+0000 osd.11 (osd.11) 229 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:47:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:22 smithi121 bash[18489]: cluster 2024-02-12T16:47:20.672223+0000 osd.11 (osd.11) 230 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:47:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:22 smithi121 bash[18489]: cluster 2024-02-12T16:47:21.190002+0000 osd.8 (osd.8) 153 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:47:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:22 smithi121 bash[18489]: cluster 2024-02-12T16:47:21.623183+0000 osd.7 (osd.7) 17 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:47:22.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:22 smithi121 bash[18489]: cluster 2024-02-12T16:47:21.686884+0000 osd.7 (osd.7) 18 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:47:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:22 smithi169 bash[19604]: cluster 2024-02-12T16:47:20.633344+0000 osd.11 (osd.11) 229 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:47:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:22 smithi169 bash[19604]: cluster 2024-02-12T16:47:20.672223+0000 osd.11 (osd.11) 230 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:47:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:22 smithi169 bash[19604]: cluster 2024-02-12T16:47:21.190002+0000 osd.8 (osd.8) 153 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:47:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:22 smithi169 bash[19604]: cluster 2024-02-12T16:47:21.623183+0000 osd.7 (osd.7) 17 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:47:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:22 smithi169 bash[19604]: cluster 2024-02-12T16:47:21.686884+0000 osd.7 (osd.7) 18 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:47:22.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset noscrub 2024-02-12T16:47:23.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:23 smithi121 bash[18489]: cluster 2024-02-12T16:47:21.333121+0000 osd.8 (osd.8) 154 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:47:23.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:23 smithi121 bash[18489]: cluster 2024-02-12T16:47:21.631981+0000 osd.11 (osd.11) 231 : cluster [DBG] 1.b scrub starts 2024-02-12T16:47:23.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:23 smithi121 bash[18489]: cluster 2024-02-12T16:47:21.796757+0000 osd.11 (osd.11) 232 : cluster [DBG] 1.b scrub ok 2024-02-12T16:47:23.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:23 smithi121 bash[18489]: cluster 2024-02-12T16:47:22.344417+0000 mgr.y (mgr.14146) 1895 : cluster [DBG] pgmap v2795: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:23.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:23 smithi121 bash[18489]: cluster 2024-02-12T16:47:22.671968+0000 osd.7 (osd.7) 19 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:47:23.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:23 smithi121 bash[18489]: cluster 2024-02-12T16:47:22.703028+0000 osd.7 (osd.7) 20 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:47:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:23 smithi169 bash[19604]: cluster 2024-02-12T16:47:21.333121+0000 osd.8 (osd.8) 154 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:47:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:23 smithi169 bash[19604]: cluster 2024-02-12T16:47:21.631981+0000 osd.11 (osd.11) 231 : cluster [DBG] 1.b scrub starts 2024-02-12T16:47:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:23 smithi169 bash[19604]: cluster 2024-02-12T16:47:21.796757+0000 osd.11 (osd.11) 232 : cluster [DBG] 1.b scrub ok 2024-02-12T16:47:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:23 smithi169 bash[19604]: cluster 2024-02-12T16:47:22.344417+0000 mgr.y (mgr.14146) 1895 : cluster [DBG] pgmap v2795: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:23 smithi169 bash[19604]: cluster 2024-02-12T16:47:22.671968+0000 osd.7 (osd.7) 19 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:47:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:23 smithi169 bash[19604]: cluster 2024-02-12T16:47:22.703028+0000 osd.7 (osd.7) 20 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:47:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:23 smithi123 bash[19119]: cluster 2024-02-12T16:47:21.333121+0000 osd.8 (osd.8) 154 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:47:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:23 smithi123 bash[19119]: cluster 2024-02-12T16:47:21.631981+0000 osd.11 (osd.11) 231 : cluster [DBG] 1.b scrub starts 2024-02-12T16:47:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:23 smithi123 bash[19119]: cluster 2024-02-12T16:47:21.796757+0000 osd.11 (osd.11) 232 : cluster [DBG] 1.b scrub ok 2024-02-12T16:47:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:23 smithi123 bash[19119]: cluster 2024-02-12T16:47:22.344417+0000 mgr.y (mgr.14146) 1895 : cluster [DBG] pgmap v2795: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:23 smithi123 bash[19119]: cluster 2024-02-12T16:47:22.671968+0000 osd.7 (osd.7) 19 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:47:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:23 smithi123 bash[19119]: cluster 2024-02-12T16:47:22.703028+0000 osd.7 (osd.7) 20 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:47:24.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:24 smithi121 bash[18489]: cluster 2024-02-12T16:47:22.874486+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:47:24.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:24 smithi121 bash[18489]: cluster 2024-02-12T16:47:22.917320+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:47:24.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:24 smithi121 bash[18489]: cluster 2024-02-12T16:47:23.648638+0000 osd.7 (osd.7) 21 : cluster [DBG] 1.d scrub starts 2024-02-12T16:47:24.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:24 smithi121 bash[18489]: cluster 2024-02-12T16:47:23.741231+0000 osd.7 (osd.7) 22 : cluster [DBG] 1.d scrub ok 2024-02-12T16:47:24.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:24 smithi169 bash[19604]: cluster 2024-02-12T16:47:22.874486+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:47:24.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:24 smithi169 bash[19604]: cluster 2024-02-12T16:47:22.917320+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:47:24.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:24 smithi169 bash[19604]: cluster 2024-02-12T16:47:23.648638+0000 osd.7 (osd.7) 21 : cluster [DBG] 1.d scrub starts 2024-02-12T16:47:24.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:24 smithi169 bash[19604]: cluster 2024-02-12T16:47:23.741231+0000 osd.7 (osd.7) 22 : cluster [DBG] 1.d scrub ok 2024-02-12T16:47:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:24 smithi123 bash[19119]: cluster 2024-02-12T16:47:22.874486+0000 osd.5 (osd.5) 229 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:47:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:24 smithi123 bash[19119]: cluster 2024-02-12T16:47:22.917320+0000 osd.5 (osd.5) 230 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:47:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:24 smithi123 bash[19119]: cluster 2024-02-12T16:47:23.648638+0000 osd.7 (osd.7) 21 : cluster [DBG] 1.d scrub starts 2024-02-12T16:47:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:24 smithi123 bash[19119]: cluster 2024-02-12T16:47:23.741231+0000 osd.7 (osd.7) 22 : cluster [DBG] 1.d scrub ok 2024-02-12T16:47:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:25 smithi169 bash[19604]: cluster 2024-02-12T16:47:23.992161+0000 osd.11 (osd.11) 233 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:25 smithi169 bash[19604]: cluster 2024-02-12T16:47:23.992251+0000 osd.11 (osd.11) 234 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:25 smithi169 bash[19604]: cluster 2024-02-12T16:47:24.145500+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e1034: 12 total, 12 up, 10 in 2024-02-12T16:47:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:25 smithi169 bash[19604]: cluster 2024-02-12T16:47:24.345084+0000 mgr.y (mgr.14146) 1896 : cluster [DBG] pgmap v2797: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:25 smithi169 bash[19604]: cluster 2024-02-12T16:47:24.604021+0000 osd.7 (osd.7) 23 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:47:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:25 smithi169 bash[19604]: cluster 2024-02-12T16:47:24.679614+0000 osd.7 (osd.7) 24 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:47:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:25 smithi123 bash[19119]: cluster 2024-02-12T16:47:23.992161+0000 osd.11 (osd.11) 233 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:25 smithi123 bash[19119]: cluster 2024-02-12T16:47:23.992251+0000 osd.11 (osd.11) 234 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:25 smithi123 bash[19119]: cluster 2024-02-12T16:47:24.145500+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e1034: 12 total, 12 up, 10 in 2024-02-12T16:47:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:25 smithi123 bash[19119]: cluster 2024-02-12T16:47:24.345084+0000 mgr.y (mgr.14146) 1896 : cluster [DBG] pgmap v2797: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:25 smithi123 bash[19119]: cluster 2024-02-12T16:47:24.604021+0000 osd.7 (osd.7) 23 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:47:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:25 smithi123 bash[19119]: cluster 2024-02-12T16:47:24.679614+0000 osd.7 (osd.7) 24 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:47:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:25 smithi121 bash[18489]: cluster 2024-02-12T16:47:23.992161+0000 osd.11 (osd.11) 233 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:25 smithi121 bash[18489]: cluster 2024-02-12T16:47:23.992251+0000 osd.11 (osd.11) 234 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:25 smithi121 bash[18489]: cluster 2024-02-12T16:47:24.145500+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e1034: 12 total, 12 up, 10 in 2024-02-12T16:47:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:25 smithi121 bash[18489]: cluster 2024-02-12T16:47:24.345084+0000 mgr.y (mgr.14146) 1896 : cluster [DBG] pgmap v2797: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:25.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:25 smithi121 bash[18489]: cluster 2024-02-12T16:47:24.604021+0000 osd.7 (osd.7) 23 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:47:25.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:25 smithi121 bash[18489]: cluster 2024-02-12T16:47:24.679614+0000 osd.7 (osd.7) 24 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:47:25.923 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:47:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:24.836696+0000 osd.5 (osd.5) 231 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:47:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:24.863364+0000 osd.5 (osd.5) 232 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:47:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.241148+0000 osd.1 (osd.1) 182 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:47:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.436563+0000 osd.10 (osd.10) 246 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:47:26.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.455164+0000 osd.1 (osd.1) 183 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:47:26.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.540468+0000 osd.10 (osd.10) 247 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:47:26.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.594726+0000 osd.7 (osd.7) 25 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:47:26.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.661796+0000 osd.7 (osd.7) 26 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:47:26.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:26 smithi169 bash[19604]: cluster 2024-02-12T16:47:26.056280+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e1035: 12 total, 12 up, 10 in 2024-02-12T16:47:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:24.836696+0000 osd.5 (osd.5) 231 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:47:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:24.863364+0000 osd.5 (osd.5) 232 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:47:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.241148+0000 osd.1 (osd.1) 182 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:47:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.436563+0000 osd.10 (osd.10) 246 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:47:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.455164+0000 osd.1 (osd.1) 183 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:47:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.540468+0000 osd.10 (osd.10) 247 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:47:26.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.594726+0000 osd.7 (osd.7) 25 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:47:26.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.661796+0000 osd.7 (osd.7) 26 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:47:26.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:26 smithi123 bash[19119]: cluster 2024-02-12T16:47:26.056280+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e1035: 12 total, 12 up, 10 in 2024-02-12T16:47:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:24.836696+0000 osd.5 (osd.5) 231 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:47:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:24.863364+0000 osd.5 (osd.5) 232 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:47:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.241148+0000 osd.1 (osd.1) 182 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:47:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.436563+0000 osd.10 (osd.10) 246 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:47:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.455164+0000 osd.1 (osd.1) 183 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:47:26.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.540468+0000 osd.10 (osd.10) 247 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:47:26.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.594726+0000 osd.7 (osd.7) 25 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:47:26.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.661796+0000 osd.7 (osd.7) 26 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:47:26.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:26 smithi121 bash[18489]: cluster 2024-02-12T16:47:26.056280+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e1035: 12 total, 12 up, 10 in 2024-02-12T16:47:27.277 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:27 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.847778+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:47:27.277 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:27 smithi123 bash[19119]: cluster 2024-02-12T16:47:25.849739+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:47:27.278 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:27 smithi123 bash[19119]: cluster 2024-02-12T16:47:26.345748+0000 mgr.y (mgr.14146) 1897 : cluster [DBG] pgmap v2799: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:27.278 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:27 smithi123 bash[19119]: cluster 2024-02-12T16:47:26.895194+0000 osd.7 (osd.7) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:27.278 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:27 smithi123 bash[19119]: cluster 2024-02-12T16:47:26.895300+0000 osd.7 (osd.7) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:27 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.847778+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:47:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:27 smithi121 bash[18489]: cluster 2024-02-12T16:47:25.849739+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:47:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:27 smithi121 bash[18489]: cluster 2024-02-12T16:47:26.345748+0000 mgr.y (mgr.14146) 1897 : cluster [DBG] pgmap v2799: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:27 smithi121 bash[18489]: cluster 2024-02-12T16:47:26.895194+0000 osd.7 (osd.7) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:27 smithi121 bash[18489]: cluster 2024-02-12T16:47:26.895300+0000 osd.7 (osd.7) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:27 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.847778+0000 osd.4 (osd.4) 171 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:47:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:27 smithi169 bash[19604]: cluster 2024-02-12T16:47:25.849739+0000 osd.4 (osd.4) 172 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:47:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:27 smithi169 bash[19604]: cluster 2024-02-12T16:47:26.345748+0000 mgr.y (mgr.14146) 1897 : cluster [DBG] pgmap v2799: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:27 smithi169 bash[19604]: cluster 2024-02-12T16:47:26.895194+0000 osd.7 (osd.7) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:27 smithi169 bash[19604]: cluster 2024-02-12T16:47:26.895300+0000 osd.7 (osd.7) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:28.123 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:28 smithi169 bash[19604]: cluster 2024-02-12T16:47:27.174637+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e1036: 12 total, 12 up, 10 in 2024-02-12T16:47:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:28 smithi169 bash[19604]: cluster 2024-02-12T16:47:27.431639+0000 osd.10 (osd.10) 248 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:47:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:28 smithi169 bash[19604]: cluster 2024-02-12T16:47:27.482035+0000 osd.10 (osd.10) 249 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:47:28.604 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:28 smithi123 bash[19119]: cluster 2024-02-12T16:47:27.174637+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e1036: 12 total, 12 up, 10 in 2024-02-12T16:47:28.604 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:28 smithi123 bash[19119]: cluster 2024-02-12T16:47:27.431639+0000 osd.10 (osd.10) 248 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:47:28.605 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:28 smithi123 bash[19119]: cluster 2024-02-12T16:47:27.482035+0000 osd.10 (osd.10) 249 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:47:28.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:28 smithi121 bash[18489]: cluster 2024-02-12T16:47:27.174637+0000 mon.a (mon.0) 3388 : cluster [DBG] osdmap e1036: 12 total, 12 up, 10 in 2024-02-12T16:47:28.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:28 smithi121 bash[18489]: cluster 2024-02-12T16:47:27.431639+0000 osd.10 (osd.10) 248 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:47:28.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:28 smithi121 bash[18489]: cluster 2024-02-12T16:47:27.482035+0000 osd.10 (osd.10) 249 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:47:28.836 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:47:28.839 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:47:28.839 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:47:28.839 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:47:29.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:29 smithi121 bash[18489]: cluster 2024-02-12T16:47:27.581131+0000 osd.11 (osd.11) 235 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:47:29.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:29 smithi121 bash[18489]: cluster 2024-02-12T16:47:27.584598+0000 osd.11 (osd.11) 236 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:47:29.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:29 smithi121 bash[18489]: cluster 2024-02-12T16:47:28.346174+0000 mgr.y (mgr.14146) 1898 : cluster [DBG] pgmap v2801: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:29.451 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:29 smithi123 bash[19119]: cluster 2024-02-12T16:47:27.581131+0000 osd.11 (osd.11) 235 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:47:29.451 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:29 smithi123 bash[19119]: cluster 2024-02-12T16:47:27.584598+0000 osd.11 (osd.11) 236 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:47:29.451 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:29 smithi123 bash[19119]: cluster 2024-02-12T16:47:28.346174+0000 mgr.y (mgr.14146) 1898 : cluster [DBG] pgmap v2801: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:29.452 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-12T16:47:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:29 smithi169 bash[19604]: cluster 2024-02-12T16:47:27.581131+0000 osd.11 (osd.11) 235 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:47:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:29 smithi169 bash[19604]: cluster 2024-02-12T16:47:27.584598+0000 osd.11 (osd.11) 236 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:47:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:29 smithi169 bash[19604]: cluster 2024-02-12T16:47:28.346174+0000 mgr.y (mgr.14146) 1898 : cluster [DBG] pgmap v2801: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:30.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:29 smithi121 bash[18489]: debug 2024-02-12T16:47:29.805+0000 7fbba85de700 -1 mon.a@0(leader).osd e1036 definitely_dead 0 2024-02-12T16:47:30.202 INFO:teuthology.orchestra.run.smithi121.stderr:marked in osd.6. 2024-02-12T16:47:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:30 smithi169 bash[19604]: cluster 2024-02-12T16:47:28.806642+0000 osd.5 (osd.5) 233 : cluster [DBG] 1.f scrub starts 2024-02-12T16:47:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:30 smithi169 bash[19604]: cluster 2024-02-12T16:47:28.964453+0000 osd.5 (osd.5) 234 : cluster [DBG] 1.f scrub ok 2024-02-12T16:47:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:30 smithi169 bash[19604]: cluster 2024-02-12T16:47:29.199865+0000 osd.8 (osd.8) 155 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:47:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:30 smithi169 bash[19604]: cluster 2024-02-12T16:47:29.203310+0000 osd.8 (osd.8) 156 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:47:30.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:30 smithi169 bash[19604]: audit 2024-02-12T16:47:29.806960+0000 mon.a (mon.0) 3389 : audit [DBG] from='client.? v1:172.21.15.121:0/2463168376' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T16:47:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:30 smithi123 bash[19119]: cluster 2024-02-12T16:47:28.806642+0000 osd.5 (osd.5) 233 : cluster [DBG] 1.f scrub starts 2024-02-12T16:47:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:30 smithi123 bash[19119]: cluster 2024-02-12T16:47:28.964453+0000 osd.5 (osd.5) 234 : cluster [DBG] 1.f scrub ok 2024-02-12T16:47:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:30 smithi123 bash[19119]: cluster 2024-02-12T16:47:29.199865+0000 osd.8 (osd.8) 155 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:47:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:30 smithi123 bash[19119]: cluster 2024-02-12T16:47:29.203310+0000 osd.8 (osd.8) 156 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:47:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:30 smithi123 bash[19119]: audit 2024-02-12T16:47:29.806960+0000 mon.a (mon.0) 3389 : audit [DBG] from='client.? v1:172.21.15.121:0/2463168376' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T16:47:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:30 smithi121 bash[18489]: cluster 2024-02-12T16:47:28.806642+0000 osd.5 (osd.5) 233 : cluster [DBG] 1.f scrub starts 2024-02-12T16:47:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:30 smithi121 bash[18489]: cluster 2024-02-12T16:47:28.964453+0000 osd.5 (osd.5) 234 : cluster [DBG] 1.f scrub ok 2024-02-12T16:47:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:30 smithi121 bash[18489]: cluster 2024-02-12T16:47:29.199865+0000 osd.8 (osd.8) 155 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:47:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:30 smithi121 bash[18489]: cluster 2024-02-12T16:47:29.203310+0000 osd.8 (osd.8) 156 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:47:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:30 smithi121 bash[18489]: audit 2024-02-12T16:47:29.806960+0000 mon.a (mon.0) 3389 : audit [DBG] from='client.? v1:172.21.15.121:0/2463168376' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-12T16:47:31.349 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-02-12T16:47:31.349 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-12T16:47:31.349 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-02-12T16:47:31.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd in 9 2024-02-12T16:47:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:31 smithi169 bash[19604]: audit 2024-02-12T16:47:30.190539+0000 mon.a (mon.0) 3390 : audit [INF] from='client.? v1:172.21.15.121:0/2463168376' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-12T16:47:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:31 smithi169 bash[19604]: cluster 2024-02-12T16:47:30.190668+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e1037: 12 total, 12 up, 11 in 2024-02-12T16:47:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:31 smithi169 bash[19604]: cluster 2024-02-12T16:47:30.346766+0000 mgr.y (mgr.14146) 1899 : cluster [DBG] pgmap v2803: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:31 smithi123 bash[19119]: audit 2024-02-12T16:47:30.190539+0000 mon.a (mon.0) 3390 : audit [INF] from='client.? v1:172.21.15.121:0/2463168376' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-12T16:47:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:31 smithi123 bash[19119]: cluster 2024-02-12T16:47:30.190668+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e1037: 12 total, 12 up, 11 in 2024-02-12T16:47:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:31 smithi123 bash[19119]: cluster 2024-02-12T16:47:30.346766+0000 mgr.y (mgr.14146) 1899 : cluster [DBG] pgmap v2803: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:31 smithi121 bash[18489]: audit 2024-02-12T16:47:30.190539+0000 mon.a (mon.0) 3390 : audit [INF] from='client.? v1:172.21.15.121:0/2463168376' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-12T16:47:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:31 smithi121 bash[18489]: cluster 2024-02-12T16:47:30.190668+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e1037: 12 total, 12 up, 11 in 2024-02-12T16:47:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:31 smithi121 bash[18489]: cluster 2024-02-12T16:47:30.346766+0000 mgr.y (mgr.14146) 1899 : cluster [DBG] pgmap v2803: 31 pgs: 31 active+clean; 0 B data, 8.0 GiB used, 886 GiB / 894 GiB avail 2024-02-12T16:47:32.222 INFO:teuthology.orchestra.run.smithi121.stderr:noscrub is unset 2024-02-12T16:47:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:32 smithi169 bash[19604]: cluster 2024-02-12T16:47:31.210219+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e1038: 12 total, 12 up, 11 in 2024-02-12T16:47:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:32 smithi169 bash[19604]: cluster 2024-02-12T16:47:31.419723+0000 osd.10 (osd.10) 250 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:47:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:32 smithi169 bash[19604]: cluster 2024-02-12T16:47:31.445079+0000 osd.10 (osd.10) 251 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:47:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:32 smithi169 bash[19604]: cluster 2024-02-12T16:47:31.648433+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:32.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:32 smithi169 bash[19604]: cluster 2024-02-12T16:47:31.648511+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:32.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:32 smithi169 bash[19604]: audit 2024-02-12T16:47:32.143902+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? v1:172.21.15.121:0/3793790596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:47:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:32 smithi123 bash[19119]: cluster 2024-02-12T16:47:31.210219+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e1038: 12 total, 12 up, 11 in 2024-02-12T16:47:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:32 smithi123 bash[19119]: cluster 2024-02-12T16:47:31.419723+0000 osd.10 (osd.10) 250 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:47:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:32 smithi123 bash[19119]: cluster 2024-02-12T16:47:31.445079+0000 osd.10 (osd.10) 251 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:47:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:32 smithi123 bash[19119]: cluster 2024-02-12T16:47:31.648433+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:32 smithi123 bash[19119]: cluster 2024-02-12T16:47:31.648511+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:32 smithi123 bash[19119]: audit 2024-02-12T16:47:32.143902+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? v1:172.21.15.121:0/3793790596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:47:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:32 smithi121 bash[18489]: cluster 2024-02-12T16:47:31.210219+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e1038: 12 total, 12 up, 11 in 2024-02-12T16:47:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:32 smithi121 bash[18489]: cluster 2024-02-12T16:47:31.419723+0000 osd.10 (osd.10) 250 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:47:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:32 smithi121 bash[18489]: cluster 2024-02-12T16:47:31.445079+0000 osd.10 (osd.10) 251 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:47:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:32 smithi121 bash[18489]: cluster 2024-02-12T16:47:31.648433+0000 osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:32 smithi121 bash[18489]: cluster 2024-02-12T16:47:31.648511+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:32 smithi121 bash[18489]: audit 2024-02-12T16:47:32.143902+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? v1:172.21.15.121:0/3793790596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-12T16:47:33.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-12T16:47:33.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: cluster 2024-02-12T16:47:31.860427+0000 osd.5 (osd.5) 235 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:47:33.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: cluster 2024-02-12T16:47:31.975423+0000 osd.5 (osd.5) 236 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:47:33.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: audit 2024-02-12T16:47:32.211783+0000 mon.a (mon.0) 3394 : audit [INF] from='client.? v1:172.21.15.121:0/3793790596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:47:33.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: cluster 2024-02-12T16:47:32.211878+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e1039: 12 total, 12 up, 11 in 2024-02-12T16:47:33.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: cluster 2024-02-12T16:47:32.331439+0000 osd.1 (osd.1) 184 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:47:33.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: cluster 2024-02-12T16:47:32.347290+0000 mgr.y (mgr.14146) 1900 : cluster [DBG] pgmap v2806: 31 pgs: 3 peering, 28 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:33.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: cluster 2024-02-12T16:47:32.452207+0000 osd.1 (osd.1) 185 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:47:33.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:33 smithi121 bash[18489]: cluster 2024-02-12T16:47:33.155529+0000 osd.8 (osd.8) 157 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:47:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: cluster 2024-02-12T16:47:31.860427+0000 osd.5 (osd.5) 235 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:47:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: cluster 2024-02-12T16:47:31.975423+0000 osd.5 (osd.5) 236 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:47:33.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: audit 2024-02-12T16:47:32.211783+0000 mon.a (mon.0) 3394 : audit [INF] from='client.? v1:172.21.15.121:0/3793790596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:47:33.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: cluster 2024-02-12T16:47:32.211878+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e1039: 12 total, 12 up, 11 in 2024-02-12T16:47:33.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: cluster 2024-02-12T16:47:32.331439+0000 osd.1 (osd.1) 184 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:47:33.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: cluster 2024-02-12T16:47:32.347290+0000 mgr.y (mgr.14146) 1900 : cluster [DBG] pgmap v2806: 31 pgs: 3 peering, 28 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:33.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: cluster 2024-02-12T16:47:32.452207+0000 osd.1 (osd.1) 185 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:47:33.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:33 smithi169 bash[19604]: cluster 2024-02-12T16:47:33.155529+0000 osd.8 (osd.8) 157 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:47:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: cluster 2024-02-12T16:47:31.860427+0000 osd.5 (osd.5) 235 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:47:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: cluster 2024-02-12T16:47:31.975423+0000 osd.5 (osd.5) 236 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:47:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: audit 2024-02-12T16:47:32.211783+0000 mon.a (mon.0) 3394 : audit [INF] from='client.? v1:172.21.15.121:0/3793790596' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-12T16:47:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: cluster 2024-02-12T16:47:32.211878+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e1039: 12 total, 12 up, 11 in 2024-02-12T16:47:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: cluster 2024-02-12T16:47:32.331439+0000 osd.1 (osd.1) 184 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:47:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: cluster 2024-02-12T16:47:32.347290+0000 mgr.y (mgr.14146) 1900 : cluster [DBG] pgmap v2806: 31 pgs: 3 peering, 28 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:33.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: cluster 2024-02-12T16:47:32.452207+0000 osd.1 (osd.1) 185 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:47:33.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:33 smithi123 bash[19119]: cluster 2024-02-12T16:47:33.155529+0000 osd.8 (osd.8) 157 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:47:34.248 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:47:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:34 smithi169 bash[19604]: cluster 2024-02-12T16:47:32.845075+0000 osd.4 (osd.4) 173 : cluster [DBG] 1.e scrub starts 2024-02-12T16:47:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:34 smithi169 bash[19604]: cluster 2024-02-12T16:47:32.983180+0000 osd.4 (osd.4) 174 : cluster [DBG] 1.e scrub ok 2024-02-12T16:47:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:34 smithi169 bash[19604]: cluster 2024-02-12T16:47:33.273329+0000 osd.8 (osd.8) 158 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:47:34.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:34 smithi169 bash[19604]: cluster 2024-02-12T16:47:33.303874+0000 osd.1 (osd.1) 186 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:47:34.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:34 smithi169 bash[19604]: cluster 2024-02-12T16:47:33.508708+0000 osd.1 (osd.1) 187 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:47:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:34 smithi123 bash[19119]: cluster 2024-02-12T16:47:32.845075+0000 osd.4 (osd.4) 173 : cluster [DBG] 1.e scrub starts 2024-02-12T16:47:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:34 smithi123 bash[19119]: cluster 2024-02-12T16:47:32.983180+0000 osd.4 (osd.4) 174 : cluster [DBG] 1.e scrub ok 2024-02-12T16:47:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:34 smithi123 bash[19119]: cluster 2024-02-12T16:47:33.273329+0000 osd.8 (osd.8) 158 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:47:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:34 smithi123 bash[19119]: cluster 2024-02-12T16:47:33.303874+0000 osd.1 (osd.1) 186 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:47:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:34 smithi123 bash[19119]: cluster 2024-02-12T16:47:33.508708+0000 osd.1 (osd.1) 187 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:47:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:34 smithi121 bash[18489]: cluster 2024-02-12T16:47:32.845075+0000 osd.4 (osd.4) 173 : cluster [DBG] 1.e scrub starts 2024-02-12T16:47:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:34 smithi121 bash[18489]: cluster 2024-02-12T16:47:32.983180+0000 osd.4 (osd.4) 174 : cluster [DBG] 1.e scrub ok 2024-02-12T16:47:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:34 smithi121 bash[18489]: cluster 2024-02-12T16:47:33.273329+0000 osd.8 (osd.8) 158 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:47:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:34 smithi121 bash[18489]: cluster 2024-02-12T16:47:33.303874+0000 osd.1 (osd.1) 186 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:47:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:34 smithi121 bash[18489]: cluster 2024-02-12T16:47:33.508708+0000 osd.1 (osd.1) 187 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:47:35.473 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:35 smithi123 bash[19119]: cluster 2024-02-12T16:47:34.347939+0000 mgr.y (mgr.14146) 1901 : cluster [DBG] pgmap v2807: 31 pgs: 3 peering, 28 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:35.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:35 smithi169 bash[19604]: cluster 2024-02-12T16:47:34.347939+0000 mgr.y (mgr.14146) 1901 : cluster [DBG] pgmap v2807: 31 pgs: 3 peering, 28 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:35 smithi121 bash[18489]: cluster 2024-02-12T16:47:34.347939+0000 mgr.y (mgr.14146) 1901 : cluster [DBG] pgmap v2807: 31 pgs: 3 peering, 28 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:36.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:36 smithi169 bash[19604]: cluster 2024-02-12T16:47:34.838475+0000 osd.4 (osd.4) 175 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:47:36.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:36 smithi169 bash[19604]: cluster 2024-02-12T16:47:34.935841+0000 osd.4 (osd.4) 176 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:47:36.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:36 smithi169 bash[19604]: cluster 2024-02-12T16:47:35.468690+0000 osd.10 (osd.10) 252 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:47:36.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:36 smithi169 bash[19604]: cluster 2024-02-12T16:47:35.509122+0000 osd.10 (osd.10) 253 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:47:36.639 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:36 smithi123 bash[19119]: cluster 2024-02-12T16:47:34.838475+0000 osd.4 (osd.4) 175 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:47:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:36 smithi123 bash[19119]: cluster 2024-02-12T16:47:34.935841+0000 osd.4 (osd.4) 176 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:47:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:36 smithi123 bash[19119]: cluster 2024-02-12T16:47:35.468690+0000 osd.10 (osd.10) 252 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:47:36.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:36 smithi123 bash[19119]: cluster 2024-02-12T16:47:35.509122+0000 osd.10 (osd.10) 253 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:47:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:36 smithi121 bash[18489]: cluster 2024-02-12T16:47:34.838475+0000 osd.4 (osd.4) 175 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:47:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:36 smithi121 bash[18489]: cluster 2024-02-12T16:47:34.935841+0000 osd.4 (osd.4) 176 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:47:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:36 smithi121 bash[18489]: cluster 2024-02-12T16:47:35.468690+0000 osd.10 (osd.10) 252 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:47:36.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:36 smithi121 bash[18489]: cluster 2024-02-12T16:47:35.509122+0000 osd.10 (osd.10) 253 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:47:37.195 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:47:37.195 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:47:37.195 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2024-02-12T16:47:37.195 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2024-02-12T16:47:37.195 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:47:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:37 smithi169 bash[19604]: cluster 2024-02-12T16:47:36.348625+0000 mgr.y (mgr.14146) 1902 : cluster [DBG] pgmap v2808: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:37 smithi123 bash[19119]: cluster 2024-02-12T16:47:36.348625+0000 mgr.y (mgr.14146) 1902 : cluster [DBG] pgmap v2808: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:37 smithi121 bash[18489]: cluster 2024-02-12T16:47:36.348625+0000 mgr.y (mgr.14146) 1902 : cluster [DBG] pgmap v2808: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:37.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-12T16:47:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:39 smithi121 bash[18489]: cluster 2024-02-12T16:47:38.349318+0000 mgr.y (mgr.14146) 1903 : cluster [DBG] pgmap v2809: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:39 smithi169 bash[19604]: cluster 2024-02-12T16:47:38.349318+0000 mgr.y (mgr.14146) 1903 : cluster [DBG] pgmap v2809: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:39 smithi123 bash[19119]: cluster 2024-02-12T16:47:38.349318+0000 mgr.y (mgr.14146) 1903 : cluster [DBG] pgmap v2809: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:40 smithi169 bash[19604]: cluster 2024-02-12T16:47:39.528843+0000 osd.2 (osd.2) 149 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:47:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:40 smithi123 bash[19119]: cluster 2024-02-12T16:47:39.528843+0000 osd.2 (osd.2) 149 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:47:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:40 smithi121 bash[18489]: cluster 2024-02-12T16:47:39.528843+0000 osd.2 (osd.2) 149 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:47:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:41 smithi169 bash[19604]: cluster 2024-02-12T16:47:39.633263+0000 osd.2 (osd.2) 150 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:47:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:41 smithi169 bash[19604]: cluster 2024-02-12T16:47:40.350025+0000 mgr.y (mgr.14146) 1904 : cluster [DBG] pgmap v2810: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:41 smithi123 bash[19119]: cluster 2024-02-12T16:47:39.633263+0000 osd.2 (osd.2) 150 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:47:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:41 smithi123 bash[19119]: cluster 2024-02-12T16:47:40.350025+0000 mgr.y (mgr.14146) 1904 : cluster [DBG] pgmap v2810: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:41 smithi121 bash[18489]: cluster 2024-02-12T16:47:39.633263+0000 osd.2 (osd.2) 150 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:47:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:41 smithi121 bash[18489]: cluster 2024-02-12T16:47:40.350025+0000 mgr.y (mgr.14146) 1904 : cluster [DBG] pgmap v2810: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:41 smithi121 bash[18489]: debug 2024-02-12T16:47:41.625+0000 7fbba85de700 -1 mon.a@0(leader).osd e1039 definitely_dead 0 2024-02-12T16:47:42.443 INFO:teuthology.orchestra.run.smithi121.stderr:marked in osd.9. 2024-02-12T16:47:42.641 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:47:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:42 smithi169 bash[19604]: audit 2024-02-12T16:47:41.629966+0000 mon.a (mon.0) 3396 : audit [DBG] from='client.? v1:172.21.15.121:0/3111145890' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-02-12T16:47:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:42 smithi123 bash[19119]: audit 2024-02-12T16:47:41.629966+0000 mon.a (mon.0) 3396 : audit [DBG] from='client.? v1:172.21.15.121:0/3111145890' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-02-12T16:47:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:42 smithi121 bash[18489]: audit 2024-02-12T16:47:41.629966+0000 mon.a (mon.0) 3396 : audit [DBG] from='client.? v1:172.21.15.121:0/3111145890' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-02-12T16:47:43.218 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-02-12T16:47:43.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-12T16:47:43.444 INFO:teuthology.orchestra.run.smithi121.stderr:nodeep-scrub is unset 2024-02-12T16:47:43.502 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:43 smithi121 bash[18489]: cluster 2024-02-12T16:47:42.252730+0000 osd.3 (osd.3) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:43.502 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:43 smithi121 bash[18489]: cluster 2024-02-12T16:47:42.252850+0000 osd.3 (osd.3) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:43.502 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:43 smithi121 bash[18489]: cluster 2024-02-12T16:47:42.350713+0000 mgr.y (mgr.14146) 1905 : cluster [DBG] pgmap v2811: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:43.502 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:43 smithi121 bash[18489]: audit 2024-02-12T16:47:42.431878+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? v1:172.21.15.121:0/3111145890' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-02-12T16:47:43.502 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:43 smithi121 bash[18489]: cluster 2024-02-12T16:47:42.431985+0000 mon.a (mon.0) 3398 : cluster [DBG] osdmap e1040: 12 total, 12 up, 12 in 2024-02-12T16:47:43.502 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:43 smithi121 bash[18489]: audit 2024-02-12T16:47:42.578814+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? v1:172.21.15.121:0/100927444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:47:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:43 smithi169 bash[19604]: cluster 2024-02-12T16:47:42.252730+0000 osd.3 (osd.3) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:43 smithi169 bash[19604]: cluster 2024-02-12T16:47:42.252850+0000 osd.3 (osd.3) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:43 smithi169 bash[19604]: cluster 2024-02-12T16:47:42.350713+0000 mgr.y (mgr.14146) 1905 : cluster [DBG] pgmap v2811: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:43 smithi169 bash[19604]: audit 2024-02-12T16:47:42.431878+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? v1:172.21.15.121:0/3111145890' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-02-12T16:47:43.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:43 smithi169 bash[19604]: cluster 2024-02-12T16:47:42.431985+0000 mon.a (mon.0) 3398 : cluster [DBG] osdmap e1040: 12 total, 12 up, 12 in 2024-02-12T16:47:43.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:43 smithi169 bash[19604]: audit 2024-02-12T16:47:42.578814+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? v1:172.21.15.121:0/100927444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:47:43.846 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:43 smithi123 bash[19119]: cluster 2024-02-12T16:47:42.252730+0000 osd.3 (osd.3) 134 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:43.846 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:43 smithi123 bash[19119]: cluster 2024-02-12T16:47:42.252850+0000 osd.3 (osd.3) 135 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:43.846 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:43 smithi123 bash[19119]: cluster 2024-02-12T16:47:42.350713+0000 mgr.y (mgr.14146) 1905 : cluster [DBG] pgmap v2811: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:43.846 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:43 smithi123 bash[19119]: audit 2024-02-12T16:47:42.431878+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? v1:172.21.15.121:0/3111145890' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-02-12T16:47:43.846 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:43 smithi123 bash[19119]: cluster 2024-02-12T16:47:42.431985+0000 mon.a (mon.0) 3398 : cluster [DBG] osdmap e1040: 12 total, 12 up, 12 in 2024-02-12T16:47:43.846 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:43 smithi123 bash[19119]: audit 2024-02-12T16:47:42.578814+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? v1:172.21.15.121:0/100927444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-12T16:47:44.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:44 smithi121 bash[18489]: cluster 2024-02-12T16:47:43.427805+0000 mon.a (mon.0) 3400 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:47:44.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:44 smithi121 bash[18489]: cluster 2024-02-12T16:47:43.427841+0000 mon.a (mon.0) 3401 : cluster [INF] Cluster is now healthy 2024-02-12T16:47:44.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:44 smithi121 bash[18489]: audit 2024-02-12T16:47:43.434474+0000 mon.a (mon.0) 3402 : audit [INF] from='client.? v1:172.21.15.121:0/100927444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:47:44.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:44 smithi121 bash[18489]: cluster 2024-02-12T16:47:43.434558+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e1041: 12 total, 12 up, 12 in 2024-02-12T16:47:44.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:44 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.086730+0000 osd.9 (osd.9) 169 : cluster [DBG] 1.a scrub starts 2024-02-12T16:47:44.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:44 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.125316+0000 osd.9 (osd.9) 170 : cluster [DBG] 1.a scrub ok 2024-02-12T16:47:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:44 smithi169 bash[19604]: cluster 2024-02-12T16:47:43.427805+0000 mon.a (mon.0) 3400 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:47:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:44 smithi169 bash[19604]: cluster 2024-02-12T16:47:43.427841+0000 mon.a (mon.0) 3401 : cluster [INF] Cluster is now healthy 2024-02-12T16:47:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:44 smithi169 bash[19604]: audit 2024-02-12T16:47:43.434474+0000 mon.a (mon.0) 3402 : audit [INF] from='client.? v1:172.21.15.121:0/100927444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:47:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:44 smithi169 bash[19604]: cluster 2024-02-12T16:47:43.434558+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e1041: 12 total, 12 up, 12 in 2024-02-12T16:47:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:44 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.086730+0000 osd.9 (osd.9) 169 : cluster [DBG] 1.a scrub starts 2024-02-12T16:47:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:44 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.125316+0000 osd.9 (osd.9) 170 : cluster [DBG] 1.a scrub ok 2024-02-12T16:47:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:44 smithi123 bash[19119]: cluster 2024-02-12T16:47:43.427805+0000 mon.a (mon.0) 3400 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-12T16:47:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:44 smithi123 bash[19119]: cluster 2024-02-12T16:47:43.427841+0000 mon.a (mon.0) 3401 : cluster [INF] Cluster is now healthy 2024-02-12T16:47:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:44 smithi123 bash[19119]: audit 2024-02-12T16:47:43.434474+0000 mon.a (mon.0) 3402 : audit [INF] from='client.? v1:172.21.15.121:0/100927444' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-12T16:47:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:44 smithi123 bash[19119]: cluster 2024-02-12T16:47:43.434558+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e1041: 12 total, 12 up, 12 in 2024-02-12T16:47:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:44 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.086730+0000 osd.9 (osd.9) 169 : cluster [DBG] 1.a scrub starts 2024-02-12T16:47:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:44 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.125316+0000 osd.9 (osd.9) 170 : cluster [DBG] 1.a scrub ok 2024-02-12T16:47:45.550 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:47:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:47:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:47:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26785.0:106018 8.18 8:1c94a6d8:::benchmark_data_smithi169_115677_object23390:head [delete] snapc 0=[] ondisk+write+known_if_redirected e882)", 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:40:58.366796+0000", 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 407.176039033, 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0697817030000001, 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26785", 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 106018 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.366796+0000", 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.366796+0000", 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999948 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.366791+0000", 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8396000000000001e-05 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.366809+0000", 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7898e-05 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.366837+0000", 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1780000000000001e-05 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.366849+0000", 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.11686326499999999 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.483712+0000", 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4620000000000001e-06 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:58.483718+0000", 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.93851068500000001 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:59.422229+0000", 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103213 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:59.422332+0000", 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1778000000000001e-05 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3", 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:59.422424+0000", 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030546219999999999 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:59.425479+0000", 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01101571 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:59.436494+0000", 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1337000000000001e-05 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:59.436536+0000", 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1655e-05 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:40:59.436577+0000", 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0697817030000001 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66885 9.a 9:5e6d2de8:::benchmark_data_smithi169_124168_object66884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.200337+0000", 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 284.34249796900002, 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43735970499999999, 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66885 2024-02-12T16:47:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200337+0000", 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200337+0000", 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999919 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200329+0000", 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4385000000000002e-05 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200383+0000", 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2992e-05 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200396+0000", 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.668e-06 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200401+0000", 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7512e-05 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200428+0000", 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1780000000000006e-05 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200510+0000", 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011618899999999999 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200626+0000", 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032965379999999999 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.203923+0000", 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43371046899999999 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637633+0000", 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7926999999999998e-05 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637671+0000", 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5201000000000001e-05 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637696+0000", 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43735970499999999 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66886 9.a 9:550d5e92:::benchmark_data_smithi169_124168_object66885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.200694+0000", 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 284.342140986, 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43711446999999998, 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66886 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200694+0000", 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200694+0000", 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999924 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200686+0000", 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2293999999999996e-05 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200758+0000", 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6050000000000003e-06 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200765+0000", 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9689999999999994e-06 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.200773+0000", 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014839090000000001 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.202257+0000", 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141908 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.202399+0000", 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016407999999999999 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.202563+0000", 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001471794 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.204035+0000", 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43371074599999998 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637745+0000", 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637782+0000", 2024-02-12T16:47:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6067e-05 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637808+0000", 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.43711446999999998 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66894 9.6 9:6205fe83:::benchmark_data_smithi169_124168_object66893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.206009+0000", 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 284.33682552300002, 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42733294599999999, 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66894 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206009+0000", 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206009+0000", 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206008+0000", 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2534999999999998e-05 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206060+0000", 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9890000000000004e-06 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206070+0000", 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3869999999999997e-06 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206074+0000", 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6932999999999999e-05 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206121+0000", 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013759500000000001 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206259+0000", 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017055299999999999 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.206429+0000", 2024-02-12T16:47:45.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021563609999999999 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.208586+0000", 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42446620299999999 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.633052+0000", 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014058000000000001 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.633193+0000", 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014942699999999999 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.633342+0000", 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42733294599999999 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66911 9.6 9:657f120d:::benchmark_data_smithi169_124168_object66910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.222601+0000", 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 284.32023403699998, 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.41113147999999999, 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.572 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66911 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222601+0000", 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222601+0000", 2024-02-12T16:47:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222598+0000", 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1854999999999998e-05 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222640+0000", 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3413e-05 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222654+0000", 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7359999999999998e-06 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222659+0000", 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9249999999999998e-05 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222709+0000", 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012672900000000001 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222835+0000", 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153769 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222989+0000", 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002040807 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.225030+0000", 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.40848518299999997 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.633515+0000", 2024-02-12T16:47:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013322 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.633648+0000", 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3867000000000002e-05 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.633732+0000", 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.41113147999999999 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66912 9.a 9:59b67dda:::benchmark_data_smithi169_124168_object66911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.222732+0000", 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 284.32010246700003, 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.415159051, 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66912 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222732+0000", 2024-02-12T16:47:45.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222732+0000", 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222731+0000", 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2879000000000006e-05 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222794+0000", 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9889999999999992e-06 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222802+0000", 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4450000000000001e-06 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222805+0000", 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6322000000000001e-05 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222852+0000", 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1661000000000003e-05 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.222933+0000", 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8853999999999999e-05 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.223032+0000", 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018522230000000001 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.224884+0000", 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.41295535700000002 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637840+0000", 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9592000000000001e-05 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637869+0000", 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.179e-05 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637891+0000", 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.415159051 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:201046 9.a 9:5e7b3634:::benchmark_data_smithi169_124168_object50362:head [delete] snapc 0=[] ondisk+write+known_if_redirected e969)", 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:10.833641+0000", 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 154.70919333399999, 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34230059000000002, 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 201046 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.833641+0000", 2024-02-12T16:47:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.833641+0000", 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.833639+0000", 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.397e-06 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.833649+0000", 2024-02-12T16:47:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0613000000000001e-05 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.833669+0000", 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8700000000000003e-06 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.833676+0000", 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050070050000000001 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.838683+0000", 2024-02-12T16:47:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000137544 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.838821+0000", 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018068399999999999 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.839002+0000", 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084375000000000006 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:10.847439+0000", 2024-02-12T16:47:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32842255199999998 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:11.175862+0000", 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3266999999999997e-05 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:11.175915+0000", 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6982999999999999e-05 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:11.175942+0000", 2024-02-12T16:47:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34230059000000002 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236324 9.6 9:6b638078:::benchmark_data_smithi169_124168_object85640:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.400831+0000", 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 115.14200365000001, 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48114621299999999, 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236324 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.400831+0000", 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.400831+0000", 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.400829+0000", 2024-02-12T16:47:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9040000000000003e-06 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.400836+0000", 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3045e-05 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.400849+0000", 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2120000000000003e-06 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.400857+0000", 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034251210000000002 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.404282+0000", 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113275 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.404396+0000", 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0593000000000006e-05 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.404476+0000", 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041447699999999999 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.408621+0000", 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47320846100000002 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.881830+0000", 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2306999999999996e-05 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.881912+0000", 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5333000000000006e-05 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.881977+0000", 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.48114621299999999 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236347 9.f 9:f549ca6a:::benchmark_data_smithi169_124168_object85663:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.420047+0000", 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 115.122787661, 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47569077900000001, 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236347 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420047+0000", 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420047+0000", 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420044+0000", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5750000000000007e-06 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420054+0000", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5187999999999999e-05 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420069+0000", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9210000000000003e-06 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420076+0000", 2024-02-12T16:47:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4196999999999995e-05 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420140+0000", 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011571800000000001 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420256+0000", 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010479099999999999 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.420360+0000", 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092661559999999994 2024-02-12T16:47:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.429627+0000", 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.465841699 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.895468+0000", 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013965700000000001 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.895608+0000", 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012978100000000001 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.895738+0000", 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47569077900000001 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236350 9.f 9:f1368a16:::benchmark_data_smithi169_124168_object85666:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.424640+0000", 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 115.118194811, 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47952399200000001, 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.598 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236350 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.424640+0000", 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.424640+0000", 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.424638+0000", 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5000000000000005e-06 2024-02-12T16:47:45.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.424647+0000", 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2427e-05 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.424660+0000", 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1379999999999998e-06 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.424665+0000", 2024-02-12T16:47:45.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054637700000000002 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425211+0000", 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013630000000000001 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425348+0000", 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016759000000000001 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425515+0000", 2024-02-12T16:47:45.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0042944009999999998 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.429810+0000", 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47427902199999999 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.904089+0000", 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2188999999999999e-05 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.904131+0000", 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3111999999999997e-05 2024-02-12T16:47:45.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.904164+0000", 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47952399200000001 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236353 9.f 9:fa07f442:::benchmark_data_smithi169_124168_object85669:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.431928+0000", 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 115.110907024, 2024-02-12T16:47:45.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47239493399999999, 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236353 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.431928+0000", 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.431928+0000", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.431925+0000", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1070000000000005e-06 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.431933+0000", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3919e-05 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.431947+0000", 2024-02-12T16:47:45.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5199999999999997e-06 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.431953+0000", 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1081999999999997e-05 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.431994+0000", 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1742000000000004e-05 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.432086+0000", 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000123418 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.432209+0000", 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0078878449999999992 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.440097+0000", 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46411088700000003 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.904208+0000", 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5783000000000003e-05 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.904284+0000", 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8868999999999997e-05 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.904323+0000", 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47239493399999999 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236354 9.f 9:fb41a7ad:::benchmark_data_smithi169_124168_object85670:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.433385+0000", 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 115.10944983500001, 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47608149700000002, 2024-02-12T16:47:45.608 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236354 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433385+0000", 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433385+0000", 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:45.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433383+0000", 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8830000000000006e-06 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433391+0000", 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0761000000000001e-05 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433402+0000", 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3769999999999998e-06 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433406+0000", 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027613059999999998 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.436168+0000", 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2342999999999993e-05 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.436260+0000", 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3126000000000003e-05 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.436353+0000", 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039010809999999998 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.440254+0000", 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.469142004 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.909396+0000", 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7421e-05 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.909434+0000", 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2566e-05 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.909466+0000", 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47608149700000002 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236355 9.6 9:6be22e41:::benchmark_data_smithi169_124168_object85671:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.433439+0000", 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 115.10939544999999, 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46036730599999998, 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.613 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236355 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433439+0000", 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433439+0000", 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:47:45.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433438+0000", 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.442e-06 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433443+0000", 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5009999999999998e-06 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433450+0000", 2024-02-12T16:47:45.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2119999999999999e-06 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433453+0000", 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9033999999999999e-05 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433492+0000", 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9701000000000001e-05 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433582+0000", 2024-02-12T16:47:45.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000138014 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433720+0000", 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043191360000000003 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.438039+0000", 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.45542792700000001 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.893467+0000", 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019995200000000001 2024-02-12T16:47:45.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.893667+0000", 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013930099999999999 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.893807+0000", 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46036730599999998 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.618 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:265177 9.12 9.3fdc3ef2 (undecoded) ondisk+write+known_if_redirected e996)", 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:19.059883+0000", 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.482952065999996, 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.98957258100000001, 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265177 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.059883+0000", 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.059883+0000", 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.059882+0000", 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.126e-06 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.059886+0000", 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2489999999999998e-06 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.059892+0000", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.378e-06 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.059897+0000", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0051314840000000004 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.065029+0000", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5379999999999998e-06 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.065032+0000", 2024-02-12T16:47:45.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.984403374 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049436+0000", 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9306000000000001e-05 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049455+0000", 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.98957258100000001 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.622 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:265256 9.12 9.9cab8872 (undecoded) ondisk+write+known_if_redirected e996)", 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:19.116610+0000", 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.42622489, 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.93304926799999999, 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265256 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.116610+0000", 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.116610+0000", 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.116608+0000", 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6919999999999999e-06 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.116614+0000", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4980000000000006e-06 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.116623+0000", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5190000000000003e-06 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.116627+0000", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082232420000000004 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.124850+0000", 2024-02-12T16:47:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5610000000000004e-06 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.124857+0000", 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.92478877699999995 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049646+0000", 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3291e-05 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049659+0000", 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.93304926799999999 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:265274 9.12 9.12c16a52 (undecoded) ondisk+write+known_if_redirected e996)", 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:19.132748+0000", 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.410087086000004, 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.91697698900000002, 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265274 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.132748+0000", 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.132748+0000", 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.132746+0000", 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.472e-06 2024-02-12T16:47:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.132752+0000", 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2225999999999999e-05 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.132764+0000", 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8119999999999998e-06 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.132769+0000", 2024-02-12T16:47:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012615549999999999 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.134031+0000", 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.905e-06 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.134038+0000", 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.91566952300000004 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049707+0000", 2024-02-12T16:47:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7350000000000002e-05 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049725+0000", 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.91697698900000002 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:265320 9.12 9.1862bf72 (undecoded) ondisk+write+known_if_redirected e996)", 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:19.168013+0000", 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.374821159999996, 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.88172950000000005, 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:47:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265320 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168013+0000", 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168013+0000", 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168011+0000", 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4130000000000003e-06 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168018+0000", 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2853999999999999e-05 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168031+0000", 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6449999999999999e-06 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168037+0000", 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1919999999999999e-05 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168069+0000", 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5530000000000003e-06 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.168074+0000", 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.88165973500000006 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049734+0000", 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3710000000000006e-06 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049743+0000", 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.88172950000000005 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.635 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:265326 9.12 9.35096ab2 (undecoded) ondisk+write+known_if_redirected e996)", 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:19.184698+0000", 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.358136642999995, 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.86509519099999999, 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265326 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184698+0000", 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184698+0000", 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184696+0000", 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6510000000000004e-06 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184703+0000", 2024-02-12T16:47:45.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.491e-05 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184718+0000", 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5470000000000002e-06 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184724+0000", 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0237e-05 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184775+0000", 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6440000000000003e-06 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.184781+0000", 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.86499770200000003 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049779+0000", 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.416e-05 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049793+0000", 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.86509519099999999 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.639 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:265347 9.12 9.ced8bd12 (undecoded) ondisk+write+known_if_redirected e996)", 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:19.207459+0000", 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.335375161000002, 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.84235215200000002, 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265347 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207459+0000", 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207459+0000", 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207458+0000", 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.998e-06 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207464+0000", 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8849999999999999e-06 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207472+0000", 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2039999999999999e-06 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207476+0000", 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6281000000000001e-05 2024-02-12T16:47:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207502+0000", 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9990000000000002e-06 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.207506+0000", 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.842299609 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049806+0000", 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7760000000000003e-06 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049812+0000", 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.84235215200000002 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:265351 9.12 9.d0489bf2 (undecoded) ondisk+write+known_if_redirected e996)", 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:19.211697+0000", 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.331137384000002, 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.83815298100000002, 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:47:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 265351 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211697+0000", 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211697+0000", 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:47:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211696+0000", 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1719999999999998e-06 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211701+0000", 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0553999999999999e-05 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211712+0000", 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4329999999999997e-06 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211716+0000", 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2614000000000002e-05 2024-02-12T16:47:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211759+0000", 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7780000000000007e-06 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for peered", 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:19.211767+0000", 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.83807450800000005 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049841+0000", 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.217e-06 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:47:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:20.049850+0000", 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.83815298100000002 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:47:45.648 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:47:45.648 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:45 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.344026+0000 osd.8 (osd.8) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:45.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:45 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.344116+0000 osd.8 (osd.8) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:45.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:45 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.351455+0000 mgr.y (mgr.14146) 1906 : cluster [DBG] pgmap v2814: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:45.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:45 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.454658+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e1042: 12 total, 12 up, 12 in 2024-02-12T16:47:45.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:45 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.863849+0000 osd.6 (osd.6) 196 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:47:45.649 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:45 smithi123 bash[19119]: audit 2024-02-12T16:47:45.164499+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:45 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.344026+0000 osd.8 (osd.8) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:45 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.344116+0000 osd.8 (osd.8) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:45 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.351455+0000 mgr.y (mgr.14146) 1906 : cluster [DBG] pgmap v2814: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:45 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.454658+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e1042: 12 total, 12 up, 12 in 2024-02-12T16:47:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:45 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.863849+0000 osd.6 (osd.6) 196 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:47:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:45 smithi169 bash[19604]: audit 2024-02-12T16:47:45.164499+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:45 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.344026+0000 osd.8 (osd.8) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:45 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.344116+0000 osd.8 (osd.8) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:45 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.351455+0000 mgr.y (mgr.14146) 1906 : cluster [DBG] pgmap v2814: 31 pgs: 31 active+clean; 0 B data, 8.9 GiB used, 975 GiB / 983 GiB avail 2024-02-12T16:47:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:45 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.454658+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e1042: 12 total, 12 up, 12 in 2024-02-12T16:47:45.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:45 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.863849+0000 osd.6 (osd.6) 196 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:47:45.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:45 smithi121 bash[18489]: audit 2024-02-12T16:47:45.164499+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:46.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-12T16:47:46.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:46 smithi123 bash[19119]: cluster 2024-02-12T16:47:44.992909+0000 osd.6 (osd.6) 197 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:47:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:46 smithi169 bash[19604]: cluster 2024-02-12T16:47:44.992909+0000 osd.6 (osd.6) 197 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:47:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:46 smithi121 bash[18489]: cluster 2024-02-12T16:47:44.992909+0000 osd.6 (osd.6) 197 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:47:47.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:47 smithi169 bash[19604]: cluster 2024-02-12T16:47:46.352152+0000 mgr.y (mgr.14146) 1907 : cluster [DBG] pgmap v2816: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:47 smithi123 bash[19119]: cluster 2024-02-12T16:47:46.352152+0000 mgr.y (mgr.14146) 1907 : cluster [DBG] pgmap v2816: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:47.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:47 smithi121 bash[18489]: cluster 2024-02-12T16:47:46.352152+0000 mgr.y (mgr.14146) 1907 : cluster [DBG] pgmap v2816: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:48 smithi169 bash[19604]: cluster 2024-02-12T16:47:47.013627+0000 osd.0 (osd.0) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:48 smithi169 bash[19604]: cluster 2024-02-12T16:47:47.013717+0000 osd.0 (osd.0) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:48 smithi169 bash[19604]: cluster 2024-02-12T16:47:47.207198+0000 osd.0 (osd.0) 169 : cluster [DBG] 1.c scrub starts 2024-02-12T16:47:48.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:48 smithi169 bash[19604]: cluster 2024-02-12T16:47:47.305287+0000 osd.0 (osd.0) 170 : cluster [DBG] 1.c scrub ok 2024-02-12T16:47:48.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:48 smithi169 bash[19604]: cluster 2024-02-12T16:47:47.476452+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e1043: 12 total, 12 up, 12 in 2024-02-12T16:47:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:48 smithi123 bash[19119]: cluster 2024-02-12T16:47:47.013627+0000 osd.0 (osd.0) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:48 smithi123 bash[19119]: cluster 2024-02-12T16:47:47.013717+0000 osd.0 (osd.0) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:48 smithi123 bash[19119]: cluster 2024-02-12T16:47:47.207198+0000 osd.0 (osd.0) 169 : cluster [DBG] 1.c scrub starts 2024-02-12T16:47:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:48 smithi123 bash[19119]: cluster 2024-02-12T16:47:47.305287+0000 osd.0 (osd.0) 170 : cluster [DBG] 1.c scrub ok 2024-02-12T16:47:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:48 smithi123 bash[19119]: cluster 2024-02-12T16:47:47.476452+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e1043: 12 total, 12 up, 12 in 2024-02-12T16:47:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:48 smithi121 bash[18489]: cluster 2024-02-12T16:47:47.013627+0000 osd.0 (osd.0) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:47:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:48 smithi121 bash[18489]: cluster 2024-02-12T16:47:47.013717+0000 osd.0 (osd.0) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:47:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:48 smithi121 bash[18489]: cluster 2024-02-12T16:47:47.207198+0000 osd.0 (osd.0) 169 : cluster [DBG] 1.c scrub starts 2024-02-12T16:47:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:48 smithi121 bash[18489]: cluster 2024-02-12T16:47:47.305287+0000 osd.0 (osd.0) 170 : cluster [DBG] 1.c scrub ok 2024-02-12T16:47:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:48 smithi121 bash[18489]: cluster 2024-02-12T16:47:47.476452+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e1043: 12 total, 12 up, 12 in 2024-02-12T16:47:49.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:49 smithi121 bash[18489]: cluster 2024-02-12T16:47:48.352739+0000 mgr.y (mgr.14146) 1908 : cluster [DBG] pgmap v2818: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:49 smithi169 bash[19604]: cluster 2024-02-12T16:47:48.352739+0000 mgr.y (mgr.14146) 1908 : cluster [DBG] pgmap v2818: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:49 smithi123 bash[19119]: cluster 2024-02-12T16:47:48.352739+0000 mgr.y (mgr.14146) 1908 : cluster [DBG] pgmap v2818: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:51.016 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:47:51.423 INFO:teuthology.orchestra.run.smithi121.stdout:mon.a: {} 2024-02-12T16:47:51.442 INFO:teuthology.orchestra.run.smithi121.stdout:mon.c: {} 2024-02-12T16:47:51.463 INFO:teuthology.orchestra.run.smithi121.stdout:mon.b: {} 2024-02-12T16:47:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-12T16:47:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-12T16:47:51.466 INFO:teuthology.orchestra.run.smithi121.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-12T16:47:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:51 smithi169 bash[19604]: cluster 2024-02-12T16:47:50.353465+0000 mgr.y (mgr.14146) 1909 : cluster [DBG] pgmap v2819: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:51 smithi169 bash[19604]: audit 2024-02-12T16:47:51.400403+0000 mon.a (mon.0) 3407 : audit [DBG] from='client.? v1:172.21.15.121:0/2011001327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:47:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:51 smithi123 bash[19119]: cluster 2024-02-12T16:47:50.353465+0000 mgr.y (mgr.14146) 1909 : cluster [DBG] pgmap v2819: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:51 smithi123 bash[19119]: audit 2024-02-12T16:47:51.400403+0000 mon.a (mon.0) 3407 : audit [DBG] from='client.? v1:172.21.15.121:0/2011001327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:47:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:51 smithi121 bash[18489]: cluster 2024-02-12T16:47:50.353465+0000 mgr.y (mgr.14146) 1909 : cluster [DBG] pgmap v2819: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:51 smithi121 bash[18489]: audit 2024-02-12T16:47:51.400403+0000 mon.a (mon.0) 3407 : audit [DBG] from='client.? v1:172.21.15.121:0/2011001327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-12T16:47:52.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 0 1 2024-02-12T16:47:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:53 smithi169 bash[19604]: cluster 2024-02-12T16:47:52.354290+0000 mgr.y (mgr.14146) 1910 : cluster [DBG] pgmap v2820: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:53.818 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:53 smithi123 bash[19119]: cluster 2024-02-12T16:47:52.354290+0000 mgr.y (mgr.14146) 1910 : cluster [DBG] pgmap v2820: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:53 smithi121 bash[18489]: cluster 2024-02-12T16:47:52.354290+0000 mgr.y (mgr.14146) 1910 : cluster [DBG] pgmap v2820: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:53.980 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:47:53.980 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:47:53.980 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:47:53.981 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:47:54.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-12T16:47:55.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:55 smithi169 bash[19604]: cluster 2024-02-12T16:47:54.355021+0000 mgr.y (mgr.14146) 1911 : cluster [DBG] pgmap v2821: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:55.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:55 smithi169 bash[19604]: audit 2024-02-12T16:47:55.184970+0000 mon.a (mon.0) 3408 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:55 smithi123 bash[19119]: cluster 2024-02-12T16:47:54.355021+0000 mgr.y (mgr.14146) 1911 : cluster [DBG] pgmap v2821: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:55 smithi123 bash[19119]: audit 2024-02-12T16:47:55.184970+0000 mon.a (mon.0) 3408 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:55 smithi121 bash[18489]: cluster 2024-02-12T16:47:54.355021+0000 mgr.y (mgr.14146) 1911 : cluster [DBG] pgmap v2821: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:55 smithi121 bash[18489]: audit 2024-02-12T16:47:55.184970+0000 mon.a (mon.0) 3408 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:57.051 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:47:57.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:57 smithi123 bash[19119]: cluster 2024-02-12T16:47:56.189284+0000 osd.3 (osd.3) 136 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:47:57.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:57 smithi123 bash[19119]: cluster 2024-02-12T16:47:56.337946+0000 osd.3 (osd.3) 137 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:47:57.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:57 smithi123 bash[19119]: cluster 2024-02-12T16:47:56.355522+0000 mgr.y (mgr.14146) 1912 : cluster [DBG] pgmap v2822: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:57 smithi121 bash[18489]: cluster 2024-02-12T16:47:56.189284+0000 osd.3 (osd.3) 136 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:47:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:57 smithi121 bash[18489]: cluster 2024-02-12T16:47:56.337946+0000 osd.3 (osd.3) 137 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:47:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:57 smithi121 bash[18489]: cluster 2024-02-12T16:47:56.355522+0000 mgr.y (mgr.14146) 1912 : cluster [DBG] pgmap v2822: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:57 smithi169 bash[19604]: cluster 2024-02-12T16:47:56.189284+0000 osd.3 (osd.3) 136 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:47:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:57 smithi169 bash[19604]: cluster 2024-02-12T16:47:56.337946+0000 osd.3 (osd.3) 137 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:47:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:57 smithi169 bash[19604]: cluster 2024-02-12T16:47:56.355522+0000 mgr.y (mgr.14146) 1912 : cluster [DBG] pgmap v2822: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:58 smithi123 bash[19119]: audit 2024-02-12T16:47:57.596540+0000 mon.a (mon.0) 3409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:58 smithi123 bash[19119]: audit 2024-02-12T16:47:57.922095+0000 mon.a (mon.0) 3410 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:58 smithi123 bash[19119]: audit 2024-02-12T16:47:57.922968+0000 mon.a (mon.0) 3411 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:58.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:58 smithi123 bash[19119]: audit 2024-02-12T16:47:57.928793+0000 mon.a (mon.0) 3412 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:58 smithi121 bash[18489]: audit 2024-02-12T16:47:57.596540+0000 mon.a (mon.0) 3409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:58 smithi121 bash[18489]: audit 2024-02-12T16:47:57.922095+0000 mon.a (mon.0) 3410 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:58 smithi121 bash[18489]: audit 2024-02-12T16:47:57.922968+0000 mon.a (mon.0) 3411 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:58 smithi121 bash[18489]: audit 2024-02-12T16:47:57.928793+0000 mon.a (mon.0) 3412 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:58 smithi169 bash[19604]: audit 2024-02-12T16:47:57.596540+0000 mon.a (mon.0) 3409 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:47:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:58 smithi169 bash[19604]: audit 2024-02-12T16:47:57.922095+0000 mon.a (mon.0) 3410 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:47:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:58 smithi169 bash[19604]: audit 2024-02-12T16:47:57.922968+0000 mon.a (mon.0) 3411 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:47:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:58 smithi169 bash[19604]: audit 2024-02-12T16:47:57.928793+0000 mon.a (mon.0) 3412 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:47:59.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:47:59 smithi123 bash[19119]: cluster 2024-02-12T16:47:58.356147+0000 mgr.y (mgr.14146) 1913 : cluster [DBG] pgmap v2823: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:47:59 smithi121 bash[18489]: cluster 2024-02-12T16:47:58.356147+0000 mgr.y (mgr.14146) 1913 : cluster [DBG] pgmap v2823: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:47:59.426 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:47:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:47:59 smithi169 bash[19604]: cluster 2024-02-12T16:47:58.356147+0000 mgr.y (mgr.14146) 1913 : cluster [DBG] pgmap v2823: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:01.214 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.0 to 1 (10000) 2024-02-12T16:48:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:01 smithi169 bash[19604]: audit 2024-02-12T16:48:00.196723+0000 mon.a (mon.0) 3413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:48:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:01 smithi169 bash[19604]: cluster 2024-02-12T16:48:00.356904+0000 mgr.y (mgr.14146) 1914 : cluster [DBG] pgmap v2824: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:01 smithi169 bash[19604]: audit 2024-02-12T16:48:00.465128+0000 mon.a (mon.0) 3414 : audit [INF] from='client.? v1:172.21.15.121:0/2089468601' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:48:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:01 smithi123 bash[19119]: audit 2024-02-12T16:48:00.196723+0000 mon.a (mon.0) 3413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:48:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:01 smithi123 bash[19119]: cluster 2024-02-12T16:48:00.356904+0000 mgr.y (mgr.14146) 1914 : cluster [DBG] pgmap v2824: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:01 smithi123 bash[19119]: audit 2024-02-12T16:48:00.465128+0000 mon.a (mon.0) 3414 : audit [INF] from='client.? v1:172.21.15.121:0/2089468601' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:48:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:01 smithi121 bash[18489]: audit 2024-02-12T16:48:00.196723+0000 mon.a (mon.0) 3413 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:48:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:01 smithi121 bash[18489]: cluster 2024-02-12T16:48:00.356904+0000 mgr.y (mgr.14146) 1914 : cluster [DBG] pgmap v2824: 31 pgs: 31 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:01 smithi121 bash[18489]: audit 2024-02-12T16:48:00.465128+0000 mon.a (mon.0) 3414 : audit [INF] from='client.? v1:172.21.15.121:0/2089468601' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:48:02.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-12T16:48:02.300 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:48:02.301 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:48:02.301 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2024-02-12T16:48:02.301 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2024-02-12T16:48:02.301 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:48:02.351 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:02 smithi123 bash[19119]: audit 2024-02-12T16:48:01.199352+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? v1:172.21.15.121:0/2089468601' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:48:02.351 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:02 smithi123 bash[19119]: cluster 2024-02-12T16:48:01.199556+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e1044: 12 total, 12 up, 12 in 2024-02-12T16:48:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:02 smithi121 bash[18489]: audit 2024-02-12T16:48:01.199352+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? v1:172.21.15.121:0/2089468601' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:48:02.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:02 smithi121 bash[18489]: cluster 2024-02-12T16:48:01.199556+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e1044: 12 total, 12 up, 12 in 2024-02-12T16:48:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:02 smithi169 bash[19604]: audit 2024-02-12T16:48:01.199352+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? v1:172.21.15.121:0/2089468601' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:48:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:02 smithi169 bash[19604]: cluster 2024-02-12T16:48:01.199556+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e1044: 12 total, 12 up, 12 in 2024-02-12T16:48:02.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-12T16:48:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:03 smithi169 bash[19604]: cluster 2024-02-12T16:48:02.213530+0000 mon.a (mon.0) 3417 : cluster [DBG] osdmap e1045: 12 total, 12 up, 12 in 2024-02-12T16:48:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:03 smithi169 bash[19604]: cluster 2024-02-12T16:48:02.357571+0000 mgr.y (mgr.14146) 1915 : cluster [DBG] pgmap v2827: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:03 smithi123 bash[19119]: cluster 2024-02-12T16:48:02.213530+0000 mon.a (mon.0) 3417 : cluster [DBG] osdmap e1045: 12 total, 12 up, 12 in 2024-02-12T16:48:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:03 smithi123 bash[19119]: cluster 2024-02-12T16:48:02.357571+0000 mgr.y (mgr.14146) 1915 : cluster [DBG] pgmap v2827: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:03 smithi121 bash[18489]: cluster 2024-02-12T16:48:02.213530+0000 mon.a (mon.0) 3417 : cluster [DBG] osdmap e1045: 12 total, 12 up, 12 in 2024-02-12T16:48:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:03 smithi121 bash[18489]: cluster 2024-02-12T16:48:02.357571+0000 mgr.y (mgr.14146) 1915 : cluster [DBG] pgmap v2827: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:05 smithi169 bash[19604]: cluster 2024-02-12T16:48:04.358401+0000 mgr.y (mgr.14146) 1916 : cluster [DBG] pgmap v2828: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:05 smithi123 bash[19119]: cluster 2024-02-12T16:48:04.358401+0000 mgr.y (mgr.14146) 1916 : cluster [DBG] pgmap v2828: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:05 smithi121 bash[18489]: cluster 2024-02-12T16:48:04.358401+0000 mgr.y (mgr.14146) 1916 : cluster [DBG] pgmap v2828: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:06.801 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:48:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:07 smithi123 bash[19119]: cluster 2024-02-12T16:48:06.359219+0000 mgr.y (mgr.14146) 1917 : cluster [DBG] pgmap v2829: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:07 smithi121 bash[18489]: cluster 2024-02-12T16:48:06.359219+0000 mgr.y (mgr.14146) 1917 : cluster [DBG] pgmap v2829: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:07 smithi169 bash[19604]: cluster 2024-02-12T16:48:06.359219+0000 mgr.y (mgr.14146) 1917 : cluster [DBG] pgmap v2829: 31 pgs: 1 peering, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:07.774 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:48:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:09 smithi169 bash[19604]: cluster 2024-02-12T16:48:08.360008+0000 mgr.y (mgr.14146) 1918 : cluster [DBG] pgmap v2830: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:09.877 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:09 smithi121 bash[18489]: cluster 2024-02-12T16:48:08.360008+0000 mgr.y (mgr.14146) 1918 : cluster [DBG] pgmap v2830: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:09.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:09 smithi123 bash[19119]: cluster 2024-02-12T16:48:08.360008+0000 mgr.y (mgr.14146) 1918 : cluster [DBG] pgmap v2830: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:10.438 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-12T16:48:10.756 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:48:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:48:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:48:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:118837 9.1 e942/935 9:8c30544e:::benchmark_data_smithi169_124168_object118836:head v 942'7297, mlcod=925'510)", 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:36.452465+0000", 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 274.29845912100001, 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.056742094, 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.452465+0000", 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.452465+0000", 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.452464+0000", 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5591000000000003e-05 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.452529+0000", 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0912e-05 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.452540+0000", 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6080000000000001e-06 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.452547+0000", 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061365050000000004 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.458683+0000", 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9062000000000001e-05 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.458712+0000", 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.050460897999999997 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.509173+0000", 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4227999999999998e-05 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:36.509208+0000", 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.056742094 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:158404 9.1 e953/935 9:87c05c94:::benchmark_data_smithi169_124168_object7720:head v 953'9683, mlcod=926'1580)", 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:16.170498+0000", 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 234.580426155, 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.057622547000000003, 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.170498+0000", 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.170498+0000", 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.170497+0000", 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.833e-06 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.170506+0000", 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1056999999999999e-05 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.170517+0000", 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3809999999999998e-06 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.170522+0000", 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047594509999999996 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.175282+0000", 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1659000000000003e-05 2024-02-12T16:48:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.175314+0000", 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.052756253000000003 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.228070+0000", 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1122999999999998e-05 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:16.228121+0000", 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.057622547000000003 2024-02-12T16:48:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:164891 9.1 e955/935 9:880c1031:::benchmark_data_smithi169_124168_object14207:head v 955'10075, mlcod=926'1810)", 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:23.479608+0000", 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 227.271316498, 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.058010915000000003, 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.479608+0000", 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.479608+0000", 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.479605+0000", 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7292999999999999e-05 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.479622+0000", 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1030999999999999e-05 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.479643+0000", 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4516e-05 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.479658+0000", 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026274698999999999 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.505933+0000", 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9477e-05 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.505952+0000", 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031620429999999998 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.537572+0000", 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6505999999999998e-05 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:23.537619+0000", 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.058010915000000003 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:170258 9.1 e956/935 9:88938403:::benchmark_data_smithi169_124168_object19574:head v 956'10420, mlcod=926'2067)", 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:30.609641+0000", 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 220.141283133, 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036684199000000001, 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.609641+0000", 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.609641+0000", 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.609640+0000", 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4780000000000002e-06 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.609648+0000", 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2539999999999992e-06 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.609656+0000", 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.403e-06 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.609660+0000", 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070491410000000001 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.616709+0000", 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0009e-05 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.616740+0000", 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.029558557999999999 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.646298+0000", 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7563e-05 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:30.646326+0000", 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036684199000000001 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:171362 9.1 e956/935 9:80547fd3:::benchmark_data_smithi169_124168_object20678:head v 956'10491, mlcod=926'2119)", 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:32.076243+0000", 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 218.67468117499999, 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036438237999999998, 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.076243+0000", 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.076243+0000", 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.076242+0000", 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6739999999999992e-06 2024-02-12T16:48:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.076252+0000", 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3753e-05 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.076266+0000", 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4430000000000002e-06 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.076271+0000", 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040428030000000002 2024-02-12T16:48:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.080314+0000", 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8875999999999999e-05 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.080343+0000", 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032299422000000001 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.112642+0000", 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9558000000000003e-05 2024-02-12T16:48:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:32.112682+0000", 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036438237999999998 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:192985 9.1 e966/965 9:81c3a0a5:::benchmark_data_smithi169_124168_object42301:head v 966'11791, mlcod=927'3270)", 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:01.319147+0000", 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 189.431777678, 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036538832, 2024-02-12T16:48:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.319147+0000", 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.319147+0000", 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.319146+0000", 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3899999999999999e-06 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.319155+0000", 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6979999999999998e-06 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.319163+0000", 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6100000000000002e-06 2024-02-12T16:48:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.319167+0000", 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035139530000000002 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.322681+0000", 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7754e-05 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.322698+0000", 2024-02-12T16:48:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.032943348999999997 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.355642+0000", 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4103000000000001e-05 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:01.355686+0000", 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.036538832 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:194502 9.1 e967/965 9:8d3cb812:::benchmark_data_smithi169_124168_object43818:head v 967'11900, mlcod=928'3342)", 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:02.981105+0000", 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 187.769819351, 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.038258205000000003, 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.981105+0000", 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.981105+0000", 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.981103+0000", 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1080999999999999e-05 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.981114+0000", 2024-02-12T16:48:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3134e-05 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.981127+0000", 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3489999999999997e-06 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.981134+0000", 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061849180000000002 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.987319+0000", 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9247999999999998e-05 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:02.987348+0000", 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031976623000000003 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:03.019325+0000", 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8821e-05 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:03.019363+0000", 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.038258205000000003 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:195413 9.1 e967/965 9:83afd6fb:::benchmark_data_smithi169_124168_object44729:head v 967'11954, mlcod=928'3405)", 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:04.089770+0000", 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 186.66115433600001, 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044358124999999998, 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.089770+0000", 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.089770+0000", 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.089768+0000", 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2079000000000001e-05 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.089780+0000", 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6586999999999999e-05 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.089797+0000", 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8210000000000004e-06 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.089804+0000", 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057492890000000003 2024-02-12T16:48:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.095553+0000", 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2384000000000001e-05 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.095575+0000", 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.038502963000000001 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.134078+0000", 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.99e-05 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.134128+0000", 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.044358124999999998 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:195415 9.1 e967/965 9:8c2a78de:::benchmark_data_smithi169_124168_object44731:head v 967'11955, mlcod=928'3405)", 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:04.094255+0000", 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 186.656669929, 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039894605, 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.094255+0000", 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.094255+0000", 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.094252+0000", 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1745e-05 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.094264+0000", 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3132000000000001e-05 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.094277+0000", 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3770000000000004e-06 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.094283+0000", 2024-02-12T16:48:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083857250000000001 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.102668+0000", 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1838999999999998e-05 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.102700+0000", 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.031430687999999998 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.134131+0000", 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8247999999999999e-05 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:04.134149+0000", 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.039894605 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:254734 9.11 e992/991 9:8e6a81a9:::benchmark_data_smithi169_124168_object104050:head v 992'15422, mlcod=992'15420)", 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:12.205491+0000", 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 118.545433831, 2024-02-12T16:48:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010922540999999999, 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205491+0000", 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205491+0000", 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205489+0000", 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2683e-05 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205502+0000", 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4079e-05 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205516+0000", 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5290000000000003e-06 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205523+0000", 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3600999999999997e-05 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205577+0000", 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8240999999999999e-05 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.205605+0000", 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710447999999999 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.216316+0000", 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7724000000000002e-05 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:12.216413+0000", 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010922540999999999 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:272278 9.0 e999/983 9:07d294dd:::benchmark_data_smithi169_124168_object121594:head v 999'16218, mlcod=999'16216)", 2024-02-12T16:48:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:26.050647+0000", 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.700277976, 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021195281999999999, 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050647+0000", 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050647+0000", 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050644+0000", 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2728000000000001e-05 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050657+0000", 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4854e-05 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050672+0000", 2024-02-12T16:48:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9270000000000002e-06 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050678+0000", 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8594000000000003e-05 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050726+0000", 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0548999999999999e-05 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.050747+0000", 2024-02-12T16:48:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021043579999999999 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.071791+0000", 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1264000000000002e-05 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.071842+0000", 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021195281999999999 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:272336 9.10 e999/989 9:09b83d8d:::benchmark_data_smithi169_124168_object121652:head v 999'16289, mlcod=999'16287)", 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:26.109207+0000", 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 104.641717763, 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021859139, 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109207+0000", 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109207+0000", 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109204+0000", 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5173e-05 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109219+0000", 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8831e-05 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109238+0000", 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7389999999999994e-06 2024-02-12T16:48:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109248+0000", 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0222000000000006e-05 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109318+0000", 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9612e-05 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.109348+0000", 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021655708999999999 2024-02-12T16:48:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.131004+0000", 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2422999999999995e-05 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:26.131066+0000", 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021859139 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:274560 9.10 e999/989 9:0a75dd0d:::benchmark_data_smithi169_124168_object123876:head v 999'16362, mlcod=999'16360)", 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:27.732268+0000", 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 103.018656746, 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011860735000000001, 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732268+0000", 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732268+0000", 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732266+0000", 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6039e-05 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732282+0000", 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8729999999999999e-05 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732300+0000", 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2039999999999997e-06 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732309+0000", 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5497000000000001e-05 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732364+0000", 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6937999999999999e-05 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732391+0000", 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697034 2024-02-12T16:48:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.744088+0000", 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0559000000000002e-05 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.744129+0000", 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011860735000000001 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.26977.0:274563 9.10 e999/989 9:09ee7060:::benchmark_data_smithi169_124168_object123879:head v 999'16363, mlcod=999'16360)", 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:46:27.732322+0000", 2024-02-12T16:48:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 103.018602957, 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011825904, 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732322+0000", 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732322+0000", 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732321+0000", 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7820000000000001e-06 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732326+0000", 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.302e-06 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732332+0000", 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0880000000000002e-06 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.732337+0000", 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024969150000000002 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.734834+0000", 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3230999999999998e-05 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.734857+0000", 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092740410000000002 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.744131+0000", 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6200000000000001e-05 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:46:27.744148+0000", 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011825904 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "replica_scrub(pg: 1.c,from:0'0,to:0'0,epoch:1042/1037,start:1:30000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:47:47.261585+0000", 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.489339580999999, 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034657299999999998, 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261585+0000", 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261585+0000", 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261583+0000", 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4246999999999999e-05 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261597+0000", 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3125e-05 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261620+0000", 2024-02-12T16:48:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8516999999999999e-05 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261639+0000", 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021544500000000001 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261854+0000", 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8397e-05 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261872+0000", 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9181999999999997e-05 2024-02-12T16:48:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:47.261932+0000", 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034657299999999998 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "MOSDScrubReserve(1.3 REQUEST e1043)", 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:47:56.191489+0000", 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.559435864999999, 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032361299999999999, 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191489+0000", 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191489+0000", 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999948 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191484+0000", 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1012000000000001e-05 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191505+0000", 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4236000000000002e-05 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191539+0000", 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.411e-05 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191553+0000", 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015862999999999999 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191712+0000", 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0618999999999999e-05 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191752+0000", 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0105000000000001e-05 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.191812+0000", 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032361299999999999 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "replica_scrub(pg: 1.3,from:0'0,to:0'0,epoch:1043/1037,start:1:c0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:47:56.274062+0000", 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.476862402, 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029981300000000001, 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274062+0000", 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274062+0000", 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274059+0000", 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6730000000000001e-05 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274075+0000", 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3955999999999998e-05 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274109+0000", 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4236000000000001e-05 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274124+0000", 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016348 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274287+0000", 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7275999999999999e-05 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274304+0000", 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7592e-05 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.274362+0000", 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029981300000000001 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "MOSDScrubReserve(1.3 RELEASE e1043)", 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:47:56.341360+0000", 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.409564694, 2024-02-12T16:48:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000227488, 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341360+0000", 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341360+0000", 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:48:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341358+0000", 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1429999999999998e-06 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341367+0000", 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3636e-05 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341380+0000", 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2283000000000001e-05 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341393+0000", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000160082 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341553+0000", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4579999999999999e-05 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341567+0000", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0143999999999999e-05 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:47:56.341587+0000", 2024-02-12T16:48:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000227488 2024-02-12T16:48:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:48:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:48:10.829 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:48:10.829 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:10 smithi169 bash[19604]: audit 2024-02-12T16:48:10.303349+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? v1:172.21.15.121:0/2592694688' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:48:10.830 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:10 smithi123 bash[19119]: audit 2024-02-12T16:48:10.303349+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? v1:172.21.15.121:0/2592694688' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:48:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:10 smithi121 bash[18489]: audit 2024-02-12T16:48:10.303349+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? v1:172.21.15.121:0/2592694688' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-12T16:48:11.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 1 1 2024-02-12T16:48:11.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-02-12T16:48:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:11 smithi169 bash[19604]: cluster 2024-02-12T16:48:10.360684+0000 mgr.y (mgr.14146) 1919 : cluster [DBG] pgmap v2831: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:11 smithi169 bash[19604]: audit 2024-02-12T16:48:10.428693+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? v1:172.21.15.121:0/2592694688' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:48:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:11 smithi169 bash[19604]: cluster 2024-02-12T16:48:10.428775+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e1046: 12 total, 12 up, 12 in 2024-02-12T16:48:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:11 smithi123 bash[19119]: cluster 2024-02-12T16:48:10.360684+0000 mgr.y (mgr.14146) 1919 : cluster [DBG] pgmap v2831: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:11 smithi123 bash[19119]: audit 2024-02-12T16:48:10.428693+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? v1:172.21.15.121:0/2592694688' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:48:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:11 smithi123 bash[19119]: cluster 2024-02-12T16:48:10.428775+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e1046: 12 total, 12 up, 12 in 2024-02-12T16:48:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:11 smithi121 bash[18489]: cluster 2024-02-12T16:48:10.360684+0000 mgr.y (mgr.14146) 1919 : cluster [DBG] pgmap v2831: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:11 smithi121 bash[18489]: audit 2024-02-12T16:48:10.428693+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? v1:172.21.15.121:0/2592694688' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-12T16:48:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:11 smithi121 bash[18489]: cluster 2024-02-12T16:48:10.428775+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e1046: 12 total, 12 up, 12 in 2024-02-12T16:48:13.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:13 smithi169 bash[19604]: cluster 2024-02-12T16:48:12.361361+0000 mgr.y (mgr.14146) 1920 : cluster [DBG] pgmap v2833: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:13 smithi123 bash[19119]: cluster 2024-02-12T16:48:12.361361+0000 mgr.y (mgr.14146) 1920 : cluster [DBG] pgmap v2833: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:13 smithi121 bash[18489]: cluster 2024-02-12T16:48:12.361361+0000 mgr.y (mgr.14146) 1920 : cluster [DBG] pgmap v2833: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:15 smithi169 bash[19604]: cluster 2024-02-12T16:48:14.362049+0000 mgr.y (mgr.14146) 1921 : cluster [DBG] pgmap v2834: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:15 smithi123 bash[19119]: cluster 2024-02-12T16:48:14.362049+0000 mgr.y (mgr.14146) 1921 : cluster [DBG] pgmap v2834: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:15.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:15 smithi121 bash[18489]: cluster 2024-02-12T16:48:14.362049+0000 mgr.y (mgr.14146) 1921 : cluster [DBG] pgmap v2834: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:15.976 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:48:16.288 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:48:17.291 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:17 smithi121 bash[18489]: cluster 2024-02-12T16:48:16.362647+0000 mgr.y (mgr.14146) 1922 : cluster [DBG] pgmap v2835: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:17 smithi123 bash[19119]: cluster 2024-02-12T16:48:16.362647+0000 mgr.y (mgr.14146) 1922 : cluster [DBG] pgmap v2835: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:17 smithi169 bash[19604]: cluster 2024-02-12T16:48:16.362647+0000 mgr.y (mgr.14146) 1922 : cluster [DBG] pgmap v2835: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: cluster 2024-02-12T16:48:18.363354+0000 mgr.y (mgr.14146) 1923 : cluster [DBG] pgmap v2836: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.160940+0000 mon.a (mon.0) 3421 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07983575203404575]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.161576+0000 mon.a (mon.0) 3422 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09402193123599238]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.162095+0000 mon.a (mon.0) 3423 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08758636521016291]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.162611+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09686735864090346]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.163157+0000 mon.a (mon.0) 3425 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09314250564892909]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.163654+0000 mon.a (mon.0) 3426 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08497513091379234]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.164137+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08448330293764608]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.164639+0000 mon.a (mon.0) 3428 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08346231064023946]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.165133+0000 mon.a (mon.0) 3429 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0898416407371798]}]: dispatch 2024-02-12T16:48:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.165614+0000 mon.a (mon.0) 3430 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.0849533681013228]}]: dispatch 2024-02-12T16:48:19.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.166108+0000 mon.a (mon.0) 3431 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08309658205609381]}]: dispatch 2024-02-12T16:48:19.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.166587+0000 mon.a (mon.0) 3432 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.085280138562442]}]: dispatch 2024-02-12T16:48:19.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:19 smithi169 bash[19604]: audit 2024-02-12T16:48:19.167067+0000 mon.a (mon.0) 3433 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"2\": \"53811\", \"3\": \"36655\", \"9\": \"60036\"}"}]: dispatch 2024-02-12T16:48:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: cluster 2024-02-12T16:48:18.363354+0000 mgr.y (mgr.14146) 1923 : cluster [DBG] pgmap v2836: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.160940+0000 mon.a (mon.0) 3421 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07983575203404575]}]: dispatch 2024-02-12T16:48:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.161576+0000 mon.a (mon.0) 3422 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09402193123599238]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.162095+0000 mon.a (mon.0) 3423 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08758636521016291]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.162611+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09686735864090346]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.163157+0000 mon.a (mon.0) 3425 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09314250564892909]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.163654+0000 mon.a (mon.0) 3426 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08497513091379234]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.164137+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08448330293764608]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.164639+0000 mon.a (mon.0) 3428 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08346231064023946]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.165133+0000 mon.a (mon.0) 3429 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0898416407371798]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.165614+0000 mon.a (mon.0) 3430 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.0849533681013228]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.166108+0000 mon.a (mon.0) 3431 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08309658205609381]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.166587+0000 mon.a (mon.0) 3432 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.085280138562442]}]: dispatch 2024-02-12T16:48:19.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:19 smithi123 bash[19119]: audit 2024-02-12T16:48:19.167067+0000 mon.a (mon.0) 3433 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"2\": \"53811\", \"3\": \"36655\", \"9\": \"60036\"}"}]: dispatch 2024-02-12T16:48:19.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: cluster 2024-02-12T16:48:18.363354+0000 mgr.y (mgr.14146) 1923 : cluster [DBG] pgmap v2836: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:19.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.160940+0000 mon.a (mon.0) 3421 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07983575203404575]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.161576+0000 mon.a (mon.0) 3422 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09402193123599238]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.162095+0000 mon.a (mon.0) 3423 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08758636521016291]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.162611+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09686735864090346]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.163157+0000 mon.a (mon.0) 3425 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09314250564892909]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.163654+0000 mon.a (mon.0) 3426 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08497513091379234]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.164137+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08448330293764608]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.164639+0000 mon.a (mon.0) 3428 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08346231064023946]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.165133+0000 mon.a (mon.0) 3429 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0898416407371798]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.165614+0000 mon.a (mon.0) 3430 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.0849533681013228]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.166108+0000 mon.a (mon.0) 3431 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08309658205609381]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.166587+0000 mon.a (mon.0) 3432 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.085280138562442]}]: dispatch 2024-02-12T16:48:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:19 smithi121 bash[18489]: audit 2024-02-12T16:48:19.167067+0000 mon.a (mon.0) 3433 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"2\": \"53811\", \"3\": \"36655\", \"9\": \"60036\"}"}]: dispatch 2024-02-12T16:48:20.022 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:48:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:48:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-02-12T16:48:20.022 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:48:20.439 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.1 to 1 (10000) 2024-02-12T16:48:20.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.424078+0000 mon.a (mon.0) 3434 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07983575203404575]}]': finished 2024-02-12T16:48:20.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.424245+0000 mon.a (mon.0) 3435 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09402193123599238]}]': finished 2024-02-12T16:48:20.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.424451+0000 mon.a (mon.0) 3436 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08758636521016291]}]': finished 2024-02-12T16:48:20.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.424622+0000 mon.a (mon.0) 3437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09686735864090346]}]': finished 2024-02-12T16:48:20.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.424861+0000 mon.a (mon.0) 3438 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09314250564892909]}]': finished 2024-02-12T16:48:20.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.425054+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08497513091379234]}]': finished 2024-02-12T16:48:20.713 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.425249+0000 mon.a (mon.0) 3440 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08448330293764608]}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.425386+0000 mon.a (mon.0) 3441 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08346231064023946]}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.425534+0000 mon.a (mon.0) 3442 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0898416407371798]}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.425682+0000 mon.a (mon.0) 3443 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.0849533681013228]}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.425839+0000 mon.a (mon.0) 3444 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08309658205609381]}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.425985+0000 mon.a (mon.0) 3445 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.085280138562442]}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.426134+0000 mon.a (mon.0) 3446 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"2\": \"53811\", \"3\": \"36655\", \"9\": \"60036\"}"}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: cluster 2024-02-12T16:48:19.426265+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e1047: 12 total, 12 up, 12 in 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:19.427266+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? v1:172.21.15.121:0/3548574318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: audit 2024-02-12T16:48:20.427098+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? v1:172.21.15.121:0/3548574318' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:48:20.714 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:20 smithi169 bash[19604]: cluster 2024-02-12T16:48:20.427316+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e1048: 12 total, 12 up, 12 in 2024-02-12T16:48:20.715 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-02-12T16:48:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.424078+0000 mon.a (mon.0) 3434 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07983575203404575]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.424245+0000 mon.a (mon.0) 3435 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09402193123599238]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.424451+0000 mon.a (mon.0) 3436 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08758636521016291]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.424622+0000 mon.a (mon.0) 3437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09686735864090346]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.424861+0000 mon.a (mon.0) 3438 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09314250564892909]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.425054+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08497513091379234]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.425249+0000 mon.a (mon.0) 3440 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08448330293764608]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.425386+0000 mon.a (mon.0) 3441 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08346231064023946]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.425534+0000 mon.a (mon.0) 3442 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0898416407371798]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.425682+0000 mon.a (mon.0) 3443 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.0849533681013228]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.425839+0000 mon.a (mon.0) 3444 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08309658205609381]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.425985+0000 mon.a (mon.0) 3445 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.085280138562442]}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.426134+0000 mon.a (mon.0) 3446 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"2\": \"53811\", \"3\": \"36655\", \"9\": \"60036\"}"}]': finished 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: cluster 2024-02-12T16:48:19.426265+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e1047: 12 total, 12 up, 12 in 2024-02-12T16:48:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:19.427266+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? v1:172.21.15.121:0/3548574318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:48:20.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: audit 2024-02-12T16:48:20.427098+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? v1:172.21.15.121:0/3548574318' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:48:20.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:20 smithi123 bash[19119]: cluster 2024-02-12T16:48:20.427316+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e1048: 12 total, 12 up, 12 in 2024-02-12T16:48:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.424078+0000 mon.a (mon.0) 3434 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07983575203404575]}]': finished 2024-02-12T16:48:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.424245+0000 mon.a (mon.0) 3435 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09402193123599238]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.424451+0000 mon.a (mon.0) 3436 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08758636521016291]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.424622+0000 mon.a (mon.0) 3437 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.09686735864090346]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.424861+0000 mon.a (mon.0) 3438 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09314250564892909]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.425054+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08497513091379234]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.425249+0000 mon.a (mon.0) 3440 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08448330293764608]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.425386+0000 mon.a (mon.0) 3441 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08346231064023946]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.425534+0000 mon.a (mon.0) 3442 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.0898416407371798]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.425682+0000 mon.a (mon.0) 3443 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.0849533681013228]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.425839+0000 mon.a (mon.0) 3444 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08309658205609381]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.425985+0000 mon.a (mon.0) 3445 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.085280138562442]}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.426134+0000 mon.a (mon.0) 3446 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"2\": \"53811\", \"3\": \"36655\", \"9\": \"60036\"}"}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: cluster 2024-02-12T16:48:19.426265+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e1047: 12 total, 12 up, 12 in 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:19.427266+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? v1:172.21.15.121:0/3548574318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: audit 2024-02-12T16:48:20.427098+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? v1:172.21.15.121:0/3548574318' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:48:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:20 smithi121 bash[18489]: cluster 2024-02-12T16:48:20.427316+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e1048: 12 total, 12 up, 12 in 2024-02-12T16:48:21.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-12T16:48:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:21 smithi121 bash[18489]: cluster 2024-02-12T16:48:20.364057+0000 mgr.y (mgr.14146) 1924 : cluster [DBG] pgmap v2838: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:21 smithi121 bash[18489]: cluster 2024-02-12T16:48:20.537572+0000 osd.10 (osd.10) 254 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:21.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:21 smithi121 bash[18489]: cluster 2024-02-12T16:48:20.537665+0000 osd.10 (osd.10) 255 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:21 smithi169 bash[19604]: cluster 2024-02-12T16:48:20.364057+0000 mgr.y (mgr.14146) 1924 : cluster [DBG] pgmap v2838: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:21 smithi169 bash[19604]: cluster 2024-02-12T16:48:20.537572+0000 osd.10 (osd.10) 254 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:21 smithi169 bash[19604]: cluster 2024-02-12T16:48:20.537665+0000 osd.10 (osd.10) 255 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:21 smithi123 bash[19119]: cluster 2024-02-12T16:48:20.364057+0000 mgr.y (mgr.14146) 1924 : cluster [DBG] pgmap v2838: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:21 smithi123 bash[19119]: cluster 2024-02-12T16:48:20.537572+0000 osd.10 (osd.10) 254 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:21 smithi123 bash[19119]: cluster 2024-02-12T16:48:20.537665+0000 osd.10 (osd.10) 255 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:22 smithi169 bash[19604]: cluster 2024-02-12T16:48:21.448127+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e1049: 12 total, 12 up, 12 in 2024-02-12T16:48:22.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:22 smithi123 bash[19119]: cluster 2024-02-12T16:48:21.448127+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e1049: 12 total, 12 up, 12 in 2024-02-12T16:48:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:22 smithi121 bash[18489]: cluster 2024-02-12T16:48:21.448127+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e1049: 12 total, 12 up, 12 in 2024-02-12T16:48:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:23 smithi169 bash[19604]: cluster 2024-02-12T16:48:22.364839+0000 mgr.y (mgr.14146) 1925 : cluster [DBG] pgmap v2841: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:23 smithi123 bash[19119]: cluster 2024-02-12T16:48:22.364839+0000 mgr.y (mgr.14146) 1925 : cluster [DBG] pgmap v2841: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:23 smithi121 bash[18489]: cluster 2024-02-12T16:48:22.364839+0000 mgr.y (mgr.14146) 1925 : cluster [DBG] pgmap v2841: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:25.496 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:48:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:25 smithi169 bash[19604]: cluster 2024-02-12T16:48:24.365563+0000 mgr.y (mgr.14146) 1926 : cluster [DBG] pgmap v2842: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:25 smithi123 bash[19119]: cluster 2024-02-12T16:48:24.365563+0000 mgr.y (mgr.14146) 1926 : cluster [DBG] pgmap v2842: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:25 smithi121 bash[18489]: cluster 2024-02-12T16:48:24.365563+0000 mgr.y (mgr.14146) 1926 : cluster [DBG] pgmap v2842: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:26.056 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:48:27.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:27 smithi121 bash[18489]: cluster 2024-02-12T16:48:26.084316+0000 mon.a (mon.0) 3452 : cluster [DBG] osdmap e1050: 12 total, 12 up, 12 in 2024-02-12T16:48:27.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:27 smithi121 bash[18489]: cluster 2024-02-12T16:48:26.366132+0000 mgr.y (mgr.14146) 1927 : cluster [DBG] pgmap v2844: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:27 smithi123 bash[19119]: cluster 2024-02-12T16:48:26.084316+0000 mon.a (mon.0) 3452 : cluster [DBG] osdmap e1050: 12 total, 12 up, 12 in 2024-02-12T16:48:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:27 smithi123 bash[19119]: cluster 2024-02-12T16:48:26.366132+0000 mgr.y (mgr.14146) 1927 : cluster [DBG] pgmap v2844: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:27 smithi169 bash[19604]: cluster 2024-02-12T16:48:26.084316+0000 mon.a (mon.0) 3452 : cluster [DBG] osdmap e1050: 12 total, 12 up, 12 in 2024-02-12T16:48:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:27 smithi169 bash[19604]: cluster 2024-02-12T16:48:26.366132+0000 mgr.y (mgr.14146) 1927 : cluster [DBG] pgmap v2844: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:48:28.543 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:48:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:28 smithi169 bash[19604]: cluster 2024-02-12T16:48:27.528102+0000 osd.11 (osd.11) 237 : cluster [DBG] 1.b scrub starts 2024-02-12T16:48:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:28 smithi169 bash[19604]: cluster 2024-02-12T16:48:27.690541+0000 osd.11 (osd.11) 238 : cluster [DBG] 1.b scrub ok 2024-02-12T16:48:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:28 smithi123 bash[19119]: cluster 2024-02-12T16:48:27.528102+0000 osd.11 (osd.11) 237 : cluster [DBG] 1.b scrub starts 2024-02-12T16:48:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:28 smithi123 bash[19119]: cluster 2024-02-12T16:48:27.690541+0000 osd.11 (osd.11) 238 : cluster [DBG] 1.b scrub ok 2024-02-12T16:48:28.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:28 smithi121 bash[18489]: cluster 2024-02-12T16:48:27.528102+0000 osd.11 (osd.11) 237 : cluster [DBG] 1.b scrub starts 2024-02-12T16:48:28.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:28 smithi121 bash[18489]: cluster 2024-02-12T16:48:27.690541+0000 osd.11 (osd.11) 238 : cluster [DBG] 1.b scrub ok 2024-02-12T16:48:29.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-02-12T16:48:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:29 smithi169 bash[19604]: cluster 2024-02-12T16:48:28.366812+0000 mgr.y (mgr.14146) 1928 : cluster [DBG] pgmap v2845: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:29 smithi123 bash[19119]: cluster 2024-02-12T16:48:28.366812+0000 mgr.y (mgr.14146) 1928 : cluster [DBG] pgmap v2845: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:29 smithi121 bash[18489]: cluster 2024-02-12T16:48:28.366812+0000 mgr.y (mgr.14146) 1928 : cluster [DBG] pgmap v2845: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:30.331 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-12T16:48:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:30 smithi123 bash[19119]: cluster 2024-02-12T16:48:29.440065+0000 osd.10 (osd.10) 256 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:48:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:30 smithi123 bash[19119]: cluster 2024-02-12T16:48:29.564667+0000 osd.10 (osd.10) 257 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:48:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:30 smithi123 bash[19119]: audit 2024-02-12T16:48:29.818936+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? v1:172.21.15.121:0/3965685981' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:48:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:30 smithi121 bash[18489]: cluster 2024-02-12T16:48:29.440065+0000 osd.10 (osd.10) 256 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:48:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:30 smithi121 bash[18489]: cluster 2024-02-12T16:48:29.564667+0000 osd.10 (osd.10) 257 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:48:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:30 smithi121 bash[18489]: audit 2024-02-12T16:48:29.818936+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? v1:172.21.15.121:0/3965685981' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:48:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:30 smithi169 bash[19604]: cluster 2024-02-12T16:48:29.440065+0000 osd.10 (osd.10) 256 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:48:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:30 smithi169 bash[19604]: cluster 2024-02-12T16:48:29.564667+0000 osd.10 (osd.10) 257 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:48:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:30 smithi169 bash[19604]: audit 2024-02-12T16:48:29.818936+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? v1:172.21.15.121:0/3965685981' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-12T16:48:31.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 2 1 2024-02-12T16:48:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:31 smithi121 bash[18489]: audit 2024-02-12T16:48:30.319065+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? v1:172.21.15.121:0/3965685981' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:48:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:31 smithi121 bash[18489]: cluster 2024-02-12T16:48:30.319235+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e1051: 12 total, 12 up, 12 in 2024-02-12T16:48:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:31 smithi121 bash[18489]: cluster 2024-02-12T16:48:30.367486+0000 mgr.y (mgr.14146) 1929 : cluster [DBG] pgmap v2847: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:31 smithi121 bash[18489]: cluster 2024-02-12T16:48:31.084546+0000 osd.11 (osd.11) 239 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:31 smithi121 bash[18489]: cluster 2024-02-12T16:48:31.084632+0000 osd.11 (osd.11) 240 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:31.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:31 smithi121 bash[18489]: cluster 2024-02-12T16:48:31.085923+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e1052: 12 total, 12 up, 12 in 2024-02-12T16:48:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:31 smithi123 bash[19119]: audit 2024-02-12T16:48:30.319065+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? v1:172.21.15.121:0/3965685981' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:48:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:31 smithi123 bash[19119]: cluster 2024-02-12T16:48:30.319235+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e1051: 12 total, 12 up, 12 in 2024-02-12T16:48:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:31 smithi123 bash[19119]: cluster 2024-02-12T16:48:30.367486+0000 mgr.y (mgr.14146) 1929 : cluster [DBG] pgmap v2847: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:31 smithi123 bash[19119]: cluster 2024-02-12T16:48:31.084546+0000 osd.11 (osd.11) 239 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:31 smithi123 bash[19119]: cluster 2024-02-12T16:48:31.084632+0000 osd.11 (osd.11) 240 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:31 smithi123 bash[19119]: cluster 2024-02-12T16:48:31.085923+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e1052: 12 total, 12 up, 12 in 2024-02-12T16:48:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:31 smithi169 bash[19604]: audit 2024-02-12T16:48:30.319065+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? v1:172.21.15.121:0/3965685981' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-12T16:48:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:31 smithi169 bash[19604]: cluster 2024-02-12T16:48:30.319235+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e1051: 12 total, 12 up, 12 in 2024-02-12T16:48:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:31 smithi169 bash[19604]: cluster 2024-02-12T16:48:30.367486+0000 mgr.y (mgr.14146) 1929 : cluster [DBG] pgmap v2847: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:31 smithi169 bash[19604]: cluster 2024-02-12T16:48:31.084546+0000 osd.11 (osd.11) 239 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:31.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:31 smithi169 bash[19604]: cluster 2024-02-12T16:48:31.084632+0000 osd.11 (osd.11) 240 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:31.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:31 smithi169 bash[19604]: cluster 2024-02-12T16:48:31.085923+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e1052: 12 total, 12 up, 12 in 2024-02-12T16:48:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:32 smithi123 bash[19119]: cluster 2024-02-12T16:48:31.354275+0000 osd.1 (osd.1) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:32 smithi123 bash[19119]: cluster 2024-02-12T16:48:31.354336+0000 osd.1 (osd.1) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:32.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:32 smithi123 bash[19119]: cluster 2024-02-12T16:48:32.088004+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e1053: 12 total, 12 up, 12 in 2024-02-12T16:48:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:32 smithi121 bash[18489]: cluster 2024-02-12T16:48:31.354275+0000 osd.1 (osd.1) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:32 smithi121 bash[18489]: cluster 2024-02-12T16:48:31.354336+0000 osd.1 (osd.1) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:32.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:32 smithi121 bash[18489]: cluster 2024-02-12T16:48:32.088004+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e1053: 12 total, 12 up, 12 in 2024-02-12T16:48:32.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:32 smithi169 bash[19604]: cluster 2024-02-12T16:48:31.354275+0000 osd.1 (osd.1) 188 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:32.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:32 smithi169 bash[19604]: cluster 2024-02-12T16:48:31.354336+0000 osd.1 (osd.1) 189 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:32.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:32 smithi169 bash[19604]: cluster 2024-02-12T16:48:32.088004+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e1053: 12 total, 12 up, 12 in 2024-02-12T16:48:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:33 smithi123 bash[19119]: cluster 2024-02-12T16:48:32.368074+0000 mgr.y (mgr.14146) 1930 : cluster [DBG] pgmap v2850: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:33 smithi123 bash[19119]: cluster 2024-02-12T16:48:32.497241+0000 osd.11 (osd.11) 241 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:48:33.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:33 smithi123 bash[19119]: cluster 2024-02-12T16:48:32.521050+0000 osd.11 (osd.11) 242 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:48:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:33 smithi121 bash[18489]: cluster 2024-02-12T16:48:32.368074+0000 mgr.y (mgr.14146) 1930 : cluster [DBG] pgmap v2850: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:33 smithi121 bash[18489]: cluster 2024-02-12T16:48:32.497241+0000 osd.11 (osd.11) 241 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:48:33.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:33 smithi121 bash[18489]: cluster 2024-02-12T16:48:32.521050+0000 osd.11 (osd.11) 242 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:48:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:33 smithi169 bash[19604]: cluster 2024-02-12T16:48:32.368074+0000 mgr.y (mgr.14146) 1930 : cluster [DBG] pgmap v2850: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:33 smithi169 bash[19604]: cluster 2024-02-12T16:48:32.497241+0000 osd.11 (osd.11) 241 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:48:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:33 smithi169 bash[19604]: cluster 2024-02-12T16:48:32.521050+0000 osd.11 (osd.11) 242 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:48:33.998 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:48:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:34 smithi123 bash[19119]: cluster 2024-02-12T16:48:32.983108+0000 osd.4 (osd.4) 177 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:48:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:34 smithi123 bash[19119]: cluster 2024-02-12T16:48:33.063100+0000 osd.4 (osd.4) 178 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:48:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:34 smithi123 bash[19119]: cluster 2024-02-12T16:48:33.401020+0000 osd.7 (osd.7) 29 : cluster [DBG] 1.d scrub starts 2024-02-12T16:48:34.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:34 smithi123 bash[19119]: cluster 2024-02-12T16:48:33.516127+0000 osd.7 (osd.7) 30 : cluster [DBG] 1.d scrub ok 2024-02-12T16:48:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:34 smithi121 bash[18489]: cluster 2024-02-12T16:48:32.983108+0000 osd.4 (osd.4) 177 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:48:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:34 smithi121 bash[18489]: cluster 2024-02-12T16:48:33.063100+0000 osd.4 (osd.4) 178 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:48:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:34 smithi121 bash[18489]: cluster 2024-02-12T16:48:33.401020+0000 osd.7 (osd.7) 29 : cluster [DBG] 1.d scrub starts 2024-02-12T16:48:34.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:34 smithi121 bash[18489]: cluster 2024-02-12T16:48:33.516127+0000 osd.7 (osd.7) 30 : cluster [DBG] 1.d scrub ok 2024-02-12T16:48:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:34 smithi169 bash[19604]: cluster 2024-02-12T16:48:32.983108+0000 osd.4 (osd.4) 177 : cluster [DBG] 1.10 deep-scrub starts 2024-02-12T16:48:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:34 smithi169 bash[19604]: cluster 2024-02-12T16:48:33.063100+0000 osd.4 (osd.4) 178 : cluster [DBG] 1.10 deep-scrub ok 2024-02-12T16:48:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:34 smithi169 bash[19604]: cluster 2024-02-12T16:48:33.401020+0000 osd.7 (osd.7) 29 : cluster [DBG] 1.d scrub starts 2024-02-12T16:48:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:34 smithi169 bash[19604]: cluster 2024-02-12T16:48:33.516127+0000 osd.7 (osd.7) 30 : cluster [DBG] 1.d scrub ok 2024-02-12T16:48:35.388 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:35 smithi169 bash[19604]: cluster 2024-02-12T16:48:34.295964+0000 osd.9 (osd.9) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:35.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:35 smithi169 bash[19604]: cluster 2024-02-12T16:48:34.296022+0000 osd.9 (osd.9) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:35.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:35 smithi169 bash[19604]: cluster 2024-02-12T16:48:34.350152+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e1054: 12 total, 12 up, 12 in 2024-02-12T16:48:35.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:35 smithi169 bash[19604]: cluster 2024-02-12T16:48:34.361836+0000 osd.10 (osd.10) 258 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:48:35.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:35 smithi169 bash[19604]: cluster 2024-02-12T16:48:34.368678+0000 mgr.y (mgr.14146) 1931 : cluster [DBG] pgmap v2852: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:35.389 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:35 smithi169 bash[19604]: cluster 2024-02-12T16:48:34.444981+0000 osd.10 (osd.10) 259 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:48:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:35 smithi123 bash[19119]: cluster 2024-02-12T16:48:34.295964+0000 osd.9 (osd.9) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:35 smithi123 bash[19119]: cluster 2024-02-12T16:48:34.296022+0000 osd.9 (osd.9) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:35 smithi123 bash[19119]: cluster 2024-02-12T16:48:34.350152+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e1054: 12 total, 12 up, 12 in 2024-02-12T16:48:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:35 smithi123 bash[19119]: cluster 2024-02-12T16:48:34.361836+0000 osd.10 (osd.10) 258 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:48:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:35 smithi123 bash[19119]: cluster 2024-02-12T16:48:34.368678+0000 mgr.y (mgr.14146) 1931 : cluster [DBG] pgmap v2852: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:35.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:35 smithi123 bash[19119]: cluster 2024-02-12T16:48:34.444981+0000 osd.10 (osd.10) 259 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:48:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:35 smithi121 bash[18489]: cluster 2024-02-12T16:48:34.295964+0000 osd.9 (osd.9) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:35 smithi121 bash[18489]: cluster 2024-02-12T16:48:34.296022+0000 osd.9 (osd.9) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:35 smithi121 bash[18489]: cluster 2024-02-12T16:48:34.350152+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e1054: 12 total, 12 up, 12 in 2024-02-12T16:48:35.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:35 smithi121 bash[18489]: cluster 2024-02-12T16:48:34.361836+0000 osd.10 (osd.10) 258 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:48:35.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:35 smithi121 bash[18489]: cluster 2024-02-12T16:48:34.368678+0000 mgr.y (mgr.14146) 1931 : cluster [DBG] pgmap v2852: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:35.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:35 smithi121 bash[18489]: cluster 2024-02-12T16:48:34.444981+0000 osd.10 (osd.10) 259 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:48:35.904 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:48:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:36 smithi123 bash[19119]: cluster 2024-02-12T16:48:35.394244+0000 osd.10 (osd.10) 260 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:48:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:36 smithi123 bash[19119]: cluster 2024-02-12T16:48:35.492741+0000 osd.10 (osd.10) 261 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:48:36.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:36 smithi123 bash[19119]: cluster 2024-02-12T16:48:36.088623+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e1055: 12 total, 12 up, 12 in 2024-02-12T16:48:36.733 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:36 smithi121 bash[18489]: cluster 2024-02-12T16:48:35.394244+0000 osd.10 (osd.10) 260 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:48:36.733 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:36 smithi121 bash[18489]: cluster 2024-02-12T16:48:35.492741+0000 osd.10 (osd.10) 261 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:48:36.733 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:36 smithi121 bash[18489]: cluster 2024-02-12T16:48:36.088623+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e1055: 12 total, 12 up, 12 in 2024-02-12T16:48:36.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:36 smithi169 bash[19604]: cluster 2024-02-12T16:48:35.394244+0000 osd.10 (osd.10) 260 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:48:36.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:36 smithi169 bash[19604]: cluster 2024-02-12T16:48:35.492741+0000 osd.10 (osd.10) 261 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:48:36.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:36 smithi169 bash[19604]: cluster 2024-02-12T16:48:36.088623+0000 mon.a (mon.0) 3459 : cluster [DBG] osdmap e1055: 12 total, 12 up, 12 in 2024-02-12T16:48:37.318 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:48:37.318 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:48:37.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:48:37.318 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59820 8.a 8:5b47d031:::benchmark_data_smithi169_115677_object59819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.828872+0000", 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 484.48058583599999, 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33076042500000002, 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59820 2024-02-12T16:48:37.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828872+0000", 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828872+0000", 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828869+0000", 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4998999999999999e-05 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828894+0000", 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0559e-05 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828905+0000", 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0240000000000003e-06 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828911+0000", 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3707999999999994e-05 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828975+0000", 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016124100000000001 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.829136+0000", 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017426400000000001 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:48:37.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.829310+0000", 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061200600000000001 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.835430+0000", 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32392584499999999 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.159356+0000", 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015318999999999999 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.159509+0000", 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012287499999999999 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.159632+0000", 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33076042500000002 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59821 8.a 8:5035d20a:::benchmark_data_smithi169_115677_object59820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:48:37.324 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.828927+0000", 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 484.48053088500001, 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.330987068, 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59821 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828927+0000", 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828927+0000", 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828926+0000", 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9612000000000001e-05 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828945+0000", 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.942e-06 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828951+0000", 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.262e-06 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828956+0000", 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023954620000000001 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.831351+0000", 2024-02-12T16:48:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013886599999999999 2024-02-12T16:48:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.831490+0000", 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015294700000000001 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.831643+0000", 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003790837 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.835434+0000", 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32432144200000002 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.159755+0000", 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4017999999999993e-05 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.159849+0000", 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4604999999999996e-05 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.159914+0000", 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.330987068 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59826 8.a 8:50ae57c1:::benchmark_data_smithi169_115677_object59825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.830530+0000", 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 484.47892734599998, 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32976924200000002, 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59826 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830530+0000", 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830530+0000", 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830529+0000", 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2765999999999998e-05 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830561+0000", 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2029999999999998e-06 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830571+0000", 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.964e-06 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830577+0000", 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029620599999999999 2024-02-12T16:48:37.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.833539+0000", 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115852 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.833654+0000", 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015285 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.833807+0000", 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017532890000000001 2024-02-12T16:48:37.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.835561+0000", 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32445918099999999 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.160020+0000", 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000163881 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.160184+0000", 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011577400000000001 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.160299+0000", 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32976924200000002 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59845 8.a 8:51d2d893:::benchmark_data_smithi169_115677_object59844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.841356+0000", 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 484.46810176399998, 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32050469399999998, 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59845 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841356+0000", 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841356+0000", 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841354+0000", 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1528000000000001e-05 2024-02-12T16:48:37.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841376+0000", 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1583000000000001e-05 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841387+0000", 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0459999999999996e-06 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841393+0000", 2024-02-12T16:48:37.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2901999999999997e-05 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841435+0000", 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000130116 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841566+0000", 2024-02-12T16:48:37.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015812500000000001 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.841724+0000", 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001826236 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.843550+0000", 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31816730500000001 2024-02-12T16:48:37.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.161717+0000", 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5681999999999997e-05 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.161803+0000", 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7538999999999999e-05 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.161860+0000", 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32050469399999998 2024-02-12T16:48:37.342 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67852 8.a 8:5415d222:::benchmark_data_smithi169_115677_object67851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.713848+0000", 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 480.595610018, 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.228732983, 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:48:37.343 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67852 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.713848+0000", 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.713848+0000", 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.713846+0000", 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2339000000000001e-05 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.713868+0000", 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2680999999999999e-05 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.713881+0000", 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5060000000000001e-06 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.713887+0000", 2024-02-12T16:48:37.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7941000000000001e-05 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.713935+0000", 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011934000000000001 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.714055+0000", 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015273700000000001 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:48:37.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.714207+0000", 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030281560000000002 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717236+0000", 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.225107108 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.942343+0000", 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000126048 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.942469+0000", 2024-02-12T16:48:37.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000111731 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.942580+0000", 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.228732983 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67872 8.a 8:5d7fd3e8:::benchmark_data_smithi169_115677_object67871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.723636+0000", 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 480.58582150699999, 2024-02-12T16:48:37.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21909031700000001, 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67872 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723636+0000", 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723636+0000", 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723634+0000", 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8107e-05 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723662+0000", 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5413e-05 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723677+0000", 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2799999999999998e-06 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723684+0000", 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9833999999999998e-05 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723734+0000", 2024-02-12T16:48:37.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107605 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723842+0000", 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000108108 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.723950+0000", 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001892775 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.725843+0000", 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21680148899999999 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.942644+0000", 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7503000000000002e-05 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.942702+0000", 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4609000000000001e-05 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.942726+0000", 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21909031700000001 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:122565 9.3 9:cc37c853:::benchmark_data_smithi169_124168_object122564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:43:39.707120+0000", 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 297.60233775400002, 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18511791599999999, 2024-02-12T16:48:37.354 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 122565 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707120+0000", 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707120+0000", 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707117+0000", 2024-02-12T16:48:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7097000000000002e-05 2024-02-12T16:48:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707144+0000", 2024-02-12T16:48:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7540000000000001e-05 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707161+0000", 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.934e-06 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707168+0000", 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1063999999999997e-05 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707229+0000", 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016063899999999999 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707390+0000", 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017013400000000001 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.707560+0000", 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0084398960000000006 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.716000+0000", 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.176067318 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892067+0000", 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5276000000000001e-05 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892163+0000", 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5148999999999995e-05 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892238+0000", 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18511791599999999 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:122570 9.3 9:c9cc0ff0:::benchmark_data_smithi169_124168_object122569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:43:39.709412+0000", 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 297.600045373, 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18323223499999999, 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.361 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 122570 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.709412+0000", 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.709412+0000", 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.709410+0000", 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5063e-05 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.709435+0000", 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4681999999999999e-05 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.709449+0000", 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.405e-06 2024-02-12T16:48:37.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.709458+0000", 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002150183 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.711608+0000", 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014510400000000001 2024-02-12T16:48:37.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.711753+0000", 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020412699999999999 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.711957+0000", 2024-02-12T16:48:37.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042030870000000003 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.716160+0000", 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17623802199999999 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892398+0000", 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013976 2024-02-12T16:48:37.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892538+0000", 2024-02-12T16:48:37.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106292 2024-02-12T16:48:37.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892644+0000", 2024-02-12T16:48:37.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18323223499999999 2024-02-12T16:48:37.368 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.368 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.369 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.369 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:122585 9.3 9:cc34baa6:::benchmark_data_smithi169_124168_object122584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:48:37.369 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:43:39.727902+0000", 2024-02-12T16:48:37.370 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 297.581555272, 2024-02-12T16:48:37.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16496565499999999, 2024-02-12T16:48:37.370 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.370 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.370 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 122585 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.727902+0000", 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.727902+0000", 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.727899+0000", 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4785e-05 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.727934+0000", 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1639999999999999e-05 2024-02-12T16:48:37.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.727955+0000", 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4599999999999992e-06 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.727965+0000", 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0825000000000001e-05 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.728026+0000", 2024-02-12T16:48:37.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017178199999999999 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.728197+0000", 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019126500000000001 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.728389+0000", 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042118030000000001 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.732600+0000", 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.1601195 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892720+0000", 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8020000000000002e-05 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892798+0000", 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0001999999999998e-05 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:39.892868+0000", 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16496565499999999 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:136555 9.3 9:c2057f54:::benchmark_data_smithi169_124168_object136554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e948)", 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:43:52.483432+0000", 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 284.82602544000002, 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14121768100000001, 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.376 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 136555 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483432+0000", 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483432+0000", 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483430+0000", 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3952999999999998e-05 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483464+0000", 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2298e-05 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483476+0000", 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3300000000000001e-06 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483484+0000", 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3777999999999999e-05 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483537+0000", 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000155456 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483693+0000", 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016882099999999999 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.483862+0000", 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004860718 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.488722+0000", 2024-02-12T16:48:37.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.135822737 2024-02-12T16:48:37.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.624545+0000", 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1278999999999994e-05 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.624606+0000", 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3439000000000002e-05 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:52.624650+0000", 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14121768100000001 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:157670 9.3 9:cea0c24e:::benchmark_data_smithi169_124168_object6986:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:44:14.973442+0000", 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 262.33601527000002, 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.193624302, 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.383 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 157670 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973442+0000", 2024-02-12T16:48:37.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973442+0000", 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973439+0000", 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2157999999999999e-05 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973451+0000", 2024-02-12T16:48:37.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1373999999999999e-05 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973472+0000", 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2740000000000007e-06 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973481+0000", 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7286000000000001e-05 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973528+0000", 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000125952 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973654+0000", 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107044 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.973761+0000", 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046449459999999996 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.978406+0000", 2024-02-12T16:48:37.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18855271700000001 2024-02-12T16:48:37.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.166959+0000", 2024-02-12T16:48:37.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7114999999999996e-05 2024-02-12T16:48:37.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.167036+0000", 2024-02-12T16:48:37.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0763999999999999e-05 2024-02-12T16:48:37.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.167067+0000", 2024-02-12T16:48:37.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.193624302 2024-02-12T16:48:37.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.390 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.391 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:157679 9.3 9:c0b6ff2d:::benchmark_data_smithi169_124168_object6995:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:48:37.391 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:44:14.990269+0000", 2024-02-12T16:48:37.391 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 262.31918873799998, 2024-02-12T16:48:37.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.177136929, 2024-02-12T16:48:37.391 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.392 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.392 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.392 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.392 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.392 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 157679 2024-02-12T16:48:37.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.393 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990269+0000", 2024-02-12T16:48:37.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990269+0000", 2024-02-12T16:48:37.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:48:37.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990266+0000", 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1348e-05 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990277+0000", 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8658999999999999e-05 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990296+0000", 2024-02-12T16:48:37.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0979999999999996e-06 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990303+0000", 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2042999999999999e-05 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990395+0000", 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020136399999999999 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990596+0000", 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025055900000000002 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.990847+0000", 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067948210000000004 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:14.997641+0000", 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.169506829 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.167148+0000", 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014224499999999999 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.167291+0000", 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115192 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.167406+0000", 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.177136929 2024-02-12T16:48:37.398 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:158403 9.3 9:ce7a838e:::benchmark_data_smithi169_124168_object7719:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:44:15.873899+0000", 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 261.43555876400001, 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.303531457, 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.399 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 158403 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.873899+0000", 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.873899+0000", 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.873897+0000", 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6329999999999996e-06 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.873904+0000", 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3695e-05 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.873918+0000", 2024-02-12T16:48:37.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.178e-06 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.873924+0000", 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8457000000000001e-05 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.873963+0000", 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7250000000000007e-05 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.874050+0000", 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5153000000000004e-05 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.874125+0000", 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004056339 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.878181+0000", 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.29914511500000002 2024-02-12T16:48:37.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177326+0000", 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8801000000000002e-05 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177385+0000", 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5009000000000002e-05 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177430+0000", 2024-02-12T16:48:37.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.303531457 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:158424 9.3 9:c1d8f513:::benchmark_data_smithi169_124168_object7740:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:44:15.895499+0000", 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 261.41395888300002, 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28228352000000001, 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 158424 2024-02-12T16:48:37.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.895499+0000", 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.895499+0000", 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.895495+0000", 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2802e-05 2024-02-12T16:48:37.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.895508+0000", 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2733000000000001e-05 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.895531+0000", 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0287e-05 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.895541+0000", 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086309699999999995 2024-02-12T16:48:37.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.896404+0000", 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015081200000000001 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.896555+0000", 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019892400000000001 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.896754+0000", 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056612820000000001 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.902415+0000", 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.27513432700000001 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177550+0000", 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012655699999999999 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177676+0000", 2024-02-12T16:48:37.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000105998 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177782+0000", 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.28228352000000001 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:158439 9.3 9:c398ad1f:::benchmark_data_smithi169_124168_object7755:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:44:15.909574+0000", 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 261.39988367900003, 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26837416400000003, 2024-02-12T16:48:37.410 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 158439 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909574+0000", 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909574+0000", 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:37.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909572+0000", 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.267e-06 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909579+0000", 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2164000000000001e-05 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909591+0000", 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1130000000000003e-06 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909598+0000", 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5538999999999998e-05 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909654+0000", 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012924599999999999 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909783+0000", 2024-02-12T16:48:37.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015082800000000001 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.909934+0000", 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048352380000000004 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.914769+0000", 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26307514599999998 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177844+0000", 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3053999999999998e-05 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177887+0000", 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0903999999999998e-05 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.177948+0000", 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.26837416400000003 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:158470 9.3 9:cea51599:::benchmark_data_smithi169_124168_object7786:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:44:15.956206+0000", 2024-02-12T16:48:37.415 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 261.35325105200002, 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22207233100000001, 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 158470 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956206+0000", 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956206+0000", 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999962 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956203+0000", 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4363e-05 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956217+0000", 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2677e-05 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956240+0000", 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0223e-05 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956250+0000", 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9676000000000003e-05 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956310+0000", 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116459 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956426+0000", 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100987 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.956527+0000", 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00444809 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:15.960975+0000", 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21707849200000001 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.178054+0000", 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012545199999999999 2024-02-12T16:48:37.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.178179+0000", 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9580999999999994e-05 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:44:16.178279+0000", 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22207233100000001 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.420 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:234958 9.3 9:c83cffb6:::benchmark_data_smithi169_124168_object84274:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:45:48.776078+0000", 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 168.53337998800001, 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15306916000000001, 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 234958 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.776078+0000", 2024-02-12T16:48:37.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.776078+0000", 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.776076+0000", 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0460000000000002e-06 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.776082+0000", 2024-02-12T16:48:37.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1739000000000001e-05 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.776094+0000", 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.152e-06 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.776099+0000", 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004171091 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.780270+0000", 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014301200000000001 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.780413+0000", 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013966299999999999 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.780553+0000", 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015207013 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.795760+0000", 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.133254816 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929014+0000", 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0613999999999999e-05 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929075+0000", 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.169e-05 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929147+0000", 2024-02-12T16:48:37.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15306916000000001 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:234969 9.3 9:c1a424ca:::benchmark_data_smithi169_124168_object84285:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:45:48.784371+0000", 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 168.525086268, 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14492028400000001, 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.426 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 234969 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784371+0000", 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784371+0000", 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784369+0000", 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9060000000000002e-06 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784377+0000", 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5936000000000001e-05 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784393+0000", 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6780000000000003e-06 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784399+0000", 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013751799999999999 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.785775+0000", 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012133699999999999 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.785896+0000", 2024-02-12T16:48:37.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5074000000000006e-05 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.785981+0000", 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0097945140000000007 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.795775+0000", 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.133422068 2024-02-12T16:48:37.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929198+0000", 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1647999999999997e-05 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929249+0000", 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2385999999999997e-05 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929292+0000", 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14492028400000001 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.431 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:234970 9.3 9:cc4f204b:::benchmark_data_smithi169_124168_object84286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:45:48.784495+0000", 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 168.52496245699999, 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14493974700000001, 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 234970 2024-02-12T16:48:37.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784495+0000", 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784495+0000", 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784494+0000", 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8369999999999996e-06 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784499+0000", 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7410000000000004e-06 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784509+0000", 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8969999999999999e-06 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.784513+0000", 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057678310000000002 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.790281+0000", 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000125007 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.790406+0000", 2024-02-12T16:48:37.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013817500000000001 2024-02-12T16:48:37.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.790544+0000", 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053978009999999998 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.795942+0000", 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.13339551299999999 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929338+0000", 2024-02-12T16:48:37.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5463000000000001e-05 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929393+0000", 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1607999999999999e-05 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929435+0000", 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.14493974700000001 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:234979 9.3 9:cb4e731c:::benchmark_data_smithi169_124168_object84295:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:45:48.796248+0000", 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 168.51320932799999, 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.133324264, 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 234979 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:48:37.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796248+0000", 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796248+0000", 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796245+0000", 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5950000000000005e-06 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796255+0000", 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2105e-05 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796267+0000", 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7219999999999996e-06 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796273+0000", 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1289999999999999e-05 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796324+0000", 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014906500000000001 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796473+0000", 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015229499999999999 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 3", 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.796625+0000", 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004445526 2024-02-12T16:48:37.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.801071+0000", 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.128408564 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929479+0000", 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7246999999999997e-05 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929527+0000", 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5756e-05 2024-02-12T16:48:37.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:45:48.929572+0000", 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.133324264 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:48:37.444 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:48:37.601 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:37 smithi121 bash[18489]: cluster 2024-02-12T16:48:36.369272+0000 mgr.y (mgr.14146) 1932 : cluster [DBG] pgmap v2854: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:37 smithi123 bash[19119]: cluster 2024-02-12T16:48:36.369272+0000 mgr.y (mgr.14146) 1932 : cluster [DBG] pgmap v2854: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:37.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:37 smithi169 bash[19604]: cluster 2024-02-12T16:48:36.369272+0000 mgr.y (mgr.14146) 1932 : cluster [DBG] pgmap v2854: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:38.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-02-12T16:48:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:39 smithi121 bash[18489]: cluster 2024-02-12T16:48:38.370048+0000 mgr.y (mgr.14146) 1933 : cluster [DBG] pgmap v2855: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:39 smithi169 bash[19604]: cluster 2024-02-12T16:48:38.370048+0000 mgr.y (mgr.14146) 1933 : cluster [DBG] pgmap v2855: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:39 smithi123 bash[19119]: cluster 2024-02-12T16:48:38.370048+0000 mgr.y (mgr.14146) 1933 : cluster [DBG] pgmap v2855: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:40.411 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.2 to 1 (10000) 2024-02-12T16:48:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:40 smithi121 bash[18489]: cluster 2024-02-12T16:48:38.826918+0000 osd.5 (osd.5) 237 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:40 smithi121 bash[18489]: cluster 2024-02-12T16:48:38.827007+0000 osd.5 (osd.5) 238 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:40 smithi121 bash[18489]: cluster 2024-02-12T16:48:39.381895+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e1056: 12 total, 12 up, 12 in 2024-02-12T16:48:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:40 smithi121 bash[18489]: audit 2024-02-12T16:48:39.679990+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.121:0/3417779208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:48:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:40 smithi121 bash[18489]: cluster 2024-02-12T16:48:40.016172+0000 osd.8 (osd.8) 161 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:48:40.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:40 smithi121 bash[18489]: cluster 2024-02-12T16:48:40.088550+0000 osd.8 (osd.8) 162 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:48:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:40 smithi169 bash[19604]: cluster 2024-02-12T16:48:38.826918+0000 osd.5 (osd.5) 237 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:40 smithi169 bash[19604]: cluster 2024-02-12T16:48:38.827007+0000 osd.5 (osd.5) 238 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:40 smithi169 bash[19604]: cluster 2024-02-12T16:48:39.381895+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e1056: 12 total, 12 up, 12 in 2024-02-12T16:48:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:40 smithi169 bash[19604]: audit 2024-02-12T16:48:39.679990+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.121:0/3417779208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:48:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:40 smithi169 bash[19604]: cluster 2024-02-12T16:48:40.016172+0000 osd.8 (osd.8) 161 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:48:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:40 smithi169 bash[19604]: cluster 2024-02-12T16:48:40.088550+0000 osd.8 (osd.8) 162 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:48:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:40 smithi123 bash[19119]: cluster 2024-02-12T16:48:38.826918+0000 osd.5 (osd.5) 237 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:40 smithi123 bash[19119]: cluster 2024-02-12T16:48:38.827007+0000 osd.5 (osd.5) 238 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:40 smithi123 bash[19119]: cluster 2024-02-12T16:48:39.381895+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e1056: 12 total, 12 up, 12 in 2024-02-12T16:48:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:40 smithi123 bash[19119]: audit 2024-02-12T16:48:39.679990+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.121:0/3417779208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:48:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:40 smithi123 bash[19119]: cluster 2024-02-12T16:48:40.016172+0000 osd.8 (osd.8) 161 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:48:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:40 smithi123 bash[19119]: cluster 2024-02-12T16:48:40.088550+0000 osd.8 (osd.8) 162 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:48:41.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-12T16:48:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:39.939936+0000 osd.5 (osd.5) 239 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:48:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:39.969938+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:39.970010+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:39.982463+0000 osd.5 (osd.5) 240 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:40.370679+0000 mgr.y (mgr.14146) 1934 : cluster [DBG] pgmap v2857: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: audit 2024-02-12T16:48:40.401463+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? v1:172.21.15.121:0/3417779208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:40.401521+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e1057: 12 total, 12 up, 12 in 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:40.501713+0000 osd.1 (osd.1) 190 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:40.708112+0000 osd.1 (osd.1) 191 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:40.909025+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:41.009086+0000 osd.8 (osd.8) 163 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:48:41.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:41 smithi121 bash[18489]: cluster 2024-02-12T16:48:41.132023+0000 osd.8 (osd.8) 164 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:48:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:39.939936+0000 osd.5 (osd.5) 239 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:48:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:39.969938+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:39.970010+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:39.982463+0000 osd.5 (osd.5) 240 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:48:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:40.370679+0000 mgr.y (mgr.14146) 1934 : cluster [DBG] pgmap v2857: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:41.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: audit 2024-02-12T16:48:40.401463+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? v1:172.21.15.121:0/3417779208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:48:41.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:40.401521+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e1057: 12 total, 12 up, 12 in 2024-02-12T16:48:41.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:40.501713+0000 osd.1 (osd.1) 190 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:48:41.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:40.708112+0000 osd.1 (osd.1) 191 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:48:41.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:40.909025+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:48:41.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:41.009086+0000 osd.8 (osd.8) 163 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:48:41.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:41 smithi169 bash[19604]: cluster 2024-02-12T16:48:41.132023+0000 osd.8 (osd.8) 164 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:48:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:39.939936+0000 osd.5 (osd.5) 239 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:48:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:39.969938+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:39.970010+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:39.982463+0000 osd.5 (osd.5) 240 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:48:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:40.370679+0000 mgr.y (mgr.14146) 1934 : cluster [DBG] pgmap v2857: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: audit 2024-02-12T16:48:40.401463+0000 mon.a (mon.0) 3462 : audit [INF] from='client.? v1:172.21.15.121:0/3417779208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:48:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:40.401521+0000 mon.a (mon.0) 3463 : cluster [DBG] osdmap e1057: 12 total, 12 up, 12 in 2024-02-12T16:48:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:40.501713+0000 osd.1 (osd.1) 190 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:48:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:40.708112+0000 osd.1 (osd.1) 191 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:48:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:40.909025+0000 osd.6 (osd.6) 198 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:48:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:41.009086+0000 osd.8 (osd.8) 163 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:48:41.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:41 smithi123 bash[19119]: cluster 2024-02-12T16:48:41.132023+0000 osd.8 (osd.8) 164 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:48:42.792 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:48:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:42 smithi169 bash[19604]: cluster 2024-02-12T16:48:41.049239+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:48:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:42 smithi169 bash[19604]: cluster 2024-02-12T16:48:41.411197+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e1058: 12 total, 12 up, 12 in 2024-02-12T16:48:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:42 smithi123 bash[19119]: cluster 2024-02-12T16:48:41.049239+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:48:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:42 smithi123 bash[19119]: cluster 2024-02-12T16:48:41.411197+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e1058: 12 total, 12 up, 12 in 2024-02-12T16:48:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:42 smithi121 bash[18489]: cluster 2024-02-12T16:48:41.049239+0000 osd.6 (osd.6) 199 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:48:42.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:42 smithi121 bash[18489]: cluster 2024-02-12T16:48:41.411197+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e1058: 12 total, 12 up, 12 in 2024-02-12T16:48:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:43 smithi169 bash[19604]: cluster 2024-02-12T16:48:42.371435+0000 mgr.y (mgr.14146) 1935 : cluster [DBG] pgmap v2860: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:43 smithi123 bash[19119]: cluster 2024-02-12T16:48:42.371435+0000 mgr.y (mgr.14146) 1935 : cluster [DBG] pgmap v2860: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:43 smithi121 bash[18489]: cluster 2024-02-12T16:48:42.371435+0000 mgr.y (mgr.14146) 1935 : cluster [DBG] pgmap v2860: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:44 smithi169 bash[19604]: cluster 2024-02-12T16:48:43.322477+0000 osd.7 (osd.7) 31 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:48:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:44 smithi169 bash[19604]: cluster 2024-02-12T16:48:43.421898+0000 osd.7 (osd.7) 32 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:48:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:44 smithi123 bash[19119]: cluster 2024-02-12T16:48:43.322477+0000 osd.7 (osd.7) 31 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:48:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:44 smithi123 bash[19119]: cluster 2024-02-12T16:48:43.421898+0000 osd.7 (osd.7) 32 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:48:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:44 smithi121 bash[18489]: cluster 2024-02-12T16:48:43.322477+0000 osd.7 (osd.7) 31 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:48:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:44 smithi121 bash[18489]: cluster 2024-02-12T16:48:43.421898+0000 osd.7 (osd.7) 32 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:48:45.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:45 smithi169 bash[19604]: cluster 2024-02-12T16:48:44.256365+0000 osd.6 (osd.6) 200 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:45.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:45 smithi169 bash[19604]: cluster 2024-02-12T16:48:44.256455+0000 osd.6 (osd.6) 201 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:45.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:45 smithi169 bash[19604]: cluster 2024-02-12T16:48:44.372119+0000 mgr.y (mgr.14146) 1936 : cluster [DBG] pgmap v2861: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:45.763 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:45 smithi169 bash[19604]: cluster 2024-02-12T16:48:44.433371+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e1059: 12 total, 12 up, 12 in 2024-02-12T16:48:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:45 smithi123 bash[19119]: cluster 2024-02-12T16:48:44.256365+0000 osd.6 (osd.6) 200 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:45 smithi123 bash[19119]: cluster 2024-02-12T16:48:44.256455+0000 osd.6 (osd.6) 201 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:45 smithi123 bash[19119]: cluster 2024-02-12T16:48:44.372119+0000 mgr.y (mgr.14146) 1936 : cluster [DBG] pgmap v2861: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:45 smithi123 bash[19119]: cluster 2024-02-12T16:48:44.433371+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e1059: 12 total, 12 up, 12 in 2024-02-12T16:48:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:45 smithi121 bash[18489]: cluster 2024-02-12T16:48:44.256365+0000 osd.6 (osd.6) 200 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:45 smithi121 bash[18489]: cluster 2024-02-12T16:48:44.256455+0000 osd.6 (osd.6) 201 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:45 smithi121 bash[18489]: cluster 2024-02-12T16:48:44.372119+0000 mgr.y (mgr.14146) 1936 : cluster [DBG] pgmap v2861: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:45 smithi121 bash[18489]: cluster 2024-02-12T16:48:44.433371+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e1059: 12 total, 12 up, 12 in 2024-02-12T16:48:45.967 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:48:46.010 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:48:46.011 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:48:46.011 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-02-12T16:48:46.011 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:48:46.745 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-02-12T16:48:47.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:45.444732+0000 osd.11 (osd.11) 243 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:48:47.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:45.521643+0000 osd.11 (osd.11) 244 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:48:47.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:45.810583+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:47.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:45.810681+0000 osd.2 (osd.2) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:47.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:46.160604+0000 osd.0 (osd.0) 171 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:48:47.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:46.265070+0000 osd.0 (osd.0) 172 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:48:47.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:46.372731+0000 mgr.y (mgr.14146) 1937 : cluster [DBG] pgmap v2863: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:47.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:47 smithi121 bash[18489]: cluster 2024-02-12T16:48:46.441504+0000 mon.a (mon.0) 3466 : cluster [DBG] osdmap e1060: 12 total, 12 up, 12 in 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:45.444732+0000 osd.11 (osd.11) 243 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:45.521643+0000 osd.11 (osd.11) 244 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:45.810583+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:45.810681+0000 osd.2 (osd.2) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:46.160604+0000 osd.0 (osd.0) 171 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:46.265070+0000 osd.0 (osd.0) 172 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:46.372731+0000 mgr.y (mgr.14146) 1937 : cluster [DBG] pgmap v2863: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:47 smithi123 bash[19119]: cluster 2024-02-12T16:48:46.441504+0000 mon.a (mon.0) 3466 : cluster [DBG] osdmap e1060: 12 total, 12 up, 12 in 2024-02-12T16:48:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:45.444732+0000 osd.11 (osd.11) 243 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:48:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:45.521643+0000 osd.11 (osd.11) 244 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:48:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:45.810583+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:45.810681+0000 osd.2 (osd.2) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:46.160604+0000 osd.0 (osd.0) 171 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:48:47.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:46.265070+0000 osd.0 (osd.0) 172 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:48:47.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:46.372731+0000 mgr.y (mgr.14146) 1937 : cluster [DBG] pgmap v2863: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:47.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:47 smithi169 bash[19604]: cluster 2024-02-12T16:48:46.441504+0000 mon.a (mon.0) 3466 : cluster [DBG] osdmap e1060: 12 total, 12 up, 12 in 2024-02-12T16:48:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:48 smithi123 bash[19119]: cluster 2024-02-12T16:48:46.924517+0000 osd.5 (osd.5) 241 : cluster [DBG] 1.18 deep-scrub starts 2024-02-12T16:48:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:48 smithi123 bash[19119]: cluster 2024-02-12T16:48:46.968259+0000 osd.5 (osd.5) 242 : cluster [DBG] 1.18 deep-scrub ok 2024-02-12T16:48:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:48 smithi121 bash[18489]: cluster 2024-02-12T16:48:46.924517+0000 osd.5 (osd.5) 241 : cluster [DBG] 1.18 deep-scrub starts 2024-02-12T16:48:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:48 smithi121 bash[18489]: cluster 2024-02-12T16:48:46.968259+0000 osd.5 (osd.5) 242 : cluster [DBG] 1.18 deep-scrub ok 2024-02-12T16:48:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:48 smithi169 bash[19604]: cluster 2024-02-12T16:48:46.924517+0000 osd.5 (osd.5) 241 : cluster [DBG] 1.18 deep-scrub starts 2024-02-12T16:48:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:48 smithi169 bash[19604]: cluster 2024-02-12T16:48:46.968259+0000 osd.5 (osd.5) 242 : cluster [DBG] 1.18 deep-scrub ok 2024-02-12T16:48:49.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:49 smithi123 bash[19119]: cluster 2024-02-12T16:48:48.373417+0000 mgr.y (mgr.14146) 1938 : cluster [DBG] pgmap v2865: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:49 smithi121 bash[18489]: cluster 2024-02-12T16:48:48.373417+0000 mgr.y (mgr.14146) 1938 : cluster [DBG] pgmap v2865: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:49 smithi169 bash[19604]: cluster 2024-02-12T16:48:48.373417+0000 mgr.y (mgr.14146) 1938 : cluster [DBG] pgmap v2865: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-12T16:48:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:50 smithi123 bash[19119]: cluster 2024-02-12T16:48:49.204031+0000 osd.7 (osd.7) 33 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:48:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:50 smithi123 bash[19119]: cluster 2024-02-12T16:48:49.268338+0000 osd.7 (osd.7) 34 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:48:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:50 smithi123 bash[19119]: audit 2024-02-12T16:48:49.598929+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? v1:172.21.15.121:0/3201295736' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:48:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:50 smithi123 bash[19119]: cluster 2024-02-12T16:48:49.926108+0000 osd.7 (osd.7) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:50.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:50 smithi123 bash[19119]: cluster 2024-02-12T16:48:49.926203+0000 osd.7 (osd.7) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:50.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:50 smithi121 bash[18489]: cluster 2024-02-12T16:48:49.204031+0000 osd.7 (osd.7) 33 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:48:50.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:50 smithi121 bash[18489]: cluster 2024-02-12T16:48:49.268338+0000 osd.7 (osd.7) 34 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:48:50.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:50 smithi121 bash[18489]: audit 2024-02-12T16:48:49.598929+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? v1:172.21.15.121:0/3201295736' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:48:50.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:50 smithi121 bash[18489]: cluster 2024-02-12T16:48:49.926108+0000 osd.7 (osd.7) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:50.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:50 smithi121 bash[18489]: cluster 2024-02-12T16:48:49.926203+0000 osd.7 (osd.7) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:50 smithi169 bash[19604]: cluster 2024-02-12T16:48:49.204031+0000 osd.7 (osd.7) 33 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:48:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:50 smithi169 bash[19604]: cluster 2024-02-12T16:48:49.268338+0000 osd.7 (osd.7) 34 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:48:50.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:50 smithi169 bash[19604]: audit 2024-02-12T16:48:49.598929+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? v1:172.21.15.121:0/3201295736' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-12T16:48:50.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:50 smithi169 bash[19604]: cluster 2024-02-12T16:48:49.926108+0000 osd.7 (osd.7) 35 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:50.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:50 smithi169 bash[19604]: cluster 2024-02-12T16:48:49.926203+0000 osd.7 (osd.7) 36 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:50.924 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 4 1 2024-02-12T16:48:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:51 smithi123 bash[19119]: audit 2024-02-12T16:48:50.126008+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? v1:172.21.15.121:0/3201295736' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:48:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:51 smithi123 bash[19119]: cluster 2024-02-12T16:48:50.126183+0000 mon.a (mon.0) 3469 : cluster [DBG] osdmap e1061: 12 total, 12 up, 12 in 2024-02-12T16:48:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:51 smithi123 bash[19119]: cluster 2024-02-12T16:48:50.374077+0000 mgr.y (mgr.14146) 1939 : cluster [DBG] pgmap v2867: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:51 smithi123 bash[19119]: cluster 2024-02-12T16:48:50.575375+0000 osd.1 (osd.1) 192 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:48:51.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:51 smithi123 bash[19119]: cluster 2024-02-12T16:48:50.738150+0000 osd.1 (osd.1) 193 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:48:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:51 smithi121 bash[18489]: audit 2024-02-12T16:48:50.126008+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? v1:172.21.15.121:0/3201295736' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:48:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:51 smithi121 bash[18489]: cluster 2024-02-12T16:48:50.126183+0000 mon.a (mon.0) 3469 : cluster [DBG] osdmap e1061: 12 total, 12 up, 12 in 2024-02-12T16:48:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:51 smithi121 bash[18489]: cluster 2024-02-12T16:48:50.374077+0000 mgr.y (mgr.14146) 1939 : cluster [DBG] pgmap v2867: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:51 smithi121 bash[18489]: cluster 2024-02-12T16:48:50.575375+0000 osd.1 (osd.1) 192 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:48:51.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:51 smithi121 bash[18489]: cluster 2024-02-12T16:48:50.738150+0000 osd.1 (osd.1) 193 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:48:51.527 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:48:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:51 smithi169 bash[19604]: audit 2024-02-12T16:48:50.126008+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? v1:172.21.15.121:0/3201295736' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-12T16:48:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:51 smithi169 bash[19604]: cluster 2024-02-12T16:48:50.126183+0000 mon.a (mon.0) 3469 : cluster [DBG] osdmap e1061: 12 total, 12 up, 12 in 2024-02-12T16:48:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:51 smithi169 bash[19604]: cluster 2024-02-12T16:48:50.374077+0000 mgr.y (mgr.14146) 1939 : cluster [DBG] pgmap v2867: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:51 smithi169 bash[19604]: cluster 2024-02-12T16:48:50.575375+0000 osd.1 (osd.1) 192 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:48:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:51 smithi169 bash[19604]: cluster 2024-02-12T16:48:50.738150+0000 osd.1 (osd.1) 193 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:48:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:53 smithi169 bash[19604]: cluster 2024-02-12T16:48:51.991766+0000 osd.6 (osd.6) 202 : cluster [DBG] 1.7 deep-scrub starts 2024-02-12T16:48:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:53 smithi169 bash[19604]: cluster 2024-02-12T16:48:52.123164+0000 osd.6 (osd.6) 203 : cluster [DBG] 1.7 deep-scrub ok 2024-02-12T16:48:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:53 smithi169 bash[19604]: cluster 2024-02-12T16:48:52.374826+0000 mgr.y (mgr.14146) 1940 : cluster [DBG] pgmap v2868: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:53 smithi169 bash[19604]: cluster 2024-02-12T16:48:52.623879+0000 osd.1 (osd.1) 194 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:48:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:53 smithi169 bash[19604]: cluster 2024-02-12T16:48:52.838438+0000 osd.1 (osd.1) 195 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:48:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:53 smithi123 bash[19119]: cluster 2024-02-12T16:48:51.991766+0000 osd.6 (osd.6) 202 : cluster [DBG] 1.7 deep-scrub starts 2024-02-12T16:48:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:53 smithi123 bash[19119]: cluster 2024-02-12T16:48:52.123164+0000 osd.6 (osd.6) 203 : cluster [DBG] 1.7 deep-scrub ok 2024-02-12T16:48:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:53 smithi123 bash[19119]: cluster 2024-02-12T16:48:52.374826+0000 mgr.y (mgr.14146) 1940 : cluster [DBG] pgmap v2868: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:53 smithi123 bash[19119]: cluster 2024-02-12T16:48:52.623879+0000 osd.1 (osd.1) 194 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:48:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:53 smithi123 bash[19119]: cluster 2024-02-12T16:48:52.838438+0000 osd.1 (osd.1) 195 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:48:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:53 smithi121 bash[18489]: cluster 2024-02-12T16:48:51.991766+0000 osd.6 (osd.6) 202 : cluster [DBG] 1.7 deep-scrub starts 2024-02-12T16:48:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:53 smithi121 bash[18489]: cluster 2024-02-12T16:48:52.123164+0000 osd.6 (osd.6) 203 : cluster [DBG] 1.7 deep-scrub ok 2024-02-12T16:48:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:53 smithi121 bash[18489]: cluster 2024-02-12T16:48:52.374826+0000 mgr.y (mgr.14146) 1940 : cluster [DBG] pgmap v2868: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:53 smithi121 bash[18489]: cluster 2024-02-12T16:48:52.623879+0000 osd.1 (osd.1) 194 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:48:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:53 smithi121 bash[18489]: cluster 2024-02-12T16:48:52.838438+0000 osd.1 (osd.1) 195 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:48:54.700 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:48:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:48:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:48:54.700 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:48:54.700 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:48:54.739 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:54 smithi169 bash[19604]: cluster 2024-02-12T16:48:53.255430+0000 osd.9 (osd.9) 173 : cluster [DBG] 1.a scrub starts 2024-02-12T16:48:54.740 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:54 smithi169 bash[19604]: cluster 2024-02-12T16:48:53.345569+0000 osd.9 (osd.9) 174 : cluster [DBG] 1.a scrub ok 2024-02-12T16:48:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:54 smithi123 bash[19119]: cluster 2024-02-12T16:48:53.255430+0000 osd.9 (osd.9) 173 : cluster [DBG] 1.a scrub starts 2024-02-12T16:48:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:54 smithi123 bash[19119]: cluster 2024-02-12T16:48:53.345569+0000 osd.9 (osd.9) 174 : cluster [DBG] 1.a scrub ok 2024-02-12T16:48:54.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:54 smithi121 bash[18489]: cluster 2024-02-12T16:48:53.255430+0000 osd.9 (osd.9) 173 : cluster [DBG] 1.a scrub starts 2024-02-12T16:48:54.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:54 smithi121 bash[18489]: cluster 2024-02-12T16:48:53.345569+0000 osd.9 (osd.9) 174 : cluster [DBG] 1.a scrub ok 2024-02-12T16:48:55.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-02-12T16:48:55.724 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:48:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:55 smithi169 bash[19604]: cluster 2024-02-12T16:48:53.667987+0000 osd.10 (osd.10) 262 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:48:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:55 smithi169 bash[19604]: cluster 2024-02-12T16:48:53.713935+0000 osd.10 (osd.10) 263 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:48:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:55 smithi169 bash[19604]: cluster 2024-02-12T16:48:54.375527+0000 mgr.y (mgr.14146) 1941 : cluster [DBG] pgmap v2869: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:55 smithi123 bash[19119]: cluster 2024-02-12T16:48:53.667987+0000 osd.10 (osd.10) 262 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:48:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:55 smithi123 bash[19119]: cluster 2024-02-12T16:48:53.713935+0000 osd.10 (osd.10) 263 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:48:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:55 smithi123 bash[19119]: cluster 2024-02-12T16:48:54.375527+0000 mgr.y (mgr.14146) 1941 : cluster [DBG] pgmap v2869: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:55 smithi121 bash[18489]: cluster 2024-02-12T16:48:53.667987+0000 osd.10 (osd.10) 262 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:48:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:55 smithi121 bash[18489]: cluster 2024-02-12T16:48:53.713935+0000 osd.10 (osd.10) 263 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:48:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:55 smithi121 bash[18489]: cluster 2024-02-12T16:48:54.375527+0000 mgr.y (mgr.14146) 1941 : cluster [DBG] pgmap v2869: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: cluster 2024-02-12T16:48:55.181698+0000 osd.4 (osd.4) 181 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: cluster 2024-02-12T16:48:55.205819+0000 osd.0 (osd.0) 173 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: cluster 2024-02-12T16:48:55.273571+0000 osd.4 (osd.4) 182 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: cluster 2024-02-12T16:48:55.294546+0000 osd.0 (osd.0) 174 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: audit 2024-02-12T16:48:56.380159+0000 mon.a (mon.0) 3470 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: audit 2024-02-12T16:48:56.380590+0000 mon.a (mon.0) 3471 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: audit 2024-02-12T16:48:56.381088+0000 mon.a (mon.0) 3472 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:48:56.791 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:56 smithi121 bash[18489]: audit 2024-02-12T16:48:56.408118+0000 mon.a (mon.0) 3473 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: cluster 2024-02-12T16:48:55.181698+0000 osd.4 (osd.4) 181 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: cluster 2024-02-12T16:48:55.205819+0000 osd.0 (osd.0) 173 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: cluster 2024-02-12T16:48:55.273571+0000 osd.4 (osd.4) 182 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: cluster 2024-02-12T16:48:55.294546+0000 osd.0 (osd.0) 174 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: audit 2024-02-12T16:48:56.380159+0000 mon.a (mon.0) 3470 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: audit 2024-02-12T16:48:56.380590+0000 mon.a (mon.0) 3471 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: audit 2024-02-12T16:48:56.381088+0000 mon.a (mon.0) 3472 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:48:56.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:56 smithi169 bash[19604]: audit 2024-02-12T16:48:56.408118+0000 mon.a (mon.0) 3473 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: cluster 2024-02-12T16:48:55.181698+0000 osd.4 (osd.4) 181 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: cluster 2024-02-12T16:48:55.205819+0000 osd.0 (osd.0) 173 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: cluster 2024-02-12T16:48:55.273571+0000 osd.4 (osd.4) 182 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: cluster 2024-02-12T16:48:55.294546+0000 osd.0 (osd.0) 174 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: audit 2024-02-12T16:48:56.380159+0000 mon.a (mon.0) 3470 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: audit 2024-02-12T16:48:56.380590+0000 mon.a (mon.0) 3471 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: audit 2024-02-12T16:48:56.381088+0000 mon.a (mon.0) 3472 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:48:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:56 smithi123 bash[19119]: audit 2024-02-12T16:48:56.408118+0000 mon.a (mon.0) 3473 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:48:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:57 smithi169 bash[19604]: cluster 2024-02-12T16:48:56.169585+0000 osd.7 (osd.7) 37 : cluster [DBG] 1.9 deep-scrub starts 2024-02-12T16:48:57.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:57 smithi169 bash[19604]: cluster 2024-02-12T16:48:56.268040+0000 osd.7 (osd.7) 38 : cluster [DBG] 1.9 deep-scrub ok 2024-02-12T16:48:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:57 smithi123 bash[19119]: cluster 2024-02-12T16:48:56.169585+0000 osd.7 (osd.7) 37 : cluster [DBG] 1.9 deep-scrub starts 2024-02-12T16:48:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:57 smithi123 bash[19119]: cluster 2024-02-12T16:48:56.268040+0000 osd.7 (osd.7) 38 : cluster [DBG] 1.9 deep-scrub ok 2024-02-12T16:48:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:57 smithi121 bash[18489]: cluster 2024-02-12T16:48:56.169585+0000 osd.7 (osd.7) 37 : cluster [DBG] 1.9 deep-scrub starts 2024-02-12T16:48:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:57 smithi121 bash[18489]: cluster 2024-02-12T16:48:56.268040+0000 osd.7 (osd.7) 38 : cluster [DBG] 1.9 deep-scrub ok 2024-02-12T16:48:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:58 smithi169 bash[19604]: cluster 2024-02-12T16:48:56.769856+0000 osd.3 (osd.3) 138 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:58 smithi169 bash[19604]: cluster 2024-02-12T16:48:56.769973+0000 osd.3 (osd.3) 139 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:58 smithi169 bash[19604]: cluster 2024-02-12T16:48:57.203237+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:48:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:58 smithi169 bash[19604]: cluster 2024-02-12T16:48:57.229263+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:48:58.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:58 smithi169 bash[19604]: cluster 2024-02-12T16:48:57.453442+0000 mon.a (mon.0) 3474 : cluster [DBG] osdmap e1062: 12 total, 12 up, 12 in 2024-02-12T16:48:58.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:58 smithi121 bash[18489]: cluster 2024-02-12T16:48:56.769856+0000 osd.3 (osd.3) 138 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:58.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:58 smithi121 bash[18489]: cluster 2024-02-12T16:48:56.769973+0000 osd.3 (osd.3) 139 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:58.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:58 smithi121 bash[18489]: cluster 2024-02-12T16:48:57.203237+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:48:58.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:58 smithi121 bash[18489]: cluster 2024-02-12T16:48:57.229263+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:48:58.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:58 smithi121 bash[18489]: cluster 2024-02-12T16:48:57.453442+0000 mon.a (mon.0) 3474 : cluster [DBG] osdmap e1062: 12 total, 12 up, 12 in 2024-02-12T16:48:58.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:58 smithi123 bash[19119]: cluster 2024-02-12T16:48:56.769856+0000 osd.3 (osd.3) 138 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:48:58.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:58 smithi123 bash[19119]: cluster 2024-02-12T16:48:56.769973+0000 osd.3 (osd.3) 139 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:48:58.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:58 smithi123 bash[19119]: cluster 2024-02-12T16:48:57.203237+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:48:58.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:58 smithi123 bash[19119]: cluster 2024-02-12T16:48:57.229263+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:48:58.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:58 smithi123 bash[19119]: cluster 2024-02-12T16:48:57.453442+0000 mon.a (mon.0) 3474 : cluster [DBG] osdmap e1062: 12 total, 12 up, 12 in 2024-02-12T16:48:59.490 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.4 to 1 (10000) 2024-02-12T16:48:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:59 smithi169 bash[19604]: cluster 2024-02-12T16:48:58.376779+0000 mgr.y (mgr.14146) 1943 : cluster [DBG] pgmap v2872: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:48:59 smithi169 bash[19604]: audit 2024-02-12T16:48:59.267731+0000 mon.a (mon.0) 3475 : audit [INF] from='client.? v1:172.21.15.121:0/2705248562' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:48:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:59 smithi123 bash[19119]: cluster 2024-02-12T16:48:58.376779+0000 mgr.y (mgr.14146) 1943 : cluster [DBG] pgmap v2872: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:48:59 smithi123 bash[19119]: audit 2024-02-12T16:48:59.267731+0000 mon.a (mon.0) 3475 : audit [INF] from='client.? v1:172.21.15.121:0/2705248562' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:48:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:59 smithi121 bash[18489]: cluster 2024-02-12T16:48:58.376779+0000 mgr.y (mgr.14146) 1943 : cluster [DBG] pgmap v2872: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:48:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:48:59 smithi121 bash[18489]: audit 2024-02-12T16:48:59.267731+0000 mon.a (mon.0) 3475 : audit [INF] from='client.? v1:172.21.15.121:0/2705248562' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:49:00.190 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:49:00.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-12T16:49:00.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:00 smithi169 bash[19604]: cluster 2024-02-12T16:48:58.964932+0000 osd.5 (osd.5) 243 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:49:00.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:00 smithi169 bash[19604]: cluster 2024-02-12T16:48:59.075252+0000 osd.5 (osd.5) 244 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:49:00.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:00 smithi169 bash[19604]: audit 2024-02-12T16:48:59.480758+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? v1:172.21.15.121:0/2705248562' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:49:00.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:00 smithi169 bash[19604]: cluster 2024-02-12T16:48:59.480809+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e1063: 12 total, 12 up, 12 in 2024-02-12T16:49:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:00 smithi123 bash[19119]: cluster 2024-02-12T16:48:58.964932+0000 osd.5 (osd.5) 243 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:49:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:00 smithi123 bash[19119]: cluster 2024-02-12T16:48:59.075252+0000 osd.5 (osd.5) 244 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:49:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:00 smithi123 bash[19119]: audit 2024-02-12T16:48:59.480758+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? v1:172.21.15.121:0/2705248562' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:49:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:00 smithi123 bash[19119]: cluster 2024-02-12T16:48:59.480809+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e1063: 12 total, 12 up, 12 in 2024-02-12T16:49:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:00 smithi121 bash[18489]: cluster 2024-02-12T16:48:58.964932+0000 osd.5 (osd.5) 243 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:49:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:00 smithi121 bash[18489]: cluster 2024-02-12T16:48:59.075252+0000 osd.5 (osd.5) 244 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:49:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:00 smithi121 bash[18489]: audit 2024-02-12T16:48:59.480758+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? v1:172.21.15.121:0/2705248562' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:49:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:00 smithi121 bash[18489]: cluster 2024-02-12T16:48:59.480809+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e1063: 12 total, 12 up, 12 in 2024-02-12T16:49:01.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:01 smithi169 bash[19604]: cluster 2024-02-12T16:48:59.766852+0000 osd.10 (osd.10) 264 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:49:01.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:01 smithi169 bash[19604]: cluster 2024-02-12T16:48:59.831428+0000 osd.10 (osd.10) 265 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:49:01.622 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:01 smithi169 bash[19604]: cluster 2024-02-12T16:49:00.377362+0000 mgr.y (mgr.14146) 1944 : cluster [DBG] pgmap v2874: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:01 smithi123 bash[19119]: cluster 2024-02-12T16:48:59.766852+0000 osd.10 (osd.10) 264 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:49:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:01 smithi123 bash[19119]: cluster 2024-02-12T16:48:59.831428+0000 osd.10 (osd.10) 265 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:49:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:01 smithi123 bash[19119]: cluster 2024-02-12T16:49:00.377362+0000 mgr.y (mgr.14146) 1944 : cluster [DBG] pgmap v2874: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:01 smithi121 bash[18489]: cluster 2024-02-12T16:48:59.766852+0000 osd.10 (osd.10) 264 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:49:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:01 smithi121 bash[18489]: cluster 2024-02-12T16:48:59.831428+0000 osd.10 (osd.10) 265 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:49:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:01 smithi121 bash[18489]: cluster 2024-02-12T16:49:00.377362+0000 mgr.y (mgr.14146) 1944 : cluster [DBG] pgmap v2874: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:02 smithi169 bash[19604]: cluster 2024-02-12T16:49:01.223650+0000 osd.4 (osd.4) 183 : cluster [DBG] 1.e scrub starts 2024-02-12T16:49:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:02 smithi169 bash[19604]: cluster 2024-02-12T16:49:01.374971+0000 osd.4 (osd.4) 184 : cluster [DBG] 1.e scrub ok 2024-02-12T16:49:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:02 smithi169 bash[19604]: audit 2024-02-12T16:49:01.809526+0000 mon.a (mon.0) 3478 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:49:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:02 smithi169 bash[19604]: audit 2024-02-12T16:49:02.096516+0000 mon.a (mon.0) 3479 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:02 smithi169 bash[19604]: audit 2024-02-12T16:49:02.098284+0000 mon.a (mon.0) 3480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:02.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:02 smithi169 bash[19604]: audit 2024-02-12T16:49:02.108234+0000 mon.a (mon.0) 3481 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:49:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:02 smithi123 bash[19119]: cluster 2024-02-12T16:49:01.223650+0000 osd.4 (osd.4) 183 : cluster [DBG] 1.e scrub starts 2024-02-12T16:49:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:02 smithi123 bash[19119]: cluster 2024-02-12T16:49:01.374971+0000 osd.4 (osd.4) 184 : cluster [DBG] 1.e scrub ok 2024-02-12T16:49:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:02 smithi123 bash[19119]: audit 2024-02-12T16:49:01.809526+0000 mon.a (mon.0) 3478 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:49:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:02 smithi123 bash[19119]: audit 2024-02-12T16:49:02.096516+0000 mon.a (mon.0) 3479 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:02 smithi123 bash[19119]: audit 2024-02-12T16:49:02.098284+0000 mon.a (mon.0) 3480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:02.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:02 smithi123 bash[19119]: audit 2024-02-12T16:49:02.108234+0000 mon.a (mon.0) 3481 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:49:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:02 smithi121 bash[18489]: cluster 2024-02-12T16:49:01.223650+0000 osd.4 (osd.4) 183 : cluster [DBG] 1.e scrub starts 2024-02-12T16:49:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:02 smithi121 bash[18489]: cluster 2024-02-12T16:49:01.374971+0000 osd.4 (osd.4) 184 : cluster [DBG] 1.e scrub ok 2024-02-12T16:49:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:02 smithi121 bash[18489]: audit 2024-02-12T16:49:01.809526+0000 mon.a (mon.0) 3478 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:49:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:02 smithi121 bash[18489]: audit 2024-02-12T16:49:02.096516+0000 mon.a (mon.0) 3479 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:49:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:02 smithi121 bash[18489]: audit 2024-02-12T16:49:02.098284+0000 mon.a (mon.0) 3480 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:49:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:02 smithi121 bash[18489]: audit 2024-02-12T16:49:02.108234+0000 mon.a (mon.0) 3481 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:160558 8.c 8:3f13e905:::benchmark_data_smithi169_115677_object77930:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.420497+0000", 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 441.12003146000001, 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033072887000000002, 2024-02-12T16:49:03.547 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 160558 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.420497+0000", 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.420497+0000", 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.420494+0000", 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4608999999999999e-05 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.420539+0000", 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6614000000000001e-05 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.420555+0000", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.52e-06 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.420563+0000", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001711534 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422274+0000", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000123161 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422397+0000", 2024-02-12T16:49:03.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000133237 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422531+0000", 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024786656000000001 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.447317+0000", 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061492489999999999 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.453467+0000", 2024-02-12T16:49:03.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5810000000000003e-05 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.453522+0000", 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7117999999999998e-05 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.453570+0000", 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033072887000000002 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:160562 8.c 8:3ecc64b7:::benchmark_data_smithi169_115677_object77934:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.422631+0000", 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 441.11789688699997, 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.034999303000000002, 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 160562 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422631+0000", 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422631+0000", 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422629+0000", 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0679999999999997e-06 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422637+0000", 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1212999999999999e-05 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422648+0000", 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9099999999999996e-06 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.422653+0000", 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049542459999999998 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.427607+0000", 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012953800000000001 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.427737+0000", 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000153936 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.427891+0000", 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.019442292 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.556 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.447333+0000", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010196656 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.457530+0000", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2117999999999998e-05 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.457582+0000", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8866000000000003e-05 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.457631+0000", 2024-02-12T16:49:03.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.034999303000000002 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:160566 8.c 8:3c288309:::benchmark_data_smithi169_115677_object77938:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:42.426445+0000", 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 441.11408330099999, 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031304721000000001, 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 160566 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.558 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.426445+0000", 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.426445+0000", 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.426442+0000", 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0679000000000001e-05 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.426452+0000", 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9632000000000001e-05 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.426472+0000", 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8089999999999999e-06 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.426481+0000", 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0064262470000000004 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.432907+0000", 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000153029 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.433060+0000", 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013543100000000001 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.433195+0000", 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014359248 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.447555+0000", 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010120726 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.457675+0000", 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.282e-05 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.457718+0000", 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1390000000000003e-05 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:42.457750+0000", 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031304721000000001 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:161287 8.c 8:3482d9c1:::benchmark_data_smithi169_115677_object78659:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:43.072491+0000", 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 440.46803683299999, 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032600687000000003, 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.563 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 161287 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.072491+0000", 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.072491+0000", 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.072491+0000", 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.399e-06 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.072494+0000", 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4299999999999999e-06 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.072498+0000", 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4489999999999999e-06 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.072502+0000", 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011963952 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.084466+0000", 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013354099999999999 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.084599+0000", 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015680300000000001 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.084756+0000", 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010995661 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.095752+0000", 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092257269999999995 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.104978+0000", 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5888000000000003e-05 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.105043+0000", 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8532999999999998e-05 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.105092+0000", 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032600687000000003 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:161825 8.c 8:32c211ff:::benchmark_data_smithi169_115677_object79197:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:43.573252+0000", 2024-02-12T16:49:03.568 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 439.96727645300001, 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032670588, 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 161825 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.573252+0000", 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.573252+0000", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.573249+0000", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0320000000000003e-06 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.573257+0000", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4143e-05 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.573272+0000", 2024-02-12T16:49:03.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7970000000000003e-06 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.573278+0000", 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015386583000000001 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.588665+0000", 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6158999999999999e-05 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.588761+0000", 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109176 2024-02-12T16:49:03.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.588870+0000", 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087362429999999994 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.597607+0000", 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081929700000000008 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.605800+0000", 2024-02-12T16:49:03.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7909000000000001e-05 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.605887+0000", 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4768999999999999e-05 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.605922+0000", 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032670588 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:161982 8.c 8:30e4dc80:::benchmark_data_smithi169_115677_object79354:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:43.705801+0000", 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 439.83472669299999, 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030992546999999999, 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 161982 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.705801+0000", 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.705801+0000", 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.705799+0000", 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4630000000000004e-06 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.705807+0000", 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4569000000000001e-05 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.705821+0000", 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7100000000000001e-06 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.705828+0000", 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0085649239999999998 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.714393+0000", 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010435800000000001 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.714497+0000", 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011706599999999999 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.714614+0000", 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012626722999999999 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.727241+0000", 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094732740000000003 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.736714+0000", 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5501999999999999e-05 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.736760+0000", 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4082000000000003e-05 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.578 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.736794+0000", 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030992546999999999 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:161987 8.c 8:3aa941cf:::benchmark_data_smithi169_115677_object79359:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:43.707086+0000", 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 439.83344194300003, 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032383426, 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.579 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 161987 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.707086+0000", 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.707086+0000", 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.707084+0000", 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9970000000000004e-06 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.707090+0000", 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1274000000000001e-05 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.707102+0000", 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8650000000000003e-06 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.707108+0000", 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015478929000000001 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.722587+0000", 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000118546 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.722706+0000", 2024-02-12T16:49:03.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013727499999999999 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.722843+0000", 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044198049999999997 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.727263+0000", 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012117411999999999 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.739380+0000", 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5896000000000002e-05 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.739426+0000", 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3263999999999998e-05 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.739470+0000", 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032383426 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.584 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:161992 8.c 8:3b0ac0ab:::benchmark_data_smithi169_115677_object79364:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:43.711511+0000", 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 439.82901738800001, 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.037895789999999999, 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 161992 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.585 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.711511+0000", 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.711511+0000", 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.711508+0000", 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6650000000000003e-06 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.711516+0000", 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4071e-05 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.711530+0000", 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7649999999999999e-06 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.711536+0000", 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01575797 2024-02-12T16:49:03.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.727294+0000", 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117484 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.727411+0000", 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013847999999999999 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.727550+0000", 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016591265000000001 2024-02-12T16:49:03.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.744141+0000", 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051469510000000003 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.749288+0000", 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3295999999999999e-05 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.749341+0000", 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5423e-05 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.589 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.749407+0000", 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.037895789999999999 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:162000 8.c 8:30dd7e47:::benchmark_data_smithi169_115677_object79372:head [delete] snapc 0=[] ondisk+write+known_if_redirected e901)", 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:43.718578+0000", 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 439.821950044, 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031010243999999999, 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.590 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 162000 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.718578+0000", 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.718578+0000", 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.591 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.718575+0000", 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8859999999999999e-06 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.718585+0000", 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6045e-05 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.718601+0000", 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1840000000000002e-06 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.592 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.718608+0000", 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013823218999999999 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.732431+0000", 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014055900000000001 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.732572+0000", 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000104558 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.732676+0000", 2024-02-12T16:49:03.593 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011474013 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.744150+0000", 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053104889999999998 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.749461+0000", 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5146000000000001e-05 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.749546+0000", 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2394000000000001e-05 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.594 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:43.749588+0000", 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031010243999999999 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:165098 8.c 8:3d9bc57a:::benchmark_data_smithi169_115677_object82470:head [delete] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:47.160705+0000", 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 436.37982263599997, 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.040062981999999997, 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.595 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 165098 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.160705+0000", 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.160705+0000", 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.596 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.160703+0000", 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4290000000000001e-06 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.160711+0000", 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5868000000000001e-05 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.160727+0000", 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1890000000000001e-06 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.160734+0000", 2024-02-12T16:49:03.597 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040187740000000001 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.164753+0000", 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000133285 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.164887+0000", 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000143342 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.165030+0000", 2024-02-12T16:49:03.598 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0099860139999999997 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.175016+0000", 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025669253999999999 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.200685+0000", 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5866999999999999e-05 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.200741+0000", 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7419999999999998e-05 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.599 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.200768+0000", 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.040062981999999997 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:165100 8.c 8:3b0edb3e:::benchmark_data_smithi169_115677_object82472:head [delete] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:47.162700+0000", 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 436.37782858399999, 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.038219599, 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 165100 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.162700+0000", 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.601 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.162700+0000", 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.162697+0000", 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5189999999999992e-06 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.162706+0000", 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6407999999999999e-05 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.602 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.162722+0000", 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1810000000000004e-06 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.162729+0000", 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069155120000000004 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.169645+0000", 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012518699999999999 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.603 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.169770+0000", 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013432400000000001 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.169904+0000", 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051343450000000002 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.175039+0000", 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.025790607 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.604 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.200829+0000", 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1353000000000001e-05 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.200881+0000", 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8615999999999998e-05 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.200919+0000", 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.038219599 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:165117 8.c 8:34f82946:::benchmark_data_smithi169_115677_object82489:head [delete] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:47.179063+0000", 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 436.36146466700001, 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032328477000000001, 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 165117 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.179063+0000", 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.179063+0000", 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.179060+0000", 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0242e-05 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.179071+0000", 2024-02-12T16:49:03.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8329e-05 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.179089+0000", 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.588e-06 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.179097+0000", 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071390550000000001 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.186236+0000", 2024-02-12T16:49:03.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131929 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.186368+0000", 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016253799999999999 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 8", 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.186530+0000", 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004039323 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.190569+0000", 2024-02-12T16:49:03.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020748294 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.211318+0000", 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5782999999999999e-05 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.211353+0000", 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8430999999999998e-05 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:47.211392+0000", 2024-02-12T16:49:03.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032328477000000001 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:56533 9.8 9:1ee5b695:::benchmark_data_smithi169_124168_object56532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e928)", 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:42:57.066345+0000", 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 366.47418331400002, 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.042588697000000002, 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:03.611 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 56533 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066345+0000", 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066345+0000", 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066343+0000", 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.283e-05 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066366+0000", 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0795000000000001e-05 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066376+0000", 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7929999999999997e-06 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066381+0000", 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1569000000000001e-05 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066423+0000", 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9080999999999995e-05 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066522+0000", 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103063 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 11", 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.066625+0000", 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015761250000000001 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.068201+0000", 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.040457875999999997 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.108659+0000", 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016848300000000001 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.108827+0000", 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000106063 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:42:57.108933+0000", 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.042588697000000002 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26977.0:76449 9.2 e932/916 9:4c4e309b:::benchmark_data_smithi169_124168_object76448:head v 932'4795, mlcod=932'4792)", 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:43:05.984654+0000", 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.55587446999999, 2024-02-12T16:49:03.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031394316999999998, 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984654+0000", 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984654+0000", 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984653+0000", 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6508999999999999e-05 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984690+0000", 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1979999999999998e-06 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984696+0000", 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5950000000000001e-06 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984700+0000", 2024-02-12T16:49:03.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029195670000000001 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.987620+0000", 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1379e-05 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.987641+0000", 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028364072000000001 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:06.016005+0000", 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2616e-05 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:06.016048+0000", 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031394316999999998 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.26977.0:76465 9.2 e932/916 9:46158318:::benchmark_data_smithi169_124168_object76464:head v 932'4796, mlcod=932'4792)", 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:43:05.984705+0000", 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 357.55582333500001, 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031368351000000003, 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984705+0000", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984705+0000", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999995 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984704+0000", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3224000000000001e-05 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984787+0000", 2024-02-12T16:49:03.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7479999999999996e-06 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984794+0000", 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8659999999999997e-06 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.984799+0000", 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063014569999999999 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.991100+0000", 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5423999999999999e-05 2024-02-12T16:49:03.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:05.991126+0000", 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024927588000000001 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:06.016053+0000", 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9669000000000001e-05 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:43:06.016073+0000", 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031368351000000003 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.623 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.a GRANT e1061)", 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:48:53.255049+0000", 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.28547949, 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026743600000000002, 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255049+0000", 2024-02-12T16:49:03.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255049+0000", 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999962 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255045+0000", 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.327e-05 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255058+0000", 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8603e-05 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255087+0000", 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0888999999999999e-05 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255108+0000", 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013698200000000001 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255245+0000", 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6742000000000002e-05 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255281+0000", 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0129999999999996e-06 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255288+0000", 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7690000000000001e-05 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.255316+0000", 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026743600000000002 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.627 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "rep_scrubmap(1.a e1040 from shard 6)", 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:48:53.327760+0000", 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.212767867, 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016629676999999999, 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.327760+0000", 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.327760+0000", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999938 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.327754+0000", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3105e-05 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.327777+0000", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5837000000000003e-05 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.327813+0000", 2024-02-12T16:49:03.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8791e-05 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.327832+0000", 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016521291 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.344353+0000", 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4409999999999999e-06 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.344363+0000", 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7206e-05 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.344390+0000", 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016629676999999999 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.6 REQUEST e1061)", 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:48:53.667342+0000", 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 9.8731863309999994, 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031753800000000002, 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667342+0000", 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667342+0000", 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999943 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667336+0000", 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9596999999999999e-05 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667356+0000", 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6822000000000001e-05 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667393+0000", 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5647000000000001e-05 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667408+0000", 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015169700000000001 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667560+0000", 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8325999999999999e-05 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667598+0000", 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0936000000000001e-05 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:53.667659+0000", 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031753800000000002 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.9 REQUEST e1061)", 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:48:56.167706+0000", 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.3728225170000004, 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030240099999999999, 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167706+0000", 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167706+0000", 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999933 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167699+0000", 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5242e-05 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167724+0000", 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8347999999999999e-05 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167763+0000", 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5917e-05 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167779+0000", 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011678700000000001 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167895+0000", 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2015999999999999e-05 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.167937+0000", 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0538999999999997e-05 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:56.168008+0000", 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030240099999999999 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDScrubReserve(1.1 REQUEST e1062)", 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:48:59.766183+0000", 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 3.774345007, 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027982199999999999, 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766183+0000", 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766183+0000", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999938 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766177+0000", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1157000000000001e-05 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766198+0000", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5162e-05 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766233+0000", 2024-02-12T16:49:03.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5140000000000001e-05 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766248+0000", 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012457500000000001 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766373+0000", 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9246000000000001e-05 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766402+0000", 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0921999999999997e-05 2024-02-12T16:49:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:48:59.766463+0000", 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027982199999999999 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:03.643 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:49:03.644 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:03 smithi169 bash[19604]: cluster 2024-02-12T16:49:02.378057+0000 mgr.y (mgr.14146) 1945 : cluster [DBG] pgmap v2875: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:03 smithi123 bash[19119]: cluster 2024-02-12T16:49:02.378057+0000 mgr.y (mgr.14146) 1945 : cluster [DBG] pgmap v2875: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:03 smithi121 bash[18489]: cluster 2024-02-12T16:49:02.378057+0000 mgr.y (mgr.14146) 1945 : cluster [DBG] pgmap v2875: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:04.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-02-12T16:49:04.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:04 smithi169 bash[19604]: cluster 2024-02-12T16:49:03.636295+0000 osd.8 (osd.8) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:04.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:04 smithi169 bash[19604]: cluster 2024-02-12T16:49:03.636433+0000 osd.8 (osd.8) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:04 smithi123 bash[19119]: cluster 2024-02-12T16:49:03.636295+0000 osd.8 (osd.8) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:04 smithi123 bash[19119]: cluster 2024-02-12T16:49:03.636433+0000 osd.8 (osd.8) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:04 smithi121 bash[18489]: cluster 2024-02-12T16:49:03.636295+0000 osd.8 (osd.8) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:04 smithi121 bash[18489]: cluster 2024-02-12T16:49:03.636433+0000 osd.8 (osd.8) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:05.035 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:49:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:05 smithi169 bash[19604]: cluster 2024-02-12T16:49:04.378727+0000 mgr.y (mgr.14146) 1946 : cluster [DBG] pgmap v2876: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:05 smithi169 bash[19604]: cluster 2024-02-12T16:49:04.506607+0000 mon.a (mon.0) 3482 : cluster [DBG] osdmap e1064: 12 total, 12 up, 12 in 2024-02-12T16:49:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:05 smithi123 bash[19119]: cluster 2024-02-12T16:49:04.378727+0000 mgr.y (mgr.14146) 1946 : cluster [DBG] pgmap v2876: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:05 smithi123 bash[19119]: cluster 2024-02-12T16:49:04.506607+0000 mon.a (mon.0) 3482 : cluster [DBG] osdmap e1064: 12 total, 12 up, 12 in 2024-02-12T16:49:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:05 smithi121 bash[18489]: cluster 2024-02-12T16:49:04.378727+0000 mgr.y (mgr.14146) 1946 : cluster [DBG] pgmap v2876: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:05 smithi121 bash[18489]: cluster 2024-02-12T16:49:04.506607+0000 mon.a (mon.0) 3482 : cluster [DBG] osdmap e1064: 12 total, 12 up, 12 in 2024-02-12T16:49:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:07 smithi123 bash[19119]: cluster 2024-02-12T16:49:06.171220+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:07 smithi123 bash[19119]: cluster 2024-02-12T16:49:06.171294+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:07.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:07 smithi123 bash[19119]: cluster 2024-02-12T16:49:06.379311+0000 mgr.y (mgr.14146) 1947 : cluster [DBG] pgmap v2878: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:07 smithi121 bash[18489]: cluster 2024-02-12T16:49:06.171220+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:07 smithi121 bash[18489]: cluster 2024-02-12T16:49:06.171294+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:07.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:07 smithi121 bash[18489]: cluster 2024-02-12T16:49:06.379311+0000 mgr.y (mgr.14146) 1947 : cluster [DBG] pgmap v2878: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:07 smithi169 bash[19604]: cluster 2024-02-12T16:49:06.171220+0000 osd.0 (osd.0) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:07 smithi169 bash[19604]: cluster 2024-02-12T16:49:06.171294+0000 osd.0 (osd.0) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:07 smithi169 bash[19604]: cluster 2024-02-12T16:49:06.379311+0000 mgr.y (mgr.14146) 1947 : cluster [DBG] pgmap v2878: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:08.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:08 smithi123 bash[19119]: cluster 2024-02-12T16:49:07.113872+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e1065: 12 total, 12 up, 12 in 2024-02-12T16:49:08.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:08 smithi121 bash[18489]: cluster 2024-02-12T16:49:07.113872+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e1065: 12 total, 12 up, 12 in 2024-02-12T16:49:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:08 smithi169 bash[19604]: cluster 2024-02-12T16:49:07.113872+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e1065: 12 total, 12 up, 12 in 2024-02-12T16:49:09.035 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:49:09.138 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-12T16:49:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:09 smithi123 bash[19119]: cluster 2024-02-12T16:49:08.379908+0000 mgr.y (mgr.14146) 1948 : cluster [DBG] pgmap v2880: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:09.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:09 smithi123 bash[19119]: audit 2024-02-12T16:49:08.798104+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? v1:172.21.15.121:0/2220497270' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:49:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:09 smithi121 bash[18489]: cluster 2024-02-12T16:49:08.379908+0000 mgr.y (mgr.14146) 1948 : cluster [DBG] pgmap v2880: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:09.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:09 smithi121 bash[18489]: audit 2024-02-12T16:49:08.798104+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? v1:172.21.15.121:0/2220497270' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:49:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:09 smithi169 bash[19604]: cluster 2024-02-12T16:49:08.379908+0000 mgr.y (mgr.14146) 1948 : cluster [DBG] pgmap v2880: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:09 smithi169 bash[19604]: audit 2024-02-12T16:49:08.798104+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? v1:172.21.15.121:0/2220497270' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-12T16:49:09.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 5 1 2024-02-12T16:49:10.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:10 smithi169 bash[19604]: audit 2024-02-12T16:49:09.126880+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? v1:172.21.15.121:0/2220497270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:49:10.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:10 smithi169 bash[19604]: cluster 2024-02-12T16:49:09.127016+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e1066: 12 total, 12 up, 12 in 2024-02-12T16:49:10.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:10 smithi169 bash[19604]: cluster 2024-02-12T16:49:09.200802+0000 osd.2 (osd.2) 153 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:49:10.378 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:10 smithi169 bash[19604]: cluster 2024-02-12T16:49:09.303748+0000 osd.2 (osd.2) 154 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:49:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:10 smithi123 bash[19119]: audit 2024-02-12T16:49:09.126880+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? v1:172.21.15.121:0/2220497270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:49:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:10 smithi123 bash[19119]: cluster 2024-02-12T16:49:09.127016+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e1066: 12 total, 12 up, 12 in 2024-02-12T16:49:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:10 smithi123 bash[19119]: cluster 2024-02-12T16:49:09.200802+0000 osd.2 (osd.2) 153 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:49:10.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:10 smithi123 bash[19119]: cluster 2024-02-12T16:49:09.303748+0000 osd.2 (osd.2) 154 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:49:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:10 smithi121 bash[18489]: audit 2024-02-12T16:49:09.126880+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? v1:172.21.15.121:0/2220497270' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-12T16:49:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:10 smithi121 bash[18489]: cluster 2024-02-12T16:49:09.127016+0000 mon.a (mon.0) 3486 : cluster [DBG] osdmap e1066: 12 total, 12 up, 12 in 2024-02-12T16:49:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:10 smithi121 bash[18489]: cluster 2024-02-12T16:49:09.200802+0000 osd.2 (osd.2) 153 : cluster [DBG] 1.14 deep-scrub starts 2024-02-12T16:49:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:10 smithi121 bash[18489]: cluster 2024-02-12T16:49:09.303748+0000 osd.2 (osd.2) 154 : cluster [DBG] 1.14 deep-scrub ok 2024-02-12T16:49:11.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:11 smithi121 bash[18489]: cluster 2024-02-12T16:49:10.380555+0000 mgr.y (mgr.14146) 1949 : cluster [DBG] pgmap v2882: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:11 smithi169 bash[19604]: cluster 2024-02-12T16:49:10.380555+0000 mgr.y (mgr.14146) 1949 : cluster [DBG] pgmap v2882: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:11 smithi123 bash[19119]: cluster 2024-02-12T16:49:10.380555+0000 mgr.y (mgr.14146) 1949 : cluster [DBG] pgmap v2882: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:12.347 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:49:12.348 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:49:12.348 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-02-12T16:49:12.348 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:49:12.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-02-12T16:49:13.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:13 smithi169 bash[19604]: cluster 2024-02-12T16:49:12.381268+0000 mgr.y (mgr.14146) 1950 : cluster [DBG] pgmap v2883: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:13 smithi123 bash[19119]: cluster 2024-02-12T16:49:12.381268+0000 mgr.y (mgr.14146) 1950 : cluster [DBG] pgmap v2883: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:13 smithi121 bash[18489]: cluster 2024-02-12T16:49:12.381268+0000 mgr.y (mgr.14146) 1950 : cluster [DBG] pgmap v2883: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:14.674 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:49:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:15 smithi169 bash[19604]: cluster 2024-02-12T16:49:14.382002+0000 mgr.y (mgr.14146) 1951 : cluster [DBG] pgmap v2884: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:15.843 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:15 smithi121 bash[18489]: cluster 2024-02-12T16:49:14.382002+0000 mgr.y (mgr.14146) 1951 : cluster [DBG] pgmap v2884: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:15 smithi123 bash[19119]: cluster 2024-02-12T16:49:14.382002+0000 mgr.y (mgr.14146) 1951 : cluster [DBG] pgmap v2884: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:17.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:17 smithi123 bash[19119]: cluster 2024-02-12T16:49:16.382589+0000 mgr.y (mgr.14146) 1952 : cluster [DBG] pgmap v2885: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:17 smithi121 bash[18489]: cluster 2024-02-12T16:49:16.382589+0000 mgr.y (mgr.14146) 1952 : cluster [DBG] pgmap v2885: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:17.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:17 smithi169 bash[19604]: cluster 2024-02-12T16:49:16.382589+0000 mgr.y (mgr.14146) 1952 : cluster [DBG] pgmap v2885: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:17.793 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:49:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.5 to 1 (10000) 2024-02-12T16:49:18.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:18 smithi169 bash[19604]: audit 2024-02-12T16:49:18.348735+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? v1:172.21.15.121:0/671486454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:49:18.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:18 smithi123 bash[19119]: audit 2024-02-12T16:49:18.348735+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? v1:172.21.15.121:0/671486454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:49:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:18 smithi121 bash[18489]: audit 2024-02-12T16:49:18.348735+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? v1:172.21.15.121:0/671486454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:49:19.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-12T16:49:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:19 smithi121 bash[18489]: cluster 2024-02-12T16:49:18.383333+0000 mgr.y (mgr.14146) 1953 : cluster [DBG] pgmap v2886: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:19 smithi121 bash[18489]: audit 2024-02-12T16:49:18.405328+0000 mon.a (mon.0) 3488 : audit [INF] from='client.? v1:172.21.15.121:0/671486454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:49:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:19 smithi121 bash[18489]: cluster 2024-02-12T16:49:18.405397+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e1067: 12 total, 12 up, 12 in 2024-02-12T16:49:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:19 smithi169 bash[19604]: cluster 2024-02-12T16:49:18.383333+0000 mgr.y (mgr.14146) 1953 : cluster [DBG] pgmap v2886: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:19.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:19 smithi169 bash[19604]: audit 2024-02-12T16:49:18.405328+0000 mon.a (mon.0) 3488 : audit [INF] from='client.? v1:172.21.15.121:0/671486454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:49:19.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:19 smithi169 bash[19604]: cluster 2024-02-12T16:49:18.405397+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e1067: 12 total, 12 up, 12 in 2024-02-12T16:49:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:19 smithi123 bash[19119]: cluster 2024-02-12T16:49:18.383333+0000 mgr.y (mgr.14146) 1953 : cluster [DBG] pgmap v2886: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:19 smithi123 bash[19119]: audit 2024-02-12T16:49:18.405328+0000 mon.a (mon.0) 3488 : audit [INF] from='client.? v1:172.21.15.121:0/671486454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:49:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:19 smithi123 bash[19119]: cluster 2024-02-12T16:49:18.405397+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e1067: 12 total, 12 up, 12 in 2024-02-12T16:49:20.699 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.502950+0000 mon.a (mon.0) 3490 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07669213814351329]}]: dispatch 2024-02-12T16:49:20.699 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.503544+0000 mon.a (mon.0) 3491 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09970392438191042]}]: dispatch 2024-02-12T16:49:20.699 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.503948+0000 mon.a (mon.0) 3492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09646542633675945]}]: dispatch 2024-02-12T16:49:20.699 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.504335+0000 mon.a (mon.0) 3493 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10294204489371399]}]: dispatch 2024-02-12T16:49:20.699 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.504760+0000 mon.a (mon.0) 3494 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09432312306601932]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.505175+0000 mon.a (mon.0) 3495 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08178976258233023]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.505577+0000 mon.a (mon.0) 3496 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08114825626194376]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.505953+0000 mon.a (mon.0) 3497 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07707137089842865]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.506348+0000 mon.a (mon.0) 3498 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09116209416226288]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.506716+0000 mon.a (mon.0) 3499 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08760460585561398]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.507152+0000 mon.a (mon.0) 3500 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07673315314352605]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.507549+0000 mon.a (mon.0) 3501 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08191048699272797]}]: dispatch 2024-02-12T16:49:20.700 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:20 smithi169 bash[19604]: audit 2024-02-12T16:49:19.507931+0000 mon.a (mon.0) 3502 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"36982\", \"9\": \"60363\"}"}]: dispatch 2024-02-12T16:49:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.502950+0000 mon.a (mon.0) 3490 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07669213814351329]}]: dispatch 2024-02-12T16:49:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.503544+0000 mon.a (mon.0) 3491 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09970392438191042]}]: dispatch 2024-02-12T16:49:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.503948+0000 mon.a (mon.0) 3492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09646542633675945]}]: dispatch 2024-02-12T16:49:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.504335+0000 mon.a (mon.0) 3493 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10294204489371399]}]: dispatch 2024-02-12T16:49:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.504760+0000 mon.a (mon.0) 3494 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09432312306601932]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.505175+0000 mon.a (mon.0) 3495 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08178976258233023]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.505577+0000 mon.a (mon.0) 3496 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08114825626194376]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.505953+0000 mon.a (mon.0) 3497 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07707137089842865]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.506348+0000 mon.a (mon.0) 3498 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09116209416226288]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.506716+0000 mon.a (mon.0) 3499 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08760460585561398]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.507152+0000 mon.a (mon.0) 3500 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07673315314352605]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.507549+0000 mon.a (mon.0) 3501 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08191048699272797]}]: dispatch 2024-02-12T16:49:20.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:20 smithi123 bash[19119]: audit 2024-02-12T16:49:19.507931+0000 mon.a (mon.0) 3502 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"36982\", \"9\": \"60363\"}"}]: dispatch 2024-02-12T16:49:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.502950+0000 mon.a (mon.0) 3490 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07669213814351329]}]: dispatch 2024-02-12T16:49:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.503544+0000 mon.a (mon.0) 3491 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09970392438191042]}]: dispatch 2024-02-12T16:49:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.503948+0000 mon.a (mon.0) 3492 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09646542633675945]}]: dispatch 2024-02-12T16:49:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.504335+0000 mon.a (mon.0) 3493 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10294204489371399]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.504760+0000 mon.a (mon.0) 3494 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09432312306601932]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.505175+0000 mon.a (mon.0) 3495 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08178976258233023]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.505577+0000 mon.a (mon.0) 3496 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08114825626194376]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.505953+0000 mon.a (mon.0) 3497 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07707137089842865]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.506348+0000 mon.a (mon.0) 3498 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09116209416226288]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.506716+0000 mon.a (mon.0) 3499 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08760460585561398]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.507152+0000 mon.a (mon.0) 3500 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07673315314352605]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.507549+0000 mon.a (mon.0) 3501 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08191048699272797]}]: dispatch 2024-02-12T16:49:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:20 smithi121 bash[18489]: audit 2024-02-12T16:49:19.507931+0000 mon.a (mon.0) 3502 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"36982\", \"9\": \"60363\"}"}]: dispatch 2024-02-12T16:49:20.950 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:49:20.951 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:49:20.951 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:49:20.951 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:49:20.951 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:49:21.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-02-12T16:49:21.694 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: cluster 2024-02-12T16:49:20.384159+0000 mgr.y (mgr.14146) 1954 : cluster [DBG] pgmap v2888: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:21.694 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.428924+0000 mon.a (mon.0) 3503 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07669213814351329]}]': finished 2024-02-12T16:49:21.694 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.429058+0000 mon.a (mon.0) 3504 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09970392438191042]}]': finished 2024-02-12T16:49:21.694 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.429165+0000 mon.a (mon.0) 3505 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09646542633675945]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.429255+0000 mon.a (mon.0) 3506 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10294204489371399]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.429451+0000 mon.a (mon.0) 3507 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09432312306601932]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.429583+0000 mon.a (mon.0) 3508 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08178976258233023]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.429729+0000 mon.a (mon.0) 3509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08114825626194376]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.429889+0000 mon.a (mon.0) 3510 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07707137089842865]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.430033+0000 mon.a (mon.0) 3511 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09116209416226288]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.430213+0000 mon.a (mon.0) 3512 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08760460585561398]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.430354+0000 mon.a (mon.0) 3513 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07673315314352605]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.430496+0000 mon.a (mon.0) 3514 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08191048699272797]}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: audit 2024-02-12T16:49:20.430639+0000 mon.a (mon.0) 3515 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"36982\", \"9\": \"60363\"}"}]': finished 2024-02-12T16:49:21.695 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:21 smithi169 bash[19604]: cluster 2024-02-12T16:49:20.430758+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e1068: 12 total, 12 up, 12 in 2024-02-12T16:49:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: cluster 2024-02-12T16:49:20.384159+0000 mgr.y (mgr.14146) 1954 : cluster [DBG] pgmap v2888: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.428924+0000 mon.a (mon.0) 3503 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07669213814351329]}]': finished 2024-02-12T16:49:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.429058+0000 mon.a (mon.0) 3504 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09970392438191042]}]': finished 2024-02-12T16:49:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.429165+0000 mon.a (mon.0) 3505 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09646542633675945]}]': finished 2024-02-12T16:49:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.429255+0000 mon.a (mon.0) 3506 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10294204489371399]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.429451+0000 mon.a (mon.0) 3507 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09432312306601932]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.429583+0000 mon.a (mon.0) 3508 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08178976258233023]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.429729+0000 mon.a (mon.0) 3509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08114825626194376]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.429889+0000 mon.a (mon.0) 3510 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07707137089842865]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.430033+0000 mon.a (mon.0) 3511 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09116209416226288]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.430213+0000 mon.a (mon.0) 3512 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08760460585561398]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.430354+0000 mon.a (mon.0) 3513 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07673315314352605]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.430496+0000 mon.a (mon.0) 3514 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08191048699272797]}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: audit 2024-02-12T16:49:20.430639+0000 mon.a (mon.0) 3515 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"36982\", \"9\": \"60363\"}"}]': finished 2024-02-12T16:49:21.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:21 smithi123 bash[19119]: cluster 2024-02-12T16:49:20.430758+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e1068: 12 total, 12 up, 12 in 2024-02-12T16:49:21.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: cluster 2024-02-12T16:49:20.384159+0000 mgr.y (mgr.14146) 1954 : cluster [DBG] pgmap v2888: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.428924+0000 mon.a (mon.0) 3503 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07669213814351329]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.429058+0000 mon.a (mon.0) 3504 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.09970392438191042]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.429165+0000 mon.a (mon.0) 3505 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09646542633675945]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.429255+0000 mon.a (mon.0) 3506 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10294204489371399]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.429451+0000 mon.a (mon.0) 3507 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09432312306601932]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.429583+0000 mon.a (mon.0) 3508 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08178976258233023]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.429729+0000 mon.a (mon.0) 3509 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08114825626194376]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.429889+0000 mon.a (mon.0) 3510 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07707137089842865]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.430033+0000 mon.a (mon.0) 3511 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09116209416226288]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.430213+0000 mon.a (mon.0) 3512 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08760460585561398]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.430354+0000 mon.a (mon.0) 3513 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07673315314352605]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.430496+0000 mon.a (mon.0) 3514 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08191048699272797]}]': finished 2024-02-12T16:49:21.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: audit 2024-02-12T16:49:20.430639+0000 mon.a (mon.0) 3515 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"36982\", \"9\": \"60363\"}"}]': finished 2024-02-12T16:49:21.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:21 smithi121 bash[18489]: cluster 2024-02-12T16:49:20.430758+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e1068: 12 total, 12 up, 12 in 2024-02-12T16:49:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:22 smithi169 bash[19604]: cluster 2024-02-12T16:49:21.432124+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e1069: 12 total, 12 up, 12 in 2024-02-12T16:49:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:22 smithi169 bash[19604]: cluster 2024-02-12T16:49:21.574718+0000 osd.10 (osd.10) 266 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:22 smithi169 bash[19604]: cluster 2024-02-12T16:49:21.574808+0000 osd.10 (osd.10) 267 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:22.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:22 smithi123 bash[19119]: cluster 2024-02-12T16:49:21.432124+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e1069: 12 total, 12 up, 12 in 2024-02-12T16:49:22.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:22 smithi123 bash[19119]: cluster 2024-02-12T16:49:21.574718+0000 osd.10 (osd.10) 266 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:22.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:22 smithi123 bash[19119]: cluster 2024-02-12T16:49:21.574808+0000 osd.10 (osd.10) 267 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:22 smithi121 bash[18489]: cluster 2024-02-12T16:49:21.432124+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e1069: 12 total, 12 up, 12 in 2024-02-12T16:49:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:22 smithi121 bash[18489]: cluster 2024-02-12T16:49:21.574718+0000 osd.10 (osd.10) 266 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:22.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:22 smithi121 bash[18489]: cluster 2024-02-12T16:49:21.574808+0000 osd.10 (osd.10) 267 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:23 smithi169 bash[19604]: cluster 2024-02-12T16:49:22.384850+0000 mgr.y (mgr.14146) 1955 : cluster [DBG] pgmap v2891: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:23 smithi169 bash[19604]: cluster 2024-02-12T16:49:22.448440+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e1070: 12 total, 12 up, 12 in 2024-02-12T16:49:23.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:23 smithi123 bash[19119]: cluster 2024-02-12T16:49:22.384850+0000 mgr.y (mgr.14146) 1955 : cluster [DBG] pgmap v2891: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:23.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:23 smithi123 bash[19119]: cluster 2024-02-12T16:49:22.448440+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e1070: 12 total, 12 up, 12 in 2024-02-12T16:49:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:23 smithi121 bash[18489]: cluster 2024-02-12T16:49:22.384850+0000 mgr.y (mgr.14146) 1955 : cluster [DBG] pgmap v2891: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:23 smithi121 bash[18489]: cluster 2024-02-12T16:49:22.448440+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e1070: 12 total, 12 up, 12 in 2024-02-12T16:49:24.006 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:49:24.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:24 smithi169 bash[19604]: cluster 2024-02-12T16:49:23.561877+0000 osd.3 (osd.3) 140 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:49:24.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:24 smithi169 bash[19604]: cluster 2024-02-12T16:49:23.701643+0000 osd.3 (osd.3) 141 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:49:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:24 smithi123 bash[19119]: cluster 2024-02-12T16:49:23.561877+0000 osd.3 (osd.3) 140 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:49:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:24 smithi123 bash[19119]: cluster 2024-02-12T16:49:23.701643+0000 osd.3 (osd.3) 141 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:49:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:24 smithi121 bash[18489]: cluster 2024-02-12T16:49:23.561877+0000 osd.3 (osd.3) 140 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:49:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:24 smithi121 bash[18489]: cluster 2024-02-12T16:49:23.701643+0000 osd.3 (osd.3) 141 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:49:25.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:25 smithi121 bash[18489]: cluster 2024-02-12T16:49:24.385509+0000 mgr.y (mgr.14146) 1956 : cluster [DBG] pgmap v2893: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:25 smithi169 bash[19604]: cluster 2024-02-12T16:49:24.385509+0000 mgr.y (mgr.14146) 1956 : cluster [DBG] pgmap v2893: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:25 smithi123 bash[19119]: cluster 2024-02-12T16:49:24.385509+0000 mgr.y (mgr.14146) 1956 : cluster [DBG] pgmap v2893: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:26.478 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:49:27.385 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:27 smithi121 bash[18489]: cluster 2024-02-12T16:49:26.386363+0000 mgr.y (mgr.14146) 1957 : cluster [DBG] pgmap v2894: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:27.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:27 smithi123 bash[19119]: cluster 2024-02-12T16:49:26.386363+0000 mgr.y (mgr.14146) 1957 : cluster [DBG] pgmap v2894: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:27.528 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:27 smithi169 bash[19604]: cluster 2024-02-12T16:49:26.386363+0000 mgr.y (mgr.14146) 1957 : cluster [DBG] pgmap v2894: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:28.141 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-12T16:49:28.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:28 smithi123 bash[19119]: audit 2024-02-12T16:49:27.833272+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? v1:172.21.15.121:0/2323060907' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:49:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:28 smithi121 bash[18489]: audit 2024-02-12T16:49:27.833272+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? v1:172.21.15.121:0/2323060907' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:49:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:28 smithi169 bash[19604]: audit 2024-02-12T16:49:27.833272+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? v1:172.21.15.121:0/2323060907' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-12T16:49:28.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 6 1 2024-02-12T16:49:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:29 smithi121 bash[18489]: audit 2024-02-12T16:49:28.131473+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? v1:172.21.15.121:0/2323060907' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:49:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:29 smithi121 bash[18489]: cluster 2024-02-12T16:49:28.131532+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e1071: 12 total, 12 up, 12 in 2024-02-12T16:49:29.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:29 smithi121 bash[18489]: cluster 2024-02-12T16:49:28.387025+0000 mgr.y (mgr.14146) 1958 : cluster [DBG] pgmap v2896: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:29.412 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:29 smithi169 bash[19604]: audit 2024-02-12T16:49:28.131473+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? v1:172.21.15.121:0/2323060907' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:49:29.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:29 smithi169 bash[19604]: cluster 2024-02-12T16:49:28.131532+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e1071: 12 total, 12 up, 12 in 2024-02-12T16:49:29.413 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:29 smithi169 bash[19604]: cluster 2024-02-12T16:49:28.387025+0000 mgr.y (mgr.14146) 1958 : cluster [DBG] pgmap v2896: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:29 smithi123 bash[19119]: audit 2024-02-12T16:49:28.131473+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? v1:172.21.15.121:0/2323060907' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-12T16:49:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:29 smithi123 bash[19119]: cluster 2024-02-12T16:49:28.131532+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e1071: 12 total, 12 up, 12 in 2024-02-12T16:49:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:29 smithi123 bash[19119]: cluster 2024-02-12T16:49:28.387025+0000 mgr.y (mgr.14146) 1958 : cluster [DBG] pgmap v2896: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:29.692 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:49:29.692 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:49:29.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:49:29.692 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106015 8.12 8:4d0dbb88:::benchmark_data_smithi169_115677_object23387:head [delete] snapc 0=[] ondisk+write+known_if_redirected e882)", 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:58.360267+0000", 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 511.32477991299999, 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0432394679999999, 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106015 2024-02-12T16:49:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360267+0000", 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360267+0000", 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360264+0000", 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9529999999999999e-06 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360272+0000", 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0678e-05 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360283+0000", 2024-02-12T16:49:29.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7360000000000004e-06 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.360290+0000", 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.166646664 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.526936+0000", 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2949999999999996e-06 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.526944+0000", 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.87383641700000003 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.400780+0000", 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109707 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.400890+0000", 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9071000000000002e-05 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 0", 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.400979+0000", 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024375270000000001 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403416+0000", 2024-02-12T16:49:29.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.241e-05 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403429+0000", 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1751000000000001e-05 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403470+0000", 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5691000000000001e-05 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.403506+0000", 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0432394679999999 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106019 8.16 8.e008b6b6 (undecoded) ondisk+write+known_if_redirected e882)", 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:58.368844+0000", 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 511.31620278899999, 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.012615437, 2024-02-12T16:49:29.700 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106019 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368844+0000", 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.701 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368844+0000", 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368840+0000", 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0348e-05 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368851+0000", 2024-02-12T16:49:29.702 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3125000000000001e-05 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368864+0000", 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3629999999999996e-06 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.368871+0000", 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15464666199999999 2024-02-12T16:49:29.703 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.523518+0000", 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.977e-06 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:58.523526+0000", 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.857852783 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.704 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381379+0000", 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0574999999999993e-05 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381459+0000", 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.012615437 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.705 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106036 8.16 8.8b727756 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.408773+0000", 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.27627340800001, 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97285082300000003, 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106036 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.706 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408773+0000", 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408773+0000", 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408771+0000", 2024-02-12T16:49:29.707 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0509999999999997e-06 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408779+0000", 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2703e-05 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408832+0000", 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115853 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.708 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408948+0000", 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0973999999999997e-05 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408999+0000", 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.524e-06 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409007+0000", 2024-02-12T16:49:29.709 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97260508099999998 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381612+0000", 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1589e-05 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381624+0000", 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97285082300000003 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.710 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106038 8.16 8.8e8b3fd6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.408988+0000", 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.27605900899999, 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97266427899999996, 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.711 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106038 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408988+0000", 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408988+0000", 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.712 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408986+0000", 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0949999999999998e-06 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.408993+0000", 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4509999999999992e-06 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409001+0000", 2024-02-12T16:49:29.713 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7910000000000001e-06 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409007+0000", 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5868000000000002e-05 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409053+0000", 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2830000000000004e-06 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.714 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.409060+0000", 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97258542299999995 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381646+0000", 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.993e-06 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381652+0000", 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.97266427899999996 2024-02-12T16:49:29.715 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106043 8.16 8.f5bba616 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.415033+0000", 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.27001360700001, 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96665297100000003, 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.716 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106043 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415033+0000", 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415033+0000", 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:29.717 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415031+0000", 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6410000000000002e-06 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415039+0000", 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3165999999999999e-05 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.718 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415092+0000", 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6299999999999993e-06 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415102+0000", 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1401000000000004e-05 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415163+0000", 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0858e-05 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.719 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.415174+0000", 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96650533500000002 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381679+0000", 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.708e-06 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381686+0000", 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.96665297100000003 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106103 8.16 8.720be176 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.457924+0000", 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.227122323, 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.92378728799999998, 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.721 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106103 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457924+0000", 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457924+0000", 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457921+0000", 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457931+0000", 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5812999999999999e-05 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457947+0000", 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3550000000000005e-06 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.457956+0000", 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8015e-05 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.724 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.458004+0000", 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2286000000000001e-05 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.458016+0000", 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.92368782999999999 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381704+0000", 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.79e-06 2024-02-12T16:49:29.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381712+0000", 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.92378728799999998 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106122 8.16 8.5e63e396 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.468415+0000", 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.21663137399997, 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91331797599999998, 2024-02-12T16:49:29.726 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106122 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468415+0000", 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468415+0000", 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468414+0000", 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8350000000000003e-06 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468419+0000", 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5565e-05 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468444+0000", 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0990000000000004e-06 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468450+0000", 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8056000000000002e-05 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468498+0000", 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4099999999999997e-06 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.468508+0000", 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91321966300000001 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381727+0000", 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8629999999999996e-06 2024-02-12T16:49:29.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381733+0000", 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.91331797599999998 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106151 8.16 8.48750896 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.481942+0000", 2024-02-12T16:49:29.731 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.20310429800003, 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89981281300000004, 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106151 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481942+0000", 2024-02-12T16:49:29.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481942+0000", 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481940+0000", 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1019999999999999e-06 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481947+0000", 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.381e-05 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481961+0000", 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0169999999999999e-06 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.481968+0000", 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4472000000000003e-05 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.482023+0000", 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4709999999999996e-06 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.482032+0000", 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89971682799999997 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381749+0000", 2024-02-12T16:49:29.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0480000000000004e-06 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381755+0000", 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89981281300000004 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106158 8.16 8.eba0cfd6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.485343+0000", 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.19970339499997, 2024-02-12T16:49:29.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89643377400000002, 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106158 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485343+0000", 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485343+0000", 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485341+0000", 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3170000000000002e-06 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485349+0000", 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4054e-05 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485363+0000", 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.674e-06 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485370+0000", 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8694000000000003e-05 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485439+0000", 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1878999999999999e-05 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.485451+0000", 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89631989000000001 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381771+0000", 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4330000000000001e-06 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381777+0000", 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.89643377400000002 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106172 8.16 8.6585dbd6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.493796+0000", 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.19125058100002, 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88803040600000005, 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106172 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493796+0000", 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493796+0000", 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493794+0000", 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2820000000000004e-06 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493800+0000", 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4489999999999999e-06 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493808+0000", 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9259999999999999e-06 2024-02-12T16:49:29.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493813+0000", 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5454000000000002e-05 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493859+0000", 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3679999999999996e-06 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.493866+0000", 2024-02-12T16:49:29.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88795410500000005 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381820+0000", 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3890000000000002e-06 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381826+0000", 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.88803040600000005 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106277 8.16 8.7fbde36 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.560827+0000", 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.12421954799999, 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82102122700000002, 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106277 2024-02-12T16:49:29.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560827+0000", 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560827+0000", 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560825+0000", 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2920000000000001e-06 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560831+0000", 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0268000000000001e-05 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560841+0000", 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8590000000000004e-06 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560846+0000", 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8593000000000001e-05 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560885+0000", 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3249999999999998e-06 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.560890+0000", 2024-02-12T16:49:29.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82095215499999996 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381842+0000", 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410000000000002e-06 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381848+0000", 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.82102122700000002 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106283 8.16 8.641c5ef6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.563678+0000", 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.12136826900002, 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81818920500000003, 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106283 2024-02-12T16:49:29.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563678+0000", 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563678+0000", 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563677+0000", 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6919999999999996e-06 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563684+0000", 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2099000000000001e-05 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563696+0000", 2024-02-12T16:49:29.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7170000000000002e-06 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563702+0000", 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9125999999999997e-05 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563751+0000", 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7589999999999997e-06 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.563759+0000", 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81810208600000001 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381861+0000", 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1639999999999998e-06 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381868+0000", 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.81818920500000003 2024-02-12T16:49:29.756 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106306 8.16 8.f15ebc16 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.579963+0000", 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.10508360400001, 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80192957499999995, 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.757 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106306 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579963+0000", 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579963+0000", 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579962+0000", 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7659999999999998e-06 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579966+0000", 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4961e-05 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579981+0000", 2024-02-12T16:49:29.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2300000000000002e-06 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.579986+0000", 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4341000000000001e-05 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.580030+0000", 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1549999999999998e-06 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.580035+0000", 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80185269400000003 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381888+0000", 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6859999999999997e-06 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381893+0000", 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.80192957499999995 2024-02-12T16:49:29.761 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106316 8.16 8.8ca12656 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.588489+0000", 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.09655758299999, 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.79342752000000005, 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.762 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106316 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588489+0000", 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588489+0000", 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588487+0000", 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2480000000000001e-06 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588493+0000", 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2643000000000001e-05 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588506+0000", 2024-02-12T16:49:29.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8010000000000004e-06 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588514+0000", 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4776e-05 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588559+0000", 2024-02-12T16:49:29.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6049999999999997e-06 2024-02-12T16:49:29.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.588564+0000", 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.79334808199999995 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381912+0000", 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1019999999999999e-06 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381917+0000", 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.79342752000000005 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.767 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106402 8.16 8.59a95bf6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.683695+0000", 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 510.00135152799999, 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.69824561600000001, 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.768 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106402 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683695+0000", 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683695+0000", 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683694+0000", 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1589999999999998e-06 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683700+0000", 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7410000000000004e-06 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683709+0000", 2024-02-12T16:49:29.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3750000000000002e-06 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683715+0000", 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0726999999999998e-05 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683756+0000", 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2310000000000004e-06 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.683761+0000", 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.69817597200000003 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381937+0000", 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9210000000000002e-06 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381941+0000", 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.69824561600000001 2024-02-12T16:49:29.772 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106432 8.16 8.75870976 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.724576+0000", 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 509.96047101099998, 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65739000999999997, 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.773 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106432 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724576+0000", 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724576+0000", 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724574+0000", 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2650000000000004e-06 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724579+0000", 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7809999999999994e-06 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724588+0000", 2024-02-12T16:49:29.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6969999999999996e-06 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724595+0000", 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4279000000000001e-05 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724639+0000", 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1219999999999995e-06 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.724647+0000", 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65731423300000003 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381961+0000", 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0840000000000002e-06 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381966+0000", 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65739000999999997 2024-02-12T16:49:29.777 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:106443 8.16 8.4816aff6 (undecoded) ondisk+write+known_if_redirected e883)", 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:59.750934+0000", 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 509.93411208499998, 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63104969, 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:29.778 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 106443 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750934+0000", 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750934+0000", 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:29.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750932+0000", 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.525e-06 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750940+0000", 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6228999999999999e-05 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750966+0000", 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6800000000000005e-06 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.750976+0000", 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6941999999999997e-05 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.751043+0000", 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0635000000000001e-05 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for peered", 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:59.751053+0000", 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63092733499999998 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381981+0000", 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6239999999999999e-06 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:00.381984+0000", 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.63104969 2024-02-12T16:49:29.782 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:267740 9.19 9:9fc78f6e:::benchmark_data_smithi169_124168_object117056:head [delete] snapc 0=[] ondisk+write+known_if_redirected e998)", 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:22.793345+0000", 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 186.891701786, 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20791299299999999, 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:29.783 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267740 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.793345+0000", 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.793345+0000", 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:29.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.793343+0000", 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2810000000000003e-06 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.793350+0000", 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1943000000000001e-05 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.793362+0000", 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8340000000000004e-06 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.793368+0000", 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021685760000000002 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.795537+0000", 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000127157 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.795664+0000", 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025954500000000002 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 11", 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.795923+0000", 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001919259 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.797843+0000", 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20331655300000001 2024-02-12T16:49:29.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:29.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.001159+0000", 2024-02-12T16:49:29.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3869000000000002e-05 2024-02-12T16:49:29.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.001213+0000", 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4855999999999999e-05 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.001258+0000", 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20791299299999999 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:267807 9.9 9:96cbca32:::benchmark_data_smithi169_124168_object117123:head [delete] snapc 0=[] ondisk+write+known_if_redirected e998)", 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:22.838631+0000", 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 186.84641510899999, 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17146666799999999, 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267807 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838631+0000", 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838631+0000", 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838629+0000", 2024-02-12T16:49:29.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0450000000000001e-06 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838637+0000", 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3762e-05 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838651+0000", 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7660000000000004e-06 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838657+0000", 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1298999999999997e-05 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838718+0000", 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013226000000000001 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.838850+0000", 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000588022 2024-02-12T16:49:29.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 11", 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.839438+0000", 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0064881449999999998 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.845927+0000", 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.16408829599999999 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.010015+0000", 2024-02-12T16:49:29.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8019000000000003e-05 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.010053+0000", 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5297999999999999e-05 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.010098+0000", 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17146666799999999 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.795 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:267831 9.19 9:9832a454:::benchmark_data_smithi169_124168_object117147:head [delete] snapc 0=[] ondisk+write+known_if_redirected e998)", 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:22.852454+0000", 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 186.83259217599999, 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.155454904, 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267831 2024-02-12T16:49:29.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852454+0000", 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852454+0000", 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:29.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852452+0000", 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4009999999999997e-06 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852460+0000", 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6286999999999998e-05 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852476+0000", 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5129999999999999e-06 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852484+0000", 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7924999999999999e-05 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852542+0000", 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013925299999999999 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.852681+0000", 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00041838500000000001 2024-02-12T16:49:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 11", 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.853099+0000", 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002902862 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:22.856002+0000", 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15181941199999999 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:29.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.007822+0000", 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5562000000000002e-05 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.007867+0000", 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2049999999999999e-05 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:23.007909+0000", 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.155454904 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.801 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:29.802 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:29.802 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:49:30.429 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-02-12T16:49:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:31 smithi169 bash[19604]: cluster 2024-02-12T16:49:30.387732+0000 mgr.y (mgr.14146) 1959 : cluster [DBG] pgmap v2897: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:31 smithi123 bash[19119]: cluster 2024-02-12T16:49:30.387732+0000 mgr.y (mgr.14146) 1959 : cluster [DBG] pgmap v2897: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:31 smithi121 bash[18489]: cluster 2024-02-12T16:49:30.387732+0000 mgr.y (mgr.14146) 1959 : cluster [DBG] pgmap v2897: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:33.720 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:49:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:33 smithi169 bash[19604]: cluster 2024-02-12T16:49:32.101314+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:33 smithi169 bash[19604]: cluster 2024-02-12T16:49:32.101404+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:33 smithi169 bash[19604]: cluster 2024-02-12T16:49:32.388434+0000 mgr.y (mgr.14146) 1960 : cluster [DBG] pgmap v2898: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:33 smithi169 bash[19604]: cluster 2024-02-12T16:49:32.456871+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e1072: 12 total, 12 up, 12 in 2024-02-12T16:49:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:33 smithi123 bash[19119]: cluster 2024-02-12T16:49:32.101314+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:33 smithi123 bash[19119]: cluster 2024-02-12T16:49:32.101404+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:33 smithi123 bash[19119]: cluster 2024-02-12T16:49:32.388434+0000 mgr.y (mgr.14146) 1960 : cluster [DBG] pgmap v2898: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:33 smithi123 bash[19119]: cluster 2024-02-12T16:49:32.456871+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e1072: 12 total, 12 up, 12 in 2024-02-12T16:49:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:33 smithi121 bash[18489]: cluster 2024-02-12T16:49:32.101314+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:33 smithi121 bash[18489]: cluster 2024-02-12T16:49:32.101404+0000 osd.1 (osd.1) 197 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:33 smithi121 bash[18489]: cluster 2024-02-12T16:49:32.388434+0000 mgr.y (mgr.14146) 1960 : cluster [DBG] pgmap v2898: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:33 smithi121 bash[18489]: cluster 2024-02-12T16:49:32.456871+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e1072: 12 total, 12 up, 12 in 2024-02-12T16:49:35.258 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:49:35.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:35 smithi169 bash[19604]: cluster 2024-02-12T16:49:34.389099+0000 mgr.y (mgr.14146) 1961 : cluster [DBG] pgmap v2900: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:35 smithi123 bash[19119]: cluster 2024-02-12T16:49:34.389099+0000 mgr.y (mgr.14146) 1961 : cluster [DBG] pgmap v2900: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:35 smithi121 bash[18489]: cluster 2024-02-12T16:49:34.389099+0000 mgr.y (mgr.14146) 1961 : cluster [DBG] pgmap v2900: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:37.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:37 smithi123 bash[19119]: cluster 2024-02-12T16:49:36.389755+0000 mgr.y (mgr.14146) 1962 : cluster [DBG] pgmap v2901: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:37.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:37 smithi121 bash[18489]: cluster 2024-02-12T16:49:36.389755+0000 mgr.y (mgr.14146) 1962 : cluster [DBG] pgmap v2901: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:37 smithi169 bash[19604]: cluster 2024-02-12T16:49:36.389755+0000 mgr.y (mgr.14146) 1962 : cluster [DBG] pgmap v2901: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:38.150 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.6 to 1 (10000) 2024-02-12T16:49:38.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:38 smithi123 bash[19119]: audit 2024-02-12T16:49:37.203887+0000 mon.a (mon.0) 3523 : audit [INF] from='client.? v1:172.21.15.121:0/3082034608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:49:38.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:38 smithi121 bash[18489]: audit 2024-02-12T16:49:37.203887+0000 mon.a (mon.0) 3523 : audit [INF] from='client.? v1:172.21.15.121:0/3082034608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:49:38.431 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:38 smithi169 bash[19604]: audit 2024-02-12T16:49:37.203887+0000 mon.a (mon.0) 3523 : audit [INF] from='client.? v1:172.21.15.121:0/3082034608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:49:38.597 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:49:38.598 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:49:38.598 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-02-12T16:49:38.598 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:49:38.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-12T16:49:39.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-02-12T16:49:39.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:39 smithi121 bash[18489]: audit 2024-02-12T16:49:38.139159+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? v1:172.21.15.121:0/3082034608' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:49:39.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:39 smithi121 bash[18489]: cluster 2024-02-12T16:49:38.139281+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e1073: 12 total, 12 up, 12 in 2024-02-12T16:49:39.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:39 smithi121 bash[18489]: cluster 2024-02-12T16:49:38.390385+0000 mgr.y (mgr.14146) 1963 : cluster [DBG] pgmap v2903: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:39 smithi169 bash[19604]: audit 2024-02-12T16:49:38.139159+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? v1:172.21.15.121:0/3082034608' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:49:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:39 smithi169 bash[19604]: cluster 2024-02-12T16:49:38.139281+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e1073: 12 total, 12 up, 12 in 2024-02-12T16:49:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:39 smithi169 bash[19604]: cluster 2024-02-12T16:49:38.390385+0000 mgr.y (mgr.14146) 1963 : cluster [DBG] pgmap v2903: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:39 smithi123 bash[19119]: audit 2024-02-12T16:49:38.139159+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? v1:172.21.15.121:0/3082034608' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:49:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:39 smithi123 bash[19119]: cluster 2024-02-12T16:49:38.139281+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e1073: 12 total, 12 up, 12 in 2024-02-12T16:49:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:39 smithi123 bash[19119]: cluster 2024-02-12T16:49:38.390385+0000 mgr.y (mgr.14146) 1963 : cluster [DBG] pgmap v2903: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:41 smithi169 bash[19604]: cluster 2024-02-12T16:49:40.391124+0000 mgr.y (mgr.14146) 1964 : cluster [DBG] pgmap v2904: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:41 smithi123 bash[19119]: cluster 2024-02-12T16:49:40.391124+0000 mgr.y (mgr.14146) 1964 : cluster [DBG] pgmap v2904: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:41 smithi121 bash[18489]: cluster 2024-02-12T16:49:40.391124+0000 mgr.y (mgr.14146) 1964 : cluster [DBG] pgmap v2904: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:43.706 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:49:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:43 smithi169 bash[19604]: cluster 2024-02-12T16:49:42.391853+0000 mgr.y (mgr.14146) 1965 : cluster [DBG] pgmap v2905: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:43.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:43 smithi123 bash[19119]: cluster 2024-02-12T16:49:42.391853+0000 mgr.y (mgr.14146) 1965 : cluster [DBG] pgmap v2905: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:43 smithi121 bash[18489]: cluster 2024-02-12T16:49:42.391853+0000 mgr.y (mgr.14146) 1965 : cluster [DBG] pgmap v2905: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:44.128 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:49:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:44 smithi169 bash[19604]: cluster 2024-02-12T16:49:43.280438+0000 osd.4 (osd.4) 185 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:49:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:44 smithi169 bash[19604]: cluster 2024-02-12T16:49:43.356017+0000 osd.4 (osd.4) 186 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:49:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:44 smithi169 bash[19604]: cluster 2024-02-12T16:49:43.805574+0000 osd.11 (osd.11) 245 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:44.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:44 smithi169 bash[19604]: cluster 2024-02-12T16:49:43.805667+0000 osd.11 (osd.11) 246 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:44.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:44 smithi121 bash[18489]: cluster 2024-02-12T16:49:43.280438+0000 osd.4 (osd.4) 185 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:49:44.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:44 smithi121 bash[18489]: cluster 2024-02-12T16:49:43.356017+0000 osd.4 (osd.4) 186 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:49:44.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:44 smithi121 bash[18489]: cluster 2024-02-12T16:49:43.805574+0000 osd.11 (osd.11) 245 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:44.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:44 smithi121 bash[18489]: cluster 2024-02-12T16:49:43.805667+0000 osd.11 (osd.11) 246 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:44 smithi123 bash[19119]: cluster 2024-02-12T16:49:43.280438+0000 osd.4 (osd.4) 185 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:49:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:44 smithi123 bash[19119]: cluster 2024-02-12T16:49:43.356017+0000 osd.4 (osd.4) 186 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:49:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:44 smithi123 bash[19119]: cluster 2024-02-12T16:49:43.805574+0000 osd.11 (osd.11) 245 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:44.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:44 smithi123 bash[19119]: cluster 2024-02-12T16:49:43.805667+0000 osd.11 (osd.11) 246 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:45.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:45 smithi169 bash[19604]: cluster 2024-02-12T16:49:44.392574+0000 mgr.y (mgr.14146) 1966 : cluster [DBG] pgmap v2906: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:45.620 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:45 smithi169 bash[19604]: cluster 2024-02-12T16:49:44.467247+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e1074: 12 total, 12 up, 12 in 2024-02-12T16:49:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:45 smithi123 bash[19119]: cluster 2024-02-12T16:49:44.392574+0000 mgr.y (mgr.14146) 1966 : cluster [DBG] pgmap v2906: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:45 smithi123 bash[19119]: cluster 2024-02-12T16:49:44.467247+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e1074: 12 total, 12 up, 12 in 2024-02-12T16:49:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:45 smithi121 bash[18489]: cluster 2024-02-12T16:49:44.392574+0000 mgr.y (mgr.14146) 1966 : cluster [DBG] pgmap v2906: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:45 smithi121 bash[18489]: cluster 2024-02-12T16:49:44.467247+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e1074: 12 total, 12 up, 12 in 2024-02-12T16:49:47.372 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:49:47.373 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-02-12T16:49:47.373 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-02-12T16:49:47.373 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-02-12T16:49:47.373 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:49:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:47 smithi123 bash[19119]: cluster 2024-02-12T16:49:45.723123+0000 osd.9 (osd.9) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:47 smithi123 bash[19119]: cluster 2024-02-12T16:49:45.723187+0000 osd.9 (osd.9) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:47 smithi123 bash[19119]: cluster 2024-02-12T16:49:46.072218+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.d scrub starts 2024-02-12T16:49:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:47 smithi123 bash[19119]: cluster 2024-02-12T16:49:46.169366+0000 osd.8 (osd.8) 167 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:49:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:47 smithi123 bash[19119]: cluster 2024-02-12T16:49:46.186616+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.d scrub ok 2024-02-12T16:49:47.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:47 smithi123 bash[19119]: cluster 2024-02-12T16:49:46.393321+0000 mgr.y (mgr.14146) 1967 : cluster [DBG] pgmap v2908: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:47.390 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:47 smithi123 bash[19119]: cluster 2024-02-12T16:49:46.472571+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e1075: 12 total, 12 up, 12 in 2024-02-12T16:49:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:47 smithi121 bash[18489]: cluster 2024-02-12T16:49:45.723123+0000 osd.9 (osd.9) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:47 smithi121 bash[18489]: cluster 2024-02-12T16:49:45.723187+0000 osd.9 (osd.9) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:47 smithi121 bash[18489]: cluster 2024-02-12T16:49:46.072218+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.d scrub starts 2024-02-12T16:49:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:47 smithi121 bash[18489]: cluster 2024-02-12T16:49:46.169366+0000 osd.8 (osd.8) 167 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:49:47.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:47 smithi121 bash[18489]: cluster 2024-02-12T16:49:46.186616+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.d scrub ok 2024-02-12T16:49:47.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:47 smithi121 bash[18489]: cluster 2024-02-12T16:49:46.393321+0000 mgr.y (mgr.14146) 1967 : cluster [DBG] pgmap v2908: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:47.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:47 smithi121 bash[18489]: cluster 2024-02-12T16:49:46.472571+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e1075: 12 total, 12 up, 12 in 2024-02-12T16:49:47.428 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:47 smithi169 bash[19604]: cluster 2024-02-12T16:49:45.723123+0000 osd.9 (osd.9) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:47.428 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:47 smithi169 bash[19604]: cluster 2024-02-12T16:49:45.723187+0000 osd.9 (osd.9) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:47.428 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:47 smithi169 bash[19604]: cluster 2024-02-12T16:49:46.072218+0000 osd.7 (osd.7) 41 : cluster [DBG] 1.d scrub starts 2024-02-12T16:49:47.428 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:47 smithi169 bash[19604]: cluster 2024-02-12T16:49:46.169366+0000 osd.8 (osd.8) 167 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:49:47.428 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:47 smithi169 bash[19604]: cluster 2024-02-12T16:49:46.186616+0000 osd.7 (osd.7) 42 : cluster [DBG] 1.d scrub ok 2024-02-12T16:49:47.428 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:47 smithi169 bash[19604]: cluster 2024-02-12T16:49:46.393321+0000 mgr.y (mgr.14146) 1967 : cluster [DBG] pgmap v2908: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:47.428 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:47 smithi169 bash[19604]: cluster 2024-02-12T16:49:46.472571+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e1075: 12 total, 12 up, 12 in 2024-02-12T16:49:48.098 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-02-12T16:49:48.146 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-12T16:49:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:48 smithi123 bash[19119]: cluster 2024-02-12T16:49:46.288810+0000 osd.8 (osd.8) 168 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:49:48.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:48 smithi123 bash[19119]: audit 2024-02-12T16:49:47.225578+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? v1:172.21.15.121:0/3273371735' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:49:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:48 smithi121 bash[18489]: cluster 2024-02-12T16:49:46.288810+0000 osd.8 (osd.8) 168 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:49:48.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:48 smithi121 bash[18489]: audit 2024-02-12T16:49:47.225578+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? v1:172.21.15.121:0/3273371735' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:49:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:48 smithi169 bash[19604]: cluster 2024-02-12T16:49:46.288810+0000 osd.8 (osd.8) 168 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:49:48.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:48 smithi169 bash[19604]: audit 2024-02-12T16:49:47.225578+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? v1:172.21.15.121:0/3273371735' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-12T16:49:48.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 8 1 2024-02-12T16:49:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:49 smithi121 bash[18489]: cluster 2024-02-12T16:49:47.517028+0000 osd.11 (osd.11) 247 : cluster [DBG] 1.b scrub starts 2024-02-12T16:49:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:49 smithi121 bash[18489]: cluster 2024-02-12T16:49:47.688787+0000 osd.11 (osd.11) 248 : cluster [DBG] 1.b scrub ok 2024-02-12T16:49:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:49 smithi121 bash[18489]: audit 2024-02-12T16:49:48.136143+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? v1:172.21.15.121:0/3273371735' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:49:49.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:49 smithi121 bash[18489]: cluster 2024-02-12T16:49:48.136237+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e1076: 12 total, 12 up, 12 in 2024-02-12T16:49:49.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:49 smithi121 bash[18489]: cluster 2024-02-12T16:49:48.394012+0000 mgr.y (mgr.14146) 1968 : cluster [DBG] pgmap v2911: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:49 smithi169 bash[19604]: cluster 2024-02-12T16:49:47.517028+0000 osd.11 (osd.11) 247 : cluster [DBG] 1.b scrub starts 2024-02-12T16:49:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:49 smithi169 bash[19604]: cluster 2024-02-12T16:49:47.688787+0000 osd.11 (osd.11) 248 : cluster [DBG] 1.b scrub ok 2024-02-12T16:49:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:49 smithi169 bash[19604]: audit 2024-02-12T16:49:48.136143+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? v1:172.21.15.121:0/3273371735' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:49:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:49 smithi169 bash[19604]: cluster 2024-02-12T16:49:48.136237+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e1076: 12 total, 12 up, 12 in 2024-02-12T16:49:49.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:49 smithi169 bash[19604]: cluster 2024-02-12T16:49:48.394012+0000 mgr.y (mgr.14146) 1968 : cluster [DBG] pgmap v2911: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:49.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:49 smithi123 bash[19119]: cluster 2024-02-12T16:49:47.517028+0000 osd.11 (osd.11) 247 : cluster [DBG] 1.b scrub starts 2024-02-12T16:49:49.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:49 smithi123 bash[19119]: cluster 2024-02-12T16:49:47.688787+0000 osd.11 (osd.11) 248 : cluster [DBG] 1.b scrub ok 2024-02-12T16:49:49.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:49 smithi123 bash[19119]: audit 2024-02-12T16:49:48.136143+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? v1:172.21.15.121:0/3273371735' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-12T16:49:49.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:49 smithi123 bash[19119]: cluster 2024-02-12T16:49:48.136237+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e1076: 12 total, 12 up, 12 in 2024-02-12T16:49:49.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:49 smithi123 bash[19119]: cluster 2024-02-12T16:49:48.394012+0000 mgr.y (mgr.14146) 1968 : cluster [DBG] pgmap v2911: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:51 smithi169 bash[19604]: cluster 2024-02-12T16:49:49.887666+0000 osd.5 (osd.5) 245 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:51 smithi169 bash[19604]: cluster 2024-02-12T16:49:49.887794+0000 osd.5 (osd.5) 246 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:51 smithi169 bash[19604]: cluster 2024-02-12T16:49:50.150508+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e1077: 12 total, 12 up, 12 in 2024-02-12T16:49:51.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:51 smithi169 bash[19604]: cluster 2024-02-12T16:49:50.394610+0000 mgr.y (mgr.14146) 1969 : cluster [DBG] pgmap v2913: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:51 smithi123 bash[19119]: cluster 2024-02-12T16:49:49.887666+0000 osd.5 (osd.5) 245 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:51 smithi123 bash[19119]: cluster 2024-02-12T16:49:49.887794+0000 osd.5 (osd.5) 246 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:51 smithi123 bash[19119]: cluster 2024-02-12T16:49:50.150508+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e1077: 12 total, 12 up, 12 in 2024-02-12T16:49:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:51 smithi123 bash[19119]: cluster 2024-02-12T16:49:50.394610+0000 mgr.y (mgr.14146) 1969 : cluster [DBG] pgmap v2913: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:51 smithi121 bash[18489]: cluster 2024-02-12T16:49:49.887666+0000 osd.5 (osd.5) 245 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:51 smithi121 bash[18489]: cluster 2024-02-12T16:49:49.887794+0000 osd.5 (osd.5) 246 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:51 smithi121 bash[18489]: cluster 2024-02-12T16:49:50.150508+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e1077: 12 total, 12 up, 12 in 2024-02-12T16:49:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:51 smithi121 bash[18489]: cluster 2024-02-12T16:49:50.394610+0000 mgr.y (mgr.14146) 1969 : cluster [DBG] pgmap v2913: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:50.444295+0000 osd.11 (osd.11) 249 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:49:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:50.468682+0000 osd.11 (osd.11) 250 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:49:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:50.718676+0000 osd.5 (osd.5) 247 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:49:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:50.800073+0000 osd.5 (osd.5) 248 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:49:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:50.982843+0000 osd.6 (osd.6) 204 : cluster [DBG] 1.f scrub starts 2024-02-12T16:49:52.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:51.085078+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:51.085147+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:51.129878+0000 osd.6 (osd.6) 205 : cluster [DBG] 1.f scrub ok 2024-02-12T16:49:52.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:52 smithi169 bash[19604]: cluster 2024-02-12T16:49:51.179439+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e1078: 12 total, 12 up, 12 in 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:50.444295+0000 osd.11 (osd.11) 249 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:50.468682+0000 osd.11 (osd.11) 250 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:50.718676+0000 osd.5 (osd.5) 247 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:50.800073+0000 osd.5 (osd.5) 248 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:50.982843+0000 osd.6 (osd.6) 204 : cluster [DBG] 1.f scrub starts 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:51.085078+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:51.085147+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:52.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:51.129878+0000 osd.6 (osd.6) 205 : cluster [DBG] 1.f scrub ok 2024-02-12T16:49:52.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:52 smithi123 bash[19119]: cluster 2024-02-12T16:49:51.179439+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e1078: 12 total, 12 up, 12 in 2024-02-12T16:49:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:50.444295+0000 osd.11 (osd.11) 249 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:49:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:50.468682+0000 osd.11 (osd.11) 250 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:49:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:50.718676+0000 osd.5 (osd.5) 247 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:49:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:50.800073+0000 osd.5 (osd.5) 248 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:49:52.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:50.982843+0000 osd.6 (osd.6) 204 : cluster [DBG] 1.f scrub starts 2024-02-12T16:49:52.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:51.085078+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:49:52.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:51.085147+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:49:52.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:51.129878+0000 osd.6 (osd.6) 205 : cluster [DBG] 1.f scrub ok 2024-02-12T16:49:52.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:52 smithi121 bash[18489]: cluster 2024-02-12T16:49:51.179439+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e1078: 12 total, 12 up, 12 in 2024-02-12T16:49:52.885 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.c/config 2024-02-12T16:49:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:53 smithi169 bash[19604]: cluster 2024-02-12T16:49:52.395336+0000 mgr.y (mgr.14146) 1970 : cluster [DBG] pgmap v2915: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:53.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:53 smithi169 bash[19604]: cluster 2024-02-12T16:49:52.758747+0000 osd.1 (osd.1) 198 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:49:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:53 smithi123 bash[19119]: cluster 2024-02-12T16:49:52.395336+0000 mgr.y (mgr.14146) 1970 : cluster [DBG] pgmap v2915: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:53.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:53 smithi123 bash[19119]: cluster 2024-02-12T16:49:52.758747+0000 osd.1 (osd.1) 198 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:49:53.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:53 smithi121 bash[18489]: cluster 2024-02-12T16:49:52.395336+0000 mgr.y (mgr.14146) 1970 : cluster [DBG] pgmap v2915: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:53.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:53 smithi121 bash[18489]: cluster 2024-02-12T16:49:52.758747+0000 osd.1 (osd.1) 198 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:49:53.747 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:49:54.367 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:54 smithi169 bash[19604]: cluster 2024-02-12T16:49:52.972559+0000 osd.1 (osd.1) 199 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:49:54.368 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:54 smithi169 bash[19604]: cluster 2024-02-12T16:49:53.168888+0000 osd.8 (osd.8) 169 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:49:54.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:54 smithi123 bash[19119]: cluster 2024-02-12T16:49:52.972559+0000 osd.1 (osd.1) 199 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:49:54.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:54 smithi123 bash[19119]: cluster 2024-02-12T16:49:53.168888+0000 osd.8 (osd.8) 169 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:49:54.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:54 smithi121 bash[18489]: cluster 2024-02-12T16:49:52.972559+0000 osd.1 (osd.1) 199 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:49:54.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:54 smithi121 bash[18489]: cluster 2024-02-12T16:49:53.168888+0000 osd.8 (osd.8) 169 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:49:55.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:55 smithi121 bash[18489]: cluster 2024-02-12T16:49:53.238211+0000 osd.8 (osd.8) 170 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:49:55.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:55 smithi121 bash[18489]: cluster 2024-02-12T16:49:53.836581+0000 osd.10 (osd.10) 268 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:49:55.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:55 smithi121 bash[18489]: cluster 2024-02-12T16:49:53.930733+0000 osd.10 (osd.10) 269 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:49:55.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:55 smithi121 bash[18489]: cluster 2024-02-12T16:49:54.396033+0000 mgr.y (mgr.14146) 1971 : cluster [DBG] pgmap v2916: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:55.361 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:55 smithi121 bash[18489]: cluster 2024-02-12T16:49:54.561175+0000 osd.0 (osd.0) 177 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:49:55.361 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:55 smithi121 bash[18489]: cluster 2024-02-12T16:49:54.674777+0000 osd.0 (osd.0) 178 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:49:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:55 smithi169 bash[19604]: cluster 2024-02-12T16:49:53.238211+0000 osd.8 (osd.8) 170 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:49:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:55 smithi169 bash[19604]: cluster 2024-02-12T16:49:53.836581+0000 osd.10 (osd.10) 268 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:49:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:55 smithi169 bash[19604]: cluster 2024-02-12T16:49:53.930733+0000 osd.10 (osd.10) 269 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:49:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:55 smithi169 bash[19604]: cluster 2024-02-12T16:49:54.396033+0000 mgr.y (mgr.14146) 1971 : cluster [DBG] pgmap v2916: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:55 smithi169 bash[19604]: cluster 2024-02-12T16:49:54.561175+0000 osd.0 (osd.0) 177 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:49:55.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:55 smithi169 bash[19604]: cluster 2024-02-12T16:49:54.674777+0000 osd.0 (osd.0) 178 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:49:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:55 smithi123 bash[19119]: cluster 2024-02-12T16:49:53.238211+0000 osd.8 (osd.8) 170 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:49:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:55 smithi123 bash[19119]: cluster 2024-02-12T16:49:53.836581+0000 osd.10 (osd.10) 268 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:49:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:55 smithi123 bash[19119]: cluster 2024-02-12T16:49:53.930733+0000 osd.10 (osd.10) 269 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:49:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:55 smithi123 bash[19119]: cluster 2024-02-12T16:49:54.396033+0000 mgr.y (mgr.14146) 1971 : cluster [DBG] pgmap v2916: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:55 smithi123 bash[19119]: cluster 2024-02-12T16:49:54.561175+0000 osd.0 (osd.0) 177 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:49:55.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:55 smithi123 bash[19119]: cluster 2024-02-12T16:49:54.674777+0000 osd.0 (osd.0) 178 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:49:56.282 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-02-12T16:49:56.282 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59811 8.d 8:ba970246:::benchmark_data_smithi169_115677_object59810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.823440+0000", 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 563.45218875800003, 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34065904400000002, 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.283 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59811 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823440+0000", 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823440+0000", 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.284 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823439+0000", 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3519999999999998e-05 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823462+0000", 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0699e-05 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823473+0000", 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1129999999999997e-06 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.285 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.823479+0000", 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024674580000000001 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.825947+0000", 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00023116099999999999 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.826178+0000", 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000178221 2024-02-12T16:49:56.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.826356+0000", 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006138274 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.832494+0000", 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33121484800000001 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.163709+0000", 2024-02-12T16:49:56.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0861000000000001e-05 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.163750+0000", 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034960299999999998 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.164100+0000", 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.34065904400000002 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59822 8.d 8:bd106337:::benchmark_data_smithi169_115677_object59821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.828921+0000", 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 563.44670777399995, 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33677383700000002, 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59822 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.289 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828921+0000", 2024-02-12T16:49:56.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828921+0000", 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828919+0000", 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3802999999999999e-05 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828943+0000", 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.521e-05 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828958+0000", 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5080000000000002e-06 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.828965+0000", 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00116552 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830130+0000", 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013362500000000001 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830264+0000", 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000191225 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.830455+0000", 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034418320000000001 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.833897+0000", 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33036304500000002 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.164260+0000", 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051311100000000001 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.164773+0000", 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00092227200000000002 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.165695+0000", 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.33677383700000002 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59837 8.d 8:bce42dd4:::benchmark_data_smithi169_115677_object59836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.836326+0000", 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 563.43930280799998, 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32974853500000001, 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.295 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59837 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836326+0000", 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836326+0000", 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836324+0000", 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1529000000000003e-05 2024-02-12T16:49:56.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836356+0000", 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3139000000000001e-05 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836369+0000", 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2120000000000002e-06 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836374+0000", 2024-02-12T16:49:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9558999999999998e-05 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836423+0000", 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015596699999999999 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836579+0000", 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000199817 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.836779+0000", 2024-02-12T16:49:56.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0019116210000000001 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.838691+0000", 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32708284300000001 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.165774+0000", 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9215000000000001e-05 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.165833+0000", 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00024211 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166075+0000", 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.32974853500000001 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59858 8.d 8:b9b6efa3:::benchmark_data_smithi169_115677_object59857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.851078+0000", 2024-02-12T16:49:56.301 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 563.42455113200003, 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.315281547, 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59858 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851078+0000", 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851078+0000", 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851076+0000", 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1844e-05 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851098+0000", 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3205e-05 2024-02-12T16:49:56.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851111+0000", 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7649999999999993e-06 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851119+0000", 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5439000000000003e-05 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851164+0000", 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000108528 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851273+0000", 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000129102 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.851402+0000", 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034160670000000001 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.854818+0000", 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31130226599999999 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166120+0000", 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3087999999999998e-05 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166163+0000", 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019629000000000001 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166360+0000", 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.315281547 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:59860 8.d 8:b39a1c77:::benchmark_data_smithi169_115677_object59859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:32.852616+0000", 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 563.42301294000004, 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31403088899999998, 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.307 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 59860 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852616+0000", 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852616+0000", 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852615+0000", 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9853000000000001e-05 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852635+0000", 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0856000000000001e-05 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852646+0000", 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7729999999999999e-06 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.852650+0000", 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00041727000000000002 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.853068+0000", 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00016523699999999999 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.853233+0000", 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000208827 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.853442+0000", 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001487012 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:32.854929+0000", 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.311478274 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166407+0000", 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7182999999999999e-05 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166454+0000", 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019307600000000001 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:33.166647+0000", 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.31403088899999998 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67828 8.d 8:bbb16db0:::benchmark_data_smithi169_115677_object67827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.312 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.702612+0000", 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 559.57301759699999, 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24922882399999999, 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67828 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702612+0000", 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702612+0000", 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702609+0000", 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2708e-05 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702632+0000", 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2493999999999999e-05 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702644+0000", 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3149999999999999e-06 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702650+0000", 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2055000000000003e-05 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702692+0000", 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101516 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702793+0000", 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000113244 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.702906+0000", 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004734266 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.707641+0000", 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24380940600000001 2024-02-12T16:49:56.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951450+0000", 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6556000000000003e-05 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951537+0000", 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030377900000000001 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951840+0000", 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24922882399999999 2024-02-12T16:49:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67829 8.d 8:b4a23540:::benchmark_data_smithi169_115677_object67828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.704059+0000", 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 559.57157045999998, 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24810375800000001, 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67829 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.318 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704059+0000", 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704059+0000", 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704057+0000", 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0411000000000002e-05 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704078+0000", 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1507e-05 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704089+0000", 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7840000000000003e-06 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704094+0000", 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00087896600000000001 2024-02-12T16:49:56.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.704973+0000", 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000179872 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.705153+0000", 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018307100000000001 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.705336+0000", 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002813701 2024-02-12T16:49:56.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.708149+0000", 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24374633600000001 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951896+0000", 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9889999999999997e-05 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.951966+0000", 2024-02-12T16:49:56.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019684999999999999 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952163+0000", 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24810375800000001 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67857 8.d 8:b50e86ce:::benchmark_data_smithi169_115677_object67856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.716963+0000", 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 559.55866648300002, 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23546748300000001, 2024-02-12T16:49:56.323 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67857 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716963+0000", 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716963+0000", 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716960+0000", 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9241999999999999e-05 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.716989+0000", 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0531999999999999e-05 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717010+0000", 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717018+0000", 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0773999999999998e-05 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717079+0000", 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017184899999999999 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717251+0000", 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019772499999999999 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.717449+0000", 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026654080000000002 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.720114+0000", 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23208046700000001 2024-02-12T16:49:56.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952194+0000", 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8588999999999999e-05 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952233+0000", 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00019727699999999999 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.952430+0000", 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.23546748300000001 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67883 8.d 8:b86b9654:::benchmark_data_smithi169_115677_object67882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.730676+0000", 2024-02-12T16:49:56.329 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 559.54495342300004, 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.223248011, 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67883 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730676+0000", 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730676+0000", 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999967 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730672+0000", 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9053000000000001e-05 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730702+0000", 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6235e-05 2024-02-12T16:49:56.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730718+0000", 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7000000000000008e-06 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730725+0000", 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9512999999999998e-05 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730785+0000", 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015649500000000001 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.730941+0000", 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000138787 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.731080+0000", 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021907390000000001 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.733271+0000", 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.22032115799999999 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953592+0000", 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2156e-05 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953644+0000", 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00027951300000000001 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953924+0000", 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.223248011 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26785.0:67899 8.d 8:b3bc5adf:::benchmark_data_smithi169_115677_object67898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e874)", 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:40:36.756677+0000", 2024-02-12T16:49:56.335 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 559.51895185800004, 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197548369, 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26785", 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 67899 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756677+0000", 2024-02-12T16:49:56.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756677+0000", 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999971 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756674+0000", 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5304999999999999e-05 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756700+0000", 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5061e-05 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756715+0000", 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9519999999999999e-06 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756721+0000", 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5686000000000002e-05 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756776+0000", 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015034 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.756927+0000", 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017099899999999999 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 2", 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.757098+0000", 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021726800000000002 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.759270+0000", 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19469667800000001 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.953967+0000", 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1038000000000002e-05 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.954018+0000", 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000207672 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:40:36.954226+0000", 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.197548369 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 898/889 [PushOp(8:e9d21599:::benchmark_data_smithi169_115677_object70736:head, version: 876'4346, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e9d21599:::benchmark_data_smithi169_115677_object70736:head@876'4346, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:35.267146+0000", 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 501.00848340499999, 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19494173300000001, 2024-02-12T16:49:56.341 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267146+0000", 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267146+0000", 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267145+0000", 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0874000000000001e-05 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267196+0000", 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7368999999999999e-05 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267213+0000", 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1689999999999998e-06 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.267222+0000", 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18740217200000001 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.454625+0000", 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3699999999999995e-06 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.454633+0000", 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074546960000000002 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.462088+0000", 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.19494173300000001 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.7 898/889 [PushOp(8:e69a8233:::benchmark_data_smithi169_115677_object70702:head, version: 876'4345, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e69a8233:::benchmark_data_smithi169_115677_object70702:head@876'4345, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:35.279424+0000", 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 500.996205449, 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18170157200000001, 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279424+0000", 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279424+0000", 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279423+0000", 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4529000000000001e-05 2024-02-12T16:49:56.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279457+0000", 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7051999999999999e-05 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279474+0000", 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7479999999999993e-06 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.279482+0000", 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.179421944 2024-02-12T16:49:56.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.458904+0000", 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7399999999999998e-06 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.458911+0000", 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022147529999999999 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.461125+0000", 2024-02-12T16:49:56.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18170157200000001 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 898/889 [PushOp(8:e94d15ac:::benchmark_data_smithi169_115677_object70790:head, version: 876'4350, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e94d15ac:::benchmark_data_smithi169_115677_object70790:head@876'4350, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:35.291573+0000", 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 500.98405673500002, 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17258230399999999, 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291573+0000", 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291573+0000", 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291571+0000", 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000003e-05 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291607+0000", 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3944999999999999e-05 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291621+0000", 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5499999999999995e-06 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.291630+0000", 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17034442399999999 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.461974+0000", 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.216e-06 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.461981+0000", 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002173473 2024-02-12T16:49:56.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:35.464155+0000", 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.17258230399999999 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 899/889 [PushOp(8:e9f88ef0:::benchmark_data_smithi169_115677_object75347:head, version: 877'4655, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e9f88ef0:::benchmark_data_smithi169_115677_object75347:head@877'4655, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:39.921862+0000", 2024-02-12T16:49:56.353 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 496.35376719800001, 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18364805100000001, 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921862+0000", 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921862+0000", 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.999999 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921861+0000", 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4548000000000006e-05 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921925+0000", 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6736999999999999e-05 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921942+0000", 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.166e-05 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:39.921954+0000", 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.181234334 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:40.103188+0000", 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9610000000000005e-06 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:40.103196+0000", 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002313973 2024-02-12T16:49:56.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:40.105510+0000", 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.18364805100000001 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDPGPush(8.17 900/889 [PushOp(8:eeb3a8f1:::benchmark_data_smithi169_115677_object76694:head, version: 878'4743, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:eeb3a8f1:::benchmark_data_smithi169_115677_object76694:head@878'4743, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-12T16:49:56.357 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:41:41.135011+0000", 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 495.14061819300002, 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21948980300000001, 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135011+0000", 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135011+0000", 2024-02-12T16:49:56.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135010+0000", 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8582000000000001e-05 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135088+0000", 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4621e-05 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135113+0000", 2024-02-12T16:49:56.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3378e-05 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.135126+0000", 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.20962963100000001 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.344756+0000", 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1490000000000002e-06 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.344761+0000", 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0097400060000000007 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:41:41.354501+0000", 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.21948980300000001 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:291100 9.8 9:1568bc1a:::benchmark_data_smithi169_124168_object140416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1013)", 2024-02-12T16:49:56.361 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:46.053249+0000", 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 190.22238005700001, 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028123028000000001, 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 291100 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.053249+0000", 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.053249+0000", 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.053248+0000", 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8290000000000002e-06 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.053254+0000", 2024-02-12T16:49:56.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0332e-05 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.053265+0000", 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5909999999999998e-06 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.053269+0000", 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027179249999999999 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.055987+0000", 2024-02-12T16:49:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000128582 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.056116+0000", 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00060458300000000001 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.056720+0000", 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.018122539999999999 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.074843+0000", 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0064466000000000002 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.081290+0000", 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3544999999999997e-05 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.081333+0000", 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9174999999999997e-05 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:46.081372+0000", 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028123028000000001 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:295397 9.5 9:a49ba2ad:::benchmark_data_smithi169_124168_object144713:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1014)", 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:48.905218+0000", 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 187.370410961, 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027997483, 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.367 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 295397 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905218+0000", 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905218+0000", 2024-02-12T16:49:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905217+0000", 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9579999999999998e-06 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905222+0000", 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0027000000000001e-05 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905232+0000", 2024-02-12T16:49:56.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6199999999999998e-06 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905237+0000", 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0156e-05 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905277+0000", 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6459999999999996e-05 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905363+0000", 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00047765399999999999 2024-02-12T16:49:56.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.905841+0000", 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002300369 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.908141+0000", 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024991121000000002 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.933132+0000", 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9332000000000002e-05 2024-02-12T16:49:56.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.933172+0000", 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.409e-05 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:48.933216+0000", 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027997483 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:296399 9.8 9:1188e885:::benchmark_data_smithi169_124168_object145715:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1014)", 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:49.587177+0000", 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 186.68845202099999, 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028411342999999999, 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 296399 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.587177+0000", 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.587177+0000", 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999986 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.587176+0000", 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5739999999999997e-06 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.587181+0000", 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6752000000000002e-05 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.587197+0000", 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0975000000000001e-05 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.587208+0000", 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004987465 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.592196+0000", 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020231200000000001 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.592398+0000", 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00063250399999999999 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.593031+0000", 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0032439909999999999 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.596275+0000", 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.019220706000000001 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.615495+0000", 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9957000000000003e-05 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.615545+0000", 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331000000000002e-05 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:49.615589+0000", 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028411342999999999 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:298225 9.18 9:1a42559d:::benchmark_data_smithi169_124168_object147541:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:52.427474+0000", 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 183.848155185, 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031778733000000003, 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.378 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 298225 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427474+0000", 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427474+0000", 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999981 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427472+0000", 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.798e-06 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427479+0000", 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8166000000000001e-05 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427507+0000", 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1029999999999996e-06 2024-02-12T16:49:56.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427514+0000", 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5741000000000002e-05 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427550+0000", 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4529000000000002e-05 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.427644+0000", 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00044835699999999998 2024-02-12T16:49:56.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.428093+0000", 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033595000000000001 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.431452+0000", 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02773726 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.459189+0000", 2024-02-12T16:49:56.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6894999999999999e-05 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.459216+0000", 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6501000000000002e-05 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.459253+0000", 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031778733000000003 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.26977.0:298228 9.13 9:ca83f99b:::benchmark_data_smithi169_124168_object147544:head [delete] snapc 0=[] ondisk+write+known_if_redirected e1017)", 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-02-12T16:46:52.431014+0000", 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 183.84461557399999, 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028201502999999999, 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.26977", 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 298228 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-02-12T16:49:56.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.431014+0000", 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.431014+0000", 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4294967295.9999976 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.431012+0000", 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6799999999999993e-06 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.431019+0000", 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1389000000000001e-05 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.431041+0000", 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9260000000000003e-06 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.431048+0000", 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015977210000000001 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.432645+0000", 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013436500000000001 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.432780+0000", 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049448700000000005 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for subops from 5", 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.433274+0000", 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025268719999999999 2024-02-12T16:49:56.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op_commit", 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.435801+0000", 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023329560999999999 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.459131+0000", 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1103999999999997e-05 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.459182+0000", 2024-02-12T16:49:56.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3553999999999998e-05 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-02-12T16:46:52.459215+0000", 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028201502999999999 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-02-12T16:49:56.389 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-02-12T16:49:56.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-12T16:49:57.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:57 smithi169 bash[19604]: cluster 2024-02-12T16:49:56.396689+0000 mgr.y (mgr.14146) 1972 : cluster [DBG] pgmap v2917: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:57.389 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:57 smithi123 bash[19119]: cluster 2024-02-12T16:49:56.396689+0000 mgr.y (mgr.14146) 1972 : cluster [DBG] pgmap v2917: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:57.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:57 smithi121 bash[18489]: cluster 2024-02-12T16:49:56.396689+0000 mgr.y (mgr.14146) 1972 : cluster [DBG] pgmap v2917: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:58.152 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.8 to 1 (10000) 2024-02-12T16:49:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:58 smithi121 bash[18489]: cluster 2024-02-12T16:49:56.827555+0000 osd.10 (osd.10) 270 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:49:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:58 smithi121 bash[18489]: cluster 2024-02-12T16:49:56.926855+0000 osd.10 (osd.10) 271 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:49:58.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:58 smithi121 bash[18489]: audit 2024-02-12T16:49:57.354697+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? v1:172.21.15.121:0/1414011833' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:49:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:58 smithi169 bash[19604]: cluster 2024-02-12T16:49:56.827555+0000 osd.10 (osd.10) 270 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:49:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:58 smithi169 bash[19604]: cluster 2024-02-12T16:49:56.926855+0000 osd.10 (osd.10) 271 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:49:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:58 smithi169 bash[19604]: audit 2024-02-12T16:49:57.354697+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? v1:172.21.15.121:0/1414011833' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:49:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:58 smithi123 bash[19119]: cluster 2024-02-12T16:49:56.827555+0000 osd.10 (osd.10) 270 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:49:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:58 smithi123 bash[19119]: cluster 2024-02-12T16:49:56.926855+0000 osd.10 (osd.10) 271 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:49:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:58 smithi123 bash[19119]: audit 2024-02-12T16:49:57.354697+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? v1:172.21.15.121:0/1414011833' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:49:58.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 8 1 2024-02-12T16:49:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:59 smithi121 bash[18489]: audit 2024-02-12T16:49:58.139944+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? v1:172.21.15.121:0/1414011833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:49:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:59 smithi121 bash[18489]: cluster 2024-02-12T16:49:58.140040+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e1079: 12 total, 12 up, 12 in 2024-02-12T16:49:59.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:49:59 smithi121 bash[18489]: cluster 2024-02-12T16:49:58.397337+0000 mgr.y (mgr.14146) 1973 : cluster [DBG] pgmap v2919: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:59 smithi169 bash[19604]: audit 2024-02-12T16:49:58.139944+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? v1:172.21.15.121:0/1414011833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:49:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:59 smithi169 bash[19604]: cluster 2024-02-12T16:49:58.140040+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e1079: 12 total, 12 up, 12 in 2024-02-12T16:49:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:49:59 smithi169 bash[19604]: cluster 2024-02-12T16:49:58.397337+0000 mgr.y (mgr.14146) 1973 : cluster [DBG] pgmap v2919: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:49:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:59 smithi123 bash[19119]: audit 2024-02-12T16:49:58.139944+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? v1:172.21.15.121:0/1414011833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:49:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:59 smithi123 bash[19119]: cluster 2024-02-12T16:49:58.140040+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e1079: 12 total, 12 up, 12 in 2024-02-12T16:49:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:49:59 smithi123 bash[19119]: cluster 2024-02-12T16:49:58.397337+0000 mgr.y (mgr.14146) 1973 : cluster [DBG] pgmap v2919: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:00 smithi169 bash[19604]: cluster 2024-02-12T16:49:58.881885+0000 osd.10 (osd.10) 272 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:50:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:00 smithi169 bash[19604]: cluster 2024-02-12T16:49:59.013628+0000 osd.10 (osd.10) 273 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:50:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:00 smithi169 bash[19604]: cluster 2024-02-12T16:50:00.000185+0000 mon.a (mon.0) 3536 : cluster [INF] overall HEALTH_OK 2024-02-12T16:50:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:00 smithi123 bash[19119]: cluster 2024-02-12T16:49:58.881885+0000 osd.10 (osd.10) 272 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:50:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:00 smithi123 bash[19119]: cluster 2024-02-12T16:49:59.013628+0000 osd.10 (osd.10) 273 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:50:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:00 smithi123 bash[19119]: cluster 2024-02-12T16:50:00.000185+0000 mon.a (mon.0) 3536 : cluster [INF] overall HEALTH_OK 2024-02-12T16:50:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:00 smithi121 bash[18489]: cluster 2024-02-12T16:49:58.881885+0000 osd.10 (osd.10) 272 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:50:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:00 smithi121 bash[18489]: cluster 2024-02-12T16:49:59.013628+0000 osd.10 (osd.10) 273 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:50:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:00 smithi121 bash[18489]: cluster 2024-02-12T16:50:00.000185+0000 mon.a (mon.0) 3536 : cluster [INF] overall HEALTH_OK 2024-02-12T16:50:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:01 smithi169 bash[19604]: cluster 2024-02-12T16:50:00.097307+0000 osd.7 (osd.7) 43 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:50:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:01 smithi169 bash[19604]: cluster 2024-02-12T16:50:00.154222+0000 osd.7 (osd.7) 44 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:50:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:01 smithi169 bash[19604]: cluster 2024-02-12T16:50:00.398100+0000 mgr.y (mgr.14146) 1974 : cluster [DBG] pgmap v2920: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:01 smithi123 bash[19119]: cluster 2024-02-12T16:50:00.097307+0000 osd.7 (osd.7) 43 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:50:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:01 smithi123 bash[19119]: cluster 2024-02-12T16:50:00.154222+0000 osd.7 (osd.7) 44 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:50:01.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:01 smithi123 bash[19119]: cluster 2024-02-12T16:50:00.398100+0000 mgr.y (mgr.14146) 1974 : cluster [DBG] pgmap v2920: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:01 smithi121 bash[18489]: cluster 2024-02-12T16:50:00.097307+0000 osd.7 (osd.7) 43 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:50:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:01 smithi121 bash[18489]: cluster 2024-02-12T16:50:00.154222+0000 osd.7 (osd.7) 44 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:50:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:01 smithi121 bash[18489]: cluster 2024-02-12T16:50:00.398100+0000 mgr.y (mgr.14146) 1974 : cluster [DBG] pgmap v2920: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:01.748 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:02 smithi169 bash[19604]: audit 2024-02-12T16:50:02.112358+0000 mon.a (mon.0) 3537 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:02 smithi123 bash[19119]: audit 2024-02-12T16:50:02.112358+0000 mon.a (mon.0) 3537 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:02 smithi121 bash[18489]: audit 2024-02-12T16:50:02.112358+0000 mon.a (mon.0) 3537 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:03.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.398827+0000 mgr.y (mgr.14146) 1975 : cluster [DBG] pgmap v2921: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: audit 2024-02-12T16:50:02.450410+0000 mon.a (mon.0) 3538 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: audit 2024-02-12T16:50:02.451836+0000 mon.a (mon.0) 3539 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: audit 2024-02-12T16:50:02.461386+0000 mon.a (mon.0) 3540 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.579128+0000 osd.0 (osd.0) 179 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.664468+0000 osd.1 (osd.1) 200 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.671358+0000 osd.0 (osd.0) 180 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.674367+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.674433+0000 osd.3 (osd.3) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.871440+0000 osd.1 (osd.1) 201 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:50:03.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:03 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.903732+0000 osd.6 (osd.6) 206 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:50:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.398827+0000 mgr.y (mgr.14146) 1975 : cluster [DBG] pgmap v2921: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: audit 2024-02-12T16:50:02.450410+0000 mon.a (mon.0) 3538 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: audit 2024-02-12T16:50:02.451836+0000 mon.a (mon.0) 3539 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: audit 2024-02-12T16:50:02.461386+0000 mon.a (mon.0) 3540 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:50:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.579128+0000 osd.0 (osd.0) 179 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:50:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.664468+0000 osd.1 (osd.1) 200 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:50:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.671358+0000 osd.0 (osd.0) 180 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:50:03.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.674367+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:03.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.674433+0000 osd.3 (osd.3) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:03.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.871440+0000 osd.1 (osd.1) 201 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:50:03.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:03 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.903732+0000 osd.6 (osd.6) 206 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.398827+0000 mgr.y (mgr.14146) 1975 : cluster [DBG] pgmap v2921: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: audit 2024-02-12T16:50:02.450410+0000 mon.a (mon.0) 3538 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: audit 2024-02-12T16:50:02.451836+0000 mon.a (mon.0) 3539 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: audit 2024-02-12T16:50:02.461386+0000 mon.a (mon.0) 3540 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.579128+0000 osd.0 (osd.0) 179 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.664468+0000 osd.1 (osd.1) 200 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.671358+0000 osd.0 (osd.0) 180 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.674367+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.674433+0000 osd.3 (osd.3) 143 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.871440+0000 osd.1 (osd.1) 201 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:50:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:03 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.903732+0000 osd.6 (osd.6) 206 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:50:03.775 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.901709+0000 osd.5 (osd.5) 249 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:02.981357+0000 osd.5 (osd.5) 250 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.037807+0000 osd.6 (osd.6) 207 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.468064+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e1080: 12 total, 12 up, 12 in 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.945763+0000 mon.a (mon.0) 3542 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2478541222,config=2903391811,health=3696959945,kv=2542851637}) 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.947617+0000 mon.a (mon.0) 3543 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=3506548954,logm=833205723}) 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.949315+0000 mon.a (mon.0) 3544 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419429912}) 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.951144+0000 mon.a (mon.0) 3545 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1373575944}) 2024-02-12T16:50:04.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.953002+0000 mon.a (mon.0) 3546 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=216247478}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.954723+0000 mon.a (mon.0) 3547 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406284940}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.960028+0000 mon.a (mon.0) 3548 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1549526032}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.964630+0000 mon.a (mon.0) 3549 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496208905}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.969309+0000 mon.a (mon.0) 3550 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3900824583}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.974157+0000 mon.a (mon.0) 3551 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646432729}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.978471+0000 mon.a (mon.0) 3552 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3988777940}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.981090+0000 mon.a (mon.0) 3553 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=37,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=28} crc {logm=3027694865,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=4083815489,mon_config_key=3810721106}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.982070+0000 mon.a (mon.0) 3554 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1123477410}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.983140+0000 mon.a (mon.0) 3555 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2234197760}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.984205+0000 mon.a (mon.0) 3556 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3824029627,monmap=2720272532,osd_metadata=4093509710,osd_pg_creating=2098255515,osdmap=1785660982}) 2024-02-12T16:50:04.800 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:04 smithi169 bash[19604]: cluster 2024-02-12T16:50:03.991967+0000 mon.a (mon.0) 3557 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=45} crc {osdmap=2673219995}) 2024-02-12T16:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.901709+0000 osd.5 (osd.5) 249 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:02.981357+0000 osd.5 (osd.5) 250 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.037807+0000 osd.6 (osd.6) 207 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:50:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.468064+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e1080: 12 total, 12 up, 12 in 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.945763+0000 mon.a (mon.0) 3542 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2478541222,config=2903391811,health=3696959945,kv=2542851637}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.947617+0000 mon.a (mon.0) 3543 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=3506548954,logm=833205723}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.949315+0000 mon.a (mon.0) 3544 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419429912}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.951144+0000 mon.a (mon.0) 3545 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1373575944}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.953002+0000 mon.a (mon.0) 3546 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=216247478}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.954723+0000 mon.a (mon.0) 3547 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406284940}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.960028+0000 mon.a (mon.0) 3548 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1549526032}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.964630+0000 mon.a (mon.0) 3549 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496208905}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.969309+0000 mon.a (mon.0) 3550 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3900824583}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.974157+0000 mon.a (mon.0) 3551 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646432729}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.978471+0000 mon.a (mon.0) 3552 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3988777940}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.981090+0000 mon.a (mon.0) 3553 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=37,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=28} crc {logm=3027694865,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=4083815489,mon_config_key=3810721106}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.982070+0000 mon.a (mon.0) 3554 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1123477410}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.983140+0000 mon.a (mon.0) 3555 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2234197760}) 2024-02-12T16:50:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.984205+0000 mon.a (mon.0) 3556 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3824029627,monmap=2720272532,osd_metadata=4093509710,osd_pg_creating=2098255515,osdmap=1785660982}) 2024-02-12T16:50:04.891 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:04 smithi123 bash[19119]: cluster 2024-02-12T16:50:03.991967+0000 mon.a (mon.0) 3557 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=45} crc {osdmap=2673219995}) 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.901709+0000 osd.5 (osd.5) 249 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:02.981357+0000 osd.5 (osd.5) 250 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.037807+0000 osd.6 (osd.6) 207 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.468064+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e1080: 12 total, 12 up, 12 in 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.945763+0000 mon.a (mon.0) 3542 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2478541222,config=2903391811,health=3696959945,kv=2542851637}) 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.947617+0000 mon.a (mon.0) 3543 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=3506548954,logm=833205723}) 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.949315+0000 mon.a (mon.0) 3544 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419429912}) 2024-02-12T16:50:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.951144+0000 mon.a (mon.0) 3545 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1373575944}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.953002+0000 mon.a (mon.0) 3546 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=216247478}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.954723+0000 mon.a (mon.0) 3547 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3406284940}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.960028+0000 mon.a (mon.0) 3548 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1549526032}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.964630+0000 mon.a (mon.0) 3549 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496208905}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.969309+0000 mon.a (mon.0) 3550 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3900824583}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.974157+0000 mon.a (mon.0) 3551 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646432729}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.978471+0000 mon.a (mon.0) 3552 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3988777940}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.981090+0000 mon.a (mon.0) 3553 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=37,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=28} crc {logm=3027694865,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=4083815489,mon_config_key=3810721106}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.982070+0000 mon.a (mon.0) 3554 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1123477410}) 2024-02-12T16:50:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.983140+0000 mon.a (mon.0) 3555 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2234197760}) 2024-02-12T16:50:04.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.984205+0000 mon.a (mon.0) 3556 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=18,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=64} crc {mon_config_key=3824029627,monmap=2720272532,osd_metadata=4093509710,osd_pg_creating=2098255515,osdmap=1785660982}) 2024-02-12T16:50:04.913 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:04 smithi121 bash[18489]: cluster 2024-02-12T16:50:03.991967+0000 mon.a (mon.0) 3557 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=45} crc {osdmap=2673219995}) 2024-02-12T16:50:05.788 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:05 smithi121 bash[18489]: cluster 2024-02-12T16:50:04.399657+0000 mgr.y (mgr.14146) 1976 : cluster [DBG] pgmap v2923: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:05 smithi169 bash[19604]: cluster 2024-02-12T16:50:04.399657+0000 mgr.y (mgr.14146) 1976 : cluster [DBG] pgmap v2923: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:05 smithi123 bash[19119]: cluster 2024-02-12T16:50:04.399657+0000 mgr.y (mgr.14146) 1976 : cluster [DBG] pgmap v2923: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:05.938 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:50:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:50:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-02-12T16:50:05.939 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:50:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:06 smithi169 bash[19604]: cluster 2024-02-12T16:50:05.036028+0000 osd.7 (osd.7) 45 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:50:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:06 smithi169 bash[19604]: cluster 2024-02-12T16:50:05.147971+0000 osd.7 (osd.7) 46 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:50:06.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:06 smithi169 bash[19604]: cluster 2024-02-12T16:50:05.270332+0000 osd.4 (osd.4) 189 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:50:06.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:06 smithi169 bash[19604]: cluster 2024-02-12T16:50:05.364121+0000 osd.4 (osd.4) 190 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:50:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:06 smithi123 bash[19119]: cluster 2024-02-12T16:50:05.036028+0000 osd.7 (osd.7) 45 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:50:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:06 smithi123 bash[19119]: cluster 2024-02-12T16:50:05.147971+0000 osd.7 (osd.7) 46 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:50:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:06 smithi123 bash[19119]: cluster 2024-02-12T16:50:05.270332+0000 osd.4 (osd.4) 189 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:50:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:06 smithi123 bash[19119]: cluster 2024-02-12T16:50:05.364121+0000 osd.4 (osd.4) 190 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:50:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:06 smithi121 bash[18489]: cluster 2024-02-12T16:50:05.036028+0000 osd.7 (osd.7) 45 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:50:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:06 smithi121 bash[18489]: cluster 2024-02-12T16:50:05.147971+0000 osd.7 (osd.7) 46 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:50:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:06 smithi121 bash[18489]: cluster 2024-02-12T16:50:05.270332+0000 osd.4 (osd.4) 189 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:50:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:06 smithi121 bash[18489]: cluster 2024-02-12T16:50:05.364121+0000 osd.4 (osd.4) 190 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:50:07.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-12T16:50:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:07 smithi121 bash[18489]: cluster 2024-02-12T16:50:06.279921+0000 osd.9 (osd.9) 177 : cluster [DBG] 1.a scrub starts 2024-02-12T16:50:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:07 smithi121 bash[18489]: cluster 2024-02-12T16:50:06.371563+0000 osd.9 (osd.9) 178 : cluster [DBG] 1.a scrub ok 2024-02-12T16:50:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:07 smithi121 bash[18489]: cluster 2024-02-12T16:50:06.400382+0000 mgr.y (mgr.14146) 1977 : cluster [DBG] pgmap v2924: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:07 smithi121 bash[18489]: cluster 2024-02-12T16:50:06.758457+0000 osd.1 (osd.1) 202 : cluster [DBG] 1.16 deep-scrub starts 2024-02-12T16:50:07.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:07 smithi121 bash[18489]: cluster 2024-02-12T16:50:06.911718+0000 osd.1 (osd.1) 203 : cluster [DBG] 1.16 deep-scrub ok 2024-02-12T16:50:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:07 smithi169 bash[19604]: cluster 2024-02-12T16:50:06.279921+0000 osd.9 (osd.9) 177 : cluster [DBG] 1.a scrub starts 2024-02-12T16:50:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:07 smithi169 bash[19604]: cluster 2024-02-12T16:50:06.371563+0000 osd.9 (osd.9) 178 : cluster [DBG] 1.a scrub ok 2024-02-12T16:50:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:07 smithi169 bash[19604]: cluster 2024-02-12T16:50:06.400382+0000 mgr.y (mgr.14146) 1977 : cluster [DBG] pgmap v2924: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:07 smithi169 bash[19604]: cluster 2024-02-12T16:50:06.758457+0000 osd.1 (osd.1) 202 : cluster [DBG] 1.16 deep-scrub starts 2024-02-12T16:50:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:07 smithi169 bash[19604]: cluster 2024-02-12T16:50:06.911718+0000 osd.1 (osd.1) 203 : cluster [DBG] 1.16 deep-scrub ok 2024-02-12T16:50:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:07 smithi123 bash[19119]: cluster 2024-02-12T16:50:06.279921+0000 osd.9 (osd.9) 177 : cluster [DBG] 1.a scrub starts 2024-02-12T16:50:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:07 smithi123 bash[19119]: cluster 2024-02-12T16:50:06.371563+0000 osd.9 (osd.9) 178 : cluster [DBG] 1.a scrub ok 2024-02-12T16:50:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:07 smithi123 bash[19119]: cluster 2024-02-12T16:50:06.400382+0000 mgr.y (mgr.14146) 1977 : cluster [DBG] pgmap v2924: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:07 smithi123 bash[19119]: cluster 2024-02-12T16:50:06.758457+0000 osd.1 (osd.1) 202 : cluster [DBG] 1.16 deep-scrub starts 2024-02-12T16:50:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:07 smithi123 bash[19119]: cluster 2024-02-12T16:50:06.911718+0000 osd.1 (osd.1) 203 : cluster [DBG] 1.16 deep-scrub ok 2024-02-12T16:50:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-02-12T16:50:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:09 smithi169 bash[19604]: cluster 2024-02-12T16:50:08.145723+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:09 smithi169 bash[19604]: cluster 2024-02-12T16:50:08.145812+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:09 smithi169 bash[19604]: cluster 2024-02-12T16:50:08.401084+0000 mgr.y (mgr.14146) 1978 : cluster [DBG] pgmap v2925: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:09 smithi169 bash[19604]: cluster 2024-02-12T16:50:08.490436+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e1081: 12 total, 12 up, 12 in 2024-02-12T16:50:09.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:09 smithi169 bash[19604]: audit 2024-02-12T16:50:08.681526+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? v1:172.21.15.121:0/63017399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:50:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:09 smithi123 bash[19119]: cluster 2024-02-12T16:50:08.145723+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:09 smithi123 bash[19119]: cluster 2024-02-12T16:50:08.145812+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:09 smithi123 bash[19119]: cluster 2024-02-12T16:50:08.401084+0000 mgr.y (mgr.14146) 1978 : cluster [DBG] pgmap v2925: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:09 smithi123 bash[19119]: cluster 2024-02-12T16:50:08.490436+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e1081: 12 total, 12 up, 12 in 2024-02-12T16:50:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:09 smithi123 bash[19119]: audit 2024-02-12T16:50:08.681526+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? v1:172.21.15.121:0/63017399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:50:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:09 smithi121 bash[18489]: cluster 2024-02-12T16:50:08.145723+0000 osd.6 (osd.6) 208 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:09 smithi121 bash[18489]: cluster 2024-02-12T16:50:08.145812+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:09 smithi121 bash[18489]: cluster 2024-02-12T16:50:08.401084+0000 mgr.y (mgr.14146) 1978 : cluster [DBG] pgmap v2925: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:09 smithi121 bash[18489]: cluster 2024-02-12T16:50:08.490436+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e1081: 12 total, 12 up, 12 in 2024-02-12T16:50:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:09 smithi121 bash[18489]: audit 2024-02-12T16:50:08.681526+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? v1:172.21.15.121:0/63017399' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-02-12T16:50:10.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 9 1 2024-02-12T16:50:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:10 smithi121 bash[18489]: audit 2024-02-12T16:50:09.494650+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.121:0/63017399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:50:10.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:10 smithi121 bash[18489]: cluster 2024-02-12T16:50:09.494775+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e1082: 12 total, 12 up, 12 in 2024-02-12T16:50:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:10 smithi169 bash[19604]: audit 2024-02-12T16:50:09.494650+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.121:0/63017399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:50:10.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:10 smithi169 bash[19604]: cluster 2024-02-12T16:50:09.494775+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e1082: 12 total, 12 up, 12 in 2024-02-12T16:50:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:10 smithi123 bash[19119]: audit 2024-02-12T16:50:09.494650+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.121:0/63017399' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-02-12T16:50:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:10 smithi123 bash[19119]: cluster 2024-02-12T16:50:09.494775+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e1082: 12 total, 12 up, 12 in 2024-02-12T16:50:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:11 smithi169 bash[19604]: cluster 2024-02-12T16:50:09.969404+0000 osd.10 (osd.10) 274 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:50:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:11 smithi169 bash[19604]: cluster 2024-02-12T16:50:10.019724+0000 osd.10 (osd.10) 275 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:50:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:11 smithi169 bash[19604]: cluster 2024-02-12T16:50:10.088960+0000 osd.7 (osd.7) 47 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:50:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:11 smithi169 bash[19604]: cluster 2024-02-12T16:50:10.115116+0000 osd.7 (osd.7) 48 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:50:11.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:11 smithi169 bash[19604]: cluster 2024-02-12T16:50:10.401710+0000 mgr.y (mgr.14146) 1979 : cluster [DBG] pgmap v2928: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:11 smithi123 bash[19119]: cluster 2024-02-12T16:50:09.969404+0000 osd.10 (osd.10) 274 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:50:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:11 smithi123 bash[19119]: cluster 2024-02-12T16:50:10.019724+0000 osd.10 (osd.10) 275 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:50:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:11 smithi123 bash[19119]: cluster 2024-02-12T16:50:10.088960+0000 osd.7 (osd.7) 47 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:50:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:11 smithi123 bash[19119]: cluster 2024-02-12T16:50:10.115116+0000 osd.7 (osd.7) 48 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:50:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:11 smithi123 bash[19119]: cluster 2024-02-12T16:50:10.401710+0000 mgr.y (mgr.14146) 1979 : cluster [DBG] pgmap v2928: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:11 smithi121 bash[18489]: cluster 2024-02-12T16:50:09.969404+0000 osd.10 (osd.10) 274 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:50:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:11 smithi121 bash[18489]: cluster 2024-02-12T16:50:10.019724+0000 osd.10 (osd.10) 275 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:50:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:11 smithi121 bash[18489]: cluster 2024-02-12T16:50:10.088960+0000 osd.7 (osd.7) 47 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:50:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:11 smithi121 bash[18489]: cluster 2024-02-12T16:50:10.115116+0000 osd.7 (osd.7) 48 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:50:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:11 smithi121 bash[18489]: cluster 2024-02-12T16:50:10.401710+0000 mgr.y (mgr.14146) 1979 : cluster [DBG] pgmap v2928: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:12.088 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:13.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:13 smithi121 bash[18489]: cluster 2024-02-12T16:50:12.402445+0000 mgr.y (mgr.14146) 1980 : cluster [DBG] pgmap v2929: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:13.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:13 smithi169 bash[19604]: cluster 2024-02-12T16:50:12.402445+0000 mgr.y (mgr.14146) 1980 : cluster [DBG] pgmap v2929: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:13 smithi123 bash[19119]: cluster 2024-02-12T16:50:12.402445+0000 mgr.y (mgr.14146) 1980 : cluster [DBG] pgmap v2929: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:14.752 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:15.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:15 smithi121 bash[18489]: cluster 2024-02-12T16:50:14.198085+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:15.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:15 smithi121 bash[18489]: cluster 2024-02-12T16:50:14.198176+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:15.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:15 smithi121 bash[18489]: cluster 2024-02-12T16:50:14.403175+0000 mgr.y (mgr.14146) 1981 : cluster [DBG] pgmap v2930: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:15.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:15 smithi121 bash[18489]: cluster 2024-02-12T16:50:14.529212+0000 mon.a (mon.0) 3562 : cluster [DBG] osdmap e1083: 12 total, 12 up, 12 in 2024-02-12T16:50:15.785 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:15 smithi121 bash[18489]: cluster 2024-02-12T16:50:15.256225+0000 osd.11 (osd.11) 251 : cluster [DBG] 1.1e deep-scrub starts 2024-02-12T16:50:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:15 smithi169 bash[19604]: cluster 2024-02-12T16:50:14.198085+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:15 smithi169 bash[19604]: cluster 2024-02-12T16:50:14.198176+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:15 smithi169 bash[19604]: cluster 2024-02-12T16:50:14.403175+0000 mgr.y (mgr.14146) 1981 : cluster [DBG] pgmap v2930: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:15 smithi169 bash[19604]: cluster 2024-02-12T16:50:14.529212+0000 mon.a (mon.0) 3562 : cluster [DBG] osdmap e1083: 12 total, 12 up, 12 in 2024-02-12T16:50:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:15 smithi169 bash[19604]: cluster 2024-02-12T16:50:15.256225+0000 osd.11 (osd.11) 251 : cluster [DBG] 1.1e deep-scrub starts 2024-02-12T16:50:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:15 smithi123 bash[19119]: cluster 2024-02-12T16:50:14.198085+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:15 smithi123 bash[19119]: cluster 2024-02-12T16:50:14.198176+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:15 smithi123 bash[19119]: cluster 2024-02-12T16:50:14.403175+0000 mgr.y (mgr.14146) 1981 : cluster [DBG] pgmap v2930: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:15 smithi123 bash[19119]: cluster 2024-02-12T16:50:14.529212+0000 mon.a (mon.0) 3562 : cluster [DBG] osdmap e1083: 12 total, 12 up, 12 in 2024-02-12T16:50:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:15 smithi123 bash[19119]: cluster 2024-02-12T16:50:15.256225+0000 osd.11 (osd.11) 251 : cluster [DBG] 1.1e deep-scrub starts 2024-02-12T16:50:15.950 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:50:15.951 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-02-12T16:50:15.951 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-02-12T16:50:15.951 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-02-12T16:50:15.951 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:50:17.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-12T16:50:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:17 smithi121 bash[18489]: cluster 2024-02-12T16:50:15.328466+0000 osd.11 (osd.11) 252 : cluster [DBG] 1.1e deep-scrub ok 2024-02-12T16:50:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:17 smithi121 bash[18489]: cluster 2024-02-12T16:50:16.403839+0000 mgr.y (mgr.14146) 1982 : cluster [DBG] pgmap v2932: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:17 smithi169 bash[19604]: cluster 2024-02-12T16:50:15.328466+0000 osd.11 (osd.11) 252 : cluster [DBG] 1.1e deep-scrub ok 2024-02-12T16:50:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:17 smithi169 bash[19604]: cluster 2024-02-12T16:50:16.403839+0000 mgr.y (mgr.14146) 1982 : cluster [DBG] pgmap v2932: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:17.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:17 smithi123 bash[19119]: cluster 2024-02-12T16:50:15.328466+0000 osd.11 (osd.11) 252 : cluster [DBG] 1.1e deep-scrub ok 2024-02-12T16:50:17.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:17 smithi123 bash[19119]: cluster 2024-02-12T16:50:16.403839+0000 mgr.y (mgr.14146) 1982 : cluster [DBG] pgmap v2932: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:18 smithi121 bash[18489]: cluster 2024-02-12T16:50:16.789677+0000 osd.8 (osd.8) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:18 smithi121 bash[18489]: cluster 2024-02-12T16:50:16.789769+0000 osd.8 (osd.8) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:18 smithi121 bash[18489]: cluster 2024-02-12T16:50:17.145177+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e1084: 12 total, 12 up, 12 in 2024-02-12T16:50:18.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:18 smithi121 bash[18489]: cluster 2024-02-12T16:50:17.254736+0000 osd.7 (osd.7) 49 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:50:18.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:18 smithi121 bash[18489]: cluster 2024-02-12T16:50:17.349373+0000 osd.7 (osd.7) 50 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:50:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:18 smithi169 bash[19604]: cluster 2024-02-12T16:50:16.789677+0000 osd.8 (osd.8) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:18 smithi169 bash[19604]: cluster 2024-02-12T16:50:16.789769+0000 osd.8 (osd.8) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:18 smithi169 bash[19604]: cluster 2024-02-12T16:50:17.145177+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e1084: 12 total, 12 up, 12 in 2024-02-12T16:50:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:18 smithi169 bash[19604]: cluster 2024-02-12T16:50:17.254736+0000 osd.7 (osd.7) 49 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:50:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:18 smithi169 bash[19604]: cluster 2024-02-12T16:50:17.349373+0000 osd.7 (osd.7) 50 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:50:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:18 smithi123 bash[19119]: cluster 2024-02-12T16:50:16.789677+0000 osd.8 (osd.8) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:18 smithi123 bash[19119]: cluster 2024-02-12T16:50:16.789769+0000 osd.8 (osd.8) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:18 smithi123 bash[19119]: cluster 2024-02-12T16:50:17.145177+0000 mon.a (mon.0) 3563 : cluster [DBG] osdmap e1084: 12 total, 12 up, 12 in 2024-02-12T16:50:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:18 smithi123 bash[19119]: cluster 2024-02-12T16:50:17.254736+0000 osd.7 (osd.7) 49 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:50:18.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:18 smithi123 bash[19119]: cluster 2024-02-12T16:50:17.349373+0000 osd.7 (osd.7) 50 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:50:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.9 to 1 (10000) 2024-02-12T16:50:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:19 smithi169 bash[19604]: cluster 2024-02-12T16:50:18.404437+0000 mgr.y (mgr.14146) 1983 : cluster [DBG] pgmap v2934: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:19.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:19 smithi169 bash[19604]: audit 2024-02-12T16:50:18.959317+0000 mon.a (mon.0) 3564 : audit [INF] from='client.? v1:172.21.15.121:0/3720554221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:50:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:19 smithi123 bash[19119]: cluster 2024-02-12T16:50:18.404437+0000 mgr.y (mgr.14146) 1983 : cluster [DBG] pgmap v2934: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:19.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:19 smithi123 bash[19119]: audit 2024-02-12T16:50:18.959317+0000 mon.a (mon.0) 3564 : audit [INF] from='client.? v1:172.21.15.121:0/3720554221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:50:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:19 smithi121 bash[18489]: cluster 2024-02-12T16:50:18.404437+0000 mgr.y (mgr.14146) 1983 : cluster [DBG] pgmap v2934: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:19 smithi121 bash[18489]: audit 2024-02-12T16:50:18.959317+0000 mon.a (mon.0) 3564 : audit [INF] from='client.? v1:172.21.15.121:0/3720554221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:50:19.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 9 1 2024-02-12T16:50:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:20 smithi121 bash[18489]: cluster 2024-02-12T16:50:19.042167+0000 osd.10 (osd.10) 276 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:50:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:20 smithi121 bash[18489]: cluster 2024-02-12T16:50:19.112195+0000 osd.10 (osd.10) 277 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:50:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:20 smithi121 bash[18489]: audit 2024-02-12T16:50:19.152358+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? v1:172.21.15.121:0/3720554221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:50:20.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:20 smithi121 bash[18489]: cluster 2024-02-12T16:50:19.152533+0000 mon.a (mon.0) 3566 : cluster [DBG] osdmap e1085: 12 total, 12 up, 12 in 2024-02-12T16:50:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:20 smithi169 bash[19604]: cluster 2024-02-12T16:50:19.042167+0000 osd.10 (osd.10) 276 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:50:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:20 smithi169 bash[19604]: cluster 2024-02-12T16:50:19.112195+0000 osd.10 (osd.10) 277 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:50:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:20 smithi169 bash[19604]: audit 2024-02-12T16:50:19.152358+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? v1:172.21.15.121:0/3720554221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:50:20.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:20 smithi169 bash[19604]: cluster 2024-02-12T16:50:19.152533+0000 mon.a (mon.0) 3566 : cluster [DBG] osdmap e1085: 12 total, 12 up, 12 in 2024-02-12T16:50:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:20 smithi123 bash[19119]: cluster 2024-02-12T16:50:19.042167+0000 osd.10 (osd.10) 276 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:50:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:20 smithi123 bash[19119]: cluster 2024-02-12T16:50:19.112195+0000 osd.10 (osd.10) 277 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:50:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:20 smithi123 bash[19119]: audit 2024-02-12T16:50:19.152358+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? v1:172.21.15.121:0/3720554221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:50:20.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:20 smithi123 bash[19119]: cluster 2024-02-12T16:50:19.152533+0000 mon.a (mon.0) 3566 : cluster [DBG] osdmap e1085: 12 total, 12 up, 12 in 2024-02-12T16:50:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: cluster 2024-02-12T16:50:20.405136+0000 mgr.y (mgr.14146) 1984 : cluster [DBG] pgmap v2936: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.524200+0000 mon.a (mon.0) 3567 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07611543990867582]}]: dispatch 2024-02-12T16:50:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.524772+0000 mon.a (mon.0) 3568 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10065076203989101]}]: dispatch 2024-02-12T16:50:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.525363+0000 mon.a (mon.0) 3569 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09788043773815991]}]: dispatch 2024-02-12T16:50:21.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.525823+0000 mon.a (mon.0) 3570 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10367306166492181]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.526266+0000 mon.a (mon.0) 3571 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0940000730126043]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.526740+0000 mon.a (mon.0) 3572 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08155707199834682]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.527241+0000 mon.a (mon.0) 3573 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0805375864373931]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.527674+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07599393870280446]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.528122+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09128117548133796]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.528564+0000 mon.a (mon.0) 3576 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08889916042632115]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.529002+0000 mon.a (mon.0) 3577 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07566287599954472]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.529443+0000 mon.a (mon.0) 3578 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08129480330874889]}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: audit 2024-02-12T16:50:20.529868+0000 mon.a (mon.0) 3579 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"37309\"}"}]: dispatch 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: cluster 2024-02-12T16:50:20.827015+0000 osd.7 (osd.7) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:21.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:21 smithi169 bash[19604]: cluster 2024-02-12T16:50:20.827119+0000 osd.7 (osd.7) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: cluster 2024-02-12T16:50:20.405136+0000 mgr.y (mgr.14146) 1984 : cluster [DBG] pgmap v2936: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.524200+0000 mon.a (mon.0) 3567 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07611543990867582]}]: dispatch 2024-02-12T16:50:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.524772+0000 mon.a (mon.0) 3568 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10065076203989101]}]: dispatch 2024-02-12T16:50:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.525363+0000 mon.a (mon.0) 3569 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09788043773815991]}]: dispatch 2024-02-12T16:50:21.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.525823+0000 mon.a (mon.0) 3570 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10367306166492181]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.526266+0000 mon.a (mon.0) 3571 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0940000730126043]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.526740+0000 mon.a (mon.0) 3572 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08155707199834682]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.527241+0000 mon.a (mon.0) 3573 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0805375864373931]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.527674+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07599393870280446]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.528122+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09128117548133796]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.528564+0000 mon.a (mon.0) 3576 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08889916042632115]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.529002+0000 mon.a (mon.0) 3577 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07566287599954472]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.529443+0000 mon.a (mon.0) 3578 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08129480330874889]}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: audit 2024-02-12T16:50:20.529868+0000 mon.a (mon.0) 3579 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"37309\"}"}]: dispatch 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: cluster 2024-02-12T16:50:20.827015+0000 osd.7 (osd.7) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:21.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:21 smithi123 bash[19119]: cluster 2024-02-12T16:50:20.827119+0000 osd.7 (osd.7) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: cluster 2024-02-12T16:50:20.405136+0000 mgr.y (mgr.14146) 1984 : cluster [DBG] pgmap v2936: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.524200+0000 mon.a (mon.0) 3567 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07611543990867582]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.524772+0000 mon.a (mon.0) 3568 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10065076203989101]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.525363+0000 mon.a (mon.0) 3569 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09788043773815991]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.525823+0000 mon.a (mon.0) 3570 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10367306166492181]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.526266+0000 mon.a (mon.0) 3571 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0940000730126043]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.526740+0000 mon.a (mon.0) 3572 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08155707199834682]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.527241+0000 mon.a (mon.0) 3573 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0805375864373931]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.527674+0000 mon.a (mon.0) 3574 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07599393870280446]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.528122+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09128117548133796]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.528564+0000 mon.a (mon.0) 3576 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08889916042632115]}]: dispatch 2024-02-12T16:50:21.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.529002+0000 mon.a (mon.0) 3577 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07566287599954472]}]: dispatch 2024-02-12T16:50:21.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.529443+0000 mon.a (mon.0) 3578 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08129480330874889]}]: dispatch 2024-02-12T16:50:21.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: audit 2024-02-12T16:50:20.529868+0000 mon.a (mon.0) 3579 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"37309\"}"}]: dispatch 2024-02-12T16:50:21.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: cluster 2024-02-12T16:50:20.827015+0000 osd.7 (osd.7) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:21.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:21 smithi121 bash[18489]: cluster 2024-02-12T16:50:20.827119+0000 osd.7 (osd.7) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:22.440 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.197787+0000 mon.a (mon.0) 3580 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07611543990867582]}]': finished 2024-02-12T16:50:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.197905+0000 mon.a (mon.0) 3581 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10065076203989101]}]': finished 2024-02-12T16:50:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198001+0000 mon.a (mon.0) 3582 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09788043773815991]}]': finished 2024-02-12T16:50:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198113+0000 mon.a (mon.0) 3583 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10367306166492181]}]': finished 2024-02-12T16:50:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198206+0000 mon.a (mon.0) 3584 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0940000730126043]}]': finished 2024-02-12T16:50:22.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198336+0000 mon.a (mon.0) 3585 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08155707199834682]}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198397+0000 mon.a (mon.0) 3586 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0805375864373931]}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198494+0000 mon.a (mon.0) 3587 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07599393870280446]}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198635+0000 mon.a (mon.0) 3588 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09128117548133796]}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198733+0000 mon.a (mon.0) 3589 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08889916042632115]}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198812+0000 mon.a (mon.0) 3590 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07566287599954472]}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198884+0000 mon.a (mon.0) 3591 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08129480330874889]}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: audit 2024-02-12T16:50:21.198957+0000 mon.a (mon.0) 3592 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"37309\"}"}]': finished 2024-02-12T16:50:22.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:22 smithi169 bash[19604]: cluster 2024-02-12T16:50:21.199054+0000 mon.a (mon.0) 3593 : cluster [DBG] osdmap e1086: 12 total, 12 up, 12 in 2024-02-12T16:50:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.197787+0000 mon.a (mon.0) 3580 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07611543990867582]}]': finished 2024-02-12T16:50:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.197905+0000 mon.a (mon.0) 3581 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10065076203989101]}]': finished 2024-02-12T16:50:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198001+0000 mon.a (mon.0) 3582 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09788043773815991]}]': finished 2024-02-12T16:50:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198113+0000 mon.a (mon.0) 3583 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10367306166492181]}]': finished 2024-02-12T16:50:22.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198206+0000 mon.a (mon.0) 3584 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0940000730126043]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198336+0000 mon.a (mon.0) 3585 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08155707199834682]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198397+0000 mon.a (mon.0) 3586 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0805375864373931]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198494+0000 mon.a (mon.0) 3587 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07599393870280446]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198635+0000 mon.a (mon.0) 3588 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09128117548133796]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198733+0000 mon.a (mon.0) 3589 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08889916042632115]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198812+0000 mon.a (mon.0) 3590 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07566287599954472]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198884+0000 mon.a (mon.0) 3591 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08129480330874889]}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: audit 2024-02-12T16:50:21.198957+0000 mon.a (mon.0) 3592 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"37309\"}"}]': finished 2024-02-12T16:50:22.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:22 smithi123 bash[19119]: cluster 2024-02-12T16:50:21.199054+0000 mon.a (mon.0) 3593 : cluster [DBG] osdmap e1086: 12 total, 12 up, 12 in 2024-02-12T16:50:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.197787+0000 mon.a (mon.0) 3580 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.07611543990867582]}]': finished 2024-02-12T16:50:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.197905+0000 mon.a (mon.0) 3581 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.10065076203989101]}]': finished 2024-02-12T16:50:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198001+0000 mon.a (mon.0) 3582 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09788043773815991]}]': finished 2024-02-12T16:50:22.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198113+0000 mon.a (mon.0) 3583 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.10367306166492181]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198206+0000 mon.a (mon.0) 3584 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.0940000730126043]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198336+0000 mon.a (mon.0) 3585 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08155707199834682]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198397+0000 mon.a (mon.0) 3586 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.0805375864373931]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198494+0000 mon.a (mon.0) 3587 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.07599393870280446]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198635+0000 mon.a (mon.0) 3588 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.09128117548133796]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198733+0000 mon.a (mon.0) 3589 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08889916042632115]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198812+0000 mon.a (mon.0) 3590 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.07566287599954472]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198884+0000 mon.a (mon.0) 3591 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08129480330874889]}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: audit 2024-02-12T16:50:21.198957+0000 mon.a (mon.0) 3592 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "format": "json", "weights": "{\"3\": \"37309\"}"}]': finished 2024-02-12T16:50:22.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:22 smithi121 bash[18489]: cluster 2024-02-12T16:50:21.199054+0000 mon.a (mon.0) 3593 : cluster [DBG] osdmap e1086: 12 total, 12 up, 12 in 2024-02-12T16:50:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:23 smithi169 bash[19604]: cluster 2024-02-12T16:50:22.203980+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e1087: 12 total, 12 up, 12 in 2024-02-12T16:50:23.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:23 smithi169 bash[19604]: cluster 2024-02-12T16:50:22.405794+0000 mgr.y (mgr.14146) 1985 : cluster [DBG] pgmap v2939: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:23 smithi123 bash[19119]: cluster 2024-02-12T16:50:22.203980+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e1087: 12 total, 12 up, 12 in 2024-02-12T16:50:23.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:23 smithi123 bash[19119]: cluster 2024-02-12T16:50:22.405794+0000 mgr.y (mgr.14146) 1985 : cluster [DBG] pgmap v2939: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:23.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:23 smithi121 bash[18489]: cluster 2024-02-12T16:50:22.203980+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e1087: 12 total, 12 up, 12 in 2024-02-12T16:50:23.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:23 smithi121 bash[18489]: cluster 2024-02-12T16:50:22.405794+0000 mgr.y (mgr.14146) 1985 : cluster [DBG] pgmap v2939: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:24.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:24 smithi121 bash[18489]: cluster 2024-02-12T16:50:22.904958+0000 osd.0 (osd.0) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:24.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:24 smithi121 bash[18489]: cluster 2024-02-12T16:50:22.905055+0000 osd.0 (osd.0) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:24.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:24 smithi121 bash[18489]: cluster 2024-02-12T16:50:22.964938+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:50:24.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:24 smithi121 bash[18489]: cluster 2024-02-12T16:50:23.077957+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:50:24.414 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:24 smithi121 bash[18489]: cluster 2024-02-12T16:50:23.222797+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e1088: 12 total, 12 up, 12 in 2024-02-12T16:50:24.473 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:24 smithi169 bash[19604]: cluster 2024-02-12T16:50:22.904958+0000 osd.0 (osd.0) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:24 smithi169 bash[19604]: cluster 2024-02-12T16:50:22.905055+0000 osd.0 (osd.0) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:24 smithi169 bash[19604]: cluster 2024-02-12T16:50:22.964938+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:50:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:24 smithi169 bash[19604]: cluster 2024-02-12T16:50:23.077957+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:50:24.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:24 smithi169 bash[19604]: cluster 2024-02-12T16:50:23.222797+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e1088: 12 total, 12 up, 12 in 2024-02-12T16:50:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:24 smithi123 bash[19119]: cluster 2024-02-12T16:50:22.904958+0000 osd.0 (osd.0) 181 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:24 smithi123 bash[19119]: cluster 2024-02-12T16:50:22.905055+0000 osd.0 (osd.0) 182 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:24 smithi123 bash[19119]: cluster 2024-02-12T16:50:22.964938+0000 osd.5 (osd.5) 251 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:50:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:24 smithi123 bash[19119]: cluster 2024-02-12T16:50:23.077957+0000 osd.5 (osd.5) 252 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:50:24.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:24 smithi123 bash[19119]: cluster 2024-02-12T16:50:23.222797+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e1088: 12 total, 12 up, 12 in 2024-02-12T16:50:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:25 smithi169 bash[19604]: audit 2024-02-12T16:50:24.412151+0000 mon.c (mon.1) 110 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:50:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:25 smithi169 bash[19604]: audit 2024-02-12T16:50:24.412580+0000 mon.c (mon.1) 111 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:50:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:25 smithi169 bash[19604]: audit 2024-02-12T16:50:24.413034+0000 mon.c (mon.1) 112 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:50:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:25 smithi169 bash[19604]: audit 2024-02-12T16:50:24.439683+0000 mon.c (mon.1) 113 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:25 smithi169 bash[19604]: audit 2024-02-12T16:50:24.443289+0000 mon.c (mon.1) 114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:25 smithi169 bash[19604]: audit 2024-02-12T16:50:24.445239+0000 mon.c (mon.1) 115 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:25.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:25 smithi169 bash[19604]: audit 2024-02-12T16:50:24.452923+0000 mon.a (mon.0) 3596 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:50:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:25 smithi123 bash[19119]: audit 2024-02-12T16:50:24.412151+0000 mon.c (mon.1) 110 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:50:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:25 smithi123 bash[19119]: audit 2024-02-12T16:50:24.412580+0000 mon.c (mon.1) 111 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:50:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:25 smithi123 bash[19119]: audit 2024-02-12T16:50:24.413034+0000 mon.c (mon.1) 112 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:50:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:25 smithi123 bash[19119]: audit 2024-02-12T16:50:24.439683+0000 mon.c (mon.1) 113 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:25 smithi123 bash[19119]: audit 2024-02-12T16:50:24.443289+0000 mon.c (mon.1) 114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:25.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:25 smithi123 bash[19119]: audit 2024-02-12T16:50:24.445239+0000 mon.c (mon.1) 115 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:25.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:25 smithi123 bash[19119]: audit 2024-02-12T16:50:24.452923+0000 mon.a (mon.0) 3596 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:50:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:25 smithi121 bash[18489]: audit 2024-02-12T16:50:24.412151+0000 mon.c (mon.1) 110 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:50:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:25 smithi121 bash[18489]: audit 2024-02-12T16:50:24.412580+0000 mon.c (mon.1) 111 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:50:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:25 smithi121 bash[18489]: audit 2024-02-12T16:50:24.413034+0000 mon.c (mon.1) 112 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:50:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:25 smithi121 bash[18489]: audit 2024-02-12T16:50:24.439683+0000 mon.c (mon.1) 113 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:50:25.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:25 smithi121 bash[18489]: audit 2024-02-12T16:50:24.443289+0000 mon.c (mon.1) 114 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:50:25.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:25 smithi121 bash[18489]: audit 2024-02-12T16:50:24.445239+0000 mon.c (mon.1) 115 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:50:25.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:25 smithi121 bash[18489]: audit 2024-02-12T16:50:24.452923+0000 mon.a (mon.0) 3596 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:50:26.601 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-12T16:50:26.601 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-02-12T16:50:26.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26785.0:106004 8.10 8:0c16ca9a:::benchmark_data_smithi169_115677_object23376:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e882)", 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 568.23589300799995, 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.07780718, 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26785", 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/1260804721", 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106004 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.602 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355590+0000", 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2119999999999999e-06 2024-02-12T16:50:26.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355593+0000", 2024-02-12T16:50:26.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1560000000000003e-06 2024-02-12T16:50:26.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355598+0000", 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8820000000000003e-06 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.355602+0000", 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.149617209 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.505219+0000", 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5519999999999998e-06 2024-02-12T16:50:26.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:58.505224+0000", 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.928077547 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:59.433301+0000", 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6343999999999998e-05 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:40:59.433397+0000", 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.07780718 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:122532 9.1 9:8f6a5060:::benchmark_data_smithi169_124168_object122531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.678976+0000", 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 406.91250756099998, 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20243556900000001, 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 122532 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.678976+0000", 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.678976+0000", 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999847 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.678960+0000", 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9978000000000001e-05 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679020+0000", 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9273999999999999e-05 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679040+0000", 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9089999999999999e-06 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679045+0000", 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0336999999999997e-05 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679085+0000", 2024-02-12T16:50:26.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014626099999999999 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679231+0000", 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174907 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.679406+0000", 2024-02-12T16:50:26.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0091274470000000003 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.688534+0000", 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.015220551000000001 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.703754+0000", 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17174315200000001 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875497+0000", 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0059140199999999999 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.881411+0000", 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20243556900000001 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPushReply(9.1 943/935 [PushReplyOp(9:85e0efaa:::benchmark_data_smithi169_124168_object9727:head)])", 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.693117+0000", 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 406.89836595499997, 2024-02-12T16:50:26.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20235745399999999, 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693117+0000", 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693117+0000", 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693116+0000", 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7070000000000003e-06 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693123+0000", 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1909999999999997e-06 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693129+0000", 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8430000000000003e-06 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.693133+0000", 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19611678199999999 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.889250+0000", 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.006224907 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.895475+0000", 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20235745399999999 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.617 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:122555 9.1 9:8efe72fe:::benchmark_data_smithi169_124168_object122554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.697530+0000", 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 406.89395365000001, 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20841303999999999, 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 122555 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.618 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697530+0000", 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697530+0000", 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999938 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697523+0000", 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5047000000000002e-05 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697589+0000", 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.117e-05 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697600+0000", 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5139999999999998e-06 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.697606+0000", 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18391962000000001 2024-02-12T16:50:26.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.881526+0000", 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00026639600000000002 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.881792+0000", 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032232199999999999 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:50:26.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.882115+0000", 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0071066419999999998 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.889221+0000", 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013541442000000001 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.902763+0000", 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1150999999999997e-05 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.902834+0000", 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003108857 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.905943+0000", 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20841303999999999 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.623 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:122572 9.c 9:3e05ac4a:::benchmark_data_smithi169_124168_object122571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:43:39.711516+0000", 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 406.87996748799998, 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20045165000000001, 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 122572 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711516+0000", 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711516+0000", 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711513+0000", 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5548000000000001e-05 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.711609+0000", 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.163957779 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875567+0000", 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6982999999999999e-05 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.875594+0000", 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058365700000000001 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.876177+0000", 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035321000000000001 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.876531+0000", 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012794799999999999 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.877810+0000", 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0093099129999999995 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.887120+0000", 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.024732095999999999 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.911852+0000", 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7362999999999999e-05 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.911919+0000", 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8062000000000002e-05 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:43:39.911967+0000", 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20045165000000001 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:157639 9.1 9:88ef1387:::benchmark_data_smithi169_124168_object6955:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:14.941858+0000", 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 371.64962544399998, 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19744788699999999, 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157639 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.941858+0000", 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.941858+0000", 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.941856+0000", 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6490000000000002e-06 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.941862+0000", 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.842e-06 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.941872+0000", 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8359999999999999e-06 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.941877+0000", 2024-02-12T16:50:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023957359999999999 2024-02-12T16:50:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.944273+0000", 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122583 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.944395+0000", 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012613300000000001 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.944522+0000", 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019360698999999999 2024-02-12T16:50:26.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.963882+0000", 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000323679 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.964206+0000", 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4674e-05 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.964241+0000", 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17506511399999999 2024-02-12T16:50:26.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.139306+0000", 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19744788699999999 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:157643 9.1 9:821e6249:::benchmark_data_smithi169_124168_object6959:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:50:26.636 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:14.943052+0000", 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 371.64843089700003, 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199840459, 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157643 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.943052+0000", 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.943052+0000", 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.943051+0000", 2024-02-12T16:50:26.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.519e-06 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.943056+0000", 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4659999999999996e-06 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.943063+0000", 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0069999999999999e-06 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.943067+0000", 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0068218890000000003 2024-02-12T16:50:26.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.949889+0000", 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133905 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.950023+0000", 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013071799999999999 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.950153+0000", 2024-02-12T16:50:26.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.013736533 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:14.963890+0000", 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.17547883 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.139369+0000", 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5430999999999998e-05 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.139424+0000", 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0034686809999999999 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.142893+0000", 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199840459 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:158386 9.1 9:8104dbe8:::benchmark_data_smithi169_124168_object7702:head [delete] snapc 0=[] ondisk+write+known_if_redirected e953)", 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:15.851512+0000", 2024-02-12T16:50:26.642 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 370.73997153599998, 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.303392045, 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158386 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.851512+0000", 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.851512+0000", 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.851509+0000", 2024-02-12T16:50:26.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0026e-05 2024-02-12T16:50:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.851519+0000", 2024-02-12T16:50:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2274e-05 2024-02-12T16:50:26.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.851531+0000", 2024-02-12T16:50:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5289999999999997e-06 2024-02-12T16:50:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.851538+0000", 2024-02-12T16:50:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002791239 2024-02-12T16:50:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.854329+0000", 2024-02-12T16:50:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010089 2024-02-12T16:50:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.854430+0000", 2024-02-12T16:50:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104413 2024-02-12T16:50:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-02-12T16:50:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.854534+0000", 2024-02-12T16:50:26.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044288909999999999 2024-02-12T16:50:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:50:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.858963+0000", 2024-02-12T16:50:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29059449100000001 2024-02-12T16:50:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149558+0000", 2024-02-12T16:50:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106622 2024-02-12T16:50:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149664+0000", 2024-02-12T16:50:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0052394850000000003 2024-02-12T16:50:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.154904+0000", 2024-02-12T16:50:26.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.303392045 2024-02-12T16:50:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.653 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.26977.0:158386 9.1 e953/935 ondisk, result = 0)", 2024-02-12T16:50:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:15.862676+0000", 2024-02-12T16:50:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 370.72880738800001, 2024-02-12T16:50:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29225242499999998, 2024-02-12T16:50:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-02-12T16:50:26.654 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.862676+0000", 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.862676+0000", 2024-02-12T16:50:26.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:50:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.862676+0000", 2024-02-12T16:50:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3759999999999999e-06 2024-02-12T16:50:26.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.862679+0000", 2024-02-12T16:50:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3110000000000001e-06 2024-02-12T16:50:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.862682+0000", 2024-02-12T16:50:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9720000000000001e-06 2024-02-12T16:50:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.862685+0000", 2024-02-12T16:50:26.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28682546399999997 2024-02-12T16:50:26.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149511+0000", 2024-02-12T16:50:26.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0743e-05 2024-02-12T16:50:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149541+0000", 2024-02-12T16:50:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053869 2024-02-12T16:50:26.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.154928+0000", 2024-02-12T16:50:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29225242499999998 2024-02-12T16:50:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.661 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.26977.0:158403 9.3 e953/916 9:ce7a838e:::benchmark_data_smithi169_124168_object7719:head v 953'9918, mlcod=953'9916)", 2024-02-12T16:50:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:15.872111+0000", 2024-02-12T16:50:26.662 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 370.71937269199998, 2024-02-12T16:50:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30277821399999999, 2024-02-12T16:50:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.872111+0000", 2024-02-12T16:50:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.872111+0000", 2024-02-12T16:50:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.872108+0000", 2024-02-12T16:50:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.094e-05 2024-02-12T16:50:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:15.872119+0000", 2024-02-12T16:50:26.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277395636 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149515+0000", 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0752000000000001e-05 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149535+0000", 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117026 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149652+0000", 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1828999999999994e-05 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.149714+0000", 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025133694000000002 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.174848+0000", 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0821000000000001e-05 2024-02-12T16:50:26.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:16.174889+0000", 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30277821399999999 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192784 9.1 9.32f55671 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.043718+0000", 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.54776488800002, 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.98590883900000004, 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192784 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043718+0000", 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043718+0000", 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043715+0000", 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5410000000000006e-06 2024-02-12T16:50:26.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043725+0000", 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2668e-05 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043737+0000", 2024-02-12T16:50:26.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5359999999999998e-06 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043744+0000", 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010749 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043852+0000", 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2509e-05 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.043864+0000", 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.98573347200000005 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029597+0000", 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9726e-05 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029627+0000", 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.98590883900000004 2024-02-12T16:50:26.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192798 9.1 9.19e54d11 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.059436+0000", 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.53204729999999, 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.97022514900000001, 2024-02-12T16:50:26.678 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192798 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059436+0000", 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059436+0000", 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999914 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059427+0000", 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4865000000000001e-05 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059442+0000", 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6242000000000001e-05 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059458+0000", 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3760000000000002e-06 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059466+0000", 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9993000000000005e-05 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059546+0000", 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7586999999999999e-05 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.059563+0000", 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.97009120699999996 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029655+0000", 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5570000000000001e-06 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029661+0000", 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.97022514900000001 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192801 9.1 9.7b636171 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.063588+0000", 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.52789483399999, 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.96609848899999995, 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192801 2024-02-12T16:50:26.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063588+0000", 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063588+0000", 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063586+0000", 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6480000000000003e-06 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063594+0000", 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5608e-05 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063609+0000", 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6919999999999996e-06 2024-02-12T16:50:26.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063616+0000", 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.177e-05 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063678+0000", 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3061000000000001e-05 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.063691+0000", 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.96599105500000004 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029682+0000", 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9339999999999997e-06 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029687+0000", 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.96609848899999995 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192856 9.1 9.73cf0da1 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.136028+0000", 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.45545503199997, 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.893684119, 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192856 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136028+0000", 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136028+0000", 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136026+0000", 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6610000000000006e-06 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136035+0000", 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136046+0000", 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3759999999999996e-06 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136052+0000", 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2160999999999997e-05 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136104+0000", 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0599999999999997e-06 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.136113+0000", 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.89359413799999998 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029708+0000", 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.831e-06 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029712+0000", 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.893684119 2024-02-12T16:50:26.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192876 9.1 9.af209d01 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.166138+0000", 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.42534487799998, 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.86360691199999995, 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192876 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166138+0000", 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166138+0000", 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166135+0000", 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9690000000000006e-06 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166145+0000", 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7716000000000001e-05 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166163+0000", 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2840000000000006e-06 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166171+0000", 2024-02-12T16:50:26.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9659000000000005e-05 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166241+0000", 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4141999999999999e-05 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.166255+0000", 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.86348572199999996 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029741+0000", 2024-02-12T16:50:26.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7720000000000002e-06 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029745+0000", 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.86360691199999995 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192891 9.1 9.81418ab1 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.187004+0000", 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.40447907200002, 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.84276802799999995, 2024-02-12T16:50:26.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192891 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187004+0000", 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187004+0000", 2024-02-12T16:50:26.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187001+0000", 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1583000000000001e-05 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187013+0000", 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8413999999999999e-05 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187041+0000", 2024-02-12T16:50:26.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.973e-06 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187048+0000", 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4997000000000002e-05 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187103+0000", 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.044e-05 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.187113+0000", 2024-02-12T16:50:26.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.84265375099999995 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029767+0000", 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.045e-06 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029772+0000", 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.84276802799999995 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.702 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192901 9.1 9.27719f41 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.215264+0000", 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.37621903500002, 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.81453462700000001, 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192901 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215264+0000", 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215264+0000", 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215260+0000", 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2507000000000001e-05 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215273+0000", 2024-02-12T16:50:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2252e-05 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215295+0000", 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9179999999999994e-06 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215303+0000", 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3768000000000003e-05 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215367+0000", 2024-02-12T16:50:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0630000000000006e-06 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.215375+0000", 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.81441755000000005 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029792+0000", 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5169999999999996e-06 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029799+0000", 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.81453462700000001 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192921 9.1 9.a877b651 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.252965+0000", 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.33851805699999, 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.776867207, 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192921 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.252965+0000", 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.252965+0000", 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.252963+0000", 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0469999999999994e-06 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.252971+0000", 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6368999999999999e-05 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.252987+0000", 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.865e-06 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.252992+0000", 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7078999999999999e-05 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.253059+0000", 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.181e-05 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.253071+0000", 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77675591300000002 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029827+0000", 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2979999999999999e-06 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029832+0000", 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.776867207 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:192932 9.1 9.da944a51 (undecoded) ondisk+write+known_if_redirected e964)", 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:44:59.294828+0000", 2024-02-12T16:50:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 327.29665567299998, 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.73503404900000002, 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 192932 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294828+0000", 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294828+0000", 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294824+0000", 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3655000000000001e-05 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294838+0000", 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8986000000000001e-05 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294857+0000", 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8979999999999996e-06 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294865+0000", 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2460999999999996e-05 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294937+0000", 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5101e-05 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:44:59.294952+0000", 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.73490433300000002 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029857+0000", 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0100000000000003e-06 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:45:00.029862+0000", 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.73503404900000002 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.26977.0:254311 9.11 9:8b388cf4:::benchmark_data_smithi169_124168_object103627:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e991)", 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-02-12T16:46:11.731127+0000", 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 254.86035590899999, 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19930630599999999, 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.26977", 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 254311 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731127+0000", 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731127+0000", 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731127+0000", 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.64e-06 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-02-12T16:50:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731129+0000", 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2559999999999998e-06 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731132+0000", 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0320000000000002e-06 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731134+0000", 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055728600000000004 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731691+0000", 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3032e-05 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.731704+0000", 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.192946486 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.924651+0000", 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5327000000000004e-05 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.924746+0000", 2024-02-12T16:50:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000410035 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1,6,7", 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.925156+0000", 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021982460000000001 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.927354+0000", 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0027857060000000002 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.930140+0000", 2024-02-12T16:50:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5151999999999998e-05 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.930175+0000", 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5472999999999997e-05 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.930211+0000", 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8424999999999997e-05 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.930269+0000", 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016456399999999999 2024-02-12T16:50:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-02-12T16:46:11.930434+0000", 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19930630599999999 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-12T16:50:26.722 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-12T16:50:27.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:27 smithi121 bash[18489]: cluster 2024-02-12T16:50:26.407263+0000 mgr.y (mgr.14146) 1987 : cluster [DBG] pgmap v2942: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:27.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:27 smithi121 bash[18489]: cluster 2024-02-12T16:50:26.539547+0000 osd.10 (osd.10) 278 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:27.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:27 smithi121 bash[18489]: cluster 2024-02-12T16:50:26.539641+0000 osd.10 (osd.10) 279 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:27.549 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:27 smithi169 bash[19604]: cluster 2024-02-12T16:50:26.407263+0000 mgr.y (mgr.14146) 1987 : cluster [DBG] pgmap v2942: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:27.604 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:27 smithi169 bash[19604]: cluster 2024-02-12T16:50:26.539547+0000 osd.10 (osd.10) 278 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:27.604 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:27 smithi169 bash[19604]: cluster 2024-02-12T16:50:26.539641+0000 osd.10 (osd.10) 279 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:27 smithi123 bash[19119]: cluster 2024-02-12T16:50:26.407263+0000 mgr.y (mgr.14146) 1987 : cluster [DBG] pgmap v2942: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:27 smithi123 bash[19119]: cluster 2024-02-12T16:50:26.539547+0000 osd.10 (osd.10) 278 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:27 smithi123 bash[19119]: cluster 2024-02-12T16:50:26.539641+0000 osd.10 (osd.10) 279 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:27.826 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-12T16:50:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:28 smithi121 bash[18489]: cluster 2024-02-12T16:50:27.153221+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e1089: 12 total, 12 up, 12 in 2024-02-12T16:50:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:28 smithi121 bash[18489]: cluster 2024-02-12T16:50:27.243551+0000 osd.2 (osd.2) 157 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:50:28.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:28 smithi121 bash[18489]: cluster 2024-02-12T16:50:27.346249+0000 osd.2 (osd.2) 158 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:50:28.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:28 smithi121 bash[18489]: cluster 2024-02-12T16:50:27.477870+0000 osd.4 (osd.4) 191 : cluster [DBG] 1.e scrub starts 2024-02-12T16:50:28.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:28 smithi121 bash[18489]: cluster 2024-02-12T16:50:27.574568+0000 osd.4 (osd.4) 192 : cluster [DBG] 1.e scrub ok 2024-02-12T16:50:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:28 smithi169 bash[19604]: cluster 2024-02-12T16:50:27.153221+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e1089: 12 total, 12 up, 12 in 2024-02-12T16:50:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:28 smithi169 bash[19604]: cluster 2024-02-12T16:50:27.243551+0000 osd.2 (osd.2) 157 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:50:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:28 smithi169 bash[19604]: cluster 2024-02-12T16:50:27.346249+0000 osd.2 (osd.2) 158 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:50:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:28 smithi169 bash[19604]: cluster 2024-02-12T16:50:27.477870+0000 osd.4 (osd.4) 191 : cluster [DBG] 1.e scrub starts 2024-02-12T16:50:28.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:28 smithi169 bash[19604]: cluster 2024-02-12T16:50:27.574568+0000 osd.4 (osd.4) 192 : cluster [DBG] 1.e scrub ok 2024-02-12T16:50:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:28 smithi123 bash[19119]: cluster 2024-02-12T16:50:27.153221+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e1089: 12 total, 12 up, 12 in 2024-02-12T16:50:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:28 smithi123 bash[19119]: cluster 2024-02-12T16:50:27.243551+0000 osd.2 (osd.2) 157 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:50:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:28 smithi123 bash[19119]: cluster 2024-02-12T16:50:27.346249+0000 osd.2 (osd.2) 158 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:50:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:28 smithi123 bash[19119]: cluster 2024-02-12T16:50:27.477870+0000 osd.4 (osd.4) 191 : cluster [DBG] 1.e scrub starts 2024-02-12T16:50:28.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:28 smithi123 bash[19119]: cluster 2024-02-12T16:50:27.574568+0000 osd.4 (osd.4) 192 : cluster [DBG] 1.e scrub ok 2024-02-12T16:50:29.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:29 smithi169 bash[19604]: cluster 2024-02-12T16:50:28.407907+0000 mgr.y (mgr.14146) 1988 : cluster [DBG] pgmap v2944: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:29.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:29 smithi123 bash[19119]: cluster 2024-02-12T16:50:28.407907+0000 mgr.y (mgr.14146) 1988 : cluster [DBG] pgmap v2944: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:29 smithi121 bash[18489]: cluster 2024-02-12T16:50:28.407907+0000 mgr.y (mgr.14146) 1988 : cluster [DBG] pgmap v2944: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:30.185 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-02-12T16:50:30.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:30 smithi169 bash[19604]: audit 2024-02-12T16:50:29.311657+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? v1:172.21.15.121:0/1458807798' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:50:30.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:30 smithi123 bash[19119]: audit 2024-02-12T16:50:29.311657+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? v1:172.21.15.121:0/1458807798' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:50:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:30 smithi121 bash[18489]: audit 2024-02-12T16:50:29.311657+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? v1:172.21.15.121:0/1458807798' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-02-12T16:50:30.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 10 1 2024-02-12T16:50:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:31 smithi121 bash[18489]: audit 2024-02-12T16:50:30.173905+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.121:0/1458807798' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:50:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:31 smithi121 bash[18489]: cluster 2024-02-12T16:50:30.174023+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e1090: 12 total, 12 up, 12 in 2024-02-12T16:50:31.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:31 smithi121 bash[18489]: cluster 2024-02-12T16:50:30.408565+0000 mgr.y (mgr.14146) 1989 : cluster [DBG] pgmap v2946: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:31 smithi169 bash[19604]: audit 2024-02-12T16:50:30.173905+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.121:0/1458807798' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:50:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:31 smithi169 bash[19604]: cluster 2024-02-12T16:50:30.174023+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e1090: 12 total, 12 up, 12 in 2024-02-12T16:50:31.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:31 smithi169 bash[19604]: cluster 2024-02-12T16:50:30.408565+0000 mgr.y (mgr.14146) 1989 : cluster [DBG] pgmap v2946: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:31 smithi123 bash[19119]: audit 2024-02-12T16:50:30.173905+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.121:0/1458807798' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-02-12T16:50:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:31 smithi123 bash[19119]: cluster 2024-02-12T16:50:30.174023+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e1090: 12 total, 12 up, 12 in 2024-02-12T16:50:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:31 smithi123 bash[19119]: cluster 2024-02-12T16:50:30.408565+0000 mgr.y (mgr.14146) 1989 : cluster [DBG] pgmap v2946: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:32.629 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:50:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:33 smithi169 bash[19604]: cluster 2024-02-12T16:50:32.409358+0000 mgr.y (mgr.14146) 1990 : cluster [DBG] pgmap v2947: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:33.812 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:33 smithi123 bash[19119]: cluster 2024-02-12T16:50:32.409358+0000 mgr.y (mgr.14146) 1990 : cluster [DBG] pgmap v2947: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:33 smithi121 bash[18489]: cluster 2024-02-12T16:50:32.409358+0000 mgr.y (mgr.14146) 1990 : cluster [DBG] pgmap v2947: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:35.692 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:50:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:50:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2024-02-12T16:50:35.692 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:50:35.740 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:35 smithi123 bash[19119]: cluster 2024-02-12T16:50:34.410150+0000 mgr.y (mgr.14146) 1991 : cluster [DBG] pgmap v2948: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:35.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:35 smithi169 bash[19604]: cluster 2024-02-12T16:50:34.410150+0000 mgr.y (mgr.14146) 1991 : cluster [DBG] pgmap v2948: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:35 smithi121 bash[18489]: cluster 2024-02-12T16:50:34.410150+0000 mgr.y (mgr.14146) 1991 : cluster [DBG] pgmap v2948: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:36.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-12T16:50:37.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:37 smithi121 bash[18489]: cluster 2024-02-12T16:50:36.410932+0000 mgr.y (mgr.14146) 1992 : cluster [DBG] pgmap v2949: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:37.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:37 smithi121 bash[18489]: cluster 2024-02-12T16:50:36.556201+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:50:37.183 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:37 smithi121 bash[18489]: cluster 2024-02-12T16:50:36.706813+0000 osd.3 (osd.3) 145 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:50:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:37 smithi169 bash[19604]: cluster 2024-02-12T16:50:36.410932+0000 mgr.y (mgr.14146) 1992 : cluster [DBG] pgmap v2949: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:37 smithi169 bash[19604]: cluster 2024-02-12T16:50:36.556201+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:50:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:37 smithi169 bash[19604]: cluster 2024-02-12T16:50:36.706813+0000 osd.3 (osd.3) 145 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:50:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:37 smithi123 bash[19119]: cluster 2024-02-12T16:50:36.410932+0000 mgr.y (mgr.14146) 1992 : cluster [DBG] pgmap v2949: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:37 smithi123 bash[19119]: cluster 2024-02-12T16:50:36.556201+0000 osd.3 (osd.3) 144 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:50:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:37 smithi123 bash[19119]: cluster 2024-02-12T16:50:36.706813+0000 osd.3 (osd.3) 145 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:50:39.484 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.10 to 1 (10000) 2024-02-12T16:50:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:39 smithi169 bash[19604]: cluster 2024-02-12T16:50:38.411698+0000 mgr.y (mgr.14146) 1993 : cluster [DBG] pgmap v2950: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:39 smithi169 bash[19604]: audit 2024-02-12T16:50:39.164021+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? v1:172.21.15.121:0/2349593630' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:50:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:39 smithi123 bash[19119]: cluster 2024-02-12T16:50:38.411698+0000 mgr.y (mgr.14146) 1993 : cluster [DBG] pgmap v2950: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:39 smithi123 bash[19119]: audit 2024-02-12T16:50:39.164021+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? v1:172.21.15.121:0/2349593630' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:50:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:39 smithi121 bash[18489]: cluster 2024-02-12T16:50:38.411698+0000 mgr.y (mgr.14146) 1993 : cluster [DBG] pgmap v2950: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:39 smithi121 bash[18489]: audit 2024-02-12T16:50:39.164021+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? v1:172.21.15.121:0/2349593630' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:50:40.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 10 1 2024-02-12T16:50:40.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:40 smithi169 bash[19604]: audit 2024-02-12T16:50:39.473891+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? v1:172.21.15.121:0/2349593630' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:50:40.799 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:40 smithi169 bash[19604]: cluster 2024-02-12T16:50:39.474011+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e1091: 12 total, 12 up, 12 in 2024-02-12T16:50:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:40 smithi123 bash[19119]: audit 2024-02-12T16:50:39.473891+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? v1:172.21.15.121:0/2349593630' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:50:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:40 smithi123 bash[19119]: cluster 2024-02-12T16:50:39.474011+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e1091: 12 total, 12 up, 12 in 2024-02-12T16:50:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:40 smithi121 bash[18489]: audit 2024-02-12T16:50:39.473891+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? v1:172.21.15.121:0/2349593630' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:50:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:40 smithi121 bash[18489]: cluster 2024-02-12T16:50:39.474011+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e1091: 12 total, 12 up, 12 in 2024-02-12T16:50:41.159 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:50:41.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:41 smithi169 bash[19604]: cluster 2024-02-12T16:50:40.412378+0000 mgr.y (mgr.14146) 1994 : cluster [DBG] pgmap v2952: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:41.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:41 smithi123 bash[19119]: cluster 2024-02-12T16:50:40.412378+0000 mgr.y (mgr.14146) 1994 : cluster [DBG] pgmap v2952: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:41 smithi121 bash[18489]: cluster 2024-02-12T16:50:40.412378+0000 mgr.y (mgr.14146) 1994 : cluster [DBG] pgmap v2952: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:43 smithi169 bash[19604]: cluster 2024-02-12T16:50:42.413226+0000 mgr.y (mgr.14146) 1995 : cluster [DBG] pgmap v2953: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:43.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:43 smithi123 bash[19119]: cluster 2024-02-12T16:50:42.413226+0000 mgr.y (mgr.14146) 1995 : cluster [DBG] pgmap v2953: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:43 smithi121 bash[18489]: cluster 2024-02-12T16:50:42.413226+0000 mgr.y (mgr.14146) 1995 : cluster [DBG] pgmap v2953: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:44.047 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:50:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2024-02-12T16:50:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2024-02-12T16:50:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2024-02-12T16:50:44.047 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:50:44.731 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-12T16:50:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:45 smithi169 bash[19604]: cluster 2024-02-12T16:50:44.414030+0000 mgr.y (mgr.14146) 1996 : cluster [DBG] pgmap v2954: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:45.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:45 smithi123 bash[19119]: cluster 2024-02-12T16:50:44.414030+0000 mgr.y (mgr.14146) 1996 : cluster [DBG] pgmap v2954: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:45 smithi121 bash[18489]: cluster 2024-02-12T16:50:44.414030+0000 mgr.y (mgr.14146) 1996 : cluster [DBG] pgmap v2954: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:47.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:47 smithi121 bash[18489]: cluster 2024-02-12T16:50:46.414722+0000 mgr.y (mgr.14146) 1997 : cluster [DBG] pgmap v2955: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:47 smithi169 bash[19604]: cluster 2024-02-12T16:50:46.414722+0000 mgr.y (mgr.14146) 1997 : cluster [DBG] pgmap v2955: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:47.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:47 smithi123 bash[19119]: cluster 2024-02-12T16:50:46.414722+0000 mgr.y (mgr.14146) 1997 : cluster [DBG] pgmap v2955: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:49.488 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-02-12T16:50:49.533 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.b/config 2024-02-12T16:50:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:49 smithi169 bash[19604]: cluster 2024-02-12T16:50:48.415439+0000 mgr.y (mgr.14146) 1998 : cluster [DBG] pgmap v2956: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:49 smithi169 bash[19604]: audit 2024-02-12T16:50:48.683147+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? v1:172.21.15.121:0/1269603556' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:50:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:49 smithi123 bash[19119]: cluster 2024-02-12T16:50:48.415439+0000 mgr.y (mgr.14146) 1998 : cluster [DBG] pgmap v2956: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:49 smithi123 bash[19119]: audit 2024-02-12T16:50:48.683147+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? v1:172.21.15.121:0/1269603556' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:50:49.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:49 smithi121 bash[18489]: cluster 2024-02-12T16:50:48.415439+0000 mgr.y (mgr.14146) 1998 : cluster [DBG] pgmap v2956: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:49.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:49 smithi121 bash[18489]: audit 2024-02-12T16:50:48.683147+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? v1:172.21.15.121:0/1269603556' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-02-12T16:50:50.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 11 1 2024-02-12T16:50:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:50 smithi121 bash[18489]: audit 2024-02-12T16:50:49.476086+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? v1:172.21.15.121:0/1269603556' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:50:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:50 smithi121 bash[18489]: cluster 2024-02-12T16:50:49.476213+0000 mon.a (mon.0) 3606 : cluster [DBG] osdmap e1092: 12 total, 12 up, 12 in 2024-02-12T16:50:50.744 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:50 smithi123 bash[19119]: audit 2024-02-12T16:50:49.476086+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? v1:172.21.15.121:0/1269603556' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:50:50.744 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:50 smithi123 bash[19119]: cluster 2024-02-12T16:50:49.476213+0000 mon.a (mon.0) 3606 : cluster [DBG] osdmap e1092: 12 total, 12 up, 12 in 2024-02-12T16:50:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:50 smithi169 bash[19604]: audit 2024-02-12T16:50:49.476086+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? v1:172.21.15.121:0/1269603556' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-02-12T16:50:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:50 smithi169 bash[19604]: cluster 2024-02-12T16:50:49.476213+0000 mon.a (mon.0) 3606 : cluster [DBG] osdmap e1092: 12 total, 12 up, 12 in 2024-02-12T16:50:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:51 smithi169 bash[19604]: cluster 2024-02-12T16:50:50.416090+0000 mgr.y (mgr.14146) 1999 : cluster [DBG] pgmap v2958: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:51 smithi169 bash[19604]: cluster 2024-02-12T16:50:50.633738+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.d scrub starts 2024-02-12T16:50:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:51 smithi169 bash[19604]: cluster 2024-02-12T16:50:50.750419+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.d scrub ok 2024-02-12T16:50:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:51 smithi123 bash[19119]: cluster 2024-02-12T16:50:50.416090+0000 mgr.y (mgr.14146) 1999 : cluster [DBG] pgmap v2958: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:51 smithi123 bash[19119]: cluster 2024-02-12T16:50:50.633738+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.d scrub starts 2024-02-12T16:50:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:51 smithi123 bash[19119]: cluster 2024-02-12T16:50:50.750419+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.d scrub ok 2024-02-12T16:50:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:51 smithi121 bash[18489]: cluster 2024-02-12T16:50:50.416090+0000 mgr.y (mgr.14146) 1999 : cluster [DBG] pgmap v2958: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:51 smithi121 bash[18489]: cluster 2024-02-12T16:50:50.633738+0000 osd.7 (osd.7) 53 : cluster [DBG] 1.d scrub starts 2024-02-12T16:50:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:51 smithi121 bash[18489]: cluster 2024-02-12T16:50:50.750419+0000 osd.7 (osd.7) 54 : cluster [DBG] 1.d scrub ok 2024-02-12T16:50:52.447 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-02-12T16:50:52.447 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2024-02-12T16:50:52.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2024-02-12T16:50:52.448 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2024-02-12T16:50:52.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.448 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:11 9.d 9.a8d11bfd (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.448 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.175715+0000", 2024-02-12T16:50:52.448 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26589323600001, 2024-02-12T16:50:52.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64770053699999997, 2024-02-12T16:50:52.449 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.449 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.450 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 11 2024-02-12T16:50:52.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.450 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175715+0000", 2024-02-12T16:50:52.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175715+0000", 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175712+0000", 2024-02-12T16:50:52.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7687000000000002e-05 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175810+0000", 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9610999999999997e-05 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175850+0000", 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9979999999999998e-06 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175860+0000", 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64751687700000005 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823377+0000", 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8554999999999999e-05 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823415+0000", 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64770053699999997 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:24 9.d 9.1c59118d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.175929+0000", 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26567932199998, 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64759051999999995, 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.455 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 24 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175929+0000", 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175929+0000", 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.175927+0000", 2024-02-12T16:50:52.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021789300000000001 2024-02-12T16:50:52.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176145+0000", 2024-02-12T16:50:52.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7478999999999998e-05 2024-02-12T16:50:52.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176182+0000", 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8719999999999997e-06 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176192+0000", 2024-02-12T16:50:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64730112900000003 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823493+0000", 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5669000000000001e-05 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823519+0000", 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64759051999999995 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.460 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:26 9.d 9.83ee18ed (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.176227+0000", 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26538138400002, 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64731014499999995, 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 26 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.461 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176227+0000", 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176227+0000", 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:50:52.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176225+0000", 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000230604 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176456+0000", 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4382000000000001e-05 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176470+0000", 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9782000000000001e-05 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176490+0000", 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64704144900000005 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823531+0000", 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.152e-06 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823537+0000", 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64731014499999995 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:29 9.d 9.3cf8248d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.176576+0000", 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26503229000002, 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.646997551, 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.465 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 29 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176576+0000", 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176576+0000", 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176574+0000", 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014659699999999999 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176721+0000", 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2809999999999999e-05 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176734+0000", 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9348999999999999e-05 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.176773+0000", 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64679205900000003 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823565+0000", 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1899999999999995e-06 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823573+0000", 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.646997551 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.470 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:47 9.d 9.59200e7d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177055+0000", 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26455255000002, 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64655418099999995, 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 47 2024-02-12T16:50:52.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.472 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177055+0000", 2024-02-12T16:50:52.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177055+0000", 2024-02-12T16:50:52.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:50:52.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177054+0000", 2024-02-12T16:50:52.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6201e-05 2024-02-12T16:50:52.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177100+0000", 2024-02-12T16:50:52.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2798e-05 2024-02-12T16:50:52.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177113+0000", 2024-02-12T16:50:52.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5350000000000004e-06 2024-02-12T16:50:52.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177121+0000", 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64648264099999997 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823604+0000", 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7559999999999996e-06 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823610+0000", 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64655418099999995 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.478 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66 9.d 9.5f28cf2d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177228+0000", 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26437971600001, 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64642686800000004, 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.479 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177228+0000", 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177228+0000", 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177227+0000", 2024-02-12T16:50:52.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7198e-05 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177324+0000", 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0577999999999999e-05 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177334+0000", 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0620000000000007e-06 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177343+0000", 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64628943900000002 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823633+0000", 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2263e-05 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823655+0000", 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64642686800000004 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:69 9.d 9.6c15de4d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177361+0000", 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26424684099999, 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64630940299999995, 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.483 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 69 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177361+0000", 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177361+0000", 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177359+0000", 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3695000000000001e-05 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177403+0000", 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1970000000000001e-05 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177415+0000", 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7847999999999999e-05 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177433+0000", 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64623225900000003 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823665+0000", 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3229999999999997e-06 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823670+0000", 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64630940299999995 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:77 9.d 9.dc88835d (undecoded) ondisk+retry+write+known_if_redirected e923)", 2024-02-12T16:50:52.487 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:40.177450+0000", 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 492.26415742699999, 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64624936799999999, 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "reached pg", 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 77 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177450+0000", 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177450+0000", 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177449+0000", 2024-02-12T16:50:52.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1155999999999997e-05 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177490+0000", 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3303e-05 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177504+0000", 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0199999999999994e-06 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.177512+0000", 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64618072800000004 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823692+0000", 2024-02-12T16:50:52.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:40.823700+0000", 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.64624936799999999 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:16995 9.d 9:b1101a0d:::benchmark_data_smithi169_124168_object16994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e925)", 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:44.072076+0000", 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 488.36953237, 2024-02-12T16:50:52.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32354703800000001, 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 16995 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072076+0000", 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072076+0000", 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072074+0000", 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8467000000000002e-05 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072123+0000", 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2581e-05 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072135+0000", 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0599999999999996e-06 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072141+0000", 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6850000000000001e-05 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072178+0000", 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4089e-05 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072272+0000", 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7096999999999995e-05 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072369+0000", 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059913000000000002 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.072968+0000", 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32242240300000002 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395391+0000", 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133757 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395524+0000", 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8088e-05 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395623+0000", 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32354703800000001 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:17001 9.d 9:bc5e5ff6:::benchmark_data_smithi169_124168_object17000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e925)", 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:42:44.076282+0000", 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 488.36532613100002, 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31989659399999998, 2024-02-12T16:50:52.498 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 17001 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076282+0000", 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076282+0000", 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076280+0000", 2024-02-12T16:50:52.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000147696 2024-02-12T16:50:52.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076428+0000", 2024-02-12T16:50:52.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2692e-05 2024-02-12T16:50:52.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076441+0000", 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3680000000000004e-06 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076445+0000", 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8293000000000001e-05 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076483+0000", 2024-02-12T16:50:52.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012841400000000001 2024-02-12T16:50:52.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076612+0000", 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017282800000000001 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.076785+0000", 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050753599999999995 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.077292+0000", 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31846323399999998 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395755+0000", 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000151831 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.395907+0000", 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027103400000000002 2024-02-12T16:50:52.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:42:44.396178+0000", 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31989659399999998 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66895 9.d 9:bbd5c3e8:::benchmark_data_smithi169_124168_object66894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.209519+0000", 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 471.23208870100001, 2024-02-12T16:50:52.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42805788099999997, 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66895 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209519+0000", 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209519+0000", 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209517+0000", 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5612000000000005e-05 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209612+0000", 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1770000000000001e-05 2024-02-12T16:50:52.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209624+0000", 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1610000000000002e-06 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209629+0000", 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5796e-05 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209675+0000", 2024-02-12T16:50:52.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102526 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209778+0000", 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107908 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.209885+0000", 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001571228 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.211457+0000", 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42588541000000002 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637342+0000", 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000154523 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637497+0000", 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0442e-05 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.637577+0000", 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42805788099999997 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66902 9.d 9:b99a6cc2:::benchmark_data_smithi169_124168_object66901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.215250+0000", 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 471.226358151, 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42328817499999999, 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.513 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66902 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215250+0000", 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215250+0000", 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215249+0000", 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6382e-05 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215345+0000", 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.374e-06 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215353+0000", 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.101e-06 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215356+0000", 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8970000000000001e-05 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215395+0000", 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8777999999999996e-05 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215483+0000", 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7287999999999994e-05 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215581+0000", 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001935855 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.217517+0000", 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42092468100000002 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638441+0000", 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3287e-05 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638495+0000", 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3380000000000001e-05 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638538+0000", 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42328817499999999 2024-02-12T16:50:52.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:66906 9.d 9:bfe77eb9:::benchmark_data_smithi169_124168_object66905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:43:01.215599+0000", 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 471.22600861400002, 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42307459600000002, 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 66906 2024-02-12T16:50:52.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215599+0000", 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215599+0000", 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215598+0000", 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0842999999999998e-05 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215679+0000", 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.216e-06 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215686+0000", 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5140000000000003e-06 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.215691+0000", 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078608499999999995 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.216477+0000", 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2298000000000004e-05 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.216559+0000", 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012651700000000001 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.216686+0000", 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00096934199999999999 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.217655+0000", 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42093917199999997 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638594+0000", 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3069000000000003e-05 2024-02-12T16:50:52.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638647+0000", 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6625999999999999e-05 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:43:01.638674+0000", 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.42307459600000002 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:190681 9.d 9:b10494f5:::benchmark_data_smithi169_124168_object39997:head [delete] snapc 0=[] ondisk+write+known_if_redirected e962)", 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:56.139260+0000", 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 356.30234756499999, 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34157827200000002, 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 190681 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.525 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139260+0000", 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139260+0000", 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139259+0000", 2024-02-12T16:50:52.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8810000000000001e-06 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139264+0000", 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0587999999999999e-05 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139275+0000", 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2410000000000003e-06 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139280+0000", 2024-02-12T16:50:52.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3096999999999997e-05 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139333+0000", 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013068199999999999 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139464+0000", 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013298300000000001 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.139597+0000", 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099421409999999998 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.149539+0000", 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.3311229 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.480662+0000", 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9438000000000006e-05 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.480762+0000", 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7084999999999998e-05 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.480839+0000", 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.34157827200000002 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:190685 9.d 9:b92e8f1a:::benchmark_data_smithi169_124168_object40001:head [delete] snapc 0=[] ondisk+write+known_if_redirected e962)", 2024-02-12T16:50:52.530 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:56.144302+0000", 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 356.29730621099998, 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.33674124799999999, 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 190685 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144302+0000", 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144302+0000", 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144299+0000", 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1000000000000004e-06 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144307+0000", 2024-02-12T16:50:52.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1881e-05 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144319+0000", 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9899999999999997e-06 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144324+0000", 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5302999999999995e-05 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144389+0000", 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113212 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144503+0000", 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4234999999999995e-05 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.144597+0000", 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0089107230000000006 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.153508+0000", 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32741821999999998 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.480926+0000", 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7076999999999995e-05 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.480993+0000", 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0056000000000003e-05 2024-02-12T16:50:52.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481043+0000", 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.33674124799999999 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:190689 9.d 9:bbf42016:::benchmark_data_smithi169_124168_object40005:head [delete] snapc 0=[] ondisk+write+known_if_redirected e962)", 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:56.145819+0000", 2024-02-12T16:50:52.536 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 356.29578845999998, 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.33539933500000002, 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 190689 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.145819+0000", 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.145819+0000", 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.145818+0000", 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7519999999999998e-06 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.145822+0000", 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1160000000000001e-06 2024-02-12T16:50:52.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.145829+0000", 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4539999999999999e-06 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.145833+0000", 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003733044 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.149566+0000", 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013600400000000001 2024-02-12T16:50:52.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.149702+0000", 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015870899999999999 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.149861+0000", 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081975959999999997 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.158058+0000", 2024-02-12T16:50:52.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32306120399999999 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481119+0000", 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2599000000000002e-05 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481172+0000", 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6841000000000001e-05 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481219+0000", 2024-02-12T16:50:52.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.33539933500000002 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:190692 9.d 9:b3080036:::benchmark_data_smithi169_124168_object40008:head [delete] snapc 0=[] ondisk+write+known_if_redirected e962)", 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:56.152850+0000", 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 356.28875779800001, 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32854391999999999, 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.542 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 190692 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.152850+0000", 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.152850+0000", 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.152848+0000", 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3790000000000006e-06 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.152856+0000", 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5792000000000001e-05 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.152872+0000", 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7789999999999998e-06 2024-02-12T16:50:52.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.152878+0000", 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064731299999999995 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.153526+0000", 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9526999999999999e-05 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.153605+0000", 2024-02-12T16:50:52.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6339e-05 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.153702+0000", 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081876180000000007 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.161889+0000", 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31940675699999999 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481296+0000", 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2917000000000002e-05 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481349+0000", 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5058000000000001e-05 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481394+0000", 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32854391999999999 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:190694 9.d 9:b792f13d:::benchmark_data_smithi169_124168_object40010:head [delete] snapc 0=[] ondisk+write+known_if_redirected e962)", 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:44:56.155108+0000", 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 356.28649964700003, 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32644309500000002, 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 190694 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.155108+0000", 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.155108+0000", 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.155106+0000", 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.328e-06 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.155114+0000", 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1770000000000001e-05 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.155125+0000", 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0091e-05 2024-02-12T16:50:52.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.155136+0000", 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029586590000000002 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.158094+0000", 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133516 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.158228+0000", 2024-02-12T16:50:52.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016235500000000001 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.158390+0000", 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036480390000000001 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.162038+0000", 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.31943509399999998 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481473+0000", 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2193000000000001e-05 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481515+0000", 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5941e-05 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:44:56.481551+0000", 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.32644309500000002 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.553 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236351 9.d 9:b7d08d3e:::benchmark_data_smithi169_124168_object85667:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.425838+0000", 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 302.015769433, 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47076204199999999, 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236351 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.554 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425838+0000", 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425838+0000", 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425836+0000", 2024-02-12T16:50:52.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0389e-05 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425846+0000", 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0373e-05 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425866+0000", 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8800000000000008e-06 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425874+0000", 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7064999999999996e-05 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.425941+0000", 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5775999999999999e-05 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.426037+0000", 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7157999999999995e-05 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.426124+0000", 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041037859999999999 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.430228+0000", 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46621120999999999 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.896439+0000", 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112011 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.896551+0000", 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9326000000000002e-05 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.896600+0000", 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.47076204199999999 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.26977.0:236356 9.d 9:b756670f:::benchmark_data_smithi169_124168_object85672:head [delete] snapc 0=[] ondisk+write+known_if_redirected e975)", 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2024-02-12T16:45:50.433590+0000", 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 302.00801812499998, 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46314349300000002, 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.26977", 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.169:0/3650630517", 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 236356 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2024-02-12T16:50:52.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433590+0000", 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433590+0000", 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433587+0000", 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6420000000000007e-06 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433595+0000", 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5152e-05 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433610+0000", 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.793e-06 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433617+0000", 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8615999999999997e-05 2024-02-12T16:50:52.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433665+0000", 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121931 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433787+0000", 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146119 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1", 2024-02-12T16:50:52.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.433933+0000", 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049517889999999998 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.438885+0000", 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.45776330199999998 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.896649+0000", 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5101000000000001e-05 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.896694+0000", 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9582999999999997e-05 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2024-02-12T16:45:50.896733+0000", 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.46314349300000002 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-02-12T16:50:52.565 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-02-12T16:50:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:53 smithi169 bash[19604]: cluster 2024-02-12T16:50:52.416852+0000 mgr.y (mgr.14146) 2000 : cluster [DBG] pgmap v2959: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:53 smithi123 bash[19119]: cluster 2024-02-12T16:50:52.416852+0000 mgr.y (mgr.14146) 2000 : cluster [DBG] pgmap v2959: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:53 smithi121 bash[18489]: cluster 2024-02-12T16:50:52.416852+0000 mgr.y (mgr.14146) 2000 : cluster [DBG] pgmap v2959: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:55.046 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:50:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:55 smithi169 bash[19604]: cluster 2024-02-12T16:50:54.417489+0000 mgr.y (mgr.14146) 2001 : cluster [DBG] pgmap v2960: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:55 smithi123 bash[19119]: cluster 2024-02-12T16:50:54.417489+0000 mgr.y (mgr.14146) 2001 : cluster [DBG] pgmap v2960: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:55 smithi121 bash[18489]: cluster 2024-02-12T16:50:54.417489+0000 mgr.y (mgr.14146) 2001 : cluster [DBG] pgmap v2960: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:57 smithi169 bash[19604]: cluster 2024-02-12T16:50:56.199370+0000 osd.1 (osd.1) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:57 smithi169 bash[19604]: cluster 2024-02-12T16:50:56.199446+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:57 smithi169 bash[19604]: cluster 2024-02-12T16:50:56.418127+0000 mgr.y (mgr.14146) 2002 : cluster [DBG] pgmap v2961: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:57 smithi123 bash[19119]: cluster 2024-02-12T16:50:56.199370+0000 osd.1 (osd.1) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:57 smithi123 bash[19119]: cluster 2024-02-12T16:50:56.199446+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:57 smithi123 bash[19119]: cluster 2024-02-12T16:50:56.418127+0000 mgr.y (mgr.14146) 2002 : cluster [DBG] pgmap v2961: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:57 smithi121 bash[18489]: cluster 2024-02-12T16:50:56.199370+0000 osd.1 (osd.1) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:57 smithi121 bash[18489]: cluster 2024-02-12T16:50:56.199446+0000 osd.1 (osd.1) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:57 smithi121 bash[18489]: cluster 2024-02-12T16:50:56.418127+0000 mgr.y (mgr.14146) 2002 : cluster [DBG] pgmap v2961: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:58.456 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:58 smithi121 bash[18489]: cluster 2024-02-12T16:50:56.484033+0000 osd.8 (osd.8) 173 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:50:58.456 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:58 smithi121 bash[18489]: cluster 2024-02-12T16:50:56.556646+0000 osd.8 (osd.8) 174 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:50:58.456 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:58 smithi121 bash[18489]: cluster 2024-02-12T16:50:56.765561+0000 osd.11 (osd.11) 253 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:58.456 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:58 smithi121 bash[18489]: cluster 2024-02-12T16:50:56.765675+0000 osd.11 (osd.11) 254 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:58.457 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:58 smithi121 bash[18489]: cluster 2024-02-12T16:50:57.163664+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e1093: 12 total, 12 up, 12 in 2024-02-12T16:50:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:58 smithi169 bash[19604]: cluster 2024-02-12T16:50:56.484033+0000 osd.8 (osd.8) 173 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:50:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:58 smithi169 bash[19604]: cluster 2024-02-12T16:50:56.556646+0000 osd.8 (osd.8) 174 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:50:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:58 smithi169 bash[19604]: cluster 2024-02-12T16:50:56.765561+0000 osd.11 (osd.11) 253 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:58 smithi169 bash[19604]: cluster 2024-02-12T16:50:56.765675+0000 osd.11 (osd.11) 254 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:58 smithi169 bash[19604]: cluster 2024-02-12T16:50:57.163664+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e1093: 12 total, 12 up, 12 in 2024-02-12T16:50:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:58 smithi123 bash[19119]: cluster 2024-02-12T16:50:56.484033+0000 osd.8 (osd.8) 173 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:50:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:58 smithi123 bash[19119]: cluster 2024-02-12T16:50:56.556646+0000 osd.8 (osd.8) 174 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:50:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:58 smithi123 bash[19119]: cluster 2024-02-12T16:50:56.765561+0000 osd.11 (osd.11) 253 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:50:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:58 smithi123 bash[19119]: cluster 2024-02-12T16:50:56.765675+0000 osd.11 (osd.11) 254 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:50:58.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:58 smithi123 bash[19119]: cluster 2024-02-12T16:50:57.163664+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e1093: 12 total, 12 up, 12 in 2024-02-12T16:50:59.184 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.11 to 1 (10000) 2024-02-12T16:50:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:59 smithi169 bash[19604]: cluster 2024-02-12T16:50:57.615086+0000 osd.4 (osd.4) 193 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:50:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:59 smithi169 bash[19604]: cluster 2024-02-12T16:50:57.665110+0000 osd.4 (osd.4) 194 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:50:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:59 smithi169 bash[19604]: cluster 2024-02-12T16:50:58.418861+0000 mgr.y (mgr.14146) 2003 : cluster [DBG] pgmap v2963: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:50:59 smithi169 bash[19604]: audit 2024-02-12T16:50:58.803381+0000 mon.a (mon.0) 3608 : audit [INF] from='client.? v1:172.21.15.121:0/1952419097' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:50:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:59 smithi123 bash[19119]: cluster 2024-02-12T16:50:57.615086+0000 osd.4 (osd.4) 193 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:50:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:59 smithi123 bash[19119]: cluster 2024-02-12T16:50:57.665110+0000 osd.4 (osd.4) 194 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:50:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:59 smithi123 bash[19119]: cluster 2024-02-12T16:50:58.418861+0000 mgr.y (mgr.14146) 2003 : cluster [DBG] pgmap v2963: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:50:59 smithi123 bash[19119]: audit 2024-02-12T16:50:58.803381+0000 mon.a (mon.0) 3608 : audit [INF] from='client.? v1:172.21.15.121:0/1952419097' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:50:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:59 smithi121 bash[18489]: cluster 2024-02-12T16:50:57.615086+0000 osd.4 (osd.4) 193 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:50:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:59 smithi121 bash[18489]: cluster 2024-02-12T16:50:57.665110+0000 osd.4 (osd.4) 194 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:50:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:59 smithi121 bash[18489]: cluster 2024-02-12T16:50:58.418861+0000 mgr.y (mgr.14146) 2003 : cluster [DBG] pgmap v2963: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:50:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:50:59 smithi121 bash[18489]: audit 2024-02-12T16:50:58.803381+0000 mon.a (mon.0) 3608 : audit [INF] from='client.? v1:172.21.15.121:0/1952419097' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:50:59.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 11 1 2024-02-12T16:51:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:00 smithi169 bash[19604]: cluster 2024-02-12T16:50:58.933976+0000 osd.9 (osd.9) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:00 smithi169 bash[19604]: cluster 2024-02-12T16:50:58.934063+0000 osd.9 (osd.9) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:00 smithi169 bash[19604]: audit 2024-02-12T16:50:59.175394+0000 mon.a (mon.0) 3609 : audit [INF] from='client.? v1:172.21.15.121:0/1952419097' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:51:00.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:00 smithi169 bash[19604]: cluster 2024-02-12T16:50:59.175476+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e1094: 12 total, 12 up, 12 in 2024-02-12T16:51:00.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:00 smithi123 bash[19119]: cluster 2024-02-12T16:50:58.933976+0000 osd.9 (osd.9) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:00 smithi123 bash[19119]: cluster 2024-02-12T16:50:58.934063+0000 osd.9 (osd.9) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:00 smithi123 bash[19119]: audit 2024-02-12T16:50:59.175394+0000 mon.a (mon.0) 3609 : audit [INF] from='client.? v1:172.21.15.121:0/1952419097' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:51:00.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:00 smithi123 bash[19119]: cluster 2024-02-12T16:50:59.175476+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e1094: 12 total, 12 up, 12 in 2024-02-12T16:51:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:00 smithi121 bash[18489]: cluster 2024-02-12T16:50:58.933976+0000 osd.9 (osd.9) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:00 smithi121 bash[18489]: cluster 2024-02-12T16:50:58.934063+0000 osd.9 (osd.9) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:00 smithi121 bash[18489]: audit 2024-02-12T16:50:59.175394+0000 mon.a (mon.0) 3609 : audit [INF] from='client.? v1:172.21.15.121:0/1952419097' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:51:00.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:00 smithi121 bash[18489]: cluster 2024-02-12T16:50:59.175476+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e1094: 12 total, 12 up, 12 in 2024-02-12T16:51:01.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:01 smithi169 bash[19604]: cluster 2024-02-12T16:51:00.419619+0000 mgr.y (mgr.14146) 2004 : cluster [DBG] pgmap v2965: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:01.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:01 smithi123 bash[19119]: cluster 2024-02-12T16:51:00.419619+0000 mgr.y (mgr.14146) 2004 : cluster [DBG] pgmap v2965: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:01.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:01 smithi121 bash[18489]: cluster 2024-02-12T16:51:00.419619+0000 mgr.y (mgr.14146) 2004 : cluster [DBG] pgmap v2965: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:02 smithi169 bash[19604]: cluster 2024-02-12T16:51:00.565881+0000 osd.11 (osd.11) 255 : cluster [DBG] 1.b scrub starts 2024-02-12T16:51:02.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:02 smithi169 bash[19604]: cluster 2024-02-12T16:51:00.729686+0000 osd.11 (osd.11) 256 : cluster [DBG] 1.b scrub ok 2024-02-12T16:51:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:02 smithi123 bash[19119]: cluster 2024-02-12T16:51:00.565881+0000 osd.11 (osd.11) 255 : cluster [DBG] 1.b scrub starts 2024-02-12T16:51:02.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:02 smithi123 bash[19119]: cluster 2024-02-12T16:51:00.729686+0000 osd.11 (osd.11) 256 : cluster [DBG] 1.b scrub ok 2024-02-12T16:51:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:02 smithi121 bash[18489]: cluster 2024-02-12T16:51:00.565881+0000 osd.11 (osd.11) 255 : cluster [DBG] 1.b scrub starts 2024-02-12T16:51:02.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:02 smithi121 bash[18489]: cluster 2024-02-12T16:51:00.729686+0000 osd.11 (osd.11) 256 : cluster [DBG] 1.b scrub ok 2024-02-12T16:51:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:03 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.012322+0000 osd.5 (osd.5) 253 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:51:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:03 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.096401+0000 osd.5 (osd.5) 254 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:51:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:03 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.420331+0000 mgr.y (mgr.14146) 2005 : cluster [DBG] pgmap v2966: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:03 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.916357+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:51:03.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:03 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.981755+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:51:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:03 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.012322+0000 osd.5 (osd.5) 253 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:51:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:03 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.096401+0000 osd.5 (osd.5) 254 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:51:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:03 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.420331+0000 mgr.y (mgr.14146) 2005 : cluster [DBG] pgmap v2966: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:03 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.916357+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:51:03.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:03 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.981755+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:51:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:03 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.012322+0000 osd.5 (osd.5) 253 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:51:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:03 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.096401+0000 osd.5 (osd.5) 254 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:51:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:03 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.420331+0000 mgr.y (mgr.14146) 2005 : cluster [DBG] pgmap v2966: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:03 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.916357+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:51:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:03 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.981755+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:51:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:04 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.861301+0000 osd.5 (osd.5) 255 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:04 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.861390+0000 osd.5 (osd.5) 256 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:04 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.993442+0000 osd.3 (osd.3) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:04 smithi169 bash[19604]: cluster 2024-02-12T16:51:02.993569+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:04.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:04 smithi169 bash[19604]: cluster 2024-02-12T16:51:03.198986+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e1095: 12 total, 12 up, 12 in 2024-02-12T16:51:04.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:04 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.861301+0000 osd.5 (osd.5) 255 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:04.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:04 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.861390+0000 osd.5 (osd.5) 256 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:04.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:04 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.993442+0000 osd.3 (osd.3) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:04.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:04 smithi123 bash[19119]: cluster 2024-02-12T16:51:02.993569+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:04.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:04 smithi123 bash[19119]: cluster 2024-02-12T16:51:03.198986+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e1095: 12 total, 12 up, 12 in 2024-02-12T16:51:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:04 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.861301+0000 osd.5 (osd.5) 255 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:04 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.861390+0000 osd.5 (osd.5) 256 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:04 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.993442+0000 osd.3 (osd.3) 146 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:04 smithi121 bash[18489]: cluster 2024-02-12T16:51:02.993569+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:04.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:04 smithi121 bash[18489]: cluster 2024-02-12T16:51:03.198986+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e1095: 12 total, 12 up, 12 in 2024-02-12T16:51:04.710 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:05 smithi169 bash[19604]: cluster 2024-02-12T16:51:04.421103+0000 mgr.y (mgr.14146) 2006 : cluster [DBG] pgmap v2968: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:05 smithi169 bash[19604]: cluster 2024-02-12T16:51:04.672761+0000 osd.6 (osd.6) 210 : cluster [DBG] 1.f scrub starts 2024-02-12T16:51:05.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:05 smithi169 bash[19604]: cluster 2024-02-12T16:51:04.815753+0000 osd.6 (osd.6) 211 : cluster [DBG] 1.f scrub ok 2024-02-12T16:51:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:05 smithi123 bash[19119]: cluster 2024-02-12T16:51:04.421103+0000 mgr.y (mgr.14146) 2006 : cluster [DBG] pgmap v2968: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:05 smithi123 bash[19119]: cluster 2024-02-12T16:51:04.672761+0000 osd.6 (osd.6) 210 : cluster [DBG] 1.f scrub starts 2024-02-12T16:51:05.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:05 smithi123 bash[19119]: cluster 2024-02-12T16:51:04.815753+0000 osd.6 (osd.6) 211 : cluster [DBG] 1.f scrub ok 2024-02-12T16:51:05.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:05 smithi121 bash[18489]: cluster 2024-02-12T16:51:04.421103+0000 mgr.y (mgr.14146) 2006 : cluster [DBG] pgmap v2968: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:05.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:05 smithi121 bash[18489]: cluster 2024-02-12T16:51:04.672761+0000 osd.6 (osd.6) 210 : cluster [DBG] 1.f scrub starts 2024-02-12T16:51:05.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:05 smithi121 bash[18489]: cluster 2024-02-12T16:51:04.815753+0000 osd.6 (osd.6) 211 : cluster [DBG] 1.f scrub ok 2024-02-12T16:51:06.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:06 smithi121 bash[18489]: cluster 2024-02-12T16:51:04.519534+0000 osd.8 (osd.8) 175 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:51:06.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:06 smithi121 bash[18489]: cluster 2024-02-12T16:51:04.640209+0000 osd.8 (osd.8) 176 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:51:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:06 smithi169 bash[19604]: cluster 2024-02-12T16:51:04.519534+0000 osd.8 (osd.8) 175 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:51:06.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:06 smithi169 bash[19604]: cluster 2024-02-12T16:51:04.640209+0000 osd.8 (osd.8) 176 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:51:06.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:06 smithi123 bash[19119]: cluster 2024-02-12T16:51:04.519534+0000 osd.8 (osd.8) 175 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:51:06.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:06 smithi123 bash[19119]: cluster 2024-02-12T16:51:04.640209+0000 osd.8 (osd.8) 176 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:51:07.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:07 smithi169 bash[19604]: cluster 2024-02-12T16:51:06.421774+0000 mgr.y (mgr.14146) 2007 : cluster [DBG] pgmap v2969: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:07.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:07 smithi123 bash[19119]: cluster 2024-02-12T16:51:06.421774+0000 mgr.y (mgr.14146) 2007 : cluster [DBG] pgmap v2969: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:07.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:07 smithi121 bash[18489]: cluster 2024-02-12T16:51:06.421774+0000 mgr.y (mgr.14146) 2007 : cluster [DBG] pgmap v2969: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:08 smithi169 bash[19604]: cluster 2024-02-12T16:51:06.857855+0000 osd.10 (osd.10) 280 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:51:08.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:08 smithi169 bash[19604]: cluster 2024-02-12T16:51:06.957591+0000 osd.10 (osd.10) 281 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:51:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:08 smithi123 bash[19119]: cluster 2024-02-12T16:51:06.857855+0000 osd.10 (osd.10) 280 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:51:08.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:08 smithi123 bash[19119]: cluster 2024-02-12T16:51:06.957591+0000 osd.10 (osd.10) 281 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:51:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:08 smithi121 bash[18489]: cluster 2024-02-12T16:51:06.857855+0000 osd.10 (osd.10) 280 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:51:08.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:08 smithi121 bash[18489]: cluster 2024-02-12T16:51:06.957591+0000 osd.10 (osd.10) 281 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:51:09.246 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-02-12T16:51:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:09 smithi169 bash[19604]: audit 2024-02-12T16:51:08.385543+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? v1:172.21.15.121:0/2803697115' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:51:09.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:09 smithi169 bash[19604]: cluster 2024-02-12T16:51:08.422400+0000 mgr.y (mgr.14146) 2008 : cluster [DBG] pgmap v2970: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:09.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:09 smithi123 bash[19119]: audit 2024-02-12T16:51:08.385543+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? v1:172.21.15.121:0/2803697115' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:51:09.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:09 smithi123 bash[19119]: cluster 2024-02-12T16:51:08.422400+0000 mgr.y (mgr.14146) 2008 : cluster [DBG] pgmap v2970: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:09.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:09 smithi121 bash[18489]: audit 2024-02-12T16:51:08.385543+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? v1:172.21.15.121:0/2803697115' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-02-12T16:51:09.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:09 smithi121 bash[18489]: cluster 2024-02-12T16:51:08.422400+0000 mgr.y (mgr.14146) 2008 : cluster [DBG] pgmap v2970: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:10.015 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 3 1 2024-02-12T16:51:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:10 smithi121 bash[18489]: cluster 2024-02-12T16:51:08.816696+0000 osd.10 (osd.10) 282 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:51:10.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:10 smithi121 bash[18489]: cluster 2024-02-12T16:51:08.949586+0000 osd.10 (osd.10) 283 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:51:10.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:10 smithi121 bash[18489]: audit 2024-02-12T16:51:09.236859+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? v1:172.21.15.121:0/2803697115' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:51:10.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:10 smithi121 bash[18489]: cluster 2024-02-12T16:51:09.236930+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e1096: 12 total, 12 up, 12 in 2024-02-12T16:51:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:10 smithi169 bash[19604]: cluster 2024-02-12T16:51:08.816696+0000 osd.10 (osd.10) 282 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:51:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:10 smithi169 bash[19604]: cluster 2024-02-12T16:51:08.949586+0000 osd.10 (osd.10) 283 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:51:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:10 smithi169 bash[19604]: audit 2024-02-12T16:51:09.236859+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? v1:172.21.15.121:0/2803697115' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:51:10.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:10 smithi169 bash[19604]: cluster 2024-02-12T16:51:09.236930+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e1096: 12 total, 12 up, 12 in 2024-02-12T16:51:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:10 smithi123 bash[19119]: cluster 2024-02-12T16:51:08.816696+0000 osd.10 (osd.10) 282 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:51:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:10 smithi123 bash[19119]: cluster 2024-02-12T16:51:08.949586+0000 osd.10 (osd.10) 283 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:51:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:10 smithi123 bash[19119]: audit 2024-02-12T16:51:09.236859+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? v1:172.21.15.121:0/2803697115' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-02-12T16:51:10.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:10 smithi123 bash[19119]: cluster 2024-02-12T16:51:09.236930+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e1096: 12 total, 12 up, 12 in 2024-02-12T16:51:11.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:11 smithi169 bash[19604]: cluster 2024-02-12T16:51:10.423125+0000 mgr.y (mgr.14146) 2009 : cluster [DBG] pgmap v2972: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:11.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:11 smithi123 bash[19119]: cluster 2024-02-12T16:51:10.423125+0000 mgr.y (mgr.14146) 2009 : cluster [DBG] pgmap v2972: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:11.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:11 smithi121 bash[18489]: cluster 2024-02-12T16:51:10.423125+0000 mgr.y (mgr.14146) 2009 : cluster [DBG] pgmap v2972: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:13.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:13 smithi169 bash[19604]: cluster 2024-02-12T16:51:12.423982+0000 mgr.y (mgr.14146) 2010 : cluster [DBG] pgmap v2973: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:13.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:13 smithi123 bash[19119]: cluster 2024-02-12T16:51:12.423982+0000 mgr.y (mgr.14146) 2010 : cluster [DBG] pgmap v2973: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:13 smithi121 bash[18489]: cluster 2024-02-12T16:51:12.423982+0000 mgr.y (mgr.14146) 2010 : cluster [DBG] pgmap v2973: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:14 smithi169 bash[19604]: cluster 2024-02-12T16:51:13.215002+0000 osd.1 (osd.1) 206 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:51:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:14 smithi169 bash[19604]: cluster 2024-02-12T16:51:13.428326+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:51:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:14 smithi169 bash[19604]: cluster 2024-02-12T16:51:13.535946+0000 osd.11 (osd.11) 257 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:51:14.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:14 smithi169 bash[19604]: cluster 2024-02-12T16:51:13.560657+0000 osd.11 (osd.11) 258 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:51:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:14 smithi123 bash[19119]: cluster 2024-02-12T16:51:13.215002+0000 osd.1 (osd.1) 206 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:51:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:14 smithi123 bash[19119]: cluster 2024-02-12T16:51:13.428326+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:51:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:14 smithi123 bash[19119]: cluster 2024-02-12T16:51:13.535946+0000 osd.11 (osd.11) 257 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:51:14.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:14 smithi123 bash[19119]: cluster 2024-02-12T16:51:13.560657+0000 osd.11 (osd.11) 258 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:51:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:14 smithi121 bash[18489]: cluster 2024-02-12T16:51:13.215002+0000 osd.1 (osd.1) 206 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:51:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:14 smithi121 bash[18489]: cluster 2024-02-12T16:51:13.428326+0000 osd.1 (osd.1) 207 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:51:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:14 smithi121 bash[18489]: cluster 2024-02-12T16:51:13.535946+0000 osd.11 (osd.11) 257 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:51:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:14 smithi121 bash[18489]: cluster 2024-02-12T16:51:13.560657+0000 osd.11 (osd.11) 258 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:51:15.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:15 smithi169 bash[19604]: cluster 2024-02-12T16:51:14.424758+0000 mgr.y (mgr.14146) 2011 : cluster [DBG] pgmap v2974: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:15 smithi123 bash[19119]: cluster 2024-02-12T16:51:14.424758+0000 mgr.y (mgr.14146) 2011 : cluster [DBG] pgmap v2974: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:15.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:15 smithi121 bash[18489]: cluster 2024-02-12T16:51:14.424758+0000 mgr.y (mgr.14146) 2011 : cluster [DBG] pgmap v2974: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:16 smithi121 bash[18489]: cluster 2024-02-12T16:51:15.063022+0000 osd.2 (osd.2) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:16 smithi121 bash[18489]: cluster 2024-02-12T16:51:15.063134+0000 osd.2 (osd.2) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:16.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:16 smithi121 bash[18489]: cluster 2024-02-12T16:51:15.502271+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e1097: 12 total, 12 up, 12 in 2024-02-12T16:51:16.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:16 smithi169 bash[19604]: cluster 2024-02-12T16:51:15.063022+0000 osd.2 (osd.2) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:16.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:16 smithi169 bash[19604]: cluster 2024-02-12T16:51:15.063134+0000 osd.2 (osd.2) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:16.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:16 smithi169 bash[19604]: cluster 2024-02-12T16:51:15.502271+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e1097: 12 total, 12 up, 12 in 2024-02-12T16:51:16.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:16 smithi123 bash[19119]: cluster 2024-02-12T16:51:15.063022+0000 osd.2 (osd.2) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:16.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:16 smithi123 bash[19119]: cluster 2024-02-12T16:51:15.063134+0000 osd.2 (osd.2) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:16.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:16 smithi123 bash[19119]: cluster 2024-02-12T16:51:15.502271+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e1097: 12 total, 12 up, 12 in 2024-02-12T16:51:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:17 smithi169 bash[19604]: cluster 2024-02-12T16:51:15.969507+0000 osd.4 (osd.4) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:17 smithi169 bash[19604]: cluster 2024-02-12T16:51:15.969588+0000 osd.4 (osd.4) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:17 smithi169 bash[19604]: cluster 2024-02-12T16:51:16.425534+0000 mgr.y (mgr.14146) 2012 : cluster [DBG] pgmap v2976: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:17.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:17 smithi169 bash[19604]: cluster 2024-02-12T16:51:16.512147+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e1098: 12 total, 12 up, 12 in 2024-02-12T16:51:17.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:17 smithi123 bash[19119]: cluster 2024-02-12T16:51:15.969507+0000 osd.4 (osd.4) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:17.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:17 smithi123 bash[19119]: cluster 2024-02-12T16:51:15.969588+0000 osd.4 (osd.4) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:17.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:17 smithi123 bash[19119]: cluster 2024-02-12T16:51:16.425534+0000 mgr.y (mgr.14146) 2012 : cluster [DBG] pgmap v2976: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:17.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:17 smithi123 bash[19119]: cluster 2024-02-12T16:51:16.512147+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e1098: 12 total, 12 up, 12 in 2024-02-12T16:51:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:17 smithi121 bash[18489]: cluster 2024-02-12T16:51:15.969507+0000 osd.4 (osd.4) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:17 smithi121 bash[18489]: cluster 2024-02-12T16:51:15.969588+0000 osd.4 (osd.4) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:17 smithi121 bash[18489]: cluster 2024-02-12T16:51:16.425534+0000 mgr.y (mgr.14146) 2012 : cluster [DBG] pgmap v2976: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:17.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:17 smithi121 bash[18489]: cluster 2024-02-12T16:51:16.512147+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e1098: 12 total, 12 up, 12 in 2024-02-12T16:51:19.537 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.3 to 1 (10000) 2024-02-12T16:51:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:19 smithi169 bash[19604]: cluster 2024-02-12T16:51:18.069977+0000 osd.6 (osd.6) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:19 smithi169 bash[19604]: cluster 2024-02-12T16:51:18.070061+0000 osd.6 (osd.6) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:19 smithi169 bash[19604]: cluster 2024-02-12T16:51:18.426143+0000 mgr.y (mgr.14146) 2013 : cluster [DBG] pgmap v2978: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:19 smithi169 bash[19604]: cluster 2024-02-12T16:51:18.521431+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e1099: 12 total, 12 up, 12 in 2024-02-12T16:51:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:19 smithi169 bash[19604]: audit 2024-02-12T16:51:18.609022+0000 mon.a (mon.0) 3618 : audit [INF] from='client.? v1:172.21.15.121:0/519559956' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:51:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:19 smithi169 bash[19604]: cluster 2024-02-12T16:51:18.822115+0000 osd.8 (osd.8) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:19 smithi169 bash[19604]: cluster 2024-02-12T16:51:18.822203+0000 osd.8 (osd.8) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:19 smithi123 bash[19119]: cluster 2024-02-12T16:51:18.069977+0000 osd.6 (osd.6) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:19 smithi123 bash[19119]: cluster 2024-02-12T16:51:18.070061+0000 osd.6 (osd.6) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:19 smithi123 bash[19119]: cluster 2024-02-12T16:51:18.426143+0000 mgr.y (mgr.14146) 2013 : cluster [DBG] pgmap v2978: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:19 smithi123 bash[19119]: cluster 2024-02-12T16:51:18.521431+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e1099: 12 total, 12 up, 12 in 2024-02-12T16:51:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:19 smithi123 bash[19119]: audit 2024-02-12T16:51:18.609022+0000 mon.a (mon.0) 3618 : audit [INF] from='client.? v1:172.21.15.121:0/519559956' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:51:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:19 smithi123 bash[19119]: cluster 2024-02-12T16:51:18.822115+0000 osd.8 (osd.8) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:19.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:19 smithi123 bash[19119]: cluster 2024-02-12T16:51:18.822203+0000 osd.8 (osd.8) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:19.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:19 smithi121 bash[18489]: cluster 2024-02-12T16:51:18.069977+0000 osd.6 (osd.6) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:19.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:19 smithi121 bash[18489]: cluster 2024-02-12T16:51:18.070061+0000 osd.6 (osd.6) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:19 smithi121 bash[18489]: cluster 2024-02-12T16:51:18.426143+0000 mgr.y (mgr.14146) 2013 : cluster [DBG] pgmap v2978: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:19 smithi121 bash[18489]: cluster 2024-02-12T16:51:18.521431+0000 mon.a (mon.0) 3617 : cluster [DBG] osdmap e1099: 12 total, 12 up, 12 in 2024-02-12T16:51:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:19 smithi121 bash[18489]: audit 2024-02-12T16:51:18.609022+0000 mon.a (mon.0) 3618 : audit [INF] from='client.? v1:172.21.15.121:0/519559956' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:51:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:19 smithi121 bash[18489]: cluster 2024-02-12T16:51:18.822115+0000 osd.8 (osd.8) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:19.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:19 smithi121 bash[18489]: cluster 2024-02-12T16:51:18.822203+0000 osd.8 (osd.8) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:20.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-12T16:51:20.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:20 smithi121 bash[18489]: audit 2024-02-12T16:51:19.527510+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? v1:172.21.15.121:0/519559956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:51:20.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:20 smithi121 bash[18489]: cluster 2024-02-12T16:51:19.527670+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e1100: 12 total, 12 up, 12 in 2024-02-12T16:51:20.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:20 smithi121 bash[18489]: cluster 2024-02-12T16:51:20.528673+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e1101: 12 total, 12 up, 12 in 2024-02-12T16:51:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:20 smithi169 bash[19604]: audit 2024-02-12T16:51:19.527510+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? v1:172.21.15.121:0/519559956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:51:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:20 smithi169 bash[19604]: cluster 2024-02-12T16:51:19.527670+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e1100: 12 total, 12 up, 12 in 2024-02-12T16:51:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:20 smithi169 bash[19604]: cluster 2024-02-12T16:51:20.528673+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e1101: 12 total, 12 up, 12 in 2024-02-12T16:51:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:20 smithi123 bash[19119]: audit 2024-02-12T16:51:19.527510+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? v1:172.21.15.121:0/519559956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:51:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:20 smithi123 bash[19119]: cluster 2024-02-12T16:51:19.527670+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e1100: 12 total, 12 up, 12 in 2024-02-12T16:51:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:20 smithi123 bash[19119]: cluster 2024-02-12T16:51:20.528673+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e1101: 12 total, 12 up, 12 in 2024-02-12T16:51:21.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:21 smithi123 bash[19119]: cluster 2024-02-12T16:51:20.426565+0000 mgr.y (mgr.14146) 2014 : cluster [DBG] pgmap v2981: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:21.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:21 smithi121 bash[18489]: cluster 2024-02-12T16:51:20.426565+0000 mgr.y (mgr.14146) 2014 : cluster [DBG] pgmap v2981: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:22.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:21 smithi169 bash[19604]: cluster 2024-02-12T16:51:20.426565+0000 mgr.y (mgr.14146) 2014 : cluster [DBG] pgmap v2981: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:23 smithi123 bash[19119]: cluster 2024-02-12T16:51:22.229401+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:51:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:23 smithi123 bash[19119]: cluster 2024-02-12T16:51:22.427275+0000 mgr.y (mgr.14146) 2015 : cluster [DBG] pgmap v2983: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:23 smithi123 bash[19119]: cluster 2024-02-12T16:51:22.440158+0000 osd.1 (osd.1) 209 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:51:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:23 smithi123 bash[19119]: cluster 2024-02-12T16:51:22.593435+0000 osd.0 (osd.0) 183 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:51:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:23 smithi123 bash[19119]: cluster 2024-02-12T16:51:22.708856+0000 osd.0 (osd.0) 184 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:51:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:23 smithi121 bash[18489]: cluster 2024-02-12T16:51:22.229401+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:51:23.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:23 smithi121 bash[18489]: cluster 2024-02-12T16:51:22.427275+0000 mgr.y (mgr.14146) 2015 : cluster [DBG] pgmap v2983: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:23.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:23 smithi121 bash[18489]: cluster 2024-02-12T16:51:22.440158+0000 osd.1 (osd.1) 209 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:51:23.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:23 smithi121 bash[18489]: cluster 2024-02-12T16:51:22.593435+0000 osd.0 (osd.0) 183 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:51:23.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:23 smithi121 bash[18489]: cluster 2024-02-12T16:51:22.708856+0000 osd.0 (osd.0) 184 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:51:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:23 smithi169 bash[19604]: cluster 2024-02-12T16:51:22.229401+0000 osd.1 (osd.1) 208 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:51:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:23 smithi169 bash[19604]: cluster 2024-02-12T16:51:22.427275+0000 mgr.y (mgr.14146) 2015 : cluster [DBG] pgmap v2983: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:23 smithi169 bash[19604]: cluster 2024-02-12T16:51:22.440158+0000 osd.1 (osd.1) 209 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:51:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:23 smithi169 bash[19604]: cluster 2024-02-12T16:51:22.593435+0000 osd.0 (osd.0) 183 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:51:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:23 smithi169 bash[19604]: cluster 2024-02-12T16:51:22.708856+0000 osd.0 (osd.0) 184 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:51:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:24 smithi123 bash[19119]: cluster 2024-02-12T16:51:22.794193+0000 osd.4 (osd.4) 197 : cluster [DBG] 1.1a deep-scrub starts 2024-02-12T16:51:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:24 smithi123 bash[19119]: cluster 2024-02-12T16:51:22.890638+0000 osd.4 (osd.4) 198 : cluster [DBG] 1.1a deep-scrub ok 2024-02-12T16:51:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:24 smithi123 bash[19119]: cluster 2024-02-12T16:51:23.225345+0000 osd.1 (osd.1) 210 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:51:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:24 smithi123 bash[19119]: cluster 2024-02-12T16:51:23.379808+0000 osd.1 (osd.1) 211 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:51:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:24 smithi123 bash[19119]: cluster 2024-02-12T16:51:23.533495+0000 osd.3 (osd.3) 148 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:51:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:24 smithi123 bash[19119]: cluster 2024-02-12T16:51:23.682323+0000 osd.3 (osd.3) 149 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:51:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:24 smithi123 bash[19119]: audit 2024-02-12T16:51:24.462702+0000 mon.c (mon.1) 116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:24 smithi121 bash[18489]: cluster 2024-02-12T16:51:22.794193+0000 osd.4 (osd.4) 197 : cluster [DBG] 1.1a deep-scrub starts 2024-02-12T16:51:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:24 smithi121 bash[18489]: cluster 2024-02-12T16:51:22.890638+0000 osd.4 (osd.4) 198 : cluster [DBG] 1.1a deep-scrub ok 2024-02-12T16:51:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:24 smithi121 bash[18489]: cluster 2024-02-12T16:51:23.225345+0000 osd.1 (osd.1) 210 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:51:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:24 smithi121 bash[18489]: cluster 2024-02-12T16:51:23.379808+0000 osd.1 (osd.1) 211 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:51:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:24 smithi121 bash[18489]: cluster 2024-02-12T16:51:23.533495+0000 osd.3 (osd.3) 148 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:51:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:24 smithi121 bash[18489]: cluster 2024-02-12T16:51:23.682323+0000 osd.3 (osd.3) 149 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:51:24.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:24 smithi121 bash[18489]: audit 2024-02-12T16:51:24.462702+0000 mon.c (mon.1) 116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:24 smithi169 bash[19604]: cluster 2024-02-12T16:51:22.794193+0000 osd.4 (osd.4) 197 : cluster [DBG] 1.1a deep-scrub starts 2024-02-12T16:51:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:24 smithi169 bash[19604]: cluster 2024-02-12T16:51:22.890638+0000 osd.4 (osd.4) 198 : cluster [DBG] 1.1a deep-scrub ok 2024-02-12T16:51:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:24 smithi169 bash[19604]: cluster 2024-02-12T16:51:23.225345+0000 osd.1 (osd.1) 210 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:51:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:24 smithi169 bash[19604]: cluster 2024-02-12T16:51:23.379808+0000 osd.1 (osd.1) 211 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:51:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:24 smithi169 bash[19604]: cluster 2024-02-12T16:51:23.533495+0000 osd.3 (osd.3) 148 : cluster [DBG] 1.c deep-scrub starts 2024-02-12T16:51:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:24 smithi169 bash[19604]: cluster 2024-02-12T16:51:23.682323+0000 osd.3 (osd.3) 149 : cluster [DBG] 1.c deep-scrub ok 2024-02-12T16:51:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:24 smithi169 bash[19604]: audit 2024-02-12T16:51:24.462702+0000 mon.c (mon.1) 116 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:25.104 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: cluster 2024-02-12T16:51:24.427977+0000 mgr.y (mgr.14146) 2016 : cluster [DBG] pgmap v2984: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: cluster 2024-02-12T16:51:24.720941+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: cluster 2024-02-12T16:51:24.721000+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: audit 2024-02-12T16:51:24.790886+0000 mon.c (mon.1) 117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: audit 2024-02-12T16:51:24.792438+0000 mon.c (mon.1) 118 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: audit 2024-02-12T16:51:24.797700+0000 mon.a (mon.0) 3622 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: cluster 2024-02-12T16:51:24.951950+0000 osd.7 (osd.7) 59 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:51:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:25 smithi123 bash[19119]: cluster 2024-02-12T16:51:24.971096+0000 osd.7 (osd.7) 60 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:51:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: cluster 2024-02-12T16:51:24.427977+0000 mgr.y (mgr.14146) 2016 : cluster [DBG] pgmap v2984: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: cluster 2024-02-12T16:51:24.720941+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: cluster 2024-02-12T16:51:24.721000+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: audit 2024-02-12T16:51:24.790886+0000 mon.c (mon.1) 117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:25.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: audit 2024-02-12T16:51:24.792438+0000 mon.c (mon.1) 118 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:25.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: audit 2024-02-12T16:51:24.797700+0000 mon.a (mon.0) 3622 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:25.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: cluster 2024-02-12T16:51:24.951950+0000 osd.7 (osd.7) 59 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:51:25.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:25 smithi121 bash[18489]: cluster 2024-02-12T16:51:24.971096+0000 osd.7 (osd.7) 60 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: cluster 2024-02-12T16:51:24.427977+0000 mgr.y (mgr.14146) 2016 : cluster [DBG] pgmap v2984: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: cluster 2024-02-12T16:51:24.720941+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: cluster 2024-02-12T16:51:24.721000+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: audit 2024-02-12T16:51:24.790886+0000 mon.c (mon.1) 117 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: audit 2024-02-12T16:51:24.792438+0000 mon.c (mon.1) 118 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: audit 2024-02-12T16:51:24.797700+0000 mon.a (mon.0) 3622 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: cluster 2024-02-12T16:51:24.951950+0000 osd.7 (osd.7) 59 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:51:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:25 smithi169 bash[19604]: cluster 2024-02-12T16:51:24.971096+0000 osd.7 (osd.7) 60 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:51:26.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:26 smithi121 bash[18489]: cluster 2024-02-12T16:51:24.780252+0000 osd.10 (osd.10) 284 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:51:26.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:26 smithi121 bash[18489]: cluster 2024-02-12T16:51:24.871883+0000 osd.10 (osd.10) 285 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:51:26.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:26 smithi121 bash[18489]: cluster 2024-02-12T16:51:25.574418+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e1102: 12 total, 12 up, 12 in 2024-02-12T16:51:26.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:26 smithi121 bash[18489]: cluster 2024-02-12T16:51:25.684268+0000 osd.6 (osd.6) 214 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:51:26.724 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:26 smithi121 bash[18489]: cluster 2024-02-12T16:51:25.811286+0000 osd.6 (osd.6) 215 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:51:26.724 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:26 smithi121 bash[18489]: cluster 2024-02-12T16:51:26.105683+0000 osd.5 (osd.5) 257 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:51:26.724 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:26 smithi121 bash[18489]: cluster 2024-02-12T16:51:26.213842+0000 osd.5 (osd.5) 258 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:51:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:26 smithi123 bash[19119]: cluster 2024-02-12T16:51:24.780252+0000 osd.10 (osd.10) 284 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:51:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:26 smithi123 bash[19119]: cluster 2024-02-12T16:51:24.871883+0000 osd.10 (osd.10) 285 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:51:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:26 smithi123 bash[19119]: cluster 2024-02-12T16:51:25.574418+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e1102: 12 total, 12 up, 12 in 2024-02-12T16:51:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:26 smithi123 bash[19119]: cluster 2024-02-12T16:51:25.684268+0000 osd.6 (osd.6) 214 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:51:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:26 smithi123 bash[19119]: cluster 2024-02-12T16:51:25.811286+0000 osd.6 (osd.6) 215 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:51:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:26 smithi123 bash[19119]: cluster 2024-02-12T16:51:26.105683+0000 osd.5 (osd.5) 257 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:51:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:26 smithi123 bash[19119]: cluster 2024-02-12T16:51:26.213842+0000 osd.5 (osd.5) 258 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:51:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:26 smithi169 bash[19604]: cluster 2024-02-12T16:51:24.780252+0000 osd.10 (osd.10) 284 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:51:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:26 smithi169 bash[19604]: cluster 2024-02-12T16:51:24.871883+0000 osd.10 (osd.10) 285 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:51:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:26 smithi169 bash[19604]: cluster 2024-02-12T16:51:25.574418+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e1102: 12 total, 12 up, 12 in 2024-02-12T16:51:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:26 smithi169 bash[19604]: cluster 2024-02-12T16:51:25.684268+0000 osd.6 (osd.6) 214 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:51:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:26 smithi169 bash[19604]: cluster 2024-02-12T16:51:25.811286+0000 osd.6 (osd.6) 215 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:51:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:26 smithi169 bash[19604]: cluster 2024-02-12T16:51:26.105683+0000 osd.5 (osd.5) 257 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:51:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:26 smithi169 bash[19604]: cluster 2024-02-12T16:51:26.213842+0000 osd.5 (osd.5) 258 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:51:27.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:27 smithi123 bash[19119]: cluster 2024-02-12T16:51:26.428635+0000 mgr.y (mgr.14146) 2017 : cluster [DBG] pgmap v2986: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:27.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:27 smithi121 bash[18489]: cluster 2024-02-12T16:51:26.428635+0000 mgr.y (mgr.14146) 2017 : cluster [DBG] pgmap v2986: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:28.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:27 smithi169 bash[19604]: cluster 2024-02-12T16:51:26.428635+0000 mgr.y (mgr.14146) 2017 : cluster [DBG] pgmap v2986: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:29.595 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-12T16:51:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:29 smithi123 bash[19119]: cluster 2024-02-12T16:51:28.429302+0000 mgr.y (mgr.14146) 2018 : cluster [DBG] pgmap v2987: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:29 smithi123 bash[19119]: audit 2024-02-12T16:51:28.987537+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? v1:172.21.15.121:0/4202643304' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:51:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:29 smithi121 bash[18489]: cluster 2024-02-12T16:51:28.429302+0000 mgr.y (mgr.14146) 2018 : cluster [DBG] pgmap v2987: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:29 smithi121 bash[18489]: audit 2024-02-12T16:51:28.987537+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? v1:172.21.15.121:0/4202643304' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:51:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:29 smithi169 bash[19604]: cluster 2024-02-12T16:51:28.429302+0000 mgr.y (mgr.14146) 2018 : cluster [DBG] pgmap v2987: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:29 smithi169 bash[19604]: audit 2024-02-12T16:51:28.987537+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? v1:172.21.15.121:0/4202643304' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-12T16:51:30.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd reweight 7 1 2024-02-12T16:51:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:30 smithi121 bash[18489]: audit 2024-02-12T16:51:29.584685+0000 mon.a (mon.0) 3625 : audit [INF] from='client.? v1:172.21.15.121:0/4202643304' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:51:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:30 smithi121 bash[18489]: cluster 2024-02-12T16:51:29.584799+0000 mon.a (mon.0) 3626 : cluster [DBG] osdmap e1103: 12 total, 12 up, 12 in 2024-02-12T16:51:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:30 smithi123 bash[19119]: audit 2024-02-12T16:51:29.584685+0000 mon.a (mon.0) 3625 : audit [INF] from='client.? v1:172.21.15.121:0/4202643304' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:51:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:30 smithi123 bash[19119]: cluster 2024-02-12T16:51:29.584799+0000 mon.a (mon.0) 3626 : cluster [DBG] osdmap e1103: 12 total, 12 up, 12 in 2024-02-12T16:51:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:30 smithi169 bash[19604]: audit 2024-02-12T16:51:29.584685+0000 mon.a (mon.0) 3625 : audit [INF] from='client.? v1:172.21.15.121:0/4202643304' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-12T16:51:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:30 smithi169 bash[19604]: cluster 2024-02-12T16:51:29.584799+0000 mon.a (mon.0) 3626 : cluster [DBG] osdmap e1103: 12 total, 12 up, 12 in 2024-02-12T16:51:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:31 smithi123 bash[19119]: cluster 2024-02-12T16:51:29.710014+0000 osd.10 (osd.10) 286 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:51:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:31 smithi123 bash[19119]: cluster 2024-02-12T16:51:29.756696+0000 osd.10 (osd.10) 287 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:51:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:31 smithi123 bash[19119]: cluster 2024-02-12T16:51:30.429797+0000 mgr.y (mgr.14146) 2019 : cluster [DBG] pgmap v2989: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:31 smithi121 bash[18489]: cluster 2024-02-12T16:51:29.710014+0000 osd.10 (osd.10) 286 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:51:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:31 smithi121 bash[18489]: cluster 2024-02-12T16:51:29.756696+0000 osd.10 (osd.10) 287 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:51:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:31 smithi121 bash[18489]: cluster 2024-02-12T16:51:30.429797+0000 mgr.y (mgr.14146) 2019 : cluster [DBG] pgmap v2989: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:31 smithi169 bash[19604]: cluster 2024-02-12T16:51:29.710014+0000 osd.10 (osd.10) 286 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:51:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:31 smithi169 bash[19604]: cluster 2024-02-12T16:51:29.756696+0000 osd.10 (osd.10) 287 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:51:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:31 smithi169 bash[19604]: cluster 2024-02-12T16:51:30.429797+0000 mgr.y (mgr.14146) 2019 : cluster [DBG] pgmap v2989: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:32.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:32 smithi123 bash[19119]: cluster 2024-02-12T16:51:31.049728+0000 osd.7 (osd.7) 61 : cluster [DBG] 1.15 deep-scrub starts 2024-02-12T16:51:32.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:32 smithi123 bash[19119]: cluster 2024-02-12T16:51:31.160915+0000 osd.7 (osd.7) 62 : cluster [DBG] 1.15 deep-scrub ok 2024-02-12T16:51:32.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:32 smithi123 bash[19119]: cluster 2024-02-12T16:51:32.118289+0000 osd.5 (osd.5) 259 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:51:32.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:32 smithi123 bash[19119]: cluster 2024-02-12T16:51:32.189782+0000 osd.5 (osd.5) 260 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:51:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:32 smithi121 bash[18489]: cluster 2024-02-12T16:51:31.049728+0000 osd.7 (osd.7) 61 : cluster [DBG] 1.15 deep-scrub starts 2024-02-12T16:51:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:32 smithi121 bash[18489]: cluster 2024-02-12T16:51:31.160915+0000 osd.7 (osd.7) 62 : cluster [DBG] 1.15 deep-scrub ok 2024-02-12T16:51:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:32 smithi121 bash[18489]: cluster 2024-02-12T16:51:32.118289+0000 osd.5 (osd.5) 259 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:51:32.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:32 smithi121 bash[18489]: cluster 2024-02-12T16:51:32.189782+0000 osd.5 (osd.5) 260 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:51:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:32 smithi169 bash[19604]: cluster 2024-02-12T16:51:31.049728+0000 osd.7 (osd.7) 61 : cluster [DBG] 1.15 deep-scrub starts 2024-02-12T16:51:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:32 smithi169 bash[19604]: cluster 2024-02-12T16:51:31.160915+0000 osd.7 (osd.7) 62 : cluster [DBG] 1.15 deep-scrub ok 2024-02-12T16:51:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:32 smithi169 bash[19604]: cluster 2024-02-12T16:51:32.118289+0000 osd.5 (osd.5) 259 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:51:33.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:32 smithi169 bash[19604]: cluster 2024-02-12T16:51:32.189782+0000 osd.5 (osd.5) 260 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:51:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:33 smithi123 bash[19119]: cluster 2024-02-12T16:51:32.056921+0000 osd.7 (osd.7) 63 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:51:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:33 smithi123 bash[19119]: cluster 2024-02-12T16:51:32.153272+0000 osd.7 (osd.7) 64 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:51:33.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:33 smithi123 bash[19119]: cluster 2024-02-12T16:51:32.430489+0000 mgr.y (mgr.14146) 2020 : cluster [DBG] pgmap v2990: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:33 smithi121 bash[18489]: cluster 2024-02-12T16:51:32.056921+0000 osd.7 (osd.7) 63 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:51:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:33 smithi121 bash[18489]: cluster 2024-02-12T16:51:32.153272+0000 osd.7 (osd.7) 64 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:51:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:33 smithi121 bash[18489]: cluster 2024-02-12T16:51:32.430489+0000 mgr.y (mgr.14146) 2020 : cluster [DBG] pgmap v2990: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:33 smithi169 bash[19604]: cluster 2024-02-12T16:51:32.056921+0000 osd.7 (osd.7) 63 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:51:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:33 smithi169 bash[19604]: cluster 2024-02-12T16:51:32.153272+0000 osd.7 (osd.7) 64 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:51:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:33 smithi169 bash[19604]: cluster 2024-02-12T16:51:32.430489+0000 mgr.y (mgr.14146) 2020 : cluster [DBG] pgmap v2990: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:34 smithi123 bash[19119]: audit 2024-02-12T16:51:34.437152+0000 mon.c (mon.1) 119 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:51:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:34 smithi123 bash[19119]: audit 2024-02-12T16:51:34.437558+0000 mon.c (mon.1) 120 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:51:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:34 smithi123 bash[19119]: audit 2024-02-12T16:51:34.437931+0000 mon.c (mon.1) 121 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:51:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:34 smithi123 bash[19119]: audit 2024-02-12T16:51:34.498895+0000 mon.c (mon.1) 122 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:34 smithi121 bash[18489]: audit 2024-02-12T16:51:34.437152+0000 mon.c (mon.1) 119 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:51:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:34 smithi121 bash[18489]: audit 2024-02-12T16:51:34.437558+0000 mon.c (mon.1) 120 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:51:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:34 smithi121 bash[18489]: audit 2024-02-12T16:51:34.437931+0000 mon.c (mon.1) 121 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:51:34.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:34 smithi121 bash[18489]: audit 2024-02-12T16:51:34.498895+0000 mon.c (mon.1) 122 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:34 smithi169 bash[19604]: audit 2024-02-12T16:51:34.437152+0000 mon.c (mon.1) 119 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:51:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:34 smithi169 bash[19604]: audit 2024-02-12T16:51:34.437558+0000 mon.c (mon.1) 120 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:51:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:34 smithi169 bash[19604]: audit 2024-02-12T16:51:34.437931+0000 mon.c (mon.1) 121 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:51:35.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:34 smithi169 bash[19604]: audit 2024-02-12T16:51:34.498895+0000 mon.c (mon.1) 122 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:51:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:35 smithi123 bash[19119]: cluster 2024-02-12T16:51:34.688596+0000 osd.4 (osd.4) 199 : cluster [DBG] 1.e scrub starts 2024-02-12T16:51:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:35 smithi123 bash[19119]: cluster 2024-02-12T16:51:34.779168+0000 osd.4 (osd.4) 200 : cluster [DBG] 1.e scrub ok 2024-02-12T16:51:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:35 smithi123 bash[19119]: audit 2024-02-12T16:51:34.810965+0000 mon.a (mon.0) 3627 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:35 smithi123 bash[19119]: audit 2024-02-12T16:51:34.817644+0000 mon.c (mon.1) 123 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:35 smithi123 bash[19119]: audit 2024-02-12T16:51:34.819368+0000 mon.c (mon.1) 124 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:36.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:35 smithi123 bash[19119]: audit 2024-02-12T16:51:34.826399+0000 mon.a (mon.0) 3628 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:35 smithi121 bash[18489]: cluster 2024-02-12T16:51:34.688596+0000 osd.4 (osd.4) 199 : cluster [DBG] 1.e scrub starts 2024-02-12T16:51:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:35 smithi121 bash[18489]: cluster 2024-02-12T16:51:34.779168+0000 osd.4 (osd.4) 200 : cluster [DBG] 1.e scrub ok 2024-02-12T16:51:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:35 smithi121 bash[18489]: audit 2024-02-12T16:51:34.810965+0000 mon.a (mon.0) 3627 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:35 smithi121 bash[18489]: audit 2024-02-12T16:51:34.817644+0000 mon.c (mon.1) 123 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:36.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:35 smithi121 bash[18489]: audit 2024-02-12T16:51:34.819368+0000 mon.c (mon.1) 124 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:36.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:35 smithi121 bash[18489]: audit 2024-02-12T16:51:34.826399+0000 mon.a (mon.0) 3628 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:35 smithi169 bash[19604]: cluster 2024-02-12T16:51:34.688596+0000 osd.4 (osd.4) 199 : cluster [DBG] 1.e scrub starts 2024-02-12T16:51:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:35 smithi169 bash[19604]: cluster 2024-02-12T16:51:34.779168+0000 osd.4 (osd.4) 200 : cluster [DBG] 1.e scrub ok 2024-02-12T16:51:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:35 smithi169 bash[19604]: audit 2024-02-12T16:51:34.810965+0000 mon.a (mon.0) 3627 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:35 smithi169 bash[19604]: audit 2024-02-12T16:51:34.817644+0000 mon.c (mon.1) 123 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:51:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:35 smithi169 bash[19604]: audit 2024-02-12T16:51:34.819368+0000 mon.c (mon.1) 124 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:51:36.298 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:35 smithi169 bash[19604]: audit 2024-02-12T16:51:34.826399+0000 mon.a (mon.0) 3628 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:51:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:37 smithi169 bash[19604]: cluster 2024-02-12T16:51:36.431965+0000 mgr.y (mgr.14146) 2022 : cluster [DBG] pgmap v2992: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:37 smithi123 bash[19119]: cluster 2024-02-12T16:51:36.431965+0000 mgr.y (mgr.14146) 2022 : cluster [DBG] pgmap v2992: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:37 smithi121 bash[18489]: cluster 2024-02-12T16:51:36.431965+0000 mgr.y (mgr.14146) 2022 : cluster [DBG] pgmap v2992: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:38.444 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:38 smithi121 bash[18489]: cluster 2024-02-12T16:51:36.540322+0000 osd.9 (osd.9) 181 : cluster [DBG] 1.a deep-scrub starts 2024-02-12T16:51:38.444 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:38 smithi121 bash[18489]: cluster 2024-02-12T16:51:36.632605+0000 osd.9 (osd.9) 182 : cluster [DBG] 1.a deep-scrub ok 2024-02-12T16:51:38.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:38 smithi169 bash[19604]: cluster 2024-02-12T16:51:36.540322+0000 osd.9 (osd.9) 181 : cluster [DBG] 1.a deep-scrub starts 2024-02-12T16:51:38.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:38 smithi169 bash[19604]: cluster 2024-02-12T16:51:36.632605+0000 osd.9 (osd.9) 182 : cluster [DBG] 1.a deep-scrub ok 2024-02-12T16:51:38.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:38 smithi123 bash[19119]: cluster 2024-02-12T16:51:36.540322+0000 osd.9 (osd.9) 181 : cluster [DBG] 1.a deep-scrub starts 2024-02-12T16:51:38.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:38 smithi123 bash[19119]: cluster 2024-02-12T16:51:36.632605+0000 osd.9 (osd.9) 182 : cluster [DBG] 1.a deep-scrub ok 2024-02-12T16:51:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:reweighted osd.7 to 1 (10000) 2024-02-12T16:51:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:39 smithi169 bash[19604]: cluster 2024-02-12T16:51:38.432677+0000 mgr.y (mgr.14146) 2023 : cluster [DBG] pgmap v2993: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:39 smithi169 bash[19604]: cluster 2024-02-12T16:51:38.584396+0000 osd.10 (osd.10) 288 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:51:39.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:39 smithi169 bash[19604]: audit 2024-02-12T16:51:38.790649+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? v1:172.21.15.121:0/1770389044' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:51:39.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:39 smithi123 bash[19119]: cluster 2024-02-12T16:51:38.432677+0000 mgr.y (mgr.14146) 2023 : cluster [DBG] pgmap v2993: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:39 smithi123 bash[19119]: cluster 2024-02-12T16:51:38.584396+0000 osd.10 (osd.10) 288 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:51:39.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:39 smithi123 bash[19119]: audit 2024-02-12T16:51:38.790649+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? v1:172.21.15.121:0/1770389044' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:51:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:39 smithi121 bash[18489]: cluster 2024-02-12T16:51:38.432677+0000 mgr.y (mgr.14146) 2023 : cluster [DBG] pgmap v2993: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:39 smithi121 bash[18489]: cluster 2024-02-12T16:51:38.584396+0000 osd.10 (osd.10) 288 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:51:39.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:39 smithi121 bash[18489]: audit 2024-02-12T16:51:38.790649+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? v1:172.21.15.121:0/1770389044' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:51:39.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-12T16:51:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:40 smithi121 bash[18489]: cluster 2024-02-12T16:51:38.654273+0000 osd.10 (osd.10) 289 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:51:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:40 smithi121 bash[18489]: audit 2024-02-12T16:51:39.185534+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? v1:172.21.15.121:0/1770389044' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:51:40.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:40 smithi121 bash[18489]: cluster 2024-02-12T16:51:39.185582+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e1104: 12 total, 12 up, 12 in 2024-02-12T16:51:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:40 smithi169 bash[19604]: cluster 2024-02-12T16:51:38.654273+0000 osd.10 (osd.10) 289 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:51:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:40 smithi169 bash[19604]: audit 2024-02-12T16:51:39.185534+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? v1:172.21.15.121:0/1770389044' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:51:40.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:40 smithi169 bash[19604]: cluster 2024-02-12T16:51:39.185582+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e1104: 12 total, 12 up, 12 in 2024-02-12T16:51:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:40 smithi123 bash[19119]: cluster 2024-02-12T16:51:38.654273+0000 osd.10 (osd.10) 289 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:51:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:40 smithi123 bash[19119]: audit 2024-02-12T16:51:39.185534+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? v1:172.21.15.121:0/1770389044' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:51:40.640 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:40 smithi123 bash[19119]: cluster 2024-02-12T16:51:39.185582+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e1104: 12 total, 12 up, 12 in 2024-02-12T16:51:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:41 smithi169 bash[19604]: cluster 2024-02-12T16:51:40.433335+0000 mgr.y (mgr.14146) 2024 : cluster [DBG] pgmap v2995: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:41 smithi123 bash[19119]: cluster 2024-02-12T16:51:40.433335+0000 mgr.y (mgr.14146) 2024 : cluster [DBG] pgmap v2995: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:41.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:41 smithi121 bash[18489]: cluster 2024-02-12T16:51:40.433335+0000 mgr.y (mgr.14146) 2024 : cluster [DBG] pgmap v2995: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:42 smithi169 bash[19604]: cluster 2024-02-12T16:51:41.293647+0000 osd.0 (osd.0) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:42 smithi169 bash[19604]: cluster 2024-02-12T16:51:41.293742+0000 osd.0 (osd.0) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:42.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:42 smithi123 bash[19119]: cluster 2024-02-12T16:51:41.293647+0000 osd.0 (osd.0) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:42 smithi123 bash[19119]: cluster 2024-02-12T16:51:41.293742+0000 osd.0 (osd.0) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:42 smithi121 bash[18489]: cluster 2024-02-12T16:51:41.293647+0000 osd.0 (osd.0) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:42 smithi121 bash[18489]: cluster 2024-02-12T16:51:41.293742+0000 osd.0 (osd.0) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:43 smithi169 bash[19604]: cluster 2024-02-12T16:51:42.211142+0000 mon.a (mon.0) 3632 : cluster [DBG] osdmap e1105: 12 total, 12 up, 12 in 2024-02-12T16:51:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:43 smithi169 bash[19604]: cluster 2024-02-12T16:51:42.233372+0000 osd.2 (osd.2) 161 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:51:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:43 smithi169 bash[19604]: cluster 2024-02-12T16:51:42.338522+0000 osd.2 (osd.2) 162 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:51:43.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:43 smithi169 bash[19604]: cluster 2024-02-12T16:51:42.434059+0000 mgr.y (mgr.14146) 2025 : cluster [DBG] pgmap v2997: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:43 smithi123 bash[19119]: cluster 2024-02-12T16:51:42.211142+0000 mon.a (mon.0) 3632 : cluster [DBG] osdmap e1105: 12 total, 12 up, 12 in 2024-02-12T16:51:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:43 smithi123 bash[19119]: cluster 2024-02-12T16:51:42.233372+0000 osd.2 (osd.2) 161 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:51:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:43 smithi123 bash[19119]: cluster 2024-02-12T16:51:42.338522+0000 osd.2 (osd.2) 162 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:51:43.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:43 smithi123 bash[19119]: cluster 2024-02-12T16:51:42.434059+0000 mgr.y (mgr.14146) 2025 : cluster [DBG] pgmap v2997: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:43 smithi121 bash[18489]: cluster 2024-02-12T16:51:42.211142+0000 mon.a (mon.0) 3632 : cluster [DBG] osdmap e1105: 12 total, 12 up, 12 in 2024-02-12T16:51:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:43 smithi121 bash[18489]: cluster 2024-02-12T16:51:42.233372+0000 osd.2 (osd.2) 161 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:51:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:43 smithi121 bash[18489]: cluster 2024-02-12T16:51:42.338522+0000 osd.2 (osd.2) 162 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:51:43.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:43 smithi121 bash[18489]: cluster 2024-02-12T16:51:42.434059+0000 mgr.y (mgr.14146) 2025 : cluster [DBG] pgmap v2997: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:44.772 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:45.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:45 smithi169 bash[19604]: cluster 2024-02-12T16:51:44.434916+0000 mgr.y (mgr.14146) 2026 : cluster [DBG] pgmap v2998: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:45.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:45 smithi123 bash[19119]: cluster 2024-02-12T16:51:44.434916+0000 mgr.y (mgr.14146) 2026 : cluster [DBG] pgmap v2998: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:45 smithi121 bash[18489]: cluster 2024-02-12T16:51:44.434916+0000 mgr.y (mgr.14146) 2026 : cluster [DBG] pgmap v2998: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:47.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:47 smithi169 bash[19604]: cluster 2024-02-12T16:51:46.162374+0000 osd.10 (osd.10) 290 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:47.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:47 smithi169 bash[19604]: cluster 2024-02-12T16:51:46.162464+0000 osd.10 (osd.10) 291 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:47.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:47 smithi169 bash[19604]: cluster 2024-02-12T16:51:46.435652+0000 mgr.y (mgr.14146) 2027 : cluster [DBG] pgmap v2999: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:47.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:47 smithi169 bash[19604]: cluster 2024-02-12T16:51:46.498355+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e1106: 12 total, 12 up, 12 in 2024-02-12T16:51:47.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:47 smithi121 bash[18489]: cluster 2024-02-12T16:51:46.162374+0000 osd.10 (osd.10) 290 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:47.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:47 smithi121 bash[18489]: cluster 2024-02-12T16:51:46.162464+0000 osd.10 (osd.10) 291 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:47.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:47 smithi121 bash[18489]: cluster 2024-02-12T16:51:46.435652+0000 mgr.y (mgr.14146) 2027 : cluster [DBG] pgmap v2999: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:47.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:47 smithi121 bash[18489]: cluster 2024-02-12T16:51:46.498355+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e1106: 12 total, 12 up, 12 in 2024-02-12T16:51:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:47 smithi123 bash[19119]: cluster 2024-02-12T16:51:46.162374+0000 osd.10 (osd.10) 290 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:47 smithi123 bash[19119]: cluster 2024-02-12T16:51:46.162464+0000 osd.10 (osd.10) 291 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:47 smithi123 bash[19119]: cluster 2024-02-12T16:51:46.435652+0000 mgr.y (mgr.14146) 2027 : cluster [DBG] pgmap v2999: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:47.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:47 smithi123 bash[19119]: cluster 2024-02-12T16:51:46.498355+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e1106: 12 total, 12 up, 12 in 2024-02-12T16:51:48.522 INFO:teuthology.orchestra.run.smithi121.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-12T16:51:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:48 smithi169 bash[19604]: audit 2024-02-12T16:51:48.278035+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? v1:172.21.15.121:0/1717842435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:51:48.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:48 smithi123 bash[19119]: audit 2024-02-12T16:51:48.278035+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? v1:172.21.15.121:0/1717842435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:51:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:48 smithi121 bash[18489]: audit 2024-02-12T16:51:48.278035+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? v1:172.21.15.121:0/1717842435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-12T16:51:49.335 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-12T16:51:49.335 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-12T16:51:49.335 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-12T16:51:49.335 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-12T16:51:49.335 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-12T16:51:49.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T16:51:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:49 smithi121 bash[18489]: cluster 2024-02-12T16:51:48.436392+0000 mgr.y (mgr.14146) 2028 : cluster [DBG] pgmap v3001: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:49 smithi121 bash[18489]: audit 2024-02-12T16:51:48.510830+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? v1:172.21.15.121:0/1717842435' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:51:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:49 smithi121 bash[18489]: cluster 2024-02-12T16:51:48.510939+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e1107: 12 total, 12 up, 12 in 2024-02-12T16:51:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:49 smithi169 bash[19604]: cluster 2024-02-12T16:51:48.436392+0000 mgr.y (mgr.14146) 2028 : cluster [DBG] pgmap v3001: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:49 smithi169 bash[19604]: audit 2024-02-12T16:51:48.510830+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? v1:172.21.15.121:0/1717842435' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:51:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:49 smithi169 bash[19604]: cluster 2024-02-12T16:51:48.510939+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e1107: 12 total, 12 up, 12 in 2024-02-12T16:51:49.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:49 smithi123 bash[19119]: cluster 2024-02-12T16:51:48.436392+0000 mgr.y (mgr.14146) 2028 : cluster [DBG] pgmap v3001: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:49 smithi123 bash[19119]: audit 2024-02-12T16:51:48.510830+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? v1:172.21.15.121:0/1717842435' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-12T16:51:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:49 smithi123 bash[19119]: cluster 2024-02-12T16:51:48.510939+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e1107: 12 total, 12 up, 12 in 2024-02-12T16:51:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:51 smithi169 bash[19604]: cluster 2024-02-12T16:51:50.437115+0000 mgr.y (mgr.14146) 2029 : cluster [DBG] pgmap v3003: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:51 smithi123 bash[19119]: cluster 2024-02-12T16:51:50.437115+0000 mgr.y (mgr.14146) 2029 : cluster [DBG] pgmap v3003: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:51 smithi121 bash[18489]: cluster 2024-02-12T16:51:50.437115+0000 mgr.y (mgr.14146) 2029 : cluster [DBG] pgmap v3003: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:52.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:52 smithi169 bash[19604]: cluster 2024-02-12T16:51:51.669680+0000 osd.3 (osd.3) 150 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:51:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:52 smithi123 bash[19119]: cluster 2024-02-12T16:51:51.669680+0000 osd.3 (osd.3) 150 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:51:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:52 smithi121 bash[18489]: cluster 2024-02-12T16:51:51.669680+0000 osd.3 (osd.3) 150 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:51:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:53 smithi169 bash[19604]: cluster 2024-02-12T16:51:51.813839+0000 osd.3 (osd.3) 151 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:51:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:53 smithi169 bash[19604]: cluster 2024-02-12T16:51:52.437834+0000 mgr.y (mgr.14146) 2030 : cluster [DBG] pgmap v3004: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:53 smithi169 bash[19604]: cluster 2024-02-12T16:51:52.693499+0000 osd.11 (osd.11) 259 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:51:53.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:53 smithi169 bash[19604]: cluster 2024-02-12T16:51:52.771021+0000 osd.11 (osd.11) 260 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:51:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:53 smithi123 bash[19119]: cluster 2024-02-12T16:51:51.813839+0000 osd.3 (osd.3) 151 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:51:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:53 smithi123 bash[19119]: cluster 2024-02-12T16:51:52.437834+0000 mgr.y (mgr.14146) 2030 : cluster [DBG] pgmap v3004: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:53 smithi123 bash[19119]: cluster 2024-02-12T16:51:52.693499+0000 osd.11 (osd.11) 259 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:51:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:53 smithi123 bash[19119]: cluster 2024-02-12T16:51:52.771021+0000 osd.11 (osd.11) 260 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:51:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:53 smithi121 bash[18489]: cluster 2024-02-12T16:51:51.813839+0000 osd.3 (osd.3) 151 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:51:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:53 smithi121 bash[18489]: cluster 2024-02-12T16:51:52.437834+0000 mgr.y (mgr.14146) 2030 : cluster [DBG] pgmap v3004: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:53 smithi121 bash[18489]: cluster 2024-02-12T16:51:52.693499+0000 osd.11 (osd.11) 259 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:51:53.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:53 smithi121 bash[18489]: cluster 2024-02-12T16:51:52.771021+0000 osd.11 (osd.11) 260 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:51:54.139 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:51:55.722 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:55 smithi121 bash[18489]: cluster 2024-02-12T16:51:54.438556+0000 mgr.y (mgr.14146) 2031 : cluster [DBG] pgmap v3005: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:55.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:55 smithi169 bash[19604]: cluster 2024-02-12T16:51:54.438556+0000 mgr.y (mgr.14146) 2031 : cluster [DBG] pgmap v3005: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:55 smithi123 bash[19119]: cluster 2024-02-12T16:51:54.438556+0000 mgr.y (mgr.14146) 2031 : cluster [DBG] pgmap v3005: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:57 smithi169 bash[19604]: cluster 2024-02-12T16:51:56.439306+0000 mgr.y (mgr.14146) 2032 : cluster [DBG] pgmap v3006: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:57.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:57 smithi123 bash[19119]: cluster 2024-02-12T16:51:56.439306+0000 mgr.y (mgr.14146) 2032 : cluster [DBG] pgmap v3006: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:57 smithi121 bash[18489]: cluster 2024-02-12T16:51:56.439306+0000 mgr.y (mgr.14146) 2032 : cluster [DBG] pgmap v3006: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:57.756 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:51:57.756 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":1107,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","created":"2024-02-12T15:47:18.273813+0000","modified":"2024-02-12T16:51:48.497857+0000","last_up_change":"2024-02-12T16:42:40.138095+0000","last_in_change":"2024-02-12T16:47:42.419336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":42,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T15:51:35.886878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"957","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"663","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"3a3badac-97ea-4ee1-ad82-1b67a4fe94f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1044,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1815232143}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1815232143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":1815232143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1815232143}]},"public_addr":"172.21.15.121:6801/1815232143","cluster_addr":"172.21.15.121:6802/1815232143","heartbeat_back_addr":"172.21.15.121:6804/1815232143","heartbeat_front_addr":"172.21.15.121:6803/1815232143","state":["exists","up"]},{"osd":1,"uuid":"c6140309-5703-4538-9739-276fbbdaa880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1020,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2968433670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2968433670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":2968433670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2968433670}]},"public_addr":"172.21.15.121:6805/2968433670","cluster_addr":"172.21.15.121:6806/2968433670","heartbeat_back_addr":"172.21.15.121:6808/2968433670","heartbeat_front_addr":"172.21.15.121:6807/2968433670","state":["exists","up"]},{"osd":2,"uuid":"7c81e1a4-cae0-48d4-98ab-0becb6b31170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":889,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":2158187852}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":2158187852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":2158187852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":2158187852}]},"public_addr":"172.21.15.121:6809/2158187852","cluster_addr":"172.21.15.121:6810/2158187852","heartbeat_back_addr":"172.21.15.121:6812/2158187852","heartbeat_front_addr":"172.21.15.121:6811/2158187852","state":["exists","up"]},{"osd":3,"uuid":"5f824526-596d-42e1-8b7e-cd8676ded6dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":414,"up_from":522,"up_thru":1100,"down_at":415,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":2551349828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":2551349828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6816","nonce":2551349828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":2551349828}]},"public_addr":"172.21.15.121:6813/2551349828","cluster_addr":"172.21.15.121:6814/2551349828","heartbeat_back_addr":"172.21.15.121:6816/2551349828","heartbeat_front_addr":"172.21.15.121:6815/2551349828","state":["exists","up"]},{"osd":4,"uuid":"14c2f878-99ce-4b51-8d9c-131c1d883b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":1021,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6800","nonce":3447700123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":3447700123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":3447700123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":3447700123}]},"public_addr":"172.21.15.123:6800/3447700123","cluster_addr":"172.21.15.123:6801/3447700123","heartbeat_back_addr":"172.21.15.123:6803/3447700123","heartbeat_front_addr":"172.21.15.123:6802/3447700123","state":["exists","up"]},{"osd":5,"uuid":"75ecf26a-d7ed-46e7-a40e-6f719e2c3602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":1086,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":961317306}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":961317306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":961317306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":961317306}]},"public_addr":"172.21.15.123:6804/961317306","cluster_addr":"172.21.15.123:6805/961317306","heartbeat_back_addr":"172.21.15.123:6807/961317306","heartbeat_front_addr":"172.21.15.123:6806/961317306","state":["exists","up"]},{"osd":6,"uuid":"71e715e6-eac1-4386-927e-51df0cc419c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":1037,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":1377261784}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":1377261784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":1377261784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":1377261784}]},"public_addr":"172.21.15.123:6808/1377261784","cluster_addr":"172.21.15.123:6809/1377261784","heartbeat_back_addr":"172.21.15.123:6811/1377261784","heartbeat_front_addr":"172.21.15.123:6810/1377261784","state":["exists","up"]},{"osd":7,"uuid":"b419b04f-0c0c-495a-aa9b-25c334159515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":56,"last_clean_end":848,"up_from":923,"up_thru":1068,"down_at":849,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":4284140534}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":4284140534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":4284140534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":4284140534}]},"public_addr":"172.21.15.123:6812/4284140534","cluster_addr":"172.21.15.123:6813/4284140534","heartbeat_back_addr":"172.21.15.123:6815/4284140534","heartbeat_front_addr":"172.21.15.123:6814/4284140534","state":["exists","up"]},{"osd":8,"uuid":"1591e7bc-5b86-4401-803d-5325062ffb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6800","nonce":2551075225}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6801","nonce":2551075225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6803","nonce":2551075225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6802","nonce":2551075225}]},"public_addr":"172.21.15.169:6800/2551075225","cluster_addr":"172.21.15.169:6801/2551075225","heartbeat_back_addr":"172.21.15.169:6803/2551075225","heartbeat_front_addr":"172.21.15.169:6802/2551075225","state":["exists","up"]},{"osd":9,"uuid":"522d61df-8dc1-4915-83fd-f94d0d572c15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6804","nonce":2339555071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6805","nonce":2339555071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6807","nonce":2339555071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6806","nonce":2339555071}]},"public_addr":"172.21.15.169:6804/2339555071","cluster_addr":"172.21.15.169:6805/2339555071","heartbeat_back_addr":"172.21.15.169:6807/2339555071","heartbeat_front_addr":"172.21.15.169:6806/2339555071","state":["exists","up"]},{"osd":10,"uuid":"91f74dca-503a-45d1-9647-df4804d58515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":1057,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6808","nonce":391306610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6809","nonce":391306610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6811","nonce":391306610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6810","nonce":391306610}]},"public_addr":"172.21.15.169:6808/391306610","cluster_addr":"172.21.15.169:6809/391306610","heartbeat_back_addr":"172.21.15.169:6811/391306610","heartbeat_front_addr":"172.21.15.169:6810/391306610","state":["exists","up"]},{"osd":11,"uuid":"788b4f47-eadf-4601-a5b4-51fb6f04fbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":93,"up_thru":1015,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6812","nonce":3696812305}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6813","nonce":3696812305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6815","nonce":3696812305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6814","nonce":3696812305}]},"public_addr":"172.21.15.169:6812/3696812305","cluster_addr":"172.21.15.169:6813/3696812305","heartbeat_back_addr":"172.21.15.169:6815/3696812305","heartbeat_front_addr":"172.21.15.169:6814/3696812305","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:41.293746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:56.199449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:15.063139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:02.993573+0000","dead_epoch":414},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:15.969593+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:02.861397+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:18.070065+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:24.721002+0000","dead_epoch":848},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:18.822208+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:58.934068+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:46.162468+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:56.765682+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.1b","osds":[10,5]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/2474849789":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/253689008":"2024-02-13T15:48:17.261083+0000","172.21.15.121:6800/3531151725":"2024-02-13T15:47:47.232761+0000","172.21.15.121:0/1778611055":"2024-02-13T15:47:47.232761+0000","172.21.15.121:6800/28718860":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/445290372":"2024-02-13T15:47:47.232761+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T16:51:58.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:58 smithi169 bash[19604]: audit 2024-02-12T16:51:57.748637+0000 mon.a (mon.0) 3637 : audit [DBG] from='client.? v1:172.21.15.121:0/2802616411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:51:58.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:58 smithi121 bash[18489]: audit 2024-02-12T16:51:57.748637+0000 mon.a (mon.0) 3637 : audit [DBG] from='client.? v1:172.21.15.121:0/2802616411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:51:58.550 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-12T16:51:58.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-12T16:51:58.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:58 smithi123 bash[19119]: audit 2024-02-12T16:51:57.748637+0000 mon.a (mon.0) 3637 : audit [DBG] from='client.? v1:172.21.15.121:0/2802616411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:51:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:59 smithi169 bash[19604]: cluster 2024-02-12T16:51:57.940228+0000 osd.11 (osd.11) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:59 smithi169 bash[19604]: cluster 2024-02-12T16:51:57.940320+0000 osd.11 (osd.11) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:59 smithi169 bash[19604]: cluster 2024-02-12T16:51:58.190350+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e1108: 12 total, 12 up, 12 in 2024-02-12T16:51:59.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:51:59 smithi169 bash[19604]: cluster 2024-02-12T16:51:58.439746+0000 mgr.y (mgr.14146) 2033 : cluster [DBG] pgmap v3008: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:59 smithi123 bash[19119]: cluster 2024-02-12T16:51:57.940228+0000 osd.11 (osd.11) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:59 smithi123 bash[19119]: cluster 2024-02-12T16:51:57.940320+0000 osd.11 (osd.11) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:59 smithi123 bash[19119]: cluster 2024-02-12T16:51:58.190350+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e1108: 12 total, 12 up, 12 in 2024-02-12T16:51:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:51:59 smithi123 bash[19119]: cluster 2024-02-12T16:51:58.439746+0000 mgr.y (mgr.14146) 2033 : cluster [DBG] pgmap v3008: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:51:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:59 smithi121 bash[18489]: cluster 2024-02-12T16:51:57.940228+0000 osd.11 (osd.11) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:51:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:59 smithi121 bash[18489]: cluster 2024-02-12T16:51:57.940320+0000 osd.11 (osd.11) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:51:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:59 smithi121 bash[18489]: cluster 2024-02-12T16:51:58.190350+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e1108: 12 total, 12 up, 12 in 2024-02-12T16:51:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:51:59 smithi121 bash[18489]: cluster 2024-02-12T16:51:58.439746+0000 mgr.y (mgr.14146) 2033 : cluster [DBG] pgmap v3008: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:01.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:01 smithi169 bash[19604]: cluster 2024-02-12T16:52:00.440395+0000 mgr.y (mgr.14146) 2034 : cluster [DBG] pgmap v3009: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:01.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:01 smithi169 bash[19604]: cluster 2024-02-12T16:52:00.561856+0000 osd.8 (osd.8) 179 : cluster [DBG] 1.1c deep-scrub starts 2024-02-12T16:52:01.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:01 smithi169 bash[19604]: cluster 2024-02-12T16:52:00.635618+0000 osd.8 (osd.8) 180 : cluster [DBG] 1.1c deep-scrub ok 2024-02-12T16:52:01.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:01 smithi169 bash[19604]: cluster 2024-02-12T16:52:00.855868+0000 osd.9 (osd.9) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:01.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:01 smithi169 bash[19604]: cluster 2024-02-12T16:52:00.855956+0000 osd.9 (osd.9) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:01 smithi123 bash[19119]: cluster 2024-02-12T16:52:00.440395+0000 mgr.y (mgr.14146) 2034 : cluster [DBG] pgmap v3009: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:01 smithi123 bash[19119]: cluster 2024-02-12T16:52:00.561856+0000 osd.8 (osd.8) 179 : cluster [DBG] 1.1c deep-scrub starts 2024-02-12T16:52:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:01 smithi123 bash[19119]: cluster 2024-02-12T16:52:00.635618+0000 osd.8 (osd.8) 180 : cluster [DBG] 1.1c deep-scrub ok 2024-02-12T16:52:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:01 smithi123 bash[19119]: cluster 2024-02-12T16:52:00.855868+0000 osd.9 (osd.9) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:01 smithi123 bash[19119]: cluster 2024-02-12T16:52:00.855956+0000 osd.9 (osd.9) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:01 smithi121 bash[18489]: cluster 2024-02-12T16:52:00.440395+0000 mgr.y (mgr.14146) 2034 : cluster [DBG] pgmap v3009: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:01 smithi121 bash[18489]: cluster 2024-02-12T16:52:00.561856+0000 osd.8 (osd.8) 179 : cluster [DBG] 1.1c deep-scrub starts 2024-02-12T16:52:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:01 smithi121 bash[18489]: cluster 2024-02-12T16:52:00.635618+0000 osd.8 (osd.8) 180 : cluster [DBG] 1.1c deep-scrub ok 2024-02-12T16:52:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:01 smithi121 bash[18489]: cluster 2024-02-12T16:52:00.855868+0000 osd.9 (osd.9) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:01 smithi121 bash[18489]: cluster 2024-02-12T16:52:00.855956+0000 osd.9 (osd.9) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:02.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:02 smithi169 bash[19604]: cluster 2024-02-12T16:52:01.511125+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e1109: 12 total, 12 up, 12 in 2024-02-12T16:52:02.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:02 smithi123 bash[19119]: cluster 2024-02-12T16:52:01.511125+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e1109: 12 total, 12 up, 12 in 2024-02-12T16:52:02.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:02 smithi121 bash[18489]: cluster 2024-02-12T16:52:01.511125+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e1109: 12 total, 12 up, 12 in 2024-02-12T16:52:03.352 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:03 smithi169 bash[19604]: cluster 2024-02-12T16:52:02.355918+0000 osd.7 (osd.7) 65 : cluster [DBG] 1.d scrub starts 2024-02-12T16:52:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:03 smithi169 bash[19604]: cluster 2024-02-12T16:52:02.441176+0000 mgr.y (mgr.14146) 2035 : cluster [DBG] pgmap v3011: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:03.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:03 smithi169 bash[19604]: cluster 2024-02-12T16:52:02.464894+0000 osd.7 (osd.7) 66 : cluster [DBG] 1.d scrub ok 2024-02-12T16:52:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:03 smithi123 bash[19119]: cluster 2024-02-12T16:52:02.355918+0000 osd.7 (osd.7) 65 : cluster [DBG] 1.d scrub starts 2024-02-12T16:52:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:03 smithi123 bash[19119]: cluster 2024-02-12T16:52:02.441176+0000 mgr.y (mgr.14146) 2035 : cluster [DBG] pgmap v3011: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:03 smithi123 bash[19119]: cluster 2024-02-12T16:52:02.464894+0000 osd.7 (osd.7) 66 : cluster [DBG] 1.d scrub ok 2024-02-12T16:52:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:03 smithi121 bash[18489]: cluster 2024-02-12T16:52:02.355918+0000 osd.7 (osd.7) 65 : cluster [DBG] 1.d scrub starts 2024-02-12T16:52:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:03 smithi121 bash[18489]: cluster 2024-02-12T16:52:02.441176+0000 mgr.y (mgr.14146) 2035 : cluster [DBG] pgmap v3011: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:03 smithi121 bash[18489]: cluster 2024-02-12T16:52:02.464894+0000 osd.7 (osd.7) 66 : cluster [DBG] 1.d scrub ok 2024-02-12T16:52:05.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:05 smithi169 bash[19604]: cluster 2024-02-12T16:52:04.441878+0000 mgr.y (mgr.14146) 2036 : cluster [DBG] pgmap v3012: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:05 smithi123 bash[19119]: cluster 2024-02-12T16:52:04.441878+0000 mgr.y (mgr.14146) 2036 : cluster [DBG] pgmap v3012: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:05 smithi121 bash[18489]: cluster 2024-02-12T16:52:04.441878+0000 mgr.y (mgr.14146) 2036 : cluster [DBG] pgmap v3012: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:06 smithi169 bash[19604]: cluster 2024-02-12T16:52:04.792705+0000 osd.5 (osd.5) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:06 smithi169 bash[19604]: cluster 2024-02-12T16:52:04.792814+0000 osd.5 (osd.5) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:06 smithi169 bash[19604]: cluster 2024-02-12T16:52:05.203533+0000 osd.5 (osd.5) 263 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:52:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:06 smithi169 bash[19604]: cluster 2024-02-12T16:52:05.282986+0000 osd.5 (osd.5) 264 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:52:06.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:06 smithi169 bash[19604]: cluster 2024-02-12T16:52:05.534166+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e1110: 12 total, 12 up, 12 in 2024-02-12T16:52:06.812 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:06 smithi121 bash[18489]: cluster 2024-02-12T16:52:04.792705+0000 osd.5 (osd.5) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:06.812 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:06 smithi121 bash[18489]: cluster 2024-02-12T16:52:04.792814+0000 osd.5 (osd.5) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:06.812 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:06 smithi121 bash[18489]: cluster 2024-02-12T16:52:05.203533+0000 osd.5 (osd.5) 263 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:52:06.812 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:06 smithi121 bash[18489]: cluster 2024-02-12T16:52:05.282986+0000 osd.5 (osd.5) 264 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:52:06.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:06 smithi121 bash[18489]: cluster 2024-02-12T16:52:05.534166+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e1110: 12 total, 12 up, 12 in 2024-02-12T16:52:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:06 smithi123 bash[19119]: cluster 2024-02-12T16:52:04.792705+0000 osd.5 (osd.5) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:06 smithi123 bash[19119]: cluster 2024-02-12T16:52:04.792814+0000 osd.5 (osd.5) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:06 smithi123 bash[19119]: cluster 2024-02-12T16:52:05.203533+0000 osd.5 (osd.5) 263 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:52:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:06 smithi123 bash[19119]: cluster 2024-02-12T16:52:05.282986+0000 osd.5 (osd.5) 264 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:52:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:06 smithi123 bash[19119]: cluster 2024-02-12T16:52:05.534166+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e1110: 12 total, 12 up, 12 in 2024-02-12T16:52:07.140 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:52:07.141 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":1110,"fsid":"eb2349b2-c9bd-11ee-95b9-87774f69a715","created":"2024-02-12T15:47:18.273813+0000","modified":"2024-02-12T16:52:05.522518+0000","last_up_change":"2024-02-12T16:42:40.138095+0000","last_in_change":"2024-02-12T16:47:42.419336+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":42,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-12T15:51:35.886878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"957","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"663","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"3a3badac-97ea-4ee1-ad82-1b67a4fe94f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1044,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1815232143}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1815232143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":1815232143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1815232143}]},"public_addr":"172.21.15.121:6801/1815232143","cluster_addr":"172.21.15.121:6802/1815232143","heartbeat_back_addr":"172.21.15.121:6804/1815232143","heartbeat_front_addr":"172.21.15.121:6803/1815232143","state":["exists","up"]},{"osd":1,"uuid":"c6140309-5703-4538-9739-276fbbdaa880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1020,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2968433670}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2968433670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":2968433670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2968433670}]},"public_addr":"172.21.15.121:6805/2968433670","cluster_addr":"172.21.15.121:6806/2968433670","heartbeat_back_addr":"172.21.15.121:6808/2968433670","heartbeat_front_addr":"172.21.15.121:6807/2968433670","state":["exists","up"]},{"osd":2,"uuid":"7c81e1a4-cae0-48d4-98ab-0becb6b31170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":889,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":2158187852}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":2158187852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":2158187852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":2158187852}]},"public_addr":"172.21.15.121:6809/2158187852","cluster_addr":"172.21.15.121:6810/2158187852","heartbeat_back_addr":"172.21.15.121:6812/2158187852","heartbeat_front_addr":"172.21.15.121:6811/2158187852","state":["exists","up"]},{"osd":3,"uuid":"5f824526-596d-42e1-8b7e-cd8676ded6dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":414,"up_from":522,"up_thru":1100,"down_at":415,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":2551349828}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":2551349828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6816","nonce":2551349828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":2551349828}]},"public_addr":"172.21.15.121:6813/2551349828","cluster_addr":"172.21.15.121:6814/2551349828","heartbeat_back_addr":"172.21.15.121:6816/2551349828","heartbeat_front_addr":"172.21.15.121:6815/2551349828","state":["exists","up"]},{"osd":4,"uuid":"14c2f878-99ce-4b51-8d9c-131c1d883b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":1021,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6800","nonce":3447700123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":3447700123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":3447700123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":3447700123}]},"public_addr":"172.21.15.123:6800/3447700123","cluster_addr":"172.21.15.123:6801/3447700123","heartbeat_back_addr":"172.21.15.123:6803/3447700123","heartbeat_front_addr":"172.21.15.123:6802/3447700123","state":["exists","up"]},{"osd":5,"uuid":"75ecf26a-d7ed-46e7-a40e-6f719e2c3602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":1086,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":961317306}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":961317306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":961317306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":961317306}]},"public_addr":"172.21.15.123:6804/961317306","cluster_addr":"172.21.15.123:6805/961317306","heartbeat_back_addr":"172.21.15.123:6807/961317306","heartbeat_front_addr":"172.21.15.123:6806/961317306","state":["exists","up"]},{"osd":6,"uuid":"71e715e6-eac1-4386-927e-51df0cc419c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":1037,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":1377261784}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":1377261784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":1377261784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":1377261784}]},"public_addr":"172.21.15.123:6808/1377261784","cluster_addr":"172.21.15.123:6809/1377261784","heartbeat_back_addr":"172.21.15.123:6811/1377261784","heartbeat_front_addr":"172.21.15.123:6810/1377261784","state":["exists","up"]},{"osd":7,"uuid":"b419b04f-0c0c-495a-aa9b-25c334159515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":56,"last_clean_end":848,"up_from":923,"up_thru":1068,"down_at":849,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":4284140534}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":4284140534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":4284140534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":4284140534}]},"public_addr":"172.21.15.123:6812/4284140534","cluster_addr":"172.21.15.123:6813/4284140534","heartbeat_back_addr":"172.21.15.123:6815/4284140534","heartbeat_front_addr":"172.21.15.123:6814/4284140534","state":["exists","up"]},{"osd":8,"uuid":"1591e7bc-5b86-4401-803d-5325062ffb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6800","nonce":2551075225}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6801","nonce":2551075225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6803","nonce":2551075225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6802","nonce":2551075225}]},"public_addr":"172.21.15.169:6800/2551075225","cluster_addr":"172.21.15.169:6801/2551075225","heartbeat_back_addr":"172.21.15.169:6803/2551075225","heartbeat_front_addr":"172.21.15.169:6802/2551075225","state":["exists","up"]},{"osd":9,"uuid":"522d61df-8dc1-4915-83fd-f94d0d572c15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6804","nonce":2339555071}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6805","nonce":2339555071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6807","nonce":2339555071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6806","nonce":2339555071}]},"public_addr":"172.21.15.169:6804/2339555071","cluster_addr":"172.21.15.169:6805/2339555071","heartbeat_back_addr":"172.21.15.169:6807/2339555071","heartbeat_front_addr":"172.21.15.169:6806/2339555071","state":["exists","up"]},{"osd":10,"uuid":"91f74dca-503a-45d1-9647-df4804d58515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":1057,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6808","nonce":391306610}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6809","nonce":391306610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6811","nonce":391306610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6810","nonce":391306610}]},"public_addr":"172.21.15.169:6808/391306610","cluster_addr":"172.21.15.169:6809/391306610","heartbeat_back_addr":"172.21.15.169:6811/391306610","heartbeat_front_addr":"172.21.15.169:6810/391306610","state":["exists","up"]},{"osd":11,"uuid":"788b4f47-eadf-4601-a5b4-51fb6f04fbdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":93,"up_thru":1015,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6812","nonce":3696812305}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6813","nonce":3696812305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6815","nonce":3696812305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.169:6814","nonce":3696812305}]},"public_addr":"172.21.15.169:6812/3696812305","cluster_addr":"172.21.15.169:6813/3696812305","heartbeat_back_addr":"172.21.15.169:6815/3696812305","heartbeat_front_addr":"172.21.15.169:6814/3696812305","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:41.293746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:50:56.199449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:15.063139+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:02.993573+0000","dead_epoch":414},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:15.969593+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:52:04.792820+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:18.070065+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:24.721002+0000","dead_epoch":848},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:18.822208+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:52:00.855961+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:46.162468+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-12T16:51:57.940325+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.1b","osds":[10,5]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/2474849789":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/253689008":"2024-02-13T15:48:17.261083+0000","172.21.15.121:6800/3531151725":"2024-02-13T15:47:47.232761+0000","172.21.15.121:0/1778611055":"2024-02-13T15:47:47.232761+0000","172.21.15.121:6800/28718860":"2024-02-13T15:48:17.261083+0000","172.21.15.121:0/445290372":"2024-02-13T15:47:47.232761+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-12T16:52:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:07 smithi169 bash[19604]: cluster 2024-02-12T16:52:06.442318+0000 mgr.y (mgr.14146) 2037 : cluster [DBG] pgmap v3014: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:07 smithi169 bash[19604]: cluster 2024-02-12T16:52:06.601463+0000 osd.11 (osd.11) 263 : cluster [DBG] 1.b scrub starts 2024-02-12T16:52:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:07 smithi169 bash[19604]: cluster 2024-02-12T16:52:06.762179+0000 osd.11 (osd.11) 264 : cluster [DBG] 1.b scrub ok 2024-02-12T16:52:07.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:07 smithi169 bash[19604]: audit 2024-02-12T16:52:07.133244+0000 mon.a (mon.0) 3641 : audit [DBG] from='client.? v1:172.21.15.121:0/610437901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:52:07.867 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:07 smithi121 bash[18489]: cluster 2024-02-12T16:52:06.442318+0000 mgr.y (mgr.14146) 2037 : cluster [DBG] pgmap v3014: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:07.867 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:07 smithi121 bash[18489]: cluster 2024-02-12T16:52:06.601463+0000 osd.11 (osd.11) 263 : cluster [DBG] 1.b scrub starts 2024-02-12T16:52:07.867 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:07 smithi121 bash[18489]: cluster 2024-02-12T16:52:06.762179+0000 osd.11 (osd.11) 264 : cluster [DBG] 1.b scrub ok 2024-02-12T16:52:07.867 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:07 smithi121 bash[18489]: audit 2024-02-12T16:52:07.133244+0000 mon.a (mon.0) 3641 : audit [DBG] from='client.? v1:172.21.15.121:0/610437901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:52:07.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-12T16:52:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:07 smithi123 bash[19119]: cluster 2024-02-12T16:52:06.442318+0000 mgr.y (mgr.14146) 2037 : cluster [DBG] pgmap v3014: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:07 smithi123 bash[19119]: cluster 2024-02-12T16:52:06.601463+0000 osd.11 (osd.11) 263 : cluster [DBG] 1.b scrub starts 2024-02-12T16:52:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:07 smithi123 bash[19119]: cluster 2024-02-12T16:52:06.762179+0000 osd.11 (osd.11) 264 : cluster [DBG] 1.b scrub ok 2024-02-12T16:52:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:07 smithi123 bash[19119]: audit 2024-02-12T16:52:07.133244+0000 mon.a (mon.0) 3641 : audit [DBG] from='client.? v1:172.21.15.121:0/610437901' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-12T16:52:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:09 smithi123 bash[19119]: cluster 2024-02-12T16:52:08.443070+0000 mgr.y (mgr.14146) 2038 : cluster [DBG] pgmap v3015: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:09 smithi123 bash[19119]: cluster 2024-02-12T16:52:08.450379+0000 osd.7 (osd.7) 67 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:52:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:09 smithi123 bash[19119]: cluster 2024-02-12T16:52:08.519013+0000 osd.7 (osd.7) 68 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:52:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:09 smithi121 bash[18489]: cluster 2024-02-12T16:52:08.443070+0000 mgr.y (mgr.14146) 2038 : cluster [DBG] pgmap v3015: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:09 smithi121 bash[18489]: cluster 2024-02-12T16:52:08.450379+0000 osd.7 (osd.7) 67 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:52:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:09 smithi121 bash[18489]: cluster 2024-02-12T16:52:08.519013+0000 osd.7 (osd.7) 68 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:52:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:09 smithi169 bash[19604]: cluster 2024-02-12T16:52:08.443070+0000 mgr.y (mgr.14146) 2038 : cluster [DBG] pgmap v3015: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:09 smithi169 bash[19604]: cluster 2024-02-12T16:52:08.450379+0000 osd.7 (osd.7) 67 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:52:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:09 smithi169 bash[19604]: cluster 2024-02-12T16:52:08.519013+0000 osd.7 (osd.7) 68 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:52:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:10 smithi123 bash[19119]: cluster 2024-02-12T16:52:08.756224+0000 osd.4 (osd.4) 201 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:52:10.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:10 smithi123 bash[19119]: cluster 2024-02-12T16:52:08.806325+0000 osd.4 (osd.4) 202 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:52:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:10 smithi121 bash[18489]: cluster 2024-02-12T16:52:08.756224+0000 osd.4 (osd.4) 201 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:52:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:10 smithi121 bash[18489]: cluster 2024-02-12T16:52:08.806325+0000 osd.4 (osd.4) 202 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:52:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:10 smithi169 bash[19604]: cluster 2024-02-12T16:52:08.756224+0000 osd.4 (osd.4) 201 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:52:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:10 smithi169 bash[19604]: cluster 2024-02-12T16:52:08.806325+0000 osd.4 (osd.4) 202 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:52:11.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:11 smithi123 bash[19119]: cluster 2024-02-12T16:52:10.443967+0000 mgr.y (mgr.14146) 2039 : cluster [DBG] pgmap v3016: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:11 smithi121 bash[18489]: cluster 2024-02-12T16:52:10.443967+0000 mgr.y (mgr.14146) 2039 : cluster [DBG] pgmap v3016: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:12.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:11 smithi169 bash[19604]: cluster 2024-02-12T16:52:10.443967+0000 mgr.y (mgr.14146) 2039 : cluster [DBG] pgmap v3016: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:12.670 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:13.847 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:13 smithi121 bash[18489]: cluster 2024-02-12T16:52:12.444711+0000 mgr.y (mgr.14146) 2040 : cluster [DBG] pgmap v3017: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:13.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:13 smithi123 bash[19119]: cluster 2024-02-12T16:52:12.444711+0000 mgr.y (mgr.14146) 2040 : cluster [DBG] pgmap v3017: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:13 smithi169 bash[19604]: cluster 2024-02-12T16:52:12.444711+0000 mgr.y (mgr.14146) 2040 : cluster [DBG] pgmap v3017: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:15.865 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:15 smithi121 bash[18489]: cluster 2024-02-12T16:52:13.865573+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:15.866 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:15 smithi121 bash[18489]: cluster 2024-02-12T16:52:13.865711+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:15.866 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:15 smithi121 bash[18489]: cluster 2024-02-12T16:52:14.445529+0000 mgr.y (mgr.14146) 2041 : cluster [DBG] pgmap v3018: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:15.866 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:15 smithi121 bash[18489]: cluster 2024-02-12T16:52:14.584843+0000 mon.a (mon.0) 3642 : cluster [DBG] osdmap e1111: 12 total, 12 up, 12 in 2024-02-12T16:52:15.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:15 smithi123 bash[19119]: cluster 2024-02-12T16:52:13.865573+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:15 smithi123 bash[19119]: cluster 2024-02-12T16:52:13.865711+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:15 smithi123 bash[19119]: cluster 2024-02-12T16:52:14.445529+0000 mgr.y (mgr.14146) 2041 : cluster [DBG] pgmap v3018: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:15.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:15 smithi123 bash[19119]: cluster 2024-02-12T16:52:14.584843+0000 mon.a (mon.0) 3642 : cluster [DBG] osdmap e1111: 12 total, 12 up, 12 in 2024-02-12T16:52:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:15 smithi169 bash[19604]: cluster 2024-02-12T16:52:13.865573+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:15 smithi169 bash[19604]: cluster 2024-02-12T16:52:13.865711+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:15 smithi169 bash[19604]: cluster 2024-02-12T16:52:14.445529+0000 mgr.y (mgr.14146) 2041 : cluster [DBG] pgmap v3018: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:15 smithi169 bash[19604]: cluster 2024-02-12T16:52:14.584843+0000 mon.a (mon.0) 3642 : cluster [DBG] osdmap e1111: 12 total, 12 up, 12 in 2024-02-12T16:52:16.919 INFO:teuthology.orchestra.run.smithi121.stdout:34359739110 2024-02-12T16:52:16.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-12T16:52:17.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:17 smithi169 bash[19604]: cluster 2024-02-12T16:52:16.446242+0000 mgr.y (mgr.14146) 2042 : cluster [DBG] pgmap v3020: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:17 smithi123 bash[19119]: cluster 2024-02-12T16:52:16.446242+0000 mgr.y (mgr.14146) 2042 : cluster [DBG] pgmap v3020: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:17 smithi121 bash[18489]: cluster 2024-02-12T16:52:16.446242+0000 mgr.y (mgr.14146) 2042 : cluster [DBG] pgmap v3020: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:19.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:19 smithi169 bash[19604]: cluster 2024-02-12T16:52:18.446949+0000 mgr.y (mgr.14146) 2043 : cluster [DBG] pgmap v3021: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:19 smithi123 bash[19119]: cluster 2024-02-12T16:52:18.446949+0000 mgr.y (mgr.14146) 2043 : cluster [DBG] pgmap v3021: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:19.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:19 smithi121 bash[18489]: cluster 2024-02-12T16:52:18.446949+0000 mgr.y (mgr.14146) 2043 : cluster [DBG] pgmap v3021: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:20.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:20 smithi169 bash[19604]: cluster 2024-02-12T16:52:19.473985+0000 osd.1 (osd.1) 212 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:52:20.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:20 smithi169 bash[19604]: cluster 2024-02-12T16:52:19.688196+0000 osd.1 (osd.1) 213 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:52:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:20 smithi123 bash[19119]: cluster 2024-02-12T16:52:19.473985+0000 osd.1 (osd.1) 212 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:52:20.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:20 smithi123 bash[19119]: cluster 2024-02-12T16:52:19.688196+0000 osd.1 (osd.1) 213 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:52:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:20 smithi121 bash[18489]: cluster 2024-02-12T16:52:19.473985+0000 osd.1 (osd.1) 212 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:52:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:20 smithi121 bash[18489]: cluster 2024-02-12T16:52:19.688196+0000 osd.1 (osd.1) 213 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:52:21.714 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:21.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:21 smithi169 bash[19604]: cluster 2024-02-12T16:52:20.447644+0000 mgr.y (mgr.14146) 2044 : cluster [DBG] pgmap v3022: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:21.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:21 smithi123 bash[19119]: cluster 2024-02-12T16:52:20.447644+0000 mgr.y (mgr.14146) 2044 : cluster [DBG] pgmap v3022: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:21.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:21 smithi121 bash[18489]: cluster 2024-02-12T16:52:20.447644+0000 mgr.y (mgr.14146) 2044 : cluster [DBG] pgmap v3022: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:22.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:22 smithi169 bash[19604]: cluster 2024-02-12T16:52:21.316446+0000 osd.6 (osd.6) 216 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:22 smithi169 bash[19604]: cluster 2024-02-12T16:52:21.316536+0000 osd.6 (osd.6) 217 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:22.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:22 smithi169 bash[19604]: cluster 2024-02-12T16:52:22.184989+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e1112: 12 total, 12 up, 12 in 2024-02-12T16:52:22.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:22 smithi123 bash[19119]: cluster 2024-02-12T16:52:21.316446+0000 osd.6 (osd.6) 216 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:22.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:22 smithi123 bash[19119]: cluster 2024-02-12T16:52:21.316536+0000 osd.6 (osd.6) 217 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:22.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:22 smithi123 bash[19119]: cluster 2024-02-12T16:52:22.184989+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e1112: 12 total, 12 up, 12 in 2024-02-12T16:52:22.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:22 smithi121 bash[18489]: cluster 2024-02-12T16:52:21.316446+0000 osd.6 (osd.6) 216 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:22.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:22 smithi121 bash[18489]: cluster 2024-02-12T16:52:21.316536+0000 osd.6 (osd.6) 217 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:22.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:22 smithi121 bash[18489]: cluster 2024-02-12T16:52:22.184989+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e1112: 12 total, 12 up, 12 in 2024-02-12T16:52:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:23 smithi169 bash[19604]: cluster 2024-02-12T16:52:22.171505+0000 osd.1 (osd.1) 214 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:23 smithi169 bash[19604]: cluster 2024-02-12T16:52:22.171596+0000 osd.1 (osd.1) 215 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:23 smithi169 bash[19604]: cluster 2024-02-12T16:52:22.448268+0000 mgr.y (mgr.14146) 2045 : cluster [DBG] pgmap v3024: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:23 smithi123 bash[19119]: cluster 2024-02-12T16:52:22.171505+0000 osd.1 (osd.1) 214 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:23 smithi123 bash[19119]: cluster 2024-02-12T16:52:22.171596+0000 osd.1 (osd.1) 215 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:23 smithi123 bash[19119]: cluster 2024-02-12T16:52:22.448268+0000 mgr.y (mgr.14146) 2045 : cluster [DBG] pgmap v3024: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:23 smithi121 bash[18489]: cluster 2024-02-12T16:52:22.171505+0000 osd.1 (osd.1) 214 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:23 smithi121 bash[18489]: cluster 2024-02-12T16:52:22.171596+0000 osd.1 (osd.1) 215 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:23 smithi121 bash[18489]: cluster 2024-02-12T16:52:22.448268+0000 mgr.y (mgr.14146) 2045 : cluster [DBG] pgmap v3024: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:24.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:24 smithi169 bash[19604]: cluster 2024-02-12T16:52:23.484085+0000 osd.8 (osd.8) 181 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:52:24.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:24 smithi169 bash[19604]: cluster 2024-02-12T16:52:23.606039+0000 osd.8 (osd.8) 182 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:52:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:24 smithi123 bash[19119]: cluster 2024-02-12T16:52:23.484085+0000 osd.8 (osd.8) 181 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:52:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:24 smithi123 bash[19119]: cluster 2024-02-12T16:52:23.606039+0000 osd.8 (osd.8) 182 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:52:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:24 smithi121 bash[18489]: cluster 2024-02-12T16:52:23.484085+0000 osd.8 (osd.8) 181 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:52:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:24 smithi121 bash[18489]: cluster 2024-02-12T16:52:23.606039+0000 osd.8 (osd.8) 182 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:52:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:25 smithi121 bash[18489]: cluster 2024-02-12T16:52:23.712313+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:25 smithi121 bash[18489]: cluster 2024-02-12T16:52:23.712403+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:25.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:25 smithi121 bash[18489]: cluster 2024-02-12T16:52:24.448972+0000 mgr.y (mgr.14146) 2046 : cluster [DBG] pgmap v3025: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:25.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:25 smithi121 bash[18489]: cluster 2024-02-12T16:52:24.534065+0000 mon.a (mon.0) 3644 : cluster [DBG] osdmap e1113: 12 total, 12 up, 12 in 2024-02-12T16:52:25.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:25 smithi169 bash[19604]: cluster 2024-02-12T16:52:23.712313+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:25 smithi169 bash[19604]: cluster 2024-02-12T16:52:23.712403+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:25 smithi169 bash[19604]: cluster 2024-02-12T16:52:24.448972+0000 mgr.y (mgr.14146) 2046 : cluster [DBG] pgmap v3025: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:25.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:25 smithi169 bash[19604]: cluster 2024-02-12T16:52:24.534065+0000 mon.a (mon.0) 3644 : cluster [DBG] osdmap e1113: 12 total, 12 up, 12 in 2024-02-12T16:52:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:25 smithi123 bash[19119]: cluster 2024-02-12T16:52:23.712313+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:25 smithi123 bash[19119]: cluster 2024-02-12T16:52:23.712403+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:25 smithi123 bash[19119]: cluster 2024-02-12T16:52:24.448972+0000 mgr.y (mgr.14146) 2046 : cluster [DBG] pgmap v3025: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:25 smithi123 bash[19119]: cluster 2024-02-12T16:52:24.534065+0000 mon.a (mon.0) 3644 : cluster [DBG] osdmap e1113: 12 total, 12 up, 12 in 2024-02-12T16:52:25.988 INFO:teuthology.orchestra.run.smithi121.stdout:55834575580 2024-02-12T16:52:25.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-12T16:52:27.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:27 smithi169 bash[19604]: cluster 2024-02-12T16:52:26.449661+0000 mgr.y (mgr.14146) 2047 : cluster [DBG] pgmap v3027: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:27.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:27 smithi123 bash[19119]: cluster 2024-02-12T16:52:26.449661+0000 mgr.y (mgr.14146) 2047 : cluster [DBG] pgmap v3027: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:27.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:27 smithi121 bash[18489]: cluster 2024-02-12T16:52:26.449661+0000 mgr.y (mgr.14146) 2047 : cluster [DBG] pgmap v3027: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:29 smithi169 bash[19604]: cluster 2024-02-12T16:52:28.315508+0000 osd.10 (osd.10) 292 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:52:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:29 smithi169 bash[19604]: cluster 2024-02-12T16:52:28.421317+0000 osd.10 (osd.10) 293 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:52:29.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:29 smithi169 bash[19604]: cluster 2024-02-12T16:52:28.450288+0000 mgr.y (mgr.14146) 2048 : cluster [DBG] pgmap v3028: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:29 smithi123 bash[19119]: cluster 2024-02-12T16:52:28.315508+0000 osd.10 (osd.10) 292 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:52:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:29 smithi123 bash[19119]: cluster 2024-02-12T16:52:28.421317+0000 osd.10 (osd.10) 293 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:52:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:29 smithi123 bash[19119]: cluster 2024-02-12T16:52:28.450288+0000 mgr.y (mgr.14146) 2048 : cluster [DBG] pgmap v3028: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:29 smithi121 bash[18489]: cluster 2024-02-12T16:52:28.315508+0000 osd.10 (osd.10) 292 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:52:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:29 smithi121 bash[18489]: cluster 2024-02-12T16:52:28.421317+0000 osd.10 (osd.10) 293 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:52:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:29 smithi121 bash[18489]: cluster 2024-02-12T16:52:28.450288+0000 mgr.y (mgr.14146) 2048 : cluster [DBG] pgmap v3028: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:30.788 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:30.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:30 smithi169 bash[19604]: cluster 2024-02-12T16:52:29.466243+0000 osd.11 (osd.11) 265 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:52:30.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:30 smithi169 bash[19604]: cluster 2024-02-12T16:52:29.486920+0000 osd.11 (osd.11) 266 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:52:30.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:30 smithi123 bash[19119]: cluster 2024-02-12T16:52:29.466243+0000 osd.11 (osd.11) 265 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:52:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:30 smithi123 bash[19119]: cluster 2024-02-12T16:52:29.486920+0000 osd.11 (osd.11) 266 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:52:30.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:30 smithi121 bash[18489]: cluster 2024-02-12T16:52:29.466243+0000 osd.11 (osd.11) 265 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:52:30.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:30 smithi121 bash[18489]: cluster 2024-02-12T16:52:29.486920+0000 osd.11 (osd.11) 266 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:52:31.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:31 smithi169 bash[19604]: cluster 2024-02-12T16:52:30.376394+0000 osd.1 (osd.1) 216 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:52:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:31 smithi169 bash[19604]: cluster 2024-02-12T16:52:30.450944+0000 mgr.y (mgr.14146) 2049 : cluster [DBG] pgmap v3029: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:31 smithi169 bash[19604]: cluster 2024-02-12T16:52:30.589922+0000 osd.1 (osd.1) 217 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:52:31.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:31 smithi123 bash[19119]: cluster 2024-02-12T16:52:30.376394+0000 osd.1 (osd.1) 216 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:52:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:31 smithi123 bash[19119]: cluster 2024-02-12T16:52:30.450944+0000 mgr.y (mgr.14146) 2049 : cluster [DBG] pgmap v3029: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:31 smithi123 bash[19119]: cluster 2024-02-12T16:52:30.589922+0000 osd.1 (osd.1) 217 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:52:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:31 smithi121 bash[18489]: cluster 2024-02-12T16:52:30.376394+0000 osd.1 (osd.1) 216 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:52:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:31 smithi121 bash[18489]: cluster 2024-02-12T16:52:30.450944+0000 mgr.y (mgr.14146) 2049 : cluster [DBG] pgmap v3029: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:31 smithi121 bash[18489]: cluster 2024-02-12T16:52:30.589922+0000 osd.1 (osd.1) 217 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:52:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:33 smithi169 bash[19604]: cluster 2024-02-12T16:52:32.021604+0000 osd.6 (osd.6) 218 : cluster [DBG] 1.f scrub starts 2024-02-12T16:52:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:33 smithi169 bash[19604]: cluster 2024-02-12T16:52:32.165726+0000 osd.6 (osd.6) 219 : cluster [DBG] 1.f scrub ok 2024-02-12T16:52:33.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:33 smithi169 bash[19604]: cluster 2024-02-12T16:52:32.451716+0000 mgr.y (mgr.14146) 2050 : cluster [DBG] pgmap v3030: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:33.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:33 smithi123 bash[19119]: cluster 2024-02-12T16:52:32.021604+0000 osd.6 (osd.6) 218 : cluster [DBG] 1.f scrub starts 2024-02-12T16:52:33.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:33 smithi123 bash[19119]: cluster 2024-02-12T16:52:32.165726+0000 osd.6 (osd.6) 219 : cluster [DBG] 1.f scrub ok 2024-02-12T16:52:33.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:33 smithi123 bash[19119]: cluster 2024-02-12T16:52:32.451716+0000 mgr.y (mgr.14146) 2050 : cluster [DBG] pgmap v3030: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:33.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:33 smithi121 bash[18489]: cluster 2024-02-12T16:52:32.021604+0000 osd.6 (osd.6) 218 : cluster [DBG] 1.f scrub starts 2024-02-12T16:52:33.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:33 smithi121 bash[18489]: cluster 2024-02-12T16:52:32.165726+0000 osd.6 (osd.6) 219 : cluster [DBG] 1.f scrub ok 2024-02-12T16:52:33.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:33 smithi121 bash[18489]: cluster 2024-02-12T16:52:32.451716+0000 mgr.y (mgr.14146) 2050 : cluster [DBG] pgmap v3030: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:34 smithi169 bash[19604]: cluster 2024-02-12T16:52:33.273936+0000 osd.10 (osd.10) 294 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:52:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:34 smithi169 bash[19604]: cluster 2024-02-12T16:52:33.407759+0000 osd.10 (osd.10) 295 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:52:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:34 smithi169 bash[19604]: cluster 2024-02-12T16:52:33.507424+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:34 smithi169 bash[19604]: cluster 2024-02-12T16:52:33.507516+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:34 smithi169 bash[19604]: cluster 2024-02-12T16:52:33.536393+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e1114: 12 total, 12 up, 12 in 2024-02-12T16:52:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:34 smithi169 bash[19604]: cluster 2024-02-12T16:52:33.802257+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:34.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:34 smithi169 bash[19604]: cluster 2024-02-12T16:52:33.802359+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:34 smithi123 bash[19119]: cluster 2024-02-12T16:52:33.273936+0000 osd.10 (osd.10) 294 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:52:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:34 smithi123 bash[19119]: cluster 2024-02-12T16:52:33.407759+0000 osd.10 (osd.10) 295 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:52:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:34 smithi123 bash[19119]: cluster 2024-02-12T16:52:33.507424+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:34 smithi123 bash[19119]: cluster 2024-02-12T16:52:33.507516+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:34 smithi123 bash[19119]: cluster 2024-02-12T16:52:33.536393+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e1114: 12 total, 12 up, 12 in 2024-02-12T16:52:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:34 smithi123 bash[19119]: cluster 2024-02-12T16:52:33.802257+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:34 smithi123 bash[19119]: cluster 2024-02-12T16:52:33.802359+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:34 smithi121 bash[18489]: cluster 2024-02-12T16:52:33.273936+0000 osd.10 (osd.10) 294 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:52:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:34 smithi121 bash[18489]: cluster 2024-02-12T16:52:33.407759+0000 osd.10 (osd.10) 295 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:52:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:34 smithi121 bash[18489]: cluster 2024-02-12T16:52:33.507424+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:34.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:34 smithi121 bash[18489]: cluster 2024-02-12T16:52:33.507516+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:34.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:34 smithi121 bash[18489]: cluster 2024-02-12T16:52:33.536393+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e1114: 12 total, 12 up, 12 in 2024-02-12T16:52:34.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:34 smithi121 bash[18489]: cluster 2024-02-12T16:52:33.802257+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:34.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:34 smithi121 bash[18489]: cluster 2024-02-12T16:52:33.802359+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:34.917 INFO:teuthology.orchestra.run.smithi121.stdout:90194313944 2024-02-12T16:52:34.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-12T16:52:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:35 smithi123 bash[19119]: cluster 2024-02-12T16:52:34.445747+0000 osd.7 (osd.7) 71 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:52:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:35 smithi123 bash[19119]: cluster 2024-02-12T16:52:34.452539+0000 mgr.y (mgr.14146) 2051 : cluster [DBG] pgmap v3032: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:35 smithi123 bash[19119]: cluster 2024-02-12T16:52:34.468492+0000 osd.7 (osd.7) 72 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:52:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:35 smithi123 bash[19119]: cluster 2024-02-12T16:52:34.545738+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e1115: 12 total, 12 up, 12 in 2024-02-12T16:52:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:35 smithi123 bash[19119]: audit 2024-02-12T16:52:34.834056+0000 mon.c (mon.1) 125 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:52:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:35 smithi121 bash[18489]: cluster 2024-02-12T16:52:34.445747+0000 osd.7 (osd.7) 71 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:52:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:35 smithi121 bash[18489]: cluster 2024-02-12T16:52:34.452539+0000 mgr.y (mgr.14146) 2051 : cluster [DBG] pgmap v3032: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:35 smithi121 bash[18489]: cluster 2024-02-12T16:52:34.468492+0000 osd.7 (osd.7) 72 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:52:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:35 smithi121 bash[18489]: cluster 2024-02-12T16:52:34.545738+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e1115: 12 total, 12 up, 12 in 2024-02-12T16:52:35.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:35 smithi121 bash[18489]: audit 2024-02-12T16:52:34.834056+0000 mon.c (mon.1) 125 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:52:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:35 smithi169 bash[19604]: cluster 2024-02-12T16:52:34.445747+0000 osd.7 (osd.7) 71 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:52:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:35 smithi169 bash[19604]: cluster 2024-02-12T16:52:34.452539+0000 mgr.y (mgr.14146) 2051 : cluster [DBG] pgmap v3032: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:35 smithi169 bash[19604]: cluster 2024-02-12T16:52:34.468492+0000 osd.7 (osd.7) 72 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:52:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:35 smithi169 bash[19604]: cluster 2024-02-12T16:52:34.545738+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e1115: 12 total, 12 up, 12 in 2024-02-12T16:52:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:35 smithi169 bash[19604]: audit 2024-02-12T16:52:34.834056+0000 mon.c (mon.1) 125 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:52:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:37 smithi169 bash[19604]: cluster 2024-02-12T16:52:36.368400+0000 osd.10 (osd.10) 296 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:52:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:37 smithi169 bash[19604]: cluster 2024-02-12T16:52:36.453325+0000 mgr.y (mgr.14146) 2052 : cluster [DBG] pgmap v3034: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:37 smithi169 bash[19604]: cluster 2024-02-12T16:52:36.475185+0000 osd.10 (osd.10) 297 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:52:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:37 smithi123 bash[19119]: cluster 2024-02-12T16:52:36.368400+0000 osd.10 (osd.10) 296 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:52:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:37 smithi123 bash[19119]: cluster 2024-02-12T16:52:36.453325+0000 mgr.y (mgr.14146) 2052 : cluster [DBG] pgmap v3034: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:37 smithi123 bash[19119]: cluster 2024-02-12T16:52:36.475185+0000 osd.10 (osd.10) 297 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:52:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:37 smithi121 bash[18489]: cluster 2024-02-12T16:52:36.368400+0000 osd.10 (osd.10) 296 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:52:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:37 smithi121 bash[18489]: cluster 2024-02-12T16:52:36.453325+0000 mgr.y (mgr.14146) 2052 : cluster [DBG] pgmap v3034: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:37 smithi121 bash[18489]: cluster 2024-02-12T16:52:36.475185+0000 osd.10 (osd.10) 297 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:52:39.712 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:39 smithi169 bash[19604]: cluster 2024-02-12T16:52:38.390169+0000 osd.10 (osd.10) 298 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:52:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:39 smithi169 bash[19604]: cluster 2024-02-12T16:52:38.439555+0000 osd.10 (osd.10) 299 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:52:39.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:39 smithi169 bash[19604]: cluster 2024-02-12T16:52:38.454084+0000 mgr.y (mgr.14146) 2053 : cluster [DBG] pgmap v3035: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:39.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:39 smithi123 bash[19119]: cluster 2024-02-12T16:52:38.390169+0000 osd.10 (osd.10) 298 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:52:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:39 smithi123 bash[19119]: cluster 2024-02-12T16:52:38.439555+0000 osd.10 (osd.10) 299 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:52:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:39 smithi123 bash[19119]: cluster 2024-02-12T16:52:38.454084+0000 mgr.y (mgr.14146) 2053 : cluster [DBG] pgmap v3035: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:39 smithi121 bash[18489]: cluster 2024-02-12T16:52:38.390169+0000 osd.10 (osd.10) 298 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:52:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:39 smithi121 bash[18489]: cluster 2024-02-12T16:52:38.439555+0000 osd.10 (osd.10) 299 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:52:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:39 smithi121 bash[18489]: cluster 2024-02-12T16:52:38.454084+0000 mgr.y (mgr.14146) 2053 : cluster [DBG] pgmap v3035: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:41.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:41 smithi121 bash[18489]: audit 2024-02-12T16:52:40.243805+0000 mon.a (mon.0) 3647 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:52:41.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:41 smithi121 bash[18489]: cluster 2024-02-12T16:52:40.454919+0000 mgr.y (mgr.14146) 2054 : cluster [DBG] pgmap v3036: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:41.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:41 smithi121 bash[18489]: audit 2024-02-12T16:52:40.533582+0000 mon.c (mon.1) 126 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:52:41.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:41 smithi121 bash[18489]: audit 2024-02-12T16:52:40.535592+0000 mon.c (mon.1) 127 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:52:41.411 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:41 smithi121 bash[18489]: audit 2024-02-12T16:52:40.545180+0000 mon.a (mon.0) 3648 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:52:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:41 smithi169 bash[19604]: audit 2024-02-12T16:52:40.243805+0000 mon.a (mon.0) 3647 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:52:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:41 smithi169 bash[19604]: cluster 2024-02-12T16:52:40.454919+0000 mgr.y (mgr.14146) 2054 : cluster [DBG] pgmap v3036: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:41 smithi169 bash[19604]: audit 2024-02-12T16:52:40.533582+0000 mon.c (mon.1) 126 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:52:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:41 smithi169 bash[19604]: audit 2024-02-12T16:52:40.535592+0000 mon.c (mon.1) 127 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:52:41.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:41 smithi169 bash[19604]: audit 2024-02-12T16:52:40.545180+0000 mon.a (mon.0) 3648 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:52:41.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:41 smithi123 bash[19119]: audit 2024-02-12T16:52:40.243805+0000 mon.a (mon.0) 3647 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:52:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:41 smithi123 bash[19119]: cluster 2024-02-12T16:52:40.454919+0000 mgr.y (mgr.14146) 2054 : cluster [DBG] pgmap v3036: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:41 smithi123 bash[19119]: audit 2024-02-12T16:52:40.533582+0000 mon.c (mon.1) 126 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:52:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:41 smithi123 bash[19119]: audit 2024-02-12T16:52:40.535592+0000 mon.c (mon.1) 127 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:52:41.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:41 smithi123 bash[19119]: audit 2024-02-12T16:52:40.545180+0000 mon.a (mon.0) 3648 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:52:42.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:42 smithi123 bash[19119]: cluster 2024-02-12T16:52:42.153890+0000 osd.9 (osd.9) 185 : cluster [DBG] 1.a scrub starts 2024-02-12T16:52:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:42 smithi123 bash[19119]: cluster 2024-02-12T16:52:42.246439+0000 osd.9 (osd.9) 186 : cluster [DBG] 1.a scrub ok 2024-02-12T16:52:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:42 smithi121 bash[18489]: cluster 2024-02-12T16:52:42.153890+0000 osd.9 (osd.9) 185 : cluster [DBG] 1.a scrub starts 2024-02-12T16:52:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:42 smithi121 bash[18489]: cluster 2024-02-12T16:52:42.246439+0000 osd.9 (osd.9) 186 : cluster [DBG] 1.a scrub ok 2024-02-12T16:52:42.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:42 smithi169 bash[19604]: cluster 2024-02-12T16:52:42.153890+0000 osd.9 (osd.9) 185 : cluster [DBG] 1.a scrub starts 2024-02-12T16:52:42.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:42 smithi169 bash[19604]: cluster 2024-02-12T16:52:42.246439+0000 osd.9 (osd.9) 186 : cluster [DBG] 1.a scrub ok 2024-02-12T16:52:43.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:43 smithi123 bash[19119]: cluster 2024-02-12T16:52:42.455777+0000 mgr.y (mgr.14146) 2055 : cluster [DBG] pgmap v3037: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:43 smithi121 bash[18489]: cluster 2024-02-12T16:52:42.455777+0000 mgr.y (mgr.14146) 2055 : cluster [DBG] pgmap v3037: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:43.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:43 smithi169 bash[19604]: cluster 2024-02-12T16:52:42.455777+0000 mgr.y (mgr.14146) 2055 : cluster [DBG] pgmap v3037: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:43.876 INFO:teuthology.orchestra.run.smithi121.stdout:2241972928870 2024-02-12T16:52:43.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-12T16:52:45.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:45 smithi169 bash[19604]: cluster 2024-02-12T16:52:44.456521+0000 mgr.y (mgr.14146) 2056 : cluster [DBG] pgmap v3038: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:45.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:45 smithi123 bash[19119]: cluster 2024-02-12T16:52:44.456521+0000 mgr.y (mgr.14146) 2056 : cluster [DBG] pgmap v3038: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:45 smithi121 bash[18489]: cluster 2024-02-12T16:52:44.456521+0000 mgr.y (mgr.14146) 2056 : cluster [DBG] pgmap v3038: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:46 smithi169 bash[19604]: cluster 2024-02-12T16:52:45.404805+0000 osd.1 (osd.1) 218 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:52:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:46 smithi169 bash[19604]: cluster 2024-02-12T16:52:45.497467+0000 osd.4 (osd.4) 205 : cluster [DBG] 1.e scrub starts 2024-02-12T16:52:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:46 smithi169 bash[19604]: cluster 2024-02-12T16:52:45.561075+0000 osd.1 (osd.1) 219 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:52:46.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:46 smithi169 bash[19604]: cluster 2024-02-12T16:52:45.591740+0000 osd.4 (osd.4) 206 : cluster [DBG] 1.e scrub ok 2024-02-12T16:52:46.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:46 smithi123 bash[19119]: cluster 2024-02-12T16:52:45.404805+0000 osd.1 (osd.1) 218 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:52:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:46 smithi123 bash[19119]: cluster 2024-02-12T16:52:45.497467+0000 osd.4 (osd.4) 205 : cluster [DBG] 1.e scrub starts 2024-02-12T16:52:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:46 smithi123 bash[19119]: cluster 2024-02-12T16:52:45.561075+0000 osd.1 (osd.1) 219 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:52:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:46 smithi123 bash[19119]: cluster 2024-02-12T16:52:45.591740+0000 osd.4 (osd.4) 206 : cluster [DBG] 1.e scrub ok 2024-02-12T16:52:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:46 smithi121 bash[18489]: cluster 2024-02-12T16:52:45.404805+0000 osd.1 (osd.1) 218 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:52:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:46 smithi121 bash[18489]: cluster 2024-02-12T16:52:45.497467+0000 osd.4 (osd.4) 205 : cluster [DBG] 1.e scrub starts 2024-02-12T16:52:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:46 smithi121 bash[18489]: cluster 2024-02-12T16:52:45.561075+0000 osd.1 (osd.1) 219 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:52:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:46 smithi121 bash[18489]: cluster 2024-02-12T16:52:45.591740+0000 osd.4 (osd.4) 206 : cluster [DBG] 1.e scrub ok 2024-02-12T16:52:47.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:47 smithi169 bash[19604]: cluster 2024-02-12T16:52:46.457210+0000 mgr.y (mgr.14146) 2057 : cluster [DBG] pgmap v3039: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:47.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:47 smithi123 bash[19119]: cluster 2024-02-12T16:52:46.457210+0000 mgr.y (mgr.14146) 2057 : cluster [DBG] pgmap v3039: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:47.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:47 smithi121 bash[18489]: cluster 2024-02-12T16:52:46.457210+0000 mgr.y (mgr.14146) 2057 : cluster [DBG] pgmap v3039: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:48.674 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:49.788 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:49 smithi121 bash[18489]: cluster 2024-02-12T16:52:48.457960+0000 mgr.y (mgr.14146) 2058 : cluster [DBG] pgmap v3040: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:49.788 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:49 smithi121 bash[18489]: cluster 2024-02-12T16:52:49.157652+0000 osd.8 (osd.8) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:49.789 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:49 smithi121 bash[18489]: cluster 2024-02-12T16:52:49.157750+0000 osd.8 (osd.8) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:49 smithi169 bash[19604]: cluster 2024-02-12T16:52:48.457960+0000 mgr.y (mgr.14146) 2058 : cluster [DBG] pgmap v3040: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:49 smithi169 bash[19604]: cluster 2024-02-12T16:52:49.157652+0000 osd.8 (osd.8) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:49.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:49 smithi169 bash[19604]: cluster 2024-02-12T16:52:49.157750+0000 osd.8 (osd.8) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:49.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:49 smithi123 bash[19119]: cluster 2024-02-12T16:52:48.457960+0000 mgr.y (mgr.14146) 2058 : cluster [DBG] pgmap v3040: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:49 smithi123 bash[19119]: cluster 2024-02-12T16:52:49.157652+0000 osd.8 (osd.8) 183 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:49.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:49 smithi123 bash[19119]: cluster 2024-02-12T16:52:49.157750+0000 osd.8 (osd.8) 184 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:50 smithi169 bash[19604]: cluster 2024-02-12T16:52:49.422766+0000 osd.5 (osd.5) 265 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:52:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:50 smithi169 bash[19604]: cluster 2024-02-12T16:52:49.528603+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e1116: 12 total, 12 up, 12 in 2024-02-12T16:52:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:50 smithi169 bash[19604]: cluster 2024-02-12T16:52:49.542068+0000 osd.5 (osd.5) 266 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:52:50.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:50 smithi123 bash[19119]: cluster 2024-02-12T16:52:49.422766+0000 osd.5 (osd.5) 265 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:52:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:50 smithi123 bash[19119]: cluster 2024-02-12T16:52:49.528603+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e1116: 12 total, 12 up, 12 in 2024-02-12T16:52:50.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:50 smithi123 bash[19119]: cluster 2024-02-12T16:52:49.542068+0000 osd.5 (osd.5) 266 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:52:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:50 smithi121 bash[18489]: cluster 2024-02-12T16:52:49.422766+0000 osd.5 (osd.5) 265 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:52:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:50 smithi121 bash[18489]: cluster 2024-02-12T16:52:49.528603+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e1116: 12 total, 12 up, 12 in 2024-02-12T16:52:50.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:50 smithi121 bash[18489]: cluster 2024-02-12T16:52:49.542068+0000 osd.5 (osd.5) 266 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:52:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:51 smithi123 bash[19119]: cluster 2024-02-12T16:52:50.282894+0000 osd.7 (osd.7) 73 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:52:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:51 smithi123 bash[19119]: cluster 2024-02-12T16:52:50.391496+0000 osd.7 (osd.7) 74 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:52:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:51 smithi123 bash[19119]: cluster 2024-02-12T16:52:50.458585+0000 mgr.y (mgr.14146) 2059 : cluster [DBG] pgmap v3042: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:51 smithi123 bash[19119]: cluster 2024-02-12T16:52:50.458781+0000 osd.0 (osd.0) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:51 smithi123 bash[19119]: cluster 2024-02-12T16:52:50.458864+0000 osd.0 (osd.0) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:51 smithi123 bash[19119]: cluster 2024-02-12T16:52:50.546422+0000 mon.a (mon.0) 3650 : cluster [DBG] osdmap e1117: 12 total, 12 up, 12 in 2024-02-12T16:52:51.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:51 smithi121 bash[18489]: cluster 2024-02-12T16:52:50.282894+0000 osd.7 (osd.7) 73 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:52:51.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:51 smithi121 bash[18489]: cluster 2024-02-12T16:52:50.391496+0000 osd.7 (osd.7) 74 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:52:51.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:51 smithi121 bash[18489]: cluster 2024-02-12T16:52:50.458585+0000 mgr.y (mgr.14146) 2059 : cluster [DBG] pgmap v3042: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:51.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:51 smithi121 bash[18489]: cluster 2024-02-12T16:52:50.458781+0000 osd.0 (osd.0) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:51.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:51 smithi121 bash[18489]: cluster 2024-02-12T16:52:50.458864+0000 osd.0 (osd.0) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:51.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:51 smithi121 bash[18489]: cluster 2024-02-12T16:52:50.546422+0000 mon.a (mon.0) 3650 : cluster [DBG] osdmap e1117: 12 total, 12 up, 12 in 2024-02-12T16:52:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:51 smithi169 bash[19604]: cluster 2024-02-12T16:52:50.282894+0000 osd.7 (osd.7) 73 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:52:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:51 smithi169 bash[19604]: cluster 2024-02-12T16:52:50.391496+0000 osd.7 (osd.7) 74 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:52:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:51 smithi169 bash[19604]: cluster 2024-02-12T16:52:50.458585+0000 mgr.y (mgr.14146) 2059 : cluster [DBG] pgmap v3042: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:51 smithi169 bash[19604]: cluster 2024-02-12T16:52:50.458781+0000 osd.0 (osd.0) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:52:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:51 smithi169 bash[19604]: cluster 2024-02-12T16:52:50.458864+0000 osd.0 (osd.0) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:52:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:51 smithi169 bash[19604]: cluster 2024-02-12T16:52:50.546422+0000 mon.a (mon.0) 3650 : cluster [DBG] osdmap e1117: 12 total, 12 up, 12 in 2024-02-12T16:52:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:52 smithi123 bash[19119]: cluster 2024-02-12T16:52:51.462306+0000 osd.0 (osd.0) 189 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:52:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:52 smithi123 bash[19119]: cluster 2024-02-12T16:52:51.574950+0000 osd.0 (osd.0) 190 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:52:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:52 smithi121 bash[18489]: cluster 2024-02-12T16:52:51.462306+0000 osd.0 (osd.0) 189 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:52:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:52 smithi121 bash[18489]: cluster 2024-02-12T16:52:51.574950+0000 osd.0 (osd.0) 190 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:52:52.961 INFO:teuthology.orchestra.run.smithi121.stdout:137438954187 2024-02-12T16:52:52.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-12T16:52:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:52 smithi169 bash[19604]: cluster 2024-02-12T16:52:51.462306+0000 osd.0 (osd.0) 189 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:52:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:52 smithi169 bash[19604]: cluster 2024-02-12T16:52:51.574950+0000 osd.0 (osd.0) 190 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:52:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:53 smithi123 bash[19119]: cluster 2024-02-12T16:52:52.459471+0000 mgr.y (mgr.14146) 2060 : cluster [DBG] pgmap v3044: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:53 smithi121 bash[18489]: cluster 2024-02-12T16:52:52.459471+0000 mgr.y (mgr.14146) 2060 : cluster [DBG] pgmap v3044: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:53 smithi169 bash[19604]: cluster 2024-02-12T16:52:52.459471+0000 mgr.y (mgr.14146) 2060 : cluster [DBG] pgmap v3044: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:54.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:54 smithi123 bash[19119]: cluster 2024-02-12T16:52:53.220693+0000 osd.10 (osd.10) 300 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:52:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:54 smithi123 bash[19119]: cluster 2024-02-12T16:52:53.292668+0000 osd.10 (osd.10) 301 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:52:54.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:54 smithi121 bash[18489]: cluster 2024-02-12T16:52:53.220693+0000 osd.10 (osd.10) 300 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:52:54.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:54 smithi121 bash[18489]: cluster 2024-02-12T16:52:53.292668+0000 osd.10 (osd.10) 301 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:52:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:54 smithi169 bash[19604]: cluster 2024-02-12T16:52:53.220693+0000 osd.10 (osd.10) 300 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:52:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:54 smithi169 bash[19604]: cluster 2024-02-12T16:52:53.292668+0000 osd.10 (osd.10) 301 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:52:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:55 smithi123 bash[19119]: cluster 2024-02-12T16:52:54.460221+0000 mgr.y (mgr.14146) 2061 : cluster [DBG] pgmap v3045: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:55 smithi121 bash[18489]: cluster 2024-02-12T16:52:54.460221+0000 mgr.y (mgr.14146) 2061 : cluster [DBG] pgmap v3045: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:55 smithi169 bash[19604]: cluster 2024-02-12T16:52:54.460221+0000 mgr.y (mgr.14146) 2061 : cluster [DBG] pgmap v3045: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:56.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:56 smithi123 bash[19119]: cluster 2024-02-12T16:52:55.427588+0000 osd.2 (osd.2) 165 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:52:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:56 smithi123 bash[19119]: cluster 2024-02-12T16:52:55.532390+0000 osd.2 (osd.2) 166 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:52:56.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:56 smithi121 bash[18489]: cluster 2024-02-12T16:52:55.427588+0000 osd.2 (osd.2) 165 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:52:56.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:56 smithi121 bash[18489]: cluster 2024-02-12T16:52:55.532390+0000 osd.2 (osd.2) 166 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:52:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:56 smithi169 bash[19604]: cluster 2024-02-12T16:52:55.427588+0000 osd.2 (osd.2) 165 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:52:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:56 smithi169 bash[19604]: cluster 2024-02-12T16:52:55.532390+0000 osd.2 (osd.2) 166 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:52:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:52:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:57 smithi123 bash[19119]: cluster 2024-02-12T16:52:56.460904+0000 mgr.y (mgr.14146) 2062 : cluster [DBG] pgmap v3046: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:57 smithi123 bash[19119]: cluster 2024-02-12T16:52:56.558729+0000 osd.3 (osd.3) 154 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:52:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:57 smithi123 bash[19119]: cluster 2024-02-12T16:52:56.705432+0000 osd.3 (osd.3) 155 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:52:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:57 smithi121 bash[18489]: cluster 2024-02-12T16:52:56.460904+0000 mgr.y (mgr.14146) 2062 : cluster [DBG] pgmap v3046: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:57 smithi121 bash[18489]: cluster 2024-02-12T16:52:56.558729+0000 osd.3 (osd.3) 154 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:52:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:57 smithi121 bash[18489]: cluster 2024-02-12T16:52:56.705432+0000 osd.3 (osd.3) 155 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:52:58.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:57 smithi169 bash[19604]: cluster 2024-02-12T16:52:56.460904+0000 mgr.y (mgr.14146) 2062 : cluster [DBG] pgmap v3046: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:58.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:57 smithi169 bash[19604]: cluster 2024-02-12T16:52:56.558729+0000 osd.3 (osd.3) 154 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:52:58.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:57 smithi169 bash[19604]: cluster 2024-02-12T16:52:56.705432+0000 osd.3 (osd.3) 155 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:52:59.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:59 smithi123 bash[19119]: cluster 2024-02-12T16:52:58.145610+0000 osd.6 (osd.6) 220 : cluster [DBG] 1.7 deep-scrub starts 2024-02-12T16:52:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:59 smithi123 bash[19119]: cluster 2024-02-12T16:52:58.274914+0000 osd.6 (osd.6) 221 : cluster [DBG] 1.7 deep-scrub ok 2024-02-12T16:52:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:59 smithi123 bash[19119]: cluster 2024-02-12T16:52:58.461576+0000 mgr.y (mgr.14146) 2063 : cluster [DBG] pgmap v3047: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:59 smithi123 bash[19119]: cluster 2024-02-12T16:52:58.509134+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:52:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:59 smithi123 bash[19119]: cluster 2024-02-12T16:52:58.512555+0000 osd.3 (osd.3) 156 : cluster [DBG] 1.c scrub starts 2024-02-12T16:52:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:59 smithi123 bash[19119]: cluster 2024-02-12T16:52:58.603792+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:52:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:52:59 smithi123 bash[19119]: cluster 2024-02-12T16:52:58.659583+0000 osd.3 (osd.3) 157 : cluster [DBG] 1.c scrub ok 2024-02-12T16:52:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:59 smithi121 bash[18489]: cluster 2024-02-12T16:52:58.145610+0000 osd.6 (osd.6) 220 : cluster [DBG] 1.7 deep-scrub starts 2024-02-12T16:52:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:59 smithi121 bash[18489]: cluster 2024-02-12T16:52:58.274914+0000 osd.6 (osd.6) 221 : cluster [DBG] 1.7 deep-scrub ok 2024-02-12T16:52:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:59 smithi121 bash[18489]: cluster 2024-02-12T16:52:58.461576+0000 mgr.y (mgr.14146) 2063 : cluster [DBG] pgmap v3047: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:52:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:59 smithi121 bash[18489]: cluster 2024-02-12T16:52:58.509134+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:52:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:59 smithi121 bash[18489]: cluster 2024-02-12T16:52:58.512555+0000 osd.3 (osd.3) 156 : cluster [DBG] 1.c scrub starts 2024-02-12T16:52:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:59 smithi121 bash[18489]: cluster 2024-02-12T16:52:58.603792+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:52:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:52:59 smithi121 bash[18489]: cluster 2024-02-12T16:52:58.659583+0000 osd.3 (osd.3) 157 : cluster [DBG] 1.c scrub ok 2024-02-12T16:53:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:59 smithi169 bash[19604]: cluster 2024-02-12T16:52:58.145610+0000 osd.6 (osd.6) 220 : cluster [DBG] 1.7 deep-scrub starts 2024-02-12T16:53:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:59 smithi169 bash[19604]: cluster 2024-02-12T16:52:58.274914+0000 osd.6 (osd.6) 221 : cluster [DBG] 1.7 deep-scrub ok 2024-02-12T16:53:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:59 smithi169 bash[19604]: cluster 2024-02-12T16:52:58.461576+0000 mgr.y (mgr.14146) 2063 : cluster [DBG] pgmap v3047: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:59 smithi169 bash[19604]: cluster 2024-02-12T16:52:58.509134+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:53:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:59 smithi169 bash[19604]: cluster 2024-02-12T16:52:58.512555+0000 osd.3 (osd.3) 156 : cluster [DBG] 1.c scrub starts 2024-02-12T16:53:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:59 smithi169 bash[19604]: cluster 2024-02-12T16:52:58.603792+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:53:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:52:59 smithi169 bash[19604]: cluster 2024-02-12T16:52:58.659583+0000 osd.3 (osd.3) 157 : cluster [DBG] 1.c scrub ok 2024-02-12T16:53:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:00 smithi123 bash[19119]: cluster 2024-02-12T16:53:00.192299+0000 osd.5 (osd.5) 267 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:53:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:00 smithi123 bash[19119]: cluster 2024-02-12T16:53:00.264964+0000 osd.5 (osd.5) 268 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:53:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:00 smithi121 bash[18489]: cluster 2024-02-12T16:53:00.192299+0000 osd.5 (osd.5) 267 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:53:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:00 smithi121 bash[18489]: cluster 2024-02-12T16:53:00.264964+0000 osd.5 (osd.5) 268 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:53:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:00 smithi169 bash[19604]: cluster 2024-02-12T16:53:00.192299+0000 osd.5 (osd.5) 267 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:53:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:00 smithi169 bash[19604]: cluster 2024-02-12T16:53:00.264964+0000 osd.5 (osd.5) 268 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:53:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:01 smithi123 bash[19119]: cluster 2024-02-12T16:53:00.422482+0000 osd.4 (osd.4) 207 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:53:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:01 smithi123 bash[19119]: cluster 2024-02-12T16:53:00.462315+0000 mgr.y (mgr.14146) 2064 : cluster [DBG] pgmap v3048: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:01.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:01 smithi123 bash[19119]: cluster 2024-02-12T16:53:00.515421+0000 osd.4 (osd.4) 208 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:53:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:01 smithi121 bash[18489]: cluster 2024-02-12T16:53:00.422482+0000 osd.4 (osd.4) 207 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:53:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:01 smithi121 bash[18489]: cluster 2024-02-12T16:53:00.462315+0000 mgr.y (mgr.14146) 2064 : cluster [DBG] pgmap v3048: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:01 smithi121 bash[18489]: cluster 2024-02-12T16:53:00.515421+0000 osd.4 (osd.4) 208 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:53:02.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:01 smithi169 bash[19604]: cluster 2024-02-12T16:53:00.422482+0000 osd.4 (osd.4) 207 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:53:02.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:01 smithi169 bash[19604]: cluster 2024-02-12T16:53:00.462315+0000 mgr.y (mgr.14146) 2064 : cluster [DBG] pgmap v3048: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:02.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:01 smithi169 bash[19604]: cluster 2024-02-12T16:53:00.515421+0000 osd.4 (osd.4) 208 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:53:02.053 INFO:teuthology.orchestra.run.smithi121.stdout:167503725253 2024-02-12T16:53:02.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-12T16:53:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:03 smithi123 bash[19119]: cluster 2024-02-12T16:53:02.463061+0000 mgr.y (mgr.14146) 2065 : cluster [DBG] pgmap v3049: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:03 smithi121 bash[18489]: cluster 2024-02-12T16:53:02.463061+0000 mgr.y (mgr.14146) 2065 : cluster [DBG] pgmap v3049: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:03 smithi169 bash[19604]: cluster 2024-02-12T16:53:02.463061+0000 mgr.y (mgr.14146) 2065 : cluster [DBG] pgmap v3049: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:05 smithi123 bash[19119]: cluster 2024-02-12T16:53:04.463770+0000 mgr.y (mgr.14146) 2066 : cluster [DBG] pgmap v3050: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:05 smithi121 bash[18489]: cluster 2024-02-12T16:53:04.463770+0000 mgr.y (mgr.14146) 2066 : cluster [DBG] pgmap v3050: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:05 smithi169 bash[19604]: cluster 2024-02-12T16:53:04.463770+0000 mgr.y (mgr.14146) 2066 : cluster [DBG] pgmap v3050: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:06.853 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:06 smithi123 bash[19119]: cluster 2024-02-12T16:53:05.185681+0000 osd.10 (osd.10) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:06 smithi123 bash[19119]: cluster 2024-02-12T16:53:05.185775+0000 osd.10 (osd.10) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:06 smithi123 bash[19119]: cluster 2024-02-12T16:53:05.610449+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e1118: 12 total, 12 up, 12 in 2024-02-12T16:53:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:06 smithi121 bash[18489]: cluster 2024-02-12T16:53:05.185681+0000 osd.10 (osd.10) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:06 smithi121 bash[18489]: cluster 2024-02-12T16:53:05.185775+0000 osd.10 (osd.10) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:06 smithi121 bash[18489]: cluster 2024-02-12T16:53:05.610449+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e1118: 12 total, 12 up, 12 in 2024-02-12T16:53:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:06 smithi169 bash[19604]: cluster 2024-02-12T16:53:05.185681+0000 osd.10 (osd.10) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:06 smithi169 bash[19604]: cluster 2024-02-12T16:53:05.185775+0000 osd.10 (osd.10) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:06 smithi169 bash[19604]: cluster 2024-02-12T16:53:05.610449+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e1118: 12 total, 12 up, 12 in 2024-02-12T16:53:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:07 smithi123 bash[19119]: cluster 2024-02-12T16:53:06.464425+0000 mgr.y (mgr.14146) 2067 : cluster [DBG] pgmap v3052: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:07 smithi121 bash[18489]: cluster 2024-02-12T16:53:06.464425+0000 mgr.y (mgr.14146) 2067 : cluster [DBG] pgmap v3052: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:07 smithi169 bash[19604]: cluster 2024-02-12T16:53:06.464425+0000 mgr.y (mgr.14146) 2067 : cluster [DBG] pgmap v3052: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:09 smithi123 bash[19119]: cluster 2024-02-12T16:53:08.465115+0000 mgr.y (mgr.14146) 2068 : cluster [DBG] pgmap v3053: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:09 smithi121 bash[18489]: cluster 2024-02-12T16:53:08.465115+0000 mgr.y (mgr.14146) 2068 : cluster [DBG] pgmap v3053: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:10.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:09 smithi169 bash[19604]: cluster 2024-02-12T16:53:08.465115+0000 mgr.y (mgr.14146) 2068 : cluster [DBG] pgmap v3053: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:10.849 INFO:teuthology.orchestra.run.smithi121.stdout:206158430912 2024-02-12T16:53:10.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-12T16:53:11.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:11 smithi123 bash[19119]: cluster 2024-02-12T16:53:10.465834+0000 mgr.y (mgr.14146) 2069 : cluster [DBG] pgmap v3054: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:11 smithi121 bash[18489]: cluster 2024-02-12T16:53:10.465834+0000 mgr.y (mgr.14146) 2069 : cluster [DBG] pgmap v3054: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:12.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:11 smithi169 bash[19604]: cluster 2024-02-12T16:53:10.465834+0000 mgr.y (mgr.14146) 2069 : cluster [DBG] pgmap v3054: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:12.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:12 smithi123 bash[19119]: cluster 2024-02-12T16:53:12.218495+0000 osd.8 (osd.8) 186 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:53:12.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:12 smithi121 bash[18489]: cluster 2024-02-12T16:53:12.218495+0000 osd.8 (osd.8) 186 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:53:13.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:12 smithi169 bash[19604]: cluster 2024-02-12T16:53:12.218495+0000 osd.8 (osd.8) 186 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:53:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:13 smithi121 bash[18489]: cluster 2024-02-12T16:53:12.453591+0000 osd.11 (osd.11) 267 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:53:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:13 smithi121 bash[18489]: cluster 2024-02-12T16:53:12.456629+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.d scrub starts 2024-02-12T16:53:13.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:13 smithi121 bash[18489]: cluster 2024-02-12T16:53:12.466548+0000 mgr.y (mgr.14146) 2070 : cluster [DBG] pgmap v3055: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:13.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:13 smithi121 bash[18489]: cluster 2024-02-12T16:53:12.536007+0000 osd.11 (osd.11) 268 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:53:13.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:13 smithi121 bash[18489]: cluster 2024-02-12T16:53:12.577685+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.d scrub ok 2024-02-12T16:53:13.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:13 smithi121 bash[18489]: cluster 2024-02-12T16:53:13.139514+0000 osd.5 (osd.5) 269 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:53:13.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:13 smithi121 bash[18489]: cluster 2024-02-12T16:53:13.219134+0000 osd.5 (osd.5) 270 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:53:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:13 smithi169 bash[19604]: cluster 2024-02-12T16:53:12.453591+0000 osd.11 (osd.11) 267 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:53:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:13 smithi169 bash[19604]: cluster 2024-02-12T16:53:12.456629+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.d scrub starts 2024-02-12T16:53:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:13 smithi169 bash[19604]: cluster 2024-02-12T16:53:12.466548+0000 mgr.y (mgr.14146) 2070 : cluster [DBG] pgmap v3055: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:13 smithi169 bash[19604]: cluster 2024-02-12T16:53:12.536007+0000 osd.11 (osd.11) 268 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:53:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:13 smithi169 bash[19604]: cluster 2024-02-12T16:53:12.577685+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.d scrub ok 2024-02-12T16:53:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:13 smithi169 bash[19604]: cluster 2024-02-12T16:53:13.139514+0000 osd.5 (osd.5) 269 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:53:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:13 smithi169 bash[19604]: cluster 2024-02-12T16:53:13.219134+0000 osd.5 (osd.5) 270 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:53:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:13 smithi123 bash[19119]: cluster 2024-02-12T16:53:12.453591+0000 osd.11 (osd.11) 267 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:53:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:13 smithi123 bash[19119]: cluster 2024-02-12T16:53:12.456629+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.d scrub starts 2024-02-12T16:53:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:13 smithi123 bash[19119]: cluster 2024-02-12T16:53:12.466548+0000 mgr.y (mgr.14146) 2070 : cluster [DBG] pgmap v3055: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:13 smithi123 bash[19119]: cluster 2024-02-12T16:53:12.536007+0000 osd.11 (osd.11) 268 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:53:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:13 smithi123 bash[19119]: cluster 2024-02-12T16:53:12.577685+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.d scrub ok 2024-02-12T16:53:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:13 smithi123 bash[19119]: cluster 2024-02-12T16:53:13.139514+0000 osd.5 (osd.5) 269 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:53:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:13 smithi123 bash[19119]: cluster 2024-02-12T16:53:13.219134+0000 osd.5 (osd.5) 270 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:53:15.648 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:15.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:15 smithi121 bash[18489]: cluster 2024-02-12T16:53:14.467268+0000 mgr.y (mgr.14146) 2071 : cluster [DBG] pgmap v3056: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:15 smithi169 bash[19604]: cluster 2024-02-12T16:53:14.467268+0000 mgr.y (mgr.14146) 2071 : cluster [DBG] pgmap v3056: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:15 smithi123 bash[19119]: cluster 2024-02-12T16:53:14.467268+0000 mgr.y (mgr.14146) 2071 : cluster [DBG] pgmap v3056: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:17.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:17 smithi121 bash[18489]: cluster 2024-02-12T16:53:16.467943+0000 mgr.y (mgr.14146) 2072 : cluster [DBG] pgmap v3057: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:17.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:17 smithi169 bash[19604]: cluster 2024-02-12T16:53:16.467943+0000 mgr.y (mgr.14146) 2072 : cluster [DBG] pgmap v3057: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:17 smithi123 bash[19119]: cluster 2024-02-12T16:53:16.467943+0000 mgr.y (mgr.14146) 2072 : cluster [DBG] pgmap v3057: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:18 smithi169 bash[19604]: cluster 2024-02-12T16:53:17.216766+0000 osd.4 (osd.4) 209 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:53:18.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:18 smithi169 bash[19604]: cluster 2024-02-12T16:53:17.267223+0000 osd.4 (osd.4) 210 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:53:18.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:18 smithi123 bash[19119]: cluster 2024-02-12T16:53:17.216766+0000 osd.4 (osd.4) 209 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:53:18.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:18 smithi123 bash[19119]: cluster 2024-02-12T16:53:17.267223+0000 osd.4 (osd.4) 210 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:53:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:18 smithi121 bash[18489]: cluster 2024-02-12T16:53:17.216766+0000 osd.4 (osd.4) 209 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:53:18.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:18 smithi121 bash[18489]: cluster 2024-02-12T16:53:17.267223+0000 osd.4 (osd.4) 210 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:53:19.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:19 smithi169 bash[19604]: cluster 2024-02-12T16:53:18.468643+0000 mgr.y (mgr.14146) 2073 : cluster [DBG] pgmap v3058: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:19.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:19 smithi123 bash[19119]: cluster 2024-02-12T16:53:18.468643+0000 mgr.y (mgr.14146) 2073 : cluster [DBG] pgmap v3058: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:19.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:19 smithi121 bash[18489]: cluster 2024-02-12T16:53:18.468643+0000 mgr.y (mgr.14146) 2073 : cluster [DBG] pgmap v3058: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:19.899 INFO:teuthology.orchestra.run.smithi121.stdout:3964254814337 2024-02-12T16:53:19.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-02-12T16:53:21.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:21 smithi169 bash[19604]: cluster 2024-02-12T16:53:20.469466+0000 mgr.y (mgr.14146) 2074 : cluster [DBG] pgmap v3059: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:21.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:21 smithi123 bash[19119]: cluster 2024-02-12T16:53:20.469466+0000 mgr.y (mgr.14146) 2074 : cluster [DBG] pgmap v3059: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:21.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:21 smithi121 bash[18489]: cluster 2024-02-12T16:53:20.469466+0000 mgr.y (mgr.14146) 2074 : cluster [DBG] pgmap v3059: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:23 smithi169 bash[19604]: cluster 2024-02-12T16:53:22.470206+0000 mgr.y (mgr.14146) 2075 : cluster [DBG] pgmap v3060: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:23 smithi169 bash[19604]: cluster 2024-02-12T16:53:22.794261+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:23.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:23 smithi169 bash[19604]: cluster 2024-02-12T16:53:22.794391+0000 osd.3 (osd.3) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:23 smithi123 bash[19119]: cluster 2024-02-12T16:53:22.470206+0000 mgr.y (mgr.14146) 2075 : cluster [DBG] pgmap v3060: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:23 smithi123 bash[19119]: cluster 2024-02-12T16:53:22.794261+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:23.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:23 smithi123 bash[19119]: cluster 2024-02-12T16:53:22.794391+0000 osd.3 (osd.3) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:23 smithi121 bash[18489]: cluster 2024-02-12T16:53:22.470206+0000 mgr.y (mgr.14146) 2075 : cluster [DBG] pgmap v3060: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:23 smithi121 bash[18489]: cluster 2024-02-12T16:53:22.794261+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:23 smithi121 bash[18489]: cluster 2024-02-12T16:53:22.794391+0000 osd.3 (osd.3) 159 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:24.693 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:24 smithi123 bash[19119]: cluster 2024-02-12T16:53:23.545660+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e1119: 12 total, 12 up, 12 in 2024-02-12T16:53:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:24 smithi121 bash[18489]: cluster 2024-02-12T16:53:23.545660+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e1119: 12 total, 12 up, 12 in 2024-02-12T16:53:25.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:24 smithi169 bash[19604]: cluster 2024-02-12T16:53:23.545660+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e1119: 12 total, 12 up, 12 in 2024-02-12T16:53:25.856 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:25 smithi121 bash[18489]: cluster 2024-02-12T16:53:24.470918+0000 mgr.y (mgr.14146) 2076 : cluster [DBG] pgmap v3062: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:25.856 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:25 smithi121 bash[18489]: cluster 2024-02-12T16:53:24.863159+0000 osd.9 (osd.9) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:25.856 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:25 smithi121 bash[18489]: cluster 2024-02-12T16:53:24.863243+0000 osd.9 (osd.9) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:25 smithi123 bash[19119]: cluster 2024-02-12T16:53:24.470918+0000 mgr.y (mgr.14146) 2076 : cluster [DBG] pgmap v3062: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:25 smithi123 bash[19119]: cluster 2024-02-12T16:53:24.863159+0000 osd.9 (osd.9) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:25.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:25 smithi123 bash[19119]: cluster 2024-02-12T16:53:24.863243+0000 osd.9 (osd.9) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:25 smithi169 bash[19604]: cluster 2024-02-12T16:53:24.470918+0000 mgr.y (mgr.14146) 2076 : cluster [DBG] pgmap v3062: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:25 smithi169 bash[19604]: cluster 2024-02-12T16:53:24.863159+0000 osd.9 (osd.9) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:25 smithi169 bash[19604]: cluster 2024-02-12T16:53:24.863243+0000 osd.9 (osd.9) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:26.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:26 smithi123 bash[19119]: cluster 2024-02-12T16:53:25.563350+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e1120: 12 total, 12 up, 12 in 2024-02-12T16:53:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:26 smithi121 bash[18489]: cluster 2024-02-12T16:53:25.563350+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e1120: 12 total, 12 up, 12 in 2024-02-12T16:53:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:26 smithi169 bash[19604]: cluster 2024-02-12T16:53:25.563350+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e1120: 12 total, 12 up, 12 in 2024-02-12T16:53:27.866 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:27 smithi121 bash[18489]: cluster 2024-02-12T16:53:26.471570+0000 mgr.y (mgr.14146) 2077 : cluster [DBG] pgmap v3064: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:27.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:27 smithi123 bash[19119]: cluster 2024-02-12T16:53:26.471570+0000 mgr.y (mgr.14146) 2077 : cluster [DBG] pgmap v3064: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:27 smithi169 bash[19604]: cluster 2024-02-12T16:53:26.471570+0000 mgr.y (mgr.14146) 2077 : cluster [DBG] pgmap v3064: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:28.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:28 smithi121 bash[18489]: cluster 2024-02-12T16:53:28.079134+0000 osd.11 (osd.11) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:28.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:28 smithi121 bash[18489]: cluster 2024-02-12T16:53:28.079255+0000 osd.11 (osd.11) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:28.870 INFO:teuthology.orchestra.run.smithi121.stdout:279172874931 2024-02-12T16:53:28.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-02-12T16:53:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:28 smithi123 bash[19119]: cluster 2024-02-12T16:53:28.079134+0000 osd.11 (osd.11) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:28.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:28 smithi123 bash[19119]: cluster 2024-02-12T16:53:28.079255+0000 osd.11 (osd.11) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:29.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:28 smithi169 bash[19604]: cluster 2024-02-12T16:53:28.079134+0000 osd.11 (osd.11) 269 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:29.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:28 smithi169 bash[19604]: cluster 2024-02-12T16:53:28.079255+0000 osd.11 (osd.11) 270 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:29 smithi123 bash[19119]: cluster 2024-02-12T16:53:28.255450+0000 osd.5 (osd.5) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:29 smithi123 bash[19119]: cluster 2024-02-12T16:53:28.255541+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:29 smithi123 bash[19119]: cluster 2024-02-12T16:53:28.472300+0000 mgr.y (mgr.14146) 2078 : cluster [DBG] pgmap v3065: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:29.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:29 smithi123 bash[19119]: cluster 2024-02-12T16:53:28.575269+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e1121: 12 total, 12 up, 12 in 2024-02-12T16:53:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:29 smithi121 bash[18489]: cluster 2024-02-12T16:53:28.255450+0000 osd.5 (osd.5) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:29 smithi121 bash[18489]: cluster 2024-02-12T16:53:28.255541+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:29 smithi121 bash[18489]: cluster 2024-02-12T16:53:28.472300+0000 mgr.y (mgr.14146) 2078 : cluster [DBG] pgmap v3065: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:29.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:29 smithi121 bash[18489]: cluster 2024-02-12T16:53:28.575269+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e1121: 12 total, 12 up, 12 in 2024-02-12T16:53:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:29 smithi169 bash[19604]: cluster 2024-02-12T16:53:28.255450+0000 osd.5 (osd.5) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:29 smithi169 bash[19604]: cluster 2024-02-12T16:53:28.255541+0000 osd.5 (osd.5) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:29 smithi169 bash[19604]: cluster 2024-02-12T16:53:28.472300+0000 mgr.y (mgr.14146) 2078 : cluster [DBG] pgmap v3065: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:30.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:29 smithi169 bash[19604]: cluster 2024-02-12T16:53:28.575269+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e1121: 12 total, 12 up, 12 in 2024-02-12T16:53:31.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:31 smithi123 bash[19119]: cluster 2024-02-12T16:53:30.473150+0000 mgr.y (mgr.14146) 2079 : cluster [DBG] pgmap v3067: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:31 smithi121 bash[18489]: cluster 2024-02-12T16:53:30.473150+0000 mgr.y (mgr.14146) 2079 : cluster [DBG] pgmap v3067: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:32.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:31 smithi169 bash[19604]: cluster 2024-02-12T16:53:30.473150+0000 mgr.y (mgr.14146) 2079 : cluster [DBG] pgmap v3067: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:33.669 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:33 smithi123 bash[19119]: cluster 2024-02-12T16:53:32.473937+0000 mgr.y (mgr.14146) 2080 : cluster [DBG] pgmap v3068: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:33 smithi121 bash[18489]: cluster 2024-02-12T16:53:32.473937+0000 mgr.y (mgr.14146) 2080 : cluster [DBG] pgmap v3068: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:33 smithi169 bash[19604]: cluster 2024-02-12T16:53:32.473937+0000 mgr.y (mgr.14146) 2080 : cluster [DBG] pgmap v3068: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:35 smithi123 bash[19119]: cluster 2024-02-12T16:53:34.442094+0000 osd.11 (osd.11) 271 : cluster [DBG] 1.b scrub starts 2024-02-12T16:53:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:35 smithi123 bash[19119]: cluster 2024-02-12T16:53:34.474691+0000 mgr.y (mgr.14146) 2081 : cluster [DBG] pgmap v3069: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:35.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:35 smithi123 bash[19119]: cluster 2024-02-12T16:53:34.604397+0000 osd.11 (osd.11) 272 : cluster [DBG] 1.b scrub ok 2024-02-12T16:53:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:35 smithi121 bash[18489]: cluster 2024-02-12T16:53:34.442094+0000 osd.11 (osd.11) 271 : cluster [DBG] 1.b scrub starts 2024-02-12T16:53:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:35 smithi121 bash[18489]: cluster 2024-02-12T16:53:34.474691+0000 mgr.y (mgr.14146) 2081 : cluster [DBG] pgmap v3069: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:35 smithi121 bash[18489]: cluster 2024-02-12T16:53:34.604397+0000 osd.11 (osd.11) 272 : cluster [DBG] 1.b scrub ok 2024-02-12T16:53:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:35 smithi169 bash[19604]: cluster 2024-02-12T16:53:34.442094+0000 osd.11 (osd.11) 271 : cluster [DBG] 1.b scrub starts 2024-02-12T16:53:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:35 smithi169 bash[19604]: cluster 2024-02-12T16:53:34.474691+0000 mgr.y (mgr.14146) 2081 : cluster [DBG] pgmap v3069: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:35 smithi169 bash[19604]: cluster 2024-02-12T16:53:34.604397+0000 osd.11 (osd.11) 272 : cluster [DBG] 1.b scrub ok 2024-02-12T16:53:36.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:36 smithi123 bash[19119]: cluster 2024-02-12T16:53:35.382456+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:36 smithi123 bash[19119]: cluster 2024-02-12T16:53:35.382547+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:36.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:36 smithi123 bash[19119]: cluster 2024-02-12T16:53:35.593311+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e1122: 12 total, 12 up, 12 in 2024-02-12T16:53:36.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:36 smithi121 bash[18489]: cluster 2024-02-12T16:53:35.382456+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:36.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:36 smithi121 bash[18489]: cluster 2024-02-12T16:53:35.382547+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:36.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:36 smithi121 bash[18489]: cluster 2024-02-12T16:53:35.593311+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e1122: 12 total, 12 up, 12 in 2024-02-12T16:53:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:36 smithi169 bash[19604]: cluster 2024-02-12T16:53:35.382456+0000 osd.2 (osd.2) 167 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:36 smithi169 bash[19604]: cluster 2024-02-12T16:53:35.382547+0000 osd.2 (osd.2) 168 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:37.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:36 smithi169 bash[19604]: cluster 2024-02-12T16:53:35.593311+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e1122: 12 total, 12 up, 12 in 2024-02-12T16:53:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:317827580588 2024-02-12T16:53:37.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-02-12T16:53:37.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:37 smithi123 bash[19119]: cluster 2024-02-12T16:53:36.475157+0000 mgr.y (mgr.14146) 2082 : cluster [DBG] pgmap v3071: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:37 smithi121 bash[18489]: cluster 2024-02-12T16:53:36.475157+0000 mgr.y (mgr.14146) 2082 : cluster [DBG] pgmap v3071: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:38.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:37 smithi169 bash[19604]: cluster 2024-02-12T16:53:36.475157+0000 mgr.y (mgr.14146) 2082 : cluster [DBG] pgmap v3071: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:38.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:38 smithi123 bash[19119]: cluster 2024-02-12T16:53:37.208388+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:38.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:38 smithi123 bash[19119]: cluster 2024-02-12T16:53:37.208476+0000 osd.6 (osd.6) 223 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:38.892 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:38 smithi123 bash[19119]: cluster 2024-02-12T16:53:37.405465+0000 osd.7 (osd.7) 79 : cluster [DBG] 1.2 deep-scrub starts 2024-02-12T16:53:38.893 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:38 smithi123 bash[19119]: cluster 2024-02-12T16:53:37.471994+0000 osd.7 (osd.7) 80 : cluster [DBG] 1.2 deep-scrub ok 2024-02-12T16:53:38.893 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:38 smithi123 bash[19119]: cluster 2024-02-12T16:53:37.615699+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e1123: 12 total, 12 up, 12 in 2024-02-12T16:53:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:38 smithi121 bash[18489]: cluster 2024-02-12T16:53:37.208388+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:38 smithi121 bash[18489]: cluster 2024-02-12T16:53:37.208476+0000 osd.6 (osd.6) 223 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:38 smithi121 bash[18489]: cluster 2024-02-12T16:53:37.405465+0000 osd.7 (osd.7) 79 : cluster [DBG] 1.2 deep-scrub starts 2024-02-12T16:53:38.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:38 smithi121 bash[18489]: cluster 2024-02-12T16:53:37.471994+0000 osd.7 (osd.7) 80 : cluster [DBG] 1.2 deep-scrub ok 2024-02-12T16:53:38.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:38 smithi121 bash[18489]: cluster 2024-02-12T16:53:37.615699+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e1123: 12 total, 12 up, 12 in 2024-02-12T16:53:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:38 smithi169 bash[19604]: cluster 2024-02-12T16:53:37.208388+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:38 smithi169 bash[19604]: cluster 2024-02-12T16:53:37.208476+0000 osd.6 (osd.6) 223 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:38 smithi169 bash[19604]: cluster 2024-02-12T16:53:37.405465+0000 osd.7 (osd.7) 79 : cluster [DBG] 1.2 deep-scrub starts 2024-02-12T16:53:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:38 smithi169 bash[19604]: cluster 2024-02-12T16:53:37.471994+0000 osd.7 (osd.7) 80 : cluster [DBG] 1.2 deep-scrub ok 2024-02-12T16:53:39.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:38 smithi169 bash[19604]: cluster 2024-02-12T16:53:37.615699+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e1123: 12 total, 12 up, 12 in 2024-02-12T16:53:39.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:39 smithi123 bash[19119]: cluster 2024-02-12T16:53:38.475742+0000 mgr.y (mgr.14146) 2083 : cluster [DBG] pgmap v3073: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:39 smithi121 bash[18489]: cluster 2024-02-12T16:53:38.475742+0000 mgr.y (mgr.14146) 2083 : cluster [DBG] pgmap v3073: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:40.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:39 smithi169 bash[19604]: cluster 2024-02-12T16:53:38.475742+0000 mgr.y (mgr.14146) 2083 : cluster [DBG] pgmap v3073: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:40 smithi123 bash[19119]: cluster 2024-02-12T16:53:39.291651+0000 osd.8 (osd.8) 187 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:53:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:40 smithi123 bash[19119]: cluster 2024-02-12T16:53:39.425340+0000 osd.8 (osd.8) 188 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:53:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:40 smithi123 bash[19119]: cluster 2024-02-12T16:53:39.544999+0000 osd.1 (osd.1) 220 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:53:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:40 smithi123 bash[19119]: cluster 2024-02-12T16:53:39.756294+0000 osd.1 (osd.1) 221 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:53:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:40 smithi123 bash[19119]: cluster 2024-02-12T16:53:39.857669+0000 osd.4 (osd.4) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:40 smithi123 bash[19119]: cluster 2024-02-12T16:53:39.857757+0000 osd.4 (osd.4) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:40 smithi123 bash[19119]: audit 2024-02-12T16:53:40.553552+0000 mon.c (mon.1) 128 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:53:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:40 smithi121 bash[18489]: cluster 2024-02-12T16:53:39.291651+0000 osd.8 (osd.8) 187 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:53:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:40 smithi121 bash[18489]: cluster 2024-02-12T16:53:39.425340+0000 osd.8 (osd.8) 188 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:53:40.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:40 smithi121 bash[18489]: cluster 2024-02-12T16:53:39.544999+0000 osd.1 (osd.1) 220 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:53:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:40 smithi121 bash[18489]: cluster 2024-02-12T16:53:39.756294+0000 osd.1 (osd.1) 221 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:53:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:40 smithi121 bash[18489]: cluster 2024-02-12T16:53:39.857669+0000 osd.4 (osd.4) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:40 smithi121 bash[18489]: cluster 2024-02-12T16:53:39.857757+0000 osd.4 (osd.4) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:40 smithi121 bash[18489]: audit 2024-02-12T16:53:40.553552+0000 mon.c (mon.1) 128 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:53:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:40 smithi169 bash[19604]: cluster 2024-02-12T16:53:39.291651+0000 osd.8 (osd.8) 187 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:53:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:40 smithi169 bash[19604]: cluster 2024-02-12T16:53:39.425340+0000 osd.8 (osd.8) 188 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:53:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:40 smithi169 bash[19604]: cluster 2024-02-12T16:53:39.544999+0000 osd.1 (osd.1) 220 : cluster [DBG] 1.13 scrub starts 2024-02-12T16:53:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:40 smithi169 bash[19604]: cluster 2024-02-12T16:53:39.756294+0000 osd.1 (osd.1) 221 : cluster [DBG] 1.13 scrub ok 2024-02-12T16:53:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:40 smithi169 bash[19604]: cluster 2024-02-12T16:53:39.857669+0000 osd.4 (osd.4) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:40 smithi169 bash[19604]: cluster 2024-02-12T16:53:39.857757+0000 osd.4 (osd.4) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:40 smithi169 bash[19604]: audit 2024-02-12T16:53:40.553552+0000 mon.c (mon.1) 128 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:53:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:41 smithi121 bash[18489]: cluster 2024-02-12T16:53:40.476494+0000 mgr.y (mgr.14146) 2084 : cluster [DBG] pgmap v3074: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:41 smithi121 bash[18489]: cluster 2024-02-12T16:53:40.634797+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e1124: 12 total, 12 up, 12 in 2024-02-12T16:53:41.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:41 smithi121 bash[18489]: audit 2024-02-12T16:53:40.860625+0000 mon.a (mon.0) 3658 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:41 smithi169 bash[19604]: cluster 2024-02-12T16:53:40.476494+0000 mgr.y (mgr.14146) 2084 : cluster [DBG] pgmap v3074: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:41 smithi169 bash[19604]: cluster 2024-02-12T16:53:40.634797+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e1124: 12 total, 12 up, 12 in 2024-02-12T16:53:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:41 smithi169 bash[19604]: audit 2024-02-12T16:53:40.860625+0000 mon.a (mon.0) 3658 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:42.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:41 smithi123 bash[19119]: cluster 2024-02-12T16:53:40.476494+0000 mgr.y (mgr.14146) 2084 : cluster [DBG] pgmap v3074: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:42.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:41 smithi123 bash[19119]: cluster 2024-02-12T16:53:40.634797+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e1124: 12 total, 12 up, 12 in 2024-02-12T16:53:42.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:41 smithi123 bash[19119]: audit 2024-02-12T16:53:40.860625+0000 mon.a (mon.0) 3658 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:42.560 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:43 smithi121 bash[18489]: cluster 2024-02-12T16:53:42.120795+0000 osd.10 (osd.10) 304 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:53:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:43 smithi121 bash[18489]: cluster 2024-02-12T16:53:42.253263+0000 osd.10 (osd.10) 305 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:53:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:43 smithi121 bash[18489]: cluster 2024-02-12T16:53:42.477212+0000 mgr.y (mgr.14146) 2085 : cluster [DBG] pgmap v3076: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:43 smithi169 bash[19604]: cluster 2024-02-12T16:53:42.120795+0000 osd.10 (osd.10) 304 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:53:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:43 smithi169 bash[19604]: cluster 2024-02-12T16:53:42.253263+0000 osd.10 (osd.10) 305 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:53:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:43 smithi169 bash[19604]: cluster 2024-02-12T16:53:42.477212+0000 mgr.y (mgr.14146) 2085 : cluster [DBG] pgmap v3076: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:44.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:43 smithi123 bash[19119]: cluster 2024-02-12T16:53:42.120795+0000 osd.10 (osd.10) 304 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:53:44.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:43 smithi123 bash[19119]: cluster 2024-02-12T16:53:42.253263+0000 osd.10 (osd.10) 305 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:53:44.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:43 smithi123 bash[19119]: cluster 2024-02-12T16:53:42.477212+0000 mgr.y (mgr.14146) 2085 : cluster [DBG] pgmap v3076: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:45 smithi121 bash[18489]: cluster 2024-02-12T16:53:44.477915+0000 mgr.y (mgr.14146) 2086 : cluster [DBG] pgmap v3077: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:45 smithi169 bash[19604]: cluster 2024-02-12T16:53:44.477915+0000 mgr.y (mgr.14146) 2086 : cluster [DBG] pgmap v3077: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:46.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:45 smithi123 bash[19119]: cluster 2024-02-12T16:53:44.477915+0000 mgr.y (mgr.14146) 2086 : cluster [DBG] pgmap v3077: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:47.212 INFO:teuthology.orchestra.run.smithi121.stdout:356482286245 2024-02-12T16:53:47.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-02-12T16:53:47.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:47 smithi169 bash[19604]: cluster 2024-02-12T16:53:46.478683+0000 mgr.y (mgr.14146) 2087 : cluster [DBG] pgmap v3078: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:47 smithi123 bash[19119]: cluster 2024-02-12T16:53:46.478683+0000 mgr.y (mgr.14146) 2087 : cluster [DBG] pgmap v3078: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:47 smithi121 bash[18489]: cluster 2024-02-12T16:53:46.478683+0000 mgr.y (mgr.14146) 2087 : cluster [DBG] pgmap v3078: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:48 smithi123 bash[19119]: cluster 2024-02-12T16:53:47.412846+0000 osd.11 (osd.11) 273 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:53:48.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:48 smithi123 bash[19119]: cluster 2024-02-12T16:53:47.436898+0000 osd.11 (osd.11) 274 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:53:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:48 smithi121 bash[18489]: cluster 2024-02-12T16:53:47.412846+0000 osd.11 (osd.11) 273 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:53:48.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:48 smithi121 bash[18489]: cluster 2024-02-12T16:53:47.436898+0000 osd.11 (osd.11) 274 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:53:48.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:48 smithi169 bash[19604]: cluster 2024-02-12T16:53:47.412846+0000 osd.11 (osd.11) 273 : cluster [DBG] 1.4 scrub starts 2024-02-12T16:53:48.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:48 smithi169 bash[19604]: cluster 2024-02-12T16:53:47.436898+0000 osd.11 (osd.11) 274 : cluster [DBG] 1.4 scrub ok 2024-02-12T16:53:49.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:49 smithi123 bash[19119]: cluster 2024-02-12T16:53:48.479477+0000 mgr.y (mgr.14146) 2088 : cluster [DBG] pgmap v3079: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:49.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:49 smithi121 bash[18489]: cluster 2024-02-12T16:53:48.479477+0000 mgr.y (mgr.14146) 2088 : cluster [DBG] pgmap v3079: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:49.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:49 smithi169 bash[19604]: cluster 2024-02-12T16:53:48.479477+0000 mgr.y (mgr.14146) 2088 : cluster [DBG] pgmap v3079: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:50.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:50 smithi123 bash[19119]: cluster 2024-02-12T16:53:49.568977+0000 osd.1 (osd.1) 222 : cluster [DBG] 1.1d deep-scrub starts 2024-02-12T16:53:50.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:50 smithi123 bash[19119]: cluster 2024-02-12T16:53:49.771530+0000 osd.1 (osd.1) 223 : cluster [DBG] 1.1d deep-scrub ok 2024-02-12T16:53:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:50 smithi121 bash[18489]: cluster 2024-02-12T16:53:49.568977+0000 osd.1 (osd.1) 222 : cluster [DBG] 1.1d deep-scrub starts 2024-02-12T16:53:50.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:50 smithi121 bash[18489]: cluster 2024-02-12T16:53:49.771530+0000 osd.1 (osd.1) 223 : cluster [DBG] 1.1d deep-scrub ok 2024-02-12T16:53:50.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:50 smithi169 bash[19604]: cluster 2024-02-12T16:53:49.568977+0000 osd.1 (osd.1) 222 : cluster [DBG] 1.1d deep-scrub starts 2024-02-12T16:53:50.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:50 smithi169 bash[19604]: cluster 2024-02-12T16:53:49.771530+0000 osd.1 (osd.1) 223 : cluster [DBG] 1.1d deep-scrub ok 2024-02-12T16:53:51.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:51 smithi123 bash[19119]: cluster 2024-02-12T16:53:50.184593+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:51.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:51 smithi123 bash[19119]: cluster 2024-02-12T16:53:50.184690+0000 osd.1 (osd.1) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:51 smithi123 bash[19119]: cluster 2024-02-12T16:53:50.373221+0000 mon.a (mon.0) 3659 : cluster [DBG] osdmap e1125: 12 total, 12 up, 12 in 2024-02-12T16:53:51.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:51 smithi123 bash[19119]: cluster 2024-02-12T16:53:50.480102+0000 mgr.y (mgr.14146) 2089 : cluster [DBG] pgmap v3081: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:51 smithi121 bash[18489]: cluster 2024-02-12T16:53:50.184593+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:51 smithi121 bash[18489]: cluster 2024-02-12T16:53:50.184690+0000 osd.1 (osd.1) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:51 smithi121 bash[18489]: cluster 2024-02-12T16:53:50.373221+0000 mon.a (mon.0) 3659 : cluster [DBG] osdmap e1125: 12 total, 12 up, 12 in 2024-02-12T16:53:51.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:51 smithi121 bash[18489]: cluster 2024-02-12T16:53:50.480102+0000 mgr.y (mgr.14146) 2089 : cluster [DBG] pgmap v3081: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:51.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:51 smithi169 bash[19604]: cluster 2024-02-12T16:53:50.184593+0000 osd.1 (osd.1) 224 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:51 smithi169 bash[19604]: cluster 2024-02-12T16:53:50.184690+0000 osd.1 (osd.1) 225 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:51 smithi169 bash[19604]: cluster 2024-02-12T16:53:50.373221+0000 mon.a (mon.0) 3659 : cluster [DBG] osdmap e1125: 12 total, 12 up, 12 in 2024-02-12T16:53:51.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:51 smithi169 bash[19604]: cluster 2024-02-12T16:53:50.480102+0000 mgr.y (mgr.14146) 2089 : cluster [DBG] pgmap v3081: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:52 smithi123 bash[19119]: cluster 2024-02-12T16:53:51.393611+0000 osd.0 (osd.0) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:52 smithi123 bash[19119]: cluster 2024-02-12T16:53:51.393664+0000 osd.0 (osd.0) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:52 smithi123 bash[19119]: audit 2024-02-12T16:53:51.563988+0000 mon.a (mon.0) 3660 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:52 smithi123 bash[19119]: audit 2024-02-12T16:53:51.870238+0000 mon.c (mon.1) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:53:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:52 smithi123 bash[19119]: audit 2024-02-12T16:53:51.871945+0000 mon.c (mon.1) 130 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:53:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:52 smithi123 bash[19119]: audit 2024-02-12T16:53:51.879991+0000 mon.a (mon.0) 3661 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:52 smithi123 bash[19119]: cluster 2024-02-12T16:53:52.384875+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e1126: 12 total, 12 up, 12 in 2024-02-12T16:53:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:52 smithi121 bash[18489]: cluster 2024-02-12T16:53:51.393611+0000 osd.0 (osd.0) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:52 smithi121 bash[18489]: cluster 2024-02-12T16:53:51.393664+0000 osd.0 (osd.0) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:52 smithi121 bash[18489]: audit 2024-02-12T16:53:51.563988+0000 mon.a (mon.0) 3660 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:52 smithi121 bash[18489]: audit 2024-02-12T16:53:51.870238+0000 mon.c (mon.1) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:53:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:52 smithi121 bash[18489]: audit 2024-02-12T16:53:51.871945+0000 mon.c (mon.1) 130 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:53:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:52 smithi121 bash[18489]: audit 2024-02-12T16:53:51.879991+0000 mon.a (mon.0) 3661 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:52.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:52 smithi121 bash[18489]: cluster 2024-02-12T16:53:52.384875+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e1126: 12 total, 12 up, 12 in 2024-02-12T16:53:53.014 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:53:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:52 smithi169 bash[19604]: cluster 2024-02-12T16:53:51.393611+0000 osd.0 (osd.0) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:52 smithi169 bash[19604]: cluster 2024-02-12T16:53:51.393664+0000 osd.0 (osd.0) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:52 smithi169 bash[19604]: audit 2024-02-12T16:53:51.563988+0000 mon.a (mon.0) 3660 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:52 smithi169 bash[19604]: audit 2024-02-12T16:53:51.870238+0000 mon.c (mon.1) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:53:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:52 smithi169 bash[19604]: audit 2024-02-12T16:53:51.871945+0000 mon.c (mon.1) 130 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:53:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:52 smithi169 bash[19604]: audit 2024-02-12T16:53:51.879991+0000 mon.a (mon.0) 3661 : audit [INF] from='mgr.14146 ' entity='mgr.y' 2024-02-12T16:53:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:52 smithi169 bash[19604]: cluster 2024-02-12T16:53:52.384875+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e1126: 12 total, 12 up, 12 in 2024-02-12T16:53:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:53 smithi123 bash[19119]: cluster 2024-02-12T16:53:52.260410+0000 osd.10 (osd.10) 306 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:53:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:53 smithi123 bash[19119]: cluster 2024-02-12T16:53:52.365390+0000 osd.10 (osd.10) 307 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:53:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:53 smithi123 bash[19119]: cluster 2024-02-12T16:53:52.480704+0000 mgr.y (mgr.14146) 2090 : cluster [DBG] pgmap v3083: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:53 smithi123 bash[19119]: cluster 2024-02-12T16:53:52.610058+0000 osd.1 (osd.1) 226 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:53:53.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:53 smithi123 bash[19119]: cluster 2024-02-12T16:53:52.765251+0000 osd.1 (osd.1) 227 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:53:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:53 smithi121 bash[18489]: cluster 2024-02-12T16:53:52.260410+0000 osd.10 (osd.10) 306 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:53:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:53 smithi121 bash[18489]: cluster 2024-02-12T16:53:52.365390+0000 osd.10 (osd.10) 307 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:53:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:53 smithi121 bash[18489]: cluster 2024-02-12T16:53:52.480704+0000 mgr.y (mgr.14146) 2090 : cluster [DBG] pgmap v3083: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:53 smithi121 bash[18489]: cluster 2024-02-12T16:53:52.610058+0000 osd.1 (osd.1) 226 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:53:53.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:53 smithi121 bash[18489]: cluster 2024-02-12T16:53:52.765251+0000 osd.1 (osd.1) 227 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:53:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:53 smithi169 bash[19604]: cluster 2024-02-12T16:53:52.260410+0000 osd.10 (osd.10) 306 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:53:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:53 smithi169 bash[19604]: cluster 2024-02-12T16:53:52.365390+0000 osd.10 (osd.10) 307 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:53:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:53 smithi169 bash[19604]: cluster 2024-02-12T16:53:52.480704+0000 mgr.y (mgr.14146) 2090 : cluster [DBG] pgmap v3083: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:53 smithi169 bash[19604]: cluster 2024-02-12T16:53:52.610058+0000 osd.1 (osd.1) 226 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:53:54.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:53 smithi169 bash[19604]: cluster 2024-02-12T16:53:52.765251+0000 osd.1 (osd.1) 227 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:53:54.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:54 smithi121 bash[18489]: cluster 2024-02-12T16:53:53.375959+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:54.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:54 smithi121 bash[18489]: cluster 2024-02-12T16:53:53.376060+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:54.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:54 smithi121 bash[18489]: cluster 2024-02-12T16:53:53.571355+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e1127: 12 total, 12 up, 12 in 2024-02-12T16:53:54.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:54 smithi123 bash[19119]: cluster 2024-02-12T16:53:53.375959+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:54.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:54 smithi123 bash[19119]: cluster 2024-02-12T16:53:53.376060+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:54.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:54 smithi123 bash[19119]: cluster 2024-02-12T16:53:53.571355+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e1127: 12 total, 12 up, 12 in 2024-02-12T16:53:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:54 smithi169 bash[19604]: cluster 2024-02-12T16:53:53.375959+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:53:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:54 smithi169 bash[19604]: cluster 2024-02-12T16:53:53.376060+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:53:55.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:54 smithi169 bash[19604]: cluster 2024-02-12T16:53:53.571355+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e1127: 12 total, 12 up, 12 in 2024-02-12T16:53:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:55 smithi123 bash[19119]: cluster 2024-02-12T16:53:54.481471+0000 mgr.y (mgr.14146) 2091 : cluster [DBG] pgmap v3085: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:55 smithi123 bash[19119]: cluster 2024-02-12T16:53:54.557608+0000 osd.7 (osd.7) 83 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:53:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:55 smithi123 bash[19119]: cluster 2024-02-12T16:53:54.659162+0000 osd.7 (osd.7) 84 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:53:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:55 smithi121 bash[18489]: cluster 2024-02-12T16:53:54.481471+0000 mgr.y (mgr.14146) 2091 : cluster [DBG] pgmap v3085: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:55 smithi121 bash[18489]: cluster 2024-02-12T16:53:54.557608+0000 osd.7 (osd.7) 83 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:53:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:55 smithi121 bash[18489]: cluster 2024-02-12T16:53:54.659162+0000 osd.7 (osd.7) 84 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:53:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:55 smithi169 bash[19604]: cluster 2024-02-12T16:53:54.481471+0000 mgr.y (mgr.14146) 2091 : cluster [DBG] pgmap v3085: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:55 smithi169 bash[19604]: cluster 2024-02-12T16:53:54.557608+0000 osd.7 (osd.7) 83 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:53:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:55 smithi169 bash[19604]: cluster 2024-02-12T16:53:54.659162+0000 osd.7 (osd.7) 84 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:53:56.735 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:56 smithi121 bash[18489]: cluster 2024-02-12T16:53:55.344741+0000 osd.10 (osd.10) 308 : cluster [DBG] 1.19 deep-scrub starts 2024-02-12T16:53:56.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:56 smithi121 bash[18489]: cluster 2024-02-12T16:53:55.445351+0000 osd.10 (osd.10) 309 : cluster [DBG] 1.19 deep-scrub ok 2024-02-12T16:53:56.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:56 smithi121 bash[18489]: cluster 2024-02-12T16:53:56.282274+0000 osd.5 (osd.5) 273 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:53:56.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:56 smithi121 bash[18489]: cluster 2024-02-12T16:53:56.395334+0000 osd.5 (osd.5) 274 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:53:56.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:56 smithi123 bash[19119]: cluster 2024-02-12T16:53:55.344741+0000 osd.10 (osd.10) 308 : cluster [DBG] 1.19 deep-scrub starts 2024-02-12T16:53:56.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:56 smithi123 bash[19119]: cluster 2024-02-12T16:53:55.445351+0000 osd.10 (osd.10) 309 : cluster [DBG] 1.19 deep-scrub ok 2024-02-12T16:53:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:56 smithi123 bash[19119]: cluster 2024-02-12T16:53:56.282274+0000 osd.5 (osd.5) 273 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:53:56.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:56 smithi123 bash[19119]: cluster 2024-02-12T16:53:56.395334+0000 osd.5 (osd.5) 274 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:53:57.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:56 smithi169 bash[19604]: cluster 2024-02-12T16:53:55.344741+0000 osd.10 (osd.10) 308 : cluster [DBG] 1.19 deep-scrub starts 2024-02-12T16:53:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:56 smithi169 bash[19604]: cluster 2024-02-12T16:53:55.445351+0000 osd.10 (osd.10) 309 : cluster [DBG] 1.19 deep-scrub ok 2024-02-12T16:53:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:56 smithi169 bash[19604]: cluster 2024-02-12T16:53:56.282274+0000 osd.5 (osd.5) 273 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:53:57.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:56 smithi169 bash[19604]: cluster 2024-02-12T16:53:56.395334+0000 osd.5 (osd.5) 274 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:53:57.429 INFO:teuthology.orchestra.run.smithi121.stdout:399431959199 2024-02-12T16:53:57.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-12T16:53:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:57 smithi123 bash[19119]: cluster 2024-02-12T16:53:56.038007+0000 osd.6 (osd.6) 224 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:53:57.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:57 smithi123 bash[19119]: cluster 2024-02-12T16:53:56.183993+0000 osd.6 (osd.6) 225 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:53:57.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:57 smithi123 bash[19119]: cluster 2024-02-12T16:53:56.482223+0000 mgr.y (mgr.14146) 2092 : cluster [DBG] pgmap v3086: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:57 smithi121 bash[18489]: cluster 2024-02-12T16:53:56.038007+0000 osd.6 (osd.6) 224 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:53:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:57 smithi121 bash[18489]: cluster 2024-02-12T16:53:56.183993+0000 osd.6 (osd.6) 225 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:53:57.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:57 smithi121 bash[18489]: cluster 2024-02-12T16:53:56.482223+0000 mgr.y (mgr.14146) 2092 : cluster [DBG] pgmap v3086: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:57 smithi169 bash[19604]: cluster 2024-02-12T16:53:56.038007+0000 osd.6 (osd.6) 224 : cluster [DBG] 1.f deep-scrub starts 2024-02-12T16:53:58.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:57 smithi169 bash[19604]: cluster 2024-02-12T16:53:56.183993+0000 osd.6 (osd.6) 225 : cluster [DBG] 1.f deep-scrub ok 2024-02-12T16:53:58.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:57 smithi169 bash[19604]: cluster 2024-02-12T16:53:56.482223+0000 mgr.y (mgr.14146) 2092 : cluster [DBG] pgmap v3086: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:59.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:59 smithi123 bash[19119]: cluster 2024-02-12T16:53:58.428038+0000 osd.9 (osd.9) 189 : cluster [DBG] 1.a scrub starts 2024-02-12T16:53:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:59 smithi123 bash[19119]: cluster 2024-02-12T16:53:58.482907+0000 mgr.y (mgr.14146) 2093 : cluster [DBG] pgmap v3087: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:59.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:53:59 smithi123 bash[19119]: cluster 2024-02-12T16:53:58.522325+0000 osd.9 (osd.9) 190 : cluster [DBG] 1.a scrub ok 2024-02-12T16:53:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:59 smithi121 bash[18489]: cluster 2024-02-12T16:53:58.428038+0000 osd.9 (osd.9) 189 : cluster [DBG] 1.a scrub starts 2024-02-12T16:53:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:59 smithi121 bash[18489]: cluster 2024-02-12T16:53:58.482907+0000 mgr.y (mgr.14146) 2093 : cluster [DBG] pgmap v3087: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:53:59.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:53:59 smithi121 bash[18489]: cluster 2024-02-12T16:53:58.522325+0000 osd.9 (osd.9) 190 : cluster [DBG] 1.a scrub ok 2024-02-12T16:54:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:59 smithi169 bash[19604]: cluster 2024-02-12T16:53:58.428038+0000 osd.9 (osd.9) 189 : cluster [DBG] 1.a scrub starts 2024-02-12T16:54:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:59 smithi169 bash[19604]: cluster 2024-02-12T16:53:58.482907+0000 mgr.y (mgr.14146) 2093 : cluster [DBG] pgmap v3087: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:00.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:53:59 smithi169 bash[19604]: cluster 2024-02-12T16:53:58.522325+0000 osd.9 (osd.9) 190 : cluster [DBG] 1.a scrub ok 2024-02-12T16:54:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:00 smithi123 bash[19119]: cluster 2024-02-12T16:53:59.196489+0000 osd.0 (osd.0) 193 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:54:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:00 smithi123 bash[19119]: cluster 2024-02-12T16:53:59.314668+0000 osd.0 (osd.0) 194 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:54:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:00 smithi121 bash[18489]: cluster 2024-02-12T16:53:59.196489+0000 osd.0 (osd.0) 193 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:54:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:00 smithi121 bash[18489]: cluster 2024-02-12T16:53:59.314668+0000 osd.0 (osd.0) 194 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:54:01.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:00 smithi169 bash[19604]: cluster 2024-02-12T16:53:59.196489+0000 osd.0 (osd.0) 193 : cluster [DBG] 1.11 scrub starts 2024-02-12T16:54:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:00 smithi169 bash[19604]: cluster 2024-02-12T16:53:59.314668+0000 osd.0 (osd.0) 194 : cluster [DBG] 1.11 scrub ok 2024-02-12T16:54:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:01 smithi123 bash[19119]: cluster 2024-02-12T16:54:00.483672+0000 mgr.y (mgr.14146) 2094 : cluster [DBG] pgmap v3088: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:01 smithi121 bash[18489]: cluster 2024-02-12T16:54:00.483672+0000 mgr.y (mgr.14146) 2094 : cluster [DBG] pgmap v3088: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:02.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:01 smithi169 bash[19604]: cluster 2024-02-12T16:54:00.483672+0000 mgr.y (mgr.14146) 2094 : cluster [DBG] pgmap v3088: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:02.221 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:03.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:03 smithi121 bash[18489]: cluster 2024-02-12T16:54:02.484314+0000 mgr.y (mgr.14146) 2095 : cluster [DBG] pgmap v3089: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:03 smithi123 bash[19119]: cluster 2024-02-12T16:54:02.484314+0000 mgr.y (mgr.14146) 2095 : cluster [DBG] pgmap v3089: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:04.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:03 smithi169 bash[19604]: cluster 2024-02-12T16:54:02.484314+0000 mgr.y (mgr.14146) 2095 : cluster [DBG] pgmap v3089: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:04 smithi123 bash[19119]: cluster 2024-02-12T16:54:02.852665+0000 osd.2 (osd.2) 169 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:54:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:04 smithi123 bash[19119]: cluster 2024-02-12T16:54:02.957622+0000 osd.2 (osd.2) 170 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:54:04.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:04 smithi121 bash[18489]: cluster 2024-02-12T16:54:02.852665+0000 osd.2 (osd.2) 169 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:54:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:04 smithi121 bash[18489]: cluster 2024-02-12T16:54:02.957622+0000 osd.2 (osd.2) 170 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:54:05.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:04 smithi169 bash[19604]: cluster 2024-02-12T16:54:02.852665+0000 osd.2 (osd.2) 169 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:54:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:04 smithi169 bash[19604]: cluster 2024-02-12T16:54:02.957622+0000 osd.2 (osd.2) 170 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:54:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:05 smithi123 bash[19119]: cluster 2024-02-12T16:54:04.403503+0000 osd.10 (osd.10) 310 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:54:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:05 smithi123 bash[19119]: cluster 2024-02-12T16:54:04.448614+0000 osd.10 (osd.10) 311 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:54:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:05 smithi123 bash[19119]: cluster 2024-02-12T16:54:04.485024+0000 mgr.y (mgr.14146) 2096 : cluster [DBG] pgmap v3090: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:05 smithi123 bash[19119]: cluster 2024-02-12T16:54:04.551942+0000 osd.7 (osd.7) 85 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:54:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:05 smithi123 bash[19119]: cluster 2024-02-12T16:54:04.583519+0000 osd.7 (osd.7) 86 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:54:05.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:05 smithi123 bash[19119]: cluster 2024-02-12T16:54:04.686183+0000 osd.3 (osd.3) 160 : cluster [DBG] 1.c scrub starts 2024-02-12T16:54:05.906 INFO:teuthology.orchestra.run.smithi121.stdout:34359739133 2024-02-12T16:54:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:05 smithi121 bash[18489]: cluster 2024-02-12T16:54:04.403503+0000 osd.10 (osd.10) 310 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:54:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:05 smithi121 bash[18489]: cluster 2024-02-12T16:54:04.448614+0000 osd.10 (osd.10) 311 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:54:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:05 smithi121 bash[18489]: cluster 2024-02-12T16:54:04.485024+0000 mgr.y (mgr.14146) 2096 : cluster [DBG] pgmap v3090: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:05 smithi121 bash[18489]: cluster 2024-02-12T16:54:04.551942+0000 osd.7 (osd.7) 85 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:54:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:05 smithi121 bash[18489]: cluster 2024-02-12T16:54:04.583519+0000 osd.7 (osd.7) 86 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:54:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:05 smithi121 bash[18489]: cluster 2024-02-12T16:54:04.686183+0000 osd.3 (osd.3) 160 : cluster [DBG] 1.c scrub starts 2024-02-12T16:54:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:05 smithi169 bash[19604]: cluster 2024-02-12T16:54:04.403503+0000 osd.10 (osd.10) 310 : cluster [DBG] 1.6 scrub starts 2024-02-12T16:54:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:05 smithi169 bash[19604]: cluster 2024-02-12T16:54:04.448614+0000 osd.10 (osd.10) 311 : cluster [DBG] 1.6 scrub ok 2024-02-12T16:54:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:05 smithi169 bash[19604]: cluster 2024-02-12T16:54:04.485024+0000 mgr.y (mgr.14146) 2096 : cluster [DBG] pgmap v3090: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:05 smithi169 bash[19604]: cluster 2024-02-12T16:54:04.551942+0000 osd.7 (osd.7) 85 : cluster [DBG] 1.0 scrub starts 2024-02-12T16:54:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:05 smithi169 bash[19604]: cluster 2024-02-12T16:54:04.583519+0000 osd.7 (osd.7) 86 : cluster [DBG] 1.0 scrub ok 2024-02-12T16:54:06.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:05 smithi169 bash[19604]: cluster 2024-02-12T16:54:04.686183+0000 osd.3 (osd.3) 160 : cluster [DBG] 1.c scrub starts 2024-02-12T16:54:06.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359739110 got 34359739133 for osd.0 2024-02-12T16:54:06.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-12T16:54:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:06 smithi123 bash[19119]: cluster 2024-02-12T16:54:04.834643+0000 osd.3 (osd.3) 161 : cluster [DBG] 1.c scrub ok 2024-02-12T16:54:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:06 smithi123 bash[19119]: audit 2024-02-12T16:54:05.900360+0000 mon.a (mon.0) 3664 : audit [DBG] from='client.? v1:172.21.15.121:0/14926459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:54:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:06 smithi123 bash[19119]: cluster 2024-02-12T16:54:06.553536+0000 osd.4 (osd.4) 213 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:54:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:06 smithi121 bash[18489]: cluster 2024-02-12T16:54:04.834643+0000 osd.3 (osd.3) 161 : cluster [DBG] 1.c scrub ok 2024-02-12T16:54:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:06 smithi121 bash[18489]: audit 2024-02-12T16:54:05.900360+0000 mon.a (mon.0) 3664 : audit [DBG] from='client.? v1:172.21.15.121:0/14926459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:54:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:06 smithi121 bash[18489]: cluster 2024-02-12T16:54:06.553536+0000 osd.4 (osd.4) 213 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:54:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:06 smithi169 bash[19604]: cluster 2024-02-12T16:54:04.834643+0000 osd.3 (osd.3) 161 : cluster [DBG] 1.c scrub ok 2024-02-12T16:54:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:06 smithi169 bash[19604]: audit 2024-02-12T16:54:05.900360+0000 mon.a (mon.0) 3664 : audit [DBG] from='client.? v1:172.21.15.121:0/14926459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-12T16:54:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:06 smithi169 bash[19604]: cluster 2024-02-12T16:54:06.553536+0000 osd.4 (osd.4) 213 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:54:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:07 smithi123 bash[19119]: cluster 2024-02-12T16:54:06.435735+0000 osd.10 (osd.10) 312 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:54:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:07 smithi123 bash[19119]: cluster 2024-02-12T16:54:06.485727+0000 mgr.y (mgr.14146) 2097 : cluster [DBG] pgmap v3091: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:07 smithi123 bash[19119]: cluster 2024-02-12T16:54:06.503421+0000 osd.10 (osd.10) 313 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:54:07.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:07 smithi123 bash[19119]: cluster 2024-02-12T16:54:06.648668+0000 osd.4 (osd.4) 214 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:54:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:07 smithi121 bash[18489]: cluster 2024-02-12T16:54:06.435735+0000 osd.10 (osd.10) 312 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:54:07.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:07 smithi121 bash[18489]: cluster 2024-02-12T16:54:06.485727+0000 mgr.y (mgr.14146) 2097 : cluster [DBG] pgmap v3091: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:07.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:07 smithi121 bash[18489]: cluster 2024-02-12T16:54:06.503421+0000 osd.10 (osd.10) 313 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:54:07.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:07 smithi121 bash[18489]: cluster 2024-02-12T16:54:06.648668+0000 osd.4 (osd.4) 214 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:54:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:07 smithi169 bash[19604]: cluster 2024-02-12T16:54:06.435735+0000 osd.10 (osd.10) 312 : cluster [DBG] 1.1 scrub starts 2024-02-12T16:54:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:07 smithi169 bash[19604]: cluster 2024-02-12T16:54:06.485727+0000 mgr.y (mgr.14146) 2097 : cluster [DBG] pgmap v3091: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:07 smithi169 bash[19604]: cluster 2024-02-12T16:54:06.503421+0000 osd.10 (osd.10) 313 : cluster [DBG] 1.1 scrub ok 2024-02-12T16:54:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:07 smithi169 bash[19604]: cluster 2024-02-12T16:54:06.648668+0000 osd.4 (osd.4) 214 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:54:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:09 smithi121 bash[18489]: cluster 2024-02-12T16:54:08.486353+0000 mgr.y (mgr.14146) 2098 : cluster [DBG] pgmap v3092: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:09.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:09 smithi121 bash[18489]: cluster 2024-02-12T16:54:08.773539+0000 osd.8 (osd.8) 189 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:09.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:09 smithi121 bash[18489]: cluster 2024-02-12T16:54:08.773636+0000 osd.8 (osd.8) 190 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:10.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:09 smithi169 bash[19604]: cluster 2024-02-12T16:54:08.486353+0000 mgr.y (mgr.14146) 2098 : cluster [DBG] pgmap v3092: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:09 smithi169 bash[19604]: cluster 2024-02-12T16:54:08.773539+0000 osd.8 (osd.8) 189 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:09 smithi169 bash[19604]: cluster 2024-02-12T16:54:08.773636+0000 osd.8 (osd.8) 190 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:10.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:09 smithi123 bash[19119]: cluster 2024-02-12T16:54:08.486353+0000 mgr.y (mgr.14146) 2098 : cluster [DBG] pgmap v3092: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:10.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:09 smithi123 bash[19119]: cluster 2024-02-12T16:54:08.773539+0000 osd.8 (osd.8) 189 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:10.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:09 smithi123 bash[19119]: cluster 2024-02-12T16:54:08.773636+0000 osd.8 (osd.8) 190 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:10 smithi121 bash[18489]: cluster 2024-02-12T16:54:09.651096+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e1128: 12 total, 12 up, 12 in 2024-02-12T16:54:11.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:10 smithi169 bash[19604]: cluster 2024-02-12T16:54:09.651096+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e1128: 12 total, 12 up, 12 in 2024-02-12T16:54:11.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:10 smithi123 bash[19119]: cluster 2024-02-12T16:54:09.651096+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e1128: 12 total, 12 up, 12 in 2024-02-12T16:54:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:12.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:11 smithi169 bash[19604]: cluster 2024-02-12T16:54:10.486998+0000 mgr.y (mgr.14146) 2099 : cluster [DBG] pgmap v3094: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:12.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:11 smithi123 bash[19119]: cluster 2024-02-12T16:54:10.486998+0000 mgr.y (mgr.14146) 2099 : cluster [DBG] pgmap v3094: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:12.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:11 smithi121 bash[18489]: cluster 2024-02-12T16:54:10.486998+0000 mgr.y (mgr.14146) 2099 : cluster [DBG] pgmap v3094: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:13 smithi169 bash[19604]: cluster 2024-02-12T16:54:12.487512+0000 mgr.y (mgr.14146) 2100 : cluster [DBG] pgmap v3095: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:13 smithi169 bash[19604]: cluster 2024-02-12T16:54:12.495259+0000 osd.7 (osd.7) 87 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:54:14.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:13 smithi169 bash[19604]: cluster 2024-02-12T16:54:12.599269+0000 osd.7 (osd.7) 88 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:54:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:13 smithi123 bash[19119]: cluster 2024-02-12T16:54:12.487512+0000 mgr.y (mgr.14146) 2100 : cluster [DBG] pgmap v3095: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:13 smithi123 bash[19119]: cluster 2024-02-12T16:54:12.495259+0000 osd.7 (osd.7) 87 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:54:14.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:13 smithi123 bash[19119]: cluster 2024-02-12T16:54:12.599269+0000 osd.7 (osd.7) 88 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:54:14.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:13 smithi121 bash[18489]: cluster 2024-02-12T16:54:12.487512+0000 mgr.y (mgr.14146) 2100 : cluster [DBG] pgmap v3095: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:14.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:13 smithi121 bash[18489]: cluster 2024-02-12T16:54:12.495259+0000 osd.7 (osd.7) 87 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:54:14.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:13 smithi121 bash[18489]: cluster 2024-02-12T16:54:12.599269+0000 osd.7 (osd.7) 88 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:54:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:14 smithi169 bash[19604]: cluster 2024-02-12T16:54:14.492070+0000 osd.4 (osd.4) 215 : cluster [DBG] 1.e scrub starts 2024-02-12T16:54:15.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:14 smithi169 bash[19604]: cluster 2024-02-12T16:54:14.584494+0000 osd.4 (osd.4) 216 : cluster [DBG] 1.e scrub ok 2024-02-12T16:54:15.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:14 smithi123 bash[19119]: cluster 2024-02-12T16:54:14.492070+0000 osd.4 (osd.4) 215 : cluster [DBG] 1.e scrub starts 2024-02-12T16:54:15.140 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:14 smithi123 bash[19119]: cluster 2024-02-12T16:54:14.584494+0000 osd.4 (osd.4) 216 : cluster [DBG] 1.e scrub ok 2024-02-12T16:54:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:14 smithi121 bash[18489]: cluster 2024-02-12T16:54:14.492070+0000 osd.4 (osd.4) 215 : cluster [DBG] 1.e scrub starts 2024-02-12T16:54:15.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:14 smithi121 bash[18489]: cluster 2024-02-12T16:54:14.584494+0000 osd.4 (osd.4) 216 : cluster [DBG] 1.e scrub ok 2024-02-12T16:54:15.214 INFO:teuthology.orchestra.run.smithi121.stdout:55834575602 2024-02-12T16:54:15.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:15 smithi121 bash[18489]: cluster 2024-02-12T16:54:14.487974+0000 mgr.y (mgr.14146) 2101 : cluster [DBG] pgmap v3096: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:15.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:15 smithi121 bash[18489]: audit 2024-02-12T16:54:15.208076+0000 mon.a (mon.0) 3666 : audit [DBG] from='client.? v1:172.21.15.121:0/2380847046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:54:15.993 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575580 got 55834575602 for osd.1 2024-02-12T16:54:15.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-12T16:54:16.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:15 smithi169 bash[19604]: cluster 2024-02-12T16:54:14.487974+0000 mgr.y (mgr.14146) 2101 : cluster [DBG] pgmap v3096: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:16.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:15 smithi169 bash[19604]: audit 2024-02-12T16:54:15.208076+0000 mon.a (mon.0) 3666 : audit [DBG] from='client.? v1:172.21.15.121:0/2380847046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:54:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:15 smithi123 bash[19119]: cluster 2024-02-12T16:54:14.487974+0000 mgr.y (mgr.14146) 2101 : cluster [DBG] pgmap v3096: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:15 smithi123 bash[19119]: audit 2024-02-12T16:54:15.208076+0000 mon.a (mon.0) 3666 : audit [DBG] from='client.? v1:172.21.15.121:0/2380847046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-12T16:54:17.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:17 smithi169 bash[19604]: cluster 2024-02-12T16:54:16.488665+0000 mgr.y (mgr.14146) 2102 : cluster [DBG] pgmap v3097: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:17.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:17 smithi123 bash[19119]: cluster 2024-02-12T16:54:16.488665+0000 mgr.y (mgr.14146) 2102 : cluster [DBG] pgmap v3097: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:17.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:17 smithi121 bash[18489]: cluster 2024-02-12T16:54:16.488665+0000 mgr.y (mgr.14146) 2102 : cluster [DBG] pgmap v3097: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:19.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:19 smithi123 bash[19119]: cluster 2024-02-12T16:54:18.489342+0000 mgr.y (mgr.14146) 2103 : cluster [DBG] pgmap v3098: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:19.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:19 smithi121 bash[18489]: cluster 2024-02-12T16:54:18.489342+0000 mgr.y (mgr.14146) 2103 : cluster [DBG] pgmap v3098: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:19 smithi169 bash[19604]: cluster 2024-02-12T16:54:18.489342+0000 mgr.y (mgr.14146) 2103 : cluster [DBG] pgmap v3098: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:20.791 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:20 smithi123 bash[19119]: cluster 2024-02-12T16:54:19.109192+0000 osd.5 (osd.5) 275 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:54:20.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:20 smithi123 bash[19119]: cluster 2024-02-12T16:54:19.181045+0000 osd.5 (osd.5) 276 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:54:20.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:20 smithi121 bash[18489]: cluster 2024-02-12T16:54:19.109192+0000 osd.5 (osd.5) 275 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:54:20.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:20 smithi121 bash[18489]: cluster 2024-02-12T16:54:19.181045+0000 osd.5 (osd.5) 276 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:54:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:20 smithi169 bash[19604]: cluster 2024-02-12T16:54:19.109192+0000 osd.5 (osd.5) 275 : cluster [DBG] 1.8 scrub starts 2024-02-12T16:54:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:20 smithi169 bash[19604]: cluster 2024-02-12T16:54:19.181045+0000 osd.5 (osd.5) 276 : cluster [DBG] 1.8 scrub ok 2024-02-12T16:54:21.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:21 smithi121 bash[18489]: cluster 2024-02-12T16:54:20.490065+0000 mgr.y (mgr.14146) 2104 : cluster [DBG] pgmap v3099: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:21.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:21 smithi123 bash[19119]: cluster 2024-02-12T16:54:20.490065+0000 mgr.y (mgr.14146) 2104 : cluster [DBG] pgmap v3099: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:22.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:21 smithi169 bash[19604]: cluster 2024-02-12T16:54:20.490065+0000 mgr.y (mgr.14146) 2104 : cluster [DBG] pgmap v3099: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:23.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:23 smithi123 bash[19119]: cluster 2024-02-12T16:54:22.490772+0000 mgr.y (mgr.14146) 2105 : cluster [DBG] pgmap v3100: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:23.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:23 smithi121 bash[18489]: cluster 2024-02-12T16:54:22.490772+0000 mgr.y (mgr.14146) 2105 : cluster [DBG] pgmap v3100: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:24.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:23 smithi169 bash[19604]: cluster 2024-02-12T16:54:22.490772+0000 mgr.y (mgr.14146) 2105 : cluster [DBG] pgmap v3100: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:24.160 INFO:teuthology.orchestra.run.smithi121.stdout:90194313966 2024-02-12T16:54:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:24 smithi123 bash[19119]: cluster 2024-02-12T16:54:23.414354+0000 osd.10 (osd.10) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:24.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:24 smithi123 bash[19119]: cluster 2024-02-12T16:54:23.414447+0000 osd.10 (osd.10) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:24 smithi123 bash[19119]: cluster 2024-02-12T16:54:23.567147+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e1129: 12 total, 12 up, 12 in 2024-02-12T16:54:24.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:24 smithi123 bash[19119]: audit 2024-02-12T16:54:24.153881+0000 mon.a (mon.0) 3668 : audit [DBG] from='client.? v1:172.21.15.121:0/334281064' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:54:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:24 smithi121 bash[18489]: cluster 2024-02-12T16:54:23.414354+0000 osd.10 (osd.10) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:24 smithi121 bash[18489]: cluster 2024-02-12T16:54:23.414447+0000 osd.10 (osd.10) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:24.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:24 smithi121 bash[18489]: cluster 2024-02-12T16:54:23.567147+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e1129: 12 total, 12 up, 12 in 2024-02-12T16:54:24.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:24 smithi121 bash[18489]: audit 2024-02-12T16:54:24.153881+0000 mon.a (mon.0) 3668 : audit [DBG] from='client.? v1:172.21.15.121:0/334281064' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:54:24.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 90194313944 got 90194313966 for osd.2 2024-02-12T16:54:24.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-12T16:54:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:24 smithi169 bash[19604]: cluster 2024-02-12T16:54:23.414354+0000 osd.10 (osd.10) 314 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:24 smithi169 bash[19604]: cluster 2024-02-12T16:54:23.414447+0000 osd.10 (osd.10) 315 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:24 smithi169 bash[19604]: cluster 2024-02-12T16:54:23.567147+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e1129: 12 total, 12 up, 12 in 2024-02-12T16:54:25.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:24 smithi169 bash[19604]: audit 2024-02-12T16:54:24.153881+0000 mon.a (mon.0) 3668 : audit [DBG] from='client.? v1:172.21.15.121:0/334281064' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-12T16:54:25.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:25 smithi123 bash[19119]: cluster 2024-02-12T16:54:24.491498+0000 mgr.y (mgr.14146) 2106 : cluster [DBG] pgmap v3102: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:25.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:25 smithi121 bash[18489]: cluster 2024-02-12T16:54:24.491498+0000 mgr.y (mgr.14146) 2106 : cluster [DBG] pgmap v3102: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:26.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:25 smithi169 bash[19604]: cluster 2024-02-12T16:54:24.491498+0000 mgr.y (mgr.14146) 2106 : cluster [DBG] pgmap v3102: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:26.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:26 smithi123 bash[19119]: cluster 2024-02-12T16:54:25.126466+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:54:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:26 smithi123 bash[19119]: cluster 2024-02-12T16:54:25.252251+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:54:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:26 smithi123 bash[19119]: cluster 2024-02-12T16:54:25.644500+0000 osd.3 (osd.3) 162 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:54:26.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:26 smithi123 bash[19119]: cluster 2024-02-12T16:54:25.787820+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:54:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:26 smithi121 bash[18489]: cluster 2024-02-12T16:54:25.126466+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:54:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:26 smithi121 bash[18489]: cluster 2024-02-12T16:54:25.252251+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:54:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:26 smithi121 bash[18489]: cluster 2024-02-12T16:54:25.644500+0000 osd.3 (osd.3) 162 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:54:26.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:26 smithi121 bash[18489]: cluster 2024-02-12T16:54:25.787820+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:54:27.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:26 smithi169 bash[19604]: cluster 2024-02-12T16:54:25.126466+0000 osd.6 (osd.6) 226 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:54:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:26 smithi169 bash[19604]: cluster 2024-02-12T16:54:25.252251+0000 osd.6 (osd.6) 227 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:54:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:26 smithi169 bash[19604]: cluster 2024-02-12T16:54:25.644500+0000 osd.3 (osd.3) 162 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:54:27.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:26 smithi169 bash[19604]: cluster 2024-02-12T16:54:25.787820+0000 osd.3 (osd.3) 163 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:54:27.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:27 smithi123 bash[19119]: cluster 2024-02-12T16:54:26.492166+0000 mgr.y (mgr.14146) 2107 : cluster [DBG] pgmap v3103: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:27.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:27 smithi121 bash[18489]: cluster 2024-02-12T16:54:26.492166+0000 mgr.y (mgr.14146) 2107 : cluster [DBG] pgmap v3103: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:28.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:27 smithi169 bash[19604]: cluster 2024-02-12T16:54:26.492166+0000 mgr.y (mgr.14146) 2107 : cluster [DBG] pgmap v3103: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:29.765 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:29.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:29 smithi123 bash[19119]: cluster 2024-02-12T16:54:28.492845+0000 mgr.y (mgr.14146) 2108 : cluster [DBG] pgmap v3104: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:29.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:29 smithi121 bash[18489]: cluster 2024-02-12T16:54:28.492845+0000 mgr.y (mgr.14146) 2108 : cluster [DBG] pgmap v3104: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:30.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:29 smithi169 bash[19604]: cluster 2024-02-12T16:54:28.492845+0000 mgr.y (mgr.14146) 2108 : cluster [DBG] pgmap v3104: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:30.860 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: cluster 2024-02-12T16:54:29.515423+0000 osd.4 (osd.4) 217 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:54:30.860 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: cluster 2024-02-12T16:54:29.565731+0000 osd.4 (osd.4) 218 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:54:30.860 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.602403+0000 mon.a (mon.0) 3669 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:54:30.860 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.602729+0000 mon.a (mon.0) 3670 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:54:30.860 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.603008+0000 mon.a (mon.0) 3671 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:54:30.860 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.660761+0000 mon.a (mon.0) 3672 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:54:30.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.966370+0000 mon.a (mon.0) 3673 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:54:30.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.968449+0000 mon.a (mon.0) 3674 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:54:30.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.970334+0000 mon.a (mon.0) 3675 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:54:30.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:30 smithi121 bash[18489]: audit 2024-02-12T16:54:29.981731+0000 mon.a (mon.0) 3676 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:54:30.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: cluster 2024-02-12T16:54:29.515423+0000 osd.4 (osd.4) 217 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: cluster 2024-02-12T16:54:29.565731+0000 osd.4 (osd.4) 218 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.602403+0000 mon.a (mon.0) 3669 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.602729+0000 mon.a (mon.0) 3670 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.603008+0000 mon.a (mon.0) 3671 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.660761+0000 mon.a (mon.0) 3672 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.966370+0000 mon.a (mon.0) 3673 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.968449+0000 mon.a (mon.0) 3674 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.970334+0000 mon.a (mon.0) 3675 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:54:30.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:30 smithi123 bash[19119]: audit 2024-02-12T16:54:29.981731+0000 mon.a (mon.0) 3676 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: cluster 2024-02-12T16:54:29.515423+0000 osd.4 (osd.4) 217 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: cluster 2024-02-12T16:54:29.565731+0000 osd.4 (osd.4) 218 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.602403+0000 mon.a (mon.0) 3669 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.602729+0000 mon.a (mon.0) 3670 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.603008+0000 mon.a (mon.0) 3671 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.660761+0000 mon.a (mon.0) 3672 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.966370+0000 mon.a (mon.0) 3673 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.968449+0000 mon.a (mon.0) 3674 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.970334+0000 mon.a (mon.0) 3675 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:54:31.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:30 smithi169 bash[19604]: audit 2024-02-12T16:54:29.981731+0000 mon.a (mon.0) 3676 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:54:31.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:31 smithi123 bash[19119]: cluster 2024-02-12T16:54:30.493581+0000 mgr.y (mgr.14146) 2109 : cluster [DBG] pgmap v3105: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:31.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:31 smithi123 bash[19119]: cluster 2024-02-12T16:54:30.577302+0000 osd.7 (osd.7) 89 : cluster [DBG] 1.d scrub starts 2024-02-12T16:54:31.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:31 smithi123 bash[19119]: cluster 2024-02-12T16:54:30.691450+0000 osd.7 (osd.7) 90 : cluster [DBG] 1.d scrub ok 2024-02-12T16:54:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:31 smithi121 bash[18489]: cluster 2024-02-12T16:54:30.493581+0000 mgr.y (mgr.14146) 2109 : cluster [DBG] pgmap v3105: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:31 smithi121 bash[18489]: cluster 2024-02-12T16:54:30.577302+0000 osd.7 (osd.7) 89 : cluster [DBG] 1.d scrub starts 2024-02-12T16:54:31.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:31 smithi121 bash[18489]: cluster 2024-02-12T16:54:30.691450+0000 osd.7 (osd.7) 90 : cluster [DBG] 1.d scrub ok 2024-02-12T16:54:32.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:31 smithi169 bash[19604]: cluster 2024-02-12T16:54:30.493581+0000 mgr.y (mgr.14146) 2109 : cluster [DBG] pgmap v3105: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:31 smithi169 bash[19604]: cluster 2024-02-12T16:54:30.577302+0000 osd.7 (osd.7) 89 : cluster [DBG] 1.d scrub starts 2024-02-12T16:54:32.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:31 smithi169 bash[19604]: cluster 2024-02-12T16:54:30.691450+0000 osd.7 (osd.7) 90 : cluster [DBG] 1.d scrub ok 2024-02-12T16:54:33.426 INFO:teuthology.orchestra.run.smithi121.stdout:2241972928892 2024-02-12T16:54:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:33 smithi123 bash[19119]: cluster 2024-02-12T16:54:32.494188+0000 mgr.y (mgr.14146) 2110 : cluster [DBG] pgmap v3106: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:33 smithi123 bash[19119]: audit 2024-02-12T16:54:33.420665+0000 mon.a (mon.0) 3677 : audit [DBG] from='client.? v1:172.21.15.121:0/1365083972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:54:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:33 smithi121 bash[18489]: cluster 2024-02-12T16:54:32.494188+0000 mgr.y (mgr.14146) 2110 : cluster [DBG] pgmap v3106: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:33 smithi121 bash[18489]: audit 2024-02-12T16:54:33.420665+0000 mon.a (mon.0) 3677 : audit [DBG] from='client.? v1:172.21.15.121:0/1365083972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:54:34.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:33 smithi169 bash[19604]: cluster 2024-02-12T16:54:32.494188+0000 mgr.y (mgr.14146) 2110 : cluster [DBG] pgmap v3106: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:34.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:33 smithi169 bash[19604]: audit 2024-02-12T16:54:33.420665+0000 mon.a (mon.0) 3677 : audit [DBG] from='client.? v1:172.21.15.121:0/1365083972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-12T16:54:34.167 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2241972928870 got 2241972928892 for osd.3 2024-02-12T16:54:34.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-12T16:54:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:35 smithi123 bash[19119]: cluster 2024-02-12T16:54:34.494905+0000 mgr.y (mgr.14146) 2111 : cluster [DBG] pgmap v3107: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:35 smithi123 bash[19119]: cluster 2024-02-12T16:54:35.207847+0000 osd.8 (osd.8) 191 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:54:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:35 smithi121 bash[18489]: cluster 2024-02-12T16:54:34.494905+0000 mgr.y (mgr.14146) 2111 : cluster [DBG] pgmap v3107: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:35.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:35 smithi121 bash[18489]: cluster 2024-02-12T16:54:35.207847+0000 osd.8 (osd.8) 191 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:54:36.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:35 smithi169 bash[19604]: cluster 2024-02-12T16:54:34.494905+0000 mgr.y (mgr.14146) 2111 : cluster [DBG] pgmap v3107: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:36.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:35 smithi169 bash[19604]: cluster 2024-02-12T16:54:35.207847+0000 osd.8 (osd.8) 191 : cluster [DBG] 1.1c scrub starts 2024-02-12T16:54:37.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:37 smithi169 bash[19604]: cluster 2024-02-12T16:54:35.288392+0000 osd.8 (osd.8) 192 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:54:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:37 smithi169 bash[19604]: cluster 2024-02-12T16:54:36.114806+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:37 smithi169 bash[19604]: cluster 2024-02-12T16:54:36.114923+0000 osd.3 (osd.3) 165 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:37 smithi169 bash[19604]: cluster 2024-02-12T16:54:36.495579+0000 mgr.y (mgr.14146) 2112 : cluster [DBG] pgmap v3108: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:37.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:37 smithi169 bash[19604]: cluster 2024-02-12T16:54:36.615680+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e1130: 12 total, 12 up, 12 in 2024-02-12T16:54:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:37 smithi123 bash[19119]: cluster 2024-02-12T16:54:35.288392+0000 osd.8 (osd.8) 192 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:54:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:37 smithi123 bash[19119]: cluster 2024-02-12T16:54:36.114806+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:37 smithi123 bash[19119]: cluster 2024-02-12T16:54:36.114923+0000 osd.3 (osd.3) 165 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:37.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:37 smithi123 bash[19119]: cluster 2024-02-12T16:54:36.495579+0000 mgr.y (mgr.14146) 2112 : cluster [DBG] pgmap v3108: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:37.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:37 smithi123 bash[19119]: cluster 2024-02-12T16:54:36.615680+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e1130: 12 total, 12 up, 12 in 2024-02-12T16:54:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:37 smithi121 bash[18489]: cluster 2024-02-12T16:54:35.288392+0000 osd.8 (osd.8) 192 : cluster [DBG] 1.1c scrub ok 2024-02-12T16:54:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:37 smithi121 bash[18489]: cluster 2024-02-12T16:54:36.114806+0000 osd.3 (osd.3) 164 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:37.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:37 smithi121 bash[18489]: cluster 2024-02-12T16:54:36.114923+0000 osd.3 (osd.3) 165 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:37.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:37 smithi121 bash[18489]: cluster 2024-02-12T16:54:36.495579+0000 mgr.y (mgr.14146) 2112 : cluster [DBG] pgmap v3108: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:37.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:37 smithi121 bash[18489]: cluster 2024-02-12T16:54:36.615680+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e1130: 12 total, 12 up, 12 in 2024-02-12T16:54:38.967 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:39.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:39 smithi123 bash[19119]: cluster 2024-02-12T16:54:38.496325+0000 mgr.y (mgr.14146) 2113 : cluster [DBG] pgmap v3110: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:39 smithi121 bash[18489]: cluster 2024-02-12T16:54:38.496325+0000 mgr.y (mgr.14146) 2113 : cluster [DBG] pgmap v3110: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:40.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:39 smithi169 bash[19604]: cluster 2024-02-12T16:54:38.496325+0000 mgr.y (mgr.14146) 2113 : cluster [DBG] pgmap v3110: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:41.876 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:41 smithi121 bash[18489]: cluster 2024-02-12T16:54:39.980763+0000 osd.5 (osd.5) 277 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:54:41.876 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:41 smithi121 bash[18489]: cluster 2024-02-12T16:54:40.059689+0000 osd.5 (osd.5) 278 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:54:41.876 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:41 smithi121 bash[18489]: cluster 2024-02-12T16:54:40.497145+0000 mgr.y (mgr.14146) 2114 : cluster [DBG] pgmap v3111: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:41 smithi123 bash[19119]: cluster 2024-02-12T16:54:39.980763+0000 osd.5 (osd.5) 277 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:54:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:41 smithi123 bash[19119]: cluster 2024-02-12T16:54:40.059689+0000 osd.5 (osd.5) 278 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:54:41.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:41 smithi123 bash[19119]: cluster 2024-02-12T16:54:40.497145+0000 mgr.y (mgr.14146) 2114 : cluster [DBG] pgmap v3111: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:41 smithi169 bash[19604]: cluster 2024-02-12T16:54:39.980763+0000 osd.5 (osd.5) 277 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:54:42.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:41 smithi169 bash[19604]: cluster 2024-02-12T16:54:40.059689+0000 osd.5 (osd.5) 278 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:54:42.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:41 smithi169 bash[19604]: cluster 2024-02-12T16:54:40.497145+0000 mgr.y (mgr.14146) 2114 : cluster [DBG] pgmap v3111: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:42.293 INFO:teuthology.orchestra.run.smithi121.stdout:137438954209 2024-02-12T16:54:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:42 smithi121 bash[18489]: cluster 2024-02-12T16:54:41.587283+0000 osd.11 (osd.11) 275 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:54:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:42 smithi121 bash[18489]: cluster 2024-02-12T16:54:41.657848+0000 osd.11 (osd.11) 276 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:54:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:42 smithi121 bash[18489]: cluster 2024-02-12T16:54:41.959289+0000 osd.9 (osd.9) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:42 smithi121 bash[18489]: cluster 2024-02-12T16:54:41.959378+0000 osd.9 (osd.9) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:42 smithi121 bash[18489]: audit 2024-02-12T16:54:42.287473+0000 mon.a (mon.0) 3679 : audit [DBG] from='client.? v1:172.21.15.121:0/59078440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:54:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:42 smithi123 bash[19119]: cluster 2024-02-12T16:54:41.587283+0000 osd.11 (osd.11) 275 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:54:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:42 smithi123 bash[19119]: cluster 2024-02-12T16:54:41.657848+0000 osd.11 (osd.11) 276 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:54:42.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:42 smithi123 bash[19119]: cluster 2024-02-12T16:54:41.959289+0000 osd.9 (osd.9) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:42 smithi123 bash[19119]: cluster 2024-02-12T16:54:41.959378+0000 osd.9 (osd.9) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:42.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:42 smithi123 bash[19119]: audit 2024-02-12T16:54:42.287473+0000 mon.a (mon.0) 3679 : audit [DBG] from='client.? v1:172.21.15.121:0/59078440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:54:43.039 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438954187 got 137438954209 for osd.4 2024-02-12T16:54:43.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-12T16:54:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:42 smithi169 bash[19604]: cluster 2024-02-12T16:54:41.587283+0000 osd.11 (osd.11) 275 : cluster [DBG] 1.1e scrub starts 2024-02-12T16:54:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:42 smithi169 bash[19604]: cluster 2024-02-12T16:54:41.657848+0000 osd.11 (osd.11) 276 : cluster [DBG] 1.1e scrub ok 2024-02-12T16:54:43.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:42 smithi169 bash[19604]: cluster 2024-02-12T16:54:41.959289+0000 osd.9 (osd.9) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:42 smithi169 bash[19604]: cluster 2024-02-12T16:54:41.959378+0000 osd.9 (osd.9) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:43.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:42 smithi169 bash[19604]: audit 2024-02-12T16:54:42.287473+0000 mon.a (mon.0) 3679 : audit [DBG] from='client.? v1:172.21.15.121:0/59078440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-12T16:54:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:43 smithi123 bash[19119]: cluster 2024-02-12T16:54:42.497861+0000 mgr.y (mgr.14146) 2115 : cluster [DBG] pgmap v3112: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:43.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:43 smithi123 bash[19119]: cluster 2024-02-12T16:54:42.571985+0000 mon.a (mon.0) 3680 : cluster [DBG] osdmap e1131: 12 total, 12 up, 12 in 2024-02-12T16:54:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:43 smithi121 bash[18489]: cluster 2024-02-12T16:54:42.497861+0000 mgr.y (mgr.14146) 2115 : cluster [DBG] pgmap v3112: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:43.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:43 smithi121 bash[18489]: cluster 2024-02-12T16:54:42.571985+0000 mon.a (mon.0) 3680 : cluster [DBG] osdmap e1131: 12 total, 12 up, 12 in 2024-02-12T16:54:44.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:43 smithi169 bash[19604]: cluster 2024-02-12T16:54:42.497861+0000 mgr.y (mgr.14146) 2115 : cluster [DBG] pgmap v3112: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:44.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:43 smithi169 bash[19604]: cluster 2024-02-12T16:54:42.571985+0000 mon.a (mon.0) 3680 : cluster [DBG] osdmap e1131: 12 total, 12 up, 12 in 2024-02-12T16:54:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:44 smithi123 bash[19119]: cluster 2024-02-12T16:54:43.688841+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:54:44.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:44 smithi123 bash[19119]: cluster 2024-02-12T16:54:43.757248+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:54:44.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:44 smithi121 bash[18489]: cluster 2024-02-12T16:54:43.688841+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:54:44.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:44 smithi121 bash[18489]: cluster 2024-02-12T16:54:43.757248+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:54:45.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:44 smithi169 bash[19604]: cluster 2024-02-12T16:54:43.688841+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.2 scrub starts 2024-02-12T16:54:45.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:44 smithi169 bash[19604]: cluster 2024-02-12T16:54:43.757248+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.2 scrub ok 2024-02-12T16:54:45.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:45 smithi123 bash[19119]: cluster 2024-02-12T16:54:44.498556+0000 mgr.y (mgr.14146) 2116 : cluster [DBG] pgmap v3114: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:45.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:45 smithi121 bash[18489]: cluster 2024-02-12T16:54:44.498556+0000 mgr.y (mgr.14146) 2116 : cluster [DBG] pgmap v3114: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:46.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:45 smithi169 bash[19604]: cluster 2024-02-12T16:54:44.498556+0000 mgr.y (mgr.14146) 2116 : cluster [DBG] pgmap v3114: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:46.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:46 smithi123 bash[19119]: cluster 2024-02-12T16:54:45.016608+0000 osd.5 (osd.5) 279 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:46.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:46 smithi123 bash[19119]: cluster 2024-02-12T16:54:45.016724+0000 osd.5 (osd.5) 280 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:46.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:46 smithi123 bash[19119]: cluster 2024-02-12T16:54:45.584593+0000 mon.a (mon.0) 3681 : cluster [DBG] osdmap e1132: 12 total, 12 up, 12 in 2024-02-12T16:54:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:46 smithi121 bash[18489]: cluster 2024-02-12T16:54:45.016608+0000 osd.5 (osd.5) 279 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:46 smithi121 bash[18489]: cluster 2024-02-12T16:54:45.016724+0000 osd.5 (osd.5) 280 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:46.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:46 smithi121 bash[18489]: cluster 2024-02-12T16:54:45.584593+0000 mon.a (mon.0) 3681 : cluster [DBG] osdmap e1132: 12 total, 12 up, 12 in 2024-02-12T16:54:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:46 smithi169 bash[19604]: cluster 2024-02-12T16:54:45.016608+0000 osd.5 (osd.5) 279 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:47.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:46 smithi169 bash[19604]: cluster 2024-02-12T16:54:45.016724+0000 osd.5 (osd.5) 280 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:47.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:46 smithi169 bash[19604]: cluster 2024-02-12T16:54:45.584593+0000 mon.a (mon.0) 3681 : cluster [DBG] osdmap e1132: 12 total, 12 up, 12 in 2024-02-12T16:54:47.837 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:47.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:47 smithi123 bash[19119]: cluster 2024-02-12T16:54:46.499269+0000 mgr.y (mgr.14146) 2117 : cluster [DBG] pgmap v3116: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:47.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:47 smithi121 bash[18489]: cluster 2024-02-12T16:54:46.499269+0000 mgr.y (mgr.14146) 2117 : cluster [DBG] pgmap v3116: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:48.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:47 smithi169 bash[19604]: cluster 2024-02-12T16:54:46.499269+0000 mgr.y (mgr.14146) 2117 : cluster [DBG] pgmap v3116: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:48 smithi123 bash[19119]: cluster 2024-02-12T16:54:47.392000+0000 osd.6 (osd.6) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:48 smithi123 bash[19119]: cluster 2024-02-12T16:54:47.392088+0000 osd.6 (osd.6) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:48.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:48 smithi123 bash[19119]: cluster 2024-02-12T16:54:48.256839+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e1133: 12 total, 12 up, 12 in 2024-02-12T16:54:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:48 smithi121 bash[18489]: cluster 2024-02-12T16:54:47.392000+0000 osd.6 (osd.6) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:48 smithi121 bash[18489]: cluster 2024-02-12T16:54:47.392088+0000 osd.6 (osd.6) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:48.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:48 smithi121 bash[18489]: cluster 2024-02-12T16:54:48.256839+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e1133: 12 total, 12 up, 12 in 2024-02-12T16:54:49.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:48 smithi169 bash[19604]: cluster 2024-02-12T16:54:47.392000+0000 osd.6 (osd.6) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:49.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:48 smithi169 bash[19604]: cluster 2024-02-12T16:54:47.392088+0000 osd.6 (osd.6) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:49.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:48 smithi169 bash[19604]: cluster 2024-02-12T16:54:48.256839+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e1133: 12 total, 12 up, 12 in 2024-02-12T16:54:49.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:49 smithi123 bash[19119]: cluster 2024-02-12T16:54:48.499942+0000 mgr.y (mgr.14146) 2118 : cluster [DBG] pgmap v3118: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:49.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:49 smithi121 bash[18489]: cluster 2024-02-12T16:54:48.499942+0000 mgr.y (mgr.14146) 2118 : cluster [DBG] pgmap v3118: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:50.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:49 smithi169 bash[19604]: cluster 2024-02-12T16:54:48.499942+0000 mgr.y (mgr.14146) 2118 : cluster [DBG] pgmap v3118: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:51.496 INFO:teuthology.orchestra.run.smithi121.stdout:167503725275 2024-02-12T16:54:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:51 smithi123 bash[19119]: cluster 2024-02-12T16:54:50.106997+0000 osd.4 (osd.4) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:51 smithi123 bash[19119]: cluster 2024-02-12T16:54:50.107087+0000 osd.4 (osd.4) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:51 smithi123 bash[19119]: cluster 2024-02-12T16:54:50.500598+0000 mgr.y (mgr.14146) 2119 : cluster [DBG] pgmap v3119: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:51.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:51 smithi123 bash[19119]: cluster 2024-02-12T16:54:50.611799+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e1134: 12 total, 12 up, 12 in 2024-02-12T16:54:51.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:51 smithi123 bash[19119]: audit 2024-02-12T16:54:51.489688+0000 mon.a (mon.0) 3684 : audit [DBG] from='client.? v1:172.21.15.121:0/614159042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:54:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:51 smithi121 bash[18489]: cluster 2024-02-12T16:54:50.106997+0000 osd.4 (osd.4) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:51 smithi121 bash[18489]: cluster 2024-02-12T16:54:50.107087+0000 osd.4 (osd.4) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:51 smithi121 bash[18489]: cluster 2024-02-12T16:54:50.500598+0000 mgr.y (mgr.14146) 2119 : cluster [DBG] pgmap v3119: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:51 smithi121 bash[18489]: cluster 2024-02-12T16:54:50.611799+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e1134: 12 total, 12 up, 12 in 2024-02-12T16:54:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:51 smithi121 bash[18489]: audit 2024-02-12T16:54:51.489688+0000 mon.a (mon.0) 3684 : audit [DBG] from='client.? v1:172.21.15.121:0/614159042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:54:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:51 smithi169 bash[19604]: cluster 2024-02-12T16:54:50.106997+0000 osd.4 (osd.4) 219 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:51 smithi169 bash[19604]: cluster 2024-02-12T16:54:50.107087+0000 osd.4 (osd.4) 220 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:51 smithi169 bash[19604]: cluster 2024-02-12T16:54:50.500598+0000 mgr.y (mgr.14146) 2119 : cluster [DBG] pgmap v3119: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:51 smithi169 bash[19604]: cluster 2024-02-12T16:54:50.611799+0000 mon.a (mon.0) 3683 : cluster [DBG] osdmap e1134: 12 total, 12 up, 12 in 2024-02-12T16:54:52.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:51 smithi169 bash[19604]: audit 2024-02-12T16:54:51.489688+0000 mon.a (mon.0) 3684 : audit [DBG] from='client.? v1:172.21.15.121:0/614159042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-12T16:54:52.254 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503725253 got 167503725275 for osd.5 2024-02-12T16:54:52.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-12T16:54:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:52 smithi123 bash[19119]: cluster 2024-02-12T16:54:50.721905+0000 osd.10 (osd.10) 316 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:54:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:52 smithi123 bash[19119]: cluster 2024-02-12T16:54:50.856134+0000 osd.10 (osd.10) 317 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:54:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:52 smithi123 bash[19119]: cluster 2024-02-12T16:54:52.061516+0000 osd.11 (osd.11) 277 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:52.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:52 smithi123 bash[19119]: cluster 2024-02-12T16:54:52.061604+0000 osd.11 (osd.11) 278 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:52 smithi121 bash[18489]: cluster 2024-02-12T16:54:50.721905+0000 osd.10 (osd.10) 316 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:54:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:52 smithi121 bash[18489]: cluster 2024-02-12T16:54:50.856134+0000 osd.10 (osd.10) 317 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:54:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:52 smithi121 bash[18489]: cluster 2024-02-12T16:54:52.061516+0000 osd.11 (osd.11) 277 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:52 smithi121 bash[18489]: cluster 2024-02-12T16:54:52.061604+0000 osd.11 (osd.11) 278 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:52 smithi169 bash[19604]: cluster 2024-02-12T16:54:50.721905+0000 osd.10 (osd.10) 316 : cluster [DBG] 1.1b scrub starts 2024-02-12T16:54:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:52 smithi169 bash[19604]: cluster 2024-02-12T16:54:50.856134+0000 osd.10 (osd.10) 317 : cluster [DBG] 1.1b scrub ok 2024-02-12T16:54:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:52 smithi169 bash[19604]: cluster 2024-02-12T16:54:52.061516+0000 osd.11 (osd.11) 277 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:54:53.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:52 smithi169 bash[19604]: cluster 2024-02-12T16:54:52.061604+0000 osd.11 (osd.11) 278 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:54:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:53 smithi123 bash[19119]: cluster 2024-02-12T16:54:52.501294+0000 mgr.y (mgr.14146) 2120 : cluster [DBG] pgmap v3121: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:53 smithi123 bash[19119]: cluster 2024-02-12T16:54:52.624451+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e1135: 12 total, 12 up, 12 in 2024-02-12T16:54:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:53 smithi121 bash[18489]: cluster 2024-02-12T16:54:52.501294+0000 mgr.y (mgr.14146) 2120 : cluster [DBG] pgmap v3121: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:53.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:53 smithi121 bash[18489]: cluster 2024-02-12T16:54:52.624451+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e1135: 12 total, 12 up, 12 in 2024-02-12T16:54:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:53 smithi169 bash[19604]: cluster 2024-02-12T16:54:52.501294+0000 mgr.y (mgr.14146) 2120 : cluster [DBG] pgmap v3121: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:53 smithi169 bash[19604]: cluster 2024-02-12T16:54:52.624451+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e1135: 12 total, 12 up, 12 in 2024-02-12T16:54:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:55 smithi123 bash[19119]: cluster 2024-02-12T16:54:54.501977+0000 mgr.y (mgr.14146) 2121 : cluster [DBG] pgmap v3123: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:55 smithi121 bash[18489]: cluster 2024-02-12T16:54:54.501977+0000 mgr.y (mgr.14146) 2121 : cluster [DBG] pgmap v3123: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:55 smithi169 bash[19604]: cluster 2024-02-12T16:54:54.501977+0000 mgr.y (mgr.14146) 2121 : cluster [DBG] pgmap v3123: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:57.050 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:54:57.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:57 smithi169 bash[19604]: cluster 2024-02-12T16:54:56.502652+0000 mgr.y (mgr.14146) 2122 : cluster [DBG] pgmap v3124: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:57 smithi123 bash[19119]: cluster 2024-02-12T16:54:56.502652+0000 mgr.y (mgr.14146) 2122 : cluster [DBG] pgmap v3124: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:57 smithi121 bash[18489]: cluster 2024-02-12T16:54:56.502652+0000 mgr.y (mgr.14146) 2122 : cluster [DBG] pgmap v3124: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:59.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:59 smithi123 bash[19119]: cluster 2024-02-12T16:54:58.503403+0000 mgr.y (mgr.14146) 2123 : cluster [DBG] pgmap v3125: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:59.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:54:59 smithi123 bash[19119]: cluster 2024-02-12T16:54:58.864311+0000 osd.1 (osd.1) 228 : cluster [DBG] 1.13 deep-scrub starts 2024-02-12T16:54:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:59 smithi121 bash[18489]: cluster 2024-02-12T16:54:58.503403+0000 mgr.y (mgr.14146) 2123 : cluster [DBG] pgmap v3125: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:54:59.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:54:59 smithi121 bash[18489]: cluster 2024-02-12T16:54:58.864311+0000 osd.1 (osd.1) 228 : cluster [DBG] 1.13 deep-scrub starts 2024-02-12T16:55:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:59 smithi169 bash[19604]: cluster 2024-02-12T16:54:58.503403+0000 mgr.y (mgr.14146) 2123 : cluster [DBG] pgmap v3125: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:00.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:54:59 smithi169 bash[19604]: cluster 2024-02-12T16:54:58.864311+0000 osd.1 (osd.1) 228 : cluster [DBG] 1.13 deep-scrub starts 2024-02-12T16:55:00.516 INFO:teuthology.orchestra.run.smithi121.stdout:206158430934 2024-02-12T16:55:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:00 smithi123 bash[19119]: cluster 2024-02-12T16:54:59.072519+0000 osd.1 (osd.1) 229 : cluster [DBG] 1.13 deep-scrub ok 2024-02-12T16:55:00.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:00 smithi123 bash[19119]: cluster 2024-02-12T16:54:59.825228+0000 osd.1 (osd.1) 230 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:55:00.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:00 smithi123 bash[19119]: audit 2024-02-12T16:55:00.510578+0000 mon.a (mon.0) 3686 : audit [DBG] from='client.? v1:172.21.15.121:0/977166562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:55:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:00 smithi121 bash[18489]: cluster 2024-02-12T16:54:59.072519+0000 osd.1 (osd.1) 229 : cluster [DBG] 1.13 deep-scrub ok 2024-02-12T16:55:00.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:00 smithi121 bash[18489]: cluster 2024-02-12T16:54:59.825228+0000 osd.1 (osd.1) 230 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:55:00.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:00 smithi121 bash[18489]: audit 2024-02-12T16:55:00.510578+0000 mon.a (mon.0) 3686 : audit [DBG] from='client.? v1:172.21.15.121:0/977166562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:55:01.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:00 smithi169 bash[19604]: cluster 2024-02-12T16:54:59.072519+0000 osd.1 (osd.1) 229 : cluster [DBG] 1.13 deep-scrub ok 2024-02-12T16:55:01.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:00 smithi169 bash[19604]: cluster 2024-02-12T16:54:59.825228+0000 osd.1 (osd.1) 230 : cluster [DBG] 1.16 scrub starts 2024-02-12T16:55:01.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:00 smithi169 bash[19604]: audit 2024-02-12T16:55:00.510578+0000 mon.a (mon.0) 3686 : audit [DBG] from='client.? v1:172.21.15.121:0/977166562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-12T16:55:01.315 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430912 got 206158430934 for osd.6 2024-02-12T16:55:01.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-12T16:55:01.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:01 smithi121 bash[18489]: cluster 2024-02-12T16:54:59.980462+0000 osd.1 (osd.1) 231 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:55:01.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:01 smithi121 bash[18489]: cluster 2024-02-12T16:55:00.462086+0000 osd.11 (osd.11) 279 : cluster [DBG] 1.b deep-scrub starts 2024-02-12T16:55:01.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:01 smithi121 bash[18489]: cluster 2024-02-12T16:55:00.504130+0000 mgr.y (mgr.14146) 2124 : cluster [DBG] pgmap v3126: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:01.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:01 smithi121 bash[18489]: cluster 2024-02-12T16:55:00.621278+0000 osd.11 (osd.11) 280 : cluster [DBG] 1.b deep-scrub ok 2024-02-12T16:55:01.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:01 smithi169 bash[19604]: cluster 2024-02-12T16:54:59.980462+0000 osd.1 (osd.1) 231 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:55:01.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:01 smithi169 bash[19604]: cluster 2024-02-12T16:55:00.462086+0000 osd.11 (osd.11) 279 : cluster [DBG] 1.b deep-scrub starts 2024-02-12T16:55:01.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:01 smithi169 bash[19604]: cluster 2024-02-12T16:55:00.504130+0000 mgr.y (mgr.14146) 2124 : cluster [DBG] pgmap v3126: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:01.838 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:01 smithi169 bash[19604]: cluster 2024-02-12T16:55:00.621278+0000 osd.11 (osd.11) 280 : cluster [DBG] 1.b deep-scrub ok 2024-02-12T16:55:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:01 smithi123 bash[19119]: cluster 2024-02-12T16:54:59.980462+0000 osd.1 (osd.1) 231 : cluster [DBG] 1.16 scrub ok 2024-02-12T16:55:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:01 smithi123 bash[19119]: cluster 2024-02-12T16:55:00.462086+0000 osd.11 (osd.11) 279 : cluster [DBG] 1.b deep-scrub starts 2024-02-12T16:55:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:01 smithi123 bash[19119]: cluster 2024-02-12T16:55:00.504130+0000 mgr.y (mgr.14146) 2124 : cluster [DBG] pgmap v3126: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:01 smithi123 bash[19119]: cluster 2024-02-12T16:55:00.621278+0000 osd.11 (osd.11) 280 : cluster [DBG] 1.b deep-scrub ok 2024-02-12T16:55:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:03 smithi123 bash[19119]: cluster 2024-02-12T16:55:02.475448+0000 osd.2 (osd.2) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:03 smithi123 bash[19119]: cluster 2024-02-12T16:55:02.475536+0000 osd.2 (osd.2) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:03 smithi123 bash[19119]: cluster 2024-02-12T16:55:02.504790+0000 mgr.y (mgr.14146) 2125 : cluster [DBG] pgmap v3127: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:03.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:03 smithi123 bash[19119]: cluster 2024-02-12T16:55:02.583188+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e1136: 12 total, 12 up, 12 in 2024-02-12T16:55:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:03 smithi121 bash[18489]: cluster 2024-02-12T16:55:02.475448+0000 osd.2 (osd.2) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:03 smithi121 bash[18489]: cluster 2024-02-12T16:55:02.475536+0000 osd.2 (osd.2) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:03 smithi121 bash[18489]: cluster 2024-02-12T16:55:02.504790+0000 mgr.y (mgr.14146) 2125 : cluster [DBG] pgmap v3127: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:03 smithi121 bash[18489]: cluster 2024-02-12T16:55:02.583188+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e1136: 12 total, 12 up, 12 in 2024-02-12T16:55:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:03 smithi169 bash[19604]: cluster 2024-02-12T16:55:02.475448+0000 osd.2 (osd.2) 171 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:03 smithi169 bash[19604]: cluster 2024-02-12T16:55:02.475536+0000 osd.2 (osd.2) 172 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:03 smithi169 bash[19604]: cluster 2024-02-12T16:55:02.504790+0000 mgr.y (mgr.14146) 2125 : cluster [DBG] pgmap v3127: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:04.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:03 smithi169 bash[19604]: cluster 2024-02-12T16:55:02.583188+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e1136: 12 total, 12 up, 12 in 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:03.580558+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:03.580668+0000 osd.1 (osd.1) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:03.994327+0000 mon.a (mon.0) 3688 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1687772211,config=2903391811,health=2581032988,kv=3430585634}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:03.996090+0000 mon.a (mon.0) 3689 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2773047387,logm=735719948}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:03.997824+0000 mon.a (mon.0) 3690 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1627044826}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:03.999564+0000 mon.a (mon.0) 3691 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=554505229}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.001220+0000 mon.a (mon.0) 3692 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146252636}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.002802+0000 mon.a (mon.0) 3693 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576361151}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.010346+0000 mon.a (mon.0) 3694 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087609774}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.016052+0000 mon.a (mon.0) 3695 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3105205299}) 2024-02-12T16:55:04.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.021640+0000 mon.a (mon.0) 3696 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864570146}) 2024-02-12T16:55:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.026285+0000 mon.a (mon.0) 3697 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885205015}) 2024-02-12T16:55:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.030665+0000 mon.a (mon.0) 3698 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2221781948}) 2024-02-12T16:55:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.032774+0000 mon.a (mon.0) 3699 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=10,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=50} crc {logm=3585606402,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3663688687,mon_config_key=1160482841}) 2024-02-12T16:55:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.033630+0000 mon.a (mon.0) 3700 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1954259964}) 2024-02-12T16:55:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.034896+0000 mon.a (mon.0) 3701 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=96,monmap=4} crc {mon_config_key=898625644,monmap=3481771112}) 2024-02-12T16:55:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.036586+0000 mon.a (mon.0) 3702 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {monmap=1,osd_metadata=12,osd_pg_creating=1,osdmap=86} crc {monmap=3678701417,osd_metadata=4093509710,osd_pg_creating=636262315,osdmap=3223263619}) 2024-02-12T16:55:04.890 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:04 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.046229+0000 mon.a (mon.0) 3703 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=27} crc {osdmap=1446020844}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:03.580558+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:03.580668+0000 osd.1 (osd.1) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:03.994327+0000 mon.a (mon.0) 3688 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1687772211,config=2903391811,health=2581032988,kv=3430585634}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:03.996090+0000 mon.a (mon.0) 3689 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2773047387,logm=735719948}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:03.997824+0000 mon.a (mon.0) 3690 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1627044826}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:03.999564+0000 mon.a (mon.0) 3691 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=554505229}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.001220+0000 mon.a (mon.0) 3692 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146252636}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.002802+0000 mon.a (mon.0) 3693 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576361151}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.010346+0000 mon.a (mon.0) 3694 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087609774}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.016052+0000 mon.a (mon.0) 3695 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3105205299}) 2024-02-12T16:55:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.021640+0000 mon.a (mon.0) 3696 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864570146}) 2024-02-12T16:55:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.026285+0000 mon.a (mon.0) 3697 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885205015}) 2024-02-12T16:55:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.030665+0000 mon.a (mon.0) 3698 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2221781948}) 2024-02-12T16:55:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.032774+0000 mon.a (mon.0) 3699 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=10,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=50} crc {logm=3585606402,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3663688687,mon_config_key=1160482841}) 2024-02-12T16:55:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.033630+0000 mon.a (mon.0) 3700 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1954259964}) 2024-02-12T16:55:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.034896+0000 mon.a (mon.0) 3701 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=96,monmap=4} crc {mon_config_key=898625644,monmap=3481771112}) 2024-02-12T16:55:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.036586+0000 mon.a (mon.0) 3702 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {monmap=1,osd_metadata=12,osd_pg_creating=1,osdmap=86} crc {monmap=3678701417,osd_metadata=4093509710,osd_pg_creating=636262315,osdmap=3223263619}) 2024-02-12T16:55:04.912 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:04 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.046229+0000 mon.a (mon.0) 3703 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=27} crc {osdmap=1446020844}) 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:03.580558+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:03.580668+0000 osd.1 (osd.1) 233 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:03.994327+0000 mon.a (mon.0) 3688 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1687772211,config=2903391811,health=2581032988,kv=3430585634}) 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:03.996090+0000 mon.a (mon.0) 3689 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2773047387,logm=735719948}) 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:03.997824+0000 mon.a (mon.0) 3690 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1627044826}) 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:03.999564+0000 mon.a (mon.0) 3691 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=554505229}) 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.001220+0000 mon.a (mon.0) 3692 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146252636}) 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.002802+0000 mon.a (mon.0) 3693 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576361151}) 2024-02-12T16:55:05.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.010346+0000 mon.a (mon.0) 3694 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3087609774}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.016052+0000 mon.a (mon.0) 3695 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3105205299}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.021640+0000 mon.a (mon.0) 3696 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864570146}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.026285+0000 mon.a (mon.0) 3697 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885205015}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.030665+0000 mon.a (mon.0) 3698 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2221781948}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.032774+0000 mon.a (mon.0) 3699 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {logm=10,mdsmap=3,mgr=17,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=50} crc {logm=3585606402,mdsmap=1531151339,mgr=4141026994,mgr_command_descs=3987371412,mgr_metadata=2291712237,mgrstat=3663688687,mon_config_key=1160482841}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.033630+0000 mon.a (mon.0) 3700 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1954259964}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.034896+0000 mon.a (mon.0) 3701 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=96,monmap=4} crc {mon_config_key=898625644,monmap=3481771112}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.036586+0000 mon.a (mon.0) 3702 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {monmap=1,osd_metadata=12,osd_pg_creating=1,osdmap=86} crc {monmap=3678701417,osd_metadata=4093509710,osd_pg_creating=636262315,osdmap=3223263619}) 2024-02-12T16:55:05.049 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:04 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.046229+0000 mon.a (mon.0) 3703 : cluster [DBG] scrub ok on 0,1,2: ScrubResult(keys {osdmap=27} crc {osdmap=1446020844}) 2024-02-12T16:55:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:05 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.505544+0000 mgr.y (mgr.14146) 2126 : cluster [DBG] pgmap v3129: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:05 smithi123 bash[19119]: cluster 2024-02-12T16:55:04.600151+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e1137: 12 total, 12 up, 12 in 2024-02-12T16:55:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:05 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.505544+0000 mgr.y (mgr.14146) 2126 : cluster [DBG] pgmap v3129: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:05 smithi121 bash[18489]: cluster 2024-02-12T16:55:04.600151+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e1137: 12 total, 12 up, 12 in 2024-02-12T16:55:06.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:05 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.505544+0000 mgr.y (mgr.14146) 2126 : cluster [DBG] pgmap v3129: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:06.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:05 smithi169 bash[19604]: cluster 2024-02-12T16:55:04.600151+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e1137: 12 total, 12 up, 12 in 2024-02-12T16:55:06.108 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:06 smithi123 bash[19119]: cluster 2024-02-12T16:55:05.187151+0000 osd.6 (osd.6) 230 : cluster [DBG] 1.f scrub starts 2024-02-12T16:55:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:06 smithi123 bash[19119]: cluster 2024-02-12T16:55:05.220147+0000 osd.0 (osd.0) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:06.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:06 smithi123 bash[19119]: cluster 2024-02-12T16:55:05.220237+0000 osd.0 (osd.0) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:06 smithi123 bash[19119]: cluster 2024-02-12T16:55:05.331567+0000 osd.6 (osd.6) 231 : cluster [DBG] 1.f scrub ok 2024-02-12T16:55:06.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:06 smithi123 bash[19119]: cluster 2024-02-12T16:55:05.599256+0000 mon.a (mon.0) 3705 : cluster [DBG] osdmap e1138: 12 total, 12 up, 12 in 2024-02-12T16:55:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:06 smithi121 bash[18489]: cluster 2024-02-12T16:55:05.187151+0000 osd.6 (osd.6) 230 : cluster [DBG] 1.f scrub starts 2024-02-12T16:55:06.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:06 smithi121 bash[18489]: cluster 2024-02-12T16:55:05.220147+0000 osd.0 (osd.0) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:06.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:06 smithi121 bash[18489]: cluster 2024-02-12T16:55:05.220237+0000 osd.0 (osd.0) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:06.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:06 smithi121 bash[18489]: cluster 2024-02-12T16:55:05.331567+0000 osd.6 (osd.6) 231 : cluster [DBG] 1.f scrub ok 2024-02-12T16:55:06.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:06 smithi121 bash[18489]: cluster 2024-02-12T16:55:05.599256+0000 mon.a (mon.0) 3705 : cluster [DBG] osdmap e1138: 12 total, 12 up, 12 in 2024-02-12T16:55:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:06 smithi169 bash[19604]: cluster 2024-02-12T16:55:05.187151+0000 osd.6 (osd.6) 230 : cluster [DBG] 1.f scrub starts 2024-02-12T16:55:07.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:06 smithi169 bash[19604]: cluster 2024-02-12T16:55:05.220147+0000 osd.0 (osd.0) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:06 smithi169 bash[19604]: cluster 2024-02-12T16:55:05.220237+0000 osd.0 (osd.0) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:06 smithi169 bash[19604]: cluster 2024-02-12T16:55:05.331567+0000 osd.6 (osd.6) 231 : cluster [DBG] 1.f scrub ok 2024-02-12T16:55:07.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:06 smithi169 bash[19604]: cluster 2024-02-12T16:55:05.599256+0000 mon.a (mon.0) 3705 : cluster [DBG] osdmap e1138: 12 total, 12 up, 12 in 2024-02-12T16:55:07.716 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:07 smithi121 bash[18489]: cluster 2024-02-12T16:55:05.979527+0000 osd.1 (osd.1) 234 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:55:07.716 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:07 smithi121 bash[18489]: cluster 2024-02-12T16:55:06.196427+0000 osd.1 (osd.1) 235 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:55:07.716 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:07 smithi121 bash[18489]: cluster 2024-02-12T16:55:06.506194+0000 mgr.y (mgr.14146) 2127 : cluster [DBG] pgmap v3132: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:07 smithi123 bash[19119]: cluster 2024-02-12T16:55:05.979527+0000 osd.1 (osd.1) 234 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:55:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:07 smithi123 bash[19119]: cluster 2024-02-12T16:55:06.196427+0000 osd.1 (osd.1) 235 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:55:07.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:07 smithi123 bash[19119]: cluster 2024-02-12T16:55:06.506194+0000 mgr.y (mgr.14146) 2127 : cluster [DBG] pgmap v3132: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:07 smithi169 bash[19604]: cluster 2024-02-12T16:55:05.979527+0000 osd.1 (osd.1) 234 : cluster [DBG] 1.1d scrub starts 2024-02-12T16:55:08.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:07 smithi169 bash[19604]: cluster 2024-02-12T16:55:06.196427+0000 osd.1 (osd.1) 235 : cluster [DBG] 1.1d scrub ok 2024-02-12T16:55:08.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:07 smithi169 bash[19604]: cluster 2024-02-12T16:55:06.506194+0000 mgr.y (mgr.14146) 2127 : cluster [DBG] pgmap v3132: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:09.730 INFO:teuthology.orchestra.run.smithi121.stdout:3964254814359 2024-02-12T16:55:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:09 smithi123 bash[19119]: cluster 2024-02-12T16:55:08.506897+0000 mgr.y (mgr.14146) 2128 : cluster [DBG] pgmap v3133: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:09.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:09 smithi123 bash[19119]: cluster 2024-02-12T16:55:08.852582+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:09 smithi123 bash[19119]: cluster 2024-02-12T16:55:08.852705+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:09 smithi123 bash[19119]: cluster 2024-02-12T16:55:09.051495+0000 osd.5 (osd.5) 281 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:55:09.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:09 smithi123 bash[19119]: cluster 2024-02-12T16:55:09.168893+0000 osd.5 (osd.5) 282 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:55:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:09 smithi121 bash[18489]: cluster 2024-02-12T16:55:08.506897+0000 mgr.y (mgr.14146) 2128 : cluster [DBG] pgmap v3133: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:09 smithi121 bash[18489]: cluster 2024-02-12T16:55:08.852582+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:09.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:09 smithi121 bash[18489]: cluster 2024-02-12T16:55:08.852705+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:09.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:09 smithi121 bash[18489]: cluster 2024-02-12T16:55:09.051495+0000 osd.5 (osd.5) 281 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:55:09.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:09 smithi121 bash[18489]: cluster 2024-02-12T16:55:09.168893+0000 osd.5 (osd.5) 282 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:55:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:09 smithi169 bash[19604]: cluster 2024-02-12T16:55:08.506897+0000 mgr.y (mgr.14146) 2128 : cluster [DBG] pgmap v3133: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:09 smithi169 bash[19604]: cluster 2024-02-12T16:55:08.852582+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:09 smithi169 bash[19604]: cluster 2024-02-12T16:55:08.852705+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:09 smithi169 bash[19604]: cluster 2024-02-12T16:55:09.051495+0000 osd.5 (osd.5) 281 : cluster [DBG] 1.12 scrub starts 2024-02-12T16:55:10.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:09 smithi169 bash[19604]: cluster 2024-02-12T16:55:09.168893+0000 osd.5 (osd.5) 282 : cluster [DBG] 1.12 scrub ok 2024-02-12T16:55:10.467 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3964254814337 got 3964254814359 for osd.7 2024-02-12T16:55:10.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-02-12T16:55:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:10 smithi123 bash[19119]: cluster 2024-02-12T16:55:09.619327+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e1139: 12 total, 12 up, 12 in 2024-02-12T16:55:10.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:10 smithi123 bash[19119]: audit 2024-02-12T16:55:09.724492+0000 mon.a (mon.0) 3707 : audit [DBG] from='client.? v1:172.21.15.121:0/1560796670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:55:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:10 smithi121 bash[18489]: cluster 2024-02-12T16:55:09.619327+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e1139: 12 total, 12 up, 12 in 2024-02-12T16:55:10.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:10 smithi121 bash[18489]: audit 2024-02-12T16:55:09.724492+0000 mon.a (mon.0) 3707 : audit [DBG] from='client.? v1:172.21.15.121:0/1560796670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:55:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:10 smithi169 bash[19604]: cluster 2024-02-12T16:55:09.619327+0000 mon.a (mon.0) 3706 : cluster [DBG] osdmap e1139: 12 total, 12 up, 12 in 2024-02-12T16:55:11.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:10 smithi169 bash[19604]: audit 2024-02-12T16:55:09.724492+0000 mon.a (mon.0) 3707 : audit [DBG] from='client.? v1:172.21.15.121:0/1560796670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-12T16:55:11.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:11 smithi121 bash[18489]: cluster 2024-02-12T16:55:10.507516+0000 mgr.y (mgr.14146) 2129 : cluster [DBG] pgmap v3135: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:12.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:11 smithi169 bash[19604]: cluster 2024-02-12T16:55:10.507516+0000 mgr.y (mgr.14146) 2129 : cluster [DBG] pgmap v3135: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:12.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:11 smithi123 bash[19119]: cluster 2024-02-12T16:55:10.507516+0000 mgr.y (mgr.14146) 2129 : cluster [DBG] pgmap v3135: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:13 smithi121 bash[18489]: cluster 2024-02-12T16:55:11.756669+0000 osd.10 (osd.10) 318 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:55:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:13 smithi121 bash[18489]: cluster 2024-02-12T16:55:11.862294+0000 osd.10 (osd.10) 319 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:55:13.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:13 smithi121 bash[18489]: cluster 2024-02-12T16:55:12.508188+0000 mgr.y (mgr.14146) 2130 : cluster [DBG] pgmap v3136: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:14.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:13 smithi169 bash[19604]: cluster 2024-02-12T16:55:11.756669+0000 osd.10 (osd.10) 318 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:55:14.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:13 smithi169 bash[19604]: cluster 2024-02-12T16:55:11.862294+0000 osd.10 (osd.10) 319 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:55:14.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:13 smithi169 bash[19604]: cluster 2024-02-12T16:55:12.508188+0000 mgr.y (mgr.14146) 2130 : cluster [DBG] pgmap v3136: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:13 smithi123 bash[19119]: cluster 2024-02-12T16:55:11.756669+0000 osd.10 (osd.10) 318 : cluster [DBG] 1.19 scrub starts 2024-02-12T16:55:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:13 smithi123 bash[19119]: cluster 2024-02-12T16:55:11.862294+0000 osd.10 (osd.10) 319 : cluster [DBG] 1.19 scrub ok 2024-02-12T16:55:14.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:13 smithi123 bash[19119]: cluster 2024-02-12T16:55:12.508188+0000 mgr.y (mgr.14146) 2130 : cluster [DBG] pgmap v3136: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:14.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:14 smithi121 bash[18489]: cluster 2024-02-12T16:55:13.890062+0000 osd.0 (osd.0) 197 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:55:15.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:14 smithi169 bash[19604]: cluster 2024-02-12T16:55:13.890062+0000 osd.0 (osd.0) 197 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:55:15.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:14 smithi123 bash[19119]: cluster 2024-02-12T16:55:13.890062+0000 osd.0 (osd.0) 197 : cluster [DBG] 1.11 deep-scrub starts 2024-02-12T16:55:15.251 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:16.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:15 smithi121 bash[18489]: cluster 2024-02-12T16:55:13.758289+0000 osd.2 (osd.2) 173 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:55:16.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:15 smithi121 bash[18489]: cluster 2024-02-12T16:55:13.864683+0000 osd.2 (osd.2) 174 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:55:16.020 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:15 smithi121 bash[18489]: cluster 2024-02-12T16:55:13.998283+0000 osd.0 (osd.0) 198 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:55:16.021 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:15 smithi121 bash[18489]: cluster 2024-02-12T16:55:14.508883+0000 mgr.y (mgr.14146) 2131 : cluster [DBG] pgmap v3137: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:16.021 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:15 smithi121 bash[18489]: cluster 2024-02-12T16:55:14.663704+0000 osd.9 (osd.9) 193 : cluster [DBG] 1.a scrub starts 2024-02-12T16:55:16.021 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:15 smithi121 bash[18489]: cluster 2024-02-12T16:55:14.754847+0000 osd.9 (osd.9) 194 : cluster [DBG] 1.a scrub ok 2024-02-12T16:55:16.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:15 smithi169 bash[19604]: cluster 2024-02-12T16:55:13.758289+0000 osd.2 (osd.2) 173 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:55:16.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:15 smithi169 bash[19604]: cluster 2024-02-12T16:55:13.864683+0000 osd.2 (osd.2) 174 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:55:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:15 smithi169 bash[19604]: cluster 2024-02-12T16:55:13.998283+0000 osd.0 (osd.0) 198 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:55:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:15 smithi169 bash[19604]: cluster 2024-02-12T16:55:14.508883+0000 mgr.y (mgr.14146) 2131 : cluster [DBG] pgmap v3137: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:15 smithi169 bash[19604]: cluster 2024-02-12T16:55:14.663704+0000 osd.9 (osd.9) 193 : cluster [DBG] 1.a scrub starts 2024-02-12T16:55:16.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:15 smithi169 bash[19604]: cluster 2024-02-12T16:55:14.754847+0000 osd.9 (osd.9) 194 : cluster [DBG] 1.a scrub ok 2024-02-12T16:55:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:15 smithi123 bash[19119]: cluster 2024-02-12T16:55:13.758289+0000 osd.2 (osd.2) 173 : cluster [DBG] 1.14 scrub starts 2024-02-12T16:55:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:15 smithi123 bash[19119]: cluster 2024-02-12T16:55:13.864683+0000 osd.2 (osd.2) 174 : cluster [DBG] 1.14 scrub ok 2024-02-12T16:55:16.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:15 smithi123 bash[19119]: cluster 2024-02-12T16:55:13.998283+0000 osd.0 (osd.0) 198 : cluster [DBG] 1.11 deep-scrub ok 2024-02-12T16:55:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:15 smithi123 bash[19119]: cluster 2024-02-12T16:55:14.508883+0000 mgr.y (mgr.14146) 2131 : cluster [DBG] pgmap v3137: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:15 smithi123 bash[19119]: cluster 2024-02-12T16:55:14.663704+0000 osd.9 (osd.9) 193 : cluster [DBG] 1.a scrub starts 2024-02-12T16:55:16.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:15 smithi123 bash[19119]: cluster 2024-02-12T16:55:14.754847+0000 osd.9 (osd.9) 194 : cluster [DBG] 1.a scrub ok 2024-02-12T16:55:16.862 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:16 smithi121 bash[18489]: cluster 2024-02-12T16:55:15.637470+0000 osd.3 (osd.3) 166 : cluster [DBG] 1.c scrub starts 2024-02-12T16:55:16.862 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:16 smithi121 bash[18489]: cluster 2024-02-12T16:55:15.783814+0000 osd.3 (osd.3) 167 : cluster [DBG] 1.c scrub ok 2024-02-12T16:55:17.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:16 smithi169 bash[19604]: cluster 2024-02-12T16:55:15.637470+0000 osd.3 (osd.3) 166 : cluster [DBG] 1.c scrub starts 2024-02-12T16:55:17.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:16 smithi169 bash[19604]: cluster 2024-02-12T16:55:15.783814+0000 osd.3 (osd.3) 167 : cluster [DBG] 1.c scrub ok 2024-02-12T16:55:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:16 smithi123 bash[19119]: cluster 2024-02-12T16:55:15.637470+0000 osd.3 (osd.3) 166 : cluster [DBG] 1.c scrub starts 2024-02-12T16:55:17.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:16 smithi123 bash[19119]: cluster 2024-02-12T16:55:15.783814+0000 osd.3 (osd.3) 167 : cluster [DBG] 1.c scrub ok 2024-02-12T16:55:18.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:17 smithi169 bash[19604]: cluster 2024-02-12T16:55:16.509576+0000 mgr.y (mgr.14146) 2132 : cluster [DBG] pgmap v3138: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:18.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:17 smithi169 bash[19604]: cluster 2024-02-12T16:55:16.730881+0000 osd.7 (osd.7) 95 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:55:18.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:17 smithi169 bash[19604]: cluster 2024-02-12T16:55:16.833037+0000 osd.7 (osd.7) 96 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:55:18.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:17 smithi123 bash[19119]: cluster 2024-02-12T16:55:16.509576+0000 mgr.y (mgr.14146) 2132 : cluster [DBG] pgmap v3138: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:18.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:17 smithi123 bash[19119]: cluster 2024-02-12T16:55:16.730881+0000 osd.7 (osd.7) 95 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:55:18.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:17 smithi123 bash[19119]: cluster 2024-02-12T16:55:16.833037+0000 osd.7 (osd.7) 96 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:55:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:17 smithi121 bash[18489]: cluster 2024-02-12T16:55:16.509576+0000 mgr.y (mgr.14146) 2132 : cluster [DBG] pgmap v3138: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:17 smithi121 bash[18489]: cluster 2024-02-12T16:55:16.730881+0000 osd.7 (osd.7) 95 : cluster [DBG] 1.15 scrub starts 2024-02-12T16:55:18.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:17 smithi121 bash[18489]: cluster 2024-02-12T16:55:16.833037+0000 osd.7 (osd.7) 96 : cluster [DBG] 1.15 scrub ok 2024-02-12T16:55:18.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:18 smithi121 bash[18489]: cluster 2024-02-12T16:55:17.602251+0000 osd.11 (osd.11) 281 : cluster [DBG] 1.4 deep-scrub starts 2024-02-12T16:55:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:18 smithi121 bash[18489]: cluster 2024-02-12T16:55:17.625650+0000 osd.11 (osd.11) 282 : cluster [DBG] 1.4 deep-scrub ok 2024-02-12T16:55:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:18 smithi121 bash[18489]: cluster 2024-02-12T16:55:17.952458+0000 osd.8 (osd.8) 193 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:55:18.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:18 smithi121 bash[18489]: cluster 2024-02-12T16:55:18.076674+0000 osd.8 (osd.8) 194 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:55:18.961 INFO:teuthology.orchestra.run.smithi121.stdout:279172874953 2024-02-12T16:55:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:18 smithi169 bash[19604]: cluster 2024-02-12T16:55:17.602251+0000 osd.11 (osd.11) 281 : cluster [DBG] 1.4 deep-scrub starts 2024-02-12T16:55:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:18 smithi169 bash[19604]: cluster 2024-02-12T16:55:17.625650+0000 osd.11 (osd.11) 282 : cluster [DBG] 1.4 deep-scrub ok 2024-02-12T16:55:19.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:18 smithi169 bash[19604]: cluster 2024-02-12T16:55:17.952458+0000 osd.8 (osd.8) 193 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:55:19.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:18 smithi169 bash[19604]: cluster 2024-02-12T16:55:18.076674+0000 osd.8 (osd.8) 194 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:55:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:18 smithi123 bash[19119]: cluster 2024-02-12T16:55:17.602251+0000 osd.11 (osd.11) 281 : cluster [DBG] 1.4 deep-scrub starts 2024-02-12T16:55:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:18 smithi123 bash[19119]: cluster 2024-02-12T16:55:17.625650+0000 osd.11 (osd.11) 282 : cluster [DBG] 1.4 deep-scrub ok 2024-02-12T16:55:19.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:18 smithi123 bash[19119]: cluster 2024-02-12T16:55:17.952458+0000 osd.8 (osd.8) 193 : cluster [DBG] 1.17 scrub starts 2024-02-12T16:55:19.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:18 smithi123 bash[19119]: cluster 2024-02-12T16:55:18.076674+0000 osd.8 (osd.8) 194 : cluster [DBG] 1.17 scrub ok 2024-02-12T16:55:19.723 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874931 got 279172874953 for osd.8 2024-02-12T16:55:19.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-02-12T16:55:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:19 smithi169 bash[19604]: cluster 2024-02-12T16:55:18.510220+0000 mgr.y (mgr.14146) 2133 : cluster [DBG] pgmap v3139: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:19 smithi169 bash[19604]: cluster 2024-02-12T16:55:18.710574+0000 osd.10 (osd.10) 320 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:55:20.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:19 smithi169 bash[19604]: cluster 2024-02-12T16:55:18.805216+0000 osd.10 (osd.10) 321 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:55:20.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:19 smithi169 bash[19604]: audit 2024-02-12T16:55:18.955069+0000 mon.a (mon.0) 3708 : audit [DBG] from='client.? v1:172.21.15.121:0/2361553009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-12T16:55:20.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:19 smithi123 bash[19119]: cluster 2024-02-12T16:55:18.510220+0000 mgr.y (mgr.14146) 2133 : cluster [DBG] pgmap v3139: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:20.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:19 smithi123 bash[19119]: cluster 2024-02-12T16:55:18.710574+0000 osd.10 (osd.10) 320 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:55:20.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:19 smithi123 bash[19119]: cluster 2024-02-12T16:55:18.805216+0000 osd.10 (osd.10) 321 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:55:20.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:19 smithi123 bash[19119]: audit 2024-02-12T16:55:18.955069+0000 mon.a (mon.0) 3708 : audit [DBG] from='client.? v1:172.21.15.121:0/2361553009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-12T16:55:20.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:19 smithi121 bash[18489]: cluster 2024-02-12T16:55:18.510220+0000 mgr.y (mgr.14146) 2133 : cluster [DBG] pgmap v3139: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:20.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:19 smithi121 bash[18489]: cluster 2024-02-12T16:55:18.710574+0000 osd.10 (osd.10) 320 : cluster [DBG] 1.5 scrub starts 2024-02-12T16:55:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:19 smithi121 bash[18489]: cluster 2024-02-12T16:55:18.805216+0000 osd.10 (osd.10) 321 : cluster [DBG] 1.5 scrub ok 2024-02-12T16:55:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:19 smithi121 bash[18489]: audit 2024-02-12T16:55:18.955069+0000 mon.a (mon.0) 3708 : audit [DBG] from='client.? v1:172.21.15.121:0/2361553009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-12T16:55:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:20 smithi169 bash[19604]: cluster 2024-02-12T16:55:20.264284+0000 osd.4 (osd.4) 221 : cluster [DBG] 1.e scrub starts 2024-02-12T16:55:21.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:20 smithi169 bash[19604]: cluster 2024-02-12T16:55:20.357021+0000 osd.4 (osd.4) 222 : cluster [DBG] 1.e scrub ok 2024-02-12T16:55:21.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:20 smithi123 bash[19119]: cluster 2024-02-12T16:55:20.264284+0000 osd.4 (osd.4) 221 : cluster [DBG] 1.e scrub starts 2024-02-12T16:55:21.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:20 smithi123 bash[19119]: cluster 2024-02-12T16:55:20.357021+0000 osd.4 (osd.4) 222 : cluster [DBG] 1.e scrub ok 2024-02-12T16:55:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:20 smithi121 bash[18489]: cluster 2024-02-12T16:55:20.264284+0000 osd.4 (osd.4) 221 : cluster [DBG] 1.e scrub starts 2024-02-12T16:55:21.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:20 smithi121 bash[18489]: cluster 2024-02-12T16:55:20.357021+0000 osd.4 (osd.4) 222 : cluster [DBG] 1.e scrub ok 2024-02-12T16:55:22.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:21 smithi169 bash[19604]: cluster 2024-02-12T16:55:20.510963+0000 mgr.y (mgr.14146) 2134 : cluster [DBG] pgmap v3140: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:22.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:21 smithi123 bash[19119]: cluster 2024-02-12T16:55:20.510963+0000 mgr.y (mgr.14146) 2134 : cluster [DBG] pgmap v3140: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:22.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:21 smithi121 bash[18489]: cluster 2024-02-12T16:55:20.510963+0000 mgr.y (mgr.14146) 2134 : cluster [DBG] pgmap v3140: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:22 smithi169 bash[19604]: cluster 2024-02-12T16:55:21.826539+0000 osd.8 (osd.8) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:23.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:22 smithi169 bash[19604]: cluster 2024-02-12T16:55:21.826631+0000 osd.8 (osd.8) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:22 smithi123 bash[19119]: cluster 2024-02-12T16:55:21.826539+0000 osd.8 (osd.8) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:23.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:22 smithi123 bash[19119]: cluster 2024-02-12T16:55:21.826631+0000 osd.8 (osd.8) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:22 smithi121 bash[18489]: cluster 2024-02-12T16:55:21.826539+0000 osd.8 (osd.8) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:23.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:22 smithi121 bash[18489]: cluster 2024-02-12T16:55:21.826631+0000 osd.8 (osd.8) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:23 smithi169 bash[19604]: cluster 2024-02-12T16:55:22.511548+0000 mgr.y (mgr.14146) 2135 : cluster [DBG] pgmap v3141: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:24.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:23 smithi169 bash[19604]: cluster 2024-02-12T16:55:22.692586+0000 mon.a (mon.0) 3709 : cluster [DBG] osdmap e1140: 12 total, 12 up, 12 in 2024-02-12T16:55:24.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:23 smithi123 bash[19119]: cluster 2024-02-12T16:55:22.511548+0000 mgr.y (mgr.14146) 2135 : cluster [DBG] pgmap v3141: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:24.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:23 smithi123 bash[19119]: cluster 2024-02-12T16:55:22.692586+0000 mon.a (mon.0) 3709 : cluster [DBG] osdmap e1140: 12 total, 12 up, 12 in 2024-02-12T16:55:24.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:23 smithi121 bash[18489]: cluster 2024-02-12T16:55:22.511548+0000 mgr.y (mgr.14146) 2135 : cluster [DBG] pgmap v3141: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:24.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:23 smithi121 bash[18489]: cluster 2024-02-12T16:55:22.692586+0000 mon.a (mon.0) 3709 : cluster [DBG] osdmap e1140: 12 total, 12 up, 12 in 2024-02-12T16:55:24.526 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:25.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:25 smithi121 bash[18489]: cluster 2024-02-12T16:55:24.512202+0000 mgr.y (mgr.14146) 2136 : cluster [DBG] pgmap v3143: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:25.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:25 smithi121 bash[18489]: cluster 2024-02-12T16:55:25.206820+0000 osd.4 (osd.4) 223 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:55:25.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:25 smithi121 bash[18489]: cluster 2024-02-12T16:55:25.306104+0000 osd.4 (osd.4) 224 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:55:26.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:25 smithi169 bash[19604]: cluster 2024-02-12T16:55:24.512202+0000 mgr.y (mgr.14146) 2136 : cluster [DBG] pgmap v3143: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:25 smithi169 bash[19604]: cluster 2024-02-12T16:55:25.206820+0000 osd.4 (osd.4) 223 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:55:26.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:25 smithi169 bash[19604]: cluster 2024-02-12T16:55:25.306104+0000 osd.4 (osd.4) 224 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:55:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:25 smithi123 bash[19119]: cluster 2024-02-12T16:55:24.512202+0000 mgr.y (mgr.14146) 2136 : cluster [DBG] pgmap v3143: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:26.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:25 smithi123 bash[19119]: cluster 2024-02-12T16:55:25.206820+0000 osd.4 (osd.4) 223 : cluster [DBG] 1.1a scrub starts 2024-02-12T16:55:26.139 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:25 smithi123 bash[19119]: cluster 2024-02-12T16:55:25.306104+0000 osd.4 (osd.4) 224 : cluster [DBG] 1.1a scrub ok 2024-02-12T16:55:27.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:27 smithi169 bash[19604]: cluster 2024-02-12T16:55:26.512886+0000 mgr.y (mgr.14146) 2137 : cluster [DBG] pgmap v3144: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:27.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:27 smithi169 bash[19604]: cluster 2024-02-12T16:55:26.669867+0000 osd.7 (osd.7) 97 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:55:27.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:27 smithi169 bash[19604]: cluster 2024-02-12T16:55:26.697613+0000 osd.7 (osd.7) 98 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:55:27.638 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:27 smithi121 bash[18489]: cluster 2024-02-12T16:55:26.512886+0000 mgr.y (mgr.14146) 2137 : cluster [DBG] pgmap v3144: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:27.638 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:27 smithi121 bash[18489]: cluster 2024-02-12T16:55:26.669867+0000 osd.7 (osd.7) 97 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:55:27.638 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:27 smithi121 bash[18489]: cluster 2024-02-12T16:55:26.697613+0000 osd.7 (osd.7) 98 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:55:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:27 smithi123 bash[19119]: cluster 2024-02-12T16:55:26.512886+0000 mgr.y (mgr.14146) 2137 : cluster [DBG] pgmap v3144: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:27 smithi123 bash[19119]: cluster 2024-02-12T16:55:26.669867+0000 osd.7 (osd.7) 97 : cluster [DBG] 1.0 deep-scrub starts 2024-02-12T16:55:27.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:27 smithi123 bash[19119]: cluster 2024-02-12T16:55:26.697613+0000 osd.7 (osd.7) 98 : cluster [DBG] 1.0 deep-scrub ok 2024-02-12T16:55:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:317827580610 2024-02-12T16:55:28.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:28 smithi121 bash[18489]: audit 2024-02-12T16:55:28.050613+0000 mon.a (mon.0) 3710 : audit [DBG] from='client.? v1:172.21.15.121:0/3984520761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-12T16:55:28.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:28 smithi169 bash[19604]: audit 2024-02-12T16:55:28.050613+0000 mon.a (mon.0) 3710 : audit [DBG] from='client.? v1:172.21.15.121:0/3984520761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-12T16:55:28.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:28 smithi123 bash[19119]: audit 2024-02-12T16:55:28.050613+0000 mon.a (mon.0) 3710 : audit [DBG] from='client.? v1:172.21.15.121:0/3984520761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-12T16:55:28.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827580588 got 317827580610 for osd.9 2024-02-12T16:55:28.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-02-12T16:55:29.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:29 smithi169 bash[19604]: cluster 2024-02-12T16:55:28.513604+0000 mgr.y (mgr.14146) 2138 : cluster [DBG] pgmap v3145: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:29.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:29 smithi123 bash[19119]: cluster 2024-02-12T16:55:28.513604+0000 mgr.y (mgr.14146) 2138 : cluster [DBG] pgmap v3145: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:29.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:29 smithi121 bash[18489]: cluster 2024-02-12T16:55:28.513604+0000 mgr.y (mgr.14146) 2138 : cluster [DBG] pgmap v3145: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:30.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:30 smithi169 bash[19604]: audit 2024-02-12T16:55:29.987831+0000 mon.a (mon.0) 3711 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:30.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:30 smithi123 bash[19119]: audit 2024-02-12T16:55:29.987831+0000 mon.a (mon.0) 3711 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:30.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:30 smithi121 bash[18489]: audit 2024-02-12T16:55:29.987831+0000 mon.a (mon.0) 3711 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:31.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:31 smithi123 bash[19119]: cluster 2024-02-12T16:55:29.769756+0000 osd.10 (osd.10) 322 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:55:31.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:31 smithi123 bash[19119]: cluster 2024-02-12T16:55:29.819657+0000 osd.10 (osd.10) 323 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:55:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:31 smithi123 bash[19119]: audit 2024-02-12T16:55:30.313252+0000 mon.a (mon.0) 3712 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:31 smithi123 bash[19119]: audit 2024-02-12T16:55:30.314768+0000 mon.a (mon.0) 3713 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:31 smithi123 bash[19119]: audit 2024-02-12T16:55:30.325525+0000 mon.a (mon.0) 3714 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:55:31.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:31 smithi123 bash[19119]: cluster 2024-02-12T16:55:30.514330+0000 mgr.y (mgr.14146) 2139 : cluster [DBG] pgmap v3146: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:31.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:31 smithi121 bash[18489]: cluster 2024-02-12T16:55:29.769756+0000 osd.10 (osd.10) 322 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:55:31.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:31 smithi121 bash[18489]: cluster 2024-02-12T16:55:29.819657+0000 osd.10 (osd.10) 323 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:55:31.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:31 smithi121 bash[18489]: audit 2024-02-12T16:55:30.313252+0000 mon.a (mon.0) 3712 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:31.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:31 smithi121 bash[18489]: audit 2024-02-12T16:55:30.314768+0000 mon.a (mon.0) 3713 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:31.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:31 smithi121 bash[18489]: audit 2024-02-12T16:55:30.325525+0000 mon.a (mon.0) 3714 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:55:31.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:31 smithi121 bash[18489]: cluster 2024-02-12T16:55:30.514330+0000 mgr.y (mgr.14146) 2139 : cluster [DBG] pgmap v3146: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:31.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:31 smithi169 bash[19604]: cluster 2024-02-12T16:55:29.769756+0000 osd.10 (osd.10) 322 : cluster [DBG] 1.6 deep-scrub starts 2024-02-12T16:55:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:31 smithi169 bash[19604]: cluster 2024-02-12T16:55:29.819657+0000 osd.10 (osd.10) 323 : cluster [DBG] 1.6 deep-scrub ok 2024-02-12T16:55:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:31 smithi169 bash[19604]: audit 2024-02-12T16:55:30.313252+0000 mon.a (mon.0) 3712 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:31 smithi169 bash[19604]: audit 2024-02-12T16:55:30.314768+0000 mon.a (mon.0) 3713 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:31 smithi169 bash[19604]: audit 2024-02-12T16:55:30.325525+0000 mon.a (mon.0) 3714 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:55:31.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:31 smithi169 bash[19604]: cluster 2024-02-12T16:55:30.514330+0000 mgr.y (mgr.14146) 2139 : cluster [DBG] pgmap v3146: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:33.630 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:33.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:33 smithi123 bash[19119]: cluster 2024-02-12T16:55:32.515128+0000 mgr.y (mgr.14146) 2140 : cluster [DBG] pgmap v3147: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:33.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:33 smithi121 bash[18489]: cluster 2024-02-12T16:55:32.515128+0000 mgr.y (mgr.14146) 2140 : cluster [DBG] pgmap v3147: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:34.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:33 smithi169 bash[19604]: cluster 2024-02-12T16:55:32.515128+0000 mgr.y (mgr.14146) 2140 : cluster [DBG] pgmap v3147: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:34.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:34 smithi123 bash[19119]: cluster 2024-02-12T16:55:32.835311+0000 osd.5 (osd.5) 283 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:55:34.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:34 smithi123 bash[19119]: cluster 2024-02-12T16:55:32.907534+0000 osd.5 (osd.5) 284 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:55:34.904 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:34 smithi121 bash[18489]: cluster 2024-02-12T16:55:32.835311+0000 osd.5 (osd.5) 283 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:55:34.905 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:34 smithi121 bash[18489]: cluster 2024-02-12T16:55:32.907534+0000 osd.5 (osd.5) 284 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:55:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:34 smithi169 bash[19604]: cluster 2024-02-12T16:55:32.835311+0000 osd.5 (osd.5) 283 : cluster [DBG] 1.8 deep-scrub starts 2024-02-12T16:55:35.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:34 smithi169 bash[19604]: cluster 2024-02-12T16:55:32.907534+0000 osd.5 (osd.5) 284 : cluster [DBG] 1.8 deep-scrub ok 2024-02-12T16:55:35.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:35 smithi123 bash[19119]: cluster 2024-02-12T16:55:34.515862+0000 mgr.y (mgr.14146) 2141 : cluster [DBG] pgmap v3148: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:35.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:35 smithi121 bash[18489]: cluster 2024-02-12T16:55:34.515862+0000 mgr.y (mgr.14146) 2141 : cluster [DBG] pgmap v3148: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:36.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:35 smithi169 bash[19604]: cluster 2024-02-12T16:55:34.515862+0000 mgr.y (mgr.14146) 2141 : cluster [DBG] pgmap v3148: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:36.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:36 smithi123 bash[19119]: cluster 2024-02-12T16:55:35.599619+0000 osd.7 (osd.7) 99 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:55:36.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:36 smithi123 bash[19119]: cluster 2024-02-12T16:55:35.692727+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:55:36.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:36 smithi121 bash[18489]: cluster 2024-02-12T16:55:35.599619+0000 osd.7 (osd.7) 99 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:55:36.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:36 smithi121 bash[18489]: cluster 2024-02-12T16:55:35.692727+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:55:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:36 smithi169 bash[19604]: cluster 2024-02-12T16:55:35.599619+0000 osd.7 (osd.7) 99 : cluster [DBG] 1.9 scrub starts 2024-02-12T16:55:37.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:36 smithi169 bash[19604]: cluster 2024-02-12T16:55:35.692727+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.9 scrub ok 2024-02-12T16:55:37.282 INFO:teuthology.orchestra.run.smithi121.stdout:356482286267 2024-02-12T16:55:37.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:37 smithi123 bash[19119]: cluster 2024-02-12T16:55:36.516515+0000 mgr.y (mgr.14146) 2142 : cluster [DBG] pgmap v3149: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:37.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:37 smithi123 bash[19119]: audit 2024-02-12T16:55:37.276643+0000 mon.a (mon.0) 3715 : audit [DBG] from='client.? v1:172.21.15.121:0/3023280926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-12T16:55:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:37 smithi121 bash[18489]: cluster 2024-02-12T16:55:36.516515+0000 mgr.y (mgr.14146) 2142 : cluster [DBG] pgmap v3149: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:37.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:37 smithi121 bash[18489]: audit 2024-02-12T16:55:37.276643+0000 mon.a (mon.0) 3715 : audit [DBG] from='client.? v1:172.21.15.121:0/3023280926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-12T16:55:38.006 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482286245 got 356482286267 for osd.10 2024-02-12T16:55:38.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-02-12T16:55:38.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:37 smithi169 bash[19604]: cluster 2024-02-12T16:55:36.516515+0000 mgr.y (mgr.14146) 2142 : cluster [DBG] pgmap v3149: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:38.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:37 smithi169 bash[19604]: audit 2024-02-12T16:55:37.276643+0000 mon.a (mon.0) 3715 : audit [DBG] from='client.? v1:172.21.15.121:0/3023280926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-12T16:55:39.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:39 smithi123 bash[19119]: cluster 2024-02-12T16:55:38.517226+0000 mgr.y (mgr.14146) 2143 : cluster [DBG] pgmap v3150: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:39.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:39 smithi123 bash[19119]: cluster 2024-02-12T16:55:38.980806+0000 osd.8 (osd.8) 197 : cluster [DBG] 1.1c deep-scrub starts 2024-02-12T16:55:39.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:39 smithi123 bash[19119]: cluster 2024-02-12T16:55:39.052997+0000 osd.8 (osd.8) 198 : cluster [DBG] 1.1c deep-scrub ok 2024-02-12T16:55:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:39 smithi121 bash[18489]: cluster 2024-02-12T16:55:38.517226+0000 mgr.y (mgr.14146) 2143 : cluster [DBG] pgmap v3150: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:39 smithi121 bash[18489]: cluster 2024-02-12T16:55:38.980806+0000 osd.8 (osd.8) 197 : cluster [DBG] 1.1c deep-scrub starts 2024-02-12T16:55:39.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:39 smithi121 bash[18489]: cluster 2024-02-12T16:55:39.052997+0000 osd.8 (osd.8) 198 : cluster [DBG] 1.1c deep-scrub ok 2024-02-12T16:55:40.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:39 smithi169 bash[19604]: cluster 2024-02-12T16:55:38.517226+0000 mgr.y (mgr.14146) 2143 : cluster [DBG] pgmap v3150: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:39 smithi169 bash[19604]: cluster 2024-02-12T16:55:38.980806+0000 osd.8 (osd.8) 197 : cluster [DBG] 1.1c deep-scrub starts 2024-02-12T16:55:40.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:39 smithi169 bash[19604]: cluster 2024-02-12T16:55:39.052997+0000 osd.8 (osd.8) 198 : cluster [DBG] 1.1c deep-scrub ok 2024-02-12T16:55:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:40 smithi123 bash[19119]: audit 2024-02-12T16:55:40.521858+0000 mon.a (mon.0) 3716 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:55:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:40 smithi123 bash[19119]: audit 2024-02-12T16:55:40.522266+0000 mon.a (mon.0) 3717 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:55:40.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:40 smithi123 bash[19119]: audit 2024-02-12T16:55:40.522632+0000 mon.a (mon.0) 3718 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:55:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:40 smithi123 bash[19119]: audit 2024-02-12T16:55:40.571895+0000 mon.a (mon.0) 3719 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:40 smithi123 bash[19119]: audit 2024-02-12T16:55:40.576048+0000 mon.a (mon.0) 3720 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:40 smithi123 bash[19119]: audit 2024-02-12T16:55:40.577716+0000 mon.a (mon.0) 3721 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:40.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:40 smithi123 bash[19119]: audit 2024-02-12T16:55:40.586677+0000 mon.a (mon.0) 3722 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:55:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:40 smithi121 bash[18489]: audit 2024-02-12T16:55:40.521858+0000 mon.a (mon.0) 3716 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:55:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:40 smithi121 bash[18489]: audit 2024-02-12T16:55:40.522266+0000 mon.a (mon.0) 3717 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:55:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:40 smithi121 bash[18489]: audit 2024-02-12T16:55:40.522632+0000 mon.a (mon.0) 3718 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:55:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:40 smithi121 bash[18489]: audit 2024-02-12T16:55:40.571895+0000 mon.a (mon.0) 3719 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:40 smithi121 bash[18489]: audit 2024-02-12T16:55:40.576048+0000 mon.a (mon.0) 3720 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:40 smithi121 bash[18489]: audit 2024-02-12T16:55:40.577716+0000 mon.a (mon.0) 3721 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:40.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:40 smithi121 bash[18489]: audit 2024-02-12T16:55:40.586677+0000 mon.a (mon.0) 3722 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:55:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:40 smithi169 bash[19604]: audit 2024-02-12T16:55:40.521858+0000 mon.a (mon.0) 3716 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-12T16:55:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:40 smithi169 bash[19604]: audit 2024-02-12T16:55:40.522266+0000 mon.a (mon.0) 3717 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-12T16:55:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:40 smithi169 bash[19604]: audit 2024-02-12T16:55:40.522632+0000 mon.a (mon.0) 3718 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-12T16:55:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:40 smithi169 bash[19604]: audit 2024-02-12T16:55:40.571895+0000 mon.a (mon.0) 3719 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-12T16:55:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:40 smithi169 bash[19604]: audit 2024-02-12T16:55:40.576048+0000 mon.a (mon.0) 3720 : audit [DBG] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-12T16:55:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:40 smithi169 bash[19604]: audit 2024-02-12T16:55:40.577716+0000 mon.a (mon.0) 3721 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-12T16:55:41.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:40 smithi169 bash[19604]: audit 2024-02-12T16:55:40.586677+0000 mon.a (mon.0) 3722 : audit [INF] from='mgr.14146 v1:172.21.15.121:0/429257444' entity='mgr.y' 2024-02-12T16:55:42.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:42 smithi169 bash[19604]: cluster 2024-02-12T16:55:40.660869+0000 osd.10 (osd.10) 324 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:55:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:42 smithi169 bash[19604]: cluster 2024-02-12T16:55:40.729399+0000 osd.10 (osd.10) 325 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:55:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:42 smithi169 bash[19604]: cluster 2024-02-12T16:55:41.615511+0000 osd.10 (osd.10) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:42 smithi169 bash[19604]: cluster 2024-02-12T16:55:41.615597+0000 osd.10 (osd.10) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:42.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:42 smithi169 bash[19604]: cluster 2024-02-12T16:55:41.740033+0000 osd.3 (osd.3) 168 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:55:42.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:42 smithi123 bash[19119]: cluster 2024-02-12T16:55:40.660869+0000 osd.10 (osd.10) 324 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:55:42.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:42 smithi123 bash[19119]: cluster 2024-02-12T16:55:40.729399+0000 osd.10 (osd.10) 325 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:55:42.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:42 smithi123 bash[19119]: cluster 2024-02-12T16:55:41.615511+0000 osd.10 (osd.10) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:42.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:42 smithi123 bash[19119]: cluster 2024-02-12T16:55:41.615597+0000 osd.10 (osd.10) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:42.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:42 smithi123 bash[19119]: cluster 2024-02-12T16:55:41.740033+0000 osd.3 (osd.3) 168 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:55:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:42 smithi121 bash[18489]: cluster 2024-02-12T16:55:40.660869+0000 osd.10 (osd.10) 324 : cluster [DBG] 1.1 deep-scrub starts 2024-02-12T16:55:42.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:42 smithi121 bash[18489]: cluster 2024-02-12T16:55:40.729399+0000 osd.10 (osd.10) 325 : cluster [DBG] 1.1 deep-scrub ok 2024-02-12T16:55:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:42 smithi121 bash[18489]: cluster 2024-02-12T16:55:41.615511+0000 osd.10 (osd.10) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:42 smithi121 bash[18489]: cluster 2024-02-12T16:55:41.615597+0000 osd.10 (osd.10) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:42.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:42 smithi121 bash[18489]: cluster 2024-02-12T16:55:41.740033+0000 osd.3 (osd.3) 168 : cluster [DBG] 1.3 scrub starts 2024-02-12T16:55:42.807 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:43.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:43 smithi169 bash[19604]: cluster 2024-02-12T16:55:41.884811+0000 osd.3 (osd.3) 169 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:55:43.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:43 smithi169 bash[19604]: cluster 2024-02-12T16:55:42.267094+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e1141: 12 total, 12 up, 12 in 2024-02-12T16:55:43.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:43 smithi169 bash[19604]: cluster 2024-02-12T16:55:42.518654+0000 mgr.y (mgr.14146) 2145 : cluster [DBG] pgmap v3153: 31 pgs: 1 active+clean+scrubbing+deep, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:43.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:43 smithi123 bash[19119]: cluster 2024-02-12T16:55:41.884811+0000 osd.3 (osd.3) 169 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:55:43.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:43 smithi123 bash[19119]: cluster 2024-02-12T16:55:42.267094+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e1141: 12 total, 12 up, 12 in 2024-02-12T16:55:43.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:43 smithi123 bash[19119]: cluster 2024-02-12T16:55:42.518654+0000 mgr.y (mgr.14146) 2145 : cluster [DBG] pgmap v3153: 31 pgs: 1 active+clean+scrubbing+deep, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:43 smithi121 bash[18489]: cluster 2024-02-12T16:55:41.884811+0000 osd.3 (osd.3) 169 : cluster [DBG] 1.3 scrub ok 2024-02-12T16:55:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:43 smithi121 bash[18489]: cluster 2024-02-12T16:55:42.267094+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e1141: 12 total, 12 up, 12 in 2024-02-12T16:55:43.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:43 smithi121 bash[18489]: cluster 2024-02-12T16:55:42.518654+0000 mgr.y (mgr.14146) 2145 : cluster [DBG] pgmap v3153: 31 pgs: 1 active+clean+scrubbing+deep, 30 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:44.470 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:44 smithi121 bash[18489]: cluster 2024-02-12T16:55:43.882260+0000 osd.6 (osd.6) 232 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:55:44.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:44 smithi169 bash[19604]: cluster 2024-02-12T16:55:43.882260+0000 osd.6 (osd.6) 232 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:55:44.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:44 smithi123 bash[19119]: cluster 2024-02-12T16:55:43.882260+0000 osd.6 (osd.6) 232 : cluster [DBG] 1.7 scrub starts 2024-02-12T16:55:45.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:45 smithi169 bash[19604]: cluster 2024-02-12T16:55:44.015392+0000 osd.6 (osd.6) 233 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:55:45.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:45 smithi169 bash[19604]: cluster 2024-02-12T16:55:44.519427+0000 mgr.y (mgr.14146) 2146 : cluster [DBG] pgmap v3154: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:45.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:45 smithi123 bash[19119]: cluster 2024-02-12T16:55:44.015392+0000 osd.6 (osd.6) 233 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:55:45.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:45 smithi123 bash[19119]: cluster 2024-02-12T16:55:44.519427+0000 mgr.y (mgr.14146) 2146 : cluster [DBG] pgmap v3154: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:45.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:45 smithi121 bash[18489]: cluster 2024-02-12T16:55:44.015392+0000 osd.6 (osd.6) 233 : cluster [DBG] 1.7 scrub ok 2024-02-12T16:55:45.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:45 smithi121 bash[18489]: cluster 2024-02-12T16:55:44.519427+0000 mgr.y (mgr.14146) 2146 : cluster [DBG] pgmap v3154: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:46.645 INFO:teuthology.orchestra.run.smithi121.stdout:399431959220 2024-02-12T16:55:47.425 INFO:tasks.cephadm.ceph_manager.ceph:need seq 399431959199 got 399431959220 for osd.11 2024-02-12T16:55:47.425 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-12T16:55:47.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:55:47.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:47 smithi169 bash[19604]: cluster 2024-02-12T16:55:46.520105+0000 mgr.y (mgr.14146) 2147 : cluster [DBG] pgmap v3155: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:47.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:47 smithi169 bash[19604]: audit 2024-02-12T16:55:46.639463+0000 mon.a (mon.0) 3724 : audit [DBG] from='client.? v1:172.21.15.121:0/4066735598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-12T16:55:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:47 smithi123 bash[19119]: cluster 2024-02-12T16:55:46.520105+0000 mgr.y (mgr.14146) 2147 : cluster [DBG] pgmap v3155: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:47.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:47 smithi123 bash[19119]: audit 2024-02-12T16:55:46.639463+0000 mon.a (mon.0) 3724 : audit [DBG] from='client.? v1:172.21.15.121:0/4066735598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-12T16:55:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:47 smithi121 bash[18489]: cluster 2024-02-12T16:55:46.520105+0000 mgr.y (mgr.14146) 2147 : cluster [DBG] pgmap v3155: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:47.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:47 smithi121 bash[18489]: audit 2024-02-12T16:55:46.639463+0000 mon.a (mon.0) 3724 : audit [DBG] from='client.? v1:172.21.15.121:0/4066735598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-12T16:55:49.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:49 smithi123 bash[19119]: cluster 2024-02-12T16:55:48.520884+0000 mgr.y (mgr.14146) 2148 : cluster [DBG] pgmap v3156: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:49.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:49 smithi121 bash[18489]: cluster 2024-02-12T16:55:48.520884+0000 mgr.y (mgr.14146) 2148 : cluster [DBG] pgmap v3156: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:50.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:49 smithi169 bash[19604]: cluster 2024-02-12T16:55:48.520884+0000 mgr.y (mgr.14146) 2148 : cluster [DBG] pgmap v3156: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:51 smithi123 bash[19119]: cluster 2024-02-12T16:55:50.521674+0000 mgr.y (mgr.14146) 2149 : cluster [DBG] pgmap v3157: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:51 smithi123 bash[19119]: cluster 2024-02-12T16:55:50.963595+0000 osd.9 (osd.9) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:51.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:51 smithi123 bash[19119]: cluster 2024-02-12T16:55:50.963685+0000 osd.9 (osd.9) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:51 smithi121 bash[18489]: cluster 2024-02-12T16:55:50.521674+0000 mgr.y (mgr.14146) 2149 : cluster [DBG] pgmap v3157: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:51.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:51 smithi121 bash[18489]: cluster 2024-02-12T16:55:50.963595+0000 osd.9 (osd.9) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:51.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:51 smithi121 bash[18489]: cluster 2024-02-12T16:55:50.963685+0000 osd.9 (osd.9) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:51 smithi169 bash[19604]: cluster 2024-02-12T16:55:50.521674+0000 mgr.y (mgr.14146) 2149 : cluster [DBG] pgmap v3157: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:51 smithi169 bash[19604]: cluster 2024-02-12T16:55:50.963595+0000 osd.9 (osd.9) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:52.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:51 smithi169 bash[19604]: cluster 2024-02-12T16:55:50.963685+0000 osd.9 (osd.9) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:52.223 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:55:52.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:52 smithi123 bash[19119]: cluster 2024-02-12T16:55:51.587674+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e1142: 12 total, 12 up, 12 in 2024-02-12T16:55:52.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:52 smithi121 bash[18489]: cluster 2024-02-12T16:55:51.587674+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e1142: 12 total, 12 up, 12 in 2024-02-12T16:55:53.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:52 smithi169 bash[19604]: cluster 2024-02-12T16:55:51.587674+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e1142: 12 total, 12 up, 12 in 2024-02-12T16:55:53.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:53 smithi121 bash[18489]: cluster 2024-02-12T16:55:52.522406+0000 mgr.y (mgr.14146) 2150 : cluster [DBG] pgmap v3159: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:53.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:53 smithi123 bash[19119]: cluster 2024-02-12T16:55:52.522406+0000 mgr.y (mgr.14146) 2150 : cluster [DBG] pgmap v3159: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:54.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:53 smithi169 bash[19604]: cluster 2024-02-12T16:55:52.522406+0000 mgr.y (mgr.14146) 2150 : cluster [DBG] pgmap v3159: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:54.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:54 smithi123 bash[19119]: cluster 2024-02-12T16:55:53.595654+0000 osd.7 (osd.7) 101 : cluster [DBG] 1.d scrub starts 2024-02-12T16:55:54.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:54 smithi123 bash[19119]: cluster 2024-02-12T16:55:53.698122+0000 osd.7 (osd.7) 102 : cluster [DBG] 1.d scrub ok 2024-02-12T16:55:54.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:54 smithi121 bash[18489]: cluster 2024-02-12T16:55:53.595654+0000 osd.7 (osd.7) 101 : cluster [DBG] 1.d scrub starts 2024-02-12T16:55:54.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:54 smithi121 bash[18489]: cluster 2024-02-12T16:55:53.698122+0000 osd.7 (osd.7) 102 : cluster [DBG] 1.d scrub ok 2024-02-12T16:55:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:54 smithi169 bash[19604]: cluster 2024-02-12T16:55:53.595654+0000 osd.7 (osd.7) 101 : cluster [DBG] 1.d scrub starts 2024-02-12T16:55:55.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:54 smithi169 bash[19604]: cluster 2024-02-12T16:55:53.698122+0000 osd.7 (osd.7) 102 : cluster [DBG] 1.d scrub ok 2024-02-12T16:55:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:55 smithi123 bash[19119]: cluster 2024-02-12T16:55:54.268305+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:55 smithi123 bash[19119]: cluster 2024-02-12T16:55:54.268414+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:55.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:55 smithi123 bash[19119]: cluster 2024-02-12T16:55:54.366132+0000 osd.4 (osd.4) 225 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:55:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:55 smithi123 bash[19119]: cluster 2024-02-12T16:55:54.416592+0000 osd.4 (osd.4) 226 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:55:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:55 smithi123 bash[19119]: cluster 2024-02-12T16:55:54.523184+0000 mgr.y (mgr.14146) 2151 : cluster [DBG] pgmap v3160: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:55.889 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:55 smithi123 bash[19119]: cluster 2024-02-12T16:55:54.612000+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e1143: 12 total, 12 up, 12 in 2024-02-12T16:55:55.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:55 smithi121 bash[18489]: cluster 2024-02-12T16:55:54.268305+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:55.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:55 smithi121 bash[18489]: cluster 2024-02-12T16:55:54.268414+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:55.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:55 smithi121 bash[18489]: cluster 2024-02-12T16:55:54.366132+0000 osd.4 (osd.4) 225 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:55:55.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:55 smithi121 bash[18489]: cluster 2024-02-12T16:55:54.416592+0000 osd.4 (osd.4) 226 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:55:55.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:55 smithi121 bash[18489]: cluster 2024-02-12T16:55:54.523184+0000 mgr.y (mgr.14146) 2151 : cluster [DBG] pgmap v3160: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:55.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:55 smithi121 bash[18489]: cluster 2024-02-12T16:55:54.612000+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e1143: 12 total, 12 up, 12 in 2024-02-12T16:55:55.919 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:55:55.920 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:55:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:55 smithi169 bash[19604]: cluster 2024-02-12T16:55:54.268305+0000 osd.5 (osd.5) 285 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:56.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:55 smithi169 bash[19604]: cluster 2024-02-12T16:55:54.268414+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:55 smithi169 bash[19604]: cluster 2024-02-12T16:55:54.366132+0000 osd.4 (osd.4) 225 : cluster [DBG] 1.10 scrub starts 2024-02-12T16:55:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:55 smithi169 bash[19604]: cluster 2024-02-12T16:55:54.416592+0000 osd.4 (osd.4) 226 : cluster [DBG] 1.10 scrub ok 2024-02-12T16:55:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:55 smithi169 bash[19604]: cluster 2024-02-12T16:55:54.523184+0000 mgr.y (mgr.14146) 2151 : cluster [DBG] pgmap v3160: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:56.048 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:55 smithi169 bash[19604]: cluster 2024-02-12T16:55:54.612000+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e1143: 12 total, 12 up, 12 in 2024-02-12T16:55:56.669 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":3161,"stamp":"2024-02-12T16:55:54.609497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10197372,"kb_used_data":9180,"kb_used_omap":92,"kb_used_meta":10188003,"kb_avail":1114744452,"statfs":{"total":1151940427776,"available":1141498318848,"internally_reserved":0,"allocated":9400320,"data_stored":4344645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95164,"internal_metadata":10432515140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13402112,"num_objects":-206,"num_object_clones":0,"num_object_copies":-412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-46,"num_objects_unfound":0,"num_objects_dirty":-206,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-292936,"num_write_kb":-9349025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-20060,"num_bytes_recovered":-1314553856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.320132"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":533,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.117107+0000","last_change":"2024-02-12T16:54:41.657969+0000","last_active":"2024-02-12T16:55:23.117107+0000","last_peered":"2024-02-12T16:55:23.117107+0000","last_clean":"2024-02-12T16:55:23.117107+0000","last_became_active":"2024-02-12T16:41:18.947003+0000","last_became_peered":"2024-02-12T16:41:18.947003+0000","last_unstale":"2024-02-12T16:55:23.117107+0000","last_undegraded":"2024-02-12T16:55:23.117107+0000","last_fullsized":"2024-02-12T16:55:23.117107+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:41.657854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:15.328480+0000","last_clean_scrub_stamp":"2024-02-12T16:54:41.657854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":568,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.113646+0000","last_change":"2024-02-12T16:55:06.196564+0000","last_active":"2024-02-12T16:55:23.113646+0000","last_peered":"2024-02-12T16:55:23.113646+0000","last_clean":"2024-02-12T16:55:23.113646+0000","last_became_active":"2024-02-12T16:41:17.943883+0000","last_became_peered":"2024-02-12T16:41:17.943883+0000","last_unstale":"2024-02-12T16:55:23.113646+0000","last_undegraded":"2024-02-12T16:55:23.113646+0000","last_fullsized":"2024-02-12T16:55:23.113646+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:06.196437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:49.771537+0000","last_clean_scrub_stamp":"2024-02-12T16:55:06.196437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":575,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.269777+0000","last_change":"2024-02-12T16:55:39.053092+0000","last_active":"2024-02-12T16:55:42.269777+0000","last_peered":"2024-02-12T16:55:42.269777+0000","last_clean":"2024-02-12T16:55:42.269777+0000","last_became_active":"2024-02-12T16:47:31.213335+0000","last_became_peered":"2024-02-12T16:47:31.213335+0000","last_unstale":"2024-02-12T16:55:42.269777+0000","last_undegraded":"2024-02-12T16:55:42.269777+0000","last_fullsized":"2024-02-12T16:55:42.269777+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:39.053002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:39.053002+0000","last_clean_scrub_stamp":"2024-02-12T16:55:39.053002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":375,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.036522+0000","last_change":"2024-02-12T16:54:50.856266+0000","last_active":"2024-02-12T16:55:43.036522+0000","last_peered":"2024-02-12T16:55:43.036522+0000","last_clean":"2024-02-12T16:55:43.036522+0000","last_became_active":"2024-02-12T16:39:09.436840+0000","last_became_peered":"2024-02-12T16:39:09.436840+0000","last_unstale":"2024-02-12T16:55:43.036522+0000","last_undegraded":"2024-02-12T16:55:43.036522+0000","last_fullsized":"2024-02-12T16:55:43.036522+0000","mapping_epoch":845,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":846,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:50.856141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_clean_scrub_stamp":"2024-02-12T16:54:50.856141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":606,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.270884+0000","last_change":"2024-02-12T16:55:25.306208+0000","last_active":"2024-02-12T16:55:42.270884+0000","last_peered":"2024-02-12T16:55:42.270884+0000","last_clean":"2024-02-12T16:55:42.270884+0000","last_became_active":"2024-02-12T16:42:40.975420+0000","last_became_peered":"2024-02-12T16:42:40.975420+0000","last_unstale":"2024-02-12T16:55:42.270884+0000","last_undegraded":"2024-02-12T16:55:42.270884+0000","last_fullsized":"2024-02-12T16:55:42.270884+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:25.306108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.890642+0000","last_clean_scrub_stamp":"2024-02-12T16:55:25.306108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":591,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.036540+0000","last_change":"2024-02-12T16:55:11.862493+0000","last_active":"2024-02-12T16:55:43.036540+0000","last_peered":"2024-02-12T16:55:43.036540+0000","last_clean":"2024-02-12T16:55:43.036540+0000","last_became_active":"2024-02-12T16:48:02.220402+0000","last_became_peered":"2024-02-12T16:48:02.220402+0000","last_unstale":"2024-02-12T16:55:43.036540+0000","last_undegraded":"2024-02-12T16:55:43.036540+0000","last_fullsized":"2024-02-12T16:55:43.036540+0000","mapping_epoch":1044,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:11.862304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:55.445363+0000","last_clean_scrub_stamp":"2024-02-12T16:55:11.862304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":910,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.745710+0000","last_change":"2024-02-12T16:54:40.059795+0000","last_active":"2024-02-12T16:55:42.745710+0000","last_peered":"2024-02-12T16:55:42.745710+0000","last_clean":"2024-02-12T16:55:42.745710+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:55:42.745710+0000","last_undegraded":"2024-02-12T16:55:42.745710+0000","last_fullsized":"2024-02-12T16:55:42.745710+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:40.059694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:48:46.968263+0000","last_clean_scrub_stamp":"2024-02-12T16:54:40.059694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":558,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.269903+0000","last_change":"2024-02-12T16:55:18.076770+0000","last_active":"2024-02-12T16:55:42.269903+0000","last_peered":"2024-02-12T16:55:42.269903+0000","last_clean":"2024-02-12T16:55:42.269903+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:55:42.269903+0000","last_undegraded":"2024-02-12T16:55:42.269903+0000","last_fullsized":"2024-02-12T16:55:42.269903+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:18.076678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:55:18.076678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":552,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.113585+0000","last_change":"2024-02-12T16:54:59.980562+0000","last_active":"2024-02-12T16:55:23.113585+0000","last_peered":"2024-02-12T16:55:23.113585+0000","last_clean":"2024-02-12T16:55:23.113585+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:55:23.113585+0000","last_undegraded":"2024-02-12T16:55:23.113585+0000","last_fullsized":"2024-02-12T16:55:23.113585+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:59.980467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:06.911728+0000","last_clean_scrub_stamp":"2024-02-12T16:54:59.980467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":553,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.134427+0000","last_change":"2024-02-12T16:55:16.833161+0000","last_active":"2024-02-12T16:55:43.134427+0000","last_peered":"2024-02-12T16:55:43.134427+0000","last_clean":"2024-02-12T16:55:43.134427+0000","last_became_active":"2024-02-12T16:49:21.796837+0000","last_became_peered":"2024-02-12T16:49:21.796837+0000","last_unstale":"2024-02-12T16:55:43.134427+0000","last_undegraded":"2024-02-12T16:55:43.134427+0000","last_fullsized":"2024-02-12T16:55:43.134427+0000","mapping_epoch":1068,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":1069,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:16.833043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:31.160930+0000","last_clean_scrub_stamp":"2024-02-12T16:55:16.833043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":818,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.129599+0000","last_change":"2024-02-12T16:55:13.864823+0000","last_active":"2024-02-12T16:55:43.129599+0000","last_peered":"2024-02-12T16:55:43.129599+0000","last_clean":"2024-02-12T16:55:43.129599+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:55:43.129599+0000","last_undegraded":"2024-02-12T16:55:43.129599+0000","last_fullsized":"2024-02-12T16:55:43.129599+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:13.864693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:09.303752+0000","last_clean_scrub_stamp":"2024-02-12T16:55:13.864693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":297,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.134182+0000","last_change":"2024-02-12T16:55:26.697755+0000","last_active":"2024-02-12T16:55:43.134182+0000","last_peered":"2024-02-12T16:55:43.134182+0000","last_clean":"2024-02-12T16:55:43.134182+0000","last_became_active":"2024-02-12T16:42:41.980413+0000","last_became_peered":"2024-02-12T16:42:41.980413+0000","last_unstale":"2024-02-12T16:55:43.134182+0000","last_undegraded":"2024-02-12T16:55:43.134182+0000","last_fullsized":"2024-02-12T16:55:43.134182+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:26.697620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:26.697620+0000","last_clean_scrub_stamp":"2024-02-12T16:55:26.697620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":976,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.036364+0000","last_change":"2024-02-12T16:55:18.805323+0000","last_active":"2024-02-12T16:55:43.036364+0000","last_peered":"2024-02-12T16:55:43.036364+0000","last_clean":"2024-02-12T16:55:43.036364+0000","last_became_active":"2024-02-12T16:48:42.347481+0000","last_became_peered":"2024-02-12T16:48:42.347481+0000","last_unstale":"2024-02-12T16:55:43.036364+0000","last_undegraded":"2024-02-12T16:55:43.036364+0000","last_fullsized":"2024-02-12T16:55:43.036364+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:18.805222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:55:18.805222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1285,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.117059+0000","last_change":"2024-02-12T16:55:17.625772+0000","last_active":"2024-02-12T16:55:23.117059+0000","last_peered":"2024-02-12T16:55:23.117059+0000","last_clean":"2024-02-12T16:55:23.117059+0000","last_became_active":"2024-02-12T16:41:18.947424+0000","last_became_peered":"2024-02-12T16:41:18.947424+0000","last_unstale":"2024-02-12T16:55:23.117059+0000","last_undegraded":"2024-02-12T16:55:23.117059+0000","last_fullsized":"2024-02-12T16:55:23.117059+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:17.625656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:17.625656+0000","last_clean_scrub_stamp":"2024-02-12T16:55:17.625656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":147,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:44.015487+0000","last_change":"2024-02-12T16:55:44.015487+0000","last_active":"2024-02-12T16:55:44.015487+0000","last_peered":"2024-02-12T16:55:44.015487+0000","last_clean":"2024-02-12T16:55:44.015487+0000","last_became_active":"2024-02-12T16:47:31.216533+0000","last_became_peered":"2024-02-12T16:47:31.216533+0000","last_unstale":"2024-02-12T16:55:44.015487+0000","last_undegraded":"2024-02-12T16:55:44.015487+0000","last_fullsized":"2024-02-12T16:55:44.015487+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:44.015397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:58.274918+0000","last_clean_scrub_stamp":"2024-02-12T16:55:44.015397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":922,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.036503+0000","last_change":"2024-02-12T16:55:29.819766+0000","last_active":"2024-02-12T16:55:43.036503+0000","last_peered":"2024-02-12T16:55:43.036503+0000","last_clean":"2024-02-12T16:55:43.036503+0000","last_became_active":"2024-02-12T16:47:43.444392+0000","last_became_peered":"2024-02-12T16:47:43.444392+0000","last_unstale":"2024-02-12T16:55:43.036503+0000","last_undegraded":"2024-02-12T16:55:43.036503+0000","last_fullsized":"2024-02-12T16:55:43.036503+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:29.819662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:29.819662+0000","last_clean_scrub_stamp":"2024-02-12T16:55:29.819662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":965,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.036452+0000","last_change":"2024-02-12T16:55:40.729541+0000","last_active":"2024-02-12T16:55:43.036452+0000","last_peered":"2024-02-12T16:55:43.036452+0000","last_clean":"2024-02-12T16:55:43.036452+0000","last_became_active":"2024-02-12T16:48:20.617116+0000","last_became_peered":"2024-02-12T16:48:20.617116+0000","last_unstale":"2024-02-12T16:55:43.036452+0000","last_undegraded":"2024-02-12T16:55:43.036452+0000","last_fullsized":"2024-02-12T16:55:43.036452+0000","mapping_epoch":1047,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1048,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:40.729408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.729408+0000","last_clean_scrub_stamp":"2024-02-12T16:55:40.729408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":785,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:41.884938+0000","last_change":"2024-02-12T16:55:41.884907+0000","last_active":"2024-02-12T16:55:41.884938+0000","last_peered":"2024-02-12T16:55:41.884938+0000","last_clean":"2024-02-12T16:55:41.884938+0000","last_became_active":"2024-02-12T16:47:31.215614+0000","last_became_peered":"2024-02-12T16:47:31.215614+0000","last_unstale":"2024-02-12T16:55:41.884938+0000","last_undegraded":"2024-02-12T16:55:41.884938+0000","last_fullsized":"2024-02-12T16:55:41.884938+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:41.884816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:46:39.182350+0000","last_clean_scrub_stamp":"2024-02-12T16:55:41.884816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1249,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.134250+0000","last_change":"2024-02-12T16:54:43.757481+0000","last_active":"2024-02-12T16:55:43.134250+0000","last_peered":"2024-02-12T16:55:43.134250+0000","last_clean":"2024-02-12T16:55:43.134250+0000","last_became_active":"2024-02-12T16:42:41.321180+0000","last_became_peered":"2024-02-12T16:42:41.321180+0000","last_unstale":"2024-02-12T16:55:43.134250+0000","last_undegraded":"2024-02-12T16:55:43.134250+0000","last_fullsized":"2024-02-12T16:55:43.134250+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:43.757263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:37.472009+0000","last_clean_scrub_stamp":"2024-02-12T16:54:43.757263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":673,"reported_epoch":1142,"state":"active+clean","last_fresh":"2024-02-12T16:55:53.698219+0000","last_change":"2024-02-12T16:55:53.698219+0000","last_active":"2024-02-12T16:55:53.698219+0000","last_peered":"2024-02-12T16:55:53.698219+0000","last_clean":"2024-02-12T16:55:53.698219+0000","last_became_active":"2024-02-12T16:42:40.975171+0000","last_became_peered":"2024-02-12T16:42:40.975171+0000","last_unstale":"2024-02-12T16:55:53.698219+0000","last_undegraded":"2024-02-12T16:55:53.698219+0000","last_fullsized":"2024-02-12T16:55:53.698219+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:53.698127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:55:53.698127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":69,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.113551+0000","last_change":"2024-02-12T16:55:15.783912+0000","last_active":"2024-02-12T16:55:23.113551+0000","last_peered":"2024-02-12T16:55:23.113551+0000","last_clean":"2024-02-12T16:55:23.113551+0000","last_became_active":"2024-02-12T16:51:20.593832+0000","last_became_peered":"2024-02-12T16:51:20.593832+0000","last_unstale":"2024-02-12T16:55:23.113551+0000","last_undegraded":"2024-02-12T16:55:23.113551+0000","last_fullsized":"2024-02-12T16:55:23.113551+0000","mapping_epoch":1100,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1101,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:15.783819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:23.682328+0000","last_clean_scrub_stamp":"2024-02-12T16:55:15.783819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":142,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.116659+0000","last_change":"2024-02-12T16:55:05.331663+0000","last_active":"2024-02-12T16:55:23.116659+0000","last_peered":"2024-02-12T16:55:23.116659+0000","last_clean":"2024-02-12T16:55:23.116659+0000","last_became_active":"2024-02-12T16:47:31.213215+0000","last_became_peered":"2024-02-12T16:47:31.213215+0000","last_unstale":"2024-02-12T16:55:23.116659+0000","last_undegraded":"2024-02-12T16:55:23.116659+0000","last_fullsized":"2024-02-12T16:55:23.116659+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:05.331571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:56.183999+0000","last_clean_scrub_stamp":"2024-02-12T16:55:05.331571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":722,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.271053+0000","last_change":"2024-02-12T16:55:20.357147+0000","last_active":"2024-02-12T16:55:42.271053+0000","last_peered":"2024-02-12T16:55:42.271053+0000","last_clean":"2024-02-12T16:55:42.271053+0000","last_became_active":"2024-02-12T16:42:40.975493+0000","last_became_peered":"2024-02-12T16:42:40.975493+0000","last_unstale":"2024-02-12T16:55:42.271053+0000","last_undegraded":"2024-02-12T16:55:42.271053+0000","last_fullsized":"2024-02-12T16:55:42.271053+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:20.357026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_clean_scrub_stamp":"2024-02-12T16:55:20.357026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":1244,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:43.134150+0000","last_change":"2024-02-12T16:55:35.692950+0000","last_active":"2024-02-12T16:55:43.134150+0000","last_peered":"2024-02-12T16:55:43.134150+0000","last_clean":"2024-02-12T16:55:43.134150+0000","last_became_active":"2024-02-12T16:47:43.444507+0000","last_became_peered":"2024-02-12T16:47:43.444507+0000","last_unstale":"2024-02-12T16:55:43.134150+0000","last_undegraded":"2024-02-12T16:55:43.134150+0000","last_fullsized":"2024-02-12T16:55:43.134150+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:35.692738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:48:56.268062+0000","last_clean_scrub_stamp":"2024-02-12T16:55:35.692738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":904,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.745683+0000","last_change":"2024-02-12T16:55:32.907642+0000","last_active":"2024-02-12T16:55:42.745683+0000","last_peered":"2024-02-12T16:55:42.745683+0000","last_clean":"2024-02-12T16:55:42.745683+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:55:42.745683+0000","last_undegraded":"2024-02-12T16:55:42.745683+0000","last_fullsized":"2024-02-12T16:55:42.745683+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:32.907539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:32.907539+0000","last_clean_scrub_stamp":"2024-02-12T16:55:32.907539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":298,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.117038+0000","last_change":"2024-02-12T16:55:00.621406+0000","last_active":"2024-02-12T16:55:23.117038+0000","last_peered":"2024-02-12T16:55:23.117038+0000","last_clean":"2024-02-12T16:55:23.117038+0000","last_became_active":"2024-02-12T16:41:17.958639+0000","last_became_peered":"2024-02-12T16:41:17.958639+0000","last_unstale":"2024-02-12T16:55:23.117038+0000","last_undegraded":"2024-02-12T16:55:23.117038+0000","last_fullsized":"2024-02-12T16:55:23.117038+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:00.621284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:00.621284+0000","last_clean_scrub_stamp":"2024-02-12T16:55:00.621284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":143,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.116667+0000","last_change":"2024-02-12T16:55:14.754945+0000","last_active":"2024-02-12T16:55:23.116667+0000","last_peered":"2024-02-12T16:55:23.116667+0000","last_clean":"2024-02-12T16:55:23.116667+0000","last_became_active":"2024-02-12T16:47:43.439741+0000","last_became_peered":"2024-02-12T16:47:43.439741+0000","last_unstale":"2024-02-12T16:55:23.116667+0000","last_undegraded":"2024-02-12T16:55:23.116667+0000","last_fullsized":"2024-02-12T16:55:23.116667+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:14.754852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:36.632610+0000","last_clean_scrub_stamp":"2024-02-12T16:55:14.754852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":477,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.270995+0000","last_change":"2024-02-12T16:54:29.565844+0000","last_active":"2024-02-12T16:55:42.270995+0000","last_peered":"2024-02-12T16:55:42.270995+0000","last_clean":"2024-02-12T16:55:42.270995+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:55:42.270995+0000","last_undegraded":"2024-02-12T16:55:42.270995+0000","last_fullsized":"2024-02-12T16:55:42.270995+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:29.565737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:48:33.063109+0000","last_clean_scrub_stamp":"2024-02-12T16:54:29.565737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":135,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:22.690331+0000","last_change":"2024-02-12T16:55:13.998488+0000","last_active":"2024-02-12T16:55:22.690331+0000","last_peered":"2024-02-12T16:55:22.690331+0000","last_clean":"2024-02-12T16:55:22.690331+0000","last_became_active":"2024-02-12T16:48:02.884693+0000","last_became_peered":"2024-02-12T16:48:02.884693+0000","last_unstale":"2024-02-12T16:55:22.690331+0000","last_undegraded":"2024-02-12T16:55:22.690331+0000","last_fullsized":"2024-02-12T16:55:22.690331+0000","mapping_epoch":1044,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:13.998294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:13.998294+0000","last_clean_scrub_stamp":"2024-02-12T16:55:13.998294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":793,"reported_epoch":1141,"state":"active+clean","last_fresh":"2024-02-12T16:55:42.745439+0000","last_change":"2024-02-12T16:55:09.168990+0000","last_active":"2024-02-12T16:55:42.745439+0000","last_peered":"2024-02-12T16:55:42.745439+0000","last_clean":"2024-02-12T16:55:42.745439+0000","last_became_active":"2024-02-12T16:50:22.212417+0000","last_became_peered":"2024-02-12T16:50:22.212417+0000","last_unstale":"2024-02-12T16:55:42.745439+0000","last_undegraded":"2024-02-12T16:55:42.745439+0000","last_fullsized":"2024-02-12T16:55:42.745439+0000","mapping_epoch":1086,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1087,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:09.168898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:55:09.168898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":735,"reported_epoch":1140,"state":"active+clean","last_fresh":"2024-02-12T16:55:23.113547+0000","last_change":"2024-02-12T16:54:59.072627+0000","last_active":"2024-02-12T16:55:23.113547+0000","last_peered":"2024-02-12T16:55:23.113547+0000","last_clean":"2024-02-12T16:55:23.113547+0000","last_became_active":"2024-02-12T16:41:17.950595+0000","last_became_peered":"2024-02-12T16:41:17.950595+0000","last_unstale":"2024-02-12T16:55:23.113547+0000","last_undegraded":"2024-02-12T16:55:23.113547+0000","last_fullsized":"2024-02-12T16:55:23.113547+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:59.072524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:54:59.072524+0000","last_clean_scrub_stamp":"2024-02-12T16:54:59.072524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431959222,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897600,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":896832,"kb_avail":92847552,"statfs":{"total":95995035648,"available":95075893248,"internally_reserved":0,"allocated":778240,"data_stored":359590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":918355968},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":0.873},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.248},"max":{"1min":1.653,"5min":1.653,"15min":3.9159999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0680000000000001,"15min":1.0920000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.33500000000000002},"max":{"1min":1.1870000000000001,"5min":1.6399999999999999,"15min":5.7009999999999996},"last":0.95599999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:55:05 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.99299999999999999,"15min":0.998},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.58099999999999996},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.6240000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.2330000000000001,"15min":1.1659999999999999},"min":{"1min":0.70999999999999996,"5min":0.68600000000000005,"15min":0.38100000000000001},"max":{"1min":1.5049999999999999,"5min":1.7809999999999999,"15min":2.202},"last":1.1499999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":1.734},"min":{"1min":0.625,"5min":0.60999999999999999,"15min":0.308},"max":{"1min":1.3200000000000001,"5min":1.4910000000000001,"15min":219.56100000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.23,"5min":1.129,"15min":1.7010000000000001},"min":{"1min":0.80000000000000004,"5min":0.65300000000000002,"15min":0.26900000000000002},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":219.79300000000001},"last":1.25}]},{"osd":3,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.113,"15min":1.3899999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.26300000000000001},"max":{"1min":1.45,"5min":2.0979999999999999,"15min":46.445},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.325,"15min":1.444},"min":{"1min":0.80700000000000005,"5min":0.753,"15min":0.28299999999999997},"max":{"1min":1.696,"5min":1.9770000000000001,"15min":46.774999999999999},"last":1.3340000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.93300000000000005,"15min":0.91800000000000004},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.4239999999999999,"5min":6.1669999999999998,"15min":6.2039999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.304,"5min":1.234,"15min":1.121},"min":{"1min":0.48099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.4989999999999997,"5min":5.7469999999999999,"15min":7.3920000000000003},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.359,"15min":1.2669999999999999},"min":{"1min":0.83799999999999997,"5min":0.70599999999999996,"15min":0.37},"max":{"1min":2.1520000000000001,"5min":2.1579999999999999,"15min":5.1280000000000001},"last":1.042},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.167,"15min":1.202},"min":{"1min":0.90400000000000003,"5min":0.77400000000000002,"15min":0.38300000000000001},"max":{"1min":1.405,"5min":1.8009999999999999,"15min":5.8200000000000003},"last":1.099}]},{"osd":6,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.137,"15min":1.143},"min":{"1min":0.80400000000000005,"5min":0.73799999999999999,"15min":0.35199999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":6.202},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.254,"15min":1.2190000000000001},"min":{"1min":0.878,"5min":0.81499999999999995,"15min":0.40600000000000003},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":6.1289999999999996},"last":1.02}]},{"osd":7,"last update":"Mon Feb 12 16:55:05 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3460000000000001,"15min":1.3029999999999999},"min":{"1min":1.0649999999999999,"5min":0.73199999999999998,"15min":0.38200000000000001},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":5.9660000000000002},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5760000000000001,"15min":1.2729999999999999},"min":{"1min":1.2749999999999999,"5min":0.747,"15min":0.41299999999999998},"max":{"1min":1.736,"5min":2.2240000000000002,"15min":5.0739999999999998},"last":1.4590000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.302,"15min":1.224},"min":{"1min":0.998,"5min":0.55900000000000005,"15min":0.32800000000000001},"max":{"1min":2.141,"5min":2.141,"15min":6.306},"last":1.278},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5720000000000001,"15min":1.2969999999999999},"min":{"1min":0.89600000000000002,"5min":0.66100000000000003,"15min":0.35199999999999998},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":3.7530000000000001},"last":1.4019999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.613,"15min":1.3440000000000001},"min":{"1min":1.173,"5min":0.80000000000000004,"15min":0.437},"max":{"1min":2.331,"5min":2.331,"15min":3.1480000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.718,"15min":1.4299999999999999},"min":{"1min":1.421,"5min":1.1100000000000001,"15min":0.47399999999999998},"max":{"1min":2.488,"5min":2.488,"15min":3.21},"last":1.5469999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.371,"15min":1.407},"min":{"1min":1.1479999999999999,"5min":0.99099999999999999,"15min":0.39700000000000002},"max":{"1min":1.583,"5min":1.909,"15min":9.6159999999999997},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.508,"5min":1.516,"15min":1.488},"min":{"1min":1.2430000000000001,"5min":1.054,"15min":0.39500000000000002},"max":{"1min":2.4460000000000002,"5min":2.4460000000000002,"15min":8.2129999999999992},"last":1.357}]}]},{"osd":10,"up_from":83,"seq":356482286271,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899340,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":898560,"kb_avail":92845812,"statfs":{"total":95995035648,"available":95074111488,"internally_reserved":0,"allocated":790528,"data_stored":365503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":920125440},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:14 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.95099999999999996,"15min":0.97399999999999998},"min":{"1min":0.755,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.466,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":0.99199999999999999,"15min":0.99099999999999999},"min":{"1min":0.79600000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.359,"5min":1.8129999999999999,"15min":1.863},"last":0.95099999999999996}]},{"osd":1,"last update":"Mon Feb 12 16:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.85699999999999998,"15min":1.044},"min":{"1min":0.44,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2909999999999999,"5min":5.524,"15min":5.524},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0549999999999999,"15min":1.1060000000000001},"min":{"1min":0.61199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.542,"5min":5.5609999999999999,"15min":7.2000000000000002},"last":1.3220000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.94499999999999995,"15min":0.96499999999999997},"min":{"1min":0.69599999999999995,"5min":0.55100000000000005,"15min":0.314},"max":{"1min":1.385,"5min":1.544,"15min":9.4179999999999993},"last":1.325},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94399999999999995,"15min":0.95499999999999996},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.34100000000000003},"max":{"1min":1.169,"5min":1.4890000000000001,"15min":5.6289999999999996},"last":0.88300000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.008,"15min":1.042},"min":{"1min":0.71199999999999997,"5min":0.68000000000000005,"15min":0.28599999999999998},"max":{"1min":1.4339999999999999,"5min":1.4710000000000001,"15min":9.3699999999999992},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1299999999999999,"15min":1.073},"min":{"1min":0.84399999999999997,"5min":0.67100000000000004,"15min":0.28699999999999998},"max":{"1min":1.673,"5min":1.675,"15min":5.6769999999999996},"last":1.3819999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:54:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.294,"15min":1.1950000000000001},"min":{"1min":0.998,"5min":0.68300000000000005,"15min":0.32500000000000001},"max":{"1min":1.601,"5min":2.1320000000000001,"15min":5.8380000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.415,"15min":1.2250000000000001},"min":{"1min":1.0529999999999999,"5min":0.89200000000000002,"15min":0.38600000000000001},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":5.649},"last":1.5800000000000001}]},{"osd":5,"last update":"Mon Feb 12 16:54:57 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.2569999999999999,"15min":1.468},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.38800000000000001},"max":{"1min":1.4259999999999999,"5min":1.9510000000000001,"15min":74.137},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.3360000000000001,"15min":1.4950000000000001},"min":{"1min":0.97799999999999998,"5min":0.78800000000000003,"15min":0.44},"max":{"1min":1.802,"5min":1.835,"15min":74.063999999999993},"last":1.22}]},{"osd":6,"last update":"Mon Feb 12 16:54:57 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.3859999999999999,"15min":1.3080000000000001},"min":{"1min":1.0640000000000001,"5min":0.874,"15min":0.38400000000000001},"max":{"1min":1.7250000000000001,"5min":2.0219999999999998,"15min":5.7729999999999997},"last":1.29},{"interface":"front","average":{"1min":1.542,"5min":1.5580000000000001,"15min":1.389},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.44800000000000001},"max":{"1min":1.972,"5min":2.0840000000000001,"15min":5.7960000000000003},"last":1.6519999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:54:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4419999999999999,"15min":2.4169999999999998},"min":{"1min":1.111,"5min":0.996,"15min":0.45000000000000001},"max":{"1min":2.0430000000000001,"5min":2.0659999999999998,"15min":74.007000000000005},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.323,"15min":2.21},"min":{"1min":1.0720000000000001,"5min":0.92600000000000005,"15min":0.42299999999999999},"max":{"1min":1.629,"5min":1.8169999999999999,"15min":73.819000000000003},"last":1.5580000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:54:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.97299999999999998,"5min":0.84599999999999997,"15min":0.34100000000000003},"max":{"1min":1.758,"5min":1.9259999999999999,"15min":7.5810000000000004},"last":1.468},{"interface":"front","average":{"1min":1.589,"5min":1.55,"15min":1.407},"min":{"1min":1.1699999999999999,"5min":0.81899999999999995,"15min":0.41299999999999998},"max":{"1min":2.1379999999999999,"5min":2.2050000000000001,"15min":9.5090000000000003},"last":1.704}]},{"osd":9,"last update":"Mon Feb 12 16:54:57 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.47,"15min":1.345},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.35399999999999998},"max":{"1min":1.8260000000000001,"5min":2.218,"15min":5.8029999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.629,"5min":1.647,"15min":1.4319999999999999},"min":{"1min":1.2170000000000001,"5min":1.1479999999999999,"15min":0.378},"max":{"1min":2.093,"5min":2.2789999999999999,"15min":5.7519999999999998},"last":1.7470000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.4570000000000001,"15min":1.4430000000000001},"min":{"1min":1.2390000000000001,"5min":0.94299999999999995,"15min":0.39900000000000002},"max":{"1min":1.756,"5min":1.8680000000000001,"15min":8.7569999999999997},"last":1.623},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.708,"15min":1.581},"min":{"1min":1.4199999999999999,"5min":1.272,"15min":0.44700000000000001},"max":{"1min":2.1299999999999999,"5min":2.1819999999999999,"15min":7.891},"last":1.788}]}]},{"osd":9,"up_from":74,"seq":317827580615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898432,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":897664,"kb_avail":92846720,"statfs":{"total":95995035648,"available":95075041280,"internally_reserved":0,"allocated":778240,"data_stored":359590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":919207936},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98399999999999999,"15min":0.83499999999999996},"min":{"1min":0.71999999999999997,"5min":0.68600000000000005,"15min":0.25700000000000001},"max":{"1min":1.472,"5min":1.518,"15min":5.1180000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1200000000000001,"15min":0.96199999999999997},"min":{"1min":0.75600000000000001,"5min":0.46800000000000003,"15min":0.28699999999999998},"max":{"1min":1.494,"5min":1.6479999999999999,"15min":5.0039999999999996},"last":1.0449999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2569999999999999,"15min":1.5800000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.29399999999999998},"max":{"1min":1.8979999999999999,"5min":1.919,"15min":157.74100000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.997,"5min":0.96999999999999997,"15min":1.492},"min":{"1min":0.71499999999999997,"5min":0.64900000000000002,"15min":0.317},"max":{"1min":1.327,"5min":1.3700000000000001,"15min":157.709},"last":1.1870000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.079,"15min":0.96399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.311},"max":{"1min":1.4610000000000001,"5min":1.742,"15min":5.0780000000000003},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.272,"5min":1.2569999999999999,"15min":1.0600000000000001},"min":{"1min":0.67400000000000004,"5min":0.52700000000000002,"15min":0.33500000000000002},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":5.4100000000000001},"last":1.2909999999999999}]},{"osd":3,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.1899999999999999,"15min":1.0469999999999999},"min":{"1min":0.82299999999999995,"5min":0.67200000000000004,"15min":0.34399999999999997},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":5.4699999999999998},"last":0.879},{"interface":"front","average":{"1min":1.377,"5min":1.3109999999999999,"15min":1.159},"min":{"1min":0.95699999999999996,"5min":0.78900000000000003,"15min":0.376},"max":{"1min":1.79,"5min":1.8200000000000001,"15min":5.4420000000000002},"last":1.1499999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.345,"15min":1.1519999999999999},"min":{"1min":1.0549999999999999,"5min":0.65400000000000003,"15min":0.36199999999999999},"max":{"1min":1.601,"5min":2.133,"15min":5.2949999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.466,"15min":1.2549999999999999},"min":{"1min":1.202,"5min":0.74399999999999999,"15min":0.42599999999999999},"max":{"1min":1.6719999999999999,"5min":2.2480000000000002,"15min":5.3170000000000002},"last":1.327}]},{"osd":5,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.458,"15min":1.236},"min":{"1min":1.1539999999999999,"5min":0.72199999999999998,"15min":0.373},"max":{"1min":1.8959999999999999,"5min":1.9590000000000001,"15min":5.6280000000000001},"last":1.234},{"interface":"front","average":{"1min":1.585,"5min":1.4199999999999999,"15min":1.1970000000000001},"min":{"1min":1.2310000000000001,"5min":0.60799999999999998,"15min":0.42599999999999999},"max":{"1min":1.9790000000000001,"5min":2.1909999999999998,"15min":5.085},"last":1.3540000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.421,"15min":1.228},"min":{"1min":0.97199999999999998,"5min":0.60299999999999998,"15min":0.42699999999999999},"max":{"1min":1.8440000000000001,"5min":1.9550000000000001,"15min":5.2729999999999997},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.2150000000000001,"15min":1.115},"min":{"1min":0.92800000000000005,"5min":0.68000000000000005,"15min":0.40699999999999997},"max":{"1min":1.4350000000000001,"5min":1.8799999999999999,"15min":5.0499999999999998},"last":1.3919999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.95799999999999996,"15min":1.169},"min":{"1min":0.45600000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":6.7050000000000001,"15min":7.9489999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":1.1950000000000001,"15min":1.2529999999999999},"min":{"1min":0.53700000000000003,"5min":0.44800000000000001,"15min":0.43099999999999999},"max":{"1min":1.3220000000000001,"5min":6.7720000000000002,"15min":7.9160000000000004},"last":0.56799999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.4610000000000001,"15min":1.3069999999999999},"min":{"1min":1.147,"5min":0.61699999999999999,"15min":0.38100000000000001},"max":{"1min":1.8460000000000001,"5min":2.3250000000000002,"15min":5.2460000000000004},"last":1.415},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.321,"15min":1.21},"min":{"1min":0.96399999999999997,"5min":0.72099999999999997,"15min":0.35299999999999998},"max":{"1min":1.585,"5min":1.9430000000000001,"15min":5.2160000000000002},"last":1.306}]},{"osd":10,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.6120000000000001,"15min":1.607},"min":{"1min":0.97299999999999998,"5min":0.88400000000000001,"15min":0.38500000000000001},"max":{"1min":1.925,"5min":2.089,"15min":66.311999999999998},"last":1.492},{"interface":"front","average":{"1min":1.665,"5min":1.6950000000000001,"15min":1.7090000000000001},"min":{"1min":1.365,"5min":0.879,"15min":0.34599999999999997},"max":{"1min":2.1339999999999999,"5min":2.157,"15min":59.326999999999998},"last":1.544}]},{"osd":11,"last update":"Mon Feb 12 16:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.411,"15min":1.304},"min":{"1min":1.0920000000000001,"5min":0.83999999999999997,"15min":0.371},"max":{"1min":1.839,"5min":1.8819999999999999,"15min":5.6609999999999996},"last":1.47},{"interface":"front","average":{"1min":1.704,"5min":1.6870000000000001,"15min":1.4390000000000001},"min":{"1min":1.369,"5min":1.2310000000000001,"15min":0.441},"max":{"1min":2.29,"5min":2.29,"15min":5.5659999999999998},"last":1.4430000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874961,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898380,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":897600,"kb_avail":92846772,"statfs":{"total":95995035648,"available":95075094528,"internally_reserved":0,"allocated":790528,"data_stored":365503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":919142400},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.02,"15min":0.86599999999999999},"min":{"1min":0.61799999999999999,"5min":0.60999999999999999,"15min":0.27500000000000002},"max":{"1min":1.387,"5min":1.732,"15min":5.96},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.0269999999999999,"15min":0.91500000000000004},"min":{"1min":0.69099999999999995,"5min":0.60799999999999998,"15min":0.308},"max":{"1min":1.7849999999999999,"5min":1.9119999999999999,"15min":6.7729999999999997},"last":1.387}]},{"osd":1,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.27900000000000003},"max":{"1min":1.048,"5min":1.4339999999999999,"15min":8.1210000000000004},"last":1.022},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.101,"15min":0.98899999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.27300000000000002},"max":{"1min":1.5109999999999999,"5min":1.5960000000000001,"15min":8.4819999999999993},"last":1.746}]},{"osd":2,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":1.9399999999999999,"5min":3.2320000000000002,"15min":8.6430000000000007},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2609999999999999,"15min":1.091},"min":{"1min":0.48999999999999999,"5min":0.378,"15min":0.26800000000000002},"max":{"1min":2.0070000000000001,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.84099999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.157,"15min":1.0700000000000001},"min":{"1min":0.70899999999999996,"5min":0.60199999999999998,"15min":0.29399999999999998},"max":{"1min":1.631,"5min":1.6830000000000001,"15min":10.231},"last":1.613},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.98099999999999998,"15min":1.0089999999999999},"min":{"1min":0.70499999999999996,"5min":0.66400000000000003,"15min":0.34200000000000003},"max":{"1min":1.361,"5min":1.6519999999999999,"15min":10.265000000000001},"last":1.2310000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.302,"15min":1.1459999999999999},"min":{"1min":0.93400000000000005,"5min":0.66200000000000003,"15min":0.313},"max":{"1min":1.593,"5min":1.873,"15min":3.3620000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.381,"15min":1.127},"min":{"1min":1.143,"5min":0.628,"15min":0.38200000000000001},"max":{"1min":1.9379999999999999,"5min":2.1840000000000002,"15min":4.4500000000000002},"last":1.827}]},{"osd":5,"last update":"Mon Feb 12 16:55:06 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.1299999999999999,"15min":1.071},"min":{"1min":0.81899999999999995,"5min":0.624,"15min":0.38},"max":{"1min":1.5529999999999999,"5min":1.633,"15min":4.5069999999999997},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.4330000000000001,"15min":1.254},"min":{"1min":1.0940000000000001,"5min":0.86699999999999999,"15min":0.42399999999999999},"max":{"1min":1.9319999999999999,"5min":2.097,"15min":6.681},"last":1.903}]},{"osd":6,"last update":"Mon Feb 12 16:55:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.49,"15min":1.2749999999999999},"min":{"1min":0.97599999999999998,"5min":0.94699999999999995,"15min":0.43099999999999999},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":5.5369999999999999},"last":2.0070000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.224,"15min":1.115},"min":{"1min":0.83199999999999996,"5min":0.78800000000000003,"15min":0.41399999999999998},"max":{"1min":1.702,"5min":1.829,"15min":4.5140000000000002},"last":1.4530000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:55:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.173,"15min":1.1419999999999999},"min":{"1min":0.81299999999999994,"5min":0.77800000000000002,"15min":0.40500000000000003},"max":{"1min":1.462,"5min":1.75,"15min":6.5970000000000004},"last":1.661},{"interface":"front","average":{"1min":1.577,"5min":1.593,"15min":1.323},"min":{"1min":1.1719999999999999,"5min":0.93400000000000005,"15min":0.41599999999999998},"max":{"1min":2.0550000000000002,"5min":2.27,"15min":4.0069999999999997},"last":2.0659999999999998}]},{"osd":9,"last update":"Mon Feb 12 16:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.5029999999999999,"15min":1.2889999999999999},"min":{"1min":0.58399999999999996,"5min":0.52500000000000002,"15min":0.40600000000000003},"max":{"1min":2.0110000000000001,"5min":2.3340000000000001,"15min":4.407},"last":1.9530000000000001},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.6919999999999999,"15min":1.4259999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.49199999999999999},"max":{"1min":2.1139999999999999,"5min":2.4430000000000001,"15min":4.5380000000000003},"last":2.1139999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.2949999999999999,"15min":1.1499999999999999},"min":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.39500000000000002},"max":{"1min":1.663,"5min":1.8759999999999999,"15min":4.5289999999999999},"last":1.702},{"interface":"front","average":{"1min":1.355,"5min":1.365,"15min":1.24},"min":{"1min":1.1579999999999999,"5min":0.875,"15min":0.35399999999999998},"max":{"1min":1.7709999999999999,"5min":2,"15min":5.4169999999999998},"last":1.504}]},{"osd":11,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3180000000000001,"15min":1.254},"min":{"1min":0.49199999999999999,"5min":0.441,"15min":0.437},"max":{"1min":1.853,"5min":1.9670000000000001,"15min":6.1699999999999999},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4830000000000001,"15min":1.3720000000000001},"min":{"1min":0.81499999999999995,"5min":0.65600000000000003,"15min":0.45800000000000002},"max":{"1min":1.897,"5min":2.0299999999999998,"15min":5.7220000000000004},"last":1.847}]}]},{"osd":7,"up_from":923,"seq":3964254814369,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319244,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":318463,"kb_avail":93425908,"statfs":{"total":95995035648,"available":95668129792,"internally_reserved":0,"allocated":790528,"data_stored":365503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":326106164},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.93200000000000005,"15min":0.89700000000000002},"min":{"1min":0.746,"5min":0.62,"15min":0.27100000000000002},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":6.4630000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.99399999999999999,"15min":0.92100000000000004},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.30199999999999999},"max":{"1min":1.351,"5min":1.3640000000000001,"15min":5.2110000000000003},"last":1.089}]},{"osd":1,"last update":"Mon Feb 12 16:46:47 2024","interfaces":[{"interface":"back","average":{"1min":2.778,"5min":3.3290000000000002,"15min":1.8089999999999999},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":32.298000000000002,"5min":177.82400000000001,"15min":177.82400000000001},"last":1.131},{"interface":"front","average":{"1min":2.7519999999999998,"5min":3.153,"15min":1.429},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":31.824000000000002,"5min":177.30500000000001,"15min":177.30500000000001},"last":1.47}]},{"osd":2,"last update":"Mon Feb 12 16:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.004,"15min":0.876},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.318},"max":{"1min":1.1779999999999999,"5min":1.383,"15min":5.2359999999999998},"last":1.171},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.071,"15min":1.0389999999999999},"min":{"1min":0.81899999999999995,"5min":0.61699999999999999,"15min":0.32900000000000001},"max":{"1min":1.3089999999999999,"5min":1.627,"15min":5.3630000000000004},"last":1.282}]},{"osd":3,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1100000000000001,"15min":1.151},"min":{"1min":0.84599999999999997,"5min":0.69999999999999996,"15min":0.38200000000000001},"max":{"1min":1.409,"5min":1.5049999999999999,"15min":9.1029999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.298,"5min":1.28,"15min":1.2569999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.443},"max":{"1min":1.5640000000000001,"5min":1.78,"15min":9.1470000000000002},"last":1.5}]},{"osd":4,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.84999999999999998},"min":{"1min":0.71899999999999997,"5min":0.63200000000000001,"15min":0.28199999999999997},"max":{"1min":1.2490000000000001,"5min":1.4810000000000001,"15min":5.4199999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.1160000000000001,"15min":1.0289999999999999},"min":{"1min":0.82699999999999996,"5min":0.66200000000000003,"15min":0.33100000000000002},"max":{"1min":1.444,"5min":1.5529999999999999,"15min":8.4719999999999995},"last":1.3069999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.133,"15min":1.149},"min":{"1min":0.81799999999999995,"5min":0.625,"15min":0.36399999999999999},"max":{"1min":1.415,"5min":1.7230000000000001,"15min":6.4909999999999997},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.2749999999999999,"15min":1.3089999999999999},"min":{"1min":0.80000000000000004,"5min":0.70599999999999996,"15min":0.36399999999999999},"max":{"1min":1.6930000000000001,"5min":1.6950000000000001,"15min":5.7069999999999999},"last":1.3400000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1930000000000001,"15min":1.091},"min":{"1min":0.67900000000000005,"5min":0.49199999999999999,"15min":0.38400000000000001},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":6.5069999999999997},"last":1.613},{"interface":"front","average":{"1min":1.212,"5min":1.2090000000000001,"15min":1.052},"min":{"1min":0.94199999999999995,"5min":0.66600000000000004,"15min":0.36899999999999999},"max":{"1min":1.542,"5min":1.6040000000000001,"15min":5.508},"last":1.407}]},{"osd":8,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4319999999999999,"15min":1.272},"min":{"1min":1.026,"5min":1.026,"15min":0.437},"max":{"1min":1.7310000000000001,"5min":2.355,"15min":5.6719999999999997},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.472,"15min":1.25},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.46999999999999997},"max":{"1min":1.925,"5min":2.093,"15min":6.5880000000000001},"last":1.855}]},{"osd":9,"last update":"Mon Feb 12 16:54:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.506,"15min":1.528},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":1.756,"5min":1.944,"15min":2.1890000000000001},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.585,"5min":1.53,"15min":1.4870000000000001},"min":{"1min":1.3759999999999999,"5min":1.103,"15min":1.034},"max":{"1min":1.8200000000000001,"5min":2.2400000000000002,"15min":2.4380000000000002},"last":1.6399999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.5449999999999999,"15min":1.3720000000000001},"min":{"1min":1.361,"5min":0.98199999999999998,"15min":0.50600000000000001},"max":{"1min":1.8759999999999999,"5min":1.9179999999999999,"15min":5.6559999999999997},"last":1.9510000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.5649999999999999,"15min":1.2869999999999999},"min":{"1min":1.3380000000000001,"5min":1.0089999999999999,"15min":0.48099999999999998},"max":{"1min":1.8420000000000001,"5min":2.0539999999999998,"15min":6.5650000000000004},"last":1.75}]},{"osd":11,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.599,"15min":1.476},"min":{"1min":1.2889999999999999,"5min":1.1160000000000001,"15min":0.53300000000000003},"max":{"1min":1.9950000000000001,"5min":2.0680000000000001,"15min":9.7409999999999997},"last":2.0419999999999998},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.6020000000000001,"15min":1.452},"min":{"1min":1.3340000000000001,"5min":1.0980000000000001,"15min":0.437},"max":{"1min":1.8859999999999999,"5min":2.0979999999999999,"15min":5.5670000000000002},"last":1.907}]}]},{"osd":6,"up_from":48,"seq":206158430945,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895168,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":894400,"kb_avail":92849984,"statfs":{"total":95995035648,"available":95078383616,"internally_reserved":0,"allocated":778240,"data_stored":359590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915865600},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.03,"15min":0.996},"min":{"1min":0.78300000000000003,"5min":0.70499999999999996,"15min":0.68899999999999995},"max":{"1min":1.2310000000000001,"5min":1.538,"15min":1.538},"last":1.153},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0529999999999999,"15min":1.0489999999999999},"min":{"1min":0.70099999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.3520000000000001,"5min":1.512,"15min":1.639},"last":0.93799999999999994}]},{"osd":1,"last update":"Mon Feb 12 16:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":1.7509999999999999,"15min":1.3100000000000001},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74099999999999999,"5min":118.962,"15min":118.962},"last":1.6060000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.8360000000000001,"15min":1.395},"min":{"1min":0.28799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.1379999999999999,"5min":122.48,"15min":122.48},"last":1.0760000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.0169999999999999,"15min":0.89200000000000002},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.308},"max":{"1min":1.349,"5min":1.6160000000000001,"15min":6.3380000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.044,"5min":1.095,"15min":0.93799999999999994},"min":{"1min":0.82099999999999995,"5min":0.68500000000000005,"15min":0.33700000000000002},"max":{"1min":1.264,"5min":1.597,"15min":4.6900000000000004},"last":1.325}]},{"osd":3,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.141,"15min":1.448},"min":{"1min":0.90100000000000002,"5min":0.79300000000000004,"15min":0.316},"max":{"1min":1.4710000000000001,"5min":1.494,"15min":124.44499999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.149,"15min":1.4590000000000001},"min":{"1min":0.85899999999999999,"5min":0.78900000000000003,"15min":0.374},"max":{"1min":1.3120000000000001,"5min":1.722,"15min":124.407},"last":1.1930000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.115,"15min":0.93200000000000005},"min":{"1min":0.71699999999999997,"5min":0.56499999999999995,"15min":0.29699999999999999},"max":{"1min":1.476,"5min":1.6200000000000001,"15min":3.5},"last":1.125},{"interface":"front","average":{"1min":1.171,"5min":1.1579999999999999,"15min":0.997},"min":{"1min":0.73499999999999999,"5min":0.69099999999999995,"15min":0.29899999999999999},"max":{"1min":1.591,"5min":1.841,"15min":7.2149999999999999},"last":0.90100000000000002}]},{"osd":5,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1990000000000001,"15min":2.0579999999999998},"min":{"1min":0.85699999999999998,"5min":0.60499999999999998,"15min":0.42499999999999999},"max":{"1min":1.546,"5min":1.885,"15min":281.48500000000001},"last":1.361},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2549999999999999,"15min":2.1230000000000002},"min":{"1min":0.77600000000000002,"5min":0.76800000000000002,"15min":0.34499999999999997},"max":{"1min":1.4330000000000001,"5min":1.9950000000000001,"15min":282.96199999999999},"last":1.075}]},{"osd":7,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2849999999999999,"15min":1.048},"min":{"1min":0.81399999999999995,"5min":0.72399999999999998,"15min":0.34300000000000003},"max":{"1min":1.516,"5min":1.8720000000000001,"15min":6.8869999999999996},"last":1.242},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.345,"15min":1.1439999999999999},"min":{"1min":1.014,"5min":0.92500000000000004,"15min":0.40000000000000002},"max":{"1min":1.6459999999999999,"5min":1.9319999999999999,"15min":4.0510000000000002},"last":1.292}]},{"osd":8,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.5309999999999999,"15min":1.266},"min":{"1min":1.121,"5min":0.88800000000000001,"15min":0.40899999999999997},"max":{"1min":1.7549999999999999,"5min":2.0870000000000002,"15min":5.5279999999999996},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.556,"15min":1.296},"min":{"1min":1.1339999999999999,"5min":1.127,"15min":0.40500000000000003},"max":{"1min":1.855,"5min":2.395,"15min":5.5700000000000003},"last":1.4650000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.6319999999999999,"15min":1.339},"min":{"1min":1.167,"5min":1.135,"15min":0.46500000000000002},"max":{"1min":2.0030000000000001,"5min":2.468,"15min":3.383},"last":1.532},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.716,"15min":1.395},"min":{"1min":1.3160000000000001,"5min":1.2629999999999999,"15min":0.41299999999999998},"max":{"1min":1.9890000000000001,"5min":2.407,"15min":3.4119999999999999},"last":1.6459999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.6080000000000001,"15min":1.4219999999999999},"min":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":0.46700000000000003},"max":{"1min":1.827,"5min":2.2040000000000002,"15min":4.516},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.694,"5min":1.7230000000000001,"15min":1.5069999999999999},"min":{"1min":1.3720000000000001,"5min":1.3220000000000001,"15min":0.54500000000000004},"max":{"1min":2.2080000000000002,"5min":2.3330000000000002,"15min":4.5469999999999997},"last":1.7030000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.5549999999999999,"15min":1.3580000000000001},"min":{"1min":1.256,"5min":1.1399999999999999,"15min":0.51300000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":5.3799999999999999},"last":1.492},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.6639999999999999,"15min":1.4390000000000001},"min":{"1min":1.3740000000000001,"5min":1.1970000000000001,"15min":0.45600000000000002},"max":{"1min":2.1890000000000001,"5min":2.2320000000000002,"15min":5.6130000000000004},"last":1.587}]}]},{"osd":1,"up_from":13,"seq":55834575622,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1191552,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":1190784,"kb_avail":92553600,"statfs":{"total":95995035648,"available":94774886400,"internally_reserved":0,"allocated":778240,"data_stored":359590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1219362816},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.89900000000000002,"15min":0.88300000000000001},"min":{"1min":0.69299999999999995,"5min":0.48699999999999999,"15min":0.23899999999999999},"max":{"1min":1.0940000000000001,"5min":2.3380000000000001,"15min":7.532},"last":0.874},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89800000000000002,"15min":0.82099999999999995},"min":{"1min":0.70099999999999996,"5min":0.42799999999999999,"15min":0.254},"max":{"1min":1.4590000000000001,"5min":1.976,"15min":7.3540000000000001},"last":0.90100000000000002}]},{"osd":2,"last update":"Mon Feb 12 16:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.995,"15min":0.93899999999999995},"min":{"1min":0.78000000000000003,"5min":0.65800000000000003,"15min":0.29399999999999998},"max":{"1min":1.367,"5min":1.367,"15min":7.54},"last":1.113},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.94699999999999995,"15min":0.91000000000000003},"min":{"1min":0.67200000000000004,"5min":0.62,"15min":0.25900000000000001},"max":{"1min":1.2869999999999999,"5min":1.6519999999999999,"15min":7.9539999999999997},"last":1.083}]},{"osd":3,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":0.98899999999999999,"15min":1.1679999999999999},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":4.1820000000000004,"5min":8.9920000000000009,"15min":26.664000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.466,"5min":1.113,"15min":1.389},"min":{"1min":0.50700000000000001,"5min":0.34599999999999997,"15min":0.26200000000000001},"max":{"1min":5.2080000000000002,"5min":8.9610000000000003,"15min":27.148},"last":1.133}]},{"osd":4,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.1839999999999999,"15min":1.069},"min":{"1min":0.91800000000000004,"5min":0.61899999999999999,"15min":0.34699999999999998},"max":{"1min":1.5469999999999999,"5min":1.673,"15min":7.6029999999999998},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.226,"15min":1.125},"min":{"1min":0.79500000000000004,"5min":0.57699999999999996,"15min":0.36699999999999999},"max":{"1min":1.7909999999999999,"5min":1.839,"15min":7.7149999999999999},"last":1.224}]},{"osd":5,"last update":"Mon Feb 12 16:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1439999999999999,"15min":1.097},"min":{"1min":0.82499999999999996,"5min":0.81299999999999994,"15min":0.32100000000000001},"max":{"1min":1.508,"5min":1.718,"15min":8.0190000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2130000000000001,"15min":1.177},"min":{"1min":0.97399999999999998,"5min":0.78500000000000003,"15min":0.34799999999999998},"max":{"1min":1.587,"5min":2.1800000000000002,"15min":8.7050000000000001},"last":1.3460000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:54:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2849999999999999,"15min":1.2030000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.39000000000000001},"max":{"1min":1.7070000000000001,"5min":2.2210000000000001,"15min":7.9450000000000003},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.3700000000000001,"15min":1.2889999999999999},"min":{"1min":0.99199999999999999,"5min":0.70499999999999996,"15min":0.40999999999999998},"max":{"1min":1.7270000000000001,"5min":2.0510000000000002,"15min":8.7669999999999995},"last":1.496}]},{"osd":7,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.292,"15min":1.1919999999999999},"min":{"1min":0.874,"5min":0.751,"15min":0.44600000000000001},"max":{"1min":1.615,"5min":1.8779999999999999,"15min":2.75},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4059999999999999,"15min":1.167},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.38800000000000001},"max":{"1min":1.591,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":1.1479999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6279999999999999,"15min":1.4590000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.48499999999999999},"max":{"1min":1.944,"5min":2.5960000000000001,"15min":8.8100000000000005},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.536,"15min":1.496},"min":{"1min":1.2250000000000001,"5min":0.84299999999999997,"15min":0.54300000000000004},"max":{"1min":1.792,"5min":2.2389999999999999,"15min":8.9269999999999996},"last":1.611}]},{"osd":9,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.544,"15min":1.3320000000000001},"min":{"1min":1.2390000000000001,"5min":1,"15min":0.40200000000000002},"max":{"1min":1.7789999999999999,"5min":2.4289999999999998,"15min":8.1539999999999999},"last":1.411},{"interface":"front","average":{"1min":1.641,"5min":1.6539999999999999,"15min":1.425},"min":{"1min":1.3959999999999999,"5min":1.1679999999999999,"15min":0.48099999999999998},"max":{"1min":2.0870000000000002,"5min":2.3639999999999999,"15min":8.2409999999999997},"last":1.538}]},{"osd":10,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.6379999999999999,"15min":1.474},"min":{"1min":1.278,"5min":0.80200000000000005,"15min":0.44900000000000001},"max":{"1min":1.952,"5min":2.3620000000000001,"15min":8.1140000000000008},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.6799999999999999,"15min":1.5549999999999999},"min":{"1min":1.2410000000000001,"5min":1.1519999999999999,"15min":0.51200000000000001},"max":{"1min":2.048,"5min":2.7149999999999999,"15min":8.8469999999999995},"last":1.6779999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.639,"15min":1.6659999999999999},"min":{"1min":1.099,"5min":1.099,"15min":0.53800000000000003},"max":{"1min":2.1400000000000001,"5min":2.823,"15min":14.862},"last":1.792},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.623,"15min":1.5049999999999999},"min":{"1min":0.89800000000000002,"5min":0.87,"15min":0.51800000000000002},"max":{"1min":2.0659999999999998,"5min":2.2930000000000001,"15min":15.76},"last":1.6359999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739156,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887680,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":886912,"kb_avail":92857472,"statfs":{"total":95995035648,"available":95086051328,"internally_reserved":0,"allocated":778240,"data_stored":359590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":908197888},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84199999999999997,"15min":1.254},"min":{"1min":0.53600000000000003,"5min":0.33200000000000002,"15min":0.24099999999999999},"max":{"1min":1.3240000000000001,"5min":1.4079999999999999,"15min":118.657},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.004,"15min":1.4159999999999999},"min":{"1min":0.52500000000000002,"5min":0.51100000000000001,"15min":0.29199999999999998},"max":{"1min":1.405,"5min":1.9930000000000001,"15min":118.58},"last":1.1399999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.91200000000000003,"15min":0.81399999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.27500000000000002},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":4.0999999999999996},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.97599999999999998,"15min":0.877},"min":{"1min":0.621,"5min":0.41799999999999998,"15min":0.28599999999999998},"max":{"1min":1.405,"5min":1.909,"15min":4.0469999999999997},"last":0.77400000000000002}]},{"osd":3,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0509999999999999,"15min":1.679},"min":{"1min":0.72499999999999998,"5min":0.47599999999999998,"15min":0.32900000000000001},"max":{"1min":1.722,"5min":1.722,"15min":193.114},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.98599999999999999,"15min":1.625},"min":{"1min":0.57299999999999995,"5min":0.39000000000000001,"15min":0.26600000000000001},"max":{"1min":1.234,"5min":1.7729999999999999,"15min":193.22},"last":0.98899999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.232,"15min":1.002},"min":{"1min":0.78800000000000003,"5min":0.58299999999999996,"15min":0.35699999999999998},"max":{"1min":1.847,"5min":1.847,"15min":2.21},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.196,"15min":1.0249999999999999},"min":{"1min":0.70499999999999996,"5min":0.65900000000000003,"15min":0.35199999999999998},"max":{"1min":1.754,"5min":1.754,"15min":3.9580000000000002},"last":1.2989999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.248,"15min":1.0840000000000001},"min":{"1min":0.67700000000000005,"5min":0.50700000000000001,"15min":0.33700000000000002},"max":{"1min":1.7909999999999999,"5min":1.8680000000000001,"15min":8.532},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.371,"15min":1.2070000000000001},"min":{"1min":0.73499999999999999,"5min":0.60399999999999998,"15min":0.41299999999999998},"max":{"1min":1.927,"5min":2.0099999999999998,"15min":8.6519999999999992},"last":1.4470000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:54:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3169999999999999,"15min":1.0980000000000001},"min":{"1min":0.91700000000000004,"5min":0.46800000000000003,"15min":0.38300000000000001},"max":{"1min":1.837,"5min":1.964,"15min":6.6950000000000003},"last":1.353},{"interface":"front","average":{"1min":1.351,"5min":1.327,"15min":1.123},"min":{"1min":1.075,"5min":0.63700000000000001,"15min":0.38300000000000001},"max":{"1min":1.825,"5min":2.0950000000000002,"15min":2.1779999999999999},"last":1.5089999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:39:13 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.24,"15min":2.706},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.374},"max":{"1min":1.835,"5min":6.1130000000000004,"15min":191.47200000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4019999999999999,"15min":2.823},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":1.897,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.081}]},{"osd":8,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.552,"15min":1.28},"min":{"1min":0.88500000000000001,"5min":0.55300000000000005,"15min":0.442},"max":{"1min":2.0369999999999999,"5min":2.2109999999999999,"15min":6.7400000000000002},"last":1.714},{"interface":"front","average":{"1min":1.649,"5min":1.6339999999999999,"15min":1.3799999999999999},"min":{"1min":0.91300000000000003,"5min":0.72699999999999998,"15min":0.51600000000000001},"max":{"1min":2.2389999999999999,"5min":2.3420000000000001,"15min":6.8570000000000002},"last":1.6579999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:54:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4810000000000001,"15min":1.2450000000000001},"min":{"1min":0.97699999999999998,"5min":0.79200000000000004,"15min":0.42299999999999999},"max":{"1min":1.919,"5min":1.9750000000000001,"15min":4.1130000000000004},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.544,"15min":1.278},"min":{"1min":1.1599999999999999,"5min":0.67600000000000005,"15min":0.47499999999999998},"max":{"1min":1.9279999999999999,"5min":2.246,"15min":6.7770000000000001},"last":1.5569999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.5429999999999999,"15min":1.3340000000000001},"min":{"1min":0.91100000000000003,"5min":0.71599999999999997,"15min":0.46100000000000002},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":6.0880000000000001},"last":1.671},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6479999999999999,"15min":1.415},"min":{"1min":1.101,"5min":0.86899999999999999,"15min":0.52400000000000002},"max":{"1min":2.472,"5min":2.472,"15min":6.1319999999999997},"last":1.734}]},{"osd":11,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4990000000000001,"15min":1.361},"min":{"1min":0.97299999999999998,"5min":0.75900000000000001,"15min":0.47599999999999998},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":5.8079999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5009999999999999,"15min":1.3500000000000001},"min":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.50600000000000001},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":5.7140000000000004},"last":1.6080000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313984,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594572,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93150580,"statfs":{"total":95995035648,"available":95386193920,"internally_reserved":0,"allocated":790528,"data_stored":365503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.88900000000000001,"15min":0.76200000000000001},"min":{"1min":0.48599999999999999,"5min":0.43099999999999999,"15min":0.223},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":3.996},"last":1.103},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0209999999999999,"15min":0.84299999999999997},"min":{"1min":0.627,"5min":0.53700000000000003,"15min":0.23400000000000001},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":4.9690000000000003},"last":1.056}]},{"osd":1,"last update":"Mon Feb 12 16:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.93700000000000006,"15min":1.212},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.27200000000000002},"max":{"1min":1.7410000000000001,"5min":2.1899999999999999,"15min":43.488999999999997},"last":1.238},{"interface":"front","average":{"1min":1.016,"5min":0.99199999999999999,"15min":1.2649999999999999},"min":{"1min":0.495,"5min":0.38700000000000001,"15min":0.22800000000000001},"max":{"1min":1.329,"5min":1.55,"15min":47.810000000000002},"last":1.298}]},{"osd":3,"last update":"Mon Feb 12 16:55:04 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.036,"15min":2.133},"min":{"1min":0.73899999999999999,"5min":0.439,"15min":0.28499999999999998},"max":{"1min":1.4179999999999999,"5min":2.0859999999999999,"15min":193.74799999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.254,"15min":2.2909999999999999},"min":{"1min":0.92500000000000004,"5min":0.55600000000000005,"15min":0.38100000000000001},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":193.80000000000001},"last":1.3520000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2549999999999999,"15min":1.018},"min":{"1min":0.751,"5min":0.70099999999999996,"15min":0.309},"max":{"1min":1.7929999999999999,"5min":2.5470000000000002,"15min":3.0470000000000002},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.3180000000000001,"15min":1.0800000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.32400000000000001},"max":{"1min":2.258,"5min":2.2789999999999999,"15min":6.9409999999999998},"last":1.466}]},{"osd":5,"last update":"Mon Feb 12 16:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2070000000000001,"15min":1.04},"min":{"1min":0.83299999999999996,"5min":0.65500000000000003,"15min":0.31900000000000001},"max":{"1min":1.593,"5min":1.9350000000000001,"15min":5.1319999999999997},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.365,"15min":1.115},"min":{"1min":0.68600000000000005,"5min":0.60999999999999999,"15min":0.30599999999999999},"max":{"1min":1.984,"5min":2.7290000000000001,"15min":4.2089999999999996},"last":1.536}]},{"osd":6,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.425,"15min":1.165},"min":{"1min":0.88700000000000001,"5min":0.75900000000000001,"15min":0.39800000000000002},"max":{"1min":2.3650000000000002,"5min":2.5230000000000001,"15min":7.0049999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3260000000000001,"15min":1.131},"min":{"1min":0.79600000000000004,"5min":0.67000000000000004,"15min":0.36099999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":4.0940000000000003},"last":1.611}]},{"osd":7,"last update":"Mon Feb 12 16:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.58399999999999996,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":1.9019999999999999,"5min":6.2560000000000002,"15min":11.044},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2529999999999999,"15min":1.248},"min":{"1min":0.44600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.4219999999999997,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":1.6499999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:55:04 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.6080000000000001,"15min":1.494},"min":{"1min":1.2410000000000001,"5min":0.73199999999999998,"15min":0.36899999999999999},"max":{"1min":2.069,"5min":2.3650000000000002,"15min":59.164999999999999},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.546,"15min":1.4550000000000001},"min":{"1min":1.0860000000000001,"5min":0.90300000000000002,"15min":0.44500000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":58.993000000000002},"last":1.9259999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.556,"15min":1.272},"min":{"1min":0.96599999999999997,"5min":0.79700000000000004,"15min":0.435},"max":{"1min":2.1749999999999998,"5min":2.8490000000000002,"15min":5.2210000000000001},"last":1.718},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6319999999999999,"15min":1.367},"min":{"1min":0.95699999999999996,"5min":0.83299999999999996,"15min":0.41299999999999998},"max":{"1min":2.4969999999999999,"5min":2.9700000000000002,"15min":5.2549999999999999},"last":1.8160000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.6419999999999999,"15min":2.798},"min":{"1min":1.095,"5min":0.879,"15min":0.45600000000000002},"max":{"1min":2.702,"5min":2.702,"15min":407.577},"last":1.875},{"interface":"front","average":{"1min":1.782,"5min":1.698,"15min":2.778},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.47799999999999998},"max":{"1min":2.6360000000000001,"5min":2.6360000000000001,"15min":407.45100000000002},"last":1.6699999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.5840000000000001,"15min":1.5029999999999999},"min":{"1min":0.875,"5min":0.875,"15min":0.47099999999999997},"max":{"1min":2.2890000000000001,"5min":2.5,"15min":31.835999999999999},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.645,"15min":1.548},"min":{"1min":1.006,"5min":0.93999999999999995,"15min":0.53500000000000003},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":31.978999999999999},"last":1.8440000000000001}]}]},{"osd":3,"up_from":522,"seq":2241972928908,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930752,"kb_used_data":760,"kb_used_omap":91,"kb_used_meta":929892,"kb_avail":92814400,"statfs":{"total":95995035648,"available":95041945600,"internally_reserved":0,"allocated":778240,"data_stored":359590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":952209424},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.86099999999999999,"15min":0.872},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0289999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.89600000000000002,"15min":0.90600000000000003},"min":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":1.036}]},{"osd":1,"last update":"Mon Feb 12 16:50:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.89600000000000002,"15min":1.909},"min":{"1min":0.63500000000000001,"5min":0.26900000000000002,"15min":0.26200000000000001},"max":{"1min":1.359,"5min":3.819,"15min":245.10900000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.056,"5min":1.1639999999999999,"15min":2.089},"min":{"1min":0.79300000000000004,"5min":0.313,"15min":0.313},"max":{"1min":1.6859999999999999,"5min":5.0279999999999996,"15min":245.21199999999999},"last":1.1599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.91500000000000004,"15min":1.206},"min":{"1min":0.56899999999999995,"5min":0.51100000000000001,"15min":0.30399999999999999},"max":{"1min":1.0209999999999999,"5min":1.911,"15min":81.769999999999996},"last":1.004},{"interface":"front","average":{"1min":1.115,"5min":1.1359999999999999,"15min":1.357},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.32300000000000001},"max":{"1min":1.516,"5min":2.262,"15min":81.876999999999995},"last":1.232}]},{"osd":4,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.075,"15min":1.008},"min":{"1min":0.65600000000000003,"5min":0.60899999999999999,"15min":0.32000000000000001},"max":{"1min":1.3360000000000001,"5min":1.6719999999999999,"15min":4.806},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.3069999999999999,"15min":1.1200000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.318},"max":{"1min":1.4950000000000001,"5min":2.0840000000000001,"15min":8.0069999999999997},"last":1.472}]},{"osd":5,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.075,"15min":1.0429999999999999},"min":{"1min":1.018,"5min":0.73399999999999999,"15min":0.318},"max":{"1min":1.482,"5min":1.623,"15min":10.058},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2769999999999999,"15min":1.2250000000000001},"min":{"1min":1.0529999999999999,"5min":0.872,"15min":0.35899999999999999},"max":{"1min":1.46,"5min":2.3679999999999999,"15min":8.032},"last":1.3939999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.1799999999999999,"15min":1.139},"min":{"1min":1.0049999999999999,"5min":0.751,"15min":0.379},"max":{"1min":1.6439999999999999,"5min":1.7509999999999999,"15min":9.7710000000000008},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.381,"15min":1.25},"min":{"1min":0.96499999999999997,"5min":0.91500000000000004,"15min":0.44500000000000001},"max":{"1min":1.8700000000000001,"5min":2.2410000000000001,"15min":4.9790000000000001},"last":1.637}]},{"osd":7,"last update":"Mon Feb 12 16:55:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2370000000000001,"15min":1.155},"min":{"1min":0.93200000000000005,"5min":0.70499999999999996,"15min":0.42399999999999999},"max":{"1min":1.5620000000000001,"5min":1.944,"15min":3.1150000000000002},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3300000000000001,"15min":1.169},"min":{"1min":1.1719999999999999,"5min":0.80700000000000005,"15min":0.54200000000000004},"max":{"1min":1.6419999999999999,"5min":1.988,"15min":3},"last":1.722}]},{"osd":8,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.4810000000000001,"15min":1.272},"min":{"1min":1.1699999999999999,"5min":0.76000000000000001,"15min":0.42699999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":8.2710000000000008},"last":1.782},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6020000000000001,"15min":1.4330000000000001},"min":{"1min":1.4179999999999999,"5min":1.113,"15min":0.47899999999999998},"max":{"1min":1.998,"5min":2.4780000000000002,"15min":5.0279999999999996},"last":1.8560000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.458,"15min":1.288},"min":{"1min":1.3600000000000001,"5min":0.80800000000000005,"15min":0.42599999999999999},"max":{"1min":1.7729999999999999,"5min":2.306,"15min":8.1159999999999997},"last":1.881},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.597,"15min":1.4259999999999999},"min":{"1min":1.452,"5min":1.1659999999999999,"15min":0.502},"max":{"1min":1.899,"5min":2.5459999999999998,"15min":4.9080000000000004},"last":1.756}]},{"osd":10,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5509999999999999,"15min":4.0490000000000004},"min":{"1min":1.234,"5min":1.089,"15min":0.45500000000000002},"max":{"1min":1.873,"5min":2.1389999999999998,"15min":738.99900000000002},"last":1.9870000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.6439999999999999,"15min":4.1230000000000002},"min":{"1min":1.3200000000000001,"5min":1.1399999999999999,"15min":0.51000000000000001},"max":{"1min":2.044,"5min":2.2250000000000001,"15min":746.45299999999997},"last":1.839}]},{"osd":11,"last update":"Mon Feb 12 16:54:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5449999999999999,"15min":1.5629999999999999},"min":{"1min":1.161,"5min":1.0920000000000001,"15min":0.55000000000000004},"max":{"1min":1.9570000000000001,"5min":2.2269999999999999,"15min":21.888000000000002},"last":1.9139999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.55,"15min":1.52},"min":{"1min":1.2849999999999999,"5min":0.83899999999999997,"15min":0.51400000000000001},"max":{"1min":1.875,"5min":2.3199999999999998,"15min":22.079999999999998},"last":1.583}]}]},{"osd":4,"up_from":32,"seq":137438954224,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890828,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":890048,"kb_avail":92854324,"statfs":{"total":95995035648,"available":95082827776,"internally_reserved":0,"allocated":790528,"data_stored":365503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911409152},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.92200000000000004,"15min":0.74199999999999999},"min":{"1min":0.70099999999999996,"5min":0.65500000000000003,"15min":0.26100000000000001},"max":{"1min":1.1359999999999999,"5min":1.383,"15min":3.218},"last":1.081},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0489999999999999,"15min":0.82499999999999996},"min":{"1min":0.72899999999999998,"5min":0.68799999999999994,"15min":0.31},"max":{"1min":1.454,"5min":1.835,"15min":3.3410000000000002},"last":1.1439999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.98699999999999999,"15min":0.90600000000000003},"min":{"1min":0.69099999999999995,"5min":0.62,"15min":0.27600000000000002},"max":{"1min":1.147,"5min":2.0449999999999999,"15min":7.7069999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.069,"15min":0.94999999999999996},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.32300000000000001},"max":{"1min":1.379,"5min":2.2869999999999999,"15min":5.2039999999999997},"last":1.3839999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:44:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":3.0840000000000001,"15min":1.8340000000000001},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3100000000000001,"5min":212.685,"15min":212.685},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":2.9910000000000001,"15min":1.7490000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.3540000000000001,"5min":210.95699999999999,"15min":210.95699999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0980000000000001,"15min":1.0669999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.33600000000000002},"max":{"1min":1.3360000000000001,"5min":1.992,"15min":17.876999999999999},"last":1.452},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.115,"15min":1.1180000000000001},"min":{"1min":0.87,"5min":0.71999999999999997,"15min":0.312},"max":{"1min":1.5009999999999999,"5min":1.877,"15min":17.623999999999999},"last":1.28}]},{"osd":5,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.99399999999999999,"15min":0.86899999999999999},"min":{"1min":0.64300000000000002,"5min":0.54400000000000004,"15min":0.28699999999999998},"max":{"1min":1.3839999999999999,"5min":2.0910000000000002,"15min":4.7370000000000001},"last":1.008},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3149999999999999,"15min":1.1000000000000001},"min":{"1min":1.1879999999999999,"5min":0.66400000000000003,"15min":0.35699999999999998},"max":{"1min":1.4810000000000001,"5min":2.2879999999999998,"15min":4.8150000000000004},"last":1.577}]},{"osd":6,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.98199999999999998,"15min":0.88},"min":{"1min":0.80200000000000005,"5min":0.68799999999999994,"15min":0.29899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3300000000000001,"15min":3.4830000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2889999999999999,"15min":1.093},"min":{"1min":0.92000000000000004,"5min":0.82499999999999996,"15min":0.36499999999999999},"max":{"1min":1.4790000000000001,"5min":2.3809999999999998,"15min":3.508},"last":1.526}]},{"osd":7,"last update":"Mon Feb 12 16:55:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.052,"15min":1.018},"min":{"1min":0.73999999999999999,"5min":0.58999999999999997,"15min":0.33600000000000002},"max":{"1min":1.298,"5min":1.5049999999999999,"15min":3.569},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2110000000000001,"15min":1.02},"min":{"1min":0.85799999999999998,"5min":0.83499999999999996,"15min":0.376},"max":{"1min":1.4179999999999999,"5min":1.9359999999999999,"15min":4.0709999999999997},"last":1.494}]},{"osd":8,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4359999999999999,"15min":1.2170000000000001},"min":{"1min":1.038,"5min":0.93500000000000005,"15min":0.44400000000000001},"max":{"1min":1.7649999999999999,"5min":2.915,"15min":3.8279999999999998},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4770000000000001,"15min":1.1890000000000001},"min":{"1min":1.0760000000000001,"5min":1.0169999999999999,"15min":0.38700000000000001},"max":{"1min":1.7749999999999999,"5min":2.718,"15min":4.6470000000000002},"last":1.641}]},{"osd":9,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5509999999999999,"15min":1.2869999999999999},"min":{"1min":1.1599999999999999,"5min":0.82499999999999996,"15min":0.46899999999999997},"max":{"1min":1.712,"5min":2.7690000000000001,"15min":3.319},"last":1.95},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.6559999999999999,"15min":1.3819999999999999},"min":{"1min":1.27,"5min":1.111,"15min":0.56699999999999995},"max":{"1min":1.948,"5min":2.8199999999999998,"15min":3.3519999999999999},"last":1.845}]},{"osd":10,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.577,"15min":2.1259999999999999},"min":{"1min":1.212,"5min":1.0920000000000001,"15min":0.41699999999999998},"max":{"1min":1.8420000000000001,"5min":2.7919999999999998,"15min":241.36600000000001},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.6299999999999999,"15min":2.109},"min":{"1min":1.3740000000000001,"5min":1.204,"15min":0.45500000000000002},"max":{"1min":1.77,"5min":2.5390000000000001,"15min":241.27799999999999},"last":1.921}]},{"osd":11,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.657,"15min":1.3300000000000001},"min":{"1min":1.3660000000000001,"5min":1.242,"15min":0.496},"max":{"1min":1.863,"5min":2.7090000000000001,"15min":3.4710000000000001},"last":1.702},{"interface":"front","average":{"1min":1.573,"5min":1.6399999999999999,"15min":1.409},"min":{"1min":1.3220000000000001,"5min":1.2470000000000001,"15min":0.55200000000000005},"max":{"1min":1.905,"5min":2.6659999999999999,"15min":3.4580000000000002},"last":2.024}]}]},{"osd":5,"up_from":39,"seq":167503725288,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893824,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":893056,"kb_avail":92851328,"statfs":{"total":95995035648,"available":95079759872,"internally_reserved":0,"allocated":778240,"data_stored":359590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914489344},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.74199999999999999,"15min":1.2609999999999999},"min":{"1min":0.28899999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2649999999999999,"5min":3.7610000000000001,"15min":38.366},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.94799999999999995,"15min":1.385},"min":{"1min":0.317,"5min":0.28899999999999998,"15min":0.255},"max":{"1min":4.915,"5min":5.1459999999999999,"15min":37.310000000000002},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.129,"15min":2.7229999999999999},"min":{"1min":0.80200000000000005,"5min":0.61799999999999999,"15min":0.23499999999999999},"max":{"1min":1.3440000000000001,"5min":1.518,"15min":387.76299999999998},"last":1.54},{"interface":"front","average":{"1min":1.23,"5min":1.25,"15min":2.8660000000000001},"min":{"1min":1.014,"5min":0.82899999999999996,"15min":0.34100000000000003},"max":{"1min":1.4379999999999999,"5min":1.615,"15min":387.88099999999997},"last":1.6659999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0189999999999999},"min":{"1min":0.69099999999999995,"5min":0.68000000000000005,"15min":0.33600000000000002},"max":{"1min":1.2609999999999999,"5min":1.3979999999999999,"15min":5.1109999999999998},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.97799999999999998,"15min":0.91800000000000004},"min":{"1min":0.71199999999999997,"5min":0.66600000000000004,"15min":0.26900000000000002},"max":{"1min":1.379,"5min":1.5149999999999999,"15min":7.3470000000000004},"last":1.236}]},{"osd":3,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.006,"15min":1.0640000000000001},"min":{"1min":0.72099999999999997,"5min":0.66400000000000003,"15min":0.28299999999999997},"max":{"1min":1.4139999999999999,"5min":1.4990000000000001,"15min":5.8879999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.0229999999999999,"15min":0.97299999999999998},"min":{"1min":0.71399999999999997,"5min":0.66400000000000003,"15min":0.315},"max":{"1min":1.151,"5min":1.3859999999999999,"15min":4.7839999999999998},"last":1.101}]},{"osd":4,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.03,"15min":0.97799999999999998},"min":{"1min":0.74399999999999999,"5min":0.44700000000000001,"15min":0.255},"max":{"1min":1.4450000000000001,"5min":1.766,"15min":5.7460000000000004},"last":1.3460000000000001},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.0269999999999999,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.60099999999999998,"15min":0.28199999999999997},"max":{"1min":1.2609999999999999,"5min":1.4950000000000001,"15min":5.069},"last":1.147}]},{"osd":6,"last update":"Mon Feb 12 16:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1619999999999999,"15min":1.111},"min":{"1min":0.93300000000000005,"5min":0.79600000000000004,"15min":0.35099999999999998},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":5.1760000000000002},"last":1.268},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.1240000000000001,"15min":1.042},"min":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.28699999999999998},"max":{"1min":1.4730000000000001,"5min":1.7210000000000001,"15min":7.3029999999999999},"last":1.4019999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:54:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.248,"15min":1.2749999999999999},"min":{"1min":0.76200000000000001,"5min":0.752,"15min":0.59999999999999998},"max":{"1min":1.53,"5min":1.742,"15min":4.8600000000000003},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.2709999999999999,"15min":1.294},"min":{"1min":0.91400000000000003,"5min":0.52100000000000002,"15min":0.42999999999999999},"max":{"1min":1.4850000000000001,"5min":1.661,"15min":4.8890000000000002},"last":1.4770000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:55:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.458,"15min":1.3420000000000001},"min":{"1min":1.077,"5min":0.74099999999999999,"15min":0.46100000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":8.0579999999999998},"last":1.8069999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4159999999999999,"15min":1.2290000000000001},"min":{"1min":1.085,"5min":0.90500000000000003,"15min":0.38200000000000001},"max":{"1min":1.629,"5min":2.1269999999999998,"15min":4.9279999999999999},"last":1.5640000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5589999999999999,"15min":1.421},"min":{"1min":1.298,"5min":0.97299999999999998,"15min":0.45300000000000001},"max":{"1min":1.756,"5min":2.0600000000000001,"15min":9.0830000000000002},"last":1.9830000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.5,"15min":1.3129999999999999},"min":{"1min":1.071,"5min":0.89500000000000002,"15min":0.41699999999999998},"max":{"1min":1.78,"5min":2.1859999999999999,"15min":6.5419999999999998},"last":1.958}]},{"osd":10,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5049999999999999,"15min":1.4199999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.43099999999999999},"max":{"1min":2.0550000000000002,"5min":2.0640000000000001,"15min":8.3879999999999999},"last":2.0150000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.613,"15min":1.5249999999999999},"min":{"1min":1.238,"5min":0.68899999999999995,"15min":0.43099999999999999},"max":{"1min":2.0640000000000001,"5min":2.093,"15min":8.1270000000000007},"last":2.1429999999999998}]},{"osd":11,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.5580000000000001,"15min":1.5089999999999999},"min":{"1min":1.298,"5min":1.0089999999999999,"15min":0.47199999999999998},"max":{"1min":1.9490000000000001,"5min":2.109,"15min":7.4210000000000003},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.706,"5min":1.6659999999999999,"15min":1.5900000000000001},"min":{"1min":1.3999999999999999,"5min":1,"15min":0.52300000000000002},"max":{"1min":2.1509999999999998,"5min":2.1920000000000002,"15min":8.1539999999999999},"last":2.0750000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:55:56.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0e714d9a4bd2a821113e6318adb87bd06cf81ec1 shell --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-12T16:55:57.547 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:57 smithi169 bash[19604]: cluster 2024-02-12T16:55:55.699133+0000 osd.11 (osd.11) 283 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:57 smithi169 bash[19604]: cluster 2024-02-12T16:55:55.699243+0000 osd.11 (osd.11) 284 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:57 smithi169 bash[19604]: audit 2024-02-12T16:55:55.902928+0000 mgr.y (mgr.14146) 2152 : audit [DBG] from='client.17715 v1:172.21.15.121:0/2245616811' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:55:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:57 smithi169 bash[19604]: cluster 2024-02-12T16:55:56.523869+0000 mgr.y (mgr.14146) 2153 : cluster [DBG] pgmap v3162: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:57.548 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:57 smithi169 bash[19604]: cluster 2024-02-12T16:55:56.616958+0000 mon.a (mon.0) 3727 : cluster [DBG] osdmap e1144: 12 total, 12 up, 12 in 2024-02-12T16:55:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:57 smithi123 bash[19119]: cluster 2024-02-12T16:55:55.699133+0000 osd.11 (osd.11) 283 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:57 smithi123 bash[19119]: cluster 2024-02-12T16:55:55.699243+0000 osd.11 (osd.11) 284 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:57 smithi123 bash[19119]: audit 2024-02-12T16:55:55.902928+0000 mgr.y (mgr.14146) 2152 : audit [DBG] from='client.17715 v1:172.21.15.121:0/2245616811' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:55:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:57 smithi123 bash[19119]: cluster 2024-02-12T16:55:56.523869+0000 mgr.y (mgr.14146) 2153 : cluster [DBG] pgmap v3162: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:57.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:57 smithi123 bash[19119]: cluster 2024-02-12T16:55:56.616958+0000 mon.a (mon.0) 3727 : cluster [DBG] osdmap e1144: 12 total, 12 up, 12 in 2024-02-12T16:55:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:57 smithi121 bash[18489]: cluster 2024-02-12T16:55:55.699133+0000 osd.11 (osd.11) 283 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:57.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:57 smithi121 bash[18489]: cluster 2024-02-12T16:55:55.699243+0000 osd.11 (osd.11) 284 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:57.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:57 smithi121 bash[18489]: audit 2024-02-12T16:55:55.902928+0000 mgr.y (mgr.14146) 2152 : audit [DBG] from='client.17715 v1:172.21.15.121:0/2245616811' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-12T16:55:57.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:57 smithi121 bash[18489]: cluster 2024-02-12T16:55:56.523869+0000 mgr.y (mgr.14146) 2153 : cluster [DBG] pgmap v3162: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:57.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:57 smithi121 bash[18489]: cluster 2024-02-12T16:55:56.616958+0000 mon.a (mon.0) 3727 : cluster [DBG] osdmap e1144: 12 total, 12 up, 12 in 2024-02-12T16:55:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:59 smithi123 bash[19119]: cluster 2024-02-12T16:55:57.995600+0000 osd.5 (osd.5) 287 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:55:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:59 smithi123 bash[19119]: cluster 2024-02-12T16:55:58.076267+0000 osd.5 (osd.5) 288 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:55:59.638 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:59 smithi123 bash[19119]: cluster 2024-02-12T16:55:58.144431+0000 osd.3 (osd.3) 170 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:59 smithi123 bash[19119]: cluster 2024-02-12T16:55:58.144531+0000 osd.3 (osd.3) 171 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:59 smithi123 bash[19119]: cluster 2024-02-12T16:55:58.289737+0000 mon.a (mon.0) 3728 : cluster [DBG] osdmap e1145: 12 total, 12 up, 12 in 2024-02-12T16:55:59.639 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:55:59 smithi123 bash[19119]: cluster 2024-02-12T16:55:58.524520+0000 mgr.y (mgr.14146) 2154 : cluster [DBG] pgmap v3165: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:59 smithi121 bash[18489]: cluster 2024-02-12T16:55:57.995600+0000 osd.5 (osd.5) 287 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:55:59.660 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:59 smithi121 bash[18489]: cluster 2024-02-12T16:55:58.076267+0000 osd.5 (osd.5) 288 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:55:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:59 smithi121 bash[18489]: cluster 2024-02-12T16:55:58.144431+0000 osd.3 (osd.3) 170 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:59 smithi121 bash[18489]: cluster 2024-02-12T16:55:58.144531+0000 osd.3 (osd.3) 171 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:59 smithi121 bash[18489]: cluster 2024-02-12T16:55:58.289737+0000 mon.a (mon.0) 3728 : cluster [DBG] osdmap e1145: 12 total, 12 up, 12 in 2024-02-12T16:55:59.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:55:59 smithi121 bash[18489]: cluster 2024-02-12T16:55:58.524520+0000 mgr.y (mgr.14146) 2154 : cluster [DBG] pgmap v3165: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:55:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:59 smithi169 bash[19604]: cluster 2024-02-12T16:55:57.995600+0000 osd.5 (osd.5) 287 : cluster [DBG] 1.18 scrub starts 2024-02-12T16:55:59.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:59 smithi169 bash[19604]: cluster 2024-02-12T16:55:58.076267+0000 osd.5 (osd.5) 288 : cluster [DBG] 1.18 scrub ok 2024-02-12T16:55:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:59 smithi169 bash[19604]: cluster 2024-02-12T16:55:58.144431+0000 osd.3 (osd.3) 170 : cluster [DBG] purged_snaps scrub starts 2024-02-12T16:55:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:59 smithi169 bash[19604]: cluster 2024-02-12T16:55:58.144531+0000 osd.3 (osd.3) 171 : cluster [DBG] purged_snaps scrub ok 2024-02-12T16:55:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:59 smithi169 bash[19604]: cluster 2024-02-12T16:55:58.289737+0000 mon.a (mon.0) 3728 : cluster [DBG] osdmap e1145: 12 total, 12 up, 12 in 2024-02-12T16:55:59.798 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:55:59 smithi169 bash[19604]: cluster 2024-02-12T16:55:58.524520+0000 mgr.y (mgr.14146) 2154 : cluster [DBG] pgmap v3165: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:01.474 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/mon.a/config 2024-02-12T16:56:01.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:56:01 smithi123 bash[19119]: cluster 2024-02-12T16:56:00.525276+0000 mgr.y (mgr.14146) 2155 : cluster [DBG] pgmap v3166: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:01.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:56:01 smithi121 bash[18489]: cluster 2024-02-12T16:56:00.525276+0000 mgr.y (mgr.14146) 2155 : cluster [DBG] pgmap v3166: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:02.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:56:01 smithi169 bash[19604]: cluster 2024-02-12T16:56:00.525276+0000 mgr.y (mgr.14146) 2155 : cluster [DBG] pgmap v3166: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:03.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:56:03 smithi123 bash[19119]: cluster 2024-02-12T16:56:02.525995+0000 mgr.y (mgr.14146) 2156 : cluster [DBG] pgmap v3167: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:03.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:56:03 smithi121 bash[18489]: cluster 2024-02-12T16:56:02.525995+0000 mgr.y (mgr.14146) 2156 : cluster [DBG] pgmap v3167: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:04.047 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:56:03 smithi169 bash[19604]: cluster 2024-02-12T16:56:02.525995+0000 mgr.y (mgr.14146) 2156 : cluster [DBG] pgmap v3167: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:05.197 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-12T16:56:05.200 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-12T16:56:05.888 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:56:05 smithi123 bash[19119]: cluster 2024-02-12T16:56:04.526437+0000 mgr.y (mgr.14146) 2157 : cluster [DBG] pgmap v3168: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:05.910 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:56:05 smithi121 bash[18489]: cluster 2024-02-12T16:56:04.526437+0000 mgr.y (mgr.14146) 2157 : cluster [DBG] pgmap v3168: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:05.970 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":3168,"stamp":"2024-02-12T16:56:04.526206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10197228,"kb_used_data":8112,"kb_used_omap":92,"kb_used_meta":10188003,"kb_avail":1114744596,"statfs":{"total":1151940427776,"available":1141498466304,"internally_reserved":0,"allocated":8306688,"data_stored":3803568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95164,"internal_metadata":10432515140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13402112,"num_objects":-206,"num_object_clones":0,"num_object_copies":-412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-46,"num_objects_unfound":0,"num_objects_dirty":-206,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-292936,"num_write_kb":-9349025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-20060,"num_bytes_recovered":-1314553856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.320132"},"pg_stats":[{"pgid":"1.1e","version":"0'0","reported_seq":537,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138016+0000","last_change":"2024-02-12T16:54:41.657969+0000","last_active":"2024-02-12T16:55:57.138016+0000","last_peered":"2024-02-12T16:55:57.138016+0000","last_clean":"2024-02-12T16:55:57.138016+0000","last_became_active":"2024-02-12T16:41:18.947003+0000","last_became_peered":"2024-02-12T16:41:18.947003+0000","last_unstale":"2024-02-12T16:55:57.138016+0000","last_undegraded":"2024-02-12T16:55:57.138016+0000","last_fullsized":"2024-02-12T16:55:57.138016+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:41.657854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:15.328480+0000","last_clean_scrub_stamp":"2024-02-12T16:54:41.657854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"0'0","reported_seq":572,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.616352+0000","last_change":"2024-02-12T16:55:06.196564+0000","last_active":"2024-02-12T16:55:56.616352+0000","last_peered":"2024-02-12T16:55:56.616352+0000","last_clean":"2024-02-12T16:55:56.616352+0000","last_became_active":"2024-02-12T16:41:17.943883+0000","last_became_peered":"2024-02-12T16:41:17.943883+0000","last_unstale":"2024-02-12T16:55:56.616352+0000","last_undegraded":"2024-02-12T16:55:56.616352+0000","last_fullsized":"2024-02-12T16:55:56.616352+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:06.196437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:49.771537+0000","last_clean_scrub_stamp":"2024-02-12T16:55:06.196437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"0'0","reported_seq":578,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.621903+0000","last_change":"2024-02-12T16:55:39.053092+0000","last_active":"2024-02-12T16:55:56.621903+0000","last_peered":"2024-02-12T16:55:56.621903+0000","last_clean":"2024-02-12T16:55:56.621903+0000","last_became_active":"2024-02-12T16:47:31.213335+0000","last_became_peered":"2024-02-12T16:47:31.213335+0000","last_unstale":"2024-02-12T16:55:56.621903+0000","last_undegraded":"2024-02-12T16:55:56.621903+0000","last_fullsized":"2024-02-12T16:55:56.621903+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:39.053002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:39.053002+0000","last_clean_scrub_stamp":"2024-02-12T16:55:39.053002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"0'0","reported_seq":378,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138003+0000","last_change":"2024-02-12T16:54:50.856266+0000","last_active":"2024-02-12T16:55:57.138003+0000","last_peered":"2024-02-12T16:55:57.138003+0000","last_clean":"2024-02-12T16:55:57.138003+0000","last_became_active":"2024-02-12T16:39:09.436840+0000","last_became_peered":"2024-02-12T16:39:09.436840+0000","last_unstale":"2024-02-12T16:55:57.138003+0000","last_undegraded":"2024-02-12T16:55:57.138003+0000","last_fullsized":"2024-02-12T16:55:57.138003+0000","mapping_epoch":845,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":846,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:50.856141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:41:21.337430+0000","last_clean_scrub_stamp":"2024-02-12T16:54:50.856141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"0'0","reported_seq":609,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138465+0000","last_change":"2024-02-12T16:55:25.306208+0000","last_active":"2024-02-12T16:55:57.138465+0000","last_peered":"2024-02-12T16:55:57.138465+0000","last_clean":"2024-02-12T16:55:57.138465+0000","last_became_active":"2024-02-12T16:42:40.975420+0000","last_became_peered":"2024-02-12T16:42:40.975420+0000","last_unstale":"2024-02-12T16:55:57.138465+0000","last_undegraded":"2024-02-12T16:55:57.138465+0000","last_fullsized":"2024-02-12T16:55:57.138465+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:25.306108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:22.890642+0000","last_clean_scrub_stamp":"2024-02-12T16:55:25.306108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"0'0","reported_seq":594,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.137865+0000","last_change":"2024-02-12T16:55:11.862493+0000","last_active":"2024-02-12T16:55:57.137865+0000","last_peered":"2024-02-12T16:55:57.137865+0000","last_clean":"2024-02-12T16:55:57.137865+0000","last_became_active":"2024-02-12T16:48:02.220402+0000","last_became_peered":"2024-02-12T16:48:02.220402+0000","last_unstale":"2024-02-12T16:55:57.137865+0000","last_undegraded":"2024-02-12T16:55:57.137865+0000","last_fullsized":"2024-02-12T16:55:57.137865+0000","mapping_epoch":1044,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:11.862304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:55.445363+0000","last_clean_scrub_stamp":"2024-02-12T16:55:11.862304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"0'0","reported_seq":918,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:58.076442+0000","last_change":"2024-02-12T16:55:58.076405+0000","last_active":"2024-02-12T16:55:58.076442+0000","last_peered":"2024-02-12T16:55:58.076442+0000","last_clean":"2024-02-12T16:55:58.076442+0000","last_became_active":"2024-02-12T16:30:49.376949+0000","last_became_peered":"2024-02-12T16:30:49.376949+0000","last_unstale":"2024-02-12T16:55:58.076442+0000","last_undegraded":"2024-02-12T16:55:58.076442+0000","last_fullsized":"2024-02-12T16:55:58.076442+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:58.076273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:48:46.968263+0000","last_clean_scrub_stamp":"2024-02-12T16:55:58.076273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"0'0","reported_seq":561,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.621910+0000","last_change":"2024-02-12T16:55:18.076770+0000","last_active":"2024-02-12T16:55:56.621910+0000","last_peered":"2024-02-12T16:55:56.621910+0000","last_clean":"2024-02-12T16:55:56.621910+0000","last_became_active":"2024-02-12T16:30:55.400680+0000","last_became_peered":"2024-02-12T16:30:55.400680+0000","last_unstale":"2024-02-12T16:55:56.621910+0000","last_undegraded":"2024-02-12T16:55:56.621910+0000","last_fullsized":"2024-02-12T16:55:56.621910+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:18.076678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:55:18.076678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"0'0","reported_seq":556,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.616287+0000","last_change":"2024-02-12T16:54:59.980562+0000","last_active":"2024-02-12T16:55:56.616287+0000","last_peered":"2024-02-12T16:55:56.616287+0000","last_clean":"2024-02-12T16:55:56.616287+0000","last_became_active":"2024-02-12T16:30:55.402237+0000","last_became_peered":"2024-02-12T16:30:55.402237+0000","last_unstale":"2024-02-12T16:55:56.616287+0000","last_undegraded":"2024-02-12T16:55:56.616287+0000","last_fullsized":"2024-02-12T16:55:56.616287+0000","mapping_epoch":665,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:59.980467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:50:06.911728+0000","last_clean_scrub_stamp":"2024-02-12T16:54:59.980467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"0'0","reported_seq":556,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138994+0000","last_change":"2024-02-12T16:55:16.833161+0000","last_active":"2024-02-12T16:55:57.138994+0000","last_peered":"2024-02-12T16:55:57.138994+0000","last_clean":"2024-02-12T16:55:57.138994+0000","last_became_active":"2024-02-12T16:49:21.796837+0000","last_became_peered":"2024-02-12T16:49:21.796837+0000","last_unstale":"2024-02-12T16:55:57.138994+0000","last_undegraded":"2024-02-12T16:55:57.138994+0000","last_fullsized":"2024-02-12T16:55:57.138994+0000","mapping_epoch":1068,"log_start":"0'0","ondisk_log_start":"0'0","created":663,"last_epoch_clean":1069,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:16.833043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:31.160930+0000","last_clean_scrub_stamp":"2024-02-12T16:55:16.833043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"0'0","reported_seq":821,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.134299+0000","last_change":"2024-02-12T16:55:13.864823+0000","last_active":"2024-02-12T16:55:57.134299+0000","last_peered":"2024-02-12T16:55:57.134299+0000","last_clean":"2024-02-12T16:55:57.134299+0000","last_became_active":"2024-02-12T16:35:16.894715+0000","last_became_peered":"2024-02-12T16:35:16.894715+0000","last_unstale":"2024-02-12T16:55:57.134299+0000","last_undegraded":"2024-02-12T16:55:57.134299+0000","last_fullsized":"2024-02-12T16:55:57.134299+0000","mapping_epoch":780,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:13.864693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:49:09.303752+0000","last_clean_scrub_stamp":"2024-02-12T16:55:13.864693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":300,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.139035+0000","last_change":"2024-02-12T16:55:26.697755+0000","last_active":"2024-02-12T16:55:57.139035+0000","last_peered":"2024-02-12T16:55:57.139035+0000","last_clean":"2024-02-12T16:55:57.139035+0000","last_became_active":"2024-02-12T16:42:41.980413+0000","last_became_peered":"2024-02-12T16:42:41.980413+0000","last_unstale":"2024-02-12T16:55:57.139035+0000","last_undegraded":"2024-02-12T16:55:57.139035+0000","last_fullsized":"2024-02-12T16:55:57.139035+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:26.697620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:26.697620+0000","last_clean_scrub_stamp":"2024-02-12T16:55:26.697620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":979,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138154+0000","last_change":"2024-02-12T16:55:18.805323+0000","last_active":"2024-02-12T16:55:57.138154+0000","last_peered":"2024-02-12T16:55:57.138154+0000","last_clean":"2024-02-12T16:55:57.138154+0000","last_became_active":"2024-02-12T16:48:42.347481+0000","last_became_peered":"2024-02-12T16:48:42.347481+0000","last_unstale":"2024-02-12T16:55:57.138154+0000","last_undegraded":"2024-02-12T16:55:57.138154+0000","last_fullsized":"2024-02-12T16:55:57.138154+0000","mapping_epoch":1057,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1058,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:18.805222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:21:49.784906+0000","last_clean_scrub_stamp":"2024-02-12T16:55:18.805222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1289,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138145+0000","last_change":"2024-02-12T16:55:17.625772+0000","last_active":"2024-02-12T16:55:57.138145+0000","last_peered":"2024-02-12T16:55:57.138145+0000","last_clean":"2024-02-12T16:55:57.138145+0000","last_became_active":"2024-02-12T16:41:18.947424+0000","last_became_peered":"2024-02-12T16:41:18.947424+0000","last_unstale":"2024-02-12T16:55:57.138145+0000","last_undegraded":"2024-02-12T16:55:57.138145+0000","last_fullsized":"2024-02-12T16:55:57.138145+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:17.625656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:17.625656+0000","last_clean_scrub_stamp":"2024-02-12T16:55:17.625656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":151,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.621223+0000","last_change":"2024-02-12T16:55:44.015487+0000","last_active":"2024-02-12T16:55:56.621223+0000","last_peered":"2024-02-12T16:55:56.621223+0000","last_clean":"2024-02-12T16:55:56.621223+0000","last_became_active":"2024-02-12T16:47:31.216533+0000","last_became_peered":"2024-02-12T16:47:31.216533+0000","last_unstale":"2024-02-12T16:55:56.621223+0000","last_undegraded":"2024-02-12T16:55:56.621223+0000","last_fullsized":"2024-02-12T16:55:56.621223+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:44.015397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:52:58.274918+0000","last_clean_scrub_stamp":"2024-02-12T16:55:44.015397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":925,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138113+0000","last_change":"2024-02-12T16:55:29.819766+0000","last_active":"2024-02-12T16:55:57.138113+0000","last_peered":"2024-02-12T16:55:57.138113+0000","last_clean":"2024-02-12T16:55:57.138113+0000","last_became_active":"2024-02-12T16:47:43.444392+0000","last_became_peered":"2024-02-12T16:47:43.444392+0000","last_unstale":"2024-02-12T16:55:57.138113+0000","last_undegraded":"2024-02-12T16:55:57.138113+0000","last_fullsized":"2024-02-12T16:55:57.138113+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:29.819662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:29.819662+0000","last_clean_scrub_stamp":"2024-02-12T16:55:29.819662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":968,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.137857+0000","last_change":"2024-02-12T16:55:40.729541+0000","last_active":"2024-02-12T16:55:57.137857+0000","last_peered":"2024-02-12T16:55:57.137857+0000","last_clean":"2024-02-12T16:55:57.137857+0000","last_became_active":"2024-02-12T16:48:20.617116+0000","last_became_peered":"2024-02-12T16:48:20.617116+0000","last_unstale":"2024-02-12T16:55:57.137857+0000","last_undegraded":"2024-02-12T16:55:57.137857+0000","last_fullsized":"2024-02-12T16:55:57.137857+0000","mapping_epoch":1047,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1048,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:40.729408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:40.729408+0000","last_clean_scrub_stamp":"2024-02-12T16:55:40.729408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":789,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.816452+0000","last_change":"2024-02-12T16:55:41.884907+0000","last_active":"2024-02-12T16:55:56.816452+0000","last_peered":"2024-02-12T16:55:56.816452+0000","last_clean":"2024-02-12T16:55:56.816452+0000","last_became_active":"2024-02-12T16:47:31.215614+0000","last_became_peered":"2024-02-12T16:47:31.215614+0000","last_unstale":"2024-02-12T16:55:56.816452+0000","last_undegraded":"2024-02-12T16:55:56.816452+0000","last_fullsized":"2024-02-12T16:55:56.816452+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:41.884816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:46:39.182350+0000","last_clean_scrub_stamp":"2024-02-12T16:55:41.884816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1252,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.139074+0000","last_change":"2024-02-12T16:54:43.757481+0000","last_active":"2024-02-12T16:55:57.139074+0000","last_peered":"2024-02-12T16:55:57.139074+0000","last_clean":"2024-02-12T16:55:57.139074+0000","last_became_active":"2024-02-12T16:42:41.321180+0000","last_became_peered":"2024-02-12T16:42:41.321180+0000","last_unstale":"2024-02-12T16:55:57.139074+0000","last_undegraded":"2024-02-12T16:55:57.139074+0000","last_fullsized":"2024-02-12T16:55:57.139074+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:43.757263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:37.472009+0000","last_clean_scrub_stamp":"2024-02-12T16:54:43.757263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":676,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.139097+0000","last_change":"2024-02-12T16:55:53.698219+0000","last_active":"2024-02-12T16:55:57.139097+0000","last_peered":"2024-02-12T16:55:57.139097+0000","last_clean":"2024-02-12T16:55:57.139097+0000","last_became_active":"2024-02-12T16:42:40.975171+0000","last_became_peered":"2024-02-12T16:42:40.975171+0000","last_unstale":"2024-02-12T16:55:57.139097+0000","last_undegraded":"2024-02-12T16:55:57.139097+0000","last_fullsized":"2024-02-12T16:55:57.139097+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:53.698127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:03:59.776674+0000","last_clean_scrub_stamp":"2024-02-12T16:55:53.698127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":73,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.816452+0000","last_change":"2024-02-12T16:55:15.783912+0000","last_active":"2024-02-12T16:55:56.816452+0000","last_peered":"2024-02-12T16:55:56.816452+0000","last_clean":"2024-02-12T16:55:56.816452+0000","last_became_active":"2024-02-12T16:51:20.593832+0000","last_became_peered":"2024-02-12T16:51:20.593832+0000","last_unstale":"2024-02-12T16:55:56.816452+0000","last_undegraded":"2024-02-12T16:55:56.816452+0000","last_fullsized":"2024-02-12T16:55:56.816452+0000","mapping_epoch":1100,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1101,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:15.783819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:23.682328+0000","last_clean_scrub_stamp":"2024-02-12T16:55:15.783819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":146,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.621127+0000","last_change":"2024-02-12T16:55:05.331663+0000","last_active":"2024-02-12T16:55:56.621127+0000","last_peered":"2024-02-12T16:55:56.621127+0000","last_clean":"2024-02-12T16:55:56.621127+0000","last_became_active":"2024-02-12T16:47:31.213215+0000","last_became_peered":"2024-02-12T16:47:31.213215+0000","last_unstale":"2024-02-12T16:55:56.621127+0000","last_undegraded":"2024-02-12T16:55:56.621127+0000","last_fullsized":"2024-02-12T16:55:56.621127+0000","mapping_epoch":1037,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1038,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:05.331571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:53:56.183999+0000","last_clean_scrub_stamp":"2024-02-12T16:55:05.331571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":725,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138673+0000","last_change":"2024-02-12T16:55:20.357147+0000","last_active":"2024-02-12T16:55:57.138673+0000","last_peered":"2024-02-12T16:55:57.138673+0000","last_clean":"2024-02-12T16:55:57.138673+0000","last_became_active":"2024-02-12T16:42:40.975493+0000","last_became_peered":"2024-02-12T16:42:40.975493+0000","last_unstale":"2024-02-12T16:55:57.138673+0000","last_undegraded":"2024-02-12T16:55:57.138673+0000","last_fullsized":"2024-02-12T16:55:57.138673+0000","mapping_epoch":923,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":924,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:20.357026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:42:42.373026+0000","last_clean_scrub_stamp":"2024-02-12T16:55:20.357026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":1247,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138880+0000","last_change":"2024-02-12T16:55:35.692950+0000","last_active":"2024-02-12T16:55:57.138880+0000","last_peered":"2024-02-12T16:55:57.138880+0000","last_clean":"2024-02-12T16:55:57.138880+0000","last_became_active":"2024-02-12T16:47:43.444507+0000","last_became_peered":"2024-02-12T16:47:43.444507+0000","last_unstale":"2024-02-12T16:55:57.138880+0000","last_undegraded":"2024-02-12T16:55:57.138880+0000","last_fullsized":"2024-02-12T16:55:57.138880+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:35.692738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:48:56.268062+0000","last_clean_scrub_stamp":"2024-02-12T16:55:35.692738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":907,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.749136+0000","last_change":"2024-02-12T16:55:32.907642+0000","last_active":"2024-02-12T16:55:56.749136+0000","last_peered":"2024-02-12T16:55:56.749136+0000","last_clean":"2024-02-12T16:55:56.749136+0000","last_became_active":"2024-02-12T16:30:49.376796+0000","last_became_peered":"2024-02-12T16:30:49.376796+0000","last_unstale":"2024-02-12T16:55:56.749136+0000","last_undegraded":"2024-02-12T16:55:56.749136+0000","last_fullsized":"2024-02-12T16:55:56.749136+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:32.907539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:32.907539+0000","last_clean_scrub_stamp":"2024-02-12T16:55:32.907539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":302,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138137+0000","last_change":"2024-02-12T16:55:00.621406+0000","last_active":"2024-02-12T16:55:57.138137+0000","last_peered":"2024-02-12T16:55:57.138137+0000","last_clean":"2024-02-12T16:55:57.138137+0000","last_became_active":"2024-02-12T16:41:17.958639+0000","last_became_peered":"2024-02-12T16:41:17.958639+0000","last_unstale":"2024-02-12T16:55:57.138137+0000","last_undegraded":"2024-02-12T16:55:57.138137+0000","last_fullsized":"2024-02-12T16:55:57.138137+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:00.621284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:00.621284+0000","last_clean_scrub_stamp":"2024-02-12T16:55:00.621284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":147,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138154+0000","last_change":"2024-02-12T16:55:14.754945+0000","last_active":"2024-02-12T16:55:57.138154+0000","last_peered":"2024-02-12T16:55:57.138154+0000","last_clean":"2024-02-12T16:55:57.138154+0000","last_became_active":"2024-02-12T16:47:43.439741+0000","last_became_peered":"2024-02-12T16:47:43.439741+0000","last_unstale":"2024-02-12T16:55:57.138154+0000","last_undegraded":"2024-02-12T16:55:57.138154+0000","last_fullsized":"2024-02-12T16:55:57.138154+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":365,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:14.754852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:51:36.632610+0000","last_clean_scrub_stamp":"2024-02-12T16:55:14.754852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":485,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.138594+0000","last_change":"2024-02-12T16:55:54.416700+0000","last_active":"2024-02-12T16:55:57.138594+0000","last_peered":"2024-02-12T16:55:57.138594+0000","last_clean":"2024-02-12T16:55:57.138594+0000","last_became_active":"2024-02-12T16:36:17.849909+0000","last_became_peered":"2024-02-12T16:36:17.849909+0000","last_unstale":"2024-02-12T16:55:57.138594+0000","last_undegraded":"2024-02-12T16:55:57.138594+0000","last_fullsized":"2024-02-12T16:55:57.138594+0000","mapping_epoch":796,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":797,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:54.416597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:48:33.063109+0000","last_clean_scrub_stamp":"2024-02-12T16:55:54.416597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"0'0","reported_seq":139,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:57.134526+0000","last_change":"2024-02-12T16:55:13.998488+0000","last_active":"2024-02-12T16:55:57.134526+0000","last_peered":"2024-02-12T16:55:57.134526+0000","last_clean":"2024-02-12T16:55:57.134526+0000","last_became_active":"2024-02-12T16:48:02.884693+0000","last_became_peered":"2024-02-12T16:48:02.884693+0000","last_unstale":"2024-02-12T16:55:57.134526+0000","last_undegraded":"2024-02-12T16:55:57.134526+0000","last_fullsized":"2024-02-12T16:55:57.134526+0000","mapping_epoch":1044,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:13.998294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:55:13.998294+0000","last_clean_scrub_stamp":"2024-02-12T16:55:13.998294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"0'0","reported_seq":796,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.749035+0000","last_change":"2024-02-12T16:55:09.168990+0000","last_active":"2024-02-12T16:55:56.749035+0000","last_peered":"2024-02-12T16:55:56.749035+0000","last_clean":"2024-02-12T16:55:56.749035+0000","last_became_active":"2024-02-12T16:50:22.212417+0000","last_became_peered":"2024-02-12T16:50:22.212417+0000","last_unstale":"2024-02-12T16:55:56.749035+0000","last_undegraded":"2024-02-12T16:55:56.749035+0000","last_fullsized":"2024-02-12T16:55:56.749035+0000","mapping_epoch":1086,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":1087,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:55:09.168898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:23:44.185848+0000","last_clean_scrub_stamp":"2024-02-12T16:55:09.168898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"0'0","reported_seq":739,"reported_epoch":1144,"state":"active+clean","last_fresh":"2024-02-12T16:55:56.616374+0000","last_change":"2024-02-12T16:54:59.072627+0000","last_active":"2024-02-12T16:55:56.616374+0000","last_peered":"2024-02-12T16:55:56.616374+0000","last_clean":"2024-02-12T16:55:56.616374+0000","last_became_active":"2024-02-12T16:41:17.950595+0000","last_became_peered":"2024-02-12T16:41:17.950595+0000","last_unstale":"2024-02-12T16:55:56.616374+0000","last_undegraded":"2024-02-12T16:55:56.616374+0000","last_fullsized":"2024-02-12T16:55:56.616374+0000","mapping_epoch":889,"log_start":"0'0","ondisk_log_start":"0'0","created":410,"last_epoch_clean":890,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-12T16:54:59.072524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-12T16:54:59.072524+0000","last_clean_scrub_stamp":"2024-02-12T16:54:59.072524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":93,"seq":399431959224,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897648,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":896832,"kb_avail":92847504,"statfs":{"total":95995035648,"available":95075844096,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":918355968},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":0.873},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.248},"max":{"1min":1.653,"5min":1.653,"15min":3.9159999999999999},"last":1.323},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0680000000000001,"15min":1.0920000000000001},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.33500000000000002},"max":{"1min":1.1870000000000001,"5min":1.6399999999999999,"15min":5.7009999999999996},"last":1.087}]},{"osd":1,"last update":"Mon Feb 12 16:55:05 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.99299999999999999,"15min":0.998},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.58099999999999996},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.6240000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.2330000000000001,"15min":1.1659999999999999},"min":{"1min":0.70999999999999996,"5min":0.68600000000000005,"15min":0.38100000000000001},"max":{"1min":1.5049999999999999,"5min":1.7809999999999999,"15min":2.202},"last":1.6459999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.95099999999999996,"15min":1.734},"min":{"1min":0.625,"5min":0.60999999999999999,"15min":0.308},"max":{"1min":1.3200000000000001,"5min":1.4910000000000001,"15min":219.56100000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.23,"5min":1.129,"15min":1.7010000000000001},"min":{"1min":0.80000000000000004,"5min":0.65300000000000002,"15min":0.26900000000000002},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":219.79300000000001},"last":1.0600000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:56 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.117,"15min":1.371},"min":{"1min":0.71799999999999997,"5min":0.57599999999999996,"15min":0.26300000000000001},"max":{"1min":1.651,"5min":1.8959999999999999,"15min":46.445},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.484,"5min":1.367,"15min":1.462},"min":{"1min":1.0980000000000001,"5min":0.753,"15min":0.28299999999999997},"max":{"1min":1.944,"5min":1.9770000000000001,"15min":46.774999999999999},"last":1.542}]},{"osd":4,"last update":"Mon Feb 12 16:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.93300000000000005,"15min":0.91800000000000004},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.4239999999999999,"5min":6.1669999999999998,"15min":6.2039999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.304,"5min":1.234,"15min":1.121},"min":{"1min":0.48099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.4989999999999997,"5min":5.7469999999999999,"15min":7.3920000000000003},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.359,"15min":1.2669999999999999},"min":{"1min":0.83799999999999997,"5min":0.70599999999999996,"15min":0.37},"max":{"1min":2.1520000000000001,"5min":2.1579999999999999,"15min":5.1280000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.167,"15min":1.202},"min":{"1min":0.90400000000000003,"5min":0.77400000000000002,"15min":0.38300000000000001},"max":{"1min":1.405,"5min":1.8009999999999999,"15min":5.8200000000000003},"last":1.159}]},{"osd":6,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.137,"15min":1.143},"min":{"1min":0.80400000000000005,"5min":0.73799999999999999,"15min":0.35199999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":6.202},"last":1.474},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.254,"15min":1.2190000000000001},"min":{"1min":0.878,"5min":0.81499999999999995,"15min":0.40600000000000003},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":6.1289999999999996},"last":1.1799999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:55:05 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3460000000000001,"15min":1.3029999999999999},"min":{"1min":1.0649999999999999,"5min":0.73199999999999998,"15min":0.38200000000000001},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":5.9660000000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5760000000000001,"15min":1.2729999999999999},"min":{"1min":1.2749999999999999,"5min":0.747,"15min":0.41299999999999998},"max":{"1min":1.736,"5min":2.2240000000000002,"15min":5.0739999999999998},"last":1.71}]},{"osd":8,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.302,"15min":1.224},"min":{"1min":0.998,"5min":0.55900000000000005,"15min":0.32800000000000001},"max":{"1min":2.141,"5min":2.141,"15min":6.306},"last":1.351},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5720000000000001,"15min":1.2969999999999999},"min":{"1min":0.89600000000000002,"5min":0.66100000000000003,"15min":0.35199999999999998},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":3.7530000000000001},"last":1.764}]},{"osd":9,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.613,"15min":1.3440000000000001},"min":{"1min":1.173,"5min":0.80000000000000004,"15min":0.437},"max":{"1min":2.331,"5min":2.331,"15min":3.1480000000000001},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.718,"15min":1.4299999999999999},"min":{"1min":1.421,"5min":1.1100000000000001,"15min":0.47399999999999998},"max":{"1min":2.488,"5min":2.488,"15min":3.21},"last":1.851}]},{"osd":10,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.371,"15min":1.407},"min":{"1min":1.1479999999999999,"5min":0.99099999999999999,"15min":0.39700000000000002},"max":{"1min":1.583,"5min":1.909,"15min":9.6159999999999997},"last":1.375},{"interface":"front","average":{"1min":1.508,"5min":1.516,"15min":1.488},"min":{"1min":1.2430000000000001,"5min":1.054,"15min":0.39500000000000002},"max":{"1min":2.4460000000000002,"5min":2.4460000000000002,"15min":8.2129999999999992},"last":1.583}]}]},{"osd":10,"up_from":83,"seq":356482286273,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899244,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":898560,"kb_avail":92845908,"statfs":{"total":95995035648,"available":95074209792,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":920125440},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:14 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.95099999999999996,"15min":0.97399999999999998},"min":{"1min":0.755,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.466,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.103},{"interface":"front","average":{"1min":1.0760000000000001,"5min":0.99199999999999999,"15min":0.99099999999999999},"min":{"1min":0.79600000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.359,"5min":1.8129999999999999,"15min":1.863},"last":1.0309999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.85699999999999998,"15min":1.044},"min":{"1min":0.44,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2909999999999999,"5min":5.524,"15min":5.524},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0549999999999999,"15min":1.1060000000000001},"min":{"1min":0.61199999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.542,"5min":5.5609999999999999,"15min":7.2000000000000002},"last":1.3220000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69399999999999995,"5min":0.59999999999999998,"15min":0.314},"max":{"1min":1.4770000000000001,"5min":1.544,"15min":9.4179999999999993},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.91400000000000003,"15min":0.93400000000000005},"min":{"1min":0.66000000000000003,"5min":0.57499999999999996,"15min":0.34100000000000003},"max":{"1min":1.1259999999999999,"5min":1.387,"15min":5.6289999999999996},"last":0.88800000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.008,"15min":1.042},"min":{"1min":0.71199999999999997,"5min":0.68000000000000005,"15min":0.28599999999999998},"max":{"1min":1.4339999999999999,"5min":1.4710000000000001,"15min":9.3699999999999992},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1299999999999999,"15min":1.073},"min":{"1min":0.84399999999999997,"5min":0.67100000000000004,"15min":0.28699999999999998},"max":{"1min":1.673,"5min":1.675,"15min":5.6769999999999996},"last":1.0609999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.2889999999999999,"15min":1.1930000000000001},"min":{"1min":0.88800000000000001,"5min":0.68300000000000005,"15min":0.32500000000000001},"max":{"1min":1.643,"5min":2.1320000000000001,"15min":5.8380000000000001},"last":1.375},{"interface":"front","average":{"1min":1.478,"5min":1.4279999999999999,"15min":1.2350000000000001},"min":{"1min":1.3200000000000001,"5min":0.89200000000000002,"15min":0.38600000000000001},"max":{"1min":1.788,"5min":2.0329999999999999,"15min":5.649},"last":1.4159999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.456},"min":{"1min":0.78900000000000003,"5min":0.73899999999999999,"15min":0.38800000000000001},"max":{"1min":1.8,"5min":1.9510000000000001,"15min":74.137},"last":1.153},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3580000000000001,"15min":1.49},"min":{"1min":0.98899999999999999,"5min":0.78800000000000003,"15min":0.44},"max":{"1min":1.7649999999999999,"5min":1.835,"15min":74.063999999999993},"last":1.3500000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.304},"min":{"1min":1.0860000000000001,"5min":0.874,"15min":0.38400000000000001},"max":{"1min":1.8480000000000001,"5min":2.0219999999999998,"15min":5.7729999999999997},"last":1.264},{"interface":"front","average":{"1min":1.583,"5min":1.5620000000000001,"15min":1.391},"min":{"1min":1.3380000000000001,"5min":0.76500000000000001,"15min":0.44800000000000001},"max":{"1min":1.8999999999999999,"5min":1.972,"15min":5.7960000000000003},"last":1.54}]},{"osd":7,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.4470000000000001,"15min":2.1640000000000001},"min":{"1min":1.099,"5min":0.996,"15min":0.45000000000000001},"max":{"1min":1.95,"5min":2.0659999999999998,"15min":74.007000000000005},"last":1.171},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.321,"15min":1.9690000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.42299999999999999},"max":{"1min":1.754,"5min":1.754,"15min":73.819000000000003},"last":1.2909999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.3380000000000001,"15min":1.2969999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.34100000000000003},"max":{"1min":1.835,"5min":1.9259999999999999,"15min":7.5810000000000004},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.552,"15min":1.417},"min":{"1min":1.0660000000000001,"5min":0.98799999999999999,"15min":0.41299999999999998},"max":{"1min":2.145,"5min":2.2050000000000001,"15min":9.5090000000000003},"last":1.254}]},{"osd":9,"last update":"Mon Feb 12 16:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.472,"15min":1.3300000000000001},"min":{"1min":0.92400000000000004,"5min":0.88800000000000001,"15min":0.35399999999999998},"max":{"1min":1.8540000000000001,"5min":2.218,"15min":5.8029999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.659,"15min":1.4319999999999999},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.378},"max":{"1min":1.9970000000000001,"5min":2.2789999999999999,"15min":5.7519999999999998},"last":1.4770000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.4570000000000001,"15min":1.4430000000000001},"min":{"1min":1.2390000000000001,"5min":0.94299999999999995,"15min":0.39900000000000002},"max":{"1min":1.756,"5min":1.8680000000000001,"15min":8.7569999999999997},"last":1.431},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.708,"15min":1.581},"min":{"1min":1.4199999999999999,"5min":1.272,"15min":0.44700000000000001},"max":{"1min":2.1299999999999999,"5min":2.1819999999999999,"15min":7.891},"last":1.5900000000000001}]}]},{"osd":9,"up_from":74,"seq":317827580617,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898348,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":897664,"kb_avail":92846804,"statfs":{"total":95995035648,"available":95075127296,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":919207936},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96399999999999997,"15min":0.84599999999999997},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.25700000000000001},"max":{"1min":1.4330000000000001,"5min":1.518,"15min":5.1180000000000003},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.111,"15min":0.96499999999999997},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.28699999999999998},"max":{"1min":1.3839999999999999,"5min":1.6479999999999999,"15min":5.0039999999999996},"last":0.80700000000000005}]},{"osd":1,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.2629999999999999,"15min":1.603},"min":{"1min":0.66600000000000004,"5min":0.63,"15min":0.29799999999999999},"max":{"1min":1.9059999999999999,"5min":1.919,"15min":157.74100000000001},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.026,"5min":0.98599999999999999,"15min":1.502},"min":{"1min":0.878,"5min":0.64900000000000002,"15min":0.317},"max":{"1min":1.1870000000000001,"5min":1.3700000000000001,"15min":157.709},"last":0.91000000000000003}]},{"osd":2,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.095,"15min":0.97699999999999998},"min":{"1min":0.69799999999999995,"5min":0.59699999999999998,"15min":0.311},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":5.0780000000000003},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.256,"15min":1.0640000000000001},"min":{"1min":0.73099999999999998,"5min":0.52700000000000002,"15min":0.33500000000000002},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":5.4100000000000001},"last":1.161}]},{"osd":3,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.206,"15min":1.0469999999999999},"min":{"1min":0.879,"5min":0.67200000000000004,"15min":0.34399999999999997},"max":{"1min":1.6579999999999999,"5min":1.9039999999999999,"15min":5.4699999999999998},"last":0.999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3340000000000001,"15min":1.1679999999999999},"min":{"1min":0.79100000000000004,"5min":0.78900000000000003,"15min":0.376},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":5.4420000000000002},"last":1.504}]},{"osd":4,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3560000000000001,"15min":1.1619999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.36199999999999999},"max":{"1min":1.9299999999999999,"5min":2.133,"15min":5.2949999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.4790000000000001,"15min":1.2689999999999999},"min":{"1min":1.1679999999999999,"5min":0.97099999999999997,"15min":0.42599999999999999},"max":{"1min":2.0259999999999998,"5min":2.2480000000000002,"15min":5.3170000000000002},"last":1.3979999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.488,"15min":1.246},"min":{"1min":1.03,"5min":0.72199999999999998,"15min":0.373},"max":{"1min":2.117,"5min":2.117,"15min":5.6280000000000001},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.4750000000000001,"15min":1.2250000000000001},"min":{"1min":1.117,"5min":0.94099999999999995,"15min":0.42599999999999999},"max":{"1min":2.1899999999999999,"5min":2.1909999999999998,"15min":5.085},"last":1.4330000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4410000000000001,"15min":1.244},"min":{"1min":0.97499999999999998,"5min":0.60299999999999998,"15min":0.42699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":5.2729999999999997},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.208,"15min":1.119},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.40699999999999997},"max":{"1min":1.3919999999999999,"5min":1.8799999999999999,"15min":5.0499999999999998},"last":0.73599999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.95799999999999996,"15min":1.169},"min":{"1min":0.45600000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":6.7050000000000001,"15min":7.9489999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":1.1950000000000001,"15min":1.2529999999999999},"min":{"1min":0.53700000000000003,"5min":0.44800000000000001,"15min":0.43099999999999999},"max":{"1min":1.3220000000000001,"5min":6.7720000000000002,"15min":7.9160000000000004},"last":0.56799999999999995}]},{"osd":8,"last update":"Mon Feb 12 16:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.508,"15min":1.3260000000000001},"min":{"1min":0.88600000000000001,"5min":0.61699999999999999,"15min":0.38100000000000001},"max":{"1min":2.085,"5min":2.3250000000000002,"15min":5.2460000000000004},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.329,"15min":1.2170000000000001},"min":{"1min":1.0609999999999999,"5min":0.74199999999999999,"15min":0.35299999999999998},"max":{"1min":1.706,"5min":1.9430000000000001,"15min":5.2160000000000002},"last":1.113}]},{"osd":10,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.6120000000000001,"15min":1.607},"min":{"1min":0.97299999999999998,"5min":0.88400000000000001,"15min":0.38500000000000001},"max":{"1min":1.925,"5min":2.089,"15min":66.311999999999998},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.6950000000000001,"15min":1.7090000000000001},"min":{"1min":1.365,"5min":0.879,"15min":0.34599999999999997},"max":{"1min":2.1339999999999999,"5min":2.157,"15min":59.326999999999998},"last":1.4850000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.411,"15min":1.304},"min":{"1min":1.0920000000000001,"5min":0.83999999999999997,"15min":0.371},"max":{"1min":1.839,"5min":1.8819999999999999,"15min":5.6609999999999996},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.704,"5min":1.6870000000000001,"15min":1.4390000000000001},"min":{"1min":1.369,"5min":1.2310000000000001,"15min":0.441},"max":{"1min":2.29,"5min":2.29,"15min":5.5659999999999998},"last":1.5600000000000001}]}]},{"osd":8,"up_from":65,"seq":279172874963,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898416,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":897600,"kb_avail":92846736,"statfs":{"total":95995035648,"available":95075057664,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":919142400},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.02,"15min":0.86599999999999999},"min":{"1min":0.61799999999999999,"5min":0.60999999999999999,"15min":0.27500000000000002},"max":{"1min":1.387,"5min":1.732,"15min":5.96},"last":1.577},{"interface":"front","average":{"1min":1.036,"5min":1.0269999999999999,"15min":0.91500000000000004},"min":{"1min":0.69099999999999995,"5min":0.60799999999999998,"15min":0.308},"max":{"1min":1.7849999999999999,"5min":1.9119999999999999,"15min":6.7729999999999997},"last":0.86799999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.67400000000000004,"5min":0.63200000000000001,"15min":0.27900000000000003},"max":{"1min":1.048,"5min":1.4339999999999999,"15min":8.1210000000000004},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.101,"15min":0.98899999999999999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.27300000000000002},"max":{"1min":1.5109999999999999,"5min":1.5960000000000001,"15min":8.4819999999999993},"last":1.1259999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.85899999999999999,"15min":0.877},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.29299999999999998},"max":{"1min":1.9399999999999999,"5min":3.2320000000000002,"15min":8.6430000000000007},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2609999999999999,"15min":1.091},"min":{"1min":0.48999999999999999,"5min":0.378,"15min":0.26800000000000002},"max":{"1min":2.0070000000000001,"5min":8.9399999999999995,"15min":8.9399999999999995},"last":0.84099999999999997}]},{"osd":3,"last update":"Mon Feb 12 16:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.157,"15min":1.0700000000000001},"min":{"1min":0.70899999999999996,"5min":0.60199999999999998,"15min":0.29399999999999998},"max":{"1min":1.631,"5min":1.6830000000000001,"15min":10.231},"last":1.661},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.98099999999999998,"15min":1.0089999999999999},"min":{"1min":0.70499999999999996,"5min":0.66400000000000003,"15min":0.34200000000000003},"max":{"1min":1.361,"5min":1.6519999999999999,"15min":10.265000000000001},"last":1.2030000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.302,"15min":1.1459999999999999},"min":{"1min":0.93400000000000005,"5min":0.66200000000000003,"15min":0.313},"max":{"1min":1.593,"5min":1.873,"15min":3.3620000000000001},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.381,"15min":1.127},"min":{"1min":1.143,"5min":0.628,"15min":0.38200000000000001},"max":{"1min":1.9379999999999999,"5min":2.1840000000000002,"15min":4.4500000000000002},"last":1.8129999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:55:06 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.1299999999999999,"15min":1.071},"min":{"1min":0.81899999999999995,"5min":0.624,"15min":0.38},"max":{"1min":1.5529999999999999,"5min":1.633,"15min":4.5069999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.518,"5min":1.4330000000000001,"15min":1.254},"min":{"1min":1.0940000000000001,"5min":0.86699999999999999,"15min":0.42399999999999999},"max":{"1min":1.9319999999999999,"5min":2.097,"15min":6.681},"last":1.738}]},{"osd":6,"last update":"Mon Feb 12 16:55:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.49,"15min":1.2749999999999999},"min":{"1min":0.97599999999999998,"5min":0.94699999999999995,"15min":0.43099999999999999},"max":{"1min":2.1139999999999999,"5min":2.1139999999999999,"15min":5.5369999999999999},"last":1.893},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.224,"15min":1.115},"min":{"1min":0.83199999999999996,"5min":0.78800000000000003,"15min":0.41399999999999998},"max":{"1min":1.702,"5min":1.829,"15min":4.5140000000000002},"last":1.478}]},{"osd":7,"last update":"Mon Feb 12 16:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.149,"15min":1.141},"min":{"1min":0.97899999999999998,"5min":0.77800000000000002,"15min":0.40500000000000003},"max":{"1min":1.661,"5min":1.71,"15min":6.5970000000000004},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.5860000000000001,"15min":1.3680000000000001},"min":{"1min":1.0780000000000001,"5min":0.93400000000000005,"15min":0.41599999999999998},"max":{"1min":2.0659999999999998,"5min":2.27,"15min":4.0069999999999997},"last":1.9750000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.5029999999999999,"15min":1.2889999999999999},"min":{"1min":0.58399999999999996,"5min":0.52500000000000002,"15min":0.40600000000000003},"max":{"1min":2.0110000000000001,"5min":2.3340000000000001,"15min":4.407},"last":2.052},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.6919999999999999,"15min":1.4259999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.49199999999999999},"max":{"1min":2.1139999999999999,"5min":2.4430000000000001,"15min":4.5380000000000003},"last":2.113}]},{"osd":10,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.2949999999999999,"15min":1.1499999999999999},"min":{"1min":0.80000000000000004,"5min":0.71999999999999997,"15min":0.39500000000000002},"max":{"1min":1.663,"5min":1.8759999999999999,"15min":4.5289999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.365,"15min":1.24},"min":{"1min":1.1579999999999999,"5min":0.875,"15min":0.35399999999999998},"max":{"1min":1.7709999999999999,"5min":2,"15min":5.4169999999999998},"last":1.345}]},{"osd":11,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3180000000000001,"15min":1.254},"min":{"1min":0.49199999999999999,"5min":0.441,"15min":0.437},"max":{"1min":1.853,"5min":1.9670000000000001,"15min":6.1699999999999999},"last":1.429},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4830000000000001,"15min":1.3720000000000001},"min":{"1min":0.81499999999999995,"5min":0.65600000000000003,"15min":0.45800000000000002},"max":{"1min":1.897,"5min":2.0299999999999998,"15min":5.7220000000000004},"last":1.508}]}]},{"osd":7,"up_from":923,"seq":3964254814371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319280,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":318463,"kb_avail":93425872,"statfs":{"total":95995035648,"available":95668092928,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":326106164},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.93200000000000005,"15min":0.89700000000000002},"min":{"1min":0.746,"5min":0.62,"15min":0.27100000000000002},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":6.4630000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.99399999999999999,"15min":0.92100000000000004},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.30199999999999999},"max":{"1min":1.351,"5min":1.3640000000000001,"15min":5.2110000000000003},"last":0.90400000000000003}]},{"osd":1,"last update":"Mon Feb 12 16:46:47 2024","interfaces":[{"interface":"back","average":{"1min":2.778,"5min":3.3290000000000002,"15min":1.8089999999999999},"min":{"1min":0.35099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":32.298000000000002,"5min":177.82400000000001,"15min":177.82400000000001},"last":1.131},{"interface":"front","average":{"1min":2.7519999999999998,"5min":3.153,"15min":1.429},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":31.824000000000002,"5min":177.30500000000001,"15min":177.30500000000001},"last":1.47}]},{"osd":2,"last update":"Mon Feb 12 16:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.004,"15min":0.876},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.318},"max":{"1min":1.1779999999999999,"5min":1.383,"15min":5.2359999999999998},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.071,"15min":1.0389999999999999},"min":{"1min":0.81899999999999995,"5min":0.61699999999999999,"15min":0.32900000000000001},"max":{"1min":1.3089999999999999,"5min":1.627,"15min":5.3630000000000004},"last":0.72199999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1100000000000001,"15min":1.151},"min":{"1min":0.84599999999999997,"5min":0.69999999999999996,"15min":0.38200000000000001},"max":{"1min":1.409,"5min":1.5049999999999999,"15min":9.1029999999999998},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.28,"15min":1.2569999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.443},"max":{"1min":1.5640000000000001,"5min":1.78,"15min":9.1470000000000002},"last":1.1899999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.84999999999999998},"min":{"1min":0.71899999999999997,"5min":0.63200000000000001,"15min":0.28199999999999997},"max":{"1min":1.2490000000000001,"5min":1.4810000000000001,"15min":5.4199999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.107,"5min":1.1160000000000001,"15min":1.0289999999999999},"min":{"1min":0.82699999999999996,"5min":0.66200000000000003,"15min":0.33100000000000002},"max":{"1min":1.444,"5min":1.5529999999999999,"15min":8.4719999999999995},"last":1.079}]},{"osd":5,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.133,"15min":1.149},"min":{"1min":0.81799999999999995,"5min":0.625,"15min":0.36399999999999999},"max":{"1min":1.415,"5min":1.7230000000000001,"15min":6.4909999999999997},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.2749999999999999,"15min":1.3089999999999999},"min":{"1min":0.80000000000000004,"5min":0.70599999999999996,"15min":0.36399999999999999},"max":{"1min":1.6930000000000001,"5min":1.6950000000000001,"15min":5.7069999999999999},"last":1.2909999999999999}]},{"osd":6,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1930000000000001,"15min":1.091},"min":{"1min":0.67900000000000005,"5min":0.49199999999999999,"15min":0.38400000000000001},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":6.5069999999999997},"last":1.135},{"interface":"front","average":{"1min":1.212,"5min":1.2090000000000001,"15min":1.052},"min":{"1min":0.94199999999999995,"5min":0.66600000000000004,"15min":0.36899999999999999},"max":{"1min":1.542,"5min":1.6040000000000001,"15min":5.508},"last":1.0369999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4319999999999999,"15min":1.272},"min":{"1min":1.026,"5min":1.026,"15min":0.437},"max":{"1min":1.7310000000000001,"5min":2.355,"15min":5.6719999999999997},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.472,"15min":1.25},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.46999999999999997},"max":{"1min":1.925,"5min":2.093,"15min":6.5880000000000001},"last":1.3600000000000001}]},{"osd":9,"last update":"Mon Feb 12 16:55:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5069999999999999,"15min":1.53},"min":{"1min":1.27,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":1.891,"5min":1.944,"15min":2.1890000000000001},"last":1.232},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.54,"15min":1.494},"min":{"1min":1.238,"5min":1.103,"15min":1.034},"max":{"1min":1.8939999999999999,"5min":2.2400000000000002,"15min":2.4380000000000002},"last":1.4399999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.5449999999999999,"15min":1.3720000000000001},"min":{"1min":1.361,"5min":0.98199999999999998,"15min":0.50600000000000001},"max":{"1min":1.8759999999999999,"5min":1.9179999999999999,"15min":5.6559999999999997},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.534,"5min":1.5649999999999999,"15min":1.2869999999999999},"min":{"1min":1.3380000000000001,"5min":1.0089999999999999,"15min":0.48099999999999998},"max":{"1min":1.8420000000000001,"5min":2.0539999999999998,"15min":6.5650000000000004},"last":1.3979999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:55:09 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.599,"15min":1.476},"min":{"1min":1.2889999999999999,"5min":1.1160000000000001,"15min":0.53300000000000003},"max":{"1min":1.9950000000000001,"5min":2.0680000000000001,"15min":9.7409999999999997},"last":1.464},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.6020000000000001,"15min":1.452},"min":{"1min":1.3340000000000001,"5min":1.0980000000000001,"15min":0.437},"max":{"1min":1.8859999999999999,"5min":2.0979999999999999,"15min":5.5670000000000002},"last":1.4730000000000001}]}]},{"osd":6,"up_from":48,"seq":206158430947,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895084,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":894400,"kb_avail":92850068,"statfs":{"total":95995035648,"available":95078469632,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":915865600},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.03,"15min":0.996},"min":{"1min":0.78300000000000003,"5min":0.70499999999999996,"15min":0.68899999999999995},"max":{"1min":1.2310000000000001,"5min":1.538,"15min":1.538},"last":1.083},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0529999999999999,"15min":1.0489999999999999},"min":{"1min":0.70099999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.3520000000000001,"5min":1.512,"15min":1.639},"last":1.2689999999999999}]},{"osd":1,"last update":"Mon Feb 12 16:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":1.7509999999999999,"15min":1.3100000000000001},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74099999999999999,"5min":118.962,"15min":118.962},"last":1.6060000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.8360000000000001,"15min":1.395},"min":{"1min":0.28799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.1379999999999999,"5min":122.48,"15min":122.48},"last":1.0760000000000001}]},{"osd":2,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.0169999999999999,"15min":0.89200000000000002},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.308},"max":{"1min":1.349,"5min":1.6160000000000001,"15min":6.3380000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.044,"5min":1.095,"15min":0.93799999999999994},"min":{"1min":0.82099999999999995,"5min":0.68500000000000005,"15min":0.33700000000000002},"max":{"1min":1.264,"5min":1.597,"15min":4.6900000000000004},"last":1.242}]},{"osd":3,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.141,"15min":1.448},"min":{"1min":0.90100000000000002,"5min":0.79300000000000004,"15min":0.316},"max":{"1min":1.4710000000000001,"5min":1.494,"15min":124.44499999999999},"last":1.431},{"interface":"front","average":{"1min":1.101,"5min":1.149,"15min":1.4590000000000001},"min":{"1min":0.85899999999999999,"5min":0.78900000000000003,"15min":0.374},"max":{"1min":1.3120000000000001,"5min":1.722,"15min":124.407},"last":1.3959999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.115,"15min":0.93200000000000005},"min":{"1min":0.71699999999999997,"5min":0.56499999999999995,"15min":0.29699999999999999},"max":{"1min":1.476,"5min":1.6200000000000001,"15min":3.5},"last":1.028},{"interface":"front","average":{"1min":1.171,"5min":1.1579999999999999,"15min":0.997},"min":{"1min":0.73499999999999999,"5min":0.69099999999999995,"15min":0.29899999999999999},"max":{"1min":1.591,"5min":1.841,"15min":7.2149999999999999},"last":0.97799999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1990000000000001,"15min":2.0579999999999998},"min":{"1min":0.85699999999999998,"5min":0.60499999999999998,"15min":0.42499999999999999},"max":{"1min":1.546,"5min":1.885,"15min":281.48500000000001},"last":1.47},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2549999999999999,"15min":2.1230000000000002},"min":{"1min":0.77600000000000002,"5min":0.76800000000000002,"15min":0.34499999999999997},"max":{"1min":1.4330000000000001,"5min":1.9950000000000001,"15min":282.96199999999999},"last":1.5009999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.2849999999999999,"15min":1.048},"min":{"1min":0.81399999999999995,"5min":0.72399999999999998,"15min":0.34300000000000003},"max":{"1min":1.516,"5min":1.8720000000000001,"15min":6.8869999999999996},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.345,"15min":1.1439999999999999},"min":{"1min":1.014,"5min":0.92500000000000004,"15min":0.40000000000000002},"max":{"1min":1.6459999999999999,"5min":1.9319999999999999,"15min":4.0510000000000002},"last":1.538}]},{"osd":8,"last update":"Mon Feb 12 16:55:12 2024","interfaces":[{"interface":"back","average":{"1min":1.411,"5min":1.5309999999999999,"15min":1.266},"min":{"1min":1.121,"5min":0.88800000000000001,"15min":0.40899999999999997},"max":{"1min":1.7549999999999999,"5min":2.0870000000000002,"15min":5.5279999999999996},"last":1.9450000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.556,"15min":1.296},"min":{"1min":1.1339999999999999,"5min":1.127,"15min":0.40500000000000003},"max":{"1min":1.855,"5min":2.395,"15min":5.5700000000000003},"last":1.875}]},{"osd":9,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.6319999999999999,"15min":1.339},"min":{"1min":1.167,"5min":1.135,"15min":0.46500000000000002},"max":{"1min":2.0030000000000001,"5min":2.468,"15min":3.383},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.716,"15min":1.395},"min":{"1min":1.3160000000000001,"5min":1.2629999999999999,"15min":0.41299999999999998},"max":{"1min":1.9890000000000001,"5min":2.407,"15min":3.4119999999999999},"last":2.0659999999999998}]},{"osd":10,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.6080000000000001,"15min":1.4219999999999999},"min":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":0.46700000000000003},"max":{"1min":1.827,"5min":2.2040000000000002,"15min":4.516},"last":1.8360000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.7230000000000001,"15min":1.5069999999999999},"min":{"1min":1.3720000000000001,"5min":1.3220000000000001,"15min":0.54500000000000004},"max":{"1min":2.2080000000000002,"5min":2.3330000000000002,"15min":4.5469999999999997},"last":1.986}]},{"osd":11,"last update":"Mon Feb 12 16:55:34 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.5549999999999999,"15min":1.3580000000000001},"min":{"1min":1.256,"5min":1.1399999999999999,"15min":0.51300000000000001},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":5.3799999999999999},"last":1.8999999999999999},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.6639999999999999,"15min":1.4390000000000001},"min":{"1min":1.3740000000000001,"5min":1.1970000000000001,"15min":0.45600000000000002},"max":{"1min":2.1890000000000001,"5min":2.2320000000000002,"15min":5.6130000000000004},"last":1.712}]}]},{"osd":1,"up_from":13,"seq":55834575624,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1191600,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":1190784,"kb_avail":92553552,"statfs":{"total":95995035648,"available":94774837248,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1219362816},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.89900000000000002,"15min":0.88300000000000001},"min":{"1min":0.69299999999999995,"5min":0.48699999999999999,"15min":0.23899999999999999},"max":{"1min":1.0940000000000001,"5min":2.3380000000000001,"15min":7.532},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89800000000000002,"15min":0.82099999999999995},"min":{"1min":0.70099999999999996,"5min":0.42799999999999999,"15min":0.254},"max":{"1min":1.4590000000000001,"5min":1.976,"15min":7.3540000000000001},"last":0.92300000000000004}]},{"osd":2,"last update":"Mon Feb 12 16:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.995,"15min":0.93899999999999995},"min":{"1min":0.78000000000000003,"5min":0.65800000000000003,"15min":0.29399999999999998},"max":{"1min":1.367,"5min":1.367,"15min":7.54},"last":1.048},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.94699999999999995,"15min":0.91000000000000003},"min":{"1min":0.67200000000000004,"5min":0.62,"15min":0.25900000000000001},"max":{"1min":1.2869999999999999,"5min":1.6519999999999999,"15min":7.9539999999999997},"last":0.97299999999999998}]},{"osd":3,"last update":"Mon Feb 12 16:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":0.98899999999999999,"15min":1.1679999999999999},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":4.1820000000000004,"5min":8.9920000000000009,"15min":26.664000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.466,"5min":1.113,"15min":1.389},"min":{"1min":0.50700000000000001,"5min":0.34599999999999997,"15min":0.26200000000000001},"max":{"1min":5.2080000000000002,"5min":8.9610000000000003,"15min":27.148},"last":1.133}]},{"osd":4,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.1839999999999999,"15min":1.069},"min":{"1min":0.91800000000000004,"5min":0.61899999999999999,"15min":0.34699999999999998},"max":{"1min":1.5469999999999999,"5min":1.673,"15min":7.6029999999999998},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.226,"15min":1.125},"min":{"1min":0.79500000000000004,"5min":0.57699999999999996,"15min":0.36699999999999999},"max":{"1min":1.7909999999999999,"5min":1.839,"15min":7.7149999999999999},"last":1.1379999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1439999999999999,"15min":1.097},"min":{"1min":0.82499999999999996,"5min":0.81299999999999994,"15min":0.32100000000000001},"max":{"1min":1.508,"5min":1.718,"15min":8.0190000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2130000000000001,"15min":1.177},"min":{"1min":0.97399999999999998,"5min":0.78500000000000003,"15min":0.34799999999999998},"max":{"1min":1.587,"5min":2.1800000000000002,"15min":8.7050000000000001},"last":1.113}]},{"osd":6,"last update":"Mon Feb 12 16:55:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.3089999999999999,"15min":1.2030000000000001},"min":{"1min":1.038,"5min":0.75800000000000001,"15min":0.39000000000000001},"max":{"1min":1.8260000000000001,"5min":2.2210000000000001,"15min":7.9450000000000003},"last":1.242},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.3899999999999999,"15min":1.2809999999999999},"min":{"1min":1.071,"5min":0.80900000000000005,"15min":0.40999999999999998},"max":{"1min":1.857,"5min":2.0510000000000002,"15min":8.7669999999999995},"last":1.3049999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.292,"15min":1.1919999999999999},"min":{"1min":0.874,"5min":0.751,"15min":0.44600000000000001},"max":{"1min":1.615,"5min":1.8779999999999999,"15min":2.75},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4059999999999999,"15min":1.167},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.38800000000000001},"max":{"1min":1.591,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":1.27}]},{"osd":8,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.6279999999999999,"15min":1.4590000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.48499999999999999},"max":{"1min":1.944,"5min":2.5960000000000001,"15min":8.8100000000000005},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.522,"5min":1.536,"15min":1.496},"min":{"1min":1.2250000000000001,"5min":0.84299999999999997,"15min":0.54300000000000004},"max":{"1min":1.792,"5min":2.2389999999999999,"15min":8.9269999999999996},"last":1.222}]},{"osd":9,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.544,"15min":1.3320000000000001},"min":{"1min":1.2390000000000001,"5min":1,"15min":0.40200000000000002},"max":{"1min":1.7789999999999999,"5min":2.4289999999999998,"15min":8.1539999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.641,"5min":1.6539999999999999,"15min":1.425},"min":{"1min":1.3959999999999999,"5min":1.1679999999999999,"15min":0.48099999999999998},"max":{"1min":2.0870000000000002,"5min":2.3639999999999999,"15min":8.2409999999999997},"last":1.4239999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.6379999999999999,"15min":1.474},"min":{"1min":1.278,"5min":0.80200000000000005,"15min":0.44900000000000001},"max":{"1min":1.952,"5min":2.3620000000000001,"15min":8.1140000000000008},"last":1.524},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.6799999999999999,"15min":1.5549999999999999},"min":{"1min":1.2410000000000001,"5min":1.1519999999999999,"15min":0.51200000000000001},"max":{"1min":2.048,"5min":2.7149999999999999,"15min":8.8469999999999995},"last":1.387}]},{"osd":11,"last update":"Mon Feb 12 16:55:17 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.639,"15min":1.6659999999999999},"min":{"1min":1.099,"5min":1.099,"15min":0.53800000000000003},"max":{"1min":2.1400000000000001,"5min":2.823,"15min":14.862},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.623,"15min":1.5049999999999999},"min":{"1min":0.89800000000000002,"5min":0.87,"15min":0.51800000000000002},"max":{"1min":2.0659999999999998,"5min":2.2930000000000001,"15min":15.76},"last":1.605}]}]},{"osd":0,"up_from":8,"seq":34359739158,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887596,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":886912,"kb_avail":92857556,"statfs":{"total":95995035648,"available":95086137344,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":908197888},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.84199999999999997,"15min":1.254},"min":{"1min":0.53600000000000003,"5min":0.33200000000000002,"15min":0.24099999999999999},"max":{"1min":1.3240000000000001,"5min":1.4079999999999999,"15min":118.657},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.004,"15min":1.4159999999999999},"min":{"1min":0.52500000000000002,"5min":0.51100000000000001,"15min":0.29199999999999998},"max":{"1min":1.405,"5min":1.9930000000000001,"15min":118.58},"last":0.872}]},{"osd":2,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.91200000000000003,"15min":0.81399999999999995},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.27500000000000002},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":4.0999999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.97599999999999998,"15min":0.877},"min":{"1min":0.621,"5min":0.41799999999999998,"15min":0.28599999999999998},"max":{"1min":1.405,"5min":1.909,"15min":4.0469999999999997},"last":1.147}]},{"osd":3,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0509999999999999,"15min":1.679},"min":{"1min":0.72499999999999998,"5min":0.47599999999999998,"15min":0.32900000000000001},"max":{"1min":1.722,"5min":1.722,"15min":193.114},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.98599999999999999,"15min":1.625},"min":{"1min":0.57299999999999995,"5min":0.39000000000000001,"15min":0.26600000000000001},"max":{"1min":1.234,"5min":1.7729999999999999,"15min":193.22},"last":0.94399999999999995}]},{"osd":4,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.232,"15min":1.002},"min":{"1min":0.78800000000000003,"5min":0.58299999999999996,"15min":0.35699999999999998},"max":{"1min":1.847,"5min":1.847,"15min":2.21},"last":1.411},{"interface":"front","average":{"1min":1.173,"5min":1.196,"15min":1.0249999999999999},"min":{"1min":0.70499999999999996,"5min":0.65900000000000003,"15min":0.35199999999999998},"max":{"1min":1.754,"5min":1.754,"15min":3.9580000000000002},"last":0.98199999999999998}]},{"osd":5,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.248,"15min":1.0840000000000001},"min":{"1min":0.67700000000000005,"5min":0.50700000000000001,"15min":0.33700000000000002},"max":{"1min":1.7909999999999999,"5min":1.8680000000000001,"15min":8.532},"last":1.127},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.371,"15min":1.2070000000000001},"min":{"1min":0.73499999999999999,"5min":0.60399999999999998,"15min":0.41299999999999998},"max":{"1min":1.927,"5min":2.0099999999999998,"15min":8.6519999999999992},"last":1.2050000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:55:52 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3080000000000001,"15min":1.1100000000000001},"min":{"1min":0.81200000000000006,"5min":0.46800000000000003,"15min":0.38300000000000001},"max":{"1min":1.885,"5min":1.964,"15min":6.6950000000000003},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.321,"15min":1.117},"min":{"1min":0.79700000000000004,"5min":0.63700000000000001,"15min":0.38300000000000001},"max":{"1min":1.901,"5min":2.0950000000000002,"15min":2.1779999999999999},"last":1.228}]},{"osd":7,"last update":"Mon Feb 12 16:39:13 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.24,"15min":2.706},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.374},"max":{"1min":1.835,"5min":6.1130000000000004,"15min":191.47200000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4019999999999999,"15min":2.823},"min":{"1min":0.59199999999999997,"5min":0.49199999999999999,"15min":0.432},"max":{"1min":1.897,"5min":6.1349999999999998,"15min":191.50299999999999},"last":1.081}]},{"osd":8,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.552,"15min":1.28},"min":{"1min":0.88500000000000001,"5min":0.55300000000000005,"15min":0.442},"max":{"1min":2.0369999999999999,"5min":2.2109999999999999,"15min":6.7400000000000002},"last":1.7290000000000001},{"interface":"front","average":{"1min":1.649,"5min":1.6339999999999999,"15min":1.3799999999999999},"min":{"1min":0.91300000000000003,"5min":0.72699999999999998,"15min":0.51600000000000001},"max":{"1min":2.2389999999999999,"5min":2.3420000000000001,"15min":6.8570000000000002},"last":1.6279999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:55:52 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.49,"15min":1.256},"min":{"1min":0.80200000000000005,"5min":0.79200000000000004,"15min":0.42299999999999999},"max":{"1min":2.4969999999999999,"5min":2.4969999999999999,"15min":4.1130000000000004},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.51,"15min":1.28},"min":{"1min":0.84199999999999997,"5min":0.67600000000000005,"15min":0.47499999999999998},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":6.7770000000000001},"last":1.6899999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.5429999999999999,"15min":1.3340000000000001},"min":{"1min":0.91100000000000003,"5min":0.71599999999999997,"15min":0.46100000000000002},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":6.0880000000000001},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6479999999999999,"15min":1.415},"min":{"1min":1.101,"5min":0.86899999999999999,"15min":0.52400000000000002},"max":{"1min":2.472,"5min":2.472,"15min":6.1319999999999997},"last":1.645}]},{"osd":11,"last update":"Mon Feb 12 16:55:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4990000000000001,"15min":1.361},"min":{"1min":0.97299999999999998,"5min":0.75900000000000001,"15min":0.47599999999999998},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":5.8079999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5009999999999999,"15min":1.3500000000000001},"min":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.50600000000000001},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":5.7140000000000004},"last":1.4470000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313986,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594608,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":593792,"kb_avail":93150544,"statfs":{"total":95995035648,"available":95386157056,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":608043008},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.88900000000000001,"15min":0.76200000000000001},"min":{"1min":0.48599999999999999,"5min":0.43099999999999999,"15min":0.223},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":3.996},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0209999999999999,"15min":0.84299999999999997},"min":{"1min":0.627,"5min":0.53700000000000003,"15min":0.23400000000000001},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":4.9690000000000003},"last":1.1830000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.93700000000000006,"15min":1.212},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.27200000000000002},"max":{"1min":1.7410000000000001,"5min":2.1899999999999999,"15min":43.488999999999997},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.016,"5min":0.99199999999999999,"15min":1.2649999999999999},"min":{"1min":0.495,"5min":0.38700000000000001,"15min":0.22800000000000001},"max":{"1min":1.329,"5min":1.55,"15min":47.810000000000002},"last":0.88400000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:04 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.036,"15min":2.133},"min":{"1min":0.73899999999999999,"5min":0.439,"15min":0.28499999999999998},"max":{"1min":1.4179999999999999,"5min":2.0859999999999999,"15min":193.74799999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.254,"15min":2.2909999999999999},"min":{"1min":0.92500000000000004,"5min":0.55600000000000005,"15min":0.38100000000000001},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":193.80000000000001},"last":1.3100000000000001}]},{"osd":4,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2549999999999999,"15min":1.018},"min":{"1min":0.751,"5min":0.70099999999999996,"15min":0.309},"max":{"1min":1.7929999999999999,"5min":2.5470000000000002,"15min":3.0470000000000002},"last":1.137},{"interface":"front","average":{"1min":1.405,"5min":1.3180000000000001,"15min":1.0800000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.32400000000000001},"max":{"1min":2.258,"5min":2.2789999999999999,"15min":6.9409999999999998},"last":1.4359999999999999}]},{"osd":5,"last update":"Mon Feb 12 16:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2070000000000001,"15min":1.04},"min":{"1min":0.83299999999999996,"5min":0.65500000000000003,"15min":0.31900000000000001},"max":{"1min":1.593,"5min":1.9350000000000001,"15min":5.1319999999999997},"last":1.155},{"interface":"front","average":{"1min":1.405,"5min":1.365,"15min":1.115},"min":{"1min":0.68600000000000005,"5min":0.60999999999999999,"15min":0.30599999999999999},"max":{"1min":1.984,"5min":2.7290000000000001,"15min":4.2089999999999996},"last":1.052}]},{"osd":6,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.425,"15min":1.165},"min":{"1min":0.88700000000000001,"5min":0.75900000000000001,"15min":0.39800000000000002},"max":{"1min":2.3650000000000002,"5min":2.5230000000000001,"15min":7.0049999999999999},"last":1.536},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3260000000000001,"15min":1.131},"min":{"1min":0.79600000000000004,"5min":0.67000000000000004,"15min":0.36099999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":4.0940000000000003},"last":1.248}]},{"osd":7,"last update":"Mon Feb 12 16:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.321,"15min":1.3520000000000001},"min":{"1min":0.58399999999999996,"5min":0.34000000000000002,"15min":0.314},"max":{"1min":1.9019999999999999,"5min":6.2560000000000002,"15min":11.044},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2529999999999999,"15min":1.248},"min":{"1min":0.44600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.4219999999999997,"5min":8.8620000000000001,"15min":8.8620000000000001},"last":1.6499999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:55:04 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.6080000000000001,"15min":1.494},"min":{"1min":1.2410000000000001,"5min":0.73199999999999998,"15min":0.36899999999999999},"max":{"1min":2.069,"5min":2.3650000000000002,"15min":59.164999999999999},"last":1.643},{"interface":"front","average":{"1min":1.569,"5min":1.546,"15min":1.4550000000000001},"min":{"1min":1.0860000000000001,"5min":0.90300000000000002,"15min":0.44500000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":58.993000000000002},"last":1.335}]},{"osd":9,"last update":"Mon Feb 12 16:55:40 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.556,"15min":1.272},"min":{"1min":0.96599999999999997,"5min":0.79700000000000004,"15min":0.435},"max":{"1min":2.1749999999999998,"5min":2.8490000000000002,"15min":5.2210000000000001},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.6319999999999999,"15min":1.367},"min":{"1min":0.95699999999999996,"5min":0.83299999999999996,"15min":0.41299999999999998},"max":{"1min":2.4969999999999999,"5min":2.9700000000000002,"15min":5.2549999999999999},"last":1.5660000000000001}]},{"osd":10,"last update":"Mon Feb 12 16:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.6419999999999999,"15min":2.798},"min":{"1min":1.095,"5min":0.879,"15min":0.45600000000000002},"max":{"1min":2.702,"5min":2.702,"15min":407.577},"last":1.8480000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.698,"15min":2.778},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.47799999999999998},"max":{"1min":2.6360000000000001,"5min":2.6360000000000001,"15min":407.45100000000002},"last":1.7629999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.5840000000000001,"15min":1.5029999999999999},"min":{"1min":0.875,"5min":0.875,"15min":0.47099999999999997},"max":{"1min":2.2890000000000001,"5min":2.5,"15min":31.835999999999999},"last":1.407},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.645,"15min":1.548},"min":{"1min":1.006,"5min":0.93999999999999995,"15min":0.53500000000000003},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":31.978999999999999},"last":1.5900000000000001}]}]},{"osd":3,"up_from":522,"seq":2241972928910,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930668,"kb_used_data":676,"kb_used_omap":91,"kb_used_meta":929892,"kb_avail":92814484,"statfs":{"total":95995035648,"available":95042031616,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94192,"internal_metadata":952209424},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.86099999999999999,"15min":0.872},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0289999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.89600000000000002,"15min":0.90600000000000003},"min":{"1min":0.58199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Feb 12 16:50:42 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.89600000000000002,"15min":1.909},"min":{"1min":0.63500000000000001,"5min":0.26900000000000002,"15min":0.26200000000000001},"max":{"1min":1.359,"5min":3.819,"15min":245.10900000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.056,"5min":1.1639999999999999,"15min":2.089},"min":{"1min":0.79300000000000004,"5min":0.313,"15min":0.313},"max":{"1min":1.6859999999999999,"5min":5.0279999999999996,"15min":245.21199999999999},"last":1.1599999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.88100000000000001,"15min":1.1899999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.30399999999999999},"max":{"1min":1.179,"5min":1.911,"15min":81.769999999999996},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.079,"5min":1.109,"15min":1.3540000000000001},"min":{"1min":0.63500000000000001,"5min":0.61799999999999999,"15min":0.32700000000000001},"max":{"1min":1.5229999999999999,"5min":2.262,"15min":81.876999999999995},"last":1.1659999999999999}]},{"osd":4,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.091,"15min":1.0089999999999999},"min":{"1min":0.67300000000000004,"5min":0.60899999999999999,"15min":0.32000000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":4.806},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.2869999999999999,"15min":1.1399999999999999},"min":{"1min":0.91300000000000003,"5min":0.74099999999999999,"15min":0.318},"max":{"1min":1.6879999999999999,"5min":1.7749999999999999,"15min":8.0069999999999997},"last":1.264}]},{"osd":5,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0980000000000001,"15min":1.0489999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.318},"max":{"1min":1.4390000000000001,"5min":1.623,"15min":10.058},"last":1.139},{"interface":"front","average":{"1min":1.163,"5min":1.2330000000000001,"15min":1.2250000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.35899999999999999},"max":{"1min":1.927,"5min":1.9359999999999999,"15min":8.032},"last":1.3520000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2150000000000001,"15min":1.1459999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.39800000000000002},"max":{"1min":1.5940000000000001,"5min":1.7509999999999999,"15min":9.7710000000000008},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.351,"15min":1.256},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.44500000000000001},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":4.9790000000000001},"last":1.4350000000000001}]},{"osd":7,"last update":"Mon Feb 12 16:55:24 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.2370000000000001,"15min":1.155},"min":{"1min":0.93200000000000005,"5min":0.70499999999999996,"15min":0.42399999999999999},"max":{"1min":1.5620000000000001,"5min":1.944,"15min":3.1150000000000002},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.3300000000000001,"15min":1.169},"min":{"1min":1.1719999999999999,"5min":0.80700000000000005,"15min":0.54200000000000004},"max":{"1min":1.6419999999999999,"5min":1.988,"15min":3},"last":1.3129999999999999}]},{"osd":8,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.53,"15min":1.29},"min":{"1min":1.099,"5min":0.76000000000000001,"15min":0.42699999999999999},"max":{"1min":1.9219999999999999,"5min":2.1669999999999998,"15min":8.2710000000000008},"last":1.732},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.593,"15min":1.4370000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.47899999999999998},"max":{"1min":1.863,"5min":2.258,"15min":5.0279999999999996},"last":1.542}]},{"osd":9,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.5029999999999999,"15min":1.302},"min":{"1min":0.95199999999999996,"5min":0.80800000000000005,"15min":0.42599999999999999},"max":{"1min":1.881,"5min":2.306,"15min":8.1159999999999997},"last":1.627},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5780000000000001,"15min":1.4350000000000001},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.502},"max":{"1min":2.0230000000000001,"5min":2.1619999999999999,"15min":4.9080000000000004},"last":1.6619999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.5880000000000001,"15min":4.069},"min":{"1min":1.01,"5min":1.01,"15min":0.52100000000000002},"max":{"1min":1.9870000000000001,"5min":2.1389999999999998,"15min":738.99900000000002},"last":1.389},{"interface":"front","average":{"1min":1.611,"5min":1.663,"15min":4.1390000000000002},"min":{"1min":1.022,"5min":1.022,"15min":0.51000000000000001},"max":{"1min":1.9910000000000001,"5min":2.2250000000000001,"15min":746.45299999999997},"last":1.5700000000000001}]},{"osd":11,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.5600000000000001,"15min":1.575},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.55000000000000004},"max":{"1min":2.004,"5min":2.1429999999999998,"15min":21.888000000000002},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.5589999999999999,"15min":1.5269999999999999},"min":{"1min":1.113,"5min":0.83899999999999997,"15min":0.51400000000000001},"max":{"1min":1.958,"5min":2.2010000000000001,"15min":22.079999999999998},"last":1.8120000000000001}]}]},{"osd":4,"up_from":32,"seq":137438954226,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890864,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":890048,"kb_avail":92854288,"statfs":{"total":95995035648,"available":95082790912,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":911409152},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.92200000000000004,"15min":0.74199999999999999},"min":{"1min":0.70099999999999996,"5min":0.65500000000000003,"15min":0.26100000000000001},"max":{"1min":1.1359999999999999,"5min":1.383,"15min":3.218},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0489999999999999,"15min":0.82499999999999996},"min":{"1min":0.72899999999999998,"5min":0.68799999999999994,"15min":0.31},"max":{"1min":1.454,"5min":1.835,"15min":3.3410000000000002},"last":1.091}]},{"osd":1,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.98699999999999999,"15min":0.90600000000000003},"min":{"1min":0.69099999999999995,"5min":0.62,"15min":0.27600000000000002},"max":{"1min":1.147,"5min":2.0449999999999999,"15min":7.7069999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.069,"15min":0.94999999999999996},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.32300000000000001},"max":{"1min":1.379,"5min":2.2869999999999999,"15min":5.2039999999999997},"last":0.98299999999999998}]},{"osd":2,"last update":"Mon Feb 12 16:44:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":3.0840000000000001,"15min":1.8340000000000001},"min":{"1min":0.377,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3100000000000001,"5min":212.685,"15min":212.685},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":2.9910000000000001,"15min":1.7490000000000001},"min":{"1min":0.33700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.3540000000000001,"5min":210.95699999999999,"15min":210.95699999999999},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0980000000000001,"15min":1.0669999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.33600000000000002},"max":{"1min":1.3360000000000001,"5min":1.992,"15min":17.876999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.115,"15min":1.1180000000000001},"min":{"1min":0.87,"5min":0.71999999999999997,"15min":0.312},"max":{"1min":1.5009999999999999,"5min":1.877,"15min":17.623999999999999},"last":1.117}]},{"osd":5,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.99399999999999999,"15min":0.86899999999999999},"min":{"1min":0.64300000000000002,"5min":0.54400000000000004,"15min":0.28699999999999998},"max":{"1min":1.3839999999999999,"5min":2.0910000000000002,"15min":4.7370000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3149999999999999,"15min":1.1000000000000001},"min":{"1min":1.1879999999999999,"5min":0.66400000000000003,"15min":0.35699999999999998},"max":{"1min":1.4810000000000001,"5min":2.2879999999999998,"15min":4.8150000000000004},"last":1.202}]},{"osd":6,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.98199999999999998,"15min":0.88},"min":{"1min":0.80200000000000005,"5min":0.68799999999999994,"15min":0.29899999999999999},"max":{"1min":1.3169999999999999,"5min":1.3300000000000001,"15min":3.4830000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2889999999999999,"15min":1.093},"min":{"1min":0.92000000000000004,"5min":0.82499999999999996,"15min":0.36499999999999999},"max":{"1min":1.4790000000000001,"5min":2.3809999999999998,"15min":3.508},"last":1.2509999999999999}]},{"osd":7,"last update":"Mon Feb 12 16:55:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.052,"15min":1.018},"min":{"1min":0.73999999999999999,"5min":0.58999999999999997,"15min":0.33600000000000002},"max":{"1min":1.298,"5min":1.5049999999999999,"15min":3.569},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2110000000000001,"15min":1.02},"min":{"1min":0.85799999999999998,"5min":0.83499999999999996,"15min":0.376},"max":{"1min":1.4179999999999999,"5min":1.9359999999999999,"15min":4.0709999999999997},"last":1.603}]},{"osd":8,"last update":"Mon Feb 12 16:55:13 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4359999999999999,"15min":1.2170000000000001},"min":{"1min":1.038,"5min":0.93500000000000005,"15min":0.44400000000000001},"max":{"1min":1.7649999999999999,"5min":2.915,"15min":3.8279999999999998},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4770000000000001,"15min":1.1890000000000001},"min":{"1min":1.0760000000000001,"5min":1.0169999999999999,"15min":0.38700000000000001},"max":{"1min":1.7749999999999999,"5min":2.718,"15min":4.6470000000000002},"last":1.5169999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.5509999999999999,"15min":1.2869999999999999},"min":{"1min":1.1599999999999999,"5min":0.82499999999999996,"15min":0.46899999999999997},"max":{"1min":1.712,"5min":2.7690000000000001,"15min":3.319},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.6559999999999999,"15min":1.3819999999999999},"min":{"1min":1.27,"5min":1.111,"15min":0.56699999999999995},"max":{"1min":1.948,"5min":2.8199999999999998,"15min":3.3519999999999999},"last":1.6659999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.577,"15min":2.1259999999999999},"min":{"1min":1.212,"5min":1.0920000000000001,"15min":0.41699999999999998},"max":{"1min":1.8420000000000001,"5min":2.7919999999999998,"15min":241.36600000000001},"last":1.468},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.6299999999999999,"15min":2.109},"min":{"1min":1.3740000000000001,"5min":1.204,"15min":0.45500000000000002},"max":{"1min":1.77,"5min":2.5390000000000001,"15min":241.27799999999999},"last":1.571}]},{"osd":11,"last update":"Mon Feb 12 16:55:01 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.657,"15min":1.3300000000000001},"min":{"1min":1.3660000000000001,"5min":1.242,"15min":0.496},"max":{"1min":1.863,"5min":2.7090000000000001,"15min":3.4710000000000001},"last":1.752},{"interface":"front","average":{"1min":1.573,"5min":1.6399999999999999,"15min":1.409},"min":{"1min":1.3220000000000001,"5min":1.2470000000000001,"15min":0.55200000000000005},"max":{"1min":1.905,"5min":2.6659999999999999,"15min":3.4580000000000002},"last":1.6970000000000001}]}]},{"osd":5,"up_from":39,"seq":167503725290,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893872,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":893056,"kb_avail":92851280,"statfs":{"total":95995035648,"available":95079710720,"internally_reserved":0,"allocated":692224,"data_stored":316964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":914489344},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 12 16:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.74199999999999999,"15min":1.2609999999999999},"min":{"1min":0.28899999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2649999999999999,"5min":3.7610000000000001,"15min":38.366},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.94799999999999995,"15min":1.385},"min":{"1min":0.317,"5min":0.28899999999999998,"15min":0.255},"max":{"1min":4.915,"5min":5.1459999999999999,"15min":37.310000000000002},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.129,"15min":2.7229999999999999},"min":{"1min":0.80200000000000005,"5min":0.61799999999999999,"15min":0.23499999999999999},"max":{"1min":1.3440000000000001,"5min":1.518,"15min":387.76299999999998},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.25,"15min":2.8660000000000001},"min":{"1min":1.014,"5min":0.82899999999999996,"15min":0.34100000000000003},"max":{"1min":1.4379999999999999,"5min":1.615,"15min":387.88099999999997},"last":1.0529999999999999}]},{"osd":2,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.0149999999999999,"15min":1.0189999999999999},"min":{"1min":0.69099999999999995,"5min":0.68000000000000005,"15min":0.33600000000000002},"max":{"1min":1.2609999999999999,"5min":1.3979999999999999,"15min":5.1109999999999998},"last":1.157},{"interface":"front","average":{"1min":0.997,"5min":0.97799999999999998,"15min":0.91800000000000004},"min":{"1min":0.71199999999999997,"5min":0.66600000000000004,"15min":0.26900000000000002},"max":{"1min":1.379,"5min":1.5149999999999999,"15min":7.3470000000000004},"last":0.79400000000000004}]},{"osd":3,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.006,"15min":1.0640000000000001},"min":{"1min":0.72099999999999997,"5min":0.66400000000000003,"15min":0.28299999999999997},"max":{"1min":1.4139999999999999,"5min":1.4990000000000001,"15min":5.8879999999999999},"last":0.997},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.0229999999999999,"15min":0.97299999999999998},"min":{"1min":0.71399999999999997,"5min":0.66400000000000003,"15min":0.315},"max":{"1min":1.151,"5min":1.3859999999999999,"15min":4.7839999999999998},"last":0.91700000000000004}]},{"osd":4,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.03,"15min":0.97799999999999998},"min":{"1min":0.74399999999999999,"5min":0.44700000000000001,"15min":0.255},"max":{"1min":1.4450000000000001,"5min":1.766,"15min":5.7460000000000004},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.0269999999999999,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.60099999999999998,"15min":0.28199999999999997},"max":{"1min":1.2609999999999999,"5min":1.4950000000000001,"15min":5.069},"last":1.1240000000000001}]},{"osd":6,"last update":"Mon Feb 12 16:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1619999999999999,"15min":1.111},"min":{"1min":0.93300000000000005,"5min":0.79600000000000004,"15min":0.35099999999999998},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":5.1760000000000002},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.1240000000000001,"15min":1.042},"min":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.28699999999999998},"max":{"1min":1.4730000000000001,"5min":1.7210000000000001,"15min":7.3029999999999999},"last":0.97999999999999998}]},{"osd":7,"last update":"Mon Feb 12 16:55:55 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.2569999999999999,"15min":1.2749999999999999},"min":{"1min":1.0620000000000001,"5min":0.76200000000000001,"15min":0.59999999999999998},"max":{"1min":1.6160000000000001,"5min":1.619,"15min":4.8600000000000003},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2729999999999999,"15min":1.292},"min":{"1min":0.995,"5min":0.52100000000000002,"15min":0.42999999999999999},"max":{"1min":1.583,"5min":1.6419999999999999,"15min":4.8890000000000002},"last":1.0800000000000001}]},{"osd":8,"last update":"Mon Feb 12 16:55:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.458,"15min":1.3420000000000001},"min":{"1min":1.077,"5min":0.74099999999999999,"15min":0.46100000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":8.0579999999999998},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4159999999999999,"15min":1.2290000000000001},"min":{"1min":1.085,"5min":0.90500000000000003,"15min":0.38200000000000001},"max":{"1min":1.629,"5min":2.1269999999999998,"15min":4.9279999999999999},"last":1.1879999999999999}]},{"osd":9,"last update":"Mon Feb 12 16:55:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.5589999999999999,"15min":1.421},"min":{"1min":1.298,"5min":0.97299999999999998,"15min":0.45300000000000001},"max":{"1min":1.756,"5min":2.0600000000000001,"15min":9.0830000000000002},"last":1.992},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.5,"15min":1.3129999999999999},"min":{"1min":1.071,"5min":0.89500000000000002,"15min":0.41699999999999998},"max":{"1min":1.78,"5min":2.1859999999999999,"15min":6.5419999999999998},"last":1.2829999999999999}]},{"osd":10,"last update":"Mon Feb 12 16:55:29 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.5049999999999999,"15min":1.4199999999999999},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.43099999999999999},"max":{"1min":2.0550000000000002,"5min":2.0640000000000001,"15min":8.3879999999999999},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.613,"15min":1.5249999999999999},"min":{"1min":1.238,"5min":0.68899999999999995,"15min":0.43099999999999999},"max":{"1min":2.0640000000000001,"5min":2.093,"15min":8.1270000000000007},"last":1.7529999999999999}]},{"osd":11,"last update":"Mon Feb 12 16:55:42 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.5580000000000001,"15min":1.5089999999999999},"min":{"1min":1.298,"5min":1.0089999999999999,"15min":0.47199999999999998},"max":{"1min":1.9490000000000001,"5min":2.109,"15min":7.4210000000000003},"last":1.956},{"interface":"front","average":{"1min":1.706,"5min":1.6659999999999999,"15min":1.5900000000000001},"min":{"1min":1.3999999999999999,"5min":1,"15min":0.52300000000000002},"max":{"1min":2.1509999999999998,"5min":2.1920000000000002,"15min":8.1539999999999999},"last":1.867}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-12T16:56:05.974 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-12T16:56:05.974 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-12T16:56:05.983 INFO:tasks.cephadm:Teardown begin 2024-02-12T16:56:05.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:56:06.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:56:06.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:56:06.031 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:56:05 smithi169 bash[19604]: cluster 2024-02-12T16:56:04.526437+0000 mgr.y (mgr.14146) 2157 : cluster [DBG] pgmap v3168: 31 pgs: 31 active+clean; 0 B data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-02-12T16:56:06.133 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-12T16:56:06.133 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-12T16:56:06.139 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-12T16:56:06.146 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-12T16:56:06.180 INFO:tasks.cephadm:Stopping all daemons... 2024-02-12T16:56:06.180 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-12T16:56:06.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.a 2024-02-12T16:56:06.233 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:56:06 smithi121 systemd[1]: Stopping Ceph mon.a for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:06.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:56:06 smithi121 bash[301088]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-mon.a 2024-02-12T16:56:06.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:56:06 smithi121 bash[18489]: debug 2024-02-12T16:56:06.442+0000 7fbbaedeb700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T16:56:06.661 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 12 16:56:06 smithi121 bash[18489]: debug 2024-02-12T16:56:06.442+0000 7fbbaedeb700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-12T16:56:07.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.a.service' 2024-02-12T16:56:07.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:07.771 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-12T16:56:07.771 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-02-12T16:56:07.771 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.b 2024-02-12T16:56:08.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:56:07 smithi123 systemd[1]: Stopping Ceph mon.b for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:08.138 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:56:07 smithi123 bash[135716]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-mon.b 2024-02-12T16:56:08.432 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:56:08 smithi123 bash[19119]: debug 2024-02-12T16:56:08.170+0000 7f0ab2ab2700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T16:56:08.432 INFO:journalctl@ceph.mon.b.smithi123.stdout:Feb 12 16:56:08 smithi123 bash[19119]: debug 2024-02-12T16:56:08.170+0000 7f0ab2ab2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-12T16:56:09.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.b.service' 2024-02-12T16:56:09.180 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:09.180 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-02-12T16:56:09.180 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-12T16:56:09.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.c 2024-02-12T16:56:09.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:56:09 smithi169 systemd[1]: Stopping Ceph mon.c for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:09.487 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:56:09 smithi169 bash[135864]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-mon.c 2024-02-12T16:56:09.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:56:09 smithi169 bash[19604]: debug 2024-02-12T16:56:09.483+0000 7ff661816700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-12T16:56:09.797 INFO:journalctl@ceph.mon.c.smithi169.stdout:Feb 12 16:56:09 smithi169 bash[19604]: debug 2024-02-12T16:56:09.483+0000 7ff661816700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-12T16:56:10.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mon.c.service' 2024-02-12T16:56:10.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:10.611 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-12T16:56:10.611 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-12T16:56:10.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.y 2024-02-12T16:56:10.899 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 16:56:10 smithi121 systemd[1]: Stopping Ceph mgr.y for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:10.899 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 16:56:10 smithi121 bash[301262]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-mgr.y 2024-02-12T16:56:10.900 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 12 16:56:10 smithi121 bash[18805]: debug 2024-02-12T16:56:10.810+0000 7fb587c3f700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:56:11.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.y.service' 2024-02-12T16:56:11.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:11.764 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-12T16:56:11.764 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-12T16:56:11.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.x 2024-02-12T16:56:12.050 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 16:56:11 smithi123 systemd[1]: Stopping Ceph mgr.x for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:12.050 INFO:journalctl@ceph.mgr.x.smithi123.stdout:Feb 12 16:56:11 smithi123 bash[135862]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-mgr.x 2024-02-12T16:56:12.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@mgr.x.service' 2024-02-12T16:56:12.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:12.826 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-12T16:56:12.826 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-12T16:56:12.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.0 2024-02-12T16:56:13.161 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 12 16:56:12 smithi121 systemd[1]: Stopping Ceph osd.0 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:13.161 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 12 16:56:12 smithi121 bash[301578]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.0 2024-02-12T16:56:13.161 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 12 16:56:13 smithi121 bash[29235]: debug 2024-02-12T16:56:13.038+0000 7f35b09ad700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:56:13.161 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 12 16:56:13 smithi121 bash[29235]: debug 2024-02-12T16:56:13.038+0000 7f35b09ad700 -1 osd.0 1145 *** Got signal Terminated *** 2024-02-12T16:56:13.161 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 12 16:56:13 smithi121 bash[29235]: debug 2024-02-12T16:56:13.038+0000 7f35b09ad700 -1 osd.0 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:56:19.411 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 12 16:56:19 smithi121 bash[301578]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-0 2024-02-12T16:56:21.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.0.service' 2024-02-12T16:56:21.320 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:21.320 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-12T16:56:21.320 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-12T16:56:21.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.1 2024-02-12T16:56:21.572 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 12 16:56:21 smithi121 systemd[1]: Stopping Ceph osd.1 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:21.572 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 12 16:56:21 smithi121 bash[301842]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.1 2024-02-12T16:56:21.910 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 12 16:56:21 smithi121 bash[34171]: debug 2024-02-12T16:56:21.562+0000 7f5c3f913700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:56:21.910 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 12 16:56:21 smithi121 bash[34171]: debug 2024-02-12T16:56:21.562+0000 7f5c3f913700 -1 osd.1 1145 *** Got signal Terminated *** 2024-02-12T16:56:21.911 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 12 16:56:21 smithi121 bash[34171]: debug 2024-02-12T16:56:21.562+0000 7f5c3f913700 -1 osd.1 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:56:27.911 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 12 16:56:27 smithi121 bash[301842]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-1 2024-02-12T16:56:27.911 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 12 16:56:27 smithi121 bash[301931]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-1 2024-02-12T16:56:29.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.1.service' 2024-02-12T16:56:29.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:29.574 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-12T16:56:29.574 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-12T16:56:29.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.2 2024-02-12T16:56:29.811 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 12 16:56:29 smithi121 systemd[1]: Stopping Ceph osd.2 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:29.811 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 12 16:56:29 smithi121 bash[302120]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.2 2024-02-12T16:56:30.160 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 12 16:56:29 smithi121 bash[39225]: debug 2024-02-12T16:56:29.802+0000 7f586f57a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:56:30.161 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 12 16:56:29 smithi121 bash[39225]: debug 2024-02-12T16:56:29.802+0000 7f586f57a700 -1 osd.2 1145 *** Got signal Terminated *** 2024-02-12T16:56:30.161 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 12 16:56:29 smithi121 bash[39225]: debug 2024-02-12T16:56:29.802+0000 7f586f57a700 -1 osd.2 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:56:36.161 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 12 16:56:35 smithi121 bash[302120]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-2 2024-02-12T16:56:37.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.2.service' 2024-02-12T16:56:37.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:37.786 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-12T16:56:37.786 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-12T16:56:37.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.3 2024-02-12T16:56:38.047 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:37 smithi121 systemd[1]: Stopping Ceph osd.3 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:38.047 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:37 smithi121 bash[302386]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.3 2024-02-12T16:56:38.411 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:38 smithi121 bash[159491]: debug 2024-02-12T16:56:38.037+0000 7f6ad2371700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:56:38.411 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:38 smithi121 bash[159491]: debug 2024-02-12T16:56:38.037+0000 7f6ad2371700 -1 osd.3 1145 *** Got signal Terminated *** 2024-02-12T16:56:38.411 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:38 smithi121 bash[159491]: debug 2024-02-12T16:56:38.037+0000 7f6ad2371700 -1 osd.3 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:56:38.911 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:38 smithi121 bash[159491]: debug 2024-02-12T16:56:38.645+0000 7f6acf2ee700 -1 osd.3 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.420455+0000 front 2024-02-12T16:56:15.420411+0000 (oldest deadline 2024-02-12T16:56:38.320672+0000) 2024-02-12T16:56:39.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:38 smithi169 bash[23226]: debug 2024-02-12T16:56:38.946+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:40.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:39 smithi123 bash[32586]: debug 2024-02-12T16:56:39.857+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:40.160 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:39 smithi121 bash[159491]: debug 2024-02-12T16:56:39.673+0000 7f6acf2ee700 -1 osd.3 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.420455+0000 front 2024-02-12T16:56:15.420411+0000 (oldest deadline 2024-02-12T16:56:38.320672+0000) 2024-02-12T16:56:40.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:39 smithi169 bash[23226]: debug 2024-02-12T16:56:39.902+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:40.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:39 smithi169 bash[33498]: debug 2024-02-12T16:56:39.898+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:41.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:40 smithi123 bash[32586]: debug 2024-02-12T16:56:40.877+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:41.138 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:40 smithi123 bash[123421]: debug 2024-02-12T16:56:40.661+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:41.160 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:40 smithi121 bash[159491]: debug 2024-02-12T16:56:40.717+0000 7f6acf2ee700 -1 osd.3 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.420455+0000 front 2024-02-12T16:56:15.420411+0000 (oldest deadline 2024-02-12T16:56:38.320672+0000) 2024-02-12T16:56:41.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:40 smithi169 bash[23226]: debug 2024-02-12T16:56:40.898+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:41.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:40 smithi169 bash[33498]: debug 2024-02-12T16:56:40.926+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:42.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:41 smithi169 bash[23226]: debug 2024-02-12T16:56:41.870+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:42.048 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:41 smithi169 bash[33498]: debug 2024-02-12T16:56:41.930+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:42.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:41 smithi169 bash[28391]: debug 2024-02-12T16:56:41.742+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:42.138 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:41 smithi123 bash[123421]: debug 2024-02-12T16:56:41.689+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:42.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:41 smithi123 bash[32586]: debug 2024-02-12T16:56:41.865+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:42.160 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:41 smithi121 bash[159491]: debug 2024-02-12T16:56:41.725+0000 7f6acf2ee700 -1 osd.3 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.420455+0000 front 2024-02-12T16:56:15.420411+0000 (oldest deadline 2024-02-12T16:56:38.320672+0000) 2024-02-12T16:56:43.047 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:42 smithi169 bash[28391]: debug 2024-02-12T16:56:42.714+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:43.048 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:42 smithi169 bash[33498]: debug 2024-02-12T16:56:42.882+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:43.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:42 smithi169 bash[23226]: debug 2024-02-12T16:56:42.854+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:43.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:42 smithi169 bash[38680]: debug 2024-02-12T16:56:42.698+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:43.138 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:42 smithi123 bash[123421]: debug 2024-02-12T16:56:42.709+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:43.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:42 smithi123 bash[32586]: debug 2024-02-12T16:56:42.889+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:43.160 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:42 smithi121 bash[159491]: debug 2024-02-12T16:56:42.773+0000 7f6acf2ee700 -1 osd.3 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.420455+0000 front 2024-02-12T16:56:15.420411+0000 (oldest deadline 2024-02-12T16:56:38.320672+0000) 2024-02-12T16:56:43.720 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:43 smithi123 bash[22467]: debug 2024-02-12T16:56:43.437+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:44.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:43 smithi169 bash[38680]: debug 2024-02-12T16:56:43.718+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:44.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:43 smithi169 bash[23226]: debug 2024-02-12T16:56:43.814+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:44.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:43 smithi169 bash[28391]: debug 2024-02-12T16:56:43.714+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:44.048 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:43 smithi169 bash[33498]: debug 2024-02-12T16:56:43.926+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:44.138 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:43 smithi123 bash[123421]: debug 2024-02-12T16:56:43.713+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:44.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:43 smithi123 bash[32586]: debug 2024-02-12T16:56:43.849+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:44.411 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 12 16:56:43 smithi121 bash[302386]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-3 2024-02-12T16:56:44.699 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:44 smithi123 bash[22467]: debug 2024-02-12T16:56:44.393+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:45.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:44 smithi169 bash[23226]: debug 2024-02-12T16:56:44.862+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:45.048 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:44 smithi169 bash[33498]: debug 2024-02-12T16:56:44.970+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:45.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:44 smithi169 bash[28391]: debug 2024-02-12T16:56:44.718+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:45.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:44 smithi169 bash[38680]: debug 2024-02-12T16:56:44.770+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:45.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:44 smithi123 bash[32586]: debug 2024-02-12T16:56:44.869+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:45.138 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:44 smithi123 bash[123421]: debug 2024-02-12T16:56:44.693+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:45.638 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:45 smithi123 bash[22467]: debug 2024-02-12T16:56:45.369+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:45.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.3.service' 2024-02-12T16:56:45.938 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:45.938 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-12T16:56:45.938 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-12T16:56:45.938 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.4 2024-02-12T16:56:45.951 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:45 smithi123 bash[32586]: debug 2024-02-12T16:56:45.825+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:45.951 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:45 smithi123 bash[123421]: debug 2024-02-12T16:56:45.677+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:45.973 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:45 smithi169 bash[28391]: debug 2024-02-12T16:56:45.690+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:45.973 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:45 smithi169 bash[38680]: debug 2024-02-12T16:56:45.794+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:46.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:45 smithi169 bash[23226]: debug 2024-02-12T16:56:45.906+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:46.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:45 smithi169 bash[33498]: debug 2024-02-12T16:56:45.974+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:46.357 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:45 smithi123 systemd[1]: Stopping Ceph osd.4 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:46.357 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:45 smithi123 bash[135992]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.4 2024-02-12T16:56:46.357 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:46 smithi123 bash[22467]: debug 2024-02-12T16:56:46.157+0000 7f322870d700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:56:46.357 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:46 smithi123 bash[22467]: debug 2024-02-12T16:56:46.157+0000 7f322870d700 -1 osd.4 1145 *** Got signal Terminated *** 2024-02-12T16:56:46.357 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:46 smithi123 bash[22467]: debug 2024-02-12T16:56:46.157+0000 7f322870d700 -1 osd.4 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:56:46.923 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:46 smithi169 bash[28391]: debug 2024-02-12T16:56:46.666+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:46.923 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:46 smithi169 bash[38680]: debug 2024-02-12T16:56:46.830+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:47.138 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:46 smithi123 bash[22467]: debug 2024-02-12T16:56:46.353+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:47.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:46 smithi123 bash[32586]: debug 2024-02-12T16:56:46.841+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:47.139 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:46 smithi123 bash[123421]: debug 2024-02-12T16:56:46.653+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:47.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:46 smithi169 bash[33498]: debug 2024-02-12T16:56:46.978+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:47.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:46 smithi169 bash[23226]: debug 2024-02-12T16:56:46.918+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:47.834 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:47 smithi123 bash[123421]: debug 2024-02-12T16:56:47.613+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:47.835 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:47 smithi123 bash[22467]: debug 2024-02-12T16:56:47.381+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:47.835 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:47 smithi123 bash[22467]: debug 2024-02-12T16:56:47.381+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:23.077988+0000 front 2024-02-12T16:56:23.077928+0000 (oldest deadline 2024-02-12T16:56:47.177533+0000) 2024-02-12T16:56:48.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:47 smithi169 bash[33498]: debug 2024-02-12T16:56:47.958+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:48.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:47 smithi169 bash[38680]: debug 2024-02-12T16:56:47.866+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:48.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:47 smithi169 bash[23226]: debug 2024-02-12T16:56:47.926+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:48.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:47 smithi169 bash[23226]: debug 2024-02-12T16:56:47.926+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:48.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:47 smithi169 bash[28391]: debug 2024-02-12T16:56:47.710+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:48.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:47 smithi123 bash[32586]: debug 2024-02-12T16:56:47.829+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:48.876 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:48 smithi123 bash[123421]: debug 2024-02-12T16:56:48.593+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:48.876 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:48 smithi123 bash[22467]: debug 2024-02-12T16:56:48.425+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:48.876 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:48 smithi123 bash[22467]: debug 2024-02-12T16:56:48.425+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:23.077988+0000 front 2024-02-12T16:56:23.077928+0000 (oldest deadline 2024-02-12T16:56:47.177533+0000) 2024-02-12T16:56:49.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:48 smithi169 bash[33498]: debug 2024-02-12T16:56:48.938+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:49.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:48 smithi169 bash[38680]: debug 2024-02-12T16:56:48.870+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:49.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:48 smithi169 bash[23226]: debug 2024-02-12T16:56:48.902+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:49.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:48 smithi169 bash[23226]: debug 2024-02-12T16:56:48.902+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:49.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:48 smithi169 bash[28391]: debug 2024-02-12T16:56:48.686+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:49.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:48 smithi169 bash[28391]: debug 2024-02-12T16:56:48.686+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:49.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:48 smithi123 bash[32586]: debug 2024-02-12T16:56:48.873+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:49.888 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:49 smithi123 bash[22467]: debug 2024-02-12T16:56:49.457+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:49.888 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:49 smithi123 bash[22467]: debug 2024-02-12T16:56:49.457+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:23.077988+0000 front 2024-02-12T16:56:23.077928+0000 (oldest deadline 2024-02-12T16:56:47.177533+0000) 2024-02-12T16:56:49.888 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:49 smithi123 bash[123421]: debug 2024-02-12T16:56:49.557+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:49.912 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:49 smithi169 bash[38680]: debug 2024-02-12T16:56:49.838+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:49.913 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:49 smithi169 bash[28391]: debug 2024-02-12T16:56:49.650+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:49.913 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:49 smithi169 bash[28391]: debug 2024-02-12T16:56:49.650+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:50.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:49 smithi169 bash[33498]: debug 2024-02-12T16:56:49.906+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:50.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:49 smithi169 bash[23226]: debug 2024-02-12T16:56:49.910+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:50.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:49 smithi169 bash[23226]: debug 2024-02-12T16:56:49.910+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:50.388 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:49 smithi123 bash[32586]: debug 2024-02-12T16:56:49.909+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:50.878 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:50 smithi123 bash[123421]: debug 2024-02-12T16:56:50.557+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:50.878 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:50 smithi123 bash[22467]: debug 2024-02-12T16:56:50.505+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.176820+0000 front 2024-02-12T16:56:17.176908+0000 (oldest deadline 2024-02-12T16:56:43.076762+0000) 2024-02-12T16:56:50.878 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:50 smithi123 bash[22467]: debug 2024-02-12T16:56:50.505+0000 7f3221e83700 -1 osd.4 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:23.077988+0000 front 2024-02-12T16:56:23.077928+0000 (oldest deadline 2024-02-12T16:56:47.177533+0000) 2024-02-12T16:56:51.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:50 smithi169 bash[33498]: debug 2024-02-12T16:56:50.878+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:51.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:50 smithi169 bash[38680]: debug 2024-02-12T16:56:50.854+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:51.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:50 smithi169 bash[23226]: debug 2024-02-12T16:56:50.874+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:51.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:50 smithi169 bash[23226]: debug 2024-02-12T16:56:50.874+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:51.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:50 smithi169 bash[28391]: debug 2024-02-12T16:56:50.682+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:51.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:50 smithi169 bash[28391]: debug 2024-02-12T16:56:50.682+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:51.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:50 smithi123 bash[32586]: debug 2024-02-12T16:56:50.873+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:51.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:51 smithi123 bash[27408]: debug 2024-02-12T16:56:51.045+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:51.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:51 smithi123 bash[123421]: debug 2024-02-12T16:56:51.509+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:52.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:51 smithi169 bash[33498]: debug 2024-02-12T16:56:51.890+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:52.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:51 smithi169 bash[38680]: debug 2024-02-12T16:56:51.866+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:52.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:51 smithi169 bash[38680]: debug 2024-02-12T16:56:51.866+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:52.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:51 smithi169 bash[23226]: debug 2024-02-12T16:56:51.874+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:52.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:51 smithi169 bash[23226]: debug 2024-02-12T16:56:51.874+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:52.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:51 smithi169 bash[28391]: debug 2024-02-12T16:56:51.722+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:52.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:51 smithi169 bash[28391]: debug 2024-02-12T16:56:51.722+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:52.095 INFO:journalctl@ceph.osd.4.smithi123.stdout:Feb 12 16:56:51 smithi123 bash[135992]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-4 2024-02-12T16:56:52.095 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:51 smithi123 bash[32586]: debug 2024-02-12T16:56:51.833+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:52.388 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:52 smithi123 bash[27408]: debug 2024-02-12T16:56:52.089+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:52.815 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:52 smithi123 bash[123421]: debug 2024-02-12T16:56:52.513+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:53.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:52 smithi169 bash[33498]: debug 2024-02-12T16:56:52.882+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:53.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:52 smithi169 bash[23226]: debug 2024-02-12T16:56:52.870+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:53.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:52 smithi169 bash[23226]: debug 2024-02-12T16:56:52.870+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:53.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:52 smithi169 bash[28391]: debug 2024-02-12T16:56:52.762+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:53.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:52 smithi169 bash[28391]: debug 2024-02-12T16:56:52.762+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:53.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:52 smithi169 bash[38680]: debug 2024-02-12T16:56:52.838+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:53.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:52 smithi169 bash[38680]: debug 2024-02-12T16:56:52.838+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:53.067 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:52 smithi123 bash[32586]: debug 2024-02-12T16:56:52.809+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:53.388 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:53 smithi123 bash[27408]: debug 2024-02-12T16:56:53.061+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:53.838 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:53 smithi123 bash[123421]: debug 2024-02-12T16:56:53.501+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:54.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:53 smithi169 bash[33498]: debug 2024-02-12T16:56:53.910+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:54.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:53 smithi169 bash[38680]: debug 2024-02-12T16:56:53.862+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:54.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:53 smithi169 bash[38680]: debug 2024-02-12T16:56:53.862+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:54.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:53 smithi169 bash[23226]: debug 2024-02-12T16:56:53.902+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:54.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:53 smithi169 bash[23226]: debug 2024-02-12T16:56:53.902+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:54.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:53 smithi169 bash[28391]: debug 2024-02-12T16:56:53.766+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:54.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:53 smithi169 bash[28391]: debug 2024-02-12T16:56:53.766+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:54.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:54 smithi123 bash[27408]: debug 2024-02-12T16:56:54.029+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:54.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:53 smithi123 bash[32586]: debug 2024-02-12T16:56:53.833+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:54.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.4.service' 2024-02-12T16:56:54.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:56:54.567 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-12T16:56:54.567 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-12T16:56:54.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.5 2024-02-12T16:56:54.803 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:54 smithi123 systemd[1]: Stopping Ceph osd.5 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:56:54.803 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:54 smithi123 bash[136259]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.5 2024-02-12T16:56:54.803 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:54 smithi123 bash[123421]: debug 2024-02-12T16:56:54.481+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:55.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:54 smithi169 bash[33498]: debug 2024-02-12T16:56:54.882+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:55.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:54 smithi169 bash[38680]: debug 2024-02-12T16:56:54.902+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:55.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:54 smithi169 bash[38680]: debug 2024-02-12T16:56:54.902+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:55.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:54 smithi169 bash[23226]: debug 2024-02-12T16:56:54.938+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:55.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:54 smithi169 bash[23226]: debug 2024-02-12T16:56:54.938+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:55.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:54 smithi169 bash[28391]: debug 2024-02-12T16:56:54.762+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:55.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:54 smithi169 bash[28391]: debug 2024-02-12T16:56:54.762+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:55.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:54 smithi123 bash[32586]: debug 2024-02-12T16:56:54.865+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:55.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:54 smithi123 bash[27408]: debug 2024-02-12T16:56:54.797+0000 7f34bbd80700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:56:55.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:54 smithi123 bash[27408]: debug 2024-02-12T16:56:54.797+0000 7f34bbd80700 -1 osd.5 1145 *** Got signal Terminated *** 2024-02-12T16:56:55.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:54 smithi123 bash[27408]: debug 2024-02-12T16:56:54.797+0000 7f34bbd80700 -1 osd.5 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:56:55.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:55 smithi123 bash[27408]: debug 2024-02-12T16:56:55.013+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:55.821 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:55 smithi123 bash[123421]: debug 2024-02-12T16:56:55.433+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:56.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[33498]: debug 2024-02-12T16:56:55.882+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:56.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[33498]: debug 2024-02-12T16:56:55.882+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:56:56.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[38680]: debug 2024-02-12T16:56:55.882+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:56.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[38680]: debug 2024-02-12T16:56:55.882+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:56.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[38680]: debug 2024-02-12T16:56:55.882+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:56:56.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[23226]: debug 2024-02-12T16:56:55.990+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:56.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[23226]: debug 2024-02-12T16:56:55.990+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:56.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[28391]: debug 2024-02-12T16:56:55.738+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:56.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:55 smithi169 bash[28391]: debug 2024-02-12T16:56:55.738+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:56.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:56 smithi123 bash[27408]: debug 2024-02-12T16:56:56.029+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:56.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:55 smithi123 bash[32586]: debug 2024-02-12T16:56:55.813+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:56.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:55 smithi123 bash[32586]: debug 2024-02-12T16:56:55.813+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:56:56.803 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:56 smithi123 bash[123421]: debug 2024-02-12T16:56:56.393+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:57.008 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[33498]: debug 2024-02-12T16:56:56.846+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:57.008 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[33498]: debug 2024-02-12T16:56:56.846+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:56:57.008 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[38680]: debug 2024-02-12T16:56:56.846+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:57.008 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[38680]: debug 2024-02-12T16:56:56.846+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:57.009 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[38680]: debug 2024-02-12T16:56:56.846+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:56:57.009 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[28391]: debug 2024-02-12T16:56:56.702+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:57.009 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[28391]: debug 2024-02-12T16:56:56.702+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:57.009 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:56 smithi169 bash[28391]: debug 2024-02-12T16:56:56.702+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:56:57.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:57 smithi123 bash[27408]: debug 2024-02-12T16:56:57.021+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:57.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:57 smithi123 bash[27408]: debug 2024-02-12T16:56:57.021+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:32.758988+0000 front 2024-02-12T16:56:32.759278+0000 (oldest deadline 2024-02-12T16:56:56.259179+0000) 2024-02-12T16:56:57.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:56 smithi123 bash[32586]: debug 2024-02-12T16:56:56.797+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:57.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:56 smithi123 bash[32586]: debug 2024-02-12T16:56:56.797+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:56:57.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[23226]: debug 2024-02-12T16:56:57.002+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:57.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[23226]: debug 2024-02-12T16:56:57.002+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:57.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:57 smithi123 bash[123421]: debug 2024-02-12T16:56:57.381+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:58.028 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[38680]: debug 2024-02-12T16:56:57.826+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:58.028 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[38680]: debug 2024-02-12T16:56:57.826+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:58.029 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[38680]: debug 2024-02-12T16:56:57.826+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:56:58.029 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[28391]: debug 2024-02-12T16:56:57.742+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:58.029 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[28391]: debug 2024-02-12T16:56:57.742+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:58.029 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[28391]: debug 2024-02-12T16:56:57.742+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:56:58.029 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[33498]: debug 2024-02-12T16:56:57.810+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:58.029 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:57 smithi169 bash[33498]: debug 2024-02-12T16:56:57.810+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:56:58.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:57 smithi123 bash[27408]: debug 2024-02-12T16:56:57.993+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:58.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:57 smithi123 bash[27408]: debug 2024-02-12T16:56:57.993+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:32.758988+0000 front 2024-02-12T16:56:32.759278+0000 (oldest deadline 2024-02-12T16:56:56.259179+0000) 2024-02-12T16:56:58.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:57 smithi123 bash[32586]: debug 2024-02-12T16:56:57.773+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:58.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:57 smithi123 bash[32586]: debug 2024-02-12T16:56:57.773+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:56:58.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[23226]: debug 2024-02-12T16:56:58.022+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:58.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[23226]: debug 2024-02-12T16:56:58.022+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:58.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:58 smithi123 bash[123421]: debug 2024-02-12T16:56:58.337+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:58.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:58 smithi123 bash[123421]: debug 2024-02-12T16:56:58.337+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:56:59.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[33498]: debug 2024-02-12T16:56:58.838+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:56:59.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[33498]: debug 2024-02-12T16:56:58.838+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[38680]: debug 2024-02-12T16:56:58.778+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[38680]: debug 2024-02-12T16:56:58.778+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[38680]: debug 2024-02-12T16:56:58.778+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[23226]: debug 2024-02-12T16:56:58.978+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[23226]: debug 2024-02-12T16:56:58.978+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[28391]: debug 2024-02-12T16:56:58.766+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[28391]: debug 2024-02-12T16:56:58.766+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:56:59.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:58 smithi169 bash[28391]: debug 2024-02-12T16:56:58.766+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:56:59.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:59 smithi123 bash[27408]: debug 2024-02-12T16:56:59.001+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.157533+0000 front 2024-02-12T16:56:25.157650+0000 (oldest deadline 2024-02-12T16:56:50.457318+0000) 2024-02-12T16:56:59.138 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:56:59 smithi123 bash[27408]: debug 2024-02-12T16:56:59.001+0000 7f34b84fc700 -1 osd.5 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:32.758988+0000 front 2024-02-12T16:56:32.759278+0000 (oldest deadline 2024-02-12T16:56:56.259179+0000) 2024-02-12T16:56:59.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:58 smithi123 bash[32586]: debug 2024-02-12T16:56:58.813+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:56:59.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:58 smithi123 bash[32586]: debug 2024-02-12T16:56:58.813+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:56:59.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:59 smithi123 bash[123421]: debug 2024-02-12T16:56:59.317+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:56:59.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:56:59 smithi123 bash[123421]: debug 2024-02-12T16:56:59.317+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:00.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[33498]: debug 2024-02-12T16:56:59.870+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:00.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[33498]: debug 2024-02-12T16:56:59.870+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[23226]: debug 2024-02-12T16:56:59.978+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[23226]: debug 2024-02-12T16:56:59.978+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[28391]: debug 2024-02-12T16:56:59.734+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[28391]: debug 2024-02-12T16:56:59.734+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[28391]: debug 2024-02-12T16:56:59.734+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[38680]: debug 2024-02-12T16:56:59.794+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[38680]: debug 2024-02-12T16:56:59.794+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:00.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:56:59 smithi169 bash[38680]: debug 2024-02-12T16:56:59.794+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:00.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:59 smithi123 bash[32586]: debug 2024-02-12T16:56:59.809+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:00.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:56:59 smithi123 bash[32586]: debug 2024-02-12T16:56:59.809+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:00.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:00 smithi123 bash[123421]: debug 2024-02-12T16:57:00.309+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:00.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:00 smithi123 bash[123421]: debug 2024-02-12T16:57:00.309+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:01.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[33498]: debug 2024-02-12T16:57:00.918+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:01.048 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[33498]: debug 2024-02-12T16:57:00.918+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:01.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[23226]: debug 2024-02-12T16:57:00.978+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:01.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[23226]: debug 2024-02-12T16:57:00.978+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:01.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[28391]: debug 2024-02-12T16:57:00.758+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:01.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[28391]: debug 2024-02-12T16:57:00.758+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:01.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[28391]: debug 2024-02-12T16:57:00.758+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:01.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[38680]: debug 2024-02-12T16:57:00.794+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:01.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[38680]: debug 2024-02-12T16:57:00.794+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:01.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:00 smithi169 bash[38680]: debug 2024-02-12T16:57:00.794+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:01.142 INFO:journalctl@ceph.osd.5.smithi123.stdout:Feb 12 16:57:00 smithi123 bash[136259]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-5 2024-02-12T16:57:01.142 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:00 smithi123 bash[32586]: debug 2024-02-12T16:57:00.785+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:01.142 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:00 smithi123 bash[32586]: debug 2024-02-12T16:57:00.785+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:01.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:01 smithi123 bash[123421]: debug 2024-02-12T16:57:01.297+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:01.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:01 smithi123 bash[123421]: debug 2024-02-12T16:57:01.297+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:02.011 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[33498]: debug 2024-02-12T16:57:01.918+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:02.011 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[33498]: debug 2024-02-12T16:57:01.918+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:02.011 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[38680]: debug 2024-02-12T16:57:01.822+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:02.011 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[38680]: debug 2024-02-12T16:57:01.822+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:02.012 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[38680]: debug 2024-02-12T16:57:01.822+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:02.012 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[28391]: debug 2024-02-12T16:57:01.730+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:02.012 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[28391]: debug 2024-02-12T16:57:01.730+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:02.012 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:01 smithi169 bash[28391]: debug 2024-02-12T16:57:01.730+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:02.017 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:01 smithi123 bash[32586]: debug 2024-02-12T16:57:01.753+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:02.017 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:01 smithi123 bash[32586]: debug 2024-02-12T16:57:01.753+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:02.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[23226]: debug 2024-02-12T16:57:02.006+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:02.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[23226]: debug 2024-02-12T16:57:02.006+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:02.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:02 smithi123 bash[123421]: debug 2024-02-12T16:57:02.293+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:02.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:02 smithi123 bash[123421]: debug 2024-02-12T16:57:02.293+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:02.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.5.service' 2024-02-12T16:57:03.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:57:03.004 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-12T16:57:03.004 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-12T16:57:03.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.6 2024-02-12T16:57:03.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[33498]: debug 2024-02-12T16:57:02.954+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:03.047 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[33498]: debug 2024-02-12T16:57:02.954+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[23226]: debug 2024-02-12T16:57:03.010+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[23226]: debug 2024-02-12T16:57:03.010+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[38680]: debug 2024-02-12T16:57:02.842+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[38680]: debug 2024-02-12T16:57:02.842+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[38680]: debug 2024-02-12T16:57:02.842+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[28391]: debug 2024-02-12T16:57:02.774+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[28391]: debug 2024-02-12T16:57:02.774+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:03.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:02 smithi169 bash[28391]: debug 2024-02-12T16:57:02.774+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:03.056 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:02 smithi123 bash[32586]: debug 2024-02-12T16:57:02.769+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:03.056 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:02 smithi123 bash[32586]: debug 2024-02-12T16:57:02.769+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:03.341 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:03 smithi123 systemd[1]: Stopping Ceph osd.6 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:57:03.341 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[136519]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.6 2024-02-12T16:57:03.341 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[32586]: debug 2024-02-12T16:57:03.249+0000 7f2763d8a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:57:03.341 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[32586]: debug 2024-02-12T16:57:03.249+0000 7f2763d8a700 -1 osd.6 1145 *** Got signal Terminated *** 2024-02-12T16:57:03.341 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[32586]: debug 2024-02-12T16:57:03.249+0000 7f2763d8a700 -1 osd.6 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:57:03.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[123421]: debug 2024-02-12T16:57:03.337+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:03.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[123421]: debug 2024-02-12T16:57:03.337+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:04.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[32586]: debug 2024-02-12T16:57:03.813+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:04.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:03 smithi123 bash[32586]: debug 2024-02-12T16:57:03.813+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:04.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[33498]: debug 2024-02-12T16:57:03.930+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:04.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[33498]: debug 2024-02-12T16:57:03.930+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:04.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[38680]: debug 2024-02-12T16:57:03.818+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:04.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[38680]: debug 2024-02-12T16:57:03.818+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:04.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[38680]: debug 2024-02-12T16:57:03.818+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:04.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[23226]: debug 2024-02-12T16:57:04.058+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:04.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[23226]: debug 2024-02-12T16:57:04.058+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:04.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[23226]: debug 2024-02-12T16:57:04.058+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:04.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[28391]: debug 2024-02-12T16:57:03.818+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:04.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[28391]: debug 2024-02-12T16:57:03.818+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:04.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:03 smithi169 bash[28391]: debug 2024-02-12T16:57:03.818+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:04.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:04 smithi123 bash[123421]: debug 2024-02-12T16:57:04.353+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:04.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:04 smithi123 bash[123421]: debug 2024-02-12T16:57:04.353+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:05.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:04 smithi123 bash[32586]: debug 2024-02-12T16:57:04.785+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:05.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:04 smithi123 bash[32586]: debug 2024-02-12T16:57:04.785+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:05.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[33498]: debug 2024-02-12T16:57:04.946+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:05.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[33498]: debug 2024-02-12T16:57:04.946+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:05.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[38680]: debug 2024-02-12T16:57:04.838+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[38680]: debug 2024-02-12T16:57:04.838+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[38680]: debug 2024-02-12T16:57:04.838+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[23226]: debug 2024-02-12T16:57:05.078+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[23226]: debug 2024-02-12T16:57:05.078+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[23226]: debug 2024-02-12T16:57:05.078+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[28391]: debug 2024-02-12T16:57:04.834+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[28391]: debug 2024-02-12T16:57:04.834+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:05.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:04 smithi169 bash[28391]: debug 2024-02-12T16:57:04.834+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:05.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:05 smithi123 bash[123421]: debug 2024-02-12T16:57:05.313+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:05.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:05 smithi123 bash[123421]: debug 2024-02-12T16:57:05.313+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:06.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:05 smithi123 bash[32586]: debug 2024-02-12T16:57:05.773+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:06.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:05 smithi123 bash[32586]: debug 2024-02-12T16:57:05.773+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:06.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[33498]: debug 2024-02-12T16:57:05.930+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:06.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[33498]: debug 2024-02-12T16:57:05.930+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:06.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[38680]: debug 2024-02-12T16:57:05.854+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:06.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[38680]: debug 2024-02-12T16:57:05.854+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:06.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[38680]: debug 2024-02-12T16:57:05.854+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:06.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[38680]: debug 2024-02-12T16:57:05.854+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:06.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[23226]: debug 2024-02-12T16:57:06.030+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:06.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[23226]: debug 2024-02-12T16:57:06.030+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:06.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[23226]: debug 2024-02-12T16:57:06.030+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:06.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[28391]: debug 2024-02-12T16:57:05.870+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:06.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[28391]: debug 2024-02-12T16:57:05.870+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:06.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[28391]: debug 2024-02-12T16:57:05.870+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:06.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:05 smithi169 bash[28391]: debug 2024-02-12T16:57:05.870+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:06.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:06 smithi123 bash[123421]: debug 2024-02-12T16:57:06.289+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:06.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:06 smithi123 bash[123421]: debug 2024-02-12T16:57:06.289+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:06.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:06 smithi123 bash[123421]: debug 2024-02-12T16:57:06.289+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:07.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:06 smithi123 bash[32586]: debug 2024-02-12T16:57:06.753+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:07.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:06 smithi123 bash[32586]: debug 2024-02-12T16:57:06.753+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:07.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:06 smithi123 bash[32586]: debug 2024-02-12T16:57:06.753+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.654009+0000 front 2024-02-12T16:56:41.654209+0000 (oldest deadline 2024-02-12T16:57:06.354196+0000) 2024-02-12T16:57:07.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[33498]: debug 2024-02-12T16:57:06.953+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:07.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[33498]: debug 2024-02-12T16:57:06.953+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:07.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[23226]: debug 2024-02-12T16:57:07.057+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[23226]: debug 2024-02-12T16:57:07.057+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[23226]: debug 2024-02-12T16:57:07.057+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[38680]: debug 2024-02-12T16:57:06.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[38680]: debug 2024-02-12T16:57:06.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[38680]: debug 2024-02-12T16:57:06.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[38680]: debug 2024-02-12T16:57:06.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[28391]: debug 2024-02-12T16:57:06.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[28391]: debug 2024-02-12T16:57:06.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:07.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[28391]: debug 2024-02-12T16:57:06.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:07.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:06 smithi169 bash[28391]: debug 2024-02-12T16:57:06.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:07.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:07 smithi123 bash[123421]: debug 2024-02-12T16:57:07.273+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:07.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:07 smithi123 bash[123421]: debug 2024-02-12T16:57:07.273+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:07.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:07 smithi123 bash[123421]: debug 2024-02-12T16:57:07.273+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:08.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:07 smithi123 bash[32586]: debug 2024-02-12T16:57:07.729+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:18.043209+0000 front 2024-02-12T16:56:18.043412+0000 (oldest deadline 2024-02-12T16:56:39.743069+0000) 2024-02-12T16:57:08.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:07 smithi123 bash[32586]: debug 2024-02-12T16:57:07.729+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:34.750994+0000 front 2024-02-12T16:56:34.750763+0000 (oldest deadline 2024-02-12T16:56:55.250811+0000) 2024-02-12T16:57:08.138 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:07 smithi123 bash[32586]: debug 2024-02-12T16:57:07.729+0000 7f275fd05700 -1 osd.6 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.654009+0000 front 2024-02-12T16:56:41.654209+0000 (oldest deadline 2024-02-12T16:57:06.354196+0000) 2024-02-12T16:57:08.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[33498]: debug 2024-02-12T16:57:07.953+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:08.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[33498]: debug 2024-02-12T16:57:07.953+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:08.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[38680]: debug 2024-02-12T16:57:07.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:08.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[38680]: debug 2024-02-12T16:57:07.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:08.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[38680]: debug 2024-02-12T16:57:07.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:08.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[38680]: debug 2024-02-12T16:57:07.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:08.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[23226]: debug 2024-02-12T16:57:08.069+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:08.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[23226]: debug 2024-02-12T16:57:08.069+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:08.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[23226]: debug 2024-02-12T16:57:08.069+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:08.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[28391]: debug 2024-02-12T16:57:07.865+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:08.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[28391]: debug 2024-02-12T16:57:07.865+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:08.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[28391]: debug 2024-02-12T16:57:07.865+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:08.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:07 smithi169 bash[28391]: debug 2024-02-12T16:57:07.865+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:08.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:08 smithi123 bash[123421]: debug 2024-02-12T16:57:08.265+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:08.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:08 smithi123 bash[123421]: debug 2024-02-12T16:57:08.265+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:08.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:08 smithi123 bash[123421]: debug 2024-02-12T16:57:08.265+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:09.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[33498]: debug 2024-02-12T16:57:08.937+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:09.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[33498]: debug 2024-02-12T16:57:08.937+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:09.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[33498]: debug 2024-02-12T16:57:08.937+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:09.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[38680]: debug 2024-02-12T16:57:08.989+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:09.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[38680]: debug 2024-02-12T16:57:08.989+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:09.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[38680]: debug 2024-02-12T16:57:08.989+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:09.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[38680]: debug 2024-02-12T16:57:08.989+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:09.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[23226]: debug 2024-02-12T16:57:09.037+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:09.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[23226]: debug 2024-02-12T16:57:09.037+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:09.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[23226]: debug 2024-02-12T16:57:09.037+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:09.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[28391]: debug 2024-02-12T16:57:08.881+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:09.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[28391]: debug 2024-02-12T16:57:08.881+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:09.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[28391]: debug 2024-02-12T16:57:08.881+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:09.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:08 smithi169 bash[28391]: debug 2024-02-12T16:57:08.881+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:09.388 INFO:journalctl@ceph.osd.6.smithi123.stdout:Feb 12 16:57:09 smithi123 bash[136519]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-6 2024-02-12T16:57:09.389 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:09 smithi123 bash[123421]: debug 2024-02-12T16:57:09.233+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:09.389 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:09 smithi123 bash[123421]: debug 2024-02-12T16:57:09.233+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:09.389 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:09 smithi123 bash[123421]: debug 2024-02-12T16:57:09.233+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:10.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[33498]: debug 2024-02-12T16:57:09.937+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:10.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[33498]: debug 2024-02-12T16:57:09.937+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:10.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[33498]: debug 2024-02-12T16:57:09.937+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:10.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[38680]: debug 2024-02-12T16:57:09.957+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:10.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[38680]: debug 2024-02-12T16:57:09.957+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:10.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[38680]: debug 2024-02-12T16:57:09.957+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:10.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[38680]: debug 2024-02-12T16:57:09.957+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:10.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[23226]: debug 2024-02-12T16:57:10.069+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:10.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[23226]: debug 2024-02-12T16:57:10.069+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:10.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[23226]: debug 2024-02-12T16:57:10.069+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:10.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[28391]: debug 2024-02-12T16:57:09.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:10.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[28391]: debug 2024-02-12T16:57:09.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:10.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[28391]: debug 2024-02-12T16:57:09.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:10.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:09 smithi169 bash[28391]: debug 2024-02-12T16:57:09.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:10.634 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:10 smithi123 bash[123421]: debug 2024-02-12T16:57:10.249+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:10.634 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:10 smithi123 bash[123421]: debug 2024-02-12T16:57:10.249+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:10.634 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:10 smithi123 bash[123421]: debug 2024-02-12T16:57:10.249+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:11.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.6.service' 2024-02-12T16:57:11.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:57:11.221 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-12T16:57:11.221 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-12T16:57:11.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.7 2024-02-12T16:57:11.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[33498]: debug 2024-02-12T16:57:10.925+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:11.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[33498]: debug 2024-02-12T16:57:10.925+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:11.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[33498]: debug 2024-02-12T16:57:10.925+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:11.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[38680]: debug 2024-02-12T16:57:10.985+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[38680]: debug 2024-02-12T16:57:10.985+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[38680]: debug 2024-02-12T16:57:10.985+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[38680]: debug 2024-02-12T16:57:10.985+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[23226]: debug 2024-02-12T16:57:11.041+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[23226]: debug 2024-02-12T16:57:11.041+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[23226]: debug 2024-02-12T16:57:11.041+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[28391]: debug 2024-02-12T16:57:10.893+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[28391]: debug 2024-02-12T16:57:10.893+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[28391]: debug 2024-02-12T16:57:10.893+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:11.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:10 smithi169 bash[28391]: debug 2024-02-12T16:57:10.893+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:11.484 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 systemd[1]: Stopping Ceph osd.7 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:57:11.484 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 bash[123421]: debug 2024-02-12T16:57:11.285+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:11.484 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 bash[123421]: debug 2024-02-12T16:57:11.285+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:11.484 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 bash[123421]: debug 2024-02-12T16:57:11.285+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:11.484 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 bash[136782]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.7 2024-02-12T16:57:11.484 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 bash[123421]: debug 2024-02-12T16:57:11.477+0000 7f77c5dcb700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:57:11.484 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 bash[123421]: debug 2024-02-12T16:57:11.477+0000 7f77c5dcb700 -1 osd.7 1145 *** Got signal Terminated *** 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[33498]: debug 2024-02-12T16:57:11.949+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[33498]: debug 2024-02-12T16:57:11.949+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[33498]: debug 2024-02-12T16:57:11.949+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[33498]: debug 2024-02-12T16:57:11.949+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[38680]: debug 2024-02-12T16:57:12.001+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[38680]: debug 2024-02-12T16:57:12.001+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[38680]: debug 2024-02-12T16:57:12.001+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:12.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[38680]: debug 2024-02-12T16:57:12.001+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:12.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[23226]: debug 2024-02-12T16:57:11.997+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:12.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[23226]: debug 2024-02-12T16:57:11.997+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:12.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[23226]: debug 2024-02-12T16:57:11.997+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:12.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[28391]: debug 2024-02-12T16:57:11.925+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:12.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[28391]: debug 2024-02-12T16:57:11.925+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:12.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[28391]: debug 2024-02-12T16:57:11.925+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:12.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:11 smithi169 bash[28391]: debug 2024-02-12T16:57:11.925+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:12.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:11 smithi123 bash[123421]: debug 2024-02-12T16:57:11.477+0000 7f77c5dcb700 -1 osd.7 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:57:12.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:12 smithi123 bash[123421]: debug 2024-02-12T16:57:12.301+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:12.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:12 smithi123 bash[123421]: debug 2024-02-12T16:57:12.301+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:12.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:12 smithi123 bash[123421]: debug 2024-02-12T16:57:12.301+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:12.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:12 smithi123 bash[123421]: debug 2024-02-12T16:57:12.301+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:49.775147+0000 front 2024-02-12T16:56:49.775402+0000 (oldest deadline 2024-02-12T16:57:12.075343+0000) 2024-02-12T16:57:13.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[33498]: debug 2024-02-12T16:57:12.985+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:13.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[33498]: debug 2024-02-12T16:57:12.985+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:13.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[33498]: debug 2024-02-12T16:57:12.985+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:13.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[33498]: debug 2024-02-12T16:57:12.985+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:13.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.013+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:13.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.013+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:13.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.013+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:13.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.013+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:13.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[23226]: debug 2024-02-12T16:57:13.013+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:13.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[23226]: debug 2024-02-12T16:57:13.013+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:13.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[23226]: debug 2024-02-12T16:57:13.013+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:13.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[28391]: debug 2024-02-12T16:57:12.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:13.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[28391]: debug 2024-02-12T16:57:12.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:13.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[28391]: debug 2024-02-12T16:57:12.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:13.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[28391]: debug 2024-02-12T16:57:12.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:13.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:12 smithi169 bash[28391]: debug 2024-02-12T16:57:12.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:13.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:13 smithi123 bash[123421]: debug 2024-02-12T16:57:13.308+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:13.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:13 smithi123 bash[123421]: debug 2024-02-12T16:57:13.308+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:13.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:13 smithi123 bash[123421]: debug 2024-02-12T16:57:13.308+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:13.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:13 smithi123 bash[123421]: debug 2024-02-12T16:57:13.308+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:49.775147+0000 front 2024-02-12T16:56:49.775402+0000 (oldest deadline 2024-02-12T16:57:12.075343+0000) 2024-02-12T16:57:14.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.981+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:14.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.981+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:14.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.981+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:14.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[38680]: debug 2024-02-12T16:57:13.981+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:14.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[23226]: debug 2024-02-12T16:57:14.065+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:14.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[23226]: debug 2024-02-12T16:57:14.065+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:14.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[23226]: debug 2024-02-12T16:57:14.065+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:14.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[33498]: debug 2024-02-12T16:57:13.993+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:14.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[33498]: debug 2024-02-12T16:57:13.993+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:14.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[33498]: debug 2024-02-12T16:57:13.993+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:14.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[33498]: debug 2024-02-12T16:57:13.993+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:14.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[28391]: debug 2024-02-12T16:57:13.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:14.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[28391]: debug 2024-02-12T16:57:13.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:14.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[28391]: debug 2024-02-12T16:57:13.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:14.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[28391]: debug 2024-02-12T16:57:13.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:14.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:13 smithi169 bash[28391]: debug 2024-02-12T16:57:13.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:14.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:14 smithi123 bash[123421]: debug 2024-02-12T16:57:14.324+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:14.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:14 smithi123 bash[123421]: debug 2024-02-12T16:57:14.324+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:14.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:14 smithi123 bash[123421]: debug 2024-02-12T16:57:14.324+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:14.639 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:14 smithi123 bash[123421]: debug 2024-02-12T16:57:14.324+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:49.775147+0000 front 2024-02-12T16:56:49.775402+0000 (oldest deadline 2024-02-12T16:57:12.075343+0000) 2024-02-12T16:57:15.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[38680]: debug 2024-02-12T16:57:14.933+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:15.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[38680]: debug 2024-02-12T16:57:14.933+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:15.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[38680]: debug 2024-02-12T16:57:14.933+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:15.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[38680]: debug 2024-02-12T16:57:14.933+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:15.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:15.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:15.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:15.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:15.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[23226]: debug 2024-02-12T16:57:15.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:15.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[23226]: debug 2024-02-12T16:57:15.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:15.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[23226]: debug 2024-02-12T16:57:15.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:15.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[23226]: debug 2024-02-12T16:57:15.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:15.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[28391]: debug 2024-02-12T16:57:14.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:15.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[28391]: debug 2024-02-12T16:57:14.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:15.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[28391]: debug 2024-02-12T16:57:14.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:15.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[28391]: debug 2024-02-12T16:57:14.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:15.300 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:14 smithi169 bash[28391]: debug 2024-02-12T16:57:14.933+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:15.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:15 smithi123 bash[123421]: debug 2024-02-12T16:57:15.364+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:15.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:15 smithi123 bash[123421]: debug 2024-02-12T16:57:15.364+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:15.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:15 smithi123 bash[123421]: debug 2024-02-12T16:57:15.364+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:15.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:15 smithi123 bash[123421]: debug 2024-02-12T16:57:15.364+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:49.775147+0000 front 2024-02-12T16:56:49.775402+0000 (oldest deadline 2024-02-12T16:57:12.075343+0000) 2024-02-12T16:57:16.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.961+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:16.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.961+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:16.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.961+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:16.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[33498]: debug 2024-02-12T16:57:15.961+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:16.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[38680]: debug 2024-02-12T16:57:15.917+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:16.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[38680]: debug 2024-02-12T16:57:15.917+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:16.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[38680]: debug 2024-02-12T16:57:15.917+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:16.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[38680]: debug 2024-02-12T16:57:15.917+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:16.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[23226]: debug 2024-02-12T16:57:16.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:16.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[23226]: debug 2024-02-12T16:57:16.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:16.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[23226]: debug 2024-02-12T16:57:16.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:16.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[23226]: debug 2024-02-12T16:57:16.061+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:16.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[28391]: debug 2024-02-12T16:57:15.977+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:16.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[28391]: debug 2024-02-12T16:57:15.977+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:16.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[28391]: debug 2024-02-12T16:57:15.977+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:16.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[28391]: debug 2024-02-12T16:57:15.977+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:16.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:15 smithi169 bash[28391]: debug 2024-02-12T16:57:15.977+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:16.888 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:16 smithi123 bash[123421]: debug 2024-02-12T16:57:16.384+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:15.965001+0000 front 2024-02-12T16:56:15.964958+0000 (oldest deadline 2024-02-12T16:56:40.665156+0000) 2024-02-12T16:57:16.888 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:16 smithi123 bash[123421]: debug 2024-02-12T16:57:16.384+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.670385+0000 front 2024-02-12T16:56:31.670411+0000 (oldest deadline 2024-02-12T16:56:57.570421+0000) 2024-02-12T16:57:16.888 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:16 smithi123 bash[123421]: debug 2024-02-12T16:57:16.384+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.573257+0000 front 2024-02-12T16:56:41.573397+0000 (oldest deadline 2024-02-12T16:57:06.273214+0000) 2024-02-12T16:57:16.888 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:16 smithi123 bash[123421]: debug 2024-02-12T16:57:16.384+0000 7f77bfd42700 -1 osd.7 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:49.775147+0000 front 2024-02-12T16:56:49.775402+0000 (oldest deadline 2024-02-12T16:57:12.075343+0000) 2024-02-12T16:57:17.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[33498]: debug 2024-02-12T16:57:16.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:17.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[33498]: debug 2024-02-12T16:57:16.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:17.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[33498]: debug 2024-02-12T16:57:16.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:17.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[33498]: debug 2024-02-12T16:57:16.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:17.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[38680]: debug 2024-02-12T16:57:16.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:17.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[38680]: debug 2024-02-12T16:57:16.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:17.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[38680]: debug 2024-02-12T16:57:16.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:17.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[38680]: debug 2024-02-12T16:57:16.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:17.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.021+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:17.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.021+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:17.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.021+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:17.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.021+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:17.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[28391]: debug 2024-02-12T16:57:16.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:17.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[28391]: debug 2024-02-12T16:57:16.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:17.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[28391]: debug 2024-02-12T16:57:16.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:17.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[28391]: debug 2024-02-12T16:57:16.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:17.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:16 smithi169 bash[28391]: debug 2024-02-12T16:57:16.941+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:17.638 INFO:journalctl@ceph.osd.7.smithi123.stdout:Feb 12 16:57:17 smithi123 bash[136782]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-7 2024-02-12T16:57:18.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[38680]: debug 2024-02-12T16:57:17.873+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:18.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[38680]: debug 2024-02-12T16:57:17.873+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[38680]: debug 2024-02-12T16:57:17.873+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[38680]: debug 2024-02-12T16:57:17.873+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[33498]: debug 2024-02-12T16:57:17.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[33498]: debug 2024-02-12T16:57:17.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[33498]: debug 2024-02-12T16:57:17.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[33498]: debug 2024-02-12T16:57:17.977+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.981+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:18.298 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.981+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:18.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.981+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:18.299 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[23226]: debug 2024-02-12T16:57:17.981+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:18.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[28391]: debug 2024-02-12T16:57:17.917+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:18.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[28391]: debug 2024-02-12T16:57:17.917+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:18.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[28391]: debug 2024-02-12T16:57:17.917+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:18.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[28391]: debug 2024-02-12T16:57:17.917+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:18.299 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:17 smithi169 bash[28391]: debug 2024-02-12T16:57:17.917+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:19.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.7.service' 2024-02-12T16:57:19.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:57:19.270 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-12T16:57:19.271 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-02-12T16:57:19.271 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.8 2024-02-12T16:57:19.283 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[38680]: debug 2024-02-12T16:57:18.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:19.283 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[38680]: debug 2024-02-12T16:57:18.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:19.283 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[38680]: debug 2024-02-12T16:57:18.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:19.283 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[38680]: debug 2024-02-12T16:57:18.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:19.284 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[23226]: debug 2024-02-12T16:57:18.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:19.284 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[23226]: debug 2024-02-12T16:57:18.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:19.284 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[23226]: debug 2024-02-12T16:57:18.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:19.284 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[23226]: debug 2024-02-12T16:57:18.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:19.284 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[28391]: debug 2024-02-12T16:57:18.949+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:19.284 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[28391]: debug 2024-02-12T16:57:18.949+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:19.284 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[28391]: debug 2024-02-12T16:57:18.949+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:19.285 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[28391]: debug 2024-02-12T16:57:18.949+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:19.285 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:18 smithi169 bash[28391]: debug 2024-02-12T16:57:18.949+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:19.285 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[33498]: debug 2024-02-12T16:57:19.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:19.285 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[33498]: debug 2024-02-12T16:57:19.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:19.285 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[33498]: debug 2024-02-12T16:57:19.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:19.285 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[33498]: debug 2024-02-12T16:57:19.009+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:19.547 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 systemd[1]: Stopping Ceph osd.8 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:57:19.547 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[136007]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.8 2024-02-12T16:57:19.928 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[23226]: debug 2024-02-12T16:57:19.577+0000 7fefba4d3700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:57:19.928 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[23226]: debug 2024-02-12T16:57:19.577+0000 7fefba4d3700 -1 osd.8 1145 *** Got signal Terminated *** 2024-02-12T16:57:19.928 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[23226]: debug 2024-02-12T16:57:19.577+0000 7fefba4d3700 -1 osd.8 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:57:19.928 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[38680]: debug 2024-02-12T16:57:19.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:19.928 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[38680]: debug 2024-02-12T16:57:19.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:19.928 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[38680]: debug 2024-02-12T16:57:19.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:19.928 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[38680]: debug 2024-02-12T16:57:19.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:20.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[23226]: debug 2024-02-12T16:57:19.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:20.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[23226]: debug 2024-02-12T16:57:19.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:20.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[23226]: debug 2024-02-12T16:57:19.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:20.297 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[23226]: debug 2024-02-12T16:57:19.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[33498]: debug 2024-02-12T16:57:20.053+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[33498]: debug 2024-02-12T16:57:20.053+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[33498]: debug 2024-02-12T16:57:20.053+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[33498]: debug 2024-02-12T16:57:20.053+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[28391]: debug 2024-02-12T16:57:19.921+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[28391]: debug 2024-02-12T16:57:19.921+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[28391]: debug 2024-02-12T16:57:19.921+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[28391]: debug 2024-02-12T16:57:19.921+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:20.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:19 smithi169 bash[28391]: debug 2024-02-12T16:57:19.921+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:21.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[23226]: debug 2024-02-12T16:57:20.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:21.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[23226]: debug 2024-02-12T16:57:20.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[23226]: debug 2024-02-12T16:57:20.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[23226]: debug 2024-02-12T16:57:20.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[23226]: debug 2024-02-12T16:57:20.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:58.184744+0000 front 2024-02-12T16:56:58.184863+0000 (oldest deadline 2024-02-12T16:57:20.484951+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[28391]: debug 2024-02-12T16:57:20.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[28391]: debug 2024-02-12T16:57:20.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[28391]: debug 2024-02-12T16:57:20.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[28391]: debug 2024-02-12T16:57:20.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[28391]: debug 2024-02-12T16:57:20.877+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:21.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[38680]: debug 2024-02-12T16:57:20.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:21.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[38680]: debug 2024-02-12T16:57:20.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:21.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[38680]: debug 2024-02-12T16:57:20.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:21.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[38680]: debug 2024-02-12T16:57:20.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:21.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:20 smithi169 bash[38680]: debug 2024-02-12T16:57:20.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:21.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[33498]: debug 2024-02-12T16:57:21.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:21.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[33498]: debug 2024-02-12T16:57:21.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:21.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[33498]: debug 2024-02-12T16:57:21.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:21.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[33498]: debug 2024-02-12T16:57:21.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:21.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[33498]: debug 2024-02-12T16:57:21.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:22.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[23226]: debug 2024-02-12T16:57:21.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:22.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[23226]: debug 2024-02-12T16:57:21.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[23226]: debug 2024-02-12T16:57:21.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[23226]: debug 2024-02-12T16:57:21.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[23226]: debug 2024-02-12T16:57:21.965+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:58.184744+0000 front 2024-02-12T16:56:58.184863+0000 (oldest deadline 2024-02-12T16:57:20.484951+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[28391]: debug 2024-02-12T16:57:21.853+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[28391]: debug 2024-02-12T16:57:21.853+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[28391]: debug 2024-02-12T16:57:21.853+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[28391]: debug 2024-02-12T16:57:21.853+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[28391]: debug 2024-02-12T16:57:21.853+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:22.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[28391]: debug 2024-02-12T16:57:21.853+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:22.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[38680]: debug 2024-02-12T16:57:21.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:22.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[38680]: debug 2024-02-12T16:57:21.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:22.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[38680]: debug 2024-02-12T16:57:21.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:22.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[38680]: debug 2024-02-12T16:57:21.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:22.049 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:21 smithi169 bash[38680]: debug 2024-02-12T16:57:21.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:22.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[33498]: debug 2024-02-12T16:57:22.101+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:22.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[33498]: debug 2024-02-12T16:57:22.101+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:22.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[33498]: debug 2024-02-12T16:57:22.101+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:22.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[33498]: debug 2024-02-12T16:57:22.101+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:22.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[33498]: debug 2024-02-12T16:57:22.101+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:23.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[23226]: debug 2024-02-12T16:57:22.985+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:23.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[23226]: debug 2024-02-12T16:57:22.985+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[23226]: debug 2024-02-12T16:57:22.985+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[23226]: debug 2024-02-12T16:57:22.985+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[23226]: debug 2024-02-12T16:57:22.985+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:58.184744+0000 front 2024-02-12T16:56:58.184863+0000 (oldest deadline 2024-02-12T16:57:20.484951+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[38680]: debug 2024-02-12T16:57:22.781+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[38680]: debug 2024-02-12T16:57:22.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[38680]: debug 2024-02-12T16:57:22.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[38680]: debug 2024-02-12T16:57:22.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:23.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[38680]: debug 2024-02-12T16:57:22.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:23.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[28391]: debug 2024-02-12T16:57:22.805+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:23.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[28391]: debug 2024-02-12T16:57:22.805+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:23.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[28391]: debug 2024-02-12T16:57:22.805+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:23.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[28391]: debug 2024-02-12T16:57:22.805+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:23.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[28391]: debug 2024-02-12T16:57:22.805+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:23.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:22 smithi169 bash[28391]: debug 2024-02-12T16:57:22.805+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:23.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[33498]: debug 2024-02-12T16:57:23.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:23.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[33498]: debug 2024-02-12T16:57:23.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:23.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[33498]: debug 2024-02-12T16:57:23.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:23.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[33498]: debug 2024-02-12T16:57:23.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:23.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[33498]: debug 2024-02-12T16:57:23.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:24.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[23226]: debug 2024-02-12T16:57:23.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.870702+0000 front 2024-02-12T16:56:17.870255+0000 (oldest deadline 2024-02-12T16:56:38.370312+0000) 2024-02-12T16:57:24.047 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[23226]: debug 2024-02-12T16:57:23.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.973007+0000 front 2024-02-12T16:56:25.973171+0000 (oldest deadline 2024-02-12T16:56:47.673090+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[23226]: debug 2024-02-12T16:57:23.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.478667+0000 front 2024-02-12T16:56:41.478690+0000 (oldest deadline 2024-02-12T16:57:03.778867+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[23226]: debug 2024-02-12T16:57:23.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.682435+0000 front 2024-02-12T16:56:50.682541+0000 (oldest deadline 2024-02-12T16:57:14.182238+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[23226]: debug 2024-02-12T16:57:23.941+0000 7fefb5c4d700 -1 osd.8 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:58.184744+0000 front 2024-02-12T16:56:58.184863+0000 (oldest deadline 2024-02-12T16:57:20.484951+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[38680]: debug 2024-02-12T16:57:23.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[38680]: debug 2024-02-12T16:57:23.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[38680]: debug 2024-02-12T16:57:23.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[38680]: debug 2024-02-12T16:57:23.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:24.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[38680]: debug 2024-02-12T16:57:23.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:24.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[28391]: debug 2024-02-12T16:57:23.773+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:24.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[28391]: debug 2024-02-12T16:57:23.773+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:24.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[28391]: debug 2024-02-12T16:57:23.773+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:24.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[28391]: debug 2024-02-12T16:57:23.773+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:24.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[28391]: debug 2024-02-12T16:57:23.773+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:24.049 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:23 smithi169 bash[28391]: debug 2024-02-12T16:57:23.773+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:24.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[33498]: debug 2024-02-12T16:57:24.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:24.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[33498]: debug 2024-02-12T16:57:24.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:24.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[33498]: debug 2024-02-12T16:57:24.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:24.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[33498]: debug 2024-02-12T16:57:24.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:24.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[33498]: debug 2024-02-12T16:57:24.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:25.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[38680]: debug 2024-02-12T16:57:24.741+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:25.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[38680]: debug 2024-02-12T16:57:24.741+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:25.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[38680]: debug 2024-02-12T16:57:24.741+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[38680]: debug 2024-02-12T16:57:24.741+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[38680]: debug 2024-02-12T16:57:24.741+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[28391]: debug 2024-02-12T16:57:24.761+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[28391]: debug 2024-02-12T16:57:24.761+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[28391]: debug 2024-02-12T16:57:24.761+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[28391]: debug 2024-02-12T16:57:24.761+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[28391]: debug 2024-02-12T16:57:24.761+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:25.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:24 smithi169 bash[28391]: debug 2024-02-12T16:57:24.761+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:25.539 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[33498]: debug 2024-02-12T16:57:25.069+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:25.539 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[33498]: debug 2024-02-12T16:57:25.069+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:25.540 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[33498]: debug 2024-02-12T16:57:25.069+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:25.540 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[33498]: debug 2024-02-12T16:57:25.069+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:25.540 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[33498]: debug 2024-02-12T16:57:25.069+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:25.797 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[136007]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-8 2024-02-12T16:57:25.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[38680]: debug 2024-02-12T16:57:25.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:25.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[38680]: debug 2024-02-12T16:57:25.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:25.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[38680]: debug 2024-02-12T16:57:25.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:25.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[38680]: debug 2024-02-12T16:57:25.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:25.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[38680]: debug 2024-02-12T16:57:25.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:26.047 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[28391]: debug 2024-02-12T16:57:25.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:26.047 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[28391]: debug 2024-02-12T16:57:25.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:26.047 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[28391]: debug 2024-02-12T16:57:25.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:26.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[28391]: debug 2024-02-12T16:57:25.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:26.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[28391]: debug 2024-02-12T16:57:25.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:26.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:25 smithi169 bash[28391]: debug 2024-02-12T16:57:25.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:26.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[33498]: debug 2024-02-12T16:57:26.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:26.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[33498]: debug 2024-02-12T16:57:26.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:26.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[33498]: debug 2024-02-12T16:57:26.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:26.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[33498]: debug 2024-02-12T16:57:26.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:26.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[33498]: debug 2024-02-12T16:57:26.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:27.047 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[28391]: debug 2024-02-12T16:57:26.813+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:27.047 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[28391]: debug 2024-02-12T16:57:26.813+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[28391]: debug 2024-02-12T16:57:26.813+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[28391]: debug 2024-02-12T16:57:26.813+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[28391]: debug 2024-02-12T16:57:26.813+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[28391]: debug 2024-02-12T16:57:26.813+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[38680]: debug 2024-02-12T16:57:26.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[38680]: debug 2024-02-12T16:57:26.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[38680]: debug 2024-02-12T16:57:26.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[38680]: debug 2024-02-12T16:57:26.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:27.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:26 smithi169 bash[38680]: debug 2024-02-12T16:57:26.765+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:27.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[33498]: debug 2024-02-12T16:57:27.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:27.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[33498]: debug 2024-02-12T16:57:27.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:27.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[33498]: debug 2024-02-12T16:57:27.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:27.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[33498]: debug 2024-02-12T16:57:27.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:27.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[33498]: debug 2024-02-12T16:57:27.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:28.095 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[38680]: debug 2024-02-12T16:57:27.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:28.095 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[38680]: debug 2024-02-12T16:57:27.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[38680]: debug 2024-02-12T16:57:27.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[38680]: debug 2024-02-12T16:57:27.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[38680]: debug 2024-02-12T16:57:27.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[28391]: debug 2024-02-12T16:57:27.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[28391]: debug 2024-02-12T16:57:27.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[28391]: debug 2024-02-12T16:57:27.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[28391]: debug 2024-02-12T16:57:27.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[28391]: debug 2024-02-12T16:57:27.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:28.096 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:27 smithi169 bash[28391]: debug 2024-02-12T16:57:27.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:28.353 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[33498]: debug 2024-02-12T16:57:28.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:28.353 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[33498]: debug 2024-02-12T16:57:28.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:28.353 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[33498]: debug 2024-02-12T16:57:28.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:28.353 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[33498]: debug 2024-02-12T16:57:28.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:28.353 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[33498]: debug 2024-02-12T16:57:28.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:29.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.8.service' 2024-02-12T16:57:29.078 INFO:journalctl@ceph.osd.8.smithi169.stdout:Feb 12 16:57:29 smithi169 systemd[1]: Stopped Ceph osd.8 for eb2349b2-c9bd-11ee-95b9-87774f69a715. 2024-02-12T16:57:29.078 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[28391]: debug 2024-02-12T16:57:28.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:29.078 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[28391]: debug 2024-02-12T16:57:28.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:29.078 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[28391]: debug 2024-02-12T16:57:28.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:29.078 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[28391]: debug 2024-02-12T16:57:28.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:29.078 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[28391]: debug 2024-02-12T16:57:28.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:29.078 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[28391]: debug 2024-02-12T16:57:28.793+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:29.079 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[38680]: debug 2024-02-12T16:57:28.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:29.079 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[38680]: debug 2024-02-12T16:57:28.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:29.079 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[38680]: debug 2024-02-12T16:57:28.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:29.079 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[38680]: debug 2024-02-12T16:57:28.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:29.079 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:28 smithi169 bash[38680]: debug 2024-02-12T16:57:28.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:29.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:57:29.091 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-02-12T16:57:29.091 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-02-12T16:57:29.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.9 2024-02-12T16:57:29.344 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 systemd[1]: Stopping Ceph osd.9 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:57:29.344 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[136271]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.9 2024-02-12T16:57:29.344 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[33498]: debug 2024-02-12T16:57:29.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:29.344 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[33498]: debug 2024-02-12T16:57:29.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:29.345 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[33498]: debug 2024-02-12T16:57:29.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:29.345 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[33498]: debug 2024-02-12T16:57:29.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:29.345 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[33498]: debug 2024-02-12T16:57:29.077+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:29.797 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.337+0000 7f47f8052700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:57:29.797 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.337+0000 7f47f8052700 -1 osd.9 1145 *** Got signal Terminated *** 2024-02-12T16:57:29.797 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.337+0000 7f47f8052700 -1 osd.9 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:57:30.297 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.829+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.829+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.829+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.829+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.829+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[28391]: debug 2024-02-12T16:57:29.829+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[33498]: debug 2024-02-12T16:57:30.029+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[33498]: debug 2024-02-12T16:57:30.029+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[33498]: debug 2024-02-12T16:57:30.029+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[33498]: debug 2024-02-12T16:57:30.029+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:30.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[33498]: debug 2024-02-12T16:57:30.029+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:30.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[38680]: debug 2024-02-12T16:57:29.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:30.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[38680]: debug 2024-02-12T16:57:29.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:30.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[38680]: debug 2024-02-12T16:57:29.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:30.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[38680]: debug 2024-02-12T16:57:29.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:30.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:29 smithi169 bash[38680]: debug 2024-02-12T16:57:29.829+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:31.077 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[28391]: debug 2024-02-12T16:57:30.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:31.077 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[28391]: debug 2024-02-12T16:57:30.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:31.077 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[28391]: debug 2024-02-12T16:57:30.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:31.077 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[28391]: debug 2024-02-12T16:57:30.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:31.077 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[28391]: debug 2024-02-12T16:57:30.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:31.077 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[28391]: debug 2024-02-12T16:57:30.809+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:31.078 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[38680]: debug 2024-02-12T16:57:30.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:31.078 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[38680]: debug 2024-02-12T16:57:30.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:31.078 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[38680]: debug 2024-02-12T16:57:30.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:31.078 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[38680]: debug 2024-02-12T16:57:30.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:31.078 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[38680]: debug 2024-02-12T16:57:30.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:31.078 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:30 smithi169 bash[38680]: debug 2024-02-12T16:57:30.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:31.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[33498]: debug 2024-02-12T16:57:31.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:31.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[33498]: debug 2024-02-12T16:57:31.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:31.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[33498]: debug 2024-02-12T16:57:31.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:31.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[33498]: debug 2024-02-12T16:57:31.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:31.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[33498]: debug 2024-02-12T16:57:31.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:32.114 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[28391]: debug 2024-02-12T16:57:31.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[28391]: debug 2024-02-12T16:57:31.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[28391]: debug 2024-02-12T16:57:31.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[28391]: debug 2024-02-12T16:57:31.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[28391]: debug 2024-02-12T16:57:31.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[28391]: debug 2024-02-12T16:57:31.845+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[38680]: debug 2024-02-12T16:57:31.861+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[38680]: debug 2024-02-12T16:57:31.861+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[38680]: debug 2024-02-12T16:57:31.861+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[38680]: debug 2024-02-12T16:57:31.861+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[38680]: debug 2024-02-12T16:57:31.861+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:32.115 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:31 smithi169 bash[38680]: debug 2024-02-12T16:57:31.861+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:32.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[33498]: debug 2024-02-12T16:57:32.105+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:32.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[33498]: debug 2024-02-12T16:57:32.105+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:32.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[33498]: debug 2024-02-12T16:57:32.105+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:32.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[33498]: debug 2024-02-12T16:57:32.105+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:32.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[33498]: debug 2024-02-12T16:57:32.105+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[28391]: debug 2024-02-12T16:57:32.801+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[28391]: debug 2024-02-12T16:57:32.801+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[28391]: debug 2024-02-12T16:57:32.801+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[28391]: debug 2024-02-12T16:57:32.801+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[28391]: debug 2024-02-12T16:57:32.801+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[28391]: debug 2024-02-12T16:57:32.801+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[38680]: debug 2024-02-12T16:57:32.857+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[38680]: debug 2024-02-12T16:57:32.857+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[38680]: debug 2024-02-12T16:57:32.857+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:33.089 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[38680]: debug 2024-02-12T16:57:32.857+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:33.090 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[38680]: debug 2024-02-12T16:57:32.857+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:33.090 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:32 smithi169 bash[38680]: debug 2024-02-12T16:57:32.857+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:33.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[33498]: debug 2024-02-12T16:57:33.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:33.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[33498]: debug 2024-02-12T16:57:33.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:33.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[33498]: debug 2024-02-12T16:57:33.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:33.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[33498]: debug 2024-02-12T16:57:33.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:33.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[33498]: debug 2024-02-12T16:57:33.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:33.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[33498]: debug 2024-02-12T16:57:33.085+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:34.129 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[38680]: debug 2024-02-12T16:57:33.881+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:34.129 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[38680]: debug 2024-02-12T16:57:33.881+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:34.129 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[38680]: debug 2024-02-12T16:57:33.881+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:34.129 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[38680]: debug 2024-02-12T16:57:33.881+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:34.129 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[38680]: debug 2024-02-12T16:57:33.881+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:34.129 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[38680]: debug 2024-02-12T16:57:33.881+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:34.130 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[28391]: debug 2024-02-12T16:57:33.797+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.565850+0000 front 2024-02-12T16:56:17.565697+0000 (oldest deadline 2024-02-12T16:56:41.665915+0000) 2024-02-12T16:57:34.130 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[28391]: debug 2024-02-12T16:57:33.797+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:24.469264+0000 front 2024-02-12T16:56:24.468869+0000 (oldest deadline 2024-02-12T16:56:47.969004+0000) 2024-02-12T16:57:34.130 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[28391]: debug 2024-02-12T16:57:33.797+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.470766+0000 front 2024-02-12T16:56:31.470542+0000 (oldest deadline 2024-02-12T16:56:56.170505+0000) 2024-02-12T16:57:34.130 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[28391]: debug 2024-02-12T16:57:33.797+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:41.472240+0000 front 2024-02-12T16:56:41.472371+0000 (oldest deadline 2024-02-12T16:57:04.972035+0000) 2024-02-12T16:57:34.130 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[28391]: debug 2024-02-12T16:57:33.797+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:50.874014+0000 front 2024-02-12T16:56:50.874194+0000 (oldest deadline 2024-02-12T16:57:11.973990+0000) 2024-02-12T16:57:34.130 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[28391]: debug 2024-02-12T16:57:33.797+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:56.075864+0000 front 2024-02-12T16:56:56.075753+0000 (oldest deadline 2024-02-12T16:57:21.375611+0000) 2024-02-12T16:57:34.130 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:33 smithi169 bash[28391]: debug 2024-02-12T16:57:33.797+0000 7f47f1fc9700 -1 osd.9 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:07.277376+0000 front 2024-02-12T16:57:07.277485+0000 (oldest deadline 2024-02-12T16:57:33.177324+0000) 2024-02-12T16:57:34.520 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[33498]: debug 2024-02-12T16:57:34.125+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:34.520 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[33498]: debug 2024-02-12T16:57:34.125+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:34.520 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[33498]: debug 2024-02-12T16:57:34.125+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:34.520 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[33498]: debug 2024-02-12T16:57:34.125+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:34.520 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[33498]: debug 2024-02-12T16:57:34.125+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:34.520 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[33498]: debug 2024-02-12T16:57:34.125+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:35.292 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[33498]: debug 2024-02-12T16:57:35.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:35.292 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[33498]: debug 2024-02-12T16:57:35.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:35.292 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[33498]: debug 2024-02-12T16:57:35.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:35.292 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[33498]: debug 2024-02-12T16:57:35.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:35.292 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[33498]: debug 2024-02-12T16:57:35.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:35.292 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[33498]: debug 2024-02-12T16:57:35.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:35.292 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[38680]: debug 2024-02-12T16:57:34.921+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:35.293 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[38680]: debug 2024-02-12T16:57:34.921+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:35.293 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[38680]: debug 2024-02-12T16:57:34.921+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:35.293 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[38680]: debug 2024-02-12T16:57:34.921+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:35.293 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[38680]: debug 2024-02-12T16:57:34.921+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:35.293 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:34 smithi169 bash[38680]: debug 2024-02-12T16:57:34.921+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:35.547 INFO:journalctl@ceph.osd.9.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[136271]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-9 2024-02-12T16:57:36.197 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[38680]: debug 2024-02-12T16:57:35.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:36.197 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[38680]: debug 2024-02-12T16:57:35.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:36.197 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[38680]: debug 2024-02-12T16:57:35.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:36.197 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[38680]: debug 2024-02-12T16:57:35.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:36.197 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[38680]: debug 2024-02-12T16:57:35.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:36.197 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:35 smithi169 bash[38680]: debug 2024-02-12T16:57:35.893+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:36.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[33498]: debug 2024-02-12T16:57:36.193+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:36.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[33498]: debug 2024-02-12T16:57:36.193+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:36.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[33498]: debug 2024-02-12T16:57:36.193+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:36.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[33498]: debug 2024-02-12T16:57:36.193+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:36.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[33498]: debug 2024-02-12T16:57:36.193+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:36.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[33498]: debug 2024-02-12T16:57:36.193+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:37.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[38680]: debug 2024-02-12T16:57:36.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:37.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[38680]: debug 2024-02-12T16:57:36.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:37.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[38680]: debug 2024-02-12T16:57:36.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:37.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[38680]: debug 2024-02-12T16:57:36.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:37.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[38680]: debug 2024-02-12T16:57:36.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:37.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:36 smithi169 bash[38680]: debug 2024-02-12T16:57:36.937+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:37.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[33498]: debug 2024-02-12T16:57:37.157+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:37.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[33498]: debug 2024-02-12T16:57:37.157+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:37.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[33498]: debug 2024-02-12T16:57:37.157+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:37.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[33498]: debug 2024-02-12T16:57:37.157+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:37.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[33498]: debug 2024-02-12T16:57:37.157+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:37.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[33498]: debug 2024-02-12T16:57:37.157+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:37.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.9.service' 2024-02-12T16:57:37.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:57:37.793 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-02-12T16:57:37.794 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-02-12T16:57:37.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.10 2024-02-12T16:57:38.039 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[38680]: debug 2024-02-12T16:57:37.909+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:38.039 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[38680]: debug 2024-02-12T16:57:37.909+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:38.040 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[38680]: debug 2024-02-12T16:57:37.909+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:38.040 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[38680]: debug 2024-02-12T16:57:37.909+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:38.040 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[38680]: debug 2024-02-12T16:57:37.909+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:38.040 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[38680]: debug 2024-02-12T16:57:37.909+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:38.040 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 systemd[1]: Stopping Ceph osd.10 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:57:38.040 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:37 smithi169 bash[136535]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.10 2024-02-12T16:57:38.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.033+0000 7fdbed8d8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:57:38.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.033+0000 7fdbed8d8700 -1 osd.10 1145 *** Got signal Terminated *** 2024-02-12T16:57:38.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.033+0000 7fdbed8d8700 -1 osd.10 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:57:38.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:38.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:38.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:38.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:38.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:38.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[33498]: debug 2024-02-12T16:57:38.113+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:39.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[33498]: debug 2024-02-12T16:57:39.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:39.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[33498]: debug 2024-02-12T16:57:39.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:39.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[33498]: debug 2024-02-12T16:57:39.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:39.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[33498]: debug 2024-02-12T16:57:39.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:39.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[33498]: debug 2024-02-12T16:57:39.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:39.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[33498]: debug 2024-02-12T16:57:39.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:39.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[33498]: debug 2024-02-12T16:57:39.073+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:13.463119+0000 front 2024-02-12T16:57:13.463144+0000 (oldest deadline 2024-02-12T16:57:38.162962+0000) 2024-02-12T16:57:39.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[38680]: debug 2024-02-12T16:57:38.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:39.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[38680]: debug 2024-02-12T16:57:38.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:39.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[38680]: debug 2024-02-12T16:57:38.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:39.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[38680]: debug 2024-02-12T16:57:38.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:39.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[38680]: debug 2024-02-12T16:57:38.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:39.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[38680]: debug 2024-02-12T16:57:38.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:39.299 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:38 smithi169 bash[38680]: debug 2024-02-12T16:57:38.889+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:40.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[38680]: debug 2024-02-12T16:57:39.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[38680]: debug 2024-02-12T16:57:39.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[38680]: debug 2024-02-12T16:57:39.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[38680]: debug 2024-02-12T16:57:39.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[38680]: debug 2024-02-12T16:57:39.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[38680]: debug 2024-02-12T16:57:39.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:39 smithi169 bash[38680]: debug 2024-02-12T16:57:39.865+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[33498]: debug 2024-02-12T16:57:40.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[33498]: debug 2024-02-12T16:57:40.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[33498]: debug 2024-02-12T16:57:40.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[33498]: debug 2024-02-12T16:57:40.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:40.298 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[33498]: debug 2024-02-12T16:57:40.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:40.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[33498]: debug 2024-02-12T16:57:40.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:40.299 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[33498]: debug 2024-02-12T16:57:40.089+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:13.463119+0000 front 2024-02-12T16:57:13.463144+0000 (oldest deadline 2024-02-12T16:57:38.162962+0000) 2024-02-12T16:57:41.136 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[38680]: debug 2024-02-12T16:57:40.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:41.136 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[38680]: debug 2024-02-12T16:57:40.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:41.136 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[38680]: debug 2024-02-12T16:57:40.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:41.136 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[38680]: debug 2024-02-12T16:57:40.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:41.136 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[38680]: debug 2024-02-12T16:57:40.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:41.136 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[38680]: debug 2024-02-12T16:57:40.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:41.136 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:40 smithi169 bash[38680]: debug 2024-02-12T16:57:40.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:41.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[33498]: debug 2024-02-12T16:57:41.129+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:41.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[33498]: debug 2024-02-12T16:57:41.129+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:41.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[33498]: debug 2024-02-12T16:57:41.129+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:41.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[33498]: debug 2024-02-12T16:57:41.129+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:41.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[33498]: debug 2024-02-12T16:57:41.129+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:41.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[33498]: debug 2024-02-12T16:57:41.129+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:41.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[33498]: debug 2024-02-12T16:57:41.129+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:13.463119+0000 front 2024-02-12T16:57:13.463144+0000 (oldest deadline 2024-02-12T16:57:38.162962+0000) 2024-02-12T16:57:42.152 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[38680]: debug 2024-02-12T16:57:41.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:42.152 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[38680]: debug 2024-02-12T16:57:41.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:42.152 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[38680]: debug 2024-02-12T16:57:41.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:42.153 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[38680]: debug 2024-02-12T16:57:41.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:42.153 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[38680]: debug 2024-02-12T16:57:41.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:42.153 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[38680]: debug 2024-02-12T16:57:41.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:42.153 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:41 smithi169 bash[38680]: debug 2024-02-12T16:57:41.825+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:42.547 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[33498]: debug 2024-02-12T16:57:42.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:17.146401+0000 front 2024-02-12T16:56:17.146495+0000 (oldest deadline 2024-02-12T16:56:39.446256+0000) 2024-02-12T16:57:42.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[33498]: debug 2024-02-12T16:57:42.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:31.650752+0000 front 2024-02-12T16:56:31.650718+0000 (oldest deadline 2024-02-12T16:56:55.150862+0000) 2024-02-12T16:57:42.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[33498]: debug 2024-02-12T16:57:42.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:42.153691+0000 front 2024-02-12T16:56:42.153636+0000 (oldest deadline 2024-02-12T16:57:08.053587+0000) 2024-02-12T16:57:42.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[33498]: debug 2024-02-12T16:57:42.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6802 osd.4 since back 2024-02-12T16:56:48.054922+0000 front 2024-02-12T16:56:48.054891+0000 (oldest deadline 2024-02-12T16:57:11.554445+0000) 2024-02-12T16:57:42.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[33498]: debug 2024-02-12T16:57:42.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:57.957015+0000 front 2024-02-12T16:56:57.957216+0000 (oldest deadline 2024-02-12T16:57:20.857001+0000) 2024-02-12T16:57:42.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[33498]: debug 2024-02-12T16:57:42.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:05.360610+0000 front 2024-02-12T16:57:05.360584+0000 (oldest deadline 2024-02-12T16:57:32.361259+0000) 2024-02-12T16:57:42.548 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[33498]: debug 2024-02-12T16:57:42.149+0000 7fdbeb056700 -1 osd.10 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:13.463119+0000 front 2024-02-12T16:57:13.463144+0000 (oldest deadline 2024-02-12T16:57:38.162962+0000) 2024-02-12T16:57:43.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[38680]: debug 2024-02-12T16:57:42.789+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:43.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[38680]: debug 2024-02-12T16:57:42.789+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:43.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[38680]: debug 2024-02-12T16:57:42.789+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:43.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[38680]: debug 2024-02-12T16:57:42.789+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:43.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[38680]: debug 2024-02-12T16:57:42.789+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:43.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[38680]: debug 2024-02-12T16:57:42.789+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:43.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:42 smithi169 bash[38680]: debug 2024-02-12T16:57:42.789+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:44.297 INFO:journalctl@ceph.osd.10.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[136535]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-10 2024-02-12T16:57:44.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[38680]: debug 2024-02-12T16:57:43.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:44.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[38680]: debug 2024-02-12T16:57:43.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:44.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[38680]: debug 2024-02-12T16:57:43.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:44.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[38680]: debug 2024-02-12T16:57:43.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:44.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[38680]: debug 2024-02-12T16:57:43.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:44.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[38680]: debug 2024-02-12T16:57:43.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:44.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:43 smithi169 bash[38680]: debug 2024-02-12T16:57:43.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:45.166 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:44 smithi169 bash[38680]: debug 2024-02-12T16:57:44.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:45.166 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:44 smithi169 bash[38680]: debug 2024-02-12T16:57:44.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:45.166 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:44 smithi169 bash[38680]: debug 2024-02-12T16:57:44.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:45.166 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:44 smithi169 bash[38680]: debug 2024-02-12T16:57:44.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:45.166 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:44 smithi169 bash[38680]: debug 2024-02-12T16:57:44.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:45.166 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:44 smithi169 bash[38680]: debug 2024-02-12T16:57:44.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:45.166 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:44 smithi169 bash[38680]: debug 2024-02-12T16:57:44.837+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:46.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:45 smithi169 bash[38680]: debug 2024-02-12T16:57:45.845+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:46.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:45 smithi169 bash[38680]: debug 2024-02-12T16:57:45.845+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:46.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:45 smithi169 bash[38680]: debug 2024-02-12T16:57:45.845+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:46.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:45 smithi169 bash[38680]: debug 2024-02-12T16:57:45.845+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:46.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:45 smithi169 bash[38680]: debug 2024-02-12T16:57:45.845+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:46.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:45 smithi169 bash[38680]: debug 2024-02-12T16:57:45.845+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:46.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:45 smithi169 bash[38680]: debug 2024-02-12T16:57:45.845+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:46.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.10.service' 2024-02-12T16:57:46.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:57:46.274 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-02-12T16:57:46.274 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-02-12T16:57:46.274 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.11 2024-02-12T16:57:46.515 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 systemd[1]: Stopping Ceph osd.11 for eb2349b2-c9bd-11ee-95b9-87774f69a715... 2024-02-12T16:57:46.515 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[136795]: Error response from daemon: No such container: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd.11 2024-02-12T16:57:46.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.509+0000 7fc964744700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-12T16:57:46.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.509+0000 7fc964744700 -1 osd.11 1145 *** Got signal Terminated *** 2024-02-12T16:57:46.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.509+0000 7fc964744700 -1 osd.11 1145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-12T16:57:47.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:47.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:47.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:47.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:47.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:47.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:47.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:47.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:46 smithi169 bash[38680]: debug 2024-02-12T16:57:46.797+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.169:6802 osd.8 since back 2024-02-12T16:57:22.403314+0000 front 2024-02-12T16:57:22.403472+0000 (oldest deadline 2024-02-12T16:57:45.903188+0000) 2024-02-12T16:57:48.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:48.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:48.297 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:48.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:48.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:48.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:48.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:48.298 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:47 smithi169 bash[38680]: debug 2024-02-12T16:57:47.813+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.169:6802 osd.8 since back 2024-02-12T16:57:22.403314+0000 front 2024-02-12T16:57:22.403472+0000 (oldest deadline 2024-02-12T16:57:45.903188+0000) 2024-02-12T16:57:49.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:49.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:49.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:49.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:49.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:49.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:49.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:49.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:48 smithi169 bash[38680]: debug 2024-02-12T16:57:48.785+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.169:6802 osd.8 since back 2024-02-12T16:57:22.403314+0000 front 2024-02-12T16:57:22.403472+0000 (oldest deadline 2024-02-12T16:57:45.903188+0000) 2024-02-12T16:57:50.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:50.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:50.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:50.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:50.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:50.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:50.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:50.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:49 smithi169 bash[38680]: debug 2024-02-12T16:57:49.777+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.169:6802 osd.8 since back 2024-02-12T16:57:22.403314+0000 front 2024-02-12T16:57:22.403472+0000 (oldest deadline 2024-02-12T16:57:45.903188+0000) 2024-02-12T16:57:51.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6803 osd.0 since back 2024-02-12T16:56:12.787863+0000 front 2024-02-12T16:56:12.787739+0000 (oldest deadline 2024-02-12T16:56:42.188280+0000) 2024-02-12T16:57:51.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6807 osd.1 since back 2024-02-12T16:56:25.690363+0000 front 2024-02-12T16:56:25.690317+0000 (oldest deadline 2024-02-12T16:56:50.990006+0000) 2024-02-12T16:57:51.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6811 osd.2 since back 2024-02-12T16:56:30.990808+0000 front 2024-02-12T16:56:30.990845+0000 (oldest deadline 2024-02-12T16:56:55.090925+0000) 2024-02-12T16:57:51.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.121:6815 osd.3 since back 2024-02-12T16:56:40.893627+0000 front 2024-02-12T16:56:40.893781+0000 (oldest deadline 2024-02-12T16:57:05.593667+0000) 2024-02-12T16:57:51.047 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6806 osd.5 since back 2024-02-12T16:56:55.596225+0000 front 2024-02-12T16:56:55.596280+0000 (oldest deadline 2024-02-12T16:57:20.296300+0000) 2024-02-12T16:57:51.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6810 osd.6 since back 2024-02-12T16:57:04.799774+0000 front 2024-02-12T16:57:04.799990+0000 (oldest deadline 2024-02-12T16:57:30.699757+0000) 2024-02-12T16:57:51.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.123:6814 osd.7 since back 2024-02-12T16:57:14.201735+0000 front 2024-02-12T16:57:14.201764+0000 (oldest deadline 2024-02-12T16:57:38.301556+0000) 2024-02-12T16:57:51.048 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:50 smithi169 bash[38680]: debug 2024-02-12T16:57:50.793+0000 7fc95eebc700 -1 osd.11 1145 heartbeat_check: no reply from 172.21.15.169:6802 osd.8 since back 2024-02-12T16:57:22.403314+0000 front 2024-02-12T16:57:22.403472+0000 (oldest deadline 2024-02-12T16:57:45.903188+0000) 2024-02-12T16:57:52.797 INFO:journalctl@ceph.osd.11.smithi169.stdout:Feb 12 16:57:52 smithi169 bash[136795]: ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715-osd-11 2024-02-12T16:57:54.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb2349b2-c9bd-11ee-95b9-87774f69a715@osd.11.service' 2024-02-12T16:57:54.532 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-12T16:57:54.532 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-02-12T16:57:54.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-12T16:57:57.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-12T16:58:00.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-12T16:58:02.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:58:02.838 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:58:02.854 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-12T16:58:02.869 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-12T16:58:02.871 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/ubuntu@smithi121.front.sepia.ceph.com/crash 2024-02-12T16:58:02.872 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash -- . 2024-02-12T16:58:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-12T16:58:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2024-02-12T16:58:02.914 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/ubuntu@smithi123.front.sepia.ceph.com/crash 2024-02-12T16:58:02.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash -- . 2024-02-12T16:58:03.044 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-12T16:58:03.044 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-02-12T16:58:03.046 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/ubuntu@smithi169.front.sepia.ceph.com/crash 2024-02-12T16:58:03.047 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash -- . 2024-02-12T16:58:03.074 INFO:teuthology.orchestra.run.smithi169.stderr:tar: /var/lib/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-12T16:58:03.074 INFO:teuthology.orchestra.run.smithi169.stderr:tar: Error is not recoverable: exiting now 2024-02-12T16:58:03.075 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-12T16:58:03.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(MON_DOWN\)' | egrep -v 'osds down' | egrep -v 'mons down' | head -n 1 2024-02-12T16:58:03.211 INFO:teuthology.orchestra.run.smithi121.stdout:2024-02-12T16:10:00.000101+0000 mon.a (mon.0) 1168 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:58:03.212 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-12T16:58:03.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[SEC\]' /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(MON_DOWN\)' | egrep -v 'osds down' | egrep -v 'mons down' | head -n 1 2024-02-12T16:58:03.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]' /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(MON_DOWN\)' | egrep -v 'osds down' | egrep -v 'mons down' | head -n 1 2024-02-12T16:58:03.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[WRN\]' /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(MON_DOWN\)' | egrep -v 'osds down' | egrep -v 'mons down' | head -n 1 2024-02-12T16:58:03.355 INFO:teuthology.orchestra.run.smithi121.stdout:2024-02-12T16:10:00.000101+0000 mon.a (mon.0) 1168 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-12T16:58:03.355 INFO:tasks.cephadm:Compressing logs... 2024-02-12T16:58:03.356 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-12T16:58:03.398 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-12T16:58:03.400 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-12T16:58:03.431 INFO:teuthology.orchestra.run.smithi169.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-02-12T16:58:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-02-12T16:58:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-12T16:58:03.441 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-12T16:58:03.448 INFO:teuthology.orchestra.run.smithi123.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-02-12T16:58:03.448 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-12T16:58:03.449 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-12T16:58:03.449 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.8.log 2024-02-12T16:58:03.450 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.7.log 2024-02-12T16:58:03.450 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.8.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.0.log 2024-02-12T16:58:03.451 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.a.log 2024-02-12T16:58:03.452 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-12T16:58:03.452 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mgr.y.log 2024-02-12T16:58:03.453 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.1.log 2024-02-12T16:58:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-12T16:58:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.2.log 2024-02-12T16:58:03.455 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-12T16:58:03.455 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.6.log 2024-02-12T16:58:03.455 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-12T16:58:03.457 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.3.log 2024-02-12T16:58:03.457 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log 2024-02-12T16:58:03.459 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-12T16:58:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.11.log 2024-02-12T16:58:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.9.log 2024-02-12T16:58:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-12T16:58:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.10.log 2024-02-12T16:58:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log: --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log 2024-02-12T16:58:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.c.log 2024-02-12T16:58:03.460 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.8.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.8.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.7.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.8.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.b.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.7.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.5.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mgr.x.log 2024-02-12T16:58:03.461 INFO:teuthology.orchestra.run.smithi123.stderr: 90.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-12T16:58:03.462 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.4.log 2024-02-12T16:58:03.462 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.c.log: /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.8.log: 90.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-12T16:58:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log 2024-02-12T16:58:03.470 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.3.log: /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-12T16:58:03.473 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mgr.x.log: /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.4.log: /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log: 82.5% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-12T16:58:03.476 INFO:teuthology.orchestra.run.smithi121.stderr: 86.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-12T16:58:03.478 INFO:teuthology.orchestra.run.smithi121.stderr: 81.9% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-12T16:58:03.487 INFO:teuthology.orchestra.run.smithi169.stderr: 82.5% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-12T16:58:03.512 INFO:teuthology.orchestra.run.smithi169.stderr: 91.9% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-12T16:58:03.515 INFO:teuthology.orchestra.run.smithi123.stderr: 91.9% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-12T16:58:03.535 INFO:teuthology.orchestra.run.smithi121.stderr: 92.0% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-12T16:58:03.541 INFO:teuthology.orchestra.run.smithi123.stderr: 95.5% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.8.log.gz 2024-02-12T16:58:03.559 INFO:teuthology.orchestra.run.smithi121.stderr: 95.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.8.log.gz 2024-02-12T16:58:03.560 INFO:teuthology.orchestra.run.smithi123.stderr: 95.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.7.log.gz 2024-02-12T16:58:03.567 INFO:teuthology.orchestra.run.smithi121.stderr: 95.5% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/objectstore_tool.7.log.gz 2024-02-12T16:58:03.582 INFO:teuthology.orchestra.run.smithi123.stderr: 93.6% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mgr.x.log.gz 2024-02-12T16:58:03.588 INFO:teuthology.orchestra.run.smithi123.stderr: 90.3% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-12T16:58:03.616 INFO:teuthology.orchestra.run.smithi123.stderr: 86.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-12T16:58:03.803 INFO:teuthology.orchestra.run.smithi169.stderr: 90.3% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-12T16:58:03.805 INFO:teuthology.orchestra.run.smithi169.stderr: 86.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-12T16:58:04.215 INFO:teuthology.orchestra.run.smithi121.stderr: 90.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-12T16:58:04.925 INFO:teuthology.orchestra.run.smithi121.stderr: 90.8% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mgr.y.log.gz 2024-02-12T16:58:05.417 INFO:teuthology.orchestra.run.smithi169.stderr: 91.3% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.c.log.gz 2024-02-12T16:58:05.517 INFO:teuthology.orchestra.run.smithi123.stderr: 91.1% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.b.log.gz 2024-02-12T16:58:12.354 INFO:teuthology.orchestra.run.smithi121.stderr: 89.9% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-mon.a.log.gz 2024-02-12T16:59:19.114 INFO:teuthology.orchestra.run.smithi169.stderr: 93.8% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.10.log.gz 2024-02-12T16:59:26.368 INFO:teuthology.orchestra.run.smithi169.stderr: 94.0% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.8.log.gz 2024-02-12T16:59:29.930 INFO:teuthology.orchestra.run.smithi169.stderr: 94.1% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.9.log.gz 2024-02-12T16:59:50.778 INFO:teuthology.orchestra.run.smithi123.stderr: 94.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.6.log.gz 2024-02-12T17:00:10.038 INFO:teuthology.orchestra.run.smithi123.stderr: 94.0% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.4.log.gz 2024-02-12T17:00:33.197 INFO:teuthology.orchestra.run.smithi123.stderr: 93.3% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.7.log.gz 2024-02-12T17:00:35.035 INFO:teuthology.orchestra.run.smithi123.stderr: 94.0% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.5.log.gz 2024-02-12T17:00:35.037 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-02-12T17:00:35.037 INFO:teuthology.orchestra.run.smithi123.stderr:real 2m31.634s 2024-02-12T17:00:35.037 INFO:teuthology.orchestra.run.smithi123.stderr:user 4m7.037s 2024-02-12T17:00:35.037 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m10.427s 2024-02-12T17:00:46.314 INFO:teuthology.orchestra.run.smithi169.stderr: 92.7% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.11.log.gz 2024-02-12T17:00:46.315 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-02-12T17:00:46.315 INFO:teuthology.orchestra.run.smithi169.stderr:real 2m42.910s 2024-02-12T17:00:46.315 INFO:teuthology.orchestra.run.smithi169.stderr:user 4m4.594s 2024-02-12T17:00:46.315 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m9.951s 2024-02-12T17:02:05.922 INFO:teuthology.orchestra.run.smithi121.stderr: 93.2% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.2.log.gz 2024-02-12T17:02:56.223 INFO:teuthology.orchestra.run.smithi121.stderr: 93.7% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.0.log.gz 2024-02-12T17:03:33.075 INFO:teuthology.orchestra.run.smithi121.stderr: 92.8% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.3.log.gz 2024-02-12T17:04:20.800 INFO:teuthology.orchestra.run.smithi121.stderr: 92.5% -- replaced with /var/log/ceph/eb2349b2-c9bd-11ee-95b9-87774f69a715/ceph-osd.1.log.gz 2024-02-12T17:04:20.801 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-12T17:04:20.801 INFO:teuthology.orchestra.run.smithi121.stderr:real 6m17.400s 2024-02-12T17:04:20.801 INFO:teuthology.orchestra.run.smithi121.stderr:user 6m10.507s 2024-02-12T17:04:20.801 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m16.582s 2024-02-12T17:04:20.802 INFO:tasks.cephadm:Archiving logs... 2024-02-12T17:04:20.803 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/ubuntu@smithi121.front.sepia.ceph.com/log 2024-02-12T17:04:20.804 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-12T17:05:21.922 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/ubuntu@smithi123.front.sepia.ceph.com/log 2024-02-12T17:05:21.923 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-12T17:05:59.460 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/ubuntu@smithi169.front.sepia.ceph.com/log 2024-02-12T17:05:59.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-12T17:06:38.257 INFO:tasks.cephadm:Removing cluster... 2024-02-12T17:06:38.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 --force 2024-02-12T17:06:40.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 --force 2024-02-12T17:06:42.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb2349b2-c9bd-11ee-95b9-87774f69a715 --force 2024-02-12T17:06:44.718 INFO:tasks.cephadm:Removing cephadm ... 2024-02-12T17:06:44.718 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-12T17:06:44.726 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-12T17:06:44.825 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-12T17:06:44.844 INFO:tasks.cephadm:Teardown complete 2024-02-12T17:06:44.845 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-12T17:06:44.854 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-12T17:06:44.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-12T17:06:44.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-12T17:06:44.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-12T17:06:44.978 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-12T17:06:44.987 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-12T17:06:44.987 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T17:06:44.989 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T17:06:44.992 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-12T17:06:45.038 INFO:teuthology.orchestra.run.smithi169.stdout: remote refid st t when poll reach delay offset jitter 2024-02-12T17:06:45.038 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================== 2024-02-12T17:06:45.039 INFO:teuthology.orchestra.run.smithi169.stdout:*hv01.front.sepi 44.4.53.2 2 u 67 128 377 0.159 2.994 0.358 2024-02-12T17:06:45.039 INFO:teuthology.orchestra.run.smithi169.stdout:+hv02.front.sepi 63.231.80.2 3 u 29 128 377 0.118 0.966 0.407 2024-02-12T17:06:45.039 INFO:teuthology.orchestra.run.smithi169.stdout:+hv03.front.sepi 74.6.168.72 3 u 29 128 377 0.138 0.457 0.297 2024-02-12T17:06:45.039 INFO:teuthology.orchestra.run.smithi169.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-02-12T17:06:45.045 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2024-02-12T17:06:45.046 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2024-02-12T17:06:45.046 INFO:teuthology.orchestra.run.smithi121.stdout:*hv01.front.sepi 44.4.53.2 2 u 147 256 377 0.180 6.447 0.710 2024-02-12T17:06:45.046 INFO:teuthology.orchestra.run.smithi121.stdout:+hv02.front.sepi 63.231.80.2 3 u 98 256 377 0.157 4.753 0.686 2024-02-12T17:06:45.046 INFO:teuthology.orchestra.run.smithi121.stdout:+hv03.front.sepi 74.6.168.72 3 u 139 256 377 0.146 4.894 1.356 2024-02-12T17:06:45.046 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-02-12T17:06:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2024-02-12T17:06:45.066 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2024-02-12T17:06:45.066 INFO:teuthology.orchestra.run.smithi123.stdout:*hv01.front.sepi 44.4.53.2 2 u 70 256 377 0.166 1.592 0.316 2024-02-12T17:06:45.066 INFO:teuthology.orchestra.run.smithi123.stdout:+hv02.front.sepi 63.231.80.2 3 u 128 128 377 0.125 0.676 0.832 2024-02-12T17:06:45.066 INFO:teuthology.orchestra.run.smithi123.stdout:+hv03.front.sepi 74.6.168.72 3 u 10 256 377 0.137 -0.467 0.097 2024-02-12T17:06:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-02-12T17:06:45.067 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-12T17:06:45.076 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-12T17:06:45.076 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-12T17:06:45.101 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-12T17:06:45.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-12T17:06:45.168 INFO:teuthology.task.internal:Duration was 5225.724582 seconds 2024-02-12T17:06:45.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-12T17:06:45.201 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-12T17:06:45.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-12T17:06:45.204 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-12T17:06:45.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-12T17:06:45.346 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-12T17:06:45.347 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-02-12T17:06:45.347 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-12T17:06:45.370 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-02-12T17:06:45.370 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-12T17:06:45.440 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-02-12T17:06:45.440 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-12T17:06:45.499 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-12T17:06:45.499 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-12T17:06:45.501 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-12T17:06:45.504 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-12T17:06:45.676 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-12T17:06:45.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-12T17:06:45.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-12T17:06:45.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-12T17:06:46.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-12T17:06:46.098 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-12T17:06:46.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-12T17:06:46.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-12T17:06:46.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-12T17:06:46.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-12T17:06:46.190 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:06:46.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:06:46.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:06:46.210 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-02-12T17:06:46.228 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-02-12T17:06:46.232 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-02-12T17:06:46.266 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:06:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T17:06:46.281 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:06:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T17:06:46.317 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-12T17:06:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-12T17:06:46.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-12T17:06:46.331 INFO:teuthology.task.internal:Transferring archived files... 2024-02-12T17:06:46.332 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/smithi121 2024-02-12T17:06:46.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-12T17:06:46.379 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/smithi123 2024-02-12T17:06:46.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-12T17:06:46.419 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633/remote/smithi169 2024-02-12T17:06:46.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-12T17:06:46.459 INFO:teuthology.task.internal:Removing archive directory... 2024-02-12T17:06:46.460 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-12T17:06:46.462 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-12T17:06:46.464 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-12T17:06:46.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-12T17:06:46.509 INFO:teuthology.task.internal:Not uploading archives. 2024-02-12T17:06:46.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-12T17:06:46.568 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-12T17:06:46.569 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-12T17:06:46.571 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-12T17:06:46.573 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-12T17:06:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: 659550 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 17:06 /home/ubuntu/cephtest 2024-02-12T17:06:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: 528234 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 17:06 /home/ubuntu/cephtest 2024-02-12T17:06:46.581 INFO:teuthology.orchestra.run.smithi169.stdout: 659551 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 17:06 /home/ubuntu/cephtest 2024-02-12T17:06:46.583 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-12T17:06:46.602 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-12T17:06:46.669 INFO:teuthology.nuke:Checking targets against current locks 2024-02-12T17:06:46.695 DEBUG:teuthology.nuke:shortname: smithi121 2024-02-12T17:06:46.696 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T17:06:46.717 DEBUG:teuthology.nuke:shortname: smithi123 2024-02-12T17:06:46.717 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T17:06:46.721 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 15:29:11.215556', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE='} 2024-02-12T17:06:46.735 DEBUG:teuthology.nuke:shortname: smithi169 2024-02-12T17:06:46.736 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-12T17:06:46.739 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 15:29:11.214572', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE='} 2024-02-12T17:06:46.761 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-12_15:26:58-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7556633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '18.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-12 15:29:11.213430', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE='} 2024-02-12T17:06:46.770 INFO:teuthology.orchestra.console.smithi121:Power off 2024-02-12T17:06:46.771 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-12T17:06:46.794 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-02-12T17:06:46.794 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:46.807 INFO:teuthology.orchestra.console.smithi123:Power off 2024-02-12T17:06:46.808 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-12T17:06:46.821 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-12T17:06:46.829 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-02-12T17:06:46.829 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:46.845 INFO:teuthology.orchestra.console.smithi169:Power off 2024-02-12T17:06:46.846 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-12T17:06:46.857 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-02-12T17:06:46.870 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-02-12T17:06:46.870 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:46.890 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-02-12T17:06:50.822 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:50.858 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:50.891 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:50.932 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-12T17:06:50.968 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-02-12T17:06:51.000 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-02-12T17:06:54.937 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:54.969 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:55.001 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:55.050 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-02-12T17:06:55.078 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-02-12T17:06:55.110 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-02-12T17:06:55.151 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-02-12T17:06:55.179 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-02-12T17:06:59.115 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-12T17:06:59.227 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-02-12T17:06:59.328 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-02-12T17:06:59.433 INFO:teuthology.run:Summary data: description: rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} duration: 5225.724582433701 failure_reason: '"2024-02-12T16:10:00.000101+0000 mon.a (mon.0) 1168 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-02-12T17:06:59.434 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-12T17:06:59.515 INFO:teuthology.run:FAIL